[PATCH 0/2] UBIFS: Free space fixup on first mount

[PATCH 0/2] UBIFS: Free space fixup on first mount

[Matthew L. Creech]

Hi,

The following changes add a superblock option to UBIFS indicating that
it needs to be scanned for LEBs with empty pages the first time it's
mounted.  This is needed because images programmed using "nandwrite"
may contain empty pages, which UBIFS intends to be erased but which
may be programmed as real data.  This causes problems on some NAND
flashes, as described here:

http://www.linux-mtd.infradead.org/faq/ubifs.html#L_why_ubiformat

When this flag is set, the FS is scanned for any in-use LEBs with one
or more free pages, and these LEBs are remapped (which erases the
blank pages).


Changes since v3 (Artem):
- Free buffer used for reading superblock
- Remove unnecessary buffer-length alignment
- Fixup the log area

[PATCH 1/2] UBIFS: add the fixup function

[Matthew L. Creech]

This patch adds the 'ubifs_fixup_free_space()' function which scans all
LEBs in the filesystem for those that are in-use but have one or more
empty pages, then re-maps the LEBs in order to erase the empty portions.
Afterward it removes the "space_fixup" flag from the UBIFS superblock.

Signed-off-by: Matthew L. Creech <mlcreech@gmail.com>
---
 fs/ubifs/sb.c    |  152 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 fs/ubifs/ubifs.h |    1 +
 2 files changed, 153 insertions(+), 0 deletions(-)

diff --git a/fs/ubifs/sb.c b/fs/ubifs/sb.c
index 93d6928..3f61a5d 100644
--- a/fs/ubifs/sb.c
+++ b/fs/ubifs/sb.c
@@ -652,3 +652,155 @@ out:
 	kfree(sup);
 	return err;
 }
+
+/**
+ * fixup_leb_free_space - fixup/unmap an LEB containing free space.
+ * @c: UBIFS file-system description object
+ * @lnum: LEB number
+ * @len: number of used bytes in LEB (starting at offset 0)
+ *
+ * This function reads the contents of the given LEB number @lnum, then fixes
+ * it up, so that any empty min. I/O units are actually erased on flash (rather
+ * than being just all-0xff real data). If the LEB is completely empty, it is
+ * simply unmapped.
+ */
+static int fixup_leb_free_space(struct ubifs_info *c, int lnum, int len)
+{
+       int err;
+       void *sbuf = c->sbuf;
+
+       ubifs_assert(len >= 0);
+       ubifs_assert(len % c->min_io_size == 0);
+       ubifs_assert(len < c->leb_size);
+
+       if (len == 0) {
+               dbg_mnt("unmap empty LEB %d", lnum);
+               return ubi_leb_unmap(c->ubi, lnum);
+       }
+
+       dbg_mnt("fixup LEB %d, data len %d", lnum, len);
+       err = ubi_read(c->ubi, lnum, sbuf, 0, len);
+       if (err && err != -EBADMSG)
+               return err;
+
+       return ubi_leb_change(c->ubi, lnum, sbuf, len, UBI_UNKNOWN);
+}
+
+/**
+ * fixup_free_space - find & remap all LEBs containing free space.
+ * @c: UBIFS file-system description object
+ *
+ * This function walks through all LEBs in the filesystem and fiexes up those
+ * containing free/empty space.
+ */
+static int fixup_free_space(struct ubifs_info *c)
+{
+       int lnum, err = 0;
+       struct ubifs_lprops *lprops;
+
+       ubifs_get_lprops(c);
+
+       /* Fixup LEBs in the master area */
+       for (lnum = UBIFS_MST_LNUM; lnum < UBIFS_LOG_LNUM; lnum++) {
+               err = fixup_leb_free_space(c, lnum,
+                                          c->mst_offs + c->mst_node_alsz);
+               if (err)
+                       goto out;
+       }
+
+       /* Unmap unused log LEBs */
+       lnum = ubifs_next_log_lnum(c, c->lhead_lnum);
+       while (lnum != c->ltail_lnum) {
+               err = fixup_leb_free_space(c, lnum, 0);
+               if (err)
+                       goto out;
+               lnum = ubifs_next_log_lnum(c, lnum);
+       }
+
+       /* Fixup the current log head */
+       err = fixup_leb_free_space(c, c->lhead_lnum, c->lhead_offs);
+       if (err)
+               goto out;
+
+       /* Fixup LEBs in the LPT area */
+       for (lnum = c->lpt_first; lnum <= c->lpt_last; lnum++) {
+               int free = c->ltab[lnum - c->lpt_first].free;
+
+               if (free > 0) {
+                       err = fixup_leb_free_space(c, lnum, c->leb_size - free);
+                       if (err)
+                               goto out;
+               }
+       }
+
+       /* Unmap LEBs in the orphans area */
+       for (lnum = c->orph_first; lnum <= c->orph_last; lnum++) {
+               err = fixup_leb_free_space(c, lnum, 0);
+               if (err)
+                       goto out;
+       }
+
+       /* Fixup LEBs in the main area */
+       for (lnum = c->main_first; lnum < c->leb_cnt; lnum++) {
+               lprops = ubifs_lpt_lookup(c, lnum);
+               if (IS_ERR(lprops)) {
+                       err = PTR_ERR(lprops);
+                       goto out;
+               }
+
+               if (lprops->free > 0) {
+                       err = fixup_leb_free_space(c, lnum,
+                                                  c->leb_size - lprops->free);
+                       if (err)
+                               goto out;
+               }
+       }
+
+out:
+       ubifs_release_lprops(c);
+       return err;
+}
+
+/**
+ * ubifs_fixup_free_space - find & fix all LEBs with free space.
+ * @c: UBIFS file-system description object
+ *
+ * This function fixes up LEBs containing free space on first mount, if the
+ * appropriate flag was set when the FS was created. Each LEB with one or more
+ * empty min. I/O unit(i.e. free-space-count > 0) is re-written, to make sure
+ * the free space is actually erased. E.g., this is necessary for some NAND
+ * chips, since the free space may have been programmed like real "0xff" data
+ * (generating a non-0xff ECC), causing future writes to the not-really-erased
+ * NAND pages to behave badly. After fixup, the superblock space fixup flag is
+ * cleared, so that this is skipped for all future mounts.
+ */
+int ubifs_fixup_free_space(struct ubifs_info *c)
+{
+       int err;
+       struct ubifs_sb_node *sup;
+
+       ubifs_assert(c->space_fixup);
+       ubifs_assert(!c->ro_mount);
+
+       ubifs_msg("start fixing up free space");
+
+       err = fixup_free_space(c);
+       if (err)
+               return err;
+
+       sup = ubifs_read_sb_node(c);
+       if (IS_ERR(sup))
+               return PTR_ERR(sup);
+
+       /* Free-space fixup is no longer required */
+       c->space_fixup = 0;
+       sup->flags &= cpu_to_le32(~UBIFS_FLG_SPACE_FIXUP);
+
+       err = ubifs_write_sb_node(c, sup);
+       kfree(sup);
+       if (err)
+               return err;
+
+       ubifs_msg("free space fixup complete");
+       return err;
+}
diff --git a/fs/ubifs/ubifs.h b/fs/ubifs/ubifs.h
index 6f0bfa9..43b3195 100644
--- a/fs/ubifs/ubifs.h
+++ b/fs/ubifs/ubifs.h
@@ -1635,6 +1635,7 @@ int ubifs_write_master(struct ubifs_info *c);
 int ubifs_read_superblock(struct ubifs_info *c);
 struct ubifs_sb_node *ubifs_read_sb_node(struct ubifs_info *c);
 int ubifs_write_sb_node(struct ubifs_info *c, struct ubifs_sb_node *sup);
+int ubifs_fixup_free_space(struct ubifs_info *c);
 
 /* replay.c */
 int ubifs_validate_entry(struct ubifs_info *c,
-- 
1.6.3.3

[PATCH 2/2] UBIFS: fix-up free space on mount if flag is set

[Matthew L. Creech]

If a UBIFS filesystem is being mounted read-write, or is being remounted
from read-only to read-write, check for the "space_fixup" flag and fix
all LEBs containing empty space if necessary.

Signed-off-by: Matthew L. Creech <mlcreech@gmail.com>
---
 fs/ubifs/super.c |   12 ++++++++++++
 1 files changed, 12 insertions(+), 0 deletions(-)

diff --git a/fs/ubifs/super.c b/fs/ubifs/super.c
index 17b92af..59e8858 100644
--- a/fs/ubifs/super.c
+++ b/fs/ubifs/super.c
@@ -1396,6 +1396,12 @@ static int mount_ubifs(struct ubifs_info *c)
 	} else
 		ubifs_assert(c->lst.taken_empty_lebs > 0);
 
+	if (!c->ro_mount && c->space_fixup) {
+		err = ubifs_fixup_free_space(c);
+		if (err)
+			goto out_infos;
+	}
+
 	err = dbg_check_filesystem(c);
 	if (err)
 		goto out_infos;
@@ -1677,6 +1683,12 @@ static int ubifs_remount_rw(struct ubifs_info *c)
 		ubifs_msg("deferred recovery completed");
 	}
 
+	if (c->space_fixup) {
+		err = ubifs_fixup_free_space(c);
+		if (err)
+			goto out;
+	}
+
 	dbg_gen("re-mounted read-write");
 	c->remounting_rw = 0;
 	err = dbg_check_space_info(c);
-- 
1.6.3.3

Re: [PATCH 1/2] UBIFS: add the fixup function

[Artem Bityutskiy]

On Fri, 2011-05-06 at 18:58 -0400, Matthew L. Creech wrote:
> This patch adds the 'ubifs_fixup_free_space()' function which scans all
> LEBs in the filesystem for those that are in-use but have one or more
> empty pages, then re-maps the LEBs in order to erase the empty portions.
> Afterward it removes the "space_fixup" flag from the UBIFS superblock.
> 
> Signed-off-by: Matthew L. Creech <mlcreech@gmail.com>

...

> +static int fixup_leb_free_space(struct ubifs_info *c, int lnum, int len)

I've renamed it to fixup_leb() which seems to be a nicer name.

> +{
> +       int err;
> +       void *sbuf = c->sbuf;
> +
> +       ubifs_assert(len >= 0);
> +       ubifs_assert(len % c->min_io_size == 0);
> +       ubifs_assert(len < c->leb_size);
> +
> +       if (len == 0) {
> +               dbg_mnt("unmap empty LEB %d", lnum);
> +               return ubi_leb_unmap(c->ubi, lnum);
> +       }
> +
> +       dbg_mnt("fixup LEB %d, data len %d", lnum, len);
> +       err = ubi_read(c->ubi, lnum, sbuf, 0, len);
> +       if (err && err != -EBADMSG)
> +               return err;

I've removed the 'err != -EBADMSG' check because I think we do want to
fail in case of unrecoverable ECC errors.

And I've pushed it to ubifs-2.6.git, thanks.

-- 
Best Regards,
Artem Bityutskiy (Артём Битюцкий)

Re: [PATCH 2/2] UBIFS: fix-up free space on mount if flag is set

[Artem Bityutskiy]

On Fri, 2011-05-06 at 18:58 -0400, Matthew L. Creech wrote:
> If a UBIFS filesystem is being mounted read-write, or is being remounted
> from read-only to read-write, check for the "space_fixup" flag and fix
> all LEBs containing empty space if necessary.
> 
> Signed-off-by: Matthew L. Creech <mlcreech@gmail.com>

Pushed to l2-mtd-2.6.git, thanks!

Also, it would be nice to have a piece of documentation for this feature
in the mtd web site - could you cook a patch for that too please? And
please, add links to this doc from other relevant places - you posted
links AFAIR.

-- 
Best Regards,
Artem Bityutskiy (Артём Битюцкий)

Re: [PATCH 2/2] UBIFS: fix-up free space on mount if flag is set

[Artem Bityutskiy]

On Fri, 2011-05-06 at 18:58 -0400, Matthew L. Creech wrote:
> +	if (c->space_fixup) {
> +		err = ubifs_fixup_free_space(c);
> +		if (err)
> +			goto out;
> +	}
> +
>  	dbg_gen("re-mounted read-write");
>  	c->remounting_rw = 0;
>  	err = dbg_check_space_info(c);

Note, I've moved the 'ubifs_fixup_free_space()' a bit down to make sure
we first print "re-mounted" and "deferred recovery completed" messages,
and then start fixing up. I think this is a bit neater.

-- 
Best Regards,
Artem Bityutskiy (Артём Битюцкий)

Re: [PATCH 2/2] UBIFS: fix-up free space on mount if flag is set

[Artem Bityutskiy]

On Thu, 2011-05-12 at 14:09 +0300, Artem Bityutskiy wrote:
> On Fri, 2011-05-06 at 18:58 -0400, Matthew L. Creech wrote:
> > +	if (c->space_fixup) {
> > +		err = ubifs_fixup_free_space(c);
> > +		if (err)
> > +			goto out;
> > +	}
> > +
> >  	dbg_gen("re-mounted read-write");
> >  	c->remounting_rw = 0;
> >  	err = dbg_check_space_info(c);
> 
> Note, I've moved the 'ubifs_fixup_free_space()' a bit down to make sure
> we first print "re-mounted" and "deferred recovery completed" messages,
> and then start fixing up. I think this is a bit neater.

I do not know if that was me or you, but I've found out that the a lot
of code uses spaces for indentation instead of tabs, and checkpatch.pl
complains:

WARNING: please, no spaces at the start of a line
#119: FILE: fs/ubifs/sb.c:746:
+                       err = PTR_ERR(lprops);$

ERROR: code indent should use tabs where possible
#120: FILE: fs/ubifs/sb.c:747:
+                       goto out;$

.... a lot of this ....

But I've just fixed this.

-- 
Best Regards,
Artem Bityutskiy (Артём Битюцкий)

RE: [PATCH 2/2] UBIFS: fix-up free space on mount if flag is set

[Atlant Schmidt]

Artem:

  I don't know what the coding standards are like on
  your project, but in very much of the C/C++ developer
  world, <tab>s are frowned upon in code as being fragile,
  primarily because there are varying standards for just
  how wide a <tab> should be. By default, they're 8 spaces,
  but I've worked at several shops where they are only
  4 spaces and one shop where they are deemed to be 3!

  Because of this, many coding standards (such as the
  WebKit coding standards and my current employer's
  standard) outright ban <tab> characters in code and
  I'd recommend you do so as well, no matter what the
  current scripts enforce. (It's certainly easy enough
  to expand all the tabs in an existing code base so
  as to bring it into compliance with a "no tabs" rule.)

                            Atlant

-----Original Message-----
From: linux-mtd-bounces@lists.infradead.org [mailto:linux-mtd-bounces@lists.infradead.org] On Behalf Of Artem Bityutskiy
Sent: Friday, May 13, 2011 03:59
To: Matthew L. Creech
Cc: linux-mtd@lists.infradead.org
Subject: Re: [PATCH 2/2] UBIFS: fix-up free space on mount if flag is set

On Thu, 2011-05-12 at 14:09 +0300, Artem Bityutskiy wrote:
> On Fri, 2011-05-06 at 18:58 -0400, Matthew L. Creech wrote:
> > +   if (c->space_fixup) {
> > +           err = ubifs_fixup_free_space(c);
> > +           if (err)
> > +                   goto out;
> > +   }
> > +
> >     dbg_gen("re-mounted read-write");
> >     c->remounting_rw = 0;
> >     err = dbg_check_space_info(c);
>
> Note, I've moved the 'ubifs_fixup_free_space()' a bit down to make sure
> we first print "re-mounted" and "deferred recovery completed" messages,
> and then start fixing up. I think this is a bit neater.

I do not know if that was me or you, but I've found out that the a lot
of code uses spaces for indentation instead of tabs, and checkpatch.pl
complains:

WARNING: please, no spaces at the start of a line
#119: FILE: fs/ubifs/sb.c:746:
+                       err = PTR_ERR(lprops);$

ERROR: code indent should use tabs where possible
#120: FILE: fs/ubifs/sb.c:747:
+                       goto out;$

.... a lot of this ....

But I've just fixed this.

--
Best Regards,
Artem Bityutskiy (Артём Битюцкий)


______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

This e-mail and the information, including any attachments, it contains are intended to be a confidential communication only to the person or entity to whom it is addressed and may contain information that is privileged. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please immediately notify the sender and destroy the original message.

Thank you.

Please consider the environment before printing this email.

Re: [PATCH 2/2] UBIFS: fix-up free space on mount if flag is set

[Michael Cashwell]

Any many shops/projects feel strongly the other way.

I don't buy the "tabs are fragile" argument. It makes no sense. Even if a project uses spaces but some people indent to 8, some to 4 and some to 3 you still end up with a hard-to-read mess. Mandating the use of spaces is solving the wrong problem.

The problem is that there must be agreement on what the indentation value is. Once you have that it then the code is readable and clean regardless of whether you use spaces or tabs to do it.

-Mike

On May 13, 2011, at 6:59 AM, Atlant Schmidt wrote:

> Artem:
> 
>  I don't know what the coding standards are like on
>  your project, but in very much of the C/C++ developer
>  world, <tab>s are frowned upon in code as being fragile,
>  primarily because there are varying standards for just
>  how wide a <tab> should be. By default, they're 8 spaces,
>  but I've worked at several shops where they are only
>  4 spaces and one shop where they are deemed to be 3!
> 
>  Because of this, many coding standards (such as the
>  WebKit coding standards and my current employer's
>  standard) outright ban <tab> characters in code and
>  I'd recommend you do so as well, no matter what the
>  current scripts enforce. (It's certainly easy enough
>  to expand all the tabs in an existing code base so
>  as to bring it into compliance with a "no tabs" rule.)
> 
>                            Atlant
> 
> -----Original Message-----
> From: linux-mtd-bounces@lists.infradead.org [mailto:linux-mtd-bounces@lists.infradead.org] On Behalf Of Artem Bityutskiy
> Sent: Friday, May 13, 2011 03:59
> To: Matthew L. Creech
> Cc: linux-mtd@lists.infradead.org
> Subject: Re: [PATCH 2/2] UBIFS: fix-up free space on mount if flag is set
> 
> On Thu, 2011-05-12 at 14:09 +0300, Artem Bityutskiy wrote:
>> On Fri, 2011-05-06 at 18:58 -0400, Matthew L. Creech wrote:
>>> +   if (c->space_fixup) {
>>> +           err = ubifs_fixup_free_space(c);
>>> +           if (err)
>>> +                   goto out;
>>> +   }
>>> +
>>>    dbg_gen("re-mounted read-write");
>>>    c->remounting_rw = 0;
>>>    err = dbg_check_space_info(c);
>> 
>> Note, I've moved the 'ubifs_fixup_free_space()' a bit down to make sure
>> we first print "re-mounted" and "deferred recovery completed" messages,
>> and then start fixing up. I think this is a bit neater.
> 
> I do not know if that was me or you, but I've found out that the a lot
> of code uses spaces for indentation instead of tabs, and checkpatch.pl
> complains:
> 
> WARNING: please, no spaces at the start of a line
> #119: FILE: fs/ubifs/sb.c:746:
> +                       err = PTR_ERR(lprops);$
> 
> ERROR: code indent should use tabs where possible
> #120: FILE: fs/ubifs/sb.c:747:
> +                       goto out;$
> 
> .... a lot of this ....
> 
> But I've just fixed this.
> 
> --
> Best Regards,
> Artem Bityutskiy (Артём Битюцкий)
> 
> 
> ______________________________________________________
> Linux MTD discussion mailing list
> http://lists.infradead.org/mailman/listinfo/linux-mtd/
> 
> This e-mail and the information, including any attachments, it contains are intended to be a confidential communication only to the person or entity to whom it is addressed and may contain information that is privileged. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please immediately notify the sender and destroy the original message.
> 
> Thank you.
> 
> Please consider the environment before printing this email.
> 
> ______________________________________________________
> Linux MTD discussion mailing list
> http://lists.infradead.org/mailman/listinfo/linux-mtd/

RE: [PATCH 2/2] UBIFS: fix-up free space on mount if flag is set

[Artem Bityutskiy]

Hi,

On Fri, 2011-05-13 at 06:59 -0400, Atlant Schmidt wrote:
> Artem:
> 
>   I don't know what the coding standards are like on
>   your project,

The kenrel standard, described in Documentation/CodingStyle

>  but in very much of the C/C++ developer
>   world, <tab>s are frowned upon in code as being fragile,
>   primarily because there are varying standards for just
>   how wide a <tab> should be.

In the kernel we just stand that it must be 8 spaces.

>  By default, they're 8 spaces,
>   but I've worked at several shops where they are only
>   4 spaces and one shop where they are deemed to be 3!

Well, there were many flamewars about this, but we just assume that
anyone looking at the kernel source code has to setup their editors to
have tab=8 spaces.

>   Because of this, many coding standards (such as the
>   WebKit coding standards and my current employer's
>   standard) outright ban <tab> characters in code and
>   I'd recommend you do so as well, no matter what the
>   current scripts enforce. (It's certainly easy enough
>   to expand all the tabs in an existing code base so
>   as to bring it into compliance with a "no tabs" rule.)

Well, UBIFS is part of the kernel and it follows the kernel coding
style.

-- 
Best Regards,
Artem Bityutskiy (Артём Битюцкий)

RE: [PATCH 2/2] UBIFS: fix-up free space on mount if flag is set

[Artem Bityutskiy]

On Fri, 2011-05-13 at 15:29 +0300, Artem Bityutskiy wrote:
> Well, UBIFS is part of the kernel and it follows the kernel coding
> style.

Note, in this case I'm talking only about tabs as indentation, like

<tab>if (a)
<tab><tab>if (b)
<tab><tab><tab>function();

The other thing is tabs as "dynamic spaces", like:

int a<tab>=0;
int aa<tab>=0;

Some people use tabs as "dynamic spaces" because many editors will align
things. I personally do not appreciate this at all, and the kernel
Coding style does not require this. I only use tabs for indentation at
the beginning of the line.

-- 
Best Regards,
Artem Bityutskiy (Артём Битюцкий)

[PATCH] UBIFS: don't fail on -EBADMSG when fixing free space

[Ben Gardiner]

The free space fixup was introduced so that UBIFS on NAND flashes where the
empty pages at the end need to be dropped when flashing or else -74
(aka -EBADMSG) failures will occur -- but due to the limited functionality of
flash-programming facilities the emtpy pages cannot be dropped.

In fixup_leb, which is called during free space fixup, the now-expected return
code of -EBADMSG is treated as a failure.

Don't fail when -EBADMSG is encountered during ubi_read.

Signed-off-by: Ben Gardiner <bengardiner@nanometrics.ca>
CC: Matthew L. Creech <mlcreech@gmail.com>

---
I tested this patch on-top-of l2 head : 'bbf2b37 UBIFS: fix extremely rare
mount failure' using Matthew's 'mkfs.ubifs: add "-F" option for "free-space
fixup"' patch to create the UBIFS image with the flag set.

The test system was a da850evm, the image was flashed with U-boot's plain-old
'nand write' and the rootfs is mounted with 'ubi.mtd=X,2048 root=ubi0:rootfs
rootfstype=ubifs' bootargs.

Without this patch the first attemp to start fixing free space fails
with -EBADMSG:

UBIFS: start fixing up free space
UBI error: ubi_io_read: error -74 (ECC error) while reading 4096 bytes from PEB 18:4096, read 4096 bytes
VFS: Cannot open root device "ubi0:rootfs" or unknown-block(0,0)
[...]

---
 fs/ubifs/sb.c |    8 +++++++-
 1 files changed, 7 insertions(+), 1 deletions(-)

diff --git a/fs/ubifs/sb.c b/fs/ubifs/sb.c
index c606f01..6e7da54 100644
--- a/fs/ubifs/sb.c
+++ b/fs/ubifs/sb.c
@@ -1,3 +1,4 @@
+#define DEBUG
 /*
  * This file is part of UBIFS.
  *
@@ -679,7 +680,12 @@ static int fixup_leb(struct ubifs_info *c, int lnum, int len)
 
 	dbg_mnt("fixup LEB %d, data len %d", lnum, len);
 	err = ubi_read(c->ubi, lnum, c->sbuf, 0, len);
-	if (err)
+	/*
+	 * Don't fail on -EBADMSG since these are precisely the error codes that
+	 * are returned by ubi_red in the cases where free-space fix-ups are
+	 * required.
+	 */
+	if (err && err != -EBADMSG)
 		return err;
 
 	return ubi_leb_change(c->ubi, lnum, c->sbuf, len, UBI_UNKNOWN);
-- 
1.7.4.1

[PATCH] UBIFS: don't fail on -EBADMSG when fixing free space

[Ben Gardiner]

The free space fixup was introduced so that UBIFS on NAND flashes where the
empty pages at the end need to be dropped when flashing or else -74
(aka -EBADMSG) failures will occur -- but due to the limited functionality of
flash-programming facilities the emtpy pages cannot be dropped.

In fixup_leb, which is called during free space fixup, the now-expected return
code of -EBADMSG is treated as a failure.

Don't fail when -EBADMSG is encountered during ubi_read.

Signed-off-by: Ben Gardiner <bengardiner@nanometrics.ca>
CC: Matthew L. Creech <mlcreech@gmail.com>

---
I tested this patch on-top-of l2 head : 'bbf2b37 UBIFS: fix extremely rare
mount failure' using Matthew's 'mkfs.ubifs: add "-F" option for "free-space
fixup"' patch to create the UBIFS image with the flag set.

The test system was a da850evm, the image was flashed with U-boot's plain-old
'nand write' and the rootfs is mounted with 'ubi.mtd=X,2048 root=ubi0:rootfs
rootfstype=ubifs' bootargs.

Without this patch the first attemp to start fixing free space fails
with -EBADMSG:

UBIFS: start fixing up free space
UBI error: ubi_io_read: error -74 (ECC error) while reading 4096 bytes from PEB 18:4096, read 4096 bytes
VFS: Cannot open root device "ubi0:rootfs" or unknown-block(0,0)
[...]

---
 fs/ubifs/sb.c |    8 +++++++-
 1 files changed, 7 insertions(+), 1 deletions(-)

diff --git a/fs/ubifs/sb.c b/fs/ubifs/sb.c
index c606f01..6e7da54 100644
--- a/fs/ubifs/sb.c
+++ b/fs/ubifs/sb.c
@@ -1,3 +1,4 @@
+#define DEBUG
 /*
  * This file is part of UBIFS.
  *
@@ -679,7 +680,12 @@ static int fixup_leb(struct ubifs_info *c, int lnum, int len)
 
 	dbg_mnt("fixup LEB %d, data len %d", lnum, len);
 	err = ubi_read(c->ubi, lnum, c->sbuf, 0, len);
-	if (err)
+	/*
+	 * Don't fail on -EBADMSG since these are precisely the error codes that
+	 * are returned by ubi_red in the cases where free-space fix-ups are
+	 * required.
+	 */
+	if (err && err != -EBADMSG)
 		return err;
 
 	return ubi_leb_change(c->ubi, lnum, c->sbuf, len, UBI_UNKNOWN);
-- 
1.7.4.1

Re: [PATCH] UBIFS: don't fail on -EBADMSG when fixing free space

[Matthew L. Creech]

On Wed, May 18, 2011 at 4:47 PM, Ben Gardiner
<bengardiner@nanometrics.ca> wrote:
> +       /*
> +        * Don't fail on -EBADMSG since these are precisely the error codes that
> +        * are returned by ubi_red in the cases where free-space fix-ups are
> +        * required.
> +        */

Hi Ben,

Off-hand, I don't see how this comment can be true.  It seems like
we'll have 2 cases:

1. The pages in question are actually erased, in which case they'll be
0xff-filled by ubi_read() without error, or
2. The pages were programmed with all 0xff data, in which case the ECC
info should be correct

There's a third possibility, that the ECC info is bad because there's
a real error, but that's not something we can reliably recover from.
Am I overlooking another scenario?

-- 
Matthew L. Creech

Re: [PATCH] UBIFS: don't fail on -EBADMSG when fixing free space

[Matthew L. Creech]

On Wed, May 18, 2011 at 4:47 PM, Ben Gardiner
<bengardiner@nanometrics.ca> wrote:
> +       /*
> +        * Don't fail on -EBADMSG since these are precisely the error codes that
> +        * are returned by ubi_red in the cases where free-space fix-ups are
> +        * required.
> +        */

Hi Ben,

Off-hand, I don't see how this comment can be true.  It seems like
we'll have 2 cases:

1. The pages in question are actually erased, in which case they'll be
0xff-filled by ubi_read() without error, or
2. The pages were programmed with all 0xff data, in which case the ECC
info should be correct

There's a third possibility, that the ECC info is bad because there's
a real error, but that's not something we can reliably recover from.
Am I overlooking another scenario?

-- 
Matthew L. Creech

[PATCH] UBIFS: document the "free space fixup" flag

[Matthew L. Creech]

Several parts of the UBI and UBIFS documentation already mention the need to
use ubiformat when flashing UBIFS images.  Add a new FAQ entry for the "free
space fixup" flag as an alternative in recent kernels, and put links to it in
these existing locations.
---
 doc/ubi.xml   |    6 ++++++
 faq/ubi.xml   |    8 +++++---
 faq/ubifs.xml |   34 +++++++++++++++++++++++++++++++++-
 3 files changed, 44 insertions(+), 4 deletions(-)

diff --git a/doc/ubi.xml b/doc/ubi.xml
index d22884f..b3ca4ee 100644
--- a/doc/ubi.xml
+++ b/doc/ubi.xml
@@ -699,6 +699,12 @@ and the writes did not fail. But later we observed weird ECC errors. It took a
 while to find out the problem. In other words, this is also relevant to JFFS2
 images.</p>
 
+<p>An alternative to this approach is to enable the "free space fixup" option
+when generating the UBIFS file system using <code>mkfs.ubifs</code>.  This will
+allow your flasher to not have to worry about <code>0xFF</code> bytes at the end
+of PEBs, which is particularly useful if you need to use an industrial flash
+programmer to write a UBI image.  More information is available
+<a href="../faq/ubifs.html#L_free_space_fixup">here</a>.</p>
 
 
 <h2><a name="L_torturing">Marking eraseblocks as bad</a></h2>
diff --git a/faq/ubi.xml b/faq/ubi.xml
index 8a48a48..f616cfe 100644
--- a/faq/ubi.xml
+++ b/faq/ubi.xml
@@ -567,9 +567,11 @@ any image, and check if you still have these errors.</p>
 related to how you flash the UBI/UBIFS images. One typical problem is related
 to ECC calculation algorithm - read
 <a name="ubifs.html#L_why_ubiformat">here</a> for more information. Make sure
-that you use <a name="../doc/ubifs.html#L_usptools">ubiformat</a>), or make sure
-your flashing program skips 0xFF properly (see
-<a href="../doc/ubi.html#L_flasher_algo">here</a>).</p>
+that you use <a href="../doc/ubifs.html#L_usptools">ubiformat</a>, or make sure
+either that your flashing program skips <code>0xFF</code> properly (see
+<a href="../doc/ubi.html#L_flasher_algo">here</a>) or that your UBIFS image
+was generated with the "free space fixup" flag set (see
+<a href="ubifs.html#L_free_space_fixup">here</a>).</p>
 
 <p>Another possibility is that your flash reports that it supports
 <a href="../doc/ubi.html#L_subpage">sub-pages</a>, but does not actually support
diff --git a/faq/ubifs.xml b/faq/ubifs.xml
index f1e063c..6e010b5 100644
--- a/faq/ubifs.xml
+++ b/faq/ubifs.xml
@@ -20,6 +20,7 @@
 	<li><a href="ubifs.html#L_emptyflash">May an empty UBI volume be mounted?</a></li>
 	<li><a href="ubifs.html#L_max_leb_cnt">What is the purpose of -c (--max-leb-cnt) mkfs.ubifs option?</a></li>
 	<li><a href="ubifs.html#L_why_ubiformat">Why do I have to use ubiformat?</a></li>
+	<li><a href="ubifs.html#L_free_space_fixup">What is the the purpose of the -F (--space-fixup) mkfs.ubifs option?</a></li>
 	<li><a href="ubifs.html#L_mkfs_ubifs_comp">How do I compile mkfs.ubifs?</a></li>
 	<li><a href="ubifs.html#L_favor_lzo">What is "favor LZO" compression?</a></li>
 	<li><a href="ubifs.html#L_loop_mount">Can UBIFS mount loop-back devices?</a></li>
@@ -397,7 +398,9 @@ vol_flags=autoresize
 (<a href="../doc/ubi.html#L_flasher_algo">here</a> you may find some
 hints). Please, read
 <a href="ubifs.html#L_why_ubiformat">this</a> section for more information
-why you should use <code>ubiformat</code>.</p>
+why you should use <code>ubiformat</code>, or generate your UBIFS images with
+the <a href="ubifs.html#L_free_space_fixup">free space fixup</a> flag if that
+is not possible.</p>
 
 
 
@@ -594,6 +597,35 @@ written once and only once after the erasure. If you use
 <code>nandwrite</code>, some pages are written twice - once by
 <code>nandwrite</code>, and once by UBIFS.</p>
 
+<p>If you can not use <code>ubiformat</code>, an alternative is to set the
+"free space fixup" flag when generating the UBIFS image (see
+<a href="ubifs.html#L_free_space_fixup">here</a>).</p>
+
+
+<h2><a name="L_free_space_fixup">What is the the purpose of the -F
+(--space-fixup) mkfs.ubifs option?</a></h2>
+
+<p>Because of subtle ECC errors that can arise when programming NAND flash
+(see <a href="ubifs.html#L_why_ubiformat">here</a>), <code>ubiformat</code>
+is the recommended way of flashing a UBI image which contains a UBIFS file
+system.  However, this is not always possible - for example, some embedded
+devices are manufactured using an industrial NAND flash programmer which has
+no knowledge of UBI or UBIFS.</p>
+
+<p>The <code>-F</code> option causes <code>mkfs.ubifs</code> to set a special
+flag in the superblock, which triggers a "free space fixup" procedure in the
+kernel the very first time the filesystem is mounted.  This fixup procedure
+involves finding all empty pages in the UBIFS file system and re-erasing them.
+This ensures that NAND pages which contain all <code>0xFF</code> data get
+fully erased, which removes any problematic non-<code>0xFF</code> data from
+their OOB areas.</p>
+
+<p>This option is supported if you are running a kernel version
+<code>2.6.40</code> or higher.  Note that <code>ubiformat</code> is still the
+preferred flashing method if the image is not being flashed for the first time,
+since it preserves existing erase counters (while using <code>nandwrite</code>
+or its equivalent does not).</p>
+
 
 
 <h2><a name="L_mkfs_ubifs_comp">How do I compile mkfs.ubifs?</a></h2>
-- 
1.7.4.1

Re: [PATCH] UBIFS: don't fail on -EBADMSG when fixing free space

[Ben Gardiner]

Hi Matthew,

On Wed, May 18, 2011 at 5:41 PM, Matthew L. Creech <mlcreech@gmail.com> wrote:
> On Wed, May 18, 2011 at 4:47 PM, Ben Gardiner
> <bengardiner@nanometrics.ca> wrote:
>> +       /*
>> +        * Don't fail on -EBADMSG since these are precisely the error codes that
>> +        * are returned by ubi_red in the cases where free-space fix-ups are
>> +        * required.
>> +        */
>
> Hi Ben,
>
> Off-hand, I don't see how this comment can be true.  It seems like
> we'll have 2 cases:
>
> 1. The pages in question are actually erased, in which case they'll be
> 0xff-filled by ubi_read() without error, or
> 2. The pages were programmed with all 0xff data, in which case the ECC
> info should be correct
>
> There's a third possibility, that the ECC info is bad because there's
> a real error, but that's not something we can reliably recover from.
> Am I overlooking another scenario?

Perhaps I am running into a nuance of the davinci nand driver. Back
when you started the 'Programming ubinized images' thread [1] some of
you problem description caught me eye:

On Mon, Apr 25, 2011 at 2:37 PM, Matthew L. Creech <mlcreech@gmail.com> wrote:
> We encountered one case in which we were re-flashing a device for
> testing using U-Boot's "nand erase", and got the "ubi_io_read: error
> -74" error from the FAQ.  That's no big deal, since we never do this
> in the field, and clearly "nand erase" isn't something we'd want to do
> even without this problem since it loses erase-counter info.

Because the "ubi_io_read: error -74 (ECC error)" is precisely what I
am encountering on my hardware when I do not flash with a utility that
drops empty pages at the end of eraseblocks. I imagined that this was
also the case for you. But I have also read that there are
peculiarities of the davinci nand driver (both in u-boot and linux).

So, at least on my hardware, the -74 error is expected when the 0xff
pages are not dropped and so without the 'err != -EBADMSG' exception
the free space fixup will cause the volume to fail mount for me:

UBIFS: start fixing up free space
UBI error: ubi_io_read: error -74 (ECC error) while reading 4096 bytes
from PEB 18:4096, read 4096 bytes
VFS: Cannot open root device "ubi0:rootfs" or unknown-block(0,0)
[...]

Whereas, when the same image is flashed using a u-boot 'nand write'
variant that drops 0xff pages [2] there are no such -74 errors
encountered.

UBIFS: start fixing up free space
UBIFS: free space fixup complete
UBIFS: mounted UBI device 0, volume 1, name "rootfs"
[...]

The nand I am using is the micron part that ships with the da850evm;
it has eraseblocks of 128KiB and pages of 2048. Here is a dump of
eraseblock 18 from the ubinized image:

00240000  55 42 49 23 01 00 00 00  00 00 00 00 00 00 00 00  |UBI#............|
00240010  00 00 08 00 00 00 10 00  04 6c 24 45 00 00 00 00  |.........l$E....|
00240020  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00240030  00 00 00 00 00 00 00 00  00 00 00 00 38 be e8 75  |............8..u|
00240040  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................|
*
00240800  55 42 49 21 01 01 00 00  00 00 00 01 00 00 00 01  |UBI!............|
00240810  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00240830  00 00 00 00 00 00 00 00  00 00 00 00 5f 7f ac 08  |............_...|
00240840  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................|
*
00241000  31 18 10 06 5f 3f 84 94  65 32 01 00 00 00 00 00  |1..._?..e2......|
00241010  00 02 00 00 07 00 00 00  53 2a 00 00 00 00 00 00  |........S*......|
00241020  00 00 00 00 00 00 00 00  02 00 00 00 03 00 00 00  |................|
00241030  f0 01 00 00 e0 8a 01 00  44 00 00 00 e3 01 00 00  |........D.......|
00241040  f0 01 00 00 00 90 01 00  b0 c7 18 00 00 00 00 00  |................|
00241050  00 18 05 00 00 00 00 00  00 49 05 00 00 00 00 00  |.........I......|
00241060  50 77 8a 03 00 00 00 00  78 fc 04 00 00 00 00 00  |Pw......x.......|
00241070  78 b8 01 00 00 00 00 00  08 00 00 00 12 0a 00 00  |x...............|
00241080  08 00 00 00 00 10 00 00  08 00 00 00 1e 0a 00 00  |................|
00241090  00 00 00 00 00 00 00 00  0b 00 00 00 01 00 00 00  |................|
002410a0  0d 00 00 00 f1 01 00 00  00 00 00 00 00 00 00 00  |................|
002410b0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00241200  31 18 10 06 ef de 0e ee  00 00 00 00 00 00 00 00  |1...............|
00241210  1c 00 00 00 05 00 00 00  e4 05 00 00 00 00 00 00  |................|
00241220  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00241800  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................|
*
00260000  55 42 49 23 01 00 00 00  00 00 00 00 00 00 00 00  |UBI#............|
00260010  00 00 08 00 00 00 10 00  04 6c 24 45 00 00 00 00  |.........l$E....|
00260020  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00260030  00 00 00 00 00 00 00 00  00 00 00 00 38 be e8 75  |............8..u|
00260040  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................|
*
00260800  55 42 49 21 01 01 00 00  00 00 00 01 00 00 00 02  |UBI!............|
[...]

The data between 0x00241800 and 0x00260000 is all 0xff, so there are
trailing empty pages in this block.

Here is a u-boot nand-dump of eraseblock 18 + 0x1800 when flashed
using the usual u-boot 'nand write':
        ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff
[...]
        ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff
OOB:
        ff ff ff ff ff ff ff ff
        ff ff ff ff ff ff ff ff
        ff ff ff ff ff ff ff ff
        3f 27 56 f5 29 d8 61 d9
        9d 14 3f 27 56 f5 29 d8
        61 d9 9d 14 3f 27 56 f5
        29 d8 61 d9 9d 14 3f 27
        56 f5 29 d8 61 d9 9d 14

and here it is again when flashed with the 'nand write' variant that
drops 0xff pages [2]:

        ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff
[...]
        ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff
OOB:
        ff ff ff ff ff ff ff ff
        ff ff ff ff ff ff ff ff
        ff ff ff ff ff ff ff ff
        ff ff ff ff ff ff ff ff
        ff ff ff ff ff ff ff ff
        ff ff ff ff ff ff ff ff
        ff ff ff ff ff ff ff ff
        ff ff ff ff ff ff ff ff

The former state of eraseblock 18 causes "ubi_io_read: error -74 (ECC
error)" during free-space-fixup and the latter does not.

So I guess my particular situation is a problem with the davinci nand
driver's ECC for 0xFF data and is _not_ covered by the free space
fixup? It would be really nice if the free space fixup supported both
1) setups that are such that the flash cannot be written to more than
once and 2) setups that are such that they return bogus -74 errors.
Both are caused by not dropping trailing 0xff pages when writing.

Best Regards,
Ben Gardiner

[1] http://article.gmane.org/gmane.linux.drivers.mtd/34890
[2] http://article.gmane.org/gmane.comp.boot-loaders.u-boot/98740

---
Nanometrics Inc.
http://www.nanometrics.ca

Re: [PATCH] UBIFS: don't fail on -EBADMSG when fixing free space

[Ben Gardiner]

Hi Matthew,

On Wed, May 18, 2011 at 5:41 PM, Matthew L. Creech <mlcreech@gmail.com> wrote:
> On Wed, May 18, 2011 at 4:47 PM, Ben Gardiner
> <bengardiner@nanometrics.ca> wrote:
>> +       /*
>> +        * Don't fail on -EBADMSG since these are precisely the error codes that
>> +        * are returned by ubi_red in the cases where free-space fix-ups are
>> +        * required.
>> +        */
>
> Hi Ben,
>
> Off-hand, I don't see how this comment can be true.  It seems like
> we'll have 2 cases:
>
> 1. The pages in question are actually erased, in which case they'll be
> 0xff-filled by ubi_read() without error, or
> 2. The pages were programmed with all 0xff data, in which case the ECC
> info should be correct
>
> There's a third possibility, that the ECC info is bad because there's
> a real error, but that's not something we can reliably recover from.
> Am I overlooking another scenario?

Perhaps I am running into a nuance of the davinci nand driver. Back
when you started the 'Programming ubinized images' thread [1] some of
you problem description caught me eye:

On Mon, Apr 25, 2011 at 2:37 PM, Matthew L. Creech <mlcreech@gmail.com> wrote:
> We encountered one case in which we were re-flashing a device for
> testing using U-Boot's "nand erase", and got the "ubi_io_read: error
> -74" error from the FAQ.  That's no big deal, since we never do this
> in the field, and clearly "nand erase" isn't something we'd want to do
> even without this problem since it loses erase-counter info.

Because the "ubi_io_read: error -74 (ECC error)" is precisely what I
am encountering on my hardware when I do not flash with a utility that
drops empty pages at the end of eraseblocks. I imagined that this was
also the case for you. But I have also read that there are
peculiarities of the davinci nand driver (both in u-boot and linux).

So, at least on my hardware, the -74 error is expected when the 0xff
pages are not dropped and so without the 'err != -EBADMSG' exception
the free space fixup will cause the volume to fail mount for me:

UBIFS: start fixing up free space
UBI error: ubi_io_read: error -74 (ECC error) while reading 4096 bytes
from PEB 18:4096, read 4096 bytes
VFS: Cannot open root device "ubi0:rootfs" or unknown-block(0,0)
[...]

Whereas, when the same image is flashed using a u-boot 'nand write'
variant that drops 0xff pages [2] there are no such -74 errors
encountered.

UBIFS: start fixing up free space
UBIFS: free space fixup complete
UBIFS: mounted UBI device 0, volume 1, name "rootfs"
[...]

The nand I am using is the micron part that ships with the da850evm;
it has eraseblocks of 128KiB and pages of 2048. Here is a dump of
eraseblock 18 from the ubinized image:

00240000  55 42 49 23 01 00 00 00  00 00 00 00 00 00 00 00  |UBI#............|
00240010  00 00 08 00 00 00 10 00  04 6c 24 45 00 00 00 00  |.........l$E....|
00240020  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00240030  00 00 00 00 00 00 00 00  00 00 00 00 38 be e8 75  |............8..u|
00240040  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................|
*
00240800  55 42 49 21 01 01 00 00  00 00 00 01 00 00 00 01  |UBI!............|
00240810  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00240830  00 00 00 00 00 00 00 00  00 00 00 00 5f 7f ac 08  |............_...|
00240840  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................|
*
00241000  31 18 10 06 5f 3f 84 94  65 32 01 00 00 00 00 00  |1..._?..e2......|
00241010  00 02 00 00 07 00 00 00  53 2a 00 00 00 00 00 00  |........S*......|
00241020  00 00 00 00 00 00 00 00  02 00 00 00 03 00 00 00  |................|
00241030  f0 01 00 00 e0 8a 01 00  44 00 00 00 e3 01 00 00  |........D.......|
00241040  f0 01 00 00 00 90 01 00  b0 c7 18 00 00 00 00 00  |................|
00241050  00 18 05 00 00 00 00 00  00 49 05 00 00 00 00 00  |.........I......|
00241060  50 77 8a 03 00 00 00 00  78 fc 04 00 00 00 00 00  |Pw......x.......|
00241070  78 b8 01 00 00 00 00 00  08 00 00 00 12 0a 00 00  |x...............|
00241080  08 00 00 00 00 10 00 00  08 00 00 00 1e 0a 00 00  |................|
00241090  00 00 00 00 00 00 00 00  0b 00 00 00 01 00 00 00  |................|
002410a0  0d 00 00 00 f1 01 00 00  00 00 00 00 00 00 00 00  |................|
002410b0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00241200  31 18 10 06 ef de 0e ee  00 00 00 00 00 00 00 00  |1...............|
00241210  1c 00 00 00 05 00 00 00  e4 05 00 00 00 00 00 00  |................|
00241220  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00241800  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................|
*
00260000  55 42 49 23 01 00 00 00  00 00 00 00 00 00 00 00  |UBI#............|
00260010  00 00 08 00 00 00 10 00  04 6c 24 45 00 00 00 00  |.........l$E....|
00260020  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00260030  00 00 00 00 00 00 00 00  00 00 00 00 38 be e8 75  |............8..u|
00260040  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................|
*
00260800  55 42 49 21 01 01 00 00  00 00 00 01 00 00 00 02  |UBI!............|
[...]

The data between 0x00241800 and 0x00260000 is all 0xff, so there are
trailing empty pages in this block.

Here is a u-boot nand-dump of eraseblock 18 + 0x1800 when flashed
using the usual u-boot 'nand write':
        ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff
[...]
        ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff
OOB:
        ff ff ff ff ff ff ff ff
        ff ff ff ff ff ff ff ff
        ff ff ff ff ff ff ff ff
        3f 27 56 f5 29 d8 61 d9
        9d 14 3f 27 56 f5 29 d8
        61 d9 9d 14 3f 27 56 f5
        29 d8 61 d9 9d 14 3f 27
        56 f5 29 d8 61 d9 9d 14

and here it is again when flashed with the 'nand write' variant that
drops 0xff pages [2]:

        ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff
[...]
        ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff
OOB:
        ff ff ff ff ff ff ff ff
        ff ff ff ff ff ff ff ff
        ff ff ff ff ff ff ff ff
        ff ff ff ff ff ff ff ff
        ff ff ff ff ff ff ff ff
        ff ff ff ff ff ff ff ff
        ff ff ff ff ff ff ff ff
        ff ff ff ff ff ff ff ff

The former state of eraseblock 18 causes "ubi_io_read: error -74 (ECC
error)" during free-space-fixup and the latter does not.

So I guess my particular situation is a problem with the davinci nand
driver's ECC for 0xFF data and is _not_ covered by the free space
fixup? It would be really nice if the free space fixup supported both
1) setups that are such that the flash cannot be written to more than
once and 2) setups that are such that they return bogus -74 errors.
Both are caused by not dropping trailing 0xff pages when writing.

Best Regards,
Ben Gardiner

[1] http://article.gmane.org/gmane.linux.drivers.mtd/34890
[2] http://article.gmane.org/gmane.comp.boot-loaders.u-boot/98740

---
Nanometrics Inc.
http://www.nanometrics.ca

Re: [PATCH] UBIFS: don't fail on -EBADMSG when fixing free space

[Matthew L. Creech]

On Thu, May 19, 2011 at 9:28 AM, Ben Gardiner
<bengardiner@nanometrics.ca> wrote:
>
> So I guess my particular situation is a problem with the davinci nand
> driver's ECC for 0xFF data and is _not_ covered by the free space
> fixup?
>

Yeah, that's what it sounds like.

Can you reproduce the -EBADMSG with a simpler test case that leaves
UBI/UBIFS out of the equation?  Maybe enable write-verify in U-Boot
and try using "nand write" again, or use mtd-utils "nandwrite" from an
initramfs image then try to read back the data -- that should give a
simpler test case that shows whether the ECC handling of empty space
is wrong.

If so, that probably needs to be fixed at the MTD level, since it will
affect more than this particular UBIFS case.

-- 
Matthew L. Creech

Re: [PATCH] UBIFS: don't fail on -EBADMSG when fixing free space

[Matthew L. Creech]

On Thu, May 19, 2011 at 9:28 AM, Ben Gardiner
<bengardiner@nanometrics.ca> wrote:
>
> So I guess my particular situation is a problem with the davinci nand
> driver's ECC for 0xFF data and is _not_ covered by the free space
> fixup?
>

Yeah, that's what it sounds like.

Can you reproduce the -EBADMSG with a simpler test case that leaves
UBI/UBIFS out of the equation?  Maybe enable write-verify in U-Boot
and try using "nand write" again, or use mtd-utils "nandwrite" from an
initramfs image then try to read back the data -- that should give a
simpler test case that shows whether the ECC handling of empty space
is wrong.

If so, that probably needs to be fixed at the MTD level, since it will
affect more than this particular UBIFS case.

-- 
Matthew L. Creech

Re: [PATCH] UBIFS: don't fail on -EBADMSG when fixing free space

[Artem Bityutskiy]

[Rmoving Adiran Hunter from CC - he's left Nokia and his e-mail does not
work anyway]

Hi,

On Thu, 2011-05-19 at 09:28 -0400, Ben Gardiner wrote:
> On Mon, Apr 25, 2011 at 2:37 PM, Matthew L. Creech <mlcreech@gmail.com> wrote:
> > We encountered one case in which we were re-flashing a device for
> > testing using U-Boot's "nand erase", and got the "ubi_io_read: error
> > -74" error from the FAQ.  That's no big deal, since we never do this
> > in the field, and clearly "nand erase" isn't something we'd want to do
> > even without this problem since it loses erase-counter info.
> 
> Because the "ubi_io_read: error -74 (ECC error)" is precisely what I
> am encountering on my hardware when I do not flash with a utility that
> drops empty pages at the end of eraseblocks. I imagined that this was
> also the case for you. But I have also read that there are
> peculiarities of the davinci nand driver (both in u-boot and linux).
> 
> So, at least on my hardware, the -74 error is expected when the 0xff
> pages are not dropped and so without the 'err != -EBADMSG' exception
> the free space fixup will cause the volume to fail mount for me:

I am confused. The fix Matthew made is about the following situation:

1. You have completely erased flash (MTD partition) - no one has ever
written there. If you now read the flash, you'll get all 0xFFs with no
errors.

2. You use a "dumb" flasher to program an UBI image. This flasher will
write empty NAND pages "as is". If you now read the flash after the
"dumb" programming, you should have no errors.

3. You mount UBIFS for the very first time. It tries to fix up your
flash. Whatever eraseblock UBIFS reads, it should not encounter any
error.

Isn't it weird that a freshly programmed flash cannot be read without
-EBADMSG (ECC correction failure).

Why Matthew's patches are needed then? They are needed to _prevent_
UBIFS from writing to the NAND pages which have been programmed with all
0xFFs.


-- 
Best Regards,
Artem Bityutskiy (Артём Битюцкий)

Re: [PATCH] UBIFS: don't fail on -EBADMSG when fixing free space

[Artem Bityutskiy]

[Rmoving Adiran Hunter from CC - he's left Nokia and his e-mail does not
work anyway]

Hi,

On Thu, 2011-05-19 at 09:28 -0400, Ben Gardiner wrote:
> On Mon, Apr 25, 2011 at 2:37 PM, Matthew L. Creech <mlcreech@gmail.com> wrote:
> > We encountered one case in which we were re-flashing a device for
> > testing using U-Boot's "nand erase", and got the "ubi_io_read: error
> > -74" error from the FAQ.  That's no big deal, since we never do this
> > in the field, and clearly "nand erase" isn't something we'd want to do
> > even without this problem since it loses erase-counter info.
> 
> Because the "ubi_io_read: error -74 (ECC error)" is precisely what I
> am encountering on my hardware when I do not flash with a utility that
> drops empty pages at the end of eraseblocks. I imagined that this was
> also the case for you. But I have also read that there are
> peculiarities of the davinci nand driver (both in u-boot and linux).
> 
> So, at least on my hardware, the -74 error is expected when the 0xff
> pages are not dropped and so without the 'err != -EBADMSG' exception
> the free space fixup will cause the volume to fail mount for me:

I am confused. The fix Matthew made is about the following situation:

1. You have completely erased flash (MTD partition) - no one has ever
written there. If you now read the flash, you'll get all 0xFFs with no
errors.

2. You use a "dumb" flasher to program an UBI image. This flasher will
write empty NAND pages "as is". If you now read the flash after the
"dumb" programming, you should have no errors.

3. You mount UBIFS for the very first time. It tries to fix up your
flash. Whatever eraseblock UBIFS reads, it should not encounter any
error.

Isn't it weird that a freshly programmed flash cannot be read without
-EBADMSG (ECC correction failure).

Why Matthew's patches are needed then? They are needed to _prevent_
UBIFS from writing to the NAND pages which have been programmed with all
0xFFs.


-- 
Best Regards,
Artem Bityutskiy (Артём Битюцкий)

Re: [PATCH] UBIFS: don't fail on -EBADMSG when fixing free space

[Artem Bityutskiy]

On Thu, 2011-05-19 at 09:28 -0400, Ben Gardiner wrote:
> So, at least on my hardware, the -74 error is expected when the 0xff
> pages are not dropped and so without the 'err != -EBADMSG' exception
> the free space fixup will cause the volume to fail mount for me:

Still confused. If you have empty and erased flash, then you program it
with all 0xFFs, you should be able to read it with no errors. If this is
not the case for you, you should fix the driver. UBIFS cannot help in
this case - we consider it as "broken flash driver" case.

I have not read your e-mail carefully because of limited amount of time,
but with quick reading I became confused - too much information :-)

-- 
Best Regards,
Artem Bityutskiy (Артём Битюцкий)

Re: [PATCH] UBIFS: don't fail on -EBADMSG when fixing free space

[Artem Bityutskiy]

On Thu, 2011-05-19 at 09:28 -0400, Ben Gardiner wrote:
> So, at least on my hardware, the -74 error is expected when the 0xff
> pages are not dropped and so without the 'err != -EBADMSG' exception
> the free space fixup will cause the volume to fail mount for me:

Still confused. If you have empty and erased flash, then you program it
with all 0xFFs, you should be able to read it with no errors. If this is
not the case for you, you should fix the driver. UBIFS cannot help in
this case - we consider it as "broken flash driver" case.

I have not read your e-mail carefully because of limited amount of time,
but with quick reading I became confused - too much information :-)

-- 
Best Regards,
Artem Bityutskiy (Артём Битюцкий)

Re: [PATCH] UBIFS: document the "free space fixup" flag

[Artem Bityutskiy]

On Thu, 2011-05-19 at 01:32 -0400, Matthew L. Creech wrote:
> Several parts of the UBI and UBIFS documentation already mention the need to
> use ubiformat when flashing UBIFS images.  Add a new FAQ entry for the "free
> space fixup" flag as an alternative in recent kernels, and put links to it in
> these existing locations.
> ---

Pushed, thanks. I've only added few changes as a patch on top of yours,
also pushed.

> -why you should use <code>ubiformat</code>.</p>
> +why you should use <code>ubiformat</code>, or generate your UBIFS images with
> +the <a href="ubifs.html#L_free_space_fixup">free space fixup</a> flag if that
> +is not possible.</p>

I've removed this change because the first link already contains the
"fixup" link.

Pluse I've updated news and added a bit more information about the fixup
thingy.

Thank you!

-- 
Best Regards,
Artem Bityutskiy (Артём Битюцкий)

Re: [PATCH] UBIFS: don't fail on -EBADMSG when fixing free space

[Ben Gardiner]

Hi Artem,

On Fri, May 20, 2011 at 2:21 AM, Artem Bityutskiy <dedekind1@gmail.com> wrote:
> On Thu, 2011-05-19 at 09:28 -0400, Ben Gardiner wrote:
>> [...]
>>
>> Because the "ubi_io_read: error -74 (ECC error)" is precisely what I
>> am encountering on my hardware when I do not flash with a utility that
>> drops empty pages at the end of eraseblocks. I imagined that this was
>> also the case for you. But I have also read that there are
>> peculiarities of the davinci nand driver (both in u-boot and linux).
>>
>> So, at least on my hardware, the -74 error is expected when the 0xff
>> pages are not dropped and so without the 'err != -EBADMSG' exception
>> the free space fixup will cause the volume to fail mount for me:
>
> I am confused. The fix Matthew made is about the following situation:

Sorry for the confusion. In the first 'Programming ubinized images'
thread I saw that Matthew mentioned seeing lots of -74 errors so I was
assuming that his and my problems are one and the same. Now that I
have tested his patch I am seeing that this is not the case.

> 1. You have completely erased flash (MTD partition) - no one has ever
> written there. If you now read the flash, you'll get all 0xFFs with no
> errors.
>
> 2. You use a "dumb" flasher to program an UBI image. This flasher will
> write empty NAND pages "as is". If you now read the flash after the
> "dumb" programming, you should have no errors.
>
> 3. You mount UBIFS for the very first time. It tries to fix up your
> flash. Whatever eraseblock UBIFS reads, it should not encounter any
> error.

Yes I agree with the context for the patch. I can say so far that on
da850evm 1. occurs as you described whereas for 2. and 3. ECC errors
are encountered on read.

> Isn't it weird that a freshly programmed flash cannot be read without
> -EBADMSG (ECC correction failure).

Yes it is very weird. :) I had been assuming up to this point that the
-74 errors were the result of writing the 0xff pages when they should
be dropped. I can see now that this is not the case.

On Fri, May 20, 2011 at 2:29 AM, Artem Bityutskiy <dedekind1@gmail.com> wrote:
> On Thu, 2011-05-19 at 09:28 -0400, Ben Gardiner wrote:
>> So, at least on my hardware, the -74 error is expected when the 0xff
>> pages are not dropped and so without the 'err != -EBADMSG' exception
>> the free space fixup will cause the volume to fail mount for me:
>
> Still confused. If you have empty and erased flash, then you program it
> with all 0xFFs, you should be able to read it with no errors.

Right. In retrospect this seems to be a self-evident statement :)
I had been previously attributing the -74 errors to a combination of
1) writing 0xff pages when one should not and 2) lack of subpage
writing support workaround with -O 2048. But it is clearly (to me now)
a reading operation failure.

> If this is
> not the case for you, you should fix the driver. UBIFS cannot help in
> this case - we consider it as "broken flash driver" case.

Thank you. Yes, this appears to be the case and I'm glad to hear
confirmation of this from an expert such as yourself.

> I have not read your e-mail carefully because of limited amount of time,
> but with quick reading I became confused - too much information :-)

No problem. Thank you very much for what you have read and for your
insights into the ongoing NAND flash problems on da850evm.

Best Regards,
Ben Gardiner

---
Nanometrics Inc.
http://www.nanometrics.ca

Re: [PATCH] UBIFS: don't fail on -EBADMSG when fixing free space

[Ben Gardiner]

Hi Artem,

On Fri, May 20, 2011 at 2:21 AM, Artem Bityutskiy <dedekind1@gmail.com> wrote:
> On Thu, 2011-05-19 at 09:28 -0400, Ben Gardiner wrote:
>> [...]
>>
>> Because the "ubi_io_read: error -74 (ECC error)" is precisely what I
>> am encountering on my hardware when I do not flash with a utility that
>> drops empty pages at the end of eraseblocks. I imagined that this was
>> also the case for you. But I have also read that there are
>> peculiarities of the davinci nand driver (both in u-boot and linux).
>>
>> So, at least on my hardware, the -74 error is expected when the 0xff
>> pages are not dropped and so without the 'err != -EBADMSG' exception
>> the free space fixup will cause the volume to fail mount for me:
>
> I am confused. The fix Matthew made is about the following situation:

Sorry for the confusion. In the first 'Programming ubinized images'
thread I saw that Matthew mentioned seeing lots of -74 errors so I was
assuming that his and my problems are one and the same. Now that I
have tested his patch I am seeing that this is not the case.

> 1. You have completely erased flash (MTD partition) - no one has ever
> written there. If you now read the flash, you'll get all 0xFFs with no
> errors.
>
> 2. You use a "dumb" flasher to program an UBI image. This flasher will
> write empty NAND pages "as is". If you now read the flash after the
> "dumb" programming, you should have no errors.
>
> 3. You mount UBIFS for the very first time. It tries to fix up your
> flash. Whatever eraseblock UBIFS reads, it should not encounter any
> error.

Yes I agree with the context for the patch. I can say so far that on
da850evm 1. occurs as you described whereas for 2. and 3. ECC errors
are encountered on read.

> Isn't it weird that a freshly programmed flash cannot be read without
> -EBADMSG (ECC correction failure).

Yes it is very weird. :) I had been assuming up to this point that the
-74 errors were the result of writing the 0xff pages when they should
be dropped. I can see now that this is not the case.

On Fri, May 20, 2011 at 2:29 AM, Artem Bityutskiy <dedekind1@gmail.com> wrote:
> On Thu, 2011-05-19 at 09:28 -0400, Ben Gardiner wrote:
>> So, at least on my hardware, the -74 error is expected when the 0xff
>> pages are not dropped and so without the 'err != -EBADMSG' exception
>> the free space fixup will cause the volume to fail mount for me:
>
> Still confused. If you have empty and erased flash, then you program it
> with all 0xFFs, you should be able to read it with no errors.

Right. In retrospect this seems to be a self-evident statement :)
I had been previously attributing the -74 errors to a combination of
1) writing 0xff pages when one should not and 2) lack of subpage
writing support workaround with -O 2048. But it is clearly (to me now)
a reading operation failure.

> If this is
> not the case for you, you should fix the driver. UBIFS cannot help in
> this case - we consider it as "broken flash driver" case.

Thank you. Yes, this appears to be the case and I'm glad to hear
confirmation of this from an expert such as yourself.

> I have not read your e-mail carefully because of limited amount of time,
> but with quick reading I became confused - too much information :-)

No problem. Thank you very much for what you have read and for your
insights into the ongoing NAND flash problems on da850evm.

Best Regards,
Ben Gardiner

---
Nanometrics Inc.
http://www.nanometrics.ca