[RFC] read-only filesystem support for NAND flash devices

[RFC] read-only filesystem support for NAND flash devices

[Vitaly Wool]

Hi folks,

there had recently been some talks around $Subject... and I'm personally 
interested in implementing that, both professionally and as an 
open-source guy :)

The problem, as most of you know well, is that currently existing 
compressed read-only filesystems don't support bad block handling. OTOH, 
it's quite an attractive idea to use those for root filesystem of an 
embedded device.
As a filesystem is read-only, wear-out can't occur during the normal 
use, only on root filesystem upgrade. Therefore writing it using a tool 
like nandwrite that bypasses bad blocks seems to be enough, and the only 
thing needed is either an in-MTD layer that makes a partition look as if 
there's no babd blocks, or  similar means within a filesystem.

The only known implementation goes the former way (though I recall some 
attempts for squashfs to implement the latter one). It's at 
http://lists.infradead.org/pipermail/linux-mtd/2004-May/009695.html.
The main drawbacks I see in the implementation are:
- adds more functions/vars to struct mtd_info which is apparently a bad 
idea (it's already overcomplicated)
- modifies a lot of generic files (i. e. CFI command set 
implementations, etc.)
- possibly exposed to FTL/NFTL patents

What I'd like to suggest is to rework this patch to get rid of these 
(and probably some more) drawbacks.
Ideas:
1. Implement it on top of NAND layer.
    - use NAND bad block table, but make its construction lazy in order 
to decrease boot time (might be a useful idea anyway)
    - use tree-like structure to calculate the offset between the block 
number supplied and the actual block to read (alternatively, just add 
the how-many-to-add info to BBT entries)
2. Implement it on top of MTD layer.
    - might be reasonable to use mtdblock_ro for that
    - might be reasonable to make bad block table global for all MTD 
devices, not for NAND only and, once again, add an option to construct 
it lazily.

Comments, corrections and suggestions are welcome.

Thx,
   Vitaly

Re: [RFC] read-only filesystem support for NAND flash devices

[Josh Boyer]

On 5/10/06, Vitaly Wool <vwool@ru.mvista.com> wrote:

> As a filesystem is read-only, wear-out can't occur during the normal
> use, only on root filesystem upgrade. Therefore writing it using a tool
> like nandwrite that bypasses bad blocks seems to be enough, and the only
> thing needed is either an in-MTD layer that makes a partition look as if
> there's no babd blocks, or  similar means within a filesystem.

Not quite the case.  You need bad block skipping, yes.  But NAND can
get bit flips in good blocks still.  How do you deal with that?  You
can't leave the block in that state forever because it will continue
to get bit flips and then your data will be unusable.

josh

Re: [RFC] read-only filesystem support for NAND flash devices

[Vitaly Wool]

Josh Boyer wrote:
>> As a filesystem is read-only, wear-out can't occur during the normal
>> use, only on root filesystem upgrade. Therefore writing it using a tool
>> like nandwrite that bypasses bad blocks seems to be enough, and the only
>> thing needed is either an in-MTD layer that makes a partition look as if
>> there's no babd blocks, or  similar means within a filesystem.
> Not quite the case.  You need bad block skipping, yes.  But NAND can
> get bit flips in good blocks still.  How do you deal with that?  You
> can't leave the block in that state forever because it will continue
> to get bit flips and then your data will be unusable.
Yep, I know about the issue. The recommended way to go here AFAIK is to 
mark the block as bad and copy its contents to a free one.
However, this will make mapping a lot more complicated so I'd like to go 
another way, i. e. erasing and rewriting this very block, and if it 
wears out, cache the data read from it and schedule moving the data 
forward within the partition in a background. Does that make sense?

Vitaly

Re: [RFC] read-only filesystem support for NAND flash devices

[Jörn Engel]

On Wed, 10 May 2006 15:20:44 +0400, Vitaly Wool wrote:
> Josh Boyer wrote:
>
> >Not quite the case.  You need bad block skipping, yes.  But NAND can
> >get bit flips in good blocks still.  How do you deal with that?  You
> >can't leave the block in that state forever because it will continue
> >to get bit flips and then your data will be unusable.
> Yep, I know about the issue. The recommended way to go here AFAIK is to 
> mark the block as bad and copy its contents to a free one.
> However, this will make mapping a lot more complicated so I'd like to go 
> another way, i. e. erasing and rewriting this very block, and if it 
> wears out, cache the data read from it and schedule moving the data 
> forward within the partition in a background. Does that make sense?

Or you could just ignore the issue.  The way I read it, your RFC is
basically "Don't do the real thing, just a quick hack to make it work
at all."  Which is perfectly fine to me.  It is not perfect, but that
is well documented and if people actually use it and care a lot about
some particular deficiency, you can solve it then.

Jörn

-- 
Anything that can go wrong, will.
-- Finagle's Law

Re: [RFC] read-only filesystem support for NAND flash devices

[Josh Boyer]

On 5/10/06, Jörn Engel <joern@wohnheim.fh-wedel.de> wrote:
> On Wed, 10 May 2006 15:20:44 +0400, Vitaly Wool wrote:
> > Josh Boyer wrote:
> >
> > >Not quite the case.  You need bad block skipping, yes.  But NAND can
> > >get bit flips in good blocks still.  How do you deal with that?  You
> > >can't leave the block in that state forever because it will continue
> > >to get bit flips and then your data will be unusable.
> > Yep, I know about the issue. The recommended way to go here AFAIK is to
> > mark the block as bad and copy its contents to a free one.
> > However, this will make mapping a lot more complicated so I'd like to go
> > another way, i. e. erasing and rewriting this very block, and if it
> > wears out, cache the data read from it and schedule moving the data
> > forward within the partition in a background. Does that make sense?
>
> Or you could just ignore the issue.  The way I read it, your RFC is
> basically "Don't do the real thing, just a quick hack to make it work
> at all."  Which is perfectly fine to me.  It is not perfect, but that
> is well documented and if people actually use it and care a lot about
> some particular deficiency, you can solve it then.

That's a perfectly acceptable approach to me.  I was just pointing
something out in case someone already cared enough about it to solve
it now.

josh

Re: [RFC] read-only filesystem support for NAND flash devices

[Russ Dill]

> > Not quite the case.  You need bad block skipping, yes.  But NAND can
> > get bit flips in good blocks still.  How do you deal with that?  You
> > can't leave the block in that state forever because it will continue
> > to get bit flips and then your data will be unusable.
> Yep, I know about the issue. The recommended way to go here AFAIK is to
> mark the block as bad and copy its contents to a free one.
> However, this will make mapping a lot more complicated so I'd like to go
> another way, i. e. erasing and rewriting this very block, and if it
> wears out, cache the data read from it and schedule moving the data
> forward within the partition in a background. Does that make sense?

Then you'll run out of blocks. There are two seperate issues with NAND:

1) Bad blocks, NAND chips are shipped with a certain number of bad
blocks, during the chips life time, some of the other blocks will go
bad.

2) Reading from a sector degrades the data, but just a little tiny
bit. After several thousand reads, you may get a bit flip in your data
(which can be corrected by ECC). At this point, you do not have a bad
block, you just need to rewrite the data. You can't rewrite it to the
current block, because then a power loss would mean disaster. Also,
you can't mark the currently block as bad, because if you did, you
would decrease the usable life of your system (tons of good blocks
marked as bad)

So really,  for issue 2, you need some sort of free block list, so you
can reuse them. Which also means, you can expect the order of the
blocks on your filesystem to start to look more and more random as
time goes on. In other words, you also need some algorithm to find
which block is where.

Free block list? Bad block list? Tracking of block numbers? Sounds
like a full on FTL to me.

Re: [RFC] read-only filesystem support for NAND flash devices

[David Woodhouse]

On Thu, 2006-05-11 at 12:06 -0700, Russ Dill wrote:
> Free block list? Bad block list? Tracking of block numbers? Sounds
> like a full on FTL to me. 

A full read-write FTL needs to handle 512-byte writes if it's going to
pretend to be a normal block device. A simple FTL for read-only stuff
like cramfs/squashfs doesn't need that.

We should probably do this kind of thing as an FTL -- we already have
the 'mtd_blkdevs' infrastructure to help you by doing most of the
interfacing to MTD and block layers for you. And then you can use any
read-only file system on it.

-- 
dwmw2

Re: [RFC] read-only filesystem support for NAND flash devices

[Russ Dill]

> > Free block list? Bad block list? Tracking of block numbers? Sounds
> > like a full on FTL to me.
>
> A full read-write FTL needs to handle 512-byte writes if it's going to
> pretend to be a normal block device. A simple FTL for read-only stuff
> like cramfs/squashfs doesn't need that.
>
> We should probably do this kind of thing as an FTL -- we already have
> the 'mtd_blkdevs' infrastructure to help you by doing most of the
> interfacing to MTD and block layers for you. And then you can use any
> read-only file system on it.

True, I also realized another possible shortcut. If a blocks gets an
ECC correction on a read, rewrite it to a free block, erase the block,
rewrite it from the data written to the free block, and then erase the
free block. You'd need to put some info in the OOB so you could know
what the state is on power recovery.

Re: [RFC] read-only filesystem support for NAND flash devices

[Josh Boyer]

On 5/11/06, Russ Dill <russ.dill@gmail.com> wrote:
> > > Free block list? Bad block list? Tracking of block numbers? Sounds
> > > like a full on FTL to me.
> >
> > A full read-write FTL needs to handle 512-byte writes if it's going to
> > pretend to be a normal block device. A simple FTL for read-only stuff
> > like cramfs/squashfs doesn't need that.
> >
> > We should probably do this kind of thing as an FTL -- we already have
> > the 'mtd_blkdevs' infrastructure to help you by doing most of the
> > interfacing to MTD and block layers for you. And then you can use any
> > read-only file system on it.
>
> True, I also realized another possible shortcut. If a blocks gets an
> ECC correction on a read, rewrite it to a free block, erase the block,
> rewrite it from the data written to the free block, and then erase the
> free block. You'd need to put some info in the OOB so you could know
> what the state is on power recovery.

Um, or simply leave the copied block alone, erase the original block
and use it as the spare now.  You only need a single block for
scrubbing.

josh

Re: [RFC] read-only filesystem support for NAND flash devices

[Russ Dill]

On 5/11/06, Josh Boyer <jwboyer@gmail.com> wrote:
> On 5/11/06, Russ Dill <russ.dill@gmail.com> wrote:
> > > > Free block list? Bad block list? Tracking of block numbers? Sounds
> > > > like a full on FTL to me.
> > >
> > > A full read-write FTL needs to handle 512-byte writes if it's going to
> > > pretend to be a normal block device. A simple FTL for read-only stuff
> > > like cramfs/squashfs doesn't need that.
> > >
> > > We should probably do this kind of thing as an FTL -- we already have
> > > the 'mtd_blkdevs' infrastructure to help you by doing most of the
> > > interfacing to MTD and block layers for you. And then you can use any
> > > read-only file system on it.
> >
> > True, I also realized another possible shortcut. If a blocks gets an
> > ECC correction on a read, rewrite it to a free block, erase the block,
> > rewrite it from the data written to the free block, and then erase the
> > free block. You'd need to put some info in the OOB so you could know
> > what the state is on power recovery.
>
> Um, or simply leave the copied block alone, erase the original block
> and use it as the spare now.  You only need a single block for
> scrubbing.

At some point, the original block needs to be reused though, I suppose
it could go onto your freelist at scantime. I'm just looking for ways
to decrease mount time. If any block could be anywhere, you will have
long mount times.

If, on the other hand, you have a free block list past the end of the
image, you only need to scan this at mount, or even lazy, if a block
isn't available when you read it, then at that point, scan the free
list until it is found.

Re: [RFC] read-only filesystem support for NAND flash devices

[Vitaly Wool]

Russ Dill wrote:
> True, I also realized another possible shortcut. If a blocks gets an
> ECC correction on a read, rewrite it to a free block, erase the block,
> rewrite it from the data written to the free block, and then erase the
> free block. You'd need to put some info in the OOB so you could know
> what the state is on power recovery.
>
Oh thanks, that's the idea I'm going to use. It's a pretty reasonable 
assumption that 1 byte out of OOB is available as there's not going to 
be anything in OOB but ECC code in case of RO filesystem.

Vitaly

Re: [RFC] read-only filesystem support for NAND flash devices

[Vitaly Wool]

Russ Dill wrote:
> Free block list? Bad block list? Tracking of block numbers? Sounds
> like a full on FTL to me.
I'd rather go modifying mtdblock_ro. I don't want to implement FTL 
because 1) as David pointed out, I don't need to handle writes; 2) I'm 
afraid we may end up dealing with patent issues here again.

Vitaly

Re: [RFC] read-only filesystem support for NAND flash devices

[David Woodhouse]

On Fri, 2006-05-12 at 12:52 +0400, Vitaly Wool wrote:
> I'd rather go modifying mtdblock_ro. I don't want to implement FTL 
> because 1) as David pointed out, I don't need to handle writes; 2) I'm
> afraid we may end up dealing with patent issues here again. 

mtdblock_ro _is_ an FTL. Just a very very primitive one :)

By all means extend it until it does what we need -- but don't call it
mtdblock_ro.

-- 
dwmw2

Re: [RFC] read-only filesystem support for NAND flash devices

[Atsushi Nemoto]

On Wed, 10 May 2006 06:00:57 -0500, "Josh Boyer" <jwboyer@gmail.com> wrote:
> Not quite the case.  You need bad block skipping, yes.  But NAND can
> get bit flips in good blocks still.  How do you deal with that?  You
> can't leave the block in that state forever because it will continue
> to get bit flips and then your data will be unusable.

If I mounted a JFFS2 filesystem on NAND and _never_ write anything (or
mounted read-only), does current MTD/NAND/JFFS2 layer handle those bit
flips?  I've thought JFFS2 only reassign blocks on writing, so JFFS2
also suffer those "bit flips on read" problem, is it right?

---
Atsushi Nemoto

Re: [RFC] read-only filesystem support for NAND flash devices

[Josh Boyer]

On 5/10/06, Atsushi Nemoto <anemo@mba.ocn.ne.jp> wrote:
> On Wed, 10 May 2006 06:00:57 -0500, "Josh Boyer" <jwboyer@gmail.com> wrote:
> > Not quite the case.  You need bad block skipping, yes.  But NAND can
> > get bit flips in good blocks still.  How do you deal with that?  You
> > can't leave the block in that state forever because it will continue
> > to get bit flips and then your data will be unusable.
>
> If I mounted a JFFS2 filesystem on NAND and _never_ write anything (or
> mounted read-only), does current MTD/NAND/JFFS2 layer handle those bit
> flips?  I've thought JFFS2 only reassign blocks on writing, so JFFS2
> also suffer those "bit flips on read" problem, is it right?

If you truly never write anything, then yes it probably suffers from
the same problem.  In a typical JFFS2 setup where writes do occur, the
theory is that garbage collection will eventually reclaim that block. 
Whether that actually happens or not is largely determined on a per
use case basis.

josh

Re: [RFC] read-only filesystem support for NAND flash devices

[Jörn Engel]

On Wed, 10 May 2006 12:20:58 -0500, Josh Boyer wrote:
> On 5/10/06, Atsushi Nemoto <anemo@mba.ocn.ne.jp> wrote:
> >
> >If I mounted a JFFS2 filesystem on NAND and _never_ write anything (or
> >mounted read-only), does current MTD/NAND/JFFS2 layer handle those bit
> >flips?  I've thought JFFS2 only reassign blocks on writing, so JFFS2
> >also suffer those "bit flips on read" problem, is it right?
> 
> If you truly never write anything, then yes it probably suffers from
> the same problem.  In a typical JFFS2 setup where writes do occur, the
> theory is that garbage collection will eventually reclaim that block. 
> Whether that actually happens or not is largely determined on a per
> use case basis.

The fact remains that jffs2 only scrubs bad blocks by accident on a
random basis.  And it couldn't do more anyway.  Nothing in mtd
indicates a correctable error that would require scrubbing.

Jörn

-- 
This above all: to thine own self be true.
-- Shakespeare

Re: [RFC] read-only filesystem support for NAND flash devices

[Atsushi Nemoto]

On Wed, 10 May 2006 12:20:58 -0500, "Josh Boyer" <jwboyer@gmail.com> wrote:
> > If I mounted a JFFS2 filesystem on NAND and _never_ write anything (or
> > mounted read-only), does current MTD/NAND/JFFS2 layer handle those bit
> > flips?  I've thought JFFS2 only reassign blocks on writing, so JFFS2
> > also suffer those "bit flips on read" problem, is it right?
> 
> If you truly never write anything, then yes it probably suffers from
> the same problem.  In a typical JFFS2 setup where writes do occur, the
> theory is that garbage collection will eventually reclaim that block. 
> Whether that actually happens or not is largely determined on a per
> use case basis.

Thank you.  Then something like periodical garbage collection will
save the bit flips, but of course unnecessary writes should shorten
the life of the device ...  so ambivalent.

---
Atsushi Nemoto