Page corruption when writing non-sequential pages in an MLC NAND eraseblock

Page corruption when writing non-sequential pages in an MLC NAND eraseblock

[Avery Pennarun]

Hi everyone,

Does this sound familiar?

Setup:
- Linux 2.6.37 or 3.2.26 (apologies for the old versions, but I
couldn't find any related patches in newer kernels and these are
embedded systems that are a bit hard to upgrade)
- Micron MT29F16G08CBACAWP - MLC NAND
- 1 Mbyte eraseblocks
- 1 kbyte pages
- Tested on two different hardware platforms (MIPS and ARM devices)

Steps:
- Disable ECC to avoid any confusion (ECC turns out to not affect the
test results, but I wanted to make sure)
- Erase any eraseblock
- Write all-zeroes to page 0x18 of that eraseblock; read it back -> ok
- Write all-zeroes to page 0x12 of that eraseblock; read it back ->
FAIL, all 0xff instead
- Read page 0x18 -> completely random data (about 12% of bits are flipped)

More details:
- It doesn't happen with a related Micron SLC NAND
- If you write the pages in the opposite order there is no problem
- Delays before/after reading and writing, and the sequence of reads,
has no effect
- This turns out to happen for any pair of pages, where pairs are
always exactly 6 pages apart (n and n+6), other than the first and
last 4 pages in each eraseblock, which are paired with different pages
to make the math work out

I've read about the concept of MLC "paired pages" causing corruption
elsewhere, but it supposedly only happens when you get a power
failure.  It's happening for me during normal runtime, which seems
wrong.

Has anyone heard of this?

Thanks,

Avery

Re: Page corruption when writing non-sequential pages in an MLC NAND eraseblock

[Ricard Wanderlof]

On Wed, 2 Oct 2013, Avery Pennarun wrote:

> Does this sound familiar?
>
> Setup:
> - Linux 2.6.37 or 3.2.26 (apologies for the old versions, but I
> couldn't find any related patches in newer kernels and these are
> embedded systems that are a bit hard to upgrade)
> - Micron MT29F16G08CBACAWP - MLC NAND
> - 1 Mbyte eraseblocks
> - 1 kbyte pages
> - Tested on two different hardware platforms (MIPS and ARM devices)
>
> Steps:
> - Disable ECC to avoid any confusion (ECC turns out to not affect the
> test results, but I wanted to make sure)
> - Erase any eraseblock
> - Write all-zeroes to page 0x18 of that eraseblock; read it back -> ok
> - Write all-zeroes to page 0x12 of that eraseblock; read it back ->
> FAIL, all 0xff instead
> - Read page 0x18 -> completely random data (about 12% of bits are flipped)
>
> More details:
> - It doesn't happen with a related Micron SLC NAND
> - If you write the pages in the opposite order there is no problem
> - Delays before/after reading and writing, and the sequence of reads,
> has no effect
> - This turns out to happen for any pair of pages, where pairs are
> always exactly 6 pages apart (n and n+6), other than the first and
> last 4 pages in each eraseblock, which are paired with different pages
> to make the math work out
>
> I've read about the concept of MLC "paired pages" causing corruption
> elsewhere, but it supposedly only happens when you get a power
> failure.  It's happening for me during normal runtime, which seems
> wrong.

Not much help, but AFAIU, you can get so-called 'write disturb' when 
writing to the other page in a pair, even without power failure during the 
write. But to my mind, 12% flipped bits sounds very excessive. I don't 
have any experience with MLC though.

Instictively it sounds like there is some other mechanism causing 
problems, for instance if mtd does the write to the pages in two goes, 
rather than in one write, as I think MLC in general can only accept one 
write operation per page, in contrast to SLC where it is legal to write 
the main and oob data at different times (normally 2 or 3 writes per page 
is legal, after that the page must be erased or the data can not be 
considered stable).

/Ricard
-- 
Ricard Wolf Wanderlöf                           ricardw(at)axis.com
Axis Communications AB, Lund, Sweden            www.axis.com
Phone +46 46 272 2016                           Fax +46 46 13 61 30

Re: Page corruption when writing non-sequential pages in an MLC NAND eraseblock

[Romain Izard]

On 2013-10-01, Avery Pennarun <apenwarr@gmail.com> wrote:
> Hi everyone,
>
> Does this sound familiar?
>
> Setup:
> - Linux 2.6.37 or 3.2.26 (apologies for the old versions, but I
> couldn't find any related patches in newer kernels and these are
> embedded systems that are a bit hard to upgrade)
> - Micron MT29F16G08CBACAWP - MLC NAND
> - 1 Mbyte eraseblocks
> - 1 kbyte pages
> - Tested on two different hardware platforms (MIPS and ARM devices)
>
> Steps:
> - Disable ECC to avoid any confusion (ECC turns out to not affect the
> test results, but I wanted to make sure)
> - Erase any eraseblock
> - Write all-zeroes to page 0x18 of that eraseblock; read it back -> ok
> - Write all-zeroes to page 0x12 of that eraseblock; read it back ->
> FAIL, all 0xff instead
> - Read page 0x18 -> completely random data (about 12% of bits are flipped)
>
> More details:
> - It doesn't happen with a related Micron SLC NAND
> - If you write the pages in the opposite order there is no problem
> - Delays before/after reading and writing, and the sequence of reads,
> has no effect
> - This turns out to happen for any pair of pages, where pairs are
> always exactly 6 pages apart (n and n+6), other than the first and
> last 4 pages in each eraseblock, which are paired with different pages
> to make the math work out
>
> I've read about the concept of MLC "paired pages" causing corruption
> elsewhere, but it supposedly only happens when you get a power
> failure.  It's happening for me during normal runtime, which seems
> wrong.

For me you're violating one key MLC assumption: the pages within a block
must be written in a strictly increasing order. It should be written
somewhere in the component's datasheet.

Best regards,
-- 
Romain Izard

Re: Page corruption when writing non-sequential pages in an MLC NAND eraseblock

[Avery Pennarun]

On Wed, Oct 2, 2013 at 3:18 AM, Romain Izard <romain.izard.pro@gmail.com> wrote:
> On 2013-10-01, Avery Pennarun <apenwarr@gmail.com> wrote:
>> Steps:
>> - Disable ECC to avoid any confusion (ECC turns out to not affect the
>> test results, but I wanted to make sure)
>> - Erase any eraseblock
>> - Write all-zeroes to page 0x18 of that eraseblock; read it back -> ok
>> - Write all-zeroes to page 0x12 of that eraseblock; read it back ->
>> FAIL, all 0xff instead
>> - Read page 0x18 -> completely random data (about 12% of bits are flipped)
>
> For me you're violating one key MLC assumption: the pages within a block
> must be written in a strictly increasing order. It should be written
> somewhere in the component's datasheet.

Hmm.  I haven't heard of this constraint, but it certainly seems to be
happening in this case (although in fact random-access writes *mostly*
work other than this pairing effect, so it seems like they intended it
to work properly).

This makes /dev/mtdblock completely unusable as it reorders writes.
Is /dev/mtdblock known to not work with MLC chips?

Thanks,

Avery

Re: Page corruption when writing non-sequential pages in an MLC NAND eraseblock

[Avery Pennarun]

On Wed, Oct 2, 2013 at 12:58 AM, Ricard Wanderlof
<ricard.wanderlof@axis.com> wrote:
> On Wed, 2 Oct 2013, Avery Pennarun wrote:
>> Steps:
>> - Disable ECC to avoid any confusion (ECC turns out to not affect the
>> test results, but I wanted to make sure)
>> - Erase any eraseblock
>> - Write all-zeroes to page 0x18 of that eraseblock; read it back -> ok
>> - Write all-zeroes to page 0x12 of that eraseblock; read it back ->
>> FAIL, all 0xff instead
>> - Read page 0x18 -> completely random data (about 12% of bits are flipped)
>
> Not much help, but AFAIU, you can get so-called 'write disturb' when writing
> to the other page in a pair, even without power failure during the write.
> But to my mind, 12% flipped bits sounds very excessive. I don't have any
> experience with MLC though.
>
> Instictively it sounds like there is some other mechanism causing problems,
> for instance if mtd does the write to the pages in two goes, rather than in
> one write, as I think MLC in general can only accept one write operation per
> page, in contrast to SLC where it is legal to write the main and oob data at
> different times (normally 2 or 3 writes per page is legal, after that the
> page must be erased or the data can not be considered stable).

Thanks Ricard.  Based on this suggestion, I added some trace printk's
to the nand driver and did find that it was writing the data, then the
oob, for each page.  So I commented out all the code that writes to
the oob (since I have ECC disabled anyhow for this test).  The traces
now show just two 4096-byte page writes.  But the problem still
occurs.  So I think that rules out this theory, unless I missed some
ways the nand can be written.

Note that I was slightly mistaken earlier: the number of flipped bits
seems to sometimes be 12%, sometimes 25%.  It's a little suspicious
that this is almostly exactly 1/8 and 1/4 respectively.

Re: Page corruption when writing non-sequential pages in an MLC NAND eraseblock

[Ricard Wanderlof]

On Wed, 2 Oct 2013, Avery Pennarun wrote:

> On Wed, Oct 2, 2013 at 3:18 AM, Romain Izard <romain.izard.pro@gmail.com> wrote:
>> On 2013-10-01, Avery Pennarun <apenwarr@gmail.com> wrote:
>>> Steps:
>>> - Disable ECC to avoid any confusion (ECC turns out to not affect the
>>> test results, but I wanted to make sure)
>>> - Erase any eraseblock
>>> - Write all-zeroes to page 0x18 of that eraseblock; read it back -> ok
>>> - Write all-zeroes to page 0x12 of that eraseblock; read it back ->
>>> FAIL, all 0xff instead
>>> - Read page 0x18 -> completely random data (about 12% of bits are flipped)
>>
>> For me you're violating one key MLC assumption: the pages within a block
>> must be written in a strictly increasing order. It should be written
>> somewhere in the component's datasheet.
>
> Hmm.  I haven't heard of this constraint, but it certainly seems to be
> happening in this case (although in fact random-access writes *mostly*
> work other than this pairing effect, so it seems like they intended it
> to work properly).

Just because they place the constraint on the user that the pages must be 
written in order doesn't mean that the device necessarily will malfunction 
if the user violates this. It just means that the designers have the 
freedom to introduce dependencies between the page writes if it makes the 
design easier/better/possible/whatever. It could very well be that for a 
given flash it is possible to do some writes out of order, but to avoid a) 
tying down the actual possibilities for a given flash and b) complicating 
the facts for the user they just specify that 'page writes must be 
executed in order'.

I must admit I wasn't aware of this fact either, but I have no hands-on 
experience with MLC flashes on the other hand.

> This makes /dev/mtdblock completely unusable as it reorders writes.
> Is /dev/mtdblock known to not work with MLC chips?

I don't think /dev/mtdblock reorders writes; isn't it just exposing the 
underlying /dev/mtd device as a block device essentially in order to make 
the mount command happy?

I would think the two most common uses of flash are either using a flash 
file system such as jffs2 or ubifs, and since each data node has its own 
header and identifier, there is no reason to do anything else but write 
the nodes in order within blocks, and the other use would be to store 
binary blobs, in which again the data would just be stored sequentially 
from start to end.

/Ricard
-- 
Ricard Wolf Wanderlöf                           ricardw(at)axis.com
Axis Communications AB, Lund, Sweden            www.axis.com
Phone +46 46 272 2016                           Fax +46 46 13 61 30

Re: Page corruption when writing non-sequential pages in an MLC NAND eraseblock

[Avery Pennarun]

On Thu, Oct 3, 2013 at 12:40 AM, Ricard Wanderlof
<ricard.wanderlof@axis.com> wrote:
> Just because they place the constraint on the user that the pages must be
> written in order doesn't mean that the device necessarily will malfunction
> if the user violates this. It just means that the designers have the freedom
> to introduce dependencies between the page writes if it makes the design
> easier/better/possible/whatever. It could very well be that for a given
> flash it is possible to do some writes out of order, but to avoid a) tying
> down the actual possibilities for a given flash and b) complicating the
> facts for the user they just specify that 'page writes must be executed in
> order'.
>
> I must admit I wasn't aware of this fact either, but I have no hands-on
> experience with MLC flashes on the other hand.

It seems rather under-documented to me, and I wouldn't be overly
surprised if it's not so much "standard behaviour" as "behaviour some
chip makers decided they could get away with based on usage patterns."
 But ok.

>> This makes /dev/mtdblock completely unusable as it reorders writes.
>> Is /dev/mtdblock known to not work with MLC chips?
>
> I don't think /dev/mtdblock reorders writes; isn't it just exposing the
> underlying /dev/mtd device as a block device essentially in order to make
> the mount command happy?
>
> I would think the two most common uses of flash are either using a flash
> file system such as jffs2 or ubifs, and since each data node has its own
> header and identifier, there is no reason to do anything else but write the
> nodes in order within blocks, and the other use would be to store binary
> blobs, in which again the data would just be stored sequentially from start
> to end.

/dev/mtdblock seems to do very complicated things.  For example, when
using it, you don't need to use flash_erase before writing to blocks.
Also, my device has 1024k eraseblocks, but if I do:

   dd if=/tmp/whatever of=/dev/mtdblock9 bs=1024k

it ends up *reading* from the flash, not just writing to it.  Given
how many ECC errors it also produces on my setup, I think it's also
reordering writes.  This isn't particularly surprising to me since as
soon as you go through the Linux block layer, you're at the mercy of
the disk I/O schedulers etc.  Perhaps it *mostly* won't reorder writes
in simple cases, but I don't think there's any guarantee of this.

Re: Page corruption when writing non-sequential pages in an MLC NAND eraseblock

[Romain Izard]

2013/10/3 Avery Pennarun <apenwarr@gmail.com>:
> On Thu, Oct 3, 2013 at 12:40 AM, Ricard Wanderlof
> <ricard.wanderlof@axis.com> wrote:
>> Just because they place the constraint on the user that the pages must be
>> written in order doesn't mean that the device necessarily will malfunction
>> if the user violates this. It just means that the designers have the freedom
>> to introduce dependencies between the page writes if it makes the design
>> easier/better/possible/whatever. It could very well be that for a given
>> flash it is possible to do some writes out of order, but to avoid a) tying
>> down the actual possibilities for a given flash and b) complicating the
>> facts for the user they just specify that 'page writes must be executed in
>> order'.
>>
>> I must admit I wasn't aware of this fact either, but I have no hands-on
>> experience with MLC flashes on the other hand.
>
> It seems rather under-documented to me, and I wouldn't be overly
> surprised if it's not so much "standard behaviour" as "behaviour some
> chip makers decided they could get away with based on usage patterns."
>  But ok.
>

It's right there in the datasheet for both MLC manufacturers I have worked with.
And it seems that even SLC flash with large blocks [1] can also have the
same constraint.

There is very interesting technical presentation from Micron [2] that explains
all the issues with NAND flash, and that states that the write order
is necessary
to reduce the 'program disturb' issues.

[1] http://permalink.gmane.org/gmane.linux.drivers.mtd/20182
[2] https://www.micron.com/~/media/Documents/Products/Presentation/flash_mem_summit_jcooke_inconvenient_truths_nand.pdf

Regards,
-- 
Romain Izard

Re: Page corruption when writing non-sequential pages in an MLC NAND eraseblock

[Ricard Wanderlof]

On Thu, 3 Oct 2013, Romain Izard wrote:

> It's right there in the datasheet for both MLC manufacturers I have worked with.
> And it seems that even SLC flash with large blocks [1] can also have the
> same constraint.
>
> There is very interesting technical presentation from Micron [2] that explains
> all the issues with NAND flash, and that states that the write order
> is necessary
> to reduce the 'program disturb' issues.
>
> [1] http://permalink.gmane.org/gmane.linux.drivers.mtd/20182
> [2] https://www.micron.com/~/media/Documents/Products/Presentation/flash_mem_summit_jcooke_inconvenient_truths_nand.pdf

Interesting, I hadn't heard that sequential writes were a requirement for 
large-page SLC flash. I just checked the data sheet for a Samsung K9F1208 
and it doesn't mention it at all. That's just one of many of course, and 
it could still be that sequential writes reduce the likelyhood of program 
disturb errors.

/Ricard
-- 
Ricard Wolf Wanderlöf                           ricardw(at)axis.com
Axis Communications AB, Lund, Sweden            www.axis.com
Phone +46 46 272 2016                           Fax +46 46 13 61 30