RFC: detect and manage power cut on MLC NAND

RFC: detect and manage power cut on MLC NAND

[Andrea Scian]

Dear All,

for testing and research purpose, I'm working with an Embedded Linux
product which mounts an MLC NAND.

I know that one of the trouble with MLC flash on Linux with UBI/UBIFS is
the power cut management (ref.
http://www.linux-mtd.infradead.org/doc/ubifs.html#L_unstable_bits)

IIUC the solution inside MTD/UBI/UBIFS is not so simple and thus not so
quick to implement.

I'm wondering if anyone of you already think about managing power cut in
a different way: many embedded/industrial products already have some
big/huge bypass capacitor that are able to keep the system alive for
hundreds of milliseconds (this is useful for power supply
micro-interruptions), which are commonly enough to complete the latest
NAND operation.
Most of the has a GPIO/IRQ that tells whenever such a power cut happens,
so the kernel can take some actions with it (e.g., in our case, complete
the last NAND operation and prevent from starting newer operation, which
may corrupt the NAND pages).

WDYT about this?
If it sounds reasonable is there any suggestion where to place such a code?

Best Regards,

-- 

Andrea SCIAN

DAVE Embedded Systems

Re: RFC: detect and manage power cut on MLC NAND

[Richard Weinberger]

On Tue, Mar 10, 2015 at 12:57 PM, Andrea Scian <rnd4@dave-tech.it> wrote:
>
> Dear All,
>
> for testing and research purpose, I'm working with an Embedded Linux
> product which mounts an MLC NAND.
>
> I know that one of the trouble with MLC flash on Linux with UBI/UBIFS is
> the power cut management (ref.
> http://www.linux-mtd.infradead.org/doc/ubifs.html#L_unstable_bits)
>
> IIUC the solution inside MTD/UBI/UBIFS is not so simple and thus not so
> quick to implement.
>
> I'm wondering if anyone of you already think about managing power cut in
> a different way: many embedded/industrial products already have some
> big/huge bypass capacitor that are able to keep the system alive for
> hundreds of milliseconds (this is useful for power supply
> micro-interruptions), which are commonly enough to complete the latest
> NAND operation.
> Most of the has a GPIO/IRQ that tells whenever such a power cut happens,
> so the kernel can take some actions with it (e.g., in our case, complete
> the last NAND operation and prevent from starting newer operation, which
> may corrupt the NAND pages).
>
> WDYT about this?
> If it sounds reasonable is there any suggestion where to place such a code?

Customers often use DYI uninterruptible power supplies using capacitors.
But managing a power cut is the least problem you have with MLC NAND.

-- 
Thanks,
//richard

Re: RFC: detect and manage power cut on MLC NAND

[Artem Bityutskiy]

On Tue, 2015-03-10 at 13:51 +0100, Richard Weinberger wrote:
> > WDYT about this?
> > If it sounds reasonable is there any suggestion where to place such a code?
> 
> Customers often use DYI uninterruptible power supplies using capacitors.
> But managing a power cut is the least problem you have with MLC NAND.

Why is it the least problem, what is the hardest one? I thought this one
is the hardest.

Artem.

Re: RFC: detect and manage power cut on MLC NAND

[Artem Bityutskiy]

On Tue, 2015-03-10 at 12:57 +0100, Andrea Scian wrote:
> Dear All,
> 
> for testing and research purpose, I'm working with an Embedded Linux
> product which mounts an MLC NAND.
> 
> I know that one of the trouble with MLC flash on Linux with UBI/UBIFS is
> the power cut management (ref.
> http://www.linux-mtd.infradead.org/doc/ubifs.html#L_unstable_bits)
> 
> IIUC the solution inside MTD/UBI/UBIFS is not so simple and thus not so
> quick to implement.
> 
> I'm wondering if anyone of you already think about managing power cut in
> a different way: many embedded/industrial products already have some
> big/huge bypass capacitor that are able to keep the system alive for
> hundreds of milliseconds (this is useful for power supply
> micro-interruptions), which are commonly enough to complete the latest
> NAND operation.
> Most of the has a GPIO/IRQ that tells whenever such a power cut happens,
> so the kernel can take some actions with it (e.g., in our case, complete
> the last NAND operation and prevent from starting newer operation, which
> may corrupt the NAND pages).
> 
> WDYT about this?
> If it sounds reasonable is there any suggestion where to place such a code?

Ccing Boris, he has been looking into this area lately.

Artem.

Re: RFC: detect and manage power cut on MLC NAND

[Richard Weinberger]

Hi!

Am 11.03.2015 um 08:20 schrieb Artem Bityutskiy:
> On Tue, 2015-03-10 at 13:51 +0100, Richard Weinberger wrote:
>>> WDYT about this?
>>> If it sounds reasonable is there any suggestion where to place such a code?
>>
>> Customers often use DYI uninterruptible power supplies using capacitors.
>> But managing a power cut is the least problem you have with MLC NAND.
> 
> Why is it the least problem, what is the hardest one? I thought this one
> is the hardest.

IMHO the hardest ones are the problems we don't know yet as NAND vendors
are not really chatty about the MLC constraints.
We don't know much about data retention for example. At least we have not
much hard facts. Most of our knowledge is hearsay.

What I wanted to say is that soldering a big capacitor onto your board
will not make MLC NAND magically work perfectly fine. :-)

Thanks,
//richard

Re: RFC: detect and manage power cut on MLC NAND

[Artem Bityutskiy]

On Wed, 2015-03-11 at 09:57 +0100, Richard Weinberger wrote:
> Hi!
> 
> Am 11.03.2015 um 08:20 schrieb Artem Bityutskiy:
> > On Tue, 2015-03-10 at 13:51 +0100, Richard Weinberger wrote:
> >>> WDYT about this?
> >>> If it sounds reasonable is there any suggestion where to place such a code?
> >>
> >> Customers often use DYI uninterruptible power supplies using capacitors.
> >> But managing a power cut is the least problem you have with MLC NAND.
> > 
> > Why is it the least problem, what is the hardest one? I thought this one
> > is the hardest.
> 
> IMHO the hardest ones are the problems we don't know yet as NAND vendors
> are not really chatty about the MLC constraints.
> We don't know much about data retention for example. At least we have not
> much hard facts. Most of our knowledge is hearsay.

Well, but from the problems we know paired pages seems to be the biggest
one. E.g., what do we do if VID header gets corrupted because of an
interrupted write to the page paired with the VID header page? Sounds
like a hard problem to me.

As for other things, I am not sure. I remember back when I worked for
Nokia, vendors did tell us everything we needed to know about their
flash product.

Re: RFC: detect and manage power cut on MLC NAND

[Richard Weinberger]

Am 11.03.2015 um 10:05 schrieb Artem Bityutskiy:
> On Wed, 2015-03-11 at 09:57 +0100, Richard Weinberger wrote:
>> Hi!
>>
>> Am 11.03.2015 um 08:20 schrieb Artem Bityutskiy:
>>> On Tue, 2015-03-10 at 13:51 +0100, Richard Weinberger wrote:
>>>>> WDYT about this?
>>>>> If it sounds reasonable is there any suggestion where to place such a code?
>>>>
>>>> Customers often use DYI uninterruptible power supplies using capacitors.
>>>> But managing a power cut is the least problem you have with MLC NAND.
>>>
>>> Why is it the least problem, what is the hardest one? I thought this one
>>> is the hardest.
>>
>> IMHO the hardest ones are the problems we don't know yet as NAND vendors
>> are not really chatty about the MLC constraints.
>> We don't know much about data retention for example. At least we have not
>> much hard facts. Most of our knowledge is hearsay.
> 
> Well, but from the problems we know paired pages seems to be the biggest
> one. E.g., what do we do if VID header gets corrupted because of an
> interrupted write to the page paired with the VID header page? Sounds
> like a hard problem to me.

It is a hard problem. But at least we know about it.

> As for other things, I am not sure. I remember back when I worked for
> Nokia, vendors did tell us everything we needed to know about their
> flash product.
> 

Nokia is/was not really a small player.

Thanks,
//richard

Re: RFC: detect and manage power cut on MLC NAND

[Andrea Scian]

Hi all,

and thanks for you feedback.
You can find my comments below.

Il 11/03/2015 10:09, Richard Weinberger ha scritto:
> Am 11.03.2015 um 10:05 schrieb Artem Bityutskiy:
>> On Wed, 2015-03-11 at 09:57 +0100, Richard Weinberger wrote:
>>> Hi!
>>>
>>> Am 11.03.2015 um 08:20 schrieb Artem Bityutskiy:
>>>> On Tue, 2015-03-10 at 13:51 +0100, Richard Weinberger wrote:
>>>>>> WDYT about this?
>>>>>> If it sounds reasonable is there any suggestion where to place such a code?
>>>>> Customers often use DYI uninterruptible power supplies using capacitors.
>>>>> But managing a power cut is the least problem you have with MLC NAND.
>>>> Why is it the least problem, what is the hardest one? I thought this one
>>>> is the hardest.
>>> IMHO the hardest ones are the problems we don't know yet as NAND vendors
>>> are not really chatty about the MLC constraints.
>>> We don't know much about data retention for example. At least we have not
>>> much hard facts. Most of our knowledge is hearsay.
>> Well, but from the problems we know paired pages seems to be the biggest
>> one. E.g., what do we do if VID header gets corrupted because of an
>> interrupted write to the page paired with the VID header page? Sounds
>> like a hard problem to me.
> It is a hard problem. But at least we know about it.
>

IIUC can I summarize like the following

1) power cut is one of the big problem we know about MLC NAND flash with UBI
2) we are unaware of other big issue with MLC NAND

Unfortunately I don't really know the MTD/UBI/UBIFS internals but
(correct me if I'm wrong) to me solving the power cut issue only by
software (by using proper data structure, redundancy, NAND flash
operation jornal and so on) is pretty hard to implement inside the Linux
MTD stack.

On the other hand, stuff like scrubbing (e.g. having a task like GC that
periodically scrub the required NAND erase block), paired page (e.g.
carefully selecting LEB to PEB mapping to avoid reading/writing data
into paired page in a wrong way) are still to be implemented but have,
somehow, less impact on the whole MTD stack.

For sure I'm missing some other MLC NAND issue, but the above are the
one that I'm aware of.

WDYT?

Best Regards,

-- 

Andrea SCIAN

DAVE Embedded Systems

RE: RFC: detect and manage power cut on MLC NAND

[Jeff Lauruhn (jlauruhn)]

Power loss is actually very complex.  The Write Protect (WP) pin was added to NAND help lock the NAND when a power loss event is detected.  I have extensive information on NAND and would be happy to discuss.   


Jeff Lauruhn
NAND Application Engineer
Embedded Business Unit
Micron Technology, Inc


-----Original Message-----
From: linux-mtd [mailto:linux-mtd-bounces@lists.infradead.org] On Behalf Of Andrea Scian
Sent: Wednesday, March 11, 2015 10:01 AM
To: Richard Weinberger; dedekind1@gmail.com
Cc: mtd_mailinglist
Subject: Re: RFC: detect and manage power cut on MLC NAND


Hi all,

and thanks for you feedback.
You can find my comments below.

Il 11/03/2015 10:09, Richard Weinberger ha scritto:
> Am 11.03.2015 um 10:05 schrieb Artem Bityutskiy:
>> On Wed, 2015-03-11 at 09:57 +0100, Richard Weinberger wrote:
>>> Hi!
>>>
>>> Am 11.03.2015 um 08:20 schrieb Artem Bityutskiy:
>>>> On Tue, 2015-03-10 at 13:51 +0100, Richard Weinberger wrote:
>>>>>> WDYT about this?
>>>>>> If it sounds reasonable is there any suggestion where to place such a code?
>>>>> Customers often use DYI uninterruptible power supplies using capacitors.
>>>>> But managing a power cut is the least problem you have with MLC NAND.
>>>> Why is it the least problem, what is the hardest one? I thought 
>>>> this one is the hardest.
>>> IMHO the hardest ones are the problems we don't know yet as NAND 
>>> vendors are not really chatty about the MLC constraints.
>>> We don't know much about data retention for example. At least we 
>>> have not much hard facts. Most of our knowledge is hearsay.
>> Well, but from the problems we know paired pages seems to be the 
>> biggest one. E.g., what do we do if VID header gets corrupted because 
>> of an interrupted write to the page paired with the VID header page? 
>> Sounds like a hard problem to me.
> It is a hard problem. But at least we know about it.
>

IIUC can I summarize like the following

1) power cut is one of the big problem we know about MLC NAND flash with UBI
2) we are unaware of other big issue with MLC NAND

Unfortunately I don't really know the MTD/UBI/UBIFS internals but (correct me if I'm wrong) to me solving the power cut issue only by software (by using proper data structure, redundancy, NAND flash operation jornal and so on) is pretty hard to implement inside the Linux MTD stack.

On the other hand, stuff like scrubbing (e.g. having a task like GC that periodically scrub the required NAND erase block), paired page (e.g.
carefully selecting LEB to PEB mapping to avoid reading/writing data into paired page in a wrong way) are still to be implemented but have, somehow, less impact on the whole MTD stack.

For sure I'm missing some other MLC NAND issue, but the above are the one that I'm aware of.

WDYT?

Best Regards,

-- 

Andrea SCIAN

DAVE Embedded Systems


______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

Re: RFC: detect and manage power cut on MLC NAND

[Richard Weinberger]

Jeff,

Am 11.03.2015 um 18:23 schrieb Jeff Lauruhn (jlauruhn):
> Power loss is actually very complex.  The Write Protect (WP) pin was added to NAND help lock the NAND when a power loss event is detected.  I have extensive information on NAND and would be happy to discuss.   
> 

it would be wonderful if you could share knowledge on MLC NAND.
I'm very interested in supporting MLC in UBI (and UBIFS).
UBI was designed with SLC NAND in mind, now with MLC some things have changed.

Thanks,
//richard

> Jeff Lauruhn
> NAND Application Engineer
> Embedded Business Unit
> Micron Technology, Inc
> 
> 
> -----Original Message-----
> From: linux-mtd [mailto:linux-mtd-bounces@lists.infradead.org] On Behalf Of Andrea Scian
> Sent: Wednesday, March 11, 2015 10:01 AM
> To: Richard Weinberger; dedekind1@gmail.com
> Cc: mtd_mailinglist
> Subject: Re: RFC: detect and manage power cut on MLC NAND
> 
> 
> Hi all,
> 
> and thanks for you feedback.
> You can find my comments below.
> 
> Il 11/03/2015 10:09, Richard Weinberger ha scritto:
>> Am 11.03.2015 um 10:05 schrieb Artem Bityutskiy:
>>> On Wed, 2015-03-11 at 09:57 +0100, Richard Weinberger wrote:
>>>> Hi!
>>>>
>>>> Am 11.03.2015 um 08:20 schrieb Artem Bityutskiy:
>>>>> On Tue, 2015-03-10 at 13:51 +0100, Richard Weinberger wrote:
>>>>>>> WDYT about this?
>>>>>>> If it sounds reasonable is there any suggestion where to place such a code?
>>>>>> Customers often use DYI uninterruptible power supplies using capacitors.
>>>>>> But managing a power cut is the least problem you have with MLC NAND.
>>>>> Why is it the least problem, what is the hardest one? I thought 
>>>>> this one is the hardest.
>>>> IMHO the hardest ones are the problems we don't know yet as NAND 
>>>> vendors are not really chatty about the MLC constraints.
>>>> We don't know much about data retention for example. At least we 
>>>> have not much hard facts. Most of our knowledge is hearsay.
>>> Well, but from the problems we know paired pages seems to be the 
>>> biggest one. E.g., what do we do if VID header gets corrupted because 
>>> of an interrupted write to the page paired with the VID header page? 
>>> Sounds like a hard problem to me.
>> It is a hard problem. But at least we know about it.
>>
> 
> IIUC can I summarize like the following
> 
> 1) power cut is one of the big problem we know about MLC NAND flash with UBI
> 2) we are unaware of other big issue with MLC NAND
> 
> Unfortunately I don't really know the MTD/UBI/UBIFS internals but (correct me if I'm wrong) to me solving the power cut issue only by software (by using proper data structure, redundancy, NAND flash operation jornal and so on) is pretty hard to implement inside the Linux MTD stack.
> 
> On the other hand, stuff like scrubbing (e.g. having a task like GC that periodically scrub the required NAND erase block), paired page (e.g.
> carefully selecting LEB to PEB mapping to avoid reading/writing data into paired page in a wrong way) are still to be implemented but have, somehow, less impact on the whole MTD stack.
> 
> For sure I'm missing some other MLC NAND issue, but the above are the one that I'm aware of.
> 
> WDYT?
> 
> Best Regards,
> 

RE: RFC: detect and manage power cut on MLC NAND

[Jeff Lauruhn (jlauruhn)]

Glad to help out.  I train FAE's and customers on many aspects of NAND including MLC.  


Jeff Lauruhn
NAND Application Engineer
Embedded Business Unit
Micron Technology, Inc


-----Original Message-----
From: linux-mtd [mailto:linux-mtd-bounces@lists.infradead.org] On Behalf Of Richard Weinberger
Sent: Wednesday, March 11, 2015 10:29 AM
To: Jeff Lauruhn (jlauruhn); Andrea Scian; dedekind1@gmail.com
Cc: Boris Brezillon; mtd_mailinglist
Subject: Re: RFC: detect and manage power cut on MLC NAND

Jeff,

Am 11.03.2015 um 18:23 schrieb Jeff Lauruhn (jlauruhn):
> Power loss is actually very complex.  The Write Protect (WP) pin was added to NAND help lock the NAND when a power loss event is detected.  I have extensive information on NAND and would be happy to discuss.   
> 

it would be wonderful if you could share knowledge on MLC NAND.
I'm very interested in supporting MLC in UBI (and UBIFS).
UBI was designed with SLC NAND in mind, now with MLC some things have changed.

Thanks,
//richard

> Jeff Lauruhn
> NAND Application Engineer
> Embedded Business Unit
> Micron Technology, Inc
> 
> 
> -----Original Message-----
> From: linux-mtd [mailto:linux-mtd-bounces@lists.infradead.org] On 
> Behalf Of Andrea Scian
> Sent: Wednesday, March 11, 2015 10:01 AM
> To: Richard Weinberger; dedekind1@gmail.com
> Cc: mtd_mailinglist
> Subject: Re: RFC: detect and manage power cut on MLC NAND
> 
> 
> Hi all,
> 
> and thanks for you feedback.
> You can find my comments below.
> 
> Il 11/03/2015 10:09, Richard Weinberger ha scritto:
>> Am 11.03.2015 um 10:05 schrieb Artem Bityutskiy:
>>> On Wed, 2015-03-11 at 09:57 +0100, Richard Weinberger wrote:
>>>> Hi!
>>>>
>>>> Am 11.03.2015 um 08:20 schrieb Artem Bityutskiy:
>>>>> On Tue, 2015-03-10 at 13:51 +0100, Richard Weinberger wrote:
>>>>>>> WDYT about this?
>>>>>>> If it sounds reasonable is there any suggestion where to place such a code?
>>>>>> Customers often use DYI uninterruptible power supplies using capacitors.
>>>>>> But managing a power cut is the least problem you have with MLC NAND.
>>>>> Why is it the least problem, what is the hardest one? I thought 
>>>>> this one is the hardest.
>>>> IMHO the hardest ones are the problems we don't know yet as NAND 
>>>> vendors are not really chatty about the MLC constraints.
>>>> We don't know much about data retention for example. At least we 
>>>> have not much hard facts. Most of our knowledge is hearsay.
>>> Well, but from the problems we know paired pages seems to be the 
>>> biggest one. E.g., what do we do if VID header gets corrupted 
>>> because of an interrupted write to the page paired with the VID header page?
>>> Sounds like a hard problem to me.
>> It is a hard problem. But at least we know about it.
>>
> 
> IIUC can I summarize like the following
> 
> 1) power cut is one of the big problem we know about MLC NAND flash 
> with UBI
> 2) we are unaware of other big issue with MLC NAND
> 
> Unfortunately I don't really know the MTD/UBI/UBIFS internals but (correct me if I'm wrong) to me solving the power cut issue only by software (by using proper data structure, redundancy, NAND flash operation jornal and so on) is pretty hard to implement inside the Linux MTD stack.
> 
> On the other hand, stuff like scrubbing (e.g. having a task like GC that periodically scrub the required NAND erase block), paired page (e.g.
> carefully selecting LEB to PEB mapping to avoid reading/writing data into paired page in a wrong way) are still to be implemented but have, somehow, less impact on the whole MTD stack.
> 
> For sure I'm missing some other MLC NAND issue, but the above are the one that I'm aware of.
> 
> WDYT?
> 
> Best Regards,
> 

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

RE: RFC: detect and manage power cut on MLC NAND

[Ricard Wanderlof]

On Wed, 11 Mar 2015, Jeff Lauruhn (jlauruhn) wrote:

> IIUC can I summarize like the following
> 
> 1) power cut is one of the big problem we know about MLC NAND flash with UBI
> 2) we are unaware of other big issue with MLC NAND
> 
> Unfortunately I don't really know the MTD/UBI/UBIFS internals but 
> (correct me if I'm wrong) to me solving the power cut issue only by 
> software (by using proper data structure, redundancy, NAND flash 
> operation jornal and so on) is pretty hard to implement inside the Linux 
> MTD stack.
> 
> On the other hand, stuff like scrubbing (e.g. having a task like GC that 
> periodically scrub the required NAND erase block), paired page (e.g. 
> carefully selecting LEB to PEB mapping to avoid reading/writing data 
> into paired page in a wrong way) are still to be implemented but have, 
> somehow, less impact on the whole MTD stack.

I may be going off in the wrong direction, but there seem to be two 
principal differences between SLC and MLC nand from a software point of 
view.

1. Write endurance is much lower, like 3000 erase/write cycles for MLC
   compared to 100000 for SLC.
2. The probability of bits changing 'under your feet' are much higher,
   being it because of the data retention simply being worse due to the
   underlying technology or because of page pairing.
   - This also includes bits that look ok but which aren't e.g. from the
     result of power loss during an erase operation, which might leave
     the bits in a state which looks erased, but which on subsequent use
     will not behave accordingly.

Generally, the basic view of any memory from a software point of view is 
that if 'the data was written it will not change', which is true a lot of 
the time with SLC, given a couple of precautions (like ECC and never 
trusting an erased page unless we know the erase operation had completed 
properly), but much less so with MLC. I.e. with SLC, we can take our 'the 
data will not change' paradigm and patch on a few fixes for the relatively 
few cases where this is not true, but with MLC this will approach will 
tend to fall apart and we must rethink our approach if not from the bottom 
then a bit further down than was the case with SLC.

/Ricard
-- 
Ricard Wolf Wanderlöf                           ricardw(at)axis.com
Axis Communications AB, Lund, Sweden            www.axis.com
Phone +46 46 272 2016                           Fax +46 46 13 61 30

Re: RFC: detect and manage power cut on MLC NAND

[Richard Weinberger]

Am 11.03.2015 um 22:16 schrieb Jeff Lauruhn (jlauruhn):
> Glad to help out.  I train FAE's and customers on many aspects of NAND including MLC.  

UBI (and UBIFS) was designed with SLC NAND in mind, so far we know that we have to
address the following constraints when we want UBI on MLC NAND:

1. Avoid repeating bit patterns. This can be solved by scrambling. Boris did some great work in this area.
2. Paired pages. We'll have choose pages we write to very carefully to not loss already written data in case of a power cut.
3. Read disturb. Happens also on SLC but not that early. I'm working in this.
4. Data retention. i.e, blocks that have not been erased for a long time have to be re-erased. I'm working in this too.
5. Unstable bits (not MLC specific).
6. What did I miss?

Jeff, what do you think?
Can you point us to some hard facts? I'm specially interested in numbers on read disturb and data retention.

Thanks,
//richard

RE: RFC: detect and manage power cut on MLC NAND

[Jeff Lauruhn (jlauruhn)]

Jeff Lauruhn
NAND Application Engineer
Embedded Business Unit
Micron Technology, Inc


-----Original Message-----
From: Richard Weinberger [mailto:richard@nod.at] 
Sent: Thursday, March 12, 2015 3:28 AM
To: Jeff Lauruhn (jlauruhn)
Cc: Andrea Scian; dedekind1@gmail.com; Boris Brezillon; mtd_mailinglist
Subject: Re: RFC: detect and manage power cut on MLC NAND

Am 11.03.2015 um 22:16 schrieb Jeff Lauruhn (jlauruhn):
> Glad to help out.  I train FAE's and customers on many aspects of NAND including MLC.  

UBI (and UBIFS) was designed with SLC NAND in mind, so far we know that we have to address the following constraints when we want UBI on MLC NAND:


1. Avoid repeating bit patterns. This can be solved by scrambling. Boris did some great work in this area.
2. Paired pages. We'll have choose pages we write to very carefully to not loss already written data in case of a power cut.
	For MLC we store 4 bits in the same cell has 
3. Read disturb. Happens also on SLC but not that early. I'm working in this.
		
4. Data retention. i.e, blocks that have not been erased for a long time have to be re-erased. I'm working in this too.
	
5. Unstable bits (not MLC specific).
	Two types.  Data retention and Disturbs (read and program).   Data retention (charge loss) tends to shift left,  
6. What did I miss?

Jeff, what do you think?
Can you point us to some hard facts? I'm specially interested in numbers on read disturb and data retention.
I wish there were numbers, it would make my job a lot easier, but NAND doesn't work that way.  Data retention is dependent on process node (35nm, 25nm, 20nm 15nm for example) P/E cycles and temperature. We generally specify our NAND using JEDC standards, x numbers of years at 55C with 10% cycling.   If you apply our recommended ECC, then you will be able to store data and recover it after x numbers of years.  But temperature, P/E, process size and usage have major effects on data retention so we recommend actively managing your NAND.  This is what you do and what I find so interesting about your group.

I can speak in generalities for now, and when I get more specifics I can predict and recommend solutions.

Data Retention is characterized by a loss of charge.  We program a bit from 1 to 0 (just the opposite of what people think).  Over time the charge will leave the gate, this is normal NAND behavior.  I say that the distribution of charged cell shifts left toward uncharged. Why is SLC better than MLC?  First SLC was first and used older larger lithographies.  You could store 10's of thousands of electrons on a 40nm gate and you only had two states L0 erased (0volts) or L1 programmed (1.5v for example). If you lose a few, it didn't make much difference and there was a lot of room between 0 and 1.5 volts.  Newer processes are 20nm and MLC.  With a smaller gate there are just a few hundred electrons and they need to be disturbed in one of 4 levels L0 (0 volts), L1(.5 volts), L2(1volts) or L3 (1.5 volts).  Now adding or losing a few electrons can have a larger effect.  We determine a programmed cell by measuring between the L0 and L1.  If the levels have shifted, they can shift to the point where you can't reliably tell the difference between L0 and L1.  When there are more levels and they are closer together it makes it just that much more challenging.  


Program/Read Disturb are characterized by charge gain or a shift right.  The problem is the same.

These affects are not instantaneous, they happen over long periods of time.  Instead of trying to predict every case, we recommend actively managing NAND, and that's what your team does.  Read the data an use ECC to see how much the data has changed.  Create a threshold and when the data hits the threshold move the data to a fresh block.  


Thanks,
//richard

Re: RFC: detect and manage power cut on MLC NAND

[Boris Brezillon]

Hello Jeff,

I'm joining the discussion to ask more questions about MLC NANDs ;-).

Could you tell us more about how block wear impact the voltage level
stored in NAND cells.

1/ Are all pages in a block impacted the same way ?
2/ Is wear more likely to induce voltage increase, voltage decrease
   or is it unpredictable ?
3/ Is it possible to have more than one working voltage threshold
   (read-retry mode): I did some testing on my Hynix chip (I know you
   work for Micron but that's the only MLC chip I have :-)), and I
   managed to get less bitflips by trying another read-retry mode even
   if the previous one was allowing me to successfully fix existing
   bitflips.
4/ Do you have any numbers/statistics that could
   help us choose the more appropriate read-retry mode according to the
   number of P/E cycles ?
5/ Any other things you'd like to share regarding read-retry ?


Apart from that, we're currently trying to find the most appropriate
way to deal with paired pages, and this sounds rather complicated.
The current idea is to expose paired pages information up to the UBIFS
layer, and let UBIFS decide when it should stop writing on pages paired
with already written pages.
Moreover, we have a few pages we need to protect (UBI metadata: EC and
VID headers) in order to keep UBI/UBIFS consistent.
Do you have anything to share on this topic (ideas, solutions we should
consider, constraints we're not aware of, ...)


Thanks for your valuable information.

Best Regards,

Boris

-- 
Boris Brezillon, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

RE: RFC: detect and manage power cut on MLC NAND

[Jeff Lauruhn (jlauruhn)]

Jeff Lauruhn

-----Original Message-----
From: linux-mtd [mailto:linux-mtd-bounces@lists.infradead.org] On Behalf Of Boris Brezillon
Sent: Friday, March 13, 2015 1:32 PM
To: Jeff Lauruhn (jlauruhn)
Cc: Richard Weinberger; dedekind1@gmail.com; mtd_mailinglist; Andrea Scian
Subject: Re: RFC: detect and manage power cut on MLC NAND

Hello Jeff,

I'm joining the discussion to ask more questions about MLC NANDs ;-).

Could you tell us more about how block wear impact the voltage level stored in NAND cells.

1/ Are all pages in a block impacted the same way ?
	Yes, because of block erase, P/E cycles affect all the pages in a block.    
2/ Is wear more likely to induce voltage increase, voltage decrease
   or is it unpredictable ?   Wear is a very well known a NAND characteristic.   During P/E cycling there is a potential for electrons to get permanently trapped in the oxide.  The more P/E cycles the more electrons get trapped.  Over many P/E cycles cells well get to a point where they look permanent programmed and can't be erased or programmed.  As cells begin to fail, ECC can be used to recover the data.  If too many bits fail in page the device will respond with a FAIL status after a P/E cycle.
	
3/ Is it possible to have more than one working voltage threshold
   (read-retry mode): I did some testing on my Hynix chip (I know you
   work for Micron but that's the only MLC chip I have :-)), and I
   managed to get less bitflips by trying another read-retry mode even
   if the previous one was allowing me to successfully fix existing
   bitflips.
Read Retry is available on some newer  products.  RR was introduced to help maintain and improve data retention and P/E cycles as geometry shrinks and bit/cell increase.  If the device supports RR, we have predefined RR Options, based on the most  likely chance of success.  Start with option 1 and step through the options until you get a successful read.  The DS usually has pretty good information.  

4/ Do you have any numbers/statistics that could
   help us choose the more appropriate read-retry mode according to the
   number of P/E cycles ?  I don't have numbers or statistics, but I can tell you that the RR steps are generally defined based on known NAND behavior.  Go to the Micron website and put in this PN MT29F128G08CBCCB and you will find good information on RR.
   
5/ Any other things you'd like to share regarding read-retry ? 
RR isn't available on all devices.   From your prospective I would give them the option to use RR if it's available.  

Apart from that, we're currently trying to find the most appropriate way to deal with paired pages, and this sounds rather complicated.
The current idea is to expose paired pages information up to the UBIFS layer, and let UBIFS decide when it should stop writing on pages paired with already written pages.
Moreover, we have a few pages we need to protect (UBI metadata: EC and VID headers) in order to keep UBI/UBIFS consistent.
Do you have anything to share on this topic (ideas, solutions we should consider, constraints we're not aware of, ...)

This is one of the reasons I came to this site.  I have a great deal of device knowledge and I need to know more about how end users use the device.  

Most designs today employ power loss detection and employ elegant shutdown to the NAND.  In addition, we provide Write Protect, which provides an extra layer of protection against power loss.  There is still a chance that if the power event happens during a program to a page, the previously programmed shared page can also be corrupted.  It's not clear to me how to keep track of shared pages for every device out there.  It's not like a parameter page that you can read.  It's an interesting problem.    

Thanks for your valuable information.

Best Regards,

Boris

--
Boris Brezillon, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

Re: RFC: detect and manage power cut on MLC NAND

[Andrea Marson - DAVE Embedded Systems]

Hi Jeff,

thank you for your availability.
I would like to go back to your statement about Write Protect pin:

 > Power loss is actually very complex.  The Write Protect (WP) pin was 
 > added to NAND help lock the NAND when a power loss event is
 > detected.  I have extensive information on NAND and would be happy to 
 > discuss.

IIUC WP must respect several constraints. For example it must not be 
transitioned while NAND is busy. What if it is set low while NAND is 
ready - that is it can accept new commands - but an erase operation is 
in progress? Does the erase operation complete anyway?


Best regards,
Andrea MARSON
DAVE Embedded Systems
www.dave.eu


>
>
>
> Jeff Lauruhn
>
> -----Original Message-----
> From: linux-mtd [mailto:linux-mtd-bounces@lists.infradead.org] On Behalf Of Boris Brezillon
> Sent: Friday, March 13, 2015 1:32 PM
> To: Jeff Lauruhn (jlauruhn)
> Cc: Richard Weinberger; dedekind1@gmail.com; mtd_mailinglist; Andrea Scian
> Subject: Re: RFC: detect and manage power cut on MLC NAND
>
> Hello Jeff,
>
> I'm joining the discussion to ask more questions about MLC NANDs ;-).
>
> Could you tell us more about how block wear impact the voltage level stored in NAND cells.
>
> 1/ Are all pages in a block impacted the same way ?
> 	Yes, because of block erase, P/E cycles affect all the pages in a block.
> 2/ Is wear more likely to induce voltage increase, voltage decrease
>     or is it unpredictable ?   Wear is a very well known a NAND characteristic.   During P/E cycling there is a potential for electrons to get permanently trapped in the oxide.  The more P/E cycles the more electrons get trapped.  Over many P/E cycles cells well get to a point where they look permanent programmed and can't be erased or programmed.  As cells begin to fail, ECC can be used to recover the data.  If too many bits fail in page the device will respond with a FAIL status after a P/E cycle.
> 	
> 3/ Is it possible to have more than one working voltage threshold
>     (read-retry mode): I did some testing on my Hynix chip (I know you
>     work for Micron but that's the only MLC chip I have :-)), and I
>     managed to get less bitflips by trying another read-retry mode even
>     if the previous one was allowing me to successfully fix existing
>     bitflips.
> Read Retry is available on some newer  products.  RR was introduced to help maintain and improve data retention and P/E cycles as geometry shrinks and bit/cell increase.  If the device supports RR, we have predefined RR Options, based on the most  likely chance of success.  Start with option 1 and step through the options until you get a successful read.  The DS usually has pretty good information.
>
> 4/ Do you have any numbers/statistics that could
>     help us choose the more appropriate read-retry mode according to the
>     number of P/E cycles ?  I don't have numbers or statistics, but I can tell you that the RR steps are generally defined based on known NAND behavior.  Go to the Micron website and put in this PN MT29F128G08CBCCB and you will find good information on RR.
>
> 5/ Any other things you'd like to share regarding read-retry ?
> RR isn't available on all devices.   From your prospective I would give them the option to use RR if it's available.
>
> Apart from that, we're currently trying to find the most appropriate way to deal with paired pages, and this sounds rather complicated.
> The current idea is to expose paired pages information up to the UBIFS layer, and let UBIFS decide when it should stop writing on pages paired with already written pages.
> Moreover, we have a few pages we need to protect (UBI metadata: EC and VID headers) in order to keep UBI/UBIFS consistent.
> Do you have anything to share on this topic (ideas, solutions we should consider, constraints we're not aware of, ...)
>
> This is one of the reasons I came to this site.  I have a great deal of device knowledge and I need to know more about how end users use the device.
>
> Most designs today employ power loss detection and employ elegant shutdown to the NAND.  In addition, we provide Write Protect, which provides an extra layer of protection against power loss.  There is still a chance that if the power event happens during a program to a page, the previously programmed shared page can also be corrupted.  It's not clear to me how to keep track of shared pages for every device out there.  It's not like a parameter page that you can read.  It's an interesting problem.
>
> Thanks for your valuable information.
>
> Best Regards,
>
> Boris
>
> --
> Boris Brezillon, Free Electrons
> Embedded Linux and Kernel engineering
> http://free-electrons.com
>
> ______________________________________________________
> Linux MTD discussion mailing list
> http://lists.infradead.org/mailman/listinfo/linux-mtd/
>
> ______________________________________________________
> Linux MTD discussion mailing list
> http://lists.infradead.org/mailman/listinfo/linux-mtd/
>

Re: RFC: detect and manage power cut on MLC NAND

[Richard Weinberger]

Hi Jeff,

Am 12.03.2015 um 23:57 schrieb Jeff Lauruhn (jlauruhn):
> 
> 
> Jeff Lauruhn
> NAND Application Engineer
> Embedded Business Unit
> Micron Technology, Inc
> 
> 
> -----Original Message-----
> From: Richard Weinberger [mailto:richard@nod.at] 
> Sent: Thursday, March 12, 2015 3:28 AM
> To: Jeff Lauruhn (jlauruhn)
> Cc: Andrea Scian; dedekind1@gmail.com; Boris Brezillon; mtd_mailinglist
> Subject: Re: RFC: detect and manage power cut on MLC NAND
> 
> Am 11.03.2015 um 22:16 schrieb Jeff Lauruhn (jlauruhn):
>> Glad to help out.  I train FAE's and customers on many aspects of NAND including MLC.  
> 
> UBI (and UBIFS) was designed with SLC NAND in mind, so far we know that we have to address the following constraints when we want UBI on MLC NAND:
> 
> 
> 1. Avoid repeating bit patterns. This can be solved by scrambling. Boris did some great work in this area.
> 2. Paired pages. We'll have choose pages we write to very carefully to not loss already written data in case of a power cut.
> 	For MLC we store 4 bits in the same cell has 
> 3. Read disturb. Happens also on SLC but not that early. I'm working in this.
> 		
> 4. Data retention. i.e, blocks that have not been erased for a long time have to be re-erased. I'm working in this too.
> 	
> 5. Unstable bits (not MLC specific).
> 	Two types.  Data retention and Disturbs (read and program).   Data retention (charge loss) tends to shift left,  
> 6. What did I miss?
> 
> Jeff, what do you think?

can you please say something on the "TODO" list? Did we miss something?
Do you have kind of a design document?

> Can you point us to some hard facts? I'm specially interested in numbers on read disturb and data retention.
> I wish there were numbers, it would make my job a lot easier, but NAND doesn't work that way.  Data retention is dependent on process node (35nm, 25nm, 20nm 15nm for example) P/E cycles and temperature. We generally specify our NAND using JEDC standards, x numbers of years at 55C with 10% cycling.   If you apply our recommended ECC, then you will be able to store data and recover it after x numbers of years.  But temperature, P/E, process size and usage have major effects on data retention so we recommend actively managing your NAND.  This is what you do and what I find so interesting about your group.

But I'm sure there are some rough numbers. Do we have to expect read disturb after say 100 reads?
https://www.micron.com/~/media/Documents/Products/Presentation/flash_mem_summit_jcooke_inconvenient_truths_nand.pdf
Are the numbers on page 20 still valid?

> I can speak in generalities for now, and when I get more specifics I can predict and recommend solutions.
> 
> Data Retention is characterized by a loss of charge.  We program a bit from 1 to 0 (just the opposite of what people think).  Over time the charge will leave the gate, this is normal NAND behavior.  I say that the distribution of charged cell shifts left toward uncharged. Why is SLC better than MLC?  First SLC was first and used older larger lithographies.  You could store 10's of thousands of electrons on a 40nm gate and you only had two states L0 erased (0volts) or L1 programmed (1.5v for example). If you lose a few, it didn't make much difference and there was a lot of room between 0 and 1.5 volts.  Newer processes are 20nm and MLC.  With a smaller gate there are just a few hundred electrons and they need to be disturbed in one of 4 levels L0 (0 volts), L1(.5 volts), L2(1volts) or L3 (1.5 volts).  Now adding or losing a few electrons can have a larger effect.  We determine a programmed cell by measuring between the L0 and L1.  If the levels have shifted, they can shift to the poi
nt where you can't reliably tell the difference between L0 and L1.  When there are more levels and they are closer together it makes it just that much more challenging.  
> 
> 
> Program/Read Disturb are characterized by charge gain or a shift right.  The problem is the same.
> 
> These affects are not instantaneous, they happen over long periods of time.  Instead of trying to predict every case, we recommend actively managing NAND, and that's what your team does.  Read the data an use ECC to see how much the data has changed.  Create a threshold and when the data hits the threshold move the data to a fresh block.  

Thanks,
//richard

Re: RFC: detect and manage power cut on MLC NAND

[Boris Brezillon]

Hi Jeff,

On Fri, 13 Mar 2015 23:51:53 +0000
"Jeff Lauruhn (jlauruhn)" <jlauruhn@micron.com> wrote:
> 
> Hello Jeff,
> 
> I'm joining the discussion to ask more questions about MLC NANDs ;-).
> 
> Could you tell us more about how block wear impact the voltage level stored in NAND cells.
> 
> 1/ Are all pages in a block impacted the same way ?
> 	Yes, because of block erase, P/E cycles affect all the pages in a block.

Okay, that's what I thought.

> 2/ Is wear more likely to induce voltage increase, voltage decrease
>    or is it unpredictable ?   Wear is a very well known a NAND characteristic.   During P/E cycling there is a potential for electrons to get permanently trapped in the oxide.  The more P/E cycles the more electrons get trapped.  Over many P/E cycles cells well get to a point where they look permanent programmed and can't be erased or programmed.  As cells begin to fail, ECC can be used to recover the data.  If too many bits fail in page the device will respond with a FAIL status after a P/E cycle.

So voltage thresholds tends to increase with wear, right ?

> 	
> 3/ Is it possible to have more than one working voltage threshold
>    (read-retry mode): I did some testing on my Hynix chip (I know you
>    work for Micron but that's the only MLC chip I have :-)), and I
>    managed to get less bitflips by trying another read-retry mode even
>    if the previous one was allowing me to successfully fix existing
>    bitflips.
> Read Retry is available on some newer  products.  RR was introduced to help maintain and improve data retention and P/E cycles as geometry shrinks and bit/cell increase.  If the device supports RR, we have predefined RR Options, based on the most  likely chance of success.  Start with option 1 and step through the options until you get a successful read.  The DS usually has pretty good information.

When you say you have "predefined RR Options, based on the most  likely
chance of success", does this mean these options are internally
evolving during the NAND block lifetime, or is RR mode 0 always
encoding the same threshold config.
In the latter case, maybe we should start with a different RR mode
depending on the number of P/E cycles already done on the block, so
that we have more chance to successfully read the page on our first
read.

 

> 
> 4/ Do you have any numbers/statistics that could
>    help us choose the more appropriate read-retry mode according to the
>    number of P/E cycles ?  I don't have numbers or statistics, but I can tell you that the RR steps are generally defined based on known NAND behavior.  Go to the Micron website and put in this PN MT29F128G08CBCCB and you will find good information on RR.

Okay, I'll have a look at the datasheet you pointed out (the Hynix one
was not even talking about read-retry, I had to search in Allwinner
code to understand how to change read-retry mode).

>    
> 5/ Any other things you'd like to share regarding read-retry ? 
> RR isn't available on all devices.   From your prospective I would give them the option to use RR if it's available.

Yes, that's already done this way: we use RR on devices providing
this feature. IIRC, only Micron chips are supported so far, but I
added support for one of the Hynix chip.
The whole problem here is that each vendor implement RR in their own
way (using ONFI params for Micron, OTP area and private commands for
Hynix, and probably something else for Samsung chips).

Anyway, that's just a matter of adding a NAND chip database + vendor
specific code to deal with each read retry implementation (even if
that would have helped us a lot if chip vendors had agreed on a
standard way to control RR).

> 
> Apart from that, we're currently trying to find the most appropriate way to deal with paired pages, and this sounds rather complicated.
> The current idea is to expose paired pages information up to the UBIFS layer, and let UBIFS decide when it should stop writing on pages paired with already written pages.
> Moreover, we have a few pages we need to protect (UBI metadata: EC and VID headers) in order to keep UBI/UBIFS consistent.
> Do you have anything to share on this topic (ideas, solutions we should consider, constraints we're not aware of, ...)
> 
> This is one of the reasons I came to this site.  I have a great deal of device knowledge and I need to know more about how end users use the device.  
> 
> Most designs today employ power loss detection and employ elegant shutdown to the NAND.  In addition, we provide Write Protect, which provides an extra layer of protection against power loss.  There is still a chance that if the power event happens during a program to a page, the previously programmed shared page can also be corrupted.  It's not clear to me how to keep track of shared pages for every device out there.  It's not like a parameter page that you can read.  It's an interesting problem.

Of course, preventing page corruption is a good approach, but some
board designers are just simply not taking these constraints into
account, and detecting power loss in order to assert the WP pin is not
possible in such designs.

I think we should also find a solution to recover from corruptions
induced by paired pages write, and that's the direction we're currently
investigating.

But if someone have real examples (boards) supporting power loss
detection + WP pin control in such cases, maybe we can start thinking
about a standard way to deal with that in Linux.

Thanks again for your answers.

Best Regards,

Boris

-- 
Boris Brezillon, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

Re: RFC: detect and manage power cut on MLC NAND

[Ricard Wanderlof]

Hi Jeff,

I have a question regarding MLC:s, probably not so much something we can 
do anything about, but I'm curious just the same:

If I understand correctly, page pairing in MLC's means that of the two 
bits in a cell, one is allocated to one page and another one to a 
completely different page. This means (among other things) that rewriting 
one page may impact the other, paired, page.

My question is: why is it done this way? Is it to distribute bit flips 
more evenly?

An initial trivial allocation would otherwise be to put the paired bits in 
the same byte, for two reasons a) to avoid page-pairing issues, and b) 
because it simply would be easier to write both bits in a cell at the same 
time rather than at different times.

Granted, without page pairing, any sort of failure or disturb in one bit 
cell would would require twice the amount of ECC as both bits would likely 
be corrupted, on the other hand, we'd avoid having data in one part of the 
flash be corrupted by operations in another part of the flash.

/Ricard
-- 
Ricard Wolf Wanderlöf                           ricardw(at)axis.com
Axis Communications AB, Lund, Sweden            www.axis.com
Phone +46 46 272 2016                           Fax +46 46 13 61 30

RE: RFC: detect and manage power cut on MLC NAND

[Jeff Lauruhn (jlauruhn)]

"What if it is set low while NAND is ready - that is it can accept new commands - but an erase operation is in progress?".  Again, the likelihood is low, in this case a block that was intended to be erased my not be fully erase, it can be fully erased when power returns.  No data will lost because the block was intended to be erased anyway. 

"Does the erase operation complete anyway?"  There has been a lot of work done to mitigate power loss on NAND, but I haven't ever seen a design from any NAND vendor that is 100%.  Surprise power loss should be avoided on NAND or consider power detection and elegant shutdown circuitry.


Jeff Lauruhn
NAND Application Engineer
Embedded Business Unit


-----Original Message-----
From: Andrea Marson - DAVE Embedded Systems [mailto:andrea.marson@dave.eu] 
Sent: Saturday, March 14, 2015 2:46 AM
To: Jeff Lauruhn (jlauruhn)
Cc: Boris Brezillon; Andrea Scian; Richard Weinberger; mtd_mailinglist; dedekind1@gmail.com
Subject: Re: RFC: detect and manage power cut on MLC NAND

Hi Jeff,

thank you for your availability.
I would like to go back to your statement about Write Protect pin:

 > Power loss is actually very complex.  The Write Protect (WP) pin was  > added to NAND help lock the NAND when a power loss event is  > detected.  I have extensive information on NAND and would be happy to  > discuss.

IIUC WP must respect several constraints. For example it must not be transitioned while NAND is busy. What if it is set low while NAND is ready - that is it can accept new commands - but an erase operation is in progress? Does the erase operation complete anyway?


Best regards,
Andrea MARSON
DAVE Embedded Systems
www.dave.eu


>
>
>
> Jeff Lauruhn
>
> -----Original Message-----
> From: linux-mtd [mailto:linux-mtd-bounces@lists.infradead.org] On 
> Behalf Of Boris Brezillon
> Sent: Friday, March 13, 2015 1:32 PM
> To: Jeff Lauruhn (jlauruhn)
> Cc: Richard Weinberger; dedekind1@gmail.com; mtd_mailinglist; Andrea 
> Scian
> Subject: Re: RFC: detect and manage power cut on MLC NAND
>
> Hello Jeff,
>
> I'm joining the discussion to ask more questions about MLC NANDs ;-).
>
> Could you tell us more about how block wear impact the voltage level stored in NAND cells.
>
> 1/ Are all pages in a block impacted the same way ?
> 	Yes, because of block erase, P/E cycles affect all the pages in a block.
> 2/ Is wear more likely to induce voltage increase, voltage decrease
>     or is it unpredictable ?   Wear is a very well known a NAND characteristic.   During P/E cycling there is a potential for electrons to get permanently trapped in the oxide.  The more P/E cycles the more electrons get trapped.  Over many P/E cycles cells well get to a point where they look permanent programmed and can't be erased or programmed.  As cells begin to fail, ECC can be used to recover the data.  If too many bits fail in page the device will respond with a FAIL status after a P/E cycle.
> 	
> 3/ Is it possible to have more than one working voltage threshold
>     (read-retry mode): I did some testing on my Hynix chip (I know you
>     work for Micron but that's the only MLC chip I have :-)), and I
>     managed to get less bitflips by trying another read-retry mode even
>     if the previous one was allowing me to successfully fix existing
>     bitflips.
> Read Retry is available on some newer  products.  RR was introduced to help maintain and improve data retention and P/E cycles as geometry shrinks and bit/cell increase.  If the device supports RR, we have predefined RR Options, based on the most  likely chance of success.  Start with option 1 and step through the options until you get a successful read.  The DS usually has pretty good information.
>
> 4/ Do you have any numbers/statistics that could
>     help us choose the more appropriate read-retry mode according to the
>     number of P/E cycles ?  I don't have numbers or statistics, but I can tell you that the RR steps are generally defined based on known NAND behavior.  Go to the Micron website and put in this PN MT29F128G08CBCCB and you will find good information on RR.
>
> 5/ Any other things you'd like to share regarding read-retry ?
> RR isn't available on all devices.   From your prospective I would give them the option to use RR if it's available.
>
> Apart from that, we're currently trying to find the most appropriate way to deal with paired pages, and this sounds rather complicated.
> The current idea is to expose paired pages information up to the UBIFS layer, and let UBIFS decide when it should stop writing on pages paired with already written pages.
> Moreover, we have a few pages we need to protect (UBI metadata: EC and VID headers) in order to keep UBI/UBIFS consistent.
> Do you have anything to share on this topic (ideas, solutions we 
> should consider, constraints we're not aware of, ...)
>
> This is one of the reasons I came to this site.  I have a great deal of device knowledge and I need to know more about how end users use the device.
>
> Most designs today employ power loss detection and employ elegant shutdown to the NAND.  In addition, we provide Write Protect, which provides an extra layer of protection against power loss.  There is still a chance that if the power event happens during a program to a page, the previously programmed shared page can also be corrupted.  It's not clear to me how to keep track of shared pages for every device out there.  It's not like a parameter page that you can read.  It's an interesting problem.
>
> Thanks for your valuable information.
>
> Best Regards,
>
> Boris
>
> --
> Boris Brezillon, Free Electrons
> Embedded Linux and Kernel engineering
> http://free-electrons.com
>
> ______________________________________________________
> Linux MTD discussion mailing list
> http://lists.infradead.org/mailman/listinfo/linux-mtd/
>
> ______________________________________________________
> Linux MTD discussion mailing list
> http://lists.infradead.org/mailman/listinfo/linux-mtd/
>

RE: RFC: detect and manage power cut on MLC NAND

[Jeff Lauruhn (jlauruhn)]

Ricard;
I wish I could add some images, they would really help.  But, what you call paired pages we call lower/upper pagers.  Any yes, it's all about the distribution.  I need to look in to other vendors but under program operations for Micron we have a requirement "Within a block, pages must be programmed sequentially from the least significant page address to the most significant page address (i.e. 0, 1, 2, 3, .). Programming pages out of order within a block is prohibited." This ensures that the lower page is programmed, before the upper page and that they are programmed together.  This sequence ensures the best reliability.  When we program the lower page it gets programmed into L0 or L1 state, when we program the upper page we either move L0 or L1 to its final location and we end up with L0, L1, L2, or L3.  Programming takes longer because there are 4 levels on the same gate so tighter distribution is required.  Power loss isn't as big an issue as it was in the past, most vendors are aware and have power loss considerations, but there are still vendors who take the risk.  In the case of a power loss during a upper page program it's always a good idea to check the condition of the lower page. 

Jeff Lauruhn
NAND Application Engineer
Embedded Business Unit


-----Original Message-----
From: Ricard Wanderlof [mailto:ricard.wanderlof@axis.com] 
Sent: Monday, March 16, 2015 2:02 AM
To: Jeff Lauruhn (jlauruhn)
Cc: mtd_mailinglist
Subject: Re: RFC: detect and manage power cut on MLC NAND


Hi Jeff,

I have a question regarding MLC:s, probably not so much something we can do anything about, but I'm curious just the same:

If I understand correctly, page pairing in MLC's means that of the two bits in a cell, one is allocated to one page and another one to a completely different page. This means (among other things) that rewriting one page may impact the other, paired, page.


My question is: why is it done this way? Is it to distribute bit flips more evenly?

An initial trivial allocation would otherwise be to put the paired bits in the same byte, for two reasons a) to avoid page-pairing issues, and b) because it simply would be easier to write both bits in a cell at the same time rather than at different times.

Granted, without page pairing, any sort of failure or disturb in one bit cell would would require twice the amount of ECC as both bits would likely be corrupted, on the other hand, we'd avoid having data in one part of the flash be corrupted by operations in another part of the flash.

/Ricard
-- 
Ricard Wolf Wanderlöf                           ricardw(at)axis.com
Axis Communications AB, Lund, Sweden            www.axis.com
Phone +46 46 272 2016                           Fax +46 46 13 61 30

RE: RFC: detect and manage power cut on MLC NAND

[Jeff Lauruhn (jlauruhn)]

Good morning Boris;
RR is a new feature and not available on all parts few.  I'm not sure about others, but since these are features, you simply enable of disable via SET FEATURE/GET FEATURE.  If you already provide that SET/GET FEATURE functionality then an end-user determine if their device supports a feature and then write the code to enable when they need it on their particular design.


Jeff Lauruhn
NAND Application Engineer
Embedded Business Unit


-----Original Message-----
From: Boris Brezillon [mailto:boris.brezillon@free-electrons.com] 
Sent: Saturday, March 14, 2015 3:32 AM
To: Jeff Lauruhn (jlauruhn)
Cc: Richard Weinberger; dedekind1@gmail.com; mtd_mailinglist; Andrea Scian
Subject: Re: RFC: detect and manage power cut on MLC NAND

Hi Jeff,

On Fri, 13 Mar 2015 23:51:53 +0000
"Jeff Lauruhn (jlauruhn)" <jlauruhn@micron.com> wrote:
> 
> Hello Jeff,
> 
> I'm joining the discussion to ask more questions about MLC NANDs ;-).
> 
> Could you tell us more about how block wear impact the voltage level stored in NAND cells.
> 
> 1/ Are all pages in a block impacted the same way ?
> 	Yes, because of block erase, P/E cycles affect all the pages in a block.

Okay, that's what I thought.

> 2/ Is wear more likely to induce voltage increase, voltage decrease
>    or is it unpredictable ?   Wear is a very well known a NAND characteristic.   During P/E cycling there is a potential for electrons to get permanently trapped in the oxide.  The more P/E cycles the more electrons get trapped.  Over many P/E cycles cells well get to a point where they look permanent programmed and can't be erased or programmed.  As cells begin to fail, ECC can be used to recover the data.  If too many bits fail in page the device will respond with a FAIL status after a P/E cycle.

So voltage thresholds tends to increase with wear, right ?

> 	
> 3/ Is it possible to have more than one working voltage threshold
>    (read-retry mode): I did some testing on my Hynix chip (I know you
>    work for Micron but that's the only MLC chip I have :-)), and I
>    managed to get less bitflips by trying another read-retry mode even
>    if the previous one was allowing me to successfully fix existing
>    bitflips.
> Read Retry is available on some newer  products.  RR was introduced to help maintain and improve data retention and P/E cycles as geometry shrinks and bit/cell increase.  If the device supports RR, we have predefined RR Options, based on the most  likely chance of success.  Start with option 1 and step through the options until you get a successful read.  The DS usually has pretty good information.

When you say you have "predefined RR Options, based on the most  likely chance of success", does this mean these options are internally evolving during the NAND block lifetime, or is RR mode 0 always encoding the same threshold config.
In the latter case, maybe we should start with a different RR mode depending on the number of P/E cycles already done on the block, so that we have more chance to successfully read the page on our first read.

 

> 
> 4/ Do you have any numbers/statistics that could
>    help us choose the more appropriate read-retry mode according to the
>    number of P/E cycles ?  I don't have numbers or statistics, but I can tell you that the RR steps are generally defined based on known NAND behavior.  Go to the Micron website and put in this PN MT29F128G08CBCCB and you will find good information on RR.

Okay, I'll have a look at the datasheet you pointed out (the Hynix one was not even talking about read-retry, I had to search in Allwinner code to understand how to change read-retry mode).

>    
> 5/ Any other things you'd like to share regarding read-retry ? 
> RR isn't available on all devices.   From your prospective I would give them the option to use RR if it's available.

Yes, that's already done this way: we use RR on devices providing this feature. IIRC, only Micron chips are supported so far, but I added support for one of the Hynix chip.
The whole problem here is that each vendor implement RR in their own way (using ONFI params for Micron, OTP area and private commands for Hynix, and probably something else for Samsung chips).

Anyway, that's just a matter of adding a NAND chip database + vendor specific code to deal with each read retry implementation (even if that would have helped us a lot if chip vendors had agreed on a standard way to control RR).

> 
> Apart from that, we're currently trying to find the most appropriate way to deal with paired pages, and this sounds rather complicated.
> The current idea is to expose paired pages information up to the UBIFS layer, and let UBIFS decide when it should stop writing on pages paired with already written pages.
> Moreover, we have a few pages we need to protect (UBI metadata: EC and VID headers) in order to keep UBI/UBIFS consistent.
> Do you have anything to share on this topic (ideas, solutions we 
> should consider, constraints we're not aware of, ...)
> 
> This is one of the reasons I came to this site.  I have a great deal of device knowledge and I need to know more about how end users use the device.  
> 
> Most designs today employ power loss detection and employ elegant shutdown to the NAND.  In addition, we provide Write Protect, which provides an extra layer of protection against power loss.  There is still a chance that if the power event happens during a program to a page, the previously programmed shared page can also be corrupted.  It's not clear to me how to keep track of shared pages for every device out there.  It's not like a parameter page that you can read.  It's an interesting problem.

Of course, preventing page corruption is a good approach, but some board designers are just simply not taking these constraints into account, and detecting power loss in order to assert the WP pin is not possible in such designs.

I think we should also find a solution to recover from corruptions induced by paired pages write, and that's the direction we're currently investigating.

But if someone have real examples (boards) supporting power loss detection + WP pin control in such cases, maybe we can start thinking about a standard way to deal with that in Linux.

Thanks again for your answers.

Best Regards,

Boris

--
Boris Brezillon, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

Re: RFC: detect and manage power cut on MLC NAND

[Andrea Scian]

Dear Jeff,

Il 16/03/2015 17:02, Jeff Lauruhn (jlauruhn) ha scritto:
> "What if it is set low while NAND is ready - that is it can accept new commands - but an erase operation is in progress?".  Again, the likelihood is low, in this case a block that was intended to be erased my not be fully erase, it can be fully erased when power returns.  No data will lost because the block was intended to be erased anyway. 

AFAIK the real problem here is how to detect that the in-progress erase
has not been terminated completely
The block seems to be erased (it contains all 0xFF) but it can lead to
some failure when writing or reading
(I have some indirect experience in it, when doing some power-cut test
on MLC but I cannot say for sure that it was caused by not completed
erase operation. See also this reference
http://www.linux-mtd.infradead.org/doc/ubifs.html#L_unstable_bits)

> "Does the erase operation complete anyway?"  There has been a lot of work done to mitigate power loss on NAND, but I haven't ever seen a design from any NAND vendor that is 100%.  Surprise power loss should be avoided on NAND or consider power detection and elegant shutdown circuitry.

IIUC we have two workaround here:
- power detection and clean shutdown. This implies, of course, some
hardware implementation and a piece of software able to intercept the
hardware event, block all the NAND operation apart the one that is
currently running. This is the topic of my initial RFC. I got a first
implementation but, IMHO, is too strictly related to the NAND controller
(I've placed the event handler inside it and just lock the NAND access
by hacking chip->select_chip())
I think that power fail detection can be useful in some other context
too inside Linux kernel, so probably it may have a more general
implementation. However (and unfortunately) I'm not so involved in
mainline kernel so I don't really know if this topic has been discussed
or not

- detection of interrupted operations: having some kind of journal that
record the last running operation and fix it on next reboot. AFAIK this
is what commercial FTL and flash memory controller do, using a lot of
patented piece of software.
I'm not so aware of UBI/MTD internals but I think this is hard to
implement using a general approach. It should be easier to implement
such a logic having some additional hardware support (the first that
come into my mind is battery backed SRAM, e.g. the one that we find
inside RTC devices)

Best Regards,

-- 

Andrea SCIAN

DAVE Embedded Systems

Re: RFC: detect and manage power cut on MLC NAND

[Andrea Scian]

Dear Jeff,

Il 16/03/2015 22:11, Jeff Lauruhn (jlauruhn) ha scritto:
> Good morning Boris;
> RR is a new feature and not available on all parts few.  I'm not sure 
> about others, but since these are features, you simply enable of 
> disable via SET FEATURE/GET FEATURE.  If you already provide that 
> SET/GET FEATURE functionality then an end-user determine if their 
> device supports a feature and then write the code to enable when they 
> need it on their particular design.

I can confirm this. In fact I'm currently working with two Micron NAND:

MT29F32G08CBACAWP
MT29F32G08CBADAWP

The latter should be "just" a newer die revision of the former (at
least, this is what our distributor says)

There's a technology change between the two and, in fact, the latter
supports RR while there's no mention of such a feature inside rev C.

Jeff, could you please help me in understanding which if the following
sentences are true and which are false?
- rev D is more "robust" than rev C because it has RR (so an additional
feature that improve error correction)
- rev D is "robust" like rev C, if rev D is used with RR
- if RR is not used rev D is more error prone than rev C

I think this is crucial to understand how RR works and how much is
needed inside MTD/UBI code.
I hope that the above information are not under NDA ;-)

Thanks in advance,

--

Andrea SCIAN

DAVE Embedded Systems

Re: RFC: detect and manage power cut on MLC NAND

[Boris Brezillon]

Hi Andrea,

I'll let Jeff answer this question, but I'd like to share my
understanding.

On Tue, 17 Mar 2015 10:30:30 +0100
Andrea Scian <rnd4@dave-tech.it> wrote:

> 
> 
> Dear Jeff,
> 
> Il 16/03/2015 22:11, Jeff Lauruhn (jlauruhn) ha scritto:
> > Good morning Boris;
> > RR is a new feature and not available on all parts few.  I'm not sure 
> > about others, but since these are features, you simply enable of 
> > disable via SET FEATURE/GET FEATURE.  If you already provide that 
> > SET/GET FEATURE functionality then an end-user determine if their 
> > device supports a feature and then write the code to enable when they 
> > need it on their particular design.
> 
> I can confirm this. In fact I'm currently working with two Micron NAND:
> 
> MT29F32G08CBACAWP
> MT29F32G08CBADAWP
> 
> The latter should be "just" a newer die revision of the former (at
> least, this is what our distributor says)
> 
> There's a technology change between the two and, in fact, the latter
> supports RR while there's no mention of such a feature inside rev C.
> 
> Jeff, could you please help me in understanding which if the following
> sentences are true and which are false?
> - rev D is more "robust" than rev C because it has RR (so an additional
> feature that improve error correction)
> - rev D is "robust" like rev C, if rev D is used with RR
> - if RR is not used rev D is more error prone than rev C

RR shouldn't change NAND robustness (or sensitivity to read/write
disturbance generating bitflips).

AFAIU RR will help you improve your NAND lifetime, because you're
allowed to change voltage thresholds which means you can fix errors
that were previously considered as unfixable and lead to blocks being
marked bad earlier.

I'll let Jeff correct me if I'm wrong ;-).

> 
> I think this is crucial to understand how RR works and how much is
> needed inside MTD/UBI code.

Hopefully this can all be handled in the MTD layer, with some help
from the UBI layer to feed the wear information (number of P/E cycles
on each block).

Best Regards,

Boris

-- 
Boris Brezillon, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

RE: RFC: detect and manage power cut on MLC NAND

[Jeff Lauruhn (jlauruhn)]

Very nice explanation!  Not sure if I could have done better myself.  

Jeff Lauruhn
NAND Application Engineer
Embedded Business Unit
Micron Technology, Inc


-----Original Message-----
From: Boris Brezillon [mailto:boris.brezillon@free-electrons.com] 
Sent: Tuesday, March 17, 2015 3:02 AM
To: Andrea Scian
Cc: Jeff Lauruhn (jlauruhn); Richard Weinberger; dedekind1@gmail.com; mtd_mailinglist
Subject: Re: RFC: detect and manage power cut on MLC NAND

Hi Andrea,

I'll let Jeff answer this question, but I'd like to share my understanding.

On Tue, 17 Mar 2015 10:30:30 +0100
Andrea Scian <rnd4@dave-tech.it> wrote:

> 
> 
> Dear Jeff,
> 
> Il 16/03/2015 22:11, Jeff Lauruhn (jlauruhn) ha scritto:
> > Good morning Boris;
> > RR is a new feature and not available on all parts few.  I'm not 
> > sure about others, but since these are features, you simply enable 
> > of disable via SET FEATURE/GET FEATURE.  If you already provide that 
> > SET/GET FEATURE functionality then an end-user determine if their 
> > device supports a feature and then write the code to enable when 
> > they need it on their particular design.
> 
> I can confirm this. In fact I'm currently working with two Micron NAND:
> 
> MT29F32G08CBACAWP
> MT29F32G08CBADAWP
> 
> The latter should be "just" a newer die revision of the former (at 
> least, this is what our distributor says)
> 
> There's a technology change between the two and, in fact, the latter 
> supports RR while there's no mention of such a feature inside rev C.
> 
> Jeff, could you please help me in understanding which if the following 
> sentences are true and which are false?
> - rev D is more "robust" than rev C because it has RR (so an 
> additional feature that improve error correction)
> - rev D is "robust" like rev C, if rev D is used with RR
> - if RR is not used rev D is more error prone than rev C

RR shouldn't change NAND robustness (or sensitivity to read/write disturbance generating bitflips).

AFAIU RR will help you improve your NAND lifetime, because you're allowed to change voltage thresholds which means you can fix errors that were previously considered as unfixable and lead to blocks being marked bad earlier.

I'll let Jeff correct me if I'm wrong ;-).

> 
> I think this is crucial to understand how RR works and how much is 
> needed inside MTD/UBI code.

Hopefully this can all be handled in the MTD layer, with some help from the UBI layer to feed the wear information (number of P/E cycles on each block).

Best Regards,

Boris

--
Boris Brezillon, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

RE: RFC: detect and manage power cut on MLC NAND

[Jeff Lauruhn (jlauruhn)]

Yes, MT29F32G08CBADAWP is available under NDA.  These are actually two different process nodes.  MT29F32G08CBADAWP is one of our latest processes.  


Jeff Lauruhn
NAND Application Engineer
Embedded Business Unit


-----Original Message-----
From: Andrea Scian [mailto:rnd4@dave-tech.it] 
Sent: Tuesday, March 17, 2015 2:30 AM
To: Jeff Lauruhn (jlauruhn)
Cc: Boris Brezillon; Richard Weinberger; dedekind1@gmail.com; mtd_mailinglist
Subject: Re: RFC: detect and manage power cut on MLC NAND



Dear Jeff,

Il 16/03/2015 22:11, Jeff Lauruhn (jlauruhn) ha scritto:
> Good morning Boris;
> RR is a new feature and not available on all parts few.  I'm not sure 
> about others, but since these are features, you simply enable of 
> disable via SET FEATURE/GET FEATURE.  If you already provide that 
> SET/GET FEATURE functionality then an end-user determine if their 
> device supports a feature and then write the code to enable when they 
> need it on their particular design.

I can confirm this. In fact I'm currently working with two Micron NAND:

MT29F32G08CBACAWP
MT29F32G08CBADAWP

The latter should be "just" a newer die revision of the former (at least, this is what our distributor says)

There's a technology change between the two and, in fact, the latter supports RR while there's no mention of such a feature inside rev C.

Jeff, could you please help me in understanding which if the following sentences are true and which are false?
- rev D is more "robust" than rev C because it has RR (so an additional feature that improve error correction)
- rev D is "robust" like rev C, if rev D is used with RR
- if RR is not used rev D is more error prone than rev C

I think this is crucial to understand how RR works and how much is needed inside MTD/UBI code.
I hope that the above information are not under NDA ;-)

Thanks in advance,

--

Andrea SCIAN

DAVE Embedded Systems

Re: RFC: detect and manage power cut on MLC NAND (linux-mtd Digest, Vol 144, Issue 70)

[Andrea Marson]

Hello,

I would like to discuss about another couple of topics: partitioning and 
SLC emulation.

1) IIUC read/program disturb effects exhibit at block level.
In a typical embedded linux systems there are software parts - 
bootloader, kernel image etc. - that virtually are never changed (almost 
...) but are read many times. Other parts - application libraries, log 
files etc. - are read and wrote many times instead.
If these two kinds of software are stored in different MTD partitions - 
ket's say partition A for bootloader, kernel etc. and partition B for 
application libraries, log files etc. - can we say that read/write 
operations performed on partition B have no disturb effects on partition A?

2) IIUC Boris has worked on SLC emulation too. This seems to be a 
promising feature because it would allow to partition NAND flash and to 
create higher reliability partition (at the cost of halving the size). 
Is it possibile to implement such functionality in software stack 
only(MTD/UBI) or is it necessary that NAND memory supports specific 
features?

Regards,
Andrea Marson


> Very nice explanation!  Not sure if I could have done better myself.
>
> Jeff Lauruhn
> NAND Application Engineer
> Embedded Business Unit
> Micron Technology, Inc
>
>
> -----Original Message-----
> From: Boris Brezillon [mailto:boris.brezillon@free-electrons.com]
> Sent: Tuesday, March 17, 2015 3:02 AM
> To: Andrea Scian
> Cc: Jeff Lauruhn (jlauruhn); Richard Weinberger; dedekind1@gmail.com; mtd_mailinglist
> Subject: Re: RFC: detect and manage power cut on MLC NAND
>
> Hi Andrea,
>
> I'll let Jeff answer this question, but I'd like to share my understanding.
>
> On Tue, 17 Mar 2015 10:30:30 +0100
> Andrea Scian <rnd4@dave-tech.it> wrote:
>
>>
>>
>> Dear Jeff,
>>
>> Il 16/03/2015 22:11, Jeff Lauruhn (jlauruhn) ha scritto:
>>> Good morning Boris;
>>> RR is a new feature and not available on all parts few.  I'm not
>>> sure about others, but since these are features, you simply enable
>>> of disable via SET FEATURE/GET FEATURE.  If you already provide that
>>> SET/GET FEATURE functionality then an end-user determine if their
>>> device supports a feature and then write the code to enable when
>>> they need it on their particular design.
>>
>> I can confirm this. In fact I'm currently working with two Micron NAND:
>>
>> MT29F32G08CBACAWP
>> MT29F32G08CBADAWP
>>
>> The latter should be "just" a newer die revision of the former (at
>> least, this is what our distributor says)
>>
>> There's a technology change between the two and, in fact, the latter
>> supports RR while there's no mention of such a feature inside rev C.
>>
>> Jeff, could you please help me in understanding which if the following
>> sentences are true and which are false?
>> - rev D is more "robust" than rev C because it has RR (so an
>> additional feature that improve error correction)
>> - rev D is "robust" like rev C, if rev D is used with RR
>> - if RR is not used rev D is more error prone than rev C
>
> RR shouldn't change NAND robustness (or sensitivity to read/write disturbance generating bitflips).
>
> AFAIU RR will help you improve your NAND lifetime, because you're allowed to change voltage thresholds which means you can fix errors that were previously considered as unfixable and lead to blocks being marked bad earlier.
>
> I'll let Jeff correct me if I'm wrong ;-).
>
>>
>> I think this is crucial to understand how RR works and how much is
>> needed inside MTD/UBI code.
>
> Hopefully this can all be handled in the MTD layer, with some help from the UBI layer to feed the wear information (number of P/E cycles on each block).
>
> Best Regards,
>
> Boris
>
> --
> Boris Brezillon, Free Electrons
> Embedded Linux and Kernel engineering
> http://free-electrons.com
>
>
>

Re: RFC: detect and manage power cut on MLC NAND (linux-mtd Digest, Vol 144, Issue 70)

[Boris Brezillon]

Hi Andrea,

On Wed, 18 Mar 2015 09:45:18 +0100
Andrea Marson <andrea.marson@dave.eu> wrote:

> Hello,
> 
> I would like to discuss about another couple of topics: partitioning and 
> SLC emulation.
> 
> 1) IIUC read/program disturb effects exhibit at block level.
> In a typical embedded linux systems there are software parts - 
> bootloader, kernel image etc. - that virtually are never changed (almost 
> ...) but are read many times. Other parts - application libraries, log 
> files etc. - are read and wrote many times instead.
> If these two kinds of software are stored in different MTD partitions - 
> ket's say partition A for bootloader, kernel etc. and partition B for 
> application libraries, log files etc. - can we say that read/write 
> operations performed on partition B have no disturb effects on partition A?

AFAIK, read/write disturb effects only occur on pages of the same
block, so we shouldn't see bitflips on partition A caused by read/write
on partition B.

> 
> 2) IIUC Boris has worked on SLC emulation too.

No I didn't. What makes you think I did ?

> This seems to be a 
> promising feature because it would allow to partition NAND flash and to 
> create higher reliability partition (at the cost of halving the size). 
> Is it possibile to implement such functionality in software stack 
> only(MTD/UBI) or is it necessary that NAND memory supports specific 
> features?

I guess you're talking about the slides for my ELC talk.
Note that I'm just talking about hypothetical changes, and no
development has been started on this topic yet.

IIUC, you want to select SLC/MLC mode on a per-partition basis, that
should be feasible, and this would be particularly useful for bootloader
partitions (where UBI is not involved).

Best Regards,

Boris

-- 
Boris Brezillon, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

Re: RFC: detect and manage power cut on MLC NAND (linux-mtd Digest, Vol 144, Issue 70)

[Andrea Marson]

>> 2) IIUC Boris has worked on SLC emulation too.
> No I didn't. What makes you think I did ?
My bad. I misunderstood a sentence in your slides for ELC talk :(

>
>> This seems to be a
>> promising feature because it would allow to partition NAND flash and to
>> create higher reliability partition (at the cost of halving the size).
>> Is it possibile to implement such functionality in software stack
>> only(MTD/UBI) or is it necessary that NAND memory supports specific
>> features?
> I guess you're talking about the slides for my ELC talk.
> Note that I'm just talking about hypothetical changes, and no
> development has been started on this topic yet.
I see.


>
> IIUC, you want to select SLC/MLC mode on a per-partition basis, that
> should be feasible, and this would be particularly useful for bootloader
> partitions (where UBI is not involved).
That's exactly what I was thinking about.
However it is still not clear to me how this should be implemented. I 
have read a datasheet saying that "SLC mode" consists of halving block 
addresses, that is higher bit of block address lines must be set to low. 
I wonder if this strategy prevents paired page related issues. I think 
one should work at page addressing level instead, in order to use pages 
that are not paired only. Do you think it makes sense?

Andrea

Re: RFC: detect and manage power cut on MLC NAND (linux-mtd Digest, Vol 144, Issue 70)

[Boris Brezillon]

On Wed, 18 Mar 2015 10:56:15 +0100
Andrea Marson <andrea.marson@dave.eu> wrote:

> >
> > IIUC, you want to select SLC/MLC mode on a per-partition basis, that
> > should be feasible, and this would be particularly useful for bootloader
> > partitions (where UBI is not involved).
> That's exactly what I was thinking about.
> However it is still not clear to me how this should be implemented. I 
> have read a datasheet saying that "SLC mode" consists of halving block 
> addresses, that is higher bit of block address lines must be set to low. 
> I wonder if this strategy prevents paired page related issues. I think 
> one should work at page addressing level instead, in order to use pages 
> that are not paired only. Do you think it makes sense?

Yes, actually implementing the SLC mode in software is quite easy, the
only thing we need is a description of which pages are paired together
in a block.
Moreover, by doing that in SW we won't have to deal with each vendor
implementation of "SLC mode".



-- 
Boris Brezillon, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

Re: RFC: detect and manage power cut on MLC NAND (linux-mtd Digest, Vol 144, Issue 70)

[Richard Weinberger]

Am 18.03.2015 um 10:06 schrieb Boris Brezillon:
>> 1) IIUC read/program disturb effects exhibit at block level.
>> In a typical embedded linux systems there are software parts - 
>> bootloader, kernel image etc. - that virtually are never changed (almost 
>> ...) but are read many times. Other parts - application libraries, log 
>> files etc. - are read and wrote many times instead.
>> If these two kinds of software are stored in different MTD partitions - 
>> ket's say partition A for bootloader, kernel etc. and partition B for 
>> application libraries, log files etc. - can we say that read/write 
>> operations performed on partition B have no disturb effects on partition A?
> 
> AFAIK, read/write disturb effects only occur on pages of the same
> block, so we shouldn't see bitflips on partition A caused by read/write
> on partition B.

I already saw corruptions on nearby blocks but I'm not sure if really
plain read disturb was the root cause as this NAND chips showed in general
funny symptoms.

Jeff, can you tell us more?

Thanks,
//richard

RE: RFC: detect and manage power cut on MLC NAND (linux-mtd Digest, Vol 144, Issue 70)

[Jeff Lauruhn (jlauruhn)]

Disturb is a block level affect, as long as partition A and B are in different blocks there will be no disturb between them.   Disturbs, does not damage cells; ERASE returns cells to undisturbed levels.  Disturbed bits are effectively managed with error correction codes (ECC).

Officially I would say don't use SLC emulation, but technically I know what your doing.   The reason I say no is because we have very precise recipes designed to create very tight distibutions, and although the first pass distributions might look like an SLC, they are really designed with the expectation of the upper page being programmed.  Not a true SLC. 

With MLC lithography of 25 nm and less  the difference between each level (L0, L1, L2 and L3) is just a few 10's of electrons.  The distribution have to be very tight, in order to meet retention requirements.

Jeff Lauruhn
NAND Application Engineer
Embedded Business Unit


-----Original Message-----
From: Andrea Marson [mailto:andrea.marson@dave.eu] 
Sent: Wednesday, March 18, 2015 1:45 AM
To: Boris Brezillon; Jeff Lauruhn (jlauruhn)
Cc: linux-mtd@lists.infradead.org; Andrea Scian; Richard Weinberger; dedekind1@gmail.com
Subject: Re: RFC: detect and manage power cut on MLC NAND (linux-mtd Digest, Vol 144, Issue 70)

Hello,

I would like to discuss about another couple of topics: partitioning and SLC emulation.

1) IIUC read/program disturb effects exhibit at block level.
In a typical embedded linux systems there are software parts - bootloader, kernel image etc. - that virtually are never changed (almost
...) but are read many times. Other parts - application libraries, log files etc. - are read and wrote many times instead.
If these two kinds of software are stored in different MTD partitions - ket's say partition A for bootloader, kernel etc. and partition B for application libraries, log files etc. - can we say that read/write operations performed on partition B have no disturb effects on partition A?

2) IIUC Boris has worked on SLC emulation too. This seems to be a promising feature because it would allow to partition NAND flash and to create higher reliability partition (at the cost of halving the size). 
Is it possibile to implement such functionality in software stack
only(MTD/UBI) or is it necessary that NAND memory supports specific features?

Regards,
Andrea Marson


> Very nice explanation!  Not sure if I could have done better myself.
>
> Jeff Lauruhn
> NAND Application Engineer
> Embedded Business Unit
> Micron Technology, Inc
>
>
> -----Original Message-----
> From: Boris Brezillon [mailto:boris.brezillon@free-electrons.com]
> Sent: Tuesday, March 17, 2015 3:02 AM
> To: Andrea Scian
> Cc: Jeff Lauruhn (jlauruhn); Richard Weinberger; dedekind1@gmail.com; 
> mtd_mailinglist
> Subject: Re: RFC: detect and manage power cut on MLC NAND
>
> Hi Andrea,
>
> I'll let Jeff answer this question, but I'd like to share my understanding.
>
> On Tue, 17 Mar 2015 10:30:30 +0100
> Andrea Scian <rnd4@dave-tech.it> wrote:
>
>>
>>
>> Dear Jeff,
>>
>> Il 16/03/2015 22:11, Jeff Lauruhn (jlauruhn) ha scritto:
>>> Good morning Boris;
>>> RR is a new feature and not available on all parts few.  I'm not 
>>> sure about others, but since these are features, you simply enable 
>>> of disable via SET FEATURE/GET FEATURE.  If you already provide that 
>>> SET/GET FEATURE functionality then an end-user determine if their 
>>> device supports a feature and then write the code to enable when 
>>> they need it on their particular design.
>>
>> I can confirm this. In fact I'm currently working with two Micron NAND:
>>
>> MT29F32G08CBACAWP
>> MT29F32G08CBADAWP
>>
>> The latter should be "just" a newer die revision of the former (at 
>> least, this is what our distributor says)
>>
>> There's a technology change between the two and, in fact, the latter 
>> supports RR while there's no mention of such a feature inside rev C.
>>
>> Jeff, could you please help me in understanding which if the 
>> following sentences are true and which are false?
>> - rev D is more "robust" than rev C because it has RR (so an 
>> additional feature that improve error correction)
>> - rev D is "robust" like rev C, if rev D is used with RR
>> - if RR is not used rev D is more error prone than rev C
>
> RR shouldn't change NAND robustness (or sensitivity to read/write disturbance generating bitflips).
>
> AFAIU RR will help you improve your NAND lifetime, because you're allowed to change voltage thresholds which means you can fix errors that were previously considered as unfixable and lead to blocks being marked bad earlier.
>
> I'll let Jeff correct me if I'm wrong ;-).
>
>>
>> I think this is crucial to understand how RR works and how much is 
>> needed inside MTD/UBI code.
>
> Hopefully this can all be handled in the MTD layer, with some help from the UBI layer to feed the wear information (number of P/E cycles on each block).
>
> Best Regards,
>
> Boris
>
> --
> Boris Brezillon, Free Electrons
> Embedded Linux and Kernel engineering
> http://free-electrons.com
>
>
>

RE: RFC: detect and manage power cut on MLC NAND (linux-mtd Digest, Vol 144, Issue 70)

[Jeff Lauruhn (jlauruhn)]

Disturb is a block level affect, but in operation data is often layout in continuous blocks, so if one block is seeing a lot of reads or P/E cycles, there's a high probability the blocks around it are also seeing very similar read and P/E patterns. 



Jeff Lauruhn
NAND Application Engineer
Embedded Business Unit



-----Original Message-----
From: Richard Weinberger [mailto:richard@nod.at] 
Sent: Wednesday, March 18, 2015 5:07 AM
To: Boris Brezillon; Andrea Marson
Cc: Jeff Lauruhn (jlauruhn); linux-mtd@lists.infradead.org; Andrea Scian; dedekind1@gmail.com
Subject: Re: RFC: detect and manage power cut on MLC NAND (linux-mtd Digest, Vol 144, Issue 70)

Am 18.03.2015 um 10:06 schrieb Boris Brezillon:
>> 1) IIUC read/program disturb effects exhibit at block level.
>> In a typical embedded linux systems there are software parts - 
>> bootloader, kernel image etc. - that virtually are never changed 
>> (almost
>> ...) but are read many times. Other parts - application libraries, 
>> log files etc. - are read and wrote many times instead.
>> If these two kinds of software are stored in different MTD partitions 
>> - ket's say partition A for bootloader, kernel etc. and partition B 
>> for application libraries, log files etc. - can we say that 
>> read/write operations performed on partition B have no disturb effects on partition A?
> 
> AFAIK, read/write disturb effects only occur on pages of the same 
> block, so we shouldn't see bitflips on partition A caused by 
> read/write on partition B.

I already saw corruptions on nearby blocks but I'm not sure if really plain read disturb was the root cause as this NAND chips showed in general funny symptoms.

Jeff, can you tell us more?

Thanks,
//richard

Re: RFC: detect and manage power cut on MLC NAND

[Andrea Marson]

> Disturb is a block level affect, as long as partition A and B are in different blocks there will be no disturb between them.   Disturbs, does not damage cells; ERASE returns cells to undisturbed levels.
I think there are two options here: MTD partitioning and UBI 
partitioning. AFAIK one should prefer UBI partitioning to preserve 
device-wide wear leveling. Boris, am I right?

> Officially I would say don't use SLC emulation, but technically I know what your doing.   The reason I say no is because we have very precise recipes designed to create very tight distibutions, and although the first pass distributions might look like an SLC, they are really designed with the expectation of the upper page being programmed.  Not a true SLC.
> With MLC lithography of 25 nm and less  the difference between each level (L0, L1, L2 and L3) is just a few 10's of electrons.  The distribution have to be very tight, in order to meet retention requirements.
This is quite interesting, however I'm afraid I have not fully 
understood it. Let me try to rephrase it. Please correct me if I'm wrong.

1) Technically speaking, it is possible to use an MLC memory in SLC 
mode, even if this is not recommended because MLC is not designed for 
this usage.

2) As indicated by Boris, the easiest way to implement this thing is to 
avoid the use of paired pages, according to paired page table provided 
by datasheet.

3) This technique does not transform an MLC NAND to an SLC magically. 
Thus data retention and lifetime are not increased. However all paired 
pages issues disappear. It is not clear if there are further drawbacks 
that reduce flash reliability if used this way.

Thank you,
Andrea

Re: RFC: detect and manage power cut on MLC NAND

[Boris Brezillon]

On Thu, 19 Mar 2015 09:47:21 +0100
Andrea Marson <andrea.marson@dave.eu> wrote:

> > Disturb is a block level affect, as long as partition A and B are in different blocks there will be no disturb between them.   Disturbs, does not damage cells; ERASE returns cells to undisturbed levels.
> I think there are two options here: MTD partitioning and UBI 
> partitioning. AFAIK one should prefer UBI partitioning to preserve 
> device-wide wear leveling. Boris, am I right?

Both of them act at block level, meaning that your the partition size
must be a multiple of the block size (logical block size in case of UBI
volume and physical block size in case of MTD partition).
IOW, you shouldn't bother whether you're using UBI on top of MTD or
directly using MTD partitions, both are immune to cross partition/volume
read/write disturbance.


> 
> > Officially I would say don't use SLC emulation, but technically I know what your doing.   The reason I say no is because we have very precise recipes designed to create very tight distibutions, and although the first pass distributions might look like an SLC, they are really designed with the expectation of the upper page being programmed.  Not a true SLC.
> > With MLC lithography of 25 nm and less  the difference between each level (L0, L1, L2 and L3) is just a few 10's of electrons.  The distribution have to be very tight, in order to meet retention requirements.
> This is quite interesting, however I'm afraid I have not fully 
> understood it.

Me neither :-/.

> Let me try to rephrase it. Please correct me if I'm wrong.
> 
> 1) Technically speaking, it is possible to use an MLC memory in SLC 
> mode, even if this is not recommended because MLC is not designed for 
> this usage.

That's what I understood, but I'm not sure to understand the
constraints brought by SLC mode (only programming one of the paired
pages).
Jeff, Are you trying to explain what's described here [1] in slide 8
(BTW I'm not sure to understand this diagram).
If that's the case, could you explain us, how the NAND chip knows which
threshold should be used (does it somehow store the information of
which page has already been programmed)

[1]http://www.bswd.com/FMS09/FMS09-T2A-Grunzke.pdf

-- 
Boris Brezillon, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

RE: RFC: detect and manage power cut on MLC NAND

[Jeff Lauruhn (jlauruhn)]

Boris,
You seem to have a pretty good understanding of NAND.  Now that I have a diagram, it will make this easier to explain.  Refer to page 8 of the presentation.  When we program and MLC device we program the Lower Page first.  In a perfect world all bits would set to discrete values like -.5v and 2.0v, called Vt erased and Vt programmed.  But, in reality Vts end up in a distribution, due to process, location of cell with a block, couplings, previous state of the cell, etc.  The diagram we are looking at is a representation of the distribution of all cells in the die, so if we programmed only the lower page and measured the potential on all cells, they would fall into 2 distributions around -.5v and 2.0v, I refer to them as L0 and L1.  L0 is defined as erased, or 1h and L1 is defined as programmed or 0h.  And yes there is a bit, to indicate if the middle or upper page is programmed.  Now, program the middle page (this is MLC 2bits/cell btw).  First read the lower page and get it's current value.  If the lower page of a cell is 1 and you want to store a 1 in the upper page of the cell, then tighten the distribution around -.5 and call this new L0.  If the lower page of the cell is 1 and you want to upper page to be 0, then move and tighten L0 to new L1.  If the lower page is 0 and you want the upper page to be a 0, then add charge to L1 and tighten, call this new L2.  Finally if you lower page is 0 and you want the upper page to be 1 add a little more charge and tighten to the new L3.  Now if you could measure the potential of every cell in the array they would fall into 4 tighter distributions, L0 to L3, representing all for conditions. 00, 01, 10, 11. I'm not going to go into the TLC, just follow the arrows.  

If you have a power loss during a program, only the page you're programming gets affected, not all distributions.  The programming process is designed to make it easier to recover the lower page if there's a power loss while programming the upper page.   

These guys have a pretty good explaination.  http://www.supertalent.com/datasheets/SLC_vs_MLC%20whitepaper.pdf 

Hope this helps.

Jeff Lauruhn
NAND Application Engineer


-----Original Message-----
From: Boris Brezillon [mailto:boris.brezillon@free-electrons.com] 
Sent: Thursday, March 19, 2015 2:13 AM
To: Andrea Marson
Cc: Jeff Lauruhn (jlauruhn); linux-mtd@lists.infradead.org; Andrea Scian; Richard Weinberger; dedekind1@gmail.com
Subject: Re: RFC: detect and manage power cut on MLC NAND

On Thu, 19 Mar 2015 09:47:21 +0100
Andrea Marson <andrea.marson@dave.eu> wrote:

> > Disturb is a block level affect, as long as partition A and B are in different blocks there will be no disturb between them.   Disturbs, does not damage cells; ERASE returns cells to undisturbed levels.
> I think there are two options here: MTD partitioning and UBI 
> partitioning. AFAIK one should prefer UBI partitioning to preserve 
> device-wide wear leveling. Boris, am I right?

Both of them act at block level, meaning that your the partition size must be a multiple of the block size (logical block size in case of UBI volume and physical block size in case of MTD partition).
IOW, you shouldn't bother whether you're using UBI on top of MTD or directly using MTD partitions, both are immune to cross partition/volume read/write disturbance.


> 
> > Officially I would say don't use SLC emulation, but technically I know what your doing.   The reason I say no is because we have very precise recipes designed to create very tight distibutions, and although the first pass distributions might look like an SLC, they are really designed with the expectation of the upper page being programmed.  Not a true SLC.
> > With MLC lithography of 25 nm and less  the difference between each level (L0, L1, L2 and L3) is just a few 10's of electrons.  The distribution have to be very tight, in order to meet retention requirements.
> This is quite interesting, however I'm afraid I have not fully 
> understood it.

Me neither :-/.

> Let me try to rephrase it. Please correct me if I'm wrong.
> 
> 1) Technically speaking, it is possible to use an MLC memory in SLC 
> mode, even if this is not recommended because MLC is not designed for 
> this usage.

That's what I understood, but I'm not sure to understand the constraints brought by SLC mode (only programming one of the paired pages).
Jeff, Are you trying to explain what's described here [1] in slide 8 (BTW I'm not sure to understand this diagram).
If that's the case, could you explain us, how the NAND chip knows which threshold should be used (does it somehow store the information of which page has already been programmed)

[1]http://www.bswd.com/FMS09/FMS09-T2A-Grunzke.pdf

--
Boris Brezillon, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

RE: RFC: detect and manage power cut on MLC NAND

[Jeff Lauruhn (jlauruhn)]

Jeff Lauruhn
NAND Application Engineer

-----Original Message-----
From: Andrea Marson [mailto:andrea.marson@dave.eu] 
Sent: Thursday, March 19, 2015 1:47 AM
To: Jeff Lauruhn (jlauruhn); Boris Brezillon
Cc: linux-mtd@lists.infradead.org; Andrea Scian; Richard Weinberger; dedekind1@gmail.com
Subject: Re: RFC: detect and manage power cut on MLC NAND

> Disturb is a block level affect, as long as partition A and B are in different blocks there will be no disturb between them.   Disturbs, does not damage cells; ERASE returns cells to undisturbed levels.
I think there are two options here: MTD partitioning and UBI partitioning. AFAIK one should prefer UBI partitioning to preserve device-wide wear leveling. Boris, am I right?

> Officially I would say don't use SLC emulation, but technically I know what your doing.   The reason I say no is because we have very precise recipes designed to create very tight distibutions, and although the first pass distributions might look like an SLC, they are really designed with the expectation of the upper page being programmed.  Not a true SLC.
> With MLC lithography of 25 nm and less  the difference between each level (L0, L1, L2 and L3) is just a few 10's of electrons.  The distribution have to be very tight, in order to meet retention requirements.
This is quite interesting, however I'm afraid I have not fully understood it. Let me try to rephrase it. Please correct me if I'm wrong.

1) Technically speaking, it is possible to use an MLC memory in SLC mode, even if this is not recommended because MLC is not designed for this usage.
	There are devices that support SLC Mode.  You can set a feature and treat blocks like SLC, this is the preferred method, because the P/E recipes used to set the values are SLC specific.  

	Also, there are people who program just to lower pages of MLC device which could be referred to SLC like.  This is not the preferred method because the P/E recipes would be set for MLC specifics assuming all pages would be used. 

2) As indicated by Boris, the easiest way to implement this thing is to avoid the use of paired pages, according to paired page table provided by datasheet.
	Boris has a good understanding of NAND.  

3) This technique does not transform an MLC NAND to an SLC magically. 
Thus data retention and lifetime are not increased. However all paired pages issues disappear. It is not clear if there are further drawbacks that reduce flash reliability if used this way.

	True.  This is SLC like, half the capacity of MLC, data retention and life time expectations could be in the range of SLC but it would be hard to know for sure. 

Thank you,
Andrea

RE: RFC: detect and manage power cut on MLC NAND

[Iwo Mergler]

Hi all,


I've been thinking a little about the paired pages problem.

The patented mitigation methods I've seen so far are

1)  SanDisk: Use a separate log mechanism to back up LSB page
while writing MSB page.

2) M-Systems: Within the block, interleave backup copies of LSB
pages when writing MSB pages.

3) Micron: Use as a half-capacity SLC device by writing LSB and MSB
pages such that the levels are reinforced. Jeff mentioned this one.


Here is another idea. I assume it's patented already, in which case
I'd like to hear about it. If not, consider this published as of now :-)


Get UBI to map the paired pages into a single write unit. Jumbo page
if you will.  So, instead of a block with 64 pages, we get 32 pages,
twice as large.

Thus, the paired pages will be written in quick succession. A power
cut during this is reduced to the unstable bits issue we already have
with SLC.

UBI could take the risk and split the first page pair between EC and
VID headers, replacing the EC info with average in case of unlikely
failure.

Would this work?


Best regards,

Iwo


________________________________________
From: linux-mtd [linux-mtd-bounces@lists.infradead.org] On Behalf Of Boris Brezillon [boris.brezillon@free-electrons.com]
Sent: Thursday, 19 March 2015 8:12 PM
To: Andrea Marson
Cc: Andrea Scian; Jeff Lauruhn (jlauruhn); linux-mtd@lists.infradead.org; dedekind1@gmail.com; Richard Weinberger
Subject: Re: RFC: detect and manage power cut on MLC NAND

On Thu, 19 Mar 2015 09:47:21 +0100
Andrea Marson <andrea.marson@dave.eu> wrote:

> > Disturb is a block level affect, as long as partition A and B are in different blocks there will be no disturb between them.   Disturbs, does not damage cells; ERASE returns cells to undisturbed levels.
> I think there are two options here: MTD partitioning and UBI
> partitioning. AFAIK one should prefer UBI partitioning to preserve
> device-wide wear leveling. Boris, am I right?

Both of them act at block level, meaning that your the partition size
must be a multiple of the block size (logical block size in case of UBI
volume and physical block size in case of MTD partition).
IOW, you shouldn't bother whether you're using UBI on top of MTD or
directly using MTD partitions, both are immune to cross partition/volume
read/write disturbance.


>
> > Officially I would say don't use SLC emulation, but technically I know what your doing.   The reason I say no is because we have very precise recipes designed to create very tight distibutions, and although the first pass distributions might look like an SLC, they are really designed with the expectation of the upper page being programmed.  Not a true SLC.
> > With MLC lithography of 25 nm and less  the difference between each level (L0, L1, L2 and L3) is just a few 10's of electrons.  The distribution have to be very tight, in order to meet retention requirements.
> This is quite interesting, however I'm afraid I have not fully
> understood it.

Me neither :-/.

> Let me try to rephrase it. Please correct me if I'm wrong.
>
> 1) Technically speaking, it is possible to use an MLC memory in SLC
> mode, even if this is not recommended because MLC is not designed for
> this usage.

That's what I understood, but I'm not sure to understand the
constraints brought by SLC mode (only programming one of the paired
pages).
Jeff, Are you trying to explain what's described here [1] in slide 8
(BTW I'm not sure to understand this diagram).
If that's the case, could you explain us, how the NAND chip knows which
threshold should be used (does it somehow store the information of
which page has already been programmed)

[1]http://www.bswd.com/FMS09/FMS09-T2A-Grunzke.pdf

--
Boris Brezillon, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

Re: RFC: detect and manage power cut on MLC NAND

[nick]

On 2015-03-19 08:25 PM, Iwo Mergler wrote:
> 
> Hi all,
> 
> 
> I've been thinking a little about the paired pages problem.
> 
> The patented mitigation methods I've seen so far are
> 
> 1)  SanDisk: Use a separate log mechanism to back up LSB page
> while writing MSB page.
> 
> 2) M-Systems: Within the block, interleave backup copies of LSB
> pages when writing MSB pages.
> 
> 3) Micron: Use as a half-capacity SLC device by writing LSB and MSB
> pages such that the levels are reinforced. Jeff mentioned this one.
> 
> 
> Here is another idea. I assume it's patented already, in which case
> I'd like to hear about it. If not, consider this published as of now :-)
> 
> 
> Get UBI to map the paired pages into a single write unit. Jumbo page
> if you will.  So, instead of a block with 64 pages, we get 32 pages,
> twice as large.
> 
> Thus, the paired pages will be written in quick succession. A power
> cut during this is reduced to the unstable bits issue we already have
> with SLC.
> 
> UBI could take the risk and split the first page pair between EC and
> VID headers, replacing the EC info with average in case of unlikely
> failure.
> 
> Would this work?
> 
> 
> Best regards,
> 
> Iwo
> 
> 
> 
Iwo,
Not a bad idea but what happens if the power gets cut after the first 2 pages then we lose
30 pages worth of data.  Unless the hardware is very fast and can move the pages over in
a millisecond or less then this can't really worth. However if we make these transactions
atomic in nature this may work better as CPU instruction respond is in the nanosecond range
even on low cost embedded boards using UBI for raw flash embedded on to themselves. 
Nick
________________________________________
> From: linux-mtd [linux-mtd-bounces@lists.infradead.org] On Behalf Of Boris Brezillon [boris.brezillon@free-electrons.com]
> Sent: Thursday, 19 March 2015 8:12 PM
> To: Andrea Marson
> Cc: Andrea Scian; Jeff Lauruhn (jlauruhn); linux-mtd@lists.infradead.org; dedekind1@gmail.com; Richard Weinberger
> Subject: Re: RFC: detect and manage power cut on MLC NAND
> 
> On Thu, 19 Mar 2015 09:47:21 +0100
> Andrea Marson <andrea.marson@dave.eu> wrote:
> 
>>> Disturb is a block level affect, as long as partition A and B are in different blocks there will be no disturb between them.   Disturbs, does not damage cells; ERASE returns cells to undisturbed levels.
>> I think there are two options here: MTD partitioning and UBI
>> partitioning. AFAIK one should prefer UBI partitioning to preserve
>> device-wide wear leveling. Boris, am I right?
> 
> Both of them act at block level, meaning that your the partition size
> must be a multiple of the block size (logical block size in case of UBI
> volume and physical block size in case of MTD partition).
> IOW, you shouldn't bother whether you're using UBI on top of MTD or
> directly using MTD partitions, both are immune to cross partition/volume
> read/write disturbance.
> 
> 
>>
>>> Officially I would say don't use SLC emulation, but technically I know what your doing.   The reason I say no is because we have very precise recipes designed to create very tight distibutions, and although the first pass distributions might look like an SLC, they are really designed with the expectation of the upper page being programmed.  Not a true SLC.
>>> With MLC lithography of 25 nm and less  the difference between each level (L0, L1, L2 and L3) is just a few 10's of electrons.  The distribution have to be very tight, in order to meet retention requirements.
>> This is quite interesting, however I'm afraid I have not fully
>> understood it.
> 
> Me neither :-/.
> 
>> Let me try to rephrase it. Please correct me if I'm wrong.
>>
>> 1) Technically speaking, it is possible to use an MLC memory in SLC
>> mode, even if this is not recommended because MLC is not designed for
>> this usage.
> 
> That's what I understood, but I'm not sure to understand the
> constraints brought by SLC mode (only programming one of the paired
> pages).
> Jeff, Are you trying to explain what's described here [1] in slide 8
> (BTW I'm not sure to understand this diagram).
> If that's the case, could you explain us, how the NAND chip knows which
> threshold should be used (does it somehow store the information of
> which page has already been programmed)
> 
> [1]http://www.bswd.com/FMS09/FMS09-T2A-Grunzke.pdf
> 
> --
> Boris Brezillon, Free Electrons
> Embedded Linux and Kernel engineering
> http://free-electrons.com
> 
> ______________________________________________________
> Linux MTD discussion mailing list
> http://lists.infradead.org/mailman/listinfo/linux-mtd/
> 
> ______________________________________________________
> Linux MTD discussion mailing list
> http://lists.infradead.org/mailman/listinfo/linux-mtd/
> 

RE: RFC: detect and manage power cut on MLC NAND

[Iwo Mergler]

On Fri, 20 Mar 2015 14:38:03 +1100
nick <xerofoify@gmail.com> wrote:
> 
> 
> On 2015-03-19 08:25 PM, Iwo Mergler wrote:
> > 
> > Get UBI to map the paired pages into a single write unit. Jumbo page
> > if you will.  So, instead of a block with 64 pages, we get 32 pages,
> > twice as large.
> > 
> > Thus, the paired pages will be written in quick succession. A power
> > cut during this is reduced to the unstable bits issue we already
> > have with SLC.
> > 
> > UBI could take the risk and split the first page pair between EC and
> > VID headers, replacing the EC info with average in case of unlikely
> > failure.
> > 
> > Would this work?
> > 
> > 
> > Best regards,
> > 
> > Iwo
> > 
> > 
> > 
> Iwo,
> Not a bad idea but what happens if the power gets cut after the first
> 2 pages then we lose 30 pages worth of data.  Unless the hardware is
> very fast and can move the pages over in a millisecond or less then
> this can't really worth. However if we make these transactions atomic
> in nature this may work better as CPU instruction respond is in the
> nanosecond range even on low cost embedded boards using UBI for raw
> flash embedded on to themselves. Nick

I'm not sure I understand - why would you lose any data outside the
affected paired pages?

Also, write order within the block will still be respected. That is,
when writing the first page pair, the other pages will be erased.


Best regards,

Iwo

Re: RFC: detect and manage power cut on MLC NAND

[Andrea Marson]

> 1) Technically speaking, it is possible to use an MLC memory in SLC mode, even if this is not recommended because MLC is not designed for this usage.
> 	There are devices that support SLC Mode.
Can you please indicate Micron part numbers supporting SLC mode? It 
would be really interesting to test these devices on our platforms.

Thank you,
Andrea

Re: RFC: detect and manage power cut on MLC NAND

[Boris Brezillon]

Hi Iwo,

On Fri, 20 Mar 2015 11:25:18 +1100
Iwo Mergler <Iwo.Mergler@netcommwireless.com> wrote:

> 
> Hi all,
> 
> 
> I've been thinking a little about the paired pages problem.
> 
> The patented mitigation methods I've seen so far are
> 
> 1)  SanDisk: Use a separate log mechanism to back up LSB page
> while writing MSB page.
> 
> 2) M-Systems: Within the block, interleave backup copies of LSB
> pages when writing MSB pages.
> 
> 3) Micron: Use as a half-capacity SLC device by writing LSB and MSB
> pages such that the levels are reinforced. Jeff mentioned this one.

All these solutions involves storing a backup of the first paired page,
before writing the 2nd one (I guess that's even worse with TLC NANDs:
you'll have to save the 2nd page when writing the 3rd one).
If that's possible, I think we should find a way to avoid this
duplication.

> 
> 
> Here is another idea. I assume it's patented already, in which case
> I'd like to hear about it. If not, consider this published as of now :-)
> 
> 
> Get UBI to map the paired pages into a single write unit. Jumbo page
> if you will.  So, instead of a block with 64 pages, we get 32 pages,
> twice as large.
> 
> Thus, the paired pages will be written in quick succession. A power
> cut during this is reduced to the unstable bits issue we already have
> with SLC.

That's actually a good idea, but I'd like to have feedback from Jeff
and/or Qi regarding the feasibility of this proposal.
I've read many times that MLC pages should be programmed in ascending
order (0, 1, 2, 3, ..., N), and if you take a look at MLC datasheet
you'll see that paired pages are not contiguous (here is an example
[1], page 55-56 describe how pages are paired together).

My question is: is there a reason for interleaving paired pages with
other pages (write disturbance mitigation ?) ?

If there is no specific reason but to annoy software developers :-),
then I think we should consider this Jumbo page approach, because it
would contain the paired pages handling in the NAND layer (instead of
exposing complexity to the UBI/UBIFS layers) while allowing us to use
almost all the NAND chip capacity.

> 
> UBI could take the risk and split the first page pair between EC and
> VID headers, replacing the EC info with average in case of unlikely
> failure.

You want to do that in order to avoid using 2 Jumbo pages to only store
a few bytes ?
Artem suggested another solution to deal with that: duplicate the EC
header in the VID header so that EC information can be recovered when
writing to the page paired with the EC header.
You'd still have to prevent any write on the page paired with the EC
header, but you're still saving one page with this approach and avoid
any corruption of UBI metadata caused by paired pages.

Maybe there are better solution, and I've been thinking about a
different approach to avoid writing the EC and VID headers in 2
different pages: store EC header information in a specific UBI volume
(containing one or two LEBs).
This volume would store a log of EC information update, so that when
you erase a page you don't have to write the EC header right away, but
should just add an entry in the UBI EC log.
This way you write EC and VID information in one go when someone maps a
LEB (attach it to a specific volume).

These are just thoughts ;-).

[1]http://www.szyuda88.com/uploadfile/cfile/2013419135343223.pdf
-- 
Boris Brezillon, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

Re: RFC: detect and manage power cut on MLC NAND

[Nick Krause]

On Fri, Mar 20, 2015 at 4:26 AM, Boris Brezillon
<boris.brezillon@free-electrons.com> wrote:
> Hi Iwo,
>
> On Fri, 20 Mar 2015 11:25:18 +1100
> Iwo Mergler <Iwo.Mergler@netcommwireless.com> wrote:
>
>>
>> Hi all,
>>
>>
>> I've been thinking a little about the paired pages problem.
>>
>> The patented mitigation methods I've seen so far are
>>
>> 1)  SanDisk: Use a separate log mechanism to back up LSB page
>> while writing MSB page.
>>
>> 2) M-Systems: Within the block, interleave backup copies of LSB
>> pages when writing MSB pages.
>>
>> 3) Micron: Use as a half-capacity SLC device by writing LSB and MSB
>> pages such that the levels are reinforced. Jeff mentioned this one.
>
> All these solutions involves storing a backup of the first paired page,
> before writing the 2nd one (I guess that's even worse with TLC NANDs:
> you'll have to save the 2nd page when writing the 3rd one).
> If that's possible, I think we should find a way to avoid this
> duplication.
>
>>
>>
>> Here is another idea. I assume it's patented already, in which case
>> I'd like to hear about it. If not, consider this published as of now :-)
>>
>>
>> Get UBI to map the paired pages into a single write unit. Jumbo page
>> if you will.  So, instead of a block with 64 pages, we get 32 pages,
>> twice as large.
>>
>> Thus, the paired pages will be written in quick succession. A power
>> cut during this is reduced to the unstable bits issue we already have
>> with SLC.
>
> That's actually a good idea, but I'd like to have feedback from Jeff
> and/or Qi regarding the feasibility of this proposal.
> I've read many times that MLC pages should be programmed in ascending
> order (0, 1, 2, 3, ..., N), and if you take a look at MLC datasheet
> you'll see that paired pages are not contiguous (here is an example
> [1], page 55-56 describe how pages are paired together).
>
> My question is: is there a reason for interleaving paired pages with
> other pages (write disturbance mitigation ?) ?
>
> If there is no specific reason but to annoy software developers :-),
> then I think we should consider this Jumbo page approach, because it
> would contain the paired pages handling in the NAND layer (instead of
> exposing complexity to the UBI/UBIFS layers) while allowing us to use
> almost all the NAND chip capacity.
>
>>
>> UBI could take the risk and split the first page pair between EC and
>> VID headers, replacing the EC info with average in case of unlikely
>> failure.
>
> You want to do that in order to avoid using 2 Jumbo pages to only store
> a few bytes ?
> Artem suggested another solution to deal with that: duplicate the EC
> header in the VID header so that EC information can be recovered when
> writing to the page paired with the EC header.
> You'd still have to prevent any write on the page paired with the EC
> header, but you're still saving one page with this approach and avoid
> any corruption of UBI metadata caused by paired pages.
>
> Maybe there are better solution, and I've been thinking about a
> different approach to avoid writing the EC and VID headers in 2
> different pages: store EC header information in a specific UBI volume
> (containing one or two LEBs).
> This volume would store a log of EC information update, so that when
> you erase a page you don't have to write the EC header right away, but
> should just add an entry in the UBI EC log.
> This way you write EC and VID information in one go when someone maps a
> LEB (attach it to a specific volume).
>
> These are just thoughts ;-).
>
> [1]http://www.szyuda88.com/uploadfile/cfile/2013419135343223.pdf
> --
> Boris Brezillon, Free Electrons
> Embedded Linux and Kernel engineering
> http://free-electrons.com
>
> ______________________________________________________
> Linux MTD discussion mailing list
> http://lists.infradead.org/mailman/listinfo/linux-mtd/
Iwo,
I was assuming the pages were still in ram and volatile for my example.
Nick

RE: RFC: detect and manage power cut on MLC NAND

[Iwo Mergler]

On Fri, 20 Mar 2015 19:26:11 +1100
Boris Brezillon <boris.brezillon@free-electrons.com> wrote:

> I've read many times that MLC pages should be programmed in ascending
> order (0, 1, 2, 3, ..., N), and if you take a look at MLC datasheet
> you'll see that paired pages are not contiguous (here is an example
> [1], page 55-56 describe how pages are paired together).
> 
> My question is: is there a reason for interleaving paired pages with
> other pages (write disturbance mitigation ?) ?
> 
> If there is no specific reason but to annoy software developers :-),


I don't know for sure, but it seems that groups of pages are sharing
some resources. For instance, I've seen a few bit error patterns which
looked like this (256MB SLC, pages in one block):

-------------------------------X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X

The pages marked with 'X' have a single bit error at the same bit
offset in every page.

That is, even on a SLC device, there is some sort of page interleaving
pattern at work. In this case, maybe they share a broken sense amplifier
or similar.

I currently think that the rule about the ascending order is a
generalisation for unknown NAND architecture. In my example, it
may well be slightly better to write (0,2,4,6,8,10,12,14,1,3,5,
7,9,11,13,15,16,18,20,22,24,26,28,30,17,19,21,23,25,27,29,31).

Unless someone tells me otherwise, I guess that the paired pages
probably should be written in sequence anyway (0,4,1,5,etc).
Some of the backup methods appear suggestive of this.


Best regards,

Iwo

RE: RFC: detect and manage power cut on MLC NAND

[Iwo Mergler]

On Fri, 20 Mar 2015 19:26:11 +1100
Boris Brezillon <boris.brezillon@free-electrons.com> wrote:

> You want to do that in order to avoid using 2 Jumbo pages to only
> store a few bytes ?

Yes. I'm already a little unhappy about using two whole pages
for UBI, and don't want to be guilty of doubling that.

> Artem suggested another solution to deal with that: duplicate the EC
> header in the VID header so that EC information can be recovered when
> writing to the page paired with the EC header.

Sounds good to me. It won't avoid the loss of the VID header,
but it will safeguard the EC. It's free, too.

> You'd still have to prevent any write on the page paired with the EC
> header, but you're still saving one page with this approach and avoid
> any corruption of UBI metadata caused by paired pages.

Ah, you probably meant VID header here, right?

> Maybe there are better solution, and I've been thinking about a
> different approach to avoid writing the EC and VID headers in 2
> different pages: store EC header information in a specific UBI volume
> (containing one or two LEBs).
> This volume would store a log of EC information update, so that when
> you erase a page you don't have to write the EC header right away, but
> should just add an entry in the UBI EC log.
> This way you write EC and VID information in one go when someone maps
> a LEB (attach it to a specific volume).

I like that. Maybe UBI could try grouping erases to reduce the
stress on the EC log - write the (future) ECs for a number of
PEBs to the log, then delete the PEBS at leisure, then mark
them as done in the log.

In fact, you could even go and pad the log pages with future
ECs for active PEBs, to be used in case they need erasing later.
Just never erase a PEB that doesn't have an EC log entry.

Would it be possible to push that further, and defer the writing
of the VID header until the first payload write is pending? Up to
that point, it shouldn't matter where the LEB is mapped.

I'm aware that UBIFS would have to cooperate here (first page is
64 bytes smaller), but VID and integrated EC could share the
page 0 with the first payload nodes of the LEB.


Best regards,

Iwo

RE: RFC: detect and manage power cut on MLC NAND

[Iwo Mergler]

Hi all,


I probably don't know enough about the silicon implementation
of MLC paired pages, but my feeling is that there should be a
way to recover one of the pages if the paired write fails, at least
in some cases.

Something along the lines of using both bits to map to a single
good one.

2 bit MLC stores 4 levels - 1.0, 0.7, 0.3, 0.0. Obviously, the actual
voltage levels will be somewhat different, so take this as
electrons on the floating gate: 1.0=minimum, 0.0 maximum.

I imagine that there are two ways to achieve that - small step
for low page and large step for high page, or the other way 'round.

Assuming the first, the low page write would subtract 0.3 from
the erased (1.0) cell if the bit is 0. That leaves the cell at
either ~1.0 (1) or 0.7 (0).

Lvl    LH
===========
1.0 => 1u
0.7 => 0u

Then, the high page write would subtract either nothing (1) or
0.7 (0):

Lvl    LH
===========
1.0 => 11
0.7 => 01
0.3 => 10
0.0 => 00

So the MLC decoder logic gets 3 priority encoded bits from the
sense amplifiers: 111, 011, 001, 000. The decoder turns this
into 11, 01, 10, 00.

The process of writing a 0 to the high page, transitions low page
0-bits through 1 and back to 0, as the level moves down.

Low page 1 bits transition from 1 through 0 and back to 1.

So a half-completed high page 0-write can flip a low page bit
both ways.

We can detect an incorrect 0-1 transition in the low page,
because it's marked by a 0 bit in the high page.

We can't detect an incorrect 1-0 transition in the low page.

So assuming a failed high page write, this is what we get:

LH

11 = nothing happens, reads back as 11
     Correct level for both.

01 = Level stays at 0.7, reads back as 01,
     Correct level for low page.

10 = Level between 1.0 and 0.3, reads back as 11, 01 or 10.
     01 is wrong for low page, but can't be distinguished from 10.

00 = Level between 0.7 and 0.0, reads back as 01, 10, or 00.
     10 is wrong for low page, but can be distinguished from 01.

So, there are two bit combinations (50%) that have an
undetectable failure, and this failure will happen about half
the time, for a total of 25% unfixable failure rate.

Not acceptable in the general case, but might be good enough
for things like UBI EC & VID headers, if we ensure that the
high page contains 1s at the offsets at which the low page
stores the header.


Now, on the other hand, if the low page write uses the larger
step, there shouldn't be any paired page problem at all, since
the high page write wouldn't cross the low page thresholds
on the way:

Lvl    LH
===========
1.0 => 1u
0.3 => 0u

Lvl    LH
===========
1.0 => 11
0.7 => 10
0.3 => 01
0.0 => 00


Which makes me think I'm misunderstanding something. If not,
why isn't his scheme used in the first place?

What would happen if we reverse the paired page writing order?


Jeff, Qi, is the mechanism I described here anywhere near reality?


Best regards,

Iwo

Re: RFC: detect and manage power cut on MLC NAND

[Artem Bityutskiy]

On Mon, 2015-03-23 at 13:18 +1100, Iwo Mergler wrote:
> On Fri, 20 Mar 2015 19:26:11 +1100
> Boris Brezillon <boris.brezillon@free-electrons.com> wrote:
> 
> > You want to do that in order to avoid using 2 Jumbo pages to only
> > store a few bytes ?
> 
> Yes. I'm already a little unhappy about using two whole pages
> for UBI, and don't want to be guilty of doubling that.
> 
> > Artem suggested another solution to deal with that: duplicate the EC
> > header in the VID header so that EC information can be recovered when
> > writing to the page paired with the EC header.
> 
> Sounds good to me. It won't avoid the loss of the VID header,
> but it will safeguard the EC. It's free, too.
> 
> > You'd still have to prevent any write on the page paired with the EC
> > header, but you're still saving one page with this approach and avoid
> > any corruption of UBI metadata caused by paired pages.
> 
> Ah, you probably meant VID header here, right?

If we sacrifice another page, we could store a copy of the VID header
there, in case the first copy gets corrupted. IOW, we'd have:

	ECH, VIDH#1, VIDH#2, User Data.

Re: RFC: detect and manage power cut on MLC NAND

[Boris Brezillon]

Hi Artem,

On Mon, 23 Mar 2015 09:06:31 +0200
Artem Bityutskiy <dedekind1@gmail.com> wrote:

> > 
> > > You'd still have to prevent any write on the page paired with the EC
> > > header, but you're still saving one page with this approach and avoid
> > > any corruption of UBI metadata caused by paired pages.
> > 
> > Ah, you probably meant VID header here, right?
> 
> If we sacrifice another page, we could store a copy of the VID header
> there, in case the first copy gets corrupted. IOW, we'd have:
> 
> 	ECH, VIDH#1, VIDH#2, User Data.

What do you mean by 'sacrificing another page', are you talking about
the one paired with the VID header page ?
If that's the case, then we cannot write anything on it (even a VID
header backup), because if the PROGRAM operation is interrupted both
pages (VIDH#1 and #2) could be corrupted.

Best Regards,

Boris


-- 
Boris Brezillon, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

RE: RFC: detect and manage power cut on MLC NAND

[Jeff Lauruhn (jlauruhn)]

This is a very simplified description, but actually it's more like this: 

First pass, program the lower page.  If you the lower page is 1, do nothing.  If the lower page is 0 subtract 0.7v to 0.3.  Lower page is SLC like, two distributions spread apart by 0.7V.

Lvl    LH
===========
1.0 => 1u
0.3 => 0u

Now, program the upper page.  First, read lower page, if lower page is 1 and upper page is 1, do nothing (11).  If lower page is 1 and upper page is 0, then subtract -0.3 and call that 01.  Next if lower page is 0 and upper page is 1 do nothing and if lower page is 0 and upper page is 0 subtract 0.3v and call it 00.  Notice that state of lower page is on right of 11, 01,10, 00.  

Lvl    LH
===========
1.0 => 11
0.7 => 01
0.3 => 10
0.0 => 00

Now what happens if there's a power loss during the programming of the upper page?  The upper page data will most likely be lost, and the lower page may be changed, but there's a good chance of recovery, because it will be in the range of SLC.  It is highly recommended to read and refresh data after a power loss.
 

Jeff Lauruhn
NAND Application Engineer
Embedded Business Unit

-----Original Message-----
From: Iwo Mergler [mailto:Iwo.Mergler@netcommwireless.com] 
Sent: Sunday, March 22, 2015 9:09 PM
To: Richard Weinberger; dedekind1@gmail.com
Cc: Andrea Scian; mtd_mailinglist; Jeff Lauruhn (jlauruhn); Qi Wang 王起 (qiwang)
Subject: RE: RFC: detect and manage power cut on MLC NAND


Hi all,


I probably don't know enough about the silicon implementation of MLC paired pages, but my feeling is that there should be a way to recover one of the pages if the paired write fails, at least in some cases.

Something along the lines of using both bits to map to a single good one.

2 bit MLC stores 4 levels - 1.0, 0.7, 0.3, 0.0. Obviously, the actual voltage levels will be somewhat different, so take this as electrons on the floating gate: 1.0=minimum, 0.0 maximum.

I imagine that there are two ways to achieve that - small step for low page and large step for high page, or the other way 'round.

Assuming the first, the low page write would subtract 0.3 from the erased (1.0) cell if the bit is 0. That leaves the cell at either ~1.0 (1) or 0.7 (0).

Lvl    LH
===========
1.0 => 1u
0.7 => 0u

Then, the high page write would subtract either nothing (1) or
0.7 (0):

Lvl    LH
===========
1.0 => 11
0.7 => 01
0.3 => 10
0.0 => 00

So the MLC decoder logic gets 3 priority encoded bits from the sense amplifiers: 111, 011, 001, 000. The decoder turns this into 11, 01, 10, 00.

The process of writing a 0 to the high page, transitions low page 0-bits through 1 and back to 0, as the level moves down.

Low page 1 bits transition from 1 through 0 and back to 1.

So a half-completed high page 0-write can flip a low page bit both ways.

We can detect an incorrect 0-1 transition in the low page, because it's marked by a 0 bit in the high page.

We can't detect an incorrect 1-0 transition in the low page.

So assuming a failed high page write, this is what we get:

LH

11 = nothing happens, reads back as 11
     Correct level for both.

01 = Level stays at 0.7, reads back as 01,
     Correct level for low page.

10 = Level between 1.0 and 0.3, reads back as 11, 01 or 10.
     01 is wrong for low page, but can't be distinguished from 10.

00 = Level between 0.7 and 0.0, reads back as 01, 10, or 00.
     10 is wrong for low page, but can be distinguished from 01.

So, there are two bit combinations (50%) that have an undetectable failure, and this failure will happen about half the time, for a total of 25% unfixable failure rate.

Not acceptable in the general case, but might be good enough for things like UBI EC & VID headers, if we ensure that the high page contains 1s at the offsets at which the low page stores the header.


Now, on the other hand, if the low page write uses the larger step, there shouldn't be any paired page problem at all, since the high page write wouldn't cross the low page thresholds on the way:

Lvl    LH
===========
1.0 => 1u
0.3 => 0u

Lvl    LH
===========
1.0 => 11
0.7 => 10
0.3 => 01
0.0 => 00


Which makes me think I'm misunderstanding something. If not, why isn't his scheme used in the first place?

What would happen if we reverse the paired page writing order?
Not recommended, we want pages programmed in sequence to mitigate disturbs and obtain the highest reliability.


Jeff, Qi, is the mechanism I described here anywhere near reality?
It's a simplified view, but fairly accurate.  


Best regards,

Iwo

RE: RFC: detect and manage power cut on MLC NAND

[Iwo Mergler]

Hi Jeff,


thanks for the info. That sounds like very good news
to me - it appears that the paired page problem isn't
as bad as I thought.

> Lvl    LH
> ===========
> 1.0 => 11
> 0.7 => 01
> 0.3 => 10
> 0.0 => 00
         ||
Is this  HL?

This seems to match my second scenario (large step
for low page) - we do not lose the low page when
the high page write fails. Here is what I understood:

The low page distribution centres around 1.0 and 0.3,
with a spread of 0.7. Reading the low page as MLC at
this point will classify everything above 0.5 as 1
and below as 0.

So the distance of a '1' from the threshold is 0.5,
and the distance of a '0' is 0.2. It's asymmetric,
with SLC distance on one side and MLC on the other.

The high page write (only when writing a '0'), moves
the charge towards 0.0, by 0.3. So if the first page
contains '1', the charge is lowered to 0.7, if the
first page contains '0' the charge goes down to 0.0.

At no time is the charge crossing the 0.5 threshold.
An aborted high page write will have reduced the '1'
distance by 0.0-0.3 or increased the '0' distance
by 0.0-0.3.

Why are we worried about the low page at all? The way
I understand the gate charge changes, the low page
situation can not be, at any point, worse than the
normal programmed MLC state.


Losing the high page isn't conceptually any worse
than aborted writes on SLC. This has been called
the "unstable bits problem".

As far as I understand, UBI/UBIFS currently don't
mitigate for aborted write. The worry here is that an
aborted write (SLC or MLC) gets you a page where the
0s are weak - they may pass ECC on read now but fail
next month. Or if the abort was early, weaken some
1s in an erased-looking page.

Aborted erases are handled by UBI via the EC header
write immediately after erase. An erased block without
an EC header is erased again at boot time, to avoid
partially erased blocks (weak 1s).

Aborted writes are harder. UBI could deal with its
own headers via sequence number and always rewrite
the last block when it has UBI headers but no payload.
Seems a bit wasteful, though.

UBIFS also has an idea which few blocks may have
received a power cut during the last write - I think
around 5 candidates or so. Rewriting them all every
mount time seems even more wasteful.

Then of course, the rewrite can also be aborted, so
UBI's last atomic LEB change probably also needs
redoing at attach time.


Jeff, what's the situation with aborted writes?
Is this problem real? I understand that there are
charge pumps on board the NAND chip - do they
store sufficient energy to complete a page write?

What about the typical system, which has, say, a
3.3V supply rail and a system reset firing when
that falls below 3.0V or so. The processor core
and NAND operate at 1.8V, so there are a few ms
between the last possible NAND command and the
breakdown of the NAND supply.

Would that be safe from aborted writes?


Best regards,

Iwo


________________________________________
From: Jeff Lauruhn (jlauruhn) [jlauruhn@micron.com]
Sent: Tuesday, 24 March 2015 8:15 AM
To: Iwo Mergler; Richard Weinberger; dedekind1@gmail.com
Cc: Andrea Scian; mtd_mailinglist; Qi Wang 王起 (qiwang)
Subject: RE: RFC: detect and manage power cut on MLC NAND

This is a very simplified description, but actually it's more like this:

First pass, program the lower page.  If you the lower page is 1, do nothing.  If the lower page is 0 subtract 0.7v to 0.3.  Lower page is SLC like, two distributions spread apart by 0.7V.

Lvl    LH
===========
1.0 => 1u
0.3 => 0u

Now, program the upper page.  First, read lower page, if lower page is 1 and upper page is 1, do nothing (11).  If lower page is 1 and upper page is 0, then subtract -0.3 and call that 01.  Next if lower page is 0 and upper page is 1 do nothing and if lower page is 0 and upper page is 0 subtract 0.3v and call it 00.  Notice that state of lower page is on right of 11, 01,10, 00.

Lvl    LH
===========
1.0 => 11
0.7 => 01
0.3 => 10
0.0 => 00

Now what happens if there's a power loss during the programming of the upper page?  The upper page data will most likely be lost, and the lower page may be changed, but there's a good chance of recovery, because it will be in the range of SLC.  It is highly recommended to read and refresh data after a power loss.


Jeff Lauruhn
NAND Application Engineer
Embedded Business Unit

-----Original Message-----
From: Iwo Mergler [mailto:Iwo.Mergler@netcommwireless.com]
Sent: Sunday, March 22, 2015 9:09 PM
To: Richard Weinberger; dedekind1@gmail.com
Cc: Andrea Scian; mtd_mailinglist; Jeff Lauruhn (jlauruhn); Qi Wang 王起 (qiwang)
Subject: RE: RFC: detect and manage power cut on MLC NAND


Hi all,


I probably don't know enough about the silicon implementation of MLC paired pages, but my feeling is that there should be a way to recover one of the pages if the paired write fails, at least in some cases.

Something along the lines of using both bits to map to a single good one.

2 bit MLC stores 4 levels - 1.0, 0.7, 0.3, 0.0. Obviously, the actual voltage levels will be somewhat different, so take this as electrons on the floating gate: 1.0=minimum, 0.0 maximum.

I imagine that there are two ways to achieve that - small step for low page and large step for high page, or the other way 'round.

Assuming the first, the low page write would subtract 0.3 from the erased (1.0) cell if the bit is 0. That leaves the cell at either ~1.0 (1) or 0.7 (0).

Lvl    LH
===========
1.0 => 1u
0.7 => 0u

Then, the high page write would subtract either nothing (1) or
0.7 (0):

Lvl    LH
===========
1.0 => 11
0.7 => 01
0.3 => 10
0.0 => 00

So the MLC decoder logic gets 3 priority encoded bits from the sense amplifiers: 111, 011, 001, 000. The decoder turns this into 11, 01, 10, 00.

The process of writing a 0 to the high page, transitions low page 0-bits through 1 and back to 0, as the level moves down.

Low page 1 bits transition from 1 through 0 and back to 1.

So a half-completed high page 0-write can flip a low page bit both ways.

We can detect an incorrect 0-1 transition in the low page, because it's marked by a 0 bit in the high page.

We can't detect an incorrect 1-0 transition in the low page.

So assuming a failed high page write, this is what we get:

LH

11 = nothing happens, reads back as 11
     Correct level for both.

01 = Level stays at 0.7, reads back as 01,
     Correct level for low page.

10 = Level between 1.0 and 0.3, reads back as 11, 01 or 10.
     01 is wrong for low page, but can't be distinguished from 10.

00 = Level between 0.7 and 0.0, reads back as 01, 10, or 00.
     10 is wrong for low page, but can be distinguished from 01.

So, there are two bit combinations (50%) that have an undetectable failure, and this failure will happen about half the time, for a total of 25% unfixable failure rate.

Not acceptable in the general case, but might be good enough for things like UBI EC & VID headers, if we ensure that the high page contains 1s at the offsets at which the low page stores the header.


Now, on the other hand, if the low page write uses the larger step, there shouldn't be any paired page problem at all, since the high page write wouldn't cross the low page thresholds on the way:

Lvl    LH
===========
1.0 => 1u
0.3 => 0u

Lvl    LH
===========
1.0 => 11
0.7 => 10
0.3 => 01
0.0 => 00


Which makes me think I'm misunderstanding something. If not, why isn't his scheme used in the first place?

What would happen if we reverse the paired page writing order?
Not recommended, we want pages programmed in sequence to mitigate disturbs and obtain the highest reliability.


Jeff, Qi, is the mechanism I described here anywhere near reality?
It's a simplified view, but fairly accurate.


Best regards,

Iwo

Re: RFC: detect and manage power cut on MLC NAND

[Artem Bityutskiy]

On Mon, 2015-03-23 at 20:05 +0100, Boris Brezillon wrote:
> > If we sacrifice another page, we could store a copy of the VID header
> > there, in case the first copy gets corrupted. IOW, we'd have:
> > 
> > 	ECH, VIDH#1, VIDH#2, User Data.
> 
> What do you mean by 'sacrificing another page', are you talking about
> the one paired with the VID header page ?
> If that's the case, then we cannot write anything on it (even a VID
> header backup), because if the PROGRAM operation is interrupted both
> pages (VIDH#1 and #2) could be corrupted.

Just this:

Page 0: ECH
Page 1: VIDH#1 (+all the ECH data)
Page 2: VIDH#2 (same as VIDH#1)

VIDH#1 and #2 are not in paired pages, so I expect one of them to always
be good.

If pages 0,1;2,3;4,5;etc are paired, we may lose ECH sometimes, should
not be catastrophic. I wonder though, if this pairing scheme is
realistic?

In case of a larger pairing step we seem to be always OK and will always
have a valid ECH and VIDH.

Do I miss something?

RE: RFC: detect and manage power cut on MLC NAND

[Jeff Lauruhn (jlauruhn)]

>From a simplified point of view you're right.  In reality the program/erase recipes are actually quite advanced in order to get very tight distributions on a full page.  The lower/upper page sequence is designed to provide the most reliable results and optimally we would like the lower and upper page programmed 100% of the time.  There's been a lot of work done over the years to improve power loss and it's much better than in the past, but it's still something to be avoided on NAND.  It's always best to check the integrity of the page after a power loss event. 

I want to make sure I understand your meaning of "aborted write".  If by aborted write you are referring to stopping the command, not a power loss event, then yes the internal circuitry will complete the write.  

"What about the typical system, which has, say, a 3.3V supply rail and a system reset firing when that falls below 3.0V or so. The processor core and NAND operate at 1.8V, so there are a few ms between the last possible NAND command and the breakdown of the NAND supply."

I have to be careful here because it's very dependent on the design and I really need to know the specifics to make a definitive statement, but a few ms should be enough time to protect the NAND.  WP# is your friend here.


Jeff Lauruhn
NAND Application Engineer
Embedded Business Unit
Micron Technology, Inc


-----Original Message-----
From: Iwo Mergler [mailto:Iwo.Mergler@netcommwireless.com] 
Sent: Monday, March 23, 2015 6:18 PM
To: Jeff Lauruhn (jlauruhn); Richard Weinberger; dedekind1@gmail.com
Cc: Andrea Scian; mtd_mailinglist; Qi Wang 王起 (qiwang)
Subject: RE: RFC: detect and manage power cut on MLC NAND

Hi Jeff,


thanks for the info. That sounds like very good news to me - it appears that the paired page problem isn't as bad as I thought.

> Lvl    LH
> ===========
> 1.0 => 11
> 0.7 => 01
> 0.3 => 10
> 0.0 => 00
         ||
Is this  HL?

This seems to match my second scenario (large step for low page) - we do not lose the low page when the high page write fails. Here is what I understood:

The low page distribution centres around 1.0 and 0.3, with a spread of 0.7. Reading the low page as MLC at this point will classify everything above 0.5 as 1 and below as 0.

So the distance of a '1' from the threshold is 0.5, and the distance of a '0' is 0.2. It's asymmetric, with SLC distance on one side and MLC on the other.

The high page write (only when writing a '0'), moves the charge towards 0.0, by 0.3. So if the first page contains '1', the charge is lowered to 0.7, if the first page contains '0' the charge goes down to 0.0.

At no time is the charge crossing the 0.5 threshold.
An aborted high page write will have reduced the '1'
distance by 0.0-0.3 or increased the '0' distance by 0.0-0.3.

Why are we worried about the low page at all? The way I understand the gate charge changes, the low page situation can not be, at any point, worse than the normal programmed MLC state.


Losing the high page isn't conceptually any worse than aborted writes on SLC. This has been called the "unstable bits problem".

As far as I understand, UBI/UBIFS currently don't mitigate for aborted write. The worry here is that an aborted write (SLC or MLC) gets you a page where the 0s are weak - they may pass ECC on read now but fail next month. Or if the abort was early, weaken some 1s in an erased-looking page.

Aborted erases are handled by UBI via the EC header write immediately after erase. An erased block without an EC header is erased again at boot time, to avoid partially erased blocks (weak 1s).

Aborted writes are harder. UBI could deal with its own headers via sequence number and always rewrite the last block when it has UBI headers but no payload.
Seems a bit wasteful, though.

UBIFS also has an idea which few blocks may have received a power cut during the last write - I think around 5 candidates or so. Rewriting them all every mount time seems even more wasteful.

Then of course, the rewrite can also be aborted, so UBI's last atomic LEB change probably also needs redoing at attach time.


Jeff, what's the situation with aborted writes?
Is this problem real? I understand that there are charge pumps on board the NAND chip - do they store sufficient energy to complete a page write?

What about the typical system, which has, say, a 3.3V supply rail and a system reset firing when that falls below 3.0V or so. The processor core and NAND operate at 1.8V, so there are a few ms between the last possible NAND command and the breakdown of the NAND supply.

Would that be safe from aborted writes?


Best regards,

Iwo


________________________________________
From: Jeff Lauruhn (jlauruhn) [jlauruhn@micron.com]
Sent: Tuesday, 24 March 2015 8:15 AM
To: Iwo Mergler; Richard Weinberger; dedekind1@gmail.com
Cc: Andrea Scian; mtd_mailinglist; Qi Wang 王起 (qiwang)
Subject: RE: RFC: detect and manage power cut on MLC NAND

This is a very simplified description, but actually it's more like this:

First pass, program the lower page.  If you the lower page is 1, do nothing.  If the lower page is 0 subtract 0.7v to 0.3.  Lower page is SLC like, two distributions spread apart by 0.7V.

Lvl    LH
===========
1.0 => 1u
0.3 => 0u

Now, program the upper page.  First, read lower page, if lower page is 1 and upper page is 1, do nothing (11).  If lower page is 1 and upper page is 0, then subtract -0.3 and call that 01.  Next if lower page is 0 and upper page is 1 do nothing and if lower page is 0 and upper page is 0 subtract 0.3v and call it 00.  Notice that state of lower page is on right of 11, 01,10, 00.

Lvl    LH
===========
1.0 => 11
0.7 => 01
0.3 => 10
0.0 => 00

Now what happens if there's a power loss during the programming of the upper page?  The upper page data will most likely be lost, and the lower page may be changed, but there's a good chance of recovery, because it will be in the range of SLC.  It is highly recommended to read and refresh data after a power loss.


Jeff Lauruhn
NAND Application Engineer
Embedded Business Unit

-----Original Message-----
From: Iwo Mergler [mailto:Iwo.Mergler@netcommwireless.com]
Sent: Sunday, March 22, 2015 9:09 PM
To: Richard Weinberger; dedekind1@gmail.com
Cc: Andrea Scian; mtd_mailinglist; Jeff Lauruhn (jlauruhn); Qi Wang 王起 (qiwang)
Subject: RE: RFC: detect and manage power cut on MLC NAND


Hi all,


I probably don't know enough about the silicon implementation of MLC paired pages, but my feeling is that there should be a way to recover one of the pages if the paired write fails, at least in some cases.

Something along the lines of using both bits to map to a single good one.

2 bit MLC stores 4 levels - 1.0, 0.7, 0.3, 0.0. Obviously, the actual voltage levels will be somewhat different, so take this as electrons on the floating gate: 1.0=minimum, 0.0 maximum.

I imagine that there are two ways to achieve that - small step for low page and large step for high page, or the other way 'round.

Assuming the first, the low page write would subtract 0.3 from the erased (1.0) cell if the bit is 0. That leaves the cell at either ~1.0 (1) or 0.7 (0).

Lvl    LH
===========
1.0 => 1u
0.7 => 0u

Then, the high page write would subtract either nothing (1) or
0.7 (0):

Lvl    LH
===========
1.0 => 11
0.7 => 01
0.3 => 10
0.0 => 00

So the MLC decoder logic gets 3 priority encoded bits from the sense amplifiers: 111, 011, 001, 000. The decoder turns this into 11, 01, 10, 00.

The process of writing a 0 to the high page, transitions low page 0-bits through 1 and back to 0, as the level moves down.

Low page 1 bits transition from 1 through 0 and back to 1.

So a half-completed high page 0-write can flip a low page bit both ways.

We can detect an incorrect 0-1 transition in the low page, because it's marked by a 0 bit in the high page.

We can't detect an incorrect 1-0 transition in the low page.

So assuming a failed high page write, this is what we get:

LH

11 = nothing happens, reads back as 11
     Correct level for both.

01 = Level stays at 0.7, reads back as 01,
     Correct level for low page.

10 = Level between 1.0 and 0.3, reads back as 11, 01 or 10.
     01 is wrong for low page, but can't be distinguished from 10.

00 = Level between 0.7 and 0.0, reads back as 01, 10, or 00.
     10 is wrong for low page, but can be distinguished from 01.

So, there are two bit combinations (50%) that have an undetectable failure, and this failure will happen about half the time, for a total of 25% unfixable failure rate.

Not acceptable in the general case, but might be good enough for things like UBI EC & VID headers, if we ensure that the high page contains 1s at the offsets at which the low page stores the header.


Now, on the other hand, if the low page write uses the larger step, there shouldn't be any paired page problem at all, since the high page write wouldn't cross the low page thresholds on the way:

Lvl    LH
===========
1.0 => 1u
0.3 => 0u

Lvl    LH
===========
1.0 => 11
0.7 => 10
0.3 => 01
0.0 => 00


Which makes me think I'm misunderstanding something. If not, why isn't his scheme used in the first place?

What would happen if we reverse the paired page writing order?
Not recommended, we want pages programmed in sequence to mitigate disturbs and obtain the highest reliability.


Jeff, Qi, is the mechanism I described here anywhere near reality?
It's a simplified view, but fairly accurate.


Best regards,

Iwo

RE: RFC: detect and manage power cut on MLC NAND

[Iwo Mergler]

> From a simplified point of view you're right.  In reality the
> program/erase recipes are actually quite advanced in order to get
> very tight distributions on a full page.  The lower/upper page
> sequence is designed to provide the most reliable results and
> optimally we would like the lower and upper page programmed 100% of
> the time.   There's been a lot of work done over the years to improve
> power loss and it's much better than in the past, but it's still
> something to be avoided on NAND.  It's always best to check the
> integrity of the page after a power loss event. 

Is there any way to check the page integrity beyond ECC?

I'm concerned that the power loss could yield an OK looking
page, but with not so tight charge distribution.

Maybe the hardware that can achieve tight distributions during
programming, can be accessed to measure distribution of a
programmed page?

> I want to make sure I understand your meaning of "aborted write".  If
> by aborted write you are referring to stopping the command, not a
> power loss event, then yes the internal circuitry will complete the
> write.  

I did mean power loss.

> I have to be careful here because it's very dependent on the design
> and I really need to know the specifics to make a definitive
> statement, but a few ms should be enough time to protect the NAND.
> WP# is your friend here.

The design is somewhat hypothetical - let's assume that we can
guarantee the NAND supply for 10ms after system reset asserts.

At reset time, the NAND controller will abort any command sequence in
progress, so the final "program page" command will be sent either before
the reset, or not at all. The command byte may be cut short on the bus.

Would I gain anything by tying WP# to the system RESET# in this case?

In other words, is there a way to electrically mangle the "program page"
command on the bus (i.e. short timing or such) that could do something
else than do nothing or start a page write? Would a simultaneous WP#
help?


Apologies for all the questions. I'm very happy to talk to someone
at the coal face of modern NAND manufacturing. :-)


Best regards,

Iwo



________________________________________
From: Jeff Lauruhn (jlauruhn) [jlauruhn@micron.com]
Sent: Wednesday, 25 March 2015 3:50 AM
To: Iwo Mergler; Richard Weinberger; dedekind1@gmail.com
Cc: Andrea Scian; mtd_mailinglist; Qi Wang 王起 (qiwang)
Subject: RE: RFC: detect and manage power cut on MLC NAND

>From a simplified point of view you're right.  In reality the program/erase recipes are actually quite advanced in order to get very tight distributions on a full page.  The lower/upper page sequence is designed to provide the most reliable results and optimally we would like the lower and upper page programmed 100% of the time.  There's been a lot of work done over the years to improve power loss and it's much better than in the past, but it's still something to be avoided on NAND.  It's always best to check the integrity of the page after a power loss event.

I want to make sure I understand your meaning of "aborted write".  If by aborted write you are referring to stopping the command, not a power loss event, then yes the internal circuitry will complete the write.

"What about the typical system, which has, say, a 3.3V supply rail and a system reset firing when that falls below 3.0V or so. The processor core and NAND operate at 1.8V, so there are a few ms between the last possible NAND command and the breakdown of the NAND supply."

I have to be careful here because it's very dependent on the design and I really need to know the specifics to make a definitive statement, but a few ms should be enough time to protect the NAND.  WP# is your friend here.


Jeff Lauruhn
NAND Application Engineer
Embedded Business Unit
Micron Technology, Inc


-----Original Message-----
From: Iwo Mergler [mailto:Iwo.Mergler@netcommwireless.com]
Sent: Monday, March 23, 2015 6:18 PM
To: Jeff Lauruhn (jlauruhn); Richard Weinberger; dedekind1@gmail.com
Cc: Andrea Scian; mtd_mailinglist; Qi Wang 王起 (qiwang)
Subject: RE: RFC: detect and manage power cut on MLC NAND

Hi Jeff,


thanks for the info. That sounds like very good news to me - it appears that the paired page problem isn't as bad as I thought.

> Lvl    LH
> ===========
> 1.0 => 11
> 0.7 => 01
> 0.3 => 10
> 0.0 => 00
         ||
Is this  HL?

This seems to match my second scenario (large step for low page) - we do not lose the low page when the high page write fails. Here is what I understood:

The low page distribution centres around 1.0 and 0.3, with a spread of 0.7. Reading the low page as MLC at this point will classify everything above 0.5 as 1 and below as 0.

So the distance of a '1' from the threshold is 0.5, and the distance of a '0' is 0.2. It's asymmetric, with SLC distance on one side and MLC on the other.

The high page write (only when writing a '0'), moves the charge towards 0.0, by 0.3. So if the first page contains '1', the charge is lowered to 0.7, if the first page contains '0' the charge goes down to 0.0.

At no time is the charge crossing the 0.5 threshold.
An aborted high page write will have reduced the '1'
distance by 0.0-0.3 or increased the '0' distance by 0.0-0.3.

Why are we worried about the low page at all? The way I understand the gate charge changes, the low page situation can not be, at any point, worse than the normal programmed MLC state.


Losing the high page isn't conceptually any worse than aborted writes on SLC. This has been called the "unstable bits problem".

As far as I understand, UBI/UBIFS currently don't mitigate for aborted write. The worry here is that an aborted write (SLC or MLC) gets you a page where the 0s are weak - they may pass ECC on read now but fail next month. Or if the abort was early, weaken some 1s in an erased-looking page.

Aborted erases are handled by UBI via the EC header write immediately after erase. An erased block without an EC header is erased again at boot time, to avoid partially erased blocks (weak 1s).

Aborted writes are harder. UBI could deal with its own headers via sequence number and always rewrite the last block when it has UBI headers but no payload.
Seems a bit wasteful, though.

UBIFS also has an idea which few blocks may have received a power cut during the last write - I think around 5 candidates or so. Rewriting them all every mount time seems even more wasteful.

Then of course, the rewrite can also be aborted, so UBI's last atomic LEB change probably also needs redoing at attach time.


Jeff, what's the situation with aborted writes?
Is this problem real? I understand that there are charge pumps on board the NAND chip - do they store sufficient energy to complete a page write?

What about the typical system, which has, say, a 3.3V supply rail and a system reset firing when that falls below 3.0V or so. The processor core and NAND operate at 1.8V, so there are a few ms between the last possible NAND command and the breakdown of the NAND supply.

Would that be safe from aborted writes?


Best regards,

Iwo


________________________________________
From: Jeff Lauruhn (jlauruhn) [jlauruhn@micron.com]
Sent: Tuesday, 24 March 2015 8:15 AM
To: Iwo Mergler; Richard Weinberger; dedekind1@gmail.com
Cc: Andrea Scian; mtd_mailinglist; Qi Wang 王起 (qiwang)
Subject: RE: RFC: detect and manage power cut on MLC NAND

This is a very simplified description, but actually it's more like this:

First pass, program the lower page.  If you the lower page is 1, do nothing.  If the lower page is 0 subtract 0.7v to 0.3.  Lower page is SLC like, two distributions spread apart by 0.7V.

Lvl    LH
===========
1.0 => 1u
0.3 => 0u

Now, program the upper page.  First, read lower page, if lower page is 1 and upper page is 1, do nothing (11).  If lower page is 1 and upper page is 0, then subtract -0.3 and call that 01.  Next if lower page is 0 and upper page is 1 do nothing and if lower page is 0 and upper page is 0 subtract 0.3v and call it 00.  Notice that state of lower page is on right of 11, 01,10, 00.

Lvl    LH
===========
1.0 => 11
0.7 => 01
0.3 => 10
0.0 => 00

Now what happens if there's a power loss during the programming of the upper page?  The upper page data will most likely be lost, and the lower page may be changed, but there's a good chance of recovery, because it will be in the range of SLC.  It is highly recommended to read and refresh data after a power loss.


Jeff Lauruhn
NAND Application Engineer
Embedded Business Unit

-----Original Message-----
From: Iwo Mergler [mailto:Iwo.Mergler@netcommwireless.com]
Sent: Sunday, March 22, 2015 9:09 PM
To: Richard Weinberger; dedekind1@gmail.com
Cc: Andrea Scian; mtd_mailinglist; Jeff Lauruhn (jlauruhn); Qi Wang 王起 (qiwang)
Subject: RE: RFC: detect and manage power cut on MLC NAND


Hi all,


I probably don't know enough about the silicon implementation of MLC paired pages, but my feeling is that there should be a way to recover one of the pages if the paired write fails, at least in some cases.

Something along the lines of using both bits to map to a single good one.

2 bit MLC stores 4 levels - 1.0, 0.7, 0.3, 0.0. Obviously, the actual voltage levels will be somewhat different, so take this as electrons on the floating gate: 1.0=minimum, 0.0 maximum.

I imagine that there are two ways to achieve that - small step for low page and large step for high page, or the other way 'round.

Assuming the first, the low page write would subtract 0.3 from the erased (1.0) cell if the bit is 0. That leaves the cell at either ~1.0 (1) or 0.7 (0).

Lvl    LH
===========
1.0 => 1u
0.7 => 0u

Then, the high page write would subtract either nothing (1) or
0.7 (0):

Lvl    LH
===========
1.0 => 11
0.7 => 01
0.3 => 10
0.0 => 00

So the MLC decoder logic gets 3 priority encoded bits from the sense amplifiers: 111, 011, 001, 000. The decoder turns this into 11, 01, 10, 00.

The process of writing a 0 to the high page, transitions low page 0-bits through 1 and back to 0, as the level moves down.

Low page 1 bits transition from 1 through 0 and back to 1.

So a half-completed high page 0-write can flip a low page bit both ways.

We can detect an incorrect 0-1 transition in the low page, because it's marked by a 0 bit in the high page.

We can't detect an incorrect 1-0 transition in the low page.

So assuming a failed high page write, this is what we get:

LH

11 = nothing happens, reads back as 11
     Correct level for both.

01 = Level stays at 0.7, reads back as 01,
     Correct level for low page.

10 = Level between 1.0 and 0.3, reads back as 11, 01 or 10.
     01 is wrong for low page, but can't be distinguished from 10.

00 = Level between 0.7 and 0.0, reads back as 01, 10, or 00.
     10 is wrong for low page, but can be distinguished from 01.

So, there are two bit combinations (50%) that have an undetectable failure, and this failure will happen about half the time, for a total of 25% unfixable failure rate.

Not acceptable in the general case, but might be good enough for things like UBI EC & VID headers, if we ensure that the high page contains 1s at the offsets at which the low page stores the header.


Now, on the other hand, if the low page write uses the larger step, there shouldn't be any paired page problem at all, since the high page write wouldn't cross the low page thresholds on the way:

Lvl    LH
===========
1.0 => 1u
0.3 => 0u

Lvl    LH
===========
1.0 => 11
0.7 => 10
0.3 => 01
0.0 => 00


Which makes me think I'm misunderstanding something. If not, why isn't his scheme used in the first place?

What would happen if we reverse the paired page writing order?
Not recommended, we want pages programmed in sequence to mitigate disturbs and obtain the highest reliability.


Jeff, Qi, is the mechanism I described here anywhere near reality?
It's a simplified view, but fairly accurate.


Best regards,

Iwo

RE: RFC: detect and manage power cut on MLC NAND

[Ricard Wanderlof]

On Wed, 25 Mar 2015, Iwo Mergler wrote:

> > From a simplified point of view you're right.  In reality the
> > program/erase recipes are actually quite advanced in order to get
> > very tight distributions on a full page.  The lower/upper page
> > sequence is designed to provide the most reliable results and
> > optimally we would like the lower and upper page programmed 100% of
> > the time.   There's been a lot of work done over the years to improve
> > power loss and it's much better than in the past, but it's still
> > something to be avoided on NAND.  It's always best to check the
> > integrity of the page after a power loss event.
> 
> Is there any way to check the page integrity beyond ECC?
>
> I'm concerned that the power loss could yield an OK looking
> page, but with not so tight charge distribution.
> 
> Maybe the hardware that can achieve tight distributions during
> programming, can be accessed to measure distribution of a
> programmed page?

What would be interesting from a software perspective would be if in some 
special mode one could read the read the memory cells and get an analog 
value with several bits of resolution, alowing the software to make an 
assessment as to how "good" the bits are. This would be in contrast to the 
normal, high speed, read mode. But perhaps matters are not that simple, 
either there is no such value to be had (but as I understand it in certain 
MLC flashes it is possible to shift the read thresholds, thus one could 
accomplish this by successive approximation. Sure, that means that one 
could do it entirely in software using existing devices, but it is a 
rather cumbersome process however), or there are other factors that govern 
the read thresholds which are not known outside the chip (or rather, 
outside the manufacturers lab!).

> > I have to be careful here because it's very dependent on the design
> > and I really need to know the specifics to make a definitive
> > statement, but a few ms should be enough time to protect the NAND.
> > WP# is your friend here.
> 
> The design is somewhat hypothetical - let's assume that we can
> guarantee the NAND supply for 10ms after system reset asserts.
> 
> At reset time, the NAND controller will abort any command sequence in
> progress, so the final "program page" command will be sent either before
> the reset, or not at all. The command byte may be cut short on the bus.

It would seem to me that the only thing really needed to guarantee that 
writes (or erase operations) are not cut short by power loss, is as Iwo 
says that the system design is such that when power loss occurs, there is 
enough power to maintain valid supply voltage levels to allow the NAND to 
complete operations in the worst case, after system reset is asserted.

Admittedly we don't always have the luxury of well-designed hardware, but 
having clear design rules for the hardware guys would help a long way in 
future designs.

> I'm very happy to talk to someone at the coal face of modern NAND 
> manufacturing. :-)

Agreed, I think we're very many that appreciate Jeff's contributions on 
the list, me included. NAND data sheets are often not so forthcoming, and 
there ends up being a lot of speculation about how things actually work, 
so it's really nice to have someone with real knowledge to discuss this 
with.

/Ricard
-- 
Ricard Wolf Wanderlöf                           ricardw(at)axis.com
Axis Communications AB, Lund, Sweden            www.axis.com
Phone +46 46 272 2016                           Fax +46 46 13 61 30

RE: RFC: detect and manage power cut on MLC NAND

[Jeff Lauruhn (jlauruhn)]

Jeff Lauruhn
NAND Application Engineer
Embedded Business Unit
Micron Technology, Inc


-----Original Message-----
From: Ricard Wanderlof [mailto:ricard.wanderlof@axis.com] 
Sent: Wednesday, March 25, 2015 1:33 AM
To: Iwo Mergler
Cc: Jeff Lauruhn (jlauruhn); Richard Weinberger; dedekind1@gmail.com; Andrea Scian; Qi Wang 王起 (qiwang); mtd_mailinglist
Subject: RE: RFC: detect and manage power cut on MLC NAND


On Wed, 25 Mar 2015, Iwo Mergler wrote:

> > From a simplified point of view you're right.  In reality the 
> > program/erase recipes are actually quite advanced in order to get 
> > very tight distributions on a full page.  The lower/upper page 
> > sequence is designed to provide the most reliable results and 
> > optimally we would like the lower and upper page programmed 100% of
> > the time.   There's been a lot of work done over the years to improve
> > power loss and it's much better than in the past, but it's still 
> > something to be avoided on NAND.  It's always best to check the 
> > integrity of the page after a power loss event.
> 
> Is there any way to check the page integrity beyond ECC?
>
> I'm concerned that the power loss could yield an OK looking page, but 
> with not so tight charge distribution.
> 
> Maybe the hardware that can achieve tight distributions during 
> programming, can be accessed to measure distribution of a programmed 
> page?

What would be interesting from a software perspective would be if in some special mode one could read the read the memory cells and get an analog value with several bits of resolution, alowing the software to make an assessment as to how "good" the bits are. This would be in contrast to the normal, high speed, read mode. But perhaps matters are not that simple, either there is no such value to be had (but as I understand it in certain MLC flashes it is possible to shift the read thresholds, thus one could accomplish this by successive approximation. Sure, that means that one could do it entirely in software using existing devices, but it is a rather cumbersome process however), or there are other factors that govern the read thresholds which are not known outside the chip (or rather, outside the manufacturers lab!).

No special analog modes on production devices, but we are moving in the direction of giving more control to the end user.  As lithography goes down and bits per cell goes up we are adding we are trying to come up with manageable ways to recover data.  There's no analog read out on the roadmap, but new features like read retry, which generally assumes charge loss and allows the end user to try different read reference voltages and other read offset features are on the road map.  

Let me explain the read process a bit.  When we program and erase too, we set a target value L0 and L1 in the case of SLC and we get a distribution around that those values.  But when we read we apply a voltage to the gate of the cell we intend to read that is between L0 and L1, call it Vread, if the cell is erased, Vread is greater than the voltage threshold Vt of L0 and the cell will conduct and we will sense a current flow between the Drain and Source and the sense circuit registers a 1 for that cell.  If the cell is programmed, Vread will not be high enough to overcome the Vt of the cell and we will sense no current flow between the drain and source and the sense circuit registers a 0 for that cell.  The sense circuit is very simple because there needs to be 2K of them so we can sense the whole page simultaneously.  The type of circuitry required to measure an analog value would make the die huge.  If it was possible, it would already be designed in.  

Instead of measuring the cell voltage, it was easier to allow the end user to move Vread to maybe compensate for the shift in distribution.  
              

> > I have to be careful here because it's very dependent on the design 
> > and I really need to know the specifics to make a definitive 
> > statement, but a few ms should be enough time to protect the NAND.
> > WP# is your friend here.
> 
> The design is somewhat hypothetical - let's assume that we can 
> guarantee the NAND supply for 10ms after system reset asserts.
> 
> At reset time, the NAND controller will abort any command sequence in 
> progress, so the final "program page" command will be sent either 
> before the reset, or not at all. The command byte may be cut short on the bus.

It would seem to me that the only thing really needed to guarantee that writes (or erase operations) are not cut short by power loss, is as Iwo says that the system design is such that when power loss occurs, there is enough power to maintain valid supply voltage levels to allow the NAND to complete operations in the worst case, after system reset is asserted.

Admittedly we don't always have the luxury of well-designed hardware, but having clear design rules for the hardware guys would help a long way in future designs.

> I'm very happy to talk to someone at the coal face of modern NAND 
> manufacturing. :-)

Agreed, I think we're very many that appreciate Jeff's contributions on the list, me included. NAND data sheets are often not so forthcoming, and there ends up being a lot of speculation about how things actually work, so it's really nice to have someone with real knowledge to discuss this with.

/Ricard
-- 
Ricard Wolf Wanderlöf                           ricardw(at)axis.com
Axis Communications AB, Lund, Sweden            www.axis.com
Phone +46 46 272 2016                           Fax +46 46 13 61 30

RE: RFC: detect and manage power cut on MLC NAND

[Ricard Wanderlof]

On Thu, 26 Mar 2015, Jeff Lauruhn (jlauruhn) wrote:

> Let me explain the read process a bit.  When we program and erase too, 
> we set a target value L0 and L1 in the case of SLC and we get a 
> ...
> registers a 0 for that cell.  The sense circuit is very simple because 
> there needs to be 2K of them so we can sense the whole page 
> simultaneously.  The type of circuitry required to measure an analog 
> value would make the die huge.  If it was possible, it would already be 
> designed in.

> Instead of measuring the cell voltage, it was easier to allow the end 
> user to move Vread to maybe compensate for the shift in distribution.

Makes perfect sense. Thanks for the response!

/Ricard
-- 
Ricard Wolf Wanderlöf                           ricardw(at)axis.com
Axis Communications AB, Lund, Sweden            www.axis.com
Phone +46 46 272 2016                           Fax +46 46 13 61 30

RE: RFC: detect and manage power cut on MLC NAND

[Andrea Marson - DAVE Embedded Systems]

Hi Jeff,

thank you for your availability. I'd really love to discuss in more 
detail these matters.

About Write Protect (WP) pin, IIUC it must respect several constraints. 
For example it must not be transitioned while NAND is busy. What if it 
is set low while NAND is ready - that is it can accept new commands - 
but an erase operation is in progress? Does the erase operation complete 
anyway?

Andrea Marson
*DAVE Embedded Systems*



> Power loss is actually very complex.  The Write Protect (WP) pin was added =
> to NAND help lock the NAND when a power loss event is detected.  I have ext=
> ensive information on NAND and would be happy to discuss.  =20
>
>
> Jeff Lauruhn
> NAND Application Engineer
> Embedded Business Unit
> Micron Technology, Inc
>
>
> -----Original Message-----
> From: linux-mtd [mailto:linux-mtd-bounces@lists.infradead.org] On Behalf Of=
>   Andrea Scian
> Sent: Wednesday, March 11, 2015 10:01 AM
> To: Richard Weinberger; dedekind1@gmail.com
> Cc: mtd_mailinglist
> Subject: Re: RFC: detect and manage power cut on MLC NAND
>
>
> Hi all,
>
> and thanks for you feedback.
> You can find my comments below.
>
> Il 11/03/2015 10:09, Richard Weinberger ha scritto:
>> Am 11.03.2015 um 10:05 schrieb Artem Bityutskiy:
>>> On Wed, 2015-03-11 at 09:57 +0100, Richard Weinberger wrote:
>>>> Hi!
>>>>
>>>> Am 11.03.2015 um 08:20 schrieb Artem Bityutskiy:
>>>>> On Tue, 2015-03-10 at 13:51 +0100, Richard Weinberger wrote:
>>>>>>> WDYT about this?
>>>>>>> If it sounds reasonable is there any suggestion where to place such =
> a code?
>>>>>> Customers often use DYI uninterruptible power supplies using capacito=
> rs.
>>>>>> But managing a power cut is the least problem you have with MLC NAND.
>>>>> Why is it the least problem, what is the hardest one? I thought=20
>>>>> this one is the hardest.
>>>> IMHO the hardest ones are the problems we don't know yet as NAND=20
>>>> vendors are not really chatty about the MLC constraints.
>>>> We don't know much about data retention for example. At least we=20
>>>> have not much hard facts. Most of our knowledge is hearsay.
>>> Well, but from the problems we know paired pages seems to be the=20
>>> biggest one. E.g., what do we do if VID header gets corrupted because=20
>>> of an interrupted write to the page paired with the VID header page?=20
>>> Sounds like a hard problem to me.
>> It is a hard problem. But at least we know about it.
>>
> IIUC can I summarize like the following
>
> 1) power cut is one of the big problem we know about MLC NAND flash with UB=
> I
> 2) we are unaware of other big issue with MLC NAND
>
> Unfortunately I don't really know the MTD/UBI/UBIFS internals but (correct =
> me if I'm wrong) to me solving the power cut issue only by software (by usi=
> ng proper data structure, redundancy, NAND flash operation jornal and so on=
> ) is pretty hard to implement inside the Linux MTD stack.
>
> On the other hand, stuff like scrubbing (e.g. having a task like GC that pe=
> riodically scrub the required NAND erase block), paired page (e.g.
> carefully selecting LEB to PEB mapping to avoid reading/writing data into p=
> aired page in a wrong way) are still to be implemented but have, somehow, l=
> ess impact on the whole MTD stack.
>
> For sure I'm missing some other MLC NAND issue, but the above are the one t=
> hat I'm aware of.
>
> WDYT?
>
> Best Regards,
>
> --=20
>
> Andrea SCIAN
>
> DAVE Embedded Systems
>
>
>