* RFC: detect and manage power cut on MLC NAND @ 2015-03-10 11:57 Andrea Scian 2015-03-10 12:51 ` Richard Weinberger 2015-03-11 7:21 ` Artem Bityutskiy 0 siblings, 2 replies; 59+ messages in thread From: Andrea Scian @ 2015-03-10 11:57 UTC (permalink / raw) To: mtd_mailinglist Dear All, for testing and research purpose, I'm working with an Embedded Linux product which mounts an MLC NAND. I know that one of the trouble with MLC flash on Linux with UBI/UBIFS is the power cut management (ref. http://www.linux-mtd.infradead.org/doc/ubifs.html#L_unstable_bits) IIUC the solution inside MTD/UBI/UBIFS is not so simple and thus not so quick to implement. I'm wondering if anyone of you already think about managing power cut in a different way: many embedded/industrial products already have some big/huge bypass capacitor that are able to keep the system alive for hundreds of milliseconds (this is useful for power supply micro-interruptions), which are commonly enough to complete the latest NAND operation. Most of the has a GPIO/IRQ that tells whenever such a power cut happens, so the kernel can take some actions with it (e.g., in our case, complete the last NAND operation and prevent from starting newer operation, which may corrupt the NAND pages). WDYT about this? If it sounds reasonable is there any suggestion where to place such a code? Best Regards, -- Andrea SCIAN DAVE Embedded Systems ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: RFC: detect and manage power cut on MLC NAND 2015-03-10 11:57 RFC: detect and manage power cut on MLC NAND Andrea Scian @ 2015-03-10 12:51 ` Richard Weinberger 2015-03-11 7:20 ` Artem Bityutskiy 2015-03-11 7:21 ` Artem Bityutskiy 1 sibling, 1 reply; 59+ messages in thread From: Richard Weinberger @ 2015-03-10 12:51 UTC (permalink / raw) To: Andrea Scian; +Cc: mtd_mailinglist On Tue, Mar 10, 2015 at 12:57 PM, Andrea Scian <rnd4@dave-tech.it> wrote: > > Dear All, > > for testing and research purpose, I'm working with an Embedded Linux > product which mounts an MLC NAND. > > I know that one of the trouble with MLC flash on Linux with UBI/UBIFS is > the power cut management (ref. > http://www.linux-mtd.infradead.org/doc/ubifs.html#L_unstable_bits) > > IIUC the solution inside MTD/UBI/UBIFS is not so simple and thus not so > quick to implement. > > I'm wondering if anyone of you already think about managing power cut in > a different way: many embedded/industrial products already have some > big/huge bypass capacitor that are able to keep the system alive for > hundreds of milliseconds (this is useful for power supply > micro-interruptions), which are commonly enough to complete the latest > NAND operation. > Most of the has a GPIO/IRQ that tells whenever such a power cut happens, > so the kernel can take some actions with it (e.g., in our case, complete > the last NAND operation and prevent from starting newer operation, which > may corrupt the NAND pages). > > WDYT about this? > If it sounds reasonable is there any suggestion where to place such a code? Customers often use DYI uninterruptible power supplies using capacitors. But managing a power cut is the least problem you have with MLC NAND. -- Thanks, //richard ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: RFC: detect and manage power cut on MLC NAND 2015-03-10 12:51 ` Richard Weinberger @ 2015-03-11 7:20 ` Artem Bityutskiy 2015-03-11 8:57 ` Richard Weinberger 0 siblings, 1 reply; 59+ messages in thread From: Artem Bityutskiy @ 2015-03-11 7:20 UTC (permalink / raw) To: Richard Weinberger; +Cc: Andrea Scian, mtd_mailinglist On Tue, 2015-03-10 at 13:51 +0100, Richard Weinberger wrote: > > WDYT about this? > > If it sounds reasonable is there any suggestion where to place such a code? > > Customers often use DYI uninterruptible power supplies using capacitors. > But managing a power cut is the least problem you have with MLC NAND. Why is it the least problem, what is the hardest one? I thought this one is the hardest. Artem. ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: RFC: detect and manage power cut on MLC NAND 2015-03-11 7:20 ` Artem Bityutskiy @ 2015-03-11 8:57 ` Richard Weinberger 2015-03-11 9:05 ` Artem Bityutskiy 0 siblings, 1 reply; 59+ messages in thread From: Richard Weinberger @ 2015-03-11 8:57 UTC (permalink / raw) To: dedekind1; +Cc: Andrea Scian, mtd_mailinglist Hi! Am 11.03.2015 um 08:20 schrieb Artem Bityutskiy: > On Tue, 2015-03-10 at 13:51 +0100, Richard Weinberger wrote: >>> WDYT about this? >>> If it sounds reasonable is there any suggestion where to place such a code? >> >> Customers often use DYI uninterruptible power supplies using capacitors. >> But managing a power cut is the least problem you have with MLC NAND. > > Why is it the least problem, what is the hardest one? I thought this one > is the hardest. IMHO the hardest ones are the problems we don't know yet as NAND vendors are not really chatty about the MLC constraints. We don't know much about data retention for example. At least we have not much hard facts. Most of our knowledge is hearsay. What I wanted to say is that soldering a big capacitor onto your board will not make MLC NAND magically work perfectly fine. :-) Thanks, //richard ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: RFC: detect and manage power cut on MLC NAND 2015-03-11 8:57 ` Richard Weinberger @ 2015-03-11 9:05 ` Artem Bityutskiy 2015-03-11 9:09 ` Richard Weinberger 0 siblings, 1 reply; 59+ messages in thread From: Artem Bityutskiy @ 2015-03-11 9:05 UTC (permalink / raw) To: Richard Weinberger; +Cc: Andrea Scian, mtd_mailinglist On Wed, 2015-03-11 at 09:57 +0100, Richard Weinberger wrote: > Hi! > > Am 11.03.2015 um 08:20 schrieb Artem Bityutskiy: > > On Tue, 2015-03-10 at 13:51 +0100, Richard Weinberger wrote: > >>> WDYT about this? > >>> If it sounds reasonable is there any suggestion where to place such a code? > >> > >> Customers often use DYI uninterruptible power supplies using capacitors. > >> But managing a power cut is the least problem you have with MLC NAND. > > > > Why is it the least problem, what is the hardest one? I thought this one > > is the hardest. > > IMHO the hardest ones are the problems we don't know yet as NAND vendors > are not really chatty about the MLC constraints. > We don't know much about data retention for example. At least we have not > much hard facts. Most of our knowledge is hearsay. Well, but from the problems we know paired pages seems to be the biggest one. E.g., what do we do if VID header gets corrupted because of an interrupted write to the page paired with the VID header page? Sounds like a hard problem to me. As for other things, I am not sure. I remember back when I worked for Nokia, vendors did tell us everything we needed to know about their flash product. ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: RFC: detect and manage power cut on MLC NAND 2015-03-11 9:05 ` Artem Bityutskiy @ 2015-03-11 9:09 ` Richard Weinberger 2015-03-11 17:01 ` Andrea Scian 2015-03-23 4:08 ` Iwo Mergler 0 siblings, 2 replies; 59+ messages in thread From: Richard Weinberger @ 2015-03-11 9:09 UTC (permalink / raw) To: dedekind1; +Cc: Andrea Scian, mtd_mailinglist Am 11.03.2015 um 10:05 schrieb Artem Bityutskiy: > On Wed, 2015-03-11 at 09:57 +0100, Richard Weinberger wrote: >> Hi! >> >> Am 11.03.2015 um 08:20 schrieb Artem Bityutskiy: >>> On Tue, 2015-03-10 at 13:51 +0100, Richard Weinberger wrote: >>>>> WDYT about this? >>>>> If it sounds reasonable is there any suggestion where to place such a code? >>>> >>>> Customers often use DYI uninterruptible power supplies using capacitors. >>>> But managing a power cut is the least problem you have with MLC NAND. >>> >>> Why is it the least problem, what is the hardest one? I thought this one >>> is the hardest. >> >> IMHO the hardest ones are the problems we don't know yet as NAND vendors >> are not really chatty about the MLC constraints. >> We don't know much about data retention for example. At least we have not >> much hard facts. Most of our knowledge is hearsay. > > Well, but from the problems we know paired pages seems to be the biggest > one. E.g., what do we do if VID header gets corrupted because of an > interrupted write to the page paired with the VID header page? Sounds > like a hard problem to me. It is a hard problem. But at least we know about it. > As for other things, I am not sure. I remember back when I worked for > Nokia, vendors did tell us everything we needed to know about their > flash product. > Nokia is/was not really a small player. Thanks, //richard ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: RFC: detect and manage power cut on MLC NAND 2015-03-11 9:09 ` Richard Weinberger @ 2015-03-11 17:01 ` Andrea Scian 2015-03-11 17:23 ` Jeff Lauruhn (jlauruhn) 2015-03-23 4:08 ` Iwo Mergler 1 sibling, 1 reply; 59+ messages in thread From: Andrea Scian @ 2015-03-11 17:01 UTC (permalink / raw) To: Richard Weinberger, dedekind1; +Cc: mtd_mailinglist Hi all, and thanks for you feedback. You can find my comments below. Il 11/03/2015 10:09, Richard Weinberger ha scritto: > Am 11.03.2015 um 10:05 schrieb Artem Bityutskiy: >> On Wed, 2015-03-11 at 09:57 +0100, Richard Weinberger wrote: >>> Hi! >>> >>> Am 11.03.2015 um 08:20 schrieb Artem Bityutskiy: >>>> On Tue, 2015-03-10 at 13:51 +0100, Richard Weinberger wrote: >>>>>> WDYT about this? >>>>>> If it sounds reasonable is there any suggestion where to place such a code? >>>>> Customers often use DYI uninterruptible power supplies using capacitors. >>>>> But managing a power cut is the least problem you have with MLC NAND. >>>> Why is it the least problem, what is the hardest one? I thought this one >>>> is the hardest. >>> IMHO the hardest ones are the problems we don't know yet as NAND vendors >>> are not really chatty about the MLC constraints. >>> We don't know much about data retention for example. At least we have not >>> much hard facts. Most of our knowledge is hearsay. >> Well, but from the problems we know paired pages seems to be the biggest >> one. E.g., what do we do if VID header gets corrupted because of an >> interrupted write to the page paired with the VID header page? Sounds >> like a hard problem to me. > It is a hard problem. But at least we know about it. > IIUC can I summarize like the following 1) power cut is one of the big problem we know about MLC NAND flash with UBI 2) we are unaware of other big issue with MLC NAND Unfortunately I don't really know the MTD/UBI/UBIFS internals but (correct me if I'm wrong) to me solving the power cut issue only by software (by using proper data structure, redundancy, NAND flash operation jornal and so on) is pretty hard to implement inside the Linux MTD stack. On the other hand, stuff like scrubbing (e.g. having a task like GC that periodically scrub the required NAND erase block), paired page (e.g. carefully selecting LEB to PEB mapping to avoid reading/writing data into paired page in a wrong way) are still to be implemented but have, somehow, less impact on the whole MTD stack. For sure I'm missing some other MLC NAND issue, but the above are the one that I'm aware of. WDYT? Best Regards, -- Andrea SCIAN DAVE Embedded Systems ^ permalink raw reply [flat|nested] 59+ messages in thread
* RE: RFC: detect and manage power cut on MLC NAND 2015-03-11 17:01 ` Andrea Scian @ 2015-03-11 17:23 ` Jeff Lauruhn (jlauruhn) 2015-03-11 17:29 ` Richard Weinberger 2015-03-12 9:32 ` Ricard Wanderlof 0 siblings, 2 replies; 59+ messages in thread From: Jeff Lauruhn (jlauruhn) @ 2015-03-11 17:23 UTC (permalink / raw) To: Andrea Scian, Richard Weinberger, dedekind1; +Cc: mtd_mailinglist Power loss is actually very complex. The Write Protect (WP) pin was added to NAND help lock the NAND when a power loss event is detected. I have extensive information on NAND and would be happy to discuss. Jeff Lauruhn NAND Application Engineer Embedded Business Unit Micron Technology, Inc -----Original Message----- From: linux-mtd [mailto:linux-mtd-bounces@lists.infradead.org] On Behalf Of Andrea Scian Sent: Wednesday, March 11, 2015 10:01 AM To: Richard Weinberger; dedekind1@gmail.com Cc: mtd_mailinglist Subject: Re: RFC: detect and manage power cut on MLC NAND Hi all, and thanks for you feedback. You can find my comments below. Il 11/03/2015 10:09, Richard Weinberger ha scritto: > Am 11.03.2015 um 10:05 schrieb Artem Bityutskiy: >> On Wed, 2015-03-11 at 09:57 +0100, Richard Weinberger wrote: >>> Hi! >>> >>> Am 11.03.2015 um 08:20 schrieb Artem Bityutskiy: >>>> On Tue, 2015-03-10 at 13:51 +0100, Richard Weinberger wrote: >>>>>> WDYT about this? >>>>>> If it sounds reasonable is there any suggestion where to place such a code? >>>>> Customers often use DYI uninterruptible power supplies using capacitors. >>>>> But managing a power cut is the least problem you have with MLC NAND. >>>> Why is it the least problem, what is the hardest one? I thought >>>> this one is the hardest. >>> IMHO the hardest ones are the problems we don't know yet as NAND >>> vendors are not really chatty about the MLC constraints. >>> We don't know much about data retention for example. At least we >>> have not much hard facts. Most of our knowledge is hearsay. >> Well, but from the problems we know paired pages seems to be the >> biggest one. E.g., what do we do if VID header gets corrupted because >> of an interrupted write to the page paired with the VID header page? >> Sounds like a hard problem to me. > It is a hard problem. But at least we know about it. > IIUC can I summarize like the following 1) power cut is one of the big problem we know about MLC NAND flash with UBI 2) we are unaware of other big issue with MLC NAND Unfortunately I don't really know the MTD/UBI/UBIFS internals but (correct me if I'm wrong) to me solving the power cut issue only by software (by using proper data structure, redundancy, NAND flash operation jornal and so on) is pretty hard to implement inside the Linux MTD stack. On the other hand, stuff like scrubbing (e.g. having a task like GC that periodically scrub the required NAND erase block), paired page (e.g. carefully selecting LEB to PEB mapping to avoid reading/writing data into paired page in a wrong way) are still to be implemented but have, somehow, less impact on the whole MTD stack. For sure I'm missing some other MLC NAND issue, but the above are the one that I'm aware of. WDYT? Best Regards, -- Andrea SCIAN DAVE Embedded Systems ______________________________________________________ Linux MTD discussion mailing list http://lists.infradead.org/mailman/listinfo/linux-mtd/ ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: RFC: detect and manage power cut on MLC NAND 2015-03-11 17:23 ` Jeff Lauruhn (jlauruhn) @ 2015-03-11 17:29 ` Richard Weinberger 2015-03-11 21:16 ` Jeff Lauruhn (jlauruhn) 2015-03-12 9:32 ` Ricard Wanderlof 1 sibling, 1 reply; 59+ messages in thread From: Richard Weinberger @ 2015-03-11 17:29 UTC (permalink / raw) To: Jeff Lauruhn (jlauruhn), Andrea Scian, dedekind1 Cc: Boris Brezillon, mtd_mailinglist Jeff, Am 11.03.2015 um 18:23 schrieb Jeff Lauruhn (jlauruhn): > Power loss is actually very complex. The Write Protect (WP) pin was added to NAND help lock the NAND when a power loss event is detected. I have extensive information on NAND and would be happy to discuss. > it would be wonderful if you could share knowledge on MLC NAND. I'm very interested in supporting MLC in UBI (and UBIFS). UBI was designed with SLC NAND in mind, now with MLC some things have changed. Thanks, //richard > Jeff Lauruhn > NAND Application Engineer > Embedded Business Unit > Micron Technology, Inc > > > -----Original Message----- > From: linux-mtd [mailto:linux-mtd-bounces@lists.infradead.org] On Behalf Of Andrea Scian > Sent: Wednesday, March 11, 2015 10:01 AM > To: Richard Weinberger; dedekind1@gmail.com > Cc: mtd_mailinglist > Subject: Re: RFC: detect and manage power cut on MLC NAND > > > Hi all, > > and thanks for you feedback. > You can find my comments below. > > Il 11/03/2015 10:09, Richard Weinberger ha scritto: >> Am 11.03.2015 um 10:05 schrieb Artem Bityutskiy: >>> On Wed, 2015-03-11 at 09:57 +0100, Richard Weinberger wrote: >>>> Hi! >>>> >>>> Am 11.03.2015 um 08:20 schrieb Artem Bityutskiy: >>>>> On Tue, 2015-03-10 at 13:51 +0100, Richard Weinberger wrote: >>>>>>> WDYT about this? >>>>>>> If it sounds reasonable is there any suggestion where to place such a code? >>>>>> Customers often use DYI uninterruptible power supplies using capacitors. >>>>>> But managing a power cut is the least problem you have with MLC NAND. >>>>> Why is it the least problem, what is the hardest one? I thought >>>>> this one is the hardest. >>>> IMHO the hardest ones are the problems we don't know yet as NAND >>>> vendors are not really chatty about the MLC constraints. >>>> We don't know much about data retention for example. At least we >>>> have not much hard facts. Most of our knowledge is hearsay. >>> Well, but from the problems we know paired pages seems to be the >>> biggest one. E.g., what do we do if VID header gets corrupted because >>> of an interrupted write to the page paired with the VID header page? >>> Sounds like a hard problem to me. >> It is a hard problem. But at least we know about it. >> > > IIUC can I summarize like the following > > 1) power cut is one of the big problem we know about MLC NAND flash with UBI > 2) we are unaware of other big issue with MLC NAND > > Unfortunately I don't really know the MTD/UBI/UBIFS internals but (correct me if I'm wrong) to me solving the power cut issue only by software (by using proper data structure, redundancy, NAND flash operation jornal and so on) is pretty hard to implement inside the Linux MTD stack. > > On the other hand, stuff like scrubbing (e.g. having a task like GC that periodically scrub the required NAND erase block), paired page (e.g. > carefully selecting LEB to PEB mapping to avoid reading/writing data into paired page in a wrong way) are still to be implemented but have, somehow, less impact on the whole MTD stack. > > For sure I'm missing some other MLC NAND issue, but the above are the one that I'm aware of. > > WDYT? > > Best Regards, > ^ permalink raw reply [flat|nested] 59+ messages in thread
* RE: RFC: detect and manage power cut on MLC NAND 2015-03-11 17:29 ` Richard Weinberger @ 2015-03-11 21:16 ` Jeff Lauruhn (jlauruhn) 2015-03-12 10:28 ` Richard Weinberger 0 siblings, 1 reply; 59+ messages in thread From: Jeff Lauruhn (jlauruhn) @ 2015-03-11 21:16 UTC (permalink / raw) To: Richard Weinberger, Andrea Scian, dedekind1 Cc: Boris Brezillon, mtd_mailinglist Glad to help out. I train FAE's and customers on many aspects of NAND including MLC. Jeff Lauruhn NAND Application Engineer Embedded Business Unit Micron Technology, Inc -----Original Message----- From: linux-mtd [mailto:linux-mtd-bounces@lists.infradead.org] On Behalf Of Richard Weinberger Sent: Wednesday, March 11, 2015 10:29 AM To: Jeff Lauruhn (jlauruhn); Andrea Scian; dedekind1@gmail.com Cc: Boris Brezillon; mtd_mailinglist Subject: Re: RFC: detect and manage power cut on MLC NAND Jeff, Am 11.03.2015 um 18:23 schrieb Jeff Lauruhn (jlauruhn): > Power loss is actually very complex. The Write Protect (WP) pin was added to NAND help lock the NAND when a power loss event is detected. I have extensive information on NAND and would be happy to discuss. > it would be wonderful if you could share knowledge on MLC NAND. I'm very interested in supporting MLC in UBI (and UBIFS). UBI was designed with SLC NAND in mind, now with MLC some things have changed. Thanks, //richard > Jeff Lauruhn > NAND Application Engineer > Embedded Business Unit > Micron Technology, Inc > > > -----Original Message----- > From: linux-mtd [mailto:linux-mtd-bounces@lists.infradead.org] On > Behalf Of Andrea Scian > Sent: Wednesday, March 11, 2015 10:01 AM > To: Richard Weinberger; dedekind1@gmail.com > Cc: mtd_mailinglist > Subject: Re: RFC: detect and manage power cut on MLC NAND > > > Hi all, > > and thanks for you feedback. > You can find my comments below. > > Il 11/03/2015 10:09, Richard Weinberger ha scritto: >> Am 11.03.2015 um 10:05 schrieb Artem Bityutskiy: >>> On Wed, 2015-03-11 at 09:57 +0100, Richard Weinberger wrote: >>>> Hi! >>>> >>>> Am 11.03.2015 um 08:20 schrieb Artem Bityutskiy: >>>>> On Tue, 2015-03-10 at 13:51 +0100, Richard Weinberger wrote: >>>>>>> WDYT about this? >>>>>>> If it sounds reasonable is there any suggestion where to place such a code? >>>>>> Customers often use DYI uninterruptible power supplies using capacitors. >>>>>> But managing a power cut is the least problem you have with MLC NAND. >>>>> Why is it the least problem, what is the hardest one? I thought >>>>> this one is the hardest. >>>> IMHO the hardest ones are the problems we don't know yet as NAND >>>> vendors are not really chatty about the MLC constraints. >>>> We don't know much about data retention for example. At least we >>>> have not much hard facts. Most of our knowledge is hearsay. >>> Well, but from the problems we know paired pages seems to be the >>> biggest one. E.g., what do we do if VID header gets corrupted >>> because of an interrupted write to the page paired with the VID header page? >>> Sounds like a hard problem to me. >> It is a hard problem. But at least we know about it. >> > > IIUC can I summarize like the following > > 1) power cut is one of the big problem we know about MLC NAND flash > with UBI > 2) we are unaware of other big issue with MLC NAND > > Unfortunately I don't really know the MTD/UBI/UBIFS internals but (correct me if I'm wrong) to me solving the power cut issue only by software (by using proper data structure, redundancy, NAND flash operation jornal and so on) is pretty hard to implement inside the Linux MTD stack. > > On the other hand, stuff like scrubbing (e.g. having a task like GC that periodically scrub the required NAND erase block), paired page (e.g. > carefully selecting LEB to PEB mapping to avoid reading/writing data into paired page in a wrong way) are still to be implemented but have, somehow, less impact on the whole MTD stack. > > For sure I'm missing some other MLC NAND issue, but the above are the one that I'm aware of. > > WDYT? > > Best Regards, > ______________________________________________________ Linux MTD discussion mailing list http://lists.infradead.org/mailman/listinfo/linux-mtd/ ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: RFC: detect and manage power cut on MLC NAND 2015-03-11 21:16 ` Jeff Lauruhn (jlauruhn) @ 2015-03-12 10:28 ` Richard Weinberger 2015-03-12 22:57 ` Jeff Lauruhn (jlauruhn) 0 siblings, 1 reply; 59+ messages in thread From: Richard Weinberger @ 2015-03-12 10:28 UTC (permalink / raw) To: Jeff Lauruhn (jlauruhn) Cc: Andrea Scian, Boris Brezillon, mtd_mailinglist, dedekind1 Am 11.03.2015 um 22:16 schrieb Jeff Lauruhn (jlauruhn): > Glad to help out. I train FAE's and customers on many aspects of NAND including MLC. UBI (and UBIFS) was designed with SLC NAND in mind, so far we know that we have to address the following constraints when we want UBI on MLC NAND: 1. Avoid repeating bit patterns. This can be solved by scrambling. Boris did some great work in this area. 2. Paired pages. We'll have choose pages we write to very carefully to not loss already written data in case of a power cut. 3. Read disturb. Happens also on SLC but not that early. I'm working in this. 4. Data retention. i.e, blocks that have not been erased for a long time have to be re-erased. I'm working in this too. 5. Unstable bits (not MLC specific). 6. What did I miss? Jeff, what do you think? Can you point us to some hard facts? I'm specially interested in numbers on read disturb and data retention. Thanks, //richard ^ permalink raw reply [flat|nested] 59+ messages in thread
* RE: RFC: detect and manage power cut on MLC NAND 2015-03-12 10:28 ` Richard Weinberger @ 2015-03-12 22:57 ` Jeff Lauruhn (jlauruhn) 2015-03-13 20:31 ` Boris Brezillon 2015-03-14 10:03 ` Richard Weinberger 0 siblings, 2 replies; 59+ messages in thread From: Jeff Lauruhn (jlauruhn) @ 2015-03-12 22:57 UTC (permalink / raw) To: Richard Weinberger Cc: Andrea Scian, Boris Brezillon, mtd_mailinglist, dedekind1 Jeff Lauruhn NAND Application Engineer Embedded Business Unit Micron Technology, Inc -----Original Message----- From: Richard Weinberger [mailto:richard@nod.at] Sent: Thursday, March 12, 2015 3:28 AM To: Jeff Lauruhn (jlauruhn) Cc: Andrea Scian; dedekind1@gmail.com; Boris Brezillon; mtd_mailinglist Subject: Re: RFC: detect and manage power cut on MLC NAND Am 11.03.2015 um 22:16 schrieb Jeff Lauruhn (jlauruhn): > Glad to help out. I train FAE's and customers on many aspects of NAND including MLC. UBI (and UBIFS) was designed with SLC NAND in mind, so far we know that we have to address the following constraints when we want UBI on MLC NAND: 1. Avoid repeating bit patterns. This can be solved by scrambling. Boris did some great work in this area. 2. Paired pages. We'll have choose pages we write to very carefully to not loss already written data in case of a power cut. For MLC we store 4 bits in the same cell has 3. Read disturb. Happens also on SLC but not that early. I'm working in this. 4. Data retention. i.e, blocks that have not been erased for a long time have to be re-erased. I'm working in this too. 5. Unstable bits (not MLC specific). Two types. Data retention and Disturbs (read and program). Data retention (charge loss) tends to shift left, 6. What did I miss? Jeff, what do you think? Can you point us to some hard facts? I'm specially interested in numbers on read disturb and data retention. I wish there were numbers, it would make my job a lot easier, but NAND doesn't work that way. Data retention is dependent on process node (35nm, 25nm, 20nm 15nm for example) P/E cycles and temperature. We generally specify our NAND using JEDC standards, x numbers of years at 55C with 10% cycling. If you apply our recommended ECC, then you will be able to store data and recover it after x numbers of years. But temperature, P/E, process size and usage have major effects on data retention so we recommend actively managing your NAND. This is what you do and what I find so interesting about your group. I can speak in generalities for now, and when I get more specifics I can predict and recommend solutions. Data Retention is characterized by a loss of charge. We program a bit from 1 to 0 (just the opposite of what people think). Over time the charge will leave the gate, this is normal NAND behavior. I say that the distribution of charged cell shifts left toward uncharged. Why is SLC better than MLC? First SLC was first and used older larger lithographies. You could store 10's of thousands of electrons on a 40nm gate and you only had two states L0 erased (0volts) or L1 programmed (1.5v for example). If you lose a few, it didn't make much difference and there was a lot of room between 0 and 1.5 volts. Newer processes are 20nm and MLC. With a smaller gate there are just a few hundred electrons and they need to be disturbed in one of 4 levels L0 (0 volts), L1(.5 volts), L2(1volts) or L3 (1.5 volts). Now adding or losing a few electrons can have a larger effect. We determine a programmed cell by measuring between the L0 and L1. If the levels have shifted, they can shift to the point where you can't reliably tell the difference between L0 and L1. When there are more levels and they are closer together it makes it just that much more challenging. Program/Read Disturb are characterized by charge gain or a shift right. The problem is the same. These affects are not instantaneous, they happen over long periods of time. Instead of trying to predict every case, we recommend actively managing NAND, and that's what your team does. Read the data an use ECC to see how much the data has changed. Create a threshold and when the data hits the threshold move the data to a fresh block. Thanks, //richard ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: RFC: detect and manage power cut on MLC NAND 2015-03-12 22:57 ` Jeff Lauruhn (jlauruhn) @ 2015-03-13 20:31 ` Boris Brezillon 2015-03-13 23:51 ` Jeff Lauruhn (jlauruhn) 2015-03-16 9:01 ` Ricard Wanderlof 2015-03-14 10:03 ` Richard Weinberger 1 sibling, 2 replies; 59+ messages in thread From: Boris Brezillon @ 2015-03-13 20:31 UTC (permalink / raw) To: Jeff Lauruhn (jlauruhn) Cc: Richard Weinberger, dedekind1, mtd_mailinglist, Andrea Scian Hello Jeff, I'm joining the discussion to ask more questions about MLC NANDs ;-). Could you tell us more about how block wear impact the voltage level stored in NAND cells. 1/ Are all pages in a block impacted the same way ? 2/ Is wear more likely to induce voltage increase, voltage decrease or is it unpredictable ? 3/ Is it possible to have more than one working voltage threshold (read-retry mode): I did some testing on my Hynix chip (I know you work for Micron but that's the only MLC chip I have :-)), and I managed to get less bitflips by trying another read-retry mode even if the previous one was allowing me to successfully fix existing bitflips. 4/ Do you have any numbers/statistics that could help us choose the more appropriate read-retry mode according to the number of P/E cycles ? 5/ Any other things you'd like to share regarding read-retry ? Apart from that, we're currently trying to find the most appropriate way to deal with paired pages, and this sounds rather complicated. The current idea is to expose paired pages information up to the UBIFS layer, and let UBIFS decide when it should stop writing on pages paired with already written pages. Moreover, we have a few pages we need to protect (UBI metadata: EC and VID headers) in order to keep UBI/UBIFS consistent. Do you have anything to share on this topic (ideas, solutions we should consider, constraints we're not aware of, ...) Thanks for your valuable information. Best Regards, Boris -- Boris Brezillon, Free Electrons Embedded Linux and Kernel engineering http://free-electrons.com ^ permalink raw reply [flat|nested] 59+ messages in thread
* RE: RFC: detect and manage power cut on MLC NAND 2015-03-13 20:31 ` Boris Brezillon @ 2015-03-13 23:51 ` Jeff Lauruhn (jlauruhn) 2015-03-14 9:46 ` Andrea Marson - DAVE Embedded Systems 2015-03-14 10:32 ` Boris Brezillon 2015-03-16 9:01 ` Ricard Wanderlof 1 sibling, 2 replies; 59+ messages in thread From: Jeff Lauruhn (jlauruhn) @ 2015-03-13 23:51 UTC (permalink / raw) To: Boris Brezillon Cc: Andrea Scian, Richard Weinberger, mtd_mailinglist, dedekind1 Jeff Lauruhn -----Original Message----- From: linux-mtd [mailto:linux-mtd-bounces@lists.infradead.org] On Behalf Of Boris Brezillon Sent: Friday, March 13, 2015 1:32 PM To: Jeff Lauruhn (jlauruhn) Cc: Richard Weinberger; dedekind1@gmail.com; mtd_mailinglist; Andrea Scian Subject: Re: RFC: detect and manage power cut on MLC NAND Hello Jeff, I'm joining the discussion to ask more questions about MLC NANDs ;-). Could you tell us more about how block wear impact the voltage level stored in NAND cells. 1/ Are all pages in a block impacted the same way ? Yes, because of block erase, P/E cycles affect all the pages in a block. 2/ Is wear more likely to induce voltage increase, voltage decrease or is it unpredictable ? Wear is a very well known a NAND characteristic. During P/E cycling there is a potential for electrons to get permanently trapped in the oxide. The more P/E cycles the more electrons get trapped. Over many P/E cycles cells well get to a point where they look permanent programmed and can't be erased or programmed. As cells begin to fail, ECC can be used to recover the data. If too many bits fail in page the device will respond with a FAIL status after a P/E cycle. 3/ Is it possible to have more than one working voltage threshold (read-retry mode): I did some testing on my Hynix chip (I know you work for Micron but that's the only MLC chip I have :-)), and I managed to get less bitflips by trying another read-retry mode even if the previous one was allowing me to successfully fix existing bitflips. Read Retry is available on some newer products. RR was introduced to help maintain and improve data retention and P/E cycles as geometry shrinks and bit/cell increase. If the device supports RR, we have predefined RR Options, based on the most likely chance of success. Start with option 1 and step through the options until you get a successful read. The DS usually has pretty good information. 4/ Do you have any numbers/statistics that could help us choose the more appropriate read-retry mode according to the number of P/E cycles ? I don't have numbers or statistics, but I can tell you that the RR steps are generally defined based on known NAND behavior. Go to the Micron website and put in this PN MT29F128G08CBCCB and you will find good information on RR. 5/ Any other things you'd like to share regarding read-retry ? RR isn't available on all devices. From your prospective I would give them the option to use RR if it's available. Apart from that, we're currently trying to find the most appropriate way to deal with paired pages, and this sounds rather complicated. The current idea is to expose paired pages information up to the UBIFS layer, and let UBIFS decide when it should stop writing on pages paired with already written pages. Moreover, we have a few pages we need to protect (UBI metadata: EC and VID headers) in order to keep UBI/UBIFS consistent. Do you have anything to share on this topic (ideas, solutions we should consider, constraints we're not aware of, ...) This is one of the reasons I came to this site. I have a great deal of device knowledge and I need to know more about how end users use the device. Most designs today employ power loss detection and employ elegant shutdown to the NAND. In addition, we provide Write Protect, which provides an extra layer of protection against power loss. There is still a chance that if the power event happens during a program to a page, the previously programmed shared page can also be corrupted. It's not clear to me how to keep track of shared pages for every device out there. It's not like a parameter page that you can read. It's an interesting problem. Thanks for your valuable information. Best Regards, Boris -- Boris Brezillon, Free Electrons Embedded Linux and Kernel engineering http://free-electrons.com ______________________________________________________ Linux MTD discussion mailing list http://lists.infradead.org/mailman/listinfo/linux-mtd/ ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: RFC: detect and manage power cut on MLC NAND 2015-03-13 23:51 ` Jeff Lauruhn (jlauruhn) @ 2015-03-14 9:46 ` Andrea Marson - DAVE Embedded Systems 2015-03-16 16:02 ` Jeff Lauruhn (jlauruhn) 2015-03-14 10:32 ` Boris Brezillon 1 sibling, 1 reply; 59+ messages in thread From: Andrea Marson - DAVE Embedded Systems @ 2015-03-14 9:46 UTC (permalink / raw) To: Jeff Lauruhn (jlauruhn) Cc: Boris Brezillon, dedekind1, mtd_mailinglist, Richard Weinberger, Andrea Scian Hi Jeff, thank you for your availability. I would like to go back to your statement about Write Protect pin: > Power loss is actually very complex. The Write Protect (WP) pin was > added to NAND help lock the NAND when a power loss event is > detected. I have extensive information on NAND and would be happy to > discuss. IIUC WP must respect several constraints. For example it must not be transitioned while NAND is busy. What if it is set low while NAND is ready - that is it can accept new commands - but an erase operation is in progress? Does the erase operation complete anyway? Best regards, Andrea MARSON DAVE Embedded Systems www.dave.eu > > > > Jeff Lauruhn > > -----Original Message----- > From: linux-mtd [mailto:linux-mtd-bounces@lists.infradead.org] On Behalf Of Boris Brezillon > Sent: Friday, March 13, 2015 1:32 PM > To: Jeff Lauruhn (jlauruhn) > Cc: Richard Weinberger; dedekind1@gmail.com; mtd_mailinglist; Andrea Scian > Subject: Re: RFC: detect and manage power cut on MLC NAND > > Hello Jeff, > > I'm joining the discussion to ask more questions about MLC NANDs ;-). > > Could you tell us more about how block wear impact the voltage level stored in NAND cells. > > 1/ Are all pages in a block impacted the same way ? > Yes, because of block erase, P/E cycles affect all the pages in a block. > 2/ Is wear more likely to induce voltage increase, voltage decrease > or is it unpredictable ? Wear is a very well known a NAND characteristic. During P/E cycling there is a potential for electrons to get permanently trapped in the oxide. The more P/E cycles the more electrons get trapped. Over many P/E cycles cells well get to a point where they look permanent programmed and can't be erased or programmed. As cells begin to fail, ECC can be used to recover the data. If too many bits fail in page the device will respond with a FAIL status after a P/E cycle. > > 3/ Is it possible to have more than one working voltage threshold > (read-retry mode): I did some testing on my Hynix chip (I know you > work for Micron but that's the only MLC chip I have :-)), and I > managed to get less bitflips by trying another read-retry mode even > if the previous one was allowing me to successfully fix existing > bitflips. > Read Retry is available on some newer products. RR was introduced to help maintain and improve data retention and P/E cycles as geometry shrinks and bit/cell increase. If the device supports RR, we have predefined RR Options, based on the most likely chance of success. Start with option 1 and step through the options until you get a successful read. The DS usually has pretty good information. > > 4/ Do you have any numbers/statistics that could > help us choose the more appropriate read-retry mode according to the > number of P/E cycles ? I don't have numbers or statistics, but I can tell you that the RR steps are generally defined based on known NAND behavior. Go to the Micron website and put in this PN MT29F128G08CBCCB and you will find good information on RR. > > 5/ Any other things you'd like to share regarding read-retry ? > RR isn't available on all devices. From your prospective I would give them the option to use RR if it's available. > > Apart from that, we're currently trying to find the most appropriate way to deal with paired pages, and this sounds rather complicated. > The current idea is to expose paired pages information up to the UBIFS layer, and let UBIFS decide when it should stop writing on pages paired with already written pages. > Moreover, we have a few pages we need to protect (UBI metadata: EC and VID headers) in order to keep UBI/UBIFS consistent. > Do you have anything to share on this topic (ideas, solutions we should consider, constraints we're not aware of, ...) > > This is one of the reasons I came to this site. I have a great deal of device knowledge and I need to know more about how end users use the device. > > Most designs today employ power loss detection and employ elegant shutdown to the NAND. In addition, we provide Write Protect, which provides an extra layer of protection against power loss. There is still a chance that if the power event happens during a program to a page, the previously programmed shared page can also be corrupted. It's not clear to me how to keep track of shared pages for every device out there. It's not like a parameter page that you can read. It's an interesting problem. > > Thanks for your valuable information. > > Best Regards, > > Boris > > -- > Boris Brezillon, Free Electrons > Embedded Linux and Kernel engineering > http://free-electrons.com > > ______________________________________________________ > Linux MTD discussion mailing list > http://lists.infradead.org/mailman/listinfo/linux-mtd/ > > ______________________________________________________ > Linux MTD discussion mailing list > http://lists.infradead.org/mailman/listinfo/linux-mtd/ > ^ permalink raw reply [flat|nested] 59+ messages in thread
* RE: RFC: detect and manage power cut on MLC NAND 2015-03-14 9:46 ` Andrea Marson - DAVE Embedded Systems @ 2015-03-16 16:02 ` Jeff Lauruhn (jlauruhn) 2015-03-17 8:00 ` Andrea Scian 0 siblings, 1 reply; 59+ messages in thread From: Jeff Lauruhn (jlauruhn) @ 2015-03-16 16:02 UTC (permalink / raw) To: Andrea Marson - DAVE Embedded Systems Cc: Boris Brezillon, dedekind1, mtd_mailinglist, Richard Weinberger, Andrea Scian "What if it is set low while NAND is ready - that is it can accept new commands - but an erase operation is in progress?". Again, the likelihood is low, in this case a block that was intended to be erased my not be fully erase, it can be fully erased when power returns. No data will lost because the block was intended to be erased anyway. "Does the erase operation complete anyway?" There has been a lot of work done to mitigate power loss on NAND, but I haven't ever seen a design from any NAND vendor that is 100%. Surprise power loss should be avoided on NAND or consider power detection and elegant shutdown circuitry. Jeff Lauruhn NAND Application Engineer Embedded Business Unit -----Original Message----- From: Andrea Marson - DAVE Embedded Systems [mailto:andrea.marson@dave.eu] Sent: Saturday, March 14, 2015 2:46 AM To: Jeff Lauruhn (jlauruhn) Cc: Boris Brezillon; Andrea Scian; Richard Weinberger; mtd_mailinglist; dedekind1@gmail.com Subject: Re: RFC: detect and manage power cut on MLC NAND Hi Jeff, thank you for your availability. I would like to go back to your statement about Write Protect pin: > Power loss is actually very complex. The Write Protect (WP) pin was > added to NAND help lock the NAND when a power loss event is > detected. I have extensive information on NAND and would be happy to > discuss. IIUC WP must respect several constraints. For example it must not be transitioned while NAND is busy. What if it is set low while NAND is ready - that is it can accept new commands - but an erase operation is in progress? Does the erase operation complete anyway? Best regards, Andrea MARSON DAVE Embedded Systems www.dave.eu > > > > Jeff Lauruhn > > -----Original Message----- > From: linux-mtd [mailto:linux-mtd-bounces@lists.infradead.org] On > Behalf Of Boris Brezillon > Sent: Friday, March 13, 2015 1:32 PM > To: Jeff Lauruhn (jlauruhn) > Cc: Richard Weinberger; dedekind1@gmail.com; mtd_mailinglist; Andrea > Scian > Subject: Re: RFC: detect and manage power cut on MLC NAND > > Hello Jeff, > > I'm joining the discussion to ask more questions about MLC NANDs ;-). > > Could you tell us more about how block wear impact the voltage level stored in NAND cells. > > 1/ Are all pages in a block impacted the same way ? > Yes, because of block erase, P/E cycles affect all the pages in a block. > 2/ Is wear more likely to induce voltage increase, voltage decrease > or is it unpredictable ? Wear is a very well known a NAND characteristic. During P/E cycling there is a potential for electrons to get permanently trapped in the oxide. The more P/E cycles the more electrons get trapped. Over many P/E cycles cells well get to a point where they look permanent programmed and can't be erased or programmed. As cells begin to fail, ECC can be used to recover the data. If too many bits fail in page the device will respond with a FAIL status after a P/E cycle. > > 3/ Is it possible to have more than one working voltage threshold > (read-retry mode): I did some testing on my Hynix chip (I know you > work for Micron but that's the only MLC chip I have :-)), and I > managed to get less bitflips by trying another read-retry mode even > if the previous one was allowing me to successfully fix existing > bitflips. > Read Retry is available on some newer products. RR was introduced to help maintain and improve data retention and P/E cycles as geometry shrinks and bit/cell increase. If the device supports RR, we have predefined RR Options, based on the most likely chance of success. Start with option 1 and step through the options until you get a successful read. The DS usually has pretty good information. > > 4/ Do you have any numbers/statistics that could > help us choose the more appropriate read-retry mode according to the > number of P/E cycles ? I don't have numbers or statistics, but I can tell you that the RR steps are generally defined based on known NAND behavior. Go to the Micron website and put in this PN MT29F128G08CBCCB and you will find good information on RR. > > 5/ Any other things you'd like to share regarding read-retry ? > RR isn't available on all devices. From your prospective I would give them the option to use RR if it's available. > > Apart from that, we're currently trying to find the most appropriate way to deal with paired pages, and this sounds rather complicated. > The current idea is to expose paired pages information up to the UBIFS layer, and let UBIFS decide when it should stop writing on pages paired with already written pages. > Moreover, we have a few pages we need to protect (UBI metadata: EC and VID headers) in order to keep UBI/UBIFS consistent. > Do you have anything to share on this topic (ideas, solutions we > should consider, constraints we're not aware of, ...) > > This is one of the reasons I came to this site. I have a great deal of device knowledge and I need to know more about how end users use the device. > > Most designs today employ power loss detection and employ elegant shutdown to the NAND. In addition, we provide Write Protect, which provides an extra layer of protection against power loss. There is still a chance that if the power event happens during a program to a page, the previously programmed shared page can also be corrupted. It's not clear to me how to keep track of shared pages for every device out there. It's not like a parameter page that you can read. It's an interesting problem. > > Thanks for your valuable information. > > Best Regards, > > Boris > > -- > Boris Brezillon, Free Electrons > Embedded Linux and Kernel engineering > http://free-electrons.com > > ______________________________________________________ > Linux MTD discussion mailing list > http://lists.infradead.org/mailman/listinfo/linux-mtd/ > > ______________________________________________________ > Linux MTD discussion mailing list > http://lists.infradead.org/mailman/listinfo/linux-mtd/ > ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: RFC: detect and manage power cut on MLC NAND 2015-03-16 16:02 ` Jeff Lauruhn (jlauruhn) @ 2015-03-17 8:00 ` Andrea Scian 0 siblings, 0 replies; 59+ messages in thread From: Andrea Scian @ 2015-03-17 8:00 UTC (permalink / raw) To: Jeff Lauruhn (jlauruhn), Andrea Marson - DAVE Embedded Systems Cc: Boris Brezillon, dedekind1, mtd_mailinglist, Richard Weinberger Dear Jeff, Il 16/03/2015 17:02, Jeff Lauruhn (jlauruhn) ha scritto: > "What if it is set low while NAND is ready - that is it can accept new commands - but an erase operation is in progress?". Again, the likelihood is low, in this case a block that was intended to be erased my not be fully erase, it can be fully erased when power returns. No data will lost because the block was intended to be erased anyway. AFAIK the real problem here is how to detect that the in-progress erase has not been terminated completely The block seems to be erased (it contains all 0xFF) but it can lead to some failure when writing or reading (I have some indirect experience in it, when doing some power-cut test on MLC but I cannot say for sure that it was caused by not completed erase operation. See also this reference http://www.linux-mtd.infradead.org/doc/ubifs.html#L_unstable_bits) > "Does the erase operation complete anyway?" There has been a lot of work done to mitigate power loss on NAND, but I haven't ever seen a design from any NAND vendor that is 100%. Surprise power loss should be avoided on NAND or consider power detection and elegant shutdown circuitry. IIUC we have two workaround here: - power detection and clean shutdown. This implies, of course, some hardware implementation and a piece of software able to intercept the hardware event, block all the NAND operation apart the one that is currently running. This is the topic of my initial RFC. I got a first implementation but, IMHO, is too strictly related to the NAND controller (I've placed the event handler inside it and just lock the NAND access by hacking chip->select_chip()) I think that power fail detection can be useful in some other context too inside Linux kernel, so probably it may have a more general implementation. However (and unfortunately) I'm not so involved in mainline kernel so I don't really know if this topic has been discussed or not - detection of interrupted operations: having some kind of journal that record the last running operation and fix it on next reboot. AFAIK this is what commercial FTL and flash memory controller do, using a lot of patented piece of software. I'm not so aware of UBI/MTD internals but I think this is hard to implement using a general approach. It should be easier to implement such a logic having some additional hardware support (the first that come into my mind is battery backed SRAM, e.g. the one that we find inside RTC devices) Best Regards, -- Andrea SCIAN DAVE Embedded Systems ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: RFC: detect and manage power cut on MLC NAND 2015-03-13 23:51 ` Jeff Lauruhn (jlauruhn) 2015-03-14 9:46 ` Andrea Marson - DAVE Embedded Systems @ 2015-03-14 10:32 ` Boris Brezillon 2015-03-16 21:11 ` Jeff Lauruhn (jlauruhn) 1 sibling, 1 reply; 59+ messages in thread From: Boris Brezillon @ 2015-03-14 10:32 UTC (permalink / raw) To: Jeff Lauruhn (jlauruhn) Cc: Andrea Scian, Richard Weinberger, mtd_mailinglist, dedekind1 Hi Jeff, On Fri, 13 Mar 2015 23:51:53 +0000 "Jeff Lauruhn (jlauruhn)" <jlauruhn@micron.com> wrote: > > Hello Jeff, > > I'm joining the discussion to ask more questions about MLC NANDs ;-). > > Could you tell us more about how block wear impact the voltage level stored in NAND cells. > > 1/ Are all pages in a block impacted the same way ? > Yes, because of block erase, P/E cycles affect all the pages in a block. Okay, that's what I thought. > 2/ Is wear more likely to induce voltage increase, voltage decrease > or is it unpredictable ? Wear is a very well known a NAND characteristic. During P/E cycling there is a potential for electrons to get permanently trapped in the oxide. The more P/E cycles the more electrons get trapped. Over many P/E cycles cells well get to a point where they look permanent programmed and can't be erased or programmed. As cells begin to fail, ECC can be used to recover the data. If too many bits fail in page the device will respond with a FAIL status after a P/E cycle. So voltage thresholds tends to increase with wear, right ? > > 3/ Is it possible to have more than one working voltage threshold > (read-retry mode): I did some testing on my Hynix chip (I know you > work for Micron but that's the only MLC chip I have :-)), and I > managed to get less bitflips by trying another read-retry mode even > if the previous one was allowing me to successfully fix existing > bitflips. > Read Retry is available on some newer products. RR was introduced to help maintain and improve data retention and P/E cycles as geometry shrinks and bit/cell increase. If the device supports RR, we have predefined RR Options, based on the most likely chance of success. Start with option 1 and step through the options until you get a successful read. The DS usually has pretty good information. When you say you have "predefined RR Options, based on the most likely chance of success", does this mean these options are internally evolving during the NAND block lifetime, or is RR mode 0 always encoding the same threshold config. In the latter case, maybe we should start with a different RR mode depending on the number of P/E cycles already done on the block, so that we have more chance to successfully read the page on our first read. > > 4/ Do you have any numbers/statistics that could > help us choose the more appropriate read-retry mode according to the > number of P/E cycles ? I don't have numbers or statistics, but I can tell you that the RR steps are generally defined based on known NAND behavior. Go to the Micron website and put in this PN MT29F128G08CBCCB and you will find good information on RR. Okay, I'll have a look at the datasheet you pointed out (the Hynix one was not even talking about read-retry, I had to search in Allwinner code to understand how to change read-retry mode). > > 5/ Any other things you'd like to share regarding read-retry ? > RR isn't available on all devices. From your prospective I would give them the option to use RR if it's available. Yes, that's already done this way: we use RR on devices providing this feature. IIRC, only Micron chips are supported so far, but I added support for one of the Hynix chip. The whole problem here is that each vendor implement RR in their own way (using ONFI params for Micron, OTP area and private commands for Hynix, and probably something else for Samsung chips). Anyway, that's just a matter of adding a NAND chip database + vendor specific code to deal with each read retry implementation (even if that would have helped us a lot if chip vendors had agreed on a standard way to control RR). > > Apart from that, we're currently trying to find the most appropriate way to deal with paired pages, and this sounds rather complicated. > The current idea is to expose paired pages information up to the UBIFS layer, and let UBIFS decide when it should stop writing on pages paired with already written pages. > Moreover, we have a few pages we need to protect (UBI metadata: EC and VID headers) in order to keep UBI/UBIFS consistent. > Do you have anything to share on this topic (ideas, solutions we should consider, constraints we're not aware of, ...) > > This is one of the reasons I came to this site. I have a great deal of device knowledge and I need to know more about how end users use the device. > > Most designs today employ power loss detection and employ elegant shutdown to the NAND. In addition, we provide Write Protect, which provides an extra layer of protection against power loss. There is still a chance that if the power event happens during a program to a page, the previously programmed shared page can also be corrupted. It's not clear to me how to keep track of shared pages for every device out there. It's not like a parameter page that you can read. It's an interesting problem. Of course, preventing page corruption is a good approach, but some board designers are just simply not taking these constraints into account, and detecting power loss in order to assert the WP pin is not possible in such designs. I think we should also find a solution to recover from corruptions induced by paired pages write, and that's the direction we're currently investigating. But if someone have real examples (boards) supporting power loss detection + WP pin control in such cases, maybe we can start thinking about a standard way to deal with that in Linux. Thanks again for your answers. Best Regards, Boris -- Boris Brezillon, Free Electrons Embedded Linux and Kernel engineering http://free-electrons.com ^ permalink raw reply [flat|nested] 59+ messages in thread
* RE: RFC: detect and manage power cut on MLC NAND 2015-03-14 10:32 ` Boris Brezillon @ 2015-03-16 21:11 ` Jeff Lauruhn (jlauruhn) 2015-03-17 9:30 ` Andrea Scian 0 siblings, 1 reply; 59+ messages in thread From: Jeff Lauruhn (jlauruhn) @ 2015-03-16 21:11 UTC (permalink / raw) To: Boris Brezillon Cc: Andrea Scian, Richard Weinberger, mtd_mailinglist, dedekind1 Good morning Boris; RR is a new feature and not available on all parts few. I'm not sure about others, but since these are features, you simply enable of disable via SET FEATURE/GET FEATURE. If you already provide that SET/GET FEATURE functionality then an end-user determine if their device supports a feature and then write the code to enable when they need it on their particular design. Jeff Lauruhn NAND Application Engineer Embedded Business Unit -----Original Message----- From: Boris Brezillon [mailto:boris.brezillon@free-electrons.com] Sent: Saturday, March 14, 2015 3:32 AM To: Jeff Lauruhn (jlauruhn) Cc: Richard Weinberger; dedekind1@gmail.com; mtd_mailinglist; Andrea Scian Subject: Re: RFC: detect and manage power cut on MLC NAND Hi Jeff, On Fri, 13 Mar 2015 23:51:53 +0000 "Jeff Lauruhn (jlauruhn)" <jlauruhn@micron.com> wrote: > > Hello Jeff, > > I'm joining the discussion to ask more questions about MLC NANDs ;-). > > Could you tell us more about how block wear impact the voltage level stored in NAND cells. > > 1/ Are all pages in a block impacted the same way ? > Yes, because of block erase, P/E cycles affect all the pages in a block. Okay, that's what I thought. > 2/ Is wear more likely to induce voltage increase, voltage decrease > or is it unpredictable ? Wear is a very well known a NAND characteristic. During P/E cycling there is a potential for electrons to get permanently trapped in the oxide. The more P/E cycles the more electrons get trapped. Over many P/E cycles cells well get to a point where they look permanent programmed and can't be erased or programmed. As cells begin to fail, ECC can be used to recover the data. If too many bits fail in page the device will respond with a FAIL status after a P/E cycle. So voltage thresholds tends to increase with wear, right ? > > 3/ Is it possible to have more than one working voltage threshold > (read-retry mode): I did some testing on my Hynix chip (I know you > work for Micron but that's the only MLC chip I have :-)), and I > managed to get less bitflips by trying another read-retry mode even > if the previous one was allowing me to successfully fix existing > bitflips. > Read Retry is available on some newer products. RR was introduced to help maintain and improve data retention and P/E cycles as geometry shrinks and bit/cell increase. If the device supports RR, we have predefined RR Options, based on the most likely chance of success. Start with option 1 and step through the options until you get a successful read. The DS usually has pretty good information. When you say you have "predefined RR Options, based on the most likely chance of success", does this mean these options are internally evolving during the NAND block lifetime, or is RR mode 0 always encoding the same threshold config. In the latter case, maybe we should start with a different RR mode depending on the number of P/E cycles already done on the block, so that we have more chance to successfully read the page on our first read. > > 4/ Do you have any numbers/statistics that could > help us choose the more appropriate read-retry mode according to the > number of P/E cycles ? I don't have numbers or statistics, but I can tell you that the RR steps are generally defined based on known NAND behavior. Go to the Micron website and put in this PN MT29F128G08CBCCB and you will find good information on RR. Okay, I'll have a look at the datasheet you pointed out (the Hynix one was not even talking about read-retry, I had to search in Allwinner code to understand how to change read-retry mode). > > 5/ Any other things you'd like to share regarding read-retry ? > RR isn't available on all devices. From your prospective I would give them the option to use RR if it's available. Yes, that's already done this way: we use RR on devices providing this feature. IIRC, only Micron chips are supported so far, but I added support for one of the Hynix chip. The whole problem here is that each vendor implement RR in their own way (using ONFI params for Micron, OTP area and private commands for Hynix, and probably something else for Samsung chips). Anyway, that's just a matter of adding a NAND chip database + vendor specific code to deal with each read retry implementation (even if that would have helped us a lot if chip vendors had agreed on a standard way to control RR). > > Apart from that, we're currently trying to find the most appropriate way to deal with paired pages, and this sounds rather complicated. > The current idea is to expose paired pages information up to the UBIFS layer, and let UBIFS decide when it should stop writing on pages paired with already written pages. > Moreover, we have a few pages we need to protect (UBI metadata: EC and VID headers) in order to keep UBI/UBIFS consistent. > Do you have anything to share on this topic (ideas, solutions we > should consider, constraints we're not aware of, ...) > > This is one of the reasons I came to this site. I have a great deal of device knowledge and I need to know more about how end users use the device. > > Most designs today employ power loss detection and employ elegant shutdown to the NAND. In addition, we provide Write Protect, which provides an extra layer of protection against power loss. There is still a chance that if the power event happens during a program to a page, the previously programmed shared page can also be corrupted. It's not clear to me how to keep track of shared pages for every device out there. It's not like a parameter page that you can read. It's an interesting problem. Of course, preventing page corruption is a good approach, but some board designers are just simply not taking these constraints into account, and detecting power loss in order to assert the WP pin is not possible in such designs. I think we should also find a solution to recover from corruptions induced by paired pages write, and that's the direction we're currently investigating. But if someone have real examples (boards) supporting power loss detection + WP pin control in such cases, maybe we can start thinking about a standard way to deal with that in Linux. Thanks again for your answers. Best Regards, Boris -- Boris Brezillon, Free Electrons Embedded Linux and Kernel engineering http://free-electrons.com ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: RFC: detect and manage power cut on MLC NAND 2015-03-16 21:11 ` Jeff Lauruhn (jlauruhn) @ 2015-03-17 9:30 ` Andrea Scian 2015-03-17 10:02 ` Boris Brezillon 2015-03-17 17:04 ` Jeff Lauruhn (jlauruhn) 0 siblings, 2 replies; 59+ messages in thread From: Andrea Scian @ 2015-03-17 9:30 UTC (permalink / raw) To: Jeff Lauruhn (jlauruhn) Cc: Boris Brezillon, dedekind1, mtd_mailinglist, Richard Weinberger Dear Jeff, Il 16/03/2015 22:11, Jeff Lauruhn (jlauruhn) ha scritto: > Good morning Boris; > RR is a new feature and not available on all parts few. I'm not sure > about others, but since these are features, you simply enable of > disable via SET FEATURE/GET FEATURE. If you already provide that > SET/GET FEATURE functionality then an end-user determine if their > device supports a feature and then write the code to enable when they > need it on their particular design. I can confirm this. In fact I'm currently working with two Micron NAND: MT29F32G08CBACAWP MT29F32G08CBADAWP The latter should be "just" a newer die revision of the former (at least, this is what our distributor says) There's a technology change between the two and, in fact, the latter supports RR while there's no mention of such a feature inside rev C. Jeff, could you please help me in understanding which if the following sentences are true and which are false? - rev D is more "robust" than rev C because it has RR (so an additional feature that improve error correction) - rev D is "robust" like rev C, if rev D is used with RR - if RR is not used rev D is more error prone than rev C I think this is crucial to understand how RR works and how much is needed inside MTD/UBI code. I hope that the above information are not under NDA ;-) Thanks in advance, -- Andrea SCIAN DAVE Embedded Systems ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: RFC: detect and manage power cut on MLC NAND 2015-03-17 9:30 ` Andrea Scian @ 2015-03-17 10:02 ` Boris Brezillon 2015-03-17 16:42 ` Jeff Lauruhn (jlauruhn) 2015-03-17 17:04 ` Jeff Lauruhn (jlauruhn) 1 sibling, 1 reply; 59+ messages in thread From: Boris Brezillon @ 2015-03-17 10:02 UTC (permalink / raw) To: Andrea Scian Cc: Richard Weinberger, Jeff Lauruhn (jlauruhn), mtd_mailinglist, dedekind1 Hi Andrea, I'll let Jeff answer this question, but I'd like to share my understanding. On Tue, 17 Mar 2015 10:30:30 +0100 Andrea Scian <rnd4@dave-tech.it> wrote: > > > Dear Jeff, > > Il 16/03/2015 22:11, Jeff Lauruhn (jlauruhn) ha scritto: > > Good morning Boris; > > RR is a new feature and not available on all parts few. I'm not sure > > about others, but since these are features, you simply enable of > > disable via SET FEATURE/GET FEATURE. If you already provide that > > SET/GET FEATURE functionality then an end-user determine if their > > device supports a feature and then write the code to enable when they > > need it on their particular design. > > I can confirm this. In fact I'm currently working with two Micron NAND: > > MT29F32G08CBACAWP > MT29F32G08CBADAWP > > The latter should be "just" a newer die revision of the former (at > least, this is what our distributor says) > > There's a technology change between the two and, in fact, the latter > supports RR while there's no mention of such a feature inside rev C. > > Jeff, could you please help me in understanding which if the following > sentences are true and which are false? > - rev D is more "robust" than rev C because it has RR (so an additional > feature that improve error correction) > - rev D is "robust" like rev C, if rev D is used with RR > - if RR is not used rev D is more error prone than rev C RR shouldn't change NAND robustness (or sensitivity to read/write disturbance generating bitflips). AFAIU RR will help you improve your NAND lifetime, because you're allowed to change voltage thresholds which means you can fix errors that were previously considered as unfixable and lead to blocks being marked bad earlier. I'll let Jeff correct me if I'm wrong ;-). > > I think this is crucial to understand how RR works and how much is > needed inside MTD/UBI code. Hopefully this can all be handled in the MTD layer, with some help from the UBI layer to feed the wear information (number of P/E cycles on each block). Best Regards, Boris -- Boris Brezillon, Free Electrons Embedded Linux and Kernel engineering http://free-electrons.com ^ permalink raw reply [flat|nested] 59+ messages in thread
* RE: RFC: detect and manage power cut on MLC NAND 2015-03-17 10:02 ` Boris Brezillon @ 2015-03-17 16:42 ` Jeff Lauruhn (jlauruhn) 2015-03-18 8:45 ` RFC: detect and manage power cut on MLC NAND (linux-mtd Digest, Vol 144, Issue 70) Andrea Marson 0 siblings, 1 reply; 59+ messages in thread From: Jeff Lauruhn (jlauruhn) @ 2015-03-17 16:42 UTC (permalink / raw) To: Boris Brezillon, Andrea Scian Cc: Richard Weinberger, mtd_mailinglist, dedekind1 Very nice explanation! Not sure if I could have done better myself. Jeff Lauruhn NAND Application Engineer Embedded Business Unit Micron Technology, Inc -----Original Message----- From: Boris Brezillon [mailto:boris.brezillon@free-electrons.com] Sent: Tuesday, March 17, 2015 3:02 AM To: Andrea Scian Cc: Jeff Lauruhn (jlauruhn); Richard Weinberger; dedekind1@gmail.com; mtd_mailinglist Subject: Re: RFC: detect and manage power cut on MLC NAND Hi Andrea, I'll let Jeff answer this question, but I'd like to share my understanding. On Tue, 17 Mar 2015 10:30:30 +0100 Andrea Scian <rnd4@dave-tech.it> wrote: > > > Dear Jeff, > > Il 16/03/2015 22:11, Jeff Lauruhn (jlauruhn) ha scritto: > > Good morning Boris; > > RR is a new feature and not available on all parts few. I'm not > > sure about others, but since these are features, you simply enable > > of disable via SET FEATURE/GET FEATURE. If you already provide that > > SET/GET FEATURE functionality then an end-user determine if their > > device supports a feature and then write the code to enable when > > they need it on their particular design. > > I can confirm this. In fact I'm currently working with two Micron NAND: > > MT29F32G08CBACAWP > MT29F32G08CBADAWP > > The latter should be "just" a newer die revision of the former (at > least, this is what our distributor says) > > There's a technology change between the two and, in fact, the latter > supports RR while there's no mention of such a feature inside rev C. > > Jeff, could you please help me in understanding which if the following > sentences are true and which are false? > - rev D is more "robust" than rev C because it has RR (so an > additional feature that improve error correction) > - rev D is "robust" like rev C, if rev D is used with RR > - if RR is not used rev D is more error prone than rev C RR shouldn't change NAND robustness (or sensitivity to read/write disturbance generating bitflips). AFAIU RR will help you improve your NAND lifetime, because you're allowed to change voltage thresholds which means you can fix errors that were previously considered as unfixable and lead to blocks being marked bad earlier. I'll let Jeff correct me if I'm wrong ;-). > > I think this is crucial to understand how RR works and how much is > needed inside MTD/UBI code. Hopefully this can all be handled in the MTD layer, with some help from the UBI layer to feed the wear information (number of P/E cycles on each block). Best Regards, Boris -- Boris Brezillon, Free Electrons Embedded Linux and Kernel engineering http://free-electrons.com ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: RFC: detect and manage power cut on MLC NAND (linux-mtd Digest, Vol 144, Issue 70) 2015-03-17 16:42 ` Jeff Lauruhn (jlauruhn) @ 2015-03-18 8:45 ` Andrea Marson 2015-03-18 9:07 ` Boris Brezillon 2015-03-18 16:12 ` Jeff Lauruhn (jlauruhn) 0 siblings, 2 replies; 59+ messages in thread From: Andrea Marson @ 2015-03-18 8:45 UTC (permalink / raw) To: Boris Brezillon, Jeff Lauruhn (jlauruhn) Cc: Andrea Scian, dedekind1, linux-mtd, Richard Weinberger Hello, I would like to discuss about another couple of topics: partitioning and SLC emulation. 1) IIUC read/program disturb effects exhibit at block level. In a typical embedded linux systems there are software parts - bootloader, kernel image etc. - that virtually are never changed (almost ...) but are read many times. Other parts - application libraries, log files etc. - are read and wrote many times instead. If these two kinds of software are stored in different MTD partitions - ket's say partition A for bootloader, kernel etc. and partition B for application libraries, log files etc. - can we say that read/write operations performed on partition B have no disturb effects on partition A? 2) IIUC Boris has worked on SLC emulation too. This seems to be a promising feature because it would allow to partition NAND flash and to create higher reliability partition (at the cost of halving the size). Is it possibile to implement such functionality in software stack only(MTD/UBI) or is it necessary that NAND memory supports specific features? Regards, Andrea Marson > Very nice explanation! Not sure if I could have done better myself. > > Jeff Lauruhn > NAND Application Engineer > Embedded Business Unit > Micron Technology, Inc > > > -----Original Message----- > From: Boris Brezillon [mailto:boris.brezillon@free-electrons.com] > Sent: Tuesday, March 17, 2015 3:02 AM > To: Andrea Scian > Cc: Jeff Lauruhn (jlauruhn); Richard Weinberger; dedekind1@gmail.com; mtd_mailinglist > Subject: Re: RFC: detect and manage power cut on MLC NAND > > Hi Andrea, > > I'll let Jeff answer this question, but I'd like to share my understanding. > > On Tue, 17 Mar 2015 10:30:30 +0100 > Andrea Scian <rnd4@dave-tech.it> wrote: > >> >> >> Dear Jeff, >> >> Il 16/03/2015 22:11, Jeff Lauruhn (jlauruhn) ha scritto: >>> Good morning Boris; >>> RR is a new feature and not available on all parts few. I'm not >>> sure about others, but since these are features, you simply enable >>> of disable via SET FEATURE/GET FEATURE. If you already provide that >>> SET/GET FEATURE functionality then an end-user determine if their >>> device supports a feature and then write the code to enable when >>> they need it on their particular design. >> >> I can confirm this. In fact I'm currently working with two Micron NAND: >> >> MT29F32G08CBACAWP >> MT29F32G08CBADAWP >> >> The latter should be "just" a newer die revision of the former (at >> least, this is what our distributor says) >> >> There's a technology change between the two and, in fact, the latter >> supports RR while there's no mention of such a feature inside rev C. >> >> Jeff, could you please help me in understanding which if the following >> sentences are true and which are false? >> - rev D is more "robust" than rev C because it has RR (so an >> additional feature that improve error correction) >> - rev D is "robust" like rev C, if rev D is used with RR >> - if RR is not used rev D is more error prone than rev C > > RR shouldn't change NAND robustness (or sensitivity to read/write disturbance generating bitflips). > > AFAIU RR will help you improve your NAND lifetime, because you're allowed to change voltage thresholds which means you can fix errors that were previously considered as unfixable and lead to blocks being marked bad earlier. > > I'll let Jeff correct me if I'm wrong ;-). > >> >> I think this is crucial to understand how RR works and how much is >> needed inside MTD/UBI code. > > Hopefully this can all be handled in the MTD layer, with some help from the UBI layer to feed the wear information (number of P/E cycles on each block). > > Best Regards, > > Boris > > -- > Boris Brezillon, Free Electrons > Embedded Linux and Kernel engineering > http://free-electrons.com > > > ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: RFC: detect and manage power cut on MLC NAND (linux-mtd Digest, Vol 144, Issue 70) 2015-03-18 8:45 ` RFC: detect and manage power cut on MLC NAND (linux-mtd Digest, Vol 144, Issue 70) Andrea Marson @ 2015-03-18 9:07 ` Boris Brezillon 2015-03-18 9:56 ` Andrea Marson 2015-03-18 12:07 ` Richard Weinberger 2015-03-18 16:12 ` Jeff Lauruhn (jlauruhn) 1 sibling, 2 replies; 59+ messages in thread From: Boris Brezillon @ 2015-03-18 9:07 UTC (permalink / raw) To: Andrea Marson Cc: Andrea Scian, Jeff Lauruhn (jlauruhn), linux-mtd, dedekind1, Richard Weinberger Hi Andrea, On Wed, 18 Mar 2015 09:45:18 +0100 Andrea Marson <andrea.marson@dave.eu> wrote: > Hello, > > I would like to discuss about another couple of topics: partitioning and > SLC emulation. > > 1) IIUC read/program disturb effects exhibit at block level. > In a typical embedded linux systems there are software parts - > bootloader, kernel image etc. - that virtually are never changed (almost > ...) but are read many times. Other parts - application libraries, log > files etc. - are read and wrote many times instead. > If these two kinds of software are stored in different MTD partitions - > ket's say partition A for bootloader, kernel etc. and partition B for > application libraries, log files etc. - can we say that read/write > operations performed on partition B have no disturb effects on partition A? AFAIK, read/write disturb effects only occur on pages of the same block, so we shouldn't see bitflips on partition A caused by read/write on partition B. > > 2) IIUC Boris has worked on SLC emulation too. No I didn't. What makes you think I did ? > This seems to be a > promising feature because it would allow to partition NAND flash and to > create higher reliability partition (at the cost of halving the size). > Is it possibile to implement such functionality in software stack > only(MTD/UBI) or is it necessary that NAND memory supports specific > features? I guess you're talking about the slides for my ELC talk. Note that I'm just talking about hypothetical changes, and no development has been started on this topic yet. IIUC, you want to select SLC/MLC mode on a per-partition basis, that should be feasible, and this would be particularly useful for bootloader partitions (where UBI is not involved). Best Regards, Boris -- Boris Brezillon, Free Electrons Embedded Linux and Kernel engineering http://free-electrons.com ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: RFC: detect and manage power cut on MLC NAND (linux-mtd Digest, Vol 144, Issue 70) 2015-03-18 9:07 ` Boris Brezillon @ 2015-03-18 9:56 ` Andrea Marson 2015-03-18 10:03 ` Boris Brezillon 2015-03-18 12:07 ` Richard Weinberger 1 sibling, 1 reply; 59+ messages in thread From: Andrea Marson @ 2015-03-18 9:56 UTC (permalink / raw) To: Boris Brezillon Cc: Andrea Scian, Jeff Lauruhn (jlauruhn), linux-mtd, dedekind1, Richard Weinberger >> 2) IIUC Boris has worked on SLC emulation too. > No I didn't. What makes you think I did ? My bad. I misunderstood a sentence in your slides for ELC talk :( > >> This seems to be a >> promising feature because it would allow to partition NAND flash and to >> create higher reliability partition (at the cost of halving the size). >> Is it possibile to implement such functionality in software stack >> only(MTD/UBI) or is it necessary that NAND memory supports specific >> features? > I guess you're talking about the slides for my ELC talk. > Note that I'm just talking about hypothetical changes, and no > development has been started on this topic yet. I see. > > IIUC, you want to select SLC/MLC mode on a per-partition basis, that > should be feasible, and this would be particularly useful for bootloader > partitions (where UBI is not involved). That's exactly what I was thinking about. However it is still not clear to me how this should be implemented. I have read a datasheet saying that "SLC mode" consists of halving block addresses, that is higher bit of block address lines must be set to low. I wonder if this strategy prevents paired page related issues. I think one should work at page addressing level instead, in order to use pages that are not paired only. Do you think it makes sense? Andrea ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: RFC: detect and manage power cut on MLC NAND (linux-mtd Digest, Vol 144, Issue 70) 2015-03-18 9:56 ` Andrea Marson @ 2015-03-18 10:03 ` Boris Brezillon 0 siblings, 0 replies; 59+ messages in thread From: Boris Brezillon @ 2015-03-18 10:03 UTC (permalink / raw) To: Andrea Marson Cc: Andrea Scian, Jeff Lauruhn (jlauruhn), linux-mtd, dedekind1, Richard Weinberger On Wed, 18 Mar 2015 10:56:15 +0100 Andrea Marson <andrea.marson@dave.eu> wrote: > > > > IIUC, you want to select SLC/MLC mode on a per-partition basis, that > > should be feasible, and this would be particularly useful for bootloader > > partitions (where UBI is not involved). > That's exactly what I was thinking about. > However it is still not clear to me how this should be implemented. I > have read a datasheet saying that "SLC mode" consists of halving block > addresses, that is higher bit of block address lines must be set to low. > I wonder if this strategy prevents paired page related issues. I think > one should work at page addressing level instead, in order to use pages > that are not paired only. Do you think it makes sense? Yes, actually implementing the SLC mode in software is quite easy, the only thing we need is a description of which pages are paired together in a block. Moreover, by doing that in SW we won't have to deal with each vendor implementation of "SLC mode". -- Boris Brezillon, Free Electrons Embedded Linux and Kernel engineering http://free-electrons.com ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: RFC: detect and manage power cut on MLC NAND (linux-mtd Digest, Vol 144, Issue 70) 2015-03-18 9:07 ` Boris Brezillon 2015-03-18 9:56 ` Andrea Marson @ 2015-03-18 12:07 ` Richard Weinberger 2015-03-18 17:11 ` Jeff Lauruhn (jlauruhn) 1 sibling, 1 reply; 59+ messages in thread From: Richard Weinberger @ 2015-03-18 12:07 UTC (permalink / raw) To: Boris Brezillon, Andrea Marson Cc: Andrea Scian, Jeff Lauruhn (jlauruhn), linux-mtd, dedekind1 Am 18.03.2015 um 10:06 schrieb Boris Brezillon: >> 1) IIUC read/program disturb effects exhibit at block level. >> In a typical embedded linux systems there are software parts - >> bootloader, kernel image etc. - that virtually are never changed (almost >> ...) but are read many times. Other parts - application libraries, log >> files etc. - are read and wrote many times instead. >> If these two kinds of software are stored in different MTD partitions - >> ket's say partition A for bootloader, kernel etc. and partition B for >> application libraries, log files etc. - can we say that read/write >> operations performed on partition B have no disturb effects on partition A? > > AFAIK, read/write disturb effects only occur on pages of the same > block, so we shouldn't see bitflips on partition A caused by read/write > on partition B. I already saw corruptions on nearby blocks but I'm not sure if really plain read disturb was the root cause as this NAND chips showed in general funny symptoms. Jeff, can you tell us more? Thanks, //richard ^ permalink raw reply [flat|nested] 59+ messages in thread
* RE: RFC: detect and manage power cut on MLC NAND (linux-mtd Digest, Vol 144, Issue 70) 2015-03-18 12:07 ` Richard Weinberger @ 2015-03-18 17:11 ` Jeff Lauruhn (jlauruhn) 0 siblings, 0 replies; 59+ messages in thread From: Jeff Lauruhn (jlauruhn) @ 2015-03-18 17:11 UTC (permalink / raw) To: Richard Weinberger, Boris Brezillon, Andrea Marson Cc: Andrea Scian, linux-mtd, dedekind1 Disturb is a block level affect, but in operation data is often layout in continuous blocks, so if one block is seeing a lot of reads or P/E cycles, there's a high probability the blocks around it are also seeing very similar read and P/E patterns. Jeff Lauruhn NAND Application Engineer Embedded Business Unit -----Original Message----- From: Richard Weinberger [mailto:richard@nod.at] Sent: Wednesday, March 18, 2015 5:07 AM To: Boris Brezillon; Andrea Marson Cc: Jeff Lauruhn (jlauruhn); linux-mtd@lists.infradead.org; Andrea Scian; dedekind1@gmail.com Subject: Re: RFC: detect and manage power cut on MLC NAND (linux-mtd Digest, Vol 144, Issue 70) Am 18.03.2015 um 10:06 schrieb Boris Brezillon: >> 1) IIUC read/program disturb effects exhibit at block level. >> In a typical embedded linux systems there are software parts - >> bootloader, kernel image etc. - that virtually are never changed >> (almost >> ...) but are read many times. Other parts - application libraries, >> log files etc. - are read and wrote many times instead. >> If these two kinds of software are stored in different MTD partitions >> - ket's say partition A for bootloader, kernel etc. and partition B >> for application libraries, log files etc. - can we say that >> read/write operations performed on partition B have no disturb effects on partition A? > > AFAIK, read/write disturb effects only occur on pages of the same > block, so we shouldn't see bitflips on partition A caused by > read/write on partition B. I already saw corruptions on nearby blocks but I'm not sure if really plain read disturb was the root cause as this NAND chips showed in general funny symptoms. Jeff, can you tell us more? Thanks, //richard ^ permalink raw reply [flat|nested] 59+ messages in thread
* RE: RFC: detect and manage power cut on MLC NAND (linux-mtd Digest, Vol 144, Issue 70) 2015-03-18 8:45 ` RFC: detect and manage power cut on MLC NAND (linux-mtd Digest, Vol 144, Issue 70) Andrea Marson 2015-03-18 9:07 ` Boris Brezillon @ 2015-03-18 16:12 ` Jeff Lauruhn (jlauruhn) 2015-03-19 8:47 ` RFC: detect and manage power cut on MLC NAND Andrea Marson 1 sibling, 1 reply; 59+ messages in thread From: Jeff Lauruhn (jlauruhn) @ 2015-03-18 16:12 UTC (permalink / raw) To: Andrea Marson, Boris Brezillon Cc: Andrea Scian, dedekind1, linux-mtd, Richard Weinberger Disturb is a block level affect, as long as partition A and B are in different blocks there will be no disturb between them. Disturbs, does not damage cells; ERASE returns cells to undisturbed levels. Disturbed bits are effectively managed with error correction codes (ECC). Officially I would say don't use SLC emulation, but technically I know what your doing. The reason I say no is because we have very precise recipes designed to create very tight distibutions, and although the first pass distributions might look like an SLC, they are really designed with the expectation of the upper page being programmed. Not a true SLC. With MLC lithography of 25 nm and less the difference between each level (L0, L1, L2 and L3) is just a few 10's of electrons. The distribution have to be very tight, in order to meet retention requirements. Jeff Lauruhn NAND Application Engineer Embedded Business Unit -----Original Message----- From: Andrea Marson [mailto:andrea.marson@dave.eu] Sent: Wednesday, March 18, 2015 1:45 AM To: Boris Brezillon; Jeff Lauruhn (jlauruhn) Cc: linux-mtd@lists.infradead.org; Andrea Scian; Richard Weinberger; dedekind1@gmail.com Subject: Re: RFC: detect and manage power cut on MLC NAND (linux-mtd Digest, Vol 144, Issue 70) Hello, I would like to discuss about another couple of topics: partitioning and SLC emulation. 1) IIUC read/program disturb effects exhibit at block level. In a typical embedded linux systems there are software parts - bootloader, kernel image etc. - that virtually are never changed (almost ...) but are read many times. Other parts - application libraries, log files etc. - are read and wrote many times instead. If these two kinds of software are stored in different MTD partitions - ket's say partition A for bootloader, kernel etc. and partition B for application libraries, log files etc. - can we say that read/write operations performed on partition B have no disturb effects on partition A? 2) IIUC Boris has worked on SLC emulation too. This seems to be a promising feature because it would allow to partition NAND flash and to create higher reliability partition (at the cost of halving the size). Is it possibile to implement such functionality in software stack only(MTD/UBI) or is it necessary that NAND memory supports specific features? Regards, Andrea Marson > Very nice explanation! Not sure if I could have done better myself. > > Jeff Lauruhn > NAND Application Engineer > Embedded Business Unit > Micron Technology, Inc > > > -----Original Message----- > From: Boris Brezillon [mailto:boris.brezillon@free-electrons.com] > Sent: Tuesday, March 17, 2015 3:02 AM > To: Andrea Scian > Cc: Jeff Lauruhn (jlauruhn); Richard Weinberger; dedekind1@gmail.com; > mtd_mailinglist > Subject: Re: RFC: detect and manage power cut on MLC NAND > > Hi Andrea, > > I'll let Jeff answer this question, but I'd like to share my understanding. > > On Tue, 17 Mar 2015 10:30:30 +0100 > Andrea Scian <rnd4@dave-tech.it> wrote: > >> >> >> Dear Jeff, >> >> Il 16/03/2015 22:11, Jeff Lauruhn (jlauruhn) ha scritto: >>> Good morning Boris; >>> RR is a new feature and not available on all parts few. I'm not >>> sure about others, but since these are features, you simply enable >>> of disable via SET FEATURE/GET FEATURE. If you already provide that >>> SET/GET FEATURE functionality then an end-user determine if their >>> device supports a feature and then write the code to enable when >>> they need it on their particular design. >> >> I can confirm this. In fact I'm currently working with two Micron NAND: >> >> MT29F32G08CBACAWP >> MT29F32G08CBADAWP >> >> The latter should be "just" a newer die revision of the former (at >> least, this is what our distributor says) >> >> There's a technology change between the two and, in fact, the latter >> supports RR while there's no mention of such a feature inside rev C. >> >> Jeff, could you please help me in understanding which if the >> following sentences are true and which are false? >> - rev D is more "robust" than rev C because it has RR (so an >> additional feature that improve error correction) >> - rev D is "robust" like rev C, if rev D is used with RR >> - if RR is not used rev D is more error prone than rev C > > RR shouldn't change NAND robustness (or sensitivity to read/write disturbance generating bitflips). > > AFAIU RR will help you improve your NAND lifetime, because you're allowed to change voltage thresholds which means you can fix errors that were previously considered as unfixable and lead to blocks being marked bad earlier. > > I'll let Jeff correct me if I'm wrong ;-). > >> >> I think this is crucial to understand how RR works and how much is >> needed inside MTD/UBI code. > > Hopefully this can all be handled in the MTD layer, with some help from the UBI layer to feed the wear information (number of P/E cycles on each block). > > Best Regards, > > Boris > > -- > Boris Brezillon, Free Electrons > Embedded Linux and Kernel engineering > http://free-electrons.com > > > ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: RFC: detect and manage power cut on MLC NAND 2015-03-18 16:12 ` Jeff Lauruhn (jlauruhn) @ 2015-03-19 8:47 ` Andrea Marson 2015-03-19 9:12 ` Boris Brezillon 2015-03-19 18:00 ` Jeff Lauruhn (jlauruhn) 0 siblings, 2 replies; 59+ messages in thread From: Andrea Marson @ 2015-03-19 8:47 UTC (permalink / raw) To: Jeff Lauruhn (jlauruhn), Boris Brezillon Cc: Andrea Scian, dedekind1, linux-mtd, Richard Weinberger > Disturb is a block level affect, as long as partition A and B are in different blocks there will be no disturb between them. Disturbs, does not damage cells; ERASE returns cells to undisturbed levels. I think there are two options here: MTD partitioning and UBI partitioning. AFAIK one should prefer UBI partitioning to preserve device-wide wear leveling. Boris, am I right? > Officially I would say don't use SLC emulation, but technically I know what your doing. The reason I say no is because we have very precise recipes designed to create very tight distibutions, and although the first pass distributions might look like an SLC, they are really designed with the expectation of the upper page being programmed. Not a true SLC. > With MLC lithography of 25 nm and less the difference between each level (L0, L1, L2 and L3) is just a few 10's of electrons. The distribution have to be very tight, in order to meet retention requirements. This is quite interesting, however I'm afraid I have not fully understood it. Let me try to rephrase it. Please correct me if I'm wrong. 1) Technically speaking, it is possible to use an MLC memory in SLC mode, even if this is not recommended because MLC is not designed for this usage. 2) As indicated by Boris, the easiest way to implement this thing is to avoid the use of paired pages, according to paired page table provided by datasheet. 3) This technique does not transform an MLC NAND to an SLC magically. Thus data retention and lifetime are not increased. However all paired pages issues disappear. It is not clear if there are further drawbacks that reduce flash reliability if used this way. Thank you, Andrea ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: RFC: detect and manage power cut on MLC NAND 2015-03-19 8:47 ` RFC: detect and manage power cut on MLC NAND Andrea Marson @ 2015-03-19 9:12 ` Boris Brezillon 2015-03-19 17:45 ` Jeff Lauruhn (jlauruhn) 2015-03-20 0:25 ` Iwo Mergler 2015-03-19 18:00 ` Jeff Lauruhn (jlauruhn) 1 sibling, 2 replies; 59+ messages in thread From: Boris Brezillon @ 2015-03-19 9:12 UTC (permalink / raw) To: Andrea Marson Cc: Andrea Scian, Jeff Lauruhn (jlauruhn), linux-mtd, dedekind1, Richard Weinberger On Thu, 19 Mar 2015 09:47:21 +0100 Andrea Marson <andrea.marson@dave.eu> wrote: > > Disturb is a block level affect, as long as partition A and B are in different blocks there will be no disturb between them. Disturbs, does not damage cells; ERASE returns cells to undisturbed levels. > I think there are two options here: MTD partitioning and UBI > partitioning. AFAIK one should prefer UBI partitioning to preserve > device-wide wear leveling. Boris, am I right? Both of them act at block level, meaning that your the partition size must be a multiple of the block size (logical block size in case of UBI volume and physical block size in case of MTD partition). IOW, you shouldn't bother whether you're using UBI on top of MTD or directly using MTD partitions, both are immune to cross partition/volume read/write disturbance. > > > Officially I would say don't use SLC emulation, but technically I know what your doing. The reason I say no is because we have very precise recipes designed to create very tight distibutions, and although the first pass distributions might look like an SLC, they are really designed with the expectation of the upper page being programmed. Not a true SLC. > > With MLC lithography of 25 nm and less the difference between each level (L0, L1, L2 and L3) is just a few 10's of electrons. The distribution have to be very tight, in order to meet retention requirements. > This is quite interesting, however I'm afraid I have not fully > understood it. Me neither :-/. > Let me try to rephrase it. Please correct me if I'm wrong. > > 1) Technically speaking, it is possible to use an MLC memory in SLC > mode, even if this is not recommended because MLC is not designed for > this usage. That's what I understood, but I'm not sure to understand the constraints brought by SLC mode (only programming one of the paired pages). Jeff, Are you trying to explain what's described here [1] in slide 8 (BTW I'm not sure to understand this diagram). If that's the case, could you explain us, how the NAND chip knows which threshold should be used (does it somehow store the information of which page has already been programmed) [1]http://www.bswd.com/FMS09/FMS09-T2A-Grunzke.pdf -- Boris Brezillon, Free Electrons Embedded Linux and Kernel engineering http://free-electrons.com ^ permalink raw reply [flat|nested] 59+ messages in thread
* RE: RFC: detect and manage power cut on MLC NAND 2015-03-19 9:12 ` Boris Brezillon @ 2015-03-19 17:45 ` Jeff Lauruhn (jlauruhn) 2015-03-20 0:25 ` Iwo Mergler 1 sibling, 0 replies; 59+ messages in thread From: Jeff Lauruhn (jlauruhn) @ 2015-03-19 17:45 UTC (permalink / raw) To: Boris Brezillon, Andrea Marson Cc: Andrea Scian, dedekind1, linux-mtd, Richard Weinberger Boris, You seem to have a pretty good understanding of NAND. Now that I have a diagram, it will make this easier to explain. Refer to page 8 of the presentation. When we program and MLC device we program the Lower Page first. In a perfect world all bits would set to discrete values like -.5v and 2.0v, called Vt erased and Vt programmed. But, in reality Vts end up in a distribution, due to process, location of cell with a block, couplings, previous state of the cell, etc. The diagram we are looking at is a representation of the distribution of all cells in the die, so if we programmed only the lower page and measured the potential on all cells, they would fall into 2 distributions around -.5v and 2.0v, I refer to them as L0 and L1. L0 is defined as erased, or 1h and L1 is defined as programmed or 0h. And yes there is a bit, to indicate if the middle or upper page is programmed. Now, program the middle page (this is MLC 2bits/cell btw). First read the lower page and get it's current value. If the lower page of a cell is 1 and you want to store a 1 in the upper page of the cell, then tighten the distribution around -.5 and call this new L0. If the lower page of the cell is 1 and you want to upper page to be 0, then move and tighten L0 to new L1. If the lower page is 0 and you want the upper page to be a 0, then add charge to L1 and tighten, call this new L2. Finally if you lower page is 0 and you want the upper page to be 1 add a little more charge and tighten to the new L3. Now if you could measure the potential of every cell in the array they would fall into 4 tighter distributions, L0 to L3, representing all for conditions. 00, 01, 10, 11. I'm not going to go into the TLC, just follow the arrows. If you have a power loss during a program, only the page you're programming gets affected, not all distributions. The programming process is designed to make it easier to recover the lower page if there's a power loss while programming the upper page. These guys have a pretty good explaination. http://www.supertalent.com/datasheets/SLC_vs_MLC%20whitepaper.pdf Hope this helps. Jeff Lauruhn NAND Application Engineer -----Original Message----- From: Boris Brezillon [mailto:boris.brezillon@free-electrons.com] Sent: Thursday, March 19, 2015 2:13 AM To: Andrea Marson Cc: Jeff Lauruhn (jlauruhn); linux-mtd@lists.infradead.org; Andrea Scian; Richard Weinberger; dedekind1@gmail.com Subject: Re: RFC: detect and manage power cut on MLC NAND On Thu, 19 Mar 2015 09:47:21 +0100 Andrea Marson <andrea.marson@dave.eu> wrote: > > Disturb is a block level affect, as long as partition A and B are in different blocks there will be no disturb between them. Disturbs, does not damage cells; ERASE returns cells to undisturbed levels. > I think there are two options here: MTD partitioning and UBI > partitioning. AFAIK one should prefer UBI partitioning to preserve > device-wide wear leveling. Boris, am I right? Both of them act at block level, meaning that your the partition size must be a multiple of the block size (logical block size in case of UBI volume and physical block size in case of MTD partition). IOW, you shouldn't bother whether you're using UBI on top of MTD or directly using MTD partitions, both are immune to cross partition/volume read/write disturbance. > > > Officially I would say don't use SLC emulation, but technically I know what your doing. The reason I say no is because we have very precise recipes designed to create very tight distibutions, and although the first pass distributions might look like an SLC, they are really designed with the expectation of the upper page being programmed. Not a true SLC. > > With MLC lithography of 25 nm and less the difference between each level (L0, L1, L2 and L3) is just a few 10's of electrons. The distribution have to be very tight, in order to meet retention requirements. > This is quite interesting, however I'm afraid I have not fully > understood it. Me neither :-/. > Let me try to rephrase it. Please correct me if I'm wrong. > > 1) Technically speaking, it is possible to use an MLC memory in SLC > mode, even if this is not recommended because MLC is not designed for > this usage. That's what I understood, but I'm not sure to understand the constraints brought by SLC mode (only programming one of the paired pages). Jeff, Are you trying to explain what's described here [1] in slide 8 (BTW I'm not sure to understand this diagram). If that's the case, could you explain us, how the NAND chip knows which threshold should be used (does it somehow store the information of which page has already been programmed) [1]http://www.bswd.com/FMS09/FMS09-T2A-Grunzke.pdf -- Boris Brezillon, Free Electrons Embedded Linux and Kernel engineering http://free-electrons.com ^ permalink raw reply [flat|nested] 59+ messages in thread
* RE: RFC: detect and manage power cut on MLC NAND 2015-03-19 9:12 ` Boris Brezillon 2015-03-19 17:45 ` Jeff Lauruhn (jlauruhn) @ 2015-03-20 0:25 ` Iwo Mergler 2015-03-20 3:38 ` nick 2015-03-20 8:26 ` Boris Brezillon 1 sibling, 2 replies; 59+ messages in thread From: Iwo Mergler @ 2015-03-20 0:25 UTC (permalink / raw) To: Boris Brezillon, Andrea Marson Cc: Andrea Scian, Jeff Lauruhn (jlauruhn), linux-mtd, Richard Weinberger, dedekind1 Hi all, I've been thinking a little about the paired pages problem. The patented mitigation methods I've seen so far are 1) SanDisk: Use a separate log mechanism to back up LSB page while writing MSB page. 2) M-Systems: Within the block, interleave backup copies of LSB pages when writing MSB pages. 3) Micron: Use as a half-capacity SLC device by writing LSB and MSB pages such that the levels are reinforced. Jeff mentioned this one. Here is another idea. I assume it's patented already, in which case I'd like to hear about it. If not, consider this published as of now :-) Get UBI to map the paired pages into a single write unit. Jumbo page if you will. So, instead of a block with 64 pages, we get 32 pages, twice as large. Thus, the paired pages will be written in quick succession. A power cut during this is reduced to the unstable bits issue we already have with SLC. UBI could take the risk and split the first page pair between EC and VID headers, replacing the EC info with average in case of unlikely failure. Would this work? Best regards, Iwo ________________________________________ From: linux-mtd [linux-mtd-bounces@lists.infradead.org] On Behalf Of Boris Brezillon [boris.brezillon@free-electrons.com] Sent: Thursday, 19 March 2015 8:12 PM To: Andrea Marson Cc: Andrea Scian; Jeff Lauruhn (jlauruhn); linux-mtd@lists.infradead.org; dedekind1@gmail.com; Richard Weinberger Subject: Re: RFC: detect and manage power cut on MLC NAND On Thu, 19 Mar 2015 09:47:21 +0100 Andrea Marson <andrea.marson@dave.eu> wrote: > > Disturb is a block level affect, as long as partition A and B are in different blocks there will be no disturb between them. Disturbs, does not damage cells; ERASE returns cells to undisturbed levels. > I think there are two options here: MTD partitioning and UBI > partitioning. AFAIK one should prefer UBI partitioning to preserve > device-wide wear leveling. Boris, am I right? Both of them act at block level, meaning that your the partition size must be a multiple of the block size (logical block size in case of UBI volume and physical block size in case of MTD partition). IOW, you shouldn't bother whether you're using UBI on top of MTD or directly using MTD partitions, both are immune to cross partition/volume read/write disturbance. > > > Officially I would say don't use SLC emulation, but technically I know what your doing. The reason I say no is because we have very precise recipes designed to create very tight distibutions, and although the first pass distributions might look like an SLC, they are really designed with the expectation of the upper page being programmed. Not a true SLC. > > With MLC lithography of 25 nm and less the difference between each level (L0, L1, L2 and L3) is just a few 10's of electrons. The distribution have to be very tight, in order to meet retention requirements. > This is quite interesting, however I'm afraid I have not fully > understood it. Me neither :-/. > Let me try to rephrase it. Please correct me if I'm wrong. > > 1) Technically speaking, it is possible to use an MLC memory in SLC > mode, even if this is not recommended because MLC is not designed for > this usage. That's what I understood, but I'm not sure to understand the constraints brought by SLC mode (only programming one of the paired pages). Jeff, Are you trying to explain what's described here [1] in slide 8 (BTW I'm not sure to understand this diagram). If that's the case, could you explain us, how the NAND chip knows which threshold should be used (does it somehow store the information of which page has already been programmed) [1]http://www.bswd.com/FMS09/FMS09-T2A-Grunzke.pdf -- Boris Brezillon, Free Electrons Embedded Linux and Kernel engineering http://free-electrons.com ______________________________________________________ Linux MTD discussion mailing list http://lists.infradead.org/mailman/listinfo/linux-mtd/ ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: RFC: detect and manage power cut on MLC NAND 2015-03-20 0:25 ` Iwo Mergler @ 2015-03-20 3:38 ` nick 2015-03-20 5:40 ` Iwo Mergler 2015-03-20 8:26 ` Boris Brezillon 1 sibling, 1 reply; 59+ messages in thread From: nick @ 2015-03-20 3:38 UTC (permalink / raw) To: Iwo Mergler, Boris Brezillon, Andrea Marson Cc: Andrea Scian, Jeff Lauruhn (jlauruhn), linux-mtd, dedekind1, Richard Weinberger On 2015-03-19 08:25 PM, Iwo Mergler wrote: > > Hi all, > > > I've been thinking a little about the paired pages problem. > > The patented mitigation methods I've seen so far are > > 1) SanDisk: Use a separate log mechanism to back up LSB page > while writing MSB page. > > 2) M-Systems: Within the block, interleave backup copies of LSB > pages when writing MSB pages. > > 3) Micron: Use as a half-capacity SLC device by writing LSB and MSB > pages such that the levels are reinforced. Jeff mentioned this one. > > > Here is another idea. I assume it's patented already, in which case > I'd like to hear about it. If not, consider this published as of now :-) > > > Get UBI to map the paired pages into a single write unit. Jumbo page > if you will. So, instead of a block with 64 pages, we get 32 pages, > twice as large. > > Thus, the paired pages will be written in quick succession. A power > cut during this is reduced to the unstable bits issue we already have > with SLC. > > UBI could take the risk and split the first page pair between EC and > VID headers, replacing the EC info with average in case of unlikely > failure. > > Would this work? > > > Best regards, > > Iwo > > > Iwo, Not a bad idea but what happens if the power gets cut after the first 2 pages then we lose 30 pages worth of data. Unless the hardware is very fast and can move the pages over in a millisecond or less then this can't really worth. However if we make these transactions atomic in nature this may work better as CPU instruction respond is in the nanosecond range even on low cost embedded boards using UBI for raw flash embedded on to themselves. Nick ________________________________________ > From: linux-mtd [linux-mtd-bounces@lists.infradead.org] On Behalf Of Boris Brezillon [boris.brezillon@free-electrons.com] > Sent: Thursday, 19 March 2015 8:12 PM > To: Andrea Marson > Cc: Andrea Scian; Jeff Lauruhn (jlauruhn); linux-mtd@lists.infradead.org; dedekind1@gmail.com; Richard Weinberger > Subject: Re: RFC: detect and manage power cut on MLC NAND > > On Thu, 19 Mar 2015 09:47:21 +0100 > Andrea Marson <andrea.marson@dave.eu> wrote: > >>> Disturb is a block level affect, as long as partition A and B are in different blocks there will be no disturb between them. Disturbs, does not damage cells; ERASE returns cells to undisturbed levels. >> I think there are two options here: MTD partitioning and UBI >> partitioning. AFAIK one should prefer UBI partitioning to preserve >> device-wide wear leveling. Boris, am I right? > > Both of them act at block level, meaning that your the partition size > must be a multiple of the block size (logical block size in case of UBI > volume and physical block size in case of MTD partition). > IOW, you shouldn't bother whether you're using UBI on top of MTD or > directly using MTD partitions, both are immune to cross partition/volume > read/write disturbance. > > >> >>> Officially I would say don't use SLC emulation, but technically I know what your doing. The reason I say no is because we have very precise recipes designed to create very tight distibutions, and although the first pass distributions might look like an SLC, they are really designed with the expectation of the upper page being programmed. Not a true SLC. >>> With MLC lithography of 25 nm and less the difference between each level (L0, L1, L2 and L3) is just a few 10's of electrons. The distribution have to be very tight, in order to meet retention requirements. >> This is quite interesting, however I'm afraid I have not fully >> understood it. > > Me neither :-/. > >> Let me try to rephrase it. Please correct me if I'm wrong. >> >> 1) Technically speaking, it is possible to use an MLC memory in SLC >> mode, even if this is not recommended because MLC is not designed for >> this usage. > > That's what I understood, but I'm not sure to understand the > constraints brought by SLC mode (only programming one of the paired > pages). > Jeff, Are you trying to explain what's described here [1] in slide 8 > (BTW I'm not sure to understand this diagram). > If that's the case, could you explain us, how the NAND chip knows which > threshold should be used (does it somehow store the information of > which page has already been programmed) > > [1]http://www.bswd.com/FMS09/FMS09-T2A-Grunzke.pdf > > -- > Boris Brezillon, Free Electrons > Embedded Linux and Kernel engineering > http://free-electrons.com > > ______________________________________________________ > Linux MTD discussion mailing list > http://lists.infradead.org/mailman/listinfo/linux-mtd/ > > ______________________________________________________ > Linux MTD discussion mailing list > http://lists.infradead.org/mailman/listinfo/linux-mtd/ > ^ permalink raw reply [flat|nested] 59+ messages in thread
* RE: RFC: detect and manage power cut on MLC NAND 2015-03-20 3:38 ` nick @ 2015-03-20 5:40 ` Iwo Mergler 0 siblings, 0 replies; 59+ messages in thread From: Iwo Mergler @ 2015-03-20 5:40 UTC (permalink / raw) To: nick, Boris Brezillon, Andrea Marson Cc: Andrea Scian, Jeff Lauruhn (jlauruhn), linux-mtd, dedekind1, Richard Weinberger On Fri, 20 Mar 2015 14:38:03 +1100 nick <xerofoify@gmail.com> wrote: > > > On 2015-03-19 08:25 PM, Iwo Mergler wrote: > > > > Get UBI to map the paired pages into a single write unit. Jumbo page > > if you will. So, instead of a block with 64 pages, we get 32 pages, > > twice as large. > > > > Thus, the paired pages will be written in quick succession. A power > > cut during this is reduced to the unstable bits issue we already > > have with SLC. > > > > UBI could take the risk and split the first page pair between EC and > > VID headers, replacing the EC info with average in case of unlikely > > failure. > > > > Would this work? > > > > > > Best regards, > > > > Iwo > > > > > > > Iwo, > Not a bad idea but what happens if the power gets cut after the first > 2 pages then we lose 30 pages worth of data. Unless the hardware is > very fast and can move the pages over in a millisecond or less then > this can't really worth. However if we make these transactions atomic > in nature this may work better as CPU instruction respond is in the > nanosecond range even on low cost embedded boards using UBI for raw > flash embedded on to themselves. Nick I'm not sure I understand - why would you lose any data outside the affected paired pages? Also, write order within the block will still be respected. That is, when writing the first page pair, the other pages will be erased. Best regards, Iwo ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: RFC: detect and manage power cut on MLC NAND 2015-03-20 0:25 ` Iwo Mergler 2015-03-20 3:38 ` nick @ 2015-03-20 8:26 ` Boris Brezillon 2015-03-20 17:15 ` Nick Krause ` (2 more replies) 1 sibling, 3 replies; 59+ messages in thread From: Boris Brezillon @ 2015-03-20 8:26 UTC (permalink / raw) To: Iwo Mergler Cc: Jeff Lauruhn (jlauruhn), dedekind1, Richard Weinberger, Andrea Scian, linux-mtd, Andrea Marson Hi Iwo, On Fri, 20 Mar 2015 11:25:18 +1100 Iwo Mergler <Iwo.Mergler@netcommwireless.com> wrote: > > Hi all, > > > I've been thinking a little about the paired pages problem. > > The patented mitigation methods I've seen so far are > > 1) SanDisk: Use a separate log mechanism to back up LSB page > while writing MSB page. > > 2) M-Systems: Within the block, interleave backup copies of LSB > pages when writing MSB pages. > > 3) Micron: Use as a half-capacity SLC device by writing LSB and MSB > pages such that the levels are reinforced. Jeff mentioned this one. All these solutions involves storing a backup of the first paired page, before writing the 2nd one (I guess that's even worse with TLC NANDs: you'll have to save the 2nd page when writing the 3rd one). If that's possible, I think we should find a way to avoid this duplication. > > > Here is another idea. I assume it's patented already, in which case > I'd like to hear about it. If not, consider this published as of now :-) > > > Get UBI to map the paired pages into a single write unit. Jumbo page > if you will. So, instead of a block with 64 pages, we get 32 pages, > twice as large. > > Thus, the paired pages will be written in quick succession. A power > cut during this is reduced to the unstable bits issue we already have > with SLC. That's actually a good idea, but I'd like to have feedback from Jeff and/or Qi regarding the feasibility of this proposal. I've read many times that MLC pages should be programmed in ascending order (0, 1, 2, 3, ..., N), and if you take a look at MLC datasheet you'll see that paired pages are not contiguous (here is an example [1], page 55-56 describe how pages are paired together). My question is: is there a reason for interleaving paired pages with other pages (write disturbance mitigation ?) ? If there is no specific reason but to annoy software developers :-), then I think we should consider this Jumbo page approach, because it would contain the paired pages handling in the NAND layer (instead of exposing complexity to the UBI/UBIFS layers) while allowing us to use almost all the NAND chip capacity. > > UBI could take the risk and split the first page pair between EC and > VID headers, replacing the EC info with average in case of unlikely > failure. You want to do that in order to avoid using 2 Jumbo pages to only store a few bytes ? Artem suggested another solution to deal with that: duplicate the EC header in the VID header so that EC information can be recovered when writing to the page paired with the EC header. You'd still have to prevent any write on the page paired with the EC header, but you're still saving one page with this approach and avoid any corruption of UBI metadata caused by paired pages. Maybe there are better solution, and I've been thinking about a different approach to avoid writing the EC and VID headers in 2 different pages: store EC header information in a specific UBI volume (containing one or two LEBs). This volume would store a log of EC information update, so that when you erase a page you don't have to write the EC header right away, but should just add an entry in the UBI EC log. This way you write EC and VID information in one go when someone maps a LEB (attach it to a specific volume). These are just thoughts ;-). [1]http://www.szyuda88.com/uploadfile/cfile/2013419135343223.pdf -- Boris Brezillon, Free Electrons Embedded Linux and Kernel engineering http://free-electrons.com ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: RFC: detect and manage power cut on MLC NAND 2015-03-20 8:26 ` Boris Brezillon @ 2015-03-20 17:15 ` Nick Krause 2015-03-22 23:45 ` Iwo Mergler 2015-03-23 2:18 ` Iwo Mergler 2 siblings, 0 replies; 59+ messages in thread From: Nick Krause @ 2015-03-20 17:15 UTC (permalink / raw) To: Boris Brezillon Cc: Iwo Mergler, Jeff Lauruhn (jlauruhn), dedekind1, Richard Weinberger, Andrea Scian, linux-mtd, Andrea Marson On Fri, Mar 20, 2015 at 4:26 AM, Boris Brezillon <boris.brezillon@free-electrons.com> wrote: > Hi Iwo, > > On Fri, 20 Mar 2015 11:25:18 +1100 > Iwo Mergler <Iwo.Mergler@netcommwireless.com> wrote: > >> >> Hi all, >> >> >> I've been thinking a little about the paired pages problem. >> >> The patented mitigation methods I've seen so far are >> >> 1) SanDisk: Use a separate log mechanism to back up LSB page >> while writing MSB page. >> >> 2) M-Systems: Within the block, interleave backup copies of LSB >> pages when writing MSB pages. >> >> 3) Micron: Use as a half-capacity SLC device by writing LSB and MSB >> pages such that the levels are reinforced. Jeff mentioned this one. > > All these solutions involves storing a backup of the first paired page, > before writing the 2nd one (I guess that's even worse with TLC NANDs: > you'll have to save the 2nd page when writing the 3rd one). > If that's possible, I think we should find a way to avoid this > duplication. > >> >> >> Here is another idea. I assume it's patented already, in which case >> I'd like to hear about it. If not, consider this published as of now :-) >> >> >> Get UBI to map the paired pages into a single write unit. Jumbo page >> if you will. So, instead of a block with 64 pages, we get 32 pages, >> twice as large. >> >> Thus, the paired pages will be written in quick succession. A power >> cut during this is reduced to the unstable bits issue we already have >> with SLC. > > That's actually a good idea, but I'd like to have feedback from Jeff > and/or Qi regarding the feasibility of this proposal. > I've read many times that MLC pages should be programmed in ascending > order (0, 1, 2, 3, ..., N), and if you take a look at MLC datasheet > you'll see that paired pages are not contiguous (here is an example > [1], page 55-56 describe how pages are paired together). > > My question is: is there a reason for interleaving paired pages with > other pages (write disturbance mitigation ?) ? > > If there is no specific reason but to annoy software developers :-), > then I think we should consider this Jumbo page approach, because it > would contain the paired pages handling in the NAND layer (instead of > exposing complexity to the UBI/UBIFS layers) while allowing us to use > almost all the NAND chip capacity. > >> >> UBI could take the risk and split the first page pair between EC and >> VID headers, replacing the EC info with average in case of unlikely >> failure. > > You want to do that in order to avoid using 2 Jumbo pages to only store > a few bytes ? > Artem suggested another solution to deal with that: duplicate the EC > header in the VID header so that EC information can be recovered when > writing to the page paired with the EC header. > You'd still have to prevent any write on the page paired with the EC > header, but you're still saving one page with this approach and avoid > any corruption of UBI metadata caused by paired pages. > > Maybe there are better solution, and I've been thinking about a > different approach to avoid writing the EC and VID headers in 2 > different pages: store EC header information in a specific UBI volume > (containing one or two LEBs). > This volume would store a log of EC information update, so that when > you erase a page you don't have to write the EC header right away, but > should just add an entry in the UBI EC log. > This way you write EC and VID information in one go when someone maps a > LEB (attach it to a specific volume). > > These are just thoughts ;-). > > [1]http://www.szyuda88.com/uploadfile/cfile/2013419135343223.pdf > -- > Boris Brezillon, Free Electrons > Embedded Linux and Kernel engineering > http://free-electrons.com > > ______________________________________________________ > Linux MTD discussion mailing list > http://lists.infradead.org/mailman/listinfo/linux-mtd/ Iwo, I was assuming the pages were still in ram and volatile for my example. Nick ^ permalink raw reply [flat|nested] 59+ messages in thread
* RE: RFC: detect and manage power cut on MLC NAND 2015-03-20 8:26 ` Boris Brezillon 2015-03-20 17:15 ` Nick Krause @ 2015-03-22 23:45 ` Iwo Mergler 2015-03-23 2:18 ` Iwo Mergler 2 siblings, 0 replies; 59+ messages in thread From: Iwo Mergler @ 2015-03-22 23:45 UTC (permalink / raw) To: Boris Brezillon Cc: Jeff Lauruhn (jlauruhn), dedekind1, Richard Weinberger, Andrea Scian, linux-mtd, Andrea Marson On Fri, 20 Mar 2015 19:26:11 +1100 Boris Brezillon <boris.brezillon@free-electrons.com> wrote: > I've read many times that MLC pages should be programmed in ascending > order (0, 1, 2, 3, ..., N), and if you take a look at MLC datasheet > you'll see that paired pages are not contiguous (here is an example > [1], page 55-56 describe how pages are paired together). > > My question is: is there a reason for interleaving paired pages with > other pages (write disturbance mitigation ?) ? > > If there is no specific reason but to annoy software developers :-), I don't know for sure, but it seems that groups of pages are sharing some resources. For instance, I've seen a few bit error patterns which looked like this (256MB SLC, pages in one block): -------------------------------X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X The pages marked with 'X' have a single bit error at the same bit offset in every page. That is, even on a SLC device, there is some sort of page interleaving pattern at work. In this case, maybe they share a broken sense amplifier or similar. I currently think that the rule about the ascending order is a generalisation for unknown NAND architecture. In my example, it may well be slightly better to write (0,2,4,6,8,10,12,14,1,3,5, 7,9,11,13,15,16,18,20,22,24,26,28,30,17,19,21,23,25,27,29,31). Unless someone tells me otherwise, I guess that the paired pages probably should be written in sequence anyway (0,4,1,5,etc). Some of the backup methods appear suggestive of this. Best regards, Iwo ^ permalink raw reply [flat|nested] 59+ messages in thread
* RE: RFC: detect and manage power cut on MLC NAND 2015-03-20 8:26 ` Boris Brezillon 2015-03-20 17:15 ` Nick Krause 2015-03-22 23:45 ` Iwo Mergler @ 2015-03-23 2:18 ` Iwo Mergler 2015-03-23 7:06 ` Artem Bityutskiy 2 siblings, 1 reply; 59+ messages in thread From: Iwo Mergler @ 2015-03-23 2:18 UTC (permalink / raw) To: Boris Brezillon Cc: Jeff Lauruhn (jlauruhn), dedekind1, Richard Weinberger, Andrea Scian, linux-mtd, Andrea Marson On Fri, 20 Mar 2015 19:26:11 +1100 Boris Brezillon <boris.brezillon@free-electrons.com> wrote: > You want to do that in order to avoid using 2 Jumbo pages to only > store a few bytes ? Yes. I'm already a little unhappy about using two whole pages for UBI, and don't want to be guilty of doubling that. > Artem suggested another solution to deal with that: duplicate the EC > header in the VID header so that EC information can be recovered when > writing to the page paired with the EC header. Sounds good to me. It won't avoid the loss of the VID header, but it will safeguard the EC. It's free, too. > You'd still have to prevent any write on the page paired with the EC > header, but you're still saving one page with this approach and avoid > any corruption of UBI metadata caused by paired pages. Ah, you probably meant VID header here, right? > Maybe there are better solution, and I've been thinking about a > different approach to avoid writing the EC and VID headers in 2 > different pages: store EC header information in a specific UBI volume > (containing one or two LEBs). > This volume would store a log of EC information update, so that when > you erase a page you don't have to write the EC header right away, but > should just add an entry in the UBI EC log. > This way you write EC and VID information in one go when someone maps > a LEB (attach it to a specific volume). I like that. Maybe UBI could try grouping erases to reduce the stress on the EC log - write the (future) ECs for a number of PEBs to the log, then delete the PEBS at leisure, then mark them as done in the log. In fact, you could even go and pad the log pages with future ECs for active PEBs, to be used in case they need erasing later. Just never erase a PEB that doesn't have an EC log entry. Would it be possible to push that further, and defer the writing of the VID header until the first payload write is pending? Up to that point, it shouldn't matter where the LEB is mapped. I'm aware that UBIFS would have to cooperate here (first page is 64 bytes smaller), but VID and integrated EC could share the page 0 with the first payload nodes of the LEB. Best regards, Iwo ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: RFC: detect and manage power cut on MLC NAND 2015-03-23 2:18 ` Iwo Mergler @ 2015-03-23 7:06 ` Artem Bityutskiy 2015-03-23 19:05 ` Boris Brezillon 0 siblings, 1 reply; 59+ messages in thread From: Artem Bityutskiy @ 2015-03-23 7:06 UTC (permalink / raw) To: Iwo Mergler Cc: Boris Brezillon, Jeff Lauruhn (jlauruhn), Richard Weinberger, Andrea Scian, linux-mtd, Andrea Marson On Mon, 2015-03-23 at 13:18 +1100, Iwo Mergler wrote: > On Fri, 20 Mar 2015 19:26:11 +1100 > Boris Brezillon <boris.brezillon@free-electrons.com> wrote: > > > You want to do that in order to avoid using 2 Jumbo pages to only > > store a few bytes ? > > Yes. I'm already a little unhappy about using two whole pages > for UBI, and don't want to be guilty of doubling that. > > > Artem suggested another solution to deal with that: duplicate the EC > > header in the VID header so that EC information can be recovered when > > writing to the page paired with the EC header. > > Sounds good to me. It won't avoid the loss of the VID header, > but it will safeguard the EC. It's free, too. > > > You'd still have to prevent any write on the page paired with the EC > > header, but you're still saving one page with this approach and avoid > > any corruption of UBI metadata caused by paired pages. > > Ah, you probably meant VID header here, right? If we sacrifice another page, we could store a copy of the VID header there, in case the first copy gets corrupted. IOW, we'd have: ECH, VIDH#1, VIDH#2, User Data. ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: RFC: detect and manage power cut on MLC NAND 2015-03-23 7:06 ` Artem Bityutskiy @ 2015-03-23 19:05 ` Boris Brezillon 2015-03-24 7:05 ` Artem Bityutskiy 0 siblings, 1 reply; 59+ messages in thread From: Boris Brezillon @ 2015-03-23 19:05 UTC (permalink / raw) To: dedekind1 Cc: Iwo Mergler, Jeff Lauruhn (jlauruhn), Richard Weinberger, Andrea Scian, linux-mtd, Andrea Marson Hi Artem, On Mon, 23 Mar 2015 09:06:31 +0200 Artem Bityutskiy <dedekind1@gmail.com> wrote: > > > > > You'd still have to prevent any write on the page paired with the EC > > > header, but you're still saving one page with this approach and avoid > > > any corruption of UBI metadata caused by paired pages. > > > > Ah, you probably meant VID header here, right? > > If we sacrifice another page, we could store a copy of the VID header > there, in case the first copy gets corrupted. IOW, we'd have: > > ECH, VIDH#1, VIDH#2, User Data. What do you mean by 'sacrificing another page', are you talking about the one paired with the VID header page ? If that's the case, then we cannot write anything on it (even a VID header backup), because if the PROGRAM operation is interrupted both pages (VIDH#1 and #2) could be corrupted. Best Regards, Boris -- Boris Brezillon, Free Electrons Embedded Linux and Kernel engineering http://free-electrons.com ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: RFC: detect and manage power cut on MLC NAND 2015-03-23 19:05 ` Boris Brezillon @ 2015-03-24 7:05 ` Artem Bityutskiy 0 siblings, 0 replies; 59+ messages in thread From: Artem Bityutskiy @ 2015-03-24 7:05 UTC (permalink / raw) To: Boris Brezillon Cc: Iwo Mergler, Jeff Lauruhn (jlauruhn), Richard Weinberger, Andrea Scian, linux-mtd, Andrea Marson On Mon, 2015-03-23 at 20:05 +0100, Boris Brezillon wrote: > > If we sacrifice another page, we could store a copy of the VID header > > there, in case the first copy gets corrupted. IOW, we'd have: > > > > ECH, VIDH#1, VIDH#2, User Data. > > What do you mean by 'sacrificing another page', are you talking about > the one paired with the VID header page ? > If that's the case, then we cannot write anything on it (even a VID > header backup), because if the PROGRAM operation is interrupted both > pages (VIDH#1 and #2) could be corrupted. Just this: Page 0: ECH Page 1: VIDH#1 (+all the ECH data) Page 2: VIDH#2 (same as VIDH#1) VIDH#1 and #2 are not in paired pages, so I expect one of them to always be good. If pages 0,1;2,3;4,5;etc are paired, we may lose ECH sometimes, should not be catastrophic. I wonder though, if this pairing scheme is realistic? In case of a larger pairing step we seem to be always OK and will always have a valid ECH and VIDH. Do I miss something? ^ permalink raw reply [flat|nested] 59+ messages in thread
* RE: RFC: detect and manage power cut on MLC NAND 2015-03-19 8:47 ` RFC: detect and manage power cut on MLC NAND Andrea Marson 2015-03-19 9:12 ` Boris Brezillon @ 2015-03-19 18:00 ` Jeff Lauruhn (jlauruhn) 2015-03-20 8:07 ` Andrea Marson 1 sibling, 1 reply; 59+ messages in thread From: Jeff Lauruhn (jlauruhn) @ 2015-03-19 18:00 UTC (permalink / raw) To: Andrea Marson, Boris Brezillon Cc: Andrea Scian, dedekind1, linux-mtd, Richard Weinberger Jeff Lauruhn NAND Application Engineer -----Original Message----- From: Andrea Marson [mailto:andrea.marson@dave.eu] Sent: Thursday, March 19, 2015 1:47 AM To: Jeff Lauruhn (jlauruhn); Boris Brezillon Cc: linux-mtd@lists.infradead.org; Andrea Scian; Richard Weinberger; dedekind1@gmail.com Subject: Re: RFC: detect and manage power cut on MLC NAND > Disturb is a block level affect, as long as partition A and B are in different blocks there will be no disturb between them. Disturbs, does not damage cells; ERASE returns cells to undisturbed levels. I think there are two options here: MTD partitioning and UBI partitioning. AFAIK one should prefer UBI partitioning to preserve device-wide wear leveling. Boris, am I right? > Officially I would say don't use SLC emulation, but technically I know what your doing. The reason I say no is because we have very precise recipes designed to create very tight distibutions, and although the first pass distributions might look like an SLC, they are really designed with the expectation of the upper page being programmed. Not a true SLC. > With MLC lithography of 25 nm and less the difference between each level (L0, L1, L2 and L3) is just a few 10's of electrons. The distribution have to be very tight, in order to meet retention requirements. This is quite interesting, however I'm afraid I have not fully understood it. Let me try to rephrase it. Please correct me if I'm wrong. 1) Technically speaking, it is possible to use an MLC memory in SLC mode, even if this is not recommended because MLC is not designed for this usage. There are devices that support SLC Mode. You can set a feature and treat blocks like SLC, this is the preferred method, because the P/E recipes used to set the values are SLC specific. Also, there are people who program just to lower pages of MLC device which could be referred to SLC like. This is not the preferred method because the P/E recipes would be set for MLC specifics assuming all pages would be used. 2) As indicated by Boris, the easiest way to implement this thing is to avoid the use of paired pages, according to paired page table provided by datasheet. Boris has a good understanding of NAND. 3) This technique does not transform an MLC NAND to an SLC magically. Thus data retention and lifetime are not increased. However all paired pages issues disappear. It is not clear if there are further drawbacks that reduce flash reliability if used this way. True. This is SLC like, half the capacity of MLC, data retention and life time expectations could be in the range of SLC but it would be hard to know for sure. Thank you, Andrea ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: RFC: detect and manage power cut on MLC NAND 2015-03-19 18:00 ` Jeff Lauruhn (jlauruhn) @ 2015-03-20 8:07 ` Andrea Marson 0 siblings, 0 replies; 59+ messages in thread From: Andrea Marson @ 2015-03-20 8:07 UTC (permalink / raw) To: Jeff Lauruhn (jlauruhn) Cc: Boris Brezillon, dedekind1, linux-mtd, Richard Weinberger, Andrea Scian > 1) Technically speaking, it is possible to use an MLC memory in SLC mode, even if this is not recommended because MLC is not designed for this usage. > There are devices that support SLC Mode. Can you please indicate Micron part numbers supporting SLC mode? It would be really interesting to test these devices on our platforms. Thank you, Andrea ^ permalink raw reply [flat|nested] 59+ messages in thread
* RE: RFC: detect and manage power cut on MLC NAND 2015-03-17 9:30 ` Andrea Scian 2015-03-17 10:02 ` Boris Brezillon @ 2015-03-17 17:04 ` Jeff Lauruhn (jlauruhn) 1 sibling, 0 replies; 59+ messages in thread From: Jeff Lauruhn (jlauruhn) @ 2015-03-17 17:04 UTC (permalink / raw) To: Andrea Scian Cc: Boris Brezillon, dedekind1, mtd_mailinglist, Richard Weinberger Yes, MT29F32G08CBADAWP is available under NDA. These are actually two different process nodes. MT29F32G08CBADAWP is one of our latest processes. Jeff Lauruhn NAND Application Engineer Embedded Business Unit -----Original Message----- From: Andrea Scian [mailto:rnd4@dave-tech.it] Sent: Tuesday, March 17, 2015 2:30 AM To: Jeff Lauruhn (jlauruhn) Cc: Boris Brezillon; Richard Weinberger; dedekind1@gmail.com; mtd_mailinglist Subject: Re: RFC: detect and manage power cut on MLC NAND Dear Jeff, Il 16/03/2015 22:11, Jeff Lauruhn (jlauruhn) ha scritto: > Good morning Boris; > RR is a new feature and not available on all parts few. I'm not sure > about others, but since these are features, you simply enable of > disable via SET FEATURE/GET FEATURE. If you already provide that > SET/GET FEATURE functionality then an end-user determine if their > device supports a feature and then write the code to enable when they > need it on their particular design. I can confirm this. In fact I'm currently working with two Micron NAND: MT29F32G08CBACAWP MT29F32G08CBADAWP The latter should be "just" a newer die revision of the former (at least, this is what our distributor says) There's a technology change between the two and, in fact, the latter supports RR while there's no mention of such a feature inside rev C. Jeff, could you please help me in understanding which if the following sentences are true and which are false? - rev D is more "robust" than rev C because it has RR (so an additional feature that improve error correction) - rev D is "robust" like rev C, if rev D is used with RR - if RR is not used rev D is more error prone than rev C I think this is crucial to understand how RR works and how much is needed inside MTD/UBI code. I hope that the above information are not under NDA ;-) Thanks in advance, -- Andrea SCIAN DAVE Embedded Systems ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: RFC: detect and manage power cut on MLC NAND 2015-03-13 20:31 ` Boris Brezillon 2015-03-13 23:51 ` Jeff Lauruhn (jlauruhn) @ 2015-03-16 9:01 ` Ricard Wanderlof 2015-03-16 17:27 ` Jeff Lauruhn (jlauruhn) 1 sibling, 1 reply; 59+ messages in thread From: Ricard Wanderlof @ 2015-03-16 9:01 UTC (permalink / raw) To: Jeff Lauruhn (jlauruhn); +Cc: mtd_mailinglist Hi Jeff, I have a question regarding MLC:s, probably not so much something we can do anything about, but I'm curious just the same: If I understand correctly, page pairing in MLC's means that of the two bits in a cell, one is allocated to one page and another one to a completely different page. This means (among other things) that rewriting one page may impact the other, paired, page. My question is: why is it done this way? Is it to distribute bit flips more evenly? An initial trivial allocation would otherwise be to put the paired bits in the same byte, for two reasons a) to avoid page-pairing issues, and b) because it simply would be easier to write both bits in a cell at the same time rather than at different times. Granted, without page pairing, any sort of failure or disturb in one bit cell would would require twice the amount of ECC as both bits would likely be corrupted, on the other hand, we'd avoid having data in one part of the flash be corrupted by operations in another part of the flash. /Ricard -- Ricard Wolf Wanderlöf ricardw(at)axis.com Axis Communications AB, Lund, Sweden www.axis.com Phone +46 46 272 2016 Fax +46 46 13 61 30 ^ permalink raw reply [flat|nested] 59+ messages in thread
* RE: RFC: detect and manage power cut on MLC NAND 2015-03-16 9:01 ` Ricard Wanderlof @ 2015-03-16 17:27 ` Jeff Lauruhn (jlauruhn) 0 siblings, 0 replies; 59+ messages in thread From: Jeff Lauruhn (jlauruhn) @ 2015-03-16 17:27 UTC (permalink / raw) To: Ricard Wanderlof; +Cc: mtd_mailinglist Ricard; I wish I could add some images, they would really help. But, what you call paired pages we call lower/upper pagers. Any yes, it's all about the distribution. I need to look in to other vendors but under program operations for Micron we have a requirement "Within a block, pages must be programmed sequentially from the least significant page address to the most significant page address (i.e. 0, 1, 2, 3, .). Programming pages out of order within a block is prohibited." This ensures that the lower page is programmed, before the upper page and that they are programmed together. This sequence ensures the best reliability. When we program the lower page it gets programmed into L0 or L1 state, when we program the upper page we either move L0 or L1 to its final location and we end up with L0, L1, L2, or L3. Programming takes longer because there are 4 levels on the same gate so tighter distribution is required. Power loss isn't as big an issue as it was in the past, most vendors are aware and have power loss considerations, but there are still vendors who take the risk. In the case of a power loss during a upper page program it's always a good idea to check the condition of the lower page. Jeff Lauruhn NAND Application Engineer Embedded Business Unit -----Original Message----- From: Ricard Wanderlof [mailto:ricard.wanderlof@axis.com] Sent: Monday, March 16, 2015 2:02 AM To: Jeff Lauruhn (jlauruhn) Cc: mtd_mailinglist Subject: Re: RFC: detect and manage power cut on MLC NAND Hi Jeff, I have a question regarding MLC:s, probably not so much something we can do anything about, but I'm curious just the same: If I understand correctly, page pairing in MLC's means that of the two bits in a cell, one is allocated to one page and another one to a completely different page. This means (among other things) that rewriting one page may impact the other, paired, page. My question is: why is it done this way? Is it to distribute bit flips more evenly? An initial trivial allocation would otherwise be to put the paired bits in the same byte, for two reasons a) to avoid page-pairing issues, and b) because it simply would be easier to write both bits in a cell at the same time rather than at different times. Granted, without page pairing, any sort of failure or disturb in one bit cell would would require twice the amount of ECC as both bits would likely be corrupted, on the other hand, we'd avoid having data in one part of the flash be corrupted by operations in another part of the flash. /Ricard -- Ricard Wolf Wanderlöf ricardw(at)axis.com Axis Communications AB, Lund, Sweden www.axis.com Phone +46 46 272 2016 Fax +46 46 13 61 30 ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: RFC: detect and manage power cut on MLC NAND 2015-03-12 22:57 ` Jeff Lauruhn (jlauruhn) 2015-03-13 20:31 ` Boris Brezillon @ 2015-03-14 10:03 ` Richard Weinberger 1 sibling, 0 replies; 59+ messages in thread From: Richard Weinberger @ 2015-03-14 10:03 UTC (permalink / raw) To: Jeff Lauruhn (jlauruhn) Cc: Andrea Scian, Boris Brezillon, mtd_mailinglist, dedekind1 Hi Jeff, Am 12.03.2015 um 23:57 schrieb Jeff Lauruhn (jlauruhn): > > > Jeff Lauruhn > NAND Application Engineer > Embedded Business Unit > Micron Technology, Inc > > > -----Original Message----- > From: Richard Weinberger [mailto:richard@nod.at] > Sent: Thursday, March 12, 2015 3:28 AM > To: Jeff Lauruhn (jlauruhn) > Cc: Andrea Scian; dedekind1@gmail.com; Boris Brezillon; mtd_mailinglist > Subject: Re: RFC: detect and manage power cut on MLC NAND > > Am 11.03.2015 um 22:16 schrieb Jeff Lauruhn (jlauruhn): >> Glad to help out. I train FAE's and customers on many aspects of NAND including MLC. > > UBI (and UBIFS) was designed with SLC NAND in mind, so far we know that we have to address the following constraints when we want UBI on MLC NAND: > > > 1. Avoid repeating bit patterns. This can be solved by scrambling. Boris did some great work in this area. > 2. Paired pages. We'll have choose pages we write to very carefully to not loss already written data in case of a power cut. > For MLC we store 4 bits in the same cell has > 3. Read disturb. Happens also on SLC but not that early. I'm working in this. > > 4. Data retention. i.e, blocks that have not been erased for a long time have to be re-erased. I'm working in this too. > > 5. Unstable bits (not MLC specific). > Two types. Data retention and Disturbs (read and program). Data retention (charge loss) tends to shift left, > 6. What did I miss? > > Jeff, what do you think? can you please say something on the "TODO" list? Did we miss something? Do you have kind of a design document? > Can you point us to some hard facts? I'm specially interested in numbers on read disturb and data retention. > I wish there were numbers, it would make my job a lot easier, but NAND doesn't work that way. Data retention is dependent on process node (35nm, 25nm, 20nm 15nm for example) P/E cycles and temperature. We generally specify our NAND using JEDC standards, x numbers of years at 55C with 10% cycling. If you apply our recommended ECC, then you will be able to store data and recover it after x numbers of years. But temperature, P/E, process size and usage have major effects on data retention so we recommend actively managing your NAND. This is what you do and what I find so interesting about your group. But I'm sure there are some rough numbers. Do we have to expect read disturb after say 100 reads? https://www.micron.com/~/media/Documents/Products/Presentation/flash_mem_summit_jcooke_inconvenient_truths_nand.pdf Are the numbers on page 20 still valid? > I can speak in generalities for now, and when I get more specifics I can predict and recommend solutions. > > Data Retention is characterized by a loss of charge. We program a bit from 1 to 0 (just the opposite of what people think). Over time the charge will leave the gate, this is normal NAND behavior. I say that the distribution of charged cell shifts left toward uncharged. Why is SLC better than MLC? First SLC was first and used older larger lithographies. You could store 10's of thousands of electrons on a 40nm gate and you only had two states L0 erased (0volts) or L1 programmed (1.5v for example). If you lose a few, it didn't make much difference and there was a lot of room between 0 and 1.5 volts. Newer processes are 20nm and MLC. With a smaller gate there are just a few hundred electrons and they need to be disturbed in one of 4 levels L0 (0 volts), L1(.5 volts), L2(1volts) or L3 (1.5 volts). Now adding or losing a few electrons can have a larger effect. We determine a programmed cell by measuring between the L0 and L1. If the levels have shifted, they can shift to the poi nt where you can't reliably tell the difference between L0 and L1. When there are more levels and they are closer together it makes it just that much more challenging. > > > Program/Read Disturb are characterized by charge gain or a shift right. The problem is the same. > > These affects are not instantaneous, they happen over long periods of time. Instead of trying to predict every case, we recommend actively managing NAND, and that's what your team does. Read the data an use ECC to see how much the data has changed. Create a threshold and when the data hits the threshold move the data to a fresh block. Thanks, //richard ^ permalink raw reply [flat|nested] 59+ messages in thread
* RE: RFC: detect and manage power cut on MLC NAND 2015-03-11 17:23 ` Jeff Lauruhn (jlauruhn) 2015-03-11 17:29 ` Richard Weinberger @ 2015-03-12 9:32 ` Ricard Wanderlof 1 sibling, 0 replies; 59+ messages in thread From: Ricard Wanderlof @ 2015-03-12 9:32 UTC (permalink / raw) To: Jeff Lauruhn (jlauruhn) Cc: Andrea Scian, dedekind1, mtd_mailinglist, Richard Weinberger On Wed, 11 Mar 2015, Jeff Lauruhn (jlauruhn) wrote: > IIUC can I summarize like the following > > 1) power cut is one of the big problem we know about MLC NAND flash with UBI > 2) we are unaware of other big issue with MLC NAND > > Unfortunately I don't really know the MTD/UBI/UBIFS internals but > (correct me if I'm wrong) to me solving the power cut issue only by > software (by using proper data structure, redundancy, NAND flash > operation jornal and so on) is pretty hard to implement inside the Linux > MTD stack. > > On the other hand, stuff like scrubbing (e.g. having a task like GC that > periodically scrub the required NAND erase block), paired page (e.g. > carefully selecting LEB to PEB mapping to avoid reading/writing data > into paired page in a wrong way) are still to be implemented but have, > somehow, less impact on the whole MTD stack. I may be going off in the wrong direction, but there seem to be two principal differences between SLC and MLC nand from a software point of view. 1. Write endurance is much lower, like 3000 erase/write cycles for MLC compared to 100000 for SLC. 2. The probability of bits changing 'under your feet' are much higher, being it because of the data retention simply being worse due to the underlying technology or because of page pairing. - This also includes bits that look ok but which aren't e.g. from the result of power loss during an erase operation, which might leave the bits in a state which looks erased, but which on subsequent use will not behave accordingly. Generally, the basic view of any memory from a software point of view is that if 'the data was written it will not change', which is true a lot of the time with SLC, given a couple of precautions (like ECC and never trusting an erased page unless we know the erase operation had completed properly), but much less so with MLC. I.e. with SLC, we can take our 'the data will not change' paradigm and patch on a few fixes for the relatively few cases where this is not true, but with MLC this will approach will tend to fall apart and we must rethink our approach if not from the bottom then a bit further down than was the case with SLC. /Ricard -- Ricard Wolf Wanderlöf ricardw(at)axis.com Axis Communications AB, Lund, Sweden www.axis.com Phone +46 46 272 2016 Fax +46 46 13 61 30 ^ permalink raw reply [flat|nested] 59+ messages in thread
* RE: RFC: detect and manage power cut on MLC NAND 2015-03-11 9:09 ` Richard Weinberger 2015-03-11 17:01 ` Andrea Scian @ 2015-03-23 4:08 ` Iwo Mergler 2015-03-23 21:15 ` Jeff Lauruhn (jlauruhn) 1 sibling, 1 reply; 59+ messages in thread From: Iwo Mergler @ 2015-03-23 4:08 UTC (permalink / raw) To: Richard Weinberger, dedekind1 Cc: Andrea Scian, Jeff Lauruhn (jlauruhn), mtd_mailinglist, Qi Wang Hi all, I probably don't know enough about the silicon implementation of MLC paired pages, but my feeling is that there should be a way to recover one of the pages if the paired write fails, at least in some cases. Something along the lines of using both bits to map to a single good one. 2 bit MLC stores 4 levels - 1.0, 0.7, 0.3, 0.0. Obviously, the actual voltage levels will be somewhat different, so take this as electrons on the floating gate: 1.0=minimum, 0.0 maximum. I imagine that there are two ways to achieve that - small step for low page and large step for high page, or the other way 'round. Assuming the first, the low page write would subtract 0.3 from the erased (1.0) cell if the bit is 0. That leaves the cell at either ~1.0 (1) or 0.7 (0). Lvl LH =========== 1.0 => 1u 0.7 => 0u Then, the high page write would subtract either nothing (1) or 0.7 (0): Lvl LH =========== 1.0 => 11 0.7 => 01 0.3 => 10 0.0 => 00 So the MLC decoder logic gets 3 priority encoded bits from the sense amplifiers: 111, 011, 001, 000. The decoder turns this into 11, 01, 10, 00. The process of writing a 0 to the high page, transitions low page 0-bits through 1 and back to 0, as the level moves down. Low page 1 bits transition from 1 through 0 and back to 1. So a half-completed high page 0-write can flip a low page bit both ways. We can detect an incorrect 0-1 transition in the low page, because it's marked by a 0 bit in the high page. We can't detect an incorrect 1-0 transition in the low page. So assuming a failed high page write, this is what we get: LH 11 = nothing happens, reads back as 11 Correct level for both. 01 = Level stays at 0.7, reads back as 01, Correct level for low page. 10 = Level between 1.0 and 0.3, reads back as 11, 01 or 10. 01 is wrong for low page, but can't be distinguished from 10. 00 = Level between 0.7 and 0.0, reads back as 01, 10, or 00. 10 is wrong for low page, but can be distinguished from 01. So, there are two bit combinations (50%) that have an undetectable failure, and this failure will happen about half the time, for a total of 25% unfixable failure rate. Not acceptable in the general case, but might be good enough for things like UBI EC & VID headers, if we ensure that the high page contains 1s at the offsets at which the low page stores the header. Now, on the other hand, if the low page write uses the larger step, there shouldn't be any paired page problem at all, since the high page write wouldn't cross the low page thresholds on the way: Lvl LH =========== 1.0 => 1u 0.3 => 0u Lvl LH =========== 1.0 => 11 0.7 => 10 0.3 => 01 0.0 => 00 Which makes me think I'm misunderstanding something. If not, why isn't his scheme used in the first place? What would happen if we reverse the paired page writing order? Jeff, Qi, is the mechanism I described here anywhere near reality? Best regards, Iwo ^ permalink raw reply [flat|nested] 59+ messages in thread
* RE: RFC: detect and manage power cut on MLC NAND 2015-03-23 4:08 ` Iwo Mergler @ 2015-03-23 21:15 ` Jeff Lauruhn (jlauruhn) 2015-03-24 1:17 ` Iwo Mergler 0 siblings, 1 reply; 59+ messages in thread From: Jeff Lauruhn (jlauruhn) @ 2015-03-23 21:15 UTC (permalink / raw) To: Iwo Mergler, Richard Weinberger, dedekind1 Cc: Andrea Scian, Qi Wang 王起 (qiwang), mtd_mailinglist This is a very simplified description, but actually it's more like this: First pass, program the lower page. If you the lower page is 1, do nothing. If the lower page is 0 subtract 0.7v to 0.3. Lower page is SLC like, two distributions spread apart by 0.7V. Lvl LH =========== 1.0 => 1u 0.3 => 0u Now, program the upper page. First, read lower page, if lower page is 1 and upper page is 1, do nothing (11). If lower page is 1 and upper page is 0, then subtract -0.3 and call that 01. Next if lower page is 0 and upper page is 1 do nothing and if lower page is 0 and upper page is 0 subtract 0.3v and call it 00. Notice that state of lower page is on right of 11, 01,10, 00. Lvl LH =========== 1.0 => 11 0.7 => 01 0.3 => 10 0.0 => 00 Now what happens if there's a power loss during the programming of the upper page? The upper page data will most likely be lost, and the lower page may be changed, but there's a good chance of recovery, because it will be in the range of SLC. It is highly recommended to read and refresh data after a power loss. Jeff Lauruhn NAND Application Engineer Embedded Business Unit -----Original Message----- From: Iwo Mergler [mailto:Iwo.Mergler@netcommwireless.com] Sent: Sunday, March 22, 2015 9:09 PM To: Richard Weinberger; dedekind1@gmail.com Cc: Andrea Scian; mtd_mailinglist; Jeff Lauruhn (jlauruhn); Qi Wang 王起 (qiwang) Subject: RE: RFC: detect and manage power cut on MLC NAND Hi all, I probably don't know enough about the silicon implementation of MLC paired pages, but my feeling is that there should be a way to recover one of the pages if the paired write fails, at least in some cases. Something along the lines of using both bits to map to a single good one. 2 bit MLC stores 4 levels - 1.0, 0.7, 0.3, 0.0. Obviously, the actual voltage levels will be somewhat different, so take this as electrons on the floating gate: 1.0=minimum, 0.0 maximum. I imagine that there are two ways to achieve that - small step for low page and large step for high page, or the other way 'round. Assuming the first, the low page write would subtract 0.3 from the erased (1.0) cell if the bit is 0. That leaves the cell at either ~1.0 (1) or 0.7 (0). Lvl LH =========== 1.0 => 1u 0.7 => 0u Then, the high page write would subtract either nothing (1) or 0.7 (0): Lvl LH =========== 1.0 => 11 0.7 => 01 0.3 => 10 0.0 => 00 So the MLC decoder logic gets 3 priority encoded bits from the sense amplifiers: 111, 011, 001, 000. The decoder turns this into 11, 01, 10, 00. The process of writing a 0 to the high page, transitions low page 0-bits through 1 and back to 0, as the level moves down. Low page 1 bits transition from 1 through 0 and back to 1. So a half-completed high page 0-write can flip a low page bit both ways. We can detect an incorrect 0-1 transition in the low page, because it's marked by a 0 bit in the high page. We can't detect an incorrect 1-0 transition in the low page. So assuming a failed high page write, this is what we get: LH 11 = nothing happens, reads back as 11 Correct level for both. 01 = Level stays at 0.7, reads back as 01, Correct level for low page. 10 = Level between 1.0 and 0.3, reads back as 11, 01 or 10. 01 is wrong for low page, but can't be distinguished from 10. 00 = Level between 0.7 and 0.0, reads back as 01, 10, or 00. 10 is wrong for low page, but can be distinguished from 01. So, there are two bit combinations (50%) that have an undetectable failure, and this failure will happen about half the time, for a total of 25% unfixable failure rate. Not acceptable in the general case, but might be good enough for things like UBI EC & VID headers, if we ensure that the high page contains 1s at the offsets at which the low page stores the header. Now, on the other hand, if the low page write uses the larger step, there shouldn't be any paired page problem at all, since the high page write wouldn't cross the low page thresholds on the way: Lvl LH =========== 1.0 => 1u 0.3 => 0u Lvl LH =========== 1.0 => 11 0.7 => 10 0.3 => 01 0.0 => 00 Which makes me think I'm misunderstanding something. If not, why isn't his scheme used in the first place? What would happen if we reverse the paired page writing order? Not recommended, we want pages programmed in sequence to mitigate disturbs and obtain the highest reliability. Jeff, Qi, is the mechanism I described here anywhere near reality? It's a simplified view, but fairly accurate. Best regards, Iwo ^ permalink raw reply [flat|nested] 59+ messages in thread
* RE: RFC: detect and manage power cut on MLC NAND 2015-03-23 21:15 ` Jeff Lauruhn (jlauruhn) @ 2015-03-24 1:17 ` Iwo Mergler 2015-03-24 16:50 ` Jeff Lauruhn (jlauruhn) 0 siblings, 1 reply; 59+ messages in thread From: Iwo Mergler @ 2015-03-24 1:17 UTC (permalink / raw) To: Jeff Lauruhn (jlauruhn), Richard Weinberger, dedekind1 Cc: Andrea Scian, Qi Wang 王起 (qiwang), mtd_mailinglist Hi Jeff, thanks for the info. That sounds like very good news to me - it appears that the paired page problem isn't as bad as I thought. > Lvl LH > =========== > 1.0 => 11 > 0.7 => 01 > 0.3 => 10 > 0.0 => 00 || Is this HL? This seems to match my second scenario (large step for low page) - we do not lose the low page when the high page write fails. Here is what I understood: The low page distribution centres around 1.0 and 0.3, with a spread of 0.7. Reading the low page as MLC at this point will classify everything above 0.5 as 1 and below as 0. So the distance of a '1' from the threshold is 0.5, and the distance of a '0' is 0.2. It's asymmetric, with SLC distance on one side and MLC on the other. The high page write (only when writing a '0'), moves the charge towards 0.0, by 0.3. So if the first page contains '1', the charge is lowered to 0.7, if the first page contains '0' the charge goes down to 0.0. At no time is the charge crossing the 0.5 threshold. An aborted high page write will have reduced the '1' distance by 0.0-0.3 or increased the '0' distance by 0.0-0.3. Why are we worried about the low page at all? The way I understand the gate charge changes, the low page situation can not be, at any point, worse than the normal programmed MLC state. Losing the high page isn't conceptually any worse than aborted writes on SLC. This has been called the "unstable bits problem". As far as I understand, UBI/UBIFS currently don't mitigate for aborted write. The worry here is that an aborted write (SLC or MLC) gets you a page where the 0s are weak - they may pass ECC on read now but fail next month. Or if the abort was early, weaken some 1s in an erased-looking page. Aborted erases are handled by UBI via the EC header write immediately after erase. An erased block without an EC header is erased again at boot time, to avoid partially erased blocks (weak 1s). Aborted writes are harder. UBI could deal with its own headers via sequence number and always rewrite the last block when it has UBI headers but no payload. Seems a bit wasteful, though. UBIFS also has an idea which few blocks may have received a power cut during the last write - I think around 5 candidates or so. Rewriting them all every mount time seems even more wasteful. Then of course, the rewrite can also be aborted, so UBI's last atomic LEB change probably also needs redoing at attach time. Jeff, what's the situation with aborted writes? Is this problem real? I understand that there are charge pumps on board the NAND chip - do they store sufficient energy to complete a page write? What about the typical system, which has, say, a 3.3V supply rail and a system reset firing when that falls below 3.0V or so. The processor core and NAND operate at 1.8V, so there are a few ms between the last possible NAND command and the breakdown of the NAND supply. Would that be safe from aborted writes? Best regards, Iwo ________________________________________ From: Jeff Lauruhn (jlauruhn) [jlauruhn@micron.com] Sent: Tuesday, 24 March 2015 8:15 AM To: Iwo Mergler; Richard Weinberger; dedekind1@gmail.com Cc: Andrea Scian; mtd_mailinglist; Qi Wang 王起 (qiwang) Subject: RE: RFC: detect and manage power cut on MLC NAND This is a very simplified description, but actually it's more like this: First pass, program the lower page. If you the lower page is 1, do nothing. If the lower page is 0 subtract 0.7v to 0.3. Lower page is SLC like, two distributions spread apart by 0.7V. Lvl LH =========== 1.0 => 1u 0.3 => 0u Now, program the upper page. First, read lower page, if lower page is 1 and upper page is 1, do nothing (11). If lower page is 1 and upper page is 0, then subtract -0.3 and call that 01. Next if lower page is 0 and upper page is 1 do nothing and if lower page is 0 and upper page is 0 subtract 0.3v and call it 00. Notice that state of lower page is on right of 11, 01,10, 00. Lvl LH =========== 1.0 => 11 0.7 => 01 0.3 => 10 0.0 => 00 Now what happens if there's a power loss during the programming of the upper page? The upper page data will most likely be lost, and the lower page may be changed, but there's a good chance of recovery, because it will be in the range of SLC. It is highly recommended to read and refresh data after a power loss. Jeff Lauruhn NAND Application Engineer Embedded Business Unit -----Original Message----- From: Iwo Mergler [mailto:Iwo.Mergler@netcommwireless.com] Sent: Sunday, March 22, 2015 9:09 PM To: Richard Weinberger; dedekind1@gmail.com Cc: Andrea Scian; mtd_mailinglist; Jeff Lauruhn (jlauruhn); Qi Wang 王起 (qiwang) Subject: RE: RFC: detect and manage power cut on MLC NAND Hi all, I probably don't know enough about the silicon implementation of MLC paired pages, but my feeling is that there should be a way to recover one of the pages if the paired write fails, at least in some cases. Something along the lines of using both bits to map to a single good one. 2 bit MLC stores 4 levels - 1.0, 0.7, 0.3, 0.0. Obviously, the actual voltage levels will be somewhat different, so take this as electrons on the floating gate: 1.0=minimum, 0.0 maximum. I imagine that there are two ways to achieve that - small step for low page and large step for high page, or the other way 'round. Assuming the first, the low page write would subtract 0.3 from the erased (1.0) cell if the bit is 0. That leaves the cell at either ~1.0 (1) or 0.7 (0). Lvl LH =========== 1.0 => 1u 0.7 => 0u Then, the high page write would subtract either nothing (1) or 0.7 (0): Lvl LH =========== 1.0 => 11 0.7 => 01 0.3 => 10 0.0 => 00 So the MLC decoder logic gets 3 priority encoded bits from the sense amplifiers: 111, 011, 001, 000. The decoder turns this into 11, 01, 10, 00. The process of writing a 0 to the high page, transitions low page 0-bits through 1 and back to 0, as the level moves down. Low page 1 bits transition from 1 through 0 and back to 1. So a half-completed high page 0-write can flip a low page bit both ways. We can detect an incorrect 0-1 transition in the low page, because it's marked by a 0 bit in the high page. We can't detect an incorrect 1-0 transition in the low page. So assuming a failed high page write, this is what we get: LH 11 = nothing happens, reads back as 11 Correct level for both. 01 = Level stays at 0.7, reads back as 01, Correct level for low page. 10 = Level between 1.0 and 0.3, reads back as 11, 01 or 10. 01 is wrong for low page, but can't be distinguished from 10. 00 = Level between 0.7 and 0.0, reads back as 01, 10, or 00. 10 is wrong for low page, but can be distinguished from 01. So, there are two bit combinations (50%) that have an undetectable failure, and this failure will happen about half the time, for a total of 25% unfixable failure rate. Not acceptable in the general case, but might be good enough for things like UBI EC & VID headers, if we ensure that the high page contains 1s at the offsets at which the low page stores the header. Now, on the other hand, if the low page write uses the larger step, there shouldn't be any paired page problem at all, since the high page write wouldn't cross the low page thresholds on the way: Lvl LH =========== 1.0 => 1u 0.3 => 0u Lvl LH =========== 1.0 => 11 0.7 => 10 0.3 => 01 0.0 => 00 Which makes me think I'm misunderstanding something. If not, why isn't his scheme used in the first place? What would happen if we reverse the paired page writing order? Not recommended, we want pages programmed in sequence to mitigate disturbs and obtain the highest reliability. Jeff, Qi, is the mechanism I described here anywhere near reality? It's a simplified view, but fairly accurate. Best regards, Iwo ^ permalink raw reply [flat|nested] 59+ messages in thread
* RE: RFC: detect and manage power cut on MLC NAND 2015-03-24 1:17 ` Iwo Mergler @ 2015-03-24 16:50 ` Jeff Lauruhn (jlauruhn) 2015-03-25 3:38 ` Iwo Mergler 0 siblings, 1 reply; 59+ messages in thread From: Jeff Lauruhn (jlauruhn) @ 2015-03-24 16:50 UTC (permalink / raw) To: Iwo Mergler, Richard Weinberger, dedekind1 Cc: Andrea Scian, Qi Wang 王起 (qiwang), mtd_mailinglist >From a simplified point of view you're right. In reality the program/erase recipes are actually quite advanced in order to get very tight distributions on a full page. The lower/upper page sequence is designed to provide the most reliable results and optimally we would like the lower and upper page programmed 100% of the time. There's been a lot of work done over the years to improve power loss and it's much better than in the past, but it's still something to be avoided on NAND. It's always best to check the integrity of the page after a power loss event. I want to make sure I understand your meaning of "aborted write". If by aborted write you are referring to stopping the command, not a power loss event, then yes the internal circuitry will complete the write. "What about the typical system, which has, say, a 3.3V supply rail and a system reset firing when that falls below 3.0V or so. The processor core and NAND operate at 1.8V, so there are a few ms between the last possible NAND command and the breakdown of the NAND supply." I have to be careful here because it's very dependent on the design and I really need to know the specifics to make a definitive statement, but a few ms should be enough time to protect the NAND. WP# is your friend here. Jeff Lauruhn NAND Application Engineer Embedded Business Unit Micron Technology, Inc -----Original Message----- From: Iwo Mergler [mailto:Iwo.Mergler@netcommwireless.com] Sent: Monday, March 23, 2015 6:18 PM To: Jeff Lauruhn (jlauruhn); Richard Weinberger; dedekind1@gmail.com Cc: Andrea Scian; mtd_mailinglist; Qi Wang 王起 (qiwang) Subject: RE: RFC: detect and manage power cut on MLC NAND Hi Jeff, thanks for the info. That sounds like very good news to me - it appears that the paired page problem isn't as bad as I thought. > Lvl LH > =========== > 1.0 => 11 > 0.7 => 01 > 0.3 => 10 > 0.0 => 00 || Is this HL? This seems to match my second scenario (large step for low page) - we do not lose the low page when the high page write fails. Here is what I understood: The low page distribution centres around 1.0 and 0.3, with a spread of 0.7. Reading the low page as MLC at this point will classify everything above 0.5 as 1 and below as 0. So the distance of a '1' from the threshold is 0.5, and the distance of a '0' is 0.2. It's asymmetric, with SLC distance on one side and MLC on the other. The high page write (only when writing a '0'), moves the charge towards 0.0, by 0.3. So if the first page contains '1', the charge is lowered to 0.7, if the first page contains '0' the charge goes down to 0.0. At no time is the charge crossing the 0.5 threshold. An aborted high page write will have reduced the '1' distance by 0.0-0.3 or increased the '0' distance by 0.0-0.3. Why are we worried about the low page at all? The way I understand the gate charge changes, the low page situation can not be, at any point, worse than the normal programmed MLC state. Losing the high page isn't conceptually any worse than aborted writes on SLC. This has been called the "unstable bits problem". As far as I understand, UBI/UBIFS currently don't mitigate for aborted write. The worry here is that an aborted write (SLC or MLC) gets you a page where the 0s are weak - they may pass ECC on read now but fail next month. Or if the abort was early, weaken some 1s in an erased-looking page. Aborted erases are handled by UBI via the EC header write immediately after erase. An erased block without an EC header is erased again at boot time, to avoid partially erased blocks (weak 1s). Aborted writes are harder. UBI could deal with its own headers via sequence number and always rewrite the last block when it has UBI headers but no payload. Seems a bit wasteful, though. UBIFS also has an idea which few blocks may have received a power cut during the last write - I think around 5 candidates or so. Rewriting them all every mount time seems even more wasteful. Then of course, the rewrite can also be aborted, so UBI's last atomic LEB change probably also needs redoing at attach time. Jeff, what's the situation with aborted writes? Is this problem real? I understand that there are charge pumps on board the NAND chip - do they store sufficient energy to complete a page write? What about the typical system, which has, say, a 3.3V supply rail and a system reset firing when that falls below 3.0V or so. The processor core and NAND operate at 1.8V, so there are a few ms between the last possible NAND command and the breakdown of the NAND supply. Would that be safe from aborted writes? Best regards, Iwo ________________________________________ From: Jeff Lauruhn (jlauruhn) [jlauruhn@micron.com] Sent: Tuesday, 24 March 2015 8:15 AM To: Iwo Mergler; Richard Weinberger; dedekind1@gmail.com Cc: Andrea Scian; mtd_mailinglist; Qi Wang 王起 (qiwang) Subject: RE: RFC: detect and manage power cut on MLC NAND This is a very simplified description, but actually it's more like this: First pass, program the lower page. If you the lower page is 1, do nothing. If the lower page is 0 subtract 0.7v to 0.3. Lower page is SLC like, two distributions spread apart by 0.7V. Lvl LH =========== 1.0 => 1u 0.3 => 0u Now, program the upper page. First, read lower page, if lower page is 1 and upper page is 1, do nothing (11). If lower page is 1 and upper page is 0, then subtract -0.3 and call that 01. Next if lower page is 0 and upper page is 1 do nothing and if lower page is 0 and upper page is 0 subtract 0.3v and call it 00. Notice that state of lower page is on right of 11, 01,10, 00. Lvl LH =========== 1.0 => 11 0.7 => 01 0.3 => 10 0.0 => 00 Now what happens if there's a power loss during the programming of the upper page? The upper page data will most likely be lost, and the lower page may be changed, but there's a good chance of recovery, because it will be in the range of SLC. It is highly recommended to read and refresh data after a power loss. Jeff Lauruhn NAND Application Engineer Embedded Business Unit -----Original Message----- From: Iwo Mergler [mailto:Iwo.Mergler@netcommwireless.com] Sent: Sunday, March 22, 2015 9:09 PM To: Richard Weinberger; dedekind1@gmail.com Cc: Andrea Scian; mtd_mailinglist; Jeff Lauruhn (jlauruhn); Qi Wang 王起 (qiwang) Subject: RE: RFC: detect and manage power cut on MLC NAND Hi all, I probably don't know enough about the silicon implementation of MLC paired pages, but my feeling is that there should be a way to recover one of the pages if the paired write fails, at least in some cases. Something along the lines of using both bits to map to a single good one. 2 bit MLC stores 4 levels - 1.0, 0.7, 0.3, 0.0. Obviously, the actual voltage levels will be somewhat different, so take this as electrons on the floating gate: 1.0=minimum, 0.0 maximum. I imagine that there are two ways to achieve that - small step for low page and large step for high page, or the other way 'round. Assuming the first, the low page write would subtract 0.3 from the erased (1.0) cell if the bit is 0. That leaves the cell at either ~1.0 (1) or 0.7 (0). Lvl LH =========== 1.0 => 1u 0.7 => 0u Then, the high page write would subtract either nothing (1) or 0.7 (0): Lvl LH =========== 1.0 => 11 0.7 => 01 0.3 => 10 0.0 => 00 So the MLC decoder logic gets 3 priority encoded bits from the sense amplifiers: 111, 011, 001, 000. The decoder turns this into 11, 01, 10, 00. The process of writing a 0 to the high page, transitions low page 0-bits through 1 and back to 0, as the level moves down. Low page 1 bits transition from 1 through 0 and back to 1. So a half-completed high page 0-write can flip a low page bit both ways. We can detect an incorrect 0-1 transition in the low page, because it's marked by a 0 bit in the high page. We can't detect an incorrect 1-0 transition in the low page. So assuming a failed high page write, this is what we get: LH 11 = nothing happens, reads back as 11 Correct level for both. 01 = Level stays at 0.7, reads back as 01, Correct level for low page. 10 = Level between 1.0 and 0.3, reads back as 11, 01 or 10. 01 is wrong for low page, but can't be distinguished from 10. 00 = Level between 0.7 and 0.0, reads back as 01, 10, or 00. 10 is wrong for low page, but can be distinguished from 01. So, there are two bit combinations (50%) that have an undetectable failure, and this failure will happen about half the time, for a total of 25% unfixable failure rate. Not acceptable in the general case, but might be good enough for things like UBI EC & VID headers, if we ensure that the high page contains 1s at the offsets at which the low page stores the header. Now, on the other hand, if the low page write uses the larger step, there shouldn't be any paired page problem at all, since the high page write wouldn't cross the low page thresholds on the way: Lvl LH =========== 1.0 => 1u 0.3 => 0u Lvl LH =========== 1.0 => 11 0.7 => 10 0.3 => 01 0.0 => 00 Which makes me think I'm misunderstanding something. If not, why isn't his scheme used in the first place? What would happen if we reverse the paired page writing order? Not recommended, we want pages programmed in sequence to mitigate disturbs and obtain the highest reliability. Jeff, Qi, is the mechanism I described here anywhere near reality? It's a simplified view, but fairly accurate. Best regards, Iwo ^ permalink raw reply [flat|nested] 59+ messages in thread
* RE: RFC: detect and manage power cut on MLC NAND 2015-03-24 16:50 ` Jeff Lauruhn (jlauruhn) @ 2015-03-25 3:38 ` Iwo Mergler 2015-03-25 8:33 ` Ricard Wanderlof 0 siblings, 1 reply; 59+ messages in thread From: Iwo Mergler @ 2015-03-25 3:38 UTC (permalink / raw) To: Jeff Lauruhn (jlauruhn), Richard Weinberger, dedekind1 Cc: Andrea Scian, Qi Wang 王起 (qiwang), mtd_mailinglist > From a simplified point of view you're right. In reality the > program/erase recipes are actually quite advanced in order to get > very tight distributions on a full page. The lower/upper page > sequence is designed to provide the most reliable results and > optimally we would like the lower and upper page programmed 100% of > the time. There's been a lot of work done over the years to improve > power loss and it's much better than in the past, but it's still > something to be avoided on NAND. It's always best to check the > integrity of the page after a power loss event. Is there any way to check the page integrity beyond ECC? I'm concerned that the power loss could yield an OK looking page, but with not so tight charge distribution. Maybe the hardware that can achieve tight distributions during programming, can be accessed to measure distribution of a programmed page? > I want to make sure I understand your meaning of "aborted write". If > by aborted write you are referring to stopping the command, not a > power loss event, then yes the internal circuitry will complete the > write. I did mean power loss. > I have to be careful here because it's very dependent on the design > and I really need to know the specifics to make a definitive > statement, but a few ms should be enough time to protect the NAND. > WP# is your friend here. The design is somewhat hypothetical - let's assume that we can guarantee the NAND supply for 10ms after system reset asserts. At reset time, the NAND controller will abort any command sequence in progress, so the final "program page" command will be sent either before the reset, or not at all. The command byte may be cut short on the bus. Would I gain anything by tying WP# to the system RESET# in this case? In other words, is there a way to electrically mangle the "program page" command on the bus (i.e. short timing or such) that could do something else than do nothing or start a page write? Would a simultaneous WP# help? Apologies for all the questions. I'm very happy to talk to someone at the coal face of modern NAND manufacturing. :-) Best regards, Iwo ________________________________________ From: Jeff Lauruhn (jlauruhn) [jlauruhn@micron.com] Sent: Wednesday, 25 March 2015 3:50 AM To: Iwo Mergler; Richard Weinberger; dedekind1@gmail.com Cc: Andrea Scian; mtd_mailinglist; Qi Wang 王起 (qiwang) Subject: RE: RFC: detect and manage power cut on MLC NAND >From a simplified point of view you're right. In reality the program/erase recipes are actually quite advanced in order to get very tight distributions on a full page. The lower/upper page sequence is designed to provide the most reliable results and optimally we would like the lower and upper page programmed 100% of the time. There's been a lot of work done over the years to improve power loss and it's much better than in the past, but it's still something to be avoided on NAND. It's always best to check the integrity of the page after a power loss event. I want to make sure I understand your meaning of "aborted write". If by aborted write you are referring to stopping the command, not a power loss event, then yes the internal circuitry will complete the write. "What about the typical system, which has, say, a 3.3V supply rail and a system reset firing when that falls below 3.0V or so. The processor core and NAND operate at 1.8V, so there are a few ms between the last possible NAND command and the breakdown of the NAND supply." I have to be careful here because it's very dependent on the design and I really need to know the specifics to make a definitive statement, but a few ms should be enough time to protect the NAND. WP# is your friend here. Jeff Lauruhn NAND Application Engineer Embedded Business Unit Micron Technology, Inc -----Original Message----- From: Iwo Mergler [mailto:Iwo.Mergler@netcommwireless.com] Sent: Monday, March 23, 2015 6:18 PM To: Jeff Lauruhn (jlauruhn); Richard Weinberger; dedekind1@gmail.com Cc: Andrea Scian; mtd_mailinglist; Qi Wang 王起 (qiwang) Subject: RE: RFC: detect and manage power cut on MLC NAND Hi Jeff, thanks for the info. That sounds like very good news to me - it appears that the paired page problem isn't as bad as I thought. > Lvl LH > =========== > 1.0 => 11 > 0.7 => 01 > 0.3 => 10 > 0.0 => 00 || Is this HL? This seems to match my second scenario (large step for low page) - we do not lose the low page when the high page write fails. Here is what I understood: The low page distribution centres around 1.0 and 0.3, with a spread of 0.7. Reading the low page as MLC at this point will classify everything above 0.5 as 1 and below as 0. So the distance of a '1' from the threshold is 0.5, and the distance of a '0' is 0.2. It's asymmetric, with SLC distance on one side and MLC on the other. The high page write (only when writing a '0'), moves the charge towards 0.0, by 0.3. So if the first page contains '1', the charge is lowered to 0.7, if the first page contains '0' the charge goes down to 0.0. At no time is the charge crossing the 0.5 threshold. An aborted high page write will have reduced the '1' distance by 0.0-0.3 or increased the '0' distance by 0.0-0.3. Why are we worried about the low page at all? The way I understand the gate charge changes, the low page situation can not be, at any point, worse than the normal programmed MLC state. Losing the high page isn't conceptually any worse than aborted writes on SLC. This has been called the "unstable bits problem". As far as I understand, UBI/UBIFS currently don't mitigate for aborted write. The worry here is that an aborted write (SLC or MLC) gets you a page where the 0s are weak - they may pass ECC on read now but fail next month. Or if the abort was early, weaken some 1s in an erased-looking page. Aborted erases are handled by UBI via the EC header write immediately after erase. An erased block without an EC header is erased again at boot time, to avoid partially erased blocks (weak 1s). Aborted writes are harder. UBI could deal with its own headers via sequence number and always rewrite the last block when it has UBI headers but no payload. Seems a bit wasteful, though. UBIFS also has an idea which few blocks may have received a power cut during the last write - I think around 5 candidates or so. Rewriting them all every mount time seems even more wasteful. Then of course, the rewrite can also be aborted, so UBI's last atomic LEB change probably also needs redoing at attach time. Jeff, what's the situation with aborted writes? Is this problem real? I understand that there are charge pumps on board the NAND chip - do they store sufficient energy to complete a page write? What about the typical system, which has, say, a 3.3V supply rail and a system reset firing when that falls below 3.0V or so. The processor core and NAND operate at 1.8V, so there are a few ms between the last possible NAND command and the breakdown of the NAND supply. Would that be safe from aborted writes? Best regards, Iwo ________________________________________ From: Jeff Lauruhn (jlauruhn) [jlauruhn@micron.com] Sent: Tuesday, 24 March 2015 8:15 AM To: Iwo Mergler; Richard Weinberger; dedekind1@gmail.com Cc: Andrea Scian; mtd_mailinglist; Qi Wang 王起 (qiwang) Subject: RE: RFC: detect and manage power cut on MLC NAND This is a very simplified description, but actually it's more like this: First pass, program the lower page. If you the lower page is 1, do nothing. If the lower page is 0 subtract 0.7v to 0.3. Lower page is SLC like, two distributions spread apart by 0.7V. Lvl LH =========== 1.0 => 1u 0.3 => 0u Now, program the upper page. First, read lower page, if lower page is 1 and upper page is 1, do nothing (11). If lower page is 1 and upper page is 0, then subtract -0.3 and call that 01. Next if lower page is 0 and upper page is 1 do nothing and if lower page is 0 and upper page is 0 subtract 0.3v and call it 00. Notice that state of lower page is on right of 11, 01,10, 00. Lvl LH =========== 1.0 => 11 0.7 => 01 0.3 => 10 0.0 => 00 Now what happens if there's a power loss during the programming of the upper page? The upper page data will most likely be lost, and the lower page may be changed, but there's a good chance of recovery, because it will be in the range of SLC. It is highly recommended to read and refresh data after a power loss. Jeff Lauruhn NAND Application Engineer Embedded Business Unit -----Original Message----- From: Iwo Mergler [mailto:Iwo.Mergler@netcommwireless.com] Sent: Sunday, March 22, 2015 9:09 PM To: Richard Weinberger; dedekind1@gmail.com Cc: Andrea Scian; mtd_mailinglist; Jeff Lauruhn (jlauruhn); Qi Wang 王起 (qiwang) Subject: RE: RFC: detect and manage power cut on MLC NAND Hi all, I probably don't know enough about the silicon implementation of MLC paired pages, but my feeling is that there should be a way to recover one of the pages if the paired write fails, at least in some cases. Something along the lines of using both bits to map to a single good one. 2 bit MLC stores 4 levels - 1.0, 0.7, 0.3, 0.0. Obviously, the actual voltage levels will be somewhat different, so take this as electrons on the floating gate: 1.0=minimum, 0.0 maximum. I imagine that there are two ways to achieve that - small step for low page and large step for high page, or the other way 'round. Assuming the first, the low page write would subtract 0.3 from the erased (1.0) cell if the bit is 0. That leaves the cell at either ~1.0 (1) or 0.7 (0). Lvl LH =========== 1.0 => 1u 0.7 => 0u Then, the high page write would subtract either nothing (1) or 0.7 (0): Lvl LH =========== 1.0 => 11 0.7 => 01 0.3 => 10 0.0 => 00 So the MLC decoder logic gets 3 priority encoded bits from the sense amplifiers: 111, 011, 001, 000. The decoder turns this into 11, 01, 10, 00. The process of writing a 0 to the high page, transitions low page 0-bits through 1 and back to 0, as the level moves down. Low page 1 bits transition from 1 through 0 and back to 1. So a half-completed high page 0-write can flip a low page bit both ways. We can detect an incorrect 0-1 transition in the low page, because it's marked by a 0 bit in the high page. We can't detect an incorrect 1-0 transition in the low page. So assuming a failed high page write, this is what we get: LH 11 = nothing happens, reads back as 11 Correct level for both. 01 = Level stays at 0.7, reads back as 01, Correct level for low page. 10 = Level between 1.0 and 0.3, reads back as 11, 01 or 10. 01 is wrong for low page, but can't be distinguished from 10. 00 = Level between 0.7 and 0.0, reads back as 01, 10, or 00. 10 is wrong for low page, but can be distinguished from 01. So, there are two bit combinations (50%) that have an undetectable failure, and this failure will happen about half the time, for a total of 25% unfixable failure rate. Not acceptable in the general case, but might be good enough for things like UBI EC & VID headers, if we ensure that the high page contains 1s at the offsets at which the low page stores the header. Now, on the other hand, if the low page write uses the larger step, there shouldn't be any paired page problem at all, since the high page write wouldn't cross the low page thresholds on the way: Lvl LH =========== 1.0 => 1u 0.3 => 0u Lvl LH =========== 1.0 => 11 0.7 => 10 0.3 => 01 0.0 => 00 Which makes me think I'm misunderstanding something. If not, why isn't his scheme used in the first place? What would happen if we reverse the paired page writing order? Not recommended, we want pages programmed in sequence to mitigate disturbs and obtain the highest reliability. Jeff, Qi, is the mechanism I described here anywhere near reality? It's a simplified view, but fairly accurate. Best regards, Iwo ^ permalink raw reply [flat|nested] 59+ messages in thread
* RE: RFC: detect and manage power cut on MLC NAND 2015-03-25 3:38 ` Iwo Mergler @ 2015-03-25 8:33 ` Ricard Wanderlof 2015-03-26 1:57 ` Jeff Lauruhn (jlauruhn) 0 siblings, 1 reply; 59+ messages in thread From: Ricard Wanderlof @ 2015-03-25 8:33 UTC (permalink / raw) To: Iwo Mergler Cc: Jeff Lauruhn (jlauruhn), dedekind1, Richard Weinberger, Andrea Scian, Qi Wang 王起 (qiwang), mtd_mailinglist On Wed, 25 Mar 2015, Iwo Mergler wrote: > > From a simplified point of view you're right. In reality the > > program/erase recipes are actually quite advanced in order to get > > very tight distributions on a full page. The lower/upper page > > sequence is designed to provide the most reliable results and > > optimally we would like the lower and upper page programmed 100% of > > the time. There's been a lot of work done over the years to improve > > power loss and it's much better than in the past, but it's still > > something to be avoided on NAND. It's always best to check the > > integrity of the page after a power loss event. > > Is there any way to check the page integrity beyond ECC? > > I'm concerned that the power loss could yield an OK looking > page, but with not so tight charge distribution. > > Maybe the hardware that can achieve tight distributions during > programming, can be accessed to measure distribution of a > programmed page? What would be interesting from a software perspective would be if in some special mode one could read the read the memory cells and get an analog value with several bits of resolution, alowing the software to make an assessment as to how "good" the bits are. This would be in contrast to the normal, high speed, read mode. But perhaps matters are not that simple, either there is no such value to be had (but as I understand it in certain MLC flashes it is possible to shift the read thresholds, thus one could accomplish this by successive approximation. Sure, that means that one could do it entirely in software using existing devices, but it is a rather cumbersome process however), or there are other factors that govern the read thresholds which are not known outside the chip (or rather, outside the manufacturers lab!). > > I have to be careful here because it's very dependent on the design > > and I really need to know the specifics to make a definitive > > statement, but a few ms should be enough time to protect the NAND. > > WP# is your friend here. > > The design is somewhat hypothetical - let's assume that we can > guarantee the NAND supply for 10ms after system reset asserts. > > At reset time, the NAND controller will abort any command sequence in > progress, so the final "program page" command will be sent either before > the reset, or not at all. The command byte may be cut short on the bus. It would seem to me that the only thing really needed to guarantee that writes (or erase operations) are not cut short by power loss, is as Iwo says that the system design is such that when power loss occurs, there is enough power to maintain valid supply voltage levels to allow the NAND to complete operations in the worst case, after system reset is asserted. Admittedly we don't always have the luxury of well-designed hardware, but having clear design rules for the hardware guys would help a long way in future designs. > I'm very happy to talk to someone at the coal face of modern NAND > manufacturing. :-) Agreed, I think we're very many that appreciate Jeff's contributions on the list, me included. NAND data sheets are often not so forthcoming, and there ends up being a lot of speculation about how things actually work, so it's really nice to have someone with real knowledge to discuss this with. /Ricard -- Ricard Wolf Wanderlöf ricardw(at)axis.com Axis Communications AB, Lund, Sweden www.axis.com Phone +46 46 272 2016 Fax +46 46 13 61 30 ^ permalink raw reply [flat|nested] 59+ messages in thread
* RE: RFC: detect and manage power cut on MLC NAND 2015-03-25 8:33 ` Ricard Wanderlof @ 2015-03-26 1:57 ` Jeff Lauruhn (jlauruhn) 2015-03-26 8:55 ` Ricard Wanderlof 0 siblings, 1 reply; 59+ messages in thread From: Jeff Lauruhn (jlauruhn) @ 2015-03-26 1:57 UTC (permalink / raw) To: Ricard Wanderlof, Iwo Mergler Cc: Andrea Scian, Richard Weinberger, Qi Wang 王起 (qiwang), mtd_mailinglist, dedekind1 Jeff Lauruhn NAND Application Engineer Embedded Business Unit Micron Technology, Inc -----Original Message----- From: Ricard Wanderlof [mailto:ricard.wanderlof@axis.com] Sent: Wednesday, March 25, 2015 1:33 AM To: Iwo Mergler Cc: Jeff Lauruhn (jlauruhn); Richard Weinberger; dedekind1@gmail.com; Andrea Scian; Qi Wang 王起 (qiwang); mtd_mailinglist Subject: RE: RFC: detect and manage power cut on MLC NAND On Wed, 25 Mar 2015, Iwo Mergler wrote: > > From a simplified point of view you're right. In reality the > > program/erase recipes are actually quite advanced in order to get > > very tight distributions on a full page. The lower/upper page > > sequence is designed to provide the most reliable results and > > optimally we would like the lower and upper page programmed 100% of > > the time. There's been a lot of work done over the years to improve > > power loss and it's much better than in the past, but it's still > > something to be avoided on NAND. It's always best to check the > > integrity of the page after a power loss event. > > Is there any way to check the page integrity beyond ECC? > > I'm concerned that the power loss could yield an OK looking page, but > with not so tight charge distribution. > > Maybe the hardware that can achieve tight distributions during > programming, can be accessed to measure distribution of a programmed > page? What would be interesting from a software perspective would be if in some special mode one could read the read the memory cells and get an analog value with several bits of resolution, alowing the software to make an assessment as to how "good" the bits are. This would be in contrast to the normal, high speed, read mode. But perhaps matters are not that simple, either there is no such value to be had (but as I understand it in certain MLC flashes it is possible to shift the read thresholds, thus one could accomplish this by successive approximation. Sure, that means that one could do it entirely in software using existing devices, but it is a rather cumbersome process however), or there are other factors that govern the read thresholds which are not known outside the chip (or rather, outside the manufacturers lab!). No special analog modes on production devices, but we are moving in the direction of giving more control to the end user. As lithography goes down and bits per cell goes up we are adding we are trying to come up with manageable ways to recover data. There's no analog read out on the roadmap, but new features like read retry, which generally assumes charge loss and allows the end user to try different read reference voltages and other read offset features are on the road map. Let me explain the read process a bit. When we program and erase too, we set a target value L0 and L1 in the case of SLC and we get a distribution around that those values. But when we read we apply a voltage to the gate of the cell we intend to read that is between L0 and L1, call it Vread, if the cell is erased, Vread is greater than the voltage threshold Vt of L0 and the cell will conduct and we will sense a current flow between the Drain and Source and the sense circuit registers a 1 for that cell. If the cell is programmed, Vread will not be high enough to overcome the Vt of the cell and we will sense no current flow between the drain and source and the sense circuit registers a 0 for that cell. The sense circuit is very simple because there needs to be 2K of them so we can sense the whole page simultaneously. The type of circuitry required to measure an analog value would make the die huge. If it was possible, it would already be designed in. Instead of measuring the cell voltage, it was easier to allow the end user to move Vread to maybe compensate for the shift in distribution. > > I have to be careful here because it's very dependent on the design > > and I really need to know the specifics to make a definitive > > statement, but a few ms should be enough time to protect the NAND. > > WP# is your friend here. > > The design is somewhat hypothetical - let's assume that we can > guarantee the NAND supply for 10ms after system reset asserts. > > At reset time, the NAND controller will abort any command sequence in > progress, so the final "program page" command will be sent either > before the reset, or not at all. The command byte may be cut short on the bus. It would seem to me that the only thing really needed to guarantee that writes (or erase operations) are not cut short by power loss, is as Iwo says that the system design is such that when power loss occurs, there is enough power to maintain valid supply voltage levels to allow the NAND to complete operations in the worst case, after system reset is asserted. Admittedly we don't always have the luxury of well-designed hardware, but having clear design rules for the hardware guys would help a long way in future designs. > I'm very happy to talk to someone at the coal face of modern NAND > manufacturing. :-) Agreed, I think we're very many that appreciate Jeff's contributions on the list, me included. NAND data sheets are often not so forthcoming, and there ends up being a lot of speculation about how things actually work, so it's really nice to have someone with real knowledge to discuss this with. /Ricard -- Ricard Wolf Wanderlöf ricardw(at)axis.com Axis Communications AB, Lund, Sweden www.axis.com Phone +46 46 272 2016 Fax +46 46 13 61 30 ^ permalink raw reply [flat|nested] 59+ messages in thread
* RE: RFC: detect and manage power cut on MLC NAND 2015-03-26 1:57 ` Jeff Lauruhn (jlauruhn) @ 2015-03-26 8:55 ` Ricard Wanderlof 0 siblings, 0 replies; 59+ messages in thread From: Ricard Wanderlof @ 2015-03-26 8:55 UTC (permalink / raw) To: Jeff Lauruhn (jlauruhn) Cc: Iwo Mergler, Qi Wang 王起 (qiwang), dedekind1, Richard Weinberger, Andrea Scian, Ricard Wanderlöf, mtd_mailinglist On Thu, 26 Mar 2015, Jeff Lauruhn (jlauruhn) wrote: > Let me explain the read process a bit. When we program and erase too, > we set a target value L0 and L1 in the case of SLC and we get a > ... > registers a 0 for that cell. The sense circuit is very simple because > there needs to be 2K of them so we can sense the whole page > simultaneously. The type of circuitry required to measure an analog > value would make the die huge. If it was possible, it would already be > designed in. > Instead of measuring the cell voltage, it was easier to allow the end > user to move Vread to maybe compensate for the shift in distribution. Makes perfect sense. Thanks for the response! /Ricard -- Ricard Wolf Wanderlöf ricardw(at)axis.com Axis Communications AB, Lund, Sweden www.axis.com Phone +46 46 272 2016 Fax +46 46 13 61 30 ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: RFC: detect and manage power cut on MLC NAND 2015-03-10 11:57 RFC: detect and manage power cut on MLC NAND Andrea Scian 2015-03-10 12:51 ` Richard Weinberger @ 2015-03-11 7:21 ` Artem Bityutskiy 1 sibling, 0 replies; 59+ messages in thread From: Artem Bityutskiy @ 2015-03-11 7:21 UTC (permalink / raw) To: Andrea Scian, Boris Brezillon; +Cc: mtd_mailinglist On Tue, 2015-03-10 at 12:57 +0100, Andrea Scian wrote: > Dear All, > > for testing and research purpose, I'm working with an Embedded Linux > product which mounts an MLC NAND. > > I know that one of the trouble with MLC flash on Linux with UBI/UBIFS is > the power cut management (ref. > http://www.linux-mtd.infradead.org/doc/ubifs.html#L_unstable_bits) > > IIUC the solution inside MTD/UBI/UBIFS is not so simple and thus not so > quick to implement. > > I'm wondering if anyone of you already think about managing power cut in > a different way: many embedded/industrial products already have some > big/huge bypass capacitor that are able to keep the system alive for > hundreds of milliseconds (this is useful for power supply > micro-interruptions), which are commonly enough to complete the latest > NAND operation. > Most of the has a GPIO/IRQ that tells whenever such a power cut happens, > so the kernel can take some actions with it (e.g., in our case, complete > the last NAND operation and prevent from starting newer operation, which > may corrupt the NAND pages). > > WDYT about this? > If it sounds reasonable is there any suggestion where to place such a code? Ccing Boris, he has been looking into this area lately. Artem. ^ permalink raw reply [flat|nested] 59+ messages in thread
* RE: RFC: detect and manage power cut on MLC NAND
@ 2015-03-12 10:31 Andrea Marson - DAVE Embedded Systems
0 siblings, 0 replies; 59+ messages in thread
From: Andrea Marson - DAVE Embedded Systems @ 2015-03-12 10:31 UTC (permalink / raw)
To: jlauruhn
Cc: Andrea Scian - DAVE Embedded Systems, dedekind1, linux-mtd, richard
Hi Jeff,
thank you for your availability. I'd really love to discuss in more
detail these matters.
About Write Protect (WP) pin, IIUC it must respect several constraints.
For example it must not be transitioned while NAND is busy. What if it
is set low while NAND is ready - that is it can accept new commands -
but an erase operation is in progress? Does the erase operation complete
anyway?
Andrea Marson
*DAVE Embedded Systems*
> Power loss is actually very complex. The Write Protect (WP) pin was added =
> to NAND help lock the NAND when a power loss event is detected. I have ext=
> ensive information on NAND and would be happy to discuss. =20
>
>
> Jeff Lauruhn
> NAND Application Engineer
> Embedded Business Unit
> Micron Technology, Inc
>
>
> -----Original Message-----
> From: linux-mtd [mailto:linux-mtd-bounces@lists.infradead.org] On Behalf Of=
> Andrea Scian
> Sent: Wednesday, March 11, 2015 10:01 AM
> To: Richard Weinberger; dedekind1@gmail.com
> Cc: mtd_mailinglist
> Subject: Re: RFC: detect and manage power cut on MLC NAND
>
>
> Hi all,
>
> and thanks for you feedback.
> You can find my comments below.
>
> Il 11/03/2015 10:09, Richard Weinberger ha scritto:
>> Am 11.03.2015 um 10:05 schrieb Artem Bityutskiy:
>>> On Wed, 2015-03-11 at 09:57 +0100, Richard Weinberger wrote:
>>>> Hi!
>>>>
>>>> Am 11.03.2015 um 08:20 schrieb Artem Bityutskiy:
>>>>> On Tue, 2015-03-10 at 13:51 +0100, Richard Weinberger wrote:
>>>>>>> WDYT about this?
>>>>>>> If it sounds reasonable is there any suggestion where to place such =
> a code?
>>>>>> Customers often use DYI uninterruptible power supplies using capacito=
> rs.
>>>>>> But managing a power cut is the least problem you have with MLC NAND.
>>>>> Why is it the least problem, what is the hardest one? I thought=20
>>>>> this one is the hardest.
>>>> IMHO the hardest ones are the problems we don't know yet as NAND=20
>>>> vendors are not really chatty about the MLC constraints.
>>>> We don't know much about data retention for example. At least we=20
>>>> have not much hard facts. Most of our knowledge is hearsay.
>>> Well, but from the problems we know paired pages seems to be the=20
>>> biggest one. E.g., what do we do if VID header gets corrupted because=20
>>> of an interrupted write to the page paired with the VID header page?=20
>>> Sounds like a hard problem to me.
>> It is a hard problem. But at least we know about it.
>>
> IIUC can I summarize like the following
>
> 1) power cut is one of the big problem we know about MLC NAND flash with UB=
> I
> 2) we are unaware of other big issue with MLC NAND
>
> Unfortunately I don't really know the MTD/UBI/UBIFS internals but (correct =
> me if I'm wrong) to me solving the power cut issue only by software (by usi=
> ng proper data structure, redundancy, NAND flash operation jornal and so on=
> ) is pretty hard to implement inside the Linux MTD stack.
>
> On the other hand, stuff like scrubbing (e.g. having a task like GC that pe=
> riodically scrub the required NAND erase block), paired page (e.g.
> carefully selecting LEB to PEB mapping to avoid reading/writing data into p=
> aired page in a wrong way) are still to be implemented but have, somehow, l=
> ess impact on the whole MTD stack.
>
> For sure I'm missing some other MLC NAND issue, but the above are the one t=
> hat I'm aware of.
>
> WDYT?
>
> Best Regards,
>
> --=20
>
> Andrea SCIAN
>
> DAVE Embedded Systems
>
>
>
^ permalink raw reply [flat|nested] 59+ messages in thread
[parent not found: <mailman.37176.1426610573.22890.linux-mtd@lists.infradead.org>]
end of thread, other threads:[~2015-03-26 8:55 UTC | newest] Thread overview: 59+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2015-03-10 11:57 RFC: detect and manage power cut on MLC NAND Andrea Scian 2015-03-10 12:51 ` Richard Weinberger 2015-03-11 7:20 ` Artem Bityutskiy 2015-03-11 8:57 ` Richard Weinberger 2015-03-11 9:05 ` Artem Bityutskiy 2015-03-11 9:09 ` Richard Weinberger 2015-03-11 17:01 ` Andrea Scian 2015-03-11 17:23 ` Jeff Lauruhn (jlauruhn) 2015-03-11 17:29 ` Richard Weinberger 2015-03-11 21:16 ` Jeff Lauruhn (jlauruhn) 2015-03-12 10:28 ` Richard Weinberger 2015-03-12 22:57 ` Jeff Lauruhn (jlauruhn) 2015-03-13 20:31 ` Boris Brezillon 2015-03-13 23:51 ` Jeff Lauruhn (jlauruhn) 2015-03-14 9:46 ` Andrea Marson - DAVE Embedded Systems 2015-03-16 16:02 ` Jeff Lauruhn (jlauruhn) 2015-03-17 8:00 ` Andrea Scian 2015-03-14 10:32 ` Boris Brezillon 2015-03-16 21:11 ` Jeff Lauruhn (jlauruhn) 2015-03-17 9:30 ` Andrea Scian 2015-03-17 10:02 ` Boris Brezillon 2015-03-17 16:42 ` Jeff Lauruhn (jlauruhn) 2015-03-18 8:45 ` RFC: detect and manage power cut on MLC NAND (linux-mtd Digest, Vol 144, Issue 70) Andrea Marson 2015-03-18 9:07 ` Boris Brezillon 2015-03-18 9:56 ` Andrea Marson 2015-03-18 10:03 ` Boris Brezillon 2015-03-18 12:07 ` Richard Weinberger 2015-03-18 17:11 ` Jeff Lauruhn (jlauruhn) 2015-03-18 16:12 ` Jeff Lauruhn (jlauruhn) 2015-03-19 8:47 ` RFC: detect and manage power cut on MLC NAND Andrea Marson 2015-03-19 9:12 ` Boris Brezillon 2015-03-19 17:45 ` Jeff Lauruhn (jlauruhn) 2015-03-20 0:25 ` Iwo Mergler 2015-03-20 3:38 ` nick 2015-03-20 5:40 ` Iwo Mergler 2015-03-20 8:26 ` Boris Brezillon 2015-03-20 17:15 ` Nick Krause 2015-03-22 23:45 ` Iwo Mergler 2015-03-23 2:18 ` Iwo Mergler 2015-03-23 7:06 ` Artem Bityutskiy 2015-03-23 19:05 ` Boris Brezillon 2015-03-24 7:05 ` Artem Bityutskiy 2015-03-19 18:00 ` Jeff Lauruhn (jlauruhn) 2015-03-20 8:07 ` Andrea Marson 2015-03-17 17:04 ` Jeff Lauruhn (jlauruhn) 2015-03-16 9:01 ` Ricard Wanderlof 2015-03-16 17:27 ` Jeff Lauruhn (jlauruhn) 2015-03-14 10:03 ` Richard Weinberger 2015-03-12 9:32 ` Ricard Wanderlof 2015-03-23 4:08 ` Iwo Mergler 2015-03-23 21:15 ` Jeff Lauruhn (jlauruhn) 2015-03-24 1:17 ` Iwo Mergler 2015-03-24 16:50 ` Jeff Lauruhn (jlauruhn) 2015-03-25 3:38 ` Iwo Mergler 2015-03-25 8:33 ` Ricard Wanderlof 2015-03-26 1:57 ` Jeff Lauruhn (jlauruhn) 2015-03-26 8:55 ` Ricard Wanderlof 2015-03-11 7:21 ` Artem Bityutskiy -- strict thread matches above, loose matches on Subject: below -- 2015-03-12 10:31 Andrea Marson - DAVE Embedded Systems [not found] <mailman.37176.1426610573.22890.linux-mtd@lists.infradead.org>
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.