* Re: Regarding UBI scalability [not found] <7824366.270131233573513030.JavaMail.weblogic@epml10> @ 2009-02-04 9:47 ` Adrian Hunter 2009-02-05 9:40 ` Brijesh Singh 0 siblings, 1 reply; 25+ messages in thread From: Adrian Hunter @ 2009-02-04 9:47 UTC (permalink / raw) To: brij.singh; +Cc: AMIT KUMARSHARMA, linux-mtd ext BRIJESH SINGH wrote: > Hi, > > Artem Bityutskiy wrote: >> On Mon, 2009-02-02 at 13:07 +0200, Adrian Hunter wrote: >>> I would suggest an intermediate step. Create UBI2 which is >>> similar to UBI but stores eraseblock information in one place, >>> instead of at the beginning of each eraseblock. Such an approach >>> might be OK up to as much as 64GiB, and would probably perform >>> better than a fully scalable version. >>> >>> Then look at creating UBI3, which is fully scalable. >> Yes, I assume UBI2 should store mapping/erasure information in separate >> tables, not in each eraseblock. So we should get rid of eraseblock >> headers. > > Yes that is what I meant. You could probably make do with as little as > 12 bytes per eraseblock so a 64GiB flash with 512KiB eraseblock size > would need 1536KiB table, which could be read in a second or two, so > mount time is OK. > > I have an idea for how to update the table relatively efficiently if you > are interested. > > I am definitely interested. But apart from on flash headers, I am also interested in memory consumption scaling. > UBIFS solved this problem quite interestingly. Can something similar be borrowed for UBI? > > Thanks and Regards, > Brijesh I would leave the memory consumption issue for UBI3. Do you have a target memory consumption in mind? ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Regarding UBI scalability 2009-02-04 9:47 ` Regarding UBI scalability Adrian Hunter @ 2009-02-05 9:40 ` Brijesh Singh 2009-02-08 9:48 ` Corentin Chary 0 siblings, 1 reply; 25+ messages in thread From: Brijesh Singh @ 2009-02-05 9:40 UTC (permalink / raw) To: Adrian Hunter; +Cc: AMIT KUMARSHARMA, linux-mtd On Wed, Feb 4, 2009 at 3:17 PM, Adrian Hunter <ext-adrian.hunter@nokia.com> wrote: > ext BRIJESH SINGH wrote: >> Hi, >> >> Artem Bityutskiy wrote: >>> On Mon, 2009-02-02 at 13:07 +0200, Adrian Hunter wrote: >>>> I would suggest an intermediate step. Create UBI2 which is >>>> similar to UBI but stores eraseblock information in one place, >>>> instead of at the beginning of each eraseblock. Such an approach >>>> might be OK up to as much as 64GiB, and would probably perform >>>> better than a fully scalable version. >>>> >>>> Then look at creating UBI3, which is fully scalable. >>> Yes, I assume UBI2 should store mapping/erasure information in separate >>> tables, not in each eraseblock. So we should get rid of eraseblock >>> headers. >> >> Yes that is what I meant. You could probably make do with as little as >> 12 bytes per eraseblock so a 64GiB flash with 512KiB eraseblock size >> would need 1536KiB table, which could be read in a second or two, so >> mount time is OK. >> >> I have an idea for how to update the table relatively efficiently if you >> are interested. >> >> I am definitely interested. But apart from on flash headers, I am also interested in memory consumption scaling. >> UBIFS solved this problem quite interestingly. Can something similar be borrowed for UBI? >> >> Thanks and Regards, >> Brijesh > > I would leave the memory consumption issue for UBI3. > > Do you have a target memory consumption in mind? Yes,this needs to be done.I agree with the approach, better to keep it for UBI3. ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Regarding UBI scalability 2009-02-05 9:40 ` Brijesh Singh @ 2009-02-08 9:48 ` Corentin Chary 2009-02-08 10:31 ` Kyungmin Park 0 siblings, 1 reply; 25+ messages in thread From: Corentin Chary @ 2009-02-08 9:48 UTC (permalink / raw) To: Brijesh Singh; +Cc: AMIT KUMARSHARMA, linux-mtd, Adrian Hunter On Thu, Feb 5, 2009 at 10:40 AM, Brijesh Singh <brij.singh@samsung.com> wrote: > On Wed, Feb 4, 2009 at 3:17 PM, Adrian Hunter > <ext-adrian.hunter@nokia.com> wrote: >> ext BRIJESH SINGH wrote: >>> Hi, >>> >>> Artem Bityutskiy wrote: >>>> On Mon, 2009-02-02 at 13:07 +0200, Adrian Hunter wrote: >>>>> I would suggest an intermediate step. Create UBI2 which is >>>>> similar to UBI but stores eraseblock information in one place, >>>>> instead of at the beginning of each eraseblock. Such an approach >>>>> might be OK up to as much as 64GiB, and would probably perform >>>>> better than a fully scalable version. >>>>> >>>>> Then look at creating UBI3, which is fully scalable. >>>> Yes, I assume UBI2 should store mapping/erasure information in separate >>>> tables, not in each eraseblock. So we should get rid of eraseblock >>>> headers. >>> >>> Yes that is what I meant. You could probably make do with as little as >>> 12 bytes per eraseblock so a 64GiB flash with 512KiB eraseblock size >>> would need 1536KiB table, which could be read in a second or two, so >>> mount time is OK. >>> >>> I have an idea for how to update the table relatively efficiently if you >>> are interested. >>> >>> I am definitely interested. But apart from on flash headers, I am also interested in memory consumption scaling. >>> UBIFS solved this problem quite interestingly. Can something similar be borrowed for UBI? >>> >>> Thanks and Regards, >>> Brijesh >> >> I would leave the memory consumption issue for UBI3. As you are working on UBI2, have you a git tree or something ? I may have some time to help :) Thanks -- Corentin Chary http://xf.iksaif.net ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Regarding UBI scalability 2009-02-08 9:48 ` Corentin Chary @ 2009-02-08 10:31 ` Kyungmin Park 2009-02-09 8:46 ` Artem Bityutskiy 0 siblings, 1 reply; 25+ messages in thread From: Kyungmin Park @ 2009-02-08 10:31 UTC (permalink / raw) To: Corentin Chary; +Cc: AMIT KUMARSHARMA, linux-mtd, Adrian Hunter, Brijesh Singh Hi, On Sun, Feb 8, 2009 at 6:48 PM, Corentin Chary <corentin.chary@gmail.com> wrote: > On Thu, Feb 5, 2009 at 10:40 AM, Brijesh Singh <brij.singh@samsung.com> wrote: >> On Wed, Feb 4, 2009 at 3:17 PM, Adrian Hunter >> <ext-adrian.hunter@nokia.com> wrote: >>> ext BRIJESH SINGH wrote: >>>> Hi, >>>> >>>> Artem Bityutskiy wrote: >>>>> On Mon, 2009-02-02 at 13:07 +0200, Adrian Hunter wrote: >>>>>> I would suggest an intermediate step. Create UBI2 which is >>>>>> similar to UBI but stores eraseblock information in one place, >>>>>> instead of at the beginning of each eraseblock. Such an approach >>>>>> might be OK up to as much as 64GiB, and would probably perform >>>>>> better than a fully scalable version. >>>>>> >>>>>> Then look at creating UBI3, which is fully scalable. >>>>> Yes, I assume UBI2 should store mapping/erasure information in separate >>>>> tables, not in each eraseblock. So we should get rid of eraseblock >>>>> headers. >>>> >>>> Yes that is what I meant. You could probably make do with as little as >>>> 12 bytes per eraseblock so a 64GiB flash with 512KiB eraseblock size >>>> would need 1536KiB table, which could be read in a second or two, so >>>> mount time is OK. >>>> >>>> I have an idea for how to update the table relatively efficiently if you >>>> are interested. >>>> >>>> I am definitely interested. But apart from on flash headers, I am also interested in memory consumption scaling. >>>> UBIFS solved this problem quite interestingly. Can something similar be borrowed for UBI? >>>> >>>> Thanks and Regards, >>>> Brijesh >>> >>> I would leave the memory consumption issue for UBI3. > > As you are working on UBI2, have you a git tree or something ? I may > have some time to help :) > Thanks > One more requested feature. How about to scan ubi image and bad block table at simultaneously? Now there's two full device scan, bbt and ubi at booting time. As internal LEA mapping layout, how about to add bbt layout? It's also helpful to MLC since MLC doesn't have read oob so it calls page read command to build bbt. As you know it takes long time. I'm not sure if internal LEA maping is supported, it's meaningful work or not. How do you think? Thank you, Kyungmin Park ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Regarding UBI scalability 2009-02-08 10:31 ` Kyungmin Park @ 2009-02-09 8:46 ` Artem Bityutskiy 0 siblings, 0 replies; 25+ messages in thread From: Artem Bityutskiy @ 2009-02-09 8:46 UTC (permalink / raw) To: Kyungmin Park Cc: AMIT KUMARSHARMA, linux-mtd, Adrian Hunter, Corentin Chary, Brijesh Singh On Sun, 2009-02-08 at 19:31 +0900, Kyungmin Park wrote: > One more requested feature. > How about to scan ubi image and bad block table at simultaneously? > Now there's two full device scan, bbt and ubi at booting time. > As internal LEA mapping layout, how about to add bbt layout? > > It's also helpful to MLC since MLC doesn't have read oob so it calls > page read command to build bbt. > As you know it takes long time. > > I'm not sure if internal LEA maping is supported, it's meaningful work or not. > > How do you think? Well, doing scanning twice is not very good indeed. I guess what you could do is to make MTD avoid scanning and building the bad block table at all and build it lazily. But then you need 2 bits per eraseblock, not just one. IOW, sure, it is possible to optimize this. -- Best regards, Artem Bityutskiy (Битюцкий Артём) ^ permalink raw reply [flat|nested] 25+ messages in thread
* Regarding UBI scalability
@ 2009-01-30 12:45 BRIJESH SINGH
2009-02-02 9:31 ` Artem Bityutskiy
0 siblings, 1 reply; 25+ messages in thread
From: BRIJESH SINGH @ 2009-01-30 12:45 UTC (permalink / raw)
To: linux-mtd
Hi,
I have gone through UBI documentation which clearly mentions that UBI doesn't scale very well.(Boot time and Memory consumption increases with size of device.)This is very similar to JFFS2 scalability issues. Log, journal(like UBIFS LPT) combination might surely help on this issue.
I would like to work on this enhancement with the community. Before that I just wanted to check if any one is already working on it.
As this a well known issue, I am sure somebody will have some design suggestions. Any design suggestions will be appreciated.
Thanks,
Brijesh
^ permalink raw reply [flat|nested] 25+ messages in thread* Re: Regarding UBI scalability 2009-01-30 12:45 BRIJESH SINGH @ 2009-02-02 9:31 ` Artem Bityutskiy 2009-02-02 10:17 ` Amit Kumar Sharma 0 siblings, 1 reply; 25+ messages in thread From: Artem Bityutskiy @ 2009-02-02 9:31 UTC (permalink / raw) To: brij.singh; +Cc: linux-mtd On Fri, 2009-01-30 at 12:45 +0000, BRIJESH SINGH wrote: > Hi, > I have gone through UBI documentation which clearly mentions that > UBI doesn't scale very well.(Boot time and Memory consumption > increases with size of device.)This is very similar to JFFS2 > scalability issues. Log, journal(like UBIFS LPT) combination might > surely help on this issue. Right. > I would like to work on this enhancement with the community. Before > that I just wanted to check if any one is already working on it. No, I do not think so. > As this a well known issue, I am sure somebody will have some design > suggestions. Any design suggestions will be appreciated. You could look at the old JFFS3 design document: http://www.linux-mtd.infradead.org/doc/JFFS3design.pdf at the "The superblock" section, which describes the idea of how to have a data structure which refers everything else in the media: journal, root nodes of the trees, etc. An then you should think about how to store the EBA table on flash. And how to store the erase counters on the flash. Many ideas may be borrowed from UBIFS in this area. I may help you in terms of design suggestions / review, and some code review. But I do not really have time to seriously work on this. -- Best regards, Artem Bityutskiy (Битюцкий Артём) ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Regarding UBI scalability 2009-02-02 9:31 ` Artem Bityutskiy @ 2009-02-02 10:17 ` Amit Kumar Sharma 2009-02-02 11:07 ` Adrian Hunter 0 siblings, 1 reply; 25+ messages in thread From: Amit Kumar Sharma @ 2009-02-02 10:17 UTC (permalink / raw) To: dedekind, brij.singh; +Cc: linux-mtd Hi ----- Original Message ----- From: "Artem Bityutskiy" <dedekind@infradead.org> To: <brij.singh@samsung.com> Cc: <linux-mtd@lists.infradead.org> Sent: Monday, February 02, 2009 3:01 PM Subject: Re: Regarding UBI scalability > On Fri, 2009-01-30 at 12:45 +0000, BRIJESH SINGH wrote: >> Hi, >> I have gone through UBI documentation which clearly >> mentions that >> UBI doesn't scale very well.(Boot time and Memory >> consumption >> increases with size of device.)This is very similar to >> JFFS2 >> scalability issues. Log, journal(like UBIFS LPT) >> combination might >> surely help on this issue. > > Right > >> I would like to work on this enhancement with the >> community. Before >> that I just wanted to check if any one is already working >> on it. > > No, I do not think so. > >> As this a well known issue, I am sure somebody will >> have some design >> suggestions. Any design suggestions will be appreciated. > > You could look at the old JFFS3 design document: > > http://www.linux-mtd.infradead.org/doc/JFFS3design.pdf > > at the "The superblock" section, which describes the idea > of how to have > a data structure which refers everything else in the > media: journal, > root nodes of the trees, etc. > > An then you should think about how to store the EBA table > on flash. And > how to store the erase counters on the flash. Many ideas > may be borrowed > from UBIFS in this area. > > I may help you in terms of design suggestions / review, > and some code > review. But I do not really have time to seriously work on > this. > > -- > Best regards, > Artem Bityutskiy (???????? ?????) Thanks for your comments Brijesh will check all your suggestion and we will discuss final design with you for your comments. Thanks Amit > > > ______________________________________________________ > Linux MTD discussion mailing list > http://lists.infradead.org/mailman/listinfo/linux-mtd/ > ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Regarding UBI scalability 2009-02-02 10:17 ` Amit Kumar Sharma @ 2009-02-02 11:07 ` Adrian Hunter 2009-02-02 10:57 ` Artem Bityutskiy 0 siblings, 1 reply; 25+ messages in thread From: Adrian Hunter @ 2009-02-02 11:07 UTC (permalink / raw) To: Amit Kumar Sharma; +Cc: linux-mtd, brij.singh Amit Kumar Sharma wrote: > Hi > ----- Original Message ----- > From: "Artem Bityutskiy" <dedekind@infradead.org> > To: <brij.singh@samsung.com> > Cc: <linux-mtd@lists.infradead.org> > Sent: Monday, February 02, 2009 3:01 PM > Subject: Re: Regarding UBI scalability > > >> On Fri, 2009-01-30 at 12:45 +0000, BRIJESH SINGH wrote: >>> Hi, >>> I have gone through UBI documentation which clearly >>> mentions that >>> UBI doesn't scale very well.(Boot time and Memory >>> consumption >>> increases with size of device.)This is very similar to >>> JFFS2 >>> scalability issues. Log, journal(like UBIFS LPT) >>> combination might >>> surely help on this issue. >> Right >> >>> I would like to work on this enhancement with the >>> community. Before >>> that I just wanted to check if any one is already working >>> on it. >> No, I do not think so. >> >>> As this a well known issue, I am sure somebody will >>> have some design >>> suggestions. Any design suggestions will be appreciated. >> You could look at the old JFFS3 design document: >> >> http://www.linux-mtd.infradead.org/doc/JFFS3design.pdf >> >> at the "The superblock" section, which describes the idea >> of how to have >> a data structure which refers everything else in the >> media: journal, >> root nodes of the trees, etc. >> >> An then you should think about how to store the EBA table >> on flash. And >> how to store the erase counters on the flash. Many ideas >> may be borrowed >> from UBIFS in this area. >> >> I may help you in terms of design suggestions / review, >> and some code >> review. But I do not really have time to seriously work on >> this. >> >> -- >> Best regards, >> Artem Bityutskiy (???????? ?????) > > Thanks for your comments Brijesh will check all your > suggestion and we will discuss final design with you for > your comments. > > Thanks > Amit I would suggest an intermediate step. Create UBI2 which is similar to UBI but stores eraseblock information in one place, instead of at the beginning of each eraseblock. Such an approach might be OK up to as much as 64GiB, and would probably perform better than a fully scalable version. Then look at creating UBI3, which is fully scalable. ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Regarding UBI scalability 2009-02-02 11:07 ` Adrian Hunter @ 2009-02-02 10:57 ` Artem Bityutskiy 2009-02-02 11:18 ` Adrian Hunter 0 siblings, 1 reply; 25+ messages in thread From: Artem Bityutskiy @ 2009-02-02 10:57 UTC (permalink / raw) To: Adrian Hunter; +Cc: Amit Kumar Sharma, linux-mtd, brij.singh On Mon, 2009-02-02 at 13:07 +0200, Adrian Hunter wrote: > I would suggest an intermediate step. Create UBI2 which is > similar to UBI but stores eraseblock information in one place, > instead of at the beginning of each eraseblock. Such an approach > might be OK up to as much as 64GiB, and would probably perform > better than a fully scalable version. > > Then look at creating UBI3, which is fully scalable. Yes, I assume UBI2 should store mapping/erasure information in separate tables, not in each eraseblock. So we should get rid of eraseblock headers. -- Best regards, Artem Bityutskiy (Битюцкий Артём) ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Regarding UBI scalability 2009-02-02 10:57 ` Artem Bityutskiy @ 2009-02-02 11:18 ` Adrian Hunter 2009-02-02 20:07 ` Jamie Lokier 2009-02-02 23:44 ` Corentin Chary 0 siblings, 2 replies; 25+ messages in thread From: Adrian Hunter @ 2009-02-02 11:18 UTC (permalink / raw) To: dedekind; +Cc: Amit Kumar Sharma, linux-mtd, brij.singh Artem Bityutskiy wrote: > On Mon, 2009-02-02 at 13:07 +0200, Adrian Hunter wrote: >> I would suggest an intermediate step. Create UBI2 which is >> similar to UBI but stores eraseblock information in one place, >> instead of at the beginning of each eraseblock. Such an approach >> might be OK up to as much as 64GiB, and would probably perform >> better than a fully scalable version. >> >> Then look at creating UBI3, which is fully scalable. > > Yes, I assume UBI2 should store mapping/erasure information in separate > tables, not in each eraseblock. So we should get rid of eraseblock > headers. Yes that is what I meant. You could probably make do with as little as 12 bytes per eraseblock so a 64GiB flash with 512KiB eraseblock size would need 1536KiB table, which could be read in a second or two, so mount time is OK. I have an idea for how to update the table relatively efficiently if you are interested. ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Regarding UBI scalability 2009-02-02 11:18 ` Adrian Hunter @ 2009-02-02 20:07 ` Jamie Lokier 2009-02-02 23:44 ` Corentin Chary 1 sibling, 0 replies; 25+ messages in thread From: Jamie Lokier @ 2009-02-02 20:07 UTC (permalink / raw) To: Adrian Hunter; +Cc: Amit Kumar Sharma, linux-mtd, brij.singh You might also look at the method LogFS uses. LogFS claims to be scalable and very fast for large flashes already. -- Jamie ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Regarding UBI scalability 2009-02-02 11:18 ` Adrian Hunter 2009-02-02 20:07 ` Jamie Lokier @ 2009-02-02 23:44 ` Corentin Chary 2009-02-03 10:35 ` Brijesh Singh 2009-02-03 10:46 ` Artem Bityutskiy 1 sibling, 2 replies; 25+ messages in thread From: Corentin Chary @ 2009-02-02 23:44 UTC (permalink / raw) To: Adrian Hunter; +Cc: Amit Kumar Sharma, linux-mtd, brij.singh On Mon, Feb 2, 2009 at 12:18 PM, Adrian Hunter <ext-adrian.hunter@nokia.com> wrote: > Artem Bityutskiy wrote: >> On Mon, 2009-02-02 at 13:07 +0200, Adrian Hunter wrote: >>> I would suggest an intermediate step. Create UBI2 which is >>> similar to UBI but stores eraseblock information in one place, >>> instead of at the beginning of each eraseblock. Such an approach >>> might be OK up to as much as 64GiB, and would probably perform >>> better than a fully scalable version. >>> >>> Then look at creating UBI3, which is fully scalable. >> >> Yes, I assume UBI2 should store mapping/erasure information in separate >> tables, not in each eraseblock. So we should get rid of eraseblock >> headers. > > Yes that is what I meant. You could probably make do with as little as > 12 bytes per eraseblock so a 64GiB flash with 512KiB eraseblock size > would need 1536KiB table, which could be read in a second or two, so > mount time is OK. Hi, I was wondering how it is possible to get atomic operations using such tables ? The first thing that come to my mind is TFAT (two tables, one for operation in progress, the other considered as always "valid"). But I'm not sure it' the good way to do such a thing. -- Corentin Chary http://xf.iksaif.net ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Regarding UBI scalability 2009-02-02 23:44 ` Corentin Chary @ 2009-02-03 10:35 ` Brijesh Singh 2009-02-03 10:48 ` Artem Bityutskiy 2009-02-04 9:29 ` Adrian Hunter 2009-02-03 10:46 ` Artem Bityutskiy 1 sibling, 2 replies; 25+ messages in thread From: Brijesh Singh @ 2009-02-03 10:35 UTC (permalink / raw) To: Corentin Chary; +Cc: Amit Kumar Sharma, linux-mtd, Adrian Hunter On Tue, Feb 3, 2009 at 5:14 AM, Corentin Chary <corentin.chary@gmail.com> wrote: > On Mon, Feb 2, 2009 at 12:18 PM, Adrian Hunter > <ext-adrian.hunter@nokia.com> wrote: >> Artem Bityutskiy wrote: >>> On Mon, 2009-02-02 at 13:07 +0200, Adrian Hunter wrote: >>>> I would suggest an intermediate step. Create UBI2 which is >>>> similar to UBI but stores eraseblock information in one place, >>>> instead of at the beginning of each eraseblock. Such an approach >>>> might be OK up to as much as 64GiB, and would probably perform >>>> better than a fully scalable version. >>>> >>>> Then look at creating UBI3, which is fully scalable. >>> >>> Yes, I assume UBI2 should store mapping/erasure information in separate >>> tables, not in each eraseblock. So we should get rid of eraseblock >>> headers. >> >> Yes that is what I meant. You could probably make do with as little as >> 12 bytes per eraseblock so a 64GiB flash with 512KiB eraseblock size >> would need 1536KiB table, which could be read in a second or two, so >> mount time is OK. Adrian,to my understanding, this is minimum info needed per physical erase block... Erase count -8 bytes Lnum -4 bytes Volume ID -4 byte(Can make it 1 byte for now as vol limit=128) Header CRC -4 bytes(Only for static volumes.) i.e. Minimum 20 bytes. Is there any other way to make it 12? And how to update the table relatively efficiently? > Hi, > I was wondering how it is possible to get atomic operations using such tables ? May be keeping a commit block and journal will help.(*Like UBIFS design) Thanks, Brijesh ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Regarding UBI scalability 2009-02-03 10:35 ` Brijesh Singh @ 2009-02-03 10:48 ` Artem Bityutskiy 2009-02-03 11:27 ` Enrico Scholz 2009-02-04 9:29 ` Adrian Hunter 1 sibling, 1 reply; 25+ messages in thread From: Artem Bityutskiy @ 2009-02-03 10:48 UTC (permalink / raw) To: Brijesh Singh; +Cc: Amit Kumar Sharma, linux-mtd, Adrian Hunter, Corentin Chary On Tue, 2009-02-03 at 16:05 +0530, Brijesh Singh wrote: > On Tue, Feb 3, 2009 at 5:14 AM, Corentin Chary <corentin.chary@gmail.com> wrote: > > On Mon, Feb 2, 2009 at 12:18 PM, Adrian Hunter > > <ext-adrian.hunter@nokia.com> wrote: > >> Artem Bityutskiy wrote: > >>> On Mon, 2009-02-02 at 13:07 +0200, Adrian Hunter wrote: > >>>> I would suggest an intermediate step. Create UBI2 which is > >>>> similar to UBI but stores eraseblock information in one place, > >>>> instead of at the beginning of each eraseblock. Such an approach > >>>> might be OK up to as much as 64GiB, and would probably perform > >>>> better than a fully scalable version. > >>>> > >>>> Then look at creating UBI3, which is fully scalable. > >>> > >>> Yes, I assume UBI2 should store mapping/erasure information in separate > >>> tables, not in each eraseblock. So we should get rid of eraseblock > >>> headers. > >> > >> Yes that is what I meant. You could probably make do with as little as > >> 12 bytes per eraseblock so a 64GiB flash with 512KiB eraseblock size > >> would need 1536KiB table, which could be read in a second or two, so > >> mount time is OK. > > Adrian,to my understanding, this is minimum info needed per physical > erase block... > > Erase count -8 bytes > Lnum -4 bytes > Volume ID -4 byte(Can make it 1 byte for now as > vol limit=128) > Header CRC -4 bytes(Only for static volumes.) > i.e. Minimum 20 bytes. Is there any other way to make it 12? > And how to update the table relatively efficiently? For UBI2 I would just forget about static volumes support. They are not so necessary, and a rare user needs them. This would simplify things. -- Best regards, Artem Bityutskiy (Битюцкий Артём) ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Regarding UBI scalability 2009-02-03 10:48 ` Artem Bityutskiy @ 2009-02-03 11:27 ` Enrico Scholz 2009-02-04 7:41 ` Artem Bityutskiy 0 siblings, 1 reply; 25+ messages in thread From: Enrico Scholz @ 2009-02-03 11:27 UTC (permalink / raw) To: linux-mtd Artem Bityutskiy <dedekind@infradead.org> writes: > For UBI2 I would just forget about static volumes support. They > are not so necessary, and a rare user needs them I store my kernel, the 2nd level bootloader and bootenv in such volumes. Having the number of used bytes available without introducing special storage formats simplifies things in the initial bootloader and when writing images into the volumes. This is nothing which can not be changed, but static volumes are not a rare needs imo. Enrico ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Regarding UBI scalability 2009-02-03 11:27 ` Enrico Scholz @ 2009-02-04 7:41 ` Artem Bityutskiy 0 siblings, 0 replies; 25+ messages in thread From: Artem Bityutskiy @ 2009-02-04 7:41 UTC (permalink / raw) To: Enrico Scholz; +Cc: linux-mtd On Tue, 2009-02-03 at 12:27 +0100, Enrico Scholz wrote: > Artem Bityutskiy <dedekind@infradead.org> writes: > > > For UBI2 I would just forget about static volumes support. They > > are not so necessary, and a rare user needs them > > I store my kernel, the 2nd level bootloader and bootenv in such > volumes. Having the number of used bytes available without > introducing special storage formats simplifies things in the > initial bootloader and when writing images into the volumes. > > This is nothing which can not be changed, but static volumes are > not a rare needs imo. Ok, thanks for this information. Probably static volumes are not that useless as I thought. -- Best regards, Artem Bityutskiy (Битюцкий Артём) ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Regarding UBI scalability 2009-02-03 10:35 ` Brijesh Singh 2009-02-03 10:48 ` Artem Bityutskiy @ 2009-02-04 9:29 ` Adrian Hunter 2009-02-05 9:37 ` Brijesh Singh 1 sibling, 1 reply; 25+ messages in thread From: Adrian Hunter @ 2009-02-04 9:29 UTC (permalink / raw) To: Brijesh Singh; +Cc: Amit Kumar Sharma, linux-mtd, Corentin Chary Brijesh Singh wrote: > On Tue, Feb 3, 2009 at 5:14 AM, Corentin Chary <corentin.chary@gmail.com> wrote: >> On Mon, Feb 2, 2009 at 12:18 PM, Adrian Hunter >> <ext-adrian.hunter@nokia.com> wrote: >>> Artem Bityutskiy wrote: >>>> On Mon, 2009-02-02 at 13:07 +0200, Adrian Hunter wrote: >>>>> I would suggest an intermediate step. Create UBI2 which is >>>>> similar to UBI but stores eraseblock information in one place, >>>>> instead of at the beginning of each eraseblock. Such an approach >>>>> might be OK up to as much as 64GiB, and would probably perform >>>>> better than a fully scalable version. >>>>> >>>>> Then look at creating UBI3, which is fully scalable. >>>> Yes, I assume UBI2 should store mapping/erasure information in separate >>>> tables, not in each eraseblock. So we should get rid of eraseblock >>>> headers. >>> Yes that is what I meant. You could probably make do with as little as >>> 12 bytes per eraseblock so a 64GiB flash with 512KiB eraseblock size >>> would need 1536KiB table, which could be read in a second or two, so >>> mount time is OK. > > Adrian,to my understanding, this is minimum info needed per physical > erase block... > > Erase count -8 bytes > Lnum -4 bytes > Volume ID -4 byte(Can make it 1 byte for now as > vol limit=128) > Header CRC -4 bytes(Only for static volumes.) > i.e. Minimum 20 bytes. Is there any other way to make it 12? 8 bytes for erase counter is many more than needed i.e. SLC NAND 100,000 - 1,000,000 max. erase cycles MLC NAND 10000 - 10,000 max. erase cycles Leave out the header CRC for dynamic volumes Combine lnum and volume id. Maybe go so far as to pack the information into bit fields like LPT does. i.e. if the flash media only has a million eraseblocks then only 20 bits are needed for lnum. Maybe it is MLC NAND, so say 15 bits for erase counter. That leaves 5 bites for volume id, and fits into 5 bytes for dynmaic volumes. > And how to update the table relatively efficiently? Each update to UBI typically requires writing a whole NAND page. That page can be filled with unchanged information from the first eraseblock of UBI information. Hence after enough updates the first eraseblock will end up containing only out-of-date information and can then be unmapped. Wash, rinse, repeat. > >> Hi, >> I was wondering how it is possible to get atomic operations using such tables ? > May be keeping a commit block and journal will help.(*Like UBIFS design) > > Thanks, > Brijesh > ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Regarding UBI scalability 2009-02-04 9:29 ` Adrian Hunter @ 2009-02-05 9:37 ` Brijesh Singh 2009-02-05 11:17 ` Adrian Hunter 0 siblings, 1 reply; 25+ messages in thread From: Brijesh Singh @ 2009-02-05 9:37 UTC (permalink / raw) To: Adrian Hunter; +Cc: linux-mtd On Wed, Feb 4, 2009 at 2:59 PM, Adrian Hunter <ext-adrian.hunter@nokia.com> wrote: > Brijesh Singh wrote: >> >> On Tue, Feb 3, 2009 at 5:14 AM, Corentin Chary <corentin.chary@gmail.com> >> wrote: >>> >>> On Mon, Feb 2, 2009 at 12:18 PM, Adrian Hunter >>> <ext-adrian.hunter@nokia.com> wrote: >>>> >>>> Artem Bityutskiy wrote: >>>>> >>>>> On Mon, 2009-02-02 at 13:07 +0200, Adrian Hunter wrote: >>>>>> >>>>>> I would suggest an intermediate step. Create UBI2 which is >>>>>> similar to UBI but stores eraseblock information in one place, >>>>>> instead of at the beginning of each eraseblock. Such an approach >>>>>> might be OK up to as much as 64GiB, and would probably perform >>>>>> better than a fully scalable version. >>>>>> >>>>>> Then look at creating UBI3, which is fully scalable. >>>>> >>>>> Yes, I assume UBI2 should store mapping/erasure information in separate >>>>> tables, not in each eraseblock. So we should get rid of eraseblock >>>>> headers. >>>> >>>> Yes that is what I meant. You could probably make do with as little as >>>> 12 bytes per eraseblock so a 64GiB flash with 512KiB eraseblock size >>>> would need 1536KiB table, which could be read in a second or two, so >>>> mount time is OK. >> >> Adrian,to my understanding, this is minimum info needed per physical >> erase block... >> >> Erase count -8 bytes >> Lnum -4 bytes >> Volume ID -4 byte(Can make it 1 byte for now as >> vol limit=128) >> Header CRC -4 bytes(Only for static volumes.) >> i.e. Minimum 20 bytes. Is there any other way to make it 12? > > 8 bytes for erase counter is many more than needed i.e. > SLC NAND 100,000 - 1,000,000 max. erase cycles > MLC NAND 10000 - 10,000 max. erase cycles > > Leave out the header CRC for dynamic volumes > > Combine lnum and volume id. > > Maybe go so far as to pack the information into bit fields > like LPT does. i.e. if the flash media only has a million > eraseblocks then only 20 bits are needed for lnum. Maybe it > is MLC NAND, so say 15 bits for erase counter. That leaves 5 > bites for volume id, and fits into 5 bytes for dynmaic volumes. Makes sense. >> And how to update the table relatively efficiently? > > Each update to UBI typically requires writing a whole NAND > page. That page can be filled with unchanged information > from the first eraseblock of UBI information. Hence after > enough updates the first eraseblock will end up containing > only out-of-date information and can then be unmapped. Does it mean there is no separate commit block?And do we move the journal with valid entries when it is full? I find some problems with it. 1) Why process extra information for each write? 2) We will need to scan the entire journal for each mount. 3) Will work well with only NAND. Instead, if there is a separate commit block, this can be read once and processed. I was wondering, how about combining both the approaches. i.e. A commit block and journal as you suggested?Commit time will be reduced dramatically.(As One page now contains group info). And boot time will be fixed for good unmounts. Besides, it is simple and can be easily scaled. Best Regards, Brijesh ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Regarding UBI scalability 2009-02-05 9:37 ` Brijesh Singh @ 2009-02-05 11:17 ` Adrian Hunter 2009-02-11 7:50 ` Brijesh Singh 0 siblings, 1 reply; 25+ messages in thread From: Adrian Hunter @ 2009-02-05 11:17 UTC (permalink / raw) To: Brijesh Singh; +Cc: linux-mtd Brijesh Singh wrote: > On Wed, Feb 4, 2009 at 2:59 PM, Adrian Hunter > <ext-adrian.hunter@nokia.com> wrote: >> Brijesh Singh wrote: >>> On Tue, Feb 3, 2009 at 5:14 AM, Corentin Chary <corentin.chary@gmail.com> >>> wrote: >>>> On Mon, Feb 2, 2009 at 12:18 PM, Adrian Hunter >>>> <ext-adrian.hunter@nokia.com> wrote: >>>>> Artem Bityutskiy wrote: >>>>>> On Mon, 2009-02-02 at 13:07 +0200, Adrian Hunter wrote: >>>>>>> I would suggest an intermediate step. Create UBI2 which is >>>>>>> similar to UBI but stores eraseblock information in one place, >>>>>>> instead of at the beginning of each eraseblock. Such an approach >>>>>>> might be OK up to as much as 64GiB, and would probably perform >>>>>>> better than a fully scalable version. >>>>>>> >>>>>>> Then look at creating UBI3, which is fully scalable. >>>>>> Yes, I assume UBI2 should store mapping/erasure information in separate >>>>>> tables, not in each eraseblock. So we should get rid of eraseblock >>>>>> headers. >>>>> Yes that is what I meant. You could probably make do with as little as >>>>> 12 bytes per eraseblock so a 64GiB flash with 512KiB eraseblock size >>>>> would need 1536KiB table, which could be read in a second or two, so >>>>> mount time is OK. >>> Adrian,to my understanding, this is minimum info needed per physical >>> erase block... >>> >>> Erase count -8 bytes >>> Lnum -4 bytes >>> Volume ID -4 byte(Can make it 1 byte for now as >>> vol limit=128) >>> Header CRC -4 bytes(Only for static volumes.) >>> i.e. Minimum 20 bytes. Is there any other way to make it 12? >> 8 bytes for erase counter is many more than needed i.e. >> SLC NAND 100,000 - 1,000,000 max. erase cycles >> MLC NAND 10000 - 10,000 max. erase cycles >> >> Leave out the header CRC for dynamic volumes >> >> Combine lnum and volume id. >> >> Maybe go so far as to pack the information into bit fields >> like LPT does. i.e. if the flash media only has a million >> eraseblocks then only 20 bits are needed for lnum. Maybe it >> is MLC NAND, so say 15 bits for erase counter. That leaves 5 >> bites for volume id, and fits into 5 bytes for dynmaic volumes. > > Makes sense. > >>> And how to update the table relatively efficiently? >> Each update to UBI typically requires writing a whole NAND >> page. That page can be filled with unchanged information >> from the first eraseblock of UBI information. Hence after >> enough updates the first eraseblock will end up containing >> only out-of-date information and can then be unmapped. > > Does it mean there is no separate commit block?And do we move the > journal with valid entries when it is full? I was describing my idea for UBI2. There would be no journal, no commit. When writing new mappings, also write some existing mappings from one of the eraseblocks until it no longer contains any needed information. It is like garbage-collection as you go. > I find some problems with it. > 1) Why process extra information for each write? > 2) We will need to scan the entire journal for each mount. > 3) Will work well with only NAND. > Instead, if there is a separate commit block, this can be read once > and processed. > > I was wondering, how about combining both the approaches. i.e. A > commit block and journal as you suggested?Commit time will be reduced > dramatically.(As One page now contains group info). And boot time will > be fixed for good unmounts. Besides, it is simple and can be easily > scaled. Are you talking here about UBI3? ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Regarding UBI scalability 2009-02-05 11:17 ` Adrian Hunter @ 2009-02-11 7:50 ` Brijesh Singh 0 siblings, 0 replies; 25+ messages in thread From: Brijesh Singh @ 2009-02-11 7:50 UTC (permalink / raw) To: Adrian Hunter; +Cc: linux-mtd Hi, Sorry for delayed response. On Thu, Feb 5, 2009 at 4:47 PM, Adrian Hunter <ext-adrian.hunter@nokia.com> wrote: > Brijesh Singh wrote: >> On Wed, Feb 4, 2009 at 2:59 PM, Adrian Hunter >> <ext-adrian.hunter@nokia.com> wrote: >>> Brijesh Singh wrote: >>>> On Tue, Feb 3, 2009 at 5:14 AM, Corentin Chary <corentin.chary@gmail.com> >>>> wrote: >>>>> On Mon, Feb 2, 2009 at 12:18 PM, Adrian Hunter >>>>> <ext-adrian.hunter@nokia.com> wrote: >>>>>> Artem Bityutskiy wrote: >>>>>>> On Mon, 2009-02-02 at 13:07 +0200, Adrian Hunter wrote: >>>>>>>> I would suggest an intermediate step. Create UBI2 which is >>>>>>>> similar to UBI but stores eraseblock information in one place, >>>>>>>> instead of at the beginning of each eraseblock. Such an approach >>>>>>>> might be OK up to as much as 64GiB, and would probably perform >>>>>>>> better than a fully scalable version. >>>>>>>> >>>>>>>> Then look at creating UBI3, which is fully scalable. >>>>>>> Yes, I assume UBI2 should store mapping/erasure information in separate >>>>>>> tables, not in each eraseblock. So we should get rid of eraseblock >>>>>>> headers. >>>>>> Yes that is what I meant. You could probably make do with as little as >>>>>> 12 bytes per eraseblock so a 64GiB flash with 512KiB eraseblock size >>>>>> would need 1536KiB table, which could be read in a second or two, so >>>>>> mount time is OK. >>>> Adrian,to my understanding, this is minimum info needed per physical >>>> erase block... >>>> >>>> Erase count -8 bytes >>>> Lnum -4 bytes >>>> Volume ID -4 byte(Can make it 1 byte for now as >>>> vol limit=128) >>>> Header CRC -4 bytes(Only for static volumes.) >>>> i.e. Minimum 20 bytes. Is there any other way to make it 12? >>> 8 bytes for erase counter is many more than needed i.e. >>> SLC NAND 100,000 - 1,000,000 max. erase cycles >>> MLC NAND 10000 - 10,000 max. erase cycles >>> >>> Leave out the header CRC for dynamic volumes >>> >>> Combine lnum and volume id. >>> >>> Maybe go so far as to pack the information into bit fields >>> like LPT does. i.e. if the flash media only has a million >>> eraseblocks then only 20 bits are needed for lnum. Maybe it >>> is MLC NAND, so say 15 bits for erase counter. That leaves 5 >>> bites for volume id, and fits into 5 bytes for dynmaic volumes. >> >> Makes sense. >> >>>> And how to update the table relatively efficiently? >>> Each update to UBI typically requires writing a whole NAND >>> page. That page can be filled with unchanged information >>> from the first eraseblock of UBI information. Hence after >>> enough updates the first eraseblock will end up containing >>> only out-of-date information and can then be unmapped. >> >> Does it mean there is no separate commit block?And do we move the >> journal with valid entries when it is full? > > I was describing my idea for UBI2. There would be no journal, > no commit. When writing new mappings, also write some existing > mappings from one of the eraseblocks until it no longer > contains any needed information. It is like garbage-collection > as you go. OK. >> I find some problems with it. >> 1) Why process extra information for each write? >> 2) We will need to scan the entire journal for each mount. >> 3) Will work well with only NAND. >> Instead, if there is a separate commit block, this can be read once >> and processed. >> >> I was wondering, how about combining both the approaches. i.e. A >> commit block and journal as you suggested?Commit time will be reduced >> dramatically.(As One page now contains group info). And boot time will >> be fixed for good unmounts. Besides, it is simple and can be easily >> scaled. > > Are you talking here about UBI3? No.Sorry,but I was looking at UBI2 in different way.I will write a document then may be we can discuss this.It will take some time for internal review. Thanks and Regards, Brijesh ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Regarding UBI scalability 2009-02-02 23:44 ` Corentin Chary 2009-02-03 10:35 ` Brijesh Singh @ 2009-02-03 10:46 ` Artem Bityutskiy 2009-02-03 11:13 ` Corentin Chary 1 sibling, 1 reply; 25+ messages in thread From: Artem Bityutskiy @ 2009-02-03 10:46 UTC (permalink / raw) To: Corentin Chary; +Cc: Amit Kumar Sharma, linux-mtd, Adrian Hunter, brij.singh On Tue, 2009-02-03 at 00:44 +0100, Corentin Chary wrote: > I was wondering how it is possible to get atomic operations using such tables ? > The first thing that come to my mind is TFAT (two tables, one for > operation in progress, > the other considered as always "valid"). But I'm not sure it' the good > way to do such a thing. Journal + commit mechanisms should allow doing this. -- Best regards, Artem Bityutskiy (Битюцкий Артём) ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Regarding UBI scalability 2009-02-03 10:46 ` Artem Bityutskiy @ 2009-02-03 11:13 ` Corentin Chary 2009-02-03 11:51 ` Brijesh Singh 2009-02-04 7:45 ` Artem Bityutskiy 0 siblings, 2 replies; 25+ messages in thread From: Corentin Chary @ 2009-02-03 11:13 UTC (permalink / raw) To: dedekind; +Cc: Amit Kumar Sharma, linux-mtd, Adrian Hunter, brij.singh On Tue, Feb 3, 2009 at 11:46 AM, Artem Bityutskiy <dedekind@infradead.org> wrote: > On Tue, 2009-02-03 at 00:44 +0100, Corentin Chary wrote: >> I was wondering how it is possible to get atomic operations using such tables ? >> The first thing that come to my mind is TFAT (two tables, one for >> operation in progress, >> the other considered as always "valid"). But I'm not sure it' the good >> way to do such a thing. > > Journal + commit mechanisms should allow doing this. I'm asking some other questions, because in UFFS we are working on something like that, but not for UBI2. But if I can help for UBI2, I'll. A journal node, would probably take less than 16 B. With min_io_size >= 512 B (sometime 2KB), a journal for map/unmap operation will take some time before being written. I see two solutions, based on periodic journal write with padding to have min_io_size bytes to write: - based on time (every X ms) - based on changes (every X changes) - based on both But there is probably more clever solutions. One other thing is the table will have to move on the flash for wear leveling, but for that we can use the "anchors" as defined in JFFS3 pdf. -- Corentin Chary http://xf.iksaif.net ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Regarding UBI scalability 2009-02-03 11:13 ` Corentin Chary @ 2009-02-03 11:51 ` Brijesh Singh 2009-02-04 7:45 ` Artem Bityutskiy 1 sibling, 0 replies; 25+ messages in thread From: Brijesh Singh @ 2009-02-03 11:51 UTC (permalink / raw) To: Corentin Chary; +Cc: Amit Kumar Sharma, linux-mtd, Adrian Hunter On Tue, Feb 3, 2009 at 4:43 PM, Corentin Chary <corentin.chary@gmail.com> wrote: > On Tue, Feb 3, 2009 at 11:46 AM, Artem Bityutskiy > <dedekind@infradead.org> wrote: >> On Tue, 2009-02-03 at 00:44 +0100, Corentin Chary wrote: >>> I was wondering how it is possible to get atomic operations using such tables ? >>> The first thing that come to my mind is TFAT (two tables, one for >>> operation in progress, >>> the other considered as always "valid"). But I'm not sure it' the good >>> way to do such a thing. >> >> Journal + commit mechanisms should allow doing this. > > I'm asking some other questions, because in UFFS we are working on > something like that, but not for UBI2. But if I can help for UBI2, > I'll. > > A journal node, would probably take less than 16 B. With min_io_size >>= 512 B (sometime 2KB), a journal for map/unmap operation will take > some time before being written. > > I see two solutions, based on periodic journal write with padding to > have min_io_size bytes to write: > - based on time (every X ms) > - based on changes (every X changes) > - based on both > But there is probably more clever solutions. I will check this.As of now I don't find a way to skip writing journal node.For consistency, you have to. > > One other thing is the table will have to move on the flash for wear > leveling, but for that we can use the "anchors" as defined in JFFS3 > pdf. This will move on flash. In UBI,there is a small trick. Journal and commit can be on internal volumes. So it easily gets wear-leveled. And the Super block will know where are they?Super block can be logged. Fixed in 0-5 blocks. Best Regards, Brijesh ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Regarding UBI scalability 2009-02-03 11:13 ` Corentin Chary 2009-02-03 11:51 ` Brijesh Singh @ 2009-02-04 7:45 ` Artem Bityutskiy 1 sibling, 0 replies; 25+ messages in thread From: Artem Bityutskiy @ 2009-02-04 7:45 UTC (permalink / raw) To: Corentin Chary; +Cc: Amit Kumar Sharma, linux-mtd, Adrian Hunter, brij.singh On Tue, 2009-02-03 at 12:13 +0100, Corentin Chary wrote: > On Tue, Feb 3, 2009 at 11:46 AM, Artem Bityutskiy > <dedekind@infradead.org> wrote: > > On Tue, 2009-02-03 at 00:44 +0100, Corentin Chary wrote: > >> I was wondering how it is possible to get atomic operations using such tables ? > >> The first thing that come to my mind is TFAT (two tables, one for > >> operation in progress, > >> the other considered as always "valid"). But I'm not sure it' the good > >> way to do such a thing. > > > > Journal + commit mechanisms should allow doing this. > > I'm asking some other questions, because in UFFS we are working on > something like that, but not for UBI2. But if I can help for UBI2, > I'll. > > A journal node, would probably take less than 16 B. With min_io_size > >= 512 B (sometime 2KB), a journal for map/unmap operation will take > some time before being written. Well, to guarantee integrity, I do not see any other solution but doing 2x2KiB writes on 2KiB-page NAND: start update (1), (write the LEB somewhere), end update (2), (erase the old PEB). That's the price you pay. -- Best regards, Artem Bityutskiy (Битюцкий Артём) ^ permalink raw reply [flat|nested] 25+ messages in thread
end of thread, other threads:[~2009-02-11 7:50 UTC | newest]
Thread overview: 25+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <7824366.270131233573513030.JavaMail.weblogic@epml10>
2009-02-04 9:47 ` Regarding UBI scalability Adrian Hunter
2009-02-05 9:40 ` Brijesh Singh
2009-02-08 9:48 ` Corentin Chary
2009-02-08 10:31 ` Kyungmin Park
2009-02-09 8:46 ` Artem Bityutskiy
2009-01-30 12:45 BRIJESH SINGH
2009-02-02 9:31 ` Artem Bityutskiy
2009-02-02 10:17 ` Amit Kumar Sharma
2009-02-02 11:07 ` Adrian Hunter
2009-02-02 10:57 ` Artem Bityutskiy
2009-02-02 11:18 ` Adrian Hunter
2009-02-02 20:07 ` Jamie Lokier
2009-02-02 23:44 ` Corentin Chary
2009-02-03 10:35 ` Brijesh Singh
2009-02-03 10:48 ` Artem Bityutskiy
2009-02-03 11:27 ` Enrico Scholz
2009-02-04 7:41 ` Artem Bityutskiy
2009-02-04 9:29 ` Adrian Hunter
2009-02-05 9:37 ` Brijesh Singh
2009-02-05 11:17 ` Adrian Hunter
2009-02-11 7:50 ` Brijesh Singh
2009-02-03 10:46 ` Artem Bityutskiy
2009-02-03 11:13 ` Corentin Chary
2009-02-03 11:51 ` Brijesh Singh
2009-02-04 7:45 ` Artem Bityutskiy
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.