* Unstable bits and JFFS2
@ 2012-03-28 11:16 Matej Kupljen
2012-04-02 17:12 ` Brian Norris
0 siblings, 1 reply; 12+ messages in thread
From: Matej Kupljen @ 2012-03-28 11:16 UTC (permalink / raw)
To: linux-mtd
To all,
during my experiments with the UBIFS crashing, I believe that I am experiencing
the "unstable bit" issue.
Would another FS, like JFFS2 work on such device, or I'd have the problems on
such device with all file systems?
Thank you and Best Regards,
Matej
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Unstable bits and JFFS2
2012-03-28 11:16 Unstable bits and JFFS2 Matej Kupljen
@ 2012-04-02 17:12 ` Brian Norris
2012-04-04 9:17 ` Matej Kupljen
0 siblings, 1 reply; 12+ messages in thread
From: Brian Norris @ 2012-04-02 17:12 UTC (permalink / raw)
To: Matej Kupljen; +Cc: linux-mtd
Hi Matej,
On Wed, Mar 28, 2012 at 4:16 AM, Matej Kupljen <matej.kupljen@gmail.com> wrote:
> during my experiments with the UBIFS crashing, I believe that I am experiencing
> the "unstable bit" issue.
>
> Would another FS, like JFFS2 work on such device, or I'd have the problems on
> such device with all file systems?
I doubt it. JFFS2 is not supported much anymore, and it is not
designed for some of the problems with modern NAND flash.
Have you read the information here?
http://www.linux-mtd.infradead.org/doc/ubifs.html#L_unstable_bits
I'm not much of an export on UBIFS or unstable bits, but I think that
there are some unsolved problems. However, before ruling this an
unstable bits problem, it's a good idea to rule out any other possible
issues.
Brian
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Unstable bits and JFFS2
2012-04-02 17:12 ` Brian Norris
@ 2012-04-04 9:17 ` Matej Kupljen
2012-04-04 10:54 ` Atlant Schmidt
0 siblings, 1 reply; 12+ messages in thread
From: Matej Kupljen @ 2012-04-04 9:17 UTC (permalink / raw)
To: Brian Norris; +Cc: linux-mtd
Brian,
> On Wed, Mar 28, 2012 at 4:16 AM, Matej Kupljen <matej.kupljen@gmail.com> wrote:
>> during my experiments with the UBIFS crashing, I believe that I am experiencing
>> the "unstable bit" issue.
>>
>> Would another FS, like JFFS2 work on such device, or I'd have the problems on
>> such device with all file systems?
>
> I doubt it. JFFS2 is not supported much anymore, and it is not
> designed for some of the problems with modern NAND flash.
Aha, I see.
Thank you for the information.
> Have you read the information here?
>
> http://www.linux-mtd.infradead.org/doc/ubifs.html#L_unstable_bits
Yes, I have. Artem already pointed me to that FAQ.
> I'm not much of an export on UBIFS or unstable bits, but I think that
> there are some unsolved problems. However, before ruling this an
> unstable bits problem, it's a good idea to rule out any other possible
> issues.
Yes, I'll check the MTD driver.
At the moment I am running again integrity check on my UBIFS for
almost 24 hours
now, but until now no error:
# ./integck -n 0 -v -e /media/card
The only difference from previous run, where I saw one bit flip is
that previously
I run the test with power failure option set to ON. Now it is off.
Artem, you said that this unstable bits only happen during power cuts,
is this right?
Would those appear also on simulated power cuts, the ones that can
integck produce?
As for the ECC correction of the empty space, nothing is mentioned in
the manual.
Maybe I should ask the manufacturer of the chip (NXP LPC3152)??
It uses HW error correction included in the FLASH controller, with:
"The error correction code used is Reed-Solomon over GF(2^9). The
primitive polynomial
g(x) over GF(2) is:
g(x) = x9 + x4 +1"
Thank you and Best Regards,
Matej
^ permalink raw reply [flat|nested] 12+ messages in thread
* RE: Unstable bits and JFFS2
2012-04-04 9:17 ` Matej Kupljen
@ 2012-04-04 10:54 ` Atlant Schmidt
2012-04-10 7:22 ` Matej Kupljen
2012-04-13 16:27 ` Artem Bityutskiy
0 siblings, 2 replies; 12+ messages in thread
From: Atlant Schmidt @ 2012-04-04 10:54 UTC (permalink / raw)
To: 'Matej Kupljen', Brian Norris; +Cc: linux-mtd
Matej:
> Artem, you said that this unstable bits only happen
> during power cuts, is this right? Would those appear
> also on simulated power cuts, the ones that integck
> can produce?
(Note: This isn't Artem replying.)
Unstable bits can happen anytime, but *REAL* power-cuts
while writing are certainly a great way to produce them;
after all, if you've only transferred half the electrons
to the floating gate when power goes away, well, that
Flash cell is now a roll-of-the-quantum-dice each time
you read it.
Simulated power cuts (that just stop the software
processing at arbitrary and random points) can't
produce this effect.
But any time a read-disturb or write-disturb takes place,
there's some probability that a Flash cell will be left
with a "near-threshold" charge on the floating gate, so
unstable bits are a fact of life that must be faced by
any software that drives NAND Flash memory chips. This
is, of course, especially true of MLC chips and even
more-true for TLC chips (with three bits per cell).
Atlant
-----Original Message-----
From: linux-mtd-bounces@lists.infradead.org [mailto:linux-mtd-bounces@lists.infradead.org] On Behalf Of Matej Kupljen
Sent: Wednesday, April 04, 2012 05:17
To: Brian Norris
Cc: linux-mtd
Subject: Re: Unstable bits and JFFS2
Brian,
> On Wed, Mar 28, 2012 at 4:16 AM, Matej Kupljen <matej.kupljen@gmail.com> wrote:
>> during my experiments with the UBIFS crashing, I believe that I am experiencing
>> the "unstable bit" issue.
>>
>> Would another FS, like JFFS2 work on such device, or I'd have the problems on
>> such device with all file systems?
>
> I doubt it. JFFS2 is not supported much anymore, and it is not
> designed for some of the problems with modern NAND flash.
Aha, I see.
Thank you for the information.
> Have you read the information here?
>
> http://www.linux-mtd.infradead.org/doc/ubifs.html#L_unstable_bits
Yes, I have. Artem already pointed me to that FAQ.
> I'm not much of an export on UBIFS or unstable bits, but I think that
> there are some unsolved problems. However, before ruling this an
> unstable bits problem, it's a good idea to rule out any other possible
> issues.
Yes, I'll check the MTD driver.
At the moment I am running again integrity check on my UBIFS for
almost 24 hours
now, but until now no error:
# ./integck -n 0 -v -e /media/card
The only difference from previous run, where I saw one bit flip is
that previously
I run the test with power failure option set to ON. Now it is off.
Artem, you said that this unstable bits only happen during power cuts,
is this right?
Would those appear also on simulated power cuts, the ones that can
integck produce?
As for the ECC correction of the empty space, nothing is mentioned in
the manual.
Maybe I should ask the manufacturer of the chip (NXP LPC3152)??
It uses HW error correction included in the FLASH controller, with:
"The error correction code used is Reed-Solomon over GF(2^9). The
primitive polynomial
g(x) over GF(2) is:
g(x) = x9 + x4 +1"
Thank you and Best Regards,
Matej
______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/
Click https://www.mailcontrol.com/sr/kqBJlvK+JBjTndxI!oX7Ulvwl1GqFAPrim4OaE2i4BWjrlQi2a8dR8ju1tDr2izUVJQZBBFsBrWtoBNtFUgrMg== to report this email as spam.
This e-mail and the information, including any attachments, it contains are intended to be a confidential communication only to the person or entity to whom it is addressed and may contain information that is privileged. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please immediately notify the sender and destroy the original message.
Thank you.
Please consider the environment before printing this email.
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Unstable bits and JFFS2
2012-04-04 10:54 ` Atlant Schmidt
@ 2012-04-10 7:22 ` Matej Kupljen
2012-04-13 16:27 ` Artem Bityutskiy
1 sibling, 0 replies; 12+ messages in thread
From: Matej Kupljen @ 2012-04-10 7:22 UTC (permalink / raw)
To: Atlant Schmidt; +Cc: Brian Norris, linux-mtd
To all,
>> Artem, you said that this unstable bits only happen
>> during power cuts, is this right? Would those appear
>> also on simulated power cuts, the ones that integck
>> can produce?
I've been trying to find the issue with my device over last week and for now,
I didn't find anything.
Here is what I've done:
Please NOTE, that I have used THE SAME device for testing, which showed
the problems in the past.
1) I've prepared SD card and booted the device from it. Then I run the
integchk without
simulating power cuts. The test run form more then 96 hours (four
days and nights)
and nothing happened. I mean, no errors were reported.
2) I made a simple "power cut simulator", which cuts the power to the
device. The device
was booted from SD card, then the UBIFS partition on NAND Flash was mounted.
The test file was removed, and then a new file was created with
simple dd from
/dev/zero to the test file on UBIFS.
The test run for 72 hours and again, nothing happened.
3) Now I am running the test, with power cut simulator, bit this time
the device is booted
from UBIFS. It has been running for more then 12h at the moment,
and for now it
is still working as expected.
The only real thing from previous tests is that now, I have formatted
the UBIFS from
the device, when it was booted from SD card and Linux was up and running.
Previous tests were done (when I've noticed the problems) only when U-Boot was
running and the ubinized image was flashed to Flash trough U-Boot.
Could this be the problem?
Maybe different timings or something in U-Boot compared to Linux?
Thank you and Best Regards,
Matej Kupljen
^ permalink raw reply [flat|nested] 12+ messages in thread
* RE: Unstable bits and JFFS2
2012-04-04 10:54 ` Atlant Schmidt
2012-04-10 7:22 ` Matej Kupljen
@ 2012-04-13 16:27 ` Artem Bityutskiy
2012-04-13 16:40 ` Atlant Schmidt
2012-04-13 18:36 ` Wolfram Sang
1 sibling, 2 replies; 12+ messages in thread
From: Artem Bityutskiy @ 2012-04-13 16:27 UTC (permalink / raw)
To: Atlant Schmidt; +Cc: 'Matej Kupljen', Brian Norris, linux-mtd
[-- Attachment #1: Type: text/plain, Size: 1483 bytes --]
On Wed, 2012-04-04 at 06:54 -0400, Atlant Schmidt wrote:
> Matej:
>
> > Artem, you said that this unstable bits only happen
> > during power cuts, is this right? Would those appear
> > also on simulated power cuts, the ones that integck
> > can produce?
>
> (Note: This isn't Artem replying.)
>
> Unstable bits can happen anytime, but *REAL* power-cuts
> while writing are certainly a great way to produce them;
> after all, if you've only transferred half the electrons
> to the floating gate when power goes away, well, that
> Flash cell is now a roll-of-the-quantum-dice each time
> you read it.
>
> Simulated power cuts (that just stop the software
> processing at arbitrary and random points) can't
> produce this effect.
>
> But any time a read-disturb or write-disturb takes place,
> there's some probability that a Flash cell will be left
> with a "near-threshold" charge on the floating gate, so
> unstable bits are a fact of life that must be faced by
> any software that drives NAND Flash memory chips. This
> is, of course, especially true of MLC chips and even
> more-true for TLC chips (with three bits per cell).
Yeah, thanks for correcting. Yeah, read/write-disturb may make bits to
become unstable, but we assume this is a slow process which will
gradually make more and more bits flip and ECC will take care of that.
So I think Matej can exclude this.
--
Best Regards,
Artem Bityutskiy
[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 836 bytes --]
^ permalink raw reply [flat|nested] 12+ messages in thread
* RE: Unstable bits and JFFS2
2012-04-13 16:27 ` Artem Bityutskiy
@ 2012-04-13 16:40 ` Atlant Schmidt
2012-04-13 17:01 ` Artem Bityutskiy
2012-04-13 18:36 ` Wolfram Sang
1 sibling, 1 reply; 12+ messages in thread
From: Atlant Schmidt @ 2012-04-13 16:40 UTC (permalink / raw)
To: 'dedekind1@gmail.com'
Cc: 'Matej Kupljen', Brian Norris, linux-mtd
Artem:
> Yeah, thanks for correcting. Yeah, read/write-disturb may make bits
> to become unstable, but we assume this is a slow process which will
> gradually make more and more bits flip and ECC will take care of
> that. So I think Matej can exclude this.
Unfortunately, ECC can only fix those PEBs that
are actually read. If one of the PEBs in your
filesystem is being used entirely to contain
obscure data that only gets read once in a
blue moon (i.e., very rarely), then multiple
read-/write-disturbs can hit accumulate in
that PEB and when it is finally read, it may
already contains too many errors to be
corrected by the ECC.
We've seen this exact scenario occur with the
current UBI/UBIfs and will be implementing a
userland "scrubbing" system to ensure that
every PEB gets read at least once in a while
(e.g., weekly or whatever interval seems right).
Atlant
-----Original Message-----
From: Artem Bityutskiy [mailto:dedekind1@gmail.com]
Sent: Friday, April 13, 2012 12:28
To: Atlant Schmidt
Cc: 'Matej Kupljen'; Brian Norris; linux-mtd
Subject: RE: Unstable bits and JFFS2
On Wed, 2012-04-04 at 06:54 -0400, Atlant Schmidt wrote:
> Matej:
>
> > Artem, you said that this unstable bits only happen
> > during power cuts, is this right? Would those appear
> > also on simulated power cuts, the ones that integck
> > can produce?
>
> (Note: This isn't Artem replying.)
>
> Unstable bits can happen anytime, but *REAL* power-cuts
> while writing are certainly a great way to produce them;
> after all, if you've only transferred half the electrons
> to the floating gate when power goes away, well, that
> Flash cell is now a roll-of-the-quantum-dice each time
> you read it.
>
> Simulated power cuts (that just stop the software
> processing at arbitrary and random points) can't
> produce this effect.
>
> But any time a read-disturb or write-disturb takes place,
> there's some probability that a Flash cell will be left
> with a "near-threshold" charge on the floating gate, so
> unstable bits are a fact of life that must be faced by
> any software that drives NAND Flash memory chips. This
> is, of course, especially true of MLC chips and even
> more-true for TLC chips (with three bits per cell).
Yeah, thanks for correcting. Yeah, read/write-disturb may make bits to
become unstable, but we assume this is a slow process which will
gradually make more and more bits flip and ECC will take care of that.
So I think Matej can exclude this.
--
Best Regards,
Artem Bityutskiy
This e-mail and the information, including any attachments, it contains are intended to be a confidential communication only to the person or entity to whom it is addressed and may contain information that is privileged. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please immediately notify the sender and destroy the original message.
Thank you.
Please consider the environment before printing this email.
^ permalink raw reply [flat|nested] 12+ messages in thread
* RE: Unstable bits and JFFS2
2012-04-13 16:40 ` Atlant Schmidt
@ 2012-04-13 17:01 ` Artem Bityutskiy
2012-04-14 4:21 ` Ricard Wanderlof
0 siblings, 1 reply; 12+ messages in thread
From: Artem Bityutskiy @ 2012-04-13 17:01 UTC (permalink / raw)
To: Atlant Schmidt; +Cc: 'Matej Kupljen', Brian Norris, linux-mtd
[-- Attachment #1: Type: text/plain, Size: 1389 bytes --]
On Fri, 2012-04-13 at 12:40 -0400, Atlant Schmidt wrote:
> Artem:
>
> > Yeah, thanks for correcting. Yeah, read/write-disturb may make bits
> > to become unstable, but we assume this is a slow process which will
> > gradually make more and more bits flip and ECC will take care of
> > that. So I think Matej can exclude this.
>
> Unfortunately, ECC can only fix those PEBs that
> are actually read. If one of the PEBs in your
> filesystem is being used entirely to contain
> obscure data that only gets read once in a
> blue moon (i.e., very rarely), then multiple
> read-/write-disturbs can hit accumulate in
> that PEB and when it is finally read, it may
> already contains too many errors to be
> corrected by the ECC.
>
> We've seen this exact scenario occur with the
> current UBI/UBIfs and will be implementing a
> userland "scrubbing" system to ensure that
> every PEB gets read at least once in a while
> (e.g., weekly or whatever interval seems right).
This is right, people should have a user-space app which periodically
reads all /dev/ubiX_Y. Care to send a patch against mtd-www with some
explantion of this issue and extend the unstable bits issue?
But in Matej's case I do not believe he is testing long enough to hit
this scenario. But everything is possible of course.
--
Best Regards,
Artem Bityutskiy
[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 836 bytes --]
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Unstable bits and JFFS2
2012-04-13 16:27 ` Artem Bityutskiy
2012-04-13 16:40 ` Atlant Schmidt
@ 2012-04-13 18:36 ` Wolfram Sang
2012-04-13 19:49 ` Artem Bityutskiy
1 sibling, 1 reply; 12+ messages in thread
From: Wolfram Sang @ 2012-04-13 18:36 UTC (permalink / raw)
To: Artem Bityutskiy
Cc: 'Matej Kupljen', Brian Norris, linux-mtd, Atlant Schmidt
[-- Attachment #1: Type: text/plain, Size: 1504 bytes --]
Hi,
> > > Artem, you said that this unstable bits only happen
> > > during power cuts, is this right? Would those appear
> > > also on simulated power cuts, the ones that integck
> > > can produce?
A bit of hijacking this thread, yet it fits somehow to the idea I got
yesterday:
There is the Open Project Proposal again, where the CEWG funds topics which
improve embedded linux in general [1]. I wonder if it makes sense to hand in a
proposal starting to tackle the 'unstable bits' issue. I could imagine task 1
mentioned in the wiki page (1. Improve the existing power cut emulation
infrastructure in UBIFS and start emulating unstable bits. Start with emulating
only one type of unstable bits, e.g., type 1.) could be a good candidate for
such a proposal. Since I am not deeply involved with this topic:
1) Is this task still needed?
2) Is it a good task for a 2-3 week period (I'd think so)
3) Are there people around up for the task?
I'd be willing to write the proposal, if nobody else volunteers. (yet, given my
constraints, I am not the right person to actually work on it). Even if the
proposal doesn't get contracted, the topic itself might get some attention
since it will be discussed in the CEWG.
What do you think?
Regards,
Wolfram
[1] http://elinux.org/CEWG_Open_Project_Proposal_2012
--
Pengutronix e.K. | Wolfram Sang |
Industrial Linux Solutions | http://www.pengutronix.de/ |
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Unstable bits and JFFS2
2012-04-13 18:36 ` Wolfram Sang
@ 2012-04-13 19:49 ` Artem Bityutskiy
0 siblings, 0 replies; 12+ messages in thread
From: Artem Bityutskiy @ 2012-04-13 19:49 UTC (permalink / raw)
To: Wolfram Sang
Cc: 'Matej Kupljen', Brian Norris, linux-mtd, Atlant Schmidt
On Fri, 2012-04-13 at 20:36 +0200, Wolfram Sang wrote:
> Hi,
>
> > > > Artem, you said that this unstable bits only happen
> > > > during power cuts, is this right? Would those appear
> > > > also on simulated power cuts, the ones that integck
> > > > can produce?
>
> A bit of hijacking this thread, yet it fits somehow to the idea I got
> yesterday:
>
> There is the Open Project Proposal again, where the CEWG funds topics which
> improve embedded linux in general [1]. I wonder if it makes sense to hand in a
> proposal starting to tackle the 'unstable bits' issue. I could imagine task 1
> mentioned in the wiki page (1. Improve the existing power cut emulation
> infrastructure in UBIFS and start emulating unstable bits. Start with emulating
> only one type of unstable bits, e.g., type 1.) could be a good candidate for
> such a proposal. Since I am not deeply involved with this topic:
>
> 1) Is this task still needed?
> 2) Is it a good task for a 2-3 week period (I'd think so)
> 3) Are there people around up for the task?
>
> I'd be willing to write the proposal, if nobody else volunteers. (yet, given my
> constraints, I am not the right person to actually work on it). Even if the
> proposal doesn't get contracted, the topic itself might get some attention
> since it will be discussed in the CEWG.
Yes, definitely a hot topic now, I am swamped with lots of things and
never found time for writing proposals like this. Please, go ahead,
would be very appreciated. I might help reviewing it and amending.
^ permalink raw reply [flat|nested] 12+ messages in thread
* RE: Unstable bits and JFFS2
2012-04-13 17:01 ` Artem Bityutskiy
@ 2012-04-14 4:21 ` Ricard Wanderlof
2012-04-22 14:25 ` Artem Bityutskiy
0 siblings, 1 reply; 12+ messages in thread
From: Ricard Wanderlof @ 2012-04-14 4:21 UTC (permalink / raw)
To: Artem Bityutskiy
Cc: 'Matej Kupljen', Brian Norris, linux-mtd, Atlant Schmidt
On Fri, 13 Apr 2012, Artem Bityutskiy wrote:
> On Fri, 2012-04-13 at 12:40 -0400, Atlant Schmidt wrote:
>>
>> Unfortunately, ECC can only fix those PEBs that
>> are actually read. If one of the PEBs in your
>> filesystem is being used entirely to contain
>> obscure data that only gets read once in a
>> blue moon (i.e., very rarely), then multiple
>> read-/write-disturbs can hit accumulate in
>> that PEB and when it is finally read, it may
>> already contains too many errors to be
>> corrected by the ECC.
>>
>> We've seen this exact scenario occur with the
>> current UBI/UBIfs and will be implementing a
>> userland "scrubbing" system to ensure that
>> every PEB gets read at least once in a while
>> (e.g., weekly or whatever interval seems right).
>
> This is right, people should have a user-space app which periodically
> reads all /dev/ubiX_Y. Care to send a patch against mtd-www with some
> explantion of this issue and extend the unstable bits issue?
Would it not be preferable to have this functionality within UBI itself,
so that, for instance governed by some sysfs parameter, UBI reads through
all PEBs at some interval, triggering scrubbing of any blocks that require
too much ECC for comfort?
To me it seems something that userspace should not be concerned with,
given the rest of the features that UBI deals with so that userspace does
not have to.
/Ricard
--
Ricard Wolf Wanderlöf ricardw(at)axis.com
Axis Communications AB, Lund, Sweden www.axis.com
Phone +46 46 272 2016 Fax +46 46 13 61 30
^ permalink raw reply [flat|nested] 12+ messages in thread
* RE: Unstable bits and JFFS2
2012-04-14 4:21 ` Ricard Wanderlof
@ 2012-04-22 14:25 ` Artem Bityutskiy
0 siblings, 0 replies; 12+ messages in thread
From: Artem Bityutskiy @ 2012-04-22 14:25 UTC (permalink / raw)
To: Ricard Wanderlof
Cc: 'Matej Kupljen', Brian Norris, linux-mtd, Atlant Schmidt
[-- Attachment #1: Type: text/plain, Size: 361 bytes --]
On Sat, 2012-04-14 at 06:21 +0200, Ricard Wanderlof wrote:
> To me it seems something that userspace should not be concerned with,
> given the rest of the features that UBI deals with so that userspace does
> not have to.
Yes, may probably you are right. Now we just need the brave knight who'd
send the patch.
--
Best Regards,
Artem Bityutskiy
[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 836 bytes --]
^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2012-04-22 14:27 UTC | newest]
Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2012-03-28 11:16 Unstable bits and JFFS2 Matej Kupljen
2012-04-02 17:12 ` Brian Norris
2012-04-04 9:17 ` Matej Kupljen
2012-04-04 10:54 ` Atlant Schmidt
2012-04-10 7:22 ` Matej Kupljen
2012-04-13 16:27 ` Artem Bityutskiy
2012-04-13 16:40 ` Atlant Schmidt
2012-04-13 17:01 ` Artem Bityutskiy
2012-04-14 4:21 ` Ricard Wanderlof
2012-04-22 14:25 ` Artem Bityutskiy
2012-04-13 18:36 ` Wolfram Sang
2012-04-13 19:49 ` Artem Bityutskiy
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.