linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* A really interesting mass-storage BUG introduced after 2.6.18
@ 2009-01-11 11:41 Ozan Çağlayan
  2009-01-12 22:19 ` Alan Stern
  0 siblings, 1 reply; 6+ messages in thread
From: Ozan Çağlayan @ 2009-01-11 11:41 UTC (permalink / raw)
  To: linux-kernel, linux-usb

Hi,

One of our users reported[0] that his 128MB mass storage driver doesn't 
work anymore on our current stock kernel 2.6.25.20 by hotplugging but 
works flawlessly if it was plugged before booting the OS. He also said 
that it was working out of the box with our old kernel 2.6.18. I 
gathered a lot of output and debugging stuff from the user[1][2] and 
found out that the device has some problems with SCSI Inquiry commands 
then I discovered the patch of Alan Stern about the bad residue 
values[3], backported it and no way it didn't solve the problem.

I also suggested to give a try to the current stable 2.6.28 but the 
behaviour is still the same.

When hotplugged, the kernel doesn't detect the partitions in the device 
leaving the block device on its own but if it's plugged during the boot, 
the partitions are correctly detected and have their own /sys/block 
entries. Note that the device reports a "usb embedded hub" beside a mass 
storage device (dunno if it's the case with all of these kind devices).

[0]: http://bugs.pardus.org.tr/show_bug.cgi?id=8369
[1]: http://bugzilla.kernel.org/show_bug.cgi?id=11125
[2]: http://bugs.pardus.org.tr/attachment.cgi?id=3553 (Hotplugged dmesg 
output)
[3]: http://bugs.pardus.org.tr/attachment.cgi?id=3561 (Coldplug dmesg 
output, correctly detected)

Regards,

-- 

Ozan Çağlayan
<ozan_at_pardus.org.tr>


^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: A really interesting mass-storage BUG introduced after 2.6.18
  2009-01-11 11:41 A really interesting mass-storage BUG introduced after 2.6.18 Ozan Çağlayan
@ 2009-01-12 22:19 ` Alan Stern
  2009-01-13 11:28   ` Ozan Çağlayan
  0 siblings, 1 reply; 6+ messages in thread
From: Alan Stern @ 2009-01-12 22:19 UTC (permalink / raw)
  To: Ozan Çağlayan; +Cc: linux-kernel, linux-usb

On Sun, 11 Jan 2009, [UTF-8] Ozan Çağlayan wrote:

> Hi,
> 
> One of our users reported[0] that his 128MB mass storage driver doesn't 
> work anymore on our current stock kernel 2.6.25.20 by hotplugging but 
> works flawlessly if it was plugged before booting the OS. He also said 
> that it was working out of the box with our old kernel 2.6.18. I 
> gathered a lot of output and debugging stuff from the user[1][2] and 
> found out that the device has some problems with SCSI Inquiry commands 
> then I discovered the patch of Alan Stern about the bad residue 
> values[3], backported it and no way it didn't solve the problem.
> 
> I also suggested to give a try to the current stable 2.6.28 but the 
> behaviour is still the same.
> 
> When hotplugged, the kernel doesn't detect the partitions in the device 
> leaving the block device on its own but if it's plugged during the boot, 
> the partitions are correctly detected and have their own /sys/block 
> entries. Note that the device reports a "usb embedded hub" beside a mass 
> storage device (dunno if it's the case with all of these kind devices).
> 
> [0]: http://bugs.pardus.org.tr/show_bug.cgi?id=8369
> [1]: http://bugzilla.kernel.org/show_bug.cgi?id=11125
> [2]: http://bugs.pardus.org.tr/attachment.cgi?id=3553 (Hotplugged dmesg 
> output)
> [3]: http://bugs.pardus.org.tr/attachment.cgi?id=3561 (Coldplug dmesg 
> output, correctly detected)

Please ask Antoon to collect a usbmon log during a hotplug test.  The 
instructions are in Documentation/usb/usbmon.txt.

Alan Stern


^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: A really interesting mass-storage BUG introduced after 2.6.18
  2009-01-12 22:19 ` Alan Stern
@ 2009-01-13 11:28   ` Ozan Çağlayan
  2009-01-14 15:47     ` Alan Stern
  0 siblings, 1 reply; 6+ messages in thread
From: Ozan Çağlayan @ 2009-01-13 11:28 UTC (permalink / raw)
  To: Alan Stern; +Cc: linux-kernel, linux-usb

Alan Stern wrote On 13-01-2009 00:19:

> 
> Please ask Antoon to collect a usbmon log during a hotplug test.  The 
> instructions are in Documentation/usb/usbmon.txt.
> 
> Alan Stern
>

Here's the usbmon log:

ee5f0100 2494136602 S Ci:1:001:0 s a3 00 0000 0001 0004 4 <
ee5f0100 2494136621 C Ci:1:001:0 0 4 = 00010000
ee5f0100 2494136627 S Ci:1:001:0 s a3 00 0000 0002 0004 4 <
ee5f0100 2494136629 C Ci:1:001:0 0 4 = 00010000
ee5f0100 2494136631 S Ci:1:001:0 s a3 00 0000 0003 0004 4 <
ee5f0100 2494136633 C Ci:1:001:0 0 4 = 00010000
ee5f0100 2494136635 S Ci:1:001:0 s a3 00 0000 0004 0004 4 <
ee5f0100 2494136637 C Ci:1:001:0 0 4 = 00010000
ee5f0100 2494136639 S Ci:1:001:0 s a3 00 0000 0005 0004 4 <
ee5f0100 2494136641 C Ci:1:001:0 0 4 = 00010000
ee5f0100 2494136643 S Ci:1:001:0 s a3 00 0000 0006 0004 4 <
ee5f0100 2494136644 C Ci:1:001:0 0 4 = 01050100
ee5f0100 2494136646 S Ci:1:001:0 s a3 00 0000 0007 0004 4 <
ee5f0100 2494136648 C Ci:1:001:0 0 4 = 00010000
ee5f0100 2494136650 S Ci:1:001:0 s a3 00 0000 0008 0004 4 <
ee5f0100 2494136652 C Ci:1:001:0 0 4 = 00010000
f7738700 2494136654 S Ii:1:001:1 -115:2048 4 <
ee5f0100 2494136668 S Ci:1:001:0 s a3 00 0000 0001 0004 4 <
ee5f0100 2494136670 C Ci:1:001:0 0 4 = 00010000
ee5f0100 2494136672 S Ci:1:001:0 s a3 00 0000 0002 0004 4 <
ee5f0100 2494136674 C Ci:1:001:0 0 4 = 00010000
ee5f0100 2494136676 S Ci:1:001:0 s a3 00 0000 0003 0004 4 <
ee5f0100 2494136678 C Ci:1:001:0 0 4 = 00010000
ee5f0100 2494136679 S Ci:1:001:0 s a3 00 0000 0004 0004 4 <
ee5f0100 2494136681 C Ci:1:001:0 0 4 = 00010000
ee5f0100 2494136683 S Ci:1:001:0 s a3 00 0000 0005 0004 4 <
ee5f0100 2494136685 C Ci:1:001:0 0 4 = 00010000
ee5f0100 2494136693 S Ci:1:001:0 s a3 00 0000 0006 0004 4 <
ee5f0100 2494136695 C Ci:1:001:0 0 4 = 01050100
ee5f0100 2494136697 S Co:1:001:0 s 23 01 0010 0006 0000 0
ee5f0100 2494136699 C Co:1:001:0 0 0
ee5f0100 2494136701 S Ci:1:001:0 s a3 00 0000 0006 0004 4 <
ee5f0100 2494136703 C Ci:1:001:0 0 4 = 01050000
ee5f0100 2494162597 S Ci:1:001:0 s a3 00 0000 0006 0004 4 <
ee5f0100 2494162601 C Ci:1:001:0 0 4 = 01050000
ee5f0100 2494188595 S Ci:1:001:0 s a3 00 0000 0006 0004 4 <
ee5f0100 2494188599 C Ci:1:001:0 0 4 = 01050000
ee5f0100 2494214603 S Ci:1:001:0 s a3 00 0000 0006 0004 4 <
ee5f0100 2494214611 C Ci:1:001:0 0 4 = 01050000
ee5f0100 2494240596 S Ci:1:001:0 s a3 00 0000 0006 0004 4 <
ee5f0100 2494240599 C Ci:1:001:0 0 4 = 01050000
ee5f0100 2494240609 S Co:1:001:0 s 23 03 0004 0006 0000 0
ee5f0100 2494240613 C Co:1:001:0 0 0
ee5f0100 2494291596 S Ci:1:001:0 s a3 00 0000 0006 0004 4 <
ee5f0100 2494291832 C Ci:1:001:0 0 4 = 03051000
ee5f0100 2494342598 S Co:1:001:0 s 23 01 0014 0006 0000 0
ee5f0100 2494342604 C Co:1:001:0 0 0
ee5f0100 2494342618 S Ci:1:000:0 s 80 06 0100 0000 0040 64 <
ee5f0100 2494344842 C Ci:1:000:0 0 18 = 12010002 09000140 e0131525 00010103 0001
ee5f0100 2494344857 S Co:1:001:0 s 23 03 0004 0006 0000 0
ee5f0100 2494344860 C Co:1:001:0 0 0
ee5f0100 2494395600 S Ci:1:001:0 s a3 00 0000 0006 0004 4 <
ee5f0100 2494395841 C Ci:1:001:0 0 4 = 03051000
ee5f0100 2494446595 S Co:1:001:0 s 23 01 0014 0006 0000 0
ee5f0100 2494446600 C Co:1:001:0 0 0
ee5f0100 2494446605 S Co:1:000:0 s 00 05 000a 0000 0000 0
ee5f0100 2494446723 C Co:1:000:0 0 0
ee5f0100 2494458594 S Ci:1:010:0 s 80 06 0100 0000 0012 18 <
ee5f0100 2494460849 C Ci:1:010:0 0 18 = 12010002 09000140 e0131525 00010103 0001
ee5f0100 2494460864 S Ci:1:010:0 s 80 06 0200 0000 0009 9 <
ee5f0100 2494462974 C Ci:1:010:0 0 9 = 09022200 010100c0 32
ee5f0100 2494462983 S Ci:1:010:0 s 80 06 0200 0000 0022 34 <
ee5f0100 2494465224 C Ci:1:010:0 0 34 = 09022200 010100c0 32090400 00010900 00000705 81030100 0c092901 0d003264
ee5f0100 2494465238 S Ci:1:010:0 s 80 06 0300 0000 00ff 255 <
ee5f0100 2494467349 C Ci:1:010:0 0 4 = 04030904
ee5f0100 2494467358 S Ci:1:010:0 s 80 06 0303 0409 00ff 255 <
ee5f0100 2494469599 C Ci:1:010:0 0 34 = 22035500 53004200 20004500 6d006200 65006400 64006500 64002000 48007500
ee5f0100 2494469609 S Ci:1:010:0 s 80 06 0301 0409 00ff 255 <
ee5f0100 2494471724 C Ci:1:010:0 0 18 = 12035400 53004500 20004900 6e006300 2e00
ee5f0100 2494472677 S Co:1:010:0 s 00 09 0001 0000 0000 0
ee5f0100 2494472853 C Co:1:010:0 0 0
ee5f0100 2494474632 S Ci:1:010:0 s a0 06 2900 0000 000f 15 <
ee5f0100 2494474851 C Ci:1:010:0 0 9 = 0929010d 00326402 ff
ee5f0100 2494474867 S Ci:1:010:0 s 80 00 0000 0000 0002 2 <
ee5f0100 2494474974 C Ci:1:010:0 0 2 = 0100
ee5f0100 2494474986 S Ci:1:010:0 s a0 00 0000 0000 0004 4 <
ee5f0100 2494475100 C Ci:1:010:0 0 4 = 01000000
ee5f0180 2494475111 S Co:1:010:0 s 23 03 0008 0001 0000 0
ee5f0180 2494475224 C Co:1:010:0 0 0
ee5f0100 2494575599 S Ii:1:010:1 -115:2048 1 <
ee5f0400 2494576161 S Ci:1:001:0 s a3 00 0000 0007 0004 4 <
ee5f0400 2494576170 C Ci:1:001:0 0 4 = 00010000
ee5f0400 2494576175 S Ci:1:001:0 s a3 00 0000 0008 0004 4 <
ee5f0400 2494576177 C Ci:1:001:0 0 4 = 00010000
ee5f0400 2494576181 S Ci:1:010:0 s a3 00 0000 0001 0004 4 <
ee5f0400 2494576361 C Ci:1:010:0 0 4 = 01050100
ee5f0400 2494576411 S Co:1:010:0 s 23 01 0010 0001 0000 0
ee5f0400 2494576610 C Co:1:010:0 0 0
ee5f0400 2494576630 S Ci:1:010:0 s a3 00 0000 0001 0004 4 <
ee5f0400 2494576733 C Ci:1:010:0 0 4 = 01050000
ee5f0400 2494602603 S Ci:1:010:0 s a3 00 0000 0001 0004 4 <
ee5f0400 2494602737 C Ci:1:010:0 0 4 = 01050000
ee5f0400 2494628595 S Ci:1:010:0 s a3 00 0000 0001 0004 4 <
ee5f0400 2494628736 C Ci:1:010:0 0 4 = 01050000
ee5f0400 2494654595 S Ci:1:010:0 s a3 00 0000 0001 0004 4 <
ee5f0400 2494654739 C Ci:1:010:0 0 4 = 01050000
ee5f0400 2494680601 S Ci:1:010:0 s a3 00 0000 0001 0004 4 <
ee5f0400 2494680742 C Ci:1:010:0 0 4 = 01050000
ee5f0400 2494680766 S Co:1:010:0 s 23 03 0004 0001 0000 0
ee5f0400 2494680990 C Co:1:010:0 0 0
ee5f0400 2494691594 S Ci:1:010:0 s a3 00 0000 0001 0004 4 <
ee5f0400 2494691741 C Ci:1:010:0 0 4 = 03051000
ee5f0400 2494742596 S Co:1:010:0 s 23 01 0014 0001 0000 0
ee5f0400 2494742744 C Co:1:010:0 0 0
ee5f0400 2494742773 S Ci:1:000:0 s 80 06 0100 0000 0040 64 <
ee5f0400 2494744995 C Ci:1:000:0 0 18 = 12010002 00000040 e0131725 00010104 0001
ee5f0400 2494745009 S Co:1:010:0 s 23 03 0004 0001 0000 0
ee5f0400 2494745245 C Co:1:010:0 0 0
ee5f0400 2494755595 S Ci:1:010:0 s a3 00 0000 0001 0004 4 <
ee5f0400 2494755746 C Ci:1:010:0 0 4 = 03051000
ee5f0400 2494806599 S Co:1:010:0 s 23 01 0014 0001 0000 0
ee5f0400 2494806751 C Co:1:010:0 0 0
ee5f0400 2494806776 S Co:1:000:0 s 00 05 000b 0000 0000 0
ee5f0400 2494807000 C Co:1:000:0 0 0
ee5f0400 2494818596 S Ci:1:011:0 s 80 06 0100 0000 0012 18 <
ee5f0400 2494820750 C Ci:1:011:0 0 18 = 12010002 00000040 e0131725 00010104 0001
ee5f0400 2494820767 S Ci:1:011:0 s 80 06 0200 0000 0009 9 <
ee5f0400 2494822875 C Ci:1:011:0 0 9 = 09022000 010100c0 00
ee5f0400 2494822887 S Ci:1:011:0 s 80 06 0200 0000 0020 32 <
ee5f0400 2494825126 C Ci:1:011:0 0 32 = 09022000 010100c0 00090400 00020806 50000705 01020002 00070582 02000200
ee5f0400 2494825143 S Ci:1:011:0 s 80 06 0300 0000 00ff 255 <
ee5f0400 2494827251 C Ci:1:011:0 0 4 = 04030904
ee5f0400 2494827263 S Ci:1:011:0 s 80 06 0304 0409 00ff 255 <
ee5f0400 2494829501 C Ci:1:011:0 0 48 = 30035500 53004200 20004d00 61007300 73002000 53007400 6f007200 61006700
ee5f0400 2494829513 S Ci:1:011:0 s 80 06 0301 0409 00ff 255 <
ee5f0400 2494831751 C Ci:1:011:0 0 18 = 12035400 53004500 20004900 6e006300 2e00
ee5f0400 2494837439 S Co:1:011:0 s 00 09 0001 0000 0000 0
ee5f0400 2494837629 C Co:1:011:0 0 0
ee5f0e80 2495843600 S Ci:1:011:0 s a1 fe 0000 0000 0001 1 <
ee5f0e80 2495843829 C Ci:1:011:0 0 1 = 00
ee5f0e80 2495843950 S Bo:1:011:1 -115 31 = 55534243 01000000 24000000 80000612 00000024 00000000 00000000 000000
ee5f0e80 2495844077 C Bo:1:011:1 0 31 >
ee4d7280 2495844088 S Bi:1:011:2 -115 36 <
ee4d7280 2495844577 C Bi:1:011:2 0 36 = 00000000 00000000 55534220 20202020 322e3020 466c6173 68204472 69766500
ee5f0e80 2495844585 S Bi:1:011:2 -115 13 <
ee5f0e80 2495844702 C Bi:1:011:2 0 13 = 55534253 01000000 00000000 00
ee5f0e80 2495844993 S Bo:1:011:1 -115 31 = 55534243 02000000 00000000 00000600 00000000 00000000 00000000 000000
ee5f0e80 2495845078 C Bo:1:011:1 0 31 >
ee5f0e80 2495845112 S Bi:1:011:2 -115 13 <
ee5f0e80 2495845327 C Bi:1:011:2 0 13 = 55534253 02000000 00000000 01
ee5f0e80 2495845338 S Bo:1:011:1 -115 31 = 55534243 03000000 12000000 80000603 00000012 00000000 00000000 000000
ee5f0e80 2495845579 C Bo:1:011:1 0 31 >
ee4d7280 2495845595 S Bi:1:011:2 -115 18 <
ee4d7280 2495845829 C Bi:1:011:2 0 18 = 70000600 0000000a 00000000 3a000000 0000
ee5f0e80 2495845850 S Bi:1:011:2 -115 13 <
ee5f0e80 2495845955 C Bi:1:011:2 0 13 = 55534253 03000000 00000000 00



-- 
Ozan Çağlayan


^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: A really interesting mass-storage BUG introduced after 2.6.18
  2009-01-13 11:28   ` Ozan Çağlayan
@ 2009-01-14 15:47     ` Alan Stern
  2009-01-15  5:13       ` Iwo Mergler
  0 siblings, 1 reply; 6+ messages in thread
From: Alan Stern @ 2009-01-14 15:47 UTC (permalink / raw)
  To: Ozan Çağlayan; +Cc: linux-kernel, linux-usb

On Tue, 13 Jan 2009, [UTF-8] Ozan Çağlayan wrote:

> Here's the usbmon log:
...

Here's the interesting part, right at the end of the log:

> ee5f0e80 2495844993 S Bo:1:011:1 -115 31 = 55534243 02000000 00000000 00000600 00000000 00000000 00000000 000000
> ee5f0e80 2495845078 C Bo:1:011:1 0 31 >
> ee5f0e80 2495845112 S Bi:1:011:2 -115 13 <
> ee5f0e80 2495845327 C Bi:1:011:2 0 13 = 55534253 02000000 00000000 01
> ee5f0e80 2495845338 S Bo:1:011:1 -115 31 = 55534243 03000000 12000000 80000603 00000012 00000000 00000000 000000
> ee5f0e80 2495845579 C Bo:1:011:1 0 31 >
> ee4d7280 2495845595 S Bi:1:011:2 -115 18 <
> ee4d7280 2495845829 C Bi:1:011:2 0 18 = 70000600 0000000a 00000000 3a000000 0000
> ee5f0e80 2495845850 S Bi:1:011:2 -115 13 <
> ee5f0e80 2495845955 C Bi:1:011:2 0 13 = 55534253 03000000 00000000 00

This shows the computer sending a TEST UNIT READY command to the drive, 
and the drive reporting Unit Attention: Media not present.  So of 
course Linux believes there is no media loaded in the drive and doesn't 
try to read the partition table or anything else.

Presumably if the drive is plugged in during bootup then the BIOS does 
something to make this error status go away.  It's hard to guess what 
that would be, however.

Alan Stern


^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: A really interesting mass-storage BUG introduced after 2.6.18
  2009-01-14 15:47     ` Alan Stern
@ 2009-01-15  5:13       ` Iwo Mergler
  2009-01-15 15:13         ` Alan Stern
  0 siblings, 1 reply; 6+ messages in thread
From: Iwo Mergler @ 2009-01-15  5:13 UTC (permalink / raw)
  To: Alan Stern; +Cc: Ozan Çağlayan, linux-kernel, linux-usb

Alan Stern wrote:
> On Tue, 13 Jan 2009, [UTF-8] Ozan Çağlayan wrote:
> 
>> Here's the usbmon log:
> ...
> 
> Here's the interesting part, right at the end of the log:
> 
>> ee5f0e80 2495844993 S Bo:1:011:1 -115 31 = 55534243 02000000 00000000 00000600 00000000 00000000 00000000 000000
>> ee5f0e80 2495845078 C Bo:1:011:1 0 31 >
>> ee5f0e80 2495845112 S Bi:1:011:2 -115 13 <
>> ee5f0e80 2495845327 C Bi:1:011:2 0 13 = 55534253 02000000 00000000 01
>> ee5f0e80 2495845338 S Bo:1:011:1 -115 31 = 55534243 03000000 12000000 80000603 00000012 00000000 00000000 000000
>> ee5f0e80 2495845579 C Bo:1:011:1 0 31 >
>> ee4d7280 2495845595 S Bi:1:011:2 -115 18 <
>> ee4d7280 2495845829 C Bi:1:011:2 0 18 = 70000600 0000000a 00000000 3a000000 0000
>> ee5f0e80 2495845850 S Bi:1:011:2 -115 13 <
>> ee5f0e80 2495845955 C Bi:1:011:2 0 13 = 55534253 03000000 00000000 00
> 
> This shows the computer sending a TEST UNIT READY command to the drive, 
> and the drive reporting Unit Attention: Media not present.  So of 
> course Linux believes there is no media loaded in the drive and doesn't 
> try to read the partition table or anything else.
> 
> Presumably if the drive is plugged in during bootup then the BIOS does 
> something to make this error status go away.  It's hard to guess what 
> that would be, however.
> 

It could be a matter of timing. Some badly designed USB devices have a
dead time between power-up and being ready, but show up on the bus as
if they where. The "Media not present" thing could be a symptom of this.

To verify it, try inserting the device slowly. The power contacts in
the USB connector are longer than the data contacts and the device gets
more time to boot.

Regards,

Iwo



^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: A really interesting mass-storage BUG introduced after 2.6.18
  2009-01-15  5:13       ` Iwo Mergler
@ 2009-01-15 15:13         ` Alan Stern
  0 siblings, 0 replies; 6+ messages in thread
From: Alan Stern @ 2009-01-15 15:13 UTC (permalink / raw)
  To: ozan, Iwo Mergler; +Cc: Kernel development list, USB list

On Thu, 15 Jan 2009, Iwo Mergler wrote:

> > 
> > This shows the computer sending a TEST UNIT READY command to the drive, 
> > and the drive reporting Unit Attention: Media not present.  So of 
> > course Linux believes there is no media loaded in the drive and doesn't 
> > try to read the partition table or anything else.
> > 
> > Presumably if the drive is plugged in during bootup then the BIOS does 
> > something to make this error status go away.  It's hard to guess what 
> > that would be, however.
> > 
> 
> It could be a matter of timing. Some badly designed USB devices have a
> dead time between power-up and being ready, but show up on the bus as
> if they where. The "Media not present" thing could be a symptom of this.

Good point.

> To verify it, try inserting the device slowly. The power contacts in
> the USB connector are longer than the data contacts and the device gets
> more time to boot.

Another timing-sensitive values you can set is the delay_use module
parameter for usb-storage.  It defaults to 5 seconds but perhaps you
need a longer delay.

Alan Stern


^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2009-01-15 15:13 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2009-01-11 11:41 A really interesting mass-storage BUG introduced after 2.6.18 Ozan Çağlayan
2009-01-12 22:19 ` Alan Stern
2009-01-13 11:28   ` Ozan Çağlayan
2009-01-14 15:47     ` Alan Stern
2009-01-15  5:13       ` Iwo Mergler
2009-01-15 15:13         ` Alan Stern

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).