All of lore.kernel.org
 help / color / mirror / Atom feed
* Getting ath5k driver working with Thinkpad T61 [168c:1014]
@ 2008-05-14 17:24 Zack Weinberg
  2008-05-14 18:38 ` Bob Copeland
                   ` (2 more replies)
  0 siblings, 3 replies; 14+ messages in thread
From: Zack Weinberg @ 2008-05-14 17:24 UTC (permalink / raw)
  To: linux-wireless

I have a shiny new Thinkpad T61 and I'd like to get its wireless
chipset working with the ath5k driver.  I have tried
compat-wireless-2008-05-07 on top of kernel 2.6.25.1 -- I get this in
dmesg:

# modprobe ath5k
[15497.699573] ACPI: PCI Interrupt 0000:03:00.0[A] -> GSI 17 (level,
low) -> IRQ 17
[15497.699607] PCI: Setting latency timer of device 0000:03:00.0 to 64
[15497.699665] ath5k_pci 0000:03:00.0: registered as 'phy0'
[15498.709122] ath5k phy0: failed to wakeup the MAC Chip
[15498.709195] ACPI: PCI interrupt for device 0000:03:00.0 disabled
[15498.709205] ath5k_pci: probe of 0000:03:00.0 failed with error -5

# lspci -vnn -s 0000:03:00.0
03:00.0 Ethernet controller [0200]: Atheros Communications Inc. AR5212
802.11abg NIC [168c:1014] (rev 01)
	Subsystem: IBM Device [1014:058a]
	Flags: fast devsel, IRQ 17
	Memory at df3f0000 (64-bit, non-prefetchable) [size=64K]
	Capabilities: [40] Power Management version 2
	Capabilities: [50] Message Signalled Interrupts: Mask- 64bit- Queue=0/0 Enable-
	Capabilities: [60] Express Legacy Endpoint, MSI 00
	Capabilities: [90] MSI-X: Enable- Mask- TabSize=1
	Capabilities: [100] Advanced Error Reporting <?>
	Capabilities: [140] Virtual Channel <?>
	Kernel modules: ath5k

ath_info does not work:
# ath_info df3f0000
Unable to read EEPROM Magic value!

The old madwifi drivers don't work with this chip either:
# modprobe ath_pci
[  392.367300] ath_hal: 0.9.18.0 (AR5210, AR5211, AR5212, RF5111,
RF5112, RF2413, RF5413)
[  392.379623] wlan: 0.9.4
[  392.381431] ath_pci: 0.9.4
[  392.381466] PCI: Enabling device 0000:03:00.0 (0000 -> 0002)
[  392.381472] ACPI: PCI Interrupt 0000:03:00.0[A] -> GSI 17 (level,
low) -> IRQ 17
[  392.381488] PCI: Setting latency timer of device 0000:03:00.0 to 64
[  392.385856] wifi%d: unable to attach hardware: 'Hardware didn't
respond as expected' (HAL status 3)
[  392.385870] ACPI: PCI interrupt for device 0000:03:00.0 disabled

What can I do to help make this chip work in the near future?

zw

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Getting ath5k driver working with Thinkpad T61 [168c:1014]
  2008-05-14 17:24 Getting ath5k driver working with Thinkpad T61 [168c:1014] Zack Weinberg
@ 2008-05-14 18:38 ` Bob Copeland
  2008-05-14 18:45   ` Zack Weinberg
  2008-05-15 19:44 ` Nick Kossifidis
  2008-05-16  0:50 ` Nick Kossifidis
  2 siblings, 1 reply; 14+ messages in thread
From: Bob Copeland @ 2008-05-14 18:38 UTC (permalink / raw)
  To: Zack Weinberg; +Cc: linux-wireless

On Wed, May 14, 2008 at 1:24 PM, Zack Weinberg <zackw@panix.com> wrote:
> [15498.709195] ACPI: PCI interrupt for device 0000:03:00.0 disabled
> [15498.709205] ath5k_pci: probe of 0000:03:00.0 failed with error -5

Did you try power-cycling your machine?  Looks like chip is hung.

-- 
Bob Copeland %% www.bobcopeland.com

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Getting ath5k driver working with Thinkpad T61 [168c:1014]
  2008-05-14 18:38 ` Bob Copeland
@ 2008-05-14 18:45   ` Zack Weinberg
  2008-05-14 19:09     ` Bob Copeland
  0 siblings, 1 reply; 14+ messages in thread
From: Zack Weinberg @ 2008-05-14 18:45 UTC (permalink / raw)
  To: Bob Copeland; +Cc: linux-wireless

On Wed, May 14, 2008 at 2:38 PM, Bob Copeland <me@bobcopeland.com> wrote:
> On Wed, May 14, 2008 at 1:24 PM, Zack Weinberg <zackw@panix.com> wrote:
>> [15498.709195] ACPI: PCI interrupt for device 0000:03:00.0 disabled
>> [15498.709205] ath5k_pci: probe of 0000:03:00.0 failed with error -5
>
> Did you try power-cycling your machine?  Looks like chip is hung.

exactly the same thing happens immediately after cold boot with
nothing else having touched the chip.

zw

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Getting ath5k driver working with Thinkpad T61 [168c:1014]
  2008-05-14 18:45   ` Zack Weinberg
@ 2008-05-14 19:09     ` Bob Copeland
  2008-05-15 14:41       ` Zack Weinberg
  0 siblings, 1 reply; 14+ messages in thread
From: Bob Copeland @ 2008-05-14 19:09 UTC (permalink / raw)
  To: Zack Weinberg; +Cc: linux-wireless

On Wed, May 14, 2008 at 2:45 PM, Zack Weinberg <zackw@panix.com> wrote:
> On Wed, May 14, 2008 at 2:38 PM, Bob Copeland <me@bobcopeland.com> wrote:
>> On Wed, May 14, 2008 at 1:24 PM, Zack Weinberg <zackw@panix.com> wrote:
>>> [15498.709195] ACPI: PCI interrupt for device 0000:03:00.0 disabled
>>> [15498.709205] ath5k_pci: probe of 0000:03:00.0 failed with error -5
>>
>> Did you try power-cycling your machine?  Looks like chip is hung.
>
> exactly the same thing happens immediately after cold boot with
> nothing else having touched the chip.

Interesting...  can you confirm that madwifi also will not start from a cold
boot?  I only ask because I had a similar issue with my pci-e card that was
fixed a while ago.  Madwifi would work from a cold boot, though, if ath5k
had never been loaded.

Here's someone else with the same PCI ID, same symptom:

http://kerneltrap.org/mailarchive/linux-ath5k-devel/2007/12/12/496961

Perhaps the best thing to do is add printks to ath5k_hw_nic_wakeup and see
where it hangs?

-- 
Bob Copeland %% www.bobcopeland.com

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Getting ath5k driver working with Thinkpad T61 [168c:1014]
  2008-05-14 19:09     ` Bob Copeland
@ 2008-05-15 14:41       ` Zack Weinberg
  2008-05-15 16:45         ` Bob Copeland
  0 siblings, 1 reply; 14+ messages in thread
From: Zack Weinberg @ 2008-05-15 14:41 UTC (permalink / raw)
  To: Bob Copeland; +Cc: linux-wireless

On Wed, May 14, 2008 at 3:09 PM, Bob Copeland <me@bobcopeland.com> wrote:
> On Wed, May 14, 2008 at 2:45 PM, Zack Weinberg <zackw@panix.com> wrote:
>> exactly the same thing happens immediately after cold boot with
>> nothing else having touched the chip.
>
> Interesting...  can you confirm that madwifi also will not start from a cold
> boot?

Yes, madwifi will not start from a cold boot.

> Perhaps the best thing to do is add printks to ath5k_hw_nic_wakeup and see
> where it hangs?

It's not hanging. The very first call to ath5k_hw_set_power(ah,
AR5K_PM_AWAKE, true, 0); from ath5k_hw_nic_wakeup fails with -EIO, and
the driver gives up.

zw

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Getting ath5k driver working with Thinkpad T61 [168c:1014]
  2008-05-15 14:41       ` Zack Weinberg
@ 2008-05-15 16:45         ` Bob Copeland
  2008-05-15 16:53           ` Zack Weinberg
  0 siblings, 1 reply; 14+ messages in thread
From: Bob Copeland @ 2008-05-15 16:45 UTC (permalink / raw)
  To: Zack Weinberg; +Cc: linux-wireless

On Thu, May 15, 2008 at 10:41 AM, Zack Weinberg <zackw@panix.com> wrote:
>> Interesting...  can you confirm that madwifi also will not start from a cold
>> boot?
>
> Yes, madwifi will not start from a cold boot.

Ugh, well that is not good.  Does ndiswrapper work?  Unless anyone
else has ideas, I would suggest capturing a trace using mmiotrace and
ndiswrapper to see what is going on.  Someone on ath5k-devel list
might have a better idea.

>> Perhaps the best thing to do is add printks to ath5k_hw_nic_wakeup and see
>> where it hangs?
>
> It's not hanging. The very first call to ath5k_hw_set_power(ah,
> AR5K_PM_AWAKE, true, 0); from ath5k_hw_nic_wakeup fails with -EIO, and
> the driver gives up.

By 'hanging' I mean the register writes cease to do anything useful
and the card basically stops responding until you power off.  Madwifi
will even give "Card didn't behave as expected" in this case.  -EIO
means it couldn't wake up the chip after 1s, so if the first call to
ath5k_hw_set_power that doesn't work, we'll need a trace to see what
differences there are in a working setup.

-- 
Bob Copeland %% www.bobcopeland.com

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Getting ath5k driver working with Thinkpad T61 [168c:1014]
  2008-05-15 16:45         ` Bob Copeland
@ 2008-05-15 16:53           ` Zack Weinberg
  2008-05-15 20:44             ` Bob Copeland
  0 siblings, 1 reply; 14+ messages in thread
From: Zack Weinberg @ 2008-05-15 16:53 UTC (permalink / raw)
  To: Bob Copeland; +Cc: linux-wireless

On Thu, May 15, 2008 at 12:45 PM, Bob Copeland <me@bobcopeland.com> wrote:
> On Thu, May 15, 2008 at 10:41 AM, Zack Weinberg <zackw@panix.com> wrote:
>>> Interesting...  can you confirm that madwifi also will not start from a cold
>>> boot?
>>
>> Yes, madwifi will not start from a cold boot.
>
> Ugh, well that is not good.  Does ndiswrapper work?

I haven't tried it, but wireless does work under Windows (Vista,
vendor installation).

> Unless anyone
> else has ideas, I would suggest capturing a trace using mmiotrace and
> ndiswrapper to see what is going on.  Someone on ath5k-devel list
> might have a better idea.

I don't know how to do that, can you point me at instructions?

>> It's not hanging. The very first call to ath5k_hw_set_power(ah,
>> AR5K_PM_AWAKE, true, 0); from ath5k_hw_nic_wakeup fails with -EIO, and
>> the driver gives up.
>
> By 'hanging' I mean the register writes cease to do anything useful
> and the card basically stops responding until you power off.  Madwifi
> will even give "Card didn't behave as expected" in this case

Oh, okay.  That is indeed the error message I'm getting from madwifi.

zw

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Getting ath5k driver working with Thinkpad T61 [168c:1014]
  2008-05-14 17:24 Getting ath5k driver working with Thinkpad T61 [168c:1014] Zack Weinberg
  2008-05-14 18:38 ` Bob Copeland
@ 2008-05-15 19:44 ` Nick Kossifidis
  2008-05-15 19:53   ` Zack Weinberg
  2008-05-16  0:50 ` Nick Kossifidis
  2 siblings, 1 reply; 14+ messages in thread
From: Nick Kossifidis @ 2008-05-15 19:44 UTC (permalink / raw)
  To: Zack Weinberg; +Cc: linux-wireless

2008/5/14 Zack Weinberg <zackw@panix.com>:
> I have a shiny new Thinkpad T61 and I'd like to get its wireless
> chipset working with the ath5k driver.  I have tried
> compat-wireless-2008-05-07 on top of kernel 2.6.25.1 -- I get this in
> dmesg:
>
> # modprobe ath5k
> [15497.699573] ACPI: PCI Interrupt 0000:03:00.0[A] -> GSI 17 (level,
> low) -> IRQ 17
> [15497.699607] PCI: Setting latency timer of device 0000:03:00.0 to 64
> [15497.699665] ath5k_pci 0000:03:00.0: registered as 'phy0'
> [15498.709122] ath5k phy0: failed to wakeup the MAC Chip
> [15498.709195] ACPI: PCI interrupt for device 0000:03:00.0 disabled
> [15498.709205] ath5k_pci: probe of 0000:03:00.0 failed with error -5
>
> # lspci -vnn -s 0000:03:00.0
> 03:00.0 Ethernet controller [0200]: Atheros Communications Inc. AR5212
> 802.11abg NIC [168c:1014] (rev 01)
>        Subsystem: IBM Device [1014:058a]
>        Flags: fast devsel, IRQ 17
>        Memory at df3f0000 (64-bit, non-prefetchable) [size=64K]
>        Capabilities: [40] Power Management version 2
>        Capabilities: [50] Message Signalled Interrupts: Mask- 64bit- Queue=0/0 Enable-
>        Capabilities: [60] Express Legacy Endpoint, MSI 00
>        Capabilities: [90] MSI-X: Enable- Mask- TabSize=1
>        Capabilities: [100] Advanced Error Reporting <?>
>        Capabilities: [140] Virtual Channel <?>
>        Kernel modules: ath5k
>
> ath_info does not work:
> # ath_info df3f0000
> Unable to read EEPROM Magic value!
>
> The old madwifi drivers don't work with this chip either:
> # modprobe ath_pci
> [  392.367300] ath_hal: 0.9.18.0 (AR5210, AR5211, AR5212, RF5111,
> RF5112, RF2413, RF5413)
> [  392.379623] wlan: 0.9.4
> [  392.381431] ath_pci: 0.9.4
> [  392.381466] PCI: Enabling device 0000:03:00.0 (0000 -> 0002)
> [  392.381472] ACPI: PCI Interrupt 0000:03:00.0[A] -> GSI 17 (level,
> low) -> IRQ 17
> [  392.381488] PCI: Setting latency timer of device 0000:03:00.0 to 64
> [  392.385856] wifi%d: unable to attach hardware: 'Hardware didn't
> respond as expected' (HAL status 3)
> [  392.385870] ACPI: PCI interrupt for device 0000:03:00.0 disabled
>
> What can I do to help make this chip work in the near future?
>
> zw

Are you sure that ath5k from compat-wireless is being used and not the
one that comes with 2.6.25 ?
In ath5k that comes with 2.6.25 the fix Bob mentioned is not included
and PCI-E cards hang.
Have you tried blacklisting ath5k, power-cycle and only use
madwifi/ndiswrapper ?

If your card doesn't work even after disabling ath5k and power-cycle
(reboot won't work, you'll have to actually turn your system off and
on) then you might have some hw problem.


-- 
GPG ID: 0xD21DB2DB
As you read this post global entropy rises. Have Fun ;-)
Nick

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Getting ath5k driver working with Thinkpad T61 [168c:1014]
  2008-05-15 19:44 ` Nick Kossifidis
@ 2008-05-15 19:53   ` Zack Weinberg
  0 siblings, 0 replies; 14+ messages in thread
From: Zack Weinberg @ 2008-05-15 19:53 UTC (permalink / raw)
  To: Nick Kossifidis; +Cc: linux-wireless

On Thu, May 15, 2008 at 3:44 PM, Nick Kossifidis <mickflemm@gmail.com> wrote:
> Are you sure that ath5k from compat-wireless is being used and not the
> one that comes with 2.6.25 ?

Yes.

> Have you tried blacklisting ath5k, power-cycle and only use
> madwifi/ndiswrapper ?

Only madwifi, not ndiswrapper.

> If your card doesn't work even after disabling ath5k and power-cycle
> (reboot won't work, you'll have to actually turn your system off and
> on)

Did that.

> then you might have some hw problem.

Doubt it.  It works fine in Windows, after all.

zw

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Getting ath5k driver working with Thinkpad T61 [168c:1014]
  2008-05-15 16:53           ` Zack Weinberg
@ 2008-05-15 20:44             ` Bob Copeland
  0 siblings, 0 replies; 14+ messages in thread
From: Bob Copeland @ 2008-05-15 20:44 UTC (permalink / raw)
  To: Zack Weinberg; +Cc: linux-wireless, Nick Kossifidis

On Thu, May 15, 2008 at 12:53 PM, Zack Weinberg <zackw@panix.com> wrote:
>> Unless anyone
>> else has ideas, I would suggest capturing a trace using mmiotrace and
>> ndiswrapper to see what is going on.  Someone on ath5k-devel list
>> might have a better idea.
>
> I don't know how to do that, can you point me at instructions?

There are some instructions over here, but they are a little out of date:

    http://madwifi.org/wiki/DevDocs/MadwifiTrace

I think believe the info from the nouveau link below is more recent.
It's changed a lot even since the last time I ran it a month or two
ago, so you may have to do a little research.  "Built-in mmiotrace"
seems to be where it's going lately.

    http://nouveau.freedesktop.org/wiki/MmioTrace

First step would be to get your card working under ndiswrapper, I guess.

-- 
Bob Copeland %% www.bobcopeland.com

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Getting ath5k driver working with Thinkpad T61 [168c:1014]
  2008-05-14 17:24 Getting ath5k driver working with Thinkpad T61 [168c:1014] Zack Weinberg
  2008-05-14 18:38 ` Bob Copeland
  2008-05-15 19:44 ` Nick Kossifidis
@ 2008-05-16  0:50 ` Nick Kossifidis
  2008-06-03 21:45   ` Zack Weinberg
  2 siblings, 1 reply; 14+ messages in thread
From: Nick Kossifidis @ 2008-05-16  0:50 UTC (permalink / raw)
  To: Zack Weinberg; +Cc: linux-wireless

> ath_info does not work:
> # ath_info df3f0000
> Unable to read EEPROM Magic value!
>

Hm, have you tried latest ath_info from
http://madwifi.org/browser/ath_info/trunk

This one looks interesting, plz use ndiswrapper for your card and use
mmiotrace to see
what windows driver does in your situation that makes the card work.

Follow the link Bob gave you, it has instructions for 2.6.25. Remember
if you have enabled
4K stacks on your kernel configuration to disable it otherwise
ndiswrapper won't work.

-- 
GPG ID: 0xD21DB2DB
As you read this post global entropy rises. Have Fun ;-)
Nick

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Getting ath5k driver working with Thinkpad T61 [168c:1014]
  2008-05-16  0:50 ` Nick Kossifidis
@ 2008-06-03 21:45   ` Zack Weinberg
  2008-06-05 21:50     ` Bob Copeland
  0 siblings, 1 reply; 14+ messages in thread
From: Zack Weinberg @ 2008-06-03 21:45 UTC (permalink / raw)
  To: Nick Kossifidis; +Cc: linux-wireless

[-- Attachment #1: Type: text/plain, Size: 1402 bytes --]

I'm sorry to have taken so long to get back to y'all.  It turns out I
was wrong about madwifi, it does work with this chipset.  Either I was
mistaken in thinking that the device had been untouched up to the
point I tried to load ath_pci, or a change in a 2.6.25.x point release
fixed it.  ath_info also works now.

>> ath_info does not work:
>> # ath_info df3f0000

It now says:

Warning: Invalid EEPROM Magic number!
 -==Device Information==-
MAC Version:  2424  (0xa0)
MAC Revision: 5424  (0xa3)
5Ghz PHY Revision: SChip (0xa2)
2Ghz PHY Revision: SChip (0xa2)
 -==EEPROM Information==-
EEPROM Version:     5.3
EEPROM Size:        4K
Regulatory Domain:  0x62
 -==== Capabilities ====-
|  802.11a Support: yes  |
|  802.11b Support: yes  |
|  802.11g Support: yes  |
|  RFKill  Support: yes  |
|  32KHz   Crystal: no   |
 ========================
GPIO registers: CR 00018003 DO 00000000 DI 0000000a

> This one looks interesting, plz use ndiswrapper for your card and use
> mmiotrace to see
> what windows driver does in your situation that makes the card work.

Since it works now with madwifi, I am attaching mmiotrace runs for
ath5k (latest compat-wireless driver, which still doesn't work) and
madwifi.  Please let me know if you still want to see traces for the
windows driver.  (The madwifi trace also includes a successful "iwlist
wlan0 scan" operation; the ath5k trace obviously does not.)

zw

[-- Attachment #2: cpu0.ath5k.gz --]
[-- Type: application/x-gzip, Size: 33541 bytes --]

[-- Attachment #3: cpu0.madwifi.gz --]
[-- Type: application/x-gzip, Size: 489447 bytes --]

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Getting ath5k driver working with Thinkpad T61 [168c:1014]
  2008-06-03 21:45   ` Zack Weinberg
@ 2008-06-05 21:50     ` Bob Copeland
  0 siblings, 0 replies; 14+ messages in thread
From: Bob Copeland @ 2008-06-05 21:50 UTC (permalink / raw)
  To: Zack Weinberg; +Cc: Nick Kossifidis, linux-wireless

On Tue, Jun 3, 2008 at 5:45 PM, Zack Weinberg <zackw@panix.com> wrote:
> I'm sorry to have taken so long to get back to y'all.  It turns out I
> was wrong about madwifi, it does work with this chipset.

Great!

> Since it works now with madwifi, I am attaching mmiotrace runs for
> ath5k (latest compat-wireless driver, which still doesn't work) and
> madwifi.

Any changes in the dmesg output when you use ath5k?  I looked very briefly and
didn't see the usual symptom of a locked up card (reads of 0xffffffff), but it
also looks like there is only .08 seconds covered by the trace.  Could you try
another trace for a bit longer?

> Please let me know if you still want to see traces for the
> windows driver.  (The madwifi trace also includes a successful "iwlist
> wlan0 scan" operation; the ath5k trace obviously does not.)

Probably not necessary -- the madwifi trace looks fine.

Thanks!
-- 
Bob Copeland %% www.bobcopeland.com

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Getting ath5k driver working with Thinkpad T61 [168c:1014]
@ 2008-06-16 15:24 Bob Copeland
  0 siblings, 0 replies; 14+ messages in thread
From: Bob Copeland @ 2008-06-16 15:24 UTC (permalink / raw)
  To: Zack Weinberg, Nick Kossifidis, linux-wireless

> Attached.  This has two attempts at 'iwlist wlan0 scan' in there
> before I shut it back down again.  I see an awful lot of repetitive
> writes of the same value at the same address, but I don't know what
> this stuff means...

Thanks for the update.  I haven't had a chance to look at this until 
now, as I've been traveling the last couple of weeks.  I'll go through 
it this evening to see if anything jumps out.

FWIW, here's how to read the dump.  Given a line like:

    R 4 48.700287 1 0xdf3f9860 0x9d1a 0x0 0

>From left to right that would be:

Read 4 bytes at time 48.700287 (map id=1) the address 0xdf3f9860, and
receive the value 0x9d1a (the other two 0s are meaningless for now).  
Writes look the same except start with W.

>From the address, you can figure out the register: in this case it is 
0x9860 since the MMIO base address is 0xdf3f0000.  Now you can look in 
reg.h to find the name of that register if we know it (here it is
AR5K_PHY_AGCCTL).

Then you can look at the source for where that register is being used to
get an idea of what is going on.  In the above case, we're reading the
register until the hardware completes AGC calibration and sets 
AR5K_PHY_AGCCTL_CAL (bit 0) low - that eventually happens .02 seconds 
later when the value 0x9d18 is read.

-- 
Bob Copeland %% www.bobcopeland.com



^ permalink raw reply	[flat|nested] 14+ messages in thread

end of thread, other threads:[~2008-06-16 15:24 UTC | newest]

Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2008-05-14 17:24 Getting ath5k driver working with Thinkpad T61 [168c:1014] Zack Weinberg
2008-05-14 18:38 ` Bob Copeland
2008-05-14 18:45   ` Zack Weinberg
2008-05-14 19:09     ` Bob Copeland
2008-05-15 14:41       ` Zack Weinberg
2008-05-15 16:45         ` Bob Copeland
2008-05-15 16:53           ` Zack Weinberg
2008-05-15 20:44             ` Bob Copeland
2008-05-15 19:44 ` Nick Kossifidis
2008-05-15 19:53   ` Zack Weinberg
2008-05-16  0:50 ` Nick Kossifidis
2008-06-03 21:45   ` Zack Weinberg
2008-06-05 21:50     ` Bob Copeland
2008-06-16 15:24 Bob Copeland

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.