linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [Oops] 2.6.33-rc1: kernel NULL pointer dereference in check_preempt_wakeup
@ 2009-12-23  0:20 Johannes Hirte
  2009-12-23  2:09 ` Américo Wang
  0 siblings, 1 reply; 5+ messages in thread
From: Johannes Hirte @ 2009-12-23  0:20 UTC (permalink / raw)
  To: linux-kernel

With 2.6.33-rc1 I have random hangs. Even SysRq-Key works, only hard reset. 
I've had it mostly within X, where I didn't get any messages but one time on 
console with the following trace:

BUG: unable to handle kernel NULL pointer dereference at 0000000000000090
IP [<ffffffff810263f1>1 check_preempt_wakeup+0xae/0x14e
PGD 10efa1067 PUD 11767a067 PMD 0
Oops: 0000 [#1] SMP
last sysfs file: /sys/devices/pci0000:00/0000:00:10.3/tempt1_input
CPU 0
Pid: 20108, comm: ebuild.sh Not tainted 2.6.33-rc1 #1 TYAN Tiger K8W Dual AMD 
Opteron, S28755/To Be Filled By O.E.M.
RIP: 0010:[<ffffffff810263f1>] [<ffffffff810263f1>] check_preempt_wakeup+0xae/0x14e
RSP: 0018: ffff88006c0e3e50 EFALGS: 000010087
RAX: ffff880028292420 RBX: 0000000000000000 RCX: 0000000000000000
RDX: 0000000000000002 RSI: 0000000000000000 RDI: 0000000001200011
RBP: ffff880105a3eae0 R08: 0000000000000000 R09: 0000000000000001
R10: 0000000001200011 R11: ffff88011cdd0000 R12: ffff8800282123c0
R13: 0000000000000001 R14: 0000000000000000 R15: 0000000000004ed0
FS:  00007f767a6aa700(0000) GS:ffff880028200000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000090 CR§: 00000001061a3000 CR4: 00000000000006f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process ebuild.sh (pid: 20108, threadinfo ffff88006c0e2000, task ffff880105a3eae0)
Stack:
 ffff8800202123c0 ffff88009734c740 0000000001200011 ffff88009734c740
<0> 0000000000004ed0 ffffffff8102e329 0000000000000000 0000000000000286
<0> 00007f767a6aa9d0 0000000000000000 0000000000000000 ffffffff8103042d
Call Trace:
 [<ffffffff8102e329>] ?  wake_up_new_task+0x66/0x8f
 [<ffffffff8103042d>] ? do_fork+0x20f/0x273
 [<ffffffff810a5a9s>] ? alloc_fd+0x69/10a
 [<ffffffff8103a953>] ? sigprocmask+0x9f/0xc2
 [<ffffffff810020d3>] ? stub_clone+0x13/0x20
 [<ffffffff81001e6b>] ? system_call_fastpath+0x16/0x1b
Code: 8b 89 88 00 00 00 ff c0 48 85 c9 75 f2 eb 09 40 0b 9b 88 00 00 00 ff ca 39 
c2 7f f3 eb 09 4d 8b b6 88 00 00 00 ff c8 39 d0 7f f3 <49> 8b 86 90 00 00 00 48 
39 83 90 00 00 00 75 74 4d 85 f6 75 04
RIP  [<ffffffff810263f1>] check_preempt_wakeup+0xae/0x14e
 RSP <ffff88006c0e3e50>
CR2: 0000000000000090
---[ end trace f6bafb05b72ae358 ]---

Since the system didn't respond anymore, I had captured this with camera and 
copied by hand. I hope I haven't made any mistakes when copying. The picture 
wasn't very good.

regards,
  Johannes

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [Oops] 2.6.33-rc1: kernel NULL pointer dereference in  check_preempt_wakeup
  2009-12-23  0:20 [Oops] 2.6.33-rc1: kernel NULL pointer dereference in check_preempt_wakeup Johannes Hirte
@ 2009-12-23  2:09 ` Américo Wang
  2009-12-23  2:59   ` Johannes Hirte
  0 siblings, 1 reply; 5+ messages in thread
From: Américo Wang @ 2009-12-23  2:09 UTC (permalink / raw)
  To: Johannes Hirte; +Cc: linux-kernel

On Wed, Dec 23, 2009 at 8:20 AM, Johannes Hirte
<johannes.hirte@fem.tu-ilmenau.de> wrote:
> With 2.6.33-rc1 I have random hangs. Even SysRq-Key works, only hard reset.
> I've had it mostly within X, where I didn't get any messages but one time on
> console with the following trace:
>
> BUG: unable to handle kernel NULL pointer dereference at 0000000000000090
> IP [<ffffffff810263f1>1 check_preempt_wakeup+0xae/0x14e
> PGD 10efa1067 PUD 11767a067 PMD 0
> Oops: 0000 [#1] SMP
> last sysfs file: /sys/devices/pci0000:00/0000:00:10.3/tempt1_input
> CPU 0
> Pid: 20108, comm: ebuild.sh Not tainted 2.6.33-rc1 #1 TYAN Tiger K8W Dual AMD
> Opteron, S28755/To Be Filled By O.E.M.
> RIP: 0010:[<ffffffff810263f1>] [<ffffffff810263f1>] check_preempt_wakeup+0xae/0x14e
> RSP: 0018: ffff88006c0e3e50 EFALGS: 000010087
> RAX: ffff880028292420 RBX: 0000000000000000 RCX: 0000000000000000
> RDX: 0000000000000002 RSI: 0000000000000000 RDI: 0000000001200011
> RBP: ffff880105a3eae0 R08: 0000000000000000 R09: 0000000000000001
> R10: 0000000001200011 R11: ffff88011cdd0000 R12: ffff8800282123c0
> R13: 0000000000000001 R14: 0000000000000000 R15: 0000000000004ed0
> FS:  00007f767a6aa700(0000) GS:ffff880028200000(0000) knlGS:0000000000000000
> CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> CR2: 0000000000000090 CR§: 00000001061a3000 CR4: 00000000000006f0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Process ebuild.sh (pid: 20108, threadinfo ffff88006c0e2000, task ffff880105a3eae0)
> Stack:
>  ffff8800202123c0 ffff88009734c740 0000000001200011 ffff88009734c740
> <0> 0000000000004ed0 ffffffff8102e329 0000000000000000 0000000000000286
> <0> 00007f767a6aa9d0 0000000000000000 0000000000000000 ffffffff8103042d
> Call Trace:
>  [<ffffffff8102e329>] ?  wake_up_new_task+0x66/0x8f
>  [<ffffffff8103042d>] ? do_fork+0x20f/0x273
>  [<ffffffff810a5a9s>] ? alloc_fd+0x69/10a
>  [<ffffffff8103a953>] ? sigprocmask+0x9f/0xc2
>  [<ffffffff810020d3>] ? stub_clone+0x13/0x20
>  [<ffffffff81001e6b>] ? system_call_fastpath+0x16/0x1b
> Code: 8b 89 88 00 00 00 ff c0 48 85 c9 75 f2 eb 09 40 0b 9b 88 00 00 00 ff ca 39
> c2 7f f3 eb 09 4d 8b b6 88 00 00 00 ff c8 39 d0 7f f3 <49> 8b 86 90 00 00 00 48
> 39 83 90 00 00 00 75 74 4d 85 f6 75 04
> RIP  [<ffffffff810263f1>] check_preempt_wakeup+0xae/0x14e
>  RSP <ffff88006c0e3e50>
> CR2: 0000000000000090
> ---[ end trace f6bafb05b72ae358 ]---

Hmm, seems like the one Eric reported:
http://lkml.org/lkml/2009/12/21/272

>
> Since the system didn't respond anymore, I had captured this with camera and
> copied by hand. I hope I haven't made any mistakes when copying. The picture
> wasn't very good.
>

Wow, hard work. :)

Can you try Peter's patch?

http://lkml.org/lkml/2009/12/22/124

Thanks!

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [Oops] 2.6.33-rc1: kernel NULL pointer dereference in check_preempt_wakeup
  2009-12-23  2:09 ` Américo Wang
@ 2009-12-23  2:59   ` Johannes Hirte
  2009-12-23  3:09     ` Américo Wang
  0 siblings, 1 reply; 5+ messages in thread
From: Johannes Hirte @ 2009-12-23  2:59 UTC (permalink / raw)
  To: Américo Wang; +Cc: linux-kernel

Am Mittwoch 23 Dezember 2009 03:09:47 schrieb Américo Wang:
> On Wed, Dec 23, 2009 at 8:20 AM, Johannes Hirte
> 
> <johannes.hirte@fem.tu-ilmenau.de> wrote:
> > With 2.6.33-rc1 I have random hangs. Even SysRq-Key works, only hard
Aargh, should be: "Even SysRq-Key didn't work"

> > reset. I've had it mostly within X, where I didn't get any messages but
> > one time on console with the following trace:
> >
> > BUG: unable to handle kernel NULL pointer dereference at 0000000000000090
> > IP [<ffffffff810263f1>1 check_preempt_wakeup+0xae/0x14e
> > PGD 10efa1067 PUD 11767a067 PMD 0
> > Oops: 0000 [#1] SMP
> > last sysfs file: /sys/devices/pci0000:00/0000:00:10.3/tempt1_input
> > CPU 0
> > Pid: 20108, comm: ebuild.sh Not tainted 2.6.33-rc1 #1 TYAN Tiger K8W Dual
> > AMD Opteron, S28755/To Be Filled By O.E.M.
> > RIP: 0010:[<ffffffff810263f1>] [<ffffffff810263f1>]
> > check_preempt_wakeup+0xae/0x14e RSP: 0018: ffff88006c0e3e50 EFALGS:
> > 000010087
> > RAX: ffff880028292420 RBX: 0000000000000000 RCX: 0000000000000000
> > RDX: 0000000000000002 RSI: 0000000000000000 RDI: 0000000001200011
> > RBP: ffff880105a3eae0 R08: 0000000000000000 R09: 0000000000000001
> > R10: 0000000001200011 R11: ffff88011cdd0000 R12: ffff8800282123c0
> > R13: 0000000000000001 R14: 0000000000000000 R15: 0000000000004ed0
> > FS:  00007f767a6aa700(0000) GS:ffff880028200000(0000)
> > knlGS:0000000000000000 CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> > CR2: 0000000000000090 CR§: 00000001061a3000 CR4: 00000000000006f0
> > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> > Process ebuild.sh (pid: 20108, threadinfo ffff88006c0e2000, task
> > ffff880105a3eae0) Stack:
> >  ffff8800202123c0 ffff88009734c740 0000000001200011 ffff88009734c740
> > <0> 0000000000004ed0 ffffffff8102e329 0000000000000000 0000000000000286
> > <0> 00007f767a6aa9d0 0000000000000000 0000000000000000 ffffffff8103042d
> > Call Trace:
> >  [<ffffffff8102e329>] ?  wake_up_new_task+0x66/0x8f
> >  [<ffffffff8103042d>] ? do_fork+0x20f/0x273
> >  [<ffffffff810a5a9s>] ? alloc_fd+0x69/10a
> >  [<ffffffff8103a953>] ? sigprocmask+0x9f/0xc2
> >  [<ffffffff810020d3>] ? stub_clone+0x13/0x20
> >  [<ffffffff81001e6b>] ? system_call_fastpath+0x16/0x1b
> > Code: 8b 89 88 00 00 00 ff c0 48 85 c9 75 f2 eb 09 40 0b 9b 88 00 00 00
> > ff ca 39 c2 7f f3 eb 09 4d 8b b6 88 00 00 00 ff c8 39 d0 7f f3 <49> 8b 86
> > 90 00 00 00 48 39 83 90 00 00 00 75 74 4d 85 f6 75 04
> > RIP  [<ffffffff810263f1>] check_preempt_wakeup+0xae/0x14e
> >  RSP <ffff88006c0e3e50>
> > CR2: 0000000000000090
> > ---[ end trace f6bafb05b72ae358 ]---
> 
> Hmm, seems like the one Eric reported:
> http://lkml.org/lkml/2009/12/21/272
> 
> > Since the system didn't respond anymore, I had captured this with camera
> > and copied by hand. I hope I haven't made any mistakes when copying. The
> > picture wasn't very good.
> 
> Wow, hard work. :)
> 
> Can you try Peter's patch?
> 
> http://lkml.org/lkml/2009/12/22/124
> 
> Thanks!

I'll try it. The problem is, I haven't a workload, I can reproduce the hang 
reliably. :(

regards,
  Johannes

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [Oops] 2.6.33-rc1: kernel NULL pointer dereference in  check_preempt_wakeup
  2009-12-23  2:59   ` Johannes Hirte
@ 2009-12-23  3:09     ` Américo Wang
  2009-12-24 19:27       ` Johannes Hirte
  0 siblings, 1 reply; 5+ messages in thread
From: Américo Wang @ 2009-12-23  3:09 UTC (permalink / raw)
  To: Johannes Hirte; +Cc: linux-kernel

On Wed, Dec 23, 2009 at 10:59 AM, Johannes Hirte
<johannes.hirte@fem.tu-ilmenau.de> wrote:
> Am Mittwoch 23 Dezember 2009 03:09:47 schrieb Américo Wang:
>> On Wed, Dec 23, 2009 at 8:20 AM, Johannes Hirte
>>
>> <johannes.hirte@fem.tu-ilmenau.de> wrote:
>> > With 2.6.33-rc1 I have random hangs. Even SysRq-Key works, only hard
> Aargh, should be: "Even SysRq-Key didn't work"
>
>> > reset. I've had it mostly within X, where I didn't get any messages but
>> > one time on console with the following trace:
>> >
>> > BUG: unable to handle kernel NULL pointer dereference at 0000000000000090
>> > IP [<ffffffff810263f1>1 check_preempt_wakeup+0xae/0x14e
>> > PGD 10efa1067 PUD 11767a067 PMD 0
>> > Oops: 0000 [#1] SMP
>> > last sysfs file: /sys/devices/pci0000:00/0000:00:10.3/tempt1_input
>> > CPU 0
>> > Pid: 20108, comm: ebuild.sh Not tainted 2.6.33-rc1 #1 TYAN Tiger K8W Dual
>> > AMD Opteron, S28755/To Be Filled By O.E.M.
>> > RIP: 0010:[<ffffffff810263f1>] [<ffffffff810263f1>]
>> > check_preempt_wakeup+0xae/0x14e RSP: 0018: ffff88006c0e3e50 EFALGS:
>> > 000010087
>> > RAX: ffff880028292420 RBX: 0000000000000000 RCX: 0000000000000000
>> > RDX: 0000000000000002 RSI: 0000000000000000 RDI: 0000000001200011
>> > RBP: ffff880105a3eae0 R08: 0000000000000000 R09: 0000000000000001
>> > R10: 0000000001200011 R11: ffff88011cdd0000 R12: ffff8800282123c0
>> > R13: 0000000000000001 R14: 0000000000000000 R15: 0000000000004ed0
>> > FS:  00007f767a6aa700(0000) GS:ffff880028200000(0000)
>> > knlGS:0000000000000000 CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
>> > CR2: 0000000000000090 CR§: 00000001061a3000 CR4: 00000000000006f0
>> > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> > DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>> > Process ebuild.sh (pid: 20108, threadinfo ffff88006c0e2000, task
>> > ffff880105a3eae0) Stack:
>> >  ffff8800202123c0 ffff88009734c740 0000000001200011 ffff88009734c740
>> > <0> 0000000000004ed0 ffffffff8102e329 0000000000000000 0000000000000286
>> > <0> 00007f767a6aa9d0 0000000000000000 0000000000000000 ffffffff8103042d
>> > Call Trace:
>> >  [<ffffffff8102e329>] ?  wake_up_new_task+0x66/0x8f
>> >  [<ffffffff8103042d>] ? do_fork+0x20f/0x273
>> >  [<ffffffff810a5a9s>] ? alloc_fd+0x69/10a
>> >  [<ffffffff8103a953>] ? sigprocmask+0x9f/0xc2
>> >  [<ffffffff810020d3>] ? stub_clone+0x13/0x20
>> >  [<ffffffff81001e6b>] ? system_call_fastpath+0x16/0x1b
>> > Code: 8b 89 88 00 00 00 ff c0 48 85 c9 75 f2 eb 09 40 0b 9b 88 00 00 00
>> > ff ca 39 c2 7f f3 eb 09 4d 8b b6 88 00 00 00 ff c8 39 d0 7f f3 <49> 8b 86
>> > 90 00 00 00 48 39 83 90 00 00 00 75 74 4d 85 f6 75 04
>> > RIP  [<ffffffff810263f1>] check_preempt_wakeup+0xae/0x14e
>> >  RSP <ffff88006c0e3e50>
>> > CR2: 0000000000000090
>> > ---[ end trace f6bafb05b72ae358 ]---
>>
>> Hmm, seems like the one Eric reported:
>> http://lkml.org/lkml/2009/12/21/272
>>
>> > Since the system didn't respond anymore, I had captured this with camera
>> > and copied by hand. I hope I haven't made any mistakes when copying. The
>> > picture wasn't very good.
>>
>> Wow, hard work. :)
>>
>> Can you try Peter's patch?
>>
>> http://lkml.org/lkml/2009/12/22/124
>>
>> Thanks!
>
> I'll try it. The problem is, I haven't a workload, I can reproduce the hang
> reliably. :(

Amazing...

If that patch doesn't work, please try run addr2line, something like:

 addr2line -e vmlinux ffffffff810263f1

Thanks.

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [Oops] 2.6.33-rc1: kernel NULL pointer dereference in check_preempt_wakeup
  2009-12-23  3:09     ` Américo Wang
@ 2009-12-24 19:27       ` Johannes Hirte
  0 siblings, 0 replies; 5+ messages in thread
From: Johannes Hirte @ 2009-12-24 19:27 UTC (permalink / raw)
  To: Américo Wang; +Cc: linux-kernel

Am Mittwoch 23 Dezember 2009 04:09:42 schrieb Américo Wang:
> On Wed, Dec 23, 2009 at 10:59 AM, Johannes Hirte
> 
> <johannes.hirte@fem.tu-ilmenau.de> wrote:
> > Am Mittwoch 23 Dezember 2009 03:09:47 schrieb Américo Wang:
> >> On Wed, Dec 23, 2009 at 8:20 AM, Johannes Hirte
> >>
> >> <johannes.hirte@fem.tu-ilmenau.de> wrote:
> >> > With 2.6.33-rc1 I have random hangs. Even SysRq-Key works, only hard
> >
> > Aargh, should be: "Even SysRq-Key didn't work"
> >
> >> > reset. I've had it mostly within X, where I didn't get any messages
> >> > but one time on console with the following trace:
> >> >
> >> > BUG: unable to handle kernel NULL pointer dereference at
> >> > 0000000000000090 IP [<ffffffff810263f1>1
> >> > check_preempt_wakeup+0xae/0x14e
> >> > PGD 10efa1067 PUD 11767a067 PMD 0
> >> > Oops: 0000 [#1] SMP
> >> > last sysfs file: /sys/devices/pci0000:00/0000:00:10.3/tempt1_input
> >> > CPU 0
> >> > Pid: 20108, comm: ebuild.sh Not tainted 2.6.33-rc1 #1 TYAN Tiger K8W
> >> > Dual AMD Opteron, S28755/To Be Filled By O.E.M.
> >> > RIP: 0010:[<ffffffff810263f1>] [<ffffffff810263f1>]
> >> > check_preempt_wakeup+0xae/0x14e RSP: 0018: ffff88006c0e3e50 EFALGS:
> >> > 000010087
> >> > RAX: ffff880028292420 RBX: 0000000000000000 RCX: 0000000000000000
> >> > RDX: 0000000000000002 RSI: 0000000000000000 RDI: 0000000001200011
> >> > RBP: ffff880105a3eae0 R08: 0000000000000000 R09: 0000000000000001
> >> > R10: 0000000001200011 R11: ffff88011cdd0000 R12: ffff8800282123c0
> >> > R13: 0000000000000001 R14: 0000000000000000 R15: 0000000000004ed0
> >> > FS:  00007f767a6aa700(0000) GS:ffff880028200000(0000)
> >> > knlGS:0000000000000000 CS:  0010 DS: 0000 ES: 0000 CR0:
> >> > 000000008005003b CR2: 0000000000000090 CR§: 00000001061a3000 CR4:
> >> > 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> >> > 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
> >> > 0000000000000400 Process ebuild.sh (pid: 20108, threadinfo
> >> > ffff88006c0e2000, task ffff880105a3eae0) Stack:
> >> >  ffff8800202123c0 ffff88009734c740 0000000001200011 ffff88009734c740
> >> > <0> 0000000000004ed0 ffffffff8102e329 0000000000000000
> >> > 0000000000000286 <0> 00007f767a6aa9d0 0000000000000000
> >> > 0000000000000000 ffffffff8103042d Call Trace:
> >> >  [<ffffffff8102e329>] ?  wake_up_new_task+0x66/0x8f
> >> >  [<ffffffff8103042d>] ? do_fork+0x20f/0x273
> >> >  [<ffffffff810a5a9s>] ? alloc_fd+0x69/10a
> >> >  [<ffffffff8103a953>] ? sigprocmask+0x9f/0xc2
> >> >  [<ffffffff810020d3>] ? stub_clone+0x13/0x20
> >> >  [<ffffffff81001e6b>] ? system_call_fastpath+0x16/0x1b
> >> > Code: 8b 89 88 00 00 00 ff c0 48 85 c9 75 f2 eb 09 40 0b 9b 88 00 00
> >> > 00 ff ca 39 c2 7f f3 eb 09 4d 8b b6 88 00 00 00 ff c8 39 d0 7f f3 <49>
> >> > 8b 86 90 00 00 00 48 39 83 90 00 00 00 75 74 4d 85 f6 75 04
> >> > RIP  [<ffffffff810263f1>] check_preempt_wakeup+0xae/0x14e
> >> >  RSP <ffff88006c0e3e50>
> >> > CR2: 0000000000000090
> >> > ---[ end trace f6bafb05b72ae358 ]---
> >>
> >> Hmm, seems like the one Eric reported:
> >> http://lkml.org/lkml/2009/12/21/272
> >>
> >> > Since the system didn't respond anymore, I had captured this with
> >> > camera and copied by hand. I hope I haven't made any mistakes when
> >> > copying. The picture wasn't very good.
> >>
> >> Wow, hard work. :)
> >>
> >> Can you try Peter's patch?
> >>
> >> http://lkml.org/lkml/2009/12/22/124
> >>
> >> Thanks!
> >
> > I'll try it. The problem is, I haven't a workload, I can reproduce the
> > hang reliably. :(
>
> Amazing...
> 
> If that patch doesn't work, please try run addr2line, something like:
> 
>  addr2line -e vmlinux ffffffff810263f1

After more than 24 hours without any hang I think the patch fixed it. Thank 
your for the hint.

regards,
  Johannes

^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2009-12-24 19:27 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2009-12-23  0:20 [Oops] 2.6.33-rc1: kernel NULL pointer dereference in check_preempt_wakeup Johannes Hirte
2009-12-23  2:09 ` Américo Wang
2009-12-23  2:59   ` Johannes Hirte
2009-12-23  3:09     ` Américo Wang
2009-12-24 19:27       ` Johannes Hirte

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).