linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* intel_ips produces constant load of 1
@ 2011-03-19 15:38 Andreas Hartmann
  2011-03-21 18:04 ` Jesse Barnes
  0 siblings, 1 reply; 10+ messages in thread
From: Andreas Hartmann @ 2011-03-19 15:38 UTC (permalink / raw)
  To: linux-kernel

Hello,

on my MSI CR620 laptop, intel_ips produces a constant load of 1, even if
the machine is idle.

The ips-monitor hangs in D state:

ps aux | grep ips
root       593  0.0  0.0      0     0 ?        S    17:20   0:00
[ips-adjust]
root       594  0.0  0.0      0     0 ?        D    17:20   0:00
[ips-monitor]

If the module isn't loaded, the load of the machine in idle mode is 0 as
expected.

Some more information about ips on this machine:

[   61.689778] intel ips 0000:00:1f.6: CPU TDP doesn't match expected
value (found 25, expected 29)
[   61.689805] intel ips 0000:00:1f.6: PCI INT C -> GSI 18 (level, low)
-> IRQ 18
[   61.690236] intel ips 0000:00:1f.6: IPS driver initialized, MCP temp
limit 90

kernel: [17208.187382] intel ips 0000:00:1f.6: MCP limit exceeded: Avg
power 35102, limit 35000
kernel: [50357.822192] intel ips 0000:00:1f.6: MCP limit exceeded: Avg
power 39335, limit 35000Mar 17 18:46:31 linux-8fih kernel:
[50362.819118] intel ips 0000:00:1f.6: MCP limit exceeded: Avg power
39823, limit 35000Mar 17 18:46:36 linux-8fih kernel: [50367.816045]
intel ips 0000:00:1f.6: MCP limit exceeded: Avg power 40540, limit
35000Mar 17 18:46:46 linux-8fih kernel: [50377.809947] intel ips
0000:00:1f.6: MCP limit exceeded: Avg power 36915, limit 35000Mar 17
18:47:01 linux-8fih kernel: [50392.800632] intel ips 0000:00:1f.6: MCP
limit exceeded: Avg power 39427, limit 35000Mar 17 18:47:11 linux-8fih
kernel: [50402.794479] intel ips 0000:00:1f.6: MCP limit exceeded: Avg
power 39781, limit 35000
kernel: [50407.791447] intel ips 0000:00:1f.6: MCP limit exceeded: Avg
power 37437, limit 35000
kernel: [50417.785250] intel ips 0000:00:1f.6: MCP limit exceeded: Avg
power 37682, limit 35000
kernel: [50422.782204] intel ips 0000:00:1f.6: MCP limit exceeded: Avg
power 35474, limit 35000


The CPU is:
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 37
model name      : Intel(R) Core(TM) i5 CPU       M 460  @ 2.53GHz
stepping        : 5
cpu MHz         : 2533.000
cache size      : 3072 KB
physical id     : 0
siblings        : 4
core id         : 0
cpu cores       : 2
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 11
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe
syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl
xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2
ssse3 cx16 xtpr pdcm sse4_1 sse4_2 popcnt lahf_lm arat dts tpr_shadow
vnmi flexpriority ept vpid
bogomips        : 5053.43
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

(+ another 3 times)



lspci of the signal processing controller:

00:1f.6 Signal processing controller: Intel Corporation 5 Series/3400
Series Chipset Thermal Subsystem (rev 05)
        Subsystem: Micro-Star International Co., Ltd. Device 1033
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin C routed to IRQ 11
        Region 0: Memory at f6804000 (64-bit, non-prefetchable) [size=4K]
        Capabilities: [50] Power Management version 3
                Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit-
                Address: 00000000  Data: 0000


Do you need some more information? Please ask, I'll try to provide it!


Regards,
Andreas

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: intel_ips produces constant load of 1
  2011-03-19 15:38 intel_ips produces constant load of 1 Andreas Hartmann
@ 2011-03-21 18:04 ` Jesse Barnes
  2011-03-22  7:00   ` Andreas Hartmann
  2011-03-22 20:25   ` Jesse Barnes
  0 siblings, 2 replies; 10+ messages in thread
From: Jesse Barnes @ 2011-03-21 18:04 UTC (permalink / raw)
  To: Andreas Hartmann; +Cc: linux-kernel

On Sat, 19 Mar 2011 16:38:38 +0100
Andreas Hartmann <andihartmann@01019freenet.de> wrote:

> Hello,
> 
> on my MSI CR620 laptop, intel_ips produces a constant load of 1, even if
> the machine is idle.
> 
> The ips-monitor hangs in D state:
> 
> ps aux | grep ips
> root       593  0.0  0.0      0     0 ?        S    17:20   0:00
> [ips-adjust]
> root       594  0.0  0.0      0     0 ?        D    17:20   0:00
> [ips-monitor]
> 
> If the module isn't loaded, the load of the machine in idle mode is 0 as
> expected.

This is a reporting problem, and probably due to the schedule() call
and associated task state in the ips-monitor thread.  I thought setting
the task state to interruptible would prevent this, but it seems like
it's not enough for the deferrable on-stack timers?

At any rate, it's not actually causing increased CPU usage, so you can
safely ignore it until we have a fix.

-- 
Jesse Barnes, Intel Open Source Technology Center

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: intel_ips produces constant load of 1
  2011-03-21 18:04 ` Jesse Barnes
@ 2011-03-22  7:00   ` Andreas Hartmann
  2011-03-22 15:49     ` Jesse Barnes
  2011-03-22 20:25   ` Jesse Barnes
  1 sibling, 1 reply; 10+ messages in thread
From: Andreas Hartmann @ 2011-03-22  7:00 UTC (permalink / raw)
  To: Jesse Barnes; +Cc: linux-kernel

Jesse Barnes wrote:
> On Sat, 19 Mar 2011 16:38:38 +0100
> Andreas Hartmann <andihartmann@01019freenet.de> wrote:
> 
>> Hello,
>>
>> on my MSI CR620 laptop, intel_ips produces a constant load of 1, even if
>> the machine is idle.
>>
>> The ips-monitor hangs in D state:
>>
>> ps aux | grep ips
>> root       593  0.0  0.0      0     0 ?        S    17:20   0:00
>> [ips-adjust]
>> root       594  0.0  0.0      0     0 ?        D    17:20   0:00
>> [ips-monitor]
>>
>> If the module isn't loaded, the load of the machine in idle mode is 0 as
>> expected.
> 
> This is a reporting problem, and probably due to the schedule() call
> and associated task state in the ips-monitor thread.  I thought setting
> the task state to interruptible would prevent this, but it seems like
> it's not enough for the deferrable on-stack timers?
> 
> At any rate, it's not actually causing increased CPU usage, so you can
> safely ignore it until we have a fix.
> 

I've got one more question (I found
http://forum.soft32.com/linux/RFC-Intelligent-power-sharing-driver-ftopict510146.html):

Where is the difference to the functionality the bios provides? I can't
see (and hear :-)) any difference between with intel_ips and without
intel_ips.
The fan always runs at a minimal speed, very quiet.
The fan is getting loader, if the load is getting high (during
compile-sessions e.g.). If the compile session is ready, the fan get's
slower again.

If I check the "cpu MHz" in /proc/cpuinfo with or without intel_ips, I
can't see any difference. The value never exceeds the value given in the
model name. The lowest values are equal, too.


What is the added value compared to the bios functionality? How can I
check it?



Thank you for your help!
Andreas

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: intel_ips produces constant load of 1
  2011-03-22  7:00   ` Andreas Hartmann
@ 2011-03-22 15:49     ` Jesse Barnes
  0 siblings, 0 replies; 10+ messages in thread
From: Jesse Barnes @ 2011-03-22 15:49 UTC (permalink / raw)
  To: Andreas Hartmann; +Cc: linux-kernel

On Tue, 22 Mar 2011 08:00:39 +0100
Andreas Hartmann <andihartmann@freenet.de> wrote:

> Jesse Barnes wrote:
> > On Sat, 19 Mar 2011 16:38:38 +0100
> > Andreas Hartmann <andihartmann@01019freenet.de> wrote:
> > 
> >> Hello,
> >>
> >> on my MSI CR620 laptop, intel_ips produces a constant load of 1, even if
> >> the machine is idle.
> >>
> >> The ips-monitor hangs in D state:
> >>
> >> ps aux | grep ips
> >> root       593  0.0  0.0      0     0 ?        S    17:20   0:00
> >> [ips-adjust]
> >> root       594  0.0  0.0      0     0 ?        D    17:20   0:00
> >> [ips-monitor]
> >>
> >> If the module isn't loaded, the load of the machine in idle mode is 0 as
> >> expected.
> > 
> > This is a reporting problem, and probably due to the schedule() call
> > and associated task state in the ips-monitor thread.  I thought setting
> > the task state to interruptible would prevent this, but it seems like
> > it's not enough for the deferrable on-stack timers?
> > 
> > At any rate, it's not actually causing increased CPU usage, so you can
> > safely ignore it until we have a fix.
> > 
> 
> I've got one more question (I found
> http://forum.soft32.com/linux/RFC-Intelligent-power-sharing-driver-ftopict510146.html):
> 
> Where is the difference to the functionality the bios provides? I can't
> see (and hear :-)) any difference between with intel_ips and without
> intel_ips.
> The fan always runs at a minimal speed, very quiet.
> The fan is getting loader, if the load is getting high (during
> compile-sessions e.g.). If the compile session is ready, the fan get's
> slower again.
> 
> If I check the "cpu MHz" in /proc/cpuinfo with or without intel_ips, I
> can't see any difference. The value never exceeds the value given in the
> model name. The lowest values are equal, too.
> 
> 
> What is the added value compared to the bios functionality? How can I
> check it?

Loading the IPS driver will allow graphics turbo.  This can improve gfx
performance quite a bit (up to 3x on some synthetic workloads).  If
you're not really using gfx though, the driver won't really do anything
for you.

-- 
Jesse Barnes, Intel Open Source Technology Center

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: intel_ips produces constant load of 1
  2011-03-21 18:04 ` Jesse Barnes
  2011-03-22  7:00   ` Andreas Hartmann
@ 2011-03-22 20:25   ` Jesse Barnes
  2011-03-23 18:15     ` Andreas Hartmann
  2011-03-25 18:23     ` Andreas Hartmann
  1 sibling, 2 replies; 10+ messages in thread
From: Jesse Barnes @ 2011-03-22 20:25 UTC (permalink / raw)
  To: Jesse Barnes; +Cc: Andreas Hartmann, linux-kernel

On Mon, 21 Mar 2011 11:04:04 -0700
Jesse Barnes <jbarnes@virtuousgeek.org> wrote:

> > ps aux | grep ips
> > root       593  0.0  0.0      0     0 ?        S    17:20   0:00
> > [ips-adjust]
> > root       594  0.0  0.0      0     0 ?        D    17:20   0:00
> > [ips-monitor]
> > 
> > If the module isn't loaded, the load of the machine in idle mode is 0 as
> > expected.  
> 
> This is a reporting problem, and probably due to the schedule() call
> and associated task state in the ips-monitor thread.  I thought setting
> the task state to interruptible would prevent this, but it seems like
> it's not enough for the deferrable on-stack timers?
> 
> At any rate, it's not actually causing increased CPU usage, so you can
> safely ignore it until we have a fix.

Oops, one task uses interruptible correctly, but the monitor thread
doesn't.

Does this patch fix your load average?

-- 
Jesse Barnes, Intel Open Source Technology Center

diff --git a/drivers/platform/x86/intel_ips.c b/drivers/platform/x86/intel_ips.c
index 1294a39..85c8ad4 100644
--- a/drivers/platform/x86/intel_ips.c
+++ b/drivers/platform/x86/intel_ips.c
@@ -1111,7 +1111,7 @@ static int ips_monitor(void *data)
                last_msecs = jiffies_to_msecs(jiffies);
                expire = jiffies + msecs_to_jiffies(IPS_SAMPLE_PERIOD);
 
-               __set_current_state(TASK_UNINTERRUPTIBLE);
+               __set_current_state(TASK_INTERRUPTIBLE);
                mod_timer(&timer, expire);
                schedule();
 

^ permalink raw reply related	[flat|nested] 10+ messages in thread

* Re: intel_ips produces constant load of 1
  2011-03-22 20:25   ` Jesse Barnes
@ 2011-03-23 18:15     ` Andreas Hartmann
  2011-03-23 18:20       ` Jesse Barnes
  2011-03-25 18:23     ` Andreas Hartmann
  1 sibling, 1 reply; 10+ messages in thread
From: Andreas Hartmann @ 2011-03-23 18:15 UTC (permalink / raw)
  To: Jesse Barnes; +Cc: linux-kernel

Jesse Barnes wrote:
> On Mon, 21 Mar 2011 11:04:04 -0700
> Jesse Barnes <jbarnes@virtuousgeek.org> wrote:
> 
>>> ps aux | grep ips
>>> root       593  0.0  0.0      0     0 ?        S    17:20   0:00
>>> [ips-adjust]
>>> root       594  0.0  0.0      0     0 ?        D    17:20   0:00
>>> [ips-monitor]
>>>
>>> If the module isn't loaded, the load of the machine in idle mode is 0 as
>>> expected.  
>>
>> This is a reporting problem, and probably due to the schedule() call
>> and associated task state in the ips-monitor thread.  I thought setting
>> the task state to interruptible would prevent this, but it seems like
>> it's not enough for the deferrable on-stack timers?
>>
>> At any rate, it's not actually causing increased CPU usage, so you can
>> safely ignore it until we have a fix.
> 
> Oops, one task uses interruptible correctly, but the monitor thread
> doesn't.
> 
> Does this patch fix your load average?

Which patch? I can't see any patch :-).


Andreas

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: intel_ips produces constant load of 1
  2011-03-23 18:15     ` Andreas Hartmann
@ 2011-03-23 18:20       ` Jesse Barnes
  2011-03-23 19:44         ` Andreas Hartmann
  0 siblings, 1 reply; 10+ messages in thread
From: Jesse Barnes @ 2011-03-23 18:20 UTC (permalink / raw)
  To: Andreas Hartmann; +Cc: linux-kernel

On Wed, 23 Mar 2011 19:15:57 +0100
Andreas Hartmann <andihartmann@freenet.de> wrote:

> Jesse Barnes wrote:
> > On Mon, 21 Mar 2011 11:04:04 -0700
> > Jesse Barnes <jbarnes@virtuousgeek.org> wrote:
> > 
> >>> ps aux | grep ips
> >>> root       593  0.0  0.0      0     0 ?        S    17:20   0:00
> >>> [ips-adjust]
> >>> root       594  0.0  0.0      0     0 ?        D    17:20   0:00
> >>> [ips-monitor]
> >>>
> >>> If the module isn't loaded, the load of the machine in idle mode is 0 as
> >>> expected.  
> >>
> >> This is a reporting problem, and probably due to the schedule() call
> >> and associated task state in the ips-monitor thread.  I thought setting
> >> the task state to interruptible would prevent this, but it seems like
> >> it's not enough for the deferrable on-stack timers?
> >>
> >> At any rate, it's not actually causing increased CPU usage, so you can
> >> safely ignore it until we have a fix.
> > 
> > Oops, one task uses interruptible correctly, but the monitor thread
> > doesn't.
> > 
> > Does this patch fix your load average?
> 
> Which patch? I can't see any patch :-).
> 

Did I forget to paste it?  See below.

-- 
Jesse Barnes, Intel Open Source Technology Center

diff --git a/drivers/platform/x86/intel_ips.c b/drivers/platform/x86/intel_ips.c
index 1294a39..85c8ad4 100644
--- a/drivers/platform/x86/intel_ips.c
+++ b/drivers/platform/x86/intel_ips.c
@@ -1111,7 +1111,7 @@ static int ips_monitor(void *data)
                last_msecs = jiffies_to_msecs(jiffies);
                expire = jiffies + msecs_to_jiffies(IPS_SAMPLE_PERIOD);
 
-               __set_current_state(TASK_UNINTERRUPTIBLE);
+               __set_current_state(TASK_INTERRUPTIBLE);
                mod_timer(&timer, expire);
                schedule();
 

^ permalink raw reply related	[flat|nested] 10+ messages in thread

* Re: intel_ips produces constant load of 1
  2011-03-23 18:20       ` Jesse Barnes
@ 2011-03-23 19:44         ` Andreas Hartmann
  0 siblings, 0 replies; 10+ messages in thread
From: Andreas Hartmann @ 2011-03-23 19:44 UTC (permalink / raw)
  To: Jesse Barnes; +Cc: linux-kernel

Jesse Barnes wrote:
> On Wed, 23 Mar 2011 19:15:57 +0100
> Andreas Hartmann <andihartmann@freenet.de> wrote:
> 
>> Jesse Barnes wrote:
>>> On Mon, 21 Mar 2011 11:04:04 -0700
>>> Jesse Barnes <jbarnes@virtuousgeek.org> wrote:
>>>
>>>>> ps aux | grep ips
>>>>> root       593  0.0  0.0      0     0 ?        S    17:20   0:00
>>>>> [ips-adjust]
>>>>> root       594  0.0  0.0      0     0 ?        D    17:20   0:00
>>>>> [ips-monitor]
>>>>>
>>>>> If the module isn't loaded, the load of the machine in idle mode is 0 as
>>>>> expected.  
>>>>
>>>> This is a reporting problem, and probably due to the schedule() call
>>>> and associated task state in the ips-monitor thread.  I thought setting
>>>> the task state to interruptible would prevent this, but it seems like
>>>> it's not enough for the deferrable on-stack timers?
>>>>
>>>> At any rate, it's not actually causing increased CPU usage, so you can
>>>> safely ignore it until we have a fix.
>>>
>>> Oops, one task uses interruptible correctly, but the monitor thread
>>> doesn't.
>>>
>>> Does this patch fix your load average?
>>
>> Which patch? I can't see any patch :-).
>>
> 
> Did I forget to paste it?  See below.
> 

Uuuups - it was below the sig, which I switch off :-). Found it. Will
test it tomorrow.


Andreas

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: intel_ips produces constant load of 1
  2011-03-22 20:25   ` Jesse Barnes
  2011-03-23 18:15     ` Andreas Hartmann
@ 2011-03-25 18:23     ` Andreas Hartmann
  2011-03-25 18:24       ` Jesse Barnes
  1 sibling, 1 reply; 10+ messages in thread
From: Andreas Hartmann @ 2011-03-25 18:23 UTC (permalink / raw)
  To: Jesse Barnes, linux-kernel

Jesse Barnes wrote:
> On Mon, 21 Mar 2011 11:04:04 -0700
> Jesse Barnes <jbarnes@virtuousgeek.org> wrote:
> 
>>> ps aux | grep ips
>>> root       593  0.0  0.0      0     0 ?        S    17:20   0:00
>>> [ips-adjust]
>>> root       594  0.0  0.0      0     0 ?        D    17:20   0:00
>>> [ips-monitor]
>>>
>>> If the module isn't loaded, the load of the machine in idle mode is 0 as
>>> expected.  
>>
>> This is a reporting problem, and probably due to the schedule() call
>> and associated task state in the ips-monitor thread.  I thought setting
>> the task state to interruptible would prevent this, but it seems like
>> it's not enough for the deferrable on-stack timers?
>>
>> At any rate, it's not actually causing increased CPU usage, so you can
>> safely ignore it until we have a fix.
> 
> Oops, one task uses interruptible correctly, but the monitor thread
> doesn't.
> 
> Does this patch fix your load average?

The patch works fine:

ps aux | grep ips
root     22427  0.0  0.0      0     0 ?        S    19:15   0:00
[ips-adjust]
root     22428  0.0  0.0      0     0 ?        S    19:15   0:00
[ips-monitor]



btw: what does this mean (first line):


kernel: [ 6610.030205] intel ips 0000:00:1f.6: CPU TDP doesn't match
expected value (found 25, expected 29)
kernel: [ 6610.030218] intel ips 0000:00:1f.6: PCI INT C -> GSI 18
(level, low) -> IRQ 18
kernel: [ 6610.030733] intel ips 0000:00:1f.6: IPS driver initialized,
MCP temp limit 90


Thank you,
kind regards,
Andreas

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: intel_ips produces constant load of 1
  2011-03-25 18:23     ` Andreas Hartmann
@ 2011-03-25 18:24       ` Jesse Barnes
  0 siblings, 0 replies; 10+ messages in thread
From: Jesse Barnes @ 2011-03-25 18:24 UTC (permalink / raw)
  To: Andreas Hartmann; +Cc: linux-kernel

On Fri, 25 Mar 2011 19:23:15 +0100
Andreas Hartmann <andihartmann@freenet.de> wrote:

> Jesse Barnes wrote:
> > On Mon, 21 Mar 2011 11:04:04 -0700
> > Jesse Barnes <jbarnes@virtuousgeek.org> wrote:
> > 
> >>> ps aux | grep ips
> >>> root       593  0.0  0.0      0     0 ?        S    17:20   0:00
> >>> [ips-adjust]
> >>> root       594  0.0  0.0      0     0 ?        D    17:20   0:00
> >>> [ips-monitor]
> >>>
> >>> If the module isn't loaded, the load of the machine in idle mode is 0 as
> >>> expected.  
> >>
> >> This is a reporting problem, and probably due to the schedule() call
> >> and associated task state in the ips-monitor thread.  I thought setting
> >> the task state to interruptible would prevent this, but it seems like
> >> it's not enough for the deferrable on-stack timers?
> >>
> >> At any rate, it's not actually causing increased CPU usage, so you can
> >> safely ignore it until we have a fix.
> > 
> > Oops, one task uses interruptible correctly, but the monitor thread
> > doesn't.
> > 
> > Does this patch fix your load average?
> 
> The patch works fine:
> 
> ps aux | grep ips
> root     22427  0.0  0.0      0     0 ?        S    19:15   0:00
> [ips-adjust]
> root     22428  0.0  0.0      0     0 ?        S    19:15   0:00
> [ips-monitor]

Great, thanks for testing.  I'll get the fix over to Matthew.

> btw: what does this mean (first line):
> 
> 
> kernel: [ 6610.030205] intel ips 0000:00:1f.6: CPU TDP doesn't match
> expected value (found 25, expected 29)

Each SKU has a default TDP.  In your case that's 29W.  But the BIOS can
override that value with a lower one if the platform (chassis, board,
fans, etc) isn't designed to dissipate the full amount.


-- 
Jesse Barnes, Intel Open Source Technology Center

^ permalink raw reply	[flat|nested] 10+ messages in thread

end of thread, other threads:[~2011-03-25 18:24 UTC | newest]

Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2011-03-19 15:38 intel_ips produces constant load of 1 Andreas Hartmann
2011-03-21 18:04 ` Jesse Barnes
2011-03-22  7:00   ` Andreas Hartmann
2011-03-22 15:49     ` Jesse Barnes
2011-03-22 20:25   ` Jesse Barnes
2011-03-23 18:15     ` Andreas Hartmann
2011-03-23 18:20       ` Jesse Barnes
2011-03-23 19:44         ` Andreas Hartmann
2011-03-25 18:23     ` Andreas Hartmann
2011-03-25 18:24       ` Jesse Barnes

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).