* 2.5.67-mm4 & IRQ balancing
@ 2003-04-18 23:58 Philippe Gramoullé
2003-04-19 0:51 ` Andrew Morton
0 siblings, 1 reply; 14+ messages in thread
From: Philippe Gramoullé @ 2003-04-18 23:58 UTC (permalink / raw)
To: linux-kernel
Hello,
It may be not related to -mm4 only but as i hadn't checked before ( with 2.5.x kernels),
I just wonder about /proc/interrupts output:
$ cat /proc/interrupts
CPU0 CPU1
0: 47851610 0 IO-APIC-edge timer
1: 51789 0 IO-APIC-edge i8042
2: 0 0 XT-PIC cascade
3: 171 0 IO-APIC-edge serial
8: 772066 0 IO-APIC-edge rtc
12: 3 0 IO-APIC-edge i8042, i8042, i8042, i8042
15: 58 1 IO-APIC-edge ide1
16: 47047 0 IO-APIC-level ohci1394
18: 391753 0 IO-APIC-level EMU10K1
19: 911863 0 IO-APIC-level uhci-hcd
20: 261806 0 IO-APIC-level eth0
22: 273648 0 IO-APIC-level aic7xxx
23: 0 0 IO-APIC-level uhci-hcd
NMI: 47853468 47852927
LOC: 47860500 47860630
ERR: 0
MIS: 0
Shouldn't the interrupts be balanced on both CPUs ?
DELL MT 530 Ws , SMP Xeon 1.5Ghz, 512 Mo RAM on Debian Unstable.
Thanks,
Philippe
^ permalink raw reply [flat|nested] 14+ messages in thread* Re: 2.5.67-mm4 & IRQ balancing
2003-04-18 23:58 2.5.67-mm4 & IRQ balancing Philippe Gramoullé
@ 2003-04-19 0:51 ` Andrew Morton
2003-04-19 13:39 ` Philippe Gramoullé
0 siblings, 1 reply; 14+ messages in thread
From: Andrew Morton @ 2003-04-19 0:51 UTC (permalink / raw)
To: Philippe Gramoullé; +Cc: linux-kernel
Philippe Gramoullé <philippe.gramoulle@mmania.com> wrote:
>
>
> Hello,
>
> It may be not related to -mm4 only but as i hadn't checked before ( with 2.5.x kernels),
> I just wonder about /proc/interrupts output:
>
> $ cat /proc/interrupts
> CPU0 CPU1
> 0: 47851610 0 IO-APIC-edge timer
> 1: 51789 0 IO-APIC-edge i8042
> 2: 0 0 XT-PIC cascade
> 3: 171 0 IO-APIC-edge serial
> 8: 772066 0 IO-APIC-edge rtc
> 12: 3 0 IO-APIC-edge i8042, i8042, i8042, i8042
> 15: 58 1 IO-APIC-edge ide1
> 16: 47047 0 IO-APIC-level ohci1394
> 18: 391753 0 IO-APIC-level EMU10K1
> 19: 911863 0 IO-APIC-level uhci-hcd
> 20: 261806 0 IO-APIC-level eth0
> 22: 273648 0 IO-APIC-level aic7xxx
It is supposed to do that.
You might as well beat the rush; boot with the `noirqbalance' option and
run http://people.redhat.com/arjanv/irqbalance/. We want to pull the
irq balancer out of the kernel altogether.
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: 2.5.67-mm4 & IRQ balancing
2003-04-19 0:51 ` Andrew Morton
@ 2003-04-19 13:39 ` Philippe Gramoullé
2003-04-19 20:38 ` Andrew Morton
0 siblings, 1 reply; 14+ messages in thread
From: Philippe Gramoullé @ 2003-04-19 13:39 UTC (permalink / raw)
To: Andrew Morton; +Cc: linux-kernel
Hello,
On Fri, 18 Apr 2003 17:51:16 -0700
Andrew Morton <akpm@digeo.com> wrote:
| $ cat /proc/interrupts
| > CPU0 CPU1
| > 0: 47851610 0 IO-APIC-edge timer
| > 1: 51789 0 IO-APIC-edge i8042
| > 2: 0 0 XT-PIC cascade
| > 3: 171 0 IO-APIC-edge serial
| > 8: 772066 0 IO-APIC-edge rtc
| > 12: 3 0 IO-APIC-edge i8042, i8042, i8042, i8042
| > 15: 58 1 IO-APIC-edge ide1
| > 16: 47047 0 IO-APIC-level ohci1394
| > 18: 391753 0 IO-APIC-level EMU10K1
| > 19: 911863 0 IO-APIC-level uhci-hcd
| > 20: 261806 0 IO-APIC-level eth0
| > 22: 273648 0 IO-APIC-level aic7xxx
|
| It is supposed to do that.
Ok.
|
| You might as well beat the rush; boot with the `noirqbalance' option and
| run http://people.redhat.com/arjanv/irqbalance/. We want to pull the
| irq balancer out of the kernel altogether.
Ok, i booted with noriqbalance, removed nmi_watchdog=1 and ran irqbalance 0.06.
Now after few minutes of activity :
# cat /proc/interrupts
CPU0 CPU1
0: 73897 577734 IO-APIC-edge timer
1: 3297 18 IO-APIC-edge i8042
2: 0 0 XT-PIC cascade
3: 177 0 IO-APIC-edge serial
8: 1 0 IO-APIC-edge rtc
12: 3 0 IO-APIC-edge i8042, i8042, i8042, i8042
15: 10 1 IO-APIC-edge ide1
18: 0 0 IO-APIC-level EMU10K1
19: 8366 0 IO-APIC-level uhci-hcd
20: 1306 0 IO-APIC-level eth0
22: 30753 965 IO-APIC-level aic7xxx
23: 0 0 IO-APIC-level uhci-hcd
NMI: 0 0
LOC: 649138 649349
ERR: 0
MIS: 0
and about 30 seconds later ( mail checking )
# cat /proc/interrupts
CPU0 CPU1
0: 73897 676905 IO-APIC-edge timer
1: 3571 18 IO-APIC-edge i8042
2: 0 0 XT-PIC cascade
3: 177 0 IO-APIC-edge serial
8: 1 0 IO-APIC-edge rtc
12: 3 0 IO-APIC-edge i8042, i8042, i8042, i8042
15: 10 1 IO-APIC-edge ide1
18: 0 0 IO-APIC-level EMU10K1
19: 15866 0 IO-APIC-level uhci-hcd
20: 1377 0 IO-APIC-level eth0
22: 34408 965 IO-APIC-level aic7xxx
23: 0 0 IO-APIC-level uhci-hcd
NMI: 0 0
LOC: 748312 748523
ERR: 0
MIS: 0
Is this what you are looking for ? and are the values changes meaningful ?
Thanks,
Philippe
^ permalink raw reply [flat|nested] 14+ messages in thread* Re: 2.5.67-mm4 & IRQ balancing
2003-04-19 13:39 ` Philippe Gramoullé
@ 2003-04-19 20:38 ` Andrew Morton
2003-04-22 10:06 ` Philippe Gramoullé
2003-04-23 19:41 ` Bill Davidsen
0 siblings, 2 replies; 14+ messages in thread
From: Andrew Morton @ 2003-04-19 20:38 UTC (permalink / raw)
To: Philippe Gramoullé; +Cc: linux-kernel
Philippe Gramoullé <philippe.gramoulle@mmania.com> wrote:
>
> [ SMP IRQ distribution ]
>
> Is this what you are looking for ? and are the values changes meaningful ?
Looks good to me. But it didn't affect your machine at all, did it?
This stuff only counts when the machine is doing a lot of work. The current
IRQ balancer works well under high interrupt frequencies, but does quite the
wrong thing if you're doing a lot of softirq work at low interrupt
frequencies (gige routing with NAPI).
My gut feel is that we'll never get this right with a single in-kernel IRQ
balancer. So the proposal is to pull the IRQ balancer out altogether and to
then merge Arjan's userspace balancer into the main kernel tree.
It's a little radical to go placing userspace daemons into the kernel tree,
but I think it is appropriate - this thing is very tightly coupled to the
kernel.
The proposal has these advantages:
- No version skew problems: if the format of /proc/interrupts changes, we
patch the irq balance daemon at the same time.
- Can build irqbalanced into the intial initramfs image as part of kernel
build. (lacking klibc, we would need to statically link against glibc)
- Doing it in userspace means that we can do more things.
- The balancer can "know about" the differences between NICs, disk
controllers, etc.
- The balancer can be controlled by config files: "I am a router"
- The balancer can support non-x86 architectures
Anyway, that's the theory. None of it has been done yet.
^ permalink raw reply [flat|nested] 14+ messages in thread* Re: 2.5.67-mm4 & IRQ balancing
2003-04-19 20:38 ` Andrew Morton
@ 2003-04-22 10:06 ` Philippe Gramoullé
2003-04-22 10:32 ` Arjan van de Ven
2003-04-23 19:41 ` Bill Davidsen
1 sibling, 1 reply; 14+ messages in thread
From: Philippe Gramoullé @ 2003-04-22 10:06 UTC (permalink / raw)
To: Andrew Morton; +Cc: linux-kernel
Hello,
On Sat, 19 Apr 2003 13:38:37 -0700
Andrew Morton <akpm@digeo.com> wrote:
| Philippe Gramoullé <philippe.gramoulle@mmania.com> wrote:
| >
| > [ SMP IRQ distribution ]
| >
| > Is this what you are looking for ? and are the values changes meaningful ?
|
| Looks good to me. But it didn't affect your machine at all, did it?
Well, no, i didn't really felt a clear change.
|
| This stuff only counts when the machine is doing a lot of work. The current
| IRQ balancer works well under high interrupt frequencies, but does quite the
| wrong thing if you're doing a lot of softirq work at low interrupt
| frequencies (gige routing with NAPI).
This box is used as a desktop box, so quite a lots of open applications, but no
real high load/IO except few kernel compiles and BK consistency checks.
Thanks,
Philippe
Booted with "noirqbalance" & started irqbalance:
$ cat /proc/interrupts
CPU0 CPU1
0: 73897 247288143 IO-APIC-edge timer
1: 38421 56 IO-APIC-edge i8042
2: 0 0 XT-PIC cascade
3: 177 0 IO-APIC-edge serial
8: 107607 0 IO-APIC-edge rtc
12: 3 0 IO-APIC-edge i8042, i8042, i8042, i8042
15: 32 118 IO-APIC-edge ide1
18: 12602 1159 IO-APIC-level EMU10K1
19: 454172 15987 IO-APIC-level uhci-hcd
20: 494005 0 IO-APIC-level eth0
22: 717483 38681 IO-APIC-level aic7xxx
23: 0 0 IO-APIC-level uhci-hcd
NMI: 0 0
LOC: 247366287 247364170
ERR: 0
MIS: 0
^ permalink raw reply [flat|nested] 14+ messages in thread* Re: 2.5.67-mm4 & IRQ balancing
2003-04-19 20:38 ` Andrew Morton
2003-04-22 10:06 ` Philippe Gramoullé
@ 2003-04-23 19:41 ` Bill Davidsen
2003-04-23 20:13 ` Zwane Mwaikambo
2003-04-23 20:15 ` Andrew Morton
1 sibling, 2 replies; 14+ messages in thread
From: Bill Davidsen @ 2003-04-23 19:41 UTC (permalink / raw)
To: Andrew Morton; +Cc: Philippe Gramoullé, linux-kernel
On Sat, 19 Apr 2003, Andrew Morton wrote:
> It's a little radical to go placing userspace daemons into the kernel tree,
> but I think it is appropriate - this thing is very tightly coupled to the
> kernel.
>
> The proposal has these advantages:
>
> - No version skew problems: if the format of /proc/interrupts changes, we
> patch the irq balance daemon at the same time.
>
> - Can build irqbalanced into the intial initramfs image as part of kernel
> build. (lacking klibc, we would need to statically link against glibc)
Why, please? Unless you postulate that (a) the default kernel balance
would be so bad the machine wouldn't boot, or (b) that the interface would
be done only once at boot time, there's no reason for the user program to
be in initramfs, is there? Let the distribution put it where other system
things like ifconfig live.
Feel free to explain what I'm missing.
> - Doing it in userspace means that we can do more things.
>
> - The balancer can "know about" the differences between NICs, disk
> controllers, etc.
>
> - The balancer can be controlled by config files: "I am a router"
>
> - The balancer can support non-x86 architectures
>
>
> Anyway, that's the theory. None of it has been done yet.
I do agree that the program would have to match the /proc if done as you
propose, but wouldn't it be better to design an interface once and then
NOT have it change? And does it belong in /proc at all, given that other
things are being moved out?
I like the idea of being able to tune the int processing with a user
program. I don't think I share your vision of making a user program part
of the kernel to allow diddling an interface which might be better getting
right the first time, and protecting against "features" being added.
Hopefully it will be minimalist, and may well benefit from a totally
different user program for various machine types.
--
bill davidsen <davidsen@tmr.com>
CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: 2.5.67-mm4 & IRQ balancing
2003-04-23 19:41 ` Bill Davidsen
@ 2003-04-23 20:13 ` Zwane Mwaikambo
2003-04-24 21:32 ` Bill Davidsen
2003-04-23 20:15 ` Andrew Morton
1 sibling, 1 reply; 14+ messages in thread
From: Zwane Mwaikambo @ 2003-04-23 20:13 UTC (permalink / raw)
To: Bill Davidsen; +Cc: Andrew Morton, Philippe Gramoullé, linux-kernel
On Wed, 23 Apr 2003, Bill Davidsen wrote:
> I like the idea of being able to tune the int processing with a user
> program. I don't think I share your vision of making a user program part
You've actually been able to do this with echo(1) for a while, just not
'automagically'
> of the kernel to allow diddling an interface which might be better getting
> right the first time, and protecting against "features" being added.
> Hopefully it will be minimalist, and may well benefit from a totally
> different user program for various machine types.
The smp interrupt affinity interface hasn't changed for a while (since
inception?), we're only now deciding on where to put the autotune aspect
of it.
Zwane
--
function.linuxpower.ca
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: 2.5.67-mm4 & IRQ balancing
2003-04-23 20:13 ` Zwane Mwaikambo
@ 2003-04-24 21:32 ` Bill Davidsen
2003-04-25 9:03 ` Arjan van de Ven
0 siblings, 1 reply; 14+ messages in thread
From: Bill Davidsen @ 2003-04-24 21:32 UTC (permalink / raw)
To: Zwane Mwaikambo; +Cc: Andrew Morton, Philippe Gramoullé, linux-kernel
On Wed, 23 Apr 2003, Zwane Mwaikambo wrote:
> On Wed, 23 Apr 2003, Bill Davidsen wrote:
>
> > I like the idea of being able to tune the int processing with a user
> > program. I don't think I share your vision of making a user program part
>
> You've actually been able to do this with echo(1) for a while, just not
> 'automagically'
>
> > of the kernel to allow diddling an interface which might be better getting
> > right the first time, and protecting against "features" being added.
> > Hopefully it will be minimalist, and may well benefit from a totally
> > different user program for various machine types.
>
> The smp interrupt affinity interface hasn't changed for a while (since
> inception?), we're only now deciding on where to put the autotune aspect
> of it.
So the usermode program would not have to be part of kernel source as
previously stated, if I read that right, it just has to conform to a
standard. And everybody can write one and try to measure the difference it
makes.
--
bill davidsen <davidsen@tmr.com>
CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: 2.5.67-mm4 & IRQ balancing
2003-04-23 19:41 ` Bill Davidsen
2003-04-23 20:13 ` Zwane Mwaikambo
@ 2003-04-23 20:15 ` Andrew Morton
1 sibling, 0 replies; 14+ messages in thread
From: Andrew Morton @ 2003-04-23 20:15 UTC (permalink / raw)
To: Bill Davidsen; +Cc: philippe.gramoulle, linux-kernel
Bill Davidsen <davidsen@tmr.com> wrote:
>
> > - Can build irqbalanced into the intial initramfs image as part of kernel
> > build. (lacking klibc, we would need to statically link against glibc)
>
> Why, please? Unless you postulate that (a) the default kernel balance
> would be so bad the machine wouldn't boot, or (b) that the interface would
> be done only once at boot time, there's no reason for the user program to
> be in initramfs, is there? Let the distribution put it where other system
> things like ifconfig live.
Mainly as an exercise in using the initramfs infrastructure for something
real. It's doubtful that irqbalanced would be started that way in real life.
^ permalink raw reply [flat|nested] 14+ messages in thread
* RE: 2.5.67-mm4 & IRQ balancing
@ 2003-04-19 1:03 Nakajima, Jun
0 siblings, 0 replies; 14+ messages in thread
From: Nakajima, Jun @ 2003-04-19 1:03 UTC (permalink / raw)
To: Philippe Gramoullé, linux-kernel
Can you try to stress some (e.g. aic7xxx) of them (use dd, for example), and check what happens? I assume they are not HT-capable.
Thanks,
Jun
> -----Original Message-----
> From: Philippe Gramoullé [mailto:philippe.gramoulle@mmania.com]
> Sent: Friday, April 18, 2003 4:59 PM
> To: linux-kernel@vger.kernel.org
> Subject: 2.5.67-mm4 & IRQ balancing
> Importance: High
>
>
> Hello,
>
> It may be not related to -mm4 only but as i hadn't checked before ( with
> 2.5.x kernels),
> I just wonder about /proc/interrupts output:
>
> $ cat /proc/interrupts
> CPU0 CPU1
> 0: 47851610 0 IO-APIC-edge timer
> 1: 51789 0 IO-APIC-edge i8042
> 2: 0 0 XT-PIC cascade
> 3: 171 0 IO-APIC-edge serial
> 8: 772066 0 IO-APIC-edge rtc
> 12: 3 0 IO-APIC-edge i8042, i8042, i8042, i8042
> 15: 58 1 IO-APIC-edge ide1
> 16: 47047 0 IO-APIC-level ohci1394
> 18: 391753 0 IO-APIC-level EMU10K1
> 19: 911863 0 IO-APIC-level uhci-hcd
> 20: 261806 0 IO-APIC-level eth0
> 22: 273648 0 IO-APIC-level aic7xxx
> 23: 0 0 IO-APIC-level uhci-hcd
> NMI: 47853468 47852927
> LOC: 47860500 47860630
> ERR: 0
> MIS: 0
>
> Shouldn't the interrupts be balanced on both CPUs ?
>
> DELL MT 530 Ws , SMP Xeon 1.5Ghz, 512 Mo RAM on Debian Unstable.
>
> Thanks,
>
> Philippe
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: 2.5.67-mm4 & IRQ balancing
@ 2003-04-19 12:21 Chuck Ebbert
2003-04-19 13:14 ` Philippe Gramoullé
0 siblings, 1 reply; 14+ messages in thread
From: Chuck Ebbert @ 2003-04-19 12:21 UTC (permalink / raw)
To: Philippe Gramoullé; +Cc: linux-kernel
Philippe wrote:
> CPU0 CPU1
> 0: 47851610 0 IO-APIC-edge timer
> 1: 51789 0 IO-APIC-edge i8042
> 2: 0 0 XT-PIC cascade
> 3: 171 0 IO-APIC-edge serial
> 8: 772066 0 IO-APIC-edge rtc
> 12: 3 0 IO-APIC-edge i8042, i8042, i8042, i8042
> 15: 58 1 IO-APIC-edge ide1
> 16: 47047 0 IO-APIC-level ohci1394
> 18: 391753 0 IO-APIC-level EMU10K1
> 19: 911863 0 IO-APIC-level uhci-hcd
> 20: 261806 0 IO-APIC-level eth0
> 22: 273648 0 IO-APIC-level aic7xxx
> 23: 0 0 IO-APIC-level uhci-hcd
> NMI: 47853468 47852927
> LOC: 47860500 47860630
> ERR: 0
> MIS: 0
I wonder what is the reason for all the NMIs? And why arent't the local
APIC interrupt counters in sync?
With 2.5.66 I have twice as many interrupts on CPU1 as you. :)
CPU0 CPU1
0: 250666330 0 IO-APIC-edge timer
1: 545 1 IO-APIC-edge i8042
2: 0 0 XT-PIC cascade
12: 124 0 IO-APIC-edge i8042
15: 21 1 IO-APIC-edge ide1
18: 8484 0 IO-APIC-level ide3
19: 4679 0 IO-APIC-level uhci-hcd, eth0
NMI: 0 0
LOC: 250677924 250677924
ERR: 0
MIS: 0
------
Chuck
^ permalink raw reply [flat|nested] 14+ messages in thread* Re: 2.5.67-mm4 & IRQ balancing
2003-04-19 12:21 Chuck Ebbert
@ 2003-04-19 13:14 ` Philippe Gramoullé
0 siblings, 0 replies; 14+ messages in thread
From: Philippe Gramoullé @ 2003-04-19 13:14 UTC (permalink / raw)
To: Chuck Ebbert; +Cc: linux-kernel
Hello,
Well, my guess is that because i boot with nmi_watchdog=1 kernel boot option ?? I don't really know.
This was what i used to troubleshoot hard lockups with previous kernels ( i may remove it now
as since a patch was made to fix that, everything runs smoothly :)
Thanks,
Philippe
On Sat, 19 Apr 2003 08:21:38 -0400
Chuck Ebbert <76306.1226@compuserve.com> wrote:
| Philippe wrote:
|
|
| > CPU0 CPU1
| > 0: 47851610 0 IO-APIC-edge timer
| > 1: 51789 0 IO-APIC-edge i8042
| > 2: 0 0 XT-PIC cascade
| > 3: 171 0 IO-APIC-edge serial
| > 8: 772066 0 IO-APIC-edge rtc
| > 12: 3 0 IO-APIC-edge i8042, i8042, i8042, i8042
| > 15: 58 1 IO-APIC-edge ide1
| > 16: 47047 0 IO-APIC-level ohci1394
| > 18: 391753 0 IO-APIC-level EMU10K1
| > 19: 911863 0 IO-APIC-level uhci-hcd
| > 20: 261806 0 IO-APIC-level eth0
| > 22: 273648 0 IO-APIC-level aic7xxx
| > 23: 0 0 IO-APIC-level uhci-hcd
| > NMI: 47853468 47852927
| > LOC: 47860500 47860630
| > ERR: 0
| > MIS: 0
|
|
| I wonder what is the reason for all the NMIs? And why arent't the local
| APIC interrupt counters in sync?
|
| With 2.5.66 I have twice as many interrupts on CPU1 as you. :)
|
|
| CPU0 CPU1
| 0: 250666330 0 IO-APIC-edge timer
| 1: 545 1 IO-APIC-edge i8042
| 2: 0 0 XT-PIC cascade
| 12: 124 0 IO-APIC-edge i8042
| 15: 21 1 IO-APIC-edge ide1
| 18: 8484 0 IO-APIC-level ide3
| 19: 4679 0 IO-APIC-level uhci-hcd, eth0
| NMI: 0 0
| LOC: 250677924 250677924
| ERR: 0
| MIS: 0
|
|
|
| ------
| Chuck
| -
| To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
| the body of a message to majordomo@vger.kernel.org
| More majordomo info at http://vger.kernel.org/majordomo-info.html
| Please read the FAQ at http://www.tux.org/lkml/
^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2003-04-25 8:52 UTC | newest]
Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2003-04-18 23:58 2.5.67-mm4 & IRQ balancing Philippe Gramoullé
2003-04-19 0:51 ` Andrew Morton
2003-04-19 13:39 ` Philippe Gramoullé
2003-04-19 20:38 ` Andrew Morton
2003-04-22 10:06 ` Philippe Gramoullé
2003-04-22 10:32 ` Arjan van de Ven
2003-04-23 19:41 ` Bill Davidsen
2003-04-23 20:13 ` Zwane Mwaikambo
2003-04-24 21:32 ` Bill Davidsen
2003-04-25 9:03 ` Arjan van de Ven
2003-04-23 20:15 ` Andrew Morton
2003-04-19 1:03 Nakajima, Jun
2003-04-19 12:21 Chuck Ebbert
2003-04-19 13:14 ` Philippe Gramoullé
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).