* userspace irq balancer
[not found] <200305191314.06216.pbadari@us.ibm.com>
@ 2003-05-19 22:07 ` Dave Hansen
2003-05-19 22:11 ` Arjan van de Ven
0 siblings, 1 reply; 76+ messages in thread
From: Dave Hansen @ 2003-05-19 22:07 UTC (permalink / raw)
To: Badari Pulavarty, lkml
Cc: Gerrit Huizenga, John Stultz, James Cleverdon, Andrew Morton,
arjanv, Keith Mannthey
On Mon, 2003-05-19 at 13:14, Badari Pulavarty wrote:
> ---------- Forwarded Message ----------
>
> Subject: Re: 2.5 closeout
> Date: Mon, 19 May 2003 11:40:05 -0700
> From: Andrew Morton <akpm@digeo.com>
> To: Badari Pulavarty <pbadari@us.ibm.com>
>
> The attribution wasn't very accurate. That's me, Arjan, perhaps
> Jeff Garzik.
>
> The problem is that kird does the wrong thing for some workloads (packet
> forwarding mainly - routing). We'll never get that right, so we'd like to
> deprecate the in-kernel IRQ balancer and merge Arjan's user-space balancer
> into the kernel tree instead.
>
> http://people.redhat.com/~arjanv/irqbalance/
The only thing I'm concerned about is how it's going to be packaged.
I'm envisioning explaining how to get the daemon out of its initrd
image, set it up and run it, especially before distros have it
integrated. The stuff that's in the kernel now isn't horribly broken;
it's just not optimal for some relatively unusual cases.
Can we leave the current code as-is, and make the added intelligence
from the userspace one an optional thing for those with unusal setups?
--
Dave Hansen
haveblue@us.ibm.com
^ permalink raw reply [flat|nested] 76+ messages in thread
* Re: userspace irq balancer
2003-05-19 22:07 ` userspace irq balancer Dave Hansen
@ 2003-05-19 22:11 ` Arjan van de Ven
2003-05-19 22:22 ` Dave Hansen
0 siblings, 1 reply; 76+ messages in thread
From: Arjan van de Ven @ 2003-05-19 22:11 UTC (permalink / raw)
To: Dave Hansen
Cc: Badari Pulavarty, lkml, Gerrit Huizenga, John Stultz,
James Cleverdon, Andrew Morton, arjanv, Keith Mannthey
On Mon, May 19, 2003 at 03:07:36PM -0700, Dave Hansen wrote:
> The only thing I'm concerned about is how it's going to be packaged.
> I'm envisioning explaining how to get the daemon out of its initrd
> image, set it up and run it, especially before distros have it
> integrated. The stuff that's in the kernel now isn't horribly broken;
> it's just not optimal for some relatively unusual cases.
as for distros: RHL8 and later ship with it on the RH side
(default enabled as of RHL9).
As for where to start it: I really think an initscript is the logical
place; there has been some discussion about doing it
from the initramfs but I don't see real benifit from that; from starting
init to running the initscripts isn't exactly THIS interrupt/performance
heavy.
^ permalink raw reply [flat|nested] 76+ messages in thread
* Re: userspace irq balancer
2003-05-19 22:11 ` Arjan van de Ven
@ 2003-05-19 22:22 ` Dave Hansen
2003-05-20 3:25 ` David S. Miller
0 siblings, 1 reply; 76+ messages in thread
From: Dave Hansen @ 2003-05-19 22:22 UTC (permalink / raw)
To: Arjan van de Ven
Cc: Badari Pulavarty, lkml, Gerrit Huizenga, John Stultz,
James Cleverdon, Andrew Morton, Keith Mannthey
On Mon, 2003-05-19 at 15:11, Arjan van de Ven wrote:
> On Mon, May 19, 2003 at 03:07:36PM -0700, Dave Hansen wrote:
>
> > The only thing I'm concerned about is how it's going to be packaged.
> > I'm envisioning explaining how to get the daemon out of its initrd
> > image, set it up and run it, especially before distros have it
> > integrated. The stuff that's in the kernel now isn't horribly broken;
> > it's just not optimal for some relatively unusual cases.
>
> as for distros: RHL8 and later ship with it on the RH side
> (default enabled as of RHL9).
But, do you see the need for ripping out the current code? For those of
us that are still running a slightly more primitive distro, it would be
nice to have some pretty effective default behavior, like what is in the
kernel now.
> As for where to start it: I really think an initscript is the logical
> place; there has been some discussion about doing it
> from the initramfs but I don't see real benifit from that; from starting
> init to running the initscripts isn't exactly THIS interrupt/performance
> heavy.
Yeah, I don't think we need it the second the kernel boots :) Do you
really think this is a 2.6 showstopper? Since it will require distro
cooperation anyway, and those are many months from releasing a 2.6
distro, do we really need it in place for 2.6.0?
--
Dave Hansen
haveblue@us.ibm.com
^ permalink raw reply [flat|nested] 76+ messages in thread
* Re: userspace irq balancer
2003-05-19 22:22 ` Dave Hansen
@ 2003-05-20 3:25 ` David S. Miller
2003-05-20 3:46 ` William Lee Irwin III
0 siblings, 1 reply; 76+ messages in thread
From: David S. Miller @ 2003-05-20 3:25 UTC (permalink / raw)
To: Dave Hansen
Cc: Arjan van de Ven, Badari Pulavarty, lkml, Gerrit Huizenga,
John Stultz, James Cleverdon, Andrew Morton, Keith Mannthey
On Mon, 2003-05-19 at 15:22, Dave Hansen wrote:
> > as for distros: RHL8 and later ship with it on the RH side
> > (default enabled as of RHL9).
>
> But, do you see the need for ripping out the current code? For those of
> us that are still running a slightly more primitive distro, it would be
> nice to have some pretty effective default behavior, like what is in the
> kernel now.
You have to install new modutils to even use modules with the 2.5.x
kernel, given that why are we even talking about the "inconvenience"
of installing the usermode IRQ balancer as being a blocker for
ripping out the in-kernel stuff?
The in-kernel stuff MUST go. It went in because "some benchmark went
faster", but with no "why" describing why it might have improved
performance. We KNOW it absolutely sucks for routing and firewall
applications. The in-kernel bits were all a shamans dance, with zero
technical "here is why this makes things go faster" description
attached. If I remember properly, the changelog message when the
in-kernel irq balancing went in was of the form "this makes some
specweb run go faster".
--
David S. Miller <davem@redhat.com>
^ permalink raw reply [flat|nested] 76+ messages in thread
* Re: userspace irq balancer
2003-05-20 3:25 ` David S. Miller
@ 2003-05-20 3:46 ` William Lee Irwin III
2003-05-20 5:03 ` Dave Hansen
0 siblings, 1 reply; 76+ messages in thread
From: William Lee Irwin III @ 2003-05-20 3:46 UTC (permalink / raw)
To: David S. Miller
Cc: Dave Hansen, Arjan van de Ven, Badari Pulavarty, lkml,
Gerrit Huizenga, John Stultz, James Cleverdon, Andrew Morton,
Keith Mannthey
On Mon, May 19, 2003 at 08:25:31PM -0700, David S. Miller wrote:
> The in-kernel stuff MUST go. It went in because "some benchmark went
> faster", but with no "why" describing why it might have improved
> performance. We KNOW it absolutely sucks for routing and firewall
> applications. The in-kernel bits were all a shamans dance, with zero
> technical "here is why this makes things go faster" description
> attached. If I remember properly, the changelog message when the
> in-kernel irq balancing went in was of the form "this makes some
> specweb run go faster".
Absolutely. Not to mention the code for the in-kernel algorithm has
historically broken i386 ports using certain modes of Intel's
interrupt controllers.
Far better would be to validate that the affinity specified is feasible
to program into the interrupt controller in a system call and leave the
algorithm to userspace.
-- wli
^ permalink raw reply [flat|nested] 76+ messages in thread
* Re: userspace irq balancer
2003-05-20 3:46 ` William Lee Irwin III
@ 2003-05-20 5:03 ` Dave Hansen
2003-05-20 5:53 ` Martin J. Bligh
2003-05-20 9:00 ` Arjan van de Ven
0 siblings, 2 replies; 76+ messages in thread
From: Dave Hansen @ 2003-05-20 5:03 UTC (permalink / raw)
To: William Lee Irwin III
Cc: David S. Miller, Arjan van de Ven, Badari Pulavarty, lkml,
Gerrit Huizenga, John Stultz, James Cleverdon, Andrew Morton,
Keith Mannthey
On Mon, 2003-05-19 at 20:46, William Lee Irwin III wrote:
> On Mon, May 19, 2003 at 08:25:31PM -0700, David S. Miller wrote:
> > The in-kernel stuff MUST go. It went in because "some benchmark went
> > faster", but with no "why" describing why it might have improved
> > performance. We KNOW it absolutely sucks for routing and firewall
> > applications. The in-kernel bits were all a shamans dance, with zero
> > technical "here is why this makes things go faster" description
> > attached. If I remember properly, the changelog message when the
> > in-kernel irq balancing went in was of the form "this makes some
> > specweb run go faster".
>
> Absolutely. Not to mention the code for the in-kernel algorithm has
> historically broken i386 ports using certain modes of Intel's
> interrupt controllers.
OK, I just went and actually looked at the code again. After
suppressing my gag reflex, I started to remember all of the problems
we've had with it, including fixing it for Intel's own clustered APIC
mode.
Does anyone have a patch to tear it out already? Is the current proc
interface acceptable, or do we want a syscall interface like wli
suggests?
--
Dave Hansen
haveblue@us.ibm.com
^ permalink raw reply [flat|nested] 76+ messages in thread
* Re: userspace irq balancer
2003-05-20 5:03 ` Dave Hansen
@ 2003-05-20 5:53 ` Martin J. Bligh
2003-05-20 6:13 ` David S. Miller
2003-05-20 9:00 ` Arjan van de Ven
1 sibling, 1 reply; 76+ messages in thread
From: Martin J. Bligh @ 2003-05-20 5:53 UTC (permalink / raw)
To: Dave Hansen, William Lee Irwin III
Cc: David S. Miller, Arjan van de Ven, Badari Pulavarty, lkml,
Gerrit Huizenga, John Stultz, James Cleverdon, Andrew Morton,
Keith Mannthey
--Dave Hansen <haveblue@us.ibm.com> wrote (on Monday, May 19, 2003 22:03:50 -0700):
> On Mon, 2003-05-19 at 20:46, William Lee Irwin III wrote:
>> On Mon, May 19, 2003 at 08:25:31PM -0700, David S. Miller wrote:
>> > The in-kernel stuff MUST go. It went in because "some benchmark went
>> > faster", but with no "why" describing why it might have improved
>> > performance. We KNOW it absolutely sucks for routing and firewall
>> > applications. The in-kernel bits were all a shamans dance, with zero
>> > technical "here is why this makes things go faster" description
>> > attached. If I remember properly, the changelog message when the
>> > in-kernel irq balancing went in was of the form "this makes some
>> > specweb run go faster".
>>
>> Absolutely. Not to mention the code for the in-kernel algorithm has
>> historically broken i386 ports using certain modes of Intel's
>> interrupt controllers.
>
> OK, I just went and actually looked at the code again. After
> suppressing my gag reflex, I started to remember all of the problems
> we've had with it, including fixing it for Intel's own clustered APIC
> mode.
>
> Does anyone have a patch to tear it out already? Is the current proc
> interface acceptable, or do we want a syscall interface like wli
> suggests?
I have no frigging idea why you'd want to tear something out that works
well already, and has a shitload of work put into it.
Make it a config option if you don't like it, Keith has a patch to do
that already - it's trivial. That way everyone can have what they want.
M.
^ permalink raw reply [flat|nested] 76+ messages in thread
* Re: userspace irq balancer
2003-05-20 5:53 ` Martin J. Bligh
@ 2003-05-20 6:13 ` David S. Miller
2003-05-20 6:36 ` Dave Hansen
2003-05-20 14:01 ` Martin J. Bligh
0 siblings, 2 replies; 76+ messages in thread
From: David S. Miller @ 2003-05-20 6:13 UTC (permalink / raw)
To: mbligh
Cc: haveblue, wli, arjanv, pbadari, linux-kernel, gh, johnstul,
jamesclv, akpm, mannthey
From: "Martin J. Bligh" <mbligh@aracnet.com>
Date: Mon, 19 May 2003 22:53:11 -0700
I have no frigging idea why you'd want to tear something out that
works well already, and has a shitload of work put into it.
It's pretty fundamentally broken for having had so much work
put into it. Show me something other than "SpecWEB run for IBM
ran faster" as a reason for keeping this code in there. Can you
even do this?
It is crap, from the very beginning, would you like to know why?
How does the in-kernel IRQ load balancing measure "load" and
"busyness"? Herein lies the most absolutely fundamental problem with
this code, it fails to recognize that we end up with most of our
networking "load" from softint context.
We can process thousands of packets for one hardware interrupt. Are
you able to comprehend this?
Measuring hardware interrupts in some was as "load" is about
as far from the truth as you can get.
This is just the tip of the iceberg.
rm -rf in-kernel-irqbalance;
And hey, if _YOU_ want a broken system which uses this bogus algorithm,
YOU CAN DO THIS with the userland thing if you want.
^ permalink raw reply [flat|nested] 76+ messages in thread
* Re: userspace irq balancer
2003-05-20 6:13 ` David S. Miller
@ 2003-05-20 6:36 ` Dave Hansen
2003-05-20 6:40 ` David S. Miller
2003-05-20 14:01 ` Martin J. Bligh
1 sibling, 1 reply; 76+ messages in thread
From: Dave Hansen @ 2003-05-20 6:36 UTC (permalink / raw)
To: David S. Miller
Cc: Martin J. Bligh, William Lee Irwin III, arjanv, Badari Pulavarty,
lkml, Gerrit Huizenga, John Stultz, James Cleverdon,
Andrew Morton, Keith Mannthey
On Mon, 2003-05-19 at 23:13, David S. Miller wrote:
> From: "Martin J. Bligh" <mbligh@aracnet.com>
> Date: Mon, 19 May 2003 22:53:11 -0700
>
> I have no frigging idea why you'd want to tear something out that
> works well already, and has a shitload of work put into it.
>
> It's pretty fundamentally broken for having had so much work
> put into it. Show me something other than "SpecWEB run for IBM
> ran faster" as a reason for keeping this code in there. Can you
> even do this?
I don't even think we can do that. That code was being integrated
around the same time that our Specweb setup decided to go south on us
and start physically frying itself. We never got a chance to run it.
BTW, I don't think there are any other kernel developers running Specweb
on 2.5 kernels. If there are, please speak up!
Andrew Theurer posted some positive results here, which were quite
marginal in the case with 1 nic. 4.7% with two.
http://marc.theaimsgroup.com/?l=linux-kernel&m=104212930819212&w=2
--
Dave Hansen
haveblue@us.ibm.com
^ permalink raw reply [flat|nested] 76+ messages in thread
* Re: userspace irq balancer
2003-05-20 6:36 ` Dave Hansen
@ 2003-05-20 6:40 ` David S. Miller
2003-05-20 14:07 ` Andrew Theurer
2003-05-21 11:00 ` Kai Bankett
0 siblings, 2 replies; 76+ messages in thread
From: David S. Miller @ 2003-05-20 6:40 UTC (permalink / raw)
To: haveblue
Cc: mbligh, wli, arjanv, pbadari, linux-kernel, gh, johnstul,
jamesclv, akpm, mannthey
From: Dave Hansen <haveblue@us.ibm.com>
Date: 19 May 2003 23:36:23 -0700
I don't even think we can do that. That code was being integrated
around the same time that our Specweb setup decided to go south on us
and start physically frying itself.
This gets more amusing by the second. Let's kill this code
already. People who like the current algorithms can push
them into the userspace solution.
^ permalink raw reply [flat|nested] 76+ messages in thread
* Re: userspace irq balancer
2003-05-20 6:40 ` David S. Miller
@ 2003-05-20 14:07 ` Andrew Theurer
2003-05-20 14:21 ` Jeff Garzik
[not found] ` <20030520.163833.104040023.davem@redhat.com>
2003-05-21 11:00 ` Kai Bankett
1 sibling, 2 replies; 76+ messages in thread
From: Andrew Theurer @ 2003-05-20 14:07 UTC (permalink / raw)
To: David S. Miller, haveblue
Cc: mbligh, wli, arjanv, pbadari, linux-kernel, gh, johnstul,
jamesclv, akpm, mannthey
On Tuesday 20 May 2003 01:40, David S. Miller wrote:
> From: Dave Hansen <haveblue@us.ibm.com>
> Date: 19 May 2003 23:36:23 -0700
>
> I don't even think we can do that. That code was being integrated
> around the same time that our Specweb setup decided to go south on us
> and start physically frying itself.
>
> This gets more amusing by the second. Let's kill this code
> already. People who like the current algorithms can push
> them into the userspace solution.
Remember this all started with some idea of "fairness" among cpus and very
little to do with performance. particularly on P4 with HT, where the first
logical cpu got all the ints and tasks running on that cpu were slower than
other cpus. This was in most cases the highest performing situation, -but-
it was unfair to the tasks running on cpu0. irq_balance fixed this with a
random target cpu that was in theory supposed to not change often enough to
preserve cache warmth. In practice is the target cpus changed too often
which thrashed cache and the HW overhead of changing the destination that
often was way way to high.
Although kirq was a step in the right direction (compared to irq_balance), I'd
rather see it in user space in the long term, too. That way we can make
policy changes much much easier. IMO, networking performance was always
better with all net card ints going to only one cpu, -until- that cpu would
be saturated. This situation point can come much sooner with HT since the
core is shared, and as far as I know, there is no way to bias the core to the
one sibling handling ints when int load is high. The only thing better than
all ints to cpu0 is aligning irq a process affinity together, which is 99%
unrealistic for all actual workloads.
Now, if someone can figure out how/when the first cpu is saturated, and
measure int load properly, maybe we can have a policy that keeps all ints on
cpu0, spills some ints to another cpu when that cpu is saturated, -and-
modifies find_busiest_queue to compensate nr_running on cpus with high int
load to make the process thingy more fair.
If kirq gets ripped out, at least have some default policy that is somewhat
harmless, like destination cpu = int_number % nr_cpus. I think Suse8 had
this, and it performed reasonably well.
-Andrew Theurer
^ permalink raw reply [flat|nested] 76+ messages in thread
* Re: userspace irq balancer
2003-05-20 14:07 ` Andrew Theurer
@ 2003-05-20 14:21 ` Jeff Garzik
2003-05-20 14:35 ` Andrew Theurer
[not found] ` <20030520.163833.104040023.davem@redhat.com>
1 sibling, 1 reply; 76+ messages in thread
From: Jeff Garzik @ 2003-05-20 14:21 UTC (permalink / raw)
To: Andrew Theurer
Cc: David S. Miller, haveblue, mbligh, wli, arjanv, pbadari,
linux-kernel, gh, johnstul, jamesclv, akpm, mannthey
On Tue, May 20, 2003 at 09:07:41AM -0500, Andrew Theurer wrote:
> On Tuesday 20 May 2003 01:40, David S. Miller wrote:
> > From: Dave Hansen <haveblue@us.ibm.com>
> > Date: 19 May 2003 23:36:23 -0700
> >
> > I don't even think we can do that. That code was being integrated
> > around the same time that our Specweb setup decided to go south on us
> > and start physically frying itself.
> >
> > This gets more amusing by the second. Let's kill this code
> > already. People who like the current algorithms can push
> > them into the userspace solution.
>
> Remember this all started with some idea of "fairness" among cpus and very
> little to do with performance. particularly on P4 with HT, where the first
> logical cpu got all the ints and tasks running on that cpu were slower than
> other cpus. This was in most cases the highest performing situation, -but-
> it was unfair to the tasks running on cpu0. irq_balance fixed this with a
> random target cpu that was in theory supposed to not change often enough to
> preserve cache warmth. In practice is the target cpus changed too often
> which thrashed cache and the HW overhead of changing the destination that
> often was way way to high.
You call that a fix? ;-) I call that working around a bug.
If tasks run slower on cpuX than cpuY because of a heavier int load,
that's the fault of the scheduler not the irqbalancer, be it in-kernel
or userspace. If there's a lesser-utilized cpu the task needs to be
migrated to that cpu from the irq-loaded one, when CPU accounting
notices the kernel interrupt handling having an impact.
Jeff
^ permalink raw reply [flat|nested] 76+ messages in thread
* Re: userspace irq balancer
2003-05-20 14:21 ` Jeff Garzik
@ 2003-05-20 14:35 ` Andrew Theurer
0 siblings, 0 replies; 76+ messages in thread
From: Andrew Theurer @ 2003-05-20 14:35 UTC (permalink / raw)
To: Jeff Garzik
Cc: David S. Miller, haveblue, mbligh, wli, arjanv, pbadari,
linux-kernel, gh, johnstul, jamesclv, akpm, mannthey
On Tuesday 20 May 2003 09:21, Jeff Garzik wrote:
> On Tue, May 20, 2003 at 09:07:41AM -0500, Andrew Theurer wrote:
> > On Tuesday 20 May 2003 01:40, David S. Miller wrote:
> > > From: Dave Hansen <haveblue@us.ibm.com>
> > > Date: 19 May 2003 23:36:23 -0700
> > >
> > > I don't even think we can do that. That code was being integrated
> > > around the same time that our Specweb setup decided to go south on
> > > us and start physically frying itself.
> > >
> > > This gets more amusing by the second. Let's kill this code
> > > already. People who like the current algorithms can push
> > > them into the userspace solution.
> >
> > Remember this all started with some idea of "fairness" among cpus and
> > very little to do with performance. particularly on P4 with HT, where
> > the first logical cpu got all the ints and tasks running on that cpu were
> > slower than other cpus. This was in most cases the highest performing
> > situation, -but- it was unfair to the tasks running on cpu0. irq_balance
> > fixed this with a random target cpu that was in theory supposed to not
> > change often enough to preserve cache warmth. In practice is the target
> > cpus changed too often which thrashed cache and the HW overhead of
> > changing the destination that often was way way to high.
>
> You call that a fix? ;-) I call that working around a bug.
>
> If tasks run slower on cpuX than cpuY because of a heavier int load,
> that's the fault of the scheduler not the irqbalancer, be it in-kernel
> or userspace. If there's a lesser-utilized cpu the task needs to be
> migrated to that cpu from the irq-loaded one, when CPU accounting
> notices the kernel interrupt handling having an impact.
On paper it sounds good but it may be more difficult (and not perfect) in
practice. For example, if you have one runnable task per cpu, moving it to
another cpu would not help. To preserve fairness, all you could do is swap
the tasks around, which would thrash the cache pretty well. Also, the int
load may be very dynamic, so you need to be really careful and make sure you
are measuring sustained int load.
However I do agree, we need something in the scheduler, in particular
something to modify max_load and load in find_busiest_queue, based on a int
load average for those cpus.
-Andrew
^ permalink raw reply [flat|nested] 76+ messages in thread
[parent not found: <20030520.163833.104040023.davem@redhat.com>]
* Re: userspace irq balancer
[not found] ` <20030520.163833.104040023.davem@redhat.com>
@ 2003-05-21 14:58 ` Martin J. Bligh
2003-05-21 22:55 ` David S. Miller
0 siblings, 1 reply; 76+ messages in thread
From: Martin J. Bligh @ 2003-05-21 14:58 UTC (permalink / raw)
To: David S. Miller, habanero
Cc: haveblue, wli, arjanv, pbadari, linux-kernel, gh, johnstul,
jamesclv, akpm, mannthey
> There is zero reason why IRQ balancing should be in any way
> different. It's POLICY, and POLICY == USERSPACE. It is the end
> of the argument.
Despite whatever political wrangling there is between userspace and
kernelspace implementations (and some very valid points about other
arches), there is still a dearth of testing, as far as I can see.
I can't see anything wrong with making it a config option for now,
and letting people choose what they want to do, until we have more
information as to which performs better under a variety of workloads.
That seems the most pragmatic way forward.
M.
^ permalink raw reply [flat|nested] 76+ messages in thread
* Re: userspace irq balancer
2003-05-21 14:58 ` Martin J. Bligh
@ 2003-05-21 22:55 ` David S. Miller
0 siblings, 0 replies; 76+ messages in thread
From: David S. Miller @ 2003-05-21 22:55 UTC (permalink / raw)
To: mbligh
Cc: habanero, haveblue, wli, arjanv, pbadari, linux-kernel, gh,
johnstul, jamesclv, akpm, mannthey
From: "Martin J. Bligh" <mbligh@aracnet.com>
Date: Wed, 21 May 2003 07:58:11 -0700
Despite whatever political wrangling there is between userspace and
kernelspace implementations (and some very valid points about other
arches), there is still a dearth of testing, as far as I can see.
I've never in my life heard the argument that we kept something
in the kernel that didn't belong there due to "userland testing".
That's a bogus argument.
When I ripped RARP out of the kernel, we didn't immediately have
a replacement, but one showed up shortly. So what?
And in this ase we already have Arjan's stuff. So start testing
his code instead of whining about keeping the current stuff in
the tree.
^ permalink raw reply [flat|nested] 76+ messages in thread
* Re: userspace irq balancer
2003-05-20 6:40 ` David S. Miller
2003-05-20 14:07 ` Andrew Theurer
@ 2003-05-21 11:00 ` Kai Bankett
1 sibling, 0 replies; 76+ messages in thread
From: Kai Bankett @ 2003-05-21 11:00 UTC (permalink / raw)
To: David S. Miller
Cc: haveblue, mbligh, wli, arjanv, pbadari, linux-kernel, gh,
johnstul, jamesclv, akpm, mannthey
[-- Attachment #1: Type: text/plain, Size: 1049 bytes --]
David S. Miller wrote:
> From: Dave Hansen <haveblue@us.ibm.com>
> Date: 19 May 2003 23:36:23 -0700
>
> I don't even think we can do that. That code was being integrated
> around the same time that our Specweb setup decided to go south on us
> and start physically frying itself.
>
>This gets more amusing by the second. Let's kill this code
>already. People who like the current algorithms can push
>them into the userspace solution.
>
>
>
That´s also my feeling.
After having more and more performance loss (at least for my
system/usage) and no real good solution in view to fix the problems at
this location in the kernel, I would prefer to get this thing ripped of
as soon as possible.
By now I do not see any gain with that piece of code (relative to
process/interrupt distribution). And please don´t show me benchmarks - I
know what I feel.
Userland would be first of all the right place to do any further testing.
And please - don´t tell me it´s that complicated to remove this piece of
code from source tree.
Kai.
[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/x-pkcs7-signature, Size: 3237 bytes --]
^ permalink raw reply [flat|nested] 76+ messages in thread
* Re: userspace irq balancer
2003-05-20 6:13 ` David S. Miller
2003-05-20 6:36 ` Dave Hansen
@ 2003-05-20 14:01 ` Martin J. Bligh
1 sibling, 0 replies; 76+ messages in thread
From: Martin J. Bligh @ 2003-05-20 14:01 UTC (permalink / raw)
To: David S. Miller
Cc: haveblue, wli, arjanv, pbadari, linux-kernel, gh, johnstul,
jamesclv, akpm, mannthey
> How does the in-kernel IRQ load balancing measure "load" and
> "busyness"? Herein lies the most absolutely fundamental problem with
> this code, it fails to recognize that we end up with most of our
> networking "load" from softint context.
OK, that's a great observation, and probably fixable. What were the
author's comments when you told him that?
> rm -rf in-kernel-irqbalance;
It's *very* late in the day to be ripping out such chunks of code.
1. Prove new code works better for you => make it a config option.
2. Prove new code works better for everyone => rip it out.
I think we're at 1, not 2.
Note that the userspace stuff doesn't even require that the kernel
stuff be disabled ... it should just override it (I can believe
there maybe is a bug that needs fixing, but it works by design).
M.
^ permalink raw reply [flat|nested] 76+ messages in thread
* Re: userspace irq balancer
2003-05-20 5:03 ` Dave Hansen
2003-05-20 5:53 ` Martin J. Bligh
@ 2003-05-20 9:00 ` Arjan van de Ven
2003-05-20 9:14 ` William Lee Irwin III
2003-05-20 9:17 ` Andrew Morton
1 sibling, 2 replies; 76+ messages in thread
From: Arjan van de Ven @ 2003-05-20 9:00 UTC (permalink / raw)
To: Dave Hansen
Cc: William Lee Irwin III, David S. Miller, Arjan van de Ven,
Badari Pulavarty, lkml, Gerrit Huizenga, John Stultz,
James Cleverdon, Andrew Morton, Keith Mannthey
On Mon, May 19, 2003 at 10:03:50PM -0700, Dave Hansen wrote:
> Does anyone have a patch to tear it out already? Is the current proc
> interface acceptable, or do we want a syscall interface like wli
> suggests?
I have no problems with the proc interface; it's ascii so reasonably
extendible in the future for, say, when 64 cpus on
32 bit linux get supported. It's also not THAT inefficient since my code
only uses it when some binding changes, not all the time.
^ permalink raw reply [flat|nested] 76+ messages in thread
* Re: userspace irq balancer
2003-05-20 9:00 ` Arjan van de Ven
@ 2003-05-20 9:14 ` William Lee Irwin III
2003-05-20 9:17 ` Andrew Morton
1 sibling, 0 replies; 76+ messages in thread
From: William Lee Irwin III @ 2003-05-20 9:14 UTC (permalink / raw)
To: Arjan van de Ven
Cc: Dave Hansen, David S. Miller, Badari Pulavarty, lkml,
Gerrit Huizenga, John Stultz, James Cleverdon, Andrew Morton,
Keith Mannthey
On Mon, May 19, 2003 at 10:03:50PM -0700, Dave Hansen wrote:
>> Does anyone have a patch to tear it out already? Is the current proc
>> interface acceptable, or do we want a syscall interface like wli
>> suggests?
On Tue, May 20, 2003 at 09:00:18AM +0000, Arjan van de Ven wrote:
> I have no problems with the proc interface; it's ascii so reasonably
> extendible in the future for, say, when 64 cpus on
> 32 bit linux get supported. It's also not THAT inefficient since my code
> only uses it when some binding changes, not all the time.
Sorry about that; I forgot about the /proc/ part and thought the thing
was based on system calls as it stood. I wouldn't want a redundant
interface to be added.
My current cpumask_t patches handle extending the /proc/ interface to
handle an arbitrary-sized cpumask, so I should have realized this.
-- wli
^ permalink raw reply [flat|nested] 76+ messages in thread
* Re: userspace irq balancer
2003-05-20 9:00 ` Arjan van de Ven
2003-05-20 9:14 ` William Lee Irwin III
@ 2003-05-20 9:17 ` Andrew Morton
[not found] ` <20030520.172230.102567463.davem@redhat.com>
1 sibling, 1 reply; 76+ messages in thread
From: Andrew Morton @ 2003-05-20 9:17 UTC (permalink / raw)
To: Arjan van de Ven
Cc: haveblue, wli, davem, arjanv, pbadari, linux-kernel, gh,
johnstul, jamesclv, mannthey
Arjan van de Ven <arjanv@redhat.com> wrote:
>
> On Mon, May 19, 2003 at 10:03:50PM -0700, Dave Hansen wrote:
> > Does anyone have a patch to tear it out already? Is the current proc
> > interface acceptable, or do we want a syscall interface like wli
> > suggests?
>
> I have no problems with the proc interface; it's ascii so reasonably
> extendible in the future for, say, when 64 cpus on
> 32 bit linux get supported. It's also not THAT inefficient since my code
> only uses it when some binding changes, not all the time.
Concerns have been expressed that the /proc interface may be a bit racy.
One thing we do need to do is to write a /proc stresstest tool which pokes
numbers into the /proc files at high rates, run that under traffic for a
few hours.
There is no need to pull out the existing balancer until the userspace
solution is proven - it can be turned off with `noirqbalance' until that
work has been performed.
Nobody has tried improving the current balancer. From a quick look it
appears that it could work reasonably for the problematic packet-forwarding
workload if the when-to-start-balancing threshold is reduced from 1000/sec
to (say) 10/sec. Don't know - I've never seen a description of how the
algorithm should be improved.
^ permalink raw reply [flat|nested] 76+ messages in thread
* RE: userspace irq balancer
@ 2003-05-20 15:41 Nakajima, Jun
2003-05-21 13:54 ` James Cleverdon
0 siblings, 1 reply; 76+ messages in thread
From: Nakajima, Jun @ 2003-05-20 15:41 UTC (permalink / raw)
To: Martin J. Bligh, David S. Miller
Cc: haveblue, wli, arjanv, pbadari, linux-kernel, gh, johnstul,
jamesclv, akpm, mannthey
The in-kernel load balance does not move IRQs that are bound to particular CPUs. If the user-level can do it better, just set affinity, and I believe that is the current implementation.
The in-kernel one is simply trying to emulate functionality in the chipset, thus it's not so intelligent, of course. The major reason we need to do it in software is that x86 Linux does not update TPR(s).
Thanks,
Jun
> -----Original Message-----
> From: Martin J. Bligh [mailto:mbligh@aracnet.com]
> Sent: Tuesday, May 20, 2003 7:01 AM
> To: David S. Miller
> Cc: haveblue@us.ibm.com; wli@holomorphy.com; arjanv@redhat.com;
> pbadari@us.ibm.com; linux-kernel@vger.kernel.org; gh@us.ibm.com;
> johnstul@us.ibm.com; jamesclv@us.ibm.com; akpm@digeo.com;
> mannthey@us.ibm.com
> Subject: Re: userspace irq balancer
>
> > How does the in-kernel IRQ load balancing measure "load" and
> > "busyness"? Herein lies the most absolutely fundamental problem with
> > this code, it fails to recognize that we end up with most of our
> > networking "load" from softint context.
>
> OK, that's a great observation, and probably fixable. What were the
> author's comments when you told him that?
>
> > rm -rf in-kernel-irqbalance;
>
> It's *very* late in the day to be ripping out such chunks of code.
> 1. Prove new code works better for you => make it a config option.
> 2. Prove new code works better for everyone => rip it out.
>
> I think we're at 1, not 2.
>
> Note that the userspace stuff doesn't even require that the kernel
> stuff be disabled ... it should just override it (I can believe
> there maybe is a bug that needs fixing, but it works by design).
>
> M.
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
^ permalink raw reply [flat|nested] 76+ messages in thread
* Re: userspace irq balancer
2003-05-20 15:41 Nakajima, Jun
@ 2003-05-21 13:54 ` James Cleverdon
2003-05-21 22:56 ` Zwane Mwaikambo
0 siblings, 1 reply; 76+ messages in thread
From: James Cleverdon @ 2003-05-21 13:54 UTC (permalink / raw)
To: Nakajima, Jun, Martin J. Bligh, David S. Miller
Cc: haveblue, wli, arjanv, pbadari, linux-kernel, gh, johnstul, akpm,
mannthey
On Tuesday 20 May 2003 08:41 am, Nakajima, Jun wrote:
> The in-kernel load balance does not move IRQs that are bound to particular
> CPUs. If the user-level can do it better, just set affinity, and I believe
> that is the current implementation.
>
> The in-kernel one is simply trying to emulate functionality in the chipset,
> thus it's not so intelligent, of course. The major reason we need to do it
> in software is that x86 Linux does not update TPR(s).
>
> Thanks,
> Jun
It may be time to think about using the TPRs again, and see if HW interrupt
routing helps Arjan's test case. Of course for any system using clustered
APIC mode, we will still need to decide which APIC cluster gets which IRQ....
> > -----Original Message-----
> > From: Martin J. Bligh [mailto:mbligh@aracnet.com]
> > Sent: Tuesday, May 20, 2003 7:01 AM
> > To: David S. Miller
> > Cc: haveblue@us.ibm.com; wli@holomorphy.com; arjanv@redhat.com;
> > pbadari@us.ibm.com; linux-kernel@vger.kernel.org; gh@us.ibm.com;
> > johnstul@us.ibm.com; jamesclv@us.ibm.com; akpm@digeo.com;
> > mannthey@us.ibm.com
> > Subject: Re: userspace irq balancer
> >
> > > How does the in-kernel IRQ load balancing measure "load" and
> > > "busyness"? Herein lies the most absolutely fundamental problem with
> > > this code, it fails to recognize that we end up with most of our
> > > networking "load" from softint context.
> >
> > OK, that's a great observation, and probably fixable. What were the
> > author's comments when you told him that?
> >
> > > rm -rf in-kernel-irqbalance;
> >
> > It's *very* late in the day to be ripping out such chunks of code.
> > 1. Prove new code works better for you => make it a config option.
> > 2. Prove new code works better for everyone => rip it out.
> >
> > I think we're at 1, not 2.
> >
> > Note that the userspace stuff doesn't even require that the kernel
> > stuff be disabled ... it should just override it (I can believe
> > there maybe is a bug that needs fixing, but it works by design).
> >
> > M.
--
James Cleverdon
IBM xSeries Linux Solutions
{jamesclv(Unix, preferred), cleverdj(Notes)} at us dot ibm dot com
^ permalink raw reply [flat|nested] 76+ messages in thread
* Re: userspace irq balancer
2003-05-21 13:54 ` James Cleverdon
@ 2003-05-21 22:56 ` Zwane Mwaikambo
0 siblings, 0 replies; 76+ messages in thread
From: Zwane Mwaikambo @ 2003-05-21 22:56 UTC (permalink / raw)
To: James Cleverdon
Cc: Nakajima, Jun, Martin J. Bligh, David S. Miller, haveblue, wli,
arjanv, pbadari, linux-kernel, gh, johnstul, akpm, mannthey
On Wed, 21 May 2003, James Cleverdon wrote:
> It may be time to think about using the TPRs again, and see if HW interrupt
> routing helps Arjan's test case. Of course for any system using clustered
> APIC mode, we will still need to decide which APIC cluster gets which IRQ....
You can build cpu masks of capable clusters easily, even for NUMAQ
Zwane
--
function.linuxpower.ca
^ permalink raw reply [flat|nested] 76+ messages in thread
* Re: userspace irq balancer
@ 2003-05-21 16:31 James Bottomley
2003-05-21 20:16 ` Arjan van de Ven
0 siblings, 1 reply; 76+ messages in thread
From: James Bottomley @ 2003-05-21 16:31 UTC (permalink / raw)
To: Linux Kernel
I'm interested in using this for voyager. However, I have a problem in
that voyager may have CPUs that can't accept interrupts (this is global
on voyager, but may be per-interrupt on NUMA like systems). I think
before we move to a userspace solution, some thought about how to cope
with this is needed.
I have several suggestions:
1. Place the masks into /proc/irq/<n>/smp_affinity at start of day and
have the userspace irqbalancer take this as the maximal mask
2. Have a separate file /proc/irq/<n>/mask(?) to expose the mask always
3. Some other method...
Comments would be welcome
James
^ permalink raw reply [flat|nested] 76+ messages in thread
* RE: userspace irq balancer
@ 2003-05-21 21:43 Nakajima, Jun
2003-05-22 0:29 ` Gerrit Huizenga
0 siblings, 1 reply; 76+ messages in thread
From: Nakajima, Jun @ 2003-05-21 21:43 UTC (permalink / raw)
To: jamesclv; +Cc: haveblue, pbadari, linux-kernel, gh, johnstul, mannthey
Again, since the userland is using /proc/irq/N/smp_affinity, the in-kernel one won't touch whatever settings done by the userlannd. So I don't think we have issues here - if the userland has more knowledge, then it simply uses binding. If not, use the generic but dumb one in the kernel. Same thing as scheduling. If the dumb one has a critical problem, we should fix it.
At the same time, I don't believe a single almighty userland policy exists, either. One might need to write or modify his program to do the best for the system anyway. Or a very simple script might just work fine.
Jun
> -----Original Message-----
> From: James Cleverdon [mailto:jamesclv@us.ibm.com]
> Sent: Wednesday, May 21, 2003 7:27 AM
> To: David S. Miller; akpm@digeo.com
> Cc: arjanv@redhat.com; haveblue@us.ibm.com; wli@holomorphy.com;
> pbadari@us.ibm.com; linux-kernel@vger.kernel.org; gh@us.ibm.com;
> johnstul@us.ibm.com; mannthey@us.ibm.com
> Subject: Re: userspace irq balancer
>
> On Tuesday 20 May 2003 05:22 pm, David S. Miller wrote:
> > From: Andrew Morton <akpm@digeo.com>
> > Date: Tue, 20 May 2003 02:17:12 -0700
> >
> > Concerns have been expressed that the /proc interface may be a bit
> racy.
> > One thing we do need to do is to write a /proc stresstest tool which
> > pokes numbers into the /proc files at high rates, run that under traffic
> > for a few hours.
> >
> > This issue is %100 independant of whether policy belongs in the
> > kernel or not. Also, the /proc race problem exists and should be
> > fixed regardless.
> >
> > Nobody has tried improving the current balancer.
> >
> > Policy does not belong in the kernel. I don't care what algorithm
> > people decide to use, but such decisions do NOT belong in the kernel.
>
> You keep saying that, but suppose I want to try HW IRQ balancing using the
> TPR
> registers. How could I do that from userspace? And if I could, wouldn't
> the
> benefit of real time IRQ routing be lost?
>
> It seems to me that only long term interrupt policy can be done from
> userland.
> Anything that does fast responses to fluctuating load must be inside the
> kernel.
>
> At the moment we don't do any fast IRQ policy. Even the original
> irq_balance
> only looked for idle CPUs after an interrupt was serviced. However,
> suppose
> you had a P4 with hyperthreading turned on. If an IRQ is to be delivered
> to
> the main thread but it is busy and its sibling is idle, why shouldn't we
> deliver the interrupt to the idle sibling? They both share the same
> caches,
> etc, so cache warmth isn't a problem.
>
> --
> James Cleverdon
> IBM xSeries Linux Solutions
> {jamesclv(Unix, preferred), cleverdj(Notes)} at us dot ibm dot com
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
^ permalink raw reply [flat|nested] 76+ messages in thread
* Re: userspace irq balancer
2003-05-21 21:43 Nakajima, Jun
@ 2003-05-22 0:29 ` Gerrit Huizenga
2003-05-22 1:28 ` Martin J. Bligh
2003-05-22 2:04 ` William Lee Irwin III
0 siblings, 2 replies; 76+ messages in thread
From: Gerrit Huizenga @ 2003-05-22 0:29 UTC (permalink / raw)
To: Nakajima, Jun
Cc: jamesclv, haveblue, pbadari, linux-kernel, johnstul, mannthey
Yeah, I suppose this userland policy change means we should pull
the scheduler policy decisions out of the kernel and write user level
HT, NUMA, SMP and UP schedulers. Also, the IO schedulers should
probably be pulled out - I'm sure AS and CFQ and linus_scheduler
could be user land policies, as well as the elevator. Memory
placement and swapping policies, too.
Oh, wait, some people actually do this - they call it, what,
Workload Management or some such thing. But I don't know any
style of workload management that leaves *no* default, semi-sane
policy in the kernel.
gerrit
On Wed, 21 May 2003 14:43:18 PDT, "Nakajima, Jun" wrote:
> Again, since the userland is using /proc/irq/N/smp_affinity, the
> in-kernel one won't touch whatever settings done by the userlannd. So
> I don't think we have issues here - if the userland has more knowledge,
> then it simply uses binding. If not, use the generic but dumb one in the
> kernel. Same thing as scheduling. If the dumb one has a critical problem,
> we should fix it.
>
> At the same time, I don't believe a single almighty userland policy
> exists, either. One might need to write or modify his program to do the
> best for the system anyway. Or a very simple script might just work fine.
>
> Jun
> > -----Original Message-----
> > From: James Cleverdon [mailto:jamesclv@us.ibm.com]
> > Sent: Wednesday, May 21, 2003 7:27 AM
> > To: David S. Miller; akpm@digeo.com
> > Cc: arjanv@redhat.com; haveblue@us.ibm.com; wli@holomorphy.com;
> > pbadari@us.ibm.com; linux-kernel@vger.kernel.org; gh@us.ibm.com;
> > johnstul@us.ibm.com; mannthey@us.ibm.com
> > Subject: Re: userspace irq balancer
> >
> > On Tuesday 20 May 2003 05:22 pm, David S. Miller wrote:
> > > From: Andrew Morton <akpm@digeo.com>
> > > Date: Tue, 20 May 2003 02:17:12 -0700
> > >
> > > Concerns have been expressed that the /proc interface may be a bit
> > racy.
> > > One thing we do need to do is to write a /proc stresstest tool which
> > > pokes numbers into the /proc files at high rates, run that under traffic
> > > for a few hours.
> > >
> > > This issue is %100 independant of whether policy belongs in the
> > > kernel or not. Also, the /proc race problem exists and should be
> > > fixed regardless.
> > >
> > > Nobody has tried improving the current balancer.
> > >
> > > Policy does not belong in the kernel. I don't care what algorithm
> > > people decide to use, but such decisions do NOT belong in the kernel.
> >
> > You keep saying that, but suppose I want to try HW IRQ balancing using the
> > TPR
> > registers. How could I do that from userspace? And if I could, wouldn't
> > the
> > benefit of real time IRQ routing be lost?
> >
> > It seems to me that only long term interrupt policy can be done from
> > userland.
> > Anything that does fast responses to fluctuating load must be inside the
> > kernel.
> >
> > At the moment we don't do any fast IRQ policy. Even the original
> > irq_balance
> > only looked for idle CPUs after an interrupt was serviced. However,
> > suppose
> > you had a P4 with hyperthreading turned on. If an IRQ is to be delivered
> > to
> > the main thread but it is busy and its sibling is idle, why shouldn't we
> > deliver the interrupt to the idle sibling? They both share the same
> > caches,
> > etc, so cache warmth isn't a problem.
> >
> > --
> > James Cleverdon
> > IBM xSeries Linux Solutions
> > {jamesclv(Unix, preferred), cleverdj(Notes)} at us dot ibm dot com
> >
> > -
> > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
> > Please read the FAQ at http://www.tux.org/lkml/
>
>
^ permalink raw reply [flat|nested] 76+ messages in thread
* Re: userspace irq balancer
2003-05-22 0:29 ` Gerrit Huizenga
@ 2003-05-22 1:28 ` Martin J. Bligh
2003-05-22 1:44 ` Gerrit Huizenga
2003-05-22 2:04 ` William Lee Irwin III
1 sibling, 1 reply; 76+ messages in thread
From: Martin J. Bligh @ 2003-05-22 1:28 UTC (permalink / raw)
To: Gerrit Huizenga, Nakajima, Jun
Cc: jamesclv, haveblue, pbadari, linux-kernel, johnstul, mannthey
> Yeah, I suppose this userland policy change means we should pull
> the scheduler policy decisions out of the kernel and write user level
> HT, NUMA, SMP and UP schedulers. Also, the IO schedulers should
> probably be pulled out - I'm sure AS and CFQ and linus_scheduler
> could be user land policies, as well as the elevator. Memory
> placement and swapping policies, too.
>
> Oh, wait, some people actually do this - they call it, what,
> Workload Management or some such thing. But I don't know any
> style of workload management that leaves *no* default, semi-sane
> policy in the kernel.
I think the word you're groping for here is "microkernel".
M.
^ permalink raw reply [flat|nested] 76+ messages in thread
* Re: userspace irq balancer
2003-05-22 1:28 ` Martin J. Bligh
@ 2003-05-22 1:44 ` Gerrit Huizenga
2003-05-22 2:03 ` William Lee Irwin III
0 siblings, 1 reply; 76+ messages in thread
From: Gerrit Huizenga @ 2003-05-22 1:44 UTC (permalink / raw)
To: Martin J. Bligh
Cc: Nakajima, Jun, jamesclv, haveblue, pbadari, linux-kernel,
johnstul, mannthey
On Wed, 21 May 2003 18:28:56 PDT, "Martin J. Bligh" wrote:
> > Yeah, I suppose this userland policy change means we should pull
> > the scheduler policy decisions out of the kernel and write user level
> > HT, NUMA, SMP and UP schedulers. Also, the IO schedulers should
> > probably be pulled out - I'm sure AS and CFQ and linus_scheduler
> > could be user land policies, as well as the elevator. Memory
> > placement and swapping policies, too.
> >
> > Oh, wait, some people actually do this - they call it, what,
> > Workload Management or some such thing. But I don't know any
> > style of workload management that leaves *no* default, semi-sane
> > policy in the kernel.
>
> I think the word you're groping for here is "microkernel".
>
> M.
Oh, yeah. Page replacement policy in user level. That one was
a real winner.
gerrit
^ permalink raw reply [flat|nested] 76+ messages in thread
* Re: userspace irq balancer
2003-05-22 1:44 ` Gerrit Huizenga
@ 2003-05-22 2:03 ` William Lee Irwin III
0 siblings, 0 replies; 76+ messages in thread
From: William Lee Irwin III @ 2003-05-22 2:03 UTC (permalink / raw)
To: Gerrit Huizenga
Cc: Martin J. Bligh, Nakajima, Jun, jamesclv, haveblue, pbadari,
linux-kernel, johnstul, mannthey
On Wed, 21 May 2003 18:28:56 PDT, "Martin J. Bligh" wrote:
>> I think the word you're groping for here is "microkernel".
On Wed, May 21, 2003 at 06:44:46PM -0700, Gerrit Huizenga wrote:
> Oh, yeah. Page replacement policy in user level. That one was
> a real winner.
That's incorrect. Page replacement policy at the user-level was
proposed but AIUI not included in Mach. You're thinking of external
pagers, which are grossly inefficient but have only to do with the
I/O to fetch and store pages belonging to a memory object, not with
page replacement. Mach's page replacement policy was global SEGQ.
This is specifically discussed by Vahalia.
-- wli
^ permalink raw reply [flat|nested] 76+ messages in thread
* Re: userspace irq balancer
2003-05-22 0:29 ` Gerrit Huizenga
2003-05-22 1:28 ` Martin J. Bligh
@ 2003-05-22 2:04 ` William Lee Irwin III
2003-05-22 2:12 ` Zwane Mwaikambo
` (2 more replies)
1 sibling, 3 replies; 76+ messages in thread
From: William Lee Irwin III @ 2003-05-22 2:04 UTC (permalink / raw)
To: Gerrit Huizenga
Cc: Nakajima, Jun, jamesclv, haveblue, pbadari, linux-kernel,
johnstul, mannthey
On Wed, May 21, 2003 at 05:29:45PM -0700, Gerrit Huizenga wrote:
> Yeah, I suppose this userland policy change means we should pull
> the scheduler policy decisions out of the kernel and write user level
> HT, NUMA, SMP and UP schedulers. Also, the IO schedulers should
> probably be pulled out - I'm sure AS and CFQ and linus_scheduler
> could be user land policies, as well as the elevator. Memory
> placement and swapping policies, too.
> Oh, wait, some people actually do this - they call it, what,
> Workload Management or some such thing. But I don't know any
> style of workload management that leaves *no* default, semi-sane
> policy in the kernel.
This is not the case. Interrupt arbitration for sane things generally
balances interrupt load automatically in-hardware. AIUI the TPR was
intended to enable the hardware to do such a thing for xAPIC. Linux
doesn't use the TPR now, which results in decisions made by the
hardware on xAPIC -based SMP systems that are highly detrimental to
performance.
Allowing userspace to exploit more specific knowledge and perform
either static or userspace-controlled dynamic interrupt affinity
is not equivalent to having an insane default policy in-kernel.
The task scheduler, the io scheduler, and memory entitlement policies
are very different issues. They deal entirely with managing software
constructs and resource allocation. Memory placement policies sit at
least two or three levels above anything hardware memory management
can do and it is safe to say it's infeasible to implement NUMA memory
placement policies in hardware.
Interrupt load balancing is very much doable in hardware and prior to
xAPIC it was done so in all cases; for xAPIC the hardware mechanism
became strictly bound to the TPR and had less optimal tiebreak
resolution decisions (something on the order of defaulting to the
lowest APIC ID in the event of a tie, which always occurs if the TPR
isn't frobbed). This naturally creates a problem, which these userspace
and kernel mechanisms are meant to address.
The difficulty with the in-kernel policy is that its decisions are not
optimal for all cases, and it has implementation issues that prevent it
from being fully generally used, i.e. it does not handle the physical
DESTMOD case for pre-xAPIC systems with multiple APIC buses, which
amounts to a very simple incompleteness of what to all outward
appearances is an already large and feature-rich implementation; the
kernel code merely refrains from calling it in that case as a brute-
force workaround. Furthermore the complexity of the decisions to be
made is inappropriate for the kernel. It needs unusual (and slow)
manipulation of hardware to be done in code requiring fast response
times in various cases and that is called at an uncontrollable rate. It
has heuristics which may be inaccurate or wrong for various cases.
IMHO Linux on Pentium IV should use the TPR in conjunction with _very_
simplistic interrupt load accounting by default and all more
sophisticated logic should be punted straight to userspace as an
administrative API.
To quote chapter and verse:
IA-32 Intel Architecture Software Developer's Manual
Volume 3: System Programming Guide
Section 8.6.2.4 "Lowest Priority Delivery Mode"
"In operating systems that use the lowest-priority delivery mode but do
not update the TPR, the TPR information saved in the chipset will
potentially cause the interrupt to always be delivered to the same
processor from the logical set. This behavior is functionally backward
compatible with the P6 family processor but may result in unexpected
performance implications."
i.e. frob the fscking TPR as recommended by the APIC docs every once in
a while by default, punt anything (and everything) fancier up to
userspace, and get the code that doesn't even understand what the fsck
DESTMOD means the Hell out of the kernel and the Hell away from my
IO-APIC RTE's.
-- wli
^ permalink raw reply [flat|nested] 76+ messages in thread
* Re: userspace irq balancer
2003-05-22 2:04 ` William Lee Irwin III
@ 2003-05-22 2:12 ` Zwane Mwaikambo
2003-05-22 3:57 ` Martin J. Bligh
2003-05-22 14:18 ` James Cleverdon
2 siblings, 0 replies; 76+ messages in thread
From: Zwane Mwaikambo @ 2003-05-22 2:12 UTC (permalink / raw)
To: William Lee Irwin III
Cc: Gerrit Huizenga, Nakajima, Jun, jamesclv, haveblue, pbadari,
linux-kernel, johnstul, mannthey
On Wed, 21 May 2003, William Lee Irwin III wrote:
> This is not the case. Interrupt arbitration for sane things generally
> balances interrupt load automatically in-hardware. AIUI the TPR was
> intended to enable the hardware to do such a thing for xAPIC. Linux
> doesn't use the TPR now, which results in decisions made by the
> hardware on xAPIC -based SMP systems that are highly detrimental to
> performance.
Well using the APIC arbitration round robin thing isn't all that smart
either unless you use the TPR, so TPR would be a win everywhere.
> IMHO Linux on Pentium IV should use the TPR in conjunction with _very_
> simplistic interrupt load accounting by default and all more
> sophisticated logic should be punted straight to userspace as an
> administrative API.
>
> i.e. frob the fscking TPR as recommended by the APIC docs every once in
> a while by default, punt anything (and everything) fancier up to
> userspace, and get the code that doesn't even understand what the fsck
> DESTMOD means the Hell out of the kernel and the Hell away from my
> IO-APIC RTE's.
Word... This is all rather tired, if we have a working irq affinity user
accessible interface this can all go away, so how about we just work
towards that means, and then remove kirqd when everyone is happy
(personally i like Arjan's/RH9 userland irqbalance).
Zwane
--
function.linuxpower.ca
^ permalink raw reply [flat|nested] 76+ messages in thread
* Re: userspace irq balancer
2003-05-22 2:04 ` William Lee Irwin III
2003-05-22 2:12 ` Zwane Mwaikambo
@ 2003-05-22 3:57 ` Martin J. Bligh
2003-05-22 17:24 ` Bill Davidsen
2003-05-22 14:18 ` James Cleverdon
2 siblings, 1 reply; 76+ messages in thread
From: Martin J. Bligh @ 2003-05-22 3:57 UTC (permalink / raw)
To: William Lee Irwin III, Gerrit Huizenga
Cc: Nakajima, Jun, jamesclv, haveblue, pbadari, linux-kernel,
johnstul, mannthey
> The task scheduler, the io scheduler, and memory entitlement policies
> are very different issues. They deal entirely with managing software
> constructs and resource allocation.
So we should expose low-level hardware stuff to userspace to manage,
but not higher level software constructs? I fail to see the abiding
logic there. If anything, the inverse ought to be true.
> IMHO Linux on Pentium IV should use the TPR in conjunction with _very_
> simplistic interrupt load accounting by default and all more
> sophisticated logic should be punted straight to userspace as an
> administrative API.
I'd be happy with that - sounds to me like you're arguing for the same
thing. Sane default in kernel, can override from userspace if you like.
However, I've yet to see an implementation of the TPR usage that got
good performance numbers ... I'd love to see that happen.
M.
^ permalink raw reply [flat|nested] 76+ messages in thread
* Re: userspace irq balancer
2003-05-22 3:57 ` Martin J. Bligh
@ 2003-05-22 17:24 ` Bill Davidsen
2003-05-22 22:44 ` David S. Miller
0 siblings, 1 reply; 76+ messages in thread
From: Bill Davidsen @ 2003-05-22 17:24 UTC (permalink / raw)
To: Dave Hansen, David S. Miller, Andrew Theurer, Martin J. Bligh; +Cc: lkml
On 19 May 2003, Dave Hansen wrote:
> On Mon, 2003-05-19 at 15:11, Arjan van de Ven wrote:
[...snip...]
> But, do you see the need for ripping out the current code? For those of
> us that are still running a slightly more primitive distro, it would be
> nice to have some pretty effective default behavior, like what is in the
> kernel now.
Ripping out the current code and having useful default behaviour are
hopefully not mutually exclusive.
On 19 May 2003, David S. Miller wrote:
> You have to install new modutils to even use modules with the 2.5.x
> kernel, given that why are we even talking about the "inconvenience"
> of installing the usermode IRQ balancer as being a blocker for
> ripping out the in-kernel stuff?
But you don't have to use modules at all, while running without
interrupts isn't an option.
> The in-kernel stuff MUST go. It went in because "some benchmark went
> faster", but with no "why" describing why it might have improved
> performance. We KNOW it absolutely sucks for routing and firewall
> applications. The in-kernel bits were all a shamans dance, with zero
> technical "here is why this makes things go faster" description
> attached. If I remember properly, the changelog message when the
> in-kernel irq balancing went in was of the form "this makes some
> specweb run go faster".
Perhaps I misread Linus' recent post about not breaking things to the
user, and I know he was talking about executables in particular, but if
this new code is so great, why can't it start with some default values
which will be no worse than what we have? Because there will be people
who haven't installed the latest userspace int-diddler.
Deliberately making the initial tuning useless to promote use of the
user space software seems counter productive. There will be many users
who will find a way to make a tuned kernel worse than any default, so if
the default is usable it will avoid people shooting themselves in the
foot. Can you imagine someone leaving an interrupt unserviced at all? If
there's a way someone will :-(
I'm not defending the existing code, just speaking for the idea of
leaving something useful in its place.
On Tue, 20 May 2003, Andrew Theurer wrote:
[...snip...]
> If kirq gets ripped out, at least have some default policy that is somewhat
> harmless, like destination cpu = int_number % nr_cpus. I think Suse8 had
> this, and it performed reasonably well.
As long as it's something sane, it doesn't have to be optimal. If SuSE
used it, it's likely to be good enough.
On Wed, 21 May 2003, Martin J. Bligh wrote:
[...snip...]
> I'd be happy with that - sounds to me like you're arguing for the same
> thing. Sane default in kernel, can override from userspace if you like.
> However, I've yet to see an implementation of the TPR usage that got
> good performance numbers ... I'd love to see that happen.
I may be misreading the intent in David's messages, as long as the
default is useful it certainly doesn't have to be what is in place now.
Of couse it would be useful to actually see some numbers, just because
existing code is somewhat ugly, confusing, and ill-justified and the new
is a pretty algorithm with great flexibility, does not mean that the
actual benchmarks will reflect better performance.
--
bill davidsen <davidsen@tmr.com>
CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.
^ permalink raw reply [flat|nested] 76+ messages in thread
* Re: userspace irq balancer
2003-05-22 17:24 ` Bill Davidsen
@ 2003-05-22 22:44 ` David S. Miller
2003-05-26 22:24 ` Andrea Arcangeli
0 siblings, 1 reply; 76+ messages in thread
From: David S. Miller @ 2003-05-22 22:44 UTC (permalink / raw)
To: davidsen; +Cc: haveblue, habanero, mbligh, linux-kernel
From: Bill Davidsen <davidsen@tmr.com>
Date: Thu, 22 May 2003 13:24:26 -0400 (EDT)
> You have to install new modutils to even use modules with the 2.5.x
> kernel, given that why are we even talking about the "inconvenience"
> of installing the usermode IRQ balancer as being a blocker for
> ripping out the in-kernel stuff?
But you don't have to use modules at all, while running without
interrupts isn't an option.
Interrupts WORK without the IRQ balancing.
Come one people, get real already.
^ permalink raw reply [flat|nested] 76+ messages in thread
* Re: userspace irq balancer
2003-05-22 22:44 ` David S. Miller
@ 2003-05-26 22:24 ` Andrea Arcangeli
2003-05-26 23:26 ` Andrew Morton
0 siblings, 1 reply; 76+ messages in thread
From: Andrea Arcangeli @ 2003-05-26 22:24 UTC (permalink / raw)
To: David S. Miller; +Cc: davidsen, haveblue, habanero, mbligh, linux-kernel
On Thu, May 22, 2003 at 03:44:10PM -0700, David S. Miller wrote:
> From: Bill Davidsen <davidsen@tmr.com>
> Date: Thu, 22 May 2003 13:24:26 -0400 (EDT)
>
> > You have to install new modutils to even use modules with the 2.5.x
> > kernel, given that why are we even talking about the "inconvenience"
> > of installing the usermode IRQ balancer as being a blocker for
> > ripping out the in-kernel stuff?
>
> But you don't have to use modules at all, while running without
> interrupts isn't an option.
>
> Interrupts WORK without the IRQ balancing.
>
> Come one people, get real already.
this is not the point, the whole point of irq balancing is performance,
not functionality.
Tell me how to do what I'm doing in my 2.4 tree in userspace (this was my
partially rewritten implementation based on Ingo's original code that I found
very suboptimal while reading it):
#ifdef CONFIG_SMP
#define IRQ_BALANCE_INTERVAL (HZ/50)
typedef struct {
unsigned int cpu;
unsigned long timestamp;
} ____cacheline_aligned irq_balance_t;
static irq_balance_t irq_balance[NR_IRQS] __cacheline_aligned;
extern unsigned long irq_affinity [NR_IRQS];
#define IRQ_ALLOWED(cpu,allowed_mask) \
((1UL << cpu) & (allowed_mask))
static unsigned long move(unsigned int curr_cpu, unsigned long allowed_mask, unsigned long now, int direction)
{
unsigned int cpu = curr_cpu;
unsigned int phys_id;
phys_id = cpu_logical_map(cpu);
if (IRQ_ALLOWED(phys_id, allowed_mask) && idle_cpu(phys_id))
return cpu;
goto inside;
do {
if (unlikely(cpu == curr_cpu))
return cpu;
inside:
if (direction == 1) {
cpu++;
if (cpu >= smp_num_cpus)
cpu = 0;
} else {
cpu--;
if (cpu == -1)
cpu = smp_num_cpus-1;
}
phys_id = cpu_logical_map(cpu);
} while (!IRQ_ALLOWED(phys_id, allowed_mask) || !idle_cpu(phys_id));
return cpu;
}
#endif /* CONFIG_SMP */
static inline void balance_irq(int irq)
{
#if CONFIG_SMP
irq_balance_t *entry = irq_balance + irq;
unsigned long now = jiffies;
if (unlikely(time_after(now, entry->timestamp + IRQ_BALANCE_INTERVAL))) {
unsigned long allowed_mask;
unsigned int new_cpu;
int random_number;
entry->timestamp = now;
rdtscl(random_number);
random_number &= 1;
allowed_mask = cpu_online_map & irq_affinity[irq];
new_cpu = move(entry->cpu, allowed_mask, now, random_number);
if (entry->cpu != new_cpu) {
entry->cpu = new_cpu;
set_ioapic_affinity(irq, clustered_apic_mode == 0 ?
1UL << new_cpu : cpu_present_to_apicid(entry->cpu) );
}
}
#endif
}
this is much better than whatever you can do in userspace. You'll never be able
to react quickly to avoid wasting idle timeslices of cpus in userspace.
Unfortunately I didn't see number comparisons of this with the userspace
balancer, but the above in the numbers is an order of magnitude better
than all other 2.4 patches I seen floating around. the other patches are
even reprogramming the IO-apic when no irq routing change is done, and
they reprogram at an overkill frequency, so no surprise the above is
much faster and no surprise the irq balancer as well would be much
faster than the other patches.
Would be interesting to compare the above with the userspace balancer in
a system with some cpu sometime idle, like some network bound file
serving or similar.
To you it seems obvious no kernel-only optimization like the above (not
remotely doable in userspace) can ever exist, this isn't true as far as I can
tell, and to me those heuristics make lots of sense, especially on a 64way smp
not under 100% cpu utilization.
and if you put the userspace balancer on top of the above the above will
be only a branch cost (not even an extern call) (unlike the other
patches that were reprogramming the ioapic regardless).
And overall I believe it doesn't make much sense to leave irq balancing in
userspace, the advantages seems too few and the disavantages seems way too
many.
The portability argument is a plain symptom of the lack of an API to write the
algorithm in a portable way: the proof is the /proc/irq interface that for
istance is still duplicated across all ports. Grep for register_irq_proc,
you'll find it in sparc64 in i386 etc.. fix that and the portability argument
will go away. Infact I wish that the sparc64 port will finally go in sync with
the rest of the kernel with regards to irqs. x86, alpha x86-64 ia64 are
all just using the exact same logic for irq handling, sparc is the only
relevant arch that refuses to go in sync. If you miss functionalty from
the current irq API in all major ports just extend it and put sparc in
sync so finally irq.c can be moved to kernel and not under arch/. If
sparc64 is so much smarter than the rest of the archs in handling
interrupts then just make it the standard for all ports. Then be sure
there will be no risk of portability issues while writing an irq
balancing algorithm in kernel. I would like to see that cleanup
happening, that wouldn't only help stuff like irq balancing but it
should also help writing portable realtime hooks that needs to deal with
those bits in irq.c. Deferring the need of this further with the excuse
that irq balancing in userspace is better doesn't sound good to me. Sure
I also don't like to see the irq balancing code cut and pasted a dozen
of times in the kernel, but keeping it in userspace just hides the
problem, that the only common API is provided to userspace and not to
the kernel itself.
Maybe you leave sparc64 with its own implementation to avoid one
indirect call per irq? But do you really think those ugly if else are
faster than a indirect call? what when you'll have to support one more
chipset?
if (tlb_type == cheetah || tlb_type == cheetah_plus) {
/* We set it to our Safari AID. */
__asm__ __volatile__("ldxa [%%g0] %1, %0"
: "=r" (tid)
: "i" (ASI_SAFARI_CONFIG));
tid = ((tid & (0x3ffUL<<17)) << 9);
tid &= IMAP_AID_SAFARI;
} else if (this_is_starfire == 0) {
/* We set it to our UPA MID. */
__asm__ __volatile__("ldxa [%%g0] %1, %0"
: "=r" (tid)
: "i" (ASI_UPA_CONFIG));
tid = ((tid & UPA_CONFIG_MID) << 9);
tid &= IMAP_TID_UPA;
} else {
tid = (starfire_translate(imap, smp_processor_id()) << 26);
tid &= IMAP_TID_UPA;
}
Andrea
^ permalink raw reply [flat|nested] 76+ messages in thread
* Re: userspace irq balancer
2003-05-26 22:24 ` Andrea Arcangeli
@ 2003-05-26 23:26 ` Andrew Morton
2003-05-26 23:34 ` Andrea Arcangeli
0 siblings, 1 reply; 76+ messages in thread
From: Andrew Morton @ 2003-05-26 23:26 UTC (permalink / raw)
To: Andrea Arcangeli
Cc: davem, davidsen, haveblue, habanero, mbligh, linux-kernel
Andrea Arcangeli <andrea@suse.de> wrote:
>
> if (IRQ_ALLOWED(phys_id, allowed_mask) && idle_cpu(phys_id))
> return cpu;
How hard would it be to make this HT-aware?
idle_cpu(phys_id) && idle_cpu_siblings(phys_id)
or whatever.
^ permalink raw reply [flat|nested] 76+ messages in thread
* Re: userspace irq balancer
2003-05-26 23:26 ` Andrew Morton
@ 2003-05-26 23:34 ` Andrea Arcangeli
2003-05-26 23:43 ` David S. Miller
2003-05-27 1:17 ` Andrea Arcangeli
0 siblings, 2 replies; 76+ messages in thread
From: Andrea Arcangeli @ 2003-05-26 23:34 UTC (permalink / raw)
To: Andrew Morton; +Cc: davem, davidsen, haveblue, habanero, mbligh, linux-kernel
On Mon, May 26, 2003 at 04:26:16PM -0700, Andrew Morton wrote:
> Andrea Arcangeli <andrea@suse.de> wrote:
> >
> > if (IRQ_ALLOWED(phys_id, allowed_mask) && idle_cpu(phys_id))
> > return cpu;
>
> How hard would it be to make this HT-aware?
>
> idle_cpu(phys_id) && idle_cpu_siblings(phys_id)
>
> or whatever.
yeah! that was the obvious next step. as fast path the additional && is
sure good. Maybe that's enough after all, and we might search only for
fully idle cpus, however I wouldn't dislike to search for a fallback
(partially) logical idle cpu if none physical cpu is (fully) idle.
Andrea
^ permalink raw reply [flat|nested] 76+ messages in thread
* Re: userspace irq balancer
2003-05-26 23:34 ` Andrea Arcangeli
@ 2003-05-26 23:43 ` David S. Miller
[not found] ` <20030527000639.GA3767@dualathlon.random>
2003-05-27 1:17 ` Andrea Arcangeli
1 sibling, 1 reply; 76+ messages in thread
From: David S. Miller @ 2003-05-26 23:43 UTC (permalink / raw)
To: andrea; +Cc: akpm, davidsen, haveblue, habanero, mbligh, linux-kernel
From: Andrea Arcangeli <andrea@suse.de>
Date: Tue, 27 May 2003 01:34:46 +0200
On Mon, May 26, 2003 at 04:26:16PM -0700, Andrew Morton wrote:
> How hard would it be to make this HT-aware?
yeah! that was the obvious next step.
So what is idle defined as? How are you going to measure things like
softirq load? How much weight will softirq load get compared to
hardirq load? How will process load be factored into this and what
weight will this get?
All of these questions have no answer as far as the kernel is
concerned, because this is a policy decision and something the user
ought to be able to configure to suite his needs.
All you've said today is that IRQ balancing needs to be more like the
cpufreq drivers. The hardware programming and some of the delicate
time sensitive details are done in the kernel, but deciding how and
when to do these things belongs as some userspace action.
I still contend that Arjan's usermode irq balancer solves one realm of
those problems. And there is nothing that prevents his work from
being extended to upload policies for the things you have brought up
today.
Finally, claiming this is a performance issue is moot. We've already
shown that if the current IRQ load balancer in 2.5.x improves
performance for any network based things there is no reasonable reason
WHY this is the case bacause it's behavior is anti-networking in
nature in that it thinks hardware IRQ load equates to real load which
it does not.
^ permalink raw reply [flat|nested] 76+ messages in thread
* Re: userspace irq balancer
2003-05-26 23:34 ` Andrea Arcangeli
2003-05-26 23:43 ` David S. Miller
@ 2003-05-27 1:17 ` Andrea Arcangeli
2003-05-27 1:20 ` David S. Miller
1 sibling, 1 reply; 76+ messages in thread
From: Andrea Arcangeli @ 2003-05-27 1:17 UTC (permalink / raw)
To: Andrew Morton; +Cc: davem, davidsen, haveblue, habanero, mbligh, linux-kernel
On Tue, May 27, 2003 at 01:34:46AM +0200, Andrea Arcangeli wrote:
> On Mon, May 26, 2003 at 04:26:16PM -0700, Andrew Morton wrote:
> > Andrea Arcangeli <andrea@suse.de> wrote:
> > >
> > > if (IRQ_ALLOWED(phys_id, allowed_mask) && idle_cpu(phys_id))
> > > return cpu;
> >
> > How hard would it be to make this HT-aware?
> >
> > idle_cpu(phys_id) && idle_cpu_siblings(phys_id)
> >
> > or whatever.
>
> yeah! that was the obvious next step. as fast path the additional && is
> sure good. Maybe that's enough after all, and we might search only for
> fully idle cpus, however I wouldn't dislike to search for a fallback
> (partially) logical idle cpu if none physical cpu is (fully) idle.
I'm going to try this (if it compiles ;). the ksoftirqd check is the one
for the NAPI workload brought to attention by Dave. the idea is that
statistically the softirq load will follow the hardirq load. Both wants
to go into an idle cpu. But we don't want to mistake the softirq load
for unrelated cpu load. So we don't want to separate a ksoftirqd load
from the irq load or we could keep bouncing over and over again.
For HT I take the trivial approch you mentioned above that is to switch
only if the physical cpu is completely idle.
--- ./arch/i386/kernel/io_apic.c.~1~ 2003-05-27 02:45:34.000000000 +0200
+++ ./arch/i386/kernel/io_apic.c 2003-05-27 03:00:32.000000000 +0200
@@ -217,13 +217,18 @@ extern unsigned long irq_affinity [NR_IR
#define IRQ_ALLOWED(cpu,allowed_mask) \
((1UL << cpu) & (allowed_mask))
+#define ksoftirqd_is_running(phys_id) (cpu_curr(phys_id) == ksoftirqd_task(phys_id))
+#define __irq_idle_cpu(phys_id) (idle_cpu(phys_id) || ksoftirqd_is_running(phys_id))
+#define irq_idle_cpu(phys_id) (__irq_idle_cpu(phys_id) &&
+ (smp_num_siblings <= 1 || __irq_idle_cpu(cpu_sibling_map[phys_id])))
+
static unsigned long move(unsigned int curr_cpu, unsigned long allowed_mask, unsigned long now, int direction)
{
unsigned int cpu = curr_cpu;
unsigned int phys_id;
phys_id = cpu_logical_map(cpu);
- if (IRQ_ALLOWED(phys_id, allowed_mask) && idle_cpu(phys_id))
+ if (IRQ_ALLOWED(phys_id, allowed_mask) && irq_idle_cpu(phys_id))
return cpu;
goto inside;
@@ -243,7 +248,7 @@ inside:
}
phys_id = cpu_logical_map(cpu);
- } while (!IRQ_ALLOWED(phys_id, allowed_mask) || !idle_cpu(phys_id));
+ } while (!IRQ_ALLOWED(phys_id, allowed_mask) || !irq_idle_cpu(phys_id));
return cpu;
}
>
> Andrea
Andrea
^ permalink raw reply [flat|nested] 76+ messages in thread
* Re: userspace irq balancer
2003-05-27 1:17 ` Andrea Arcangeli
@ 2003-05-27 1:20 ` David S. Miller
2003-05-27 1:33 ` Andrea Arcangeli
0 siblings, 1 reply; 76+ messages in thread
From: David S. Miller @ 2003-05-27 1:20 UTC (permalink / raw)
To: andrea; +Cc: akpm, davidsen, haveblue, habanero, mbligh, linux-kernel
From: Andrea Arcangeli <andrea@suse.de>
Date: Tue, 27 May 2003 03:17:50 +0200
I'm going to try this (if it compiles ;). the ksoftirqd check is
the one for the NAPI workload brought to attention by Dave.
Ksoftirqd should not be running on a properly functioning system.
In fact, I know lots of people who are simply making ksoftirqd
only run if we do the softirq loop N times where N is very large
in order to avoid the performance problems that result from ksoftirqd.
^ permalink raw reply [flat|nested] 76+ messages in thread
* Re: userspace irq balancer
2003-05-27 1:20 ` David S. Miller
@ 2003-05-27 1:33 ` Andrea Arcangeli
0 siblings, 0 replies; 76+ messages in thread
From: Andrea Arcangeli @ 2003-05-27 1:33 UTC (permalink / raw)
To: David S. Miller; +Cc: akpm, davidsen, haveblue, habanero, mbligh, linux-kernel
On Mon, May 26, 2003 at 06:20:26PM -0700, David S. Miller wrote:
> From: Andrea Arcangeli <andrea@suse.de>
> Date: Tue, 27 May 2003 03:17:50 +0200
>
> I'm going to try this (if it compiles ;). the ksoftirqd check is
> the one for the NAPI workload brought to attention by Dave.
>
> Ksoftirqd should not be running on a properly functioning system.
>
> In fact, I know lots of people who are simply making ksoftirqd
> only run if we do the softirq loop N times where N is very large
> in order to avoid the performance problems that result from ksoftirqd.
note that what those lots of people are doing is very legitimate. I'll
also change this in my current tree right now, so we'll see if these
lots of people will be fine with my next tree. would be interesting
which N those lots of people are using though.
I very well recall that we waited people to ask if the N == 1 was too
low when we introduced ksoftirqd, Linus just said he would increase it
if it was generating spurious reschedules during load peaks.
Note however that the larger N the more it will tend to hang the box
(i.e. less fair) with NAPI. I certainly don't doubt network performance
can increase slightly.
Andrea
^ permalink raw reply [flat|nested] 76+ messages in thread
* Re: userspace irq balancer
2003-05-22 2:04 ` William Lee Irwin III
2003-05-22 2:12 ` Zwane Mwaikambo
2003-05-22 3:57 ` Martin J. Bligh
@ 2003-05-22 14:18 ` James Cleverdon
2003-05-22 14:43 ` William Lee Irwin III
2 siblings, 1 reply; 76+ messages in thread
From: James Cleverdon @ 2003-05-22 14:18 UTC (permalink / raw)
To: William Lee Irwin III, Gerrit Huizenga
Cc: Nakajima, Jun, haveblue, pbadari, linux-kernel, johnstul, mannthey
[-- Attachment #1: Type: text/plain, Size: 1065 bytes --]
On Wednesday 21 May 2003 07:04 pm, William Lee Irwin III wrote:
[ Snip! ]
> ...
> IMHO Linux on Pentium IV should use the TPR in conjunction with _very_
> simplistic interrupt load accounting by default and all more
> sophisticated logic should be punted straight to userspace as an
> administrative API.
>
[ Snip! ]
>
> i.e. frob the fscking TPR as recommended by the APIC docs every once in
> a while by default, punt anything (and everything) fancier up to
> userspace, and get the code that doesn't even understand what the fsck
> DESTMOD means the Hell out of the kernel and the Hell away from my
> IO-APIC RTE's.
>
>
> -- wli
Here's my old very stupid TPR patch . It lacks TPRing soft ints for kernel
preemption, etc. Because the xTPR logic only compares the top nibble of the
TPR and I don't want to mask out IRQs unnecessarily, it only tracks busy/idle
and IRQ/no-IRQ.
Simple enough for you, Bill? 8^)
--
James Cleverdon
IBM xSeries Linux Solutions
{jamesclv(Unix, preferred), cleverdj(Notes)} at us dot ibm dot com
[-- Attachment #2: tpr_dyn-2003-01-10_2.5.55 --]
[-- Type: text/x-diff, Size: 3480 bytes --]
diff -pru j55/arch/i386/kernel/irq.c t55/arch/i386/kernel/irq.c
--- j55/arch/i386/kernel/irq.c Wed Jan 8 20:03:51 2003
+++ t55/arch/i386/kernel/irq.c Fri Jan 10 18:01:03 2003
@@ -329,6 +329,7 @@ asmlinkage unsigned int do_IRQ(struct pt
struct irqaction * action;
unsigned int status;
+ apic_adj_tpr(TPR_IRQ);
irq_enter();
#ifdef CONFIG_DEBUG_STACKOVERFLOW
@@ -406,6 +407,7 @@ out:
spin_unlock(&desc->lock);
irq_exit();
+ apic_adj_tpr(-TPR_IRQ);
return 1;
}
diff -pru j55/arch/i386/kernel/process.c t55/arch/i386/kernel/process.c
--- j55/arch/i386/kernel/process.c Wed Jan 8 20:03:48 2003
+++ t55/arch/i386/kernel/process.c Fri Jan 10 17:59:13 2003
@@ -143,7 +143,9 @@ void cpu_idle (void)
irq_stat[smp_processor_id()].idle_timestamp = jiffies;
while (!need_resched())
idle();
+ apic_set_tpr(TPR_TASK);
schedule();
+ apic_set_tpr(TPR_IDLE);
}
}
diff -pru j55/include/asm-i386/apic.h t55/include/asm-i386/apic.h
--- j55/include/asm-i386/apic.h Wed Jan 8 20:04:27 2003
+++ t55/include/asm-i386/apic.h Fri Jan 10 17:59:13 2003
@@ -64,6 +64,22 @@ static inline void ack_APIC_irq(void)
apic_write_around(APIC_EOI, 0);
}
+static inline void apic_set_tpr(unsigned long val)
+{
+ unsigned long value;
+
+ value = apic_read(APIC_TASKPRI);
+ apic_write_around(APIC_TASKPRI, (value & ~APIC_TPRI_MASK) + val);
+}
+
+static inline void apic_adj_tpr(long adj)
+{
+ unsigned long value;
+
+ value = apic_read(APIC_TASKPRI);
+ apic_write_around(APIC_TASKPRI, value + adj);
+}
+
extern int get_maxlvt(void);
extern void clear_local_APIC(void);
extern void connect_bsp_APIC (void);
@@ -96,6 +112,15 @@ extern unsigned int nmi_watchdog;
#define NMI_LOCAL_APIC 2
#define NMI_INVALID 3
+#else /* CONFIG_X86_LOCAL_APIC */
+#define apic_set_tpr(val)
+#define apic_adj_tpr(adj)
#endif /* CONFIG_X86_LOCAL_APIC */
+/* Priority values for apic_adj_tpr() and apic_set_tpr() */
+/* xAPICs only do priority comparisons on the upper nibble. */
+#define TPR_IDLE (0x00L)
+#define TPR_TASK (0x10L)
+#define TPR_IRQ (0x10L)
+
#endif /* __ASM_APIC_H */
diff -pru j55/include/asm-i386/mach-summit/mach_apic.h t55/include/asm-i386/mach-summit/mach_apic.h
--- j55/include/asm-i386/mach-summit/mach_apic.h Fri Jan 10 16:16:44 2003
+++ t55/include/asm-i386/mach-summit/mach_apic.h Fri Jan 10 19:24:52 2003
@@ -4,7 +4,7 @@
extern int x86_summit;
#define esr_disable (1)
-#define no_balance_irq (0)
+#define no_balance_irq (1)
#define XAPIC_DEST_CPUS_MASK 0x0Fu
#define XAPIC_DEST_CLUSTER_MASK 0xF0u
@@ -13,7 +13,7 @@ extern int x86_summit;
((phys_apic) & XAPIC_DEST_CLUSTER_MASK) )
#define APIC_DFR_VALUE (x86_summit ? APIC_DFR_CLUSTER : APIC_DFR_FLAT)
-#define TARGET_CPUS (x86_summit ? XAPIC_DEST_CPUS_MASK : cpu_online_map)
+#define TARGET_CPUS (x86_summit ? xapic_round_robin_cpu_apic_id() : cpu_online_map)
#define APIC_BROADCAST_ID (x86_summit ? 0xFF : 0x0F)
#define check_apicid_used(bitmap, apicid) (0)
@@ -106,4 +106,20 @@ static inline int check_phys_apicid_pres
return (1);
}
+/*
+ * xapic_round_robin_cpu_apic_id -- Distribute the interrupts using a simple
+ * round robin scheme.
+ */
+static inline int xapic_round_robin_cpu_apic_id(void)
+{
+ int val;
+ static unsigned next_cpu = 0;
+
+ if (next_cpu >= NR_CPUS || cpu_2_logical_apicid[next_cpu] == BAD_APICID)
+ next_cpu = 0;
+ val = cpu_to_logical_apicid(next_cpu) | XAPIC_DEST_CPUS_MASK;
+ ++next_cpu;
+ return (val);
+}
+
#endif /* __ASM_MACH_APIC_H */
^ permalink raw reply [flat|nested] 76+ messages in thread
* Re: userspace irq balancer
2003-05-22 14:18 ` James Cleverdon
@ 2003-05-22 14:43 ` William Lee Irwin III
2003-05-22 15:30 ` James Cleverdon
0 siblings, 1 reply; 76+ messages in thread
From: William Lee Irwin III @ 2003-05-22 14:43 UTC (permalink / raw)
To: James Cleverdon
Cc: Gerrit Huizenga, Nakajima, Jun, haveblue, pbadari, linux-kernel,
johnstul, mannthey
On Thu, May 22, 2003 at 07:18:06AM -0700, James Cleverdon wrote:
> Here's my old very stupid TPR patch . It lacks TPRing soft ints for kernel
> preemption, etc. Because the xTPR logic only compares the top nibble of the
> TPR and I don't want to mask out IRQs unnecessarily, it only tracks busy/idle
> and IRQ/no-IRQ.
> Simple enough for you, Bill? 8^)
Simple enough, yes. But I hesitate to endorse it without making sure
it's not too simple.
It's much closer to the right direction, which is actually following
hardware docs and then punting the fancy (potentially more performant)
bits up into userspace. When properly tuned, it should actually have a
useful interaction with explicit irq balancing via retargeting IO-APIC
RTE destinations as interrupts targeted at a destination specifying
multiple cpus won't always target a single cpu when TPR's are adjusted.
The only real issue with the TPR is that it's an spl-like ranking of
interrupts, assuming a static prioritization based on vector number.
That doesn't really agree with the Linux model and is undesirable in
various scenarios; however, it's how the hardware works and so can't
be avoided (and the disastrous attempt to avoid it didn't DTRT anyway).
-- wli
^ permalink raw reply [flat|nested] 76+ messages in thread
* Re: userspace irq balancer
2003-05-22 14:43 ` William Lee Irwin III
@ 2003-05-22 15:30 ` James Cleverdon
2003-05-22 15:45 ` William Lee Irwin III
0 siblings, 1 reply; 76+ messages in thread
From: James Cleverdon @ 2003-05-22 15:30 UTC (permalink / raw)
To: William Lee Irwin III
Cc: Gerrit Huizenga, Nakajima, Jun, haveblue, pbadari, linux-kernel,
johnstul, mannthey, Andrew Theurer
On Thursday 22 May 2003 07:43 am, William Lee Irwin III wrote:
> On Thu, May 22, 2003 at 07:18:06AM -0700, James Cleverdon wrote:
> > Here's my old very stupid TPR patch . It lacks TPRing soft ints for
> > kernel preemption, etc. Because the xTPR logic only compares the top
> > nibble of the TPR and I don't want to mask out IRQs unnecessarily, it
> > only tracks busy/idle and IRQ/no-IRQ.
> > Simple enough for you, Bill? 8^)
>
> Simple enough, yes. But I hesitate to endorse it without making sure
> it's not too simple.
>
> It's much closer to the right direction, which is actually following
> hardware docs and then punting the fancy (potentially more performant)
> bits up into userspace. When properly tuned, it should actually have a
> useful interaction with explicit irq balancing via retargeting IO-APIC
> RTE destinations as interrupts targeted at a destination specifying
> multiple cpus won't always target a single cpu when TPR's are adjusted.
>
> The only real issue with the TPR is that it's an spl-like ranking of
> interrupts, assuming a static prioritization based on vector number.
> That doesn't really agree with the Linux model and is undesirable in
> various scenarios; however, it's how the hardware works and so can't
> be avoided (and the disastrous attempt to avoid it didn't DTRT anyway).
>
>
> -- wli
Serial APICs have always had a spl-like effect built into them. The effective
TPR value of a given local APIC is:
max(TPR, highest vector currently in progress) & 0xF0
Parallel APICs don't do that because they don't have serial priority
arbitration; instead they use the xTPRs in the bridge chips.
So, I suppose an argument could be made for setting the TPR to the vector
number on entry of do_IRQ. I don't think that would be a good idea. It
could interfere with IRQ nesting during a non-DMA IDE interrupt handler. And
of course, an IRQ's vector has little to do with the IRQ itself, thanks to
the vector hashing scheme used to avoid the (stupid) 2 latches per APIC level
HW limitation of most i586 and i686 CPUs.
--
James Cleverdon
IBM xSeries Linux Solutions
{jamesclv(Unix, preferred), cleverdj(Notes)} at us dot ibm dot com
^ permalink raw reply [flat|nested] 76+ messages in thread
* Re: userspace irq balancer
2003-05-22 15:30 ` James Cleverdon
@ 2003-05-22 15:45 ` William Lee Irwin III
0 siblings, 0 replies; 76+ messages in thread
From: William Lee Irwin III @ 2003-05-22 15:45 UTC (permalink / raw)
To: James Cleverdon
Cc: Gerrit Huizenga, Nakajima, Jun, haveblue, pbadari, linux-kernel,
johnstul, mannthey, Andrew Theurer
On Thu, May 22, 2003 at 08:30:29AM -0700, James Cleverdon wrote:
> Serial APICs have always had a spl-like effect built into them.
> The effective > TPR value of a given local APIC is:
> max(TPR, highest vector currently in progress) & 0xF0
> Parallel APICs don't do that because they don't have serial priority
> arbitration; instead they use the xTPRs in the bridge chips.
> So, I suppose an argument could be made for setting the TPR to the vector
> number on entry of do_IRQ. I don't think that would be a good idea. It
> could interfere with IRQ nesting during a non-DMA IDE interrupt handler. And
> of course, an IRQ's vector has little to do with the IRQ itself, thanks to
> the vector hashing scheme used to avoid the (stupid) 2 latches per APIC level
> HW limitation of most i586 and i686 CPUs.
The code to deal with the 2 latches per APIC level is already
problematic in other ways. I'm not sure how much we can be allowed to
mix the issues. But I wouldn't mind at least hearing about alternative
methods of dealing with that that interact better with the rest of the
APIC mechanics. My interest in particular is vector exhaustion, but as
you point out rearrangements of that can also serve to make the TPR
more meaningful than it is now, and perhaps reduce mutual interference
between devices generating many interrupts and those generating few.
-- wli
^ permalink raw reply [flat|nested] 76+ messages in thread
* RE: userspace irq balancer
@ 2003-05-24 1:10 Nakajima, Jun
0 siblings, 0 replies; 76+ messages in thread
From: Nakajima, Jun @ 2003-05-24 1:10 UTC (permalink / raw)
To: jamesclv
Cc: Gerrit Huizenga, haveblue, pbadari, linux-kernel, johnstul,
mannthey, Andrew Theurer
> So, I suppose an argument could be made for setting the TPR to the vector
> number on entry of do_IRQ. I don't think that would be a good idea.
I agree. If we start spl-like ranking of interrupts, we need to modify disable/enable_irq(), etc. as well, causing possible impacts to device derivers.
One thing that might be helpful here is to have 4-level of priorities, for example:
Idle (0)
User (0x10)
Kernel (0x20)
Interrupt (0x30)
Jun
> -----Original Message-----
> From: James Cleverdon [mailto:jamesclv@us.ibm.com]
> Sent: Thursday, May 22, 2003 8:30 AM
> To: William Lee Irwin III
> Cc: Gerrit Huizenga; Nakajima, Jun; haveblue@us.ibm.com;
> pbadari@us.ibm.com; linux-kernel@vger.kernel.org; johnstul@us.ibm.com;
> mannthey@us.ibm.com; Andrew Theurer
> Subject: Re: userspace irq balancer
>
> On Thursday 22 May 2003 07:43 am, William Lee Irwin III wrote:
> > On Thu, May 22, 2003 at 07:18:06AM -0700, James Cleverdon wrote:
> > > Here's my old very stupid TPR patch . It lacks TPRing soft ints for
> > > kernel preemption, etc. Because the xTPR logic only compares the top
> > > nibble of the TPR and I don't want to mask out IRQs unnecessarily, it
> > > only tracks busy/idle and IRQ/no-IRQ.
> > > Simple enough for you, Bill? 8^)
> >
> > Simple enough, yes. But I hesitate to endorse it without making sure
> > it's not too simple.
> >
> > It's much closer to the right direction, which is actually following
> > hardware docs and then punting the fancy (potentially more performant)
> > bits up into userspace. When properly tuned, it should actually have a
> > useful interaction with explicit irq balancing via retargeting IO-APIC
> > RTE destinations as interrupts targeted at a destination specifying
> > multiple cpus won't always target a single cpu when TPR's are adjusted.
> >
> > The only real issue with the TPR is that it's an spl-like ranking of
> > interrupts, assuming a static prioritization based on vector number.
> > That doesn't really agree with the Linux model and is undesirable in
> > various scenarios; however, it's how the hardware works and so can't
> > be avoided (and the disastrous attempt to avoid it didn't DTRT anyway).
> >
> >
> > -- wli
>
> Serial APICs have always had a spl-like effect built into them. The
> effective
> TPR value of a given local APIC is:
> max(TPR, highest vector currently in progress) & 0xF0
> Parallel APICs don't do that because they don't have serial priority
> arbitration; instead they use the xTPRs in the bridge chips.
>
> So, I suppose an argument could be made for setting the TPR to the vector
> number on entry of do_IRQ. I don't think that would be a good idea. It
> could interfere with IRQ nesting during a non-DMA IDE interrupt handler.
> And
> of course, an IRQ's vector has little to do with the IRQ itself, thanks to
> the vector hashing scheme used to avoid the (stupid) 2 latches per APIC
> level
> HW limitation of most i586 and i686 CPUs.
>
>
> --
> James Cleverdon
> IBM xSeries Linux Solutions
> {jamesclv(Unix, preferred), cleverdj(Notes)} at us dot ibm dot com
^ permalink raw reply [flat|nested] 76+ messages in thread
end of thread, other threads:[~2003-06-13 18:08 UTC | newest]
Thread overview: 76+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <200305191314.06216.pbadari@us.ibm.com>
2003-05-19 22:07 ` userspace irq balancer Dave Hansen
2003-05-19 22:11 ` Arjan van de Ven
2003-05-19 22:22 ` Dave Hansen
2003-05-20 3:25 ` David S. Miller
2003-05-20 3:46 ` William Lee Irwin III
2003-05-20 5:03 ` Dave Hansen
2003-05-20 5:53 ` Martin J. Bligh
2003-05-20 6:13 ` David S. Miller
2003-05-20 6:36 ` Dave Hansen
2003-05-20 6:40 ` David S. Miller
2003-05-20 14:07 ` Andrew Theurer
2003-05-20 14:21 ` Jeff Garzik
2003-05-20 14:35 ` Andrew Theurer
[not found] ` <20030520.163833.104040023.davem@redhat.com>
2003-05-21 14:58 ` Martin J. Bligh
2003-05-21 22:55 ` David S. Miller
2003-05-21 11:00 ` Kai Bankett
2003-05-20 14:01 ` Martin J. Bligh
2003-05-20 9:00 ` Arjan van de Ven
2003-05-20 9:14 ` William Lee Irwin III
2003-05-20 9:17 ` Andrew Morton
[not found] ` <20030520.172230.102567463.davem@redhat.com>
2003-05-21 14:27 ` James Cleverdon
2003-05-20 15:41 Nakajima, Jun
2003-05-21 13:54 ` James Cleverdon
2003-05-21 22:56 ` Zwane Mwaikambo
2003-05-21 16:31 James Bottomley
2003-05-21 20:16 ` Arjan van de Ven
2003-05-21 21:43 Nakajima, Jun
2003-05-22 0:29 ` Gerrit Huizenga
2003-05-22 1:28 ` Martin J. Bligh
2003-05-22 1:44 ` Gerrit Huizenga
2003-05-22 2:03 ` William Lee Irwin III
2003-05-22 2:04 ` William Lee Irwin III
2003-05-22 2:12 ` Zwane Mwaikambo
2003-05-22 3:57 ` Martin J. Bligh
2003-05-22 17:24 ` Bill Davidsen
2003-05-22 22:44 ` David S. Miller
2003-05-26 22:24 ` Andrea Arcangeli
2003-05-26 23:26 ` Andrew Morton
2003-05-26 23:34 ` Andrea Arcangeli
2003-05-26 23:43 ` David S. Miller
[not found] ` <20030527000639.GA3767@dualathlon.random>
2003-05-27 0:15 ` David S. Miller
2003-05-27 0:41 ` Andrea Arcangeli
2003-05-27 0:48 ` David S. Miller
2003-05-27 1:09 ` Andrea Arcangeli
2003-05-27 1:13 ` David S. Miller
2003-05-27 1:26 ` Andrea Arcangeli
2003-05-27 6:11 ` David S. Miller
2003-05-27 11:53 ` Andrea Arcangeli
2003-05-27 22:04 ` David S. Miller
2003-05-27 22:27 ` Andrea Arcangeli
2003-05-27 23:55 ` David S. Miller
2003-06-13 6:22 ` David S. Miller
2003-06-13 18:23 ` Andrea Arcangeli
2003-05-27 1:16 ` Dave Jones
2003-05-27 1:17 ` David S. Miller
2003-05-27 9:07 ` Arjan van de Ven
2003-05-27 9:10 ` David S. Miller
2003-05-27 1:28 ` Andrea Arcangeli
2003-05-27 1:53 ` William Lee Irwin III
2003-05-27 1:59 ` Andrew Morton
2003-05-27 2:10 ` William Lee Irwin III
2003-05-27 2:15 ` Zwane Mwaikambo
2003-05-27 2:44 ` William Lee Irwin III
2003-05-27 2:45 ` Zwane Mwaikambo
2003-05-27 4:22 ` William Lee Irwin III
2003-05-27 2:15 ` Andrea Arcangeli
2003-05-27 2:14 ` Andrea Arcangeli
2003-05-27 2:26 ` William Lee Irwin III
2003-05-27 1:17 ` Andrea Arcangeli
2003-05-27 1:20 ` David S. Miller
2003-05-27 1:33 ` Andrea Arcangeli
2003-05-22 14:18 ` James Cleverdon
2003-05-22 14:43 ` William Lee Irwin III
2003-05-22 15:30 ` James Cleverdon
2003-05-22 15:45 ` William Lee Irwin III
2003-05-24 1:10 Nakajima, Jun
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).