Re: CONFIG_IRQBALANCE for 64-bit x86 ?

[Nick Piggin <nickpiggin@yahoo.com.au>]

On Tuesday 20 November 2007 16:37, Arjan van de Ven wrote:
> On Tue, 20 Nov 2007 15:17:15 +1100

> > For that matter, I'd like to know why it has been decided that the
> > best place for IRQ balancing is in userspace. It should be in kernel
> > IMO, and it would probably allow better power saving, performance,
> > fairness, etc. if it were to be integrated with the task balancer as
> > well.
>
> actually.... no. IRQ balancing is not a "fast" decision; every time you

I didn't say anything of the sort. But IRQ load could still fluctuate
a lot more rapidly than we'd like to wake up the irqbalancer.


> move an interrupt around, you end up causing a really a TON of cache
> line bounces, and generally really bad performance

All the more reason why the kernel should do it. When I say move it to
the kernel, I don't mean because I want to move IRQs 1 000 000 times
per second and can't sustain enough context switches to do it in
userspace. Userspace basically has insufficient information to do it
as well as kernel.

We do task balancing in the kernel too, it's a pretty similar problem
(although granted it is less feasible for userspace because tasks are
created and destroyed very often)


> (esp if you do it 
> for networking ones, since you destroy the packet reassembly stuff in
> the tcp/ip stack).
>
> Instead, what ends up working is if you do high level categories of
> interrupt classes and balance within those (so that no 2 networking
> irqs are on the same core/package unless you have more nics than cores)

Sure, but you say that like it is difficult information for the kernel
to know about. Actually it is much easier. Note that you can still
bind interrupts to specific CPUs.


> etc. Balancing on a 10 second scale seems to work quite well; no need
> to pull that complexity into the kernel....

My perspective is that it isn't a good idea to have such a critical
piece of infrastructure outside the kernel.

I want the kernel to balance interrupts and tasks fairly; maybe move
interrupts closer to the tasks they are interacting with (instead of,
or combined with our current policy of moving tasks near the interrupts,
which can be much more damaging for cache and NUMA); move all interrupts
to a single core when there is enough capacity and we are balancing for
power savings; do exponential interrupt balancing backoff when it isn't
required; etc. Not easy to do all that in userspace.

Any reason you actually think it is a good idea, aside from the fact
that a userspace solution was able to be better than a crappy old
kernel one?