From: Mark Lord <lkml@rtr.ca>
To: Arjan van de Ven <arjan@infradead.org>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>,
Andrew Morton <akpm@linux-foundation.org>,
Linus Torvalds <torvalds@osdl.org>, Ingo Molnar <mingo@elte.hu>,
Linux Kernel <linux-kernel@vger.kernel.org>
Subject: Re: CONFIG_IRQBALANCE for 64-bit x86 ?
Date: Tue, 20 Nov 2007 15:02:43 -0500 [thread overview]
Message-ID: <47433D63.5080806@rtr.ca> (raw)
In-Reply-To: <20071120110726.4cc2995e@laptopd505.fenrus.org>
Arjan van de Ven wrote:
> On Wed, 21 Nov 2007 02:43:46 +1100
> Nick Piggin <nickpiggin@yahoo.com.au> wrote:
>
>> On Wednesday 21 November 2007 01:47, Arjan van de Ven wrote:
>>> On Tue, 20 Nov 2007 18:37:39 +1100
>>>
>>> Nick Piggin <nickpiggin@yahoo.com.au> wrote:
>>>>> actually.... no. IRQ balancing is not a "fast" decision; every
>>>>> time you
>>>> I didn't say anything of the sort. But IRQ load could still
>>>> fluctuate a lot more rapidly than we'd like to wake up the
>>>> irqbalancer.
>>> irq load fluctuates by definition. but acting on it faster isn't the
>>> right thing.
>> Of course it is, if you want to effectively use your resources.
>> Imagine if the task balancer only polled once every 10s.
>
> but unlike the task balancer, moving an irq is really expensive.
> (at least for networking and a few other similar systems)
> ANd no it's not just the cache bouncing, it's the entire reassembly of
> multiple packets etc etc that gets really messy.
>
>
>>> I assume you've read what/how irqbalance does; good luck convincing
>>> people that that kind of policy belongs in the kernel.
>> Lots of code to get topology and device information.
>
> yes this would go away in the kernel
>
>> Some constants
>> that make assumptions about the machine it is running on and may or
>> may not agree with what the task scheduler is trying to do.
>
>> Some
>> classification stuff which makes guesses about how a particular bit of
>
> you misunderstood this; the classification stuff is there to spread
> different irqs of similar class (say networking) over multiple
> cores/packages. Doing this is a system resource balancing proposition
> not just a cpu time one.
>
> You may think this spreading based on classification is a mistake, but
> it's based on the following observation:
> 1) servers with multiple network cards serving internet traffic out
> really need to load balance their loads; this is for various per-cpu
> resource reasons (such as per cpu memory pools) to be evenly used. It
> also makes sure that under network spikes on both interfaces, the
> response is sane
> 2) servers with multiple IO devices need this to be spread out, just
> think of oracle etc.
>
> for both you could argue "but we could balance this based on actual
> observed load in some way", but you can only do that if you rebalance
> at a relatively high frequency, which you really don't want to do for
> networking and probably even storage.
>
> We used to rebalance this frequently in the 2.4-early kernels based on
> a patch from Ingo. Turned out to be a really really bad idea;
> performance really tanked.
>
>> hardware or device driver wants to be balanced. Hacks to poll
>> hotplugging and topology changes.
>
> "hacks" as in "rescan".. so falls under the topology code and would
> indeed be changed to hook into hotplug inside the kernel; just
> different complexity.
>
>> I'm still convinced. Who isn't?
>
> I know you can do SOME sort of balancing in the kernel. But please
> describe the algorithm you would use; I started out with the same
> thought but when it got down to the algorithm to me at least it became
> clear "we really don't want this complexity in kernel mode".
..
Well, for my dualCore notebook, dualCore MythTV box, and QuadCore desktop,
the behaviour of the existing, working, 32-bit kernel IRQBALANCE code
outperforms the userspace utility.
Mostly, I suspect, due to it's much faster response to changing conditions.
That's something the external one could try to match, but at present it
seems tuned specifically for high-traffic network servers, not for the
average notebook or desktop.
Cheers
next prev parent reply other threads:[~2007-11-20 20:02 UTC|newest]
Thread overview: 38+ messages / expand[flat|nested] mbox.gz Atom feed top
2007-11-20 4:12 CONFIG_IRQBALANCE for 64-bit x86 ? Mark Lord
2007-11-20 4:15 ` Ismail Dönmez
2007-11-20 4:17 ` Nick Piggin
2007-11-20 4:29 ` Willy Tarreau
2007-11-20 4:37 ` Adrian Bunk
2007-11-20 5:24 ` Nick Piggin
2007-11-20 5:28 ` H. Peter Anvin
2007-11-20 5:37 ` Arjan van de Ven
2007-11-20 7:37 ` Nick Piggin
2007-11-20 14:47 ` Arjan van de Ven
2007-11-20 15:43 ` Nick Piggin
2007-11-20 19:07 ` Arjan van de Ven
2007-11-20 20:02 ` Mark Lord [this message]
2007-11-20 21:58 ` Arjan van de Ven
2007-11-20 23:17 ` Mark Lord
2007-11-22 7:54 ` Nick Piggin
2007-11-23 13:09 ` Ingo Molnar
2007-11-25 10:03 ` Nick Piggin
2007-11-20 15:47 ` Mark Lord
2007-11-20 15:52 ` Mark Lord
2007-11-20 16:02 ` Arjan van de Ven
2007-11-20 16:10 ` Mark Lord
2007-11-20 18:42 ` Mark Lord
2007-11-20 22:01 ` Ingo Molnar
2007-11-20 23:22 ` Mark Lord
2007-11-20 23:27 ` Ingo Molnar
2007-11-20 23:33 ` H. Peter Anvin
2007-11-20 23:47 ` Ingo Molnar
2007-11-20 23:50 ` H. Peter Anvin
2007-11-21 0:07 ` Ingo Molnar
2007-11-21 0:20 ` H. Peter Anvin
2007-11-21 0:36 ` Ingo Molnar
2007-11-21 0:47 ` H. Peter Anvin
2007-11-21 2:48 ` Jeff Garzik
2007-11-21 2:59 ` H. Peter Anvin
2007-11-20 23:28 ` H. Peter Anvin
2007-11-20 19:17 ` Andi Kleen
2007-11-21 2:22 Walt H
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=47433D63.5080806@rtr.ca \
--to=lkml@rtr.ca \
--cc=akpm@linux-foundation.org \
--cc=arjan@infradead.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@elte.hu \
--cc=nickpiggin@yahoo.com.au \
--cc=torvalds@osdl.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).