RE: 2.6.1 and irq balancing

RE: 2.6.1 and irq balancing

[Nakajima, Jun]

> Admittedly, the machine's load was not high when I took this sample.
> However, creating a great deal of load does not change these
statistics
> at all.  Being that there are patches available for 2.4.x kernels to
fix
> this, I don't think this at all by design, but what do I know? =)

2.6 kernels don't need a patch to it as far as I understand. Are you
saying that with significant amount of load, you did not see any
distribution of interrupts? Today's threshold in the kernel is high
because we found moving around interrupts frequently rather hurt the
cache and thus lower the performance compared to "do nothing". Can you
try to create significant load with your network (eth0 and eh1) and see
what happens? 

Jun 

> -----Original Message-----
> From: linux-kernel-owner@vger.kernel.org [mailto:linux-kernel-
> owner@vger.kernel.org] On Behalf Of Ethan Weinstein
> Sent: Saturday, January 10, 2004 9:19 PM
> To: Ed Tomlinson
> Cc: linux-kernel@vger.kernel.org; piggin@cyberone.com.au
> Subject: Re: 2.6.1 and irq balancing
> 
> Ed Tomlinson wrote:
> > Hi,
> >
> > What is the load on the box when this is happening?  If its low
think
> > this is optimal (for cache reasons).
> >
> 
> Admittedly, the machine's load was not high when I took this sample.
> However, creating a great deal of load does not change these
statistics
> at all.  Being that there are patches available for 2.4.x kernels to
fix
> this, I don't think this at all by design, but what do I know? =)
> 
> 2.6.0 running on a non-HT SMP machine I have (old Compaq proliant
> 2xPentium2) does interrupt on all CPU's with "noirqbalance" bootparam.
> 
> Regarding the keyboard, I noticed something interesting
> 
> 2.6.1-rc1 shows the i8042 in /proc/interrupts:
> 
>    1:       1871          0          0          0    IO-APIC-edge
i8042
> 
> (keyboard still does not work, though..)
> 
> 2.6.1 final does not show this at all, and [kseriod] eats a constant
5%
>   CPU.  Something's awry =)
> 
> 
> -Ethan
> -
> To unsubscribe from this list: send the line "unsubscribe
linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: 2.6.1 and irq balancing

[Bill Davidsen]

Nakajima, Jun wrote:

> 2.6 kernels don't need a patch to it as far as I understand. Are you
> saying that with significant amount of load, you did not see any
> distribution of interrupts? Today's threshold in the kernel is high
> because we found moving around interrupts frequently rather hurt the
> cache and thus lower the performance compared to "do nothing". Can you
> try to create significant load with your network (eth0 and eh1) and see
> what happens? 

How much is significant? The term doesn't really help much. I will say 
that with one NIC taking 120MB/sec of data to a TB database and copying 
to two other machine (~220MB)  my interrupts got up in in the 5k-12k 
range with essentially CPU0 doing the work, some few percent going to CPU2.

I'm not sure this is a problem in any way, but some serious load is 
needed to trigger sharing, if indeed the NIC was the source of the ints 
on CPU2.

2x Xeon-2.4GHz, HT enabled. "CPU2" from memory, it was the other 
physical CPU, not another sibling. Worked fine, didn't break, don't 
regard it as a problem.

-- 
bill davidsen <davidsen@tmr.com>
   CTO TMR Associates, Inc
   Doing interesting things with small computers since 1979

Re: 2.6.1 and irq balancing

[Zwane Mwaikambo]

On Sun, 11 Jan 2004, Bill Davidsen wrote:

> I'm not sure this is a problem in any way, but some serious load is
> needed to trigger sharing, if indeed the NIC was the source of the ints
> on CPU2.
>
> 2x Xeon-2.4GHz, HT enabled. "CPU2" from memory, it was the other
> physical CPU, not another sibling. Worked fine, didn't break, don't
> regard it as a problem.

Seems to be ok here, 2x Xeon 2.0GHz w/ HT

           CPU0       CPU1       CPU2       CPU3
  0:  322303400          0          0          0    IO-APIC-edge  timer
  1:     255712          0          0          0    IO-APIC-edge  i8042
  2:          0          0          0          0          XT-PIC  cascade
  3:        828          0          0          0    IO-APIC-edge  serial
  4:      70682          0          0          0    IO-APIC-edge  serial
  8:          1          0          0          0    IO-APIC-edge  rtc
  9:          0          0          0          0   IO-APIC-level  acpi
 12:    1244334          0          0          0    IO-APIC-edge  i8042
 14:   21708437        129   18313205       1492    IO-APIC-edge  ide0
 15:   10859481         78    9907876         64    IO-APIC-edge  ide1
 16:        655          0          0          0   IO-APIC-level  uhci_hcd
 19:      34458          0          0          0   IO-APIC-level  uhci_hcd, serial
 20:    5762897          0    1009019          0   IO-APIC-level  eth0
 22:    4985838         60    4908642         42   IO-APIC-level  ide2, ide3
 23:    3667147          0          0          0   IO-APIC-level  eth1
 48:    8459639          0          0          0   IO-APIC-level  EMU10K1
NMI:          0          0          0          0
LOC:  322346360  322346360  322346359  322346358
ERR:          0
MIS:          0

Re: 2.6.1 and irq balancing

[Martin J. Bligh]

> How much is significant? The term doesn't really help much. I will say that with one NIC taking 120MB/sec of data to a TB database and copying to two other machine (~220MB)  my interrupts got up in in the 5k-12k range with essentially CPU0 doing the work, some few percent going to CPU2.


1010 per second, IIRC. Try this patch:

diff -aurpN -X /home/fletch/.diff.exclude 290-gfp_node_strict/arch/i386/kernel/io_apic.c 310-irqbal_fast/arch/i386/kernel/io_apic.c
--- 290-gfp_node_strict/arch/i386/kernel/io_apic.c	Fri Jan  9 22:25:48 2004
+++ 310-irqbal_fast/arch/i386/kernel/io_apic.c	Fri Jan  9 22:27:55 2004
@@ -401,7 +401,7 @@ static void do_irq_balance(void)
 	unsigned long max_cpu_irq = 0, min_cpu_irq = (~0);
 	unsigned long move_this_load = 0;
 	int max_loaded = 0, min_loaded = 0;
-	unsigned long useful_load_threshold = balanced_irq_interval + 10;
+	unsigned long useful_load_threshold = balanced_irq_interval / 10;
 	int selected_irq;
 	int tmp_loaded, first_attempt = 1;
 	unsigned long tmp_cpu_irq;

M.

Re: 2.6.1 and irq balancing

[Ethan Weinstein]

Nakajima, Jun wrote:

>> Admittedly, the machine's load was not high when I took this sample.
>> However, creating a great deal of load does not change these statistics 
>> at all.  Being that there are patches available for 2.4.x kernels to 
>> fix this, I don't think this at all by design, but what do I know? =)
>>  

> 2.6 kernels don't need a patch to it as far as I understand. Are you
> saying that with significant amount of load, you did not see any
> distribution of interrupts? Today's threshold in the kernel is high
> because we found moving around interrupts frequently rather hurt the
> cache and thus lower the performance compared to "do nothing". Can you
> try to create significant load with your network (eth0 and eh1) and see
> what happens? 
> 
> Jun 

Here's the situation two days later, I created some brief periods of 
high load on eth1 and I see we have some change:


            CPU0       CPU1       CPU2       CPU3
   0:  184932542          0    2592511          0    IO-APIC-edge  timer
   1:       1875          0          0          0    IO-APIC-edge  i8042
   2:          0          0          0          0          XT-PIC  cascade
   3:    3046103          0          0          0    IO-APIC-edge  serial
   8:          2          0          0          0    IO-APIC-edge  rtc
   9:          0          0          0          0   IO-APIC-level  acpi
  14:         76          0          0          0    IO-APIC-edge  ide0
  16:    2978264          0          0          0   IO-APIC-level  sym53c8xx
  22:    7838940          0          0          0   IO-APIC-level  eth0
  48:     916078          0     125150          0   IO-APIC-level  aic79xx
  49:    1099375          0          0          0   IO-APIC-level  aic79xx
  54:   51484241        316   50560879        279   IO-APIC-level  eth1
NMI:          0          0          0          0
LOC:  187530735  187530988  187530981  187530986
ERR:          0
MIS:          0


My argument is (see below).  This is an old 2x pentium2 @400, also 
running 2.6, an old Compaq Proliant to be exact.  This machine obviously 
has no HT, so why the balanced load?


            CPU0       CPU1
   0: 1066522197 1117196193    IO-APIC-edge  timer
   1:         42         19    IO-APIC-edge  i8042
   2:          0          0          XT-PIC  cascade
   5:   23523428   23510845   IO-APIC-level  TLAN
   8:          0          4    IO-APIC-edge  rtc
   9:         15         15   IO-APIC-level  sym53c8xx
  10:    6874323    6809042   IO-APIC-level  sym53c8xx
  11:    7545802    7509034   IO-APIC-level  ida0
  14:          8          2    IO-APIC-edge  ide0
NMI:          0          0
LOC: 2183867261 2183867237
ERR:          0
MIS:          0



Ethan

Re: 2.6.1 and irq balancing

[Nick Piggin]

Ethan Weinstein wrote:

> Nakajima, Jun wrote:
>
>>> Admittedly, the machine's load was not high when I took this sample.
>>> However, creating a great deal of load does not change these 
>>> statistics at all.  Being that there are patches available for 2.4.x 
>>> kernels to fix this, I don't think this at all by design, but what 
>>> do I know? =)
>>>  
>>
>
>> 2.6 kernels don't need a patch to it as far as I understand. Are you
>> saying that with significant amount of load, you did not see any
>> distribution of interrupts? Today's threshold in the kernel is high
>> because we found moving around interrupts frequently rather hurt the
>> cache and thus lower the performance compared to "do nothing". Can you
>> try to create significant load with your network (eth0 and eh1) and see
>> what happens?
>> Jun 
>
>
> Here's the situation two days later, I created some brief periods of 
> high load on eth1 and I see we have some change:
>
>
>            CPU0       CPU1       CPU2       CPU3
>   0:  184932542          0    2592511          0    IO-APIC-edge  timer
>   1:       1875          0          0          0    IO-APIC-edge  i8042
>   2:          0          0          0          0          XT-PIC  cascade
>   3:    3046103          0          0          0    IO-APIC-edge  serial
>   8:          2          0          0          0    IO-APIC-edge  rtc
>   9:          0          0          0          0   IO-APIC-level  acpi
>  14:         76          0          0          0    IO-APIC-edge  ide0
>  16:    2978264          0          0          0   IO-APIC-level  
> sym53c8xx
>  22:    7838940          0          0          0   IO-APIC-level  eth0
>  48:     916078          0     125150          0   IO-APIC-level  aic79xx
>  49:    1099375          0          0          0   IO-APIC-level  aic79xx
>  54:   51484241        316   50560879        279   IO-APIC-level  eth1
> NMI:          0          0          0          0
> LOC:  187530735  187530988  187530981  187530986
> ERR:          0
> MIS:          0
>


Aside from the obvious imbalance between physical CPUs:
I think interrupts should be much more freely balanced between siblings
that share cache, otherwise process a running on CPU0 gets less time than
process b running on CPU1 because of the interrupt load.


>
> My argument is (see below).  This is an old 2x pentium2 @400, also 
> running 2.6, an old Compaq Proliant to be exact.  This machine 
> obviously has no HT, so why the balanced load?


IIRC the P2/3 APICs are set to a round robin delivery mode while the P4
ones are not. It is still not ideal though, while you have fairness, you now
have suboptimal performance.

Re: 2.6.1 and irq balancing

[Pavel Machek]

Hi!

> > > Greetings all,
> > > 
> > > I upgraded my server to 2.6.1, and I'm finding I'm saddled with only 
> > > interrupting on CPU0 again. 2.6.0 does this as well. This is the 
> > > Supermicro X5DPL-iGM-O (E7501 chipset), 2 Xeons@2.4ghz HT enabled. 
> > > /proc/cpuinfo is normal as per HT, displaying 4 cpus.
> > 
> > you should run the userspace irq balance daemon:
> > http://people.redhat.com/arjanv/irqbalance/
> 
> I have long wondered what is so evil about most interrupts going to
> CPU 0 that we felt we had to have a pair of irqdaemons in 2.6.  From my
> (admittedly imperfect) experience, the APIC will route an interrupt to
> CPU 1 if CPU 0 is busy with another interrupt, to CPU 2 if 0 and 1 are
> so occupied, and so on.  I see no harm in this other than the strangely
> lopsided /proc/interrupt displays, which I can live with.

Well, imagine 8 CPU machine with high interrupt load. Poor process
that gets scheduled on CPU#0 does little progress, but is shown as
eating one whole CPU.
								Pavel
-- 
When do you have a heart between your knees?
[Johanka's followup: and *two* hearts?]

RE: 2.6.1 and irq balancing

[Nakajima, Jun]

> Aside from the obvious imbalance between physical CPUs:
> I think interrupts should be much more freely balanced between
siblings
> that share cache, otherwise process a running on CPU0 gets less time
than
> process b running on CPU1 because of the interrupt load.
>
That scheduling issue is true. Today we balance interrupt load on a
package (i.e. physical CPU) basis, and we don't care which logical
processors do the interrupt handling because it should not matter in
terms of performance. 

Jun

> -----Original Message-----
> From: Nick Piggin [mailto:piggin@cyberone.com.au]
> Sent: Monday, January 12, 2004 11:05 PM
> To: Ethan Weinstein
> Cc: Nakajima, Jun; Ed Tomlinson; linux-kernel@vger.kernel.org; Kamble,
> Nitin A
> Subject: Re: 2.6.1 and irq balancing
> 
> 
> 
> Ethan Weinstein wrote:
> 
> > Nakajima, Jun wrote:
> >
> >>> Admittedly, the machine's load was not high when I took this
sample.
> >>> However, creating a great deal of load does not change these
> >>> statistics at all.  Being that there are patches available for
2.4.x
> >>> kernels to fix this, I don't think this at all by design, but what
> >>> do I know? =)
> >>>
> >>
> >
> >> 2.6 kernels don't need a patch to it as far as I understand. Are
you
> >> saying that with significant amount of load, you did not see any
> >> distribution of interrupts? Today's threshold in the kernel is high
> >> because we found moving around interrupts frequently rather hurt
the
> >> cache and thus lower the performance compared to "do nothing". Can
you
> >> try to create significant load with your network (eth0 and eh1) and
see
> >> what happens?
> >> Jun
> >
> >
> > Here's the situation two days later, I created some brief periods of
> > high load on eth1 and I see we have some change:
> >
> >
> >            CPU0       CPU1       CPU2       CPU3
> >   0:  184932542          0    2592511          0    IO-APIC-edge
timer
> >   1:       1875          0          0          0    IO-APIC-edge
i8042
> >   2:          0          0          0          0          XT-PIC
> cascade
> >   3:    3046103          0          0          0    IO-APIC-edge
serial
> >   8:          2          0          0          0    IO-APIC-edge
rtc
> >   9:          0          0          0          0   IO-APIC-level
acpi
> >  14:         76          0          0          0    IO-APIC-edge
ide0
> >  16:    2978264          0          0          0   IO-APIC-level
> > sym53c8xx
> >  22:    7838940          0          0          0   IO-APIC-level
eth0
> >  48:     916078          0     125150          0   IO-APIC-level
> aic79xx
> >  49:    1099375          0          0          0   IO-APIC-level
> aic79xx
> >  54:   51484241        316   50560879        279   IO-APIC-level
eth1
> > NMI:          0          0          0          0
> > LOC:  187530735  187530988  187530981  187530986
> > ERR:          0
> > MIS:          0
> >
> 
> 
> Aside from the obvious imbalance between physical CPUs:
> I think interrupts should be much more freely balanced between
siblings
> that share cache, otherwise process a running on CPU0 gets less time
than
> process b running on CPU1 because of the interrupt load.
> 
> 
> >
> > My argument is (see below).  This is an old 2x pentium2 @400, also
> > running 2.6, an old Compaq Proliant to be exact.  This machine
> > obviously has no HT, so why the balanced load?
> 
> 
> IIRC the P2/3 APICs are set to a round robin delivery mode while the
P4
> ones are not. It is still not ideal though, while you have fairness,
you
> now
> have suboptimal performance.
> 

RE: 2.6.1 and irq balancing

[Nakajima, Jun]

I don't see a major problem there. If you look at eth1, it has the
biggest load with order of difference. Interrupts from disk controllers
cannot be major unless you have a lot of disks (tens, hundreds) given a
time period.

>   16:    2978264          0          0          0   IO-APIC-level
> sym53c8xx
>   22:    7838940          0          0          0   IO-APIC-level
eth0
>   48:     916078          0     125150          0   IO-APIC-level
aic79xx
>   49:    1099375          0          0          0   IO-APIC-level
aic79xx
>   54:   51484241        316   50560879        279   IO-APIC-level
eth1

Timer is a bit tricky because it continuously generates interrupts 1000
per sec, and it looks higher as the results. But given a time period,
the timer interrupts should be nothing. 

>    0:  184932542          0    2592511          0    IO-APIC-edge
timer


>             CPU0       CPU1
>    0: 1066522197 1117196193    IO-APIC-edge  timer
>    1:         42         19    IO-APIC-edge  i8042
>    2:          0          0          XT-PIC  cascade
>    5:   23523428   23510845   IO-APIC-level  TLAN
>    8:          0          4    IO-APIC-edge  rtc
>    9:         15         15   IO-APIC-level  sym53c8xx
>   10:    6874323    6809042   IO-APIC-level  sym53c8xx
>   11:    7545802    7509034   IO-APIC-level  ida0
>   14:          8          2    IO-APIC-edge  ide0
> NMI:          0          0
> LOC: 2183867261 2183867237
> ERR:          0
> MIS:          0

The above is generated by the round-robin interrupt distribution
provided by the chipset. It's very easy to do this on P4P systems
(that's basically the initial Ingo's patch), but the performance tends
to be worse if you do that. In fact, we saw better results on PIII
systems if we used the one in the kernel, i.e. irqbalance, rather than
the one by the chipset (round-robin).

Jun

> -----Original Message-----
> From: Ethan Weinstein [mailto:lists@stinkfoot.org]
> Sent: Monday, January 12, 2004 10:50 PM
> To: Nakajima, Jun
> Cc: Ed Tomlinson; linux-kernel@vger.kernel.org;
piggin@cyberone.com.au;
> Kamble, Nitin A
> Subject: Re: 2.6.1 and irq balancing
> 
> Nakajima, Jun wrote:
> 
> >> Admittedly, the machine's load was not high when I took this
sample.
> >> However, creating a great deal of load does not change these
statistics
> >> at all.  Being that there are patches available for 2.4.x kernels
to
> >> fix this, I don't think this at all by design, but what do I know?
=)
> >>
> 
> > 2.6 kernels don't need a patch to it as far as I understand. Are you
> > saying that with significant amount of load, you did not see any
> > distribution of interrupts? Today's threshold in the kernel is high
> > because we found moving around interrupts frequently rather hurt the
> > cache and thus lower the performance compared to "do nothing". Can
you
> > try to create significant load with your network (eth0 and eh1) and
see
> > what happens?
> >
> > Jun
> 
> Here's the situation two days later, I created some brief periods of
> high load on eth1 and I see we have some change:
> 
> 
>             CPU0       CPU1       CPU2       CPU3
>    0:  184932542          0    2592511          0    IO-APIC-edge
timer
>    1:       1875          0          0          0    IO-APIC-edge
i8042
>    2:          0          0          0          0          XT-PIC
cascade
>    3:    3046103          0          0          0    IO-APIC-edge
serial
>    8:          2          0          0          0    IO-APIC-edge  rtc
>    9:          0          0          0          0   IO-APIC-level
acpi
>   14:         76          0          0          0    IO-APIC-edge
ide0
>   16:    2978264          0          0          0   IO-APIC-level
> sym53c8xx
>   22:    7838940          0          0          0   IO-APIC-level
eth0
>   48:     916078          0     125150          0   IO-APIC-level
aic79xx
>   49:    1099375          0          0          0   IO-APIC-level
aic79xx
>   54:   51484241        316   50560879        279   IO-APIC-level
eth1
> NMI:          0          0          0          0
> LOC:  187530735  187530988  187530981  187530986
> ERR:          0
> MIS:          0
> 
> 
> My argument is (see below).  This is an old 2x pentium2 @400, also
> running 2.6, an old Compaq Proliant to be exact.  This machine
obviously
> has no HT, so why the balanced load?
> 
> 
>             CPU0       CPU1
>    0: 1066522197 1117196193    IO-APIC-edge  timer
>    1:         42         19    IO-APIC-edge  i8042
>    2:          0          0          XT-PIC  cascade
>    5:   23523428   23510845   IO-APIC-level  TLAN
>    8:          0          4    IO-APIC-edge  rtc
>    9:         15         15   IO-APIC-level  sym53c8xx
>   10:    6874323    6809042   IO-APIC-level  sym53c8xx
>   11:    7545802    7509034   IO-APIC-level  ida0
>   14:          8          2    IO-APIC-edge  ide0
> NMI:          0          0
> LOC: 2183867261 2183867237
> ERR:          0
> MIS:          0
> 
> 
> 
> Ethan

Re: 2.6.1 and irq balancing

[Arjan van de Ven]

[-- Attachment #1: Type: text/plain, Size: 1108 bytes --]

On Sun, Jan 11, 2004 at 11:50:12AM -0500, Joe Korty wrote:
> On Sun, Jan 11, 2004 at 10:51:22AM +0100, Arjan van de Ven wrote:
> > On Sun, 2004-01-11 at 00:14, Ethan Weinstein wrote:
> > > Greetings all,
> > > 
> > > I upgraded my server to 2.6.1, and I'm finding I'm saddled with only 
> > > interrupting on CPU0 again. 2.6.0 does this as well. This is the 
> > > Supermicro X5DPL-iGM-O (E7501 chipset), 2 Xeons@2.4ghz HT enabled. 
> > > /proc/cpuinfo is normal as per HT, displaying 4 cpus.
> > 
> > you should run the userspace irq balance daemon:
> > http://people.redhat.com/arjanv/irqbalance/
> 
> I have long wondered what is so evil about most interrupts going to
> CPU 0 that we felt we had to have a pair of irqdaemons in 2.6.

well irqbalanced is a userspace balancer

> Earlier APICs had a variation where the search for where each new
> interrupt was to go started with first cpu after the one that got the
> last interrupt.  If we call this 'round-robin' allocation, then today's
> technique could be described as 'first fit'.

if it's really busy it starves cpu0 .... 

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: 2.6.1 and irq balancing

[Joe Korty]

On Sun, Jan 11, 2004 at 10:51:22AM +0100, Arjan van de Ven wrote:
> On Sun, 2004-01-11 at 00:14, Ethan Weinstein wrote:
> > Greetings all,
> > 
> > I upgraded my server to 2.6.1, and I'm finding I'm saddled with only 
> > interrupting on CPU0 again. 2.6.0 does this as well. This is the 
> > Supermicro X5DPL-iGM-O (E7501 chipset), 2 Xeons@2.4ghz HT enabled. 
> > /proc/cpuinfo is normal as per HT, displaying 4 cpus.
> 
> you should run the userspace irq balance daemon:
> http://people.redhat.com/arjanv/irqbalance/

I have long wondered what is so evil about most interrupts going to
CPU 0 that we felt we had to have a pair of irqdaemons in 2.6.  From my
(admittedly imperfect) experience, the APIC will route an interrupt to
CPU 1 if CPU 0 is busy with another interrupt, to CPU 2 if 0 and 1 are
so occupied, and so on.  I see no harm in this other than the strangely
lopsided /proc/interrupt displays, which I can live with.

Earlier APICs had a variation where the search for where each new
interrupt was to go started with first cpu after the one that got the
last interrupt.  If we call this 'round-robin' allocation, then today's
technique could be described as 'first fit'.

If I am wrong about this, I would dearly love to be corrected :)
Joe

Re: 2.6.1 and irq balancing

[Martin Schlemmer]

[-- Attachment #1: Type: text/plain, Size: 1734 bytes --]

On Sun, 2004-01-11 at 01:14, Ethan Weinstein wrote:
> Greetings all,
> 
> I upgraded my server to 2.6.1, and I'm finding I'm saddled with only 
> interrupting on CPU0 again. 2.6.0 does this as well. This is the 
> Supermicro X5DPL-iGM-O (E7501 chipset), 2 Xeons@2.4ghz HT enabled. 
> /proc/cpuinfo is normal as per HT, displaying 4 cpus.
> 2.4.2(3|4) exhibited this behaviour as well, until I applied patches 
> from here: 
> http://www.hardrock.org/kernel/2.4.23/irqbalance-2.4.23-jb.patch, et al.
> 
> 
>             CPU0       CPU1       CPU2       CPU3
>    0:    1572323          0          0          0    IO-APIC-edge  timer
>    2:          0          0          0          0          XT-PIC  cascade
>    3:      23520          0          0          0    IO-APIC-edge  serial
>    8:          2          0          0          0    IO-APIC-edge  rtc
>    9:          0          0          0          0   IO-APIC-level  acpi
>   14:         10          0          0          0    IO-APIC-edge  ide0
>   16:         30          0          0          0   IO-APIC-level  sym53c8xx
>   22:       4162          0          0          0   IO-APIC-level  eth0
>   48:       7798          0          0          0   IO-APIC-level  aic79xx
>   49:       3385          0          0          0   IO-APIC-level  aic79xx
>   54:      17062          0          0          0   IO-APIC-level  eth1
> NMI:          0          0          0          0
> LOC:    1572002    1572251    1572250    1572243
> ERR:          0
> MIS:          0
> 
> 
> THey keyboard isn't working either, but we see the i8042..
> 
> serio: i8042 KBD port at 0x60,0x64 irq 1
> 

Are you running irqbalance ?


-- 
Martin Schlemmer

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: 2.6.1 and irq balancing

[Arjan van de Ven]

[-- Attachment #1: Type: text/plain, Size: 780 bytes --]

On Sun, 2004-01-11 at 04:38, Nick Piggin wrote:
> Ed Tomlinson wrote:
> 
> >Hi,
> >
> >What is the load on the box when this is happening?  If its low think
> >this is optimal (for cache reasons).
> >  
> >
> 
> I'd rather see different interrupt sources run on different CPUs
> initially, which would help fairness a little bit, and should be
> more optimal with big interrupt loads.
> 
> 
> 0:      xxx1     0      0      0
> 1:      0     xxx2      0      0
> 2:      0        0   xxx3      0
> 3:      0        0      0   xxx4
> 
> This would delay the need for interrupt balancing in the case where
> 2 or more interrupts are heavily used.

this is what irqbalanced will do (but more inteligent than just using
the irq number as round robin seed).


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: 2.6.1 and irq balancing

[Arjan van de Ven]

[-- Attachment #1: Type: text/plain, Size: 571 bytes --]

On Sun, 2004-01-11 at 00:14, Ethan Weinstein wrote:
> Greetings all,
> 
> I upgraded my server to 2.6.1, and I'm finding I'm saddled with only 
> interrupting on CPU0 again. 2.6.0 does this as well. This is the 
> Supermicro X5DPL-iGM-O (E7501 chipset), 2 Xeons@2.4ghz HT enabled. 
> /proc/cpuinfo is normal as per HT, displaying 4 cpus.

you should run the userspace irq balance daemon:
http://people.redhat.com/arjanv/irqbalance/

(or part of the kernel-utils package if you run a RH based distro; afaik
SuSE ships it too but I don't know in what package)

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: 2.6.1 and irq balancing

[Ethan Weinstein]

Ed Tomlinson wrote:
> Hi,
> 
> What is the load on the box when this is happening?  If its low think
> this is optimal (for cache reasons).
> 

Admittedly, the machine's load was not high when I took this sample. 
However, creating a great deal of load does not change these statistics 
at all.  Being that there are patches available for 2.4.x kernels to fix 
this, I don't think this at all by design, but what do I know? =)

2.6.0 running on a non-HT SMP machine I have (old Compaq proliant 
2xPentium2) does interrupt on all CPU's with "noirqbalance" bootparam.

Regarding the keyboard, I noticed something interesting

2.6.1-rc1 shows the i8042 in /proc/interrupts:

   1:       1871          0          0          0    IO-APIC-edge  i8042

(keyboard still does not work, though..)

2.6.1 final does not show this at all, and [kseriod] eats a constant 5% 
  CPU.  Something's awry =)


-Ethan

Re: 2.6.1 and irq balancing

[Nick Piggin]

Ed Tomlinson wrote:

>Hi,
>
>What is the load on the box when this is happening?  If its low think
>this is optimal (for cache reasons).
>  
>

I'd rather see different interrupt sources run on different CPUs
initially, which would help fairness a little bit, and should be
more optimal with big interrupt loads.


0:      xxx1     0      0      0
1:      0     xxx2      0      0
2:      0        0   xxx3      0
3:      0        0      0   xxx4

This would delay the need for interrupt balancing in the case where
2 or more interrupts are heavily used.

Re: 2.6.1 and irq balancing

[Ed Tomlinson]

Hi,

What is the load on the box when this is happening?  If its low think
this is optimal (for cache reasons).

Ed Tomlinson

On January 10, 2004 06:14 pm, Ethan Weinstein wrote:
> Greetings all,
> 
> I upgraded my server to 2.6.1, and I'm finding I'm saddled with only 
> interrupting on CPU0 again. 2.6.0 does this as well. This is the 
> Supermicro X5DPL-iGM-O (E7501 chipset), 2 Xeons@2.4ghz HT enabled. 
> /proc/cpuinfo is normal as per HT, displaying 4 cpus.
> 2.4.2(3|4) exhibited this behaviour as well, until I applied patches 
> from here: 
> http://www.hardrock.org/kernel/2.4.23/irqbalance-2.4.23-jb.patch, et al.
> 
> 
>             CPU0       CPU1       CPU2       CPU3
>    0:    1572323          0          0          0    IO-APIC-edge  timer
>    2:          0          0          0          0          XT-PIC  cascade
>    3:      23520          0          0          0    IO-APIC-edge  serial
>    8:          2          0          0          0    IO-APIC-edge  rtc
>    9:          0          0          0          0   IO-APIC-level  acpi
>   14:         10          0          0          0    IO-APIC-edge  ide0
>   16:         30          0          0          0   IO-APIC-level 
> sym53c8xx\r 22:       4162          0          0          0   IO-APIC-level 
> eth0 48:       7798          0          0          0   IO-APIC-level 
> aic79xx 49:       3385          0          0          0   IO-APIC-level 
> aic79xx 54:      17062          0          0          0   IO-APIC-level 
> eth1 NMI:          0          0          0          0
> LOC:    1572002    1572251    1572250    1572243
> ERR:          0
> MIS:          0
> 
> 
> THey keyboard isn't working either, but we see the i8042..
> 
> serio: i8042 KBD port at 0x60,0x64 irq 1
> 
> 
> -Ethan
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

2.6.1 and irq balancing

[Ethan Weinstein]

Greetings all,

I upgraded my server to 2.6.1, and I'm finding I'm saddled with only 
interrupting on CPU0 again. 2.6.0 does this as well. This is the 
Supermicro X5DPL-iGM-O (E7501 chipset), 2 Xeons@2.4ghz HT enabled. 
/proc/cpuinfo is normal as per HT, displaying 4 cpus.
2.4.2(3|4) exhibited this behaviour as well, until I applied patches 
from here: 
http://www.hardrock.org/kernel/2.4.23/irqbalance-2.4.23-jb.patch, et al.


            CPU0       CPU1       CPU2       CPU3
   0:    1572323          0          0          0    IO-APIC-edge  timer
   2:          0          0          0          0          XT-PIC  cascade
   3:      23520          0          0          0    IO-APIC-edge  serial
   8:          2          0          0          0    IO-APIC-edge  rtc
   9:          0          0          0          0   IO-APIC-level  acpi
  14:         10          0          0          0    IO-APIC-edge  ide0
  16:         30          0          0          0   IO-APIC-level  sym53c8xx
  22:       4162          0          0          0   IO-APIC-level  eth0
  48:       7798          0          0          0   IO-APIC-level  aic79xx
  49:       3385          0          0          0   IO-APIC-level  aic79xx
  54:      17062          0          0          0   IO-APIC-level  eth1
NMI:          0          0          0          0
LOC:    1572002    1572251    1572250    1572243
ERR:          0
MIS:          0


THey keyboard isn't working either, but we see the i8042..

serio: i8042 KBD port at 0x60,0x64 irq 1


-Ethan