irq balancing question

irq balancing question

[Imre Gergely]

hi

i have a SMP with 2x duo-core Opteron processors, and i have an ethernet card
(eth0). i want to balance the interrupts generated by the card to all 4
processors. i can do this by writing to smp_affinity, but on another machine it
get's done by "default", without doing anything.

is this irq balancing a feature in the chipset, or driver, or kernel ? i got
the idea of the smp_affinity stuff, and i've set it to "f", but nothing
happens, i mean i see only one CPU getting the interrupts.

i'm using kernel 2.6.18, but tried with 2.6.19, too.

thanks.

Re: irq balancing question

[Arjan van de Ven]

On Fri, 2007-02-23 at 13:32 +0200, Imre Gergely wrote:
> hi
> 
> i have a SMP with 2x duo-core Opteron processors, and i have an ethernet card
> (eth0). i want to balance the interrupts generated by the card to all 4
> processors. 

you have only one card? then balancing like you say will only really
hurt you; neither linux nor the chipset will do this.

> is this irq balancing a feature in the chipset, or driver, or kernel ? i got
> the idea of the smp_affinity stuff, and i've set it to "f", but nothing
> happens, i mean i see only one CPU getting the interrupts.

if you have multiple real interrupt sources you can balance between them
using irqbalance, see http://www.irqbalance.org
But note that this program will NOT move one irq across 4 cpus all the
time, since that gives *really* bad performance.

-- 
if you want to mail me at work (you don't), use arjan (at) linux.intel.com
Test the interaction between Linux and your BIOS via http://www.linuxfirmwarekit.org

Re: irq balancing question

[Gergely Imre]

Arjan van de Ven wrote:
> On Fri, 2007-02-23 at 13:32 +0200, Imre Gergely wrote:
>> hi
>>
>> i have a SMP with 2x duo-core Opteron processors, and i have an ethernet card
>> (eth0). i want to balance the interrupts generated by the card to all 4
>> processors. 
> 
> you have only one card? then balancing like you say will only really
> hurt you; neither linux nor the chipset will do this.

in fact i have two cards, and 4 CPUs, but i was interested in then answer
Robert gave, that only _some_ machines distribute interrupts in hardware.
software distribution is obviously not good. consider this scenario:

you have one machine with 4 cpus, and two ethernet cards with a lot of
traffic on them. if you bind every card to one cpu, two of them are not used,
so you really use only half the power. not let's say you have so much traffic
(with limiting enabled, htb or something), that the two CPUs are on 100% all
the time, but the other two are doing nothing.

now if you could balance that to all 4 cpus, you could use all the power AND
no cpu would be used 100%.

but i guess the machine has to do it in hardware, and that was the real
question. irqbalance has no use here, i think, only if the hardware supports it.

is there a way to find out (some easy way) for sure, if one particular
hardware supports this interrupt distribution?

Re: irq balancing question

[Arjan van de Ven]

> in fact i have two cards, and 4 CPUs, but i was interested in then answer
> Robert gave, that only _some_ machines distribute interrupts in hardware.
> software distribution is obviously not good. consider this scenario:
> 
> you have one machine with 4 cpus, and two ethernet cards with a lot of
> traffic on them. if you bind every card to one cpu, two of them are not used,
> so you really use only half the power. not let's say you have so much traffic
> (with limiting enabled, htb or something), that the two CPUs are on 100% all
> the time, but the other two are doing nothing.
> 
> now if you could balance that to all 4 cpus, you could use all the power AND
> no cpu would be used 100%.

actually this will give you worse performance than only using 2 cores.
The reason for this is twofold
1) If you rotate the irqs, TCP and IP packet fragments will arrive at
different CPUs. This in turn means that a VERY expensive reassembly path
gets taken, compared to local-cpu-only reassembly
2) If you rotate the irqs, you bounce cachelines between the caches ALL
THE TIME, which is also very expensive.

Both make it more likely that you'll be slower than just using only 2
cores...



-- 
if you want to mail me at work (you don't), use arjan (at) linux.intel.com
Test the interaction between Linux and your BIOS via http://www.linuxfirmwarekit.org

Re: irq balancing question

[Gergely Imre]

Arjan van de Ven wrote:
>> in fact i have two cards, and 4 CPUs, but i was interested in then answer
>> Robert gave, that only _some_ machines distribute interrupts in hardware.
>> software distribution is obviously not good. consider this scenario:
>>
>> you have one machine with 4 cpus, and two ethernet cards with a lot of
>> traffic on them. if you bind every card to one cpu, two of them are not used,
>> so you really use only half the power. not let's say you have so much traffic
>> (with limiting enabled, htb or something), that the two CPUs are on 100% all
>> the time, but the other two are doing nothing.
>>
>> now if you could balance that to all 4 cpus, you could use all the power AND
>> no cpu would be used 100%.
> 
> actually this will give you worse performance than only using 2 cores.
> The reason for this is twofold
> 1) If you rotate the irqs, TCP and IP packet fragments will arrive at
> different CPUs. This in turn means that a VERY expensive reassembly path
> gets taken, compared to local-cpu-only reassembly
> 2) If you rotate the irqs, you bounce cachelines between the caches ALL
> THE TIME, which is also very expensive.
> 
> Both make it more likely that you'll be slower than just using only 2
> cores...

and i guess it doesn't matter if the distribution is being done by the
hardware, from the point of view of the kernel, i would still get the
performance penalty.

and what if CPU0 and CPU1 is actually the same CPU, only duo core, and i'm
distributing interrupts to them, and with the other card to CPU2 and 3 (which
are part of the other physical CPU) ?

i'm just trying to figure it out, i have no real knowledge of the inner
kernel workings, so i dont know. but i really would like to use all 4 cores.
just how expensive is that reassembly path ?

Re: irq balancing question

[Arjan van de Ven]

> 
> and i guess it doesn't matter if the distribution is being done by the
> hardware, from the point of view of the kernel, i would still get the
> performance penalty.

for the cache bouncing you save assuming you have an Intel cpu (eg one
where the cache is shared). You don't save on the cross-cpu reassembly,
that is an entire different algorithm path you hit there..



> i'm just trying to figure it out, i have no real knowledge of the inner
> kernel workings, so i dont know. but i really would like to use all 4 cores.
> just how expensive is that reassembly path ?

depends on your traffic to be honest, probably a question more suited
for net-dev list.

-- 
if you want to mail me at work (you don't), use arjan (at) linux.intel.com
Test the interaction between Linux and your BIOS via http://www.linuxfirmwarekit.org

Re: irq balancing question

[Gergely Imre]

Arjan van de Ven wrote:
>> and i guess it doesn't matter if the distribution is being done by the
>> hardware, from the point of view of the kernel, i would still get the
>> performance penalty.
> 
> for the cache bouncing you save assuming you have an Intel cpu (eg one
> where the cache is shared). You don't save on the cross-cpu reassembly,
> that is an entire different algorithm path you hit there..
> 
> 
> 
>> i'm just trying to figure it out, i have no real knowledge of the inner
>> kernel workings, so i dont know. but i really would like to use all 4 cores.
>> just how expensive is that reassembly path ?
> 
> depends on your traffic to be honest, probably a question more suited
> for net-dev list.
> 

i'll look into it, thanks a lot for the comments.

Re: irq balancing question

[Robert Hancock]

Imre Gergely wrote:
> hi
> 
> i have a SMP with 2x duo-core Opteron processors, and i have an ethernet card
> (eth0). i want to balance the interrupts generated by the card to all 4
> processors. i can do this by writing to smp_affinity, but on another machine it
> get's done by "default", without doing anything.
> 
> is this irq balancing a feature in the chipset, or driver, or kernel ? i got
> the idea of the smp_affinity stuff, and i've set it to "f", but nothing
> happens, i mean i see only one CPU getting the interrupts.
> 
> i'm using kernel 2.6.18, but tried with 2.6.19, too.

You need to run the irqbalance daemon to do this, unless the interrupts 
are distributed in hardware (which only some machines do).

-- 
Robert Hancock      Saskatoon, SK, Canada
To email, remove "nospam" from hancockr@nospamshaw.ca
Home Page: http://www.roberthancock.com/

RE: irq balancing question

[Pallipadi, Venkatesh]

>> >----- Original Message ----- 
>> >From: "Arjan van de Ven" <arjan@infradead.org>
>> >To: "JaniD++" <djani22@dynamicweb.hu>
>> >> On Wed, 2005-12-14 at 22:05 +0100, JaniD++ wrote:
>> >> > Hello, list,
>> >> >
>> >> > I try to tune my system with manually irq assigning, but
>> >this simple not
>> >> > works, and i don't know why. :(
>> >> > I have already read all the documentation in the kernel
>> >tree, and search
>> >in
>> >> > google, but i can not find any valuable reason.
>> >>
>> >>
>> >> which chipset? there is a chipset that is broken wrt irq 
>balancing so
>> >> the kernel refuses to do it there...
>> >
>> >This happens all of my systems, with different hardware.
>> >
>> >In the example is Intel SE7520AF2,  IntelR E7520 Chipset, +2x
>> >Xeon with HT.
>> >
>> >And the other systems is Abit IS7, intel 865, and only one P4
>> >CPU with HT,
>> >but the issue is the same.
>> >
>>
>> Which kernel and which architecture (i386 or x86-64?)
>
>i386, and kernel 2.6.14 - 2.6.15-rc3

Things should work with 2.6.15-rc5. 
There was a bug with this that was fixed recently. The patch here
http://kernel.org/git/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commi
t;h=fe655d3a06488c8a188461bca493e9f23fc8c448

>
>(the intel xeon CPU can work x86-64 kernels?)
>

Yes. If your CPUs have EM64T capability, then they can run x86-64
kernels.

Thanks,
Venki

Re: irq balancing question

[JaniD++]

----- Original Message ----- 
From: "Pallipadi, Venkatesh" <venkatesh.pallipadi@intel.com>
To: "JaniD++" <djani22@dynamicweb.hu>; "Arjan van de Ven"
<arjan@infradead.org>
Cc: <linux-kernel@vger.kernel.org>
Sent: Thursday, December 15, 2005 1:48 AM
Subject: RE: irq balancing question


>
>
> >-----Original Message-----
> >From: linux-kernel-owner@vger.kernel.org
> >[mailto:linux-kernel-owner@vger.kernel.org] On Behalf Of JaniD++
> >Sent: Wednesday, December 14, 2005 1:32 PM
> >To: Arjan van de Ven
> >Cc: linux-kernel@vger.kernel.org
> >Subject: Re: irq balancing question
> >
> >Hi,
> >
> >----- Original Message ----- 
> >From: "Arjan van de Ven" <arjan@infradead.org>
> >To: "JaniD++" <djani22@dynamicweb.hu>
> >Cc: <linux-kernel@vger.kernel.org>
> >Sent: Wednesday, December 14, 2005 10:16 PM
> >Subject: Re: irq balancing question
> >
> >
> >> On Wed, 2005-12-14 at 22:05 +0100, JaniD++ wrote:
> >> > Hello, list,
> >> >
> >> > I try to tune my system with manually irq assigning, but
> >this simple not
> >> > works, and i don't know why. :(
> >> > I have already read all the documentation in the kernel
> >tree, and search
> >in
> >> > google, but i can not find any valuable reason.
> >>
> >>
> >> which chipset? there is a chipset that is broken wrt irq balancing so
> >> the kernel refuses to do it there...
> >
> >This happens all of my systems, with different hardware.
> >
> >In the example is Intel SE7520AF2,  IntelR E7520 Chipset, +2x
> >Xeon with HT.
> >
> >And the other systems is Abit IS7, intel 865, and only one P4
> >CPU with HT,
> >but the issue is the same.
> >
>
> Which kernel and which architecture (i386 or x86-64?)

i386, and kernel 2.6.14 - 2.6.15-rc3

(the intel xeon CPU can work x86-64 kernels?)

Cheers,
Janos

>
> Thanks,
> Venki
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

RE: irq balancing question

[Pallipadi, Venkatesh]

 

>-----Original Message-----
>From: linux-kernel-owner@vger.kernel.org 
>[mailto:linux-kernel-owner@vger.kernel.org] On Behalf Of JaniD++
>Sent: Wednesday, December 14, 2005 1:32 PM
>To: Arjan van de Ven
>Cc: linux-kernel@vger.kernel.org
>Subject: Re: irq balancing question
>
>Hi,
>
>----- Original Message ----- 
>From: "Arjan van de Ven" <arjan@infradead.org>
>To: "JaniD++" <djani22@dynamicweb.hu>
>Cc: <linux-kernel@vger.kernel.org>
>Sent: Wednesday, December 14, 2005 10:16 PM
>Subject: Re: irq balancing question
>
>
>> On Wed, 2005-12-14 at 22:05 +0100, JaniD++ wrote:
>> > Hello, list,
>> >
>> > I try to tune my system with manually irq assigning, but 
>this simple not
>> > works, and i don't know why. :(
>> > I have already read all the documentation in the kernel 
>tree, and search
>in
>> > google, but i can not find any valuable reason.
>>
>>
>> which chipset? there is a chipset that is broken wrt irq balancing so
>> the kernel refuses to do it there...
>
>This happens all of my systems, with different hardware.
>
>In the example is Intel SE7520AF2,  IntelR E7520 Chipset, +2x 
>Xeon with HT.
>
>And the other systems is Abit IS7, intel 865, and only one P4 
>CPU with HT,
>but the issue is the same.
>

Which kernel and which architecture (i386 or x86-64?)

Thanks,
Venki

Re: irq balancing question

[JaniD++]

Hi,

----- Original Message ----- 
From: "Arjan van de Ven" <arjan@infradead.org>
To: "JaniD++" <djani22@dynamicweb.hu>
Cc: <linux-kernel@vger.kernel.org>
Sent: Wednesday, December 14, 2005 10:16 PM
Subject: Re: irq balancing question


> On Wed, 2005-12-14 at 22:05 +0100, JaniD++ wrote:
> > Hello, list,
> >
> > I try to tune my system with manually irq assigning, but this simple not
> > works, and i don't know why. :(
> > I have already read all the documentation in the kernel tree, and search
in
> > google, but i can not find any valuable reason.
>
>
> which chipset? there is a chipset that is broken wrt irq balancing so
> the kernel refuses to do it there...

This happens all of my systems, with different hardware.

In the example is Intel SE7520AF2,  IntelR E7520 Chipset, +2x Xeon with HT.

And the other systems is Abit IS7, intel 865, and only one P4 CPU with HT,
but the issue is the same.

Cheers,
Janos


>

Re: irq balancing question

[Arjan van de Ven]

On Wed, 2005-12-14 at 22:05 +0100, JaniD++ wrote:
> Hello, list,
> 
> I try to tune my system with manually irq assigning, but this simple not
> works, and i don't know why. :(
> I have already read all the documentation in the kernel tree, and search in
> google, but i can not find any valuable reason.


which chipset? there is a chipset that is broken wrt irq balancing so
the kernel refuses to do it there...

irq balancing question

[JaniD++]

Hello, list,

I try to tune my system with manually irq assigning, but this simple not
works, and i don't know why. :(
I have already read all the documentation in the kernel tree, and search in
google, but i can not find any valuable reason.

[root@dy-xeon-1 proc]# cat irq/217/smp_affinity
f
[root@dy-xeon-1 proc]# echo 1 > irq/217/smp_affinity
[root@dy-xeon-1 proc]# cat irq/217/smp_affinity
f
[root@dy-xeon-1 proc]# /bin/echo 1 > irq/217/smp_affinity
[root@dy-xeon-1 proc]# cat irq/217/smp_affinity
f
[root@dy-xeon-1 proc]# cat interrupts
           CPU0       CPU1       CPU2       CPU3
  0:        117          0          0   50302311    IO-APIC-edge  timer
  1:          0          0          0        560    IO-APIC-edge  i8042
  8:          0          0          0       8369    IO-APIC-edge  rtc
  9:          0          0          0          0   IO-APIC-level  acpi
 14:          0          0          0    9707311    IO-APIC-edge  ide0
169:          0          0          0          1   IO-APIC-level
uhci_hcd:usb2
177:          0          0          0          0   IO-APIC-level
uhci_hcd:usb3
185:          0          0          0          0   IO-APIC-level
uhci_hcd:usb4
193:          0          0          0    4335777   IO-APIC-level
ehci_hcd:usb1
209:          0          0          0 1777636996   IO-APIC-level  eth0
217:          0          0          0 2669520977   IO-APIC-level  eth1
NMI:          0          0          0          0
LOC:   50307523   50307625   50307006   50302885
ERR:          0
MIS:          0
[root@dy-xeon-1 proc]#

Can somebody help me?

Cheers,
Janos

irq balancing question

[JaniD++]

Hello,list,

I want to change the irq bounding, but i can't.

[root@dy-xeon-1 irq]# cat 217/smp_affinity
f
[root@dy-xeon-1 irq]# echo 1 > 217/smp_affinity
[root@dy-xeon-1 irq]# cat 217/smp_affinity
f
[root@dy-xeon-1 irq]# /bin/echo 1 > 217/smp_affinity
[root@dy-xeon-1 irq]# cat 217/smp_affinity
f
[root@dy-xeon-1 irq]#

           CPU0       CPU1       CPU2       CPU3
  0:        117          0          0   17139594    IO-APIC-edge  timer
  8:          0          0          0       2842    IO-APIC-edge  rtc
  9:          0          0          0          0   IO-APIC-level  acpi
 14:          0          0          0    2701302    IO-APIC-edge  ide0
169:          0          0          0          0   IO-APIC-level
uhci_hcd:usb2
177:          0          0          0          0   IO-APIC-level
uhci_hcd:usb3
185:          0          0          0          0   IO-APIC-level
uhci_hcd:usb4
193:          0          0          0          0   IO-APIC-level
ehci_hcd:usb1
209:          0          0          0  211910641   IO-APIC-level  eth0
217:          0          0          0  494228730   IO-APIC-level  eth1
NMI:          0          0          0          0
LOC:   17139481   17139480   17139479   17139375
ERR:          0
MIS:          0

I have allready read all the documentation about this, and try to find
answer in google, but found nothing. :(

My kernel is 2.6.15-rc3, but with 2.6.14.2 is the same.
The system is dual xeon + HT.

My kernel config file is here:
http://download.netcenter.hu/bughunt/smp/config.txt


Thanks,
Janos

Re: irq balancing question

[JaniD++]

----- Original Message ----- 
From: "Pallipadi, Venkatesh" <venkatesh.pallipadi@intel.com>
To: "JaniD++" <djani22@dynamicweb.hu>
Cc: <linux-kernel@vger.kernel.org>
Sent: Thursday, December 15, 2005 3:00 PM
Subject: RE: irq balancing question


> >> >----- Original Message ----- 
> >> >From: "Arjan van de Ven" <arjan@infradead.org>
> >> >To: "JaniD++" <djani22@dynamicweb.hu>
> >> >> On Wed, 2005-12-14 at 22:05 +0100, JaniD++ wrote:
> >> >> > Hello, list,
> >> >> >
> >> >> > I try to tune my system with manually irq assigning, but
> >> >this simple not
> >> >> > works, and i don't know why. :(
> >> >> > I have already read all the documentation in the kernel
> >> >tree, and search
> >> >in
> >> >> > google, but i can not find any valuable reason.
> >> >>
> >> >>
> >> >> which chipset? there is a chipset that is broken wrt irq
> >balancing so
> >> >> the kernel refuses to do it there...
> >> >
> >> >This happens all of my systems, with different hardware.
> >> >
> >> >In the example is Intel SE7520AF2,  IntelR E7520 Chipset, +2x
> >> >Xeon with HT.
> >> >
> >> >And the other systems is Abit IS7, intel 865, and only one P4
> >> >CPU with HT,
> >> >but the issue is the same.
> >> >
> >>
> >> Which kernel and which architecture (i386 or x86-64?)
> >
> >i386, and kernel 2.6.14 - 2.6.15-rc3
>
> Things should work with 2.6.15-rc5.
> There was a bug with this that was fixed recently. The patch here
> http://kernel.org/git/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commi
> t;h=fe655d3a06488c8a188461bca493e9f23fc8c448

Ahh, thanks! :-)
This fix the problem!

>
> >
> >(the intel xeon CPU can work x86-64 kernels?)
> >
>
> Yes. If your CPUs have EM64T capability, then they can run x86-64
> kernels.

OK, i will try it! ;-)

Thanks,
Janos

>
> Thanks,
> Venki
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/