Should kirqd work on HT?

Should kirqd work on HT?

[Nigel Cunningham]

Hi all.

I've noticed this problem for a while, but only now decided to ask.
Interrupt balancing doesn't do anything on my system.

           CPU0       CPU1
  0:   31931808          0    IO-APIC-edge  timer
  1:      76595          0    IO-APIC-edge  i8042
  8:          1          0    IO-APIC-edge  rtc
  9:          1          0   IO-APIC-level  acpi
 14:        122          1    IO-APIC-edge  ide0
 16:    4074456          0   IO-APIC-level  uhci_hcd, uhci_hcd, radeon@PCI:1:0:0
 17:    4295132          0   IO-APIC-level  Intel ICH5
 18:    2070933          0   IO-APIC-level  libata, uhci_hcd, eth0
 19:     887311          0   IO-APIC-level  uhci_hcd
 22:     572530          0   IO-APIC-level  ath0
NMI:   31931749   31931636 (I've since disabled the nmi_watchdog)
LOC:   31931252   31931251
ERR:          0
MIS:          0

I enabled the debugging and found that it doesn't think it's worth the
effort. Is that correct? Not a complaint, just curious!

Regards,

Nigel
-- 
Nigel Cunningham
Software Engineer, Canberra, Australia
http://www.cyclades.com

Ph: +61 (2) 6292 8028      Mob: +61 (417) 100 574

Re: Should kirqd work on HT?

[Kwijibo]

My guess is that irqbalance is not running.

Nigel Cunningham wrote:

>Hi all.
>
>I've noticed this problem for a while, but only now decided to ask.
>Interrupt balancing doesn't do anything on my system.
>
>           CPU0       CPU1
>  0:   31931808          0    IO-APIC-edge  timer
>  1:      76595          0    IO-APIC-edge  i8042
>  8:          1          0    IO-APIC-edge  rtc
>  9:          1          0   IO-APIC-level  acpi
> 14:        122          1    IO-APIC-edge  ide0
> 16:    4074456          0   IO-APIC-level  uhci_hcd, uhci_hcd, radeon@PCI:1:0:0
> 17:    4295132          0   IO-APIC-level  Intel ICH5
> 18:    2070933          0   IO-APIC-level  libata, uhci_hcd, eth0
> 19:     887311          0   IO-APIC-level  uhci_hcd
> 22:     572530          0   IO-APIC-level  ath0
>NMI:   31931749   31931636 (I've since disabled the nmi_watchdog)
>LOC:   31931252   31931251
>ERR:          0
>MIS:          0
>
>I enabled the debugging and found that it doesn't think it's worth the
>effort. Is that correct? Not a complaint, just curious!
>
>Regards,
>
>Nigel
>  
>

Re: Should kirqd work on HT?

[Jeff Garzik]

Nigel Cunningham wrote:
> Hi all.
> 
> I've noticed this problem for a while, but only now decided to ask.
> Interrupt balancing doesn't do anything on my system.
> 
>            CPU0       CPU1
>   0:   31931808          0    IO-APIC-edge  timer
>   1:      76595          0    IO-APIC-edge  i8042
>   8:          1          0    IO-APIC-edge  rtc
>   9:          1          0   IO-APIC-level  acpi
>  14:        122          1    IO-APIC-edge  ide0
>  16:    4074456          0   IO-APIC-level  uhci_hcd, uhci_hcd, radeon@PCI:1:0:0
>  17:    4295132          0   IO-APIC-level  Intel ICH5
>  18:    2070933          0   IO-APIC-level  libata, uhci_hcd, eth0
>  19:     887311          0   IO-APIC-level  uhci_hcd
>  22:     572530          0   IO-APIC-level  ath0
> NMI:   31931749   31931636 (I've since disabled the nmi_watchdog)
> LOC:   31931252   31931251
> ERR:          0
> MIS:          0
> 
> I enabled the debugging and found that it doesn't think it's worth the
> effort. Is that correct? Not a complaint, just curious!

What are the results of running irqbalanced?

	Jeff

Re: Should kirqd work on HT?

[Nigel Cunningham]

Hi.

On Sat, 2005-02-19 at 17:44, Kwijibo wrote:
> My guess is that irqbalance is not running.

No. It is.

USER       PID %CPU %MEM   VSZ  RSS TTY      STAT START   TIME COMMAND
root       301  0.0  0.0     0    0 ?        SW   16:52   0:00 [kirqd]

The debugging info reports that it doesn't think it's worth doing the
balancing.

Regards,

Nigel
-- 
Nigel Cunningham
Software Engineer, Canberra, Australia
http://www.cyclades.com

Ph: +61 (2) 6292 8028      Mob: +61 (417) 100 574

Re: Should kirqd work on HT?

[Nigel Cunningham]

Hi Jeff.

On Sat, 2005-02-19 at 17:52, Jeff Garzik wrote:
> Nigel Cunningham wrote:
> What are the results of running irqbalanced?

You mean the debugging output? I can reenable it and record the results
if that's what you mean.

>From memory though, it got to not_worth_the_effort via this code:

        if (tmp_loaded == -1) {
         /* In the case of small number of heavy interrupt sources,
          * loading some of the cpus too much. We use Ingo's original
          * approach to rotate them around.
          */
                if (!first_attempt && imbalance >= useful_load_threshold) {
                        rotate_irqs_among_cpus(useful_load_threshold);
                        return;
                }
                goto not_worth_the_effort;
        }


(arch/i386/kernel/io_apic.c:449.

Regards,

Nigel
-- 
Nigel Cunningham
Software Engineer, Canberra, Australia
http://www.cyclades.com

Ph: +61 (2) 6292 8028      Mob: +61 (417) 100 574

Re: Should kirqd work on HT?

[Arjan van de Ven]

On Sat, 2005-02-19 at 20:02 +1100, Nigel Cunningham wrote:
> Hi.
> 
> On Sat, 2005-02-19 at 17:44, Kwijibo wrote:
> > My guess is that irqbalance is not running.
> 
> No. It is.
> 
> USER       PID %CPU %MEM   VSZ  RSS TTY      STAT START   TIME COMMAND
> root       301  0.0  0.0     0    0 ?        SW   16:52   0:00 [kirqd]
> 
> The debugging info reports that it doesn't think it's worth doing the
> balancing.

I guess the question was about the userspace irqbalance....

Re: Should kirqd work on HT?

[Arjan van de Ven]

On Sat, 2005-02-19 at 20:07 +1100, Nigel Cunningham wrote:
> Hi Jeff.
> 
> On Sat, 2005-02-19 at 17:52, Jeff Garzik wrote:
> > Nigel Cunningham wrote:
> > What are the results of running irqbalanced?
> 
> You mean the debugging output? I can reenable it and record the results
> if that's what you mean.

no Jeff meant
http://people.redhat.com/arjanv/irqbalance/
that app most likely....

Re: Should kirqd work on HT?

[Nigel Cunningham]

Hi,

On Sat, 2005-02-19 at 20:36, Arjan van de Ven wrote:
> On Sat, 2005-02-19 at 20:07 +1100, Nigel Cunningham wrote:
> > Hi Jeff.
> > 
> > On Sat, 2005-02-19 at 17:52, Jeff Garzik wrote:
> > > Nigel Cunningham wrote:
> > > What are the results of running irqbalanced?
> > 
> > You mean the debugging output? I can reenable it and record the results
> > if that's what you mean.
> 
> no Jeff meant
> http://people.redhat.com/arjanv/irqbalance/
> that app most likely....

Oh... Ok. Never heard of it before.

I'll download it now, but I'm about to go to bed now so the answer will
be a while.

Thanks for the reply.

Nigel

-- 
Nigel Cunningham
Software Engineer, Canberra, Australia
http://www.cyclades.com

Ph: +61 (2) 6292 8028      Mob: +61 (417) 100 574

Re: Should kirqd work on HT?

[Nigel Cunningham]

Hi again.

Didn't realise it was going to take nothing to install, so I've done it.
IRQs are running on cpu 1 now. Is there some documentation somewhere?
I'm wondering whether I can compile kirqd out.

Thanks and regards,

Nigel

On Sat, 2005-02-19 at 20:36, Arjan van de Ven wrote:
> On Sat, 2005-02-19 at 20:07 +1100, Nigel Cunningham wrote:
> > Hi Jeff.
> > 
> > On Sat, 2005-02-19 at 17:52, Jeff Garzik wrote:
> > > Nigel Cunningham wrote:
> > > What are the results of running irqbalanced?
> > 
> > You mean the debugging output? I can reenable it and record the results
> > if that's what you mean.
> 
> no Jeff meant
> http://people.redhat.com/arjanv/irqbalance/
> that app most likely....
> 
-- 
Nigel Cunningham
Software Engineer, Canberra, Australia
http://www.cyclades.com

Ph: +61 (2) 6292 8028      Mob: +61 (417) 100 574

Re: Should kirqd work on HT?

[Arjan van de Ven]

On Sat, 2005-02-19 at 21:58 +1100, Nigel Cunningham wrote:
> Hi again.
> 
> Didn't realise it was going to take nothing to install, so I've done it.
> IRQs are running on cpu 1 now. Is there some documentation somewhere?
> I'm wondering whether I can compile kirqd out.

with irqbalance running yes you can compile kirqd out..

Re: Should kirqd work on HT?

[Martin J. Bligh]

> I've noticed this problem for a while, but only now decided to ask.
> Interrupt balancing doesn't do anything on my system.
> 
>            CPU0       CPU1
>   0:   31931808          0    IO-APIC-edge  timer
>   1:      76595          0    IO-APIC-edge  i8042
>   8:          1          0    IO-APIC-edge  rtc
>   9:          1          0   IO-APIC-level  acpi
>  14:        122          1    IO-APIC-edge  ide0
>  16:    4074456          0   IO-APIC-level  uhci_hcd, uhci_hcd, radeon@PCI:1:0:0
>  17:    4295132          0   IO-APIC-level  Intel ICH5
>  18:    2070933          0   IO-APIC-level  libata, uhci_hcd, eth0
>  19:     887311          0   IO-APIC-level  uhci_hcd
>  22:     572530          0   IO-APIC-level  ath0
> NMI:   31931749   31931636 (I've since disabled the nmi_watchdog)
> LOC:   31931252   31931251
> ERR:          0
> MIS:          0
> 
> I enabled the debugging and found that it doesn't think it's worth the
> effort. Is that correct? Not a complaint, just curious!

I think it's nothing to do with HT, just the rate you're sending intterrupts
at isn't high enough to bother rotating.

M.

Re: Should kirqd work on HT?

[Jeff Garzik]

Martin J. Bligh wrote:
> --Jeff Garzik <jgarzik@pobox.com> wrote (on Saturday, February 19, 2005 11:30:53 -0500):
> 
> 
>>Pallipadi, Venkatesh wrote:
>>
>>>You are right. Kernel balancer doesn't move around the irqs, unless it
>>>has too many interrupts. The logic is moving around interrupts all the
>>>time will not be good on caches. So, there is a threshold above which
>>>the balancer start moving things around.
>>>
>>>You should see them moving around if you do 'ping -f' or a big 'dd' from
>>>the disk.
>>
>>If kirqd is moving NIC interrupts, it's broken.
>>
>>(and another reason why irqbalanced is preferable)
> 
> 
> Why is it broken to move NIC interrupts? Obviously you don't want to
> rotate them around a lot, but in the interests of fairness to other 
> processes, it seems reasonable to migrate them occasionally (IIRC, kirqd
> rate limits to once a second or something).

This has been explained to you before, search your email archives...

The main problem that has been seen in the field SMP packet ordering, 
but a secondary problem observed is cache misses.  Just NAPI mitigates 
this somewhat (no pun intended).

Overall, kirqd should be avoided except in special situations.  It 
doesn't know about such policy things as network-specific or 
storage-specific irq balancing (and shouldn't).

	Jeff

Re: Should kirqd work on HT?

[Martin J. Bligh]

--Jeff Garzik <jgarzik@pobox.com> wrote (on Saturday, February 19, 2005 11:30:53 -0500):

> Pallipadi, Venkatesh wrote:
>> You are right. Kernel balancer doesn't move around the irqs, unless it
>> has too many interrupts. The logic is moving around interrupts all the
>> time will not be good on caches. So, there is a threshold above which
>> the balancer start moving things around.
>> 
>> You should see them moving around if you do 'ping -f' or a big 'dd' from
>> the disk.
> 
> If kirqd is moving NIC interrupts, it's broken.
> 
> (and another reason why irqbalanced is preferable)

Why is it broken to move NIC interrupts? Obviously you don't want to
rotate them around a lot, but in the interests of fairness to other 
processes, it seems reasonable to migrate them occasionally (IIRC, kirqd
rate limits to once a second or something).

M.

Re: Should kirqd work on HT?

[Jeff Garzik]

Pallipadi, Venkatesh wrote:
> You are right. Kernel balancer doesn't move around the irqs, unless it
> has too many interrupts. The logic is moving around interrupts all the
> time will not be good on caches. So, there is a threshold above which
> the balancer start moving things around.
> 
> You should see them moving around if you do 'ping -f' or a big 'dd' from
> the disk.

If kirqd is moving NIC interrupts, it's broken.

(and another reason why irqbalanced is preferable)

	Jeff

RE: Should kirqd work on HT?

[Pallipadi, Venkatesh]

You are right. Kernel balancer doesn't move around the irqs, unless it
has too many interrupts. The logic is moving around interrupts all the
time will not be good on caches. So, there is a threshold above which
the balancer start moving things around.

You should see them moving around if you do 'ping -f' or a big 'dd' from
the disk.

Thanks,
Venki 

>-----Original Message-----
>From: linux-kernel-owner@vger.kernel.org 
>[mailto:linux-kernel-owner@vger.kernel.org] On Behalf Of Nigel 
>Cunningham
>Sent: Saturday, February 19, 2005 1:02 AM
>To: kwijibo@zianet.com
>Cc: Linux Kernel Mailing List
>Subject: Re: Should kirqd work on HT?
>
>Hi.
>
>On Sat, 2005-02-19 at 17:44, Kwijibo wrote:
>> My guess is that irqbalance is not running.
>
>No. It is.
>
>USER       PID %CPU %MEM   VSZ  RSS TTY      STAT START   TIME COMMAND
>root       301  0.0  0.0     0    0 ?        SW   16:52   0:00 [kirqd]
>
>The debugging info reports that it doesn't think it's worth doing the
>balancing.
>
>Regards,
>
>Nigel
>-- 
>Nigel Cunningham
>Software Engineer, Canberra, Australia
>http://www.cyclades.com
>
>Ph: +61 (2) 6292 8028      Mob: +61 (417) 100 574
>
>-
>To unsubscribe from this list: send the line "unsubscribe 
>linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/
>