[PATCH] genirq: Set initial default irq affinity to just CPU0

[PATCH] genirq: Set initial default irq affinity to just CPU0

[Kumar Gala]

Commit 18404756765c713a0be4eb1082920c04822ce588 introduced a regression
on a subset of SMP based PPC systems whose interrupt controller only
allow setting an irq to a single processor.  The previous behavior
was only CPU0 was initially setup to get interrupts.  Revert back
to that behavior.

Signed-off-by: Kumar Gala <galak@kernel.crashing.org>
---
 kernel/irq/manage.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/kernel/irq/manage.c b/kernel/irq/manage.c
index c498a1b..728d36a 100644
--- a/kernel/irq/manage.c
+++ b/kernel/irq/manage.c
@@ -17,7 +17,7 @@
 
 #ifdef CONFIG_SMP
 
-cpumask_t irq_default_affinity = CPU_MASK_ALL;
+cpumask_t irq_default_affinity = CPU_MASK_CPU0;
 
 /**
  *	synchronize_irq - wait for pending IRQ handlers (on other CPUs)
-- 
1.5.5.1

Re: [PATCH] genirq: Set initial default irq affinity to just CPU0

[David Miller]

From: Kumar Gala <galak@kernel.crashing.org>
Date: Fri, 24 Oct 2008 10:57:38 -0500

> Commit 18404756765c713a0be4eb1082920c04822ce588 introduced a regression
> on a subset of SMP based PPC systems whose interrupt controller only
> allow setting an irq to a single processor.  The previous behavior
> was only CPU0 was initially setup to get interrupts.  Revert back
> to that behavior.
> 
> Signed-off-by: Kumar Gala <galak@kernel.crashing.org>

I really don't remember getting all of my interrupts only on cpu 0
on sparc64 before any of these changes.  I therefore find all of
this quite mysterious. :-)

Re: [PATCH] genirq: Set initial default irq affinity to just CPU0

[Benjamin Herrenschmidt]

On Fri, 2008-10-24 at 16:18 -0700, David Miller wrote:
> From: Kumar Gala <galak@kernel.crashing.org>
> Date: Fri, 24 Oct 2008 10:57:38 -0500
> 
> > Commit 18404756765c713a0be4eb1082920c04822ce588 introduced a regression
> > on a subset of SMP based PPC systems whose interrupt controller only
> > allow setting an irq to a single processor.  The previous behavior
> > was only CPU0 was initially setup to get interrupts.  Revert back
> > to that behavior.
> > 
> > Signed-off-by: Kumar Gala <galak@kernel.crashing.org>
> 
> I really don't remember getting all of my interrupts only on cpu 0
> on sparc64 before any of these changes.  I therefore find all of
> this quite mysterious. :-)

Well, I don't know how you do it but on powerpc, we explicitely fill the
affinity masks at boot time when we can spread interrupts... Maybe we
should change it the other way around and limit the mask when we can't ?
It's hard to tell for sure at this stage.

Ben.

Re: [PATCH] genirq: Set initial default irq affinity to just CPU0

[David Miller]

From: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Date: Sun, 26 Oct 2008 08:33:09 +1100

> Well, I don't know how you do it but on powerpc, we explicitely fill the
> affinity masks at boot time when we can spread interrupts... Maybe we
> should change it the other way around and limit the mask when we can't ?
> It's hard to tell for sure at this stage.

On sparc64 we look at the cpu mask configured for the interrupt and do
one of two things:

1) If all bits are set, we round robin assign a cpu at IRQ enable time.

2) Else we pick the first bit set in the mask.

One modification I want to make is to make case #1 NUMA aware.

But back to my original wonder, since I've always tipped off of this
generic IRQ layer cpu mask, when was it ever defaulting to zero
and causing the behvaior your powerpc guys actually want? :-)

Re: [PATCH] genirq: Set initial default irq affinity to just CPU0

[Benjamin Herrenschmidt]

On Sat, 2008-10-25 at 21:04 -0700, David Miller wrote:
> But back to my original wonder, since I've always tipped off of this
> generic IRQ layer cpu mask, when was it ever defaulting to zero
> and causing the behvaior your powerpc guys actually want? :-)

Well, I'm not sure what Kumar wants. Most powerpc SMP setups actually
want to spread interrupts to all CPUs, and those who can't tend to just
not implement set_affinity... So Kumar must have a special case of MPIC
usage here on FSL platforms.

In any case, the platform limitations should be dealt with there or the
user could break it by manipulating affinity via /proc anyway.

By yeah, I do expect default affinity to be all CPUs and in fact, I even
have an -OLD- comment in the code that says

 	/* let the mpic know we want intrs. default affinitya is 0xffffffff ...

Now, I've tried to track that down but it's hard because the generic code
seem to have changed in many ways around affinity handling...

So it looks like nowadays, the generic setup_irq() will call
irq_select_affinity() when an interrupt is first requested. Unless
you set CONFIG_AUTO_IRQ_AFFINITY and implement your own
irq_select_affinity(), thus, you will get the default one which copies
the content of this global irq_default_affinity to the interrupt.

However it does that _after_ your IRQ startup() has been called
(yes, this is very fishy), and so after you did your irq_choose_cpu()...

This is all very messy, along with hooks for balancing and other confusing
stuff that I suspect keeps changing. I'll have to spend more time next
week to sort out what exactly is happening on powerpc and whether we
get our interrupts spread or not...

That's the downside of having more generic irq code I suppose: now people
keep rewriting half of the generic code with x86 exclusively in mind and
we have to be extra careful :-)

Cheers,
Ben.

Re: [PATCH] genirq: Set initial default irq affinity to just CPU0

[Kumar Gala]

On Oct 26, 2008, at 1:33 AM, Benjamin Herrenschmidt wrote:

> On Sat, 2008-10-25 at 21:04 -0700, David Miller wrote:
>> But back to my original wonder, since I've always tipped off of this
>> generic IRQ layer cpu mask, when was it ever defaulting to zero
>> and causing the behvaior your powerpc guys actually want? :-)
>
> Well, I'm not sure what Kumar wants. Most powerpc SMP setups actually
> want to spread interrupts to all CPUs, and those who can't tend to  
> just
> not implement set_affinity... So Kumar must have a special case of  
> MPIC
> usage here on FSL platforms.
>
> In any case, the platform limitations should be dealt with there or  
> the
> user could break it by manipulating affinity via /proc anyway.
>
> By yeah, I do expect default affinity to be all CPUs and in fact, I  
> even
> have an -OLD- comment in the code that says
>
> 	/* let the mpic know we want intrs. default affinitya is  
> 0xffffffff ...

While we have the comment the code appears not to really follow it.   
We appear to write 1 << hard_smp_processor_id().

- k

Re: [PATCH] genirq: Set initial default irq affinity to just CPU0

[Benjamin Herrenschmidt]

On Mon, 2008-10-27 at 08:43 -0500, Kumar Gala wrote:
> 
> While we have the comment the code appears not to really follow it.   
> We appear to write 1 << hard_smp_processor_id().

That code is called by each CPU that gets onlined and OR's it's
bit in the mask.

Ben.

Re: [PATCH] genirq: Set initial default irq affinity to just CPU0

[Kumar Gala]

On Oct 27, 2008, at 3:27 PM, Benjamin Herrenschmidt wrote:

> On Mon, 2008-10-27 at 08:43 -0500, Kumar Gala wrote:
>>
>> While we have the comment the code appears not to really follow it.
>> We appear to write 1 << hard_smp_processor_id().
>
> That code is called by each CPU that gets onlined and OR's it's
> bit in the mask.

ahh, I see now.

- k