[PATCH 1/2 v2] sparc32: implement SMP IPIs using the generic functions

[PATCH 1/2 v2] sparc32: implement SMP IPIs using the generic functions

[Daniel Hellstrom]

The current sparc32 SMP IPI generation is implemented the
cross call function. The cross call function uses IRQ15 the
NMI, this is has the effect that IPIs will interrupt IRQ
critical areas and hang the system. Typically on/after
spin_lock_irqsave calls can be aborted.

The cross call functionality must still exist to flush
cache/TLBS.

This patch provides CPU models a custom way to implement
generation of IPIs on the generic code's request. The
typical approach is to generate an IRQ for each IPI case.

After this patch each sparc32 SMP CPU model needs to
implement IPIs in order to function properly.

Signed-off-by: Daniel Hellstrom <daniel@gaisler.com>
---
 arch/sparc/Kconfig              |    2 +-
 arch/sparc/include/asm/smp_32.h |   15 ++-------------
 arch/sparc/kernel/smp_32.c      |   34 +++++++++++++++++++++++++++++++++-
 3 files changed, 36 insertions(+), 15 deletions(-)

diff --git a/arch/sparc/Kconfig b/arch/sparc/Kconfig
index e48f471..cef291e 100644
--- a/arch/sparc/Kconfig
+++ b/arch/sparc/Kconfig
@@ -25,6 +25,7 @@ config SPARC
 	select HAVE_DMA_ATTRS
 	select HAVE_DMA_API_DEBUG
 	select HAVE_ARCH_JUMP_LABEL
+	select USE_GENERIC_SMP_HELPERS if SMP
 
 config SPARC32
 	def_bool !64BIT
@@ -43,7 +44,6 @@ config SPARC64
 	select HAVE_DYNAMIC_FTRACE
 	select HAVE_FTRACE_MCOUNT_RECORD
 	select HAVE_SYSCALL_TRACEPOINTS
-	select USE_GENERIC_SMP_HELPERS if SMP
 	select RTC_DRV_CMOS
 	select RTC_DRV_BQ4802
 	select RTC_DRV_SUN4V
diff --git a/arch/sparc/include/asm/smp_32.h b/arch/sparc/include/asm/smp_32.h
index d82d7f4..8390ef6 100644
--- a/arch/sparc/include/asm/smp_32.h
+++ b/arch/sparc/include/asm/smp_32.h
@@ -73,19 +73,8 @@ static inline void xc4(smpfunc_t func, unsigned long arg1, unsigned long arg2,
 			   unsigned long arg3, unsigned long arg4)
 { smp_cross_call(func, cpu_online_map, arg1, arg2, arg3, arg4); }
 
-static inline int smp_call_function(void (*func)(void *info), void *info, int wait)
-{
-	xc1((smpfunc_t)func, (unsigned long)info);
-	return 0;
-}
-
-static inline int smp_call_function_single(int cpuid, void (*func) (void *info),
-					   void *info, int wait)
-{
-	smp_cross_call((smpfunc_t)func, cpumask_of_cpu(cpuid),
-		       (unsigned long) info, 0, 0, 0);
-	return 0;
-}
+extern void arch_send_call_function_single_ipi(int cpu);
+extern void arch_send_call_function_ipi_mask(const struct cpumask *mask);
 
 static inline int cpu_logical_map(int cpu)
 {
diff --git a/arch/sparc/kernel/smp_32.c b/arch/sparc/kernel/smp_32.c
index 91c10fb..a1bb3a8 100644
--- a/arch/sparc/kernel/smp_32.c
+++ b/arch/sparc/kernel/smp_32.c
@@ -125,13 +125,45 @@ struct linux_prom_registers smp_penguin_ctable __cpuinitdata = { 0 };
 
 void smp_send_reschedule(int cpu)
 {
-	/* See sparc64 */
+	/*
+	 * CPU model dependent way of implementing IPI generation targeting
+	 * a single CPU. The trap handler needs only to do trap entry/return
+	 * to call schedule.
+	 */
+	switch (sparc_cpu_model) {
+	default:
+		BUG();
+	}
 }
 
 void smp_send_stop(void)
 {
 }
 
+void arch_send_call_function_single_ipi(int cpu)
+{
+	/*
+	 * CPU model dependent way of implementing IPI generation targeting
+	 * a single CPU
+	 */
+	switch (sparc_cpu_model) {
+	default:
+		BUG();
+	}
+}
+
+void arch_send_call_function_ipi_mask(const struct cpumask *mask)
+{
+	/*
+	 * CPU model dependent way of implementing IPI generation targeting
+	 * a set of CPUs
+	 */
+	switch (sparc_cpu_model) {
+	default:
+		BUG();
+	}
+}
+
 void smp_flush_cache_all(void)
 {
 	xc0((smpfunc_t) BTFIXUP_CALL(local_flush_cache_all));
-- 
1.5.4

Re: [PATCH 1/2 v2] sparc32: implement SMP IPIs using the generic

[David Miller]

From: Daniel Hellstrom <daniel@gaisler.com>
Date: Wed,  2 Feb 2011 20:40:55 +0100

> The current sparc32 SMP IPI generation is implemented the
> cross call function. The cross call function uses IRQ15 the
> NMI, this is has the effect that IPIs will interrupt IRQ
> critical areas and hang the system. Typically on/after
> spin_lock_irqsave calls can be aborted.
> 
> The cross call functionality must still exist to flush
> cache/TLBS.
> 
> This patch provides CPU models a custom way to implement
> generation of IPIs on the generic code's request. The
> typical approach is to generate an IRQ for each IPI case.
> 
> After this patch each sparc32 SMP CPU model needs to
> implement IPIs in order to function properly.
> 
> Signed-off-by: Daniel Hellstrom <daniel@gaisler.com>

Ok, this looks mostly fine to me.

Once we have the sun4m/sun4d versions written I can think about
applying this.

We might want to abstract out the common code of the dispatch and
service stuff you have for LEON, otherwise we'll have 3 copies of
that code that sets and clears the mask state.

Perhaps something using BTFIXUP or similar.

Re: [PATCH 1/2 v2] sparc32: implement SMP IPIs using the generic

[Daniel Hellstrom]

David Miller wrote:

>From: Daniel Hellstrom <daniel@gaisler.com>
>Date: Wed,  2 Feb 2011 20:40:55 +0100
>
>  
>
>>The current sparc32 SMP IPI generation is implemented the
>>cross call function. The cross call function uses IRQ15 the
>>NMI, this is has the effect that IPIs will interrupt IRQ
>>critical areas and hang the system. Typically on/after
>>spin_lock_irqsave calls can be aborted.
>>
>>The cross call functionality must still exist to flush
>>cache/TLBS.
>>
>>This patch provides CPU models a custom way to implement
>>generation of IPIs on the generic code's request. The
>>typical approach is to generate an IRQ for each IPI case.
>>
>>After this patch each sparc32 SMP CPU model needs to
>>implement IPIs in order to function properly.
>>
>>Signed-off-by: Daniel Hellstrom <daniel@gaisler.com>
>>    
>>
>
>Ok, this looks mostly fine to me.
>
>Once we have the sun4m/sun4d versions written I can think about
>applying this.
>
>We might want to abstract out the common code of the dispatch and
>service stuff you have for LEON, otherwise we'll have 3 copies of
>that code that sets and clears the mask state.
>
>Perhaps something using BTFIXUP or similar.
>  
>
I was thinking so too, BTFIXUPs for CPU model specific "IPI operations" 
to avoid the switch-statement in the arch_ functions.

Daniel

Re: [PATCH 1/2 v2] sparc32: implement SMP IPIs using the generic

[David Miller]

From: Daniel Hellstrom <daniel@gaisler.com>
Date: Thu, 03 Feb 2011 09:50:05 +0100

> David Miller wrote:
> 
>>Ok, this looks mostly fine to me.
>>
>>Once we have the sun4m/sun4d versions written I can think about
>>applying this.
>>
>>We might want to abstract out the common code of the dispatch and
>>service stuff you have for LEON, otherwise we'll have 3 copies of
>>that code that sets and clears the mask state.
>>
>>Perhaps something using BTFIXUP or similar.
>>  
> I was thinking so too, BTFIXUPs for CPU model specific "IPI
> operations" to avoid the switch-statement in the arch_ functions.

Exactly.

Re: [PATCH 1/2 v2] sparc32: implement SMP IPIs using the generic

[Daniel Hellstrom]

David Miller wrote:

>From: Daniel Hellstrom <daniel@gaisler.com>
>Date: Thu, 03 Feb 2011 09:50:05 +0100
>
>  
>
>>David Miller wrote:
>>
>>    
>>
>>>Ok, this looks mostly fine to me.
>>>
>>>Once we have the sun4m/sun4d versions written I can think about
>>>applying this.
>>>
>>>We might want to abstract out the common code of the dispatch and
>>>service stuff you have for LEON, otherwise we'll have 3 copies of
>>>that code that sets and clears the mask state.
>>>
>>>Perhaps something using BTFIXUP or similar.
>>> 
>>>      
>>>
>>I was thinking so too, BTFIXUPs for CPU model specific "IPI
>>operations" to avoid the switch-statement in the arch_ functions.
>>    
>>
>
>Exactly.
>  
>
I have made new IPI patches that uses BTFIXUPs as we discussed above.

 From another discussion we had:


David Miller wrote:

>I think we can make this work on sun4m/sun4c/sun4d, which have several
>software interrupt vectors available.
>
>sun4m has 15 soft interrupts, in the sun4m_irq_percpu->{pending,clear,set}
>registers, these live sequentially starting at bit 16, as per the definition
>of the SUN4M_SOFT_INT() macro.
>
>The only catch is that we'll need to peek at the ->pending register(s) to
>determine if we have a hardware or software interrupt pending at a given
>PIL level (or both).
>  
>
I have made an implementation for sun4m which I think will work, I added 
code to the IRQ traphandler to check for the IPIs as you suggested above 
and clearing the pending register. The previous sun4m code had some IPI 
checks already which I beleive was old unused code that was never 
called, so I removed that.. Do you have any ideas what was going on in 
the old code (smp_reschedule_irq)? Please see patch and commit message.

I don't have the sun4d-SMP hardware so it would be great if someone 
could test the patches?

>I'm not sure how the software interrupt triggering works on sun4c, but that
>doesn't matter since we only need this on SMP.
>  
>
I skip sun4c in this patch.

>The sun4d code has a sun4d_send_ipi() interface from which the necessary
>code can be constructed.
>  
>
I started making a sun4d patch however I don't know the hardware enough 
to make a guess. My problem is that I think I can generate an IPI but 
don't know how to ACK them, detect them, which IRQs to use etc. Is there 
a way to distinguish between soft and hard IRQs on sun4d? If I had a 
piece of document about sun4d it could be possible, but guessing from 
looking at unimplemented code... I hope someone else can take over here, 
probably it is very similar to the sun4m implementation.

Also, this time I added some IPI IRQ statistics to the genirq 
arch_show_interrupts() function, I hope that it may help someone trying 
to implement IPIs on sun4d.

After booting on a LEON4 dualcore board:

# cat /proc/interrupts
           CPU0       CPU1
  2:       2808          0      leon-edge      timer
  3:       2809       2741      leon-per-cpu   ticker
  5:        198          0      leon-edge      apbuart
 13:          0          0      leon-pcilvl    GRPCI2_JUMP
 20:          0          0    grpci2-pcilvl    GRPCI2_ERR
RES:         29          6      IPI rescheduling interrupts
CAL:         35         32      IPI function call interrupts
NMI:          1          1      Non-maskable interrupts


Daniel