SMP MIPS and Linux 3.2

SMP MIPS and Linux 3.2

[Mikael Starvik]

I'm running Linux 3.2 on a MIPS 34K with two VPEs (in MT_SMP configuration). It works fine in UP but with SMP it deadlocks during bootup (both CPUs gets idle). Typically like this:

[    0.090000] CPU revision is: 01019550 (MIPS 34Kc)
[    0.090000] Primary instruction cache 32kB, VIPT, 4-way, linesize 32 bytes.
[    0.090000] Primary data cache 32kB, 4-way, PIPT, no aliases, linesize 32 bytes
[    0.170000] Brought up 2 CPUs
<No more output>

I have tried to enable __ARCH_WANT_INTERRUPTS_ON_CTXSW but that didn't improve anything. Anyone else got this running or have any thoughts about what the problem may be?

Best Regards
/Mikael

Re: SMP MIPS and Linux 3.2

[tiejun.chen]

Mikael Starvik wrote:
> I'm running Linux 3.2 on a MIPS 34K with two VPEs (in MT_SMP configuration). It works fine in UP but with SMP it deadlocks during bootup (both CPUs gets idle). Typically like this:
> 
> [��� 0.090000] CPU revision is: 01019550 (MIPS 34Kc)
> [��� 0.090000] Primary instruction cache 32kB, VIPT, 4-way, linesize 32 bytes.
> [��� 0.090000] Primary data cache 32kB, 4-way, PIPT, no aliases, linesize 32 bytes
> [��� 0.170000] Brought up 2 CPUs
> <No more output>
> 
> I have tried to enable __ARCH_WANT_INTERRUPTS_ON_CTXSW but that didn't improve anything. Anyone else got this running or have any thoughts about what the problem may be?
> 

I think using git-bisect is the simplest way to figure out this if you already
know one kernel version is fine for mips 34kc.

Or did you try to pass 'nosmp' into the kernel command line? If good maybe
you're hitting some locking issues. You can enable those Kconfig options to
probe-to-debug these locking problem.

Tiejun

Re: SMP MIPS and Linux 3.2

[Ralf Baechle]

On Mon, Feb 20, 2012 at 10:34:20AM +0100, Mikael Starvik wrote:

> I'm running Linux 3.2 on a MIPS 34K with two VPEs (in MT_SMP configuration). It works fine in UP but with SMP it deadlocks during bootup (both CPUs gets idle). Typically like this:
> 
> [    0.090000] CPU revision is: 01019550 (MIPS 34Kc)
> [    0.090000] Primary instruction cache 32kB, VIPT, 4-way, linesize 32 bytes.
> [    0.090000] Primary data cache 32kB, 4-way, PIPT, no aliases, linesize 32 bytes
> [    0.170000] Brought up 2 CPUs
> <No more output>
> 
> I have tried to enable __ARCH_WANT_INTERRUPTS_ON_CTXSW but that didn't
> improve anything. Anyone else got this running or have any thoughts about
> what the problem may be?

It used to work ...  Are you testing this on Malta?  In my experience if a
CPU hangs at this stage it often is because it does not receive a timer
tick, so all changes to the timer code are candidates to be reviewed.
Git bisect may be the way of least resistance here.  

  Ralf

Re: SMP MIPS and Linux 3.2

[John Crispin]

Hi,

i am getting this. looks like ralf is right and the timer irqs get lost.

running with nosmp works like a charm.

John



CPU Clock: 333MHz
Calibrating delay loop... 221.18 BogoMIPS (lpj=442368)
pid_max: default: 32768 minimum: 301
Mount-cache hash table entries: 512
CPU revision is: 0001954c (MIPS 34Kc)
Primary instruction cache 32kB, VIPT, 4-way, linesize 32 bytes.
Primary data cache 16kB, 4-way, VIPT, no aliases, linesize 32 bytes
Brought up 2 CPUs
INFO: rcu_sched detected stall on CPU 0 (t=15000 jiffies)
INFO: rcu_sched detected stall on CPU 1 (t=15000 jiffies)
Call Trace:
[<802c310c>] dump_stack+0x8/0x34
[<8007e280>] print_cpu_stall+0x40/0xf0
[<8007e630>] __rcu_pending+0x110/0x270
[<8007fa44>] rcu_check_callbacks+0x78/0x180
[<80035234>] update_process_times+0x40/0x6c
[<80069acc>] tick_handle_periodic+0x38/0x170
[<8000c8f4>] c0_compare_interrupt+0x78/0xbc
[<80077740>] handle_irq_event_percpu+0x78/0x294
[<8007ba08>] handle_percpu_irq+0x8c/0xc0
[<80076dd0>] generic_handle_irq+0x40/0x50
[<80006650>] do_IRQ+0x18/0x28
[<80004aec>] ret_from_irq+0x0/0x4
[<8002e3c0>] __do_softirq+0x84/0x1ac
[<8002e744>] do_softirq+0x78/0x80
[<8002e9f4>] irq_exit+0x80/0x8c
[<80004aec>] ret_from_irq+0x0/0x4
[<80004ce0>] r4k_wait+0x20/0x40
[<800069d8>] cpu_idle+0x7c/0xa4

Call Trace:
[<802c310c>] dump_stack+0x8/0x34
[<8007e280>] print_cpu_stall+0x40/0xf0
[<8007e630>] __rcu_pending+0x110/0x270
[<8007fae4>] rcu_check_callbacks+0x118/0x180
[<80035234>] update_process_times+0x40/0x6c
[<80069acc>] tick_handle_periodic+0x38/0x170
[<8000c8f4>] c0_compare_interrupt+0x78/0xbc
[<80077740>] handle_irq_event_percpu+0x78/0x294
[<8007ba08>] handle_percpu_irq+0x8c/0xc0
[<80076dd0>] generic_handle_irq+0x40/0x50
[<80006650>] do_IRQ+0x18/0x28
[<80004aec>] ret_from_irq+0x0/0x4
[<8006fcf0>] generic_exec_single+0x98/0xfc
[<8006ff3c>] smp_call_function_single+0x1e8/0x1f0
[<800704e0>] smp_call_function+0x28/0x38
[<80070510>] on_each_cpu+0x20/0x64
[<800bbdac>] do_tune_cpucache+0x130/0x238
[<800bc09c>] enable_cpucache+0x60/0xf8
[<800bc4a0>] kmem_cache_create+0x36c/0x554
[<803705e0>] shmem_init+0x60/0xf8
[<80366998>] kernel_init+0x94/0x154
[<80006954>] kernel_thread_helper+0x10/0x18

Re: SMP MIPS and Linux 3.2

[JoeJ]

Mikael,

 Just wanted to know if you could bring-up 3.2 on mips34k core with SMVP
support enabled? Do you see any boot-up issues or memory related issues
here? I am trying to bring-up SMVP on 34k using 2.6.35 kernel and current
hit on issues during system boot-up (application load). Could you provide me
the following details w.r.t kernel configuration if possible. 
1. Is CONFIG_PREEMPT enabled ?
2. What is page size configured?
3. Do you have timer interrupts to both vpe0 & vpe1?
4. Is the kernel downloaded from kernel.org or mips repository. 
5. If you can share the kernel configuration file you are using, it will be
helpful. 

Also, any other bugs that you encountered during SMVP bring-up using 3.2
kernel?

Regards,
Joe. 




Tiejun Chen-2 wrote:
> 
> Mikael Starvik wrote:
>> I'm running Linux 3.2 on a MIPS 34K with two VPEs (in MT_SMP
>> configuration). It works fine in UP but with SMP it deadlocks during
>> bootup (both CPUs gets idle). Typically like this:
>> 
>> [��� 0.090000] CPU revision is: 01019550 (MIPS 34Kc)
>> [��� 0.090000] Primary instruction cache 32kB, VIPT, 4-way, linesize 32
>> bytes.
>> [��� 0.090000] Primary data cache 32kB, 4-way, PIPT, no aliases, linesize
>> 32 bytes
>> [��� 0.170000] Brought up 2 CPUs
>> <No more output>
>> 
>> I have tried to enable __ARCH_WANT_INTERRUPTS_ON_CTXSW but that didn't
>> improve anything. Anyone else got this running or have any thoughts about
>> what the problem may be?
>> 
> 
> I think using git-bisect is the simplest way to figure out this if you
> already
> know one kernel version is fine for mips 34kc.
> 
> Or did you try to pass 'nosmp' into the kernel command line? If good maybe
> you're hitting some locking issues. You can enable those Kconfig options
> to
> probe-to-debug these locking problem.
> 
> Tiejun
> 
> 
> 
> 

-- 
View this message in context: http://old.nabble.com/SMP-MIPS-and-Linux-3.2-tp33355620p33777466.html
Sent from the linux-mips main mailing list archive at Nabble.com.

RE: SMP MIPS and Linux 3.2

[Gandham, Raghu]

Hi Deng-Cheng, 

>Do you know why it didn't happen?

I must have forgotten to upload this patch along with other patches I submitted a while ago. I will verify that this patch is valid as is and submit it.

Raghu

>-----Original Message-----
>From: Deng-Cheng Zhu [mailto:dengcheng.zhu@gmail.com]
>Sent: Thursday, February 23, 2012 2:11 AM
>To: Mikael Starvik; Gandham, Raghu
>Cc: linux-mips@linux-mips.org
>Subject: Re: SMP MIPS and Linux 3.2
>
>I should have contacted the author (Raghu Gandham) of a fix for this
>issue to get it into the mainline. But it slipped out of my mind...
>
>The patch link is here:
>http://git.linux-mips.org/?p=linux-
>mti.git;a=commitdiff;h=5460815027d802697b879644c74f0e8365254020
>
>Hi, Raghu
>
>Do you know why it didn't happen?
>
>
>Deng-Cheng
>
>On Wed, Feb 22, 2012 at 6:57 PM, Mikael Starvik
><mikael.starvik@axis.com> wrote:
>>
>> Found it! There are no calls to scheduler_ipi() from the MIPS parts in
>vanilla 3.2.
>>
>> /Mikael
>>
>> -----Original Message-----
>> From: Mikael Starvik
>> Sent: den 20 februari 2012 10:34
>> To: 'linux-mips@linux-mips.org'
>> Subject: SMP MIPS and Linux 3.2
>>
>> I'm running Linux 3.2 on a MIPS 34K with two VPEs (in MT_SMP
>configuration). It works fine in UP but with SMP it deadlocks during
>bootup (both CPUs gets idle). Typically like this:
>>
>> [    0.090000] CPU revision is: 01019550 (MIPS 34Kc) [    0.090000]
>Primary instruction cache 32kB, VIPT, 4-way, linesize 32 bytes.
>> [    0.090000] Primary data cache 32kB, 4-way, PIPT, no aliases,
>linesize 32 bytes [    0.170000] Brought up 2 CPUs <No more output>
>>
>> I have tried to enable __ARCH_WANT_INTERRUPTS_ON_CTXSW but that didn't
>improve anything. Anyone else got this running or have any thoughts
>about what the problem may be?
>>
>> Best Regards
>> /Mikael
>>

Re: SMP MIPS and Linux 3.2

[John Crispin]

Hi,

the proposed patch will only fix this problem for SoCs with gic_present=1

i have just sent a patch [1] that makes it work on lantiq 34kc socs that
have gic_present=0

John

[1] http://www.linux-mips.org/archives/linux-mips/2012-02/msg00140.html



On 23/02/12 11:11, Deng-Cheng Zhu wrote:
> I should have contacted the author (Raghu Gandham) of a fix for this
> issue to get it into the mainline. But it slipped out of my mind...
> 
> The patch link is here:
> http://git.linux-mips.org/?p=linux-mti.git;a=commitdiff;h=5460815027d802697b879644c74f0e8365254020
> 
> Hi, Raghu
> 
> Do you know why it didn't happen?
> 
> 
> Deng-Cheng
> 
> On Wed, Feb 22, 2012 at 6:57 PM, Mikael Starvik <mikael.starvik@axis.com> wrote:
>>
>> Found it! There are no calls to scheduler_ipi() from the MIPS parts in vanilla 3.2.
>>
>> /Mikael
>>
>> -----Original Message-----
>> From: Mikael Starvik
>> Sent: den 20 februari 2012 10:34
>> To: 'linux-mips@linux-mips.org'
>> Subject: SMP MIPS and Linux 3.2
>>
>> I'm running Linux 3.2 on a MIPS 34K with two VPEs (in MT_SMP configuration). It works fine in UP but with SMP it deadlocks during bootup (both CPUs gets idle). Typically like this:
>>
>> [    0.090000] CPU revision is: 01019550 (MIPS 34Kc) [    0.090000] Primary instruction cache 32kB, VIPT, 4-way, linesize 32 bytes.
>> [    0.090000] Primary data cache 32kB, 4-way, PIPT, no aliases, linesize 32 bytes [    0.170000] Brought up 2 CPUs <No more output>
>>
>> I have tried to enable __ARCH_WANT_INTERRUPTS_ON_CTXSW but that didn't improve anything. Anyone else got this running or have any thoughts about what the problem may be?
>>
>> Best Regards
>> /Mikael
>>
> 
> 

Re: SMP MIPS and Linux 3.2

[Deng-Cheng Zhu]

I should have contacted the author (Raghu Gandham) of a fix for this
issue to get it into the mainline. But it slipped out of my mind...

The patch link is here:
http://git.linux-mips.org/?p=linux-mti.git;a=commitdiff;h=5460815027d802697b879644c74f0e8365254020

Hi, Raghu

Do you know why it didn't happen?


Deng-Cheng

On Wed, Feb 22, 2012 at 6:57 PM, Mikael Starvik <mikael.starvik@axis.com> wrote:
>
> Found it! There are no calls to scheduler_ipi() from the MIPS parts in vanilla 3.2.
>
> /Mikael
>
> -----Original Message-----
> From: Mikael Starvik
> Sent: den 20 februari 2012 10:34
> To: 'linux-mips@linux-mips.org'
> Subject: SMP MIPS and Linux 3.2
>
> I'm running Linux 3.2 on a MIPS 34K with two VPEs (in MT_SMP configuration). It works fine in UP but with SMP it deadlocks during bootup (both CPUs gets idle). Typically like this:
>
> [    0.090000] CPU revision is: 01019550 (MIPS 34Kc) [    0.090000] Primary instruction cache 32kB, VIPT, 4-way, linesize 32 bytes.
> [    0.090000] Primary data cache 32kB, 4-way, PIPT, no aliases, linesize 32 bytes [    0.170000] Brought up 2 CPUs <No more output>
>
> I have tried to enable __ARCH_WANT_INTERRUPTS_ON_CTXSW but that didn't improve anything. Anyone else got this running or have any thoughts about what the problem may be?
>
> Best Regards
> /Mikael
>

RE: SMP MIPS and Linux 3.2

[Mikael Starvik]

Found it! There are no calls to scheduler_ipi() from the MIPS parts in vanilla 3.2.

/Mikael

-----Original Message-----
From: Mikael Starvik 
Sent: den 20 februari 2012 10:34
To: 'linux-mips@linux-mips.org'
Subject: SMP MIPS and Linux 3.2

I'm running Linux 3.2 on a MIPS 34K with two VPEs (in MT_SMP configuration). It works fine in UP but with SMP it deadlocks during bootup (both CPUs gets idle). Typically like this:

[    0.090000] CPU revision is: 01019550 (MIPS 34Kc) [    0.090000] Primary instruction cache 32kB, VIPT, 4-way, linesize 32 bytes.
[    0.090000] Primary data cache 32kB, 4-way, PIPT, no aliases, linesize 32 bytes [    0.170000] Brought up 2 CPUs <No more output>

I have tried to enable __ARCH_WANT_INTERRUPTS_ON_CTXSW but that didn't improve anything. Anyone else got this running or have any thoughts about what the problem may be?

Best Regards
/Mikael

SMP MIPS and Linux 3.2

[Mikael Starvik]

I'm running Linux 3.2 on a MIPS 34K with two VPEs (in MT_SMP configuration). It works fine in UP but with SMP it deadlocks during bootup (both CPUs gets idle). Typically like this:

[    0.090000] CPU revision is: 01019550 (MIPS 34Kc)
[    0.090000] Primary instruction cache 32kB, VIPT, 4-way, linesize 32 bytes.
[    0.090000] Primary data cache 32kB, 4-way, PIPT, no aliases, linesize 32 bytes
[    0.170000] Brought up 2 CPUs
<No more output>

I have tried to enable __ARCH_WANT_INTERRUPTS_ON_CTXSW but that didn't improve anything. Anyone else got this running or have any thoughts about what the problem may be?

Best Regards
/Mikael