2.6.34 hangs during boot on PB11MPCore

2.6.34 hangs during boot on PB11MPCore

[Bjoern Brandenburg]

Hi all,

we're having a bit of trouble to get 2.6.34 to run on a PB11MPCore board with
SMP enabled. If we don't specify maxcpus=1, then the system gets stuck when
trying to launch init (via NFS).

We've tried three kernel versions:
- 2.6.33-arm1 [1]
  - boot log: http://pastebin.com/UqhfJNgz
- 2.6.34-vanilla
  - boot log with maxcpus=1: http://pastebin.com/NHHeYpbb
  - boot log with failure: http://pastebin.com/uy6y8msZ
  - config: http://pastebin.com/v9dttg0L
- 2.6.34-arm  [2]
  - boot log with maxcpus=1: http://pastebin.com/rCy2x9AP
  - boot log with failure: http://pastebin.com/spam.php?i=mpedVqp3

Of these, only 2.6.33-arm1 boots with more than one CPU enabled. The 2.6.34
versions only boot to login if we restrict the number of CPUs. The configuration
used in all cases is based on the one provided by ARM [3]. For 2.6.34, we used
'make oldconfig' and accepted all defaults.

Trying to boot 2.6.34-vanilla results in:
-----
[...]
Looking up port of RPC 100003/3 on 152.2.128.169
Looking up port of RPC 100005/3 on 152.2.128.169
VFS: Mounted root (nfs filesystem) on device 0:11.
Freeing init memory: 148K
BUG: soft lockup - CPU#0 stuck for 61s! [swapper:0]
Modules linked in:

Pid: 0, comm:              swapper
CPU: 0    Not tainted  (2.6.34 #2)
PC is at handle_IRQ_event+0x1c/0xf8
LR is at handle_level_irq+0xa4/0x150
pc : [<c0086084>]    lr : [<c0088130>]    psr: 40000113
sp : c0433e60  ip : 009564e3  fp : ffff8e11
r10: c0434080  r9 : c0888200  r8 : c0433f70
r7 : 00000029  r6 : 00000029  r5 : c043509c  r4 : dfdce980
r3 : 00000080  r2 : 00000000  r1 : dfdce980  r0 : 00000029
Flags: nZcv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment kernel
Control: 00c5787d  Table: 8fefc00a  DAC: 00000017
INFO: task init:1 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
init          D c03423c0     0     1      0 0x00000000
[<c03423c0>] (schedule+0x228/0x5f8) from [<c03427e4>] (io_schedule+0x54/0x84)
[<c03427e4>] (io_schedule+0x54/0x84) from [<c0097140>] (sync_page+0x50/0x5c)
[<c0097140>] (sync_page+0x50/0x5c) from [<c0342b8c>]
(__wait_on_bit_lock+0x6c/0xb8)
[<c0342b8c>] (__wait_on_bit_lock+0x6c/0xb8) from [<c00970c8>]
(__lock_page+0x94/0xac)
[<c00970c8>] (__lock_page+0x94/0xac) from [<c0097334>]
(find_lock_page+0x50/0x68)
[<c0097334>] (find_lock_page+0x50/0x68) from [<c0097a6c>]
(filemap_fault+0x1a0/0x420)
[<c0097a6c>] (filemap_fault+0x1a0/0x420) from [<c00ae48c>]
(__do_fault+0x54/0x49c)
[<c00ae48c>] (__do_fault+0x54/0x49c) from [<c00af79c>]
(handle_mm_fault+0x114/0x898)
[<c00af79c>] (handle_mm_fault+0x114/0x898) from [<c00381e4>]
(do_page_fault+0x208/0x300)
[<c00381e4>] (do_page_fault+0x208/0x300) from [<c002d45c>]
(do_PrefetchAbort+0x34/0x98)
[<c002d45c>] (do_PrefetchAbort+0x34/0x98) from [<c002e080>]
(ret_from_exception+0x0/0x10)
Exception stack(0xdfc3dfb0 to 0xdfc3dff8)
dfa0:                                     beee1bdd 0000007e 000b1b14 beee1c80
dfc0: beee1bdc 00000003 beee1c80 00000204 00000204 20400000 00000053 00000000
dfe0: 4014884c beee1bd8 0008976c 4014884c 60000010 ffffffff
INFO: task events/3:18 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
events/3      D c03423c0     0    18      2 0x00000000
[<c03423c0>] (schedule+0x228/0x5f8) from [<c0342a68>]
(schedule_timeout+0x168/0x1cc)
[<c0342a68>] (schedule_timeout+0x168/0x1cc) from [<c0342054>]
(wait_for_common+0xb8/0x170)
[<c0342054>] (wait_for_common+0xb8/0x170) from [<c008a2c0>]
(synchronize_sched+0x50/0x64)
[<c008a2c0>] (synchronize_sched+0x50/0x64) from [<c02ccb7c>]
(dev_deactivate+0x168/0x1e8)
[<c02ccb7c>] (dev_deactivate+0x168/0x1e8) from [<c02c9194>]
(linkwatch_do_dev+0x9c/0xf4)
[<c02c9194>] (linkwatch_do_dev+0x9c/0xf4) from [<c02c9608>]
(__linkwatch_run_queue+0x1bc/0x1fc)
[<c02c9608>] (__linkwatch_run_queue+0x1bc/0x1fc) from [<c02c966c>]
(linkwatch_event+0x24/0x34)
[<c02c966c>] (linkwatch_event+0x24/0x34) from [<c0061e2c>]
(worker_thread+0x14c/0x1f0)
[<c0061e2c>] (worker_thread+0x14c/0x1f0) from [<c0065d8c>] (kthread+0x80/0x88)
[<c0065d8c>] (kthread+0x80/0x88) from [<c002f390>] (kernel_thread_exit+0x0/0x8)
-----

Trying to boot 2.6.34-arm results in:
-----
[...]
VFS: Mounted root (nfs filesystem) on device 0:12.
Freeing init memory: 160K
nfs: server 152.2.128.169 not responding, still trying
nfs: server 152.2.128.169 not responding, still trying
-----
Nothing happens after this for a while, so I gave up. I was using 152.2.128.169
at the same time via ssh, so I'm pretty sure that it was responding just fine.

Are these known problem? Are there additional patches that we should try to get
2.6.34 to run with SMP enabled?

Thanks,
Bj?rn


[1]
http://linux-arm.org/git?p=linux-2.6-stable.git;a=commit;h=
729b7925dd8b9f6e5f4e4466408f91a2b9f51c3a
[2]
http://linux-arm.org/git?p=linux-2.6.git;a=commit;h=
9bc7891fd8946a6de950902f25b870e04144add2
[3]
http://linux-arm.org/git?p=ael.git;a=blob;f=kernel/config/2.6.33-arm1/config-2.6
.33-arm1-realview-v6-smp;h=7637d63580ab87ed25f8102b874d3e3e2fae1322;hb=2010q2

2.6.34 hangs during boot on PB11MPCore

[Daniel Mack]

On Sat, May 29, 2010 at 06:02:28PM -0400, Bjoern Brandenburg wrote:
> We've tried three kernel versions:
> - 2.6.33-arm1 [1]
>   - boot log: http://pastebin.com/UqhfJNgz
> - 2.6.34-vanilla
>   - boot log with maxcpus=1: http://pastebin.com/NHHeYpbb
>   - boot log with failure: http://pastebin.com/uy6y8msZ
>   - config: http://pastebin.com/v9dttg0L
> - 2.6.34-arm  [2]
>   - boot log with maxcpus=1: http://pastebin.com/rCy2x9AP
>   - boot log with failure: http://pastebin.com/spam.php?i=mpedVqp3
> 
> Of these, only 2.6.33-arm1 boots with more than one CPU enabled. The 2.6.34
> versions only boot to login if we restrict the number of CPUs. The configuration
> used in all cases is based on the one provided by ARM [3]. For 2.6.34, we used
> 'make oldconfig' and accepted all defaults.

This sounds like a perfect case for giving 'git bisect' a try. It will
point you to the commit between 2.6.33 and 2.6.34 which broke it for
you.

See for example http://www.landley.net/writing/git-quick.html#bisect for
a detailed explanation of how to use it.


Daniel

2.6.34 hangs during boot on PB11MPCore

[Linus Walleij]

2010/5/30 Bjoern Brandenburg <bbb.lst@gmail.com>:

> we're having a bit of trouble to get 2.6.34 to run on a PB11MPCore board with
> SMP enabled. If we don't specify maxcpus=1, then the system gets stuck when
> trying to launch init (via NFS).

I had the same problem earlier on the -next tree. However I thought it was
all my fault for some reason and used the defconfig with SMP disabled.
Now I can't access the machine for the moment :-/

Yours,
Linus Walleij

2.6.34 hangs during boot on PB11MPCore

[Bjoern Brandenburg]

On Sun, May 30, 2010 at 3:36 AM, Daniel Mack <daniel@caiaq.de> wrote:
> On Sat, May 29, 2010 at 06:02:28PM -0400, Bjoern Brandenburg wrote:
>> We've tried three kernel versions:
>> - 2.6.33-arm1 [1]
>> ? - boot log: http://pastebin.com/UqhfJNgz
>> - 2.6.34-vanilla
>> ? - boot log with maxcpus=1: http://pastebin.com/NHHeYpbb
>> ? - boot log with failure: http://pastebin.com/uy6y8msZ
>> ? - config: http://pastebin.com/v9dttg0L
>> - 2.6.34-arm ?[2]
>> ? - boot log with maxcpus=1: http://pastebin.com/rCy2x9AP
>> ? - boot log with failure: http://pastebin.com/spam.php?i=mpedVqp3
>>
>> Of these, only 2.6.33-arm1 boots with more than one CPU enabled. The 2.6.34
>> versions only boot to login if we restrict the number of CPUs. The configuration
>> used in all cases is based on the one provided by ARM [3]. For 2.6.34, we used
>> 'make oldconfig' and accepted all defaults.
>
> This sounds like a perfect case for giving 'git bisect' a try. It will
> point you to the commit between 2.6.33 and 2.6.34 which broke it for
> you.
>
> See for example http://www.landley.net/writing/git-quick.html#bisect for
> a detailed explanation of how to use it.

Thanks. I gave it a shot.  Unfortunately, 2.6.33 vanilla doesn't work
either, and their is no common history between 2.6.33-arm1 and 2.6.34
(either vanilla or -arm). Thus, I don't have a good commit to start
from.

Basically, as far as I can tell, no vanilla kernel has ever worked
with SMP enabled on the  PB11MPCore, but 2.6.28 and 2.6.33 worked with
vendor patches applied. Some of these patches were picked up for
2.6.34, but even with those it still doesn't work. Worse, now even the
vendor patches provided at http://linux-arm.org/git?p=linux-2.6.git
don't work anymore, either. I realize that ARM's 2.6.34 repository is
still in a state of flux, thus my question whether this is a known
issue.

I'll try to see if I can pinpoint what was dropped between 2.6.33-arm1
and 2.6.34-arm.

Thanks,
Bj?rn

2.6.34 hangs during boot on PB11MPCore

[Bjoern Brandenburg]

On Sun, May 30, 2010 at 3:27 PM, Bjoern Brandenburg <bbb.lst@gmail.com> wrote:
> On Sun, May 30, 2010 at 3:36 AM, Daniel Mack <daniel@caiaq.de> wrote:
>> On Sat, May 29, 2010 at 06:02:28PM -0400, Bjoern Brandenburg wrote:
>>> We've tried three kernel versions:
>>> - 2.6.33-arm1 [1]
>>> ? - boot log: http://pastebin.com/UqhfJNgz
>>> - 2.6.34-vanilla
>>> ? - boot log with maxcpus=1: http://pastebin.com/NHHeYpbb
>>> ? - boot log with failure: http://pastebin.com/uy6y8msZ
>>> ? - config: http://pastebin.com/v9dttg0L
>>> - 2.6.34-arm ?[2]
>>> ? - boot log with maxcpus=1: http://pastebin.com/rCy2x9AP
>>> ? - boot log with failure: http://pastebin.com/spam.php?i=mpedVqp3
>>>
>>> Of these, only 2.6.33-arm1 boots with more than one CPU enabled. The 2.6.34
>>> versions only boot to login if we restrict the number of CPUs. The configuration
>>> used in all cases is based on the one provided by ARM [3]. For 2.6.34, we used
>>> 'make oldconfig' and accepted all defaults.
>>
>> This sounds like a perfect case for giving 'git bisect' a try. It will
>> point you to the commit between 2.6.33 and 2.6.34 which broke it for
>> you.
>>
>> See for example http://www.landley.net/writing/git-quick.html#bisect for
>> a detailed explanation of how to use it.
>
> Thanks. I gave it a shot. ?Unfortunately, 2.6.33 vanilla doesn't work
> either, and their is no common history between 2.6.33-arm1 and 2.6.34
> (either vanilla or -arm). Thus, I don't have a good commit to start
> from.
>
> Basically, as far as I can tell, no vanilla kernel has ever worked
> with SMP enabled on the ?PB11MPCore, but 2.6.28 and 2.6.33 worked with
> vendor patches applied. Some of these patches were picked up for
> 2.6.34, but even with those it still doesn't work. Worse, now even the
> vendor patches provided at http://linux-arm.org/git?p=linux-2.6.git
> don't work anymore, either. I realize that ARM's 2.6.34 repository is
> still in a state of flux, thus my question whether this is a known
> issue.
>
> I'll try to see if I can pinpoint what was dropped between 2.6.33-arm1
> and 2.6.34-arm.

Progress: I can get 2.6.34-arm to boot with all 4 CPUs after
cherry-picking the following commits (which seemed relevant but
absent):

60060ca ARM: Handle instruction cache maintenance fault  properly
3f64e83 ARM errata: Eviction Buffer not empty after Cache Sync on L220
3b009b5 ARM: change definition of cpu_relax() for ARM11MPCore

Let's see which is the critical one...

- Bj?rn

2.6.34 hangs during boot on PB11MPCore

[Bjoern Brandenburg]

On Sun, May 30, 2010 at 5:05 PM, Bjoern Brandenburg <bbb.lst@gmail.com> wrote:
> On Sun, May 30, 2010 at 3:27 PM, Bjoern Brandenburg <bbb.lst@gmail.com> wrote:
>>
>> I'll try to see if I can pinpoint what was dropped between 2.6.33-arm1
>> and 2.6.34-arm.
>
> Progress: I can get 2.6.34-arm to boot with all 4 CPUs after
> cherry-picking the following commits (which seemed relevant but
> absent):
>
> 60060ca ARM: Handle instruction cache maintenance fault ?properly
> 3f64e83 ARM errata: Eviction Buffer not empty after Cache Sync on L220
> 3b009b5 ARM: change definition of cpu_relax() for ARM11MPCore
>
> Let's see which is the critical one...

It's 3f64e83 "ARM errata: Eviction Buffer not empty after Cache Sync
on L220" [1]. With this commit cherry-picked (on top of the 'rebased'
branch in ARM's repository, i.e., 2.6.34-arm), the system boots to X11
and runs some simple FS tests; the other ones don't make a difference.

Are there plans for getting this and the other patches in the
'rebased' branch into mainline (for .35 or .36)?

- Bj?rn

[1]  http://linux-arm.org/git?p=linux-2.6-stable.git;a=commitdiff;h=3f64e83e9935106acd3264a350260d74c618f263

2.6.34 hangs during boot on PB11MPCore

[Catalin Marinas]

On Sun, 2010-05-30 at 22:38 +0100, Bjoern Brandenburg wrote:
> On Sun, May 30, 2010 at 5:05 PM, Bjoern Brandenburg <bbb.lst@gmail.com> wrote:
> > On Sun, May 30, 2010 at 3:27 PM, Bjoern Brandenburg <bbb.lst@gmail.com> wrote:
> >>
> >> I'll try to see if I can pinpoint what was dropped between 2.6.33-arm1
> >> and 2.6.34-arm.
> >
> > Progress: I can get 2.6.34-arm to boot with all 4 CPUs after
> > cherry-picking the following commits (which seemed relevant but
> > absent):
> >
> > 60060ca ARM: Handle instruction cache maintenance fault  properly
> > 3f64e83 ARM errata: Eviction Buffer not empty after Cache Sync on L220
> > 3b009b5 ARM: change definition of cpu_relax() for ARM11MPCore
> >
> > Let's see which is the critical one...
> 
> It's 3f64e83 "ARM errata: Eviction Buffer not empty after Cache Sync
> on L220" [1]. With this commit cherry-picked (on top of the 'rebased'
> branch in ARM's repository, i.e., 2.6.34-arm), the system boots to X11
> and runs some simple FS tests; the other ones don't make a difference.
> 
> Are there plans for getting this and the other patches in the
> 'rebased' branch into mainline (for .35 or .36)?

Thanks for the investigation. I recall I got something similar in the
past though I could no longer reproduce it with 2.6.34 (-arm) on the
PB11MPCore I have. Could you try reverting commit e7c5650f606 (ARM:
Change the mandatory barriers implementation) on a vanilla 2.6.34
kernel?

I asked for clarification from hardware people here in ARM and the above
errata workaround doesn't seem apply to the L220 revision on the
PB11MPCore board (I need to reconfirm).

Anyway, some L220 revisions have an issue with a DSB followed by a cache
sync leading to hardware deadlock (this sequence was introduced in Linux
by the above commit). We could push the L220 erratum workaround (484863)
though the workaround I have implemented in the above commit is no
longer recommended in the errata document since it may cause other
problems with some L220 revisions.

-- 
Catalin

2.6.34 hangs during boot on PB11MPCore

[Bjoern Brandenburg]

On Sun, May 30, 2010 at 6:46 PM, Catalin Marinas
<catalin.marinas@arm.com> wrote:
> On Sun, 2010-05-30 at 22:38 +0100, Bjoern Brandenburg wrote:
>> On Sun, May 30, 2010 at 5:05 PM, Bjoern Brandenburg <bbb.lst@gmail.com> wrote:
>> > On Sun, May 30, 2010 at 3:27 PM, Bjoern Brandenburg <bbb.lst@gmail.com> wrote:
>> >>
>> >> I'll try to see if I can pinpoint what was dropped between 2.6.33-arm1
>> >> and 2.6.34-arm.
>> >
>> > Progress: I can get 2.6.34-arm to boot with all 4 CPUs after
>> > cherry-picking the following commits (which seemed relevant but
>> > absent):
>> >
>> > 60060ca ARM: Handle instruction cache maintenance fault ?properly
>> > 3f64e83 ARM errata: Eviction Buffer not empty after Cache Sync on L220
>> > 3b009b5 ARM: change definition of cpu_relax() for ARM11MPCore
>> >
>> > Let's see which is the critical one...
>>
>> It's 3f64e83 "ARM errata: Eviction Buffer not empty after Cache Sync
>> on L220" [1]. With this commit cherry-picked (on top of the 'rebased'
>> branch in ARM's repository, i.e., 2.6.34-arm), the system boots to X11
>> and runs some simple FS tests; the other ones don't make a difference.
>>
>> Are there plans for getting this and the other patches in the
>> 'rebased' branch into mainline (for .35 or .36)?
>
> Thanks for the investigation. I recall I got something similar in the
> past though I could no longer reproduce it with 2.6.34 (-arm) on the
> PB11MPCore I have. Could you try reverting commit e7c5650f606 (ARM:
> Change the mandatory barriers implementation) on a vanilla 2.6.34
> kernel?

That doesn't seem to help. v2.6.34 with e7c5650f606 reverted still hangs.

Looking up port of RPC 100003/3 on 152.2.128.169
Looking up port of RPC 100005/3 on 152.2.128.169
VFS: Mounted root (nfs filesystem) on device 0:11.
Freeing init memory: 148K
nfs: server 152.2.128.169 not responding, still trying
nfs: server 152.2.128.169 not responding, still trying
nfs: server 152.2.128.169 not responding, still trying
nfs: server 152.2.128.169 not responding, still trying
nfs: server 152.2.128.169 not responding, still trying
nfs: server 152.2.128.169 not responding, still trying
nfs: server 152.2.128.169 not responding, still trying
nfs: server 152.2.128.169 not responding, still trying
nfs: server 152.2.128.169 not responding, still trying
INFO: task init:1 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
init          D c03429a0     0     1      0 0x00000000
[<c03429a0>] (schedule+0x228/0x5f8) from [<c0342dc4>] (io_schedule+0x54/0x84)
[<c0342dc4>] (io_schedule+0x54/0x84) from [<c0098b1c>] (sync_page+0x50/0x5c)
[<c0098b1c>] (sync_page+0x50/0x5c) from [<c034316c>]
(__wait_on_bit_lock+0x6c/0xb8)
[<c034316c>] (__wait_on_bit_lock+0x6c/0xb8) from [<c0098aa4>]
(__lock_page+0x94/0xac)
[<c0098aa4>] (__lock_page+0x94/0xac) from [<c0098ccc>]
(find_lock_page+0x50/0x68)
[<c0098ccc>] (find_lock_page+0x50/0x68) from [<c0099404>]
(filemap_fault+0x1a0/0x420)
[<c0099404>] (filemap_fault+0x1a0/0x420) from [<c00afde8>]
(__do_fault+0x54/0x49c)
[<c00afde8>] (__do_fault+0x54/0x49c) from [<c00b10f8>]
(handle_mm_fault+0x114/0x898)
[<c00b10f8>] (handle_mm_fault+0x114/0x898) from [<c00381e4>]
(do_page_fault+0x208/0x300)
[<c00381e4>] (do_page_fault+0x208/0x300) from [<c002d4f4>]
(do_DataAbort+0x34/0x98)
[<c002d4f4>] (do_DataAbort+0x34/0x98) from [<c002e080>]
(ret_from_exception+0x0/0x10)
Exception stack(0xdfc3dfb0 to 0xdfc3dff8)
dfa0:                                     400f5776 400e4000 001406e8 4001ef60
dfc0: 400e9260 4001eb80 400f8ea8 00000002 0000017a 40027548 402246e8 bed80b24
dfe0: 00008002 bed80a68 40003cfc 4000bba8 20000010 ffffffff
INFO: task init:1 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
init          D c03429a0     0     1      0 0x00000000
[<c03429a0>] (schedule+0x228/0x5f8) from [<c0342dc4>] (io_schedule+0x54/0x84)
[<c0342dc4>] (io_schedule+0x54/0x84) from [<c0098b1c>] (sync_page+0x50/0x5c)
[<c0098b1c>] (sync_page+0x50/0x5c) from [<c034316c>]
(__wait_on_bit_lock+0x6c/0xb8)
[<c034316c>] (__wait_on_bit_lock+0x6c/0xb8) from [<c0098aa4>]
(__lock_page+0x94/0xac)
[<c0098aa4>] (__lock_page+0x94/0xac) from [<c0098ccc>]
(find_lock_page+0x50/0x68)
[<c0098ccc>] (find_lock_page+0x50/0x68) from [<c0099404>]
(filemap_fault+0x1a0/0x420)
[<c0099404>] (filemap_fault+0x1a0/0x420) from [<c00afde8>]
(__do_fault+0x54/0x49c)
[<c00afde8>] (__do_fault+0x54/0x49c) from [<c00b10f8>]
(handle_mm_fault+0x114/0x898)
[<c00b10f8>] (handle_mm_fault+0x114/0x898) from [<c00381e4>]
(do_page_fault+0x208/0x300)
[<c00381e4>] (do_page_fault+0x208/0x300) from [<c002d4f4>]
(do_DataAbort+0x34/0x98)
[<c002d4f4>] (do_DataAbort+0x34/0x98) from [<c002e080>]
(ret_from_exception+0x0/0x10)
Exception stack(0xdfc3dfb0 to 0xdfc3dff8)
dfa0:                                     400f5776 400e4000 001406e8 4001ef60
dfc0: 400e9260 4001eb80 400f8ea8 00000002 0000017a 40027548 402246e8 bed80b24
dfe0: 00008002 bed80a68 40003cfc 4000bba8 20000010 ffffffff

2.6.35-rc1 also doesn't boot.

Anything else that I should try? I'd be happy to help out with testing
patches; it'd be nice to have our box working with a vanilla kernel in
the future.

Thanks,
Bj?rn

2.6.34 hangs during boot on PB11MPCore

[Bjoern Brandenburg]

On Sun, May 30, 2010 at 9:04 PM, Bjoern Brandenburg <bbb.lst@gmail.com> wrote:
> On Sun, May 30, 2010 at 6:46 PM, Catalin Marinas
> <catalin.marinas@arm.com> wrote:
>> On Sun, 2010-05-30 at 22:38 +0100, Bjoern Brandenburg wrote:
>>> On Sun, May 30, 2010 at 5:05 PM, Bjoern Brandenburg <bbb.lst@gmail.com> wrote:
>>> > On Sun, May 30, 2010 at 3:27 PM, Bjoern Brandenburg <bbb.lst@gmail.com> wrote:
>>> >>
>>> >> I'll try to see if I can pinpoint what was dropped between 2.6.33-arm1
>>> >> and 2.6.34-arm.
>>> >
>>> > Progress: I can get 2.6.34-arm to boot with all 4 CPUs after
>>> > cherry-picking the following commits (which seemed relevant but
>>> > absent):
>>> >
>>> > 60060ca ARM: Handle instruction cache maintenance fault ?properly
>>> > 3f64e83 ARM errata: Eviction Buffer not empty after Cache Sync on L220
>>> > 3b009b5 ARM: change definition of cpu_relax() for ARM11MPCore
>>> >
>>> > Let's see which is the critical one...
>>>
>>> It's 3f64e83 "ARM errata: Eviction Buffer not empty after Cache Sync
>>> on L220" [1]. With this commit cherry-picked (on top of the 'rebased'
>>> branch in ARM's repository, i.e., 2.6.34-arm), the system boots to X11
>>> and runs some simple FS tests; the other ones don't make a difference.
>>>
>>> Are there plans for getting this and the other patches in the
>>> 'rebased' branch into mainline (for .35 or .36)?
>>
>> Thanks for the investigation. I recall I got something similar in the
>> past though I could no longer reproduce it with 2.6.34 (-arm) on the
>> PB11MPCore I have. Could you try reverting commit e7c5650f606 (ARM:
>> Change the mandatory barriers implementation) on a vanilla 2.6.34
>> kernel?
>
> That doesn't seem to help. v2.6.34 with e7c5650f606 reverted still hangs.

Given that 2.6.34-arm contains patches for missing synchronization
primitives in the network driver, it is maybe not too surprising that
the vanilla kernel won't boot with an NFS root file system.

I just tried 2.6.34-arm (i.e., the 'rebased' branch in the ARM repo)
with e7c5650f606 reverted (and without cherry-picking 3f64e83)   and
that kernel boots ok!

- Bj?rn

2.6.34 hangs during boot on PB11MPCore

[Shilimkar, Santosh]

> -----Original Message-----
> From: linux-arm-kernel-bounces at lists.infradead.org [mailto:linux-arm-kernel-
> bounces at lists.infradead.org] On Behalf Of Bjoern Brandenburg
> Sent: Monday, May 31, 2010 6:46 AM
> To: Catalin Marinas
> Cc: bastoni at cs.unc.edu; Daniel Mack; linux-arm-kernel at lists.infradead.org; linus.ml.walleij at gmail.com
> Subject: Re: 2.6.34 hangs during boot on PB11MPCore
> 
> On Sun, May 30, 2010 at 9:04 PM, Bjoern Brandenburg <bbb.lst@gmail.com> wrote:
> > On Sun, May 30, 2010 at 6:46 PM, Catalin Marinas
> > <catalin.marinas@arm.com> wrote:
> >> On Sun, 2010-05-30 at 22:38 +0100, Bjoern Brandenburg wrote:
> >>> On Sun, May 30, 2010 at 5:05 PM, Bjoern Brandenburg <bbb.lst@gmail.com> wrote:
> >>> > On Sun, May 30, 2010 at 3:27 PM, Bjoern Brandenburg <bbb.lst@gmail.com> wrote:
> >>> >>
> >>> >> I'll try to see if I can pinpoint what was dropped between 2.6.33-arm1
> >>> >> and 2.6.34-arm.
> >>> >
> >>> > Progress: I can get 2.6.34-arm to boot with all 4 CPUs after
> >>> > cherry-picking the following commits (which seemed relevant but
> >>> > absent):
> >>> >
> >>> > 60060ca ARM: Handle instruction cache maintenance fault ?properly
> >>> > 3f64e83 ARM errata: Eviction Buffer not empty after Cache Sync on L220
> >>> > 3b009b5 ARM: change definition of cpu_relax() for ARM11MPCore
> >>> >
> >>> > Let's see which is the critical one...
> >>>
> >>> It's 3f64e83 "ARM errata: Eviction Buffer not empty after Cache Sync
> >>> on L220" [1]. With this commit cherry-picked (on top of the 'rebased'
> >>> branch in ARM's repository, i.e., 2.6.34-arm), the system boots to X11
> >>> and runs some simple FS tests; the other ones don't make a difference.
> >>>
> >>> Are there plans for getting this and the other patches in the
> >>> 'rebased' branch into mainline (for .35 or .36)?
> >>
> >> Thanks for the investigation. I recall I got something similar in the
> >> past though I could no longer reproduce it with 2.6.34 (-arm) on the
> >> PB11MPCore I have. Could you try reverting commit e7c5650f606 (ARM:
> >> Change the mandatory barriers implementation) on a vanilla 2.6.34
> >> kernel?
> >
> > That doesn't seem to help. v2.6.34 with e7c5650f606 reverted still hangs.
> 
> Given that 2.6.34-arm contains patches for missing synchronization
> primitives in the network driver, it is maybe not too surprising that
> the vanilla kernel won't boot with an NFS root file system.
> 
> I just tried 2.6.34-arm (i.e., the 'rebased' branch in the ARM repo)
> with e7c5650f606 reverted (and without cherry-picking 3f64e83)   and
> that kernel boots ok!
> 
We have similar observation on SPI based network driver. By playing with
buffer alignment, the issue seems to be going away. Do you have link
to " synchronization primitives in the network driver"  patches ??

Regards,
Santosh

2.6.34 hangs during boot on PB11MPCore

[Catalin Marinas]

On Mon, 2010-05-31 at 02:16 +0100, Bjoern Brandenburg wrote:
> On Sun, May 30, 2010 at 9:04 PM, Bjoern Brandenburg <bbb.lst@gmail.com> wrote:
> > On Sun, May 30, 2010 at 6:46 PM, Catalin Marinas
> > <catalin.marinas@arm.com> wrote:
> >> On Sun, 2010-05-30 at 22:38 +0100, Bjoern Brandenburg wrote:
> >>> On Sun, May 30, 2010 at 5:05 PM, Bjoern Brandenburg <bbb.lst@gmail.com> wrote:
> >>> > On Sun, May 30, 2010 at 3:27 PM, Bjoern Brandenburg <bbb.lst@gmail.com> wrote:
> >>> >>
> >>> >> I'll try to see if I can pinpoint what was dropped between 2.6.33-arm1
> >>> >> and 2.6.34-arm.
> >>> >
> >>> > Progress: I can get 2.6.34-arm to boot with all 4 CPUs after
> >>> > cherry-picking the following commits (which seemed relevant but
> >>> > absent):
> >>> >
> >>> > 60060ca ARM: Handle instruction cache maintenance fault  properly
> >>> > 3f64e83 ARM errata: Eviction Buffer not empty after Cache Sync on L220
> >>> > 3b009b5 ARM: change definition of cpu_relax() for ARM11MPCore
> >>> >
> >>> > Let's see which is the critical one...
> >>>
> >>> It's 3f64e83 "ARM errata: Eviction Buffer not empty after Cache Sync
> >>> on L220" [1]. With this commit cherry-picked (on top of the 'rebased'
> >>> branch in ARM's repository, i.e., 2.6.34-arm), the system boots to X11
> >>> and runs some simple FS tests; the other ones don't make a difference.
> >>>
> >>> Are there plans for getting this and the other patches in the
> >>> 'rebased' branch into mainline (for .35 or .36)?
> >>
> >> Thanks for the investigation. I recall I got something similar in the
> >> past though I could no longer reproduce it with 2.6.34 (-arm) on the
> >> PB11MPCore I have. Could you try reverting commit e7c5650f606 (ARM:
> >> Change the mandatory barriers implementation) on a vanilla 2.6.34
> >> kernel?
> >
> > That doesn't seem to help. v2.6.34 with e7c5650f606 reverted still hangs.
> 
> Given that 2.6.34-arm contains patches for missing synchronization
> primitives in the network driver, it is maybe not too surprising that
> the vanilla kernel won't boot with an NFS root file system.
> 
> I just tried 2.6.34-arm (i.e., the 'rebased' branch in the ARM repo)
> with e7c5650f606 reverted (and without cherry-picking 3f64e83)   and
> that kernel boots ok!

OK. So the L220 erratum I had in mind was 425331 (Potential deadlock in
certain system configurations if Write Buffer not empty when specific
maintenance operations are received) though doesn't seem to apply to
this revision of the L220. I'll try to get more information tomorrow and
maybe make the e7c5650f606 commit not apply to ARMv6 SMP systems (though
an L220 workaround would be better).

Regarding the SMSC 911x driver synchronisation - I've been asking SMSC
for more than a year to look into this. It fails on SMP systems without
my patch apply but SMSC states that such patch isn't needed. I even
consider reverting the RealView move to the new 911x driver. Something
like below (and make sure you enable the other smc911x.c driver):


Revert "arm: convert realview platform to use smsc911x"

From: Catalin Marinas <catalin.marinas@arm.com>

This reverts commit c5142e84f3a39c6396ce2ddde01b1e420a67464a.

Signed-off-by: Steve Glendinning <steve.glendinning@smsc.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
---
 arch/arm/mach-realview/core.c |   17 ++++++++---------
 1 files changed, 8 insertions(+), 9 deletions(-)

diff --git a/arch/arm/mach-realview/core.c b/arch/arm/mach-realview/core.c
index d5a9573..98fcacd 100644
--- a/arch/arm/mach-realview/core.c
+++ b/arch/arm/mach-realview/core.c
@@ -28,7 +28,7 @@
 #include <linux/clocksource.h>
 #include <linux/clockchips.h>
 #include <linux/io.h>
-#include <linux/smsc911x.h>
+#include <linux/smc911x.h>
 #include <linux/ata_platform.h>
 #include <linux/amba/mmci.h>
 #include <linux/gfp.h>
@@ -151,15 +151,14 @@ int realview_flash_register(struct resource *res, u32 num)
 	return platform_device_register(&realview_flash_device);
 }
 
-static struct smsc911x_platform_config smsc911x_config = {
-	.flags		= SMSC911X_USE_32BIT,
-	.irq_polarity	= SMSC911X_IRQ_POLARITY_ACTIVE_HIGH,
-	.irq_type	= SMSC911X_IRQ_TYPE_PUSH_PULL,
-	.phy_interface	= PHY_INTERFACE_MODE_MII,
+static struct smc911x_platdata realview_smc911x_platdata = {
+	.flags		= SMC911X_USE_32BIT,
+	.irq_flags	= IRQF_SHARED,
+	.irq_polarity	= 1,
 };
 
 static struct platform_device realview_eth_device = {
-	.name		= "smsc911x",
+	.name		= "smc911x",
 	.id		= 0,
 	.num_resources	= 2,
 };
@@ -169,8 +168,8 @@ int realview_eth_register(const char *name, struct resource *res)
 	if (name)
 		realview_eth_device.name = name;
 	realview_eth_device.resource = res;
-	if (strcmp(realview_eth_device.name, "smsc911x") == 0)
-		realview_eth_device.dev.platform_data = &smsc911x_config;
+	if (strcmp(realview_eth_device.name, "smc911x") == 0)
+		realview_eth_device.dev.platform_data = &realview_smc911x_platdata;
 
 	return platform_device_register(&realview_eth_device);
 }



-- 
Catalin

2.6.34 hangs during boot on PB11MPCore

[Colin Tuckley]

> -----Original Message-----
> From: linux-arm-kernel-bounces at lists.infradead.org [mailto:linux-arm-
> kernel-bounces at lists.infradead.org] On Behalf Of Catalin Marinas

> Regarding the SMSC 911x driver synchronisation - I've been asking SMSC
> for more than a year to look into this.

I'm still waiting for them to take up our offer of a PB11MPCore system for testing.

One thing Bjoern hasn't mentioned... is the board firmware up to date? I.e. has the fpga image been updated to the latest release (from the latest CD).

Colin

--
Colin Tuckley - ARM Ltd.
110 Fulbourn Rd
Cambridge, CB1 9NJ
Tel: +44 1223 400536

-- IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium.  Thank you.

2.6.34 hangs during boot on PB11MPCore

[Bjoern Brandenburg]

On Mon, May 31, 2010 at 2:26 AM, Shilimkar, Santosh
<santosh.shilimkar@ti.com> wrote:
>> -----Original Message-----
>> From: linux-arm-kernel-bounces at lists.infradead.org [mailto:linux-arm-kernel-
>> bounces at lists.infradead.org] On Behalf Of Bjoern Brandenburg
>> Sent: Monday, May 31, 2010 6:46 AM
>> To: Catalin Marinas
>> Cc: bastoni at cs.unc.edu; Daniel Mack; linux-arm-kernel at lists.infradead.org; linus.ml.walleij at gmail.com
>> Subject: Re: 2.6.34 hangs during boot on PB11MPCore
>>
>> On Sun, May 30, 2010 at 9:04 PM, Bjoern Brandenburg <bbb.lst@gmail.com> wrote:
>> > On Sun, May 30, 2010 at 6:46 PM, Catalin Marinas
>> > <catalin.marinas@arm.com> wrote:
>> >> On Sun, 2010-05-30 at 22:38 +0100, Bjoern Brandenburg wrote:
>> >>> On Sun, May 30, 2010 at 5:05 PM, Bjoern Brandenburg <bbb.lst@gmail.com> wrote:
>> >>> > On Sun, May 30, 2010 at 3:27 PM, Bjoern Brandenburg <bbb.lst@gmail.com> wrote:
>> >>> >>
>> >>> >> I'll try to see if I can pinpoint what was dropped between 2.6.33-arm1
>> >>> >> and 2.6.34-arm.
>> >>> >
>> >>> > Progress: I can get 2.6.34-arm to boot with all 4 CPUs after
>> >>> > cherry-picking the following commits (which seemed relevant but
>> >>> > absent):
>> >>> >
>> >>> > 60060ca ARM: Handle instruction cache maintenance fault ?properly
>> >>> > 3f64e83 ARM errata: Eviction Buffer not empty after Cache Sync on L220
>> >>> > 3b009b5 ARM: change definition of cpu_relax() for ARM11MPCore
>> >>> >
>> >>> > Let's see which is the critical one...
>> >>>
>> >>> It's 3f64e83 "ARM errata: Eviction Buffer not empty after Cache Sync
>> >>> on L220" [1]. With this commit cherry-picked (on top of the 'rebased'
>> >>> branch in ARM's repository, i.e., 2.6.34-arm), the system boots to X11
>> >>> and runs some simple FS tests; the other ones don't make a difference.
>> >>>
>> >>> Are there plans for getting this and the other patches in the
>> >>> 'rebased' branch into mainline (for .35 or .36)?
>> >>
>> >> Thanks for the investigation. I recall I got something similar in the
>> >> past though I could no longer reproduce it with 2.6.34 (-arm) on the
>> >> PB11MPCore I have. Could you try reverting commit e7c5650f606 (ARM:
>> >> Change the mandatory barriers implementation) on a vanilla 2.6.34
>> >> kernel?
>> >
>> > That doesn't seem to help. v2.6.34 with e7c5650f606 reverted still hangs.
>>
>> Given that 2.6.34-arm contains patches for missing synchronization
>> primitives in the network driver, it is maybe not too surprising that
>> the vanilla kernel won't boot with an NFS root file system.
>>
>> I just tried 2.6.34-arm (i.e., the 'rebased' branch in the ARM repo)
>> with e7c5650f606 reverted (and without cherry-picking 3f64e83) ? and
>> that kernel boots ok!
>>
> We have similar observation on SPI based network driver. By playing with
> buffer alignment, the issue seems to be going away. Do you have link
> to " synchronization primitives in the network driver" ?patches ??

Hi  Santosh,

I was referring to the below patch. Without it, none of the tested
kernels could boot on our box.

- Bjoern

http://linux-arm.org/git?p=linux-2.6.git;a=commit;h=840cb637f8781516b9a4abc8f39508b30a63f1b0
commit 840cb637f8781516b9a4abc8f39508b30a63f1b0
Author: Catalin Marinas <catalin.marinas@arm.com>
Date:   Tue Jun 1 10:05:05 2010 +0100

    smsc911x: Add spinlocks around registers access

    On SMP systems, the SMSC911x registers may be accessed by multiple CPUs
    and this seems to put the chip in an inconsistent state. The patch adds
    spinlocks to the smsc911x_reg_read, smsc911x_reg_write,
    smsc911x_rx_readfifo and smsc911x_tx_writefifo functions.

    Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>

2.6.34 hangs during boot on PB11MPCore

[Bjoern Brandenburg]

On Tue, Jun 1, 2010 at 3:46 AM, Colin Tuckley <Colin.Tuckley@arm.com> wrote:
>> -----Original Message-----
>> From: linux-arm-kernel-bounces at lists.infradead.org [mailto:linux-arm-
>> kernel-bounces at lists.infradead.org] On Behalf Of Catalin Marinas
>
>> Regarding the SMSC 911x driver synchronisation - I've been asking SMSC
>> for more than a year to look into this.
>
> I'm still waiting for them to take up our offer of a PB11MPCore system for testing.
>
> One thing Bjoern hasn't mentioned... is the board firmware up to date? I.e. has the fpga image been updated to the latest release (from the latest CD).
>

Mhhh... Despite much looking through the arm.com site, I couldn't
figure out what the "latest CD" is. We got our box last September and
haven't installed any firmware updates since. Thus it's quite possible
that the firmware is out of date.
Is this the revision ID that you're looking for?

# cat /proc/cpuinfo
Processor	: ARMv6-compatible processor rev 0 (v6l)
processor	: 0
BogoMIPS	: 83.55

processor	: 1
BogoMIPS	: 83.55

processor	: 2
BogoMIPS	: 83.55

processor	: 3
BogoMIPS	: 83.55

Features	: swp half thumb fastmult vfp edsp java
CPU implementer	: 0x41
CPU architecture: 7
CPU variant	: 0x0
CPU part	: 0xb02
CPU revision	: 0

Hardware	: ARM-RealView PB11MPCore
Revision	: 2159f503
Serial		: 0000000000000000

- Bj?rn