[REGRESSION in 3.18][PPC] PA Semi fails to boot after: of/base: Fix PowerPC address parsing hack

[REGRESSION in 3.18][PPC] PA Semi fails to boot after: of/base: Fix PowerPC address parsing hack

[Steven Rostedt]

A bug in ftrace was reported to me that affects ARM and ARM64 but not
x86. Looking at the code it appears to affect PowerPC as well. So I
booted up my old PA Semi, to give it a try. The last time I booted it
was for a 3.17 kernel. Unfortunately, for 4.0-rc2 it crashed with:

Unable to handle kernel paging request for data at address 0x00000000
Faulting instruction address: 0xc0000000005cef88
Oops: Kernel access of bad area, sig: 11 [#1]
SMP NR_CPUS=2 PA Semi PWRficient
Modules linked in:
CPU: 1 PID: 0 Comm: swapper/1 Not tainted 4.0.0-rc2-test #50
task: c00000003816cb60 ti: c0000000381a4000 task.ti: c0000000381a4000
NIP: c0000000005cef88 LR: c00000000007c1a0 CTR: c00000000007c184
REGS: c0000000381a7a00 TRAP: 0300   Not tainted  (4.0.0-rc2-test)
MSR: 9000000000009032 <SF,HV,EE,ME,IR,DR,RI>  CR: 22000028  XER: 00000000
DAR: 0000000000000000 DSISR: 40000000 SOFTE: 0 
GPR00: c00000000007c1a0 c0000000381a7c80 c000000000af4b98 0000000000000001 
GPR04: 0000000000000000 0000000000000000 00000000000004ba 000000003d6de000 
GPR08: 0100000000000000 0000000000000000 c0000000381a4080 0000000000000000 
GPR12: 0000000024044042 c00000000ffff300 ffffffffffffffed 0000000000000000 
GPR16: c000000000afb920 c0000000381a4000 c0000000009ad648 c0000000009ae580 
GPR20: c0000000381a4080 c0000000381a4000 c0000000381a4080 c0000000381a4000 
GPR24: c0000000381a4000 c0000000381a4000 c000000000afb880 c0000000381a4000 
GPR28: c0000000009f8790 0000000000000000 c0000000381a4000 c000000000b02168 
NIP [c0000000005cef88] .check_astate+0x28/0x50
LR [c00000000007c1a0] sleep_common+0x14/0x74
Call Trace:
[c0000000381a7c80] [c000000000afb880] 0xc000000000afb880 (unreliable)
[c0000000381a7cf0] [c00000000007c1a0] sleep_common+0x14/0x74
[c0000000381a7d30] [c0000000000130f0] .arch_cpu_idle+0x70/0x160
[c0000000381a7db0] [c0000000000d6660] .cpu_startup_entry+0x320/0x5a0
[c0000000381a7ee0] [c000000000034570] .start_secondary+0x290/0x2c0
[c0000000381a7f90] [c000000000008bfc] start_secondary_prolog+0x10/0x14
Instruction dump:
60000000 60000000 7c0802a6 f8010010 f821ff91 60000000 60000000 3d220003 
39296870 a86d0038 e9290010 7c0004ac <7c004c2c> 0c000000 4c00012c 5463103a 
---[ end trace 40e864a431826b26 ]---

I kicked off a ktest bisect, and it came down to this commit:

commit 746c9e9f92dde2789908e51a354ba90a1962a2eb
Author: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Date:   Fri Nov 14 17:55:03 2014 +1100

    of/base: Fix PowerPC address parsing hack

When I revert this from v4.0-rc2, I can successfully boot my PA Semi
again.

-- Steve

Re: [REGRESSION in 3.18][PPC] PA Semi fails to boot after: of/base: Fix PowerPC address parsing hack

[Benjamin Herrenschmidt]

On Thu, 2015-03-05 at 17:12 -0500, Steven Rostedt wrote:
> A bug in ftrace was reported to me that affects ARM and ARM64 but not
> x86. Looking at the code it appears to affect PowerPC as well. So I
> booted up my old PA Semi, to give it a try. The last time I booted it
> was for a 3.17 kernel. Unfortunately, for 4.0-rc2 it crashed with:

Argh. Well, we have one of these here but Michael who owns it is off til
Tuesday. 

Can you shoot me the DT (/proc/device-tree in a tarball) ? Olof, can the
DT be updated on this thing or should we add workarounds to Linux if
something is really missing ?

Cheers,
Ben.

> Unable to handle kernel paging request for data at address 0x00000000
> Faulting instruction address: 0xc0000000005cef88
> Oops: Kernel access of bad area, sig: 11 [#1]
> SMP NR_CPUS=2 PA Semi PWRficient
> Modules linked in:
> CPU: 1 PID: 0 Comm: swapper/1 Not tainted 4.0.0-rc2-test #50
> task: c00000003816cb60 ti: c0000000381a4000 task.ti: c0000000381a4000
> NIP: c0000000005cef88 LR: c00000000007c1a0 CTR: c00000000007c184
> REGS: c0000000381a7a00 TRAP: 0300   Not tainted  (4.0.0-rc2-test)
> MSR: 9000000000009032 <SF,HV,EE,ME,IR,DR,RI>  CR: 22000028  XER: 00000000
> DAR: 0000000000000000 DSISR: 40000000 SOFTE: 0 
> GPR00: c00000000007c1a0 c0000000381a7c80 c000000000af4b98 0000000000000001 
> GPR04: 0000000000000000 0000000000000000 00000000000004ba 000000003d6de000 
> GPR08: 0100000000000000 0000000000000000 c0000000381a4080 0000000000000000 
> GPR12: 0000000024044042 c00000000ffff300 ffffffffffffffed 0000000000000000 
> GPR16: c000000000afb920 c0000000381a4000 c0000000009ad648 c0000000009ae580 
> GPR20: c0000000381a4080 c0000000381a4000 c0000000381a4080 c0000000381a4000 
> GPR24: c0000000381a4000 c0000000381a4000 c000000000afb880 c0000000381a4000 
> GPR28: c0000000009f8790 0000000000000000 c0000000381a4000 c000000000b02168 
> NIP [c0000000005cef88] .check_astate+0x28/0x50
> LR [c00000000007c1a0] sleep_common+0x14/0x74
> Call Trace:
> [c0000000381a7c80] [c000000000afb880] 0xc000000000afb880 (unreliable)
> [c0000000381a7cf0] [c00000000007c1a0] sleep_common+0x14/0x74
> [c0000000381a7d30] [c0000000000130f0] .arch_cpu_idle+0x70/0x160
> [c0000000381a7db0] [c0000000000d6660] .cpu_startup_entry+0x320/0x5a0
> [c0000000381a7ee0] [c000000000034570] .start_secondary+0x290/0x2c0
> [c0000000381a7f90] [c000000000008bfc] start_secondary_prolog+0x10/0x14
> Instruction dump:
> 60000000 60000000 7c0802a6 f8010010 f821ff91 60000000 60000000 3d220003 
> 39296870 a86d0038 e9290010 7c0004ac <7c004c2c> 0c000000 4c00012c 5463103a 
> ---[ end trace 40e864a431826b26 ]---
> 
> I kicked off a ktest bisect, and it came down to this commit:
> 
> commit 746c9e9f92dde2789908e51a354ba90a1962a2eb
> Author: Benjamin Herrenschmidt <benh@kernel.crashing.org>
> Date:   Fri Nov 14 17:55:03 2014 +1100
> 
>     of/base: Fix PowerPC address parsing hack
> 
> When I revert this from v4.0-rc2, I can successfully boot my PA Semi
> again.
> 
> -- Steve

Re: [REGRESSION in 3.18][PPC] PA Semi fails to boot after: of/base: Fix PowerPC address parsing hack

[Steven Rostedt]

[-- Attachment #1: Type: text/plain, Size: 178 bytes --]

On Fri, 06 Mar 2015 15:18:42 +1100
Benjamin Herrenschmidt <benh@kernel.crashing.org> wrote:

 
> Can you shoot me the DT (/proc/device-tree in a tarball) ?

Attached.

-- Steve


[-- Attachment #2: device-tree.tar.bz2 --]
[-- Type: application/x-bzip, Size: 6488 bytes --]

Re: [REGRESSION in 3.18][PPC] PA Semi fails to boot after: of/base: Fix PowerPC address parsing hack

[Benjamin Herrenschmidt]

On Fri, 2015-03-06 at 10:00 -0500, Steven Rostedt wrote:
> On Fri, 06 Mar 2015 15:18:42 +1100
> Benjamin Herrenschmidt <benh@kernel.crashing.org> wrote:
> 
>  
> > Can you shoot me the DT (/proc/device-tree in a tarball) ?
> 
> Attached.

This is indeed a bug in their DT. We might want to add quirks for
that unless it can be fixed (or has been via FW update). Olof ?

In the meantime, try that patch:

diff --git a/drivers/of/address.c b/drivers/of/address.c
index ad29069..78a7dcb 100644
--- a/drivers/of/address.c
+++ b/drivers/of/address.c
@@ -450,12 +450,17 @@ static struct of_bus *of_match_bus(struct device_node *np)
 	return NULL;
 }
 
-static int of_empty_ranges_quirk(void)
+static int of_empty_ranges_quirk(struct device_node *np)
 {
 	if (IS_ENABLED(CONFIG_PPC)) {
-		/* To save cycles, we cache the result */
+		/* To save cycles, we cache the result for global "Mac" setting */
 		static int quirk_state = -1;
 
+		/* PA-SEMI sdc DT bug */
+		if (of_device_is_compatible(np, "1682m-sdc"))
+			return true;
+
+		/* Make quirk cached */
 		if (quirk_state < 0)
 			quirk_state =
 				of_machine_is_compatible("Power Macintosh") ||
@@ -490,7 +495,7 @@ static int of_translate_one(struct device_node *parent, struct of_bus *bus,
 	 * This code is only enabled on powerpc. --gcl
 	 */
 	ranges = of_get_property(parent, rprop, &rlen);
-	if (ranges == NULL && !of_empty_ranges_quirk()) {
+	if (ranges == NULL && !of_empty_ranges_quirk(parent)) {
 		pr_debug("OF: no ranges; cannot translate\n");
 		return 1;
 	}

Re: [REGRESSION in 3.18][PPC] PA Semi fails to boot after: of/base: Fix PowerPC address parsing hack

[Steven Rostedt]

On Sat, 07 Mar 2015 09:56:22 +1100
Benjamin Herrenschmidt <benh@kernel.crashing.org> wrote:


> In the meantime, try that patch:
> 

It boots for me.

Tested-by: Steven Rostedt <rostedt@goodmis.org>

-- Steve

Re: [REGRESSION in 3.18][PPC] PA Semi fails to boot after: of/base: Fix PowerPC address parsing hack

[Olof Johansson]

On Fri, Mar 6, 2015 at 2:56 PM, Benjamin Herrenschmidt
<benh@kernel.crashing.org> wrote:
> On Fri, 2015-03-06 at 10:00 -0500, Steven Rostedt wrote:
>> On Fri, 06 Mar 2015 15:18:42 +1100
>> Benjamin Herrenschmidt <benh@kernel.crashing.org> wrote:
>>
>>
>> > Can you shoot me the DT (/proc/device-tree in a tarball) ?
>>
>> Attached.
>
> This is indeed a bug in their DT. We might want to add quirks for
> that unless it can be fixed (or has been via FW update). Olof ?

FW updates on this platform are highly unlikely. Quirk it is.


-Olof

Re: [REGRESSION in 3.18][PPC] PA Semi fails to boot after: of/base: Fix PowerPC address parsing hack

[Benjamin Herrenschmidt]

On Fri, 2015-03-06 at 15:50 -0800, Olof Johansson wrote:
> On Fri, Mar 6, 2015 at 2:56 PM, Benjamin Herrenschmidt
> <benh@kernel.crashing.org> wrote:
> > On Fri, 2015-03-06 at 10:00 -0500, Steven Rostedt wrote:
> >> On Fri, 06 Mar 2015 15:18:42 +1100
> >> Benjamin Herrenschmidt <benh@kernel.crashing.org> wrote:
> >>
> >>
> >> > Can you shoot me the DT (/proc/device-tree in a tarball) ?
> >>
> >> Attached.
> >
> > This is indeed a bug in their DT. We might want to add quirks for
> > that unless it can be fixed (or has been via FW update). Olof ?
> 
> FW updates on this platform are highly unlikely. Quirk it is.

Oh I was not expecting a new FW, I was mostly wondering whether Steven
had the latest one since I *think* Michael has been testing with the
PA board we got here and didn't see that problem ... anyway, I'll check
with him early next week and clean up / submit that patch.

Cheers,
Ben.

> 
> -Olof
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: [REGRESSION in 3.18][PPC] PA Semi fails to boot after: of/base: Fix PowerPC address parsing hack

[Michael Ellerman]

On Sat, 2015-03-07 at 15:02 +1100, Benjamin Herrenschmidt wrote:
> On Fri, 2015-03-06 at 15:50 -0800, Olof Johansson wrote:
> > On Fri, Mar 6, 2015 at 2:56 PM, Benjamin Herrenschmidt
> > <benh@kernel.crashing.org> wrote:
> > > On Fri, 2015-03-06 at 10:00 -0500, Steven Rostedt wrote:
> > >> On Fri, 06 Mar 2015 15:18:42 +1100
> > >> Benjamin Herrenschmidt <benh@kernel.crashing.org> wrote:
> > >>
> > >>
> > >> > Can you shoot me the DT (/proc/device-tree in a tarball) ?
> > >>
> > >> Attached.
> > >
> > > This is indeed a bug in their DT. We might want to add quirks for
> > > that unless it can be fixed (or has been via FW update). Olof ?
> > 
> > FW updates on this platform are highly unlikely. Quirk it is.
> 
> Oh I was not expecting a new FW, I was mostly wondering whether Steven
> had the latest one since I *think* Michael has been testing with the
> PA board we got here and didn't see that problem ... anyway, I'll check
> with him early next week and clean up / submit that patch.

Yeah I have been testing semi-regularly.

4.0-rc2 boots fine on mine.

But mine is an Athena, Steve's is an Electra. So they're not identical.

Mine is running:

  CFE version PAS-2.0.29 for ATHENA (64bit,MP,BE,PPC)
  Build Date: Mon Jun 30 11:47:25 PDT 2008 (mpl@mitch-1)

Steve is your CFE older than that?

Olof do you remember if that version or something newer is available for
Electra?

cheers

Re: [REGRESSION in 3.18][PPC] PA Semi fails to boot after: of/base: Fix PowerPC address parsing hack

[Olof Johansson]

On Mon, Mar 9, 2015 at 5:28 PM, Michael Ellerman <mpe@ellerman.id.au> wrote:
> On Sat, 2015-03-07 at 15:02 +1100, Benjamin Herrenschmidt wrote:
>> On Fri, 2015-03-06 at 15:50 -0800, Olof Johansson wrote:
>> > On Fri, Mar 6, 2015 at 2:56 PM, Benjamin Herrenschmidt
>> > <benh@kernel.crashing.org> wrote:
>> > > On Fri, 2015-03-06 at 10:00 -0500, Steven Rostedt wrote:
>> > >> On Fri, 06 Mar 2015 15:18:42 +1100
>> > >> Benjamin Herrenschmidt <benh@kernel.crashing.org> wrote:
>> > >>
>> > >>
>> > >> > Can you shoot me the DT (/proc/device-tree in a tarball) ?
>> > >>
>> > >> Attached.
>> > >
>> > > This is indeed a bug in their DT. We might want to add quirks for
>> > > that unless it can be fixed (or has been via FW update). Olof ?
>> >
>> > FW updates on this platform are highly unlikely. Quirk it is.
>>
>> Oh I was not expecting a new FW, I was mostly wondering whether Steven
>> had the latest one since I *think* Michael has been testing with the
>> PA board we got here and didn't see that problem ... anyway, I'll check
>> with him early next week and clean up / submit that patch.
>
> Yeah I have been testing semi-regularly.
>
> 4.0-rc2 boots fine on mine.
>
> But mine is an Athena, Steve's is an Electra. So they're not identical.

I have a chitra in my boot farm, so I run every build I do on it. I
have not hit it either, which confused me.

Turns out that Steven's machine boots with idle=doze, which is the
part that makes all the difference.


FWIW, the three machines have roughly these diffs:

* Electra: First development/eval board. Funky USB on localbus, plenty
of PCI-e. Two GigE, one 10GigE XAUI. CompactFlash and IDE on localbus
too. Usually shipped with a PCI-e SATA card and a USB card.

* Chitra: Second edition dev/eval board. Moved SATA and USB on-board,
and removed some of the localbus hardware. Might have routed three
GigE out instead of 2, can't remember.

* Athena: Never released board with a smaller package chip, there's
only a few of these around. Can't comment too much on the specifics,
but it's similar to Chitra, and the silicon is the same.

> Mine is running:
>
>   CFE version PAS-2.0.29 for ATHENA (64bit,MP,BE,PPC)
>   Build Date: Mon Jun 30 11:47:25 PDT 2008 (mpl@mitch-1)
>
> Steve is your CFE older than that?
>
> Olof do you remember if that version or something newer is available for
> Electra?

That looks about as new as they come. My board runs a .29 too.


-Olof

Re: [REGRESSION in 3.18][PPC] PA Semi fails to boot after: of/base: Fix PowerPC address parsing hack

[Michael Ellerman]

On Mon, 2015-03-09 at 17:33 -0700, Olof Johansson wrote:
> On Mon, Mar 9, 2015 at 5:28 PM, Michael Ellerman <mpe@ellerman.id.au> wrote:
> > On Sat, 2015-03-07 at 15:02 +1100, Benjamin Herrenschmidt wrote:
> >> On Fri, 2015-03-06 at 15:50 -0800, Olof Johansson wrote:
> >> > On Fri, Mar 6, 2015 at 2:56 PM, Benjamin Herrenschmidt
> >> > <benh@kernel.crashing.org> wrote:
> >> > > On Fri, 2015-03-06 at 10:00 -0500, Steven Rostedt wrote:
> >> > >> On Fri, 06 Mar 2015 15:18:42 +1100
> >> > >> Benjamin Herrenschmidt <benh@kernel.crashing.org> wrote:
> >> > >>
> >> > >>
> >> > >> > Can you shoot me the DT (/proc/device-tree in a tarball) ?
> >> > >>
> >> > >> Attached.
> >> > >
> >> > > This is indeed a bug in their DT. We might want to add quirks for
> >> > > that unless it can be fixed (or has been via FW update). Olof ?
> >> >
> >> > FW updates on this platform are highly unlikely. Quirk it is.
> >>
> >> Oh I was not expecting a new FW, I was mostly wondering whether Steven
> >> had the latest one since I *think* Michael has been testing with the
> >> PA board we got here and didn't see that problem ... anyway, I'll check
> >> with him early next week and clean up / submit that patch.
> >
> > Yeah I have been testing semi-regularly.
> >
> > 4.0-rc2 boots fine on mine.
> >
> > But mine is an Athena, Steve's is an Electra. So they're not identical.
> 
> I have a chitra in my boot farm, so I run every build I do on it. I
> have not hit it either, which confused me.
> 
> Turns out that Steven's machine boots with idle=doze, which is the
> part that makes all the difference.

Aha, that is the key.

So mine also crashes with idle=doze, even with the newer firmware.

So we'll have to fix this with the quirk in the kernel.

cheers

Re: [REGRESSION in 3.18][PPC] PA Semi fails to boot after: of/base: Fix PowerPC address parsing hack

[Steven Rostedt]

On Tue, 10 Mar 2015 11:28:03 +1100
Michael Ellerman <mpe@ellerman.id.au> wrote:

> 
> Mine is running:
> 
>   CFE version PAS-2.0.29 for ATHENA (64bit,MP,BE,PPC)
>   Build Date: Mon Jun 30 11:47:25 PDT 2008 (mpl@mitch-1)
> 
> Steve is your CFE older than that?

Seems so:

CFE version PAS-2.0.20 for ELECTRA (64bit,MP,BE,PPC)
Build Date: Tue Nov  6 22:35:48 PST 2007 (mpl@mitch-1)

-- Steve