ppc64 qemu test failure since commit f9aa67142 ("powerpc/64s: Consolidate Alignment 0x600 interrupt")

ppc64 qemu test failure since commit f9aa67142 ("powerpc/64s: Consolidate Alignment 0x600 interrupt")

[Guenter Roeck]

Nicholas,

some of my qemu tests for ppc64 started failing on mainline (and -next).
You can find a test log at
http://kerneltests.org/builders/qemu-ppc64-master/builds/580/steps/qemubuildcommand/logs/stdio

The scripts to run the test are available at
https://github.com/groeck/linux-build-test/tree/master/rootfs/ppc64

Bisect points to commit f9aa67142ef26 ("powerpc/64s: Consolidate Alignment 0x600
interrupt"). Bisect log is attached.

Since I don't have the means to run the code on a real system, I have no idea
if the problem is caused by qemu or by the code. It is interesting, though, that
only the 'mac99' tests are affected.

Please let me know if there is anything I can do to help tracking down the
problem.

Thanks,
Guenter

---
# bad: [b66484cd74706fa8681d051840fe4b18a3da40ff] Merge branch 'akpm' (patches from Andrew)
# good: [87840a2b7e048018d18d60bdac5c09224de85370] Merge branch 'i2c/for-4.9' of git://git.kernel.org/pub/scm/linux/kernel/git/wsa/linux
git bisect start 'HEAD' '87840a2'
# bad: [6afd563d4bbc1924b7de9e053324c007e0d36476] Merge tag 'armsoc-drivers' of git://git.kernel.org/pub/scm/linux/kernel/git/arm/arm-soc
git bisect bad 6afd563d4bbc1924b7de9e053324c007e0d36476
# bad: [07021b43597f506cc525d139ed1a94e79cf184f2] Merge tag 'powerpc-4.9-1' of git://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux
git bisect bad 07021b43597f506cc525d139ed1a94e79cf184f2
# bad: [e909fb83d39292679118761426d7784715ad79ad] powerpc: Never giveup a reclaimed thread when enabling kernel {fp, altivec, vsx}
git bisect bad e909fb83d39292679118761426d7784715ad79ad
# good: [313483dd72289ff0b7dff4207032d94dbce39cf6] powerpc/powernv: Unfreeze PE on allocation
git bisect good 313483dd72289ff0b7dff4207032d94dbce39cf6
# bad: [792cbddd628282027f99f5499f2899613e07f8f9] powerpc/64s: Consolidate VSX Unavailable 0xf40 interrupt
git bisect bad 792cbddd628282027f99f5499f2899613e07f8f9
# bad: [11e87346b9b885f3b088efb6afda604e91233026] powerpc/64s: Consolidate Program 0x700 interrupt
git bisect bad 11e87346b9b885f3b088efb6afda604e91233026
# good: [57f266497d81e16141bd2c9009e91dad34ea5f70] powerpc: Use gas sections for arranging exception vectors
git bisect good 57f266497d81e16141bd2c9009e91dad34ea5f70
# good: [2b9af6e40e477fbfe39777dda9a609ae359d3dd8] powerpc/64s: Consolidate Data Segment 0x380 interrupt
git bisect good 2b9af6e40e477fbfe39777dda9a609ae359d3dd8
# good: [8d04631ad7cbbe27d058ae79e3d41009a8e006d1] powerpc/64s: Consolidate Instruction Segment 0x480 interrupt
git bisect good 8d04631ad7cbbe27d058ae79e3d41009a8e006d1
# bad: [f9aa67142ef2697990d1f36aa3d59320820bcfae] powerpc/64s: Consolidate Alignment 0x600 interrupt
git bisect bad f9aa67142ef2697990d1f36aa3d59320820bcfae
# good: [c138e58890dd5128a5706ed30c5deab6736320d6] powerpc/64s: Consolidate External 0x500 interrupt
git bisect good c138e58890dd5128a5706ed30c5deab6736320d6
# first bad commit: [f9aa67142ef2697990d1f36aa3d59320820bcfae] powerpc/64s: Consolidate Alignment 0x600 interrupt

Re: ppc64 qemu test failure since commit f9aa67142 ("powerpc/64s: Consolidate Alignment 0x600 interrupt")

[Nicholas Piggin]

On Sun, 9 Oct 2016 08:21:21 -0700
Guenter Roeck <linux@roeck-us.net> wrote:

> Nicholas,
> 
> some of my qemu tests for ppc64 started failing on mainline (and -next).
> You can find a test log at
> http://kerneltests.org/builders/qemu-ppc64-master/builds/580/steps/qemubuildcommand/logs/stdio
> 
> The scripts to run the test are available at
> https://github.com/groeck/linux-build-test/tree/master/rootfs/ppc64
> 
> Bisect points to commit f9aa67142ef26 ("powerpc/64s: Consolidate Alignment 0x600
> interrupt"). Bisect log is attached.
> 
> Since I don't have the means to run the code on a real system, I have no idea
> if the problem is caused by qemu or by the code. It is interesting, though, that
> only the 'mac99' tests are affected.
> 
> Please let me know if there is anything I can do to help tracking down the
> problem.

Thanks for this. That patch just moves a small amount of code, so it's likely
that it's caused something to get placed out of range of its caller, or the
linker started generating a stub for some reason. I can't immediately see the
problem, but it could be specific to your exact toolchain.

Something that might help, would you be able to put the compiled vmlinux binaries
from before/after the bad patch somewhere I can grab them?

Thanks,
Nick

Re: ppc64 qemu test failure since commit f9aa67142 ("powerpc/64s: Consolidate Alignment 0x600 interrupt")

[Michael Ellerman]

Guenter Roeck <linux@roeck-us.net> writes:

> Nicholas,
>
> some of my qemu tests for ppc64 started failing on mainline (and -next).
> You can find a test log at
> http://kerneltests.org/builders/qemu-ppc64-master/builds/580/steps/qemubuildcommand/logs/stdio
>
> The scripts to run the test are available at
> https://github.com/groeck/linux-build-test/tree/master/rootfs/ppc64
>
> Bisect points to commit f9aa67142ef26 ("powerpc/64s: Consolidate Alignment 0x600
> interrupt"). Bisect log is attached.
>
> Since I don't have the means to run the code on a real system, I have no idea
> if the problem is caused by qemu or by the code. It is interesting, though, that
> only the 'mac99' tests are affected.

Thanks for the report.

I do test qemu mac99 as part of my CI, and it's booting OK, so there
must be some config difference?

I'm using pmac32_defconfig plus:

CONFIG_LOCALVERSION_AUTO=y
CONFIG_SERIAL_PMACZILOG=y
CONFIG_SERIAL_PMACZILOG_TTYS=n
CONFIG_SERIAL_PMACZILOG_CONSOLE=y
CONFIG_DEVTMPFS=y
CONFIG_DEVTMPFS_MOUNT=y

And running with:

$ qemu-system-ppc -nographic -vga none -M mac99 -m 1G -kernel vmlinux -initrd $HOME/static/ppc32-initrd.gz -append "console=ttyPZ0 init=/bin/sh"

cheers

Re: ppc64 qemu test failure since commit f9aa67142 ("powerpc/64s: Consolidate Alignment 0x600 interrupt")

[Guenter Roeck]

Hi Michael,

On 10/09/2016 11:00 PM, Michael Ellerman wrote:
> Guenter Roeck <linux@roeck-us.net> writes:
>
>> Nicholas,
>>
>> some of my qemu tests for ppc64 started failing on mainline (and -next).
>> You can find a test log at
>> http://kerneltests.org/builders/qemu-ppc64-master/builds/580/steps/qemubuildcommand/logs/stdio
>>
>> The scripts to run the test are available at
>> https://github.com/groeck/linux-build-test/tree/master/rootfs/ppc64
>>
>> Bisect points to commit f9aa67142ef26 ("powerpc/64s: Consolidate Alignment 0x600
>> interrupt"). Bisect log is attached.
>>
>> Since I don't have the means to run the code on a real system, I have no idea
>> if the problem is caused by qemu or by the code. It is interesting, though, that
>> only the 'mac99' tests are affected.
>
> Thanks for the report.
>
> I do test qemu mac99 as part of my CI, and it's booting OK, so there
> must be some config difference?
>
> I'm using pmac32_defconfig plus:
>
> CONFIG_LOCALVERSION_AUTO=y
> CONFIG_SERIAL_PMACZILOG=y
> CONFIG_SERIAL_PMACZILOG_TTYS=n
> CONFIG_SERIAL_PMACZILOG_CONSOLE=y
> CONFIG_DEVTMPFS=y
> CONFIG_DEVTMPFS_MOUNT=y
>
> And running with:
>
> $ qemu-system-ppc -nographic -vga none -M mac99 -m 1G -kernel vmlinux -initrd $HOME/static/ppc32-initrd.gz -append "console=ttyPZ0 init=/bin/sh"
>

I am building a 64bit image, and my test runs with qemu-system-ppc64.
My 32bit tests for mac99 are fine as well.

Guenter

Re: ppc64 qemu test failure since commit f9aa67142 ("powerpc/64s: Consolidate Alignment 0x600 interrupt")

[Guenter Roeck]

On 10/09/2016 10:49 PM, Nicholas Piggin wrote:
> On Sun, 9 Oct 2016 08:21:21 -0700
> Guenter Roeck <linux@roeck-us.net> wrote:
>
>> Nicholas,
>>
>> some of my qemu tests for ppc64 started failing on mainline (and -next).
>> You can find a test log at
>> http://kerneltests.org/builders/qemu-ppc64-master/builds/580/steps/qemubuildcommand/logs/stdio
>>
>> The scripts to run the test are available at
>> https://github.com/groeck/linux-build-test/tree/master/rootfs/ppc64
>>
>> Bisect points to commit f9aa67142ef26 ("powerpc/64s: Consolidate Alignment 0x600
>> interrupt"). Bisect log is attached.
>>
>> Since I don't have the means to run the code on a real system, I have no idea
>> if the problem is caused by qemu or by the code. It is interesting, though, that
>> only the 'mac99' tests are affected.
>>
>> Please let me know if there is anything I can do to help tracking down the
>> problem.
>
> Thanks for this. That patch just moves a small amount of code, so it's likely
> that it's caused something to get placed out of range of its caller, or the
> linker started generating a stub for some reason. I can't immediately see the
> problem, but it could be specific to your exact toolchain.
>
> Something that might help, would you be able to put the compiled vmlinux binaries
> from before/after the bad patch somewhere I can grab them?
>

http://server.roeck-us.net/qemu/ppc64/mac99/

'bad' is at f9aa67142ef26, 'good' is one commit earlier, 'tot' is from top of tree
(b66484cd7470, more specifically).

Key difference in System.map, from the bad case:

c000000000005c00 T __end_interrupts
c000000000007000 t end_virt_trampolines
c000000000008000 t 00000010.long_branch.power4_fixup_nap+0
c000000000008100 t fs_label
c000000000008100 t start_text

00000010.long_branch.power4_fixup_nap+0 does not exist in the good case,
and fs_label/start_text are at c000000000008000.

Toolchain is from poky 1.5.1, which uses gcc 4.8.1 and binutils 2.23.2.
I also tried with the toolchain from poky 1.6, using gcc 4.8.2 and binutils 2.24,
with the same result.

Guenter

Re: ppc64 qemu test failure since commit f9aa67142 ("powerpc/64s: Consolidate Alignment 0x600 interrupt")

[Nicholas Piggin]

On Mon, 10 Oct 2016 07:15:11 -0700
Guenter Roeck <linux@roeck-us.net> wrote:

> On 10/09/2016 10:49 PM, Nicholas Piggin wrote:
> > On Sun, 9 Oct 2016 08:21:21 -0700
> > Guenter Roeck <linux@roeck-us.net> wrote:
> >  
> >> Nicholas,
> >>
> >> some of my qemu tests for ppc64 started failing on mainline (and -next).
> >> You can find a test log at
> >> http://kerneltests.org/builders/qemu-ppc64-master/builds/580/steps/qemubuildcommand/logs/stdio
> >>
> >> The scripts to run the test are available at
> >> https://github.com/groeck/linux-build-test/tree/master/rootfs/ppc64
> >>
> >> Bisect points to commit f9aa67142ef26 ("powerpc/64s: Consolidate Alignment 0x600
> >> interrupt"). Bisect log is attached.
> >>
> >> Since I don't have the means to run the code on a real system, I have no idea
> >> if the problem is caused by qemu or by the code. It is interesting, though, that
> >> only the 'mac99' tests are affected.
> >>
> >> Please let me know if there is anything I can do to help tracking down the
> >> problem.  
> >
> > Thanks for this. That patch just moves a small amount of code, so it's likely
> > that it's caused something to get placed out of range of its caller, or the
> > linker started generating a stub for some reason. I can't immediately see the
> > problem, but it could be specific to your exact toolchain.
> >
> > Something that might help, would you be able to put the compiled vmlinux binaries
> > from before/after the bad patch somewhere I can grab them?
> >  
> 
> http://server.roeck-us.net/qemu/ppc64/mac99/
> 
> 'bad' is at f9aa67142ef26, 'good' is one commit earlier, 'tot' is from top of tree
> (b66484cd7470, more specifically).
> 
> Key difference in System.map, from the bad case:
> 
> c000000000005c00 T __end_interrupts
> c000000000007000 t end_virt_trampolines
> c000000000008000 t 00000010.long_branch.power4_fixup_nap+0
> c000000000008100 t fs_label
> c000000000008100 t start_text
> 
> 00000010.long_branch.power4_fixup_nap+0 does not exist in the good case,
> and fs_label/start_text are at c000000000008000.
> 
> Toolchain is from poky 1.5.1, which uses gcc 4.8.1 and binutils 2.23.2.
> I also tried with the toolchain from poky 1.6, using gcc 4.8.2 and binutils 2.24,
> with the same result.

Thank you for the quick response, this points to the exact problem.
I've attached a patch which should fix the bug. There are some checks
I've got planned that will catch this type of thing at build time and
be much easier to track down.

Thanks,
Nick

From: Nicholas Piggin <npiggin@gmail.com>
Date: Tue, 11 Oct 2016 18:33:26 +1100
Subject: [PATCH] powerpc/64s: fix power4_fixup_nap placement

power4_fixup_nap is called from the "common" handlers, not the virt/real
handlers, therefore it should itself be a common handler. Placing it
down in the trampoline space caused it to go out of reach of its
callers, requiring a trampoline inserted at the start of the text
section, which breaks the fixed section address calculations.

Reported-by: Guenter Roeck <linux@roeck-us.net>
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
 arch/powerpc/kernel/exceptions-64s.S | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/arch/powerpc/kernel/exceptions-64s.S b/arch/powerpc/kernel/exceptions-64s.S
index 08992f8..f129408 100644
--- a/arch/powerpc/kernel/exceptions-64s.S
+++ b/arch/powerpc/kernel/exceptions-64s.S
@@ -1377,7 +1377,7 @@ __end_interrupts:
 DEFINE_FIXED_SYMBOL(__end_interrupts)
 
 #ifdef CONFIG_PPC_970_NAP
-TRAMP_REAL_BEGIN(power4_fixup_nap)
+EXC_COMMON_BEGIN(power4_fixup_nap)
 	andc	r9,r9,r10
 	std	r9,TI_LOCAL_FLAGS(r11)
 	ld	r10,_LINK(r1)		/* make idle task do the */
-- 
2.9.3

Re: ppc64 qemu test failure since commit f9aa67142 ("powerpc/64s: Consolidate Alignment 0x600 interrupt")

[Guenter Roeck]

On 10/11/2016 12:47 AM, Nicholas Piggin wrote:
> On Mon, 10 Oct 2016 07:15:11 -0700
> Guenter Roeck <linux@roeck-us.net> wrote:
>
>> On 10/09/2016 10:49 PM, Nicholas Piggin wrote:
>>> On Sun, 9 Oct 2016 08:21:21 -0700
>>> Guenter Roeck <linux@roeck-us.net> wrote:
>>>
>>>> Nicholas,
>>>>
>>>> some of my qemu tests for ppc64 started failing on mainline (and -next).
>>>> You can find a test log at
>>>> http://kerneltests.org/builders/qemu-ppc64-master/builds/580/steps/qemubuildcommand/logs/stdio
>>>>
>>>> The scripts to run the test are available at
>>>> https://github.com/groeck/linux-build-test/tree/master/rootfs/ppc64
>>>>
>>>> Bisect points to commit f9aa67142ef26 ("powerpc/64s: Consolidate Alignment 0x600
>>>> interrupt"). Bisect log is attached.
>>>>
>>>> Since I don't have the means to run the code on a real system, I have no idea
>>>> if the problem is caused by qemu or by the code. It is interesting, though, that
>>>> only the 'mac99' tests are affected.
>>>>
>>>> Please let me know if there is anything I can do to help tracking down the
>>>> problem.
>>>
>>> Thanks for this. That patch just moves a small amount of code, so it's likely
>>> that it's caused something to get placed out of range of its caller, or the
>>> linker started generating a stub for some reason. I can't immediately see the
>>> problem, but it could be specific to your exact toolchain.
>>>
>>> Something that might help, would you be able to put the compiled vmlinux binaries
>>> from before/after the bad patch somewhere I can grab them?
>>>
>>
>> http://server.roeck-us.net/qemu/ppc64/mac99/
>>
>> 'bad' is at f9aa67142ef26, 'good' is one commit earlier, 'tot' is from top of tree
>> (b66484cd7470, more specifically).
>>
>> Key difference in System.map, from the bad case:
>>
>> c000000000005c00 T __end_interrupts
>> c000000000007000 t end_virt_trampolines
>> c000000000008000 t 00000010.long_branch.power4_fixup_nap+0
>> c000000000008100 t fs_label
>> c000000000008100 t start_text
>>
>> 00000010.long_branch.power4_fixup_nap+0 does not exist in the good case,
>> and fs_label/start_text are at c000000000008000.
>>
>> Toolchain is from poky 1.5.1, which uses gcc 4.8.1 and binutils 2.23.2.
>> I also tried with the toolchain from poky 1.6, using gcc 4.8.2 and binutils 2.24,
>> with the same result.
>
> Thank you for the quick response, this points to the exact problem.
> I've attached a patch which should fix the bug. There are some checks
> I've got planned that will catch this type of thing at build time and
> be much easier to track down.
>
> Thanks,
> Nick
>
> From: Nicholas Piggin <npiggin@gmail.com>
> Date: Tue, 11 Oct 2016 18:33:26 +1100
> Subject: [PATCH] powerpc/64s: fix power4_fixup_nap placement
>
> power4_fixup_nap is called from the "common" handlers, not the virt/real
> handlers, therefore it should itself be a common handler. Placing it
> down in the trampoline space caused it to go out of reach of its
> callers, requiring a trampoline inserted at the start of the text
> section, which breaks the fixed section address calculations.
>
> Reported-by: Guenter Roeck <linux@roeck-us.net>
> Signed-off-by: Nicholas Piggin <npiggin@gmail.com>

Yes, that works.

Tested-by: Guenter Roeck <linux@roeck-us.net>

> ---
>  arch/powerpc/kernel/exceptions-64s.S | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/arch/powerpc/kernel/exceptions-64s.S b/arch/powerpc/kernel/exceptions-64s.S
> index 08992f8..f129408 100644
> --- a/arch/powerpc/kernel/exceptions-64s.S
> +++ b/arch/powerpc/kernel/exceptions-64s.S
> @@ -1377,7 +1377,7 @@ __end_interrupts:
>  DEFINE_FIXED_SYMBOL(__end_interrupts)
>
>  #ifdef CONFIG_PPC_970_NAP
> -TRAMP_REAL_BEGIN(power4_fixup_nap)
> +EXC_COMMON_BEGIN(power4_fixup_nap)
>  	andc	r9,r9,r10
>  	std	r9,TI_LOCAL_FLAGS(r11)
>  	ld	r10,_LINK(r1)		/* make idle task do the */
>

Re: ppc64 qemu test failure since commit f9aa67142 ("powerpc/64s: Consolidate Alignment 0x600 interrupt")

[Michael Ellerman]

On Tue, 2016-11-10 at 07:47:56 UTC, Nicholas Piggin wrote:
> On Mon, 10 Oct 2016 07:15:11 -0700
> Guenter Roeck <linux@roeck-us.net> wrote:
> 
> > On 10/09/2016 10:49 PM, Nicholas Piggin wrote:
> > > On Sun, 9 Oct 2016 08:21:21 -0700
> > > Guenter Roeck <linux@roeck-us.net> wrote:
> > >  
> > >> Nicholas,
> > >>
> > >> some of my qemu tests for ppc64 started failing on mainline (and -next).
> > >> You can find a test log at
> > >> http://kerneltests.org/builders/qemu-ppc64-master/builds/580/steps/qemubuildcommand/logs/stdio
> > >>
> > >> The scripts to run the test are available at
> > >> https://github.com/groeck/linux-build-test/tree/master/rootfs/ppc64
> > >>
> > >> Bisect points to commit f9aa67142ef26 ("powerpc/64s: Consolidate Alignment 0x600
> > >> interrupt"). Bisect log is attached.
> > >>
> > >> Since I don't have the means to run the code on a real system, I have no idea
> > >> if the problem is caused by qemu or by the code. It is interesting, though, that
> > >> only the 'mac99' tests are affected.
> > >>
> > >> Please let me know if there is anything I can do to help tracking down the
> > >> problem.  
> > >
> > > Thanks for this. That patch just moves a small amount of code, so it's likely
> > > that it's caused something to get placed out of range of its caller, or the
> > > linker started generating a stub for some reason. I can't immediately see the
> > > problem, but it could be specific to your exact toolchain.
> > >
> > > Something that might help, would you be able to put the compiled vmlinux binaries
> > > from before/after the bad patch somewhere I can grab them?
> > >  
> > 
> > http://server.roeck-us.net/qemu/ppc64/mac99/
> > 
> > 'bad' is at f9aa67142ef26, 'good' is one commit earlier, 'tot' is from top of tree
> > (b66484cd7470, more specifically).
> > 
> > Key difference in System.map, from the bad case:
> > 
> > c000000000005c00 T __end_interrupts
> > c000000000007000 t end_virt_trampolines
> > c000000000008000 t 00000010.long_branch.power4_fixup_nap+0
> > c000000000008100 t fs_label
> > c000000000008100 t start_text
> > 
> > 00000010.long_branch.power4_fixup_nap+0 does not exist in the good case,
> > and fs_label/start_text are at c000000000008000.
> > 
> > Toolchain is from poky 1.5.1, which uses gcc 4.8.1 and binutils 2.23.2.
> > I also tried with the toolchain from poky 1.6, using gcc 4.8.2 and binutils 2.24,
> > with the same result.
> 
> Thank you for the quick response, this points to the exact problem.
> I've attached a patch which should fix the bug. There are some checks
> I've got planned that will catch this type of thing at build time and
> be much easier to track down.
> 
> Thanks,
> Nick
> 
> From: Nicholas Piggin <npiggin@gmail.com>
> Date: Tue, 11 Oct 2016 18:33:26 +1100
> Subject: [PATCH] powerpc/64s: fix power4_fixup_nap placement
> 
> power4_fixup_nap is called from the "common" handlers, not the virt/real
> handlers, therefore it should itself be a common handler. Placing it
> down in the trampoline space caused it to go out of reach of its
> callers, requiring a trampoline inserted at the start of the text
> section, which breaks the fixed section address calculations.
> 
> Reported-by: Guenter Roeck <linux@roeck-us.net>
> Signed-off-by: Nicholas Piggin <npiggin@gmail.com>

Applied to powerpc next, thanks.

https://git.kernel.org/powerpc/c/7c8cb4b50f3cc6f4a8f7bfddad6fb5

cheers