[PATCH] powerpc: allow cross-compilation of ppc64 kernel

[PATCH] powerpc: allow cross-compilation of ppc64 kernel

[Laurent Vivier]

When I try to cross compile a ppc64 kernel, it generally
fails on the VDSO stage. This is true for powerpc64 cross-
compiler, but also when I try to build a ppc64le kernel
on a ppc64 host.

VDSO64L fails:

  VDSO64L arch/powerpc/kernel/vdso64/vdso64.so.dbg
/usr/bin/powerpc64-linux-gnu-ld: arch/powerpc/kernel/vdso64/sigtramp.o:
file class ELFCLASS64 incompatible with ELFCLASS32
/usr/bin/powerpc64-linux-gnu-ld: final link failed: File in wrong format

This fails because gcc calls "collect2" with
"--oformat elf32-powerpcle" with ppc64 objects, without the
"--oformat" ld works well because it use the format of the
first object as output format.

As this case is correctly managed to build the other kernel
objects, this patch replaces $(GCC) by $(LD) to generate the
VDSO objects.

Signed-off-by: Laurent Vivier <laurent@vivier.eu>
---
 arch/powerpc/kernel/vdso64/Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/arch/powerpc/kernel/vdso64/Makefile b/arch/powerpc/kernel/vdso64/Makefile
index effca94..618c420 100644
--- a/arch/powerpc/kernel/vdso64/Makefile
+++ b/arch/powerpc/kernel/vdso64/Makefile
@@ -36,7 +36,7 @@ $(obj-vdso64): %.o: %.S
 
 # actual build commands
 quiet_cmd_vdso64ld = VDSO64L $@
-      cmd_vdso64ld = $(CC) $(c_flags) -Wl,-T $^ -o $@
+      cmd_vdso64ld = $(LD) $(LDFLAGS) $(ldflags-y) -T $^ -o $@
 quiet_cmd_vdso64as = VDSO64A $@
       cmd_vdso64as = $(CC) $(a_flags) -c -o $@ $<
 
-- 
2.5.0

Re: [PATCH] powerpc: allow cross-compilation of ppc64 kernel

[Scott Wood]

On Thu, 2015-11-05 at 12:47 +0100, Laurent Vivier wrote:
> When I try to cross compile a ppc64 kernel, it generally
> fails on the VDSO stage. This is true for powerpc64 cross-
> compiler, but also when I try to build a ppc64le kernel
> on a ppc64 host.
> 
> VDSO64L fails:
> 
>   VDSO64L arch/powerpc/kernel/vdso64/vdso64.so.dbg
> /usr/bin/powerpc64-linux-gnu-ld: arch/powerpc/kernel/vdso64/sigtramp.o:
> file class ELFCLASS64 incompatible with ELFCLASS32
> /usr/bin/powerpc64-linux-gnu-ld: final link failed: File in wrong format
> 
> This fails because gcc calls "collect2" with
> "--oformat elf32-powerpcle" with ppc64 objects, without the
> "--oformat" ld works well because it use the format of the
> first object as output format.
> 
> As this case is correctly managed to build the other kernel
> objects, this patch replaces $(GCC) by $(LD) to generate the
> VDSO objects.

I cross-compile ppc64 kernels and have not seen this problem.  I do need to 
pass in -m64 as part of $(CC) if it's not the toolchain default, which is not 
nice, but the proper fix for that is to add -m64 in the makefiles -- and if I 
don't it fails way before VDSO.

Why is GCC building ppc64 object files but telling the linker --oformat elf32-
powerpcle?  Are different options somehow being passed to GCC in one case 
versus the other?

-Scott

Re: [PATCH] powerpc: allow cross-compilation of ppc64 kernel

[Laurent Vivier]

Le 06/11/2015 22:09, Scott Wood a écrit :
> On Thu, 2015-11-05 at 12:47 +0100, Laurent Vivier wrote:
>> When I try to cross compile a ppc64 kernel, it generally
>> fails on the VDSO stage. This is true for powerpc64 cross-
>> compiler, but also when I try to build a ppc64le kernel
>> on a ppc64 host.
>>
>> VDSO64L fails:
>>
>>   VDSO64L arch/powerpc/kernel/vdso64/vdso64.so.dbg
>> /usr/bin/powerpc64-linux-gnu-ld: arch/powerpc/kernel/vdso64/sigtramp.o:
>> file class ELFCLASS64 incompatible with ELFCLASS32
>> /usr/bin/powerpc64-linux-gnu-ld: final link failed: File in wrong format
>>
>> This fails because gcc calls "collect2" with
>> "--oformat elf32-powerpcle" with ppc64 objects, without the
>> "--oformat" ld works well because it use the format of the
>> first object as output format.
>>
>> As this case is correctly managed to build the other kernel
>> objects, this patch replaces $(GCC) by $(LD) to generate the
>> VDSO objects.
> 
> I cross-compile ppc64 kernels and have not seen this problem.  I do need to 
> pass in -m64 as part of $(CC) if it's not the toolchain default, which is not 
> nice, but the proper fix for that is to add -m64 in the makefiles -- and if I 
> don't it fails way before VDSO.
> 
> Why is GCC building ppc64 object files but telling the linker --oformat elf32-
> powerpcle?  Are different options somehow being passed to GCC in one case 
> versus the other?

In fact, for all the other parts of the kernel, gcc is called with
"-mlittle-endian -m64", ld with "-EL -m elf64lppc", and thus generates
the good objects and calls ld with the good options ("elf64lppc"). I
think gcc is never used to link, only to compile.
This, I think, comes from:

arch/powerpc/Makefile:

ifeq ($(CONFIG_CPU_LITTLE_ENDIAN),y)
override CC     += -mlittle-endian
override LD     += -EL
...
ifeq ($(HAS_BIARCH),y)
override CC     += -m$(CONFIG_WORD_SIZE)
override LD     += -m elf$(CONFIG_WORD_SIZE)$(LDEMULATION)

But in the case of vdso64, ld command is gcc:

arch/powerpc/kernel/vdso64/Makefile:

	cmd_vdso64ld = $(CC) $(c_flags) -Wl,-T $^ -o $@

So to  link, we use "gcc -mlittle-endian -m64"

and strace of "gcc -mlittle-endian -m64" gives me:

"/usr/libexec/gcc/ppc64-redhat-linux/5.1.1/collect2", [ ... "--oformat",
"elf32-powerpcle", "-m", "elf64lppc",...

So the format used to link is by default "elf32-powerpcle" (with the
emulation elf64lppc given by "-mlittle-endian -m64", I agree it seems
strange).

I think this is coming from the configuration of my gcc, "-dumpspecs"
gives me:

*link_target:
%{mlittle|mlittle-endian: --oformat
elf32-powerpcle;mbig|mbig-endian:;mcall-i960-old: --oformat
elf32-powerpcle;:}

When "ld" is called without "--oformat" it takes the format of the first
processed object ("elf64-powerpcle" in our case, it's why it works for
the other binaries of the kernel).

So at this point, I can:

1- either fix my compiler,
2- or fix the vdso64 linker command.

As all the others objects of the kernel are generated and linked
correctly with this build env, there is no reason to not choose 2- as 1-
is much more complex (at leasts for me = rebuild gcc whereas I just want
to build the kernel).

But, more generally, what I'm wondering is why we are using CC instead
of LD to link objects...

Laurent

Re: [PATCH] powerpc: allow cross-compilation of ppc64 kernel

[Segher Boessenkool]

On Fri, Nov 06, 2015 at 03:09:40PM -0600, Scott Wood wrote:
> Why is GCC building ppc64 object files but telling the linker --oformat elf32-
> powerpcle?  Are different options somehow being passed to GCC in one case 
> versus the other?

This was changed for GCC 6 in <http://gcc.gnu.org/r227397>, could one of
you check if that fixes this problem?  If so we'll need to backport it.


Segher

Re: [PATCH] powerpc: allow cross-compilation of ppc64 kernel

[Benjamin Herrenschmidt]

On Fri, 2015-11-06 at 15:09 -0600, Scott Wood wrote:
> On Thu, 2015-11-05 at 12:47 +0100, Laurent Vivier wrote:
> > When I try to cross compile a ppc64 kernel, it generally
> > fails on the VDSO stage. This is true for powerpc64 cross-
> > compiler, but also when I try to build a ppc64le kernel
> > on a ppc64 host.
> > 
> > VDSO64L fails:
> > 
> >   VDSO64L arch/powerpc/kernel/vdso64/vdso64.so.dbg
> > /usr/bin/powerpc64-linux-gnu-ld:
> > arch/powerpc/kernel/vdso64/sigtramp.o:
> > file class ELFCLASS64 incompatible with ELFCLASS32
> > /usr/bin/powerpc64-linux-gnu-ld: final link failed: File in wrong
> > format
> > 
> > This fails because gcc calls "collect2" with
> > "--oformat elf32-powerpcle" with ppc64 objects, without the
> > "--oformat" ld works well because it use the format of the
> > first object as output format.
> > 
> > As this case is correctly managed to build the other kernel
> > objects, this patch replaces $(GCC) by $(LD) to generate the
> > VDSO objects.

This is LE ? I think that's a bug in binutils or gcc ... I remember we
fought that a while ago for the openpower builds. It might have been
fixed in upstream toolchain.

Cheers,
Ben.

Re: [PATCH] powerpc: allow cross-compilation of ppc64 kernel

[Scott Wood]

On Fri, 2015-11-06 at 23:22 +0100, Laurent Vivier wrote:
> Le 06/11/2015 22:09, Scott Wood a écrit :
> > On Thu, 2015-11-05 at 12:47 +0100, Laurent Vivier wrote:
> > > When I try to cross compile a ppc64 kernel, it generally
> > > fails on the VDSO stage. This is true for powerpc64 cross-
> > > compiler, but also when I try to build a ppc64le kernel
> > > on a ppc64 host.
> > > 
> > > VDSO64L fails:
> > > 
> > >   VDSO64L arch/powerpc/kernel/vdso64/vdso64.so.dbg
> > > /usr/bin/powerpc64-linux-gnu-ld: arch/powerpc/kernel/vdso64/sigtramp.o:
> > > file class ELFCLASS64 incompatible with ELFCLASS32
> > > /usr/bin/powerpc64-linux-gnu-ld: final link failed: File in wrong format
> > > 
> > > This fails because gcc calls "collect2" with
> > > "--oformat elf32-powerpcle" with ppc64 objects, without the
> > > "--oformat" ld works well because it use the format of the
> > > first object as output format.
> > > 
> > > As this case is correctly managed to build the other kernel
> > > objects, this patch replaces $(GCC) by $(LD) to generate the
> > > VDSO objects.
> > 
> > I cross-compile ppc64 kernels and have not seen this problem.  I do need 
> > to 
> > pass in -m64 as part of $(CC) if it's not the toolchain default, which is 
> > not 
> > nice, but the proper fix for that is to add -m64 in the makefiles -- and 
> > if I 
> > don't it fails way before VDSO.
> > 
> > Why is GCC building ppc64 object files but telling the linker --oformat 
> > elf32-
> > powerpcle?  Are different options somehow being passed to GCC in one case 
> > versus the other?
> 
> In fact, for all the other parts of the kernel, gcc is called with
> "-mlittle-endian -m64", ld with "-EL -m elf64lppc", and thus generates
> the good objects and calls ld with the good options ("elf64lppc"). I
> think gcc is never used to link, only to compile.
> This, I think, comes from:
> 
> arch/powerpc/Makefile:
> 
> ifeq ($(CONFIG_CPU_LITTLE_ENDIAN),y)
> override CC     += -mlittle-endian
> override LD     += -EL
> ...
> ifeq ($(HAS_BIARCH),y)
> override CC     += -m$(CONFIG_WORD_SIZE)
> override LD     += -m elf$(CONFIG_WORD_SIZE)$(LDEMULATION)

OK, that works if I completely remove the CC environment setting rather than 
just remove the -m64 from it.  It looks like that's been there longer than 
I've been building 64-bit kernels, so I wonder what originally prompted me to 
add the environment setting.

Yes, the kernel itself is linked with ld rather than gcc, but that's unusual. 
 There's nothing wrong with using gcc to link.

> So at this point, I can:
> 
> 1- either fix my compiler,
> 2- or fix the vdso64 linker command.

Do #1, as that's where the bug is, and if you work around it you'll have the 
same problem building other projects if they happen to specify the endianness 
explicitly.

-Scott

Re: [PATCH] powerpc: allow cross-compilation of ppc64 kernel

[Segher Boessenkool]

On Fri, Nov 06, 2015 at 04:55:49PM -0600, Segher Boessenkool wrote:
> On Fri, Nov 06, 2015 at 03:09:40PM -0600, Scott Wood wrote:
> > Why is GCC building ppc64 object files but telling the linker --oformat elf32-
> > powerpcle?  Are different options somehow being passed to GCC in one case 
> > versus the other?
> 
> This was changed for GCC 6 in <http://gcc.gnu.org/r227397>, could one of
> you check if that fixes this problem?  If so we'll need to backport it.

Actually it already has been backported, r228089.


Segher

Re: [PATCH] powerpc: allow cross-compilation of ppc64 kernel

[Laurent Vivier]

Le 07/11/2015 00:32, Segher Boessenkool a écrit :
> On Fri, Nov 06, 2015 at 04:55:49PM -0600, Segher Boessenkool wrote:
>> On Fri, Nov 06, 2015 at 03:09:40PM -0600, Scott Wood wrote:
>>> Why is GCC building ppc64 object files but telling the linker --oformat elf32-
>>> powerpcle?  Are different options somehow being passed to GCC in one case 
>>> versus the other?
>>
>> This was changed for GCC 6 in <http://gcc.gnu.org/r227397>, could one of
>> you check if that fixes this problem?  If so we'll need to backport it.
> 
> Actually it already has been backported, r228089.

Thank you, I'm going to see if it can be included in the distro I use.

Laurent

Re: [PATCH] powerpc: allow cross-compilation of ppc64 kernel

[Laurent Vivier]

Le 07/11/2015 00:12, Benjamin Herrenschmidt a écrit :
> On Fri, 2015-11-06 at 15:09 -0600, Scott Wood wrote:
>> On Thu, 2015-11-05 at 12:47 +0100, Laurent Vivier wrote:
>>> When I try to cross compile a ppc64 kernel, it generally
>>> fails on the VDSO stage. This is true for powerpc64 cross-
>>> compiler, but also when I try to build a ppc64le kernel
>>> on a ppc64 host.
>>>
>>> VDSO64L fails:
>>>
>>>   VDSO64L arch/powerpc/kernel/vdso64/vdso64.so.dbg
>>> /usr/bin/powerpc64-linux-gnu-ld:
>>> arch/powerpc/kernel/vdso64/sigtramp.o:
>>> file class ELFCLASS64 incompatible with ELFCLASS32
>>> /usr/bin/powerpc64-linux-gnu-ld: final link failed: File in wrong
>>> format
>>>
>>> This fails because gcc calls "collect2" with
>>> "--oformat elf32-powerpcle" with ppc64 objects, without the
>>> "--oformat" ld works well because it use the format of the
>>> first object as output format.
>>>
>>> As this case is correctly managed to build the other kernel
>>> objects, this patch replaces $(GCC) by $(LD) to generate the
>>> VDSO objects.
> 
> This is LE ? I think that's a bug in binutils or gcc ... I remember we
> fought that a while ago for the openpower builds. It might have been
> fixed in upstream toolchain.

Yes, Segher has given me the commit id. It is just what I need.

Laurent

Re: [PATCH] powerpc: allow cross-compilation of ppc64 kernel

[Laurent Vivier]

Le 07/11/2015 00:24, Scott Wood a écrit :
> On Fri, 2015-11-06 at 23:22 +0100, Laurent Vivier wrote:
>> Le 06/11/2015 22:09, Scott Wood a écrit :
>>> On Thu, 2015-11-05 at 12:47 +0100, Laurent Vivier wrote:
>>>> When I try to cross compile a ppc64 kernel, it generally
>>>> fails on the VDSO stage. This is true for powerpc64 cross-
>>>> compiler, but also when I try to build a ppc64le kernel
>>>> on a ppc64 host.
>>>>
>>>> VDSO64L fails:
>>>>
>>>>   VDSO64L arch/powerpc/kernel/vdso64/vdso64.so.dbg
>>>> /usr/bin/powerpc64-linux-gnu-ld: arch/powerpc/kernel/vdso64/sigtramp.o:
>>>> file class ELFCLASS64 incompatible with ELFCLASS32
>>>> /usr/bin/powerpc64-linux-gnu-ld: final link failed: File in wrong format
>>>>
>>>> This fails because gcc calls "collect2" with
>>>> "--oformat elf32-powerpcle" with ppc64 objects, without the
>>>> "--oformat" ld works well because it use the format of the
>>>> first object as output format.
>>>>
>>>> As this case is correctly managed to build the other kernel
>>>> objects, this patch replaces $(GCC) by $(LD) to generate the
>>>> VDSO objects.
>>>
>>> I cross-compile ppc64 kernels and have not seen this problem.  I do need 
>>> to 
>>> pass in -m64 as part of $(CC) if it's not the toolchain default, which is 
>>> not 
>>> nice, but the proper fix for that is to add -m64 in the makefiles -- and 
>>> if I 
>>> don't it fails way before VDSO.
>>>
>>> Why is GCC building ppc64 object files but telling the linker --oformat 
>>> elf32-
>>> powerpcle?  Are different options somehow being passed to GCC in one case 
>>> versus the other?
>>
>> In fact, for all the other parts of the kernel, gcc is called with
>> "-mlittle-endian -m64", ld with "-EL -m elf64lppc", and thus generates
>> the good objects and calls ld with the good options ("elf64lppc"). I
>> think gcc is never used to link, only to compile.
>> This, I think, comes from:
>>
>> arch/powerpc/Makefile:
>>
>> ifeq ($(CONFIG_CPU_LITTLE_ENDIAN),y)
>> override CC     += -mlittle-endian
>> override LD     += -EL
>> ...
>> ifeq ($(HAS_BIARCH),y)
>> override CC     += -m$(CONFIG_WORD_SIZE)
>> override LD     += -m elf$(CONFIG_WORD_SIZE)$(LDEMULATION)
> 
> OK, that works if I completely remove the CC environment setting rather than 
> just remove the -m64 from it.  It looks like that's been there longer than 
> I've been building 64-bit kernels, so I wonder what originally prompted me to 
> add the environment setting.
> 
> Yes, the kernel itself is linked with ld rather than gcc, but that's unusual. 
>  There's nothing wrong with using gcc to link.
> 
>> So at this point, I can:
>>
>> 1- either fix my compiler,
>> 2- or fix the vdso64 linker command.
> 
> Do #1, as that's where the bug is, and if you work around it you'll have the 
> same problem building other projects if they happen to specify the endianness 
> explicitly.

Thank you Scott. With the help of the comment from Segher, I can choose
#1 now :)

Laurent

Re: [PATCH] powerpc: allow cross-compilation of ppc64 kernel

[Michael Ellerman]

On Sat, 2015-11-07 at 12:35 +0100, Laurent Vivier wrote:
> Le 07/11/2015 00:24, Scott Wood a écrit :
> > On Fri, 2015-11-06 at 23:22 +0100, Laurent Vivier wrote:
> > > Le 06/11/2015 22:09, Scott Wood a écrit :
> > > > On Thu, 2015-11-05 at 12:47 +0100, Laurent Vivier wrote:
> > > > > When I try to cross compile a ppc64 kernel, it generally
> > > > > fails on the VDSO stage. This is true for powerpc64 cross-
> > > > > compiler, but also when I try to build a ppc64le kernel
> > > > > on a ppc64 host.
> > > > > 
> > > > > VDSO64L fails:
> > > > > 
> > > > >   VDSO64L arch/powerpc/kernel/vdso64/vdso64.so.dbg
> > > > > /usr/bin/powerpc64-linux-gnu-ld: arch/powerpc/kernel/vdso64/sigtramp.o:
> > > > > file class ELFCLASS64 incompatible with ELFCLASS32
> > > > > /usr/bin/powerpc64-linux-gnu-ld: final link failed: File in wrong format
> > > > > 
> > > > > This fails because gcc calls "collect2" with
> > > > > "--oformat elf32-powerpcle" with ppc64 objects, without the
> > > > > "--oformat" ld works well because it use the format of the
> > > > > first object as output format.
> > > > > 
> > > > > As this case is correctly managed to build the other kernel
> > > > > objects, this patch replaces $(GCC) by $(LD) to generate the
> > > > > VDSO objects.
> > 
> > > So at this point, I can:
> > > 
> > > 1- either fix my compiler,
> > > 2- or fix the vdso64 linker command.
> > 
> Thank you Scott. With the help of the comment from Segher, I can choose
> #1 now :)

What distro/toolchain did you hit this on?

I actually wrote this same patch a while back, but from memory it broke in some
configurations. So I never merged it. I'll see if I can find my notes and work
out exactly why it didn't work, maybe we should merge it anyway, even though
the real bug is a toolchain bug.

cheers

Re: [PATCH] powerpc: allow cross-compilation of ppc64 kernel

[Laurent Vivier]

Le 10/11/2015 01:29, Michael Ellerman a écrit :
> On Sat, 2015-11-07 at 12:35 +0100, Laurent Vivier wrote:
>> Le 07/11/2015 00:24, Scott Wood a écrit :
>>> On Fri, 2015-11-06 at 23:22 +0100, Laurent Vivier wrote:
>>>> Le 06/11/2015 22:09, Scott Wood a écrit :
>>>>> On Thu, 2015-11-05 at 12:47 +0100, Laurent Vivier wrote:
>>>>>> When I try to cross compile a ppc64 kernel, it generally
>>>>>> fails on the VDSO stage. This is true for powerpc64 cross-
>>>>>> compiler, but also when I try to build a ppc64le kernel
>>>>>> on a ppc64 host.
>>>>>>
>>>>>> VDSO64L fails:
>>>>>>
>>>>>>   VDSO64L arch/powerpc/kernel/vdso64/vdso64.so.dbg
>>>>>> /usr/bin/powerpc64-linux-gnu-ld: arch/powerpc/kernel/vdso64/sigtramp.o:
>>>>>> file class ELFCLASS64 incompatible with ELFCLASS32
>>>>>> /usr/bin/powerpc64-linux-gnu-ld: final link failed: File in wrong format
>>>>>>
>>>>>> This fails because gcc calls "collect2" with
>>>>>> "--oformat elf32-powerpcle" with ppc64 objects, without the
>>>>>> "--oformat" ld works well because it use the format of the
>>>>>> first object as output format.
>>>>>>
>>>>>> As this case is correctly managed to build the other kernel
>>>>>> objects, this patch replaces $(GCC) by $(LD) to generate the
>>>>>> VDSO objects.
>>>
>>>> So at this point, I can:
>>>>
>>>> 1- either fix my compiler,
>>>> 2- or fix the vdso64 linker command.
>>>
>> Thank you Scott. With the help of the comment from Segher, I can choose
>> #1 now :)
> 
> What distro/toolchain did you hit this on?

fedora 23.

> I actually wrote this same patch a while back, but from memory it broke in some
> configurations. So I never merged it. I'll see if I can find my notes and work
> out exactly why it didn't work, maybe we should merge it anyway, even though
> the real bug is a toolchain bug.

On fedora, there is also a bug in binutils configuration where ld is not
able to manage ppc64le binaries on ppc64 host, but I've already sent a
fix for that and it is now in rawhide (will be in fedora 24).

https://bugzilla.redhat.com/show_bug.cgi?id=1275709

Laurent