how can we test the hexagon port in mainline

how can we test the hexagon port in mainline

[Christoph Hellwig]

Hi all,

the oldest supported gcc version in mainline is gcc 4.9.  But the only
hexagon crosscompiler I can find is the one Arnds website points to here:

https://mirrors.edge.kernel.org/pub/tools/crosstool/

which is a non-upstream gcc 4.6.1 port.  How are we supposed to even
build test hexagon code?

RE: how can we test the hexagon port in mainline

[Brian Cain]

> -----Original Message-----
> From: Christoph Hellwig <hch@lst.de>
...
> 
> Hi all,
> 
> the oldest supported gcc version in mainline is gcc 4.9.  But the only
> hexagon crosscompiler I can find is the one Arnds website points to here:
> 
> https://mirrors.edge.kernel.org/pub/tools/crosstool/
> 
> which is a non-upstream gcc 4.6.1 port.  How are we supposed to even
> build test hexagon code?

We have provided a clang-12-based toolchain here:

https://codelinaro.jfrog.io/artifactory/codelinaro-qemu/2021-05-12/clang+llv
m-12.0.0-cross-hexagon-unknown-linux-musl.tar.xz

Could we update https://mirrors.edge.kernel.org/pub/tools/crosstool/ to
point here?

-Brian

Re: how can we test the hexagon port in mainline

[Arnd Bergmann]

On Wed, Jun 23, 2021 at 4:39 PM Brian Cain <bcain@codeaurora.org> wrote:
>
> > -----Original Message-----
> > From: Christoph Hellwig <hch@lst.de>
> ...
> >
> > Hi all,
> >
> > the oldest supported gcc version in mainline is gcc 4.9.  But the only
> > hexagon crosscompiler I can find is the one Arnds website points to here:
> >
> > https://mirrors.edge.kernel.org/pub/tools/crosstool/
> >
> > which is a non-upstream gcc 4.6.1 port.  How are we supposed to even
> > build test hexagon code?
>
> We have provided a clang-12-based toolchain here:
>
> https://codelinaro.jfrog.io/artifactory/codelinaro-qemu/2021-05-12/clang+llv
> m-12.0.0-cross-hexagon-unknown-linux-musl.tar.xz

How close is this to the mainline clang builds from https://apt.llvm.org/?

The last time we talked, there were still a couple of bugs that prevented
me from building an unmodified kernel with an unmodified clang toolchain,
but I think you planned to address those. Are any changes left that did
not make it into the 12.0 release?

> Could we update https://mirrors.edge.kernel.org/pub/tools/crosstool/ to
> point here?

I'm happy to edit the index page on
https://mirrors.edge.kernel.org/pub/tools/crosstool,
but could you provide a stable URL that will keep pointing to the latest version
in the future so I don't have to update it for each new build?

        Arnd

Re: how can we test the hexagon port in mainline

['Christoph Hellwig']

On Wed, Jun 23, 2021 at 09:39:21AM -0500, Brian Cain wrote:
> > -----Original Message-----
> > From: Christoph Hellwig <hch@lst.de>
> ...
> > 
> > Hi all,
> > 
> > the oldest supported gcc version in mainline is gcc 4.9.  But the only
> > hexagon crosscompiler I can find is the one Arnds website points to here:
> > 
> > https://mirrors.edge.kernel.org/pub/tools/crosstool/
> > 
> > which is a non-upstream gcc 4.6.1 port.  How are we supposed to even
> > build test hexagon code?
> 
> We have provided a clang-12-based toolchain here:
> 
> https://codelinaro.jfrog.io/artifactory/codelinaro-qemu/2021-05-12/clang+llv
> m-12.0.0-cross-hexagon-unknown-linux-musl.tar.xz

Do you have something that is just a small compiler like Arnd's gcc
build?  That tarball is already 264MB with xz compression, which does
not compare very favorablt to the 7MB for gcc.  And I fear I don't
really have space for that on my setups..

RE: how can we test the hexagon port in mainline

[Brian Cain]

> -----Original Message-----
> From: Arnd Bergmann <arnd@arndb.de>
> Sent: Wednesday, June 23, 2021 9:54 AM
> To: Brian Cain <bcain@codeaurora.org>
> Cc: Christoph Hellwig <hch@lst.de>; Linus Torvalds <torvalds@linux-
> foundation.org>; Sid Manning <sidneym@codeaurora.org>; open
> list:QUALCOMM HEXAGON... <linux-hexagon@vger.kernel.org>; Linux Kernel
> Mailing List <linux-kernel@vger.kernel.org>
> Subject: Re: how can we test the hexagon port in mainline
> 
> On Wed, Jun 23, 2021 at 4:39 PM Brian Cain <bcain@codeaurora.org> wrote:
> >
> > > -----Original Message-----
> > > From: Christoph Hellwig <hch@lst.de>
> > ...
> > >
> > > Hi all,
> > >
> > > the oldest supported gcc version in mainline is gcc 4.9.  But the only
> > > hexagon crosscompiler I can find is the one Arnds website points to here:
> > >
> > > https://mirrors.edge.kernel.org/pub/tools/crosstool/
> > >
> > > which is a non-upstream gcc 4.6.1 port.  How are we supposed to even
> > > build test hexagon code?
> >
> > We have provided a clang-12-based toolchain here:
> >
> > https://codelinaro.jfrog.io/artifactory/codelinaro-qemu/2021-05-
> 12/clang+llv
> > m-12.0.0-cross-hexagon-unknown-linux-musl.tar.xz
> 
> How close is this to the mainline clang builds from https://apt.llvm.org/?

This is built from llvm-project 12.0.0 but also contains userspace toolchain features like musl and qemu-linux-user for hexagon.  I suppose for the purposes of kernel builds they're equivalent.

> The last time we talked, there were still a couple of bugs that prevented
> me from building an unmodified kernel with an unmodified clang toolchain,
> but I think you planned to address those. Are any changes left that did
> not make it into the 12.0 release?

I think those were identified building with clang 12.0.0 or post-12.0.0, so I don't think they'll be addressed.  Let me review and follow up on the status of those items.  I recall one workaround was to use the integrated assembler to avoid one of the toolchain bugs.

> > Could we update https://mirrors.edge.kernel.org/pub/tools/crosstool/ to
> > point here?
> 
> I'm happy to edit the index page on
> https://mirrors.edge.kernel.org/pub/tools/crosstool,
> but could you provide a stable URL that will keep pointing to the latest
> version
> in the future so I don't have to update it for each new build?

I don't know how to do that with the mechanisms we have so far.  But I will see about getting one created and then hopefully you could use that.

-Brian

Re: how can we test the hexagon port in mainline

['Christoph Hellwig']

It seem like it still isn't complete enought for a kernel build, though:

$ export CROSS_COMPILE=/opt/clang+llvm-12.0.0-cross-hexagon-unknown-linux-musl/x86_64-linux-gnu/bin/hexagon-unknown-linux-musl- 
$ make ARCH=hexagon LLVM=1 oldconfig
...
scripts/Kconfig.include:40: linker 'ld.lld' not found
make[1]: *** [scripts/kconfig/Makefile:77: oldconfig] Error 1
make: *** [Makefile:614: oldconfig] Error 2

Re: how can we test the hexagon port in mainline

[Arnd Bergmann]

On Wed, Jun 23, 2021 at 5:17 PM Christoph Hellwig <hch@lst.de> wrote:
>
> It seem like it still isn't complete enought for a kernel build, though:
>
> $ export CROSS_COMPILE=/opt/clang+llvm-12.0.0-cross-hexagon-unknown-linux-musl/x86_64-linux-gnu/bin/hexagon-unknown-linux-musl-
> $ make ARCH=hexagon LLVM=1 oldconfig
> ...
> scripts/Kconfig.include:40: linker 'ld.lld' not found

I tried this using the prebuilt binaries from apt.llvm.org:

$ make ARCH=hexagon LLVM=1 O=obj-hexagon CROSS_COMPILE=hexagon-linux-
LLVM_IAS=1 CC=llvm-12 defconfig modules vmlinux
<stdin>:1515:2: warning: syscall clone3 not implemented [-W#warnings]
#warning syscall clone3 not implemented
 ^
1 warning generated.

Doing the same thing with allmodconfig results in an internal error
with clang-12
while compiling kernel/locking/lockdep.c. Same thing with clang-13.
After turning
off lock debugging, it seems fine.

       Arnd

Re: how can we test the hexagon port in mainline

[Nathan Chancellor]

On Wed, Jun 23, 2021 at 05:35:38PM +0200, Arnd Bergmann wrote:
> On Wed, Jun 23, 2021 at 5:17 PM Christoph Hellwig <hch@lst.de> wrote:
> >
> > It seem like it still isn't complete enought for a kernel build, though:
> >
> > $ export CROSS_COMPILE=/opt/clang+llvm-12.0.0-cross-hexagon-unknown-linux-musl/x86_64-linux-gnu/bin/hexagon-unknown-linux-musl-
> > $ make ARCH=hexagon LLVM=1 oldconfig
> > ...
> > scripts/Kconfig.include:40: linker 'ld.lld' not found
> 
> I tried this using the prebuilt binaries from apt.llvm.org:
> 
> $ make ARCH=hexagon LLVM=1 O=obj-hexagon CROSS_COMPILE=hexagon-linux-
> LLVM_IAS=1 CC=llvm-12 defconfig modules vmlinux
> <stdin>:1515:2: warning: syscall clone3 not implemented [-W#warnings]
> #warning syscall clone3 not implemented
>  ^
> 1 warning generated.
> 
> Doing the same thing with allmodconfig results in an internal error
> with clang-12
> while compiling kernel/locking/lockdep.c. Same thing with clang-13.
> After turning
> off lock debugging, it seems fine.

I've reported this upstream with you on CC:

https://bugs.llvm.org/show_bug.cgi?id=50838

Christoph, that toolchain should work (I had to install libtinfo5 and
libc++1-7 on Debian Buster):

$ export PATH=$HOME/tmp/clang+llvm-12.0.0-cross-hexagon-unknown-linux-musl/x86_64-linux-gnu/bin:$PATH

$ make -skj"$(nproc)" ARCH=hexagon CROSS_COMPILE=hexagon-unknown-linux-musl LLVM=1 LLVM_IAS=1 defconfig all

CROSS_COMPILE is really only used to set the '--target=' flag when
LLVM=1 LLVM_IAS=1 are used, it can even be just 'hexagon-linux'.

As Arnd points out, https://apt.llvm.org is also an option, that is what
we use in our continuous integration:

https://github.com/ClangBuiltLinux/continuous-integration2

Cheers,
Nathan

Re: how can we test the hexagon port in mainline

[Christoph Hellwig]

On Wed, Jun 23, 2021 at 09:06:41PM -0700, Nathan Chancellor wrote:
> On Wed, Jun 23, 2021 at 05:35:38PM +0200, Arnd Bergmann wrote:
> > On Wed, Jun 23, 2021 at 5:17 PM Christoph Hellwig <hch@lst.de> wrote:
> > >
> > > It seem like it still isn't complete enought for a kernel build, though:
> > >
> > > $ export CROSS_COMPILE=/opt/clang+llvm-12.0.0-cross-hexagon-unknown-linux-musl/x86_64-linux-gnu/bin/hexagon-unknown-linux-musl-
> > > $ make ARCH=hexagon LLVM=1 oldconfig
> > > ...
> > > scripts/Kconfig.include:40: linker 'ld.lld' not found
> > 
> > I tried this using the prebuilt binaries from apt.llvm.org:
> > 
> > $ make ARCH=hexagon LLVM=1 O=obj-hexagon CROSS_COMPILE=hexagon-linux-
> > LLVM_IAS=1 CC=llvm-12 defconfig modules vmlinux
> > <stdin>:1515:2: warning: syscall clone3 not implemented [-W#warnings]
> > #warning syscall clone3 not implemented
> >  ^
> > 1 warning generated.
> > 
> > Doing the same thing with allmodconfig results in an internal error
> > with clang-12
> > while compiling kernel/locking/lockdep.c. Same thing with clang-13.
> > After turning
> > off lock debugging, it seems fine.
> 
> I've reported this upstream with you on CC:
> 
> https://bugs.llvm.org/show_bug.cgi?id=50838

I've not actually got any mail from that Cc..

> Christoph, that toolchain should work (I had to install libtinfo5 and
> libc++1-7 on Debian Buster):
> 
> $ export PATH=$HOME/tmp/clang+llvm-12.0.0-cross-hexagon-unknown-linux-musl/x86_64-linux-gnu/bin:$PATH
> 
> $ make -skj"$(nproc)" ARCH=hexagon CROSS_COMPILE=hexagon-unknown-linux-musl LLVM=1 LLVM_IAS=1 defconfig all

hch@brick:~/work/linux$ make -j4 ARCH=hexagon
CROSS_COMPILE=hexagon-unknown-linux-musl LLVM=1 LLVM_IAS=1 defconfig all
HOSTCC  scripts/basic/fixdep
clang: error while loading shared libraries: libtinfo.so.5: cannot open shared object file: No such file or directory

Re: how can we test the hexagon port in mainline

[Nathan Chancellor]

Hi Christoph,

On 7/7/2021 7:10 AM, Christoph Hellwig wrote:
> On Wed, Jun 23, 2021 at 09:06:41PM -0700, Nathan Chancellor wrote:
>> I've reported this upstream with you on CC:
>>
>> https://bugs.llvm.org/show_bug.cgi?id=50838
> 
> I've not actually got any mail from that Cc..

Sorry, I directed that at Arnd. I should have kept you in the loop too.

>> Christoph, that toolchain should work (I had to install libtinfo5 and
>> libc++1-7 on Debian Buster):
>>
>> $ export PATH=$HOME/tmp/clang+llvm-12.0.0-cross-hexagon-unknown-linux-musl/x86_64-linux-gnu/bin:$PATH
>>
>> $ make -skj"$(nproc)" ARCH=hexagon CROSS_COMPILE=hexagon-unknown-linux-musl LLVM=1 LLVM_IAS=1 defconfig all
> 
> hch@brick:~/work/linux$ make -j4 ARCH=hexagon
> CROSS_COMPILE=hexagon-unknown-linux-musl LLVM=1 LLVM_IAS=1 defconfig all
> HOSTCC  scripts/basic/fixdep
> clang: error while loading shared libraries: libtinfo.so.5: cannot open shared object file: No such file or directory

Hmmm, is that with libtinfo5 installed (or whatever the ncurses-compat 
equivalent is on your distribution installed)? I had that problem on 
Debian until I insta

Brian/Sid, it might be worth flipping LLVM_ENABLE_TERMINFO to OFF during 
your cmake configuration so that there are less dynamic dependencies and 
it is easier for more people to run the toolchain. Android's clang team 
did the same thing:

https://github.com/android/ndk/issues/574

With https://reviews.llvm.org/D42055, there should not be too much of a 
sacrifice.

Cheers,
Nathan

Re: how can we test the hexagon port in mainline

[Christoph Hellwig]

On Wed, Jul 07, 2021 at 10:42:27AM -0700, Nathan Chancellor wrote:
>> hch@brick:~/work/linux$ make -j4 ARCH=hexagon
>> CROSS_COMPILE=hexagon-unknown-linux-musl LLVM=1 LLVM_IAS=1 defconfig all
>> HOSTCC  scripts/basic/fixdep
>> clang: error while loading shared libraries: libtinfo.so.5: cannot open shared object file: No such file or directory
>
> Hmmm, is that with libtinfo5 installed (or whatever the ncurses-compat 
> equivalent is on your distribution installed)? I had that problem on Debian 
> until I insta

I did install libtinfo5, which just gets me to the next error:

hch@brick:~/work/linux$ export PATH=/opt/clang+llvm-12.0.0-cross-hexagon-unknown-linux-musl/x86_64-linux-gnu/bin/:$PATH
hch@brick:~/work/linux$ make -j4 ARCH=hexagon CROSS_COMPILE=hexagon-unknown-linux-musl LLVM=1 LLVM_IAS=1 defconfig all
  HOSTCC  scripts/basic/fixdep
clang: error while loading shared libraries: libc++.so.1: cannot open shared object file: No such file or directory
make[2]: *** [scripts/Makefile.host:95: scripts/basic/fixdep] Error 127
make[1]: *** [Makefile:543: scripts_basic] Error 2
make: *** [Makefile:346: __build_one_by_one] Error 2

Re: how can we test the hexagon port in mainline

[Nick Desaulniers]

On Wed, Jul 7, 2021 at 10:27 PM Christoph Hellwig <hch@lst.de> wrote:
>
> On Wed, Jul 07, 2021 at 10:42:27AM -0700, Nathan Chancellor wrote:
> >> hch@brick:~/work/linux$ make -j4 ARCH=hexagon
> >> CROSS_COMPILE=hexagon-unknown-linux-musl LLVM=1 LLVM_IAS=1 defconfig all
> >> HOSTCC  scripts/basic/fixdep
> >> clang: error while loading shared libraries: libtinfo.so.5: cannot open shared object file: No such file or directory
> >
> > Hmmm, is that with libtinfo5 installed (or whatever the ncurses-compat
> > equivalent is on your distribution installed)? I had that problem on Debian
> > until I insta
>
> I did install libtinfo5, which just gets me to the next error:
>
> hch@brick:~/work/linux$ export PATH=/opt/clang+llvm-12.0.0-cross-hexagon-unknown-linux-musl/x86_64-linux-gnu/bin/:$PATH
> hch@brick:~/work/linux$ make -j4 ARCH=hexagon CROSS_COMPILE=hexagon-unknown-linux-musl LLVM=1 LLVM_IAS=1 defconfig all
>   HOSTCC  scripts/basic/fixdep
> clang: error while loading shared libraries: libc++.so.1: cannot open shared object file: No such file or directory

^ Nathan did mention earlier in the thread that he "had to install
libtinfo5 and libc++1-7 on Debian Buster." Emphasis on the _and
libc++_ part.

I'm not sure if that binary distribution came with a libc++.so.1; if
so, that path needs to be specified via LD_LIBRARY_PATH so that the
runtime loader can find it.  Perhaps rpath wasn't set when the clang
binary was built.

We're looking into statically linked images of clang to prevent these
kinds of games.

> make[2]: *** [scripts/Makefile.host:95: scripts/basic/fixdep] Error 127
> make[1]: *** [Makefile:543: scripts_basic] Error 2
> make: *** [Makefile:346: __build_one_by_one] Error 2
>


-- 
Thanks,
~Nick Desaulniers

RE: how can we test the hexagon port in mainline

[Brian Cain]

> -----Original Message-----
> From: Nick Desaulniers <ndesaulniers@google.com>
> Sent: Thursday, July 8, 2021 12:54 PM
> To: Christoph Hellwig <hch@lst.de>
> Cc: Nathan Chancellor <nathan@kernel.org>; Brian Cain
> <bcain@codeaurora.org>; Sid Manning <sidneym@codeaurora.org>; Arnd
> Bergmann <arnd@arndb.de>; linux-hexagon@vger.kernel.org; linux-
> kernel@vger.kernel.org; clang-built-linux@googlegroups.com
> Subject: Re: how can we test the hexagon port in mainline
> 
> On Wed, Jul 7, 2021 at 10:27 PM Christoph Hellwig <hch@lst.de> wrote:
> >
> > On Wed, Jul 07, 2021 at 10:42:27AM -0700, Nathan Chancellor wrote:
> > >> hch@brick:~/work/linux$ make -j4 ARCH=hexagon
> > >> CROSS_COMPILE=hexagon-unknown-linux-musl LLVM=1 LLVM_IAS=1
> defconfig all
> > >> HOSTCC  scripts/basic/fixdep
> > >> clang: error while loading shared libraries: libtinfo.so.5: cannot open
> shared object file: No such file or directory
> > >
> > > Hmmm, is that with libtinfo5 installed (or whatever the ncurses-compat
> > > equivalent is on your distribution installed)? I had that problem on Debian
> > > until I insta
> >
> > I did install libtinfo5, which just gets me to the next error:
> >
> > hch@brick:~/work/linux$ export PATH=/opt/clang+llvm-12.0.0-cross-
> hexagon-unknown-linux-musl/x86_64-linux-gnu/bin/:$PATH
> > hch@brick:~/work/linux$ make -j4 ARCH=hexagon
> CROSS_COMPILE=hexagon-unknown-linux-musl LLVM=1 LLVM_IAS=1 defconfig
> all
> >   HOSTCC  scripts/basic/fixdep
> > clang: error while loading shared libraries: libc++.so.1: cannot open shared
> object file: No such file or directory
> 
> ^ Nathan did mention earlier in the thread that he "had to install
> libtinfo5 and libc++1-7 on Debian Buster." Emphasis on the _and
> libc++_ part.
> 
> I'm not sure if that binary distribution came with a libc++.so.1; if
> so, that path needs to be specified via LD_LIBRARY_PATH so that the
> runtime loader can find it.  Perhaps rpath wasn't set when the clang
> binary was built.

The only libc++ builds in this distribution are the target hexagon ones.  I did not include a host x86_64 libc++.so library, but it does seem like it would be more convenient if we did.

Nathan suggested disabling the terminfo dependency, I will include that change for the next release that we produce.  Also, the upcoming clang-13 release from releases.llvm.org should contain all the necessary fixes (discussed recently) to build kernel code for hexagon.

> We're looking into statically linked images of clang to prevent these
> kinds of games.

Statically linking against libc++/libc++abi at least seems like a good idea.  Let me know if we can help.

-Brian

Re: how can we test the hexagon port in mainline

[Nick Desaulniers]

On Thu, Jul 8, 2021 at 11:36 AM Brian Cain <bcain@codeaurora.org> wrote:
>
> > From: Nick Desaulniers <ndesaulniers@google.com>
> > We're looking into statically linked images of clang to prevent these
> > kinds of games.
>
> Statically linking against libc++/libc++abi at least seems like a good idea.  Let me know if we can help.

Consider subscribing/reading/posting to
https://github.com/ClangBuiltLinux/tc-build/issues/150.
-- 
Thanks,
~Nick Desaulniers