Using GNU AS from a selfmade binutils v2.35.2

Using GNU AS from a selfmade binutils v2.35.2

[Sedat Dilek]

Hi Masahiro and ClangBuiltLinux folks,

I am trying to use binaries from GNU/binutils v2.35.2 for my
Linux-kernel builds.

Background is I am doing some testing for BTF + pahole with GCC-10 and LLVM-12.

I tried several strategies.

[ STRATEGY #1 - Use a-local-binutils.conf LDCONFIG ]

[ /etc/ld.so.conf.d/a-local-binutils.conf ]

# Selfmade GNU/binutils lib configuration
/opt/binutils/lib
- EOT -

sudo ldconfig
sudo ldconfig -v -p
...shows me above lib path is in the ldconfig-cache.

In my build-script I additionally add:

BINUTILS_BIN_PATH="/opt/binutils/bin"
if [ -d ${BINUTILS_BIN_PATH} ]; then
    export PATH="${BINUTILS_BIN_PATH}:${PATH}"
fi

That's NOT including ld.bfd from /opt/binutils/bin - not including nm,
ar, strip etc.

NOTE: Symlink: lrwxrwxrwx 1 root   root     15  5. Feb 11:10
/opt/binutils -> binutils-2.35.2

[ STRATEGY #2 - Use LD_LIBRARY_PATH ]

From my build-script:

BINUTILS_BIN_PATH="/opt/binutils/bin"
BINUTILS_LIB_PATH="/opt/binutils/lib"
if [ -d ${BINUTILS_BIN_PATH} ] && [ -d ${BINUTILS_LIB_PATH} ]; then
   export PATH="${BINUTILS_BIN_PATH}:${PATH}"
   export LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:${BINUTILS_LIB_PATH}"
fi

That's not working either.
New PATH and LD_LIBRARY_PATH are set in the user's environment variables.

[ STRATEGY #3 - Use which to get full path to binary ]

### GCC version settings
GCC_MAJOR_VER="10"

### Compiler options
CC_FOR_BUILD="clang"
CXX_FOR_BUILD="clang++"

### Linker options
##LD_FOR_BUILD="ld.bfd"
LD_FOR_BUILD=$(which ld.bfd)

### GNU tools options
# NOTE: Selfmade GNU AS v2.35.2 needs to be symlinked in /usr/bin directory
# XXX: Symlink: /usr/bin/as -> /opt/binutils-2.35.2/bin/as
HOSTAR_FOR_BUILD=$(which ar)
AR_FOR_BUILD=$(which ar)
NM_FOR_BUILD=$(which nm)
STRIP_FOR_BUILD=$(which strip)
OBJCOPY_FOR_BUILD=$(which objcopy)
OBJDUMP_FOR_BUILD=$(which objdump)
READELF_FOR_BUILD=$(which readelf)
GNU_TOOLS_OPTS="HOSTCC=${CC_FOR_BUILD} HOSTCXX=${CXX_FOR_BUILD}
HOSTLD=${LD_FOR_BUILD} HOSTAR=${HOSTAR_FOR_BUILD}"
GNU_TOOLS_OPTS="$GNU_TOOLS_OPTS CC=${CC_FOR_BUILD} LD=${LD_FOR_BUILD}
AR=${AR_FOR_BUILD} NM=${NM_FOR_BUILD} STRIP=${STRIP_FOR_BUILD}"
GNU_TOOLS_OPTS="$GNU_TOOLS_OPTS OBJCOPY=${OBJCOPY_FOR_BUILD}
OBJDUMP=${OBJDUMP_FOR_BUILD} READELF=${READELF_FOR_BUILD}"

That works - means passes all binaries from GNU binutils v2.35.2 to my
make-line.

Please NOTE that I had to symlink GNU AS v2.35.2 as I saw too late I
was using Debian's GNU AS v2.35.1 within my last builds.

AFAICS there is no more AS= assignment in the top-level Makefile.
How can I say: "Please use a different ASsembler?"

[ LDD ]

When I inspect with ldd (here: GNU AS v2.35.2):

# ldd /opt/binutils/bin/as
       linux-vdso.so.1 (0x00007ffc7f4d6000)
       libopcodes-2.35.2.so =>
/opt/binutils-2.35.2/lib/libopcodes-2.35.2.so (0x00007f11f3bcc000)
       libbfd-2.35.2.so => /opt/binutils-2.35.2/lib/libbfd-2.35.2.so
(0x00007f11f3a8b000)
       libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f11f3a44000)
       libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f11f387f000)
       libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f11f3879000)
       /lib64/ld-linux-x86-64.so.2 (0x00007f11f3de3000)

So GNU AS v2.35.2 is loading from the correct places.

Is the symlink in /opt directory a problem?
binutils -> binutils-2.35.2

Can someone comment and give me a hint?

Thanks in advance.

Regards,
- Sedat -

Re: Using GNU AS from a selfmade binutils v2.35.2

[Nathan Chancellor]

On Fri, Feb 05, 2021 at 11:56:55AM +0100, Sedat Dilek wrote:
> Hi Masahiro and ClangBuiltLinux folks,
> 
> I am trying to use binaries from GNU/binutils v2.35.2 for my
> Linux-kernel builds.
> 
> Background is I am doing some testing for BTF + pahole with GCC-10 and LLVM-12.

TL;DR: GCC will use the first "as" that is found in your PATH while
clang will:
1. Use the "as" present in the folder that it is installed in.
2. Use the first "as" that is found in your PATH.

To override clang's behavior, you can pass '--prefix=' to it.

GCC:

$ echo | gcc -### -c -x assembler -o /dev/null -
...
 as --64 -o /dev/null -
...

You can prove that this works with a symlink to /bin/false.

$ cd "$(mktemp -d)"

$ ln -s /bin/false as

$ echo | gcc -c -x assembler -o /dev/null -

$ echo ${?}
0

$ echo | PATH=${PWD}:${PATH} gcc -c -x assembler -o /dev/null -

$ echo ${?}
1

As you will notice, PATH has no affect when there is an "as" binary in
the same location as "clang":

$ which clang
/usr/bin/clang

$ which as
/usr/bin/as

$ echo | clang -### -no-integrated-as -c -x assembler -o /dev/null -
...
 "/usr/bin/as" "--64" "-o" "/dev/null" "-"

$ echo | PATH=${PWD}:${PATH} clang -### -no-integrated-as -c -x assembler -o /dev/null -
...
 "/usr/bin/as" "--64" "-o" "/dev/null" "-"

But '--prefix=' does:

$ echo | clang -### --prefix=${PWD} -no-integrated-as -c -x assembler -o /dev/null -
...
 "/tmp/tmp.B8BdCWZJTL/as" "--64" "-o" "/dev/null" "-"

When there is no "as" binary in the same location as "clang", PATH works:

$ echo | /usr/lib/llvm-10/bin/clang -### -no-integrated-as -c -x assembler -o /dev/null -
...
 "/usr/bin/as" "--64" "-o" "/dev/null" "-"

$ echo | PATH=${PWD}:${PATH} /usr/lib/llvm-10/bin/clang -### -no-integrated-as -c -x assembler -o /dev/null -
...
 "/tmp/tmp.B8BdCWZJTL/as" "--64" "-o" "/dev/null" "-"

If you want to use a separate binutils while building the kernel with
clang, you can:

1. Symlink your LLVM and binutils binaries into one folder.
2. Pass '--prefix=' via KCFLAGS:

make KCFLAGS=--prefix=<binutils_bin_folder>

It is entirely possible that '--prefix=' should always be present though:

diff --git a/Makefile b/Makefile
index f5842126e89d..409822f45bfd 100644
--- a/Makefile
+++ b/Makefile
@@ -562,10 +562,10 @@ endif
 ifneq ($(shell $(CC) --version 2>&1 | head -n 1 | grep clang),)
 ifneq ($(CROSS_COMPILE),)
 CLANG_FLAGS	+= --target=$(notdir $(CROSS_COMPILE:%-=%))
+endif
 GCC_TOOLCHAIN_DIR := $(dir $(shell which $(CROSS_COMPILE)elfedit))
 CLANG_FLAGS	+= --prefix=$(GCC_TOOLCHAIN_DIR)$(notdir $(CROSS_COMPILE))
 GCC_TOOLCHAIN	:= $(realpath $(GCC_TOOLCHAIN_DIR)/..)
-endif
 ifneq ($(GCC_TOOLCHAIN),)
 CLANG_FLAGS	+= --gcc-toolchain=$(GCC_TOOLCHAIN)
 endif

Hopefully that helps.

Cheers,
Nathan

Re: Using GNU AS from a selfmade binutils v2.35.2

[Nick Desaulniers]

On Fri, Feb 5, 2021 at 1:37 PM Nathan Chancellor <nathan@kernel.org> wrote:
>
> It is entirely possible that '--prefix=' should always be present though:
>
> diff --git a/Makefile b/Makefile
> index f5842126e89d..409822f45bfd 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -562,10 +562,10 @@ endif
>  ifneq ($(shell $(CC) --version 2>&1 | head -n 1 | grep clang),)
>  ifneq ($(CROSS_COMPILE),)
>  CLANG_FLAGS    += --target=$(notdir $(CROSS_COMPILE:%-=%))
> +endif
>  GCC_TOOLCHAIN_DIR := $(dir $(shell which $(CROSS_COMPILE)elfedit))
>  CLANG_FLAGS    += --prefix=$(GCC_TOOLCHAIN_DIR)$(notdir $(CROSS_COMPILE))
>  GCC_TOOLCHAIN  := $(realpath $(GCC_TOOLCHAIN_DIR)/..)
> -endif
>  ifneq ($(GCC_TOOLCHAIN),)
>  CLANG_FLAGS    += --gcc-toolchain=$(GCC_TOOLCHAIN)
>  endif

I'm unsure. Consider the case where I'm building with `LLVM=1
LLVM_IAS=1 CROSS_COMPILE=aarch64-linux-gnu`, ie. no GNU binutils and
am cross compiling.  In that case, we should not be setting any
--prefix or --gcc-toolchain, and yet today we are.  Perhaps that is
orthogonal though?
-- 
Thanks,
~Nick Desaulniers

Re: Using GNU AS from a selfmade binutils v2.35.2

[Sedat Dilek]

On Fri, Feb 5, 2021 at 10:44 PM Nick Desaulniers
<ndesaulniers@google.com> wrote:
>
> On Fri, Feb 5, 2021 at 1:37 PM Nathan Chancellor <nathan@kernel.org> wrote:
> >
> > It is entirely possible that '--prefix=' should always be present though:
> >
> > diff --git a/Makefile b/Makefile
> > index f5842126e89d..409822f45bfd 100644
> > --- a/Makefile
> > +++ b/Makefile
> > @@ -562,10 +562,10 @@ endif
> >  ifneq ($(shell $(CC) --version 2>&1 | head -n 1 | grep clang),)
> >  ifneq ($(CROSS_COMPILE),)
> >  CLANG_FLAGS    += --target=$(notdir $(CROSS_COMPILE:%-=%))
> > +endif
> >  GCC_TOOLCHAIN_DIR := $(dir $(shell which $(CROSS_COMPILE)elfedit))
> >  CLANG_FLAGS    += --prefix=$(GCC_TOOLCHAIN_DIR)$(notdir $(CROSS_COMPILE))
> >  GCC_TOOLCHAIN  := $(realpath $(GCC_TOOLCHAIN_DIR)/..)
> > -endif
> >  ifneq ($(GCC_TOOLCHAIN),)
> >  CLANG_FLAGS    += --gcc-toolchain=$(GCC_TOOLCHAIN)
> >  endif
>
> I'm unsure. Consider the case where I'm building with `LLVM=1
> LLVM_IAS=1 CROSS_COMPILE=aarch64-linux-gnu`, ie. no GNU binutils and
> am cross compiling.  In that case, we should not be setting any
> --prefix or --gcc-toolchain, and yet today we are.  Perhaps that is
> orthogonal though?
>

Nathan and Nick, Guys - you are really great.

Big thank-you for your feedback.
But I am out - the whole today I build Linux-kernels...
"My head is empty." (Kofi Anan)

Today, I remembered it is exactly one year we met in Zurich @ 1st
ClangBuiltLinux meeting :-).

- Sedat -

P.S.: I asked Matthias Klose why binutils v2.35.2 is delayed - it's
ready on <salsa.debian.org> Git.

Re: Using GNU AS from a selfmade binutils v2.35.2

[Nathan Chancellor]

On Fri, Feb 05, 2021 at 01:44:44PM -0800, 'Nick Desaulniers' via Clang Built Linux wrote:
> On Fri, Feb 5, 2021 at 1:37 PM Nathan Chancellor <nathan@kernel.org> wrote:
> >
> > It is entirely possible that '--prefix=' should always be present though:
> >
> > diff --git a/Makefile b/Makefile
> > index f5842126e89d..409822f45bfd 100644
> > --- a/Makefile
> > +++ b/Makefile
> > @@ -562,10 +562,10 @@ endif
> >  ifneq ($(shell $(CC) --version 2>&1 | head -n 1 | grep clang),)
> >  ifneq ($(CROSS_COMPILE),)
> >  CLANG_FLAGS    += --target=$(notdir $(CROSS_COMPILE:%-=%))
> > +endif
> >  GCC_TOOLCHAIN_DIR := $(dir $(shell which $(CROSS_COMPILE)elfedit))
> >  CLANG_FLAGS    += --prefix=$(GCC_TOOLCHAIN_DIR)$(notdir $(CROSS_COMPILE))
> >  GCC_TOOLCHAIN  := $(realpath $(GCC_TOOLCHAIN_DIR)/..)
> > -endif
> >  ifneq ($(GCC_TOOLCHAIN),)
> >  CLANG_FLAGS    += --gcc-toolchain=$(GCC_TOOLCHAIN)
> >  endif
> 
> I'm unsure. Consider the case where I'm building with `LLVM=1
> LLVM_IAS=1 CROSS_COMPILE=aarch64-linux-gnu`, ie. no GNU binutils and
> am cross compiling.  In that case, we should not be setting any
> --prefix or --gcc-toolchain, and yet today we are.  Perhaps that is
> orthogonal though?

Yes, we could probably move the 'ifneq ($(LLVM_IAS),1)' block up because
all other build tools should be called directly (really, just "$(LD)")
but I do think that is orthogonal to the issue that is going on here.

I am happy to send patches if you feel this is worthwhile.

Cheers,
Nathan

Re: Using GNU AS from a selfmade binutils v2.35.2

[Nick Desaulniers]

On Fri, Feb 5, 2021 at 1:55 PM Nathan Chancellor <nathan@kernel.org> wrote:
>
> On Fri, Feb 05, 2021 at 01:44:44PM -0800, 'Nick Desaulniers' via Clang Built Linux wrote:
> > On Fri, Feb 5, 2021 at 1:37 PM Nathan Chancellor <nathan@kernel.org> wrote:
> > >
> > > It is entirely possible that '--prefix=' should always be present though:
> > >
> > > diff --git a/Makefile b/Makefile
> > > index f5842126e89d..409822f45bfd 100644
> > > --- a/Makefile
> > > +++ b/Makefile
> > > @@ -562,10 +562,10 @@ endif
> > >  ifneq ($(shell $(CC) --version 2>&1 | head -n 1 | grep clang),)
> > >  ifneq ($(CROSS_COMPILE),)
> > >  CLANG_FLAGS    += --target=$(notdir $(CROSS_COMPILE:%-=%))
> > > +endif
> > >  GCC_TOOLCHAIN_DIR := $(dir $(shell which $(CROSS_COMPILE)elfedit))
> > >  CLANG_FLAGS    += --prefix=$(GCC_TOOLCHAIN_DIR)$(notdir $(CROSS_COMPILE))
> > >  GCC_TOOLCHAIN  := $(realpath $(GCC_TOOLCHAIN_DIR)/..)
> > > -endif
> > >  ifneq ($(GCC_TOOLCHAIN),)
> > >  CLANG_FLAGS    += --gcc-toolchain=$(GCC_TOOLCHAIN)
> > >  endif
> >
> > I'm unsure. Consider the case where I'm building with `LLVM=1
> > LLVM_IAS=1 CROSS_COMPILE=aarch64-linux-gnu`, ie. no GNU binutils and
> > am cross compiling.  In that case, we should not be setting any
> > --prefix or --gcc-toolchain, and yet today we are.  Perhaps that is
> > orthogonal though?
>
> Yes, we could probably move the 'ifneq ($(LLVM_IAS),1)' block up because
> all other build tools should be called directly (really, just "$(LD)")
> but I do think that is orthogonal to the issue that is going on here.
>
> I am happy to send patches if you feel this is worthwhile.

It would be the capstone to hermetic builds, I suspect.
-- 
Thanks,
~Nick Desaulniers

Re: Using GNU AS from a selfmade binutils v2.35.2

[Sedat Dilek]

Just wanna let you know, I filed a bug to Debian BTS namely "binutils
2.35.2 release".

- Sedat -

[1] https://bugs.debian.org/982086