Re: [Buildroot] [PATCH 1/2] package/toolchain-external: ensure ARCH_LIB_DIR exist

[Mao Han <han_mao@c-sky.com>]

I'v replied the email yesterday, but I can't find it in the mailist.
So just resend one.

Hi Thomas,
On Thu, Jan 16, 2020 at 10:36:45AM +0100, Thomas De Schampheleire wrote:
> Hello Mao Han,
> 
> El lun., 13 ene. 2020 a las 4:39, Mao Han (<han_mao@c-sky.com>) escribió:
> >
> > Hi,
> > On Fri, Jan 10, 2020 at 11:57:00AM +0100, Thomas De Schampheleire wrote:
> > > Hello,
> > >
> > > El jue., 2 ene. 2020 a las 5:15, Mao Han (<han_mao@c-sky.com>) escribió:
> > > >
> > > > Different architecture may have different arch library folder structure
> > > > for their multilib toolchain. RISC-V toolchain has folder structure
> > > > like:
> > > > /sysroot/lib64/
> > > > /sysroot/lib64/lp64
> > > > /sysroot/lib64/lp64d
> > > > The first level is for ISA and the second level is for ABI.
> > > > Current buildroot only creat lib and lib64, some error is reported
> > > > while installing the external toolchain:
> > > > >>> toolchain-external-custom  Installing to staging directory
> > > > /usr/bin/install -D -m 0755 /.../build/toolchain-external-custom/toolchain-wrapper /.../host/bin/toolchain-wrapper
> > > > ln: failed to create symbolic link '/.../host/riscv64-buildroot-linux-gnu/sysroot/lib64/lp64': No such file or directory
> > > > ln: failed to create symbolic link '/.../host/riscv64-buildroot-linux-gnu/sysroot/usr/lib64/lp64': No such file or directory
> > > >
> > > > Signed-off-by: Qu Xianmiao <xianmiao_qu@c-sky.com>
> > > > Signed-off-by: Chen Hongdeng <hongdeng_chen@c-sky.com>
> > > > Signed-off-by: Guo Ren<ren_guo@c-sky.com>
> > > > Signed-off-by: Mao Han <han_mao@c-sky.com>
> > > > Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> > > > Cc: Mark Corbin <mark.corbin@embecosm.com>
> > > > ---
> > > >  toolchain/toolchain-external/pkg-toolchain-external.mk | 2 ++
> > > >  1 file changed, 2 insertions(+)
> > > >
> > > > diff --git a/toolchain/toolchain-external/pkg-toolchain-external.mk b/toolchain/toolchain-external/pkg-toolchain-external.mk
> > > > index baf719a..13b2468 100644
> > > > --- a/toolchain/toolchain-external/pkg-toolchain-external.mk
> > > > +++ b/toolchain/toolchain-external/pkg-toolchain-external.mk
> > > > @@ -454,6 +454,8 @@ create_lib_symlinks = \
> > > >         ARCH_LIB_DIR="$(call toolchain_find_libdir,$(TOOLCHAIN_EXTERNAL_CC) $(TOOLCHAIN_EXTERNAL_CFLAGS))" ; \
> > > >         if [ ! -e "$${DESTDIR}/$${ARCH_LIB_DIR}" -a ! -e "$${DESTDIR}/usr/$${ARCH_LIB_DIR}" ]; then \
> > > >                 relpath="$(call relpath_prefix,$${ARCH_LIB_DIR})" ; \
> > > > +               mkdir -p "$${DESTDIR}/$${ARCH_LIB_DIR}" ; \
> > > > +               mkdir -p "$${DESTDIR}/usr/$${ARCH_LIB_DIR}" ; \
> > > >                 ln -snf $${relpath}lib "$${DESTDIR}/$${ARCH_LIB_DIR}" ; \
> > > >                 ln -snf $${relpath}lib "$${DESTDIR}/usr/$${ARCH_LIB_DIR}" ; \
> > >
> > > This is not correct.
> > >
> > > The purpose of this code is to create a symbolic link named according
> > > to 'ARCH_LIB_DIR' inside DESTDIR and DESTDIR/usr.
> > > It is not the purpose to create a symbolic link called 'lib' inside a
> > > directory DESTDIR/ARCH_LIB_DIR.
> > > The change you made is creating a _directory_ ARCH_LIB_DIR while it
> > > should be a _symbolic link_.
> > >
> >
> > Yes, I misunderstood the option -f as it will only remove the symbol
> > file not the directory.
> > The path sysroot/lib64 is relinked to sysroot/lib after the external
> > tool installation, all the libraries are copied to sysroot/lib/lp64d.
> > The toolchain can find it's corresponding library, so I don't notice
> > the mistake.
> >
> > If I change the code like:
> > mkdir -p `dirname "$${DESTDIR}/$${ARCH_LIB_DIR}"` ; \
> > mkdir -p `dirname "$${DESTDIR}/usr/$${ARCH_LIB_DIR}"` ; \
> >
> > Two symbol links are created:
> > sysroot/:lib64 -> lib
> > sysroot/lib: lp64d -> ../lib
> > The libraries are under: lib, lib64, lib64/lp64d ...
> >
> > > Could you please clarify what is the value of ARCH_LIB_DIR in your case?
> > >
> > > And could you clarify which part of the path is missing?
> > > '/.../host/riscv64-buildroot-linux-gnu/sysroot/lib64/lp64' ?
> > > I assume that the 'lib64' part is missing?
> > >
> >
> > Yes, "lib64" is the missing part. It can be some other value like
> > "lib64xthead" when the extension is different.
> >
> > > Finally, please provide a defconfig that allows to reproduce and test
> > > this problem.
> > >
> >
> > The defconfig is attached. And the toolchain I used comes from:
> > https://github.com/riscv/riscv-gnu-toolchain
> >
> > make with:
> > git clone https://github.com/riscv/riscv-gnu-toolchain
> > cd riscv-gnu-toolchain
> > git submodule update --init --recursive
> > ./configure --prefix=/opt/riscv --enable-multilib
> > make linux
> >
> > I can't provide the toolchain binary as it is quite large, and
> > the network is also limited.
> >
> 
> I compiled the riscv toolchain as instructed, and applied your config
> (note: it was a full config, not a defconfig). The defconfig is:
> 
> BR2_riscv=y
> BR2_TOOLCHAIN_EXTERNAL=y
> BR2_TOOLCHAIN_EXTERNAL_PATH="/opt/riscv"
> BR2_TOOLCHAIN_EXTERNAL_CUSTOM_PREFIX="$(ARCH)-unknown-linux-gnu"
> BR2_TOOLCHAIN_EXTERNAL_HEADERS_5_0=y
> BR2_TOOLCHAIN_EXTERNAL_CUSTOM_GLIBC=y
> BR2_TOOLCHAIN_EXTERNAL_CXX=y
> BR2_TARGET_GENERIC_GETTY_PORT="ttyS0"
> BR2_SYSTEM_DHCP="eth0"
> BR2_LINUX_KERNEL=y
> BR2_LINUX_KERNEL_CUSTOM_VERSION=y
> BR2_LINUX_KERNEL_CUSTOM_VERSION_VALUE="5.1.12"
> BR2_LINUX_KERNEL_USE_ARCH_DEFAULT_CONFIG=y
> BR2_TARGET_ROOTFS_EXT2=y
> BR2_TARGET_OPENSBI=y
> BR2_TARGET_OPENSBI_PLAT="qemu/virt"
> BR2_PACKAGE_HOST_QEMU=y
> BR2_PACKAGE_HOST_QEMU_SYSTEM_MODE=y
>
> Using this config, I do not see the same error as you. The toolchain
> seems to install correctly (note: later we will notice it does not;
> see below), but I do get an error later when busybox is compiled:
>

The problem is originally occurs on a non-public released toolchain
with a sysroot folder structure like:
sysroot/lib64xthead/lp64d
and I tried to reproduce with a public RISC-V toolchain, and got
similar error weeks ago.
I retried with the toolchains yesterday, the error is only reproduced
with the internal one. There might be something wrong with my 
previous operation.

> ...
>   CC      libbb/xregcomp.o
>   AR      libbb/lib.a
>   LINK    busybox_unstripped
> Your linker does not support --sort-section,alignment
> Your linker does not support --sort-common
> Your linker does not support -Wl,--gc-sections
> Trying libraries: m resolv
> Failed: -Wl,--start-group  -lm -lresolv  -Wl,--end-group
> Output of:
> /home/tdescham/repo/contrib/buildroot/output/host/bin/riscv64-unknown-linux-gnu-gcc
> -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -Os
> -Wall -Wshadow -Wwrite-strings -Wundef -Wstrict-prototypes -Wunused
> -Wunused-parameter -Wunused-function -Wunused-value
> -Wmissing-prototypes -Wmissing-declarations -Wno-format-security
> -Wdeclaration-after-statement -Wold-style-definition
> -fno-builtin-strlen -finline-limit=0 -fomit-frame-pointer
> -ffunction-sections -fdata-sections -fno-guess-branch-probability
> -funsigned-char -static-libgcc -falign-functions=1 -falign-jumps=1
> -falign-labels=1 -falign-loops=1 -fno-unwind-tables
> -fno-asynchronous-unwind-tables -fno-builtin-printf -Os -o
> busybox_unstripped -Wl,--start-group applets/built-in.o archival/lib.a
> archival/libarchive/lib.a console-tools/lib.a coreutils/lib.a
> coreutils/libcoreutils/lib.a debianutils/lib.a klibc-utils/lib.a
> e2fsprogs/lib.a editors/lib.a findutils/lib.a init/lib.a libbb/lib.a
> libpwdgrp/lib.a loginutils/lib.a mailutils/lib.a miscutils/lib.a
> modutils/lib.a networking/lib.a networking/libiproute/lib.a
> networking/udhcp/lib.a printutils/lib.a procps/lib.a runit/lib.a
> selinux/lib.a shell/lib.a sysklogd/lib.a util-linux/lib.a
> util-linux/volume_id/lib.a archival/built-in.o
> archival/libarchive/built-in.o console-tools/built-in.o
> coreutils/built-in.o coreutils/libcoreutils/built-in.o
> debianutils/built-in.o klibc-utils/built-in.o e2fsprogs/built-in.o
> editors/built-in.o findutils/built-in.o init/built-in.o
> libbb/built-in.o libpwdgrp/built-in.o loginutils/built-in.o
> mailutils/built-in.o miscutils/built-in.o modutils/built-in.o
> networking/built-in.o networking/libiproute/built-in.o
> networking/udhcp/built-in.o printutils/built-in.o procps/built-in.o
> runit/built-in.o selinux/built-in.o shell/built-in.o
> sysklogd/built-in.o util-linux/built-in.o
> util-linux/volume_id/built-in.o -Wl,--end-group -Wl,--start-group -lm
> -lresolv -Wl,--end-group
> ==========
> /home/tdescham/repo/contrib/riscv/lib/gcc/riscv64-unknown-linux-gnu/9.2.0/../../../../riscv64-unknown-linux-gnu/bin/ld:
> coreutils/lib.a(mktemp.o): in function `.L5':
> mktemp.c:(.text.mktemp_main+0xb4): warning: the use of `mktemp' is
> dangerous, better use `mkstemp' or `mkdtemp'
> /home/tdescham/repo/contrib/riscv/lib/gcc/riscv64-unknown-linux-gnu/9.2.0/../../../../riscv64-unknown-linux-gnu/bin/ld:
> cannot find /lib/ld-linux-riscv64-lp64d.so.1 inside
> /home/tdescham/repo/contrib/buildroot/output/host/riscv64-buildroot-linux-gnu/sysroot
> collect2: error: ld returned 1 exit status
> Note: if build needs additional libraries, put them in CONFIG_EXTRA_LDLIBS.
> Example: CONFIG_EXTRA_LDLIBS="pthread dl tirpc audit pam"
> make[1]: *** [Makefile:718: busybox_unstripped] Error 1
> make[1]: Leaving directory
> '/home/tdescham/repo/contrib/buildroot/output/build/busybox-1.31.1'
> make: *** [package/pkg-generic.mk:260:
> /home/tdescham/repo/contrib/buildroot/output/build/busybox-1.31.1/.stamp_built]
> Error 2
> 
> 
> So the actual error here is:
> /home/tdescham/repo/contrib/riscv/lib/gcc/riscv64-unknown-linux-gnu/9.2.0/../../../../riscv64-unknown-linux-gnu/bin/ld:
> cannot find /lib/ld-linux-riscv64-lp64d.so.1 inside
> /home/tdescham/repo/contrib/buildroot/output/host/riscv64-buildroot-linux-gnu/sysroot
> collect2: error: ld returned 1 exit status
> 
> At this point, the situation in the buildroot sysroot is:
> 
> $ ls -l output/host/riscv64-buildroot-linux-gnu/sysroot/
> total 56
> drwxr-xr-x 2 tdescham tdescham 4096 Jan 16 10:05 bin/
> drwxr-xr-x 2 tdescham tdescham 4096 May  7  2019 dev/
> drwxr-xr-x 3 tdescham tdescham 4096 Jan 15 21:58 etc/
> drwxr-xr-x 2 tdescham tdescham 4096 Jan 16 08:48 lib/
> lrwxrwxrwx 1 tdescham tdescham    3 Jan 16 10:05 lib64 -> lib/
> drwxr-xr-x 2 tdescham tdescham 4096 May 28  2014 media/
> drwxr-xr-x 2 tdescham tdescham 4096 May 28  2014 mnt/
> drwxr-xr-x 2 tdescham tdescham 4096 May 28  2014 opt/
> drwxr-xr-x 2 tdescham tdescham 4096 May 28  2014 proc/
> drwxr-xr-x 2 tdescham tdescham 4096 Jan 25  2017 root/
> drwxr-xr-x 2 tdescham tdescham 4096 Feb  2  2016 run/
> drwxr-xr-x 2 tdescham tdescham 4096 Jan 15 21:58 sbin/
> drwxr-xr-x 2 tdescham tdescham 4096 May 28  2014 sys/
> drwxr-xr-x 2 tdescham tdescham 4096 May 28  2014 tmp/
> drwxr-xr-x 8 tdescham tdescham 4096 Jan 15 21:57 usr/
> 
> $ ls -l output/host/riscv64-buildroot-linux-gnu/sysroot/lib/lp64d
> lrwxrwxrwx 1 tdescham tdescham 6 Jan 16 10:05
> output/host/riscv64-buildroot-linux-gnu/sysroot/lib/lp64d -> ../lib/
> 
> and there is no ld-linux* file found anywhere, indeed.
> 
> Looking at the output of the toolchain installation, there was an error:
> >>> toolchain-external-custom  Copying external toolchain sysroot to staging...
> /bin/sh: line 0: [: too many arguments
> 
> which is caused by the fact that there are multiple 'ld*' matches:
> (extra traces added)
> 
> SYSROOT_DIR=/home/tdescham/repo/contrib/riscv/sysroot/
> ARCH_SYSROOT_DIR=/home/tdescham/repo/contrib/riscv/sysroot/
> ARCH_SUBDIR=
> ARCH_LIB_DIR=lib64/lp64d
> SUPPORT_LIB_DIR=
> no ld found yet @@@@@@@@@@@@
> /home/tdescham/repo/contrib/riscv/sysroot//lib/ld-linux-riscv32-ilp32d.so.1
> /home/tdescham/repo/contrib/riscv/sysroot//lib/ld-linux-riscv32-ilp32.so.1
> /home/tdescham/repo/contrib/riscv/sysroot//lib/ld-linux-riscv64-lp64d.so.1
> /home/tdescham/repo/contrib/riscv/sysroot//lib/ld-linux-riscv64-lp64.so.1
> /bin/sh: line 0: [: too many arguments
> done no ld found @@@@@@@@@@@@
> 
> Corresponding to this code:
> 
>     if [ ! -e $(STAGING_DIR)/lib/ld*.so.* ]; then \
>         echo "no ld found yet @@@@@@@@@@@@"; \
>         ls $${ARCH_SYSROOT_DIR}/lib/ld*.so.* ; \
>         if [ -e $${ARCH_SYSROOT_DIR}/lib/ld*.so.* ]; then \
>             echo "ld found in arch_sysroot_dir @@@@@@@@@@@@@"; \
>             cp -a $${ARCH_SYSROOT_DIR}/lib/ld*.so.* $(STAGING_DIR)/lib/ ; \
>         fi ; \
>         echo "done no ld found @@@@@@@@@@@@" ; \
>     fi ; \
> 
> 
> I'm not yet sure how this should be solved...

There is another patch in the same thread:
[PATCH 2/2] toolchain: Get ld.so name if available
http://buildroot-busybox.2317881.n4.nabble.com/PATCH-1-2-package-toolchain-external-ensure-ARCH-LIB-DIR-exist-td243604.html#a244193
Tried to get the corresponding ld.so name with certain configuration.

Thanks,
Mao Han