[Buildroot] [PATCH 1/1] package/pkg-utils.mk: fix per-package build with ncurses

[Buildroot] [PATCH 1/1] package/pkg-utils.mk: fix per-package build with ncurses

[Fabrice Fontaine]

Fix per-package build with ncurses and toolchains providing ncurses.h by
adding --update parameter to rsync call. Indeed, without this parameter,
the curses.h file installed by ncurses (with wchar support) could be
overriden by a curses.h file provided by the toolchain (without wchar
support) resulting in the following build failure on bmon or rtorrent:

rsync -a --link-dest=/tmp/instance-0/output-1/per-package/host-pkgconf/host/ /tmp/instance-0/output-1/per-package/host-pkgconf/host/ /tmp/instance-0/output-1/per-package/rtorrent/host
rsync -a --link-dest=/tmp/instance-0/output-1/per-package/host-skeleton/host/ /tmp/instance-0/output-1/per-package/host-skeleton/host/ /tmp/instance-0/output-1/per-package/rtorrent/host
rsync -a --link-dest=/tmp/instance-0/output-1/per-package/libcurl/host/ /tmp/instance-0/output-1/per-package/libcurl/host/ /tmp/instance-0/output-1/per-package/rtorrent/host
rsync -a --link-dest=/tmp/instance-0/output-1/per-package/libtorrent/host/ /tmp/instance-0/output-1/per-package/libtorrent/host/ /tmp/instance-0/output-1/per-package/rtorrent/host
rsync -a --link-dest=/tmp/instance-0/output-1/per-package/ncurses/host/ /tmp/instance-0/output-1/per-package/ncurses/host/ /tmp/instance-0/output-1/per-package/rtorrent/host
rsync -a --link-dest=/tmp/instance-0/output-1/per-package/skeleton/host/ /tmp/instance-0/output-1/per-package/skeleton/host/ /tmp/instance-0/output-1/per-package/rtorrent/host
rsync -a --link-dest=/tmp/instance-0/output-1/per-package/toolchain/host/ /tmp/instance-0/output-1/per-package/toolchain/host/ /tmp/instance-0/output-1/per-package/rtorrent/host

[...]

checking for NcursesW wide-character library... yes
checking for working ncursesw/curses.h... no
checking for working ncursesw.h... no
checking for working ncurses.h... no
configure: WARNING: could not find a working ncursesw/curses.h, ncursesw.h or ncurses.h

Fixes:
 - http://autobuild.buildroot.org/results/8c0d13e3cc3ddd2a21f99477cad4e39522e9deec
 - http://autobuild.buildroot.org/results/d2f18ccad6a0330d78e97d70c810ac89b84d2261

Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
---
 package/pkg-utils.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
index ae3c7f9da9..5af34d4ee0 100644
--- a/package/pkg-utils.mk
+++ b/package/pkg-utils.mk
@@ -184,7 +184,7 @@ ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
 define per-package-rsync
 	mkdir -p $(3)
 	$(foreach pkg,$(1),\
-		rsync -a --link-dest=$(PER_PACKAGE_DIR)/$(pkg)/$(2)/ \
+		rsync -au --link-dest=$(PER_PACKAGE_DIR)/$(pkg)/$(2)/ \
 		$(PER_PACKAGE_DIR)/$(pkg)/$(2)/ \
 		$(3)$(sep))
 endef
-- 
2.32.0

_______________________________________________
buildroot mailing list
buildroot@busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/1] package/pkg-utils.mk: fix per-package build with ncurses

[Yann E. MORIN]

Fabrice, All,

+Hervé, +Thomas, +Arnout, +Romain

Thanks for this patch, that is a very good example of a very tricky
problem we've been stumbling upon since we started doing top-level
parallel builds...

On 2021-08-29 20:39 +0200, Fabrice Fontaine spake thusly:
> Fix per-package build with ncurses and toolchains providing ncurses.h by
> adding --update parameter to rsync call. Indeed, without this parameter,
> the curses.h file installed by ncurses (with wchar support) could be
> overriden by a curses.h file provided by the toolchain (without wchar
> support) resulting in the following build failure on bmon or rtorrent:
> 
> rsync -a --link-dest=/tmp/instance-0/output-1/per-package/host-pkgconf/host/ /tmp/instance-0/output-1/per-package/host-pkgconf/host/ /tmp/instance-0/output-1/per-package/rtorrent/host
> rsync -a --link-dest=/tmp/instance-0/output-1/per-package/host-skeleton/host/ /tmp/instance-0/output-1/per-package/host-skeleton/host/ /tmp/instance-0/output-1/per-package/rtorrent/host
> rsync -a --link-dest=/tmp/instance-0/output-1/per-package/libcurl/host/ /tmp/instance-0/output-1/per-package/libcurl/host/ /tmp/instance-0/output-1/per-package/rtorrent/host
> rsync -a --link-dest=/tmp/instance-0/output-1/per-package/libtorrent/host/ /tmp/instance-0/output-1/per-package/libtorrent/host/ /tmp/instance-0/output-1/per-package/rtorrent/host
> rsync -a --link-dest=/tmp/instance-0/output-1/per-package/ncurses/host/ /tmp/instance-0/output-1/per-package/ncurses/host/ /tmp/instance-0/output-1/per-package/rtorrent/host
> rsync -a --link-dest=/tmp/instance-0/output-1/per-package/skeleton/host/ /tmp/instance-0/output-1/per-package/skeleton/host/ /tmp/instance-0/output-1/per-package/rtorrent/host
> rsync -a --link-dest=/tmp/instance-0/output-1/per-package/toolchain/host/ /tmp/instance-0/output-1/per-package/toolchain/host/ /tmp/instance-0/output-1/per-package/rtorrent/host

So, if we had had file overwrite detection in place (which I have been
reviewing and testing this WE), the build would have failed earlier, at
the end of the install step of ncurses, with:

    /home/ymorin/dev/buildroot/O/per-package/ncurses/host/riscv64-buildroot-linux-gnu/sysroot/usr/lib/pkgconfig/ncurses.pc: FAILED
    /home/ymorin/dev/buildroot/O/per-package/ncurses/host/riscv64-buildroot-linux-gnu/sysroot/usr/lib/pkgconfig/menu.pc: FAILED
    /home/ymorin/dev/buildroot/O/per-package/ncurses/host/riscv64-buildroot-linux-gnu/sysroot/usr/lib/pkgconfig/form.pc: FAILED
    /home/ymorin/dev/buildroot/O/per-package/ncurses/host/riscv64-buildroot-linux-gnu/sysroot/usr/lib/pkgconfig/panel.pc: FAILED
    /home/ymorin/dev/buildroot/O/per-package/ncurses/host/riscv64-buildroot-linux-gnu/sysroot/usr/include/curses.h: FAILED
    /home/ymorin/dev/buildroot/O/per-package/ncurses/host/riscv64-buildroot-linux-gnu/sysroot/usr/include/term.h: FAILED
    md5sum: WARNING: 6 computed checksums did NOT match
    ERROR: package ncurses has overwritten files installed by a previous package, aborting.
    make[1]: *** [package/pkg-generic.mk:467: /home/ymorin/dev/buildroot/O/build/ncurses-6.1/.stamp_installed] Error 1
    make: *** [Makefile:23: _all] Error 2

> [...]
> 
> checking for NcursesW wide-character library... yes
> checking for working ncursesw/curses.h... no
> checking for working ncursesw.h... no
> checking for working ncurses.h... no
> configure: WARNING: could not find a working ncursesw/curses.h, ncursesw.h or ncurses.h
> 
> Fixes:
>  - http://autobuild.buildroot.org/results/8c0d13e3cc3ddd2a21f99477cad4e39522e9deec
>  - http://autobuild.buildroot.org/results/d2f18ccad6a0330d78e97d70c810ac89b84d2261
> 
> Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
> ---
>  package/pkg-utils.mk | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
> index ae3c7f9da9..5af34d4ee0 100644
> --- a/package/pkg-utils.mk
> +++ b/package/pkg-utils.mk
> @@ -184,7 +184,7 @@ ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
>  define per-package-rsync
>  	mkdir -p $(3)
>  	$(foreach pkg,$(1),\
> -		rsync -a --link-dest=$(PER_PACKAGE_DIR)/$(pkg)/$(2)/ \
> +		rsync -au --link-dest=$(PER_PACKAGE_DIR)/$(pkg)/$(2)/ \

Sorry, this is not the correct solution. Indeed, we can not accept that
a package overwrites a file provided by another package. See the
explanations Thomas and I discussed earlier today:

    https://lore.kernel.org/buildroot/20210829133955.74d921e9@windsurf/
    https://lore.kernel.org/buildroot/20210829164031.GB1053080@scaer/

This case is however very special, and I have no idea (yet) on what the
best solution is. I guess the curses headers in the toolchain are due to
the presence of gdb with TUI support...

I would be tempted to say "this toolchain is not pure, we can't use it",
and defer to the Bootlin toolchain builder project to fix their
toolchains, but this is probably a bit of shuffling the dust under the
rag...

So, any input and insight would be appreciated...

Regards,
Yann E. MORIN.

>  		$(PER_PACKAGE_DIR)/$(pkg)/$(2)/ \
>  		$(3)$(sep))
>  endef
> -- 
> 2.32.0
> 
> _______________________________________________
> buildroot mailing list
> buildroot@busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/1] package/pkg-utils.mk: fix per-package build with ncurses

[Arnout Vandecappelle]

On 29/08/2021 21:37, Yann E. MORIN wrote:
> Fabrice, All,
> 
> +Hervé, +Thomas, +Arnout, +Romain
> 
> Thanks for this patch, that is a very good example of a very tricky
> problem we've been stumbling upon since we started doing top-level
> parallel builds...
> 
> On 2021-08-29 20:39 +0200, Fabrice Fontaine spake thusly:
>> Fix per-package build with ncurses and toolchains providing ncurses.h by
>> adding --update parameter to rsync call. Indeed, without this parameter,
>> the curses.h file installed by ncurses (with wchar support) could be
>> overriden by a curses.h file provided by the toolchain (without wchar
>> support) resulting in the following build failure on bmon or rtorrent:
>>
>> rsync -a --link-dest=/tmp/instance-0/output-1/per-package/host-pkgconf/host/ /tmp/instance-0/output-1/per-package/host-pkgconf/host/ /tmp/instance-0/output-1/per-package/rtorrent/host
>> rsync -a --link-dest=/tmp/instance-0/output-1/per-package/host-skeleton/host/ /tmp/instance-0/output-1/per-package/host-skeleton/host/ /tmp/instance-0/output-1/per-package/rtorrent/host
>> rsync -a --link-dest=/tmp/instance-0/output-1/per-package/libcurl/host/ /tmp/instance-0/output-1/per-package/libcurl/host/ /tmp/instance-0/output-1/per-package/rtorrent/host
>> rsync -a --link-dest=/tmp/instance-0/output-1/per-package/libtorrent/host/ /tmp/instance-0/output-1/per-package/libtorrent/host/ /tmp/instance-0/output-1/per-package/rtorrent/host
>> rsync -a --link-dest=/tmp/instance-0/output-1/per-package/ncurses/host/ /tmp/instance-0/output-1/per-package/ncurses/host/ /tmp/instance-0/output-1/per-package/rtorrent/host
>> rsync -a --link-dest=/tmp/instance-0/output-1/per-package/skeleton/host/ /tmp/instance-0/output-1/per-package/skeleton/host/ /tmp/instance-0/output-1/per-package/rtorrent/host
>> rsync -a --link-dest=/tmp/instance-0/output-1/per-package/toolchain/host/ /tmp/instance-0/output-1/per-package/toolchain/host/ /tmp/instance-0/output-1/per-package/rtorrent/host
> 
> So, if we had had file overwrite detection in place (which I have been
> reviewing and testing this WE), the build would have failed earlier, at
> the end of the install step of ncurses, with:
> 
>     /home/ymorin/dev/buildroot/O/per-package/ncurses/host/riscv64-buildroot-linux-gnu/sysroot/usr/lib/pkgconfig/ncurses.pc: FAILED
>     /home/ymorin/dev/buildroot/O/per-package/ncurses/host/riscv64-buildroot-linux-gnu/sysroot/usr/lib/pkgconfig/menu.pc: FAILED
>     /home/ymorin/dev/buildroot/O/per-package/ncurses/host/riscv64-buildroot-linux-gnu/sysroot/usr/lib/pkgconfig/form.pc: FAILED
>     /home/ymorin/dev/buildroot/O/per-package/ncurses/host/riscv64-buildroot-linux-gnu/sysroot/usr/lib/pkgconfig/panel.pc: FAILED
>     /home/ymorin/dev/buildroot/O/per-package/ncurses/host/riscv64-buildroot-linux-gnu/sysroot/usr/include/curses.h: FAILED
>     /home/ymorin/dev/buildroot/O/per-package/ncurses/host/riscv64-buildroot-linux-gnu/sysroot/usr/include/term.h: FAILED
>     md5sum: WARNING: 6 computed checksums did NOT match
>     ERROR: package ncurses has overwritten files installed by a previous package, aborting.
>     make[1]: *** [package/pkg-generic.mk:467: /home/ymorin/dev/buildroot/O/build/ncurses-6.1/.stamp_installed] Error 1
>     make: *** [Makefile:23: _all] Error 2
> 
>> [...]
>>
>> checking for NcursesW wide-character library... yes
>> checking for working ncursesw/curses.h... no
>> checking for working ncursesw.h... no
>> checking for working ncurses.h... no
>> configure: WARNING: could not find a working ncursesw/curses.h, ncursesw.h or ncurses.h
>>
>> Fixes:
>>  - http://autobuild.buildroot.org/results/8c0d13e3cc3ddd2a21f99477cad4e39522e9deec
>>  - http://autobuild.buildroot.org/results/d2f18ccad6a0330d78e97d70c810ac89b84d2261
>>
>> Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
>> ---
>>  package/pkg-utils.mk | 2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
>> index ae3c7f9da9..5af34d4ee0 100644
>> --- a/package/pkg-utils.mk
>> +++ b/package/pkg-utils.mk
>> @@ -184,7 +184,7 @@ ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
>>  define per-package-rsync
>>  	mkdir -p $(3)
>>  	$(foreach pkg,$(1),\
>> -		rsync -a --link-dest=$(PER_PACKAGE_DIR)/$(pkg)/$(2)/ \
>> +		rsync -au --link-dest=$(PER_PACKAGE_DIR)/$(pkg)/$(2)/ \
> 
> Sorry, this is not the correct solution. Indeed, we can not accept that
> a package overwrites a file provided by another package. See the
> explanations Thomas and I discussed earlier today:
> 
>     https://lore.kernel.org/buildroot/20210829133955.74d921e9@windsurf/
>     https://lore.kernel.org/buildroot/20210829164031.GB1053080@scaer/
> 
> This case is however very special, and I have no idea (yet) on what the
> best solution is. I guess the curses headers in the toolchain are due to
> the presence of gdb with TUI support...
> 
> I would be tempted to say "this toolchain is not pure, we can't use it",
> and defer to the Bootlin toolchain builder project to fix their
> toolchains, but this is probably a bit of shuffling the dust under the
> rag...
> 
> So, any input and insight would be appreciated...

 I think we need to start a wiki page for this discussion, on the mailing list
the conclusions will get lost too easily... I don't have time for that now
though, maybe tonight.

 Regards,
 Arnout

> 
> Regards,
> Yann E. MORIN.
> 
>>  		$(PER_PACKAGE_DIR)/$(pkg)/$(2)/ \
>>  		$(3)$(sep))
>>  endef
>> -- 
>> 2.32.0
>>
>> _______________________________________________
>> buildroot mailing list
>> buildroot@busybox.net
>> http://lists.busybox.net/mailman/listinfo/buildroot
> 
_______________________________________________
buildroot mailing list
buildroot@busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/1] package/pkg-utils.mk: fix per-package build with ncurses

[Thomas Petazzoni]

On Sun, 29 Aug 2021 21:37:48 +0200
"Yann E. MORIN" <yann.morin.1998@free.fr> wrote:

> Sorry, this is not the correct solution. Indeed, we can not accept that
> a package overwrites a file provided by another package. See the
> explanations Thomas and I discussed earlier today:
> 
>     https://lore.kernel.org/buildroot/20210829133955.74d921e9@windsurf/
>     https://lore.kernel.org/buildroot/20210829164031.GB1053080@scaer/
> 
> This case is however very special, and I have no idea (yet) on what the
> best solution is. I guess the curses headers in the toolchain are due to
> the presence of gdb with TUI support...

gdb with TUI support would mean to have ncurses (library and headers)
installed for the host, but here it's in the sysroot of the toolchain,
while for the target we only build gdbserver.

The two build failures that Fabrice is pointing to are RISC-V 64-bit,
and I believe until recently gdbserver was not available on RISC-V
64-bit. So it means that when we enable BR2_PACKAGE_GDB=y when building
our RISC-V 64-bit toolchains, instead of getting just the default of
gdbserver, we get the full gdb, which pulls in ncurses. This is clearly
a bug in the toolchain: we don't want to provide a full gdb for the
target.

Confirmed with:

$ tar tvf aarch64--glibc--bleeding-edge-2020.08-1.tar.bz2 | grep ncurses.h
lrwxrwxrwx root/root         0 2020-09-30 21:49 aarch64--glibc--bleeding-edge-2020.08-1/include/ncurses/ncurses.h -> curses.h

$ tar tvf riscv64--glibc--bleeding-edge-2020.08-1.tar.bz2 | grep ncurses.h
lrwxrwxrwx root/root         0 2020-10-01 10:22 riscv64--glibc--bleeding-edge-2020.08-1/riscv64-buildroot-linux-gnu/sysroot/usr/include/ncurses.h -> curses.h
lrwxrwxrwx root/root         0 2020-10-01 10:21 riscv64--glibc--bleeding-edge-2020.08-1/include/ncurses/ncurses.h -> curses.h

But in fact, I see we don't even provide gdbserver anywhere in fact,
since we're building the toolchain tarball with "make sdk", that
doesn't preserve the target directory. So we're building the target gdb
package for no reason...

Meh. Stuff to fix.

Thomas
-- 
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot