[Buildroot] 'make foo-menuconfig' broken when host-ncurses was built

[Buildroot] 'make foo-menuconfig' broken when host-ncurses was built

[Arnout Vandecappelle]

 Hi all,

 My colleague David observed that since a month or so, the dialogs of 'make
linux-menuconfig' looked pretty garbled. After a bit of debugging, it turns out
that the problem is mixing host-ncurses with recent (6.0+) system ncurses. Based
on the output, I suspect it's mixing an 8-bit ncurses library with widechar
termcap files.

 This issue is caused by dde090c299 linux: fix passing of host CFLAGS and LDFLAGS

 What happens is that kconfig uses pkg-config to discover the ncurses
installation. Since our host-ncurses doesn't install a pc file (and in linux.mk
we anyway don't pass the approprate PKG_CONFIG_* variables), this will pick up
the system's pkg-config settings. However, we set HOSTCC="/usr/bin/gcc -O2
-I.../host/include -L.../host/lib -Wl,-rpath,.../host/lib" on the linux make
command line, so it *will* actually pick up our host-ncurses library. On my
system, 'pkg-config --libs ncurses' returns '-lncurses -ltinfo'. Our ncurses
only has libncurses.so, so libtinfo.so will be picked up from the host.

 The question is how to resolve this. The correct thing to do would be to
install the .pc file, and to do what Thomas proposed a while ago:
$(HOST_DIR)/bin/pkg-config returns the host config, $(CROSS_COMPILE)pkg-config
returns the target config. But this will probably result in breakage of a lot of
packages...

 Any better ideas?

 Regards,
 Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH] package/ncurses: enable widechar for host build

[Arnout Vandecappelle]

Kconfig uses pkg-config to find the ncurses or ncursesw library. If the
ncursesw package is found with pkg-config, it will #include <ncursesw.h>.
Since Buildroot's host-ncurses doesn't install a .pc file, and linux.mk
anyway doesn't pass the pkg-config options to find the host pkg-config
files, Kconfig will always find the system's ncursesw.h.

However, since commit dde090c299 (linux: fix passing of host CFLAGS and
LDFLAGS) HOST_LDFLAGS is passed to the linux build system. Thus, if
host-ncurses was already built before 'make linux-menuconfig' is called,
the build will pick up libncurses from the host directory, which is NOT
widechar. Thus, two different ncurses configurations are mixed into the
final mconf program. This will result in serious breakage in the
rendering of the menus (lots of @ and question mark characters).

As a workaround, just build host-ncurses with widechar support. That
makes it compatible with ncursesw.h picked up from the host. Clearly,
this is not a robust solution. But it's simple, it works, and it
shouldn't hurt to build host-ncurses with widechar support (we can
safely assume that the system doesn't have a non-widechare uClibc
library...).

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Cc: David De Grave <david.degrave@essensium.com>
---
 package/ncurses/ncurses.mk | 1 +
 1 file changed, 1 insertion(+)

diff --git a/package/ncurses/ncurses.mk b/package/ncurses/ncurses.mk
index 90cf4a4dc5..9942d3f0bd 100644
--- a/package/ncurses/ncurses.mk
+++ b/package/ncurses/ncurses.mk
@@ -153,6 +153,7 @@ HOST_NCURSES_CONF_OPTS = \
 	--without-cxx \
 	--without-cxx-binding \
 	--without-ada \
+	--enable-widec \
 	--without-normal
 
 $(eval $(autotools-package))
-- 
2.17.0

[Buildroot] [PATCH] package/ncurses: enable widechar for host build

[Scott Fan]

Hi, Arnout:

I suggest you to adjust your patch from

HOST_NCURSES_CONF_OPTS = \
--with-shared \
--without-gpm \
--without-manpages \
--without-cxx \
--without-cxx-binding \
--without-ada \
--enable-widec \
--without-normal

to

HOST_NCURSES_CONF_OPTS = \
--enable-widec \
--with-shared \
--without-gpm \
--without-manpages \
--without-cxx \
--without-cxx-binding \
--without-ada \
--without-normal

just for more cleanly.


Thanks.

Scott Fan


On Fri, Apr 27, 2018 at 3:08 AM Arnout Vandecappelle (Essensium/Mind) <
arnout@mind.be> wrote:

> Kconfig uses pkg-config to find the ncurses or ncursesw library. If the
> ncursesw package is found with pkg-config, it will #include <ncursesw.h>.
> Since Buildroot's host-ncurses doesn't install a .pc file, and linux.mk
> anyway doesn't pass the pkg-config options to find the host pkg-config
> files, Kconfig will always find the system's ncursesw.h.
>
> However, since commit dde090c299 (linux: fix passing of host CFLAGS and
> LDFLAGS) HOST_LDFLAGS is passed to the linux build system. Thus, if
> host-ncurses was already built before 'make linux-menuconfig' is called,
> the build will pick up libncurses from the host directory, which is NOT
> widechar. Thus, two different ncurses configurations are mixed into the
> final mconf program. This will result in serious breakage in the
> rendering of the menus (lots of @ and question mark characters).
>
> As a workaround, just build host-ncurses with widechar support. That
> makes it compatible with ncursesw.h picked up from the host. Clearly,
> this is not a robust solution. But it's simple, it works, and it
> shouldn't hurt to build host-ncurses with widechar support (we can
> safely assume that the system doesn't have a non-widechare uClibc
> library...).
>
> Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
> Cc: David De Grave <david.degrave@essensium.com>
> ---
>  package/ncurses/ncurses.mk | 1 +
>  1 file changed, 1 insertion(+)
>
> diff --git a/package/ncurses/ncurses.mk b/package/ncurses/ncurses.mk
> index 90cf4a4dc5..9942d3f0bd 100644
> --- a/package/ncurses/ncurses.mk
> +++ b/package/ncurses/ncurses.mk
> @@ -153,6 +153,7 @@ HOST_NCURSES_CONF_OPTS = \
>         --without-cxx \
>         --without-cxx-binding \
>         --without-ada \
> +       --enable-widec \
>         --without-normal
>
>  $(eval $(autotools-package))
> --
> 2.17.0
>
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20180427/bb98b8ee/attachment.html>

[Buildroot] 'make foo-menuconfig' broken when host-ncurses was built

[Peter Korsgaard]

>>>>> "Arnout" == Arnout Vandecappelle <arnout@mind.be> writes:

 >  Hi all,
 >  My colleague David observed that since a month or so, the dialogs of 'make
 > linux-menuconfig' looked pretty garbled. After a bit of debugging, it turns out
 > that the problem is mixing host-ncurses with recent (6.0+) system ncurses. Based
 > on the output, I suspect it's mixing an 8-bit ncurses library with widechar
 > termcap files.

 >  This issue is caused by dde090c299 linux: fix passing of host CFLAGS and LDFLAGS

 >  What happens is that kconfig uses pkg-config to discover the ncurses
 > installation. Since our host-ncurses doesn't install a pc file (and in linux.mk
 > we anyway don't pass the approprate PKG_CONFIG_* variables), this will pick up
 > the system's pkg-config settings. However, we set HOSTCC="/usr/bin/gcc -O2
 > -I.../host/include -L.../host/lib -Wl,-rpath,.../host/lib" on the linux make
 > command line, so it *will* actually pick up our host-ncurses library. On my
 > system, 'pkg-config --libs ncurses' returns '-lncurses -ltinfo'. Our ncurses
 > only has libncurses.so, so libtinfo.so will be picked up from the host.

Ahh :/

 >  The question is how to resolve this. The correct thing to do would be to
 > install the .pc file, and to do what Thomas proposed a while ago:
 > $(HOST_DIR)/bin/pkg-config returns the host config, $(CROSS_COMPILE)pkg-config
 > returns the target config. But this will probably result in breakage of a lot of
 > packages...

 >  Any better ideas?

Is that really needed? Can't we just change linux.mk to use
HOST_MAKE_ENV instead of TARGET_MAKE_ENV (it is a target package, but it
only needs to link against host libraries). That wil ensure that the
PKG_CONFIG_* environment variables are set, so if we were to install a
ncurses(w) pc file for the host, kconfig should find and use it?

-- 
Bye, Peter Korsgaard

[Buildroot] 'make foo-menuconfig' broken when host-ncurses was built

[Arnout Vandecappelle]

On 01-05-18 22:12, Peter Korsgaard wrote:
>>>>>> "Arnout" == Arnout Vandecappelle <arnout@mind.be> writes:
> 
>  >  Hi all,
>  >  My colleague David observed that since a month or so, the dialogs of 'make
>  > linux-menuconfig' looked pretty garbled. After a bit of debugging, it turns out
>  > that the problem is mixing host-ncurses with recent (6.0+) system ncurses. Based
>  > on the output, I suspect it's mixing an 8-bit ncurses library with widechar
>  > termcap files.
> 
>  >  This issue is caused by dde090c299 linux: fix passing of host CFLAGS and LDFLAGS
> 
>  >  What happens is that kconfig uses pkg-config to discover the ncurses
>  > installation. Since our host-ncurses doesn't install a pc file (and in linux.mk
>  > we anyway don't pass the approprate PKG_CONFIG_* variables), this will pick up
>  > the system's pkg-config settings. However, we set HOSTCC="/usr/bin/gcc -O2
>  > -I.../host/include -L.../host/lib -Wl,-rpath,.../host/lib" on the linux make
>  > command line, so it *will* actually pick up our host-ncurses library. On my
>  > system, 'pkg-config --libs ncurses' returns '-lncurses -ltinfo'. Our ncurses
>  > only has libncurses.so, so libtinfo.so will be picked up from the host.
> 
> Ahh :/

 See the patch I sent: my analysis was not entirely correct. The issue is not in
libtinfo.so. The issue is that we include the system's ncursesw.h (which is
actually called ncurses.h), but linking with buildroot's ncurses library that
doesn't support widechar. Hence the ^@ everywhere.

 By the way, the issue is not limited to pkg-config: older versions of
check-lxdialog.sh would just add -I/usr/include/ncurses if they find an
ncurses.h there, and that ncurses.h might be a widechar version...

> 
>  >  The question is how to resolve this. The correct thing to do would be to
>  > install the .pc file, and to do what Thomas proposed a while ago:
>  > $(HOST_DIR)/bin/pkg-config returns the host config, $(CROSS_COMPILE)pkg-config
>  > returns the target config. But this will probably result in breakage of a lot of
>  > packages...
> 
>  >  Any better ideas?
> 
> Is that really needed? Can't we just change linux.mk to use
> HOST_MAKE_ENV instead of TARGET_MAKE_ENV (it is a target package, but it
> only needs to link against host libraries). That wil ensure that the
> PKG_CONFIG_* environment variables are set, so if we were to install a
> ncurses(w) pc file for the host, kconfig should find and use it?

 Unfortunately, we have a fragment in pkg-kconfig.mk that *removes* the
PKG_CONFIG_* stuff from the environment...

 Also, the same issue may happen for *any* package using (recent) kconfig. I
didn't find an example that does it, but it's possible that a kconfig-package
adds TARGET_CONFIGURE_OPTS to FOO_MAKE_ENV, and this would cause the same issue
(TARGET_CONFIGURE_OPTS includes LDFLAGS_FOR_BUILD="$(HOST_LDFLAGS)" and
HOST_LDFLAGS adds -L$(HOST_DIR)/lib).

 So my patch enabling widechar is the simplest solution :-). Not perfect of
course, because that patch will break again if the host has an ncurses.h which
is NOT widechar...

 Regards,
 Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH] package/ncurses: enable widechar for host build

[Arnout Vandecappelle]

Kconfig uses either pkg-config or hard-coded /usr/include paths to find
the ncurses or ncursesw library. If ncursesw is found, it will include
<ncursesw.h>. Since Buildroot's host-ncurses doesn't install a .pc file,
and linux.mk anyway doesn't pass the pkg-config options to find the host
pkg-config files, Kconfig will always find the system's ncursesw.h.

However, since commit dde090c299 (linux: fix passing of host CFLAGS and
LDFLAGS) HOST_LDFLAGS is passed to the linux build system. Thus, if
host-ncurses was already built before 'make linux-menuconfig' is called,
the build will pick up libncurses from the host directory, which is NOT
widechar. Thus, two different ncurses configurations are mixed into the
final mconf program. This will result in serious breakage in the
rendering of the menus (lots of @ and question mark characters).

As a workaround, just build host-ncurses with widechar support. That
makes it compatible with ncursesw.h picked up from the host. Clearly,
this is not a robust solution. But it's simple, it works, and it
shouldn't hurt to build host-ncurses with widechar support (we can
safely assume that the system doesn't have a non-widechar uClibc
library...).

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Cc: David De Grave <david.degrave@essensium.com>
Cc: Scott Fan <fancp2007@gmail.com>
---
v2:
 - fix typo in commit message;
 - extend commit message;
 - keep alphabetic ordering of configure options (Scott)

Note: I haven't tested what happens on a system which doesn't have
ncursesw.h. But I guess such a system is so exotic that it's not
worthwhile to do something special for it.
---
 package/ncurses/ncurses.mk | 1 +
 1 file changed, 1 insertion(+)

diff --git a/package/ncurses/ncurses.mk b/package/ncurses/ncurses.mk
index 90cf4a4dc5..f9cd0d3fbd 100644
--- a/package/ncurses/ncurses.mk
+++ b/package/ncurses/ncurses.mk
@@ -147,6 +147,7 @@ define HOST_NCURSES_BUILD_CMDS
 endef
 
 HOST_NCURSES_CONF_OPTS = \
+	--enable-widec \
 	--with-shared \
 	--without-gpm \
 	--without-manpages \
-- 
2.17.0

[Buildroot] [PATCH] package/ncurses: enable widechar for host build

[Yann E. MORIN]

Arnout, All,

On 2018-05-03 09:58 +0200, Arnout Vandecappelle (Essensium/Mind) spake thusly:
> Kconfig uses either pkg-config or hard-coded /usr/include paths to find
> the ncurses or ncursesw library. If ncursesw is found, it will include
> <ncursesw.h>. Since Buildroot's host-ncurses doesn't install a .pc file,
> and linux.mk anyway doesn't pass the pkg-config options to find the host
> pkg-config files, Kconfig will always find the system's ncursesw.h.
> 
> However, since commit dde090c299 (linux: fix passing of host CFLAGS and
> LDFLAGS) HOST_LDFLAGS is passed to the linux build system. Thus, if
> host-ncurses was already built before 'make linux-menuconfig' is called,
> the build will pick up libncurses from the host directory, which is NOT
> widechar. Thus, two different ncurses configurations are mixed into the
> final mconf program. This will result in serious breakage in the
> rendering of the menus (lots of @ and question mark characters).

What about overriding HOSTCC when calling linux-menuconfig:

    diff --git a/linux/linux.mk b/linux/linux.mk
    index 9e646baffc..75a256ff9e 100644
    --- a/linux/linux.mk
    +++ b/linux/linux.mk
    @@ -235,7 +235,7 @@ LINUX_KCONFIG_FILE = $(call qstrip,$(BR2_LINUX_KERNEL_CUSTOM_CONFIG_FILE))
     endif
     LINUX_KCONFIG_FRAGMENT_FILES = $(call qstrip,$(BR2_LINUX_KERNEL_CONFIG_FRAGMENT_FILES))
     LINUX_KCONFIG_EDITORS = menuconfig xconfig gconfig nconfig
    -LINUX_KCONFIG_OPTS = $(LINUX_MAKE_FLAGS)
    +LINUX_KCONFIG_OPTS = $(LINUX_MAKE_FLAGS) HOSTCC="$(HOSTCC)"
     
     # If no package has yet set it, set it from the Kconfig option
     LINUX_NEEDS_MODULES ?= $(BR2_LINUX_NEEDS_MODULES)

We should not need any library from HOST_DIR when we run menuconfig.

Note: we may still want to enable widechar in host-ncurses for other
reasons, though. But with the proposdal above, there is no lopnger any
mix of system ncurses vs our own.

Regards,
Yann E. MORIN.

> As a workaround, just build host-ncurses with widechar support. That
> makes it compatible with ncursesw.h picked up from the host. Clearly,
> this is not a robust solution. But it's simple, it works, and it
> shouldn't hurt to build host-ncurses with widechar support (we can
> safely assume that the system doesn't have a non-widechar uClibc
> library...).
> 
> Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
> Cc: David De Grave <david.degrave@essensium.com>
> Cc: Scott Fan <fancp2007@gmail.com>
> ---
> v2:
>  - fix typo in commit message;
>  - extend commit message;
>  - keep alphabetic ordering of configure options (Scott)
> 
> Note: I haven't tested what happens on a system which doesn't have
> ncursesw.h. But I guess such a system is so exotic that it's not
> worthwhile to do something special for it.
> ---
>  package/ncurses/ncurses.mk | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/package/ncurses/ncurses.mk b/package/ncurses/ncurses.mk
> index 90cf4a4dc5..f9cd0d3fbd 100644
> --- a/package/ncurses/ncurses.mk
> +++ b/package/ncurses/ncurses.mk
> @@ -147,6 +147,7 @@ define HOST_NCURSES_BUILD_CMDS
>  endef
>  
>  HOST_NCURSES_CONF_OPTS = \
> +	--enable-widec \
>  	--with-shared \
>  	--without-gpm \
>  	--without-manpages \
> -- 
> 2.17.0
> 
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH] package/ncurses: enable widechar for host build

[Thomas Petazzoni]

Hello,

On Thu, 3 May 2018 21:45:07 +0200, Yann E. MORIN wrote:

> What about overriding HOSTCC when calling linux-menuconfig:
> 
>     diff --git a/linux/linux.mk b/linux/linux.mk
>     index 9e646baffc..75a256ff9e 100644
>     --- a/linux/linux.mk
>     +++ b/linux/linux.mk
>     @@ -235,7 +235,7 @@ LINUX_KCONFIG_FILE = $(call qstrip,$(BR2_LINUX_KERNEL_CUSTOM_CONFIG_FILE))
>      endif
>      LINUX_KCONFIG_FRAGMENT_FILES = $(call qstrip,$(BR2_LINUX_KERNEL_CONFIG_FRAGMENT_FILES))
>      LINUX_KCONFIG_EDITORS = menuconfig xconfig gconfig nconfig
>     -LINUX_KCONFIG_OPTS = $(LINUX_MAKE_FLAGS)
>     +LINUX_KCONFIG_OPTS = $(LINUX_MAKE_FLAGS) HOSTCC="$(HOSTCC)"
>      
>      # If no package has yet set it, set it from the Kconfig option
>      LINUX_NEEDS_MODULES ?= $(BR2_LINUX_NEEDS_MODULES)
> 
> We should not need any library from HOST_DIR when we run menuconfig.
> 
> Note: we may still want to enable widechar in host-ncurses for other
> reasons, though. But with the proposdal above, there is no lopnger any
> mix of system ncurses vs our own.

Arnout, any comment on Yann's proposal? I have to say my preference
goes to Yann's solution, as it's the one that ensures there is no
mixing of ncurses between the one we build in HOST_DIR and the one
provided by the system.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH] package/ncurses: enable widechar for host build

[Arnout Vandecappelle]

On 08-05-18 15:31, Thomas Petazzoni wrote:
> Hello,
> 
> On Thu, 3 May 2018 21:45:07 +0200, Yann E. MORIN wrote:
> 
>> What about overriding HOSTCC when calling linux-menuconfig:
>>
>>     diff --git a/linux/linux.mk b/linux/linux.mk
>>     index 9e646baffc..75a256ff9e 100644
>>     --- a/linux/linux.mk
>>     +++ b/linux/linux.mk
>>     @@ -235,7 +235,7 @@ LINUX_KCONFIG_FILE = $(call qstrip,$(BR2_LINUX_KERNEL_CUSTOM_CONFIG_FILE))
>>      endif
>>      LINUX_KCONFIG_FRAGMENT_FILES = $(call qstrip,$(BR2_LINUX_KERNEL_CONFIG_FRAGMENT_FILES))
>>      LINUX_KCONFIG_EDITORS = menuconfig xconfig gconfig nconfig
>>     -LINUX_KCONFIG_OPTS = $(LINUX_MAKE_FLAGS)
>>     +LINUX_KCONFIG_OPTS = $(LINUX_MAKE_FLAGS) HOSTCC="$(HOSTCC)"
>>      
>>      # If no package has yet set it, set it from the Kconfig option
>>      LINUX_NEEDS_MODULES ?= $(BR2_LINUX_NEEDS_MODULES)
>>
>> We should not need any library from HOST_DIR when we run menuconfig.
>>
>> Note: we may still want to enable widechar in host-ncurses for other
>> reasons, though. But with the proposdal above, there is no lopnger any
>> mix of system ncurses vs our own.
> 
> Arnout, any comment on Yann's proposal? I have to say my preference
> goes to Yann's solution, as it's the one that ensures there is no
> mixing of ncurses between the one we build in HOST_DIR and the one
> provided by the system.

 As noted in another thread, the same problem might occur for another package.
Basically, kconfig will fail if we pass the normal HOST_CONFIGURE_OPTS. But I
guess it's not likely that we ever have a package that does that, since kconfig
packages are normally target packages, and kconfig doesn't listen to
CFLAGS_FOR_BUILD/LDFLAGS_FOR_BUILD.

 So OK, Yann, can you prepare a patch? I've marked this patch as Rejected.

 Regards,
 Arnout


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF