* [Buildroot] [PATCH 1/1] Change host locale from C to en_US.UTF-8 for reproducable builds @ 2018-01-28 11:22 Julius Kriukas 2018-01-30 22:38 ` Thomas Petazzoni 2018-01-31 21:36 ` [Buildroot] [PATCH v2] " Julius Kriukas 0 siblings, 2 replies; 19+ messages in thread From: Julius Kriukas @ 2018-01-28 11:22 UTC (permalink / raw) To: buildroot When BR2_REPRODUCIBLE is enabled building systemd fails with error: [1/1080] Generating systemd.bg.catalog with a meson_exe.py custom command. FAILED: catalog/systemd.bg.catalog /buildroot/output/host/bin/python3 /buildroot/output/host/bin/meson --internal exe /buildroot/output/build/systemd-236/build/meson-private/meson_exe_sed_232a0623cc7ce2cd67ec72ed784b76307102ed76.dat Warning: You are using 'ANSI_X3.4-1968' which is not a Unicode-compatible locale. You might see errors if you use UTF-8 strings as filenames, as strings, or as file contents. Please switch to a UTF-8 locale for your platform. ... UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 1079: ordinal not in range(128) package/pkg-generic.mk:247: recipe for target '/buildroot/output/build/systemd-236/.stamp_built' failed make: *** [/buildroot/output/build/systemd-236/.stamp_built] Error 1 Other packages using meson build system might fail as well. This patch changes default host system locale from C to en_US.UTF-8 to fix this issue. Any Unicode-compatible locale would be suitable here. Signed-off-by: Julius Kriukas <julius@kriukas.lt> --- Makefile | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Makefile b/Makefile index 7d8ab51a8c..5991c6a221 100644 --- a/Makefile +++ b/Makefile @@ -250,8 +250,8 @@ endif # timezone and locale may affect build output ifeq ($(BR2_REPRODUCIBLE),y) export TZ = UTC -export LANG = C -export LC_ALL = C +export LANG = en_US.UTF-8 +export LC_ALL = en_US.UTF-8 export GZIP = -n BR2_VERSION_GIT_EPOCH = $(shell GIT_DIR=$(TOPDIR)/.git $(GIT) log -1 --format=%at) export SOURCE_DATE_EPOCH ?= $(if $(wildcard $(TOPDIR)/.git),$(BR2_VERSION_GIT_EPOCH),$(BR2_VERSION_EPOCH)) -- 2.15.1 ^ permalink raw reply related [flat|nested] 19+ messages in thread
* [Buildroot] [PATCH 1/1] Change host locale from C to en_US.UTF-8 for reproducable builds 2018-01-28 11:22 [Buildroot] [PATCH 1/1] Change host locale from C to en_US.UTF-8 for reproducable builds Julius Kriukas @ 2018-01-30 22:38 ` Thomas Petazzoni 2018-01-31 21:34 ` Julius Kriukas 2018-01-31 21:36 ` [Buildroot] [PATCH v2] " Julius Kriukas 1 sibling, 1 reply; 19+ messages in thread From: Thomas Petazzoni @ 2018-01-30 22:38 UTC (permalink / raw) To: buildroot Hello, On Sun, 28 Jan 2018 13:22:05 +0200, Julius Kriukas wrote: > When BR2_REPRODUCIBLE is enabled building systemd fails with error: > > [1/1080] Generating systemd.bg.catalog with a meson_exe.py custom command. > FAILED: catalog/systemd.bg.catalog > /buildroot/output/host/bin/python3 /buildroot/output/host/bin/meson --internal exe /buildroot/output/build/systemd-236/build/meson-private/meson_exe_sed_232a0623cc7ce2cd67ec72ed784b76307102ed76.dat > Warning: You are using 'ANSI_X3.4-1968' which is not a Unicode-compatible locale. > You might see errors if you use UTF-8 strings as filenames, as strings, or as file contents. > Please switch to a UTF-8 locale for your platform. > ... > UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 1079: ordinal not in range(128) > package/pkg-generic.mk:247: recipe for target '/buildroot/output/build/systemd-236/.stamp_built' failed > make: *** [/buildroot/output/build/systemd-236/.stamp_built] Error 1 > > Other packages using meson build system might fail as well. > > This patch changes default host system locale from C to en_US.UTF-8 to > fix this issue. Any Unicode-compatible locale would be suitable here. > > Signed-off-by: Julius Kriukas <julius@kriukas.lt> Thanks for your patch! I was about to apply it, but one issue I see is that we have no guarantee that en_US.UTF-8 is available. support/dependencies/dependencies.sh checks that we have at least one UTF-8 locale available if BR2_NEEDS_HOST_UTF8_LOCALE=y, but: - BR2_NEEDS_HOST_UTF8_LOCALE is only selected when a uClibc toolchain is being built - it only ensures one UTF-8 locale is available, not specifically en_US.UTF-8 Perhaps we need another BR2_NEEDS_HOST_EN_US_UTF8_LOCALE, which would be used by support/dependencies/dependencies.sh to ensure that en_US.UTF-8 is available. This new option would be selected by BR2_REPRODUCIBLE. Arnout, Peter, Yann? Thoughts? Best regards, Thomas -- Thomas Petazzoni, CTO, Free Electrons Embedded Linux and Kernel engineering http://free-electrons.com ^ permalink raw reply [flat|nested] 19+ messages in thread
* [Buildroot] [PATCH 1/1] Change host locale from C to en_US.UTF-8 for reproducable builds 2018-01-30 22:38 ` Thomas Petazzoni @ 2018-01-31 21:34 ` Julius Kriukas 0 siblings, 0 replies; 19+ messages in thread From: Julius Kriukas @ 2018-01-31 21:34 UTC (permalink / raw) To: buildroot On 01/31/2018 12:38 AM, Thomas Petazzoni wrote: > Hello, > > On Sun, 28 Jan 2018 13:22:05 +0200, Julius Kriukas wrote: >> When BR2_REPRODUCIBLE is enabled building systemd fails with error: >> >> [1/1080] Generating systemd.bg.catalog with a meson_exe.py custom command. >> FAILED: catalog/systemd.bg.catalog >> /buildroot/output/host/bin/python3 /buildroot/output/host/bin/meson --internal exe /buildroot/output/build/systemd-236/build/meson-private/meson_exe_sed_232a0623cc7ce2cd67ec72ed784b76307102ed76.dat >> Warning: You are using 'ANSI_X3.4-1968' which is not a Unicode-compatible locale. >> You might see errors if you use UTF-8 strings as filenames, as strings, or as file contents. >> Please switch to a UTF-8 locale for your platform. >> ... >> UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 1079: ordinal not in range(128) >> package/pkg-generic.mk:247: recipe for target '/buildroot/output/build/systemd-236/.stamp_built' failed >> make: *** [/buildroot/output/build/systemd-236/.stamp_built] Error 1 >> >> Other packages using meson build system might fail as well. >> >> This patch changes default host system locale from C to en_US.UTF-8 to >> fix this issue. Any Unicode-compatible locale would be suitable here. >> >> Signed-off-by: Julius Kriukas <julius@kriukas.lt> > > Thanks for your patch! I was about to apply it, but one issue I see is > that we have no guarantee that en_US.UTF-8 is available. > > support/dependencies/dependencies.sh checks that we have at least one > UTF-8 locale available if BR2_NEEDS_HOST_UTF8_LOCALE=y, but: > > - BR2_NEEDS_HOST_UTF8_LOCALE is only selected when a uClibc toolchain > is being built > > - it only ensures one UTF-8 locale is available, not specifically > en_US.UTF-8 > > Perhaps we need another BR2_NEEDS_HOST_EN_US_UTF8_LOCALE, which would > be used by support/dependencies/dependencies.sh to ensure that > en_US.UTF-8 is available. This new option would be selected by > BR2_REPRODUCIBLE. > > Arnout, Peter, Yann? Thoughts? > > Best regards, > > Thomas > I think Debian uses C.UTF-8 locale in similar circumstances, but I suppose it is too much Debian specific (for example it seems C.UTF-8 locale is not available by default on CentOS 7 systems). I favor Thomas proposal to add en_US.UTF-8 availability detection with a new flag. It seemed overly complicated at first but it solves the issue and I do not have anything better to suggest. -- Julius Kriukas ^ permalink raw reply [flat|nested] 19+ messages in thread
* [Buildroot] [PATCH v2] Change host locale from C to en_US.UTF-8 for reproducable builds 2018-01-28 11:22 [Buildroot] [PATCH 1/1] Change host locale from C to en_US.UTF-8 for reproducable builds Julius Kriukas 2018-01-30 22:38 ` Thomas Petazzoni @ 2018-01-31 21:36 ` Julius Kriukas 2018-02-04 15:19 ` Yann E. MORIN ` (2 more replies) 1 sibling, 3 replies; 19+ messages in thread From: Julius Kriukas @ 2018-01-31 21:36 UTC (permalink / raw) To: buildroot When BR2_REPRODUCIBLE is enabled building systemd fails with error: [1/1080] Generating systemd.bg.catalog with a meson_exe.py custom command. FAILED: catalog/systemd.bg.catalog /buildroot/output/host/bin/python3 /buildroot/output/host/bin/meson --internal exe /buildroot/output/build/systemd-236/build/meson-private/meson_exe_sed_232a0623cc7ce2cd67ec72ed784b76307102ed76.dat Warning: You are using 'ANSI_X3.4-1968' which is not a Unicode-compatible locale. You might see errors if you use UTF-8 strings as filenames, as strings, or as file contents. Please switch to a UTF-8 locale for your platform. ... UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 1079: ordinal not in range(128) package/pkg-generic.mk:247: recipe for target '/buildroot/output/build/systemd-236/.stamp_built' failed make: *** [/buildroot/output/build/systemd-236/.stamp_built] Error 1 Other packages using meson build system might fail as well. This patch changes default host system locale from C to en_US.UTF-8 to fix this issue. It also introduces BR2_NEEDS_HOST_EN_US_UTF8_LOCALE hidden flag that checks if this locale is available on the host system. If locale is not available error message is show and build process is stopped. --- Changes v1 -> v2: - Added BR2_NEEDS_HOST_EN_US_UTF8_LOCALE flag that is selected by BR2_REPRODUCIBLE. - Extended support/dependencies/dependencies.sh to check for en_US.UTF-8 locale availability if BR2_NEEDS_HOST_EN_US_UTF8_LOCALE is set. Signed-off-by: Julius Kriukas <julius@kriukas.lt> --- Config.in | 6 ++++++ Makefile | 4 ++-- support/dependencies/dependencies.sh | 13 +++++++++++++ 3 files changed, 21 insertions(+), 2 deletions(-) diff --git a/Config.in b/Config.in index 62d67ce957..bd14226edc 100644 --- a/Config.in +++ b/Config.in @@ -94,6 +94,11 @@ config BR2_HOSTARCH_NEEDS_IA32_COMPILER config BR2_NEEDS_HOST_UTF8_LOCALE bool +# Hidden boolean selected by packages that explicitly need en_US.UTF-8 locale +# to be available on host system. +config BR2_NEEDS_HOST_EN_US_UTF8_LOCALE + bool + source "arch/Config.in" menu "Build options" @@ -715,6 +720,7 @@ config BR2_REPRODUCIBLE bool "Make the build reproducible (experimental)" # SOURCE_DATE_EPOCH support in toolchain-wrapper requires GCC 4.4 depends on BR2_TOOLCHAIN_GCC_AT_LEAST_4_4 + select BR2_NEEDS_HOST_EN_US_UTF8_LOCALE help This option will remove all sources of non-reproducibility from the build process. For a given Buildroot configuration, diff --git a/Makefile b/Makefile index 7d8ab51a8c..5991c6a221 100644 --- a/Makefile +++ b/Makefile @@ -250,8 +250,8 @@ endif # timezone and locale may affect build output ifeq ($(BR2_REPRODUCIBLE),y) export TZ = UTC -export LANG = C -export LC_ALL = C +export LANG = en_US.UTF-8 +export LC_ALL = en_US.UTF-8 export GZIP = -n BR2_VERSION_GIT_EPOCH = $(shell GIT_DIR=$(TOPDIR)/.git $(GIT) log -1 --format=%at) export SOURCE_DATE_EPOCH ?= $(if $(wildcard $(TOPDIR)/.git),$(BR2_VERSION_GIT_EPOCH),$(BR2_VERSION_EPOCH)) diff --git a/support/dependencies/dependencies.sh b/support/dependencies/dependencies.sh index a195c62c8c..4a5b55e9da 100755 --- a/support/dependencies/dependencies.sh +++ b/support/dependencies/dependencies.sh @@ -207,6 +207,19 @@ if grep ^BR2_NEEDS_HOST_UTF8_LOCALE=y $BR2_CONFIG > /dev/null; then fi fi +if grep ^BR2_NEEDS_HOST_EN_US_UTF8_LOCALE=y $BR2_CONFIG > /dev/null; then + if ! which locale > /dev/null ; then + echo + echo "You need locale support on your build machine" + exit 1 ; + fi + if ! LC_ALL=en_US.UTF-8 locale -c charmap | grep '^UTF-8$' > /dev/null; then + echo + echo "You need en_US.UTF-8 locale suppport on the host system" + exit 1 ; + fi +fi + if grep -q ^BR2_NEEDS_HOST_JAVA=y $BR2_CONFIG ; then check_prog_host "java" JAVA_GCJ=$(java -version 2>&1 | grep gcj) -- 2.15.1 ^ permalink raw reply related [flat|nested] 19+ messages in thread
* [Buildroot] [PATCH v2] Change host locale from C to en_US.UTF-8 for reproducable builds 2018-01-31 21:36 ` [Buildroot] [PATCH v2] " Julius Kriukas @ 2018-02-04 15:19 ` Yann E. MORIN 2018-02-06 16:32 ` Thomas Petazzoni 2018-02-12 21:34 ` [Buildroot] [PATCH v3] systemd: switch to en_US.UTF-8 locale when building Julius Kriukas 2 siblings, 0 replies; 19+ messages in thread From: Yann E. MORIN @ 2018-02-04 15:19 UTC (permalink / raw) To: buildroot Julius, All, On 2018-01-31 23:36 +0200, Julius Kriukas spake thusly: > When BR2_REPRODUCIBLE is enabled building systemd fails with error: > > [1/1080] Generating systemd.bg.catalog with a meson_exe.py custom command. > FAILED: catalog/systemd.bg.catalog > /buildroot/output/host/bin/python3 /buildroot/output/host/bin/meson --internal exe /buildroot/output/build/systemd-236/build/meson-private/meson_exe_sed_232a0623cc7ce2cd67ec72ed784b76307102ed76.dat > Warning: You are using 'ANSI_X3.4-1968' which is not a Unicode-compatible locale. > You might see errors if you use UTF-8 strings as filenames, as strings, or as file contents. > Please switch to a UTF-8 locale for your platform. > ... > UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 1079: ordinal not in range(128) > package/pkg-generic.mk:247: recipe for target '/buildroot/output/build/systemd-236/.stamp_built' failed > make: *** [/buildroot/output/build/systemd-236/.stamp_built] Error 1 > > Other packages using meson build system might fail as well. > > This patch changes default host system locale from C to en_US.UTF-8 to > fix this issue. It also introduces BR2_NEEDS_HOST_EN_US_UTF8_LOCALE > hidden flag that checks if this locale is available on the host system. > If locale is not available error message is show and build process is > stopped. Yeah, I like this change. Yet, people that already use an UTF-8 locale (e.g fr_FR.UTF-8, or en_GB.UTF-8 or whatever_else.UTF-8) will be very disapointed that their build now fail when they were previously successful... :-/ But see below... > diff --git a/Config.in b/Config.in > index 62d67ce957..bd14226edc 100644 > --- a/Config.in > +++ b/Config.in > @@ -94,6 +94,11 @@ config BR2_HOSTARCH_NEEDS_IA32_COMPILER > config BR2_NEEDS_HOST_UTF8_LOCALE > bool > > +# Hidden boolean selected by packages that explicitly need en_US.UTF-8 locale > +# to be available on host system. > +config BR2_NEEDS_HOST_EN_US_UTF8_LOCALE > + bool So, this option is supposed to be selectable by packages, but it is only ever selected by the reproducible option, and never used to directly set the locale. What I would have expected, too, would be that systemd also selects BR2_NEEDS_HOST_EN_US_UTF8_LOCALE, and the build commands for systemd are changed such as (simplified for brevity): SYSTEMD_ENV = $(TARGET_MAKE_ENV) LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 define SYSTEMD_CONFIGURE_CMDS $(SYSTEMD_ENV) meson $(SYSTEMD_CONF_OPTS) $(@D) $(@D)/build endef define SYSTEMD_BUILD_CMDS $(SYSTEMD_ENV) ninja $(SYSTEMD_NINJA_OPTS) -C $(@D)/build endef ... and so on... > source "arch/Config.in" > > menu "Build options" > @@ -715,6 +720,7 @@ config BR2_REPRODUCIBLE > bool "Make the build reproducible (experimental)" > # SOURCE_DATE_EPOCH support in toolchain-wrapper requires GCC 4.4 > depends on BR2_TOOLCHAIN_GCC_AT_LEAST_4_4 > + select BR2_NEEDS_HOST_EN_US_UTF8_LOCALE If it were me, this option would ultimately disappear, and we would only do reproducible builds. There is no point in hell to generate non-reproducible builds... :-/ (Yeah, I know, not everyone is at ease reading tools' output messages not in their language.) Regards, Yann E. MORIN. > help > This option will remove all sources of non-reproducibility > from the build process. For a given Buildroot configuration, > diff --git a/Makefile b/Makefile > index 7d8ab51a8c..5991c6a221 100644 > --- a/Makefile > +++ b/Makefile > @@ -250,8 +250,8 @@ endif > # timezone and locale may affect build output > ifeq ($(BR2_REPRODUCIBLE),y) > export TZ = UTC > -export LANG = C > -export LC_ALL = C > +export LANG = en_US.UTF-8 > +export LC_ALL = en_US.UTF-8 > export GZIP = -n > BR2_VERSION_GIT_EPOCH = $(shell GIT_DIR=$(TOPDIR)/.git $(GIT) log -1 --format=%at) > export SOURCE_DATE_EPOCH ?= $(if $(wildcard $(TOPDIR)/.git),$(BR2_VERSION_GIT_EPOCH),$(BR2_VERSION_EPOCH)) > diff --git a/support/dependencies/dependencies.sh b/support/dependencies/dependencies.sh > index a195c62c8c..4a5b55e9da 100755 > --- a/support/dependencies/dependencies.sh > +++ b/support/dependencies/dependencies.sh > @@ -207,6 +207,19 @@ if grep ^BR2_NEEDS_HOST_UTF8_LOCALE=y $BR2_CONFIG > /dev/null; then > fi > fi > > +if grep ^BR2_NEEDS_HOST_EN_US_UTF8_LOCALE=y $BR2_CONFIG > /dev/null; then > + if ! which locale > /dev/null ; then > + echo > + echo "You need locale support on your build machine" > + exit 1 ; > + fi > + if ! LC_ALL=en_US.UTF-8 locale -c charmap | grep '^UTF-8$' > /dev/null; then > + echo > + echo "You need en_US.UTF-8 locale suppport on the host system" > + exit 1 ; > + fi > +fi > + > if grep -q ^BR2_NEEDS_HOST_JAVA=y $BR2_CONFIG ; then > check_prog_host "java" > JAVA_GCJ=$(java -version 2>&1 | grep gcj) > -- > 2.15.1 > -- .-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' ^ permalink raw reply [flat|nested] 19+ messages in thread
* [Buildroot] [PATCH v2] Change host locale from C to en_US.UTF-8 for reproducable builds 2018-01-31 21:36 ` [Buildroot] [PATCH v2] " Julius Kriukas 2018-02-04 15:19 ` Yann E. MORIN @ 2018-02-06 16:32 ` Thomas Petazzoni 2018-02-12 21:34 ` [Buildroot] [PATCH v3] systemd: switch to en_US.UTF-8 locale when building Julius Kriukas 2 siblings, 0 replies; 19+ messages in thread From: Thomas Petazzoni @ 2018-02-06 16:32 UTC (permalink / raw) To: buildroot Hello, On Wed, 31 Jan 2018 23:36:29 +0200, Julius Kriukas wrote: > When BR2_REPRODUCIBLE is enabled building systemd fails with error: > > [1/1080] Generating systemd.bg.catalog with a meson_exe.py custom command. > FAILED: catalog/systemd.bg.catalog > /buildroot/output/host/bin/python3 /buildroot/output/host/bin/meson --internal exe /buildroot/output/build/systemd-236/build/meson-private/meson_exe_sed_232a0623cc7ce2cd67ec72ed784b76307102ed76.dat > Warning: You are using 'ANSI_X3.4-1968' which is not a Unicode-compatible locale. > You might see errors if you use UTF-8 strings as filenames, as strings, or as file contents. > Please switch to a UTF-8 locale for your platform. > ... > UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 1079: ordinal not in range(128) > package/pkg-generic.mk:247: recipe for target '/buildroot/output/build/systemd-236/.stamp_built' failed > make: *** [/buildroot/output/build/systemd-236/.stamp_built] Error 1 > > Other packages using meson build system might fail as well. > > This patch changes default host system locale from C to en_US.UTF-8 to > fix this issue. It also introduces BR2_NEEDS_HOST_EN_US_UTF8_LOCALE > hidden flag that checks if this locale is available on the host system. > If locale is not available error message is show and build process is > stopped. > > --- > Changes v1 -> v2: > - Added BR2_NEEDS_HOST_EN_US_UTF8_LOCALE flag that is selected by > BR2_REPRODUCIBLE. > - Extended support/dependencies/dependencies.sh to check for > en_US.UTF-8 locale availability if BR2_NEEDS_HOST_EN_US_UTF8_LOCALE > is set. > > Signed-off-by: Julius Kriukas <julius@kriukas.lt> Your SoB line should be before the "---" that separates the commit log and the changelog. However, I have a bigger issue with the patch: you're fixing the issue for BR2_REPRODUCIBLE, but the problem actually appears also with a non-BR2_REPRODUCIBLE build. Look at: http://autobuild.buildroot.net/results/257/25757c41d1a83edbe2a7c47b63ce2e43eb6141dc/build-end.log This configuration does *not* have BR2_REPRODUCIBLE enabled. The system on which it runs has several UTF-8 locales enabled: $ locale -a C C.UTF-8 POSIX en_US.utf8 fr_FR.utf8 But the POSIX locale is used: $ locale LANG= LANGUAGE= LC_CTYPE="POSIX" LC_NUMERIC="POSIX" LC_TIME="POSIX" LC_COLLATE="POSIX" LC_MONETARY="POSIX" LC_MESSAGES="POSIX" LC_PAPER="POSIX" LC_NAME="POSIX" LC_ADDRESS="POSIX" LC_TELEPHONE="POSIX" LC_MEASUREMENT="POSIX" LC_IDENTIFICATION="POSIX" LC_ALL= And Meson fails in this situation. I believe it should be fixed in Meson. We should not have to set a specific locale for the thing to build. The need for the UTF-8 locale to be present is more or less OK, but the need to have such a locale in *use* is annoying. So perhaps your fix is needed because Meson will in the end really need at least one UTF-8 locale available, but it is not complete because it doesn't fix the non-BR2_REPRODUCIBLE situation. Best regards, Thomas -- Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons) Embedded Linux and Kernel engineering http://bootlin.com ^ permalink raw reply [flat|nested] 19+ messages in thread
* [Buildroot] [PATCH v3] systemd: switch to en_US.UTF-8 locale when building 2018-01-31 21:36 ` [Buildroot] [PATCH v2] " Julius Kriukas 2018-02-04 15:19 ` Yann E. MORIN 2018-02-06 16:32 ` Thomas Petazzoni @ 2018-02-12 21:34 ` Julius Kriukas 2018-02-17 18:06 ` John Keeping ` (2 more replies) 2 siblings, 3 replies; 19+ messages in thread From: Julius Kriukas @ 2018-02-12 21:34 UTC (permalink / raw) To: buildroot When BR2_REPRODUCIBLE is enabled or host uses non UTF-8 capable locale building systemd fails with an error: [1/1080] Generating systemd.bg.catalog with a meson_exe.py custom command. FAILED: catalog/systemd.bg.catalog /buildroot/output/host/bin/python3 /buildroot/output/host/bin/meson --internal exe /buildroot/output/build/systemd-236/build/meson-private/meson_exe_sed_232a0623cc7ce2cd67ec72ed784b76307102ed76.dat Warning: You are using 'ANSI_X3.4-1968' which is not a Unicode-compatible locale. You might see errors if you use UTF-8 strings as filenames, as strings, or as file contents. Please switch to a UTF-8 locale for your platform. ... UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 1079: ordinal not in range(128) package/pkg-generic.mk:247: recipe for target '/buildroot/output/build/systemd-236/.stamp_built' failed make: *** [/buildroot/output/build/systemd-236/.stamp_built] Error 1 This patch changes default host system locale from C to en_US.UTF-8 when building systemd package to fix this issue. It also introduces BR2_NEEDS_HOST_EN_US_UTF8_LOCALE flag that checks if this locale is available on the host system. If locale is not available error message is show and build process is stopped. Signed-off-by: Julius Kriukas <julius@kriukas.lt> --- Changes v1 -> v2: - Added BR2_NEEDS_HOST_EN_US_UTF8_LOCALE flag that is selected by BR2_REPRODUCIBLE. - Extended support/dependencies/dependencies.sh to check for en_US.UTF-8 locale availability if BR2_NEEDS_HOST_EN_US_UTF8_LOCALE is set. Changes v2 -> v3: - Always select BR2_NEEDS_HOST_EN_US_UTF8_LOCALE by systemd package. - Do not select BR2_NEEDS_HOST_EN_US_UTF8_LOCALE by BR2_REPRODUCIBLE. - Keep global locale set to C if BR2_REPRODUCIBLE=y. - Use 'grep -q' instead of '> /dev/null' in dependencies.sh script. --- Config.in | 5 +++++ package/systemd/Config.in | 1 + package/systemd/systemd.mk | 10 ++++++---- support/dependencies/dependencies.sh | 13 +++++++++++++ 4 files changed, 25 insertions(+), 4 deletions(-) diff --git a/Config.in b/Config.in index 0002df5176..8d471dfb2e 100644 --- a/Config.in +++ b/Config.in @@ -94,6 +94,11 @@ config BR2_HOSTARCH_NEEDS_IA32_COMPILER config BR2_NEEDS_HOST_UTF8_LOCALE bool +# Hidden boolean selected by packages that explicitly need en_US.UTF-8 locale +# to be available on host system. +config BR2_NEEDS_HOST_EN_US_UTF8_LOCALE + bool + source "arch/Config.in" menu "Build options" diff --git a/package/systemd/Config.in b/package/systemd/Config.in index 661f40dee4..742b3f1f1d 100644 --- a/package/systemd/Config.in +++ b/package/systemd/Config.in @@ -27,6 +27,7 @@ menuconfig BR2_PACKAGE_SYSTEMD select BR2_PACKAGE_BUSYBOX_SHOW_OTHERS # kmod-tools select BR2_PACKAGE_KMOD_TOOLS select BR2_TARGET_TZ_INFO + select BR2_NEEDS_HOST_EN_US_UTF8_LOCALE help systemd is a system and service manager for Linux, compatible with SysV and LSB init scripts. systemd provides diff --git a/package/systemd/systemd.mk b/package/systemd/systemd.mk index 1c31ebb34c..58798f32df 100644 --- a/package/systemd/systemd.mk +++ b/package/systemd/systemd.mk @@ -385,23 +385,25 @@ endef SYSTEMD_NINJA_OPTS = $(if $(VERBOSE),-v) -j$(PARALLEL_JOBS) +SYSTEMD_ENV = $(TARGET_MAKE_ENV) LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 + define SYSTEMD_CONFIGURE_CMDS rm -rf $(@D)/build mkdir -p $(@D)/build - $(TARGET_MAKE_ENV) meson $(SYSTEMD_CONF_OPTS) $(@D) $(@D)/build + $(SYSTEMD_ENV) meson $(SYSTEMD_CONF_OPTS) $(@D) $(@D)/build endef define SYSTEMD_BUILD_CMDS - $(TARGET_MAKE_ENV) ninja $(SYSTEMD_NINJA_OPTS) -C $(@D)/build + $(SYSTEMD_ENV) ninja $(SYSTEMD_NINJA_OPTS) -C $(@D)/build endef define SYSTEMD_INSTALL_TARGET_CMDS - $(TARGET_MAKE_ENV) DESTDIR=$(TARGET_DIR) ninja $(SYSTEMD_NINJA_OPTS) \ + $(SYSTEMD_ENV) DESTDIR=$(TARGET_DIR) ninja $(SYSTEMD_NINJA_OPTS) \ -C $(@D)/build install endef define SYSTEMD_INSTALL_STAGING_CMDS - $(TARGET_MAKE_ENV) DESTDIR=$(STAGING_DIR) ninja $(SYSTEMD_NINJA_OPTS) \ + $(SYSTEMD_ENV) DESTDIR=$(STAGING_DIR) ninja $(SYSTEMD_NINJA_OPTS) \ -C $(@D)/build install endef diff --git a/support/dependencies/dependencies.sh b/support/dependencies/dependencies.sh index 1804e85508..881f07008c 100755 --- a/support/dependencies/dependencies.sh +++ b/support/dependencies/dependencies.sh @@ -207,6 +207,19 @@ if grep ^BR2_NEEDS_HOST_UTF8_LOCALE=y $BR2_CONFIG > /dev/null; then fi fi +if grep -q ^BR2_NEEDS_HOST_EN_US_UTF8_LOCALE=y $BR2_CONFIG; then + if ! which locale > /dev/null ; then + echo + echo "You need locale support on your build machine" + exit 1 ; + fi + if ! LC_ALL=en_US.UTF-8 locale -c charmap | grep -q '^UTF-8$'; then + echo + echo "You need en_US.UTF-8 locale suppport on the host system" + exit 1 ; + fi +fi + if grep -q ^BR2_NEEDS_HOST_JAVA=y $BR2_CONFIG ; then check_prog_host "java" JAVA_GCJ=$(java -version 2>&1 | grep gcj) -- 2.15.1 ^ permalink raw reply related [flat|nested] 19+ messages in thread
* [Buildroot] [PATCH v3] systemd: switch to en_US.UTF-8 locale when building 2018-02-12 21:34 ` [Buildroot] [PATCH v3] systemd: switch to en_US.UTF-8 locale when building Julius Kriukas @ 2018-02-17 18:06 ` John Keeping 2018-02-22 23:02 ` Julius Kriukas 2018-02-22 23:00 ` [Buildroot] [PATCH v4] " Julius Kriukas 2018-02-23 9:06 ` [Buildroot] [PATCH v3] " Arnout Vandecappelle 2 siblings, 1 reply; 19+ messages in thread From: John Keeping @ 2018-02-17 18:06 UTC (permalink / raw) To: buildroot On Mon, 12 Feb 2018 23:34:00 +0200, Julius Kriukas wrote: > When BR2_REPRODUCIBLE is enabled or host uses non UTF-8 capable locale > building systemd fails with an error: > > [1/1080] Generating systemd.bg.catalog with a meson_exe.py custom command. > FAILED: catalog/systemd.bg.catalog > /buildroot/output/host/bin/python3 /buildroot/output/host/bin/meson --internal exe /buildroot/output/build/systemd-236/build/meson-private/meson_exe_sed_232a0623cc7ce2cd67ec72ed784b76307102ed76.dat > Warning: You are using 'ANSI_X3.4-1968' which is not a Unicode-compatible locale. > You might see errors if you use UTF-8 strings as filenames, as strings, or as file contents. > Please switch to a UTF-8 locale for your platform. > ... > UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 1079: ordinal not in range(128) > package/pkg-generic.mk:247: recipe for target '/buildroot/output/build/systemd-236/.stamp_built' failed > make: *** [/buildroot/output/build/systemd-236/.stamp_built] Error 1 > > This patch changes default host system locale from C to en_US.UTF-8 when > building systemd package to fix this issue. It also introduces > BR2_NEEDS_HOST_EN_US_UTF8_LOCALE flag that checks if this locale is available on > the host system. If locale is not available error message is show and build > process is stopped. > > Signed-off-by: Julius Kriukas <julius@kriukas.lt> Tested-by: John Keeping <john@metanate.com> And, FWIW, with the indentation in Config.in fixed, Reviewed-by: John Keeping <john@metanate.com> > --- > Changes v1 -> v2: > - Added BR2_NEEDS_HOST_EN_US_UTF8_LOCALE flag that is selected by > BR2_REPRODUCIBLE. > - Extended support/dependencies/dependencies.sh to check for > en_US.UTF-8 locale availability if BR2_NEEDS_HOST_EN_US_UTF8_LOCALE > is set. > > Changes v2 -> v3: > - Always select BR2_NEEDS_HOST_EN_US_UTF8_LOCALE by systemd package. > - Do not select BR2_NEEDS_HOST_EN_US_UTF8_LOCALE by BR2_REPRODUCIBLE. > - Keep global locale set to C if BR2_REPRODUCIBLE=y. > - Use 'grep -q' instead of '> /dev/null' in dependencies.sh script. > --- > Config.in | 5 +++++ > package/systemd/Config.in | 1 + > package/systemd/systemd.mk | 10 ++++++---- > support/dependencies/dependencies.sh | 13 +++++++++++++ > 4 files changed, 25 insertions(+), 4 deletions(-) > > diff --git a/Config.in b/Config.in > index 0002df5176..8d471dfb2e 100644 > --- a/Config.in > +++ b/Config.in > @@ -94,6 +94,11 @@ config BR2_HOSTARCH_NEEDS_IA32_COMPILER > config BR2_NEEDS_HOST_UTF8_LOCALE > bool > > +# Hidden boolean selected by packages that explicitly need en_US.UTF-8 locale > +# to be available on host system. > +config BR2_NEEDS_HOST_EN_US_UTF8_LOCALE > + bool > + > source "arch/Config.in" > > menu "Build options" > diff --git a/package/systemd/Config.in b/package/systemd/Config.in > index 661f40dee4..742b3f1f1d 100644 > --- a/package/systemd/Config.in > +++ b/package/systemd/Config.in > @@ -27,6 +27,7 @@ menuconfig BR2_PACKAGE_SYSTEMD > select BR2_PACKAGE_BUSYBOX_SHOW_OTHERS # kmod-tools > select BR2_PACKAGE_KMOD_TOOLS > select BR2_TARGET_TZ_INFO > + select BR2_NEEDS_HOST_EN_US_UTF8_LOCALE This should use TAB for indentation. > help > systemd is a system and service manager for Linux, > compatible with SysV and LSB init scripts. systemd provides > diff --git a/package/systemd/systemd.mk b/package/systemd/systemd.mk > index 1c31ebb34c..58798f32df 100644 > --- a/package/systemd/systemd.mk > +++ b/package/systemd/systemd.mk > @@ -385,23 +385,25 @@ endef > > SYSTEMD_NINJA_OPTS = $(if $(VERBOSE),-v) -j$(PARALLEL_JOBS) > > +SYSTEMD_ENV = $(TARGET_MAKE_ENV) LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 > + > define SYSTEMD_CONFIGURE_CMDS > rm -rf $(@D)/build > mkdir -p $(@D)/build > - $(TARGET_MAKE_ENV) meson $(SYSTEMD_CONF_OPTS) $(@D) $(@D)/build > + $(SYSTEMD_ENV) meson $(SYSTEMD_CONF_OPTS) $(@D) $(@D)/build > endef > > define SYSTEMD_BUILD_CMDS > - $(TARGET_MAKE_ENV) ninja $(SYSTEMD_NINJA_OPTS) -C $(@D)/build > + $(SYSTEMD_ENV) ninja $(SYSTEMD_NINJA_OPTS) -C $(@D)/build > endef > > define SYSTEMD_INSTALL_TARGET_CMDS > - $(TARGET_MAKE_ENV) DESTDIR=$(TARGET_DIR) ninja $(SYSTEMD_NINJA_OPTS) \ > + $(SYSTEMD_ENV) DESTDIR=$(TARGET_DIR) ninja $(SYSTEMD_NINJA_OPTS) \ > -C $(@D)/build install > endef > > define SYSTEMD_INSTALL_STAGING_CMDS > - $(TARGET_MAKE_ENV) DESTDIR=$(STAGING_DIR) ninja $(SYSTEMD_NINJA_OPTS) \ > + $(SYSTEMD_ENV) DESTDIR=$(STAGING_DIR) ninja $(SYSTEMD_NINJA_OPTS) \ > -C $(@D)/build install > endef > > diff --git a/support/dependencies/dependencies.sh b/support/dependencies/dependencies.sh > index 1804e85508..881f07008c 100755 > --- a/support/dependencies/dependencies.sh > +++ b/support/dependencies/dependencies.sh > @@ -207,6 +207,19 @@ if grep ^BR2_NEEDS_HOST_UTF8_LOCALE=y $BR2_CONFIG > /dev/null; then > fi > fi > > +if grep -q ^BR2_NEEDS_HOST_EN_US_UTF8_LOCALE=y $BR2_CONFIG; then > + if ! which locale > /dev/null ; then > + echo > + echo "You need locale support on your build machine" > + exit 1 ; > + fi > + if ! LC_ALL=en_US.UTF-8 locale -c charmap | grep -q '^UTF-8$'; then > + echo > + echo "You need en_US.UTF-8 locale suppport on the host system" > + exit 1 ; > + fi > +fi > + > if grep -q ^BR2_NEEDS_HOST_JAVA=y $BR2_CONFIG ; then > check_prog_host "java" > JAVA_GCJ=$(java -version 2>&1 | grep gcj) ^ permalink raw reply [flat|nested] 19+ messages in thread
* [Buildroot] [PATCH v3] systemd: switch to en_US.UTF-8 locale when building 2018-02-17 18:06 ` John Keeping @ 2018-02-22 23:02 ` Julius Kriukas 0 siblings, 0 replies; 19+ messages in thread From: Julius Kriukas @ 2018-02-22 23:02 UTC (permalink / raw) To: buildroot On 02/17/2018 08:06 PM, John Keeping wrote: > On Mon, 12 Feb 2018 23:34:00 +0200, Julius Kriukas wrote: > >> When BR2_REPRODUCIBLE is enabled or host uses non UTF-8 capable locale >> building systemd fails with an error: >> >> [1/1080] Generating systemd.bg.catalog with a meson_exe.py custom command. >> FAILED: catalog/systemd.bg.catalog >> /buildroot/output/host/bin/python3 /buildroot/output/host/bin/meson --internal exe /buildroot/output/build/systemd-236/build/meson-private/meson_exe_sed_232a0623cc7ce2cd67ec72ed784b76307102ed76.dat >> Warning: You are using 'ANSI_X3.4-1968' which is not a Unicode-compatible locale. >> You might see errors if you use UTF-8 strings as filenames, as strings, or as file contents. >> Please switch to a UTF-8 locale for your platform. >> ... >> UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 1079: ordinal not in range(128) >> package/pkg-generic.mk:247: recipe for target '/buildroot/output/build/systemd-236/.stamp_built' failed >> make: *** [/buildroot/output/build/systemd-236/.stamp_built] Error 1 >> >> This patch changes default host system locale from C to en_US.UTF-8 when >> building systemd package to fix this issue. It also introduces >> BR2_NEEDS_HOST_EN_US_UTF8_LOCALE flag that checks if this locale is available on >> the host system. If locale is not available error message is show and build >> process is stopped. >> >> Signed-off-by: Julius Kriukas <julius@kriukas.lt> > > Tested-by: John Keeping <john@metanate.com> > > And, FWIW, with the indentation in Config.in fixed, > > Reviewed-by: John Keeping <john@metanate.com> > >> --- >> Changes v1 -> v2: >> - Added BR2_NEEDS_HOST_EN_US_UTF8_LOCALE flag that is selected by >> BR2_REPRODUCIBLE. >> - Extended support/dependencies/dependencies.sh to check for >> en_US.UTF-8 locale availability if BR2_NEEDS_HOST_EN_US_UTF8_LOCALE >> is set. >> >> Changes v2 -> v3: >> - Always select BR2_NEEDS_HOST_EN_US_UTF8_LOCALE by systemd package. >> - Do not select BR2_NEEDS_HOST_EN_US_UTF8_LOCALE by BR2_REPRODUCIBLE. >> - Keep global locale set to C if BR2_REPRODUCIBLE=y. >> - Use 'grep -q' instead of '> /dev/null' in dependencies.sh script. >> --- >> Config.in | 5 +++++ >> package/systemd/Config.in | 1 + >> package/systemd/systemd.mk | 10 ++++++---- >> support/dependencies/dependencies.sh | 13 +++++++++++++ >> 4 files changed, 25 insertions(+), 4 deletions(-) >> >> diff --git a/Config.in b/Config.in >> index 0002df5176..8d471dfb2e 100644 >> --- a/Config.in >> +++ b/Config.in >> @@ -94,6 +94,11 @@ config BR2_HOSTARCH_NEEDS_IA32_COMPILER >> config BR2_NEEDS_HOST_UTF8_LOCALE >> bool >> >> +# Hidden boolean selected by packages that explicitly need en_US.UTF-8 locale >> +# to be available on host system. >> +config BR2_NEEDS_HOST_EN_US_UTF8_LOCALE >> + bool >> + >> source "arch/Config.in" >> >> menu "Build options" >> diff --git a/package/systemd/Config.in b/package/systemd/Config.in >> index 661f40dee4..742b3f1f1d 100644 >> --- a/package/systemd/Config.in >> +++ b/package/systemd/Config.in >> @@ -27,6 +27,7 @@ menuconfig BR2_PACKAGE_SYSTEMD >> select BR2_PACKAGE_BUSYBOX_SHOW_OTHERS # kmod-tools >> select BR2_PACKAGE_KMOD_TOOLS >> select BR2_TARGET_TZ_INFO >> + select BR2_NEEDS_HOST_EN_US_UTF8_LOCALE > > This should use TAB for indentation. > >> help >> systemd is a system and service manager for Linux, >> compatible with SysV and LSB init scripts. systemd provides >> diff --git a/package/systemd/systemd.mk b/package/systemd/systemd.mk >> index 1c31ebb34c..58798f32df 100644 >> --- a/package/systemd/systemd.mk >> +++ b/package/systemd/systemd.mk >> @@ -385,23 +385,25 @@ endef >> >> SYSTEMD_NINJA_OPTS = $(if $(VERBOSE),-v) -j$(PARALLEL_JOBS) >> >> +SYSTEMD_ENV = $(TARGET_MAKE_ENV) LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 >> + >> define SYSTEMD_CONFIGURE_CMDS >> rm -rf $(@D)/build >> mkdir -p $(@D)/build >> - $(TARGET_MAKE_ENV) meson $(SYSTEMD_CONF_OPTS) $(@D) $(@D)/build >> + $(SYSTEMD_ENV) meson $(SYSTEMD_CONF_OPTS) $(@D) $(@D)/build >> endef >> >> define SYSTEMD_BUILD_CMDS >> - $(TARGET_MAKE_ENV) ninja $(SYSTEMD_NINJA_OPTS) -C $(@D)/build >> + $(SYSTEMD_ENV) ninja $(SYSTEMD_NINJA_OPTS) -C $(@D)/build >> endef >> >> define SYSTEMD_INSTALL_TARGET_CMDS >> - $(TARGET_MAKE_ENV) DESTDIR=$(TARGET_DIR) ninja $(SYSTEMD_NINJA_OPTS) \ >> + $(SYSTEMD_ENV) DESTDIR=$(TARGET_DIR) ninja $(SYSTEMD_NINJA_OPTS) \ >> -C $(@D)/build install >> endef >> >> define SYSTEMD_INSTALL_STAGING_CMDS >> - $(TARGET_MAKE_ENV) DESTDIR=$(STAGING_DIR) ninja $(SYSTEMD_NINJA_OPTS) \ >> + $(SYSTEMD_ENV) DESTDIR=$(STAGING_DIR) ninja $(SYSTEMD_NINJA_OPTS) \ >> -C $(@D)/build install >> endef >> >> diff --git a/support/dependencies/dependencies.sh b/support/dependencies/dependencies.sh >> index 1804e85508..881f07008c 100755 >> --- a/support/dependencies/dependencies.sh >> +++ b/support/dependencies/dependencies.sh >> @@ -207,6 +207,19 @@ if grep ^BR2_NEEDS_HOST_UTF8_LOCALE=y $BR2_CONFIG > /dev/null; then >> fi >> fi >> >> +if grep -q ^BR2_NEEDS_HOST_EN_US_UTF8_LOCALE=y $BR2_CONFIG; then >> + if ! which locale > /dev/null ; then >> + echo >> + echo "You need locale support on your build machine" >> + exit 1 ; >> + fi >> + if ! LC_ALL=en_US.UTF-8 locale -c charmap | grep -q '^UTF-8$'; then >> + echo >> + echo "You need en_US.UTF-8 locale suppport on the host system" >> + exit 1 ; >> + fi >> +fi >> + >> if grep -q ^BR2_NEEDS_HOST_JAVA=y $BR2_CONFIG ; then >> check_prog_host "java" >> JAVA_GCJ=$(java -version 2>&1 | grep gcj) Hi John. Thanks for spotting the indentation issue and review. I am submitting v4 of this patch with the issue fixed. -- Julius Kriukas ^ permalink raw reply [flat|nested] 19+ messages in thread
* [Buildroot] [PATCH v4] systemd: switch to en_US.UTF-8 locale when building 2018-02-12 21:34 ` [Buildroot] [PATCH v3] systemd: switch to en_US.UTF-8 locale when building Julius Kriukas 2018-02-17 18:06 ` John Keeping @ 2018-02-22 23:00 ` Julius Kriukas 2018-02-28 21:31 ` Yann E. MORIN 2018-03-02 21:45 ` Thomas Petazzoni 2018-02-23 9:06 ` [Buildroot] [PATCH v3] " Arnout Vandecappelle 2 siblings, 2 replies; 19+ messages in thread From: Julius Kriukas @ 2018-02-22 23:00 UTC (permalink / raw) To: buildroot When BR2_REPRODUCIBLE is enabled or host uses non UTF-8 capable locale building systemd fails with an error: [1/1080] Generating systemd.bg.catalog with a meson_exe.py custom command. FAILED: catalog/systemd.bg.catalog /buildroot/output/host/bin/python3 /buildroot/output/host/bin/meson --internal exe /buildroot/output/build/systemd-236/build/meson-private/meson_exe_sed_232a0623cc7ce2cd67ec72ed784b76307102ed76.dat Warning: You are using 'ANSI_X3.4-1968' which is not a Unicode-compatible locale. You might see errors if you use UTF-8 strings as filenames, as strings, or as file contents. Please switch to a UTF-8 locale for your platform. ... UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 1079: ordinal not in range(128) package/pkg-generic.mk:247: recipe for target '/buildroot/output/build/systemd-236/.stamp_built' failed make: *** [/buildroot/output/build/systemd-236/.stamp_built] Error 1 This patch changes default host system locale from C to en_US.UTF-8 when building systemd package to fix this issue. It also introduces BR2_NEEDS_HOST_EN_US_UTF8_LOCALE flag that checks if this locale is available on the host system. If locale is not available error message is show and build process is stopped. Signed-off-by: Julius Kriukas <julius@kriukas.lt> --- Changes v1 -> v2: - Added BR2_NEEDS_HOST_EN_US_UTF8_LOCALE flag that is selected by BR2_REPRODUCIBLE. - Extended support/dependencies/dependencies.sh to check for en_US.UTF-8 locale availability if BR2_NEEDS_HOST_EN_US_UTF8_LOCALE is set. Changes v2 -> v3: - Always select BR2_NEEDS_HOST_EN_US_UTF8_LOCALE by systemd package. - Do not select BR2_NEEDS_HOST_EN_US_UTF8_LOCALE by BR2_REPRODUCIBLE. - Keep global locale set to C if BR2_REPRODUCIBLE=y. - Use 'grep -q' instead of '> /dev/null' in dependencies.sh script. Changes v3 -> v4: - Fix indentation in package/systemd/Config.in, spaces -> tab. --- Config.in | 5 +++++ package/systemd/Config.in | 1 + package/systemd/systemd.mk | 10 ++++++---- support/dependencies/dependencies.sh | 13 +++++++++++++ 4 files changed, 25 insertions(+), 4 deletions(-) diff --git a/Config.in b/Config.in index 0002df5176..8d471dfb2e 100644 --- a/Config.in +++ b/Config.in @@ -94,6 +94,11 @@ config BR2_HOSTARCH_NEEDS_IA32_COMPILER config BR2_NEEDS_HOST_UTF8_LOCALE bool +# Hidden boolean selected by packages that explicitly need en_US.UTF-8 locale +# to be available on host system. +config BR2_NEEDS_HOST_EN_US_UTF8_LOCALE + bool + source "arch/Config.in" menu "Build options" diff --git a/package/systemd/Config.in b/package/systemd/Config.in index 661f40dee4..e83f75c733 100644 --- a/package/systemd/Config.in +++ b/package/systemd/Config.in @@ -27,6 +27,7 @@ menuconfig BR2_PACKAGE_SYSTEMD select BR2_PACKAGE_BUSYBOX_SHOW_OTHERS # kmod-tools select BR2_PACKAGE_KMOD_TOOLS select BR2_TARGET_TZ_INFO + select BR2_NEEDS_HOST_EN_US_UTF8_LOCALE help systemd is a system and service manager for Linux, compatible with SysV and LSB init scripts. systemd provides diff --git a/package/systemd/systemd.mk b/package/systemd/systemd.mk index 126727d602..9a6a9cfcbc 100644 --- a/package/systemd/systemd.mk +++ b/package/systemd/systemd.mk @@ -385,23 +385,25 @@ endef SYSTEMD_NINJA_OPTS = $(if $(VERBOSE),-v) -j$(PARALLEL_JOBS) +SYSTEMD_ENV = $(TARGET_MAKE_ENV) LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 + define SYSTEMD_CONFIGURE_CMDS rm -rf $(@D)/build mkdir -p $(@D)/build - $(TARGET_MAKE_ENV) meson $(SYSTEMD_CONF_OPTS) $(@D) $(@D)/build + $(SYSTEMD_ENV) meson $(SYSTEMD_CONF_OPTS) $(@D) $(@D)/build endef define SYSTEMD_BUILD_CMDS - $(TARGET_MAKE_ENV) ninja $(SYSTEMD_NINJA_OPTS) -C $(@D)/build + $(SYSTEMD_ENV) ninja $(SYSTEMD_NINJA_OPTS) -C $(@D)/build endef define SYSTEMD_INSTALL_TARGET_CMDS - $(TARGET_MAKE_ENV) DESTDIR=$(TARGET_DIR) ninja $(SYSTEMD_NINJA_OPTS) \ + $(SYSTEMD_ENV) DESTDIR=$(TARGET_DIR) ninja $(SYSTEMD_NINJA_OPTS) \ -C $(@D)/build install endef define SYSTEMD_INSTALL_STAGING_CMDS - $(TARGET_MAKE_ENV) DESTDIR=$(STAGING_DIR) ninja $(SYSTEMD_NINJA_OPTS) \ + $(SYSTEMD_ENV) DESTDIR=$(STAGING_DIR) ninja $(SYSTEMD_NINJA_OPTS) \ -C $(@D)/build install endef diff --git a/support/dependencies/dependencies.sh b/support/dependencies/dependencies.sh index 1804e85508..881f07008c 100755 --- a/support/dependencies/dependencies.sh +++ b/support/dependencies/dependencies.sh @@ -207,6 +207,19 @@ if grep ^BR2_NEEDS_HOST_UTF8_LOCALE=y $BR2_CONFIG > /dev/null; then fi fi +if grep -q ^BR2_NEEDS_HOST_EN_US_UTF8_LOCALE=y $BR2_CONFIG; then + if ! which locale > /dev/null ; then + echo + echo "You need locale support on your build machine" + exit 1 ; + fi + if ! LC_ALL=en_US.UTF-8 locale -c charmap | grep -q '^UTF-8$'; then + echo + echo "You need en_US.UTF-8 locale suppport on the host system" + exit 1 ; + fi +fi + if grep -q ^BR2_NEEDS_HOST_JAVA=y $BR2_CONFIG ; then check_prog_host "java" JAVA_GCJ=$(java -version 2>&1 | grep gcj) -- 2.16.1 ^ permalink raw reply related [flat|nested] 19+ messages in thread
* [Buildroot] [PATCH v4] systemd: switch to en_US.UTF-8 locale when building 2018-02-22 23:00 ` [Buildroot] [PATCH v4] " Julius Kriukas @ 2018-02-28 21:31 ` Yann E. MORIN 2018-02-28 22:13 ` Thomas Petazzoni 2018-03-02 21:45 ` Thomas Petazzoni 1 sibling, 1 reply; 19+ messages in thread From: Yann E. MORIN @ 2018-02-28 21:31 UTC (permalink / raw) To: buildroot Julius, All, On 2018-02-23 01:00 +0200, Julius Kriukas spake thusly: > When BR2_REPRODUCIBLE is enabled or host uses non UTF-8 capable locale > building systemd fails with an error: As far as I can see, this patch is now no longer about REPRODCIBLE, but only about systemd; that's good. But then, the commit log should drop the part about REPRODUCIBLE. > [1/1080] Generating systemd.bg.catalog with a meson_exe.py custom command. > FAILED: catalog/systemd.bg.catalog > /buildroot/output/host/bin/python3 /buildroot/output/host/bin/meson --internal exe /buildroot/output/build/systemd-236/build/meson-private/meson_exe_sed_232a0623cc7ce2cd67ec72ed784b76307102ed76.dat > Warning: You are using 'ANSI_X3.4-1968' which is not a Unicode-compatible locale. > You might see errors if you use UTF-8 strings as filenames, as strings, or as file contents. > Please switch to a UTF-8 locale for your platform. > ... > UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 1079: ordinal not in range(128) > package/pkg-generic.mk:247: recipe for target '/buildroot/output/build/systemd-236/.stamp_built' failed > make: *** [/buildroot/output/build/systemd-236/.stamp_built] Error 1 > > This patch changes default host system locale from C to en_US.UTF-8 when > building systemd package to fix this issue. It also introduces > BR2_NEEDS_HOST_EN_US_UTF8_LOCALE flag that checks if this locale is available on > the host system. If locale is not available error message is show and build > process is stopped. OK, so I do not have en_USUTF-8 on my machine. I have en_GB.UTF-8 (which I am using). So I'm left out in the cold with this patch. However, C.UTF-8 is virtually available everywhere. And I was able to build systemd with C.UTF-8 without any problem. So, I would prefer, if we have to force anything, that we force the use of C.UTF-8. But then that also means that we override the users' environments, users that may prefer to see localised messages. So, as a stop-gap measure, I'm OK with this patch as long as it is switched to using C.UTF-8. Also, rename the variable, something like: BR2_NEEDS_HOST_UTF8_LOCALE_CURRENT Note: this is different from the existing BR2_NEEDS_HOST_UTF8_LOCALE: it onlhy requires that a UTF-8 locale be present on the system, while we now require that it be the current locale. Regards, Yann E. MORIN. > Signed-off-by: Julius Kriukas <julius@kriukas.lt> > > --- > Changes v1 -> v2: > - Added BR2_NEEDS_HOST_EN_US_UTF8_LOCALE flag that is selected by > BR2_REPRODUCIBLE. > - Extended support/dependencies/dependencies.sh to check for > en_US.UTF-8 locale availability if BR2_NEEDS_HOST_EN_US_UTF8_LOCALE > is set. > > Changes v2 -> v3: > - Always select BR2_NEEDS_HOST_EN_US_UTF8_LOCALE by systemd package. > - Do not select BR2_NEEDS_HOST_EN_US_UTF8_LOCALE by BR2_REPRODUCIBLE. > - Keep global locale set to C if BR2_REPRODUCIBLE=y. > - Use 'grep -q' instead of '> /dev/null' in dependencies.sh script. > > Changes v3 -> v4: > - Fix indentation in package/systemd/Config.in, spaces -> tab. > --- > Config.in | 5 +++++ > package/systemd/Config.in | 1 + > package/systemd/systemd.mk | 10 ++++++---- > support/dependencies/dependencies.sh | 13 +++++++++++++ > 4 files changed, 25 insertions(+), 4 deletions(-) > > diff --git a/Config.in b/Config.in > index 0002df5176..8d471dfb2e 100644 > --- a/Config.in > +++ b/Config.in > @@ -94,6 +94,11 @@ config BR2_HOSTARCH_NEEDS_IA32_COMPILER > config BR2_NEEDS_HOST_UTF8_LOCALE > bool > > +# Hidden boolean selected by packages that explicitly need en_US.UTF-8 locale > +# to be available on host system. > +config BR2_NEEDS_HOST_EN_US_UTF8_LOCALE > + bool > + > source "arch/Config.in" > > menu "Build options" > diff --git a/package/systemd/Config.in b/package/systemd/Config.in > index 661f40dee4..e83f75c733 100644 > --- a/package/systemd/Config.in > +++ b/package/systemd/Config.in > @@ -27,6 +27,7 @@ menuconfig BR2_PACKAGE_SYSTEMD > select BR2_PACKAGE_BUSYBOX_SHOW_OTHERS # kmod-tools > select BR2_PACKAGE_KMOD_TOOLS > select BR2_TARGET_TZ_INFO > + select BR2_NEEDS_HOST_EN_US_UTF8_LOCALE > help > systemd is a system and service manager for Linux, > compatible with SysV and LSB init scripts. systemd provides > diff --git a/package/systemd/systemd.mk b/package/systemd/systemd.mk > index 126727d602..9a6a9cfcbc 100644 > --- a/package/systemd/systemd.mk > +++ b/package/systemd/systemd.mk > @@ -385,23 +385,25 @@ endef > > SYSTEMD_NINJA_OPTS = $(if $(VERBOSE),-v) -j$(PARALLEL_JOBS) > > +SYSTEMD_ENV = $(TARGET_MAKE_ENV) LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 > + > define SYSTEMD_CONFIGURE_CMDS > rm -rf $(@D)/build > mkdir -p $(@D)/build > - $(TARGET_MAKE_ENV) meson $(SYSTEMD_CONF_OPTS) $(@D) $(@D)/build > + $(SYSTEMD_ENV) meson $(SYSTEMD_CONF_OPTS) $(@D) $(@D)/build > endef > > define SYSTEMD_BUILD_CMDS > - $(TARGET_MAKE_ENV) ninja $(SYSTEMD_NINJA_OPTS) -C $(@D)/build > + $(SYSTEMD_ENV) ninja $(SYSTEMD_NINJA_OPTS) -C $(@D)/build > endef > > define SYSTEMD_INSTALL_TARGET_CMDS > - $(TARGET_MAKE_ENV) DESTDIR=$(TARGET_DIR) ninja $(SYSTEMD_NINJA_OPTS) \ > + $(SYSTEMD_ENV) DESTDIR=$(TARGET_DIR) ninja $(SYSTEMD_NINJA_OPTS) \ > -C $(@D)/build install > endef > > define SYSTEMD_INSTALL_STAGING_CMDS > - $(TARGET_MAKE_ENV) DESTDIR=$(STAGING_DIR) ninja $(SYSTEMD_NINJA_OPTS) \ > + $(SYSTEMD_ENV) DESTDIR=$(STAGING_DIR) ninja $(SYSTEMD_NINJA_OPTS) \ > -C $(@D)/build install > endef > > diff --git a/support/dependencies/dependencies.sh b/support/dependencies/dependencies.sh > index 1804e85508..881f07008c 100755 > --- a/support/dependencies/dependencies.sh > +++ b/support/dependencies/dependencies.sh > @@ -207,6 +207,19 @@ if grep ^BR2_NEEDS_HOST_UTF8_LOCALE=y $BR2_CONFIG > /dev/null; then > fi > fi > > +if grep -q ^BR2_NEEDS_HOST_EN_US_UTF8_LOCALE=y $BR2_CONFIG; then > + if ! which locale > /dev/null ; then > + echo > + echo "You need locale support on your build machine" > + exit 1 ; > + fi > + if ! LC_ALL=en_US.UTF-8 locale -c charmap | grep -q '^UTF-8$'; then > + echo > + echo "You need en_US.UTF-8 locale suppport on the host system" > + exit 1 ; > + fi > +fi > + > if grep -q ^BR2_NEEDS_HOST_JAVA=y $BR2_CONFIG ; then > check_prog_host "java" > JAVA_GCJ=$(java -version 2>&1 | grep gcj) > -- > 2.16.1 > -- .-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' ^ permalink raw reply [flat|nested] 19+ messages in thread
* [Buildroot] [PATCH v4] systemd: switch to en_US.UTF-8 locale when building 2018-02-28 21:31 ` Yann E. MORIN @ 2018-02-28 22:13 ` Thomas Petazzoni 2018-03-01 18:06 ` Yann E. MORIN 0 siblings, 1 reply; 19+ messages in thread From: Thomas Petazzoni @ 2018-02-28 22:13 UTC (permalink / raw) To: buildroot Hello, On Wed, 28 Feb 2018 22:31:04 +0100, Yann E. MORIN wrote: > OK, so I do not have en_USUTF-8 on my machine. I have en_GB.UTF-8 (which > I am using). So I'm left out in the cold with this patch. > > However, C.UTF-8 is virtually available everywhere. And I was able to > build systemd with C.UTF-8 without any problem. > > So, I would prefer, if we have to force anything, that we force the > use of C.UTF-8. > > But then that also means that we override the users' environments, users > that may prefer to see localised messages. > > So, as a stop-gap measure, I'm OK with this patch as long as it is > switched to using C.UTF-8. Also, rename the variable, something like: > > BR2_NEEDS_HOST_UTF8_LOCALE_CURRENT > > Note: this is different from the existing BR2_NEEDS_HOST_UTF8_LOCALE: it > onlhy requires that a UTF-8 locale be present on the system, while we > now require that it be the current locale. Now, we don't require it to be the current locale: > > +SYSTEMD_ENV = $(TARGET_MAKE_ENV) LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 The systemd package ensures to pass LANG/LC_ALL pointing to an UTF-8 locale. So even if your current locale is something else, as long as en_US.UTF-8 is *available*, things are going to work. That being said, I also dislike this solution, and think the sytemd build system should be fixed instead. Thomas -- Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons) Embedded Linux and Kernel engineering http://bootlin.com ^ permalink raw reply [flat|nested] 19+ messages in thread
* [Buildroot] [PATCH v4] systemd: switch to en_US.UTF-8 locale when building 2018-02-28 22:13 ` Thomas Petazzoni @ 2018-03-01 18:06 ` Yann E. MORIN 2018-03-01 18:50 ` Peter Korsgaard 0 siblings, 1 reply; 19+ messages in thread From: Yann E. MORIN @ 2018-03-01 18:06 UTC (permalink / raw) To: buildroot Thomas, All, On 2018-02-28 23:13 +0100, Thomas Petazzoni spake thusly: > On Wed, 28 Feb 2018 22:31:04 +0100, Yann E. MORIN wrote: [--SNIP--] > > Note: this is different from the existing BR2_NEEDS_HOST_UTF8_LOCALE: it > > onlhy requires that a UTF-8 locale be present on the system, while we > > now require that it be the current locale. > Now, we don't require it to be the current locale: > > > +SYSTEMD_ENV = $(TARGET_MAKE_ENV) LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 > The systemd package ensures to pass LANG/LC_ALL pointing to an UTF-8 > locale. Ah, yes, you're right, sorry... I was confused now, because (without this patch) I had to change my locale to be able to build systemd. My bad, sorry... > That being said, I also dislike this solution, and think the sytemd > build system should be fixed instead. Yup, that's really sad... But properly fixing it is a totally different endeavour... :-/ Regards, Yann E. MORIN. -- .-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' ^ permalink raw reply [flat|nested] 19+ messages in thread
* [Buildroot] [PATCH v4] systemd: switch to en_US.UTF-8 locale when building 2018-03-01 18:06 ` Yann E. MORIN @ 2018-03-01 18:50 ` Peter Korsgaard 2018-03-01 19:12 ` Yann E. MORIN 0 siblings, 1 reply; 19+ messages in thread From: Peter Korsgaard @ 2018-03-01 18:50 UTC (permalink / raw) To: buildroot >>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes: Hi, > Ah, yes, you're right, sorry... > I was confused now, because (without this patch) I had to change my > locale to be able to build systemd. My bad, sorry... Wait, are you saying that it doesn't build with en_GB.UTF-8 locale? That's odd. -- Bye, Peter Korsgaard ^ permalink raw reply [flat|nested] 19+ messages in thread
* [Buildroot] [PATCH v4] systemd: switch to en_US.UTF-8 locale when building 2018-03-01 18:50 ` Peter Korsgaard @ 2018-03-01 19:12 ` Yann E. MORIN 2018-03-01 20:04 ` Peter Korsgaard 0 siblings, 1 reply; 19+ messages in thread From: Yann E. MORIN @ 2018-03-01 19:12 UTC (permalink / raw) To: buildroot Peter,All, On 2018-03-01 19:50 +0100, Peter Korsgaard spake thusly: > >>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes: > > Ah, yes, you're right, sorry... > > > I was confused now, because (without this patch) I had to change my > > locale to be able to build systemd. My bad, sorry... > > Wait, are you saying that it doesn't build with en_GB.UTF-8 locale? > That's odd. No, no, don't worry! In my locale, en_GB.UTF-8, systemd-v237 *does* build fine. But on my autobuilder, which only has C, POSIX, and C.UTF-8, the default being POSIX, I had to explicitly set it to C.UTF-8 to test without the patch. So, don't worry, all is well when the current locale is any of *.UTF-8. Regards, Yann E. MORIN. -- .-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' ^ permalink raw reply [flat|nested] 19+ messages in thread
* [Buildroot] [PATCH v4] systemd: switch to en_US.UTF-8 locale when building 2018-03-01 19:12 ` Yann E. MORIN @ 2018-03-01 20:04 ` Peter Korsgaard 0 siblings, 0 replies; 19+ messages in thread From: Peter Korsgaard @ 2018-03-01 20:04 UTC (permalink / raw) To: buildroot >>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes: Hi, >> Wait, are you saying that it doesn't build with en_GB.UTF-8 locale? >> That's odd. > No, no, don't worry! > In my locale, en_GB.UTF-8, systemd-v237 *does* build fine. > But on my autobuilder, which only has C, POSIX, and C.UTF-8, the default > being POSIX, I had to explicitly set it to C.UTF-8 to test without the > patch. > So, don't worry, all is well when the current locale is any of *.UTF-8. *Pheeuw* ;) -- Bye, Peter Korsgaard ^ permalink raw reply [flat|nested] 19+ messages in thread
* [Buildroot] [PATCH v4] systemd: switch to en_US.UTF-8 locale when building 2018-02-22 23:00 ` [Buildroot] [PATCH v4] " Julius Kriukas 2018-02-28 21:31 ` Yann E. MORIN @ 2018-03-02 21:45 ` Thomas Petazzoni [not found] ` <87vaecbsf2.fsf@paral.in> 1 sibling, 1 reply; 19+ messages in thread From: Thomas Petazzoni @ 2018-03-02 21:45 UTC (permalink / raw) To: buildroot Hello, On Fri, 23 Feb 2018 01:00:55 +0200, Julius Kriukas wrote: > When BR2_REPRODUCIBLE is enabled or host uses non UTF-8 capable locale > building systemd fails with an error: > > [1/1080] Generating systemd.bg.catalog with a meson_exe.py custom command. > FAILED: catalog/systemd.bg.catalog > /buildroot/output/host/bin/python3 /buildroot/output/host/bin/meson --internal exe /buildroot/output/build/systemd-236/build/meson-private/meson_exe_sed_232a0623cc7ce2cd67ec72ed784b76307102ed76.dat > Warning: You are using 'ANSI_X3.4-1968' which is not a Unicode-compatible locale. > You might see errors if you use UTF-8 strings as filenames, as strings, or as file contents. > Please switch to a UTF-8 locale for your platform. > ... > UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 1079: ordinal not in range(128) > package/pkg-generic.mk:247: recipe for target '/buildroot/output/build/systemd-236/.stamp_built' failed > make: *** [/buildroot/output/build/systemd-236/.stamp_built] Error 1 > > This patch changes default host system locale from C to en_US.UTF-8 when > building systemd package to fix this issue. It also introduces > BR2_NEEDS_HOST_EN_US_UTF8_LOCALE flag that checks if this locale is available on > the host system. If locale is not available error message is show and build > process is stopped. > > Signed-off-by: Julius Kriukas <julius@kriukas.lt> > > --- > Changes v1 -> v2: > - Added BR2_NEEDS_HOST_EN_US_UTF8_LOCALE flag that is selected by > BR2_REPRODUCIBLE. > - Extended support/dependencies/dependencies.sh to check for > en_US.UTF-8 locale availability if BR2_NEEDS_HOST_EN_US_UTF8_LOCALE > is set. I've changed to use C.UTF-8 instead of en_US.UTF-8 and applied to master. Thanks! Thomas -- Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons) Embedded Linux and Kernel engineering https://bootlin.com ^ permalink raw reply [flat|nested] 19+ messages in thread
[parent not found: <87vaecbsf2.fsf@paral.in>]
* [Buildroot] [PATCH v4] systemd: switch to en_US.UTF-8 locale when building [not found] ` <87vaecbsf2.fsf@paral.in> @ 2018-03-04 9:01 ` Yann E. MORIN 0 siblings, 0 replies; 19+ messages in thread From: Yann E. MORIN @ 2018-03-04 9:01 UTC (permalink / raw) To: buildroot Christian, All, On 2018-03-03 19:22 -0500, Christian Stewart spake thusly: > Thomas Petazzoni <thomas.petazzoni@bootlin.com> writes: > > I've changed to use C.UTF-8 instead of en_US.UTF-8 and applied to > > master. Thanks! > > This has broken the build on Gentoo, unfortunately. Grr.. Gentoo... Grr... The major distros (Debian and derivatives, Fedora...) all have C.UTF-8. Gentoo is again playing solo... Sigh... :-/ I'm onto a fix for that, now... > Is it possible to compile with an en_US.UTF-8 or other language-specific > UTF-8 locale, such that the build again works on Gentoo? Not everyone on Earth is American, by far, so we already ruled out using a hard-coded en_US.UTF-8 locale; for example, I don't have it on my system (although I do have, and do use, an english locale, en_GB.UTF-8). I'm onto a fix now... Regards, Yann E. MORIN. -- .-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' ^ permalink raw reply [flat|nested] 19+ messages in thread
* [Buildroot] [PATCH v3] systemd: switch to en_US.UTF-8 locale when building 2018-02-12 21:34 ` [Buildroot] [PATCH v3] systemd: switch to en_US.UTF-8 locale when building Julius Kriukas 2018-02-17 18:06 ` John Keeping 2018-02-22 23:00 ` [Buildroot] [PATCH v4] " Julius Kriukas @ 2018-02-23 9:06 ` Arnout Vandecappelle 2 siblings, 0 replies; 19+ messages in thread From: Arnout Vandecappelle @ 2018-02-23 9:06 UTC (permalink / raw) To: buildroot Hi Julius, I wanted to take the time to think of better solutions, but since I don't find the time, here is a very quick review. On 12-02-18 22:34, Julius Kriukas wrote: > When BR2_REPRODUCIBLE is enabled or host uses non UTF-8 capable locale > building systemd fails with an error: > > [1/1080] Generating systemd.bg.catalog with a meson_exe.py custom command. > FAILED: catalog/systemd.bg.catalog > /buildroot/output/host/bin/python3 /buildroot/output/host/bin/meson --internal exe /buildroot/output/build/systemd-236/build/meson-private/meson_exe_sed_232a0623cc7ce2cd67ec72ed784b76307102ed76.dat > Warning: You are using 'ANSI_X3.4-1968' which is not a Unicode-compatible locale. > You might see errors if you use UTF-8 strings as filenames, as strings, or as file contents. > Please switch to a UTF-8 locale for your platform. > ... > UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 1079: ordinal not in range(128) As someone mentioned (IIRC), it's really not OK that some Python script (is it meson or ninja?) is relying on a locale being set to work correctly. So it would be much preferable to find out what the real problem is, and to have an upstreamable patch that fixes it. That said, getting to the bottom of this is work, someone has to do it, and in the mean time we've got a broken system, and we're about to release 2018.02. So I'm OK to accept this workaround for the time being (and as we all now, that can be a very long time :-). I would, however, like to mention explicitly in the commit message that this is just a workaround and that really one of the underlying scripts (either meson or ninja or something systemd-specific, TBD) is broken. > package/pkg-generic.mk:247: recipe for target '/buildroot/output/build/systemd-236/.stamp_built' failed > make: *** [/buildroot/output/build/systemd-236/.stamp_built] Error 1 > > This patch changes default host system locale from C to en_US.UTF-8 when > building systemd package to fix this issue. It also introduces > BR2_NEEDS_HOST_EN_US_UTF8_LOCALE flag that checks if this locale is available on > the host system. If locale is not available error message is show and build > process is stopped. The part that introduces BR2_NEEDS_HOST_EN_US_UTF8_LOCALE and uses it for BR2_REPRODUCIBLE should be a separate patch. Indeed, when we find a better solution for the systemd/meson issue, then we can revert this patch, but the BR2_REPRODUCIBLE bit should stay. Regards, Arnout > > Signed-off-by: Julius Kriukas <julius@kriukas.lt> -- Arnout Vandecappelle arnout at mind be Senior Embedded Software Architect +32-16-286500 Essensium/Mind http://www.mind.be G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF ^ permalink raw reply [flat|nested] 19+ messages in thread
end of thread, other threads:[~2018-03-04 9:01 UTC | newest]
Thread overview: 19+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-01-28 11:22 [Buildroot] [PATCH 1/1] Change host locale from C to en_US.UTF-8 for reproducable builds Julius Kriukas
2018-01-30 22:38 ` Thomas Petazzoni
2018-01-31 21:34 ` Julius Kriukas
2018-01-31 21:36 ` [Buildroot] [PATCH v2] " Julius Kriukas
2018-02-04 15:19 ` Yann E. MORIN
2018-02-06 16:32 ` Thomas Petazzoni
2018-02-12 21:34 ` [Buildroot] [PATCH v3] systemd: switch to en_US.UTF-8 locale when building Julius Kriukas
2018-02-17 18:06 ` John Keeping
2018-02-22 23:02 ` Julius Kriukas
2018-02-22 23:00 ` [Buildroot] [PATCH v4] " Julius Kriukas
2018-02-28 21:31 ` Yann E. MORIN
2018-02-28 22:13 ` Thomas Petazzoni
2018-03-01 18:06 ` Yann E. MORIN
2018-03-01 18:50 ` Peter Korsgaard
2018-03-01 19:12 ` Yann E. MORIN
2018-03-01 20:04 ` Peter Korsgaard
2018-03-02 21:45 ` Thomas Petazzoni
[not found] ` <87vaecbsf2.fsf@paral.in>
2018-03-04 9:01 ` Yann E. MORIN
2018-02-23 9:06 ` [Buildroot] [PATCH v3] " Arnout Vandecappelle
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.