All of lore.kernel.org
 help / color / mirror / Atom feed
* [Buildroot] [PATCH 1/1] Change host locale from C to en_US.UTF-8 for reproducable builds
@ 2018-01-28 11:22 Julius Kriukas
  2018-01-30 22:38 ` Thomas Petazzoni
  2018-01-31 21:36 ` [Buildroot] [PATCH v2] " Julius Kriukas
  0 siblings, 2 replies; 19+ messages in thread
From: Julius Kriukas @ 2018-01-28 11:22 UTC (permalink / raw)
  To: buildroot

When BR2_REPRODUCIBLE is enabled building systemd fails with error:

[1/1080] Generating systemd.bg.catalog with a meson_exe.py custom command.
FAILED: catalog/systemd.bg.catalog
/buildroot/output/host/bin/python3 /buildroot/output/host/bin/meson --internal exe /buildroot/output/build/systemd-236/build/meson-private/meson_exe_sed_232a0623cc7ce2cd67ec72ed784b76307102ed76.dat
Warning: You are using 'ANSI_X3.4-1968' which is not a Unicode-compatible locale.
You might see errors if you use UTF-8 strings as filenames, as strings, or as file contents.
Please switch to a UTF-8 locale for your platform.
...
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 1079: ordinal not in range(128)
package/pkg-generic.mk:247: recipe for target '/buildroot/output/build/systemd-236/.stamp_built' failed
make: *** [/buildroot/output/build/systemd-236/.stamp_built] Error 1

Other packages using meson build system might fail as well.

This patch changes default host system locale from C to en_US.UTF-8 to
fix this issue. Any Unicode-compatible locale would be suitable here.

Signed-off-by: Julius Kriukas <julius@kriukas.lt>
---
 Makefile | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/Makefile b/Makefile
index 7d8ab51a8c..5991c6a221 100644
--- a/Makefile
+++ b/Makefile
@@ -250,8 +250,8 @@ endif
 # timezone and locale may affect build output
 ifeq ($(BR2_REPRODUCIBLE),y)
 export TZ = UTC
-export LANG = C
-export LC_ALL = C
+export LANG = en_US.UTF-8
+export LC_ALL = en_US.UTF-8
 export GZIP = -n
 BR2_VERSION_GIT_EPOCH = $(shell GIT_DIR=$(TOPDIR)/.git $(GIT) log -1 --format=%at)
 export SOURCE_DATE_EPOCH ?= $(if $(wildcard $(TOPDIR)/.git),$(BR2_VERSION_GIT_EPOCH),$(BR2_VERSION_EPOCH))
-- 
2.15.1

^ permalink raw reply related	[flat|nested] 19+ messages in thread

* [Buildroot] [PATCH 1/1] Change host locale from C to en_US.UTF-8 for reproducable builds
  2018-01-28 11:22 [Buildroot] [PATCH 1/1] Change host locale from C to en_US.UTF-8 for reproducable builds Julius Kriukas
@ 2018-01-30 22:38 ` Thomas Petazzoni
  2018-01-31 21:34   ` Julius Kriukas
  2018-01-31 21:36 ` [Buildroot] [PATCH v2] " Julius Kriukas
  1 sibling, 1 reply; 19+ messages in thread
From: Thomas Petazzoni @ 2018-01-30 22:38 UTC (permalink / raw)
  To: buildroot

Hello,

On Sun, 28 Jan 2018 13:22:05 +0200, Julius Kriukas wrote:
> When BR2_REPRODUCIBLE is enabled building systemd fails with error:
> 
> [1/1080] Generating systemd.bg.catalog with a meson_exe.py custom command.
> FAILED: catalog/systemd.bg.catalog
> /buildroot/output/host/bin/python3 /buildroot/output/host/bin/meson --internal exe /buildroot/output/build/systemd-236/build/meson-private/meson_exe_sed_232a0623cc7ce2cd67ec72ed784b76307102ed76.dat
> Warning: You are using 'ANSI_X3.4-1968' which is not a Unicode-compatible locale.
> You might see errors if you use UTF-8 strings as filenames, as strings, or as file contents.
> Please switch to a UTF-8 locale for your platform.
> ...
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 1079: ordinal not in range(128)
> package/pkg-generic.mk:247: recipe for target '/buildroot/output/build/systemd-236/.stamp_built' failed
> make: *** [/buildroot/output/build/systemd-236/.stamp_built] Error 1
> 
> Other packages using meson build system might fail as well.
> 
> This patch changes default host system locale from C to en_US.UTF-8 to
> fix this issue. Any Unicode-compatible locale would be suitable here.
> 
> Signed-off-by: Julius Kriukas <julius@kriukas.lt>

Thanks for your patch! I was about to apply it, but one issue I see is
that we have no guarantee that en_US.UTF-8 is available.

support/dependencies/dependencies.sh checks that we have at least one
UTF-8 locale available if BR2_NEEDS_HOST_UTF8_LOCALE=y, but:

 - BR2_NEEDS_HOST_UTF8_LOCALE is only selected when a uClibc toolchain
   is being built

 - it only ensures one UTF-8 locale is available, not specifically
   en_US.UTF-8

Perhaps we need another BR2_NEEDS_HOST_EN_US_UTF8_LOCALE, which would
be used by support/dependencies/dependencies.sh to ensure that
en_US.UTF-8 is available. This new option would be selected by
BR2_REPRODUCIBLE.

Arnout, Peter, Yann? Thoughts?

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

^ permalink raw reply	[flat|nested] 19+ messages in thread

* [Buildroot] [PATCH 1/1] Change host locale from C to en_US.UTF-8 for reproducable builds
  2018-01-30 22:38 ` Thomas Petazzoni
@ 2018-01-31 21:34   ` Julius Kriukas
  0 siblings, 0 replies; 19+ messages in thread
From: Julius Kriukas @ 2018-01-31 21:34 UTC (permalink / raw)
  To: buildroot

On 01/31/2018 12:38 AM, Thomas Petazzoni wrote:
> Hello,
> 
> On Sun, 28 Jan 2018 13:22:05 +0200, Julius Kriukas wrote:
>> When BR2_REPRODUCIBLE is enabled building systemd fails with error:
>>
>> [1/1080] Generating systemd.bg.catalog with a meson_exe.py custom command.
>> FAILED: catalog/systemd.bg.catalog
>> /buildroot/output/host/bin/python3 /buildroot/output/host/bin/meson --internal exe /buildroot/output/build/systemd-236/build/meson-private/meson_exe_sed_232a0623cc7ce2cd67ec72ed784b76307102ed76.dat
>> Warning: You are using 'ANSI_X3.4-1968' which is not a Unicode-compatible locale.
>> You might see errors if you use UTF-8 strings as filenames, as strings, or as file contents.
>> Please switch to a UTF-8 locale for your platform.
>> ...
>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 1079: ordinal not in range(128)
>> package/pkg-generic.mk:247: recipe for target '/buildroot/output/build/systemd-236/.stamp_built' failed
>> make: *** [/buildroot/output/build/systemd-236/.stamp_built] Error 1
>>
>> Other packages using meson build system might fail as well.
>>
>> This patch changes default host system locale from C to en_US.UTF-8 to
>> fix this issue. Any Unicode-compatible locale would be suitable here.
>>
>> Signed-off-by: Julius Kriukas <julius@kriukas.lt>
> 
> Thanks for your patch! I was about to apply it, but one issue I see is
> that we have no guarantee that en_US.UTF-8 is available.
> 
> support/dependencies/dependencies.sh checks that we have at least one
> UTF-8 locale available if BR2_NEEDS_HOST_UTF8_LOCALE=y, but:
> 
>   - BR2_NEEDS_HOST_UTF8_LOCALE is only selected when a uClibc toolchain
>     is being built
> 
>   - it only ensures one UTF-8 locale is available, not specifically
>     en_US.UTF-8
> 
> Perhaps we need another BR2_NEEDS_HOST_EN_US_UTF8_LOCALE, which would
> be used by support/dependencies/dependencies.sh to ensure that
> en_US.UTF-8 is available. This new option would be selected by
> BR2_REPRODUCIBLE.
> 
> Arnout, Peter, Yann? Thoughts?
> 
> Best regards,
> 
> Thomas
> 

I think Debian uses C.UTF-8 locale in similar circumstances, but I 
suppose it is too much Debian specific (for example it seems C.UTF-8 
locale is not available by default on CentOS 7 systems).

I favor Thomas proposal to add en_US.UTF-8 availability detection with a 
new flag. It seemed overly complicated at first but it solves the issue 
and I do not have anything better to suggest.

-- 
Julius Kriukas

^ permalink raw reply	[flat|nested] 19+ messages in thread

* [Buildroot] [PATCH v2] Change host locale from C to en_US.UTF-8 for reproducable builds
  2018-01-28 11:22 [Buildroot] [PATCH 1/1] Change host locale from C to en_US.UTF-8 for reproducable builds Julius Kriukas
  2018-01-30 22:38 ` Thomas Petazzoni
@ 2018-01-31 21:36 ` Julius Kriukas
  2018-02-04 15:19   ` Yann E. MORIN
                     ` (2 more replies)
  1 sibling, 3 replies; 19+ messages in thread
From: Julius Kriukas @ 2018-01-31 21:36 UTC (permalink / raw)
  To: buildroot

When BR2_REPRODUCIBLE is enabled building systemd fails with error:

[1/1080] Generating systemd.bg.catalog with a meson_exe.py custom command.
FAILED: catalog/systemd.bg.catalog
/buildroot/output/host/bin/python3 /buildroot/output/host/bin/meson --internal exe /buildroot/output/build/systemd-236/build/meson-private/meson_exe_sed_232a0623cc7ce2cd67ec72ed784b76307102ed76.dat
Warning: You are using 'ANSI_X3.4-1968' which is not a Unicode-compatible locale.
You might see errors if you use UTF-8 strings as filenames, as strings, or as file contents.
Please switch to a UTF-8 locale for your platform.
...
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 1079: ordinal not in range(128)
package/pkg-generic.mk:247: recipe for target '/buildroot/output/build/systemd-236/.stamp_built' failed
make: *** [/buildroot/output/build/systemd-236/.stamp_built] Error 1

Other packages using meson build system might fail as well.

This patch changes default host system locale from C to en_US.UTF-8 to
fix this issue. It also introduces BR2_NEEDS_HOST_EN_US_UTF8_LOCALE
hidden flag that checks if this locale is available on the host system.
If locale is not available error message is show and build process is
stopped.

---
Changes v1 -> v2:
  - Added BR2_NEEDS_HOST_EN_US_UTF8_LOCALE flag that is selected by
    BR2_REPRODUCIBLE.
  - Extended support/dependencies/dependencies.sh to check for
    en_US.UTF-8 locale availability if BR2_NEEDS_HOST_EN_US_UTF8_LOCALE
    is set.

Signed-off-by: Julius Kriukas <julius@kriukas.lt>
---
 Config.in                            |  6 ++++++
 Makefile                             |  4 ++--
 support/dependencies/dependencies.sh | 13 +++++++++++++
 3 files changed, 21 insertions(+), 2 deletions(-)

diff --git a/Config.in b/Config.in
index 62d67ce957..bd14226edc 100644
--- a/Config.in
+++ b/Config.in
@@ -94,6 +94,11 @@ config BR2_HOSTARCH_NEEDS_IA32_COMPILER
 config BR2_NEEDS_HOST_UTF8_LOCALE
 	bool
 
+# Hidden boolean selected by packages that explicitly need en_US.UTF-8 locale 
+# to be available on host system.
+config BR2_NEEDS_HOST_EN_US_UTF8_LOCALE
+	bool
+
 source "arch/Config.in"
 
 menu "Build options"
@@ -715,6 +720,7 @@ config BR2_REPRODUCIBLE
 	bool "Make the build reproducible (experimental)"
 	# SOURCE_DATE_EPOCH support in toolchain-wrapper requires GCC 4.4
 	depends on BR2_TOOLCHAIN_GCC_AT_LEAST_4_4
+	select BR2_NEEDS_HOST_EN_US_UTF8_LOCALE
 	help
 	  This option will remove all sources of non-reproducibility
 	  from the build process. For a given Buildroot configuration,
diff --git a/Makefile b/Makefile
index 7d8ab51a8c..5991c6a221 100644
--- a/Makefile
+++ b/Makefile
@@ -250,8 +250,8 @@ endif
 # timezone and locale may affect build output
 ifeq ($(BR2_REPRODUCIBLE),y)
 export TZ = UTC
-export LANG = C
-export LC_ALL = C
+export LANG = en_US.UTF-8
+export LC_ALL = en_US.UTF-8
 export GZIP = -n
 BR2_VERSION_GIT_EPOCH = $(shell GIT_DIR=$(TOPDIR)/.git $(GIT) log -1 --format=%at)
 export SOURCE_DATE_EPOCH ?= $(if $(wildcard $(TOPDIR)/.git),$(BR2_VERSION_GIT_EPOCH),$(BR2_VERSION_EPOCH))
diff --git a/support/dependencies/dependencies.sh b/support/dependencies/dependencies.sh
index a195c62c8c..4a5b55e9da 100755
--- a/support/dependencies/dependencies.sh
+++ b/support/dependencies/dependencies.sh
@@ -207,6 +207,19 @@ if grep ^BR2_NEEDS_HOST_UTF8_LOCALE=y $BR2_CONFIG > /dev/null; then
 	fi
 fi
 
+if grep ^BR2_NEEDS_HOST_EN_US_UTF8_LOCALE=y $BR2_CONFIG > /dev/null; then
+	if ! which locale > /dev/null ; then
+		echo
+		echo "You need locale support on your build machine"
+		exit 1 ;
+	fi
+	if ! LC_ALL=en_US.UTF-8 locale -c charmap | grep '^UTF-8$' > /dev/null; then
+		echo
+		echo "You need en_US.UTF-8 locale suppport on the host system"
+		exit 1 ;
+	fi
+fi
+
 if grep -q ^BR2_NEEDS_HOST_JAVA=y $BR2_CONFIG ; then
 	check_prog_host "java"
 	JAVA_GCJ=$(java -version 2>&1 | grep gcj)
-- 
2.15.1

^ permalink raw reply related	[flat|nested] 19+ messages in thread

* [Buildroot] [PATCH v2] Change host locale from C to en_US.UTF-8 for reproducable builds
  2018-01-31 21:36 ` [Buildroot] [PATCH v2] " Julius Kriukas
@ 2018-02-04 15:19   ` Yann E. MORIN
  2018-02-06 16:32   ` Thomas Petazzoni
  2018-02-12 21:34   ` [Buildroot] [PATCH v3] systemd: switch to en_US.UTF-8 locale when building Julius Kriukas
  2 siblings, 0 replies; 19+ messages in thread
From: Yann E. MORIN @ 2018-02-04 15:19 UTC (permalink / raw)
  To: buildroot

Julius, All,

On 2018-01-31 23:36 +0200, Julius Kriukas spake thusly:
> When BR2_REPRODUCIBLE is enabled building systemd fails with error:
> 
> [1/1080] Generating systemd.bg.catalog with a meson_exe.py custom command.
> FAILED: catalog/systemd.bg.catalog
> /buildroot/output/host/bin/python3 /buildroot/output/host/bin/meson --internal exe /buildroot/output/build/systemd-236/build/meson-private/meson_exe_sed_232a0623cc7ce2cd67ec72ed784b76307102ed76.dat
> Warning: You are using 'ANSI_X3.4-1968' which is not a Unicode-compatible locale.
> You might see errors if you use UTF-8 strings as filenames, as strings, or as file contents.
> Please switch to a UTF-8 locale for your platform.
> ...
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 1079: ordinal not in range(128)
> package/pkg-generic.mk:247: recipe for target '/buildroot/output/build/systemd-236/.stamp_built' failed
> make: *** [/buildroot/output/build/systemd-236/.stamp_built] Error 1
> 
> Other packages using meson build system might fail as well.
> 
> This patch changes default host system locale from C to en_US.UTF-8 to
> fix this issue. It also introduces BR2_NEEDS_HOST_EN_US_UTF8_LOCALE
> hidden flag that checks if this locale is available on the host system.
> If locale is not available error message is show and build process is
> stopped.

Yeah, I like this change.

Yet, people that already use an UTF-8 locale (e.g fr_FR.UTF-8, or
en_GB.UTF-8 or whatever_else.UTF-8) will be very disapointed that their
build now fail when they were previously successful... :-/

But see below...

> diff --git a/Config.in b/Config.in
> index 62d67ce957..bd14226edc 100644
> --- a/Config.in
> +++ b/Config.in
> @@ -94,6 +94,11 @@ config BR2_HOSTARCH_NEEDS_IA32_COMPILER
>  config BR2_NEEDS_HOST_UTF8_LOCALE
>  	bool
>  
> +# Hidden boolean selected by packages that explicitly need en_US.UTF-8 locale 
> +# to be available on host system.
> +config BR2_NEEDS_HOST_EN_US_UTF8_LOCALE
> +	bool

So, this option is supposed to be selectable by packages, but it is only
ever selected by the reproducible option, and never used to directly set
the locale.

What I would have expected, too, would be that systemd also selects
BR2_NEEDS_HOST_EN_US_UTF8_LOCALE, and the build commands for systemd are
changed such as (simplified for brevity):

    SYSTEMD_ENV = $(TARGET_MAKE_ENV) LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8

    define SYSTEMD_CONFIGURE_CMDS
        $(SYSTEMD_ENV) meson $(SYSTEMD_CONF_OPTS) $(@D) $(@D)/build
    endef

    define SYSTEMD_BUILD_CMDS
        $(SYSTEMD_ENV) ninja $(SYSTEMD_NINJA_OPTS) -C $(@D)/build
    endef

... and so on...

>  source "arch/Config.in"
>  
>  menu "Build options"
> @@ -715,6 +720,7 @@ config BR2_REPRODUCIBLE
>  	bool "Make the build reproducible (experimental)"
>  	# SOURCE_DATE_EPOCH support in toolchain-wrapper requires GCC 4.4
>  	depends on BR2_TOOLCHAIN_GCC_AT_LEAST_4_4
> +	select BR2_NEEDS_HOST_EN_US_UTF8_LOCALE

If it were me, this option would ultimately disappear, and we would only
do reproducible builds. There is no point in hell to generate
non-reproducible builds... :-/

(Yeah, I know, not everyone is at ease reading tools' output messages
not in their language.)

Regards,
Yann E. MORIN.

>  	help
>  	  This option will remove all sources of non-reproducibility
>  	  from the build process. For a given Buildroot configuration,
> diff --git a/Makefile b/Makefile
> index 7d8ab51a8c..5991c6a221 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -250,8 +250,8 @@ endif
>  # timezone and locale may affect build output
>  ifeq ($(BR2_REPRODUCIBLE),y)
>  export TZ = UTC
> -export LANG = C
> -export LC_ALL = C
> +export LANG = en_US.UTF-8
> +export LC_ALL = en_US.UTF-8
>  export GZIP = -n
>  BR2_VERSION_GIT_EPOCH = $(shell GIT_DIR=$(TOPDIR)/.git $(GIT) log -1 --format=%at)
>  export SOURCE_DATE_EPOCH ?= $(if $(wildcard $(TOPDIR)/.git),$(BR2_VERSION_GIT_EPOCH),$(BR2_VERSION_EPOCH))
> diff --git a/support/dependencies/dependencies.sh b/support/dependencies/dependencies.sh
> index a195c62c8c..4a5b55e9da 100755
> --- a/support/dependencies/dependencies.sh
> +++ b/support/dependencies/dependencies.sh
> @@ -207,6 +207,19 @@ if grep ^BR2_NEEDS_HOST_UTF8_LOCALE=y $BR2_CONFIG > /dev/null; then
>  	fi
>  fi
>  
> +if grep ^BR2_NEEDS_HOST_EN_US_UTF8_LOCALE=y $BR2_CONFIG > /dev/null; then
> +	if ! which locale > /dev/null ; then
> +		echo
> +		echo "You need locale support on your build machine"
> +		exit 1 ;
> +	fi
> +	if ! LC_ALL=en_US.UTF-8 locale -c charmap | grep '^UTF-8$' > /dev/null; then
> +		echo
> +		echo "You need en_US.UTF-8 locale suppport on the host system"
> +		exit 1 ;
> +	fi
> +fi
> +
>  if grep -q ^BR2_NEEDS_HOST_JAVA=y $BR2_CONFIG ; then
>  	check_prog_host "java"
>  	JAVA_GCJ=$(java -version 2>&1 | grep gcj)
> -- 
> 2.15.1
> 

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

^ permalink raw reply	[flat|nested] 19+ messages in thread

* [Buildroot] [PATCH v2] Change host locale from C to en_US.UTF-8 for reproducable builds
  2018-01-31 21:36 ` [Buildroot] [PATCH v2] " Julius Kriukas
  2018-02-04 15:19   ` Yann E. MORIN
@ 2018-02-06 16:32   ` Thomas Petazzoni
  2018-02-12 21:34   ` [Buildroot] [PATCH v3] systemd: switch to en_US.UTF-8 locale when building Julius Kriukas
  2 siblings, 0 replies; 19+ messages in thread
From: Thomas Petazzoni @ 2018-02-06 16:32 UTC (permalink / raw)
  To: buildroot

Hello,

On Wed, 31 Jan 2018 23:36:29 +0200, Julius Kriukas wrote:
> When BR2_REPRODUCIBLE is enabled building systemd fails with error:
> 
> [1/1080] Generating systemd.bg.catalog with a meson_exe.py custom command.
> FAILED: catalog/systemd.bg.catalog
> /buildroot/output/host/bin/python3 /buildroot/output/host/bin/meson --internal exe /buildroot/output/build/systemd-236/build/meson-private/meson_exe_sed_232a0623cc7ce2cd67ec72ed784b76307102ed76.dat
> Warning: You are using 'ANSI_X3.4-1968' which is not a Unicode-compatible locale.
> You might see errors if you use UTF-8 strings as filenames, as strings, or as file contents.
> Please switch to a UTF-8 locale for your platform.
> ...
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 1079: ordinal not in range(128)
> package/pkg-generic.mk:247: recipe for target '/buildroot/output/build/systemd-236/.stamp_built' failed
> make: *** [/buildroot/output/build/systemd-236/.stamp_built] Error 1
> 
> Other packages using meson build system might fail as well.
> 
> This patch changes default host system locale from C to en_US.UTF-8 to
> fix this issue. It also introduces BR2_NEEDS_HOST_EN_US_UTF8_LOCALE
> hidden flag that checks if this locale is available on the host system.
> If locale is not available error message is show and build process is
> stopped.
> 
> ---
> Changes v1 -> v2:
>   - Added BR2_NEEDS_HOST_EN_US_UTF8_LOCALE flag that is selected by
>     BR2_REPRODUCIBLE.
>   - Extended support/dependencies/dependencies.sh to check for
>     en_US.UTF-8 locale availability if BR2_NEEDS_HOST_EN_US_UTF8_LOCALE
>     is set.
> 
> Signed-off-by: Julius Kriukas <julius@kriukas.lt>

Your SoB line should be before the "---" that separates the commit log
and the changelog.

However, I have a bigger issue with the patch: you're fixing the issue
for BR2_REPRODUCIBLE, but the problem actually appears also with a
non-BR2_REPRODUCIBLE build. Look at:

  http://autobuild.buildroot.net/results/257/25757c41d1a83edbe2a7c47b63ce2e43eb6141dc/build-end.log

This configuration does *not* have BR2_REPRODUCIBLE enabled. The system
on which it runs has several UTF-8 locales enabled:

$ locale -a
C
C.UTF-8
POSIX
en_US.utf8
fr_FR.utf8

But the POSIX locale is used:

$ locale
LANG=
LANGUAGE=
LC_CTYPE="POSIX"
LC_NUMERIC="POSIX"
LC_TIME="POSIX"
LC_COLLATE="POSIX"
LC_MONETARY="POSIX"
LC_MESSAGES="POSIX"
LC_PAPER="POSIX"
LC_NAME="POSIX"
LC_ADDRESS="POSIX"
LC_TELEPHONE="POSIX"
LC_MEASUREMENT="POSIX"
LC_IDENTIFICATION="POSIX"
LC_ALL=

And Meson fails in this situation. I believe it should be fixed in
Meson. We should not have to set a specific locale for the thing to
build. The need for the UTF-8 locale to be present is more or less OK,
but the need to have such a locale in *use* is annoying.

So perhaps your fix is needed because Meson will in the end really need
at least one UTF-8 locale available, but it is not complete because it
doesn't fix the non-BR2_REPRODUCIBLE situation.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
http://bootlin.com

^ permalink raw reply	[flat|nested] 19+ messages in thread

* [Buildroot] [PATCH v3] systemd: switch to en_US.UTF-8 locale when building
  2018-01-31 21:36 ` [Buildroot] [PATCH v2] " Julius Kriukas
  2018-02-04 15:19   ` Yann E. MORIN
  2018-02-06 16:32   ` Thomas Petazzoni
@ 2018-02-12 21:34   ` Julius Kriukas
  2018-02-17 18:06     ` John Keeping
                       ` (2 more replies)
  2 siblings, 3 replies; 19+ messages in thread
From: Julius Kriukas @ 2018-02-12 21:34 UTC (permalink / raw)
  To: buildroot

When BR2_REPRODUCIBLE is enabled or host uses non UTF-8 capable locale
building systemd fails with an error:

[1/1080] Generating systemd.bg.catalog with a meson_exe.py custom command.
FAILED: catalog/systemd.bg.catalog
/buildroot/output/host/bin/python3 /buildroot/output/host/bin/meson --internal exe /buildroot/output/build/systemd-236/build/meson-private/meson_exe_sed_232a0623cc7ce2cd67ec72ed784b76307102ed76.dat
Warning: You are using 'ANSI_X3.4-1968' which is not a Unicode-compatible locale.
You might see errors if you use UTF-8 strings as filenames, as strings, or as file contents.
Please switch to a UTF-8 locale for your platform.
...
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 1079: ordinal not in range(128)
package/pkg-generic.mk:247: recipe for target '/buildroot/output/build/systemd-236/.stamp_built' failed
make: *** [/buildroot/output/build/systemd-236/.stamp_built] Error 1

This patch changes default host system locale from C to en_US.UTF-8 when
building systemd package to fix this issue. It also introduces
BR2_NEEDS_HOST_EN_US_UTF8_LOCALE flag that checks if this locale is available on
the host system. If locale is not available error message is show and build
process is stopped.

Signed-off-by: Julius Kriukas <julius@kriukas.lt>

---
Changes v1 -> v2:
  - Added BR2_NEEDS_HOST_EN_US_UTF8_LOCALE flag that is selected by
    BR2_REPRODUCIBLE.
  - Extended support/dependencies/dependencies.sh to check for
    en_US.UTF-8 locale availability if BR2_NEEDS_HOST_EN_US_UTF8_LOCALE
    is set.

Changes v2 -> v3:
  - Always select BR2_NEEDS_HOST_EN_US_UTF8_LOCALE by systemd package.
  - Do not select BR2_NEEDS_HOST_EN_US_UTF8_LOCALE by BR2_REPRODUCIBLE.
  - Keep global locale set to C if BR2_REPRODUCIBLE=y.
  - Use 'grep -q' instead of '> /dev/null' in dependencies.sh script.
---
 Config.in                            |  5 +++++
 package/systemd/Config.in            |  1 +
 package/systemd/systemd.mk           | 10 ++++++----
 support/dependencies/dependencies.sh | 13 +++++++++++++
 4 files changed, 25 insertions(+), 4 deletions(-)

diff --git a/Config.in b/Config.in
index 0002df5176..8d471dfb2e 100644
--- a/Config.in
+++ b/Config.in
@@ -94,6 +94,11 @@ config BR2_HOSTARCH_NEEDS_IA32_COMPILER
 config BR2_NEEDS_HOST_UTF8_LOCALE
 	bool
 
+# Hidden boolean selected by packages that explicitly need en_US.UTF-8 locale
+# to be available on host system.
+config BR2_NEEDS_HOST_EN_US_UTF8_LOCALE
+	bool
+
 source "arch/Config.in"
 
 menu "Build options"
diff --git a/package/systemd/Config.in b/package/systemd/Config.in
index 661f40dee4..742b3f1f1d 100644
--- a/package/systemd/Config.in
+++ b/package/systemd/Config.in
@@ -27,6 +27,7 @@ menuconfig BR2_PACKAGE_SYSTEMD
 	select BR2_PACKAGE_BUSYBOX_SHOW_OTHERS # kmod-tools
 	select BR2_PACKAGE_KMOD_TOOLS
 	select BR2_TARGET_TZ_INFO
+        select BR2_NEEDS_HOST_EN_US_UTF8_LOCALE
 	help
 	  systemd is a system and service manager for Linux,
 	  compatible with SysV and LSB init scripts. systemd provides
diff --git a/package/systemd/systemd.mk b/package/systemd/systemd.mk
index 1c31ebb34c..58798f32df 100644
--- a/package/systemd/systemd.mk
+++ b/package/systemd/systemd.mk
@@ -385,23 +385,25 @@ endef
 
 SYSTEMD_NINJA_OPTS = $(if $(VERBOSE),-v) -j$(PARALLEL_JOBS)
 
+SYSTEMD_ENV = $(TARGET_MAKE_ENV) LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8
+
 define SYSTEMD_CONFIGURE_CMDS
 	rm -rf $(@D)/build
 	mkdir -p $(@D)/build
-	$(TARGET_MAKE_ENV) meson $(SYSTEMD_CONF_OPTS) $(@D) $(@D)/build
+	$(SYSTEMD_ENV) meson $(SYSTEMD_CONF_OPTS) $(@D) $(@D)/build
 endef
 
 define SYSTEMD_BUILD_CMDS
-	$(TARGET_MAKE_ENV) ninja $(SYSTEMD_NINJA_OPTS) -C $(@D)/build
+	$(SYSTEMD_ENV) ninja $(SYSTEMD_NINJA_OPTS) -C $(@D)/build
 endef
 
 define SYSTEMD_INSTALL_TARGET_CMDS
-	$(TARGET_MAKE_ENV) DESTDIR=$(TARGET_DIR) ninja $(SYSTEMD_NINJA_OPTS) \
+	$(SYSTEMD_ENV) DESTDIR=$(TARGET_DIR) ninja $(SYSTEMD_NINJA_OPTS) \
 		-C $(@D)/build install
 endef
 
 define SYSTEMD_INSTALL_STAGING_CMDS
-	$(TARGET_MAKE_ENV) DESTDIR=$(STAGING_DIR) ninja $(SYSTEMD_NINJA_OPTS) \
+	$(SYSTEMD_ENV) DESTDIR=$(STAGING_DIR) ninja $(SYSTEMD_NINJA_OPTS) \
 		-C $(@D)/build install
 endef
 
diff --git a/support/dependencies/dependencies.sh b/support/dependencies/dependencies.sh
index 1804e85508..881f07008c 100755
--- a/support/dependencies/dependencies.sh
+++ b/support/dependencies/dependencies.sh
@@ -207,6 +207,19 @@ if grep ^BR2_NEEDS_HOST_UTF8_LOCALE=y $BR2_CONFIG > /dev/null; then
 	fi
 fi
 
+if grep -q ^BR2_NEEDS_HOST_EN_US_UTF8_LOCALE=y $BR2_CONFIG; then
+	if ! which locale > /dev/null ; then
+		echo
+		echo "You need locale support on your build machine"
+		exit 1 ;
+	fi
+	if ! LC_ALL=en_US.UTF-8 locale -c charmap | grep -q '^UTF-8$'; then
+		echo
+		echo "You need en_US.UTF-8 locale suppport on the host system"
+		exit 1 ;
+	fi
+fi
+
 if grep -q ^BR2_NEEDS_HOST_JAVA=y $BR2_CONFIG ; then
 	check_prog_host "java"
 	JAVA_GCJ=$(java -version 2>&1 | grep gcj)
-- 
2.15.1

^ permalink raw reply related	[flat|nested] 19+ messages in thread

* [Buildroot] [PATCH v3] systemd: switch to en_US.UTF-8 locale when building
  2018-02-12 21:34   ` [Buildroot] [PATCH v3] systemd: switch to en_US.UTF-8 locale when building Julius Kriukas
@ 2018-02-17 18:06     ` John Keeping
  2018-02-22 23:02       ` Julius Kriukas
  2018-02-22 23:00     ` [Buildroot] [PATCH v4] " Julius Kriukas
  2018-02-23  9:06     ` [Buildroot] [PATCH v3] " Arnout Vandecappelle
  2 siblings, 1 reply; 19+ messages in thread
From: John Keeping @ 2018-02-17 18:06 UTC (permalink / raw)
  To: buildroot

On Mon, 12 Feb 2018 23:34:00 +0200, Julius Kriukas wrote:

> When BR2_REPRODUCIBLE is enabled or host uses non UTF-8 capable locale
> building systemd fails with an error:
> 
> [1/1080] Generating systemd.bg.catalog with a meson_exe.py custom command.
> FAILED: catalog/systemd.bg.catalog
> /buildroot/output/host/bin/python3 /buildroot/output/host/bin/meson --internal exe /buildroot/output/build/systemd-236/build/meson-private/meson_exe_sed_232a0623cc7ce2cd67ec72ed784b76307102ed76.dat
> Warning: You are using 'ANSI_X3.4-1968' which is not a Unicode-compatible locale.
> You might see errors if you use UTF-8 strings as filenames, as strings, or as file contents.
> Please switch to a UTF-8 locale for your platform.
> ...
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 1079: ordinal not in range(128)
> package/pkg-generic.mk:247: recipe for target '/buildroot/output/build/systemd-236/.stamp_built' failed
> make: *** [/buildroot/output/build/systemd-236/.stamp_built] Error 1
> 
> This patch changes default host system locale from C to en_US.UTF-8 when
> building systemd package to fix this issue. It also introduces
> BR2_NEEDS_HOST_EN_US_UTF8_LOCALE flag that checks if this locale is available on
> the host system. If locale is not available error message is show and build
> process is stopped.
> 
> Signed-off-by: Julius Kriukas <julius@kriukas.lt>

Tested-by: John Keeping <john@metanate.com>

And, FWIW, with the indentation in Config.in fixed,

Reviewed-by: John Keeping <john@metanate.com>

> ---
> Changes v1 -> v2:
>   - Added BR2_NEEDS_HOST_EN_US_UTF8_LOCALE flag that is selected by
>     BR2_REPRODUCIBLE.
>   - Extended support/dependencies/dependencies.sh to check for
>     en_US.UTF-8 locale availability if BR2_NEEDS_HOST_EN_US_UTF8_LOCALE
>     is set.
> 
> Changes v2 -> v3:
>   - Always select BR2_NEEDS_HOST_EN_US_UTF8_LOCALE by systemd package.
>   - Do not select BR2_NEEDS_HOST_EN_US_UTF8_LOCALE by BR2_REPRODUCIBLE.
>   - Keep global locale set to C if BR2_REPRODUCIBLE=y.
>   - Use 'grep -q' instead of '> /dev/null' in dependencies.sh script.
> ---
>  Config.in                            |  5 +++++
>  package/systemd/Config.in            |  1 +
>  package/systemd/systemd.mk           | 10 ++++++----
>  support/dependencies/dependencies.sh | 13 +++++++++++++
>  4 files changed, 25 insertions(+), 4 deletions(-)
> 
> diff --git a/Config.in b/Config.in
> index 0002df5176..8d471dfb2e 100644
> --- a/Config.in
> +++ b/Config.in
> @@ -94,6 +94,11 @@ config BR2_HOSTARCH_NEEDS_IA32_COMPILER
>  config BR2_NEEDS_HOST_UTF8_LOCALE
>  	bool
>  
> +# Hidden boolean selected by packages that explicitly need en_US.UTF-8 locale
> +# to be available on host system.
> +config BR2_NEEDS_HOST_EN_US_UTF8_LOCALE
> +	bool
> +
>  source "arch/Config.in"
>  
>  menu "Build options"
> diff --git a/package/systemd/Config.in b/package/systemd/Config.in
> index 661f40dee4..742b3f1f1d 100644
> --- a/package/systemd/Config.in
> +++ b/package/systemd/Config.in
> @@ -27,6 +27,7 @@ menuconfig BR2_PACKAGE_SYSTEMD
>  	select BR2_PACKAGE_BUSYBOX_SHOW_OTHERS # kmod-tools
>  	select BR2_PACKAGE_KMOD_TOOLS
>  	select BR2_TARGET_TZ_INFO
> +        select BR2_NEEDS_HOST_EN_US_UTF8_LOCALE

This should use TAB for indentation.

>  	help
>  	  systemd is a system and service manager for Linux,
>  	  compatible with SysV and LSB init scripts. systemd provides
> diff --git a/package/systemd/systemd.mk b/package/systemd/systemd.mk
> index 1c31ebb34c..58798f32df 100644
> --- a/package/systemd/systemd.mk
> +++ b/package/systemd/systemd.mk
> @@ -385,23 +385,25 @@ endef
>  
>  SYSTEMD_NINJA_OPTS = $(if $(VERBOSE),-v) -j$(PARALLEL_JOBS)
>  
> +SYSTEMD_ENV = $(TARGET_MAKE_ENV) LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8
> +
>  define SYSTEMD_CONFIGURE_CMDS
>  	rm -rf $(@D)/build
>  	mkdir -p $(@D)/build
> -	$(TARGET_MAKE_ENV) meson $(SYSTEMD_CONF_OPTS) $(@D) $(@D)/build
> +	$(SYSTEMD_ENV) meson $(SYSTEMD_CONF_OPTS) $(@D) $(@D)/build
>  endef
>  
>  define SYSTEMD_BUILD_CMDS
> -	$(TARGET_MAKE_ENV) ninja $(SYSTEMD_NINJA_OPTS) -C $(@D)/build
> +	$(SYSTEMD_ENV) ninja $(SYSTEMD_NINJA_OPTS) -C $(@D)/build
>  endef
>  
>  define SYSTEMD_INSTALL_TARGET_CMDS
> -	$(TARGET_MAKE_ENV) DESTDIR=$(TARGET_DIR) ninja $(SYSTEMD_NINJA_OPTS) \
> +	$(SYSTEMD_ENV) DESTDIR=$(TARGET_DIR) ninja $(SYSTEMD_NINJA_OPTS) \
>  		-C $(@D)/build install
>  endef
>  
>  define SYSTEMD_INSTALL_STAGING_CMDS
> -	$(TARGET_MAKE_ENV) DESTDIR=$(STAGING_DIR) ninja $(SYSTEMD_NINJA_OPTS) \
> +	$(SYSTEMD_ENV) DESTDIR=$(STAGING_DIR) ninja $(SYSTEMD_NINJA_OPTS) \
>  		-C $(@D)/build install
>  endef
>  
> diff --git a/support/dependencies/dependencies.sh b/support/dependencies/dependencies.sh
> index 1804e85508..881f07008c 100755
> --- a/support/dependencies/dependencies.sh
> +++ b/support/dependencies/dependencies.sh
> @@ -207,6 +207,19 @@ if grep ^BR2_NEEDS_HOST_UTF8_LOCALE=y $BR2_CONFIG > /dev/null; then
>  	fi
>  fi
>  
> +if grep -q ^BR2_NEEDS_HOST_EN_US_UTF8_LOCALE=y $BR2_CONFIG; then
> +	if ! which locale > /dev/null ; then
> +		echo
> +		echo "You need locale support on your build machine"
> +		exit 1 ;
> +	fi
> +	if ! LC_ALL=en_US.UTF-8 locale -c charmap | grep -q '^UTF-8$'; then
> +		echo
> +		echo "You need en_US.UTF-8 locale suppport on the host system"
> +		exit 1 ;
> +	fi
> +fi
> +
>  if grep -q ^BR2_NEEDS_HOST_JAVA=y $BR2_CONFIG ; then
>  	check_prog_host "java"
>  	JAVA_GCJ=$(java -version 2>&1 | grep gcj)

^ permalink raw reply	[flat|nested] 19+ messages in thread

* [Buildroot] [PATCH v4] systemd: switch to en_US.UTF-8 locale when building
  2018-02-12 21:34   ` [Buildroot] [PATCH v3] systemd: switch to en_US.UTF-8 locale when building Julius Kriukas
  2018-02-17 18:06     ` John Keeping
@ 2018-02-22 23:00     ` Julius Kriukas
  2018-02-28 21:31       ` Yann E. MORIN
  2018-03-02 21:45       ` Thomas Petazzoni
  2018-02-23  9:06     ` [Buildroot] [PATCH v3] " Arnout Vandecappelle
  2 siblings, 2 replies; 19+ messages in thread
From: Julius Kriukas @ 2018-02-22 23:00 UTC (permalink / raw)
  To: buildroot

When BR2_REPRODUCIBLE is enabled or host uses non UTF-8 capable locale
building systemd fails with an error:

[1/1080] Generating systemd.bg.catalog with a meson_exe.py custom command.
FAILED: catalog/systemd.bg.catalog
/buildroot/output/host/bin/python3 /buildroot/output/host/bin/meson --internal exe /buildroot/output/build/systemd-236/build/meson-private/meson_exe_sed_232a0623cc7ce2cd67ec72ed784b76307102ed76.dat
Warning: You are using 'ANSI_X3.4-1968' which is not a Unicode-compatible locale.
You might see errors if you use UTF-8 strings as filenames, as strings, or as file contents.
Please switch to a UTF-8 locale for your platform.
...
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 1079: ordinal not in range(128)
package/pkg-generic.mk:247: recipe for target '/buildroot/output/build/systemd-236/.stamp_built' failed
make: *** [/buildroot/output/build/systemd-236/.stamp_built] Error 1

This patch changes default host system locale from C to en_US.UTF-8 when
building systemd package to fix this issue. It also introduces
BR2_NEEDS_HOST_EN_US_UTF8_LOCALE flag that checks if this locale is available on
the host system. If locale is not available error message is show and build
process is stopped.

Signed-off-by: Julius Kriukas <julius@kriukas.lt>

---
Changes v1 -> v2:
  - Added BR2_NEEDS_HOST_EN_US_UTF8_LOCALE flag that is selected by
    BR2_REPRODUCIBLE.
  - Extended support/dependencies/dependencies.sh to check for
    en_US.UTF-8 locale availability if BR2_NEEDS_HOST_EN_US_UTF8_LOCALE
    is set.

Changes v2 -> v3:
  - Always select BR2_NEEDS_HOST_EN_US_UTF8_LOCALE by systemd package.
  - Do not select BR2_NEEDS_HOST_EN_US_UTF8_LOCALE by BR2_REPRODUCIBLE.
  - Keep global locale set to C if BR2_REPRODUCIBLE=y.
  - Use 'grep -q' instead of '> /dev/null' in dependencies.sh script.

Changes v3 -> v4:
  - Fix indentation in package/systemd/Config.in, spaces -> tab.
---
 Config.in                            |  5 +++++
 package/systemd/Config.in            |  1 +
 package/systemd/systemd.mk           | 10 ++++++----
 support/dependencies/dependencies.sh | 13 +++++++++++++
 4 files changed, 25 insertions(+), 4 deletions(-)

diff --git a/Config.in b/Config.in
index 0002df5176..8d471dfb2e 100644
--- a/Config.in
+++ b/Config.in
@@ -94,6 +94,11 @@ config BR2_HOSTARCH_NEEDS_IA32_COMPILER
 config BR2_NEEDS_HOST_UTF8_LOCALE
 	bool
 
+# Hidden boolean selected by packages that explicitly need en_US.UTF-8 locale
+# to be available on host system.
+config BR2_NEEDS_HOST_EN_US_UTF8_LOCALE
+	bool
+
 source "arch/Config.in"
 
 menu "Build options"
diff --git a/package/systemd/Config.in b/package/systemd/Config.in
index 661f40dee4..e83f75c733 100644
--- a/package/systemd/Config.in
+++ b/package/systemd/Config.in
@@ -27,6 +27,7 @@ menuconfig BR2_PACKAGE_SYSTEMD
 	select BR2_PACKAGE_BUSYBOX_SHOW_OTHERS # kmod-tools
 	select BR2_PACKAGE_KMOD_TOOLS
 	select BR2_TARGET_TZ_INFO
+	select BR2_NEEDS_HOST_EN_US_UTF8_LOCALE
 	help
 	  systemd is a system and service manager for Linux,
 	  compatible with SysV and LSB init scripts. systemd provides
diff --git a/package/systemd/systemd.mk b/package/systemd/systemd.mk
index 126727d602..9a6a9cfcbc 100644
--- a/package/systemd/systemd.mk
+++ b/package/systemd/systemd.mk
@@ -385,23 +385,25 @@ endef
 
 SYSTEMD_NINJA_OPTS = $(if $(VERBOSE),-v) -j$(PARALLEL_JOBS)
 
+SYSTEMD_ENV = $(TARGET_MAKE_ENV) LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8
+
 define SYSTEMD_CONFIGURE_CMDS
 	rm -rf $(@D)/build
 	mkdir -p $(@D)/build
-	$(TARGET_MAKE_ENV) meson $(SYSTEMD_CONF_OPTS) $(@D) $(@D)/build
+	$(SYSTEMD_ENV) meson $(SYSTEMD_CONF_OPTS) $(@D) $(@D)/build
 endef
 
 define SYSTEMD_BUILD_CMDS
-	$(TARGET_MAKE_ENV) ninja $(SYSTEMD_NINJA_OPTS) -C $(@D)/build
+	$(SYSTEMD_ENV) ninja $(SYSTEMD_NINJA_OPTS) -C $(@D)/build
 endef
 
 define SYSTEMD_INSTALL_TARGET_CMDS
-	$(TARGET_MAKE_ENV) DESTDIR=$(TARGET_DIR) ninja $(SYSTEMD_NINJA_OPTS) \
+	$(SYSTEMD_ENV) DESTDIR=$(TARGET_DIR) ninja $(SYSTEMD_NINJA_OPTS) \
 		-C $(@D)/build install
 endef
 
 define SYSTEMD_INSTALL_STAGING_CMDS
-	$(TARGET_MAKE_ENV) DESTDIR=$(STAGING_DIR) ninja $(SYSTEMD_NINJA_OPTS) \
+	$(SYSTEMD_ENV) DESTDIR=$(STAGING_DIR) ninja $(SYSTEMD_NINJA_OPTS) \
 		-C $(@D)/build install
 endef
 
diff --git a/support/dependencies/dependencies.sh b/support/dependencies/dependencies.sh
index 1804e85508..881f07008c 100755
--- a/support/dependencies/dependencies.sh
+++ b/support/dependencies/dependencies.sh
@@ -207,6 +207,19 @@ if grep ^BR2_NEEDS_HOST_UTF8_LOCALE=y $BR2_CONFIG > /dev/null; then
 	fi
 fi
 
+if grep -q ^BR2_NEEDS_HOST_EN_US_UTF8_LOCALE=y $BR2_CONFIG; then
+	if ! which locale > /dev/null ; then
+		echo
+		echo "You need locale support on your build machine"
+		exit 1 ;
+	fi
+	if ! LC_ALL=en_US.UTF-8 locale -c charmap | grep -q '^UTF-8$'; then
+		echo
+		echo "You need en_US.UTF-8 locale suppport on the host system"
+		exit 1 ;
+	fi
+fi
+
 if grep -q ^BR2_NEEDS_HOST_JAVA=y $BR2_CONFIG ; then
 	check_prog_host "java"
 	JAVA_GCJ=$(java -version 2>&1 | grep gcj)
-- 
2.16.1

^ permalink raw reply related	[flat|nested] 19+ messages in thread

* [Buildroot] [PATCH v3] systemd: switch to en_US.UTF-8 locale when building
  2018-02-17 18:06     ` John Keeping
@ 2018-02-22 23:02       ` Julius Kriukas
  0 siblings, 0 replies; 19+ messages in thread
From: Julius Kriukas @ 2018-02-22 23:02 UTC (permalink / raw)
  To: buildroot

On 02/17/2018 08:06 PM, John Keeping wrote:
> On Mon, 12 Feb 2018 23:34:00 +0200, Julius Kriukas wrote:
> 
>> When BR2_REPRODUCIBLE is enabled or host uses non UTF-8 capable locale
>> building systemd fails with an error:
>>
>> [1/1080] Generating systemd.bg.catalog with a meson_exe.py custom command.
>> FAILED: catalog/systemd.bg.catalog
>> /buildroot/output/host/bin/python3 /buildroot/output/host/bin/meson --internal exe /buildroot/output/build/systemd-236/build/meson-private/meson_exe_sed_232a0623cc7ce2cd67ec72ed784b76307102ed76.dat
>> Warning: You are using 'ANSI_X3.4-1968' which is not a Unicode-compatible locale.
>> You might see errors if you use UTF-8 strings as filenames, as strings, or as file contents.
>> Please switch to a UTF-8 locale for your platform.
>> ...
>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 1079: ordinal not in range(128)
>> package/pkg-generic.mk:247: recipe for target '/buildroot/output/build/systemd-236/.stamp_built' failed
>> make: *** [/buildroot/output/build/systemd-236/.stamp_built] Error 1
>>
>> This patch changes default host system locale from C to en_US.UTF-8 when
>> building systemd package to fix this issue. It also introduces
>> BR2_NEEDS_HOST_EN_US_UTF8_LOCALE flag that checks if this locale is available on
>> the host system. If locale is not available error message is show and build
>> process is stopped.
>>
>> Signed-off-by: Julius Kriukas <julius@kriukas.lt>
> 
> Tested-by: John Keeping <john@metanate.com>
> 
> And, FWIW, with the indentation in Config.in fixed,
> 
> Reviewed-by: John Keeping <john@metanate.com>
> 
>> ---
>> Changes v1 -> v2:
>>    - Added BR2_NEEDS_HOST_EN_US_UTF8_LOCALE flag that is selected by
>>      BR2_REPRODUCIBLE.
>>    - Extended support/dependencies/dependencies.sh to check for
>>      en_US.UTF-8 locale availability if BR2_NEEDS_HOST_EN_US_UTF8_LOCALE
>>      is set.
>>
>> Changes v2 -> v3:
>>    - Always select BR2_NEEDS_HOST_EN_US_UTF8_LOCALE by systemd package.
>>    - Do not select BR2_NEEDS_HOST_EN_US_UTF8_LOCALE by BR2_REPRODUCIBLE.
>>    - Keep global locale set to C if BR2_REPRODUCIBLE=y.
>>    - Use 'grep -q' instead of '> /dev/null' in dependencies.sh script.
>> ---
>>   Config.in                            |  5 +++++
>>   package/systemd/Config.in            |  1 +
>>   package/systemd/systemd.mk           | 10 ++++++----
>>   support/dependencies/dependencies.sh | 13 +++++++++++++
>>   4 files changed, 25 insertions(+), 4 deletions(-)
>>
>> diff --git a/Config.in b/Config.in
>> index 0002df5176..8d471dfb2e 100644
>> --- a/Config.in
>> +++ b/Config.in
>> @@ -94,6 +94,11 @@ config BR2_HOSTARCH_NEEDS_IA32_COMPILER
>>   config BR2_NEEDS_HOST_UTF8_LOCALE
>>   	bool
>>   
>> +# Hidden boolean selected by packages that explicitly need en_US.UTF-8 locale
>> +# to be available on host system.
>> +config BR2_NEEDS_HOST_EN_US_UTF8_LOCALE
>> +	bool
>> +
>>   source "arch/Config.in"
>>   
>>   menu "Build options"
>> diff --git a/package/systemd/Config.in b/package/systemd/Config.in
>> index 661f40dee4..742b3f1f1d 100644
>> --- a/package/systemd/Config.in
>> +++ b/package/systemd/Config.in
>> @@ -27,6 +27,7 @@ menuconfig BR2_PACKAGE_SYSTEMD
>>   	select BR2_PACKAGE_BUSYBOX_SHOW_OTHERS # kmod-tools
>>   	select BR2_PACKAGE_KMOD_TOOLS
>>   	select BR2_TARGET_TZ_INFO
>> +        select BR2_NEEDS_HOST_EN_US_UTF8_LOCALE
> 
> This should use TAB for indentation.
> 
>>   	help
>>   	  systemd is a system and service manager for Linux,
>>   	  compatible with SysV and LSB init scripts. systemd provides
>> diff --git a/package/systemd/systemd.mk b/package/systemd/systemd.mk
>> index 1c31ebb34c..58798f32df 100644
>> --- a/package/systemd/systemd.mk
>> +++ b/package/systemd/systemd.mk
>> @@ -385,23 +385,25 @@ endef
>>   
>>   SYSTEMD_NINJA_OPTS = $(if $(VERBOSE),-v) -j$(PARALLEL_JOBS)
>>   
>> +SYSTEMD_ENV = $(TARGET_MAKE_ENV) LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8
>> +
>>   define SYSTEMD_CONFIGURE_CMDS
>>   	rm -rf $(@D)/build
>>   	mkdir -p $(@D)/build
>> -	$(TARGET_MAKE_ENV) meson $(SYSTEMD_CONF_OPTS) $(@D) $(@D)/build
>> +	$(SYSTEMD_ENV) meson $(SYSTEMD_CONF_OPTS) $(@D) $(@D)/build
>>   endef
>>   
>>   define SYSTEMD_BUILD_CMDS
>> -	$(TARGET_MAKE_ENV) ninja $(SYSTEMD_NINJA_OPTS) -C $(@D)/build
>> +	$(SYSTEMD_ENV) ninja $(SYSTEMD_NINJA_OPTS) -C $(@D)/build
>>   endef
>>   
>>   define SYSTEMD_INSTALL_TARGET_CMDS
>> -	$(TARGET_MAKE_ENV) DESTDIR=$(TARGET_DIR) ninja $(SYSTEMD_NINJA_OPTS) \
>> +	$(SYSTEMD_ENV) DESTDIR=$(TARGET_DIR) ninja $(SYSTEMD_NINJA_OPTS) \
>>   		-C $(@D)/build install
>>   endef
>>   
>>   define SYSTEMD_INSTALL_STAGING_CMDS
>> -	$(TARGET_MAKE_ENV) DESTDIR=$(STAGING_DIR) ninja $(SYSTEMD_NINJA_OPTS) \
>> +	$(SYSTEMD_ENV) DESTDIR=$(STAGING_DIR) ninja $(SYSTEMD_NINJA_OPTS) \
>>   		-C $(@D)/build install
>>   endef
>>   
>> diff --git a/support/dependencies/dependencies.sh b/support/dependencies/dependencies.sh
>> index 1804e85508..881f07008c 100755
>> --- a/support/dependencies/dependencies.sh
>> +++ b/support/dependencies/dependencies.sh
>> @@ -207,6 +207,19 @@ if grep ^BR2_NEEDS_HOST_UTF8_LOCALE=y $BR2_CONFIG > /dev/null; then
>>   	fi
>>   fi
>>   
>> +if grep -q ^BR2_NEEDS_HOST_EN_US_UTF8_LOCALE=y $BR2_CONFIG; then
>> +	if ! which locale > /dev/null ; then
>> +		echo
>> +		echo "You need locale support on your build machine"
>> +		exit 1 ;
>> +	fi
>> +	if ! LC_ALL=en_US.UTF-8 locale -c charmap | grep -q '^UTF-8$'; then
>> +		echo
>> +		echo "You need en_US.UTF-8 locale suppport on the host system"
>> +		exit 1 ;
>> +	fi
>> +fi
>> +
>>   if grep -q ^BR2_NEEDS_HOST_JAVA=y $BR2_CONFIG ; then
>>   	check_prog_host "java"
>>   	JAVA_GCJ=$(java -version 2>&1 | grep gcj)

Hi John. Thanks for spotting the indentation issue and review. I am 
submitting v4 of this patch with the issue fixed.

-- 
Julius Kriukas

^ permalink raw reply	[flat|nested] 19+ messages in thread

* [Buildroot] [PATCH v3] systemd: switch to en_US.UTF-8 locale when building
  2018-02-12 21:34   ` [Buildroot] [PATCH v3] systemd: switch to en_US.UTF-8 locale when building Julius Kriukas
  2018-02-17 18:06     ` John Keeping
  2018-02-22 23:00     ` [Buildroot] [PATCH v4] " Julius Kriukas
@ 2018-02-23  9:06     ` Arnout Vandecappelle
  2 siblings, 0 replies; 19+ messages in thread
From: Arnout Vandecappelle @ 2018-02-23  9:06 UTC (permalink / raw)
  To: buildroot

 Hi Julius,

 I wanted to take the time to think of better solutions, but since I don't find
the time, here is a very quick review.


On 12-02-18 22:34, Julius Kriukas wrote:
> When BR2_REPRODUCIBLE is enabled or host uses non UTF-8 capable locale
> building systemd fails with an error:
> 
> [1/1080] Generating systemd.bg.catalog with a meson_exe.py custom command.
> FAILED: catalog/systemd.bg.catalog
> /buildroot/output/host/bin/python3 /buildroot/output/host/bin/meson --internal exe /buildroot/output/build/systemd-236/build/meson-private/meson_exe_sed_232a0623cc7ce2cd67ec72ed784b76307102ed76.dat
> Warning: You are using 'ANSI_X3.4-1968' which is not a Unicode-compatible locale.
> You might see errors if you use UTF-8 strings as filenames, as strings, or as file contents.
> Please switch to a UTF-8 locale for your platform.
> ...
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 1079: ordinal not in range(128)

 As someone mentioned (IIRC), it's really not OK that some Python script (is it
meson or ninja?) is relying on a locale being set to work correctly. So it would
be much preferable to find out what the real problem is, and to have an
upstreamable patch that fixes it.

 That said, getting to the bottom of this is work, someone has to do it, and in
the mean time we've got a broken system, and we're about to release 2018.02. So
I'm OK to accept this workaround for the time being (and as we all now, that can
be a very long time :-).

 I would, however, like to mention explicitly in the commit message that this is
just a workaround and that really one of the underlying scripts (either meson or
ninja or something systemd-specific, TBD) is broken.


> package/pkg-generic.mk:247: recipe for target '/buildroot/output/build/systemd-236/.stamp_built' failed
> make: *** [/buildroot/output/build/systemd-236/.stamp_built] Error 1
> 
> This patch changes default host system locale from C to en_US.UTF-8 when
> building systemd package to fix this issue. It also introduces
> BR2_NEEDS_HOST_EN_US_UTF8_LOCALE flag that checks if this locale is available on
> the host system. If locale is not available error message is show and build
> process is stopped.

 The part that introduces BR2_NEEDS_HOST_EN_US_UTF8_LOCALE and uses it for
BR2_REPRODUCIBLE should be a separate patch. Indeed, when we find a better
solution for the systemd/meson issue, then we can revert this patch, but the
BR2_REPRODUCIBLE bit should stay.

 Regards,
 Arnout

> 
> Signed-off-by: Julius Kriukas <julius@kriukas.lt>

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

^ permalink raw reply	[flat|nested] 19+ messages in thread

* [Buildroot] [PATCH v4] systemd: switch to en_US.UTF-8 locale when building
  2018-02-22 23:00     ` [Buildroot] [PATCH v4] " Julius Kriukas
@ 2018-02-28 21:31       ` Yann E. MORIN
  2018-02-28 22:13         ` Thomas Petazzoni
  2018-03-02 21:45       ` Thomas Petazzoni
  1 sibling, 1 reply; 19+ messages in thread
From: Yann E. MORIN @ 2018-02-28 21:31 UTC (permalink / raw)
  To: buildroot

Julius, All,

On 2018-02-23 01:00 +0200, Julius Kriukas spake thusly:
> When BR2_REPRODUCIBLE is enabled or host uses non UTF-8 capable locale
> building systemd fails with an error:

As far as I can see, this patch is now no longer about REPRODCIBLE, but
only about systemd; that's good. But then, the commit log should drop
the part about REPRODUCIBLE.

> [1/1080] Generating systemd.bg.catalog with a meson_exe.py custom command.
> FAILED: catalog/systemd.bg.catalog
> /buildroot/output/host/bin/python3 /buildroot/output/host/bin/meson --internal exe /buildroot/output/build/systemd-236/build/meson-private/meson_exe_sed_232a0623cc7ce2cd67ec72ed784b76307102ed76.dat
> Warning: You are using 'ANSI_X3.4-1968' which is not a Unicode-compatible locale.
> You might see errors if you use UTF-8 strings as filenames, as strings, or as file contents.
> Please switch to a UTF-8 locale for your platform.
> ...
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 1079: ordinal not in range(128)
> package/pkg-generic.mk:247: recipe for target '/buildroot/output/build/systemd-236/.stamp_built' failed
> make: *** [/buildroot/output/build/systemd-236/.stamp_built] Error 1
> 
> This patch changes default host system locale from C to en_US.UTF-8 when
> building systemd package to fix this issue. It also introduces
> BR2_NEEDS_HOST_EN_US_UTF8_LOCALE flag that checks if this locale is available on
> the host system. If locale is not available error message is show and build
> process is stopped.

OK, so I do not have en_USUTF-8 on my machine. I have en_GB.UTF-8 (which
I am using). So I'm left out in the cold with this patch.

However, C.UTF-8 is virtually available everywhere. And I was able to
build systemd with C.UTF-8 without any problem.

So, I would  prefer, if we have to force anything, that we force the
use of C.UTF-8.

But then that also means that we override the users' environments, users
that may prefer to see localised messages.

So, as a stop-gap measure, I'm OK with this patch as long as it is
switched to using C.UTF-8. Also, rename the variable, something like:

    BR2_NEEDS_HOST_UTF8_LOCALE_CURRENT

Note: this is different from the existing BR2_NEEDS_HOST_UTF8_LOCALE: it
onlhy requires that a UTF-8 locale be present on the system, while we
now require that it be the current locale.

Regards,
Yann E. MORIN.

> Signed-off-by: Julius Kriukas <julius@kriukas.lt>
> 
> ---
> Changes v1 -> v2:
>   - Added BR2_NEEDS_HOST_EN_US_UTF8_LOCALE flag that is selected by
>     BR2_REPRODUCIBLE.
>   - Extended support/dependencies/dependencies.sh to check for
>     en_US.UTF-8 locale availability if BR2_NEEDS_HOST_EN_US_UTF8_LOCALE
>     is set.
> 
> Changes v2 -> v3:
>   - Always select BR2_NEEDS_HOST_EN_US_UTF8_LOCALE by systemd package.
>   - Do not select BR2_NEEDS_HOST_EN_US_UTF8_LOCALE by BR2_REPRODUCIBLE.
>   - Keep global locale set to C if BR2_REPRODUCIBLE=y.
>   - Use 'grep -q' instead of '> /dev/null' in dependencies.sh script.
> 
> Changes v3 -> v4:
>   - Fix indentation in package/systemd/Config.in, spaces -> tab.
> ---
>  Config.in                            |  5 +++++
>  package/systemd/Config.in            |  1 +
>  package/systemd/systemd.mk           | 10 ++++++----
>  support/dependencies/dependencies.sh | 13 +++++++++++++
>  4 files changed, 25 insertions(+), 4 deletions(-)
> 
> diff --git a/Config.in b/Config.in
> index 0002df5176..8d471dfb2e 100644
> --- a/Config.in
> +++ b/Config.in
> @@ -94,6 +94,11 @@ config BR2_HOSTARCH_NEEDS_IA32_COMPILER
>  config BR2_NEEDS_HOST_UTF8_LOCALE
>  	bool
>  
> +# Hidden boolean selected by packages that explicitly need en_US.UTF-8 locale
> +# to be available on host system.
> +config BR2_NEEDS_HOST_EN_US_UTF8_LOCALE
> +	bool
> +
>  source "arch/Config.in"
>  
>  menu "Build options"
> diff --git a/package/systemd/Config.in b/package/systemd/Config.in
> index 661f40dee4..e83f75c733 100644
> --- a/package/systemd/Config.in
> +++ b/package/systemd/Config.in
> @@ -27,6 +27,7 @@ menuconfig BR2_PACKAGE_SYSTEMD
>  	select BR2_PACKAGE_BUSYBOX_SHOW_OTHERS # kmod-tools
>  	select BR2_PACKAGE_KMOD_TOOLS
>  	select BR2_TARGET_TZ_INFO
> +	select BR2_NEEDS_HOST_EN_US_UTF8_LOCALE
>  	help
>  	  systemd is a system and service manager for Linux,
>  	  compatible with SysV and LSB init scripts. systemd provides
> diff --git a/package/systemd/systemd.mk b/package/systemd/systemd.mk
> index 126727d602..9a6a9cfcbc 100644
> --- a/package/systemd/systemd.mk
> +++ b/package/systemd/systemd.mk
> @@ -385,23 +385,25 @@ endef
>  
>  SYSTEMD_NINJA_OPTS = $(if $(VERBOSE),-v) -j$(PARALLEL_JOBS)
>  
> +SYSTEMD_ENV = $(TARGET_MAKE_ENV) LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8
> +
>  define SYSTEMD_CONFIGURE_CMDS
>  	rm -rf $(@D)/build
>  	mkdir -p $(@D)/build
> -	$(TARGET_MAKE_ENV) meson $(SYSTEMD_CONF_OPTS) $(@D) $(@D)/build
> +	$(SYSTEMD_ENV) meson $(SYSTEMD_CONF_OPTS) $(@D) $(@D)/build
>  endef
>  
>  define SYSTEMD_BUILD_CMDS
> -	$(TARGET_MAKE_ENV) ninja $(SYSTEMD_NINJA_OPTS) -C $(@D)/build
> +	$(SYSTEMD_ENV) ninja $(SYSTEMD_NINJA_OPTS) -C $(@D)/build
>  endef
>  
>  define SYSTEMD_INSTALL_TARGET_CMDS
> -	$(TARGET_MAKE_ENV) DESTDIR=$(TARGET_DIR) ninja $(SYSTEMD_NINJA_OPTS) \
> +	$(SYSTEMD_ENV) DESTDIR=$(TARGET_DIR) ninja $(SYSTEMD_NINJA_OPTS) \
>  		-C $(@D)/build install
>  endef
>  
>  define SYSTEMD_INSTALL_STAGING_CMDS
> -	$(TARGET_MAKE_ENV) DESTDIR=$(STAGING_DIR) ninja $(SYSTEMD_NINJA_OPTS) \
> +	$(SYSTEMD_ENV) DESTDIR=$(STAGING_DIR) ninja $(SYSTEMD_NINJA_OPTS) \
>  		-C $(@D)/build install
>  endef
>  
> diff --git a/support/dependencies/dependencies.sh b/support/dependencies/dependencies.sh
> index 1804e85508..881f07008c 100755
> --- a/support/dependencies/dependencies.sh
> +++ b/support/dependencies/dependencies.sh
> @@ -207,6 +207,19 @@ if grep ^BR2_NEEDS_HOST_UTF8_LOCALE=y $BR2_CONFIG > /dev/null; then
>  	fi
>  fi
>  
> +if grep -q ^BR2_NEEDS_HOST_EN_US_UTF8_LOCALE=y $BR2_CONFIG; then
> +	if ! which locale > /dev/null ; then
> +		echo
> +		echo "You need locale support on your build machine"
> +		exit 1 ;
> +	fi
> +	if ! LC_ALL=en_US.UTF-8 locale -c charmap | grep -q '^UTF-8$'; then
> +		echo
> +		echo "You need en_US.UTF-8 locale suppport on the host system"
> +		exit 1 ;
> +	fi
> +fi
> +
>  if grep -q ^BR2_NEEDS_HOST_JAVA=y $BR2_CONFIG ; then
>  	check_prog_host "java"
>  	JAVA_GCJ=$(java -version 2>&1 | grep gcj)
> -- 
> 2.16.1
> 

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

^ permalink raw reply	[flat|nested] 19+ messages in thread

* [Buildroot] [PATCH v4] systemd: switch to en_US.UTF-8 locale when building
  2018-02-28 21:31       ` Yann E. MORIN
@ 2018-02-28 22:13         ` Thomas Petazzoni
  2018-03-01 18:06           ` Yann E. MORIN
  0 siblings, 1 reply; 19+ messages in thread
From: Thomas Petazzoni @ 2018-02-28 22:13 UTC (permalink / raw)
  To: buildroot

Hello,

On Wed, 28 Feb 2018 22:31:04 +0100, Yann E. MORIN wrote:

> OK, so I do not have en_USUTF-8 on my machine. I have en_GB.UTF-8 (which
> I am using). So I'm left out in the cold with this patch.
> 
> However, C.UTF-8 is virtually available everywhere. And I was able to
> build systemd with C.UTF-8 without any problem.
> 
> So, I would  prefer, if we have to force anything, that we force the
> use of C.UTF-8.
> 
> But then that also means that we override the users' environments, users
> that may prefer to see localised messages.
> 
> So, as a stop-gap measure, I'm OK with this patch as long as it is
> switched to using C.UTF-8. Also, rename the variable, something like:
> 
>     BR2_NEEDS_HOST_UTF8_LOCALE_CURRENT
> 
> Note: this is different from the existing BR2_NEEDS_HOST_UTF8_LOCALE: it
> onlhy requires that a UTF-8 locale be present on the system, while we
> now require that it be the current locale.

Now, we don't require it to be the current locale:


> > +SYSTEMD_ENV = $(TARGET_MAKE_ENV) LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8

The systemd package ensures to pass LANG/LC_ALL pointing to an UTF-8
locale.

So even if your current locale is something else, as long as
en_US.UTF-8 is *available*, things are going to work.

That being said, I also dislike this solution, and think the sytemd
build system should be fixed instead.

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
http://bootlin.com

^ permalink raw reply	[flat|nested] 19+ messages in thread

* [Buildroot] [PATCH v4] systemd: switch to en_US.UTF-8 locale when building
  2018-02-28 22:13         ` Thomas Petazzoni
@ 2018-03-01 18:06           ` Yann E. MORIN
  2018-03-01 18:50             ` Peter Korsgaard
  0 siblings, 1 reply; 19+ messages in thread
From: Yann E. MORIN @ 2018-03-01 18:06 UTC (permalink / raw)
  To: buildroot

Thomas, All,

On 2018-02-28 23:13 +0100, Thomas Petazzoni spake thusly:
> On Wed, 28 Feb 2018 22:31:04 +0100, Yann E. MORIN wrote:
[--SNIP--]
> > Note: this is different from the existing BR2_NEEDS_HOST_UTF8_LOCALE: it
> > onlhy requires that a UTF-8 locale be present on the system, while we
> > now require that it be the current locale.
> Now, we don't require it to be the current locale:
> > > +SYSTEMD_ENV = $(TARGET_MAKE_ENV) LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8
> The systemd package ensures to pass LANG/LC_ALL pointing to an UTF-8
> locale.

Ah, yes, you're right, sorry...

I was confused now, because (without this patch) I had to change my
locale to be able to build systemd. My bad, sorry...

> That being said, I also dislike this solution, and think the sytemd
> build system should be fixed instead.

Yup, that's really sad... But properly fixing it is a totally different
endeavour... :-/

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

^ permalink raw reply	[flat|nested] 19+ messages in thread

* [Buildroot] [PATCH v4] systemd: switch to en_US.UTF-8 locale when building
  2018-03-01 18:06           ` Yann E. MORIN
@ 2018-03-01 18:50             ` Peter Korsgaard
  2018-03-01 19:12               ` Yann E. MORIN
  0 siblings, 1 reply; 19+ messages in thread
From: Peter Korsgaard @ 2018-03-01 18:50 UTC (permalink / raw)
  To: buildroot

>>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes:

Hi,

 > Ah, yes, you're right, sorry...

 > I was confused now, because (without this patch) I had to change my
 > locale to be able to build systemd. My bad, sorry...

Wait, are you saying that it doesn't build with en_GB.UTF-8 locale?
That's odd.

-- 
Bye, Peter Korsgaard

^ permalink raw reply	[flat|nested] 19+ messages in thread

* [Buildroot] [PATCH v4] systemd: switch to en_US.UTF-8 locale when building
  2018-03-01 18:50             ` Peter Korsgaard
@ 2018-03-01 19:12               ` Yann E. MORIN
  2018-03-01 20:04                 ` Peter Korsgaard
  0 siblings, 1 reply; 19+ messages in thread
From: Yann E. MORIN @ 2018-03-01 19:12 UTC (permalink / raw)
  To: buildroot

Peter,All,

On 2018-03-01 19:50 +0100, Peter Korsgaard spake thusly:
> >>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes:
>  > Ah, yes, you're right, sorry...
> 
>  > I was confused now, because (without this patch) I had to change my
>  > locale to be able to build systemd. My bad, sorry...
> 
> Wait, are you saying that it doesn't build with en_GB.UTF-8 locale?
> That's odd.

No, no, don't worry!

In my locale, en_GB.UTF-8, systemd-v237 *does* build fine.

But on my autobuilder, which only has C, POSIX, and C.UTF-8, the default
being POSIX, I had to explicitly set it to C.UTF-8 to test without the
patch.

So, don't worry, all is well when the current locale is any of *.UTF-8.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

^ permalink raw reply	[flat|nested] 19+ messages in thread

* [Buildroot] [PATCH v4] systemd: switch to en_US.UTF-8 locale when building
  2018-03-01 19:12               ` Yann E. MORIN
@ 2018-03-01 20:04                 ` Peter Korsgaard
  0 siblings, 0 replies; 19+ messages in thread
From: Peter Korsgaard @ 2018-03-01 20:04 UTC (permalink / raw)
  To: buildroot

>>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes:

Hi,

 >> Wait, are you saying that it doesn't build with en_GB.UTF-8 locale?
 >> That's odd.

 > No, no, don't worry!

 > In my locale, en_GB.UTF-8, systemd-v237 *does* build fine.

 > But on my autobuilder, which only has C, POSIX, and C.UTF-8, the default
 > being POSIX, I had to explicitly set it to C.UTF-8 to test without the
 > patch.

 > So, don't worry, all is well when the current locale is any of *.UTF-8.

*Pheeuw* ;)

-- 
Bye, Peter Korsgaard

^ permalink raw reply	[flat|nested] 19+ messages in thread

* [Buildroot] [PATCH v4] systemd: switch to en_US.UTF-8 locale when building
  2018-02-22 23:00     ` [Buildroot] [PATCH v4] " Julius Kriukas
  2018-02-28 21:31       ` Yann E. MORIN
@ 2018-03-02 21:45       ` Thomas Petazzoni
       [not found]         ` <87vaecbsf2.fsf@paral.in>
  1 sibling, 1 reply; 19+ messages in thread
From: Thomas Petazzoni @ 2018-03-02 21:45 UTC (permalink / raw)
  To: buildroot

Hello,

On Fri, 23 Feb 2018 01:00:55 +0200, Julius Kriukas wrote:
> When BR2_REPRODUCIBLE is enabled or host uses non UTF-8 capable locale
> building systemd fails with an error:
> 
> [1/1080] Generating systemd.bg.catalog with a meson_exe.py custom command.
> FAILED: catalog/systemd.bg.catalog
> /buildroot/output/host/bin/python3 /buildroot/output/host/bin/meson --internal exe /buildroot/output/build/systemd-236/build/meson-private/meson_exe_sed_232a0623cc7ce2cd67ec72ed784b76307102ed76.dat
> Warning: You are using 'ANSI_X3.4-1968' which is not a Unicode-compatible locale.
> You might see errors if you use UTF-8 strings as filenames, as strings, or as file contents.
> Please switch to a UTF-8 locale for your platform.
> ...
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 1079: ordinal not in range(128)
> package/pkg-generic.mk:247: recipe for target '/buildroot/output/build/systemd-236/.stamp_built' failed
> make: *** [/buildroot/output/build/systemd-236/.stamp_built] Error 1
> 
> This patch changes default host system locale from C to en_US.UTF-8 when
> building systemd package to fix this issue. It also introduces
> BR2_NEEDS_HOST_EN_US_UTF8_LOCALE flag that checks if this locale is available on
> the host system. If locale is not available error message is show and build
> process is stopped.
> 
> Signed-off-by: Julius Kriukas <julius@kriukas.lt>
> 
> ---
> Changes v1 -> v2:
>   - Added BR2_NEEDS_HOST_EN_US_UTF8_LOCALE flag that is selected by
>     BR2_REPRODUCIBLE.
>   - Extended support/dependencies/dependencies.sh to check for
>     en_US.UTF-8 locale availability if BR2_NEEDS_HOST_EN_US_UTF8_LOCALE
>     is set.

I've changed to use C.UTF-8 instead of en_US.UTF-8 and applied to
master. Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
https://bootlin.com

^ permalink raw reply	[flat|nested] 19+ messages in thread

* [Buildroot] [PATCH v4] systemd: switch to en_US.UTF-8 locale when building
       [not found]         ` <87vaecbsf2.fsf@paral.in>
@ 2018-03-04  9:01           ` Yann E. MORIN
  0 siblings, 0 replies; 19+ messages in thread
From: Yann E. MORIN @ 2018-03-04  9:01 UTC (permalink / raw)
  To: buildroot

Christian, All,

On 2018-03-03 19:22 -0500, Christian Stewart spake thusly:
> Thomas Petazzoni <thomas.petazzoni@bootlin.com> writes:
> > I've changed to use C.UTF-8 instead of en_US.UTF-8 and applied to
> > master. Thanks!
> 
> This has broken the build on Gentoo, unfortunately.

Grr.. Gentoo... Grr...

The major distros (Debian and derivatives, Fedora...) all have C.UTF-8.
Gentoo is again playing solo... Sigh... :-/

I'm onto a fix for that, now...

> Is it possible to compile with an en_US.UTF-8 or other language-specific
> UTF-8 locale, such that the build again works on Gentoo?

Not everyone on Earth is American, by far, so we already ruled out using
a hard-coded en_US.UTF-8 locale; for example, I don't have it on my
system (although I do have, and do use, an english locale, en_GB.UTF-8).

I'm onto a fix now...

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

^ permalink raw reply	[flat|nested] 19+ messages in thread

end of thread, other threads:[~2018-03-04  9:01 UTC | newest]

Thread overview: 19+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-01-28 11:22 [Buildroot] [PATCH 1/1] Change host locale from C to en_US.UTF-8 for reproducable builds Julius Kriukas
2018-01-30 22:38 ` Thomas Petazzoni
2018-01-31 21:34   ` Julius Kriukas
2018-01-31 21:36 ` [Buildroot] [PATCH v2] " Julius Kriukas
2018-02-04 15:19   ` Yann E. MORIN
2018-02-06 16:32   ` Thomas Petazzoni
2018-02-12 21:34   ` [Buildroot] [PATCH v3] systemd: switch to en_US.UTF-8 locale when building Julius Kriukas
2018-02-17 18:06     ` John Keeping
2018-02-22 23:02       ` Julius Kriukas
2018-02-22 23:00     ` [Buildroot] [PATCH v4] " Julius Kriukas
2018-02-28 21:31       ` Yann E. MORIN
2018-02-28 22:13         ` Thomas Petazzoni
2018-03-01 18:06           ` Yann E. MORIN
2018-03-01 18:50             ` Peter Korsgaard
2018-03-01 19:12               ` Yann E. MORIN
2018-03-01 20:04                 ` Peter Korsgaard
2018-03-02 21:45       ` Thomas Petazzoni
     [not found]         ` <87vaecbsf2.fsf@paral.in>
2018-03-04  9:01           ` Yann E. MORIN
2018-02-23  9:06     ` [Buildroot] [PATCH v3] " Arnout Vandecappelle

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.