[U-Boot] [PATCH] buildman: Translate more strings to latin-1

[U-Boot] [PATCH] buildman: Translate more strings to latin-1

[Daniel Schwierzeck]

This is a follow-up patch for commit fbeb33752999e7317113199ef89873d6b6916814.

This fixes following exception:

Exception in thread Thread-7:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/u-boot/tools/buildman/builderthread.py", line 475, in run
    self.RunJob(job)
  File "/u-boot/tools/buildman/builderthread.py", line 456, in RunJob
    self._WriteResult(result, job.keep_outputs)
  File "/u-boot/tools/buildman/builderthread.py", line 333, in _WriteResult
    print >>fd, dump_result.stdout,
UnicodeEncodeError: 'ascii' codec can't encode characters in position 75-76: ordinal not in range(128)

Signed-off-by: Daniel Schwierzeck <daniel.schwierzeck@gmail.com>

---

 tools/buildman/builderthread.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tools/buildman/builderthread.py b/tools/buildman/builderthread.py
index acaf5007f5..0a460878f3 100644
--- a/tools/buildman/builderthread.py
+++ b/tools/buildman/builderthread.py
@@ -330,7 +330,7 @@ class BuilderThread(threading.Thread):
                     objdump = self.builder.GetObjdumpFile(result.commit_upto,
                                     result.brd.target, fname)
                     with open(objdump, 'w') as fd:
-                        print >>fd, dump_result.stdout,
+                        print >>fd, dump_result.stdout.encode('latin-1', 'ignore'),
                     for line in dump_result.stdout.splitlines():
                         fields = line.split()
                         if len(fields) > 5 and fields[1] == '.rodata':
-- 
2.13.0

[U-Boot] [PATCH] buildman: Translate more strings to latin-1

[Simon Glass]

Hi Daniel,

On 31 May 2017 at 07:40, Daniel Schwierzeck
<daniel.schwierzeck@gmail.com> wrote:
> This is a follow-up patch for commit fbeb33752999e7317113199ef89873d6b6916814.
>
> This fixes following exception:
>
> Exception in thread Thread-7:
> Traceback (most recent call last):
>   File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
>     self.run()
>   File "/u-boot/tools/buildman/builderthread.py", line 475, in run
>     self.RunJob(job)
>   File "/u-boot/tools/buildman/builderthread.py", line 456, in RunJob
>     self._WriteResult(result, job.keep_outputs)
>   File "/u-boot/tools/buildman/builderthread.py", line 333, in _WriteResult
>     print >>fd, dump_result.stdout,
> UnicodeEncodeError: 'ascii' codec can't encode characters in position 75-76: ordinal not in range(128)
>
> Signed-off-by: Daniel Schwierzeck <daniel.schwierzeck@gmail.com>
>
> ---
>
>  tools/buildman/builderthread.py | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/tools/buildman/builderthread.py b/tools/buildman/builderthread.py
> index acaf5007f5..0a460878f3 100644
> --- a/tools/buildman/builderthread.py
> +++ b/tools/buildman/builderthread.py
> @@ -330,7 +330,7 @@ class BuilderThread(threading.Thread):
>                      objdump = self.builder.GetObjdumpFile(result.commit_upto,
>                                      result.brd.target, fname)
>                      with open(objdump, 'w') as fd:
> -                        print >>fd, dump_result.stdout,
> +                        print >>fd, dump_result.stdout.encode('latin-1', 'ignore'),

I am worried that this will produce gibberish in the output. Do you
know what specific characters are causing problems? Can we encode to
utf-8? BTW I recently sent a patman series to try to resolve unicode
issues in patman.

>                      for line in dump_result.stdout.splitlines():
>                          fields = line.split()
>                          if len(fields) > 5 and fields[1] == '.rodata':
> --
> 2.13.0
>

Regards,
Simon

[U-Boot] [PATCH] buildman: Translate more strings to latin-1

[Daniel Schwierzeck]

Am 01.06.2017 um 15:21 schrieb Simon Glass:
> Hi Daniel,
> 
> On 31 May 2017 at 07:40, Daniel Schwierzeck
> <daniel.schwierzeck@gmail.com> wrote:
>> This is a follow-up patch for commit fbeb33752999e7317113199ef89873d6b6916814.
>>
>> This fixes following exception:
>>
>> Exception in thread Thread-7:
>> Traceback (most recent call last):
>>   File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
>>     self.run()
>>   File "/u-boot/tools/buildman/builderthread.py", line 475, in run
>>     self.RunJob(job)
>>   File "/u-boot/tools/buildman/builderthread.py", line 456, in RunJob
>>     self._WriteResult(result, job.keep_outputs)
>>   File "/u-boot/tools/buildman/builderthread.py", line 333, in _WriteResult
>>     print >>fd, dump_result.stdout,
>> UnicodeEncodeError: 'ascii' codec can't encode characters in position 75-76: ordinal not in range(128)
>>
>> Signed-off-by: Daniel Schwierzeck <daniel.schwierzeck@gmail.com>
>>
>> ---
>>
>>  tools/buildman/builderthread.py | 2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/tools/buildman/builderthread.py b/tools/buildman/builderthread.py
>> index acaf5007f5..0a460878f3 100644
>> --- a/tools/buildman/builderthread.py
>> +++ b/tools/buildman/builderthread.py
>> @@ -330,7 +330,7 @@ class BuilderThread(threading.Thread):
>>                      objdump = self.builder.GetObjdumpFile(result.commit_upto,
>>                                      result.brd.target, fname)
>>                      with open(objdump, 'w') as fd:
>> -                        print >>fd, dump_result.stdout,
>> +                        print >>fd, dump_result.stdout.encode('latin-1', 'ignore'),
> 
> I am worried that this will produce gibberish in the output. Do you
> know what specific characters are causing problems? Can we encode to
> utf-8? BTW I recently sent a patman series to try to resolve unicode
> issues in patman.
> 

objdump uses localized output. If I use LC_ALL=C, the exception doesn't
occur. My system's default language is German. Thus the offending
character is the German 'ö':

...
Sektionen:
Idx Name          Größe     VMA       LMA       Datei-Off Ausr.
  0 .text         00032a94  9f000000  9f000000  000000c0  2**4
                  CONTENTS, ALLOC, LOAD, CODE
  1 .rodata       00008e50  9f032aa0  9f032aa0  00032b60  2**4
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
...

Actually I'd chosen UTF-8 too, but I followed Tom's patch which used
latin-1 ;)

So we should always encode the output of tools such as binutils to
UTF-8, shouldn't we? I can send an updated patch.

-- 
- Daniel

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20170602/a9d58869/attachment.sig>

[U-Boot] [PATCH] buildman: Translate more strings to latin-1

[Tom Rini]

On Fri, Jun 02, 2017 at 11:18:27AM +0200, Daniel Schwierzeck wrote:
> 
> 
> Am 01.06.2017 um 15:21 schrieb Simon Glass:
> > Hi Daniel,
> > 
> > On 31 May 2017 at 07:40, Daniel Schwierzeck
> > <daniel.schwierzeck@gmail.com> wrote:
> >> This is a follow-up patch for commit fbeb33752999e7317113199ef89873d6b6916814.
> >>
> >> This fixes following exception:
> >>
> >> Exception in thread Thread-7:
> >> Traceback (most recent call last):
> >>   File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
> >>     self.run()
> >>   File "/u-boot/tools/buildman/builderthread.py", line 475, in run
> >>     self.RunJob(job)
> >>   File "/u-boot/tools/buildman/builderthread.py", line 456, in RunJob
> >>     self._WriteResult(result, job.keep_outputs)
> >>   File "/u-boot/tools/buildman/builderthread.py", line 333, in _WriteResult
> >>     print >>fd, dump_result.stdout,
> >> UnicodeEncodeError: 'ascii' codec can't encode characters in position 75-76: ordinal not in range(128)
> >>
> >> Signed-off-by: Daniel Schwierzeck <daniel.schwierzeck@gmail.com>
> >>
> >> ---
> >>
> >>  tools/buildman/builderthread.py | 2 +-
> >>  1 file changed, 1 insertion(+), 1 deletion(-)
> >>
> >> diff --git a/tools/buildman/builderthread.py b/tools/buildman/builderthread.py
> >> index acaf5007f5..0a460878f3 100644
> >> --- a/tools/buildman/builderthread.py
> >> +++ b/tools/buildman/builderthread.py
> >> @@ -330,7 +330,7 @@ class BuilderThread(threading.Thread):
> >>                      objdump = self.builder.GetObjdumpFile(result.commit_upto,
> >>                                      result.brd.target, fname)
> >>                      with open(objdump, 'w') as fd:
> >> -                        print >>fd, dump_result.stdout,
> >> +                        print >>fd, dump_result.stdout.encode('latin-1', 'ignore'),
> > 
> > I am worried that this will produce gibberish in the output. Do you
> > know what specific characters are causing problems? Can we encode to
> > utf-8? BTW I recently sent a patman series to try to resolve unicode
> > issues in patman.
> > 
> 
> objdump uses localized output. If I use LC_ALL=C, the exception doesn't
> occur. My system's default language is German. Thus the offending
> character is the German 'ö':
> 
> ...
> Sektionen:
> Idx Name          Größe     VMA       LMA       Datei-Off Ausr.
>   0 .text         00032a94  9f000000  9f000000  000000c0  2**4
>                   CONTENTS, ALLOC, LOAD, CODE
>   1 .rodata       00008e50  9f032aa0  9f032aa0  00032b60  2**4
>                   CONTENTS, ALLOC, LOAD, READONLY, DATA
> ...
> 
> Actually I'd chosen UTF-8 too, but I followed Tom's patch which used
> latin-1 ;)
> 
> So we should always encode the output of tools such as binutils to
> UTF-8, shouldn't we? I can send an updated patch.

I'd be fine with UTF-8 everywhere too, I just borrowed an existing
example that was latin-1 :)

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20170602/b79e210e/attachment.sig>

[U-Boot] [PATCH] buildman: Translate more strings to latin-1

[Simon Glass]

Hi Daniel,

On 2 June 2017 at 07:21, Tom Rini <trini@konsulko.com> wrote:
> On Fri, Jun 02, 2017 at 11:18:27AM +0200, Daniel Schwierzeck wrote:
>>
>>
>> Am 01.06.2017 um 15:21 schrieb Simon Glass:
>> > Hi Daniel,
>> >
>> > On 31 May 2017 at 07:40, Daniel Schwierzeck
>> > <daniel.schwierzeck@gmail.com> wrote:
>> >> This is a follow-up patch for commit fbeb33752999e7317113199ef89873d6b6916814.
>> >>
>> >> This fixes following exception:
>> >>
>> >> Exception in thread Thread-7:
>> >> Traceback (most recent call last):
>> >>   File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
>> >>     self.run()
>> >>   File "/u-boot/tools/buildman/builderthread.py", line 475, in run
>> >>     self.RunJob(job)
>> >>   File "/u-boot/tools/buildman/builderthread.py", line 456, in RunJob
>> >>     self._WriteResult(result, job.keep_outputs)
>> >>   File "/u-boot/tools/buildman/builderthread.py", line 333, in _WriteResult
>> >>     print >>fd, dump_result.stdout,
>> >> UnicodeEncodeError: 'ascii' codec can't encode characters in position 75-76: ordinal not in range(128)
>> >>
>> >> Signed-off-by: Daniel Schwierzeck <daniel.schwierzeck@gmail.com>
>> >>
>> >> ---
>> >>
>> >>  tools/buildman/builderthread.py | 2 +-
>> >>  1 file changed, 1 insertion(+), 1 deletion(-)
>> >>
>> >> diff --git a/tools/buildman/builderthread.py b/tools/buildman/builderthread.py
>> >> index acaf5007f5..0a460878f3 100644
>> >> --- a/tools/buildman/builderthread.py
>> >> +++ b/tools/buildman/builderthread.py
>> >> @@ -330,7 +330,7 @@ class BuilderThread(threading.Thread):
>> >>                      objdump = self.builder.GetObjdumpFile(result.commit_upto,
>> >>                                      result.brd.target, fname)
>> >>                      with open(objdump, 'w') as fd:
>> >> -                        print >>fd, dump_result.stdout,
>> >> +                        print >>fd, dump_result.stdout.encode('latin-1', 'ignore'),
>> >
>> > I am worried that this will produce gibberish in the output. Do you
>> > know what specific characters are causing problems? Can we encode to
>> > utf-8? BTW I recently sent a patman series to try to resolve unicode
>> > issues in patman.
>> >
>>
>> objdump uses localized output. If I use LC_ALL=C, the exception doesn't
>> occur. My system's default language is German. Thus the offending
>> character is the German 'ö':
>>
>> ...
>> Sektionen:
>> Idx Name          Größe     VMA       LMA       Datei-Off Ausr.
>>   0 .text         00032a94  9f000000  9f000000  000000c0  2**4
>>                   CONTENTS, ALLOC, LOAD, CODE
>>   1 .rodata       00008e50  9f032aa0  9f032aa0  00032b60  2**4
>>                   CONTENTS, ALLOC, LOAD, READONLY, DATA
>> ...
>>
>> Actually I'd chosen UTF-8 too, but I followed Tom's patch which used
>> latin-1 ;)
>>
>> So we should always encode the output of tools such as binutils to
>> UTF-8, shouldn't we? I can send an updated patch.
>
> I'd be fine with UTF-8 everywhere too, I just borrowed an existing
> example that was latin-1 :)

Are you planning to send an updated patch?

Regards,
Simon

[U-Boot] [PATCH] buildman: Translate more strings to latin-1

[Daniel Schwierzeck]

Am 07.06.2017 um 13:55 schrieb Simon Glass:
...
>>>
>>> Actually I'd chosen UTF-8 too, but I followed Tom's patch which used
>>> latin-1 ;)
>>>
>>> So we should always encode the output of tools such as binutils to
>>> UTF-8, shouldn't we? I can send an updated patch.
>>
>> I'd be fine with UTF-8 everywhere too, I just borrowed an existing
>> example that was latin-1 :)
> 
> Are you planning to send an updated patch?
> 

sure, I forgot to send it ;)

-- 
- Daniel

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20170607/943c3761/attachment.sig>

[U-Boot] [PATCH v2] buildman: Translate binutils output to UTF-8

[Daniel Schwierzeck]

binutils are using localized output. On systems with locales and special
characters this leads to exceptions like this:

Exception in thread Thread-7:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/u-boot/tools/buildman/builderthread.py", line 475, in run
    self.RunJob(job)
  File "/u-boot/tools/buildman/builderthread.py", line 456, in RunJob
    self._WriteResult(result, job.keep_outputs)
  File "/u-boot/tools/buildman/builderthread.py", line 333, in _WriteResult
    print >>fd, dump_result.stdout,
UnicodeEncodeError: 'ascii' codec can't encode characters in position 75-76: ordinal not in range(128)

Always encode the value of CommandResult::stdout to UTF-8 before
writing it to stdout or a log file.

Signed-off-by: Daniel Schwierzeck <daniel.schwierzeck@gmail.com>

---

Changes in v2:
- encode all occurences of CommandResult::stdout after spawning binutils to UTF-8
- reworded commit message

 tools/buildman/builderthread.py | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/tools/buildman/builderthread.py b/tools/buildman/builderthread.py
index acaf5007f5..4ca2f2eb90 100644
--- a/tools/buildman/builderthread.py
+++ b/tools/buildman/builderthread.py
@@ -280,13 +280,13 @@ class BuilderThread(threading.Thread):
         outfile = os.path.join(build_dir, 'log')
         with open(outfile, 'w') as fd:
             if result.stdout:
-                fd.write(result.stdout.encode('latin-1', 'ignore'))
+                fd.write(result.stdout.encode('UTF-8', 'ignore'))
 
         errfile = self.builder.GetErrFile(result.commit_upto,
                 result.brd.target)
         if result.stderr:
             with open(errfile, 'w') as fd:
-                fd.write(result.stderr.encode('latin-1', 'ignore'))
+                fd.write(result.stderr.encode('UTF-8', 'ignore'))
         elif os.path.exists(errfile):
             os.remove(errfile)
 
@@ -319,7 +319,7 @@ class BuilderThread(threading.Thread):
                     nm = self.builder.GetFuncSizesFile(result.commit_upto,
                                     result.brd.target, fname)
                     with open(nm, 'w') as fd:
-                        print >>fd, nm_result.stdout,
+                        print >>fd, nm_result.stdout.encode('UTF-8'),
 
                 cmd = ['%sobjdump' % self.toolchain.cross, '-h', fname]
                 dump_result = command.RunPipe([cmd], capture=True,
@@ -330,7 +330,7 @@ class BuilderThread(threading.Thread):
                     objdump = self.builder.GetObjdumpFile(result.commit_upto,
                                     result.brd.target, fname)
                     with open(objdump, 'w') as fd:
-                        print >>fd, dump_result.stdout,
+                        print >>fd, dump_result.stdout.encode('UTF-8'),
                     for line in dump_result.stdout.splitlines():
                         fields = line.split()
                         if len(fields) > 5 and fields[1] == '.rodata':
@@ -352,7 +352,7 @@ class BuilderThread(threading.Thread):
                 sizes = self.builder.GetSizesFile(result.commit_upto,
                                 result.brd.target)
                 with open(sizes, 'w') as fd:
-                    print >>fd, '\n'.join(lines)
+                    print >>fd, '\n'.join(lines).encode('UTF-8')
 
         # Write out the configuration files, with a special case for SPL
         for dirname in ['', 'spl', 'tpl']:
-- 
2.13.0

[U-Boot] [PATCH v2] buildman: Translate binutils output to UTF-8

[Simon Glass]

Hi Daniel,

On 7 June 2017 at 06:19, Daniel Schwierzeck
<daniel.schwierzeck@gmail.com> wrote:
> binutils are using localized output. On systems with locales and special
> characters this leads to exceptions like this:
>
> Exception in thread Thread-7:
> Traceback (most recent call last):
>   File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
>     self.run()
>   File "/u-boot/tools/buildman/builderthread.py", line 475, in run
>     self.RunJob(job)
>   File "/u-boot/tools/buildman/builderthread.py", line 456, in RunJob
>     self._WriteResult(result, job.keep_outputs)
>   File "/u-boot/tools/buildman/builderthread.py", line 333, in _WriteResult
>     print >>fd, dump_result.stdout,
> UnicodeEncodeError: 'ascii' codec can't encode characters in position 75-76: ordinal not in range(128)
>
> Always encode the value of CommandResult::stdout to UTF-8 before
> writing it to stdout or a log file.
>
> Signed-off-by: Daniel Schwierzeck <daniel.schwierzeck@gmail.com>
>
> ---
>
> Changes in v2:
> - encode all occurences of CommandResult::stdout after spawning binutils to UTF-8
> - reworded commit message
>
>  tools/buildman/builderthread.py | 10 +++++-----
>  1 file changed, 5 insertions(+), 5 deletions(-)
>

I'm still seeing a rare failure, e.g.:

https://travis-ci.org/sglass68/u-boot/jobs/240350621

Any thoughts on that?

Regards,
Simon

[U-Boot] [PATCH v2] buildman: Translate binutils output to UTF-8

[Daniel Schwierzeck]

Am 07.06.2017 um 17:52 schrieb Simon Glass:
> Hi Daniel,
> 
> On 7 June 2017 at 06:19, Daniel Schwierzeck
> <daniel.schwierzeck@gmail.com> wrote:
>> binutils are using localized output. On systems with locales and special
>> characters this leads to exceptions like this:
>>
>> Exception in thread Thread-7:
>> Traceback (most recent call last):
>>   File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
>>     self.run()
>>   File "/u-boot/tools/buildman/builderthread.py", line 475, in run
>>     self.RunJob(job)
>>   File "/u-boot/tools/buildman/builderthread.py", line 456, in RunJob
>>     self._WriteResult(result, job.keep_outputs)
>>   File "/u-boot/tools/buildman/builderthread.py", line 333, in _WriteResult
>>     print >>fd, dump_result.stdout,
>> UnicodeEncodeError: 'ascii' codec can't encode characters in position 75-76: ordinal not in range(128)
>>
>> Always encode the value of CommandResult::stdout to UTF-8 before
>> writing it to stdout or a log file.
>>
>> Signed-off-by: Daniel Schwierzeck <daniel.schwierzeck@gmail.com>
>>
>> ---
>>
>> Changes in v2:
>> - encode all occurences of CommandResult::stdout after spawning binutils to UTF-8
>> - reworded commit message
>>
>>  tools/buildman/builderthread.py | 10 +++++-----
>>  1 file changed, 5 insertions(+), 5 deletions(-)
>>
> 
> I'm still seeing a rare failure, e.g.:
> 
> https://travis-ci.org/sglass68/u-boot/jobs/240350621
> 
> Any thoughts on that?
> 

I'll look into it

-- 
- Daniel

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20170607/178e7e0e/attachment.sig>

[U-Boot] [PATCH 0/2] Fix unicode handling in buildman

[Daniel Schwierzeck]

This series fixes and optimizes the unicode handling in buildman.
It's based on the similar series for patman and applies on top
of u-boot-dm/master.

This has been tested by forcibly adding unicode characters in
the stdout and stderr stream.

Examples:

The patch

--- a/arch/arm/config.mk
+++ b/arch/arm/config.mk
@@ -69,7 +69,7 @@ checkgcc6:
        @if test "$(call cc-name)" = "gcc" -a \
                        "$(call cc-version)" -lt "0600"; then \
                echo -n '*** Your GCC is older than 6.0 and will not be '; \
-               echo 'supported starting in v2018.01.'; \
+               echo 'supported starting <C3><B6> in v2018.01.'; \
        fi

called with `tools/buildman/buildman -P xilinx_zynqmp_ep` results in stdout:

*** Your GCC is older than 6.0 and will not be supported starting ? in v2018.01.

The patch

--- a/arch/sandbox/cpu/os.c
+++ b/arch/sandbox/cpu/os.c
@@ -40,6 +40,7 @@ ssize_t os_read(int fd, void *buf, size_t count)
 ssize_t os_read_no_block(int fd, void *buf, size_t count)
 {
        const int flags = fcntl(fd, F_GETFL, 0);
+       const char dummy[] = "aaa<C3><A4><C3><B6>cc";

        fcntl(fd, F_SETFL, flags | O_NONBLOCK);
        return os_read(fd, buf, count);

called with `tools/buildman/buildman -P sandbox` results in stderr:

arch/sandbox/cpu/os.c: In function ?os_read_no_block?:
arch/sandbox/cpu/os.c:43:13: warning: unused variable ?dummy? [-Wunused-variable]
  const char dummy[] = "aaa??cc";
             ^


Daniel Schwierzeck (2):
  buildman: disable localized and unicode output of all build tools
  buildman: properly translate strings for log and err files to ASCII

 tools/buildman/builderthread.py | 6 ++++--
 tools/buildman/toolchain.py     | 6 +++++-
 2 files changed, 9 insertions(+), 3 deletions(-)

-- 
2.13.0

[U-Boot] [PATCH 1/2] buildman: disable localized and unicode output of all build tools

[Daniel Schwierzeck]

Build tools like Make, gcc or binutils support localized output
or unicode encoded output dependent on the default system locale.
This is not useful for buildman, where we want reproducible
warning or error messages or where the output of binutils is
further processed.

Signed-off-by: Daniel Schwierzeck <daniel.schwierzeck@gmail.com>
---

 tools/buildman/toolchain.py | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/tools/buildman/toolchain.py b/tools/buildman/toolchain.py
index 5cf97ac814..2076323d5d 100644
--- a/tools/buildman/toolchain.py
+++ b/tools/buildman/toolchain.py
@@ -144,7 +144,9 @@ class Toolchain:
         """Returns an environment for using the toolchain.
 
         Thie takes the current environment and adds CROSS_COMPILE so that
-        the tool chain will operate correctly.
+        the tool chain will operate correctly. This also disables localized
+        output and possibly unicode encoded output of all build tools by
+        adding LC_ALL=C.
 
         Args:
             full_path: Return the full path in CROSS_COMPILE and don't set
@@ -159,6 +161,8 @@ class Toolchain:
             env['CROSS_COMPILE'] = wrapper + self.cross
             env['PATH'] = self.path + ':' + env['PATH']
 
+        env['LC_ALL'] = 'C'
+
         return env
 
 
-- 
2.13.0

[U-Boot] [PATCH 2/2] buildman: properly translate strings for log and err files to ASCII

[Daniel Schwierzeck]

The build output can still produce unicode encoded output. But in
the buildman's log and err files we only want plain ASCII characters.

To handle all situations with unicode and non-unicode output, encode
the stdout and stderr strings to UTF-8 and afterwards to ASCII with
replacing all special characters.

Signed-off-by: Daniel Schwierzeck <daniel.schwierzeck@gmail.com>

---

 tools/buildman/builderthread.py | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/tools/buildman/builderthread.py b/tools/buildman/builderthread.py
index acaf5007f5..9e8ca80c5b 100644
--- a/tools/buildman/builderthread.py
+++ b/tools/buildman/builderthread.py
@@ -280,13 +280,15 @@ class BuilderThread(threading.Thread):
         outfile = os.path.join(build_dir, 'log')
         with open(outfile, 'w') as fd:
             if result.stdout:
-                fd.write(result.stdout.encode('latin-1', 'ignore'))
+                # We don't want unicode characters in log files
+                fd.write(result.stdout.decode('UTF-8').encode('ASCII', 'replace'))
 
         errfile = self.builder.GetErrFile(result.commit_upto,
                 result.brd.target)
         if result.stderr:
             with open(errfile, 'w') as fd:
-                fd.write(result.stderr.encode('latin-1', 'ignore'))
+                # We don't want unicode characters in log files
+                fd.write(result.stderr.decode('UTF-8').encode('ASCII', 'replace'))
         elif os.path.exists(errfile):
             os.remove(errfile)
 
-- 
2.13.0

[U-Boot] [PATCH 2/2] buildman: properly translate strings for log and err files to ASCII

[sjg at google.com]

The build output can still produce unicode encoded output. But in
the buildman's log and err files we only want plain ASCII characters.

To handle all situations with unicode and non-unicode output, encode
the stdout and stderr strings to UTF-8 and afterwards to ASCII with
replacing all special characters.

Signed-off-by: Daniel Schwierzeck <daniel.schwierzeck@gmail.com>

---

 tools/buildman/builderthread.py | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

Applied to u-boot-dm, thanks!

[U-Boot] [PATCH 1/2] buildman: disable localized and unicode output of all build tools

[sjg at google.com]

Build tools like Make, gcc or binutils support localized output
or unicode encoded output dependent on the default system locale.
This is not useful for buildman, where we want reproducible
warning or error messages or where the output of binutils is
further processed.

Signed-off-by: Daniel Schwierzeck <daniel.schwierzeck@gmail.com>
---

 tools/buildman/toolchain.py | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

Applied to u-boot-dm, thanks!

[U-Boot] [PATCH v2] buildman: Translate binutils output to UTF-8

[Simon Glass]

Hi Daniel,

On 7 June 2017 at 10:03, Daniel Schwierzeck
<daniel.schwierzeck@gmail.com> wrote:
>
>
> Am 07.06.2017 um 17:52 schrieb Simon Glass:
>> Hi Daniel,
>>
>> On 7 June 2017 at 06:19, Daniel Schwierzeck
>> <daniel.schwierzeck@gmail.com> wrote:
>>> binutils are using localized output. On systems with locales and special
>>> characters this leads to exceptions like this:
>>>
>>> Exception in thread Thread-7:
>>> Traceback (most recent call last):
>>>   File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
>>>     self.run()
>>>   File "/u-boot/tools/buildman/builderthread.py", line 475, in run
>>>     self.RunJob(job)
>>>   File "/u-boot/tools/buildman/builderthread.py", line 456, in RunJob
>>>     self._WriteResult(result, job.keep_outputs)
>>>   File "/u-boot/tools/buildman/builderthread.py", line 333, in _WriteResult
>>>     print >>fd, dump_result.stdout,
>>> UnicodeEncodeError: 'ascii' codec can't encode characters in position 75-76: ordinal not in range(128)
>>>
>>> Always encode the value of CommandResult::stdout to UTF-8 before
>>> writing it to stdout or a log file.
>>>
>>> Signed-off-by: Daniel Schwierzeck <daniel.schwierzeck@gmail.com>
>>>
>>> ---
>>>
>>> Changes in v2:
>>> - encode all occurences of CommandResult::stdout after spawning binutils to UTF-8
>>> - reworded commit message
>>>
>>>  tools/buildman/builderthread.py | 10 +++++-----
>>>  1 file changed, 5 insertions(+), 5 deletions(-)
>>>
>>
>> I'm still seeing a rare failure, e.g.:
>>
>> https://travis-ci.org/sglass68/u-boot/jobs/240350621
>>
>> Any thoughts on that?
>>
>
> I'll look into it

Thanks. I believe the two new patches you sent supersede this one.

Regards,
Simon

[U-Boot] [PATCH v2] buildman: Translate binutils output to UTF-8

[Daniel Schwierzeck]

Am 09.06.2017 um 14:27 schrieb Simon Glass:
> Hi Daniel,
> 
> On 7 June 2017 at 10:03, Daniel Schwierzeck
> <daniel.schwierzeck@gmail.com> wrote:
>>
>>
>> Am 07.06.2017 um 17:52 schrieb Simon Glass:
>>> Hi Daniel,
>>>
>>> On 7 June 2017 at 06:19, Daniel Schwierzeck
>>> <daniel.schwierzeck@gmail.com> wrote:
>>>> binutils are using localized output. On systems with locales and special
>>>> characters this leads to exceptions like this:
>>>>
>>>> Exception in thread Thread-7:
>>>> Traceback (most recent call last):
>>>>   File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
>>>>     self.run()
>>>>   File "/u-boot/tools/buildman/builderthread.py", line 475, in run
>>>>     self.RunJob(job)
>>>>   File "/u-boot/tools/buildman/builderthread.py", line 456, in RunJob
>>>>     self._WriteResult(result, job.keep_outputs)
>>>>   File "/u-boot/tools/buildman/builderthread.py", line 333, in _WriteResult
>>>>     print >>fd, dump_result.stdout,
>>>> UnicodeEncodeError: 'ascii' codec can't encode characters in position 75-76: ordinal not in range(128)
>>>>
>>>> Always encode the value of CommandResult::stdout to UTF-8 before
>>>> writing it to stdout or a log file.
>>>>
>>>> Signed-off-by: Daniel Schwierzeck <daniel.schwierzeck@gmail.com>
>>>>
>>>> ---
>>>>
>>>> Changes in v2:
>>>> - encode all occurences of CommandResult::stdout after spawning binutils to UTF-8
>>>> - reworded commit message
>>>>
>>>>  tools/buildman/builderthread.py | 10 +++++-----
>>>>  1 file changed, 5 insertions(+), 5 deletions(-)
>>>>
>>>
>>> I'm still seeing a rare failure, e.g.:
>>>
>>> https://travis-ci.org/sglass68/u-boot/jobs/240350621
>>>
>>> Any thoughts on that?
>>>
>>
>> I'll look into it
> 
> Thanks. I believe the two new patches you sent supersede this one.
> 

yes, this one is superseded.

-- 
- Daniel

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20170609/2fac34cd/attachment.sig>