* [U-Boot] [PATCH v2 00/15] scripts: Convert to Python 3
@ 2019-10-25 1:03 Simon Glass
2019-10-25 1:03 ` [U-Boot] [PATCH v2 01/15] patman: Adjust 'command' to return strings instead of bytes Simon Glass
` (14 more replies)
0 siblings, 15 replies; 23+ messages in thread
From: Simon Glass @ 2019-10-25 1:03 UTC (permalink / raw)
To: u-boot
This converts some of the scripts that I've been involved in to use
Python 3.
There is still one problem in that fdt_property_stub() doesn't work
correctly in pylibfdt:
ftest.TestFunctional.testExtractAllEntries Traceback (most recent call last):
TypeError: in method 'fdt_property_stub', argument 3 of type 'char const *'
Changes in v2:
- Add a few more patches to correct remaining problems
Simon Glass (15):
patman: Adjust 'command' to return strings instead of bytes
patman: Move to use Python 3
buildman: Convert to Python 3
test_fdt: Move to use Python 3
test_dtoc: Move to use Python 3
microcode_tool: Convert to Python 3
move_config: Convert to Python 3
rkmux: Convert to Python 3
pylibfdt: Convert to Python 3
pylibfdt: Sync up with upstream
pylibfdt: Correct the type for fdt_property_stub()
binman: Remember the pre-reset entry size
binman: Convert a few tests to Python 3
dtoc: Convert fdt.py to Python 3
binman: Move to use Python 3
scripts/dtc/pylibfdt/Makefile | 2 +-
scripts/dtc/pylibfdt/libfdt.i_shipped | 51 ++++++++++-----
scripts/dtc/pylibfdt/setup.py | 2 +-
tools/binman/binman.py | 2 +-
tools/binman/cbfs_util_test.py | 2 +-
tools/binman/entry.py | 25 +++----
tools/binman/entry_test.py | 15 -----
tools/binman/etype/intel_fit.py | 2 +-
tools/binman/ftest.py | 16 ++---
tools/buildman/board.py | 7 +-
tools/buildman/bsettings.py | 20 +++---
tools/buildman/builder.py | 45 +++++++------
tools/buildman/builderthread.py | 24 ++++---
tools/buildman/buildman.py | 10 +--
tools/buildman/control.py | 40 ++++++------
tools/buildman/func_test.py | 16 ++---
tools/buildman/test.py | 22 +++----
tools/buildman/toolchain.py | 93 ++++++++++++++-------------
tools/dtoc/dtoc.py | 2 +-
tools/dtoc/fdt.py | 17 ++---
tools/dtoc/test_dtoc.py | 1 +
tools/dtoc/test_fdt.py | 2 +-
tools/microcode-tool.py | 28 ++++----
tools/moveconfig.py | 82 +++++++++++------------
tools/patman/command.py | 31 ++++++---
tools/patman/patman.py | 2 +-
tools/patman/tools.py | 29 ++++++---
tools/rkmux.py | 16 ++---
28 files changed, 313 insertions(+), 291 deletions(-)
mode change 100644 => 100755 tools/dtoc/test_dtoc.py
--
2.24.0.rc0.303.g954a862665-goog
^ permalink raw reply [flat|nested] 23+ messages in thread
* [U-Boot] [PATCH v2 01/15] patman: Adjust 'command' to return strings instead of bytes
2019-10-25 1:03 [U-Boot] [PATCH v2 00/15] scripts: Convert to Python 3 Simon Glass
@ 2019-10-25 1:03 ` Simon Glass
2019-10-25 1:03 ` [U-Boot] [PATCH v2 02/15] patman: Move to use Python 3 Simon Glass
` (13 subsequent siblings)
14 siblings, 0 replies; 23+ messages in thread
From: Simon Glass @ 2019-10-25 1:03 UTC (permalink / raw)
To: u-boot
At present all the 'command' methods return bytes. Most of the time we
actually want strings, so change this. We still need to keep the internal
representation as bytes since otherwise unicode strings might break over
a read() boundary (e.g. 4KB), causing errors. But we can convert the end
result to strings.
Add a 'binary' parameter to cover the few cases where bytes are needed.
Signed-off-by: Simon Glass <sjg@chromium.org>
---
Changes in v2: None
tools/binman/cbfs_util_test.py | 2 +-
tools/binman/ftest.py | 2 +-
tools/patman/command.py | 31 +++++++++++++++++++++++--------
tools/patman/tools.py | 29 +++++++++++++++++++++--------
4 files changed, 46 insertions(+), 18 deletions(-)
diff --git a/tools/binman/cbfs_util_test.py b/tools/binman/cbfs_util_test.py
index 772c794eceb..ddc2e09e358 100755
--- a/tools/binman/cbfs_util_test.py
+++ b/tools/binman/cbfs_util_test.py
@@ -56,7 +56,7 @@ class TestCbfs(unittest.TestCase):
cls.have_lz4 = True
try:
tools.Run('lz4', '--no-frame-crc', '-c',
- tools.GetInputFilename('u-boot.bin'))
+ tools.GetInputFilename('u-boot.bin'), binary=True)
except:
cls.have_lz4 = False
diff --git a/tools/binman/ftest.py b/tools/binman/ftest.py
index 7000de9d420..88daaf20a87 100644
--- a/tools/binman/ftest.py
+++ b/tools/binman/ftest.py
@@ -170,7 +170,7 @@ class TestFunctional(unittest.TestCase):
cls.have_lz4 = True
try:
tools.Run('lz4', '--no-frame-crc', '-c',
- os.path.join(cls._indir, 'u-boot.bin'))
+ os.path.join(cls._indir, 'u-boot.bin'), binary=True)
except:
cls.have_lz4 = False
diff --git a/tools/patman/command.py b/tools/patman/command.py
index 16299f3f5be..5fbd2c4a3e9 100644
--- a/tools/patman/command.py
+++ b/tools/patman/command.py
@@ -4,6 +4,7 @@
import os
import cros_subprocess
+import tools
"""Shell command ease-ups for Python."""
@@ -31,6 +32,13 @@ class CommandResult:
self.return_code = return_code
self.exception = exception
+ def ToOutput(self, binary):
+ if not binary:
+ self.stdout = tools.ToString(self.stdout)
+ self.stderr = tools.ToString(self.stderr)
+ self.combined = tools.ToString(self.combined)
+ return self
+
# This permits interception of RunPipe for test purposes. If it is set to
# a function, then that function is called with the pipe list being
@@ -41,7 +49,7 @@ test_result = None
def RunPipe(pipe_list, infile=None, outfile=None,
capture=False, capture_stderr=False, oneline=False,
- raise_on_error=True, cwd=None, **kwargs):
+ raise_on_error=True, cwd=None, binary=False, **kwargs):
"""
Perform a command pipeline, with optional input/output filenames.
@@ -67,7 +75,7 @@ def RunPipe(pipe_list, infile=None, outfile=None,
else:
return test_result
# No result: fall through to normal processing
- result = CommandResult()
+ result = CommandResult(b'', b'', b'')
last_pipe = None
pipeline = list(pipe_list)
user_pipestr = '|'.join([' '.join(pipe) for pipe in pipe_list])
@@ -93,29 +101,36 @@ def RunPipe(pipe_list, infile=None, outfile=None,
if raise_on_error:
raise Exception("Error running '%s': %s" % (user_pipestr, str))
result.return_code = 255
- return result
+ return result.ToOutput(binary)
if capture:
result.stdout, result.stderr, result.combined = (
last_pipe.CommunicateFilter(None))
if result.stdout and oneline:
- result.output = result.stdout.rstrip('\r\n')
+ result.output = result.stdout.rstrip(b'\r\n')
result.return_code = last_pipe.wait()
else:
result.return_code = os.waitpid(last_pipe.pid, 0)[1]
if raise_on_error and result.return_code:
raise Exception("Error running '%s'" % user_pipestr)
- return result
+ return result.ToOutput(binary)
def Output(*cmd, **kwargs):
kwargs['raise_on_error'] = kwargs.get('raise_on_error', True)
return RunPipe([cmd], capture=True, **kwargs).stdout
def OutputOneLine(*cmd, **kwargs):
+ """Run a command and output it as a single-line string
+
+ The command us expected to produce a single line of output
+
+ Returns:
+ String containing output of command
+ """
raise_on_error = kwargs.pop('raise_on_error', True)
- return (RunPipe([cmd], capture=True, oneline=True,
- raise_on_error=raise_on_error,
- **kwargs).stdout.strip())
+ result = RunPipe([cmd], capture=True, oneline=True,
+ raise_on_error=raise_on_error, **kwargs).stdout.strip()
+ return result
def Run(*cmd, **kwargs):
return RunPipe([cmd], **kwargs).stdout
diff --git a/tools/patman/tools.py b/tools/patman/tools.py
index 4a7fcdad214..3feddb292fc 100644
--- a/tools/patman/tools.py
+++ b/tools/patman/tools.py
@@ -186,7 +186,7 @@ def PathHasFile(path_spec, fname):
return True
return False
-def Run(name, *args):
+def Run(name, *args, **kwargs):
"""Run a tool with some arguments
This runs a 'tool', which is a program used by binman to process files and
@@ -201,13 +201,14 @@ def Run(name, *args):
CommandResult object
"""
try:
+ binary = kwargs.get('binary')
env = None
if tool_search_paths:
env = dict(os.environ)
env['PATH'] = ':'.join(tool_search_paths) + ':' + env['PATH']
all_args = (name,) + args
result = command.RunPipe([all_args], capture=True, capture_stderr=True,
- env=env, raise_on_error=False)
+ env=env, raise_on_error=False, binary=binary)
if result.return_code:
raise Exception("Error %d running '%s': %s" %
(result.return_code,' '.join(all_args),
@@ -375,7 +376,7 @@ def ToBytes(string):
"""Convert a str type into a bytes type
Args:
- string: string to convert value
+ string: string to convert
Returns:
Python 3: A bytes type
@@ -385,6 +386,18 @@ def ToBytes(string):
return string.encode('utf-8')
return string
+def ToString(bval):
+ """Convert a bytes type into a str type
+
+ Args:
+ bval: bytes value to convert
+
+ Returns:
+ Python 3: A bytes type
+ Python 2: A string type
+ """
+ return bval.decode('utf-8')
+
def Compress(indata, algo, with_header=True):
"""Compress some data using a given algorithm
@@ -406,14 +419,14 @@ def Compress(indata, algo, with_header=True):
fname = GetOutputFilename('%s.comp.tmp' % algo)
WriteFile(fname, indata)
if algo == 'lz4':
- data = Run('lz4', '--no-frame-crc', '-c', fname)
+ data = Run('lz4', '--no-frame-crc', '-c', fname, binary=True)
# cbfstool uses a very old version of lzma
elif algo == 'lzma':
outfname = GetOutputFilename('%s.comp.otmp' % algo)
Run('lzma_alone', 'e', fname, outfname, '-lc1', '-lp0', '-pb0', '-d8')
data = ReadFile(outfname)
elif algo == 'gzip':
- data = Run('gzip', '-c', fname)
+ data = Run('gzip', '-c', fname, binary=True)
else:
raise ValueError("Unknown algorithm '%s'" % algo)
if with_header:
@@ -446,13 +459,13 @@ def Decompress(indata, algo, with_header=True):
with open(fname, 'wb') as fd:
fd.write(indata)
if algo == 'lz4':
- data = Run('lz4', '-dc', fname)
+ data = Run('lz4', '-dc', fname, binary=True)
elif algo == 'lzma':
outfname = GetOutputFilename('%s.decomp.otmp' % algo)
Run('lzma_alone', 'd', fname, outfname)
- data = ReadFile(outfname)
+ data = ReadFile(outfname, binary=True)
elif algo == 'gzip':
- data = Run('gzip', '-cd', fname)
+ data = Run('gzip', '-cd', fname, binary=True)
else:
raise ValueError("Unknown algorithm '%s'" % algo)
return data
--
2.24.0.rc0.303.g954a862665-goog
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [U-Boot] [PATCH v2 02/15] patman: Move to use Python 3
2019-10-25 1:03 [U-Boot] [PATCH v2 00/15] scripts: Convert to Python 3 Simon Glass
2019-10-25 1:03 ` [U-Boot] [PATCH v2 01/15] patman: Adjust 'command' to return strings instead of bytes Simon Glass
@ 2019-10-25 1:03 ` Simon Glass
2019-10-25 1:03 ` [U-Boot] [PATCH v2 03/15] buildman: Convert to " Simon Glass
` (12 subsequent siblings)
14 siblings, 0 replies; 23+ messages in thread
From: Simon Glass @ 2019-10-25 1:03 UTC (permalink / raw)
To: u-boot
Update this tool to use Python 3 to meet the 2020 deadline.
Signed-off-by: Simon Glass <sjg@chromium.org>
---
Changes in v2: None
tools/patman/patman.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tools/patman/patman.py b/tools/patman/patman.py
index 9605a36eff2..fe82f24c673 100755
--- a/tools/patman/patman.py
+++ b/tools/patman/patman.py
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
# SPDX-License-Identifier: GPL-2.0+
#
# Copyright (c) 2011 The Chromium OS Authors.
--
2.24.0.rc0.303.g954a862665-goog
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [U-Boot] [PATCH v2 03/15] buildman: Convert to Python 3
2019-10-25 1:03 [U-Boot] [PATCH v2 00/15] scripts: Convert to Python 3 Simon Glass
2019-10-25 1:03 ` [U-Boot] [PATCH v2 01/15] patman: Adjust 'command' to return strings instead of bytes Simon Glass
2019-10-25 1:03 ` [U-Boot] [PATCH v2 02/15] patman: Move to use Python 3 Simon Glass
@ 2019-10-25 1:03 ` Simon Glass
2019-10-30 1:47 ` Simon Glass
2019-10-25 1:03 ` [U-Boot] [PATCH v2 04/15] test_fdt: Move to use " Simon Glass
` (11 subsequent siblings)
14 siblings, 1 reply; 23+ messages in thread
From: Simon Glass @ 2019-10-25 1:03 UTC (permalink / raw)
To: u-boot
Convert buildman to Python 3 and make it use that, to meet the 2020
deadline.
Signed-off-by: Simon Glass <sjg@chromium.org>
---
Changes in v2: None
tools/buildman/board.py | 7 +--
tools/buildman/bsettings.py | 20 +++----
tools/buildman/builder.py | 45 ++++++++--------
tools/buildman/builderthread.py | 24 ++++-----
tools/buildman/buildman.py | 10 ++--
tools/buildman/control.py | 40 +++++++-------
tools/buildman/func_test.py | 16 +++---
tools/buildman/test.py | 22 ++++----
tools/buildman/toolchain.py | 93 +++++++++++++++++----------------
9 files changed, 139 insertions(+), 138 deletions(-)
diff --git a/tools/buildman/board.py b/tools/buildman/board.py
index 2a1d021574c..0060ac36cc3 100644
--- a/tools/buildman/board.py
+++ b/tools/buildman/board.py
@@ -1,6 +1,7 @@
# SPDX-License-Identifier: GPL-2.0+
# Copyright (c) 2012 The Chromium OS Authors.
+from collections import OrderedDict
import re
class Expr:
@@ -155,7 +156,7 @@ class Boards:
key is board.target
value is board
"""
- board_dict = {}
+ board_dict = OrderedDict()
for board in self._boards:
board_dict[board.target] = board
return board_dict
@@ -166,7 +167,7 @@ class Boards:
Returns:
List of Board objects that are marked selected
"""
- board_dict = {}
+ board_dict = OrderedDict()
for board in self._boards:
if board.build_it:
board_dict[board.target] = board
@@ -259,7 +260,7 @@ class Boards:
due to each argument, arranged by argument.
List of errors found
"""
- result = {}
+ result = OrderedDict()
warnings = []
terms = self._BuildTerms(args)
diff --git a/tools/buildman/bsettings.py b/tools/buildman/bsettings.py
index 03d7439aa54..0b7208da373 100644
--- a/tools/buildman/bsettings.py
+++ b/tools/buildman/bsettings.py
@@ -1,9 +1,9 @@
# SPDX-License-Identifier: GPL-2.0+
# Copyright (c) 2012 The Chromium OS Authors.
-import ConfigParser
+import configparser
import os
-import StringIO
+import io
def Setup(fname=''):
@@ -15,20 +15,20 @@ def Setup(fname=''):
global settings
global config_fname
- settings = ConfigParser.SafeConfigParser()
+ settings = configparser.SafeConfigParser()
if fname is not None:
config_fname = fname
if config_fname == '':
config_fname = '%s/.buildman' % os.getenv('HOME')
if not os.path.exists(config_fname):
- print 'No config file found ~/.buildman\nCreating one...\n'
+ print('No config file found ~/.buildman\nCreating one...\n')
CreateBuildmanConfigFile(config_fname)
- print 'To install tool chains, please use the --fetch-arch option'
+ print('To install tool chains, please use the --fetch-arch option')
if config_fname:
settings.read(config_fname)
def AddFile(data):
- settings.readfp(StringIO.StringIO(data))
+ settings.readfp(io.StringIO(data))
def GetItems(section):
"""Get the items from a section of the config.
@@ -41,7 +41,7 @@ def GetItems(section):
"""
try:
return settings.items(section)
- except ConfigParser.NoSectionError as e:
+ except configparser.NoSectionError as e:
return []
except:
raise
@@ -68,10 +68,10 @@ def CreateBuildmanConfigFile(config_fname):
try:
f = open(config_fname, 'w')
except IOError:
- print "Couldn't create buildman config file '%s'\n" % config_fname
+ print("Couldn't create buildman config file '%s'\n" % config_fname)
raise
- print >>f, '''[toolchain]
+ print('''[toolchain]
# name = path
# e.g. x86 = /opt/gcc-4.6.3-nolibc/x86_64-linux
@@ -93,5 +93,5 @@ openrisc = or1k
# snapper-boards=ENABLE_AT91_TEST=1
# snapper9260=${snapper-boards} BUILD_TAG=442
# snapper9g45=${snapper-boards} BUILD_TAG=443
-'''
+''', file=f)
f.close();
diff --git a/tools/buildman/builder.py b/tools/buildman/builder.py
index fbb236676c7..3018867c39a 100644
--- a/tools/buildman/builder.py
+++ b/tools/buildman/builder.py
@@ -9,7 +9,7 @@ from datetime import datetime, timedelta
import glob
import os
import re
-import Queue
+import queue
import shutil
import signal
import string
@@ -92,11 +92,10 @@ u-boot/ source directory
"""
# Possible build outcomes
-OUTCOME_OK, OUTCOME_WARNING, OUTCOME_ERROR, OUTCOME_UNKNOWN = range(4)
+OUTCOME_OK, OUTCOME_WARNING, OUTCOME_ERROR, OUTCOME_UNKNOWN = list(range(4))
# Translate a commit subject into a valid filename (and handle unicode)
-trans_valid_chars = string.maketrans('/: ', '---')
-trans_valid_chars = trans_valid_chars.decode('latin-1')
+trans_valid_chars = str.maketrans('/: ', '---')
BASE_CONFIG_FILENAMES = [
'u-boot.cfg', 'u-boot-spl.cfg', 'u-boot-tpl.cfg'
@@ -122,8 +121,8 @@ class Config:
def __hash__(self):
val = 0
for fname in self.config:
- for key, value in self.config[fname].iteritems():
- print key, value
+ for key, value in self.config[fname].items():
+ print(key, value)
val = val ^ hash(key) & hash(value)
return val
@@ -293,8 +292,8 @@ class Builder:
self._re_dtb_warning = re.compile('(.*): Warning .*')
self._re_note = re.compile('(.*):(\d*):(\d*): note: this is the location of the previous.*')
- self.queue = Queue.Queue()
- self.out_queue = Queue.Queue()
+ self.queue = queue.Queue()
+ self.out_queue = queue.Queue()
for i in range(self.num_threads):
t = builderthread.BuilderThread(self, i, incremental,
per_board_out_dir)
@@ -781,7 +780,7 @@ class Builder:
config = {}
environment = {}
- for board in boards_selected.itervalues():
+ for board in boards_selected.values():
outcome = self.GetBuildOutcome(commit_upto, board.target,
read_func_sizes, read_config,
read_environment)
@@ -814,13 +813,13 @@ class Builder:
tconfig = Config(self.config_filenames, board.target)
for fname in self.config_filenames:
if outcome.config:
- for key, value in outcome.config[fname].iteritems():
+ for key, value in outcome.config[fname].items():
tconfig.Add(fname, key, value)
config[board.target] = tconfig
tenvironment = Environment(board.target)
if outcome.environment:
- for key, value in outcome.environment.iteritems():
+ for key, value in outcome.environment.items():
tenvironment.Add(key, value)
environment[board.target] = tenvironment
@@ -1040,12 +1039,12 @@ class Builder:
# We now have a list of image size changes sorted by arch
# Print out a summary of these
- for arch, target_list in arch_list.iteritems():
+ for arch, target_list in arch_list.items():
# Get total difference for each type
totals = {}
for result in target_list:
total = 0
- for name, diff in result.iteritems():
+ for name, diff in result.items():
if name.startswith('_'):
continue
total += diff
@@ -1250,7 +1249,7 @@ class Builder:
if self._show_unknown:
self.AddOutcome(board_selected, arch_list, unknown_boards, '?',
self.col.MAGENTA)
- for arch, target_list in arch_list.iteritems():
+ for arch, target_list in arch_list.items():
Print('%10s: %s' % (arch, target_list))
self._error_lines += 1
if better_err:
@@ -1283,13 +1282,13 @@ class Builder:
environment_minus = {}
environment_change = {}
base = tbase.environment
- for key, value in tenvironment.environment.iteritems():
+ for key, value in tenvironment.environment.items():
if key not in base:
environment_plus[key] = value
- for key, value in base.iteritems():
+ for key, value in base.items():
if key not in tenvironment.environment:
environment_minus[key] = value
- for key, value in base.iteritems():
+ for key, value in base.items():
new_value = tenvironment.environment.get(key)
if new_value and value != new_value:
desc = '%s -> %s' % (value, new_value)
@@ -1342,15 +1341,15 @@ class Builder:
config_minus = {}
config_change = {}
base = tbase.config[name]
- for key, value in tconfig.config[name].iteritems():
+ for key, value in tconfig.config[name].items():
if key not in base:
config_plus[key] = value
all_config_plus[key] = value
- for key, value in base.iteritems():
+ for key, value in base.items():
if key not in tconfig.config[name]:
config_minus[key] = value
all_config_minus[key] = value
- for key, value in base.iteritems():
+ for key, value in base.items():
new_value = tconfig.config.get(key)
if new_value and value != new_value:
desc = '%s -> %s' % (value, new_value)
@@ -1368,7 +1367,7 @@ class Builder:
summary[target] = '\n'.join(lines)
lines_by_target = {}
- for target, lines in summary.iteritems():
+ for target, lines in summary.items():
if lines in lines_by_target:
lines_by_target[lines].append(target)
else:
@@ -1392,7 +1391,7 @@ class Builder:
Print('%s:' % arch)
_OutputConfigInfo(lines)
- for lines, targets in lines_by_target.iteritems():
+ for lines, targets in lines_by_target.items():
if not lines:
continue
Print('%s :' % ' '.join(sorted(targets)))
@@ -1566,7 +1565,7 @@ class Builder:
self.ProcessResult(None)
# Create jobs to build all commits for each board
- for brd in board_selected.itervalues():
+ for brd in board_selected.values():
job = builderthread.BuilderJob()
job.board = brd
job.commits = commits
diff --git a/tools/buildman/builderthread.py b/tools/buildman/builderthread.py
index 8a9d47cd5e4..570c1f6595c 100644
--- a/tools/buildman/builderthread.py
+++ b/tools/buildman/builderthread.py
@@ -28,7 +28,7 @@ def Mkdir(dirname, parents = False):
except OSError as err:
if err.errno == errno.EEXIST:
if os.path.realpath('.') == os.path.realpath(dirname):
- print "Cannot create the current working directory '%s'!" % dirname
+ print("Cannot create the current working directory '%s'!" % dirname)
sys.exit(1)
pass
else:
@@ -291,15 +291,13 @@ class BuilderThread(threading.Thread):
outfile = os.path.join(build_dir, 'log')
with open(outfile, 'w') as fd:
if result.stdout:
- # We don't want unicode characters in log files
- fd.write(result.stdout.decode('UTF-8').encode('ASCII', 'replace'))
+ fd.write(result.stdout)
errfile = self.builder.GetErrFile(result.commit_upto,
result.brd.target)
if result.stderr:
with open(errfile, 'w') as fd:
- # We don't want unicode characters in log files
- fd.write(result.stderr.decode('UTF-8').encode('ASCII', 'replace'))
+ fd.write(result.stderr)
elif os.path.exists(errfile):
os.remove(errfile)
@@ -314,17 +312,17 @@ class BuilderThread(threading.Thread):
else:
fd.write('%s' % result.return_code)
with open(os.path.join(build_dir, 'toolchain'), 'w') as fd:
- print >>fd, 'gcc', result.toolchain.gcc
- print >>fd, 'path', result.toolchain.path
- print >>fd, 'cross', result.toolchain.cross
- print >>fd, 'arch', result.toolchain.arch
+ print('gcc', result.toolchain.gcc, file=fd)
+ print('path', result.toolchain.path, file=fd)
+ print('cross', result.toolchain.cross, file=fd)
+ print('arch', result.toolchain.arch, file=fd)
fd.write('%s' % result.return_code)
# Write out the image and function size information and an objdump
env = result.toolchain.MakeEnvironment(self.builder.full_path)
with open(os.path.join(build_dir, 'env'), 'w') as fd:
for var in sorted(env.keys()):
- print >>fd, '%s="%s"' % (var, env[var])
+ print('%s="%s"' % (var, env[var]), file=fd)
lines = []
for fname in ['u-boot', 'spl/u-boot-spl']:
cmd = ['%snm' % self.toolchain.cross, '--size-sort', fname]
@@ -335,7 +333,7 @@ class BuilderThread(threading.Thread):
nm = self.builder.GetFuncSizesFile(result.commit_upto,
result.brd.target, fname)
with open(nm, 'w') as fd:
- print >>fd, nm_result.stdout,
+ print(nm_result.stdout, end=' ', file=fd)
cmd = ['%sobjdump' % self.toolchain.cross, '-h', fname]
dump_result = command.RunPipe([cmd], capture=True,
@@ -346,7 +344,7 @@ class BuilderThread(threading.Thread):
objdump = self.builder.GetObjdumpFile(result.commit_upto,
result.brd.target, fname)
with open(objdump, 'w') as fd:
- print >>fd, dump_result.stdout,
+ print(dump_result.stdout, end=' ', file=fd)
for line in dump_result.stdout.splitlines():
fields = line.split()
if len(fields) > 5 and fields[1] == '.rodata':
@@ -378,7 +376,7 @@ class BuilderThread(threading.Thread):
sizes = self.builder.GetSizesFile(result.commit_upto,
result.brd.target)
with open(sizes, 'w') as fd:
- print >>fd, '\n'.join(lines)
+ print('\n'.join(lines), file=fd)
# Write out the configuration files, with a special case for SPL
for dirname in ['', 'spl', 'tpl']:
diff --git a/tools/buildman/buildman.py b/tools/buildman/buildman.py
index f17aa15e7c5..30a8690f935 100755
--- a/tools/buildman/buildman.py
+++ b/tools/buildman/buildman.py
@@ -1,4 +1,4 @@
-#!/usr/bin/env python2
+#!/usr/bin/env python3
# SPDX-License-Identifier: GPL-2.0+
#
# Copyright (c) 2012 The Chromium OS Authors.
@@ -6,6 +6,8 @@
"""See README for more information"""
+from __future__ import print_function
+
import multiprocessing
import os
import re
@@ -46,11 +48,11 @@ def RunTests(skip_net_tests):
suite = unittest.TestLoader().loadTestsFromTestCase(module)
suite.run(result)
- print result
+ print(result)
for test, err in result.errors:
- print err
+ print(err)
for test, err in result.failures:
- print err
+ print(err)
options, args = cmdline.ParseArgs()
diff --git a/tools/buildman/control.py b/tools/buildman/control.py
index fcf531c5f14..8905dd3e528 100644
--- a/tools/buildman/control.py
+++ b/tools/buildman/control.py
@@ -59,31 +59,31 @@ def ShowActions(series, why_selected, boards_selected, builder, options,
board_warnings: List of warnings obtained from board selected
"""
col = terminal.Color()
- print 'Dry run, so not doing much. But I would do this:'
- print
+ print('Dry run, so not doing much. But I would do this:')
+ print()
if series:
commits = series.commits
else:
commits = None
- print GetActionSummary(False, commits, boards_selected,
- options)
- print 'Build directory: %s' % builder.base_dir
+ print(GetActionSummary(False, commits, boards_selected,
+ options))
+ print('Build directory: %s' % builder.base_dir)
if commits:
for upto in range(0, len(series.commits), options.step):
commit = series.commits[upto]
- print ' ', col.Color(col.YELLOW, commit.hash[:8], bright=False),
- print commit.subject
- print
+ print(' ', col.Color(col.YELLOW, commit.hash[:8], bright=False), end=' ')
+ print(commit.subject)
+ print()
for arg in why_selected:
if arg != 'all':
- print arg, ': %d boards' % len(why_selected[arg])
+ print(arg, ': %d boards' % len(why_selected[arg]))
if options.verbose:
- print ' %s' % ' '.join(why_selected[arg])
- print ('Total boards to build for each commit: %d\n' %
- len(why_selected['all']))
+ print(' %s' % ' '.join(why_selected[arg]))
+ print(('Total boards to build for each commit: %d\n' %
+ len(why_selected['all'])))
if board_warnings:
for warning in board_warnings:
- print col.Color(col.YELLOW, warning)
+ print(col.Color(col.YELLOW, warning))
def CheckOutputDir(output_dir):
"""Make sure that the output directory is not within the current directory
@@ -146,17 +146,17 @@ def DoBuildman(options, args, toolchains=None, make_func=None, boards=None,
if options.fetch_arch:
if options.fetch_arch == 'list':
sorted_list = toolchains.ListArchs()
- print col.Color(col.BLUE, 'Available architectures: %s\n' %
- ' '.join(sorted_list))
+ print(col.Color(col.BLUE, 'Available architectures: %s\n' %
+ ' '.join(sorted_list)))
return 0
else:
fetch_arch = options.fetch_arch
if fetch_arch == 'all':
fetch_arch = ','.join(toolchains.ListArchs())
- print col.Color(col.CYAN, '\nDownloading toolchains: %s' %
- fetch_arch)
+ print(col.Color(col.CYAN, '\nDownloading toolchains: %s' %
+ fetch_arch))
for arch in fetch_arch.split(','):
- print
+ print()
ret = toolchains.FetchAndInstall(arch)
if ret:
return ret
@@ -167,7 +167,7 @@ def DoBuildman(options, args, toolchains=None, make_func=None, boards=None,
toolchains.Scan(options.list_tool_chains and options.verbose)
if options.list_tool_chains:
toolchains.List()
- print
+ print()
return 0
# Work out how many commits to build. We want to build everything on the
@@ -191,7 +191,7 @@ def DoBuildman(options, args, toolchains=None, make_func=None, boards=None,
sys.exit(col.Color(col.RED, "Range '%s' has no commits" %
options.branch))
if msg:
- print col.Color(col.YELLOW, msg)
+ print(col.Color(col.YELLOW, msg))
count += 1 # Build upstream commit also
if not count:
diff --git a/tools/buildman/func_test.py b/tools/buildman/func_test.py
index f90b8ea7f5d..4c3d497294d 100644
--- a/tools/buildman/func_test.py
+++ b/tools/buildman/func_test.py
@@ -270,7 +270,7 @@ class TestFunctional(unittest.TestCase):
stdout=''.join(commit_log[:count]))
# Not handled, so abort
- print 'git log', args
+ print('git log', args)
sys.exit(1)
def _HandleCommandGitConfig(self, args):
@@ -286,7 +286,7 @@ class TestFunctional(unittest.TestCase):
stdout='refs/heads/master\n')
# Not handled, so abort
- print 'git config', args
+ print('git config', args)
sys.exit(1)
def _HandleCommandGit(self, in_args):
@@ -320,7 +320,7 @@ class TestFunctional(unittest.TestCase):
return command.CommandResult(return_code=0)
# Not handled, so abort
- print 'git', git_args, sub_cmd, args
+ print('git', git_args, sub_cmd, args)
sys.exit(1)
def _HandleCommandNm(self, args):
@@ -351,7 +351,7 @@ class TestFunctional(unittest.TestCase):
if pipe_list[1] == ['wc', '-l']:
wc = True
else:
- print 'invalid pipe', kwargs
+ print('invalid pipe', kwargs)
sys.exit(1)
cmd = pipe_list[0][0]
args = pipe_list[0][1:]
@@ -371,7 +371,7 @@ class TestFunctional(unittest.TestCase):
if not result:
# Not handled, so abort
- print 'unknown command', kwargs
+ print('unknown command', kwargs)
sys.exit(1)
if wc:
@@ -404,14 +404,14 @@ class TestFunctional(unittest.TestCase):
return command.CommandResult(return_code=0)
# Not handled, so abort
- print 'make', stage
+ print('make', stage)
sys.exit(1)
# Example function to print output lines
def print_lines(self, lines):
- print len(lines)
+ print(len(lines))
for line in lines:
- print line
+ print(line)
#self.print_lines(terminal.GetPrintTestLines())
def testNoBoards(self):
diff --git a/tools/buildman/test.py b/tools/buildman/test.py
index de02f61be65..9e7336b419b 100644
--- a/tools/buildman/test.py
+++ b/tools/buildman/test.py
@@ -220,11 +220,11 @@ class TestBuild(unittest.TestCase):
self.assertEqual(lines[1].text, '02: %s' % commits[1][1])
col = terminal.Color()
- self.assertSummary(lines[2].text, 'sandbox', 'w+', ['board4'],
+ self.assertSummary(lines[2].text, 'arm', 'w+', ['board1'],
outcome=OUTCOME_WARN)
- self.assertSummary(lines[3].text, 'arm', 'w+', ['board1'],
+ self.assertSummary(lines[3].text, 'powerpc', 'w+', ['board2', 'board3'],
outcome=OUTCOME_WARN)
- self.assertSummary(lines[4].text, 'powerpc', 'w+', ['board2', 'board3'],
+ self.assertSummary(lines[4].text, 'sandbox', 'w+', ['board4'],
outcome=OUTCOME_WARN)
# Second commit: The warnings should be listed
@@ -234,10 +234,10 @@ class TestBuild(unittest.TestCase):
# Third commit: Still fails
self.assertEqual(lines[6].text, '03: %s' % commits[2][1])
- self.assertSummary(lines[7].text, 'sandbox', '+', ['board4'])
- self.assertSummary(lines[8].text, 'arm', '', ['board1'],
+ self.assertSummary(lines[7].text, 'arm', '', ['board1'],
outcome=OUTCOME_OK)
- self.assertSummary(lines[9].text, 'powerpc', '+', ['board2', 'board3'])
+ self.assertSummary(lines[8].text, 'powerpc', '+', ['board2', 'board3'])
+ self.assertSummary(lines[9].text, 'sandbox', '+', ['board4'])
# Expect a compiler error
self.assertEqual(lines[10].text, '+%s' %
@@ -245,8 +245,6 @@ class TestBuild(unittest.TestCase):
# Fourth commit: Compile errors are fixed, just have warning for board3
self.assertEqual(lines[11].text, '04: %s' % commits[3][1])
- self.assertSummary(lines[12].text, 'sandbox', 'w+', ['board4'],
- outcome=OUTCOME_WARN)
expect = '%10s: ' % 'powerpc'
expect += ' ' + col.Color(col.GREEN, '')
expect += ' '
@@ -254,7 +252,9 @@ class TestBuild(unittest.TestCase):
expect += ' ' + col.Color(col.YELLOW, 'w+')
expect += ' '
expect += col.Color(col.YELLOW, ' %s' % 'board3')
- self.assertEqual(lines[13].text, expect)
+ self.assertEqual(lines[12].text, expect)
+ self.assertSummary(lines[13].text, 'sandbox', 'w+', ['board4'],
+ outcome=OUTCOME_WARN)
# Compile error fixed
self.assertEqual(lines[14].text, '-%s' %
@@ -267,9 +267,9 @@ class TestBuild(unittest.TestCase):
# Fifth commit
self.assertEqual(lines[16].text, '05: %s' % commits[4][1])
- self.assertSummary(lines[17].text, 'sandbox', '+', ['board4'])
- self.assertSummary(lines[18].text, 'powerpc', '', ['board3'],
+ self.assertSummary(lines[17].text, 'powerpc', '', ['board3'],
outcome=OUTCOME_OK)
+ self.assertSummary(lines[18].text, 'sandbox', '+', ['board4'])
# The second line of errors[3] is a duplicate, so buildman will drop it
expect = errors[3].rstrip().split('\n')
diff --git a/tools/buildman/toolchain.py b/tools/buildman/toolchain.py
index a65737fdf84..fbe57ca6f58 100644
--- a/tools/buildman/toolchain.py
+++ b/tools/buildman/toolchain.py
@@ -4,18 +4,19 @@
import re
import glob
-from HTMLParser import HTMLParser
+from html.parser import HTMLParser
import os
import sys
import tempfile
-import urllib2
+import urllib.request, urllib.error, urllib.parse
import bsettings
import command
import terminal
+import tools
(PRIORITY_FULL_PREFIX, PRIORITY_PREFIX_GCC, PRIORITY_PREFIX_GCC_PATH,
- PRIORITY_CALC) = range(4)
+ PRIORITY_CALC) = list(range(4))
# Simple class to collect links from a page
class MyHTMLParser(HTMLParser):
@@ -100,15 +101,15 @@ class Toolchain:
raise_on_error=False)
self.ok = result.return_code == 0
if verbose:
- print 'Tool chain test: ',
+ print('Tool chain test: ', end=' ')
if self.ok:
- print "OK, arch='%s', priority %d" % (self.arch,
- self.priority)
+ print("OK, arch='%s', priority %d" % (self.arch,
+ self.priority))
else:
- print 'BAD'
- print 'Command: ', cmd
- print result.stdout
- print result.stderr
+ print('BAD')
+ print('Command: ', cmd)
+ print(result.stdout)
+ print(result.stderr)
else:
self.ok = True
@@ -138,7 +139,7 @@ class Toolchain:
value = ''
for name, value in bsettings.GetItems('toolchain-wrapper'):
if not value:
- print "Warning: Wrapper not found"
+ print("Warning: Wrapper not found")
if value:
value = value + ' '
@@ -227,11 +228,11 @@ class Toolchains:
"""
toolchains = bsettings.GetItems('toolchain')
if show_warning and not toolchains:
- print ("Warning: No tool chains. Please run 'buildman "
+ print(("Warning: No tool chains. Please run 'buildman "
"--fetch-arch all' to download all available toolchains, or "
"add a [toolchain] section to your buildman config file "
"%s. See README for details" %
- bsettings.config_fname)
+ bsettings.config_fname))
paths = []
for name, value in toolchains:
@@ -272,10 +273,10 @@ class Toolchains:
if add_it:
self.toolchains[toolchain.arch] = toolchain
elif verbose:
- print ("Toolchain '%s' at priority %d will be ignored because "
+ print(("Toolchain '%s' at priority %d will be ignored because "
"another toolchain for arch '%s' has priority %d" %
(toolchain.gcc, toolchain.priority, toolchain.arch,
- self.toolchains[toolchain.arch].priority))
+ self.toolchains[toolchain.arch].priority)))
def ScanPath(self, path, verbose):
"""Scan a path for a valid toolchain
@@ -289,9 +290,9 @@ class Toolchains:
fnames = []
for subdir in ['.', 'bin', 'usr/bin']:
dirname = os.path.join(path, subdir)
- if verbose: print " - looking in '%s'" % dirname
+ if verbose: print(" - looking in '%s'" % dirname)
for fname in glob.glob(dirname + '/*gcc'):
- if verbose: print " - found '%s'" % fname
+ if verbose: print(" - found '%s'" % fname)
fnames.append(fname)
return fnames
@@ -321,9 +322,9 @@ class Toolchains:
Args:
verbose: True to print out progress information
"""
- if verbose: print 'Scanning for tool chains'
+ if verbose: print('Scanning for tool chains')
for name, value in self.prefixes:
- if verbose: print " - scanning prefix '%s'" % value
+ if verbose: print(" - scanning prefix '%s'" % value)
if os.path.exists(value):
self.Add(value, True, verbose, PRIORITY_FULL_PREFIX, name)
continue
@@ -335,10 +336,10 @@ class Toolchains:
for f in fname_list:
self.Add(f, True, verbose, PRIORITY_PREFIX_GCC_PATH, name)
if not fname_list:
- raise ValueError, ("No tool chain found for prefix '%s'" %
+ raise ValueError("No tool chain found for prefix '%s'" %
value)
for path in self.paths:
- if verbose: print " - scanning path '%s'" % path
+ if verbose: print(" - scanning path '%s'" % path)
fnames = self.ScanPath(path, verbose)
for fname in fnames:
self.Add(fname, True, verbose)
@@ -346,13 +347,13 @@ class Toolchains:
def List(self):
"""List out the selected toolchains for each architecture"""
col = terminal.Color()
- print col.Color(col.BLUE, 'List of available toolchains (%d):' %
- len(self.toolchains))
+ print(col.Color(col.BLUE, 'List of available toolchains (%d):' %
+ len(self.toolchains)))
if len(self.toolchains):
- for key, value in sorted(self.toolchains.iteritems()):
- print '%-10s: %s' % (key, value.gcc)
+ for key, value in sorted(self.toolchains.items()):
+ print('%-10s: %s' % (key, value.gcc))
else:
- print 'None'
+ print('None')
def Select(self, arch):
"""Returns the toolchain for a given architecture
@@ -370,7 +371,7 @@ class Toolchains:
return self.toolchains[alias]
if not arch in self.toolchains:
- raise ValueError, ("No tool chain found for arch '%s'" % arch)
+ raise ValueError("No tool chain found for arch '%s'" % arch)
return self.toolchains[arch]
def ResolveReferences(self, var_dict, args):
@@ -464,9 +465,9 @@ class Toolchains:
links = []
for version in versions:
url = '%s/%s/%s/' % (base, arch, version)
- print 'Checking: %s' % url
- response = urllib2.urlopen(url)
- html = response.read()
+ print('Checking: %s' % url)
+ response = urllib.request.urlopen(url)
+ html = tools.ToString(response.read())
parser = MyHTMLParser(fetch_arch)
parser.feed(html)
if fetch_arch == 'list':
@@ -488,10 +489,10 @@ class Toolchains:
Full path to the downloaded archive file in that directory,
or None if there was an error while downloading
"""
- print 'Downloading: %s' % url
+ print('Downloading: %s' % url)
leaf = url.split('/')[-1]
tmpdir = tempfile.mkdtemp('.buildman')
- response = urllib2.urlopen(url)
+ response = urllib.request.urlopen(url)
fname = os.path.join(tmpdir, leaf)
fd = open(fname, 'wb')
meta = response.info()
@@ -504,7 +505,7 @@ class Toolchains:
while True:
buffer = response.read(block_size)
if not buffer:
- print chr(8) * (len(status) + 1), '\r',
+ print(chr(8) * (len(status) + 1), '\r', end=' ')
break
done += len(buffer)
@@ -512,11 +513,11 @@ class Toolchains:
status = r'%10d MiB [%3d%%]' % (done / 1024 / 1024,
done * 100 / size)
status = status + chr(8) * (len(status) + 1)
- print status,
+ print(status, end=' ')
sys.stdout.flush()
fd.close()
if done != size:
- print 'Error, failed to download'
+ print('Error, failed to download')
os.remove(fname)
fname = None
return tmpdir, fname
@@ -565,11 +566,11 @@ class Toolchains:
"""
# Fist get the URL for this architecture
col = terminal.Color()
- print col.Color(col.BLUE, "Downloading toolchain for arch '%s'" % arch)
+ print(col.Color(col.BLUE, "Downloading toolchain for arch '%s'" % arch))
url = self.LocateArchUrl(arch)
if not url:
- print ("Cannot find toolchain for arch '%s' - use 'list' to list" %
- arch)
+ print(("Cannot find toolchain for arch '%s' - use 'list' to list" %
+ arch))
return 2
home = os.environ['HOME']
dest = os.path.join(home, '.buildman-toolchains')
@@ -580,28 +581,28 @@ class Toolchains:
tmpdir, tarfile = self.Download(url)
if not tarfile:
return 1
- print col.Color(col.GREEN, 'Unpacking to: %s' % dest),
+ print(col.Color(col.GREEN, 'Unpacking to: %s' % dest), end=' ')
sys.stdout.flush()
path = self.Unpack(tarfile, dest)
os.remove(tarfile)
os.rmdir(tmpdir)
- print
+ print()
# Check that the toolchain works
- print col.Color(col.GREEN, 'Testing')
+ print(col.Color(col.GREEN, 'Testing'))
dirpath = os.path.join(dest, path)
compiler_fname_list = self.ScanPath(dirpath, True)
if not compiler_fname_list:
- print 'Could not locate C compiler - fetch failed.'
+ print('Could not locate C compiler - fetch failed.')
return 1
if len(compiler_fname_list) != 1:
- print col.Color(col.RED, 'Warning, ambiguous toolchains: %s' %
- ', '.join(compiler_fname_list))
+ print(col.Color(col.RED, 'Warning, ambiguous toolchains: %s' %
+ ', '.join(compiler_fname_list)))
toolchain = Toolchain(compiler_fname_list[0], True, True)
# Make sure that it will be found by buildman
if not self.TestSettingsHasPath(dirpath):
- print ("Adding 'download' to config file '%s'" %
- bsettings.config_fname)
+ print(("Adding 'download' to config file '%s'" %
+ bsettings.config_fname))
bsettings.SetItem('toolchain', 'download', '%s/*/*' % dest)
return 0
--
2.24.0.rc0.303.g954a862665-goog
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [U-Boot] [PATCH v2 04/15] test_fdt: Move to use Python 3
2019-10-25 1:03 [U-Boot] [PATCH v2 00/15] scripts: Convert to Python 3 Simon Glass
` (2 preceding siblings ...)
2019-10-25 1:03 ` [U-Boot] [PATCH v2 03/15] buildman: Convert to " Simon Glass
@ 2019-10-25 1:03 ` Simon Glass
2019-10-25 1:03 ` [U-Boot] [PATCH v2 05/15] test_dtoc: " Simon Glass
` (10 subsequent siblings)
14 siblings, 0 replies; 23+ messages in thread
From: Simon Glass @ 2019-10-25 1:03 UTC (permalink / raw)
To: u-boot
Update this test to use Python 3 to meet the 2020 deadline.
Signed-off-by: Simon Glass <sjg@chromium.org>
---
Changes in v2: None
tools/dtoc/test_fdt.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tools/dtoc/test_fdt.py b/tools/dtoc/test_fdt.py
index 028c8cbaa80..3316757e61e 100755
--- a/tools/dtoc/test_fdt.py
+++ b/tools/dtoc/test_fdt.py
@@ -1,4 +1,4 @@
-#!/usr/bin/python
+#!/usr/bin/env python3
# SPDX-License-Identifier: GPL-2.0+
# Copyright (c) 2018 Google, Inc
# Written by Simon Glass <sjg@chromium.org>
--
2.24.0.rc0.303.g954a862665-goog
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [U-Boot] [PATCH v2 05/15] test_dtoc: Move to use Python 3
2019-10-25 1:03 [U-Boot] [PATCH v2 00/15] scripts: Convert to Python 3 Simon Glass
` (3 preceding siblings ...)
2019-10-25 1:03 ` [U-Boot] [PATCH v2 04/15] test_fdt: Move to use " Simon Glass
@ 2019-10-25 1:03 ` Simon Glass
2019-10-25 1:03 ` [U-Boot] [PATCH v2 06/15] microcode_tool: Convert to " Simon Glass
` (9 subsequent siblings)
14 siblings, 0 replies; 23+ messages in thread
From: Simon Glass @ 2019-10-25 1:03 UTC (permalink / raw)
To: u-boot
Update this test to use Python 3 to meet the 2020 deadline.
Also make it executable while we are here.
Signed-off-by: Simon Glass <sjg@chromium.org>
---
Changes in v2: None
tools/dtoc/dtoc.py | 2 +-
tools/dtoc/test_dtoc.py | 1 +
2 files changed, 2 insertions(+), 1 deletion(-)
mode change 100644 => 100755 tools/dtoc/test_dtoc.py
diff --git a/tools/dtoc/dtoc.py b/tools/dtoc/dtoc.py
index 514e0dd4a34..b3596a5918f 100755
--- a/tools/dtoc/dtoc.py
+++ b/tools/dtoc/dtoc.py
@@ -1,4 +1,4 @@
-#!/usr/bin/env python2
+#!/usr/bin/env python3
# SPDX-License-Identifier: GPL-2.0+
#
# Copyright (C) 2016 Google, Inc
diff --git a/tools/dtoc/test_dtoc.py b/tools/dtoc/test_dtoc.py
old mode 100644
new mode 100755
index b915b278560..d733b706558
--- a/tools/dtoc/test_dtoc.py
+++ b/tools/dtoc/test_dtoc.py
@@ -1,3 +1,4 @@
+#!/usr/bin/env python3
# SPDX-License-Identifier: GPL-2.0+
# Copyright (c) 2012 The Chromium OS Authors.
#
--
2.24.0.rc0.303.g954a862665-goog
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [U-Boot] [PATCH v2 06/15] microcode_tool: Convert to Python 3
2019-10-25 1:03 [U-Boot] [PATCH v2 00/15] scripts: Convert to Python 3 Simon Glass
` (4 preceding siblings ...)
2019-10-25 1:03 ` [U-Boot] [PATCH v2 05/15] test_dtoc: " Simon Glass
@ 2019-10-25 1:03 ` Simon Glass
2019-10-25 1:03 ` [U-Boot] [PATCH v2 07/15] move_config: " Simon Glass
` (8 subsequent siblings)
14 siblings, 0 replies; 23+ messages in thread
From: Simon Glass @ 2019-10-25 1:03 UTC (permalink / raw)
To: u-boot
Convert this tool to Python 3 and make it use that, to meet the 2020
deadline.
Signed-off-by: Simon Glass <sjg@chromium.org>
---
Changes in v2: None
tools/microcode-tool.py | 28 ++++++++++++++--------------
1 file changed, 14 insertions(+), 14 deletions(-)
diff --git a/tools/microcode-tool.py b/tools/microcode-tool.py
index 249a33b8cac..24c02c4fca1 100755
--- a/tools/microcode-tool.py
+++ b/tools/microcode-tool.py
@@ -1,4 +1,4 @@
-#!/usr/bin/env python2
+#!/usr/bin/env python3
# SPDX-License-Identifier: GPL-2.0+
#
# Copyright (c) 2014 Google, Inc
@@ -126,15 +126,15 @@ def List(date, microcodes, model):
microcodes: Dict of Microcode objects indexed by name
model: Model string to search for, or None
"""
- print 'Date: %s' % date
+ print('Date: %s' % date)
if model:
mcode_list, tried = FindMicrocode(microcodes, model.lower())
- print 'Matching models %s:' % (', '.join(tried))
+ print('Matching models %s:' % (', '.join(tried)))
else:
- print 'All models:'
- mcode_list = [microcodes[m] for m in microcodes.keys()]
+ print('All models:')
+ mcode_list = [microcodes[m] for m in list(microcodes.keys())]
for mcode in mcode_list:
- print '%-20s: model %s' % (mcode.name, mcode.model)
+ print('%-20s: model %s' % (mcode.name, mcode.model))
def FindMicrocode(microcodes, model):
"""Find all the microcode chunks which match the given model.
@@ -164,7 +164,7 @@ def FindMicrocode(microcodes, model):
for i in range(3):
abbrev = model[:-i] if i else model
tried.append(abbrev)
- for mcode in microcodes.values():
+ for mcode in list(microcodes.values()):
if mcode.model.startswith(abbrev):
found.append(mcode)
if found:
@@ -229,17 +229,17 @@ data = <%s
args += [mcode.words[i] for i in range(7)]
args.append(words)
if outfile == '-':
- print out % tuple(args)
+ print(out % tuple(args))
else:
if not outfile:
if not os.path.exists(MICROCODE_DIR):
- print >> sys.stderr, "Creating directory '%s'" % MICROCODE_DIR
+ print("Creating directory '%s'" % MICROCODE_DIR, file=sys.stderr)
os.makedirs(MICROCODE_DIR)
outfile = os.path.join(MICROCODE_DIR, mcode.name + '.dtsi')
- print >> sys.stderr, "Writing microcode for '%s' to '%s'" % (
- ', '.join([mcode.name for mcode in mcodes]), outfile)
+ print("Writing microcode for '%s' to '%s'" % (
+ ', '.join([mcode.name for mcode in mcodes]), outfile), file=sys.stderr)
with open(outfile, 'w') as fd:
- print >> fd, out % tuple(args)
+ print(out % tuple(args), file=fd)
def MicrocodeTool():
"""Run the microcode tool"""
@@ -289,14 +289,14 @@ def MicrocodeTool():
if cmd == 'list':
List(date, microcodes, options.model)
elif cmd == 'license':
- print '\n'.join(license_text)
+ print('\n'.join(license_text))
elif cmd == 'create':
if not options.model:
parser.error('You must specify a model to create')
model = options.model.lower()
if options.model == 'all':
options.multiple = True
- mcode_list = microcodes.values()
+ mcode_list = list(microcodes.values())
tried = []
else:
mcode_list, tried = FindMicrocode(microcodes, model)
--
2.24.0.rc0.303.g954a862665-goog
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [U-Boot] [PATCH v2 07/15] move_config: Convert to Python 3
2019-10-25 1:03 [U-Boot] [PATCH v2 00/15] scripts: Convert to Python 3 Simon Glass
` (5 preceding siblings ...)
2019-10-25 1:03 ` [U-Boot] [PATCH v2 06/15] microcode_tool: Convert to " Simon Glass
@ 2019-10-25 1:03 ` Simon Glass
2019-10-25 1:03 ` [U-Boot] [PATCH v2 08/15] rkmux: " Simon Glass
` (7 subsequent siblings)
14 siblings, 0 replies; 23+ messages in thread
From: Simon Glass @ 2019-10-25 1:03 UTC (permalink / raw)
To: u-boot
Convert this tool to Python 3 and make it use that, to meet the 2020
deadline.
Signed-off-by: Simon Glass <sjg@chromium.org>
---
Changes in v2: None
tools/moveconfig.py | 82 ++++++++++++++++++++++-----------------------
1 file changed, 41 insertions(+), 41 deletions(-)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index b99417e9d63..e2ff4cfc88b 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -1,4 +1,4 @@
-#!/usr/bin/env python2
+#!/usr/bin/env python3
# SPDX-License-Identifier: GPL-2.0+
#
# Author: Masahiro Yamada <yamada.masahiro@socionext.com>
@@ -304,7 +304,7 @@ import glob
import multiprocessing
import optparse
import os
-import Queue
+import queue
import re
import shutil
import subprocess
@@ -450,8 +450,8 @@ def get_matched_defconfigs(defconfigs_file):
line = line.split(' ')[0] # handle 'git log' input
matched = get_matched_defconfig(line)
if not matched:
- print >> sys.stderr, "warning: %s:%d: no defconfig matched '%s'" % \
- (defconfigs_file, i + 1, line)
+ print("warning: %s:%d: no defconfig matched '%s'" % \
+ (defconfigs_file, i + 1, line), file=sys.stderr)
defconfigs += matched
@@ -494,11 +494,11 @@ def show_diff(a, b, file_path, color_enabled):
for line in diff:
if line[0] == '-' and line[1] != '-':
- print color_text(color_enabled, COLOR_RED, line),
+ print(color_text(color_enabled, COLOR_RED, line), end=' ')
elif line[0] == '+' and line[1] != '+':
- print color_text(color_enabled, COLOR_GREEN, line),
+ print(color_text(color_enabled, COLOR_GREEN, line), end=' ')
else:
- print line,
+ print(line, end=' ')
def extend_matched_lines(lines, matched, pre_patterns, post_patterns, extend_pre,
extend_post):
@@ -554,9 +554,9 @@ def extend_matched_lines(lines, matched, pre_patterns, post_patterns, extend_pre
def confirm(options, prompt):
if not options.yes:
while True:
- choice = raw_input('{} [y/n]: '.format(prompt))
+ choice = input('{} [y/n]: '.format(prompt))
choice = choice.lower()
- print choice
+ print(choice)
if choice == 'y' or choice == 'n':
break
@@ -809,10 +809,10 @@ def try_expand(line):
val= val.strip('\"')
if re.search("[*+-/]|<<|SZ_+|\(([^\)]+)\)", val):
newval = hex(eval(val, SIZES))
- print "\tExpanded expression %s to %s" % (val, newval)
+ print("\tExpanded expression %s to %s" % (val, newval))
return cfg+'='+newval
except:
- print "\tFailed to expand expression in %s" % line
+ print("\tFailed to expand expression in %s" % line)
return line
@@ -838,7 +838,7 @@ class Progress:
def show(self):
"""Display the progress."""
- print ' %d defconfigs out of %d\r' % (self.current, self.total),
+ print(' %d defconfigs out of %d\r' % (self.current, self.total), end=' ')
sys.stdout.flush()
@@ -1236,7 +1236,7 @@ class Slot:
"Tool chain for '%s' is missing. Do nothing.\n" % arch)
self.finish(False)
return
- env = toolchain.MakeEnvironment(False)
+ env = toolchain.MakeEnvironment(False)
cmd = list(self.make_cmd)
cmd.append('KCONFIG_IGNORE_DUPLICATES=1')
@@ -1312,7 +1312,7 @@ class Slot:
log += '\n'.join([ ' ' + s for s in self.log.split('\n') ])
# Some threads are running in parallel.
# Print log atomically to not mix up logs from different threads.
- print >> (sys.stdout if success else sys.stderr), log
+ print(log, file=(sys.stdout if success else sys.stderr))
if not success:
if self.options.exit_on_error:
@@ -1411,8 +1411,8 @@ class Slots:
msg = "The following boards were not processed due to error:\n"
msg += boards
msg += "(the list has been saved in %s)\n" % output_file
- print >> sys.stderr, color_text(self.options.color, COLOR_LIGHT_RED,
- msg)
+ print(color_text(self.options.color, COLOR_LIGHT_RED,
+ msg), file=sys.stderr)
with open(output_file, 'w') as f:
f.write(boards)
@@ -1431,8 +1431,8 @@ class Slots:
msg += "It is highly recommended to check them manually:\n"
msg += boards
msg += "(the list has been saved in %s)\n" % output_file
- print >> sys.stderr, color_text(self.options.color, COLOR_YELLOW,
- msg)
+ print(color_text(self.options.color, COLOR_YELLOW,
+ msg), file=sys.stderr)
with open(output_file, 'w') as f:
f.write(boards)
@@ -1448,11 +1448,11 @@ class ReferenceSource:
commit: commit to git-clone
"""
self.src_dir = tempfile.mkdtemp()
- print "Cloning git repo to a separate work directory..."
+ print("Cloning git repo to a separate work directory...")
subprocess.check_output(['git', 'clone', os.getcwd(), '.'],
cwd=self.src_dir)
- print "Checkout '%s' to build the original autoconf.mk." % \
- subprocess.check_output(['git', 'rev-parse', '--short', commit]).strip()
+ print("Checkout '%s' to build the original autoconf.mk." % \
+ subprocess.check_output(['git', 'rev-parse', '--short', commit]).strip())
subprocess.check_output(['git', 'checkout', commit],
stderr=subprocess.STDOUT, cwd=self.src_dir)
@@ -1480,14 +1480,14 @@ def move_config(toolchains, configs, options, db_queue):
"""
if len(configs) == 0:
if options.force_sync:
- print 'No CONFIG is specified. You are probably syncing defconfigs.',
+ print('No CONFIG is specified. You are probably syncing defconfigs.', end=' ')
elif options.build_db:
- print 'Building %s database' % CONFIG_DATABASE
+ print('Building %s database' % CONFIG_DATABASE)
else:
- print 'Neither CONFIG nor --force-sync is specified. Nothing will happen.',
+ print('Neither CONFIG nor --force-sync is specified. Nothing will happen.', end=' ')
else:
- print 'Move ' + ', '.join(configs),
- print '(jobs: %d)\n' % options.jobs
+ print('Move ' + ', '.join(configs), end=' ')
+ print('(jobs: %d)\n' % options.jobs)
if options.git_ref:
reference_src = ReferenceSource(options.git_ref)
@@ -1517,7 +1517,7 @@ def move_config(toolchains, configs, options, db_queue):
while not slots.empty():
time.sleep(SLEEP_TIME)
- print ''
+ print('')
slots.show_failed_boards()
slots.show_suspicious_boards()
@@ -1691,15 +1691,15 @@ def do_imply_config(config_list, add_imply, imply_flags, skip_added,
for config in config_list:
defconfigs = defconfig_db.get(config)
if not defconfigs:
- print '%s not found in any defconfig' % config
+ print('%s not found in any defconfig' % config)
continue
# Get the set of defconfigs without this one (since a config cannot
# imply itself)
non_defconfigs = all_defconfigs - defconfigs
num_defconfigs = len(defconfigs)
- print '%s found in %d/%d defconfigs' % (config, num_defconfigs,
- len(all_configs))
+ print('%s found in %d/%d defconfigs' % (config, num_defconfigs,
+ len(all_configs)))
# This will hold the results: key=config, value=defconfigs containing it
imply_configs = {}
@@ -1736,7 +1736,7 @@ def do_imply_config(config_list, add_imply, imply_flags, skip_added,
if common_defconfigs:
skip = False
if find_superset:
- for prev in imply_configs.keys():
+ for prev in list(imply_configs.keys()):
prev_count = len(imply_configs[prev])
count = len(common_defconfigs)
if (prev_count > count and
@@ -1806,15 +1806,15 @@ def do_imply_config(config_list, add_imply, imply_flags, skip_added,
add_list[fname].append(linenum)
if show and kconfig_info != 'skip':
- print '%5d : %-30s%-25s %s' % (num_common, iconfig.ljust(30),
- kconfig_info, missing_str)
+ print('%5d : %-30s%-25s %s' % (num_common, iconfig.ljust(30),
+ kconfig_info, missing_str))
# Having collected a list of things to add, now we add them. We process
# each file from the largest line number to the smallest so that
# earlier additions do not affect our line numbers. E.g. if we added an
# imply at line 20 it would change the position of each line after
# that.
- for fname, linenums in add_list.iteritems():
+ for fname, linenums in add_list.items():
for linenum in sorted(linenums, reverse=True):
add_imply_rule(config[CONFIG_LEN:], fname, linenum)
@@ -1891,11 +1891,11 @@ def main():
for flag in options.imply_flags.split(','):
bad = flag not in IMPLY_FLAGS
if bad:
- print "Invalid flag '%s'" % flag
+ print("Invalid flag '%s'" % flag)
if flag == 'help' or bad:
- print "Imply flags: (separate with ',')"
- for name, info in IMPLY_FLAGS.iteritems():
- print ' %-15s: %s' % (name, info[1])
+ print("Imply flags: (separate with ',')")
+ for name, info in IMPLY_FLAGS.items():
+ print(' %-15s: %s' % (name, info[1]))
parser.print_usage()
sys.exit(1)
imply_flags |= IMPLY_FLAGS[flag][0]
@@ -1905,14 +1905,14 @@ def main():
return
config_db = {}
- db_queue = Queue.Queue()
+ db_queue = queue.Queue()
t = DatabaseThread(config_db, db_queue)
t.setDaemon(True)
t.start()
if not options.cleanup_headers_only:
check_clean_directory()
- bsettings.Setup('')
+ bsettings.Setup('')
toolchains = toolchain.Toolchains()
toolchains.GetSettings()
toolchains.Scan(verbose=False)
@@ -1939,7 +1939,7 @@ def main():
if options.build_db:
with open(CONFIG_DATABASE, 'w') as fd:
- for defconfig, configs in config_db.iteritems():
+ for defconfig, configs in config_db.items():
fd.write('%s\n' % defconfig)
for config in sorted(configs.keys()):
fd.write(' %s=%s\n' % (config, configs[config]))
--
2.24.0.rc0.303.g954a862665-goog
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [U-Boot] [PATCH v2 08/15] rkmux: Convert to Python 3
2019-10-25 1:03 [U-Boot] [PATCH v2 00/15] scripts: Convert to Python 3 Simon Glass
` (6 preceding siblings ...)
2019-10-25 1:03 ` [U-Boot] [PATCH v2 07/15] move_config: " Simon Glass
@ 2019-10-25 1:03 ` Simon Glass
2019-10-25 1:03 ` [U-Boot] [PATCH v2 09/15] pylibfdt: " Simon Glass
` (6 subsequent siblings)
14 siblings, 0 replies; 23+ messages in thread
From: Simon Glass @ 2019-10-25 1:03 UTC (permalink / raw)
To: u-boot
Convert this tool to Python 3 and make it use that, to meet the 2020
deadline.
Signed-off-by: Simon Glass <sjg@chromium.org>
---
Changes in v2: None
tools/rkmux.py | 16 ++++++++--------
1 file changed, 8 insertions(+), 8 deletions(-)
diff --git a/tools/rkmux.py b/tools/rkmux.py
index 11c192a0737..1226ee201c3 100755
--- a/tools/rkmux.py
+++ b/tools/rkmux.py
@@ -1,4 +1,4 @@
-#!/usr/bin/env python2
+#!/usr/bin/env python3
# Script to create enums from datasheet register tables
#
@@ -43,8 +43,8 @@ class RegField:
self.desc.append(desc)
def Show(self):
- print self
- print
+ print(self)
+ print()
self.__init__()
def __str__(self):
@@ -65,11 +65,11 @@ class Printer:
self.output_footer()
def output_header(self):
- print '/* %s */' % self.name
- print 'enum {'
+ print('/* %s */' % self.name)
+ print('enum {')
def output_footer(self):
- print '};';
+ print('};');
def output_regfield(self, regfield):
lines = regfield.desc
@@ -97,7 +97,7 @@ class Printer:
self.first = False
self.output_header()
else:
- print
+ print()
out_enum(field, 'shift', bit_low)
out_enum(field, 'mask', mask)
next_val = -1
@@ -175,7 +175,7 @@ def out_enum(field, suffix, value, skip_val=False):
val_str = '%d' % value
str += '%s= %s' % ('\t' * tabs, val_str)
- print '\t%s,' % str
+ print('\t%s,' % str)
# Process a CSV file, e.g. from tabula
def process_csv(name, fd):
--
2.24.0.rc0.303.g954a862665-goog
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [U-Boot] [PATCH v2 09/15] pylibfdt: Convert to Python 3
2019-10-25 1:03 [U-Boot] [PATCH v2 00/15] scripts: Convert to Python 3 Simon Glass
` (7 preceding siblings ...)
2019-10-25 1:03 ` [U-Boot] [PATCH v2 08/15] rkmux: " Simon Glass
@ 2019-10-25 1:03 ` Simon Glass
2019-10-25 1:03 ` [U-Boot] [PATCH v2 10/15] pylibfdt: Sync up with upstream Simon Glass
` (5 subsequent siblings)
14 siblings, 0 replies; 23+ messages in thread
From: Simon Glass @ 2019-10-25 1:03 UTC (permalink / raw)
To: u-boot
Build this swig module with Python 3.
Signed-off-by: Simon Glass <sjg@chromium.org>
---
Changes in v2: None
scripts/dtc/pylibfdt/Makefile | 2 +-
scripts/dtc/pylibfdt/libfdt.i_shipped | 2 +-
scripts/dtc/pylibfdt/setup.py | 2 +-
tools/binman/entry.py | 16 ++--------------
tools/binman/entry_test.py | 15 ---------------
tools/binman/etype/intel_fit.py | 2 +-
6 files changed, 6 insertions(+), 33 deletions(-)
diff --git a/scripts/dtc/pylibfdt/Makefile b/scripts/dtc/pylibfdt/Makefile
index 15e66ad44d7..42342c75bb1 100644
--- a/scripts/dtc/pylibfdt/Makefile
+++ b/scripts/dtc/pylibfdt/Makefile
@@ -21,7 +21,7 @@ quiet_cmd_pymod = PYMOD $@
CPPFLAGS="$(HOSTCFLAGS) -I$(LIBFDT_srcdir)" OBJDIR=$(obj) \
SOURCES="$(PYLIBFDT_srcs)" \
SWIG_OPTS="-I$(LIBFDT_srcdir) -I$(LIBFDT_srcdir)/.." \
- $(PYTHON2) $< --quiet build_ext --inplace
+ $(PYTHON3) $< --quiet build_ext --inplace
$(obj)/_libfdt.so: $(src)/setup.py $(PYLIBFDT_srcs) FORCE
$(call if_changed,pymod)
diff --git a/scripts/dtc/pylibfdt/libfdt.i_shipped b/scripts/dtc/pylibfdt/libfdt.i_shipped
index 76e61e98bdf..53b70f8f5e7 100644
--- a/scripts/dtc/pylibfdt/libfdt.i_shipped
+++ b/scripts/dtc/pylibfdt/libfdt.i_shipped
@@ -624,7 +624,7 @@ class Fdt(FdtRo):
Raises:
FdtException if no parent found or other error occurs
"""
- val = val.encode('utf-8') + '\0'
+ val = val.encode('utf-8') + b'\0'
return check_err(fdt_setprop(self._fdt, nodeoffset, prop_name,
val, len(val)), quiet)
diff --git a/scripts/dtc/pylibfdt/setup.py b/scripts/dtc/pylibfdt/setup.py
index 4f7cf042bfe..992cdec30f5 100755
--- a/scripts/dtc/pylibfdt/setup.py
+++ b/scripts/dtc/pylibfdt/setup.py
@@ -1,4 +1,4 @@
-#!/usr/bin/env python2
+#!/usr/bin/env python3
"""
setup.py file for SWIG libfdt
diff --git a/tools/binman/entry.py b/tools/binman/entry.py
index 409c0dca934..5bf5be4794b 100644
--- a/tools/binman/entry.py
+++ b/tools/binman/entry.py
@@ -7,16 +7,7 @@
from __future__ import print_function
from collections import namedtuple
-
-# importlib was introduced in Python 2.7 but there was a report of it not
-# working in 2.7.12, so we work around this:
-# http://lists.denx.de/pipermail/u-boot/2016-October/269729.html
-try:
- import importlib
- have_importlib = True
-except:
- have_importlib = False
-
+import importlib
import os
import sys
@@ -119,10 +110,7 @@ class Entry(object):
old_path = sys.path
sys.path.insert(0, os.path.join(our_path, 'etype'))
try:
- if have_importlib:
- module = importlib.import_module(module_name)
- else:
- module = __import__(module_name)
+ module = importlib.import_module(module_name)
except ImportError as e:
raise ValueError("Unknown entry type '%s' in node '%s' (expected etype/%s.py, error '%s'" %
(etype, node_path, module_name, e))
diff --git a/tools/binman/entry_test.py b/tools/binman/entry_test.py
index 13f58645168..277e10b5859 100644
--- a/tools/binman/entry_test.py
+++ b/tools/binman/entry_test.py
@@ -39,21 +39,6 @@ class TestEntry(unittest.TestCase):
else:
import entry
- def test1EntryNoImportLib(self):
- """Test that we can import Entry subclassess successfully"""
- sys.modules['importlib'] = None
- global entry
- self._ReloadEntry()
- entry.Entry.Create(None, self.GetNode(), 'u-boot')
- self.assertFalse(entry.have_importlib)
-
- def test2EntryImportLib(self):
- del sys.modules['importlib']
- global entry
- self._ReloadEntry()
- entry.Entry.Create(None, self.GetNode(), 'u-boot-spl')
- self.assertTrue(entry.have_importlib)
-
def testEntryContents(self):
"""Test the Entry bass class"""
import entry
diff --git a/tools/binman/etype/intel_fit.py b/tools/binman/etype/intel_fit.py
index 23606d27d04..2a34a05f955 100644
--- a/tools/binman/etype/intel_fit.py
+++ b/tools/binman/etype/intel_fit.py
@@ -27,6 +27,6 @@ class Entry_intel_fit(Entry_blob):
self.align = 16
def ObtainContents(self):
- data = struct.pack('<8sIHBB', '_FIT_ ', 1, 0x100, 0x80, 0x7d)
+ data = struct.pack('<8sIHBB', b'_FIT_ ', 1, 0x100, 0x80, 0x7d)
self.SetContents(data)
return True
--
2.24.0.rc0.303.g954a862665-goog
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [U-Boot] [PATCH v2 10/15] pylibfdt: Sync up with upstream
2019-10-25 1:03 [U-Boot] [PATCH v2 00/15] scripts: Convert to Python 3 Simon Glass
` (8 preceding siblings ...)
2019-10-25 1:03 ` [U-Boot] [PATCH v2 09/15] pylibfdt: " Simon Glass
@ 2019-10-25 1:03 ` Simon Glass
2019-10-25 1:03 ` [U-Boot] [PATCH v2 11/15] pylibfdt: Correct the type for fdt_property_stub() Simon Glass
` (4 subsequent siblings)
14 siblings, 0 replies; 23+ messages in thread
From: Simon Glass @ 2019-10-25 1:03 UTC (permalink / raw)
To: u-boot
Sync up the libfdt Python bindings with upstream, commit:
430419c (tests: fix some python warnings)
Signed-off-by: Simon Glass <sjg@chromium.org>
---
Changes in v2: None
scripts/dtc/pylibfdt/libfdt.i_shipped | 45 ++++++++++++++++++++-------
1 file changed, 33 insertions(+), 12 deletions(-)
diff --git a/scripts/dtc/pylibfdt/libfdt.i_shipped b/scripts/dtc/pylibfdt/libfdt.i_shipped
index 53b70f8f5e7..e2aa7bb01e9 100644
--- a/scripts/dtc/pylibfdt/libfdt.i_shipped
+++ b/scripts/dtc/pylibfdt/libfdt.i_shipped
@@ -92,7 +92,7 @@ def check_err(val, quiet=()):
Raises
FdtException if val < 0
"""
- if val < 0:
+ if isinstance(val, int) and val < 0:
if -val not in quiet:
raise FdtException(val)
return val
@@ -417,7 +417,7 @@ class FdtRo(object):
quiet)
if isinstance(pdata, (int)):
return pdata
- return Property(prop_name, bytearray(pdata[0]))
+ return Property(prop_name, bytes(pdata[0]))
def get_phandle(self, nodeoffset):
"""Get the phandle of a node
@@ -431,6 +431,18 @@ class FdtRo(object):
"""
return fdt_get_phandle(self._fdt, nodeoffset)
+ def get_alias(self, name):
+ """Get the full path referenced by a given alias
+
+ Args:
+ name: name of the alias to lookup
+
+ Returns:
+ Full path to the node for the alias named 'name', if it exists
+ None, if the given alias or the /aliases node does not exist
+ """
+ return fdt_get_alias(self._fdt, name)
+
def parent_offset(self, nodeoffset, quiet=()):
"""Get the offset of a node's parent
@@ -727,8 +739,10 @@ class FdtSw(FdtRo):
# First create the device tree with a node and property:
sw = FdtSw()
- with sw.add_node('node'):
- sw.property_u32('reg', 2)
+ sw.finish_reservemap()
+ with sw.add_node(''):
+ with sw.add_node('node'):
+ sw.property_u32('reg', 2)
fdt = sw.as_fdt()
# Now we can use it as a real device tree
@@ -1029,17 +1043,24 @@ typedef uint32_t fdt32_t;
if (!$1)
$result = Py_None;
else
- $result = Py_BuildValue("s#", $1, *arg4);
+ %#if PY_VERSION_HEX >= 0x03000000
+ $result = Py_BuildValue("y#", $1, *arg4);
+ %#else
+ $result = Py_BuildValue("s#", $1, *arg4);
+ %#endif
}
/* typemap used for fdt_setprop() */
%typemap(in) (const void *val) {
- $1 = PyString_AsString($input); /* char *str */
-}
-
-/* typemap used for fdt_add_reservemap_entry() */
-%typemap(in) uint64_t {
- $1 = PyLong_AsUnsignedLong($input);
+ %#if PY_VERSION_HEX >= 0x03000000
+ if (!PyBytes_Check($input)) {
+ SWIG_exception_fail(SWIG_TypeError, "bytes expected in method '" "$symname"
+ "', argument " "$argnum"" of type '" "$type""'");
+ }
+ $1 = PyBytes_AsString($input);
+ %#else
+ $1 = PyString_AsString($input); /* char *str */
+ %#endif
}
/* typemaps used for fdt_next_node() */
@@ -1061,7 +1082,7 @@ typedef uint32_t fdt32_t;
}
%typemap(argout) uint64_t * {
- PyObject *val = PyLong_FromUnsignedLong(*arg$argnum);
+ PyObject *val = PyLong_FromUnsignedLongLong(*arg$argnum);
if (!result) {
if (PyTuple_GET_SIZE(resultobj) == 0)
resultobj = val;
--
2.24.0.rc0.303.g954a862665-goog
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [U-Boot] [PATCH v2 11/15] pylibfdt: Correct the type for fdt_property_stub()
2019-10-25 1:03 [U-Boot] [PATCH v2 00/15] scripts: Convert to Python 3 Simon Glass
` (9 preceding siblings ...)
2019-10-25 1:03 ` [U-Boot] [PATCH v2 10/15] pylibfdt: Sync up with upstream Simon Glass
@ 2019-10-25 1:03 ` Simon Glass
2019-10-25 1:03 ` [U-Boot] [PATCH v2 12/15] binman: Remember the pre-reset entry size Simon Glass
` (3 subsequent siblings)
14 siblings, 0 replies; 23+ messages in thread
From: Simon Glass @ 2019-10-25 1:03 UTC (permalink / raw)
To: u-boot
This function should use a void * type, not char *. This causes an error:
TypeError: in method 'fdt_property_stub', argument 3 of type 'char const *'
Fix it.
Signed-off-by: Simon Glass <sjg@chromium.org>
---
Changes in v2: None
scripts/dtc/pylibfdt/libfdt.i_shipped | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/scripts/dtc/pylibfdt/libfdt.i_shipped b/scripts/dtc/pylibfdt/libfdt.i_shipped
index e2aa7bb01e9..fae0b27d7d0 100644
--- a/scripts/dtc/pylibfdt/libfdt.i_shipped
+++ b/scripts/dtc/pylibfdt/libfdt.i_shipped
@@ -18,7 +18,7 @@
* a struct called fdt_property. That struct causes swig to create a class in
* libfdt.py called fdt_property(), which confuses things.
*/
-static int fdt_property_stub(void *fdt, const char *name, const char *val,
+static int fdt_property_stub(void *fdt, const char *name, const void *val,
int len)
{
return fdt_property(fdt, name, val, len);
@@ -1113,6 +1113,6 @@ int fdt_property_cell(void *fdt, const char *name, uint32_t val);
* This function has a stub since the name fdt_property is used for both a
* function and a struct, which confuses SWIG.
*/
-int fdt_property_stub(void *fdt, const char *name, const char *val, int len);
+int fdt_property_stub(void *fdt, const char *name, const void *val, int len);
%include <../libfdt/libfdt.h>
--
2.24.0.rc0.303.g954a862665-goog
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [U-Boot] [PATCH v2 12/15] binman: Remember the pre-reset entry size
2019-10-25 1:03 [U-Boot] [PATCH v2 00/15] scripts: Convert to Python 3 Simon Glass
` (10 preceding siblings ...)
2019-10-25 1:03 ` [U-Boot] [PATCH v2 11/15] pylibfdt: Correct the type for fdt_property_stub() Simon Glass
@ 2019-10-25 1:03 ` Simon Glass
2019-10-25 1:03 ` [U-Boot] [PATCH v2 13/15] binman: Convert a few tests to Python 3 Simon Glass
` (2 subsequent siblings)
14 siblings, 0 replies; 23+ messages in thread
From: Simon Glass @ 2019-10-25 1:03 UTC (permalink / raw)
To: u-boot
When preparing to possible expand or contract an entry we reset the size
to the original value from the binman device-tree definition, which is
often None.
This causes binman to forget the original size of the entry. Remember this
so that it can be used when needed.
Signed-off-by: Simon Glass <sjg@chromium.org>
---
Changes in v2: None
tools/binman/entry.py | 9 ++++++++-
1 file changed, 8 insertions(+), 1 deletion(-)
diff --git a/tools/binman/entry.py b/tools/binman/entry.py
index 5bf5be4794b..b6f1b2c93fb 100644
--- a/tools/binman/entry.py
+++ b/tools/binman/entry.py
@@ -47,6 +47,8 @@ class Entry(object):
offset: Offset of entry within the section, None if not known yet (in
which case it will be calculated by Pack())
size: Entry size in bytes, None if not known
+ pre_reset_size: size as it was before ResetForPack(). This allows us to
+ keep track of the size we started with and detect size changes
uncomp_size: Size of uncompressed data in bytes, if the entry is
compressed, else None
contents_size: Size of contents in bytes, 0 by default
@@ -71,6 +73,7 @@ class Entry(object):
self.name = node and (name_prefix + node.name) or 'none'
self.offset = None
self.size = None
+ self.pre_reset_size = None
self.uncomp_size = None
self.data = None
self.contents_size = 0
@@ -314,6 +317,7 @@ class Entry(object):
self.Detail('ResetForPack: offset %s->%s, size %s->%s' %
(ToHex(self.offset), ToHex(self.orig_offset),
ToHex(self.size), ToHex(self.orig_size)))
+ self.pre_reset_size = self.size
self.offset = self.orig_offset
self.size = self.orig_size
@@ -757,7 +761,10 @@ features to produce new behaviours.
True if the data did not result in a resize of this entry, False if
the entry must be resized
"""
- self.contents_size = self.size
+ if self.size is not None:
+ self.contents_size = self.size
+ else:
+ self.contents_size = self.pre_reset_size
ok = self.ProcessContentsUpdate(data)
self.Detail('WriteData: size=%x, ok=%s' % (len(data), ok))
section_ok = self.section.WriteChildData(self)
--
2.24.0.rc0.303.g954a862665-goog
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [U-Boot] [PATCH v2 13/15] binman: Convert a few tests to Python 3
2019-10-25 1:03 [U-Boot] [PATCH v2 00/15] scripts: Convert to Python 3 Simon Glass
` (11 preceding siblings ...)
2019-10-25 1:03 ` [U-Boot] [PATCH v2 12/15] binman: Remember the pre-reset entry size Simon Glass
@ 2019-10-25 1:03 ` Simon Glass
2019-10-25 1:03 ` [U-Boot] [PATCH v2 14/15] dtoc: Convert fdt.py " Simon Glass
2019-10-25 1:03 ` [U-Boot] [PATCH v2 15/15] binman: Move to use " Simon Glass
14 siblings, 0 replies; 23+ messages in thread
From: Simon Glass @ 2019-10-25 1:03 UTC (permalink / raw)
To: u-boot
Some tests have crept in with Python 2 strings and constructs. Convert
then.
Signed-off-by: Simon Glass <sjg@chromium.org>
---
Changes in v2: None
tools/binman/ftest.py | 14 +++++++-------
1 file changed, 7 insertions(+), 7 deletions(-)
diff --git a/tools/binman/ftest.py b/tools/binman/ftest.py
index 88daaf20a87..5ca1c073f23 100644
--- a/tools/binman/ftest.py
+++ b/tools/binman/ftest.py
@@ -2109,7 +2109,7 @@ class TestFunctional(unittest.TestCase):
data = self.data = self._DoReadFileRealDtb('115_fdtmap.dts')
fdtmap_data = data[len(U_BOOT_DATA):]
magic = fdtmap_data[:8]
- self.assertEqual('_FDTMAP_', magic)
+ self.assertEqual(b'_FDTMAP_', magic)
self.assertEqual(tools.GetBytes(0, 8), fdtmap_data[8:16])
fdt_data = fdtmap_data[16:]
@@ -2152,7 +2152,7 @@ class TestFunctional(unittest.TestCase):
dtb = fdt.Fdt.FromData(fdt_data)
fdt_size = dtb.GetFdtObj().totalsize()
hdr_data = data[-8:]
- self.assertEqual('BinM', hdr_data[:4])
+ self.assertEqual(b'BinM', hdr_data[:4])
offset = struct.unpack('<I', hdr_data[4:])[0] & 0xffffffff
self.assertEqual(fdtmap_pos - 0x400, offset - (1 << 32))
@@ -2161,7 +2161,7 @@ class TestFunctional(unittest.TestCase):
data = self.data = self._DoReadFileRealDtb('117_fdtmap_hdr_start.dts')
fdtmap_pos = 0x100 + len(U_BOOT_DATA)
hdr_data = data[:8]
- self.assertEqual('BinM', hdr_data[:4])
+ self.assertEqual(b'BinM', hdr_data[:4])
offset = struct.unpack('<I', hdr_data[4:])[0]
self.assertEqual(fdtmap_pos, offset)
@@ -2170,7 +2170,7 @@ class TestFunctional(unittest.TestCase):
data = self.data = self._DoReadFileRealDtb('118_fdtmap_hdr_pos.dts')
fdtmap_pos = 0x100 + len(U_BOOT_DATA)
hdr_data = data[0x80:0x88]
- self.assertEqual('BinM', hdr_data[:4])
+ self.assertEqual(b'BinM', hdr_data[:4])
offset = struct.unpack('<I', hdr_data[4:])[0]
self.assertEqual(fdtmap_pos, offset)
@@ -2431,9 +2431,9 @@ class TestFunctional(unittest.TestCase):
' section 100 %x section 100' % section_size,
' cbfs 100 400 cbfs 0',
' u-boot 138 4 u-boot 38',
-' u-boot-dtb 180 10f u-boot-dtb 80 3c9',
+' u-boot-dtb 180 105 u-boot-dtb 80 3c9',
' u-boot-dtb 500 %x u-boot-dtb 400 3c9' % fdt_size,
-' fdtmap %x 3b4 fdtmap %x' %
+' fdtmap %x 3bd fdtmap %x' %
(fdtmap_offset, fdtmap_offset),
' image-header bf8 8 image-header bf8',
]
@@ -2518,7 +2518,7 @@ class TestFunctional(unittest.TestCase):
data = self._RunExtractCmd('section')
cbfs_data = data[:0x400]
cbfs = cbfs_util.CbfsReader(cbfs_data)
- self.assertEqual(['u-boot', 'u-boot-dtb', ''], cbfs.files.keys())
+ self.assertEqual(['u-boot', 'u-boot-dtb', ''], list(cbfs.files.keys()))
dtb_data = data[0x400:]
dtb = self._decompress(dtb_data)
self.assertEqual(EXTRACT_DTB_SIZE, len(dtb))
--
2.24.0.rc0.303.g954a862665-goog
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [U-Boot] [PATCH v2 14/15] dtoc: Convert fdt.py to Python 3
2019-10-25 1:03 [U-Boot] [PATCH v2 00/15] scripts: Convert to Python 3 Simon Glass
` (12 preceding siblings ...)
2019-10-25 1:03 ` [U-Boot] [PATCH v2 13/15] binman: Convert a few tests to Python 3 Simon Glass
@ 2019-10-25 1:03 ` Simon Glass
2019-10-25 1:03 ` [U-Boot] [PATCH v2 15/15] binman: Move to use " Simon Glass
14 siblings, 0 replies; 23+ messages in thread
From: Simon Glass @ 2019-10-25 1:03 UTC (permalink / raw)
To: u-boot
Drop the now-unused Python 2 code to keep code coverage at 100%.
Signed-off-by: Simon Glass <sjg@chromium.org>
---
Changes in v2: None
tools/dtoc/fdt.py | 17 ++++-------------
1 file changed, 4 insertions(+), 13 deletions(-)
diff --git a/tools/dtoc/fdt.py b/tools/dtoc/fdt.py
index 6770be79fbe..1b7b730359a 100644
--- a/tools/dtoc/fdt.py
+++ b/tools/dtoc/fdt.py
@@ -56,9 +56,6 @@ def BytesToValue(data):
is_string = False
break
for ch in string:
- # Handle Python 2 treating bytes as str
- if type(ch) == str:
- ch = ord(ch)
if ch < 32 or ch > 127:
is_string = False
break
@@ -66,15 +63,9 @@ def BytesToValue(data):
is_string = False
if is_string:
if count == 1:
- if sys.version_info[0] >= 3: # pragma: no cover
- return TYPE_STRING, strings[0].decode()
- else:
- return TYPE_STRING, strings[0]
+ return TYPE_STRING, strings[0].decode()
else:
- if sys.version_info[0] >= 3: # pragma: no cover
- return TYPE_STRING, [s.decode() for s in strings[:-1]]
- else:
- return TYPE_STRING, strings[:-1]
+ return TYPE_STRING, [s.decode() for s in strings[:-1]]
if size % 4:
if size == 1:
return TYPE_BYTE, tools.ToChar(data[0])
@@ -415,8 +406,8 @@ class Node:
prop_name: Name of property to set
val: String value to set (will be \0-terminated in DT)
"""
- if sys.version_info[0] >= 3: # pragma: no cover
- val = bytes(val, 'utf-8')
+ if type(val) == str:
+ val = val.encode('utf-8')
self._CheckProp(prop_name).props[prop_name].SetData(val + b'\0')
def AddString(self, prop_name, val):
--
2.24.0.rc0.303.g954a862665-goog
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [U-Boot] [PATCH v2 15/15] binman: Move to use Python 3
2019-10-25 1:03 [U-Boot] [PATCH v2 00/15] scripts: Convert to Python 3 Simon Glass
` (13 preceding siblings ...)
2019-10-25 1:03 ` [U-Boot] [PATCH v2 14/15] dtoc: Convert fdt.py " Simon Glass
@ 2019-10-25 1:03 ` Simon Glass
14 siblings, 0 replies; 23+ messages in thread
From: Simon Glass @ 2019-10-25 1:03 UTC (permalink / raw)
To: u-boot
Update this tool to use Python 3 to meet the 2020 deadline.
Unfortunately this introduces a test failure due to a problem in pylibfdt
on Python 3. I will investigate.
Signed-off-by: Simon Glass <sjg@chromium.org>
---
Changes in v2:
- Add a few more patches to correct remaining problems
tools/binman/binman.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tools/binman/binman.py b/tools/binman/binman.py
index 8bd5868df26..9e6fd721175 100755
--- a/tools/binman/binman.py
+++ b/tools/binman/binman.py
@@ -1,4 +1,4 @@
-#!/usr/bin/env python2
+#!/usr/bin/env python3
# SPDX-License-Identifier: GPL-2.0+
# Copyright (c) 2016 Google, Inc
--
2.24.0.rc0.303.g954a862665-goog
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [U-Boot] [PATCH v2 03/15] buildman: Convert to Python 3
2019-10-25 1:03 ` [U-Boot] [PATCH v2 03/15] buildman: Convert to " Simon Glass
@ 2019-10-30 1:47 ` Simon Glass
2019-10-30 13:15 ` Tom Rini
0 siblings, 1 reply; 23+ messages in thread
From: Simon Glass @ 2019-10-30 1:47 UTC (permalink / raw)
To: u-boot
Hi Tom,
On Thu, 24 Oct 2019 at 19:04, Simon Glass <sjg@chromium.org> wrote:
>
> Convert buildman to Python 3 and make it use that, to meet the 2020
> deadline.
>
> Signed-off-by: Simon Glass <sjg@chromium.org>
> ---
>
> Changes in v2: None
>
> tools/buildman/board.py | 7 +--
> tools/buildman/bsettings.py | 20 +++----
> tools/buildman/builder.py | 45 ++++++++--------
> tools/buildman/builderthread.py | 24 ++++-----
> tools/buildman/buildman.py | 10 ++--
> tools/buildman/control.py | 40 +++++++-------
> tools/buildman/func_test.py | 16 +++---
> tools/buildman/test.py | 22 ++++----
> tools/buildman/toolchain.py | 93 +++++++++++++++++----------------
> 9 files changed, 139 insertions(+), 138 deletions(-)
I am a bit mystified by this:
https://gitlab.denx.de/u-boot/custodians/u-boot-dm/-/jobs/25186
It runs locally and I cannot see why it is not using unicode. Are
there instructions on how to get to the same Python version as gitlab?
I fear a steep learning curve coming.
Regards,
Simon
^ permalink raw reply [flat|nested] 23+ messages in thread
* [U-Boot] [PATCH v2 03/15] buildman: Convert to Python 3
2019-10-30 1:47 ` Simon Glass
@ 2019-10-30 13:15 ` Tom Rini
2019-10-30 15:12 ` Simon Glass
0 siblings, 1 reply; 23+ messages in thread
From: Tom Rini @ 2019-10-30 13:15 UTC (permalink / raw)
To: u-boot
On Tue, Oct 29, 2019 at 07:47:55PM -0600, Simon Glass wrote:
> Hi Tom,
>
> On Thu, 24 Oct 2019 at 19:04, Simon Glass <sjg@chromium.org> wrote:
> >
> > Convert buildman to Python 3 and make it use that, to meet the 2020
> > deadline.
> >
> > Signed-off-by: Simon Glass <sjg@chromium.org>
> > ---
> >
> > Changes in v2: None
> >
> > tools/buildman/board.py | 7 +--
> > tools/buildman/bsettings.py | 20 +++----
> > tools/buildman/builder.py | 45 ++++++++--------
> > tools/buildman/builderthread.py | 24 ++++-----
> > tools/buildman/buildman.py | 10 ++--
> > tools/buildman/control.py | 40 +++++++-------
> > tools/buildman/func_test.py | 16 +++---
> > tools/buildman/test.py | 22 ++++----
> > tools/buildman/toolchain.py | 93 +++++++++++++++++----------------
> > 9 files changed, 139 insertions(+), 138 deletions(-)
>
> I am a bit mystified by this:
>
> https://gitlab.denx.de/u-boot/custodians/u-boot-dm/-/jobs/25186
>
> It runs locally and I cannot see why it is not using unicode. Are
> there instructions on how to get to the same Python version as gitlab?
> I fear a steep learning curve coming.
First, I think the problem is this series will also need the test.py
python3 series as that does a few more things to get and provide a
python3 environment to use. I'll try and throw something together to
test that shortly.
Second, to have the same environment locally:
$ sudo docker pull trini/u-boot-gitlab-ci-runner:bionic-20191010-20Oct2019
And change the tag to whatever .gitlab-ci.yml says we use. That one is
what the test.py series brings us up to, and is newer than your failing
job.
$ sudo docker run --rm -it -v /your/local/u-boot/clone/..:/build/u-boot \
trini/u-boot-gitlab-ci-runner:bionic-20191010-20Oct2019 /bin/bash
And don't pass --rm if you don't want the image deleted once you exit.
That will put you in a bash prompt as the user we run everything as.
Follow the steps in the job as described in .gitlab-ci.yml to do a build
by hand.
And yes, you want to pass in one level before your clone so that in the
container you have /build/u-boot/u-boot
--
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20191030/f9b08d59/attachment.sig>
^ permalink raw reply [flat|nested] 23+ messages in thread
* [U-Boot] [PATCH v2 03/15] buildman: Convert to Python 3
2019-10-30 13:15 ` Tom Rini
@ 2019-10-30 15:12 ` Simon Glass
2019-10-30 15:58 ` Tom Rini
0 siblings, 1 reply; 23+ messages in thread
From: Simon Glass @ 2019-10-30 15:12 UTC (permalink / raw)
To: u-boot
Hi Tom,
On Wed, 30 Oct 2019 at 07:15, Tom Rini <trini@konsulko.com> wrote:
>
> On Tue, Oct 29, 2019 at 07:47:55PM -0600, Simon Glass wrote:
> > Hi Tom,
> >
> > On Thu, 24 Oct 2019 at 19:04, Simon Glass <sjg@chromium.org> wrote:
> > >
> > > Convert buildman to Python 3 and make it use that, to meet the 2020
> > > deadline.
> > >
> > > Signed-off-by: Simon Glass <sjg@chromium.org>
> > > ---
> > >
> > > Changes in v2: None
> > >
> > > tools/buildman/board.py | 7 +--
> > > tools/buildman/bsettings.py | 20 +++----
> > > tools/buildman/builder.py | 45 ++++++++--------
> > > tools/buildman/builderthread.py | 24 ++++-----
> > > tools/buildman/buildman.py | 10 ++--
> > > tools/buildman/control.py | 40 +++++++-------
> > > tools/buildman/func_test.py | 16 +++---
> > > tools/buildman/test.py | 22 ++++----
> > > tools/buildman/toolchain.py | 93 +++++++++++++++++----------------
> > > 9 files changed, 139 insertions(+), 138 deletions(-)
> >
> > I am a bit mystified by this:
> >
> > https://gitlab.denx.de/u-boot/custodians/u-boot-dm/-/jobs/25186
> >
> > It runs locally and I cannot see why it is not using unicode. Are
> > there instructions on how to get to the same Python version as gitlab?
> > I fear a steep learning curve coming.
>
> First, I think the problem is this series will also need the test.py
> python3 series as that does a few more things to get and provide a
> python3 environment to use. I'll try and throw something together to
> test that shortly.
>
> Second, to have the same environment locally:
> $ sudo docker pull trini/u-boot-gitlab-ci-runner:bionic-20191010-20Oct2019
>
> And change the tag to whatever .gitlab-ci.yml says we use. That one is
> what the test.py series brings us up to, and is newer than your failing
> job.
>
> $ sudo docker run --rm -it -v /your/local/u-boot/clone/..:/build/u-boot \
> trini/u-boot-gitlab-ci-runner:bionic-20191010-20Oct2019 /bin/bash
>
> And don't pass --rm if you don't want the image deleted once you exit.
> That will put you in a bash prompt as the user we run everything as.
> Follow the steps in the job as described in .gitlab-ci.yml to do a build
> by hand.
>
> And yes, you want to pass in one level before your clone so that in the
> container you have /build/u-boot/u-boot
Thank you, that is very helpful!
It seems like the default file encoding for open() is different. I'll
try again and hopefully send a v2 series soon.
Regards,
Simon
^ permalink raw reply [flat|nested] 23+ messages in thread
* [U-Boot] [PATCH v2 03/15] buildman: Convert to Python 3
2019-10-30 15:12 ` Simon Glass
@ 2019-10-30 15:58 ` Tom Rini
2019-10-30 16:15 ` Simon Glass
0 siblings, 1 reply; 23+ messages in thread
From: Tom Rini @ 2019-10-30 15:58 UTC (permalink / raw)
To: u-boot
On Wed, Oct 30, 2019 at 09:12:39AM -0600, Simon Glass wrote:
> Hi Tom,
>
> On Wed, 30 Oct 2019 at 07:15, Tom Rini <trini@konsulko.com> wrote:
> >
> > On Tue, Oct 29, 2019 at 07:47:55PM -0600, Simon Glass wrote:
> > > Hi Tom,
> > >
> > > On Thu, 24 Oct 2019 at 19:04, Simon Glass <sjg@chromium.org> wrote:
> > > >
> > > > Convert buildman to Python 3 and make it use that, to meet the 2020
> > > > deadline.
> > > >
> > > > Signed-off-by: Simon Glass <sjg@chromium.org>
> > > > ---
> > > >
> > > > Changes in v2: None
> > > >
> > > > tools/buildman/board.py | 7 +--
> > > > tools/buildman/bsettings.py | 20 +++----
> > > > tools/buildman/builder.py | 45 ++++++++--------
> > > > tools/buildman/builderthread.py | 24 ++++-----
> > > > tools/buildman/buildman.py | 10 ++--
> > > > tools/buildman/control.py | 40 +++++++-------
> > > > tools/buildman/func_test.py | 16 +++---
> > > > tools/buildman/test.py | 22 ++++----
> > > > tools/buildman/toolchain.py | 93 +++++++++++++++++----------------
> > > > 9 files changed, 139 insertions(+), 138 deletions(-)
> > >
> > > I am a bit mystified by this:
> > >
> > > https://gitlab.denx.de/u-boot/custodians/u-boot-dm/-/jobs/25186
> > >
> > > It runs locally and I cannot see why it is not using unicode. Are
> > > there instructions on how to get to the same Python version as gitlab?
> > > I fear a steep learning curve coming.
> >
> > First, I think the problem is this series will also need the test.py
> > python3 series as that does a few more things to get and provide a
> > python3 environment to use. I'll try and throw something together to
> > test that shortly.
> >
> > Second, to have the same environment locally:
> > $ sudo docker pull trini/u-boot-gitlab-ci-runner:bionic-20191010-20Oct2019
> >
> > And change the tag to whatever .gitlab-ci.yml says we use. That one is
> > what the test.py series brings us up to, and is newer than your failing
> > job.
> >
> > $ sudo docker run --rm -it -v /your/local/u-boot/clone/..:/build/u-boot \
> > trini/u-boot-gitlab-ci-runner:bionic-20191010-20Oct2019 /bin/bash
> >
> > And don't pass --rm if you don't want the image deleted once you exit.
> > That will put you in a bash prompt as the user we run everything as.
> > Follow the steps in the job as described in .gitlab-ci.yml to do a build
> > by hand.
> >
> > And yes, you want to pass in one level before your clone so that in the
> > container you have /build/u-boot/u-boot
>
> Thank you, that is very helpful!
>
> It seems like the default file encoding for open() is different. I'll
> try again and hopefully send a v2 series soon.
Ah, yeah. That reminded me of
http://patchwork.ozlabs.org/patch/1183213/ which ends up with telling
open to use utf-8 and then possibly some other places we need to
decode() and allow for errors to replace characters as needed rather
than be fatal. I think you might have a case of the latter where it's
found 'ä' or similar in the log and can't just turn it to ascii without
being allowed to replace.
--
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20191030/4fa89626/attachment.sig>
^ permalink raw reply [flat|nested] 23+ messages in thread
* [U-Boot] [PATCH v2 03/15] buildman: Convert to Python 3
2019-10-30 15:58 ` Tom Rini
@ 2019-10-30 16:15 ` Simon Glass
2019-10-31 3:07 ` Simon Glass
0 siblings, 1 reply; 23+ messages in thread
From: Simon Glass @ 2019-10-30 16:15 UTC (permalink / raw)
To: u-boot
Hi Tom,
On Wed, 30 Oct 2019 at 09:58, Tom Rini <trini@konsulko.com> wrote:
>
> On Wed, Oct 30, 2019 at 09:12:39AM -0600, Simon Glass wrote:
> > Hi Tom,
> >
> > On Wed, 30 Oct 2019 at 07:15, Tom Rini <trini@konsulko.com> wrote:
> > >
> > > On Tue, Oct 29, 2019 at 07:47:55PM -0600, Simon Glass wrote:
> > > > Hi Tom,
> > > >
> > > > On Thu, 24 Oct 2019 at 19:04, Simon Glass <sjg@chromium.org> wrote:
> > > > >
> > > > > Convert buildman to Python 3 and make it use that, to meet the 2020
> > > > > deadline.
> > > > >
> > > > > Signed-off-by: Simon Glass <sjg@chromium.org>
> > > > > ---
> > > > >
> > > > > Changes in v2: None
> > > > >
> > > > > tools/buildman/board.py | 7 +--
> > > > > tools/buildman/bsettings.py | 20 +++----
> > > > > tools/buildman/builder.py | 45 ++++++++--------
> > > > > tools/buildman/builderthread.py | 24 ++++-----
> > > > > tools/buildman/buildman.py | 10 ++--
> > > > > tools/buildman/control.py | 40 +++++++-------
> > > > > tools/buildman/func_test.py | 16 +++---
> > > > > tools/buildman/test.py | 22 ++++----
> > > > > tools/buildman/toolchain.py | 93 +++++++++++++++++----------------
> > > > > 9 files changed, 139 insertions(+), 138 deletions(-)
> > > >
> > > > I am a bit mystified by this:
> > > >
> > > > https://gitlab.denx.de/u-boot/custodians/u-boot-dm/-/jobs/25186
> > > >
> > > > It runs locally and I cannot see why it is not using unicode. Are
> > > > there instructions on how to get to the same Python version as gitlab?
> > > > I fear a steep learning curve coming.
> > >
> > > First, I think the problem is this series will also need the test.py
> > > python3 series as that does a few more things to get and provide a
> > > python3 environment to use. I'll try and throw something together to
> > > test that shortly.
> > >
> > > Second, to have the same environment locally:
> > > $ sudo docker pull trini/u-boot-gitlab-ci-runner:bionic-20191010-20Oct2019
> > >
> > > And change the tag to whatever .gitlab-ci.yml says we use. That one is
> > > what the test.py series brings us up to, and is newer than your failing
> > > job.
> > >
> > > $ sudo docker run --rm -it -v /your/local/u-boot/clone/..:/build/u-boot \
> > > trini/u-boot-gitlab-ci-runner:bionic-20191010-20Oct2019 /bin/bash
> > >
> > > And don't pass --rm if you don't want the image deleted once you exit.
> > > That will put you in a bash prompt as the user we run everything as.
> > > Follow the steps in the job as described in .gitlab-ci.yml to do a build
> > > by hand.
> > >
> > > And yes, you want to pass in one level before your clone so that in the
> > > container you have /build/u-boot/u-boot
> >
> > Thank you, that is very helpful!
> >
> > It seems like the default file encoding for open() is different. I'll
> > try again and hopefully send a v2 series soon.
>
> Ah, yeah. That reminded me of
> http://patchwork.ozlabs.org/patch/1183213/ which ends up with telling
> open to use utf-8 and then possibly some other places we need to
> decode() and allow for errors to replace characters as needed rather
> than be fatal. I think you might have a case of the latter where it's
> found 'ä' or similar in the log and can't just turn it to ascii without
> being allowed to replace.
I think I have that patch. My tree is building here:
https://gitlab.denx.de/u-boot/custodians/u-boot-dm/pipelines/1158
But perhaps I picked up an earlier version? Is it close to being applied?
Regards,
Simon
^ permalink raw reply [flat|nested] 23+ messages in thread
* [U-Boot] [PATCH v2 03/15] buildman: Convert to Python 3
2019-10-30 16:15 ` Simon Glass
@ 2019-10-31 3:07 ` Simon Glass
2019-10-31 3:11 ` Tom Rini
0 siblings, 1 reply; 23+ messages in thread
From: Simon Glass @ 2019-10-31 3:07 UTC (permalink / raw)
To: u-boot
Hi Tom,
On Wed, 30 Oct 2019 at 10:15, Simon Glass <sjg@chromium.org> wrote:
>
> Hi Tom,
>
> On Wed, 30 Oct 2019 at 09:58, Tom Rini <trini@konsulko.com> wrote:
> >
> > On Wed, Oct 30, 2019 at 09:12:39AM -0600, Simon Glass wrote:
> > > Hi Tom,
> > >
> > > On Wed, 30 Oct 2019 at 07:15, Tom Rini <trini@konsulko.com> wrote:
> > > >
> > > > On Tue, Oct 29, 2019 at 07:47:55PM -0600, Simon Glass wrote:
> > > > > Hi Tom,
> > > > >
> > > > > On Thu, 24 Oct 2019 at 19:04, Simon Glass <sjg@chromium.org> wrote:
> > > > > >
> > > > > > Convert buildman to Python 3 and make it use that, to meet the 2020
> > > > > > deadline.
> > > > > >
> > > > > > Signed-off-by: Simon Glass <sjg@chromium.org>
> > > > > > ---
> > > > > >
> > > > > > Changes in v2: None
> > > > > >
> > > > > > tools/buildman/board.py | 7 +--
> > > > > > tools/buildman/bsettings.py | 20 +++----
> > > > > > tools/buildman/builder.py | 45 ++++++++--------
> > > > > > tools/buildman/builderthread.py | 24 ++++-----
> > > > > > tools/buildman/buildman.py | 10 ++--
> > > > > > tools/buildman/control.py | 40 +++++++-------
> > > > > > tools/buildman/func_test.py | 16 +++---
> > > > > > tools/buildman/test.py | 22 ++++----
> > > > > > tools/buildman/toolchain.py | 93 +++++++++++++++++----------------
> > > > > > 9 files changed, 139 insertions(+), 138 deletions(-)
> > > > >
> > > > > I am a bit mystified by this:
> > > > >
> > > > > https://gitlab.denx.de/u-boot/custodians/u-boot-dm/-/jobs/25186
> > > > >
> > > > > It runs locally and I cannot see why it is not using unicode. Are
> > > > > there instructions on how to get to the same Python version as gitlab?
> > > > > I fear a steep learning curve coming.
> > > >
> > > > First, I think the problem is this series will also need the test.py
> > > > python3 series as that does a few more things to get and provide a
> > > > python3 environment to use. I'll try and throw something together to
> > > > test that shortly.
> > > >
> > > > Second, to have the same environment locally:
> > > > $ sudo docker pull trini/u-boot-gitlab-ci-runner:bionic-20191010-20Oct2019
> > > >
> > > > And change the tag to whatever .gitlab-ci.yml says we use. That one is
> > > > what the test.py series brings us up to, and is newer than your failing
> > > > job.
> > > >
> > > > $ sudo docker run --rm -it -v /your/local/u-boot/clone/..:/build/u-boot \
> > > > trini/u-boot-gitlab-ci-runner:bionic-20191010-20Oct2019 /bin/bash
> > > >
> > > > And don't pass --rm if you don't want the image deleted once you exit.
> > > > That will put you in a bash prompt as the user we run everything as.
> > > > Follow the steps in the job as described in .gitlab-ci.yml to do a build
> > > > by hand.
> > > >
> > > > And yes, you want to pass in one level before your clone so that in the
> > > > container you have /build/u-boot/u-boot
> > >
> > > Thank you, that is very helpful!
> > >
> > > It seems like the default file encoding for open() is different. I'll
> > > try again and hopefully send a v2 series soon.
> >
> > Ah, yeah. That reminded me of
> > http://patchwork.ozlabs.org/patch/1183213/ which ends up with telling
> > open to use utf-8 and then possibly some other places we need to
> > decode() and allow for errors to replace characters as needed rather
> > than be fatal. I think you might have a case of the latter where it's
> > found 'ä' or similar in the log and can't just turn it to ascii without
> > being allowed to replace.
>
> I think I have that patch. My tree is building here:
>
> https://gitlab.denx.de/u-boot/custodians/u-boot-dm/pipelines/1158
>
> But perhaps I picked up an earlier version? Is it close to being applied?
Good news is that I got a passing build on gitlab. I should be able
send the series out tomorrow. The last patch is RFC but everything
else should be ready for review.
Regards,
Simon
^ permalink raw reply [flat|nested] 23+ messages in thread
* [U-Boot] [PATCH v2 03/15] buildman: Convert to Python 3
2019-10-31 3:07 ` Simon Glass
@ 2019-10-31 3:11 ` Tom Rini
0 siblings, 0 replies; 23+ messages in thread
From: Tom Rini @ 2019-10-31 3:11 UTC (permalink / raw)
To: u-boot
On Wed, Oct 30, 2019 at 09:07:05PM -0600, Simon Glass wrote:
> Hi Tom,
>
> On Wed, 30 Oct 2019 at 10:15, Simon Glass <sjg@chromium.org> wrote:
> >
> > Hi Tom,
> >
> > On Wed, 30 Oct 2019 at 09:58, Tom Rini <trini@konsulko.com> wrote:
> > >
> > > On Wed, Oct 30, 2019 at 09:12:39AM -0600, Simon Glass wrote:
> > > > Hi Tom,
> > > >
> > > > On Wed, 30 Oct 2019 at 07:15, Tom Rini <trini@konsulko.com> wrote:
> > > > >
> > > > > On Tue, Oct 29, 2019 at 07:47:55PM -0600, Simon Glass wrote:
> > > > > > Hi Tom,
> > > > > >
> > > > > > On Thu, 24 Oct 2019 at 19:04, Simon Glass <sjg@chromium.org> wrote:
> > > > > > >
> > > > > > > Convert buildman to Python 3 and make it use that, to meet the 2020
> > > > > > > deadline.
> > > > > > >
> > > > > > > Signed-off-by: Simon Glass <sjg@chromium.org>
> > > > > > > ---
> > > > > > >
> > > > > > > Changes in v2: None
> > > > > > >
> > > > > > > tools/buildman/board.py | 7 +--
> > > > > > > tools/buildman/bsettings.py | 20 +++----
> > > > > > > tools/buildman/builder.py | 45 ++++++++--------
> > > > > > > tools/buildman/builderthread.py | 24 ++++-----
> > > > > > > tools/buildman/buildman.py | 10 ++--
> > > > > > > tools/buildman/control.py | 40 +++++++-------
> > > > > > > tools/buildman/func_test.py | 16 +++---
> > > > > > > tools/buildman/test.py | 22 ++++----
> > > > > > > tools/buildman/toolchain.py | 93 +++++++++++++++++----------------
> > > > > > > 9 files changed, 139 insertions(+), 138 deletions(-)
> > > > > >
> > > > > > I am a bit mystified by this:
> > > > > >
> > > > > > https://gitlab.denx.de/u-boot/custodians/u-boot-dm/-/jobs/25186
> > > > > >
> > > > > > It runs locally and I cannot see why it is not using unicode. Are
> > > > > > there instructions on how to get to the same Python version as gitlab?
> > > > > > I fear a steep learning curve coming.
> > > > >
> > > > > First, I think the problem is this series will also need the test.py
> > > > > python3 series as that does a few more things to get and provide a
> > > > > python3 environment to use. I'll try and throw something together to
> > > > > test that shortly.
> > > > >
> > > > > Second, to have the same environment locally:
> > > > > $ sudo docker pull trini/u-boot-gitlab-ci-runner:bionic-20191010-20Oct2019
> > > > >
> > > > > And change the tag to whatever .gitlab-ci.yml says we use. That one is
> > > > > what the test.py series brings us up to, and is newer than your failing
> > > > > job.
> > > > >
> > > > > $ sudo docker run --rm -it -v /your/local/u-boot/clone/..:/build/u-boot \
> > > > > trini/u-boot-gitlab-ci-runner:bionic-20191010-20Oct2019 /bin/bash
> > > > >
> > > > > And don't pass --rm if you don't want the image deleted once you exit.
> > > > > That will put you in a bash prompt as the user we run everything as.
> > > > > Follow the steps in the job as described in .gitlab-ci.yml to do a build
> > > > > by hand.
> > > > >
> > > > > And yes, you want to pass in one level before your clone so that in the
> > > > > container you have /build/u-boot/u-boot
> > > >
> > > > Thank you, that is very helpful!
> > > >
> > > > It seems like the default file encoding for open() is different. I'll
> > > > try again and hopefully send a v2 series soon.
> > >
> > > Ah, yeah. That reminded me of
> > > http://patchwork.ozlabs.org/patch/1183213/ which ends up with telling
> > > open to use utf-8 and then possibly some other places we need to
> > > decode() and allow for errors to replace characters as needed rather
> > > than be fatal. I think you might have a case of the latter where it's
> > > found 'ä' or similar in the log and can't just turn it to ascii without
> > > being allowed to replace.
> >
> > I think I have that patch. My tree is building here:
> >
> > https://gitlab.denx.de/u-boot/custodians/u-boot-dm/pipelines/1158
> >
> > But perhaps I picked up an earlier version? Is it close to being applied?
>
> Good news is that I got a passing build on gitlab. I should be able
> send the series out tomorrow. The last patch is RFC but everything
> else should be ready for review.
Great, thanks. I'm testing the test.py series to merge currently.
--
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20191030/efb797bd/attachment.sig>
^ permalink raw reply [flat|nested] 23+ messages in thread
end of thread, other threads:[~2019-10-31 3:11 UTC | newest]
Thread overview: 23+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-10-25 1:03 [U-Boot] [PATCH v2 00/15] scripts: Convert to Python 3 Simon Glass
2019-10-25 1:03 ` [U-Boot] [PATCH v2 01/15] patman: Adjust 'command' to return strings instead of bytes Simon Glass
2019-10-25 1:03 ` [U-Boot] [PATCH v2 02/15] patman: Move to use Python 3 Simon Glass
2019-10-25 1:03 ` [U-Boot] [PATCH v2 03/15] buildman: Convert to " Simon Glass
2019-10-30 1:47 ` Simon Glass
2019-10-30 13:15 ` Tom Rini
2019-10-30 15:12 ` Simon Glass
2019-10-30 15:58 ` Tom Rini
2019-10-30 16:15 ` Simon Glass
2019-10-31 3:07 ` Simon Glass
2019-10-31 3:11 ` Tom Rini
2019-10-25 1:03 ` [U-Boot] [PATCH v2 04/15] test_fdt: Move to use " Simon Glass
2019-10-25 1:03 ` [U-Boot] [PATCH v2 05/15] test_dtoc: " Simon Glass
2019-10-25 1:03 ` [U-Boot] [PATCH v2 06/15] microcode_tool: Convert to " Simon Glass
2019-10-25 1:03 ` [U-Boot] [PATCH v2 07/15] move_config: " Simon Glass
2019-10-25 1:03 ` [U-Boot] [PATCH v2 08/15] rkmux: " Simon Glass
2019-10-25 1:03 ` [U-Boot] [PATCH v2 09/15] pylibfdt: " Simon Glass
2019-10-25 1:03 ` [U-Boot] [PATCH v2 10/15] pylibfdt: Sync up with upstream Simon Glass
2019-10-25 1:03 ` [U-Boot] [PATCH v2 11/15] pylibfdt: Correct the type for fdt_property_stub() Simon Glass
2019-10-25 1:03 ` [U-Boot] [PATCH v2 12/15] binman: Remember the pre-reset entry size Simon Glass
2019-10-25 1:03 ` [U-Boot] [PATCH v2 13/15] binman: Convert a few tests to Python 3 Simon Glass
2019-10-25 1:03 ` [U-Boot] [PATCH v2 14/15] dtoc: Convert fdt.py " Simon Glass
2019-10-25 1:03 ` [U-Boot] [PATCH v2 15/15] binman: Move to use " Simon Glass
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.