[U-Boot] [PATCH v2 00/10] Python 3.x support for patman & dtoc

[U-Boot] [PATCH v2 00/10] Python 3.x support for patman & dtoc

[Paul Burton]

This series makes patman & dtoc run on python 3.x, which allows them to
work without hacks on distributions where the python binary is python
3.x. All changes are made in a way that continue to work on python 2.x,
so both tools should work on either major version of python.

Paul Burton (10):
  patman: Replace tabs with spaces
  patman: Make print statements python 3.x safe
  patman: Make exception handling python 3.x safe
  patman: Import 'configparser' lower case to be  python 3.x safe
  patman: Decode stdout/stderr as utf8, be python 3.x safe
  patman: Use items() to iterate over dictionaries
  patman: Fix doctest StringIO import for python 3.x
  dtoc: Use items() to iterate over dictionaries in python 3.x
  dtoc: Decode strings for struct.unpack on python 3.x
  dtoc: Make integer division python 3.x safe

 tools/dtoc/dtoc.py              | 10 +++++-----
 tools/dtoc/fdt_fallback.py      |  2 +-
 tools/dtoc/fdt_util.py          |  3 +++
 tools/patman/checkpatch.py      | 16 ++++++++--------
 tools/patman/command.py         |  2 +-
 tools/patman/cros_subprocess.py |  6 +++---
 tools/patman/get_maintainer.py  |  2 +-
 tools/patman/gitutil.py         | 18 +++++++++---------
 tools/patman/patchstream.py     |  6 +++---
 tools/patman/patman.py          | 12 ++++++------
 tools/patman/series.py          | 42 +++++++++++++++++++++--------------------
 tools/patman/settings.py        | 37 ++++++++++++++++++++++--------------
 tools/patman/terminal.py        | 12 +++++++-----
 tools/patman/test.py            |  2 +-
 14 files changed, 93 insertions(+), 77 deletions(-)

-- 
2.10.0

[U-Boot] [PATCH v2 01/10] patman: Replace tabs with spaces

[Paul Burton]

In preparation for running on python 3.x, which will refuse to run
scripts which mix tabs & spaces for indentation, replace 2 tab
characters present in series.py with spaces.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
Acked-by: Simon Glass <sjg@chromium.org>
---

Changes in v2: None

 tools/patman/series.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tools/patman/series.py b/tools/patman/series.py
index cc6f80b..1ce30c6 100644
--- a/tools/patman/series.py
+++ b/tools/patman/series.py
@@ -225,7 +225,7 @@ class Series(dict):
                                                raise_on_error=raise_on_error)
             list += gitutil.BuildEmailList(commit.cc_list,
                                            raise_on_error=raise_on_error)
-	    if add_maintainers:
+            if add_maintainers:
                 list += get_maintainer.GetMaintainer(commit.patch)
             all_ccs += list
             print >>fd, commit.patch, ', '.join(set(list))
@@ -259,7 +259,7 @@ class Series(dict):
         """
         git_prefix = gitutil.GetDefaultSubjectPrefix()
         if git_prefix:
-	    git_prefix = '%s][' % git_prefix
+            git_prefix = '%s][' % git_prefix
         else:
             git_prefix = ''
 
-- 
2.10.0

[U-Boot] [PATCH v2 02/10] patman: Make print statements python 3.x safe

[Paul Burton]

In python 3.x, print must be used as a function call. Convert all print
statements to the function call style, importing from __future__ where
we print with no trailing newline or print to a file object.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
Acked-by: Simon Glass <sjg@chromium.org>
---

Changes in v2: None

 tools/patman/checkpatch.py     | 16 ++++++++--------
 tools/patman/get_maintainer.py |  2 +-
 tools/patman/gitutil.py        |  6 +++---
 tools/patman/patchstream.py    |  6 +++---
 tools/patman/patman.py         | 12 ++++++------
 tools/patman/series.py         | 38 ++++++++++++++++++++------------------
 tools/patman/settings.py       | 16 +++++++++-------
 tools/patman/terminal.py       | 12 +++++++-----
 tools/patman/test.py           |  2 +-
 9 files changed, 58 insertions(+), 52 deletions(-)

diff --git a/tools/patman/checkpatch.py b/tools/patman/checkpatch.py
index 3eef6de..be78fc5 100644
--- a/tools/patman/checkpatch.py
+++ b/tools/patman/checkpatch.py
@@ -83,7 +83,7 @@ def CheckPatch(fname, verbose=False):
 
     for line in result.stdout.splitlines():
         if verbose:
-            print line
+            print(line)
 
         # A blank line indicates the end of a message
         if not line and item:
@@ -151,17 +151,17 @@ def CheckPatches(verbose, args):
             error_count += result.errors
             warning_count += result.warnings
             check_count += result.checks
-            print '%d errors, %d warnings, %d checks for %s:' % (result.errors,
-                    result.warnings, result.checks, col.Color(col.BLUE, fname))
+            print('%d errors, %d warnings, %d checks for %s:' % (result.errors,
+                    result.warnings, result.checks, col.Color(col.BLUE, fname)))
             if (len(result.problems) != result.errors + result.warnings +
                     result.checks):
-                print "Internal error: some problems lost"
+                print("Internal error: some problems lost")
             for item in result.problems:
-                print GetWarningMsg(col, item.get('type', '<unknown>'),
+                print(GetWarningMsg(col, item.get('type', '<unknown>'),
                         item.get('file', '<unknown>'),
-                        item.get('line', 0), item.get('msg', 'message'))
+                        item.get('line', 0), item.get('msg', 'message')))
             print
-            #print stdout
+            #print(stdout)
     if error_count or warning_count or check_count:
         str = 'checkpatch.pl found %d error(s), %d warning(s), %d checks(s)'
         color = col.GREEN
@@ -169,6 +169,6 @@ def CheckPatches(verbose, args):
             color = col.YELLOW
         if error_count:
             color = col.RED
-        print col.Color(color, str % (error_count, warning_count, check_count))
+        print(col.Color(color, str % (error_count, warning_count, check_count)))
         return False
     return True
diff --git a/tools/patman/get_maintainer.py b/tools/patman/get_maintainer.py
index 00b4939..2deb5db 100644
--- a/tools/patman/get_maintainer.py
+++ b/tools/patman/get_maintainer.py
@@ -40,7 +40,7 @@ def GetMaintainer(fname, verbose=False):
     get_maintainer = FindGetMaintainer()
     if not get_maintainer:
         if verbose:
-            print "WARNING: Couldn't find get_maintainer.pl"
+            print("WARNING: Couldn't find get_maintainer.pl")
         return []
 
     stdout = command.Output(get_maintainer, '--norolestats', fname)
diff --git a/tools/patman/gitutil.py b/tools/patman/gitutil.py
index bb7c9e0..c0fe093 100644
--- a/tools/patman/gitutil.py
+++ b/tools/patman/gitutil.py
@@ -491,7 +491,7 @@ def LookupEmail(lookup_name, alias=None, raise_on_error=True, level=0):
         if raise_on_error:
             raise OSError, msg
         else:
-            print col.Color(col.RED, msg)
+            print(col.Color(col.RED, msg))
             return out_list
 
     if lookup_name:
@@ -500,7 +500,7 @@ def LookupEmail(lookup_name, alias=None, raise_on_error=True, level=0):
             if raise_on_error:
                 raise ValueError, msg
             else:
-                print col.Color(col.RED, msg)
+                print(col.Color(col.RED, msg))
                 return out_list
         for item in alias[lookup_name]:
             todo = LookupEmail(item, alias, raise_on_error, level + 1)
@@ -508,7 +508,7 @@ def LookupEmail(lookup_name, alias=None, raise_on_error=True, level=0):
                 if not new_item in out_list:
                     out_list.append(new_item)
 
-    #print "No match for alias '%s'" % lookup_name
+    #print("No match for alias '%s'" % lookup_name)
     return out_list
 
 def GetTopLevel():
diff --git a/tools/patman/patchstream.py b/tools/patman/patchstream.py
index 69d5cfb..cd4667f 100644
--- a/tools/patman/patchstream.py
+++ b/tools/patman/patchstream.py
@@ -480,12 +480,12 @@ def FixPatches(series, fnames):
         commit.patch = fname
         result = FixPatch(backup_dir, fname, series, commit)
         if result:
-            print '%d warnings for %s:' % (len(result), fname)
+            print('%d warnings for %s:' % (len(result), fname))
             for warn in result:
-                print '\t', warn
+                print('\t', warn)
             print
         count += 1
-    print 'Cleaned %d patches' % count
+    print('Cleaned %d patches' % count)
     return series
 
 def InsertCoverLetter(fname, series, count):
diff --git a/tools/patman/patman.py b/tools/patman/patman.py
index fe50eb4..fdbee67 100755
--- a/tools/patman/patman.py
+++ b/tools/patman/patman.py
@@ -93,11 +93,11 @@ elif options.test:
         suite.run(result)
 
     # TODO: Surely we can just 'print' result?
-    print result
+    print(result)
     for test, err in result.errors:
-        print err
+        print(err)
     for test, err in result.failures:
-        print err
+        print(err)
 
 # Called from git with a patch filename as argument
 # Printout a list of additional CC recipients for this patch
@@ -110,7 +110,7 @@ elif options.cc_cmd:
             for cc in match.group(2).split(', '):
                 cc = cc.strip()
                 if cc:
-                    print cc
+                    print(cc)
     fd.close()
 
 elif options.full_help:
@@ -166,12 +166,12 @@ else:
                 options.dry_run, not options.ignore_bad_tags, cc_file,
                 in_reply_to=options.in_reply_to, thread=options.thread)
     else:
-        print col.Color(col.RED, "Not sending emails due to errors/warnings")
+        print(col.Color(col.RED, "Not sending emails due to errors/warnings"))
 
     # For a dry run, just show our actions as a sanity check
     if options.dry_run:
         series.ShowActions(args, cmd, options.process_tags)
         if not its_a_go:
-            print col.Color(col.RED, "Email would not be sent")
+            print(col.Color(col.RED, "Email would not be sent"))
 
     os.remove(cc_file)
diff --git a/tools/patman/series.py b/tools/patman/series.py
index 1ce30c6..38a452e 100644
--- a/tools/patman/series.py
+++ b/tools/patman/series.py
@@ -3,6 +3,8 @@
 # SPDX-License-Identifier:	GPL-2.0+
 #
 
+from __future__ import print_function
+
 import itertools
 import os
 
@@ -101,38 +103,38 @@ class Series(dict):
         cc_set = set(gitutil.BuildEmailList(self.cc));
 
         col = terminal.Color()
-        print 'Dry run, so not doing much. But I would do this:'
-        print
-        print 'Send a total of %d patch%s with %scover letter.' % (
+        print('Dry run, so not doing much. But I would do this:')
+        print()
+        print('Send a total of %d patch%s with %scover letter.' % (
                 len(args), '' if len(args) == 1 else 'es',
-                self.get('cover') and 'a ' or 'no ')
+                self.get('cover') and 'a ' or 'no '))
 
         # TODO: Colour the patches according to whether they passed checks
         for upto in range(len(args)):
             commit = self.commits[upto]
-            print col.Color(col.GREEN, '   %s' % args[upto])
+            print(col.Color(col.GREEN, '   %s' % args[upto]))
             cc_list = list(self._generated_cc[commit.patch])
             for email in set(cc_list) - to_set - cc_set:
                 if email == None:
                     email = col.Color(col.YELLOW, "<alias '%s' not found>"
                             % tag)
                 if email:
-                    print '      Cc: ',email
+                    print('      Cc: ', email)
         print
         for item in to_set:
-            print 'To:\t ', item
+            print('To:\t ', item)
         for item in cc_set - to_set:
-            print 'Cc:\t ', item
-        print 'Version: ', self.get('version')
-        print 'Prefix:\t ', self.get('prefix')
+            print('Cc:\t ', item)
+        print('Version: ', self.get('version'))
+        print('Prefix:\t ', self.get('prefix'))
         if self.cover:
-            print 'Cover: %d lines' % len(self.cover)
+            print('Cover: %d lines' % len(self.cover))
             cover_cc = gitutil.BuildEmailList(self.get('cover_cc', ''))
             all_ccs = itertools.chain(cover_cc, *self._generated_cc.values())
             for email in set(all_ccs) - to_set - cc_set:
-                    print '      Cc: ',email
+                    print('      Cc: ', email)
         if cmd:
-            print 'Git command: %s' % cmd
+            print('Git command: %s' % cmd)
 
     def MakeChangeLog(self, commit):
         """Create a list of changes for each version.
@@ -191,13 +193,13 @@ class Series(dict):
                 else:
                     if version > 1:
                         str = 'Change log missing for v%d' % version
-                        print col.Color(col.RED, str)
+                        print(col.Color(col.RED, str))
             for version in changes_copy:
                 str = 'Change log for unknown version v%d' % version
-                print col.Color(col.RED, str)
+                print(col.Color(col.RED, str))
         elif self.changes:
             str = 'Change log exists, but no version is set'
-            print col.Color(col.RED, str)
+            print(col.Color(col.RED, str))
 
     def MakeCcFile(self, process_tags, cover_fname, raise_on_error,
                    add_maintainers):
@@ -228,12 +230,12 @@ class Series(dict):
             if add_maintainers:
                 list += get_maintainer.GetMaintainer(commit.patch)
             all_ccs += list
-            print >>fd, commit.patch, ', '.join(set(list))
+            print(commit.patch, ', '.join(set(list)), file=fd)
             self._generated_cc[commit.patch] = list
 
         if cover_fname:
             cover_cc = gitutil.BuildEmailList(self.get('cover_cc', ''))
-            print >>fd, cover_fname, ', '.join(set(cover_cc + all_ccs))
+            print(cover_fname, ', '.join(set(cover_cc + all_ccs)), file=fd)
 
         fd.close()
         return fname
diff --git a/tools/patman/settings.py b/tools/patman/settings.py
index ba2a68f..01f6c38 100644
--- a/tools/patman/settings.py
+++ b/tools/patman/settings.py
@@ -3,6 +3,8 @@
 # SPDX-License-Identifier:	GPL-2.0+
 #
 
+from __future__ import print_function
+
 import ConfigParser
 import os
 import re
@@ -156,7 +158,7 @@ def ReadGitAliases(fname):
     try:
         fd = open(fname, 'r')
     except IOError:
-        print "Warning: Cannot find alias file '%s'" % fname
+        print("Warning: Cannot find alias file '%s'" % fname)
         return
 
     re_line = re.compile('alias\s+(\S+)\s+(.*)')
@@ -167,7 +169,7 @@ def ReadGitAliases(fname):
 
         m = re_line.match(line)
         if not m:
-            print "Warning: Alias file line '%s' not understood" % line
+            print("Warning: Alias file line '%s' not understood" % line)
             continue
 
         list = alias.get(m.group(1), [])
@@ -200,10 +202,10 @@ def CreatePatmanConfigFile(config_fname):
     try:
         f = open(config_fname, 'w')
     except IOError:
-        print "Couldn't create patman config file\n"
+        print("Couldn't create patman config file\n")
         raise
 
-    print >>f, "[alias]\nme: %s <%s>" % (name, email)
+    print("[alias]\nme: %s <%s>" % (name, email), file=f)
     f.close();
 
 def _UpdateDefaults(parser, config):
@@ -233,7 +235,7 @@ def _UpdateDefaults(parser, config):
                 val = config.getint('settings', name)
             parser.set_default(name, val)
         else:
-            print "WARNING: Unknown setting %s" % name
+            print("WARNING: Unknown setting %s" % name)
 
 def _ReadAliasFile(fname):
     """Read in the U-Boot git alias file if it exists.
@@ -258,7 +260,7 @@ def _ReadAliasFile(fname):
                     continue
                 alias[words[1]] = [s.strip() for s in words[2].split(',')]
         if bad_line:
-            print bad_line
+            print(bad_line)
 
 def Setup(parser, project_name, config_fname=''):
     """Set up the settings module by reading config files.
@@ -276,7 +278,7 @@ def Setup(parser, project_name, config_fname=''):
         config_fname = '%s/.patman' % os.getenv('HOME')
 
     if not os.path.exists(config_fname):
-        print "No config file found ~/.patman\nCreating one...\n"
+        print("No config file found ~/.patman\nCreating one...\n")
         CreatePatmanConfigFile(config_fname)
 
     config.read(config_fname)
diff --git a/tools/patman/terminal.py b/tools/patman/terminal.py
index e78a7c1..d8a9e4c 100644
--- a/tools/patman/terminal.py
+++ b/tools/patman/terminal.py
@@ -8,6 +8,8 @@
 This module handles terminal interaction including ANSI color codes.
 """
 
+from __future__ import print_function
+
 import os
 import sys
 
@@ -52,9 +54,9 @@ def Print(text='', newline=True, colour=None):
         if colour:
             col = Color()
             text = col.Color(colour, text)
-        print text,
+        print(text, end='')
         if newline:
-            print
+            print()
 
 def SetPrintTestMode():
     """Go into test mode, where all printing is recorded"""
@@ -79,11 +81,11 @@ def EchoPrintTestLines():
     for line in print_test_list:
         if line.colour:
             col = Color()
-            print col.Color(line.colour, line.text),
+            print(col.Color(line.colour, line.text), end='')
         else:
-            print line.text,
+            print(line.text, end='')
         if line.newline:
-            print
+            print()
 
 
 class Color(object):
diff --git a/tools/patman/test.py b/tools/patman/test.py
index e8f7472..8c39f66 100644
--- a/tools/patman/test.py
+++ b/tools/patman/test.py
@@ -181,7 +181,7 @@ index 0000000..2234c87
         elif data_type == 'indent':
             indent = tab
         else:
-            print 'not implemented'
+            print('not implemented')
         return data % (signoff, tab, indent, tab)
 
     def SetupData(self, data_type):
-- 
2.10.0

[U-Boot] [PATCH v2 03/10] patman: Make exception handling python 3.x safe

[Paul Burton]

Syntax for exception handling is a little more strict in python 3.x.
Convert all uses to a form accepted by both python 2.x & python 3.x.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
Acked-by: Simon Glass <sjg@chromium.org>
---

Changes in v2: None

 tools/patman/command.py         |  2 +-
 tools/patman/cros_subprocess.py |  2 +-
 tools/patman/gitutil.py         | 12 ++++++------
 3 files changed, 8 insertions(+), 8 deletions(-)

diff --git a/tools/patman/command.py b/tools/patman/command.py
index d1f0ca5..bebc495 100644
--- a/tools/patman/command.py
+++ b/tools/patman/command.py
@@ -85,7 +85,7 @@ def RunPipe(pipe_list, infile=None, outfile=None,
 
         try:
             last_pipe = cros_subprocess.Popen(cmd, cwd=cwd, **kwargs)
-        except Exception, err:
+        except Exception as err:
             result.exception = err
             if raise_on_error:
                 raise Exception("Error running '%s': %s" % (user_pipestr, str))
diff --git a/tools/patman/cros_subprocess.py b/tools/patman/cros_subprocess.py
index 0fc4a06..ebd4300 100644
--- a/tools/patman/cros_subprocess.py
+++ b/tools/patman/cros_subprocess.py
@@ -166,7 +166,7 @@ class Popen(subprocess.Popen):
         while read_set or write_set:
             try:
                 rlist, wlist, _ = select.select(read_set, write_set, [], 0.2)
-            except select.error, e:
+            except select.error as e:
                 if e.args[0] == errno.EINTR:
                     continue
                 raise
diff --git a/tools/patman/gitutil.py b/tools/patman/gitutil.py
index c0fe093..0d23079 100644
--- a/tools/patman/gitutil.py
+++ b/tools/patman/gitutil.py
@@ -139,7 +139,7 @@ def GetUpstream(git_dir, branch):
         leaf = merge.split('/')[-1]
         return '%s/%s' % (remote, leaf), None
     else:
-        raise ValueError, ("Cannot determine upstream branch for branch "
+        raise ValueError("Cannot determine upstream branch for branch "
                 "'%s' remote='%s', merge='%s'" % (branch, remote, merge))
 
 
@@ -224,7 +224,7 @@ def Checkout(commit_hash, git_dir=None, work_tree=None, force=False):
     result = command.RunPipe([pipe], capture=True, raise_on_error=False,
                              capture_stderr=True)
     if result.return_code != 0:
-        raise OSError, 'git checkout (%s): %s' % (pipe, result.stderr)
+        raise OSError('git checkout (%s): %s' % (pipe, result.stderr))
 
 def Clone(git_dir, output_dir):
     """Checkout the selected commit for this build
@@ -236,7 +236,7 @@ def Clone(git_dir, output_dir):
     result = command.RunPipe([pipe], capture=True, cwd=output_dir,
                              capture_stderr=True)
     if result.return_code != 0:
-        raise OSError, 'git clone: %s' % result.stderr
+        raise OSError('git clone: %s' % result.stderr)
 
 def Fetch(git_dir=None, work_tree=None):
     """Fetch from the origin repo
@@ -252,7 +252,7 @@ def Fetch(git_dir=None, work_tree=None):
     pipe.append('fetch')
     result = command.RunPipe([pipe], capture=True, capture_stderr=True)
     if result.return_code != 0:
-        raise OSError, 'git fetch: %s' % result.stderr
+        raise OSError('git fetch: %s' % result.stderr)
 
 def CreatePatches(start, count, series):
     """Create a series of patches from the top of the current branch.
@@ -489,7 +489,7 @@ def LookupEmail(lookup_name, alias=None, raise_on_error=True, level=0):
     if level > 10:
         msg = "Recursive email alias at '%s'" % lookup_name
         if raise_on_error:
-            raise OSError, msg
+            raise OSError(msg)
         else:
             print(col.Color(col.RED, msg))
             return out_list
@@ -498,7 +498,7 @@ def LookupEmail(lookup_name, alias=None, raise_on_error=True, level=0):
         if not lookup_name in alias:
             msg = "Alias '%s' not found" % lookup_name
             if raise_on_error:
-                raise ValueError, msg
+                raise ValueError(msg)
             else:
                 print(col.Color(col.RED, msg))
                 return out_list
-- 
2.10.0

[U-Boot] [PATCH v2 04/10] patman: Import 'configparser' lower case to be python 3.x safe

[Paul Burton]

In python 3.x module names used in import statements are case sensitive,
and the configparser module is named in all lower-case. Import it as such
in order to avoid errors when running with python 3.x.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
Acked-by: Simon Glass <sjg@chromium.org>
---

Changes in v2: None

 tools/patman/settings.py | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/tools/patman/settings.py b/tools/patman/settings.py
index 01f6c38..3caf379 100644
--- a/tools/patman/settings.py
+++ b/tools/patman/settings.py
@@ -5,7 +5,11 @@
 
 from __future__ import print_function
 
-import ConfigParser
+try:
+    import configparser as ConfigParser
+except:
+    import ConfigParser
+
 import os
 import re
 
-- 
2.10.0

[U-Boot] [PATCH v2 05/10] patman: Decode stdout/stderr as utf8, be python 3.x safe

[Paul Burton]

In python 3.x reading stdout or stdin will produce a bytestring rather
than a string. Decode it in CommunicateFilter such that the rest of the
code can continue to deal with strings. This works fine with python 2.x
too.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
Acked-by: Simon Glass <sjg@chromium.org>
---

Changes in v2: None

 tools/patman/cros_subprocess.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tools/patman/cros_subprocess.py b/tools/patman/cros_subprocess.py
index ebd4300..922e560 100644
--- a/tools/patman/cros_subprocess.py
+++ b/tools/patman/cros_subprocess.py
@@ -189,7 +189,7 @@ class Popen(subprocess.Popen):
                 data = ""
                 # We will get an error on read if the pty is closed
                 try:
-                    data = os.read(self.stdout.fileno(), 1024)
+                    data = os.read(self.stdout.fileno(), 1024).decode('utf8')
                 except OSError:
                     pass
                 if data == "":
@@ -204,7 +204,7 @@ class Popen(subprocess.Popen):
                 data = ""
                 # We will get an error on read if the pty is closed
                 try:
-                    data = os.read(self.stderr.fileno(), 1024)
+                    data = os.read(self.stderr.fileno(), 1024).decode('utf8')
                 except OSError:
                     pass
                 if data == "":
-- 
2.10.0

[U-Boot] [PATCH v2 06/10] patman: Use items() to iterate over dictionaries

[Paul Burton]

In python 3.x the iteritems() method has been removed from dictionaries,
and the items() method does effectively the same thing. On python 2.x
using items() is a little less efficient since it involves copying data,
but as speed isn't a concern in this code switch to using items() anyway
for simplicity.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>

---

Changes in v2:
- Just use items() for all python versions

 tools/patman/settings.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tools/patman/settings.py b/tools/patman/settings.py
index 3caf379..7ef0ab0 100644
--- a/tools/patman/settings.py
+++ b/tools/patman/settings.py
@@ -94,7 +94,7 @@ class _ProjectConfigParser(ConfigParser.SafeConfigParser):
         if not self.has_section(project_settings):
             self.add_section(project_settings)
         project_defaults = _default_settings.get(project_name, {})
-        for setting_name, setting_value in project_defaults.iteritems():
+        for setting_name, setting_value in project_defaults.items():
             self.set(project_settings, setting_name, setting_value)
 
     def get(self, section, option, *args, **kwargs):
-- 
2.10.0

[U-Boot] [PATCH v2 07/10] patman: Fix doctest StringIO import for python 3.x

[Paul Burton]

In python 3.x StringIO is no longer a module, and the class can instead
be found in the io module. Adjust the code in the doctest input to
account for both.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>

---

Changes in v2:
- New patch, need found by running --test.

 tools/patman/settings.py | 13 ++++++++-----
 1 file changed, 8 insertions(+), 5 deletions(-)

diff --git a/tools/patman/settings.py b/tools/patman/settings.py
index 7ef0ab0..5f207f5 100644
--- a/tools/patman/settings.py
+++ b/tools/patman/settings.py
@@ -36,7 +36,10 @@ class _ProjectConfigParser(ConfigParser.SafeConfigParser):
     - Merge general default settings/aliases with project-specific ones.
 
     # Sample config used for tests below...
-    >>> import StringIO
+    >>> try:
+    ...     from StringIO import StringIO
+    ... except ImportError:
+    ...     from io import StringIO
     >>> sample_config = '''
     ... [alias]
     ... me: Peter P. <likesspiders@example.com>
@@ -54,25 +57,25 @@ class _ProjectConfigParser(ConfigParser.SafeConfigParser):
 
     # Check to make sure that bogus project gets general alias.
     >>> config = _ProjectConfigParser("zzz")
-    >>> config.readfp(StringIO.StringIO(sample_config))
+    >>> config.readfp(StringIO(sample_config))
     >>> config.get("alias", "enemies")
     'Evil <evil@example.com>'
 
     # Check to make sure that alias gets overridden by project.
     >>> config = _ProjectConfigParser("sm")
-    >>> config.readfp(StringIO.StringIO(sample_config))
+    >>> config.readfp(StringIO(sample_config))
     >>> config.get("alias", "enemies")
     'Green G. <ugly@example.com>'
 
     # Check to make sure that settings get merged with project.
     >>> config = _ProjectConfigParser("linux")
-    >>> config.readfp(StringIO.StringIO(sample_config))
+    >>> config.readfp(StringIO(sample_config))
     >>> sorted(config.items("settings"))
     [('am_hero', 'True'), ('process_tags', 'False')]
 
     # Check to make sure that settings works with unknown project.
     >>> config = _ProjectConfigParser("unknown")
-    >>> config.readfp(StringIO.StringIO(sample_config))
+    >>> config.readfp(StringIO(sample_config))
     >>> sorted(config.items("settings"))
     [('am_hero', 'True')]
     """
-- 
2.10.0

[U-Boot] [PATCH v2 08/10] dtoc: Use items() to iterate over dictionaries in python 3.x

[Paul Burton]

In python 3.x the iteritems() method has been removed from dictionaries,
and the items() method does effectively the same thing. On python 2.x
using items() is a little less efficient since it involves copying data,
but as speed isn't a concern in the affected code switch to using
items() anyway for simplicity.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>

---

Changes in v2:
- Just use items() for all python versions

 tools/dtoc/dtoc.py         | 8 ++++----
 tools/dtoc/fdt_fallback.py | 2 +-
 2 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/tools/dtoc/dtoc.py b/tools/dtoc/dtoc.py
index 518aa51..12aa990 100755
--- a/tools/dtoc/dtoc.py
+++ b/tools/dtoc/dtoc.py
@@ -224,14 +224,14 @@ class DtbPlatdata:
             fields = {}
 
             # Get a list of all the valid properties in this node.
-            for name, prop in node.props.iteritems():
+            for name, prop in node.props.items():
                 if name not in PROP_IGNORE_LIST and name[0] != '#':
                     fields[name] = copy.deepcopy(prop)
 
             # If we've seen this node_name before, update the existing struct.
             if node_name in structs:
                 struct = structs[node_name]
-                for name, prop in fields.iteritems():
+                for name, prop in fields.items():
                     oldprop = struct.get(name)
                     if oldprop:
                         oldprop.Widen(prop)
@@ -246,7 +246,7 @@ class DtbPlatdata:
         for node in self._valid_nodes:
             node_name = self.GetCompatName(node)
             struct = structs[node_name]
-            for name, prop in node.props.iteritems():
+            for name, prop in node.props.items():
                 if name not in PROP_IGNORE_LIST and name[0] != '#':
                     prop.Widen(struct[name])
             upto += 1
@@ -298,7 +298,7 @@ class DtbPlatdata:
             var_name = Conv_name_to_c(node.name)
             self.Buf('static struct %s%s %s%s = {\n' %
                 (STRUCT_PREFIX, struct_name, VAL_PREFIX, var_name))
-            for pname, prop in node.props.iteritems():
+            for pname, prop in node.props.items():
                 if pname in PROP_IGNORE_LIST or pname[0] == '#':
                     continue
                 ptype = TYPE_NAMES[prop.type]
diff --git a/tools/dtoc/fdt_fallback.py b/tools/dtoc/fdt_fallback.py
index 0c0ebbc..8a531ff 100644
--- a/tools/dtoc/fdt_fallback.py
+++ b/tools/dtoc/fdt_fallback.py
@@ -58,7 +58,7 @@ class Node(NodeBase):
         This fills in the props and subnodes properties, recursively
         searching into subnodes so that the entire tree is built.
         """
-        for name, byte_list_str in self._fdt.GetProps(self.path).iteritems():
+        for name, byte_list_str in self._fdt.GetProps(self.path).items():
             prop = Prop(self, name, byte_list_str)
             self.props[name] = prop
 
-- 
2.10.0

[U-Boot] [PATCH v2 09/10] dtoc: Decode strings for struct.unpack on python 3.x

[Paul Burton]

On python 3.x struct.unpack will complain if we provide it with a
string since it expects to operate on a bytes object. In order to
satisfy this requirement, encode the string to a bytes object when
running on python 3.x.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
Acked-by: Simon Glass <sjg@chromium.org>
---

Changes in v2: None

 tools/dtoc/fdt_util.py | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/tools/dtoc/fdt_util.py b/tools/dtoc/fdt_util.py
index 3a10838..e6d523b 100644
--- a/tools/dtoc/fdt_util.py
+++ b/tools/dtoc/fdt_util.py
@@ -8,6 +8,7 @@
 
 import os
 import struct
+import sys
 import tempfile
 
 import command
@@ -22,6 +23,8 @@ def fdt32_to_cpu(val):
     Return:
         A native-endian integer value
     """
+    if sys.version_info > (3, 0):
+        val = val.encode('raw_unicode_escape')
     return struct.unpack('>I', val)[0]
 
 def EnsureCompiled(fname):
-- 
2.10.0

[U-Boot] [PATCH v2 10/10] dtoc: Make integer division python 3.x safe

[Paul Burton]

If we use the '/' operator then python 3.x will produce a float, and
refuse to multiply the string sequence in Conv_name_to_c by it with:

    TypeError: can't multiply sequence by non-int of type 'float'

Use the '//' operator instead to enforce that we want integer rather
than floating point division.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
Acked-by: Simon Glass <sjg@chromium.org>

---

Changes in v2: None

 tools/dtoc/dtoc.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tools/dtoc/dtoc.py b/tools/dtoc/dtoc.py
index 12aa990..11050b6 100755
--- a/tools/dtoc/dtoc.py
+++ b/tools/dtoc/dtoc.py
@@ -60,7 +60,7 @@ def Conv_name_to_c(name):
 def TabTo(num_tabs, str):
     if len(str) >= num_tabs * 8:
         return str + ' '
-    return str + '\t' * (num_tabs - len(str) / 8)
+    return str + '\t' * (num_tabs - len(str) // 8)
 
 class DtbPlatdata:
     """Provide a means to convert device tree binary data to platform data
-- 
2.10.0

[U-Boot] [PATCH v2 06/10] patman: Use items() to iterate over dictionaries

[Simon Glass]

On 27 September 2016 at 09:03, Paul Burton <paul.burton@imgtec.com> wrote:
> In python 3.x the iteritems() method has been removed from dictionaries,
> and the items() method does effectively the same thing. On python 2.x
> using items() is a little less efficient since it involves copying data,
> but as speed isn't a concern in this code switch to using items() anyway
> for simplicity.
>
> Signed-off-by: Paul Burton <paul.burton@imgtec.com>
>
> ---
>
> Changes in v2:
> - Just use items() for all python versions
>
>  tools/patman/settings.py | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)

Acked-by: Simon Glass <sjg@chromium.org>

[U-Boot] [PATCH v2 07/10] patman: Fix doctest StringIO import for python 3.x

[Simon Glass]

On 27 September 2016 at 09:03, Paul Burton <paul.burton@imgtec.com> wrote:
> In python 3.x StringIO is no longer a module, and the class can instead
> be found in the io module. Adjust the code in the doctest input to
> account for both.
>
> Signed-off-by: Paul Burton <paul.burton@imgtec.com>
>
> ---
>
> Changes in v2:
> - New patch, need found by running --test.
>
>  tools/patman/settings.py | 13 ++++++++-----
>  1 file changed, 8 insertions(+), 5 deletions(-)

Acked-by: Simon Glass <sjg@chromium.org>

[U-Boot] [PATCH v2 08/10] dtoc: Use items() to iterate over dictionaries in python 3.x

[Simon Glass]

On 27 September 2016 at 09:03, Paul Burton <paul.burton@imgtec.com> wrote:
> In python 3.x the iteritems() method has been removed from dictionaries,
> and the items() method does effectively the same thing. On python 2.x
> using items() is a little less efficient since it involves copying data,
> but as speed isn't a concern in the affected code switch to using
> items() anyway for simplicity.
>
> Signed-off-by: Paul Burton <paul.burton@imgtec.com>
>
> ---
>
> Changes in v2:
> - Just use items() for all python versions
>
>  tools/dtoc/dtoc.py         | 8 ++++----
>  tools/dtoc/fdt_fallback.py | 2 +-
>  2 files changed, 5 insertions(+), 5 deletions(-)

Acked-by: Simon Glass <sjg@chromium.org>

[U-Boot] [PATCH v2 05/10] patman: Decode stdout/stderr as utf8, be python 3.x safe

[Simon Glass]

Hi Paul,

On 27 September 2016 at 09:03, Paul Burton <paul.burton@imgtec.com> wrote:
> In python 3.x reading stdout or stdin will produce a bytestring rather
> than a string. Decode it in CommunicateFilter such that the rest of the
> code can continue to deal with strings. This works fine with python 2.x
> too.
>
> Signed-off-by: Paul Burton <paul.burton@imgtec.com>
> Acked-by: Simon Glass <sjg@chromium.org>
> ---
>
> Changes in v2: None
>
>  tools/patman/cros_subprocess.py | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/tools/patman/cros_subprocess.py b/tools/patman/cros_subprocess.py
> index ebd4300..922e560 100644
> --- a/tools/patman/cros_subprocess.py
> +++ b/tools/patman/cros_subprocess.py
> @@ -189,7 +189,7 @@ class Popen(subprocess.Popen):
>                  data = ""
>                  # We will get an error on read if the pty is closed
>                  try:
> -                    data = os.read(self.stdout.fileno(), 1024)
> +                    data = os.read(self.stdout.fileno(), 1024).decode('utf8')
>                  except OSError:
>                      pass
>                  if data == "":
> @@ -204,7 +204,7 @@ class Popen(subprocess.Popen):
>                  data = ""
>                  # We will get an error on read if the pty is closed
>                  try:
> -                    data = os.read(self.stderr.fileno(), 1024)
> +                    data = os.read(self.stderr.fileno(), 1024).decode('utf8')
>                  except OSError:
>                      pass
>                  if data == "":
> --
> 2.10.0
>

Unfortunately this causes an error:

$ buildman -b dm-push sandbox
boards.cfg is up to date. Nothing to do.
Building 16 commits for 3 boards (3 threads, 3 jobs per thread)
Traceback (most recent call last):
  File "/home/sglass/bin/buildman", line 64, in <module>
    ret_code = control.DoBuildman(options, args)
  File "/scratch/sglass/cosarm/src/third_party/u-boot/files/tools/buildman/control.py",
line 300, in DoBuildman
    options.keep_outputs, options.verbose)
  File "/scratch/sglass/cosarm/src/third_party/u-boot/files/tools/buildman/builder.py",
line 1433, in BuildBoards
    self._PrepareOutputSpace()
  File "/scratch/sglass/cosarm/src/third_party/u-boot/files/tools/buildman/builder.py",
line 1399, in _PrepareOutputSpace
    dir_list.append(self._GetOutputDir(commit_upto))
  File "/scratch/sglass/cosarm/src/third_party/u-boot/files/tools/buildman/builder.py",
line 449, in _GetOutputDir
    subject = commit.subject.translate(trans_valid_chars)
TypeError: character mapping must return integer, None or unicode

Regards,
Simon

[U-Boot] [PATCH v2 01/10] patman: Replace tabs with spaces

[Simon Glass]

On 27 September 2016 at 09:03, Paul Burton <paul.burton@imgtec.com> wrote:
> In preparation for running on python 3.x, which will refuse to run
> scripts which mix tabs & spaces for indentation, replace 2 tab
> characters present in series.py with spaces.
>
> Signed-off-by: Paul Burton <paul.burton@imgtec.com>
> Acked-by: Simon Glass <sjg@chromium.org>
> ---
>
> Changes in v2: None
>
>  tools/patman/series.py | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)

Applied to u-boot-dm, thanks!

[U-Boot] [PATCH v2 02/10] patman: Make print statements python 3.x safe

[Simon Glass]

On 27 September 2016 at 09:03, Paul Burton <paul.burton@imgtec.com> wrote:
> In python 3.x, print must be used as a function call. Convert all print
> statements to the function call style, importing from __future__ where
> we print with no trailing newline or print to a file object.
>
> Signed-off-by: Paul Burton <paul.burton@imgtec.com>
> Acked-by: Simon Glass <sjg@chromium.org>
> ---
>
> Changes in v2: None
>
>  tools/patman/checkpatch.py     | 16 ++++++++--------
>  tools/patman/get_maintainer.py |  2 +-
>  tools/patman/gitutil.py        |  6 +++---
>  tools/patman/patchstream.py    |  6 +++---
>  tools/patman/patman.py         | 12 ++++++------
>  tools/patman/series.py         | 38 ++++++++++++++++++++------------------
>  tools/patman/settings.py       | 16 +++++++++-------
>  tools/patman/terminal.py       | 12 +++++++-----
>  tools/patman/test.py           |  2 +-
>  9 files changed, 58 insertions(+), 52 deletions(-)

Applied to u-boot-dm, thanks!

[U-Boot] [PATCH v2 03/10] patman: Make exception handling python 3.x safe

[Simon Glass]

On 27 September 2016 at 09:03, Paul Burton <paul.burton@imgtec.com> wrote:
> Syntax for exception handling is a little more strict in python 3.x.
> Convert all uses to a form accepted by both python 2.x & python 3.x.
>
> Signed-off-by: Paul Burton <paul.burton@imgtec.com>
> Acked-by: Simon Glass <sjg@chromium.org>
> ---
>
> Changes in v2: None
>
>  tools/patman/command.py         |  2 +-
>  tools/patman/cros_subprocess.py |  2 +-
>  tools/patman/gitutil.py         | 12 ++++++------
>  3 files changed, 8 insertions(+), 8 deletions(-)

Applied to u-boot-dm, thanks!

[U-Boot] [PATCH v2 04/10] patman: Import 'configparser' lower case to be python 3.x safe

[Simon Glass]

On 27 September 2016 at 09:03, Paul Burton <paul.burton@imgtec.com> wrote:
> In python 3.x module names used in import statements are case sensitive,
> and the configparser module is named in all lower-case. Import it as such
> in order to avoid errors when running with python 3.x.
>
> Signed-off-by: Paul Burton <paul.burton@imgtec.com>
> Acked-by: Simon Glass <sjg@chromium.org>
> ---
>
> Changes in v2: None
>
>  tools/patman/settings.py | 6 +++++-
>  1 file changed, 5 insertions(+), 1 deletion(-)

Applied to u-boot-dm, thanks!

[U-Boot] [PATCH v2 06/10] patman: Use items() to iterate over dictionaries

[Simon Glass]

On 27 September 2016 at 11:55, Simon Glass <sjg@chromium.org> wrote:
> On 27 September 2016 at 09:03, Paul Burton <paul.burton@imgtec.com> wrote:
>> In python 3.x the iteritems() method has been removed from dictionaries,
>> and the items() method does effectively the same thing. On python 2.x
>> using items() is a little less efficient since it involves copying data,
>> but as speed isn't a concern in this code switch to using items() anyway
>> for simplicity.
>>
>> Signed-off-by: Paul Burton <paul.burton@imgtec.com>
>>
>> ---
>>
>> Changes in v2:
>> - Just use items() for all python versions
>>
>>  tools/patman/settings.py | 2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> Acked-by: Simon Glass <sjg@chromium.org>

Applied to u-boot-dm, thanks!

[U-Boot] [PATCH v2 07/10] patman: Fix doctest StringIO import for python 3.x

[Simon Glass]

On 27 September 2016 at 11:55, Simon Glass <sjg@chromium.org> wrote:
> On 27 September 2016 at 09:03, Paul Burton <paul.burton@imgtec.com> wrote:
>> In python 3.x StringIO is no longer a module, and the class can instead
>> be found in the io module. Adjust the code in the doctest input to
>> account for both.
>>
>> Signed-off-by: Paul Burton <paul.burton@imgtec.com>
>>
>> ---
>>
>> Changes in v2:
>> - New patch, need found by running --test.
>>
>>  tools/patman/settings.py | 13 ++++++++-----
>>  1 file changed, 8 insertions(+), 5 deletions(-)
>
> Acked-by: Simon Glass <sjg@chromium.org>

Applied to u-boot-dm, thanks!

[U-Boot] [PATCH v2 08/10] dtoc: Use items() to iterate over dictionaries in python 3.x

[Simon Glass]

On 27 September 2016 at 11:55, Simon Glass <sjg@chromium.org> wrote:
> On 27 September 2016 at 09:03, Paul Burton <paul.burton@imgtec.com> wrote:
>> In python 3.x the iteritems() method has been removed from dictionaries,
>> and the items() method does effectively the same thing. On python 2.x
>> using items() is a little less efficient since it involves copying data,
>> but as speed isn't a concern in the affected code switch to using
>> items() anyway for simplicity.
>>
>> Signed-off-by: Paul Burton <paul.burton@imgtec.com>
>>
>> ---
>>
>> Changes in v2:
>> - Just use items() for all python versions
>>
>>  tools/dtoc/dtoc.py         | 8 ++++----
>>  tools/dtoc/fdt_fallback.py | 2 +-
>>  2 files changed, 5 insertions(+), 5 deletions(-)
>
> Acked-by: Simon Glass <sjg@chromium.org>

Applied to u-boot-dm, thanks!

[U-Boot] [PATCH v2 09/10] dtoc: Decode strings for struct.unpack on python 3.x

[Simon Glass]

On 27 September 2016 at 09:03, Paul Burton <paul.burton@imgtec.com> wrote:
> On python 3.x struct.unpack will complain if we provide it with a
> string since it expects to operate on a bytes object. In order to
> satisfy this requirement, encode the string to a bytes object when
> running on python 3.x.
>
> Signed-off-by: Paul Burton <paul.burton@imgtec.com>
> Acked-by: Simon Glass <sjg@chromium.org>
> ---
>
> Changes in v2: None
>
>  tools/dtoc/fdt_util.py | 3 +++
>  1 file changed, 3 insertions(+)

Applied to u-boot-dm, thanks!

[U-Boot] [PATCH v2 10/10] dtoc: Make integer division python 3.x safe

[Simon Glass]

On 27 September 2016 at 09:03, Paul Burton <paul.burton@imgtec.com> wrote:
> If we use the '/' operator then python 3.x will produce a float, and
> refuse to multiply the string sequence in Conv_name_to_c by it with:
>
>     TypeError: can't multiply sequence by non-int of type 'float'
>
> Use the '//' operator instead to enforce that we want integer rather
> than floating point division.
>
> Signed-off-by: Paul Burton <paul.burton@imgtec.com>
> Acked-by: Simon Glass <sjg@chromium.org>
>
> ---
>
> Changes in v2: None
>
>  tools/dtoc/dtoc.py | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)

Applied to u-boot-dm, thanks!