All of lore.kernel.org
 help / color / mirror / Atom feed
* [U-Boot] [PATCH v2 00/10] patman: Fix unicode operation and add a functional test
@ 2017-05-29 21:31 Simon Glass
  2017-05-29 21:31 ` [U-Boot] [PATCH v2 01/10] patman: encode CC list to UTF-8 Simon Glass
                   ` (10 more replies)
  0 siblings, 11 replies; 24+ messages in thread
From: Simon Glass @ 2017-05-29 21:31 UTC (permalink / raw)
  To: u-boot

With more recent versions of Python 2.7 (and perhaps patman itself)
something has changed such that patman sometimes reports unicode decode
errors, particularly when getting a list of maintainers for code covered
by a patch.

This series attempts to correct this. It also adds a functional test so
that operation can quickly be verified. This new test tests much more of
the code than the existing unit tests.

Changes in v2:
- Expand the patch to cover all cases
- Drop RFC tag
- Rewrite commit message
- Add new clean-up patches and a functional test

Philipp Tomsich (1):
  patman: encode CC list to UTF-8

Simon Glass (9):
  patman: Adjust handling of unicode email address
  patman: Don't convert input data to unicode
  patman: Rename 'str' variable in EmailPatches()
  patman: Don't report unicode character
  patman: Don't return the series in FixPatches()
  patman: Add unicode to test patches
  patman: Add a maintainer test feature to MakeCcFile()
  patman: Rename 'list' variable in MakeCcFile()
  patman: Add a functional test

 tools/patman/cros_subprocess.py                    |   4 -
 tools/patman/func_test.py                          | 242 +++++++++++++++++++++
 tools/patman/gitutil.py                            |   8 +-
 tools/patman/patchstream.py                        |  23 +-
 tools/patman/patman.py                             |  10 +-
 tools/patman/series.py                             |  28 ++-
 tools/patman/test.py                               |   9 +
 tools/patman/test/0000-cover-letter.patch          |  23 ++
 .../test/0001-pci-Correct-cast-for-sandbox.patch   |  48 ++++
 ...-cast-for-sandbox-in-fdtdec_setup_memory_.patch |  73 +++++++
 tools/patman/test/test01.txt                       |  56 +++++
 11 files changed, 493 insertions(+), 31 deletions(-)
 create mode 100644 tools/patman/func_test.py
 create mode 100644 tools/patman/test/0000-cover-letter.patch
 create mode 100644 tools/patman/test/0001-pci-Correct-cast-for-sandbox.patch
 create mode 100644 tools/patman/test/0002-fdt-Correct-cast-for-sandbox-in-fdtdec_setup_memory_.patch
 create mode 100644 tools/patman/test/test01.txt

-- 
2.13.0.219.gdb65acc882-goog

^ permalink raw reply	[flat|nested] 24+ messages in thread

* [U-Boot] [PATCH v2 01/10] patman: encode CC list to UTF-8
  2017-05-29 21:31 [U-Boot] [PATCH v2 00/10] patman: Fix unicode operation and add a functional test Simon Glass
@ 2017-05-29 21:31 ` Simon Glass
  2017-06-07  1:47   ` sjg at google.com
  2017-05-29 21:31 ` [U-Boot] [PATCH v2 02/10] patman: Adjust handling of unicode email address Simon Glass
                   ` (9 subsequent siblings)
  10 siblings, 1 reply; 24+ messages in thread
From: Simon Glass @ 2017-05-29 21:31 UTC (permalink / raw)
  To: u-boot

From: Philipp Tomsich <philipp.tomsich@theobroma-systems.com>

This change encodes the CC list to UTF-8 to avoid failures on
maintainer-addresses that include non-ASCII characters (observed on
Debian 7.11 with Python 2.7.3).

Without this, I get the following failure:
  Traceback (most recent call last):
    File "tools/patman/patman", line 159, in <module>
      options.add_maintainers)
    File "[snip]/u-boot/tools/patman/series.py", line 234, in MakeCcFile
      print(commit.patch, ', '.join(set(list)), file=fd)
  UnicodeEncodeError: 'ascii' codec can't encode character u'\xfc' in position 81: ordinal not in range(128)
from Heiko's email address:
  [..., u'"Heiko St\xfcbner" <heiko@sntech.de>', ...]

While with this change added this encodes to:
  "=?UTF-8?q?Heiko=20St=C3=BCbner?= <heiko@sntech.de>"

Signed-off-by: Philipp Tomsich <philipp.tomsich@theobroma-systems.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
Signed-off-by: Simon Glass <sjg@chromium.org>
---

Changes in v2: None

 tools/patman/series.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tools/patman/series.py b/tools/patman/series.py
index c1b86521aa..134a381ae3 100644
--- a/tools/patman/series.py
+++ b/tools/patman/series.py
@@ -119,7 +119,7 @@ class Series(dict):
                     email = col.Color(col.YELLOW, "<alias '%s' not found>"
                             % tag)
                 if email:
-                    print('      Cc: ', email)
+                    print('      Cc: ', email.encode('utf-8'))
         print
         for item in to_set:
             print('To:\t ', item)
@@ -230,7 +230,7 @@ class Series(dict):
             if add_maintainers:
                 list += get_maintainer.GetMaintainer(commit.patch)
             all_ccs += list
-            print(commit.patch, ', '.join(set(list)), file=fd)
+            print(commit.patch, ', '.join(set(list)).encode('utf-8'), file=fd)
             self._generated_cc[commit.patch] = list
 
         if cover_fname:
-- 
2.13.0.219.gdb65acc882-goog

^ permalink raw reply related	[flat|nested] 24+ messages in thread

* [U-Boot] [PATCH v2 02/10] patman: Adjust handling of unicode email address
  2017-05-29 21:31 [U-Boot] [PATCH v2 00/10] patman: Fix unicode operation and add a functional test Simon Glass
  2017-05-29 21:31 ` [U-Boot] [PATCH v2 01/10] patman: encode CC list to UTF-8 Simon Glass
@ 2017-05-29 21:31 ` Simon Glass
  2017-06-07  1:47   ` sjg at google.com
  2017-05-29 21:31 ` [U-Boot] [PATCH v2 03/10] patman: Don't convert input data to unicode Simon Glass
                   ` (8 subsequent siblings)
  10 siblings, 1 reply; 24+ messages in thread
From: Simon Glass @ 2017-05-29 21:31 UTC (permalink / raw)
  To: u-boot

Don't mess with the email address when outputting them. Just make sure
they are encoded with utf-8.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

Changes in v2:
- Expand the patch to cover all cases
- Drop RFC tag
- Rewrite commit message

 tools/patman/gitutil.py |  2 ++
 tools/patman/series.py  | 10 +++++++---
 2 files changed, 9 insertions(+), 3 deletions(-)

diff --git a/tools/patman/gitutil.py b/tools/patman/gitutil.py
index 0d23079a3a..167bda38fc 100644
--- a/tools/patman/gitutil.py
+++ b/tools/patman/gitutil.py
@@ -407,6 +407,8 @@ def EmailPatches(series, cover_fname, args, dry_run, raise_on_error, cc_fname,
         cc = []
     cmd = ['git', 'send-email', '--annotate']
     if in_reply_to:
+        if type(in_reply_to) != str:
+            in_reply_to = in_reply_to.encode('utf-8')
         cmd.append('--in-reply-to="%s"' % in_reply_to)
     if thread:
         cmd.append('--thread')
diff --git a/tools/patman/series.py b/tools/patman/series.py
index 134a381ae3..7a3534fee9 100644
--- a/tools/patman/series.py
+++ b/tools/patman/series.py
@@ -119,7 +119,7 @@ class Series(dict):
                     email = col.Color(col.YELLOW, "<alias '%s' not found>"
                             % tag)
                 if email:
-                    print('      Cc: ', email.encode('utf-8'))
+                    print('      Cc: ', email)
         print
         for item in to_set:
             print('To:\t ', item)
@@ -229,13 +229,17 @@ class Series(dict):
                                            raise_on_error=raise_on_error)
             if add_maintainers:
                 list += get_maintainer.GetMaintainer(commit.patch)
+            list = [m.encode('utf-8') if type(m) != str else m for m in list]
             all_ccs += list
-            print(commit.patch, ', '.join(set(list)).encode('utf-8'), file=fd)
+            print(commit.patch, ', '.join(set(list)), file=fd)
             self._generated_cc[commit.patch] = list
 
         if cover_fname:
             cover_cc = gitutil.BuildEmailList(self.get('cover_cc', ''))
-            cc_list = ', '.join([x.decode('utf-8') for x in set(cover_cc + all_ccs)])
+            cover_cc = [m.encode('utf-8') if type(m) != str else m
+                        for m in cover_cc]
+            cc_list = ', '.join([x.decode('utf-8')
+                                 for x in set(cover_cc + all_ccs)])
             print(cover_fname, cc_list.encode('utf-8'), file=fd)
 
         fd.close()
-- 
2.13.0.219.gdb65acc882-goog

^ permalink raw reply related	[flat|nested] 24+ messages in thread

* [U-Boot] [PATCH v2 03/10] patman: Don't convert input data to unicode
  2017-05-29 21:31 [U-Boot] [PATCH v2 00/10] patman: Fix unicode operation and add a functional test Simon Glass
  2017-05-29 21:31 ` [U-Boot] [PATCH v2 01/10] patman: encode CC list to UTF-8 Simon Glass
  2017-05-29 21:31 ` [U-Boot] [PATCH v2 02/10] patman: Adjust handling of unicode email address Simon Glass
@ 2017-05-29 21:31 ` Simon Glass
  2017-06-07  1:47   ` sjg at google.com
  2017-05-29 21:31 ` [U-Boot] [PATCH v2 04/10] patman: Rename 'str' variable in EmailPatches() Simon Glass
                   ` (7 subsequent siblings)
  10 siblings, 1 reply; 24+ messages in thread
From: Simon Glass @ 2017-05-29 21:31 UTC (permalink / raw)
  To: u-boot

The communication filter reads data in blocks and converts each block to
unicode (if necessary) one at a time. In the unlikely event that a unicode
character in the input spans a block this will not work. We get an error
like:

UnicodeDecodeError: 'utf8' codec can't decode bytes in position 1022-1023:
   unexpected end of data

There is no need to change the input to unicode, so the easiest fix is to
drop this feature.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

Changes in v2: None

 tools/patman/cros_subprocess.py | 4 ----
 1 file changed, 4 deletions(-)

diff --git a/tools/patman/cros_subprocess.py b/tools/patman/cros_subprocess.py
index 7c76014340..ebd4300dfd 100644
--- a/tools/patman/cros_subprocess.py
+++ b/tools/patman/cros_subprocess.py
@@ -190,8 +190,6 @@ class Popen(subprocess.Popen):
                 # We will get an error on read if the pty is closed
                 try:
                     data = os.read(self.stdout.fileno(), 1024)
-                    if isinstance(data, bytes):
-                        data = data.decode('utf-8')
                 except OSError:
                     pass
                 if data == "":
@@ -207,8 +205,6 @@ class Popen(subprocess.Popen):
                 # We will get an error on read if the pty is closed
                 try:
                     data = os.read(self.stderr.fileno(), 1024)
-                    if isinstance(data, bytes):
-                        data = data.decode('utf-8')
                 except OSError:
                     pass
                 if data == "":
-- 
2.13.0.219.gdb65acc882-goog

^ permalink raw reply related	[flat|nested] 24+ messages in thread

* [U-Boot] [PATCH v2 04/10] patman: Rename 'str' variable in EmailPatches()
  2017-05-29 21:31 [U-Boot] [PATCH v2 00/10] patman: Fix unicode operation and add a functional test Simon Glass
                   ` (2 preceding siblings ...)
  2017-05-29 21:31 ` [U-Boot] [PATCH v2 03/10] patman: Don't convert input data to unicode Simon Glass
@ 2017-05-29 21:31 ` Simon Glass
  2017-06-07  1:47   ` sjg at google.com
  2017-05-29 21:31 ` [U-Boot] [PATCH v2 05/10] patman: Don't report unicode character Simon Glass
                   ` (6 subsequent siblings)
  10 siblings, 1 reply; 24+ messages in thread
From: Simon Glass @ 2017-05-29 21:31 UTC (permalink / raw)
  To: u-boot

This is not a good variable name in Python because 'str' is a type. It
shows up highlighted in some editors. Rename it.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

Changes in v2: None

 tools/patman/gitutil.py | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/tools/patman/gitutil.py b/tools/patman/gitutil.py
index 167bda38fc..08be9377ce 100644
--- a/tools/patman/gitutil.py
+++ b/tools/patman/gitutil.py
@@ -419,10 +419,10 @@ def EmailPatches(series, cover_fname, args, dry_run, raise_on_error, cc_fname,
     if cover_fname:
         cmd.append(cover_fname)
     cmd += args
-    str = ' '.join(cmd)
+    cmdstr = ' '.join(cmd)
     if not dry_run:
-        os.system(str)
-    return str
+        os.system(cmdstr)
+    return cmdstr
 
 
 def LookupEmail(lookup_name, alias=None, raise_on_error=True, level=0):
-- 
2.13.0.219.gdb65acc882-goog

^ permalink raw reply related	[flat|nested] 24+ messages in thread

* [U-Boot] [PATCH v2 05/10] patman: Don't report unicode character
  2017-05-29 21:31 [U-Boot] [PATCH v2 00/10] patman: Fix unicode operation and add a functional test Simon Glass
                   ` (3 preceding siblings ...)
  2017-05-29 21:31 ` [U-Boot] [PATCH v2 04/10] patman: Rename 'str' variable in EmailPatches() Simon Glass
@ 2017-05-29 21:31 ` Simon Glass
  2017-06-07  1:47   ` sjg at google.com
  2017-05-29 21:31 ` [U-Boot] [PATCH v2 06/10] patman: Don't return the series in FixPatches() Simon Glass
                   ` (5 subsequent siblings)
  10 siblings, 1 reply; 24+ messages in thread
From: Simon Glass @ 2017-05-29 21:31 UTC (permalink / raw)
  To: u-boot

Unicode characters may appear in input patches so we should not warn about
them. Drop this warning.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

Changes in v2: None

 tools/patman/patchstream.py | 9 ---------
 1 file changed, 9 deletions(-)

diff --git a/tools/patman/patchstream.py b/tools/patman/patchstream.py
index cd4667f61c..324c65442f 100644
--- a/tools/patman/patchstream.py
+++ b/tools/patman/patchstream.py
@@ -308,15 +308,6 @@ class PatchStream:
 
         # Well that means this is an ordinary line
         else:
-            pos = 1
-            # Look for ugly ASCII characters
-            for ch in line:
-                # TODO: Would be nicer to report source filename and line
-                if ord(ch) > 0x80:
-                    self.warn.append("Line %d/%d ('%s') has funny ascii char" %
-                        (self.linenum, pos, line))
-                pos += 1
-
             # Look for space before tab
             m = re_space_before_tab.match(line)
             if m:
-- 
2.13.0.219.gdb65acc882-goog

^ permalink raw reply related	[flat|nested] 24+ messages in thread

* [U-Boot] [PATCH v2 06/10] patman: Don't return the series in FixPatches()
  2017-05-29 21:31 [U-Boot] [PATCH v2 00/10] patman: Fix unicode operation and add a functional test Simon Glass
                   ` (4 preceding siblings ...)
  2017-05-29 21:31 ` [U-Boot] [PATCH v2 05/10] patman: Don't report unicode character Simon Glass
@ 2017-05-29 21:31 ` Simon Glass
  2017-06-07  1:47   ` sjg at google.com
  2017-05-29 21:31 ` [U-Boot] [PATCH v2 07/10] patman: Add unicode to test patches Simon Glass
                   ` (4 subsequent siblings)
  10 siblings, 1 reply; 24+ messages in thread
From: Simon Glass @ 2017-05-29 21:31 UTC (permalink / raw)
  To: u-boot

There is no need for this function to return the same object that was
passed in. Drop the return value.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

Changes in v2: None

 tools/patman/patchstream.py | 1 -
 tools/patman/patman.py      | 4 ++--
 2 files changed, 2 insertions(+), 3 deletions(-)

diff --git a/tools/patman/patchstream.py b/tools/patman/patchstream.py
index 324c65442f..6098728aa1 100644
--- a/tools/patman/patchstream.py
+++ b/tools/patman/patchstream.py
@@ -477,7 +477,6 @@ def FixPatches(series, fnames):
             print
         count += 1
     print('Cleaned %d patches' % count)
-    return series
 
 def InsertCoverLetter(fname, series, count):
     """Inserts a cover letter with the required info into patch 0
diff --git a/tools/patman/patman.py b/tools/patman/patman.py
index fdbee67f55..63da3f3200 100755
--- a/tools/patman/patman.py
+++ b/tools/patman/patman.py
@@ -141,8 +141,8 @@ else:
                 series)
 
     # Fix up the patch files to our liking, and insert the cover letter
-    series = patchstream.FixPatches(series, args)
-    if series and cover_fname and series.get('cover'):
+    patchstream.FixPatches(series, args)
+    if cover_fname and series.get('cover'):
         patchstream.InsertCoverLetter(cover_fname, series, options.count)
 
     # Do a few checks on the series
-- 
2.13.0.219.gdb65acc882-goog

^ permalink raw reply related	[flat|nested] 24+ messages in thread

* [U-Boot] [PATCH v2 07/10] patman: Add unicode to test patches
  2017-05-29 21:31 [U-Boot] [PATCH v2 00/10] patman: Fix unicode operation and add a functional test Simon Glass
                   ` (5 preceding siblings ...)
  2017-05-29 21:31 ` [U-Boot] [PATCH v2 06/10] patman: Don't return the series in FixPatches() Simon Glass
@ 2017-05-29 21:31 ` Simon Glass
  2017-06-07  1:47   ` sjg at google.com
  2017-05-29 21:31 ` [U-Boot] [PATCH v2 08/10] patman: Add a maintainer test feature to MakeCcFile() Simon Glass
                   ` (3 subsequent siblings)
  10 siblings, 1 reply; 24+ messages in thread
From: Simon Glass @ 2017-05-29 21:31 UTC (permalink / raw)
  To: u-boot

Add some unicode to the test patches to make sure that patman does the
right thing.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

Changes in v2: None

 tools/patman/test.py | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/tools/patman/test.py b/tools/patman/test.py
index 8c39f66e73..20dc9c1e0d 100644
--- a/tools/patman/test.py
+++ b/tools/patman/test.py
@@ -1,3 +1,4 @@
+# -*- coding: utf-8 -*-
 #
 # Copyright (c) 2011 The Chromium OS Authors.
 #
@@ -31,6 +32,10 @@ Subject: [PATCH (resend) 3/7] Tegra2: Add more clock support
 
 This adds functions to enable/disable clocks and reset to on-chip peripherals.
 
+cmd/pci.c:152:11: warning: format ‘%llx’ expects argument of type
+   ‘long long unsigned int’, but argument 3 has type
+   ‘u64 {aka long unsigned int}’ [-Wformat=]
+
 BUG=chromium-os:13875
 TEST=build U-Boot for Seaboard, boot
 
@@ -53,6 +58,10 @@ Subject: [PATCH (resend) 3/7] Tegra2: Add more clock support
 
 This adds functions to enable/disable clocks and reset to on-chip peripherals.
 
+cmd/pci.c:152:11: warning: format ‘%llx’ expects argument of type
+   ‘long long unsigned int’, but argument 3 has type
+   ‘u64 {aka long unsigned int}’ [-Wformat=]
+
 Signed-off-by: Simon Glass <sjg@chromium.org>
 ---
 
-- 
2.13.0.219.gdb65acc882-goog

^ permalink raw reply related	[flat|nested] 24+ messages in thread

* [U-Boot] [PATCH v2 08/10] patman: Add a maintainer test feature to MakeCcFile()
  2017-05-29 21:31 [U-Boot] [PATCH v2 00/10] patman: Fix unicode operation and add a functional test Simon Glass
                   ` (6 preceding siblings ...)
  2017-05-29 21:31 ` [U-Boot] [PATCH v2 07/10] patman: Add unicode to test patches Simon Glass
@ 2017-05-29 21:31 ` Simon Glass
  2017-06-07  1:47   ` sjg at google.com
  2017-05-29 21:31 ` [U-Boot] [PATCH v2 09/10] patman: Rename 'list' variable in MakeCcFile() Simon Glass
                   ` (2 subsequent siblings)
  10 siblings, 1 reply; 24+ messages in thread
From: Simon Glass @ 2017-05-29 21:31 UTC (permalink / raw)
  To: u-boot

Allow the add_maintainers parameter to be a list of maintainers, thus
allowing us to simulate calling the script in tests without actually
needing it to work.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

Changes in v2: None

 tools/patman/series.py | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/tools/patman/series.py b/tools/patman/series.py
index 7a3534fee9..395b9ea99b 100644
--- a/tools/patman/series.py
+++ b/tools/patman/series.py
@@ -212,7 +212,9 @@ class Series(dict):
             cover_fname: If non-None the name of the cover letter.
             raise_on_error: True to raise an error when an alias fails to match,
                 False to just print a message.
-            add_maintainers: Call the get_maintainers to CC maintainers
+            add_maintainers: Either:
+                True/False to call the get_maintainers to CC maintainers
+                List of maintainers to include (for testing)
         Return:
             Filename of temp file created
         """
@@ -227,7 +229,9 @@ class Series(dict):
                                                raise_on_error=raise_on_error)
             list += gitutil.BuildEmailList(commit.cc_list,
                                            raise_on_error=raise_on_error)
-            if add_maintainers:
+            if type(add_maintainers) == type(list):
+                list += add_maintainers
+            elif add_maintainers:
                 list += get_maintainer.GetMaintainer(commit.patch)
             list = [m.encode('utf-8') if type(m) != str else m for m in list]
             all_ccs += list
-- 
2.13.0.219.gdb65acc882-goog

^ permalink raw reply related	[flat|nested] 24+ messages in thread

* [U-Boot] [PATCH v2 09/10] patman: Rename 'list' variable in MakeCcFile()
  2017-05-29 21:31 [U-Boot] [PATCH v2 00/10] patman: Fix unicode operation and add a functional test Simon Glass
                   ` (7 preceding siblings ...)
  2017-05-29 21:31 ` [U-Boot] [PATCH v2 08/10] patman: Add a maintainer test feature to MakeCcFile() Simon Glass
@ 2017-05-29 21:31 ` Simon Glass
  2017-06-07  1:47   ` sjg at google.com
  2017-05-29 21:31 ` [U-Boot] [PATCH v2 10/10] patman: Add a functional test Simon Glass
  2017-06-05 21:41 ` [U-Boot] [PATCH v2 00/10] patman: Fix unicode operation and add " Simon Glass
  10 siblings, 1 reply; 24+ messages in thread
From: Simon Glass @ 2017-05-29 21:31 UTC (permalink / raw)
  To: u-boot

This is not a good variable name in Python because 'list' is a type. It
shows up highlighted in some editors. Rename it.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

Changes in v2: None

 tools/patman/series.py | 20 ++++++++++----------
 1 file changed, 10 insertions(+), 10 deletions(-)

diff --git a/tools/patman/series.py b/tools/patman/series.py
index 395b9ea99b..d3947a7c2a 100644
--- a/tools/patman/series.py
+++ b/tools/patman/series.py
@@ -223,20 +223,20 @@ class Series(dict):
         fd = open(fname, 'w')
         all_ccs = []
         for commit in self.commits:
-            list = []
+            cc = []
             if process_tags:
-                list += gitutil.BuildEmailList(commit.tags,
+                cc += gitutil.BuildEmailList(commit.tags,
                                                raise_on_error=raise_on_error)
-            list += gitutil.BuildEmailList(commit.cc_list,
+            cc += gitutil.BuildEmailList(commit.cc_list,
                                            raise_on_error=raise_on_error)
-            if type(add_maintainers) == type(list):
-                list += add_maintainers
+            if type(add_maintainers) == type(cc):
+                cc += add_maintainers
             elif add_maintainers:
-                list += get_maintainer.GetMaintainer(commit.patch)
-            list = [m.encode('utf-8') if type(m) != str else m for m in list]
-            all_ccs += list
-            print(commit.patch, ', '.join(set(list)), file=fd)
-            self._generated_cc[commit.patch] = list
+                cc += get_maintainer.GetMaintainer(commit.patch)
+            cc = [m.encode('utf-8') if type(m) != str else m for m in cc]
+            all_ccs += cc
+            print(commit.patch, ', '.join(set(cc)), file=fd)
+            self._generated_cc[commit.patch] = cc
 
         if cover_fname:
             cover_cc = gitutil.BuildEmailList(self.get('cover_cc', ''))
-- 
2.13.0.219.gdb65acc882-goog

^ permalink raw reply related	[flat|nested] 24+ messages in thread

* [U-Boot] [PATCH v2 10/10] patman: Add a functional test
  2017-05-29 21:31 [U-Boot] [PATCH v2 00/10] patman: Fix unicode operation and add a functional test Simon Glass
                   ` (8 preceding siblings ...)
  2017-05-29 21:31 ` [U-Boot] [PATCH v2 09/10] patman: Rename 'list' variable in MakeCcFile() Simon Glass
@ 2017-05-29 21:31 ` Simon Glass
  2017-06-07  1:47   ` sjg at google.com
  2017-06-05 21:41 ` [U-Boot] [PATCH v2 00/10] patman: Fix unicode operation and add " Simon Glass
  10 siblings, 1 reply; 24+ messages in thread
From: Simon Glass @ 2017-05-29 21:31 UTC (permalink / raw)
  To: u-boot

The existing test (patman --test) only covers basic checkpatch output.
We have had some problems with unicode processing and could use test
coverage for the various tags patman supports.

Add a new functional test which runs most of the patman flow on a few
test commits and checks that the results are correct.

See the documentation in the test for a description of what it does.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

Changes in v2:
- Add new clean-up patches and a functional test

 tools/patman/func_test.py                          | 242 +++++++++++++++++++++
 tools/patman/patchstream.py                        |  13 ++
 tools/patman/patman.py                             |   6 +-
 tools/patman/test/0000-cover-letter.patch          |  23 ++
 .../test/0001-pci-Correct-cast-for-sandbox.patch   |  48 ++++
 ...-cast-for-sandbox-in-fdtdec_setup_memory_.patch |  73 +++++++
 tools/patman/test/test01.txt                       |  56 +++++
 7 files changed, 459 insertions(+), 2 deletions(-)
 create mode 100644 tools/patman/func_test.py
 create mode 100644 tools/patman/test/0000-cover-letter.patch
 create mode 100644 tools/patman/test/0001-pci-Correct-cast-for-sandbox.patch
 create mode 100644 tools/patman/test/0002-fdt-Correct-cast-for-sandbox-in-fdtdec_setup_memory_.patch
 create mode 100644 tools/patman/test/test01.txt

diff --git a/tools/patman/func_test.py b/tools/patman/func_test.py
new file mode 100644
index 0000000000..2c0da84b30
--- /dev/null
+++ b/tools/patman/func_test.py
@@ -0,0 +1,242 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2017 Google, Inc
+#
+# SPDX-License-Identifier:	GPL-2.0+
+#
+
+import contextlib
+import os
+import re
+import shutil
+import sys
+import tempfile
+import unittest
+
+import gitutil
+import patchstream
+import settings
+
+
+ at contextlib.contextmanager
+def capture():
+    import sys
+    from cStringIO import StringIO
+    oldout,olderr = sys.stdout, sys.stderr
+    try:
+        out=[StringIO(), StringIO()]
+        sys.stdout,sys.stderr = out
+        yield out
+    finally:
+        sys.stdout,sys.stderr = oldout, olderr
+        out[0] = out[0].getvalue()
+        out[1] = out[1].getvalue()
+
+
+class TestFunctional(unittest.TestCase):
+    def setUp(self):
+        self.tmpdir = tempfile.mkdtemp(prefix='patman.')
+
+    def tearDown(self):
+        shutil.rmtree(self.tmpdir)
+
+    @staticmethod
+    def GetPath(fname):
+        return os.path.join(os.path.dirname(os.path.realpath(sys.argv[0])),
+                            'test', fname)
+
+    @classmethod
+    def GetText(self, fname):
+        return open(self.GetPath(fname)).read()
+
+    @classmethod
+    def GetPatchName(self, subject):
+        fname = re.sub('[ :]', '-', subject)
+        return fname.replace('--', '-')
+
+    def CreatePatchesForTest(self, series):
+        cover_fname = None
+        fname_list = []
+        for i, commit in enumerate(series.commits):
+            clean_subject = self.GetPatchName(commit.subject)
+            src_fname = '%04d-%s.patch' % (i + 1, clean_subject[:52])
+            fname = os.path.join(self.tmpdir, src_fname)
+            shutil.copy(self.GetPath(src_fname), fname)
+            fname_list.append(fname)
+        if series.get('cover'):
+            src_fname = '0000-cover-letter.patch'
+            cover_fname = os.path.join(self.tmpdir, src_fname)
+            fname = os.path.join(self.tmpdir, src_fname)
+            shutil.copy(self.GetPath(src_fname), fname)
+
+        return cover_fname, fname_list
+
+    def testBasic(self):
+        """Tests the basic flow of patman
+
+        This creates a series from some hard-coded patches build from a simple
+        tree with the following metadata in the top commit:
+
+            Series-to: u-boot
+            Series-prefix: RFC
+            Series-cc: Stefan Brüns <stefan.bruens@rwth-aachen.de>
+            Cover-letter-cc: Lord Mëlchett <clergy@palace.gov>
+            Series-version: 2
+            Series-changes: 4
+            - Some changes
+
+            Cover-letter:
+            test: A test patch series
+            This is a test of how the cover
+            leter
+            works
+            END
+
+        and this in the first commit:
+
+            Series-notes:
+            some notes
+            about some things
+            from the first commit
+            END
+
+            Commit-notes:
+            Some notes about
+            the first commit
+            END
+
+        with the following commands:
+
+           git log -n2 --reverse >/path/to/tools/patman/test/test01.txt
+           git format-patch --subject-prefix RFC --cover-letter HEAD~2
+           mv 00* /path/to/tools/patman/test
+
+        It checks these aspects:
+            - git log can be processed by patchstream
+            - emailing patches uses the correct command
+            - CC file has information on each commit
+            - cover letter has the expected text and subject
+            - each patch has the correct subject
+            - dry-run information prints out correctly
+            - unicode is handled correctly
+            - Series-to, Series-cc, Series-prefix, Cover-letter
+            - Cover-letter-cc, Series-version, Series-changes, Series-notes
+            - Commit-notes
+        """
+        process_tags = True
+        ignore_bad_tags = True
+        stefan = u'Stefan Brüns <stefan.bruens@rwth-aachen.de>'
+        rick = 'Richard III <richard@palace.gov>'
+        mel = u'Lord Mëlchett <clergy@palace.gov>'
+        ed = u'Lond Edmund Blackaddër <weasel at blackadder.org'
+        fred = 'Fred Bloggs <f.bloggs@napier.net>'
+        add_maintainers = [stefan, rick]
+        dry_run = True
+        in_reply_to = mel
+        count = 2
+        settings.alias = {
+                'fdt': ['simon'],
+                'u-boot': ['u-boot at lists.denx.de'],
+                'simon': [ed],
+                'fred': [fred],
+        }
+
+        text = self.GetText('test01.txt')
+        series = patchstream.GetMetaDataForTest(text)
+        cover_fname, args = self.CreatePatchesForTest(series)
+        with capture() as out:
+            patchstream.FixPatches(series, args)
+            if cover_fname and series.get('cover'):
+                patchstream.InsertCoverLetter(cover_fname, series, count)
+            series.DoChecks()
+            cc_file = series.MakeCcFile(process_tags, cover_fname,
+                                        not ignore_bad_tags, add_maintainers)
+            cmd = gitutil.EmailPatches(series, cover_fname, args,
+                    dry_run, not ignore_bad_tags, cc_file,
+                    in_reply_to=in_reply_to, thread=None)
+            series.ShowActions(args, cmd, process_tags)
+        cc_lines = open(cc_file).read().splitlines()
+        os.remove(cc_file)
+
+        lines = out[0].splitlines()
+        #print '\n'.join(lines)
+        self.assertEqual('Cleaned %s patches' % len(series.commits), lines[0])
+        self.assertEqual('Change log missing for v2', lines[1])
+        self.assertEqual('Change log missing for v3', lines[2])
+        self.assertEqual('Change log for unknown version v4', lines[3])
+        self.assertEqual("Alias 'pci' not found", lines[4])
+        self.assertIn('Dry run', lines[5])
+        self.assertIn('Send a total of %d patches' % count, lines[7])
+        line = 8
+        for i, commit in enumerate(series.commits):
+            self.assertEqual('   %s' % args[i], lines[line + 0])
+            line += 1
+            while 'Cc:' in lines[line]:
+                line += 1
+        self.assertEqual('To:	  u-boot at lists.denx.de', lines[line])
+        self.assertEqual('Cc:	  %s' % stefan.encode('utf-8'), lines[line + 1])
+        self.assertEqual('Version:  3', lines[line + 2])
+        self.assertEqual('Prefix:\t  RFC', lines[line + 3])
+        self.assertEqual('Cover: 4 lines', lines[line + 4])
+        line += 5
+        self.assertEqual('      Cc:  %s' % mel.encode('utf-8'), lines[line + 0])
+        self.assertEqual('      Cc:  %s' % rick, lines[line + 1])
+        self.assertEqual('      Cc:  %s' % fred, lines[line + 2])
+        self.assertEqual('      Cc:  %s' % ed.encode('utf-8'), lines[line + 3])
+        expected = ('Git command: git send-email --annotate '
+                    '--in-reply-to="%s" --to "u-boot at lists.denx.de" '
+                    '--cc "%s" --cc-cmd "%s --cc-cmd %s" %s %s'
+                    % (in_reply_to, stefan, sys.argv[0], cc_file, cover_fname,
+                       ' '.join(args))).encode('utf-8')
+        line += 4
+        self.assertEqual(expected, lines[line])
+
+        self.assertEqual(('%s %s, %s' % (args[0], rick, stefan))
+                         .encode('utf-8'), cc_lines[0])
+        self.assertEqual(('%s %s, %s, %s, %s' % (args[1], fred, rick, stefan,
+                                            ed)).encode('utf-8'), cc_lines[1])
+
+        expected = '''
+This is a test of how the cover
+leter
+works
+
+some notes
+about some things
+from the first commit
+
+Changes in v4:
+- Some changes
+
+Simon Glass (2):
+  pci: Correct cast for sandbox
+  fdt: Correct cast for sandbox in fdtdec_setup_memory_size()
+
+ cmd/pci.c                   | 3 ++-
+ fs/fat/fat.c                | 1 +
+ lib/efi_loader/efi_memory.c | 1 +
+ lib/fdtdec.c                | 3 ++-
+ 4 files changed, 6 insertions(+), 2 deletions(-)
+
+--\x20
+2.7.4
+
+'''
+        lines = open(cover_fname).read().splitlines()
+        #print '\n'.join(lines)
+        self.assertEqual(
+                'Subject: [RFC PATCH v3 0/2] test: A test patch series',
+                lines[3])
+        self.assertEqual(expected.splitlines(), lines[7:])
+
+        for i, fname in enumerate(args):
+            lines = open(fname).read().splitlines()
+            #print '\n'.join(lines)
+            subject = [line for line in lines if line.startswith('Subject')]
+            self.assertEqual('Subject: [RFC %d/%d]' % (i + 1, count),
+                             subject[0][:18])
+            if i == 0:
+                # Check that we got our commit notes
+                self.assertEqual('---', lines[17])
+                self.assertEqual('Some notes about', lines[18])
+                self.assertEqual('the first commit', lines[19])
diff --git a/tools/patman/patchstream.py b/tools/patman/patchstream.py
index 6098728aa1..1b9136aa5c 100644
--- a/tools/patman/patchstream.py
+++ b/tools/patman/patchstream.py
@@ -424,6 +424,19 @@ def GetMetaData(start, count):
     """
     return GetMetaDataForList('HEAD~%d' % start, None, count)
 
+def GetMetaDataForTest(text):
+    """Process metadata from a file containing a git log. Used for tests
+
+    Args:
+        text:
+    """
+    series = Series()
+    ps = PatchStream(series, is_log=True)
+    for line in text.splitlines():
+        ps.ProcessLine(line)
+    ps.Finalize()
+    return series
+
 def FixPatch(backup_dir, fname, series, commit):
     """Fix up a patch file, by adding/removing as required.
 
diff --git a/tools/patman/patman.py b/tools/patman/patman.py
index 63da3f3200..4b3bc78745 100755
--- a/tools/patman/patman.py
+++ b/tools/patman/patman.py
@@ -82,11 +82,13 @@ if __name__ != "__main__":
 # Run our meagre tests
 elif options.test:
     import doctest
+    import func_test
 
     sys.argv = [sys.argv[0]]
-    suite = unittest.TestLoader().loadTestsFromTestCase(test.TestPatch)
     result = unittest.TestResult()
-    suite.run(result)
+    for module in (test.TestPatch, func_test.TestFunctional):
+        suite = unittest.TestLoader().loadTestsFromTestCase(module)
+        suite.run(result)
 
     for module in ['gitutil', 'settings']:
         suite = doctest.DocTestSuite(module)
diff --git a/tools/patman/test/0000-cover-letter.patch b/tools/patman/test/0000-cover-letter.patch
new file mode 100644
index 0000000000..29062015bc
--- /dev/null
+++ b/tools/patman/test/0000-cover-letter.patch
@@ -0,0 +1,23 @@
+From 5ab48490f03051875ab13d288a4bf32b507d76fd Mon Sep 17 00:00:00 2001
+From: Simon Glass <sjg@chromium.org>
+Date: Sat, 27 May 2017 20:52:11 -0600
+Subject: [RFC 0/2] *** SUBJECT HERE ***
+MIME-Version: 1.0
+Content-Type: text/plain; charset=UTF-8
+Content-Transfer-Encoding: 8bit
+
+*** BLURB HERE ***
+
+Simon Glass (2):
+  pci: Correct cast for sandbox
+  fdt: Correct cast for sandbox in fdtdec_setup_memory_size()
+
+ cmd/pci.c                   | 3 ++-
+ fs/fat/fat.c                | 1 +
+ lib/efi_loader/efi_memory.c | 1 +
+ lib/fdtdec.c                | 3 ++-
+ 4 files changed, 6 insertions(+), 2 deletions(-)
+
+-- 
+2.7.4
+
diff --git a/tools/patman/test/0001-pci-Correct-cast-for-sandbox.patch b/tools/patman/test/0001-pci-Correct-cast-for-sandbox.patch
new file mode 100644
index 0000000000..7191176f75
--- /dev/null
+++ b/tools/patman/test/0001-pci-Correct-cast-for-sandbox.patch
@@ -0,0 +1,48 @@
+From b9da5f937bd5ea4931ea17459bf79b2905d9594d Mon Sep 17 00:00:00 2001
+From: Simon Glass <sjg@chromium.org>
+Date: Sat, 15 Apr 2017 15:39:08 -0600
+Subject: [RFC 1/2] pci: Correct cast for sandbox
+MIME-Version: 1.0
+Content-Type: text/plain; charset=UTF-8
+Content-Transfer-Encoding: 8bit
+
+This gives a warning with some native compilers:
+
+cmd/pci.c:152:11: warning: format ‘%llx’ expects argument of type
+   ‘long long unsigned int’, but argument 3 has type
+   ‘u64 {aka long unsigned int}’ [-Wformat=]
+
+Fix it with a cast.
+
+Signed-off-by: Simon Glass <sjg@chromium.org>
+Series-notes:
+some notes
+about some things
+from the first commit
+END
+
+Commit-notes:
+Some notes about
+the first commit
+END
+---
+ cmd/pci.c | 3 ++-
+ 1 file changed, 2 insertions(+), 1 deletion(-)
+
+diff --git a/cmd/pci.c b/cmd/pci.c
+index 41b4fff..fe27b4f 100644
+--- a/cmd/pci.c
++++ b/cmd/pci.c
+@@ -150,7 +150,8 @@ int pci_bar_show(struct udevice *dev)
+ 		if ((!is_64 && size_low) || (is_64 && size)) {
+ 			size = ~size + 1;
+ 			printf(" %d   %#016llx  %#016llx  %d     %s   %s\n",
+-			       bar_id, base, size, is_64 ? 64 : 32,
++			       bar_id, (unsigned long long)base,
++			       (unsigned long long)size, is_64 ? 64 : 32,
+ 			       is_io ? "I/O" : "MEM",
+ 			       prefetchable ? "Prefetchable" : "");
+ 		}
+-- 
+2.7.4
+
diff --git a/tools/patman/test/0002-fdt-Correct-cast-for-sandbox-in-fdtdec_setup_memory_.patch b/tools/patman/test/0002-fdt-Correct-cast-for-sandbox-in-fdtdec_setup_memory_.patch
new file mode 100644
index 0000000000..e3284973a0
--- /dev/null
+++ b/tools/patman/test/0002-fdt-Correct-cast-for-sandbox-in-fdtdec_setup_memory_.patch
@@ -0,0 +1,73 @@
+From 5ab48490f03051875ab13d288a4bf32b507d76fd Mon Sep 17 00:00:00 2001
+From: Simon Glass <sjg@chromium.org>
+Date: Sat, 15 Apr 2017 15:39:08 -0600
+Subject: [RFC 2/2] fdt: Correct cast for sandbox in fdtdec_setup_memory_size()
+MIME-Version: 1.0
+Content-Type: text/plain; charset=UTF-8
+Content-Transfer-Encoding: 8bit
+
+This gives a warning with some native compilers:
+
+lib/fdtdec.c:1203:8: warning: format ‘%llx’ expects argument of type
+   ‘long long unsigned int’, but argument 3 has type
+   ‘long unsigned int’ [-Wformat=]
+
+Fix it with a cast.
+
+Signed-off-by: Simon Glass <sjg@chromium.org>
+Series-to: u-boot
+Series-prefix: RFC
+Series-cc: Stefan Brüns <stefan.bruens@rwth-aachen.de>
+Cover-letter-cc: Lord Mëlchett <clergy@palace.gov>
+Series-version: 3
+Patch-cc: fred
+Series-changes: 4
+- Some changes
+
+Cover-letter:
+test: A test patch series
+This is a test of how the cover
+leter
+works
+END
+---
+ fs/fat/fat.c                | 1 +
+ lib/efi_loader/efi_memory.c | 1 +
+ lib/fdtdec.c                | 3 ++-
+ 3 files changed, 4 insertions(+), 1 deletion(-)
+
+diff --git a/fs/fat/fat.c b/fs/fat/fat.c
+index a71bad1..ba169dc 100644
+--- a/fs/fat/fat.c
++++ b/fs/fat/fat.c
+@@ -1,3 +1,4 @@
++
+ /*
+  * fat.c
+  *
+diff --git a/lib/efi_loader/efi_memory.c b/lib/efi_loader/efi_memory.c
+index db2ae19..05f75d1 100644
+--- a/lib/efi_loader/efi_memory.c
++++ b/lib/efi_loader/efi_memory.c
+@@ -1,3 +1,4 @@
++
+ /*
+  *  EFI application memory management
+  *
+diff --git a/lib/fdtdec.c b/lib/fdtdec.c
+index c072e54..942244f 100644
+--- a/lib/fdtdec.c
++++ b/lib/fdtdec.c
+@@ -1200,7 +1200,8 @@ int fdtdec_setup_memory_size(void)
+ 	}
+ 
+ 	gd->ram_size = (phys_size_t)(res.end - res.start + 1);
+-	debug("%s: Initial DRAM size %llx\n", __func__, (u64)gd->ram_size);
++	debug("%s: Initial DRAM size %llx\n", __func__,
++	      (unsigned long long)gd->ram_size);
+ 
+ 	return 0;
+ }
+-- 
+2.7.4
+
diff --git a/tools/patman/test/test01.txt b/tools/patman/test/test01.txt
new file mode 100644
index 0000000000..8ad9587aef
--- /dev/null
+++ b/tools/patman/test/test01.txt
@@ -0,0 +1,56 @@
+commit b9da5f937bd5ea4931ea17459bf79b2905d9594d
+Author: Simon Glass <sjg@chromium.org>
+Date:   Sat Apr 15 15:39:08 2017 -0600
+
+    pci: Correct cast for sandbox
+    
+    This gives a warning with some native compilers:
+    
+    cmd/pci.c:152:11: warning: format ‘%llx’ expects argument of type
+       ‘long long unsigned int’, but argument 3 has type
+       ‘u64 {aka long unsigned int}’ [-Wformat=]
+    
+    Fix it with a cast.
+    
+    Signed-off-by: Simon Glass <sjg@chromium.org>
+    Series-notes:
+    some notes
+    about some things
+    from the first commit
+    END
+    
+    Commit-notes:
+    Some notes about
+    the first commit
+    END
+
+commit 5ab48490f03051875ab13d288a4bf32b507d76fd
+Author: Simon Glass <sjg@chromium.org>
+Date:   Sat Apr 15 15:39:08 2017 -0600
+
+    fdt: Correct cast for sandbox in fdtdec_setup_memory_size()
+    
+    This gives a warning with some native compilers:
+    
+    lib/fdtdec.c:1203:8: warning: format ‘%llx’ expects argument of type
+       ‘long long unsigned int’, but argument 3 has type
+       ‘long unsigned int’ [-Wformat=]
+    
+    Fix it with a cast.
+    
+    Signed-off-by: Simon Glass <sjg@chromium.org>
+    Series-to: u-boot
+    Series-prefix: RFC
+    Series-cc: Stefan Brüns <stefan.bruens@rwth-aachen.de>
+    Cover-letter-cc: Lord Mëlchett <clergy@palace.gov>
+    Series-version: 3
+    Patch-cc: fred
+    Series-changes: 4
+    - Some changes
+    
+    Cover-letter:
+    test: A test patch series
+    This is a test of how the cover
+    leter
+    works
+    END
-- 
2.13.0.219.gdb65acc882-goog

^ permalink raw reply related	[flat|nested] 24+ messages in thread

* [U-Boot] [PATCH v2 00/10] patman: Fix unicode operation and add a functional test
  2017-05-29 21:31 [U-Boot] [PATCH v2 00/10] patman: Fix unicode operation and add a functional test Simon Glass
                   ` (9 preceding siblings ...)
  2017-05-29 21:31 ` [U-Boot] [PATCH v2 10/10] patman: Add a functional test Simon Glass
@ 2017-06-05 21:41 ` Simon Glass
  2017-06-05 22:50   ` Dr. Philipp Tomsich
  10 siblings, 1 reply; 24+ messages in thread
From: Simon Glass @ 2017-06-05 21:41 UTC (permalink / raw)
  To: u-boot

Hi,

On 29 May 2017 at 15:31, Simon Glass <sjg@chromium.org> wrote:
> With more recent versions of Python 2.7 (and perhaps patman itself)
> something has changed such that patman sometimes reports unicode decode
> errors, particularly when getting a list of maintainers for code covered
> by a patch.
>
> This series attempts to correct this. It also adds a functional test so
> that operation can quickly be verified. This new test tests much more of
> the code than the existing unit tests.
>
> Changes in v2:
> - Expand the patch to cover all cases
> - Drop RFC tag
> - Rewrite commit message
> - Add new clean-up patches and a functional test
>
> Philipp Tomsich (1):
>   patman: encode CC list to UTF-8
>
> Simon Glass (9):
>   patman: Adjust handling of unicode email address
>   patman: Don't convert input data to unicode
>   patman: Rename 'str' variable in EmailPatches()
>   patman: Don't report unicode character
>   patman: Don't return the series in FixPatches()
>   patman: Add unicode to test patches
>   patman: Add a maintainer test feature to MakeCcFile()
>   patman: Rename 'list' variable in MakeCcFile()
>   patman: Add a functional test
>
>  tools/patman/cros_subprocess.py                    |   4 -
>  tools/patman/func_test.py                          | 242 +++++++++++++++++++++
>  tools/patman/gitutil.py                            |   8 +-
>  tools/patman/patchstream.py                        |  23 +-
>  tools/patman/patman.py                             |  10 +-
>  tools/patman/series.py                             |  28 ++-
>  tools/patman/test.py                               |   9 +
>  tools/patman/test/0000-cover-letter.patch          |  23 ++
>  .../test/0001-pci-Correct-cast-for-sandbox.patch   |  48 ++++
>  ...-cast-for-sandbox-in-fdtdec_setup_memory_.patch |  73 +++++++
>  tools/patman/test/test01.txt                       |  56 +++++
>  11 files changed, 493 insertions(+), 31 deletions(-)
>  create mode 100644 tools/patman/func_test.py
>  create mode 100644 tools/patman/test/0000-cover-letter.patch
>  create mode 100644 tools/patman/test/0001-pci-Correct-cast-for-sandbox.patch
>  create mode 100644 tools/patman/test/0002-fdt-Correct-cast-for-sandbox-in-fdtdec_setup_memory_.patch
>  create mode 100644 tools/patman/test/test01.txt
>
> --
> 2.13.0.219.gdb65acc882-goog
>

Any comments on this one? I'm tempted to apply it to get people testing it!

Regards,
Simon

^ permalink raw reply	[flat|nested] 24+ messages in thread

* [U-Boot] [PATCH v2 00/10] patman: Fix unicode operation and add a functional test
  2017-06-05 21:41 ` [U-Boot] [PATCH v2 00/10] patman: Fix unicode operation and add " Simon Glass
@ 2017-06-05 22:50   ` Dr. Philipp Tomsich
  2017-06-06 21:40     ` Simon Glass
  0 siblings, 1 reply; 24+ messages in thread
From: Dr. Philipp Tomsich @ 2017-06-05 22:50 UTC (permalink / raw)
  To: u-boot

I have this on my tree since you sent it out and my last round of
patch-submissions was already done with this version of patman.
There were a few UTF-8 characters involved and all worked as
expected.

So you can consider it
	Tested-by: Philipp Tomsich <philipp.tomsich at theobroma-systems.com <mailto:philipp.tomsich@theobroma-systems.com>>

Cheers,
Phil.

> On 05 Jun 2017, at 23:41, Simon Glass <sjg@chromium.org> wrote:
> 
> Hi,
> 
> On 29 May 2017 at 15:31, Simon Glass <sjg@chromium.org> wrote:
>> With more recent versions of Python 2.7 (and perhaps patman itself)
>> something has changed such that patman sometimes reports unicode decode
>> errors, particularly when getting a list of maintainers for code covered
>> by a patch.
>> 
>> This series attempts to correct this. It also adds a functional test so
>> that operation can quickly be verified. This new test tests much more of
>> the code than the existing unit tests.
>> 
>> Changes in v2:
>> - Expand the patch to cover all cases
>> - Drop RFC tag
>> - Rewrite commit message
>> - Add new clean-up patches and a functional test
>> 
>> Philipp Tomsich (1):
>>  patman: encode CC list to UTF-8
>> 
>> Simon Glass (9):
>>  patman: Adjust handling of unicode email address
>>  patman: Don't convert input data to unicode
>>  patman: Rename 'str' variable in EmailPatches()
>>  patman: Don't report unicode character
>>  patman: Don't return the series in FixPatches()
>>  patman: Add unicode to test patches
>>  patman: Add a maintainer test feature to MakeCcFile()
>>  patman: Rename 'list' variable in MakeCcFile()
>>  patman: Add a functional test
>> 
>> tools/patman/cros_subprocess.py                    |   4 -
>> tools/patman/func_test.py                          | 242 +++++++++++++++++++++
>> tools/patman/gitutil.py                            |   8 +-
>> tools/patman/patchstream.py                        |  23 +-
>> tools/patman/patman.py                             |  10 +-
>> tools/patman/series.py                             |  28 ++-
>> tools/patman/test.py                               |   9 +
>> tools/patman/test/0000-cover-letter.patch          |  23 ++
>> .../test/0001-pci-Correct-cast-for-sandbox.patch   |  48 ++++
>> ...-cast-for-sandbox-in-fdtdec_setup_memory_.patch |  73 +++++++
>> tools/patman/test/test01.txt                       |  56 +++++
>> 11 files changed, 493 insertions(+), 31 deletions(-)
>> create mode 100644 tools/patman/func_test.py
>> create mode 100644 tools/patman/test/0000-cover-letter.patch
>> create mode 100644 tools/patman/test/0001-pci-Correct-cast-for-sandbox.patch
>> create mode 100644 tools/patman/test/0002-fdt-Correct-cast-for-sandbox-in-fdtdec_setup_memory_.patch
>> create mode 100644 tools/patman/test/test01.txt
>> 
>> --
>> 2.13.0.219.gdb65acc882-goog
>> 
> 
> Any comments on this one? I'm tempted to apply it to get people testing it!
> 
> Regards,
> Simon

^ permalink raw reply	[flat|nested] 24+ messages in thread

* [U-Boot] [PATCH v2 00/10] patman: Fix unicode operation and add a functional test
  2017-06-05 22:50   ` Dr. Philipp Tomsich
@ 2017-06-06 21:40     ` Simon Glass
  0 siblings, 0 replies; 24+ messages in thread
From: Simon Glass @ 2017-06-06 21:40 UTC (permalink / raw)
  To: u-boot

On 5 June 2017 at 16:50, Dr. Philipp Tomsich
<philipp.tomsich@theobroma-systems.com> wrote:
> I have this on my tree since you sent it out and my last round of
> patch-submissions was already done with this version of patman.
> There were a few UTF-8 characters involved and all worked as
> expected.
>
> So you can consider it
> Tested-by: Philipp Tomsich <philipp.tomsich@theobroma-systems.com>

Thanks - I've added your tag.

- Simon

^ permalink raw reply	[flat|nested] 24+ messages in thread

* [U-Boot] [PATCH v2 10/10] patman: Add a functional test
  2017-05-29 21:31 ` [U-Boot] [PATCH v2 10/10] patman: Add a functional test Simon Glass
@ 2017-06-07  1:47   ` sjg at google.com
  0 siblings, 0 replies; 24+ messages in thread
From: sjg at google.com @ 2017-06-07  1:47 UTC (permalink / raw)
  To: u-boot

The existing test (patman --test) only covers basic checkpatch output.
We have had some problems with unicode processing and could use test
coverage for the various tags patman supports.

Add a new functional test which runs most of the patman flow on a few
test commits and checks that the results are correct.

See the documentation in the test for a description of what it does.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

Changes in v2:
- Add new clean-up patches and a functional test

 tools/patman/func_test.py                          | 242 +++++++++++++++++++++
 tools/patman/patchstream.py                        |  13 ++
 tools/patman/patman.py                             |   6 +-
 tools/patman/test/0000-cover-letter.patch          |  23 ++
 .../test/0001-pci-Correct-cast-for-sandbox.patch   |  48 ++++
 ...-cast-for-sandbox-in-fdtdec_setup_memory_.patch |  73 +++++++
 tools/patman/test/test01.txt                       |  56 +++++
 7 files changed, 459 insertions(+), 2 deletions(-)
 create mode 100644 tools/patman/func_test.py
 create mode 100644 tools/patman/test/0000-cover-letter.patch
 create mode 100644 tools/patman/test/0001-pci-Correct-cast-for-sandbox.patch
 create mode 100644
tools/patman/test/0002-fdt-Correct-cast-for-sandbox-in-fdtdec_setup_memory_.patch
 create mode 100644 tools/patman/test/test01.txt

Applied to u-boot-dm, thanks!

^ permalink raw reply	[flat|nested] 24+ messages in thread

* [U-Boot] [PATCH v2 09/10] patman: Rename 'list' variable in MakeCcFile()
  2017-05-29 21:31 ` [U-Boot] [PATCH v2 09/10] patman: Rename 'list' variable in MakeCcFile() Simon Glass
@ 2017-06-07  1:47   ` sjg at google.com
  0 siblings, 0 replies; 24+ messages in thread
From: sjg at google.com @ 2017-06-07  1:47 UTC (permalink / raw)
  To: u-boot

This is not a good variable name in Python because 'list' is a type. It
shows up highlighted in some editors. Rename it.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

Changes in v2: None

 tools/patman/series.py | 20 ++++++++++----------
 1 file changed, 10 insertions(+), 10 deletions(-)

Applied to u-boot-dm, thanks!

^ permalink raw reply	[flat|nested] 24+ messages in thread

* [U-Boot] [PATCH v2 08/10] patman: Add a maintainer test feature to MakeCcFile()
  2017-05-29 21:31 ` [U-Boot] [PATCH v2 08/10] patman: Add a maintainer test feature to MakeCcFile() Simon Glass
@ 2017-06-07  1:47   ` sjg at google.com
  0 siblings, 0 replies; 24+ messages in thread
From: sjg at google.com @ 2017-06-07  1:47 UTC (permalink / raw)
  To: u-boot

Allow the add_maintainers parameter to be a list of maintainers, thus
allowing us to simulate calling the script in tests without actually
needing it to work.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

Changes in v2: None

 tools/patman/series.py | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

Applied to u-boot-dm, thanks!

^ permalink raw reply	[flat|nested] 24+ messages in thread

* [U-Boot] [PATCH v2 07/10] patman: Add unicode to test patches
  2017-05-29 21:31 ` [U-Boot] [PATCH v2 07/10] patman: Add unicode to test patches Simon Glass
@ 2017-06-07  1:47   ` sjg at google.com
  0 siblings, 0 replies; 24+ messages in thread
From: sjg at google.com @ 2017-06-07  1:47 UTC (permalink / raw)
  To: u-boot

Add some unicode to the test patches to make sure that patman does the
right thing.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

Changes in v2: None

 tools/patman/test.py | 9 +++++++++
 1 file changed, 9 insertions(+)

Applied to u-boot-dm, thanks!

^ permalink raw reply	[flat|nested] 24+ messages in thread

* [U-Boot] [PATCH v2 06/10] patman: Don't return the series in FixPatches()
  2017-05-29 21:31 ` [U-Boot] [PATCH v2 06/10] patman: Don't return the series in FixPatches() Simon Glass
@ 2017-06-07  1:47   ` sjg at google.com
  0 siblings, 0 replies; 24+ messages in thread
From: sjg at google.com @ 2017-06-07  1:47 UTC (permalink / raw)
  To: u-boot

There is no need for this function to return the same object that was
passed in. Drop the return value.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

Changes in v2: None

 tools/patman/patchstream.py | 1 -
 tools/patman/patman.py      | 4 ++--
 2 files changed, 2 insertions(+), 3 deletions(-)

Applied to u-boot-dm, thanks!

^ permalink raw reply	[flat|nested] 24+ messages in thread

* [U-Boot] [PATCH v2 04/10] patman: Rename 'str' variable in EmailPatches()
  2017-05-29 21:31 ` [U-Boot] [PATCH v2 04/10] patman: Rename 'str' variable in EmailPatches() Simon Glass
@ 2017-06-07  1:47   ` sjg at google.com
  0 siblings, 0 replies; 24+ messages in thread
From: sjg at google.com @ 2017-06-07  1:47 UTC (permalink / raw)
  To: u-boot

This is not a good variable name in Python because 'str' is a type. It
shows up highlighted in some editors. Rename it.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

Changes in v2: None

 tools/patman/gitutil.py | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

Applied to u-boot-dm, thanks!

^ permalink raw reply	[flat|nested] 24+ messages in thread

* [U-Boot] [PATCH v2 05/10] patman: Don't report unicode character
  2017-05-29 21:31 ` [U-Boot] [PATCH v2 05/10] patman: Don't report unicode character Simon Glass
@ 2017-06-07  1:47   ` sjg at google.com
  0 siblings, 0 replies; 24+ messages in thread
From: sjg at google.com @ 2017-06-07  1:47 UTC (permalink / raw)
  To: u-boot

Unicode characters may appear in input patches so we should not warn about
them. Drop this warning.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

Changes in v2: None

 tools/patman/patchstream.py | 9 ---------
 1 file changed, 9 deletions(-)

Applied to u-boot-dm, thanks!

^ permalink raw reply	[flat|nested] 24+ messages in thread

* [U-Boot] [PATCH v2 03/10] patman: Don't convert input data to unicode
  2017-05-29 21:31 ` [U-Boot] [PATCH v2 03/10] patman: Don't convert input data to unicode Simon Glass
@ 2017-06-07  1:47   ` sjg at google.com
  0 siblings, 0 replies; 24+ messages in thread
From: sjg at google.com @ 2017-06-07  1:47 UTC (permalink / raw)
  To: u-boot

The communication filter reads data in blocks and converts each block to
unicode (if necessary) one at a time. In the unlikely event that a unicode
character in the input spans a block this will not work. We get an error
like:

UnicodeDecodeError: 'utf8' codec can't decode bytes in position 1022-1023:
   unexpected end of data

There is no need to change the input to unicode, so the easiest fix is to
drop this feature.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

Changes in v2: None

 tools/patman/cros_subprocess.py | 4 ----
 1 file changed, 4 deletions(-)

Applied to u-boot-dm, thanks!

^ permalink raw reply	[flat|nested] 24+ messages in thread

* [U-Boot] [PATCH v2 01/10] patman: encode CC list to UTF-8
  2017-05-29 21:31 ` [U-Boot] [PATCH v2 01/10] patman: encode CC list to UTF-8 Simon Glass
@ 2017-06-07  1:47   ` sjg at google.com
  0 siblings, 0 replies; 24+ messages in thread
From: sjg at google.com @ 2017-06-07  1:47 UTC (permalink / raw)
  To: u-boot

From: Philipp Tomsich <philipp.tomsich@theobroma-systems.com>

This change encodes the CC list to UTF-8 to avoid failures on
maintainer-addresses that include non-ASCII characters (observed on
Debian 7.11 with Python 2.7.3).

Without this, I get the following failure:
  Traceback (most recent call last):
    File "tools/patman/patman", line 159, in <module>
      options.add_maintainers)
    File "[snip]/u-boot/tools/patman/series.py", line 234, in MakeCcFile
      print(commit.patch, ', '.join(set(list)), file=fd)
  UnicodeEncodeError: 'ascii' codec can't encode character u'\xfc' in
position 81: ordinal not in range(128)
from Heiko's email address:
  [..., u'"Heiko St\xfcbner" <heiko@sntech.de>', ...]

While with this change added this encodes to:
  "=?UTF-8?q?Heiko=20St=C3=BCbner?= <heiko@sntech.de>"

Signed-off-by: Philipp Tomsich <philipp.tomsich@theobroma-systems.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
Signed-off-by: Simon Glass <sjg@chromium.org>
---

Changes in v2: None

 tools/patman/series.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

Applied to u-boot-dm, thanks!

^ permalink raw reply	[flat|nested] 24+ messages in thread

* [U-Boot] [PATCH v2 02/10] patman: Adjust handling of unicode email address
  2017-05-29 21:31 ` [U-Boot] [PATCH v2 02/10] patman: Adjust handling of unicode email address Simon Glass
@ 2017-06-07  1:47   ` sjg at google.com
  0 siblings, 0 replies; 24+ messages in thread
From: sjg at google.com @ 2017-06-07  1:47 UTC (permalink / raw)
  To: u-boot

Don't mess with the email address when outputting them. Just make sure
they are encoded with utf-8.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

Changes in v2:
- Expand the patch to cover all cases
- Drop RFC tag
- Rewrite commit message

 tools/patman/gitutil.py |  2 ++
 tools/patman/series.py  | 10 +++++++---
 2 files changed, 9 insertions(+), 3 deletions(-)

Applied to u-boot-dm, thanks!

^ permalink raw reply	[flat|nested] 24+ messages in thread

end of thread, other threads:[~2017-06-07  1:47 UTC | newest]

Thread overview: 24+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-05-29 21:31 [U-Boot] [PATCH v2 00/10] patman: Fix unicode operation and add a functional test Simon Glass
2017-05-29 21:31 ` [U-Boot] [PATCH v2 01/10] patman: encode CC list to UTF-8 Simon Glass
2017-06-07  1:47   ` sjg at google.com
2017-05-29 21:31 ` [U-Boot] [PATCH v2 02/10] patman: Adjust handling of unicode email address Simon Glass
2017-06-07  1:47   ` sjg at google.com
2017-05-29 21:31 ` [U-Boot] [PATCH v2 03/10] patman: Don't convert input data to unicode Simon Glass
2017-06-07  1:47   ` sjg at google.com
2017-05-29 21:31 ` [U-Boot] [PATCH v2 04/10] patman: Rename 'str' variable in EmailPatches() Simon Glass
2017-06-07  1:47   ` sjg at google.com
2017-05-29 21:31 ` [U-Boot] [PATCH v2 05/10] patman: Don't report unicode character Simon Glass
2017-06-07  1:47   ` sjg at google.com
2017-05-29 21:31 ` [U-Boot] [PATCH v2 06/10] patman: Don't return the series in FixPatches() Simon Glass
2017-06-07  1:47   ` sjg at google.com
2017-05-29 21:31 ` [U-Boot] [PATCH v2 07/10] patman: Add unicode to test patches Simon Glass
2017-06-07  1:47   ` sjg at google.com
2017-05-29 21:31 ` [U-Boot] [PATCH v2 08/10] patman: Add a maintainer test feature to MakeCcFile() Simon Glass
2017-06-07  1:47   ` sjg at google.com
2017-05-29 21:31 ` [U-Boot] [PATCH v2 09/10] patman: Rename 'list' variable in MakeCcFile() Simon Glass
2017-06-07  1:47   ` sjg at google.com
2017-05-29 21:31 ` [U-Boot] [PATCH v2 10/10] patman: Add a functional test Simon Glass
2017-06-07  1:47   ` sjg at google.com
2017-06-05 21:41 ` [U-Boot] [PATCH v2 00/10] patman: Fix unicode operation and add " Simon Glass
2017-06-05 22:50   ` Dr. Philipp Tomsich
2017-06-06 21:40     ` Simon Glass

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.