[U-Boot] [PATCH v2 0/8] test/py: Fixes for python 3.x

[U-Boot] [PATCH v2 0/8] test/py: Fixes for python 3.x

[Paul Burton]

This series allows test/py to run on python 3.x in addition to the
already supported python 2.x. With it applied I currently see only one
failure (test_ut[ut_dm_usb_flash]) which doesn't appear to be python
related when running with:

$ ./test/py/test.py --bd sandbox --build

Thanks,
    Paul

Paul Burton (8):
  test/py: Make print statements python 3.x safe
  test/py: Use range() rather than xrange()
  test/py: Import 'configparser' lower case to be python 3.x safe
  test/py: Import StringIO from io module for python 3.x
  test/py: Encode/decode strings for stdio
  test/py: fit: Open files as binary files
  test/py: hush_if_test: Use open() in place of file()
  test/py: vboot: Remove stderr redirect from openssl command

 test/py/conftest.py                | 18 +++++++++++++-----
 test/py/multiplexed_log.py         |  4 ++--
 test/py/test.py                    |  6 ++++--
 test/py/tests/test_fit.py          | 14 ++++++++------
 test/py/tests/test_hush_if_test.py |  2 +-
 test/py/tests/test_ut.py           |  2 +-
 test/py/tests/test_vboot.py        |  4 ++--
 test/py/u_boot_console_sandbox.py  |  4 ++--
 test/py/u_boot_spawn.py            | 12 ++++++------
 test/py/u_boot_utils.py            |  4 ++--
 10 files changed, 41 insertions(+), 29 deletions(-)

-- 
2.14.1

[U-Boot] [PATCH v2 1/8] test/py: Make print statements python 3.x safe

[Paul Burton]

In python 3.x print must be called as a function rather than used as a
statement. Update uses of print to the function call syntax in order to
be python 3.x safe.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
Reviewed-by: Stephen Warren <swarren@nvidia.com>
---

Changes in v2: None

 test/py/conftest.py               |  2 +-
 test/py/test.py                   |  6 ++++--
 test/py/tests/test_fit.py         | 12 +++++++-----
 test/py/u_boot_console_sandbox.py |  2 +-
 test/py/u_boot_spawn.py           |  2 +-
 5 files changed, 14 insertions(+), 10 deletions(-)

diff --git a/test/py/conftest.py b/test/py/conftest.py
index 65e1d75626..ced96f1006 100644
--- a/test/py/conftest.py
+++ b/test/py/conftest.py
@@ -545,7 +545,7 @@ def pytest_runtest_protocol(item, nextitem):
         # is fixed, if this exception still exists, it will then be logged as
         # part of the test's stdout.
         import traceback
-        print 'Exception occurred while logging runtest status:'
+        print('Exception occurred while logging runtest status:')
         traceback.print_exc()
         # FIXME: Can we force a test failure here?
 
diff --git a/test/py/test.py b/test/py/test.py
index 74e560a4d3..215f2651a7 100755
--- a/test/py/test.py
+++ b/test/py/test.py
@@ -8,6 +8,8 @@
 # Wrapper script to invoke pytest with the directory name that contains the
 # U-Boot tests.
 
+from __future__ import print_function
+
 import os
 import os.path
 import sys
@@ -27,7 +29,7 @@ except:
     traceback.print_exc()
     # Hint to the user that they likely simply haven't installed the required
     # dependencies.
-    print >>sys.stderr, '''
+    print('''
 exec(py.test) failed; perhaps you are missing some dependencies?
-See test/py/README.md for the list.'''
+See test/py/README.md for the list.''', file=sys.stderr)
     sys.exit(1)
diff --git a/test/py/tests/test_fit.py b/test/py/tests/test_fit.py
index 7e6b96dae4..29b2491f89 100755
--- a/test/py/tests/test_fit.py
+++ b/test/py/tests/test_fit.py
@@ -4,6 +4,8 @@
 #
 # Sanity check of the FIT handling in U-Boot
 
+from __future__ import print_function
+
 import os
 import pytest
 import struct
@@ -153,7 +155,7 @@ def test_fit(u_boot_console):
         src = make_fname('u-boot.dts')
         dtb = make_fname('u-boot.dtb')
         with open(src, 'w') as fd:
-            print >> fd, base_fdt
+            print(base_fdt, file=fd)
         util.run_and_log(cons, ['dtc', src, '-O', 'dtb', '-o', dtb])
         return dtb
 
@@ -167,7 +169,7 @@ def test_fit(u_boot_console):
         """
         its = make_fname('test.its')
         with open(its, 'w') as fd:
-            print >> fd, base_its % params
+            print(base_its % params, file=fd)
         return its
 
     def make_fit(mkimage, params):
@@ -186,7 +188,7 @@ def test_fit(u_boot_console):
         its = make_its(params)
         util.run_and_log(cons, [mkimage, '-f', its, fit])
         with open(make_fname('u-boot.dts'), 'w') as fd:
-            print >> fd, base_fdt
+            print(base_fdt, file=fd)
         return fit
 
     def make_kernel(filename, text):
@@ -202,7 +204,7 @@ def test_fit(u_boot_console):
         for i in range(100):
             data += 'this %s %d is unlikely to boot\n' % (text, i)
         with open(fname, 'w') as fd:
-            print >> fd, data
+            print(data, file=fd)
         return fname
 
     def make_ramdisk(filename, text):
@@ -216,7 +218,7 @@ def test_fit(u_boot_console):
         for i in range(100):
             data += '%s %d was seldom used in the middle ages\n' % (text, i)
         with open(fname, 'w') as fd:
-            print >> fd, data
+            print(data, file=fd)
         return fname
 
     def find_matching(text, match):
diff --git a/test/py/u_boot_console_sandbox.py b/test/py/u_boot_console_sandbox.py
index 647e1f879f..9a2fe9cb59 100644
--- a/test/py/u_boot_console_sandbox.py
+++ b/test/py/u_boot_console_sandbox.py
@@ -42,7 +42,7 @@ class ConsoleSandbox(ConsoleBase):
         bcfg = self.config.buildconfig
         config_spl = bcfg.get('config_spl', 'n') == 'y'
         fname = '/spl/u-boot-spl' if config_spl else '/u-boot'
-        print fname
+        print(fname)
         cmd = []
         if self.config.gdbserver:
             cmd += ['gdbserver', self.config.gdbserver]
diff --git a/test/py/u_boot_spawn.py b/test/py/u_boot_spawn.py
index 3a0fbfad90..15ca4ac838 100644
--- a/test/py/u_boot_spawn.py
+++ b/test/py/u_boot_spawn.py
@@ -59,7 +59,7 @@ class Spawn(object):
                     os.chdir(cwd)
                 os.execvp(args[0], args)
             except:
-                print 'CHILD EXECEPTION:'
+                print('CHILD EXECEPTION:')
                 import traceback
                 traceback.print_exc()
             finally:
-- 
2.14.1

[U-Boot] [PATCH v2 2/8] test/py: Use range() rather than xrange()

[Paul Burton]

In python 3.x the xrange() function has been removed, and range()
returns an iterator much like Python 2.x's xrange(). Simply use range()
in place of xrange() in order to work on both python 2.x & 3.x. This
will mean a small cost on python 2.x since range() will return a list
there rather than an iterator, but the cost should be negligible.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
Reviewed-by: Stephen Warren <swarren@nvidia.com>
---

Changes in v2: None

 test/py/u_boot_console_sandbox.py | 2 +-
 test/py/u_boot_spawn.py           | 6 +++---
 test/py/u_boot_utils.py           | 4 ++--
 3 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/test/py/u_boot_console_sandbox.py b/test/py/u_boot_console_sandbox.py
index 9a2fe9cb59..5c91d926c5 100644
--- a/test/py/u_boot_console_sandbox.py
+++ b/test/py/u_boot_console_sandbox.py
@@ -82,7 +82,7 @@ class ConsoleSandbox(ConsoleBase):
 
         p = self.p
         self.p = None
-        for i in xrange(100):
+        for i in range(100):
             ret = not p.isalive()
             if ret:
                 break
diff --git a/test/py/u_boot_spawn.py b/test/py/u_boot_spawn.py
index 15ca4ac838..638c5dd31d 100644
--- a/test/py/u_boot_spawn.py
+++ b/test/py/u_boot_spawn.py
@@ -135,7 +135,7 @@ class Spawn(object):
             the expected time.
         """
 
-        for pi in xrange(len(patterns)):
+        for pi in range(len(patterns)):
             if type(patterns[pi]) == type(''):
                 patterns[pi] = re.compile(patterns[pi])
 
@@ -144,7 +144,7 @@ class Spawn(object):
             while True:
                 earliest_m = None
                 earliest_pi = None
-                for pi in xrange(len(patterns)):
+                for pi in range(len(patterns)):
                     pattern = patterns[pi]
                     m = pattern.search(self.buf)
                     if not m:
@@ -199,7 +199,7 @@ class Spawn(object):
         """
 
         os.close(self.fd)
-        for i in xrange(100):
+        for i in range(100):
             if not self.isalive():
                 break
             time.sleep(0.1)
diff --git a/test/py/u_boot_utils.py b/test/py/u_boot_utils.py
index 2ba4baed07..264508da4e 100644
--- a/test/py/u_boot_utils.py
+++ b/test/py/u_boot_utils.py
@@ -120,7 +120,7 @@ def wait_until_open_succeeds(fn):
         An open file handle to the file.
     """
 
-    for i in xrange(100):
+    for i in range(100):
         fh = attempt_to_open_file(fn)
         if fh:
             return fh
@@ -143,7 +143,7 @@ def wait_until_file_open_fails(fn, ignore_errors):
         Nothing.
     """
 
-    for i in xrange(100):
+    for i in range(100):
         fh = attempt_to_open_file(fn)
         if not fh:
             return
-- 
2.14.1

[U-Boot] [PATCH v2 3/8] test/py: Import 'configparser' lower case to be python 3.x safe

[Paul Burton]

In python 3.x the configparser module is named with all lower case.
Import it as such in order to avoid errors when running on python 3.x,
and fall back to the CamelCase version in order to keep working with
python 2.x.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
Reviewed-by: Stephen Warren <swarren@nvidia.com>

---

Changes in v2:
- Import python 2.x ConfigParser as the python 3.x style configparser, rather than the other way around.

 test/py/conftest.py | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/test/py/conftest.py b/test/py/conftest.py
index ced96f1006..bf55bd3d13 100644
--- a/test/py/conftest.py
+++ b/test/py/conftest.py
@@ -19,11 +19,15 @@ import os
 import os.path
 import pytest
 from _pytest.runner import runtestprotocol
-import ConfigParser
 import re
 import StringIO
 import sys
 
+try:
+    import configparser
+except:
+    import ConfigParser as configparser
+
 # Globals: The HTML log file, and the connection to the U-Boot console.
 log = None
 console = None
@@ -167,7 +171,7 @@ def pytest_configure(config):
         with open(dot_config, 'rt') as f:
             ini_str = '[root]\n' + f.read()
             ini_sio = StringIO.StringIO(ini_str)
-            parser = ConfigParser.RawConfigParser()
+            parser = configparser.RawConfigParser()
             parser.readfp(ini_sio)
             ubconfig.buildconfig.update(parser.items('root'))
 
-- 
2.14.1

[U-Boot] [PATCH v2 4/8] test/py: Import StringIO from io module for python 3.x

[Paul Burton]

In python 3.x the StringIO module is gone, and instead StringIO can be
imported from the io module. Do this in order to run on python 3.x, and
fall back to importing StringIO as a module in order to continue working
with python 2.x.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
Reviewed-by: Stephen Warren <swarren@nvidia.com>

---

Changes in v2:
- Try python 3.x style, fall back to python 2.x style rather than the reverse.

 test/py/conftest.py | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/test/py/conftest.py b/test/py/conftest.py
index bf55bd3d13..0d564942d0 100644
--- a/test/py/conftest.py
+++ b/test/py/conftest.py
@@ -20,7 +20,6 @@ import os.path
 import pytest
 from _pytest.runner import runtestprotocol
 import re
-import StringIO
 import sys
 
 try:
@@ -28,6 +27,11 @@ try:
 except:
     import ConfigParser as configparser
 
+try:
+    from io import StringIO
+except ImportError:
+    from StringIO import StringIO
+
 # Globals: The HTML log file, and the connection to the U-Boot console.
 log = None
 console = None
@@ -170,7 +174,7 @@ def pytest_configure(config):
 
         with open(dot_config, 'rt') as f:
             ini_str = '[root]\n' + f.read()
-            ini_sio = StringIO.StringIO(ini_str)
+            ini_sio = StringIO(ini_str)
             parser = configparser.RawConfigParser()
             parser.readfp(ini_sio)
             ubconfig.buildconfig.update(parser.items('root'))
-- 
2.14.1

[U-Boot] [PATCH v2 5/8] test/py: Encode/decode strings for stdio

[Paul Burton]

When reading stdin python 3.x will give us byte arrays, and when writing
stdout or stderr it will expect byte arrays. In order to insulate the
rest of the code from this difference, call encode or decode at
appropriate points when reading or writing stdio files. This works fine
on python 2.x too.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
Reviewed-by: Stephen Warren <swarren@nvidia.com>
---

Changes in v2: None

 test/py/multiplexed_log.py | 4 ++--
 test/py/tests/test_ut.py   | 2 +-
 test/py/u_boot_spawn.py    | 4 ++--
 3 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/test/py/multiplexed_log.py b/test/py/multiplexed_log.py
index bf926c3e77..21bdcb7309 100644
--- a/test/py/multiplexed_log.py
+++ b/test/py/multiplexed_log.py
@@ -140,11 +140,11 @@ class RunAndLog(object):
             if stdout:
                 if stderr:
                     output += 'stdout:\n'
-                output += stdout
+                output += stdout.decode('utf8')
             if stderr:
                 if stdout:
                     output += 'stderr:\n'
-                output += stderr
+                output += stderr.decode('utf8')
             exit_status = p.returncode
             exception = None
         except subprocess.CalledProcessError as cpe:
diff --git a/test/py/tests/test_ut.py b/test/py/tests/test_ut.py
index 5c25a2d465..1f40e2c2d0 100644
--- a/test/py/tests/test_ut.py
+++ b/test/py/tests/test_ut.py
@@ -14,7 +14,7 @@ def test_ut_dm_init(u_boot_console):
         data = 'this is a test'
         data += '\x00' * ((4 * 1024 * 1024) - len(data))
         with open(fn, 'wb') as fh:
-            fh.write(data)
+            fh.write(data.encode('utf-8'))
 
     fn = u_boot_console.config.source_dir + '/spi.bin'
     if not os.path.exists(fn):
diff --git a/test/py/u_boot_spawn.py b/test/py/u_boot_spawn.py
index 638c5dd31d..751302a529 100644
--- a/test/py/u_boot_spawn.py
+++ b/test/py/u_boot_spawn.py
@@ -114,7 +114,7 @@ class Spawn(object):
             Nothing.
         """
 
-        os.write(self.fd, data)
+        os.write(self.fd, data.encode('utf8'))
 
     def expect(self, patterns):
         """Wait for the sub-process to emit specific data.
@@ -172,7 +172,7 @@ class Spawn(object):
                 events = self.poll.poll(poll_maxwait)
                 if not events:
                     raise Timeout()
-                c = os.read(self.fd, 1024)
+                c = os.read(self.fd, 1024).decode('utf8')
                 if not c:
                     raise EOFError()
                 if self.logfile_read:
-- 
2.14.1

[U-Boot] [PATCH v2 6/8] test/py: fit: Open files as binary files

[Paul Burton]

The read_file() function in test_fit is used with files that are not
text files, as well as some that are. It is never used in a way that
requires it to decode text files to characters, so open all files in
binary mode such that read() doesn't attempt to decode characters for
files which are not text files.

Without this test_fit fails on python 3.x when reading an FDT in
run_fit_test() with:

  UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd0 in position
                       0: invalid continuation byte

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
Reviewed-by: Stephen Warren <swarren@nvidia.com>
---

Changes in v2: None

 test/py/tests/test_fit.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/test/py/tests/test_fit.py b/test/py/tests/test_fit.py
index 29b2491f89..01ecde27d5 100755
--- a/test/py/tests/test_fit.py
+++ b/test/py/tests/test_fit.py
@@ -143,7 +143,7 @@ def test_fit(u_boot_console):
         Returns:
             Contents of file as a string
         """
-        with open(fname, 'r') as fd:
+        with open(fname, 'rb') as fd:
             return fd.read()
 
     def make_dtb():
-- 
2.14.1

[U-Boot] [PATCH v2 7/8] test/py: hush_if_test: Use open() in place of file()

[Paul Burton]

In python 3.x the file() function has been removed. Use open() instead,
which works on both python 2.x & 3.x, and is described as the preferred
method of opening a file by python 2.x documentation anyway.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
Reviewed-by: Stephen Warren <swarren@nvidia.com>
---

Changes in v2: None

 test/py/tests/test_hush_if_test.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/test/py/tests/test_hush_if_test.py b/test/py/tests/test_hush_if_test.py
index c8f4208d31..cdad22b429 100644
--- a/test/py/tests/test_hush_if_test.py
+++ b/test/py/tests/test_hush_if_test.py
@@ -149,7 +149,7 @@ def test_hush_if_test_host_file_exists(u_boot_console):
     exec_hush_if(u_boot_console, expr, False)
 
     try:
-        with file(test_file, 'wb'):
+        with open(test_file, 'wb'):
             pass
         assert os.path.exists(test_file)
 
-- 
2.14.1

[U-Boot] [PATCH v2 8/8] test/py: vboot: Remove stderr redirect from openssl command

[Paul Burton]

The openssl command specified in test_with_algo() ultimately ends up
being run by RunAndLog::run(), which uses it to construct a Popen object
with the default shell=False. The stderr redirect in the command is
therefore simply passed to openssl as an argument. With at least openssl
1.1.0f this causes openssl, and therefore test_vboot, to fail with:

  genpkey: Use -help for summary.
  Exit code: 1

Any stderr output ought to be captured & stored in the RunAndLog
object's output field and returned from run() via run_and_log() to
test_with_algo() which then ignores it anyway, so we can drop the
shell-like redirection with no ill effects. With this fix test_vboot now
passes for me.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
Reviewed-by: Stephen Warren <swarren@nvidia.com>

---

Changes in v2:
- Place the % operator at the end of the last string line, not the start of the next.

 test/py/tests/test_vboot.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/test/py/tests/test_vboot.py b/test/py/tests/test_vboot.py
index 6e62820743..b1170aa4b0 100644
--- a/test/py/tests/test_vboot.py
+++ b/test/py/tests/test_vboot.py
@@ -174,8 +174,8 @@ def test_vboot(u_boot_console):
     public_exponent = 65537
     util.run_and_log(cons, 'openssl genpkey -algorithm RSA -out %sdev.key '
                      '-pkeyopt rsa_keygen_bits:2048 '
-                     '-pkeyopt rsa_keygen_pubexp:%d '
-                     '2>/dev/null'  % (tmpdir, public_exponent))
+                     '-pkeyopt rsa_keygen_pubexp:%d' %
+                     (tmpdir, public_exponent))
 
     # Create a certificate containing the public key
     util.run_and_log(cons, 'openssl req -batch -new -x509 -key %sdev.key -out '
-- 
2.14.1

[U-Boot] [U-Boot, v2, 4/8] test/py: Import StringIO from io module for python 3.x

[Tom Rini]

On Thu, Sep 14, 2017 at 02:34:46PM -0700, Paul Burton wrote:

> In python 3.x the StringIO module is gone, and instead StringIO can be
> imported from the io module. Do this in order to run on python 3.x, and
> fall back to importing StringIO as a module in order to continue working
> with python 2.x.
> 
> Signed-off-by: Paul Burton <paul.burton@imgtec.com>
> Reviewed-by: Stephen Warren <swarren@nvidia.com>

This breaks python2.7.  There we have both StringIO and io.StringIO so
we succeed in 'from io import StringIO'.  But io.StringIO() needs
unicode, and we don't have that by default in python2.  My quick attempt
here failed.  Thanks!

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20170921/bda58c3e/attachment-0001.sig>

[U-Boot] [U-Boot, v2, 4/8] test/py: Import StringIO from io module for python 3.x

[Paul Burton]

Hi Tom,

On Thursday, 21 September 2017 09:27:02 PDT Tom Rini wrote:
> On Thu, Sep 14, 2017 at 02:34:46PM -0700, Paul Burton wrote:
> > In python 3.x the StringIO module is gone, and instead StringIO can be
> > imported from the io module. Do this in order to run on python 3.x, and
> > fall back to importing StringIO as a module in order to continue working
> > with python 2.x.
> > 
> > Signed-off-by: Paul Burton <paul.burton@imgtec.com>
> > Reviewed-by: Stephen Warren <swarren@nvidia.com>
> 
> This breaks python2.7.  There we have both StringIO and io.StringIO so
> we succeed in 'from io import StringIO'.  But io.StringIO() needs
> unicode, and we don't have that by default in python2.  My quick attempt
> here failed.  Thanks!

Hmm, in that case back to v1 for this patch? (ie. try the StringIO.StringIO 
import first?)

  https://patchwork.ozlabs.org/patch/813933/

Would you like me to submit a v3 or could you just take this one from v1?

Thanks,
    Paul
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: This is a digitally signed message part.
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20170921/ae89f6c5/attachment.sig>

[U-Boot] [U-Boot, v2, 4/8] test/py: Import StringIO from io module for python 3.x

[Stephen Warren]

On 09/21/2017 10:38 AM, Paul Burton wrote:
> Hi Tom,
> 
> On Thursday, 21 September 2017 09:27:02 PDT Tom Rini wrote:
>> On Thu, Sep 14, 2017 at 02:34:46PM -0700, Paul Burton wrote:
>>> In python 3.x the StringIO module is gone, and instead StringIO can be
>>> imported from the io module. Do this in order to run on python 3.x, and
>>> fall back to importing StringIO as a module in order to continue working
>>> with python 2.x.
>>>
>>> Signed-off-by: Paul Burton <paul.burton@imgtec.com>
>>> Reviewed-by: Stephen Warren <swarren@nvidia.com>
>>
>> This breaks python2.7.  There we have both StringIO and io.StringIO so
>> we succeed in 'from io import StringIO'.  But io.StringIO() needs
>> unicode, and we don't have that by default in python2.  My quick attempt
>> here failed.  Thanks!
> 
> Hmm, in that case back to v1 for this patch? (ie. try the StringIO.StringIO
> import first?)
> 
>    https://patchwork.ozlabs.org/patch/813933/

If v1 works, that's fine by me too.

[U-Boot] [U-Boot, v2, 4/8] test/py: Import StringIO from io module for python 3.x

[Tom Rini]

On Thu, Sep 21, 2017 at 12:37:19PM -0600, Stephen Warren wrote:
> On 09/21/2017 10:38 AM, Paul Burton wrote:
> >Hi Tom,
> >
> >On Thursday, 21 September 2017 09:27:02 PDT Tom Rini wrote:
> >>On Thu, Sep 14, 2017 at 02:34:46PM -0700, Paul Burton wrote:
> >>>In python 3.x the StringIO module is gone, and instead StringIO can be
> >>>imported from the io module. Do this in order to run on python 3.x, and
> >>>fall back to importing StringIO as a module in order to continue working
> >>>with python 2.x.
> >>>
> >>>Signed-off-by: Paul Burton <paul.burton@imgtec.com>
> >>>Reviewed-by: Stephen Warren <swarren@nvidia.com>
> >>
> >>This breaks python2.7.  There we have both StringIO and io.StringIO so
> >>we succeed in 'from io import StringIO'.  But io.StringIO() needs
> >>unicode, and we don't have that by default in python2.  My quick attempt
> >>here failed.  Thanks!
> >
> >Hmm, in that case back to v1 for this patch? (ie. try the StringIO.StringIO
> >import first?)
> >
> >   https://patchwork.ozlabs.org/patch/813933/
> 
> If v1 works, that's fine by me too.

OK, thanks.  It does work and I'll grab just that.

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20170921/c44f79b7/attachment.sig>

[U-Boot] [U-Boot, v2, 5/8] test/py: Encode/decode strings for stdio

[Tom Rini]

On Thu, Sep 14, 2017 at 02:34:47PM -0700, Paul Burton wrote:

> When reading stdin python 3.x will give us byte arrays, and when writing
> stdout or stderr it will expect byte arrays. In order to insulate the
> rest of the code from this difference, call encode or decode at
> appropriate points when reading or writing stdio files. This works fine
> on python 2.x too.
> 
> Signed-off-by: Paul Burton <paul.burton@imgtec.com>
> Reviewed-by: Stephen Warren <swarren@nvidia.com>

OK, this almost works.  When I add in the following only the vboot test
fails and I can't understand why:

diff --git a/test/py/multiplexed_log.py b/test/py/multiplexed_log.py
index 21bdcb730917..d23bec0b9668 100644
--- a/test/py/multiplexed_log.py
+++ b/test/py/multiplexed_log.py
@@ -326,7 +326,7 @@ $(document).ready(function () {
 
         data = data.replace(chr(13), '')
         data = ''.join((c in self._nonprint) and ('%%%02x' % ord(c)) or
-                       c for c in data)
+                       c for c in data.encode('utf-8'))
         data = cgi.escape(data)
         return data
 
diff --git a/test/py/u_boot_spawn.py b/test/py/u_boot_spawn.py
index 751302a529e8..0490230c83dc 100644
--- a/test/py/u_boot_spawn.py
+++ b/test/py/u_boot_spawn.py
@@ -136,7 +136,7 @@ class Spawn(object):
         """
 
         for pi in range(len(patterns)):
-            if type(patterns[pi]) == type(''):
+            if type(patterns[pi]) != type(re.compile('')):
                 patterns[pi] = re.compile(patterns[pi])
 
         tstart_s = time.time()

Please run 'make tests' on a host with python==python2, thanks!

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20170921/55ca55dd/attachment.sig>

[U-Boot] [PATCH v2 0/8] test/py: Fixes for python 3.x

[Simon Glass]

Hi Paul,

On 14 September 2017 at 15:34, Paul Burton <paul.burton@imgtec.com> wrote:
> This series allows test/py to run on python 3.x in addition to the
> already supported python 2.x. With it applied I currently see only one
> failure (test_ut[ut_dm_usb_flash]) which doesn't appear to be python
> related when running with:
>
> $ ./test/py/test.py --bd sandbox --build
>
> Thanks,
>     Paul
>
> Paul Burton (8):
>   test/py: Make print statements python 3.x safe
>   test/py: Use range() rather than xrange()
>   test/py: Import 'configparser' lower case to be python 3.x safe
>   test/py: Import StringIO from io module for python 3.x
>   test/py: Encode/decode strings for stdio
>   test/py: fit: Open files as binary files
>   test/py: hush_if_test: Use open() in place of file()
>   test/py: vboot: Remove stderr redirect from openssl command
>
>  test/py/conftest.py                | 18 +++++++++++++-----
>  test/py/multiplexed_log.py         |  4 ++--
>  test/py/test.py                    |  6 ++++--
>  test/py/tests/test_fit.py          | 14 ++++++++------
>  test/py/tests/test_hush_if_test.py |  2 +-
>  test/py/tests/test_ut.py           |  2 +-
>  test/py/tests/test_vboot.py        |  4 ++--
>  test/py/u_boot_console_sandbox.py  |  4 ++--
>  test/py/u_boot_spawn.py            | 12 ++++++------
>  test/py/u_boot_utils.py            |  4 ++--
>  10 files changed, 41 insertions(+), 29 deletions(-)
>
> --
> 2.14.1
>

I never did see a v3 of this but I would like to get these patches applied.

I'll apply the ones that seem cause no problems. If you have time,
please respin these ones:

>   test/py: Import StringIO from io module for python 3.x
>   test/py: Encode/decode strings for stdio

Regards,
Simon

[U-Boot] [PATCH v2 1/8] test/py: Make print statements python 3.x safe

[Simon Glass]

On 14 September 2017 at 15:34, Paul Burton <paul.burton@imgtec.com> wrote:
> In python 3.x print must be called as a function rather than used as a
> statement. Update uses of print to the function call syntax in order to
> be python 3.x safe.
>
> Signed-off-by: Paul Burton <paul.burton@imgtec.com>
> Reviewed-by: Stephen Warren <swarren@nvidia.com>
> ---
>
> Changes in v2: None
>
>  test/py/conftest.py               |  2 +-
>  test/py/test.py                   |  6 ++++--
>  test/py/tests/test_fit.py         | 12 +++++++-----
>  test/py/u_boot_console_sandbox.py |  2 +-
>  test/py/u_boot_spawn.py           |  2 +-
>  5 files changed, 14 insertions(+), 10 deletions(-)

Applied to u-boot-dm, thanks!

[U-Boot] [PATCH v2 2/8] test/py: Use range() rather than xrange()

[Simon Glass]

On 14 September 2017 at 15:34, Paul Burton <paul.burton@imgtec.com> wrote:
> In python 3.x the xrange() function has been removed, and range()
> returns an iterator much like Python 2.x's xrange(). Simply use range()
> in place of xrange() in order to work on both python 2.x & 3.x. This
> will mean a small cost on python 2.x since range() will return a list
> there rather than an iterator, but the cost should be negligible.
>
> Signed-off-by: Paul Burton <paul.burton@imgtec.com>
> Reviewed-by: Stephen Warren <swarren@nvidia.com>
> ---
>
> Changes in v2: None
>
>  test/py/u_boot_console_sandbox.py | 2 +-
>  test/py/u_boot_spawn.py           | 6 +++---
>  test/py/u_boot_utils.py           | 4 ++--
>  3 files changed, 6 insertions(+), 6 deletions(-)

Applied to u-boot-dm, thanks!

[U-Boot] [PATCH v2 3/8] test/py: Import 'configparser' lower case to be python 3.x safe

[Simon Glass]

On 14 September 2017 at 15:34, Paul Burton <paul.burton@imgtec.com> wrote:
> In python 3.x the configparser module is named with all lower case.
> Import it as such in order to avoid errors when running on python 3.x,
> and fall back to the CamelCase version in order to keep working with
> python 2.x.
>
> Signed-off-by: Paul Burton <paul.burton@imgtec.com>
> Reviewed-by: Stephen Warren <swarren@nvidia.com>
>
> ---
>
> Changes in v2:
> - Import python 2.x ConfigParser as the python 3.x style configparser, rather than the other way around.
>
>  test/py/conftest.py | 8 ++++++--
>  1 file changed, 6 insertions(+), 2 deletions(-)

Applied to u-boot-dm, thanks!

[U-Boot] [PATCH v2 6/8] test/py: fit: Open files as binary files

[Simon Glass]

On 14 September 2017 at 15:34, Paul Burton <paul.burton@imgtec.com> wrote:
> The read_file() function in test_fit is used with files that are not
> text files, as well as some that are. It is never used in a way that
> requires it to decode text files to characters, so open all files in
> binary mode such that read() doesn't attempt to decode characters for
> files which are not text files.
>
> Without this test_fit fails on python 3.x when reading an FDT in
> run_fit_test() with:
>
>   UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd0 in position
>                        0: invalid continuation byte
>
> Signed-off-by: Paul Burton <paul.burton@imgtec.com>
> Reviewed-by: Stephen Warren <swarren@nvidia.com>
> ---
>
> Changes in v2: None
>
>  test/py/tests/test_fit.py | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)

Applied to u-boot-dm, thanks!

[U-Boot] [PATCH v2 7/8] test/py: hush_if_test: Use open() in place of file()

[Simon Glass]

On 14 September 2017 at 15:34, Paul Burton <paul.burton@imgtec.com> wrote:
> In python 3.x the file() function has been removed. Use open() instead,
> which works on both python 2.x & 3.x, and is described as the preferred
> method of opening a file by python 2.x documentation anyway.
>
> Signed-off-by: Paul Burton <paul.burton@imgtec.com>
> Reviewed-by: Stephen Warren <swarren@nvidia.com>
> ---
>
> Changes in v2: None
>
>  test/py/tests/test_hush_if_test.py | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)

Applied to u-boot-dm, thanks!

[U-Boot] [PATCH v2 8/8] test/py: vboot: Remove stderr redirect from openssl command

[Simon Glass]

On 14 September 2017 at 15:34, Paul Burton <paul.burton@imgtec.com> wrote:
> The openssl command specified in test_with_algo() ultimately ends up
> being run by RunAndLog::run(), which uses it to construct a Popen object
> with the default shell=False. The stderr redirect in the command is
> therefore simply passed to openssl as an argument. With at least openssl
> 1.1.0f this causes openssl, and therefore test_vboot, to fail with:
>
>   genpkey: Use -help for summary.
>   Exit code: 1
>
> Any stderr output ought to be captured & stored in the RunAndLog
> object's output field and returned from run() via run_and_log() to
> test_with_algo() which then ignores it anyway, so we can drop the
> shell-like redirection with no ill effects. With this fix test_vboot now
> passes for me.
>
> Signed-off-by: Paul Burton <paul.burton@imgtec.com>
> Reviewed-by: Stephen Warren <swarren@nvidia.com>
>
> ---
>
> Changes in v2:
> - Place the % operator at the end of the last string line, not the start of the next.
>
>  test/py/tests/test_vboot.py | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
>

Applied to u-boot-dm, thanks!

[U-Boot] [PATCH v2 3/8] test/py: Import 'configparser' lower case to be python 3.x safe

[Michal Simek]

On 14.9.2017 23:34, Paul Burton wrote:
> In python 3.x the configparser module is named with all lower case.
> Import it as such in order to avoid errors when running on python 3.x,
> and fall back to the CamelCase version in order to keep working with
> python 2.x.
> 
> Signed-off-by: Paul Burton <paul.burton@imgtec.com>
> Reviewed-by: Stephen Warren <swarren@nvidia.com>
> 
> ---
> 
> Changes in v2:
> - Import python 2.x ConfigParser as the python 3.x style configparser, rather than the other way around.
> 
>  test/py/conftest.py | 8 ++++++--
>  1 file changed, 6 insertions(+), 2 deletions(-)
> 
> diff --git a/test/py/conftest.py b/test/py/conftest.py
> index ced96f1006..bf55bd3d13 100644
> --- a/test/py/conftest.py
> +++ b/test/py/conftest.py
> @@ -19,11 +19,15 @@ import os
>  import os.path
>  import pytest
>  from _pytest.runner import runtestprotocol
> -import ConfigParser
>  import re
>  import StringIO
>  import sys
>  
> +try:
> +    import configparser
> +except:
> +    import ConfigParser as configparser
> +
>  # Globals: The HTML log file, and the connection to the U-Boot console.
>  log = None
>  console = None
> @@ -167,7 +171,7 @@ def pytest_configure(config):
>          with open(dot_config, 'rt') as f:
>              ini_str = '[root]\n' + f.read()
>              ini_sio = StringIO.StringIO(ini_str)
> -            parser = ConfigParser.RawConfigParser()
> +            parser = configparser.RawConfigParser()
>              parser.readfp(ini_sio)
>              ubconfig.buildconfig.update(parser.items('root'))
>  
> 

I have bisect issue with test/py framework on my PC and I end up on this
patch.

Below is the issue I am getting.

And here are my python version.
[u-boot](test)$ python3 --version
Python 3.4.3
[u-boot](test)$ python --version
Python 2.7.6
[u-boot](test)$ which python
/usr/bin/python


Do you know what could be the issue?

Thanks,
Michal


=====================================================================================================
FAILURES
======================================================================================================
_____________________________________________________________________________________________________
test_help
_____________________________________________________________________________________________________

u_boot_console = <u_boot_console_exec_attach.ConsoleExecAttach object at
0x7f1da3e6f290>

    def test_help(u_boot_console):
        """Test that the "help" command can be executed."""

>       u_boot_console.run_command('help')

test/py/tests/test_help.py:8:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
test/py/u_boot_console_base.py:181: in run_command
    self.logstream.write(self.prompt, implicit=True)
test/py/multiplexed_log.py:65: in write
    self.logfile.write(self, data, implicit)
test/py/multiplexed_log.py:688: in write
    self.f.write(self._escape(data))
test/py/multiplexed_log.py:335: in _escape
    c for c in data)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

.0 = <iterator object@0x7f1da3d9bb90>

    data = ''.join((c in self._nonprint) and ('%%%02x' % ord(c)) or
>                  c for c in data)
E   UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position
32: ordinal not in range(128)

test/py/multiplexed_log.py:335: UnicodeDecodeError
===============================================================================================
130 tests deselected
================================================================================================
=====================================================================================
1 failed, 130 deselected in 7.32 seconds
======================================================================================

[U-Boot] [PATCH v2 3/8] test/py: Import 'configparser' lower case to be python 3.x safe

[Stephen Warren]

On 08/06/2018 04:42 AM, Michal Simek wrote:
> On 14.9.2017 23:34, Paul Burton wrote:
>> In python 3.x the configparser module is named with all lower case.
>> Import it as such in order to avoid errors when running on python 3.x,
>> and fall back to the CamelCase version in order to keep working with
>> python 2.x.
...
> I have bisect issue with test/py framework on my PC and I end up on this
> patch.
> 
> Below is the issue I am getting.
> 
> And here are my python version.
> [u-boot](test)$ python3 --version
> Python 3.4.3
> [u-boot](test)$ python --version
> Python 2.7.6
> [u-boot](test)$ which python
> /usr/bin/python
> 
> 
> Do you know what could be the issue?

I honestly can't think why the configuration parsing could cause the 
kind of issue that was shown in the test log. It seems like there's a 
bug in the test/py code w.r.t. receiving characters from the target 
outside the ASCII range, which should be applicable in all cases, but I 
can't see how that would only happen based on how configparser was imported.

> 
> Thanks,
> Michal
> 
> 
> =====================================================================================================
> FAILURES
> ======================================================================================================
> _____________________________________________________________________________________________________
> test_help
> _____________________________________________________________________________________________________
> 
> u_boot_console = <u_boot_console_exec_attach.ConsoleExecAttach object at
> 0x7f1da3e6f290>
> 
>      def test_help(u_boot_console):
>          """Test that the "help" command can be executed."""
> 
>>        u_boot_console.run_command('help')
> 
> test/py/tests/test_help.py:8:
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> test/py/u_boot_console_base.py:181: in run_command
>      self.logstream.write(self.prompt, implicit=True)
> test/py/multiplexed_log.py:65: in write
>      self.logfile.write(self, data, implicit)
> test/py/multiplexed_log.py:688: in write
>      self.f.write(self._escape(data))
> test/py/multiplexed_log.py:335: in _escape
>      c for c in data)
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> 
> .0 = <iterator object@0x7f1da3d9bb90>
> 
>      data = ''.join((c in self._nonprint) and ('%%%02x' % ord(c)) or
>>                   c for c in data)
> E   UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position
> 32: ordinal not in range(128)
> 
> test/py/multiplexed_log.py:335: UnicodeDecodeError
> ===============================================================================================
> 130 tests deselected
> ================================================================================================
> =====================================================================================
> 1 failed, 130 deselected in 7.32 seconds
> ======================================================================================
> 
> 
> _______________________________________________
> U-Boot mailing list
> U-Boot at lists.denx.de
> https://lists.denx.de/listinfo/u-boot
> 

[U-Boot] [PATCH v2 3/8] test/py: Import 'configparser' lower case to be python 3.x safe

[Michal Simek]

On 20.8.2018 23:19, Stephen Warren wrote:
> On 08/06/2018 04:42 AM, Michal Simek wrote:
>> On 14.9.2017 23:34, Paul Burton wrote:
>>> In python 3.x the configparser module is named with all lower case.
>>> Import it as such in order to avoid errors when running on python 3.x,
>>> and fall back to the CamelCase version in order to keep working with
>>> python 2.x.
> ...
>> I have bisect issue with test/py framework on my PC and I end up on this
>> patch.
>>
>> Below is the issue I am getting.
>>
>> And here are my python version.
>> [u-boot](test)$ python3 --version
>> Python 3.4.3
>> [u-boot](test)$ python --version
>> Python 2.7.6
>> [u-boot](test)$ which python
>> /usr/bin/python
>>
>>
>> Do you know what could be the issue?
> 
> I honestly can't think why the configuration parsing could cause the
> kind of issue that was shown in the test log. It seems like there's a
> bug in the test/py code w.r.t. receiving characters from the target
> outside the ASCII range, which should be applicable in all cases, but I
> can't see how that would only happen based on how configparser was
> imported.

hm. ok.

Thanks,
Michal