[U-Boot] [PATCH 00/10] Moving test/py to Python 3

[U-Boot] [PATCH 00/10] Moving test/py to Python 3

[Tom Rini]

Hey all,

So, here's the first pass at moving our test/py code over to Python 3.
It's not quite ready to merge as noted by the WORKAROUND / HACK patches
at the end.  The first two patches can go in any time and I will
probably pull them soon as well.  The next patch is an old one from
Marek that I deferred at the time, but it's time now so I've updated it
slightly to cover the new cases of the same code he fixed before.  For
migrating the pytest code itself, I've left the conversion split in a
few commits.  This isn't bisectable but certainly for review it's
better.  I'm honestly not sure if it's more important for history to
keep this bisectable or to have the changes broken out like this.

Finally, we have a workaround so that we can do this conversion before
we convert buildman/patman/buildman to python3 (as that will take more
work, it seems that we've followed an anti-pattern and python3 makes
that much more of a pain to live with), a work-around to skip the FS
tests just like we are today and then finally disabling EFI tests.  I've
already told Heinrich on IRC that this will take some work to fix so
it's on his list.

Now that I've posted this, aside from any feedback I'm going to look at
the fs tests and see what's going on there as I'm not sure if we have
regressions in the fs code or lack of host tools being installed or
possibly both.

-- 
Tom

[U-Boot] [PATCH 01/10] gitlab-ci: Fix indentation in some stanzas

[Tom Rini]

In a number of our stanzas we had multi-line commands that were one
space short of alignment, correct this.

Signed-off-by: Tom Rini <trini@konsulko.com>
---
 .gitlab-ci.yml | 40 ++++++++++++++++++++--------------------
 1 file changed, 20 insertions(+), 20 deletions(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 967abed9f250..b91b5f67af98 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -65,11 +65,11 @@ build all 32bit ARM platforms:
   stage: world build
   script:
     - ret=0;
-     ./tools/buildman/buildman -o /tmp -P -E arm -x aarch64 || ret=$?;
-     if [[ $ret -ne 0 && $ret -ne 129 ]]; then
-       ./tools/buildman/buildman -o /tmp -sdeP;
-       exit $ret;
-     fi;
+      ./tools/buildman/buildman -o /tmp -P -E arm -x aarch64 || ret=$?;
+      if [[ $ret -ne 0 && $ret -ne 129 ]]; then
+        ./tools/buildman/buildman -o /tmp -sdeP;
+        exit $ret;
+      fi;
 
 build all 64bit ARM platforms:
   tags: [ 'all' ]
@@ -79,33 +79,33 @@ build all 64bit ARM platforms:
     - . /tmp/venv/bin/activate
     - pip install pyelftools
     - ret=0;
-     ./tools/buildman/buildman -o /tmp -P -E aarch64 || ret=$?;
-     if [[ $ret -ne 0 && $ret -ne 129 ]]; then
-       ./tools/buildman/buildman -o /tmp -sdeP;
-       exit $ret;
-     fi;
+      ./tools/buildman/buildman -o /tmp -P -E aarch64 || ret=$?;
+      if [[ $ret -ne 0 && $ret -ne 129 ]]; then
+        ./tools/buildman/buildman -o /tmp -sdeP;
+        exit $ret;
+      fi;
 
 build all PowerPC platforms:
   tags: [ 'all' ]
   stage: world build
   script:
     - ret=0;
-     ./tools/buildman/buildman -o /tmp -P -E powerpc || ret=$?;
-     if [[ $ret -ne 0 && $ret -ne 129 ]]; then
-       ./tools/buildman/buildman -o /tmp -sdeP;
-       exit $ret;
-     fi;
+      ./tools/buildman/buildman -o /tmp -P -E powerpc || ret=$?;
+      if [[ $ret -ne 0 && $ret -ne 129 ]]; then
+        ./tools/buildman/buildman -o /tmp -sdeP;
+        exit $ret;
+      fi;
 
 build all other platforms:
   tags: [ 'all' ]
   stage: world build
   script:
     - ret=0;
-     ./tools/buildman/buildman -o /tmp -P -E -x arm,powerpc || ret=$?;
-     if [[ $ret -ne 0 && $ret -ne 129 ]]; then
-       ./tools/buildman/buildman -o /tmp -sdeP;
-       exit $ret;
-     fi;
+      ./tools/buildman/buildman -o /tmp -P -E -x arm,powerpc || ret=$?;
+      if [[ $ret -ne 0 && $ret -ne 129 ]]; then
+        ./tools/buildman/buildman -o /tmp -sdeP;
+        exit $ret;
+      fi;
 
 # QA jobs for code analytics
 # static code analysis with cppcheck (we can add --enable=all later)
-- 
2.17.1

[U-Boot] [PATCH 02/10] gitlab-ci: Prepend to PATH rather than overwrite it

[Tom Rini]

We need to make sure that we prepend changes to PATH, so that any
binaries we need to provide ourselves are used not overwrite PATH for
this.  Overwriting PATH like this breaks the python virtualenv PATH
changes.

Signed-off-by: Tom Rini <trini@konsulko.com>
---
 .gitlab-ci.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index b91b5f67af98..ab294997e463 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -48,7 +48,7 @@ stages:
     # "-k something" even when $TEST_PY_TEST_SPEC doesnt need a custom
     # value.
     - export UBOOT_TRAVIS_BUILD_DIR=/tmp/.bm-work/${TEST_PY_BD};
-      export PATH=/opt/qemu/bin:/tmp/uboot-test-hooks/bin:/usr/bin:/bin;
+      export PATH=/opt/qemu/bin:/tmp/uboot-test-hooks/bin:${PATH};
       export PYTHONPATH=/tmp/uboot-test-hooks/py/travis-ci;
       if [[ "${TEST_PY_BD}" != "" ]]; then
         ./test/py/test.py --bd ${TEST_PY_BD} ${TEST_PY_ID}
-- 
2.17.1

[U-Boot] [PATCH 03/10] test/py: Fix pytest4 deprecation warnings

[Tom Rini]

From: Marek Vasut <marek.vasut@gmail.com>

Fix the following spit from pytest:

u-boot/test/py/conftest.py:438: RemovedInPytest4Warning: MarkInfo objects are deprecated as they contain merged marks which are hard to deal with correctly.
  Please use node.get_closest_marker(name) or node.iter_markers(name).
  Docs: https://docs.pytest.org/en/latest/mark.html#updating-code
    for board in mark.args:

In both cases, the later suggestion is applicable.

Signed-off-by: Marek Vasut <marek.vasut+renesas@gmail.com>
Cc: Igor Opaniuk <igor.opaniuk@linaro.org>
Cc: Tom Rini <trini@konsulko.com>
Cc: Simon Glass <sjg@chromium.org>
[trini: Update for current file with a few more cases, un-pin pytest in CI]
Signed-off-by: Tom Rini <trini@konsulko.com>
---
 .gitlab-ci.yml      |  2 +-
 .travis.yml         |  2 +-
 test/py/conftest.py | 30 ++++++++++++------------------
 3 files changed, 14 insertions(+), 20 deletions(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index ab294997e463..5a34321570ce 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -20,7 +20,7 @@ stages:
     - ln -s travis-ci /tmp/uboot-test-hooks/py/`hostname`
     - virtualenv /tmp/venv
     - . /tmp/venv/bin/activate
-    - pip install pytest==2.8.7
+    - pip install pytest
     - pip install python-subunit
     - pip install coverage
     - grub-mkimage --prefix="" -o ~/grub_x86.efi -O i386-efi normal  echo lsefimmap lsefi lsefisystab efinet tftp minicmd
diff --git a/.travis.yml b/.travis.yml
index a3e7451bcb17..f61278f9656a 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -49,7 +49,7 @@ install:
  - cat ~/.buildman
  - virtualenv /tmp/venv
  - . /tmp/venv/bin/activate
- - pip install pytest==2.8.7
+ - pip install pytest
  - pip install python-subunit
  - pip install pyelftools
  - grub-mkimage --prefix="" -o ~/grub_x86.efi -O i386-efi normal  echo lsefimmap lsefi lsefisystab efinet tftp minicmd
diff --git a/test/py/conftest.py b/test/py/conftest.py
index 00d8ef8ba99c..30c898b40a0d 100644
--- a/test/py/conftest.py
+++ b/test/py/conftest.py
@@ -431,11 +431,9 @@ def setup_boardspec(item):
         Nothing.
     """
 
-    mark = item.get_marker('boardspec')
-    if not mark:
-        return
     required_boards = []
-    for board in mark.args:
+    for boards in item.iter_markers('boardspec'):
+        board = boards.args[0]
         if board.startswith('!'):
             if ubconfig.board_type == board[1:]:
                 pytest.skip('board "%s" not supported' % ubconfig.board_type)
@@ -459,16 +457,14 @@ def setup_buildconfigspec(item):
         Nothing.
     """
 
-    mark = item.get_marker('buildconfigspec')
-    if mark:
-        for option in mark.args:
-            if not ubconfig.buildconfig.get('config_' + option.lower(), None):
-                pytest.skip('.config feature "%s" not enabled' % option.lower())
-    notmark = item.get_marker('notbuildconfigspec')
-    if notmark:
-        for option in notmark.args:
-            if ubconfig.buildconfig.get('config_' + option.lower(), None):
-                pytest.skip('.config feature "%s" enabled' % option.lower())
+    for options in item.iter_markers('buildconfigspec'):
+        option = options.args[0]
+        if not ubconfig.buildconfig.get('config_' + option.lower(), None):
+            pytest.skip('.config feature "%s" not enabled' % option.lower())
+    for option in item.iter_markers('notbuildconfigspec'):
+        option = options.args[0]
+        if ubconfig.buildconfig.get('config_' + option.lower(), None):
+            pytest.skip('.config feature "%s" enabled' % option.lower())
 
 def tool_is_in_path(tool):
     for path in os.environ["PATH"].split(os.pathsep):
@@ -491,10 +487,8 @@ def setup_requiredtool(item):
         Nothing.
     """
 
-    mark = item.get_marker('requiredtool')
-    if not mark:
-        return
-    for tool in mark.args:
+    for tools in item.iter_markers('requiredtool'):
+        tool = tools.args[0]
         if not tool_is_in_path(tool):
             pytest.skip('tool "%s" not in $PATH' % tool)
 
-- 
2.17.1

[U-Boot] [PATCH 04/10] test/py: Automated conversion to Python 3

[Tom Rini]

Use the 2to3 tool to perform numerous automatic conversions from Python
2 syntax to Python 3.  Also fix whitespace problems that Python 3
catches that Python 2 did not.

Signed-off-by: Tom Rini <trini@konsulko.com>
---
 test/py/conftest.py               |  9 ++--
 test/py/multiplexed_log.py        |  4 +-
 test/py/test.py                   |  4 +-
 test/py/tests/test_fit.py         |  2 -
 test/py/tests/test_fpga.py        | 46 +++++++++----------
 test/py/tests/test_fs/conftest.py |  4 +-
 test/py/tests/test_log.py         |  8 ++--
 test/py/tests/test_mmc_wr.py      | 74 +++++++++++++++----------------
 8 files changed, 71 insertions(+), 80 deletions(-)

diff --git a/test/py/conftest.py b/test/py/conftest.py
index 30c898b40a0d..5c19af1d5034 100644
--- a/test/py/conftest.py
+++ b/test/py/conftest.py
@@ -19,13 +19,10 @@ import os.path
 import pytest
 from _pytest.runner import runtestprotocol
 import re
-import StringIO
+import io
 import sys
 
-try:
-    import configparser
-except:
-    import ConfigParser as configparser
+import configparser
 
 # Globals: The HTML log file, and the connection to the U-Boot console.
 log = None
@@ -169,7 +166,7 @@ def pytest_configure(config):
 
         with open(dot_config, 'rt') as f:
             ini_str = '[root]\n' + f.read()
-            ini_sio = StringIO.StringIO(ini_str)
+            ini_sio = io.StringIO(ini_str)
             parser = configparser.RawConfigParser()
             parser.readfp(ini_sio)
             ubconfig.buildconfig.update(parser.items('root'))
diff --git a/test/py/multiplexed_log.py b/test/py/multiplexed_log.py
index 637a3bd257ba..de0aacc659b8 100644
--- a/test/py/multiplexed_log.py
+++ b/test/py/multiplexed_log.py
@@ -5,8 +5,8 @@
 # Generate an HTML-formatted log file containing multiple streams of data,
 # each represented in a well-delineated/-structured fashion.
 
-import cgi
 import datetime
+import html
 import os.path
 import shutil
 import subprocess
@@ -334,7 +334,7 @@ $(document).ready(function () {
         data = data.replace(chr(13), '')
         data = ''.join((ord(c) in self._nonprint) and ('%%%02x' % ord(c)) or
                        c for c in data)
-        data = cgi.escape(data)
+        data = html.escape(data)
         return data
 
     def _terminate_stream(self):
diff --git a/test/py/test.py b/test/py/test.py
index a5140945d4b9..0ce1838833f6 100755
--- a/test/py/test.py
+++ b/test/py/test.py
@@ -1,4 +1,4 @@
-#!/usr/bin/env python2
+#!/usr/bin/env python3
 # SPDX-License-Identifier: GPL-2.0
 
 # Copyright (c) 2015 Stephen Warren
@@ -7,8 +7,6 @@
 # Wrapper script to invoke pytest with the directory name that contains the
 # U-Boot tests.
 
-from __future__ import print_function
-
 import os
 import os.path
 import sys
diff --git a/test/py/tests/test_fit.py b/test/py/tests/test_fit.py
index e3210ed43fa4..4922b9dcc664 100755
--- a/test/py/tests/test_fit.py
+++ b/test/py/tests/test_fit.py
@@ -3,8 +3,6 @@
 #
 # Sanity check of the FIT handling in U-Boot
 
-from __future__ import print_function
-
 import os
 import pytest
 import struct
diff --git a/test/py/tests/test_fpga.py b/test/py/tests/test_fpga.py
index e3bb7b41c749..ca7ef8ea40d6 100644
--- a/test/py/tests/test_fpga.py
+++ b/test/py/tests/test_fpga.py
@@ -175,29 +175,29 @@ def test_fpga_load_fail(u_boot_console):
     f, dev, addr, bit, bit_size = load_file_from_var(u_boot_console, 'bitstream_load')
 
     for cmd in ['dump', 'load', 'loadb']:
-	# missing dev parameter
-	expected = 'fpga: incorrect parameters passed'
-	output = u_boot_console.run_command('fpga %s %x $filesize' % (cmd, addr))
-	#assert expected in output
-	assert expected_usage in output
-
-	# more parameters - 0 at the end
-	expected = 'fpga: more parameters passed'
-	output = u_boot_console.run_command('fpga %s %x %x $filesize 0' % (cmd, dev, addr))
-	#assert expected in output
-	assert expected_usage in output
-
-	# 0 address
-	expected = 'fpga: zero fpga_data address'
-	output = u_boot_console.run_command('fpga %s %x 0 $filesize' % (cmd, dev))
-	#assert expected in output
-	assert expected_usage in output
-
-	# 0 filesize
-	expected = 'fpga: zero size'
-	output = u_boot_console.run_command('fpga %s %x %x 0' % (cmd, dev, addr))
-	#assert expected in output
-	assert expected_usage in output
+        # missing dev parameter
+        expected = 'fpga: incorrect parameters passed'
+        output = u_boot_console.run_command('fpga %s %x $filesize' % (cmd, addr))
+        #assert expected in output
+        assert expected_usage in output
+
+        # more parameters - 0 at the end
+        expected = 'fpga: more parameters passed'
+        output = u_boot_console.run_command('fpga %s %x %x $filesize 0' % (cmd, dev, addr))
+        #assert expected in output
+        assert expected_usage in output
+
+        # 0 address
+        expected = 'fpga: zero fpga_data address'
+        output = u_boot_console.run_command('fpga %s %x 0 $filesize' % (cmd, dev))
+        #assert expected in output
+        assert expected_usage in output
+
+        # 0 filesize
+        expected = 'fpga: zero size'
+        output = u_boot_console.run_command('fpga %s %x %x 0' % (cmd, dev, addr))
+        #assert expected in output
+        assert expected_usage in output
 
 @pytest.mark.buildconfigspec('cmd_fpga')
 @pytest.mark.buildconfigspec('cmd_echo')
diff --git a/test/py/tests/test_fs/conftest.py b/test/py/tests/test_fs/conftest.py
index 9324657d2160..354d17672fe4 100644
--- a/test/py/tests/test_fs/conftest.py
+++ b/test/py/tests/test_fs/conftest.py
@@ -508,8 +508,8 @@ def fs_obj_unlink(request, u_boot_config):
 
         # Test Case 2
         check_call('mkdir %s/dir2' % mount_dir, shell=True)
-	for i in range(0, 20):
-	    check_call('mkdir %s/dir2/0123456789abcdef%02x'
+        for i in range(0, 20):
+            check_call('mkdir %s/dir2/0123456789abcdef%02x'
                                     % (mount_dir, i), shell=True)
 
         # Test Case 4
diff --git a/test/py/tests/test_log.py b/test/py/tests/test_log.py
index cb183444c6fc..75325fad61ad 100644
--- a/test/py/tests/test_log.py
+++ b/test/py/tests/test_log.py
@@ -27,9 +27,9 @@ def test_log(u_boot_console):
         """
         for i in range(max_level):
             if mask & 1:
-                assert 'log_run() log %d' % i == lines.next()
+                assert 'log_run() log %d' % i == next(lines)
             if mask & 3:
-                assert 'func() _log %d' % i == lines.next()
+                assert 'func() _log %d' % i == next(lines)
 
     def run_test(testnum):
         """Run a particular test number (the 'log test' command)
@@ -43,7 +43,7 @@ def test_log(u_boot_console):
            output = u_boot_console.run_command('log test %d' % testnum)
         split = output.replace('\r', '').splitlines()
         lines = iter(split)
-        assert 'test %d' % testnum == lines.next()
+        assert 'test %d' % testnum == next(lines)
         return lines
 
     def test0():
@@ -88,7 +88,7 @@ def test_log(u_boot_console):
     def test10():
         lines = run_test(10)
         for i in range(7):
-            assert 'log_test() level %d' % i == lines.next()
+            assert 'log_test() level %d' % i == next(lines)
 
     # TODO(sjg at chromium.org): Consider structuring this as separate tests
     cons = u_boot_console
diff --git a/test/py/tests/test_mmc_wr.py b/test/py/tests/test_mmc_wr.py
index 8b18781eac74..2ce79f58e3b1 100644
--- a/test/py/tests/test_mmc_wr.py
+++ b/test/py/tests/test_mmc_wr.py
@@ -65,41 +65,39 @@ def test_mmc_wr(u_boot_console, env__mmc_wr_config):
 
 
     for i in range(test_iterations):
-	# Generate random data
-	cmd = 'random %s %x' % (src_addr, count_bytes)
-	response = u_boot_console.run_command(cmd)
-	good_response = '%d bytes filled with random data' % (count_bytes)
-	assert good_response in response
-
-	# Select MMC device
-	cmd = 'mmc dev %d' % devid
-	if is_emmc:
-		cmd += ' %d' % partid
-	response = u_boot_console.run_command(cmd)
-	assert 'no card present' not in response
-	if is_emmc:
-		partid_response = "(part %d)" % partid
-	else:
-		partid_response = ""
-	good_response = 'mmc%d%s is current device' % (devid, partid_response)
-	assert good_response in response
-
-	# Write data
-	cmd = 'mmc write %s %x %x' % (src_addr, sector, count_sectors)
-	response = u_boot_console.run_command(cmd)
-	good_response = 'MMC write: dev # %d, block # %d, count %d ... %d blocks written: OK' % (
-		devid, sector, count_sectors, count_sectors)
-	assert good_response in response
-
-	# Read data
-	cmd = 'mmc read %s %x %x' % (dst_addr, sector, count_sectors)
-	response = u_boot_console.run_command(cmd)
-	good_response = 'MMC read: dev # %d, block # %d, count %d ... %d blocks read: OK' % (
-		devid, sector, count_sectors, count_sectors)
-	assert good_response in response
-
-	# Compare src and dst data
-	cmd = 'cmp.b %s %s %x' % (src_addr, dst_addr, count_bytes)
-	response = u_boot_console.run_command(cmd)
-	good_response = 'Total of %d byte(s) were the same' % (count_bytes)
-	assert good_response in response
+        # Generate random data
+        cmd = 'random %s %x' % (src_addr, count_bytes)
+        response = u_boot_console.run_command(cmd)
+        good_response = '%d bytes filled with random data' % (count_bytes)
+        assert good_response in response
+
+        # Select MMC device
+        cmd = 'mmc dev %d' % devid
+        if is_emmc:
+            cmd += ' %d' % partid
+        response = u_boot_console.run_command(cmd)
+        assert 'no card present' not in response
+        if is_emmc:
+            partid_response = "(part %d)" % partid
+        else:
+            partid_response = ""
+        good_response = 'mmc%d%s is current device' % (devid, partid_response)
+        assert good_response in response
+
+        # Write data
+        cmd = 'mmc write %s %x %x' % (src_addr, sector, count_sectors)
+        response = u_boot_console.run_command(cmd)
+        good_response = 'MMC write: dev # %d, block # %d, count %d ... %d blocks written: OK' % (devid, sector, count_sectors, count_sectors)
+        assert good_response in response
+
+        # Read data
+        cmd = 'mmc read %s %x %x' % (dst_addr, sector, count_sectors)
+        response = u_boot_console.run_command(cmd)
+        good_response = 'MMC read: dev # %d, block # %d, count %d ... %d blocks read: OK' % (devid, sector, count_sectors, count_sectors)
+        assert good_response in response
+
+        # Compare src and dst data
+        cmd = 'cmp.b %s %s %x' % (src_addr, dst_addr, count_bytes)
+        response = u_boot_console.run_command(cmd)
+        good_response = 'Total of %d byte(s) were the same' % (count_bytes)
+        assert good_response in response
-- 
2.17.1

[U-Boot] [PATCH 05/10] test/py: Split mark to multiple lines

[Tom Rini]

We inconsistently note multiple dependencies today in our tests,
sometimes with a single line that declares multiple and sometimes
multiple single lines.  Current pytest seems to fail on the single line
format so change to multiple declarations.

Signed-off-by: Tom Rini <trini@konsulko.com>
---
 test/py/tests/test_android/test_avb.py | 9 ++++++---
 test/py/tests/test_mmc_wr.py           | 4 +++-
 2 files changed, 9 insertions(+), 4 deletions(-)

diff --git a/test/py/tests/test_android/test_avb.py b/test/py/tests/test_android/test_avb.py
index 813242343555..20ccaf6712f2 100644
--- a/test/py/tests/test_android/test_avb.py
+++ b/test/py/tests/test_android/test_avb.py
@@ -23,7 +23,8 @@ mmc_dev = 1
 temp_addr = 0x90000000
 temp_addr2 = 0x90002000
 
- at pytest.mark.buildconfigspec('cmd_avb', 'cmd_mmc')
+ at pytest.mark.buildconfigspec('cmd_avb')
+ at pytest.mark.buildconfigspec('cmd_mmc')
 def test_avb_verify(u_boot_console):
     """Run AVB 2.0 boot verification chain with avb subset of commands
     """
@@ -36,7 +37,8 @@ def test_avb_verify(u_boot_console):
     assert response.find(success_str)
 
 
- at pytest.mark.buildconfigspec('cmd_avb', 'cmd_mmc')
+ at pytest.mark.buildconfigspec('cmd_avb')
+ at pytest.mark.buildconfigspec('cmd_mmc')
 def test_avb_mmc_uuid(u_boot_console):
     """Check if 'avb get_uuid' works, compare results with
     'part list mmc 1' output
@@ -93,7 +95,8 @@ def test_avb_is_unlocked(u_boot_console):
     assert response == 'Unlocked = 1'
 
 
- at pytest.mark.buildconfigspec('cmd_avb', 'cmd_mmc')
+ at pytest.mark.buildconfigspec('cmd_avb')
+ at pytest.mark.buildconfigspec('cmd_mmc')
 def test_avb_mmc_read(u_boot_console):
     """Test mmc read operation
     """
diff --git a/test/py/tests/test_mmc_wr.py b/test/py/tests/test_mmc_wr.py
index 2ce79f58e3b1..05e5c1ee85da 100644
--- a/test/py/tests/test_mmc_wr.py
+++ b/test/py/tests/test_mmc_wr.py
@@ -35,7 +35,9 @@ env__mmc_wr_configs = (
 
 """
 
- at pytest.mark.buildconfigspec('cmd_mmc','cmd_memory', 'cmd_random')
+ at pytest.mark.buildconfigspec('cmd_mmc')
+ at pytest.mark.buildconfigspec('cmd_memory')
+ at pytest.mark.buildconfigspec('cmd_random')
 def test_mmc_wr(u_boot_console, env__mmc_wr_config):
     """Test the "mmc write" command.
 
-- 
2.17.1

[U-Boot] [PATCH 06/10] test/py: test_ut.py: Ensure we use bytes

[Tom Rini]

In the case of some unit tests we are working with providing a fake
flash device that we have written some text strings in to.  In this case
we want to tell Python to encode things to bytes for us.

Signed-off-by: Tom Rini <trini@konsulko.com>
---
 test/py/tests/test_ut.py | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/test/py/tests/test_ut.py b/test/py/tests/test_ut.py
index 62037d2c4530..6c7b8dd2b30e 100644
--- a/test/py/tests/test_ut.py
+++ b/test/py/tests/test_ut.py
@@ -10,14 +10,14 @@ def test_ut_dm_init(u_boot_console):
 
     fn = u_boot_console.config.source_dir + '/testflash.bin'
     if not os.path.exists(fn):
-        data = 'this is a test'
-        data += '\x00' * ((4 * 1024 * 1024) - len(data))
+        data = b'this is a test'
+        data += b'\x00' * ((4 * 1024 * 1024) - len(data))
         with open(fn, 'wb') as fh:
             fh.write(data)
 
     fn = u_boot_console.config.source_dir + '/spi.bin'
     if not os.path.exists(fn):
-        data = '\x00' * (2 * 1024 * 1024)
+        data = b'\x00' * (2 * 1024 * 1024)
         with open(fn, 'wb') as fh:
             fh.write(data)
 
-- 
2.17.1

[U-Boot] [PATCH 07/10] test/py: Manual python3 fixes

[Tom Rini]

- Modern pytest is more visible in telling us about parameters that we
  had not described, so describe a few more.
- ConfigParser.readfp(...) is now configparser.read_file(...)
- As part of the "strings vs bytes" conversions in Python 3, we need to
  encode / decode our pipes in a few areas, use utf-8 for this.
- Fix a typo noticed while doing the above ("tot he" -> "to the").

Signed-off-by: Tom Rini <trini@konsulko.com>
---
 test/py/conftest.py        | 2 +-
 test/py/multiplexed_log.py | 4 ++--
 test/py/pytest.ini         | 3 +++
 test/py/u_boot_spawn.py    | 4 ++--
 4 files changed, 8 insertions(+), 5 deletions(-)

diff --git a/test/py/conftest.py b/test/py/conftest.py
index 5c19af1d5034..02b61655dcee 100644
--- a/test/py/conftest.py
+++ b/test/py/conftest.py
@@ -168,7 +168,7 @@ def pytest_configure(config):
             ini_str = '[root]\n' + f.read()
             ini_sio = io.StringIO(ini_str)
             parser = configparser.RawConfigParser()
-            parser.readfp(ini_sio)
+            parser.read_file(ini_sio)
             ubconfig.buildconfig.update(parser.items('root'))
 
     ubconfig.test_py_dir = test_py_dir
diff --git a/test/py/multiplexed_log.py b/test/py/multiplexed_log.py
index de0aacc659b8..35d95fbe0fe8 100644
--- a/test/py/multiplexed_log.py
+++ b/test/py/multiplexed_log.py
@@ -51,7 +51,7 @@ class LogfileStream(object):
         """Write data to the log stream.
 
         Args:
-            data: The data to write tot he file.
+            data: The data to write to the file.
             implicit: Boolean indicating whether data actually appeared in the
                 stream, or was implicitly generated. A valid use-case is to
                 repeat a shell prompt at the start of each separate log
@@ -133,7 +133,7 @@ class RunAndLog(object):
         self.logfile.write(self, msg)
 
         try:
-            p = subprocess.Popen(cmd, cwd=cwd,
+            p = subprocess.Popen(cmd, cwd=cwd, encoding="utf-8",
                 stdin=None, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
             (stdout, stderr) = p.communicate()
             output = ''
diff --git a/test/py/pytest.ini b/test/py/pytest.ini
index 7e400682bf25..e93d010f1fa2 100644
--- a/test/py/pytest.ini
+++ b/test/py/pytest.ini
@@ -8,3 +8,6 @@
 markers =
     boardspec: U-Boot: Describes the set of boards a test can/can't run on.
     buildconfigspec: U-Boot: Describes Kconfig/config-header constraints.
+    notbuildconfigspec: U-Boot: Describes required disabled Kconfig options.
+    requiredtool: U-Boot: Required host tools for a test.
+    slow: U-Boot: Specific test will run slowly.
diff --git a/test/py/u_boot_spawn.py b/test/py/u_boot_spawn.py
index b011a3e3da25..63119229f420 100644
--- a/test/py/u_boot_spawn.py
+++ b/test/py/u_boot_spawn.py
@@ -113,7 +113,7 @@ class Spawn(object):
             Nothing.
         """
 
-        os.write(self.fd, data)
+        os.write(self.fd, data.encode("utf-8"))
 
     def expect(self, patterns):
         """Wait for the sub-process to emit specific data.
@@ -171,7 +171,7 @@ class Spawn(object):
                 events = self.poll.poll(poll_maxwait)
                 if not events:
                     raise Timeout()
-                c = os.read(self.fd, 1024)
+                c = os.read(self.fd, 1024).decode("utf-8")
                 if not c:
                     raise EOFError()
                 if self.logfile_read:
-- 
2.17.1

[U-Boot] [PATCH 08/10] WORKAROUND: gitlab-ci: Rework how and when we use virtualenv in order to have python3

[Tom Rini]

As things stand today, buildman uses python 2.  In order to have
virtualenv use python 3, so that we can use pip to install what we want,
we need to delay any use of virtualenv until after we have used
buildman.  Further, to aid in speeding up work here, for now install
python3-pip in our task here rather than in the Docker image.

Signed-off-by: Tom Rini <trini@konsulko.com>
---
 .gitlab-ci.yml | 9 ++++-----
 1 file changed, 4 insertions(+), 5 deletions(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 5a34321570ce..a46ab5487257 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -18,11 +18,6 @@ stages:
     - git clone --depth=1 git://github.com/swarren/uboot-test-hooks.git /tmp/uboot-test-hooks
     - ln -s travis-ci /tmp/uboot-test-hooks/bin/`hostname`
     - ln -s travis-ci /tmp/uboot-test-hooks/py/`hostname`
-    - virtualenv /tmp/venv
-    - . /tmp/venv/bin/activate
-    - pip install pytest
-    - pip install python-subunit
-    - pip install coverage
     - grub-mkimage --prefix="" -o ~/grub_x86.efi -O i386-efi normal  echo lsefimmap lsefi lsefisystab efinet tftp minicmd
     - grub-mkimage --prefix="" -o ~/grub_x64.efi -O x86_64-efi normal  echo lsefimmap lsefi lsefisystab efinet tftp minicmd
     - mkdir ~/grub2-arm
@@ -47,6 +42,10 @@ stages:
     # never prevent any test from running. That way, we can always pass
     # "-k something" even when $TEST_PY_TEST_SPEC doesnt need a custom
     # value.
+    - sudo apt-get update && sudo apt-get -y install python3-pip && sudo apt-get -y --purge remove python-pytest
+    - virtualenv -p /usr/bin/python3 /tmp/venv
+    - . /tmp/venv/bin/activate
+    - pip3 install pytest python-subunit coverage
     - export UBOOT_TRAVIS_BUILD_DIR=/tmp/.bm-work/${TEST_PY_BD};
       export PATH=/opt/qemu/bin:/tmp/uboot-test-hooks/bin:${PATH};
       export PYTHONPATH=/tmp/uboot-test-hooks/py/travis-ci;
-- 
2.17.1

[U-Boot] [PATCH 09/10] WORKAROUND: test/py: Skip fs tests for now

[Tom Rini]

Currently, these tests are skipped when run in CI.  With the move to
python3 and a newer host they are being run, but showing failures that
need more investigation.  For now, skip these tests as we had been
doing.

Signed-off-by: Tom Rini <trini@konsulko.com>
---
 .gitlab-ci.yml | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index a46ab5487257..2528a5a9a2e4 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -177,6 +177,7 @@ sandbox test.py:
   variables:
     TEST_PY_BD: "sandbox"
     BUILDMAN: "^sandbox$"
+    TEST_PY_TEST_SPEC: "not fs"
   <<: *buildman_and_testpy_dfn
 
 sandbox_spl test.py:
@@ -200,6 +201,7 @@ sandbox_flattree test.py:
   variables:
     TEST_PY_BD: "sandbox_flattree"
     BUILDMAN: "^sandbox_flattree$"
+    TEST_PY_TEST_SPEC: "not fs"
   <<: *buildman_and_testpy_dfn
 
 vexpress_ca15_tc2 test.py:
-- 
2.17.1

[U-Boot] [PATCH 10/10] HACK: test.py: Disable EFI tests for now

[Tom Rini]

When running these tests currently we see both warnings about how
escape sequences need to be updated and then we see failures of some
tests related to finding expected strings and so forth.  It seems quite
likely that these tests need work to function with Python 3 and it's
better unicode support.  Disable them for now only to make testing of
other Python 3 conversions easier.

Cc: Heinrich Schuchardt <xypron.glpk@gmx.de>
Signed-off-by: Tom Rini <trini@konsulko.com>
---
 .gitlab-ci.yml | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 2528a5a9a2e4..29d25e9de5c0 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -233,7 +233,7 @@ qemu_arm test.py:
   tags: [ 'all' ]
   variables:
     TEST_PY_BD: "qemu_arm"
-    TEST_PY_TEST_SPEC: "not sleep"
+    TEST_PY_TEST_SPEC: "not sleep and not efi"
     BUILDMAN: "^qemu_arm$"
   <<: *buildman_and_testpy_dfn
 
@@ -241,7 +241,7 @@ qemu_arm64 test.py:
   tags: [ 'all' ]
   variables:
     TEST_PY_BD: "qemu_arm64"
-    TEST_PY_TEST_SPEC: "not sleep"
+    TEST_PY_TEST_SPEC: "not sleep and not efi"
     BUILDMAN: "^qemu_arm64$"
   <<: *buildman_and_testpy_dfn
 
@@ -289,7 +289,7 @@ qemu-riscv64 test.py:
   tags: [ 'all' ]
   variables:
     TEST_PY_BD: "qemu-riscv64"
-    TEST_PY_TEST_SPEC: "not sleep"
+    TEST_PY_TEST_SPEC: "not sleep and not efi"
     BUILDMAN: "^qemu-riscv64$"
   <<: *buildman_and_testpy_dfn
 
@@ -297,7 +297,7 @@ qemu-x86 test.py:
   tags: [ 'all' ]
   variables:
     TEST_PY_BD: "qemu-x86"
-    TEST_PY_TEST_SPEC: "not sleep"
+    TEST_PY_TEST_SPEC: "not sleep and not efi"
     BUILDMAN: "^qemu-x86$"
   <<: *buildman_and_testpy_dfn
 
@@ -305,7 +305,7 @@ qemu-x86_64 test.py:
   tags: [ 'all' ]
   variables:
     TEST_PY_BD: "qemu-x86_64"
-    TEST_PY_TEST_SPEC: "not sleep"
+    TEST_PY_TEST_SPEC: "not sleep and not efi"
     BUILDMAN: "^qemu-x86_64$"
   <<: *buildman_and_testpy_dfn
 
-- 
2.17.1

[U-Boot] [PATCH 04/10] test/py: Automated conversion to Python 3

[Stephen Warren]

On 10/18/19 2:53 PM, Tom Rini wrote:
> Use the 2to3 tool to perform numerous automatic conversions from Python
> 2 syntax to Python 3.  Also fix whitespace problems that Python 3
> catches that Python 2 did not.

> diff --git a/test/py/conftest.py b/test/py/conftest.py

>   import pytest
>   from _pytest.runner import runtestprotocol
>   import re
> -import StringIO
> +import io
>   import sys

I think that list used to be sorted. IIRC some PEP8 tools complain about 
unsorted import lists.

> -try:
> -    import configparser
> -except:
> -    import ConfigParser as configparser
> +import configparser

Now that this is a plain import statement, let's put it with all the 
others at the appropriate point in the sorted list above.


(Not sure if you want to merge fixes like this into this patch, or put 
them in path 7?)


I didn't validate that the large chunks of code that had TAB->spaces 
conversions didn't change anything other than the indentation whitespace...

[U-Boot] [PATCH 09/10] WORKAROUND: test/py: Skip fs tests for now

[Stephen Warren]

On 10/18/19 2:53 PM, Tom Rini wrote:
> Currently, these tests are skipped when run in CI.

How? If they're being skipped by some other mechanism, do we need to 
remove that other mechanism now that we're actively specifying "not fs" 
when running under CI?

[U-Boot] [PATCH 10/10] HACK: test.py: Disable EFI tests for now

[Stephen Warren]

On 10/18/19 2:53 PM, Tom Rini wrote:
> When running these tests currently we see both warnings about how
> escape sequences need to be updated and then we see failures of some
> tests related to finding expected strings and so forth.  It seems quite
> likely that these tests need work to function with Python 3 and it's

s/it's/its/.

[U-Boot] [PATCH 00/10] Moving test/py to Python 3

[Stephen Warren]

On 10/18/19 2:53 PM, Tom Rini wrote:
> Hey all,
> 
> So, here's the first pass at moving our test/py code over to Python 3.
> It's not quite ready to merge as noted by the WORKAROUND / HACK patches
> at the end.  The first two patches can go in any time and I will
> probably pull them soon as well.  The next patch is an old one from
> Marek that I deferred at the time, but it's time now so I've updated it
> slightly to cover the new cases of the same code he fixed before.  For
> migrating the pytest code itself, I've left the conversion split in a
> few commits.  This isn't bisectable but certainly for review it's
> better.  I'm honestly not sure if it's more important for history to
> keep this bisectable or to have the changes broken out like this.
> 
> Finally, we have a workaround so that we can do this conversion before
> we convert buildman/patman/buildman to python3 (as that will take more
> work, it seems that we've followed an anti-pattern and python3 makes
> that much more of a pain to live with), a work-around to skip the FS
> tests just like we are today and then finally disabling EFI tests.  I've
> already told Heinrich on IRC that this will take some work to fix so
> it's on his list.
> 
> Now that I've posted this, aside from any feedback I'm going to look at
> the fs tests and see what's going on there as I'm not sure if we have
> regressions in the fs code or lack of host tools being installed or
> possibly both.

Aside from the comments I posted, the entire series,
Reviewed-by: Stephen Warren <swarren@nvidia.com>

[U-Boot] [PATCH 09/10] WORKAROUND: test/py: Skip fs tests for now

[Tom Rini]

On Fri, Oct 18, 2019 at 03:16:45PM -0600, Stephen Warren wrote:
> On 10/18/19 2:53 PM, Tom Rini wrote:
> > Currently, these tests are skipped when run in CI.
> 
> How? If they're being skipped by some other mechanism, do we need to remove
> that other mechanism now that we're actively specifying "not fs" when
> running under CI?

Currently they get auto-skipped.  Thinking harder, I bet it's related to
patch 2 where I un-break PATH and now we're finding tools in /sbin and
try and run the tests.

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20191018/c97ac1e5/attachment.sig>

[U-Boot] [PATCH 04/10] test/py: Automated conversion to Python 3

[Tom Rini]

On Fri, Oct 18, 2019 at 03:12:06PM -0600, Stephen Warren wrote:
> On 10/18/19 2:53 PM, Tom Rini wrote:
> > Use the 2to3 tool to perform numerous automatic conversions from Python
> > 2 syntax to Python 3.  Also fix whitespace problems that Python 3
> > catches that Python 2 did not.
> 
> > diff --git a/test/py/conftest.py b/test/py/conftest.py
> 
> >   import pytest
> >   from _pytest.runner import runtestprotocol
> >   import re
> > -import StringIO
> > +import io
> >   import sys
> 
> I think that list used to be sorted. IIRC some PEP8 tools complain about
> unsorted import lists.
> 
> > -try:
> > -    import configparser
> > -except:
> > -    import ConfigParser as configparser
> > +import configparser
> 
> Now that this is a plain import statement, let's put it with all the others
> at the appropriate point in the sorted list above.
> 
> 
> (Not sure if you want to merge fixes like this into this patch, or put them
> in path 7?)

I'll take care of both of those in patch 7, thanks!

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20191018/1ff85084/attachment.sig>

[U-Boot] [PATCH 04/10] test/py: Automated conversion to Python 3

[Heinrich Schuchardt]

On 10/18/19 10:53 PM, Tom Rini wrote:
> Use the 2to3 tool to perform numerous automatic conversions from Python
> 2 syntax to Python 3.  Also fix whitespace problems that Python 3
> catches that Python 2 did not.
>
> Signed-off-by: Tom Rini <trini@konsulko.com>

This patch breaks

./test/py/test.py --bd=qemu-arm64 --build-dir=. -k=test_efi_

For remediation see below.

> ---
>   test/py/conftest.py               |  9 ++--
>   test/py/multiplexed_log.py        |  4 +-
>   test/py/test.py                   |  4 +-
>   test/py/tests/test_fit.py         |  2 -
>   test/py/tests/test_fpga.py        | 46 +++++++++----------
>   test/py/tests/test_fs/conftest.py |  4 +-
>   test/py/tests/test_log.py         |  8 ++--
>   test/py/tests/test_mmc_wr.py      | 74 +++++++++++++++----------------
>   8 files changed, 71 insertions(+), 80 deletions(-)
>
> diff --git a/test/py/conftest.py b/test/py/conftest.py
> index 30c898b40a0d..5c19af1d5034 100644
> --- a/test/py/conftest.py
> +++ b/test/py/conftest.py
> @@ -19,13 +19,10 @@ import os.path
>   import pytest
>   from _pytest.runner import runtestprotocol
>   import re
> -import StringIO
> +import io
>   import sys
>
> -try:
> -    import configparser
> -except:
> -    import ConfigParser as configparser
> +import configparser
>
>   # Globals: The HTML log file, and the connection to the U-Boot console.
>   log = None
> @@ -169,7 +166,7 @@ def pytest_configure(config):
>
>           with open(dot_config, 'rt') as f:
>               ini_str = '[root]\n' + f.read()
> -            ini_sio = StringIO.StringIO(ini_str)
> +            ini_sio = io.StringIO(ini_str)

This results in an error

INTERNALERROR>   File "/home/user/u-boot/test/py/conftest.py", line 169,
in pytest_configure
INTERNALERROR>     ini_sio = io.StringIO(ini_str)
INTERNALERROR> TypeError: initial_value must be unicode or None, not str

-            ini_sio = io.StringIO(ini_str)
+            ini_sio = io.StringIO(ini_str.decode())

>               parser = configparser.RawConfigParser()
>               parser.readfp(ini_sio)
>               ubconfig.buildconfig.update(parser.items('root'))
> diff --git a/test/py/multiplexed_log.py b/test/py/multiplexed_log.py
> index 637a3bd257ba..de0aacc659b8 100644
> --- a/test/py/multiplexed_log.py
> +++ b/test/py/multiplexed_log.py
> @@ -5,8 +5,8 @@
>   # Generate an HTML-formatted log file containing multiple streams of data,
>   # each represented in a well-delineated/-structured fashion.
>
> -import cgi
>   import datetime
> +import html
>   import os.path
>   import shutil
>   import subprocess
> @@ -334,7 +334,7 @@ $(document).ready(function () {
>           data = data.replace(chr(13), '')
>           data = ''.join((ord(c) in self._nonprint) and ('%%%02x' % ord(c)) or
>                          c for c in data)
> -        data = cgi.escape(data)
> +        data = html.escape(data)
>           return data
>
>       def _terminate_stream(self):
> diff --git a/test/py/test.py b/test/py/test.py
> index a5140945d4b9..0ce1838833f6 100755
> --- a/test/py/test.py
> +++ b/test/py/test.py
> @@ -1,4 +1,4 @@
> -#!/usr/bin/env python2
> +#!/usr/bin/env python3
>   # SPDX-License-Identifier: GPL-2.0
>
>   # Copyright (c) 2015 Stephen Warren
> @@ -7,8 +7,6 @@
>   # Wrapper script to invoke pytest with the directory name that contains the
>   # U-Boot tests.
>
> -from __future__ import print_function
> -
>   import os
>   import os.path
>   import sys
> diff --git a/test/py/tests/test_fit.py b/test/py/tests/test_fit.py
> index e3210ed43fa4..4922b9dcc664 100755
> --- a/test/py/tests/test_fit.py
> +++ b/test/py/tests/test_fit.py

Please, replace all occurrences of

-            print(base_fdt, file=fd)
+            fd.write(base_fdt)

in this file.

Together with the correction in test/py/conftest.py I now can run

./test/py/test.py --bd=qemu-arm64 --build-dir=. -k=test_efi_

using origin/master + patches 1-4.

Best regards

Heinrich

> @@ -3,8 +3,6 @@
>   #
>   # Sanity check of the FIT handling in U-Boot
>
> -from __future__ import print_function
> -
>   import os
>   import pytest
>   import struct
> diff --git a/test/py/tests/test_fpga.py b/test/py/tests/test_fpga.py
> index e3bb7b41c749..ca7ef8ea40d6 100644
> --- a/test/py/tests/test_fpga.py
> +++ b/test/py/tests/test_fpga.py
> @@ -175,29 +175,29 @@ def test_fpga_load_fail(u_boot_console):
>       f, dev, addr, bit, bit_size = load_file_from_var(u_boot_console, 'bitstream_load')
>
>       for cmd in ['dump', 'load', 'loadb']:
> -	# missing dev parameter
> -	expected = 'fpga: incorrect parameters passed'
> -	output = u_boot_console.run_command('fpga %s %x $filesize' % (cmd, addr))
> -	#assert expected in output
> -	assert expected_usage in output
> -
> -	# more parameters - 0 at the end
> -	expected = 'fpga: more parameters passed'
> -	output = u_boot_console.run_command('fpga %s %x %x $filesize 0' % (cmd, dev, addr))
> -	#assert expected in output
> -	assert expected_usage in output
> -
> -	# 0 address
> -	expected = 'fpga: zero fpga_data address'
> -	output = u_boot_console.run_command('fpga %s %x 0 $filesize' % (cmd, dev))
> -	#assert expected in output
> -	assert expected_usage in output
> -
> -	# 0 filesize
> -	expected = 'fpga: zero size'
> -	output = u_boot_console.run_command('fpga %s %x %x 0' % (cmd, dev, addr))
> -	#assert expected in output
> -	assert expected_usage in output
> +        # missing dev parameter
> +        expected = 'fpga: incorrect parameters passed'
> +        output = u_boot_console.run_command('fpga %s %x $filesize' % (cmd, addr))
> +        #assert expected in output
> +        assert expected_usage in output
> +
> +        # more parameters - 0 at the end
> +        expected = 'fpga: more parameters passed'
> +        output = u_boot_console.run_command('fpga %s %x %x $filesize 0' % (cmd, dev, addr))
> +        #assert expected in output
> +        assert expected_usage in output
> +
> +        # 0 address
> +        expected = 'fpga: zero fpga_data address'
> +        output = u_boot_console.run_command('fpga %s %x 0 $filesize' % (cmd, dev))
> +        #assert expected in output
> +        assert expected_usage in output
> +
> +        # 0 filesize
> +        expected = 'fpga: zero size'
> +        output = u_boot_console.run_command('fpga %s %x %x 0' % (cmd, dev, addr))
> +        #assert expected in output
> +        assert expected_usage in output
>
>   @pytest.mark.buildconfigspec('cmd_fpga')
>   @pytest.mark.buildconfigspec('cmd_echo')
> diff --git a/test/py/tests/test_fs/conftest.py b/test/py/tests/test_fs/conftest.py
> index 9324657d2160..354d17672fe4 100644
> --- a/test/py/tests/test_fs/conftest.py
> +++ b/test/py/tests/test_fs/conftest.py
> @@ -508,8 +508,8 @@ def fs_obj_unlink(request, u_boot_config):
>
>           # Test Case 2
>           check_call('mkdir %s/dir2' % mount_dir, shell=True)
> -	for i in range(0, 20):
> -	    check_call('mkdir %s/dir2/0123456789abcdef%02x'
> +        for i in range(0, 20):
> +            check_call('mkdir %s/dir2/0123456789abcdef%02x'
>                                       % (mount_dir, i), shell=True)
>
>           # Test Case 4
> diff --git a/test/py/tests/test_log.py b/test/py/tests/test_log.py
> index cb183444c6fc..75325fad61ad 100644
> --- a/test/py/tests/test_log.py
> +++ b/test/py/tests/test_log.py
> @@ -27,9 +27,9 @@ def test_log(u_boot_console):
>           """
>           for i in range(max_level):
>               if mask & 1:
> -                assert 'log_run() log %d' % i == lines.next()
> +                assert 'log_run() log %d' % i == next(lines)
>               if mask & 3:
> -                assert 'func() _log %d' % i == lines.next()
> +                assert 'func() _log %d' % i == next(lines)
>
>       def run_test(testnum):
>           """Run a particular test number (the 'log test' command)
> @@ -43,7 +43,7 @@ def test_log(u_boot_console):
>              output = u_boot_console.run_command('log test %d' % testnum)
>           split = output.replace('\r', '').splitlines()
>           lines = iter(split)
> -        assert 'test %d' % testnum == lines.next()
> +        assert 'test %d' % testnum == next(lines)
>           return lines
>
>       def test0():
> @@ -88,7 +88,7 @@ def test_log(u_boot_console):
>       def test10():
>           lines = run_test(10)
>           for i in range(7):
> -            assert 'log_test() level %d' % i == lines.next()
> +            assert 'log_test() level %d' % i == next(lines)
>
>       # TODO(sjg at chromium.org): Consider structuring this as separate tests
>       cons = u_boot_console
> diff --git a/test/py/tests/test_mmc_wr.py b/test/py/tests/test_mmc_wr.py
> index 8b18781eac74..2ce79f58e3b1 100644
> --- a/test/py/tests/test_mmc_wr.py
> +++ b/test/py/tests/test_mmc_wr.py
> @@ -65,41 +65,39 @@ def test_mmc_wr(u_boot_console, env__mmc_wr_config):
>
>
>       for i in range(test_iterations):
> -	# Generate random data
> -	cmd = 'random %s %x' % (src_addr, count_bytes)
> -	response = u_boot_console.run_command(cmd)
> -	good_response = '%d bytes filled with random data' % (count_bytes)
> -	assert good_response in response
> -
> -	# Select MMC device
> -	cmd = 'mmc dev %d' % devid
> -	if is_emmc:
> -		cmd += ' %d' % partid
> -	response = u_boot_console.run_command(cmd)
> -	assert 'no card present' not in response
> -	if is_emmc:
> -		partid_response = "(part %d)" % partid
> -	else:
> -		partid_response = ""
> -	good_response = 'mmc%d%s is current device' % (devid, partid_response)
> -	assert good_response in response
> -
> -	# Write data
> -	cmd = 'mmc write %s %x %x' % (src_addr, sector, count_sectors)
> -	response = u_boot_console.run_command(cmd)
> -	good_response = 'MMC write: dev # %d, block # %d, count %d ... %d blocks written: OK' % (
> -		devid, sector, count_sectors, count_sectors)
> -	assert good_response in response
> -
> -	# Read data
> -	cmd = 'mmc read %s %x %x' % (dst_addr, sector, count_sectors)
> -	response = u_boot_console.run_command(cmd)
> -	good_response = 'MMC read: dev # %d, block # %d, count %d ... %d blocks read: OK' % (
> -		devid, sector, count_sectors, count_sectors)
> -	assert good_response in response
> -
> -	# Compare src and dst data
> -	cmd = 'cmp.b %s %s %x' % (src_addr, dst_addr, count_bytes)
> -	response = u_boot_console.run_command(cmd)
> -	good_response = 'Total of %d byte(s) were the same' % (count_bytes)
> -	assert good_response in response
> +        # Generate random data
> +        cmd = 'random %s %x' % (src_addr, count_bytes)
> +        response = u_boot_console.run_command(cmd)
> +        good_response = '%d bytes filled with random data' % (count_bytes)
> +        assert good_response in response
> +
> +        # Select MMC device
> +        cmd = 'mmc dev %d' % devid
> +        if is_emmc:
> +            cmd += ' %d' % partid
> +        response = u_boot_console.run_command(cmd)
> +        assert 'no card present' not in response
> +        if is_emmc:
> +            partid_response = "(part %d)" % partid
> +        else:
> +            partid_response = ""
> +        good_response = 'mmc%d%s is current device' % (devid, partid_response)
> +        assert good_response in response
> +
> +        # Write data
> +        cmd = 'mmc write %s %x %x' % (src_addr, sector, count_sectors)
> +        response = u_boot_console.run_command(cmd)
> +        good_response = 'MMC write: dev # %d, block # %d, count %d ... %d blocks written: OK' % (devid, sector, count_sectors, count_sectors)
> +        assert good_response in response
> +
> +        # Read data
> +        cmd = 'mmc read %s %x %x' % (dst_addr, sector, count_sectors)
> +        response = u_boot_console.run_command(cmd)
> +        good_response = 'MMC read: dev # %d, block # %d, count %d ... %d blocks read: OK' % (devid, sector, count_sectors, count_sectors)
> +        assert good_response in response
> +
> +        # Compare src and dst data
> +        cmd = 'cmp.b %s %s %x' % (src_addr, dst_addr, count_bytes)
> +        response = u_boot_console.run_command(cmd)
> +        good_response = 'Total of %d byte(s) were the same' % (count_bytes)
> +        assert good_response in response
>

[U-Boot] [RFC v2 4/10] test/py: Automated conversion to Python 3

[Heinrich Schuchardt]

From: Tom Rini <trini@konsulko.com>

Use the 2to3 tool to perform numerous automatic conversions from Python
2 syntax to Python 3.  Also fix whitespace problems that Python 3
catches that Python 2 did not.

Signed-off-by: Tom Rini <trini@konsulko.com>

Add missing Python 3 conversion.

Signed-off-by: Heinrich Schuchardt <xypron.glpk@gmx.de>
---
 test/py/conftest.py               |  9 ++--
 test/py/multiplexed_log.py        |  4 +-
 test/py/test.py                   |  4 +-
 test/py/tests/test_fit.py         | 12 +++--
 test/py/tests/test_fpga.py        | 46 +++++++++----------
 test/py/tests/test_fs/conftest.py |  4 +-
 test/py/tests/test_log.py         |  8 ++--
 test/py/tests/test_mmc_wr.py      | 74 +++++++++++++++----------------
 8 files changed, 76 insertions(+), 85 deletions(-)

diff --git a/test/py/conftest.py b/test/py/conftest.py
index 30c898b40a..7da4b77bce 100644
--- a/test/py/conftest.py
+++ b/test/py/conftest.py
@@ -19,13 +19,10 @@ import os.path
 import pytest
 from _pytest.runner import runtestprotocol
 import re
-import StringIO
+import io
 import sys

-try:
-    import configparser
-except:
-    import ConfigParser as configparser
+import configparser

 # Globals: The HTML log file, and the connection to the U-Boot console.
 log = None
@@ -169,7 +166,7 @@ def pytest_configure(config):

         with open(dot_config, 'rt') as f:
             ini_str = '[root]\n' + f.read()
-            ini_sio = StringIO.StringIO(ini_str)
+            ini_sio = io.StringIO(ini_str.decode())
             parser = configparser.RawConfigParser()
             parser.readfp(ini_sio)
             ubconfig.buildconfig.update(parser.items('root'))
diff --git a/test/py/multiplexed_log.py b/test/py/multiplexed_log.py
index 637a3bd257..de0aacc659 100644
--- a/test/py/multiplexed_log.py
+++ b/test/py/multiplexed_log.py
@@ -5,8 +5,8 @@
 # Generate an HTML-formatted log file containing multiple streams of data,
 # each represented in a well-delineated/-structured fashion.

-import cgi
 import datetime
+import html
 import os.path
 import shutil
 import subprocess
@@ -334,7 +334,7 @@ $(document).ready(function () {
         data = data.replace(chr(13), '')
         data = ''.join((ord(c) in self._nonprint) and ('%%%02x' % ord(c)) or
                        c for c in data)
-        data = cgi.escape(data)
+        data = html.escape(data)
         return data

     def _terminate_stream(self):
diff --git a/test/py/test.py b/test/py/test.py
index a5140945d4..0ce1838833 100755
--- a/test/py/test.py
+++ b/test/py/test.py
@@ -1,4 +1,4 @@
-#!/usr/bin/env python2
+#!/usr/bin/env python3
 # SPDX-License-Identifier: GPL-2.0

 # Copyright (c) 2015 Stephen Warren
@@ -7,8 +7,6 @@
 # Wrapper script to invoke pytest with the directory name that contains the
 # U-Boot tests.

-from __future__ import print_function
-
 import os
 import os.path
 import sys
diff --git a/test/py/tests/test_fit.py b/test/py/tests/test_fit.py
index e3210ed43f..ea9286a553 100755
--- a/test/py/tests/test_fit.py
+++ b/test/py/tests/test_fit.py
@@ -3,8 +3,6 @@
 #
 # Sanity check of the FIT handling in U-Boot

-from __future__ import print_function
-
 import os
 import pytest
 import struct
@@ -155,7 +153,7 @@ def test_fit(u_boot_console):
         src = make_fname('u-boot.dts')
         dtb = make_fname('u-boot.dtb')
         with open(src, 'w') as fd:
-            print(base_fdt, file=fd)
+            fd.write(base_fdt)
         util.run_and_log(cons, ['dtc', src, '-O', 'dtb', '-o', dtb])
         return dtb

@@ -169,7 +167,7 @@ def test_fit(u_boot_console):
         """
         its = make_fname('test.its')
         with open(its, 'w') as fd:
-            print(base_its % params, file=fd)
+            fd.write(base_its % params)
         return its

     def make_fit(mkimage, params):
@@ -188,7 +186,7 @@ def test_fit(u_boot_console):
         its = make_its(params)
         util.run_and_log(cons, [mkimage, '-f', its, fit])
         with open(make_fname('u-boot.dts'), 'w') as fd:
-            print(base_fdt, file=fd)
+            fd.write(base_fdt)
         return fit

     def make_kernel(filename, text):
@@ -204,7 +202,7 @@ def test_fit(u_boot_console):
         for i in range(100):
             data += 'this %s %d is unlikely to boot\n' % (text, i)
         with open(fname, 'w') as fd:
-            print(data, file=fd)
+            fd.write(data)
         return fname

     def make_ramdisk(filename, text):
@@ -218,7 +216,7 @@ def test_fit(u_boot_console):
         for i in range(100):
             data += '%s %d was seldom used in the middle ages\n' % (text, i)
         with open(fname, 'w') as fd:
-            print(data, file=fd)
+            fd.write(data)
         return fname

     def make_compressed(filename):
diff --git a/test/py/tests/test_fpga.py b/test/py/tests/test_fpga.py
index e3bb7b41c7..ca7ef8ea40 100644
--- a/test/py/tests/test_fpga.py
+++ b/test/py/tests/test_fpga.py
@@ -175,29 +175,29 @@ def test_fpga_load_fail(u_boot_console):
     f, dev, addr, bit, bit_size = load_file_from_var(u_boot_console, 'bitstream_load')

     for cmd in ['dump', 'load', 'loadb']:
-	# missing dev parameter
-	expected = 'fpga: incorrect parameters passed'
-	output = u_boot_console.run_command('fpga %s %x $filesize' % (cmd, addr))
-	#assert expected in output
-	assert expected_usage in output
-
-	# more parameters - 0 at the end
-	expected = 'fpga: more parameters passed'
-	output = u_boot_console.run_command('fpga %s %x %x $filesize 0' % (cmd, dev, addr))
-	#assert expected in output
-	assert expected_usage in output
-
-	# 0 address
-	expected = 'fpga: zero fpga_data address'
-	output = u_boot_console.run_command('fpga %s %x 0 $filesize' % (cmd, dev))
-	#assert expected in output
-	assert expected_usage in output
-
-	# 0 filesize
-	expected = 'fpga: zero size'
-	output = u_boot_console.run_command('fpga %s %x %x 0' % (cmd, dev, addr))
-	#assert expected in output
-	assert expected_usage in output
+        # missing dev parameter
+        expected = 'fpga: incorrect parameters passed'
+        output = u_boot_console.run_command('fpga %s %x $filesize' % (cmd, addr))
+        #assert expected in output
+        assert expected_usage in output
+
+        # more parameters - 0 at the end
+        expected = 'fpga: more parameters passed'
+        output = u_boot_console.run_command('fpga %s %x %x $filesize 0' % (cmd, dev, addr))
+        #assert expected in output
+        assert expected_usage in output
+
+        # 0 address
+        expected = 'fpga: zero fpga_data address'
+        output = u_boot_console.run_command('fpga %s %x 0 $filesize' % (cmd, dev))
+        #assert expected in output
+        assert expected_usage in output
+
+        # 0 filesize
+        expected = 'fpga: zero size'
+        output = u_boot_console.run_command('fpga %s %x %x 0' % (cmd, dev, addr))
+        #assert expected in output
+        assert expected_usage in output

 @pytest.mark.buildconfigspec('cmd_fpga')
 @pytest.mark.buildconfigspec('cmd_echo')
diff --git a/test/py/tests/test_fs/conftest.py b/test/py/tests/test_fs/conftest.py
index 9324657d21..354d17672f 100644
--- a/test/py/tests/test_fs/conftest.py
+++ b/test/py/tests/test_fs/conftest.py
@@ -508,8 +508,8 @@ def fs_obj_unlink(request, u_boot_config):

         # Test Case 2
         check_call('mkdir %s/dir2' % mount_dir, shell=True)
-	for i in range(0, 20):
-	    check_call('mkdir %s/dir2/0123456789abcdef%02x'
+        for i in range(0, 20):
+            check_call('mkdir %s/dir2/0123456789abcdef%02x'
                                     % (mount_dir, i), shell=True)

         # Test Case 4
diff --git a/test/py/tests/test_log.py b/test/py/tests/test_log.py
index cb183444c6..75325fad61 100644
--- a/test/py/tests/test_log.py
+++ b/test/py/tests/test_log.py
@@ -27,9 +27,9 @@ def test_log(u_boot_console):
         """
         for i in range(max_level):
             if mask & 1:
-                assert 'log_run() log %d' % i == lines.next()
+                assert 'log_run() log %d' % i == next(lines)
             if mask & 3:
-                assert 'func() _log %d' % i == lines.next()
+                assert 'func() _log %d' % i == next(lines)

     def run_test(testnum):
         """Run a particular test number (the 'log test' command)
@@ -43,7 +43,7 @@ def test_log(u_boot_console):
            output = u_boot_console.run_command('log test %d' % testnum)
         split = output.replace('\r', '').splitlines()
         lines = iter(split)
-        assert 'test %d' % testnum == lines.next()
+        assert 'test %d' % testnum == next(lines)
         return lines

     def test0():
@@ -88,7 +88,7 @@ def test_log(u_boot_console):
     def test10():
         lines = run_test(10)
         for i in range(7):
-            assert 'log_test() level %d' % i == lines.next()
+            assert 'log_test() level %d' % i == next(lines)

     # TODO(sjg at chromium.org): Consider structuring this as separate tests
     cons = u_boot_console
diff --git a/test/py/tests/test_mmc_wr.py b/test/py/tests/test_mmc_wr.py
index 8b18781eac..2ce79f58e3 100644
--- a/test/py/tests/test_mmc_wr.py
+++ b/test/py/tests/test_mmc_wr.py
@@ -65,41 +65,39 @@ def test_mmc_wr(u_boot_console, env__mmc_wr_config):


     for i in range(test_iterations):
-	# Generate random data
-	cmd = 'random %s %x' % (src_addr, count_bytes)
-	response = u_boot_console.run_command(cmd)
-	good_response = '%d bytes filled with random data' % (count_bytes)
-	assert good_response in response
-
-	# Select MMC device
-	cmd = 'mmc dev %d' % devid
-	if is_emmc:
-		cmd += ' %d' % partid
-	response = u_boot_console.run_command(cmd)
-	assert 'no card present' not in response
-	if is_emmc:
-		partid_response = "(part %d)" % partid
-	else:
-		partid_response = ""
-	good_response = 'mmc%d%s is current device' % (devid, partid_response)
-	assert good_response in response
-
-	# Write data
-	cmd = 'mmc write %s %x %x' % (src_addr, sector, count_sectors)
-	response = u_boot_console.run_command(cmd)
-	good_response = 'MMC write: dev # %d, block # %d, count %d ... %d blocks written: OK' % (
-		devid, sector, count_sectors, count_sectors)
-	assert good_response in response
-
-	# Read data
-	cmd = 'mmc read %s %x %x' % (dst_addr, sector, count_sectors)
-	response = u_boot_console.run_command(cmd)
-	good_response = 'MMC read: dev # %d, block # %d, count %d ... %d blocks read: OK' % (
-		devid, sector, count_sectors, count_sectors)
-	assert good_response in response
-
-	# Compare src and dst data
-	cmd = 'cmp.b %s %s %x' % (src_addr, dst_addr, count_bytes)
-	response = u_boot_console.run_command(cmd)
-	good_response = 'Total of %d byte(s) were the same' % (count_bytes)
-	assert good_response in response
+        # Generate random data
+        cmd = 'random %s %x' % (src_addr, count_bytes)
+        response = u_boot_console.run_command(cmd)
+        good_response = '%d bytes filled with random data' % (count_bytes)
+        assert good_response in response
+
+        # Select MMC device
+        cmd = 'mmc dev %d' % devid
+        if is_emmc:
+            cmd += ' %d' % partid
+        response = u_boot_console.run_command(cmd)
+        assert 'no card present' not in response
+        if is_emmc:
+            partid_response = "(part %d)" % partid
+        else:
+            partid_response = ""
+        good_response = 'mmc%d%s is current device' % (devid, partid_response)
+        assert good_response in response
+
+        # Write data
+        cmd = 'mmc write %s %x %x' % (src_addr, sector, count_sectors)
+        response = u_boot_console.run_command(cmd)
+        good_response = 'MMC write: dev # %d, block # %d, count %d ... %d blocks written: OK' % (devid, sector, count_sectors, count_sectors)
+        assert good_response in response
+
+        # Read data
+        cmd = 'mmc read %s %x %x' % (dst_addr, sector, count_sectors)
+        response = u_boot_console.run_command(cmd)
+        good_response = 'MMC read: dev # %d, block # %d, count %d ... %d blocks read: OK' % (devid, sector, count_sectors, count_sectors)
+        assert good_response in response
+
+        # Compare src and dst data
+        cmd = 'cmp.b %s %s %x' % (src_addr, dst_addr, count_bytes)
+        response = u_boot_console.run_command(cmd)
+        good_response = 'Total of %d byte(s) were the same' % (count_bytes)
+        assert good_response in response
--
2.23.0

[U-Boot] [PATCH 07/10] test/py: Manual python3 fixes

[Heinrich Schuchardt]

On 10/18/19 10:53 PM, Tom Rini wrote:
> - Modern pytest is more visible in telling us about parameters that we
>    had not described, so describe a few more.
> - ConfigParser.readfp(...) is now configparser.read_file(...)
> - As part of the "strings vs bytes" conversions in Python 3, we need to
>    encode / decode our pipes in a few areas, use utf-8 for this.
> - Fix a typo noticed while doing the above ("tot he" -> "to the").
>
> Signed-off-by: Tom Rini <trini@konsulko.com>

This patch breaks

./test/py/test.py --bd=qemu-arm64 --build-dir=.

INTERNALERROR>   File "test/py/conftest.py", line 201, in pytest_configure
INTERNALERROR>     console =
u_boot_console_exec_attach.ConsoleExecAttach(log, ubconfig)
INTERNALERROR>   File "test/py/u_boot_console_exec_attach.py", line 41,
in __init__
INTERNALERROR>     runner.run(cmd)
INTERNALERROR>   File "test/py/multiplexed_log.py", line 174, in run
INTERNALERROR>     raise exception
INTERNALERROR> TypeError: __init__() got an unexpected keyword argument
'encoding'

Best regards

Heinrich

> ---
>   test/py/conftest.py        | 2 +-
>   test/py/multiplexed_log.py | 4 ++--
>   test/py/pytest.ini         | 3 +++
>   test/py/u_boot_spawn.py    | 4 ++--
>   4 files changed, 8 insertions(+), 5 deletions(-)
>
> diff --git a/test/py/conftest.py b/test/py/conftest.py
> index 5c19af1d5034..02b61655dcee 100644
> --- a/test/py/conftest.py
> +++ b/test/py/conftest.py
> @@ -168,7 +168,7 @@ def pytest_configure(config):
>               ini_str = '[root]\n' + f.read()
>               ini_sio = io.StringIO(ini_str)
>               parser = configparser.RawConfigParser()
> -            parser.readfp(ini_sio)
> +            parser.read_file(ini_sio)
>               ubconfig.buildconfig.update(parser.items('root'))
>
>       ubconfig.test_py_dir = test_py_dir
> diff --git a/test/py/multiplexed_log.py b/test/py/multiplexed_log.py
> index de0aacc659b8..35d95fbe0fe8 100644
> --- a/test/py/multiplexed_log.py
> +++ b/test/py/multiplexed_log.py
> @@ -51,7 +51,7 @@ class LogfileStream(object):
>           """Write data to the log stream.
>
>           Args:
> -            data: The data to write tot he file.
> +            data: The data to write to the file.
>               implicit: Boolean indicating whether data actually appeared in the
>                   stream, or was implicitly generated. A valid use-case is to
>                   repeat a shell prompt at the start of each separate log
> @@ -133,7 +133,7 @@ class RunAndLog(object):
>           self.logfile.write(self, msg)
>
>           try:
> -            p = subprocess.Popen(cmd, cwd=cwd,
> +            p = subprocess.Popen(cmd, cwd=cwd, encoding="utf-8",
>                   stdin=None, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
>               (stdout, stderr) = p.communicate()
>               output = ''
> diff --git a/test/py/pytest.ini b/test/py/pytest.ini
> index 7e400682bf25..e93d010f1fa2 100644
> --- a/test/py/pytest.ini
> +++ b/test/py/pytest.ini
> @@ -8,3 +8,6 @@
>   markers =
>       boardspec: U-Boot: Describes the set of boards a test can/can't run on.
>       buildconfigspec: U-Boot: Describes Kconfig/config-header constraints.
> +    notbuildconfigspec: U-Boot: Describes required disabled Kconfig options.
> +    requiredtool: U-Boot: Required host tools for a test.
> +    slow: U-Boot: Specific test will run slowly.
> diff --git a/test/py/u_boot_spawn.py b/test/py/u_boot_spawn.py
> index b011a3e3da25..63119229f420 100644
> --- a/test/py/u_boot_spawn.py
> +++ b/test/py/u_boot_spawn.py
> @@ -113,7 +113,7 @@ class Spawn(object):
>               Nothing.
>           """
>
> -        os.write(self.fd, data)
> +        os.write(self.fd, data.encode("utf-8"))
>
>       def expect(self, patterns):
>           """Wait for the sub-process to emit specific data.
> @@ -171,7 +171,7 @@ class Spawn(object):
>                   events = self.poll.poll(poll_maxwait)
>                   if not events:
>                       raise Timeout()
> -                c = os.read(self.fd, 1024)
> +                c = os.read(self.fd, 1024).decode("utf-8")
>                   if not c:
>                       raise EOFError()
>                   if self.logfile_read:
>

[U-Boot] [RFC v2 4/10] test/py: Automated conversion to Python 3

[Tom Rini]

On Sat, Oct 19, 2019 at 08:13:18AM +0200, Heinrich Schuchardt wrote:
> From: Tom Rini <trini@konsulko.com>
> 
> Use the 2to3 tool to perform numerous automatic conversions from Python
> 2 syntax to Python 3.  Also fix whitespace problems that Python 3
> catches that Python 2 did not.
> 
> Signed-off-by: Tom Rini <trini@konsulko.com>
> 
> Add missing Python 3 conversion.
> 
> Signed-off-by: Heinrich Schuchardt <xypron.glpk@gmx.de>

Thanks!  I'll slightly reword the commit message and pick this part up.

> ---
>  test/py/conftest.py               |  9 ++--
>  test/py/multiplexed_log.py        |  4 +-
>  test/py/test.py                   |  4 +-
>  test/py/tests/test_fit.py         | 12 +++--
>  test/py/tests/test_fpga.py        | 46 +++++++++----------
>  test/py/tests/test_fs/conftest.py |  4 +-
>  test/py/tests/test_log.py         |  8 ++--
>  test/py/tests/test_mmc_wr.py      | 74 +++++++++++++++----------------
>  8 files changed, 76 insertions(+), 85 deletions(-)
> 
> diff --git a/test/py/conftest.py b/test/py/conftest.py
> index 30c898b40a..7da4b77bce 100644
> --- a/test/py/conftest.py
> +++ b/test/py/conftest.py
> @@ -19,13 +19,10 @@ import os.path
>  import pytest
>  from _pytest.runner import runtestprotocol
>  import re
> -import StringIO
> +import io
>  import sys
> 
> -try:
> -    import configparser
> -except:
> -    import ConfigParser as configparser
> +import configparser
> 
>  # Globals: The HTML log file, and the connection to the U-Boot console.
>  log = None
> @@ -169,7 +166,7 @@ def pytest_configure(config):
> 
>          with open(dot_config, 'rt') as f:
>              ini_str = '[root]\n' + f.read()
> -            ini_sio = StringIO.StringIO(ini_str)
> +            ini_sio = io.StringIO(ini_str.decode())
>              parser = configparser.RawConfigParser()
>              parser.readfp(ini_sio)
>              ubconfig.buildconfig.update(parser.items('root'))
> diff --git a/test/py/multiplexed_log.py b/test/py/multiplexed_log.py
> index 637a3bd257..de0aacc659 100644
> --- a/test/py/multiplexed_log.py
> +++ b/test/py/multiplexed_log.py
> @@ -5,8 +5,8 @@
>  # Generate an HTML-formatted log file containing multiple streams of data,
>  # each represented in a well-delineated/-structured fashion.
> 
> -import cgi
>  import datetime
> +import html
>  import os.path
>  import shutil
>  import subprocess
> @@ -334,7 +334,7 @@ $(document).ready(function () {
>          data = data.replace(chr(13), '')
>          data = ''.join((ord(c) in self._nonprint) and ('%%%02x' % ord(c)) or
>                         c for c in data)
> -        data = cgi.escape(data)
> +        data = html.escape(data)
>          return data
> 
>      def _terminate_stream(self):
> diff --git a/test/py/test.py b/test/py/test.py
> index a5140945d4..0ce1838833 100755
> --- a/test/py/test.py
> +++ b/test/py/test.py
> @@ -1,4 +1,4 @@
> -#!/usr/bin/env python2
> +#!/usr/bin/env python3
>  # SPDX-License-Identifier: GPL-2.0
> 
>  # Copyright (c) 2015 Stephen Warren
> @@ -7,8 +7,6 @@
>  # Wrapper script to invoke pytest with the directory name that contains the
>  # U-Boot tests.
> 
> -from __future__ import print_function
> -
>  import os
>  import os.path
>  import sys
> diff --git a/test/py/tests/test_fit.py b/test/py/tests/test_fit.py
> index e3210ed43f..ea9286a553 100755
> --- a/test/py/tests/test_fit.py
> +++ b/test/py/tests/test_fit.py
> @@ -3,8 +3,6 @@
>  #
>  # Sanity check of the FIT handling in U-Boot
> 
> -from __future__ import print_function
> -
>  import os
>  import pytest
>  import struct
> @@ -155,7 +153,7 @@ def test_fit(u_boot_console):
>          src = make_fname('u-boot.dts')
>          dtb = make_fname('u-boot.dtb')
>          with open(src, 'w') as fd:
> -            print(base_fdt, file=fd)
> +            fd.write(base_fdt)
>          util.run_and_log(cons, ['dtc', src, '-O', 'dtb', '-o', dtb])
>          return dtb
> 
> @@ -169,7 +167,7 @@ def test_fit(u_boot_console):
>          """
>          its = make_fname('test.its')
>          with open(its, 'w') as fd:
> -            print(base_its % params, file=fd)
> +            fd.write(base_its % params)
>          return its
> 
>      def make_fit(mkimage, params):
> @@ -188,7 +186,7 @@ def test_fit(u_boot_console):
>          its = make_its(params)
>          util.run_and_log(cons, [mkimage, '-f', its, fit])
>          with open(make_fname('u-boot.dts'), 'w') as fd:
> -            print(base_fdt, file=fd)
> +            fd.write(base_fdt)
>          return fit
> 
>      def make_kernel(filename, text):
> @@ -204,7 +202,7 @@ def test_fit(u_boot_console):
>          for i in range(100):
>              data += 'this %s %d is unlikely to boot\n' % (text, i)
>          with open(fname, 'w') as fd:
> -            print(data, file=fd)
> +            fd.write(data)
>          return fname
> 
>      def make_ramdisk(filename, text):
> @@ -218,7 +216,7 @@ def test_fit(u_boot_console):
>          for i in range(100):
>              data += '%s %d was seldom used in the middle ages\n' % (text, i)
>          with open(fname, 'w') as fd:
> -            print(data, file=fd)
> +            fd.write(data)
>          return fname
> 
>      def make_compressed(filename):
> diff --git a/test/py/tests/test_fpga.py b/test/py/tests/test_fpga.py
> index e3bb7b41c7..ca7ef8ea40 100644
> --- a/test/py/tests/test_fpga.py
> +++ b/test/py/tests/test_fpga.py
> @@ -175,29 +175,29 @@ def test_fpga_load_fail(u_boot_console):
>      f, dev, addr, bit, bit_size = load_file_from_var(u_boot_console, 'bitstream_load')
> 
>      for cmd in ['dump', 'load', 'loadb']:
> -	# missing dev parameter
> -	expected = 'fpga: incorrect parameters passed'
> -	output = u_boot_console.run_command('fpga %s %x $filesize' % (cmd, addr))
> -	#assert expected in output
> -	assert expected_usage in output
> -
> -	# more parameters - 0 at the end
> -	expected = 'fpga: more parameters passed'
> -	output = u_boot_console.run_command('fpga %s %x %x $filesize 0' % (cmd, dev, addr))
> -	#assert expected in output
> -	assert expected_usage in output
> -
> -	# 0 address
> -	expected = 'fpga: zero fpga_data address'
> -	output = u_boot_console.run_command('fpga %s %x 0 $filesize' % (cmd, dev))
> -	#assert expected in output
> -	assert expected_usage in output
> -
> -	# 0 filesize
> -	expected = 'fpga: zero size'
> -	output = u_boot_console.run_command('fpga %s %x %x 0' % (cmd, dev, addr))
> -	#assert expected in output
> -	assert expected_usage in output
> +        # missing dev parameter
> +        expected = 'fpga: incorrect parameters passed'
> +        output = u_boot_console.run_command('fpga %s %x $filesize' % (cmd, addr))
> +        #assert expected in output
> +        assert expected_usage in output
> +
> +        # more parameters - 0 at the end
> +        expected = 'fpga: more parameters passed'
> +        output = u_boot_console.run_command('fpga %s %x %x $filesize 0' % (cmd, dev, addr))
> +        #assert expected in output
> +        assert expected_usage in output
> +
> +        # 0 address
> +        expected = 'fpga: zero fpga_data address'
> +        output = u_boot_console.run_command('fpga %s %x 0 $filesize' % (cmd, dev))
> +        #assert expected in output
> +        assert expected_usage in output
> +
> +        # 0 filesize
> +        expected = 'fpga: zero size'
> +        output = u_boot_console.run_command('fpga %s %x %x 0' % (cmd, dev, addr))
> +        #assert expected in output
> +        assert expected_usage in output
> 
>  @pytest.mark.buildconfigspec('cmd_fpga')
>  @pytest.mark.buildconfigspec('cmd_echo')
> diff --git a/test/py/tests/test_fs/conftest.py b/test/py/tests/test_fs/conftest.py
> index 9324657d21..354d17672f 100644
> --- a/test/py/tests/test_fs/conftest.py
> +++ b/test/py/tests/test_fs/conftest.py
> @@ -508,8 +508,8 @@ def fs_obj_unlink(request, u_boot_config):
> 
>          # Test Case 2
>          check_call('mkdir %s/dir2' % mount_dir, shell=True)
> -	for i in range(0, 20):
> -	    check_call('mkdir %s/dir2/0123456789abcdef%02x'
> +        for i in range(0, 20):
> +            check_call('mkdir %s/dir2/0123456789abcdef%02x'
>                                      % (mount_dir, i), shell=True)
> 
>          # Test Case 4
> diff --git a/test/py/tests/test_log.py b/test/py/tests/test_log.py
> index cb183444c6..75325fad61 100644
> --- a/test/py/tests/test_log.py
> +++ b/test/py/tests/test_log.py
> @@ -27,9 +27,9 @@ def test_log(u_boot_console):
>          """
>          for i in range(max_level):
>              if mask & 1:
> -                assert 'log_run() log %d' % i == lines.next()
> +                assert 'log_run() log %d' % i == next(lines)
>              if mask & 3:
> -                assert 'func() _log %d' % i == lines.next()
> +                assert 'func() _log %d' % i == next(lines)
> 
>      def run_test(testnum):
>          """Run a particular test number (the 'log test' command)
> @@ -43,7 +43,7 @@ def test_log(u_boot_console):
>             output = u_boot_console.run_command('log test %d' % testnum)
>          split = output.replace('\r', '').splitlines()
>          lines = iter(split)
> -        assert 'test %d' % testnum == lines.next()
> +        assert 'test %d' % testnum == next(lines)
>          return lines
> 
>      def test0():
> @@ -88,7 +88,7 @@ def test_log(u_boot_console):
>      def test10():
>          lines = run_test(10)
>          for i in range(7):
> -            assert 'log_test() level %d' % i == lines.next()
> +            assert 'log_test() level %d' % i == next(lines)
> 
>      # TODO(sjg at chromium.org): Consider structuring this as separate tests
>      cons = u_boot_console
> diff --git a/test/py/tests/test_mmc_wr.py b/test/py/tests/test_mmc_wr.py
> index 8b18781eac..2ce79f58e3 100644
> --- a/test/py/tests/test_mmc_wr.py
> +++ b/test/py/tests/test_mmc_wr.py
> @@ -65,41 +65,39 @@ def test_mmc_wr(u_boot_console, env__mmc_wr_config):
> 
> 
>      for i in range(test_iterations):
> -	# Generate random data
> -	cmd = 'random %s %x' % (src_addr, count_bytes)
> -	response = u_boot_console.run_command(cmd)
> -	good_response = '%d bytes filled with random data' % (count_bytes)
> -	assert good_response in response
> -
> -	# Select MMC device
> -	cmd = 'mmc dev %d' % devid
> -	if is_emmc:
> -		cmd += ' %d' % partid
> -	response = u_boot_console.run_command(cmd)
> -	assert 'no card present' not in response
> -	if is_emmc:
> -		partid_response = "(part %d)" % partid
> -	else:
> -		partid_response = ""
> -	good_response = 'mmc%d%s is current device' % (devid, partid_response)
> -	assert good_response in response
> -
> -	# Write data
> -	cmd = 'mmc write %s %x %x' % (src_addr, sector, count_sectors)
> -	response = u_boot_console.run_command(cmd)
> -	good_response = 'MMC write: dev # %d, block # %d, count %d ... %d blocks written: OK' % (
> -		devid, sector, count_sectors, count_sectors)
> -	assert good_response in response
> -
> -	# Read data
> -	cmd = 'mmc read %s %x %x' % (dst_addr, sector, count_sectors)
> -	response = u_boot_console.run_command(cmd)
> -	good_response = 'MMC read: dev # %d, block # %d, count %d ... %d blocks read: OK' % (
> -		devid, sector, count_sectors, count_sectors)
> -	assert good_response in response
> -
> -	# Compare src and dst data
> -	cmd = 'cmp.b %s %s %x' % (src_addr, dst_addr, count_bytes)
> -	response = u_boot_console.run_command(cmd)
> -	good_response = 'Total of %d byte(s) were the same' % (count_bytes)
> -	assert good_response in response
> +        # Generate random data
> +        cmd = 'random %s %x' % (src_addr, count_bytes)
> +        response = u_boot_console.run_command(cmd)
> +        good_response = '%d bytes filled with random data' % (count_bytes)
> +        assert good_response in response
> +
> +        # Select MMC device
> +        cmd = 'mmc dev %d' % devid
> +        if is_emmc:
> +            cmd += ' %d' % partid
> +        response = u_boot_console.run_command(cmd)
> +        assert 'no card present' not in response
> +        if is_emmc:
> +            partid_response = "(part %d)" % partid
> +        else:
> +            partid_response = ""
> +        good_response = 'mmc%d%s is current device' % (devid, partid_response)
> +        assert good_response in response
> +
> +        # Write data
> +        cmd = 'mmc write %s %x %x' % (src_addr, sector, count_sectors)
> +        response = u_boot_console.run_command(cmd)
> +        good_response = 'MMC write: dev # %d, block # %d, count %d ... %d blocks written: OK' % (devid, sector, count_sectors, count_sectors)
> +        assert good_response in response
> +
> +        # Read data
> +        cmd = 'mmc read %s %x %x' % (dst_addr, sector, count_sectors)
> +        response = u_boot_console.run_command(cmd)
> +        good_response = 'MMC read: dev # %d, block # %d, count %d ... %d blocks read: OK' % (devid, sector, count_sectors, count_sectors)
> +        assert good_response in response
> +
> +        # Compare src and dst data
> +        cmd = 'cmp.b %s %s %x' % (src_addr, dst_addr, count_bytes)
> +        response = u_boot_console.run_command(cmd)
> +        good_response = 'Total of %d byte(s) were the same' % (count_bytes)
> +        assert good_response in response
> --
> 2.23.0
> 

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20191019/c0123155/attachment.sig>

[U-Boot] [PATCH 04/10] test/py: Automated conversion to Python 3

[Tom Rini]

On Sat, Oct 19, 2019 at 07:25:37AM +0200, Heinrich Schuchardt wrote:
> On 10/18/19 10:53 PM, Tom Rini wrote:
> > Use the 2to3 tool to perform numerous automatic conversions from Python
> > 2 syntax to Python 3.  Also fix whitespace problems that Python 3
> > catches that Python 2 did not.
> > 
> > Signed-off-by: Tom Rini <trini@konsulko.com>
> 
> This patch breaks
> 
> ./test/py/test.py --bd=qemu-arm64 --build-dir=. -k=test_efi_
> 
> For remediation see below.
> 
> > ---
> >   test/py/conftest.py               |  9 ++--
> >   test/py/multiplexed_log.py        |  4 +-
> >   test/py/test.py                   |  4 +-
> >   test/py/tests/test_fit.py         |  2 -
> >   test/py/tests/test_fpga.py        | 46 +++++++++----------
> >   test/py/tests/test_fs/conftest.py |  4 +-
> >   test/py/tests/test_log.py         |  8 ++--
> >   test/py/tests/test_mmc_wr.py      | 74 +++++++++++++++----------------
> >   8 files changed, 71 insertions(+), 80 deletions(-)
> > 
> > diff --git a/test/py/conftest.py b/test/py/conftest.py
> > index 30c898b40a0d..5c19af1d5034 100644
> > --- a/test/py/conftest.py
> > +++ b/test/py/conftest.py
> > @@ -19,13 +19,10 @@ import os.path
> >   import pytest
> >   from _pytest.runner import runtestprotocol
> >   import re
> > -import StringIO
> > +import io
> >   import sys
> > 
> > -try:
> > -    import configparser
> > -except:
> > -    import ConfigParser as configparser
> > +import configparser
> > 
> >   # Globals: The HTML log file, and the connection to the U-Boot console.
> >   log = None
> > @@ -169,7 +166,7 @@ def pytest_configure(config):
> > 
> >           with open(dot_config, 'rt') as f:
> >               ini_str = '[root]\n' + f.read()
> > -            ini_sio = StringIO.StringIO(ini_str)
> > +            ini_sio = io.StringIO(ini_str)
> 
> This results in an error
> 
> INTERNALERROR>   File "/home/user/u-boot/test/py/conftest.py", line 169,
> in pytest_configure
> INTERNALERROR>     ini_sio = io.StringIO(ini_str)
> INTERNALERROR> TypeError: initial_value must be unicode or None, not str
> 
> -            ini_sio = io.StringIO(ini_str)
> +            ini_sio = io.StringIO(ini_str.decode())

With the whole series applied:
INTERNALERROR> AttributeError: 'str' object has no attribute 'decode'

> >               parser = configparser.RawConfigParser()
> >               parser.readfp(ini_sio)
> >               ubconfig.buildconfig.update(parser.items('root'))
> > diff --git a/test/py/multiplexed_log.py b/test/py/multiplexed_log.py
> > index 637a3bd257ba..de0aacc659b8 100644
> > --- a/test/py/multiplexed_log.py
> > +++ b/test/py/multiplexed_log.py
> > @@ -5,8 +5,8 @@
> >   # Generate an HTML-formatted log file containing multiple streams of data,
> >   # each represented in a well-delineated/-structured fashion.
> > 
> > -import cgi
> >   import datetime
> > +import html
> >   import os.path
> >   import shutil
> >   import subprocess
> > @@ -334,7 +334,7 @@ $(document).ready(function () {
> >           data = data.replace(chr(13), '')
> >           data = ''.join((ord(c) in self._nonprint) and ('%%%02x' % ord(c)) or
> >                          c for c in data)
> > -        data = cgi.escape(data)
> > +        data = html.escape(data)
> >           return data
> > 
> >       def _terminate_stream(self):
> > diff --git a/test/py/test.py b/test/py/test.py
> > index a5140945d4b9..0ce1838833f6 100755
> > --- a/test/py/test.py
> > +++ b/test/py/test.py
> > @@ -1,4 +1,4 @@
> > -#!/usr/bin/env python2
> > +#!/usr/bin/env python3
> >   # SPDX-License-Identifier: GPL-2.0
> > 
> >   # Copyright (c) 2015 Stephen Warren
> > @@ -7,8 +7,6 @@
> >   # Wrapper script to invoke pytest with the directory name that contains the
> >   # U-Boot tests.
> > 
> > -from __future__ import print_function
> > -
> >   import os
> >   import os.path
> >   import sys
> > diff --git a/test/py/tests/test_fit.py b/test/py/tests/test_fit.py
> > index e3210ed43fa4..4922b9dcc664 100755
> > --- a/test/py/tests/test_fit.py
> > +++ b/test/py/tests/test_fit.py
> 
> Please, replace all occurrences of
> 
> -            print(base_fdt, file=fd)
> +            fd.write(base_fdt)
> 
> in this file.
> 
> Together with the correction in test/py/conftest.py I now can run
> 
> ./test/py/test.py --bd=qemu-arm64 --build-dir=. -k=test_efi_
> 
> using origin/master + patches 1-4.

Wait, sorry.  You need to have everything applied.  A partial conversion
to python3 doesn't work, there's other changes needed and then show the
EFI problem I hit.

That said, if you re-work what I do in my patch #7 to get our input
generally encoded to fix the "doing str stuff on byte object" problems
and EFI is still happy in the end and we only have the pytest warnings
to fix, I'm fine with that.  I'm not a python expert and am happy to
take the fixes in another direction.  Thanks!

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20191019/bc20a059/attachment.sig>

[U-Boot] [PATCH 01/10] gitlab-ci: Fix indentation in some stanzas

[Simon Glass]

On Fri, 18 Oct 2019 at 14:53, Tom Rini <trini@konsulko.com> wrote:
>
> In a number of our stanzas we had multi-line commands that were one
> space short of alignment, correct this.
>
> Signed-off-by: Tom Rini <trini@konsulko.com>
> ---
>  .gitlab-ci.yml | 40 ++++++++++++++++++++--------------------
>  1 file changed, 20 insertions(+), 20 deletions(-)

Reviewed-by: Simon Glass <sjg@chromium.org>

[U-Boot] [PATCH 02/10] gitlab-ci: Prepend to PATH rather than overwrite it

[Simon Glass]

Hi Tom,

On Fri, 18 Oct 2019 at 14:53, Tom Rini <trini@konsulko.com> wrote:
>
> We need to make sure that we prepend changes to PATH, so that any
> binaries we need to provide ourselves are used not overwrite PATH for

This reads a bit wonkily.

> this.  Overwriting PATH like this breaks the python virtualenv PATH
> changes.
>
> Signed-off-by: Tom Rini <trini@konsulko.com>
> ---
>  .gitlab-ci.yml | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)

Reviewed-by: Simon Glass <sjg@chromium.org>

[U-Boot] [PATCH 03/10] test/py: Fix pytest4 deprecation warnings

[Simon Glass]

Hi Tom,

On Fri, 18 Oct 2019 at 14:53, Tom Rini <trini@konsulko.com> wrote:
>
> From: Marek Vasut <marek.vasut@gmail.com>
>
> Fix the following spit from pytest:
>
> u-boot/test/py/conftest.py:438: RemovedInPytest4Warning: MarkInfo objects are deprecated as they contain merged marks which are hard to deal with correctly.
>   Please use node.get_closest_marker(name) or node.iter_markers(name).
>   Docs: https://docs.pytest.org/en/latest/mark.html#updating-code
>     for board in mark.args:
>
> In both cases, the later suggestion is applicable.
>
> Signed-off-by: Marek Vasut <marek.vasut+renesas@gmail.com>
> Cc: Igor Opaniuk <igor.opaniuk@linaro.org>
> Cc: Tom Rini <trini@konsulko.com>
> Cc: Simon Glass <sjg@chromium.org>
> [trini: Update for current file with a few more cases, un-pin pytest in CI]
> Signed-off-by: Tom Rini <trini@konsulko.com>
> ---
>  .gitlab-ci.yml      |  2 +-
>  .travis.yml         |  2 +-
>  test/py/conftest.py | 30 ++++++++++++------------------
>  3 files changed, 14 insertions(+), 20 deletions(-)

This causes a breakage on my machine even with the rest of the patches applied.

make qcheck
./test/run quick
sandbox: +make O=/scratch/sglass/cosarm/src/third_party/u-boot/files/build-sandbox
-s sandbox_defconfig
+make O=/scratch/sglass/cosarm/src/third_party/u-boot/files/build-sandbox -s -j8
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File
"/usr/lib/python2.7/dist-packages/_pytest/main.py", line 101, in
wrap_session
INTERNALERROR>     config._do_configure()
INTERNALERROR>   File
"/usr/lib/python2.7/dist-packages/_pytest/config.py", line 921, in
_do_configure
INTERNALERROR>
self.hook.pytest_configure.call_historic(kwargs=dict(config=self))
INTERNALERROR>   File
"/usr/lib/python2.7/dist-packages/pluggy/__init__.py", line 628, in
call_historic
INTERNALERROR>     res = self._hookexec(self, self._nonwrappers +
self._wrappers, kwargs)
INTERNALERROR>   File
"/usr/lib/python2.7/dist-packages/pluggy/__init__.py", line 222, in
_hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File
"/usr/lib/python2.7/dist-packages/pluggy/__init__.py", line 216, in
<lambda>
INTERNALERROR>     firstresult=hook.spec_opts.get('firstresult'),
INTERNALERROR>   File
"/usr/lib/python2.7/dist-packages/pluggy/callers.py", line 201, in
_multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File
"/usr/lib/python2.7/dist-packages/pluggy/callers.py", line 77, in
get_result
INTERNALERROR>     _reraise(*ex)  # noqa
INTERNALERROR>   File
"/usr/lib/python2.7/dist-packages/pluggy/callers.py", line 180, in
_multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File
"/scratch/sglass/cosarm/src/third_party/u-boot/files/test/py/conftest.py",
line 169, in pytest_configure
INTERNALERROR>     ini_sio = io.StringIO(ini_str)
INTERNALERROR> TypeError: initial_value must be unicode or None, not str
sandbox_spl: +make
O=/scratch/sglass/cosarm/src/third_party/u-boot/files/build-sandbox_spl
-s sandbox_spl_defconfig
+make O=/scratch/sglass/cosarm/src/third_party/u-boot/files/build-sandbox_spl
-s -j8
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File
"/usr/lib/python2.7/dist-packages/_pytest/main.py", line 101, in
wrap_session
INTERNALERROR>     config._do_configure()
INTERNALERROR>   File
"/usr/lib/python2.7/dist-packages/_pytest/config.py", line 921, in
_do_configure
INTERNALERROR>
self.hook.pytest_configure.call_historic(kwargs=dict(config=self))
INTERNALERROR>   File
"/usr/lib/python2.7/dist-packages/pluggy/__init__.py", line 628, in
call_historic
INTERNALERROR>     res = self._hookexec(self, self._nonwrappers +
self._wrappers, kwargs)
INTERNALERROR>   File
"/usr/lib/python2.7/dist-packages/pluggy/__init__.py", line 222, in
_hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File
"/usr/lib/python2.7/dist-packages/pluggy/__init__.py", line 216, in
<lambda>
INTERNALERROR>     firstresult=hook.spec_opts.get('firstresult'),
INTERNALERROR>   File
"/usr/lib/python2.7/dist-packages/pluggy/callers.py", line 201, in
_multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File
"/usr/lib/python2.7/dist-packages/pluggy/callers.py", line 77, in
get_result
INTERNALERROR>     _reraise(*ex)  # noqa
INTERNALERROR>   File
"/usr/lib/python2.7/dist-packages/pluggy/callers.py", line 180, in
_multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File
"/scratch/sglass/cosarm/src/third_party/u-boot/files/test/py/conftest.py",
line 169, in pytest_configure
INTERNALERROR>     ini_sio = io.StringIO(ini_str)
INTERNALERROR> TypeError: initial_value must be unicode or None, not str
sandbox_flattree: +make
O=/scratch/sglass/cosarm/src/third_party/u-boot/files/build-sandbox_flattree
-s sandbox_flattree_defconfig
+make O=/scratch/sglass/cosarm/src/third_party/u-boot/files/build-sandbox_flattree
-s -j8
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File
"/usr/lib/python2.7/dist-packages/_pytest/main.py", line 101, in
wrap_session
INTERNALERROR>     config._do_configure()
INTERNALERROR>   File
"/usr/lib/python2.7/dist-packages/_pytest/config.py", line 921, in
_do_configure
INTERNALERROR>
self.hook.pytest_configure.call_historic(kwargs=dict(config=self))
INTERNALERROR>   File
"/usr/lib/python2.7/dist-packages/pluggy/__init__.py", line 628, in
call_historic
INTERNALERROR>     res = self._hookexec(self, self._nonwrappers +
self._wrappers, kwargs)
INTERNALERROR>   File
"/usr/lib/python2.7/dist-packages/pluggy/__init__.py", line 222, in
_hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File
"/usr/lib/python2.7/dist-packages/pluggy/__init__.py", line 216, in
<lambda>
INTERNALERROR>     firstresult=hook.spec_opts.get('firstresult'),
INTERNALERROR>   File
"/usr/lib/python2.7/dist-packages/pluggy/callers.py", line 201, in
_multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File
"/usr/lib/python2.7/dist-packages/pluggy/callers.py", line 77, in
get_result
INTERNALERROR>     _reraise(*ex)  # noqa
INTERNALERROR>   File
"/usr/lib/python2.7/dist-packages/pluggy/callers.py", line 180, in
_multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File
"/scratch/sglass/cosarm/src/third_party/u-boot/files/test/py/conftest.py",
line 169, in pytest_configure
INTERNALERROR>     ini_sio = io.StringIO(ini_str)
INTERNALERROR> TypeError: initial_value must be unicode or None, not str


Regards,
Simon

[U-Boot] [PATCH 03/10] test/py: Fix pytest4 deprecation warnings

[Tom Rini]

On Mon, Oct 21, 2019 at 09:25:56AM -0600, Simon Glass wrote:
> Hi Tom,
> 
> On Fri, 18 Oct 2019 at 14:53, Tom Rini <trini@konsulko.com> wrote:
> >
> > From: Marek Vasut <marek.vasut@gmail.com>
> >
> > Fix the following spit from pytest:
> >
> > u-boot/test/py/conftest.py:438: RemovedInPytest4Warning: MarkInfo objects are deprecated as they contain merged marks which are hard to deal with correctly.
> >   Please use node.get_closest_marker(name) or node.iter_markers(name).
> >   Docs: https://docs.pytest.org/en/latest/mark.html#updating-code
> >     for board in mark.args:
> >
> > In both cases, the later suggestion is applicable.
> >
> > Signed-off-by: Marek Vasut <marek.vasut+renesas@gmail.com>
> > Cc: Igor Opaniuk <igor.opaniuk@linaro.org>
> > Cc: Tom Rini <trini@konsulko.com>
> > Cc: Simon Glass <sjg@chromium.org>
> > [trini: Update for current file with a few more cases, un-pin pytest in CI]
> > Signed-off-by: Tom Rini <trini@konsulko.com>
> > ---
> >  .gitlab-ci.yml      |  2 +-
> >  .travis.yml         |  2 +-
> >  test/py/conftest.py | 30 ++++++++++++------------------
> >  3 files changed, 14 insertions(+), 20 deletions(-)
> 
> This causes a breakage on my machine even with the rest of the patches applied.
> 
> make qcheck
> ./test/run quick
> sandbox: +make O=/scratch/sglass/cosarm/src/third_party/u-boot/files/build-sandbox
> -s sandbox_defconfig
> +make O=/scratch/sglass/cosarm/src/third_party/u-boot/files/build-sandbox -s -j8
> INTERNALERROR> Traceback (most recent call last):
> INTERNALERROR>   File
> "/usr/lib/python2.7/dist-packages/_pytest/main.py", line 101, in
> wrap_session

As I've promised Heinrich as well, I will update the README.md file for
the tests.  That's invoking python2-pytest so it's not going to work.
You need to either pip3 (or virtualenv -p /usr/bin/python3) install
pytest to get the python3 version.

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20191021/091482be/attachment.sig>

[U-Boot] [PATCHv2 00/13] Moving test/py to Python 3

[Tom Rini]

Hey all,

So, here's the second pass at moving our test/py code over to Python 3.
This time, I think it's ready to merge as I've resovled the WORKAROUND /
HACK patches.  In terms of what I was calling a WORKAROUND, I'm
comfortable saying this is a feature / intentional change in how and
when we instantiate a virtualenv for python and use python3 there.  The
HACK patches have been dropped and we now have the same or better
test.py coverage in CI (test_fs tests are still skipped in Travis, but
run in GitLab so I'm going to put fixing Travis on the TODO list).

The first two patches can go in any time and I will
probably pull them soon as well.  The next patch is an old one from
Marek that I deferred at the time, but it's time now so I've updated it
slightly to cover the new cases of the same code he fixed before.  For
migrating the pytest code itself, I've left the conversion split in a
few commits.  This isn't bisectable but certainly for review it's
better.  I'm honestly not sure if it's more important for history to
keep this bisectable or to have the changes broken out like this.

In terms of changes from v1 to v2, as noted above I've fixed the test_fs
tests so that they run.  I've also fixed the EFI tests (which was
failing due to regex strings needing to be raw strings now), and fix
that same issue in other places where it was non-fatal.  I've also
introduced another patch that has a pip "requirements.txt" file that
describes all of the python modules that our tests require and the
versions there-of.  This will help make things more reproducible long
term and is something that needs to be refreshed from time to time.  The
other big change is that I've changed how test/py/test.py (which
currently is a wrapper around calling py.test) so that it instead calls
the pytest module directly (which is what py.test does).  This will
eliminate the problem of "python2-pytest was invoked, everything failed"
as test/py/test.py is using python3.  It still provides a user-friendly
error when pytest is not available.

-- 
Tom

[U-Boot] [PATCHv2 01/13] gitlab-ci: Fix indentation in some stanzas

[Tom Rini]

In a number of our stanzas we had multi-line commands that were one
space short of alignment, correct this.

Reviewed-by: Stephen Warren <swarren@nvidia.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
Signed-off-by: Tom Rini <trini@konsulko.com>
---
 .gitlab-ci.yml | 40 ++++++++++++++++++++--------------------
 1 file changed, 20 insertions(+), 20 deletions(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 967abed9f250..b91b5f67af98 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -65,11 +65,11 @@ build all 32bit ARM platforms:
   stage: world build
   script:
     - ret=0;
-     ./tools/buildman/buildman -o /tmp -P -E arm -x aarch64 || ret=$?;
-     if [[ $ret -ne 0 && $ret -ne 129 ]]; then
-       ./tools/buildman/buildman -o /tmp -sdeP;
-       exit $ret;
-     fi;
+      ./tools/buildman/buildman -o /tmp -P -E arm -x aarch64 || ret=$?;
+      if [[ $ret -ne 0 && $ret -ne 129 ]]; then
+        ./tools/buildman/buildman -o /tmp -sdeP;
+        exit $ret;
+      fi;
 
 build all 64bit ARM platforms:
   tags: [ 'all' ]
@@ -79,33 +79,33 @@ build all 64bit ARM platforms:
     - . /tmp/venv/bin/activate
     - pip install pyelftools
     - ret=0;
-     ./tools/buildman/buildman -o /tmp -P -E aarch64 || ret=$?;
-     if [[ $ret -ne 0 && $ret -ne 129 ]]; then
-       ./tools/buildman/buildman -o /tmp -sdeP;
-       exit $ret;
-     fi;
+      ./tools/buildman/buildman -o /tmp -P -E aarch64 || ret=$?;
+      if [[ $ret -ne 0 && $ret -ne 129 ]]; then
+        ./tools/buildman/buildman -o /tmp -sdeP;
+        exit $ret;
+      fi;
 
 build all PowerPC platforms:
   tags: [ 'all' ]
   stage: world build
   script:
     - ret=0;
-     ./tools/buildman/buildman -o /tmp -P -E powerpc || ret=$?;
-     if [[ $ret -ne 0 && $ret -ne 129 ]]; then
-       ./tools/buildman/buildman -o /tmp -sdeP;
-       exit $ret;
-     fi;
+      ./tools/buildman/buildman -o /tmp -P -E powerpc || ret=$?;
+      if [[ $ret -ne 0 && $ret -ne 129 ]]; then
+        ./tools/buildman/buildman -o /tmp -sdeP;
+        exit $ret;
+      fi;
 
 build all other platforms:
   tags: [ 'all' ]
   stage: world build
   script:
     - ret=0;
-     ./tools/buildman/buildman -o /tmp -P -E -x arm,powerpc || ret=$?;
-     if [[ $ret -ne 0 && $ret -ne 129 ]]; then
-       ./tools/buildman/buildman -o /tmp -sdeP;
-       exit $ret;
-     fi;
+      ./tools/buildman/buildman -o /tmp -P -E -x arm,powerpc || ret=$?;
+      if [[ $ret -ne 0 && $ret -ne 129 ]]; then
+        ./tools/buildman/buildman -o /tmp -sdeP;
+        exit $ret;
+      fi;
 
 # QA jobs for code analytics
 # static code analysis with cppcheck (we can add --enable=all later)
-- 
2.17.1

[U-Boot] [PATCHv2 02/13] gitlab-ci: Prepend to PATH rather than replace it

[Tom Rini]

Currently we set the entire PATH rather than prepend the new paths that
we need to have searched.  This however breaks parts of the "virtualenv"
that was have set up and need to use as that also will be modifying
PATH.  To fix this, prepend our new locations instead.

Reviewed-by: Stephen Warren <swarren@nvidia.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
Signed-off-by: Tom Rini <trini@konsulko.com>
---
Changes in v2:
- Reword slightly
---
 .gitlab-ci.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index b91b5f67af98..ab294997e463 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -48,7 +48,7 @@ stages:
     # "-k something" even when $TEST_PY_TEST_SPEC doesnt need a custom
     # value.
     - export UBOOT_TRAVIS_BUILD_DIR=/tmp/.bm-work/${TEST_PY_BD};
-      export PATH=/opt/qemu/bin:/tmp/uboot-test-hooks/bin:/usr/bin:/bin;
+      export PATH=/opt/qemu/bin:/tmp/uboot-test-hooks/bin:${PATH};
       export PYTHONPATH=/tmp/uboot-test-hooks/py/travis-ci;
       if [[ "${TEST_PY_BD}" != "" ]]; then
         ./test/py/test.py --bd ${TEST_PY_BD} ${TEST_PY_ID}
-- 
2.17.1

[U-Boot] [PATCHv2 03/13] test/py: Split mark to multiple lines

[Tom Rini]

We inconsistently note multiple dependencies today in our tests,
sometimes with a single line that declares multiple and sometimes
multiple single lines.  Current pytest seems to fail on the single line
format so change to multiple declarations.

Reviewed-by: Stephen Warren <swarren@nvidia.com>
Signed-off-by: Tom Rini <trini@konsulko.com>
---
 test/py/tests/test_android/test_avb.py | 9 ++++++---
 test/py/tests/test_mmc_wr.py           | 4 +++-
 2 files changed, 9 insertions(+), 4 deletions(-)

diff --git a/test/py/tests/test_android/test_avb.py b/test/py/tests/test_android/test_avb.py
index 813242343555..20ccaf6712f2 100644
--- a/test/py/tests/test_android/test_avb.py
+++ b/test/py/tests/test_android/test_avb.py
@@ -23,7 +23,8 @@ mmc_dev = 1
 temp_addr = 0x90000000
 temp_addr2 = 0x90002000
 
- at pytest.mark.buildconfigspec('cmd_avb', 'cmd_mmc')
+ at pytest.mark.buildconfigspec('cmd_avb')
+ at pytest.mark.buildconfigspec('cmd_mmc')
 def test_avb_verify(u_boot_console):
     """Run AVB 2.0 boot verification chain with avb subset of commands
     """
@@ -36,7 +37,8 @@ def test_avb_verify(u_boot_console):
     assert response.find(success_str)
 
 
- at pytest.mark.buildconfigspec('cmd_avb', 'cmd_mmc')
+ at pytest.mark.buildconfigspec('cmd_avb')
+ at pytest.mark.buildconfigspec('cmd_mmc')
 def test_avb_mmc_uuid(u_boot_console):
     """Check if 'avb get_uuid' works, compare results with
     'part list mmc 1' output
@@ -93,7 +95,8 @@ def test_avb_is_unlocked(u_boot_console):
     assert response == 'Unlocked = 1'
 
 
- at pytest.mark.buildconfigspec('cmd_avb', 'cmd_mmc')
+ at pytest.mark.buildconfigspec('cmd_avb')
+ at pytest.mark.buildconfigspec('cmd_mmc')
 def test_avb_mmc_read(u_boot_console):
     """Test mmc read operation
     """
diff --git a/test/py/tests/test_mmc_wr.py b/test/py/tests/test_mmc_wr.py
index 8b18781eac74..ab4c57974421 100644
--- a/test/py/tests/test_mmc_wr.py
+++ b/test/py/tests/test_mmc_wr.py
@@ -35,7 +35,9 @@ env__mmc_wr_configs = (
 
 """
 
- at pytest.mark.buildconfigspec('cmd_mmc','cmd_memory', 'cmd_random')
+ at pytest.mark.buildconfigspec('cmd_mmc')
+ at pytest.mark.buildconfigspec('cmd_memory')
+ at pytest.mark.buildconfigspec('cmd_random')
 def test_mmc_wr(u_boot_console, env__mmc_wr_config):
     """Test the "mmc write" command.
 
-- 
2.17.1

[U-Boot] [PATCHv2 04/13] test/py: Fix pytest4 deprecation warnings

[Tom Rini]

From: Marek Vasut <marek.vasut@gmail.com>

Fix the following spit from pytest:

u-boot/test/py/conftest.py:438: RemovedInPytest4Warning: MarkInfo objects are deprecated as they contain merged marks which are hard to deal with correctly.
  Please use node.get_closest_marker(name) or node.iter_markers(name).
  Docs: https://docs.pytest.org/en/latest/mark.html#updating-code
    for board in mark.args:

In both cases, the later suggestion is applicable.

Reviewed-by: Stephen Warren <swarren@nvidia.com>
Signed-off-by: Marek Vasut <marek.vasut+renesas@gmail.com>
Cc: Igor Opaniuk <igor.opaniuk@linaro.org>
Cc: Tom Rini <trini@konsulko.com>
Cc: Simon Glass <sjg@chromium.org>
[trini: Update for current file with a few more cases, un-pin pytest in CI]
Signed-off-by: Tom Rini <trini@konsulko.com>
---
 .gitlab-ci.yml      |  2 +-
 .travis.yml         |  2 +-
 test/py/conftest.py | 30 ++++++++++++------------------
 3 files changed, 14 insertions(+), 20 deletions(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index ab294997e463..5a34321570ce 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -20,7 +20,7 @@ stages:
     - ln -s travis-ci /tmp/uboot-test-hooks/py/`hostname`
     - virtualenv /tmp/venv
     - . /tmp/venv/bin/activate
-    - pip install pytest==2.8.7
+    - pip install pytest
     - pip install python-subunit
     - pip install coverage
     - grub-mkimage --prefix="" -o ~/grub_x86.efi -O i386-efi normal  echo lsefimmap lsefi lsefisystab efinet tftp minicmd
diff --git a/.travis.yml b/.travis.yml
index a3e7451bcb17..f61278f9656a 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -49,7 +49,7 @@ install:
  - cat ~/.buildman
  - virtualenv /tmp/venv
  - . /tmp/venv/bin/activate
- - pip install pytest==2.8.7
+ - pip install pytest
  - pip install python-subunit
  - pip install pyelftools
  - grub-mkimage --prefix="" -o ~/grub_x86.efi -O i386-efi normal  echo lsefimmap lsefi lsefisystab efinet tftp minicmd
diff --git a/test/py/conftest.py b/test/py/conftest.py
index 00d8ef8ba99c..30c898b40a0d 100644
--- a/test/py/conftest.py
+++ b/test/py/conftest.py
@@ -431,11 +431,9 @@ def setup_boardspec(item):
         Nothing.
     """
 
-    mark = item.get_marker('boardspec')
-    if not mark:
-        return
     required_boards = []
-    for board in mark.args:
+    for boards in item.iter_markers('boardspec'):
+        board = boards.args[0]
         if board.startswith('!'):
             if ubconfig.board_type == board[1:]:
                 pytest.skip('board "%s" not supported' % ubconfig.board_type)
@@ -459,16 +457,14 @@ def setup_buildconfigspec(item):
         Nothing.
     """
 
-    mark = item.get_marker('buildconfigspec')
-    if mark:
-        for option in mark.args:
-            if not ubconfig.buildconfig.get('config_' + option.lower(), None):
-                pytest.skip('.config feature "%s" not enabled' % option.lower())
-    notmark = item.get_marker('notbuildconfigspec')
-    if notmark:
-        for option in notmark.args:
-            if ubconfig.buildconfig.get('config_' + option.lower(), None):
-                pytest.skip('.config feature "%s" enabled' % option.lower())
+    for options in item.iter_markers('buildconfigspec'):
+        option = options.args[0]
+        if not ubconfig.buildconfig.get('config_' + option.lower(), None):
+            pytest.skip('.config feature "%s" not enabled' % option.lower())
+    for option in item.iter_markers('notbuildconfigspec'):
+        option = options.args[0]
+        if ubconfig.buildconfig.get('config_' + option.lower(), None):
+            pytest.skip('.config feature "%s" enabled' % option.lower())
 
 def tool_is_in_path(tool):
     for path in os.environ["PATH"].split(os.pathsep):
@@ -491,10 +487,8 @@ def setup_requiredtool(item):
         Nothing.
     """
 
-    mark = item.get_marker('requiredtool')
-    if not mark:
-        return
-    for tool in mark.args:
+    for tools in item.iter_markers('requiredtool'):
+        tool = tools.args[0]
         if not tool_is_in_path(tool):
             pytest.skip('tool "%s" not in $PATH' % tool)
 
-- 
2.17.1

[U-Boot] [PATCHv2 05/13] test/py: Automated conversion to Python 3

[Tom Rini]

Use the 2to3 tool to perform numerous automatic conversions from Python
2 syntax to Python 3.  Also fix whitespace problems that Python 3
catches that Python 2 did not.

Reviewed-by: Stephen Warren <swarren@nvidia.com>
Signed-off-by: Tom Rini <trini@konsulko.com>
---
 test/py/conftest.py               |  9 ++--
 test/py/multiplexed_log.py        |  4 +-
 test/py/test.py                   |  4 +-
 test/py/tests/test_fit.py         |  2 -
 test/py/tests/test_fpga.py        | 46 +++++++++----------
 test/py/tests/test_fs/conftest.py |  4 +-
 test/py/tests/test_log.py         |  8 ++--
 test/py/tests/test_mmc_wr.py      | 74 +++++++++++++++----------------
 8 files changed, 71 insertions(+), 80 deletions(-)

diff --git a/test/py/conftest.py b/test/py/conftest.py
index 30c898b40a0d..5c19af1d5034 100644
--- a/test/py/conftest.py
+++ b/test/py/conftest.py
@@ -19,13 +19,10 @@ import os.path
 import pytest
 from _pytest.runner import runtestprotocol
 import re
-import StringIO
+import io
 import sys
 
-try:
-    import configparser
-except:
-    import ConfigParser as configparser
+import configparser
 
 # Globals: The HTML log file, and the connection to the U-Boot console.
 log = None
@@ -169,7 +166,7 @@ def pytest_configure(config):
 
         with open(dot_config, 'rt') as f:
             ini_str = '[root]\n' + f.read()
-            ini_sio = StringIO.StringIO(ini_str)
+            ini_sio = io.StringIO(ini_str)
             parser = configparser.RawConfigParser()
             parser.readfp(ini_sio)
             ubconfig.buildconfig.update(parser.items('root'))
diff --git a/test/py/multiplexed_log.py b/test/py/multiplexed_log.py
index 637a3bd257ba..de0aacc659b8 100644
--- a/test/py/multiplexed_log.py
+++ b/test/py/multiplexed_log.py
@@ -5,8 +5,8 @@
 # Generate an HTML-formatted log file containing multiple streams of data,
 # each represented in a well-delineated/-structured fashion.
 
-import cgi
 import datetime
+import html
 import os.path
 import shutil
 import subprocess
@@ -334,7 +334,7 @@ $(document).ready(function () {
         data = data.replace(chr(13), '')
         data = ''.join((ord(c) in self._nonprint) and ('%%%02x' % ord(c)) or
                        c for c in data)
-        data = cgi.escape(data)
+        data = html.escape(data)
         return data
 
     def _terminate_stream(self):
diff --git a/test/py/test.py b/test/py/test.py
index a5140945d4b9..0ce1838833f6 100755
--- a/test/py/test.py
+++ b/test/py/test.py
@@ -1,4 +1,4 @@
-#!/usr/bin/env python2
+#!/usr/bin/env python3
 # SPDX-License-Identifier: GPL-2.0
 
 # Copyright (c) 2015 Stephen Warren
@@ -7,8 +7,6 @@
 # Wrapper script to invoke pytest with the directory name that contains the
 # U-Boot tests.
 
-from __future__ import print_function
-
 import os
 import os.path
 import sys
diff --git a/test/py/tests/test_fit.py b/test/py/tests/test_fit.py
index e3210ed43fa4..4922b9dcc664 100755
--- a/test/py/tests/test_fit.py
+++ b/test/py/tests/test_fit.py
@@ -3,8 +3,6 @@
 #
 # Sanity check of the FIT handling in U-Boot
 
-from __future__ import print_function
-
 import os
 import pytest
 import struct
diff --git a/test/py/tests/test_fpga.py b/test/py/tests/test_fpga.py
index e3bb7b41c749..ca7ef8ea40d6 100644
--- a/test/py/tests/test_fpga.py
+++ b/test/py/tests/test_fpga.py
@@ -175,29 +175,29 @@ def test_fpga_load_fail(u_boot_console):
     f, dev, addr, bit, bit_size = load_file_from_var(u_boot_console, 'bitstream_load')
 
     for cmd in ['dump', 'load', 'loadb']:
-	# missing dev parameter
-	expected = 'fpga: incorrect parameters passed'
-	output = u_boot_console.run_command('fpga %s %x $filesize' % (cmd, addr))
-	#assert expected in output
-	assert expected_usage in output
-
-	# more parameters - 0 at the end
-	expected = 'fpga: more parameters passed'
-	output = u_boot_console.run_command('fpga %s %x %x $filesize 0' % (cmd, dev, addr))
-	#assert expected in output
-	assert expected_usage in output
-
-	# 0 address
-	expected = 'fpga: zero fpga_data address'
-	output = u_boot_console.run_command('fpga %s %x 0 $filesize' % (cmd, dev))
-	#assert expected in output
-	assert expected_usage in output
-
-	# 0 filesize
-	expected = 'fpga: zero size'
-	output = u_boot_console.run_command('fpga %s %x %x 0' % (cmd, dev, addr))
-	#assert expected in output
-	assert expected_usage in output
+        # missing dev parameter
+        expected = 'fpga: incorrect parameters passed'
+        output = u_boot_console.run_command('fpga %s %x $filesize' % (cmd, addr))
+        #assert expected in output
+        assert expected_usage in output
+
+        # more parameters - 0 at the end
+        expected = 'fpga: more parameters passed'
+        output = u_boot_console.run_command('fpga %s %x %x $filesize 0' % (cmd, dev, addr))
+        #assert expected in output
+        assert expected_usage in output
+
+        # 0 address
+        expected = 'fpga: zero fpga_data address'
+        output = u_boot_console.run_command('fpga %s %x 0 $filesize' % (cmd, dev))
+        #assert expected in output
+        assert expected_usage in output
+
+        # 0 filesize
+        expected = 'fpga: zero size'
+        output = u_boot_console.run_command('fpga %s %x %x 0' % (cmd, dev, addr))
+        #assert expected in output
+        assert expected_usage in output
 
 @pytest.mark.buildconfigspec('cmd_fpga')
 @pytest.mark.buildconfigspec('cmd_echo')
diff --git a/test/py/tests/test_fs/conftest.py b/test/py/tests/test_fs/conftest.py
index 9324657d2160..354d17672fe4 100644
--- a/test/py/tests/test_fs/conftest.py
+++ b/test/py/tests/test_fs/conftest.py
@@ -508,8 +508,8 @@ def fs_obj_unlink(request, u_boot_config):
 
         # Test Case 2
         check_call('mkdir %s/dir2' % mount_dir, shell=True)
-	for i in range(0, 20):
-	    check_call('mkdir %s/dir2/0123456789abcdef%02x'
+        for i in range(0, 20):
+            check_call('mkdir %s/dir2/0123456789abcdef%02x'
                                     % (mount_dir, i), shell=True)
 
         # Test Case 4
diff --git a/test/py/tests/test_log.py b/test/py/tests/test_log.py
index cb183444c6fc..75325fad61ad 100644
--- a/test/py/tests/test_log.py
+++ b/test/py/tests/test_log.py
@@ -27,9 +27,9 @@ def test_log(u_boot_console):
         """
         for i in range(max_level):
             if mask & 1:
-                assert 'log_run() log %d' % i == lines.next()
+                assert 'log_run() log %d' % i == next(lines)
             if mask & 3:
-                assert 'func() _log %d' % i == lines.next()
+                assert 'func() _log %d' % i == next(lines)
 
     def run_test(testnum):
         """Run a particular test number (the 'log test' command)
@@ -43,7 +43,7 @@ def test_log(u_boot_console):
            output = u_boot_console.run_command('log test %d' % testnum)
         split = output.replace('\r', '').splitlines()
         lines = iter(split)
-        assert 'test %d' % testnum == lines.next()
+        assert 'test %d' % testnum == next(lines)
         return lines
 
     def test0():
@@ -88,7 +88,7 @@ def test_log(u_boot_console):
     def test10():
         lines = run_test(10)
         for i in range(7):
-            assert 'log_test() level %d' % i == lines.next()
+            assert 'log_test() level %d' % i == next(lines)
 
     # TODO(sjg at chromium.org): Consider structuring this as separate tests
     cons = u_boot_console
diff --git a/test/py/tests/test_mmc_wr.py b/test/py/tests/test_mmc_wr.py
index ab4c57974421..05e5c1ee85da 100644
--- a/test/py/tests/test_mmc_wr.py
+++ b/test/py/tests/test_mmc_wr.py
@@ -67,41 +67,39 @@ def test_mmc_wr(u_boot_console, env__mmc_wr_config):
 
 
     for i in range(test_iterations):
-	# Generate random data
-	cmd = 'random %s %x' % (src_addr, count_bytes)
-	response = u_boot_console.run_command(cmd)
-	good_response = '%d bytes filled with random data' % (count_bytes)
-	assert good_response in response
-
-	# Select MMC device
-	cmd = 'mmc dev %d' % devid
-	if is_emmc:
-		cmd += ' %d' % partid
-	response = u_boot_console.run_command(cmd)
-	assert 'no card present' not in response
-	if is_emmc:
-		partid_response = "(part %d)" % partid
-	else:
-		partid_response = ""
-	good_response = 'mmc%d%s is current device' % (devid, partid_response)
-	assert good_response in response
-
-	# Write data
-	cmd = 'mmc write %s %x %x' % (src_addr, sector, count_sectors)
-	response = u_boot_console.run_command(cmd)
-	good_response = 'MMC write: dev # %d, block # %d, count %d ... %d blocks written: OK' % (
-		devid, sector, count_sectors, count_sectors)
-	assert good_response in response
-
-	# Read data
-	cmd = 'mmc read %s %x %x' % (dst_addr, sector, count_sectors)
-	response = u_boot_console.run_command(cmd)
-	good_response = 'MMC read: dev # %d, block # %d, count %d ... %d blocks read: OK' % (
-		devid, sector, count_sectors, count_sectors)
-	assert good_response in response
-
-	# Compare src and dst data
-	cmd = 'cmp.b %s %s %x' % (src_addr, dst_addr, count_bytes)
-	response = u_boot_console.run_command(cmd)
-	good_response = 'Total of %d byte(s) were the same' % (count_bytes)
-	assert good_response in response
+        # Generate random data
+        cmd = 'random %s %x' % (src_addr, count_bytes)
+        response = u_boot_console.run_command(cmd)
+        good_response = '%d bytes filled with random data' % (count_bytes)
+        assert good_response in response
+
+        # Select MMC device
+        cmd = 'mmc dev %d' % devid
+        if is_emmc:
+            cmd += ' %d' % partid
+        response = u_boot_console.run_command(cmd)
+        assert 'no card present' not in response
+        if is_emmc:
+            partid_response = "(part %d)" % partid
+        else:
+            partid_response = ""
+        good_response = 'mmc%d%s is current device' % (devid, partid_response)
+        assert good_response in response
+
+        # Write data
+        cmd = 'mmc write %s %x %x' % (src_addr, sector, count_sectors)
+        response = u_boot_console.run_command(cmd)
+        good_response = 'MMC write: dev # %d, block # %d, count %d ... %d blocks written: OK' % (devid, sector, count_sectors, count_sectors)
+        assert good_response in response
+
+        # Read data
+        cmd = 'mmc read %s %x %x' % (dst_addr, sector, count_sectors)
+        response = u_boot_console.run_command(cmd)
+        good_response = 'MMC read: dev # %d, block # %d, count %d ... %d blocks read: OK' % (devid, sector, count_sectors, count_sectors)
+        assert good_response in response
+
+        # Compare src and dst data
+        cmd = 'cmp.b %s %s %x' % (src_addr, dst_addr, count_bytes)
+        response = u_boot_console.run_command(cmd)
+        good_response = 'Total of %d byte(s) were the same' % (count_bytes)
+        assert good_response in response
-- 
2.17.1

[U-Boot] [PATCHv2 06/13] test/py: Manual python3 fixes

[Tom Rini]

- Modern pytest is more visible in telling us about parameters that we
  had not described, so describe a few more.
- ConfigParser.readfp(...) is now configparser.read_file(...)
- As part of the "strings vs bytes" conversions in Python 3, we use the
  default encoding/decoding of utf-8 but in some places tell Python to
  replace problematic conversions rather than throw a fatal error.
- Fix a typo noticed while doing the above ("tot he" -> "to the").
- As suggested by Stephen, re-alphabetize the import list
- Per Heinrich, replace how we write contents in test_fit.py

Signed-off-by: Tom Rini <trini@konsulko.com>
---
 test/py/conftest.py        | 9 ++++-----
 test/py/multiplexed_log.py | 9 +++++----
 test/py/pytest.ini         | 3 +++
 test/py/tests/test_fit.py  | 4 ++--
 test/py/u_boot_spawn.py    | 4 ++--
 5 files changed, 16 insertions(+), 13 deletions(-)

diff --git a/test/py/conftest.py b/test/py/conftest.py
index 5c19af1d5034..bffee6b8a3a2 100644
--- a/test/py/conftest.py
+++ b/test/py/conftest.py
@@ -13,17 +13,16 @@
 # - Implementing custom pytest markers.
 
 import atexit
+import configparser
 import errno
+import io
 import os
 import os.path
 import pytest
-from _pytest.runner import runtestprotocol
 import re
-import io
+from _pytest.runner import runtestprotocol
 import sys
 
-import configparser
-
 # Globals: The HTML log file, and the connection to the U-Boot console.
 log = None
 console = None
@@ -168,7 +167,7 @@ def pytest_configure(config):
             ini_str = '[root]\n' + f.read()
             ini_sio = io.StringIO(ini_str)
             parser = configparser.RawConfigParser()
-            parser.readfp(ini_sio)
+            parser.read_file(ini_sio)
             ubconfig.buildconfig.update(parser.items('root'))
 
     ubconfig.test_py_dir = test_py_dir
diff --git a/test/py/multiplexed_log.py b/test/py/multiplexed_log.py
index de0aacc659b8..5ea4c055e072 100644
--- a/test/py/multiplexed_log.py
+++ b/test/py/multiplexed_log.py
@@ -51,7 +51,7 @@ class LogfileStream(object):
         """Write data to the log stream.
 
         Args:
-            data: The data to write tot he file.
+            data: The data to write to the file.
             implicit: Boolean indicating whether data actually appeared in the
                 stream, or was implicitly generated. A valid use-case is to
                 repeat a shell prompt at the start of each separate log
@@ -64,7 +64,8 @@ class LogfileStream(object):
 
         self.logfile.write(self, data, implicit)
         if self.chained_file:
-            self.chained_file.write(data)
+            # Chained file is console, convert things a little
+            self.chained_file.write((data.encode("ascii", "replace")).decode())
 
     def flush(self):
         """Flush the log stream, to ensure correct log interleaving.
@@ -133,7 +134,7 @@ class RunAndLog(object):
         self.logfile.write(self, msg)
 
         try:
-            p = subprocess.Popen(cmd, cwd=cwd,
+            p = subprocess.Popen(cmd, cwd=cwd, encoding="utf-8",
                 stdin=None, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
             (stdout, stderr) = p.communicate()
             output = ''
@@ -215,7 +216,7 @@ class Logfile(object):
             Nothing.
         """
 
-        self.f = open(fn, 'wt')
+        self.f = open(fn, 'wt', encoding="utf-8")
         self.last_stream = None
         self.blocks = []
         self.cur_evt = 1
diff --git a/test/py/pytest.ini b/test/py/pytest.ini
index 7e400682bf25..e93d010f1fa2 100644
--- a/test/py/pytest.ini
+++ b/test/py/pytest.ini
@@ -8,3 +8,6 @@
 markers =
     boardspec: U-Boot: Describes the set of boards a test can/can't run on.
     buildconfigspec: U-Boot: Describes Kconfig/config-header constraints.
+    notbuildconfigspec: U-Boot: Describes required disabled Kconfig options.
+    requiredtool: U-Boot: Required host tools for a test.
+    slow: U-Boot: Specific test will run slowly.
diff --git a/test/py/tests/test_fit.py b/test/py/tests/test_fit.py
index 4922b9dcc664..356d9a20f299 100755
--- a/test/py/tests/test_fit.py
+++ b/test/py/tests/test_fit.py
@@ -153,7 +153,7 @@ def test_fit(u_boot_console):
         src = make_fname('u-boot.dts')
         dtb = make_fname('u-boot.dtb')
         with open(src, 'w') as fd:
-            print(base_fdt, file=fd)
+            fd.write(base_fdt)
         util.run_and_log(cons, ['dtc', src, '-O', 'dtb', '-o', dtb])
         return dtb
 
@@ -186,7 +186,7 @@ def test_fit(u_boot_console):
         its = make_its(params)
         util.run_and_log(cons, [mkimage, '-f', its, fit])
         with open(make_fname('u-boot.dts'), 'w') as fd:
-            print(base_fdt, file=fd)
+            fd.write(base_fdt)
         return fit
 
     def make_kernel(filename, text):
diff --git a/test/py/u_boot_spawn.py b/test/py/u_boot_spawn.py
index b011a3e3da25..7fedb18d2790 100644
--- a/test/py/u_boot_spawn.py
+++ b/test/py/u_boot_spawn.py
@@ -113,7 +113,7 @@ class Spawn(object):
             Nothing.
         """
 
-        os.write(self.fd, data)
+        os.write(self.fd, data.encode(errors="replace"))
 
     def expect(self, patterns):
         """Wait for the sub-process to emit specific data.
@@ -171,7 +171,7 @@ class Spawn(object):
                 events = self.poll.poll(poll_maxwait)
                 if not events:
                     raise Timeout()
-                c = os.read(self.fd, 1024)
+                c = os.read(self.fd, 1024).decode(errors="replace")
                 if not c:
                     raise EOFError()
                 if self.logfile_read:
-- 
2.17.1

[U-Boot] [PATCHv2 07/13] test/py: test_ut.py: Ensure we use bytes

[Tom Rini]

In the case of some unit tests we are working with providing a fake
flash device that we have written some text strings in to.  In this case
we want to tell Python to encode things to bytes for us.

Reviewed-by: Stephen Warren <swarren@nvidia.com>
Signed-off-by: Tom Rini <trini@konsulko.com>
---
 test/py/tests/test_ut.py | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/test/py/tests/test_ut.py b/test/py/tests/test_ut.py
index 62037d2c4530..6c7b8dd2b30e 100644
--- a/test/py/tests/test_ut.py
+++ b/test/py/tests/test_ut.py
@@ -10,14 +10,14 @@ def test_ut_dm_init(u_boot_console):
 
     fn = u_boot_console.config.source_dir + '/testflash.bin'
     if not os.path.exists(fn):
-        data = 'this is a test'
-        data += '\x00' * ((4 * 1024 * 1024) - len(data))
+        data = b'this is a test'
+        data += b'\x00' * ((4 * 1024 * 1024) - len(data))
         with open(fn, 'wb') as fh:
             fh.write(data)
 
     fn = u_boot_console.config.source_dir + '/spi.bin'
     if not os.path.exists(fn):
-        data = '\x00' * (2 * 1024 * 1024)
+        data = b'\x00' * (2 * 1024 * 1024)
         with open(fn, 'wb') as fh:
             fh.write(data)
 
-- 
2.17.1

[U-Boot] [PATCHv2 08/13] test/py: test_efi_selftest.py: Updates for python 3 support

[Tom Rini]

- In python 3 you must use raw strings for regex as other forms are
  deprecated and would require further changes to the pattern here.
  In one case this lets us have a simpler match pattern.
- As strings are now Unicode our complex tests (Euro symbol,
  SHIFT+ALT+FN 5) we need to declare that as a bytes string and then
  decode it for use.

Cc: Heinrich Schuchardt <xypron.glpk@gmx.de>
Signed-off-by: Tom Rini <trini@konsulko.com>
---
Changes in v2:
- New patch
---
 test/py/tests/test_efi_selftest.py | 36 +++++++++++++++---------------
 1 file changed, 18 insertions(+), 18 deletions(-)

diff --git a/test/py/tests/test_efi_selftest.py b/test/py/tests/test_efi_selftest.py
index d5430f9c1275..ca01542088f1 100644
--- a/test/py/tests/test_efi_selftest.py
+++ b/test/py/tests/test_efi_selftest.py
@@ -59,7 +59,7 @@ def test_efi_selftest_text_input(u_boot_console):
 	u_boot_console.run_command(cmd='setenv efi_selftest text input')
 	output = u_boot_console.run_command(cmd='bootefi selftest',
 					    wait_for_prompt=False)
-	m = u_boot_console.p.expect(['To terminate type \'x\''])
+	m = u_boot_console.p.expect([r'To terminate type \'x\''])
 	if m != 0:
 		raise Exception('No prompt for \'text input\' test')
 	u_boot_console.drain_console()
@@ -68,7 +68,7 @@ def test_efi_selftest_text_input(u_boot_console):
 	u_boot_console.run_command(cmd=chr(4), wait_for_echo=False,
 				   send_nl=False, wait_for_prompt=False)
 	m = u_boot_console.p.expect(
-		['Unicode char 4 \(unknown\), scan code 0 \(Null\)'])
+		[r'Unicode char 4 \(unknown\), scan code 0 \(Null\)'])
 	if m != 0:
 		raise Exception('EOT failed in \'text input\' test')
 	u_boot_console.drain_console()
@@ -76,7 +76,7 @@ def test_efi_selftest_text_input(u_boot_console):
 	u_boot_console.run_command(cmd=chr(8), wait_for_echo=False,
 				   send_nl=False, wait_for_prompt=False)
 	m = u_boot_console.p.expect(
-		['Unicode char 8 \(BS\), scan code 0 \(Null\)'])
+		[r'Unicode char 8 \(BS\), scan code 0 \(Null\)'])
 	if m != 0:
 		raise Exception('BS failed in \'text input\' test')
 	u_boot_console.drain_console()
@@ -84,7 +84,7 @@ def test_efi_selftest_text_input(u_boot_console):
 	u_boot_console.run_command(cmd=chr(9), wait_for_echo=False,
 				   send_nl=False, wait_for_prompt=False)
 	m = u_boot_console.p.expect(
-		['Unicode char 9 \(TAB\), scan code 0 \(Null\)'])
+		[r'Unicode char 9 \(TAB\), scan code 0 \(Null\)'])
 	if m != 0:
 		raise Exception('BS failed in \'text input\' test')
 	u_boot_console.drain_console()
@@ -92,7 +92,7 @@ def test_efi_selftest_text_input(u_boot_console):
 	u_boot_console.run_command(cmd='a', wait_for_echo=False, send_nl=False,
 				   wait_for_prompt=False)
 	m = u_boot_console.p.expect(
-		['Unicode char 97 \(\'a\'\), scan code 0 \(Null\)'])
+		[r'Unicode char 97 \(\'a\'\), scan code 0 \(Null\)'])
 	if m != 0:
 		raise Exception('\'a\' failed in \'text input\' test')
 	u_boot_console.drain_console()
@@ -100,14 +100,14 @@ def test_efi_selftest_text_input(u_boot_console):
 	u_boot_console.run_command(cmd=chr(27) + '[A', wait_for_echo=False,
 				   send_nl=False, wait_for_prompt=False)
 	m = u_boot_console.p.expect(
-		['Unicode char 0 \(Null\), scan code 1 \(Up\)'])
+		[r'Unicode char 0 \(Null\), scan code 1 \(Up\)'])
 	if m != 0:
 		raise Exception('UP failed in \'text input\' test')
 	u_boot_console.drain_console()
 	# Euro sign
-	u_boot_console.run_command(cmd='\xe2\x82\xac', wait_for_echo=False,
+	u_boot_console.run_command(cmd=b'\xe2\x82\xac'.decode(), wait_for_echo=False,
 				   send_nl=False, wait_for_prompt=False)
-	m = u_boot_console.p.expect(['Unicode char 8364 \(\''])
+	m = u_boot_console.p.expect([r'Unicode char 8364 \(\''])
 	if m != 0:
 		raise Exception('Euro sign failed in \'text input\' test')
 	u_boot_console.drain_console()
@@ -129,7 +129,7 @@ def test_efi_selftest_text_input_ex(u_boot_console):
 	u_boot_console.run_command(cmd='setenv efi_selftest extended text input')
 	output = u_boot_console.run_command(cmd='bootefi selftest',
 					    wait_for_prompt=False)
-	m = u_boot_console.p.expect(['To terminate type \'CTRL\+x\''])
+	m = u_boot_console.p.expect([r'To terminate type \'CTRL\+x\''])
 	if m != 0:
 		raise Exception('No prompt for \'text input\' test')
 	u_boot_console.drain_console()
@@ -138,7 +138,7 @@ def test_efi_selftest_text_input_ex(u_boot_console):
 	u_boot_console.run_command(cmd=chr(4), wait_for_echo=False,
 				   send_nl=False, wait_for_prompt=False)
 	m = u_boot_console.p.expect(
-		['Unicode char 100 \\(\'d\'\\), scan code 0 \\(CTRL\\+Null\\)'])
+		[r'Unicode char 100 \(\'d\'\), scan code 0 \(CTRL\+Null\)'])
 	if m != 0:
 		raise Exception('EOT failed in \'text input\' test')
 	u_boot_console.drain_console()
@@ -146,7 +146,7 @@ def test_efi_selftest_text_input_ex(u_boot_console):
 	u_boot_console.run_command(cmd=chr(8), wait_for_echo=False,
 				   send_nl=False, wait_for_prompt=False)
 	m = u_boot_console.p.expect(
-		['Unicode char 8 \(BS\), scan code 0 \(\+Null\)'])
+		[r'Unicode char 8 \(BS\), scan code 0 \(\+Null\)'])
 	if m != 0:
 		raise Exception('BS failed in \'text input\' test')
 	u_boot_console.drain_console()
@@ -154,7 +154,7 @@ def test_efi_selftest_text_input_ex(u_boot_console):
 	u_boot_console.run_command(cmd=chr(9), wait_for_echo=False,
 				   send_nl=False, wait_for_prompt=False)
 	m = u_boot_console.p.expect(
-		['Unicode char 9 \(TAB\), scan code 0 \(\+Null\)'])
+		[r'Unicode char 9 \(TAB\), scan code 0 \(\+Null\)'])
 	if m != 0:
 		raise Exception('TAB failed in \'text input\' test')
 	u_boot_console.drain_console()
@@ -162,7 +162,7 @@ def test_efi_selftest_text_input_ex(u_boot_console):
 	u_boot_console.run_command(cmd='a', wait_for_echo=False, send_nl=False,
 				   wait_for_prompt=False)
 	m = u_boot_console.p.expect(
-		['Unicode char 97 \(\'a\'\), scan code 0 \(Null\)'])
+		[r'Unicode char 97 \(\'a\'\), scan code 0 \(Null\)'])
 	if m != 0:
 		raise Exception('\'a\' failed in \'text input\' test')
 	u_boot_console.drain_console()
@@ -170,23 +170,23 @@ def test_efi_selftest_text_input_ex(u_boot_console):
 	u_boot_console.run_command(cmd=chr(27) + '[A', wait_for_echo=False,
 				   send_nl=False, wait_for_prompt=False)
 	m = u_boot_console.p.expect(
-		['Unicode char 0 \(Null\), scan code 1 \(\+Up\)'])
+		[r'Unicode char 0 \(Null\), scan code 1 \(\+Up\)'])
 	if m != 0:
 		raise Exception('UP failed in \'text input\' test')
 	u_boot_console.drain_console()
 	# Euro sign
-	u_boot_console.run_command(cmd='\xe2\x82\xac', wait_for_echo=False,
+	u_boot_console.run_command(cmd=b'\xe2\x82\xac'.decode(), wait_for_echo=False,
 				   send_nl=False, wait_for_prompt=False)
-	m = u_boot_console.p.expect(['Unicode char 8364 \(\''])
+	m = u_boot_console.p.expect([r'Unicode char 8364 \(\''])
 	if m != 0:
 		raise Exception('Euro sign failed in \'text input\' test')
 	u_boot_console.drain_console()
 	# SHIFT+ALT+FN 5
-	u_boot_console.run_command(cmd='\x1b\x5b\x31\x35\x3b\x34\x7e',
+	u_boot_console.run_command(cmd=b'\x1b\x5b\x31\x35\x3b\x34\x7e'.decode(),
 				   wait_for_echo=False, send_nl=False,
 				   wait_for_prompt=False)
 	m = u_boot_console.p.expect(
-		['Unicode char 0 \(Null\), scan code 15 \(SHIFT\+ALT\+FN 5\)'])
+		[r'Unicode char 0 \(Null\), scan code 15 \(SHIFT\+ALT\+FN 5\)'])
 	if m != 0:
 		raise Exception('SHIFT+ALT+FN 5 failed in \'text input\' test')
 	u_boot_console.drain_console()
-- 
2.17.1

[U-Boot] [PATCHv2 09/13] test/py: Update test_fs to decode check_output calls

[Tom Rini]

The check_output function from the subprocess Python module by default
returns data as encoded bytes and leaves decoding to the application.
Given our uses of the call, it makes the most sense to immediately
decode the results.

Signed-off-by: Tom Rini <trini@konsulko.com>
---
Changes in v2:
- New patch
---
 test/py/tests/test_fs/conftest.py | 24 ++++++++++++------------
 1 file changed, 12 insertions(+), 12 deletions(-)

diff --git a/test/py/tests/test_fs/conftest.py b/test/py/tests/test_fs/conftest.py
index 354d17672fe4..1949f916197a 100644
--- a/test/py/tests/test_fs/conftest.py
+++ b/test/py/tests/test_fs/conftest.py
@@ -300,38 +300,38 @@ def fs_obj_basic(request, u_boot_config):
         # Generate the md5sums of reads that we will test against small file
         out = check_output(
             'dd if=%s bs=1M skip=0 count=1 2> /dev/null | md5sum'
-	    % small_file, shell=True)
+	    % small_file, shell=True).decode()
         md5val = [ out.split()[0] ]
 
         # Generate the md5sums of reads that we will test against big file
         # One from beginning of file.
         out = check_output(
             'dd if=%s bs=1M skip=0 count=1 2> /dev/null | md5sum'
-	    % big_file, shell=True)
+	    % big_file, shell=True).decode()
         md5val.append(out.split()[0])
 
         # One from end of file.
         out = check_output(
             'dd if=%s bs=1M skip=2499 count=1 2> /dev/null | md5sum'
-	    % big_file, shell=True)
+	    % big_file, shell=True).decode()
         md5val.append(out.split()[0])
 
         # One from the last 1MB chunk of 2GB
         out = check_output(
             'dd if=%s bs=1M skip=2047 count=1 2> /dev/null | md5sum'
-	    % big_file, shell=True)
+	    % big_file, shell=True).decode()
         md5val.append(out.split()[0])
 
         # One from the start 1MB chunk from 2GB
         out = check_output(
             'dd if=%s bs=1M skip=2048 count=1 2> /dev/null | md5sum'
-	    % big_file, shell=True)
+	    % big_file, shell=True).decode()
         md5val.append(out.split()[0])
 
         # One 1MB chunk crossing the 2GB boundary
         out = check_output(
             'dd if=%s bs=512K skip=4095 count=2 2> /dev/null | md5sum'
-	    % big_file, shell=True)
+	    % big_file, shell=True).decode()
         md5val.append(out.split()[0])
 
         umount_fs(mount_dir)
@@ -390,7 +390,7 @@ def fs_obj_ext(request, u_boot_config):
             % min_file, shell=True)
         out = check_output(
             'dd if=%s bs=1K 2> /dev/null | md5sum'
-            % min_file, shell=True)
+            % min_file, shell=True).decode()
         md5val = [ out.split()[0] ]
 
         # Calculate md5sum of Test Case 4
@@ -399,7 +399,7 @@ def fs_obj_ext(request, u_boot_config):
         check_call('dd if=%s of=%s bs=1K seek=5 count=20'
             % (min_file, tmp_file), shell=True)
         out = check_output('dd if=%s bs=1K 2> /dev/null | md5sum'
-            % tmp_file, shell=True)
+            % tmp_file, shell=True).decode()
         md5val.append(out.split()[0])
 
         # Calculate md5sum of Test Case 5
@@ -408,7 +408,7 @@ def fs_obj_ext(request, u_boot_config):
         check_call('dd if=%s of=%s bs=1K seek=5 count=5'
             % (min_file, tmp_file), shell=True)
         out = check_output('dd if=%s bs=1K 2> /dev/null | md5sum'
-            % tmp_file, shell=True)
+            % tmp_file, shell=True).decode()
         md5val.append(out.split()[0])
 
         # Calculate md5sum of Test Case 7
@@ -417,7 +417,7 @@ def fs_obj_ext(request, u_boot_config):
         check_call('dd if=%s of=%s bs=1K seek=20 count=20'
             % (min_file, tmp_file), shell=True)
         out = check_output('dd if=%s bs=1K 2> /dev/null | md5sum'
-            % tmp_file, shell=True)
+            % tmp_file, shell=True).decode()
         md5val.append(out.split()[0])
 
         check_call('rm %s' % tmp_file, shell=True)
@@ -582,11 +582,11 @@ def fs_obj_symlink(request, u_boot_config):
         # Generate the md5sums of reads that we will test against small file
         out = check_output(
             'dd if=%s bs=1M skip=0 count=1 2> /dev/null | md5sum'
-            % small_file, shell=True)
+            % small_file, shell=True).decode()
         md5val = [out.split()[0]]
         out = check_output(
             'dd if=%s bs=10M skip=0 count=1 2> /dev/null | md5sum'
-            % medium_file, shell=True)
+            % medium_file, shell=True).decode()
         md5val.extend([out.split()[0]])
 
         umount_fs(mount_dir)
-- 
2.17.1

[U-Boot] [PATCHv2 10/13] test/py: Rework test.py to be a different kind of wrapper

[Tom Rini]

Now that we have moved to being based on pytest for python3 we need to
make our test.py wrapper more robust in terms of only calling python3
rather than possibly finding and using python2.  To do this, change from
execvp()'ing pytest to invoking the package itself via python.  In the
event that pytest is unavailable we still get a user-friendly error:

pkg_resources.DistributionNotFound: The 'pytest' distribution was not found and is required by the application

Cc: Stephen Warren <swarren@nvidia.com>
Signed-off-by: Tom Rini <trini@konsulko.com>
---
Changes in v2:
- New patch
---
 test/py/test.py | 20 ++++----------------
 1 file changed, 4 insertions(+), 16 deletions(-)

diff --git a/test/py/test.py b/test/py/test.py
index 0ce1838833f6..bee88d96bc26 100755
--- a/test/py/test.py
+++ b/test/py/test.py
@@ -10,23 +10,11 @@
 import os
 import os.path
 import sys
-
-# Get rid of argv[0]
-sys.argv.pop(0)
+from pkg_resources import load_entry_point
 
 # argv; py.test test_directory_name user-supplied-arguments
-args = ['py.test', os.path.dirname(__file__) + '/tests']
+args = [os.path.dirname(__file__) + '/tests']
 args.extend(sys.argv)
 
-try:
-    os.execvp('py.test', args)
-except:
-    # Log full details of any exception for detailed analysis
-    import traceback
-    traceback.print_exc()
-    # Hint to the user that they likely simply haven't installed the required
-    # dependencies.
-    print('''
-exec(py.test) failed; perhaps you are missing some dependencies?
-See test/py/README.md for the list.''', file=sys.stderr)
-    sys.exit(1)
+if __name__ == '__main__':
+    sys.exit(load_entry_point('pytest', 'console_scripts', 'pytest')(args))
-- 
2.17.1

[U-Boot] [PATCHv2 11/13] test/py: Update docs, add requirements.txt for pip

[Tom Rini]

To be more closely aligned with Python community best practices, we need
to better document our usage of pip and make use of a requirements.txt
file that shows the versions of the tools that we are using.  This will
aide in ensuring reproducibility of our tests as well.

Signed-off-by: Tom Rini <trini@konsulko.com>
---
 test/py/README.md        | 45 +++++++++++++++++++++-------------------
 test/py/requirements.txt | 22 ++++++++++++++++++++
 2 files changed, 46 insertions(+), 21 deletions(-)
 create mode 100644 test/py/requirements.txt

diff --git a/test/py/README.md b/test/py/README.md
index 2156661d6c42..1489ff133b84 100644
--- a/test/py/README.md
+++ b/test/py/README.md
@@ -21,19 +21,24 @@ involves executing some binary and interacting with its stdin/stdout. You will
 need to implement various "hook" scripts that are called by the test suite at
 the appropriate time.
 
-On Debian or Debian-like distributions, the following packages are required.
-Some packages are required to execute any test, and others only for specific
-tests. Similar package names should exist in other distributions.
-
-| Package        | Version tested (Ubuntu 14.04) |
-| -------------- | ----------------------------- |
-| python         | 2.7.5-5ubuntu3                |
-| python-pytest  | 2.5.1-1                       |
-| python-subunit | -                             |
-| gdisk          | 0.8.8-1ubuntu0.1              |
-| dfu-util       | 0.5-1                         |
-| dtc            | 1.4.0+dfsg-1                  |
-| openssl        | 1.0.1f-1ubuntu2.22            |
+In order to run the testsuite at a minimum we require that both python3 and
+pip for python3 be installed.  All of the required python modules are
+described in the requirements.txt file in this directory and can be installed
+with the command ```pip install -r requirements.txt```
+
+In order to execute certain tests on their supported platforms other tools
+will be required.  The following is an incomplete list:
+
+| Package        |
+| -------------- |
+| gdisk          |
+| dfu-util       |
+| dtc            |
+| openssl        |
+| sudo OR guestmount |
+| e2fsprogs      |
+| dosfstools     |
+| openssl        |
 
 The test script supports either:
 
@@ -45,18 +50,16 @@ The test script supports either:
 
 ### Using `virtualenv` to provide requirements
 
-Older distributions (e.g. Ubuntu 10.04) may not provide all the required
-packages, or may provide versions that are too old to run the test suite. One
-can use the Python `virtualenv` script to locally install more up-to-date
-versions of the required packages without interfering with the OS installation.
-For example:
+The recommended way to run the test suite, in order to ensure reproducibility
+is to use `virtualenv` to set up the necessary environment.  This can be done
+via the following commands:
 
 ```bash
 $ cd /path/to/u-boot
-$ sudo apt-get install python python-virtualenv
-$ virtualenv venv
+$ sudo apt-get install python3 python3-virtualenv
+$ virtualenv -p /usr/bin/python3 venv
 $ . ./venv/bin/activate
-$ pip install pytest
+$ pip install -r test/py/requirements.txt
 ```
 
 ## Testing sandbox
diff --git a/test/py/requirements.txt b/test/py/requirements.txt
new file mode 100644
index 000000000000..cf251186f4b1
--- /dev/null
+++ b/test/py/requirements.txt
@@ -0,0 +1,22 @@
+atomicwrites==1.3.0
+attrs==19.3.0
+coverage==4.5.4
+extras==1.0.0
+fixtures==3.0.0
+importlib-metadata==0.23
+linecache2==1.0.0
+more-itertools==7.2.0
+packaging==19.2
+pbr==5.4.3
+pluggy==0.13.0
+py==1.8.0
+pyparsing==2.4.2
+pytest==5.2.1
+python-mimeparse==1.6.0
+python-subunit==1.3.0
+six==1.12.0
+testtools==2.3.0
+traceback2==1.4.0
+unittest2==1.1.0
+wcwidth==0.1.7
+zipp==0.6.0
-- 
2.17.1

[U-Boot] [PATCHv2 12/13] gitlab/travis: Rework how and when we use virtualenv in order to use python3

[Tom Rini]

As things stand today, we have tools that CI requires where "python"
must be "python2".  We need to use a virtualenv and pip in order to
ensure that our pytest tests can be run.  Rework things slightly so
that:
- On Travis-CI, we install python-pyelftools for the platforms that
  require pyelftools to be installed.
- On GitLab-CI, we move to a newer base image that includes python3-pip
  and continue to use a virtualenv per job that needs it, for the
  correct set of packages.

Signed-off-by: Tom Rini <trini@konsulko.com>
---
Changes in v2:
- Add Travis-CI support here, and stop calling it a work-around.  This
  is just the path forward until we have everything using python3 and at
  that point evaluate what the right changes, if any, are.
---
 .gitlab-ci.yml | 10 ++++------
 .travis.yml    | 30 +++++++++++++++---------------
 2 files changed, 19 insertions(+), 21 deletions(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 5a34321570ce..9b295ac710fe 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -2,7 +2,7 @@
 
 # Grab our configured image.  The source for this is found at:
 # https://gitlab.denx.de/u-boot/gitlab-ci-runner
-image: trini/u-boot-gitlab-ci-runner:bionic-20190912.1-03Oct2019
+image: trini/u-boot-gitlab-ci-runner:bionic-20191010-20Oct2019
 
 # We run some tests in different order, to catch some failures quicker.
 stages:
@@ -18,11 +18,6 @@ stages:
     - git clone --depth=1 git://github.com/swarren/uboot-test-hooks.git /tmp/uboot-test-hooks
     - ln -s travis-ci /tmp/uboot-test-hooks/bin/`hostname`
     - ln -s travis-ci /tmp/uboot-test-hooks/py/`hostname`
-    - virtualenv /tmp/venv
-    - . /tmp/venv/bin/activate
-    - pip install pytest
-    - pip install python-subunit
-    - pip install coverage
     - grub-mkimage --prefix="" -o ~/grub_x86.efi -O i386-efi normal  echo lsefimmap lsefi lsefisystab efinet tftp minicmd
     - grub-mkimage --prefix="" -o ~/grub_x64.efi -O x86_64-efi normal  echo lsefimmap lsefi lsefisystab efinet tftp minicmd
     - mkdir ~/grub2-arm
@@ -47,6 +42,9 @@ stages:
     # never prevent any test from running. That way, we can always pass
     # "-k something" even when $TEST_PY_TEST_SPEC doesnt need a custom
     # value.
+    - virtualenv -p /usr/bin/python3 /tmp/venv
+    - . /tmp/venv/bin/activate
+    - pip install -r test/py/requirements.txt
     - export UBOOT_TRAVIS_BUILD_DIR=/tmp/.bm-work/${TEST_PY_BD};
       export PATH=/opt/qemu/bin:/tmp/uboot-test-hooks/bin:${PATH};
       export PYTHONPATH=/tmp/uboot-test-hooks/py/travis-ci;
diff --git a/.travis.yml b/.travis.yml
index f61278f9656a..d15b3f742639 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -21,7 +21,9 @@ addons:
     - build-essential
     - libsdl1.2-dev
     - python
-    - python-virtualenv
+    - python-pyelftools
+    - python3-virtualenv
+    - python3-pip
     - swig
     - libpython-dev
     - iasl
@@ -47,11 +49,6 @@ install:
  - echo -e "arc = /tmp/arc_gnu_2018.09_prebuilt_uclibc_le_archs_linux_install" >> ~/.buildman
  - echo -e "\n[toolchain-alias]\nsh = sh2\n" >> ~/.buildman
  - cat ~/.buildman
- - virtualenv /tmp/venv
- - . /tmp/venv/bin/activate
- - pip install pytest
- - pip install python-subunit
- - pip install pyelftools
  - grub-mkimage --prefix="" -o ~/grub_x86.efi -O i386-efi normal  echo lsefimmap lsefi lsefisystab efinet tftp minicmd
  - grub-mkimage --prefix="" -o ~/grub_x64.efi -O x86_64-efi normal  echo lsefimmap lsefi lsefisystab efinet tftp minicmd
  - mkdir ~/grub2-arm
@@ -136,15 +133,6 @@ script:
    cp ~/grub_x64.efi $UBOOT_TRAVIS_BUILD_DIR/;
    cp ~/grub2-arm/usr/lib/grub2/arm-efi/grub.efi $UBOOT_TRAVIS_BUILD_DIR/grub_arm.efi;
    cp ~/grub2-arm64/usr/lib/grub2/arm64-efi/grub.efi $UBOOT_TRAVIS_BUILD_DIR/grub_arm64.efi;
-   if [[ "${TEST_PY_BD}" != "" ]]; then
-     ./test/py/test.py --bd ${TEST_PY_BD} ${TEST_PY_ID}
-       -k "${TEST_PY_TEST_SPEC:-not a_test_which_does_not_exist}"
-       --build-dir "$UBOOT_TRAVIS_BUILD_DIR";
-     ret=$?;
-     if [[ $ret -ne 0 ]]; then
-       exit $ret;
-     fi;
-   fi;
    if [[ -n "${TEST_PY_TOOLS}" ]]; then
      PYTHONPATH="${UBOOT_TRAVIS_BUILD_DIR}/scripts/dtc/pylibfdt"
      PATH="${UBOOT_TRAVIS_BUILD_DIR}/scripts/dtc:${PATH}"
@@ -154,6 +142,18 @@ script:
      PYTHONPATH="${UBOOT_TRAVIS_BUILD_DIR}/scripts/dtc/pylibfdt"
      PATH="${UBOOT_TRAVIS_BUILD_DIR}/scripts/dtc:${PATH}"
      ./tools/dtoc/dtoc -t;
+   fi;
+   if [[ "${TEST_PY_BD}" != "" ]]; then
+     virtualenv -p /usr/bin/python3 /tmp/venv;
+     . /tmp/venv/bin/activate;
+     pip install -r test/py/requirements.txt;
+     ./test/py/test.py --bd ${TEST_PY_BD} ${TEST_PY_ID}
+       -k "${TEST_PY_TEST_SPEC:-not a_test_which_does_not_exist}"
+       --build-dir "$UBOOT_TRAVIS_BUILD_DIR";
+     ret=$?;
+     if [[ $ret -ne 0 ]]; then
+       exit $ret;
+     fi;
    fi
 
 matrix:
-- 
2.17.1

[U-Boot] [PATCHv2 13/13] test/py: Use raw strings more to avoid deprecation warnings

[Tom Rini]

We have two further uses of raw string usage in the test/py codebase
that are used under CI.  The first of which is under the bind test and
is a direct update.  The second of which is to strip VT100 codes from
the match buffer.  While switching this to a raw string is also a direct
update, the comment it notes that problems were encountered on Ubuntu
14.04 (and whatever Python 2 version that was) that required slight
tweaks to the regex.  Replace that now that we're saying Python 3.5 is
the minimum.

Cc: Stephen Warren <swarren@nvidia.com>
Signed-off-by: Tom Rini <trini@konsulko.com>
---
Changes in v2:
- New patch
---
 test/py/tests/test_bind.py | 4 ++--
 test/py/u_boot_spawn.py    | 5 +----
 2 files changed, 3 insertions(+), 6 deletions(-)

diff --git a/test/py/tests/test_bind.py b/test/py/tests/test_bind.py
index 2d48484c6ac6..20c6050342a9 100644
--- a/test/py/tests/test_bind.py
+++ b/test/py/tests/test_bind.py
@@ -9,11 +9,11 @@ def in_tree(response, name, uclass, drv, depth, last_child):
 	lines = [x.strip() for x in response.splitlines()]
 	leaf = ' ' * 4 * depth;
 	if not last_child:
-		leaf = leaf + '\|'
+		leaf = leaf + r'\|'
 	else:
 		leaf = leaf + '`'
 	leaf = leaf + '-- ' + name
-	line = (' *{:10.10}    [0-9]*  \[ [ +] \]   {:20.20}  {}$'
+	line = (r' *{:10.10}    [0-9]*  \[ [ +] \]   {:20.20}  {}$'
 	        .format(uclass, drv, leaf))
 	prog = re.compile(line)
 	for l in lines:
diff --git a/test/py/u_boot_spawn.py b/test/py/u_boot_spawn.py
index 7fedb18d2790..8fbfd78858e5 100644
--- a/test/py/u_boot_spawn.py
+++ b/test/py/u_boot_spawn.py
@@ -42,10 +42,7 @@ class Spawn(object):
         self.after = ''
         self.timeout = None
         # http://stackoverflow.com/questions/7857352/python-regex-to-match-vt100-escape-sequences
-        # Note that re.I doesn't seem to work with this regex (or perhaps the
-        # version of Python in Ubuntu 14.04), hence the inclusion of a-z inside
-        # [] instead.
-        self.re_vt100 = re.compile('(\x1b\[|\x9b)[^@-_a-z]*[@-_a-z]|\x1b[@-_a-z]')
+        self.re_vt100 = re.compile(r'(\x1b\[|\x9b)[^@-_]*[@-_]|\x1b[@-_]', re.I)
 
         (self.pid, self.fd) = pty.fork()
         if self.pid == 0:
-- 
2.17.1

[U-Boot] [PATCHv2 08/13] test/py: test_efi_selftest.py: Updates for python 3 support

[Stephen Warren]

On 10/22/19 9:20 PM, Tom Rini wrote:
> - In python 3 you must use raw strings for regex as other forms are
>    deprecated and would require further changes to the pattern here.
>    In one case this lets us have a simpler match pattern.

I'm not sure the code was really correct even for Python 2. I'm 
surprised that the escape sequences were accepted considering they're 
meaningless. But the new code certainly looks good.

[U-Boot] [PATCHv2 10/13] test/py: Rework test.py to be a different kind of wrapper

[Stephen Warren]

On 10/22/19 9:20 PM, Tom Rini wrote:
> Now that we have moved to being based on pytest for python3 we need to
> make our test.py wrapper more robust in terms of only calling python3
> rather than possibly finding and using python2.  To do this, change from
> execvp()'ing pytest to invoking the package itself via python.  In the
> event that pytest is unavailable we still get a user-friendly error:

I guess if this works, it's fine. But I think it'd work just as well to 
simply s/py.test/py.test-3/ in the existing code?

[U-Boot] [PATCHv2 10/13] test/py: Rework test.py to be a different kind of wrapper

[Tom Rini]

On Wed, Oct 23, 2019 at 10:55:57AM -0600, Stephen Warren wrote:
> On 10/22/19 9:20 PM, Tom Rini wrote:
> > Now that we have moved to being based on pytest for python3 we need to
> > make our test.py wrapper more robust in terms of only calling python3
> > rather than possibly finding and using python2.  To do this, change from
> > execvp()'ing pytest to invoking the package itself via python.  In the
> > event that pytest is unavailable we still get a user-friendly error:
> 
> I guess if this works, it's fine. But I think it'd work just as well to
> simply s/py.test/py.test-3/ in the existing code?

We can't because pip only provides "py.test" / "pytest" binaries.

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20191023/5e3991dd/attachment.sig>

[U-Boot] [PATCHv2 10/13] test/py: Rework test.py to be a different kind of wrapper

[Stephen Warren]

On 10/23/19 10:58 AM, Tom Rini wrote:
> On Wed, Oct 23, 2019 at 10:55:57AM -0600, Stephen Warren wrote:
>> On 10/22/19 9:20 PM, Tom Rini wrote:
>>> Now that we have moved to being based on pytest for python3 we need to
>>> make our test.py wrapper more robust in terms of only calling python3
>>> rather than possibly finding and using python2.  To do this, change from
>>> execvp()'ing pytest to invoking the package itself via python.  In the
>>> event that pytest is unavailable we still get a user-friendly error:
>>
>> I guess if this works, it's fine. But I think it'd work just as well to
>> simply s/py.test/py.test-3/ in the existing code?
> 
> We can't because pip only provides "py.test" / "pytest" binaries.

Hmm. It sucks that virtualenv does things differently to Ubuntu 
packaging then. That means this series will probably break my test 
system, which doesn't use virtualenv. I guess I'll apply it and see.

[U-Boot] [PATCHv2 10/13] test/py: Rework test.py to be a different kind of wrapper

[Tom Rini]

On Wed, Oct 23, 2019 at 11:03:38AM -0600, Stephen Warren wrote:
> On 10/23/19 10:58 AM, Tom Rini wrote:
> > On Wed, Oct 23, 2019 at 10:55:57AM -0600, Stephen Warren wrote:
> > > On 10/22/19 9:20 PM, Tom Rini wrote:
> > > > Now that we have moved to being based on pytest for python3 we need to
> > > > make our test.py wrapper more robust in terms of only calling python3
> > > > rather than possibly finding and using python2.  To do this, change from
> > > > execvp()'ing pytest to invoking the package itself via python.  In the
> > > > event that pytest is unavailable we still get a user-friendly error:
> > > 
> > > I guess if this works, it's fine. But I think it'd work just as well to
> > > simply s/py.test/py.test-3/ in the existing code?
> > 
> > We can't because pip only provides "py.test" / "pytest" binaries.
> 
> Hmm. It sucks that virtualenv does things differently to Ubuntu packaging
> then. That means this series will probably break my test system, which
> doesn't use virtualenv. I guess I'll apply it and see.

This should be what works everywhere tho.  Instead of hoping that
"py.test" will be the right python version we tell python3 to use the
pytest module it has.  So if you have system python3-pytest, that works,
if you (python3) pip install pytest locally, that works and if you
virtualenv + python3 + pip install pytest, that works.

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20191023/76ecef23/attachment.sig>

[U-Boot] [PATCHv2 10/13] test/py: Rework test.py to be a different kind of wrapper

[Stephen Warren]

On 10/23/19 11:12 AM, Tom Rini wrote:
> On Wed, Oct 23, 2019 at 11:03:38AM -0600, Stephen Warren wrote:
>> On 10/23/19 10:58 AM, Tom Rini wrote:
>>> On Wed, Oct 23, 2019 at 10:55:57AM -0600, Stephen Warren wrote:
>>>> On 10/22/19 9:20 PM, Tom Rini wrote:
>>>>> Now that we have moved to being based on pytest for python3 we need to
>>>>> make our test.py wrapper more robust in terms of only calling python3
>>>>> rather than possibly finding and using python2.  To do this, change from
>>>>> execvp()'ing pytest to invoking the package itself via python.  In the
>>>>> event that pytest is unavailable we still get a user-friendly error:
>>>>
>>>> I guess if this works, it's fine. But I think it'd work just as well to
>>>> simply s/py.test/py.test-3/ in the existing code?
>>>
>>> We can't because pip only provides "py.test" / "pytest" binaries.
>>
>> Hmm. It sucks that virtualenv does things differently to Ubuntu packaging
>> then. That means this series will probably break my test system, which
>> doesn't use virtualenv. I guess I'll apply it and see.
> 
> This should be what works everywhere tho.  Instead of hoping that
> "py.test" will be the right python version we tell python3 to use the
> pytest module it has.  So if you have system python3-pytest, that works,
> if you (python3) pip install pytest locally, that works and if you
> virtualenv + python3 + pip install pytest, that works.

On Ubuntu 16.04, with everything installed from Ubuntu packages rather 
than using virtualenv, I get this:

> + ./src/u-boot/test/py/test.py --bd sandbox --build-dir /var/lib/jenkins/workspace/u-boot-github_swarren_u-boot-tegra_dev-test-py/U_BOOT_BOARD/sandbox/build/u-boot/sandbox -k 'not sleep'
> Traceback (most recent call last):
>   File "./src/u-boot/test/py/test.py", line 20, in <module>
>     sys.exit(load_entry_point('pytest', 'console_scripts', 'pytest')(args))
>   File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 542, in load_entry_point
>     return get_distribution(dist).load_entry_point(group, name)
>   File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2568, in load_entry_point
>     raise ImportError("Entry point %r not found" % ((group, name),))
> ImportError: Entry point ('console_scripts', 'pytest') not found

> [swarren at swarren-lx1 u-boot]$ dpkg -l|grep py|grep test
> ii  python-pytest    2.8.7-4
> ii  python3-pytest   2.8.7-4

I guess that's much older than the pytest in requirements.txt (5.2.1!!). 
While I think we can require a newish version of Python (3.5) since 
that's widely available on supported OSs, I don't think we want to force 
people to have a very new version of all the packages (which supported 
distros don't package yet), or change they way they run pytest (by 
requiring use of a virtualenv and requirements.txt) since that mechanism 
isn't supported on old branches of U-Boot, which will then require all 
kinds of special cases in scripts that invoke the U-Boot test system.

[U-Boot] [PATCHv2 10/13] test/py: Rework test.py to be a different kind of wrapper

[Stephen Warren]

On 10/23/19 11:29 AM, Stephen Warren wrote:
> On 10/23/19 11:12 AM, Tom Rini wrote:
>> On Wed, Oct 23, 2019 at 11:03:38AM -0600, Stephen Warren wrote:
>>> On 10/23/19 10:58 AM, Tom Rini wrote:
>>>> On Wed, Oct 23, 2019 at 10:55:57AM -0600, Stephen Warren wrote:
>>>>> On 10/22/19 9:20 PM, Tom Rini wrote:
>>>>>> Now that we have moved to being based on pytest for python3 we 
>>>>>> need to
>>>>>> make our test.py wrapper more robust in terms of only calling python3
>>>>>> rather than possibly finding and using python2.  To do this, 
>>>>>> change from
>>>>>> execvp()'ing pytest to invoking the package itself via python.  In 
>>>>>> the
>>>>>> event that pytest is unavailable we still get a user-friendly error:
>>>>>
>>>>> I guess if this works, it's fine. But I think it'd work just as 
>>>>> well to
>>>>> simply s/py.test/py.test-3/ in the existing code?
>>>>
>>>> We can't because pip only provides "py.test" / "pytest" binaries.
>>>
>>> Hmm. It sucks that virtualenv does things differently to Ubuntu 
>>> packaging
>>> then. That means this series will probably break my test system, which
>>> doesn't use virtualenv. I guess I'll apply it and see.
>>
>> This should be what works everywhere tho.  Instead of hoping that
>> "py.test" will be the right python version we tell python3 to use the
>> pytest module it has.  So if you have system python3-pytest, that works,
>> if you (python3) pip install pytest locally, that works and if you
>> virtualenv + python3 + pip install pytest, that works.
> 
> On Ubuntu 16.04, with everything installed from Ubuntu packages rather 
> than using virtualenv, I get this:
> 
>> + ./src/u-boot/test/py/test.py --bd sandbox --build-dir 
>> /var/lib/jenkins/workspace/u-boot-github_swarren_u-boot-tegra_dev-test-py/U_BOOT_BOARD/sandbox/build/u-boot/sandbox 
>> -k 'not sleep'
>> Traceback (most recent call last):
>>   File "./src/u-boot/test/py/test.py", line 20, in <module>
>>     sys.exit(load_entry_point('pytest', 'console_scripts', 
>> 'pytest')(args))
>>   File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", 
>> line 542, in load_entry_point
>>     return get_distribution(dist).load_entry_point(group, name)
>>   File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", 
>> line 2568, in load_entry_point
>>     raise ImportError("Entry point %r not found" % ((group, name),))
>> ImportError: Entry point ('console_scripts', 'pytest') not found
> 
>> [swarren at swarren-lx1 u-boot]$ dpkg -l|grep py|grep test
>> ii  python-pytest    2.8.7-4
>> ii  python3-pytest   2.8.7-4
> 
> I guess that's much older than the pytest in requirements.txt (5.2.1!!). 
> While I think we can require a newish version of Python (3.5) since 
> that's widely available on supported OSs, I don't think we want to force 
> people to have a very new version of all the packages (which supported 
> distros don't package yet), or change they way they run pytest (by 
> requiring use of a virtualenv and requirements.txt) since that mechanism 
> isn't supported on old branches of U-Boot, which will then require all 
> kinds of special cases in scripts that invoke the U-Boot test system.

Ah. The issue is the name of the entry point; the last parameter to 
load_entry_point(). "py.test" is supported everywhere, but only newer 
versions of pytesst have a "pytest" entry point. Changing that parameter 
to "py.test" makes the wrapper script work. Now every other test fails; 
I will see if there's an easy fix for that or not.

[U-Boot] [PATCHv2 10/13] test/py: Rework test.py to be a different kind of wrapper

[Tom Rini]

On Wed, Oct 23, 2019 at 12:04:30PM -0600, Stephen Warren wrote:
> On 10/23/19 11:29 AM, Stephen Warren wrote:
> > On 10/23/19 11:12 AM, Tom Rini wrote:
> > > On Wed, Oct 23, 2019 at 11:03:38AM -0600, Stephen Warren wrote:
> > > > On 10/23/19 10:58 AM, Tom Rini wrote:
> > > > > On Wed, Oct 23, 2019 at 10:55:57AM -0600, Stephen Warren wrote:
> > > > > > On 10/22/19 9:20 PM, Tom Rini wrote:
> > > > > > > Now that we have moved to being based on pytest for
> > > > > > > python3 we need to
> > > > > > > make our test.py wrapper more robust in terms of only calling python3
> > > > > > > rather than possibly finding and using python2.  To
> > > > > > > do this, change from
> > > > > > > execvp()'ing pytest to invoking the package itself
> > > > > > > via python.  In the
> > > > > > > event that pytest is unavailable we still get a user-friendly error:
> > > > > > 
> > > > > > I guess if this works, it's fine. But I think it'd work
> > > > > > just as well to
> > > > > > simply s/py.test/py.test-3/ in the existing code?
> > > > > 
> > > > > We can't because pip only provides "py.test" / "pytest" binaries.
> > > > 
> > > > Hmm. It sucks that virtualenv does things differently to Ubuntu
> > > > packaging
> > > > then. That means this series will probably break my test system, which
> > > > doesn't use virtualenv. I guess I'll apply it and see.
> > > 
> > > This should be what works everywhere tho.  Instead of hoping that
> > > "py.test" will be the right python version we tell python3 to use the
> > > pytest module it has.  So if you have system python3-pytest, that works,
> > > if you (python3) pip install pytest locally, that works and if you
> > > virtualenv + python3 + pip install pytest, that works.
> > 
> > On Ubuntu 16.04, with everything installed from Ubuntu packages rather
> > than using virtualenv, I get this:
> > 
> > > + ./src/u-boot/test/py/test.py --bd sandbox --build-dir /var/lib/jenkins/workspace/u-boot-github_swarren_u-boot-tegra_dev-test-py/U_BOOT_BOARD/sandbox/build/u-boot/sandbox
> > > -k 'not sleep'
> > > Traceback (most recent call last):
> > >   File "./src/u-boot/test/py/test.py", line 20, in <module>
> > >     sys.exit(load_entry_point('pytest', 'console_scripts',
> > > 'pytest')(args))
> > >   File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py",
> > > line 542, in load_entry_point
> > >     return get_distribution(dist).load_entry_point(group, name)
> > >   File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py",
> > > line 2568, in load_entry_point
> > >     raise ImportError("Entry point %r not found" % ((group, name),))
> > > ImportError: Entry point ('console_scripts', 'pytest') not found
> > 
> > > [swarren at swarren-lx1 u-boot]$ dpkg -l|grep py|grep test
> > > ii  python-pytest    2.8.7-4
> > > ii  python3-pytest   2.8.7-4
> > 
> > I guess that's much older than the pytest in requirements.txt (5.2.1!!).
> > While I think we can require a newish version of Python (3.5) since
> > that's widely available on supported OSs, I don't think we want to force
> > people to have a very new version of all the packages (which supported
> > distros don't package yet), or change they way they run pytest (by
> > requiring use of a virtualenv and requirements.txt) since that mechanism
> > isn't supported on old branches of U-Boot, which will then require all
> > kinds of special cases in scripts that invoke the U-Boot test system.
> 
> Ah. The issue is the name of the entry point; the last parameter to
> load_entry_point(). "py.test" is supported everywhere, but only newer
> versions of pytesst have a "pytest" entry point. Changing that parameter to
> "py.test" makes the wrapper script work. Now every other test fails; I will
> see if there's an easy fix for that or not.

We're going to have to make a cut-off point somewhere it looks like.
pytest 2.8.7 (which we had pinned to) is incompatible with pytest 4.x
and throws a lot of warnings I believe on pytest 3.x..

What I really want to avoid is the problem we have today where we've
actively pinned ourselves to a now quite out of date version and have a
lot of problems showing up.

So we aren't going to have one environment that works for history and
future but I am hoping / trying to plan such that if we have
requirements.txt we can say from here until the next new best practices
(which today would be using pipenv, but that starts out with pip install
pipenv, sigh...) we can say to use requirements.txt and for history,
python2 + pytest 2.8.7 and local setups could even dump that in to a
requirements.txt file if that makes scripting easier.

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20191023/7a81a956/attachment.sig>

[U-Boot] [PATCHv2 11/13] test/py: Update docs, add requirements.txt for pip

[Stephen Warren]

On 10/22/19 9:20 PM, Tom Rini wrote:
> To be more closely aligned with Python community best practices, we need
> to better document our usage of pip and make use of a requirements.txt
> file that shows the versions of the tools that we are using.  This will
> aide in ensuring reproducibility of our tests as well.

> +In order to execute certain tests on their supported platforms other tools
> +will be required.  The following is an incomplete list:
> +
> +| Package        |
> +| -------------- |
> +| gdisk          |
> +| dfu-util       |
> +| dtc            |

This looks like my bug from before, but may as well fix it: That package 
doesn't exist, at least on Ubuntu 16.04. There is device-tree-compiler 
though.

> +| openssl        |
> +| sudo OR guestmount |
> +| e2fsprogs      |
> +| dosfstools     |
> +| openssl        |

openssl is listed twice.

[U-Boot] [PATCHv2 11/13] test/py: Update docs, add requirements.txt for pip

[Tom Rini]

On Wed, Oct 23, 2019 at 12:30:52PM -0600, Stephen Warren wrote:
> On 10/22/19 9:20 PM, Tom Rini wrote:
> > To be more closely aligned with Python community best practices, we need
> > to better document our usage of pip and make use of a requirements.txt
> > file that shows the versions of the tools that we are using.  This will
> > aide in ensuring reproducibility of our tests as well.
> 
> > +In order to execute certain tests on their supported platforms other tools
> > +will be required.  The following is an incomplete list:
> > +
> > +| Package        |
> > +| -------------- |
> > +| gdisk          |
> > +| dfu-util       |
> > +| dtc            |
> 
> This looks like my bug from before, but may as well fix it: That package
> doesn't exist, at least on Ubuntu 16.04. There is device-tree-compiler
> though.

I wonder how best to list that.  Upstream is "dtc" and I forget what
it's called package-wise on Fedora systems and similar.  But since I
want to avoid making a distribution specific list maybe I need to add a
few words to be clear it's a tool list and not a package list.  Doing
"apt-cache search dtc" shows device-tree-compiler for example and I bet
the dnf equivalent does too.

> > +| openssl        |
> > +| sudo OR guestmount |

Note that for example guestmount comes from libguestfs-tools on
Debian/similar systems.

> > +| e2fsprogs      |
> > +| dosfstools     |
> > +| openssl        |
> 
> openssl is listed twice.

Thanks, fixed.

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20191023/412d5697/attachment.sig>

[U-Boot] [PATCHv2 06/13] test/py: Manual python3 fixes

[Stephen Warren]

On 10/22/19 9:20 PM, Tom Rini wrote:
> - Modern pytest is more visible in telling us about parameters that we
>    had not described, so describe a few more.
> - ConfigParser.readfp(...) is now configparser.read_file(...)
> - As part of the "strings vs bytes" conversions in Python 3, we use the
>    default encoding/decoding of utf-8 but in some places tell Python to
>    replace problematic conversions rather than throw a fatal error.
> - Fix a typo noticed while doing the above ("tot he" -> "to the").
> - As suggested by Stephen, re-alphabetize the import list
> - Per Heinrich, replace how we write contents in test_fit.py

> diff --git a/test/py/multiplexed_log.py b/test/py/multiplexed_log.py

> @@ -133,7 +134,7 @@ class RunAndLog(object):
>           self.logfile.write(self, msg)
>   
>           try:
> -            p = subprocess.Popen(cmd, cwd=cwd,
> +            p = subprocess.Popen(cmd, cwd=cwd, encoding="utf-8",

The encoding parameter was added in Python 3.6, so this change causes 
test.py to crash for me (with Python 3.5.2). The following will fix that:

>             p = subprocess.Popen(cmd, cwd=cwd,
>                 stdin=None, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
>             (stdout, stderr) = p.communicate()
>             if stdout is not None:
>                 stdout = stdout.decode('utf-8')
>             if stderr is not None:
>                 stderr = stderr.decode('utf-8')

I also noticed that a variety of strings in these patches use "" not '' 
in at least this patch. When I first wrote test/py, Simon made me go 
through everything and replace "" with '', so we should probably make 
new code follow the convention of using ''.

[U-Boot] [PATCHv2 06/13] test/py: Manual python3 fixes

[Tom Rini]

On Wed, Oct 23, 2019 at 12:50:12PM -0600, Stephen Warren wrote:
> On 10/22/19 9:20 PM, Tom Rini wrote:
> > - Modern pytest is more visible in telling us about parameters that we
> >    had not described, so describe a few more.
> > - ConfigParser.readfp(...) is now configparser.read_file(...)
> > - As part of the "strings vs bytes" conversions in Python 3, we use the
> >    default encoding/decoding of utf-8 but in some places tell Python to
> >    replace problematic conversions rather than throw a fatal error.
> > - Fix a typo noticed while doing the above ("tot he" -> "to the").
> > - As suggested by Stephen, re-alphabetize the import list
> > - Per Heinrich, replace how we write contents in test_fit.py
> 
> > diff --git a/test/py/multiplexed_log.py b/test/py/multiplexed_log.py
> 
> > @@ -133,7 +134,7 @@ class RunAndLog(object):
> >           self.logfile.write(self, msg)
> >           try:
> > -            p = subprocess.Popen(cmd, cwd=cwd,
> > +            p = subprocess.Popen(cmd, cwd=cwd, encoding="utf-8",
> 
> The encoding parameter was added in Python 3.6, so this change causes
> test.py to crash for me (with Python 3.5.2). The following will fix that:

Right, Ubuntu 16.04 has Python 3.5.x and later has 3.6 or higher.  It's
reasonable to ask what the minimum host version is going to be and how
much overhead things get the lower we go.  Given:

> >             p = subprocess.Popen(cmd, cwd=cwd,
> >                 stdin=None, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
> >             (stdout, stderr) = p.communicate()
> >             if stdout is not None:
> >                 stdout = stdout.decode('utf-8')
> >             if stderr is not None:
> >                 stderr = stderr.decode('utf-8')

That's not too bad, so we can go with that.  Can you please chime in on
the "Minimum Python 3 version?" thread with what I assume is your
use-case of needing to use older-than-latest-LTS distributions?

> I also noticed that a variety of strings in these patches use "" not '' in
> at least this patch. When I first wrote test/py, Simon made me go through
> everything and replace "" with '', so we should probably make new code
> follow the convention of using ''.

OK, I'll get on top of that, thanks.

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20191023/a85257a0/attachment.sig>

[U-Boot] [PATCHv2 06/13] test/py: Manual python3 fixes

[Stephen Warren]

On 10/23/19 1:01 PM, Tom Rini wrote:
> On Wed, Oct 23, 2019 at 12:50:12PM -0600, Stephen Warren wrote:
>> On 10/22/19 9:20 PM, Tom Rini wrote:
>>> - Modern pytest is more visible in telling us about parameters that we
>>>     had not described, so describe a few more.
>>> - ConfigParser.readfp(...) is now configparser.read_file(...)
>>> - As part of the "strings vs bytes" conversions in Python 3, we use the
>>>     default encoding/decoding of utf-8 but in some places tell Python to
>>>     replace problematic conversions rather than throw a fatal error.
>>> - Fix a typo noticed while doing the above ("tot he" -> "to the").
>>> - As suggested by Stephen, re-alphabetize the import list
>>> - Per Heinrich, replace how we write contents in test_fit.py
>>
>>> diff --git a/test/py/multiplexed_log.py b/test/py/multiplexed_log.py
>>
>>> @@ -133,7 +134,7 @@ class RunAndLog(object):
>>>            self.logfile.write(self, msg)
>>>            try:
>>> -            p = subprocess.Popen(cmd, cwd=cwd,
>>> +            p = subprocess.Popen(cmd, cwd=cwd, encoding="utf-8",
>>
>> The encoding parameter was added in Python 3.6, so this change causes
>> test.py to crash for me (with Python 3.5.2). The following will fix that:
> 
> Right, Ubuntu 16.04 has Python 3.5.x and later has 3.6 or higher.  It's
> reasonable to ask what the minimum host version is going to be and how
> much overhead things get the lower we go.  Given:
> 
>>>              p = subprocess.Popen(cmd, cwd=cwd,
>>>                  stdin=None, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
>>>              (stdout, stderr) = p.communicate()
>>>              if stdout is not None:
>>>                  stdout = stdout.decode('utf-8')
>>>              if stderr is not None:
>>>                  stderr = stderr.decode('utf-8')
> 
> That's not too bad, so we can go with that.  Can you please chime in on
> the "Minimum Python 3 version?" thread with what I assume is your
> use-case of needing to use older-than-latest-LTS distributions?

A very quick survey of distros that I can quickly find package lists for:

RHEL 6 - 2 only
RHEL 7 - 3.6
Ubuntu 16.04 - 3.5
Ubuntu 18.04 - 3.6
Debian oldtsable - 3.5
Debian stable and all later - 3.7

So, Ubuntu 16.04 is perhaps the only distro that still contains Python 
3.5. That said, 16.04 is still fully supported for 18 more months, so it 
feels quite reasonable to support it. I have no choice but to use it for 
now.

[U-Boot] [PATCHv2 10/13] test/py: Rework test.py to be a different kind of wrapper

[Stephen Warren]

On 10/23/19 12:17 PM, Tom Rini wrote:
> On Wed, Oct 23, 2019 at 12:04:30PM -0600, Stephen Warren wrote:
>> On 10/23/19 11:29 AM, Stephen Warren wrote:
>>> On 10/23/19 11:12 AM, Tom Rini wrote:
>>>> On Wed, Oct 23, 2019 at 11:03:38AM -0600, Stephen Warren wrote:
>>>>> On 10/23/19 10:58 AM, Tom Rini wrote:
>>>>>> On Wed, Oct 23, 2019 at 10:55:57AM -0600, Stephen Warren wrote:
>>>>>>> On 10/22/19 9:20 PM, Tom Rini wrote:
>>>>>>>> Now that we have moved to being based on pytest for
>>>>>>>> python3 we need to
>>>>>>>> make our test.py wrapper more robust in terms of only calling python3
>>>>>>>> rather than possibly finding and using python2.  To
>>>>>>>> do this, change from
>>>>>>>> execvp()'ing pytest to invoking the package itself
>>>>>>>> via python.  In the
>>>>>>>> event that pytest is unavailable we still get a user-friendly error:
>>>>>>>
>>>>>>> I guess if this works, it's fine. But I think it'd work
>>>>>>> just as well to
>>>>>>> simply s/py.test/py.test-3/ in the existing code?
>>>>>>
>>>>>> We can't because pip only provides "py.test" / "pytest" binaries.
>>>>>
>>>>> Hmm. It sucks that virtualenv does things differently to Ubuntu
>>>>> packaging
>>>>> then. That means this series will probably break my test system, which
>>>>> doesn't use virtualenv. I guess I'll apply it and see.
>>>>
>>>> This should be what works everywhere tho.  Instead of hoping that
>>>> "py.test" will be the right python version we tell python3 to use the
>>>> pytest module it has.  So if you have system python3-pytest, that works,
>>>> if you (python3) pip install pytest locally, that works and if you
>>>> virtualenv + python3 + pip install pytest, that works.
>>>
>>> On Ubuntu 16.04, with everything installed from Ubuntu packages rather
>>> than using virtualenv, I get this:
>>>
>>>> + ./src/u-boot/test/py/test.py --bd sandbox --build-dir /var/lib/jenkins/workspace/u-boot-github_swarren_u-boot-tegra_dev-test-py/U_BOOT_BOARD/sandbox/build/u-boot/sandbox
>>>> -k 'not sleep'
>>>> Traceback (most recent call last):
>>>>    File "./src/u-boot/test/py/test.py", line 20, in <module>
>>>>      sys.exit(load_entry_point('pytest', 'console_scripts',
>>>> 'pytest')(args))
>>>>    File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py",
>>>> line 542, in load_entry_point
>>>>      return get_distribution(dist).load_entry_point(group, name)
>>>>    File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py",
>>>> line 2568, in load_entry_point
>>>>      raise ImportError("Entry point %r not found" % ((group, name),))
>>>> ImportError: Entry point ('console_scripts', 'pytest') not found
>>>
>>>> [swarren at swarren-lx1 u-boot]$ dpkg -l|grep py|grep test
>>>> ii  python-pytest    2.8.7-4
>>>> ii  python3-pytest   2.8.7-4
>>>
>>> I guess that's much older than the pytest in requirements.txt (5.2.1!!).
>>> While I think we can require a newish version of Python (3.5) since
>>> that's widely available on supported OSs, I don't think we want to force
>>> people to have a very new version of all the packages (which supported
>>> distros don't package yet), or change they way they run pytest (by
>>> requiring use of a virtualenv and requirements.txt) since that mechanism
>>> isn't supported on old branches of U-Boot, which will then require all
>>> kinds of special cases in scripts that invoke the U-Boot test system.
>>
>> Ah. The issue is the name of the entry point; the last parameter to
>> load_entry_point(). "py.test" is supported everywhere, but only newer
>> versions of pytesst have a "pytest" entry point. Changing that parameter to
>> "py.test" makes the wrapper script work. Now every other test fails; I will
>> see if there's an easy fix for that or not.
> 
> We're going to have to make a cut-off point somewhere it looks like.
> pytest 2.8.7 (which we had pinned to) is incompatible with pytest 4.x
> and throws a lot of warnings I believe on pytest 3.x..
> 
> What I really want to avoid is the problem we have today where we've
> actively pinned ourselves to a now quite out of date version and have a
> lot of problems showing up.
> 
> So we aren't going to have one environment that works for history and
> future but I am hoping / trying to plan such that if we have
> requirements.txt we can say from here until the next new best practices
> (which today would be using pipenv, but that starts out with pip install
> pipenv, sigh...) we can say to use requirements.txt and for history,
> python2 + pytest 2.8.7 and local setups could even dump that in to a
> requirements.txt file if that makes scripting easier.

Indeed the pytest 4 changes were a lot of the problem, but there was 
plenty more. It turned out to be too painful to fix it all.

After a lot of pain, I've switched my system over to using a virtualenv 
when there's a test/py/requirements.txt file in the source. Using a 
virtualenv broke some of the HW remote control scripts, which were 
written for Python2 but just said "#!/usr/bin/env python". Fixing that 
to request Python 2 explicitly fixed that issue. Another pain was that 
the ftdi1 module (used by the HW remote control scripts) isn't available 
from pip so can't be installed in a virtualenv. Luckily explicitly using 
python2 for those scripts makes this irrelevant, since system libraries 
are used, since the python2 interpreter doesn't come from the virtualenv 
in that case. However, if that didn't fix the issue, then I'd have 
needed extra options to set up the virtualenv in order to rely on 
Ubuntu-provided package for the module. I'm quoting them below just in 
case someone searching the archives finds this later in the future!

> +if [ -f ./src/u-boot/test/py/requirements.txt ]; then
> +    rm -rf "${venv_dir}"
> +    # --system-site-packages: Need ftdi1 module which is not available via pip,
> +    #   but is installed on the system
> +    virtualenv --system-site-packages -p /usr/bin/python3 "${venv_dir}"
> +    . "${venv_dir}/bin/activate"
> +    # -U: upgrade: Ensure the package is installed in the venv, at the version
> +    #   requested in requirements.txt, even if already available on the host.
> +    pip install -U -r ./src/u-boot/test/py/requirements.txt
> +fi

So in summary, with the Python 3.5 fix I already sent, then this series,
Tested-by: Stephen Warren <swarren@nvidia.com>

[U-Boot] [PATCHv2 10/13] test/py: Rework test.py to be a different kind of wrapper

[Tom Rini]

On Wed, Oct 23, 2019 at 03:11:57PM -0600, Stephen Warren wrote:
> On 10/23/19 12:17 PM, Tom Rini wrote:
> > On Wed, Oct 23, 2019 at 12:04:30PM -0600, Stephen Warren wrote:
> > > On 10/23/19 11:29 AM, Stephen Warren wrote:
> > > > On 10/23/19 11:12 AM, Tom Rini wrote:
> > > > > On Wed, Oct 23, 2019 at 11:03:38AM -0600, Stephen Warren wrote:
> > > > > > On 10/23/19 10:58 AM, Tom Rini wrote:
> > > > > > > On Wed, Oct 23, 2019 at 10:55:57AM -0600, Stephen Warren wrote:
> > > > > > > > On 10/22/19 9:20 PM, Tom Rini wrote:
> > > > > > > > > Now that we have moved to being based on pytest for
> > > > > > > > > python3 we need to
> > > > > > > > > make our test.py wrapper more robust in terms of only calling python3
> > > > > > > > > rather than possibly finding and using python2.  To
> > > > > > > > > do this, change from
> > > > > > > > > execvp()'ing pytest to invoking the package itself
> > > > > > > > > via python.  In the
> > > > > > > > > event that pytest is unavailable we still get a user-friendly error:
> > > > > > > > 
> > > > > > > > I guess if this works, it's fine. But I think it'd work
> > > > > > > > just as well to
> > > > > > > > simply s/py.test/py.test-3/ in the existing code?
> > > > > > > 
> > > > > > > We can't because pip only provides "py.test" / "pytest" binaries.
> > > > > > 
> > > > > > Hmm. It sucks that virtualenv does things differently to Ubuntu
> > > > > > packaging
> > > > > > then. That means this series will probably break my test system, which
> > > > > > doesn't use virtualenv. I guess I'll apply it and see.
> > > > > 
> > > > > This should be what works everywhere tho.  Instead of hoping that
> > > > > "py.test" will be the right python version we tell python3 to use the
> > > > > pytest module it has.  So if you have system python3-pytest, that works,
> > > > > if you (python3) pip install pytest locally, that works and if you
> > > > > virtualenv + python3 + pip install pytest, that works.
> > > > 
> > > > On Ubuntu 16.04, with everything installed from Ubuntu packages rather
> > > > than using virtualenv, I get this:
> > > > 
> > > > > + ./src/u-boot/test/py/test.py --bd sandbox --build-dir /var/lib/jenkins/workspace/u-boot-github_swarren_u-boot-tegra_dev-test-py/U_BOOT_BOARD/sandbox/build/u-boot/sandbox
> > > > > -k 'not sleep'
> > > > > Traceback (most recent call last):
> > > > >    File "./src/u-boot/test/py/test.py", line 20, in <module>
> > > > >      sys.exit(load_entry_point('pytest', 'console_scripts',
> > > > > 'pytest')(args))
> > > > >    File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py",
> > > > > line 542, in load_entry_point
> > > > >      return get_distribution(dist).load_entry_point(group, name)
> > > > >    File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py",
> > > > > line 2568, in load_entry_point
> > > > >      raise ImportError("Entry point %r not found" % ((group, name),))
> > > > > ImportError: Entry point ('console_scripts', 'pytest') not found
> > > > 
> > > > > [swarren at swarren-lx1 u-boot]$ dpkg -l|grep py|grep test
> > > > > ii  python-pytest    2.8.7-4
> > > > > ii  python3-pytest   2.8.7-4
> > > > 
> > > > I guess that's much older than the pytest in requirements.txt (5.2.1!!).
> > > > While I think we can require a newish version of Python (3.5) since
> > > > that's widely available on supported OSs, I don't think we want to force
> > > > people to have a very new version of all the packages (which supported
> > > > distros don't package yet), or change they way they run pytest (by
> > > > requiring use of a virtualenv and requirements.txt) since that mechanism
> > > > isn't supported on old branches of U-Boot, which will then require all
> > > > kinds of special cases in scripts that invoke the U-Boot test system.
> > > 
> > > Ah. The issue is the name of the entry point; the last parameter to
> > > load_entry_point(). "py.test" is supported everywhere, but only newer
> > > versions of pytesst have a "pytest" entry point. Changing that parameter to
> > > "py.test" makes the wrapper script work. Now every other test fails; I will
> > > see if there's an easy fix for that or not.
> > 
> > We're going to have to make a cut-off point somewhere it looks like.
> > pytest 2.8.7 (which we had pinned to) is incompatible with pytest 4.x
> > and throws a lot of warnings I believe on pytest 3.x..
> > 
> > What I really want to avoid is the problem we have today where we've
> > actively pinned ourselves to a now quite out of date version and have a
> > lot of problems showing up.
> > 
> > So we aren't going to have one environment that works for history and
> > future but I am hoping / trying to plan such that if we have
> > requirements.txt we can say from here until the next new best practices
> > (which today would be using pipenv, but that starts out with pip install
> > pipenv, sigh...) we can say to use requirements.txt and for history,
> > python2 + pytest 2.8.7 and local setups could even dump that in to a
> > requirements.txt file if that makes scripting easier.
> 
> Indeed the pytest 4 changes were a lot of the problem, but there was plenty
> more. It turned out to be too painful to fix it all.
> 
> After a lot of pain, I've switched my system over to using a virtualenv when
> there's a test/py/requirements.txt file in the source. Using a virtualenv
> broke some of the HW remote control scripts, which were written for Python2
> but just said "#!/usr/bin/env python". Fixing that to request Python 2
> explicitly fixed that issue. Another pain was that the ftdi1 module (used by
> the HW remote control scripts) isn't available from pip so can't be
> installed in a virtualenv. Luckily explicitly using python2 for those
> scripts makes this irrelevant, since system libraries are used, since the
> python2 interpreter doesn't come from the virtualenv in that case. However,
> if that didn't fix the issue, then I'd have needed extra options to set up
> the virtualenv in order to rely on Ubuntu-provided package for the module.
> I'm quoting them below just in case someone searching the archives finds
> this later in the future!
> 
> > +if [ -f ./src/u-boot/test/py/requirements.txt ]; then
> > +    rm -rf "${venv_dir}"
> > +    # --system-site-packages: Need ftdi1 module which is not available via pip,
> > +    #   but is installed on the system
> > +    virtualenv --system-site-packages -p /usr/bin/python3 "${venv_dir}"
> > +    . "${venv_dir}/bin/activate"
> > +    # -U: upgrade: Ensure the package is installed in the venv, at the version
> > +    #   requested in requirements.txt, even if already available on the host.
> > +    pip install -U -r ./src/u-boot/test/py/requirements.txt
> > +fi
> 
> So in summary, with the Python 3.5 fix I already sent, then this series,
> Tested-by: Stephen Warren <swarren@nvidia.com>

I appreciate it, thanks!

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20191023/4b65a1f2/attachment.sig>

[U-Boot] [PATCHv2 04/13] test/py: Fix pytest4 deprecation warnings

[Simon Glass]

On Tue, 22 Oct 2019 at 21:20, Tom Rini <trini@konsulko.com> wrote:
>
> From: Marek Vasut <marek.vasut@gmail.com>
>
> Fix the following spit from pytest:
>
> u-boot/test/py/conftest.py:438: RemovedInPytest4Warning: MarkInfo objects are deprecated as they contain merged marks which are hard to deal with correctly.
>   Please use node.get_closest_marker(name) or node.iter_markers(name).
>   Docs: https://docs.pytest.org/en/latest/mark.html#updating-code
>     for board in mark.args:
>
> In both cases, the later suggestion is applicable.
>
> Reviewed-by: Stephen Warren <swarren@nvidia.com>
> Signed-off-by: Marek Vasut <marek.vasut+renesas@gmail.com>
> Cc: Igor Opaniuk <igor.opaniuk@linaro.org>
> Cc: Tom Rini <trini@konsulko.com>
> Cc: Simon Glass <sjg@chromium.org>
> [trini: Update for current file with a few more cases, un-pin pytest in CI]
> Signed-off-by: Tom Rini <trini@konsulko.com>
> ---
>  .gitlab-ci.yml      |  2 +-
>  .travis.yml         |  2 +-
>  test/py/conftest.py | 30 ++++++++++++------------------
>  3 files changed, 14 insertions(+), 20 deletions(-)

Reviewed-by: Simon Glass <sjg@chromium.org>
Tested on sandbox:
Tested-by: Simon Glass <sjg@chromium.org>

[U-Boot] [PATCHv2 05/13] test/py: Automated conversion to Python 3

[Simon Glass]

On Tue, 22 Oct 2019 at 21:20, Tom Rini <trini@konsulko.com> wrote:
>
> Use the 2to3 tool to perform numerous automatic conversions from Python
> 2 syntax to Python 3.  Also fix whitespace problems that Python 3
> catches that Python 2 did not.
>
> Reviewed-by: Stephen Warren <swarren@nvidia.com>
> Signed-off-by: Tom Rini <trini@konsulko.com>
> ---
>  test/py/conftest.py               |  9 ++--
>  test/py/multiplexed_log.py        |  4 +-
>  test/py/test.py                   |  4 +-
>  test/py/tests/test_fit.py         |  2 -
>  test/py/tests/test_fpga.py        | 46 +++++++++----------
>  test/py/tests/test_fs/conftest.py |  4 +-
>  test/py/tests/test_log.py         |  8 ++--
>  test/py/tests/test_mmc_wr.py      | 74 +++++++++++++++----------------
>  8 files changed, 71 insertions(+), 80 deletions(-)

Reviewed-by: Simon Glass <sjg@chromium.org>
Tested on sandbox:
Tested-by: Simon Glass <sjg@chromium.org>

[U-Boot] [PATCHv2 03/13] test/py: Split mark to multiple lines

[Simon Glass]

On Tue, 22 Oct 2019 at 21:20, Tom Rini <trini@konsulko.com> wrote:
>
> We inconsistently note multiple dependencies today in our tests,
> sometimes with a single line that declares multiple and sometimes
> multiple single lines.  Current pytest seems to fail on the single line
> format so change to multiple declarations.
>
> Reviewed-by: Stephen Warren <swarren@nvidia.com>
> Signed-off-by: Tom Rini <trini@konsulko.com>
> ---
>  test/py/tests/test_android/test_avb.py | 9 ++++++---
>  test/py/tests/test_mmc_wr.py           | 4 +++-
>  2 files changed, 9 insertions(+), 4 deletions(-)

Reviewed-by: Simon Glass <sjg@chromium.org>
Tested on sandbox:
Tested-by: Simon Glass <sjg@chromium.org>

[U-Boot] [PATCHv2 07/13] test/py: test_ut.py: Ensure we use bytes

[Simon Glass]

On Tue, 22 Oct 2019 at 21:20, Tom Rini <trini@konsulko.com> wrote:
>
> In the case of some unit tests we are working with providing a fake
> flash device that we have written some text strings in to.  In this case
> we want to tell Python to encode things to bytes for us.
>
> Reviewed-by: Stephen Warren <swarren@nvidia.com>
> Signed-off-by: Tom Rini <trini@konsulko.com>
> ---
>  test/py/tests/test_ut.py | 6 +++---
>  1 file changed, 3 insertions(+), 3 deletions(-)

Reviewed-by: Simon Glass <sjg@chromium.org>
Tested on sandbox:
Tested-by: Simon Glass <sjg@chromium.org>

[U-Boot] [PATCHv2 06/13] test/py: Manual python3 fixes

[Simon Glass]

On Tue, 22 Oct 2019 at 21:20, Tom Rini <trini@konsulko.com> wrote:
>
> - Modern pytest is more visible in telling us about parameters that we
>   had not described, so describe a few more.
> - ConfigParser.readfp(...) is now configparser.read_file(...)
> - As part of the "strings vs bytes" conversions in Python 3, we use the
>   default encoding/decoding of utf-8 but in some places tell Python to
>   replace problematic conversions rather than throw a fatal error.
> - Fix a typo noticed while doing the above ("tot he" -> "to the").
> - As suggested by Stephen, re-alphabetize the import list
> - Per Heinrich, replace how we write contents in test_fit.py
>
> Signed-off-by: Tom Rini <trini@konsulko.com>
> ---
>  test/py/conftest.py        | 9 ++++-----
>  test/py/multiplexed_log.py | 9 +++++----
>  test/py/pytest.ini         | 3 +++
>  test/py/tests/test_fit.py  | 4 ++--
>  test/py/u_boot_spawn.py    | 4 ++--
>  5 files changed, 16 insertions(+), 13 deletions(-)

Reviewed-by: Simon Glass <sjg@chromium.org>
Tested on sandbox:
Tested-by: Simon Glass <sjg@chromium.org>

(I agree about supporting the older Python 3, BTW)

[U-Boot] [PATCHv2 08/13] test/py: test_efi_selftest.py: Updates for python 3 support

[Simon Glass]

On Tue, 22 Oct 2019 at 21:20, Tom Rini <trini@konsulko.com> wrote:
>
> - In python 3 you must use raw strings for regex as other forms are
>   deprecated and would require further changes to the pattern here.
>   In one case this lets us have a simpler match pattern.
> - As strings are now Unicode our complex tests (Euro symbol,
>   SHIFT+ALT+FN 5) we need to declare that as a bytes string and then
>   decode it for use.
>
> Cc: Heinrich Schuchardt <xypron.glpk@gmx.de>
> Signed-off-by: Tom Rini <trini@konsulko.com>
> ---
> Changes in v2:
> - New patch
> ---
>  test/py/tests/test_efi_selftest.py | 36 +++++++++++++++---------------
>  1 file changed, 18 insertions(+), 18 deletions(-)

Reviewed-by: Simon Glass <sjg@chromium.org>
Tested on sandbox:
Tested-by: Simon Glass <sjg@chromium.org>

[U-Boot] [PATCHv2 09/13] test/py: Update test_fs to decode check_output calls

[Simon Glass]

On Tue, 22 Oct 2019 at 21:20, Tom Rini <trini@konsulko.com> wrote:
>
> The check_output function from the subprocess Python module by default
> returns data as encoded bytes and leaves decoding to the application.
> Given our uses of the call, it makes the most sense to immediately
> decode the results.
>
> Signed-off-by: Tom Rini <trini@konsulko.com>
> ---
> Changes in v2:
> - New patch
> ---
>  test/py/tests/test_fs/conftest.py | 24 ++++++++++++------------
>  1 file changed, 12 insertions(+), 12 deletions(-)

Reviewed-by: Simon Glass <sjg@chromium.org>
Tested on sandbox:
Tested-by: Simon Glass <sjg@chromium.org>

[U-Boot] [PATCHv2 10/13] test/py: Rework test.py to be a different kind of wrapper

[Simon Glass]

On Tue, 22 Oct 2019 at 21:20, Tom Rini <trini@konsulko.com> wrote:
>
> Now that we have moved to being based on pytest for python3 we need to
> make our test.py wrapper more robust in terms of only calling python3
> rather than possibly finding and using python2.  To do this, change from
> execvp()'ing pytest to invoking the package itself via python.  In the
> event that pytest is unavailable we still get a user-friendly error:
>
> pkg_resources.DistributionNotFound: The 'pytest' distribution was not found and is required by the application
>
> Cc: Stephen Warren <swarren@nvidia.com>
> Signed-off-by: Tom Rini <trini@konsulko.com>
> ---
> Changes in v2:
> - New patch
> ---
>  test/py/test.py | 20 ++++----------------
>  1 file changed, 4 insertions(+), 16 deletions(-)

Reviewed-by: Simon Glass <sjg@chromium.org>
Tested on sandbox:
Tested-by: Simon Glass <sjg@chromium.org>

[U-Boot] [PATCHv2 11/13] test/py: Update docs, add requirements.txt for pip

[Simon Glass]

On Tue, 22 Oct 2019 at 21:20, Tom Rini <trini@konsulko.com> wrote:
>
> To be more closely aligned with Python community best practices, we need
> to better document our usage of pip and make use of a requirements.txt
> file that shows the versions of the tools that we are using.  This will
> aide in ensuring reproducibility of our tests as well.
>
> Signed-off-by: Tom Rini <trini@konsulko.com>
> ---
>  test/py/README.md        | 45 +++++++++++++++++++++-------------------
>  test/py/requirements.txt | 22 ++++++++++++++++++++
>  2 files changed, 46 insertions(+), 21 deletions(-)
>  create mode 100644 test/py/requirements.txt

Reviewed-by: Simon Glass <sjg@chromium.org>
Tested on sandbox:
Tested-by: Simon Glass <sjg@chromium.org>

I don't like / use pip unless I can help it, but this seems to be correct.


- Simon

[U-Boot] [PATCHv2 12/13] gitlab/travis: Rework how and when we use virtualenv in order to use python3

[Simon Glass]

On Tue, 22 Oct 2019 at 21:20, Tom Rini <trini@konsulko.com> wrote:
>
> As things stand today, we have tools that CI requires where "python"
> must be "python2".  We need to use a virtualenv and pip in order to
> ensure that our pytest tests can be run.  Rework things slightly so
> that:
> - On Travis-CI, we install python-pyelftools for the platforms that
>   require pyelftools to be installed.
> - On GitLab-CI, we move to a newer base image that includes python3-pip
>   and continue to use a virtualenv per job that needs it, for the
>   correct set of packages.
>
> Signed-off-by: Tom Rini <trini@konsulko.com>
> ---
> Changes in v2:
> - Add Travis-CI support here, and stop calling it a work-around.  This
>   is just the path forward until we have everything using python3 and at
>   that point evaluate what the right changes, if any, are.
> ---
>  .gitlab-ci.yml | 10 ++++------
>  .travis.yml    | 30 +++++++++++++++---------------
>  2 files changed, 19 insertions(+), 21 deletions(-)

Reviewed-by: Simon Glass <sjg@chromium.org>

[U-Boot] [PATCHv2 13/13] test/py: Use raw strings more to avoid deprecation warnings

[Simon Glass]

On Tue, 22 Oct 2019 at 21:20, Tom Rini <trini@konsulko.com> wrote:
>
> We have two further uses of raw string usage in the test/py codebase
> that are used under CI.  The first of which is under the bind test and
> is a direct update.  The second of which is to strip VT100 codes from
> the match buffer.  While switching this to a raw string is also a direct
> update, the comment it notes that problems were encountered on Ubuntu
> 14.04 (and whatever Python 2 version that was) that required slight
> tweaks to the regex.  Replace that now that we're saying Python 3.5 is
> the minimum.
>
> Cc: Stephen Warren <swarren@nvidia.com>
> Signed-off-by: Tom Rini <trini@konsulko.com>
> ---
> Changes in v2:
> - New patch
> ---
>  test/py/tests/test_bind.py | 4 ++--
>  test/py/u_boot_spawn.py    | 5 +----
>  2 files changed, 3 insertions(+), 6 deletions(-)

Reviewed-by: Simon Glass <sjg@chromium.org>
Tested on sandbox:
Tested-by: Simon Glass <sjg@chromium.org>

[U-Boot] [PATCHv3 00/13] Moving test/py to Python 3

[Tom Rini]

So, here's the third and I think final pass at moving our test/py code
over to Python 3.  Over v2 I've integrated reviews/test tags and
integrated Stephen's changes to allow for Python 3.5 to be used still.

I do want to note that while I do understand the desire to use the
distribution packages for python modules I don't think that's
reproducible enough long term.  That's what lead to our current pinning
to pytest 2.8.7 (incompatible upstream changes) and there's just so many
different versions packaged in different releaes.

I'll post the minor patches for the Docker images as well in reply to
this message shortly.

Thanks again for the reviews, changes and suggestions all!

-- 
Tom

[U-Boot] [PATCHv3 01/13] gitlab-ci: Fix indentation in some stanzas

[Tom Rini]

In a number of our stanzas we had multi-line commands that were one
space short of alignment, correct this.

Reviewed-by: Stephen Warren <swarren@nvidia.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
Tested-by: Stephen Warren <swarren@nvidia.com>
Signed-off-by: Tom Rini <trini@konsulko.com>
---
 .gitlab-ci.yml | 40 ++++++++++++++++++++--------------------
 1 file changed, 20 insertions(+), 20 deletions(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 967abed9f250..b91b5f67af98 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -65,11 +65,11 @@ build all 32bit ARM platforms:
   stage: world build
   script:
     - ret=0;
-     ./tools/buildman/buildman -o /tmp -P -E arm -x aarch64 || ret=$?;
-     if [[ $ret -ne 0 && $ret -ne 129 ]]; then
-       ./tools/buildman/buildman -o /tmp -sdeP;
-       exit $ret;
-     fi;
+      ./tools/buildman/buildman -o /tmp -P -E arm -x aarch64 || ret=$?;
+      if [[ $ret -ne 0 && $ret -ne 129 ]]; then
+        ./tools/buildman/buildman -o /tmp -sdeP;
+        exit $ret;
+      fi;
 
 build all 64bit ARM platforms:
   tags: [ 'all' ]
@@ -79,33 +79,33 @@ build all 64bit ARM platforms:
     - . /tmp/venv/bin/activate
     - pip install pyelftools
     - ret=0;
-     ./tools/buildman/buildman -o /tmp -P -E aarch64 || ret=$?;
-     if [[ $ret -ne 0 && $ret -ne 129 ]]; then
-       ./tools/buildman/buildman -o /tmp -sdeP;
-       exit $ret;
-     fi;
+      ./tools/buildman/buildman -o /tmp -P -E aarch64 || ret=$?;
+      if [[ $ret -ne 0 && $ret -ne 129 ]]; then
+        ./tools/buildman/buildman -o /tmp -sdeP;
+        exit $ret;
+      fi;
 
 build all PowerPC platforms:
   tags: [ 'all' ]
   stage: world build
   script:
     - ret=0;
-     ./tools/buildman/buildman -o /tmp -P -E powerpc || ret=$?;
-     if [[ $ret -ne 0 && $ret -ne 129 ]]; then
-       ./tools/buildman/buildman -o /tmp -sdeP;
-       exit $ret;
-     fi;
+      ./tools/buildman/buildman -o /tmp -P -E powerpc || ret=$?;
+      if [[ $ret -ne 0 && $ret -ne 129 ]]; then
+        ./tools/buildman/buildman -o /tmp -sdeP;
+        exit $ret;
+      fi;
 
 build all other platforms:
   tags: [ 'all' ]
   stage: world build
   script:
     - ret=0;
-     ./tools/buildman/buildman -o /tmp -P -E -x arm,powerpc || ret=$?;
-     if [[ $ret -ne 0 && $ret -ne 129 ]]; then
-       ./tools/buildman/buildman -o /tmp -sdeP;
-       exit $ret;
-     fi;
+      ./tools/buildman/buildman -o /tmp -P -E -x arm,powerpc || ret=$?;
+      if [[ $ret -ne 0 && $ret -ne 129 ]]; then
+        ./tools/buildman/buildman -o /tmp -sdeP;
+        exit $ret;
+      fi;
 
 # QA jobs for code analytics
 # static code analysis with cppcheck (we can add --enable=all later)
-- 
2.17.1

[U-Boot] [PATCHv3 02/13] gitlab-ci: Prepend to PATH rather than replace it

[Tom Rini]

Currently we set the entire PATH rather than prepend the new paths that
we need to have searched.  This however breaks parts of the "virtualenv"
that was have set up and need to use as that also will be modifying
PATH.  To fix this, prepend our new locations instead.

Reviewed-by: Stephen Warren <swarren@nvidia.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
Tested-by: Stephen Warren <swarren@nvidia.com>
Signed-off-by: Tom Rini <trini@konsulko.com>
---
Changes in v2:
- Reword slightly
---
 .gitlab-ci.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index b91b5f67af98..ab294997e463 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -48,7 +48,7 @@ stages:
     # "-k something" even when $TEST_PY_TEST_SPEC doesnt need a custom
     # value.
     - export UBOOT_TRAVIS_BUILD_DIR=/tmp/.bm-work/${TEST_PY_BD};
-      export PATH=/opt/qemu/bin:/tmp/uboot-test-hooks/bin:/usr/bin:/bin;
+      export PATH=/opt/qemu/bin:/tmp/uboot-test-hooks/bin:${PATH};
       export PYTHONPATH=/tmp/uboot-test-hooks/py/travis-ci;
       if [[ "${TEST_PY_BD}" != "" ]]; then
         ./test/py/test.py --bd ${TEST_PY_BD} ${TEST_PY_ID}
-- 
2.17.1

[U-Boot] [PATCHv3 03/13] test/py: Split mark to multiple lines

[Tom Rini]

We inconsistently note multiple dependencies today in our tests,
sometimes with a single line that declares multiple and sometimes
multiple single lines.  Current pytest seems to fail on the single line
format so change to multiple declarations.

Reviewed-by: Stephen Warren <swarren@nvidia.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
Tested-by: Stephen Warren <swarren@nvidia.com>
Tested-by: Simon Glass <sjg@chromium.org> [on sandbox]
Signed-off-by: Tom Rini <trini@konsulko.com>
---
 test/py/tests/test_android/test_avb.py | 9 ++++++---
 test/py/tests/test_mmc_wr.py           | 4 +++-
 2 files changed, 9 insertions(+), 4 deletions(-)

diff --git a/test/py/tests/test_android/test_avb.py b/test/py/tests/test_android/test_avb.py
index 813242343555..20ccaf6712f2 100644
--- a/test/py/tests/test_android/test_avb.py
+++ b/test/py/tests/test_android/test_avb.py
@@ -23,7 +23,8 @@ mmc_dev = 1
 temp_addr = 0x90000000
 temp_addr2 = 0x90002000
 
- at pytest.mark.buildconfigspec('cmd_avb', 'cmd_mmc')
+ at pytest.mark.buildconfigspec('cmd_avb')
+ at pytest.mark.buildconfigspec('cmd_mmc')
 def test_avb_verify(u_boot_console):
     """Run AVB 2.0 boot verification chain with avb subset of commands
     """
@@ -36,7 +37,8 @@ def test_avb_verify(u_boot_console):
     assert response.find(success_str)
 
 
- at pytest.mark.buildconfigspec('cmd_avb', 'cmd_mmc')
+ at pytest.mark.buildconfigspec('cmd_avb')
+ at pytest.mark.buildconfigspec('cmd_mmc')
 def test_avb_mmc_uuid(u_boot_console):
     """Check if 'avb get_uuid' works, compare results with
     'part list mmc 1' output
@@ -93,7 +95,8 @@ def test_avb_is_unlocked(u_boot_console):
     assert response == 'Unlocked = 1'
 
 
- at pytest.mark.buildconfigspec('cmd_avb', 'cmd_mmc')
+ at pytest.mark.buildconfigspec('cmd_avb')
+ at pytest.mark.buildconfigspec('cmd_mmc')
 def test_avb_mmc_read(u_boot_console):
     """Test mmc read operation
     """
diff --git a/test/py/tests/test_mmc_wr.py b/test/py/tests/test_mmc_wr.py
index 8b18781eac74..ab4c57974421 100644
--- a/test/py/tests/test_mmc_wr.py
+++ b/test/py/tests/test_mmc_wr.py
@@ -35,7 +35,9 @@ env__mmc_wr_configs = (
 
 """
 
- at pytest.mark.buildconfigspec('cmd_mmc','cmd_memory', 'cmd_random')
+ at pytest.mark.buildconfigspec('cmd_mmc')
+ at pytest.mark.buildconfigspec('cmd_memory')
+ at pytest.mark.buildconfigspec('cmd_random')
 def test_mmc_wr(u_boot_console, env__mmc_wr_config):
     """Test the "mmc write" command.
 
-- 
2.17.1

[U-Boot] [PATCHv3 04/13] test/py: Fix pytest4 deprecation warnings

[Tom Rini]

From: Marek Vasut <marek.vasut@gmail.com>

Fix the following spit from pytest:

u-boot/test/py/conftest.py:438: RemovedInPytest4Warning: MarkInfo objects are deprecated as they contain merged marks which are hard to deal with correctly.
  Please use node.get_closest_marker(name) or node.iter_markers(name).
  Docs: https://docs.pytest.org/en/latest/mark.html#updating-code
    for board in mark.args:

In both cases, the later suggestion is applicable.

Reviewed-by: Stephen Warren <swarren@nvidia.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
Signed-off-by: Marek Vasut <marek.vasut+renesas@gmail.com>
Cc: Igor Opaniuk <igor.opaniuk@gmail.com>
[trini: Update for current file with a few more cases, un-pin pytest in CI]
Tested-by: Simon Glass <sjg@chromium.org> [on sandbox]
Tested-by: Stephen Warren <swarren@nvidia.com>
Signed-off-by: Tom Rini <trini@konsulko.com>
---
 .gitlab-ci.yml      |  2 +-
 .travis.yml         |  2 +-
 test/py/conftest.py | 30 ++++++++++++------------------
 3 files changed, 14 insertions(+), 20 deletions(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index ab294997e463..5a34321570ce 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -20,7 +20,7 @@ stages:
     - ln -s travis-ci /tmp/uboot-test-hooks/py/`hostname`
     - virtualenv /tmp/venv
     - . /tmp/venv/bin/activate
-    - pip install pytest==2.8.7
+    - pip install pytest
     - pip install python-subunit
     - pip install coverage
     - grub-mkimage --prefix="" -o ~/grub_x86.efi -O i386-efi normal  echo lsefimmap lsefi lsefisystab efinet tftp minicmd
diff --git a/.travis.yml b/.travis.yml
index a3e7451bcb17..f61278f9656a 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -49,7 +49,7 @@ install:
  - cat ~/.buildman
  - virtualenv /tmp/venv
  - . /tmp/venv/bin/activate
- - pip install pytest==2.8.7
+ - pip install pytest
  - pip install python-subunit
  - pip install pyelftools
  - grub-mkimage --prefix="" -o ~/grub_x86.efi -O i386-efi normal  echo lsefimmap lsefi lsefisystab efinet tftp minicmd
diff --git a/test/py/conftest.py b/test/py/conftest.py
index 00d8ef8ba99c..30c898b40a0d 100644
--- a/test/py/conftest.py
+++ b/test/py/conftest.py
@@ -431,11 +431,9 @@ def setup_boardspec(item):
         Nothing.
     """
 
-    mark = item.get_marker('boardspec')
-    if not mark:
-        return
     required_boards = []
-    for board in mark.args:
+    for boards in item.iter_markers('boardspec'):
+        board = boards.args[0]
         if board.startswith('!'):
             if ubconfig.board_type == board[1:]:
                 pytest.skip('board "%s" not supported' % ubconfig.board_type)
@@ -459,16 +457,14 @@ def setup_buildconfigspec(item):
         Nothing.
     """
 
-    mark = item.get_marker('buildconfigspec')
-    if mark:
-        for option in mark.args:
-            if not ubconfig.buildconfig.get('config_' + option.lower(), None):
-                pytest.skip('.config feature "%s" not enabled' % option.lower())
-    notmark = item.get_marker('notbuildconfigspec')
-    if notmark:
-        for option in notmark.args:
-            if ubconfig.buildconfig.get('config_' + option.lower(), None):
-                pytest.skip('.config feature "%s" enabled' % option.lower())
+    for options in item.iter_markers('buildconfigspec'):
+        option = options.args[0]
+        if not ubconfig.buildconfig.get('config_' + option.lower(), None):
+            pytest.skip('.config feature "%s" not enabled' % option.lower())
+    for option in item.iter_markers('notbuildconfigspec'):
+        option = options.args[0]
+        if ubconfig.buildconfig.get('config_' + option.lower(), None):
+            pytest.skip('.config feature "%s" enabled' % option.lower())
 
 def tool_is_in_path(tool):
     for path in os.environ["PATH"].split(os.pathsep):
@@ -491,10 +487,8 @@ def setup_requiredtool(item):
         Nothing.
     """
 
-    mark = item.get_marker('requiredtool')
-    if not mark:
-        return
-    for tool in mark.args:
+    for tools in item.iter_markers('requiredtool'):
+        tool = tools.args[0]
         if not tool_is_in_path(tool):
             pytest.skip('tool "%s" not in $PATH' % tool)
 
-- 
2.17.1

[U-Boot] [PATCHv3 05/13] test/py: Automated conversion to Python 3

[Tom Rini]

Use the 2to3 tool to perform numerous automatic conversions from Python
2 syntax to Python 3.  Also fix whitespace problems that Python 3
catches that Python 2 did not.

Reviewed-by: Stephen Warren <swarren@nvidia.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
Tested-by: Stephen Warren <swarren@nvidia.com>
Tested-by: Simon Glass <sjg@chromium.org> [on sandbox]
Signed-off-by: Tom Rini <trini@konsulko.com>
---
 test/py/conftest.py               |  9 ++--
 test/py/multiplexed_log.py        |  4 +-
 test/py/test.py                   |  4 +-
 test/py/tests/test_fit.py         |  2 -
 test/py/tests/test_fpga.py        | 46 +++++++++----------
 test/py/tests/test_fs/conftest.py |  4 +-
 test/py/tests/test_log.py         |  8 ++--
 test/py/tests/test_mmc_wr.py      | 74 +++++++++++++++----------------
 8 files changed, 71 insertions(+), 80 deletions(-)

diff --git a/test/py/conftest.py b/test/py/conftest.py
index 30c898b40a0d..5c19af1d5034 100644
--- a/test/py/conftest.py
+++ b/test/py/conftest.py
@@ -19,13 +19,10 @@ import os.path
 import pytest
 from _pytest.runner import runtestprotocol
 import re
-import StringIO
+import io
 import sys
 
-try:
-    import configparser
-except:
-    import ConfigParser as configparser
+import configparser
 
 # Globals: The HTML log file, and the connection to the U-Boot console.
 log = None
@@ -169,7 +166,7 @@ def pytest_configure(config):
 
         with open(dot_config, 'rt') as f:
             ini_str = '[root]\n' + f.read()
-            ini_sio = StringIO.StringIO(ini_str)
+            ini_sio = io.StringIO(ini_str)
             parser = configparser.RawConfigParser()
             parser.readfp(ini_sio)
             ubconfig.buildconfig.update(parser.items('root'))
diff --git a/test/py/multiplexed_log.py b/test/py/multiplexed_log.py
index 637a3bd257ba..de0aacc659b8 100644
--- a/test/py/multiplexed_log.py
+++ b/test/py/multiplexed_log.py
@@ -5,8 +5,8 @@
 # Generate an HTML-formatted log file containing multiple streams of data,
 # each represented in a well-delineated/-structured fashion.
 
-import cgi
 import datetime
+import html
 import os.path
 import shutil
 import subprocess
@@ -334,7 +334,7 @@ $(document).ready(function () {
         data = data.replace(chr(13), '')
         data = ''.join((ord(c) in self._nonprint) and ('%%%02x' % ord(c)) or
                        c for c in data)
-        data = cgi.escape(data)
+        data = html.escape(data)
         return data
 
     def _terminate_stream(self):
diff --git a/test/py/test.py b/test/py/test.py
index a5140945d4b9..0ce1838833f6 100755
--- a/test/py/test.py
+++ b/test/py/test.py
@@ -1,4 +1,4 @@
-#!/usr/bin/env python2
+#!/usr/bin/env python3
 # SPDX-License-Identifier: GPL-2.0
 
 # Copyright (c) 2015 Stephen Warren
@@ -7,8 +7,6 @@
 # Wrapper script to invoke pytest with the directory name that contains the
 # U-Boot tests.
 
-from __future__ import print_function
-
 import os
 import os.path
 import sys
diff --git a/test/py/tests/test_fit.py b/test/py/tests/test_fit.py
index e3210ed43fa4..4922b9dcc664 100755
--- a/test/py/tests/test_fit.py
+++ b/test/py/tests/test_fit.py
@@ -3,8 +3,6 @@
 #
 # Sanity check of the FIT handling in U-Boot
 
-from __future__ import print_function
-
 import os
 import pytest
 import struct
diff --git a/test/py/tests/test_fpga.py b/test/py/tests/test_fpga.py
index e3bb7b41c749..ca7ef8ea40d6 100644
--- a/test/py/tests/test_fpga.py
+++ b/test/py/tests/test_fpga.py
@@ -175,29 +175,29 @@ def test_fpga_load_fail(u_boot_console):
     f, dev, addr, bit, bit_size = load_file_from_var(u_boot_console, 'bitstream_load')
 
     for cmd in ['dump', 'load', 'loadb']:
-	# missing dev parameter
-	expected = 'fpga: incorrect parameters passed'
-	output = u_boot_console.run_command('fpga %s %x $filesize' % (cmd, addr))
-	#assert expected in output
-	assert expected_usage in output
-
-	# more parameters - 0 at the end
-	expected = 'fpga: more parameters passed'
-	output = u_boot_console.run_command('fpga %s %x %x $filesize 0' % (cmd, dev, addr))
-	#assert expected in output
-	assert expected_usage in output
-
-	# 0 address
-	expected = 'fpga: zero fpga_data address'
-	output = u_boot_console.run_command('fpga %s %x 0 $filesize' % (cmd, dev))
-	#assert expected in output
-	assert expected_usage in output
-
-	# 0 filesize
-	expected = 'fpga: zero size'
-	output = u_boot_console.run_command('fpga %s %x %x 0' % (cmd, dev, addr))
-	#assert expected in output
-	assert expected_usage in output
+        # missing dev parameter
+        expected = 'fpga: incorrect parameters passed'
+        output = u_boot_console.run_command('fpga %s %x $filesize' % (cmd, addr))
+        #assert expected in output
+        assert expected_usage in output
+
+        # more parameters - 0 at the end
+        expected = 'fpga: more parameters passed'
+        output = u_boot_console.run_command('fpga %s %x %x $filesize 0' % (cmd, dev, addr))
+        #assert expected in output
+        assert expected_usage in output
+
+        # 0 address
+        expected = 'fpga: zero fpga_data address'
+        output = u_boot_console.run_command('fpga %s %x 0 $filesize' % (cmd, dev))
+        #assert expected in output
+        assert expected_usage in output
+
+        # 0 filesize
+        expected = 'fpga: zero size'
+        output = u_boot_console.run_command('fpga %s %x %x 0' % (cmd, dev, addr))
+        #assert expected in output
+        assert expected_usage in output
 
 @pytest.mark.buildconfigspec('cmd_fpga')
 @pytest.mark.buildconfigspec('cmd_echo')
diff --git a/test/py/tests/test_fs/conftest.py b/test/py/tests/test_fs/conftest.py
index 9324657d2160..354d17672fe4 100644
--- a/test/py/tests/test_fs/conftest.py
+++ b/test/py/tests/test_fs/conftest.py
@@ -508,8 +508,8 @@ def fs_obj_unlink(request, u_boot_config):
 
         # Test Case 2
         check_call('mkdir %s/dir2' % mount_dir, shell=True)
-	for i in range(0, 20):
-	    check_call('mkdir %s/dir2/0123456789abcdef%02x'
+        for i in range(0, 20):
+            check_call('mkdir %s/dir2/0123456789abcdef%02x'
                                     % (mount_dir, i), shell=True)
 
         # Test Case 4
diff --git a/test/py/tests/test_log.py b/test/py/tests/test_log.py
index cb183444c6fc..75325fad61ad 100644
--- a/test/py/tests/test_log.py
+++ b/test/py/tests/test_log.py
@@ -27,9 +27,9 @@ def test_log(u_boot_console):
         """
         for i in range(max_level):
             if mask & 1:
-                assert 'log_run() log %d' % i == lines.next()
+                assert 'log_run() log %d' % i == next(lines)
             if mask & 3:
-                assert 'func() _log %d' % i == lines.next()
+                assert 'func() _log %d' % i == next(lines)
 
     def run_test(testnum):
         """Run a particular test number (the 'log test' command)
@@ -43,7 +43,7 @@ def test_log(u_boot_console):
            output = u_boot_console.run_command('log test %d' % testnum)
         split = output.replace('\r', '').splitlines()
         lines = iter(split)
-        assert 'test %d' % testnum == lines.next()
+        assert 'test %d' % testnum == next(lines)
         return lines
 
     def test0():
@@ -88,7 +88,7 @@ def test_log(u_boot_console):
     def test10():
         lines = run_test(10)
         for i in range(7):
-            assert 'log_test() level %d' % i == lines.next()
+            assert 'log_test() level %d' % i == next(lines)
 
     # TODO(sjg at chromium.org): Consider structuring this as separate tests
     cons = u_boot_console
diff --git a/test/py/tests/test_mmc_wr.py b/test/py/tests/test_mmc_wr.py
index ab4c57974421..05e5c1ee85da 100644
--- a/test/py/tests/test_mmc_wr.py
+++ b/test/py/tests/test_mmc_wr.py
@@ -67,41 +67,39 @@ def test_mmc_wr(u_boot_console, env__mmc_wr_config):
 
 
     for i in range(test_iterations):
-	# Generate random data
-	cmd = 'random %s %x' % (src_addr, count_bytes)
-	response = u_boot_console.run_command(cmd)
-	good_response = '%d bytes filled with random data' % (count_bytes)
-	assert good_response in response
-
-	# Select MMC device
-	cmd = 'mmc dev %d' % devid
-	if is_emmc:
-		cmd += ' %d' % partid
-	response = u_boot_console.run_command(cmd)
-	assert 'no card present' not in response
-	if is_emmc:
-		partid_response = "(part %d)" % partid
-	else:
-		partid_response = ""
-	good_response = 'mmc%d%s is current device' % (devid, partid_response)
-	assert good_response in response
-
-	# Write data
-	cmd = 'mmc write %s %x %x' % (src_addr, sector, count_sectors)
-	response = u_boot_console.run_command(cmd)
-	good_response = 'MMC write: dev # %d, block # %d, count %d ... %d blocks written: OK' % (
-		devid, sector, count_sectors, count_sectors)
-	assert good_response in response
-
-	# Read data
-	cmd = 'mmc read %s %x %x' % (dst_addr, sector, count_sectors)
-	response = u_boot_console.run_command(cmd)
-	good_response = 'MMC read: dev # %d, block # %d, count %d ... %d blocks read: OK' % (
-		devid, sector, count_sectors, count_sectors)
-	assert good_response in response
-
-	# Compare src and dst data
-	cmd = 'cmp.b %s %s %x' % (src_addr, dst_addr, count_bytes)
-	response = u_boot_console.run_command(cmd)
-	good_response = 'Total of %d byte(s) were the same' % (count_bytes)
-	assert good_response in response
+        # Generate random data
+        cmd = 'random %s %x' % (src_addr, count_bytes)
+        response = u_boot_console.run_command(cmd)
+        good_response = '%d bytes filled with random data' % (count_bytes)
+        assert good_response in response
+
+        # Select MMC device
+        cmd = 'mmc dev %d' % devid
+        if is_emmc:
+            cmd += ' %d' % partid
+        response = u_boot_console.run_command(cmd)
+        assert 'no card present' not in response
+        if is_emmc:
+            partid_response = "(part %d)" % partid
+        else:
+            partid_response = ""
+        good_response = 'mmc%d%s is current device' % (devid, partid_response)
+        assert good_response in response
+
+        # Write data
+        cmd = 'mmc write %s %x %x' % (src_addr, sector, count_sectors)
+        response = u_boot_console.run_command(cmd)
+        good_response = 'MMC write: dev # %d, block # %d, count %d ... %d blocks written: OK' % (devid, sector, count_sectors, count_sectors)
+        assert good_response in response
+
+        # Read data
+        cmd = 'mmc read %s %x %x' % (dst_addr, sector, count_sectors)
+        response = u_boot_console.run_command(cmd)
+        good_response = 'MMC read: dev # %d, block # %d, count %d ... %d blocks read: OK' % (devid, sector, count_sectors, count_sectors)
+        assert good_response in response
+
+        # Compare src and dst data
+        cmd = 'cmp.b %s %s %x' % (src_addr, dst_addr, count_bytes)
+        response = u_boot_console.run_command(cmd)
+        good_response = 'Total of %d byte(s) were the same' % (count_bytes)
+        assert good_response in response
-- 
2.17.1

[U-Boot] [PATCHv3 06/13] test/py: Manual python3 fixes

[Tom Rini]

- Modern pytest is more visible in telling us about parameters that we
  had not described, so describe a few more.
- ConfigParser.readfp(...) is now configparser.read_file(...)
- As part of the "strings vs bytes" conversions in Python 3, we use the
  default encoding/decoding of utf-8 but in some places tell Python to
  replace problematic conversions rather than throw a fatal error.
- Fix a typo noticed while doing the above ("tot he" -> "to the").
- As suggested by Stephen, re-alphabetize the import list
- Per Heinrich, replace how we write contents in test_fit.py

Reviewed-by: Simon Glass <sjg@chromium.org>
Tested-by: Stephen Warren <swarren@nvidia.com>
Tested-by: Simon Glass <sjg@chromium.org> [on sandbox]
Signed-off-by: Tom Rini <trini@konsulko.com>
---
Changes in v3:
- Switch to single quotes not double quotes, per Stephen
- Switch to a Python 3.5.x happy subprocess and decode set of
  statements, also from Stephen
---
 test/py/conftest.py        |  9 ++++-----
 test/py/multiplexed_log.py | 11 ++++++++---
 test/py/pytest.ini         |  3 +++
 test/py/tests/test_fit.py  |  4 ++--
 test/py/u_boot_spawn.py    |  4 ++--
 5 files changed, 19 insertions(+), 12 deletions(-)

diff --git a/test/py/conftest.py b/test/py/conftest.py
index 5c19af1d5034..bffee6b8a3a2 100644
--- a/test/py/conftest.py
+++ b/test/py/conftest.py
@@ -13,17 +13,16 @@
 # - Implementing custom pytest markers.
 
 import atexit
+import configparser
 import errno
+import io
 import os
 import os.path
 import pytest
-from _pytest.runner import runtestprotocol
 import re
-import io
+from _pytest.runner import runtestprotocol
 import sys
 
-import configparser
-
 # Globals: The HTML log file, and the connection to the U-Boot console.
 log = None
 console = None
@@ -168,7 +167,7 @@ def pytest_configure(config):
             ini_str = '[root]\n' + f.read()
             ini_sio = io.StringIO(ini_str)
             parser = configparser.RawConfigParser()
-            parser.readfp(ini_sio)
+            parser.read_file(ini_sio)
             ubconfig.buildconfig.update(parser.items('root'))
 
     ubconfig.test_py_dir = test_py_dir
diff --git a/test/py/multiplexed_log.py b/test/py/multiplexed_log.py
index de0aacc659b8..545a77430223 100644
--- a/test/py/multiplexed_log.py
+++ b/test/py/multiplexed_log.py
@@ -51,7 +51,7 @@ class LogfileStream(object):
         """Write data to the log stream.
 
         Args:
-            data: The data to write tot he file.
+            data: The data to write to the file.
             implicit: Boolean indicating whether data actually appeared in the
                 stream, or was implicitly generated. A valid use-case is to
                 repeat a shell prompt at the start of each separate log
@@ -64,7 +64,8 @@ class LogfileStream(object):
 
         self.logfile.write(self, data, implicit)
         if self.chained_file:
-            self.chained_file.write(data)
+            # Chained file is console, convert things a little
+            self.chained_file.write((data.encode('ascii', 'replace')).decode())
 
     def flush(self):
         """Flush the log stream, to ensure correct log interleaving.
@@ -136,6 +137,10 @@ class RunAndLog(object):
             p = subprocess.Popen(cmd, cwd=cwd,
                 stdin=None, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
             (stdout, stderr) = p.communicate()
+            if stdout is not None:
+                stdout = stdout.decode('utf-8')
+            if stderr is not None:
+                stderr = stderr.decode('utf-8')
             output = ''
             if stdout:
                 if stderr:
@@ -215,7 +220,7 @@ class Logfile(object):
             Nothing.
         """
 
-        self.f = open(fn, 'wt')
+        self.f = open(fn, 'wt', encoding='utf-8')
         self.last_stream = None
         self.blocks = []
         self.cur_evt = 1
diff --git a/test/py/pytest.ini b/test/py/pytest.ini
index 7e400682bf25..e93d010f1fa2 100644
--- a/test/py/pytest.ini
+++ b/test/py/pytest.ini
@@ -8,3 +8,6 @@
 markers =
     boardspec: U-Boot: Describes the set of boards a test can/can't run on.
     buildconfigspec: U-Boot: Describes Kconfig/config-header constraints.
+    notbuildconfigspec: U-Boot: Describes required disabled Kconfig options.
+    requiredtool: U-Boot: Required host tools for a test.
+    slow: U-Boot: Specific test will run slowly.
diff --git a/test/py/tests/test_fit.py b/test/py/tests/test_fit.py
index 4922b9dcc664..356d9a20f299 100755
--- a/test/py/tests/test_fit.py
+++ b/test/py/tests/test_fit.py
@@ -153,7 +153,7 @@ def test_fit(u_boot_console):
         src = make_fname('u-boot.dts')
         dtb = make_fname('u-boot.dtb')
         with open(src, 'w') as fd:
-            print(base_fdt, file=fd)
+            fd.write(base_fdt)
         util.run_and_log(cons, ['dtc', src, '-O', 'dtb', '-o', dtb])
         return dtb
 
@@ -186,7 +186,7 @@ def test_fit(u_boot_console):
         its = make_its(params)
         util.run_and_log(cons, [mkimage, '-f', its, fit])
         with open(make_fname('u-boot.dts'), 'w') as fd:
-            print(base_fdt, file=fd)
+            fd.write(base_fdt)
         return fit
 
     def make_kernel(filename, text):
diff --git a/test/py/u_boot_spawn.py b/test/py/u_boot_spawn.py
index b011a3e3da25..4f898cdefe5a 100644
--- a/test/py/u_boot_spawn.py
+++ b/test/py/u_boot_spawn.py
@@ -113,7 +113,7 @@ class Spawn(object):
             Nothing.
         """
 
-        os.write(self.fd, data)
+        os.write(self.fd, data.encode(errors='replace'))
 
     def expect(self, patterns):
         """Wait for the sub-process to emit specific data.
@@ -171,7 +171,7 @@ class Spawn(object):
                 events = self.poll.poll(poll_maxwait)
                 if not events:
                     raise Timeout()
-                c = os.read(self.fd, 1024)
+                c = os.read(self.fd, 1024).decode(errors='replace')
                 if not c:
                     raise EOFError()
                 if self.logfile_read:
-- 
2.17.1

[U-Boot] [PATCHv3 07/13] test/py: test_ut.py: Ensure we use bytes

[Tom Rini]

In the case of some unit tests we are working with providing a fake
flash device that we have written some text strings in to.  In this case
we want to tell Python to encode things to bytes for us.

Reviewed-by: Stephen Warren <swarren@nvidia.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
Tested-by: Stephen Warren <swarren@nvidia.com>
Tested-by: Simon Glass <sjg@chromium.org> [on sandbox]
Signed-off-by: Tom Rini <trini@konsulko.com>
---
 test/py/tests/test_ut.py | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/test/py/tests/test_ut.py b/test/py/tests/test_ut.py
index 62037d2c4530..6c7b8dd2b30e 100644
--- a/test/py/tests/test_ut.py
+++ b/test/py/tests/test_ut.py
@@ -10,14 +10,14 @@ def test_ut_dm_init(u_boot_console):
 
     fn = u_boot_console.config.source_dir + '/testflash.bin'
     if not os.path.exists(fn):
-        data = 'this is a test'
-        data += '\x00' * ((4 * 1024 * 1024) - len(data))
+        data = b'this is a test'
+        data += b'\x00' * ((4 * 1024 * 1024) - len(data))
         with open(fn, 'wb') as fh:
             fh.write(data)
 
     fn = u_boot_console.config.source_dir + '/spi.bin'
     if not os.path.exists(fn):
-        data = '\x00' * (2 * 1024 * 1024)
+        data = b'\x00' * (2 * 1024 * 1024)
         with open(fn, 'wb') as fh:
             fh.write(data)
 
-- 
2.17.1

[U-Boot] [PATCHv3 08/13] test/py: test_efi_selftest.py: Updates for python 3 support

[Tom Rini]

- In python 3 you must use raw strings for regex as other forms are
  deprecated and would require further changes to the pattern here.
  In one case this lets us have a simpler match pattern.
- As strings are now Unicode our complex tests (Euro symbol,
  SHIFT+ALT+FN 5) we need to declare that as a bytes string and then
  decode it for use.

Cc: Heinrich Schuchardt <xypron.glpk@gmx.de>
Reviewed-by: Simon Glass <sjg@chromium.org>
Tested-by: Stephen Warren <swarren@nvidia.com>
Tested-by: Simon Glass <sjg@chromium.org> [on sandbox]
Signed-off-by: Tom Rini <trini@konsulko.com>
---
Changes in v2:
- New patch
---
 test/py/tests/test_efi_selftest.py | 36 +++++++++++++++---------------
 1 file changed, 18 insertions(+), 18 deletions(-)

diff --git a/test/py/tests/test_efi_selftest.py b/test/py/tests/test_efi_selftest.py
index d5430f9c1275..ca01542088f1 100644
--- a/test/py/tests/test_efi_selftest.py
+++ b/test/py/tests/test_efi_selftest.py
@@ -59,7 +59,7 @@ def test_efi_selftest_text_input(u_boot_console):
 	u_boot_console.run_command(cmd='setenv efi_selftest text input')
 	output = u_boot_console.run_command(cmd='bootefi selftest',
 					    wait_for_prompt=False)
-	m = u_boot_console.p.expect(['To terminate type \'x\''])
+	m = u_boot_console.p.expect([r'To terminate type \'x\''])
 	if m != 0:
 		raise Exception('No prompt for \'text input\' test')
 	u_boot_console.drain_console()
@@ -68,7 +68,7 @@ def test_efi_selftest_text_input(u_boot_console):
 	u_boot_console.run_command(cmd=chr(4), wait_for_echo=False,
 				   send_nl=False, wait_for_prompt=False)
 	m = u_boot_console.p.expect(
-		['Unicode char 4 \(unknown\), scan code 0 \(Null\)'])
+		[r'Unicode char 4 \(unknown\), scan code 0 \(Null\)'])
 	if m != 0:
 		raise Exception('EOT failed in \'text input\' test')
 	u_boot_console.drain_console()
@@ -76,7 +76,7 @@ def test_efi_selftest_text_input(u_boot_console):
 	u_boot_console.run_command(cmd=chr(8), wait_for_echo=False,
 				   send_nl=False, wait_for_prompt=False)
 	m = u_boot_console.p.expect(
-		['Unicode char 8 \(BS\), scan code 0 \(Null\)'])
+		[r'Unicode char 8 \(BS\), scan code 0 \(Null\)'])
 	if m != 0:
 		raise Exception('BS failed in \'text input\' test')
 	u_boot_console.drain_console()
@@ -84,7 +84,7 @@ def test_efi_selftest_text_input(u_boot_console):
 	u_boot_console.run_command(cmd=chr(9), wait_for_echo=False,
 				   send_nl=False, wait_for_prompt=False)
 	m = u_boot_console.p.expect(
-		['Unicode char 9 \(TAB\), scan code 0 \(Null\)'])
+		[r'Unicode char 9 \(TAB\), scan code 0 \(Null\)'])
 	if m != 0:
 		raise Exception('BS failed in \'text input\' test')
 	u_boot_console.drain_console()
@@ -92,7 +92,7 @@ def test_efi_selftest_text_input(u_boot_console):
 	u_boot_console.run_command(cmd='a', wait_for_echo=False, send_nl=False,
 				   wait_for_prompt=False)
 	m = u_boot_console.p.expect(
-		['Unicode char 97 \(\'a\'\), scan code 0 \(Null\)'])
+		[r'Unicode char 97 \(\'a\'\), scan code 0 \(Null\)'])
 	if m != 0:
 		raise Exception('\'a\' failed in \'text input\' test')
 	u_boot_console.drain_console()
@@ -100,14 +100,14 @@ def test_efi_selftest_text_input(u_boot_console):
 	u_boot_console.run_command(cmd=chr(27) + '[A', wait_for_echo=False,
 				   send_nl=False, wait_for_prompt=False)
 	m = u_boot_console.p.expect(
-		['Unicode char 0 \(Null\), scan code 1 \(Up\)'])
+		[r'Unicode char 0 \(Null\), scan code 1 \(Up\)'])
 	if m != 0:
 		raise Exception('UP failed in \'text input\' test')
 	u_boot_console.drain_console()
 	# Euro sign
-	u_boot_console.run_command(cmd='\xe2\x82\xac', wait_for_echo=False,
+	u_boot_console.run_command(cmd=b'\xe2\x82\xac'.decode(), wait_for_echo=False,
 				   send_nl=False, wait_for_prompt=False)
-	m = u_boot_console.p.expect(['Unicode char 8364 \(\''])
+	m = u_boot_console.p.expect([r'Unicode char 8364 \(\''])
 	if m != 0:
 		raise Exception('Euro sign failed in \'text input\' test')
 	u_boot_console.drain_console()
@@ -129,7 +129,7 @@ def test_efi_selftest_text_input_ex(u_boot_console):
 	u_boot_console.run_command(cmd='setenv efi_selftest extended text input')
 	output = u_boot_console.run_command(cmd='bootefi selftest',
 					    wait_for_prompt=False)
-	m = u_boot_console.p.expect(['To terminate type \'CTRL\+x\''])
+	m = u_boot_console.p.expect([r'To terminate type \'CTRL\+x\''])
 	if m != 0:
 		raise Exception('No prompt for \'text input\' test')
 	u_boot_console.drain_console()
@@ -138,7 +138,7 @@ def test_efi_selftest_text_input_ex(u_boot_console):
 	u_boot_console.run_command(cmd=chr(4), wait_for_echo=False,
 				   send_nl=False, wait_for_prompt=False)
 	m = u_boot_console.p.expect(
-		['Unicode char 100 \\(\'d\'\\), scan code 0 \\(CTRL\\+Null\\)'])
+		[r'Unicode char 100 \(\'d\'\), scan code 0 \(CTRL\+Null\)'])
 	if m != 0:
 		raise Exception('EOT failed in \'text input\' test')
 	u_boot_console.drain_console()
@@ -146,7 +146,7 @@ def test_efi_selftest_text_input_ex(u_boot_console):
 	u_boot_console.run_command(cmd=chr(8), wait_for_echo=False,
 				   send_nl=False, wait_for_prompt=False)
 	m = u_boot_console.p.expect(
-		['Unicode char 8 \(BS\), scan code 0 \(\+Null\)'])
+		[r'Unicode char 8 \(BS\), scan code 0 \(\+Null\)'])
 	if m != 0:
 		raise Exception('BS failed in \'text input\' test')
 	u_boot_console.drain_console()
@@ -154,7 +154,7 @@ def test_efi_selftest_text_input_ex(u_boot_console):
 	u_boot_console.run_command(cmd=chr(9), wait_for_echo=False,
 				   send_nl=False, wait_for_prompt=False)
 	m = u_boot_console.p.expect(
-		['Unicode char 9 \(TAB\), scan code 0 \(\+Null\)'])
+		[r'Unicode char 9 \(TAB\), scan code 0 \(\+Null\)'])
 	if m != 0:
 		raise Exception('TAB failed in \'text input\' test')
 	u_boot_console.drain_console()
@@ -162,7 +162,7 @@ def test_efi_selftest_text_input_ex(u_boot_console):
 	u_boot_console.run_command(cmd='a', wait_for_echo=False, send_nl=False,
 				   wait_for_prompt=False)
 	m = u_boot_console.p.expect(
-		['Unicode char 97 \(\'a\'\), scan code 0 \(Null\)'])
+		[r'Unicode char 97 \(\'a\'\), scan code 0 \(Null\)'])
 	if m != 0:
 		raise Exception('\'a\' failed in \'text input\' test')
 	u_boot_console.drain_console()
@@ -170,23 +170,23 @@ def test_efi_selftest_text_input_ex(u_boot_console):
 	u_boot_console.run_command(cmd=chr(27) + '[A', wait_for_echo=False,
 				   send_nl=False, wait_for_prompt=False)
 	m = u_boot_console.p.expect(
-		['Unicode char 0 \(Null\), scan code 1 \(\+Up\)'])
+		[r'Unicode char 0 \(Null\), scan code 1 \(\+Up\)'])
 	if m != 0:
 		raise Exception('UP failed in \'text input\' test')
 	u_boot_console.drain_console()
 	# Euro sign
-	u_boot_console.run_command(cmd='\xe2\x82\xac', wait_for_echo=False,
+	u_boot_console.run_command(cmd=b'\xe2\x82\xac'.decode(), wait_for_echo=False,
 				   send_nl=False, wait_for_prompt=False)
-	m = u_boot_console.p.expect(['Unicode char 8364 \(\''])
+	m = u_boot_console.p.expect([r'Unicode char 8364 \(\''])
 	if m != 0:
 		raise Exception('Euro sign failed in \'text input\' test')
 	u_boot_console.drain_console()
 	# SHIFT+ALT+FN 5
-	u_boot_console.run_command(cmd='\x1b\x5b\x31\x35\x3b\x34\x7e',
+	u_boot_console.run_command(cmd=b'\x1b\x5b\x31\x35\x3b\x34\x7e'.decode(),
 				   wait_for_echo=False, send_nl=False,
 				   wait_for_prompt=False)
 	m = u_boot_console.p.expect(
-		['Unicode char 0 \(Null\), scan code 15 \(SHIFT\+ALT\+FN 5\)'])
+		[r'Unicode char 0 \(Null\), scan code 15 \(SHIFT\+ALT\+FN 5\)'])
 	if m != 0:
 		raise Exception('SHIFT+ALT+FN 5 failed in \'text input\' test')
 	u_boot_console.drain_console()
-- 
2.17.1

[U-Boot] [PATCHv3 09/13] test/py: Update test_fs to decode check_output calls

[Tom Rini]

The check_output function from the subprocess Python module by default
returns data as encoded bytes and leaves decoding to the application.
Given our uses of the call, it makes the most sense to immediately
decode the results.

Reviewed-by: Simon Glass <sjg@chromium.org>
Tested-by: Stephen Warren <swarren@nvidia.com>
Tested-by: Simon Glass <sjg@chromium.org> [on sandbox]
Signed-off-by: Tom Rini <trini@konsulko.com>
---
Changes in v2:
- New patch
---
 test/py/tests/test_fs/conftest.py | 24 ++++++++++++------------
 1 file changed, 12 insertions(+), 12 deletions(-)

diff --git a/test/py/tests/test_fs/conftest.py b/test/py/tests/test_fs/conftest.py
index 354d17672fe4..1949f916197a 100644
--- a/test/py/tests/test_fs/conftest.py
+++ b/test/py/tests/test_fs/conftest.py
@@ -300,38 +300,38 @@ def fs_obj_basic(request, u_boot_config):
         # Generate the md5sums of reads that we will test against small file
         out = check_output(
             'dd if=%s bs=1M skip=0 count=1 2> /dev/null | md5sum'
-	    % small_file, shell=True)
+	    % small_file, shell=True).decode()
         md5val = [ out.split()[0] ]
 
         # Generate the md5sums of reads that we will test against big file
         # One from beginning of file.
         out = check_output(
             'dd if=%s bs=1M skip=0 count=1 2> /dev/null | md5sum'
-	    % big_file, shell=True)
+	    % big_file, shell=True).decode()
         md5val.append(out.split()[0])
 
         # One from end of file.
         out = check_output(
             'dd if=%s bs=1M skip=2499 count=1 2> /dev/null | md5sum'
-	    % big_file, shell=True)
+	    % big_file, shell=True).decode()
         md5val.append(out.split()[0])
 
         # One from the last 1MB chunk of 2GB
         out = check_output(
             'dd if=%s bs=1M skip=2047 count=1 2> /dev/null | md5sum'
-	    % big_file, shell=True)
+	    % big_file, shell=True).decode()
         md5val.append(out.split()[0])
 
         # One from the start 1MB chunk from 2GB
         out = check_output(
             'dd if=%s bs=1M skip=2048 count=1 2> /dev/null | md5sum'
-	    % big_file, shell=True)
+	    % big_file, shell=True).decode()
         md5val.append(out.split()[0])
 
         # One 1MB chunk crossing the 2GB boundary
         out = check_output(
             'dd if=%s bs=512K skip=4095 count=2 2> /dev/null | md5sum'
-	    % big_file, shell=True)
+	    % big_file, shell=True).decode()
         md5val.append(out.split()[0])
 
         umount_fs(mount_dir)
@@ -390,7 +390,7 @@ def fs_obj_ext(request, u_boot_config):
             % min_file, shell=True)
         out = check_output(
             'dd if=%s bs=1K 2> /dev/null | md5sum'
-            % min_file, shell=True)
+            % min_file, shell=True).decode()
         md5val = [ out.split()[0] ]
 
         # Calculate md5sum of Test Case 4
@@ -399,7 +399,7 @@ def fs_obj_ext(request, u_boot_config):
         check_call('dd if=%s of=%s bs=1K seek=5 count=20'
             % (min_file, tmp_file), shell=True)
         out = check_output('dd if=%s bs=1K 2> /dev/null | md5sum'
-            % tmp_file, shell=True)
+            % tmp_file, shell=True).decode()
         md5val.append(out.split()[0])
 
         # Calculate md5sum of Test Case 5
@@ -408,7 +408,7 @@ def fs_obj_ext(request, u_boot_config):
         check_call('dd if=%s of=%s bs=1K seek=5 count=5'
             % (min_file, tmp_file), shell=True)
         out = check_output('dd if=%s bs=1K 2> /dev/null | md5sum'
-            % tmp_file, shell=True)
+            % tmp_file, shell=True).decode()
         md5val.append(out.split()[0])
 
         # Calculate md5sum of Test Case 7
@@ -417,7 +417,7 @@ def fs_obj_ext(request, u_boot_config):
         check_call('dd if=%s of=%s bs=1K seek=20 count=20'
             % (min_file, tmp_file), shell=True)
         out = check_output('dd if=%s bs=1K 2> /dev/null | md5sum'
-            % tmp_file, shell=True)
+            % tmp_file, shell=True).decode()
         md5val.append(out.split()[0])
 
         check_call('rm %s' % tmp_file, shell=True)
@@ -582,11 +582,11 @@ def fs_obj_symlink(request, u_boot_config):
         # Generate the md5sums of reads that we will test against small file
         out = check_output(
             'dd if=%s bs=1M skip=0 count=1 2> /dev/null | md5sum'
-            % small_file, shell=True)
+            % small_file, shell=True).decode()
         md5val = [out.split()[0]]
         out = check_output(
             'dd if=%s bs=10M skip=0 count=1 2> /dev/null | md5sum'
-            % medium_file, shell=True)
+            % medium_file, shell=True).decode()
         md5val.extend([out.split()[0]])
 
         umount_fs(mount_dir)
-- 
2.17.1

[U-Boot] [PATCHv3 10/13] test/py: Rework test.py to be a different kind of wrapper

[Tom Rini]

Now that we have moved to being based on pytest for python3 we need to
make our test.py wrapper more robust in terms of only calling python3
rather than possibly finding and using python2.  To do this, change from
execvp()'ing pytest to invoking the package itself via python.  In the
event that pytest is unavailable we still get a user-friendly error:

pkg_resources.DistributionNotFound: The 'pytest' distribution was not found and is required by the application

Reviewed-by: Simon Glass <sjg@chromium.org>
Tested-by: Stephen Warren <swarren@nvidia.com>
Tested-by: Simon Glass <sjg@chromium.org> [on sandbox]
Signed-off-by: Tom Rini <trini@konsulko.com>
---
Changes in v2:
- New patch
---
 test/py/test.py | 20 ++++----------------
 1 file changed, 4 insertions(+), 16 deletions(-)

diff --git a/test/py/test.py b/test/py/test.py
index 0ce1838833f6..bee88d96bc26 100755
--- a/test/py/test.py
+++ b/test/py/test.py
@@ -10,23 +10,11 @@
 import os
 import os.path
 import sys
-
-# Get rid of argv[0]
-sys.argv.pop(0)
+from pkg_resources import load_entry_point
 
 # argv; py.test test_directory_name user-supplied-arguments
-args = ['py.test', os.path.dirname(__file__) + '/tests']
+args = [os.path.dirname(__file__) + '/tests']
 args.extend(sys.argv)
 
-try:
-    os.execvp('py.test', args)
-except:
-    # Log full details of any exception for detailed analysis
-    import traceback
-    traceback.print_exc()
-    # Hint to the user that they likely simply haven't installed the required
-    # dependencies.
-    print('''
-exec(py.test) failed; perhaps you are missing some dependencies?
-See test/py/README.md for the list.''', file=sys.stderr)
-    sys.exit(1)
+if __name__ == '__main__':
+    sys.exit(load_entry_point('pytest', 'console_scripts', 'pytest')(args))
-- 
2.17.1

[U-Boot] [PATCHv3 11/13] test/py: Update docs, add requirements.txt for pip

[Tom Rini]

To be more closely aligned with Python community best practices, we need
to better document our usage of pip and make use of a requirements.txt
file that shows the versions of the tools that we are using.  This will
aide in ensuring reproducibility of our tests as well.

Reviewed-by: Simon Glass <sjg@chromium.org>
Tested-by: Stephen Warren <swarren@nvidia.com>
Tested-by: Simon Glass <sjg@chromium.org> [on sandbox]
Signed-off-by: Tom Rini <trini@konsulko.com>
---
Changes in v3:
- Drop duplicate openssl entry, clarify that we have a tool list and not
  package list (both from Stephen's feedback).
---
 test/py/README.md        | 47 ++++++++++++++++++++++------------------
 test/py/requirements.txt | 22 +++++++++++++++++++
 2 files changed, 48 insertions(+), 21 deletions(-)
 create mode 100644 test/py/requirements.txt

diff --git a/test/py/README.md b/test/py/README.md
index 2156661d6c42..3cbe01b73e28 100644
--- a/test/py/README.md
+++ b/test/py/README.md
@@ -21,19 +21,26 @@ involves executing some binary and interacting with its stdin/stdout. You will
 need to implement various "hook" scripts that are called by the test suite at
 the appropriate time.
 
-On Debian or Debian-like distributions, the following packages are required.
-Some packages are required to execute any test, and others only for specific
-tests. Similar package names should exist in other distributions.
-
-| Package        | Version tested (Ubuntu 14.04) |
-| -------------- | ----------------------------- |
-| python         | 2.7.5-5ubuntu3                |
-| python-pytest  | 2.5.1-1                       |
-| python-subunit | -                             |
-| gdisk          | 0.8.8-1ubuntu0.1              |
-| dfu-util       | 0.5-1                         |
-| dtc            | 1.4.0+dfsg-1                  |
-| openssl        | 1.0.1f-1ubuntu2.22            |
+In order to run the testsuite at a minimum we require that both python3 and
+pip for python3 be installed.  All of the required python modules are
+described in the requirements.txt file in this directory and can be installed
+with the command ```pip install -r requirements.txt```
+
+In order to execute certain tests on their supported platforms other tools
+will be required.  The following is an incomplete list:
+
+| Package        |
+| -------------- |
+| gdisk          |
+| dfu-util       |
+| dtc            |
+| openssl        |
+| sudo OR guestmount |
+| e2fsprogs      |
+| dosfstools     |
+
+Please use the apporirate commands for your distribution to match these tools
+up with the package that provides them.
 
 The test script supports either:
 
@@ -45,18 +52,16 @@ The test script supports either:
 
 ### Using `virtualenv` to provide requirements
 
-Older distributions (e.g. Ubuntu 10.04) may not provide all the required
-packages, or may provide versions that are too old to run the test suite. One
-can use the Python `virtualenv` script to locally install more up-to-date
-versions of the required packages without interfering with the OS installation.
-For example:
+The recommended way to run the test suite, in order to ensure reproducibility
+is to use `virtualenv` to set up the necessary environment.  This can be done
+via the following commands:
 
 ```bash
 $ cd /path/to/u-boot
-$ sudo apt-get install python python-virtualenv
-$ virtualenv venv
+$ sudo apt-get install python3 python3-virtualenv
+$ virtualenv -p /usr/bin/python3 venv
 $ . ./venv/bin/activate
-$ pip install pytest
+$ pip install -r test/py/requirements.txt
 ```
 
 ## Testing sandbox
diff --git a/test/py/requirements.txt b/test/py/requirements.txt
new file mode 100644
index 000000000000..cf251186f4b1
--- /dev/null
+++ b/test/py/requirements.txt
@@ -0,0 +1,22 @@
+atomicwrites==1.3.0
+attrs==19.3.0
+coverage==4.5.4
+extras==1.0.0
+fixtures==3.0.0
+importlib-metadata==0.23
+linecache2==1.0.0
+more-itertools==7.2.0
+packaging==19.2
+pbr==5.4.3
+pluggy==0.13.0
+py==1.8.0
+pyparsing==2.4.2
+pytest==5.2.1
+python-mimeparse==1.6.0
+python-subunit==1.3.0
+six==1.12.0
+testtools==2.3.0
+traceback2==1.4.0
+unittest2==1.1.0
+wcwidth==0.1.7
+zipp==0.6.0
-- 
2.17.1

[U-Boot] [PATCHv3 12/13] gitlab/travis: Rework how and when we use virtualenv in order to use python3

[Tom Rini]

As things stand today, we have tools that CI requires where "python"
must be "python2".  We need to use a virtualenv and pip in order to
ensure that our pytest tests can be run.  Rework things slightly so
that:
- On Travis-CI, we install python-pyelftools for the platforms that
  require pyelftools to be installed.
- On GitLab-CI, we move to a newer base image that includes python3-pip
  and continue to use a virtualenv per job that needs it, for the
  correct set of packages.

Reviewed-by: Simon Glass <sjg@chromium.org>
Tested-by: Stephen Warren <swarren@nvidia.com>
Tested-by: Simon Glass <sjg@chromium.org> [on sandbox]
Signed-off-by: Tom Rini <trini@konsulko.com>
---
Changes in v2:
- Add Travis-CI support here, and stop calling it a work-around.  This
  is just the path forward until we have everything using python3 and at
  that point evaluate what the right changes, if any, are.
---
 .gitlab-ci.yml | 10 ++++------
 .travis.yml    | 30 +++++++++++++++---------------
 2 files changed, 19 insertions(+), 21 deletions(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 5a34321570ce..9b295ac710fe 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -2,7 +2,7 @@
 
 # Grab our configured image.  The source for this is found at:
 # https://gitlab.denx.de/u-boot/gitlab-ci-runner
-image: trini/u-boot-gitlab-ci-runner:bionic-20190912.1-03Oct2019
+image: trini/u-boot-gitlab-ci-runner:bionic-20191010-20Oct2019
 
 # We run some tests in different order, to catch some failures quicker.
 stages:
@@ -18,11 +18,6 @@ stages:
     - git clone --depth=1 git://github.com/swarren/uboot-test-hooks.git /tmp/uboot-test-hooks
     - ln -s travis-ci /tmp/uboot-test-hooks/bin/`hostname`
     - ln -s travis-ci /tmp/uboot-test-hooks/py/`hostname`
-    - virtualenv /tmp/venv
-    - . /tmp/venv/bin/activate
-    - pip install pytest
-    - pip install python-subunit
-    - pip install coverage
     - grub-mkimage --prefix="" -o ~/grub_x86.efi -O i386-efi normal  echo lsefimmap lsefi lsefisystab efinet tftp minicmd
     - grub-mkimage --prefix="" -o ~/grub_x64.efi -O x86_64-efi normal  echo lsefimmap lsefi lsefisystab efinet tftp minicmd
     - mkdir ~/grub2-arm
@@ -47,6 +42,9 @@ stages:
     # never prevent any test from running. That way, we can always pass
     # "-k something" even when $TEST_PY_TEST_SPEC doesnt need a custom
     # value.
+    - virtualenv -p /usr/bin/python3 /tmp/venv
+    - . /tmp/venv/bin/activate
+    - pip install -r test/py/requirements.txt
     - export UBOOT_TRAVIS_BUILD_DIR=/tmp/.bm-work/${TEST_PY_BD};
       export PATH=/opt/qemu/bin:/tmp/uboot-test-hooks/bin:${PATH};
       export PYTHONPATH=/tmp/uboot-test-hooks/py/travis-ci;
diff --git a/.travis.yml b/.travis.yml
index f61278f9656a..d15b3f742639 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -21,7 +21,9 @@ addons:
     - build-essential
     - libsdl1.2-dev
     - python
-    - python-virtualenv
+    - python-pyelftools
+    - python3-virtualenv
+    - python3-pip
     - swig
     - libpython-dev
     - iasl
@@ -47,11 +49,6 @@ install:
  - echo -e "arc = /tmp/arc_gnu_2018.09_prebuilt_uclibc_le_archs_linux_install" >> ~/.buildman
  - echo -e "\n[toolchain-alias]\nsh = sh2\n" >> ~/.buildman
  - cat ~/.buildman
- - virtualenv /tmp/venv
- - . /tmp/venv/bin/activate
- - pip install pytest
- - pip install python-subunit
- - pip install pyelftools
  - grub-mkimage --prefix="" -o ~/grub_x86.efi -O i386-efi normal  echo lsefimmap lsefi lsefisystab efinet tftp minicmd
  - grub-mkimage --prefix="" -o ~/grub_x64.efi -O x86_64-efi normal  echo lsefimmap lsefi lsefisystab efinet tftp minicmd
  - mkdir ~/grub2-arm
@@ -136,15 +133,6 @@ script:
    cp ~/grub_x64.efi $UBOOT_TRAVIS_BUILD_DIR/;
    cp ~/grub2-arm/usr/lib/grub2/arm-efi/grub.efi $UBOOT_TRAVIS_BUILD_DIR/grub_arm.efi;
    cp ~/grub2-arm64/usr/lib/grub2/arm64-efi/grub.efi $UBOOT_TRAVIS_BUILD_DIR/grub_arm64.efi;
-   if [[ "${TEST_PY_BD}" != "" ]]; then
-     ./test/py/test.py --bd ${TEST_PY_BD} ${TEST_PY_ID}
-       -k "${TEST_PY_TEST_SPEC:-not a_test_which_does_not_exist}"
-       --build-dir "$UBOOT_TRAVIS_BUILD_DIR";
-     ret=$?;
-     if [[ $ret -ne 0 ]]; then
-       exit $ret;
-     fi;
-   fi;
    if [[ -n "${TEST_PY_TOOLS}" ]]; then
      PYTHONPATH="${UBOOT_TRAVIS_BUILD_DIR}/scripts/dtc/pylibfdt"
      PATH="${UBOOT_TRAVIS_BUILD_DIR}/scripts/dtc:${PATH}"
@@ -154,6 +142,18 @@ script:
      PYTHONPATH="${UBOOT_TRAVIS_BUILD_DIR}/scripts/dtc/pylibfdt"
      PATH="${UBOOT_TRAVIS_BUILD_DIR}/scripts/dtc:${PATH}"
      ./tools/dtoc/dtoc -t;
+   fi;
+   if [[ "${TEST_PY_BD}" != "" ]]; then
+     virtualenv -p /usr/bin/python3 /tmp/venv;
+     . /tmp/venv/bin/activate;
+     pip install -r test/py/requirements.txt;
+     ./test/py/test.py --bd ${TEST_PY_BD} ${TEST_PY_ID}
+       -k "${TEST_PY_TEST_SPEC:-not a_test_which_does_not_exist}"
+       --build-dir "$UBOOT_TRAVIS_BUILD_DIR";
+     ret=$?;
+     if [[ $ret -ne 0 ]]; then
+       exit $ret;
+     fi;
    fi
 
 matrix:
-- 
2.17.1

[U-Boot] [PATCHv3 13/13] test/py: Use raw strings more to avoid deprecation warnings

[Tom Rini]

We have two further uses of raw string usage in the test/py codebase
that are used under CI.  The first of which is under the bind test and
is a direct update.  The second of which is to strip VT100 codes from
the match buffer.  While switching this to a raw string is also a direct
update, the comment it notes that problems were encountered on Ubuntu
14.04 (and whatever Python 2 version that was) that required slight
tweaks to the regex.  Replace that now that we're saying Python 3.5 is
the minimum.

Reviewed-by: Simon Glass <sjg@chromium.org>
Tested-by: Stephen Warren <swarren@nvidia.com>
Tested-by: Simon Glass <sjg@chromium.org> [on sandbox]
Signed-off-by: Tom Rini <trini@konsulko.com>
---
Changes in v2:
- New patch
---
 test/py/tests/test_bind.py | 4 ++--
 test/py/u_boot_spawn.py    | 5 +----
 2 files changed, 3 insertions(+), 6 deletions(-)

diff --git a/test/py/tests/test_bind.py b/test/py/tests/test_bind.py
index 2d48484c6ac6..20c6050342a9 100644
--- a/test/py/tests/test_bind.py
+++ b/test/py/tests/test_bind.py
@@ -9,11 +9,11 @@ def in_tree(response, name, uclass, drv, depth, last_child):
 	lines = [x.strip() for x in response.splitlines()]
 	leaf = ' ' * 4 * depth;
 	if not last_child:
-		leaf = leaf + '\|'
+		leaf = leaf + r'\|'
 	else:
 		leaf = leaf + '`'
 	leaf = leaf + '-- ' + name
-	line = (' *{:10.10}    [0-9]*  \[ [ +] \]   {:20.20}  {}$'
+	line = (r' *{:10.10}    [0-9]*  \[ [ +] \]   {:20.20}  {}$'
 	        .format(uclass, drv, leaf))
 	prog = re.compile(line)
 	for l in lines:
diff --git a/test/py/u_boot_spawn.py b/test/py/u_boot_spawn.py
index 4f898cdefe5a..6991b78cca89 100644
--- a/test/py/u_boot_spawn.py
+++ b/test/py/u_boot_spawn.py
@@ -42,10 +42,7 @@ class Spawn(object):
         self.after = ''
         self.timeout = None
         # http://stackoverflow.com/questions/7857352/python-regex-to-match-vt100-escape-sequences
-        # Note that re.I doesn't seem to work with this regex (or perhaps the
-        # version of Python in Ubuntu 14.04), hence the inclusion of a-z inside
-        # [] instead.
-        self.re_vt100 = re.compile('(\x1b\[|\x9b)[^@-_a-z]*[@-_a-z]|\x1b[@-_a-z]')
+        self.re_vt100 = re.compile(r'(\x1b\[|\x9b)[^@-_]*[@-_]|\x1b[@-_]', re.I)
 
         (self.pid, self.fd) = pty.fork()
         if self.pid == 0:
-- 
2.17.1

[U-Boot] [PATCH 1/2] Dockerfile: Update to latest bionic tag

[Tom Rini]

Move up to the latest tag to get the usual related security fixes that
brings in.

Signed-off-by: Tom Rini <trini@konsulko.com>
---
 Dockerfile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Dockerfile b/Dockerfile
index bf840a5f1b33..6eb47b9b5016 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -2,7 +2,7 @@
 # This Dockerfile is used to build an image containing basic stuff to be used
 # to build U-Boot and run our test suites.
 
-FROM ubuntu:bionic-20190912.1
+FROM ubuntu:bionic-20191010
 MAINTAINER Tom Rini <trini@konsulko.com>
 LABEL Description=" This image is for building U-Boot inside a container"
 
-- 
2.17.1

[U-Boot] [PATCH 2/2] Dockerfile: Add python3-pip

[Tom Rini]

In order to be able to use pip within a virtualenv of python3 we need to
have python3-pip installed as well.

Signed-off-by: Tom Rini <trini@konsulko.com>
---
 Dockerfile | 1 +
 1 file changed, 1 insertion(+)

diff --git a/Dockerfile b/Dockerfile
index 6eb47b9b5016..43e739601741 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -67,6 +67,7 @@ RUN apt-get update && apt-get install -y \
 	python-dev \
 	python-pip \
 	python-virtualenv \
+	python3-pip \
 	python3-sphinx \
 	rpm2cpio \
 	sloccount \
-- 
2.17.1

[U-Boot] [PATCH 1/2] Dockerfile: Update to latest bionic tag

[Tom Rini]

On Thu, Oct 24, 2019 at 12:01:56PM -0400, Tom Rini wrote:

> Move up to the latest tag to get the usual related security fixes that
> brings in.
> 
> Signed-off-by: Tom Rini <trini@konsulko.com>

Applied to u-boot/master, thanks!

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20191030/472648ac/attachment.sig>

[U-Boot] [PATCH 2/2] Dockerfile: Add python3-pip

[Tom Rini]

On Thu, Oct 24, 2019 at 12:01:57PM -0400, Tom Rini wrote:

> In order to be able to use pip within a virtualenv of python3 we need to
> have python3-pip installed as well.
> 
> Signed-off-by: Tom Rini <trini@konsulko.com>

Applied to u-boot/master, thanks!

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20191030/29b23254/attachment.sig>

[U-Boot] [PATCHv3 01/13] gitlab-ci: Fix indentation in some stanzas

[Tom Rini]

On Thu, Oct 24, 2019 at 11:59:16AM -0400, Tom Rini wrote:

> In a number of our stanzas we had multi-line commands that were one
> space short of alignment, correct this.
> 
> Reviewed-by: Stephen Warren <swarren@nvidia.com>
> Reviewed-by: Simon Glass <sjg@chromium.org>
> Tested-by: Stephen Warren <swarren@nvidia.com>
> Signed-off-by: Tom Rini <trini@konsulko.com>

Applied to u-boot/master, thanks!

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20191101/116bb832/attachment.sig>

[U-Boot] [PATCHv3 02/13] gitlab-ci: Prepend to PATH rather than replace it

[Tom Rini]

On Thu, Oct 24, 2019 at 11:59:17AM -0400, Tom Rini wrote:

> Currently we set the entire PATH rather than prepend the new paths that
> we need to have searched.  This however breaks parts of the "virtualenv"
> that was have set up and need to use as that also will be modifying
> PATH.  To fix this, prepend our new locations instead.
> 
> Reviewed-by: Stephen Warren <swarren@nvidia.com>
> Reviewed-by: Simon Glass <sjg@chromium.org>
> Tested-by: Stephen Warren <swarren@nvidia.com>
> Signed-off-by: Tom Rini <trini@konsulko.com>

Applied to u-boot/master, thanks!

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20191101/02bf3e4f/attachment.sig>

[U-Boot] [PATCHv3 03/13] test/py: Split mark to multiple lines

[Tom Rini]

On Thu, Oct 24, 2019 at 11:59:18AM -0400, Tom Rini wrote:

> We inconsistently note multiple dependencies today in our tests,
> sometimes with a single line that declares multiple and sometimes
> multiple single lines.  Current pytest seems to fail on the single line
> format so change to multiple declarations.
> 
> Reviewed-by: Stephen Warren <swarren@nvidia.com>
> Reviewed-by: Simon Glass <sjg@chromium.org>
> Tested-by: Stephen Warren <swarren@nvidia.com>
> Tested-by: Simon Glass <sjg@chromium.org> [on sandbox]
> Signed-off-by: Tom Rini <trini@konsulko.com>

Applied to u-boot/master, thanks!

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20191101/2629c8d9/attachment.sig>

[U-Boot] [PATCHv3 04/13] test/py: Fix pytest4 deprecation warnings

[Tom Rini]

On Thu, Oct 24, 2019 at 11:59:19AM -0400, Tom Rini wrote:

> From: Marek Vasut <marek.vasut@gmail.com>
> 
> Fix the following spit from pytest:
> 
> u-boot/test/py/conftest.py:438: RemovedInPytest4Warning: MarkInfo objects are deprecated as they contain merged marks which are hard to deal with correctly.
>   Please use node.get_closest_marker(name) or node.iter_markers(name).
>   Docs: https://docs.pytest.org/en/latest/mark.html#updating-code
>     for board in mark.args:
> 
> In both cases, the later suggestion is applicable.
> 
> Reviewed-by: Stephen Warren <swarren@nvidia.com>
> Reviewed-by: Simon Glass <sjg@chromium.org>
> Signed-off-by: Marek Vasut <marek.vasut+renesas@gmail.com>
> Cc: Igor Opaniuk <igor.opaniuk@gmail.com>
> [trini: Update for current file with a few more cases, un-pin pytest in CI]
> Tested-by: Simon Glass <sjg@chromium.org> [on sandbox]
> Tested-by: Stephen Warren <swarren@nvidia.com>
> Signed-off-by: Tom Rini <trini@konsulko.com>

Applied to u-boot/master, thanks!

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20191101/82800c85/attachment.sig>

[U-Boot] [PATCHv3 05/13] test/py: Automated conversion to Python 3

[Tom Rini]

On Thu, Oct 24, 2019 at 11:59:20AM -0400, Tom Rini wrote:

> Use the 2to3 tool to perform numerous automatic conversions from Python
> 2 syntax to Python 3.  Also fix whitespace problems that Python 3
> catches that Python 2 did not.
> 
> Reviewed-by: Stephen Warren <swarren@nvidia.com>
> Reviewed-by: Simon Glass <sjg@chromium.org>
> Tested-by: Stephen Warren <swarren@nvidia.com>
> Tested-by: Simon Glass <sjg@chromium.org> [on sandbox]
> Signed-off-by: Tom Rini <trini@konsulko.com>

Applied to u-boot/master, thanks!

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20191101/2ba9b474/attachment.sig>

[U-Boot] [PATCHv3 06/13] test/py: Manual python3 fixes

[Tom Rini]

On Thu, Oct 24, 2019 at 11:59:21AM -0400, Tom Rini wrote:

> - Modern pytest is more visible in telling us about parameters that we
>   had not described, so describe a few more.
> - ConfigParser.readfp(...) is now configparser.read_file(...)
> - As part of the "strings vs bytes" conversions in Python 3, we use the
>   default encoding/decoding of utf-8 but in some places tell Python to
>   replace problematic conversions rather than throw a fatal error.
> - Fix a typo noticed while doing the above ("tot he" -> "to the").
> - As suggested by Stephen, re-alphabetize the import list
> - Per Heinrich, replace how we write contents in test_fit.py
> 
> Reviewed-by: Simon Glass <sjg@chromium.org>
> Tested-by: Stephen Warren <swarren@nvidia.com>
> Tested-by: Simon Glass <sjg@chromium.org> [on sandbox]
> Signed-off-by: Tom Rini <trini@konsulko.com>

Applied to u-boot/master, thanks!

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20191101/d7e71fa7/attachment.sig>

[U-Boot] [PATCHv3 07/13] test/py: test_ut.py: Ensure we use bytes

[Tom Rini]

On Thu, Oct 24, 2019 at 11:59:22AM -0400, Tom Rini wrote:

> In the case of some unit tests we are working with providing a fake
> flash device that we have written some text strings in to.  In this case
> we want to tell Python to encode things to bytes for us.
> 
> Reviewed-by: Stephen Warren <swarren@nvidia.com>
> Reviewed-by: Simon Glass <sjg@chromium.org>
> Tested-by: Stephen Warren <swarren@nvidia.com>
> Tested-by: Simon Glass <sjg@chromium.org> [on sandbox]
> Signed-off-by: Tom Rini <trini@konsulko.com>

Applied to u-boot/master, thanks!

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20191101/cd023b1e/attachment.sig>

[U-Boot] [PATCHv3 08/13] test/py: test_efi_selftest.py: Updates for python 3 support

[Tom Rini]

On Thu, Oct 24, 2019 at 11:59:23AM -0400, Tom Rini wrote:

> - In python 3 you must use raw strings for regex as other forms are
>   deprecated and would require further changes to the pattern here.
>   In one case this lets us have a simpler match pattern.
> - As strings are now Unicode our complex tests (Euro symbol,
>   SHIFT+ALT+FN 5) we need to declare that as a bytes string and then
>   decode it for use.
> 
> Cc: Heinrich Schuchardt <xypron.glpk@gmx.de>
> Reviewed-by: Simon Glass <sjg@chromium.org>
> Tested-by: Stephen Warren <swarren@nvidia.com>
> Tested-by: Simon Glass <sjg@chromium.org> [on sandbox]
> Signed-off-by: Tom Rini <trini@konsulko.com>

Applied to u-boot/master, thanks!

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20191101/975b105e/attachment.sig>

[U-Boot] [PATCHv3 09/13] test/py: Update test_fs to decode check_output calls

[Tom Rini]

On Thu, Oct 24, 2019 at 11:59:24AM -0400, Tom Rini wrote:

> The check_output function from the subprocess Python module by default
> returns data as encoded bytes and leaves decoding to the application.
> Given our uses of the call, it makes the most sense to immediately
> decode the results.
> 
> Reviewed-by: Simon Glass <sjg@chromium.org>
> Tested-by: Stephen Warren <swarren@nvidia.com>
> Tested-by: Simon Glass <sjg@chromium.org> [on sandbox]
> Signed-off-by: Tom Rini <trini@konsulko.com>

Applied to u-boot/master, thanks!

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20191101/aad7db11/attachment.sig>

[U-Boot] [PATCHv3 10/13] test/py: Rework test.py to be a different kind of wrapper

[Tom Rini]

On Thu, Oct 24, 2019 at 11:59:25AM -0400, Tom Rini wrote:

> Now that we have moved to being based on pytest for python3 we need to
> make our test.py wrapper more robust in terms of only calling python3
> rather than possibly finding and using python2.  To do this, change from
> execvp()'ing pytest to invoking the package itself via python.  In the
> event that pytest is unavailable we still get a user-friendly error:
> 
> pkg_resources.DistributionNotFound: The 'pytest' distribution was not found and is required by the application
> 
> Reviewed-by: Simon Glass <sjg@chromium.org>
> Tested-by: Stephen Warren <swarren@nvidia.com>
> Tested-by: Simon Glass <sjg@chromium.org> [on sandbox]
> Signed-off-by: Tom Rini <trini@konsulko.com>

Applied to u-boot/master, thanks!

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20191101/ca88a212/attachment.sig>

[U-Boot] [PATCHv3 11/13] test/py: Update docs, add requirements.txt for pip

[Tom Rini]

On Thu, Oct 24, 2019 at 11:59:26AM -0400, Tom Rini wrote:

> To be more closely aligned with Python community best practices, we need
> to better document our usage of pip and make use of a requirements.txt
> file that shows the versions of the tools that we are using.  This will
> aide in ensuring reproducibility of our tests as well.
> 
> Reviewed-by: Simon Glass <sjg@chromium.org>
> Tested-by: Stephen Warren <swarren@nvidia.com>
> Tested-by: Simon Glass <sjg@chromium.org> [on sandbox]
> Signed-off-by: Tom Rini <trini@konsulko.com>

Applied to u-boot/master, thanks!

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20191101/50cb7fdd/attachment.sig>

[U-Boot] [PATCHv3 12/13] gitlab/travis: Rework how and when we use virtualenv in order to use python3

[Tom Rini]

On Thu, Oct 24, 2019 at 11:59:27AM -0400, Tom Rini wrote:

> As things stand today, we have tools that CI requires where "python"
> must be "python2".  We need to use a virtualenv and pip in order to
> ensure that our pytest tests can be run.  Rework things slightly so
> that:
> - On Travis-CI, we install python-pyelftools for the platforms that
>   require pyelftools to be installed.
> - On GitLab-CI, we move to a newer base image that includes python3-pip
>   and continue to use a virtualenv per job that needs it, for the
>   correct set of packages.
> 
> Reviewed-by: Simon Glass <sjg@chromium.org>
> Tested-by: Stephen Warren <swarren@nvidia.com>
> Tested-by: Simon Glass <sjg@chromium.org> [on sandbox]
> Signed-off-by: Tom Rini <trini@konsulko.com>

Applied to u-boot/master, thanks!

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20191101/56dc656b/attachment.sig>

[U-Boot] [PATCHv3 13/13] test/py: Use raw strings more to avoid deprecation warnings

[Tom Rini]

On Thu, Oct 24, 2019 at 11:59:28AM -0400, Tom Rini wrote:

> We have two further uses of raw string usage in the test/py codebase
> that are used under CI.  The first of which is under the bind test and
> is a direct update.  The second of which is to strip VT100 codes from
> the match buffer.  While switching this to a raw string is also a direct
> update, the comment it notes that problems were encountered on Ubuntu
> 14.04 (and whatever Python 2 version that was) that required slight
> tweaks to the regex.  Replace that now that we're saying Python 3.5 is
> the minimum.
> 
> Reviewed-by: Simon Glass <sjg@chromium.org>
> Tested-by: Stephen Warren <swarren@nvidia.com>
> Tested-by: Simon Glass <sjg@chromium.org> [on sandbox]
> Signed-off-by: Tom Rini <trini@konsulko.com>

Applied to u-boot/master, thanks!

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20191101/87b290d2/attachment.sig>