[U-Boot] [PATCH v2 00/17] test: Various test refinements and improvements

[U-Boot] [PATCH v2 00/17] test: Various test refinements and improvements

[Simon Glass]

This series includes a number of small changes designed to make tests run
more smoothly. The overall goal is that 'make check' runs cleanly without
unnecessary output and all temporary files aare cleaned up.

Changes in v2:
- Quote @$ correctly so that quoted arguments can be passed to run_test
- Add new patch to reduce the number of tests run with sandbox_flattree
- Add a patch to run binman tests concurrently

Simon Glass (16):
  sandbox: Unprotect DATA regions in bus tests
  patman: Handle unicode in _ProjectConfigParser tests
  test/py: Fix unicode handling for log filtering
  buildman: Make the toolchain test more forgiving
  Makefile: Add a 'check' target for make
  test: Simplify the PATH setup
  test: Print the name of each test before running it
  test: Tidy up comments and variable name
  binman: Add a default path to libfdt.py
  binman: Fix up removal of temporary directories
  binman: Separate out testSplBssPad()
  buildman: dtoc: Suppress unwanted output from test
  tools: Set an initial value for indir
  patman: Don't clear progress in tout unless it was used
  test: Reduce the number of tests run with sandbox_flattree
  binman: Run tests concurrently

Stephen Warren (1):
  test/py: ignore console read exceptions after test failure

 .travis.yml                              |   1 +
 Makefile                                 |   6 +-
 arch/sandbox/cpu/os.c                    |  11 ++
 include/os.h                             |  12 ++
 test/dm/bus.c                            |  12 ++
 test/py/README.md                        |   1 +
 test/py/multiplexed_log.py               |   7 +-
 test/py/u_boot_console_base.py           |  12 +-
 test/run                                 |  57 +++++----
 tools/binman/binman.py                   |  28 ++++-
 tools/binman/cmdline.py                  |   2 +
 tools/binman/elf_test.py                 |   5 +
 tools/binman/entry_test.py               |  15 ++-
 tools/binman/fdt_test.py                 |   4 +
 tools/binman/ftest.py                    |  46 ++++----
 tools/buildman/test.py                   |   8 +-
 tools/concurrencytest/.gitignore         |   1 +
 tools/concurrencytest/README.md          |  74 ++++++++++++
 tools/concurrencytest/concurrencytest.py | 144 +++++++++++++++++++++++
 tools/dtoc/dtoc.py                       |   7 ++
 tools/dtoc/test_dtoc.py                  |   6 +-
 tools/dtoc/test_fdt.py                   |  12 +-
 tools/patman/settings.py                 |  27 +++--
 tools/patman/test_util.py                |   2 +-
 tools/patman/tools.py                    |   3 +
 tools/patman/tout.py                     |   8 +-
 26 files changed, 439 insertions(+), 72 deletions(-)
 create mode 100644 tools/concurrencytest/.gitignore
 create mode 100644 tools/concurrencytest/README.md
 create mode 100644 tools/concurrencytest/concurrencytest.py

-- 
2.19.0.605.g01d371f741-goog

[U-Boot] [PATCH v2 01/17] test/py: ignore console read exceptions after test failure

[Simon Glass]

From: Stephen Warren <swarren@nvidia.com>

After a test has failed, test/py drains the U-Boot console log to ensure
that any relevant output is captured. At this point, we don't care about
detecting any additional errors, since the test is already known to have
failed, and U-Boot will be restarted. To ensure that the test cleanup code
is not interrupted, and can correctly terminate the log sections for the
failed test, ignore any exception that occurs while reading the U-Boot
console output during this limited period of time.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
Signed-off-by: Simon Glass <sjg@chromium.org>
---

Changes in v2: None

 test/py/u_boot_console_base.py | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/test/py/u_boot_console_base.py b/test/py/u_boot_console_base.py
index a14bad6e8c5..326b2ac51fb 100644
--- a/test/py/u_boot_console_base.py
+++ b/test/py/u_boot_console_base.py
@@ -304,7 +304,17 @@ class ConsoleBase(object):
             # Wait for something U-Boot will likely never send. This will
             # cause the console output to be read and logged.
             self.p.expect(['This should never match U-Boot output'])
-        except u_boot_spawn.Timeout:
+        except:
+            # We expect a timeout, since U-Boot won't print what we waited
+            # for. Squash it when it happens.
+            #
+            # Squash any other exception too. This function is only used to
+            # drain (and log) the U-Boot console output after a failed test.
+            # The U-Boot process will be restarted, or target board reset, once
+            # this function returns. So, we don't care about detecting any
+            # additional errors, so they're squashed so that the rest of the
+            # post-test-failure cleanup code can continue operation, and
+            # correctly terminate any log sections, etc.
             pass
         finally:
             self.p.timeout = orig_timeout
-- 
2.19.0.605.g01d371f741-goog

[U-Boot] [PATCH v2 01/17] test/py: ignore console read exceptions after test failure

[Stephen Warren]

On 10/01/2018 09:12 PM, Simon Glass wrote:
> From: Stephen Warren <swarren@nvidia.com>
> 
> After a test has failed, test/py drains the U-Boot console log to ensure
> that any relevant output is captured. At this point, we don't care about
> detecting any additional errors, since the test is already known to have
> failed, and U-Boot will be restarted. To ensure that the test cleanup code
> is not interrupted, and can correctly terminate the log sections for the
> failed test, ignore any exception that occurs while reading the U-Boot
> console output during this limited period of time.

This probably shouldn't be in this series since I sent it separately as 
a standalone patch a while ago.

[U-Boot] [PATCH v2 01/17] test/py: ignore console read exceptions after test failure

[Simon Glass]

Hi Stephen,

On 4 October 2018 at 10:01, Stephen Warren <swarren@wwwdotorg.org> wrote:
> On 10/01/2018 09:12 PM, Simon Glass wrote:
>>
>> From: Stephen Warren <swarren@nvidia.com>
>>
>> After a test has failed, test/py drains the U-Boot console log to ensure
>> that any relevant output is captured. At this point, we don't care about
>> detecting any additional errors, since the test is already known to have
>> failed, and U-Boot will be restarted. To ensure that the test cleanup code
>> is not interrupted, and can correctly terminate the log sections for the
>> failed test, ignore any exception that occurs while reading the U-Boot
>> console output during this limited period of time.
>
>
> This probably shouldn't be in this series since I sent it separately as a
> standalone patch a while ago.

Sorry about that. I picked it up and assumed it would fall out when I
rebased to master, but I suppose your patch is not yet applied.

Regards,
Simon

[U-Boot] [PATCH v2 01/17] test/py: ignore console read exceptions after test failure

[sjg at google.com]

Hi Stephen,

On 4 October 2018 at 10:01, Stephen Warren <swarren@wwwdotorg.org> wrote:
> On 10/01/2018 09:12 PM, Simon Glass wrote:
>>
>> From: Stephen Warren <swarren@nvidia.com>
>>
>> After a test has failed, test/py drains the U-Boot console log to ensure
>> that any relevant output is captured. At this point, we don't care about
>> detecting any additional errors, since the test is already known to have
>> failed, and U-Boot will be restarted. To ensure that the test cleanup code
>> is not interrupted, and can correctly terminate the log sections for the
>> failed test, ignore any exception that occurs while reading the U-Boot
>> console output during this limited period of time.
>
>
> This probably shouldn't be in this series since I sent it separately as a
> standalone patch a while ago.

Sorry about that. I picked it up and assumed it would fall out when I
rebased to master, but I suppose your patch is not yet applied.

Regards,
Simon

Applied to u-boot-dm

[U-Boot] [PATCH v2 02/17] sandbox: Unprotect DATA regions in bus tests

[Simon Glass]

On my Ubuntu 18.04.1 machine two driver-model bus tests have started
failing recently. The problem appears to be that the DATA region of the
executable is protected. This does not seem correct, but perhaps there
is a reason.

To work around it, unprotect the regions in these tests before accessing
them.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

Changes in v2: None

 arch/sandbox/cpu/os.c | 11 +++++++++++
 include/os.h          | 12 ++++++++++++
 test/dm/bus.c         | 12 ++++++++++++
 3 files changed, 35 insertions(+)

diff --git a/arch/sandbox/cpu/os.c b/arch/sandbox/cpu/os.c
index 9fbcb9ef92f..d4d6d78dc74 100644
--- a/arch/sandbox/cpu/os.c
+++ b/arch/sandbox/cpu/os.c
@@ -636,3 +636,14 @@ void os_abort(void)
 {
 	abort();
 }
+
+int os_mprotect_allow(void *start, size_t len)
+{
+	int page_size = getpagesize();
+
+	/* Move start to the start of a page, len to the end */
+	start = (void *)(((ulong)start) & ~(page_size - 1));
+	len = (len + page_size * 2) & ~(page_size - 1);
+
+	return mprotect(start, len, PROT_READ | PROT_WRITE);
+}
diff --git a/include/os.h b/include/os.h
index 5c797212c25..7116f875780 100644
--- a/include/os.h
+++ b/include/os.h
@@ -334,4 +334,16 @@ void os_localtime(struct rtc_time *rt);
  * os_abort() - Raise SIGABRT to exit sandbox (e.g. to debugger)
  */
 void os_abort(void);
+
+/**
+ * os_mprotect_allow() - Remove write-protection on a region of memory
+ *
+ * The start and length will be page-aligned before use.
+ *
+ * @start:	Region start
+ * @len:	Region length in bytes
+ * @return 0 if OK, -1 on error from mprotect()
+ */
+int os_mprotect_allow(void *start, size_t len);
+
 #endif
diff --git a/test/dm/bus.c b/test/dm/bus.c
index e9a4028f047..08137a2216a 100644
--- a/test/dm/bus.c
+++ b/test/dm/bus.c
@@ -4,6 +4,9 @@
  */
 
 #include <common.h>
+#ifdef CONFIG_SANDBOX
+#include <os.h>
+#endif
 #include <dm.h>
 #include <dm/device-internal.h>
 #include <dm/test.h>
@@ -297,6 +300,11 @@ static int dm_test_bus_parent_data_uclass(struct unit_test_state *uts)
 	ut_assertok(uclass_find_device(UCLASS_TEST_BUS, 0, &bus));
 	drv = (struct driver *)bus->driver;
 	size = drv->per_child_auto_alloc_size;
+
+#ifdef CONFIG_SANDBOX
+	os_mprotect_allow(bus->uclass->uc_drv, sizeof(*bus->uclass->uc_drv));
+	os_mprotect_allow(drv, sizeof(*drv));
+#endif
 	bus->uclass->uc_drv->per_child_auto_alloc_size = size;
 	drv->per_child_auto_alloc_size = 0;
 	ret = test_bus_parent_data(uts);
@@ -440,6 +448,10 @@ static int dm_test_bus_parent_platdata_uclass(struct unit_test_state *uts)
 	ut_assertok(uclass_find_device(UCLASS_TEST_BUS, 0, &bus));
 	drv = (struct driver *)bus->driver;
 	size = drv->per_child_platdata_auto_alloc_size;
+#ifdef CONFIG_SANDBOX
+	os_mprotect_allow(bus->uclass->uc_drv, sizeof(*bus->uclass->uc_drv));
+	os_mprotect_allow(drv, sizeof(*drv));
+#endif
 	bus->uclass->uc_drv->per_child_platdata_auto_alloc_size = size;
 	drv->per_child_platdata_auto_alloc_size = 0;
 	ret = test_bus_parent_platdata(uts);
-- 
2.19.0.605.g01d371f741-goog

[U-Boot] [PATCH v2 03/17] patman: Handle unicode in _ProjectConfigParser tests

[Simon Glass]

With Python 2.7.15rc1, ConfigParser.SafeConfigParser has unfortunately
started returning unicode, for unknown reasons. Adjust the code to handle
this by converting everything to unicode. We cannot convert things to
ASCII since email addresses may be encoded with UTF-8.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

Changes in v2: None

 tools/patman/settings.py | 27 ++++++++++++++++++++-------
 1 file changed, 20 insertions(+), 7 deletions(-)

diff --git a/tools/patman/settings.py b/tools/patman/settings.py
index ca4334426ba..ea2bc74f759 100644
--- a/tools/patman/settings.py
+++ b/tools/patman/settings.py
@@ -58,25 +58,25 @@ class _ProjectConfigParser(ConfigParser.SafeConfigParser):
     >>> config = _ProjectConfigParser("zzz")
     >>> config.readfp(StringIO(sample_config))
     >>> config.get("alias", "enemies")
-    'Evil <evil@example.com>'
+    u'Evil <evil@example.com>'
 
     # Check to make sure that alias gets overridden by project.
     >>> config = _ProjectConfigParser("sm")
     >>> config.readfp(StringIO(sample_config))
     >>> config.get("alias", "enemies")
-    'Green G. <ugly@example.com>'
+    u'Green G. <ugly@example.com>'
 
     # Check to make sure that settings get merged with project.
     >>> config = _ProjectConfigParser("linux")
     >>> config.readfp(StringIO(sample_config))
     >>> sorted(config.items("settings"))
-    [('am_hero', 'True'), ('process_tags', 'False')]
+    [(u'am_hero', u'True'), (u'process_tags', u'False')]
 
     # Check to make sure that settings works with unknown project.
     >>> config = _ProjectConfigParser("unknown")
     >>> config.readfp(StringIO(sample_config))
     >>> sorted(config.items("settings"))
-    [('am_hero', 'True')]
+    [(u'am_hero', u'True')]
     """
     def __init__(self, project_name):
         """Construct _ProjectConfigParser.
@@ -99,6 +99,17 @@ class _ProjectConfigParser(ConfigParser.SafeConfigParser):
         for setting_name, setting_value in project_defaults.items():
             self.set(project_settings, setting_name, setting_value)
 
+    def _to_unicode(self, val):
+        """Make sure a value is of type 'unicode'
+
+        Args:
+            val: string or unicode object
+
+        Returns:
+            unicode version of val
+        """
+        return val if isinstance(val, unicode) else val.decode('utf-8')
+
     def get(self, section, option, *args, **kwargs):
         """Extend SafeConfigParser to try project_section before section.
 
@@ -108,14 +119,15 @@ class _ProjectConfigParser(ConfigParser.SafeConfigParser):
             See SafeConfigParser.
         """
         try:
-            return ConfigParser.SafeConfigParser.get(
+            val = ConfigParser.SafeConfigParser.get(
                 self, "%s_%s" % (self._project_name, section), option,
                 *args, **kwargs
             )
         except (ConfigParser.NoSectionError, ConfigParser.NoOptionError):
-            return ConfigParser.SafeConfigParser.get(
+            val = ConfigParser.SafeConfigParser.get(
                 self, section, option, *args, **kwargs
             )
+        return self._to_unicode(val)
 
     def items(self, section, *args, **kwargs):
         """Extend SafeConfigParser to add project_section to section.
@@ -150,7 +162,8 @@ class _ProjectConfigParser(ConfigParser.SafeConfigParser):
 
         item_dict = dict(top_items)
         item_dict.update(project_items)
-        return item_dict.items()
+        return {(self._to_unicode(item), self._to_unicode(val))
+                for item, val in item_dict.iteritems()}
 
 def ReadGitAliases(fname):
     """Read a git alias file. This is in the form used by git:
-- 
2.19.0.605.g01d371f741-goog

[U-Boot] [PATCH v2 04/17] test/py: Fix unicode handling for log filtering

[Simon Glass]

At present the unicode filtering seems to get confused at times with
this error:

  UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position
     32: ordinal not in range(128)

It seems to be due to self._nonprint being interpreted as UTF-8. Fix it
by using ordinals instead of characters, changing the string to set.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Stephen Warren <swarren@nvidia.com>
---

Changes in v2: None

 test/py/multiplexed_log.py | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/test/py/multiplexed_log.py b/test/py/multiplexed_log.py
index f23d5dec68c..637a3bd257b 100644
--- a/test/py/multiplexed_log.py
+++ b/test/py/multiplexed_log.py
@@ -314,8 +314,9 @@ $(document).ready(function () {
 
     # The set of characters that should be represented as hexadecimal codes in
     # the log file.
-    _nonprint = ('%' + ''.join(chr(c) for c in range(0, 32) if c not in (9, 10)) +
-                 ''.join(chr(c) for c in range(127, 256)))
+    _nonprint = {ord('%')}
+    _nonprint.update({c for c in range(0, 32) if c not in (9, 10)})
+    _nonprint.update({c for c in range(127, 256)})
 
     def _escape(self, data):
         """Render data format suitable for inclusion in an HTML document.
@@ -331,7 +332,7 @@ $(document).ready(function () {
         """
 
         data = data.replace(chr(13), '')
-        data = ''.join((c in self._nonprint) and ('%%%02x' % ord(c)) or
+        data = ''.join((ord(c) in self._nonprint) and ('%%%02x' % ord(c)) or
                        c for c in data)
         data = cgi.escape(data)
         return data
-- 
2.19.0.605.g01d371f741-goog

[U-Boot] [PATCH v2 04/17] test/py: Fix unicode handling for log filtering

[Michal Simek]

On 2.10.2018 05:12, Simon Glass wrote:
> At present the unicode filtering seems to get confused at times with
> this error:
> 
>   UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position
>      32: ordinal not in range(128)
> 
> It seems to be due to self._nonprint being interpreted as UTF-8. Fix it
> by using ordinals instead of characters, changing the string to set.
> 
> Signed-off-by: Simon Glass <sjg@chromium.org>
> Reviewed-by: Stephen Warren <swarren@nvidia.com>
> ---
> 
> Changes in v2: None
> 
>  test/py/multiplexed_log.py | 7 ++++---
>  1 file changed, 4 insertions(+), 3 deletions(-)
> 
> diff --git a/test/py/multiplexed_log.py b/test/py/multiplexed_log.py
> index f23d5dec68c..637a3bd257b 100644
> --- a/test/py/multiplexed_log.py
> +++ b/test/py/multiplexed_log.py
> @@ -314,8 +314,9 @@ $(document).ready(function () {
>  
>      # The set of characters that should be represented as hexadecimal codes in
>      # the log file.
> -    _nonprint = ('%' + ''.join(chr(c) for c in range(0, 32) if c not in (9, 10)) +
> -                 ''.join(chr(c) for c in range(127, 256)))
> +    _nonprint = {ord('%')}
> +    _nonprint.update({c for c in range(0, 32) if c not in (9, 10)})
> +    _nonprint.update({c for c in range(127, 256)})
>  
>      def _escape(self, data):
>          """Render data format suitable for inclusion in an HTML document.
> @@ -331,7 +332,7 @@ $(document).ready(function () {
>          """
>  
>          data = data.replace(chr(13), '')
> -        data = ''.join((c in self._nonprint) and ('%%%02x' % ord(c)) or
> +        data = ''.join((ord(c) in self._nonprint) and ('%%%02x' % ord(c)) or
>                         c for c in data)
>          data = cgi.escape(data)
>          return data
> 

Tested-by: Michal Simek <michal.simek@xilinx.com>

Thanks,
Michal

-- 
Michal Simek, Ing. (M.Eng), OpenPGP -> KeyID: FE3D1F91
w: www.monstr.eu p: +42-0-721842854
Maintainer of Linux kernel - Xilinx Microblaze
Maintainer of Linux kernel - Xilinx Zynq ARM and ZynqMP ARM64 SoCs
U-Boot custodian - Xilinx Microblaze/Zynq/ZynqMP SoCs

[U-Boot] [PATCH v2 04/17] test/py: Fix unicode handling for log filtering

[sjg at google.com]

On 2.10.2018 05:12, Simon Glass wrote:
> At present the unicode filtering seems to get confused at times with
> this error:
>
>   UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position
>      32: ordinal not in range(128)
>
> It seems to be due to self._nonprint being interpreted as UTF-8. Fix it
> by using ordinals instead of characters, changing the string to set.
>
> Signed-off-by: Simon Glass <sjg@chromium.org>
> Reviewed-by: Stephen Warren <swarren@nvidia.com>
> ---
>
> Changes in v2: None
>
>  test/py/multiplexed_log.py | 7 ++++---
>  1 file changed, 4 insertions(+), 3 deletions(-)
>
Applied to u-boot-dm

[U-Boot] [PATCH v2 05/17] buildman: Make the toolchain test more forgiving

[Simon Glass]

The filenames of the toolchains on kernel.org changes every now and then.
Fix it for the current change, and make the test use a regex so that it
has a better chance of passing with future changes too.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

Changes in v2: None

 tools/buildman/test.py | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/tools/buildman/test.py b/tools/buildman/test.py
index c36bcdf6fb7..7259f7b6507 100644
--- a/tools/buildman/test.py
+++ b/tools/buildman/test.py
@@ -422,8 +422,10 @@ class TestBuild(unittest.TestCase):
     def testToolchainDownload(self):
         """Test that we can download toolchains"""
         if use_network:
-            self.assertEqual('https://www.kernel.org/pub/tools/crosstool/files/bin/x86_64/4.9.0/x86_64-gcc-4.9.0-nolibc_arm-unknown-linux-gnueabi.tar.xz',
-                self.toolchains.LocateArchUrl('arm'))
+            url = self.toolchains.LocateArchUrl('arm')
+            self.assertRegexpMatches(url, 'https://www.kernel.org/pub/tools/'
+                    'crosstool/files/bin/x86_64/.*/'
+                    'x86_64-gcc-.*-nolibc_arm-.*linux-gnueabi.tar.xz')
 
 
 if __name__ == "__main__":
-- 
2.19.0.605.g01d371f741-goog

[U-Boot] [PATCH v2 06/17] Makefile: Add a 'check' target for make

[Simon Glass]

At present we use 'make tests' to run the tests. For many projects
'make check' is more common, so support that as well. Also add some help
to 'make help'.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

Changes in v2: None

 Makefile | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/Makefile b/Makefile
index 6869ac8f547..f578cce3663 100644
--- a/Makefile
+++ b/Makefile
@@ -1695,6 +1695,10 @@ help:
 	@echo  'Configuration targets:'
 	@$(MAKE) -f $(srctree)/scripts/kconfig/Makefile help
 	@echo  ''
+	@echo  'Test targets:'
+	@echo  ''
+	@echo  '  check           - Run all automated tests that use sandbox'
+	@echo  ''
 	@echo  'Other generic targets:'
 	@echo  '  all		  - Build all necessary images depending on configuration'
 	@echo  '  tests		  - Build U-Boot for sandbox and run tests'
@@ -1733,7 +1737,7 @@ help:
 	@echo  'Execute "make" or "make all" to build all targets marked with [*] '
 	@echo  'For further info see the ./README file'
 
-tests:
+tests check:
 	$(srctree)/test/run
 
 # Documentation targets
-- 
2.19.0.605.g01d371f741-goog

[U-Boot] [PATCH v2 07/17] test: Simplify the PATH setup

[Simon Glass]

Use 'export' to avoid repeating the path setup for each command.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

Changes in v2: None

 test/run | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)

diff --git a/test/run b/test/run
index d77a1c371b4..6b73813f9bc 100755
--- a/test/run
+++ b/test/run
@@ -16,23 +16,23 @@ run_test ./test/py/test.py --bd sandbox_spl --build -k test_ofplatdata.py
 # Run tests for the flat DT version of sandbox
 ./test/py/test.py --bd sandbox_flattree --build
 
+# Set up a path to dtc (device-tree compiler) and libfdt.py, a library it
+# provides and which is built by the sandbox_spl config.
 DTC_DIR=build-sandbox_spl/scripts/dtc
+export PYTHONPATH=${DTC_DIR}/pylibfdt
+export DTC=${DTC_DIR}/dtc
 
-PYTHONPATH=${DTC_DIR}/pylibfdt DTC=${DTC_DIR}/dtc run_test \
-	./tools/binman/binman -t
+run_test ./tools/binman/binman -t
 run_test ./tools/patman/patman --test
 run_test ./tools/buildman/buildman -t
-PYTHONPATH=${DTC_DIR}/pylibfdt DTC=${DTC_DIR}/dtc run_test ./tools/dtoc/dtoc -t
+run_test ./tools/dtoc/dtoc -t
 
 # This needs you to set up Python test coverage tools.
 # To enable Python test coverage on Debian-type distributions (e.g. Ubuntu):
 #   $ sudo apt-get install python-pytest python-coverage
-PYTHONPATH=${DTC_DIR}/pylibfdt DTC=${DTC_DIR}/dtc run_test \
-	./tools/binman/binman -T
-PYTHONPATH=${DTC_DIR}/pylibfdt DTC=${DTC_DIR}/dtc run_test \
-	./tools/dtoc/dtoc -T
-PYTHONPATH=${DTC_DIR}/pylibfdt DTC=${DTC_DIR}/dtc run_test \
-	./tools/dtoc/test_fdt -T
+run_test ./tools/binman/binman -T
+run_test ./tools/dtoc/dtoc -T
+run_test ./tools/dtoc/test_fdt -T
 
 if [ $result == 0 ]; then
 	echo "Tests passed!"
-- 
2.19.0.605.g01d371f741-goog

[U-Boot] [PATCH v2 08/17] test: Print the name of each test before running it

[Simon Glass]

At present the tests are run without any indication of what is running.
For the tests which start with a build this is pretty obvious, but for
tools it is not.

Add a name for each test we run, and print it before starting the test.
Signed-off-by: Simon Glass <sjg@chromium.org>
---

Changes in v2:
- Quote @$ correctly so that quoted arguments can be passed to run_test

 test/run | 30 +++++++++++++++++++-----------
 1 file changed, 19 insertions(+), 11 deletions(-)

diff --git a/test/run b/test/run
index 6b73813f9bc..d9901ae9f72 100755
--- a/test/run
+++ b/test/run
@@ -1,20 +1,28 @@
 #!/bin/bash
 
+# Script to run all U-Boot tests that use sandbox.
+
+# Runs a test and checks the exit code to decide if it passed
+#  $1:         Test name
+#  $2 onwards: command line to run
 run_test() {
-	$@
+	echo -n "$1: "
+	shift
+	"$@"
 	[ $? -ne 0 ] && result=$((result+1))
 }
 
 result=0
 
 # Run all tests that the standard sandbox build can support
-run_test ./test/py/test.py --bd sandbox --build
+run_test "sandbox" ./test/py/test.py --bd sandbox --build
 
 # Run tests which require sandbox_spl
-run_test ./test/py/test.py --bd sandbox_spl --build -k test_ofplatdata.py
+run_test "sandbox_spl" ./test/py/test.py --bd sandbox_spl --build \
+	-k test_ofplatdata.py
 
 # Run tests for the flat DT version of sandbox
-./test/py/test.py --bd sandbox_flattree --build
+run_test "sandbox_flattree" ./test/py/test.py --bd sandbox_flattree --build
 
 # Set up a path to dtc (device-tree compiler) and libfdt.py, a library it
 # provides and which is built by the sandbox_spl config.
@@ -22,17 +30,17 @@ DTC_DIR=build-sandbox_spl/scripts/dtc
 export PYTHONPATH=${DTC_DIR}/pylibfdt
 export DTC=${DTC_DIR}/dtc
 
-run_test ./tools/binman/binman -t
-run_test ./tools/patman/patman --test
-run_test ./tools/buildman/buildman -t
-run_test ./tools/dtoc/dtoc -t
+run_test "binman" ./tools/binman/binman -t
+run_test "patman" ./tools/patman/patman --test
+run_test "buildman" ./tools/buildman/buildman -t
+run_test "dtoc" ./tools/dtoc/dtoc -t
 
 # This needs you to set up Python test coverage tools.
 # To enable Python test coverage on Debian-type distributions (e.g. Ubuntu):
 #   $ sudo apt-get install python-pytest python-coverage
-run_test ./tools/binman/binman -T
-run_test ./tools/dtoc/dtoc -T
-run_test ./tools/dtoc/test_fdt -T
+run_test "binman code coverage" ./tools/binman/binman -T
+run_test "dtoc code coverage" ./tools/dtoc/dtoc -T
+run_test "fdt code coverage" ./tools/dtoc/test_fdt -T
 
 if [ $result == 0 ]; then
 	echo "Tests passed!"
-- 
2.19.0.605.g01d371f741-goog

[U-Boot] [PATCH v2 09/17] test: Tidy up comments and variable name

[Simon Glass]

The 'result' variable counts the number of failures in running the tests.
Rename it to 'failures' to make this more obvious. Also tidy up a few
comments.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

Changes in v2: None

 test/run | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)

diff --git a/test/run b/test/run
index d9901ae9f72..d64581ab140 100755
--- a/test/run
+++ b/test/run
@@ -9,10 +9,10 @@ run_test() {
 	echo -n "$1: "
 	shift
 	"$@"
-	[ $? -ne 0 ] && result=$((result+1))
+	[ $? -ne 0 ] && failures=$((failures+1))
 }
 
-result=0
+failures=0
 
 # Run all tests that the standard sandbox build can support
 run_test "sandbox" ./test/py/test.py --bd sandbox --build
@@ -21,7 +21,10 @@ run_test "sandbox" ./test/py/test.py --bd sandbox --build
 run_test "sandbox_spl" ./test/py/test.py --bd sandbox_spl --build \
 	-k test_ofplatdata.py
 
-# Run tests for the flat DT version of sandbox
+# Run tests for the flat-device-tree version of sandbox. This is a special
+# build which does not enable CONFIG_OF_LIVE for the live device tree, so we can
+# check that functionality is the same. The standard sandbox build (above) uses
+# CONFIG_OF_LIVE.
 run_test "sandbox_flattree" ./test/py/test.py --bd sandbox_flattree --build
 
 # Set up a path to dtc (device-tree compiler) and libfdt.py, a library it
@@ -33,6 +36,7 @@ export DTC=${DTC_DIR}/dtc
 run_test "binman" ./tools/binman/binman -t
 run_test "patman" ./tools/patman/patman --test
 run_test "buildman" ./tools/buildman/buildman -t
+run_test "fdt" ./tools/dtoc/test_fdt -t
 run_test "dtoc" ./tools/dtoc/dtoc -t
 
 # This needs you to set up Python test coverage tools.
@@ -42,7 +46,7 @@ run_test "binman code coverage" ./tools/binman/binman -T
 run_test "dtoc code coverage" ./tools/dtoc/dtoc -T
 run_test "fdt code coverage" ./tools/dtoc/test_fdt -T
 
-if [ $result == 0 ]; then
+if [ $failures == 0 ]; then
 	echo "Tests passed!"
 else
 	echo "Tests FAILED"
-- 
2.19.0.605.g01d371f741-goog

[U-Boot] [PATCH v2 10/17] binman: Add a default path to libfdt.py

[Simon Glass]

This module is often available in the sandbox_spl build created by
'make check'. Use this as a default path so that just typing 'binman -t'
(without setting PYTHONPATH) will generally run the tests.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

Changes in v2: None

 tools/binman/binman.py | 2 ++
 tools/dtoc/dtoc.py     | 5 +++++
 2 files changed, 7 insertions(+)

diff --git a/tools/binman/binman.py b/tools/binman/binman.py
index 1536e956517..f5af5359f3c 100755
--- a/tools/binman/binman.py
+++ b/tools/binman/binman.py
@@ -22,6 +22,8 @@ for dirname in ['../patman', '../dtoc', '..']:
 
 # Bring in the libfdt module
 sys.path.insert(0, 'scripts/dtc/pylibfdt')
+sys.path.insert(0, os.path.join(our_path,
+                '../../build-sandbox_spl/scripts/dtc/pylibfdt'))
 
 import cmdline
 import command
diff --git a/tools/dtoc/dtoc.py b/tools/dtoc/dtoc.py
index 827094e72ab..33b2589c561 100755
--- a/tools/dtoc/dtoc.py
+++ b/tools/dtoc/dtoc.py
@@ -34,6 +34,11 @@ import unittest
 our_path = os.path.dirname(os.path.realpath(__file__))
 sys.path.append(os.path.join(our_path, '../patman'))
 
+# Bring in the libfdt module
+sys.path.insert(0, 'scripts/dtc/pylibfdt')
+sys.path.insert(0, os.path.join(our_path,
+                '../../build-sandbox_spl/scripts/dtc/pylibfdt'))
+
 import dtb_platdata
 import test_util
 
-- 
2.19.0.605.g01d371f741-goog

[U-Boot] [PATCH v2 11/17] binman: Fix up removal of temporary directories

[Simon Glass]

At present 'make check' leaves some temporary directories around. Part of
this is because we call tools.PrepareOutputDir() twice in some cases,
without calling tools.FinaliseOutputDir() in between.

Fix this.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

Changes in v2: None

 tools/binman/elf_test.py   |  5 +++++
 tools/binman/entry_test.py |  8 ++++++--
 tools/binman/fdt_test.py   |  4 ++++
 tools/binman/ftest.py      |  8 +++-----
 tools/dtoc/test_fdt.py     | 10 +++++++---
 5 files changed, 25 insertions(+), 10 deletions(-)

diff --git a/tools/binman/elf_test.py b/tools/binman/elf_test.py
index c16f71401d1..b68530c19ba 100644
--- a/tools/binman/elf_test.py
+++ b/tools/binman/elf_test.py
@@ -10,6 +10,7 @@ import unittest
 
 import elf
 import test_util
+import tools
 
 binman_dir = os.path.dirname(os.path.realpath(sys.argv[0]))
 
@@ -46,6 +47,10 @@ class FakeSection:
 
 
 class TestElf(unittest.TestCase):
+    @classmethod
+    def setUpClass(self):
+        tools.SetInputDirs(['.'])
+
     def testAllSymbols(self):
         """Test that we can obtain a symbol from the ELF file"""
         fname = os.path.join(binman_dir, 'test', 'u_boot_ucode_ptr')
diff --git a/tools/binman/entry_test.py b/tools/binman/entry_test.py
index 69d85b4cedb..a8bc938f9e9 100644
--- a/tools/binman/entry_test.py
+++ b/tools/binman/entry_test.py
@@ -14,9 +14,14 @@ import fdt_util
 import tools
 
 class TestEntry(unittest.TestCase):
+    def setUp(self):
+        tools.PrepareOutputDir(None)
+
+    def tearDown(self):
+        tools.FinaliseOutputDir()
+
     def GetNode(self):
         binman_dir = os.path.dirname(os.path.realpath(sys.argv[0]))
-        tools.PrepareOutputDir(None)
         fname = fdt_util.EnsureCompiled(
             os.path.join(binman_dir,('test/05_simple.dts')))
         dtb = fdt.FdtScan(fname)
@@ -35,7 +40,6 @@ class TestEntry(unittest.TestCase):
         global entry
         reload(entry)
         entry.Entry.Create(None, self.GetNode(), 'u-boot-spl')
-        tools._RemoveOutputDir()
         del entry
 
     def testEntryContents(self):
diff --git a/tools/binman/fdt_test.py b/tools/binman/fdt_test.py
index 8ea098f38ae..b9167012d25 100644
--- a/tools/binman/fdt_test.py
+++ b/tools/binman/fdt_test.py
@@ -21,6 +21,10 @@ class TestFdt(unittest.TestCase):
         self._indir = tempfile.mkdtemp(prefix='binmant.')
         tools.PrepareOutputDir(self._indir, True)
 
+    @classmethod
+    def tearDownClass(self):
+        tools._FinaliseForTest()
+
     def TestFile(self, fname):
         return os.path.join(self._binman_dir, 'test', fname)
 
diff --git a/tools/binman/ftest.py b/tools/binman/ftest.py
index 57725c928e7..9ef259a699b 100644
--- a/tools/binman/ftest.py
+++ b/tools/binman/ftest.py
@@ -115,7 +115,6 @@ class TestFunctional(unittest.TestCase):
         TestFunctional._MakeInputFile('ecrw.bin', CROS_EC_RW_DATA)
         TestFunctional._MakeInputDir('devkeys')
         TestFunctional._MakeInputFile('bmpblk.bin', BMPBLK_DATA)
-        self._output_setup = False
 
         # ELF file with a '_dt_ucode_base_size' symbol
         with open(self.TestFile('u_boot_ucode_ptr')) as fd:
@@ -230,14 +229,13 @@ class TestFunctional(unittest.TestCase):
         Returns:
             Contents of device-tree binary
         """
-        if not self._output_setup:
-            tools.PrepareOutputDir(self._indir, True)
-            self._output_setup = True
+        tools.PrepareOutputDir(None)
         dtb = fdt_util.EnsureCompiled(self.TestFile(fname))
         with open(dtb) as fd:
             data = fd.read()
             TestFunctional._MakeInputFile(outfile, data)
-            return data
+        tools.FinaliseOutputDir()
+        return data
 
     def _GetDtbContentsForSplTpl(self, dtb_data, name):
         """Create a version of the main DTB for SPL or SPL
diff --git a/tools/dtoc/test_fdt.py b/tools/dtoc/test_fdt.py
index d2597020500..2e6febe8f38 100755
--- a/tools/dtoc/test_fdt.py
+++ b/tools/dtoc/test_fdt.py
@@ -60,7 +60,7 @@ class TestFdt(unittest.TestCase):
 
     @classmethod
     def tearDownClass(cls):
-        tools._FinaliseForTest()
+        tools.FinaliseOutputDir()
 
     def setUp(self):
         self.dtb = fdt.FdtScan('tools/dtoc/dtoc_test_simple.dts')
@@ -128,7 +128,7 @@ class TestNode(unittest.TestCase):
 
     @classmethod
     def tearDownClass(cls):
-        tools._FinaliseForTest()
+        tools.FinaliseOutputDir()
 
     def setUp(self):
         self.dtb = fdt.FdtScan('tools/dtoc/dtoc_test_simple.dts')
@@ -209,7 +209,7 @@ class TestProp(unittest.TestCase):
 
     @classmethod
     def tearDownClass(cls):
-        tools._FinaliseForTest()
+        tools.FinaliseOutputDir()
 
     def setUp(self):
         self.dtb = fdt.FdtScan('tools/dtoc/dtoc_test_simple.dts')
@@ -427,6 +427,10 @@ class TestFdtUtil(unittest.TestCase):
     def setUpClass(cls):
         tools.PrepareOutputDir(None)
 
+    @classmethod
+    def tearDownClass(cls):
+        tools.FinaliseOutputDir()
+
     def setUp(self):
         self.dtb = fdt.FdtScan('tools/dtoc/dtoc_test_simple.dts')
         self.node = self.dtb.GetNode('/spl-test')
-- 
2.19.0.605.g01d371f741-goog

[U-Boot] [PATCH v2 12/17] binman: Separate out testSplBssPad()

[Simon Glass]

At present this test runs binman twice, which means that the temporary
files from the first run do not get cleaned up. Split this into two tests
to fix this problem.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

Changes in v2: None

 tools/binman/ftest.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/tools/binman/ftest.py b/tools/binman/ftest.py
index 9ef259a699b..ed787745606 100644
--- a/tools/binman/ftest.py
+++ b/tools/binman/ftest.py
@@ -1022,10 +1022,12 @@ class TestFunctional(unittest.TestCase):
         data = self._DoReadFile('47_spl_bss_pad.dts')
         self.assertEqual(U_BOOT_SPL_DATA + (chr(0) * 10) + U_BOOT_DATA, data)
 
+    def testSplBssPadMissing(self):
+        """Test that a missing symbol is detected"""
         with open(self.TestFile('u_boot_ucode_ptr')) as fd:
             TestFunctional._MakeInputFile('spl/u-boot-spl', fd.read())
         with self.assertRaises(ValueError) as e:
-            data = self._DoReadFile('47_spl_bss_pad.dts')
+            self._DoReadFile('47_spl_bss_pad.dts')
         self.assertIn('Expected __bss_size symbol in spl/u-boot-spl',
                       str(e.exception))
 
-- 
2.19.0.605.g01d371f741-goog

[U-Boot] [PATCH v2 13/17] buildman: dtoc: Suppress unwanted output from test

[Simon Glass]

There are a few test cases which print output. Suppress this so that tests
can run silently in the normal case.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

Changes in v2: None

 tools/buildman/test.py  | 4 +++-
 tools/dtoc/test_dtoc.py | 6 ++++--
 2 files changed, 7 insertions(+), 3 deletions(-)

diff --git a/tools/buildman/test.py b/tools/buildman/test.py
index 7259f7b6507..e0c9d6da6a0 100644
--- a/tools/buildman/test.py
+++ b/tools/buildman/test.py
@@ -20,6 +20,7 @@ import control
 import command
 import commit
 import terminal
+import test_util
 import toolchain
 
 use_network = True
@@ -422,7 +423,8 @@ class TestBuild(unittest.TestCase):
     def testToolchainDownload(self):
         """Test that we can download toolchains"""
         if use_network:
-            url = self.toolchains.LocateArchUrl('arm')
+            with test_util.capture_sys_output() as (stdout, stderr):
+                url = self.toolchains.LocateArchUrl('arm')
             self.assertRegexpMatches(url, 'https://www.kernel.org/pub/tools/'
                     'crosstool/files/bin/x86_64/.*/'
                     'x86_64-gcc-.*-nolibc_arm-.*linux-gnueabi.tar.xz')
diff --git a/tools/dtoc/test_dtoc.py b/tools/dtoc/test_dtoc.py
index 72bcb37244e..11bead12607 100644
--- a/tools/dtoc/test_dtoc.py
+++ b/tools/dtoc/test_dtoc.py
@@ -385,7 +385,8 @@ U_BOOT_DEVICE(phandle_source2) = {
 
     def test_phandle_bad(self):
         """Test a node containing an invalid phandle fails"""
-        dtb_file = get_dtb_file('dtoc_test_phandle_bad.dts')
+        dtb_file = get_dtb_file('dtoc_test_phandle_bad.dts',
+                                capture_stderr=True)
         output = tools.GetOutputFilename('output')
         with self.assertRaises(ValueError) as e:
             dtb_platdata.run_steps(['struct'], dtb_file, False, output)
@@ -394,7 +395,8 @@ U_BOOT_DEVICE(phandle_source2) = {
 
     def test_phandle_bad2(self):
         """Test a phandle target missing its #*-cells property"""
-        dtb_file = get_dtb_file('dtoc_test_phandle_bad2.dts')
+        dtb_file = get_dtb_file('dtoc_test_phandle_bad2.dts',
+                                capture_stderr=True)
         output = tools.GetOutputFilename('output')
         with self.assertRaises(ValueError) as e:
             dtb_platdata.run_steps(['struct'], dtb_file, False, output)
-- 
2.19.0.605.g01d371f741-goog

[U-Boot] [PATCH v2 14/17] tools: Set an initial value for indir

[Simon Glass]

This variable is not documented or set up in the module. Fix this.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

Changes in v2: None

 tools/patman/tools.py | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/tools/patman/tools.py b/tools/patman/tools.py
index 1c9bf4e8100..bf099798e65 100644
--- a/tools/patman/tools.py
+++ b/tools/patman/tools.py
@@ -28,6 +28,9 @@ packages = {
     'lz4': 'liblz4-tool',
     }
 
+# List of paths to use when looking for an input file
+indir = []
+
 def PrepareOutputDir(dirname, preserve=False):
     """Select an output directory, ensuring it exists.
 
-- 
2.19.0.605.g01d371f741-goog

[U-Boot] [PATCH v2 15/17] patman: Don't clear progress in tout unless it was used

[Simon Glass]

At present calling Uninit() always called ClearProgress() which outputs
a \r character as well as spaces to remove any progress information on the
line. This can mess up the normal output of binman and other tools. Fix
this by outputing this only when progress information has actually been
previous written.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

Changes in v2: None

 tools/patman/tout.py | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/tools/patman/tout.py b/tools/patman/tout.py
index 4cd49e1c685..4957c7ae1df 100644
--- a/tools/patman/tout.py
+++ b/tools/patman/tout.py
@@ -15,6 +15,8 @@ NOTICE = 2
 INFO = 3
 DEBUG = 4
 
+in_progress = False
+
 """
 This class handles output of progress and other useful information
 to the user. It provides for simple verbosity level control and can
@@ -48,9 +50,11 @@ def UserIsPresent():
 
 def ClearProgress():
     """Clear any active progress message on the terminal."""
-    if verbose > 0 and stdout_is_tty:
+    global in_progress
+    if verbose > 0 and stdout_is_tty and in_progress:
         _stdout.write('\r%s\r' % (" " * len (_progress)))
         _stdout.flush()
+        in_progress = False
 
 def Progress(msg, warning=False, trailer='...'):
     """Display progress information.
@@ -58,6 +62,7 @@ def Progress(msg, warning=False, trailer='...'):
     Args:
         msg: Message to display.
         warning: True if this is a warning."""
+    global in_progress
     ClearProgress()
     if verbose > 0:
         _progress = msg + trailer
@@ -65,6 +70,7 @@ def Progress(msg, warning=False, trailer='...'):
             col = _color.YELLOW if warning else _color.GREEN
             _stdout.write('\r' + _color.Color(col, _progress))
             _stdout.flush()
+            in_progress = True
         else:
             _stdout.write(_progress + '\n')
 
-- 
2.19.0.605.g01d371f741-goog

[U-Boot] [PATCH v2 16/17] test: Reduce the number of tests run with sandbox_flattree

[Simon Glass]

We only need to run driver-model tests with this config, since this is the
only thing that is different when CONFIG_OF_LIVE is not defined. Filter
out the other tests to same time.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

Changes in v2:
- Add new patch to reduce the number of tests run with sandbox_flattree

 test/run | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/test/run b/test/run
index d64581ab140..fb8ff5da0cb 100755
--- a/test/run
+++ b/test/run
@@ -25,7 +25,8 @@ run_test "sandbox_spl" ./test/py/test.py --bd sandbox_spl --build \
 # build which does not enable CONFIG_OF_LIVE for the live device tree, so we can
 # check that functionality is the same. The standard sandbox build (above) uses
 # CONFIG_OF_LIVE.
-run_test "sandbox_flattree" ./test/py/test.py --bd sandbox_flattree --build
+run_test "sandbox_flattree" ./test/py/test.py --bd sandbox_flattree --build \
+	-k test_ut
 
 # Set up a path to dtc (device-tree compiler) and libfdt.py, a library it
 # provides and which is built by the sandbox_spl config.
-- 
2.19.0.605.g01d371f741-goog

[U-Boot] [PATCH v2 17/17] binman: Run tests concurrently

[Simon Glass]

At present the tests run one after the other using a single CPU. This is
not very efficient. Bring in the concurrencytest module and run the tests
concurrently, using one process for each CPU by default. A -P option
allows this to be overridden, which is necessary for code-coverage to
function correctly.

This requires fixing a few tests which are currently not fully
independent.

At some point we might consider doing this across all pytests in U-Boot.
There is a pytest version that supports specifying the number of processes
to use, but it did not work for me.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

Changes in v2:
- Add a patch to run binman tests concurrently

 .travis.yml                              |   1 +
 test/py/README.md                        |   1 +
 tools/binman/binman.py                   |  26 +++-
 tools/binman/cmdline.py                  |   2 +
 tools/binman/entry_test.py               |   7 +-
 tools/binman/ftest.py                    |  34 +++---
 tools/concurrencytest/.gitignore         |   1 +
 tools/concurrencytest/README.md          |  74 ++++++++++++
 tools/concurrencytest/concurrencytest.py | 144 +++++++++++++++++++++++
 tools/dtoc/dtoc.py                       |   2 +
 tools/dtoc/test_fdt.py                   |   2 +
 tools/patman/test_util.py                |   2 +-
 12 files changed, 274 insertions(+), 22 deletions(-)
 create mode 100644 tools/concurrencytest/.gitignore
 create mode 100644 tools/concurrencytest/README.md
 create mode 100644 tools/concurrencytest/concurrencytest.py

diff --git a/.travis.yml b/.travis.yml
index 2b759c9d686..f78749afb56 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -47,6 +47,7 @@ install:
  - virtualenv /tmp/venv
  - . /tmp/venv/bin/activate
  - pip install pytest
+ - pip install python-subunit
  - grub-mkimage -o ~/grub_x86.efi -O i386-efi normal  echo lsefimmap lsefi lsefisystab efinet tftp minicmd
  - mkdir ~/grub2-arm
  - ( cd ~/grub2-arm; wget -O - http://download.opensuse.org/ports/armv7hl/distribution/leap/42.2/repo/oss/suse/armv7hl/grub2-arm-efi-2.02~beta2-87.1.armv7hl.rpm | rpm2cpio | cpio -di )
diff --git a/test/py/README.md b/test/py/README.md
index aed2fd063a8..4d9d2b81d1e 100644
--- a/test/py/README.md
+++ b/test/py/README.md
@@ -29,6 +29,7 @@ tests. Similar package names should exist in other distributions.
 | -------------- | ----------------------------- |
 | python         | 2.7.5-5ubuntu3                |
 | python-pytest  | 2.5.1-1                       |
+| python-subunit | -                             |
 | gdisk          | 0.8.8-1ubuntu0.1              |
 | dfu-util       | 0.5-1                         |
 | dtc            | 1.4.0+dfsg-1                  |
diff --git a/tools/binman/binman.py b/tools/binman/binman.py
index f5af5359f3c..439908e6650 100755
--- a/tools/binman/binman.py
+++ b/tools/binman/binman.py
@@ -10,6 +10,7 @@
 """See README for more information"""
 
 import glob
+import multiprocessing
 import os
 import sys
 import traceback
@@ -17,7 +18,7 @@ import unittest
 
 # Bring in the patman and dtoc libraries
 our_path = os.path.dirname(os.path.realpath(__file__))
-for dirname in ['../patman', '../dtoc', '..']:
+for dirname in ['../patman', '../dtoc', '..', '../concurrencytest']:
     sys.path.insert(0, os.path.join(our_path, dirname))
 
 # Bring in the libfdt module
@@ -27,16 +28,22 @@ sys.path.insert(0, os.path.join(our_path,
 
 import cmdline
 import command
+use_concurrent = True
+try:
+    from concurrencytest import ConcurrentTestSuite, fork_for_tests
+except:
+    use_concurrent = False
 import control
 import test_util
 
-def RunTests(debug, args):
+def RunTests(debug, processes, args):
     """Run the functional tests and any embedded doctests
 
     Args:
         debug: True to enable debugging, which shows a full stack trace on error
         args: List of positional args provided to binman. This can hold a test
             name to execute (as in 'binman -t testSections', for example)
+        processes: Number of processes to use to run tests (None=same as #CPUs)
     """
     import elf_test
     import entry_test
@@ -54,19 +61,28 @@ def RunTests(debug, args):
     sys.argv = [sys.argv[0]]
     if debug:
         sys.argv.append('-D')
+    if debug:
+        sys.argv.append('-D')
 
     # Run the entry tests first ,since these need to be the first to import the
     # 'entry' module.
     test_name = args and args[0] or None
+    suite = unittest.TestSuite()
+    loader = unittest.TestLoader()
     for module in (entry_test.TestEntry, ftest.TestFunctional, fdt_test.TestFdt,
                    elf_test.TestElf, image_test.TestImage):
         if test_name:
             try:
-                suite = unittest.TestLoader().loadTestsFromName(test_name, module)
+                suite.addTests(loader.loadTestsFromName(test_name, module))
             except AttributeError:
                 continue
         else:
-            suite = unittest.TestLoader().loadTestsFromTestCase(module)
+            suite.addTests(loader.loadTestsFromTestCase(module))
+    if use_concurrent and processes != 1:
+        concurrent_suite = ConcurrentTestSuite(suite,
+                fork_for_tests(processes or multiprocessing.cpu_count()))
+        concurrent_suite.run(result)
+    else:
         suite.run(result)
 
     print result
@@ -115,7 +131,7 @@ def RunBinman(options, args):
         sys.tracebacklimit = 0
 
     if options.test:
-        ret_code = RunTests(options.debug, args[1:])
+        ret_code = RunTests(options.debug, options.processes, args[1:])
 
     elif options.test_coverage:
         RunTestCoverage()
diff --git a/tools/binman/cmdline.py b/tools/binman/cmdline.py
index f8caa7d2841..3886d52b3a0 100644
--- a/tools/binman/cmdline.py
+++ b/tools/binman/cmdline.py
@@ -46,6 +46,8 @@ def ParseArgs(argv):
     parser.add_option('-p', '--preserve', action='store_true',\
         help='Preserve temporary output directory even if option -O is not '
              'given')
+    parser.add_option('-P', '--processes', type=int,
+                      help='set number of processes to use for running tests')
     parser.add_option('-t', '--test', action='store_true',
                     default=False, help='run tests')
     parser.add_option('-T', '--test-coverage', action='store_true',
diff --git a/tools/binman/entry_test.py b/tools/binman/entry_test.py
index a8bc938f9e9..17ab2290140 100644
--- a/tools/binman/entry_test.py
+++ b/tools/binman/entry_test.py
@@ -13,6 +13,8 @@ import fdt
 import fdt_util
 import tools
 
+entry = None
+
 class TestEntry(unittest.TestCase):
     def setUp(self):
         tools.PrepareOutputDir(None)
@@ -38,7 +40,10 @@ class TestEntry(unittest.TestCase):
     def test2EntryImportLib(self):
         del sys.modules['importlib']
         global entry
-        reload(entry)
+        if entry:
+            reload(entry)
+        else:
+            import entry
         entry.Entry.Create(None, self.GetNode(), 'u-boot-spl')
         del entry
 
diff --git a/tools/binman/ftest.py b/tools/binman/ftest.py
index ed787745606..9f21027bd40 100644
--- a/tools/binman/ftest.py
+++ b/tools/binman/ftest.py
@@ -367,6 +367,16 @@ class TestFunctional(unittest.TestCase):
             os.makedirs(pathname)
         return pathname
 
+    @classmethod
+    def _SetupSplElf(self, src_fname='bss_data'):
+        """Set up an ELF file with a '_dt_ucode_base_size' symbol
+
+        Args:
+            Filename of ELF file to use as SPL
+        """
+        with open(self.TestFile(src_fname)) as fd:
+            TestFunctional._MakeInputFile('spl/u-boot-spl', fd.read())
+
     @classmethod
     def TestFile(self, fname):
         return os.path.join(self._binman_dir, 'test', fname)
@@ -715,8 +725,7 @@ class TestFunctional(unittest.TestCase):
 
     def testImagePadByte(self):
         """Test that the image pad byte can be specified"""
-        with open(self.TestFile('bss_data')) as fd:
-            TestFunctional._MakeInputFile('spl/u-boot-spl', fd.read())
+        self._SetupSplElf()
         data = self._DoReadFile('21_image_pad.dts')
         self.assertEqual(U_BOOT_SPL_DATA + (chr(0xff) * 1) + U_BOOT_DATA, data)
 
@@ -739,6 +748,7 @@ class TestFunctional(unittest.TestCase):
 
     def testPackSorted(self):
         """Test that entries can be sorted"""
+        self._SetupSplElf()
         data = self._DoReadFile('24_sorted.dts')
         self.assertEqual(chr(0) * 1 + U_BOOT_SPL_DATA + chr(0) * 2 +
                          U_BOOT_DATA, data)
@@ -781,6 +791,7 @@ class TestFunctional(unittest.TestCase):
 
     def testPackX86Rom(self):
         """Test that a basic x86 ROM can be created"""
+        self._SetupSplElf()
         data = self._DoReadFile('29_x86-rom.dts')
         self.assertEqual(U_BOOT_DATA + chr(0) * 7 + U_BOOT_SPL_DATA +
                          chr(0) * 2, data)
@@ -1017,15 +1028,13 @@ class TestFunctional(unittest.TestCase):
     def testSplBssPad(self):
         """Test that we can pad SPL's BSS with zeros"""
         # ELF file with a '__bss_size' symbol
-        with open(self.TestFile('bss_data')) as fd:
-            TestFunctional._MakeInputFile('spl/u-boot-spl', fd.read())
+        self._SetupSplElf()
         data = self._DoReadFile('47_spl_bss_pad.dts')
         self.assertEqual(U_BOOT_SPL_DATA + (chr(0) * 10) + U_BOOT_DATA, data)
 
     def testSplBssPadMissing(self):
         """Test that a missing symbol is detected"""
-        with open(self.TestFile('u_boot_ucode_ptr')) as fd:
-            TestFunctional._MakeInputFile('spl/u-boot-spl', fd.read())
+        self._SetupSplElf('u_boot_ucode_ptr')
         with self.assertRaises(ValueError) as e:
             self._DoReadFile('47_spl_bss_pad.dts')
         self.assertIn('Expected __bss_size symbol in spl/u-boot-spl',
@@ -1050,9 +1059,7 @@ class TestFunctional(unittest.TestCase):
             ucode_second: True if the microsecond entry is second instead of
                 third
         """
-        # ELF file with a '_dt_ucode_base_size' symbol
-        with open(self.TestFile('u_boot_ucode_ptr')) as fd:
-            TestFunctional._MakeInputFile('spl/u-boot-spl', fd.read())
+        self._SetupSplElf('u_boot_ucode_ptr')
         first, pos_and_size = self._RunMicrocodeTest(dts, U_BOOT_SPL_NODTB_DATA,
                                                      ucode_second=ucode_second)
         self.assertEqual('splnodtb with microc' + pos_and_size +
@@ -1094,8 +1101,7 @@ class TestFunctional(unittest.TestCase):
         addr = elf.GetSymbolAddress(elf_fname, '__image_copy_start')
         self.assertEqual(syms['_binman_u_boot_spl_prop_offset'].address, addr)
 
-        with open(self.TestFile('u_boot_binman_syms')) as fd:
-            TestFunctional._MakeInputFile('spl/u-boot-spl', fd.read())
+        self._SetupSplElf('u_boot_binman_syms')
         data = self._DoReadFile('53_symbols.dts')
         sym_values = struct.pack('<LQL', 0x24 + 0, 0x24 + 24, 0x24 + 20)
         expected = (sym_values + U_BOOT_SPL_DATA[16:] + chr(0xff) +
@@ -1727,16 +1733,14 @@ class TestFunctional(unittest.TestCase):
 
     def testElf(self):
         """Basic test of ELF entries"""
-        with open(self.TestFile('bss_data')) as fd:
-            TestFunctional._MakeInputFile('spl/u-boot-spl', fd.read())
+        self._SetupSplElf()
         with open(self.TestFile('bss_data')) as fd:
             TestFunctional._MakeInputFile('-boot', fd.read())
         data = self._DoReadFile('96_elf.dts')
 
     def testElfStripg(self):
         """Basic test of ELF entries"""
-        with open(self.TestFile('bss_data')) as fd:
-            TestFunctional._MakeInputFile('spl/u-boot-spl', fd.read())
+        self._SetupSplElf()
         with open(self.TestFile('bss_data')) as fd:
             TestFunctional._MakeInputFile('-boot', fd.read())
         data = self._DoReadFile('97_elf_strip.dts')
diff --git a/tools/concurrencytest/.gitignore b/tools/concurrencytest/.gitignore
new file mode 100644
index 00000000000..0d20b6487c6
--- /dev/null
+++ b/tools/concurrencytest/.gitignore
@@ -0,0 +1 @@
+*.pyc
diff --git a/tools/concurrencytest/README.md b/tools/concurrencytest/README.md
new file mode 100644
index 00000000000..8e65776f178
--- /dev/null
+++ b/tools/concurrencytest/README.md
@@ -0,0 +1,74 @@
+concurrencytest
+===============
+
+![testing goats](https://raw.github.com/cgoldberg/concurrencytest/master/testing-goats.png "testing goats")
+
+Python testtools extension for running unittest suites concurrently.
+
+----
+
+Install from PyPI:
+```
+pip install concurrencytest
+```
+
+----
+
+Requires:
+
+ * [testtools](https://pypi.python.org/pypi/testtools) : `pip install testtools`
+ * [python-subunit](https://pypi.python.org/pypi/python-subunit) : `pip install python-subunit`
+
+----
+
+Example:
+
+```python
+import time
+import unittest
+
+from concurrencytest import ConcurrentTestSuite, fork_for_tests
+
+
+class SampleTestCase(unittest.TestCase):
+    """Dummy tests that sleep for demo."""
+
+    def test_me_1(self):
+        time.sleep(0.5)
+
+    def test_me_2(self):
+        time.sleep(0.5)
+
+    def test_me_3(self):
+        time.sleep(0.5)
+
+    def test_me_4(self):
+        time.sleep(0.5)
+
+
+# Load tests from SampleTestCase defined above
+suite = unittest.TestLoader().loadTestsFromTestCase(SampleTestCase)
+runner = unittest.TextTestRunner()
+
+# Run tests sequentially
+runner.run(suite)
+
+# Run same tests across 4 processes
+suite = unittest.TestLoader().loadTestsFromTestCase(SampleTestCase)
+concurrent_suite = ConcurrentTestSuite(suite, fork_for_tests(4))
+runner.run(concurrent_suite)
+```
+Output:
+
+```
+....
+----------------------------------------------------------------------
+Ran 4 tests in 2.003s
+
+OK
+....
+----------------------------------------------------------------------
+Ran 4 tests in 0.504s
+
+OK
+```
diff --git a/tools/concurrencytest/concurrencytest.py b/tools/concurrencytest/concurrencytest.py
new file mode 100644
index 00000000000..418d7eed21d
--- /dev/null
+++ b/tools/concurrencytest/concurrencytest.py
@@ -0,0 +1,144 @@
+#!/usr/bin/env python
+# SPDX-License-Identifier: GPL-2.0+
+#
+# Modified by: Corey Goldberg, 2013
+#
+# Original code from:
+#   Bazaar (bzrlib.tests.__init__.py, v2.6, copied Jun 01 2013)
+#   Copyright (C) 2005-2011 Canonical Ltd
+
+"""Python testtools extension for running unittest suites concurrently.
+
+The `testtools` project provides a ConcurrentTestSuite class, but does
+not provide a `make_tests` implementation needed to use it.
+
+This allows you to parallelize a test run across a configurable number
+of worker processes. While this can speed up CPU-bound test runs, it is
+mainly useful for IO-bound tests that spend most of their time waiting for
+data to arrive from someplace else and can benefit from cocncurrency.
+
+Unix only.
+"""
+
+import os
+import sys
+import traceback
+import unittest
+from itertools import cycle
+from multiprocessing import cpu_count
+
+from subunit import ProtocolTestCase, TestProtocolClient
+from subunit.test_results import AutoTimingTestResultDecorator
+
+from testtools import ConcurrentTestSuite, iterate_tests
+
+
+_all__ = [
+    'ConcurrentTestSuite',
+    'fork_for_tests',
+    'partition_tests',
+]
+
+
+CPU_COUNT = cpu_count()
+
+
+def fork_for_tests(concurrency_num=CPU_COUNT):
+    """Implementation of `make_tests` used to construct `ConcurrentTestSuite`.
+
+    :param concurrency_num: number of processes to use.
+    """
+    def do_fork(suite):
+        """Take suite and start up multiple runners by forking (Unix only).
+
+        :param suite: TestSuite object.
+
+        :return: An iterable of TestCase-like objects which can each have
+        run(result) called on them to feed tests to result.
+        """
+        result = []
+        test_blocks = partition_tests(suite, concurrency_num)
+        # Clear the tests from the original suite so it doesn't keep them alive
+        suite._tests[:] = []
+        for process_tests in test_blocks:
+            process_suite = unittest.TestSuite(process_tests)
+            # Also clear each split list so new suite has only reference
+            process_tests[:] = []
+            c2pread, c2pwrite = os.pipe()
+            pid = os.fork()
+            if pid == 0:
+                try:
+                    stream = os.fdopen(c2pwrite, 'wb', 1)
+                    os.close(c2pread)
+                    # Leave stderr and stdout open so we can see test noise
+                    # Close stdin so that the child goes away if it decides to
+                    # read from stdin (otherwise its a roulette to see what
+                    # child actually gets keystrokes for pdb etc).
+                    sys.stdin.close()
+                    subunit_result = AutoTimingTestResultDecorator(
+                        TestProtocolClient(stream)
+                    )
+                    process_suite.run(subunit_result)
+                except:
+                    # Try and report traceback on stream, but exit with error
+                    # even if stream couldn't be created or something else
+                    # goes wrong.  The traceback is formatted to a string and
+                    # written in one go to avoid interleaving lines from
+                    # multiple failing children.
+                    try:
+                        stream.write(traceback.format_exc())
+                    finally:
+                        os._exit(1)
+                os._exit(0)
+            else:
+                os.close(c2pwrite)
+                stream = os.fdopen(c2pread, 'rb', 1)
+                test = ProtocolTestCase(stream)
+                result.append(test)
+        return result
+    return do_fork
+
+
+def partition_tests(suite, count):
+    """Partition suite into count lists of tests."""
+    # This just assigns tests in a round-robin fashion.  On one hand this
+    # splits up blocks of related tests that might run faster if they shared
+    # resources, but on the other it avoids assigning blocks of slow tests to
+    # just one partition.  So the slowest partition shouldn't be much slower
+    # than the fastest.
+    partitions = [list() for _ in range(count)]
+    tests = iterate_tests(suite)
+    for partition, test in zip(cycle(partitions), tests):
+        partition.append(test)
+    return partitions
+
+
+if __name__ == '__main__':
+    import time
+
+    class SampleTestCase(unittest.TestCase):
+        """Dummy tests that sleep for demo."""
+
+        def test_me_1(self):
+            time.sleep(0.5)
+
+        def test_me_2(self):
+            time.sleep(0.5)
+
+        def test_me_3(self):
+            time.sleep(0.5)
+
+        def test_me_4(self):
+            time.sleep(0.5)
+
+    # Load tests from SampleTestCase defined above
+    suite = unittest.TestLoader().loadTestsFromTestCase(SampleTestCase)
+    runner = unittest.TextTestRunner()
+
+    # Run tests sequentially
+    runner.run(suite)
+
+    # Run same tests across 4 processes
+    suite = unittest.TestLoader().loadTestsFromTestCase(SampleTestCase)
+    concurrent_suite = ConcurrentTestSuite(suite, fork_for_tests(4))
+    runner.run(concurrent_suite)
diff --git a/tools/dtoc/dtoc.py b/tools/dtoc/dtoc.py
index 33b2589c561..2277af9bf78 100755
--- a/tools/dtoc/dtoc.py
+++ b/tools/dtoc/dtoc.py
@@ -89,6 +89,8 @@ parser.add_option('--include-disabled', action='store_true',
                   help='Include disabled nodes')
 parser.add_option('-o', '--output', action='store', default='-',
                   help='Select output filename')
+parser.add_option('-P', '--processes', type=int,
+                  help='set number of processes to use for running tests')
 parser.add_option('-t', '--test', action='store_true', dest='test',
                   default=False, help='run tests')
 parser.add_option('-T', '--test-coverage', action='store_true',
diff --git a/tools/dtoc/test_fdt.py b/tools/dtoc/test_fdt.py
index 2e6febe8f38..8d70dd2a294 100755
--- a/tools/dtoc/test_fdt.py
+++ b/tools/dtoc/test_fdt.py
@@ -547,6 +547,8 @@ if __name__ != '__main__':
 parser = OptionParser()
 parser.add_option('-B', '--build-dir', type='string', default='b',
         help='Directory containing the build output')
+parser.add_option('-P', '--processes', type=int,
+                  help='set number of processes to use for running tests')
 parser.add_option('-t', '--test', action='store_true', dest='test',
                   default=False, help='run tests')
 parser.add_option('-T', '--test-coverage', action='store_true',
diff --git a/tools/patman/test_util.py b/tools/patman/test_util.py
index 0e79af871ab..687d40704ab 100644
--- a/tools/patman/test_util.py
+++ b/tools/patman/test_util.py
@@ -43,7 +43,7 @@ def RunTestCoverage(prog, filter_fname, exclude_list, build_dir, required=None):
     glob_list += exclude_list
     glob_list += ['*libfdt.py', '*site-packages*']
     cmd = ('PYTHONPATH=$PYTHONPATH:%s/sandbox_spl/tools python-coverage run '
-           '--omit "%s" %s -t' % (build_dir, ','.join(glob_list), prog))
+           '--omit "%s" %s -P1 -t' % (build_dir, ','.join(glob_list), prog))
     os.system(cmd)
     stdout = command.Output('python-coverage', 'report')
     lines = stdout.splitlines()
-- 
2.19.0.605.g01d371f741-goog

[U-Boot] [PATCH v2 17/17] binman: Run tests concurrently

[sjg at google.com]

At present the tests run one after the other using a single CPU. This is
not very efficient. Bring in the concurrencytest module and run the tests
concurrently, using one process for each CPU by default. A -P option
allows this to be overridden, which is necessary for code-coverage to
function correctly.

This requires fixing a few tests which are currently not fully
independent.

At some point we might consider doing this across all pytests in U-Boot.
There is a pytest version that supports specifying the number of processes
to use, but it did not work for me.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

Changes in v2:
- Add a patch to run binman tests concurrently

 .travis.yml                              |   1 +
 test/py/README.md                        |   1 +
 tools/binman/binman.py                   |  26 +++-
 tools/binman/cmdline.py                  |   2 +
 tools/binman/entry_test.py               |   7 +-
 tools/binman/ftest.py                    |  34 +++---
 tools/concurrencytest/.gitignore         |   1 +
 tools/concurrencytest/README.md          |  74 ++++++++++++
 tools/concurrencytest/concurrencytest.py | 144 +++++++++++++++++++++++
 tools/dtoc/dtoc.py                       |   2 +
 tools/dtoc/test_fdt.py                   |   2 +
 tools/patman/test_util.py                |   2 +-
 12 files changed, 274 insertions(+), 22 deletions(-)
 create mode 100644 tools/concurrencytest/.gitignore
 create mode 100644 tools/concurrencytest/README.md
 create mode 100644 tools/concurrencytest/concurrencytest.py

Applied to u-boot-dm

[U-Boot] [PATCH v2 16/17] test: Reduce the number of tests run with sandbox_flattree

[sjg at google.com]

We only need to run driver-model tests with this config, since this is the
only thing that is different when CONFIG_OF_LIVE is not defined. Filter
out the other tests to same time.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

Changes in v2:
- Add new patch to reduce the number of tests run with sandbox_flattree

 test/run | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

Applied to u-boot-dm

[U-Boot] [PATCH v2 15/17] patman: Don't clear progress in tout unless it was used

[sjg at google.com]

At present calling Uninit() always called ClearProgress() which outputs
a \r character as well as spaces to remove any progress information on the
line. This can mess up the normal output of binman and other tools. Fix
this by outputing this only when progress information has actually been
previous written.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

Changes in v2: None

 tools/patman/tout.py | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

Applied to u-boot-dm

[U-Boot] [PATCH v2 14/17] tools: Set an initial value for indir

[sjg at google.com]

This variable is not documented or set up in the module. Fix this.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

Changes in v2: None

 tools/patman/tools.py | 3 +++
 1 file changed, 3 insertions(+)

Applied to u-boot-dm

[U-Boot] [PATCH v2 13/17] buildman: dtoc: Suppress unwanted output from test

[sjg at google.com]

There are a few test cases which print output. Suppress this so that tests
can run silently in the normal case.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

Changes in v2: None

 tools/buildman/test.py  | 4 +++-
 tools/dtoc/test_dtoc.py | 6 ++++--
 2 files changed, 7 insertions(+), 3 deletions(-)

Applied to u-boot-dm

[U-Boot] [PATCH v2 12/17] binman: Separate out testSplBssPad()

[sjg at google.com]

At present this test runs binman twice, which means that the temporary
files from the first run do not get cleaned up. Split this into two tests
to fix this problem.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

Changes in v2: None

 tools/binman/ftest.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

Applied to u-boot-dm

[U-Boot] [PATCH v2 11/17] binman: Fix up removal of temporary directories

[sjg at google.com]

At present 'make check' leaves some temporary directories around. Part of
this is because we call tools.PrepareOutputDir() twice in some cases,
without calling tools.FinaliseOutputDir() in between.

Fix this.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

Changes in v2: None

 tools/binman/elf_test.py   |  5 +++++
 tools/binman/entry_test.py |  8 ++++++--
 tools/binman/fdt_test.py   |  4 ++++
 tools/binman/ftest.py      |  8 +++-----
 tools/dtoc/test_fdt.py     | 10 +++++++---
 5 files changed, 25 insertions(+), 10 deletions(-)

Applied to u-boot-dm

[U-Boot] [PATCH v2 10/17] binman: Add a default path to libfdt.py

[sjg at google.com]

This module is often available in the sandbox_spl build created by
'make check'. Use this as a default path so that just typing 'binman -t'
(without setting PYTHONPATH) will generally run the tests.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

Changes in v2: None

 tools/binman/binman.py | 2 ++
 tools/dtoc/dtoc.py     | 5 +++++
 2 files changed, 7 insertions(+)

Applied to u-boot-dm

[U-Boot] [PATCH v2 09/17] test: Tidy up comments and variable name

[sjg at google.com]

The 'result' variable counts the number of failures in running the tests.
Rename it to 'failures' to make this more obvious. Also tidy up a few
comments.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

Changes in v2: None

 test/run | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)

Applied to u-boot-dm

[U-Boot] [PATCH v2 08/17] test: Print the name of each test before running it

[sjg at google.com]

At present the tests are run without any indication of what is running.
For the tests which start with a build this is pretty obvious, but for
tools it is not.

Add a name for each test we run, and print it before starting the test.
Signed-off-by: Simon Glass <sjg@chromium.org>
---

Changes in v2:
- Quote @$ correctly so that quoted arguments can be passed to run_test

 test/run | 30 +++++++++++++++++++-----------
 1 file changed, 19 insertions(+), 11 deletions(-)

Applied to u-boot-dm

[U-Boot] [PATCH v2 07/17] test: Simplify the PATH setup

[sjg at google.com]

Use 'export' to avoid repeating the path setup for each command.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

Changes in v2: None

 test/run | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)

Applied to u-boot-dm

[U-Boot] [PATCH v2 06/17] Makefile: Add a 'check' target for make

[sjg at google.com]

At present we use 'make tests' to run the tests. For many projects
'make check' is more common, so support that as well. Also add some help
to 'make help'.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

Changes in v2: None

 Makefile | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

Applied to u-boot-dm

[U-Boot] [PATCH v2 05/17] buildman: Make the toolchain test more forgiving

[sjg at google.com]

The filenames of the toolchains on kernel.org changes every now and then.
Fix it for the current change, and make the test use a regex so that it
has a better chance of passing with future changes too.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

Changes in v2: None

 tools/buildman/test.py | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

Applied to u-boot-dm

[U-Boot] [PATCH v2 03/17] patman: Handle unicode in _ProjectConfigParser tests

[sjg at google.com]

With Python 2.7.15rc1, ConfigParser.SafeConfigParser has unfortunately
started returning unicode, for unknown reasons. Adjust the code to handle
this by converting everything to unicode. We cannot convert things to
ASCII since email addresses may be encoded with UTF-8.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

Changes in v2: None

 tools/patman/settings.py | 27 ++++++++++++++++++++-------
 1 file changed, 20 insertions(+), 7 deletions(-)

Applied to u-boot-dm

[U-Boot] [PATCH v2 02/17] sandbox: Unprotect DATA regions in bus tests

[sjg at google.com]

On my Ubuntu 18.04.1 machine two driver-model bus tests have started
failing recently. The problem appears to be that the DATA region of the
executable is protected. This does not seem correct, but perhaps there
is a reason.

To work around it, unprotect the regions in these tests before accessing
them.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

Changes in v2: None

 arch/sandbox/cpu/os.c | 11 +++++++++++
 include/os.h          | 12 ++++++++++++
 test/dm/bus.c         | 12 ++++++++++++
 3 files changed, 35 insertions(+)

Applied to u-boot-dm