[U-Boot] [PATCH 1/9] test/dm: clear unit test failure count each run

[U-Boot] [PATCH 1/9] test/dm: clear unit test failure count each run

[Stephen Warren]

From: Stephen Warren <swarren@nvidia.com>

The ut command prints a test failure count each time it is executed.
This is stored in a global variable which is never reset. Consequently,
the printed failure count accumulates across runs. Fix this by clearing
the counter each time "ut" is invoked.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
---
 test/dm/test-main.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/test/dm/test-main.c b/test/dm/test-main.c
index 91bdda83ab36..f2e004814387 100644
--- a/test/dm/test-main.c
+++ b/test/dm/test-main.c
@@ -81,6 +81,8 @@ static int dm_test_main(const char *test_name)
 	struct unit_test *test;
 	int run_count;
 
+	uts->fail_count = 0;
+
 	/*
 	 * If we have no device tree, or it only has a root node, then these
 	 * tests clearly aren't going to work...
-- 
2.7.0

[U-Boot] [PATCH 2/9] test/py: fix spawn.expect multiple match handling

[Stephen Warren]

From: Stephen Warren <swarren@nvidia.com>

Multiple patterns may be passed to spawn.expect(). The pattern which
matches at the earliest position should be designated as the match. This
aspect works correctly. When multiple patterns match at the same position,
priority should be given the the earliest entry in the list of patterns.
This aspect does not work correctly. This patch fixes it.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
---
 test/py/u_boot_spawn.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/test/py/u_boot_spawn.py b/test/py/u_boot_spawn.py
index d50807599027..7451455671b9 100644
--- a/test/py/u_boot_spawn.py
+++ b/test/py/u_boot_spawn.py
@@ -132,7 +132,7 @@ class Spawn(object):
                     m = pattern.search(self.buf)
                     if not m:
                         continue
-                    if earliest_m and m.start() > earliest_m.start():
+                    if earliest_m and m.start() >= earliest_m.start():
                         continue
                     earliest_m = m
                     earliest_pi = pi
-- 
2.7.0

[U-Boot] [PATCH 3/9] test.py: calculate bad patterns on change only

[Stephen Warren]

From: Stephen Warren <swarren@nvidia.com>

A future patch will use the bad_patterns array in multiple places. Rather
than duplicating the code to calculate it, or even sharing it in a
function and simply calling it redundantly when nothing has changed, only
re-calculate the list when some change is made to it. This reduces work.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
---
 test/py/u_boot_console_base.py | 49 +++++++++++++++++++++---------------------
 1 file changed, 25 insertions(+), 24 deletions(-)

diff --git a/test/py/u_boot_console_base.py b/test/py/u_boot_console_base.py
index c500fb3ae535..efb06cad0af0 100644
--- a/test/py/u_boot_console_base.py
+++ b/test/py/u_boot_console_base.py
@@ -23,6 +23,17 @@ pattern_stop_autoboot_prompt = re.compile('Hit any key to stop autoboot: ')
 pattern_unknown_command = re.compile('Unknown command \'.*\' - try \'help\'')
 pattern_error_notification = re.compile('## Error: ')
 
+PAT_ID = 0
+PAT_RE = 1
+
+bad_pattern_defs = (
+    ('spl_signon', pattern_u_boot_spl_signon),
+    ('main_signon', pattern_u_boot_main_signon),
+    ('stop_autoboot_prompt', pattern_stop_autoboot_prompt),
+    ('unknown_command', pattern_unknown_command),
+    ('error_notification', pattern_error_notification),
+)
+
 class ConsoleDisableCheck(object):
     """Context manager (for Python's with statement) that temporarily disables
     the specified console output error check. This is useful when deliberately
@@ -37,9 +48,11 @@ class ConsoleDisableCheck(object):
 
     def __enter__(self):
         self.console.disable_check_count[self.check_type] += 1
+        self.console.eval_bad_patterns()
 
     def __exit__(self, extype, value, traceback):
         self.console.disable_check_count[self.check_type] -= 1
+        self.console.eval_bad_patterns()
 
 class ConsoleBase(object):
     """The interface through which test functions interact with the U-Boot
@@ -77,16 +90,18 @@ class ConsoleBase(object):
         self.prompt = self.config.buildconfig['config_sys_prompt'][1:-1]
         self.prompt_escaped = re.escape(self.prompt)
         self.p = None
-        self.disable_check_count = {
-            'spl_signon': 0,
-            'main_signon': 0,
-            'unknown_command': 0,
-            'error_notification': 0,
-        }
+        self.disable_check_count = {pat[PAT_ID]: 0 for pat in bad_pattern_defs}
+        self.eval_bad_patterns()
 
         self.at_prompt = False
         self.at_prompt_logevt = None
 
+    def eval_bad_patterns(self):
+        self.bad_patterns = [pat[PAT_RE] for pat in bad_pattern_defs \
+            if self.disable_check_count[pat[PAT_ID]] == 0]
+        self.bad_pattern_ids = [pat[PAT_ID] for pat in bad_pattern_defs \
+            if self.disable_check_count[pat[PAT_ID]] == 0]
+
     def close(self):
         """Terminate the connection to the U-Boot console.
 
@@ -148,20 +163,6 @@ class ConsoleBase(object):
                 self.at_prompt_logevt != self.logstream.logfile.cur_evt:
             self.logstream.write(self.prompt, implicit=True)
 
-        bad_patterns = []
-        bad_pattern_ids = []
-        if (self.disable_check_count['spl_signon'] == 0):
-            bad_patterns.append(pattern_u_boot_spl_signon)
-            bad_pattern_ids.append('SPL signon')
-        if self.disable_check_count['main_signon'] == 0:
-            bad_patterns.append(pattern_u_boot_main_signon)
-            bad_pattern_ids.append('U-Boot main signon')
-        if self.disable_check_count['unknown_command'] == 0:
-            bad_patterns.append(pattern_unknown_command)
-            bad_pattern_ids.append('Unknown command')
-        if self.disable_check_count['error_notification'] == 0:
-            bad_patterns.append(pattern_error_notification)
-            bad_pattern_ids.append('Error notification')
         try:
             self.at_prompt = False
             if send_nl:
@@ -175,18 +176,18 @@ class ConsoleBase(object):
                     continue
                 chunk = re.escape(chunk)
                 chunk = chunk.replace('\\\n', '[\r\n]')
-                m = self.p.expect([chunk] + bad_patterns)
+                m = self.p.expect([chunk] + self.bad_patterns)
                 if m != 0:
                     self.at_prompt = False
                     raise Exception('Bad pattern found on console: ' +
-                                    bad_pattern_ids[m - 1])
+                                    self.bad_pattern_ids[m - 1])
             if not wait_for_prompt:
                 return
-            m = self.p.expect([self.prompt_escaped] + bad_patterns)
+            m = self.p.expect([self.prompt_escaped] + self.bad_patterns)
             if m != 0:
                 self.at_prompt = False
                 raise Exception('Bad pattern found on console: ' +
-                                bad_pattern_ids[m - 1])
+                                self.bad_pattern_ids[m - 1])
             self.at_prompt = True
             self.at_prompt_logevt = self.logstream.logfile.cur_evt
             # Only strip \r\n; space/TAB might be significant if testing
-- 
2.7.0

[U-Boot] [PATCH 4/9] test/py: check for bad patterns everywhere we wait

[Stephen Warren]

From: Stephen Warren <swarren@nvidia.com>

Currently, bad patterns are only honored when executing a shell command.
Other cases, such as the initial boot-up of U-Boot or when interacting
with command output rather than gathering all output prior to the shell
prompt, do not currently look for bad patterns in console output. This
patch makes sure that bad patterns are honored everywhere.

One benefit of this change is that if U-Boot sandbox fails to start up,
the error message it emits can be caught immediately, rather than relying
on a (long) timeout when waiting for the expected signon message and/or
command prompt.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
---
 test/py/u_boot_console_base.py | 26 +++++++++++++++++++-------
 1 file changed, 19 insertions(+), 7 deletions(-)

diff --git a/test/py/u_boot_console_base.py b/test/py/u_boot_console_base.py
index efb06cad0af0..71a00e863385 100644
--- a/test/py/u_boot_console_base.py
+++ b/test/py/u_boot_console_base.py
@@ -231,7 +231,10 @@ class ConsoleBase(object):
 
         if type(text) == type(''):
             text = re.escape(text)
-        self.p.expect([text])
+        m = self.p.expect([text] + self.bad_patterns)
+        if m != 0:
+            raise Exception('Bad pattern found on console: ' +
+                            self.bad_pattern_ids[m - 1])
 
     def drain_console(self):
         """Read from and log the U-Boot console for a short time.
@@ -298,8 +301,14 @@ class ConsoleBase(object):
             self.p.timeout = 30000
             self.p.logfile_read = self.logstream
             if self.config.buildconfig.get('CONFIG_SPL', False) == 'y':
-                self.p.expect([pattern_u_boot_spl_signon])
-            self.p.expect([pattern_u_boot_main_signon])
+                m = self.p.expect([pattern_u_boot_spl_signon] + self.bad_patterns)
+                if m != 0:
+                    raise Exception('Bad pattern found on console: ' +
+                                    self.bad_pattern_ids[m - 1])
+            m = self.p.expect([pattern_u_boot_main_signon] + self.bad_patterns)
+            if m != 0:
+                raise Exception('Bad pattern found on console: ' +
+                                self.bad_pattern_ids[m - 1])
             signon = self.p.after
             build_idx = signon.find(', Build:')
             if build_idx == -1:
@@ -307,12 +316,15 @@ class ConsoleBase(object):
             else:
                 self.u_boot_version_string = signon[:build_idx]
             while True:
-                match = self.p.expect([self.prompt_escaped,
-                                       pattern_stop_autoboot_prompt])
-                if match == 1:
+                m = self.p.expect([self.prompt_escaped,
+                    pattern_stop_autoboot_prompt] + self.bad_patterns)
+                if m == 0:
+                    break
+                if m == 1:
                     self.p.send(chr(3)) # CTRL-C
                     continue
-                break
+                raise Exception('Bad pattern found on console: ' +
+                                self.bad_pattern_ids[m - 2])
             self.at_prompt = True
             self.at_prompt_logevt = self.logstream.logfile.cur_evt
         except Exception as ex:
-- 
2.7.0

[U-Boot] [PATCH 5/9] test/py: detect another "bad pattern" in console output

[Stephen Warren]

From: Stephen Warren <swarren@nvidia.com>

Many error situations in U-Boot print the message:
    ### ERROR ### Please RESET the board ###

Add this to the list of bad patterns the test system detects. One
practical advantage of this change is to detect the case where sandbox
is told to use a particular DTB file, and the file cannot be opened.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
---
 test/py/u_boot_console_base.py | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/test/py/u_boot_console_base.py b/test/py/u_boot_console_base.py
index 71a00e863385..392f8cb88532 100644
--- a/test/py/u_boot_console_base.py
+++ b/test/py/u_boot_console_base.py
@@ -22,6 +22,7 @@ pattern_u_boot_main_signon = re.compile('(U-Boot \\d{4}\\.\\d{2}-[^\r\n]*)')
 pattern_stop_autoboot_prompt = re.compile('Hit any key to stop autoboot: ')
 pattern_unknown_command = re.compile('Unknown command \'.*\' - try \'help\'')
 pattern_error_notification = re.compile('## Error: ')
+pattern_error_please_reset = re.compile('### ERROR ### Please RESET the board ###')
 
 PAT_ID = 0
 PAT_RE = 1
@@ -32,6 +33,7 @@ bad_pattern_defs = (
     ('stop_autoboot_prompt', pattern_stop_autoboot_prompt),
     ('unknown_command', pattern_unknown_command),
     ('error_notification', pattern_error_notification),
+    ('error_please_reset', pattern_error_please_reset),
 )
 
 class ConsoleDisableCheck(object):
-- 
2.7.0

[U-Boot] [PATCH 6/9] test/py: correctly log xfail/xpass tests

[Stephen Warren]

From: Stephen Warren <swarren@nvidia.com>

Tests can complete in passed, skipped, xpass, xfailed, or failed, states.
Currently the U-Boot log generation code doesn't handle the xfailed or
xpass states since they aren't used. Add support for the remaining states.
Without this, tests that xfail end up being reported as skipped.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
---
 test/py/conftest.py         | 60 +++++++++++++++++++++++++++++----------------
 test/py/multiplexed_log.css |  8 ++++++
 test/py/multiplexed_log.py  | 30 ++++++++++++++++++++---
 3 files changed, 74 insertions(+), 24 deletions(-)

diff --git a/test/py/conftest.py b/test/py/conftest.py
index 9c9426aebe10..3e162cafcc4a 100644
--- a/test/py/conftest.py
+++ b/test/py/conftest.py
@@ -249,6 +249,8 @@ def u_boot_console(request):
 
 tests_not_run = set()
 tests_failed = set()
+tests_xpassed = set()
+tests_xfailed = set()
 tests_skipped = set()
 tests_passed = set()
 
@@ -289,6 +291,14 @@ def cleanup():
             log.status_skipped('%d skipped' % len(tests_skipped))
             for test in tests_skipped:
                 log.status_skipped('... ' + test)
+        if tests_xpassed:
+            log.status_xpass('%d xpass' % len(tests_xpassed))
+            for test in tests_xpassed:
+                log.status_xpass('... ' + test)
+        if tests_xfailed:
+            log.status_xfail('%d xfail' % len(tests_xfailed))
+            for test in tests_xfailed:
+                log.status_xfail('... ' + test)
         if tests_failed:
             log.status_fail('%d failed' % len(tests_failed))
             for test in tests_failed:
@@ -381,34 +391,42 @@ def pytest_runtest_protocol(item, nextitem):
     """
 
     reports = runtestprotocol(item, nextitem=nextitem)
-    failed = None
-    skipped = None
+
+    failure_cleanup = False
+    test_list = tests_passed
+    msg = 'OK'
+    msg_log = log.status_pass
     for report in reports:
         if report.outcome == 'failed':
-            failed = report
+            if hasattr(report, 'wasxfail'):
+                test_list = tests_xpassed
+                msg = 'XPASSED'
+                msg_log = log.status_xpass
+            else:
+                failure_cleanup = True
+                test_list = tests_failed
+                msg = 'FAILED:\n' + str(report.longrepr)
+                msg_log = log.status_fail
             break
         if report.outcome == 'skipped':
-            if not skipped:
-                skipped = report
-
-    if failed:
+            if hasattr(report, 'wasxfail'):
+                failure_cleanup = True
+                test_list = tests_xfailed
+                msg = 'XFAILED:\n' + str(report.longrepr)
+                msg_log = log.status_xfail
+                break
+            test_list = tests_skipped
+            msg = 'SKIPPED:\n' + str(report.longrepr)
+            msg_log = log.status_skipped
+
+    if failure_cleanup:
         console.drain_console()
-        tests_failed.add(item.name)
-    elif skipped:
-        tests_skipped.add(item.name)
-    else:
-        tests_passed.add(item.name)
+
+    test_list.add(item.name)
     tests_not_run.remove(item.name)
 
     try:
-        if failed:
-            msg = 'FAILED:\n' + str(failed.longrepr)
-            log.status_fail(msg)
-        elif skipped:
-            msg = 'SKIPPED:\n' + str(skipped.longrepr)
-            log.status_skipped(msg)
-        else:
-            log.status_pass('OK')
+        msg_log(msg)
     except:
         # If something went wrong with logging, it's better to let the test
         # process continue, which may report other exceptions that triggered
@@ -424,7 +442,7 @@ def pytest_runtest_protocol(item, nextitem):
 
     log.end_section(item.name)
 
-    if failed:
+    if failure_cleanup:
         console.cleanup_spawn()
 
     return reports
diff --git a/test/py/multiplexed_log.css b/test/py/multiplexed_log.css
index 50f7b9092983..f6240d52da66 100644
--- a/test/py/multiplexed_log.css
+++ b/test/py/multiplexed_log.css
@@ -83,6 +83,14 @@ pre {
     color: #ffff00
 }
 
+.status-xfail {
+    color: #ff7f00
+}
+
+.status-xpass {
+    color: #ff7f00
+}
+
 .status-fail {
     color: #ff0000
 }
diff --git a/test/py/multiplexed_log.py b/test/py/multiplexed_log.py
index fd3a9231a819..69a577e57720 100644
--- a/test/py/multiplexed_log.py
+++ b/test/py/multiplexed_log.py
@@ -408,7 +408,7 @@ class Logfile(object):
         """Write a note to the log file describing test(s) which passed.
 
         Args:
-            msg: A message describing passed test(s).
+            msg: A message describing the passed test(s).
 
         Returns:
             Nothing.
@@ -420,7 +420,7 @@ class Logfile(object):
         """Write a note to the log file describing skipped test(s).
 
         Args:
-            msg: A message describing passed test(s).
+            msg: A message describing the skipped test(s).
 
         Returns:
             Nothing.
@@ -428,11 +428,35 @@ class Logfile(object):
 
         self._note("status-skipped", msg)
 
+    def status_xfail(self, msg):
+        """Write a note to the log file describing xfailed test(s).
+
+        Args:
+            msg: A message describing the xfailed test(s).
+
+        Returns:
+            Nothing.
+        """
+
+        self._note("status-xfail", msg)
+
+    def status_xpass(self, msg):
+        """Write a note to the log file describing xpassed test(s).
+
+        Args:
+            msg: A message describing the xpassed test(s).
+
+        Returns:
+            Nothing.
+        """
+
+        self._note("status-xpass", msg)
+
     def status_fail(self, msg):
         """Write a note to the log file describing failed test(s).
 
         Args:
-            msg: A message describing passed test(s).
+            msg: A message describing the failed test(s).
 
         Returns:
             Nothing.
-- 
2.7.0

[U-Boot] [PATCH 7/9] test/py: pass test DTB to sandbox

[Stephen Warren]

From: Stephen Warren <swarren@nvidia.com>

This is required for at least "ut dm" to operate correctly.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
---
 test/py/u_boot_console_sandbox.py | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/test/py/u_boot_console_sandbox.py b/test/py/u_boot_console_sandbox.py
index 267f8b06508b..bbf41e788ba3 100644
--- a/test/py/u_boot_console_sandbox.py
+++ b/test/py/u_boot_console_sandbox.py
@@ -39,7 +39,12 @@ class ConsoleSandbox(ConsoleBase):
             A u_boot_spawn.Spawn object that is attached to U-Boot.
         """
 
-        return Spawn([self.config.build_dir + '/u-boot'])
+        cmd = [
+            self.config.build_dir + '/u-boot',
+            '-d',
+            self.config.build_dir + '/arch/sandbox/dts/test.dtb'
+        ]
+        return Spawn(cmd)
 
     def kill(self, sig):
         """Send a specific Unix signal to the sandbox process.
-- 
2.7.0

[U-Boot] [PATCH 8/9] test/py: run sandbox in source directory

[Stephen Warren]

From: Stephen Warren <swarren@nvidia.com>

Some unit tests expect the cwd of the sandbox process to be the root
of the source tree. Ensure that requirement is met.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
---
 test/py/u_boot_console_sandbox.py | 2 +-
 test/py/u_boot_spawn.py           | 8 ++++++--
 2 files changed, 7 insertions(+), 3 deletions(-)

diff --git a/test/py/u_boot_console_sandbox.py b/test/py/u_boot_console_sandbox.py
index bbf41e788ba3..a7263f30b8ca 100644
--- a/test/py/u_boot_console_sandbox.py
+++ b/test/py/u_boot_console_sandbox.py
@@ -44,7 +44,7 @@ class ConsoleSandbox(ConsoleBase):
             '-d',
             self.config.build_dir + '/arch/sandbox/dts/test.dtb'
         ]
-        return Spawn(cmd)
+        return Spawn(cmd, cwd=self.config.source_dir)
 
     def kill(self, sig):
         """Send a specific Unix signal to the sandbox process.
diff --git a/test/py/u_boot_spawn.py b/test/py/u_boot_spawn.py
index 7451455671b9..0f52d3e945c7 100644
--- a/test/py/u_boot_spawn.py
+++ b/test/py/u_boot_spawn.py
@@ -20,11 +20,13 @@ class Spawn(object):
     sent to the process, and responses waited for.
     """
 
-    def __init__(self, args):
+    def __init__(self, args, cwd=None):
         """Spawn (fork/exec) the sub-process.
 
         Args:
-            args: array of processs arguments. argv[0] is the command to execute.
+            args: array of processs arguments. argv[0] is the command to
+              execute.
+            cwd: the directory to run the process in, or None for no change.
 
         Returns:
             Nothing.
@@ -44,6 +46,8 @@ class Spawn(object):
                 # run under "go" (www.go.cd). Perhaps this happens under any
                 # background (non-interactive) system?
                 signal.signal(signal.SIGHUP, signal.SIG_DFL)
+                if cwd:
+                    os.chdir(cwd)
                 os.execvp(args[0], args)
             except:
                 print 'CHILD EXECEPTION:'
-- 
2.7.0

[U-Boot] [PATCH 9/9] test/py: run C-based unit tests

[Stephen Warren]

From: Stephen Warren <swarren@nvidia.com>

Add tests that execute the existing C-based unit test commands. These
simply run the command and validate the overall result. For now,
fine-grained details are not mapped into separate pytest test results in
the current implementation. However, the detail is available in the log
file for inspection, if attention is needed.

Now that the DM unit test runs under test/py, remove the manual shell
script that invokes it.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
---
 test/dm/test-dm.sh       | 16 -------------
 test/py/tests/test_ut.py | 59 ++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 59 insertions(+), 16 deletions(-)
 delete mode 100755 test/dm/test-dm.sh
 create mode 100644 test/py/tests/test_ut.py

diff --git a/test/dm/test-dm.sh b/test/dm/test-dm.sh
deleted file mode 100755
index 1a0f1509b415..000000000000
--- a/test/dm/test-dm.sh
+++ /dev/null
@@ -1,16 +0,0 @@
-#!/bin/sh
-
-die() {
-	echo $1
-	exit 1
-}
-
-NUM_CPUS=$(cat /proc/cpuinfo |grep -c processor)
-make O=sandbox sandbox_config || die "Cannot configure U-Boot"
-make O=sandbox -s -j${NUM_CPUS} || die "Cannot build U-Boot"
-dd if=/dev/zero of=spi.bin bs=1M count=2
-echo -n "this is a test" > testflash.bin
-dd if=/dev/zero bs=1M count=4 >>testflash.bin
-./sandbox/u-boot -d ./sandbox/arch/sandbox/dts/test.dtb -c "ut dm"
-rm spi.bin
-rm testflash.bin
diff --git a/test/py/tests/test_ut.py b/test/py/tests/test_ut.py
new file mode 100644
index 000000000000..b033ca54d756
--- /dev/null
+++ b/test/py/tests/test_ut.py
@@ -0,0 +1,59 @@
+# Copyright (c) 2016, NVIDIA CORPORATION. All rights reserved.
+#
+# SPDX-License-Identifier: GPL-2.0
+
+import os.path
+import pytest
+
+ at pytest.mark.buildconfigspec('ut_dm')
+def test_ut_dm(u_boot_console):
+    """Execute the "ut dm" command."""
+
+    fn = u_boot_console.config.source_dir + '/testflash.bin'
+    if not os.path.exists(fn):
+        data = 'this is a test'
+        data += '\x00' * ((4 * 1024 * 1024) - len(data))
+        with open(fn, 'wb') as fh:
+            fh.write(data)
+
+    output = u_boot_console.run_command('ut dm')
+    assert output.endswith('Failures: 0')
+
+ at pytest.mark.buildconfigspec('ut_env')
+def test_ut_env(u_boot_console):
+    """Execute the "ut env" command."""
+
+    output = u_boot_console.run_command('ut env')
+    assert output.endswith('Failures: 0')
+
+ at pytest.mark.buildconfigspec('ut_time')
+def test_ut_time(u_boot_console):
+    """Execute the "ut time" command."""
+
+    output = u_boot_console.run_command('ut time')
+    assert output.endswith('Test passed')
+
+ at pytest.mark.buildconfigspec('sandbox')
+def test_ut_cmd(u_boot_console):
+    """Execute the "ut_cmd" command."""
+
+    output = u_boot_console.run_command('ut_cmd')
+    assert output.endswith('do_ut_cmd: Everything went swimmingly')
+
+ at pytest.mark.buildconfigspec('sandbox')
+def test_ut_compression(u_boot_console):
+    """Execute the "ut_compression" command."""
+
+    output = u_boot_console.run_command('ut_compression')
+    assert output.endswith('ut_compression ok')
+
+# Even when this passes, it prints lots of scary messages such as:
+#     Must RESET board to recover
+# Equally, it fails if "ut dm" has been run first in the U-Boot session.
+# Don't enable this test until those issues have been researched/solved.
+#@pytest.mark.buildconfigspec('sandbox')
+#def test_ut_compression(u_boot_console):
+#    """Execute the "ut_image_decomp" command."""
+#
+#    output = u_boot_console.run_command('ut_image_decomp')
+#    assert output.endswith('ut_image_decomp ok')
-- 
2.7.0

[U-Boot] [PATCH 1/9] test/dm: clear unit test failure count each run

[Simon Glass]

On 27 January 2016 at 23:57, Stephen Warren <swarren@wwwdotorg.org> wrote:
> From: Stephen Warren <swarren@nvidia.com>
>
> The ut command prints a test failure count each time it is executed.
> This is stored in a global variable which is never reset. Consequently,
> the printed failure count accumulates across runs. Fix this by clearing
> the counter each time "ut" is invoked.
>
> Signed-off-by: Stephen Warren <swarren@nvidia.com>
> ---
>  test/dm/test-main.c | 2 ++
>  1 file changed, 2 insertions(+)

Acked-by: Simon Glass <sjg@chromium.org>

[U-Boot] [PATCH 2/9] test/py: fix spawn.expect multiple match handling

[Simon Glass]

On 27 January 2016 at 23:57, Stephen Warren <swarren@wwwdotorg.org> wrote:
> From: Stephen Warren <swarren@nvidia.com>
>
> Multiple patterns may be passed to spawn.expect(). The pattern which
> matches at the earliest position should be designated as the match. This
> aspect works correctly. When multiple patterns match at the same position,
> priority should be given the the earliest entry in the list of patterns.
> This aspect does not work correctly. This patch fixes it.
>
> Signed-off-by: Stephen Warren <swarren@nvidia.com>
> ---
>  test/py/u_boot_spawn.py | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)

Acked-by: Simon Glass <sjg@chromium.org>

[U-Boot] [PATCH 3/9] test.py: calculate bad patterns on change only

[Simon Glass]

On 27 January 2016 at 23:57, Stephen Warren <swarren@wwwdotorg.org> wrote:
> From: Stephen Warren <swarren@nvidia.com>
>
> A future patch will use the bad_patterns array in multiple places. Rather
> than duplicating the code to calculate it, or even sharing it in a
> function and simply calling it redundantly when nothing has changed, only
> re-calculate the list when some change is made to it. This reduces work.
>
> Signed-off-by: Stephen Warren <swarren@nvidia.com>
> ---
>  test/py/u_boot_console_base.py | 49 +++++++++++++++++++++---------------------
>  1 file changed, 25 insertions(+), 24 deletions(-)

Acked-by: Simon Glass <sjg@chromium.org>

[U-Boot] [PATCH 4/9] test/py: check for bad patterns everywhere we wait

[Simon Glass]

On 27 January 2016 at 23:57, Stephen Warren <swarren@wwwdotorg.org> wrote:
> From: Stephen Warren <swarren@nvidia.com>
>
> Currently, bad patterns are only honored when executing a shell command.
> Other cases, such as the initial boot-up of U-Boot or when interacting
> with command output rather than gathering all output prior to the shell
> prompt, do not currently look for bad patterns in console output. This
> patch makes sure that bad patterns are honored everywhere.
>
> One benefit of this change is that if U-Boot sandbox fails to start up,
> the error message it emits can be caught immediately, rather than relying
> on a (long) timeout when waiting for the expected signon message and/or
> command prompt.
>
> Signed-off-by: Stephen Warren <swarren@nvidia.com>
> ---
>  test/py/u_boot_console_base.py | 26 +++++++++++++++++++-------
>  1 file changed, 19 insertions(+), 7 deletions(-)

Acked-by: Simon Glass <sjg@chromium.org>

[U-Boot] [PATCH 5/9] test/py: detect another "bad pattern" in console output

[Simon Glass]

On 27 January 2016 at 23:57, Stephen Warren <swarren@wwwdotorg.org> wrote:
> From: Stephen Warren <swarren@nvidia.com>
>
> Many error situations in U-Boot print the message:
>     ### ERROR ### Please RESET the board ###
>
> Add this to the list of bad patterns the test system detects. One
> practical advantage of this change is to detect the case where sandbox
> is told to use a particular DTB file, and the file cannot be opened.
>
> Signed-off-by: Stephen Warren <swarren@nvidia.com>
> ---
>  test/py/u_boot_console_base.py | 2 ++
>  1 file changed, 2 insertions(+)

Acked-by: Simon Glass <sjg@chromium.org>

[U-Boot] [PATCH 6/9] test/py: correctly log xfail/xpass tests

[Simon Glass]

On 27 January 2016 at 23:57, Stephen Warren <swarren@wwwdotorg.org> wrote:
> From: Stephen Warren <swarren@nvidia.com>
>
> Tests can complete in passed, skipped, xpass, xfailed, or failed, states.
> Currently the U-Boot log generation code doesn't handle the xfailed or
> xpass states since they aren't used. Add support for the remaining states.
> Without this, tests that xfail end up being reported as skipped.
>
> Signed-off-by: Stephen Warren <swarren@nvidia.com>
> ---
>  test/py/conftest.py         | 60 +++++++++++++++++++++++++++++----------------
>  test/py/multiplexed_log.css |  8 ++++++
>  test/py/multiplexed_log.py  | 30 ++++++++++++++++++++---
>  3 files changed, 74 insertions(+), 24 deletions(-)

Acked-by: Simon Glass <sjg@chromium.org>

[U-Boot] [PATCH 7/9] test/py: pass test DTB to sandbox

[Simon Glass]

On 27 January 2016 at 23:57, Stephen Warren <swarren@wwwdotorg.org> wrote:
> From: Stephen Warren <swarren@nvidia.com>
>
> This is required for at least "ut dm" to operate correctly.
>
> Signed-off-by: Stephen Warren <swarren@nvidia.com>
> ---
>  test/py/u_boot_console_sandbox.py | 7 ++++++-
>  1 file changed, 6 insertions(+), 1 deletion(-)

Acked-by: Simon Glass <sjg@chromium.org>

[U-Boot] [PATCH 8/9] test/py: run sandbox in source directory

[Simon Glass]

On 27 January 2016 at 23:57, Stephen Warren <swarren@wwwdotorg.org> wrote:
> From: Stephen Warren <swarren@nvidia.com>
>
> Some unit tests expect the cwd of the sandbox process to be the root
> of the source tree. Ensure that requirement is met.
>
> Signed-off-by: Stephen Warren <swarren@nvidia.com>
> ---
>  test/py/u_boot_console_sandbox.py | 2 +-
>  test/py/u_boot_spawn.py           | 8 ++++++--
>  2 files changed, 7 insertions(+), 3 deletions(-)

Acked-by: Simon Glass <sjg@chromium.org>

[U-Boot] [PATCH 9/9] test/py: run C-based unit tests

[Simon Glass]

Hi Stephen,

On 27 January 2016 at 23:57, Stephen Warren <swarren@wwwdotorg.org> wrote:
> From: Stephen Warren <swarren@nvidia.com>
>
> Add tests that execute the existing C-based unit test commands. These
> simply run the command and validate the overall result. For now,
> fine-grained details are not mapped into separate pytest test results in
> the current implementation. However, the detail is available in the log
> file for inspection, if attention is needed.
>
> Now that the DM unit test runs under test/py, remove the manual shell
> script that invokes it.
>
> Signed-off-by: Stephen Warren <swarren@nvidia.com>
> ---
>  test/dm/test-dm.sh       | 16 -------------
>  test/py/tests/test_ut.py | 59 ++++++++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 59 insertions(+), 16 deletions(-)
>  delete mode 100755 test/dm/test-dm.sh
>  create mode 100644 test/py/tests/test_ut.py
>
> diff --git a/test/dm/test-dm.sh b/test/dm/test-dm.sh
> deleted file mode 100755
> index 1a0f1509b415..000000000000
> --- a/test/dm/test-dm.sh
> +++ /dev/null
> @@ -1,16 +0,0 @@
> -#!/bin/sh
> -
> -die() {
> -       echo $1
> -       exit 1
> -}
> -
> -NUM_CPUS=$(cat /proc/cpuinfo |grep -c processor)
> -make O=sandbox sandbox_config || die "Cannot configure U-Boot"
> -make O=sandbox -s -j${NUM_CPUS} || die "Cannot build U-Boot"
> -dd if=/dev/zero of=spi.bin bs=1M count=2
> -echo -n "this is a test" > testflash.bin
> -dd if=/dev/zero bs=1M count=4 >>testflash.bin
> -./sandbox/u-boot -d ./sandbox/arch/sandbox/dts/test.dtb -c "ut dm"
> -rm spi.bin
> -rm testflash.bin
> diff --git a/test/py/tests/test_ut.py b/test/py/tests/test_ut.py
> new file mode 100644
> index 000000000000..b033ca54d756
> --- /dev/null
> +++ b/test/py/tests/test_ut.py
> @@ -0,0 +1,59 @@
> +# Copyright (c) 2016, NVIDIA CORPORATION. All rights reserved.
> +#
> +# SPDX-License-Identifier: GPL-2.0
> +
> +import os.path
> +import pytest
> +
> + at pytest.mark.buildconfigspec('ut_dm')
> +def test_ut_dm(u_boot_console):
> +    """Execute the "ut dm" command."""
> +
> +    fn = u_boot_console.config.source_dir + '/testflash.bin'
> +    if not os.path.exists(fn):
> +        data = 'this is a test'
> +        data += '\x00' * ((4 * 1024 * 1024) - len(data))
> +        with open(fn, 'wb') as fh:
> +            fh.write(data)
> +

The SPI tests are currently disabled, but they will need the spi.bin file.

> +    output = u_boot_console.run_command('ut dm')
> +    assert output.endswith('Failures: 0')
> +
> + at pytest.mark.buildconfigspec('ut_env')
> +def test_ut_env(u_boot_console):
> +    """Execute the "ut env" command."""
> +
> +    output = u_boot_console.run_command('ut env')
> +    assert output.endswith('Failures: 0')
> +
> + at pytest.mark.buildconfigspec('ut_time')
> +def test_ut_time(u_boot_console):
> +    """Execute the "ut time" command."""
> +
> +    output = u_boot_console.run_command('ut time')
> +    assert output.endswith('Test passed')
> +
> + at pytest.mark.buildconfigspec('sandbox')
> +def test_ut_cmd(u_boot_console):
> +    """Execute the "ut_cmd" command."""
> +
> +    output = u_boot_console.run_command('ut_cmd')
> +    assert output.endswith('do_ut_cmd: Everything went swimmingly')
> +
> + at pytest.mark.buildconfigspec('sandbox')
> +def test_ut_compression(u_boot_console):
> +    """Execute the "ut_compression" command."""
> +
> +    output = u_boot_console.run_command('ut_compression')
> +    assert output.endswith('ut_compression ok')
> +
> +# Even when this passes, it prints lots of scary messages such as:
> +#     Must RESET board to recover
> +# Equally, it fails if "ut dm" has been run first in the U-Boot session.
> +# Don't enable this test until those issues have been researched/solved.

The messages are correct because the test is deliberating making the
output buffer to small to check that decompression doesn't overrun the
end.

One way around this would be to silence the console. Sett
dm_test_main() for how it silences the console, and records the
output. Alternatively you could just look for the 'ok' message at the
end.

> +#@pytest.mark.buildconfigspec('sandbox')
> +#def test_ut_compression(u_boot_console):
> +#    """Execute the "ut_image_decomp" command."""
> +#
> +#    output = u_boot_console.run_command('ut_image_decomp')
> +#    assert output.endswith('ut_image_decomp ok')
> --
> 2.7.0
>

Regards,
Simon

[U-Boot] [PATCH 1/9] test/dm: clear unit test failure count each run

[Simon Glass]

On 28 January 2016 at 20:46, Simon Glass <sjg@chromium.org> wrote:
> On 27 January 2016 at 23:57, Stephen Warren <swarren@wwwdotorg.org> wrote:
>> From: Stephen Warren <swarren@nvidia.com>
>>
>> The ut command prints a test failure count each time it is executed.
>> This is stored in a global variable which is never reset. Consequently,
>> the printed failure count accumulates across runs. Fix this by clearing
>> the counter each time "ut" is invoked.
>>
>> Signed-off-by: Stephen Warren <swarren@nvidia.com>
>> ---
>>  test/dm/test-main.c | 2 ++
>>  1 file changed, 2 insertions(+)
>
> Acked-by: Simon Glass <sjg@chromium.org>

Applied to u-boot-dm, thanks!

[U-Boot] [PATCH 2/9] test/py: fix spawn.expect multiple match handling

[Simon Glass]

On 28 January 2016 at 20:47, Simon Glass <sjg@chromium.org> wrote:
> On 27 January 2016 at 23:57, Stephen Warren <swarren@wwwdotorg.org> wrote:
>> From: Stephen Warren <swarren@nvidia.com>
>>
>> Multiple patterns may be passed to spawn.expect(). The pattern which
>> matches at the earliest position should be designated as the match. This
>> aspect works correctly. When multiple patterns match at the same position,
>> priority should be given the the earliest entry in the list of patterns.
>> This aspect does not work correctly. This patch fixes it.
>>
>> Signed-off-by: Stephen Warren <swarren@nvidia.com>
>> ---
>>  test/py/u_boot_spawn.py | 2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> Acked-by: Simon Glass <sjg@chromium.org>

Applied to u-boot-dm, thanks!

[U-Boot] [PATCH 3/9] test.py: calculate bad patterns on change only

[Simon Glass]

On 28 January 2016 at 20:47, Simon Glass <sjg@chromium.org> wrote:
> On 27 January 2016 at 23:57, Stephen Warren <swarren@wwwdotorg.org> wrote:
>> From: Stephen Warren <swarren@nvidia.com>
>>
>> A future patch will use the bad_patterns array in multiple places. Rather
>> than duplicating the code to calculate it, or even sharing it in a
>> function and simply calling it redundantly when nothing has changed, only
>> re-calculate the list when some change is made to it. This reduces work.
>>
>> Signed-off-by: Stephen Warren <swarren@nvidia.com>
>> ---
>>  test/py/u_boot_console_base.py | 49 +++++++++++++++++++++---------------------
>>  1 file changed, 25 insertions(+), 24 deletions(-)
>
> Acked-by: Simon Glass <sjg@chromium.org>

Applied to u-boot-dm, thanks!

[U-Boot] [PATCH 4/9] test/py: check for bad patterns everywhere we wait

[Simon Glass]

On 28 January 2016 at 20:47, Simon Glass <sjg@chromium.org> wrote:
> On 27 January 2016 at 23:57, Stephen Warren <swarren@wwwdotorg.org> wrote:
>> From: Stephen Warren <swarren@nvidia.com>
>>
>> Currently, bad patterns are only honored when executing a shell command.
>> Other cases, such as the initial boot-up of U-Boot or when interacting
>> with command output rather than gathering all output prior to the shell
>> prompt, do not currently look for bad patterns in console output. This
>> patch makes sure that bad patterns are honored everywhere.
>>
>> One benefit of this change is that if U-Boot sandbox fails to start up,
>> the error message it emits can be caught immediately, rather than relying
>> on a (long) timeout when waiting for the expected signon message and/or
>> command prompt.
>>
>> Signed-off-by: Stephen Warren <swarren@nvidia.com>
>> ---
>>  test/py/u_boot_console_base.py | 26 +++++++++++++++++++-------
>>  1 file changed, 19 insertions(+), 7 deletions(-)
>
> Acked-by: Simon Glass <sjg@chromium.org>

Applied to u-boot-dm, thanks!

[U-Boot] [PATCH 5/9] test/py: detect another "bad pattern" in console output

[Simon Glass]

On 28 January 2016 at 20:47, Simon Glass <sjg@chromium.org> wrote:
> On 27 January 2016 at 23:57, Stephen Warren <swarren@wwwdotorg.org> wrote:
>> From: Stephen Warren <swarren@nvidia.com>
>>
>> Many error situations in U-Boot print the message:
>>     ### ERROR ### Please RESET the board ###
>>
>> Add this to the list of bad patterns the test system detects. One
>> practical advantage of this change is to detect the case where sandbox
>> is told to use a particular DTB file, and the file cannot be opened.
>>
>> Signed-off-by: Stephen Warren <swarren@nvidia.com>
>> ---
>>  test/py/u_boot_console_base.py | 2 ++
>>  1 file changed, 2 insertions(+)
>
> Acked-by: Simon Glass <sjg@chromium.org>

Applied to u-boot-dm, thanks!

[U-Boot] [PATCH 6/9] test/py: correctly log xfail/xpass tests

[Simon Glass]

On 28 January 2016 at 20:47, Simon Glass <sjg@chromium.org> wrote:
> On 27 January 2016 at 23:57, Stephen Warren <swarren@wwwdotorg.org> wrote:
>> From: Stephen Warren <swarren@nvidia.com>
>>
>> Tests can complete in passed, skipped, xpass, xfailed, or failed, states.
>> Currently the U-Boot log generation code doesn't handle the xfailed or
>> xpass states since they aren't used. Add support for the remaining states.
>> Without this, tests that xfail end up being reported as skipped.
>>
>> Signed-off-by: Stephen Warren <swarren@nvidia.com>
>> ---
>>  test/py/conftest.py         | 60 +++++++++++++++++++++++++++++----------------
>>  test/py/multiplexed_log.css |  8 ++++++
>>  test/py/multiplexed_log.py  | 30 ++++++++++++++++++++---
>>  3 files changed, 74 insertions(+), 24 deletions(-)
>
> Acked-by: Simon Glass <sjg@chromium.org>

Applied to u-boot-dm, thanks!

[U-Boot] [PATCH 7/9] test/py: pass test DTB to sandbox

[Simon Glass]

On 28 January 2016 at 20:47, Simon Glass <sjg@chromium.org> wrote:
> On 27 January 2016 at 23:57, Stephen Warren <swarren@wwwdotorg.org> wrote:
>> From: Stephen Warren <swarren@nvidia.com>
>>
>> This is required for at least "ut dm" to operate correctly.
>>
>> Signed-off-by: Stephen Warren <swarren@nvidia.com>
>> ---
>>  test/py/u_boot_console_sandbox.py | 7 ++++++-
>>  1 file changed, 6 insertions(+), 1 deletion(-)
>
> Acked-by: Simon Glass <sjg@chromium.org>

Applied to u-boot-dm, thanks!

[U-Boot] [PATCH 8/9] test/py: run sandbox in source directory

[Simon Glass]

On 28 January 2016 at 20:47, Simon Glass <sjg@chromium.org> wrote:
> On 27 January 2016 at 23:57, Stephen Warren <swarren@wwwdotorg.org> wrote:
>> From: Stephen Warren <swarren@nvidia.com>
>>
>> Some unit tests expect the cwd of the sandbox process to be the root
>> of the source tree. Ensure that requirement is met.
>>
>> Signed-off-by: Stephen Warren <swarren@nvidia.com>
>> ---
>>  test/py/u_boot_console_sandbox.py | 2 +-
>>  test/py/u_boot_spawn.py           | 8 ++++++--
>>  2 files changed, 7 insertions(+), 3 deletions(-)
>
> Acked-by: Simon Glass <sjg@chromium.org>

Applied to u-boot-dm, thanks!