[U-Boot] [PATCH 0/6] test.py fixes & enhancements

[U-Boot] [PATCH 0/6] test.py fixes & enhancements

[Stephen Warren]

This builds on my previous patch "Implement pytest-based test
infrastructure". It converts a few more tests so provides a few more
examples.

Stephen Warren (6):
  test/py: require .config to exist
  test/py: parse include/autoconf.mk for config options
  test/py: limit outstanding TX data
  test/py: print implicit shell prompt later
  test/py: replace cmd_repeat.sh with Python
  test/py: convert most of test/command_ut.c to Python

 test/cmd_repeat.sh                   |  29 --------
 test/command_ut.c                    | 133 ----------------------------------
 test/py/conftest.py                  |  18 +++--
 test/py/test_env.py                  |  26 ++++++-
 test/py/test_hush_if_test.py         | 137 +++++++++++++++++++++++++++++++++++
 test/py/test_md.py                   |  10 +++
 test/py/test_shell_basics.py         |  31 ++++++++
 test/py/uboot_console_base.py        |  31 ++++----
 test/py/uboot_console_exec_attach.py |   6 +-
 test/py/uboot_console_sandbox.py     |   2 +-
 10 files changed, 233 insertions(+), 190 deletions(-)
 delete mode 100755 test/cmd_repeat.sh
 create mode 100644 test/py/test_hush_if_test.py
 create mode 100644 test/py/test_shell_basics.py

-- 
1.9.1

[U-Boot] [PATCH 1/6] test/py: require .config to exist

[Stephen Warren]

From: Stephen Warren <swarren@nvidia.com>

If .config doesn't exist, the test will currently throw an exception
because the shell prompt value can't be looked up. It's not obvious how
to resolve this. Fix the code to require that the .config file exist in
the build directory rather than ignoring the issue when it's missing.
Make the error message reference the fact that U-Boot doesn't appear to
be built, and mention how to solve that.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
---
 test/py/conftest.py | 19 ++++++++++---------
 1 file changed, 10 insertions(+), 9 deletions(-)

diff --git a/test/py/conftest.py b/test/py/conftest.py
index 4b40bdd89a60..da8618ba998c 100644
--- a/test/py/conftest.py
+++ b/test/py/conftest.py
@@ -99,15 +99,16 @@ def pytest_configure(config):
             ubconfig.__dict__[k] = v
 
     dot_config = build_dir + "/.config"
-    if os.path.exists(dot_config):
-        with open(dot_config, "rt") as f:
-            ini_str = "[root]\n" + f.read()
-            ini_sio = StringIO.StringIO(ini_str)
-            parser = ConfigParser.RawConfigParser()
-            parser.readfp(ini_sio)
-            ubconfig.buildconfig = dict(parser.items("root"))
-    else:
-        ubconfig.buildconfig = dict()
+    if not os.path.exists(dot_config):
+        raise Exception(".config file does not exist; " +
+            "try passing --build option?")
+
+    with open(dot_config, "rt") as f:
+        ini_str = "[root]\n" + f.read()
+        ini_sio = StringIO.StringIO(ini_str)
+        parser = ConfigParser.RawConfigParser()
+        parser.readfp(ini_sio)
+        ubconfig.buildconfig = dict(parser.items("root"))
 
     ubconfig.test_py_dir = test_py_dir
     ubconfig.source_dir = source_dir
-- 
1.9.1

[U-Boot] [PATCH 2/6] test/py: parse include/autoconf.mk for config options

[Stephen Warren]

From: Stephen Warren <swarren@nvidia.com>

Many config options aren't yet converted to Kconfig. Update the test
scripts to parse include/autoconf.mk to pick up the non-converted
options. This will allow tests to be marked as depending on those
options.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
---
 test/py/conftest.py | 25 ++++++++++++++-----------
 1 file changed, 14 insertions(+), 11 deletions(-)

diff --git a/test/py/conftest.py b/test/py/conftest.py
index da8618ba998c..27f6f4d69c15 100644
--- a/test/py/conftest.py
+++ b/test/py/conftest.py
@@ -98,17 +98,20 @@ def pytest_configure(config):
                 continue
             ubconfig.__dict__[k] = v
 
-    dot_config = build_dir + "/.config"
-    if not os.path.exists(dot_config):
-        raise Exception(".config file does not exist; " +
-            "try passing --build option?")
-
-    with open(dot_config, "rt") as f:
-        ini_str = "[root]\n" + f.read()
-        ini_sio = StringIO.StringIO(ini_str)
-        parser = ConfigParser.RawConfigParser()
-        parser.readfp(ini_sio)
-        ubconfig.buildconfig = dict(parser.items("root"))
+    ubconfig.buildconfig = dict()
+
+    for conf_file in (".config", "include/autoconf.mk"):
+        dot_config = build_dir + "/" + conf_file
+        if not os.path.exists(dot_config):
+            raise Exception(conf_file + " does not exist; " +
+                "try passing --build option?")
+
+        with open(dot_config, "rt") as f:
+            ini_str = "[root]\n" + f.read()
+            ini_sio = StringIO.StringIO(ini_str)
+            parser = ConfigParser.RawConfigParser()
+            parser.readfp(ini_sio)
+            ubconfig.buildconfig.update(parser.items("root"))
 
     ubconfig.test_py_dir = test_py_dir
     ubconfig.source_dir = source_dir
-- 
1.9.1

[U-Boot] [PATCH 3/6] test/py: limit outstanding TX data

[Stephen Warren]

U-Boot echoes back all characters it receives on the console. At best,
with zero overhead, it could keep up with the incoming data stream.
However, in practice there's always some delay between receiving a
character from the RX side of the console and handing it off to the TX
side of the console, which means the TX patch cannot keep up with the RX
path. This is particularly true due to U-Boot's unbuffered polling IO
model. This is exacerbated when the target performs slow operations for
each character, such as drawing on the LCD.

When sending large command-lines, larger than the UART HW's RX FIFO size,
this issue can lead to dropped characters, and hence test failures.

To work around this, modify the test scripts to that they limit the number
of outstanding characters, i.e. those sent to the target but not yet
echo'd back. This avoids

Signed-off-by: Stephen Warren <swarren@wwwdotorg.org>
---
 test/py/uboot_console_base.py        | 27 ++++++++++++---------------
 test/py/uboot_console_exec_attach.py |  6 +++++-
 test/py/uboot_console_sandbox.py     |  2 +-
 3 files changed, 18 insertions(+), 17 deletions(-)

diff --git a/test/py/uboot_console_base.py b/test/py/uboot_console_base.py
index dfd986860e75..4bf9e8303633 100644
--- a/test/py/uboot_console_base.py
+++ b/test/py/uboot_console_base.py
@@ -21,9 +21,10 @@ class ConsoleDisableCheck(object):
         self.console.disable_check_count[self.check_type] -= 1
 
 class ConsoleBase(object):
-    def __init__(self, log, config):
+    def __init__(self, log, config, max_fifo_fill):
         self.log = log
         self.config = config
+        self.max_fifo_fill = max_fifo_fill
 
         self.logstream = self.log.get_stream("console", sys.stdout)
 
@@ -63,22 +64,19 @@ class ConsoleBase(object):
             bad_patterns.append(pattern_error_notification)
             bad_pattern_ids.append("Error notification")
         try:
-            if cmd:
-                self.p.send(cmd)
-                try:
-                    m = self.p.expect([re.escape(cmd)] + bad_patterns)
-                    if m != 0:
-                        self.at_prompt = False
-                        raise Exception("Bad pattern found on console: " +
-                                        bad_pattern_ids[m - 1])
-                except Exception as ex:
+            self.at_prompt = False
+            while cmd:
+                # Limit max outstanding data, so UART FIFOs don't overflow
+                chunk = cmd[:self.max_fifo_fill]
+                cmd = cmd[self.max_fifo_fill:]
+                self.p.send(chunk)
+                m = self.p.expect([re.escape(chunk)] + bad_patterns)
+                if m != 0:
                     self.at_prompt = False
-                    print cmd
-                    self.logstream.write(cmd, implicit=True)
-                    raise
+                    raise Exception("Bad pattern found on console: " +
+                                    bad_pattern_ids[m - 1])
             self.p.send("\n")
             if not wait_for_prompt:
-                self.at_prompt = False
                 return
             m = self.p.expect([self.prompt_escaped] + bad_patterns)
             if m != 0:
@@ -88,7 +86,6 @@ class ConsoleBase(object):
             self.at_prompt = True
             return self.p.before.strip()
         except Exception as ex:
-            self.at_prompt = False
             self.log.error(str(ex))
             self.cleanup_spawn()
             raise
diff --git a/test/py/uboot_console_exec_attach.py b/test/py/uboot_console_exec_attach.py
index 7960d66107c3..0beaa4fe7102 100644
--- a/test/py/uboot_console_exec_attach.py
+++ b/test/py/uboot_console_exec_attach.py
@@ -7,7 +7,11 @@ def cmdline(app, args):
 
 class ConsoleExecAttach(ConsoleBase):
     def __init__(self, log, config):
-        super(ConsoleExecAttach, self).__init__(log, config)
+        # The max_fifo_fill value might need tweaking per-board/-SoC?
+        # 1 would be safe anywhere, but is very slow (a pexpect issue?).
+        # 16 is a common FIFO size.
+        # HW flow control would mean this could be infinite.
+        super(ConsoleExecAttach, self).__init__(log, config, max_fifo_fill=16)
 
         self.log.action("Flashing U-Boot")
         cmd = ["uboot-test-flash", config.board_type, config.board_identity]
diff --git a/test/py/uboot_console_sandbox.py b/test/py/uboot_console_sandbox.py
index c3aae3862ca9..ed3ffa5b90b1 100644
--- a/test/py/uboot_console_sandbox.py
+++ b/test/py/uboot_console_sandbox.py
@@ -4,7 +4,7 @@ from uboot_console_base import ConsoleBase
 
 class ConsoleSandbox(ConsoleBase):
     def __init__(self, log, config):
-        super(ConsoleSandbox, self).__init__(log, config)
+        super(ConsoleSandbox, self).__init__(log, config, max_fifo_fill=1024)
 
     def get_spawn(self):
         return pexpect.spawn(self.config.build_dir + "/u-boot")
-- 
1.9.1

[U-Boot] [PATCH 4/6] test/py: print implicit shell prompt later

[Stephen Warren]

Whenever a test is about to start, the test scripts log an "implicit"
version of the shell prompt. This ensures that each test's log section
includes the prompt, which makes it easier to read.

However, pytest apparently doesn't set up stdout capturing for skipped
tests. If we print this implicit prompt before we've determined whether
the test is skipped, then it may/will appear as stdout in ther terminal
where the test script was run, which looks messy.

To avoid this, only print this prompt after we've evaluated whether the
test is to be skipped. Note that internally, pytest.skip() raises an
exception, which causes the moved code not to execute, now that it's
later in the execution path.

Signed-off-by: Stephen Warren <swarren@wwwdotorg.org>
---
 test/py/conftest.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/test/py/conftest.py b/test/py/conftest.py
index 27f6f4d69c15..7d11278b5f65 100644
--- a/test/py/conftest.py
+++ b/test/py/conftest.py
@@ -184,11 +184,11 @@ def setup_envspec(item):
 
 def pytest_runtest_setup(item):
     log.start_section(item.name)
-    if console.at_prompt:
-        console.logstream.write(console.prompt, implicit=True)
     setup_boardspec(item)
     setup_buildconfigspec(item)
     setup_envspec(item)
+    if console.at_prompt:
+        console.logstream.write(console.prompt, implicit=True)
 
 def pytest_runtest_protocol(item, nextitem):
     reports = runtestprotocol(item, nextitem=nextitem)
-- 
1.9.1

[U-Boot] [PATCH 5/6] test/py: replace cmd_repeat.sh with Python

[Stephen Warren]

From: Stephen Warren <swarren@nvidia.com>

This moves this test into the new infra-structure, and also allows it to
work on real hardware, not just sandbox.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
---
 test/cmd_repeat.sh | 29 -----------------------------
 test/py/test_md.py | 10 ++++++++++
 2 files changed, 10 insertions(+), 29 deletions(-)
 delete mode 100755 test/cmd_repeat.sh

diff --git a/test/cmd_repeat.sh b/test/cmd_repeat.sh
deleted file mode 100755
index 990e79900f47..000000000000
--- a/test/cmd_repeat.sh
+++ /dev/null
@@ -1,29 +0,0 @@
-#!/bin/sh
-
-# Test for U-Boot cli including command repeat
-
-BASE="$(dirname $0)"
-. $BASE/common.sh
-
-run_test() {
-	./${OUTPUT_DIR}/u-boot <<END
-setenv ctrlc_ignore y
-md 0
-
-reset
-END
-}
-check_results() {
-	echo "Check results"
-
-	grep -q 00000100 ${tmp} || fail "Command did not repeat"
-}
-
-echo "Test CLI repeat"
-echo
-tmp="$(tempfile)"
-build_uboot
-run_test >${tmp}
-check_results ${tmp}
-rm ${tmp}
-echo "Test passed"
diff --git a/test/py/test_md.py b/test/py/test_md.py
index 49cdd2685234..9667fd4f2217 100644
--- a/test/py/test_md.py
+++ b/test/py/test_md.py
@@ -10,3 +10,13 @@ def test_md(uboot_console):
     uboot_console.run_command("mw " + addr + " " + val)
     response = uboot_console.run_command("md " + addr + " 10")
     assert(expected_response in response)
+
+ at pytest.mark.buildconfigspec("cmd_memory")
+def test_md_repeat(uboot_console):
+    addr_base = "%08x" % uboot_console.config.ram_base
+    words = 0x10
+    addr_repeat = "%08x" % (uboot_console.config.ram_base + (words * 4))
+    uboot_console.run_command("md %s %x" % (addr_base, words))
+    response = uboot_console.run_command("")
+    expected_response = addr_repeat + ": "
+    assert(expected_response in response)
-- 
1.9.1

[U-Boot] [PATCH 6/6] test/py: convert most of test/command_ut.c to Python

[Stephen Warren]

From: Stephen Warren <swarren@nvidia.com>

Most of command_ut.c simply runs shell commands which could just as
easily be invoked from the command-line. Convert all of that testing to
Python. I'd classify all of these tests as "system" tests since they use
the standard interface to the U-Boot "system".

The remaining tests rely on features only available to C code, such as
compound shell commands using \n as a separator, and validating the
return value of functions that invoke shell commands directly. I'd
classify these as unit tests, since they rely on mechanisms only
available internally.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
---
I wonder how useful do_ut_cmd() is now. It does test some things that
can't be tested from the command-line, but...
---
 test/command_ut.c             | 133 ----------------------------------------
 test/py/test_env.py           |  26 +++++++-
 test/py/test_hush_if_test.py  | 137 ++++++++++++++++++++++++++++++++++++++++++
 test/py/test_shell_basics.py  |  31 ++++++++++
 test/py/uboot_console_base.py |   4 +-
 5 files changed, 194 insertions(+), 137 deletions(-)
 create mode 100644 test/py/test_hush_if_test.py
 create mode 100644 test/py/test_shell_basics.py

diff --git a/test/command_ut.c b/test/command_ut.c
index 926573a39543..35bd35ae2e30 100644
--- a/test/command_ut.c
+++ b/test/command_ut.c
@@ -20,21 +20,6 @@ static int do_ut_cmd(cmd_tbl_t *cmdtp, int flag, int argc, char * const argv[])
 	printf("%s: Testing commands\n", __func__);
 	run_command("env default -f -a", 0);
 
-	/* run a single command */
-	run_command("setenv single 1", 0);
-	assert(!strcmp("1", getenv("single")));
-
-	/* make sure that compound statements work */
-#ifdef CONFIG_SYS_HUSH_PARSER
-	run_command("if test -n ${single} ; then setenv check 1; fi", 0);
-	assert(!strcmp("1", getenv("check")));
-	run_command("setenv check", 0);
-#endif
-
-	/* commands separated by ; */
-	run_command_list("setenv list 1; setenv list ${list}1", -1, 0);
-	assert(!strcmp("11", getenv("list")));
-
 	/* commands separated by \n */
 	run_command_list("setenv list 1\n setenv list ${list}1", -1, 0);
 	assert(!strcmp("11", getenv("list")));
@@ -43,11 +28,6 @@ static int do_ut_cmd(cmd_tbl_t *cmdtp, int flag, int argc, char * const argv[])
 	run_command_list("setenv list 1${list}\n", -1, 0);
 	assert(!strcmp("111", getenv("list")));
 
-	/* three commands in a row */
-	run_command_list("setenv list 1\n setenv list ${list}2; "
-		"setenv list ${list}3", -1, 0);
-	assert(!strcmp("123", getenv("list")));
-
 	/* a command string with \0 in it. Stuff after \0 should be ignored */
 	run_command("setenv list", 0);
 	run_command_list(test_cmd, sizeof(test_cmd), 0);
@@ -66,13 +46,6 @@ static int do_ut_cmd(cmd_tbl_t *cmdtp, int flag, int argc, char * const argv[])
 	assert(run_command_list("false", -1, 0) == 1);
 	assert(run_command_list("echo", -1, 0) == 0);
 
-	run_command("setenv foo 'setenv monty 1; setenv python 2'", 0);
-	run_command("run foo", 0);
-	assert(getenv("monty") != NULL);
-	assert(!strcmp("1", getenv("monty")));
-	assert(getenv("python") != NULL);
-	assert(!strcmp("2", getenv("python")));
-
 #ifdef CONFIG_SYS_HUSH_PARSER
 	run_command("setenv foo 'setenv black 1\nsetenv adder 2'", 0);
 	run_command("run foo", 0);
@@ -80,112 +53,6 @@ static int do_ut_cmd(cmd_tbl_t *cmdtp, int flag, int argc, char * const argv[])
 	assert(!strcmp("1", getenv("black")));
 	assert(getenv("adder") != NULL);
 	assert(!strcmp("2", getenv("adder")));
-
-	/* Test the 'test' command */
-
-#define HUSH_TEST(name, expr, expected_result) \
-	run_command("if test " expr " ; then " \
-			"setenv " #name "_" #expected_result " y; else " \
-			"setenv " #name "_" #expected_result " n; fi", 0); \
-	assert(!strcmp(#expected_result, getenv(#name "_" #expected_result))); \
-	setenv(#name "_" #expected_result, NULL);
-
-	/* Basic operators */
-	HUSH_TEST(streq, "aaa = aaa", y);
-	HUSH_TEST(streq, "aaa = bbb", n);
-
-	HUSH_TEST(strneq, "aaa != bbb", y);
-	HUSH_TEST(strneq, "aaa != aaa", n);
-
-	HUSH_TEST(strlt, "aaa < bbb", y);
-	HUSH_TEST(strlt, "bbb < aaa", n);
-
-	HUSH_TEST(strgt, "bbb > aaa", y);
-	HUSH_TEST(strgt, "aaa > bbb", n);
-
-	HUSH_TEST(eq, "123 -eq 123", y);
-	HUSH_TEST(eq, "123 -eq 456", n);
-
-	HUSH_TEST(ne, "123 -ne 456", y);
-	HUSH_TEST(ne, "123 -ne 123", n);
-
-	HUSH_TEST(lt, "123 -lt 456", y);
-	HUSH_TEST(lt_eq, "123 -lt 123", n);
-	HUSH_TEST(lt, "456 -lt 123", n);
-
-	HUSH_TEST(le, "123 -le 456", y);
-	HUSH_TEST(le_eq, "123 -le 123", y);
-	HUSH_TEST(le, "456 -le 123", n);
-
-	HUSH_TEST(gt, "456 -gt 123", y);
-	HUSH_TEST(gt_eq, "123 -gt 123", n);
-	HUSH_TEST(gt, "123 -gt 456", n);
-
-	HUSH_TEST(ge, "456 -ge 123", y);
-	HUSH_TEST(ge_eq, "123 -ge 123", y);
-	HUSH_TEST(ge, "123 -ge 456", n);
-
-	HUSH_TEST(z, "-z \"\"", y);
-	HUSH_TEST(z, "-z \"aaa\"", n);
-
-	HUSH_TEST(n, "-n \"aaa\"", y);
-	HUSH_TEST(n, "-n \"\"", n);
-
-	/* Inversion of simple tests */
-	HUSH_TEST(streq_inv, "! aaa = aaa", n);
-	HUSH_TEST(streq_inv, "! aaa = bbb", y);
-
-	HUSH_TEST(streq_inv_inv, "! ! aaa = aaa", y);
-	HUSH_TEST(streq_inv_inv, "! ! aaa = bbb", n);
-
-	/* Binary operators */
-	HUSH_TEST(or_0_0, "aaa != aaa -o bbb != bbb", n);
-	HUSH_TEST(or_0_1, "aaa != aaa -o bbb = bbb", y);
-	HUSH_TEST(or_1_0, "aaa = aaa -o bbb != bbb", y);
-	HUSH_TEST(or_1_1, "aaa = aaa -o bbb = bbb", y);
-
-	HUSH_TEST(and_0_0, "aaa != aaa -a bbb != bbb", n);
-	HUSH_TEST(and_0_1, "aaa != aaa -a bbb = bbb", n);
-	HUSH_TEST(and_1_0, "aaa = aaa -a bbb != bbb", n);
-	HUSH_TEST(and_1_1, "aaa = aaa -a bbb = bbb", y);
-
-	/* Inversion within binary operators */
-	HUSH_TEST(or_0_0_inv, "! aaa != aaa -o ! bbb != bbb", y);
-	HUSH_TEST(or_0_1_inv, "! aaa != aaa -o ! bbb = bbb", y);
-	HUSH_TEST(or_1_0_inv, "! aaa = aaa -o ! bbb != bbb", y);
-	HUSH_TEST(or_1_1_inv, "! aaa = aaa -o ! bbb = bbb", n);
-
-	HUSH_TEST(or_0_0_inv_inv, "! ! aaa != aaa -o ! ! bbb != bbb", n);
-	HUSH_TEST(or_0_1_inv_inv, "! ! aaa != aaa -o ! ! bbb = bbb", y);
-	HUSH_TEST(or_1_0_inv_inv, "! ! aaa = aaa -o ! ! bbb != bbb", y);
-	HUSH_TEST(or_1_1_inv_inv, "! ! aaa = aaa -o ! ! bbb = bbb", y);
-
-	setenv("ut_var_nonexistent", NULL);
-	setenv("ut_var_exists", "1");
-	HUSH_TEST(z_varexp_quoted, "-z \"$ut_var_nonexistent\"", y);
-	HUSH_TEST(z_varexp_quoted, "-z \"$ut_var_exists\"", n);
-	setenv("ut_var_exists", NULL);
-
-	run_command("setenv ut_var_space \" \"", 0);
-	assert(!strcmp(getenv("ut_var_space"), " "));
-	run_command("setenv ut_var_test $ut_var_space", 0);
-	assert(!getenv("ut_var_test"));
-	run_command("setenv ut_var_test \"$ut_var_space\"", 0);
-	assert(!strcmp(getenv("ut_var_test"), " "));
-	run_command("setenv ut_var_test \" 1${ut_var_space}${ut_var_space} 2 \"", 0);
-	assert(!strcmp(getenv("ut_var_test"), " 1   2 "));
-	setenv("ut_var_space", NULL);
-	setenv("ut_var_test", NULL);
-
-#ifdef CONFIG_SANDBOX
-	/* File existence */
-	HUSH_TEST(e, "-e hostfs - creating_this_file_breaks_uboot_unit_test", n);
-	run_command("sb save hostfs - creating_this_file_breaks_uboot_unit_test 0 1", 0);
-	HUSH_TEST(e, "-e hostfs - creating_this_file_breaks_uboot_unit_test", y);
-	/* Perhaps this could be replaced by an "rm" shell command one day */
-	assert(!os_unlink("creating_this_file_breaks_uboot_unit_test"));
-	HUSH_TEST(e, "-e hostfs - creating_this_file_breaks_uboot_unit_test", n);
-#endif
 #endif
 
 	assert(run_command("", 0) == 0);
diff --git a/test/py/test_env.py b/test/py/test_env.py
index 16891cd6bb15..38c305f98250 100644
--- a/test/py/test_env.py
+++ b/test/py/test_env.py
@@ -39,7 +39,7 @@ def unset_var(state_test_env, var):
         del state_test_env.env[var]
 
 def set_var(state_test_env, var, value):
-    state_test_env.uboot_console.run_command("setenv " + var + " " + value)
+    state_test_env.uboot_console.run_command("setenv " + var + " \"" + value + "\"")
     state_test_env.env[var] = value
 
 def validate_empty(state_test_env, var):
@@ -47,8 +47,10 @@ def validate_empty(state_test_env, var):
     assert response == ""
 
 def validate_set(state_test_env, var, value):
-    response = state_test_env.uboot_console.run_command("echo $" + var)
-    assert response == value
+    # echo does not preserve leading, internal, or trailing whitespace in the
+    # value. printenv does, and hence allows more complete testing.
+    response = state_test_env.uboot_console.run_command("printenv " + var)
+    assert response == (var + "=" + value)
 
 def test_env_echo_exists(state_test_env):
     """Echo a variable that exists"""
@@ -94,3 +96,21 @@ def test_env_unset_existing(state_test_env):
     var = state_test_env.set_var
     unset_var(state_test_env, var)
     validate_empty(state_test_env, var)
+
+def test_env_expansion_spaces(state_test_env):
+    var_space = None
+    var_test = None
+    try:
+        var_space = state_test_env.get_non_existent_var()
+        set_var(state_test_env, var_space, " ")
+
+        var_test = state_test_env.get_non_existent_var()
+        value = " 1${%(var_space)s}${%(var_space)s} 2 " % locals()
+        set_var(state_test_env, var_test, value)
+        value = " 1   2 "
+        validate_set(state_test_env, var_test, value)
+    finally:
+        if var_space:
+            unset_var(state_test_env, var_space)
+        if var_test:
+            unset_var(state_test_env, var_test)
diff --git a/test/py/test_hush_if_test.py b/test/py/test_hush_if_test.py
new file mode 100644
index 000000000000..3504922dcf12
--- /dev/null
+++ b/test/py/test_hush_if_test.py
@@ -0,0 +1,137 @@
+import os
+import os.path
+import pytest
+
+subtests = (
+    # Base if functionality
+
+    ("true", True),
+    ("false", False),
+
+    # Basic operators
+
+    ("test aaa = aaa", True),
+    ("test aaa = bbb", False),
+
+    ("test aaa != bbb", True),
+    ("test aaa != aaa", False),
+
+    ("test aaa < bbb", True),
+    ("test bbb < aaa", False),
+
+    ("test bbb > aaa", True),
+    ("test aaa > bbb", False),
+
+    ("test 123 -eq 123", True),
+    ("test 123 -eq 456", False),
+
+    ("test 123 -ne 456", True),
+    ("test 123 -ne 123", False),
+
+    ("test 123 -lt 456", True),
+    ("test 123 -lt 123", False),
+    ("test 456 -lt 123", False),
+
+    ("test 123 -le 456", True),
+    ("test 123 -le 123", True),
+    ("test 456 -le 123", False),
+
+    ("test 456 -gt 123", True),
+    ("test 123 -gt 123", False),
+    ("test 123 -gt 456", False),
+
+    ("test 456 -ge 123", True),
+    ("test 123 -ge 123", True),
+    ("test 123 -ge 456", False),
+
+    ("test -z \"\"", True),
+    ("test -z \"aaa\"", False),
+
+    ("test -n \"aaa\"", True),
+    ("test -n \"\"", False),
+
+    # Inversion of simple tests
+
+    ("test ! aaa = aaa", False),
+    ("test ! aaa = bbb", True),
+    ("test ! ! aaa = aaa", True),
+    ("test ! ! aaa = bbb", False),
+
+    # Binary operators
+
+    ("test aaa != aaa -o bbb != bbb", False),
+    ("test aaa != aaa -o bbb = bbb", True),
+    ("test aaa = aaa -o bbb != bbb", True),
+    ("test aaa = aaa -o bbb = bbb", True),
+
+    ("test aaa != aaa -a bbb != bbb", False),
+    ("test aaa != aaa -a bbb = bbb", False),
+    ("test aaa = aaa -a bbb != bbb", False),
+    ("test aaa = aaa -a bbb = bbb", True),
+
+    # Inversion within binary operators
+
+    ("test ! aaa != aaa -o ! bbb != bbb", True),
+    ("test ! aaa != aaa -o ! bbb = bbb", True),
+    ("test ! aaa = aaa -o ! bbb != bbb", True),
+    ("test ! aaa = aaa -o ! bbb = bbb", False),
+
+    ("test ! ! aaa != aaa -o ! ! bbb != bbb", False),
+    ("test ! ! aaa != aaa -o ! ! bbb = bbb", True),
+    ("test ! ! aaa = aaa -o ! ! bbb != bbb", True),
+    ("test ! ! aaa = aaa -o ! ! bbb = bbb", True),
+
+    # -z operator
+
+    ("test -z \"$ut_var_nonexistent\"", True),
+    ("test -z \"$ut_var_exists\"", False),
+)
+
+def exec_hush_if(uboot_console, expr, result):
+    cmd = "if " + expr + "; then echo true; else echo false; fi"
+    response = uboot_console.run_command(cmd)
+    assert response.strip() == str(result).lower()
+
+ at pytest.mark.buildconfigspec("sys_hush_parser")
+def test_hush_if_test_setup(uboot_console):
+    uboot_console.run_command("setenv ut_var_nonexistent")
+    uboot_console.run_command("setenv ut_var_exists 1")
+
+ at pytest.mark.buildconfigspec("sys_hush_parser")
+ at pytest.mark.parametrize("expr,result", subtests)
+def test_hush_if_test(uboot_console, expr, result):
+    exec_hush_if(uboot_console, expr, result)
+
+ at pytest.mark.buildconfigspec("sys_hush_parser")
+def test_hush_if_test_teardown(uboot_console):
+    uboot_console.run_command("setenv ut_var_exists")
+
+ at pytest.mark.buildconfigspec("sys_hush_parser")
+# We might test this on real filesystems via UMS, DFU, "save", etc.
+# Of those, only UMS currently allows file removal though.
+ at pytest.mark.boardspec("sandbox")
+def test_hush_if_test_host_file_exists(uboot_console):
+    test_file = uboot_console.config.result_dir + \
+        "/creating_this_file_breaks_uboot_tests"
+
+    try:
+        os.unlink(test_file)
+    except:
+        pass
+    assert not os.path.exists(test_file)
+
+    expr = "test -e hostfs - " + test_file
+    exec_hush_if(uboot_console, expr, False)
+
+    try:
+        with file(test_file, "wb"):
+            pass
+        assert os.path.exists(test_file)
+
+        expr = "test -e hostfs - " + test_file
+        exec_hush_if(uboot_console, expr, True)
+    finally:
+        os.unlink(test_file)
+
+    expr = "test -e hostfs - " + test_file
+    exec_hush_if(uboot_console, expr, False)
diff --git a/test/py/test_shell_basics.py b/test/py/test_shell_basics.py
new file mode 100644
index 000000000000..5477ea4e32ea
--- /dev/null
+++ b/test/py/test_shell_basics.py
@@ -0,0 +1,31 @@
+def test_shell_execute(uboot_console):
+    """Test any shell command"""
+    response = uboot_console.run_command("echo hello")
+    assert response.strip() == "hello"
+
+def test_shell_semicolon_two(uboot_console):
+    """Test two shell commands separate by a semi-colon"""
+    cmd = "echo hello; echo world"
+    response = uboot_console.run_command(cmd)
+    # This validation method ignores the exact whitespace between the strings
+    assert response.index("hello") < response.index("world")
+
+def test_shell_semicolon_three(uboot_console):
+    """Test three shell commands separate by a semi-colon"""
+    cmd = "setenv list 1; setenv list ${list}2; setenv list ${list}3; " + \
+        "echo ${list}"
+    response = uboot_console.run_command(cmd)
+    assert response.strip() == "123"
+    uboot_console.run_command("setenv list")
+
+def test_shell_run(uboot_console):
+    """Test the 'run' shell command"""
+    uboot_console.run_command("setenv foo 'setenv monty 1; setenv python 2'")
+    uboot_console.run_command("run foo")
+    response = uboot_console.run_command("echo $monty")
+    assert response.strip() == "1"
+    response = uboot_console.run_command("echo $python")
+    assert response.strip() == "2"
+    uboot_console.run_command("setenv foo")
+    uboot_console.run_command("setenv monty")
+    uboot_console.run_command("setenv python")
diff --git a/test/py/uboot_console_base.py b/test/py/uboot_console_base.py
index 4bf9e8303633..5ecddd88e124 100644
--- a/test/py/uboot_console_base.py
+++ b/test/py/uboot_console_base.py
@@ -84,7 +84,9 @@ class ConsoleBase(object):
                 raise Exception("Bad pattern found on console: " +
                                 bad_pattern_ids[m - 1])
             self.at_prompt = True
-            return self.p.before.strip()
+            # Only strip \r\n; space/TAB might be significant if testing
+            # indentation.
+            return self.p.before.strip("\r\n")
         except Exception as ex:
             self.log.error(str(ex))
             self.cleanup_spawn()
-- 
1.9.1

[U-Boot] [PATCH 1/6] test/py: require .config to exist

[Tom Rini]

On Tue, Nov 17, 2015 at 08:19:59PM -0700, Stephen Warren wrote:

> From: Stephen Warren <swarren@nvidia.com>
> 
> If .config doesn't exist, the test will currently throw an exception
> because the shell prompt value can't be looked up. It's not obvious how
> to resolve this. Fix the code to require that the .config file exist in
> the build directory rather than ignoring the issue when it's missing.
> Make the error message reference the fact that U-Boot doesn't appear to
> be built, and mention how to solve that.
> 
> Signed-off-by: Stephen Warren <swarren@nvidia.com>

Reviewed-by: Tom Rini <trini@konsulko.com>

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20151123/d5c00a05/attachment.sig>

[U-Boot] [PATCH 2/6] test/py: parse include/autoconf.mk for config options

[Tom Rini]

On Tue, Nov 17, 2015 at 08:20:00PM -0700, Stephen Warren wrote:

> From: Stephen Warren <swarren@nvidia.com>
> 
> Many config options aren't yet converted to Kconfig. Update the test
> scripts to parse include/autoconf.mk to pick up the non-converted
> options. This will allow tests to be marked as depending on those
> options.
> 
> Signed-off-by: Stephen Warren <swarren@nvidia.com>

Reviewed-by: Tom Rini <trini@konsulko.com>

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20151123/aaa395b9/attachment.sig>

[U-Boot] [PATCH 4/6] test/py: print implicit shell prompt later

[Tom Rini]

On Tue, Nov 17, 2015 at 08:20:02PM -0700, Stephen Warren wrote:

> Whenever a test is about to start, the test scripts log an "implicit"
> version of the shell prompt. This ensures that each test's log section
> includes the prompt, which makes it easier to read.
> 
> However, pytest apparently doesn't set up stdout capturing for skipped
> tests. If we print this implicit prompt before we've determined whether
> the test is skipped, then it may/will appear as stdout in ther terminal
> where the test script was run, which looks messy.
> 
> To avoid this, only print this prompt after we've evaluated whether the
> test is to be skipped. Note that internally, pytest.skip() raises an
> exception, which causes the moved code not to execute, now that it's
> later in the execution path.
> 
> Signed-off-by: Stephen Warren <swarren@wwwdotorg.org>

Reviewed-by: Tom Rini <trini@konsulko.com>

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20151123/5d8cdd0c/attachment.sig>

[U-Boot] [PATCH 5/6] test/py: replace cmd_repeat.sh with Python

[Tom Rini]

On Tue, Nov 17, 2015 at 08:20:03PM -0700, Stephen Warren wrote:

> From: Stephen Warren <swarren@nvidia.com>
> 
> This moves this test into the new infra-structure, and also allows it to
> work on real hardware, not just sandbox.
> 
> Signed-off-by: Stephen Warren <swarren@nvidia.com>

Reviewed-by: Tom Rini <trini@konsulko.com>

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20151123/c3e613fc/attachment.sig>