[U-Boot] [PATCH] Implement pytest-based test infrastructure

* [U-Boot] [PATCH] Implement pytest-based test infrastructure
@ 2015-11-15  6:53 Stephen Warren
  2015-11-19 14:45 ` Simon Glass
  0 siblings, 1 reply; 19+ messages in thread
From: Stephen Warren @ 2015-11-15  6:53 UTC (permalink / raw)
  To: u-boot

This tool aims to test U-Boot by executing U-Boot shell commands using the
console interface. A single top-level script exists to execute or attach
to the U-Boot console, run the entire script of tests against it, and
summarize the results. Advantages of this approach are:

- Testing is performed in the same way a user or script would interact
  with U-Boot; there can be no disconnect.
- There is no need to write or embed test-related code into U-Boot itself.
  It is asserted that writing test-related code in Python is simpler and
  more flexible that writing it all in C.
- It is reasonably simple to interact with U-Boot in this way.

A few simple tests are provided as examples. Soon, we should convert as
many as possible of the other tests in test/* and test/cmd_ut.c too.

In the future, I hope to publish (out-of-tree) the hook scripts, relay
control utilities, and udev rules I will use for my own HW setup.

See README.md for more details!

Signed-off-by: Stephen Warren <swarren@wwwdotorg.org>
---
 .gitignore                           |   1 +
 test/py/README.md                    | 287 +++++++++++++++++++++++++++++++++++
 test/py/board_jetson_tk1.py          |   1 +
 test/py/board_sandbox.py             |   1 +
 test/py/board_seaboard.py            |   1 +
 test/py/conftest.py                  | 225 +++++++++++++++++++++++++++
 test/py/multiplexed_log.css          |  70 +++++++++
 test/py/multiplexed_log.py           | 172 +++++++++++++++++++++
 test/py/pytest.ini                   |   5 +
 test/py/soc_tegra124.py              |   1 +
 test/py/soc_tegra20.py               |   1 +
 test/py/test.py                      |  12 ++
 test/py/test_000_version.py          |   9 ++
 test/py/test_env.py                  |  96 ++++++++++++
 test/py/test_help.py                 |   2 +
 test/py/test_md.py                   |  12 ++
 test/py/test_sandbox_exit.py         |  15 ++
 test/py/test_unknown_cmd.py          |   4 +
 test/py/uboot_console_base.py        | 143 +++++++++++++++++
 test/py/uboot_console_exec_attach.py |  28 ++++
 test/py/uboot_console_sandbox.py     |  22 +++
 21 files changed, 1108 insertions(+)
 create mode 100644 test/py/README.md
 create mode 100644 test/py/board_jetson_tk1.py
 create mode 100644 test/py/board_sandbox.py
 create mode 100644 test/py/board_seaboard.py
 create mode 100644 test/py/conftest.py
 create mode 100644 test/py/multiplexed_log.css
 create mode 100644 test/py/multiplexed_log.py
 create mode 100644 test/py/pytest.ini
 create mode 100644 test/py/soc_tegra124.py
 create mode 100644 test/py/soc_tegra20.py
 create mode 100755 test/py/test.py
 create mode 100644 test/py/test_000_version.py
 create mode 100644 test/py/test_env.py
 create mode 100644 test/py/test_help.py
 create mode 100644 test/py/test_md.py
 create mode 100644 test/py/test_sandbox_exit.py
 create mode 100644 test/py/test_unknown_cmd.py
 create mode 100644 test/py/uboot_console_base.py
 create mode 100644 test/py/uboot_console_exec_attach.py
 create mode 100644 test/py/uboot_console_sandbox.py

diff --git a/.gitignore b/.gitignore
index 33abbd3d0783..b276b3a160bb 100644
--- a/.gitignore
+++ b/.gitignore
@@ -20,6 +20,7 @@
 *.bin
 *.patch
 *.cfgtmp
+*.pyc
 
 # host programs on Cygwin
 *.exe
diff --git a/test/py/README.md b/test/py/README.md
new file mode 100644
index 000000000000..70104d2f3b5e
--- /dev/null
+++ b/test/py/README.md
@@ -0,0 +1,287 @@
+# U-Boot pytest suite
+
+## Introduction
+
+This tool aims to test U-Boot by executing U-Boot shell commands using the
+console interface. A single top-level script exists to execute or attach to the
+U-Boot console, run the entire script of tests against it, and summarize the
+results. Advantages of this approach are:
+
+- Testing is performed in the same way a user or script would interact with
+  U-Boot; there can be no disconnect.
+- There is no need to write or embed test-related code into U-Boot itself.
+  It is asserted that writing test-related code in Python is simpler and more
+  flexible that writing it all in C.
+- It is reasonably simple to interact with U-Boot in this way.
+
+## Requirements
+
+The test suite is implemented using pytest. Interaction with the U-Boot
+console uses pexpect. Interaction with real hardware uses the tools of your
+choice; you get to implement various "hook" scripts that are called by the
+test suite at the appropriate time.
+
+On Debian or Debian-like distributions, the following packages are required.
+Similar package names should exist in other distributions.
+
+| Package        | Version tested (Ubuntu 14.04) |
+| -------------- | ----------------------------- |
+| python         | 2.7.5-5ubuntu3                |
+| python-pytest  | 2.5.1-1                       |
+| python-pexpect | 3.1-1ubuntu0.1                |
+
+The test script supports either:
+
+- Executing a sandbox port of U-Boot on the local machine as a sub-process,
+  and interacting with it over stdin/stdout.
+- Executing external "hook" scripts to flash a U-Boot binary onto a physical
+  board, attach to the board's console stream, and reset the board. Further
+  details are described later.
+
+### Using `virtualenv` to provide requirements
+
+Older distributions (e.g. Ubuntu 10.04) may not provide all the required
+packages, or may provide versions that are too old to run the test suite. One
+can use the Python `virtualenv` script to locally install more up-to-date
+versions of the required packages without interfering with the OS installation.
+For example:
+
+```bash
+$ cd /path/to/u-boot
+$ sudo apt-get install python python-virtualenv pexpect
+$ virtualenv venv
+$ . ./venv/bin/activate
+$ pip install pytest
+```
+
+## Testing sandbox
+
+To run the testsuite on the sandbox port (U-Boot built as a native user-space
+application), simply execute:
+
+```
+./test/py/test.py --bd sandbox --build
+```
+
+The `--bd` option tells the test suite which board type is being tested. This
+lets the test suite know which features the board has, and hence exactly what
+can be tested.
+
+The `--build` option tells U-Boot to compile U-Boot. Alternatively, you may
+omit this option and build U-Boot yourself, in whatever way you choose, before
+running the test script.
+
+The test script will attach to U-Boot, execute all valid tests for the board,
+then print a summary of the test process. A complete log of the test session
+will be written to `${build_dir}/test-log.html`. This is best viewed in a web
+browser, by may be read directly as plain text, perhaps with the aid of the
+`html2text` utility.
+
+## Command-line options
+
+- `--board-type`, `--bd`, `-B` set the type of the board to be tested. For
+  example, `sandbox` or `seaboard`.
+- `--board-identity`, `--id` set the identity of the board to be tested.
+  This allows differentiation between multiple instances of the same type of
+  physical board that are attached to the same host machine. This parameter is
+  not interpreted by the test script in any way, but rather is simply passed
+  to the hook scripts described below, and may be used in any site-specific
+  way deemed necessary.
+- `--build` indicates that the test script should compile U-Boot itself
+  before running the tests. If using this option, make sure that any
+  environment variables required by the build process are already set, such as
+  `$CROSS_COMPILE`.
+- `--build-dir` sets the directory containing the compiled U-Boot binaries.
+  If omitted, this is `${source_dir}/build-${board_type}`.
+- `--result-dir` sets the directory to write results, such as log files,
+  into. If omitted, the build directory is used.
+- `--persistent-data-dir` sets the directory used to store persistent test
+  data. This is test data that may be re-used across test runs, such as file-
+  system images.
+
+`pytest` also implements a number of its own command-line options. Please see
+`pytest` documentation for complete details. Execute `py.test --version` for
+a brief summary. Note that U-Boot's test.py script passes all command-line
+arguments directly to `pytest` for processing.
+
+## Testing real hardware
+
+The tools and techniques used to interact with real hardware will vary
+radically between different host and target systems, and the whims of the user.
+For this reason, the test suite does not attempt to directly interact with real
+hardware in any way. Rather, it expects a standardized set of "hook" scripts to
+exist which implement certain actions on behalf of the test suite. This keeps
+the test suite simple and isolated from system variances unrelated to U-Boot
+features.
+
+### Hook scripts
+
+The test suite requires the following hook scripts to be executable via
+`$PATH`:
+
+#### Environment variables
+
+The following environment variables are set when running hook scripts:
+
+- `UBOOT_BOARD_TYPE` the board type being tested.
+- `UBOOT_BOARD_IDENTITY` the board identity being tested, or `na` if none was
+  specified.
+- `UBOOT_SOURCE_DIR` the U-Boot source directory.
+- `UBOOT_TEST_PY_DIR` the full path to `test/py/` in the source directory.
+- `UBOOT_BUILD_DIR` the U-Boot build directory.
+- `UBOOT_RESULT_DIR` the test result directory.
+- `UBOOT_PERSISTENT_DATA_DIR` the test peristent data directory.
+
+#### `uboot-test-console`
+
+This script provides access to the U-Boot console. The script's stdin/stdout
+should be connected to the board's console. This script should continue to run
+indefinitely, until killed. The test suite will run this script in parallel
+with all other hooks.
+
+This script may be implemented by executing e.g. `cu`, `conmux`, etc.
+
+If you are able to run U-Boot under a hardware simulator such as qemu, then
+you would likely spawn that simulator from this script. However, note that
+`uboot-test-reset` may be called multiple times per test script run, and must
+cause U-Boot to start execution from scratch each time. Hopefully your
+simulator includes a virtual reset button!
+
+#### `uboot-test-flash`
+
+Prior to running the test suite against a board, some arrangement must be made
+so that the board executes the particular U-Boot binary to be tested. Often,
+this involves writing the U-Boot binary to the board's flash ROM. The test
+suite calls this hook script for that purpose.
+
+This script should perform the entire flashing process synchronously; the
+script should only exit once flashing is complete, and a board reset will
+cause the newly flashed U-Boot binary to be executed.
+
+It is conceivable that this script will do nothing. This might be useful in
+the following cases:
+
+- Some other process has already written the desired U-Boot binary into the
+  board's flash prior to running the test suite.
+- The board allows U-Boot to be downloaded directly into RAM, and executed
+  from there. Use of this feature will reduce wear on the board's flash, so
+  may be preferable if available, and if cold boot testing of U-Boot is not
+  required. If this feature is used, the `uboot-test-reset` script should
+  peform this download, since the board could conceivably be reset multiple
+  times in a single test run.
+
+It is up to the user to determine if those situations exist, and to code this
+hook script appropriately.
+
+This script will typically be implemented by calling out to some SoC- or
+board-specific vendor flashing utility.
+
+#### `uboot-test-reset`
+
+Whenever the test suite needs to reset the target board, this script is
+executed. This is guaranteed to happen at least once, prior to executing the
+first test function. If the test script determines the remote U-Boot has
+crashed or hung, it will execute this script again to restore U-Boot to an
+operational state before running the next test function.
+
+This script will likely be implemented by communicating with some form of
+relay or electronic switch attached to the board's reset signal.
+
+The semantics of this script require that when it is executed, U-Boot will
+start running from scratch. If the U-Boot binary to be tested has been written
+to flash, pulsing the board's reset signal is likely all this script need do.
+However, in some scenarios, this script may perform other actions. For
+example, it may call out to some SoC- or board-specific vendor utility in order
+to download the U-Boot binary directly into RAM and execute it. This would
+avoid the need for `uboot-test-flash` to actually write U-Boot to flash, thus
+saving wear on the flash chip(s).
+
+### Board-type-specific configuration
+
+Each board has a different configuration and behaviour. Many of these
+differences can be automatically detected by parsing the `.config` file in the
+build directory. However, some differences can't yet be handled automatically.
+
+For each board, an optional Python module `board_${board_type}.py` may exist
+to provide board-specific information to the test script. Any global value
+defined in these modules is available for use by any test function. The data
+contained in these scripts must be purely derived from U-Boot source code.
+Hence, these configuration files are part of the U-Boot source tree too.
+
+### Execution environment configuration
+
+Each user's hardware setup may enable testing different subsets of the features
+implemented by a particular board's configuration of U-Boot. For example, a
+U-Boot configuration may support USB device mode and USB Mass Storage, but this
+can only be tested if a USB cable is connected between the board and the host
+machine running the test script.
+
+For each board, optional Python modules `boardenv_${board_type}.py` and
+`boardenv_${board_type}_${board_identity}.py` may exist to provide
+board-specific and board-identity-specific information to the test script. Any
+global value defined in these modules is available for use by any test
+function. The data contained in these is specific to a particular user's
+hardware configuration. Hence, these configuration files are not part of the
+U-Boot source tree, and should be installed outside of the source tree. Users
+should set `$PYTHONPATH` prior to running the test script to allow these
+modules to be loaded.
+
+### Configuration parameter usage
+
+The test scripts rely on the following variables being defined by the board
+module:
+
+- `ram_base` an integer indicating the address of the start of RAM. This may
+  be used by tests that read/write RAM.
+
+### Complete invocation example
+
+Assuming that you have installed the hook scripts into $HOME/ubtest/bin, and
+any required environment configuration Python modules into $HOME/ubtest/py,
+then you would likely invoke the test script as follows:
+
+If U-Boot has already been built:
+
+```bash
+PATH=$HOME/ubtest/bin:$PATH \
+    PYTHONPATH=${HOME}/ubtest/py:${PYTHONPATH} \
+    ./test/py/test.py --bd seaboard
+```
+
+If you want the test script to compile U-Boot for you too, then you likely
+need to set `$CROSS_COMPILE` to allow this, and invoke the test script as
+follow:
+
+```bash
+CROSS_COMPILE=arm-none-eabi- \
+    PATH=$HOME/ubtest/bin:$PATH \
+    PYTHONPATH=${HOME}/ubtest/py:${PYTHONPATH} \
+    ./test/py/test.py --bd seaboard --build
+```
+
+## Writing tests
+
+Please refer to the pytest documentation for details of writing pytest tests.
+Details specific to the U-Boot test suite are described below.
+
+A test fixture named `uboot_console` should be used by each test function. This
+provides the means to interact with the U-Boot console, and retrieve board and
+environment configuration information.
+
+The function `uboot_console.run_command()` executes a shell command on the
+U-Boot console, and returns all output from that command. This allows
+validation or interpretation of the command output. This function validates
+that certain strings are not seen on the U-Boot console. These include shell
+error messages and the U-Boot sign-on message (in order to detect unexpected
+board resets). See the source of `uboot_console_base.py` for a complete list of
+"bad" strings. Some test scenarios are expected to trigger these strings. Use
+`uboot_console.disable_check()` to temporarily disable checking for specific
+strings. See `test_unknown_cmd.py` for an example.
+
+Board- and board-environment configuration values may be accessed as sub-fields
+of the `uboot_console.config` object, for example
+`uboot_console.config.ram_base`.
+
+Build configuration values (from `.config`) may be accessed via the dictionary
+`uboot_console.config.buildconfig`, with keys equal to the Kconfig variable
+names.
diff --git a/test/py/board_jetson_tk1.py b/test/py/board_jetson_tk1.py
new file mode 100644
index 000000000000..3fb0753a07f2
--- /dev/null
+++ b/test/py/board_jetson_tk1.py
@@ -0,0 +1 @@
+from soc_tegra124 import *
diff --git a/test/py/board_sandbox.py b/test/py/board_sandbox.py
new file mode 100644
index 000000000000..b3ed9ec44651
--- /dev/null
+++ b/test/py/board_sandbox.py
@@ -0,0 +1 @@
+ram_base = 0
diff --git a/test/py/board_seaboard.py b/test/py/board_seaboard.py
new file mode 100644
index 000000000000..8d32b661849d
--- /dev/null
+++ b/test/py/board_seaboard.py
@@ -0,0 +1 @@
+from soc_tegra20 import *
diff --git a/test/py/conftest.py b/test/py/conftest.py
new file mode 100644
index 000000000000..4b40bdd89a60
--- /dev/null
+++ b/test/py/conftest.py
@@ -0,0 +1,225 @@
+import atexit
+import errno
+import os
+import os.path
+import pexpect
+import pytest
+from _pytest.runner import runtestprotocol
+import ConfigParser
+import StringIO
+import sys
+
+log = None
+console = None
+
+def mkdir_p(path):
+    try:
+        os.makedirs(path)
+    except OSError as exc:
+        if exc.errno == errno.EEXIST and os.path.isdir(path):
+            pass
+        else:
+            raise
+
+def pytest_addoption(parser):
+    parser.addoption("--build-dir", default=None,
+        help="U-Boot build directory (O=)")
+    parser.addoption("--result-dir", default=None,
+        help="U-Boot test result/tmp directory")
+    parser.addoption("--persistent-data-dir", default=None,
+        help="U-Boot test persistent generated data directory")
+    parser.addoption("--board-type", "--bd", "-B", default="sandbox",
+        help="U-Boot board type")
+    parser.addoption("--board-identity", "--id", default="na",
+        help="U-Boot board identity/instance")
+    parser.addoption("--build", default=False, action="store_true",
+        help="Compile U-Boot before running tests")
+
+def pytest_configure(config):
+    global log
+    global console
+    global ubconfig
+
+    test_py_dir = os.path.dirname(os.path.abspath(__file__))
+    source_dir = os.path.dirname(os.path.dirname(test_py_dir))
+
+    board_type = config.getoption("board_type")
+    board_type_fn = board_type.replace("-", "_")
+
+    board_identity = config.getoption("board_identity")
+    board_identity_fn = board_identity.replace("-", "_")
+
+    build_dir = config.getoption("build_dir")
+    if not build_dir:
+        build_dir = source_dir + "/build-" + board_type
+    mkdir_p(build_dir)
+
+    result_dir = config.getoption("result_dir")
+    if not result_dir:
+        result_dir = build_dir
+    mkdir_p(result_dir)
+
+    persistent_data_dir = config.getoption("persistent_data_dir")
+    if not persistent_data_dir:
+        persistent_data_dir = build_dir + "/persistent-data"
+    mkdir_p(persistent_data_dir)
+
+    import multiplexed_log
+    log = multiplexed_log.Logfile(result_dir + "/test-log.html")
+
+    if config.getoption("build"):
+        if build_dir != source_dir:
+            o_opt = "O=%s" % build_dir
+        else:
+            o_opt = ""
+        cmds = (
+            ["make", o_opt, "-s", board_type + "_defconfig"],
+            ["make", o_opt, "-s", "-j8"],
+        )
+        runner = log.get_runner("make", sys.stdout)
+        for cmd in cmds:
+            runner.run(cmd, cwd=source_dir)
+        runner.close()
+
+    board_type_modfn = "board_" + board_type_fn
+    ubconfig = __import__(board_type_modfn)
+
+    override_modfns = [
+        "boardenv_" + board_type_fn,
+        "boardenv_" + board_type_fn + "_" + board_identity_fn,
+    ]
+    for override_modfn in override_modfns:
+        try:
+            override_mod = __import__(override_modfn)
+        except ImportError:
+            continue
+        for (k, v) in override_mod.__dict__.iteritems():
+            if k.startswith("_"):
+                continue
+            ubconfig.__dict__[k] = v
+
+    dot_config = build_dir + "/.config"
+    if os.path.exists(dot_config):
+        with open(dot_config, "rt") as f:
+            ini_str = "[root]\n" + f.read()
+            ini_sio = StringIO.StringIO(ini_str)
+            parser = ConfigParser.RawConfigParser()
+            parser.readfp(ini_sio)
+            ubconfig.buildconfig = dict(parser.items("root"))
+    else:
+        ubconfig.buildconfig = dict()
+
+    ubconfig.test_py_dir = test_py_dir
+    ubconfig.source_dir = source_dir
+    ubconfig.build_dir = build_dir
+    ubconfig.result_dir = result_dir
+    ubconfig.persistent_data_dir = persistent_data_dir
+    ubconfig.board_type = board_type
+    ubconfig.board_identity = board_identity
+
+    env_vars = (
+        "board_type",
+        "board_identity",
+        "source_dir",
+        "test_py_dir",
+        "build_dir",
+        "result_dir",
+        "persistent_data_dir",
+    )
+    for v in env_vars:
+        os.environ["UBOOT_" + v.upper()] = getattr(ubconfig, v)
+
+    if board_type == "sandbox":
+        import uboot_console_sandbox
+        console = uboot_console_sandbox.ConsoleSandbox(log, ubconfig)
+    else:
+        import uboot_console_exec_attach
+        console = uboot_console_exec_attach.ConsoleExecAttach(log, ubconfig)
+
+ at pytest.fixture(scope="session")
+def uboot_console(request):
+    return console
+
+def cleanup():
+    if console:
+        console.close()
+    if log:
+        log.close()
+atexit.register(cleanup)
+
+def setup_boardspec(item):
+    mark = item.get_marker("boardspec")
+    if not mark:
+        return
+    required_boards = []
+    for board in mark.args:
+        if board.startswith("!"):
+            if ubconfig.board_type == board[1:]:
+                pytest.skip("board not supported")
+                return
+        else:
+            required_boards.append(board)
+    if required_boards and ubconfig.board_type not in required_boards:
+        pytest.skip("board not supported")
+
+def setup_buildconfigspec(item):
+    mark = item.get_marker("buildconfigspec")
+    if not mark:
+        return
+    for option in mark.args:
+        if not ubconfig.buildconfig.get("config_" + option.lower(), None):
+            pytest.skip(".config feature not enabled")
+
+def setup_envspec(item):
+    mark = item.get_marker("envspec")
+    if not mark:
+        return
+    for feature in mark.args:
+        if not ubconfig.__dict__.get("env__" + feature, False):
+            pytest.skip("env feature not supported")
+
+def pytest_runtest_setup(item):
+    log.start_section(item.name)
+    if console.at_prompt:
+        console.logstream.write(console.prompt, implicit=True)
+    setup_boardspec(item)
+    setup_buildconfigspec(item)
+    setup_envspec(item)
+
+def pytest_runtest_protocol(item, nextitem):
+    reports = runtestprotocol(item, nextitem=nextitem)
+    failed = None
+    skipped = None
+    for report in reports:
+        if report.outcome == "failed":
+            failed = report
+            break
+        if report.outcome == "skipped":
+            if not skipped:
+                skipped = report
+
+    try:
+        if failed:
+            msg = "FAILED:\n" + str(failed.longrepr)
+            log.status_fail(msg)
+        elif skipped:
+            msg = "SKIPPED:\n" + str(skipped.longrepr)
+            log.status_skipped(msg)
+        else:
+            log.status_pass("OK")
+    except:
+        # If something went wrong with logging, it's better to let the test
+        # process continue, which may report other exceptions that triggered
+        # the logging issue (e.g. console.log wasn't created). Hence, just
+        # squash the exception. If the test setup failed due to e.g. syntax
+        # error somewhere else, this won't be seen. However, once that issue
+        # is fixed, if this exception still exists, it will then be logged as
+        # part of the test's stdout.
+        import traceback
+        print "Exception occurred while logging runtest status:"
+        traceback.print_exc()
+        # FIXME: Can we force a test failure here?
+
+    log.end_section(item.name)
+
+    return reports
diff --git a/test/py/multiplexed_log.css b/test/py/multiplexed_log.css
new file mode 100644
index 000000000000..f0bfffe892de
--- /dev/null
+++ b/test/py/multiplexed_log.css
@@ -0,0 +1,70 @@
+body {
+    background-color: black;
+    color: #ffffff;
+}
+
+.implicit {
+    color: #808080;
+}
+
+.section {
+    border-style: solid;
+    border-color: #303030;
+    border-width: 0px 0px 0px 5px;
+    padding-left: 5px
+}
+
+.section-header {
+    background-color: #303030;
+    margin-left: -5px;
+    margin-top: 5px;
+}
+
+.section-trailer {
+    display: none;
+}
+
+.stream {
+    border-style: solid;
+    border-color: #303030;
+    border-width: 0px 0px 0px 5px;
+    padding-left: 5px
+}
+
+.stream-header {
+    background-color: #303030;
+    margin-left: -5px;
+    margin-top: 5px;
+}
+
+.stream-trailer {
+    display: none;
+}
+
+.error {
+    color: #ff0000
+}
+
+.warning {
+    color: #ffff00
+}
+
+.info {
+    color: #808080
+}
+
+.action {
+    color: #8080ff
+}
+
+.status-pass {
+    color: #00ff00
+}
+
+.status-skipped {
+    color: #ffff00
+}
+
+.status-fail {
+    color: #ff0000
+}
diff --git a/test/py/multiplexed_log.py b/test/py/multiplexed_log.py
new file mode 100644
index 000000000000..46e27de22fe5
--- /dev/null
+++ b/test/py/multiplexed_log.py
@@ -0,0 +1,172 @@
+import cgi
+import os.path
+import shutil
+import subprocess
+
+mod_dir = os.path.dirname(os.path.abspath(__file__))
+
+class LogfileStream(object):
+    def __init__(self, logfile, name, chained_file):
+        self.logfile = logfile
+        self.name = name
+        self.chained_file = chained_file
+
+    def close(self):
+        pass
+
+    def write(self, data, implicit=False):
+        self.logfile.write(self, data, implicit)
+        if self.chained_file:
+            self.chained_file.write(data)
+
+    def flush(self):
+        self.logfile.flush()
+        if self.chained_file:
+            self.chained_file.flush()
+
+class RunAndLog(object):
+    def __init__(self, logfile, name, chained_file):
+        self.logfile = logfile
+        self.name = name
+        self.chained_file = chained_file
+
+    def close(self):
+        pass
+
+    def run(self, cmd, cwd=None):
+        msg = "+" + " ".join(cmd) + "\n"
+        if self.chained_file:
+            self.chained_file.write(msg)
+        self.logfile.write(self, msg)
+
+        try:
+            p = subprocess.Popen(cmd, cwd=cwd,
+                stdin=None, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
+            (output, stderr) = p.communicate()
+            status = p.returncode
+        except subprocess.CalledProcessError as cpe:
+            output = cpe.output
+            status = cpe.returncode
+        self.logfile.write(self, output)
+        if status:
+            if self.chained_file:
+                self.chained_file.write(output)
+            raise Exception("command failed; exit code " + str(status))
+
+class Logfile(object):
+    def __init__(self, fn):
+        self.f = open(fn, "wt")
+        self.last_stream = None
+        self.linebreak = True
+        self.blocks = []
+        shutil.copy(mod_dir + "/multiplexed_log.css", os.path.dirname(fn))
+        self.f.write("""\
+<html>
+<head>
+<link rel="stylesheet" type="text/css" href="multiplexed_log.css">
+</head>
+<body>
+<tt>
+""")
+
+    def close(self):
+        self.f.write("""\
+</tt>
+</body>
+</html>
+""")
+        self.f.close()
+
+    def _escape(self, data):
+        data = data.replace(chr(13), "")
+        data = "".join((c in self._nonprint) and ("%%%02x" % ord(c)) or
+                       c for c in data)
+        data = cgi.escape(data)
+        data = data.replace(" ", "&nbsp;")
+        self.linebreak = data[-1:-1] == "\n"
+        data = data.replace(chr(10), "<br/>\n")
+        return data
+
+    def _terminate_stream(self):
+        if not self.last_stream:
+            return
+        if not self.linebreak:
+            self.f.write("<br/>\n")
+        self.f.write("<div class=\"stream-trailer\" id=\"" +
+                     self.last_stream.name + "\">End stream: " +
+                     self.last_stream.name + "</div>\n")
+        self.f.write("</div>\n")
+        self.last_stream = None
+
+    def _note(self, note_type, msg):
+        self._terminate_stream()
+        self.f.write("<div class=\"" + note_type + "\">\n")
+        self.f.write(self._escape(msg))
+        self.f.write("<br/>\n")
+        self.f.write("</div>\n")
+        self.linebreak = True
+
+    def start_section(self, marker):
+        self._terminate_stream()
+        self.blocks.append(marker)
+        blk_path = "/".join(self.blocks)
+        self.f.write("<div class=\"section\" id=\"" + blk_path + "\">\n")
+        self.f.write("<div class=\"section-header\" id=\"" + blk_path +
+                     "\">Section: " + blk_path + "</div>\n")
+
+    def end_section(self, marker):
+        if (not self.blocks) or (marker != self.blocks[-1]):
+            raise Exception("Block nesting mismatch: \"%s\" \"%s\"" %
+                            (marker, "/".join(self.blocks)))
+        self._terminate_stream()
+        blk_path = "/".join(self.blocks)
+        self.f.write("<div class=\"section-trailer\" id=\"section-trailer-" +
+                     blk_path + "\">End section: " + blk_path + "</div>\n")
+        self.f.write("</div>\n")
+        self.blocks.pop()
+
+    def error(self, msg):
+        self._note("error", msg)
+
+    def warning(self, msg):
+        self._note("warning", msg)
+
+    def info(self, msg):
+        self._note("info", msg)
+
+    def action(self, msg):
+        self._note("action", msg)
+
+    def status_pass(self, msg):
+        self._note("status-pass", msg)
+
+    def status_skipped(self, msg):
+        self._note("status-skipped", msg)
+
+    def status_fail(self, msg):
+        self._note("status-fail", msg)
+
+    def get_stream(self, name, chained_file=None):
+        return LogfileStream(self, name, chained_file)
+
+    def get_runner(self, name, chained_file=None):
+        return RunAndLog(self, name, chained_file)
+
+    _nonprint = ("^%" + "".join(chr(c) for c in range(0, 32) if c != 10) +
+                 "".join(chr(c) for c in range(127, 256)))
+
+    def write(self, stream, data, implicit=False):
+        if stream != self.last_stream:
+            self._terminate_stream()
+            self.f.write("<div class=\"stream\" id=\"%s\">\n" % stream.name)
+            self.f.write("<div class=\"stream-header\" id=\"" + stream.name +
+                         "\">Stream: " + stream.name + "</div>\n")
+        if implicit:
+            self.f.write("<span class=\"implicit\">")
+        self.f.write(self._escape(data))
+        if implicit:
+            self.f.write("</span>")
+        self.last_stream = stream
+
+    def flush(self):
+        self.f.flush()
diff --git a/test/py/pytest.ini b/test/py/pytest.ini
new file mode 100644
index 000000000000..da0d9e553a4b
--- /dev/null
+++ b/test/py/pytest.ini
@@ -0,0 +1,5 @@
+[pytest]
+markers =
+    boardspec: U-Boot: Describes the set of boards a test can/can't run on.
+    buildconfigspec: U-Boot: Describes Kconfig/config-header constraints.
+    envspec: U-Boot: Describes execution environment constraints.
diff --git a/test/py/soc_tegra124.py b/test/py/soc_tegra124.py
new file mode 100644
index 000000000000..a7427dc53261
--- /dev/null
+++ b/test/py/soc_tegra124.py
@@ -0,0 +1 @@
+ram_base = 0x80000000
diff --git a/test/py/soc_tegra20.py b/test/py/soc_tegra20.py
new file mode 100644
index 000000000000..16e3a4966c5c
--- /dev/null
+++ b/test/py/soc_tegra20.py
@@ -0,0 +1 @@
+ram_base = 0x00000000
diff --git a/test/py/test.py b/test/py/test.py
new file mode 100755
index 000000000000..b578af48eb23
--- /dev/null
+++ b/test/py/test.py
@@ -0,0 +1,12 @@
+#!/usr/bin/env python
+
+import os
+import os.path
+import sys
+
+sys.argv.pop(0)
+
+args = ["py.test", os.path.dirname(__file__)]
+args.extend(sys.argv)
+
+os.execvp("py.test", args)
diff --git a/test/py/test_000_version.py b/test/py/test_000_version.py
new file mode 100644
index 000000000000..34822fb398e5
--- /dev/null
+++ b/test/py/test_000_version.py
@@ -0,0 +1,9 @@
+# pytest runs tests the order of their module path, which is related to the
+# filename containing the test. This file is named such that it is sorted
+# first, simply as a very basic sanity check of the functionality of the U-Boot
+# command prompt.
+
+def test_version(uboot_console):
+    with uboot_console.disable_check("main_signon"):
+        response = uboot_console.run_command("version")
+    uboot_console.validate_main_signon_in_text(response)
diff --git a/test/py/test_env.py b/test/py/test_env.py
new file mode 100644
index 000000000000..16891cd6bb15
--- /dev/null
+++ b/test/py/test_env.py
@@ -0,0 +1,96 @@
+import pytest
+
+# FIXME: This might be useful for other tests;
+# perhaps refactor it into ConsoleBase or some other state object?
+class StateTestEnv(object):
+    def __init__(self, uboot_console):
+        self.uboot_console = uboot_console
+        self.get_env()
+        self.set_var = self.get_non_existent_var()
+
+    def get_env(self):
+        response = self.uboot_console.run_command("printenv")
+        self.env = {}
+        for l in response.splitlines():
+            if not "=" in l:
+                continue
+            (var, value) = l.strip().split("=")
+            self.env[var] = value
+
+    def get_existent_var(self):
+        for var in self.env:
+            return var
+
+    def get_non_existent_var(self):
+        n = 0
+        while True:
+            var = "test_env_" + str(n)
+            if var not in self.env:
+                return var
+            n += 1
+
+ at pytest.fixture(scope="module")
+def state_test_env(uboot_console):
+    return StateTestEnv(uboot_console)
+
+def unset_var(state_test_env, var):
+    state_test_env.uboot_console.run_command("setenv " + var)
+    if var in state_test_env.env:
+        del state_test_env.env[var]
+
+def set_var(state_test_env, var, value):
+    state_test_env.uboot_console.run_command("setenv " + var + " " + value)
+    state_test_env.env[var] = value
+
+def validate_empty(state_test_env, var):
+    response = state_test_env.uboot_console.run_command("echo $" + var)
+    assert response == ""
+
+def validate_set(state_test_env, var, value):
+    response = state_test_env.uboot_console.run_command("echo $" + var)
+    assert response == value
+
+def test_env_echo_exists(state_test_env):
+    """Echo a variable that exists"""
+    var = state_test_env.get_existent_var()
+    value = state_test_env.env[var]
+    validate_set(state_test_env, var, value)
+
+def test_env_echo_non_existent(state_test_env):
+    """Echo a variable that doesn't exist"""
+    var = state_test_env.set_var
+    validate_empty(state_test_env, var)
+
+def test_env_printenv_non_existent(state_test_env):
+    """Check printenv error message"""
+    var = state_test_env.set_var
+    c = state_test_env.uboot_console
+    with c.disable_check("error_notification"):
+        response = c.run_command("printenv " + var)
+    assert(response == "## Error: \"" + var + "\" not defined")
+
+def test_env_unset_non_existent(state_test_env):
+    """Unset a nonexistent variable"""
+    var = state_test_env.get_non_existent_var()
+    unset_var(state_test_env, var)
+    validate_empty(state_test_env, var)
+
+def test_env_set_non_existent(state_test_env):
+    """Set a new variable"""
+    var = state_test_env.set_var
+    value = "foo"
+    set_var(state_test_env, var, value)
+    validate_set(state_test_env, var, value)
+
+def test_env_set_existing(state_test_env):
+    """Set an existing variable"""
+    var = state_test_env.set_var
+    value = "bar"
+    set_var(state_test_env, var, value)
+    validate_set(state_test_env, var, value)
+
+def test_env_unset_existing(state_test_env):
+    """Unset a variable"""
+    var = state_test_env.set_var
+    unset_var(state_test_env, var)
+    validate_empty(state_test_env, var)
diff --git a/test/py/test_help.py b/test/py/test_help.py
new file mode 100644
index 000000000000..c2b9ace475e1
--- /dev/null
+++ b/test/py/test_help.py
@@ -0,0 +1,2 @@
+def test_help(uboot_console):
+    uboot_console.run_command("help")
diff --git a/test/py/test_md.py b/test/py/test_md.py
new file mode 100644
index 000000000000..49cdd2685234
--- /dev/null
+++ b/test/py/test_md.py
@@ -0,0 +1,12 @@
+import pytest
+
+ at pytest.mark.buildconfigspec("cmd_memory")
+def test_md(uboot_console):
+    addr = "%08x" % uboot_console.config.ram_base
+    val = "a5f09876"
+    expected_response = addr + ": " + val
+    response = uboot_console.run_command("md " + addr + " 10")
+    assert(not (expected_response in response))
+    uboot_console.run_command("mw " + addr + " " + val)
+    response = uboot_console.run_command("md " + addr + " 10")
+    assert(expected_response in response)
diff --git a/test/py/test_sandbox_exit.py b/test/py/test_sandbox_exit.py
new file mode 100644
index 000000000000..6aefa703a965
--- /dev/null
+++ b/test/py/test_sandbox_exit.py
@@ -0,0 +1,15 @@
+import pytest
+import signal
+
+ at pytest.mark.boardspec("sandbox")
+ at pytest.mark.buildconfigspec("reset")
+def test_reset(uboot_console):
+    uboot_console.run_command("reset", False)
+    assert(uboot_console.validate_exited())
+    uboot_console.ensure_spawned()
+
+ at pytest.mark.boardspec("sandbox")
+def test_ctrlc(uboot_console):
+    uboot_console.kill(signal.SIGINT)
+    assert(uboot_console.validate_exited())
+    uboot_console.ensure_spawned()
diff --git a/test/py/test_unknown_cmd.py b/test/py/test_unknown_cmd.py
new file mode 100644
index 000000000000..19ac52cc24ce
--- /dev/null
+++ b/test/py/test_unknown_cmd.py
@@ -0,0 +1,4 @@
+def test_unknown_command(uboot_console):
+    with uboot_console.disable_check("unknown_command"):
+        response = uboot_console.run_command("non_existent_cmd")
+    assert("Unknown command 'non_existent_cmd' - try 'help'" in response)
diff --git a/test/py/uboot_console_base.py b/test/py/uboot_console_base.py
new file mode 100644
index 000000000000..dfd986860e75
--- /dev/null
+++ b/test/py/uboot_console_base.py
@@ -0,0 +1,143 @@
+import multiplexed_log
+import os
+import re
+import sys
+
+pattern_uboot_spl_signon = re.compile("(U-Boot SPL \\d{4}\\.\\d{2}-[^\r\n]*)")
+pattern_uboot_main_signon = re.compile("(U-Boot \\d{4}\\.\\d{2}-[^\r\n]*)")
+pattern_stop_autoboot_prompt = re.compile("Hit any key to stop autoboot: ")
+pattern_unknown_command = re.compile("Unknown command '.*' - try 'help'")
+pattern_error_notification = re.compile("## Error: ")
+
+class ConsoleDisableCheck(object):
+    def __init__(self, console, check_type):
+        self.console = console
+        self.check_type = check_type
+
+    def __enter__(self):
+        self.console.disable_check_count[self.check_type] += 1
+
+    def __exit__(self, extype, value, traceback):
+        self.console.disable_check_count[self.check_type] -= 1
+
+class ConsoleBase(object):
+    def __init__(self, log, config):
+        self.log = log
+        self.config = config
+
+        self.logstream = self.log.get_stream("console", sys.stdout)
+
+        # Array slice removes leading/trailing quotes
+        self.prompt = self.config.buildconfig["config_sys_prompt"][1:-1]
+        self.prompt_escaped = re.escape(self.prompt)
+        self.p = None
+        self.disable_check_count = {
+            "spl_signon": 0,
+            "main_signon": 0,
+            "unknown_command": 0,
+            "error_notification": 0,
+        }
+
+        self.at_prompt = False
+
+    def close(self):
+        if self.p:
+            self.p.close()
+        self.logstream.close()
+
+    def run_command(self, cmd, wait_for_prompt=True):
+        self.ensure_spawned()
+        bad_patterns = []
+        bad_pattern_ids = []
+        if (self.disable_check_count["spl_signon"] == 0 and
+                self.uboot_spl_signon):
+            bad_patterns.append(self.uboot_spl_signon_escaped)
+            bad_pattern_ids.append("SPL signon")
+        if self.disable_check_count["main_signon"] == 0:
+            bad_patterns.append(self.uboot_main_signon_escaped)
+            bad_pattern_ids.append("U-Boot main signon")
+        if self.disable_check_count["unknown_command"] == 0:
+            bad_patterns.append(pattern_unknown_command)
+            bad_pattern_ids.append("Unknown command")
+        if self.disable_check_count["error_notification"] == 0:
+            bad_patterns.append(pattern_error_notification)
+            bad_pattern_ids.append("Error notification")
+        try:
+            if cmd:
+                self.p.send(cmd)
+                try:
+                    m = self.p.expect([re.escape(cmd)] + bad_patterns)
+                    if m != 0:
+                        self.at_prompt = False
+                        raise Exception("Bad pattern found on console: " +
+                                        bad_pattern_ids[m - 1])
+                except Exception as ex:
+                    self.at_prompt = False
+                    print cmd
+                    self.logstream.write(cmd, implicit=True)
+                    raise
+            self.p.send("\n")
+            if not wait_for_prompt:
+                self.at_prompt = False
+                return
+            m = self.p.expect([self.prompt_escaped] + bad_patterns)
+            if m != 0:
+                self.at_prompt = False
+                raise Exception("Bad pattern found on console: " +
+                                bad_pattern_ids[m - 1])
+            self.at_prompt = True
+            return self.p.before.strip()
+        except Exception as ex:
+            self.at_prompt = False
+            self.log.error(str(ex))
+            self.cleanup_spawn()
+            raise
+
+    def ensure_spawned(self):
+        if self.p:
+            return
+        try:
+            self.at_prompt = False
+            self.log.action("Starting U-Boot")
+            self.p = self.get_spawn()
+            # Real targets can take a long time to scroll large amounts of
+            # text if LCD is enabled. This value may need tweaking in the
+            # future, possibly per-test to be optimal. This works for "help"
+            # on board "seaboard".
+            self.p.timeout = 30
+            self.p.logfile_read = self.logstream
+            if self.config.buildconfig.get("CONFIG_SPL", False) == "y":
+                self.p.expect(pattern_uboot_spl_signon)
+                self.uboot_spl_signon = self.p.after
+                self.uboot_spl_signon_escaped = re.escape(self.p.after)
+            else:
+                self.uboot_spl_signon = None
+            self.p.expect(pattern_uboot_main_signon)
+            self.uboot_main_signon = self.p.after
+            self.uboot_main_signon_escaped = re.escape(self.p.after)
+            while True:
+                match = self.p.expect([self.prompt_escaped,
+                                       pattern_stop_autoboot_prompt])
+                if match == 1:
+                    self.p.send(chr(3)) # CTRL-C
+                    continue
+                break
+            self.at_prompt = True
+        except Exception as ex:
+            self.log.error(str(ex))
+            self.cleanup_spawn()
+            raise
+
+    def cleanup_spawn(self):
+        try:
+            if self.p:
+                self.p.close()
+        except:
+            pass
+        self.p = None
+
+    def validate_main_signon_in_text(self, text):
+        assert(self.uboot_main_signon in text)
+
+    def disable_check(self, check_type):
+        return ConsoleDisableCheck(self, check_type)
diff --git a/test/py/uboot_console_exec_attach.py b/test/py/uboot_console_exec_attach.py
new file mode 100644
index 000000000000..7960d66107c3
--- /dev/null
+++ b/test/py/uboot_console_exec_attach.py
@@ -0,0 +1,28 @@
+import os
+import pexpect
+from uboot_console_base import ConsoleBase
+
+def cmdline(app, args):
+    return app + ' "' + '" "'.join(args) + '"'
+
+class ConsoleExecAttach(ConsoleBase):
+    def __init__(self, log, config):
+        super(ConsoleExecAttach, self).__init__(log, config)
+
+        self.log.action("Flashing U-Boot")
+        cmd = ["uboot-test-flash", config.board_type, config.board_identity]
+        runner = self.log.get_runner(cmd[0])
+        runner.run(cmd)
+        runner.close()
+
+    def get_spawn(self):
+        args = [self.config.board_type, self.config.board_identity]
+        s = pexpect.spawn("uboot-test-console", args=args)
+
+        self.log.action("Resetting board")
+        cmd = ["uboot-test-reset"] + args
+        runner = self.log.get_runner(cmd[0])
+        runner.run(cmd)
+        runner.close()
+
+        return s
diff --git a/test/py/uboot_console_sandbox.py b/test/py/uboot_console_sandbox.py
new file mode 100644
index 000000000000..c3aae3862ca9
--- /dev/null
+++ b/test/py/uboot_console_sandbox.py
@@ -0,0 +1,22 @@
+import os
+import pexpect
+from uboot_console_base import ConsoleBase
+
+class ConsoleSandbox(ConsoleBase):
+    def __init__(self, log, config):
+        super(ConsoleSandbox, self).__init__(log, config)
+
+    def get_spawn(self):
+        return pexpect.spawn(self.config.build_dir + "/u-boot")
+
+    def kill(self, sig):
+        self.ensure_spawned()
+        self.log.action("kill %d" % sig)
+        self.p.kill(sig)
+
+    def validate_exited(self):
+        p = self.p
+        self.p = None
+        ret = p.isalive()
+        p.close()
+        return ret
-- 
1.9.1

^ permalink raw reply related	[flat|nested] 19+ messages in thread