[U-Boot] [PATCH V2 1/7] test/py: Implement pytest infrastructure

[Simon Glass <sjg@chromium.org>]

Hi Stephen,

On 2 December 2015 at 15:18, Stephen Warren <swarren@wwwdotorg.org> wrote:
> This tool aims to test U-Boot by executing U-Boot shell commands using the
> console interface. A single top-level script exists to execute or attach
> to the U-Boot console, run the entire script of tests against it, and
> summarize the results. Advantages of this approach are:
>
> - Testing is performed in the same way a user or script would interact
>   with U-Boot; there can be no disconnect.
> - There is no need to write or embed test-related code into U-Boot itself.
>   It is asserted that writing test-related code in Python is simpler and
>   more flexible that writing it all in C.
> - It is reasonably simple to interact with U-Boot in this way.
>
> A few simple tests are provided as examples. Soon, we should convert as
> many as possible of the other tests in test/* and test/cmd_ut.c too.
>
> In the future, I hope to publish (out-of-tree) the hook scripts, relay
> control utilities, and udev rules I will use for my own HW setup.
>
> See README.md for more details!
>
> Signed-off-by: Stephen Warren <swarren@wwwdotorg.org>
> Signed-off-by: Stephen Warren <swarren@nvidia.com>
> ---
> v2: Many fixes and tweaks have been squashed in. Separated out some of'
> the tests into separate commits, and added some more tests.
> ---
>  test/py/.gitignore                   |   1 +
>  test/py/README.md                    | 300 +++++++++++++++++++++++++++++++++++
>  test/py/conftest.py                  | 278 ++++++++++++++++++++++++++++++++
>  test/py/multiplexed_log.css          |  76 +++++++++
>  test/py/multiplexed_log.py           | 193 ++++++++++++++++++++++
>  test/py/pytest.ini                   |   9 ++
>  test/py/test.py                      |  24 +++
>  test/py/test_000_version.py          |  13 ++
>  test/py/test_help.py                 |   6 +
>  test/py/test_unknown_cmd.py          |   8 +
>  test/py/uboot_console_base.py        | 185 +++++++++++++++++++++
>  test/py/uboot_console_exec_attach.py |  36 +++++
>  test/py/uboot_console_sandbox.py     |  31 ++++
>  test/py/ubspawn.py                   |  97 +++++++++++
>  14 files changed, 1257 insertions(+)
>  create mode 100644 test/py/.gitignore
>  create mode 100644 test/py/README.md
>  create mode 100644 test/py/conftest.py
>  create mode 100644 test/py/multiplexed_log.css
>  create mode 100644 test/py/multiplexed_log.py
>  create mode 100644 test/py/pytest.ini
>  create mode 100755 test/py/test.py
>  create mode 100644 test/py/test_000_version.py
>  create mode 100644 test/py/test_help.py
>  create mode 100644 test/py/test_unknown_cmd.py
>  create mode 100644 test/py/uboot_console_base.py
>  create mode 100644 test/py/uboot_console_exec_attach.py
>  create mode 100644 test/py/uboot_console_sandbox.py
>  create mode 100644 test/py/ubspawn.py

This is a huge step forward for testing in U-Boot. Congratulations on
putting this together!

Tested on chromebook_link, sandbox
Tested-by: Simon Glass <sjg@chromium.org>

I've made various comments in the series as I think it needs a little
tuning. I'm also interested in how we can arrange for the existing
unit tests to be run (and results supported) by this framework.

One concern I have is about the ease of running and writing tests. It
is pretty easy at present to run a particular driver model test:

./u-boot -d test.dtb -c "ut dm uclass"

and we can run this in gdb and figure out where things are going wrong
(I do this quite a bit). Somehow we need to preserve this ease of use.
The tests should be accessible. I'm not sure how you intend to make
that work.

>
> diff --git a/test/py/.gitignore b/test/py/.gitignore
> new file mode 100644
> index 000000000000..0d20b6487c61
> --- /dev/null
> +++ b/test/py/.gitignore
> @@ -0,0 +1 @@
> +*.pyc
> diff --git a/test/py/README.md b/test/py/README.md
> new file mode 100644
> index 000000000000..23a403eb8d88
> --- /dev/null
> +++ b/test/py/README.md
> @@ -0,0 +1,300 @@
> +# U-Boot pytest suite
> +
> +## Introduction
> +
> +This tool aims to test U-Boot by executing U-Boot shell commands using the
> +console interface. A single top-level script exists to execute or attach to the
> +U-Boot console, run the entire script of tests against it, and summarize the
> +results. Advantages of this approach are:
> +
> +- Testing is performed in the same way a user or script would interact with
> +  U-Boot; there can be no disconnect.
> +- There is no need to write or embed test-related code into U-Boot itself.
> +  It is asserted that writing test-related code in Python is simpler and more
> +  flexible that writing it all in C.
> +- It is reasonably simple to interact with U-Boot in this way.
> +
> +## Requirements
> +
> +The test suite is implemented using pytest. Interaction with the U-Boot console
> +involves executing some binary and interacting with its stdin/stdout. You will
> +need to implement various "hook" scripts that are called by the test suite at
> +the appropriate time.
> +
> +On Debian or Debian-like distributions, the following packages are required.
> +Similar package names should exist in other distributions.
> +
> +| Package        | Version tested (Ubuntu 14.04) |
> +| -------------- | ----------------------------- |
> +| python         | 2.7.5-5ubuntu3                |
> +| python-pytest  | 2.5.1-1                       |
> +
> +The test script supports either:
> +
> +- Executing a sandbox port of U-Boot on the local machine as a sub-process,
> +  and interacting with it over stdin/stdout.
> +- Executing an external "hook" scripts to flash a U-Boot binary onto a
> +  physical board, attach to the board's console stream, and reset the board.
> +  Further details are described later.
> +
> +### Using `virtualenv` to provide requirements
> +
> +Older distributions (e.g. Ubuntu 10.04) may not provide all the required
> +packages, or may provide versions that are too old to run the test suite. One
> +can use the Python `virtualenv` script to locally install more up-to-date
> +versions of the required packages without interfering with the OS installation.
> +For example:
> +
> +```bash
> +$ cd /path/to/u-boot
> +$ sudo apt-get install python python-virtualenv
> +$ virtualenv venv
> +$ . ./venv/bin/activate
> +$ pip install pytest
> +```
> +
> +## Testing sandbox
> +
> +To run the testsuite on the sandbox port (U-Boot built as a native user-space
> +application), simply execute:
> +
> +```
> +./test/py/test.py --bd sandbox --build
> +```
> +
> +The `--bd` option tells the test suite which board type is being tested. This
> +lets the test suite know which features the board has, and hence exactly what
> +can be tested.

Can we use -b to fit in with buildman and patman?

> +
> +The `--build` option tells U-Boot to compile U-Boot. Alternatively, you may
> +omit this option and build U-Boot yourself, in whatever way you choose, before
> +running the test script.
> +
> +The test script will attach to U-Boot, execute all valid tests for the board,
> +then print a summary of the test process. A complete log of the test session
> +will be written to `${build_dir}/test-log.html`. This is best viewed in a web
> +browser, but may be read directly as plain text, perhaps with the aid of the
> +`html2text` utility.
> +
> +## Command-line options
> +
> +- `--board-type`, `--bd`, `-B` set the type of the board to be tested. For
> +  example, `sandbox` or `seaboard`.

-b?

> +- `--board-identity`, `--id` set the identity of the board to be tested.
> +  This allows differentiation between multiple instances of the same type of
> +  physical board that are attached to the same host machine. This parameter is
> +  not interpreted by the test script in any way, but rather is simply passed
> +  to the hook scripts described below, and may be used in any site-specific
> +  way deemed necessary.
> +- `--build` indicates that the test script should compile U-Boot itself
> +  before running the tests. If using this option, make sure that any
> +  environment variables required by the build process are already set, such as
> +  `$CROSS_COMPILE`.
> +- `--build-dir` sets the directory containing the compiled U-Boot binaries.
> +  If omitted, this is `${source_dir}/build-${board_type}`.

-d?

> +- `--result-dir` sets the directory to write results, such as log files,
> +  into. If omitted, the build directory is used.

-r?

> +- `--persistent-data-dir` sets the directory used to store persistent test
> +  data. This is test data that may be re-used across test runs, such as file-
> +  system images.

-d?

> +
> +`pytest` also implements a number of its own command-line options. Please see
> +`pytest` documentation for complete details. Execute `py.test --version` for
> +a brief summary. Note that U-Boot's test.py script passes all command-line
> +arguments directly to `pytest` for processing.
> +
> +## Testing real hardware
> +
> +The tools and techniques used to interact with real hardware will vary
> +radically between different host and target systems, and the whims of the user.
> +For this reason, the test suite does not attempt to directly interact with real
> +hardware in any way. Rather, it executes a standardized set of "hook" scripts
> +via `$PATH`. These scripts implement certain actions on behalf of the test
> +suite. This keeps the test suite simple and isolated from system variances
> +unrelated to U-Boot features.
> +
> +### Hook scripts
> +
> +#### Environment variables
> +
> +The following environment variables are set when running hook scripts:
> +
> +- `UBOOT_BOARD_TYPE` the board type being tested.

Shouldn't these be U_BOOT_BOARD_TYPE, etc.?

> +- `UBOOT_BOARD_IDENTITY` the board identity being tested, or `na` if none was
> +  specified.
> +- `UBOOT_SOURCE_DIR` the U-Boot source directory.
> +- `UBOOT_TEST_PY_DIR` the full path to `test/py/` in the source directory.
> +- `UBOOT_BUILD_DIR` the U-Boot build directory.
> +- `UBOOT_RESULT_DIR` the test result directory.
> +- `UBOOT_PERSISTENT_DATA_DIR` the test peristent data directory.
> +
> +#### `uboot-test-console`
> +
> +This script provides access to the U-Boot console. The script's stdin/stdout
> +should be connected to the board's console. This process should continue to run
> +indefinitely, until killed. The test suite will run this script in parallel
> +with all other hooks.
> +
> +This script may be implemented e.g. by exec()ing `cu`, `conmux`, etc.
> +
> +If you are able to run U-Boot under a hardware simulator such as qemu, then
> +you would likely spawn that simulator from this script. However, note that
> +`uboot-test-reset` may be called multiple times per test script run, and must

How aobut u-boot-test-reset, etc.?

> +cause U-Boot to start execution from scratch each time. Hopefully your
> +simulator includes a virtual reset button! If not, you can launch the
> +simulator from `uboot-test-reset` instead, while arranging for this console
> +process to always communicate with the current simulator instance.
> +
> +#### `uboot-test-flash`
> +
> +Prior to running the test suite against a board, some arrangement must be made
> +so that the board executes the particular U-Boot binary to be tested. Often,
> +this involves writing the U-Boot binary to the board's flash ROM. The test
> +suite calls this hook script for that purpose.
> +
> +This script should perform the entire flashing process synchronously; the
> +script should only exit once flashing is complete, and a board reset will
> +cause the newly flashed U-Boot binary to be executed.
> +
> +It is conceivable that this script will do nothing. This might be useful in
> +the following cases:
> +
> +- Some other process has already written the desired U-Boot binary into the
> +  board's flash prior to running the test suite.
> +- The board allows U-Boot to be downloaded directly into RAM, and executed
> +  from there. Use of this feature will reduce wear on the board's flash, so
> +  may be preferable if available, and if cold boot testing of U-Boot is not
> +  required. If this feature is used, the `uboot-test-reset` script should
> +  peform this download, since the board could conceivably be reset multiple
> +  times in a single test run.
> +
> +It is up to the user to determine if those situations exist, and to code this
> +hook script appropriately.
> +
> +This script will typically be implemented by calling out to some SoC- or
> +board-specific vendor flashing utility.
> +
> +#### `uboot-test-reset`
> +
> +Whenever the test suite needs to reset the target board, this script is
> +executed. This is guaranteed to happen at least once, prior to executing the
> +first test function. If any test fails, the test infra-structure will execute
> +this script again to restore U-Boot to an operational state before running the
> +next test function.
> +
> +This script will likely be implemented by communicating with some form of
> +relay or electronic switch attached to the board's reset signal.
> +
> +The semantics of this script require that when it is executed, U-Boot will
> +start running from scratch. If the U-Boot binary to be tested has been written
> +to flash, pulsing the board's reset signal is likely all this script need do.
> +However, in some scenarios, this script may perform other actions. For
> +example, it may call out to some SoC- or board-specific vendor utility in order
> +to download the U-Boot binary directly into RAM and execute it. This would
> +avoid the need for `uboot-test-flash` to actually write U-Boot to flash, thus
> +saving wear on the flash chip(s).
> +
> +### Board-type-specific configuration
> +
> +Each board has a different configuration and behaviour. Many of these
> +differences can be automatically detected by parsing the `.config` file in the
> +build directory. However, some differences can't yet be handled automatically.
> +
> +For each board, an optional Python module `uboot_board_${board_type}` may exist
> +to provide board-specific information to the test script. Any global value
> +defined in these modules is available for use by any test function. The data
> +contained in these scripts must be purely derived from U-Boot source code.
> +Hence, these configuration files are part of the U-Boot source tree too.
> +
> +### Execution environment configuration
> +
> +Each user's hardware setup may enable testing different subsets of the features
> +implemented by a particular board's configuration of U-Boot. For example, a
> +U-Boot configuration may support USB device mode and USB Mass Storage, but this
> +can only be tested if a USB cable is connected between the board and the host
> +machine running the test script.
> +
> +For each board, optional Python modules `uboot_boardenv_${board_type}` and
> +`uboot_boardenv_${board_type}_${board_identity}` may exist to provide
> +board-specific and board-identity-specific information to the test script. Any
> +global value defined in these modules is available for use by any test
> +function. The data contained in these is specific to a particular user's
> +hardware configuration. Hence, these configuration files are not part of the
> +U-Boot source tree, and should be installed outside of the source tree. Users
> +should set `$PYTHONPATH` prior to running the test script to allow these
> +modules to be loaded.
> +
> +### Board module parameter usage
> +
> +The test scripts rely on the following variables being defined by the board
> +module:
> +
> +- None at present.
> +
> +### U-Boot `.config` feature usage
> +
> +The test scripts rely on various U-Boot `.config` features, either directly in
> +order to test those features, or indirectly in order to query information from
> +the running U-Boot instance in order to test other features.
> +
> +One example is that testing of the `md` command requires knowledge of a RAM
> +address to use for the test. This data is parsed from the output of the
> +`bdinfo` command, and hence relies on CONFIG_CMD_BDI being enabled.
> +
> +For a complete list of dependencies, please search the test scripts for
> +instances of:
> +
> +- `buildconfig.get(...`
> +- `@pytest.mark.buildconfigspec(...`
> +
> +### Complete invocation example
> +
> +Assuming that you have installed the hook scripts into $HOME/ubtest/bin, and
> +any required environment configuration Python modules into $HOME/ubtest/py,
> +then you would likely invoke the test script as follows:
> +
> +If U-Boot has already been built:
> +
> +```bash
> +PATH=$HOME/ubtest/bin:$PATH \
> +    PYTHONPATH=${HOME}/ubtest/py:${PYTHONPATH} \
> +    ./test/py/test.py --bd seaboard
> +```
> +
> +If you want the test script to compile U-Boot for you too, then you likely
> +need to set `$CROSS_COMPILE` to allow this, and invoke the test script as
> +follow:
> +
> +```bash
> +CROSS_COMPILE=arm-none-eabi- \
> +    PATH=$HOME/ubtest/bin:$PATH \
> +    PYTHONPATH=${HOME}/ubtest/py:${PYTHONPATH} \
> +    ./test/py/test.py --bd seaboard --build
> +```
> +
> +## Writing tests
> +
> +Please refer to the pytest documentation for details of writing pytest tests.
> +Details specific to the U-Boot test suite are described below.
> +
> +A test fixture named `uboot_console` should be used by each test function. This
> +provides the means to interact with the U-Boot console, and retrieve board and
> +environment configuration information.
> +
> +The function `uboot_console.run_command()` executes a shell command on the
> +U-Boot console, and returns all output from that command. This allows
> +validation or interpretation of the command output. This function validates
> +that certain strings are not seen on the U-Boot console. These include shell
> +error messages and the U-Boot sign-on message (in order to detect unexpected
> +board resets). See the source of `uboot_console_base.py` for a complete list of
> +"bad" strings. Some test scenarios are expected to trigger these strings. Use
> +`uboot_console.disable_check()` to temporarily disable checking for specific
> +strings. See `test_unknown_cmd.py` for an example.
> +
> +Board- and board-environment configuration values may be accessed as sub-fields
> +of the `uboot_console.config` object, for example
> +`uboot_console.config.ram_base`.
> +
> +Build configuration values (from `.config`) may be accessed via the dictionary
> +`uboot_console.config.buildconfig`, with keys equal to the Kconfig variable
> +names.
> diff --git a/test/py/conftest.py b/test/py/conftest.py
> new file mode 100644
> index 000000000000..b6efe03a60f8
> --- /dev/null
> +++ b/test/py/conftest.py
> @@ -0,0 +1,278 @@
> +# Copyright (c) 2015 Stephen Warren
> +# Copyright (c) 2015, NVIDIA CORPORATION. All rights reserved.
> +#
> +# SPDX-License-Identifier: GPL-2.0
> +
> +import atexit
> +import errno
> +import os
> +import os.path
> +import pexpect
> +import pytest
> +from _pytest.runner import runtestprotocol
> +import ConfigParser
> +import StringIO
> +import sys
> +
> +log = None
> +console = None
> +
> +def mkdir_p(path):
> +    try:
> +        os.makedirs(path)
> +    except OSError as exc:
> +        if exc.errno == errno.EEXIST and os.path.isdir(path):
> +            pass
> +        else:
> +            raise
> +
> +def pytest_addoption(parser):
> +    parser.addoption("--build-dir", default=None,
> +        help="U-Boot build directory (O=)")

You seem to use double quote consistently throughout rather than a
single quote. That is different from the existing Python in the U-Boot
tree. It might be worth swapping it out for consistency.

> +    parser.addoption("--result-dir", default=None,
> +        help="U-Boot test result/tmp directory")
> +    parser.addoption("--persistent-data-dir", default=None,
> +        help="U-Boot test persistent generated data directory")
> +    parser.addoption("--board-type", "--bd", "-B", default="sandbox",
> +        help="U-Boot board type")
> +    parser.addoption("--board-identity", "--id", default="na",
> +        help="U-Boot board identity/instance")
> +    parser.addoption("--build", default=False, action="store_true",
> +        help="Compile U-Boot before running tests")
> +
> +def pytest_configure(config):

This series should have function comments throughout on non-trivial
functions - e.g. purpose of the function and a description of the
parameters and return value.

> +    global log
> +    global console
> +    global ubconfig
> +
> +    test_py_dir = os.path.dirname(os.path.abspath(__file__))
> +    source_dir = os.path.dirname(os.path.dirname(test_py_dir))
> +
> +    board_type = config.getoption("board_type")
> +    board_type_fn = board_type.replace("-", "_")
> +
> +    board_identity = config.getoption("board_identity")
> +    board_identity_fn = board_identity.replace("-", "_")
> +
> +    build_dir = config.getoption("build_dir")
> +    if not build_dir:
> +        build_dir = source_dir + "/build-" + board_type
> +    mkdir_p(build_dir)
> +
> +    result_dir = config.getoption("result_dir")
> +    if not result_dir:
> +        result_dir = build_dir
> +    mkdir_p(result_dir)
> +
> +    persistent_data_dir = config.getoption("persistent_data_dir")
> +    if not persistent_data_dir:
> +        persistent_data_dir = build_dir + "/persistent-data"
> +    mkdir_p(persistent_data_dir)
> +
> +    import multiplexed_log
> +    log = multiplexed_log.Logfile(result_dir + "/test-log.html")
> +
> +    if config.getoption("build"):
> +        if build_dir != source_dir:
> +            o_opt = "O=%s" % build_dir
> +        else:
> +            o_opt = ""
> +        cmds = (
> +            ["make", o_opt, "-s", board_type + "_defconfig"],
> +            ["make", o_opt, "-s", "-j8"],
> +        )
> +        runner = log.get_runner("make", sys.stdout)
> +        for cmd in cmds:
> +            runner.run(cmd, cwd=source_dir)
> +        runner.close()
> +
> +    class ArbitraryAttrContainer(object):
> +        pass
> +
> +    ubconfig = ArbitraryAttrContainer()
> +    ubconfig.brd = dict()
> +    ubconfig.env = dict()
> +
> +    modules = [
> +        (ubconfig.brd, "uboot_board_" + board_type_fn),
> +        (ubconfig.env, "uboot_boardenv_" + board_type_fn),
> +        (ubconfig.env, "uboot_boardenv_" + board_type_fn + "_" +
> +                                                board_identity_fn),
> +    ]
> +    for (sub_config, mod_name) in modules:
> +        try:
> +            mod = __import__(mod_name)
> +        except ImportError:
> +            continue
> +        sub_config.update(mod.__dict__)
> +
> +    ubconfig.buildconfig = dict()
> +
> +    for conf_file in (".config", "include/autoconf.mk"):
> +        dot_config = build_dir + "/" + conf_file
> +        if not os.path.exists(dot_config):
> +            raise Exception(conf_file + " does not exist; " +
> +                "try passing --build option?")
> +
> +        with open(dot_config, "rt") as f:
> +            ini_str = "[root]\n" + f.read()
> +            ini_sio = StringIO.StringIO(ini_str)
> +            parser = ConfigParser.RawConfigParser()
> +            parser.readfp(ini_sio)
> +            ubconfig.buildconfig.update(parser.items("root"))
> +
> +    ubconfig.test_py_dir = test_py_dir
> +    ubconfig.source_dir = source_dir
> +    ubconfig.build_dir = build_dir
> +    ubconfig.result_dir = result_dir
> +    ubconfig.persistent_data_dir = persistent_data_dir
> +    ubconfig.board_type = board_type
> +    ubconfig.board_identity = board_identity
> +
> +    env_vars = (
> +        "board_type",
> +        "board_identity",
> +        "source_dir",
> +        "test_py_dir",
> +        "build_dir",
> +        "result_dir",
> +        "persistent_data_dir",
> +    )
> +    for v in env_vars:
> +        os.environ["UBOOT_" + v.upper()] = getattr(ubconfig, v)
> +
> +    if board_type == "sandbox":
> +        import uboot_console_sandbox
> +        console = uboot_console_sandbox.ConsoleSandbox(log, ubconfig)
> +    else:
> +        import uboot_console_exec_attach
> +        console = uboot_console_exec_attach.ConsoleExecAttach(log, ubconfig)
> +
> +def pytest_generate_tests(metafunc):
> +    subconfigs = {
> +        "brd": console.config.brd,
> +        "env": console.config.env,
> +    }
> +    for fn in metafunc.fixturenames:
> +        parts = fn.split("__")
> +        if len(parts) < 2:
> +            continue
> +        if parts[0] not in subconfigs:
> +            continue
> +        subconfig = subconfigs[parts[0]]
> +        vals = []
> +        val = subconfig.get(fn, [])
> +        if val:
> +            vals = (val, )
> +        else:
> +            vals = subconfig.get(fn + "s", [])
> +        metafunc.parametrize(fn, vals)
> +
> + at pytest.fixture(scope="session")
> +def uboot_console(request):
> +    return console
> +
> +tests_not_run = set()
> +tests_failed = set()
> +tests_skipped = set()
> +tests_passed = set()
> +
> +def pytest_itemcollected(item):
> +    tests_not_run.add(item.name)
> +
> +def cleanup():
> +    if console:
> +        console.close()
> +    if log:
> +        log.status_pass("%d passed" % len(tests_passed))
> +        if tests_skipped:
> +            log.status_skipped("%d skipped" % len(tests_skipped))
> +            for test in tests_skipped:
> +                log.status_skipped("... " + test)
> +        if tests_failed:
> +            log.status_fail("%d failed" % len(tests_failed))
> +            for test in tests_failed:
> +                log.status_fail("... " + test)
> +        if tests_not_run:
> +            log.status_fail("%d not run" % len(tests_not_run))
> +            for test in tests_not_run:
> +                log.status_fail("... " + test)
> +        log.close()
> +atexit.register(cleanup)
> +
> +def setup_boardspec(item):
> +    mark = item.get_marker("boardspec")
> +    if not mark:
> +        return
> +    required_boards = []
> +    for board in mark.args:
> +        if board.startswith("!"):
> +            if ubconfig.board_type == board[1:]:
> +                pytest.skip("board not supported")
> +                return
> +        else:
> +            required_boards.append(board)
> +    if required_boards and ubconfig.board_type not in required_boards:
> +        pytest.skip("board not supported")
> +
> +def setup_buildconfigspec(item):
> +    mark = item.get_marker("buildconfigspec")
> +    if not mark:
> +        return
> +    for option in mark.args:
> +        if not ubconfig.buildconfig.get("config_" + option.lower(), None):
> +            pytest.skip(".config feature not enabled")
> +
> +def pytest_runtest_setup(item):
> +    log.start_section(item.name)
> +    setup_boardspec(item)
> +    setup_buildconfigspec(item)
> +
> +def pytest_runtest_protocol(item, nextitem):
> +    reports = runtestprotocol(item, nextitem=nextitem)
> +    failed = None
> +    skipped = None
> +    for report in reports:
> +        if report.outcome == "failed":
> +            failed = report
> +            break
> +        if report.outcome == "skipped":
> +            if not skipped:
> +                skipped = report
> +
> +    if failed:
> +        tests_failed.add(item.name)
> +    elif skipped:
> +        tests_skipped.add(item.name)
> +    else:
> +        tests_passed.add(item.name)
> +    tests_not_run.remove(item.name)
> +
> +    try:
> +        if failed:
> +            msg = "FAILED:\n" + str(failed.longrepr)
> +            log.status_fail(msg)
> +        elif skipped:
> +            msg = "SKIPPED:\n" + str(skipped.longrepr)
> +            log.status_skipped(msg)
> +        else:
> +            log.status_pass("OK")
> +    except:
> +        # If something went wrong with logging, it's better to let the test
> +        # process continue, which may report other exceptions that triggered
> +        # the logging issue (e.g. console.log wasn't created). Hence, just
> +        # squash the exception. If the test setup failed due to e.g. syntax
> +        # error somewhere else, this won't be seen. However, once that issue
> +        # is fixed, if this exception still exists, it will then be logged as
> +        # part of the test's stdout.
> +        import traceback
> +        print "Exception occurred while logging runtest status:"
> +        traceback.print_exc()
> +        # FIXME: Can we force a test failure here?
> +
> +    log.end_section(item.name)
> +
> +    if failed:
> +        console.cleanup_spawn()
> +
> +    return reports
> diff --git a/test/py/multiplexed_log.css b/test/py/multiplexed_log.css
> new file mode 100644
> index 000000000000..96d87ebe034b
> --- /dev/null
> +++ b/test/py/multiplexed_log.css
> @@ -0,0 +1,76 @@
> +/*
> + * Copyright (c) 2015 Stephen Warren
> + *
> + * SPDX-License-Identifier: GPL-2.0
> + */
> +
> +body {
> +    background-color: black;
> +    color: #ffffff;
> +}
> +
> +.implicit {
> +    color: #808080;
> +}
> +
> +.section {
> +    border-style: solid;
> +    border-color: #303030;
> +    border-width: 0px 0px 0px 5px;
> +    padding-left: 5px
> +}
> +
> +.section-header {
> +    background-color: #303030;
> +    margin-left: -5px;
> +    margin-top: 5px;
> +}
> +
> +.section-trailer {
> +    display: none;
> +}
> +
> +.stream {
> +    border-style: solid;
> +    border-color: #303030;
> +    border-width: 0px 0px 0px 5px;
> +    padding-left: 5px
> +}
> +
> +.stream-header {
> +    background-color: #303030;
> +    margin-left: -5px;
> +    margin-top: 5px;
> +}
> +
> +.stream-trailer {
> +    display: none;
> +}
> +
> +.error {
> +    color: #ff0000
> +}
> +
> +.warning {
> +    color: #ffff00
> +}
> +
> +.info {
> +    color: #808080
> +}
> +
> +.action {
> +    color: #8080ff
> +}
> +
> +.status-pass {
> +    color: #00ff00
> +}
> +
> +.status-skipped {
> +    color: #ffff00
> +}
> +
> +.status-fail {
> +    color: #ff0000
> +}
> diff --git a/test/py/multiplexed_log.py b/test/py/multiplexed_log.py
> new file mode 100644
> index 000000000000..58b9a9c50ecf
> --- /dev/null
> +++ b/test/py/multiplexed_log.py
> @@ -0,0 +1,193 @@
> +# Copyright (c) 2015 Stephen Warren
> +# Copyright (c) 2015, NVIDIA CORPORATION. All rights reserved.
> +#
> +# SPDX-License-Identifier: GPL-2.0
> +
> +import cgi
> +import os.path
> +import shutil
> +import subprocess
> +
> +mod_dir = os.path.dirname(os.path.abspath(__file__))
> +
> +class LogfileStream(object):
> +    def __init__(self, logfile, name, chained_file):
> +        self.logfile = logfile
> +        self.name = name
> +        self.chained_file = chained_file
> +
> +    def close(self):
> +        pass
> +
> +    def write(self, data, implicit=False):
> +        self.logfile.write(self, data, implicit)
> +        if self.chained_file:
> +            self.chained_file.write(data)
> +
> +    def flush(self):
> +        self.logfile.flush()
> +        if self.chained_file:
> +            self.chained_file.flush()
> +
> +class RunAndLog(object):
> +    def __init__(self, logfile, name, chained_file):
> +        self.logfile = logfile
> +        self.name = name
> +        self.chained_file = chained_file
> +
> +    def close(self):
> +        pass
> +
> +    def run(self, cmd, cwd=None):
> +        msg = "+" + " ".join(cmd) + "\n"
> +        if self.chained_file:
> +            self.chained_file.write(msg)
> +        self.logfile.write(self, msg)
> +
> +        try:
> +            p = subprocess.Popen(cmd, cwd=cwd,
> +                stdin=None, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
> +            (output, stderr) = p.communicate()
> +            status = p.returncode
> +        except subprocess.CalledProcessError as cpe:
> +            output = cpe.output
> +            status = cpe.returncode
> +        self.logfile.write(self, output)
> +        if status:
> +            if self.chained_file:
> +                self.chained_file.write(output)
> +            raise Exception("command failed; exit code " + str(status))
> +
> +class SectionCtxMgr(object):
> +    def __init__(self, log, marker):
> +        self.log = log
> +        self.marker = marker
> +
> +    def __enter__(self):
> +        self.log.start_section(self.marker)
> +
> +    def __exit__(self, extype, value, traceback):
> +        self.log.end_section(self.marker)
> +
> +class Logfile(object):
> +    def __init__(self, fn):
> +        self.f = open(fn, "wt")
> +        self.last_stream = None
> +        self.linebreak = True
> +        self.blocks = []
> +        self.cur_evt = 1
> +        shutil.copy(mod_dir + "/multiplexed_log.css", os.path.dirname(fn))
> +        self.f.write("""\
> +<html>
> +<head>
> +<link rel="stylesheet" type="text/css" href="multiplexed_log.css">
> +</head>
> +<body>
> +<tt>
> +""")
> +
> +    def close(self):
> +        self.f.write("""\
> +</tt>
> +</body>
> +</html>
> +""")
> +        self.f.close()
> +
> +    def _escape(self, data):
> +        data = data.replace(chr(13), "")
> +        data = "".join((c in self._nonprint) and ("%%%02x" % ord(c)) or
> +                       c for c in data)
> +        data = cgi.escape(data)
> +        data = data.replace(" ", "&nbsp;")
> +        self.linebreak = data[-1:-1] == "\n"
> +        data = data.replace(chr(10), "<br/>\n")
> +        return data
> +
> +    def _terminate_stream(self):
> +        self.cur_evt += 1
> +        if not self.last_stream:
> +            return
> +        if not self.linebreak:
> +            self.f.write("<br/>\n")
> +        self.f.write("<div class=\"stream-trailer\" id=\"" +
> +                     self.last_stream.name + "\">End stream: " +
> +                     self.last_stream.name + "</div>\n")
> +        self.f.write("</div>\n")
> +        self.last_stream = None
> +
> +    def _note(self, note_type, msg):
> +        self._terminate_stream()
> +        self.f.write("<div class=\"" + note_type + "\">\n")
> +        self.f.write(self._escape(msg))
> +        self.f.write("<br/>\n")
> +        self.f.write("</div>\n")
> +        self.linebreak = True
> +
> +    def start_section(self, marker):
> +        self._terminate_stream()
> +        self.blocks.append(marker)
> +        blk_path = "/".join(self.blocks)
> +        self.f.write("<div class=\"section\" id=\"" + blk_path + "\">\n")
> +        self.f.write("<div class=\"section-header\" id=\"" + blk_path +
> +                     "\">Section: " + blk_path + "</div>\n")
> +
> +    def end_section(self, marker):
> +        if (not self.blocks) or (marker != self.blocks[-1]):
> +            raise Exception("Block nesting mismatch: \"%s\" \"%s\"" %
> +                            (marker, "/".join(self.blocks)))
> +        self._terminate_stream()
> +        blk_path = "/".join(self.blocks)
> +        self.f.write("<div class=\"section-trailer\" id=\"section-trailer-" +
> +                     blk_path + "\">End section: " + blk_path + "</div>\n")
> +        self.f.write("</div>\n")
> +        self.blocks.pop()
> +
> +    def section(self, marker):
> +        return SectionCtxMgr(self, marker)
> +
> +    def error(self, msg):
> +        self._note("error", msg)
> +
> +    def warning(self, msg):
> +        self._note("warning", msg)
> +
> +    def info(self, msg):
> +        self._note("info", msg)
> +
> +    def action(self, msg):
> +        self._note("action", msg)
> +
> +    def status_pass(self, msg):
> +        self._note("status-pass", msg)
> +
> +    def status_skipped(self, msg):
> +        self._note("status-skipped", msg)
> +
> +    def status_fail(self, msg):
> +        self._note("status-fail", msg)
> +
> +    def get_stream(self, name, chained_file=None):
> +        return LogfileStream(self, name, chained_file)
> +
> +    def get_runner(self, name, chained_file=None):
> +        return RunAndLog(self, name, chained_file)
> +
> +    _nonprint = ("^%" + "".join(chr(c) for c in range(0, 32) if c != 10) +
> +                 "".join(chr(c) for c in range(127, 256)))
> +
> +    def write(self, stream, data, implicit=False):
> +        if stream != self.last_stream:
> +            self._terminate_stream()
> +            self.f.write("<div class=\"stream\" id=\"%s\">\n" % stream.name)
> +            self.f.write("<div class=\"stream-header\" id=\"" + stream.name +
> +                         "\">Stream: " + stream.name + "</div>\n")
> +        if implicit:
> +            self.f.write("<span class=\"implicit\">")
> +        self.f.write(self._escape(data))
> +        if implicit:
> +            self.f.write("</span>")
> +        self.last_stream = stream
> +
> +    def flush(self):
> +        self.f.flush()
> diff --git a/test/py/pytest.ini b/test/py/pytest.ini
> new file mode 100644
> index 000000000000..1bdff810d36e
> --- /dev/null
> +++ b/test/py/pytest.ini
> @@ -0,0 +1,9 @@
> +# Copyright (c) 2015 Stephen Warren
> +# Copyright (c) 2015, NVIDIA CORPORATION. All rights reserved.
> +#
> +# SPDX-License-Identifier: GPL-2.0
> +
> +[pytest]
> +markers =
> +    boardspec: U-Boot: Describes the set of boards a test can/can't run on.
> +    buildconfigspec: U-Boot: Describes Kconfig/config-header constraints.
> diff --git a/test/py/test.py b/test/py/test.py
> new file mode 100755
> index 000000000000..7768216a2335
> --- /dev/null
> +++ b/test/py/test.py
> @@ -0,0 +1,24 @@
> +#!/usr/bin/env python
> +
> +# Copyright (c) 2015 Stephen Warren
> +# Copyright (c) 2015, NVIDIA CORPORATION. All rights reserved.
> +#
> +# SPDX-License-Identifier: GPL-2.0
> +
> +import os
> +import os.path
> +import sys
> +
> +sys.argv.pop(0)
> +
> +args = ["py.test", os.path.dirname(__file__)]
> +args.extend(sys.argv)
> +
> +try:
> +    os.execvp("py.test", args)
> +except:
> +    import traceback
> +    traceback.print_exc()
> +    print >>sys.stderr, """
> +exec(py.test) failed; perhaps you are missing some dependencies?
> +See test/md/README.md for the list."""
> diff --git a/test/py/test_000_version.py b/test/py/test_000_version.py
> new file mode 100644
> index 000000000000..360c8fd726e0
> --- /dev/null
> +++ b/test/py/test_000_version.py
> @@ -0,0 +1,13 @@
> +# Copyright (c) 2015 Stephen Warren
> +#
> +# SPDX-License-Identifier: GPL-2.0
> +
> +# pytest runs tests the order of their module path, which is related to the
> +# filename containing the test. This file is named such that it is sorted
> +# first, simply as a very basic sanity check of the functionality of the U-Boot
> +# command prompt.
> +
> +def test_version(uboot_console):
> +    with uboot_console.disable_check("main_signon"):
> +        response = uboot_console.run_command("version")
> +    uboot_console.validate_main_signon_in_text(response)
> diff --git a/test/py/test_help.py b/test/py/test_help.py
> new file mode 100644
> index 000000000000..3cc896ee7af8
> --- /dev/null
> +++ b/test/py/test_help.py
> @@ -0,0 +1,6 @@
> +# Copyright (c) 2015 Stephen Warren
> +#
> +# SPDX-License-Identifier: GPL-2.0
> +
> +def test_help(uboot_console):
> +    uboot_console.run_command("help")
> diff --git a/test/py/test_unknown_cmd.py b/test/py/test_unknown_cmd.py
> new file mode 100644
> index 000000000000..ba12de56a294
> --- /dev/null
> +++ b/test/py/test_unknown_cmd.py
> @@ -0,0 +1,8 @@
> +# Copyright (c) 2015 Stephen Warren
> +#
> +# SPDX-License-Identifier: GPL-2.0
> +
> +def test_unknown_command(uboot_console):
> +    with uboot_console.disable_check("unknown_command"):
> +        response = uboot_console.run_command("non_existent_cmd")
> +    assert("Unknown command 'non_existent_cmd' - try 'help'" in response)
> diff --git a/test/py/uboot_console_base.py b/test/py/uboot_console_base.py
> new file mode 100644
> index 000000000000..9f13fead2e7e
> --- /dev/null
> +++ b/test/py/uboot_console_base.py
> @@ -0,0 +1,185 @@
> +# Copyright (c) 2015 Stephen Warren
> +# Copyright (c) 2015, NVIDIA CORPORATION. All rights reserved.
> +#
> +# SPDX-License-Identifier: GPL-2.0
> +
> +import multiplexed_log
> +import os
> +import pytest
> +import re
> +import sys
> +
> +pattern_uboot_spl_signon = re.compile("(U-Boot SPL \\d{4}\\.\\d{2}-[^\r\n]*)")
> +pattern_uboot_main_signon = re.compile("(U-Boot \\d{4}\\.\\d{2}-[^\r\n]*)")
> +pattern_stop_autoboot_prompt = re.compile("Hit any key to stop autoboot: ")
> +pattern_unknown_command = re.compile("Unknown command '.*' - try 'help'")
> +pattern_error_notification = re.compile("## Error: ")
> +
> +class ConsoleDisableCheck(object):
> +    def __init__(self, console, check_type):
> +        self.console = console
> +        self.check_type = check_type
> +
> +    def __enter__(self):
> +        self.console.disable_check_count[self.check_type] += 1
> +
> +    def __exit__(self, extype, value, traceback):
> +        self.console.disable_check_count[self.check_type] -= 1
> +
> +class ConsoleBase(object):
> +    def __init__(self, log, config, max_fifo_fill):
> +        self.log = log
> +        self.config = config
> +        self.max_fifo_fill = max_fifo_fill
> +
> +        self.logstream = self.log.get_stream("console", sys.stdout)
> +
> +        # Array slice removes leading/trailing quotes
> +        self.prompt = self.config.buildconfig["config_sys_prompt"][1:-1]
> +        self.prompt_escaped = re.escape(self.prompt)
> +        self.p = None
> +        self.disable_check_count = {
> +            "spl_signon": 0,
> +            "main_signon": 0,
> +            "unknown_command": 0,
> +            "error_notification": 0,
> +        }
> +
> +        self.at_prompt = False
> +        self.at_prompt_logevt = None
> +        self.ram_base = None
> +
> +    def close(self):
> +        if self.p:
> +            self.p.close()
> +        self.logstream.close()
> +
> +    def run_command(self, cmd, wait_for_echo=True, send_nl=True, wait_for_prompt=True):
> +        self.ensure_spawned()
> +
> +        if self.at_prompt and \
> +                self.at_prompt_logevt != self.logstream.logfile.cur_evt:
> +            self.logstream.write(self.prompt, implicit=True)
> +
> +        bad_patterns = []
> +        bad_pattern_ids = []
> +        if (self.disable_check_count["spl_signon"] == 0 and
> +                self.uboot_spl_signon):
> +            bad_patterns.append(self.uboot_spl_signon_escaped)
> +            bad_pattern_ids.append("SPL signon")
> +        if self.disable_check_count["main_signon"] == 0:
> +            bad_patterns.append(self.uboot_main_signon_escaped)
> +            bad_pattern_ids.append("U-Boot main signon")
> +        if self.disable_check_count["unknown_command"] == 0:
> +            bad_patterns.append(pattern_unknown_command)
> +            bad_pattern_ids.append("Unknown command")
> +        if self.disable_check_count["error_notification"] == 0:
> +            bad_patterns.append(pattern_error_notification)
> +            bad_pattern_ids.append("Error notification")
> +        try:
> +            self.at_prompt = False
> +            if send_nl:
> +                cmd += "\n"
> +            while cmd:
> +                # Limit max outstanding data, so UART FIFOs don't overflow
> +                chunk = cmd[:self.max_fifo_fill]
> +                cmd = cmd[self.max_fifo_fill:]
> +                self.p.send(chunk)
> +                if not wait_for_echo:
> +                    continue
> +                chunk = re.escape(chunk)
> +                chunk = chunk.replace("\\\n", "[\r\n]")
> +                m = self.p.expect([chunk] + bad_patterns)
> +                if m != 0:
> +                    self.at_prompt = False
> +                    raise Exception("Bad pattern found on console: " +
> +                                    bad_pattern_ids[m - 1])
> +            if not wait_for_prompt:
> +                return
> +            m = self.p.expect([self.prompt_escaped] + bad_patterns)
> +            if m != 0:
> +                self.at_prompt = False
> +                raise Exception("Bad pattern found on console: " +
> +                                bad_pattern_ids[m - 1])
> +            self.at_prompt = True
> +            self.at_prompt_logevt = self.logstream.logfile.cur_evt
> +            # Only strip \r\n; space/TAB might be significant if testing
> +            # indentation.
> +            return self.p.before.strip("\r\n")
> +        except Exception as ex:
> +            self.log.error(str(ex))
> +            self.cleanup_spawn()
> +            raise
> +
> +    def ctrlc(self):
> +        self.run_command(chr(3), wait_for_echo=False, send_nl=False)
> +
> +    def ensure_spawned(self):
> +        if self.p:
> +            return
> +        try:
> +            self.at_prompt = False
> +            self.log.action("Starting U-Boot")
> +            self.p = self.get_spawn()
> +            # Real targets can take a long time to scroll large amounts of
> +            # text if LCD is enabled. This value may need tweaking in the
> +            # future, possibly per-test to be optimal. This works for "help"
> +            # on board "seaboard".
> +            self.p.timeout = 30000
> +            self.p.logfile_read = self.logstream

Also I have found that tests fail on chromebook_link because it cannot
keep up with the pace of keyboard input. I'm not sure what the
solution is - maybe the best thing is to implement buffering in the
serial uclass, assuming that fixes it. For now I disabled LCD output.

I think it would be worth adding a test that checks for the banner and
the prompt, so we know that other test failures are not due to this
problem.

> +            if self.config.buildconfig.get("CONFIG_SPL", False) == "y":
> +                self.p.expect([pattern_uboot_spl_signon])
> +                self.uboot_spl_signon = self.p.after
> +                self.uboot_spl_signon_escaped = re.escape(self.p.after)
> +            else:
> +                self.uboot_spl_signon = None
> +            self.p.expect([pattern_uboot_main_signon])
> +            self.uboot_main_signon = self.p.after
> +            self.uboot_main_signon_escaped = re.escape(self.p.after)
> +            while True:
> +                match = self.p.expect([self.prompt_escaped,
> +                                       pattern_stop_autoboot_prompt])
> +                if match == 1:
> +                    self.p.send(chr(3)) # CTRL-C
> +                    continue
> +                break
> +            self.at_prompt = True
> +            self.at_prompt_logevt = self.logstream.logfile.cur_evt
> +        except Exception as ex:
> +            self.log.error(str(ex))
> +            self.cleanup_spawn()
> +            raise
> +
> +    def cleanup_spawn(self):
> +        try:
> +            if self.p:
> +                self.p.close()
> +        except:
> +            pass
> +        self.p = None
> +
> +    def validate_main_signon_in_text(self, text):
> +        assert(self.uboot_main_signon in text)
> +
> +    def disable_check(self, check_type):
> +        return ConsoleDisableCheck(self, check_type)
> +
> +    def find_ram_base(self):
> +        if self.config.buildconfig.get("config_cmd_bdi", "n") != "y":
> +            pytest.skip("bdinfo command not supported")
> +        if self.ram_base == -1:
> +            pytest.skip("Previously failed to find RAM bank start")
> +        if self.ram_base is not None:
> +            return self.ram_base
> +
> +        with self.log.section("find_ram_base"):
> +            response = self.run_command("bdinfo")
> +            for l in response.split("\n"):
> +                if "-> start" in l:
> +                    self.ram_base = int(l.split("=")[1].strip(), 16)
> +                    break
> +            if self.ram_base is None:
> +                self.ram_base = -1
> +                raise Exception("Failed to find RAM bank start in `bdinfo`")
> +
> +        return self.ram_base
> diff --git a/test/py/uboot_console_exec_attach.py b/test/py/uboot_console_exec_attach.py
> new file mode 100644
> index 000000000000..0267ae4dc070
> --- /dev/null
> +++ b/test/py/uboot_console_exec_attach.py
> @@ -0,0 +1,36 @@
> +# Copyright (c) 2015 Stephen Warren
> +# Copyright (c) 2015, NVIDIA CORPORATION. All rights reserved.
> +#
> +# SPDX-License-Identifier: GPL-2.0

It would be useful to have a short description at the top of each file
/ class explaining what it is for.

> +
> +from ubspawn import Spawn
> +from uboot_console_base import ConsoleBase
> +
> +def cmdline(app, args):
> +    return app + ' "' + '" "'.join(args) + '"'
> +
> +class ConsoleExecAttach(ConsoleBase):
> +    def __init__(self, log, config):
> +        # The max_fifo_fill value might need tweaking per-board/-SoC?
> +        # 1 would be safe anywhere, but is very slow (a pexpect issue?).
> +        # 16 is a common FIFO size.
> +        # HW flow control would mean this could be infinite.
> +        super(ConsoleExecAttach, self).__init__(log, config, max_fifo_fill=16)
> +
> +        self.log.action("Flashing U-Boot")
> +        cmd = ["uboot-test-flash", config.board_type, config.board_identity]
> +        runner = self.log.get_runner(cmd[0])
> +        runner.run(cmd)
> +        runner.close()
> +
[snip]

Regards,
Simon