From: Stephen Warren <swarren@wwwdotorg.org>
To: u-boot@lists.denx.de
Subject: [U-Boot] [PATCH] Implement pytest-based test infrastructure
Date: Sat, 14 Nov 2015 23:53:01 -0700 [thread overview]
Message-ID: <1447570381-1361-1-git-send-email-swarren@wwwdotorg.org> (raw)
This tool aims to test U-Boot by executing U-Boot shell commands using the
console interface. A single top-level script exists to execute or attach
to the U-Boot console, run the entire script of tests against it, and
summarize the results. Advantages of this approach are:
- Testing is performed in the same way a user or script would interact
with U-Boot; there can be no disconnect.
- There is no need to write or embed test-related code into U-Boot itself.
It is asserted that writing test-related code in Python is simpler and
more flexible that writing it all in C.
- It is reasonably simple to interact with U-Boot in this way.
A few simple tests are provided as examples. Soon, we should convert as
many as possible of the other tests in test/* and test/cmd_ut.c too.
In the future, I hope to publish (out-of-tree) the hook scripts, relay
control utilities, and udev rules I will use for my own HW setup.
See README.md for more details!
Signed-off-by: Stephen Warren <swarren@wwwdotorg.org>
---
.gitignore | 1 +
test/py/README.md | 287 +++++++++++++++++++++++++++++++++++
test/py/board_jetson_tk1.py | 1 +
test/py/board_sandbox.py | 1 +
test/py/board_seaboard.py | 1 +
test/py/conftest.py | 225 +++++++++++++++++++++++++++
test/py/multiplexed_log.css | 70 +++++++++
test/py/multiplexed_log.py | 172 +++++++++++++++++++++
test/py/pytest.ini | 5 +
test/py/soc_tegra124.py | 1 +
test/py/soc_tegra20.py | 1 +
test/py/test.py | 12 ++
test/py/test_000_version.py | 9 ++
test/py/test_env.py | 96 ++++++++++++
test/py/test_help.py | 2 +
test/py/test_md.py | 12 ++
test/py/test_sandbox_exit.py | 15 ++
test/py/test_unknown_cmd.py | 4 +
test/py/uboot_console_base.py | 143 +++++++++++++++++
test/py/uboot_console_exec_attach.py | 28 ++++
test/py/uboot_console_sandbox.py | 22 +++
21 files changed, 1108 insertions(+)
create mode 100644 test/py/README.md
create mode 100644 test/py/board_jetson_tk1.py
create mode 100644 test/py/board_sandbox.py
create mode 100644 test/py/board_seaboard.py
create mode 100644 test/py/conftest.py
create mode 100644 test/py/multiplexed_log.css
create mode 100644 test/py/multiplexed_log.py
create mode 100644 test/py/pytest.ini
create mode 100644 test/py/soc_tegra124.py
create mode 100644 test/py/soc_tegra20.py
create mode 100755 test/py/test.py
create mode 100644 test/py/test_000_version.py
create mode 100644 test/py/test_env.py
create mode 100644 test/py/test_help.py
create mode 100644 test/py/test_md.py
create mode 100644 test/py/test_sandbox_exit.py
create mode 100644 test/py/test_unknown_cmd.py
create mode 100644 test/py/uboot_console_base.py
create mode 100644 test/py/uboot_console_exec_attach.py
create mode 100644 test/py/uboot_console_sandbox.py
diff --git a/.gitignore b/.gitignore
index 33abbd3d0783..b276b3a160bb 100644
--- a/.gitignore
+++ b/.gitignore
@@ -20,6 +20,7 @@
*.bin
*.patch
*.cfgtmp
+*.pyc
# host programs on Cygwin
*.exe
diff --git a/test/py/README.md b/test/py/README.md
new file mode 100644
index 000000000000..70104d2f3b5e
--- /dev/null
+++ b/test/py/README.md
@@ -0,0 +1,287 @@
+# U-Boot pytest suite
+
+## Introduction
+
+This tool aims to test U-Boot by executing U-Boot shell commands using the
+console interface. A single top-level script exists to execute or attach to the
+U-Boot console, run the entire script of tests against it, and summarize the
+results. Advantages of this approach are:
+
+- Testing is performed in the same way a user or script would interact with
+ U-Boot; there can be no disconnect.
+- There is no need to write or embed test-related code into U-Boot itself.
+ It is asserted that writing test-related code in Python is simpler and more
+ flexible that writing it all in C.
+- It is reasonably simple to interact with U-Boot in this way.
+
+## Requirements
+
+The test suite is implemented using pytest. Interaction with the U-Boot
+console uses pexpect. Interaction with real hardware uses the tools of your
+choice; you get to implement various "hook" scripts that are called by the
+test suite at the appropriate time.
+
+On Debian or Debian-like distributions, the following packages are required.
+Similar package names should exist in other distributions.
+
+| Package | Version tested (Ubuntu 14.04) |
+| -------------- | ----------------------------- |
+| python | 2.7.5-5ubuntu3 |
+| python-pytest | 2.5.1-1 |
+| python-pexpect | 3.1-1ubuntu0.1 |
+
+The test script supports either:
+
+- Executing a sandbox port of U-Boot on the local machine as a sub-process,
+ and interacting with it over stdin/stdout.
+- Executing external "hook" scripts to flash a U-Boot binary onto a physical
+ board, attach to the board's console stream, and reset the board. Further
+ details are described later.
+
+### Using `virtualenv` to provide requirements
+
+Older distributions (e.g. Ubuntu 10.04) may not provide all the required
+packages, or may provide versions that are too old to run the test suite. One
+can use the Python `virtualenv` script to locally install more up-to-date
+versions of the required packages without interfering with the OS installation.
+For example:
+
+```bash
+$ cd /path/to/u-boot
+$ sudo apt-get install python python-virtualenv pexpect
+$ virtualenv venv
+$ . ./venv/bin/activate
+$ pip install pytest
+```
+
+## Testing sandbox
+
+To run the testsuite on the sandbox port (U-Boot built as a native user-space
+application), simply execute:
+
+```
+./test/py/test.py --bd sandbox --build
+```
+
+The `--bd` option tells the test suite which board type is being tested. This
+lets the test suite know which features the board has, and hence exactly what
+can be tested.
+
+The `--build` option tells U-Boot to compile U-Boot. Alternatively, you may
+omit this option and build U-Boot yourself, in whatever way you choose, before
+running the test script.
+
+The test script will attach to U-Boot, execute all valid tests for the board,
+then print a summary of the test process. A complete log of the test session
+will be written to `${build_dir}/test-log.html`. This is best viewed in a web
+browser, by may be read directly as plain text, perhaps with the aid of the
+`html2text` utility.
+
+## Command-line options
+
+- `--board-type`, `--bd`, `-B` set the type of the board to be tested. For
+ example, `sandbox` or `seaboard`.
+- `--board-identity`, `--id` set the identity of the board to be tested.
+ This allows differentiation between multiple instances of the same type of
+ physical board that are attached to the same host machine. This parameter is
+ not interpreted by the test script in any way, but rather is simply passed
+ to the hook scripts described below, and may be used in any site-specific
+ way deemed necessary.
+- `--build` indicates that the test script should compile U-Boot itself
+ before running the tests. If using this option, make sure that any
+ environment variables required by the build process are already set, such as
+ `$CROSS_COMPILE`.
+- `--build-dir` sets the directory containing the compiled U-Boot binaries.
+ If omitted, this is `${source_dir}/build-${board_type}`.
+- `--result-dir` sets the directory to write results, such as log files,
+ into. If omitted, the build directory is used.
+- `--persistent-data-dir` sets the directory used to store persistent test
+ data. This is test data that may be re-used across test runs, such as file-
+ system images.
+
+`pytest` also implements a number of its own command-line options. Please see
+`pytest` documentation for complete details. Execute `py.test --version` for
+a brief summary. Note that U-Boot's test.py script passes all command-line
+arguments directly to `pytest` for processing.
+
+## Testing real hardware
+
+The tools and techniques used to interact with real hardware will vary
+radically between different host and target systems, and the whims of the user.
+For this reason, the test suite does not attempt to directly interact with real
+hardware in any way. Rather, it expects a standardized set of "hook" scripts to
+exist which implement certain actions on behalf of the test suite. This keeps
+the test suite simple and isolated from system variances unrelated to U-Boot
+features.
+
+### Hook scripts
+
+The test suite requires the following hook scripts to be executable via
+`$PATH`:
+
+#### Environment variables
+
+The following environment variables are set when running hook scripts:
+
+- `UBOOT_BOARD_TYPE` the board type being tested.
+- `UBOOT_BOARD_IDENTITY` the board identity being tested, or `na` if none was
+ specified.
+- `UBOOT_SOURCE_DIR` the U-Boot source directory.
+- `UBOOT_TEST_PY_DIR` the full path to `test/py/` in the source directory.
+- `UBOOT_BUILD_DIR` the U-Boot build directory.
+- `UBOOT_RESULT_DIR` the test result directory.
+- `UBOOT_PERSISTENT_DATA_DIR` the test peristent data directory.
+
+#### `uboot-test-console`
+
+This script provides access to the U-Boot console. The script's stdin/stdout
+should be connected to the board's console. This script should continue to run
+indefinitely, until killed. The test suite will run this script in parallel
+with all other hooks.
+
+This script may be implemented by executing e.g. `cu`, `conmux`, etc.
+
+If you are able to run U-Boot under a hardware simulator such as qemu, then
+you would likely spawn that simulator from this script. However, note that
+`uboot-test-reset` may be called multiple times per test script run, and must
+cause U-Boot to start execution from scratch each time. Hopefully your
+simulator includes a virtual reset button!
+
+#### `uboot-test-flash`
+
+Prior to running the test suite against a board, some arrangement must be made
+so that the board executes the particular U-Boot binary to be tested. Often,
+this involves writing the U-Boot binary to the board's flash ROM. The test
+suite calls this hook script for that purpose.
+
+This script should perform the entire flashing process synchronously; the
+script should only exit once flashing is complete, and a board reset will
+cause the newly flashed U-Boot binary to be executed.
+
+It is conceivable that this script will do nothing. This might be useful in
+the following cases:
+
+- Some other process has already written the desired U-Boot binary into the
+ board's flash prior to running the test suite.
+- The board allows U-Boot to be downloaded directly into RAM, and executed
+ from there. Use of this feature will reduce wear on the board's flash, so
+ may be preferable if available, and if cold boot testing of U-Boot is not
+ required. If this feature is used, the `uboot-test-reset` script should
+ peform this download, since the board could conceivably be reset multiple
+ times in a single test run.
+
+It is up to the user to determine if those situations exist, and to code this
+hook script appropriately.
+
+This script will typically be implemented by calling out to some SoC- or
+board-specific vendor flashing utility.
+
+#### `uboot-test-reset`
+
+Whenever the test suite needs to reset the target board, this script is
+executed. This is guaranteed to happen at least once, prior to executing the
+first test function. If the test script determines the remote U-Boot has
+crashed or hung, it will execute this script again to restore U-Boot to an
+operational state before running the next test function.
+
+This script will likely be implemented by communicating with some form of
+relay or electronic switch attached to the board's reset signal.
+
+The semantics of this script require that when it is executed, U-Boot will
+start running from scratch. If the U-Boot binary to be tested has been written
+to flash, pulsing the board's reset signal is likely all this script need do.
+However, in some scenarios, this script may perform other actions. For
+example, it may call out to some SoC- or board-specific vendor utility in order
+to download the U-Boot binary directly into RAM and execute it. This would
+avoid the need for `uboot-test-flash` to actually write U-Boot to flash, thus
+saving wear on the flash chip(s).
+
+### Board-type-specific configuration
+
+Each board has a different configuration and behaviour. Many of these
+differences can be automatically detected by parsing the `.config` file in the
+build directory. However, some differences can't yet be handled automatically.
+
+For each board, an optional Python module `board_${board_type}.py` may exist
+to provide board-specific information to the test script. Any global value
+defined in these modules is available for use by any test function. The data
+contained in these scripts must be purely derived from U-Boot source code.
+Hence, these configuration files are part of the U-Boot source tree too.
+
+### Execution environment configuration
+
+Each user's hardware setup may enable testing different subsets of the features
+implemented by a particular board's configuration of U-Boot. For example, a
+U-Boot configuration may support USB device mode and USB Mass Storage, but this
+can only be tested if a USB cable is connected between the board and the host
+machine running the test script.
+
+For each board, optional Python modules `boardenv_${board_type}.py` and
+`boardenv_${board_type}_${board_identity}.py` may exist to provide
+board-specific and board-identity-specific information to the test script. Any
+global value defined in these modules is available for use by any test
+function. The data contained in these is specific to a particular user's
+hardware configuration. Hence, these configuration files are not part of the
+U-Boot source tree, and should be installed outside of the source tree. Users
+should set `$PYTHONPATH` prior to running the test script to allow these
+modules to be loaded.
+
+### Configuration parameter usage
+
+The test scripts rely on the following variables being defined by the board
+module:
+
+- `ram_base` an integer indicating the address of the start of RAM. This may
+ be used by tests that read/write RAM.
+
+### Complete invocation example
+
+Assuming that you have installed the hook scripts into $HOME/ubtest/bin, and
+any required environment configuration Python modules into $HOME/ubtest/py,
+then you would likely invoke the test script as follows:
+
+If U-Boot has already been built:
+
+```bash
+PATH=$HOME/ubtest/bin:$PATH \
+ PYTHONPATH=${HOME}/ubtest/py:${PYTHONPATH} \
+ ./test/py/test.py --bd seaboard
+```
+
+If you want the test script to compile U-Boot for you too, then you likely
+need to set `$CROSS_COMPILE` to allow this, and invoke the test script as
+follow:
+
+```bash
+CROSS_COMPILE=arm-none-eabi- \
+ PATH=$HOME/ubtest/bin:$PATH \
+ PYTHONPATH=${HOME}/ubtest/py:${PYTHONPATH} \
+ ./test/py/test.py --bd seaboard --build
+```
+
+## Writing tests
+
+Please refer to the pytest documentation for details of writing pytest tests.
+Details specific to the U-Boot test suite are described below.
+
+A test fixture named `uboot_console` should be used by each test function. This
+provides the means to interact with the U-Boot console, and retrieve board and
+environment configuration information.
+
+The function `uboot_console.run_command()` executes a shell command on the
+U-Boot console, and returns all output from that command. This allows
+validation or interpretation of the command output. This function validates
+that certain strings are not seen on the U-Boot console. These include shell
+error messages and the U-Boot sign-on message (in order to detect unexpected
+board resets). See the source of `uboot_console_base.py` for a complete list of
+"bad" strings. Some test scenarios are expected to trigger these strings. Use
+`uboot_console.disable_check()` to temporarily disable checking for specific
+strings. See `test_unknown_cmd.py` for an example.
+
+Board- and board-environment configuration values may be accessed as sub-fields
+of the `uboot_console.config` object, for example
+`uboot_console.config.ram_base`.
+
+Build configuration values (from `.config`) may be accessed via the dictionary
+`uboot_console.config.buildconfig`, with keys equal to the Kconfig variable
+names.
diff --git a/test/py/board_jetson_tk1.py b/test/py/board_jetson_tk1.py
new file mode 100644
index 000000000000..3fb0753a07f2
--- /dev/null
+++ b/test/py/board_jetson_tk1.py
@@ -0,0 +1 @@
+from soc_tegra124 import *
diff --git a/test/py/board_sandbox.py b/test/py/board_sandbox.py
new file mode 100644
index 000000000000..b3ed9ec44651
--- /dev/null
+++ b/test/py/board_sandbox.py
@@ -0,0 +1 @@
+ram_base = 0
diff --git a/test/py/board_seaboard.py b/test/py/board_seaboard.py
new file mode 100644
index 000000000000..8d32b661849d
--- /dev/null
+++ b/test/py/board_seaboard.py
@@ -0,0 +1 @@
+from soc_tegra20 import *
diff --git a/test/py/conftest.py b/test/py/conftest.py
new file mode 100644
index 000000000000..4b40bdd89a60
--- /dev/null
+++ b/test/py/conftest.py
@@ -0,0 +1,225 @@
+import atexit
+import errno
+import os
+import os.path
+import pexpect
+import pytest
+from _pytest.runner import runtestprotocol
+import ConfigParser
+import StringIO
+import sys
+
+log = None
+console = None
+
+def mkdir_p(path):
+ try:
+ os.makedirs(path)
+ except OSError as exc:
+ if exc.errno == errno.EEXIST and os.path.isdir(path):
+ pass
+ else:
+ raise
+
+def pytest_addoption(parser):
+ parser.addoption("--build-dir", default=None,
+ help="U-Boot build directory (O=)")
+ parser.addoption("--result-dir", default=None,
+ help="U-Boot test result/tmp directory")
+ parser.addoption("--persistent-data-dir", default=None,
+ help="U-Boot test persistent generated data directory")
+ parser.addoption("--board-type", "--bd", "-B", default="sandbox",
+ help="U-Boot board type")
+ parser.addoption("--board-identity", "--id", default="na",
+ help="U-Boot board identity/instance")
+ parser.addoption("--build", default=False, action="store_true",
+ help="Compile U-Boot before running tests")
+
+def pytest_configure(config):
+ global log
+ global console
+ global ubconfig
+
+ test_py_dir = os.path.dirname(os.path.abspath(__file__))
+ source_dir = os.path.dirname(os.path.dirname(test_py_dir))
+
+ board_type = config.getoption("board_type")
+ board_type_fn = board_type.replace("-", "_")
+
+ board_identity = config.getoption("board_identity")
+ board_identity_fn = board_identity.replace("-", "_")
+
+ build_dir = config.getoption("build_dir")
+ if not build_dir:
+ build_dir = source_dir + "/build-" + board_type
+ mkdir_p(build_dir)
+
+ result_dir = config.getoption("result_dir")
+ if not result_dir:
+ result_dir = build_dir
+ mkdir_p(result_dir)
+
+ persistent_data_dir = config.getoption("persistent_data_dir")
+ if not persistent_data_dir:
+ persistent_data_dir = build_dir + "/persistent-data"
+ mkdir_p(persistent_data_dir)
+
+ import multiplexed_log
+ log = multiplexed_log.Logfile(result_dir + "/test-log.html")
+
+ if config.getoption("build"):
+ if build_dir != source_dir:
+ o_opt = "O=%s" % build_dir
+ else:
+ o_opt = ""
+ cmds = (
+ ["make", o_opt, "-s", board_type + "_defconfig"],
+ ["make", o_opt, "-s", "-j8"],
+ )
+ runner = log.get_runner("make", sys.stdout)
+ for cmd in cmds:
+ runner.run(cmd, cwd=source_dir)
+ runner.close()
+
+ board_type_modfn = "board_" + board_type_fn
+ ubconfig = __import__(board_type_modfn)
+
+ override_modfns = [
+ "boardenv_" + board_type_fn,
+ "boardenv_" + board_type_fn + "_" + board_identity_fn,
+ ]
+ for override_modfn in override_modfns:
+ try:
+ override_mod = __import__(override_modfn)
+ except ImportError:
+ continue
+ for (k, v) in override_mod.__dict__.iteritems():
+ if k.startswith("_"):
+ continue
+ ubconfig.__dict__[k] = v
+
+ dot_config = build_dir + "/.config"
+ if os.path.exists(dot_config):
+ with open(dot_config, "rt") as f:
+ ini_str = "[root]\n" + f.read()
+ ini_sio = StringIO.StringIO(ini_str)
+ parser = ConfigParser.RawConfigParser()
+ parser.readfp(ini_sio)
+ ubconfig.buildconfig = dict(parser.items("root"))
+ else:
+ ubconfig.buildconfig = dict()
+
+ ubconfig.test_py_dir = test_py_dir
+ ubconfig.source_dir = source_dir
+ ubconfig.build_dir = build_dir
+ ubconfig.result_dir = result_dir
+ ubconfig.persistent_data_dir = persistent_data_dir
+ ubconfig.board_type = board_type
+ ubconfig.board_identity = board_identity
+
+ env_vars = (
+ "board_type",
+ "board_identity",
+ "source_dir",
+ "test_py_dir",
+ "build_dir",
+ "result_dir",
+ "persistent_data_dir",
+ )
+ for v in env_vars:
+ os.environ["UBOOT_" + v.upper()] = getattr(ubconfig, v)
+
+ if board_type == "sandbox":
+ import uboot_console_sandbox
+ console = uboot_console_sandbox.ConsoleSandbox(log, ubconfig)
+ else:
+ import uboot_console_exec_attach
+ console = uboot_console_exec_attach.ConsoleExecAttach(log, ubconfig)
+
+ at pytest.fixture(scope="session")
+def uboot_console(request):
+ return console
+
+def cleanup():
+ if console:
+ console.close()
+ if log:
+ log.close()
+atexit.register(cleanup)
+
+def setup_boardspec(item):
+ mark = item.get_marker("boardspec")
+ if not mark:
+ return
+ required_boards = []
+ for board in mark.args:
+ if board.startswith("!"):
+ if ubconfig.board_type == board[1:]:
+ pytest.skip("board not supported")
+ return
+ else:
+ required_boards.append(board)
+ if required_boards and ubconfig.board_type not in required_boards:
+ pytest.skip("board not supported")
+
+def setup_buildconfigspec(item):
+ mark = item.get_marker("buildconfigspec")
+ if not mark:
+ return
+ for option in mark.args:
+ if not ubconfig.buildconfig.get("config_" + option.lower(), None):
+ pytest.skip(".config feature not enabled")
+
+def setup_envspec(item):
+ mark = item.get_marker("envspec")
+ if not mark:
+ return
+ for feature in mark.args:
+ if not ubconfig.__dict__.get("env__" + feature, False):
+ pytest.skip("env feature not supported")
+
+def pytest_runtest_setup(item):
+ log.start_section(item.name)
+ if console.at_prompt:
+ console.logstream.write(console.prompt, implicit=True)
+ setup_boardspec(item)
+ setup_buildconfigspec(item)
+ setup_envspec(item)
+
+def pytest_runtest_protocol(item, nextitem):
+ reports = runtestprotocol(item, nextitem=nextitem)
+ failed = None
+ skipped = None
+ for report in reports:
+ if report.outcome == "failed":
+ failed = report
+ break
+ if report.outcome == "skipped":
+ if not skipped:
+ skipped = report
+
+ try:
+ if failed:
+ msg = "FAILED:\n" + str(failed.longrepr)
+ log.status_fail(msg)
+ elif skipped:
+ msg = "SKIPPED:\n" + str(skipped.longrepr)
+ log.status_skipped(msg)
+ else:
+ log.status_pass("OK")
+ except:
+ # If something went wrong with logging, it's better to let the test
+ # process continue, which may report other exceptions that triggered
+ # the logging issue (e.g. console.log wasn't created). Hence, just
+ # squash the exception. If the test setup failed due to e.g. syntax
+ # error somewhere else, this won't be seen. However, once that issue
+ # is fixed, if this exception still exists, it will then be logged as
+ # part of the test's stdout.
+ import traceback
+ print "Exception occurred while logging runtest status:"
+ traceback.print_exc()
+ # FIXME: Can we force a test failure here?
+
+ log.end_section(item.name)
+
+ return reports
diff --git a/test/py/multiplexed_log.css b/test/py/multiplexed_log.css
new file mode 100644
index 000000000000..f0bfffe892de
--- /dev/null
+++ b/test/py/multiplexed_log.css
@@ -0,0 +1,70 @@
+body {
+ background-color: black;
+ color: #ffffff;
+}
+
+.implicit {
+ color: #808080;
+}
+
+.section {
+ border-style: solid;
+ border-color: #303030;
+ border-width: 0px 0px 0px 5px;
+ padding-left: 5px
+}
+
+.section-header {
+ background-color: #303030;
+ margin-left: -5px;
+ margin-top: 5px;
+}
+
+.section-trailer {
+ display: none;
+}
+
+.stream {
+ border-style: solid;
+ border-color: #303030;
+ border-width: 0px 0px 0px 5px;
+ padding-left: 5px
+}
+
+.stream-header {
+ background-color: #303030;
+ margin-left: -5px;
+ margin-top: 5px;
+}
+
+.stream-trailer {
+ display: none;
+}
+
+.error {
+ color: #ff0000
+}
+
+.warning {
+ color: #ffff00
+}
+
+.info {
+ color: #808080
+}
+
+.action {
+ color: #8080ff
+}
+
+.status-pass {
+ color: #00ff00
+}
+
+.status-skipped {
+ color: #ffff00
+}
+
+.status-fail {
+ color: #ff0000
+}
diff --git a/test/py/multiplexed_log.py b/test/py/multiplexed_log.py
new file mode 100644
index 000000000000..46e27de22fe5
--- /dev/null
+++ b/test/py/multiplexed_log.py
@@ -0,0 +1,172 @@
+import cgi
+import os.path
+import shutil
+import subprocess
+
+mod_dir = os.path.dirname(os.path.abspath(__file__))
+
+class LogfileStream(object):
+ def __init__(self, logfile, name, chained_file):
+ self.logfile = logfile
+ self.name = name
+ self.chained_file = chained_file
+
+ def close(self):
+ pass
+
+ def write(self, data, implicit=False):
+ self.logfile.write(self, data, implicit)
+ if self.chained_file:
+ self.chained_file.write(data)
+
+ def flush(self):
+ self.logfile.flush()
+ if self.chained_file:
+ self.chained_file.flush()
+
+class RunAndLog(object):
+ def __init__(self, logfile, name, chained_file):
+ self.logfile = logfile
+ self.name = name
+ self.chained_file = chained_file
+
+ def close(self):
+ pass
+
+ def run(self, cmd, cwd=None):
+ msg = "+" + " ".join(cmd) + "\n"
+ if self.chained_file:
+ self.chained_file.write(msg)
+ self.logfile.write(self, msg)
+
+ try:
+ p = subprocess.Popen(cmd, cwd=cwd,
+ stdin=None, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
+ (output, stderr) = p.communicate()
+ status = p.returncode
+ except subprocess.CalledProcessError as cpe:
+ output = cpe.output
+ status = cpe.returncode
+ self.logfile.write(self, output)
+ if status:
+ if self.chained_file:
+ self.chained_file.write(output)
+ raise Exception("command failed; exit code " + str(status))
+
+class Logfile(object):
+ def __init__(self, fn):
+ self.f = open(fn, "wt")
+ self.last_stream = None
+ self.linebreak = True
+ self.blocks = []
+ shutil.copy(mod_dir + "/multiplexed_log.css", os.path.dirname(fn))
+ self.f.write("""\
+<html>
+<head>
+<link rel="stylesheet" type="text/css" href="multiplexed_log.css">
+</head>
+<body>
+<tt>
+""")
+
+ def close(self):
+ self.f.write("""\
+</tt>
+</body>
+</html>
+""")
+ self.f.close()
+
+ def _escape(self, data):
+ data = data.replace(chr(13), "")
+ data = "".join((c in self._nonprint) and ("%%%02x" % ord(c)) or
+ c for c in data)
+ data = cgi.escape(data)
+ data = data.replace(" ", " ")
+ self.linebreak = data[-1:-1] == "\n"
+ data = data.replace(chr(10), "<br/>\n")
+ return data
+
+ def _terminate_stream(self):
+ if not self.last_stream:
+ return
+ if not self.linebreak:
+ self.f.write("<br/>\n")
+ self.f.write("<div class=\"stream-trailer\" id=\"" +
+ self.last_stream.name + "\">End stream: " +
+ self.last_stream.name + "</div>\n")
+ self.f.write("</div>\n")
+ self.last_stream = None
+
+ def _note(self, note_type, msg):
+ self._terminate_stream()
+ self.f.write("<div class=\"" + note_type + "\">\n")
+ self.f.write(self._escape(msg))
+ self.f.write("<br/>\n")
+ self.f.write("</div>\n")
+ self.linebreak = True
+
+ def start_section(self, marker):
+ self._terminate_stream()
+ self.blocks.append(marker)
+ blk_path = "/".join(self.blocks)
+ self.f.write("<div class=\"section\" id=\"" + blk_path + "\">\n")
+ self.f.write("<div class=\"section-header\" id=\"" + blk_path +
+ "\">Section: " + blk_path + "</div>\n")
+
+ def end_section(self, marker):
+ if (not self.blocks) or (marker != self.blocks[-1]):
+ raise Exception("Block nesting mismatch: \"%s\" \"%s\"" %
+ (marker, "/".join(self.blocks)))
+ self._terminate_stream()
+ blk_path = "/".join(self.blocks)
+ self.f.write("<div class=\"section-trailer\" id=\"section-trailer-" +
+ blk_path + "\">End section: " + blk_path + "</div>\n")
+ self.f.write("</div>\n")
+ self.blocks.pop()
+
+ def error(self, msg):
+ self._note("error", msg)
+
+ def warning(self, msg):
+ self._note("warning", msg)
+
+ def info(self, msg):
+ self._note("info", msg)
+
+ def action(self, msg):
+ self._note("action", msg)
+
+ def status_pass(self, msg):
+ self._note("status-pass", msg)
+
+ def status_skipped(self, msg):
+ self._note("status-skipped", msg)
+
+ def status_fail(self, msg):
+ self._note("status-fail", msg)
+
+ def get_stream(self, name, chained_file=None):
+ return LogfileStream(self, name, chained_file)
+
+ def get_runner(self, name, chained_file=None):
+ return RunAndLog(self, name, chained_file)
+
+ _nonprint = ("^%" + "".join(chr(c) for c in range(0, 32) if c != 10) +
+ "".join(chr(c) for c in range(127, 256)))
+
+ def write(self, stream, data, implicit=False):
+ if stream != self.last_stream:
+ self._terminate_stream()
+ self.f.write("<div class=\"stream\" id=\"%s\">\n" % stream.name)
+ self.f.write("<div class=\"stream-header\" id=\"" + stream.name +
+ "\">Stream: " + stream.name + "</div>\n")
+ if implicit:
+ self.f.write("<span class=\"implicit\">")
+ self.f.write(self._escape(data))
+ if implicit:
+ self.f.write("</span>")
+ self.last_stream = stream
+
+ def flush(self):
+ self.f.flush()
diff --git a/test/py/pytest.ini b/test/py/pytest.ini
new file mode 100644
index 000000000000..da0d9e553a4b
--- /dev/null
+++ b/test/py/pytest.ini
@@ -0,0 +1,5 @@
+[pytest]
+markers =
+ boardspec: U-Boot: Describes the set of boards a test can/can't run on.
+ buildconfigspec: U-Boot: Describes Kconfig/config-header constraints.
+ envspec: U-Boot: Describes execution environment constraints.
diff --git a/test/py/soc_tegra124.py b/test/py/soc_tegra124.py
new file mode 100644
index 000000000000..a7427dc53261
--- /dev/null
+++ b/test/py/soc_tegra124.py
@@ -0,0 +1 @@
+ram_base = 0x80000000
diff --git a/test/py/soc_tegra20.py b/test/py/soc_tegra20.py
new file mode 100644
index 000000000000..16e3a4966c5c
--- /dev/null
+++ b/test/py/soc_tegra20.py
@@ -0,0 +1 @@
+ram_base = 0x00000000
diff --git a/test/py/test.py b/test/py/test.py
new file mode 100755
index 000000000000..b578af48eb23
--- /dev/null
+++ b/test/py/test.py
@@ -0,0 +1,12 @@
+#!/usr/bin/env python
+
+import os
+import os.path
+import sys
+
+sys.argv.pop(0)
+
+args = ["py.test", os.path.dirname(__file__)]
+args.extend(sys.argv)
+
+os.execvp("py.test", args)
diff --git a/test/py/test_000_version.py b/test/py/test_000_version.py
new file mode 100644
index 000000000000..34822fb398e5
--- /dev/null
+++ b/test/py/test_000_version.py
@@ -0,0 +1,9 @@
+# pytest runs tests the order of their module path, which is related to the
+# filename containing the test. This file is named such that it is sorted
+# first, simply as a very basic sanity check of the functionality of the U-Boot
+# command prompt.
+
+def test_version(uboot_console):
+ with uboot_console.disable_check("main_signon"):
+ response = uboot_console.run_command("version")
+ uboot_console.validate_main_signon_in_text(response)
diff --git a/test/py/test_env.py b/test/py/test_env.py
new file mode 100644
index 000000000000..16891cd6bb15
--- /dev/null
+++ b/test/py/test_env.py
@@ -0,0 +1,96 @@
+import pytest
+
+# FIXME: This might be useful for other tests;
+# perhaps refactor it into ConsoleBase or some other state object?
+class StateTestEnv(object):
+ def __init__(self, uboot_console):
+ self.uboot_console = uboot_console
+ self.get_env()
+ self.set_var = self.get_non_existent_var()
+
+ def get_env(self):
+ response = self.uboot_console.run_command("printenv")
+ self.env = {}
+ for l in response.splitlines():
+ if not "=" in l:
+ continue
+ (var, value) = l.strip().split("=")
+ self.env[var] = value
+
+ def get_existent_var(self):
+ for var in self.env:
+ return var
+
+ def get_non_existent_var(self):
+ n = 0
+ while True:
+ var = "test_env_" + str(n)
+ if var not in self.env:
+ return var
+ n += 1
+
+ at pytest.fixture(scope="module")
+def state_test_env(uboot_console):
+ return StateTestEnv(uboot_console)
+
+def unset_var(state_test_env, var):
+ state_test_env.uboot_console.run_command("setenv " + var)
+ if var in state_test_env.env:
+ del state_test_env.env[var]
+
+def set_var(state_test_env, var, value):
+ state_test_env.uboot_console.run_command("setenv " + var + " " + value)
+ state_test_env.env[var] = value
+
+def validate_empty(state_test_env, var):
+ response = state_test_env.uboot_console.run_command("echo $" + var)
+ assert response == ""
+
+def validate_set(state_test_env, var, value):
+ response = state_test_env.uboot_console.run_command("echo $" + var)
+ assert response == value
+
+def test_env_echo_exists(state_test_env):
+ """Echo a variable that exists"""
+ var = state_test_env.get_existent_var()
+ value = state_test_env.env[var]
+ validate_set(state_test_env, var, value)
+
+def test_env_echo_non_existent(state_test_env):
+ """Echo a variable that doesn't exist"""
+ var = state_test_env.set_var
+ validate_empty(state_test_env, var)
+
+def test_env_printenv_non_existent(state_test_env):
+ """Check printenv error message"""
+ var = state_test_env.set_var
+ c = state_test_env.uboot_console
+ with c.disable_check("error_notification"):
+ response = c.run_command("printenv " + var)
+ assert(response == "## Error: \"" + var + "\" not defined")
+
+def test_env_unset_non_existent(state_test_env):
+ """Unset a nonexistent variable"""
+ var = state_test_env.get_non_existent_var()
+ unset_var(state_test_env, var)
+ validate_empty(state_test_env, var)
+
+def test_env_set_non_existent(state_test_env):
+ """Set a new variable"""
+ var = state_test_env.set_var
+ value = "foo"
+ set_var(state_test_env, var, value)
+ validate_set(state_test_env, var, value)
+
+def test_env_set_existing(state_test_env):
+ """Set an existing variable"""
+ var = state_test_env.set_var
+ value = "bar"
+ set_var(state_test_env, var, value)
+ validate_set(state_test_env, var, value)
+
+def test_env_unset_existing(state_test_env):
+ """Unset a variable"""
+ var = state_test_env.set_var
+ unset_var(state_test_env, var)
+ validate_empty(state_test_env, var)
diff --git a/test/py/test_help.py b/test/py/test_help.py
new file mode 100644
index 000000000000..c2b9ace475e1
--- /dev/null
+++ b/test/py/test_help.py
@@ -0,0 +1,2 @@
+def test_help(uboot_console):
+ uboot_console.run_command("help")
diff --git a/test/py/test_md.py b/test/py/test_md.py
new file mode 100644
index 000000000000..49cdd2685234
--- /dev/null
+++ b/test/py/test_md.py
@@ -0,0 +1,12 @@
+import pytest
+
+ at pytest.mark.buildconfigspec("cmd_memory")
+def test_md(uboot_console):
+ addr = "%08x" % uboot_console.config.ram_base
+ val = "a5f09876"
+ expected_response = addr + ": " + val
+ response = uboot_console.run_command("md " + addr + " 10")
+ assert(not (expected_response in response))
+ uboot_console.run_command("mw " + addr + " " + val)
+ response = uboot_console.run_command("md " + addr + " 10")
+ assert(expected_response in response)
diff --git a/test/py/test_sandbox_exit.py b/test/py/test_sandbox_exit.py
new file mode 100644
index 000000000000..6aefa703a965
--- /dev/null
+++ b/test/py/test_sandbox_exit.py
@@ -0,0 +1,15 @@
+import pytest
+import signal
+
+ at pytest.mark.boardspec("sandbox")
+ at pytest.mark.buildconfigspec("reset")
+def test_reset(uboot_console):
+ uboot_console.run_command("reset", False)
+ assert(uboot_console.validate_exited())
+ uboot_console.ensure_spawned()
+
+ at pytest.mark.boardspec("sandbox")
+def test_ctrlc(uboot_console):
+ uboot_console.kill(signal.SIGINT)
+ assert(uboot_console.validate_exited())
+ uboot_console.ensure_spawned()
diff --git a/test/py/test_unknown_cmd.py b/test/py/test_unknown_cmd.py
new file mode 100644
index 000000000000..19ac52cc24ce
--- /dev/null
+++ b/test/py/test_unknown_cmd.py
@@ -0,0 +1,4 @@
+def test_unknown_command(uboot_console):
+ with uboot_console.disable_check("unknown_command"):
+ response = uboot_console.run_command("non_existent_cmd")
+ assert("Unknown command 'non_existent_cmd' - try 'help'" in response)
diff --git a/test/py/uboot_console_base.py b/test/py/uboot_console_base.py
new file mode 100644
index 000000000000..dfd986860e75
--- /dev/null
+++ b/test/py/uboot_console_base.py
@@ -0,0 +1,143 @@
+import multiplexed_log
+import os
+import re
+import sys
+
+pattern_uboot_spl_signon = re.compile("(U-Boot SPL \\d{4}\\.\\d{2}-[^\r\n]*)")
+pattern_uboot_main_signon = re.compile("(U-Boot \\d{4}\\.\\d{2}-[^\r\n]*)")
+pattern_stop_autoboot_prompt = re.compile("Hit any key to stop autoboot: ")
+pattern_unknown_command = re.compile("Unknown command '.*' - try 'help'")
+pattern_error_notification = re.compile("## Error: ")
+
+class ConsoleDisableCheck(object):
+ def __init__(self, console, check_type):
+ self.console = console
+ self.check_type = check_type
+
+ def __enter__(self):
+ self.console.disable_check_count[self.check_type] += 1
+
+ def __exit__(self, extype, value, traceback):
+ self.console.disable_check_count[self.check_type] -= 1
+
+class ConsoleBase(object):
+ def __init__(self, log, config):
+ self.log = log
+ self.config = config
+
+ self.logstream = self.log.get_stream("console", sys.stdout)
+
+ # Array slice removes leading/trailing quotes
+ self.prompt = self.config.buildconfig["config_sys_prompt"][1:-1]
+ self.prompt_escaped = re.escape(self.prompt)
+ self.p = None
+ self.disable_check_count = {
+ "spl_signon": 0,
+ "main_signon": 0,
+ "unknown_command": 0,
+ "error_notification": 0,
+ }
+
+ self.at_prompt = False
+
+ def close(self):
+ if self.p:
+ self.p.close()
+ self.logstream.close()
+
+ def run_command(self, cmd, wait_for_prompt=True):
+ self.ensure_spawned()
+ bad_patterns = []
+ bad_pattern_ids = []
+ if (self.disable_check_count["spl_signon"] == 0 and
+ self.uboot_spl_signon):
+ bad_patterns.append(self.uboot_spl_signon_escaped)
+ bad_pattern_ids.append("SPL signon")
+ if self.disable_check_count["main_signon"] == 0:
+ bad_patterns.append(self.uboot_main_signon_escaped)
+ bad_pattern_ids.append("U-Boot main signon")
+ if self.disable_check_count["unknown_command"] == 0:
+ bad_patterns.append(pattern_unknown_command)
+ bad_pattern_ids.append("Unknown command")
+ if self.disable_check_count["error_notification"] == 0:
+ bad_patterns.append(pattern_error_notification)
+ bad_pattern_ids.append("Error notification")
+ try:
+ if cmd:
+ self.p.send(cmd)
+ try:
+ m = self.p.expect([re.escape(cmd)] + bad_patterns)
+ if m != 0:
+ self.at_prompt = False
+ raise Exception("Bad pattern found on console: " +
+ bad_pattern_ids[m - 1])
+ except Exception as ex:
+ self.at_prompt = False
+ print cmd
+ self.logstream.write(cmd, implicit=True)
+ raise
+ self.p.send("\n")
+ if not wait_for_prompt:
+ self.at_prompt = False
+ return
+ m = self.p.expect([self.prompt_escaped] + bad_patterns)
+ if m != 0:
+ self.at_prompt = False
+ raise Exception("Bad pattern found on console: " +
+ bad_pattern_ids[m - 1])
+ self.at_prompt = True
+ return self.p.before.strip()
+ except Exception as ex:
+ self.at_prompt = False
+ self.log.error(str(ex))
+ self.cleanup_spawn()
+ raise
+
+ def ensure_spawned(self):
+ if self.p:
+ return
+ try:
+ self.at_prompt = False
+ self.log.action("Starting U-Boot")
+ self.p = self.get_spawn()
+ # Real targets can take a long time to scroll large amounts of
+ # text if LCD is enabled. This value may need tweaking in the
+ # future, possibly per-test to be optimal. This works for "help"
+ # on board "seaboard".
+ self.p.timeout = 30
+ self.p.logfile_read = self.logstream
+ if self.config.buildconfig.get("CONFIG_SPL", False) == "y":
+ self.p.expect(pattern_uboot_spl_signon)
+ self.uboot_spl_signon = self.p.after
+ self.uboot_spl_signon_escaped = re.escape(self.p.after)
+ else:
+ self.uboot_spl_signon = None
+ self.p.expect(pattern_uboot_main_signon)
+ self.uboot_main_signon = self.p.after
+ self.uboot_main_signon_escaped = re.escape(self.p.after)
+ while True:
+ match = self.p.expect([self.prompt_escaped,
+ pattern_stop_autoboot_prompt])
+ if match == 1:
+ self.p.send(chr(3)) # CTRL-C
+ continue
+ break
+ self.at_prompt = True
+ except Exception as ex:
+ self.log.error(str(ex))
+ self.cleanup_spawn()
+ raise
+
+ def cleanup_spawn(self):
+ try:
+ if self.p:
+ self.p.close()
+ except:
+ pass
+ self.p = None
+
+ def validate_main_signon_in_text(self, text):
+ assert(self.uboot_main_signon in text)
+
+ def disable_check(self, check_type):
+ return ConsoleDisableCheck(self, check_type)
diff --git a/test/py/uboot_console_exec_attach.py b/test/py/uboot_console_exec_attach.py
new file mode 100644
index 000000000000..7960d66107c3
--- /dev/null
+++ b/test/py/uboot_console_exec_attach.py
@@ -0,0 +1,28 @@
+import os
+import pexpect
+from uboot_console_base import ConsoleBase
+
+def cmdline(app, args):
+ return app + ' "' + '" "'.join(args) + '"'
+
+class ConsoleExecAttach(ConsoleBase):
+ def __init__(self, log, config):
+ super(ConsoleExecAttach, self).__init__(log, config)
+
+ self.log.action("Flashing U-Boot")
+ cmd = ["uboot-test-flash", config.board_type, config.board_identity]
+ runner = self.log.get_runner(cmd[0])
+ runner.run(cmd)
+ runner.close()
+
+ def get_spawn(self):
+ args = [self.config.board_type, self.config.board_identity]
+ s = pexpect.spawn("uboot-test-console", args=args)
+
+ self.log.action("Resetting board")
+ cmd = ["uboot-test-reset"] + args
+ runner = self.log.get_runner(cmd[0])
+ runner.run(cmd)
+ runner.close()
+
+ return s
diff --git a/test/py/uboot_console_sandbox.py b/test/py/uboot_console_sandbox.py
new file mode 100644
index 000000000000..c3aae3862ca9
--- /dev/null
+++ b/test/py/uboot_console_sandbox.py
@@ -0,0 +1,22 @@
+import os
+import pexpect
+from uboot_console_base import ConsoleBase
+
+class ConsoleSandbox(ConsoleBase):
+ def __init__(self, log, config):
+ super(ConsoleSandbox, self).__init__(log, config)
+
+ def get_spawn(self):
+ return pexpect.spawn(self.config.build_dir + "/u-boot")
+
+ def kill(self, sig):
+ self.ensure_spawned()
+ self.log.action("kill %d" % sig)
+ self.p.kill(sig)
+
+ def validate_exited(self):
+ p = self.p
+ self.p = None
+ ret = p.isalive()
+ p.close()
+ return ret
--
1.9.1
next reply other threads:[~2015-11-15 6:53 UTC|newest]
Thread overview: 19+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-11-15 6:53 Stephen Warren [this message]
2015-11-19 14:45 ` [U-Boot] [PATCH] Implement pytest-based test infrastructure Simon Glass
2015-11-19 17:00 ` Stephen Warren
2015-11-19 19:09 ` Stephen Warren
2015-11-21 16:49 ` Simon Glass
2015-11-22 17:30 ` Stephen Warren
2015-11-24 1:45 ` Simon Glass
2015-11-24 2:18 ` Simon Glass
2015-11-24 4:24 ` Stephen Warren
2015-11-24 4:44 ` Stephen Warren
2015-11-24 19:04 ` Simon Glass
2015-11-24 21:28 ` Stephen Warren
2015-11-27 2:52 ` Simon Glass
2015-11-30 17:13 ` Stephen Warren
2015-12-01 16:40 ` Simon Glass
2015-12-01 23:24 ` Stephen Warren
2015-12-02 13:37 ` Simon Glass
2015-11-23 23:44 ` Tom Rini
2015-11-23 23:55 ` Stephen Warren
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1447570381-1361-1-git-send-email-swarren@wwwdotorg.org \
--to=swarren@wwwdotorg.org \
--cc=u-boot@lists.denx.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.