* [U-Boot] [PATCH V3 1/7] test/py: Implement pytest infrastructure
@ 2016-01-05 22:58 Stephen Warren
2016-01-05 22:58 ` [U-Boot] [PATCH V3 2/7] test/py: test that sandbox exits when asked Stephen Warren
` (7 more replies)
0 siblings, 8 replies; 20+ messages in thread
From: Stephen Warren @ 2016-01-05 22:58 UTC (permalink / raw)
To: u-boot
This tool aims to test U-Boot by executing U-Boot shell commands using the
console interface. A single top-level script exists to execute or attach
to the U-Boot console, run the entire script of tests against it, and
summarize the results. Advantages of this approach are:
- Testing is performed in the same way a user or script would interact
with U-Boot; there can be no disconnect.
- There is no need to write or embed test-related code into U-Boot itself.
It is asserted that writing test-related code in Python is simpler and
more flexible that writing it all in C.
- It is reasonably simple to interact with U-Boot in this way.
A few simple tests are provided as examples. Soon, we should convert as
many as possible of the other tests in test/* and test/cmd_ut.c too.
The hook scripts, relay control utilities, and udev rules I use for my
own HW setup are published at https://github.com/swarren/uboot-test-hooks.
See README.md for more details!
Signed-off-by: Stephen Warren <swarren@wwwdotorg.org>
Signed-off-by: Stephen Warren <swarren@nvidia.com>
Tested-by: Michal Simek <michal.simek@xilinx.com>
Tested-by: Simon Glass <sjg@chromium.org>
---
v3:
- Rework HTML log generation so that TAB characters render as expected.
Suggested by Michal Simek.
- Move test scripts into a sub-directory.
Suggested by Michal Simek.
- s/uboot/u[-_]boot/g. Suggested by Simon Glass.
- s/"/'/g. Suggested by Simon Glass.
- Typo fixes.
- Add more documentation. Suggested by Simon Glass.
- Make "notes" in the log file be <pre> so that their formatting is
preserved. This is useful for large notes such as exception dumps.
v2:
- Many fixes and tweaks have been squashed in. Separated out some of
the tests into separate commits, and added some more tests.
---
test/py/.gitignore | 1 +
test/py/README.md | 300 ++++++++++++++++++++++++++++++
test/py/conftest.py | 335 ++++++++++++++++++++++++++++++++++
test/py/multiplexed_log.css | 88 +++++++++
test/py/multiplexed_log.py | 335 ++++++++++++++++++++++++++++++++++
test/py/pytest.ini | 11 ++
test/py/test.py | 32 ++++
test/py/tests/test_000_version.py | 20 ++
test/py/tests/test_help.py | 9 +
test/py/tests/test_unknown_cmd.py | 14 ++
test/py/u_boot_console_base.py | 260 ++++++++++++++++++++++++++
test/py/u_boot_console_exec_attach.py | 51 ++++++
test/py/u_boot_console_sandbox.py | 48 +++++
test/py/u_boot_spawn.py | 123 +++++++++++++
14 files changed, 1627 insertions(+)
create mode 100644 test/py/.gitignore
create mode 100644 test/py/README.md
create mode 100644 test/py/conftest.py
create mode 100644 test/py/multiplexed_log.css
create mode 100644 test/py/multiplexed_log.py
create mode 100644 test/py/pytest.ini
create mode 100755 test/py/test.py
create mode 100644 test/py/tests/test_000_version.py
create mode 100644 test/py/tests/test_help.py
create mode 100644 test/py/tests/test_unknown_cmd.py
create mode 100644 test/py/u_boot_console_base.py
create mode 100644 test/py/u_boot_console_exec_attach.py
create mode 100644 test/py/u_boot_console_sandbox.py
create mode 100644 test/py/u_boot_spawn.py
diff --git a/test/py/.gitignore b/test/py/.gitignore
new file mode 100644
index 000000000000..0d20b6487c61
--- /dev/null
+++ b/test/py/.gitignore
@@ -0,0 +1 @@
+*.pyc
diff --git a/test/py/README.md b/test/py/README.md
new file mode 100644
index 000000000000..7c9bb4f04c24
--- /dev/null
+++ b/test/py/README.md
@@ -0,0 +1,300 @@
+# U-Boot pytest suite
+
+## Introduction
+
+This tool aims to test U-Boot by executing U-Boot shell commands using the
+console interface. A single top-level script exists to execute or attach to the
+U-Boot console, run the entire script of tests against it, and summarize the
+results. Advantages of this approach are:
+
+- Testing is performed in the same way a user or script would interact with
+ U-Boot; there can be no disconnect.
+- There is no need to write or embed test-related code into U-Boot itself.
+ It is asserted that writing test-related code in Python is simpler and more
+ flexible that writing it all in C.
+- It is reasonably simple to interact with U-Boot in this way.
+
+## Requirements
+
+The test suite is implemented using pytest. Interaction with the U-Boot console
+involves executing some binary and interacting with its stdin/stdout. You will
+need to implement various "hook" scripts that are called by the test suite at
+the appropriate time.
+
+On Debian or Debian-like distributions, the following packages are required.
+Similar package names should exist in other distributions.
+
+| Package | Version tested (Ubuntu 14.04) |
+| -------------- | ----------------------------- |
+| python | 2.7.5-5ubuntu3 |
+| python-pytest | 2.5.1-1 |
+
+The test script supports either:
+
+- Executing a sandbox port of U-Boot on the local machine as a sub-process,
+ and interacting with it over stdin/stdout.
+- Executing an external "hook" scripts to flash a U-Boot binary onto a
+ physical board, attach to the board's console stream, and reset the board.
+ Further details are described later.
+
+### Using `virtualenv` to provide requirements
+
+Older distributions (e.g. Ubuntu 10.04) may not provide all the required
+packages, or may provide versions that are too old to run the test suite. One
+can use the Python `virtualenv` script to locally install more up-to-date
+versions of the required packages without interfering with the OS installation.
+For example:
+
+```bash
+$ cd /path/to/u-boot
+$ sudo apt-get install python python-virtualenv
+$ virtualenv venv
+$ . ./venv/bin/activate
+$ pip install pytest
+```
+
+## Testing sandbox
+
+To run the testsuite on the sandbox port (U-Boot built as a native user-space
+application), simply execute:
+
+```
+./test/py/test.py --bd sandbox --build
+```
+
+The `--bd` option tells the test suite which board type is being tested. This
+lets the test suite know which features the board has, and hence exactly what
+can be tested.
+
+The `--build` option tells U-Boot to compile U-Boot. Alternatively, you may
+omit this option and build U-Boot yourself, in whatever way you choose, before
+running the test script.
+
+The test script will attach to U-Boot, execute all valid tests for the board,
+then print a summary of the test process. A complete log of the test session
+will be written to `${build_dir}/test-log.html`. This is best viewed in a web
+browser, but may be read directly as plain text, perhaps with the aid of the
+`html2text` utility.
+
+## Command-line options
+
+- `--board-type`, `--bd`, `-B` set the type of the board to be tested. For
+ example, `sandbox` or `seaboard`.
+- `--board-identity`, `--id` set the identity of the board to be tested.
+ This allows differentiation between multiple instances of the same type of
+ physical board that are attached to the same host machine. This parameter is
+ not interpreted by the test script in any way, but rather is simply passed
+ to the hook scripts described below, and may be used in any site-specific
+ way deemed necessary.
+- `--build` indicates that the test script should compile U-Boot itself
+ before running the tests. If using this option, make sure that any
+ environment variables required by the build process are already set, such as
+ `$CROSS_COMPILE`.
+- `--build-dir` sets the directory containing the compiled U-Boot binaries.
+ If omitted, this is `${source_dir}/build-${board_type}`.
+- `--result-dir` sets the directory to write results, such as log files,
+ into. If omitted, the build directory is used.
+- `--persistent-data-dir` sets the directory used to store persistent test
+ data. This is test data that may be re-used across test runs, such as file-
+ system images.
+
+`pytest` also implements a number of its own command-line options. Please see
+`pytest` documentation for complete details. Execute `py.test --version` for
+a brief summary. Note that U-Boot's test.py script passes all command-line
+arguments directly to `pytest` for processing.
+
+## Testing real hardware
+
+The tools and techniques used to interact with real hardware will vary
+radically between different host and target systems, and the whims of the user.
+For this reason, the test suite does not attempt to directly interact with real
+hardware in any way. Rather, it executes a standardized set of "hook" scripts
+via `$PATH`. These scripts implement certain actions on behalf of the test
+suite. This keeps the test suite simple and isolated from system variances
+unrelated to U-Boot features.
+
+### Hook scripts
+
+#### Environment variables
+
+The following environment variables are set when running hook scripts:
+
+- `UBOOT_BOARD_TYPE` the board type being tested.
+- `UBOOT_BOARD_IDENTITY` the board identity being tested, or `na` if none was
+ specified.
+- `UBOOT_SOURCE_DIR` the U-Boot source directory.
+- `UBOOT_TEST_PY_DIR` the full path to `test/py/` in the source directory.
+- `UBOOT_BUILD_DIR` the U-Boot build directory.
+- `UBOOT_RESULT_DIR` the test result directory.
+- `UBOOT_PERSISTENT_DATA_DIR` the test peristent data directory.
+
+#### `u-boot-test-console`
+
+This script provides access to the U-Boot console. The script's stdin/stdout
+should be connected to the board's console. This process should continue to run
+indefinitely, until killed. The test suite will run this script in parallel
+with all other hooks.
+
+This script may be implemented e.g. by exec()ing `cu`, `conmux`, etc.
+
+If you are able to run U-Boot under a hardware simulator such as qemu, then
+you would likely spawn that simulator from this script. However, note that
+`u-boot-test-reset` may be called multiple times per test script run, and must
+cause U-Boot to start execution from scratch each time. Hopefully your
+simulator includes a virtual reset button! If not, you can launch the
+simulator from `u-boot-test-reset` instead, while arranging for this console
+process to always communicate with the current simulator instance.
+
+#### `u-boot-test-flash`
+
+Prior to running the test suite against a board, some arrangement must be made
+so that the board executes the particular U-Boot binary to be tested. Often,
+this involves writing the U-Boot binary to the board's flash ROM. The test
+suite calls this hook script for that purpose.
+
+This script should perform the entire flashing process synchronously; the
+script should only exit once flashing is complete, and a board reset will
+cause the newly flashed U-Boot binary to be executed.
+
+It is conceivable that this script will do nothing. This might be useful in
+the following cases:
+
+- Some other process has already written the desired U-Boot binary into the
+ board's flash prior to running the test suite.
+- The board allows U-Boot to be downloaded directly into RAM, and executed
+ from there. Use of this feature will reduce wear on the board's flash, so
+ may be preferable if available, and if cold boot testing of U-Boot is not
+ required. If this feature is used, the `u-boot-test-reset` script should
+ peform this download, since the board could conceivably be reset multiple
+ times in a single test run.
+
+It is up to the user to determine if those situations exist, and to code this
+hook script appropriately.
+
+This script will typically be implemented by calling out to some SoC- or
+board-specific vendor flashing utility.
+
+#### `u-boot-test-reset`
+
+Whenever the test suite needs to reset the target board, this script is
+executed. This is guaranteed to happen at least once, prior to executing the
+first test function. If any test fails, the test infra-structure will execute
+this script again to restore U-Boot to an operational state before running the
+next test function.
+
+This script will likely be implemented by communicating with some form of
+relay or electronic switch attached to the board's reset signal.
+
+The semantics of this script require that when it is executed, U-Boot will
+start running from scratch. If the U-Boot binary to be tested has been written
+to flash, pulsing the board's reset signal is likely all this script need do.
+However, in some scenarios, this script may perform other actions. For
+example, it may call out to some SoC- or board-specific vendor utility in order
+to download the U-Boot binary directly into RAM and execute it. This would
+avoid the need for `u-boot-test-flash` to actually write U-Boot to flash, thus
+saving wear on the flash chip(s).
+
+### Board-type-specific configuration
+
+Each board has a different configuration and behaviour. Many of these
+differences can be automatically detected by parsing the `.config` file in the
+build directory. However, some differences can't yet be handled automatically.
+
+For each board, an optional Python module `u_boot_board_${board_type}` may exist
+to provide board-specific information to the test script. Any global value
+defined in these modules is available for use by any test function. The data
+contained in these scripts must be purely derived from U-Boot source code.
+Hence, these configuration files are part of the U-Boot source tree too.
+
+### Execution environment configuration
+
+Each user's hardware setup may enable testing different subsets of the features
+implemented by a particular board's configuration of U-Boot. For example, a
+U-Boot configuration may support USB device mode and USB Mass Storage, but this
+can only be tested if a USB cable is connected between the board and the host
+machine running the test script.
+
+For each board, optional Python modules `u_boot_boardenv_${board_type}` and
+`u_boot_boardenv_${board_type}_${board_identity}` may exist to provide
+board-specific and board-identity-specific information to the test script. Any
+global value defined in these modules is available for use by any test
+function. The data contained in these is specific to a particular user's
+hardware configuration. Hence, these configuration files are not part of the
+U-Boot source tree, and should be installed outside of the source tree. Users
+should set `$PYTHONPATH` prior to running the test script to allow these
+modules to be loaded.
+
+### Board module parameter usage
+
+The test scripts rely on the following variables being defined by the board
+module:
+
+- None at present.
+
+### U-Boot `.config` feature usage
+
+The test scripts rely on various U-Boot `.config` features, either directly in
+order to test those features, or indirectly in order to query information from
+the running U-Boot instance in order to test other features.
+
+One example is that testing of the `md` command requires knowledge of a RAM
+address to use for the test. This data is parsed from the output of the
+`bdinfo` command, and hence relies on CONFIG_CMD_BDI being enabled.
+
+For a complete list of dependencies, please search the test scripts for
+instances of:
+
+- `buildconfig.get(...`
+- `@pytest.mark.buildconfigspec(...`
+
+### Complete invocation example
+
+Assuming that you have installed the hook scripts into $HOME/ubtest/bin, and
+any required environment configuration Python modules into $HOME/ubtest/py,
+then you would likely invoke the test script as follows:
+
+If U-Boot has already been built:
+
+```bash
+PATH=$HOME/ubtest/bin:$PATH \
+ PYTHONPATH=${HOME}/ubtest/py:${PYTHONPATH} \
+ ./test/py/test.py --bd seaboard
+```
+
+If you want the test script to compile U-Boot for you too, then you likely
+need to set `$CROSS_COMPILE` to allow this, and invoke the test script as
+follow:
+
+```bash
+CROSS_COMPILE=arm-none-eabi- \
+ PATH=$HOME/ubtest/bin:$PATH \
+ PYTHONPATH=${HOME}/ubtest/py:${PYTHONPATH} \
+ ./test/py/test.py --bd seaboard --build
+```
+
+## Writing tests
+
+Please refer to the pytest documentation for details of writing pytest tests.
+Details specific to the U-Boot test suite are described below.
+
+A test fixture named `u_boot_console` should be used by each test function. This
+provides the means to interact with the U-Boot console, and retrieve board and
+environment configuration information.
+
+The function `u_boot_console.run_command()` executes a shell command on the
+U-Boot console, and returns all output from that command. This allows
+validation or interpretation of the command output. This function validates
+that certain strings are not seen on the U-Boot console. These include shell
+error messages and the U-Boot sign-on message (in order to detect unexpected
+board resets). See the source of `u_boot_console_base.py` for a complete list of
+"bad" strings. Some test scenarios are expected to trigger these strings. Use
+`u_boot_console.disable_check()` to temporarily disable checking for specific
+strings. See `test_unknown_cmd.py` for an example.
+
+Board- and board-environment configuration values may be accessed as sub-fields
+of the `u_boot_console.config` object, for example
+`u_boot_console.config.ram_base`.
+
+Build configuration values (from `.config`) may be accessed via the dictionary
+`u_boot_console.config.buildconfig`, with keys equal to the Kconfig variable
+names.
diff --git a/test/py/conftest.py b/test/py/conftest.py
new file mode 100644
index 000000000000..3b905ed86f1c
--- /dev/null
+++ b/test/py/conftest.py
@@ -0,0 +1,335 @@
+# Copyright (c) 2015 Stephen Warren
+# Copyright (c) 2015-2016, NVIDIA CORPORATION. All rights reserved.
+#
+# SPDX-License-Identifier: GPL-2.0
+
+# Implementation of pytest run-time hook functions. These are invoked by
+# pytest at certain points during operation, e.g. startup, for each executed
+# test, at shutdown etc. These hooks perform functions such as:
+# - Parsing custom command-line options.
+# - Pullilng in user-specified board configuration.
+# - Creating the U-Boot console test fixture.
+# - Creating the HTML log file.
+# - Monitoring each test's results.
+# - Implementing custom pytest markers.
+
+import atexit
+import errno
+import os
+import os.path
+import pexpect
+import pytest
+from _pytest.runner import runtestprotocol
+import ConfigParser
+import StringIO
+import sys
+
+# Globals: The HTML log file, and the connection to the U-Boot console.
+log = None
+console = None
+
+def mkdir_p(path):
+ '''Create a directory path, including parent directories, ignoring errors
+ due to already extant directories.'''
+
+ try:
+ os.makedirs(path)
+ except OSError as exc:
+ if exc.errno == errno.EEXIST and os.path.isdir(path):
+ pass
+ else:
+ raise
+
+def pytest_addoption(parser):
+ '''pytest hook: Add custom command-line options to the cmdline parser.'''
+
+ parser.addoption('--build-dir', default=None,
+ help='U-Boot build directory (O=)')
+ parser.addoption('--result-dir', default=None,
+ help='U-Boot test result/tmp directory')
+ parser.addoption('--persistent-data-dir', default=None,
+ help='U-Boot test persistent generated data directory')
+ parser.addoption('--board-type', '--bd', '-B', default='sandbox',
+ help='U-Boot board type')
+ parser.addoption('--board-identity', '--id', default='na',
+ help='U-Boot board identity/instance')
+ parser.addoption('--build', default=False, action='store_true',
+ help='Compile U-Boot before running tests')
+
+def pytest_configure(config):
+ '''pytest hook: Perform custom initialization at startup time.'''
+
+ global log
+ global console
+ global ubconfig
+
+ test_py_dir = os.path.dirname(os.path.abspath(__file__))
+ source_dir = os.path.dirname(os.path.dirname(test_py_dir))
+
+ board_type = config.getoption('board_type')
+ board_type_filename = board_type.replace('-', '_')
+
+ board_identity = config.getoption('board_identity')
+ board_identity_filename = board_identity.replace('-', '_')
+
+ build_dir = config.getoption('build_dir')
+ if not build_dir:
+ build_dir = source_dir + '/build-' + board_type
+ mkdir_p(build_dir)
+
+ result_dir = config.getoption('result_dir')
+ if not result_dir:
+ result_dir = build_dir
+ mkdir_p(result_dir)
+
+ persistent_data_dir = config.getoption('persistent_data_dir')
+ if not persistent_data_dir:
+ persistent_data_dir = build_dir + '/persistent-data'
+ mkdir_p(persistent_data_dir)
+
+ import multiplexed_log
+ log = multiplexed_log.Logfile(result_dir + '/test-log.html')
+
+ if config.getoption('build'):
+ if build_dir != source_dir:
+ o_opt = 'O=%s' % build_dir
+ else:
+ o_opt = ''
+ cmds = (
+ ['make', o_opt, '-s', board_type + '_defconfig'],
+ ['make', o_opt, '-s', '-j8'],
+ )
+ runner = log.get_runner('make', sys.stdout)
+ for cmd in cmds:
+ runner.run(cmd, cwd=source_dir)
+ runner.close()
+
+ class ArbitraryAttributeContainer(object):
+ pass
+
+ ubconfig = ArbitraryAttributeContainer()
+ ubconfig.brd = dict()
+ ubconfig.env = dict()
+
+ modules = [
+ (ubconfig.brd, 'u_boot_board_' + board_type_filename),
+ (ubconfig.env, 'u_boot_boardenv_' + board_type_filename),
+ (ubconfig.env, 'u_boot_boardenv_' + board_type_filename + '_' +
+ board_identity_filename),
+ ]
+ for (dict_to_fill, module_name) in modules:
+ try:
+ module = __import__(module_name)
+ except ImportError:
+ continue
+ dict_to_fill.update(module.__dict__)
+
+ ubconfig.buildconfig = dict()
+
+ for conf_file in ('.config', 'include/autoconf.mk'):
+ dot_config = build_dir + '/' + conf_file
+ if not os.path.exists(dot_config):
+ raise Exception(conf_file + ' does not exist; ' +
+ 'try passing --build option?')
+
+ with open(dot_config, 'rt') as f:
+ ini_str = '[root]\n' + f.read()
+ ini_sio = StringIO.StringIO(ini_str)
+ parser = ConfigParser.RawConfigParser()
+ parser.readfp(ini_sio)
+ ubconfig.buildconfig.update(parser.items('root'))
+
+ ubconfig.test_py_dir = test_py_dir
+ ubconfig.source_dir = source_dir
+ ubconfig.build_dir = build_dir
+ ubconfig.result_dir = result_dir
+ ubconfig.persistent_data_dir = persistent_data_dir
+ ubconfig.board_type = board_type
+ ubconfig.board_identity = board_identity
+
+ env_vars = (
+ 'board_type',
+ 'board_identity',
+ 'source_dir',
+ 'test_py_dir',
+ 'build_dir',
+ 'result_dir',
+ 'persistent_data_dir',
+ )
+ for v in env_vars:
+ os.environ['U_BOOT_' + v.upper()] = getattr(ubconfig, v)
+
+ if board_type == 'sandbox':
+ import u_boot_console_sandbox
+ console = u_boot_console_sandbox.ConsoleSandbox(log, ubconfig)
+ else:
+ import u_boot_console_exec_attach
+ console = u_boot_console_exec_attach.ConsoleExecAttach(log, ubconfig)
+
+def pytest_generate_tests(metafunc):
+ '''pytest hook: parameterize test functions based on custom rules.
+
+ If a test function takes parameter(s) (fixture names) of the form brd__xxx
+ or env__xxx, the brd and env configuration dictionaries are consulted to
+ find the list of values to use for those parameters, and the test is
+ parametrized so that it runs once for each combination of values.'''
+
+ subconfigs = {
+ 'brd': console.config.brd,
+ 'env': console.config.env,
+ }
+ for fn in metafunc.fixturenames:
+ parts = fn.split('__')
+ if len(parts) < 2:
+ continue
+ if parts[0] not in subconfigs:
+ continue
+ subconfig = subconfigs[parts[0]]
+ vals = []
+ val = subconfig.get(fn, [])
+ # If that exact name is a key in the data source:
+ if val:
+ # ... use the dict value as a single parameter value.
+ vals = (val, )
+ else:
+ # ... otherwise, see if there's a key that contains a list of
+ # values to use instead.
+ vals = subconfig.get(fn + 's', [])
+ metafunc.parametrize(fn, vals)
+
+@pytest.fixture(scope='session')
+def u_boot_console(request):
+ '''Returns the value of the u_boot_console test function parameter
+ (fixture).'''
+
+ return console
+
+tests_not_run = set()
+tests_failed = set()
+tests_skipped = set()
+tests_passed = set()
+
+def pytest_itemcollected(item):
+ '''pytest hook: Called once for each test invocation found during
+ collection. This enables our custom result analysis code to see the list
+ of all tests that should eventually be run.'''
+
+ tests_not_run.add(item.name)
+
+def cleanup():
+ '''Clean up all global state. Executed (via atexit) once the entire test
+ process is complete. This includes logging the status of all tests, and
+ the identity of any failed or skipped tests.'''
+
+ if console:
+ console.close()
+ if log:
+ log.status_pass('%d passed' % len(tests_passed))
+ if tests_skipped:
+ log.status_skipped('%d skipped' % len(tests_skipped))
+ for test in tests_skipped:
+ log.status_skipped('... ' + test)
+ if tests_failed:
+ log.status_fail('%d failed' % len(tests_failed))
+ for test in tests_failed:
+ log.status_fail('... ' + test)
+ if tests_not_run:
+ log.status_fail('%d not run' % len(tests_not_run))
+ for test in tests_not_run:
+ log.status_fail('... ' + test)
+ log.close()
+atexit.register(cleanup)
+
+def setup_boardspec(item):
+ '''Process any 'boardspec' marker for a test. Such a marker lists the set
+ of board types that a test does/doesn't support. If tests are being
+ executed on an unsupported board, the test is marked to be skipped.'''
+
+ mark = item.get_marker('boardspec')
+ if not mark:
+ return
+ required_boards = []
+ for board in mark.args:
+ if board.startswith('!'):
+ if ubconfig.board_type == board[1:]:
+ pytest.skip('board not supported')
+ return
+ else:
+ required_boards.append(board)
+ if required_boards and ubconfig.board_type not in required_boards:
+ pytest.skip('board not supported')
+
+def setup_buildconfigspec(item):
+ '''Process any 'buildconfigspec' marker for a test. Such a marker lists
+ some U-Boot configuration feature that the test requires. If tests are
+ being executed on an U-Boot build that doesn't have the required feature,
+ the test is marked to be skipped.'''
+
+ mark = item.get_marker('buildconfigspec')
+ if not mark:
+ return
+ for option in mark.args:
+ if not ubconfig.buildconfig.get('config_' + option.lower(), None):
+ pytest.skip('.config feature not enabled')
+
+def pytest_runtest_setup(item):
+ '''pytest hook: Called once for each test to perform any custom
+ configuration. This hook is used to skip the test if certain conditions
+ apply.'''
+
+ log.start_section(item.name)
+ setup_boardspec(item)
+ setup_buildconfigspec(item)
+
+def pytest_runtest_protocol(item, nextitem):
+ '''pytest hook: Called to execute a test. This hook wraps the standard
+ pytest runtestprotocol() function in order to acquire visibility into, and
+ record, each test function's result.'''
+
+ reports = runtestprotocol(item, nextitem=nextitem)
+ failed = None
+ skipped = None
+ for report in reports:
+ if report.outcome == 'failed':
+ failed = report
+ break
+ if report.outcome == 'skipped':
+ if not skipped:
+ skipped = report
+
+ if failed:
+ tests_failed.add(item.name)
+ elif skipped:
+ tests_skipped.add(item.name)
+ else:
+ tests_passed.add(item.name)
+ tests_not_run.remove(item.name)
+
+ try:
+ if failed:
+ msg = 'FAILED:\n' + str(failed.longrepr)
+ log.status_fail(msg)
+ elif skipped:
+ msg = 'SKIPPED:\n' + str(skipped.longrepr)
+ log.status_skipped(msg)
+ else:
+ log.status_pass('OK')
+ except:
+ # If something went wrong with logging, it's better to let the test
+ # process continue, which may report other exceptions that triggered
+ # the logging issue (e.g. console.log wasn't created). Hence, just
+ # squash the exception. If the test setup failed due to e.g. syntax
+ # error somewhere else, this won't be seen. However, once that issue
+ # is fixed, if this exception still exists, it will then be logged as
+ # part of the test's stdout.
+ import traceback
+ print 'Exception occurred while logging runtest status:'
+ traceback.print_exc()
+ # FIXME: Can we force a test failure here?
+
+ log.end_section(item.name)
+
+ if failed:
+ console.cleanup_spawn()
+
+ return reports
diff --git a/test/py/multiplexed_log.css b/test/py/multiplexed_log.css
new file mode 100644
index 000000000000..50f7b9092983
--- /dev/null
+++ b/test/py/multiplexed_log.css
@@ -0,0 +1,88 @@
+/*
+ * Copyright (c) 2015 Stephen Warren
+ * Copyright (c) 2016, NVIDIA CORPORATION. All rights reserved.
+ *
+ * SPDX-License-Identifier: GPL-2.0
+ */
+
+/*
+ * This provides pretty formatting of the HTML log file, e.g.
+ * - colored bars beside/above log sections for easily parsed delineation.
+ * - color highlighting of various messages.
+ */
+
+body {
+ background-color: black;
+ color: #ffffff;
+}
+
+pre {
+ margin-top: 0px;
+ margin-bottom: 0px;
+}
+
+.implicit {
+ color: #808080;
+}
+
+.section {
+ border-style: solid;
+ border-color: #303030;
+ border-width: 0px 0px 0px 5px;
+ padding-left: 5px
+}
+
+.section-header {
+ background-color: #303030;
+ margin-left: -5px;
+ margin-top: 5px;
+}
+
+.section-trailer {
+ display: none;
+}
+
+.stream {
+ border-style: solid;
+ border-color: #303030;
+ border-width: 0px 0px 0px 5px;
+ padding-left: 5px
+}
+
+.stream-header {
+ background-color: #303030;
+ margin-left: -5px;
+ margin-top: 5px;
+}
+
+.stream-trailer {
+ display: none;
+}
+
+.error {
+ color: #ff0000
+}
+
+.warning {
+ color: #ffff00
+}
+
+.info {
+ color: #808080
+}
+
+.action {
+ color: #8080ff
+}
+
+.status-pass {
+ color: #00ff00
+}
+
+.status-skipped {
+ color: #ffff00
+}
+
+.status-fail {
+ color: #ff0000
+}
diff --git a/test/py/multiplexed_log.py b/test/py/multiplexed_log.py
new file mode 100644
index 000000000000..0b8f5cfa4e1b
--- /dev/null
+++ b/test/py/multiplexed_log.py
@@ -0,0 +1,335 @@
+# Copyright (c) 2015 Stephen Warren
+# Copyright (c) 2015-2016, NVIDIA CORPORATION. All rights reserved.
+#
+# SPDX-License-Identifier: GPL-2.0
+
+# Generate an HTML-formatted log file containing multiple streams of data,
+# each represented in a well-delineated/-structured fashion.
+
+import cgi
+import os.path
+import shutil
+import subprocess
+
+mod_dir = os.path.dirname(os.path.abspath(__file__))
+
+class LogfileStream(object):
+ '''A file-like object used to write a single logical stream of data into
+ a multiplexed log file. Objects of this type should be created by factory
+ functions in the Logfile class rather than directly.'''
+
+ def __init__(self, logfile, name, chained_file):
+ '''Initialize a new object.
+
+ logfile: The Logfile object to log to.
+ name: The name of this log stream.
+ chained_file: The file-like object to which all stream data should be
+ logged to in addition to logfile. Can be None.'''
+
+ self.logfile = logfile
+ self.name = name
+ self.chained_file = chained_file
+
+ def close(self):
+ '''Dummy function so that this class is "file-like".'''
+
+ pass
+
+ def write(self, data, implicit=False):
+ '''Write data to the log stream.'''
+
+ self.logfile.write(self, data, implicit)
+ if self.chained_file:
+ self.chained_file.write(data)
+
+ def flush(self):
+ '''Flush the log stream, to ensure correct log interleaving.'''
+
+ self.logfile.flush()
+ if self.chained_file:
+ self.chained_file.flush()
+
+class RunAndLog(object):
+ '''A utility object used to execute sub-processes and log their output to
+ a multiplexed log file. Objects of this type should be created by factory
+ functions in the Logfile class rather than directly.'''
+
+ def __init__(self, logfile, name, chained_file):
+ '''Initialize a new object.
+
+ logfile: The Logfile object to log to.
+ name: The name of this log stream or sub-process.
+ chained_file: The file-like object to which all stream data should be
+ logged to in addition to logfile. Can be None.'''
+
+ self.logfile = logfile
+ self.name = name
+ self.chained_file = chained_file
+
+ def close(self):
+ '''Clean up any resources managed by this object.'''
+ pass
+
+ def run(self, cmd, cwd=None):
+ '''run a command as a sub-process, and log the results.
+
+ cmd: The command to execute.
+ cwd: The directory to run the command in. Can be None to use the
+ current directory.'''
+
+ msg = "+" + " ".join(cmd) + "\n"
+ if self.chained_file:
+ self.chained_file.write(msg)
+ self.logfile.write(self, msg)
+
+ try:
+ p = subprocess.Popen(cmd, cwd=cwd,
+ stdin=None, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
+ (output, stderr) = p.communicate()
+ status = p.returncode
+ except subprocess.CalledProcessError as cpe:
+ output = cpe.output
+ status = cpe.returncode
+ self.logfile.write(self, output)
+ if status:
+ if self.chained_file:
+ self.chained_file.write(output)
+ raise Exception("command failed; exit code " + str(status))
+
+class SectionCtxMgr(object):
+ '''A context manager for Python's "with" statement, which allows a certain
+ portion of test code to be logged to a separate section of the log file.
+ Objects of this type should be created by factory functions in the Logfile
+ class rather than directly.'''
+
+ def __init__(self, log, marker):
+ '''Initialize a new object.
+
+ log: The Logfile object to log to.
+ marker: The name of the nested log section.'''
+
+ self.log = log
+ self.marker = marker
+
+ def __enter__(self):
+ self.log.start_section(self.marker)
+
+ def __exit__(self, extype, value, traceback):
+ self.log.end_section(self.marker)
+
+class Logfile(object):
+ '''Generates an HTML-formatted log file containing multiple streams of
+ data, each represented in a well-delineated/-structured fashion.'''
+
+ def __init__(self, fn):
+ '''Initialize a new object.
+
+ fn: The filename to write to.'''
+
+ self.f = open(fn, "wt")
+ self.last_stream = None
+ self.blocks = []
+ self.cur_evt = 1
+ shutil.copy(mod_dir + "/multiplexed_log.css", os.path.dirname(fn))
+ self.f.write("""\
+<html>
+<head>
+<link rel="stylesheet" type="text/css" href="multiplexed_log.css">
+</head>
+<body>
+<tt>
+""")
+
+ def close(self):
+ '''Close the log file. After calling this function, no more data may
+ be written to the log.'''
+
+ self.f.write("""\
+</tt>
+</body>
+</html>
+""")
+ self.f.close()
+
+ # The set of characters that should be represented as hexadecimal codes in
+ # the log file.
+ _nonprint = ("^%" + "".join(chr(c) for c in range(0, 32) if c not in (9, 10)) +
+ "".join(chr(c) for c in range(127, 256)))
+
+ def _escape(self, data):
+ '''Render raw log data into a format suitable for inclusion in an HTML
+ document. This includes HTML-escaping certain characters, and
+ translating control characters to a hexadecimal representation.
+
+ data: The raw string data to be escaped.'''
+
+ data = data.replace(chr(13), "")
+ data = "".join((c in self._nonprint) and ("%%%02x" % ord(c)) or
+ c for c in data)
+ data = cgi.escape(data)
+ return data
+
+ def _terminate_stream(self):
+ '''Write HTML to the log file to terminate the current stream's data.'''
+
+ self.cur_evt += 1
+ if not self.last_stream:
+ return
+ self.f.write("</pre>\n")
+ self.f.write("<div class=\"stream-trailer\" id=\"" +
+ self.last_stream.name + "\">End stream: " +
+ self.last_stream.name + "</div>\n")
+ self.f.write("</div>\n")
+ self.last_stream = None
+
+ def _note(self, note_type, msg):
+ '''Write HTML to the log file that represents some form of note or
+ one-off message.
+
+ note_type: The type of note. This must be a value supported by the
+ accompanying multiplexed_log.css.
+ msg: The note/message to log.'''
+
+ self._terminate_stream()
+ self.f.write("<div class=\"" + note_type + "\">\n<pre>")
+ self.f.write(self._escape(msg))
+ self.f.write("\n</pre></div>\n")
+
+ def start_section(self, marker):
+ '''Write HTML to the log file to mark the start of a new nested
+ section.
+
+ marker: The name of the section that is starting.'''
+
+ self._terminate_stream()
+ self.blocks.append(marker)
+ blk_path = "/".join(self.blocks)
+ self.f.write("<div class=\"section\" id=\"" + blk_path + "\">\n")
+ self.f.write("<div class=\"section-header\" id=\"" + blk_path +
+ "\">Section: " + blk_path + "</div>\n")
+
+ def end_section(self, marker):
+ '''Write HTML to the log file to mark the end of a the current nested
+ section.
+
+ marker: The name of the section that is ending.'''
+
+ if (not self.blocks) or (marker != self.blocks[-1]):
+ raise Exception("Block nesting mismatch: \"%s\" \"%s\"" %
+ (marker, "/".join(self.blocks)))
+ self._terminate_stream()
+ blk_path = "/".join(self.blocks)
+ self.f.write("<div class=\"section-trailer\" id=\"section-trailer-" +
+ blk_path + "\">End section: " + blk_path + "</div>\n")
+ self.f.write("</div>\n")
+ self.blocks.pop()
+
+ def section(self, marker):
+ '''Create a context manager for Python's "with" statement, which allows
+ a certain portion of test code to be logged to a separate section of
+ the log file.
+
+ marker: The name of the nested section.
+
+ Usage:
+ with log.section("somename"):
+ some test code'''
+
+ return SectionCtxMgr(self, marker)
+
+ def error(self, msg):
+ '''Write an error note to the log file.
+
+ msg: A message describing the error.'''
+
+ self._note("error", msg)
+
+ def warning(self, msg):
+ '''Write an warning note to the log file.
+
+ msg: A message describing the warning.'''
+
+ self._note("warning", msg)
+
+ def info(self, msg):
+ '''Write an informational note to the log file.
+
+ msg: An information message.'''
+
+ self._note("info", msg)
+
+ def action(self, msg):
+ '''Write an action note to the log file.
+
+ msg: A message describing the action that is being logged.'''
+
+ self._note("action", msg)
+
+ def status_pass(self, msg):
+ '''Write a note to the log file describing test(s) which passed.
+
+ msg: A message describing passed test(s).'''
+
+ self._note("status-pass", msg)
+
+ def status_skipped(self, msg):
+ '''Write a note to the log file describing skipped test(s).
+
+ msg: A message describing passed test(s).'''
+
+ self._note("status-skipped", msg)
+
+ def status_fail(self, msg):
+ '''Write a note to the log file describing failed test(s).
+
+ msg: A message describing passed test(s).'''
+
+ self._note("status-fail", msg)
+
+ def get_stream(self, name, chained_file=None):
+ '''Create an object to log a single stream's data into the log file.
+
+ name: The name of the stream.
+ chained_file: The file-like object to which all stream data should be
+ logged to in addition to this log. Can be None.'''
+
+ return LogfileStream(self, name, chained_file)
+
+ def get_runner(self, name, chained_file=None):
+ '''Create a utility object to execute sub-processes and log their
+ output to
+
+ name: The name of this sub-process.
+ chained_file: The file-like object to which all stream data should be
+ logged to in addition to logfile. Can be None.'''
+
+ return RunAndLog(self, name, chained_file)
+
+ def write(self, stream, data, implicit=False):
+ '''Write stream data into the log file. This function should only be
+ used by instances of LogfileStream or RunAndLog.
+
+ stream: The stream whose data is being logged.
+ data: The data to log.
+ implicit: Boolean indicating whether data actually appeared in the
+ stream, or was implicitly generated. A valid use-case is to repeat a
+ shell prompt at the start of each separate log section, so that the
+ log makes more sense.'''
+
+ if stream != self.last_stream:
+ self._terminate_stream()
+ self.f.write("<div class=\"stream\" id=\"%s\">\n" % stream.name)
+ self.f.write("<div class=\"stream-header\" id=\"" + stream.name +
+ "\">Stream: " + stream.name + "</div>\n")
+ self.f.write("<pre>")
+ if implicit:
+ self.f.write("<span class=\"implicit\">")
+ self.f.write(self._escape(data))
+ if implicit:
+ self.f.write("</span>")
+ self.last_stream = stream
+
+ def flush(self):
+ '''Flush the log stream, to ensure correct log interleaving.'''
+
+ self.f.flush()
diff --git a/test/py/pytest.ini b/test/py/pytest.ini
new file mode 100644
index 000000000000..67e514f42058
--- /dev/null
+++ b/test/py/pytest.ini
@@ -0,0 +1,11 @@
+# Copyright (c) 2015 Stephen Warren
+# Copyright (c) 2015-2016, NVIDIA CORPORATION. All rights reserved.
+#
+# SPDX-License-Identifier: GPL-2.0
+
+# Static configuration data for pytest. pytest reads this at startup time.
+
+[pytest]
+markers =
+ boardspec: U-Boot: Describes the set of boards a test can/can't run on.
+ buildconfigspec: U-Boot: Describes Kconfig/config-header constraints.
diff --git a/test/py/test.py b/test/py/test.py
new file mode 100755
index 000000000000..9c23898774ed
--- /dev/null
+++ b/test/py/test.py
@@ -0,0 +1,32 @@
+#!/usr/bin/env python
+
+# Copyright (c) 2015 Stephen Warren
+# Copyright (c) 2015-2016, NVIDIA CORPORATION. All rights reserved.
+#
+# SPDX-License-Identifier: GPL-2.0
+
+# Wrapper script to invoke pytest with the directory name that contains the
+# U-Boot tests.
+
+import os
+import os.path
+import sys
+
+# Get rid of argv[0]
+sys.argv.pop(0)
+
+# argv; py.test test_directory_name user-supplied-arguments
+args = ["py.test", os.path.dirname(__file__) + "/tests"]
+args.extend(sys.argv)
+
+try:
+ os.execvp("py.test", args)
+except:
+ # Log full details of any exception for detailed analysis
+ import traceback
+ traceback.print_exc()
+ # Hint to the user that they likely simply haven't installed the required
+ # dependencies.
+ print >>sys.stderr, """
+exec(py.test) failed; perhaps you are missing some dependencies?
+See test/py/README.md for the list."""
diff --git a/test/py/tests/test_000_version.py b/test/py/tests/test_000_version.py
new file mode 100644
index 000000000000..0fddaf4fb6d5
--- /dev/null
+++ b/test/py/tests/test_000_version.py
@@ -0,0 +1,20 @@
+# Copyright (c) 2015 Stephen Warren
+# Copyright (c) 2016, NVIDIA CORPORATION. All rights reserved.
+#
+# SPDX-License-Identifier: GPL-2.0
+
+# pytest runs tests the order of their module path, which is related to the
+# filename containing the test. This file is named such that it is sorted
+# first, simply as a very basic sanity check of the functionality of the U-Boot
+# command prompt.
+
+def test_version(u_boot_console):
+ '''Test that the "version" command prints the U-Boot version.'''
+
+ # "version" prints the U-Boot sign-on message. This is usually considered
+ # an error, so that any unexpected reboot causes an error. Here, this
+ # error detection is disabled since the sign-on message is expected.
+ with u_boot_console.disable_check('main_signon'):
+ response = u_boot_console.run_command('version')
+ # Ensure "version" printed what we expected.
+ u_boot_console.validate_main_signon_in_text(response)
diff --git a/test/py/tests/test_help.py b/test/py/tests/test_help.py
new file mode 100644
index 000000000000..894f3b5f1700
--- /dev/null
+++ b/test/py/tests/test_help.py
@@ -0,0 +1,9 @@
+# Copyright (c) 2015 Stephen Warren
+# Copyright (c) 2016, NVIDIA CORPORATION. All rights reserved.
+#
+# SPDX-License-Identifier: GPL-2.0
+
+def test_help(u_boot_console):
+ '''Test that the "help" command can be executed.'''
+
+ u_boot_console.run_command('help')
diff --git a/test/py/tests/test_unknown_cmd.py b/test/py/tests/test_unknown_cmd.py
new file mode 100644
index 000000000000..2de93e0026ff
--- /dev/null
+++ b/test/py/tests/test_unknown_cmd.py
@@ -0,0 +1,14 @@
+# Copyright (c) 2015 Stephen Warren
+# Copyright (c) 2016, NVIDIA CORPORATION. All rights reserved.
+#
+# SPDX-License-Identifier: GPL-2.0
+
+def test_unknown_command(u_boot_console):
+ '''Test that executing an unknown command causes U-Boot to print an
+ error.'''
+
+ # The "unknown command" error is actively expected here,
+ # so error detection for it is disabled.
+ with u_boot_console.disable_check('unknown_command'):
+ response = u_boot_console.run_command('non_existent_cmd')
+ assert('Unknown command \'non_existent_cmd\' - try \'help\'' in response)
diff --git a/test/py/u_boot_console_base.py b/test/py/u_boot_console_base.py
new file mode 100644
index 000000000000..9f3696947d6d
--- /dev/null
+++ b/test/py/u_boot_console_base.py
@@ -0,0 +1,260 @@
+# Copyright (c) 2015 Stephen Warren
+# Copyright (c) 2015-2016, NVIDIA CORPORATION. All rights reserved.
+#
+# SPDX-License-Identifier: GPL-2.0
+
+# Common logic to interact with U-Boot via the console. This class provides
+# the interface that tests use to execute U-Boot shell commands and wait for
+# their results. Sub-classes exist to perform board-type-specific setup
+# operations, such as spawning a sub-process for Sandbox, or attaching to the
+# serial console of real hardware.
+
+import multiplexed_log
+import os
+import pytest
+import re
+import sys
+
+# Regexes for text we expect U-Boot to send to the console.
+pattern_u_boot_spl_signon = re.compile('(U-Boot SPL \\d{4}\\.\\d{2}-[^\r\n]*)')
+pattern_u_boot_main_signon = re.compile('(U-Boot \\d{4}\\.\\d{2}-[^\r\n]*)')
+pattern_stop_autoboot_prompt = re.compile('Hit any key to stop autoboot: ')
+pattern_unknown_command = re.compile('Unknown command \'.*\' - try \'help\'')
+pattern_error_notification = re.compile('## Error: ')
+
+class ConsoleDisableCheck(object):
+ '''Context manager (for Python's with statement) that temporarily disables
+ the specified console output error check. This is useful when deliberately
+ executing a command that is known to trigger one of the error checks, in
+ order to test that the error condition is actually raised. This class is
+ used internally by ConsoleBase::disable_check(); it is not intended for
+ direct usage.'''
+
+ def __init__(self, console, check_type):
+ self.console = console
+ self.check_type = check_type
+
+ def __enter__(self):
+ self.console.disable_check_count[self.check_type] += 1
+
+ def __exit__(self, extype, value, traceback):
+ self.console.disable_check_count[self.check_type] -= 1
+
+class ConsoleBase(object):
+ '''The interface through which test functions interact with the U-Boot
+ console. This primarily involves executing shell commands, capturing their
+ results, and checking for common error conditions. Some common utilities
+ are also provided too.'''
+
+ def __init__(self, log, config, max_fifo_fill):
+ '''Initialize a U-Boot console connection. Can only usefully be called
+ by sub-classes.'''
+
+ self.log = log
+ self.config = config
+ self.max_fifo_fill = max_fifo_fill
+
+ self.logstream = self.log.get_stream('console', sys.stdout)
+
+ # Array slice removes leading/trailing quotes
+ self.prompt = self.config.buildconfig['config_sys_prompt'][1:-1]
+ self.prompt_escaped = re.escape(self.prompt)
+ self.p = None
+ self.disable_check_count = {
+ 'spl_signon': 0,
+ 'main_signon': 0,
+ 'unknown_command': 0,
+ 'error_notification': 0,
+ }
+
+ self.at_prompt = False
+ self.at_prompt_logevt = None
+ self.ram_base = None
+
+ def close(self):
+ '''Terminate the connection to the U-Boot console. Likely only useful
+ once all interaction with U-Boot is complete.'''
+
+ if self.p:
+ self.p.close()
+ self.logstream.close()
+
+ def run_command(self, cmd, wait_for_echo=True, send_nl=True,
+ wait_for_prompt=True):
+ '''Execute a command via the U-Boot console. The command is always sent
+ to U-Boot.
+
+ U-Boot echoes any command back to its output, and this function
+ typically waits for that to occur. The wait can be disabled by setting
+ wait_for_echo=False, which is useful e.g. when sending CTRL-C to
+ interrupt a long-running command such as "ums".
+
+ Command execution is typically triggered by sending a newline
+ character. This can be disabled by setting send_nl=False, which is
+ also useful when sending CTRL-C.
+
+ This function typically waits for the command to finish executing, and
+ returns the console output that it generated. This can be disabled by
+ setting wait_for_prompt=False, which is useful when invoking a long-
+ running command such as "ums".'''
+
+ self.ensure_spawned()
+
+ if self.at_prompt and \
+ self.at_prompt_logevt != self.logstream.logfile.cur_evt:
+ self.logstream.write(self.prompt, implicit=True)
+
+ bad_patterns = []
+ bad_pattern_ids = []
+ if (self.disable_check_count['spl_signon'] == 0 and
+ self.u_boot_spl_signon):
+ bad_patterns.append(self.u_boot_spl_signon_escaped)
+ bad_pattern_ids.append('SPL signon')
+ if self.disable_check_count['main_signon'] == 0:
+ bad_patterns.append(self.u_boot_main_signon_escaped)
+ bad_pattern_ids.append('U-Boot main signon')
+ if self.disable_check_count['unknown_command'] == 0:
+ bad_patterns.append(pattern_unknown_command)
+ bad_pattern_ids.append('Unknown command')
+ if self.disable_check_count['error_notification'] == 0:
+ bad_patterns.append(pattern_error_notification)
+ bad_pattern_ids.append('Error notification')
+ try:
+ self.at_prompt = False
+ if send_nl:
+ cmd += '\n'
+ while cmd:
+ # Limit max outstanding data, so UART FIFOs don't overflow
+ chunk = cmd[:self.max_fifo_fill]
+ cmd = cmd[self.max_fifo_fill:]
+ self.p.send(chunk)
+ if not wait_for_echo:
+ continue
+ chunk = re.escape(chunk)
+ chunk = chunk.replace('\\\n', '[\r\n]')
+ m = self.p.expect([chunk] + bad_patterns)
+ if m != 0:
+ self.at_prompt = False
+ raise Exception('Bad pattern found on console: ' +
+ bad_pattern_ids[m - 1])
+ if not wait_for_prompt:
+ return
+ m = self.p.expect([self.prompt_escaped] + bad_patterns)
+ if m != 0:
+ self.at_prompt = False
+ raise Exception('Bad pattern found on console: ' +
+ bad_pattern_ids[m - 1])
+ self.at_prompt = True
+ self.at_prompt_logevt = self.logstream.logfile.cur_evt
+ # Only strip \r\n; space/TAB might be significant if testing
+ # indentation.
+ return self.p.before.strip('\r\n')
+ except Exception as ex:
+ self.log.error(str(ex))
+ self.cleanup_spawn()
+ raise
+
+ def ctrlc(self):
+ '''Send a CTRL-C character to U-Boot. This is useful in order to
+ stop execution of long-running synchronous commands such as "ums".'''
+
+ self.run_command(chr(3), wait_for_echo=False, send_nl=False)
+
+ def ensure_spawned(self):
+ '''Ensure that this console object is attached to a correctly operating
+ U-Boot instance. This may require spawning a new Sandbox process or
+ resetting target hardware, as defined by the implementation sub-class.
+
+ This is an internal function and should not be called directly.'''
+
+ if self.p:
+ return
+ try:
+ self.at_prompt = False
+ self.log.action('Starting U-Boot')
+ self.p = self.get_spawn()
+ # Real targets can take a long time to scroll large amounts of
+ # text if LCD is enabled. This value may need tweaking in the
+ # future, possibly per-test to be optimal. This works for 'help'
+ # on board 'seaboard'.
+ self.p.timeout = 30000
+ self.p.logfile_read = self.logstream
+ if self.config.buildconfig.get('CONFIG_SPL', False) == 'y':
+ self.p.expect([pattern_u_boot_spl_signon])
+ self.u_boot_spl_signon = self.p.after
+ self.u_boot_spl_signon_escaped = re.escape(self.p.after)
+ else:
+ self.u_boot_spl_signon = None
+ self.p.expect([pattern_u_boot_main_signon])
+ self.u_boot_main_signon = self.p.after
+ self.u_boot_main_signon_escaped = re.escape(self.p.after)
+ while True:
+ match = self.p.expect([self.prompt_escaped,
+ pattern_stop_autoboot_prompt])
+ if match == 1:
+ self.p.send(chr(3)) # CTRL-C
+ continue
+ break
+ self.at_prompt = True
+ self.at_prompt_logevt = self.logstream.logfile.cur_evt
+ except Exception as ex:
+ self.log.error(str(ex))
+ self.cleanup_spawn()
+ raise
+
+ def cleanup_spawn(self):
+ '''Shut down all interaction with the U-Boot instance. This is used
+ when an error is detected prior to re-establishing a connection with a
+ fresh U-Boot instance.
+
+ This is an internal function and should not be called directly.'''
+
+ try:
+ if self.p:
+ self.p.close()
+ except:
+ pass
+ self.p = None
+
+ def validate_main_signon_in_text(self, text):
+ '''Validate that a command's console output includes the expected
+ U-Boot signon message. This is primarily useful for validating the
+ "version" command without duplicating the signon text regex in a test
+ function.'''
+
+ assert(self.u_boot_main_signon in text)
+
+ def disable_check(self, check_type):
+ '''Create a new Python context manager (for use with the "with"
+ statement) which temporarily disables a particular console output error
+ check. Valid values for check_type may be found in
+ self.disable_check_count above.'''
+
+ return ConsoleDisableCheck(self, check_type)
+
+ def find_ram_base(self):
+ '''Probe the running U-Boot to determine the address of the first bank
+ of RAM. This is useful for tests that test reading/writing RAM, or
+ load/save files that aren't associated with some standard address
+ typically represented in an environment variable such as
+ ${kernel_addr_r}. The value is cached so that it only needs to be
+ actively read once.'''
+
+ if self.config.buildconfig.get('config_cmd_bdi', 'n') != 'y':
+ pytest.skip('bdinfo command not supported')
+ if self.ram_base == -1:
+ pytest.skip('Previously failed to find RAM bank start')
+ if self.ram_base is not None:
+ return self.ram_base
+
+ with self.log.section('find_ram_base'):
+ response = self.run_command('bdinfo')
+ for l in response.split('\n'):
+ if '-> start' in l:
+ self.ram_base = int(l.split('=')[1].strip(), 16)
+ break
+ if self.ram_base is None:
+ self.ram_base = -1
+ raise Exception('Failed to find RAM bank start in `bdinfo`')
+
+ return self.ram_base
diff --git a/test/py/u_boot_console_exec_attach.py b/test/py/u_boot_console_exec_attach.py
new file mode 100644
index 000000000000..9fcc32c5f73b
--- /dev/null
+++ b/test/py/u_boot_console_exec_attach.py
@@ -0,0 +1,51 @@
+# Copyright (c) 2015 Stephen Warren
+# Copyright (c) 2015-2016, NVIDIA CORPORATION. All rights reserved.
+#
+# SPDX-License-Identifier: GPL-2.0
+
+# Logic to interact with U-Boot running on real hardware, typically via a
+# physical serial port.
+
+from ubspawn import Spawn
+from u_boot_console_base import ConsoleBase
+
+class ConsoleExecAttach(ConsoleBase):
+ '''Represents a physical connection to a U-Boot console, typically via a
+ serial port. This implementation executes a sub-process to attach to the
+ console, expecting that the stdin/out of the sub-process will be forwarded
+ to/from the physical hardware. This approach isolates the test infra-
+ structure from the user-/installation-specific details of how to
+ communicate with, and the identity of, serial ports etc.'''
+
+ def __init__(self, log, config):
+ '''Initialize a U-Boot console connection.
+
+ log: A multiplexed_log::Logfile instance.
+ config: A "configuration" object as defined in conftest.py.'''
+
+ # The max_fifo_fill value might need tweaking per-board/-SoC?
+ # 1 would be safe anywhere, but is very slow (a pexpect issue?).
+ # 16 is a common FIFO size.
+ # HW flow control would mean this could be infinite.
+ super(ConsoleExecAttach, self).__init__(log, config, max_fifo_fill=16)
+
+ self.log.action('Flashing U-Boot')
+ cmd = ['u-boot-test-flash', config.board_type, config.board_identity]
+ runner = self.log.get_runner(cmd[0])
+ runner.run(cmd)
+ runner.close()
+
+ def get_spawn(self):
+ '''Create and return a new "spawn" object that is attached to a
+ freshly reset board running U-Boot.'''
+
+ args = [self.config.board_type, self.config.board_identity]
+ s = Spawn(['u-boot-test-console'] + args)
+
+ self.log.action('Resetting board')
+ cmd = ['u-boot-test-reset'] + args
+ runner = self.log.get_runner(cmd[0])
+ runner.run(cmd)
+ runner.close()
+
+ return s
diff --git a/test/py/u_boot_console_sandbox.py b/test/py/u_boot_console_sandbox.py
new file mode 100644
index 000000000000..0c0343935167
--- /dev/null
+++ b/test/py/u_boot_console_sandbox.py
@@ -0,0 +1,48 @@
+# Copyright (c) 2015 Stephen Warren
+# Copyright (c) 2015-2016, NVIDIA CORPORATION. All rights reserved.
+#
+# SPDX-License-Identifier: GPL-2.0
+
+# Logic to interact with the sandbox port of U-Boot, running as a sub-process.
+
+import time
+from ubspawn import Spawn
+from u_boot_console_base import ConsoleBase
+
+class ConsoleSandbox(ConsoleBase):
+ '''Represents a connection to a sandbox U-Boot console, executed as a sub-
+ process.'''
+
+ def __init__(self, log, config):
+ '''Initialize a U-Boot console connection.
+
+ log: A multiplexed_log::Logfile instance.
+ config: A "configuration" object as defined in conftest.py.'''
+
+ super(ConsoleSandbox, self).__init__(log, config, max_fifo_fill=1024)
+
+ def get_spawn(self):
+ '''Create and return a new "spawn" object that is attached to a
+ freshly invoked U-Boot sandbox process.'''
+
+ return Spawn([self.config.build_dir + '/u-boot'])
+
+ def kill(self, sig):
+ '''Send a specific Unix signal to the sandbox process.'''
+
+ self.ensure_spawned()
+ self.log.action('kill %d' % sig)
+ self.p.kill(sig)
+
+ def validate_exited(self):
+ '''Validate that the sandbox process has exited.'''
+
+ p = self.p
+ self.p = None
+ for i in xrange(100):
+ ret = not p.isalive()
+ if ret:
+ break
+ time.sleep(0.1)
+ p.close()
+ return ret
diff --git a/test/py/u_boot_spawn.py b/test/py/u_boot_spawn.py
new file mode 100644
index 000000000000..12e417c841ed
--- /dev/null
+++ b/test/py/u_boot_spawn.py
@@ -0,0 +1,123 @@
+# Copyright (c) 2015-2016, NVIDIA CORPORATION. All rights reserved.
+#
+# SPDX-License-Identifier: GPL-2.0
+
+# Logic to spawn a sub-process and interact with its stdio.
+
+import os
+import re
+import pty
+import select
+import time
+
+class Timeout(Exception):
+ '''An exception sub-class that indicates that a timeout occurred.'''
+ pass
+
+class Spawn(object):
+ '''Represents the stdio of a freshly created sub-process. Commands may be
+ sent to the process, and responses waited for.'''
+
+ def __init__(self, args):
+ '''Spawn (fork/exec) the sub-process and initialize interaction with
+ it.
+
+ args: array of processs arguments. argv[0] is the command to execute.'''
+
+ self.waited = False
+ self.buf = ''
+ self.logfile_read = None
+ self.before = ''
+ self.after = ''
+ self.timeout = None
+
+ (self.pid, self.fd) = pty.fork()
+ if self.pid == 0:
+ try:
+ os.execvp(args[0], args)
+ except:
+ print 'CHILD EXECEPTION:'
+ import traceback
+ traceback.print_exc()
+ finally:
+ os._exit(255)
+
+ self.poll = select.poll()
+ self.poll.register(self.fd, select.POLLIN | select.POLLPRI | select.POLLERR | select.POLLHUP | select.POLLNVAL)
+
+ def kill(self, sig):
+ '''Send unix signal "sig" to the child process.'''
+
+ os.kill(self.pid, sig)
+
+ def isalive(self):
+ '''Determine whether the child process is still running.'''
+
+ if self.waited:
+ return False
+
+ w = os.waitpid(self.pid, os.WNOHANG)
+ if w[0] == 0:
+ return True
+
+ self.waited = True
+ return False
+
+ def send(self, data):
+ '''Send data to the sub-process's stdin.'''
+
+ os.write(self.fd, data)
+
+ def expect(self, patterns):
+ '''Wait for the sub-process to emit one of a set of expected patterns,
+ or for a timeout to occur.
+
+ patterns: a list of strings or regex objects that we expect to see in
+ the sub-process' stdout.'''
+
+ for pi in xrange(len(patterns)):
+ if type(patterns[pi]) == type(''):
+ patterns[pi] = re.compile(patterns[pi])
+
+ try:
+ while True:
+ earliest_m = None
+ earliest_pi = None
+ for pi in xrange(len(patterns)):
+ pattern = patterns[pi]
+ m = pattern.search(self.buf)
+ if not m:
+ continue
+ if earliest_m and m.start() > earliest_m.start():
+ continue
+ earliest_m = m
+ earliest_pi = pi
+ if earliest_m:
+ pos = earliest_m.start()
+ posafter = earliest_m.end() + 1
+ self.before = self.buf[:pos]
+ self.after = self.buf[pos:posafter]
+ self.buf = self.buf[posafter:]
+ return earliest_pi
+ events = self.poll.poll(self.timeout)
+ if not events:
+ raise Timeout()
+ c = os.read(self.fd, 1024)
+ if not c:
+ raise EOFError()
+ if self.logfile_read:
+ self.logfile_read.write(c)
+ self.buf += c
+ finally:
+ if self.logfile_read:
+ self.logfile_read.flush()
+
+ def close(self):
+ '''Close the stdio connection to the sub-process, and wait until the
+ sub-process is no longer running.'''
+
+ os.close(self.fd)
+ for i in xrange(100):
+ if not self.isalive():
+ break
+ time.sleep(0.1)
--
2.6.4
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [U-Boot] [PATCH V3 2/7] test/py: test that sandbox exits when asked
2016-01-05 22:58 [U-Boot] [PATCH V3 1/7] test/py: Implement pytest infrastructure Stephen Warren
@ 2016-01-05 22:58 ` Stephen Warren
2016-01-11 15:23 ` Simon Glass
2016-01-05 22:58 ` [U-Boot] [PATCH V3 3/7] test/py: add test of setenv/printenv/echo Stephen Warren
` (6 subsequent siblings)
7 siblings, 1 reply; 20+ messages in thread
From: Stephen Warren @ 2016-01-05 22:58 UTC (permalink / raw)
To: u-boot
Test the sandbox port's implementation of the reset command and SIGHUP
handling. These should both cause the U-Boot process to exit gracefully.
Signed-off-by: Stephen Warren <swarren@wwwdotorg.org>
Signed-off-by: Stephen Warren <swarren@nvidia.com>
---
v3:
- Move test scripts into a sub-directory.
Suggested by Michal Simek.
- s/uboot/u[-_]boot/g. Suggested by Simon Glass.
- Typo.
- s/"/'/g. Suggested by Simon Glass.
- Add more documentation. Suggested by Simon Glass.
---
test/py/tests/test_sandbox_exit.py | 24 ++++++++++++++++++++++++
1 file changed, 24 insertions(+)
create mode 100644 test/py/tests/test_sandbox_exit.py
diff --git a/test/py/tests/test_sandbox_exit.py b/test/py/tests/test_sandbox_exit.py
new file mode 100644
index 000000000000..2aa8eb4abc68
--- /dev/null
+++ b/test/py/tests/test_sandbox_exit.py
@@ -0,0 +1,24 @@
+# Copyright (c) 2015 Stephen Warren
+# Copyright (c) 2015-2016, NVIDIA CORPORATION. All rights reserved.
+#
+# SPDX-License-Identifier: GPL-2.0
+
+import pytest
+import signal
+
+ at pytest.mark.boardspec('sandbox')
+ at pytest.mark.buildconfigspec('reset')
+def test_reset(u_boot_console):
+ '''Test that the "reset" command exits sandbox process.'''
+
+ u_boot_console.run_command('reset', wait_for_prompt=False)
+ assert(u_boot_console.validate_exited())
+ u_boot_console.ensure_spawned()
+
+ at pytest.mark.boardspec('sandbox')
+def test_ctrl_c(u_boot_console):
+ '''Test that sending SIGINT to sandbox causes it to exit.'''
+
+ u_boot_console.kill(signal.SIGINT)
+ assert(u_boot_console.validate_exited())
+ u_boot_console.ensure_spawned()
--
2.6.4
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [U-Boot] [PATCH V3 3/7] test/py: add test of setenv/printenv/echo
2016-01-05 22:58 [U-Boot] [PATCH V3 1/7] test/py: Implement pytest infrastructure Stephen Warren
2016-01-05 22:58 ` [U-Boot] [PATCH V3 2/7] test/py: test that sandbox exits when asked Stephen Warren
@ 2016-01-05 22:58 ` Stephen Warren
2016-01-11 15:23 ` Simon Glass
2016-01-05 22:58 ` [U-Boot] [PATCH V3 4/7] test/py: test the md/mw commands Stephen Warren
` (5 subsequent siblings)
7 siblings, 1 reply; 20+ messages in thread
From: Stephen Warren @ 2016-01-05 22:58 UTC (permalink / raw)
To: u-boot
This tests basic environment variable functionality.
Signed-off-by: Stephen Warren <swarren@wwwdotorg.org>
Signed-off-by: Stephen Warren <swarren@nvidia.com>
---
v3:
- Handle environment values that contain an = sign.
Reported by Michal Simek.
- Move test scripts into a sub-directory.
Suggested by Michal Simek.
- s/uboot/u[-_]boot/g. Suggested by Simon Glass.
- s/"/'/g. Suggested by Simon Glass.
- Add more documentation. Suggested by Simon Glass.
- Simplify command string construction. Suggested by Simon Glass.
- Move relevant edits to command_ut.c into this patch from a later one.
Suggested by Simon Glass.
---
test/command_ut.c | 4 --
test/py/tests/test_env.py | 169 ++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 169 insertions(+), 4 deletions(-)
create mode 100644 test/py/tests/test_env.py
diff --git a/test/command_ut.c b/test/command_ut.c
index 926573a39543..c086abe3ed3e 100644
--- a/test/command_ut.c
+++ b/test/command_ut.c
@@ -20,10 +20,6 @@ static int do_ut_cmd(cmd_tbl_t *cmdtp, int flag, int argc, char * const argv[])
printf("%s: Testing commands\n", __func__);
run_command("env default -f -a", 0);
- /* run a single command */
- run_command("setenv single 1", 0);
- assert(!strcmp("1", getenv("single")));
-
/* make sure that compound statements work */
#ifdef CONFIG_SYS_HUSH_PARSER
run_command("if test -n ${single} ; then setenv check 1; fi", 0);
diff --git a/test/py/tests/test_env.py b/test/py/tests/test_env.py
new file mode 100644
index 000000000000..70e4d2a38aaf
--- /dev/null
+++ b/test/py/tests/test_env.py
@@ -0,0 +1,169 @@
+# Copyright (c) 2015 Stephen Warren
+# Copyright (c) 2015-2016, NVIDIA CORPORATION. All rights reserved.
+#
+# SPDX-License-Identifier: GPL-2.0
+
+# Test operation of shell commands relating to environment variables.
+
+import pytest
+
+# FIXME: This might be useful for other tests;
+# perhaps refactor it into ConsoleBase or some other state object?
+class StateTestEnv(object):
+ '''Container that represents the state of all U-Boot environment variables.
+ This enables quick determination of existant/non-existant variable
+ names.'''
+
+ def __init__(self, u_boot_console):
+ '''Initialize a new StateTestEnv object.
+
+ u_boot_console: A U-Boot console.'''
+
+ self.u_boot_console = u_boot_console
+ self.get_env()
+ self.set_var = self.get_non_existent_var()
+
+ def get_env(self):
+ '''Read all current environment variables from U-Boot.'''
+
+ response = self.u_boot_console.run_command('printenv')
+ self.env = {}
+ for l in response.splitlines():
+ if not '=' in l:
+ continue
+ (var, value) = l.strip().split('=', 1)
+ self.env[var] = value
+
+ def get_existent_var(self):
+ '''Return the name on an environment variable that exists.'''
+
+ for var in self.env:
+ return var
+
+ def get_non_existent_var(self):
+ '''Return the name on an environment variable that does not exist.'''
+
+ n = 0
+ while True:
+ var = 'test_env_' + str(n)
+ if var not in self.env:
+ return var
+ n += 1
+
+ at pytest.fixture(scope='module')
+def state_test_env(u_boot_console):
+ '''pytest fixture to provide a StateTestEnv object to tests.'''
+
+ return StateTestEnv(u_boot_console)
+
+def unset_var(state_test_env, var):
+ '''Unset an environment variable, both by executing a U-Boot shell command
+ and updating a StateTestEnv object.
+
+ var: The variable name to unset.'''
+
+ state_test_env.u_boot_console.run_command('setenv %s' % var)
+ if var in state_test_env.env:
+ del state_test_env.env[var]
+
+def set_var(state_test_env, var, value):
+ '''Set an environment variable, both by executing a U-Boot shell command
+ and updating a StateTestEnv object.
+
+ var: The variable name to set.
+ value: The value to set the variable to.'''
+
+ state_test_env.u_boot_console.run_command('setenv %s "%s"' % (var, value))
+ state_test_env.env[var] = value
+
+def validate_empty(state_test_env, var):
+ '''Validate that a variable is not set, using U-Boot shell commands.
+
+ var: The variable name to test.'''
+
+ response = state_test_env.u_boot_console.run_command('echo $%s' % var)
+ assert response == ''
+
+def validate_set(state_test_env, var, value):
+ '''Validate that a variable is set, using U-Boot shell commands.
+
+ var: The variable name to test.
+ value: The value the variable is expected to have.'''
+
+ # echo does not preserve leading, internal, or trailing whitespace in the
+ # value. printenv does, and hence allows more complete testing.
+ response = state_test_env.u_boot_console.run_command('printenv %s' % var)
+ assert response == ('%s=%s' % (var, value))
+
+def test_env_echo_exists(state_test_env):
+ '''Test echoing a variable that exists.'''
+
+ var = state_test_env.get_existent_var()
+ value = state_test_env.env[var]
+ validate_set(state_test_env, var, value)
+
+def test_env_echo_non_existent(state_test_env):
+ '''Test echoing a variable that doesn't exist.'''
+
+ var = state_test_env.set_var
+ validate_empty(state_test_env, var)
+
+def test_env_printenv_non_existent(state_test_env):
+ '''Test printenv error message for non-existant variables.'''
+
+ var = state_test_env.set_var
+ c = state_test_env.u_boot_console
+ with c.disable_check('error_notification'):
+ response = c.run_command('printenv %s' % var)
+ assert(response == '## Error: "%s" not defined' % var)
+
+def test_env_unset_non_existent(state_test_env):
+ '''Test unsetting a nonexistent variable.'''
+
+ var = state_test_env.get_non_existent_var()
+ unset_var(state_test_env, var)
+ validate_empty(state_test_env, var)
+
+def test_env_set_non_existent(state_test_env):
+ '''Test set a non-existant variable.'''
+
+ var = state_test_env.set_var
+ value = 'foo'
+ set_var(state_test_env, var, value)
+ validate_set(state_test_env, var, value)
+
+def test_env_set_existing(state_test_env):
+ '''Test setting an existant variable.'''
+
+ var = state_test_env.set_var
+ value = 'bar'
+ set_var(state_test_env, var, value)
+ validate_set(state_test_env, var, value)
+
+def test_env_unset_existing(state_test_env):
+ '''Test unsetting a variable.'''
+
+ var = state_test_env.set_var
+ unset_var(state_test_env, var)
+ validate_empty(state_test_env, var)
+
+def test_env_expansion_spaces(state_test_env):
+ '''Test expanding an environment variable that contains a space in its
+ value.'''
+
+ var_space = None
+ var_test = None
+ try:
+ var_space = state_test_env.get_non_existent_var()
+ set_var(state_test_env, var_space, ' ')
+
+ var_test = state_test_env.get_non_existent_var()
+ value = ' 1${%(var_space)s}${%(var_space)s} 2 ' % locals()
+ set_var(state_test_env, var_test, value)
+ value = ' 1 2 '
+ validate_set(state_test_env, var_test, value)
+ finally:
+ if var_space:
+ unset_var(state_test_env, var_space)
+ if var_test:
+ unset_var(state_test_env, var_test)
--
2.6.4
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [U-Boot] [PATCH V3 4/7] test/py: test the md/mw commands
2016-01-05 22:58 [U-Boot] [PATCH V3 1/7] test/py: Implement pytest infrastructure Stephen Warren
2016-01-05 22:58 ` [U-Boot] [PATCH V3 2/7] test/py: test that sandbox exits when asked Stephen Warren
2016-01-05 22:58 ` [U-Boot] [PATCH V3 3/7] test/py: add test of setenv/printenv/echo Stephen Warren
@ 2016-01-05 22:58 ` Stephen Warren
2016-01-11 15:23 ` Simon Glass
2016-01-05 22:58 ` [U-Boot] [PATCH V3 5/7] test/py: add test of basic shell functionality Stephen Warren
` (4 subsequent siblings)
7 siblings, 1 reply; 20+ messages in thread
From: Stephen Warren @ 2016-01-05 22:58 UTC (permalink / raw)
To: u-boot
This tests whether md/mw work, and affect each-other.
Command repeat is also tested.
test/cmd_repeat.sh is removed, since the new Python-based test does
everything it used to.
Signed-off-by: Stephen Warren <swarren@wwwdotorg.org>
Signed-off-by: Stephen Warren <swarren@nvidia.com>
---
v3:
- Add extra mw during md test to account for the possibility that the
test's write data may already be present in RAM.
Suggested by Michal Simek.
- Move test scripts into a sub-directory.
Suggested by Michal Simek.
- s/uboot/u[-_]boot/g. Suggested by Simon Glass.
- s/"/'/g. Suggested by Simon Glass.
- Add more documentation. Suggested by Simon Glass.
---
test/cmd_repeat.sh | 29 -----------------------------
test/py/tests/test_md.py | 36 ++++++++++++++++++++++++++++++++++++
2 files changed, 36 insertions(+), 29 deletions(-)
delete mode 100755 test/cmd_repeat.sh
create mode 100644 test/py/tests/test_md.py
diff --git a/test/cmd_repeat.sh b/test/cmd_repeat.sh
deleted file mode 100755
index 990e79900f47..000000000000
--- a/test/cmd_repeat.sh
+++ /dev/null
@@ -1,29 +0,0 @@
-#!/bin/sh
-
-# Test for U-Boot cli including command repeat
-
-BASE="$(dirname $0)"
-. $BASE/common.sh
-
-run_test() {
- ./${OUTPUT_DIR}/u-boot <<END
-setenv ctrlc_ignore y
-md 0
-
-reset
-END
-}
-check_results() {
- echo "Check results"
-
- grep -q 00000100 ${tmp} || fail "Command did not repeat"
-}
-
-echo "Test CLI repeat"
-echo
-tmp="$(tempfile)"
-build_uboot
-run_test >${tmp}
-check_results ${tmp}
-rm ${tmp}
-echo "Test passed"
diff --git a/test/py/tests/test_md.py b/test/py/tests/test_md.py
new file mode 100644
index 000000000000..94603c7df609
--- /dev/null
+++ b/test/py/tests/test_md.py
@@ -0,0 +1,36 @@
+# Copyright (c) 2015 Stephen Warren
+# Copyright (c) 2015-2016, NVIDIA CORPORATION. All rights reserved.
+#
+# SPDX-License-Identifier: GPL-2.0
+
+import pytest
+
+ at pytest.mark.buildconfigspec('cmd_memory')
+def test_md(u_boot_console):
+ '''Test that md reads memory as expected, and that memory can be modified
+ using the mw command.'''
+
+ ram_base = u_boot_console.find_ram_base()
+ addr = '%08x' % ram_base
+ val = 'a5f09876'
+ expected_response = addr + ': ' + val
+ u_boot_console.run_command('mw ' + addr + ' 0 10')
+ response = u_boot_console.run_command('md ' + addr + ' 10')
+ assert(not (expected_response in response))
+ u_boot_console.run_command('mw ' + addr + ' ' + val)
+ response = u_boot_console.run_command('md ' + addr + ' 10')
+ assert(expected_response in response)
+
+ at pytest.mark.buildconfigspec('cmd_memory')
+def test_md_repeat(u_boot_console):
+ '''Test command repeat (via executing an empty command) operates correctly
+ for "md"; the command must repeat and dump an incrementing address.'''
+
+ ram_base = u_boot_console.find_ram_base()
+ addr_base = '%08x' % ram_base
+ words = 0x10
+ addr_repeat = '%08x' % (ram_base + (words * 4))
+ u_boot_console.run_command('md %s %x' % (addr_base, words))
+ response = u_boot_console.run_command('')
+ expected_response = addr_repeat + ': '
+ assert(expected_response in response)
--
2.6.4
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [U-Boot] [PATCH V3 5/7] test/py: add test of basic shell functionality
2016-01-05 22:58 [U-Boot] [PATCH V3 1/7] test/py: Implement pytest infrastructure Stephen Warren
` (2 preceding siblings ...)
2016-01-05 22:58 ` [U-Boot] [PATCH V3 4/7] test/py: test the md/mw commands Stephen Warren
@ 2016-01-05 22:58 ` Stephen Warren
2016-01-11 15:23 ` Simon Glass
2016-01-05 22:58 ` [U-Boot] [PATCH V3 6/7] test/py: test the shell if command Stephen Warren
` (3 subsequent siblings)
7 siblings, 1 reply; 20+ messages in thread
From: Stephen Warren @ 2016-01-05 22:58 UTC (permalink / raw)
To: u-boot
From: Stephen Warren <swarren@nvidia.com>
This tests whether the following features of the U-Boot shell:
- Execution of a directly entered command.
- Compound commands (; delimiter).
- Quoting of arguments containing spaces.
- Executing commands from environment variables.
Signed-off-by: Stephen Warren <swarren@nvidia.com>
---
v3:
- Move test scripts into a sub-directory.
Suggested by Michal Simek.
- s/uboot/u[-_]boot/g. Suggested by Simon Glass.
- s/"/'/g. Suggested by Simon Glass.
- Add more documentation. Suggested by Simon Glass.
- Move relevant edits to command_ut.c into this patch from a later one.
Suggested by Simon Glass.
---
test/command_ut.c | 16 ---------------
test/py/tests/test_shell_basics.py | 42 ++++++++++++++++++++++++++++++++++++++
2 files changed, 42 insertions(+), 16 deletions(-)
create mode 100644 test/py/tests/test_shell_basics.py
diff --git a/test/command_ut.c b/test/command_ut.c
index c086abe3ed3e..43bd2c1771fe 100644
--- a/test/command_ut.c
+++ b/test/command_ut.c
@@ -27,10 +27,6 @@ static int do_ut_cmd(cmd_tbl_t *cmdtp, int flag, int argc, char * const argv[])
run_command("setenv check", 0);
#endif
- /* commands separated by ; */
- run_command_list("setenv list 1; setenv list ${list}1", -1, 0);
- assert(!strcmp("11", getenv("list")));
-
/* commands separated by \n */
run_command_list("setenv list 1\n setenv list ${list}1", -1, 0);
assert(!strcmp("11", getenv("list")));
@@ -39,11 +35,6 @@ static int do_ut_cmd(cmd_tbl_t *cmdtp, int flag, int argc, char * const argv[])
run_command_list("setenv list 1${list}\n", -1, 0);
assert(!strcmp("111", getenv("list")));
- /* three commands in a row */
- run_command_list("setenv list 1\n setenv list ${list}2; "
- "setenv list ${list}3", -1, 0);
- assert(!strcmp("123", getenv("list")));
-
/* a command string with \0 in it. Stuff after \0 should be ignored */
run_command("setenv list", 0);
run_command_list(test_cmd, sizeof(test_cmd), 0);
@@ -62,13 +53,6 @@ static int do_ut_cmd(cmd_tbl_t *cmdtp, int flag, int argc, char * const argv[])
assert(run_command_list("false", -1, 0) == 1);
assert(run_command_list("echo", -1, 0) == 0);
- run_command("setenv foo 'setenv monty 1; setenv python 2'", 0);
- run_command("run foo", 0);
- assert(getenv("monty") != NULL);
- assert(!strcmp("1", getenv("monty")));
- assert(getenv("python") != NULL);
- assert(!strcmp("2", getenv("python")));
-
#ifdef CONFIG_SYS_HUSH_PARSER
run_command("setenv foo 'setenv black 1\nsetenv adder 2'", 0);
run_command("run foo", 0);
diff --git a/test/py/tests/test_shell_basics.py b/test/py/tests/test_shell_basics.py
new file mode 100644
index 000000000000..719ce611d71c
--- /dev/null
+++ b/test/py/tests/test_shell_basics.py
@@ -0,0 +1,42 @@
+# Copyright (c) 2015-2016, NVIDIA CORPORATION. All rights reserved.
+#
+# SPDX-License-Identifier: GPL-2.0
+
+# Test basic shell functionality, such as commands separate by semi-colons.
+
+def test_shell_execute(u_boot_console):
+ '''Test any shell command.'''
+
+ response = u_boot_console.run_command('echo hello')
+ assert response.strip() == 'hello'
+
+def test_shell_semicolon_two(u_boot_console):
+ '''Test two shell commands separate by a semi-colon.'''
+
+ cmd = 'echo hello; echo world'
+ response = u_boot_console.run_command(cmd)
+ # This validation method ignores the exact whitespace between the strings
+ assert response.index('hello') < response.index('world')
+
+def test_shell_semicolon_three(u_boot_console):
+ '''Test three shell commands separate by a semi-colon, with variable
+ expansion dependencies between them.'''
+
+ cmd = 'setenv list 1; setenv list ${list}2; setenv list ${list}3; ' + \
+ 'echo ${list}'
+ response = u_boot_console.run_command(cmd)
+ assert response.strip() == '123'
+ u_boot_console.run_command('setenv list')
+
+def test_shell_run(u_boot_console):
+ '''Test the "run" shell command.'''
+
+ u_boot_console.run_command('setenv foo \"setenv monty 1; setenv python 2\"')
+ u_boot_console.run_command('run foo')
+ response = u_boot_console.run_command('echo $monty')
+ assert response.strip() == '1'
+ response = u_boot_console.run_command('echo $python')
+ assert response.strip() == '2'
+ u_boot_console.run_command('setenv foo')
+ u_boot_console.run_command('setenv monty')
+ u_boot_console.run_command('setenv python')
--
2.6.4
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [U-Boot] [PATCH V3 6/7] test/py: test the shell if command
2016-01-05 22:58 [U-Boot] [PATCH V3 1/7] test/py: Implement pytest infrastructure Stephen Warren
` (3 preceding siblings ...)
2016-01-05 22:58 ` [U-Boot] [PATCH V3 5/7] test/py: add test of basic shell functionality Stephen Warren
@ 2016-01-05 22:58 ` Stephen Warren
2016-01-11 15:23 ` Simon Glass
2016-01-05 22:58 ` [U-Boot] [PATCH V3 7/7] test/py: test the ums command Stephen Warren
` (2 subsequent siblings)
7 siblings, 1 reply; 20+ messages in thread
From: Stephen Warren @ 2016-01-05 22:58 UTC (permalink / raw)
To: u-boot
From: Stephen Warren <swarren@nvidia.com>
Migrate all most tests from command_ut.c into the Python test system.
This allows the tests to be run against any U-Boot binary that supports
the if command (i.e. where hush is enabled) without requiring that
binary to be permanently bloated with the code from command_ut.
Some tests in command_ut.c can only be executed from C code, since they
test internal (more unit-level) features of various U-Boot APIs. The
migrated tests can all operate directly from the U-Boot console.
Signed-off-by: Stephen Warren <swarren@nvidia.com>
---
v3:
- Move test scripts into a sub-directory.
Suggested by Michal Simek.
- s/uboot/u[-_]boot/g. Suggested by Simon Glass.
- s/"/'/g. Suggested by Simon Glass.
- Add more documentation. Suggested by Simon Glass.
- Remove inclusion of <os.h>. Suggested by Simon Glass.
---
test/command_ut.c | 116 ----------------------------
test/py/tests/test_hush_if_test.py | 154 +++++++++++++++++++++++++++++++++++++
2 files changed, 154 insertions(+), 116 deletions(-)
create mode 100644 test/py/tests/test_hush_if_test.py
diff --git a/test/command_ut.c b/test/command_ut.c
index 43bd2c1771fe..54bf62b9bc30 100644
--- a/test/command_ut.c
+++ b/test/command_ut.c
@@ -7,9 +7,6 @@
#define DEBUG
#include <common.h>
-#ifdef CONFIG_SANDBOX
-#include <os.h>
-#endif
static const char test_cmd[] = "setenv list 1\n setenv list ${list}2; "
"setenv list ${list}3\0"
@@ -20,13 +17,6 @@ static int do_ut_cmd(cmd_tbl_t *cmdtp, int flag, int argc, char * const argv[])
printf("%s: Testing commands\n", __func__);
run_command("env default -f -a", 0);
- /* make sure that compound statements work */
-#ifdef CONFIG_SYS_HUSH_PARSER
- run_command("if test -n ${single} ; then setenv check 1; fi", 0);
- assert(!strcmp("1", getenv("check")));
- run_command("setenv check", 0);
-#endif
-
/* commands separated by \n */
run_command_list("setenv list 1\n setenv list ${list}1", -1, 0);
assert(!strcmp("11", getenv("list")));
@@ -60,112 +50,6 @@ static int do_ut_cmd(cmd_tbl_t *cmdtp, int flag, int argc, char * const argv[])
assert(!strcmp("1", getenv("black")));
assert(getenv("adder") != NULL);
assert(!strcmp("2", getenv("adder")));
-
- /* Test the 'test' command */
-
-#define HUSH_TEST(name, expr, expected_result) \
- run_command("if test " expr " ; then " \
- "setenv " #name "_" #expected_result " y; else " \
- "setenv " #name "_" #expected_result " n; fi", 0); \
- assert(!strcmp(#expected_result, getenv(#name "_" #expected_result))); \
- setenv(#name "_" #expected_result, NULL);
-
- /* Basic operators */
- HUSH_TEST(streq, "aaa = aaa", y);
- HUSH_TEST(streq, "aaa = bbb", n);
-
- HUSH_TEST(strneq, "aaa != bbb", y);
- HUSH_TEST(strneq, "aaa != aaa", n);
-
- HUSH_TEST(strlt, "aaa < bbb", y);
- HUSH_TEST(strlt, "bbb < aaa", n);
-
- HUSH_TEST(strgt, "bbb > aaa", y);
- HUSH_TEST(strgt, "aaa > bbb", n);
-
- HUSH_TEST(eq, "123 -eq 123", y);
- HUSH_TEST(eq, "123 -eq 456", n);
-
- HUSH_TEST(ne, "123 -ne 456", y);
- HUSH_TEST(ne, "123 -ne 123", n);
-
- HUSH_TEST(lt, "123 -lt 456", y);
- HUSH_TEST(lt_eq, "123 -lt 123", n);
- HUSH_TEST(lt, "456 -lt 123", n);
-
- HUSH_TEST(le, "123 -le 456", y);
- HUSH_TEST(le_eq, "123 -le 123", y);
- HUSH_TEST(le, "456 -le 123", n);
-
- HUSH_TEST(gt, "456 -gt 123", y);
- HUSH_TEST(gt_eq, "123 -gt 123", n);
- HUSH_TEST(gt, "123 -gt 456", n);
-
- HUSH_TEST(ge, "456 -ge 123", y);
- HUSH_TEST(ge_eq, "123 -ge 123", y);
- HUSH_TEST(ge, "123 -ge 456", n);
-
- HUSH_TEST(z, "-z \"\"", y);
- HUSH_TEST(z, "-z \"aaa\"", n);
-
- HUSH_TEST(n, "-n \"aaa\"", y);
- HUSH_TEST(n, "-n \"\"", n);
-
- /* Inversion of simple tests */
- HUSH_TEST(streq_inv, "! aaa = aaa", n);
- HUSH_TEST(streq_inv, "! aaa = bbb", y);
-
- HUSH_TEST(streq_inv_inv, "! ! aaa = aaa", y);
- HUSH_TEST(streq_inv_inv, "! ! aaa = bbb", n);
-
- /* Binary operators */
- HUSH_TEST(or_0_0, "aaa != aaa -o bbb != bbb", n);
- HUSH_TEST(or_0_1, "aaa != aaa -o bbb = bbb", y);
- HUSH_TEST(or_1_0, "aaa = aaa -o bbb != bbb", y);
- HUSH_TEST(or_1_1, "aaa = aaa -o bbb = bbb", y);
-
- HUSH_TEST(and_0_0, "aaa != aaa -a bbb != bbb", n);
- HUSH_TEST(and_0_1, "aaa != aaa -a bbb = bbb", n);
- HUSH_TEST(and_1_0, "aaa = aaa -a bbb != bbb", n);
- HUSH_TEST(and_1_1, "aaa = aaa -a bbb = bbb", y);
-
- /* Inversion within binary operators */
- HUSH_TEST(or_0_0_inv, "! aaa != aaa -o ! bbb != bbb", y);
- HUSH_TEST(or_0_1_inv, "! aaa != aaa -o ! bbb = bbb", y);
- HUSH_TEST(or_1_0_inv, "! aaa = aaa -o ! bbb != bbb", y);
- HUSH_TEST(or_1_1_inv, "! aaa = aaa -o ! bbb = bbb", n);
-
- HUSH_TEST(or_0_0_inv_inv, "! ! aaa != aaa -o ! ! bbb != bbb", n);
- HUSH_TEST(or_0_1_inv_inv, "! ! aaa != aaa -o ! ! bbb = bbb", y);
- HUSH_TEST(or_1_0_inv_inv, "! ! aaa = aaa -o ! ! bbb != bbb", y);
- HUSH_TEST(or_1_1_inv_inv, "! ! aaa = aaa -o ! ! bbb = bbb", y);
-
- setenv("ut_var_nonexistent", NULL);
- setenv("ut_var_exists", "1");
- HUSH_TEST(z_varexp_quoted, "-z \"$ut_var_nonexistent\"", y);
- HUSH_TEST(z_varexp_quoted, "-z \"$ut_var_exists\"", n);
- setenv("ut_var_exists", NULL);
-
- run_command("setenv ut_var_space \" \"", 0);
- assert(!strcmp(getenv("ut_var_space"), " "));
- run_command("setenv ut_var_test $ut_var_space", 0);
- assert(!getenv("ut_var_test"));
- run_command("setenv ut_var_test \"$ut_var_space\"", 0);
- assert(!strcmp(getenv("ut_var_test"), " "));
- run_command("setenv ut_var_test \" 1${ut_var_space}${ut_var_space} 2 \"", 0);
- assert(!strcmp(getenv("ut_var_test"), " 1 2 "));
- setenv("ut_var_space", NULL);
- setenv("ut_var_test", NULL);
-
-#ifdef CONFIG_SANDBOX
- /* File existence */
- HUSH_TEST(e, "-e hostfs - creating_this_file_breaks_uboot_unit_test", n);
- run_command("sb save hostfs - creating_this_file_breaks_uboot_unit_test 0 1", 0);
- HUSH_TEST(e, "-e hostfs - creating_this_file_breaks_uboot_unit_test", y);
- /* Perhaps this could be replaced by an "rm" shell command one day */
- assert(!os_unlink("creating_this_file_breaks_uboot_unit_test"));
- HUSH_TEST(e, "-e hostfs - creating_this_file_breaks_uboot_unit_test", n);
-#endif
#endif
assert(run_command("", 0) == 0);
diff --git a/test/py/tests/test_hush_if_test.py b/test/py/tests/test_hush_if_test.py
new file mode 100644
index 000000000000..cf4c3aeeb731
--- /dev/null
+++ b/test/py/tests/test_hush_if_test.py
@@ -0,0 +1,154 @@
+# Copyright (c) 2015-2016, NVIDIA CORPORATION. All rights reserved.
+#
+# SPDX-License-Identifier: GPL-2.0
+
+# Test operation of the "if" shell command.
+
+import os
+import os.path
+import pytest
+
+# The list of "if test" conditions to test.
+subtests = (
+ # Base if functionality.
+
+ ('true', True),
+ ('false', False),
+
+ # Basic operators.
+
+ ('test aaa = aaa', True),
+ ('test aaa = bbb', False),
+
+ ('test aaa != bbb', True),
+ ('test aaa != aaa', False),
+
+ ('test aaa < bbb', True),
+ ('test bbb < aaa', False),
+
+ ('test bbb > aaa', True),
+ ('test aaa > bbb', False),
+
+ ('test 123 -eq 123', True),
+ ('test 123 -eq 456', False),
+
+ ('test 123 -ne 456', True),
+ ('test 123 -ne 123', False),
+
+ ('test 123 -lt 456', True),
+ ('test 123 -lt 123', False),
+ ('test 456 -lt 123', False),
+
+ ('test 123 -le 456', True),
+ ('test 123 -le 123', True),
+ ('test 456 -le 123', False),
+
+ ('test 456 -gt 123', True),
+ ('test 123 -gt 123', False),
+ ('test 123 -gt 456', False),
+
+ ('test 456 -ge 123', True),
+ ('test 123 -ge 123', True),
+ ('test 123 -ge 456', False),
+
+ ('test -z ""', True),
+ ('test -z "aaa"', False),
+
+ ('test -n "aaa"', True),
+ ('test -n ""', False),
+
+ # Inversion of simple tests.
+
+ ('test ! aaa = aaa', False),
+ ('test ! aaa = bbb', True),
+ ('test ! ! aaa = aaa', True),
+ ('test ! ! aaa = bbb', False),
+
+ # Binary operators.
+
+ ('test aaa != aaa -o bbb != bbb', False),
+ ('test aaa != aaa -o bbb = bbb', True),
+ ('test aaa = aaa -o bbb != bbb', True),
+ ('test aaa = aaa -o bbb = bbb', True),
+
+ ('test aaa != aaa -a bbb != bbb', False),
+ ('test aaa != aaa -a bbb = bbb', False),
+ ('test aaa = aaa -a bbb != bbb', False),
+ ('test aaa = aaa -a bbb = bbb', True),
+
+ # Inversion within binary operators.
+
+ ('test ! aaa != aaa -o ! bbb != bbb', True),
+ ('test ! aaa != aaa -o ! bbb = bbb', True),
+ ('test ! aaa = aaa -o ! bbb != bbb', True),
+ ('test ! aaa = aaa -o ! bbb = bbb', False),
+
+ ('test ! ! aaa != aaa -o ! ! bbb != bbb', False),
+ ('test ! ! aaa != aaa -o ! ! bbb = bbb', True),
+ ('test ! ! aaa = aaa -o ! ! bbb != bbb', True),
+ ('test ! ! aaa = aaa -o ! ! bbb = bbb', True),
+
+ # -z operator.
+
+ ('test -z "$ut_var_nonexistent"', True),
+ ('test -z "$ut_var_exists"', False),
+)
+
+def exec_hush_if(u_boot_console, expr, result):
+ '''Execute a shell "if" command, and validate its result.'''
+
+ cmd = 'if ' + expr + '; then echo true; else echo false; fi'
+ response = u_boot_console.run_command(cmd)
+ assert response.strip() == str(result).lower()
+
+ at pytest.mark.buildconfigspec('sys_hush_parser')
+def test_hush_if_test_setup(u_boot_console):
+ '''Set up environment variables used during the "if" tests.'''
+
+ u_boot_console.run_command('setenv ut_var_nonexistent')
+ u_boot_console.run_command('setenv ut_var_exists 1')
+
+ at pytest.mark.buildconfigspec('sys_hush_parser')
+ at pytest.mark.parametrize('expr,result', subtests)
+def test_hush_if_test(u_boot_console, expr, result):
+ '''Test a single "if test" condition.'''
+
+ exec_hush_if(u_boot_console, expr, result)
+
+ at pytest.mark.buildconfigspec('sys_hush_parser')
+def test_hush_if_test_teardown(u_boot_console):
+ '''Clean up environment variables used during the "if" tests.'''
+
+ u_boot_console.run_command('setenv ut_var_exists')
+
+ at pytest.mark.buildconfigspec('sys_hush_parser')
+# We might test this on real filesystems via UMS, DFU, 'save', etc.
+# Of those, only UMS currently allows file removal though.
+ at pytest.mark.boardspec('sandbox')
+def test_hush_if_test_host_file_exists(u_boot_console):
+ '''Test the "if test -e" shell command.'''
+
+ test_file = u_boot_console.config.result_dir + \
+ '/creating_this_file_breaks_u_boot_tests'
+
+ try:
+ os.unlink(test_file)
+ except:
+ pass
+ assert not os.path.exists(test_file)
+
+ expr = 'test -e hostfs - ' + test_file
+ exec_hush_if(u_boot_console, expr, False)
+
+ try:
+ with file(test_file, 'wb'):
+ pass
+ assert os.path.exists(test_file)
+
+ expr = 'test -e hostfs - ' + test_file
+ exec_hush_if(u_boot_console, expr, True)
+ finally:
+ os.unlink(test_file)
+
+ expr = 'test -e hostfs - ' + test_file
+ exec_hush_if(u_boot_console, expr, False)
--
2.6.4
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [U-Boot] [PATCH V3 7/7] test/py: test the ums command
2016-01-05 22:58 [U-Boot] [PATCH V3 1/7] test/py: Implement pytest infrastructure Stephen Warren
` (4 preceding siblings ...)
2016-01-05 22:58 ` [U-Boot] [PATCH V3 6/7] test/py: test the shell if command Stephen Warren
@ 2016-01-05 22:58 ` Stephen Warren
2016-01-11 15:23 ` Simon Glass
2016-01-08 18:13 ` [U-Boot] [PATCH V3 1/7] test/py: Implement pytest infrastructure Stephen Warren
2016-01-11 15:23 ` Simon Glass
7 siblings, 1 reply; 20+ messages in thread
From: Stephen Warren @ 2016-01-05 22:58 UTC (permalink / raw)
To: u-boot
From: Stephen Warren <swarren@nvidia.com>
This test invokes the "ums" command in U-Boot, and validates that a USB
storage device is enumerated on the test host system, and can be read
from.
Signed-off-by: Stephen Warren <swarren@nvidia.com>
---
v3:
- Move test scripts into a sub-directory.
Suggested by Michal Simek.
- s/uboot/u[-_]boot/g. Suggested by Simon Glass.
- s/"/'/g. Suggested by Simon Glass.
- Add more documentation. Suggested by Simon Glass.
- Rename boardenv USB device node variable to avoid naming conflicts if
we add support for other types of USB device later (e.g. CDC ACM, CDC
Ethernet, DFU).
---
test/py/tests/test_ums.py | 94 +++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 94 insertions(+)
create mode 100644 test/py/tests/test_ums.py
diff --git a/test/py/tests/test_ums.py b/test/py/tests/test_ums.py
new file mode 100644
index 000000000000..a137221c7a5b
--- /dev/null
+++ b/test/py/tests/test_ums.py
@@ -0,0 +1,94 @@
+# Copyright (c) 2015-2016, NVIDIA CORPORATION. All rights reserved.
+#
+# SPDX-License-Identifier: GPL-2.0
+
+# Test U-Boot's "ums" command. At present, this test only ensures that a UMS
+# device can be enumerated by the host/test machine. In the future, this test
+# should be enhanced to validate disk IO.
+
+import os
+import pytest
+import time
+
+'''
+Note: This test relies on:
+
+a) boardenv_* to contain configuration values to define which USB ports are
+available for testing. Without this, this test will be automatically skipped.
+For example:
+
+env__usb_dev_ports = (
+ {'tgt_usb_ctlr': '0', 'host_ums_dev_node': '/dev/disk/by-path/pci-0000:00:14.0-usb-0:13:1.0-scsi-0:0:0:0'},
+)
+
+env__block_devs = (
+ {'type': 'mmc', 'id': '0'}, # eMMC; always present
+ {'type': 'mmc', 'id': '1'}, # SD card; present since I plugged one in
+)
+
+b) udev rules to set permissions on devices nodes, so that sudo is not
+required. For example:
+
+ACTION=="add", SUBSYSTEM=="block", SUBSYSTEMS=="usb", KERNELS=="3-13", MODE:="666"
+
+(You may wish to change the group ID instead of setting the permissions wide
+open. All that matters is that the user ID running the test can access the
+device.)
+'''
+
+def open_ums_device(host_ums_dev_node):
+ '''Attempt to open a device node, returning either the opened file handle,
+ or None on any error.'''
+
+ try:
+ return open(host_ums_dev_node, 'rb')
+ except:
+ return None
+
+def wait_for_ums_device(host_ums_dev_node):
+ '''Continually attempt to open the device node exported by the "ums"
+ command, and either return the opened file handle, or raise an exception
+ after a timeout.'''
+
+ for i in xrange(100):
+ fh = open_ums_device(host_ums_dev_node)
+ if fh:
+ return fh
+ time.sleep(0.1)
+ raise Exception('UMS device did not appear')
+
+def wait_for_ums_device_gone(host_ums_dev_node):
+ '''Continually attempt to open the device node exported by the "ums"
+ command, and either return once the device has disappeared, or raise an
+ exception if it does not before a timeout occurs.'''
+
+ for i in xrange(100):
+ fh = open_ums_device(host_ums_dev_node)
+ if not fh:
+ return
+ fh.close()
+ time.sleep(0.1)
+ raise Exception('UMS device did not disappear')
+
+ at pytest.mark.buildconfigspec('cmd_usb_mass_storage')
+def test_ums(u_boot_console, env__usb_dev_port, env__block_devs):
+ '''Test the "ums" command; the host system must be able to enumerate a UMS
+ device when "ums" is running, and this device must disappear when "ums" is
+ aborted.'''
+
+ tgt_usb_ctlr = env__usb_dev_port['tgt_usb_ctlr']
+ host_ums_dev_node = env__usb_dev_port['host_ums_dev_node']
+
+ # We're interested in testing USB device mode on each port, not the cross-
+ # product of that with each device. So, just pick the first entry in the
+ # device list here. We'll test each block device somewhere else.
+ tgt_dev_type = env__block_devs[0]['type']
+ tgt_dev_id = env__block_devs[0]['id']
+
+ cmd = 'ums %s %s %s' % (tgt_usb_ctlr, tgt_dev_type, tgt_dev_id)
+ u_boot_console.run_command('ums 0 mmc 0', wait_for_prompt=False)
+ fh = wait_for_ums_device(host_ums_dev_node)
+ fh.read(4096)
+ fh.close()
+ u_boot_console.ctrlc()
+ wait_for_ums_device_gone(host_ums_dev_node)
--
2.6.4
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [U-Boot] [PATCH V3 1/7] test/py: Implement pytest infrastructure
2016-01-05 22:58 [U-Boot] [PATCH V3 1/7] test/py: Implement pytest infrastructure Stephen Warren
` (5 preceding siblings ...)
2016-01-05 22:58 ` [U-Boot] [PATCH V3 7/7] test/py: test the ums command Stephen Warren
@ 2016-01-08 18:13 ` Stephen Warren
2016-01-08 18:32 ` Michal Simek
2016-01-11 10:45 ` Michal Simek
2016-01-11 15:23 ` Simon Glass
7 siblings, 2 replies; 20+ messages in thread
From: Stephen Warren @ 2016-01-08 18:13 UTC (permalink / raw)
To: u-boot
On 01/05/2016 03:58 PM, Stephen Warren wrote:
> This tool aims to test U-Boot by executing U-Boot shell commands using the
> console interface. A single top-level script exists to execute or attach
> to the U-Boot console, run the entire script of tests against it, and
> summarize the results. Advantages of this approach are:
>
> - Testing is performed in the same way a user or script would interact
> with U-Boot; there can be no disconnect.
> - There is no need to write or embed test-related code into U-Boot itself.
> It is asserted that writing test-related code in Python is simpler and
> more flexible that writing it all in C.
> - It is reasonably simple to interact with U-Boot in this way.
>
> A few simple tests are provided as examples. Soon, we should convert as
> many as possible of the other tests in test/* and test/cmd_ut.c too.
>
> The hook scripts, relay control utilities, and udev rules I use for my
> own HW setup are published at https://github.com/swarren/uboot-test-hooks.
>
> See README.md for more details!
It looks like I need to send a v4 of this, since I renamed a Python
class but forgot to update all users of it. I didn't notice this, since
I had the old module lying around as a *.pyc file, so the old name worked:-(
I also have a couple of minor fixes to roll in that make the scripts
work better under a continuous integration environment (which doesn't
have a controlling TTY set when the scripts run, which need a minor
tweak to the Spawn code).
^ permalink raw reply [flat|nested] 20+ messages in thread
* [U-Boot] [PATCH V3 1/7] test/py: Implement pytest infrastructure
2016-01-08 18:13 ` [U-Boot] [PATCH V3 1/7] test/py: Implement pytest infrastructure Stephen Warren
@ 2016-01-08 18:32 ` Michal Simek
2016-01-11 15:25 ` Simon Glass
2016-01-11 10:45 ` Michal Simek
1 sibling, 1 reply; 20+ messages in thread
From: Michal Simek @ 2016-01-08 18:32 UTC (permalink / raw)
To: u-boot
On 8.1.2016 19:13, Stephen Warren wrote:
> On 01/05/2016 03:58 PM, Stephen Warren wrote:
>> This tool aims to test U-Boot by executing U-Boot shell commands using
>> the
>> console interface. A single top-level script exists to execute or attach
>> to the U-Boot console, run the entire script of tests against it, and
>> summarize the results. Advantages of this approach are:
>>
>> - Testing is performed in the same way a user or script would interact
>> with U-Boot; there can be no disconnect.
>> - There is no need to write or embed test-related code into U-Boot
>> itself.
>> It is asserted that writing test-related code in Python is simpler and
>> more flexible that writing it all in C.
>> - It is reasonably simple to interact with U-Boot in this way.
>>
>> A few simple tests are provided as examples. Soon, we should convert as
>> many as possible of the other tests in test/* and test/cmd_ut.c too.
>>
>> The hook scripts, relay control utilities, and udev rules I use for my
>> own HW setup are published at
>> https://github.com/swarren/uboot-test-hooks.
>>
>> See README.md for more details!
>
> It looks like I need to send a v4 of this, since I renamed a Python
> class but forgot to update all users of it. I didn't notice this, since
> I had the old module lying around as a *.pyc file, so the old name
> worked:-(
>
> I also have a couple of minor fixes to roll in that make the scripts
> work better under a continuous integration environment (which doesn't
> have a controlling TTY set when the scripts run, which need a minor
> tweak to the Spawn code).
btw: You can also add a note about kermit whcih I do use and works good.
Thanks,
Michal
--
Michal Simek, Ing. (M.Eng), OpenPGP -> KeyID: FE3D1F91
w: www.monstr.eu p: +42-0-721842854
Maintainer of Linux kernel - Microblaze cpu - http://www.monstr.eu/fdt/
Maintainer of Linux kernel - Xilinx Zynq ARM architecture
Microblaze U-BOOT custodian and responsible for u-boot arm zynq platform
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: OpenPGP digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20160108/c21bbd67/attachment.sig>
^ permalink raw reply [flat|nested] 20+ messages in thread
* [U-Boot] [PATCH V3 1/7] test/py: Implement pytest infrastructure
2016-01-08 18:13 ` [U-Boot] [PATCH V3 1/7] test/py: Implement pytest infrastructure Stephen Warren
2016-01-08 18:32 ` Michal Simek
@ 2016-01-11 10:45 ` Michal Simek
2016-01-11 17:00 ` Stephen Warren
1 sibling, 1 reply; 20+ messages in thread
From: Michal Simek @ 2016-01-11 10:45 UTC (permalink / raw)
To: u-boot
Hi Stephen,
On 8.1.2016 19:13, Stephen Warren wrote:
> On 01/05/2016 03:58 PM, Stephen Warren wrote:
>> This tool aims to test U-Boot by executing U-Boot shell commands using
>> the
>> console interface. A single top-level script exists to execute or attach
>> to the U-Boot console, run the entire script of tests against it, and
>> summarize the results. Advantages of this approach are:
>>
>> - Testing is performed in the same way a user or script would interact
>> with U-Boot; there can be no disconnect.
>> - There is no need to write or embed test-related code into U-Boot
>> itself.
>> It is asserted that writing test-related code in Python is simpler and
>> more flexible that writing it all in C.
>> - It is reasonably simple to interact with U-Boot in this way.
>>
>> A few simple tests are provided as examples. Soon, we should convert as
>> many as possible of the other tests in test/* and test/cmd_ut.c too.
>>
>> The hook scripts, relay control utilities, and udev rules I use for my
>> own HW setup are published at
>> https://github.com/swarren/uboot-test-hooks.
>>
>> See README.md for more details!
>
> It looks like I need to send a v4 of this, since I renamed a Python
> class but forgot to update all users of it. I didn't notice this, since
> I had the old module lying around as a *.pyc file, so the old name
> worked:-(
yep.
>
> I also have a couple of minor fixes to roll in that make the scripts
> work better under a continuous integration environment (which doesn't
> have a controlling TTY set when the scripts run, which need a minor
> tweak to the Spawn code).
I have cherry picked 3 patches from your repo.
7813ccad9ed2 test/py: reset SIGHUP handler in child processes
a2ccb34de3f5 test/py: fix ubspawn rename fallout
6dbcd7408f9e test/py: add a test for the sleep command
sleep one is missing test for cmd_misc which enables that.
There is still problem with handling special characters.
For MB I have compilation error and %5e is shown instead of ^
For example:
+make O=/home/monstr/data/disk/u-boot/build-microblaze-generic -s
microblaze-generic_defconfig
+make O=/home/monstr/data/disk/u-boot/build-microblaze-generic -s -j8
../drivers/gpio/xilinx_gpio.c: In function 'xilinx_gpio_ofdata_to_platdata':
../drivers/gpio/xilinx_gpio.c:400:13: warning: assignment makes pointer
from integer without a cast
priv->regs = dev_get_addr(dev);
%5e
But the rest looks good.
Thanks,
Michal
--
Michal Simek, Ing. (M.Eng), OpenPGP -> KeyID: FE3D1F91
w: www.monstr.eu p: +42-0-721842854
Maintainer of Linux kernel - Microblaze cpu - http://www.monstr.eu/fdt/
Maintainer of Linux kernel - Xilinx Zynq ARM architecture
Microblaze U-BOOT custodian and responsible for u-boot arm zynq platform
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: OpenPGP digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20160111/0651d0d1/attachment.sig>
^ permalink raw reply [flat|nested] 20+ messages in thread
* [U-Boot] [PATCH V3 1/7] test/py: Implement pytest infrastructure
2016-01-05 22:58 [U-Boot] [PATCH V3 1/7] test/py: Implement pytest infrastructure Stephen Warren
` (6 preceding siblings ...)
2016-01-08 18:13 ` [U-Boot] [PATCH V3 1/7] test/py: Implement pytest infrastructure Stephen Warren
@ 2016-01-11 15:23 ` Simon Glass
7 siblings, 0 replies; 20+ messages in thread
From: Simon Glass @ 2016-01-11 15:23 UTC (permalink / raw)
To: u-boot
On 5 January 2016 at 15:58, Stephen Warren <swarren@wwwdotorg.org> wrote:
> This tool aims to test U-Boot by executing U-Boot shell commands using the
> console interface. A single top-level script exists to execute or attach
> to the U-Boot console, run the entire script of tests against it, and
> summarize the results. Advantages of this approach are:
>
> - Testing is performed in the same way a user or script would interact
> with U-Boot; there can be no disconnect.
> - There is no need to write or embed test-related code into U-Boot itself.
> It is asserted that writing test-related code in Python is simpler and
> more flexible that writing it all in C.
> - It is reasonably simple to interact with U-Boot in this way.
>
> A few simple tests are provided as examples. Soon, we should convert as
> many as possible of the other tests in test/* and test/cmd_ut.c too.
>
> The hook scripts, relay control utilities, and udev rules I use for my
> own HW setup are published at https://github.com/swarren/uboot-test-hooks.
>
> See README.md for more details!
>
> Signed-off-by: Stephen Warren <swarren@wwwdotorg.org>
> Signed-off-by: Stephen Warren <swarren@nvidia.com>
> Tested-by: Michal Simek <michal.simek@xilinx.com>
> Tested-by: Simon Glass <sjg@chromium.org>
> ---
> v3:
> - Rework HTML log generation so that TAB characters render as expected.
> Suggested by Michal Simek.
> - Move test scripts into a sub-directory.
> Suggested by Michal Simek.
> - s/uboot/u[-_]boot/g. Suggested by Simon Glass.
> - s/"/'/g. Suggested by Simon Glass.
> - Typo fixes.
> - Add more documentation. Suggested by Simon Glass.
> - Make "notes" in the log file be <pre> so that their formatting is
> preserved. This is useful for large notes such as exception dumps.
>
> v2:
> - Many fixes and tweaks have been squashed in. Separated out some of
> the tests into separate commits, and added some more tests.
> ---
> test/py/.gitignore | 1 +
> test/py/README.md | 300 ++++++++++++++++++++++++++++++
> test/py/conftest.py | 335 ++++++++++++++++++++++++++++++++++
> test/py/multiplexed_log.css | 88 +++++++++
> test/py/multiplexed_log.py | 335 ++++++++++++++++++++++++++++++++++
> test/py/pytest.ini | 11 ++
> test/py/test.py | 32 ++++
> test/py/tests/test_000_version.py | 20 ++
> test/py/tests/test_help.py | 9 +
> test/py/tests/test_unknown_cmd.py | 14 ++
> test/py/u_boot_console_base.py | 260 ++++++++++++++++++++++++++
> test/py/u_boot_console_exec_attach.py | 51 ++++++
> test/py/u_boot_console_sandbox.py | 48 +++++
> test/py/u_boot_spawn.py | 123 +++++++++++++
> 14 files changed, 1627 insertions(+)
> create mode 100644 test/py/.gitignore
> create mode 100644 test/py/README.md
> create mode 100644 test/py/conftest.py
> create mode 100644 test/py/multiplexed_log.css
> create mode 100644 test/py/multiplexed_log.py
> create mode 100644 test/py/pytest.ini
> create mode 100755 test/py/test.py
> create mode 100644 test/py/tests/test_000_version.py
> create mode 100644 test/py/tests/test_help.py
> create mode 100644 test/py/tests/test_unknown_cmd.py
> create mode 100644 test/py/u_boot_console_base.py
> create mode 100644 test/py/u_boot_console_exec_attach.py
> create mode 100644 test/py/u_boot_console_sandbox.py
> create mode 100644 test/py/u_boot_spawn.py
Looks good!
Acked-by: Simon Glass <sjg@chromium.org>
One comment about function / class comments. If you like at
buildman/patman they use this sort of format:
def RunCommit(self, commit_upto, brd, work_dir, do_config, force_build,
force_build_failures):
"""Build a particular commit.
If the build is already done, and we are not forcing a build, we skip
the build and just return the previously-saved results.
Args:
commit_upto: Commit number to build (0...n-1)
brd: Board object to build
work_dir: Directory to which the source will be checked out
do_config: True to run a make <board>_defconfig on the source
force_build: Force a build even if one was previously done
force_build_failures: Force a bulid if the previous result showed
failure
Returns:
tuple containing:
- CommandResult object containing the results of the build
- boolean indicating whether 'make config' is still needed
"""
So double quotes, and the first line describes the function in a very
simple way, a bit like a git commit subject. The arguments and return
value are specifically called out, as we do in the C code.
For consistency I'd suggest doing the same.
Regards,
Simon
^ permalink raw reply [flat|nested] 20+ messages in thread
* [U-Boot] [PATCH V3 2/7] test/py: test that sandbox exits when asked
2016-01-05 22:58 ` [U-Boot] [PATCH V3 2/7] test/py: test that sandbox exits when asked Stephen Warren
@ 2016-01-11 15:23 ` Simon Glass
0 siblings, 0 replies; 20+ messages in thread
From: Simon Glass @ 2016-01-11 15:23 UTC (permalink / raw)
To: u-boot
On 5 January 2016 at 15:58, Stephen Warren <swarren@wwwdotorg.org> wrote:
> Test the sandbox port's implementation of the reset command and SIGHUP
> handling. These should both cause the U-Boot process to exit gracefully.
>
> Signed-off-by: Stephen Warren <swarren@wwwdotorg.org>
> Signed-off-by: Stephen Warren <swarren@nvidia.com>
> ---
> v3:
> - Move test scripts into a sub-directory.
> Suggested by Michal Simek.
> - s/uboot/u[-_]boot/g. Suggested by Simon Glass.
> - Typo.
> - s/"/'/g. Suggested by Simon Glass.
> - Add more documentation. Suggested by Simon Glass.
> ---
> test/py/tests/test_sandbox_exit.py | 24 ++++++++++++++++++++++++
> 1 file changed, 24 insertions(+)
> create mode 100644 test/py/tests/test_sandbox_exit.py
>
Acked-by: Simon Glass <sjg@chromium.org>
^ permalink raw reply [flat|nested] 20+ messages in thread
* [U-Boot] [PATCH V3 3/7] test/py: add test of setenv/printenv/echo
2016-01-05 22:58 ` [U-Boot] [PATCH V3 3/7] test/py: add test of setenv/printenv/echo Stephen Warren
@ 2016-01-11 15:23 ` Simon Glass
0 siblings, 0 replies; 20+ messages in thread
From: Simon Glass @ 2016-01-11 15:23 UTC (permalink / raw)
To: u-boot
On 5 January 2016 at 15:58, Stephen Warren <swarren@wwwdotorg.org> wrote:
> This tests basic environment variable functionality.
>
> Signed-off-by: Stephen Warren <swarren@wwwdotorg.org>
> Signed-off-by: Stephen Warren <swarren@nvidia.com>
> ---
> v3:
> - Handle environment values that contain an = sign.
> Reported by Michal Simek.
> - Move test scripts into a sub-directory.
> Suggested by Michal Simek.
> - s/uboot/u[-_]boot/g. Suggested by Simon Glass.
> - s/"/'/g. Suggested by Simon Glass.
> - Add more documentation. Suggested by Simon Glass.
> - Simplify command string construction. Suggested by Simon Glass.
> - Move relevant edits to command_ut.c into this patch from a later one.
> Suggested by Simon Glass.
> ---
> test/command_ut.c | 4 --
> test/py/tests/test_env.py | 169 ++++++++++++++++++++++++++++++++++++++++++++++
> 2 files changed, 169 insertions(+), 4 deletions(-)
> create mode 100644 test/py/tests/test_env.py
Acked-by: Simon Glass <sjg@chromium.org>
^ permalink raw reply [flat|nested] 20+ messages in thread
* [U-Boot] [PATCH V3 4/7] test/py: test the md/mw commands
2016-01-05 22:58 ` [U-Boot] [PATCH V3 4/7] test/py: test the md/mw commands Stephen Warren
@ 2016-01-11 15:23 ` Simon Glass
0 siblings, 0 replies; 20+ messages in thread
From: Simon Glass @ 2016-01-11 15:23 UTC (permalink / raw)
To: u-boot
On 5 January 2016 at 15:58, Stephen Warren <swarren@wwwdotorg.org> wrote:
> This tests whether md/mw work, and affect each-other.
>
> Command repeat is also tested.
>
> test/cmd_repeat.sh is removed, since the new Python-based test does
> everything it used to.
>
> Signed-off-by: Stephen Warren <swarren@wwwdotorg.org>
> Signed-off-by: Stephen Warren <swarren@nvidia.com>
> ---
> v3:
> - Add extra mw during md test to account for the possibility that the
> test's write data may already be present in RAM.
> Suggested by Michal Simek.
> - Move test scripts into a sub-directory.
> Suggested by Michal Simek.
> - s/uboot/u[-_]boot/g. Suggested by Simon Glass.
> - s/"/'/g. Suggested by Simon Glass.
> - Add more documentation. Suggested by Simon Glass.
> ---
> test/cmd_repeat.sh | 29 -----------------------------
> test/py/tests/test_md.py | 36 ++++++++++++++++++++++++++++++++++++
> 2 files changed, 36 insertions(+), 29 deletions(-)
> delete mode 100755 test/cmd_repeat.sh
> create mode 100644 test/py/tests/test_md.py
Acked-by: Simon Glass <sjg@chromium.org>
^ permalink raw reply [flat|nested] 20+ messages in thread
* [U-Boot] [PATCH V3 5/7] test/py: add test of basic shell functionality
2016-01-05 22:58 ` [U-Boot] [PATCH V3 5/7] test/py: add test of basic shell functionality Stephen Warren
@ 2016-01-11 15:23 ` Simon Glass
0 siblings, 0 replies; 20+ messages in thread
From: Simon Glass @ 2016-01-11 15:23 UTC (permalink / raw)
To: u-boot
On 5 January 2016 at 15:58, Stephen Warren <swarren@wwwdotorg.org> wrote:
> From: Stephen Warren <swarren@nvidia.com>
>
> This tests whether the following features of the U-Boot shell:
> - Execution of a directly entered command.
> - Compound commands (; delimiter).
> - Quoting of arguments containing spaces.
> - Executing commands from environment variables.
>
> Signed-off-by: Stephen Warren <swarren@nvidia.com>
> ---
> v3:
> - Move test scripts into a sub-directory.
> Suggested by Michal Simek.
> - s/uboot/u[-_]boot/g. Suggested by Simon Glass.
> - s/"/'/g. Suggested by Simon Glass.
> - Add more documentation. Suggested by Simon Glass.
> - Move relevant edits to command_ut.c into this patch from a later one.
> Suggested by Simon Glass.
> ---
> test/command_ut.c | 16 ---------------
> test/py/tests/test_shell_basics.py | 42 ++++++++++++++++++++++++++++++++++++++
> 2 files changed, 42 insertions(+), 16 deletions(-)
> create mode 100644 test/py/tests/test_shell_basics.py
Acked-by: Simon Glass <sjg@chromium.org>
^ permalink raw reply [flat|nested] 20+ messages in thread
* [U-Boot] [PATCH V3 6/7] test/py: test the shell if command
2016-01-05 22:58 ` [U-Boot] [PATCH V3 6/7] test/py: test the shell if command Stephen Warren
@ 2016-01-11 15:23 ` Simon Glass
0 siblings, 0 replies; 20+ messages in thread
From: Simon Glass @ 2016-01-11 15:23 UTC (permalink / raw)
To: u-boot
On 5 January 2016 at 15:58, Stephen Warren <swarren@wwwdotorg.org> wrote:
> From: Stephen Warren <swarren@nvidia.com>
>
> Migrate all most tests from command_ut.c into the Python test system.
> This allows the tests to be run against any U-Boot binary that supports
> the if command (i.e. where hush is enabled) without requiring that
> binary to be permanently bloated with the code from command_ut.
>
> Some tests in command_ut.c can only be executed from C code, since they
> test internal (more unit-level) features of various U-Boot APIs. The
> migrated tests can all operate directly from the U-Boot console.
>
> Signed-off-by: Stephen Warren <swarren@nvidia.com>
> ---
> v3:
> - Move test scripts into a sub-directory.
> Suggested by Michal Simek.
> - s/uboot/u[-_]boot/g. Suggested by Simon Glass.
> - s/"/'/g. Suggested by Simon Glass.
> - Add more documentation. Suggested by Simon Glass.
> - Remove inclusion of <os.h>. Suggested by Simon Glass.
> ---
> test/command_ut.c | 116 ----------------------------
> test/py/tests/test_hush_if_test.py | 154 +++++++++++++++++++++++++++++++++++++
> 2 files changed, 154 insertions(+), 116 deletions(-)
> create mode 100644 test/py/tests/test_hush_if_test.py
Acked-by: Simon Glass <sjg@chromium.org>
^ permalink raw reply [flat|nested] 20+ messages in thread
* [U-Boot] [PATCH V3 7/7] test/py: test the ums command
2016-01-05 22:58 ` [U-Boot] [PATCH V3 7/7] test/py: test the ums command Stephen Warren
@ 2016-01-11 15:23 ` Simon Glass
0 siblings, 0 replies; 20+ messages in thread
From: Simon Glass @ 2016-01-11 15:23 UTC (permalink / raw)
To: u-boot
On 5 January 2016 at 15:58, Stephen Warren <swarren@wwwdotorg.org> wrote:
> From: Stephen Warren <swarren@nvidia.com>
>
> This test invokes the "ums" command in U-Boot, and validates that a USB
> storage device is enumerated on the test host system, and can be read
> from.
>
> Signed-off-by: Stephen Warren <swarren@nvidia.com>
> ---
> v3:
> - Move test scripts into a sub-directory.
> Suggested by Michal Simek.
> - s/uboot/u[-_]boot/g. Suggested by Simon Glass.
> - s/"/'/g. Suggested by Simon Glass.
> - Add more documentation. Suggested by Simon Glass.
> - Rename boardenv USB device node variable to avoid naming conflicts if
> we add support for other types of USB device later (e.g. CDC ACM, CDC
> Ethernet, DFU).
> ---
> test/py/tests/test_ums.py | 94 +++++++++++++++++++++++++++++++++++++++++++++++
> 1 file changed, 94 insertions(+)
> create mode 100644 test/py/tests/test_ums.py
Acked-by: Simon Glass <sjg@chromium.org>
^ permalink raw reply [flat|nested] 20+ messages in thread
* [U-Boot] [PATCH V3 1/7] test/py: Implement pytest infrastructure
2016-01-08 18:32 ` Michal Simek
@ 2016-01-11 15:25 ` Simon Glass
2016-01-11 17:04 ` Stephen Warren
0 siblings, 1 reply; 20+ messages in thread
From: Simon Glass @ 2016-01-11 15:25 UTC (permalink / raw)
To: u-boot
Hi Stephen,
On 8 January 2016 at 11:32, Michal Simek <monstr@monstr.eu> wrote:
>
> On 8.1.2016 19:13, Stephen Warren wrote:
> > On 01/05/2016 03:58 PM, Stephen Warren wrote:
> >> This tool aims to test U-Boot by executing U-Boot shell commands using
> >> the
> >> console interface. A single top-level script exists to execute or attach
> >> to the U-Boot console, run the entire script of tests against it, and
> >> summarize the results. Advantages of this approach are:
> >>
> >> - Testing is performed in the same way a user or script would interact
> >> with U-Boot; there can be no disconnect.
> >> - There is no need to write or embed test-related code into U-Boot
> >> itself.
> >> It is asserted that writing test-related code in Python is simpler and
> >> more flexible that writing it all in C.
> >> - It is reasonably simple to interact with U-Boot in this way.
> >>
> >> A few simple tests are provided as examples. Soon, we should convert as
> >> many as possible of the other tests in test/* and test/cmd_ut.c too.
> >>
> >> The hook scripts, relay control utilities, and udev rules I use for my
> >> own HW setup are published at
> >> https://github.com/swarren/uboot-test-hooks.
> >>
> >> See README.md for more details!
> >
> > It looks like I need to send a v4 of this, since I renamed a Python
> > class but forgot to update all users of it. I didn't notice this, since
> > I had the old module lying around as a *.pyc file, so the old name
> > worked:-(
> >
> > I also have a couple of minor fixes to roll in that make the scripts
> > work better under a continuous integration environment (which doesn't
> > have a controlling TTY set when the scripts run, which need a minor
> > tweak to the Spawn code).
I see this now. Do I need another dependency?
/test/py/test.py --bd sandbox --build
+make O=/usr/local/google/c/cosarm/src/third_party/u-boot/files/build-sandbox
-s sandbox_defconfig
+make O=/usr/local/google/c/cosarm/src/third_party/u-boot/files/build-sandbox
-s -j8
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR> File
"/usr/lib/python2.7/dist-packages/_pytest/main.py", line 77, in
wrap_session
INTERNALERROR> config.do_configure()
INTERNALERROR> File
"/usr/lib/python2.7/dist-packages/_pytest/config.py", line 607, in
do_configure
INTERNALERROR> self.hook.pytest_configure(config=self)
INTERNALERROR> File
"/usr/lib/python2.7/dist-packages/_pytest/core.py", line 376, in
__call__
INTERNALERROR> return self._docall(methods, kwargs)
INTERNALERROR> File
"/usr/lib/python2.7/dist-packages/_pytest/core.py", line 387, in
_docall
INTERNALERROR> res = mc.execute()
INTERNALERROR> File
"/usr/lib/python2.7/dist-packages/_pytest/core.py", line 288, in
execute
INTERNALERROR> res = method(**kwargs)
INTERNALERROR> File
"/usr/local/google/c/cosarm/src/third_party/u-boot/files/test/py/conftest.py",
line 163, in pytest_configure
INTERNALERROR> import u_boot_console_sandbox
INTERNALERROR> File
"/usr/local/google/c/cosarm/src/third_party/u-boot/files/test/py/u_boot_console_sandbox.py",
line 9, in <module>
INTERNALERROR> from ubspawn import Spawn
Regards,
Simon
^ permalink raw reply [flat|nested] 20+ messages in thread
* [U-Boot] [PATCH V3 1/7] test/py: Implement pytest infrastructure
2016-01-11 10:45 ` Michal Simek
@ 2016-01-11 17:00 ` Stephen Warren
0 siblings, 0 replies; 20+ messages in thread
From: Stephen Warren @ 2016-01-11 17:00 UTC (permalink / raw)
To: u-boot
On 01/11/2016 03:45 AM, Michal Simek wrote:
> Hi Stephen,
>
> On 8.1.2016 19:13, Stephen Warren wrote:
>> On 01/05/2016 03:58 PM, Stephen Warren wrote:
>>> This tool aims to test U-Boot by executing U-Boot shell commands using
>>> the
>>> console interface. A single top-level script exists to execute or attach
>>> to the U-Boot console, run the entire script of tests against it, and
>>> summarize the results. Advantages of this approach are:
>>>
>>> - Testing is performed in the same way a user or script would interact
>>> with U-Boot; there can be no disconnect.
>>> - There is no need to write or embed test-related code into U-Boot
>>> itself.
>>> It is asserted that writing test-related code in Python is simpler and
>>> more flexible that writing it all in C.
>>> - It is reasonably simple to interact with U-Boot in this way.
>>>
>>> A few simple tests are provided as examples. Soon, we should convert as
>>> many as possible of the other tests in test/* and test/cmd_ut.c too.
>>>
>>> The hook scripts, relay control utilities, and udev rules I use for my
>>> own HW setup are published at
>>> https://github.com/swarren/uboot-test-hooks.
>>>
>>> See README.md for more details!
>>
>> It looks like I need to send a v4 of this, since I renamed a Python
>> class but forgot to update all users of it. I didn't notice this, since
>> I had the old module lying around as a *.pyc file, so the old name
>> worked:-(
>
> yep.
>
>>
>> I also have a couple of minor fixes to roll in that make the scripts
>> work better under a continuous integration environment (which doesn't
>> have a controlling TTY set when the scripts run, which need a minor
>> tweak to the Spawn code).
>
> I have cherry picked 3 patches from your repo.
> 7813ccad9ed2 test/py: reset SIGHUP handler in child processes
> a2ccb34de3f5 test/py: fix ubspawn rename fallout
> 6dbcd7408f9e test/py: add a test for the sleep command
>
> sleep one is missing test for cmd_misc which enables that.
>
> There is still problem with handling special characters.
> For MB I have compilation error and %5e is shown instead of ^
>
> For example:
>
> +make O=/home/monstr/data/disk/u-boot/build-microblaze-generic -s
> microblaze-generic_defconfig
> +make O=/home/monstr/data/disk/u-boot/build-microblaze-generic -s -j8
> ../drivers/gpio/xilinx_gpio.c: In function 'xilinx_gpio_ofdata_to_platdata':
> ../drivers/gpio/xilinx_gpio.c:400:13: warning: assignment makes pointer
> from integer without a cast
> priv->regs = dev_get_addr(dev);
> %5e
Ah yes. When I first wrote the code, the log file wasn't HTML format,
but used ^ as a delimiter for some markup, so I escaped that character
so that log data wouldn't corrupt the file format. When I switched to
HTML, I forgot to remove that character from the escape list. I'll fold
the fix into v4.
^ permalink raw reply [flat|nested] 20+ messages in thread
* [U-Boot] [PATCH V3 1/7] test/py: Implement pytest infrastructure
2016-01-11 15:25 ` Simon Glass
@ 2016-01-11 17:04 ` Stephen Warren
0 siblings, 0 replies; 20+ messages in thread
From: Stephen Warren @ 2016-01-11 17:04 UTC (permalink / raw)
To: u-boot
On 01/11/2016 08:25 AM, Simon Glass wrote:
> Hi Stephen,
>
> On 8 January 2016 at 11:32, Michal Simek <monstr@monstr.eu> wrote:
>>
>> On 8.1.2016 19:13, Stephen Warren wrote:
>>> On 01/05/2016 03:58 PM, Stephen Warren wrote:
>>>> This tool aims to test U-Boot by executing U-Boot shell commands using
>>>> the
>>>> console interface. A single top-level script exists to execute or attach
>>>> to the U-Boot console, run the entire script of tests against it, and
>>>> summarize the results. Advantages of this approach are:
>>>>
>>>> - Testing is performed in the same way a user or script would interact
>>>> with U-Boot; there can be no disconnect.
>>>> - There is no need to write or embed test-related code into U-Boot
>>>> itself.
>>>> It is asserted that writing test-related code in Python is simpler and
>>>> more flexible that writing it all in C.
>>>> - It is reasonably simple to interact with U-Boot in this way.
>>>>
>>>> A few simple tests are provided as examples. Soon, we should convert as
>>>> many as possible of the other tests in test/* and test/cmd_ut.c too.
>>>>
>>>> The hook scripts, relay control utilities, and udev rules I use for my
>>>> own HW setup are published at
>>>> https://github.com/swarren/uboot-test-hooks.
>>>>
>>>> See README.md for more details!
>>>
>>> It looks like I need to send a v4 of this, since I renamed a Python
>>> class but forgot to update all users of it. I didn't notice this, since
>>> I had the old module lying around as a *.pyc file, so the old name
>>> worked:-(
>>>
>>> I also have a couple of minor fixes to roll in that make the scripts
>>> work better under a continuous integration environment (which doesn't
>>> have a controlling TTY set when the scripts run, which need a minor
>>> tweak to the Spawn code).
>
> I see this now. Do I need another dependency?
>
> /test/py/test.py --bd sandbox --build
...
> INTERNALERROR> from ubspawn import Spawn
No, I renamed the ubspawn class (to u_boot_spawn) but forgot to update
the users of the module to use the new name. My local testing didn't
notice this since I still had the .pyc file present with the old name,
but you evidently don't.
I think something like the following should fix it for you before I post v4:
sed -i 's/ubspawn/u_boot_spawn/' test/py/*.py
^ permalink raw reply [flat|nested] 20+ messages in thread
end of thread, other threads:[~2016-01-11 17:04 UTC | newest]
Thread overview: 20+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-01-05 22:58 [U-Boot] [PATCH V3 1/7] test/py: Implement pytest infrastructure Stephen Warren
2016-01-05 22:58 ` [U-Boot] [PATCH V3 2/7] test/py: test that sandbox exits when asked Stephen Warren
2016-01-11 15:23 ` Simon Glass
2016-01-05 22:58 ` [U-Boot] [PATCH V3 3/7] test/py: add test of setenv/printenv/echo Stephen Warren
2016-01-11 15:23 ` Simon Glass
2016-01-05 22:58 ` [U-Boot] [PATCH V3 4/7] test/py: test the md/mw commands Stephen Warren
2016-01-11 15:23 ` Simon Glass
2016-01-05 22:58 ` [U-Boot] [PATCH V3 5/7] test/py: add test of basic shell functionality Stephen Warren
2016-01-11 15:23 ` Simon Glass
2016-01-05 22:58 ` [U-Boot] [PATCH V3 6/7] test/py: test the shell if command Stephen Warren
2016-01-11 15:23 ` Simon Glass
2016-01-05 22:58 ` [U-Boot] [PATCH V3 7/7] test/py: test the ums command Stephen Warren
2016-01-11 15:23 ` Simon Glass
2016-01-08 18:13 ` [U-Boot] [PATCH V3 1/7] test/py: Implement pytest infrastructure Stephen Warren
2016-01-08 18:32 ` Michal Simek
2016-01-11 15:25 ` Simon Glass
2016-01-11 17:04 ` Stephen Warren
2016-01-11 10:45 ` Michal Simek
2016-01-11 17:00 ` Stephen Warren
2016-01-11 15:23 ` Simon Glass
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.