[PATCH 0/3] move test related documentation to HTML

[PATCH 0/3] move test related documentation to HTML

[Heinrich Schuchardt]

Restructure doc/develop/index.rst and move test related READMEs to the
generated HTML documentation.

Heinrich Schuchardt (3):
  doc: structure doc/develop/index.rst
  doc: move test/README to HTML documentation
  doc: move test/py/README.md to HTML documentation

 doc/develop/index.rst      |  24 ++-
 doc/develop/py_testing.rst | 426 +++++++++++++++++++++++++++++++++++++
 doc/develop/testing.rst    |  96 +++++++++
 test/README                |  96 ---------
 test/py/README.md          | 389 ---------------------------------
 5 files changed, 543 insertions(+), 488 deletions(-)
 create mode 100644 doc/develop/py_testing.rst
 create mode 100644 doc/develop/testing.rst
 delete mode 100644 test/README
 delete mode 100644 test/py/README.md

--
2.29.2

[PATCH 1/3] doc: structure doc/develop/index.rst

[Heinrich Schuchardt]

Provide sub-chapters for 'Develop U-Boot'

Signed-off-by: Heinrich Schuchardt <xypron.glpk@gmx.de>
---
 doc/develop/index.rst | 22 +++++++++++++++++++---
 1 file changed, 19 insertions(+), 3 deletions(-)

diff --git a/doc/develop/index.rst b/doc/develop/index.rst
index 0a7e204b34..b108df8e1b 100644
--- a/doc/develop/index.rst
+++ b/doc/develop/index.rst
@@ -3,13 +3,29 @@
 Develop U-Boot
 ==============

+Implementation
+--------------

 .. toctree::
-   :maxdepth: 2
+   :maxdepth: 1

-   coccinelle
    commands
-   crash_dumps
    global_data
    logging
+
+Debugging
+---------
+
+.. toctree::
+   :maxdepth: 1
+
+   crash_dumps
    trace
+
+Testing
+-------
+
+.. toctree::
+   :maxdepth: 1
+
+   coccinelle
--
2.29.2

[PATCH 2/3] doc: move test/README to HTML documentation

[Heinrich Schuchardt]

Move test/README to the 'Develop U-Boot' chapter of the HTML documentation.

Signed-off-by: Heinrich Schuchardt <xypron.glpk@gmx.de>
---
 doc/develop/index.rst   |  1 +
 doc/develop/testing.rst | 96 +++++++++++++++++++++++++++++++++++++++++
 test/README             | 96 -----------------------------------------
 3 files changed, 97 insertions(+), 96 deletions(-)
 create mode 100644 doc/develop/testing.rst
 delete mode 100644 test/README

diff --git a/doc/develop/index.rst b/doc/develop/index.rst
index b108df8e1b..ed2e73bf56 100644
--- a/doc/develop/index.rst
+++ b/doc/develop/index.rst
@@ -29,3 +29,4 @@ Testing
    :maxdepth: 1

    coccinelle
+   testing
diff --git a/doc/develop/testing.rst b/doc/develop/testing.rst
new file mode 100644
index 0000000000..4bc9ca3a6a
--- /dev/null
+++ b/doc/develop/testing.rst
@@ -0,0 +1,96 @@
+Testing in U-Boot
+=================
+
+U-Boot has a large amount of code. This file describes how this code is
+tested and what tests you should write when adding a new feature.
+
+
+Running tests
+-------------
+
+To run most tests on sandbox, type this:
+
+    make check
+
+in the U-Boot directory. Note that only the pytest suite is run using this
+command.
+
+Some tests take ages to run. To run just the quick ones, type this:
+
+    make qcheck
+
+
+Sandbox
+-------
+U-Boot can be built as a user-space application (e.g. for Linux). This
+allows test to be executed without needing target hardware. The 'sandbox'
+target provides this feature and it is widely used in tests.
+
+
+Pytest Suite
+------------
+
+Many tests are available using the pytest suite, in test/py. This can run
+either on sandbox or on real hardware. It relies on the U-Boot console to
+inject test commands and check the result. It is slower to run than C code,
+but provides the ability to unify lots of tests and summarise their results.
+
+You can run the tests on sandbox with:
+
+	./test/py/test.py --bd sandbox --build
+
+This will produce HTML output in build-sandbox/test-log.html
+
+See test/py/README.md for more information about the pytest suite.
+
+
+tbot
+----
+
+Tbot provides a way to execute tests on target hardware. It is intended for
+trying out both U-Boot and Linux (and potentially other software) on a
+number of boards automatically. It can be used to create a continuous test
+environment. See http://www.tbot.tools for more information.
+
+
+Ad-hoc tests
+------------
+
+There are several ad-hoc tests which run outside the pytest environment:
+
+   test/fs	- File system test (shell script)
+   test/image	- FIT and legacy image tests (shell script and Python)
+   test/stdint	- A test that stdint.h can be used in U-Boot (shell script)
+   trace	- Test for the tracing feature (shell script)
+
+TODO: Move these into pytest.
+
+
+When to write tests
+-------------------
+
+If you add code to U-Boot without a test you are taking a risk. Even if you
+perform thorough manual testing at the time of submission, it may break when
+future changes are made to U-Boot. It may even break when applied to mainline,
+if other changes interact with it. A good mindset is that untested code
+probably doesn't work and should be deleted.
+
+You can assume that the Pytest suite will be run before patches are accepted
+to mainline, so this provides protection against future breakage.
+
+On the other hand there is quite a bit of code that is not covered with tests,
+or is covered sparingly. So here are some suggestions:
+
+- If you are adding a new uclass, add a sandbox driver and a test that uses it
+- If you are modifying code covered by an existing test, add a new test case
+  to cover your changes
+- If the code you are modifying has not tests, consider writing one. Even a
+  very basic test is useful, and may be picked up and enhanced by others. It
+  is much easier to add onto a test - writing a new large test can seem
+  daunting to most contributors.
+
+
+Future work
+-----------
+
+Converting existing shell scripts into pytest tests.
diff --git a/test/README b/test/README
deleted file mode 100644
index 4bc9ca3a6a..0000000000
--- a/test/README
+++ /dev/null
@@ -1,96 +0,0 @@
-Testing in U-Boot
-=================
-
-U-Boot has a large amount of code. This file describes how this code is
-tested and what tests you should write when adding a new feature.
-
-
-Running tests
--------------
-
-To run most tests on sandbox, type this:
-
-    make check
-
-in the U-Boot directory. Note that only the pytest suite is run using this
-command.
-
-Some tests take ages to run. To run just the quick ones, type this:
-
-    make qcheck
-
-
-Sandbox
--------
-U-Boot can be built as a user-space application (e.g. for Linux). This
-allows test to be executed without needing target hardware. The 'sandbox'
-target provides this feature and it is widely used in tests.
-
-
-Pytest Suite
-------------
-
-Many tests are available using the pytest suite, in test/py. This can run
-either on sandbox or on real hardware. It relies on the U-Boot console to
-inject test commands and check the result. It is slower to run than C code,
-but provides the ability to unify lots of tests and summarise their results.
-
-You can run the tests on sandbox with:
-
-	./test/py/test.py --bd sandbox --build
-
-This will produce HTML output in build-sandbox/test-log.html
-
-See test/py/README.md for more information about the pytest suite.
-
-
-tbot
-----
-
-Tbot provides a way to execute tests on target hardware. It is intended for
-trying out both U-Boot and Linux (and potentially other software) on a
-number of boards automatically. It can be used to create a continuous test
-environment. See http://www.tbot.tools for more information.
-
-
-Ad-hoc tests
-------------
-
-There are several ad-hoc tests which run outside the pytest environment:
-
-   test/fs	- File system test (shell script)
-   test/image	- FIT and legacy image tests (shell script and Python)
-   test/stdint	- A test that stdint.h can be used in U-Boot (shell script)
-   trace	- Test for the tracing feature (shell script)
-
-TODO: Move these into pytest.
-
-
-When to write tests
--------------------
-
-If you add code to U-Boot without a test you are taking a risk. Even if you
-perform thorough manual testing at the time of submission, it may break when
-future changes are made to U-Boot. It may even break when applied to mainline,
-if other changes interact with it. A good mindset is that untested code
-probably doesn't work and should be deleted.
-
-You can assume that the Pytest suite will be run before patches are accepted
-to mainline, so this provides protection against future breakage.
-
-On the other hand there is quite a bit of code that is not covered with tests,
-or is covered sparingly. So here are some suggestions:
-
-- If you are adding a new uclass, add a sandbox driver and a test that uses it
-- If you are modifying code covered by an existing test, add a new test case
-  to cover your changes
-- If the code you are modifying has not tests, consider writing one. Even a
-  very basic test is useful, and may be picked up and enhanced by others. It
-  is much easier to add onto a test - writing a new large test can seem
-  daunting to most contributors.
-
-
-Future work
------------
-
-Converting existing shell scripts into pytest tests.
--
2.29.2

[PATCH 3/3] doc: move test/py/README.md to HTML documentation

[Heinrich Schuchardt]

Convert test/py/README.md to restructured text and add it to the generated
HTML documentation.

Signed-off-by: Heinrich Schuchardt <xypron.glpk@gmx.de>
---
 doc/develop/index.rst      |   1 +
 doc/develop/py_testing.rst | 426 +++++++++++++++++++++++++++++++++++++
 test/py/README.md          | 389 ---------------------------------
 3 files changed, 427 insertions(+), 389 deletions(-)
 create mode 100644 doc/develop/py_testing.rst
 delete mode 100644 test/py/README.md

diff --git a/doc/develop/index.rst b/doc/develop/index.rst
index ed2e73bf56..83eb1307ed 100644
--- a/doc/develop/index.rst
+++ b/doc/develop/index.rst
@@ -30,3 +30,4 @@ Testing

    coccinelle
    testing
+   py_testing
diff --git a/doc/develop/py_testing.rst b/doc/develop/py_testing.rst
new file mode 100644
index 0000000000..f71e837aa9
--- /dev/null
+++ b/doc/develop/py_testing.rst
@@ -0,0 +1,426 @@
+U-Boot pytest suite
+===================
+
+Introduction
+------------
+
+This tool aims to test U-Boot by executing U-Boot shell commands using the
+console interface. A single top-level script exists to execute or attach to the
+U-Boot console, run the entire script of tests against it, and summarize the
+results. Advantages of this approach are:
+
+- Testing is performed in the same way a user or script would interact with
+  U-Boot; there can be no disconnect.
+- There is no need to write or embed test-related code into U-Boot itself.
+  It is asserted that writing test-related code in Python is simpler and more
+  flexible than writing it all in C.
+- It is reasonably simple to interact with U-Boot in this way.
+
+Requirements
+------------
+
+The test suite is implemented using pytest. Interaction with the U-Boot console
+involves executing some binary and interacting with its stdin/stdout. You will
+need to implement various "hook" scripts that are called by the test suite at
+the appropriate time.
+
+In order to run the test suite at a minimum we require that both Python 3 and
+pip for Python 3 are installed. All of the required python modules are
+described in the requirements.txt file in the /test/py/ directory and can be
+installed via the command
+
+.. code-block:: bash
+
+   pip install -r requirements.txt
+
+In order to execute certain tests on their supported platforms other tools
+will be required. The following is an incomplete list:
+
+* gdisk
+* dfu-util
+* dtc
+* openssl
+* sudo OR guestmount
+* e2fsprogs
+* util-linux
+* coreutils
+* dosfstools
+* efitools
+* mount
+* mtools
+* sbsigntool
+* udisks2
+
+Please use the appropriate commands for your distribution to match these tools
+up with the package that provides them.
+
+The test script supports either:
+
+- Executing a sandbox port of U-Boot on the local machine as a sub-process,
+  and interacting with it over stdin/stdout.
+- Executing an external "hook" scripts to flash a U-Boot binary onto a
+  physical board, attach to the board's console stream, and reset the board.
+  Further details are described later.
+
+Using `virtualenv` to provide requirements
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The recommended way to run the test suite, in order to ensure reproducibility
+is to use `virtualenv` to set up the necessary environment.  This can be done
+via the following commands:
+
+
+.. code-block:: console
+
+    $ cd /path/to/u-boot
+    $ sudo apt-get install python3 python3-virtualenv
+    $ virtualenv -p /usr/bin/python3 venv
+    $ . ./venv/bin/activate
+    $ pip install -r test/py/requirements.txt
+
+Testing sandbox
+---------------
+
+To run the test suite on the sandbox port (U-Boot built as a native user-space
+application), simply execute:
+
+.. code-block:: bash
+
+    ./test/py/test.py --bd sandbox --build
+
+The `--bd` option tells the test suite which board type is being tested. This
+lets the test suite know which features the board has, and hence exactly what
+can be tested.
+
+The `--build` option tells U-Boot to compile U-Boot. Alternatively, you may
+omit this option and build U-Boot yourself, in whatever way you choose, before
+running the test script.
+
+The test script will attach to U-Boot, execute all valid tests for the board,
+then print a summary of the test process. A complete log of the test session
+will be written to `${build_dir}/test-log.html`. This is best viewed in a web
+browser, but may be read directly as plain text, perhaps with the aid of the
+`html2text` utility.
+
+Testing under a debugger
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+If you need to run sandbox under a debugger, you may pass the command-line
+option `--gdbserver COMM`. This causes two things to happens:
+
+- Instead of running U-Boot directly, it will be run under gdbserver, with
+  debug communication via the channel `COMM`. You can attach a debugger to the
+  sandbox process in order to debug it. See `man gdbserver` and the example
+  below for details of valid values for `COMM`.
+- All timeouts in tests are disabled, allowing U-Boot an arbitrary amount of
+  time to execute commands. This is useful if U-Boot is stopped at a breakpoint
+  during debugging.
+
+A usage example is:
+
+Window 1:
+
+.. code-block:: bash
+
+    ./test/py/test.py --bd sandbox --gdbserver localhost:1234
+
+Window 2:
+
+.. code-block:: bash
+
+    gdb ./build-sandbox/u-boot -ex 'target remote localhost:1234'
+
+Alternatively, you could leave off the `-ex` option and type the command
+manually into gdb once it starts.
+
+You can use any debugger you wish, as long as it speaks the gdb remote
+protocol, or any graphical wrapper around gdb.
+
+Some tests deliberately cause the sandbox process to exit, e.g. to test the
+reset command, or sandbox's CTRL-C handling. When this happens, you will need
+to attach the debugger to the new sandbox instance. If these tests are not
+relevant to your debugging session, you can skip them using pytest's -k
+command-line option; see the next section.
+
+Command-line options
+--------------------
+
+--board-type, --bd, -B
+  set the type of the board to be tested. For example, `sandbox` or `seaboard`.
+
+--board-identity`, --id
+  sets the identity of the board to be tested. This allows differentiation
+  between multiple instances of the same type of physical board that are
+  attached to the same host machine. This parameter is not interpreted by th
+  test script in any way, but rather is simply passed to the hook scripts
+  described below, and may be used in any site-specific way deemed necessary.
+
+--build
+  indicates that the test script should compile U-Boot itself before running
+  the tests. If using this option, make sure that any environment variables
+  required by the build process are already set, such as `$CROSS_COMPILE`.
+
+--buildman
+  indicates that `--build` should use buildman to build U-Boot. There is no need
+  to set $CROSS_COMPILE` in this case since buildman handles it.
+
+--build-dir
+  sets the directory containing the compiled U-Boot binaries. If omitted, this
+  is `${source_dir}/build-${board_type}`.
+
+--result-dir
+  sets the directory to write results, such as log files, into.
+  If omitted, the build directory is used.
+
+--persistent-data-dir
+  sets the directory used to store persistent test data. This is test data that
+  may be re-used across test runs, such as file-system images.
+
+`pytest` also implements a number of its own command-line options. Commonly used
+options are mentioned below. Please see `pytest` documentation for complete
+details. Execute `py.test --version` for a brief summary. Note that U-Boot's
+test.py script passes all command-line arguments directly to `pytest` for
+processing.
+
+-k
+  selects which tests to run. The default is to run all known tests. This
+  option takes a single argument which is used to filter test names. Simple
+  logical operators are supported. For example:
+
+  - `'-k ums'` runs only tests with "ums" in their name.
+  - `'-k ut_dm'` runs only tests with "ut_dm" in their name. Note that in this
+    case, "ut_dm" is a parameter to a test rather than the test name. The full
+    test name is e.g. "test_ut[ut_dm_leak]".
+  - `'-k not reset'` runs everything except tests with "reset" in their name.
+  - `'-k ut or hush'` runs only tests with "ut" or "hush" in their name.
+  - `'-k not (ut or hush)'` runs everything except tests with "ut" or "hush" in
+    their name.
+
+-s
+  prevents pytest from hiding a test's stdout. This allows you to see
+  U-Boot's console log in real time on pytest's stdout.
+
+Testing real hardware
+---------------------
+
+The tools and techniques used to interact with real hardware will vary
+radically between different host and target systems, and the whims of the user.
+For this reason, the test suite does not attempt to directly interact with real
+hardware in any way. Rather, it executes a standardized set of "hook" scripts
+via `$PATH`. These scripts implement certain actions on behalf of the test
+suite. This keeps the test suite simple and isolated from system variances
+unrelated to U-Boot features.
+
+Hook scripts
+~~~~~~~~~~~~
+
+Environment variables
+'''''''''''''''''''''
+
+The following environment variables are set when running hook scripts:
+
+- `UBOOT_BOARD_TYPE` the board type being tested.
+- `UBOOT_BOARD_IDENTITY` the board identity being tested, or `na` if none was
+  specified.
+- `UBOOT_SOURCE_DIR` the U-Boot source directory.
+- `UBOOT_TEST_PY_DIR` the full path to `test/py/` in the source directory.
+- `UBOOT_BUILD_DIR` the U-Boot build directory.
+- `UBOOT_RESULT_DIR` the test result directory.
+- `UBOOT_PERSISTENT_DATA_DIR` the test persistent data directory.
+
+u-boot-test-console
+'''''''''''''''''''
+
+This script provides access to the U-Boot console. The script's stdin/stdout
+should be connected to the board's console. This process should continue to run
+indefinitely, until killed. The test suite will run this script in parallel
+with all other hooks.
+
+This script may be implemented e.g. by executing `cu`, `kermit`, `conmux`, etc.
+via exec().
+
+If you are able to run U-Boot under a hardware simulator such as QEMU, then
+you would likely spawn that simulator from this script. However, note that
+`u-boot-test-reset` may be called multiple times per test script run, and must
+cause U-Boot to start execution from scratch each time. Hopefully your
+simulator includes a virtual reset button! If not, you can launch the
+simulator from `u-boot-test-reset` instead, while arranging for this console
+process to always communicate with the current simulator instance.
+
+u-boot-test-flash
+'''''''''''''''''
+
+Prior to running the test suite against a board, some arrangement must be made
+so that the board executes the particular U-Boot binary to be tested. Often
+this involves writing the U-Boot binary to the board's flash ROM. The test
+suite calls this hook script for that purpose.
+
+This script should perform the entire flashing process synchronously; the
+script should only exit once flashing is complete, and a board reset will
+cause the newly flashed U-Boot binary to be executed.
+
+It is conceivable that this script will do nothing. This might be useful in
+the following cases:
+
+- Some other process has already written the desired U-Boot binary into the
+  board's flash prior to running the test suite.
+- The board allows U-Boot to be downloaded directly into RAM, and executed
+  from there. Use of this feature will reduce wear on the board's flash, so
+  may be preferable if available, and if cold boot testing of U-Boot is not
+  required. If this feature is used, the `u-boot-test-reset` script should
+  perform this download, since the board could conceivably be reset multiple
+  times in a single test run.
+
+It is up to the user to determine if those situations exist, and to code this
+hook script appropriately.
+
+This script will typically be implemented by calling out to some SoC- or
+board-specific vendor flashing utility.
+
+u-boot-test-reset
+'''''''''''''''''
+
+Whenever the test suite needs to reset the target board, this script is
+executed. This is guaranteed to happen at least once, prior to executing the
+first test function. If any test fails, the test infra-structure will execute
+this script again to restore U-Boot to an operational state before running the
+next test function.
+
+This script will likely be implemented by communicating with some form of
+relay or electronic switch attached to the board's reset signal.
+
+The semantics of this script require that when it is executed, U-Boot will
+start running from scratch. If the U-Boot binary to be tested has been written
+to flash, pulsing the board's reset signal is likely all this script needs to
+do. However, in some scenarios, this script may perform other actions. For
+example, it may call out to some SoC- or board-specific vendor utility in order
+to download the U-Boot binary directly into RAM and execute it. This would
+avoid the need for `u-boot-test-flash` to actually write U-Boot to flash, thus
+saving wear on the flash chip(s).
+
+Examples
+''''''''
+
+https://github.com/swarren/uboot-test-hooks contains some working example hook
+scripts, and may be useful as a reference when implementing hook scripts for
+your platform. These scripts are not considered part of U-Boot itself.
+
+Board-type-specific configuration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Each board has a different configuration and behaviour. Many of these
+differences can be automatically detected by parsing the `.config` file in the
+build directory. However, some differences can't yet be handled automatically.
+
+For each board, an optional Python module `u_boot_board_${board_type}` may exist
+to provide board-specific information to the test script. Any global value
+defined in these modules is available for use by any test function. The data
+contained in these scripts must be purely derived from U-Boot source code.
+Hence, these configuration files are part of the U-Boot source tree too.
+
+Execution environment configuration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Each user's hardware setup may enable testing different subsets of the features
+implemented by a particular board's configuration of U-Boot. For example, a
+U-Boot configuration may support USB device mode and USB Mass Storage, but this
+can only be tested if a USB cable is connected between the board and the host
+machine running the test script.
+
+For each board, optional Python modules `u_boot_boardenv_${board_type}` and
+`u_boot_boardenv_${board_type}_${board_identity}` may exist to provide
+board-specific and board-identity-specific information to the test script. Any
+global value defined in these modules is available for use by any test
+function. The data contained in these is specific to a particular user's
+hardware configuration. Hence, these configuration files are not part of the
+U-Boot source tree, and should be installed outside of the source tree. Users
+should set `$PYTHONPATH` prior to running the test script to allow these
+modules to be loaded.
+
+Board module parameter usage
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The test scripts rely on the following variables being defined by the board
+module:
+
+- none at present
+
+U-Boot `.config` feature usage
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The test scripts rely on various U-Boot `.config` features, either directly in
+order to test those features, or indirectly in order to query information from
+the running U-Boot instance in order to test other features.
+
+One example is that testing of the `md` command requires knowledge of a RAM
+address to use for the test. This data is parsed from the output of the
+`bdinfo` command, and hence relies on CONFIG_CMD_BDI being enabled.
+
+For a complete list of dependencies, please search the test scripts for
+instances of:
+
+- `buildconfig.get(...`
+- `@pytest.mark.buildconfigspec(...`
+- `@pytest.mark.notbuildconfigspec(...`
+
+Complete invocation example
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Assuming that you have installed the hook scripts into $HOME/ubtest/bin, and
+any required environment configuration Python modules into $HOME/ubtest/py,
+then you would likely invoke the test script as follows:
+
+If U-Boot has already been built:
+
+.. code-block:: bash
+
+    PATH=$HOME/ubtest/bin:$PATH \
+    PYTHONPATH=${HOME}/ubtest/py/${HOSTNAME}:${PYTHONPATH} \
+    ./test/py/test.py --bd seaboard
+
+If you want the test script to compile U-Boot for you too, then you likely
+need to set `$CROSS_COMPILE` to allow this, and invoke the test script as
+follows:
+
+.. code-block:: bash
+
+    CROSS_COMPILE=arm-none-eabi- \
+    PATH=$HOME/ubtest/bin:$PATH \
+    PYTHONPATH=${HOME}/ubtest/py/${HOSTNAME}:${PYTHONPATH} \
+    ./test/py/test.py --bd seaboard --build
+
+or, using buildman to handle it:
+
+.. code-block:: bash
+
+    PATH=$HOME/ubtest/bin:$PATH \
+    PYTHONPATH=${HOME}/ubtest/py/${HOSTNAME}:${PYTHONPATH} \
+    ./test/py/test.py --bd seaboard --build --buildman
+
+Writing tests
+-------------
+
+Please refer to the pytest documentation for details of writing pytest tests.
+Details specific to the U-Boot test suite are described below.
+
+A test fixture named `u_boot_console` should be used by each test function. This
+provides the means to interact with the U-Boot console, and retrieve board and
+environment configuration information.
+
+The function `u_boot_console.run_command()` executes a shell command on the
+U-Boot console, and returns all output from that command. This allows
+validation or interpretation of the command output. This function validates
+that certain strings are not seen on the U-Boot console. These include shell
+error messages and the U-Boot sign-on message (in order to detect unexpected
+board resets). See the source of `u_boot_console_base.py` for a complete list of
+"bad" strings. Some test scenarios are expected to trigger these strings. Use
+`u_boot_console.disable_check()` to temporarily disable checking for specific
+strings. See `test_unknown_cmd.py` for an example.
+
+Board- and board-environment configuration values may be accessed as sub-fields
+of the `u_boot_console.config` object, for example
+`u_boot_console.config.ram_base`.
+
+Build configuration values (from `.config`) may be accessed via the dictionary
+`u_boot_console.config.buildconfig`, with keys equal to the Kconfig variable
+names.
diff --git a/test/py/README.md b/test/py/README.md
deleted file mode 100644
index fddc104b26..0000000000
--- a/test/py/README.md
+++ /dev/null
@@ -1,389 +0,0 @@
-# U-Boot pytest suite
-
-## Introduction
-
-This tool aims to test U-Boot by executing U-Boot shell commands using the
-console interface. A single top-level script exists to execute or attach to the
-U-Boot console, run the entire script of tests against it, and summarize the
-results. Advantages of this approach are:
-
-- Testing is performed in the same way a user or script would interact with
-  U-Boot; there can be no disconnect.
-- There is no need to write or embed test-related code into U-Boot itself.
-  It is asserted that writing test-related code in Python is simpler and more
-  flexible than writing it all in C.
-- It is reasonably simple to interact with U-Boot in this way.
-
-## Requirements
-
-The test suite is implemented using pytest. Interaction with the U-Boot console
-involves executing some binary and interacting with its stdin/stdout. You will
-need to implement various "hook" scripts that are called by the test suite at
-the appropriate time.
-
-In order to run the testsuite at a minimum we require that both python3 and
-pip for python3 be installed.  All of the required python modules are
-described in the requirements.txt file in this directory and can be installed
-with the command ```pip install -r requirements.txt```
-
-In order to execute certain tests on their supported platforms other tools
-will be required.  The following is an incomplete list:
-
-| Package        |
-| -------------- |
-| gdisk          |
-| dfu-util       |
-| dtc            |
-| openssl        |
-| sudo OR guestmount |
-| e2fsprogs      |
-| util-linux     |
-| coreutils      |
-| dosfstools     |
-| efitools       |
-| mount          |
-| mtools         |
-| sbsigntool     |
-| udisks2        |
-
-
-Please use the apporirate commands for your distribution to match these tools
-up with the package that provides them.
-
-The test script supports either:
-
-- Executing a sandbox port of U-Boot on the local machine as a sub-process,
-  and interacting with it over stdin/stdout.
-- Executing an external "hook" scripts to flash a U-Boot binary onto a
-  physical board, attach to the board's console stream, and reset the board.
-  Further details are described later.
-
-### Using `virtualenv` to provide requirements
-
-The recommended way to run the test suite, in order to ensure reproducibility
-is to use `virtualenv` to set up the necessary environment.  This can be done
-via the following commands:
-
-```bash
-$ cd /path/to/u-boot
-$ sudo apt-get install python3 python3-virtualenv
-$ virtualenv -p /usr/bin/python3 venv
-$ . ./venv/bin/activate
-$ pip install -r test/py/requirements.txt
-```
-
-## Testing sandbox
-
-To run the testsuite on the sandbox port (U-Boot built as a native user-space
-application), simply execute:
-
-```
-./test/py/test.py --bd sandbox --build
-```
-
-The `--bd` option tells the test suite which board type is being tested. This
-lets the test suite know which features the board has, and hence exactly what
-can be tested.
-
-The `--build` option tells U-Boot to compile U-Boot. Alternatively, you may
-omit this option and build U-Boot yourself, in whatever way you choose, before
-running the test script.
-
-The test script will attach to U-Boot, execute all valid tests for the board,
-then print a summary of the test process. A complete log of the test session
-will be written to `${build_dir}/test-log.html`. This is best viewed in a web
-browser, but may be read directly as plain text, perhaps with the aid of the
-`html2text` utility.
-
-### Testing under a debugger
-
-If you need to run sandbox under a debugger, you may pass the command-line
-option `--gdbserver COMM`. This causes two things to happens:
-
-- Instead of running U-Boot directly, it will be run under gdbserver, with
-  debug communication via the channel `COMM`. You can attach a debugger to the
-  sandbox process in order to debug it. See `man gdbserver` and the example
-  below for details of valid values for `COMM`.
-- All timeouts in tests are disabled, allowing U-Boot an arbitrary amount of
-  time to execute commands. This is useful if U-Boot is stopped at a breakpoint
-  during debugging.
-
-A usage example is:
-
-Window 1:
-```shell
-./test/py/test.py --bd sandbox --gdbserver localhost:1234
-```
-
-Window 2:
-```shell
-gdb ./build-sandbox/u-boot -ex 'target remote localhost:1234'
-```
-
-Alternatively, you could leave off the `-ex` option and type the command
-manually into gdb once it starts.
-
-You can use any debugger you wish, so long as it speaks the gdb remote
-protocol, or any graphical wrapper around gdb.
-
-Some tests deliberately cause the sandbox process to exit, e.g. to test the
-reset command, or sandbox's CTRL-C handling. When this happens, you will need
-to attach the debugger to the new sandbox instance. If these tests are not
-relevant to your debugging session, you can skip them using pytest's -k
-command-line option; see the next section.
-
-## Command-line options
-
-- `--board-type`, `--bd`, `-B` set the type of the board to be tested. For
-  example, `sandbox` or `seaboard`.
-- `--board-identity`, `--id` set the identity of the board to be tested.
-  This allows differentiation between multiple instances of the same type of
-  physical board that are attached to the same host machine. This parameter is
-  not interpreted by the test script in any way, but rather is simply passed
-  to the hook scripts described below, and may be used in any site-specific
-  way deemed necessary.
-- `--build` indicates that the test script should compile U-Boot itself
-  before running the tests. If using this option, make sure that any
-  environment variables required by the build process are already set, such as
-  `$CROSS_COMPILE`.
-- `--buildman` indicates that `--build` should use buildman to build U-Boot.
-  There is no need to set $CROSS_COMPILE` in this case since buildman handles
-  it.
-- `--build-dir` sets the directory containing the compiled U-Boot binaries.
-  If omitted, this is `${source_dir}/build-${board_type}`.
-- `--result-dir` sets the directory to write results, such as log files,
-  into. If omitted, the build directory is used.
-- `--persistent-data-dir` sets the directory used to store persistent test
-  data. This is test data that may be re-used across test runs, such as file-
-  system images.
-
-`pytest` also implements a number of its own command-line options. Commonly used
-options are mentioned below. Please see `pytest` documentation for complete
-details. Execute `py.test --version` for a brief summary. Note that U-Boot's
-test.py script passes all command-line arguments directly to `pytest` for
-processing.
-
-- `-k` selects which tests to run. The default is to run all known tests. This
-  option takes a single argument which is used to filter test names. Simple
-  logical operators are supported. For example:
-  - `'ums'` runs only tests with "ums" in their name.
-  - `'ut_dm'` runs only tests with "ut_dm" in their name. Note that in this
-    case, "ut_dm" is a parameter to a test rather than the test name. The full
-    test name is e.g. "test_ut[ut_dm_leak]".
-  - `'not reset'` runs everything except tests with "reset" in their name.
-  - `'ut or hush'` runs only tests with "ut" or "hush" in their name.
-  - `'not (ut or hush)'` runs everything except tests with "ut" or "hush" in
-    their name.
-- `-s` prevents pytest from hiding a test's stdout. This allows you to see
-  U-Boot's console log in real time on pytest's stdout.
-
-## Testing real hardware
-
-The tools and techniques used to interact with real hardware will vary
-radically between different host and target systems, and the whims of the user.
-For this reason, the test suite does not attempt to directly interact with real
-hardware in any way. Rather, it executes a standardized set of "hook" scripts
-via `$PATH`. These scripts implement certain actions on behalf of the test
-suite. This keeps the test suite simple and isolated from system variances
-unrelated to U-Boot features.
-
-### Hook scripts
-
-#### Environment variables
-
-The following environment variables are set when running hook scripts:
-
-- `UBOOT_BOARD_TYPE` the board type being tested.
-- `UBOOT_BOARD_IDENTITY` the board identity being tested, or `na` if none was
-  specified.
-- `UBOOT_SOURCE_DIR` the U-Boot source directory.
-- `UBOOT_TEST_PY_DIR` the full path to `test/py/` in the source directory.
-- `UBOOT_BUILD_DIR` the U-Boot build directory.
-- `UBOOT_RESULT_DIR` the test result directory.
-- `UBOOT_PERSISTENT_DATA_DIR` the test persistent data directory.
-
-#### `u-boot-test-console`
-
-This script provides access to the U-Boot console. The script's stdin/stdout
-should be connected to the board's console. This process should continue to run
-indefinitely, until killed. The test suite will run this script in parallel
-with all other hooks.
-
-This script may be implemented e.g. by exec()ing `cu`, `kermit`, `conmux`, etc.
-
-If you are able to run U-Boot under a hardware simulator such as qemu, then
-you would likely spawn that simulator from this script. However, note that
-`u-boot-test-reset` may be called multiple times per test script run, and must
-cause U-Boot to start execution from scratch each time. Hopefully your
-simulator includes a virtual reset button! If not, you can launch the
-simulator from `u-boot-test-reset` instead, while arranging for this console
-process to always communicate with the current simulator instance.
-
-#### `u-boot-test-flash`
-
-Prior to running the test suite against a board, some arrangement must be made
-so that the board executes the particular U-Boot binary to be tested. Often,
-this involves writing the U-Boot binary to the board's flash ROM. The test
-suite calls this hook script for that purpose.
-
-This script should perform the entire flashing process synchronously; the
-script should only exit once flashing is complete, and a board reset will
-cause the newly flashed U-Boot binary to be executed.
-
-It is conceivable that this script will do nothing. This might be useful in
-the following cases:
-
-- Some other process has already written the desired U-Boot binary into the
-  board's flash prior to running the test suite.
-- The board allows U-Boot to be downloaded directly into RAM, and executed
-  from there. Use of this feature will reduce wear on the board's flash, so
-  may be preferable if available, and if cold boot testing of U-Boot is not
-  required. If this feature is used, the `u-boot-test-reset` script should
-  perform this download, since the board could conceivably be reset multiple
-  times in a single test run.
-
-It is up to the user to determine if those situations exist, and to code this
-hook script appropriately.
-
-This script will typically be implemented by calling out to some SoC- or
-board-specific vendor flashing utility.
-
-#### `u-boot-test-reset`
-
-Whenever the test suite needs to reset the target board, this script is
-executed. This is guaranteed to happen at least once, prior to executing the
-first test function. If any test fails, the test infra-structure will execute
-this script again to restore U-Boot to an operational state before running the
-next test function.
-
-This script will likely be implemented by communicating with some form of
-relay or electronic switch attached to the board's reset signal.
-
-The semantics of this script require that when it is executed, U-Boot will
-start running from scratch. If the U-Boot binary to be tested has been written
-to flash, pulsing the board's reset signal is likely all this script need do.
-However, in some scenarios, this script may perform other actions. For
-example, it may call out to some SoC- or board-specific vendor utility in order
-to download the U-Boot binary directly into RAM and execute it. This would
-avoid the need for `u-boot-test-flash` to actually write U-Boot to flash, thus
-saving wear on the flash chip(s).
-
-#### Examples
-
-https://github.com/swarren/uboot-test-hooks contains some working example hook
-scripts, and may be useful as a reference when implementing hook scripts for
-your platform. These scripts are not considered part of U-Boot itself.
-
-### Board-type-specific configuration
-
-Each board has a different configuration and behaviour. Many of these
-differences can be automatically detected by parsing the `.config` file in the
-build directory. However, some differences can't yet be handled automatically.
-
-For each board, an optional Python module `u_boot_board_${board_type}` may exist
-to provide board-specific information to the test script. Any global value
-defined in these modules is available for use by any test function. The data
-contained in these scripts must be purely derived from U-Boot source code.
-Hence, these configuration files are part of the U-Boot source tree too.
-
-### Execution environment configuration
-
-Each user's hardware setup may enable testing different subsets of the features
-implemented by a particular board's configuration of U-Boot. For example, a
-U-Boot configuration may support USB device mode and USB Mass Storage, but this
-can only be tested if a USB cable is connected between the board and the host
-machine running the test script.
-
-For each board, optional Python modules `u_boot_boardenv_${board_type}` and
-`u_boot_boardenv_${board_type}_${board_identity}` may exist to provide
-board-specific and board-identity-specific information to the test script. Any
-global value defined in these modules is available for use by any test
-function. The data contained in these is specific to a particular user's
-hardware configuration. Hence, these configuration files are not part of the
-U-Boot source tree, and should be installed outside of the source tree. Users
-should set `$PYTHONPATH` prior to running the test script to allow these
-modules to be loaded.
-
-### Board module parameter usage
-
-The test scripts rely on the following variables being defined by the board
-module:
-
-- None at present.
-
-### U-Boot `.config` feature usage
-
-The test scripts rely on various U-Boot `.config` features, either directly in
-order to test those features, or indirectly in order to query information from
-the running U-Boot instance in order to test other features.
-
-One example is that testing of the `md` command requires knowledge of a RAM
-address to use for the test. This data is parsed from the output of the
-`bdinfo` command, and hence relies on CONFIG_CMD_BDI being enabled.
-
-For a complete list of dependencies, please search the test scripts for
-instances of:
-
-- `buildconfig.get(...`
-- `@pytest.mark.buildconfigspec(...`
-- `@pytest.mark.notbuildconfigspec(...`
-
-### Complete invocation example
-
-Assuming that you have installed the hook scripts into $HOME/ubtest/bin, and
-any required environment configuration Python modules into $HOME/ubtest/py,
-then you would likely invoke the test script as follows:
-
-If U-Boot has already been built:
-
-```bash
-PATH=$HOME/ubtest/bin:$PATH \
-    PYTHONPATH=${HOME}/ubtest/py/${HOSTNAME}:${PYTHONPATH} \
-    ./test/py/test.py --bd seaboard
-```
-
-If you want the test script to compile U-Boot for you too, then you likely
-need to set `$CROSS_COMPILE` to allow this, and invoke the test script as
-follows:
-
-```bash
-CROSS_COMPILE=arm-none-eabi- \
-    PATH=$HOME/ubtest/bin:$PATH \
-    PYTHONPATH=${HOME}/ubtest/py/${HOSTNAME}:${PYTHONPATH} \
-    ./test/py/test.py --bd seaboard --build
-```
-
-or, using buildman to handle it:
-
-```bash
-    PATH=$HOME/ubtest/bin:$PATH \
-    PYTHONPATH=${HOME}/ubtest/py/${HOSTNAME}:${PYTHONPATH} \
-    ./test/py/test.py --bd seaboard --build --buildman
-```
-
-## Writing tests
-
-Please refer to the pytest documentation for details of writing pytest tests.
-Details specific to the U-Boot test suite are described below.
-
-A test fixture named `u_boot_console` should be used by each test function. This
-provides the means to interact with the U-Boot console, and retrieve board and
-environment configuration information.
-
-The function `u_boot_console.run_command()` executes a shell command on the
-U-Boot console, and returns all output from that command. This allows
-validation or interpretation of the command output. This function validates
-that certain strings are not seen on the U-Boot console. These include shell
-error messages and the U-Boot sign-on message (in order to detect unexpected
-board resets). See the source of `u_boot_console_base.py` for a complete list of
-"bad" strings. Some test scenarios are expected to trigger these strings. Use
-`u_boot_console.disable_check()` to temporarily disable checking for specific
-strings. See `test_unknown_cmd.py` for an example.
-
-Board- and board-environment configuration values may be accessed as sub-fields
-of the `u_boot_console.config` object, for example
-`u_boot_console.config.ram_base`.
-
-Build configuration values (from `.config`) may be accessed via the dictionary
-`u_boot_console.config.buildconfig`, with keys equal to the Kconfig variable
-names.
--
2.29.2

[PATCH 1/3] doc: structure doc/develop/index.rst

[Simon Glass]

On Mon, 18 Jan 2021 at 12:29, Heinrich Schuchardt <xypron.glpk@gmx.de> wrote:
>
> Provide sub-chapters for 'Develop U-Boot'
>
> Signed-off-by: Heinrich Schuchardt <xypron.glpk@gmx.de>
> ---
>  doc/develop/index.rst | 22 +++++++++++++++++++---
>  1 file changed, 19 insertions(+), 3 deletions(-)
>

Reviewed-by: Simon Glass <sjg@chromium.org>

[PATCH 2/3] doc: move test/README to HTML documentation

[Simon Glass]

On Mon, 18 Jan 2021 at 12:24, Heinrich Schuchardt <xypron.glpk@gmx.de> wrote:
>
> Move test/README to the 'Develop U-Boot' chapter of the HTML documentation.
>
> Signed-off-by: Heinrich Schuchardt <xypron.glpk@gmx.de>
> ---
>  doc/develop/index.rst   |  1 +
>  doc/develop/testing.rst | 96 +++++++++++++++++++++++++++++++++++++++++
>  test/README             | 96 -----------------------------------------
>  3 files changed, 97 insertions(+), 96 deletions(-)
>  create mode 100644 doc/develop/testing.rst
>  delete mode 100644 test/README

Reviewed-by: Simon Glass <sjg@chromium.org>

[PATCH 3/3] doc: move test/py/README.md to HTML documentation

[Simon Glass]

On Mon, 18 Jan 2021 at 12:29, Heinrich Schuchardt <xypron.glpk@gmx.de> wrote:
>
> Convert test/py/README.md to restructured text and add it to the generated
> HTML documentation.
>
> Signed-off-by: Heinrich Schuchardt <xypron.glpk@gmx.de>
> ---
>  doc/develop/index.rst      |   1 +
>  doc/develop/py_testing.rst | 426 +++++++++++++++++++++++++++++++++++++
>  test/py/README.md          | 389 ---------------------------------
>  3 files changed, 427 insertions(+), 389 deletions(-)
>  create mode 100644 doc/develop/py_testing.rst
>  delete mode 100644 test/py/README.md
>

Reviewed-by: Simon Glass <sjg@chromium.org>

[PATCH 1/3] doc: structure doc/develop/index.rst

[Tom Rini]

On Mon, Jan 18, 2021 at 08:24:01PM +0100, Heinrich Schuchardt wrote:

> Provide sub-chapters for 'Develop U-Boot'
> 
> Signed-off-by: Heinrich Schuchardt <xypron.glpk@gmx.de>
> Reviewed-by: Simon Glass <sjg@chromium.org>

Applied to u-boot/master, thanks!

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 659 bytes
Desc: not available
URL: <https://lists.denx.de/pipermail/u-boot/attachments/20210123/377754c5/attachment.sig>

[PATCH 2/3] doc: move test/README to HTML documentation

[Tom Rini]

On Mon, Jan 18, 2021 at 08:24:02PM +0100, Heinrich Schuchardt wrote:

> Move test/README to the 'Develop U-Boot' chapter of the HTML documentation.
> 
> Signed-off-by: Heinrich Schuchardt <xypron.glpk@gmx.de>
> Reviewed-by: Simon Glass <sjg@chromium.org>

Applied to u-boot/master, thanks!

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 659 bytes
Desc: not available
URL: <https://lists.denx.de/pipermail/u-boot/attachments/20210123/b5e18ca6/attachment.sig>

[PATCH 3/3] doc: move test/py/README.md to HTML documentation

[Tom Rini]

On Mon, Jan 18, 2021 at 08:24:03PM +0100, Heinrich Schuchardt wrote:

> Convert test/py/README.md to restructured text and add it to the generated
> HTML documentation.
> 
> Signed-off-by: Heinrich Schuchardt <xypron.glpk@gmx.de>
> Reviewed-by: Simon Glass <sjg@chromium.org>

Applied to u-boot/master, thanks!

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 659 bytes
Desc: not available
URL: <https://lists.denx.de/pipermail/u-boot/attachments/20210123/0a3c915d/attachment.sig>