All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH v2 0/4] test: Try to deal with some co-dependent tests
@ 2021-07-23  2:07 Simon Glass
  2021-07-23  2:07 ` [PATCH v2 1/4] test: Allow vboot tests to run in parallel Simon Glass
                   ` (3 more replies)
  0 siblings, 4 replies; 5+ messages in thread
From: Simon Glass @ 2021-07-23  2:07 UTC (permalink / raw)
  To: U-Boot Mailing List
  Cc: Tom Rini, Heinrich Schuchardt, Stephen Warren, Stephen Warren,
	Simon Glass, Thirupathaiah Annapureddy

Tests are supposed to be independent. With driver model tests, the
environment is reset before each test, which ensures that.

With Python tests there is no reset of the board between tests, since we
want to run all the tests as quickly as possible and without needing the
external scripts running constantly.

In principle the Python tests can be independent if they each put the
world back the way they found it, but it turns out that some are not.
This means that some tests cannot be run unless another test is run
first. It also means that tests cannot be run in parallel, e.g. on
sandbox.

This series fixes some of them. Those that remain:

   test_gpt_swap_partitions - not sure?
   test_pinmux_status - not sure?
   test_sqfs_load - cannot be run more than once!
   test_bind_unbind_with_uclass - relies on previous test

The last one would be much better done as a C test, so it doesn't have
to deal with the changing driver tree. There isn't a lot of value in
running the test on a real board, since sandbox should find any bugs
in driver model or the 'bind' command.

If the above can be resolved we can enable parallel tests. On my test
machine (32 threads) it reduces the time from 38 seconds to 7.5s

To use this feature, see the documentaiton added, or:

   pip3 install pytest-xdist

   test/py/test.py -B sandbox --build-dir /tmp/xx -q -k 'not slow' -n32

Changes in v2:
- Add documentation for how to run parallel tests

Simon Glass (4):
  test: Allow vboot tests to run in parallel
  test: Allow hush tests to run in parallel
  test: Allow tpm2 tests to run in parallel
  doc: test: Explain how to run pytests in parallel

 doc/develop/py_testing.rst         | 30 +++++++++++++++++
 test/py/tests/test_hush_if_test.py | 20 ++++--------
 test/py/tests/test_tpm2.py         | 52 ++++++++++++++++++++++++++----
 test/py/tests/test_vboot.py        | 29 +++++++++--------
 4 files changed, 98 insertions(+), 33 deletions(-)

-- 
2.32.0.432.gabb21c7263-goog


^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2021-07-23  2:08 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-07-23  2:07 [PATCH v2 0/4] test: Try to deal with some co-dependent tests Simon Glass
2021-07-23  2:07 ` [PATCH v2 1/4] test: Allow vboot tests to run in parallel Simon Glass
2021-07-23  2:07 ` [PATCH v2 2/4] test: Allow hush " Simon Glass
2021-07-23  2:07 ` [PATCH v2 3/4] test: Allow tpm2 " Simon Glass
2021-07-23  2:07 ` [PATCH v2 4/4] doc: test: Explain how to run pytests " Simon Glass

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.