Re: [PATCH 25/45] test: Support tests which can only be run manually

[Simon Glass <sjg@chromium.org>]

Hi Heinrich,

On Mon, 26 Sept 2022 at 00:56, Heinrich Schuchardt <xypron.glpk@gmx.de> wrote:
>
> On 9/25/22 17:02, Simon Glass wrote:
> > At present we normally write tests either in Python or in C. But most
> > Python tests end up doing a lot of checks which would be better done in C.
> > Checks done in C are orders of magnitude faster and it is possible to get
> > full access to U-Boot's internal workings, rather than just relying on
> > the command line.
> >
> > The model is to have a Python test set up some things and then use C code
> > (in a unit test) to check that they were done correctly. But we don't want
> > those checks to happen as part of normal test running, since each C unit
> > tests is dependent on the associate Python tests, so cannot run without
> > it.
> >
> > To acheive this, add a new UT_TESTF_MANUAL flag to use with the C 'check'
> > tests, so that they can be skipped by default when the 'ut' command is
> > used. Require that tests have a name ending with '_norun', so that pytest
> > knows to skip them.
> >
> > Signed-off-by: Simon Glass <sjg@chromium.org>
> > ---
> >
> >   arch/sandbox/cpu/spl.c        |  2 +-
> >   doc/develop/tests_writing.rst | 22 ++++++++++++++++++++++
> >   include/test/test.h           |  8 ++++++++
> >   include/test/ut.h             |  4 +++-
> >   test/cmd_ut.c                 | 16 +++++++++++++---
> >   test/dm/test-dm.c             |  2 +-
> >   test/py/conftest.py           |  8 +++++++-
> >   test/test-main.c              | 27 ++++++++++++++++++++++++++-
> >   8 files changed, 81 insertions(+), 8 deletions(-)
> >
> > diff --git a/arch/sandbox/cpu/spl.c b/arch/sandbox/cpu/spl.c
> > index 6d4981152bb..2678370481a 100644
> > --- a/arch/sandbox/cpu/spl.c
> > +++ b/arch/sandbox/cpu/spl.c
> > @@ -90,7 +90,7 @@ void spl_board_init(void)
> >               int ret;
> >
> >               ret = ut_run_list("spl", NULL, tests, count,
> > -                               state->select_unittests, 1);
> > +                               state->select_unittests, 1, false);
> >               /* continue execution into U-Boot */
> >       }
> >   }
> > diff --git a/doc/develop/tests_writing.rst b/doc/develop/tests_writing.rst
> > index 1ddf7a353a7..d3ffc550300 100644
> > --- a/doc/develop/tests_writing.rst
> > +++ b/doc/develop/tests_writing.rst
> > @@ -74,6 +74,28 @@ NOT rely on running with sandbox, but instead should function correctly on any
> >   board supported by U-Boot.
> >
> >
> > +Mixing Python and C
> > +-------------------
> > +
> > +The best of both worlds is sometimes to have a Python test set things up and
> > +perform some operations, with a 'checker' C unit test doing the checks
> > +afterwards. This can be achieved with these steps:
> > +
> > +- Add the `UT_TESTF_MANUAL` flag to the checker test so that the `ut` command
> > +  does not run it by default
> > +- Add a `_norun` suffix to the name so that pytest knows to skip it too
> > +
> > +In your Python test use the `-f` flag to the `ut` command to force the checker
> > +test to run it, e.g.::
> > +
> > +   # Do the Python part
> > +   host load ...
> > +   bootm ...
> > +
> > +   # Run the checker to make sure that everything worked
> > +   ut -f bootstd vbe_test_fixup_norun
>
> Shouldn't the documentation describes what a test looks like that can
> only be run manually?

It is just a normal test and I figure there is an example in the code.
What sort of info is missing here?

Regards,
Simon