[U-Boot] [PATCH] Implement pytest-based test infrastructure

[Stephen Warren <swarren@wwwdotorg.org>]

On 11/21/2015 09:49 AM, Simon Glass wrote:
> Hi Stephen,
> 
> On 19 November 2015 at 12:09, Stephen Warren <swarren@wwwdotorg.org> wrote:
>>
>> On 11/19/2015 10:00 AM, Stephen Warren wrote:
>>>
>>> On 11/19/2015 07:45 AM, Simon Glass wrote:
>>>>
>>>> Hi Stephen,
>>>>
>>>> On 14 November 2015 at 23:53, Stephen Warren <swarren@wwwdotorg.org>
>>>> wrote:
>>>>>
>>>>> This tool aims to test U-Boot by executing U-Boot shell commands
>>>>> using the
>>>>> console interface. A single top-level script exists to execute or attach
>>>>> to the U-Boot console, run the entire script of tests against it, and
>>>>> summarize the results. Advantages of this approach are:
>>>>>
>>>>> - Testing is performed in the same way a user or script would interact
>>>>>    with U-Boot; there can be no disconnect.
>>>>> - There is no need to write or embed test-related code into U-Boot
>>>>> itself.
>>>>>    It is asserted that writing test-related code in Python is simpler
>>>>> and
>>>>>    more flexible that writing it all in C.
>>>>> - It is reasonably simple to interact with U-Boot in this way.
>>>>>
>>>>> A few simple tests are provided as examples. Soon, we should convert as
>>>>> many as possible of the other tests in test/* and test/cmd_ut.c too.
>>>>
>>>>
>>>> It's great to see this and thank you for putting in the effort!
>>>>
>>>> It looks like a good way of doing functional tests. I still see a role
>>>> for unit tests and things like test/dm. But if we can arrange to call
>>>> all U-Boot tests (unit and functional) from one 'test.py' command that
>>>> would be a win.
>>>>
>>>> I'll look more when I can get it to work - see below.
>>
>> ...
>>>
>>> made it print a message about checking the docs for missing
>>> requirements. I can probably patch the top-level test.py to do the same.
>>
>>
>> I've pushed such a patch to:
>>
>> git://github.com/swarren/u-boot.git tegra_dev
>> (the separate pytests branch has now been deleted)
>>
>> There are also a variety of other patches there related to this testing infra-structure. I guess I'll hold off sending them to the list until there's been some general feedback on the patches I've already posted, but feel free to pull the branch down and play with it. Note that it's likely to get rebased as I work.
> 
> OK I got it working thank you. It is horribly slow though - do you
> know what is holding it up? For me to takes 12 seconds to run the
> (very basic) tests.

It looks like pexpect includes a default delay to simulate human
interaction. If you edit test/py/uboot_console_base.py ensure_spawned()
and add the following somewhere soon after the assignment to self.p:

            self.p.delaybeforesend = 0

... that will more than halve the execution time. (8.3 -> 3.5s on my
5-year-old laptop).

That said, even your 12s or my 8.3s doesn't seem like a bad price to pay
for some easy-to-use automated testing.

> Also please see dm_test_usb_tree() which uses a console buffer to
> check command output.

OK, I'll take a look.

> I wonder if we should use something like that
> for simple unit tests, and use python for the more complicated
> functional tests?

I'm not sure that's a good idea; it'd be best to settle on a single way
of executing tests so that (a) people don't have to run/implement
different kinds of tests in different ways (b) we can leverage test code
across as many tests as possible.

(Well, doing unit tests and system level tests differently might be
necessary since one calls functions and the other uses the shell "user
interface", but having multiple ways of doing e.g. system tests doesn't
seem like a good idea.)