[U-Boot] [PATCH V2 1/7] test/py: Implement pytest infrastructure

[Simon Glass <sjg@chromium.org>]

Hi Heiko,

On 16 December 2015 at 22:45, Heiko Schocher <hs@denx.de> wrote:
> Hello Stephen,
>
>
> Am 16.12.2015 um 17:27 schrieb Stephen Warren:
>>
>> On 12/16/2015 08:11 AM, Michal Simek wrote:
>>>
>>> On 9.12.2015 17:32, Stephen Warren wrote:
>>>>
>>>> On 12/02/2015 03:18 PM, Stephen Warren wrote:
>>>>>
>>>>> This tool aims to test U-Boot by executing U-Boot shell commands using
>>>>> the
>>>>> console interface. A single top-level script exists to execute or
>>>>> attach
>>>>> to the U-Boot console, run the entire script of tests against it, and
>>>>> summarize the results. Advantages of this approach are:
>>>>>
>>>>> - Testing is performed in the same way a user or script would interact
>>>>>     with U-Boot; there can be no disconnect.
>>>>> - There is no need to write or embed test-related code into U-Boot
>>>>> itself.
>>>>>     It is asserted that writing test-related code in Python is simpler
>>>>> and
>>>>>     more flexible that writing it all in C.
>>>>> - It is reasonably simple to interact with U-Boot in this way.
>>>>>
>>>>> A few simple tests are provided as examples. Soon, we should convert as
>>>>> many as possible of the other tests in test/* and test/cmd_ut.c too.
>>>>>
>>>>> In the future, I hope to publish (out-of-tree) the hook scripts, relay
>>>>> control utilities, and udev rules I will use for my own HW setup.
>>>>
>>>>
>>>> I finally got permission to publish these. Examples are at:
>>>> https://github.com/swarren/uboot-test-hooks
>>>
>>>
>>> Interesting. What's the normal setup which you have for the board?
>>> I see from your description that you use numato usb relay - I expect one
>>> with more channels for reset.
>>> Then you are able to control boot mode. Is it also via the same relay?
>>> How do you power up the board?
>>
>>
>> In my current setup I leave the board on all the time (or rather, manually
>> turn on the power when
>> I'm about to run the tests). Automating control of the power source is a
>> step I'll take later.
>
>
> Maybe you give tbot (I mentioned it in this thread) a chance?
>
> There, this things are automated, also you can do linux (and other console
> based)
> tests ... Currently added a testcase [1] which adds patches from patchwork,
> which are in a users ToDo list, to a git tree! In this case it is a u-boot
> git
> tree ... checks them with checkpatch, compiles it, and tries the new image
> on
> the board and calls testcases ... fully automated in a now weekly build [2]
> ...
> (only weekly, but thats a setup parameter in buildbot, as I have tbot and
> buildbot
> running on a raspberry pi)
> and don;t forget, I have the board not where I run tbot, the boards are
> ~1000km
> from me .. So, it is possible to add a U-Boot Testsetup on a server,
> and test boards all over the world ...

This sounds like a great development.

How can we get this so that it can be used by U-Boot people? Do you
think you could add a README to the mainline, or some scripts to aid
setting it up? I would be interested in setting up a few boards that
run continuous testing, and I suspect others would also if it were
easier.

>
>> For Tegra, there are two important signals: reset and "force recovery".
>> Each of these has a separate
>> relay, so the system currently uses 2 relays per target board. The numato
>> relay board I own has 8
>> relays, although there are a number of different models.
>>
>> On Tegra, when reset is pulsed:
>>
>> - If force-recovery is connected, the SoC enters USB recovery mode. In
>> this state, SW can be
>> downloaded over USB into RAM and executed.
>>
>> - If force-recovery is not connected, the SoC boots normally, from SW
>> stored in flash (eMMC, SPI, ...)
>>
>> The example scripts always use recovery mode to download U-Boot into RAM
>> rather than writing it to
>> flash first and then resetting. This saves wear cycles on the flash, but
>> does mean the download
>> happens in the "reset" rather than "flash" script, which may make the
>> examples a bit different than
>> for some other SoCs.
>>
>> Finally, the example scripts support two boards; my home/laptop dev setup
>> that uses a Numato relay
>> board to control the signals to the board I use there, and my work desktop
>> dev setup that uses our
>> "PM342" debug board to controll the signals. The latter works logically
>> the same as the numato relay
>> board, except it contains electronic switches driven by an FTDI chip.
>
>
> I seperated such functions into a seperate python class, so such setup
> specific things should be easy to add for other (in tbot called lab)
> setups ...
>
> Currently I have only a bdi testcase, which flashes a new image into
> the board, when it is broken ... but such relay things can be added
> of course ...
>
> bye,
> Heiko
> [1] get patchlist from patchwork
>
> https://github.com/hsdenx/tbot/commit/0bcaf4877e7aad4df2039913dcb6e85303a0b15f
>     apply them to git tree:
>
> https://github.com/hsdenx/tbot/commit/b7e2de3731252b518754cc3f71dc782559b0cad6
>     and use this on the smartweb board:
>
> https://github.com/hsdenx/tbot/commit/56a1ac18e5730ae9ffa7686acfc52877272be91d
>
> [2] weekly started testcase for the smartweb board:
>     http://xeidos.ddns.net/buildbot/builders/smartweb_dfu
>     log with the testcases from [1]
>
> http://xeidos.ddns.net/buildbot/builders/smartweb_dfu/builds/32/steps/shell/logs/tbotlog
>
> --
> DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
> HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
> _______________________________________________
> U-Boot mailing list
> U-Boot at lists.denx.de
> http://lists.denx.de/mailman/listinfo/u-boot

Regards,
Simon