[U-Boot] Sandbox DT for testing (unit tests)

[U-Boot] Sandbox DT for testing (unit tests)

[Stephen Warren]

Simon,

I noticed that under sandbox, "ut dm" needs sandbox to have been started 
with arch/sandbox/dts/test.dtb. A few questions related to that:

a) Is it safe and does it make sense to always use that DT when running 
Sandbox for tests (e.g. under test/py)?

b) Does it make sense for that DT to be the default (perhaps bundled 
into the executable like other DT-using platforms, or perhaps the 
default value for the -d option if the user supplies none)?

c) Is it possible for "ut dm" to detect if the correct DT has been 
loaded (e.g. by reading some property only in that file as a marker) and 
only execute tests that don't rely on test.dtb if test.dtb isn't in use?

I think running "ut env" and "ut time" under test/py should be very 
easy, although the test log will only report overall status, not the 
status of each individual test within the ut invocation. That 
information will still be in the log file though. I'll go add tests for 
those two at least.

[U-Boot] Sandbox DT for testing (unit tests)

[Simon Glass]

Hi Stephen,

On 26 January 2016 at 15:36, Stephen Warren <swarren@wwwdotorg.org> wrote:
> Simon,
>
> I noticed that under sandbox, "ut dm" needs sandbox to have been started
> with arch/sandbox/dts/test.dtb. A few questions related to that:
>
> a) Is it safe and does it make sense to always use that DT when running
> Sandbox for tests (e.g. under test/py)?

Yes.

>
> b) Does it make sense for that DT to be the default (perhaps bundled into
> the executable like other DT-using platforms, or perhaps the default value
> for the -d option if the user supplies none)?

There is a separate sandbox.dts which is the default with the -D
option. I don't think the test.dts should be used by default at
present.

>
> c) Is it possible for "ut dm" to detect if the correct DT has been loaded
> (e.g. by reading some property only in that file as a marker) and only
> execute tests that don't rely on test.dtb if test.dtb isn't in use?

Sure - just look for something that should be there, or perhaps check
the compatible string or model in the root node?

>
> I think running "ut env" and "ut time" under test/py should be very easy,
> although the test log will only report overall status, not the status of
> each individual test within the ut invocation. That information will still
> be in the log file though. I'll go add tests for those two at least.

Sounds good. But presumably it would not be too hard to report the
status of each individual test?

Regards,
Simon

[U-Boot] Sandbox DT for testing (unit tests)

[Stephen Warren]

On 01/26/2016 04:08 PM, Simon Glass wrote:
> Hi Stephen,
>
> On 26 January 2016 at 15:36, Stephen Warren <swarren@wwwdotorg.org> wrote:
>> Simon,
>>
>> I noticed that under sandbox, "ut dm" needs sandbox to have been started
>> with arch/sandbox/dts/test.dtb. A few questions related to that:
>>
>> a) Is it safe and does it make sense to always use that DT when running
>> Sandbox for tests (e.g. under test/py)?
>
> Yes.
>
>>
>> b) Does it make sense for that DT to be the default (perhaps bundled into
>> the executable like other DT-using platforms, or perhaps the default value
>> for the -d option if the user supplies none)?
>
> There is a separate sandbox.dts which is the default with the -D
> option. I don't think the test.dts should be used by default at
> present.
>
>>
>> c) Is it possible for "ut dm" to detect if the correct DT has been loaded
>> (e.g. by reading some property only in that file as a marker) and only
>> execute tests that don't rely on test.dtb if test.dtb isn't in use?
>
> Sure - just look for something that should be there, or perhaps check
> the compatible string or model in the root node?
>
>>
>> I think running "ut env" and "ut time" under test/py should be very easy,
>> although the test log will only report overall status, not the status of
>> each individual test within the ut invocation. That information will still
>> be in the log file though. I'll go add tests for those two at least.
>
> Sounds good. But presumably it would not be too hard to report the
> status of each individual test?

Unfortunately, this would be quite hard.

The way pytest works is that it first scans the source tree (or some 
designated tree; currently test/py/tests/) for files, classes, and 
functions which define tests, and "collects" a list of them. Then, it 
iterates over each of those tests and executes them. Each collected test 
maps to one test invocation and one reported test status. In other 
words, the set of tests to execute is calculated before actually running 
any of the tests.

Right now, I'm writing a single test_ut_env() function (and hence a 
single pytest test) which executes "ut env" on the U-Boot command-line 
and reports a single status. That appears to be the minimum granularity 
of the U-Boot ut shell interface, and hence I can't easily make pytests 
that do anything smaller than that. There are obviously other 
functions/tests for "ut time", "ut_cmd", etc.

About the only way to do anything better would be to write a custom 
"collector" implementation which somehow parsed the C source at 
collection time to determine the set of sub-tests (or perhaps hard-code 
the list in Python instead) in order to generate more pytests. Then we 
could have one pytest which executed "ut env" and saved the results, and 
all the other tests would parse out just part of the console output to 
determine the individual test status. I expect this would be very 
fragile though.

[U-Boot] Sandbox DT for testing (unit tests)

[Simon Glass]

Hi Stephen,

On 26 January 2016 at 16:28, Stephen Warren <swarren@wwwdotorg.org> wrote:
> On 01/26/2016 04:08 PM, Simon Glass wrote:
>>
>> Hi Stephen,
>>
>> On 26 January 2016 at 15:36, Stephen Warren <swarren@wwwdotorg.org> wrote:
>>>
>>> Simon,
>>>
>>> I noticed that under sandbox, "ut dm" needs sandbox to have been started
>>> with arch/sandbox/dts/test.dtb. A few questions related to that:
>>>
>>> a) Is it safe and does it make sense to always use that DT when running
>>> Sandbox for tests (e.g. under test/py)?
>>
>>
>> Yes.
>>
>>>
>>> b) Does it make sense for that DT to be the default (perhaps bundled into
>>> the executable like other DT-using platforms, or perhaps the default
>>> value
>>> for the -d option if the user supplies none)?
>>
>>
>> There is a separate sandbox.dts which is the default with the -D
>> option. I don't think the test.dts should be used by default at
>> present.
>>
>>>
>>> c) Is it possible for "ut dm" to detect if the correct DT has been loaded
>>> (e.g. by reading some property only in that file as a marker) and only
>>> execute tests that don't rely on test.dtb if test.dtb isn't in use?
>>
>>
>> Sure - just look for something that should be there, or perhaps check
>> the compatible string or model in the root node?
>>
>>>
>>> I think running "ut env" and "ut time" under test/py should be very easy,
>>> although the test log will only report overall status, not the status of
>>> each individual test within the ut invocation. That information will
>>> still
>>> be in the log file though. I'll go add tests for those two at least.
>>
>>
>> Sounds good. But presumably it would not be too hard to report the
>> status of each individual test?
>
>
> Unfortunately, this would be quite hard.
>
> The way pytest works is that it first scans the source tree (or some
> designated tree; currently test/py/tests/) for files, classes, and functions
> which define tests, and "collects" a list of them. Then, it iterates over
> each of those tests and executes them. Each collected test maps to one test
> invocation and one reported test status. In other words, the set of tests to
> execute is calculated before actually running any of the tests.
>
> Right now, I'm writing a single test_ut_env() function (and hence a single
> pytest test) which executes "ut env" on the U-Boot command-line and reports
> a single status. That appears to be the minimum granularity of the U-Boot ut
> shell interface, and hence I can't easily make pytests that do anything
> smaller than that. There are obviously other functions/tests for "ut time",
> "ut_cmd", etc.
>
> About the only way to do anything better would be to write a custom
> "collector" implementation which somehow parsed the C source at collection
> time to determine the set of sub-tests (or perhaps hard-code the list in
> Python instead) in order to generate more pytests. Then we could have one
> pytest which executed "ut env" and saved the results, and all the other
> tests would parse out just part of the console output to determine the
> individual test status. I expect this would be very fragile though.

I was thinking of this approach. The output format is consistent (and
anyway you can change it), so collecting the output should be enough.
It's really just one invocation that results in >1 test result.

Regards,
Simon