* [U-Boot] Sandbox DT for testing (unit tests) @ 2016-01-26 22:36 Stephen Warren 2016-01-26 23:08 ` Simon Glass 0 siblings, 1 reply; 4+ messages in thread From: Stephen Warren @ 2016-01-26 22:36 UTC (permalink / raw) To: u-boot Simon, I noticed that under sandbox, "ut dm" needs sandbox to have been started with arch/sandbox/dts/test.dtb. A few questions related to that: a) Is it safe and does it make sense to always use that DT when running Sandbox for tests (e.g. under test/py)? b) Does it make sense for that DT to be the default (perhaps bundled into the executable like other DT-using platforms, or perhaps the default value for the -d option if the user supplies none)? c) Is it possible for "ut dm" to detect if the correct DT has been loaded (e.g. by reading some property only in that file as a marker) and only execute tests that don't rely on test.dtb if test.dtb isn't in use? I think running "ut env" and "ut time" under test/py should be very easy, although the test log will only report overall status, not the status of each individual test within the ut invocation. That information will still be in the log file though. I'll go add tests for those two at least. ^ permalink raw reply [flat|nested] 4+ messages in thread
* [U-Boot] Sandbox DT for testing (unit tests) 2016-01-26 22:36 [U-Boot] Sandbox DT for testing (unit tests) Stephen Warren @ 2016-01-26 23:08 ` Simon Glass 2016-01-26 23:28 ` Stephen Warren 0 siblings, 1 reply; 4+ messages in thread From: Simon Glass @ 2016-01-26 23:08 UTC (permalink / raw) To: u-boot Hi Stephen, On 26 January 2016 at 15:36, Stephen Warren <swarren@wwwdotorg.org> wrote: > Simon, > > I noticed that under sandbox, "ut dm" needs sandbox to have been started > with arch/sandbox/dts/test.dtb. A few questions related to that: > > a) Is it safe and does it make sense to always use that DT when running > Sandbox for tests (e.g. under test/py)? Yes. > > b) Does it make sense for that DT to be the default (perhaps bundled into > the executable like other DT-using platforms, or perhaps the default value > for the -d option if the user supplies none)? There is a separate sandbox.dts which is the default with the -D option. I don't think the test.dts should be used by default at present. > > c) Is it possible for "ut dm" to detect if the correct DT has been loaded > (e.g. by reading some property only in that file as a marker) and only > execute tests that don't rely on test.dtb if test.dtb isn't in use? Sure - just look for something that should be there, or perhaps check the compatible string or model in the root node? > > I think running "ut env" and "ut time" under test/py should be very easy, > although the test log will only report overall status, not the status of > each individual test within the ut invocation. That information will still > be in the log file though. I'll go add tests for those two at least. Sounds good. But presumably it would not be too hard to report the status of each individual test? Regards, Simon ^ permalink raw reply [flat|nested] 4+ messages in thread
* [U-Boot] Sandbox DT for testing (unit tests) 2016-01-26 23:08 ` Simon Glass @ 2016-01-26 23:28 ` Stephen Warren 2016-01-26 23:35 ` Simon Glass 0 siblings, 1 reply; 4+ messages in thread From: Stephen Warren @ 2016-01-26 23:28 UTC (permalink / raw) To: u-boot On 01/26/2016 04:08 PM, Simon Glass wrote: > Hi Stephen, > > On 26 January 2016 at 15:36, Stephen Warren <swarren@wwwdotorg.org> wrote: >> Simon, >> >> I noticed that under sandbox, "ut dm" needs sandbox to have been started >> with arch/sandbox/dts/test.dtb. A few questions related to that: >> >> a) Is it safe and does it make sense to always use that DT when running >> Sandbox for tests (e.g. under test/py)? > > Yes. > >> >> b) Does it make sense for that DT to be the default (perhaps bundled into >> the executable like other DT-using platforms, or perhaps the default value >> for the -d option if the user supplies none)? > > There is a separate sandbox.dts which is the default with the -D > option. I don't think the test.dts should be used by default at > present. > >> >> c) Is it possible for "ut dm" to detect if the correct DT has been loaded >> (e.g. by reading some property only in that file as a marker) and only >> execute tests that don't rely on test.dtb if test.dtb isn't in use? > > Sure - just look for something that should be there, or perhaps check > the compatible string or model in the root node? > >> >> I think running "ut env" and "ut time" under test/py should be very easy, >> although the test log will only report overall status, not the status of >> each individual test within the ut invocation. That information will still >> be in the log file though. I'll go add tests for those two at least. > > Sounds good. But presumably it would not be too hard to report the > status of each individual test? Unfortunately, this would be quite hard. The way pytest works is that it first scans the source tree (or some designated tree; currently test/py/tests/) for files, classes, and functions which define tests, and "collects" a list of them. Then, it iterates over each of those tests and executes them. Each collected test maps to one test invocation and one reported test status. In other words, the set of tests to execute is calculated before actually running any of the tests. Right now, I'm writing a single test_ut_env() function (and hence a single pytest test) which executes "ut env" on the U-Boot command-line and reports a single status. That appears to be the minimum granularity of the U-Boot ut shell interface, and hence I can't easily make pytests that do anything smaller than that. There are obviously other functions/tests for "ut time", "ut_cmd", etc. About the only way to do anything better would be to write a custom "collector" implementation which somehow parsed the C source at collection time to determine the set of sub-tests (or perhaps hard-code the list in Python instead) in order to generate more pytests. Then we could have one pytest which executed "ut env" and saved the results, and all the other tests would parse out just part of the console output to determine the individual test status. I expect this would be very fragile though. ^ permalink raw reply [flat|nested] 4+ messages in thread
* [U-Boot] Sandbox DT for testing (unit tests) 2016-01-26 23:28 ` Stephen Warren @ 2016-01-26 23:35 ` Simon Glass 0 siblings, 0 replies; 4+ messages in thread From: Simon Glass @ 2016-01-26 23:35 UTC (permalink / raw) To: u-boot Hi Stephen, On 26 January 2016 at 16:28, Stephen Warren <swarren@wwwdotorg.org> wrote: > On 01/26/2016 04:08 PM, Simon Glass wrote: >> >> Hi Stephen, >> >> On 26 January 2016 at 15:36, Stephen Warren <swarren@wwwdotorg.org> wrote: >>> >>> Simon, >>> >>> I noticed that under sandbox, "ut dm" needs sandbox to have been started >>> with arch/sandbox/dts/test.dtb. A few questions related to that: >>> >>> a) Is it safe and does it make sense to always use that DT when running >>> Sandbox for tests (e.g. under test/py)? >> >> >> Yes. >> >>> >>> b) Does it make sense for that DT to be the default (perhaps bundled into >>> the executable like other DT-using platforms, or perhaps the default >>> value >>> for the -d option if the user supplies none)? >> >> >> There is a separate sandbox.dts which is the default with the -D >> option. I don't think the test.dts should be used by default at >> present. >> >>> >>> c) Is it possible for "ut dm" to detect if the correct DT has been loaded >>> (e.g. by reading some property only in that file as a marker) and only >>> execute tests that don't rely on test.dtb if test.dtb isn't in use? >> >> >> Sure - just look for something that should be there, or perhaps check >> the compatible string or model in the root node? >> >>> >>> I think running "ut env" and "ut time" under test/py should be very easy, >>> although the test log will only report overall status, not the status of >>> each individual test within the ut invocation. That information will >>> still >>> be in the log file though. I'll go add tests for those two at least. >> >> >> Sounds good. But presumably it would not be too hard to report the >> status of each individual test? > > > Unfortunately, this would be quite hard. > > The way pytest works is that it first scans the source tree (or some > designated tree; currently test/py/tests/) for files, classes, and functions > which define tests, and "collects" a list of them. Then, it iterates over > each of those tests and executes them. Each collected test maps to one test > invocation and one reported test status. In other words, the set of tests to > execute is calculated before actually running any of the tests. > > Right now, I'm writing a single test_ut_env() function (and hence a single > pytest test) which executes "ut env" on the U-Boot command-line and reports > a single status. That appears to be the minimum granularity of the U-Boot ut > shell interface, and hence I can't easily make pytests that do anything > smaller than that. There are obviously other functions/tests for "ut time", > "ut_cmd", etc. > > About the only way to do anything better would be to write a custom > "collector" implementation which somehow parsed the C source at collection > time to determine the set of sub-tests (or perhaps hard-code the list in > Python instead) in order to generate more pytests. Then we could have one > pytest which executed "ut env" and saved the results, and all the other > tests would parse out just part of the console output to determine the > individual test status. I expect this would be very fragile though. I was thinking of this approach. The output format is consistent (and anyway you can change it), so collecting the output should be enough. It's really just one invocation that results in >1 test result. Regards, Simon ^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2016-01-26 23:35 UTC | newest] Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2016-01-26 22:36 [U-Boot] Sandbox DT for testing (unit tests) Stephen Warren 2016-01-26 23:08 ` Simon Glass 2016-01-26 23:28 ` Stephen Warren 2016-01-26 23:35 ` Simon Glass
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.