* [PATCH v1] test: Allow simple glob pattern in the test name @ 2021-02-03 15:32 Andy Shevchenko 2021-02-05 3:17 ` Simon Glass 0 siblings, 1 reply; 11+ messages in thread From: Andy Shevchenko @ 2021-02-03 15:32 UTC (permalink / raw) To: u-boot When run `ut dm [test name]` allow to use simple pattern to run all tests started with given prefix. For example, to run all ACPI test cases: ut dm acpi* Signed-off-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com> --- test/dm/test-main.c | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/test/dm/test-main.c b/test/dm/test-main.c index 69458d62c869..f15527e13d7b 100644 --- a/test/dm/test-main.c +++ b/test/dm/test-main.c @@ -129,17 +129,23 @@ static bool dm_test_run_on_flattree(struct unit_test *test) static bool test_matches(const char *test_name, const char *find_name) { - if (!find_name) + size_t len = find_name ? strlen(find_name) : 0; + + /* Allow glob expansion in the test name */ + if (len && find_name[len - 1] == '*') + len--; + + if (!len) return true; - if (!strcmp(test_name, find_name)) + if (!strncmp(test_name, find_name, len)) return true; /* All tests have this prefix */ if (!strncmp(test_name, "dm_test_", 8)) test_name += 8; - if (!strcmp(test_name, find_name)) + if (!strncmp(test_name, find_name, len)) return true; return false; -- 2.30.0 ^ permalink raw reply related [flat|nested] 11+ messages in thread
* [PATCH v1] test: Allow simple glob pattern in the test name 2021-02-03 15:32 [PATCH v1] test: Allow simple glob pattern in the test name Andy Shevchenko @ 2021-02-05 3:17 ` Simon Glass 2021-02-05 17:34 ` Andy Shevchenko 0 siblings, 1 reply; 11+ messages in thread From: Simon Glass @ 2021-02-05 3:17 UTC (permalink / raw) To: u-boot Hi Andy, On Wed, 3 Feb 2021 at 08:32, Andy Shevchenko <andriy.shevchenko@linux.intel.com> wrote: > > When run `ut dm [test name]` allow to use simple pattern to run all tests > started with given prefix. For example, to run all ACPI test cases: > ut dm acpi* > > Signed-off-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com> > --- > test/dm/test-main.c | 12 +++++++++--- > 1 file changed, 9 insertions(+), 3 deletions(-) This seems like a good idea to me. Can you please send it rebased to u-boot-dm/test-working though? I have done a bit of refactoring and this function is now in test/test-main.c Regards, Simon ^ permalink raw reply [flat|nested] 11+ messages in thread
* [PATCH v1] test: Allow simple glob pattern in the test name 2021-02-05 3:17 ` Simon Glass @ 2021-02-05 17:34 ` Andy Shevchenko 2021-02-05 18:15 ` Andy Shevchenko 0 siblings, 1 reply; 11+ messages in thread From: Andy Shevchenko @ 2021-02-05 17:34 UTC (permalink / raw) To: u-boot On Thu, Feb 04, 2021 at 08:17:24PM -0700, Simon Glass wrote: > On Wed, 3 Feb 2021 at 08:32, Andy Shevchenko > <andriy.shevchenko@linux.intel.com> wrote: > > > > When run `ut dm [test name]` allow to use simple pattern to run all tests > > started with given prefix. For example, to run all ACPI test cases: > > ut dm acpi* > > > > Signed-off-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com> > > --- > > test/dm/test-main.c | 12 +++++++++--- > > 1 file changed, 9 insertions(+), 3 deletions(-) > > This seems like a good idea to me. > > Can you please send it rebased to u-boot-dm/test-working though? I > have done a bit of refactoring and this function is now in > test/test-main.c I will look at it. Btw, you have an issue there, i.e. if test case failed, all percentage after it goes red, which is wrong. -- With Best Regards, Andy Shevchenko ^ permalink raw reply [flat|nested] 11+ messages in thread
* [PATCH v1] test: Allow simple glob pattern in the test name 2021-02-05 17:34 ` Andy Shevchenko @ 2021-02-05 18:15 ` Andy Shevchenko 2021-02-05 18:18 ` Andy Shevchenko 2021-02-05 19:17 ` Andy Shevchenko 0 siblings, 2 replies; 11+ messages in thread From: Andy Shevchenko @ 2021-02-05 18:15 UTC (permalink / raw) To: u-boot On Fri, Feb 05, 2021 at 07:34:49PM +0200, Andy Shevchenko wrote: > On Thu, Feb 04, 2021 at 08:17:24PM -0700, Simon Glass wrote: ... > Btw, you have an issue there, i.e. if test case failed, all percentage after it > goes red, which is wrong. One more thing, is it known bug that either in the original code, or in your new branch the following test case is 100% failed? /* Non-existent in DTB */ ut_asserteq(FDT_ADDR_T_NONE, dev_read_addr(dev)); Can you fix this sooner than later, please? -- With Best Regards, Andy Shevchenko ^ permalink raw reply [flat|nested] 11+ messages in thread
* [PATCH v1] test: Allow simple glob pattern in the test name 2021-02-05 18:15 ` Andy Shevchenko @ 2021-02-05 18:18 ` Andy Shevchenko 2021-02-05 19:17 ` Andy Shevchenko 1 sibling, 0 replies; 11+ messages in thread From: Andy Shevchenko @ 2021-02-05 18:18 UTC (permalink / raw) To: u-boot On Fri, Feb 05, 2021 at 08:15:25PM +0200, Andy Shevchenko wrote: > On Fri, Feb 05, 2021 at 07:34:49PM +0200, Andy Shevchenko wrote: > > On Thu, Feb 04, 2021 at 08:17:24PM -0700, Simon Glass wrote: > > ... > > > Btw, you have an issue there, i.e. if test case failed, all percentage after it > > goes red, which is wrong. > > One more thing, is it known bug that either in the original code, or in your > new branch the following test case is 100% failed? > > /* Non-existent in DTB */ > ut_asserteq(FDT_ADDR_T_NONE, dev_read_addr(dev)); > > Can you fix this sooner than later, please? Another annoying issue is the 2+ run on uncleaned build folder, i.e. on the second and following runs it starts out of the sudden to ask my sudo password. The later to me is a highest priority, now I have to run rm -rf build-sandbox each time, and if I forget - wasted time! Can you fix this as soon as possible, please? -- With Best Regards, Andy Shevchenko ^ permalink raw reply [flat|nested] 11+ messages in thread
* [PATCH v1] test: Allow simple glob pattern in the test name 2021-02-05 18:15 ` Andy Shevchenko 2021-02-05 18:18 ` Andy Shevchenko @ 2021-02-05 19:17 ` Andy Shevchenko 2021-02-05 20:46 ` Andy Shevchenko 1 sibling, 1 reply; 11+ messages in thread From: Andy Shevchenko @ 2021-02-05 19:17 UTC (permalink / raw) To: u-boot On Fri, Feb 05, 2021 at 08:15:25PM +0200, Andy Shevchenko wrote: > On Fri, Feb 05, 2021 at 07:34:49PM +0200, Andy Shevchenko wrote: > > On Thu, Feb 04, 2021 at 08:17:24PM -0700, Simon Glass wrote: > > ... > > > Btw, you have an issue there, i.e. if test case failed, all percentage after it > > goes red, which is wrong. > > One more thing, is it known bug that either in the original code, or in your > new branch the following test case is 100% failed? > > /* Non-existent in DTB */ > ut_asserteq(FDT_ADDR_T_NONE, dev_read_addr(dev)); > > Can you fix this sooner than later, please? Actually it seems this very patch makes the issue visible (I suppose something with test case names). -- With Best Regards, Andy Shevchenko ^ permalink raw reply [flat|nested] 11+ messages in thread
* [PATCH v1] test: Allow simple glob pattern in the test name 2021-02-05 19:17 ` Andy Shevchenko @ 2021-02-05 20:46 ` Andy Shevchenko 2021-02-07 14:37 ` Simon Glass 0 siblings, 1 reply; 11+ messages in thread From: Andy Shevchenko @ 2021-02-05 20:46 UTC (permalink / raw) To: u-boot On Fri, Feb 05, 2021 at 09:17:27PM +0200, Andy Shevchenko wrote: > On Fri, Feb 05, 2021 at 08:15:25PM +0200, Andy Shevchenko wrote: > > On Fri, Feb 05, 2021 at 07:34:49PM +0200, Andy Shevchenko wrote: > > > On Thu, Feb 04, 2021 at 08:17:24PM -0700, Simon Glass wrote: > > > > ... > > > > > Btw, you have an issue there, i.e. if test case failed, all percentage after it > > > goes red, which is wrong. > > > > One more thing, is it known bug that either in the original code, or in your > > new branch the following test case is 100% failed? > > > > /* Non-existent in DTB */ > > ut_asserteq(FDT_ADDR_T_NONE, dev_read_addr(dev)); > > > > Can you fix this sooner than later, please? > > Actually it seems this very patch makes the issue visible (I suppose something > with test case names). Okay, actually there *is* a problem with the test suite, i.e. you may not run some test cases twice during the same session. -- With Best Regards, Andy Shevchenko ^ permalink raw reply [flat|nested] 11+ messages in thread
* [PATCH v1] test: Allow simple glob pattern in the test name 2021-02-05 20:46 ` Andy Shevchenko @ 2021-02-07 14:37 ` Simon Glass 2021-02-08 11:34 ` Andy Shevchenko 0 siblings, 1 reply; 11+ messages in thread From: Simon Glass @ 2021-02-07 14:37 UTC (permalink / raw) To: u-boot Hi Andy, On Fri, 5 Feb 2021 at 13:46, Andy Shevchenko <andriy.shevchenko@linux.intel.com> wrote: > > On Fri, Feb 05, 2021 at 09:17:27PM +0200, Andy Shevchenko wrote: > > On Fri, Feb 05, 2021 at 08:15:25PM +0200, Andy Shevchenko wrote: > > > On Fri, Feb 05, 2021 at 07:34:49PM +0200, Andy Shevchenko wrote: > > > > On Thu, Feb 04, 2021 at 08:17:24PM -0700, Simon Glass wrote: > > > > > > ... > > > > > > > Btw, you have an issue there, i.e. if test case failed, all percentage after it > > > > goes red, which is wrong. > > > > > > One more thing, is it known bug that either in the original code, or in your > > > new branch the following test case is 100% failed? > > > > > > /* Non-existent in DTB */ > > > ut_asserteq(FDT_ADDR_T_NONE, dev_read_addr(dev)); > > > > > > Can you fix this sooner than later, please? > > > > Actually it seems this very patch makes the issue visible (I suppose something > > with test case names). > > Okay, actually there *is* a problem with the test suite, i.e. you may not run > some test cases twice during the same session. If this is related to the squashfs tests? I have disabled them for now in my local tree. Regards, Simon ^ permalink raw reply [flat|nested] 11+ messages in thread
* [PATCH v1] test: Allow simple glob pattern in the test name 2021-02-07 14:37 ` Simon Glass @ 2021-02-08 11:34 ` Andy Shevchenko 2021-02-08 17:07 ` Simon Glass 0 siblings, 1 reply; 11+ messages in thread From: Andy Shevchenko @ 2021-02-08 11:34 UTC (permalink / raw) To: u-boot On Sun, Feb 07, 2021 at 07:37:55AM -0700, Simon Glass wrote: > On Fri, 5 Feb 2021 at 13:46, Andy Shevchenko > <andriy.shevchenko@linux.intel.com> wrote: > > On Fri, Feb 05, 2021 at 09:17:27PM +0200, Andy Shevchenko wrote: > > > On Fri, Feb 05, 2021 at 08:15:25PM +0200, Andy Shevchenko wrote: > > > > On Fri, Feb 05, 2021 at 07:34:49PM +0200, Andy Shevchenko wrote: > > > > > On Thu, Feb 04, 2021 at 08:17:24PM -0700, Simon Glass wrote: > > > > > > > > ... > > > > > > > > > Btw, you have an issue there, i.e. if test case failed, all percentage after it > > > > > goes red, which is wrong. > > > > > > > > One more thing, is it known bug that either in the original code, or in your > > > > new branch the following test case is 100% failed? > > > > > > > > /* Non-existent in DTB */ > > > > ut_asserteq(FDT_ADDR_T_NONE, dev_read_addr(dev)); > > > > > > > > Can you fix this sooner than later, please? > > > > > > Actually it seems this very patch makes the issue visible (I suppose something > > > with test case names). > > > > Okay, actually there *is* a problem with the test suite, i.e. you may not run > > some test cases twice during the same session. > > If this is related to the squashfs tests? I have disabled them for now > in my local tree. I don't think it's related to the certain test case, it's a test suite design issue and how some of the test cases have been written. So, if you run u-boot application and manually run `ut dm` in there, - first time: Failures: 0 - second time: Failures: 9 - third and next time: Failures: 12 Means that 12 test cases are written badly. -- With Best Regards, Andy Shevchenko ^ permalink raw reply [flat|nested] 11+ messages in thread
* [PATCH v1] test: Allow simple glob pattern in the test name 2021-02-08 11:34 ` Andy Shevchenko @ 2021-02-08 17:07 ` Simon Glass 2021-02-08 17:34 ` Andy Shevchenko 0 siblings, 1 reply; 11+ messages in thread From: Simon Glass @ 2021-02-08 17:07 UTC (permalink / raw) To: u-boot Hi Andy, On Mon, 8 Feb 2021 at 04:34, Andy Shevchenko <andriy.shevchenko@linux.intel.com> wrote: > > On Sun, Feb 07, 2021 at 07:37:55AM -0700, Simon Glass wrote: > > On Fri, 5 Feb 2021 at 13:46, Andy Shevchenko > > <andriy.shevchenko@linux.intel.com> wrote: > > > On Fri, Feb 05, 2021 at 09:17:27PM +0200, Andy Shevchenko wrote: > > > > On Fri, Feb 05, 2021 at 08:15:25PM +0200, Andy Shevchenko wrote: > > > > > On Fri, Feb 05, 2021 at 07:34:49PM +0200, Andy Shevchenko wrote: > > > > > > On Thu, Feb 04, 2021 at 08:17:24PM -0700, Simon Glass wrote: > > > > > > > > > > ... > > > > > > > > > > > Btw, you have an issue there, i.e. if test case failed, all percentage after it > > > > > > goes red, which is wrong. > > > > > > > > > > One more thing, is it known bug that either in the original code, or in your > > > > > new branch the following test case is 100% failed? > > > > > > > > > > /* Non-existent in DTB */ > > > > > ut_asserteq(FDT_ADDR_T_NONE, dev_read_addr(dev)); > > > > > > > > > > Can you fix this sooner than later, please? > > > > > > > > Actually it seems this very patch makes the issue visible (I suppose something > > > > with test case names). > > > > > > Okay, actually there *is* a problem with the test suite, i.e. you may not run > > > some test cases twice during the same session. > > > > If this is related to the squashfs tests? I have disabled them for now > > in my local tree. > > I don't think it's related to the certain test case, it's a test suite design issue and how some of the test cases have been written. So, if you run u-boot application and manually run `ut dm` in there, > - first time: Failures: 0 > - second time: Failures: 9 > - third and next time: Failures: 12 > > Means that 12 test cases are written badly. OK I see. It did not use to be possible to pass all tests without running through pytest, which does some setup. You still need to create a 'spi.bin' file for the SPI tests to pass. I'm not sure what prevents multiple runs without quitting U-Boot, but I suspect it is just some state hanging around, as we don't reset everything. For example, sandbox provides control of what happens when a sysreset is performed, and perhaps that is not reset to initial values correct by state_reset_for_test().. Regards, Simon ^ permalink raw reply [flat|nested] 11+ messages in thread
* [PATCH v1] test: Allow simple glob pattern in the test name 2021-02-08 17:07 ` Simon Glass @ 2021-02-08 17:34 ` Andy Shevchenko 0 siblings, 0 replies; 11+ messages in thread From: Andy Shevchenko @ 2021-02-08 17:34 UTC (permalink / raw) To: u-boot On Mon, Feb 08, 2021 at 10:07:55AM -0700, Simon Glass wrote: > On Mon, 8 Feb 2021 at 04:34, Andy Shevchenko > <andriy.shevchenko@linux.intel.com> wrote: > > On Sun, Feb 07, 2021 at 07:37:55AM -0700, Simon Glass wrote: > > > On Fri, 5 Feb 2021 at 13:46, Andy Shevchenko > > > <andriy.shevchenko@linux.intel.com> wrote: > > > > On Fri, Feb 05, 2021 at 09:17:27PM +0200, Andy Shevchenko wrote: > > > > > On Fri, Feb 05, 2021 at 08:15:25PM +0200, Andy Shevchenko wrote: > > > > > > On Fri, Feb 05, 2021 at 07:34:49PM +0200, Andy Shevchenko wrote: > > > > > > > On Thu, Feb 04, 2021 at 08:17:24PM -0700, Simon Glass wrote: > > > > > > > > > > > > ... > > > > > > > > > > > > > Btw, you have an issue there, i.e. if test case failed, all percentage after it > > > > > > > goes red, which is wrong. > > > > > > > > > > > > One more thing, is it known bug that either in the original code, or in your > > > > > > new branch the following test case is 100% failed? > > > > > > > > > > > > /* Non-existent in DTB */ > > > > > > ut_asserteq(FDT_ADDR_T_NONE, dev_read_addr(dev)); > > > > > > > > > > > > Can you fix this sooner than later, please? > > > > > > > > > > Actually it seems this very patch makes the issue visible (I suppose something > > > > > with test case names). > > > > > > > > Okay, actually there *is* a problem with the test suite, i.e. you may not run > > > > some test cases twice during the same session. > > > > > > If this is related to the squashfs tests? I have disabled them for now > > > in my local tree. > > > > I don't think it's related to the certain test case, it's a test suite design issue and how some of the test cases have been written. So, if you run u-boot application and manually run `ut dm` in there, > > - first time: Failures: 0 > > - second time: Failures: 9 > > - third and next time: Failures: 12 > > > > Means that 12 test cases are written badly. > > OK I see. It did not use to be possible to pass all tests without > running through pytest, which does some setup. You still need to > create a 'spi.bin' file for the SPI tests to pass. I run it _after_ the official pytest passed. So, every file is prepared. > I'm not sure what prevents multiple runs without quitting U-Boot, but > I suspect it is just some state hanging around, as we don't reset > everything. For example, sandbox provides control of what happens when > a sysreset is performed, and perhaps that is not reset to initial > values correct by state_reset_for_test().. I believe this is the case in most of the cases. -- With Best Regards, Andy Shevchenko ^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2021-02-08 17:34 UTC | newest] Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-02-03 15:32 [PATCH v1] test: Allow simple glob pattern in the test name Andy Shevchenko 2021-02-05 3:17 ` Simon Glass 2021-02-05 17:34 ` Andy Shevchenko 2021-02-05 18:15 ` Andy Shevchenko 2021-02-05 18:18 ` Andy Shevchenko 2021-02-05 19:17 ` Andy Shevchenko 2021-02-05 20:46 ` Andy Shevchenko 2021-02-07 14:37 ` Simon Glass 2021-02-08 11:34 ` Andy Shevchenko 2021-02-08 17:07 ` Simon Glass 2021-02-08 17:34 ` Andy Shevchenko
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.