* [PATCH v1] test: Allow simple glob pattern in the test name
@ 2021-02-03 15:32 Andy Shevchenko
2021-02-05 3:17 ` Simon Glass
0 siblings, 1 reply; 11+ messages in thread
From: Andy Shevchenko @ 2021-02-03 15:32 UTC (permalink / raw)
To: u-boot
When run `ut dm [test name]` allow to use simple pattern to run all tests
started with given prefix. For example, to run all ACPI test cases:
ut dm acpi*
Signed-off-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
---
test/dm/test-main.c | 12 +++++++++---
1 file changed, 9 insertions(+), 3 deletions(-)
diff --git a/test/dm/test-main.c b/test/dm/test-main.c
index 69458d62c869..f15527e13d7b 100644
--- a/test/dm/test-main.c
+++ b/test/dm/test-main.c
@@ -129,17 +129,23 @@ static bool dm_test_run_on_flattree(struct unit_test *test)
static bool test_matches(const char *test_name, const char *find_name)
{
- if (!find_name)
+ size_t len = find_name ? strlen(find_name) : 0;
+
+ /* Allow glob expansion in the test name */
+ if (len && find_name[len - 1] == '*')
+ len--;
+
+ if (!len)
return true;
- if (!strcmp(test_name, find_name))
+ if (!strncmp(test_name, find_name, len))
return true;
/* All tests have this prefix */
if (!strncmp(test_name, "dm_test_", 8))
test_name += 8;
- if (!strcmp(test_name, find_name))
+ if (!strncmp(test_name, find_name, len))
return true;
return false;
--
2.30.0
^ permalink raw reply related [flat|nested] 11+ messages in thread
* [PATCH v1] test: Allow simple glob pattern in the test name
2021-02-03 15:32 [PATCH v1] test: Allow simple glob pattern in the test name Andy Shevchenko
@ 2021-02-05 3:17 ` Simon Glass
2021-02-05 17:34 ` Andy Shevchenko
0 siblings, 1 reply; 11+ messages in thread
From: Simon Glass @ 2021-02-05 3:17 UTC (permalink / raw)
To: u-boot
Hi Andy,
On Wed, 3 Feb 2021 at 08:32, Andy Shevchenko
<andriy.shevchenko@linux.intel.com> wrote:
>
> When run `ut dm [test name]` allow to use simple pattern to run all tests
> started with given prefix. For example, to run all ACPI test cases:
> ut dm acpi*
>
> Signed-off-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
> ---
> test/dm/test-main.c | 12 +++++++++---
> 1 file changed, 9 insertions(+), 3 deletions(-)
This seems like a good idea to me.
Can you please send it rebased to u-boot-dm/test-working though? I
have done a bit of refactoring and this function is now in
test/test-main.c
Regards,
Simon
^ permalink raw reply [flat|nested] 11+ messages in thread
* [PATCH v1] test: Allow simple glob pattern in the test name
2021-02-05 3:17 ` Simon Glass
@ 2021-02-05 17:34 ` Andy Shevchenko
2021-02-05 18:15 ` Andy Shevchenko
0 siblings, 1 reply; 11+ messages in thread
From: Andy Shevchenko @ 2021-02-05 17:34 UTC (permalink / raw)
To: u-boot
On Thu, Feb 04, 2021 at 08:17:24PM -0700, Simon Glass wrote:
> On Wed, 3 Feb 2021 at 08:32, Andy Shevchenko
> <andriy.shevchenko@linux.intel.com> wrote:
> >
> > When run `ut dm [test name]` allow to use simple pattern to run all tests
> > started with given prefix. For example, to run all ACPI test cases:
> > ut dm acpi*
> >
> > Signed-off-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
> > ---
> > test/dm/test-main.c | 12 +++++++++---
> > 1 file changed, 9 insertions(+), 3 deletions(-)
>
> This seems like a good idea to me.
>
> Can you please send it rebased to u-boot-dm/test-working though? I
> have done a bit of refactoring and this function is now in
> test/test-main.c
I will look at it.
Btw, you have an issue there, i.e. if test case failed, all percentage after it
goes red, which is wrong.
--
With Best Regards,
Andy Shevchenko
^ permalink raw reply [flat|nested] 11+ messages in thread
* [PATCH v1] test: Allow simple glob pattern in the test name
2021-02-05 17:34 ` Andy Shevchenko
@ 2021-02-05 18:15 ` Andy Shevchenko
2021-02-05 18:18 ` Andy Shevchenko
2021-02-05 19:17 ` Andy Shevchenko
0 siblings, 2 replies; 11+ messages in thread
From: Andy Shevchenko @ 2021-02-05 18:15 UTC (permalink / raw)
To: u-boot
On Fri, Feb 05, 2021 at 07:34:49PM +0200, Andy Shevchenko wrote:
> On Thu, Feb 04, 2021 at 08:17:24PM -0700, Simon Glass wrote:
...
> Btw, you have an issue there, i.e. if test case failed, all percentage after it
> goes red, which is wrong.
One more thing, is it known bug that either in the original code, or in your
new branch the following test case is 100% failed?
/* Non-existent in DTB */
ut_asserteq(FDT_ADDR_T_NONE, dev_read_addr(dev));
Can you fix this sooner than later, please?
--
With Best Regards,
Andy Shevchenko
^ permalink raw reply [flat|nested] 11+ messages in thread
* [PATCH v1] test: Allow simple glob pattern in the test name
2021-02-05 18:15 ` Andy Shevchenko
@ 2021-02-05 18:18 ` Andy Shevchenko
2021-02-05 19:17 ` Andy Shevchenko
1 sibling, 0 replies; 11+ messages in thread
From: Andy Shevchenko @ 2021-02-05 18:18 UTC (permalink / raw)
To: u-boot
On Fri, Feb 05, 2021 at 08:15:25PM +0200, Andy Shevchenko wrote:
> On Fri, Feb 05, 2021 at 07:34:49PM +0200, Andy Shevchenko wrote:
> > On Thu, Feb 04, 2021 at 08:17:24PM -0700, Simon Glass wrote:
>
> ...
>
> > Btw, you have an issue there, i.e. if test case failed, all percentage after it
> > goes red, which is wrong.
>
> One more thing, is it known bug that either in the original code, or in your
> new branch the following test case is 100% failed?
>
> /* Non-existent in DTB */
> ut_asserteq(FDT_ADDR_T_NONE, dev_read_addr(dev));
>
> Can you fix this sooner than later, please?
Another annoying issue is the 2+ run on uncleaned build folder, i.e. on the
second and following runs it starts out of the sudden to ask my sudo password.
The later to me is a highest priority, now I have to run rm -rf build-sandbox
each time, and if I forget - wasted time!
Can you fix this as soon as possible, please?
--
With Best Regards,
Andy Shevchenko
^ permalink raw reply [flat|nested] 11+ messages in thread
* [PATCH v1] test: Allow simple glob pattern in the test name
2021-02-05 18:15 ` Andy Shevchenko
2021-02-05 18:18 ` Andy Shevchenko
@ 2021-02-05 19:17 ` Andy Shevchenko
2021-02-05 20:46 ` Andy Shevchenko
1 sibling, 1 reply; 11+ messages in thread
From: Andy Shevchenko @ 2021-02-05 19:17 UTC (permalink / raw)
To: u-boot
On Fri, Feb 05, 2021 at 08:15:25PM +0200, Andy Shevchenko wrote:
> On Fri, Feb 05, 2021 at 07:34:49PM +0200, Andy Shevchenko wrote:
> > On Thu, Feb 04, 2021 at 08:17:24PM -0700, Simon Glass wrote:
>
> ...
>
> > Btw, you have an issue there, i.e. if test case failed, all percentage after it
> > goes red, which is wrong.
>
> One more thing, is it known bug that either in the original code, or in your
> new branch the following test case is 100% failed?
>
> /* Non-existent in DTB */
> ut_asserteq(FDT_ADDR_T_NONE, dev_read_addr(dev));
>
> Can you fix this sooner than later, please?
Actually it seems this very patch makes the issue visible (I suppose something
with test case names).
--
With Best Regards,
Andy Shevchenko
^ permalink raw reply [flat|nested] 11+ messages in thread
* [PATCH v1] test: Allow simple glob pattern in the test name
2021-02-05 19:17 ` Andy Shevchenko
@ 2021-02-05 20:46 ` Andy Shevchenko
2021-02-07 14:37 ` Simon Glass
0 siblings, 1 reply; 11+ messages in thread
From: Andy Shevchenko @ 2021-02-05 20:46 UTC (permalink / raw)
To: u-boot
On Fri, Feb 05, 2021 at 09:17:27PM +0200, Andy Shevchenko wrote:
> On Fri, Feb 05, 2021 at 08:15:25PM +0200, Andy Shevchenko wrote:
> > On Fri, Feb 05, 2021 at 07:34:49PM +0200, Andy Shevchenko wrote:
> > > On Thu, Feb 04, 2021 at 08:17:24PM -0700, Simon Glass wrote:
> >
> > ...
> >
> > > Btw, you have an issue there, i.e. if test case failed, all percentage after it
> > > goes red, which is wrong.
> >
> > One more thing, is it known bug that either in the original code, or in your
> > new branch the following test case is 100% failed?
> >
> > /* Non-existent in DTB */
> > ut_asserteq(FDT_ADDR_T_NONE, dev_read_addr(dev));
> >
> > Can you fix this sooner than later, please?
>
> Actually it seems this very patch makes the issue visible (I suppose something
> with test case names).
Okay, actually there *is* a problem with the test suite, i.e. you may not run
some test cases twice during the same session.
--
With Best Regards,
Andy Shevchenko
^ permalink raw reply [flat|nested] 11+ messages in thread
* [PATCH v1] test: Allow simple glob pattern in the test name
2021-02-05 20:46 ` Andy Shevchenko
@ 2021-02-07 14:37 ` Simon Glass
2021-02-08 11:34 ` Andy Shevchenko
0 siblings, 1 reply; 11+ messages in thread
From: Simon Glass @ 2021-02-07 14:37 UTC (permalink / raw)
To: u-boot
Hi Andy,
On Fri, 5 Feb 2021 at 13:46, Andy Shevchenko
<andriy.shevchenko@linux.intel.com> wrote:
>
> On Fri, Feb 05, 2021 at 09:17:27PM +0200, Andy Shevchenko wrote:
> > On Fri, Feb 05, 2021 at 08:15:25PM +0200, Andy Shevchenko wrote:
> > > On Fri, Feb 05, 2021 at 07:34:49PM +0200, Andy Shevchenko wrote:
> > > > On Thu, Feb 04, 2021 at 08:17:24PM -0700, Simon Glass wrote:
> > >
> > > ...
> > >
> > > > Btw, you have an issue there, i.e. if test case failed, all percentage after it
> > > > goes red, which is wrong.
> > >
> > > One more thing, is it known bug that either in the original code, or in your
> > > new branch the following test case is 100% failed?
> > >
> > > /* Non-existent in DTB */
> > > ut_asserteq(FDT_ADDR_T_NONE, dev_read_addr(dev));
> > >
> > > Can you fix this sooner than later, please?
> >
> > Actually it seems this very patch makes the issue visible (I suppose something
> > with test case names).
>
> Okay, actually there *is* a problem with the test suite, i.e. you may not run
> some test cases twice during the same session.
If this is related to the squashfs tests? I have disabled them for now
in my local tree.
Regards,
Simon
^ permalink raw reply [flat|nested] 11+ messages in thread
* [PATCH v1] test: Allow simple glob pattern in the test name
2021-02-07 14:37 ` Simon Glass
@ 2021-02-08 11:34 ` Andy Shevchenko
2021-02-08 17:07 ` Simon Glass
0 siblings, 1 reply; 11+ messages in thread
From: Andy Shevchenko @ 2021-02-08 11:34 UTC (permalink / raw)
To: u-boot
On Sun, Feb 07, 2021 at 07:37:55AM -0700, Simon Glass wrote:
> On Fri, 5 Feb 2021 at 13:46, Andy Shevchenko
> <andriy.shevchenko@linux.intel.com> wrote:
> > On Fri, Feb 05, 2021 at 09:17:27PM +0200, Andy Shevchenko wrote:
> > > On Fri, Feb 05, 2021 at 08:15:25PM +0200, Andy Shevchenko wrote:
> > > > On Fri, Feb 05, 2021 at 07:34:49PM +0200, Andy Shevchenko wrote:
> > > > > On Thu, Feb 04, 2021 at 08:17:24PM -0700, Simon Glass wrote:
> > > >
> > > > ...
> > > >
> > > > > Btw, you have an issue there, i.e. if test case failed, all percentage after it
> > > > > goes red, which is wrong.
> > > >
> > > > One more thing, is it known bug that either in the original code, or in your
> > > > new branch the following test case is 100% failed?
> > > >
> > > > /* Non-existent in DTB */
> > > > ut_asserteq(FDT_ADDR_T_NONE, dev_read_addr(dev));
> > > >
> > > > Can you fix this sooner than later, please?
> > >
> > > Actually it seems this very patch makes the issue visible (I suppose something
> > > with test case names).
> >
> > Okay, actually there *is* a problem with the test suite, i.e. you may not run
> > some test cases twice during the same session.
>
> If this is related to the squashfs tests? I have disabled them for now
> in my local tree.
I don't think it's related to the certain test case, it's a test suite design issue and how some of the test cases have been written. So, if you run u-boot application and manually run `ut dm` in there,
- first time: Failures: 0
- second time: Failures: 9
- third and next time: Failures: 12
Means that 12 test cases are written badly.
--
With Best Regards,
Andy Shevchenko
^ permalink raw reply [flat|nested] 11+ messages in thread
* [PATCH v1] test: Allow simple glob pattern in the test name
2021-02-08 11:34 ` Andy Shevchenko
@ 2021-02-08 17:07 ` Simon Glass
2021-02-08 17:34 ` Andy Shevchenko
0 siblings, 1 reply; 11+ messages in thread
From: Simon Glass @ 2021-02-08 17:07 UTC (permalink / raw)
To: u-boot
Hi Andy,
On Mon, 8 Feb 2021 at 04:34, Andy Shevchenko
<andriy.shevchenko@linux.intel.com> wrote:
>
> On Sun, Feb 07, 2021 at 07:37:55AM -0700, Simon Glass wrote:
> > On Fri, 5 Feb 2021 at 13:46, Andy Shevchenko
> > <andriy.shevchenko@linux.intel.com> wrote:
> > > On Fri, Feb 05, 2021 at 09:17:27PM +0200, Andy Shevchenko wrote:
> > > > On Fri, Feb 05, 2021 at 08:15:25PM +0200, Andy Shevchenko wrote:
> > > > > On Fri, Feb 05, 2021 at 07:34:49PM +0200, Andy Shevchenko wrote:
> > > > > > On Thu, Feb 04, 2021 at 08:17:24PM -0700, Simon Glass wrote:
> > > > >
> > > > > ...
> > > > >
> > > > > > Btw, you have an issue there, i.e. if test case failed, all percentage after it
> > > > > > goes red, which is wrong.
> > > > >
> > > > > One more thing, is it known bug that either in the original code, or in your
> > > > > new branch the following test case is 100% failed?
> > > > >
> > > > > /* Non-existent in DTB */
> > > > > ut_asserteq(FDT_ADDR_T_NONE, dev_read_addr(dev));
> > > > >
> > > > > Can you fix this sooner than later, please?
> > > >
> > > > Actually it seems this very patch makes the issue visible (I suppose something
> > > > with test case names).
> > >
> > > Okay, actually there *is* a problem with the test suite, i.e. you may not run
> > > some test cases twice during the same session.
> >
> > If this is related to the squashfs tests? I have disabled them for now
> > in my local tree.
>
> I don't think it's related to the certain test case, it's a test suite design issue and how some of the test cases have been written. So, if you run u-boot application and manually run `ut dm` in there,
> - first time: Failures: 0
> - second time: Failures: 9
> - third and next time: Failures: 12
>
> Means that 12 test cases are written badly.
OK I see. It did not use to be possible to pass all tests without
running through pytest, which does some setup. You still need to
create a 'spi.bin' file for the SPI tests to pass.
I'm not sure what prevents multiple runs without quitting U-Boot, but
I suspect it is just some state hanging around, as we don't reset
everything. For example, sandbox provides control of what happens when
a sysreset is performed, and perhaps that is not reset to initial
values correct by state_reset_for_test()..
Regards,
Simon
^ permalink raw reply [flat|nested] 11+ messages in thread
* [PATCH v1] test: Allow simple glob pattern in the test name
2021-02-08 17:07 ` Simon Glass
@ 2021-02-08 17:34 ` Andy Shevchenko
0 siblings, 0 replies; 11+ messages in thread
From: Andy Shevchenko @ 2021-02-08 17:34 UTC (permalink / raw)
To: u-boot
On Mon, Feb 08, 2021 at 10:07:55AM -0700, Simon Glass wrote:
> On Mon, 8 Feb 2021 at 04:34, Andy Shevchenko
> <andriy.shevchenko@linux.intel.com> wrote:
> > On Sun, Feb 07, 2021 at 07:37:55AM -0700, Simon Glass wrote:
> > > On Fri, 5 Feb 2021 at 13:46, Andy Shevchenko
> > > <andriy.shevchenko@linux.intel.com> wrote:
> > > > On Fri, Feb 05, 2021 at 09:17:27PM +0200, Andy Shevchenko wrote:
> > > > > On Fri, Feb 05, 2021 at 08:15:25PM +0200, Andy Shevchenko wrote:
> > > > > > On Fri, Feb 05, 2021 at 07:34:49PM +0200, Andy Shevchenko wrote:
> > > > > > > On Thu, Feb 04, 2021 at 08:17:24PM -0700, Simon Glass wrote:
> > > > > >
> > > > > > ...
> > > > > >
> > > > > > > Btw, you have an issue there, i.e. if test case failed, all percentage after it
> > > > > > > goes red, which is wrong.
> > > > > >
> > > > > > One more thing, is it known bug that either in the original code, or in your
> > > > > > new branch the following test case is 100% failed?
> > > > > >
> > > > > > /* Non-existent in DTB */
> > > > > > ut_asserteq(FDT_ADDR_T_NONE, dev_read_addr(dev));
> > > > > >
> > > > > > Can you fix this sooner than later, please?
> > > > >
> > > > > Actually it seems this very patch makes the issue visible (I suppose something
> > > > > with test case names).
> > > >
> > > > Okay, actually there *is* a problem with the test suite, i.e. you may not run
> > > > some test cases twice during the same session.
> > >
> > > If this is related to the squashfs tests? I have disabled them for now
> > > in my local tree.
> >
> > I don't think it's related to the certain test case, it's a test suite design issue and how some of the test cases have been written. So, if you run u-boot application and manually run `ut dm` in there,
> > - first time: Failures: 0
> > - second time: Failures: 9
> > - third and next time: Failures: 12
> >
> > Means that 12 test cases are written badly.
>
> OK I see. It did not use to be possible to pass all tests without
> running through pytest, which does some setup. You still need to
> create a 'spi.bin' file for the SPI tests to pass.
I run it _after_ the official pytest passed. So, every file is prepared.
> I'm not sure what prevents multiple runs without quitting U-Boot, but
> I suspect it is just some state hanging around, as we don't reset
> everything. For example, sandbox provides control of what happens when
> a sysreset is performed, and perhaps that is not reset to initial
> values correct by state_reset_for_test()..
I believe this is the case in most of the cases.
--
With Best Regards,
Andy Shevchenko
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2021-02-08 17:34 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-02-03 15:32 [PATCH v1] test: Allow simple glob pattern in the test name Andy Shevchenko
2021-02-05 3:17 ` Simon Glass
2021-02-05 17:34 ` Andy Shevchenko
2021-02-05 18:15 ` Andy Shevchenko
2021-02-05 18:18 ` Andy Shevchenko
2021-02-05 19:17 ` Andy Shevchenko
2021-02-05 20:46 ` Andy Shevchenko
2021-02-07 14:37 ` Simon Glass
2021-02-08 11:34 ` Andy Shevchenko
2021-02-08 17:07 ` Simon Glass
2021-02-08 17:34 ` Andy Shevchenko
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.