* [PATCH] kunit: tool: fix extra trailing \n in parsed test output
@ 2020-10-20 23:32 Daniel Latypov
2020-10-30 2:34 ` David Gow
0 siblings, 1 reply; 5+ messages in thread
From: Daniel Latypov @ 2020-10-20 23:32 UTC (permalink / raw)
To: brendanhiggins
Cc: davidgow, linux-kernel, linux-kselftest, skhan, Daniel Latypov
For simplcity, strip all trailing whitespace from parsed output.
I imagine no one is printing out meaningful trailing whitespace via
KUNIT_FAIL() or similar, and that if they are, they really shouldn't.
At some point, the lines from `isolate_kunit_output()` started having
trailing \n, which results in artifacty output like this:
$ ./tools/testing/kunit/kunit.py run
[16:16:46] [FAILED] example_simple_test
[16:16:46] # example_simple_test: EXPECTATION FAILED at lib/kunit/kunit-example-test.c:29
[16:16:46] Expected 1 + 1 == 3, but
[16:16:46] 1 + 1 == 2
[16:16:46] 3 == 3
[16:16:46] not ok 1 - example_simple_test
[16:16:46]
After this change:
[16:16:46] # example_simple_test: EXPECTATION FAILED at lib/kunit/kunit-example-test.c:29
[16:16:46] Expected 1 + 1 == 3, but
[16:16:46] 1 + 1 == 2
[16:16:46] 3 == 3
[16:16:46] not ok 1 - example_simple_test
[16:16:46]
Signed-off-by: Daniel Latypov <dlatypov@google.com>
---
tools/testing/kunit/kunit_parser.py | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/tools/testing/kunit/kunit_parser.py b/tools/testing/kunit/kunit_parser.py
index 8019e3dd4c32..e68b1c66a73f 100644
--- a/tools/testing/kunit/kunit_parser.py
+++ b/tools/testing/kunit/kunit_parser.py
@@ -342,7 +342,8 @@ def parse_run_tests(kernel_output) -> TestResult:
total_tests = 0
failed_tests = 0
crashed_tests = 0
- test_result = parse_test_result(list(isolate_kunit_output(kernel_output)))
+ test_result = parse_test_result(list(
+ l.rstrip() for l in isolate_kunit_output(kernel_output)))
if test_result.status == TestStatus.NO_TESTS:
print(red('[ERROR] ') + yellow('no tests run!'))
elif test_result.status == TestStatus.FAILURE_TO_PARSE_TESTS:
base-commit: c4d6fe7311762f2e03b3c27ad38df7c40c80cc93
--
2.29.0.rc1.297.gfa9743e501-goog
^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: [PATCH] kunit: tool: fix extra trailing \n in parsed test output
2020-10-20 23:32 [PATCH] kunit: tool: fix extra trailing \n in parsed test output Daniel Latypov
@ 2020-10-30 2:34 ` David Gow
2020-10-30 5:40 ` Daniel Latypov
0 siblings, 1 reply; 5+ messages in thread
From: David Gow @ 2020-10-30 2:34 UTC (permalink / raw)
To: Daniel Latypov
Cc: Brendan Higgins, Linux Kernel Mailing List,
open list:KERNEL SELFTEST FRAMEWORK, Shuah Khan
On Wed, Oct 21, 2020 at 7:32 AM Daniel Latypov <dlatypov@google.com> wrote:
>
> For simplcity, strip all trailing whitespace from parsed output.
> I imagine no one is printing out meaningful trailing whitespace via
> KUNIT_FAIL() or similar, and that if they are, they really shouldn't.
>
> At some point, the lines from `isolate_kunit_output()` started having
> trailing \n, which results in artifacty output like this:
>
> $ ./tools/testing/kunit/kunit.py run
> [16:16:46] [FAILED] example_simple_test
> [16:16:46] # example_simple_test: EXPECTATION FAILED at lib/kunit/kunit-example-test.c:29
>
> [16:16:46] Expected 1 + 1 == 3, but
>
> [16:16:46] 1 + 1 == 2
>
> [16:16:46] 3 == 3
>
> [16:16:46] not ok 1 - example_simple_test
>
> [16:16:46]
>
> After this change:
> [16:16:46] # example_simple_test: EXPECTATION FAILED at lib/kunit/kunit-example-test.c:29
> [16:16:46] Expected 1 + 1 == 3, but
> [16:16:46] 1 + 1 == 2
> [16:16:46] 3 == 3
> [16:16:46] not ok 1 - example_simple_test
> [16:16:46]
>
> Signed-off-by: Daniel Latypov <dlatypov@google.com>
> ---
Thanks! This is a long-overdue fix, and it worked well for me.
Tested-by: David Gow <davidgow@google.com>
One comment below:
> tools/testing/kunit/kunit_parser.py | 3 ++-
> 1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/tools/testing/kunit/kunit_parser.py b/tools/testing/kunit/kunit_parser.py
> index 8019e3dd4c32..e68b1c66a73f 100644
> --- a/tools/testing/kunit/kunit_parser.py
> +++ b/tools/testing/kunit/kunit_parser.py
> @@ -342,7 +342,8 @@ def parse_run_tests(kernel_output) -> TestResult:
> total_tests = 0
> failed_tests = 0
> crashed_tests = 0
> - test_result = parse_test_result(list(isolate_kunit_output(kernel_output)))
> + test_result = parse_test_result(list(
> + l.rstrip() for l in isolate_kunit_output(kernel_output)))
Could we do this inside isolate_kunit_output() instead? That seems
like it'd be a more logical place for it (removing the newline is a
sort of isolating the output), and it'd avoid making this line quite
as horrifyingly nested.
> if test_result.status == TestStatus.NO_TESTS:
> print(red('[ERROR] ') + yellow('no tests run!'))
> elif test_result.status == TestStatus.FAILURE_TO_PARSE_TESTS:
>
> base-commit: c4d6fe7311762f2e03b3c27ad38df7c40c80cc93
> --
> 2.29.0.rc1.297.gfa9743e501-goog
>
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH] kunit: tool: fix extra trailing \n in parsed test output
2020-10-30 2:34 ` David Gow
@ 2020-10-30 5:40 ` Daniel Latypov
2020-10-30 7:22 ` David Gow
0 siblings, 1 reply; 5+ messages in thread
From: Daniel Latypov @ 2020-10-30 5:40 UTC (permalink / raw)
To: David Gow
Cc: Brendan Higgins, Linux Kernel Mailing List,
open list:KERNEL SELFTEST FRAMEWORK, Shuah Khan
On Thu, Oct 29, 2020 at 7:34 PM David Gow <davidgow@google.com> wrote:
>
> On Wed, Oct 21, 2020 at 7:32 AM Daniel Latypov <dlatypov@google.com> wrote:
> >
> > For simplcity, strip all trailing whitespace from parsed output.
> > I imagine no one is printing out meaningful trailing whitespace via
> > KUNIT_FAIL() or similar, and that if they are, they really shouldn't.
> >
> > At some point, the lines from `isolate_kunit_output()` started having
> > trailing \n, which results in artifacty output like this:
> >
> > $ ./tools/testing/kunit/kunit.py run
> > [16:16:46] [FAILED] example_simple_test
> > [16:16:46] # example_simple_test: EXPECTATION FAILED at lib/kunit/kunit-example-test.c:29
> >
> > [16:16:46] Expected 1 + 1 == 3, but
> >
> > [16:16:46] 1 + 1 == 2
> >
> > [16:16:46] 3 == 3
> >
> > [16:16:46] not ok 1 - example_simple_test
> >
> > [16:16:46]
> >
> > After this change:
> > [16:16:46] # example_simple_test: EXPECTATION FAILED at lib/kunit/kunit-example-test.c:29
> > [16:16:46] Expected 1 + 1 == 3, but
> > [16:16:46] 1 + 1 == 2
> > [16:16:46] 3 == 3
> > [16:16:46] not ok 1 - example_simple_test
> > [16:16:46]
> >
> > Signed-off-by: Daniel Latypov <dlatypov@google.com>
> > ---
>
> Thanks! This is a long-overdue fix, and it worked well for me.
>
> Tested-by: David Gow <davidgow@google.com>
>
> One comment below:
>
> > tools/testing/kunit/kunit_parser.py | 3 ++-
> > 1 file changed, 2 insertions(+), 1 deletion(-)
> >
> > diff --git a/tools/testing/kunit/kunit_parser.py b/tools/testing/kunit/kunit_parser.py
> > index 8019e3dd4c32..e68b1c66a73f 100644
> > --- a/tools/testing/kunit/kunit_parser.py
> > +++ b/tools/testing/kunit/kunit_parser.py
> > @@ -342,7 +342,8 @@ def parse_run_tests(kernel_output) -> TestResult:
> > total_tests = 0
> > failed_tests = 0
> > crashed_tests = 0
> > - test_result = parse_test_result(list(isolate_kunit_output(kernel_output)))
> > + test_result = parse_test_result(list(
> > + l.rstrip() for l in isolate_kunit_output(kernel_output)))
>
> Could we do this inside isolate_kunit_output() instead? That seems
> like it'd be a more logical place for it (removing the newline is a
> sort of isolating the output), and it'd avoid making this line quite
> as horrifyingly nested.
Good point.
We could either do it on each yield (messy), or before, i.e.
diff --git a/tools/testing/kunit/kunit_parser.py
b/tools/testing/kunit/kunit_parser.py
index 8019e3dd4c32..14d35deb96cd 100644
--- a/tools/testing/kunit/kunit_parser.py
+++ b/tools/testing/kunit/kunit_parser.py
@@ -54,6 +54,7 @@ kunit_end_re = re.compile('(List of all partitions:|'
def isolate_kunit_output(kernel_output):
started = False
for line in kernel_output:
+ line = line.rstrip() # line always has a trailing \n
if kunit_start_re.search(line):
prefix_len = len(line.split('TAP version')[0])
started = True
I had some vague concerns about this as
kunit_start_re = re.compile(r'TAP version [0-9]+$')
has that anchor at the end.
This could ostensibly make it match more things than before.
Since I'm using rstrip() out of laziness, that means strings like
'<prefix we allow for some reason>TAP version 42\t\n'
will now also match.
I don't really think that's an issue, but I'd sent this as a more
conservative change initially.
I can send the diff above as a replacement for this patch.
>
> > if test_result.status == TestStatus.NO_TESTS:
> > print(red('[ERROR] ') + yellow('no tests run!'))
> > elif test_result.status == TestStatus.FAILURE_TO_PARSE_TESTS:
> >
> > base-commit: c4d6fe7311762f2e03b3c27ad38df7c40c80cc93
> > --
> > 2.29.0.rc1.297.gfa9743e501-goog
> >
^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: [PATCH] kunit: tool: fix extra trailing \n in parsed test output
2020-10-30 5:40 ` Daniel Latypov
@ 2020-10-30 7:22 ` David Gow
2020-10-30 22:40 ` Daniel Latypov
0 siblings, 1 reply; 5+ messages in thread
From: David Gow @ 2020-10-30 7:22 UTC (permalink / raw)
To: Daniel Latypov
Cc: Brendan Higgins, Linux Kernel Mailing List,
open list:KERNEL SELFTEST FRAMEWORK, Shuah Khan
On Fri, Oct 30, 2020 at 1:41 PM Daniel Latypov <dlatypov@google.com> wrote:
>
> On Thu, Oct 29, 2020 at 7:34 PM David Gow <davidgow@google.com> wrote:
> >
> > On Wed, Oct 21, 2020 at 7:32 AM Daniel Latypov <dlatypov@google.com> wrote:
> > >
> > > For simplcity, strip all trailing whitespace from parsed output.
> > > I imagine no one is printing out meaningful trailing whitespace via
> > > KUNIT_FAIL() or similar, and that if they are, they really shouldn't.
> > >
> > > At some point, the lines from `isolate_kunit_output()` started having
> > > trailing \n, which results in artifacty output like this:
> > >
> > > $ ./tools/testing/kunit/kunit.py run
> > > [16:16:46] [FAILED] example_simple_test
> > > [16:16:46] # example_simple_test: EXPECTATION FAILED at lib/kunit/kunit-example-test.c:29
> > >
> > > [16:16:46] Expected 1 + 1 == 3, but
> > >
> > > [16:16:46] 1 + 1 == 2
> > >
> > > [16:16:46] 3 == 3
> > >
> > > [16:16:46] not ok 1 - example_simple_test
> > >
> > > [16:16:46]
> > >
> > > After this change:
> > > [16:16:46] # example_simple_test: EXPECTATION FAILED at lib/kunit/kunit-example-test.c:29
> > > [16:16:46] Expected 1 + 1 == 3, but
> > > [16:16:46] 1 + 1 == 2
> > > [16:16:46] 3 == 3
> > > [16:16:46] not ok 1 - example_simple_test
> > > [16:16:46]
> > >
> > > Signed-off-by: Daniel Latypov <dlatypov@google.com>
> > > ---
> >
> > Thanks! This is a long-overdue fix, and it worked well for me.
> >
> > Tested-by: David Gow <davidgow@google.com>
> >
> > One comment below:
> >
> > > tools/testing/kunit/kunit_parser.py | 3 ++-
> > > 1 file changed, 2 insertions(+), 1 deletion(-)
> > >
> > > diff --git a/tools/testing/kunit/kunit_parser.py b/tools/testing/kunit/kunit_parser.py
> > > index 8019e3dd4c32..e68b1c66a73f 100644
> > > --- a/tools/testing/kunit/kunit_parser.py
> > > +++ b/tools/testing/kunit/kunit_parser.py
> > > @@ -342,7 +342,8 @@ def parse_run_tests(kernel_output) -> TestResult:
> > > total_tests = 0
> > > failed_tests = 0
> > > crashed_tests = 0
> > > - test_result = parse_test_result(list(isolate_kunit_output(kernel_output)))
> > > + test_result = parse_test_result(list(
> > > + l.rstrip() for l in isolate_kunit_output(kernel_output)))
> >
> > Could we do this inside isolate_kunit_output() instead? That seems
> > like it'd be a more logical place for it (removing the newline is a
> > sort of isolating the output), and it'd avoid making this line quite
> > as horrifyingly nested.
>
> Good point.
> We could either do it on each yield (messy), or before, i.e.
>
> diff --git a/tools/testing/kunit/kunit_parser.py
> b/tools/testing/kunit/kunit_parser.py
> index 8019e3dd4c32..14d35deb96cd 100644
> --- a/tools/testing/kunit/kunit_parser.py
> +++ b/tools/testing/kunit/kunit_parser.py
> @@ -54,6 +54,7 @@ kunit_end_re = re.compile('(List of all partitions:|'
> def isolate_kunit_output(kernel_output):
> started = False
> for line in kernel_output:
> + line = line.rstrip() # line always has a trailing \n
> if kunit_start_re.search(line):
> prefix_len = len(line.split('TAP version')[0])
> started = True
>
> I had some vague concerns about this as
> kunit_start_re = re.compile(r'TAP version [0-9]+$')
> has that anchor at the end.
>
> This could ostensibly make it match more things than before.
> Since I'm using rstrip() out of laziness, that means strings like
> '<prefix we allow for some reason>TAP version 42\t\n'
> will now also match.
>
> I don't really think that's an issue, but I'd sent this as a more
> conservative change initially.
> I can send the diff above as a replacement for this patch.
I prefer this if it works. From my cursory testing, it does (though
the kunt_tool_tests.py tests will need updating). At the very least,
I'm able to get it to work with --alltests / allyesconfig (with a few
options tactically disabled), which was the main reason we needed
isolate_kunit_output() in the first place.
So, unless anyone can find a real-world case where this breaks
something, let's go with this.
Cheers,
-- David
>
> >
> > > if test_result.status == TestStatus.NO_TESTS:
> > > print(red('[ERROR] ') + yellow('no tests run!'))
> > > elif test_result.status == TestStatus.FAILURE_TO_PARSE_TESTS:
> > >
> > > base-commit: c4d6fe7311762f2e03b3c27ad38df7c40c80cc93
> > > --
> > > 2.29.0.rc1.297.gfa9743e501-goog
> > >
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH] kunit: tool: fix extra trailing \n in parsed test output
2020-10-30 7:22 ` David Gow
@ 2020-10-30 22:40 ` Daniel Latypov
0 siblings, 0 replies; 5+ messages in thread
From: Daniel Latypov @ 2020-10-30 22:40 UTC (permalink / raw)
To: David Gow
Cc: Brendan Higgins, Linux Kernel Mailing List,
open list:KERNEL SELFTEST FRAMEWORK, Shuah Khan
On Fri, Oct 30, 2020 at 12:22 AM David Gow <davidgow@google.com> wrote:
>
> On Fri, Oct 30, 2020 at 1:41 PM Daniel Latypov <dlatypov@google.com> wrote:
> >
> > On Thu, Oct 29, 2020 at 7:34 PM David Gow <davidgow@google.com> wrote:
> > >
> > > On Wed, Oct 21, 2020 at 7:32 AM Daniel Latypov <dlatypov@google.com> wrote:
> > > >
> > > > For simplcity, strip all trailing whitespace from parsed output.
> > > > I imagine no one is printing out meaningful trailing whitespace via
> > > > KUNIT_FAIL() or similar, and that if they are, they really shouldn't.
> > > >
> > > > At some point, the lines from `isolate_kunit_output()` started having
> > > > trailing \n, which results in artifacty output like this:
> > > >
> > > > $ ./tools/testing/kunit/kunit.py run
> > > > [16:16:46] [FAILED] example_simple_test
> > > > [16:16:46] # example_simple_test: EXPECTATION FAILED at lib/kunit/kunit-example-test.c:29
> > > >
> > > > [16:16:46] Expected 1 + 1 == 3, but
> > > >
> > > > [16:16:46] 1 + 1 == 2
> > > >
> > > > [16:16:46] 3 == 3
> > > >
> > > > [16:16:46] not ok 1 - example_simple_test
> > > >
> > > > [16:16:46]
> > > >
> > > > After this change:
> > > > [16:16:46] # example_simple_test: EXPECTATION FAILED at lib/kunit/kunit-example-test.c:29
> > > > [16:16:46] Expected 1 + 1 == 3, but
> > > > [16:16:46] 1 + 1 == 2
> > > > [16:16:46] 3 == 3
> > > > [16:16:46] not ok 1 - example_simple_test
> > > > [16:16:46]
> > > >
> > > > Signed-off-by: Daniel Latypov <dlatypov@google.com>
> > > > ---
> > >
> > > Thanks! This is a long-overdue fix, and it worked well for me.
> > >
> > > Tested-by: David Gow <davidgow@google.com>
> > >
> > > One comment below:
> > >
> > > > tools/testing/kunit/kunit_parser.py | 3 ++-
> > > > 1 file changed, 2 insertions(+), 1 deletion(-)
> > > >
> > > > diff --git a/tools/testing/kunit/kunit_parser.py b/tools/testing/kunit/kunit_parser.py
> > > > index 8019e3dd4c32..e68b1c66a73f 100644
> > > > --- a/tools/testing/kunit/kunit_parser.py
> > > > +++ b/tools/testing/kunit/kunit_parser.py
> > > > @@ -342,7 +342,8 @@ def parse_run_tests(kernel_output) -> TestResult:
> > > > total_tests = 0
> > > > failed_tests = 0
> > > > crashed_tests = 0
> > > > - test_result = parse_test_result(list(isolate_kunit_output(kernel_output)))
> > > > + test_result = parse_test_result(list(
> > > > + l.rstrip() for l in isolate_kunit_output(kernel_output)))
> > >
> > > Could we do this inside isolate_kunit_output() instead? That seems
> > > like it'd be a more logical place for it (removing the newline is a
> > > sort of isolating the output), and it'd avoid making this line quite
> > > as horrifyingly nested.
> >
> > Good point.
> > We could either do it on each yield (messy), or before, i.e.
> >
> > diff --git a/tools/testing/kunit/kunit_parser.py
> > b/tools/testing/kunit/kunit_parser.py
> > index 8019e3dd4c32..14d35deb96cd 100644
> > --- a/tools/testing/kunit/kunit_parser.py
> > +++ b/tools/testing/kunit/kunit_parser.py
> > @@ -54,6 +54,7 @@ kunit_end_re = re.compile('(List of all partitions:|'
> > def isolate_kunit_output(kernel_output):
> > started = False
> > for line in kernel_output:
> > + line = line.rstrip() # line always has a trailing \n
> > if kunit_start_re.search(line):
> > prefix_len = len(line.split('TAP version')[0])
> > started = True
> >
> > I had some vague concerns about this as
> > kunit_start_re = re.compile(r'TAP version [0-9]+$')
> > has that anchor at the end.
> >
> > This could ostensibly make it match more things than before.
> > Since I'm using rstrip() out of laziness, that means strings like
> > '<prefix we allow for some reason>TAP version 42\t\n'
> > will now also match.
> >
> > I don't really think that's an issue, but I'd sent this as a more
> > conservative change initially.
> > I can send the diff above as a replacement for this patch.
>
> I prefer this if it works. From my cursory testing, it does (though
> the kunt_tool_tests.py tests will need updating). At the very least,
> I'm able to get it to work with --alltests / allyesconfig (with a few
> options tactically disabled), which was the main reason we needed
> isolate_kunit_output() in the first place.
>
> So, unless anyone can find a real-world case where this breaks
> something, let's go with this.
Sounds good.
Sent https://lore.kernel.org/linux-kselftest/20201030223853.554597-1-dlatypov@google.com
>
> Cheers,
> -- David
>
> >
> > >
> > > > if test_result.status == TestStatus.NO_TESTS:
> > > > print(red('[ERROR] ') + yellow('no tests run!'))
> > > > elif test_result.status == TestStatus.FAILURE_TO_PARSE_TESTS:
> > > >
> > > > base-commit: c4d6fe7311762f2e03b3c27ad38df7c40c80cc93
> > > > --
> > > > 2.29.0.rc1.297.gfa9743e501-goog
> > > >
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2020-10-30 22:40 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-10-20 23:32 [PATCH] kunit: tool: fix extra trailing \n in parsed test output Daniel Latypov
2020-10-30 2:34 ` David Gow
2020-10-30 5:40 ` Daniel Latypov
2020-10-30 7:22 ` David Gow
2020-10-30 22:40 ` Daniel Latypov
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).