From: David Gow <davidgow@google.com>
To: Brendan Higgins <brendanhiggins@google.com>,
Arpitha Raghunandan <98.arpi@gmail.com>,
Marco Elver <elver@google.com>,
Shuah Khan <skhan@linuxfoundation.org>
Cc: David Gow <davidgow@google.com>,
kunit-dev@googlegroups.com, linux-kselftest@vger.kernel.org,
linux-kernel@vger.kernel.org
Subject: [PATCH] kunit: Fix result propagation for parameterised tests
Date: Thu, 10 Jun 2021 20:57:25 -0700 [thread overview]
Message-ID: <20210611035725.1248874-1-davidgow@google.com> (raw)
When one parameter of a parameterised test failed, its failure would be
propagated to the overall test, but not to the suite result (unless it
was the last parameter).
This is because test_case->success was being reset to the test->success
result after each parameter was used, so a failing test's result would
be overwritten by a non-failing result. The overall test result was
handled in a third variable, test_result, but this was disacarded after
the status line was printed.
Instead, just propagate the result after each parameter run.
Signed-off-by: David Gow <davidgow@google.com>
Fixes: fadb08e7c750 ("kunit: Support for Parameterized Testing")
---
This is fixing quite a serious bug where some test suites would appear
to succeed even if some of their component tests failed. It'd be nice to
get this into kunit-fixes ASAP.
(This will require a rework of some of the skip tests work, for which
I'll send out a new version soon.)
Cheers,
-- David
lib/kunit/test.c | 7 +++----
1 file changed, 3 insertions(+), 4 deletions(-)
diff --git a/lib/kunit/test.c b/lib/kunit/test.c
index 2f6cc0123232..17973a4a44c2 100644
--- a/lib/kunit/test.c
+++ b/lib/kunit/test.c
@@ -376,7 +376,7 @@ static void kunit_run_case_catch_errors(struct kunit_suite *suite,
context.test_case = test_case;
kunit_try_catch_run(try_catch, &context);
- test_case->success = test->success;
+ test_case->success &= test->success;
}
int kunit_run_tests(struct kunit_suite *suite)
@@ -388,7 +388,7 @@ int kunit_run_tests(struct kunit_suite *suite)
kunit_suite_for_each_test_case(suite, test_case) {
struct kunit test = { .param_value = NULL, .param_index = 0 };
- bool test_success = true;
+ test_case->success = true;
if (test_case->generate_params) {
/* Get initial param. */
@@ -398,7 +398,6 @@ int kunit_run_tests(struct kunit_suite *suite)
do {
kunit_run_case_catch_errors(suite, test_case, &test);
- test_success &= test_case->success;
if (test_case->generate_params) {
if (param_desc[0] == '\0') {
@@ -420,7 +419,7 @@ int kunit_run_tests(struct kunit_suite *suite)
}
} while (test.param_value);
- kunit_print_ok_not_ok(&test, true, test_success,
+ kunit_print_ok_not_ok(&test, true, test_case->success,
kunit_test_case_num(suite, test_case),
test_case->name);
}
--
2.32.0.272.g935e593368-goog
next reply other threads:[~2021-06-11 3:57 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-06-11 3:57 David Gow [this message]
2021-06-11 8:29 ` [PATCH] kunit: Fix result propagation for parameterised tests Marco Elver
2021-06-11 17:44 ` Shuah Khan
2021-06-11 23:16 ` David Gow
2021-06-11 20:26 ` Brendan Higgins
2021-06-11 23:14 ` David Gow
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210611035725.1248874-1-davidgow@google.com \
--to=davidgow@google.com \
--cc=98.arpi@gmail.com \
--cc=brendanhiggins@google.com \
--cc=elver@google.com \
--cc=kunit-dev@googlegroups.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-kselftest@vger.kernel.org \
--cc=skhan@linuxfoundation.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).