[PATCH v4 0/2] kunit: fail tests on UBSAN errors

[PATCH v4 0/2] kunit: fail tests on UBSAN errors

[Daniel Latypov]

v1 by Uriel is here: [1].
Since it's been a while, I've dropped the Reviewed-By's.

It depended on commit 83c4e7a0363b ("KUnit: KASAN Integration") which
hadn't been merged yet, so that caused some kerfuffle with applying them
previously and the series was reverted.

This revives the series but makes the kunit_fail_current_test() function
take a format string and logs the file and line number of the failing
code, addressing Alan Maguire's comments on the previous version.

As a result, the patch that makes UBSAN errors was tweaked slightly to
include an error message.

v2 -> v3:
  Try and fail to make kunit_fail_current_test() work on CONFIG_KUNIT=m
  s/_/__ on the helper func to match others in test.c
v3 -> v4:
  Revert to only enabling kunit_fail_current_test() for CONFIG_KUNIT=y

[1] https://lore.kernel.org/linux-kselftest/20200806174326.3577537-1-urielguajardojr@gmail.com/

Uriel Guajardo (2):
  kunit: support failure from dynamic analysis tools
  kunit: ubsan integration

 include/kunit/test-bug.h | 30 ++++++++++++++++++++++++++++++
 lib/kunit/test.c         | 39 +++++++++++++++++++++++++++++++++++----
 lib/ubsan.c              |  3 +++
 3 files changed, 68 insertions(+), 4 deletions(-)
 create mode 100644 include/kunit/test-bug.h


base-commit: a74e6a014c9d4d4161061f770c9b4f98372ac778
-- 
2.31.0.rc2.261.g7f71774620-goog

[PATCH v4 1/2] kunit: support failure from dynamic analysis tools

[Daniel Latypov]

From: Uriel Guajardo <urielguajardo@google.com>

Add a kunit_fail_current_test() function to fail the currently running
test, if any, with an error message.

This is largely intended for dynamic analysis tools like UBSAN and for
fakes.
E.g. say I had a fake ops struct for testing and I wanted my `free`
function to complain if it was called with an invalid argument, or
caught a double-free. Most return void and have no normal means of
signalling failure (e.g. super_operations, iommu_ops, etc.).

Key points:
* Always update current->kunit_test so anyone can use it.
  * commit 83c4e7a0363b ("KUnit: KASAN Integration") only updated it for
  CONFIG_KASAN=y

* Create a new header <kunit/test-bug.h> so non-test code doesn't have
to include all of <kunit/test.h> (e.g. lib/ubsan.c)

* Forward the file and line number to make it easier to track down
failures

* Declare the helper function for nice __printf() warnings about mismatched
format strings even when KUnit is not enabled.

Example output from kunit_fail_current_test("message"):
[15:19:34] [FAILED] example_simple_test
[15:19:34]     # example_simple_test: initializing
[15:19:34]     # example_simple_test: lib/kunit/kunit-example-test.c:24: message
[15:19:34]     not ok 1 - example_simple_test

Co-developed-by: Daniel Latypov <dlatypov@google.com>
Signed-off-by: Daniel Latypov <dlatypov@google.com>
Signed-off-by: Uriel Guajardo <urielguajardo@google.com>
Reviewed-by: Alan Maguire <alan.maguire@oracle.com>
---
 include/kunit/test-bug.h | 30 ++++++++++++++++++++++++++++++
 lib/kunit/test.c         | 39 +++++++++++++++++++++++++++++++++++----
 2 files changed, 65 insertions(+), 4 deletions(-)
 create mode 100644 include/kunit/test-bug.h

diff --git a/include/kunit/test-bug.h b/include/kunit/test-bug.h
new file mode 100644
index 000000000000..e88b74a4fd85
--- /dev/null
+++ b/include/kunit/test-bug.h
@@ -0,0 +1,30 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * KUnit API allowing dynamic analysis tools to interact with KUnit tests
+ *
+ * Copyright (C) 2020, Google LLC.
+ * Author: Uriel Guajardo <urielguajardo@google.com>
+ */
+
+#ifndef _KUNIT_TEST_BUG_H
+#define _KUNIT_TEST_BUG_H
+
+#define kunit_fail_current_test(fmt, ...) \
+	__kunit_fail_current_test(__FILE__, __LINE__, fmt, ##__VA_ARGS__)
+
+#if IS_BUILTIN(CONFIG_KUNIT)
+
+extern __printf(3, 4) void __kunit_fail_current_test(const char *file, int line,
+						    const char *fmt, ...);
+
+#else
+
+static __printf(3, 4) void __kunit_fail_current_test(const char *file, int line,
+						    const char *fmt, ...)
+{
+}
+
+#endif
+
+
+#endif /* _KUNIT_TEST_BUG_H */
diff --git a/lib/kunit/test.c b/lib/kunit/test.c
index ec9494e914ef..2f6cc0123232 100644
--- a/lib/kunit/test.c
+++ b/lib/kunit/test.c
@@ -7,6 +7,7 @@
  */
 
 #include <kunit/test.h>
+#include <kunit/test-bug.h>
 #include <linux/kernel.h>
 #include <linux/kref.h>
 #include <linux/sched/debug.h>
@@ -16,6 +17,40 @@
 #include "string-stream.h"
 #include "try-catch-impl.h"
 
+#if IS_BUILTIN(CONFIG_KUNIT)
+/*
+ * Fail the current test and print an error message to the log.
+ */
+void __kunit_fail_current_test(const char *file, int line, const char *fmt, ...)
+{
+	va_list args;
+	int len;
+	char *buffer;
+
+	if (!current->kunit_test)
+		return;
+
+	kunit_set_failure(current->kunit_test);
+
+	/* kunit_err() only accepts literals, so evaluate the args first. */
+	va_start(args, fmt);
+	len = vsnprintf(NULL, 0, fmt, args) + 1;
+	va_end(args);
+
+	buffer = kunit_kmalloc(current->kunit_test, len, GFP_KERNEL);
+	if (!buffer)
+		return;
+
+	va_start(args, fmt);
+	vsnprintf(buffer, len, fmt, args);
+	va_end(args);
+
+	kunit_err(current->kunit_test, "%s:%d: %s", file, line, buffer);
+	kunit_kfree(current->kunit_test, buffer);
+}
+EXPORT_SYMBOL_GPL(__kunit_fail_current_test);
+#endif
+
 /*
  * Append formatted message to log, size of which is limited to
  * KUNIT_LOG_SIZE bytes (including null terminating byte).
@@ -273,9 +308,7 @@ static void kunit_try_run_case(void *data)
 	struct kunit_suite *suite = ctx->suite;
 	struct kunit_case *test_case = ctx->test_case;
 
-#if (IS_ENABLED(CONFIG_KASAN) && IS_ENABLED(CONFIG_KUNIT))
 	current->kunit_test = test;
-#endif /* IS_ENABLED(CONFIG_KASAN) && IS_ENABLED(CONFIG_KUNIT) */
 
 	/*
 	 * kunit_run_case_internal may encounter a fatal error; if it does,
@@ -624,9 +657,7 @@ void kunit_cleanup(struct kunit *test)
 		spin_unlock(&test->lock);
 		kunit_remove_resource(test, res);
 	}
-#if (IS_ENABLED(CONFIG_KASAN) && IS_ENABLED(CONFIG_KUNIT))
 	current->kunit_test = NULL;
-#endif /* IS_ENABLED(CONFIG_KASAN) && IS_ENABLED(CONFIG_KUNIT)*/
 }
 EXPORT_SYMBOL_GPL(kunit_cleanup);
 
-- 
2.31.0.rc2.261.g7f71774620-goog

[PATCH v4 2/2] kunit: ubsan integration

[Daniel Latypov]

From: Uriel Guajardo <urielguajardo@google.com>

Integrates UBSAN into the KUnit testing framework. It fails KUnit tests
whenever it reports undefined behavior.

When CONFIG_KUNIT=n, nothing is printed or even formatted, so this has
no behavioral impact outside of tests.

kunit_fail_current_test() effectively does a pr_err() as well, so
there's some slight duplication, but it also ensures an error is
recorded in the debugfs entry for the running KUnit test.

Print a shorter version of the message to make it less spammy.

Co-developed-by: Daniel Latypov <dlatypov@google.com>
Signed-off-by: Daniel Latypov <dlatypov@google.com>
Signed-off-by: Uriel Guajardo <urielguajardo@google.com>
Reviewed-by: Alan Maguire <alan.maguire@oracle.com>
---
 lib/ubsan.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/lib/ubsan.c b/lib/ubsan.c
index 26229973049d..bdc380ff5d5c 100644
--- a/lib/ubsan.c
+++ b/lib/ubsan.c
@@ -14,6 +14,7 @@
 #include <linux/types.h>
 #include <linux/sched.h>
 #include <linux/uaccess.h>
+#include <kunit/test-bug.h>
 
 #include "ubsan.h"
 
@@ -141,6 +142,8 @@ static void ubsan_prologue(struct source_location *loc, const char *reason)
 		"========================================\n");
 	pr_err("UBSAN: %s in %s:%d:%d\n", reason, loc->file_name,
 		loc->line & LINE_MASK, loc->column & COLUMN_MASK);
+
+	kunit_fail_current_test("%s in %s", reason, loc->file_name);
 }
 
 static void ubsan_epilogue(void)
-- 
2.31.0.rc2.261.g7f71774620-goog

Re: [PATCH v4 1/2] kunit: support failure from dynamic analysis tools

[Brendan Higgins]

On Thu, Mar 11, 2021 at 7:23 AM Daniel Latypov <dlatypov@google.com> wrote:
>
> From: Uriel Guajardo <urielguajardo@google.com>
>
> Add a kunit_fail_current_test() function to fail the currently running
> test, if any, with an error message.
>
> This is largely intended for dynamic analysis tools like UBSAN and for
> fakes.
> E.g. say I had a fake ops struct for testing and I wanted my `free`
> function to complain if it was called with an invalid argument, or
> caught a double-free. Most return void and have no normal means of
> signalling failure (e.g. super_operations, iommu_ops, etc.).
>
> Key points:
> * Always update current->kunit_test so anyone can use it.
>   * commit 83c4e7a0363b ("KUnit: KASAN Integration") only updated it for
>   CONFIG_KASAN=y
>
> * Create a new header <kunit/test-bug.h> so non-test code doesn't have
> to include all of <kunit/test.h> (e.g. lib/ubsan.c)
>
> * Forward the file and line number to make it easier to track down
> failures
>
> * Declare the helper function for nice __printf() warnings about mismatched
> format strings even when KUnit is not enabled.
>
> Example output from kunit_fail_current_test("message"):
> [15:19:34] [FAILED] example_simple_test
> [15:19:34]     # example_simple_test: initializing
> [15:19:34]     # example_simple_test: lib/kunit/kunit-example-test.c:24: message
> [15:19:34]     not ok 1 - example_simple_test
>
> Co-developed-by: Daniel Latypov <dlatypov@google.com>
> Signed-off-by: Daniel Latypov <dlatypov@google.com>
> Signed-off-by: Uriel Guajardo <urielguajardo@google.com>
> Reviewed-by: Alan Maguire <alan.maguire@oracle.com>

Reviewed-by: Brendan Higgins <brendanhiggins@google.com>

Re: [PATCH v4 1/2] kunit: support failure from dynamic analysis tools

[Shuah Khan]

On 4/2/21 2:55 AM, Brendan Higgins wrote:
> On Thu, Mar 11, 2021 at 7:23 AM Daniel Latypov <dlatypov@google.com> wrote:
>>
>> From: Uriel Guajardo <urielguajardo@google.com>
>>
>> Add a kunit_fail_current_test() function to fail the currently running
>> test, if any, with an error message.
>>
>> This is largely intended for dynamic analysis tools like UBSAN and for
>> fakes.
>> E.g. say I had a fake ops struct for testing and I wanted my `free`
>> function to complain if it was called with an invalid argument, or
>> caught a double-free. Most return void and have no normal means of
>> signalling failure (e.g. super_operations, iommu_ops, etc.).
>>
>> Key points:
>> * Always update current->kunit_test so anyone can use it.
>>    * commit 83c4e7a0363b ("KUnit: KASAN Integration") only updated it for
>>    CONFIG_KASAN=y
>>
>> * Create a new header <kunit/test-bug.h> so non-test code doesn't have
>> to include all of <kunit/test.h> (e.g. lib/ubsan.c)
>>
>> * Forward the file and line number to make it easier to track down
>> failures
>>
>> * Declare the helper function for nice __printf() warnings about mismatched
>> format strings even when KUnit is not enabled.
>>
>> Example output from kunit_fail_current_test("message"):
>> [15:19:34] [FAILED] example_simple_test
>> [15:19:34]     # example_simple_test: initializing
>> [15:19:34]     # example_simple_test: lib/kunit/kunit-example-test.c:24: message
>> [15:19:34]     not ok 1 - example_simple_test
>>
>> Co-developed-by: Daniel Latypov <dlatypov@google.com>
>> Signed-off-by: Daniel Latypov <dlatypov@google.com>
>> Signed-off-by: Uriel Guajardo <urielguajardo@google.com>
>> Reviewed-by: Alan Maguire <alan.maguire@oracle.com>
> 
> Reviewed-by: Brendan Higgins <brendanhiggins@google.com>
> 

Please run checkpatch on your patches in the future. I am seeing
a few checkpatch readability type improvements that can be made.

Please make changes and send v2 with Brendan's Reviewed-by.

thanks,
-- Shuah

Re: [PATCH v4 1/2] kunit: support failure from dynamic analysis tools

[Daniel Latypov]

On Fri, Apr 2, 2021 at 10:53 AM Shuah Khan <skhan@linuxfoundation.org> wrote:
>
> On 4/2/21 2:55 AM, Brendan Higgins wrote:
> > On Thu, Mar 11, 2021 at 7:23 AM Daniel Latypov <dlatypov@google.com> wrote:
> >>
> >> From: Uriel Guajardo <urielguajardo@google.com>
> >>
> >> Add a kunit_fail_current_test() function to fail the currently running
> >> test, if any, with an error message.
> >>
> >> This is largely intended for dynamic analysis tools like UBSAN and for
> >> fakes.
> >> E.g. say I had a fake ops struct for testing and I wanted my `free`
> >> function to complain if it was called with an invalid argument, or
> >> caught a double-free. Most return void and have no normal means of
> >> signalling failure (e.g. super_operations, iommu_ops, etc.).
> >>
> >> Key points:
> >> * Always update current->kunit_test so anyone can use it.
> >>    * commit 83c4e7a0363b ("KUnit: KASAN Integration") only updated it for
> >>    CONFIG_KASAN=y
> >>
> >> * Create a new header <kunit/test-bug.h> so non-test code doesn't have
> >> to include all of <kunit/test.h> (e.g. lib/ubsan.c)
> >>
> >> * Forward the file and line number to make it easier to track down
> >> failures
> >>
> >> * Declare the helper function for nice __printf() warnings about mismatched
> >> format strings even when KUnit is not enabled.
> >>
> >> Example output from kunit_fail_current_test("message"):
> >> [15:19:34] [FAILED] example_simple_test
> >> [15:19:34]     # example_simple_test: initializing
> >> [15:19:34]     # example_simple_test: lib/kunit/kunit-example-test.c:24: message
> >> [15:19:34]     not ok 1 - example_simple_test
> >>
> >> Co-developed-by: Daniel Latypov <dlatypov@google.com>
> >> Signed-off-by: Daniel Latypov <dlatypov@google.com>
> >> Signed-off-by: Uriel Guajardo <urielguajardo@google.com>
> >> Reviewed-by: Alan Maguire <alan.maguire@oracle.com>
> >
> > Reviewed-by: Brendan Higgins <brendanhiggins@google.com>
> >
>
> Please run checkpatch on your patches in the future. I am seeing
> a few checkpatch readability type improvements that can be made.
>
> Please make changes and send v2 with Brendan's Reviewed-by.

Thanks for the catch.
checkpatch.pl --strict should now be happy (aside from complaining
about line wrapping)

v5 here: https://lore.kernel.org/linux-kselftest/20210402212131.835276-1-dlatypov@google.com

Note: Brendan didn't give an explicit Reviewed-by on the second patch,
not sure if that was intentional.

>
> thanks,
> -- Shuah

Re: [PATCH v4 1/2] kunit: support failure from dynamic analysis tools

[Shuah Khan]

On 4/2/21 3:25 PM, Daniel Latypov wrote:
> On Fri, Apr 2, 2021 at 10:53 AM Shuah Khan <skhan@linuxfoundation.org> wrote:
>>
>> On 4/2/21 2:55 AM, Brendan Higgins wrote:
>>> On Thu, Mar 11, 2021 at 7:23 AM Daniel Latypov <dlatypov@google.com> wrote:
>>>>
>>>> From: Uriel Guajardo <urielguajardo@google.com>
>>>>
>>>> Add a kunit_fail_current_test() function to fail the currently running
>>>> test, if any, with an error message.
>>>>
>>>> This is largely intended for dynamic analysis tools like UBSAN and for
>>>> fakes.
>>>> E.g. say I had a fake ops struct for testing and I wanted my `free`
>>>> function to complain if it was called with an invalid argument, or
>>>> caught a double-free. Most return void and have no normal means of
>>>> signalling failure (e.g. super_operations, iommu_ops, etc.).
>>>>
>>>> Key points:
>>>> * Always update current->kunit_test so anyone can use it.
>>>>     * commit 83c4e7a0363b ("KUnit: KASAN Integration") only updated it for
>>>>     CONFIG_KASAN=y
>>>>
>>>> * Create a new header <kunit/test-bug.h> so non-test code doesn't have
>>>> to include all of <kunit/test.h> (e.g. lib/ubsan.c)
>>>>
>>>> * Forward the file and line number to make it easier to track down
>>>> failures
>>>>
>>>> * Declare the helper function for nice __printf() warnings about mismatched
>>>> format strings even when KUnit is not enabled.
>>>>
>>>> Example output from kunit_fail_current_test("message"):
>>>> [15:19:34] [FAILED] example_simple_test
>>>> [15:19:34]     # example_simple_test: initializing
>>>> [15:19:34]     # example_simple_test: lib/kunit/kunit-example-test.c:24: message
>>>> [15:19:34]     not ok 1 - example_simple_test
>>>>
>>>> Co-developed-by: Daniel Latypov <dlatypov@google.com>
>>>> Signed-off-by: Daniel Latypov <dlatypov@google.com>
>>>> Signed-off-by: Uriel Guajardo <urielguajardo@google.com>
>>>> Reviewed-by: Alan Maguire <alan.maguire@oracle.com>
>>>
>>> Reviewed-by: Brendan Higgins <brendanhiggins@google.com>
>>>
>>
>> Please run checkpatch on your patches in the future. I am seeing
>> a few checkpatch readability type improvements that can be made.
>>
>> Please make changes and send v2 with Brendan's Reviewed-by.
> 
> Thanks for the catch.
> checkpatch.pl --strict should now be happy (aside from complaining
> about line wrapping)
> 
> v5 here: https://lore.kernel.org/linux-kselftest/20210402212131.835276-1-dlatypov@google.com
> 
> Note: Brendan didn't give an explicit Reviewed-by on the second patch,
> not sure if that was intentional.
> 

No worries. I applied this one as well. I was able to fix it with just
checkpatch --fix option.

All set now.

thanks,
-- Shuah

Re: [PATCH v4 1/2] kunit: support failure from dynamic analysis tools

[Shuah Khan]

On 4/2/21 3:44 PM, Shuah Khan wrote:
> On 4/2/21 3:25 PM, Daniel Latypov wrote:
>> On Fri, Apr 2, 2021 at 10:53 AM Shuah Khan <skhan@linuxfoundation.org> 
>> wrote:
>>>
>>> On 4/2/21 2:55 AM, Brendan Higgins wrote:
>>>> On Thu, Mar 11, 2021 at 7:23 AM Daniel Latypov <dlatypov@google.com> 
>>>> wrote:
>>>>>
>>>>> From: Uriel Guajardo <urielguajardo@google.com>
>>>>>
>>>>> Add a kunit_fail_current_test() function to fail the currently running
>>>>> test, if any, with an error message.
>>>>>
>>>>> This is largely intended for dynamic analysis tools like UBSAN and for
>>>>> fakes.
>>>>> E.g. say I had a fake ops struct for testing and I wanted my `free`
>>>>> function to complain if it was called with an invalid argument, or
>>>>> caught a double-free. Most return void and have no normal means of
>>>>> signalling failure (e.g. super_operations, iommu_ops, etc.).
>>>>>
>>>>> Key points:
>>>>> * Always update current->kunit_test so anyone can use it.
>>>>>     * commit 83c4e7a0363b ("KUnit: KASAN Integration") only updated 
>>>>> it for
>>>>>     CONFIG_KASAN=y
>>>>>
>>>>> * Create a new header <kunit/test-bug.h> so non-test code doesn't have
>>>>> to include all of <kunit/test.h> (e.g. lib/ubsan.c)
>>>>>
>>>>> * Forward the file and line number to make it easier to track down
>>>>> failures
>>>>>
>>>>> * Declare the helper function for nice __printf() warnings about 
>>>>> mismatched
>>>>> format strings even when KUnit is not enabled.
>>>>>
>>>>> Example output from kunit_fail_current_test("message"):
>>>>> [15:19:34] [FAILED] example_simple_test
>>>>> [15:19:34]     # example_simple_test: initializing
>>>>> [15:19:34]     # example_simple_test: 
>>>>> lib/kunit/kunit-example-test.c:24: message
>>>>> [15:19:34]     not ok 1 - example_simple_test
>>>>>
>>>>> Co-developed-by: Daniel Latypov <dlatypov@google.com>
>>>>> Signed-off-by: Daniel Latypov <dlatypov@google.com>
>>>>> Signed-off-by: Uriel Guajardo <urielguajardo@google.com>
>>>>> Reviewed-by: Alan Maguire <alan.maguire@oracle.com>
>>>>
>>>> Reviewed-by: Brendan Higgins <brendanhiggins@google.com>
>>>>
>>>
>>> Please run checkpatch on your patches in the future. I am seeing
>>> a few checkpatch readability type improvements that can be made.
>>>
>>> Please make changes and send v2 with Brendan's Reviewed-by.
>>
>> Thanks for the catch.
>> checkpatch.pl --strict should now be happy (aside from complaining
>> about line wrapping)
>>
>> v5 here: 
>> https://lore.kernel.org/linux-kselftest/20210402212131.835276-1-dlatypov@google.com 
>>
>>
>> Note: Brendan didn't give an explicit Reviewed-by on the second patch,
>> not sure if that was intentional.
>>
> 
> No worries. I applied this one as well. I was able to fix it with just
> checkpatch --fix option.
> 

Clarification. Applied 1/2 - I will wait for Brendan's ack on 2/2

thanks,
-- Shuah