From: Arpitha Raghunandan <98.arpi@gmail.com>
To: Rasmus Villemoes <linux@rasmusvillemoes.dk>,
Andy Shevchenko <andriy.shevchenko@linux.intel.com>
Cc: brendanhiggins@google.com, skhan@linuxfoundation.org,
pmladek@suse.com, rostedt@goodmis.org,
sergey.senozhatsky@gmail.com, alexandre.belloni@bootlin.com,
gregkh@linuxfoundation.org, rdunlap@infradead.org,
idryomov@gmail.com, kunit-dev@googlegroups.com,
linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org,
linux-kernel-mentees@lists.linuxfoundation.org
Subject: Re: [PATCH v2] lib: Convert test_printf.c to KUnit
Date: Fri, 23 Oct 2020 19:13:00 +0530 [thread overview]
Message-ID: <01f8ca45-60d1-ad67-f8eb-354dec411a78@gmail.com> (raw)
In-Reply-To: <0ab618c7-8c5c-00ae-8e08-0c1b99f3bf5c@rasmusvillemoes.dk>
On 23/10/20 4:36 pm, Rasmus Villemoes wrote:
> On 22/10/2020 21.16, Andy Shevchenko wrote:
>> On Thu, Oct 22, 2020 at 08:43:49PM +0530, Arpitha Raghunandan wrote:
>>> Convert test lib/test_printf.c to KUnit. More information about
>>> KUnit can be found at:
>>> https://www.kernel.org/doc/html/latest/dev-tools/kunit/index.html.
>>> KUnit provides a common framework for unit tests in the kernel.
>>> KUnit and kselftest are standardizing around KTAP, converting this
>>> test to KUnit makes this test output in KTAP which we are trying to
>>> make the standard test result format for the kernel. More about
>>> the KTAP format can be found at:
>>> https://lore.kernel.org/linux-kselftest/CY4PR13MB1175B804E31E502221BC8163FD830@CY4PR13MB1175.namprd13.prod.outlook.com/.
>>> I ran both the original and converted tests as is to produce the
>>> output for success of the test in the two cases. I also ran these
>>> tests with a small modification to show the difference in the output
>>> for failure of the test in both cases. The modification I made is:
>>> - test("127.000.000.001|127.0.0.1", "%pi4|%pI4", &sa.sin_addr, &sa.sin_addr);
>>> + test("127-000.000.001|127.0.0.1", "%pi4|%pI4", &sa.sin_addr, &sa.sin_addr);
>>>
>>> Original test success:
>>> [ 0.591262] test_printf: loaded.
>>> [ 0.591409] test_printf: all 388 tests passed
>>>
>>> Original test failure:
>>> [ 0.619345] test_printf: loaded.
>>> [ 0.619394] test_printf: vsnprintf(buf, 256, "%piS|%pIS", ...)
>>> wrote '127.000.000.001|127.0.0.1', expected
>>> '127-000.000.001|127.0.0.1'
>>> [ 0.619395] test_printf: vsnprintf(buf, 25, "%piS|%pIS", ...) wrote
>>> '127.000.000.001|127.0.0.', expected '127-000.000.001|127.0.0.'
>>> [ 0.619396] test_printf: kvasprintf(..., "%piS|%pIS", ...) returned
>>> '127.000.000.001|127.0.0.1', expected '127-000.000.001|127.0.0.1'
>>> [ 0.619495] test_printf: failed 3 out of 388 tests
>>>
>>> Converted test success:
>>> # Subtest: printf-kunit-test
>>> 1..1
>>> ok 1 - selftest
>>> ok 1 - printf-kunit-test
>>>
>>> Converted test failure:
>>> # Subtest: printf-kunit-test
>>> 1..1
>>> # selftest: EXPECTATION FAILED at lib/printf_kunit.c:82
>>> vsnprintf(buf, 256, "%pi4|%pI4", ...) wrote
>>> '127.000.000.001|127.0.0.1', expected '127-000.000.001|127.0.0.1'
>>> # selftest: EXPECTATION FAILED at lib/printf_kunit.c:82
>>> vsnprintf(buf, 5, "%pi4|%pI4", ...) wrote '127.', expected '127-'
>>> # selftest: EXPECTATION FAILED at lib/printf_kunit.c:118
>>> kvasprintf(..., "%pi4|%pI4", ...) returned
>>> '127.000.000.001|127.0.0.1', expected '127-000.000.001|127.0.0.1'
>>> not ok 1 - selftest
>>> not ok 1 - printf-kunit-test
>>
>> Not bad. Rasmus, what do you think?
>
> Much better, but that '1..1' and reporting the entire test suite as 1
> single (failing or passing) test is (also) a regression. Look at the
> original
>
>>> [ 0.591409] test_printf: all 388 tests passed
>
> or
>
>>> [ 0.619495] test_printf: failed 3 out of 388 tests
>
> That's far more informative, and I'd prefer if the summary information
> (whether in the all-good case or some-failing) included something like
> this. In particular, I have at some point spotted that I failed to
> properly hook up a new test case (or perhaps failed to re-compile, or
> somehow still ran the old kernel binary, don't remember which it was) by
> noticing that the total number of tests hadn't increased. The new output
> would not help catch such PEBKACs.
>
> Rasmus
>
Splitting the test into multiple test cases in KUnit will display the number and name of tests that pass or fail. This will be similar to the lib/list-test.c test as can be seen here: https://elixir.bootlin.com/linux/latest/source/lib/list-test.c. I will work on this for the next version of this patch.
next prev parent reply other threads:[~2020-10-23 13:43 UTC|newest]
Thread overview: 9+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-10-22 15:13 [PATCH v2] lib: Convert test_printf.c to KUnit Arpitha Raghunandan
2020-10-22 19:16 ` Andy Shevchenko
2020-10-23 11:06 ` Rasmus Villemoes
2020-10-23 13:43 ` Arpitha Raghunandan [this message]
2020-10-23 18:01 ` Petr Mladek
2020-10-24 5:08 ` Arpitha Raghunandan
2020-10-23 17:31 ` Petr Mladek
2020-10-25 12:38 ` Andy Shevchenko
2020-10-26 9:48 ` Petr Mladek
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=01f8ca45-60d1-ad67-f8eb-354dec411a78@gmail.com \
--to=98.arpi@gmail.com \
--cc=alexandre.belloni@bootlin.com \
--cc=andriy.shevchenko@linux.intel.com \
--cc=brendanhiggins@google.com \
--cc=gregkh@linuxfoundation.org \
--cc=idryomov@gmail.com \
--cc=kunit-dev@googlegroups.com \
--cc=linux-kernel-mentees@lists.linuxfoundation.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-kselftest@vger.kernel.org \
--cc=linux@rasmusvillemoes.dk \
--cc=pmladek@suse.com \
--cc=rdunlap@infradead.org \
--cc=rostedt@goodmis.org \
--cc=sergey.senozhatsky@gmail.com \
--cc=skhan@linuxfoundation.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).