[RFC v1 0/6] kunit: Add test attributes API

From: Rae Moar <rmoar@google.com>
To: shuah@kernel.org, davidgow@google.com, dlatypov@google.com,
	brendan.higgins@linux.dev
Cc: linux-kselftest@vger.kernel.org, kunit-dev@googlegroups.com,
	linux-kernel@vger.kernel.org, keescook@chromium.org,
	linux-hardening@vger.kernel.org, jstultz@google.com,
	tglx@linutronix.de, sboyd@kernel.org, Rae Moar <rmoar@google.com>
Subject: [RFC v1 0/6] kunit: Add test attributes API
Date: Sat, 10 Jun 2023 00:51:43 +0000	[thread overview]
Message-ID: <20230610005149.1145665-1-rmoar@google.com> (raw)

Hello everyone,

This is an RFC patch series to propose the addition of a test attributes
framework to KUnit.

There has been interest in filtering out "slow" KUnit tests. Most notably,
a new config, CONFIG_MEMCPY_SLOW_KUNIT_TEST, has been added to exclude
particularly slow memcpy tests
(https://lore.kernel.org/all/20230118200653.give.574-kees@kernel.org/).

This proposed attributes framework would be used to save and access test
associated data, including whether a test is slow. These attributes would
be reportable (via KTAP and command line output) and some will be
filterable.

This framework is designed to allow for the addition of other attributes in
the future. These attributes could include whether the test is flaky,
associated test files, etc.

Note that this could intersect with the discussions on how to format
test-associated data in KTAP v2 that I am also involved in
(https://lore.kernel.org/all/20230420205734.1288498-1-rmoar@google.com/).

If the overall idea seems good, I'll make sure to add tests/documentation,
and more patches marking existing tests as slow to the patch series.

Thanks!
Rae

Rae Moar (6):
  kunit: Add test attributes API structure
  kunit: Add speed attribute
  kunit: Add ability to filter attributes
  kunit: tool: Add command line interface to filter and report
    attributes
  kunit: memcpy: Mark tests as slow using test attributes
  kunit: time: Mark test as slow using test attributes

 include/kunit/attributes.h             |  41 ++++
 include/kunit/test.h                   |  62 ++++++
 kernel/time/time_test.c                |   2 +-
 lib/kunit/Makefile                     |   3 +-
 lib/kunit/attributes.c                 | 280 +++++++++++++++++++++++++
 lib/kunit/executor.c                   |  89 ++++++--
 lib/kunit/executor_test.c              |   8 +-
 lib/kunit/kunit-example-test.c         |   9 +
 lib/kunit/test.c                       |  17 +-
 lib/memcpy_kunit.c                     |   8 +-
 tools/testing/kunit/kunit.py           |  34 ++-
 tools/testing/kunit/kunit_kernel.py    |   6 +-
 tools/testing/kunit/kunit_tool_test.py |  41 ++--
 13 files changed, 536 insertions(+), 64 deletions(-)
 create mode 100644 include/kunit/attributes.h
 create mode 100644 lib/kunit/attributes.c

base-commit: fefdb43943c1a0d87e1b43ae4d03e5f9a1d058f4
-- 
2.41.0.162.gfafddb0af9-goog