Linux-kselftest Archive on lore.kernel.org
 help / color / Atom feed
From: Frank Rowand <frowand.list@gmail.com>
To: Brendan Higgins <brendanhiggins@google.com>,
	jdike@addtoit.com, richard@nod.at,
	anton.ivanov@cambridgegreys.com, arnd@arndb.de,
	keescook@chromium.org, skhan@linuxfoundation.org,
	alan.maguire@oracle.com, yzaikin@google.com, davidgow@google.com,
	akpm@linux-foundation.org, rppt@linux.ibm.com
Cc: gregkh@linuxfoundation.org, sboyd@kernel.org,
	logang@deltatee.com, mcgrof@kernel.org, knut.omang@oracle.com,
	linux-um@lists.infradead.org, linux-arch@vger.kernel.org,
	linux-kselftest@vger.kernel.org, kunit-dev@googlegroups.com,
	linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org
Subject: Re: [PATCH v2 2/7] arch: um: add linker section for KUnit test suites
Date: Tue, 4 Feb 2020 15:59:11 -0600
Message-ID: <e060bdfc-5cdb-fb62-48b0-cc54c7bc72ce@gmail.com> (raw)
In-Reply-To: <20200130230812.142642-3-brendanhiggins@google.com>

On 1/30/20 5:08 PM, Brendan Higgins wrote:
> Add a linker section to UML where KUnit can put references to its test
> suites. This patch is an early step in transitioning to dispatching all
> KUnit tests from a centralized executor rather than having each as its
> own separate late_initcall.

All architectures please.

The early versions of Kunit documented reliance on UML.  Discussion lead to
the conclusion that real architectures and real hardware would be supported.

This like this are what make me reluctant to move devicetree unittests to
KUnit.

Can you please add a section to the KUnit documentation that lists things
like the expectations, requirements, limitations, etc for a test case that
is run by KUnit?  Some examples that pop to mind from recent discussions
and my own experiences:

  - Each test case is invoked after late_init is complete.
      + Exception: the possible value of being able to run a unit test
        at a specific runlevel has been expressed.  If an actual unit
        test can be shown to require running earlier, this restriction
        will be re-visited.

  - Each test case must be idempotent.  Each test case may be called
    multiple times, and must generate the same result each time it
    is called.
      + Exception 1: a test case can be declared to not be idempotent
        [[ mechanism TBD ]], in which case KUnit will not call the
        test case a second time without the kernel rebooting.
      + Exception 2: hardware may not be deterministic, so a test that
        always passes or fails when run under UML may not always to
        so on real hardware.  <--- sentence copied from
        Documentation/dev-tools/kunit/usage.rst
          [[ This item and 1st exception do not exist yet, but will exist
          in some form if the proposed proc filesystem interface is
          added. ]]

  - KUnit provides a helpful wrapper to simplify building a UML kernel
    containing the KUnit test cases, booting the UML kernel, and
    formatting the output from the test cases.  This wrapper MUST NOT
    be required to run the test cases or to determine a test result.
    The formatting may provide additional analysis and improve
    readability of a test result.

  - .... There is more that belongs here, but I'm getting side tracked
    here, when I'm trying to instead convert devicetree unittests to
    KUnit and want to get back to that.


-Frank

> 
> Signed-off-by: Brendan Higgins <brendanhiggins@google.com>
> Reviewed-by: Stephen Boyd <sboyd@kernel.org>
> ---
>  arch/um/include/asm/common.lds.S | 4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/arch/um/include/asm/common.lds.S b/arch/um/include/asm/common.lds.S
> index 7145ce6999822..eab9ceb450efd 100644
> --- a/arch/um/include/asm/common.lds.S
> +++ b/arch/um/include/asm/common.lds.S
> @@ -52,6 +52,10 @@
>  	CON_INITCALL
>    }
>  
> +  .kunit_test_suites : {
> +	KUNIT_TEST_SUITES
> +  }
> +
>    .exitcall : {
>  	__exitcall_begin = .;
>  	*(.exitcall.exit)
> 


  reply index

Thread overview: 26+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-01-30 23:08 [PATCH v2 0/7] kunit: create a centralized executor to dispatch all KUnit tests Brendan Higgins
2020-01-30 23:08 ` [PATCH v2 1/7] vmlinux.lds.h: add linker section for KUnit test suites Brendan Higgins
2020-01-30 23:08 ` [PATCH v2 2/7] arch: um: " Brendan Higgins
2020-02-04 21:59   ` Frank Rowand [this message]
2020-02-04 22:30     ` Brendan Higgins
2020-02-04 23:17       ` Frank Rowand
2020-02-05  1:17         ` Brendan Higgins
2020-02-06 19:22           ` David Gow
2020-01-30 23:08 ` [PATCH v2 3/7] kunit: test: create a single centralized executor for all tests Brendan Higgins
2020-02-04 22:27   ` Frank Rowand
2020-02-04 22:35     ` Brendan Higgins
2020-01-30 23:08 ` [PATCH v2 4/7] init: main: add KUnit to kernel init Brendan Higgins
2020-03-02 20:00   ` Luis Chamberlain
2020-01-30 23:08 ` [PATCH v2 5/7] kunit: test: add test plan to KUnit TAP format Brendan Higgins
2020-02-04 23:01   ` Frank Rowand
2020-02-06 18:47     ` Brendan Higgins
2020-01-30 23:08 ` [PATCH v2 6/7] kunit: Add 'kunit_shutdown' option Brendan Higgins
2020-02-04 23:05   ` Frank Rowand
2020-02-04 23:05     ` Frank Rowand
2020-01-30 23:08 ` [PATCH v2 7/7] Documentation: Add kunit_shutdown to kernel-parameters.txt Brendan Higgins
2020-02-05  0:19   ` Frank Rowand
2020-02-04  7:19 ` [PATCH v2 0/7] kunit: create a centralized executor to dispatch all KUnit tests Stephen Boyd
2020-02-04 19:35   ` Brendan Higgins
2020-02-04 23:35     ` Frank Rowand
2020-02-04 21:18 ` Frank Rowand
2020-02-04 22:01   ` Brendan Higgins

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=e060bdfc-5cdb-fb62-48b0-cc54c7bc72ce@gmail.com \
    --to=frowand.list@gmail.com \
    --cc=akpm@linux-foundation.org \
    --cc=alan.maguire@oracle.com \
    --cc=anton.ivanov@cambridgegreys.com \
    --cc=arnd@arndb.de \
    --cc=brendanhiggins@google.com \
    --cc=davidgow@google.com \
    --cc=gregkh@linuxfoundation.org \
    --cc=jdike@addtoit.com \
    --cc=keescook@chromium.org \
    --cc=knut.omang@oracle.com \
    --cc=kunit-dev@googlegroups.com \
    --cc=linux-arch@vger.kernel.org \
    --cc=linux-doc@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-kselftest@vger.kernel.org \
    --cc=linux-um@lists.infradead.org \
    --cc=logang@deltatee.com \
    --cc=mcgrof@kernel.org \
    --cc=richard@nod.at \
    --cc=rppt@linux.ibm.com \
    --cc=sboyd@kernel.org \
    --cc=skhan@linuxfoundation.org \
    --cc=yzaikin@google.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-kselftest Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-kselftest/0 linux-kselftest/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-kselftest linux-kselftest/ https://lore.kernel.org/linux-kselftest \
		linux-kselftest@vger.kernel.org
	public-inbox-index linux-kselftest

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-kselftest


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git