Linux-kselftest Archive on lore.kernel.org
 help / color / Atom feed
From: Brendan Higgins <brendanhiggins@google.com>
To: Knut Omang <knut.omang@oracle.com>
Cc: "open list:KERNEL SELFTEST FRAMEWORK" 
	<linux-kselftest@vger.kernel.org>,
	Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
	"open list:DOCUMENTATION" <linux-doc@vger.kernel.org>,
	linux-kbuild <linux-kbuild@vger.kernel.org>,
	Shuah Khan <shuah@kernel.org>, Jonathan Corbet <corbet@lwn.net>,
	Masahiro Yamada <yamada.masahiro@socionext.com>,
	Michal Marek <michal.lkml@markovi.net>,
	Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
	Shreyans Devendra Doshi <0xinfosect0r@gmail.com>,
	Alan Maguire <alan.maguire@oracle.com>,
	Kevin Hilman <khilman@baylibre.com>,
	Hidenori Yamaji <hidenori.yamaji@sony.com>,
	Frank Rowand <frowand.list@gmail.com>,
	Timothy Bird <Tim.Bird@sony.com>,
	Luis Chamberlain <mcgrof@kernel.org>,
	"Theodore Ts'o" <tytso@mit.edu>, Daniel Vetter <daniel@ffwll.ch>,
	Stephen Boyd <sboyd@kernel.org>
Subject: Re: [RFC 00/19] Integration of Kernel Test Framework (KTF) into the kernel tree
Date: Tue, 13 Aug 2019 10:50:49 -0700
Message-ID: <CAFd5g47Q+bRQfyyqPx7CvcnUWnpDxO9goXtEaH8U4EWLbmhFUA@mail.gmail.com> (raw)
In-Reply-To: <b4d2cf8635103e30313773761b095f3a1cfd9a82.camel@oracle.com>

On Tue, Aug 13, 2019 at 4:29 AM Knut Omang <knut.omang@oracle.com> wrote:
>
> On Tue, 2019-08-13 at 01:17 -0700, Brendan Higgins wrote:
> > On Mon, Aug 12, 2019 at 11:11 PM Knut Omang <knut.omang@oracle.com> wrote:
> > [...]
> > > Alan Maguire (3):
> > >   ktf: Implementation of ktf support for overriding function entry and return.
> > >   ktf: A simple debugfs interface to test results
> > >   ktf: Simple coverage support
> > >
> > > Knut Omang (16):
> > >   kbuild: Fixes to rules for host-cshlib and host-cxxshlib
> > >   ktf: Introduce the main part of the kernel side of ktf
> > >   ktf: Introduce a generic netlink protocol for test result communication
> > >   ktf: An implementation of a generic associative array container
> > >   ktf: Configurable context support for network info setup
> > >   ktf: resolve: A helper utility to aid in exposing private kernel symbols to KTF tests.
> > >   ktf: Add documentation for Kernel Test Framework (KTF)
> > >   ktf: Add a small test suite with a few tests to test KTF itself
> > >   ktf: Main part of user land library for executing tests
> > >   ktf: Integration logic for running ktf tests from googletest
> > >   ktf: Internal debugging facilities
> > >   ktf: Some simple examples
> > >   ktf: Some user applications to run tests
> > >   ktf: Toplevel ktf Makefile/makefile includes and scripts to run from kselftest
> > >   kselftests: Enable building ktf
> > >   Documentation/dev-tools: Add index entry for KTF documentation
> > >
> > >  Documentation/dev-tools/index.rst                   |    1 +-
> > >  Documentation/dev-tools/ktf/concepts.rst            |  242 +++-
> > >  Documentation/dev-tools/ktf/debugging.rst           |  248 +++-
> > >  Documentation/dev-tools/ktf/examples.rst            |   26 +-
> > >  Documentation/dev-tools/ktf/features.rst            |  307 ++++-
> > >  Documentation/dev-tools/ktf/implementation.rst      |   70 +-
> > >  Documentation/dev-tools/ktf/index.rst               |   14 +-
> > >  Documentation/dev-tools/ktf/installation.rst        |   73 +-
> > >  Documentation/dev-tools/ktf/introduction.rst        |  134 ++-
> > >  Documentation/dev-tools/ktf/progref.rst             |  144 ++-
> > >  scripts/Makefile.host                               |   17 +-
> > >  tools/testing/selftests/Makefile                    |    1 +-
> > >  tools/testing/selftests/ktf/Makefile                |   21 +-
> > >  tools/testing/selftests/ktf/examples/Makefile       |   17 +-
> > >  tools/testing/selftests/ktf/examples/h2.c           |   45 +-
> > >  tools/testing/selftests/ktf/examples/h3.c           |   84 +-
> > >  tools/testing/selftests/ktf/examples/h4.c           |   62 +-
> > >  tools/testing/selftests/ktf/examples/hello.c        |   38 +-
> > >  tools/testing/selftests/ktf/examples/kgdemo.c       |   61 +-
> > >  tools/testing/selftests/ktf/kernel/Makefile         |   15 +-
> > >  tools/testing/selftests/ktf/kernel/ktf.h            |  604 +++++++-
> > >  tools/testing/selftests/ktf/kernel/ktf_context.c    |  409 +++++-
> > >  tools/testing/selftests/ktf/kernel/ktf_cov.c        |  690 ++++++++-
> > >  tools/testing/selftests/ktf/kernel/ktf_cov.h        |   94 +-
> > >  tools/testing/selftests/ktf/kernel/ktf_debugfs.c    |  356 ++++-
> > >  tools/testing/selftests/ktf/kernel/ktf_debugfs.h    |   34 +-
> > >  tools/testing/selftests/ktf/kernel/ktf_map.c        |  261 +++-
> > >  tools/testing/selftests/ktf/kernel/ktf_map.h        |  154 ++-
> > >  tools/testing/selftests/ktf/kernel/ktf_netctx.c     |  132 ++-
> > >  tools/testing/selftests/ktf/kernel/ktf_netctx.h     |   64 +-
> > >  tools/testing/selftests/ktf/kernel/ktf_nl.c         |  516 ++++++-
> > >  tools/testing/selftests/ktf/kernel/ktf_nl.h         |   15 +-
> > >  tools/testing/selftests/ktf/kernel/ktf_override.c   |   45 +-
> > >  tools/testing/selftests/ktf/kernel/ktf_override.h   |   15 +-
> > >  tools/testing/selftests/ktf/kernel/ktf_test.c       |  397 +++++-
> > >  tools/testing/selftests/ktf/kernel/ktf_test.h       |  381 ++++-
> > >  tools/testing/selftests/ktf/kernel/ktf_unlproto.h   |  105 +-
> > >  tools/testing/selftests/ktf/lib/Makefile            |   21 +-
> > >  tools/testing/selftests/ktf/lib/ktf.h               |  114 +-
> > >  tools/testing/selftests/ktf/lib/ktf_debug.cc        |   20 +-
> > >  tools/testing/selftests/ktf/lib/ktf_debug.h         |   59 +-
> > >  tools/testing/selftests/ktf/lib/ktf_int.cc          | 1031 ++++++++++++-
> > >  tools/testing/selftests/ktf/lib/ktf_int.h           |   84 +-
> > >  tools/testing/selftests/ktf/lib/ktf_run.cc          |  177 ++-
> > >  tools/testing/selftests/ktf/lib/ktf_unlproto.c      |   21 +-
> > >  tools/testing/selftests/ktf/scripts/ktf_syms.mk     |   16 +-
> > >  tools/testing/selftests/ktf/scripts/resolve         |  188 ++-
> > >  tools/testing/selftests/ktf/scripts/runtests.mk     |    3 +-
> > >  tools/testing/selftests/ktf/scripts/runtests.sh     |  100 +-
> > >  tools/testing/selftests/ktf/scripts/top_make.mk     |   14 +-
> > >  tools/testing/selftests/ktf/selftest/Makefile       |   17 +-
> > >  tools/testing/selftests/ktf/selftest/context.c      |  149 ++-
> > >  tools/testing/selftests/ktf/selftest/context.h      |   15 +-
> > >  tools/testing/selftests/ktf/selftest/context_self.h |   34 +-
> > >  tools/testing/selftests/ktf/selftest/hybrid.c       |   35 +-
> > >  tools/testing/selftests/ktf/selftest/hybrid.h       |   24 +-
> > >  tools/testing/selftests/ktf/selftest/hybrid_self.h  |   27 +-
> > >  tools/testing/selftests/ktf/selftest/ktf_syms.txt   |   17 +-
> > >  tools/testing/selftests/ktf/selftest/self.c         |  661 ++++++++-
> > >  tools/testing/selftests/ktf/user/Makefile           |   26 +-
> > >  tools/testing/selftests/ktf/user/hybrid.cc          |   39 +-
> > >  tools/testing/selftests/ktf/user/ktfcov.cc          |   68 +-
> > >  tools/testing/selftests/ktf/user/ktfrun.cc          |   20 +-
> > >  tools/testing/selftests/ktf/user/ktftest.cc         |   46 +-
> > >  64 files changed, 8909 insertions(+), 9 deletions(-)
> >
> > It also looks like all your test code lives outside of the kernel
> > source dir. I take it you intend for tests to live in
> > tools/testing/selftests/ktf/ ?
>
> Good point,
> with this RFC it would be just to add another directory under ktf/
> but this was just to find a simple way to integrate it below selftests,
> without interfering with the current structure.
>
> I imagine a tighter integration/unification between normal Kbuild targets and selftests
> targets that also took kernel module building into consideration would be a better solution.

I think tests should live alongside the code that they test, so if I
understand what you are saying, then I agree. Where do you think KTF
should go then?

> I think this is a good topic for discussion.
> As I indicate above, I think it is problematic that test code has to be explicitly
> configured in as we configure code features of the kernel, which changes the "logical"
> kernel we build.

Yep, Luis Chamberlain and I have been discussing this for a while (I
should probably try to open up that conversation and add you to it);
it's a very tricky problem. I think that in the long term a good goal
is to have a way to express code dependencies separate from the
configuration system, but that would probably mean some substantial
changes to Kbuild and friends. Even then, we have all these macros
that generate different code (not just on/off) depending on how the
kernel is configured.

> So some more "native" support for test modules are desired, IMHO.

What do you mean by "native"? Just having the tests be more aware of the code?

  reply index

Thread overview: 44+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-08-13  6:09 Knut Omang
2019-08-13  6:09 ` [RFC 01/19] kbuild: Fixes to rules for host-cshlib and host-cxxshlib Knut Omang
2019-08-13 14:01   ` Masahiro Yamada
2019-08-13 16:19     ` Knut Omang
2019-08-14  2:02       ` Masahiro Yamada
2019-08-14  5:50         ` Knut Omang
2019-08-14  5:52         ` Knut Omang
2019-08-14 12:52           ` Knut Omang
2019-08-21  1:47             ` Masahiro Yamada
2019-08-21  4:03               ` Knut Omang
2019-08-13  6:09 ` [RFC 02/19] ktf: Introduce the main part of the kernel side of ktf Knut Omang
2019-09-09  1:23   ` Brendan Higgins
2019-09-10  6:15     ` Knut Omang
2019-08-13  6:09 ` [RFC 03/19] ktf: Introduce a generic netlink protocol for test result communication Knut Omang
2019-09-09  1:28   ` Brendan Higgins
2019-09-10  6:30     ` Knut Omang
2019-08-13  6:09 ` [RFC 04/19] ktf: An implementation of a generic associative array container Knut Omang
2019-08-13  6:09 ` [RFC 05/19] ktf: Implementation of ktf support for overriding function entry and return Knut Omang
2019-08-13  6:09 ` [RFC 06/19] ktf: A simple debugfs interface to test results Knut Omang
2019-08-13  8:21   ` Greg Kroah-Hartman
2019-08-14 17:17     ` Knut Omang
2019-08-15  8:49       ` Greg Kroah-Hartman
2019-08-15 10:35         ` Knut Omang
2019-08-15 10:52           ` Greg Kroah-Hartman
2019-08-13  6:09 ` [RFC 07/19] ktf: Simple coverage support Knut Omang
2019-08-13  6:09 ` [RFC 08/19] ktf: Configurable context support for network info setup Knut Omang
2019-08-13  6:09 ` [RFC 09/19] ktf: resolve: A helper utility to aid in exposing private kernel symbols to KTF tests Knut Omang
2019-08-13  6:09 ` [RFC 10/19] ktf: Add documentation for Kernel Test Framework (KTF) Knut Omang
2019-08-13  6:09 ` [RFC 11/19] ktf: Add a small test suite with a few tests to test KTF itself Knut Omang
2019-08-13  6:09 ` [RFC 12/19] ktf: Main part of user land library for executing tests Knut Omang
2019-08-13  6:09 ` [RFC 13/19] ktf: Integration logic for running ktf tests from googletest Knut Omang
2019-08-13  6:09 ` [RFC 14/19] ktf: Internal debugging facilities Knut Omang
2019-08-13  6:09 ` [RFC 15/19] ktf: Some simple examples Knut Omang
2019-08-13  6:09 ` [RFC 16/19] ktf: Some user applications to run tests Knut Omang
2019-08-13  6:09 ` [RFC 17/19] ktf: Toplevel ktf Makefile/makefile includes and scripts to run from kselftest Knut Omang
2019-08-13  6:09 ` [RFC 18/19] kselftests: Enable building ktf Knut Omang
2019-08-13  6:09 ` [RFC 19/19] Documentation/dev-tools: Add index entry for KTF documentation Knut Omang
2019-08-13  8:10 ` [RFC 00/19] Integration of Kernel Test Framework (KTF) into the kernel tree Brendan Higgins
2019-08-13  8:17 ` Brendan Higgins
2019-08-13 11:29   ` Knut Omang
2019-08-13 17:50     ` Brendan Higgins [this message]
2019-08-13  8:23 ` Greg Kroah-Hartman
2019-08-13  9:51   ` Knut Omang
2019-08-13 17:02     ` Brendan Higgins

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=CAFd5g47Q+bRQfyyqPx7CvcnUWnpDxO9goXtEaH8U4EWLbmhFUA@mail.gmail.com \
    --to=brendanhiggins@google.com \
    --cc=0xinfosect0r@gmail.com \
    --cc=Tim.Bird@sony.com \
    --cc=alan.maguire@oracle.com \
    --cc=corbet@lwn.net \
    --cc=daniel@ffwll.ch \
    --cc=frowand.list@gmail.com \
    --cc=gregkh@linuxfoundation.org \
    --cc=hidenori.yamaji@sony.com \
    --cc=khilman@baylibre.com \
    --cc=knut.omang@oracle.com \
    --cc=linux-doc@vger.kernel.org \
    --cc=linux-kbuild@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-kselftest@vger.kernel.org \
    --cc=mcgrof@kernel.org \
    --cc=michal.lkml@markovi.net \
    --cc=sboyd@kernel.org \
    --cc=shuah@kernel.org \
    --cc=tytso@mit.edu \
    --cc=yamada.masahiro@socionext.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-kselftest Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-kselftest/0 linux-kselftest/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-kselftest linux-kselftest/ https://lore.kernel.org/linux-kselftest \
		linux-kselftest@vger.kernel.org linux-kselftest@archiver.kernel.org
	public-inbox-index linux-kselftest


Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-kselftest


AGPL code for this site: git clone https://public-inbox.org/ public-inbox