From: Kieran Bingham <kieran.bingham@ideasonboard.com>
To: Brendan Higgins <brendanhiggins@google.com>,
gregkh@linuxfoundation.org, keescook@google.com,
mcgrof@kernel.org, shuah@kernel.org
Cc: brakmo@fb.com, jdike@addtoit.com,
dri-devel@lists.freedesktop.org, linux-kselftest@vger.kernel.org,
frowand.list@gmail.com, robh@kernel.org,
linux-nvdimm@lists.01.org, richard@nod.at, knut.omang@oracle.com,
Felix Guo <felixguoxiuping@gmail.com>,
joel@jms.id.au, khilman@baylibre.com, Tim.Bird@sony.com,
linux-um@lists.infradead.org, rostedt@goodmis.org,
julia.lawall@lip6.fr, kunit-dev@googlegroups.com,
linux-kernel@vger.kernel.org, daniel@ffwll.ch,
mpe@ellerman.id.au, joe@perches.com
Subject: Re: [RFC v3 11/19] kunit: add Python libraries for handing KUnit config and kernel
Date: Thu, 29 Nov 2018 13:54:00 +0000 [thread overview]
Message-ID: <841cf4ae-501b-05ae-5863-a51010709b67@ideasonboard.com> (raw)
In-Reply-To: <20181128193636.254378-12-brendanhiggins@google.com>
Hi Brendan,
Thanks again for this series!
On 28/11/2018 19:36, Brendan Higgins wrote:
> The ultimate goal is to create minimal isolated test binaries; in the
> meantime we are using UML to provide the infrastructure to run tests, so
> define an abstract way to configure and run tests that allow us to
> change the context in which tests are built without affecting the user.
> This also makes pretty and dynamic error reporting, and a lot of other
> nice features easier.
I wonder if we could somehow generate a shared library object
'libkernel' or 'libumlinux' from a UM configured set of headers and
objects so that we could create binary targets directly ?
> kunit_config.py:
> - parse .config and Kconfig files.
>
> kunit_kernel.py: provides helper functions to:
> - configure the kernel using kunitconfig.
> - build the kernel with the appropriate configuration.
> - provide function to invoke the kernel and stream the output back.
>
> Signed-off-by: Felix Guo <felixguoxiuping@gmail.com>
> Signed-off-by: Brendan Higgins <brendanhiggins@google.com>
> ---
> tools/testing/kunit/.gitignore | 3 +
> tools/testing/kunit/kunit_config.py | 60 +++++++++++++
> tools/testing/kunit/kunit_kernel.py | 126 ++++++++++++++++++++++++++++
> 3 files changed, 189 insertions(+)
> create mode 100644 tools/testing/kunit/.gitignore
> create mode 100644 tools/testing/kunit/kunit_config.py
> create mode 100644 tools/testing/kunit/kunit_kernel.py
>
> diff --git a/tools/testing/kunit/.gitignore b/tools/testing/kunit/.gitignore
> new file mode 100644
> index 0000000000000..c791ff59a37a9
> --- /dev/null
> +++ b/tools/testing/kunit/.gitignore
> @@ -0,0 +1,3 @@
> +# Byte-compiled / optimized / DLL files
> +__pycache__/
> +*.py[cod]
> \ No newline at end of file
> diff --git a/tools/testing/kunit/kunit_config.py b/tools/testing/kunit/kunit_config.py
> new file mode 100644
> index 0000000000000..183bd5e758762
> --- /dev/null
> +++ b/tools/testing/kunit/kunit_config.py
> @@ -0,0 +1,60 @@
> +# SPDX-License-Identifier: GPL-2.0
> +
> +import collections
> +import re
> +
> +CONFIG_IS_NOT_SET_PATTERN = r'^# CONFIG_\w+ is not set$'
> +CONFIG_PATTERN = r'^CONFIG_\w+=\S+$'
> +
> +KconfigEntryBase = collections.namedtuple('KconfigEntry', ['raw_entry'])
> +
> +
> +class KconfigEntry(KconfigEntryBase):
> +
> + def __str__(self) -> str:
> + return self.raw_entry
> +
> +
> +class KconfigParseError(Exception):
> + """Error parsing Kconfig defconfig or .config."""
> +
> +
> +class Kconfig(object):
> + """Represents defconfig or .config specified using the Kconfig language."""
> +
> + def __init__(self):
> + self._entries = []
> +
> + def entries(self):
> + return set(self._entries)
> +
> + def add_entry(self, entry: KconfigEntry) -> None:
> + self._entries.append(entry)
> +
> + def is_subset_of(self, other: "Kconfig") -> bool:
> + return self.entries().issubset(other.entries())
> +
> + def write_to_file(self, path: str) -> None:
> + with open(path, 'w') as f:
> + for entry in self.entries():
> + f.write(str(entry) + '\n')
> +
> + def parse_from_string(self, blob: str) -> None:
> + """Parses a string containing KconfigEntrys and populates this Kconfig."""
> + self._entries = []
> + is_not_set_matcher = re.compile(CONFIG_IS_NOT_SET_PATTERN)
> + config_matcher = re.compile(CONFIG_PATTERN)
> + for line in blob.split('\n'):
> + line = line.strip()
> + if not line:
> + continue
> + elif config_matcher.match(line) or is_not_set_matcher.match(line):
> + self._entries.append(KconfigEntry(line))
> + elif line[0] == '#':
> + continue
> + else:
> + raise KconfigParseError('Failed to parse: ' + line)
> +
> + def read_from_file(self, path: str) -> None:
> + with open(path, 'r') as f:
> + self.parse_from_string(f.read())
> diff --git a/tools/testing/kunit/kunit_kernel.py b/tools/testing/kunit/kunit_kernel.py
> new file mode 100644
> index 0000000000000..bba7ea7ca1869
> --- /dev/null
> +++ b/tools/testing/kunit/kunit_kernel.py
> @@ -0,0 +1,126 @@
> +# SPDX-License-Identifier: GPL-2.0
> +
> +import logging
> +import subprocess
> +import os
> +
> +import kunit_config
> +
> +KCONFIG_PATH = '.config'
> +
> +class ConfigError(Exception):
> + """Represents an error trying to configure the Linux kernel."""
> +
> +
> +class BuildError(Exception):
> + """Represents an error trying to build the Linux kernel."""
> +
> +
> +class LinuxSourceTreeOperations(object):
> + """An abstraction over command line operations performed on a source tree."""
> +
> + def make_mrproper(self):
> + try:
> + subprocess.check_output(['make', 'mrproper'])
> + except OSError as e:
> + raise ConfigError('Could not call make command: ' + e)
> + except subprocess.CalledProcessError as e:
> + raise ConfigError(e.output)
> +
> + def make_olddefconfig(self):
> + try:
> + subprocess.check_output(['make', 'ARCH=um', 'olddefconfig'])
> + except OSError as e:
> + raise ConfigError('Could not call make command: ' + e)
> + except subprocess.CalledProcessError as e:
> + raise ConfigError(e.output)
> +
> + def make(self, jobs):
> + try:
> + subprocess.check_output([
> + 'make',
> + 'ARCH=um',
> + '--jobs=' + str(jobs)])
Perhaps as a future extension:
It would be nice if we could set an O= here to keep the source tree
pristine.
In fact I might even suggest that this should always be set so that the
unittesting could live along side an existing kernel build? :
O ?= $KBUILD_SRC/
O := $(O)/kunittest/$(ARCH)/build
> + except OSError as e:
> + raise BuildError('Could not call execute make: ' + e)
> + except subprocess.CalledProcessError as e:
> + raise BuildError(e.output)
> +
> + def linux_bin(self, params, timeout):
> + """Runs the Linux UML binary. Must be named 'linux'."""
> + process = subprocess.Popen(
> + ['./linux'] + params,
> + stdin=subprocess.PIPE,
> + stdout=subprocess.PIPE,
> + stderr=subprocess.PIPE)
> + process.wait(timeout=timeout)
> + return process
> +
> +
> +class LinuxSourceTree(object):
> + """Represents a Linux kernel source tree with KUnit tests."""
> +
> + def __init__(self):
> + self._kconfig = kunit_config.Kconfig()
> + self._kconfig.read_from_file('kunitconfig')
> + self._ops = LinuxSourceTreeOperations()
> +
> + def clean(self):
> + try:
> + self._ops.make_mrproper()
> + except ConfigError as e:
> + logging.error(e)
> + return False
> + return True
> +
> + def build_config(self):
> + self._kconfig.write_to_file(KCONFIG_PATH)
> + try:
> + self._ops.make_olddefconfig()
> + except ConfigError as e:
> + logging.error(e)
> + return False
> + validated_kconfig = kunit_config.Kconfig()
> + validated_kconfig.read_from_file(KCONFIG_PATH)
> + if not self._kconfig.is_subset_of(validated_kconfig):
> + logging.error('Provided Kconfig is not contained in validated .config!')
> + return False
> + return True
> +
> + def build_reconfig(self):
> + """Creates a new .config if it is not a subset of the kunitconfig."""
> + if os.path.exists(KCONFIG_PATH):
> + existing_kconfig = kunit_config.Kconfig()
> + existing_kconfig.read_from_file(KCONFIG_PATH)
> + if not self._kconfig.is_subset_of(existing_kconfig):
> + print('Regenerating .config ...')
> + os.remove(KCONFIG_PATH)
> + return self.build_config()
> + else:
> + return True
> + else:
> + print('Generating .config ...')
> + return self.build_config()
> +
> + def build_um_kernel(self, jobs):
> + try:
> + self._ops.make_olddefconfig()
> + self._ops.make(jobs)
> + except (ConfigError, BuildError) as e:
> + logging.error(e)
> + return False
> + used_kconfig = kunit_config.Kconfig()
> + used_kconfig.read_from_file(KCONFIG_PATH)
> + if not self._kconfig.is_subset_of(used_kconfig):
> + logging.error('Provided Kconfig is not contained in final config!')
> + return False
> + return True
> +
> + def run_kernel(self, args=[]):
> + timeout = None
> + args.extend(['mem=256M'])
> + process = self._ops.linux_bin(args, timeout)
> + with open('test.log', 'w') as f:
> + for line in process.stdout:
> + f.write(line.rstrip().decode('ascii') + '\n')
> + yield line.rstrip().decode('ascii')
>
--
Regards
--
Kieran
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm
next prev parent reply other threads:[~2018-11-29 13:54 UTC|newest]
Thread overview: 118+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-11-28 19:36 [RFC v3 00/19] kunit: introduce KUnit, the Linux kernel unit testing framework Brendan Higgins
2018-11-28 19:36 ` [RFC v3 01/19] kunit: test: add KUnit test runner core Brendan Higgins
2018-11-30 3:14 ` Luis Chamberlain
2018-12-01 1:51 ` Brendan Higgins
2018-12-01 2:57 ` Luis Chamberlain
2018-12-05 13:15 ` Anton Ivanov
2018-12-05 14:45 ` Arnd Bergmann
2018-12-05 14:49 ` Anton Ivanov
2018-11-30 3:28 ` Luis Chamberlain
[not found] ` <20181130032802.GG18410-dAjH6bxAqesAS62YNPtMr3dQhYtBYE6JAL8bYrjMMd8@public.gmane.org>
2018-12-01 2:08 ` Brendan Higgins
2018-12-01 3:10 ` Luis Chamberlain
[not found] ` <20181201031049.GL28501-dAjH6bxAqesAS62YNPtMr3dQhYtBYE6JAL8bYrjMMd8@public.gmane.org>
2018-12-03 22:47 ` Brendan Higgins
2018-12-01 3:02 ` Luis Chamberlain
2018-11-28 19:36 ` [RFC v3 10/19] kunit: test: add test managed resource tests Brendan Higgins
[not found] ` <20181128193636.254378-1-brendanhiggins-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org>
2018-11-28 19:36 ` [RFC v3 02/19] kunit: test: add test resource management API Brendan Higgins
2018-11-28 19:36 ` [RFC v3 03/19] kunit: test: add string_stream a std::stream like string builder Brendan Higgins
2018-11-30 3:29 ` Luis Chamberlain
2018-12-01 2:14 ` Brendan Higgins
2018-12-01 3:12 ` Luis Chamberlain
2018-12-03 10:55 ` Petr Mladek
2018-12-04 0:35 ` Brendan Higgins
2018-11-28 19:36 ` [RFC v3 04/19] kunit: test: add test_stream a std::stream like logger Brendan Higgins
2018-11-28 19:36 ` [RFC v3 05/19] kunit: test: add the concept of expectations Brendan Higgins
2018-11-28 19:36 ` [RFC v3 06/19] arch: um: enable running kunit from User Mode Linux Brendan Higgins
2018-11-28 21:26 ` Rob Herring
2018-11-30 3:37 ` Luis Chamberlain
2018-11-30 14:05 ` Rob Herring
2018-11-30 18:22 ` Luis Chamberlain
[not found] ` <20181130182203.GS18410-dAjH6bxAqesAS62YNPtMr3dQhYtBYE6JAL8bYrjMMd8@public.gmane.org>
2018-12-03 23:22 ` Brendan Higgins
2018-11-30 3:30 ` Luis Chamberlain
2018-11-28 19:36 ` [RFC v3 07/19] kunit: test: add initial tests Brendan Higgins
2018-11-30 3:40 ` Luis Chamberlain
2018-12-03 23:26 ` Brendan Higgins
2018-12-03 23:43 ` Luis Chamberlain
2018-11-28 19:36 ` [RFC v3 08/19] arch: um: add shim to trap to allow installing a fault catcher for tests Brendan Higgins
2018-11-30 3:34 ` Luis Chamberlain
[not found] ` <20181130033429.GK18410-dAjH6bxAqesAS62YNPtMr3dQhYtBYE6JAL8bYrjMMd8@public.gmane.org>
2018-12-03 23:34 ` Brendan Higgins
[not found] ` <CAFd5g45+MAVaSW8HN9x57Y8Um=TV1Oa=-K8yExPBS-4KjLyciQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2018-12-03 23:46 ` Luis Chamberlain
[not found] ` <20181203234628.GR28501-dAjH6bxAqesAS62YNPtMr3dQhYtBYE6JAL8bYrjMMd8@public.gmane.org>
2018-12-04 0:44 ` Brendan Higgins
2018-11-30 3:41 ` Luis Chamberlain
2018-12-03 23:37 ` Brendan Higgins
2018-11-28 19:36 ` [RFC v3 09/19] kunit: test: add the concept of assertions Brendan Higgins
2018-11-28 19:36 ` [RFC v3 11/19] kunit: add Python libraries for handing KUnit config and kernel Brendan Higgins
2018-11-29 13:54 ` Kieran Bingham [this message]
[not found] ` <841cf4ae-501b-05ae-5863-a51010709b67-ryLnwIuWjnjg/C1BVhZhaw@public.gmane.org>
2018-12-03 23:48 ` Brendan Higgins
2018-12-04 20:47 ` Luis Chamberlain
2018-12-06 12:32 ` Kieran Bingham
2018-12-06 15:37 ` Matthew Wilcox
2018-12-07 11:30 ` Kieran Bingham
2018-12-11 14:09 ` Petr Mladek
2018-12-11 14:41 ` Steven Rostedt
2018-12-11 17:01 ` Anton Ivanov
2019-02-09 0:40 ` Brendan Higgins
2018-12-07 1:05 ` Luis Chamberlain
2018-12-07 18:35 ` Kent Overstreet
2018-11-30 3:44 ` Luis Chamberlain
2018-12-03 23:50 ` Brendan Higgins
2018-12-04 20:48 ` Luis Chamberlain
2018-11-28 19:36 ` [RFC v3 12/19] kunit: add KUnit wrapper script and simple output parser Brendan Higgins
2018-11-28 19:36 ` [RFC v3 13/19] kunit: improve output from python wrapper Brendan Higgins
2018-11-28 19:36 ` [RFC v3 14/19] Documentation: kunit: add documentation for KUnit Brendan Higgins
2018-11-29 13:56 ` Kieran Bingham
2018-11-30 3:45 ` Luis Chamberlain
[not found] ` <20181130034525.GP18410-dAjH6bxAqesAS62YNPtMr3dQhYtBYE6JAL8bYrjMMd8@public.gmane.org>
2018-12-03 23:53 ` Brendan Higgins
2018-12-06 12:16 ` Kieran Bingham
2019-02-09 0:56 ` Brendan Higgins
2019-02-11 12:16 ` Kieran Bingham
2019-02-12 22:10 ` Brendan Higgins
2019-02-13 21:55 ` Kieran Bingham
2019-02-14 0:17 ` Brendan Higgins
2019-02-14 17:26 ` Luis Chamberlain
2019-02-14 22:07 ` Brendan Higgins
2018-11-28 19:36 ` [RFC v3 15/19] MAINTAINERS: add entry for KUnit the unit testing framework Brendan Higgins
2018-11-28 19:36 ` [RFC v3 17/19] of: unittest: migrate tests to run on KUnit Brendan Higgins
2018-11-28 20:56 ` Rob Herring
2018-11-30 0:39 ` Randy Dunlap
[not found] ` <18814973-8f0a-4647-a097-fcc3dc0b3cd3-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>
2018-12-04 0:13 ` Brendan Higgins
2018-12-04 13:40 ` Rob Herring
[not found] ` <CAL_JsqL_PivQbrJFEusdKAy-2EQtKL3OHbmyYSK9bzuTOQegqA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2018-12-05 23:42 ` Brendan Higgins
2018-12-07 0:41 ` Rob Herring
2018-12-04 0:08 ` Brendan Higgins
2019-02-13 1:44 ` Brendan Higgins
2019-02-14 20:10 ` Rob Herring
2019-02-14 21:52 ` Brendan Higgins
2019-02-18 22:56 ` Frank Rowand
2019-02-28 0:29 ` Brendan Higgins
2018-12-04 10:56 ` Frank Rowand
2018-11-28 19:36 ` [RFC v3 18/19] of: unittest: split out a couple of test cases from unittest Brendan Higgins
2018-12-04 10:58 ` Frank Rowand
2018-12-05 23:54 ` Brendan Higgins
2019-02-14 23:57 ` Frank Rowand
2019-02-15 0:56 ` Brendan Higgins
2019-02-15 2:05 ` Frank Rowand
2019-02-15 10:56 ` Brendan Higgins
2019-02-18 22:25 ` Frank Rowand
2019-02-20 20:44 ` Frank Rowand
2019-02-20 20:47 ` Frank Rowand
2019-02-28 3:52 ` Brendan Higgins
2019-03-22 0:22 ` Frank Rowand
2019-03-22 1:30 ` Brendan Higgins
2019-03-22 1:47 ` Frank Rowand
2019-03-25 22:15 ` Brendan Higgins
2019-09-20 16:57 ` Rob Herring
2019-09-21 23:57 ` Frank Rowand
2019-03-22 1:34 ` Frank Rowand
2019-03-25 22:18 ` Brendan Higgins
2018-11-28 19:36 ` [RFC v3 19/19] of: unittest: split up some super large test cases Brendan Higgins
2018-11-28 19:36 ` [RFC v3 16/19] arch: um: make UML unflatten device tree when testing Brendan Higgins
2018-11-28 21:16 ` Rob Herring
[not found] ` <CAL_JsqK5cG=QzMBFSZ31_-3ujnxqxv=jj3XYajbRLT7yWYZbfw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2018-12-04 0:00 ` Brendan Higgins
2018-11-30 3:46 ` Luis Chamberlain
2018-12-04 0:02 ` Brendan Higgins
2018-12-04 10:52 ` [RFC v3 00/19] kunit: introduce KUnit, the Linux kernel unit testing framework Frank Rowand
2018-12-04 11:40 ` Frank Rowand
2018-12-04 13:49 ` Rob Herring
2018-12-05 23:10 ` Brendan Higgins
2019-03-22 0:27 ` Frank Rowand
2019-03-25 22:04 ` Brendan Higgins
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=841cf4ae-501b-05ae-5863-a51010709b67@ideasonboard.com \
--to=kieran.bingham@ideasonboard.com \
--cc=Tim.Bird@sony.com \
--cc=brakmo@fb.com \
--cc=brendanhiggins@google.com \
--cc=daniel@ffwll.ch \
--cc=dri-devel@lists.freedesktop.org \
--cc=felixguoxiuping@gmail.com \
--cc=frowand.list@gmail.com \
--cc=gregkh@linuxfoundation.org \
--cc=jdike@addtoit.com \
--cc=joe@perches.com \
--cc=joel@jms.id.au \
--cc=julia.lawall@lip6.fr \
--cc=keescook@google.com \
--cc=khilman@baylibre.com \
--cc=knut.omang@oracle.com \
--cc=kunit-dev@googlegroups.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-kselftest@vger.kernel.org \
--cc=linux-nvdimm@lists.01.org \
--cc=linux-um@lists.infradead.org \
--cc=mcgrof@kernel.org \
--cc=mpe@ellerman.id.au \
--cc=richard@nod.at \
--cc=robh@kernel.org \
--cc=rostedt@goodmis.org \
--cc=shuah@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).