Linux-kselftest Archive on lore.kernel.org
 help / color / Atom feed
From: "Theodore Y. Ts'o" <tytso@mit.edu>
To: Tim.Bird@sony.com
Cc: skhan@linuxfoundation.org, brendanhiggins@google.com,
	yzaikin@google.com, linux-kselftest@vger.kernel.org,
	linux-ext4@vger.kernel.org, adilger.kernel@dilger.ca,
	kunit-dev@googlegroups.com
Subject: Re: [PATCH linux-kselftest/test v2] ext4: add kunit test for decoding extended timestamps
Date: Fri, 18 Oct 2019 11:27:46 -0400
Message-ID: <20191018152746.GF21137@mit.edu> (raw)
In-Reply-To: <ECADFF3FD767C149AD96A924E7EA6EAF977D01DC@USCULXMSG01.am.sony.com>

On Fri, Oct 18, 2019 at 02:40:50AM +0000, Tim.Bird@sony.com wrote:
> We're just talking past each other.  My original e-mail was a rebuttal
> to your assertion that any test that was data-driven or non-deterministic
> was a fuzzer.  I still believe that's just not the case.  This is independent
> of the mechanics or speed of how the data is input.

Apologies, I was still focused on the original context of this thread,
which was about suggested improvements to Iurii's ext4 kunit test, or
perhaps adding new features to Kunit.

> I also conceded (multiple times) that externally data-driven
> techniques are probably more aptly applied to non-unit tests. I've
> heard your pitch about speed, and I'm sympathetic.  My point is that
> I believe there is a place for data-driven tests.

I guess I would put it differently.  The key goal is it should be
really easy for developers to run, create, and extend tests.
Data-driven tests is certainly one technique to make it easier to
extend tests, and indeed fs/ext4/inode-test.c is data-driven with the
goal to make it easier to add additional tests.

Having the data for the test be external is certainly one option, and
there will be cases where it will make sense.  However, the overhead
in creating the parser for the data, and additional complexity
required to get the test data to be fed to the test program means that
that benefits need to be pretty large in order to balance the
additional costs of having an external data file, especially for
Kunit.

In terms of the abstract question, is there a place for data-driven
tests, I'm in complete agreement with you.  I've used this many times
personally, especially when writing tests which are implemented in
terms of shell scripts.  Examples of this include e2fsprogs's
regression test suite and xfstests.  I don't consider that a terribly
interesting question though; I view that as on the same order as "is
the sky blue?" or "are apple pies yummy?"

The more interesting, and more concrete question is whether there is a
place for external data-driven tests in Kunit, and there I am *much*
more skeptical.  Sure, I could imagine adding some infrastructure
where user-mode linux could read files from its "host" system.  But
there are those people who want to run KUnit tests by cross-compiling
an ARM kernel and then shipping the resulting kernel to an test board,
and then reading the output from the ARM board's serial console.  In
that case, we can't make the assumption that Kunit tests will be
running under user-mode linux, so the complexity of how we get the
external data file into the KUnit test just went up by another order
of magnitude.  It's going to be ***so*** much simpler if the test data
is embedded in the kernel built by KUnit.  That way, it works both in
the ARCH=um case, and in the "build an ARM kernel and push it to a
test board" case.

Cheers,

					- Ted

  reply index

Thread overview: 34+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-10-10  2:39 Iurii Zaikin
2019-10-10  3:46 ` Tim.Bird
2019-10-10 16:45   ` Iurii Zaikin
2019-10-10 20:29     ` Tim.Bird
2019-10-10 23:49       ` Iurii Zaikin
2019-10-10 17:11 ` Shuah Khan
2019-10-10 22:13   ` Iurii Zaikin
2019-10-11 10:05     ` Brendan Higgins
2019-10-11 13:19       ` Theodore Y. Ts'o
2019-10-12  2:38         ` Iurii Zaikin
2019-10-16 22:18         ` Brendan Higgins
2019-10-16 23:26           ` Shuah Khan
2019-10-17  0:07             ` Iurii Zaikin
2019-10-17 12:08             ` Theodore Y. Ts'o
2019-10-17 22:25               ` Tim.Bird
2019-10-17 22:56                 ` Theodore Y. Ts'o
2019-10-17 23:40                   ` Tim.Bird
2019-10-18  1:40                     ` Theodore Y. Ts'o
2019-10-18  2:40                       ` Tim.Bird
2019-10-18 15:27                         ` Theodore Y. Ts'o [this message]
2019-10-18 20:24                           ` Shuah Khan
2019-10-24  1:30                             ` Brendan Higgins
2019-10-18  1:12                 ` Brendan Higgins
2019-10-18  1:30                   ` Tim.Bird
2019-10-17 22:49               ` Shuah Khan
2019-10-17 23:07                 ` Iurii Zaikin
2019-10-17 23:12                   ` Shuah Khan
2019-10-17 23:27                     ` Iurii Zaikin
2019-10-17 23:42                       ` Shuah Khan
2019-10-17 23:54                       ` Tim.Bird
2019-10-17 23:59                         ` Shuah Khan
2019-10-18  0:11                         ` Iurii Zaikin
2019-10-18  0:38                           ` Tim.Bird
2019-10-18  1:06                             ` Iurii Zaikin

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20191018152746.GF21137@mit.edu \
    --to=tytso@mit.edu \
    --cc=Tim.Bird@sony.com \
    --cc=adilger.kernel@dilger.ca \
    --cc=brendanhiggins@google.com \
    --cc=kunit-dev@googlegroups.com \
    --cc=linux-ext4@vger.kernel.org \
    --cc=linux-kselftest@vger.kernel.org \
    --cc=skhan@linuxfoundation.org \
    --cc=yzaikin@google.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-kselftest Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-kselftest/0 linux-kselftest/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-kselftest linux-kselftest/ https://lore.kernel.org/linux-kselftest \
		linux-kselftest@vger.kernel.org
	public-inbox-index linux-kselftest

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-kselftest


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git