Linux-ext4 Archive on lore.kernel.org
 help / color / Atom feed
From: <Tim.Bird@sony.com>
To: <tytso@mit.edu>
Cc: <skhan@linuxfoundation.org>, <brendanhiggins@google.com>,
	<yzaikin@google.com>, <linux-kselftest@vger.kernel.org>,
	<linux-ext4@vger.kernel.org>, <adilger.kernel@dilger.ca>,
	<kunit-dev@googlegroups.com>
Subject: RE: [PATCH linux-kselftest/test v2] ext4: add kunit test for decoding extended timestamps
Date: Fri, 18 Oct 2019 02:40:50 +0000
Message-ID: <ECADFF3FD767C149AD96A924E7EA6EAF977D01DC@USCULXMSG01.am.sony.com> (raw)
In-Reply-To: <20191018014027.GA21137@mit.edu>



> -----Original Message-----
> From: Theodore Y. Ts'o 
> 
> On Thu, Oct 17, 2019 at 11:40:01PM +0000, Tim.Bird@sony.com wrote:
> >
> > No. Well, the data might be provided at some time independent
> > of the test compilation time, but it would not be made up on the fly.
> > So the data might be provided at run time, but that shouldn't imply
> > that the data is random, or that there is some lengthy fabrication
> > process that happens at test execution time.
> 
> So how would the data be provided?  Via a mounted file system?  There
> is no mounted file system when we're running a kunit test.  One of the
> reasons why kunit is fast is because we're not running init scripts,
> and we're not mounting a file system.
> 
> The fabrication process isn't really lengthy, though.  If I modify
> fs/ext4/inode-test.c to add or remove a test, it takes:
> 
> Elapsed time: 2.672s total, 0.001s configuring, 2.554s building, 0.116s running
> 
> Compare and contrast this with running "kvm-xfstests -c 4k generic/001"
> 
> The actual time to run the test generic/001 is 3 seconds.  But there
> is a 9 second overhead in starting the VM, for a total test time of 12
> seconds.  So sure, with kvm-xfstests I can drop a config file in
> /tmp/kvm-xfstests-tytso, which is mounted as /vtmp using 9p, so you
> could provide "user provided data" via a text file.  But the overhead
> of starting up a full KVM, mounting a file system, starting userspace,
> etc., is 9 seconds.
> 
> Contrast this with 2.5 seconds to recompile and relink
> fs/ext4/inode-test.c into the kunit library.  I wouldn't call that a
> "length fabrication process". 

I'm not sure I understand your point here at all.  I never said that
compiling the code was a lengthy fabrication process.  I said that
I was NOT envisioning a lengthy fabrication process at runtime
for the creation of the external data.  Indeed, I wasn't envisioning fabricating
the data at test runtime at all.  I was trying to clarify that I didn't envision
a human or fuzzer in the loop at test runtime, but apparently I didn't make that clear.
The clarification was based on an assumption I made about what you were asking
in your question.  Maybe that assumption was bad.

> Is it really worth it to add in some
> super-complex way to feed a data text file into a Kunit test, when
> editing the test file and rerunning the test really doesn't take that
> long?
> 
> > In this case, the cost of parsing the data file does add some overhead,
> > but it's not onerous.  I'm not sure how, or whether, kunit handles
> > the issue of reading data from a file at test time.  But it doesn't have
> > to be a file read.  I'm just talking separating data from code.
> 
> It's not the cost of parsing the data file is how to *feed* the data
> file into the test file.  How exactly are we supposed to do it?  9p?
> Some other mounted file system?  That's where all the complexity and
> overhead is going to be.
> 
> > Not necessarily.  Maybe the root privilege example is not a good one.
> > How about a test that probes the kernel config, and executes
> > some variation of the tests based on the config values it detects?
> 
> But that's even easier.  We can put "#ifdef CONFIG_xxx" into the
> fs/ext4/inode-test.c file.  Again, it doesn't take that long to
> recompile and relink the test .c file.
> 
> Apologies, but this really seems like complexity in search of a
> problem....

We're just talking past each other.  My original e-mail was a rebuttal
to your assertion that any test that was data-driven or non-deterministic
was a fuzzer.  I still believe that's just not the case.  This is independent
of the mechanics or speed of how the data is input.

I also conceded (multiple times) that externally data-driven techniques are probably
more aptly applied to non-unit tests. I've heard your pitch about speed, and I'm sympathetic.
My point is that I believe there is a place for data-driven tests.

I can live with having failed to convince you, which I'll put down to my inadequacy to
accurately communicate my ideas. :-)

Regards,
 -- Tim


  reply index

Thread overview: 34+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-10-10  2:39 Iurii Zaikin
2019-10-10  3:46 ` Tim.Bird
2019-10-10 16:45   ` Iurii Zaikin
2019-10-10 20:29     ` Tim.Bird
2019-10-10 23:49       ` Iurii Zaikin
2019-10-10 17:11 ` Shuah Khan
2019-10-10 22:13   ` Iurii Zaikin
2019-10-11 10:05     ` Brendan Higgins
2019-10-11 13:19       ` Theodore Y. Ts'o
2019-10-12  2:38         ` Iurii Zaikin
2019-10-16 22:18         ` Brendan Higgins
2019-10-16 23:26           ` Shuah Khan
2019-10-17  0:07             ` Iurii Zaikin
2019-10-17 12:08             ` Theodore Y. Ts'o
2019-10-17 22:25               ` Tim.Bird
2019-10-17 22:56                 ` Theodore Y. Ts'o
2019-10-17 23:40                   ` Tim.Bird
2019-10-18  1:40                     ` Theodore Y. Ts'o
2019-10-18  2:40                       ` Tim.Bird [this message]
2019-10-18 15:27                         ` Theodore Y. Ts'o
2019-10-18 20:24                           ` Shuah Khan
2019-10-24  1:30                             ` Brendan Higgins
2019-10-18  1:12                 ` Brendan Higgins
2019-10-18  1:30                   ` Tim.Bird
2019-10-17 22:49               ` Shuah Khan
2019-10-17 23:07                 ` Iurii Zaikin
2019-10-17 23:12                   ` Shuah Khan
2019-10-17 23:27                     ` Iurii Zaikin
2019-10-17 23:42                       ` Shuah Khan
2019-10-17 23:54                       ` Tim.Bird
2019-10-17 23:59                         ` Shuah Khan
2019-10-18  0:11                         ` Iurii Zaikin
2019-10-18  0:38                           ` Tim.Bird
2019-10-18  1:06                             ` Iurii Zaikin

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=ECADFF3FD767C149AD96A924E7EA6EAF977D01DC@USCULXMSG01.am.sony.com \
    --to=tim.bird@sony.com \
    --cc=adilger.kernel@dilger.ca \
    --cc=brendanhiggins@google.com \
    --cc=kunit-dev@googlegroups.com \
    --cc=linux-ext4@vger.kernel.org \
    --cc=linux-kselftest@vger.kernel.org \
    --cc=skhan@linuxfoundation.org \
    --cc=tytso@mit.edu \
    --cc=yzaikin@google.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-ext4 Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-ext4/0 linux-ext4/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-ext4 linux-ext4/ https://lore.kernel.org/linux-ext4 \
		linux-ext4@vger.kernel.org
	public-inbox-index linux-ext4

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-ext4


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git