Linux-ext4 Archive on lore.kernel.org
 help / color / Atom feed
From: <Tim.Bird@sony.com>
To: <yzaikin@google.com>
Cc: <skhan@linuxfoundation.org>, <tytso@mit.edu>,
	<brendanhiggins@google.com>, <linux-kselftest@vger.kernel.org>,
	<linux-ext4@vger.kernel.org>, <adilger.kernel@dilger.ca>,
	<kunit-dev@googlegroups.com>
Subject: RE: [PATCH linux-kselftest/test v2] ext4: add kunit test for decoding extended timestamps
Date: Fri, 18 Oct 2019 00:38:44 +0000
Message-ID: <ECADFF3FD767C149AD96A924E7EA6EAF977D013C@USCULXMSG01.am.sony.com> (raw)
In-Reply-To: <CAAXuY3oS=fzH0hpdjUpp_tUyypfAs=TaJxtw9L2=feUkLH2sUA@mail.gmail.com>

> -----Original Message-----
> From: Iurii Zaikin
> 
> On Thu, Oct 17, 2019 at 4:54 PM <Tim.Bird@sony.com> wrote:
> >
> > > -----Original Message-----
> > > From: Iurii Zaikin
> > >
> > > > You can do all of this and allow users to supply another set of data.
> > > > It doesn't gave to be one or the other.
> > > >
> > > What is the use case for running a unit test on a different data set than
> > > what it comes with?
> >
> > I just gave some ideas in another message (our emails crossed),
> > but one use case is to allow someone besides the test author
> > to inject additional data points, and to do so without having to re-compile
> > the code.
> >
> > They might do this for multiple reasons:
> >  - to experiment with additional data points
> >  - to try to diagnose a problem they are seeing
> >  - to fill gaps they see in existing data points
> >
> > Whether this makes sense depends on a lot of factors.  I suspect
> > the timestamp test code is not a good candidate for this, as the code
> > is simple enough that adding a new test case is pretty trivial.  For some
> > other types of tests, adding the data via an external file could be easier
> > than changing the code of the test.
> 
> I think feeding test data without recompiling looks attractive right now
> because
> in order to run a single test you need to compile and link the whole kernel.
> I believe KUnit's strategic goal is the ability to only compile the
> relevant bits,
> which is admittedly very far off.
> Normally, in application programming the amount of code that needs to be
> recompiled in order to run a test suite is small enough that the added
> complexity
> of enabling the test to get the data from external sources is not
> warranted. Typically,
> external files are used when something is not practical to include in
> the source file
> directly due to size or complexity, i.e. a large snippet of text, an
> image file, some
> binary data etc. Such needs are typically addressed by the test author
> rather than
> the core test framework.
> Now, in application programming you can do a lot of things like
> reading a file which
> is trickier in kernel. But again we've come to supporting a use case
> for test data which
> has to be fabricated through some involved process or otherwise not
> easily included
> in the source file.
I strongly agree with everything you say here.

> And if you come up with an additional test case, why not just add it
> and leave it there?
You should do exactly that.   But the part you glossed over is the 
"coming up with an additional test case" part.

Having a data-driven test can, in some circumstances, allow one
to more easily come up with additional interesting test cases.
This is where Ted is exactly right about fuzzers.  You don't want to
execute a fuzzer as part of your unit testing.  But you might want to 
execute a fuzzer to come up with additional unit test cases to add.
And fuzzers are easier to write for data files than for C code.
(I run a grave risk of de-railing the conversation by referring back to fuzzers,
just when I believe we are coming to agreement about a number of ideas. :-).
Just to be clear, I'm not promoting fuzzers for unit testing.

Regards,
 -- Tim

> Unit tests are cheap, even if a case proves to be redundant, the mere
> fact that the
> code under test made you think of such a case is sufficient to
> permanently include
> the test case into the test suite.



  reply index

Thread overview: 34+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-10-10  2:39 Iurii Zaikin
2019-10-10  3:46 ` Tim.Bird
2019-10-10 16:45   ` Iurii Zaikin
2019-10-10 20:29     ` Tim.Bird
2019-10-10 23:49       ` Iurii Zaikin
2019-10-10 17:11 ` Shuah Khan
2019-10-10 22:13   ` Iurii Zaikin
2019-10-11 10:05     ` Brendan Higgins
2019-10-11 13:19       ` Theodore Y. Ts'o
2019-10-12  2:38         ` Iurii Zaikin
2019-10-16 22:18         ` Brendan Higgins
2019-10-16 23:26           ` Shuah Khan
2019-10-17  0:07             ` Iurii Zaikin
2019-10-17 12:08             ` Theodore Y. Ts'o
2019-10-17 22:25               ` Tim.Bird
2019-10-17 22:56                 ` Theodore Y. Ts'o
2019-10-17 23:40                   ` Tim.Bird
2019-10-18  1:40                     ` Theodore Y. Ts'o
2019-10-18  2:40                       ` Tim.Bird
2019-10-18 15:27                         ` Theodore Y. Ts'o
2019-10-18 20:24                           ` Shuah Khan
2019-10-24  1:30                             ` Brendan Higgins
2019-10-18  1:12                 ` Brendan Higgins
2019-10-18  1:30                   ` Tim.Bird
2019-10-17 22:49               ` Shuah Khan
2019-10-17 23:07                 ` Iurii Zaikin
2019-10-17 23:12                   ` Shuah Khan
2019-10-17 23:27                     ` Iurii Zaikin
2019-10-17 23:42                       ` Shuah Khan
2019-10-17 23:54                       ` Tim.Bird
2019-10-17 23:59                         ` Shuah Khan
2019-10-18  0:11                         ` Iurii Zaikin
2019-10-18  0:38                           ` Tim.Bird [this message]
2019-10-18  1:06                             ` Iurii Zaikin

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=ECADFF3FD767C149AD96A924E7EA6EAF977D013C@USCULXMSG01.am.sony.com \
    --to=tim.bird@sony.com \
    --cc=adilger.kernel@dilger.ca \
    --cc=brendanhiggins@google.com \
    --cc=kunit-dev@googlegroups.com \
    --cc=linux-ext4@vger.kernel.org \
    --cc=linux-kselftest@vger.kernel.org \
    --cc=skhan@linuxfoundation.org \
    --cc=tytso@mit.edu \
    --cc=yzaikin@google.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-ext4 Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-ext4/0 linux-ext4/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-ext4 linux-ext4/ https://lore.kernel.org/linux-ext4 \
		linux-ext4@vger.kernel.org
	public-inbox-index linux-ext4

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-ext4


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git