Re: [LSF/MM TOPIC] improving storage testing

[Omar Sandoval <osandov@osandov.com>]

On Thu, Feb 14, 2019 at 01:10:40PM +0100, Lukas Czerner wrote:
> On Wed, Feb 13, 2019 at 01:07:54PM -0500, Theodore Y. Ts'o wrote:
> > 
> > 2) Documenting what are known failures should be for various tests on
> > different file systems and kernel versions.  I think we all have our
> > own way of excluding tests which are known to fail.  One extreme case
> > is where the test case was added to xfstests (generic/484), but the
> > patch to fix it got hung up because it was somewhat controversial, so
> > it was failing on all file systems.
> > 
> > Other cases might be when fixing a particular test failure is too
> > complex to backport to stable (maybe because it would drag in all
> > sorts of other changes in other subsystems), so that test is Just
> > Going To Fail for a particular stable kernel series.
> > 
> > It probably doesn't make sense to do this in xfstests, which is why we
> > all have our own individual test runners that are layered on top of
> > xfstests.  But if we want to automate running xfstests for stable
> > kernel series, some way of annotating fixes for different kernel
> > versions would be useful, perhaps some kind of centralized clearing
> > house of this information would be useful.
> 
> I think that the first step can be to require the new test to go in
> "after" the respective kernel fix. And related to that, require the test
> to include a well-defined tag (preferably both in the test itself and
> commit description) saying which commit fixed this particular problem.

For blktests, I require that regression tests include what commit they
are testing in the test comment. For xfstests, sometimes the test
mentions it, sometimes the commit mentions it, but more often you have
to search for keywords in kernel commit messages... It'd be great if
xfstests also required that the test file mentioned the commit/patch it
tests.