[LSF/MM TOPIC] [ATTEND] xfstests: what do we need to do to make it better?

[LSF/MM TOPIC] [ATTEND] xfstests: what do we need to do to make it better?

[Dave Chinner]

Given that more people are using xfstests and developing tests, we
need to consider how to make it friendlier to hack on. The current
structure of the tree is difficult to work with, the way tests are
organised and numbered make it difficult to co-ordinate new tests
and results in patch conflicts, etc.

We also see problems arising from people not really understanding how
the xfstests harness is designed and how it really is supposed to
work, so an overview of the underlying principles of operation would
probably be helpful to a lot of people. It will also save
review and rework time if we can avoid having people make the same
mistakes the first time they submit tests....

I'd also like to discuss some potential infrastructure changes to
make it easier to add new tests without conflicts with others
developing new tests. Some of the ideas Christoph and I have
previously tossed around include:

	- break tests up into groups in their own subdirectories.
	  e.g. generic tests, xfs/ext4/btrfs specific tests, stress
	  tests, performance tests, large FS tests, etc
	- change the way we define groups of tests so we don't have
	  a single registry of tests and their groups
	- allow different naming of tests, such as desciptive text
	  names rather than just plain numbers
	- allow duplicate test names in different groups

I'm sure that other users of xfstests will have some ideas on how to
improve it for the way they run it, so I'd like to gather and
incorporate these ideas into any structural change we make to
xfstests.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: [LSF/MM TOPIC] [ATTEND] xfstests: what do we need to do to make it better?

[Alex Elder]

On Wed, 2012-01-04 at 10:44 +1100, Dave Chinner wrote:
> Given that more people are using xfstests and developing tests, we
> need to consider how to make it friendlier to hack on. The current
> structure of the tree is difficult to work with, the way tests are
> organised and numbered make it difficult to co-ordinate new tests
> and results in patch conflicts, etc.

Coordination of numbers is not a big deal, the test names/numbers
can be easily fixed up at commit time.  I also thought that the
numbers--though meaningless on their own--also avoided having to
decide where a particular test belongs.  I.e., a test that exercises
several categories of things (maybe preallocation, quota, and ENOSPC)
won't be hidden in any sort of "enospc" test directory.

I do think the growing number of tests is making it a bit unwieldy
though, so I think some sort of reorganization is a good plan.

> We also see problems arising from people not really understanding how
> the xfstests harness is designed and how it really is supposed to
> work, so an overview of the underlying principles of operation would
> probably be helpful to a lot of people. It will also save
> review and rework time if we can avoid having people make the same
> mistakes the first time they submit tests....

This is very important.  And the gist of it ought to be
captured somewhere if it is not already.

> I'd also like to discuss some potential infrastructure changes to
> make it easier to add new tests without conflicts with others
> developing new tests. Some of the ideas Christoph and I have
> previously tossed around include:
> 
> 	- break tests up into groups in their own subdirectories.
> 	  e.g. generic tests, xfs/ext4/btrfs specific tests, stress
> 	  tests, performance tests, large FS tests, etc
> 	- change the way we define groups of tests so we don't have
> 	  a single registry of tests and their groups
> 	- allow different naming of tests, such as desciptive text
> 	  names rather than just plain numbers
> 	- allow duplicate test names in different groups

Despite what I said above, I don't disagree with any of this.
Perhaps the tests can be buried in one or more subdirectories,
but each FSTYP defines its own groups file to drive testing.

> I'm sure that other users of xfstests will have some ideas on how to
> improve it for the way they run it, so I'd like to gather and
> incorporate these ideas into any structural change we make to
> xfstests.

Should be a good discussion.  It might be useful to have a
proposal or two to work with as a starting point, or maybe
an outline of the types of changes (naming, directory
structure, etc.), to help keep things focused.

					-Alex

> Cheers,
> 
> Dave.

Re: [LSF/MM TOPIC] [ATTEND] xfstests: what do we need to do to make it better?

[Dave Chinner]

On Wed, Jan 04, 2012 at 11:18:11AM -0600, Alex Elder wrote:
> On Wed, 2012-01-04 at 10:44 +1100, Dave Chinner wrote:
> > Given that more people are using xfstests and developing tests, we
> > need to consider how to make it friendlier to hack on. The current
> > structure of the tree is difficult to work with, the way tests are
> > organised and numbered make it difficult to co-ordinate new tests
> > and results in patch conflicts, etc.
> 
> Coordination of numbers is not a big deal, the test names/numbers
> can be easily fixed up at commit time.  I also thought that the
> numbers--though meaningless on their own--also avoided having to
> decide where a particular test belongs.

Instead, we decide that via the global group file and via
definitions within the test itself. Yes, the global group file
allows tests to exist in multiple logical test groups, but that's
something that canbe maintained even with a broken up deirectory
heirarchy.

> I.e., a test that exercises
> several categories of things (maybe preallocation, quota, and ENOSPC)
> won't be hidden in any sort of "enospc" test directory.

No, but it will be a generic of filesystem specific test, so woul
dbe located in such a subdirectory.

Basically, I'm thinking of something more fine grained like this:

	generic/group
	       /001
	       /005
	       ....
	xfs/group
	   /002
	   /003
	   ....

So that we're not treading on each other's toes as much. Maintaining
the group files still allows logical grouping of tests (like auto,
quota, etc) so I don't see any change in functionality there. Maybe
someone will have a good idea to improve the logical grouping
implementation, but right now I'm not planning on killing that
functionality at all...

> > make it easier to add new tests without conflicts with others
> > developing new tests. Some of the ideas Christoph and I have
> > previously tossed around include:
> > 
> > 	- break tests up into groups in their own subdirectories.
> > 	  e.g. generic tests, xfs/ext4/btrfs specific tests, stress
> > 	  tests, performance tests, large FS tests, etc
> > 	- change the way we define groups of tests so we don't have
> > 	  a single registry of tests and their groups
> > 	- allow different naming of tests, such as desciptive text
> > 	  names rather than just plain numbers
> > 	- allow duplicate test names in different groups
> 
> Despite what I said above, I don't disagree with any of this.
> Perhaps the tests can be buried in one or more subdirectories,
> but each FSTYP defines its own groups file to drive testing.

I think allowing duplicate tests is a bad thing - keeping the
logical grouping effectively allows us to only keep one copy of a
test no matter where it ends up.

> > I'm sure that other users of xfstests will have some ideas on how to
> > improve it for the way they run it, so I'd like to gather and
> > incorporate these ideas into any structural change we make to
> > xfstests.
> 
> Should be a good discussion.  It might be useful to have a
> proposal or two to work with as a starting point, or maybe
> an outline of the types of changes (naming, directory
> structure, etc.), to help keep things focused.

Sure, though I don't want to done so much that it's all set in stone
before I found out what other people consider pain points that need
to be fixed...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com