Re: [PATCH RFC] fstests: allow running custom hooks

[Dave Chinner <david@fromorbit.com>]

On Wed, Jul 21, 2021 at 02:23:14AM +0000, Damien Le Moal wrote:
> On 2021/07/21 10:53, Qu Wenruo wrote:
> > On 2021/7/21 上午9:11, Dave Chinner wrote:
> > If you believe your philosophy that every test tool should be a complex
> > mess, you're free to do whatever you always do.
> > 
> > And I can always express my objection just like you.
> > 
> > So, you want to build a complex framework using the simple hook, I would
> > just say NO.
> 
> I think that Dave's opinion is actually the reverse of this: hooks, if well
> designed, can be useful and facilitate adding functionalities to a complex test
> framework. And sure, the hook infrastructure implementation can in itself be
> complex, but if well designed, its use can be, and should be, very simple.
> 
> Implementation is done once. The complexity at that stage matters much less than
> the end result usability. As a general principle, the more time one put in
> design and development, the better the end result. Here, that means hooks being
> useful, flexible, extensible, and reusable.

Yup, that's pretty much what I'm advocating for.

I'm thinking that it is something relatively simple like this:

fstests/tests/hooks
- directory containing library of hook scripts

fstests/hooks/
- directory containing symlinks to hook scripts
- kept under gitignore, as hooks are a runtime configurable state,
  not a global repository configured state.
- symlink names indicate run target and order:
  	$seq.{B,E}.X
  where:	$seq is the test that the hook runs for
  			use "global" for a hook that every test runs
  		{B,E} indicates a test Begin or End hook
		X is an integer indicating run order,
			- 0 being the first script
			- Run in ascending numerical order


Now if you want to add a 'trace-cmd clear' command to run before
the start of a specific test, you do:

$ echo trace-cmd clear > tests/hooks/trace-cmd-clear
$ ln -s tests/hooks/trace-cmd-clear hooks/btrfs-160.B.0

And now when btrfs/160 starts, the Begin hook that you've set up is
run....

If you want to do this for other tests, too, then just add symlinks
for those tests. If you want all tests to run this trace hook, link
global.B.0 to the hook script. Essentially, managing the hooks to
run becomes an exercise in linking and unlinking external hook
scripts.

This means we can add curated hook scripts such as "use trace-cmd to
trace all xfs trace points and then dump the report into
$RESULTS_DIR/$seqres.traces" and hook them into to specific tests.
The setup also allows stacked hook scripts, so we can have multiple
monitoring options running at the same time (e.g. blktrace-scratch +
trace-cmd-xfs) without having to write a custom hook scripts.

> And one of the functionality of the hook setup could be "temporary, external
> hook" for some very special case debugging as you seem to need that. What you
> want to do and what Dave is proposing are not mutually exclusive.

Yup, if you want a one-off throwaway hook script, then just add a
file in to fstests/hooks and either name it as per above or link it
to the test it should hook.

Note: this just addresses the management side of running and
curating hook scripts. There's a whole 'nother discussion about
APIs to be had, but a curated hook library inside fstests is
definitely a viable solution to that problem, too, because then the
hook scripts can be changed at the same time the rest of fstests is
changed to use the modified API....

This isn't a huge amount of work to implement, but we really need to
decide how these hooks are going to be maintained, managed and
curated before we go much further...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com