Re: [LSF/MM TOPIC] FS, MM, and stable trees

[Luis Chamberlain <mcgrof@kernel.org>]

On Wed, Mar 09, 2022 at 04:19:21PM -0500, Josef Bacik wrote:
> On Wed, Mar 09, 2022 at 11:00:49AM -0800, Luis Chamberlain wrote:
> > On Wed, Mar 09, 2022 at 01:49:18PM -0500, Josef Bacik wrote:
> > > On Wed, Mar 09, 2022 at 10:41:53AM -0800, Luis Chamberlain wrote:
> > > > On Tue, Mar 08, 2022 at 11:40:18AM -0500, Theodore Ts'o wrote:
> > > > > One of my team members has been working with Darrick to set up a set
> > > > > of xfs configs[1] recommended by Darrick, and she's stood up an
> > > > > automated test spinner using gce-xfstests which can watch a git branch
> > > > > and automatically kick off a set of tests whenever it is updated.
> > > > 
> > > > I think its important to note, as we would all know, that contrary to
> > > > most other subsystems, in so far as blktests and fstests is concerned,
> > > > simply passing a test once does not mean there is no issue given that
> > > > some test can fail with a failure rate of 1/1,000 for instance.
> > > > 
> > > 
> > > FWIW we (the btrfs team) have been running nightly runs of fstests against our
> > > devel branch for over a year and tracking the results.
> > 
> > That's wonderful, what is your steady state goal? And do you have your
> > configurations used public and also your baseline somehwere? I think
> > this later aspect could be very useful to everyone.
> > 
> 
> Yeah I post the results to http://toxicpanda.com, you can see the results from
> the runs, and http://toxicpanda.com/performance/ has the nightly performance
> numbers and graphs as well.

That's great!

But although this runs nightly, it seems this runs fstest *once* to
ensure if there are no regressions. Is that right?

> This was all put together to build into something a little more polished, but
> clearly priorities being what they are this is as far as we've taken it.  For
> configuration you can see my virt-scripts here
> https://github.com/josefbacik/virt-scripts which are what I use to generate the
> VM's to run xfstests in.
> 
> The kernel config I use is in there, I use a variety of btrfs mount options and
> mkfs options, not sure how interesting those are for people outside of btrfs.

Extremely useful.

> Right now I have a box with ZNS drives waiting for me to set this up on so that
> we can also be testing btrfs zoned support nightly, as well as my 3rd
> RaspberryPi that I'm hoping doesn't blow up this time.

Great to hear you will be covering ZNS as well.

> I have another virt setup that uses btrfs snapshots to create a one off chroot
> to run smoke tests for my development using virtme-run.  I want to replace the
> libvirtd vms with virtme-run, however I've got about a 2x performance difference
> between virtme-run and libvirtd that I'm trying to figure out, so right now all
> the nightly test VM's are using libvirtd.
> 
> Long, long term the plan is to replace my janky home setup with AWS VM's that
> can be fired from GitHub actions whenever we push branches, that way individual
> developers can get results for their patches before they're merged, and we don't
> have to rely on my terrible python+html for test results.

If you do move to AWS just keep in mind using loopback drives +
truncated files *finds* more issues than not. So when I used AWS
I got two spare nvme drives and used one to stuff the truncated
files there.

> > Yes, everyone's test setup can be different, but this is why I went with
> > a loopback/truncated file setup, it does find more issues and so far
> > these have all been real.
> > 
> > It kind of begs the question if we should adopt something like kconfig
> > on fstests to help enable a few test configs we can agree on. Thoughts?
> > 
> 
> For us (and I imagine other fs'es) the kconfigs are not interesting, it's the
> combo of different file system features that can be toggled on and off via mkfs
> as well as different mount options.  For example I run all the different mkfs
> features through normal mount options, and then again with compression turned
> on.  Thanks,

So what I mean by kconfig is not the Linux kernel kconfig, but rather
the kdevops kconfig options. kdevops essentially has a kconfig symbol
per mkfs-param-mount config we test. And it runs *ones* guest per each
of these. For example:

config FSTESTS_XFS_SECTION_REFLINK_1024
	bool "Enable testing section: xfs_reflink_1024"
	default y
	help
	  This will create a host to test the baseline of fstests using the
	  following configuration which enables reflink using 1024 byte block
	  size.

	[xfs_reflink]
	MKFS_OPTIONS='-f -m reflink=1,rmapbt=1, -i sparse=1,'
	FSTYP=xfs

The other ones can be found here for XFS:

https://github.com/mcgrof/kdevops/blob/master/workflows/fstests/xfs/Kconfig

So indeed, exactly what you mean. What I'm getting at is that it would
be good to construct these with the community. So it would beg the
question if we should embrace for instance kconfig language to be
able to configure fstests (yes I know it is xfstests but I think loose
new people who tend to assume that xfstest is only for XFS, so I only
always call it fstests).

  Luis