Re: [RFC: kdevops] Standardizing on failure rate nomenclature for expunges

From: "Theodore Ts'o" <tytso@mit.edu>
To: Amir Goldstein <amir73il@gmail.com>
Cc: Bart Van Assche <bvanassche@acm.org>,
	"Darrick J. Wong" <djwong@kernel.org>,
	Luis Chamberlain <mcgrof@kernel.org>,
	linux-fsdevel <linux-fsdevel@vger.kernel.org>,
	linux-block <linux-block@vger.kernel.org>,
	Pankaj Raghav <pankydev8@gmail.com>,
	Josef Bacik <josef@toxicpanda.com>,
	jmeneghi@redhat.com, Jan Kara <jack@suse.cz>,
	Davidlohr Bueso <dave@stgolabs.net>,
	Dan Williams <dan.j.williams@intel.com>, Jake Edge <jake@lwn.net>,
	Klaus Jensen <its@irrelevant.dk>,
	fstests <fstests@vger.kernel.org>, Zorro Lang <zlang@redhat.com>,
	Matthew Wilcox <willy@infradead.org>,
	Dave Chinner <david@fromorbit.com>
Subject: Re: [RFC: kdevops] Standardizing on failure rate nomenclature for expunges
Date: Sun, 3 Jul 2022 09:15:27 -0400	[thread overview]
Message-ID: <YsGWb8nPUySuhos/@mit.edu> (raw)
In-Reply-To: <CAOQ4uxgBtMifsNt1SDA0tz098Rt7Km6MAaNgfCeW=s=FPLtpCQ@mail.gmail.com>

On Sun, Jul 03, 2022 at 08:56:54AM +0300, Amir Goldstein wrote:
> 
> That is true for some use cases, but unfortunately, the flaky
> fstests are way too valuable and too hard to replace or improve,
> so practically, fs developers have to run them, but not everyone does.
> 
> Zorro has already proposed to properly tag the non deterministic tests
> with a specific group and I think there is really no other solution.

The non-deterministic tests are not the sole, or even the most likely
cause of flaky tests.  Or put another way, even if we used a
deterministic pseudo-random numberator seed for some of the curently
"non-determinstic tests" (and I believe we are for many of them
already anyway), it's not going to be make the flaky tests go away.

That's because with many of these tests, we are running multiple
threads either in the fstress or fsx, or in the antogonist workload
that is say, running the space utilization to full to generate ENOSPC
errors, and then deleting a bunch of files to trigger as many ENOSPC
hitter events as possible.

> The only question is whether we remove them from the 'auto' group
> (I think we should).

I wouldn't; if someone wants to exclude the non-determistic tests,
once they are tagged as belonging to a group, they can just exclude
that group.  So there's no point removing them from the auto group
IMHO.

> filesystem developers that will run ./check -g auto -g soak
> will get the exact same test coverage as today's -g auto
> and the "commoners" that run ./check -g auto will enjoy blissful
> determitic test results, at least for the default config of regularly
> tested filesystems (a.k.a, the ones tested by kernet test bot).?

First of all, there are a number of tests today which are in soak or
long_rw which are not in auto, so "-g auto -g soak" will *not* result
in the "exact same test coverage".

Secondly, as I've tested above, deterministic tests does not
necessasrily mean determinsitic test results --- unless by
"determinsitic tests" you mean "completely single-threaded tests",
which would eliminate a large amount of useful test coverage.

Cheers,

					- Ted