Re: [RFC: kdevops] Standardizing on failure rate nomenclature for expunges

[Luis Chamberlain <mcgrof@kernel.org>]

On Sun, Jul 03, 2022 at 09:32:03AM -0400, Theodore Ts'o wrote:
> On Sat, Jul 02, 2022 at 02:48:12PM -0700, Bart Van Assche wrote:
> > 
> > I strongly disagree with annotating tests with failure rates. My opinion is
> > that on a given test setup a test either should pass 100% of the time or
> > fail 100% of the time.
> 
> in the real world, while we can *strive* to
> eliminate all flaky tests, whether it is caused by buggy tests, or
> buggy kernel code, there's an old saying that the only time code is
> bug-free is when it is no longer being used.

Agreed but I will provide proof in reply to Bart shortly a bit more
related to the block layer. I thought I made the case clear enough
at LSFMM but I suppose not.

> That being said, I completely agree that annotating failure rates in
> xfstesets-dev upstream probably doesn't make much sense.  As we've
> stated before, it is highly dependent on the hardware configuration,
> and kernel version (remember, sometimes flaky tests are caused by bugs
> in other kernel subsystems --- including the loop device, which has
> not historically been bug-free(tm) either, and so bugs come and go
> across the entire kernel surface).

That does not eliminate the possible value of having failure rates for
the minimum virtualized storage arangement you can have with either
loopback devices or LVM volumes.

Nor, does it eliminate the possibility to say come up with generic
system names. Just as 0-day has names for kernel configs, we can easily
come up with names for hw profiles.

> I believe the best way to handle this is to have better test results
> analysis tools.

We're going to evaluate an ELK stack for this, but there is a difference
between historical data for random runs Vs what may be useful
generically.

> We can certainly consider having some shared test
> results database, but I'm not convinced that flat text files shared
> via git is sufficiently scalable.

How data is stored is secondary, first is order of business is if
sharing any of this information may be useful to others. I have
results dating back to 4.17.3, each kernel supported and I have
found it very valuable. I figured it may be.. but if there is no
agreement on it, we can just keep that on kdevops as-is and move
forward with our own nomenclature for hw profiles.

  Luis