Re: [ANN] oscheck: wrapper for fstests check.sh - tracking and working with baselines

[Jeff Mahoney <jeffm@suse.com>]

[-- Attachment #1.1: Type: text/plain, Size: 4049 bytes --]

On 7/13/18 4:50 PM, Luis R. Chamberlain wrote:
> On Fri, Jul 13, 2018 at 04:40:39PM -0400, Jeff Mahoney wrote:
>> On 7/13/18 12:44 PM, Luis R. Chamberlain wrote:
>>> On Fri, Jul 13, 2018 at 11:39:55AM +0300, Amir Goldstein wrote:
>>>> On Fri, Jul 13, 2018 at 5:43 AM, Luis R. Chamberlain <mcgrof@kernel.org> wrote:
>>>>> I had volunteered at the last LSF/MM to help with the stable work for
>>>>> XFS. To help with this, as part of this year's SUSE Hackweek, I've
>>>>> first generalized my own set of scripts to help track a baseline of
>>>>> results from fstests [0], and extended it to be able to easily ramp up
>>>>> with fstests on different distributions, and I've also created a
>>>>> respective baseline of results against these distributions as a
>>>>> further example of how these scripts and wrapper framework can be used
>>>>
>>>> Hi Luis!
>>>>
>>>> Thanks a lot for doing this work!
>>>>
>>>> Will take me some time to try it out, but see some questions below...
>>>>
>>>>> [1]. The distributions currently supported are:
>>>>>
>>>>>   * Debian testing
>>>>>   * OpenSUSE Leap 15.0
>>>>>   * Fedora 28
>>>>>
>>>>> The stable work starts with creating a baseline for v4.17.3. The
>>>>> results are visible as a result of expunge files which categorize the
>>>>> failures for the different sections tested.
>>>>
>>>> So the only "bad" indication is a test failure?
>>>
>>> That is correct to a certain degree, ie, if xfsprogs / the kernel
>>> config could run it we can assume it passed.
>>>
>>>> How about indication about a test that started to pass since baseline?
>>>
>>> Indeed, that is desirable.
>>>
>>> We have a few options. One is share the entire results directory for
>>> a release / section, however this is rather big. For instance for a
>>> full v4.17.3 run this is about 292 MiB alone. I don't think this
>>> scales. IMHO lgogs should only be supplied onto bug reports, not this
>>> framework.
>>>
>>> The other option is to use -R xunit to generate the report in the
>>> specified unit. I have not yet run this, or tried it, however IIRC
>>> it does record success runs? Does it also keep logs? Hopefully not.  I'm
>>> assuming it does not as of yet. I should note if one hits CTRL-C in the
>>> middle one does not get the results. An alternative was being worked on
>>> by Jeff which would sprinkle IIRC .ok files for tests which succeed,
>>> then you could just scrape the results directory to determine which
>>> tests did pass -- but you run into the same size problem as above.
>>
>> Eryu didn't like that idea, so I abandoned it.  What I have now is a -R
>> files mode that creates a bunch of files with the goal of just archiving
>> the results for later comparison or import into a results db.
>>
>> For each test, there are:
>> $seq.result.start.txt - start timestamp
>> $seq.result.stop.txt - stop timestamp
>> $seq.result.result.txt - simple result: pass/fail/expunged/notrun
>> $seq.result.detail.txt - contains the contents of $seq.notrun/$seq.expunged
>> $seq.result.{dmesg,kmemleak,full,check}.txt - contains the contents of
>> the corresponding files
> 
> This is sexy, it also gives the person interpretting the results to
> opt-in or not for the actuall full log of the output. You pick and
> choose what info you want.
> 
> This is indeed nice.
> 
>> As an aside, IIRC, -R xunit doesn't catch all kinds of failures.  Also,
>> as you mentioned, if it's interrupted, all results are lost.  This makes
>> it difficult to identify test failures that crashed or hung the test system.
> 
> OK so indeed not my preference.
> 
>> I have some basic scripts that parse the output and generate an HTML
>> report/table (and it does do what Amir asks WRT tests that started passing).
> 
> These scripts, are they for parsing your new -R files output?

Yep.

> I take it the patches are still being worked on?

Yeah.  They just need a bit of review and cleaning up and I can post them.

-Jeff

-- 
Jeff Mahoney
SUSE Labs


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]