Re: Are the xfstests exclusion files on wiki.samba.org up to date?

[ronnie sahlberg <ronniesahlberg@gmail.com>]

Thanks, good post.

I think the OP is right. The wikipage is obsolete and IMHO we should
change it to just point to our buildbot
as being the canonical truth of what tests we expect to run.
(we used to have the outdated wiki, then we had every developer having
private, secret scripts with specific subsets of tests,   and we had a
lot of regressions every release until ~5.0 or so I would say.)

At this point, the tests we run on the buildbot IS the canonical set
of the tests that are expected to work and that all of us targets.
Maybe we should look into surfacing the list of tests we perform
better. I.e. what tests do we run against windows/samba+btrfs/azure
and why?

Lets get rid of the list of tests from the wiki and let the buildbot
be the single canonical source of truth.

Regards
ronnie sahlberg


On Tue, Sep 10, 2019 at 8:23 PM Steve French <smfrench@gmail.com> wrote:
>
> We have done a lot of work (with Ronnie, Aurelien, Pavel and others)
> on xfstest automation.
>
> As you might guess it is frustrating for two reasons:
> 1) updated xfstests can be flaky (as the tests themselves are updated,
> they add subtle required features, or regress from time to time)
> 2) test automation can run into resource constraints (VMs trying to
> run tests with less memory than might be optimal - especially those
> run against Samba with "VMs inside VMs").
>
> But the good news is that we have VERY good data on which tests pass
> to various servers (just as noted, need to update the external pages)
> and we should have even more data as we go through two weeks of SMB3
> testing events in late September.
>
> In general we want to test against all typical servers and have test
> targets setup for
>  - Samba (reasonably current stable) on ext4/xfs and Samba with btrfs
> (which has various optional extensions enabled in the server)
>  - Samba with POSIX Extensions
>  - Windows (and against both NTFS and REFS server file system)
> - Azure (Cloud0
>
> But would really like to add test targets for other common servers and
> in a perfect world would like to be able to have external test
> automation pull our trees periodically to run these (similar to what
> is done for the six targets above):
>   - Macs
>   - NetApp
> (and any others of interest to the community)
>
> Obviously xfstest has a few hundred tests which only make sense for
> local file systems with block devices, but there are hundreds of
> xfstests of value, and most should run on cifs.ko and we are working
> through them but already have a very good set running.
>
> On Mon, Sep 9, 2019 at 5:50 PM Aurélien Aptel <aaptel@suse.com> wrote:
> >
> > "Murphy Zhou" <jencce.kernel@gmail.com> writes:
> > > As $subject. Is this wiki being maintained ?
> > >
> > > Looks like the last update was in January 2019.
> > >
> > > https://wiki.samba.org/index.php/Xfstesting-cifs#Exclusion_files
> >
> > We have a buildbot running xfstests relatively often here
> >
> > http://smb3-test-rhel-75.southcentralus.cloudapp.azure.com/#/
> >
> > Each group has slightly different tests run, you can see which one gets
> > run by clicking on a build:
> >
> > http://smb3-test-rhel-75.southcentralus.cloudapp.azure.com/#/builders/2/builds/249
> >
> > Overall xfstests+cifs is very finicky and frustrating to get working
> > reliably. Not to mention long. So good luck :)
> >
> > Cheers,
> > --
> > Aurélien Aptel / SUSE Labs Samba Team
> > GPG: 1839 CB5F 9F5B FB9B AA97  8C99 03C8 A49B 521B D5D3
> > SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg, DE
> > GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 247165 (AG München)
>
>
>
> --
> Thanks,
>
> Steve