Are the xfstests exclusion files on wiki.samba.org up to date?

Are the xfstests exclusion files on wiki.samba.org up to date?

[Murphy Zhou]

Hi,

As $subject. Is this wiki being maintained ?

Looks like the last update was in January 2019.

https://wiki.samba.org/index.php/Xfstesting-cifs#Exclusion_files

Thanks!
M

Re: Are the xfstests exclusion files on wiki.samba.org up to date?

[Aurélien Aptel]

"Murphy Zhou" <jencce.kernel@gmail.com> writes:
> As $subject. Is this wiki being maintained ?
>
> Looks like the last update was in January 2019.
>
> https://wiki.samba.org/index.php/Xfstesting-cifs#Exclusion_files

We have a buildbot running xfstests relatively often here

http://smb3-test-rhel-75.southcentralus.cloudapp.azure.com/#/

Each group has slightly different tests run, you can see which one gets
run by clicking on a build:

http://smb3-test-rhel-75.southcentralus.cloudapp.azure.com/#/builders/2/builds/249

Overall xfstests+cifs is very finicky and frustrating to get working
reliably. Not to mention long. So good luck :)

Cheers,
-- 
Aurélien Aptel / SUSE Labs Samba Team
GPG: 1839 CB5F 9F5B FB9B AA97  8C99 03C8 A49B 521B D5D3
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg, DE
GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 247165 (AG München)

Re: Are the xfstests exclusion files on wiki.samba.org up to date?

[Steve French]

We have done a lot of work (with Ronnie, Aurelien, Pavel and others)
on xfstest automation.

As you might guess it is frustrating for two reasons:
1) updated xfstests can be flaky (as the tests themselves are updated,
they add subtle required features, or regress from time to time)
2) test automation can run into resource constraints (VMs trying to
run tests with less memory than might be optimal - especially those
run against Samba with "VMs inside VMs").

But the good news is that we have VERY good data on which tests pass
to various servers (just as noted, need to update the external pages)
and we should have even more data as we go through two weeks of SMB3
testing events in late September.

In general we want to test against all typical servers and have test
targets setup for
 - Samba (reasonably current stable) on ext4/xfs and Samba with btrfs
(which has various optional extensions enabled in the server)
 - Samba with POSIX Extensions
 - Windows (and against both NTFS and REFS server file system)
- Azure (Cloud0

But would really like to add test targets for other common servers and
in a perfect world would like to be able to have external test
automation pull our trees periodically to run these (similar to what
is done for the six targets above):
  - Macs
  - NetApp
(and any others of interest to the community)

Obviously xfstest has a few hundred tests which only make sense for
local file systems with block devices, but there are hundreds of
xfstests of value, and most should run on cifs.ko and we are working
through them but already have a very good set running.

On Mon, Sep 9, 2019 at 5:50 PM Aurélien Aptel <aaptel@suse.com> wrote:
>
> "Murphy Zhou" <jencce.kernel@gmail.com> writes:
> > As $subject. Is this wiki being maintained ?
> >
> > Looks like the last update was in January 2019.
> >
> > https://wiki.samba.org/index.php/Xfstesting-cifs#Exclusion_files
>
> We have a buildbot running xfstests relatively often here
>
> http://smb3-test-rhel-75.southcentralus.cloudapp.azure.com/#/
>
> Each group has slightly different tests run, you can see which one gets
> run by clicking on a build:
>
> http://smb3-test-rhel-75.southcentralus.cloudapp.azure.com/#/builders/2/builds/249
>
> Overall xfstests+cifs is very finicky and frustrating to get working
> reliably. Not to mention long. So good luck :)
>
> Cheers,
> --
> Aurélien Aptel / SUSE Labs Samba Team
> GPG: 1839 CB5F 9F5B FB9B AA97  8C99 03C8 A49B 521B D5D3
> SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg, DE
> GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 247165 (AG München)



-- 
Thanks,

Steve

Re: Are the xfstests exclusion files on wiki.samba.org up to date?

[ronnie sahlberg]

Thanks, good post.

I think the OP is right. The wikipage is obsolete and IMHO we should
change it to just point to our buildbot
as being the canonical truth of what tests we expect to run.
(we used to have the outdated wiki, then we had every developer having
private, secret scripts with specific subsets of tests,   and we had a
lot of regressions every release until ~5.0 or so I would say.)

At this point, the tests we run on the buildbot IS the canonical set
of the tests that are expected to work and that all of us targets.
Maybe we should look into surfacing the list of tests we perform
better. I.e. what tests do we run against windows/samba+btrfs/azure
and why?

Lets get rid of the list of tests from the wiki and let the buildbot
be the single canonical source of truth.

Regards
ronnie sahlberg


On Tue, Sep 10, 2019 at 8:23 PM Steve French <smfrench@gmail.com> wrote:
>
> We have done a lot of work (with Ronnie, Aurelien, Pavel and others)
> on xfstest automation.
>
> As you might guess it is frustrating for two reasons:
> 1) updated xfstests can be flaky (as the tests themselves are updated,
> they add subtle required features, or regress from time to time)
> 2) test automation can run into resource constraints (VMs trying to
> run tests with less memory than might be optimal - especially those
> run against Samba with "VMs inside VMs").
>
> But the good news is that we have VERY good data on which tests pass
> to various servers (just as noted, need to update the external pages)
> and we should have even more data as we go through two weeks of SMB3
> testing events in late September.
>
> In general we want to test against all typical servers and have test
> targets setup for
>  - Samba (reasonably current stable) on ext4/xfs and Samba with btrfs
> (which has various optional extensions enabled in the server)
>  - Samba with POSIX Extensions
>  - Windows (and against both NTFS and REFS server file system)
> - Azure (Cloud0
>
> But would really like to add test targets for other common servers and
> in a perfect world would like to be able to have external test
> automation pull our trees periodically to run these (similar to what
> is done for the six targets above):
>   - Macs
>   - NetApp
> (and any others of interest to the community)
>
> Obviously xfstest has a few hundred tests which only make sense for
> local file systems with block devices, but there are hundreds of
> xfstests of value, and most should run on cifs.ko and we are working
> through them but already have a very good set running.
>
> On Mon, Sep 9, 2019 at 5:50 PM Aurélien Aptel <aaptel@suse.com> wrote:
> >
> > "Murphy Zhou" <jencce.kernel@gmail.com> writes:
> > > As $subject. Is this wiki being maintained ?
> > >
> > > Looks like the last update was in January 2019.
> > >
> > > https://wiki.samba.org/index.php/Xfstesting-cifs#Exclusion_files
> >
> > We have a buildbot running xfstests relatively often here
> >
> > http://smb3-test-rhel-75.southcentralus.cloudapp.azure.com/#/
> >
> > Each group has slightly different tests run, you can see which one gets
> > run by clicking on a build:
> >
> > http://smb3-test-rhel-75.southcentralus.cloudapp.azure.com/#/builders/2/builds/249
> >
> > Overall xfstests+cifs is very finicky and frustrating to get working
> > reliably. Not to mention long. So good luck :)
> >
> > Cheers,
> > --
> > Aurélien Aptel / SUSE Labs Samba Team
> > GPG: 1839 CB5F 9F5B FB9B AA97  8C99 03C8 A49B 521B D5D3
> > SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg, DE
> > GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 247165 (AG München)
>
>
>
> --
> Thanks,
>
> Steve

Re: Are the xfstests exclusion files on wiki.samba.org up to date?

[Aurélien Aptel]

"ronnie sahlberg" <ronniesahlberg@gmail.com> writes:
> Lets get rid of the list of tests from the wiki and let the buildbot
> be the single canonical source of truth.

Sounds good to me. Ideally this list should be exportable (perhaps
automatically from the buildbot script) in a format that can be used
directly by the check script for others to use.

-- 
Aurélien Aptel / SUSE Labs Samba Team
GPG: 1839 CB5F 9F5B FB9B AA97  8C99 03C8 A49B 521B D5D3
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg, DE
GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 247165 (AG München)

Re: Are the xfstests exclusion files on wiki.samba.org up to date?

[Murphy Zhou]

On Mon, Sep 09, 2019 at 12:50:09PM +0200, Aurélien Aptel wrote:
> "Murphy Zhou" <jencce.kernel@gmail.com> writes:
> > As $subject. Is this wiki being maintained ?
> >
> > Looks like the last update was in January 2019.
> >
> > https://wiki.samba.org/index.php/Xfstesting-cifs#Exclusion_files
> 
> We have a buildbot running xfstests relatively often here
> 
> http://smb3-test-rhel-75.southcentralus.cloudapp.azure.com/#/
> 
> Each group has slightly different tests run, you can see which one gets
> run by clicking on a build:
> 
> http://smb3-test-rhel-75.southcentralus.cloudapp.azure.com/#/builders/2/builds/249

Looks like I am unauthorised to access these sites.

> 
> Overall xfstests+cifs is very finicky and frustrating to get working
> reliably. Not to mention long. So good luck :)

Indeed. Thanks :)

> 
> Cheers,
> -- 
> Aurélien Aptel / SUSE Labs Samba Team
> GPG: 1839 CB5F 9F5B FB9B AA97  8C99 03C8 A49B 521B D5D3
> SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg, DE
> GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 247165 (AG München)

Re: Are the xfstests exclusion files on wiki.samba.org up to date?

[Murphy Zhou]

On Mon, Sep 09, 2019 at 06:23:10PM -0500, Steve French wrote:
> We have done a lot of work (with Ronnie, Aurelien, Pavel and others)
> on xfstest automation.

Kudos!

> 
> As you might guess it is frustrating for two reasons:
> 1) updated xfstests can be flaky (as the tests themselves are updated,
> they add subtle required features, or regress from time to time)
> 2) test automation can run into resource constraints (VMs trying to
> run tests with less memory than might be optimal - especially those
> run against Samba with "VMs inside VMs").
> 
> But the good news is that we have VERY good data on which tests pass
> to various servers (just as noted, need to update the external pages)
> and we should have even more data as we go through two weeks of SMB3
> testing events in late September.
> 
> In general we want to test against all typical servers and have test
> targets setup for
>  - Samba (reasonably current stable) on ext4/xfs and Samba with btrfs
> (which has various optional extensions enabled in the server)
>  - Samba with POSIX Extensions
>  - Windows (and against both NTFS and REFS server file system)
> - Azure (Cloud0

Thanks for sharing this!

Besides server type, server configuration options and client mount
options also make this matrix even bigger..

How that is handled in the buildbot ?

> 
> But would really like to add test targets for other common servers and
> in a perfect world would like to be able to have external test
> automation pull our trees periodically to run these (similar to what
> is done for the six targets above):
>   - Macs
>   - NetApp
> (and any others of interest to the community)
> 
> Obviously xfstest has a few hundred tests which only make sense for
> local file systems with block devices, but there are hundreds of
> xfstests of value, and most should run on cifs.ko and we are working
> through them but already have a very good set running.

Agree!  Thanks!

Murphy

> 
> On Mon, Sep 9, 2019 at 5:50 PM Aurélien Aptel <aaptel@suse.com> wrote:
> >
> > "Murphy Zhou" <jencce.kernel@gmail.com> writes:
> > > As $subject. Is this wiki being maintained ?
> > >
> > > Looks like the last update was in January 2019.
> > >
> > > https://wiki.samba.org/index.php/Xfstesting-cifs#Exclusion_files
> >
> > We have a buildbot running xfstests relatively often here
> >
> > http://smb3-test-rhel-75.southcentralus.cloudapp.azure.com/#/
> >
> > Each group has slightly different tests run, you can see which one gets
> > run by clicking on a build:
> >
> > http://smb3-test-rhel-75.southcentralus.cloudapp.azure.com/#/builders/2/builds/249
> >
> > Overall xfstests+cifs is very finicky and frustrating to get working
> > reliably. Not to mention long. So good luck :)
> >
> > Cheers,
> > --
> > Aurélien Aptel / SUSE Labs Samba Team
> > GPG: 1839 CB5F 9F5B FB9B AA97  8C99 03C8 A49B 521B D5D3
> > SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg, DE
> > GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 247165 (AG München)
> 
> 
> 
> -- 
> Thanks,
> 
> Steve

Re: Are the xfstests exclusion files on wiki.samba.org up to date?

[Murphy Zhou]

On Tue, Sep 10, 2019 at 01:41:46PM +0200, Aurélien Aptel wrote:
> "ronnie sahlberg" <ronniesahlberg@gmail.com> writes:
> > Lets get rid of the list of tests from the wiki and let the buildbot
> > be the single canonical source of truth.
> 
> Sounds good to me. Ideally this list should be exportable (perhaps
> automatically from the buildbot script) in a format that can be used
> directly by the check script for others to use.

That would be perfect! Thank you very much Ronnie and Aurélien!

> 
> -- 
> Aurélien Aptel / SUSE Labs Samba Team
> GPG: 1839 CB5F 9F5B FB9B AA97  8C99 03C8 A49B 521B D5D3
> SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg, DE
> GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 247165 (AG München)

Re: Are the xfstests exclusion files on wiki.samba.org up to date?

[ronnie sahlberg]

On Thu, Sep 12, 2019 at 6:50 PM Murphy Zhou <jencce.kernel@gmail.com> wrote:
>
> On Mon, Sep 09, 2019 at 06:23:10PM -0500, Steve French wrote:
> > We have done a lot of work (with Ronnie, Aurelien, Pavel and others)
> > on xfstest automation.
>
> Kudos!
>
> >
> > As you might guess it is frustrating for two reasons:
> > 1) updated xfstests can be flaky (as the tests themselves are updated,
> > they add subtle required features, or regress from time to time)
> > 2) test automation can run into resource constraints (VMs trying to
> > run tests with less memory than might be optimal - especially those
> > run against Samba with "VMs inside VMs").
> >
> > But the good news is that we have VERY good data on which tests pass
> > to various servers (just as noted, need to update the external pages)
> > and we should have even more data as we go through two weeks of SMB3
> > testing events in late September.
> >
> > In general we want to test against all typical servers and have test
> > targets setup for
> >  - Samba (reasonably current stable) on ext4/xfs and Samba with btrfs
> > (which has various optional extensions enabled in the server)
> >  - Samba with POSIX Extensions
> >  - Windows (and against both NTFS and REFS server file system)
> > - Azure (Cloud0
>
> Thanks for sharing this!
>
> Besides server type, server configuration options and client mount
> options also make this matrix even bigger..
>
> How that is handled in the buildbot ?

We have several different targets, which maps to the local.config for
xfstests so for every single test we run
we also specify the different targets to run that for.

Here is an example:
all_tests = [
        [ "cifs/100", "smb3azureseal"],
        [ "cifs/101", "smb3multiuser"],
        [ "generic/001", "smb3sign", "smb3"],
        [ "generic/002", "smb3", "smb3sign", "smb21", "smb3samba"],
        [ "generic/005", "smb3", "smb21", "smb3samba", "smb3sambabtrfs"],
        [ "generic/006", "smb3"],
        [ "generic/007", "smb3"],
        [ "generic/010", "smb3"],
        [ "generic/011", "smb3"],
        [ "generic/013", "smb3samba"],
        [ "generic/014", "smb3"],
        [ "generic/020", "smb3samba"],
        [ "generic/023", "smb3samba"],
        [ "generic/024", "smb3", "smb3samba"],
        [ "generic/028", "smb3", "smb3samba"],
...


The first argument is which xfstest to run and the remainder of the
arguments are a list of which local.config.* files to use.
That way we can run a test against different servers and also
different configurations.
I try to keep the target names meaningful, so

"smb3" just means a basic smb3   windows 2016 server running in a vm
"smb3samba" is just a generic smb3 server with samba.
etc etc.


We started doing this upstream about a year ago which is when I
started setting it up
and now I am honest but since we do this for every pull request we
send to linus,
we have SIGNIFICANTLY increased the quality of cifs.ko.

We run quite a lot of tests nowadays and I always try to remind Steve
to "please put a link to the buildbot result in the pull request"
to bring awareness to our buildbot.

For upstream, here is the most recent test:
http://smb3-test-rhel-75.southcentralus.cloudapp.azure.com/#/builders/2/builds/249

cifs-testing is the most generic test target. We also have specific
targets that are just for Azure, and other targets, but the aim is
that the canonical cifs-target tests will all pass before we pass a
pull request onto linus.


Please contact me directly if you want more info about the buildbot. I
have one setup internally at rh too that runs many, but not all, the
tests we run upstream.
Any help to maintain and expand the tests are super welcome.


Rant-off
ronnie sahlberg

>
> >
> > But would really like to add test targets for other common servers and
> > in a perfect world would like to be able to have external test
> > automation pull our trees periodically to run these (similar to what
> > is done for the six targets above):
> >   - Macs
> >   - NetApp
> > (and any others of interest to the community)
> >
> > Obviously xfstest has a few hundred tests which only make sense for
> > local file systems with block devices, but there are hundreds of
> > xfstests of value, and most should run on cifs.ko and we are working
> > through them but already have a very good set running.
>
> Agree!  Thanks!
>
> Murphy
>
> >
> > On Mon, Sep 9, 2019 at 5:50 PM Aurélien Aptel <aaptel@suse.com> wrote:
> > >
> > > "Murphy Zhou" <jencce.kernel@gmail.com> writes:
> > > > As $subject. Is this wiki being maintained ?
> > > >
> > > > Looks like the last update was in January 2019.
> > > >
> > > > https://wiki.samba.org/index.php/Xfstesting-cifs#Exclusion_files
> > >
> > > We have a buildbot running xfstests relatively often here
> > >
> > > http://smb3-test-rhel-75.southcentralus.cloudapp.azure.com/#/
> > >
> > > Each group has slightly different tests run, you can see which one gets
> > > run by clicking on a build:
> > >
> > > http://smb3-test-rhel-75.southcentralus.cloudapp.azure.com/#/builders/2/builds/249
> > >
> > > Overall xfstests+cifs is very finicky and frustrating to get working
> > > reliably. Not to mention long. So good luck :)
> > >
> > > Cheers,
> > > --
> > > Aurélien Aptel / SUSE Labs Samba Team
> > > GPG: 1839 CB5F 9F5B FB9B AA97  8C99 03C8 A49B 521B D5D3
> > > SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg, DE
> > > GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 247165 (AG München)
> >
> >
> >
> > --
> > Thanks,
> >
> > Steve

Re: Are the xfstests exclusion files on wiki.samba.org up to date?

[Murphy Zhou]

On Thu, Sep 12, 2019 at 07:06:52PM +1000, ronnie sahlberg wrote:
> On Thu, Sep 12, 2019 at 6:50 PM Murphy Zhou <jencce.kernel@gmail.com> wrote:
> >
> > On Mon, Sep 09, 2019 at 06:23:10PM -0500, Steve French wrote:
> > > We have done a lot of work (with Ronnie, Aurelien, Pavel and others)
> > > on xfstest automation.
> >
> > Kudos!
> >
> > >
> > > As you might guess it is frustrating for two reasons:
> > > 1) updated xfstests can be flaky (as the tests themselves are updated,
> > > they add subtle required features, or regress from time to time)
> > > 2) test automation can run into resource constraints (VMs trying to
> > > run tests with less memory than might be optimal - especially those
> > > run against Samba with "VMs inside VMs").
> > >
> > > But the good news is that we have VERY good data on which tests pass
> > > to various servers (just as noted, need to update the external pages)
> > > and we should have even more data as we go through two weeks of SMB3
> > > testing events in late September.
> > >
> > > In general we want to test against all typical servers and have test
> > > targets setup for
> > >  - Samba (reasonably current stable) on ext4/xfs and Samba with btrfs
> > > (which has various optional extensions enabled in the server)
> > >  - Samba with POSIX Extensions
> > >  - Windows (and against both NTFS and REFS server file system)
> > > - Azure (Cloud0
> >
> > Thanks for sharing this!
> >
> > Besides server type, server configuration options and client mount
> > options also make this matrix even bigger..
> >
> > How that is handled in the buildbot ?
> 
> We have several different targets, which maps to the local.config for
> xfstests so for every single test we run
> we also specify the different targets to run that for.
> 
> Here is an example:
> all_tests = [
>         [ "cifs/100", "smb3azureseal"],
>         [ "cifs/101", "smb3multiuser"],
>         [ "generic/001", "smb3sign", "smb3"],
>         [ "generic/002", "smb3", "smb3sign", "smb21", "smb3samba"],
>         [ "generic/005", "smb3", "smb21", "smb3samba", "smb3sambabtrfs"],
>         [ "generic/006", "smb3"],
>         [ "generic/007", "smb3"],
>         [ "generic/010", "smb3"],
>         [ "generic/011", "smb3"],
>         [ "generic/013", "smb3samba"],
>         [ "generic/014", "smb3"],
>         [ "generic/020", "smb3samba"],
>         [ "generic/023", "smb3samba"],
>         [ "generic/024", "smb3", "smb3samba"],
>         [ "generic/028", "smb3", "smb3samba"],
> ...
> 
> 
> The first argument is which xfstest to run and the remainder of the
> arguments are a list of which local.config.* files to use.
> That way we can run a test against different servers and also
> different configurations.

This is smart. :)

> I try to keep the target names meaningful, so
> 
> "smb3" just means a basic smb3   windows 2016 server running in a vm
> "smb3samba" is just a generic smb3 server with samba.
> etc etc.
> 
> 
> We started doing this upstream about a year ago which is when I
> started setting it up
> and now I am honest but since we do this for every pull request we
> send to linus,
> we have SIGNIFICANTLY increased the quality of cifs.ko.

/Thumbs up!

> 
> We run quite a lot of tests nowadays and I always try to remind Steve
> to "please put a link to the buildbot result in the pull request"
> to bring awareness to our buildbot.
> 
> For upstream, here is the most recent test:
> http://smb3-test-rhel-75.southcentralus.cloudapp.azure.com/#/builders/2/builds/249
> 
> cifs-testing is the most generic test target. We also have specific
> targets that are just for Azure, and other targets, but the aim is
> that the canonical cifs-target tests will all pass before we pass a
> pull request onto linus.
> 
> 
> Please contact me directly if you want more info about the buildbot. I
> have one setup internally at rh too that runs many, but not all, the
> tests we run upstream.

Sure! Have replied to you. Thank you very much for sharing!

If anybody resting cifs or I know that the expected setup and result of
every tests for you developers, it will save much time of investigation
and report false alarms.

After several months working on cifs, I've got a feeling that cifs is
much different from other Linux filesystems. Maybe because of Windows
or the SMB protocols. I'm wondering that xfstests which is a typical
testsuite for Unix/Linux filesystems, maybe is not that suitable for
cifs. I agree with Steve that there are hundreds of tests suitable.
If, only if, :) if we could know how MS testing NTFS or SMB protocols,
would that help or more suitable?


Thanks!
Murphy


> Any help to maintain and expand the tests are super welcome.
> 
> 
> Rant-off
> ronnie sahlberg
> 
> >
> > >
> > > But would really like to add test targets for other common servers and
> > > in a perfect world would like to be able to have external test
> > > automation pull our trees periodically to run these (similar to what
> > > is done for the six targets above):
> > >   - Macs
> > >   - NetApp
> > > (and any others of interest to the community)
> > >
> > > Obviously xfstest has a few hundred tests which only make sense for
> > > local file systems with block devices, but there are hundreds of
> > > xfstests of value, and most should run on cifs.ko and we are working
> > > through them but already have a very good set running.
> >
> > Agree!  Thanks!
> >
> > Murphy
> >
> > >
> > > On Mon, Sep 9, 2019 at 5:50 PM Aurélien Aptel <aaptel@suse.com> wrote:
> > > >
> > > > "Murphy Zhou" <jencce.kernel@gmail.com> writes:
> > > > > As $subject. Is this wiki being maintained ?
> > > > >
> > > > > Looks like the last update was in January 2019.
> > > > >
> > > > > https://wiki.samba.org/index.php/Xfstesting-cifs#Exclusion_files
> > > >
> > > > We have a buildbot running xfstests relatively often here
> > > >
> > > > http://smb3-test-rhel-75.southcentralus.cloudapp.azure.com/#/
> > > >
> > > > Each group has slightly different tests run, you can see which one gets
> > > > run by clicking on a build:
> > > >
> > > > http://smb3-test-rhel-75.southcentralus.cloudapp.azure.com/#/builders/2/builds/249
> > > >
> > > > Overall xfstests+cifs is very finicky and frustrating to get working
> > > > reliably. Not to mention long. So good luck :)
> > > >
> > > > Cheers,
> > > > --
> > > > Aurélien Aptel / SUSE Labs Samba Team
> > > > GPG: 1839 CB5F 9F5B FB9B AA97  8C99 03C8 A49B 521B D5D3
> > > > SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg, DE
> > > > GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 247165 (AG München)
> > >
> > >
> > >
> > > --
> > > Thanks,
> > >
> > > Steve

Re: Are the xfstests exclusion files on wiki.samba.org up to date?

[Murphy Zhou]

On Fri, Sep 13, 2019 at 05:03:57PM -0700, Steve French wrote:
> Actually the xfstests we pass compared favorably with other network and
> cluster fs but just different :)

Agreed :)

M
> 
> On Fri, Sep 13, 2019, 17:02 Murphy Zhou <jencce.kernel@gmail.com> wrote:
> 
> > On Thu, Sep 12, 2019 at 07:06:52PM +1000, ronnie sahlberg wrote:
> > > On Thu, Sep 12, 2019 at 6:50 PM Murphy Zhou <jencce.kernel@gmail.com>
> > wrote:
> > > >
> > > > On Mon, Sep 09, 2019 at 06:23:10PM -0500, Steve French wrote:
> > > > > We have done a lot of work (with Ronnie, Aurelien, Pavel and others)
> > > > > on xfstest automation.
> > > >
> > > > Kudos!
> > > >
> > > > >
> > > > > As you might guess it is frustrating for two reasons:
> > > > > 1) updated xfstests can be flaky (as the tests themselves are
> > updated,
> > > > > they add subtle required features, or regress from time to time)
> > > > > 2) test automation can run into resource constraints (VMs trying to
> > > > > run tests with less memory than might be optimal - especially those
> > > > > run against Samba with "VMs inside VMs").
> > > > >
> > > > > But the good news is that we have VERY good data on which tests pass
> > > > > to various servers (just as noted, need to update the external pages)
> > > > > and we should have even more data as we go through two weeks of SMB3
> > > > > testing events in late September.
> > > > >
> > > > > In general we want to test against all typical servers and have test
> > > > > targets setup for
> > > > >  - Samba (reasonably current stable) on ext4/xfs and Samba with btrfs
> > > > > (which has various optional extensions enabled in the server)
> > > > >  - Samba with POSIX Extensions
> > > > >  - Windows (and against both NTFS and REFS server file system)
> > > > > - Azure (Cloud0
> > > >
> > > > Thanks for sharing this!
> > > >
> > > > Besides server type, server configuration options and client mount
> > > > options also make this matrix even bigger..
> > > >
> > > > How that is handled in the buildbot ?
> > >
> > > We have several different targets, which maps to the local.config for
> > > xfstests so for every single test we run
> > > we also specify the different targets to run that for.
> > >
> > > Here is an example:
> > > all_tests = [
> > >         [ "cifs/100", "smb3azureseal"],
> > >         [ "cifs/101", "smb3multiuser"],
> > >         [ "generic/001", "smb3sign", "smb3"],
> > >         [ "generic/002", "smb3", "smb3sign", "smb21", "smb3samba"],
> > >         [ "generic/005", "smb3", "smb21", "smb3samba", "smb3sambabtrfs"],
> > >         [ "generic/006", "smb3"],
> > >         [ "generic/007", "smb3"],
> > >         [ "generic/010", "smb3"],
> > >         [ "generic/011", "smb3"],
> > >         [ "generic/013", "smb3samba"],
> > >         [ "generic/014", "smb3"],
> > >         [ "generic/020", "smb3samba"],
> > >         [ "generic/023", "smb3samba"],
> > >         [ "generic/024", "smb3", "smb3samba"],
> > >         [ "generic/028", "smb3", "smb3samba"],
> > > ...
> > >
> > >
> > > The first argument is which xfstest to run and the remainder of the
> > > arguments are a list of which local.config.* files to use.
> > > That way we can run a test against different servers and also
> > > different configurations.
> >
> > This is smart. :)
> >
> > > I try to keep the target names meaningful, so
> > >
> > > "smb3" just means a basic smb3   windows 2016 server running in a vm
> > > "smb3samba" is just a generic smb3 server with samba.
> > > etc etc.
> > >
> > >
> > > We started doing this upstream about a year ago which is when I
> > > started setting it up
> > > and now I am honest but since we do this for every pull request we
> > > send to linus,
> > > we have SIGNIFICANTLY increased the quality of cifs.ko.
> >
> > /Thumbs up!
> >
> > >
> > > We run quite a lot of tests nowadays and I always try to remind Steve
> > > to "please put a link to the buildbot result in the pull request"
> > > to bring awareness to our buildbot.
> > >
> > > For upstream, here is the most recent test:
> > >
> > http://smb3-test-rhel-75.southcentralus.cloudapp.azure.com/#/builders/2/builds/249
> > >
> > > cifs-testing is the most generic test target. We also have specific
> > > targets that are just for Azure, and other targets, but the aim is
> > > that the canonical cifs-target tests will all pass before we pass a
> > > pull request onto linus.
> > >
> > >
> > > Please contact me directly if you want more info about the buildbot. I
> > > have one setup internally at rh too that runs many, but not all, the
> > > tests we run upstream.
> >
> > Sure! Have replied to you. Thank you very much for sharing!
> >
> > If anybody resting cifs or I know that the expected setup and result of
> > every tests for you developers, it will save much time of investigation
> > and report false alarms.
> >
> > After several months working on cifs, I've got a feeling that cifs is
> > much different from other Linux filesystems. Maybe because of Windows
> > or the SMB protocols. I'm wondering that xfstests which is a typical
> > testsuite for Unix/Linux filesystems, maybe is not that suitable for
> > cifs. I agree with Steve that there are hundreds of tests suitable.
> > If, only if, :) if we could know how MS testing NTFS or SMB protocols,
> > would that help or more suitable?
> >
> >
> > Thanks!
> > Murphy
> >
> >
> > > Any help to maintain and expand the tests are super welcome.
> > >
> > >
> > > Rant-off
> > > ronnie sahlberg
> > >
> > > >
> > > > >
> > > > > But would really like to add test targets for other common servers
> > and
> > > > > in a perfect world would like to be able to have external test
> > > > > automation pull our trees periodically to run these (similar to what
> > > > > is done for the six targets above):
> > > > >   - Macs
> > > > >   - NetApp
> > > > > (and any others of interest to the community)
> > > > >
> > > > > Obviously xfstest has a few hundred tests which only make sense for
> > > > > local file systems with block devices, but there are hundreds of
> > > > > xfstests of value, and most should run on cifs.ko and we are working
> > > > > through them but already have a very good set running.
> > > >
> > > > Agree!  Thanks!
> > > >
> > > > Murphy
> > > >
> > > > >
> > > > > On Mon, Sep 9, 2019 at 5:50 PM Aurélien Aptel <aaptel@suse.com>
> > wrote:
> > > > > >
> > > > > > "Murphy Zhou" <jencce.kernel@gmail.com> writes:
> > > > > > > As $subject. Is this wiki being maintained ?
> > > > > > >
> > > > > > > Looks like the last update was in January 2019.
> > > > > > >
> > > > > > > https://wiki.samba.org/index.php/Xfstesting-cifs#Exclusion_files
> > > > > >
> > > > > > We have a buildbot running xfstests relatively often here
> > > > > >
> > > > > > http://smb3-test-rhel-75.southcentralus.cloudapp.azure.com/#/
> > > > > >
> > > > > > Each group has slightly different tests run, you can see which one
> > gets
> > > > > > run by clicking on a build:
> > > > > >
> > > > > >
> > http://smb3-test-rhel-75.southcentralus.cloudapp.azure.com/#/builders/2/builds/249
> > > > > >
> > > > > > Overall xfstests+cifs is very finicky and frustrating to get
> > working
> > > > > > reliably. Not to mention long. So good luck :)
> > > > > >
> > > > > > Cheers,
> > > > > > --
> > > > > > Aurélien Aptel / SUSE Labs Samba Team
> > > > > > GPG: 1839 CB5F 9F5B FB9B AA97  8C99 03C8 A49B 521B D5D3
> > > > > > SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409
> > Nürnberg, DE
> > > > > > GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 247165 (AG
> > München)
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Thanks,
> > > > >
> > > > > Steve
> >