linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Konstantin Komarov <almaz.alexandrovich@paragon-software.com>
To: "Darrick J. Wong" <djwong@kernel.org>, Theodore Ts'o <tytso@mit.edu>
Cc: Linus Torvalds <torvalds@linux-foundation.org>,
	Matthew Wilcox <willy@infradead.org>,
	"Leonidas P. Papadakos" <papadakospan@gmail.com>,
	"zajec5@gmail.com" <zajec5@gmail.com>,
	Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
	Hans de Goede <hdegoede@redhat.com>,
	linux-fsdevel <linux-fsdevel@vger.kernel.org>,
	Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
	Al Viro <viro@zeniv.linux.org.uk>
Subject: RE: [GIT PULL] vboxsf fixes for 5.14-1
Date: Thu, 5 Aug 2021 15:48:36 +0000	[thread overview]
Message-ID: <0e175373cef24e2abe76e203bb36d260@paragon-software.com> (raw)
In-Reply-To: <20210804010351.GM3601466@magnolia>

> From: Darrick J. Wong <djwong@kernel.org>
> Sent: Wednesday, August 4, 2021 4:04 AM
> To: Theodore Ts'o <tytso@mit.edu>
> Cc: Linus Torvalds <torvalds@linux-foundation.org>; Matthew Wilcox <willy@infradead.org>; Leonidas P. Papadakos
> <papadakospan@gmail.com>; Konstantin Komarov <almaz.alexandrovich@paragon-software.com>; zajec5@gmail.com; Greg Kroah-
> Hartman <gregkh@linuxfoundation.org>; Hans de Goede <hdegoede@redhat.com>; linux-fsdevel <linux-fsdevel@vger.kernel.org>;
> Linux Kernel Mailing List <linux-kernel@vger.kernel.org>; Al Viro <viro@zeniv.linux.org.uk>
> Subject: Re: [GIT PULL] vboxsf fixes for 5.14-1
> 
> On Tue, Aug 03, 2021 at 08:49:28PM -0400, Theodore Ts'o wrote:
> > On Tue, Aug 03, 2021 at 05:10:22PM -0700, Linus Torvalds wrote:
> > > The user-space FUSE thing does indeed work reasonably well.
> > >
> > > It performs horribly badly if you care about things like that, though.
> > >
> > > In fact, your own numbers kind of show that:
> > >
> > >   ntfs/default: 670 tests, 55 failures, 211 skipped, 34783 seconds
> > >   ntfs3/default: 664 tests, 67 failures, 206 skipped, 8106 seconds
> > >
> > > and that's kind of the point of ntfs3.
> >
> > Sure, although if you run fstress in parallel ntfs3 will lock up, the
> > system hard, and it has at least one lockdep deadlock complaints.
> > It's not up to me, but personally, I'd feel better if *someone* at
> > Paragon Software responded to Darrrick and my queries about their
> > quality assurance, and/or made commitments that they would at least
> > *try* to fix the problems that about 5 minutes of testing using
> > fstests turned up trivially.
> 
> <cough> Yes, my aim was to gauge their interest in actively QAing the
> driver's current problems so that it doesn't become one of the shabby
> Linux filesystem drivers, like <cough>ntfs.
> 
> Note I didn't even ask for a particular percentage of passing tests,
> because I already know that non-Unix filesystems fail the tests that
> look for the more Unix-specific behaviors.
> 
> I really only wanted them to tell /us/ what the baseline is.  IMHO the
> silence from them is a lot more telling.  Both generic/013 and
> generic/475 are basic "try to create files and read and write data to
> them" exercisers; failing those is a red flag.
> 

Hi Darrick and Theodore! First of all, apologies for the silence on your questions.
Let me please clarify and summarize the QA topic for you.

The main thing to outline is that: we have the number of autotests executed
for ntfs3 code. More specifically, we are using TeamCity as our CI tool, which
is handling autotests. Those are being executed against each commit to the
ntfs3 codebase.

Autotests are divided into the "promotion" levels, which are quite standard:
L0, L1, L2. Those levels have the division from the shortest "smoke" (L0)
to the longest set (L2). This we need to cover the ntfs3 functionality with
tests under given amount of time (feedback loop for L0 is minutes, while for
L2 is up to 24hrs).

As for suites we are using - it is the mix of open/well known suites:
- xfstests, ltp, pjd suite, fsx, dirstress, fstorture - those are of known utilites/suites
And number of internal autotests which were developed for covering various parts of
fs specs, regression autotests which are introduced to the infrastructure after bugfixes
and autotests written to test the driver operation on various data sets.

This approach is settled in Paragon for years, and ntfs3, from the first line of code written,
is being developed this way. You may refer the artifacts linked below, where the progress/coverage
during the last year is spoken by autotest results:

the 27th patch-series code (July'2021): 
https://dl.paragon-software.com/ntfs3/p27_tests.tar
25th (March'2021):
https://dl.paragon-software.com/ntfs3/p25_tests.tar
2nd (August, 2020):
https://dl.paragon-software.com/ntfs3/p2_tests.tar

Those are results on ntfs3 ran within the 'linux-next' (the most recent one given the tests start date)
As may be observed, we never skipped the "tests day" :)

There is a note should be provided on xfstests specifically. We have been using this suite
as a part of our autotests for several years already. However the suite originate for Linux
native file systems and a lot of cases are not applicable to the NTFS. This is one of the reasons
why some of "red-flag" failures are there (e.g. generic/475) - they were excluded at some point of time
and we've missed to enable it back when it was the time :)

Thank you all for this effort to run and look closer on our code, on the next patchset, the
91, 317 and 475 should be resolved. And now we are looking up to other excluded tests to find out more of such.

Hope this will resolve some of your concerns.

> --D
> 
> > I can even give them patches and configsto make it trivially easy for
> > them to run fstests using KVM or GCE....
> >
> > 				- Ted

  parent reply	other threads:[~2021-08-05 15:48 UTC|newest]

Thread overview: 42+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-07-13 10:45 Hans de Goede
2021-07-13 19:15 ` Linus Torvalds
2021-07-13 20:14   ` Al Viro
2021-07-13 20:18     ` Al Viro
2021-07-13 20:24       ` Randy Dunlap
2021-07-13 20:32         ` Al Viro
2021-07-13 21:43           ` Randy Dunlap
2021-07-14 10:50     ` Rafał Miłecki
2021-07-14 14:13       ` Christoph Hellwig
2021-07-14 14:51       ` Greg KH
2021-07-14 15:59         ` Rafał Miłecki
2021-07-14 16:05           ` Matthew Wilcox
2021-07-14 16:18             ` Rafał Miłecki
2021-07-15 21:50               ` Neal Gompa
2021-07-16 11:46               ` Leonidas P. Papadakos
2021-07-16 18:07                 ` Linus Torvalds
2021-07-30 15:55                   ` Konstantin Komarov
2021-07-30 17:23                     ` Paragon NTFSv3 (was Re: [GIT PULL] vboxsf fixes for 5.14-1) Linus Torvalds
2021-08-13 16:11                       ` Konstantin Komarov
2021-08-15 20:32                         ` Stephen Rothwell
2021-08-16  3:00                         ` Kari Argillander
2021-09-02 21:55                       ` Linus Torvalds
2021-08-03 22:48                   ` [GIT PULL] vboxsf fixes for 5.14-1 Theodore Ts'o
2021-08-03 23:44                     ` Matthew Wilcox
2021-08-04  0:04                       ` Theodore Ts'o
2021-08-04  0:10                         ` Linus Torvalds
2021-08-04  0:49                           ` Theodore Ts'o
2021-08-04  1:03                             ` Darrick J. Wong
2021-08-04  6:38                               ` Kari Argillander
2021-08-04 16:30                                 ` Theodore Ts'o
2021-08-05 15:48                               ` Konstantin Komarov [this message]
2021-08-10  7:02                                 ` Darrick J. Wong
2021-09-02 22:09                           ` NTFS testing (was: " Szabolcs Szakacsits
2021-09-03 17:48                             ` Eric Biggers
2021-09-03 21:17                               ` Szabolcs Szakacsits
2021-07-17 16:47                 ` Pali Rohár
2021-07-14 16:13           ` Darrick J. Wong
2021-07-14 16:18             ` Christoph Hellwig
2021-07-14 16:38               ` Gao Xiang
2021-07-14 20:03               ` Eric W. Biederman
2021-07-15 22:14               ` Darrick J. Wong
2021-07-13 19:17 ` pr-tracker-bot

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=0e175373cef24e2abe76e203bb36d260@paragon-software.com \
    --to=almaz.alexandrovich@paragon-software.com \
    --cc=djwong@kernel.org \
    --cc=gregkh@linuxfoundation.org \
    --cc=hdegoede@redhat.com \
    --cc=linux-fsdevel@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=papadakospan@gmail.com \
    --cc=torvalds@linux-foundation.org \
    --cc=tytso@mit.edu \
    --cc=viro@zeniv.linux.org.uk \
    --cc=willy@infradead.org \
    --cc=zajec5@gmail.com \
    --subject='RE: [GIT PULL] vboxsf fixes for 5.14-1' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).