[GIT PULL] vboxsf fixes for 5.14-1

[GIT PULL] vboxsf fixes for 5.14-1

[Hans de Goede]

Hi Linus,

Here is a pull-req with a set of patches fixing a vboxsf bug for 5.14
(I am the vboxsf maintainer).

Linus, sorry for sending this directly through you, instead of going
through some other tree, but trying to get this upstream through the
linux-fsdevel list / patch-review simply is not working.

This bugfix patch-set (3 preparation patches + 1 actual bugfix) fixes
a bug which is actually being hit by users in the wild, e.g. doing
a "git clone" on a vbox guest inside a shared-folder will fail
without his fix. Since you merge pull-req-s for a bunch of other
filesystems directly I'm hoping that you are willing to take this
one too.

This patch-set has been posted on the linux-fsdevel for the first
time on January 21th 2021, so almost 6 months ago and I've send
out several pings and a resend since then, but unfortunately
no-one on the linux-fsdevel list seems to have interest in /
time for reviewing vboxsf patches.

Regards,

Hans


The following changes since commit 6efb943b8616ec53a5e444193dccf1af9ad627b5:

  Linux 5.13-rc1 (2021-05-09 14:17:44 -0700)

are available in the Git repository at:

  git://git.kernel.org/pub/scm/linux/kernel/git/hansg/linux.git tags/vboxsf-v5.14-1

for you to fetch changes up to 52dfd86aa568e433b24357bb5fc725560f1e22d8:

  vboxsf: Add support for the atomic_open directory-inode op (2021-06-23 14:36:52 +0200)

----------------------------------------------------------------
Signed tag for a set of bugfixes for vboxsf for 5.14

This patch series adds support for the atomic_open
directory-inode op to vboxsf.

Note this is not just an enhancement this also fixes an actual issue
which users are hitting, see the commit message of the
"boxsf: Add support for the atomic_open directory-inode" patch.

----------------------------------------------------------------
Hans de Goede (4):
      vboxsf: Honor excl flag to the dir-inode create op
      vboxsf: Make vboxsf_dir_create() return the handle for the created file
      vboxsf: Add vboxsf_[create|release]_sf_handle() helpers
      vboxsf: Add support for the atomic_open directory-inode op

 fs/vboxsf/dir.c    | 76 ++++++++++++++++++++++++++++++++++++++++++++++--------
 fs/vboxsf/file.c   | 71 +++++++++++++++++++++++++++++++-------------------
 fs/vboxsf/vfsmod.h |  7 +++++
 3 files changed, 116 insertions(+), 38 deletions(-)

Re: [GIT PULL] vboxsf fixes for 5.14-1

[Linus Torvalds]

On Tue, Jul 13, 2021 at 3:45 AM Hans de Goede <hdegoede@redhat.com> wrote:
>
> Linus, sorry for sending this directly through you, instead of going
> through some other tree, but trying to get this upstream through the
> linux-fsdevel list / patch-review simply is not working.

Well, the filesystem maintainer sending their patches to me as a pull
request is actually the norm rather than the exception when it comes
to filesystems.

It's a bit different for drivers, but that's because while we have
multiple filesystems, we have multiple _thousand_ drivers, so on the
driver side I really don't want individual driver maintainers to all
send me their individual pull requests - that just wouldn't scale.

So for individual drivers, we have subsystem maintainers, but for
individual filesystems we generally don't.

(When something then touches the *common* vfs code, that's a different
thing - but something like this vboxsf thing this pull request looks
normal to me).

Even with a maintainer sending me pull requests I do obviously prefer
to see indications that other people have acked/tested/reviewed the
patches, but this is fairly small, simple and straightforward, and
absolutely nothing in this pull request makes me go "oh, that's
sketchy".

So no need to apologize at all, this all looks very regular.

               Linus

Re: [GIT PULL] vboxsf fixes for 5.14-1

[Al Viro]

On Tue, Jul 13, 2021 at 12:15:13PM -0700, Linus Torvalds wrote:
> On Tue, Jul 13, 2021 at 3:45 AM Hans de Goede <hdegoede@redhat.com> wrote:
> >
> > Linus, sorry for sending this directly through you, instead of going
> > through some other tree, but trying to get this upstream through the
> > linux-fsdevel list / patch-review simply is not working.
> 
> Well, the filesystem maintainer sending their patches to me as a pull
> request is actually the norm rather than the exception when it comes
> to filesystems.
> 
> It's a bit different for drivers, but that's because while we have
> multiple filesystems, we have multiple _thousand_ drivers, so on the
> driver side I really don't want individual driver maintainers to all
> send me their individual pull requests - that just wouldn't scale.
> 
> So for individual drivers, we have subsystem maintainers, but for
> individual filesystems we generally don't.
> 
> (When something then touches the *common* vfs code, that's a different
> thing - but something like this vboxsf thing this pull request looks
> normal to me).

To elaborate a bit - there's one case when I want it to go through
vfs.git, and that's when there's an interference between something
going on in vfs.git and the work done in filesystem.  Other than
that, I'm perfectly fine with maintainer sending pull request directly
to Linus (provided that I hadn't spotted something obviously wrong
in the series, of course, but that's not "I want it to go through
vfs.git" - that's "I don't want it in mainline until such and such
bug is resolved").

Re: [GIT PULL] vboxsf fixes for 5.14-1

[Al Viro]

On Tue, Jul 13, 2021 at 08:14:04PM +0000, Al Viro wrote:
> On Tue, Jul 13, 2021 at 12:15:13PM -0700, Linus Torvalds wrote:
> > On Tue, Jul 13, 2021 at 3:45 AM Hans de Goede <hdegoede@redhat.com> wrote:
> > >
> > > Linus, sorry for sending this directly through you, instead of going
> > > through some other tree, but trying to get this upstream through the
> > > linux-fsdevel list / patch-review simply is not working.
> > 
> > Well, the filesystem maintainer sending their patches to me as a pull
> > request is actually the norm rather than the exception when it comes
> > to filesystems.
> > 
> > It's a bit different for drivers, but that's because while we have
> > multiple filesystems, we have multiple _thousand_ drivers, so on the
> > driver side I really don't want individual driver maintainers to all
> > send me their individual pull requests - that just wouldn't scale.
> > 
> > So for individual drivers, we have subsystem maintainers, but for
> > individual filesystems we generally don't.
> > 
> > (When something then touches the *common* vfs code, that's a different
> > thing - but something like this vboxsf thing this pull request looks
> > normal to me).
> 
> To elaborate a bit - there's one case when I want it to go through
> vfs.git, and that's when there's an interference between something
> going on in vfs.git and the work done in filesystem.

Example: if there's a series changing calling conventions for some method
brewing in vfs.git and changes to filesystem's instance of that method
in the filesystem tree.  Then I'd rather it coordinated before either
gets merged.  It might be an invariant branch in either tree pulled by
both, it might be a straight pull into vfs.git and sorting the things out
there - depends upon the situation.

Re: [GIT PULL] vboxsf fixes for 5.14-1

[Randy Dunlap]

On 7/13/21 1:18 PM, Al Viro wrote:
> On Tue, Jul 13, 2021 at 08:14:04PM +0000, Al Viro wrote:
>> On Tue, Jul 13, 2021 at 12:15:13PM -0700, Linus Torvalds wrote:
>>> On Tue, Jul 13, 2021 at 3:45 AM Hans de Goede <hdegoede@redhat.com> wrote:
>>>>
...
>>>
>>> (When something then touches the *common* vfs code, that's a different
>>> thing - but something like this vboxsf thing this pull request looks
>>> normal to me).
>>
>> To elaborate a bit - there's one case when I want it to go through
>> vfs.git, and that's when there's an interference between something
>> going on in vfs.git and the work done in filesystem.
> 
> Example: if there's a series changing calling conventions for some method
> brewing in vfs.git and changes to filesystem's instance of that method
> in the filesystem tree.  Then I'd rather it coordinated before either
> gets merged.  It might be an invariant branch in either tree pulled by
> both, it might be a straight pull into vfs.git and sorting the things out
> there - depends upon the situation.
> 

Hi Al,

Where would you prefer for kernel-doc changes in fs/*.[ch] be merged?

E.g., from June 27:

https://lore.kernel.org/linux-fsdevel/20210628014613.11296-1-rdunlap@infradead.org/


thanks.
-- 
~Randy

Re: [GIT PULL] vboxsf fixes for 5.14-1

[Al Viro]

On Tue, Jul 13, 2021 at 01:24:06PM -0700, Randy Dunlap wrote:

> Hi Al,
> 
> Where would you prefer for kernel-doc changes in fs/*.[ch] be merged?
> 
> E.g., from June 27:
> 
> https://lore.kernel.org/linux-fsdevel/20210628014613.11296-1-rdunlap@infradead.org/

Umm...  I'd been under impression that kernel-doc stuff in general goes
through akpm, TBH.  I don't remember ever having a problem with your
patches of that sort; I can grab that kind of stuff, but if there's
an existing pipeline for that I'd just as well leave it there...

Re: [GIT PULL] vboxsf fixes for 5.14-1

[Randy Dunlap]

On 7/13/21 1:32 PM, Al Viro wrote:
> On Tue, Jul 13, 2021 at 01:24:06PM -0700, Randy Dunlap wrote:
> 
>> Hi Al,
>>
>> Where would you prefer for kernel-doc changes in fs/*.[ch] be merged?
>>
>> E.g., from June 27:
>>
>> https://lore.kernel.org/linux-fsdevel/20210628014613.11296-1-rdunlap@infradead.org/
> 
> Umm...  I'd been under impression that kernel-doc stuff in general goes
> through akpm, TBH.  I don't remember ever having a problem with your
> patches of that sort; I can grab that kind of stuff, but if there's
> an existing pipeline for that I'd just as well leave it there...
> 

Jon Corbet has merged some of them, but I'll be glad to send them to
akpm. Thanks.

{adding Cc: akpm for his info}

Re: [GIT PULL] vboxsf fixes for 5.14-1

[Rafał Miłecki]

Hi Alexander,

On 13.07.2021 22:14, Al Viro wrote:
> To elaborate a bit - there's one case when I want it to go through
> vfs.git, and that's when there's an interference between something
> going on in vfs.git and the work done in filesystem.  Other than
> that, I'm perfectly fine with maintainer sending pull request directly
> to Linus (provided that I hadn't spotted something obviously wrong
> in the series, of course, but that's not "I want it to go through
> vfs.git" - that's "I don't want it in mainline until such and such
> bug is resolved").

let me take this opportunity to ask about another filesystem.

Would you advise to send pull req for the fs/ntfs3 directly to Linus?

That is a pending filesystem that happens to be highly expected by many
Linux focused communities.


Paragon Software GmbH proved it's commitment by sending as many as 26
versions on it's patchset. The last set was send early April:

[PATCH v26 00/10] NTFS read-write driver GPL implementation by Paragon Software
https://marc.info/?l=linux-fsdevel&m=161738417018673&q=mbox
https://patchwork.kernel.org/project/linux-fsdevel/list/?series=460291


I'd say there weren't any serious issues raised since then.

One Tested-by, one maintenance question, one remainder, one clang-12
issue [0] [1].

It seems this filesystem only needs:
1. [Requirement] Adjusting to the meanwhile changed iov API [2]
2. [Clean up] Using fs/iomap/ helpers [3]


[0] https://marc.info/?t=161738428400012&r=1&w=2
[1] https://marc.info/?t=162143182800001&r=1&w=2
[2] https://marc.info/?l=linux-fsdevel&m=162607857810008&w=2
[3] https://marc.info/?l=linux-fsdevel&m=162152950315047&w=2

Re: [GIT PULL] vboxsf fixes for 5.14-1

[Christoph Hellwig]

> That is a pending filesystem that happens to be highly expected by many
> Linux focused communities.

This sounds like generated by a markov chain fed all kinds of marketeer
BS..

Independent of that resending a more than 3 month old page set usually
really helps to get some reviewer attention.

Re: [GIT PULL] vboxsf fixes for 5.14-1

[Greg KH]

On Wed, Jul 14, 2021 at 12:50:08PM +0200, Rafał Miłecki wrote:
> Hi Alexander,
> 
> On 13.07.2021 22:14, Al Viro wrote:
> > To elaborate a bit - there's one case when I want it to go through
> > vfs.git, and that's when there's an interference between something
> > going on in vfs.git and the work done in filesystem.  Other than
> > that, I'm perfectly fine with maintainer sending pull request directly
> > to Linus (provided that I hadn't spotted something obviously wrong
> > in the series, of course, but that's not "I want it to go through
> > vfs.git" - that's "I don't want it in mainline until such and such
> > bug is resolved").
> 
> let me take this opportunity to ask about another filesystem.
> 
> Would you advise to send pull req for the fs/ntfs3 directly to Linus?
> 
> That is a pending filesystem that happens to be highly expected by many
> Linux focused communities.
> 
> 
> Paragon Software GmbH proved it's commitment by sending as many as 26
> versions on it's patchset. The last set was send early April:
> 
> [PATCH v26 00/10] NTFS read-write driver GPL implementation by Paragon Software
> https://marc.info/?l=linux-fsdevel&m=161738417018673&q=mbox
> https://patchwork.kernel.org/project/linux-fsdevel/list/?series=460291
> 
> 
> I'd say there weren't any serious issues raised since then.
> 
> One Tested-by, one maintenance question, one remainder, one clang-12
> issue [0] [1].
> 
> It seems this filesystem only needs:
> 1. [Requirement] Adjusting to the meanwhile changed iov API [2]
> 2. [Clean up] Using fs/iomap/ helpers [3]

Why haven't those things been done and the patches resubmitted for
review?  Nothing we can do from our side when the developers don't want
to submit a new series, right?

thanks,

greg k-h

Re: [GIT PULL] vboxsf fixes for 5.14-1

[Rafał Miłecki]

On 14.07.2021 16:51, Greg KH wrote:
> On Wed, Jul 14, 2021 at 12:50:08PM +0200, Rafał Miłecki wrote:
>> Hi Alexander,
>>
>> On 13.07.2021 22:14, Al Viro wrote:
>>> To elaborate a bit - there's one case when I want it to go through
>>> vfs.git, and that's when there's an interference between something
>>> going on in vfs.git and the work done in filesystem.  Other than
>>> that, I'm perfectly fine with maintainer sending pull request directly
>>> to Linus (provided that I hadn't spotted something obviously wrong
>>> in the series, of course, but that's not "I want it to go through
>>> vfs.git" - that's "I don't want it in mainline until such and such
>>> bug is resolved").
>>
>> let me take this opportunity to ask about another filesystem.
>>
>> Would you advise to send pull req for the fs/ntfs3 directly to Linus?
>>
>> That is a pending filesystem that happens to be highly expected by many
>> Linux focused communities.
>>
>>
>> Paragon Software GmbH proved it's commitment by sending as many as 26
>> versions on it's patchset. The last set was send early April:
>>
>> [PATCH v26 00/10] NTFS read-write driver GPL implementation by Paragon Software
>> https://marc.info/?l=linux-fsdevel&m=161738417018673&q=mbox
>> https://patchwork.kernel.org/project/linux-fsdevel/list/?series=460291
>>
>>
>> I'd say there weren't any serious issues raised since then.
>>
>> One Tested-by, one maintenance question, one remainder, one clang-12
>> issue [0] [1].
>>
>> It seems this filesystem only needs:
>> 1. [Requirement] Adjusting to the meanwhile changed iov API [2]
>> 2. [Clean up] Using fs/iomap/ helpers [3]
> 
> Why haven't those things been done and the patches resubmitted for
> review?  Nothing we can do from our side when the developers don't want
> to submit a new series, right?

The real issue (broken compilation) has been pointed out 2 days ago and
is a result of a more recent commit. For months filesystem could be
pushed but it wasn't for unknown reason.

As for fs/iomap/ helpers it's unclear to me if that is really required
or could be worked on later as a clean up. Darrick joked his opinion on
using those helper is biased.

In short I'd say: missing feedback.

Re: [GIT PULL] vboxsf fixes for 5.14-1

[Matthew Wilcox]

On Wed, Jul 14, 2021 at 05:59:19PM +0200, Rafał Miłecki wrote:
> In short I'd say: missing feedback.

Uh, with all due respect: Fuck you.

I've provided feedback, and Paragon have done a fantastic job of
responding to it.  Pretending that the filesystem has simply been
ignored is hugely disrespectful of my time and those at Paragon.

I'm supportive of ntfs3 being included, FWIW.  It looks to be
in much better shape than the existing fs/ntfs.

Re: [GIT PULL] vboxsf fixes for 5.14-1

[Rafał Miłecki]

On 14.07.2021 18:05, Matthew Wilcox wrote:
> On Wed, Jul 14, 2021 at 05:59:19PM +0200, Rafał Miłecki wrote:
>> In short I'd say: missing feedback.
> 
> Uh, with all due respect: Fuck you.
> 
> I've provided feedback, and Paragon have done a fantastic job of
> responding to it.  Pretending that the filesystem has simply been
> ignored is hugely disrespectful of my time and those at Paragon.
> 
> I'm supportive of ntfs3 being included, FWIW.  It looks to be
> in much better shape than the existing fs/ntfs.

Thanks you for kind words before even trying to clarify the situation.

What I meant (but failed to write clearly) is missing feedback on *the
latest* patchset.

I highly appreciate everyone who took time and helped polishing that
filesystem to its latest form.

Re: [GIT PULL] vboxsf fixes for 5.14-1

[Neal Gompa]

On Wed, Jul 15, 2021 at 12:18 PM "Rafał Miłecki" <zajec5@gmail.com> wrote:
>
> What I meant (but failed to write clearly) is missing feedback on *the
> latest* patchset.
> 
> I highly appreciate everyone who took time and helped polishing that
> filesystem to its latest form.

As the person who tested the latest ntfs3 patchset, and had tested
many of those iterations in the past, I would really like to see
this *finally* land in Linux 5.14.

However, I get the feeling it's not going to make it for 5.14 *or*
5.15, and it seems like Paragon became discouraged by the lack of
feedback on the latest revision.

I know that compared to all you awesome folks, I'm just a lowly
user, but it's been frustrating to see nothing happen for months
with something that has a seriously high impact for a lot of people.

It's a shame, because the ntfs3 driver is miles better than the
current ntfs one, and is a solid replacement for the unmaintained
ntfs-3g FUSE implementation.


-- 
真実はいつも一つ！/ Always, there's only one truth!

Re: [GIT PULL] vboxsf fixes for 5.14-1

[Leonidas P. Papadakos]

On the subject of this merge, I have to stress that this ntfs driver (fs/ntfs3, which would probably replace fs/ntfs, right?) is an important feature, from a user perspective. It would mean having good support for a cross-platform filesystem suitable for hard drives. exFAT is welcome, but it's a simple filesystem for flash storage.

Relevant xkcd: https://xkcd.com/619/

I think Paragon has been very good about supporting this driver with 26 patchsets and in my mind it would be suitable for staging. I've seen the discussion slow down since May, and I've been excited to see this merged. This driver is already in a much better feature state than the old ntfs driver from 2001.

Re: [GIT PULL] vboxsf fixes for 5.14-1

[Linus Torvalds]

On Fri, Jul 16, 2021 at 4:49 AM Leonidas P. Papadakos
<papadakospan@gmail.com> wrote:
>
> This driver is already in a much better feature state than the old ntfs driver from 2001.

If the new ntfs code has acks from people - and it sounds like it did
get them - and Paragon is expected to be the maintainer of it, then I
think Paragon should just make a git pull request for it.

That's assuming that it continues to be all in just fs/ntfs3/ (plus
fs/Kconfig, fs/Makefile and MAINTAINERS entries and whatever
documentation) and there are no other system-wide changes. Which I
don't think it had.

We simply don't have anybody to funnel new filesystems - the fsdevel
mailing list is good for comments and get feedback, but at some point
somebody just needs to actually submit it, and that's not what fsdevel
ends up doing.

The argument that "it's already in a much better state than the old
ntfs driver" may not be a very strong technical argument (not because
of any Paragon problems - just because the old ntfs driver is not
great), but it _is_ a fairly strong argument for merging the new one
from Paragon.

And I don't think there has been any huge _complaints_ about the code,
and I don't think there's been any sign that being outside the kernel
helps.

               Linus

RE: [GIT PULL] vboxsf fixes for 5.14-1

[Konstantin Komarov]

> From: Linus Torvalds <torvalds@linux-foundation.org>
> Sent: Friday, July 16, 2021 9:07 PM
> 
> > This driver is already in a much better feature state than the old ntfs driver from 2001.
> 
> If the new ntfs code has acks from people - and it sounds like it did
> get them - and Paragon is expected to be the maintainer of it, then I
> think Paragon should just make a git pull request for it.
> 
> That's assuming that it continues to be all in just fs/ntfs3/ (plus
> fs/Kconfig, fs/Makefile and MAINTAINERS entries and whatever
> documentation) and there are no other system-wide changes. Which I
> don't think it had.
>
Hi Linus! Great to hear your feedback and clarifications on our ntfs3 code.
Greatly appreciated.
From our side, we can confirm that we will be maintaining this implementation.
Also, this is planned to be in fs/ntfs3 at this point, at least for now - until the code
and implementation becomes known and trusted within community. And then
we can discuss if it should replace the fs/ntfs and when it's convenient to do.
>
> We simply don't have anybody to funnel new filesystems - the fsdevel
> mailing list is good for comments and get feedback, but at some point
> somebody just needs to actually submit it, and that's not what fsdevel
> ends up doing.
>
Thanks for this clarification as well. This piece of infromation has not been really 
clear for us until now.
We've just sent the 27th patch series which fixes to the buildability against
current linux-next. And we'll need several days to prepare a proper pull request
before sending it to you.
 
> The argument that "it's already in a much better state than the old
> ntfs driver" may not be a very strong technical argument (not because
> of any Paragon problems - just because the old ntfs driver is not
> great), but it _is_ a fairly strong argument for merging the new one
> from Paragon.
> 
> And I don't think there has been any huge _complaints_ about the code,
> and I don't think there's been any sign that being outside the kernel
> helps.
> 
>                Linus

Paragon NTFSv3 (was Re: [GIT PULL] vboxsf fixes for 5.14-1)

[Linus Torvalds]

On Fri, Jul 30, 2021 at 8:55 AM Konstantin Komarov
<almaz.alexandrovich@paragon-software.com> wrote:
>
> We've just sent the 27th patch series which fixes to the buildability against
> current linux-next. And we'll need several days to prepare a proper pull request
> before sending it to you.

Well, I won't pull until the next merge window opens anyway (about a
month away). But it would be good to have your tree in linux-next for
at least a couple of weeks before that happens.

Added Stephen to the participants list as a heads-up for him - letting
him know where to fetch the git tree from will allow that to happen if
you haven't done so already.

The one other thing I do want when there's big new pieces like this
being added is to ask you to make sure that everything is signed-off
properly, and that there is no internal confusion about the GPLv2
inside Paragon, and that any legal people etc are all aware of this
all and are on board. The last thing we want to see is some "oops, we
didn't mean to do this" brouhaha six months later.

I doubt that's an issue, considering how public this all has been, but
I just wanted to mention it just to be very obvious about it.

                  Linus

RE: Paragon NTFSv3 (was Re: [GIT PULL] vboxsf fixes for 5.14-1)

[Konstantin Komarov]

> From: Linus Torvalds <torvalds@linux-foundation.org>
> Sent: Friday, July 30, 2021 8:24 PM
> To: Konstantin Komarov <almaz.alexandrovich@paragon-software.com>; Stephen Rothwell <sfr@canb.auug.org.au>
> Cc: Leonidas P. Papadakos <papadakospan@gmail.com>; zajec5@gmail.com; Darrick J. Wong <djwong@kernel.org>; Greg Kroah-
> Hartman <gregkh@linuxfoundation.org>; Hans de Goede <hdegoede@redhat.com>; linux-fsdevel <linux-fsdevel@vger.kernel.org>;
> Linux Kernel Mailing List <linux-kernel@vger.kernel.org>; Al Viro <viro@zeniv.linux.org.uk>; Matthew Wilcox <willy@infradead.org>
> Subject: Paragon NTFSv3 (was Re: [GIT PULL] vboxsf fixes for 5.14-1)
> 
> On Fri, Jul 30, 2021 at 8:55 AM Konstantin Komarov
> <almaz.alexandrovich@paragon-software.com> wrote:
> >
> > We've just sent the 27th patch series which fixes to the buildability against
> > current linux-next. And we'll need several days to prepare a proper pull request
> > before sending it to you.
> 
> Well, I won't pull until the next merge window opens anyway (about a
> month away). But it would be good to have your tree in linux-next for
> at least a couple of weeks before that happens.
> 
> Added Stephen to the participants list as a heads-up for him - letting
> him know where to fetch the git tree from will allow that to happen if
> you haven't done so already.
> 

Thanks for this clarification, Linus!
Stephen, please find the tree here:
https://github.com/Paragon-Software-Group/linux-ntfs3.git
It is the fork from 5.14-rc5 tag with ntfs3 patches applied.
Also, the latest changes
- fix some generic/XYZ xfstests, which were discussed
with Theodore, Darrick and others
- updates the MAINTAINERS with mailing list (also added to CC here) and scm tree link.

Please let me know if additional changes requred to get fetched into linux-next.

> The one other thing I do want when there's big new pieces like this
> being added is to ask you to make sure that everything is signed-off
> properly, and that there is no internal confusion about the GPLv2
> inside Paragon, and that any legal people etc are all aware of this
> all and are on board. The last thing we want to see is some "oops, we
> didn't mean to do this" brouhaha six months later.
> 
> I doubt that's an issue, considering how public this all has been, but
> I just wanted to mention it just to be very obvious about it.
> 
>                   Linus
Indeed, there is no internal confusion about the GPLv2 and we mean to make this contribution.

Best regards,
Konstantin.

Re: Paragon NTFSv3 (was Re: [GIT PULL] vboxsf fixes for 5.14-1)

[Stephen Rothwell]

[-- Attachment #1: Type: text/plain, Size: 3031 bytes --]

Hi Konstantin,

On Fri, 13 Aug 2021 16:11:10 +0000 Konstantin Komarov <almaz.alexandrovich@paragon-software.com> wrote:
>
> > From: Linus Torvalds <torvalds@linux-foundation.org>
> > Sent: Friday, July 30, 2021 8:24 PM
> > To: Konstantin Komarov <almaz.alexandrovich@paragon-software.com>; Stephen Rothwell <sfr@canb.auug.org.au>
> > Cc: Leonidas P. Papadakos <papadakospan@gmail.com>; zajec5@gmail.com; Darrick J. Wong <djwong@kernel.org>; Greg Kroah-
> > Hartman <gregkh@linuxfoundation.org>; Hans de Goede <hdegoede@redhat.com>; linux-fsdevel <linux-fsdevel@vger.kernel.org>;
> > Linux Kernel Mailing List <linux-kernel@vger.kernel.org>; Al Viro <viro@zeniv.linux.org.uk>; Matthew Wilcox <willy@infradead.org>
> > Subject: Paragon NTFSv3 (was Re: [GIT PULL] vboxsf fixes for 5.14-1)
> > 
> > On Fri, Jul 30, 2021 at 8:55 AM Konstantin Komarov
> > <almaz.alexandrovich@paragon-software.com> wrote:  
> > >
> > > We've just sent the 27th patch series which fixes to the buildability against
> > > current linux-next. And we'll need several days to prepare a proper pull request
> > > before sending it to you.  
> > 
> > Well, I won't pull until the next merge window opens anyway (about a
> > month away). But it would be good to have your tree in linux-next for
> > at least a couple of weeks before that happens.
> > 
> > Added Stephen to the participants list as a heads-up for him - letting
> > him know where to fetch the git tree from will allow that to happen if
> > you haven't done so already.
> >   
> 
> Thanks for this clarification, Linus!
> Stephen, please find the tree here:
> https://github.com/Paragon-Software-Group/linux-ntfs3.git
> It is the fork from 5.14-rc5 tag with ntfs3 patches applied.
> Also, the latest changes
> - fix some generic/XYZ xfstests, which were discussed
> with Theodore, Darrick and others
> - updates the MAINTAINERS with mailing list (also added to CC here) and scm tree link.
> 
> Please let me know if additional changes requred to get fetched into linux-next.

Added from today.  It looks good, we will see how it goes when integrated/built.

Thanks for adding your subsystem tree as a participant of linux-next.  As
you may know, this is not a judgement of your code.  The purpose of
linux-next is for integration testing and to lower the impact of
conflicts between subsystems in the next merge window. 

You will need to ensure that the patches/commits in your tree/series have
been:
     * submitted under GPL v2 (or later) and include the Contributor's
        Signed-off-by,
     * posted to the relevant mailing list,
     * reviewed by you (or another maintainer of your subsystem tree),
     * successfully unit tested, and 
     * destined for the current or next Linux merge window.

Basically, this should be just what you would send to Linus (or ask him
to fetch).  It is allowed to be rebased if you deem it necessary.

-- 
Cheers,
Stephen Rothwell 
sfr@canb.auug.org.au

-- 
Cheers,
Stephen Rothwell

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Paragon NTFSv3 (was Re: [GIT PULL] vboxsf fixes for 5.14-1)

[Kari Argillander]

On Fri, Aug 13, 2021 at 04:11:10PM +0000, Konstantin Komarov wrote:
> > From: Linus Torvalds <torvalds@linux-foundation.org>
> > Sent: Friday, July 30, 2021 8:24 PM
> > To: Konstantin Komarov <almaz.alexandrovich@paragon-software.com>; Stephen Rothwell <sfr@canb.auug.org.au>
> > Cc: Leonidas P. Papadakos <papadakospan@gmail.com>; zajec5@gmail.com; Darrick J. Wong <djwong@kernel.org>; Greg Kroah-
> > Hartman <gregkh@linuxfoundation.org>; Hans de Goede <hdegoede@redhat.com>; linux-fsdevel <linux-fsdevel@vger.kernel.org>;
> > Linux Kernel Mailing List <linux-kernel@vger.kernel.org>; Al Viro <viro@zeniv.linux.org.uk>; Matthew Wilcox <willy@infradead.org>
> > Subject: Paragon NTFSv3 (was Re: [GIT PULL] vboxsf fixes for 5.14-1)
> > 
> > On Fri, Jul 30, 2021 at 8:55 AM Konstantin Komarov
> > <almaz.alexandrovich@paragon-software.com> wrote:
> > >
> > > We've just sent the 27th patch series which fixes to the buildability against
> > > current linux-next. And we'll need several days to prepare a proper pull request
> > > before sending it to you.
> > 
> > Well, I won't pull until the next merge window opens anyway (about a
> > month away). But it would be good to have your tree in linux-next for
> > at least a couple of weeks before that happens.
> > 
> > Added Stephen to the participants list as a heads-up for him - letting
> > him know where to fetch the git tree from will allow that to happen if
> > you haven't done so already.
> > 
> 
> Thanks for this clarification, Linus!
> Stephen, please find the tree here:
> https://github.com/Paragon-Software-Group/linux-ntfs3.git
> It is the fork from 5.14-rc5 tag with ntfs3 patches applied.
> Also, the latest changes
> - fix some generic/XYZ xfstests, which were discussed
> with Theodore, Darrick and others
> - updates the MAINTAINERS with mailing list (also added to CC here) and scm tree link.

Can you please send this also as normal patch series to mailing lists so
we can comment there.

One thing a like to ask you to do before that is add reviewed-by tag
and signed-off-by tag as stated here
https://lore.kernel.org/linux-fsdevel/20210810054637.aap4zuiiparfl2gq@kari-VirtualBox/
and
https://lore.kernel.org/linux-fsdevel/20210810074740.mkjcow2inyjaakch@kari-VirtualBox/

> Please let me know if additional changes requred to get fetched into linux-next.
> 
> > The one other thing I do want when there's big new pieces like this
> > being added is to ask you to make sure that everything is signed-off
> > properly, and that there is no internal confusion about the GPLv2
> > inside Paragon, and that any legal people etc are all aware of this
> > all and are on board. The last thing we want to see is some "oops, we
> > didn't mean to do this" brouhaha six months later.
> > 
> > I doubt that's an issue, considering how public this all has been, but
> > I just wanted to mention it just to be very obvious about it.
> > 
> >                   Linus
> Indeed, there is no internal confusion about the GPLv2 and we mean to make this contribution.
> 
> Best regards,
> Konstantin.

Re: Paragon NTFSv3 (was Re: [GIT PULL] vboxsf fixes for 5.14-1)

[Linus Torvalds]

On Fri, Jul 30, 2021 at 10:23 AM Linus Torvalds
<torvalds@linux-foundation.org> wrote:
>
> Well, I won't pull until the next merge window opens anyway (about a
> month away). But it would be good to have your tree in linux-next for
> at least a couple of weeks before that happens.
>
> Added Stephen to the participants list as a heads-up for him - letting
> him know where to fetch the git tree from will allow that to happen if
> you haven't done so already.

Ok, so I've merged the biggest pieces of this merge window, and I
haven't actually seen a NTFSv3 pull request yet.

I wonder if you expected that being in linux-next just "automatically"
causes the pull to happen, because that's not the case. We often have
things "brewing" in linux-next for a while, and it's there for testing
but not necessarily ready for prime time.

So linux-next is a preparatory thing, not a "this will get merged"

So to actually merge things, I will expect to get an explicit pull
request with the usual diffstat and shortlog, to show that yes, you
really think it's all good, and it's ready to merge.

Don't worry about - and don't try to fix - merge conflicts with
possible other work that has been going on. Stephen fixes it for
linux-next and makes people aware of it, and I want to _know_ about
them, but I will then handle and re-do the merge conflicts myself
based on what I have actually merged up to that point.

And of course, the other side of that is that if linux-next uncovered
other issues, or if there are things holding things up, please _don't_
feel obligated to send me a pull request. There is always the next
merge window.

            Linus

Re: [GIT PULL] vboxsf fixes for 5.14-1

[Theodore Ts'o]

On Fri, Jul 16, 2021 at 11:07:29AM -0700, Linus Torvalds wrote:
> 
> The argument that "it's already in a much better state than the old
> ntfs driver" may not be a very strong technical argument (not because
> of any Paragon problems - just because the old ntfs driver is not
> great), but it _is_ a fairly strong argument for merging the new one
> from Paragon.

I'm not 100% sure that "it's better than the old driver", actually.
Konstantin has not been responding to Darrick and my questions about
what sort of QA and testing they were doing.

So over the weekend, I decided to take efforts into my own hands, and
made the relatively simple changes to fstests needed to add support
for ntfs and ntfs3 file systems.  The results show that the number
fstests failures in ntfs3 is 23% *more* than ntfs.  This includes a
potential deadlock bug, and generic/475 reliably livelocking.  Ntfs3
is also currently not container compatible, because it's not properly
handling user namespaces.

For more details, please see [1] and [2] for the complete set of test
artifacts.

[1] https://www.kernel.org/pub/linux/kernel/people/tytso/fstests-results/results-ntfs-2021-08-02.tar.xz
[2] https://www.kernel.org/pub/linux/kernel/people/tytso/fstests-results/results-ntfs3-2021-08-03.tar.xz

> And I don't think there has been any huge _complaints_ about the code,
> and I don't think there's been any sign that being outside the kernel
> helps.

Historically, the file system community at large have pushed for a
fairly high bar before a file system is merged into the kernel,
because there was a concern that once a file system got dumped into
fs/ if the maintainers weren't going to commit to continuous
improvement of their file system --- the only leverage we might have
is what effectively amounts to "hazing" to make sure that the
propsective maintainers would actually be serious about continuing to
work on the file system.

One argument for why this should be the case is that unlike a dodgy
driver that "just" causes the kernel to crash, if data ends up getting
corrupted, simply rebooting won't recover the user's data.  And once a
file system is added to mainline, it's a lot harder to remove it if it
turns out to be buggy as all h*ck. 

It's not clear this has been an effective strategy.  And there are
other ways we could handle an abandonware file system --- we could
liberally festoon its Kconfig with warnings and printk "DANGER WILL
ROBINSON" messages when someone attempts to use a dodgy file system in
mainline.  But I think whatever rationale we give for accepting --- or
holding off --- on ntfs3, we should also think about how we should be
handling requests from other file systems such as bcachefs, reiserfs4,
tux3, etc.

Maybe this should be a maintainers summit discussion topic?  I
dunno....

					- Ted

P.S.  Here is the summary of the test results of running ntfs and
ntfs3 on 5.14-rc2, with the latest ntfs3 patches applied.  Note that
for ntfs3, I had to manually exclude generic/475, since running that
test will cause the kernel to lock up and prevent the rest of the
tesets from running.  So that's really 68 fstests failures for ntfs3,
versus 55 fstests failures for ntfs.

And it's really not the absolute number of test failures that bothers
me, so much as the complete radio silence from Konstantin after you've
indicated that you are willing to take the ntfs3 merge request.  It
increases the concerns I personally have that ntfs3 might end up
becoming abandonware after it's been accepted.

ntfs/default: 670 tests, 55 failures, 211 skipped, 34783 seconds
  Failures: generic/003 generic/035 generic/053 generic/062 
    generic/087 generic/088 generic/093 generic/097 generic/099 
    generic/102 generic/105 generic/123 generic/126 generic/193 
    generic/226 generic/237 generic/260 generic/294 generic/306 
    generic/307 generic/314 generic/317 generic/318 generic/319 
    generic/321 generic/355 generic/375 generic/378 generic/409 
    generic/410 generic/411 generic/416 generic/423 generic/424 
    generic/426 generic/427 generic/441 generic/444 generic/452 
    generic/466 generic/467 generic/475 generic/477 generic/500 
    generic/525 generic/529 generic/545 generic/547 generic/553 
    generic/555 generic/564 generic/589 generic/597 generic/629 
    generic/631 

ntfs3/default: 664 tests, 67 failures, 206 skipped, 8106 seconds
  Failures: generic/013 generic/015 generic/034 generic/039 
    generic/040 generic/041 generic/056 generic/057 generic/065 
    generic/066 generic/073 generic/083 generic/090 generic/091 
    generic/092 generic/094 generic/101 generic/102 generic/104 
    generic/106 generic/107 generic/130 generic/225 generic/226 
    generic/228 generic/240 generic/258 generic/263 generic/311 
    generic/317 generic/320 generic/321 generic/322 generic/335 
    generic/336 generic/341 generic/342 generic/343 generic/348 
    generic/360 generic/361 generic/371 generic/376 generic/416 
    generic/427 generic/441 generic/476 generic/480 generic/481 
    generic/483 generic/489 generic/498 generic/502 generic/510 
    generic/512 generic/520 generic/526 generic/527 generic/534 
    generic/538 generic/547 generic/551 generic/552 generic/557 
    generic/598 generic/631 generic/640 

Other file systems for reference:

f2fs/default: 646 tests, 13 failures, 154 skipped, 1812 seconds
  Failures: generic/018 generic/026 generic/050 generic/064
    generic/066 generic/103 generic/219 generic/260 generic/342
    generic/502 generic/506 generic/526 generic/527

btrfs/default: 1075 tests, 8 failures, 219 skipped, 9143 seconds
  Failures: btrfs/012 btrfs/154 btrfs/219 btrfs/220 btrfs/235
    btrfs/241 generic/260 shared/298

xfs/4k: 922 tests, 1 failures, 136 skipped, 5452 seconds
  Failures: xfs/506

ext4/4k: 504 tests, 0 failures, 25 skipped, 6877 seconds

nfs/filestore_v3: 743 tests, 1 failures, 307 skipped, 9261 seconds
  Failures: generic/551

(Note: GCE Filestore uses a Linux kernel so this is testing the nfsv3
client versus a Linux nfsv3 server --- I think the Filestore kernel is
currently using 5.4.129 if I remember correctly.)

Re: [GIT PULL] vboxsf fixes for 5.14-1

[Matthew Wilcox]

On Tue, Aug 03, 2021 at 06:48:36PM -0400, Theodore Ts'o wrote:
> So over the weekend, I decided to take efforts into my own hands, and
> made the relatively simple changes to fstests needed to add support
> for ntfs and ntfs3 file systems.  The results show that the number
> fstests failures in ntfs3 is 23% *more* than ntfs.  This includes a
> potential deadlock bug, and generic/475 reliably livelocking.  Ntfs3
> is also currently not container compatible, because it's not properly
> handling user namespaces.

I don't understand how so many ntfs-classic xfstests pass:

config NTFS_RW
        bool "NTFS write support"
        depends on NTFS_FS
        help
          This enables the partial, but safe, write support in the NTFS driver.

          The only supported operation is overwriting existing files, without
          changing the file length.  No file or directory creation, deletion or
          renaming is possible.  Note only non-resident files can be written to
          so you may find that some very small files (<500 bytes or so) cannot
          be written to.

Are the tests really passing, or just claiming to pass?

Re: [GIT PULL] vboxsf fixes for 5.14-1

[Theodore Ts'o]

On Wed, Aug 04, 2021 at 12:44:38AM +0100, Matthew Wilcox wrote:
> 
> I don't understand how so many ntfs-classic xfstests pass:
> 
> config NTFS_RW
>         bool "NTFS write support"
>         depends on NTFS_FS
>         help
>           This enables the partial, but safe, write support in the NTFS driver.
> 
>           The only supported operation is overwriting existing files, without
>           changing the file length.  No file or directory creation, deletion or
>           renaming is possible.  Note only non-resident files can be written to
>           so you may find that some very small files (<500 bytes or so) cannot
>           be written to.
> 
> Are the tests really passing, or just claiming to pass?

This was the ntfs provided by the Debian package ntfs-3g (which is the
only source of a mkfs.ntfs that I could find, BTW).  This is a
fuse-based ntfs, not the in-kernel ntfs file system.  Apologies for
not making that clear.

<tytso.root@cwcc> {/usr/projects/linux/ext4}, level 2   (ntfs3)
1003# mkfs.ntfs /dev/cwcc-vg/scratch
Cluster size has been automatically set to 4096 bytes.
Initializing device with zeroes: 100% - Done.
Creating NTFS volume structures.
mkntfs completed successfully. Have a nice day.
<tytso.root@cwcc> {/usr/projects/linux/ext4}, level 2   (ntfs3)
1004# mount -t ntfs /dev/cwcc-vg/scratch /mnt
<tytso.root@cwcc> {/usr/projects/linux/ext4}, level 2   (ntfs3)
1005# grep /mnt /proc/mounts
/dev/mapper/cwcc--vg-scratch /mnt fuseblk rw,relatime,user_id=0,group_id=0,allow_other,blksize=4096 0 0

TBH, I had forgotten that we had an in-kernel ntfs implementation.
Whenver I've ever needed to access ntfs files, I've always used the
ntfs-3g FUSE package.

			     	  	  - Ted

Re: [GIT PULL] vboxsf fixes for 5.14-1

[Linus Torvalds]

On Tue, Aug 3, 2021 at 5:04 PM Theodore Ts'o <tytso@mit.edu> wrote:
>
> TBH, I had forgotten that we had an in-kernel ntfs implementation.

Well, that's the one we are comparing to, so forgetting it is a bit of
an oversight.

> Whenver I've ever needed to access ntfs files, I've always used the
> ntfs-3g FUSE package.

The user-space FUSE thing does indeed work reasonably well.

It performs horribly badly if you care about things like that, though.

In fact, your own numbers kind of show that:

  ntfs/default: 670 tests, 55 failures, 211 skipped, 34783 seconds
  ntfs3/default: 664 tests, 67 failures, 206 skipped, 8106 seconds

and that's kind of the point of ntfs3.

           Linus

Re: [GIT PULL] vboxsf fixes for 5.14-1

[Theodore Ts'o]

On Tue, Aug 03, 2021 at 05:10:22PM -0700, Linus Torvalds wrote:
> The user-space FUSE thing does indeed work reasonably well.
> 
> It performs horribly badly if you care about things like that, though.
> 
> In fact, your own numbers kind of show that:
> 
>   ntfs/default: 670 tests, 55 failures, 211 skipped, 34783 seconds
>   ntfs3/default: 664 tests, 67 failures, 206 skipped, 8106 seconds
> 
> and that's kind of the point of ntfs3.

Sure, although if you run fstress in parallel ntfs3 will lock up, the
system hard, and it has at least one lockdep deadlock complaints.
It's not up to me, but personally, I'd feel better if *someone* at
Paragon Software responded to Darrrick and my queries about their
quality assurance, and/or made commitments that they would at least
*try* to fix the problems that about 5 minutes of testing using
fstests turned up trivially.

I can even give them patches and configsto make it trivially easy for
them to run fstests using KVM or GCE....

				- Ted

Re: [GIT PULL] vboxsf fixes for 5.14-1

[Darrick J. Wong]

On Tue, Aug 03, 2021 at 08:49:28PM -0400, Theodore Ts'o wrote:
> On Tue, Aug 03, 2021 at 05:10:22PM -0700, Linus Torvalds wrote:
> > The user-space FUSE thing does indeed work reasonably well.
> > 
> > It performs horribly badly if you care about things like that, though.
> > 
> > In fact, your own numbers kind of show that:
> > 
> >   ntfs/default: 670 tests, 55 failures, 211 skipped, 34783 seconds
> >   ntfs3/default: 664 tests, 67 failures, 206 skipped, 8106 seconds
> > 
> > and that's kind of the point of ntfs3.
> 
> Sure, although if you run fstress in parallel ntfs3 will lock up, the
> system hard, and it has at least one lockdep deadlock complaints.
> It's not up to me, but personally, I'd feel better if *someone* at
> Paragon Software responded to Darrrick and my queries about their
> quality assurance, and/or made commitments that they would at least
> *try* to fix the problems that about 5 minutes of testing using
> fstests turned up trivially.

<cough> Yes, my aim was to gauge their interest in actively QAing the
driver's current problems so that it doesn't become one of the shabby
Linux filesystem drivers, like <cough>ntfs.

Note I didn't even ask for a particular percentage of passing tests,
because I already know that non-Unix filesystems fail the tests that
look for the more Unix-specific behaviors.

I really only wanted them to tell /us/ what the baseline is.  IMHO the
silence from them is a lot more telling.  Both generic/013 and
generic/475 are basic "try to create files and read and write data to
them" exercisers; failing those is a red flag.

--D

> I can even give them patches and configsto make it trivially easy for
> them to run fstests using KVM or GCE....
> 
> 				- Ted

Re: [GIT PULL] vboxsf fixes for 5.14-1

[Kari Argillander]

On Tue, Aug 03, 2021 at 06:03:51PM -0700, Darrick J. Wong wrote:
> On Tue, Aug 03, 2021 at 08:49:28PM -0400, Theodore Ts'o wrote:
> > On Tue, Aug 03, 2021 at 05:10:22PM -0700, Linus Torvalds wrote:
> > > The user-space FUSE thing does indeed work reasonably well.
> > > 
> > > It performs horribly badly if you care about things like that, though.
> > > 
> > > In fact, your own numbers kind of show that:
> > > 
> > >   ntfs/default: 670 tests, 55 failures, 211 skipped, 34783 seconds
> > >   ntfs3/default: 664 tests, 67 failures, 206 skipped, 8106 seconds
> > > 
> > > and that's kind of the point of ntfs3.
> > 
> > Sure, although if you run fstress in parallel ntfs3 will lock up, the
> > system hard, and it has at least one lockdep deadlock complaints.
> > It's not up to me, but personally, I'd feel better if *someone* at
> > Paragon Software responded to Darrrick and my queries about their
> > quality assurance, and/or made commitments that they would at least
> > *try* to fix the problems that about 5 minutes of testing using
> > fstests turned up trivially.
> 
> <cough> Yes, my aim was to gauge their interest in actively QAing the
> driver's current problems so that it doesn't become one of the shabby
> Linux filesystem drivers, like <cough>ntfs.
> 
> Note I didn't even ask for a particular percentage of passing tests,
> because I already know that non-Unix filesystems fail the tests that
> look for the more Unix-specific behaviors.
> 
> I really only wanted them to tell /us/ what the baseline is.  IMHO the
> silence from them is a lot more telling.  Both generic/013 and
> generic/475 are basic "try to create files and read and write data to
> them" exercisers; failing those is a red flag.
> 

Konstantin has wrote about these thing see below.

On Thu, 20 Aug 2020 10:20:26 +0000, Konstantin Komarov wrote: 
> xfstests are being one of our standard test suites among others.
> Currently we have the 'generic/339' and 'generic/013' test cases
> failing, working on it now. Other tests either pass or being skipped
> (due to missing features e.g. reflink). 
Source:
https://lore.kernel.org/linux-fsdevel/7538540ab82e4b398a0203564a1f1b23@paragon-software.com/

Also code tells that xfstests is being used in Paragon. In ntfs3/file.c:

/*
* Unwritten area
* NTFS is not able to store several unwritten areas
* Activate 'ntfs_sparse_cluster' to zero new allocated clusters
*
* Dangerous in case:
* 1G of sparsed clusters + 1 cluster of data =>
* valid_size == 1G + 1 cluster
* fallocate(1G) will zero 1G and this can be very long
* xfstest 016/086 will fail without 'ntfs_sparse_cluster'
*/
/*ntfs_sparse_cluster(inode, NULL, vcn,
 *	              min(vcn_v - vcn, clen));
 */

I'm just bringing this thing up because so many has asked and Konstantin
has not responded recently. Hopefully he will soon. Of course is it
little bit worrying that example generic/013 still fails after almoust
year has passed and Konstantin said he is working on it. And it seems that
more tests fails than beginning of review process.

> --D
> 
> > I can even give them patches and configsto make it trivially easy for
> > them to run fstests using KVM or GCE....
> > 
> > 				- Ted

Re: [GIT PULL] vboxsf fixes for 5.14-1

[Theodore Ts'o]

On Wed, Aug 04, 2021 at 09:38:10AM +0300, Kari Argillander wrote:
> Konstantin has wrote about these thing see below.
> 
> Source:
> https://lore.kernel.org/linux-fsdevel/7538540ab82e4b398a0203564a1f1b23@paragon-software.com/

Thanks for the link; that's really helpful.

> I'm just bringing this thing up because so many has asked and Konstantin
> has not responded recently. Hopefully he will soon. Of course is it
> little bit worrying that example generic/013 still fails after almoust
> year has passed and Konstantin said he is working on it. And it seems that
> more tests fails than beginning of review process.

Also interesting is that back in August 2020 Konstantin had promised
that they would be publishing their own fsck and mkfs tools.
Personally, I consider having a strong set of file system utilities to
be as important, if not more important, than the kernel code.  Perhaps
there are licensing issues which is why he hasn't been able to make
his code available?

One thing which I wonder about is whether there is anyone other than
Konstantin which is working on ntfs3?  I'm less concerned about
specific problems about the *code* --- I'll let folks like Christoph,
Dave, and Al weigh in on that front.

I'm more concerned about the long term sustainability and
maintainibility of the effort.  Programming is a team sport, and this
is especially true in the file system.  If you look at the successful
file systems, there are multiple developers involved, and ideally,
those developers work for a variety of different companies.  This way,
if a particular file system developer gets hit by a bus, laid low with
COVD-19, or gets laid off by their company due to changing business
strategies, or just decides to accept a higher paying job elsewhere,
the file system can continue to be adequately supported upstream.

If Konstantin really is the only developer working on ntfs3, that may
very well explain why generic/013 failures have been unaddressed in
over a year.  Which is why I tend to be much more concerned about
development community and development processes than just the quality
and maturity of the code.  If you have a good community and
development processes, the code qualtiy will follow.  If you don't,
that tends to be a recipe for eventual failure.

There are a large number of people on the cc line, include from folks
like Red Hat, SuSE, etc.  It would be *great* to hear that they are
also working on ntfs3, and it's not just a one engineer show.  (Also,
given the deadlock problems, lack of container compatibility, etc.,
are the Linux distros actually planning on shipping ntfs3 to their
customers?  Are they going to help make ntfs3 suitable for customers
with access to their help desks?)

> > > I can even give them patches and configs to make it trivially easy for
> > > them to run fstests using KVM or GCE....

I've since posted RFC patches to the fstests list to allow other
people to run xfstests on ntfs3.  I don't know why Konstantin hadn't
published his patches to fstests a year ago --- perhaps because of
licensing concerns with the mkfs and fsck userspace programs which
Paragon Software is using?

My fstests patches use the mkfs.ntfs and ntfsfix which ships with the
ntfs-3g package.  They are not ideal; for example ntfsfix will not
detect or fix all problems, and it is documented that for some issues,
you have to boot into Windows and run CHKDSK.  But it is the only
thing that is going to be available for any **users** of ntfs3 outside
of Paragon Software.

Some kind of update from Paragon Software about when their versions of
{mkfs,fsck}.ntfs might be made available for Linux distributions to
use would certainly be enlightening.

					- Ted

RE: [GIT PULL] vboxsf fixes for 5.14-1

[Konstantin Komarov]

> From: Darrick J. Wong <djwong@kernel.org>
> Sent: Wednesday, August 4, 2021 4:04 AM
> To: Theodore Ts'o <tytso@mit.edu>
> Cc: Linus Torvalds <torvalds@linux-foundation.org>; Matthew Wilcox <willy@infradead.org>; Leonidas P. Papadakos
> <papadakospan@gmail.com>; Konstantin Komarov <almaz.alexandrovich@paragon-software.com>; zajec5@gmail.com; Greg Kroah-
> Hartman <gregkh@linuxfoundation.org>; Hans de Goede <hdegoede@redhat.com>; linux-fsdevel <linux-fsdevel@vger.kernel.org>;
> Linux Kernel Mailing List <linux-kernel@vger.kernel.org>; Al Viro <viro@zeniv.linux.org.uk>
> Subject: Re: [GIT PULL] vboxsf fixes for 5.14-1
> 
> On Tue, Aug 03, 2021 at 08:49:28PM -0400, Theodore Ts'o wrote:
> > On Tue, Aug 03, 2021 at 05:10:22PM -0700, Linus Torvalds wrote:
> > > The user-space FUSE thing does indeed work reasonably well.
> > >
> > > It performs horribly badly if you care about things like that, though.
> > >
> > > In fact, your own numbers kind of show that:
> > >
> > >   ntfs/default: 670 tests, 55 failures, 211 skipped, 34783 seconds
> > >   ntfs3/default: 664 tests, 67 failures, 206 skipped, 8106 seconds
> > >
> > > and that's kind of the point of ntfs3.
> >
> > Sure, although if you run fstress in parallel ntfs3 will lock up, the
> > system hard, and it has at least one lockdep deadlock complaints.
> > It's not up to me, but personally, I'd feel better if *someone* at
> > Paragon Software responded to Darrrick and my queries about their
> > quality assurance, and/or made commitments that they would at least
> > *try* to fix the problems that about 5 minutes of testing using
> > fstests turned up trivially.
> 
> <cough> Yes, my aim was to gauge their interest in actively QAing the
> driver's current problems so that it doesn't become one of the shabby
> Linux filesystem drivers, like <cough>ntfs.
> 
> Note I didn't even ask for a particular percentage of passing tests,
> because I already know that non-Unix filesystems fail the tests that
> look for the more Unix-specific behaviors.
> 
> I really only wanted them to tell /us/ what the baseline is.  IMHO the
> silence from them is a lot more telling.  Both generic/013 and
> generic/475 are basic "try to create files and read and write data to
> them" exercisers; failing those is a red flag.
> 

Hi Darrick and Theodore! First of all, apologies for the silence on your questions.
Let me please clarify and summarize the QA topic for you.

The main thing to outline is that: we have the number of autotests executed
for ntfs3 code. More specifically, we are using TeamCity as our CI tool, which
is handling autotests. Those are being executed against each commit to the
ntfs3 codebase.

Autotests are divided into the "promotion" levels, which are quite standard:
L0, L1, L2. Those levels have the division from the shortest "smoke" (L0)
to the longest set (L2). This we need to cover the ntfs3 functionality with
tests under given amount of time (feedback loop for L0 is minutes, while for
L2 is up to 24hrs).

As for suites we are using - it is the mix of open/well known suites:
- xfstests, ltp, pjd suite, fsx, dirstress, fstorture - those are of known utilites/suites
And number of internal autotests which were developed for covering various parts of
fs specs, regression autotests which are introduced to the infrastructure after bugfixes
and autotests written to test the driver operation on various data sets.

This approach is settled in Paragon for years, and ntfs3, from the first line of code written,
is being developed this way. You may refer the artifacts linked below, where the progress/coverage
during the last year is spoken by autotest results:

the 27th patch-series code (July'2021): 
https://dl.paragon-software.com/ntfs3/p27_tests.tar
25th (March'2021):
https://dl.paragon-software.com/ntfs3/p25_tests.tar
2nd (August, 2020):
https://dl.paragon-software.com/ntfs3/p2_tests.tar

Those are results on ntfs3 ran within the 'linux-next' (the most recent one given the tests start date)
As may be observed, we never skipped the "tests day" :)

There is a note should be provided on xfstests specifically. We have been using this suite
as a part of our autotests for several years already. However the suite originate for Linux
native file systems and a lot of cases are not applicable to the NTFS. This is one of the reasons
why some of "red-flag" failures are there (e.g. generic/475) - they were excluded at some point of time
and we've missed to enable it back when it was the time :)

Thank you all for this effort to run and look closer on our code, on the next patchset, the
91, 317 and 475 should be resolved. And now we are looking up to other excluded tests to find out more of such.

Hope this will resolve some of your concerns.

> --D
> 
> > I can even give them patches and configsto make it trivially easy for
> > them to run fstests using KVM or GCE....
> >
> > 				- Ted

Re: [GIT PULL] vboxsf fixes for 5.14-1

[Darrick J. Wong]

On Thu, Aug 05, 2021 at 03:48:36PM +0000, Konstantin Komarov wrote:
> > From: Darrick J. Wong <djwong@kernel.org>
> > Sent: Wednesday, August 4, 2021 4:04 AM
> > To: Theodore Ts'o <tytso@mit.edu>
> > Cc: Linus Torvalds <torvalds@linux-foundation.org>; Matthew Wilcox <willy@infradead.org>; Leonidas P. Papadakos
> > <papadakospan@gmail.com>; Konstantin Komarov <almaz.alexandrovich@paragon-software.com>; zajec5@gmail.com; Greg Kroah-
> > Hartman <gregkh@linuxfoundation.org>; Hans de Goede <hdegoede@redhat.com>; linux-fsdevel <linux-fsdevel@vger.kernel.org>;
> > Linux Kernel Mailing List <linux-kernel@vger.kernel.org>; Al Viro <viro@zeniv.linux.org.uk>
> > Subject: Re: [GIT PULL] vboxsf fixes for 5.14-1
> > 
> > On Tue, Aug 03, 2021 at 08:49:28PM -0400, Theodore Ts'o wrote:
> > > On Tue, Aug 03, 2021 at 05:10:22PM -0700, Linus Torvalds wrote:
> > > > The user-space FUSE thing does indeed work reasonably well.
> > > >
> > > > It performs horribly badly if you care about things like that, though.
> > > >
> > > > In fact, your own numbers kind of show that:
> > > >
> > > >   ntfs/default: 670 tests, 55 failures, 211 skipped, 34783 seconds
> > > >   ntfs3/default: 664 tests, 67 failures, 206 skipped, 8106 seconds
> > > >
> > > > and that's kind of the point of ntfs3.
> > >
> > > Sure, although if you run fstress in parallel ntfs3 will lock up, the
> > > system hard, and it has at least one lockdep deadlock complaints.
> > > It's not up to me, but personally, I'd feel better if *someone* at
> > > Paragon Software responded to Darrrick and my queries about their
> > > quality assurance, and/or made commitments that they would at least
> > > *try* to fix the problems that about 5 minutes of testing using
> > > fstests turned up trivially.
> > 
> > <cough> Yes, my aim was to gauge their interest in actively QAing the
> > driver's current problems so that it doesn't become one of the shabby
> > Linux filesystem drivers, like <cough>ntfs.
> > 
> > Note I didn't even ask for a particular percentage of passing tests,
> > because I already know that non-Unix filesystems fail the tests that
> > look for the more Unix-specific behaviors.
> > 
> > I really only wanted them to tell /us/ what the baseline is.  IMHO the
> > silence from them is a lot more telling.  Both generic/013 and
> > generic/475 are basic "try to create files and read and write data to
> > them" exercisers; failing those is a red flag.
> > 
> 
> Hi Darrick and Theodore! First of all, apologies for the silence on your questions.
> Let me please clarify and summarize the QA topic for you.
> 
> The main thing to outline is that: we have the number of autotests executed
> for ntfs3 code. More specifically, we are using TeamCity as our CI tool, which
> is handling autotests. Those are being executed against each commit to the
> ntfs3 codebase.
> 
> Autotests are divided into the "promotion" levels, which are quite standard:
> L0, L1, L2. Those levels have the division from the shortest "smoke" (L0)
> to the longest set (L2). This we need to cover the ntfs3 functionality with
> tests under given amount of time (feedback loop for L0 is minutes, while for
> L2 is up to 24hrs).

Sounds comparable to my setup, which has these tiers:

fstests -g quick (~45 minutes) on fast ssds
fstests -g all (~3 hours) on fast ssds
fstests -g all (~12 hours) on slow(er) cheap(er) cloud storage
fstests -g long_soak (~7 days) on aging ssds

(There's also the fifth tier which is spawning dozens of VMs to fuzz
test, but I don't have enough spare time to run that and triage the
results on a regular basis.)

> As for suites we are using - it is the mix of open/well known suites:
> - xfstests, ltp, pjd suite, fsx, dirstress, fstorture - those are of known utilites/suites
> And number of internal autotests which were developed for covering various parts of
> fs specs, regression autotests which are introduced to the infrastructure after bugfixes
> and autotests written to test the driver operation on various data sets.
> 
> This approach is settled in Paragon for years, and ntfs3, from the first line of code written,
> is being developed this way. You may refer the artifacts linked below, where the progress/coverage
> during the last year is spoken by autotest results:
> 
> the 27th patch-series code (July'2021): 
> https://dl.paragon-software.com/ntfs3/p27_tests.tar
> 25th (March'2021):
> https://dl.paragon-software.com/ntfs3/p25_tests.tar
> 2nd (August, 2020):
> https://dl.paragon-software.com/ntfs3/p2_tests.tar
> 
> Those are results on ntfs3 ran within the 'linux-next' (the most recent one given the tests start date)
> As may be observed, we never skipped the "tests day" :)
> 
> There is a note should be provided on xfstests specifically. We have been using this suite
> as a part of our autotests for several years already. However the suite originate for Linux
> native file systems and a lot of cases are not applicable to the NTFS. This is one of the reasons
> why some of "red-flag" failures are there (e.g. generic/475) - they were excluded at some point of time
> and we've missed to enable it back when it was the time :)

generic/019, generic/317, generic/476 (and generic/521 and 522) are
supposed to be stress exercisers of standard functionality (read, write,
mkdir, creat, unlink) which means they ought to pass on any filesystem.

Hmm, we /dont/ have a tag for these generic exercisers.  Maybe we
should, I'll think about that.

(FWIW a minor correction: I meant to reference generic/476 and not 475,
because 475 is a looping test of crash recovery.  475 is notorious for
intermittent failures even on ext4/btrfs/xfs.)

> Thank you all for this effort to run and look closer on our code, on the next patchset, the
> 91, 317 and 475 should be resolved. And now we are looking up to other excluded tests to find out more of such.

Ok, thank you!

> Hope this will resolve some of your concerns.

It does. :)

--D

> > --D
> > 
> > > I can even give them patches and configsto make it trivially easy for
> > > them to run fstests using KVM or GCE....
> > >
> > > 				- Ted

Re: NTFS testing (was: [GIT PULL] vboxsf fixes for 5.14-1

[Szabolcs Szakacsits]

On Tue, 3 Aug 2021, Linus Torvalds wrote:
> On Tue, Aug 3, 2021 at 5:04 PM Theodore Ts'o <tytso@mit.edu> wrote:
> >
> > Whenver I've ever needed to access ntfs files, I've always used the
> > ntfs-3g FUSE package.
> 
> The user-space FUSE thing does indeed work reasonably well.
> 
> It performs horribly badly if you care about things like that, though.
> 
> In fact, your own numbers kind of show that:
> 
>   ntfs/default: 670 tests, 55 failures, 211 skipped, 34783 seconds
>   ntfs3/default: 664 tests, 67 failures, 206 skipped, 8106 seconds
> 
> and that's kind of the point of ntfs3.

In all fairness, the generic/405 test case completely distorted the overall 
timing in favour of ntfs3. Neither driver was involved in that test case.

Generic/405 test is mkfs against a 1 TB thin provision device which has 1 
MB backing size. mkfs should return an error after it hits EIO. The test 
case configuration was not correct for mkntfs on behalf of ntfs-3g because 
it missed the --fast format option, so mkntfs tried to fill the 1 TB device 
with zeros, apparently on Google Cloud Platform, for almost 8 hours. This 
had absolutely nothing to do with ntfs-3g performance, it was a pure mkfs 
test:

	https://github.com/kdave/xfstests/blob/master/tests/generic/405

Meanwhile the test case ran in 1 second on behalf of ntfs3 because its mkfs 
was not found, so the test could not be run. (And the test case got 
incorrectly categorized as success because it interpreted the "command not 
found" error as a success.)

If this mkntfs test case is ignored, as it should be, then ntfs-3g's 
runtime was (34783 - 28396) = 6387 versus ntfs3's (8106 - 1) = 8105 
seconds, i.e. the user space ntfs-3g was about 21% faster overall than the 
kernel space ntfs3.

Does this mean ntfs-3g is faster than ntfs3? Of course not. Fstests is not 
a benchmark. What we know for sure is, the unknowingly configured, untuned 
versions of the software gave different times for different workloads.

File system performance is a fairly complex topic. Ntfs-3g always aimed for 
stability, features, interoperability and portability, not for best 
possible performance. There seems to be some misconceptions, 
misinterpretations, inefficient configuration and mount options (e.g. 
missing big_writes, kernel_cache, etc). Unfortunately we did our part too 
to end up here. We will set better performance defaults in future releases.

User space drivers can have major disadvantages for certain workloads 
however how relevant are those for NTFS users? Most people use NTFS for 
file transfers in which case ntfs-3g read and write speed is about 15-20% 
less compared to ext4. For example in some quick tests ext4 read was 
3.4 GB/s versus ntfs-3g 2.8 GB/s, and write was 1.3 GB/s versus 1.1 GB/s.

Additionally there are still several technical solutions which could be 
implemented to improve all kinds of user space driver performance 
significantly.

But again, we always prefer data integrity over performance. And NTFS can 
be quite tricky with the ever changing on-disk corner cases. Does anybody 
still remember when Windows 2000 changed the NTFS on-disk format which 
massively started to trash users' data?

Please don't get me wrong, I'm not saying this is the way to go (who would 
be so crazy to write anything like that on the linux-kernel list?) I'm just 
saying this is the way we chose and support. We welcome the recent interest 
in NTFS after working on it for 20 years.

------

These are from Ted's logs which he shared earlier. It's much appreciated, 
it was highly useful. Personally I also thought the very poor ntfs-3g 
timing was due to bad configuration and/or mount options instead of an 
irrelevant test case. (Btw, the driver configuration and mount options were 
indeed not right, e.g. ACL, permission, etc related cases failed which 
could have pass.)

$ egrep ^generic/405 results-ntfs*/runtests.log
results-ntfs/runtests.log:generic/405           [21:47:08] [05:40:24] 28396s
results-ntfs3/runtests.log:generic/405          [12:12:09] [12:12:10] 1s

$ cat results-ntfs/ntfs/results-default/generic/405.full
[...]
Cluster size has been automatically set to 4096 bytes.
Initializing device with zeroes: 100% - Done.
Creating NTFS volume structures.
Failed to sync device /dev/mapper/thin-vol: Input/output error
Syncing device. FAILED

$ cat results-ntfs3/ntfs3/results-default/generic/405.full
[...]
mkfs: failed to execute mkfs.ntfs3: No such file or directory

Best regards,

	Szaka

Re: NTFS testing (was: [GIT PULL] vboxsf fixes for 5.14-1

[Eric Biggers]

On Fri, Sep 03, 2021 at 01:09:40AM +0300, Szabolcs Szakacsits wrote:
> User space drivers can have major disadvantages for certain workloads 
> however how relevant are those for NTFS users? Most people use NTFS for 
> file transfers in which case ntfs-3g read and write speed is about 15-20% 
> less compared to ext4. For example in some quick tests ext4 read was 
> 3.4 GB/s versus ntfs-3g 2.8 GB/s, and write was 1.3 GB/s versus 1.1 GB/s.

Your company's own advertising materials promoting your proprietary NTFS driver
(https://www.tuxera.com/products/tuxera-ntfs-embedded) claim that NTFS-3G is
much slower than ext4:

	Read:
		NTFS-3G: 63.4 MB/s
		ext4: 113.8 MB/s
		"Microsoft NTFS by Tuxera": 116 MB/s

	Write:
		NTFS-3G: 16.3 MB/s
		ext4: 92.4 MB/s
		"Microsoft NTFS by Tuxera": 113.3 MB/s

I'm not sure why anything you say should have any credibility when it
contradicts what your company says elsewhere, and your company has a vested
interest in not having proper NTFS support upstreamed to compete with their
proprietary driver.  (Note that Tuxera doesn't provide much support for NTFS-3G;
most of their efforts are focused on their proprietary driver.)

- Eric

Re: NTFS testing (was: [GIT PULL] vboxsf fixes for 5.14-1

[Szabolcs Szakacsits]

On Fri, 3 Sep 2021, Eric Biggers wrote:
> On Fri, Sep 03, 2021 at 01:09:40AM +0300, Szabolcs Szakacsits wrote:
> > User space drivers can have major disadvantages for certain workloads 
> > however how relevant are those for NTFS users? Most people use NTFS for 
> > file transfers in which case ntfs-3g read and write speed is about 15-20% 
> > less compared to ext4. For example in some quick tests ext4 read was 
> > 3.4 GB/s versus ntfs-3g 2.8 GB/s, and write was 1.3 GB/s versus 1.1 GB/s.
> 
> Your company's own advertising materials promoting your proprietary NTFS driver
> (https://www.tuxera.com/products/tuxera-ntfs-embedded) claim that NTFS-3G is
> much slower than ext4:

Thank you for pointing this out. And please do so whatever else you think 
is not right.

Let's see in detail.

> 	Read:
> 		NTFS-3G: 63.4 MB/s
> 		ext4: 113.8 MB/s
> 		"Microsoft NTFS by Tuxera": 116 MB/s
> 
> 	Write:
> 		NTFS-3G: 16.3 MB/s
> 		ext4: 92.4 MB/s
> 		"Microsoft NTFS by Tuxera": 113.3 MB/s

The page says under the benchmark:

 "Tested on ARM Cortex-A15 Processor / 512 MB RAM / Samsung SSD 840 PRO 256 GB, 
  USB 3.0 / Windows client and Samba over 1 GbE. Actual performance may 
  vary based on software and hardware used."

My quoted benchmark was done on 

 System on Chip: 11th Gen Intel(R) Core(TM) i5-11400 @2.60GHz (12 cores) 
	in ASUSTeK COMPUTER INC. PRIME B560-PLUS motherboard
 OS: Linux 5.10.0-8-amd64 x86_64
 Storage: Samsung SSD 970 PRO 512GB 512GB NVMe
 NTFS-3G 2017.3.23AR.6 (February 1, 2021) integrated FUSE 28 
 ext4 Intree (Linux 5.10.0-8-amd64)
 
> I'm not sure why anything you say should have any credibility 

Please don't believe me and do your own check. Both Ted's logs and the 
performance results which I have shared.

> when it contradicts what your company says elsewhere, 

The text says "Actual performance may vary based on software and hardware 
used." I'm afraid my results don't contradict. Hardware is vastly 
different, software is vastly different:

- The PC is much more powerful. Much faster multi-core CPU, RAM, 
interconnect, and storage compared to an apparently single core Cortex-A15.

- The embedded test used user space Samba, the other one didn't. Samba and 
ntfs-3g competed for one core which made the speed lower than it could have 
been. ksmbd will help a lot on this, just like ntfs3 for samba. And ntfs-3g 
could be also improved a lot, as I mentioned earlier. Isn't it great there 
are so many options?

- Today embedded often has multi-core, so the speed difference is (much) less.

- Tested embedded ntfs-3g version is unknown but it seems to be a (quite) 
old one. My test used one of the latest ones. NTFS-3G performance has 
been improving in time.

- I'm sure the embedded test didn't use the big_writes mount option. 
Otherwise I think the speed could have been around 50 MB/s. Which is still 
not great but at least 3 times faster. We explained and addressed this in 
the latest release note:

https://lore.kernel.org/linux-fsdevel/d343b1d7-6587-06a5-4b60-e4c59a585498@wanadoo.fr/

Overall, you had a good point. That comparison is not the most up-to-date 
one. We will work on it or just remove it.

> and your company has a vested interest in not having proper NTFS support 
> upstreamed

Please explain what you mean exactly by "proper"? 

Linus wrote "does indeed work reasonably well" except the horrible 
performance which was based on misinterpreting test results ntfs-3g being 
4 times slower when in fact it was 21% faster.

If "proper" means being in the kernel then I explained in my previous email 
why we chose FUSE.

> to compete with their proprietary driver.

The proprietary version enables us to pay people working on the open source 
version. The source code is available, anybody could do it. 

> (Note that Tuxera doesn't provide much support for NTFS-3G; most of their 
> efforts are focused on their proprietary driver.)

We provide both commercial and free support for NTFS-3G. We had annually at 
least one stable open source release since 2006, full changelog:

	https://github.com/tuxera/ntfs-3g/releases

And all questions and issues are answered, resolved:

	https://sourceforge.net/p/ntfs-3g/mailman/ntfs-3g-devel/

Thank you Eric again for the very honest feedback.

Best regards,

	Szaka

Re: [GIT PULL] vboxsf fixes for 5.14-1

[Pali Rohár]

On Friday 16 July 2021 14:46:35 Leonidas P. Papadakos wrote:
> It would mean having good support for a cross-platform filesystem suitable for hard drives. exFAT is welcome, but it's a simple filesystem for flash storage.

FYI: There is also another cross-platform filesystem (Linux kernel,
Windows NT kernel, Mac OS X kernel) suitable for hard disks too with
POSIX permissions about which people do not know too much. It is UDF.

Re: [GIT PULL] vboxsf fixes for 5.14-1

[Darrick J. Wong]

On Wed, Jul 14, 2021 at 05:59:19PM +0200, Rafał Miłecki wrote:
> On 14.07.2021 16:51, Greg KH wrote:
> > On Wed, Jul 14, 2021 at 12:50:08PM +0200, Rafał Miłecki wrote:
> > > Hi Alexander,
> > > 
> > > On 13.07.2021 22:14, Al Viro wrote:
> > > > To elaborate a bit - there's one case when I want it to go through
> > > > vfs.git, and that's when there's an interference between something
> > > > going on in vfs.git and the work done in filesystem.  Other than
> > > > that, I'm perfectly fine with maintainer sending pull request directly
> > > > to Linus (provided that I hadn't spotted something obviously wrong
> > > > in the series, of course, but that's not "I want it to go through
> > > > vfs.git" - that's "I don't want it in mainline until such and such
> > > > bug is resolved").
> > > 
> > > let me take this opportunity to ask about another filesystem.
> > > 
> > > Would you advise to send pull req for the fs/ntfs3 directly to Linus?
> > > 
> > > That is a pending filesystem that happens to be highly expected by many
> > > Linux focused communities.
> > > 
> > > 
> > > Paragon Software GmbH proved it's commitment by sending as many as 26
> > > versions on it's patchset. The last set was send early April:
> > > 
> > > [PATCH v26 00/10] NTFS read-write driver GPL implementation by Paragon Software
> > > https://marc.info/?l=linux-fsdevel&m=161738417018673&q=mbox
> > > https://patchwork.kernel.org/project/linux-fsdevel/list/?series=460291
> > > 
> > > 
> > > I'd say there weren't any serious issues raised since then.
> > > 
> > > One Tested-by, one maintenance question, one remainder, one clang-12
> > > issue [0] [1].
> > > 
> > > It seems this filesystem only needs:
> > > 1. [Requirement] Adjusting to the meanwhile changed iov API [2]
> > > 2. [Clean up] Using fs/iomap/ helpers [3]
> > 
> > Why haven't those things been done and the patches resubmitted for
> > review?  Nothing we can do from our side when the developers don't want
> > to submit a new series, right?
> 
> The real issue (broken compilation) has been pointed out 2 days ago and
> is a result of a more recent commit. For months filesystem could be
> pushed but it wasn't for unknown reason.
> 
> As for fs/iomap/ helpers it's unclear to me if that is really required
> or could be worked on later as a clean up. Darrick joked his opinion on
> using those helper is biased.

Porting to fs/iomap can be done after merge, so long as the ntfs3
driver doesn't depend on crazy reworking of buffer heads or whatever.
AFAICT it didn't, so ... yes, my earlier statements still apply: "later
as a clean up".

That said, I /really would/ like answers to the questions I posted here
about how well their fs driver fares while running fstests, since that's
probably the only way for community developers to ensure they don't
accidentally bitrot the driver into disk corruption land.

--D

[1] https://lore.kernel.org/linux-fsdevel/20210520161325.GA9625@magnolia/

> In short I'd say: missing feedback.

Re: [GIT PULL] vboxsf fixes for 5.14-1

[Christoph Hellwig]

On Wed, Jul 14, 2021 at 09:13:52AM -0700, Darrick J. Wong wrote:
> Porting to fs/iomap can be done after merge, so long as the ntfs3
> driver doesn't depend on crazy reworking of buffer heads or whatever.
> AFAICT it didn't, so ... yes, my earlier statements still apply: "later
> as a clean up".

I on the other hand hate piling up mor of this legacy stuff, as it tends
to not be cleaned up by the submitted.  Example: erofs still hasn't
switched to iomap despite broad claims, also we still have a huge
backlog in the switch to the new mount API.

Re: [GIT PULL] vboxsf fixes for 5.14-1

[Gao Xiang]

Hi Christoph,

On Wed, Jul 14, 2021 at 05:18:07PM +0100, Christoph Hellwig wrote:
> On Wed, Jul 14, 2021 at 09:13:52AM -0700, Darrick J. Wong wrote:
> > Porting to fs/iomap can be done after merge, so long as the ntfs3
> > driver doesn't depend on crazy reworking of buffer heads or whatever.
> > AFAICT it didn't, so ... yes, my earlier statements still apply: "later
> > as a clean up".
> 
> I on the other hand hate piling up mor of this legacy stuff, as it tends
> to not be cleaned up by the submitted.  Example: erofs still hasn't
> switched to iomap despite broad claims, also we still have a huge

Just some word of this, I've been always actively developing and
converting legacy stuffs to new APIs these years, such as new mount
APIs, xarray, which can be seen by changelog compared with other
filesystems. The iomap buffered I/O stuff is mainly because it
doesn't support tailing-packing inline, although which has been
requested for people who cares more about storage itself, like:
https://lore.kernel.org/lkml/3dbeff43-0905-3421-4652-41b7a935f3c1@gmail.com/
And I can also show the benefits by using tailing-packing inline
by taking Linux source code tree (many random tail blocks) as an
example...

I'm now working on this tailing-packing inline iomap support as
well. But Anyway, erofs didn't use buffer head in the beginning
since I can see some flew of buffer head approach itself, it just
works on raw bio and page cache interfaces for now.

Thanks,
Gao Xiang

> backlog in the switch to the new mount API.

Re: [GIT PULL] vboxsf fixes for 5.14-1

[Eric W. Biederman]

Christoph Hellwig <hch@infradead.org> writes:

> also we still have a huge backlog in the switch to the new mount API.
                                                         ^^^^^^^^^^^^^^
Speaking of code that ignored reviewer feedback.

Part of my feedback was that the new mount API has problems that make
it difficult to switch to.

Or is it your point that we let the new mount API be merged without a
conversion for all filesystems and a promise that the conversion would
get done after it was merged?

Eric

Re: [GIT PULL] vboxsf fixes for 5.14-1

[Darrick J. Wong]

On Wed, Jul 14, 2021 at 05:18:07PM +0100, Christoph Hellwig wrote:
> On Wed, Jul 14, 2021 at 09:13:52AM -0700, Darrick J. Wong wrote:
> > Porting to fs/iomap can be done after merge, so long as the ntfs3
> > driver doesn't depend on crazy reworking of buffer heads or whatever.
> > AFAICT it didn't, so ... yes, my earlier statements still apply: "later
> > as a clean up".
> 
> I on the other hand hate piling up mor of this legacy stuff, as it tends
> to not be cleaned up by the submitted.  Example: erofs still hasn't
> switched to iomap despite broad claims,

<shrug> I was letting that one go while willy tries to land all the
folio surgery on the iomap code.

> also we still have a huge backlog in the switch to the new mount API.

That's true, though having /read/ the xfs conversion series, I'm not
surprised that most maintainers don't want to do the heavy lift
themselves.

--D

Re: [GIT PULL] vboxsf fixes for 5.14-1

[pr-tracker-bot]

The pull request you sent on Tue, 13 Jul 2021 12:45:19 +0200:

> git://git.kernel.org/pub/scm/linux/kernel/git/hansg/linux.git tags/vboxsf-v5.14-1

has been merged into torvalds/linux.git:
https://git.kernel.org/torvalds/c/40226a3d96ef8ab8980f032681c8bfd46d63874e

Thank you!

-- 
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/prtracker.html