Stable NTFS-3G + NTFSPROGS 2021.8.22 Released

Stable NTFS-3G + NTFSPROGS 2021.8.22 Released

[Jean-Pierre André]

Greetings,

Main topics:

   - New stable release
   - Security advisory
   - Project moved to GitHub
   - Performance notes

The new stable release of NTFS-3G and ntfsprogs is available which 
includes important security fixes. The security advisory is available at

https://github.com/tuxera/ntfs-3g/security/advisories/GHSA-q759-8j5v-q5jp

The NTFS-3G project globally aims at providing a stable NTFS driver. The 
project's advanced branch has specifically aimed at developing, 
maturing, and releasing features for user feedback prior to feature 
integration into the project's main branch.

The parallel existence of both a stable and advanced variant maintained 
across several locations has caused some confusion. In particular, the 
Linux distributions observed different policies in selecting which 
version they use for their packaging. That led to users questioning the 
differences between features, and to additional challenges in providing 
support.

We've decided to merge the two projects and maintain a single repository 
for source code and documentation on GitHub. As the projects have always 
remained in close contact, this will cause no discontinuity in the 
released features, while enabling smoother support. The former 
repository on Sourceforge will be discontinued after a grace period, to 
allow users time to adapt to the project's new state. Please use 
GitHub's infrastructure for issue submission and release notification.

There have been some reports about very slow performance. Performance is 
a complex topic and NTFS-3G always aimed for stability, interoperability 
and portability over performance. Having said that, we did some 
investigation and benchmarking. What we have found are

1. Some distributions use an older and slower version of NTFS-3G.

2. The "big_writes" mount option is not used. This option can increase 
 >4kB IO block size write speed by 2-8 times. File transfers typically 
use 128kB which usually give a 3-4 times speed improvement. The option 
is safe to use and we plan to enable it by default in the next stable 
release.

3. Incorrect interpretation of benchmark results. For example in a 
recent public case the total runtime was completely distorted by an 
irrelevant test case hereby a wrong conclusion was made, namely NTFS-3G 
was thought to be over 4 times slower instead of 21% faster. More about 
this soon on linux-fsdevel.

In our file transfer benchmarks we have found NTFS-3G read and write 
speed was 15-20% less compared to ext4. Read was 3.4 GB/s versus 2.8 
GB/s, and write was 1.3 GB/s vs 1.1 GB/s. Nevertheless, different 
benchmarks can give different results.

The new release can be downloaded from

     https://github.com/tuxera/ntfs-3g/releases/tag/2021.8.22

Changelog is available at

     https://github.com/tuxera/ntfs-3g/wiki/NTFS-3G-Release-History

Many thanks to Rakesh Pandit, Jussi Hietanen, Erik Larsson, Szabolcs 
Szakacsits and many Tuxerians for their contributions to this release 
and to the migration to GitHub.

We also want to add special thanks to Jeremy Galindo, Akshay Ajayan, 
Kyle Zeng and Fish Wang, whose analyses were of great help in improving 
the security of the code.

With best regards,

Jean-Pierre & Tuxera Open Source Team

Re: Stable NTFS-3G + NTFSPROGS 2021.8.22 Released

[Matthew Wilcox]

On Mon, Aug 30, 2021 at 07:59:17PM +0200, Jean-Pierre André wrote:
> The NTFS-3G project globally aims at providing a stable NTFS driver. The
> project's advanced branch has specifically aimed at developing, maturing,
> and releasing features for user feedback prior to feature integration into
> the project's main branch.

So do I understand correctly ...

 - We have an NTFS filesystem from Anton Altaparmakov in fs/ntfs which was
   merged in 1997 and is read only.
 - We have Paragon's NTFS3 in the process of being merged
 - We have Tuxera's NTFS-3G hosted externally on Github that uses FUSE

Any other implementations of NTFS for Linux that we should know about?
Is there any chance that the various developers involved can agree to
cooperate on a single implementation?

Re: Stable NTFS-3G + NTFSPROGS 2021.8.22 Released

[Kari Argillander]

On Mon, Aug 30, 2021 at 08:09:38PM +0100, Matthew Wilcox wrote:
> On Mon, Aug 30, 2021 at 07:59:17PM +0200, Jean-Pierre André wrote:
> > The NTFS-3G project globally aims at providing a stable NTFS driver. The
> > project's advanced branch has specifically aimed at developing, maturing,
> > and releasing features for user feedback prior to feature integration into
> > the project's main branch.
> 
> So do I understand correctly ...
> 
>  - We have an NTFS filesystem from Anton Altaparmakov in fs/ntfs which was
>    merged in 1997 and is read only.
>  - We have Paragon's NTFS3 in the process of being merged
>  - We have Tuxera's NTFS-3G hosted externally on Github that uses FUSE
> 
> Any other implementations of NTFS for Linux that we should know about?
> Is there any chance that the various developers involved can agree to
> cooperate on a single implementation?

I would also like to here about this from ntfs-3g guys. What do you even
think about this whole ntfs3 kernel driver? My own opionion is that
kernel really needs ntfs driver at it's own because it will greatly
benefit growing iot device market and also maybe it will be in Android
in same day. Also great performance boost with ntfs3 compared to
ntfs-3g.

It will be strange situation that we continue development from both
sides.  ntfs-3g guys have only open source NTFSPROGS tools what is
available. I like to note, that Paragon has sayd that they will open
source their mkfs and fschk, but I will not count on that. That means
that then there will be two tool set also. This is not good for users.

After ntfs3 is merged to mainline we are three driver situation.
Hopefully kernel ntfs driver can be dropper in year or two.

One thing what we should at least share is testing. There is work that
has been done for xfstests that ntfs3 and ntfs-3g will be supported:
lore.kernel.org/fstests/YQoVXWRFGeY19onQ@mit.edu/

Xfstests will support also other fuse base drivers.
lore.kernel.org/fstests/20210812045950.3190-1-bhumit.attarde01@gmail.com/

There is already support for ntfs3 and ntfs-3g for kvm-xfstests. This
was done so that we know how these compare to each other. And can help
decissions that is ntfs3 ready for mainline.
github.com/tytso/xfstests-bld/commit/fa6410d922d38735a5f69345221f8eacb3ae1af

I do think that if ntfs3 will be solid and some remaining bugs will be
solved it would make no sense to use ntfs-3g. So what do you think about
all of this?

Best regards
Kari Argillander