Re: parallel file create rates (+high latency)

[Daire Byrne <daire@dneg.com>]

On Tue, 25 Jan 2022 at 22:11, Patrick Goetz <pgoetz@math.utexas.edu> wrote:
>
> IDK, 4000 images per collection, with hundreds of collections on disk?
> Say at least 500,000 files?  Maybe a million? With most files about 1GB
> in size.  I was trying to just rsync it all from the data server to a
> ZFS-based backup server in our data center, but the backup started
> failing constantly because the filesystem would change after rsync had
> already constructed an index. Even after an initial copy, a backup like
> that runs for over a week.  The strategy I'm about to try and implement
> is to NFS mount the data server's data partition to the backup server
> and then have a script walk through the directory hierarchy, rsyncing
> collections one at a time.  ZFS send/receive would probably be better,
> but the data server isn't configured with ZFS.

We've strayed slightly off topic (even if we are talking about file
creates over NFS) because you can get good parallel performance
(creates, read, writes etc) over NFS with simultaneous copies using
lots of processes if distributed across lots of directories.

Well "good" being subjective. I get 1,500 creates/s in a single
directory on a LAN NFS server from a single client and 160 creates/s
aggregate over my extreme 200ms using 10 clients & 10 different
directories. It seems fair all things considered.

But seeing as I do a lot of these kinds of big data moves (TBs) across
both the LAN and WAN, I can perhaps offer some advice from experience
that might be useful:

* walk the filesystem (locally) first to build a file list, split it
and then use rsync --files-from (e.g. https://github.com/jbd/msrsync)
to feed multiple simultaneous rsyncs.
* avoid NFS and use rsyncd directly between the servers (no ssh) so
filesystem walks are "local".

The advantage of rsync is that it will do the filesystem walks at both
ends locally and compare the directory trees as it goes along. The
other nice thing it does is open a connection between sender and
receiver and stream all the file data down it so it works really well
even for lists of small files. The TCP connection and window scaling
can sit at it's maximum without any slow remote file metadata latency
disrupting it. Avoid the encapsulation of  sshand use rsyncd instead
as it just speeds everything up.

And as always with any WAN connection, large buffers, window scaling,
no firewall DPI and maybe some fancy congestion control like BBR/2
helps.

Daire