Re: parallel file create rates (+high latency)

[Daire Byrne <daire@dneg.com>]

On Mon, 24 Jan 2022 at 19:38, J. Bruce Fields <bfields@fieldses.org> wrote:
>
> On Sun, Jan 23, 2022 at 11:53:08PM +0000, Daire Byrne wrote:
> > I've been experimenting a bit more with high latency NFSv4.2 (200ms).
> > I've noticed a difference between the file creation rates when you
> > have parallel processes running against a single client mount creating
> > files in multiple directories compared to in one shared directory.
>
> The Linux VFS requires an exclusive lock on the directory while you're
> creating a file.

Right. So when I mounted the same server/dir multiple times using
namespaces, all I was really doing was making the VFS *think* I wanted
locks on different directories even though the remote server directory
was actually the same?

> So, if L is the time in seconds required to create a single file, you're
> never going to be able to create more than 1/L files per second, because
> there's no parallelism.

And things like directory delegations can't help with this kind of
workload? You can't batch directories locks or file creates I guess.

> So, it's not surprising you'd get a higher rate when creating in
> multiple directories.
>
> Also, that lock's taken on both client and server.  So it makes sense
> that you might get a little more parallelism from multiple clients.
>
> So the usual advice is just to try to get that latency number as low as
> possible, by using a low-latency network and storage that can commit
> very quickly.  (An NFS server isn't permitted to reply to the RPC
> creating the new file until the new file actually hits stable storage.)
>
> Are you really seeing 200ms in production?

Yea, it's just a (crazy) test for now. This is the latency between two
of our offices. Running batch jobs over this kind of latency with a
NFS re-export server doing all the caching works surprisingly well.

It's just these file creations that's the deal breaker. A batch job
might create 100,000+ files in a single directory across many clients.

Maybe many containerised re-export servers in round-robin with a
common cache is the only way to get more directory locks and file
creates in flight at the same time.

Cheers,

Daire