Re: [PATCH 0/9] Multiple network connections for a single NFS mount.

[Chuck Lever <chuck.lever@oracle.com>]

> On May 30, 2019, at 6:56 PM, NeilBrown <neilb@suse.com> wrote:
> 
> On Thu, May 30 2019, Chuck Lever wrote:
> 
>> Hi Neil-
>> 
>> Thanks for chasing this a little further.
>> 
>> 
>>> On May 29, 2019, at 8:41 PM, NeilBrown <neilb@suse.com> wrote:
>>> 
>>> This patch set is based on the patches in the multipath_tcp branch of
>>> git://git.linux-nfs.org/projects/trondmy/nfs-2.6.git
>>> 
>>> I'd like to add my voice to those supporting this work and wanting to
>>> see it land.
>>> We have had customers/partners wanting this sort of functionality for
>>> years.  In SLES releases prior to SLE15, we've provide a
>>> "nosharetransport" mount option, so that several filesystem could be
>>> mounted from the same server and each would get its own TCP
>>> connection.
>> 
>> Is it well understood why splitting up the TCP connections result
>> in better performance?
>> 
>> 
>>> In SLE15 we are using this 'nconnect' feature, which is much nicer.
>>> 
>>> Partners have assured us that it improves total throughput,
>>> particularly with bonded networks, but we haven't had any concrete
>>> data until Olga Kornievskaia provided some concrete test data - thanks
>>> Olga!
>>> 
>>> My understanding, as I explain in one of the patches, is that parallel
>>> hardware is normally utilized by distributing flows, rather than
>>> packets.  This avoid out-of-order deliver of packets in a flow.
>>> So multiple flows are needed to utilizes parallel hardware.
>> 
>> Indeed.
>> 
>> However I think one of the problems is what happens in simpler scenarios.
>> We had reports that using nconnect > 1 on virtual clients made things
>> go slower. It's not always wise to establish multiple connections
>> between the same two IP addresses. It depends on the hardware on each
>> end, and the network conditions.
> 
> This is a good argument for leaving the default at '1'.  When
> documentation is added to nfs(5), we can make it clear that the optimal
> number is dependant on hardware.

Is there any visibility into the NIC hardware that can guide this setting?


>> What about situations where the network capabilities between server and
>> client change? Problem is that neither endpoint can detect that; TCP
>> usually just deals with it.
> 
> Being able to manually change (-o remount) the number of connections
> might be useful...

Ugh. I have problems with the administrative interface for this feature,
and this is one of them.

Another is what prevents your client from using a different nconnect=
setting on concurrent mounts of the same server? It's another case of a
per-mount setting being used to control a resource that is shared across
mounts.

Adding user tunables has never been known to increase the aggregate
amount of happiness in the universe. I really hope we can come up with
a better administrative interface... ideally, none would be best.


>> Related Work:
>> 
>> We now have protocol (more like conventions) for clients to discover
>> when a server has additional endpoints so that it can establish
>> connections to each of them.
>> 
>> https://datatracker.ietf.org/doc/rfc8587/
>> 
>> and
>> 
>> https://datatracker.ietf.org/doc/draft-ietf-nfsv4-rfc5661-msns-update/
>> 
>> Boiled down, the client uses fs_locations and trunking detection to
>> figure out when two IP addresses are the same server instance.
>> 
>> This facility can also be used to establish a connection over a
>> different path if network connectivity is lost.
>> 
>> There has also been some exploration of MP-TCP. The magic happens
>> under the transport socket in the network layer, and the RPC client
>> is not involved.
> 
> I would think that SCTP would be the best protocol for NFS to use as it
> supports multi-streaming - several independent streams.  That would
> require that hardware understands it of course.
> 
> Though I have examined MP-TCP closely, it looks like it is still fully
> sequenced, so it would be tricky for two RPC messages to be assembled
> into TCP frames completely independently - at least you would need
> synchronization on the sequence number.
> 
> Thanks for your thoughts,
> NeilBrown
> 
> 
>> 
>> 
>>> Comments most welcome.  I'd love to see this, or something similar,
>>> merged.
>>> 
>>> Thanks,
>>> NeilBrown
>>> 
>>> ---
>>> 
>>> NeilBrown (4):
>>>     NFS: send state management on a single connection.
>>>     SUNRPC: enhance rpc_clnt_show_stats() to report on all xprts.
>>>     SUNRPC: add links for all client xprts to debugfs
>>> 
>>> Trond Myklebust (5):
>>>     SUNRPC: Add basic load balancing to the transport switch
>>>     SUNRPC: Allow creation of RPC clients with multiple connections
>>>     NFS: Add a mount option to specify number of TCP connections to use
>>>     NFSv4: Allow multiple connections to NFSv4.x servers
>>>     pNFS: Allow multiple connections to the DS
>>>     NFS: Allow multiple connections to a NFSv2 or NFSv3 server
>>> 
>>> 
>>> fs/nfs/client.c                      |    3 +
>>> fs/nfs/internal.h                    |    2 +
>>> fs/nfs/nfs3client.c                  |    1 
>>> fs/nfs/nfs4client.c                  |   13 ++++-
>>> fs/nfs/nfs4proc.c                    |   22 +++++---
>>> fs/nfs/super.c                       |   12 ++++
>>> include/linux/nfs_fs_sb.h            |    1 
>>> include/linux/sunrpc/clnt.h          |    1 
>>> include/linux/sunrpc/sched.h         |    1 
>>> include/linux/sunrpc/xprt.h          |    1 
>>> include/linux/sunrpc/xprtmultipath.h |    2 +
>>> net/sunrpc/clnt.c                    |   98 ++++++++++++++++++++++++++++++++--
>>> net/sunrpc/debugfs.c                 |   46 ++++++++++------
>>> net/sunrpc/sched.c                   |    3 +
>>> net/sunrpc/stats.c                   |   15 +++--
>>> net/sunrpc/sunrpc.h                  |    3 +
>>> net/sunrpc/xprtmultipath.c           |   23 +++++++-
>>> 17 files changed, 204 insertions(+), 43 deletions(-)
>>> 
>>> --
>>> Signature
>>> 
>> 
>> --
>> Chuck Lever

--
Chuck Lever