[PATCH - nfs-utils] Fix fallback from tcp to udp

[PATCH - nfs-utils] Fix fallback from tcp to udp

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 1186 bytes --]


Protocol negotiation in mount.nfs does not correctly negotiate with a
server which only support NFSv3 and UDP.

When mount.nfs attempts an NFSv4 mount and fails with ECONNREFUSED
it does not fall back to NFSv3, as this is not recognised as a
"does not support NFSv4" error.
However ECONNREFUSED is a clear indication that the server doesn't
support TCP, and ipso facto does not support NFSv4.
So ECONNREFUSED should trigger a fallback from v4 to v2/3.

Once we allow that error, NFSv3 is attempted and mount.nfs talks to
rpcbind and discovers that UDP should be used for v3 and the mount
succeeds.

Signed-off-by: NeilBrown <neilb@suse.de>
Reported-by: Carsten Ziepke <kieltux@gmail.com>

diff --git a/utils/mount/stropts.c b/utils/mount/stropts.c
index a642394d2f5a..6d4fd70b7b9e 100644
--- a/utils/mount/stropts.c
+++ b/utils/mount/stropts.c
@@ -807,6 +807,9 @@ static int nfs_autonegotiate(struct nfsmount_info *mi)
 		/* Linux servers prior to 2.6.25 may return
 		 * EPERM when NFS version 4 is not supported. */
 		goto fall_back;
+	case ECONNREFUSED:
+		/* UDP-Only servers won't support v4 */
+		goto fall_back;
 	default:
 		return result;
 	}

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: [PATCH - nfs-utils] Fix fallback from tcp to udp

[Steve Dickson]

On 02/17/2014 06:43 PM, NeilBrown wrote:
> 
> Protocol negotiation in mount.nfs does not correctly negotiate with a
> server which only support NFSv3 and UDP.
> 
> When mount.nfs attempts an NFSv4 mount and fails with ECONNREFUSED
> it does not fall back to NFSv3, as this is not recognised as a
> "does not support NFSv4" error.
> However ECONNREFUSED is a clear indication that the server doesn't
> support TCP, and ipso facto does not support NFSv4.
> So ECONNREFUSED should trigger a fallback from v4 to v2/3.
I'm also pretty this is the error returned when the server is 
down or more pointy when server is rebooting... Do we really
want to fallback at this point?

Secondly, its worrisome to me that we keep making this fallback
list longer and longer... we really don't want to fall back
to v3 but I do understand we want to be compatible with 
older servers... 

steved. 
> 
> Once we allow that error, NFSv3 is attempted and mount.nfs talks to
> rpcbind and discovers that UDP should be used for v3 and the mount
> succeeds.
> 
> Signed-off-by: NeilBrown <neilb@suse.de>
> Reported-by: Carsten Ziepke <kieltux@gmail.com>
> 
> diff --git a/utils/mount/stropts.c b/utils/mount/stropts.c
> index a642394d2f5a..6d4fd70b7b9e 100644
> --- a/utils/mount/stropts.c
> +++ b/utils/mount/stropts.c
> @@ -807,6 +807,9 @@ static int nfs_autonegotiate(struct nfsmount_info *mi)
>  		/* Linux servers prior to 2.6.25 may return
>  		 * EPERM when NFS version 4 is not supported. */
>  		goto fall_back;
> +	case ECONNREFUSED:
> +		/* UDP-Only servers won't support v4 */
> +		goto fall_back;
>  	default:
>  		return result;
>  	}
> 

Re: [PATCH - nfs-utils] Fix fallback from tcp to udp

[J. Bruce Fields]

On Thu, Feb 20, 2014 at 12:50:15PM -0500, Steve Dickson wrote:
> 
> 
> On 02/17/2014 06:43 PM, NeilBrown wrote:
> > 
> > Protocol negotiation in mount.nfs does not correctly negotiate with a
> > server which only support NFSv3 and UDP.
> > 
> > When mount.nfs attempts an NFSv4 mount and fails with ECONNREFUSED
> > it does not fall back to NFSv3, as this is not recognised as a
> > "does not support NFSv4" error.
> > However ECONNREFUSED is a clear indication that the server doesn't
> > support TCP, and ipso facto does not support NFSv4.
> > So ECONNREFUSED should trigger a fallback from v4 to v2/3.
> I'm also pretty this is the error returned when the server is 
> down or more pointy when server is rebooting...

Probably worth checking that.

> Do we really want to fallback at this point?

>From a bz comment (#984901, not sure why it's private):

Any NFS server has to support either tcp or rpcbind.  But it's OK for a
server to support only of those two.  So the only way to handle both
cases while continuing to retry after ECONNREFUSED is to alternate
between trying nfs4/tcp and rpcbind until you can connect to one or the
other.

If it's the rpcbind call that succeeds first then I think we want to do
one more try of nfs4/tcp just to make sure it didn't just come up,
before falling back to v3.

The rpcbind call is done in userspace, if I understand right, so I think
this is doable.  Looking at utils/mount/ I don't understand the mount
process well enough to understand exactly how to do it.  Maybe
everything but the final nfs_sys_mount needs to be moved out of
nfs_do_mount_v3v2 into a new nfs_do_probe_v3v2 and nfs_autonegotiate
should alternate between nfs_try_mount_v4 and nfs_do_probe_v3v2 as long
as both return ECONNREFUSED, calling nfs_try_mount_v3v2 only if
nfs_try_mount_v4 has failed after a succesful nfs_do_probe_v3v2?

Except the v3v2 mount logic seems to actually modify the mount_options,
so probably that doesn't quite work.

?

--b.

Re: [PATCH - nfs-utils] Fix fallback from tcp to udp

[J. Bruce Fields]

On Thu, Feb 20, 2014 at 03:37:01PM -0500, bfields wrote:
> On Thu, Feb 20, 2014 at 12:50:15PM -0500, Steve Dickson wrote:
> > 
> > 
> > On 02/17/2014 06:43 PM, NeilBrown wrote:
> > > 
> > > Protocol negotiation in mount.nfs does not correctly negotiate with a
> > > server which only support NFSv3 and UDP.
> > > 
> > > When mount.nfs attempts an NFSv4 mount and fails with ECONNREFUSED
> > > it does not fall back to NFSv3, as this is not recognised as a
> > > "does not support NFSv4" error.
> > > However ECONNREFUSED is a clear indication that the server doesn't
> > > support TCP, and ipso facto does not support NFSv4.
> > > So ECONNREFUSED should trigger a fallback from v4 to v2/3.
> > I'm also pretty this is the error returned when the server is 
> > down or more pointy when server is rebooting...
> 
> Probably worth checking that.
> 
> > Do we really want to fallback at this point?
> 
> From a bz comment (#984901, not sure why it's private):
> 
> Any NFS server has to support either tcp or rpcbind.  But it's OK for a
> server to support only of those two.  So the only way to handle both
> cases while continuing to retry after ECONNREFUSED

(But I'm not actually convinced that's true.  In particular I don't see
ECONNREFUSED when rebooting a server.  But I'm not clear when it's
returned....)

--b.

> is to alternate
> between trying nfs4/tcp and rpcbind until you can connect to one or the
> other.
> 
> If it's the rpcbind call that succeeds first then I think we want to do
> one more try of nfs4/tcp just to make sure it didn't just come up,
> before falling back to v3.
> 
> The rpcbind call is done in userspace, if I understand right, so I think
> this is doable.  Looking at utils/mount/ I don't understand the mount
> process well enough to understand exactly how to do it.  Maybe
> everything but the final nfs_sys_mount needs to be moved out of
> nfs_do_mount_v3v2 into a new nfs_do_probe_v3v2 and nfs_autonegotiate
> should alternate between nfs_try_mount_v4 and nfs_do_probe_v3v2 as long
> as both return ECONNREFUSED, calling nfs_try_mount_v3v2 only if
> nfs_try_mount_v4 has failed after a succesful nfs_do_probe_v3v2?
> 
> Except the v3v2 mount logic seems to actually modify the mount_options,
> so probably that doesn't quite work.
> 
> ?
> 
> --b.

Re: [PATCH - nfs-utils] Fix fallback from tcp to udp

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 2807 bytes --]

On Thu, 20 Feb 2014 15:37:02 -0500 "J. Bruce Fields" <bfields@fieldses.org>
wrote:

> On Thu, Feb 20, 2014 at 12:50:15PM -0500, Steve Dickson wrote:
> > 
> > 
> > On 02/17/2014 06:43 PM, NeilBrown wrote:
> > > 
> > > Protocol negotiation in mount.nfs does not correctly negotiate with a
> > > server which only support NFSv3 and UDP.
> > > 
> > > When mount.nfs attempts an NFSv4 mount and fails with ECONNREFUSED
> > > it does not fall back to NFSv3, as this is not recognised as a
> > > "does not support NFSv4" error.
> > > However ECONNREFUSED is a clear indication that the server doesn't
> > > support TCP, and ipso facto does not support NFSv4.
> > > So ECONNREFUSED should trigger a fallback from v4 to v2/3.
> > I'm also pretty this is the error returned when the server is 
> > down or more pointy when server is rebooting...
> 
> Probably worth checking that.

It is certainly possible that there is a window during boot when a server
will RST any SYN to port 2049.  The window may be very small, but it will
usually be there.

It is possible to configure a server to start listening before enabling the
interface, and so close the window completely.  But we certainly cannot
assume any server does this.

> 
> > Do we really want to fallback at this point?
> 
> >From a bz comment (#984901, not sure why it's private):
> 
> Any NFS server has to support either tcp or rpcbind.  But it's OK for a
> server to support only of those two.  So the only way to handle both
> cases while continuing to retry after ECONNREFUSED is to alternate
> between trying nfs4/tcp and rpcbind until you can connect to one or the
> other.
> 
> If it's the rpcbind call that succeeds first then I think we want to do
> one more try of nfs4/tcp just to make sure it didn't just come up,
> before falling back to v3.
> 
> The rpcbind call is done in userspace, if I understand right, so I think
> this is doable.  Looking at utils/mount/ I don't understand the mount
> process well enough to understand exactly how to do it.  Maybe
> everything but the final nfs_sys_mount needs to be moved out of
> nfs_do_mount_v3v2 into a new nfs_do_probe_v3v2 and nfs_autonegotiate
> should alternate between nfs_try_mount_v4 and nfs_do_probe_v3v2 as long
> as both return ECONNREFUSED, calling nfs_try_mount_v3v2 only if
> nfs_try_mount_v4 has failed after a succesful nfs_do_probe_v3v2?
> 
> Except the v3v2 mount logic seems to actually modify the mount_options,
> so probably that doesn't quite work.

I had come to much the same conclusion after reading Steve's mail:  when TCP
fails we need rpcbind to be sure what to do.
I suspect it should be fairly straight forward to implement (I'm less
pessimistic than you).  I'll have a go on Monday.

Thanks,
NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: [PATCH - nfs-utils] Fix fallback from tcp to udp

[J. Bruce Fields]

On Fri, Feb 21, 2014 at 02:26:41PM +1100, NeilBrown wrote:
> On Thu, 20 Feb 2014 15:37:02 -0500 "J. Bruce Fields" <bfields@fieldses.org>
> wrote:
> > Any NFS server has to support either tcp or rpcbind.  But it's OK for a
> > server to support only of those two.  So the only way to handle both
> > cases while continuing to retry after ECONNREFUSED is to alternate
> > between trying nfs4/tcp and rpcbind until you can connect to one or the
> > other.
> > 
> > If it's the rpcbind call that succeeds first then I think we want to do
> > one more try of nfs4/tcp just to make sure it didn't just come up,
> > before falling back to v3.
> > 
> > The rpcbind call is done in userspace, if I understand right, so I think
> > this is doable.  Looking at utils/mount/ I don't understand the mount
> > process well enough to understand exactly how to do it.  Maybe
> > everything but the final nfs_sys_mount needs to be moved out of
> > nfs_do_mount_v3v2 into a new nfs_do_probe_v3v2 and nfs_autonegotiate
> > should alternate between nfs_try_mount_v4 and nfs_do_probe_v3v2 as long
> > as both return ECONNREFUSED, calling nfs_try_mount_v3v2 only if
> > nfs_try_mount_v4 has failed after a succesful nfs_do_probe_v3v2?
> > 
> > Except the v3v2 mount logic seems to actually modify the mount_options,
> > so probably that doesn't quite work.
> 
> I had come to much the same conclusion after reading Steve's mail:  when TCP
> fails we need rpcbind to be sure what to do.
> I suspect it should be fairly straight forward to implement (I'm less
> pessimistic than you).  I'll have a go on Monday.

OK, great!

Yeah, the mount code looked like a maze for me but I probably spent less
than an hour trying to trace through it, I'm sure it's not that bad.

--b.

Re: [PATCH - nfs-utils] Fix fallback from tcp to udp

[Chuck Lever]

On Feb 21, 2014, at 6:59 AM, J. Bruce Fields <bfields@fieldses.org> wrote:

> On Fri, Feb 21, 2014 at 02:26:41PM +1100, NeilBrown wrote:
>> On Thu, 20 Feb 2014 15:37:02 -0500 "J. Bruce Fields" <bfields@fieldses.org>
>> wrote:
>>> Any NFS server has to support either tcp or rpcbind.  But it's OK for a
>>> server to support only of those two.  So the only way to handle both
>>> cases while continuing to retry after ECONNREFUSED is to alternate
>>> between trying nfs4/tcp and rpcbind until you can connect to one or the
>>> other.
>>> 
>>> If it's the rpcbind call that succeeds first then I think we want to do
>>> one more try of nfs4/tcp just to make sure it didn't just come up,
>>> before falling back to v3.
>>> 
>>> The rpcbind call is done in userspace, if I understand right, so I think
>>> this is doable.  Looking at utils/mount/ I don't understand the mount
>>> process well enough to understand exactly how to do it.  Maybe
>>> everything but the final nfs_sys_mount needs to be moved out of
>>> nfs_do_mount_v3v2 into a new nfs_do_probe_v3v2 and nfs_autonegotiate
>>> should alternate between nfs_try_mount_v4 and nfs_do_probe_v3v2 as long
>>> as both return ECONNREFUSED, calling nfs_try_mount_v3v2 only if
>>> nfs_try_mount_v4 has failed after a succesful nfs_do_probe_v3v2?
>>> 
>>> Except the v3v2 mount logic seems to actually modify the mount_options,
>>> so probably that doesn't quite work.
>> 
>> I had come to much the same conclusion after reading Steve's mail:  when TCP
>> fails we need rpcbind to be sure what to do.
>> I suspect it should be fairly straight forward to implement (I'm less
>> pessimistic than you).  I'll have a go on Monday.
> 
> OK, great!
> 
> Yeah, the mount code looked like a maze for me but I probably spent less
> than an hour trying to trace through it, I'm sure it's not that bad.

Just a general comment.  Mount negotiation is a maze because

  a) we have so many legacy use cases that still MUST work, and

  b) we have no regression test suite that can confirm that mount.nfs is still operating correctly after code changes

Thus it’s very difficult to clean up over time.  It just accretes more and more logic.  We add little bits here and there because it seems safe, but that adds up.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com