[nfs-utils PATCH] retry on EPERM from NFSv4 mount attempt

[nfs-utils PATCH] retry on EPERM from NFSv4 mount attempt

[Neil Brown]

Hi,
 I recently packaged nfs-utils 1.2.1 for openSUSE and fairly quickly
 got a bug report - "-o nfsvers=3" was needed to mount NFSv3
 filesystems.

 mount.nfs in 1.2.1 will first try a v4 mount but will fall-back to v3
 if it gets ENOENT.  This works fine.
 However for kernel prior to 2.6.25, you don't get ENOENT, you get
 EPERM.
 In that case the fall-back to v3 doesn't happen and you get a failure
 to mount.

 So I think we need to fall back on EPERM as well.  See below.

Thanks,
NeilBrown


diff --git a/utils/mount/stropts.c b/utils/mount/stropts.c
index b595649..68eb82b 100644
--- a/utils/mount/stropts.c
+++ b/utils/mount/stropts.c
@@ -657,8 +657,10 @@ static int nfs_try_mount(struct nfsmount_info *mi)
 				 * To deal with legacy Linux servers that don't
 				 * automatically export a pseudo root, retry
 				 * ENOENT errors using version 3
+				 * And for Linux servers prior to 2.6.25, retry
+				 * EPERM
 				 */
-				if (errno != ENOENT)
+				if (errno != ENOENT && errno != EPERM)
 					break;
 			}
 		}

Re: [nfs-utils PATCH] retry on EPERM from NFSv4 mount attempt

[Steve Dickson]

On 11/23/2009 06:32 PM, Neil Brown wrote:
> 
> Hi,
>  I recently packaged nfs-utils 1.2.1 for openSUSE and fairly quickly
>  got a bug report - "-o nfsvers=3" was needed to mount NFSv3
>  filesystems.
> 
>  mount.nfs in 1.2.1 will first try a v4 mount but will fall-back to v3
>  if it gets ENOENT.  This works fine.
>  However for kernel prior to 2.6.25, you don't get ENOENT, you get
>  EPERM.
>  In that case the fall-back to v3 doesn't happen and you get a failure
>  to mount.
> 
>  So I think we need to fall back on EPERM as well.  See below.
I already posted this patch on the v4 mailing list
http://linux-nfs.org/pipermail/nfsv4/2009-November/011595.html
but it got shot down...  at least that's how I interpreted the
responses... 

But I do thing we need this, since there are so many server 
that will simple break if we don't... Agreed?

steved. 


> 
> Thanks,
> NeilBrown
> 
> 
> diff --git a/utils/mount/stropts.c b/utils/mount/stropts.c
> index b595649..68eb82b 100644
> --- a/utils/mount/stropts.c
> +++ b/utils/mount/stropts.c
> @@ -657,8 +657,10 @@ static int nfs_try_mount(struct nfsmount_info *mi)
>  				 * To deal with legacy Linux servers that don't
>  				 * automatically export a pseudo root, retry
>  				 * ENOENT errors using version 3
> +				 * And for Linux servers prior to 2.6.25, retry
> +				 * EPERM
>  				 */
> -				if (errno != ENOENT)
> +				if (errno != ENOENT && errno != EPERM)
>  					break;
>  			}
>  		}

Re: [nfs-utils PATCH] retry on EPERM from NFSv4 mount attempt

[J. Bruce Fields]

On Tue, Nov 24, 2009 at 09:29:47AM -0500, Steve Dickson wrote:
> 
> 
> On 11/23/2009 06:32 PM, Neil Brown wrote:
> > 
> > Hi,
> >  I recently packaged nfs-utils 1.2.1 for openSUSE and fairly quickly
> >  got a bug report - "-o nfsvers=3" was needed to mount NFSv3
> >  filesystems.
> > 
> >  mount.nfs in 1.2.1 will first try a v4 mount but will fall-back to v3
> >  if it gets ENOENT.  This works fine.
> >  However for kernel prior to 2.6.25, you don't get ENOENT, you get
> >  EPERM.
> >  In that case the fall-back to v3 doesn't happen and you get a failure
> >  to mount.
> > 
> >  So I think we need to fall back on EPERM as well.  See below.
> I already posted this patch on the v4 mailing list
> http://linux-nfs.org/pipermail/nfsv4/2009-November/011595.html
> but it got shot down...  at least that's how I interpreted the
> responses... 
> 
> But I do thing we need this, since there are so many server 
> that will simple break if we don't... Agreed?

My position is that servers should either a) turn off NFSv4 or b) add a
pseudoroot, and that we should modify initscripts to make this harder to
screw up.

(For now (without automatic pseudoroot creation) we should by default be
running rpc.nfsd with -N 4; and adding the v4 support should be
something administrators do when they add a pseudoroot.)

((If this is really totally unfeasible, then I should quickly cue up a
revert for the patch I have queued for 2.6.33 which changes this error
again, to SERVERFAULT....))

--b.

> 
> steved. 
> 
> 
> > 
> > Thanks,
> > NeilBrown
> > 
> > 
> > diff --git a/utils/mount/stropts.c b/utils/mount/stropts.c
> > index b595649..68eb82b 100644
> > --- a/utils/mount/stropts.c
> > +++ b/utils/mount/stropts.c
> > @@ -657,8 +657,10 @@ static int nfs_try_mount(struct nfsmount_info *mi)
> >  				 * To deal with legacy Linux servers that don't
> >  				 * automatically export a pseudo root, retry
> >  				 * ENOENT errors using version 3
> > +				 * And for Linux servers prior to 2.6.25, retry
> > +				 * EPERM
> >  				 */
> > -				if (errno != ENOENT)
> > +				if (errno != ENOENT && errno != EPERM)
> >  					break;
> >  			}
> >  		}
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [nfs-utils PATCH] retry on EPERM from NFSv4 mount attempt

[Peter Staubach]

J. Bruce Fields wrote:
> On Tue, Nov 24, 2009 at 09:29:47AM -0500, Steve Dickson wrote:
>>
>> On 11/23/2009 06:32 PM, Neil Brown wrote:
>>> Hi,
>>>  I recently packaged nfs-utils 1.2.1 for openSUSE and fairly quickly
>>>  got a bug report - "-o nfsvers=3" was needed to mount NFSv3
>>>  filesystems.
>>>
>>>  mount.nfs in 1.2.1 will first try a v4 mount but will fall-back to v3
>>>  if it gets ENOENT.  This works fine.
>>>  However for kernel prior to 2.6.25, you don't get ENOENT, you get
>>>  EPERM.
>>>  In that case the fall-back to v3 doesn't happen and you get a failure
>>>  to mount.
>>>
>>>  So I think we need to fall back on EPERM as well.  See below.
>> I already posted this patch on the v4 mailing list
>> http://linux-nfs.org/pipermail/nfsv4/2009-November/011595.html
>> but it got shot down...  at least that's how I interpreted the
>> responses... 
>>
>> But I do thing we need this, since there are so many server 
>> that will simple break if we don't... Agreed?
> 
> My position is that servers should either a) turn off NFSv4 or b) add a
> pseudoroot, and that we should modify initscripts to make this harder to
> screw up.
> 
> (For now (without automatic pseudoroot creation) we should by default be
> running rpc.nfsd with -N 4; and adding the v4 support should be
> something administrators do when they add a pseudoroot.)
> 
> ((If this is really totally unfeasible, then I should quickly cue up a
> revert for the patch I have queued for 2.6.33 which changes this error
> again, to SERVERFAULT....))
> 

None helps new client systems who are forced to deal with older
server deployments.  It is a bad thing to force customers to change
old servers simply because they want to deploy new clients.  New
systems are generally just supposed to fit into existing networks
and not have to mold those networks to fit the new client's view
of the world.

Is this such a bad patch, to account for our own short sightedness
when designing the original NFSv4 server implementation?  It is
even at the user level and not in the kernel...

	Thanx...

		ps

Re: [nfs-utils PATCH] retry on EPERM from NFSv4 mount attempt

[Neil Brown]

On Tue, 24 Nov 2009 15:56:16 -0500
"J. Bruce Fields" <bfields@fieldses.org> wrote:

> On Tue, Nov 24, 2009 at 09:29:47AM -0500, Steve Dickson wrote:
> > >  So I think we need to fall back on EPERM as well.  See below.
> > I already posted this patch on the v4 mailing list
> > http://linux-nfs.org/pipermail/nfsv4/2009-November/011595.html
> > but it got shot down...  at least that's how I interpreted the
> > responses... 

Thanks for the reference ... clearly I am not keeping up with my
nfs mailing lists....


> > 
> > But I do thing we need this, since there are so many server 
> > that will simple break if we don't... Agreed?

Agreed that we certainly need something.  We cannot expect people to
reconfigure their servers because they installed new software on the
client.

> 
> My position is that servers should either a) turn off NFSv4 or b) add
> a pseudoroot, and that we should modify initscripts to make this
> harder to screw up.

In the ideal world, yes.  Maybe this is best done in the kernel?
Can we synthesis an RPC-protocol-not-supported error if and NFSv4
request arrives from a client for which no pseudo-root is configured?

> 
> (For now (without automatic pseudoroot creation) we should by default
> be running rpc.nfsd with -N 4; and adding the v4 support should be
> something administrators do when they add a pseudoroot.)

Hind sight is 20/20 they say.  We probably should have done this, but
I think it is now too late to do anything useful in the init scripts.

> 
> ((If this is really totally unfeasible, then I should quickly cue up a
> revert for the patch I have queued for 2.6.33 which changes this error
> again, to SERVERFAULT....))

Maybe - maybe not.

The situation on the client is that for a command that has
traditionally always performed a v3 or (before that) a v2 mount, we are
now trying a v4 mount.
I see that v4 attempt as opportunistic.  If anything goes wrong I think
it is reasonable to go back to "the old way".

So I think this piece of code in mount.nfs should retry on any error at
all, so it would not matter whether you change again to SERVERFAULT.

But I think the best fix for the kernel is to get nfsd4_proc_compound
to return RPC_PROG_MISMATCH if there is no pseudo root, and then get
svc_process_common to handle this and fake up appropriate min/max
version numbers.

Thanks,
NeilBrown

Re: [nfs-utils PATCH] retry on EPERM from NFSv4 mount attempt

[Peter Staubach]

Neil Brown wrote:
> On Tue, 24 Nov 2009 15:56:16 -0500
> "J. Bruce Fields" <bfields@fieldses.org> wrote:
> 
>> On Tue, Nov 24, 2009 at 09:29:47AM -0500, Steve Dickson wrote:
>>>>  So I think we need to fall back on EPERM as well.  See below.
>>> I already posted this patch on the v4 mailing list
>>> http://linux-nfs.org/pipermail/nfsv4/2009-November/011595.html
>>> but it got shot down...  at least that's how I interpreted the
>>> responses... 
> 
> Thanks for the reference ... clearly I am not keeping up with my
> nfs mailing lists....
> 
> 
>>> But I do thing we need this, since there are so many server 
>>> that will simple break if we don't... Agreed?
> 
> Agreed that we certainly need something.  We cannot expect people to
> reconfigure their servers because they installed new software on the
> client.
> 
>> My position is that servers should either a) turn off NFSv4 or b) add
>> a pseudoroot, and that we should modify initscripts to make this
>> harder to screw up.
> 
> In the ideal world, yes.  Maybe this is best done in the kernel?
> Can we synthesis an RPC-protocol-not-supported error if and NFSv4
> request arrives from a client for which no pseudo-root is configured?
> 
>> (For now (without automatic pseudoroot creation) we should by default
>> be running rpc.nfsd with -N 4; and adding the v4 support should be
>> something administrators do when they add a pseudoroot.)
> 
> Hind sight is 20/20 they say.  We probably should have done this, but
> I think it is now too late to do anything useful in the init scripts.
> 
>> ((If this is really totally unfeasible, then I should quickly cue up a
>> revert for the patch I have queued for 2.6.33 which changes this error
>> again, to SERVERFAULT....))
> 
> Maybe - maybe not.
> 
> The situation on the client is that for a command that has
> traditionally always performed a v3 or (before that) a v2 mount, we are
> now trying a v4 mount.
> I see that v4 attempt as opportunistic.  If anything goes wrong I think
> it is reasonable to go back to "the old way".
> 
> So I think this piece of code in mount.nfs should retry on any error at
> all, so it would not matter whether you change again to SERVERFAULT.
> 
> But I think the best fix for the kernel is to get nfsd4_proc_compound
> to return RPC_PROG_MISMATCH if there is no pseudo root, and then get
> svc_process_common to handle this and fake up appropriate min/max
> version numbers.
> 

I think that we might be better off in the long run by taking a
step back and getting all of the plumbing right, instead of
cluttering up things to have knowledge which they have no
business knowing or worrying about.

If the NFSv4 server gets a request which involves the root file
handle and one has not been defined, then it should return the
error that is defined by the protocol.  What the client chooses
to do with the error is up to it.

		ps

Re: [nfs-utils PATCH] retry on EPERM from NFSv4 mount attempt

[Neil Brown]

On Tue, 24 Nov 2009 16:58:29 -0500
Peter Staubach <staubach@redhat.com> wrote:

> I think that we might be better off in the long run by taking a
> step back and getting all of the plumbing right, instead of
> cluttering up things to have knowledge which they have no
> business knowing or worrying about.

In principle, I completely agree.


> 
> If the NFSv4 server gets a request which involves the root file
> handle and one has not been defined, then it should return the
> error that is defined by the protocol.  What the client chooses
> to do with the error is up to it.

There is no error for "root file handle has not been defined".

The only errors available for PUTROOTFH are:
 NFS4ERR_RESOURCE - which means "I'm exchausted after all the other
                    work you made me do" and shouldn't be returned for
                    the first op in a compound (that is an implied
                    restriction, not explicit).
 NFS4ERR_SERVERFAULT which means something strange went wrong.  This is
                    probably the closest, hence Bruce's recent patch to
                    use this error code.
 NFS4ERR_WRONGSEC   which means the security mechanism used by the
                    client isn't acceptable to the server.  This is
                    certainly not usable in this context.

So NFS4ERR_SERVERFAULT would be OK simply because it is a wildcard.
But RPC_PROG_MISMATCH, which means "I don't support that version of the
protocol" would also be correct in this case and it trivial for the
client to interpret.

NeilBrown

Re: [nfs-utils PATCH] retry on EPERM from NFSv4 mount attempt

[Peter Staubach]

Neil Brown wrote:
> On Tue, 24 Nov 2009 16:58:29 -0500
> Peter Staubach <staubach@redhat.com> wrote:
> 
>> I think that we might be better off in the long run by taking a
>> step back and getting all of the plumbing right, instead of
>> cluttering up things to have knowledge which they have no
>> business knowing or worrying about.
> 
> In principle, I completely agree.
> 
> 
>> If the NFSv4 server gets a request which involves the root file
>> handle and one has not been defined, then it should return the
>> error that is defined by the protocol.  What the client chooses
>> to do with the error is up to it.
> 
> There is no error for "root file handle has not been defined".
> 
> The only errors available for PUTROOTFH are:
>  NFS4ERR_RESOURCE - which means "I'm exchausted after all the other
>                     work you made me do" and shouldn't be returned for
>                     the first op in a compound (that is an implied
>                     restriction, not explicit).
>  NFS4ERR_SERVERFAULT which means something strange went wrong.  This is
>                     probably the closest, hence Bruce's recent patch to
>                     use this error code.
>  NFS4ERR_WRONGSEC   which means the security mechanism used by the
>                     client isn't acceptable to the server.  This is
>                     certainly not usable in this context.
> 
> So NFS4ERR_SERVERFAULT would be OK simply because it is a wildcard.
> But RPC_PROG_MISMATCH, which means "I don't support that version of the
> protocol" would also be correct in this case and it trivial for the
> client to interpret.
> 

Well, to the last section, that is an RPC error and not an
NFS error.  The RPC error, VERSMISMATCH, might be close, but
once again, is an RPC error and not an NFS error.

I don't think that an RPC error is correct to return if the
server is configured to support NFSv4.  It might not be
correctly configured and completely configured, but in
theory, it is supposed to support NFSv4.

I'd probably buy into the SERVERFAULT as the most logical error.

Doing something better is tough because of the way that exports
and the NFS service are managed as distinct things.  Perhaps if
the NFS service could detect that there were no NFSv4 exports,
then it could decline to register the NFSv4 service.  Then, some
of the RPC errors would make sense.

The layering in the current implementations seems to make things
more difficult than it does make them easy.  Just my opinion.

		ps

Re: [nfs-utils PATCH] retry on EPERM from NFSv4 mount attempt

[J. Bruce Fields]

On Tue, Nov 24, 2009 at 05:29:28PM -0500, Peter Staubach wrote:
> Neil Brown wrote:
> > On Tue, 24 Nov 2009 16:58:29 -0500
> > Peter Staubach <staubach@redhat.com> wrote:
> > 
> >> I think that we might be better off in the long run by taking a
> >> step back and getting all of the plumbing right, instead of
> >> cluttering up things to have knowledge which they have no
> >> business knowing or worrying about.
> > 
> > In principle, I completely agree.
> > 
> > 
> >> If the NFSv4 server gets a request which involves the root file
> >> handle and one has not been defined, then it should return the
> >> error that is defined by the protocol.  What the client chooses
> >> to do with the error is up to it.
> > 
> > There is no error for "root file handle has not been defined".
> > 
> > The only errors available for PUTROOTFH are:
> >  NFS4ERR_RESOURCE - which means "I'm exchausted after all the other
> >                     work you made me do" and shouldn't be returned for
> >                     the first op in a compound (that is an implied
> >                     restriction, not explicit).
> >  NFS4ERR_SERVERFAULT which means something strange went wrong.  This is
> >                     probably the closest, hence Bruce's recent patch to
> >                     use this error code.
> >  NFS4ERR_WRONGSEC   which means the security mechanism used by the
> >                     client isn't acceptable to the server.  This is
> >                     certainly not usable in this context.
> > 
> > So NFS4ERR_SERVERFAULT would be OK simply because it is a wildcard.
> > But RPC_PROG_MISMATCH, which means "I don't support that version of the
> > protocol" would also be correct in this case and it trivial for the
> > client to interpret.
> > 
> 
> Well, to the last section, that is an RPC error and not an
> NFS error.  The RPC error, VERSMISMATCH, might be close, but
> once again, is an RPC error and not an NFS error.

We certainly don't want to modify the putrootfh code to return
prog_mismatch.  First, it'd be annoying to figure out how to return an
rpc error at that point.  Second, it looks odd to the client, which may
have already sent other NFSv4 rpc's without trouble, and may have
trouble handling the error at this point (it would be within its rights
to assume we do support v4 if v4 NULL succeeds).  And third, because we
already have a simple established way to return prog_mismatch: just
start rpc.nfsd with -N 4.

> Doing something better is tough because of the way that exports
> and the NFS service are managed as distinct things.  Perhaps if
> the NFS service could detect that there were no NFSv4 exports,
> then it could decline to register the NFSv4 service.  Then, some
> of the RPC errors would make sense.
> 
> The layering in the current implementations seems to make things
> more difficult than it does make them easy.  Just my opinion.

You'd have to do an upcall to mountd to find out if there's an fsid=0
export before accepting any rpc's.  Sounds awkward.  It'd be easier if
we had an interface that allowed us to tell the server on NFS startup
whether there's a pseudoroot.  But we do: /proc/net/nfsd/versions.  And
if userspace is claiming to support v4 and then denying knowledge of any
pseudoroot in the mountd upcalls, then it's buggy.

(But, OK, agreed, we may just have to deal with the fact that lots of
servers are already configured that way and work around it on the
client.)

--b.

Re: [nfs-utils PATCH] retry on EPERM from NFSv4 mount attempt

[Trond Myklebust]

On Wed, 2009-11-25 at 09:22 +1100, Neil Brown wrote: 
> On Tue, 24 Nov 2009 16:58:29 -0500
> Peter Staubach <staubach@redhat.com> wrote:
> 
> > I think that we might be better off in the long run by taking a
> > step back and getting all of the plumbing right, instead of
> > cluttering up things to have knowledge which they have no
> > business knowing or worrying about.
> 
> In principle, I completely agree.
> 
> 
> > 
> > If the NFSv4 server gets a request which involves the root file
> > handle and one has not been defined, then it should return the
> > error that is defined by the protocol.  What the client chooses
> > to do with the error is up to it.
> 
> There is no error for "root file handle has not been defined".
> 
> The only errors available for PUTROOTFH are:
>  NFS4ERR_RESOURCE - which means "I'm exchausted after all the other
>                     work you made me do" and shouldn't be returned for
>                     the first op in a compound (that is an implied
>                     restriction, not explicit).
>  NFS4ERR_SERVERFAULT which means something strange went wrong.  This is
>                     probably the closest, hence Bruce's recent patch to
>                     use this error code.
>  NFS4ERR_WRONGSEC   which means the security mechanism used by the
>                     client isn't acceptable to the server.  This is
>                     certainly not usable in this context.
> 
> So NFS4ERR_SERVERFAULT would be OK simply because it is a wildcard.
> But RPC_PROG_MISMATCH, which means "I don't support that version of the
> protocol" would also be correct in this case and it trivial for the
> client to interpret.

One response that the spec allows is to accept the PUTROOTFH, but to
return either NFS4ERR_BADHANDLE or NFS4ERR_SERVERFAULT on any operation
that attempts to use the resulting filehandle.

Cheers
  Trond

Re: [nfs-utils PATCH] retry on EPERM from NFSv4 mount attempt

[Steve Dickson]

Sorry for the delayed response, I took some time off...

On 11/24/2009 09:29 AM, Steve Dickson wrote:
> 
> 
> On 11/23/2009 06:32 PM, Neil Brown wrote:
>>
>> Hi,
>>  I recently packaged nfs-utils 1.2.1 for openSUSE and fairly quickly
>>  got a bug report - "-o nfsvers=3" was needed to mount NFSv3
>>  filesystems.
>>
>>  mount.nfs in 1.2.1 will first try a v4 mount but will fall-back to v3
>>  if it gets ENOENT.  This works fine.
>>  However for kernel prior to 2.6.25, you don't get ENOENT, you get
>>  EPERM.
>>  In that case the fall-back to v3 doesn't happen and you get a failure
>>  to mount.
>>
>>  So I think we need to fall back on EPERM as well.  See below.
> I already posted this patch on the v4 mailing list
> http://linux-nfs.org/pipermail/nfsv4/2009-November/011595.html
> but it got shot down...  at least that's how I interpreted the
> responses... 
> 
> But I do thing we need this, since there are so many server 
> that will simple break if we don't... Agreed?

There appears to be a consensus that we need to do something
but no agreement on what.... 

I believe the options are:
   1) Start servers with '-N 4' when there is no root configured.
      * The problem I see with this is, in the event v4 support is wanted 
        its yet another configuration knob that has to be turned, basically
        making it more difficult for people use v4. 

   2) Change the kernel to return NFS4ERR_SERVERFAULT when there is no
      root configured. 
      * I see this is yet another errno the mounting code has to deal with..
        We are up to two errnos, do we really want to add a third?

Neither one of these solutions deal with legacy servers or move
the "v4 technology" forward... IMHO...   

So what I suggest we do is:
   1) Finish the dynamic pseudo root support asap, since all this
      goes away when that support exists. 

   2) We commit the "roll back to v3 on EPERM errors" and we draw a 
      line in the sand saying if a server does not return one of these
      errors on v4 mounts when there is a not a root configured, the
      server is broken and needs to be fix. Point, these are the only
      two errnos that will cause a v3 roll back.
      * My only concern is how we are going to work with 
        non-linux servers and how much extra network trace will 
        this cause during automount storms... 

But in the end, I think we will have to go with the "roll back on EPERM errors"
patch, because there is simply no way a mount command can be released 
in an enterprise environment that is not backward compatible with
existing servers.

Consensus? 

steved.
 

Re: [nfs-utils PATCH] retry on EPERM from NFSv4 mount attempt

[Chuck Lever]

On Nov 30, 2009, at 8:11 AM, Steve Dickson wrote:
> Sorry for the delayed response, I took some time off...
>
> On 11/24/2009 09:29 AM, Steve Dickson wrote:
>>
>>
>> On 11/23/2009 06:32 PM, Neil Brown wrote:
>>>
>>> Hi,
>>> I recently packaged nfs-utils 1.2.1 for openSUSE and fairly quickly
>>> got a bug report - "-o nfsvers=3" was needed to mount NFSv3
>>> filesystems.
>>>
>>> mount.nfs in 1.2.1 will first try a v4 mount but will fall-back to  
>>> v3
>>> if it gets ENOENT.  This works fine.
>>> However for kernel prior to 2.6.25, you don't get ENOENT, you get
>>> EPERM.
>>> In that case the fall-back to v3 doesn't happen and you get a  
>>> failure
>>> to mount.
>>>
>>> So I think we need to fall back on EPERM as well.  See below.
>> I already posted this patch on the v4 mailing list
>> http://linux-nfs.org/pipermail/nfsv4/2009-November/011595.html
>> but it got shot down...  at least that's how I interpreted the
>> responses...
>>
>> But I do thing we need this, since there are so many server
>> that will simple break if we don't... Agreed?
>
> There appears to be a consensus that we need to do something
> but no agreement on what....
>
> I believe the options are:
>   1) Start servers with '-N 4' when there is no root configured.
>      * The problem I see with this is, in the event v4 support is  
> wanted
>        its yet another configuration knob that has to be turned,  
> basically
>        making it more difficult for people use v4.
>
>   2) Change the kernel to return NFS4ERR_SERVERFAULT when there is no
>      root configured.
>      * I see this is yet another errno the mounting code has to deal  
> with..
>        We are up to two errnos, do we really want to add a third?
>
> Neither one of these solutions deal with legacy servers or move
> the "v4 technology" forward... IMHO...

To quote Yoda, "No.  There is another."

Distributions can choose to revert the recently added behavior which  
makes mount.nfs try vers=4 by default.  That's what the mount config  
file is for, yes?  That seems like a responsible choice, given the  
number of legacy servers still in the field.

At some later point (say, when your pseudoroot work is more widely  
deployed), distributions can make vers=4 the default as shipped simply  
by flipping a switch in the mount config file.  For now, the default  
behavior stays the same, with an option for individual sites to try  
something new.  In my opinion, default behavior out of the shrinkwrap  
should always be the behavior most likely to work in any environment  
-- principle of least surprise, and all that sort of thing.

This seems like the way we normally deploy new features like this, and  
would give the many very recent changes in mount.nfs more soak time  
with early adopters before we force them on everyone.

I know you won't like that idea Steve, but at least we should discuss  
it before rejecting the idea out of hand.

-- 
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

Re: [nfs-utils PATCH] retry on EPERM from NFSv4 mount attempt

[Steve Dickson]

On 11/30/2009 11:43 AM, Chuck Lever wrote:
> On Nov 30, 2009, at 8:11 AM, Steve Dickson wrote:
>> Sorry for the delayed response, I took some time off...
>>
>> On 11/24/2009 09:29 AM, Steve Dickson wrote:
>>>
>>>
>>> On 11/23/2009 06:32 PM, Neil Brown wrote:
>>>>
>>>> Hi,
>>>> I recently packaged nfs-utils 1.2.1 for openSUSE and fairly quickly
>>>> got a bug report - "-o nfsvers=3" was needed to mount NFSv3
>>>> filesystems.
>>>>
>>>> mount.nfs in 1.2.1 will first try a v4 mount but will fall-back to v3
>>>> if it gets ENOENT.  This works fine.
>>>> However for kernel prior to 2.6.25, you don't get ENOENT, you get
>>>> EPERM.
>>>> In that case the fall-back to v3 doesn't happen and you get a failure
>>>> to mount.
>>>>
>>>> So I think we need to fall back on EPERM as well.  See below.
>>> I already posted this patch on the v4 mailing list
>>> http://linux-nfs.org/pipermail/nfsv4/2009-November/011595.html
>>> but it got shot down...  at least that's how I interpreted the
>>> responses...
>>>
>>> But I do thing we need this, since there are so many server
>>> that will simple break if we don't... Agreed?
>>
>> There appears to be a consensus that we need to do something
>> but no agreement on what....
>>
>> I believe the options are:
>>   1) Start servers with '-N 4' when there is no root configured.
>>      * The problem I see with this is, in the event v4 support is wanted
>>        its yet another configuration knob that has to be turned,
>> basically
>>        making it more difficult for people use v4.
>>
>>   2) Change the kernel to return NFS4ERR_SERVERFAULT when there is no
>>      root configured.
>>      * I see this is yet another errno the mounting code has to deal
>> with..
>>        We are up to two errnos, do we really want to add a third?
>>
>> Neither one of these solutions deal with legacy servers or move
>> the "v4 technology" forward... IMHO...
> 
> To quote Yoda, "No.  There is another."
> 
> Distributions can choose to revert the recently added behavior which
> makes mount.nfs try vers=4 by default.  That's what the mount config
> file is for, yes?  That seems like a responsible choice, given the
> number of legacy servers still in the field.
Yes... That was the main reason for the configuration file.

> 
> At some later point (say, when your pseudoroot work is more widely
> deployed), distributions can make vers=4 the default as shipped simply
> by flipping a switch in the mount config file.  For now, the default
> behavior stays the same, with an option for individual sites to try
> something new.  In my opinion, default behavior out of the shrinkwrap
> should always be the behavior most likely to work in any environment --
> principle of least surprise, and all that sort of thing.
Not matter how or when we flip the switch, we'll have to deal 
with the legacy servers...

> 
> This seems like the way we normally deploy new features like this, and
> would give the many very recent changes in mount.nfs more soak time with
> early adopters before we force them on everyone.
Take a look at the Fedora 12... Its set up exactly as you just described
with only difference vers=3 is set in the configuration file...

So in my opinion, we are at the point of no return.... ;-)

steved.
 

Re: [nfs-utils PATCH] retry on EPERM from NFSv4 mount attempt

[Chuck Lever]

On Nov 30, 2009, at 12:41 PM, Steve Dickson wrote:
> On 11/30/2009 11:43 AM, Chuck Lever wrote:
>> On Nov 30, 2009, at 8:11 AM, Steve Dickson wrote:
>>> Sorry for the delayed response, I took some time off...
>>>
>>> On 11/24/2009 09:29 AM, Steve Dickson wrote:
>>>>
>>>>
>>>> On 11/23/2009 06:32 PM, Neil Brown wrote:
>>>>>
>>>>> Hi,
>>>>> I recently packaged nfs-utils 1.2.1 for openSUSE and fairly  
>>>>> quickly
>>>>> got a bug report - "-o nfsvers=3" was needed to mount NFSv3
>>>>> filesystems.
>>>>>
>>>>> mount.nfs in 1.2.1 will first try a v4 mount but will fall-back  
>>>>> to v3
>>>>> if it gets ENOENT.  This works fine.
>>>>> However for kernel prior to 2.6.25, you don't get ENOENT, you get
>>>>> EPERM.
>>>>> In that case the fall-back to v3 doesn't happen and you get a  
>>>>> failure
>>>>> to mount.
>>>>>
>>>>> So I think we need to fall back on EPERM as well.  See below.
>>>> I already posted this patch on the v4 mailing list
>>>> http://linux-nfs.org/pipermail/nfsv4/2009-November/011595.html
>>>> but it got shot down...  at least that's how I interpreted the
>>>> responses...
>>>>
>>>> But I do thing we need this, since there are so many server
>>>> that will simple break if we don't... Agreed?
>>>
>>> There appears to be a consensus that we need to do something
>>> but no agreement on what....
>>>
>>> I believe the options are:
>>>  1) Start servers with '-N 4' when there is no root configured.
>>>     * The problem I see with this is, in the event v4 support is  
>>> wanted
>>>       its yet another configuration knob that has to be turned,
>>> basically
>>>       making it more difficult for people use v4.
>>>
>>>  2) Change the kernel to return NFS4ERR_SERVERFAULT when there is no
>>>     root configured.
>>>     * I see this is yet another errno the mounting code has to deal
>>> with..
>>>       We are up to two errnos, do we really want to add a third?
>>>
>>> Neither one of these solutions deal with legacy servers or move
>>> the "v4 technology" forward... IMHO...
>>
>> To quote Yoda, "No.  There is another."
>>
>> Distributions can choose to revert the recently added behavior which
>> makes mount.nfs try vers=4 by default.  That's what the mount config
>> file is for, yes?  That seems like a responsible choice, given the
>> number of legacy servers still in the field.
> Yes... That was the main reason for the configuration file.
>
>> At some later point (say, when your pseudoroot work is more widely
>> deployed), distributions can make vers=4 the default as shipped  
>> simply
>> by flipping a switch in the mount config file.  For now, the default
>> behavior stays the same, with an option for individual sites to try
>> something new.  In my opinion, default behavior out of the shrinkwrap
>> should always be the behavior most likely to work in any  
>> environment --
>> principle of least surprise, and all that sort of thing.
> Not matter how or when we flip the switch, we'll have to deal
> with the legacy servers...

Well, the point of waiting would be so that the proportion of legacy  
servers would become relatively insignificant.  And, a number of  
reasonable longer term solutions which have been proposed here could  
be deployed in the meantime.

>> This seems like the way we normally deploy new features like this,  
>> and
>> would give the many very recent changes in mount.nfs more soak time  
>> with
>> early adopters before we force them on everyone.
> Take a look at the Fedora 12... Its set up exactly as you just  
> described
> with only difference vers=3 is set in the configuration file...

Perhaps that should be the upstream default as well, for now, so other  
distributors aren't taken by surprise when they update to 1.2.1.  Just  
a thought.

-- 
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

Re: [nfs-utils PATCH] retry on EPERM from NFSv4 mount attempt

[Steve Dickson]

On 11/30/2009 12:52 PM, Chuck Lever wrote:
> On Nov 30, 2009, at 12:41 PM, Steve Dickson wrote:
>> On 11/30/2009 11:43 AM, Chuck Lever wrote:
>>> On Nov 30, 2009, at 8:11 AM, Steve Dickson wrote:
>>>> Sorry for the delayed response, I took some time off...
>>>>
>>>> On 11/24/2009 09:29 AM, Steve Dickson wrote:
>>>>>
>>>>>
>>>>> On 11/23/2009 06:32 PM, Neil Brown wrote:
>>>>>>
>>>>>> Hi,
>>>>>> I recently packaged nfs-utils 1.2.1 for openSUSE and fairly quickly
>>>>>> got a bug report - "-o nfsvers=3" was needed to mount NFSv3
>>>>>> filesystems.
>>>>>>
>>>>>> mount.nfs in 1.2.1 will first try a v4 mount but will fall-back to v3
>>>>>> if it gets ENOENT.  This works fine.
>>>>>> However for kernel prior to 2.6.25, you don't get ENOENT, you get
>>>>>> EPERM.
>>>>>> In that case the fall-back to v3 doesn't happen and you get a failure
>>>>>> to mount.
>>>>>>
>>>>>> So I think we need to fall back on EPERM as well.  See below.
>>>>> I already posted this patch on the v4 mailing list
>>>>> http://linux-nfs.org/pipermail/nfsv4/2009-November/011595.html
>>>>> but it got shot down...  at least that's how I interpreted the
>>>>> responses...
>>>>>
>>>>> But I do thing we need this, since there are so many server
>>>>> that will simple break if we don't... Agreed?
>>>>
>>>> There appears to be a consensus that we need to do something
>>>> but no agreement on what....
>>>>
>>>> I believe the options are:
>>>>  1) Start servers with '-N 4' when there is no root configured.
>>>>     * The problem I see with this is, in the event v4 support is wanted
>>>>       its yet another configuration knob that has to be turned,
>>>> basically
>>>>       making it more difficult for people use v4.
>>>>
>>>>  2) Change the kernel to return NFS4ERR_SERVERFAULT when there is no
>>>>     root configured.
>>>>     * I see this is yet another errno the mounting code has to deal
>>>> with..
>>>>       We are up to two errnos, do we really want to add a third?
>>>>
>>>> Neither one of these solutions deal with legacy servers or move
>>>> the "v4 technology" forward... IMHO...
>>>
>>> To quote Yoda, "No.  There is another."
>>>
>>> Distributions can choose to revert the recently added behavior which
>>> makes mount.nfs try vers=4 by default.  That's what the mount config
>>> file is for, yes?  That seems like a responsible choice, given the
>>> number of legacy servers still in the field.
>> Yes... That was the main reason for the configuration file.
>>
>>> At some later point (say, when your pseudoroot work is more widely
>>> deployed), distributions can make vers=4 the default as shipped simply
>>> by flipping a switch in the mount config file.  For now, the default
>>> behavior stays the same, with an option for individual sites to try
>>> something new.  In my opinion, default behavior out of the shrinkwrap
>>> should always be the behavior most likely to work in any environment --
>>> principle of least surprise, and all that sort of thing.
>> Not matter how or when we flip the switch, we'll have to deal
>> with the legacy servers...
> 
> Well, the point of waiting would be so that the proportion of legacy
> servers would become relatively insignificant.  And, a number of
> reasonable longer term solutions which have been proposed here could be
> deployed in the meantime.
If we don't turn the switch, the "reasonable longer term solutions" will
not be need... ;-) At the end of the day, if the pain threshold becomes
too disruptive, we (or they) can always dial the version back... 

> 
>>> This seems like the way we normally deploy new features like this, and
>>> would give the many very recent changes in mount.nfs more soak time with
>>> early adopters before we force them on everyone.
>> Take a look at the Fedora 12... Its set up exactly as you just described
>> with only difference vers=3 is set in the configuration file...
> 
> Perhaps that should be the upstream default as well, for now, so other
> distributors aren't taken by surprise when they update to 1.2.1.  Just a
> thought.
> 
The configuration file is not enabled by default with the idea being
distro could do what they want with it... If it turns out the majority of
distros are using the config file, then it should be enabled by default,
but at this point I know of only one distro using it.. 

steved.

Re: [nfs-utils PATCH] retry on EPERM from NFSv4 mount attempt

[J. Bruce Fields]

On Mon, Nov 30, 2009 at 08:11:42AM -0500, Steve Dickson wrote:
> Sorry for the delayed response, I took some time off...
> 
> On 11/24/2009 09:29 AM, Steve Dickson wrote:
> > 
> > 
> > On 11/23/2009 06:32 PM, Neil Brown wrote:
> >>
> >> Hi,
> >>  I recently packaged nfs-utils 1.2.1 for openSUSE and fairly quickly
> >>  got a bug report - "-o nfsvers=3" was needed to mount NFSv3
> >>  filesystems.
> >>
> >>  mount.nfs in 1.2.1 will first try a v4 mount but will fall-back to v3
> >>  if it gets ENOENT.  This works fine.
> >>  However for kernel prior to 2.6.25, you don't get ENOENT, you get
> >>  EPERM.
> >>  In that case the fall-back to v3 doesn't happen and you get a failure
> >>  to mount.
> >>
> >>  So I think we need to fall back on EPERM as well.  See below.
> > I already posted this patch on the v4 mailing list
> > http://linux-nfs.org/pipermail/nfsv4/2009-November/011595.html
> > but it got shot down...  at least that's how I interpreted the
> > responses... 
> > 
> > But I do thing we need this, since there are so many server 
> > that will simple break if we don't... Agreed?
> 
> There appears to be a consensus that we need to do something
> but no agreement on what.... 
> 
> I believe the options are:

Note we want to do more than one of these; in particular:

>    1) Start servers with '-N 4' when there is no root configured.

This is required.  The current behavior is a bug, and we must not start
servers with v4 support without having a pseudoroot.

>       * The problem I see with this is, in the event v4 support is wanted 
>         its yet another configuration knob that has to be turned, basically
>         making it more difficult for people use v4. 

Yes.  There are things we can do to help (the pseudoroot patches, for
example).  However, continuing to turn on v4 by default in the absence
of a pseudoroot is not an option.

>    2) Change the kernel to return NFS4ERR_SERVERFAULT when there is no
>       root configured. 
>       * I see this is yet another errno the mounting code has to deal with..
>         We are up to two errnos, do we really want to add a third?

I think SERVERFAULT is a little more accurate, but I'm open to argument.

> Neither one of these solutions deal with legacy servers or move
> the "v4 technology" forward... IMHO...   

Yes, so we also need to do one of these:

> So what I suggest we do is:
>    1) Finish the dynamic pseudo root support asap, since all this
>       goes away when that support exists. 

Sure.

>    2) We commit the "roll back to v3 on EPERM errors" and we draw a 
>       line in the sand saying if a server does not return one of these
>       errors on v4 mounts when there is a not a root configured, the
>       server is broken and needs to be fix. Point, these are the only
>       two errnos that will cause a v3 roll back.
>       * My only concern is how we are going to work with 
>         non-linux servers and how much extra network trace will 
>         this cause during automount storms... 
> 
> But in the end, I think we will have to go with the "roll back on EPERM errors"
> patch, because there is simply no way a mount command can be released 
> in an enterprise environment that is not backward compatible with
> existing servers.

OK.

--b.

Re: [nfs-utils PATCH] retry on EPERM from NFSv4 mount attempt

[Neil Brown]

On Mon, 30 Nov 2009 13:18:58 -0500
"J. Bruce Fields" <bfields@fieldses.org> wrote:

> Note we want to do more than one of these; in particular:
> 
> >    1) Start servers with '-N 4' when there is no root configured.  
> 
> This is required.  The current behavior is a bug, and we must not
> start servers with v4 support without having a pseudoroot.

Yes, you could describe the current behaviour as a bug, but I don't
think it necessarily follows than encouraging the use of "-N 4" is an
appropriate resolution to the bug.  I agree with Steve that that would
seem like a backwards step.

Completing the auto-pseudo-root work and getting that active would also
fix the bug, and would do it in a more forward-looking way.

So while there may be a case of advising people (in a FAQ?) that using 
-N4 might be appropriate if no fsid=root is configured, I don't think
there is any point in trying to make it a default through making
any changes to the nfs-utils packages.

> >    2) Change the kernel to return NFS4ERR_SERVERFAULT when there is no
> >       root configured. 
> >       * I see this is yet another errno the mounting code has to deal with..
> >         We are up to two errnos, do we really want to add a third?  
>
> I think SERVERFAULT is a little more accurate, but I'm open to argument.

I think SERVERFAULT is the only vaguely relevant error permitted by the
RFC, so it think that should be returned.
As I said previously, I think mount.nfs should fall-back from v4 to v3
on any error when no specific version has been requested.

NeilBrown

Re: [nfs-utils PATCH] retry on EPERM from NFSv4 mount attempt

[J. Bruce Fields]

On Tue, Dec 01, 2009 at 08:59:16AM +1100, Neil Brown wrote:
> On Mon, 30 Nov 2009 13:18:58 -0500
> "J. Bruce Fields" <bfields@fieldses.org> wrote:
> 
> > Note we want to do more than one of these; in particular:
> > 
> > >    1) Start servers with '-N 4' when there is no root configured.  
> > 
> > This is required.  The current behavior is a bug, and we must not
> > start servers with v4 support without having a pseudoroot.
> 
> Yes, you could describe the current behaviour as a bug, but I don't
> think it necessarily follows than encouraging the use of "-N 4" is an
> appropriate resolution to the bug.  I agree with Steve that that would
> seem like a backwards step.

I agree that there's a usability disadvantage, but basic correctness
trumps usability here.

> Completing the auto-pseudo-root work and getting that active would also
> fix the bug, and would do it in a more forward-looking way.

Yes, it's a huge problem that that hasn't been resolved yet, sorry.  I
hope we have close-to-final patches posted today or tomorrow.

By the way, on a possibly related problem: there is one corner case
where Steve's proposed approach won't work: when the filesystems
traversed on the way to the root don't themselves support nfs exports.
I'd expect that to be rare, but was surprised to see at least one person
reporting doing it:

	http://marc.info/?l=linux-nfs&m=125787146106897&w=2

As a last resort, some day we may need some way to turn off v4 in that
case.

> So while there may be a case of advising people (in a FAQ?) that using 
> -N4 might be appropriate if no fsid=root is configured, I don't think
> there is any point in trying to make it a default through making
> any changes to the nfs-utils packages.

All that's needed is a different default in e.g. /etc/sysconfig/nfs, and
a note that v4 servers need to adjust the default.  It's annoying, but
not that annoying.

--b.

Re: [nfs-utils PATCH] retry on EPERM from NFSv4 mount attempt

[Steve Dickson]

On 11/23/2009 06:32 PM, Neil Brown wrote:
> 
> Hi,
>  I recently packaged nfs-utils 1.2.1 for openSUSE and fairly quickly
>  got a bug report - "-o nfsvers=3" was needed to mount NFSv3
>  filesystems.
> 
>  mount.nfs in 1.2.1 will first try a v4 mount but will fall-back to v3
>  if it gets ENOENT.  This works fine.
>  However for kernel prior to 2.6.25, you don't get ENOENT, you get
>  EPERM.
>  In that case the fall-back to v3 doesn't happen and you get a failure
>  to mount.
> 
>  So I think we need to fall back on EPERM as well.  See below.
Committed! We have to... but the line in the sand is drawn... 

steved.