3.7-rc1 NFSv3/sec=krb5 mkdir failure

3.7-rc1 NFSv3/sec=krb5 mkdir failure

[J. Bruce Fields]

On 3.7-rc1:

	client# mount -tnfs -osec=krb5,vers=3 server:/exports/ext4 /mnt/

		server# ls -l /exports/ext4|grep TMP
		server#

	# mkdir /mnt/TMP
	mkdir: cannot create directory `/mnt/TMP': Permission denied

		server# ls -l /exports/ext4|grep TMP
		drwxr-xr-x  2 nfsnobody nfsnobody 4096 Oct 16 08:56 TMP
		server#

Wireshark also shows that the create succeeds.

--b.

Re: 3.7-rc1 NFSv3/sec=krb5 mkdir failure

[J. Bruce Fields]

Anyone get a chance to look at this?  It seems very reproduceable.

--b.

On Tue, Oct 16, 2012 at 08:58:32AM -0400, bfields wrote:
> On 3.7-rc1:
> 
> 	client# mount -tnfs -osec=krb5,vers=3 server:/exports/ext4 /mnt/
> 
> 		server# ls -l /exports/ext4|grep TMP
> 		server#
> 
> 	# mkdir /mnt/TMP
> 	mkdir: cannot create directory `/mnt/TMP': Permission denied
> 
> 		server# ls -l /exports/ext4|grep TMP
> 		drwxr-xr-x  2 nfsnobody nfsnobody 4096 Oct 16 08:56 TMP
> 		server#
> 
> Wireshark also shows that the create succeeds.
> 
> --b.

RE: 3.7-rc1 NFSv3/sec=krb5 mkdir failure

[Myklebust, Trond]

> -----Original Message-----
> From: linux-nfs-owner@vger.kernel.org [mailto:linux-nfs-
> owner@vger.kernel.org] On Behalf Of J. Bruce Fields
> Sent: Wednesday, October 24, 2012 4:03 PM
> To: linux-nfs@vger.kernel.org; Myklebust, Trond; Schumaker, Bryan
> Subject: Re: 3.7-rc1 NFSv3/sec=krb5 mkdir failure
> 
> Anyone get a chance to look at this?  It seems very reproduceable.
> 
> --b.
> 
> On Tue, Oct 16, 2012 at 08:58:32AM -0400, bfields wrote:
> > On 3.7-rc1:
> >
> > 	client# mount -tnfs -osec=krb5,vers=3 server:/exports/ext4 /mnt/
> >
> > 		server# ls -l /exports/ext4|grep TMP
> > 		server#
> >
> > 	# mkdir /mnt/TMP
> > 	mkdir: cannot create directory `/mnt/TMP': Permission denied
> >
> > 		server# ls -l /exports/ext4|grep TMP
> > 		drwxr-xr-x  2 nfsnobody nfsnobody 4096 Oct 16 08:56 TMP
> > 		server#
> >
> > Wireshark also shows that the create succeeds.

Can you share the wireshark trace?

Cheers
  Trond

Re: 3.7-rc1 NFSv3/sec=krb5 mkdir failure

[J. Bruce Fields]

[-- Attachment #1: Type: text/plain, Size: 1129 bytes --]

On Wed, Oct 24, 2012 at 08:07:55PM +0000, Myklebust, Trond wrote:
> > -----Original Message-----
> > From: linux-nfs-owner@vger.kernel.org [mailto:linux-nfs-
> > owner@vger.kernel.org] On Behalf Of J. Bruce Fields
> > Sent: Wednesday, October 24, 2012 4:03 PM
> > To: linux-nfs@vger.kernel.org; Myklebust, Trond; Schumaker, Bryan
> > Subject: Re: 3.7-rc1 NFSv3/sec=krb5 mkdir failure
> > 
> > Anyone get a chance to look at this?  It seems very reproduceable.
> > 
> > --b.
> > 
> > On Tue, Oct 16, 2012 at 08:58:32AM -0400, bfields wrote:
> > > On 3.7-rc1:
> > >
> > > 	client# mount -tnfs -osec=krb5,vers=3 server:/exports/ext4 /mnt/
> > >
> > > 		server# ls -l /exports/ext4|grep TMP
> > > 		server#
> > >
> > > 	# mkdir /mnt/TMP
> > > 	mkdir: cannot create directory `/mnt/TMP': Permission denied
> > >
> > > 		server# ls -l /exports/ext4|grep TMP
> > > 		drwxr-xr-x  2 nfsnobody nfsnobody 4096 Oct 16 08:56 TMP
> > > 		server#
> > >
> > > Wireshark also shows that the create succeeds.
> 
> Can you share the wireshark trace?

Sure.  This covers the mount and mkdir.  The mkdir call and reply are in
frames 77 and 78.

--b.

[-- Attachment #2: tmp.pcap --]
[-- Type: application/cap, Size: 11110 bytes --]

RE: 3.7-rc1 NFSv3/sec=krb5 mkdir failure

[Myklebust, Trond]

> -----Original Message-----
> From: J. Bruce Fields [mailto:bfields@fieldses.org]
> Sent: Wednesday, October 24, 2012 4:15 PM
> To: Myklebust, Trond
> Cc: linux-nfs@vger.kernel.org; Schumaker, Bryan
> Subject: Re: 3.7-rc1 NFSv3/sec=krb5 mkdir failure
> 
> On Wed, Oct 24, 2012 at 08:07:55PM +0000, Myklebust, Trond wrote:
> > > -----Original Message-----
> > > From: linux-nfs-owner@vger.kernel.org [mailto:linux-nfs-
> > > owner@vger.kernel.org] On Behalf Of J. Bruce Fields
> > > Sent: Wednesday, October 24, 2012 4:03 PM
> > > To: linux-nfs@vger.kernel.org; Myklebust, Trond; Schumaker, Bryan
> > > Subject: Re: 3.7-rc1 NFSv3/sec=krb5 mkdir failure
> > >
> > > Anyone get a chance to look at this?  It seems very reproduceable.
> > >
> > > --b.
> > >
> > > On Tue, Oct 16, 2012 at 08:58:32AM -0400, bfields wrote:
> > > > On 3.7-rc1:
> > > >
> > > > 	client# mount -tnfs -osec=krb5,vers=3 server:/exports/ext4 /mnt/
> > > >
> > > > 		server# ls -l /exports/ext4|grep TMP
> > > > 		server#
> > > >
> > > > 	# mkdir /mnt/TMP
> > > > 	mkdir: cannot create directory `/mnt/TMP': Permission denied
> > > >
> > > > 		server# ls -l /exports/ext4|grep TMP
> > > > 		drwxr-xr-x  2 nfsnobody nfsnobody 4096 Oct 16 08:56 TMP
> > > > 		server#
> > > >
> > > > Wireshark also shows that the create succeeds.
> >
> > Can you share the wireshark trace?
> 
> Sure.  This covers the mount and mkdir.  The mkdir call and reply are in
> frames 77 and 78.

Hmm.... Can you please check if the ACL is being set correctly on the server? I suspect that might be the source of the error.

Cheers
  Trond

RE: 3.7-rc1 NFSv3/sec=krb5 mkdir failure

[Myklebust, Trond]

> -----Original Message-----
> From: Myklebust, Trond
> Sent: Wednesday, October 24, 2012 4:31 PM
> To: 'J. Bruce Fields'
> Cc: linux-nfs@vger.kernel.org; Schumaker, Bryan
> Subject: RE: 3.7-rc1 NFSv3/sec=krb5 mkdir failure
> 
> > -----Original Message-----
> > From: J. Bruce Fields [mailto:bfields@fieldses.org]
> > Sent: Wednesday, October 24, 2012 4:15 PM
> > To: Myklebust, Trond
> > Cc: linux-nfs@vger.kernel.org; Schumaker, Bryan
> > Subject: Re: 3.7-rc1 NFSv3/sec=krb5 mkdir failure
> >
> > On Wed, Oct 24, 2012 at 08:07:55PM +0000, Myklebust, Trond wrote:
> > > > -----Original Message-----
> > > > From: linux-nfs-owner@vger.kernel.org [mailto:linux-nfs-
> > > > owner@vger.kernel.org] On Behalf Of J. Bruce Fields
> > > > Sent: Wednesday, October 24, 2012 4:03 PM
> > > > To: linux-nfs@vger.kernel.org; Myklebust, Trond; Schumaker, Bryan
> > > > Subject: Re: 3.7-rc1 NFSv3/sec=krb5 mkdir failure
> > > >
> > > > Anyone get a chance to look at this?  It seems very reproduceable.
> > > >
> > > > --b.
> > > >
> > > > On Tue, Oct 16, 2012 at 08:58:32AM -0400, bfields wrote:
> > > > > On 3.7-rc1:
> > > > >
> > > > > 	client# mount -tnfs -osec=krb5,vers=3 server:/exports/ext4
> > > > > /mnt/
> > > > >
> > > > > 		server# ls -l /exports/ext4|grep TMP
> > > > > 		server#
> > > > >
> > > > > 	# mkdir /mnt/TMP
> > > > > 	mkdir: cannot create directory `/mnt/TMP': Permission denied
> > > > >
> > > > > 		server# ls -l /exports/ext4|grep TMP
> > > > > 		drwxr-xr-x  2 nfsnobody nfsnobody 4096 Oct 16 08:56 TMP
> > > > > 		server#
> > > > >
> > > > > Wireshark also shows that the create succeeds.
> > >
> > > Can you share the wireshark trace?
> >
> > Sure.  This covers the mount and mkdir.  The mkdir call and reply are
> > in frames 77 and 78.
> 
> Hmm.... Can you please check if the ACL is being set correctly on the server? I
> suspect that might be the source of the error.
> 

In fact, can you see if mounting with '-onoacl' causes the whole thing to succeed?

Cheers
  Trond

Re: 3.7-rc1 NFSv3/sec=krb5 mkdir failure

[J. Bruce Fields]

On Wed, Oct 24, 2012 at 08:31:16PM +0000, Myklebust, Trond wrote:
> > -----Original Message-----
> > From: J. Bruce Fields [mailto:bfields@fieldses.org]
> > Sent: Wednesday, October 24, 2012 4:15 PM
> > To: Myklebust, Trond
> > Cc: linux-nfs@vger.kernel.org; Schumaker, Bryan
> > Subject: Re: 3.7-rc1 NFSv3/sec=krb5 mkdir failure
> > 
> > On Wed, Oct 24, 2012 at 08:07:55PM +0000, Myklebust, Trond wrote:
> > > > -----Original Message-----
> > > > From: linux-nfs-owner@vger.kernel.org [mailto:linux-nfs-
> > > > owner@vger.kernel.org] On Behalf Of J. Bruce Fields
> > > > Sent: Wednesday, October 24, 2012 4:03 PM
> > > > To: linux-nfs@vger.kernel.org; Myklebust, Trond; Schumaker, Bryan
> > > > Subject: Re: 3.7-rc1 NFSv3/sec=krb5 mkdir failure
> > > >
> > > > Anyone get a chance to look at this?  It seems very reproduceable.
> > > >
> > > > --b.
> > > >
> > > > On Tue, Oct 16, 2012 at 08:58:32AM -0400, bfields wrote:
> > > > > On 3.7-rc1:
> > > > >
> > > > > 	client# mount -tnfs -osec=krb5,vers=3 server:/exports/ext4 /mnt/
> > > > >
> > > > > 		server# ls -l /exports/ext4|grep TMP
> > > > > 		server#
> > > > >
> > > > > 	# mkdir /mnt/TMP
> > > > > 	mkdir: cannot create directory `/mnt/TMP': Permission denied
> > > > >
> > > > > 		server# ls -l /exports/ext4|grep TMP
> > > > > 		drwxr-xr-x  2 nfsnobody nfsnobody 4096 Oct 16 08:56 TMP
> > > > > 		server#
> > > > >
> > > > > Wireshark also shows that the create succeeds.
> > >
> > > Can you share the wireshark trace?
> > 
> > Sure.  This covers the mount and mkdir.  The mkdir call and reply are in
> > frames 77 and 78.
> 
> Hmm.... Can you please check if the ACL is being set correctly on the server? I suspect that might be the source of the error.

ACLs on the export and the new directory are both trivial:

	[root@pip1 ~]# ls -ld /exports/ext4/
	drwxrwxrwx. 8 root root 4096 Oct 24 16:12 /exports/ext4/
	[root@pip1 ~]# getfacl /exports/ext4
	getfacl: Removing leading '/' from absolute path names
	# file: exports/ext4
	# owner: root
	# group: root
	user::rwx
	group::rwx
	other::rwx

	[root@pip1 ~]# ls -ld /exports/ext4/TMP
	drwxr-xr-x 2 nfsnobody nfsnobody 4096 Oct 24 16:12
	/exports/ext4/TMP
	[root@pip1 ~]# getfacl /exports/ext4/TMP
	getfacl: Removing leading '/' from absolute path names
	# file: exports/ext4/TMP
	# owner: nfsnobody
	# group: nfsnobody
	user::rwx
	group::r-x
	other::r-x

	[root@pip1 ~]# 

--b.

Re: 3.7-rc1 NFSv3/sec=krb5 mkdir failure

[J. Bruce Fields]

On Wed, Oct 24, 2012 at 08:34:37PM +0000, Myklebust, Trond wrote:
> > -----Original Message-----
> > From: Myklebust, Trond
> > Sent: Wednesday, October 24, 2012 4:31 PM
> > To: 'J. Bruce Fields'
> > Cc: linux-nfs@vger.kernel.org; Schumaker, Bryan
> > Subject: RE: 3.7-rc1 NFSv3/sec=krb5 mkdir failure
> > 
> > > -----Original Message-----
> > > From: J. Bruce Fields [mailto:bfields@fieldses.org]
> > > Sent: Wednesday, October 24, 2012 4:15 PM
> > > To: Myklebust, Trond
> > > Cc: linux-nfs@vger.kernel.org; Schumaker, Bryan
> > > Subject: Re: 3.7-rc1 NFSv3/sec=krb5 mkdir failure
> > >
> > > On Wed, Oct 24, 2012 at 08:07:55PM +0000, Myklebust, Trond wrote:
> > > > > -----Original Message-----
> > > > > From: linux-nfs-owner@vger.kernel.org [mailto:linux-nfs-
> > > > > owner@vger.kernel.org] On Behalf Of J. Bruce Fields
> > > > > Sent: Wednesday, October 24, 2012 4:03 PM
> > > > > To: linux-nfs@vger.kernel.org; Myklebust, Trond; Schumaker, Bryan
> > > > > Subject: Re: 3.7-rc1 NFSv3/sec=krb5 mkdir failure
> > > > >
> > > > > Anyone get a chance to look at this?  It seems very reproduceable.
> > > > >
> > > > > --b.
> > > > >
> > > > > On Tue, Oct 16, 2012 at 08:58:32AM -0400, bfields wrote:
> > > > > > On 3.7-rc1:
> > > > > >
> > > > > > 	client# mount -tnfs -osec=krb5,vers=3 server:/exports/ext4
> > > > > > /mnt/
> > > > > >
> > > > > > 		server# ls -l /exports/ext4|grep TMP
> > > > > > 		server#
> > > > > >
> > > > > > 	# mkdir /mnt/TMP
> > > > > > 	mkdir: cannot create directory `/mnt/TMP': Permission denied
> > > > > >
> > > > > > 		server# ls -l /exports/ext4|grep TMP
> > > > > > 		drwxr-xr-x  2 nfsnobody nfsnobody 4096 Oct 16 08:56 TMP
> > > > > > 		server#
> > > > > >
> > > > > > Wireshark also shows that the create succeeds.
> > > >
> > > > Can you share the wireshark trace?
> > >
> > > Sure.  This covers the mount and mkdir.  The mkdir call and reply are
> > > in frames 77 and 78.
> > 
> > Hmm.... Can you please check if the ACL is being set correctly on the server? I
> > suspect that might be the source of the error.
> > 
> 
> In fact, can you see if mounting with '-onoacl' causes the whole thing to succeed?

That's on the client mount command?  No difference.

--b.

Re: 3.7-rc1 NFSv3/sec=krb5 mkdir failure

[J. Bruce Fields]

On Wed, Oct 24, 2012 at 04:40:59PM -0400, J. Bruce Fields wrote:
> On Wed, Oct 24, 2012 at 08:34:37PM +0000, Myklebust, Trond wrote:
> > > -----Original Message-----
> > > From: Myklebust, Trond
> > > Sent: Wednesday, October 24, 2012 4:31 PM
> > > To: 'J. Bruce Fields'
> > > Cc: linux-nfs@vger.kernel.org; Schumaker, Bryan
> > > Subject: RE: 3.7-rc1 NFSv3/sec=krb5 mkdir failure
> > > 
> > > > -----Original Message-----
> > > > From: J. Bruce Fields [mailto:bfields@fieldses.org]
> > > > Sent: Wednesday, October 24, 2012 4:15 PM
> > > > To: Myklebust, Trond
> > > > Cc: linux-nfs@vger.kernel.org; Schumaker, Bryan
> > > > Subject: Re: 3.7-rc1 NFSv3/sec=krb5 mkdir failure
> > > >
> > > > On Wed, Oct 24, 2012 at 08:07:55PM +0000, Myklebust, Trond wrote:
> > > > > > -----Original Message-----
> > > > > > From: linux-nfs-owner@vger.kernel.org [mailto:linux-nfs-
> > > > > > owner@vger.kernel.org] On Behalf Of J. Bruce Fields
> > > > > > Sent: Wednesday, October 24, 2012 4:03 PM
> > > > > > To: linux-nfs@vger.kernel.org; Myklebust, Trond; Schumaker, Bryan
> > > > > > Subject: Re: 3.7-rc1 NFSv3/sec=krb5 mkdir failure
> > > > > >
> > > > > > Anyone get a chance to look at this?  It seems very reproduceable.
> > > > > >
> > > > > > --b.
> > > > > >
> > > > > > On Tue, Oct 16, 2012 at 08:58:32AM -0400, bfields wrote:
> > > > > > > On 3.7-rc1:
> > > > > > >
> > > > > > > 	client# mount -tnfs -osec=krb5,vers=3 server:/exports/ext4
> > > > > > > /mnt/
> > > > > > >
> > > > > > > 		server# ls -l /exports/ext4|grep TMP
> > > > > > > 		server#
> > > > > > >
> > > > > > > 	# mkdir /mnt/TMP
> > > > > > > 	mkdir: cannot create directory `/mnt/TMP': Permission denied
> > > > > > >
> > > > > > > 		server# ls -l /exports/ext4|grep TMP
> > > > > > > 		drwxr-xr-x  2 nfsnobody nfsnobody 4096 Oct 16 08:56 TMP
> > > > > > > 		server#
> > > > > > >
> > > > > > > Wireshark also shows that the create succeeds.
> > > > >
> > > > > Can you share the wireshark trace?
> > > >
> > > > Sure.  This covers the mount and mkdir.  The mkdir call and reply are
> > > > in frames 77 and 78.
> > > 
> > > Hmm.... Can you please check if the ACL is being set correctly on the server? I
> > > suspect that might be the source of the error.
> > > 
> > 
> > In fact, can you see if mounting with '-onoacl' causes the whole thing to succeed?
> 
> That's on the client mount command?  No difference.

By the way, I managed to do a little bisecting while working on
something else today, and blame landed on Chuck's
ba9b584c1dc37851d9c6ca6d0d2ccba55d9aad04 "SUNRPC: Introduce
rpc_clone_client_set_auth()".  Which makes some sense if it's an ACL
problem, and indeed testing on that commit finds success with -noacl,
failure without.

I'm not sure if that explains the failure I was seeing on 3.7-rc1, since
there I didn't see any ACL traffic, and still got a failure.  (And
-noacl didn't help.)

--b.

Re: 3.7-rc1 NFSv3/sec=krb5 mkdir failure

[Chuck Lever]

On Oct 28, 2012, at 12:15 PM, J. Bruce Fields <bfields@fieldses.org> wrote:

> On Wed, Oct 24, 2012 at 04:40:59PM -0400, J. Bruce Fields wrote:
>> On Wed, Oct 24, 2012 at 08:34:37PM +0000, Myklebust, Trond wrote:
>>>> -----Original Message-----
>>>> From: Myklebust, Trond
>>>> Sent: Wednesday, October 24, 2012 4:31 PM
>>>> To: 'J. Bruce Fields'
>>>> Cc: linux-nfs@vger.kernel.org; Schumaker, Bryan
>>>> Subject: RE: 3.7-rc1 NFSv3/sec=krb5 mkdir failure
>>>> 
>>>>> -----Original Message-----
>>>>> From: J. Bruce Fields [mailto:bfields@fieldses.org]
>>>>> Sent: Wednesday, October 24, 2012 4:15 PM
>>>>> To: Myklebust, Trond
>>>>> Cc: linux-nfs@vger.kernel.org; Schumaker, Bryan
>>>>> Subject: Re: 3.7-rc1 NFSv3/sec=krb5 mkdir failure
>>>>> 
>>>>> On Wed, Oct 24, 2012 at 08:07:55PM +0000, Myklebust, Trond wrote:
>>>>>>> -----Original Message-----
>>>>>>> From: linux-nfs-owner@vger.kernel.org [mailto:linux-nfs-
>>>>>>> owner@vger.kernel.org] On Behalf Of J. Bruce Fields
>>>>>>> Sent: Wednesday, October 24, 2012 4:03 PM
>>>>>>> To: linux-nfs@vger.kernel.org; Myklebust, Trond; Schumaker, Bryan
>>>>>>> Subject: Re: 3.7-rc1 NFSv3/sec=krb5 mkdir failure
>>>>>>> 
>>>>>>> Anyone get a chance to look at this?  It seems very reproduceable.
>>>>>>> 
>>>>>>> --b.
>>>>>>> 
>>>>>>> On Tue, Oct 16, 2012 at 08:58:32AM -0400, bfields wrote:
>>>>>>>> On 3.7-rc1:
>>>>>>>> 
>>>>>>>> 	client# mount -tnfs -osec=krb5,vers=3 server:/exports/ext4
>>>>>>>> /mnt/
>>>>>>>> 
>>>>>>>> 		server# ls -l /exports/ext4|grep TMP
>>>>>>>> 		server#
>>>>>>>> 
>>>>>>>> 	# mkdir /mnt/TMP
>>>>>>>> 	mkdir: cannot create directory `/mnt/TMP': Permission denied
>>>>>>>> 
>>>>>>>> 		server# ls -l /exports/ext4|grep TMP
>>>>>>>> 		drwxr-xr-x  2 nfsnobody nfsnobody 4096 Oct 16 08:56 TMP
>>>>>>>> 		server#
>>>>>>>> 
>>>>>>>> Wireshark also shows that the create succeeds.
>>>>>> 
>>>>>> Can you share the wireshark trace?
>>>>> 
>>>>> Sure.  This covers the mount and mkdir.  The mkdir call and reply are
>>>>> in frames 77 and 78.
>>>> 
>>>> Hmm.... Can you please check if the ACL is being set correctly on the server? I
>>>> suspect that might be the source of the error.
>>>> 
>>> 
>>> In fact, can you see if mounting with '-onoacl' causes the whole thing to succeed?
>> 
>> That's on the client mount command?  No difference.
> 
> By the way, I managed to do a little bisecting while working on
> something else today, and blame landed on Chuck's
> ba9b584c1dc37851d9c6ca6d0d2ccba55d9aad04 "SUNRPC: Introduce
> rpc_clone_client_set_auth()".  Which makes some sense if it's an ACL
> problem, and indeed testing on that commit finds success with -noacl,
> failure without.

After two weeks, Bruce and I were finally able to catch up in person.

I've reproduced this on 3.7-rc5 using cthon basic tests.  The first getacl operation fails because it's mistakenly attempting to set up a fresh GSS context on a transport where one already exists.  That's in line with the kind of change that's in commit ba9b584c1.

> I'm not sure if that explains the failure I was seeing on 3.7-rc1, since
> there I didn't see any ACL traffic, and still got a failure.  (And
> -noacl didn't help.)

The failure occurs on the client just before the getacl request is issued, so you won't see any ACL-related network traffic in the failure case.  The failure prevents any ACL request from succeeding.

Anyway, I'll have a deeper look at this tomorrow.

-- 
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

Re: 3.7-rc1 NFSv3/sec=krb5 mkdir failure

[J. Bruce Fields]

On Thu, Nov 15, 2012 at 09:03:03PM -0500, Chuck Lever wrote:
> 
> On Oct 28, 2012, at 12:15 PM, J. Bruce Fields <bfields@fieldses.org> wrote:
> 
> > On Wed, Oct 24, 2012 at 04:40:59PM -0400, J. Bruce Fields wrote:
> >> On Wed, Oct 24, 2012 at 08:34:37PM +0000, Myklebust, Trond wrote:
> >>>> -----Original Message-----
> >>>> From: Myklebust, Trond
> >>>> Sent: Wednesday, October 24, 2012 4:31 PM
> >>>> To: 'J. Bruce Fields'
> >>>> Cc: linux-nfs@vger.kernel.org; Schumaker, Bryan
> >>>> Subject: RE: 3.7-rc1 NFSv3/sec=krb5 mkdir failure
> >>>> 
> >>>>> -----Original Message-----
> >>>>> From: J. Bruce Fields [mailto:bfields@fieldses.org]
> >>>>> Sent: Wednesday, October 24, 2012 4:15 PM
> >>>>> To: Myklebust, Trond
> >>>>> Cc: linux-nfs@vger.kernel.org; Schumaker, Bryan
> >>>>> Subject: Re: 3.7-rc1 NFSv3/sec=krb5 mkdir failure
> >>>>> 
> >>>>> On Wed, Oct 24, 2012 at 08:07:55PM +0000, Myklebust, Trond wrote:
> >>>>>>> -----Original Message-----
> >>>>>>> From: linux-nfs-owner@vger.kernel.org [mailto:linux-nfs-
> >>>>>>> owner@vger.kernel.org] On Behalf Of J. Bruce Fields
> >>>>>>> Sent: Wednesday, October 24, 2012 4:03 PM
> >>>>>>> To: linux-nfs@vger.kernel.org; Myklebust, Trond; Schumaker, Bryan
> >>>>>>> Subject: Re: 3.7-rc1 NFSv3/sec=krb5 mkdir failure
> >>>>>>> 
> >>>>>>> Anyone get a chance to look at this?  It seems very reproduceable.
> >>>>>>> 
> >>>>>>> --b.
> >>>>>>> 
> >>>>>>> On Tue, Oct 16, 2012 at 08:58:32AM -0400, bfields wrote:
> >>>>>>>> On 3.7-rc1:
> >>>>>>>> 
> >>>>>>>> 	client# mount -tnfs -osec=krb5,vers=3 server:/exports/ext4
> >>>>>>>> /mnt/
> >>>>>>>> 
> >>>>>>>> 		server# ls -l /exports/ext4|grep TMP
> >>>>>>>> 		server#
> >>>>>>>> 
> >>>>>>>> 	# mkdir /mnt/TMP
> >>>>>>>> 	mkdir: cannot create directory `/mnt/TMP': Permission denied
> >>>>>>>> 
> >>>>>>>> 		server# ls -l /exports/ext4|grep TMP
> >>>>>>>> 		drwxr-xr-x  2 nfsnobody nfsnobody 4096 Oct 16 08:56 TMP
> >>>>>>>> 		server#
> >>>>>>>> 
> >>>>>>>> Wireshark also shows that the create succeeds.
> >>>>>> 
> >>>>>> Can you share the wireshark trace?
> >>>>> 
> >>>>> Sure.  This covers the mount and mkdir.  The mkdir call and reply are
> >>>>> in frames 77 and 78.
> >>>> 
> >>>> Hmm.... Can you please check if the ACL is being set correctly on the server? I
> >>>> suspect that might be the source of the error.
> >>>> 
> >>> 
> >>> In fact, can you see if mounting with '-onoacl' causes the whole thing to succeed?
> >> 
> >> That's on the client mount command?  No difference.
> > 
> > By the way, I managed to do a little bisecting while working on
> > something else today, and blame landed on Chuck's
> > ba9b584c1dc37851d9c6ca6d0d2ccba55d9aad04 "SUNRPC: Introduce
> > rpc_clone_client_set_auth()".  Which makes some sense if it's an ACL
> > problem, and indeed testing on that commit finds success with -noacl,
> > failure without.
> 
> After two weeks, Bruce and I were finally able to catch up in person.
> 
> I've reproduced this on 3.7-rc5 using cthon basic tests.  The first getacl operation fails because it's mistakenly attempting to set up a fresh GSS context on a transport where one already exists.  That's in line with the kind of change that's in commit ba9b584c1.
> 
> > I'm not sure if that explains the failure I was seeing on 3.7-rc1, since
> > there I didn't see any ACL traffic, and still got a failure.  (And
> > -noacl didn't help.)
> 
> The failure occurs on the client just before the getacl request is issued, so you won't see any ACL-related network traffic in the failure case.  The failure prevents any ACL request from succeeding.
> 
> Anyway, I'll have a deeper look at this tomorrow.

Great, thanks for taking another look.

--b.

Re: 3.7-rc1 NFSv3/sec=krb5 mkdir failure

[Myklebust, Trond]

T24gVGh1LCAyMDEyLTExLTE1IGF0IDIxOjAzIC0wNTAwLCBDaHVjayBMZXZlciB3cm90ZToNCj4g
T24gT2N0IDI4LCAyMDEyLCBhdCAxMjoxNSBQTSwgSi4gQnJ1Y2UgRmllbGRzIDxiZmllbGRzQGZp
ZWxkc2VzLm9yZz4gd3JvdGU6DQo+IA0KPiA+IE9uIFdlZCwgT2N0IDI0LCAyMDEyIGF0IDA0OjQw
OjU5UE0gLTA0MDAsIEouIEJydWNlIEZpZWxkcyB3cm90ZToNCj4gPj4gT24gV2VkLCBPY3QgMjQs
IDIwMTIgYXQgMDg6MzQ6MzdQTSArMDAwMCwgTXlrbGVidXN0LCBUcm9uZCB3cm90ZToNCj4gPj4+
PiAtLS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KPiA+Pj4+IEZyb206IE15a2xlYnVzdCwgVHJv
bmQNCj4gPj4+PiBTZW50OiBXZWRuZXNkYXksIE9jdG9iZXIgMjQsIDIwMTIgNDozMSBQTQ0KPiA+
Pj4+IFRvOiAnSi4gQnJ1Y2UgRmllbGRzJw0KPiA+Pj4+IENjOiBsaW51eC1uZnNAdmdlci5rZXJu
ZWwub3JnOyBTY2h1bWFrZXIsIEJyeWFuDQo+ID4+Pj4gU3ViamVjdDogUkU6IDMuNy1yYzEgTkZT
djMvc2VjPWtyYjUgbWtkaXIgZmFpbHVyZQ0KPiA+Pj4+IA0KPiA+Pj4+PiAtLS0tLU9yaWdpbmFs
IE1lc3NhZ2UtLS0tLQ0KPiA+Pj4+PiBGcm9tOiBKLiBCcnVjZSBGaWVsZHMgW21haWx0bzpiZmll
bGRzQGZpZWxkc2VzLm9yZ10NCj4gPj4+Pj4gU2VudDogV2VkbmVzZGF5LCBPY3RvYmVyIDI0LCAy
MDEyIDQ6MTUgUE0NCj4gPj4+Pj4gVG86IE15a2xlYnVzdCwgVHJvbmQNCj4gPj4+Pj4gQ2M6IGxp
bnV4LW5mc0B2Z2VyLmtlcm5lbC5vcmc7IFNjaHVtYWtlciwgQnJ5YW4NCj4gPj4+Pj4gU3ViamVj
dDogUmU6IDMuNy1yYzEgTkZTdjMvc2VjPWtyYjUgbWtkaXIgZmFpbHVyZQ0KPiA+Pj4+PiANCj4g
Pj4+Pj4gT24gV2VkLCBPY3QgMjQsIDIwMTIgYXQgMDg6MDc6NTVQTSArMDAwMCwgTXlrbGVidXN0
LCBUcm9uZCB3cm90ZToNCj4gPj4+Pj4+PiAtLS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KPiA+
Pj4+Pj4+IEZyb206IGxpbnV4LW5mcy1vd25lckB2Z2VyLmtlcm5lbC5vcmcgW21haWx0bzpsaW51
eC1uZnMtDQo+ID4+Pj4+Pj4gb3duZXJAdmdlci5rZXJuZWwub3JnXSBPbiBCZWhhbGYgT2YgSi4g
QnJ1Y2UgRmllbGRzDQo+ID4+Pj4+Pj4gU2VudDogV2VkbmVzZGF5LCBPY3RvYmVyIDI0LCAyMDEy
IDQ6MDMgUE0NCj4gPj4+Pj4+PiBUbzogbGludXgtbmZzQHZnZXIua2VybmVsLm9yZzsgTXlrbGVi
dXN0LCBUcm9uZDsgU2NodW1ha2VyLCBCcnlhbg0KPiA+Pj4+Pj4+IFN1YmplY3Q6IFJlOiAzLjct
cmMxIE5GU3YzL3NlYz1rcmI1IG1rZGlyIGZhaWx1cmUNCj4gPj4+Pj4+PiANCj4gPj4+Pj4+PiBB
bnlvbmUgZ2V0IGEgY2hhbmNlIHRvIGxvb2sgYXQgdGhpcz8gIEl0IHNlZW1zIHZlcnkgcmVwcm9k
dWNlYWJsZS4NCj4gPj4+Pj4+PiANCj4gPj4+Pj4+PiAtLWIuDQo+ID4+Pj4+Pj4gDQo+ID4+Pj4+
Pj4gT24gVHVlLCBPY3QgMTYsIDIwMTIgYXQgMDg6NTg6MzJBTSAtMDQwMCwgYmZpZWxkcyB3cm90
ZToNCj4gPj4+Pj4+Pj4gT24gMy43LXJjMToNCj4gPj4+Pj4+Pj4gDQo+ID4+Pj4+Pj4+IAljbGll
bnQjIG1vdW50IC10bmZzIC1vc2VjPWtyYjUsdmVycz0zIHNlcnZlcjovZXhwb3J0cy9leHQ0DQo+
ID4+Pj4+Pj4+IC9tbnQvDQo+ID4+Pj4+Pj4+IA0KPiA+Pj4+Pj4+PiAJCXNlcnZlciMgbHMgLWwg
L2V4cG9ydHMvZXh0NHxncmVwIFRNUA0KPiA+Pj4+Pj4+PiAJCXNlcnZlciMNCj4gPj4+Pj4+Pj4g
DQo+ID4+Pj4+Pj4+IAkjIG1rZGlyIC9tbnQvVE1QDQo+ID4+Pj4+Pj4+IAlta2RpcjogY2Fubm90
IGNyZWF0ZSBkaXJlY3RvcnkgYC9tbnQvVE1QJzogUGVybWlzc2lvbiBkZW5pZWQNCj4gPj4+Pj4+
Pj4gDQo+ID4+Pj4+Pj4+IAkJc2VydmVyIyBscyAtbCAvZXhwb3J0cy9leHQ0fGdyZXAgVE1QDQo+
ID4+Pj4+Pj4+IAkJZHJ3eHIteHIteCAgMiBuZnNub2JvZHkgbmZzbm9ib2R5IDQwOTYgT2N0IDE2
IDA4OjU2IFRNUA0KPiA+Pj4+Pj4+PiAJCXNlcnZlciMNCj4gPj4+Pj4+Pj4gDQo+ID4+Pj4+Pj4+
IFdpcmVzaGFyayBhbHNvIHNob3dzIHRoYXQgdGhlIGNyZWF0ZSBzdWNjZWVkcy4NCj4gPj4+Pj4+
IA0KPiA+Pj4+Pj4gQ2FuIHlvdSBzaGFyZSB0aGUgd2lyZXNoYXJrIHRyYWNlPw0KPiA+Pj4+PiAN
Cj4gPj4+Pj4gU3VyZS4gIFRoaXMgY292ZXJzIHRoZSBtb3VudCBhbmQgbWtkaXIuICBUaGUgbWtk
aXIgY2FsbCBhbmQgcmVwbHkgYXJlDQo+ID4+Pj4+IGluIGZyYW1lcyA3NyBhbmQgNzguDQo+ID4+
Pj4gDQo+ID4+Pj4gSG1tLi4uLiBDYW4geW91IHBsZWFzZSBjaGVjayBpZiB0aGUgQUNMIGlzIGJl
aW5nIHNldCBjb3JyZWN0bHkgb24gdGhlIHNlcnZlcj8gSQ0KPiA+Pj4+IHN1c3BlY3QgdGhhdCBt
aWdodCBiZSB0aGUgc291cmNlIG9mIHRoZSBlcnJvci4NCj4gPj4+PiANCj4gPj4+IA0KPiA+Pj4g
SW4gZmFjdCwgY2FuIHlvdSBzZWUgaWYgbW91bnRpbmcgd2l0aCAnLW9ub2FjbCcgY2F1c2VzIHRo
ZSB3aG9sZSB0aGluZyB0byBzdWNjZWVkPw0KPiA+PiANCj4gPj4gVGhhdCdzIG9uIHRoZSBjbGll
bnQgbW91bnQgY29tbWFuZD8gIE5vIGRpZmZlcmVuY2UuDQo+ID4gDQo+ID4gQnkgdGhlIHdheSwg
SSBtYW5hZ2VkIHRvIGRvIGEgbGl0dGxlIGJpc2VjdGluZyB3aGlsZSB3b3JraW5nIG9uDQo+ID4g
c29tZXRoaW5nIGVsc2UgdG9kYXksIGFuZCBibGFtZSBsYW5kZWQgb24gQ2h1Y2sncw0KPiA+IGJh
OWI1ODRjMWRjMzc4NTFkOWM2Y2E2ZDBkMmNjYmE1NWQ5YWFkMDQgIlNVTlJQQzogSW50cm9kdWNl
DQo+ID4gcnBjX2Nsb25lX2NsaWVudF9zZXRfYXV0aCgpIi4gIFdoaWNoIG1ha2VzIHNvbWUgc2Vu
c2UgaWYgaXQncyBhbiBBQ0wNCj4gPiBwcm9ibGVtLCBhbmQgaW5kZWVkIHRlc3Rpbmcgb24gdGhh
dCBjb21taXQgZmluZHMgc3VjY2VzcyB3aXRoIC1ub2FjbCwNCj4gPiBmYWlsdXJlIHdpdGhvdXQu
DQo+IA0KPiBBZnRlciB0d28gd2Vla3MsIEJydWNlIGFuZCBJIHdlcmUgZmluYWxseSBhYmxlIHRv
IGNhdGNoIHVwIGluIHBlcnNvbi4NCj4gDQo+IEkndmUgcmVwcm9kdWNlZCB0aGlzIG9uIDMuNy1y
YzUgdXNpbmcgY3Rob24gYmFzaWMgdGVzdHMuICBUaGUgZmlyc3QgZ2V0YWNsIG9wZXJhdGlvbiBm
YWlscyBiZWNhdXNlIGl0J3MgbWlzdGFrZW5seSBhdHRlbXB0aW5nIHRvIHNldCB1cCBhIGZyZXNo
IEdTUyBjb250ZXh0IG9uIGEgdHJhbnNwb3J0IHdoZXJlIG9uZSBhbHJlYWR5IGV4aXN0cy4gIFRo
YXQncyBpbiBsaW5lIHdpdGggdGhlIGtpbmQgb2YgY2hhbmdlIHRoYXQncyBpbiBjb21taXQgYmE5
YjU4NGMxLg0KDQpXaHkgc2hvdWxkbid0IHdlIGJlIGFibGUgdG8gY29wZSB3aXRoIG11bHRpcGxl
IEdTUyBzZXNzaW9ucyBvbiB0aGUgc2FtZQ0KdHJhbnNwb3J0Pw0KDQo+ID4gSSdtIG5vdCBzdXJl
IGlmIHRoYXQgZXhwbGFpbnMgdGhlIGZhaWx1cmUgSSB3YXMgc2VlaW5nIG9uIDMuNy1yYzEsIHNp
bmNlDQo+ID4gdGhlcmUgSSBkaWRuJ3Qgc2VlIGFueSBBQ0wgdHJhZmZpYywgYW5kIHN0aWxsIGdv
dCBhIGZhaWx1cmUuICAoQW5kDQo+ID4gLW5vYWNsIGRpZG4ndCBoZWxwLikNCj4gDQo+IFRoZSBm
YWlsdXJlIG9jY3VycyBvbiB0aGUgY2xpZW50IGp1c3QgYmVmb3JlIHRoZSBnZXRhY2wgcmVxdWVz
dCBpcyBpc3N1ZWQsIHNvIHlvdSB3b24ndCBzZWUgYW55IEFDTC1yZWxhdGVkIG5ldHdvcmsgdHJh
ZmZpYyBpbiB0aGUgZmFpbHVyZSBjYXNlLiAgVGhlIGZhaWx1cmUgcHJldmVudHMgYW55IEFDTCBy
ZXF1ZXN0IGZyb20gc3VjY2VlZGluZy4NCg0KSXMgaXQgZ3NzZCB0aGF0IGlzIGZhaWxpbmcgdGhl
bj8NCg0KPiBBbnl3YXksIEknbGwgaGF2ZSBhIGRlZXBlciBsb29rIGF0IHRoaXMgdG9tb3Jyb3cu
DQoNClRoYW5rcyENCg0KLS0gDQpUcm9uZCBNeWtsZWJ1c3QNCkxpbnV4IE5GUyBjbGllbnQgbWFp
bnRhaW5lcg0KDQpOZXRBcHANClRyb25kLk15a2xlYnVzdEBuZXRhcHAuY29tDQp3d3cubmV0YXBw
LmNvbQ0K

Re: 3.7-rc1 NFSv3/sec=krb5 mkdir failure

[Chuck Lever]

Sent from my iPad

On Nov 15, 2012, at 9:26 PM, "Myklebust, Trond" <Trond.Myklebust@netapp.com> wrote:

> On Thu, 2012-11-15 at 21:03 -0500, Chuck Lever wrote:
>> On Oct 28, 2012, at 12:15 PM, J. Bruce Fields <bfields@fieldses.org> wrote:
>> 
>>> On Wed, Oct 24, 2012 at 04:40:59PM -0400, J. Bruce Fields wrote:
>>>> On Wed, Oct 24, 2012 at 08:34:37PM +0000, Myklebust, Trond wrote:
>>>>>> -----Original Message-----
>>>>>> From: Myklebust, Trond
>>>>>> Sent: Wednesday, October 24, 2012 4:31 PM
>>>>>> To: 'J. Bruce Fields'
>>>>>> Cc: linux-nfs@vger.kernel.org; Schumaker, Bryan
>>>>>> Subject: RE: 3.7-rc1 NFSv3/sec=krb5 mkdir failure
>>>>>> 
>>>>>>> -----Original Message-----
>>>>>>> From: J. Bruce Fields [mailto:bfields@fieldses.org]
>>>>>>> Sent: Wednesday, October 24, 2012 4:15 PM
>>>>>>> To: Myklebust, Trond
>>>>>>> Cc: linux-nfs@vger.kernel.org; Schumaker, Bryan
>>>>>>> Subject: Re: 3.7-rc1 NFSv3/sec=krb5 mkdir failure
>>>>>>> 
>>>>>>> On Wed, Oct 24, 2012 at 08:07:55PM +0000, Myklebust, Trond wrote:
>>>>>>>>> -----Original Message-----
>>>>>>>>> From: linux-nfs-owner@vger.kernel.org [mailto:linux-nfs-
>>>>>>>>> owner@vger.kernel.org] On Behalf Of J. Bruce Fields
>>>>>>>>> Sent: Wednesday, October 24, 2012 4:03 PM
>>>>>>>>> To: linux-nfs@vger.kernel.org; Myklebust, Trond; Schumaker, Bryan
>>>>>>>>> Subject: Re: 3.7-rc1 NFSv3/sec=krb5 mkdir failure
>>>>>>>>> 
>>>>>>>>> Anyone get a chance to look at this?  It seems very reproduceable.
>>>>>>>>> 
>>>>>>>>> --b.
>>>>>>>>> 
>>>>>>>>> On Tue, Oct 16, 2012 at 08:58:32AM -0400, bfields wrote:
>>>>>>>>>> On 3.7-rc1:
>>>>>>>>>> 
>>>>>>>>>>    client# mount -tnfs -osec=krb5,vers=3 server:/exports/ext4
>>>>>>>>>> /mnt/
>>>>>>>>>> 
>>>>>>>>>>        server# ls -l /exports/ext4|grep TMP
>>>>>>>>>>        server#
>>>>>>>>>> 
>>>>>>>>>>    # mkdir /mnt/TMP
>>>>>>>>>>    mkdir: cannot create directory `/mnt/TMP': Permission denied
>>>>>>>>>> 
>>>>>>>>>>        server# ls -l /exports/ext4|grep TMP
>>>>>>>>>>        drwxr-xr-x  2 nfsnobody nfsnobody 4096 Oct 16 08:56 TMP
>>>>>>>>>>        server#
>>>>>>>>>> 
>>>>>>>>>> Wireshark also shows that the create succeeds.
>>>>>>>> 
>>>>>>>> Can you share the wireshark trace?
>>>>>>> 
>>>>>>> Sure.  This covers the mount and mkdir.  The mkdir call and reply are
>>>>>>> in frames 77 and 78.
>>>>>> 
>>>>>> Hmm.... Can you please check if the ACL is being set correctly on the server? I
>>>>>> suspect that might be the source of the error.
>>>>>> 
>>>>> 
>>>>> In fact, can you see if mounting with '-onoacl' causes the whole thing to succeed?
>>>> 
>>>> That's on the client mount command?  No difference.
>>> 
>>> By the way, I managed to do a little bisecting while working on
>>> something else today, and blame landed on Chuck's
>>> ba9b584c1dc37851d9c6ca6d0d2ccba55d9aad04 "SUNRPC: Introduce
>>> rpc_clone_client_set_auth()".  Which makes some sense if it's an ACL
>>> problem, and indeed testing on that commit finds success with -noacl,
>>> failure without.
>> 
>> After two weeks, Bruce and I were finally able to catch up in person.
>> 
>> I've reproduced this on 3.7-rc5 using cthon basic tests.  The first getacl operation fails because it's mistakenly attempting to set up a fresh GSS context on a transport where one already exists.  That's in line with the kind of change that's in commit ba9b584c1.
> 
> Why shouldn't we be able to cope with multiple GSS sessions on the same
> transport?

Perhaps we should be able to, in general.  But the successful case here does not attempt to create a new context, it simply uses one that is already associated with the transport.  That indicates that the kernel is making an incorrect upcall request perhaps because the new code is not cloning the RPC client correctly.

> 
>>> I'm not sure if that explains the failure I was seeing on 3.7-rc1, since
>>> there I didn't see any ACL traffic, and still got a failure.  (And
>>> -noacl didn't help.)
>> 
>> The failure occurs on the client just before the getacl request is issued, so you won't see any ACL-related network traffic in the failure case.  The failure prevents any ACL request from succeeding.
> 
> Is it gssd that is failing then?

The upcall fails, yes.  That is translated into an immediate failure of the getacl operation.