Use of /etc/netgroup appears to be broken in the NFS server version which ships with Ubuntu 20.04

Use of /etc/netgroup appears to be broken in the NFS server version which ships with Ubuntu 20.04

[Patrick Goetz]

Sadly, it took me a couple of days to track this down. The /etc/netgroup 
file I'm using works perfectly on another NFS server (Ubuntu 18.04) in 
production, so this wasn't an immediate suspicion.  However, if I use 
this /etc/exports:

   /srv/nfs @cryo_em(rw,sync,fsid=0,crossmnt,no_subtree_check)
   /srv/nfs/cryosparc @cryo_em(rw,sync,fsid=2,crossmnt,no_subtree_check)

Client mounts fail:


root@javelina:~# mount -vvvt nfs4 cerebro:/cryosparc /cryosparc
mount.nfs4: timeout set for Tue Jun 15 11:53:22 2021
mount.nfs4: trying text-based options 
'vers=4.2,addr=128.xx.xx.xxx,clientaddr=129.xxx.xxx.xx'
mount.nfs4: mount(2): Permission denied
mount.nfs4: access denied by server while mounting cerebro:/cryosparc

and if I switch to specifying the host explicitly:

   /srv/nfs javelina.my.domain(rw,sync,fsid=0,crossmnt,no_subtree_check)

   /srv/nfs/cryosparc 
javelina.mydomain(rw,sync,fsid=2,crossmnt,no_subtree_check)

the mount just works.  The tcpdump error message isn't terribly helpful 
here:

11:14:02.856094 IP cerebro.my.domain.nfs > javelina.my.domain.741: Flags 
[.], ack 281, win 507, options [nop,nop,TS val 791638255 ecr 
2576087678], length 0
11:14:02.856178 IP cerebro.my.domain.nfs > javelina.my.domain.741: Flags 
[P.], seq 1:25, ack 281, win 507, options [nop,nop,TS val 791638255 ecr 
2576087678], length 24: NFS reply xid 2752089303 reply ERR 20: Auth 
Bogus Credentials (seal broken)

but after figuring out the cause of the problem, I did find a 
corroborating RHEL error report (which you'll need a RHEL account to 
access):

   https://access.redhat.com/solutions/3563601

I couldn't figure out how to determine the exact version of the NFS 
server that ships with Ubuntu 20.04.  Maybe someone could explain how to 
do this.  Running
    /usr/sbin/rpc.nfsd --version
doesn't do it.

Re: Use of /etc/netgroup appears to be broken in the NFS server version which ships with Ubuntu 20.04

[NeilBrown]

On Wed, 16 Jun 2021, Patrick Goetz wrote:
> Sadly, it took me a couple of days to track this down. The /etc/netgroup 
> file I'm using works perfectly on another NFS server (Ubuntu 18.04) in 
> production, so this wasn't an immediate suspicion.  However, if I use 
> this /etc/exports:
> 
>    /srv/nfs @cryo_em(rw,sync,fsid=0,crossmnt,no_subtree_check)
>    /srv/nfs/cryosparc @cryo_em(rw,sync,fsid=2,crossmnt,no_subtree_check)
> 
> Client mounts fail:
> 
> 
> root@javelina:~# mount -vvvt nfs4 cerebro:/cryosparc /cryosparc
> mount.nfs4: timeout set for Tue Jun 15 11:53:22 2021
> mount.nfs4: trying text-based options 
> 'vers=4.2,addr=128.xx.xx.xxx,clientaddr=129.xxx.xxx.xx'
> mount.nfs4: mount(2): Permission denied
> mount.nfs4: access denied by server while mounting cerebro:/cryosparc
> 
> and if I switch to specifying the host explicitly:
> 
>    /srv/nfs javelina.my.domain(rw,sync,fsid=0,crossmnt,no_subtree_check)
> 
>    /srv/nfs/cryosparc 
> javelina.mydomain(rw,sync,fsid=2,crossmnt,no_subtree_check)
> 
> the mount just works.  The tcpdump error message isn't terribly helpful 
> here:
> 
> 11:14:02.856094 IP cerebro.my.domain.nfs > javelina.my.domain.741: Flags 
> [.], ack 281, win 507, options [nop,nop,TS val 791638255 ecr 
> 2576087678], length 0
> 11:14:02.856178 IP cerebro.my.domain.nfs > javelina.my.domain.741: Flags 
> [P.], seq 1:25, ack 281, win 507, options [nop,nop,TS val 791638255 ecr 
> 2576087678], length 24: NFS reply xid 2752089303 reply ERR 20: Auth 
> Bogus Credentials (seal broken)
> 
> but after figuring out the cause of the problem, I did find a 
> corroborating RHEL error report (which you'll need a RHEL account to 
> access):
> 
>    https://access.redhat.com/solutions/3563601
> 
> I couldn't figure out how to determine the exact version of the NFS 
> server that ships with Ubuntu 20.04.  Maybe someone could explain how to 
> do this.  Running
>     /usr/sbin/rpc.nfsd --version
> doesn't do it.
> 
> 

The problem is unlikely to be the implementation of netgroups - that
hasn't changed in a long time.  It is more likely to be some subtle
configuration difference.

Can you provide the verbatim /etc/netgroups file, and the extact host
name that a DNS lookup of the client IP adress results in?

NeilBrown

Re: Use of /etc/netgroup appears to be broken in the NFS server version which ships with Ubuntu 20.04

[Patrick Goetz]

Hi Neil -

This is extremely embarrassing, but chalk this one up to user error 
(technically, PEBKAC).  I'm used to /etc/nsswitch.conf always including 
the files option, so didn't think to check this when in fact on Ubuntu 
20.04 they ship the nsswitch.conf with this netgroup entry.

   netgroup:       nis

Who still uses NIS?  Beats me; but once I added files to the option list 
it starting working as advertised. Very sorry to dump random noise onto 
the list.


On 6/16/21 10:11 PM, NeilBrown wrote:
> On Wed, 16 Jun 2021, Patrick Goetz wrote:
>> Sadly, it took me a couple of days to track this down. The /etc/netgroup
>> file I'm using works perfectly on another NFS server (Ubuntu 18.04) in
>> production, so this wasn't an immediate suspicion.  However, if I use
>> this /etc/exports:
>>
>>     /srv/nfs @cryo_em(rw,sync,fsid=0,crossmnt,no_subtree_check)
>>     /srv/nfs/cryosparc @cryo_em(rw,sync,fsid=2,crossmnt,no_subtree_check)
>>
>> Client mounts fail:
>>
>>
>> root@javelina:~# mount -vvvt nfs4 cerebro:/cryosparc /cryosparc
>> mount.nfs4: timeout set for Tue Jun 15 11:53:22 2021
>> mount.nfs4: trying text-based options
>> 'vers=4.2,addr=128.xx.xx.xxx,clientaddr=129.xxx.xxx.xx'
>> mount.nfs4: mount(2): Permission denied
>> mount.nfs4: access denied by server while mounting cerebro:/cryosparc
>>
>> and if I switch to specifying the host explicitly:
>>
>>     /srv/nfs javelina.my.domain(rw,sync,fsid=0,crossmnt,no_subtree_check)
>>
>>     /srv/nfs/cryosparc
>> javelina.mydomain(rw,sync,fsid=2,crossmnt,no_subtree_check)
>>
>> the mount just works.  The tcpdump error message isn't terribly helpful
>> here:
>>
>> 11:14:02.856094 IP cerebro.my.domain.nfs > javelina.my.domain.741: Flags
>> [.], ack 281, win 507, options [nop,nop,TS val 791638255 ecr
>> 2576087678], length 0
>> 11:14:02.856178 IP cerebro.my.domain.nfs > javelina.my.domain.741: Flags
>> [P.], seq 1:25, ack 281, win 507, options [nop,nop,TS val 791638255 ecr
>> 2576087678], length 24: NFS reply xid 2752089303 reply ERR 20: Auth
>> Bogus Credentials (seal broken)
>>
>> but after figuring out the cause of the problem, I did find a
>> corroborating RHEL error report (which you'll need a RHEL account to
>> access):
>>
>>     https://access.redhat.com/solutions/3563601
>>
>> I couldn't figure out how to determine the exact version of the NFS
>> server that ships with Ubuntu 20.04.  Maybe someone could explain how to
>> do this.  Running
>>      /usr/sbin/rpc.nfsd --version
>> doesn't do it.
>>
>>
> 
> The problem is unlikely to be the implementation of netgroups - that
> hasn't changed in a long time.  It is more likely to be some subtle
> configuration difference.
> 
> Can you provide the verbatim /etc/netgroups file, and the extact host
> name that a DNS lookup of the client IP adress results in?
> 
> NeilBrown
> 

Re: Use of /etc/netgroup appears to be broken in the NFS server version which ships with Ubuntu 20.04

[NeilBrown]

On Fri, 18 Jun 2021, Patrick Goetz wrote:
> Hi Neil -
> 
> This is extremely embarrassing, but chalk this one up to user error 
> (technically, PEBKAC).  I'm used to /etc/nsswitch.conf always including 
> the files option, so didn't think to check this when in fact on Ubuntu 
> 20.04 they ship the nsswitch.conf with this netgroup entry.
> 
>    netgroup:       nis
> 
> Who still uses NIS?  Beats me; but once I added files to the option list 
> it starting working as advertised. Very sorry to dump random noise onto 
> the list.

:-)
NIS has a certain elegant simplicity.  LDAP is probably objective better
in every way ... except the elegant simplicity.

Glad to know I wasn't missing anything important, and that there was an
easy solution.

NeilBrown


> 
> 
> On 6/16/21 10:11 PM, NeilBrown wrote:
> > On Wed, 16 Jun 2021, Patrick Goetz wrote:
> >> Sadly, it took me a couple of days to track this down. The /etc/netgroup
> >> file I'm using works perfectly on another NFS server (Ubuntu 18.04) in
> >> production, so this wasn't an immediate suspicion.  However, if I use
> >> this /etc/exports:
> >>
> >>     /srv/nfs @cryo_em(rw,sync,fsid=0,crossmnt,no_subtree_check)
> >>     /srv/nfs/cryosparc @cryo_em(rw,sync,fsid=2,crossmnt,no_subtree_check)
> >>
> >> Client mounts fail:
> >>
> >>
> >> root@javelina:~# mount -vvvt nfs4 cerebro:/cryosparc /cryosparc
> >> mount.nfs4: timeout set for Tue Jun 15 11:53:22 2021
> >> mount.nfs4: trying text-based options
> >> 'vers=4.2,addr=128.xx.xx.xxx,clientaddr=129.xxx.xxx.xx'
> >> mount.nfs4: mount(2): Permission denied
> >> mount.nfs4: access denied by server while mounting cerebro:/cryosparc
> >>
> >> and if I switch to specifying the host explicitly:
> >>
> >>     /srv/nfs javelina.my.domain(rw,sync,fsid=0,crossmnt,no_subtree_check)
> >>
> >>     /srv/nfs/cryosparc
> >> javelina.mydomain(rw,sync,fsid=2,crossmnt,no_subtree_check)
> >>
> >> the mount just works.  The tcpdump error message isn't terribly helpful
> >> here:
> >>
> >> 11:14:02.856094 IP cerebro.my.domain.nfs > javelina.my.domain.741: Flags
> >> [.], ack 281, win 507, options [nop,nop,TS val 791638255 ecr
> >> 2576087678], length 0
> >> 11:14:02.856178 IP cerebro.my.domain.nfs > javelina.my.domain.741: Flags
> >> [P.], seq 1:25, ack 281, win 507, options [nop,nop,TS val 791638255 ecr
> >> 2576087678], length 24: NFS reply xid 2752089303 reply ERR 20: Auth
> >> Bogus Credentials (seal broken)
> >>
> >> but after figuring out the cause of the problem, I did find a
> >> corroborating RHEL error report (which you'll need a RHEL account to
> >> access):
> >>
> >>     https://access.redhat.com/solutions/3563601
> >>
> >> I couldn't figure out how to determine the exact version of the NFS
> >> server that ships with Ubuntu 20.04.  Maybe someone could explain how to
> >> do this.  Running
> >>      /usr/sbin/rpc.nfsd --version
> >> doesn't do it.
> >>
> >>
> > 
> > The problem is unlikely to be the implementation of netgroups - that
> > hasn't changed in a long time.  It is more likely to be some subtle
> > configuration difference.
> > 
> > Can you provide the verbatim /etc/netgroups file, and the extact host
> > name that a DNS lookup of the client IP adress results in?
> > 
> > NeilBrown
> > 
> 
> 

Re: Use of /etc/netgroup appears to be broken in the NFS server version which ships with Ubuntu 20.04

[Patrick Goetz]

On 6/17/21 4:24 PM, NeilBrown wrote:
> On Fri, 18 Jun 2021, Patrick Goetz wrote:
>> Hi Neil -
>>
>> This is extremely embarrassing, but chalk this one up to user error
>> (technically, PEBKAC).  I'm used to /etc/nsswitch.conf always including
>> the files option, so didn't think to check this when in fact on Ubuntu
>> 20.04 they ship the nsswitch.conf with this netgroup entry.
>>
>>     netgroup:       nis
>>
>> Who still uses NIS?  Beats me; but once I added files to the option list
>> it starting working as advertised. Very sorry to dump random noise onto
>> the list.
> 
> :-)
> NIS has a certain elegant simplicity.  LDAP is probably objective better
> in every way ... except the elegant simplicity.
> 

I agree.  I hate LDAP (so inelegant) and I find ASN.1 to be especially 
inelegant. (I've never seen anything to indicate this is anything other 
than an anachronism.)


> Glad to know I wasn't missing anything important, and that there was an
> easy solution.
> 
> NeilBrown
> 
> 
>>
>>
>> On 6/16/21 10:11 PM, NeilBrown wrote:
>>> On Wed, 16 Jun 2021, Patrick Goetz wrote:
>>>> Sadly, it took me a couple of days to track this down. The /etc/netgroup
>>>> file I'm using works perfectly on another NFS server (Ubuntu 18.04) in
>>>> production, so this wasn't an immediate suspicion.  However, if I use
>>>> this /etc/exports:
>>>>
>>>>      /srv/nfs @cryo_em(rw,sync,fsid=0,crossmnt,no_subtree_check)
>>>>      /srv/nfs/cryosparc @cryo_em(rw,sync,fsid=2,crossmnt,no_subtree_check)
>>>>
>>>> Client mounts fail:
>>>>
>>>>
>>>> root@javelina:~# mount -vvvt nfs4 cerebro:/cryosparc /cryosparc
>>>> mount.nfs4: timeout set for Tue Jun 15 11:53:22 2021
>>>> mount.nfs4: trying text-based options
>>>> 'vers=4.2,addr=128.xx.xx.xxx,clientaddr=129.xxx.xxx.xx'
>>>> mount.nfs4: mount(2): Permission denied
>>>> mount.nfs4: access denied by server while mounting cerebro:/cryosparc
>>>>
>>>> and if I switch to specifying the host explicitly:
>>>>
>>>>      /srv/nfs javelina.my.domain(rw,sync,fsid=0,crossmnt,no_subtree_check)
>>>>
>>>>      /srv/nfs/cryosparc
>>>> javelina.mydomain(rw,sync,fsid=2,crossmnt,no_subtree_check)
>>>>
>>>> the mount just works.  The tcpdump error message isn't terribly helpful
>>>> here:
>>>>
>>>> 11:14:02.856094 IP cerebro.my.domain.nfs > javelina.my.domain.741: Flags
>>>> [.], ack 281, win 507, options [nop,nop,TS val 791638255 ecr
>>>> 2576087678], length 0
>>>> 11:14:02.856178 IP cerebro.my.domain.nfs > javelina.my.domain.741: Flags
>>>> [P.], seq 1:25, ack 281, win 507, options [nop,nop,TS val 791638255 ecr
>>>> 2576087678], length 24: NFS reply xid 2752089303 reply ERR 20: Auth
>>>> Bogus Credentials (seal broken)
>>>>
>>>> but after figuring out the cause of the problem, I did find a
>>>> corroborating RHEL error report (which you'll need a RHEL account to
>>>> access):
>>>>
>>>>      https://access.redhat.com/solutions/3563601
>>>>
>>>> I couldn't figure out how to determine the exact version of the NFS
>>>> server that ships with Ubuntu 20.04.  Maybe someone could explain how to
>>>> do this.  Running
>>>>       /usr/sbin/rpc.nfsd --version
>>>> doesn't do it.
>>>>
>>>>
>>>
>>> The problem is unlikely to be the implementation of netgroups - that
>>> hasn't changed in a long time.  It is more likely to be some subtle
>>> configuration difference.
>>>
>>> Can you provide the verbatim /etc/netgroups file, and the extact host
>>> name that a DNS lookup of the client IP adress results in?
>>>
>>> NeilBrown
>>>
>>
>>