How to debug permission denied issues?

How to debug permission denied issues?

[Paul Menzel]

Dear NFS folks,


We experience the following behavior with the Linux series 4.4 and 4.9.

When the exported directory on the server gets full to 100 % some NFS 
clients are unable to access the server anymore, but others can.

The error is, that the server says the client ID has expired. The 
server, on which the client runs on, needs to be restarted. Then it 
works again.

Can you think of an error path causing such a failure? What would be the 
right way to deal with this error?


Kind regards,

Paul

Full volume causes some NFS clients to get client ID expired responses

[Paul Menzel]

[Sorry, wrong subject.]

Dear NFS folks,


We experience the following behavior with the Linux series 4.4 and 4.9.

When the exported directory on the server gets full to 100 % some NFS
clients are unable to access the server anymore, but others can.

The error is, that the server says the client ID has expired. The
server, on which the client runs on, needs to be restarted. Then it
works again.

Can you think of an error path causing such a failure? What would be the
right way to deal with this error?


Kind regards,

Paul

Re: Full volume causes some NFS clients to get client ID expired responses

[J. Bruce Fields]

On Wed, Jun 21, 2017 at 01:48:05PM +0200, Paul Menzel wrote:
> We experience the following behavior with the Linux series 4.4 and 4.9.

So, your NFS server is a Linux server running kernel 4.4 or 4.9?  What
is the exported filesystem?

What are the clients?

> When the exported directory on the server gets full to 100 % some NFS
> clients are unable to access the server anymore, but others can.
> 
> The error is, that the server says the client ID has expired.

Where exactly are you seeing that error, in response to what operation?

Might be interesting to try a more recent kernel.

--b.

Re: How to debug permission denied issues?

[Paul Menzel]

Dear NFS folks,


On 05/16/17 13:06, Paul Menzel wrote:

> With Linux 4.9.x there are issues, that the NFSD servers denies access 
> to NFS mounts. Other clients can still connect fine. A restart of the 
> *client* fixes the issue.
> 
> We believe that this is related to a restart of a NIS server, and that 
> NFSD is now unable to authenticate the requests. Maybe some cache 
> mismatch of some client identifications?
> 
> ```
> root:sigchld:~/# mount -v -t nfs hopp:/home/joey /mnt
> mount.nfs: timeout set for Tue May 16 12:57:04 2017
> mount.nfs: trying text-based options 
> 'vers=4,addr=141.14.25.186,clientaddr=141.14.16.120'
> mount.nfs: mount(2): Permission denied
> mount.nfs: access denied by server while mounting hopp:/home/joey
> ```
> 
> Could you please point me to the right direction, how this issue can be 
> debugged?
> 
> A first step would be to get debug messages from NFSD, why exactly the 
> permission was denied to mount the directory.

It turns out, this particular issue was indeed a NIS issue, that the 
client wasn’t in the netgroup anymore. So, I am sorry for the noise.

With Wireshark we were able to find the NFSD response `Bad credentials 
(seal broken)`, leading us in the right direction.

But my question remains, is there another way besides Tcpdump and/or 
Wireshark to analyze these things. Turning on some debugging logs?


Kind regards,

Paul

How to debug permission denied issues?

[Paul Menzel]

Dear NFS folks,


With Linux 4.9.x there are issues, that the NFSD servers denies access 
to NFS mounts. Other clients can still connect fine. A restart of the 
*client* fixes the issue.

We believe that this is related to a restart of a NIS server, and that 
NFSD is now unable to authenticate the requests. Maybe some cache 
mismatch of some client identifications?

```
root:sigchld:~/# mount -v -t nfs hopp:/home/joey /mnt
mount.nfs: timeout set for Tue May 16 12:57:04 2017
mount.nfs: trying text-based options 
'vers=4,addr=141.14.25.186,clientaddr=141.14.16.120'
mount.nfs: mount(2): Permission denied
mount.nfs: access denied by server while mounting hopp:/home/joey
```

Could you please point me to the right direction, how this issue can be 
debugged?

A first step would be to get debug messages from NFSD, why exactly the 
permission was denied to mount the directory.


Kind regards,

Paul