linux-nfs.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* Need help debugging NFS issues new to 4.20 kernel
@ 2019-01-24 17:32 Jason L Tibbitts III
  2019-01-24 19:28 ` Jason L Tibbitts III
  2019-01-24 19:58 ` Trond Myklebust
  0 siblings, 2 replies; 25+ messages in thread
From: Jason L Tibbitts III @ 2019-01-24 17:32 UTC (permalink / raw)
  To: linux-nfs

I could use some help figuring out the cause of some serious NFS client
issues I'm having with the 4.20.3 kernel which I did not see under
4.19.15.

I have a network of about 130 desktops (plus a bunch of other machines,
VMs and the like) running Fedora 29 connecting to six NFS servers
running CentOS 7.6 (with the heavily patched vendor kernel
3.10.0-957.1.3).  All machines involved are x86_64.  We use kerberized
NFS4 with generally sec=krb5i.  The exports are generally made with
"(rw,async,sec=krb5i:krb5p)".

Since I booted those clients into 4.20.3 I've started seeing processes
getting stuck in the D state.  The system itself will seem OK (except
for the high load average) as long as I don't touch the hung NFS mount.
Nothing was logged to dmesg or to the journal.  So far booting back into
the 4.19.15 kernel has cleared up the problem.  I cannot yet reproduce
this on demand; I've tried but it is probably related to some specific
usage pattern.

Has anyone else seen issues like this?  Can anyone help me to get more
useful information that might point to the problem?  I still haven't
learned how to debug NFS issues properly.  And if there's a stress test
tool I could easily run that might help to reproduce the issue, I'd be
happy to run it.

I note that 4.20.4 is out; I see one sunrpc fix which I guess could be
related (sunrpc: handle ENOMEM in rpcb_getport_async) but the systems
involved have plenty of free memory so I doubt that's it.  I'll
certainly try it anyway.

Various package versions:
kernel-4.20.3-200.fc29.x86_64 (the problematic kernel)
kernel-4.19.15-300.fc29.x86_64 (the functional kernel)
nfs-utils-2.3.3-1.rc2.fc29.x86_64
gssproxy-0.8.0-6.fc29.x86_64
krb5-libs-1.16.1-25.fc29.i686

Thanks in advance for any help or advice,

 - J<

^ permalink raw reply	[flat|nested] 25+ messages in thread

end of thread, other threads:[~2019-02-25 23:15 UTC | newest]

Thread overview: 25+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-01-24 17:32 Need help debugging NFS issues new to 4.20 kernel Jason L Tibbitts III
2019-01-24 19:28 ` Jason L Tibbitts III
2019-01-24 19:58 ` Trond Myklebust
2019-01-25 19:13   ` Schumaker, Anna
2019-01-26 17:59     ` Sasha Levin
2019-01-25 19:51   ` Jason L Tibbitts III
2019-02-05 18:12     ` Jason Tibbitts
2019-02-06 12:05       ` Benjamin Coddington
     [not found]         ` <87imxwab12.fsf@hippogriff.math.uh.edu>
2019-02-07 11:13           ` Benjamin Coddington
     [not found]             ` <87d0o3aadg.fsf@hippogriff.math.uh.edu>
2019-02-08 12:01               ` Benjamin Coddington
2019-02-08 15:19                 ` Chuck Lever
2019-02-08 17:17                   ` Jason L Tibbitts III
2019-02-15 20:33                 ` Jason L Tibbitts III
2019-02-16 14:46                   ` Trond Myklebust
2019-02-20  2:13                     ` Jason L Tibbitts III
2019-02-20 15:25                     ` Jason L Tibbitts III
2019-02-20 15:37                       ` Trond Myklebust
2019-02-20 15:39                         ` Chuck Lever
2019-02-20 15:41                         ` Trond Myklebust
2019-02-21 18:19                           ` Jason L Tibbitts III
2019-02-25 19:24                             ` Jason L Tibbitts III
2019-02-25 23:15                               ` Benjamin Coddington
2019-02-20 16:25                         ` Jason L Tibbitts III
2019-02-20 16:45                           ` Trond Myklebust
2019-02-20 16:49                             ` Jason L Tibbitts III

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).