All of lore.kernel.org
 help / color / mirror / Atom feed
From: Olga Kornievskaia <aglo@umich.edu>
To: Michael Wakabayashi <mwakabayashi@vmware.com>
Cc: "linux-nfs@vger.kernel.org" <linux-nfs@vger.kernel.org>
Subject: Re: NFSv4: Mounting NFS server which is down, blocks all other NFS mounts on same machine
Date: Wed, 19 May 2021 15:15:12 -0400	[thread overview]
Message-ID: <CAN-5tyGgWx6F2s=t+0UAJJZEAEfNnv+Sq8eeBbnQYocKOOK8Jg@mail.gmail.com> (raw)
In-Reply-To: <CO1PR05MB810163D3FD8705AFF519F0B4B72D9@CO1PR05MB8101.namprd05.prod.outlook.com>

On Sun, May 16, 2021 at 11:18 PM Michael Wakabayashi
<mwakabayashi@vmware.com> wrote:
>
> Hi,
>
> We're seeing what looks like an NFSv4 issue.
>
> Mounting an NFS server that is down (ping to this NFS server's IP address does not respond) will block _all_ other NFS mount attempts even if the NFS servers are available and working properly (these subsequent mounts hang).
>
> If I kill the NFS mount process that's trying to mount the dead NFS server, the NFS mounts that were blocked will immediately unblock and mount successfully, which suggests the first mount command is blocking the other mount commands.
>
>
> I verified this behavior using a newly built mount.nfs command from the recent nfs-utils 2.5.3 package installed on a recent version of Ubuntu Cloud Image 21.04:
> * https://sourceforge.net/projects/nfs/files/nfs-utils/2.5.3/
> * https://cloud-images.ubuntu.com/releases/hirsute/release-20210513/ubuntu-21.04-server-cloudimg-amd64.ova
>
>
> The reason this looks like it is specific to NFSv4 is from the following output showing "vers=4.2":
> > $ strace /sbin/mount.nfs <unreachable-IP-address>:/path /tmp/mnt
> > [ ... cut ... ]
> > mount("<unreadhable-IP-address>:/path", "/tmp/mnt", "nfs", 0, "vers=4.2,addr=<unreachable-IP-address>,clien"...^C^Z
>
> Also, if I try the same mount.nfs commands but specifying NFSv3, the mount to the dead NFS server hangs, but the mounts to the operational NFS servers do not block and mount successfully; this bug doesn't happen when using NFSv3.
>
>
> We reported this issue under util-linux here:
> https://github.com/karelzak/util-linux/issues/1309
> [mounting nfs server which is down blocks all other nfs mounts on same machine #1309]
>
> I also found an older bug on this mailing list that had similar symptoms (but could not tell if it was the same problem or not):
> https://patchwork.kernel.org/project/linux-nfs/patch/87vaori26c.fsf@notabene.neil.brown.name/
> [[PATCH/RFC] NFSv4: don't let hanging mounts block other mounts]
>
> Thanks, Mike

Hi Mike,

This is not a helpful reply but I was curious if I could reproduce
your issue but was not successful. I'm able to initiate a mount to an
unreachable-IP-address which hangs and then do another mount to an
existing server without issues. Ubuntu 21.04 seems to be 5.11 based so
I tried upstream 5.11 and I tried the latest upstream nfs-utils
(instead of what my distro has which was an older version).

To debug, perhaps get an output of the nfs4 and sunrpc tracepoints.
Or also get output from dmesg after doing “echo t >
/proc/sysrq-trigger” to see where the mounts are hanging.

  reply	other threads:[~2021-05-19 19:15 UTC|newest]

Thread overview: 26+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-05-17  1:37 NFSv4: Mounting NFS server which is down, blocks all other NFS mounts on same machine Michael Wakabayashi
2021-05-19 19:15 ` Olga Kornievskaia [this message]
2021-05-20  9:51   ` Michael Wakabayashi
2021-05-20 10:43     ` Michael Wakabayashi
2021-05-20 23:51       ` Olga Kornievskaia
2021-05-21 19:11         ` Michael Wakabayashi
2021-05-20 18:42   ` Steve Dickson
     [not found]     ` <CO1PR05MB8101FD5E77B386A75786FF41B7299@CO1PR05MB8101.namprd05.prod.outlook.com>
2021-05-21 19:35       ` Olga Kornievskaia
2021-05-21 20:31         ` Michael Wakabayashi
2021-05-21 21:06           ` Olga Kornievskaia
2021-05-21 22:08             ` Trond Myklebust
2021-05-21 22:41               ` Olga Kornievskaia
2021-06-08  9:16                 ` Michael Wakabayashi
2021-06-08 16:10                   ` Olga Kornievskaia
2021-06-09  5:31                     ` Michael Wakabayashi
2021-06-09 13:50                       ` Olga Kornievskaia
2021-06-09 20:19                         ` Alex Romanenko
2021-06-11  5:26                           ` Michael Wakabayashi
2021-06-09 14:31                       ` Benjamin Coddington
2021-06-09 14:41                         ` Olga Kornievskaia
2021-06-09 17:14                           ` Michael Wakabayashi
2021-06-09 14:41                         ` Trond Myklebust
2021-06-09 15:00                           ` Benjamin Coddington
2021-06-09 15:19                             ` Trond Myklebust
2021-06-09  6:46                     ` Alex Romanenko
2021-05-21 22:38             ` Olga Kornievskaia

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='CAN-5tyGgWx6F2s=t+0UAJJZEAEfNnv+Sq8eeBbnQYocKOOK8Jg@mail.gmail.com' \
    --to=aglo@umich.edu \
    --cc=linux-nfs@vger.kernel.org \
    --cc=mwakabayashi@vmware.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.