nfs-utils: v3 mounts broken due to statx() returning EINVAL

nfs-utils: v3 mounts broken due to statx() returning EINVAL

[Thomas Deutschmann]

[-- Attachment #1.1: Type: text/plain, Size: 933 bytes --]

Hi,

we have some user reporting that NFS v3 mounts are broken
when using glibc-2.29 and linux-4.9.x (4.9.128) because
statx() with mask=STATX_BASIC_STATS returns EINVAL. 

Looks like this isn't happening with <nfs-utils-2.4.1 or
newer kernels.

The following workaround was confirmed to be working:

--- a/support/misc/xstat.c	2019-06-24 21:31:55.260371592 +0200
+++ b/support/misc/xstat.c	2019-06-24 21:32:29.098777436 +0200
@@ -47,6 +47,8 @@
 			statx_copy(statbuf, &stxbuf);
 			return 0;
 		}
+		if (errno == EINVAL)
+			errno = ENOSYS;
 		if (errno == ENOSYS)
 			statx_supported = 0;
 	} else


Bug: https://bugs.gentoo.org/688644

At the moment I have no clue whether this is kernel/glibc or
nfs-utils related; if the patch is safe to apply...

Any help is appreciated. Thanks!


-- 
Regards,
Thomas Deutschmann / Gentoo Linux Developer
C4DD 695F A713 8F24 2AA1 5638 5849 7EE5 1D5D 74A5


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

Re: nfs-utils: v3 mounts broken due to statx() returning EINVAL

[J. Bruce Fields]

On Tue, Oct 08, 2019 at 10:23:56PM +0200, Thomas Deutschmann wrote:
> Hi,
> 
> we have some user reporting that NFS v3 mounts are broken
> when using glibc-2.29 and linux-4.9.x (4.9.128) because
> statx() with mask=STATX_BASIC_STATS returns EINVAL. 
> 
> Looks like this isn't happening with <nfs-utils-2.4.1 or
> newer kernels.
> 
> The following workaround was confirmed to be working:
> 
> --- a/support/misc/xstat.c	2019-06-24 21:31:55.260371592 +0200
> +++ b/support/misc/xstat.c	2019-06-24 21:32:29.098777436 +0200
> @@ -47,6 +47,8 @@
>  			statx_copy(statbuf, &stxbuf);
>  			return 0;
>  		}
> +		if (errno == EINVAL)
> +			errno = ENOSYS;
>  		if (errno == ENOSYS)
>  			statx_supported = 0;
>  	} else
> 
> 
> Bug: https://bugs.gentoo.org/688644
> 
> At the moment I have no clue whether this is kernel/glibc or
> nfs-utils related; if the patch is safe to apply...

Well, sounds like nfs-utils started using statx in 2.4.1.  And just the
fact that varying the kernel version makes it sound like there was a
kernel bug causing an EINVAL return in this case, and that bug got
fixed.

One way to confirm might be running mount under strace and looking for
that EINVAL return.

I might also try looking through the kernel logs for nfs or the stat
code to see if there's a mention of this (didn't see it on a quick
look.)  Or try bisecting kernel versions to find the one where this was
fixed.

--b.

> 
> Any help is appreciated. Thanks!
> 
> 
> -- 
> Regards,
> Thomas Deutschmann / Gentoo Linux Developer
> C4DD 695F A713 8F24 2AA1 5638 5849 7EE5 1D5D 74A5
> 

Re: nfs-utils: v3 mounts broken due to statx() returning EINVAL

[Doug Nazar]

On Fri, Oct 11, 2019 at 17:07:09PM +0000, Bruce Fields wrote:
 >On Tue, Oct 08, 2019 at 10:23:56PM +0200, Thomas Deutschmann wrote:
 >> Hi,
 >>
 >> we have some user reporting that NFS v3 mounts are broken
 >> when using glibc-2.29 and linux-4.9.x (4.9.128) because
 >> statx() with mask=STATX_BASIC_STATS returns EINVAL.
 >>
 >> Looks like this isn't happening with <nfs-utils-2.4.1 or
 >> newer kernels.
 >>
 >> The following workaround was confirmed to be working:
 >>
 >> --- a/support/misc/xstat.c    2019-06-24 21:31:55.260371592 +0200
 >> +++ b/support/misc/xstat.c    2019-06-24 21:32:29.098777436 +0200
 >> @@ -47,6 +47,8 @@
 >>              statx_copy(statbuf, &stxbuf);
 >>              return 0;
 >>          }
 >> +        if (errno == EINVAL)
 >> +            errno = ENOSYS;
 >>          if (errno == ENOSYS)
 >>              statx_supported = 0;
 >>      } else
 >>
 >>
 >> Bug: https://bugs.gentoo.org/688644
 >>
 >> At the moment I have no clue whether this is kernel/glibc or
 >> nfs-utils related; if the patch is safe to apply...
 >
 >Well, sounds like nfs-utils started using statx in 2.4.1.  And just the
 >fact that varying the kernel version makes it sound like there was a
 >kernel bug causing an EINVAL return in this case, and that bug got
 >fixed.
 >
 >One way to confirm might be running mount under strace and looking for
 >that EINVAL return.

Just to provide an update on this issue, it was tracked down to glibc's 
statx emulation not supporting AT_STATX_DONT_SYNC and returning EINVAL.

So on a kernel without statx support, but a new glibc with statx 
support, nfs-utils will always fail to stat any paths.

Either this or a similar fix is required to support older kernels.

Doug

Re: nfs-utils: v3 mounts broken due to statx() returning EINVAL

[J. Bruce Fields]

On Tue, Nov 26, 2019 at 02:22:40AM -0500, Doug Nazar wrote:
> On Fri, Oct 11, 2019 at 17:07:09PM +0000, Bruce Fields wrote:
> >On Tue, Oct 08, 2019 at 10:23:56PM +0200, Thomas Deutschmann wrote:
> >> Hi,
> >>
> >> we have some user reporting that NFS v3 mounts are broken
> >> when using glibc-2.29 and linux-4.9.x (4.9.128) because
> >> statx() with mask=STATX_BASIC_STATS returns EINVAL.
> >>
> >> Looks like this isn't happening with <nfs-utils-2.4.1 or
> >> newer kernels.
> >>
> >> The following workaround was confirmed to be working:
> >>
> >> --- a/support/misc/xstat.c    2019-06-24 21:31:55.260371592 +0200
> >> +++ b/support/misc/xstat.c    2019-06-24 21:32:29.098777436 +0200
> >> @@ -47,6 +47,8 @@
> >>              statx_copy(statbuf, &stxbuf);
> >>              return 0;
> >>          }
> >> +        if (errno == EINVAL)
> >> +            errno = ENOSYS;
> >>          if (errno == ENOSYS)
> >>              statx_supported = 0;
> >>      } else
> >>
> >>
> >> Bug: https://bugs.gentoo.org/688644
> >>
> >> At the moment I have no clue whether this is kernel/glibc or
> >> nfs-utils related; if the patch is safe to apply...
> >
> >Well, sounds like nfs-utils started using statx in 2.4.1.  And just the
> >fact that varying the kernel version makes it sound like there was a
> >kernel bug causing an EINVAL return in this case, and that bug got
> >fixed.
> >
> >One way to confirm might be running mount under strace and looking for
> >that EINVAL return.
> 
> Just to provide an update on this issue, it was tracked down to glibc's
> statx emulation not supporting AT_STATX_DONT_SYNC and returning EINVAL.
> 
> So on a kernel without statx support, but a new glibc with statx support,
> nfs-utils will always fail to stat any paths.
> 
> Either this or a similar fix is required to support older kernels.

OK.  So maybe your patch is the best we can do.  I'd just add a comment
there documenting the situation.

--b.