From: Linus Torvalds <torvalds@linux-foundation.org>
To: Patrick McLean <chutzpah@gentoo.org>,
Al Viro <viro@zeniv.linux.org.uk>,
Bruce Fields <bfields@redhat.com>,
"Darrick J. Wong" <darrick.wong@oracle.com>
Cc: Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
Linux NFS Mailing List <linux-nfs@vger.kernel.org>,
stable <stable@vger.kernel.org>,
Thorsten Leemhuis <regressions@leemhuis.info>
Subject: Re: [nfsd4] potentially hardware breaking regression in 4.14-rc and 4.13.11
Date: Wed, 8 Nov 2017 18:40:22 -0800 [thread overview]
Message-ID: <CA+55aFzGDyeJctD5Y3paBnysWXbA0cMF1_7mvvzG3n2OAnNhHw@mail.gmail.com> (raw)
In-Reply-To: <a17842c3-aae7-da98-424e-4441dd727e6d@gentoo.org>
On Wed, Nov 8, 2017 at 4:43 PM, Patrick McLean <chutzpah@gentoo.org> wrote:
> As of 4.13.11 (and also with 4.14-rc) we have an issue where when
> serving nfs4 sometimes we get the following BUG. When this bug happens,
> it usually also causes the motherboard to no longer POST until we
> externally re-flash the BIOS (using the BMC web interface). If a
> motherboard does not have an external way to flash the BIOS, this would
> brick the hardware.
That sounds like your BIOS is just broken.
The kernel oops is probably just a trigger for that - possibly because
you reboot with a particular state that breaks the BIOS.
Also, are you sure you really need to reflash the BIOS? It's actually
fairly hard to overwrite the BIOS itself, but crashing with bad
hardware state (where "bad" can just mean "unexpected by the BIOS")
can cause the BIOS to not properly re-initialize things, and hang at
boot.
So not booting cleanly from a warm reset is a reasonably common BIOS failure.
And yes, reflashing tends to force a full initialization and thus
"fixes" things, but it may be a big hammer when a cold boot or just a
"reset BIOS to safe defaults" might be sufficient.
In pretty much all cases this is a sign of a nasty BIOS problem,
though, and you may want to look into a firmware update from the
vendor for that.
But on to the kernel side:
> Here is the BUG we are getting:
>> [ 58.962528] BUG: unable to handle kernel NULL pointer dereference at 0000000000000230
>> [ 58.963918] IP: vfs_statfs+0x73/0xb0
The code disassembles to
0: 83 c9 08 or $0x8,%ecx
3: 40 f6 c6 04 test $0x4,%sil
7: 0f 45 d1 cmovne %ecx,%edx
a: 89 d1 mov %edx,%ecx
c: 80 cd 04 or $0x4,%ch
f: 40 f6 c6 08 test $0x8,%sil
13: 0f 45 d1 cmovne %ecx,%edx
16: 89 d1 mov %edx,%ecx
18: 80 cd 08 or $0x8,%ch
1b: 40 f6 c6 10 test $0x10,%sil
1f: 0f 45 d1 cmovne %ecx,%edx
22: 89 d1 mov %edx,%ecx
24: 80 cd 10 or $0x10,%ch
27: 83 e6 20 and $0x20,%esi
2a:* 48 8b b7 30 02 00 00 mov 0x230(%rdi),%rsi <-- trapping instruction
31: 0f 45 d1 cmovne %ecx,%edx
34: 83 ca 20 or $0x20,%edx
37: 89 f1 mov %esi,%ecx
39: 83 e1 10 and $0x10,%ecx
3c: 89 cf mov %ecx,%edi
and all those odd cmovne and bit-ops are just the bit selection code
in flags_by_mnt(), which is inlined through calculate_f_flags (which
is _also_ inlined) into vfs_statfs().
Sadly, gcc makes a mess of it and actually generates code that looks
like the original C. I would have hoped that gcc could have turned
if (x & BIT)
y |= OTHER_BIT;
into
y |= (x & BIT) shifted-by-the-bit-difference-between BIT/OTHER_BIT;
but that doesn't happen. We actually do it by hand in some other more
critical places, but it's painful to do by hand (because the shift
direction/amount is not trivial to do in C).
Anyway, that cmovne noise makes it a bit hard to see the actual part
that matters (and that traps) but I'm almost certain that it's the
"mnt->mnt_sb->s_flags" loading that is part of calculate_f_flags()
when it then does
flags_by_sb(mnt->mnt_sb->s_flags);
and I think mnt->mnt_sb is NULL. We know it's not 'mnt' itself that is
NULL, because we wouldn't have gotten this far if it was.
Now, afaik, mnt->mnt_sb should never be NULL in the first place for a
proper path. And the vfs_statfs() code itself hasn't changed in a
while.
Which does seem to implicate nfsd as having passed in a bad path to
vfs_statfs(). But I'm not seeing any changes in nfsd either.
In particular, there are *no* nfsd changes in that 4.13.8..4.13.11
range. There is a bunch of xfs changes, though. What's the underlying
filesystem that you are exporting?
But bringing in Al Viro and Bruce Fields explicitly in case they see
something. And Darrick, just in case it might be xfs.
Linus
next prev parent reply other threads:[~2017-11-09 2:40 UTC|newest]
Thread overview: 82+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-11-09 0:43 [nfsd4] potentially hardware breaking regression in 4.14-rc and 4.13.11 Patrick McLean
2017-11-09 2:40 ` Linus Torvalds [this message]
2017-11-09 3:45 ` Al Viro
2017-11-09 19:34 ` Patrick McLean
2017-11-09 19:38 ` Al Viro
2017-11-09 19:42 ` Patrick McLean
2017-11-09 19:37 ` Al Viro
2017-11-09 19:51 ` Patrick McLean
2017-11-09 20:04 ` Linus Torvalds
2017-11-09 21:16 ` Al Viro
2017-11-10 1:58 ` Patrick McLean
2017-11-10 13:53 ` Arnd Bergmann
2017-11-10 18:42 ` Linus Torvalds
2017-11-10 23:26 ` Patrick McLean
2017-11-11 0:27 ` Patrick McLean
2017-11-11 2:36 ` Linus Torvalds
2017-11-11 2:36 ` [kernel-hardening] " Linus Torvalds
2017-11-11 2:36 ` Linus Torvalds
2017-11-11 16:13 ` Kees Cook
2017-11-11 16:13 ` [kernel-hardening] " Kees Cook
2017-11-11 16:13 ` Kees Cook
2017-11-11 17:31 ` Linus Torvalds
2017-11-11 17:31 ` [kernel-hardening] " Linus Torvalds
2017-11-11 17:31 ` Linus Torvalds
2017-11-13 22:48 ` Patrick McLean
2017-11-13 22:48 ` [kernel-hardening] " Patrick McLean
2017-11-13 22:48 ` Patrick McLean
2017-11-17 0:54 ` Kees Cook
2017-11-17 0:54 ` [kernel-hardening] " Kees Cook
2017-11-17 0:54 ` Kees Cook
2017-11-17 19:03 ` Patrick McLean
2017-11-17 19:03 ` [kernel-hardening] " Patrick McLean
2017-11-17 19:03 ` Patrick McLean
2017-11-17 21:26 ` Kees Cook
2017-11-17 21:26 ` [kernel-hardening] " Kees Cook
2017-11-17 21:26 ` Kees Cook
2017-11-18 0:27 ` Patrick McLean
2017-11-18 0:27 ` [kernel-hardening] " Patrick McLean
2017-11-18 0:27 ` Patrick McLean
2017-11-18 0:55 ` Linus Torvalds
2017-11-18 0:55 ` [kernel-hardening] " Linus Torvalds
2017-11-18 0:55 ` Linus Torvalds
2017-11-18 1:54 ` Patrick McLean
2017-11-18 1:54 ` [kernel-hardening] " Patrick McLean
2017-11-18 1:54 ` Patrick McLean
2017-11-18 5:14 ` Kees Cook
2017-11-18 5:14 ` [kernel-hardening] " Kees Cook
2017-11-18 5:14 ` Kees Cook
2017-11-18 5:29 ` Linus Torvalds
2017-11-18 5:29 ` [kernel-hardening] " Linus Torvalds
2017-11-18 5:29 ` Linus Torvalds
2017-11-18 8:20 ` Kees Cook
2017-11-18 8:20 ` [kernel-hardening] " Kees Cook
2017-11-18 8:20 ` Kees Cook
2018-02-21 22:19 ` RANDSTRUCT structs need linux/compiler_types.h (Was: [nfsd4] potentially hardware breaking regression in 4.14-rc and 4.13.11) Maciej S. Szmigiero
2018-02-21 22:47 ` Linus Torvalds
2018-02-21 22:47 ` Linus Torvalds
2018-02-21 23:34 ` Kees Cook
2018-02-21 23:34 ` Kees Cook
2018-03-05 9:27 ` Masahiro Yamada
2018-03-05 9:27 ` Masahiro Yamada
2018-03-05 19:15 ` Kees Cook
2018-03-05 19:18 ` Linus Torvalds
2018-02-21 22:52 ` Kees Cook
2018-02-21 23:24 ` Linus Torvalds
2018-02-22 0:12 ` Kees Cook
2018-02-22 0:22 ` Linus Torvalds
2018-02-22 0:23 ` Kees Cook
2018-02-22 0:27 ` Kees Cook
2017-11-11 1:13 ` [nfsd4] potentially hardware breaking regression in 4.14-rc and 4.13.11 J. Bruce Fields
2017-11-11 2:32 ` Al Viro
2017-11-10 1:47 ` Patrick McLean
2017-11-09 20:47 ` J. Bruce Fields
2017-11-09 23:07 ` Patrick McLean
2017-11-13 22:59 ` bit tweaks [was: Re: [nfsd4] potentially hardware breaking regression in 4.14-rc and 4.13.11] Rasmus Villemoes
2017-11-13 23:30 ` Linus Torvalds
2017-11-13 23:54 ` Linus Torvalds
2017-11-14 22:24 ` Rasmus Villemoes
2017-11-14 22:43 ` Linus Torvalds
2017-11-14 23:53 ` Rasmus Villemoes
2017-11-15 0:02 ` Linus Torvalds
2017-11-11 2:47 ` [nfsd4] potentially hardware breaking regression in 4.14-rc and 4.13.11 Alan Cox
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CA+55aFzGDyeJctD5Y3paBnysWXbA0cMF1_7mvvzG3n2OAnNhHw@mail.gmail.com \
--to=torvalds@linux-foundation.org \
--cc=bfields@redhat.com \
--cc=chutzpah@gentoo.org \
--cc=darrick.wong@oracle.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-nfs@vger.kernel.org \
--cc=regressions@leemhuis.info \
--cc=stable@vger.kernel.org \
--cc=viro@zeniv.linux.org.uk \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.