* Re: refcount_t: underflow; use-after-free with CIFS umount after scsi-misc commit ef2cc88e2a205b8a11a19e78db63a70d3728cdf5
2019-12-09 2:23 ` Linus Torvalds
@ 2019-12-09 2:52 ` Al Viro
2019-12-09 3:10 ` Linus Torvalds
2019-12-09 3:18 ` Steve French
2019-12-09 14:45 ` Arthur Marsh
2 siblings, 1 reply; 8+ messages in thread
From: Al Viro @ 2019-12-09 2:52 UTC (permalink / raw)
To: Linus Torvalds
Cc: Arthur Marsh, SCSI development list, Linux Kernel Mailing List,
CIFS, James E.J. Bottomley
On Sun, Dec 08, 2019 at 06:23:02PM -0800, Linus Torvalds wrote:
> On Sun, Dec 8, 2019 at 5:49 PM Arthur Marsh
> <arthur.marsh@internode.on.net> wrote:
> >
> > This still happens with 5.5.0-rc1:
>
> Does it happen 100% of the time?
>
> Your bisection result looks pretty nonsensical - not that it's
> impossible (anything is possible), but it really doesn't look very
> likely. Which makes me think maybe it's slightly timing-sensitive or
> something?
>
> Would you mind trying to re-do the bisection, and for each kernel try
> the mount thing at least a few times before you decide a kernel is
> good?
>
> Bisection is very powerful, but if _any_ of the kernels you marked
> good weren't really good (they just happened to not trigger the
> problem), bisection ends up giving completely the wrong answer. And
> with that bisection commit, there's not even a hint of what could have
> gone wrong.
FWIW, the thing that is IME absolutely incompatible with bisection
is CONFIG_GCC_PLUGIN_RANDSTRUCT. It can affect frequencies badly
enough, even in the cases when the bug isn't directly dependent
upon that thing.
I suspect that nonsense bisects spewed by CI bots lately (bisect on
x86 oops ending up at commit limited to arch/parisc, etc.) are at
least partially due to that kind of garbage...
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: refcount_t: underflow; use-after-free with CIFS umount after scsi-misc commit ef2cc88e2a205b8a11a19e78db63a70d3728cdf5
2019-12-09 2:52 ` Al Viro
@ 2019-12-09 3:10 ` Linus Torvalds
2019-12-09 3:15 ` Linus Torvalds
0 siblings, 1 reply; 8+ messages in thread
From: Linus Torvalds @ 2019-12-09 3:10 UTC (permalink / raw)
To: Al Viro
Cc: Arthur Marsh, SCSI development list, Linux Kernel Mailing List,
CIFS, James E.J. Bottomley
On Sun, Dec 8, 2019 at 6:52 PM Al Viro <viro@zeniv.linux.org.uk> wrote:
>
> FWIW, the thing that is IME absolutely incompatible with bisection
> is CONFIG_GCC_PLUGIN_RANDSTRUCT. It can affect frequencies badly
> enough, even in the cases when the bug isn't directly dependent
> upon that thing.
It will easily affect timing in major ways, yes.
You're right that at least the CI bots might want to disable it for
bisecting. Or force a particular seed for RANDSTRUCT - I seem to
recall that there was some way to make it be at least repeatable for
any particular structure.
Linus
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: refcount_t: underflow; use-after-free with CIFS umount after scsi-misc commit ef2cc88e2a205b8a11a19e78db63a70d3728cdf5
2019-12-09 3:10 ` Linus Torvalds
@ 2019-12-09 3:15 ` Linus Torvalds
0 siblings, 0 replies; 8+ messages in thread
From: Linus Torvalds @ 2019-12-09 3:15 UTC (permalink / raw)
To: Al Viro
Cc: Arthur Marsh, SCSI development list, Linux Kernel Mailing List,
CIFS, James E.J. Bottomley
On Sun, Dec 8, 2019 at 7:10 PM Linus Torvalds
<torvalds@linux-foundation.org> wrote:
>
> You're right that at least the CI bots might want to disable it for
> bisecting. Or force a particular seed for RANDSTRUCT - I seem to
> recall that there was some way to make it be at least repeatable for
> any particular structure.
Yeah, if you build in the same directory and don't do a "distclean" or
"git clean -dqfx", the seed remains in
scripts/gcc-plgins/randomize_layout_seed.h across builds, and the
result then _should_ be repeatable.
But yes, it's the kind of noise that is likely not worth fighting for
bisection (or any random bug hunting) at all, and turning off
RANDSTRUCT is probably the right thing to do.
But I don't think Arthur had RANDSTRUCT enabled, so that should be fine.
Linus
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: refcount_t: underflow; use-after-free with CIFS umount after scsi-misc commit ef2cc88e2a205b8a11a19e78db63a70d3728cdf5
2019-12-09 2:23 ` Linus Torvalds
2019-12-09 2:52 ` Al Viro
@ 2019-12-09 3:18 ` Steve French
2019-12-09 3:27 ` Steve French
2019-12-09 14:45 ` Arthur Marsh
2 siblings, 1 reply; 8+ messages in thread
From: Steve French @ 2019-12-09 3:18 UTC (permalink / raw)
To: Linus Torvalds
Cc: Arthur Marsh, SCSI development list, Linux Kernel Mailing List,
CIFS, James E.J. Bottomley, ronnie sahlberg
On Sun, Dec 8, 2019 at 8:23 PM Linus Torvalds
<torvalds@linux-foundation.org> wrote:
>
> On Sun, Dec 8, 2019 at 5:49 PM Arthur Marsh
> <arthur.marsh@internode.on.net> wrote:
> >
> > This still happens with 5.5.0-rc1:
>
> Does it happen 100% of the time?
I can reproduce it (although it was a little more difficult since WiFi doesn't
work on RC1 on some of my hardware - due to the 802.11 driver regression oops.
I was able to reproduce it to Samba localhost).
> Your bisection result looks pretty nonsensical - not that it's
> impossible (anything is possible), but it really doesn't look very
> likely. Which makes me think maybe it's slightly timing-sensitive or
> something?
The bisection result is implausible. I just did some experiments and
it looks far more likely is that it is related to commit
72e73c78c446e ("cifs: close the shared root handle on tree disconnect")
so added Ronnie to the cc. That patch added a call (at unmount time)
to close_shroot.
The idea of that patch made sense - although tree disconnect (and then
logoff of the session)
will indirectly free any open handles on the server for that session,
it is a little
cleaner to close the cached root SMB3 file handle explicitly.
void close_shroot(struct cached_fid *cfid)
{
mutex_lock(&cfid->fid_mutex);
kref_put(&cfid->refcount, smb2_close_cached_fid);
mutex_unlock(&cfid->fid_mutex);
}
Taking out the one line change in the patch from last week that calls
close_shroot from
umount (SMB2_tdis, ie tree_disconnect) I don't see the problem so far
more likely
that it is related to that commit. The problem seems to be related
to servers which
don't support directory leases. Will spin up a patch to fix this if
Ronnie hasn't already fixed it
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: refcount_t: underflow; use-after-free with CIFS umount after scsi-misc commit ef2cc88e2a205b8a11a19e78db63a70d3728cdf5
2019-12-09 3:18 ` Steve French
@ 2019-12-09 3:27 ` Steve French
0 siblings, 0 replies; 8+ messages in thread
From: Steve French @ 2019-12-09 3:27 UTC (permalink / raw)
To: Linus Torvalds
Cc: Arthur Marsh, SCSI development list, Linux Kernel Mailing List,
CIFS, James E.J. Bottomley, ronnie sahlberg
Arthur,
If you want to avoid the issue which causes the oops you can remove
this one line change (see below) but I (or Ronnie) will post a patch
for this - but wanted to discuss with him briefly first.
# git show 72e73c78c446e
commit 72e73c78c446e3c009a29b017c7fa3d79463e2aa
Author: Ronnie Sahlberg <lsahlber@redhat.com>
Date: Thu Nov 7 17:00:38 2019 +1000
cifs: close the shared root handle on tree disconnect
Signed-off-by: Ronnie Sahlberg <lsahlber@redhat.com>
Signed-off-by: Steve French <stfrench@microsoft.com>
diff --git a/fs/cifs/smb2pdu.c b/fs/cifs/smb2pdu.c
index 05149862aea4..acb70f67efc9 100644
--- a/fs/cifs/smb2pdu.c
+++ b/fs/cifs/smb2pdu.c
@@ -1807,6 +1807,8 @@ SMB2_tdis(const unsigned int xid, struct cifs_tcon *tcon)
if ((tcon->need_reconnect) || (tcon->ses->need_reconnect))
return 0;
+ close_shroot(&tcon->crfid);
+
rc = smb2_plain_req_init(SMB2_TREE_DISCONNECT, tcon, (void **) &req,
&total_len);
if (rc)
On Sun, Dec 8, 2019 at 9:18 PM Steve French <smfrench@gmail.com> wrote:
>
> On Sun, Dec 8, 2019 at 8:23 PM Linus Torvalds
> <torvalds@linux-foundation.org> wrote:
> >
> > On Sun, Dec 8, 2019 at 5:49 PM Arthur Marsh
> > <arthur.marsh@internode.on.net> wrote:
> > >
> > > This still happens with 5.5.0-rc1:
> >
> > Does it happen 100% of the time?
>
> I can reproduce it (although it was a little more difficult since WiFi doesn't
> work on RC1 on some of my hardware - due to the 802.11 driver regression oops.
> I was able to reproduce it to Samba localhost).
>
>
> > Your bisection result looks pretty nonsensical - not that it's
> > impossible (anything is possible), but it really doesn't look very
> > likely. Which makes me think maybe it's slightly timing-sensitive or
> > something?
>
> The bisection result is implausible. I just did some experiments and
> it looks far more likely is that it is related to commit
> 72e73c78c446e ("cifs: close the shared root handle on tree disconnect")
> so added Ronnie to the cc. That patch added a call (at unmount time)
> to close_shroot.
> The idea of that patch made sense - although tree disconnect (and then
> logoff of the session)
> will indirectly free any open handles on the server for that session,
> it is a little
> cleaner to close the cached root SMB3 file handle explicitly.
>
> void close_shroot(struct cached_fid *cfid)
> {
> mutex_lock(&cfid->fid_mutex);
> kref_put(&cfid->refcount, smb2_close_cached_fid);
> mutex_unlock(&cfid->fid_mutex);
> }
>
>
> Taking out the one line change in the patch from last week that calls
> close_shroot from
> umount (SMB2_tdis, ie tree_disconnect) I don't see the problem so far
> more likely
> that it is related to that commit. The problem seems to be related
> to servers which
> don't support directory leases. Will spin up a patch to fix this if
> Ronnie hasn't already fixed it
--
Thanks,
Steve
^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: refcount_t: underflow; use-after-free with CIFS umount after scsi-misc commit ef2cc88e2a205b8a11a19e78db63a70d3728cdf5
2019-12-09 2:23 ` Linus Torvalds
2019-12-09 2:52 ` Al Viro
2019-12-09 3:18 ` Steve French
@ 2019-12-09 14:45 ` Arthur Marsh
2 siblings, 0 replies; 8+ messages in thread
From: Arthur Marsh @ 2019-12-09 14:45 UTC (permalink / raw)
To: Linus Torvalds
Cc: SCSI development list, Linux Kernel Mailing List, CIFS,
James E.J. Bottomley
Hi, I ran the last good kernel with several boot-up, cifs mount, un-mount, shut down cycles without encountering the problem.
After applying the patch from <ronniesahlberg@gmail.com>:
diff --git a/fs/cifs/smb2pdu.c b/fs/cifs/smb2pdu.c
index 0ab6b1200288..d2658f51ff60 100644
--- a/fs/cifs/smb2pdu.c
+++ b/fs/cifs/smb2pdu.c
@@ -1847,7 +1847,8 @@ SMB2_tdis(const unsigned int xid, struct cifs_tcon *tcon)
if ((tcon->need_reconnect) || (tcon->ses->need_reconnect))
return 0;
- close_shroot(&tcon->crfid);
+ if (tcon->crfid.is_valid)
+ close_shroot(&tcon->crfid);
to kernel 5.5.0-rc1 I no longer experience the problem.
Regards,
Arthur.
On 9 December 2019 12:53:02 pm ACDT, Linus Torvalds <torvalds@linux-foundation.org> wrote:
>On Sun, Dec 8, 2019 at 5:49 PM Arthur Marsh
><arthur.marsh@internode.on.net> wrote:
>>
>> This still happens with 5.5.0-rc1:
>
>Does it happen 100% of the time?
>
>Your bisection result looks pretty nonsensical - not that it's
>impossible (anything is possible), but it really doesn't look very
>likely. Which makes me think maybe it's slightly timing-sensitive or
>something?
>
>Would you mind trying to re-do the bisection, and for each kernel try
>the mount thing at least a few times before you decide a kernel is
>good?
>
>Bisection is very powerful, but if _any_ of the kernels you marked
>good weren't really good (they just happened to not trigger the
>problem), bisection ends up giving completely the wrong answer. And
>with that bisection commit, there's not even a hint of what could have
>gone wrong.
>
> Linus
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
^ permalink raw reply related [flat|nested] 8+ messages in thread