* [PATCH]i[RESEND] kernel/acct.c: fix locking order when switching acct files
@ 2018-09-29 5:51 Amir Goldstein
2018-10-17 19:39 ` [PATCH][RESEND] " Amir Goldstein
0 siblings, 1 reply; 3+ messages in thread
From: Amir Goldstein @ 2018-09-29 5:51 UTC (permalink / raw)
To: Al Viro; +Cc: Miklos Szeredi, linux-unionfs, linux-fsdevel
This looks like an old bug, pre-dating the "Fixes" commit, but the
"Fixes" commit did not handle it properly.
The bug recently surfaced as a lockdep possible deadlock warning
with commit d1d04ef8572b ("ovl: stack file ops").
When acct_on() replaces one acct file with another, it takes sb_writers
lock on new file sb and calls acct_pin_kill(old) before releasing the
sb_writers lock.
If new file is on the same fs as old file, acct_pin_kill(old) fail to
file_start_write_trylock() and skip writing the old file, because
sb_writers (of new) is already taken by acct_on().
If new file is not on same fs as old file, this ordering violation
creates an unneeded dependency between new sb_writers and old sb_writers,
which may later be reported as possible deadlock.
This could result in an actual deadlock if acct file is replaced from
an old file in overlayfs over "real fs" to a new file in "real fs".
acct_on() takes freeze protection on "real fs" and tries to write to
overlayfs file. overlayfs is not freeze protected so do_acct_process()
can carry on with __kernel_write() to overlayfs file, which would
try to take freeze protection on "real fs" and deadlock.
Reproducer of lockdep possible deadlock warning:
./run --ov -s # unionmount-testsuite
touch /mnt/x /upper/y
accton /mnt/x
accton /upper/y
======================================================
WARNING: possible circular locking dependency detected
4.19.0-rc1-xfstests #3424 Not tainted
------------------------------------------------------
accton/1390 is trying to acquire lock:
00000000e0585aa5 (&acct->lock#2){+.+.}, at: acct_pin_kill+0x1b/0x76
but task is already holding lock:
000000003692505a (sb_writers#6){.+.+}, at: mnt_want_write+0x1d/0x42
Reported-by: syzbot+695726bc473f9c36a4b6@syzkaller.appspotmail.com
Fixes: 59eda0e07f43 ("new fs_pin killing logics")
Signed-off-by: Amir Goldstein <amir73il@gmail.com>
---
Al,
Welcome back. It would be nice to get an ACK (or an Applied) on this
patch. Fixes label claims to fix your commit, but I believe the bug was
already there before your commit.
As you can see, I have a reproducer to demonstrate the manifestation of
the bug in the case of switching acct file from overlayfs to real fs.
This started to manifest with overlayfs stacked f_op.
I have made another claim which seems obvious from the code about
changing acct file on same fs, but did not bother to write a reproducer
for that case.
Thanks,
Amir.
kernel/acct.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/kernel/acct.c b/kernel/acct.c
index addf7732fb56..c09355a7ae46 100644
--- a/kernel/acct.c
+++ b/kernel/acct.c
@@ -251,8 +251,8 @@ static int acct_on(struct filename *pathname)
rcu_read_lock();
old = xchg(&ns->bacct, &acct->pin);
mutex_unlock(&acct->lock);
- pin_kill(old);
mnt_drop_write(mnt);
+ pin_kill(old);
mntput(mnt);
return 0;
}
--
2.17.1
^ permalink raw reply related [flat|nested] 3+ messages in thread
* Re: [PATCH][RESEND] kernel/acct.c: fix locking order when switching acct files
2018-09-29 5:51 [PATCH]i[RESEND] kernel/acct.c: fix locking order when switching acct files Amir Goldstein
@ 2018-10-17 19:39 ` Amir Goldstein
2019-02-26 14:22 ` Amir Goldstein
0 siblings, 1 reply; 3+ messages in thread
From: Amir Goldstein @ 2018-10-17 19:39 UTC (permalink / raw)
To: Al Viro; +Cc: Miklos Szeredi, overlayfs, linux-fsdevel
On Sat, Sep 29, 2018 at 8:51 AM Amir Goldstein <amir73il@gmail.com> wrote:
>
> This looks like an old bug, pre-dating the "Fixes" commit, but the
> "Fixes" commit did not handle it properly.
>
> The bug recently surfaced as a lockdep possible deadlock warning
> with commit d1d04ef8572b ("ovl: stack file ops").
>
> When acct_on() replaces one acct file with another, it takes sb_writers
> lock on new file sb and calls acct_pin_kill(old) before releasing the
> sb_writers lock.
>
> If new file is on the same fs as old file, acct_pin_kill(old) fail to
> file_start_write_trylock() and skip writing the old file, because
> sb_writers (of new) is already taken by acct_on().
>
> If new file is not on same fs as old file, this ordering violation
> creates an unneeded dependency between new sb_writers and old sb_writers,
> which may later be reported as possible deadlock.
>
> This could result in an actual deadlock if acct file is replaced from
> an old file in overlayfs over "real fs" to a new file in "real fs".
> acct_on() takes freeze protection on "real fs" and tries to write to
> overlayfs file. overlayfs is not freeze protected so do_acct_process()
> can carry on with __kernel_write() to overlayfs file, which would
> try to take freeze protection on "real fs" and deadlock.
>
> Reproducer of lockdep possible deadlock warning:
>
> ./run --ov -s # unionmount-testsuite
> touch /mnt/x /upper/y
> accton /mnt/x
> accton /upper/y
>
> ======================================================
> WARNING: possible circular locking dependency detected
> 4.19.0-rc1-xfstests #3424 Not tainted
> ------------------------------------------------------
> accton/1390 is trying to acquire lock:
> 00000000e0585aa5 (&acct->lock#2){+.+.}, at: acct_pin_kill+0x1b/0x76
>
> but task is already holding lock:
> 000000003692505a (sb_writers#6){.+.+}, at: mnt_want_write+0x1d/0x42
>
> Reported-by: syzbot+695726bc473f9c36a4b6@syzkaller.appspotmail.com
> Fixes: 59eda0e07f43 ("new fs_pin killing logics")
> Signed-off-by: Amir Goldstein <amir73il@gmail.com>
> ---
>
> Al,
>
> Welcome back. It would be nice to get an ACK (or an Applied) on this
> patch. Fixes label claims to fix your commit, but I believe the bug was
> already there before your commit.
>
> As you can see, I have a reproducer to demonstrate the manifestation of
> the bug in the case of switching acct file from overlayfs to real fs.
> This started to manifest with overlayfs stacked f_op.
> I have made another claim which seems obvious from the code about
> changing acct file on same fs, but did not bother to write a reproducer
> for that case.
>
> Thanks,
> Amir.
PING
>
> kernel/acct.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/kernel/acct.c b/kernel/acct.c
> index addf7732fb56..c09355a7ae46 100644
> --- a/kernel/acct.c
> +++ b/kernel/acct.c
> @@ -251,8 +251,8 @@ static int acct_on(struct filename *pathname)
> rcu_read_lock();
> old = xchg(&ns->bacct, &acct->pin);
> mutex_unlock(&acct->lock);
> - pin_kill(old);
> mnt_drop_write(mnt);
> + pin_kill(old);
> mntput(mnt);
> return 0;
> }
> --
> 2.17.1
>
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [PATCH][RESEND] kernel/acct.c: fix locking order when switching acct files
2018-10-17 19:39 ` [PATCH][RESEND] " Amir Goldstein
@ 2019-02-26 14:22 ` Amir Goldstein
0 siblings, 0 replies; 3+ messages in thread
From: Amir Goldstein @ 2019-02-26 14:22 UTC (permalink / raw)
To: Al Viro; +Cc: Miklos Szeredi, overlayfs, linux-fsdevel
On Wed, Oct 17, 2018 at 10:39 PM Amir Goldstein <amir73il@gmail.com> wrote:
>
> On Sat, Sep 29, 2018 at 8:51 AM Amir Goldstein <amir73il@gmail.com> wrote:
> >
> > This looks like an old bug, pre-dating the "Fixes" commit, but the
> > "Fixes" commit did not handle it properly.
> >
> > The bug recently surfaced as a lockdep possible deadlock warning
> > with commit d1d04ef8572b ("ovl: stack file ops").
> >
> > When acct_on() replaces one acct file with another, it takes sb_writers
> > lock on new file sb and calls acct_pin_kill(old) before releasing the
> > sb_writers lock.
> >
> > If new file is on the same fs as old file, acct_pin_kill(old) fail to
> > file_start_write_trylock() and skip writing the old file, because
> > sb_writers (of new) is already taken by acct_on().
> >
> > If new file is not on same fs as old file, this ordering violation
> > creates an unneeded dependency between new sb_writers and old sb_writers,
> > which may later be reported as possible deadlock.
> >
> > This could result in an actual deadlock if acct file is replaced from
> > an old file in overlayfs over "real fs" to a new file in "real fs".
> > acct_on() takes freeze protection on "real fs" and tries to write to
> > overlayfs file. overlayfs is not freeze protected so do_acct_process()
> > can carry on with __kernel_write() to overlayfs file, which would
> > try to take freeze protection on "real fs" and deadlock.
> >
> > Reproducer of lockdep possible deadlock warning:
> >
> > ./run --ov -s # unionmount-testsuite
> > touch /mnt/x /upper/y
> > accton /mnt/x
> > accton /upper/y
> >
> > ======================================================
> > WARNING: possible circular locking dependency detected
> > 4.19.0-rc1-xfstests #3424 Not tainted
> > ------------------------------------------------------
> > accton/1390 is trying to acquire lock:
> > 00000000e0585aa5 (&acct->lock#2){+.+.}, at: acct_pin_kill+0x1b/0x76
> >
> > but task is already holding lock:
> > 000000003692505a (sb_writers#6){.+.+}, at: mnt_want_write+0x1d/0x42
> >
> > Reported-by: syzbot+695726bc473f9c36a4b6@syzkaller.appspotmail.com
> > Fixes: 59eda0e07f43 ("new fs_pin killing logics")
> > Signed-off-by: Amir Goldstein <amir73il@gmail.com>
> > ---
> >
> > Al,
> >
> > Welcome back. It would be nice to get an ACK (or an Applied) on this
> > patch. Fixes label claims to fix your commit, but I believe the bug was
> > already there before your commit.
> >
> > As you can see, I have a reproducer to demonstrate the manifestation of
> > the bug in the case of switching acct file from overlayfs to real fs.
> > This started to manifest with overlayfs stacked f_op.
> > I have made another claim which seems obvious from the code about
> > changing acct file on same fs, but did not bother to write a reproducer
> > for that case.
> >
> > Thanks,
> > Amir.
>
> PING
PING^2
>
> >
> > kernel/acct.c | 2 +-
> > 1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/kernel/acct.c b/kernel/acct.c
> > index addf7732fb56..c09355a7ae46 100644
> > --- a/kernel/acct.c
> > +++ b/kernel/acct.c
> > @@ -251,8 +251,8 @@ static int acct_on(struct filename *pathname)
> > rcu_read_lock();
> > old = xchg(&ns->bacct, &acct->pin);
> > mutex_unlock(&acct->lock);
> > - pin_kill(old);
> > mnt_drop_write(mnt);
> > + pin_kill(old);
> > mntput(mnt);
> > return 0;
> > }
> > --
> > 2.17.1
> >
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2019-02-26 14:22 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-09-29 5:51 [PATCH]i[RESEND] kernel/acct.c: fix locking order when switching acct files Amir Goldstein
2018-10-17 19:39 ` [PATCH][RESEND] " Amir Goldstein
2019-02-26 14:22 ` Amir Goldstein
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).