Re: INFO: task hung in lo_ioctl

Re: INFO: task hung in lo_ioctl

[Tetsuo Handa]

This seems to be an AB-BA deadlock where the lockdep cannot report (due to use of nested lock?).

When PID=6540 was (reported as hung) at mutex_lock_nested(&lo->lo_ctl_mutex, 1) (id=43ca8836),
it was already holding down_write_nested(&s->s_umount, SINGLE_DEPTH_NESTING) (id=566d4c39).
But when PID=6541 was (which would have been reported as hung if sysctl_hung_task_panic
were not set) at down_read(&sb->s_umount) (id=566d4c39), it was already holding
mutex_lock_nested(&lo->lo_ctl_mutex, 1) (id=43ca8836).

----------------------------------------
INFO: task syz-executor0:6540 blocked for more than 120 seconds.
      Not tainted 4.16.0+ #13
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
syz-executor0   D23560  6540   4521 0x80000004
Call Trace:
 context_switch kernel/sched/core.c:2848 [inline]
 __schedule+0x8fb/0x1ef0 kernel/sched/core.c:3490
 schedule+0xf5/0x430 kernel/sched/core.c:3549
 schedule_preempt_disabled+0x10/0x20 kernel/sched/core.c:3607
 __mutex_lock_common kernel/locking/mutex.c:833 [inline]
 __mutex_lock+0xb7f/0x1810 kernel/locking/mutex.c:893
 mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
 lo_ioctl+0x8b/0x1b70 drivers/block/loop.c:1355
 __blkdev_driver_ioctl block/ioctl.c:303 [inline]
 blkdev_ioctl+0x1759/0x1e00 block/ioctl.c:601
 ioctl_by_bdev+0xa5/0x110 fs/block_dev.c:2060
 isofs_get_last_session fs/isofs/inode.c:567 [inline]
 isofs_fill_super+0x2ba9/0x3bc0 fs/isofs/inode.c:660
 mount_bdev+0x2b7/0x370 fs/super.c:1119
 isofs_mount+0x34/0x40 fs/isofs/inode.c:1560
 mount_fs+0x66/0x2d0 fs/super.c:1222

2 locks held by syz-executor0/6540:
 #0: 00000000566d4c39 (&type->s_umount_key#49/1){+.+.}, at: alloc_super fs/super.c:211 [inline]
 #0: 00000000566d4c39 (&type->s_umount_key#49/1){+.+.}, at: sget_userns+0x3b2/0xe60 fs/super.c:502 /* down_write_nested(&s->s_umount, SINGLE_DEPTH_NESTING); */
 #1: 0000000043ca8836 (&lo->lo_ctl_mutex/1){+.+.}, at: lo_ioctl+0x8b/0x1b70 drivers/block/loop.c:1355 /* mutex_lock_nested(&lo->lo_ctl_mutex, 1); */
3 locks held by syz-executor7/6541:
 #0: 0000000043ca8836 (&lo->lo_ctl_mutex/1){+.+.}, at: lo_ioctl+0x8b/0x1b70 drivers/block/loop.c:1355 /* mutex_lock_nested(&lo->lo_ctl_mutex, 1); */
 #1: 000000007bf3d3f9 (&bdev->bd_mutex){+.+.}, at: blkdev_reread_part+0x1e/0x40 block/ioctl.c:192
 #2: 00000000566d4c39 (&type->s_umount_key#50){.+.+}, at: __get_super.part.10+0x1d3/0x280 fs/super.c:663 /* down_read(&sb->s_umount); */
----------------------------------------

sget() is using down_write_nested(&s->s_umount, SINGLE_DEPTH_NESTING)
with a comment block asserting that there is no risk of deadlock

	/*
	 * sget() can have s_umount recursion.
	 *
	 * When it cannot find a suitable sb, it allocates a new
	 * one (this one), and tries again to find a suitable old
	 * one.
	 *
	 * In case that succeeds, it will acquire the s_umount
	 * lock of the old one. Since these are clearly distrinct
	 * locks, and this object isn't exposed yet, there's no
	 * risk of deadlocks.
	 *
	 * Annotate this by putting this lock in a different
	 * subclass.
	 */

but this object (id=566d4c39) is already locked by other thread.
What is happening here?

Re: INFO: task hung in lo_ioctl

[Tetsuo Handa]

On 2018/04/05 0:23, Tetsuo Handa wrote:
> This seems to be an AB-BA deadlock where the lockdep cannot report (due to use of nested lock?).
> What is happening here?
> 

This patch does not directly fix the bug syzbot is reporting, but could be relevant.
Maybe try replacing mutex_lock_killable_nested() with mutex_lock_killable() and
check what the lockdep will say?



>From 0b006d536b2e439f347eb857e482fc304e84fd1d Mon Sep 17 00:00:00 2001
From: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
Date: Fri, 6 Apr 2018 20:52:10 +0900
Subject: [PATCH] block/loop: fix lock imbalance bug at lo_ioctl

Commit 2d1d4c1e591fd40b ("loop: don't call into filesystem while holding
lo_ctl_mutex") introduced mutex lock imbalance bug where syzbot would
trigger. The caller of loop_get_status64() assumes that mutex_unlock() is
already called by loop_get_status() but loop_get_status64() does not
always call loop_get_status().

Commit f028f3b2f987ebc6 ("loop: fix circular locking in loop_clr_fd()")
also dropped the mutex for deadlock avoidance reason. But we should
recheck whether we have to drop the mutex. Dropping the mutex at the
middle of an ioctl() request is not nice, but syzbot is reporting a
deadlock inside loop_reread_partitions() which is called by loop_set_fd()
and loop_change_fd().

Signed-off-by: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
Fixes: 2d1d4c1e591fd40b ("loop: don't call into filesystem while holding lo_ctl_mutex")
Cc: Omar Sandoval <osandov@fb.com>
Cc: Nikanth Karthikesan <knikanth@suse.de>
Cc: Jens Axboe <jens.axboe@oracle.com>
---
 drivers/block/loop.c | 17 ++++++-----------
 1 file changed, 6 insertions(+), 11 deletions(-)

diff --git a/drivers/block/loop.c b/drivers/block/loop.c
index 264abaa..64033e7 100644
--- a/drivers/block/loop.c
+++ b/drivers/block/loop.c
@@ -1362,7 +1362,7 @@ static int lo_ioctl(struct block_device *bdev, fmode_t mode,
 
 	err = mutex_lock_killable_nested(&lo->lo_ctl_mutex, 1);
 	if (err)
-		goto out_unlocked;
+		return err;
 
 	switch (cmd) {
 	case LOOP_SET_FD:
@@ -1372,10 +1372,7 @@ static int lo_ioctl(struct block_device *bdev, fmode_t mode,
 		err = loop_change_fd(lo, bdev, arg);
 		break;
 	case LOOP_CLR_FD:
-		/* loop_clr_fd would have unlocked lo_ctl_mutex on success */
 		err = loop_clr_fd(lo);
-		if (!err)
-			goto out_unlocked;
 		break;
 	case LOOP_SET_STATUS:
 		err = -EPERM;
@@ -1385,8 +1382,7 @@ static int lo_ioctl(struct block_device *bdev, fmode_t mode,
 		break;
 	case LOOP_GET_STATUS:
 		err = loop_get_status_old(lo, (struct loop_info __user *) arg);
-		/* loop_get_status() unlocks lo_ctl_mutex */
-		goto out_unlocked;
+		break;
 	case LOOP_SET_STATUS64:
 		err = -EPERM;
 		if ((mode & FMODE_WRITE) || capable(CAP_SYS_ADMIN))
@@ -1395,8 +1391,7 @@ static int lo_ioctl(struct block_device *bdev, fmode_t mode,
 		break;
 	case LOOP_GET_STATUS64:
 		err = loop_get_status64(lo, (struct loop_info64 __user *) arg);
-		/* loop_get_status() unlocks lo_ctl_mutex */
-		goto out_unlocked;
+		break;
 	case LOOP_SET_CAPACITY:
 		err = -EPERM;
 		if ((mode & FMODE_WRITE) || capable(CAP_SYS_ADMIN))
@@ -1415,9 +1410,9 @@ static int lo_ioctl(struct block_device *bdev, fmode_t mode,
 	default:
 		err = lo->ioctl ? lo->ioctl(lo, cmd, arg) : -EINVAL;
 	}
-	mutex_unlock(&lo->lo_ctl_mutex);
-
-out_unlocked:
+	/* Temporary hack for handling lock imbalance. */
+	if (__mutex_owner(&lo->lo_ctl_mutex) == current)
+		mutex_unlock(&lo->lo_ctl_mutex);
 	return err;
 }
 
-- 
1.8.3.1

Re: INFO: task hung in lo_ioctl

[Peter Zijlstra]

On Fri, Apr 06, 2018 at 09:04:18PM +0900, Tetsuo Handa wrote:
> +	/* Temporary hack for handling lock imbalance. */
> +	if (__mutex_owner(&lo->lo_ctl_mutex) == current)
> +		mutex_unlock(&lo->lo_ctl_mutex);

ARGGH.. you didn't read the comment we put on that?

Re: INFO: task hung in lo_ioctl

[Tetsuo Handa]

Peter Zijlstra wrote:
> On Fri, Apr 06, 2018 at 09:04:18PM +0900, Tetsuo Handa wrote:
> > +	/* Temporary hack for handling lock imbalance. */
> > +	if (__mutex_owner(&lo->lo_ctl_mutex) == current)
> > +		mutex_unlock(&lo->lo_ctl_mutex);
> 
> ARGGH.. you didn't read the comment we put on that?
> 

Commit 5b52330bbfe63b33 ("audit: fix auditd/kernel connection state tracking")
is using __mutex_owner(). ;-)

Of course, regarding loop module, we will be able to add a flag variable
associated with lo->lo_ctl_mutex. But that will be done after we solved
the deadlock problem. I think whether we need to drop lo->lo_ctl_mutex is
not clear. Maybe

-	err = mutex_lock_killable_nested(&lo->lo_ctl_mutex, 1);
+	err = mutex_lock_killable(&lo->lo_ctl_mutex);

on top of this patch and listen to the lockdep?

Commit f028f3b2f987ebc6 ("loop: fix circular locking in loop_clr_fd()") says

  A simple way to silence lockdep could be to mark the lo_ctl_mutex
  in ioctl to be a sub class, but this might mask some other real bugs.

and we are currently hitting a deadlock problem.

Re: INFO: task hung in lo_ioctl

[Peter Zijlstra]

On Fri, Apr 06, 2018 at 10:55:03PM +0900, Tetsuo Handa wrote:
> Peter Zijlstra wrote:
> > On Fri, Apr 06, 2018 at 09:04:18PM +0900, Tetsuo Handa wrote:
> > > +	/* Temporary hack for handling lock imbalance. */
> > > +	if (__mutex_owner(&lo->lo_ctl_mutex) == current)
> > > +		mutex_unlock(&lo->lo_ctl_mutex);
> > 
> > ARGGH.. you didn't read the comment we put on that?
> > 
> 
> Commit 5b52330bbfe63b33 ("audit: fix auditd/kernel connection state tracking")
> is using __mutex_owner(). ;-)

That got removed and the warning added.

Re: INFO: task hung in lo_ioctl

[Omar Sandoval]

On Fri, Apr 06, 2018 at 05:43:43PM +0200, Peter Zijlstra wrote:
> On Fri, Apr 06, 2018 at 10:55:03PM +0900, Tetsuo Handa wrote:
> > Peter Zijlstra wrote:
> > > On Fri, Apr 06, 2018 at 09:04:18PM +0900, Tetsuo Handa wrote:
> > > > +	/* Temporary hack for handling lock imbalance. */
> > > > +	if (__mutex_owner(&lo->lo_ctl_mutex) == current)
> > > > +		mutex_unlock(&lo->lo_ctl_mutex);
> > > 
> > > ARGGH.. you didn't read the comment we put on that?
> > > 
> > 
> > Commit 5b52330bbfe63b33 ("audit: fix auditd/kernel connection state tracking")
> > is using __mutex_owner(). ;-)
> 
> That got removed and the warning added.

Seems easy enough to fix without resorting to __mutex_owner() (untested):


diff --git a/drivers/block/loop.c b/drivers/block/loop.c
index 264abaaff662..cee258d12a1e 100644
--- a/drivers/block/loop.c
+++ b/drivers/block/loop.c
@@ -1300,12 +1300,13 @@ loop_get_status_old(struct loop_device *lo, struct loop_info __user *arg) {
 static int
 loop_get_status64(struct loop_device *lo, struct loop_info64 __user *arg) {
 	struct loop_info64 info64;
-	int err = 0;
+	int err;
 
-	if (!arg)
-		err = -EINVAL;
-	if (!err)
-		err = loop_get_status(lo, &info64);
+	if (!arg) {
+		mutex_unlock(&lo->lo_ctl_mutex);
+		return -EINVAL;
+	}
+	err = loop_get_status(lo, &info64);
 	if (!err && copy_to_user(arg, &info64, sizeof(info64)))
 		err = -EFAULT;
 

I'll test it and send it up when I get into the office.

Re: INFO: task hung in lo_ioctl

[Dmitry Vyukov]

On Fri, Apr 6, 2018 at 5:59 PM Omar Sandoval <osandov@osandov.com> wrote:
>
> On Fri, Apr 06, 2018 at 05:43:43PM +0200, Peter Zijlstra wrote:
> > On Fri, Apr 06, 2018 at 10:55:03PM +0900, Tetsuo Handa wrote:
> > > Peter Zijlstra wrote:
> > > > On Fri, Apr 06, 2018 at 09:04:18PM +0900, Tetsuo Handa wrote:
> > >
 > > +       /* Temporary hack for handling lock imbalance. */
> > > > > +       if (__mutex_owner(&lo->lo_ctl_mutex) == current)
> > > > > +               mutex_unlock(&lo->lo_ctl_mutex);
> > > >
> > > > ARGGH.. you didn't read the comment we put on that?
> > > >
> > >
> > > Commit 5b52330bbfe63b33 ("audit: fix auditd/kernel connection state tracking")
> > > is using __mutex_owner(). ;-)
> >
> > That got removed and the warning added.
>
> Seems easy enough to fix without resorting to __mutex_owner() (untested):
>
>
> diff --git a/drivers/block/loop.c b/drivers/block/loop.c
> index 264abaaff662..cee258d12a1e 100644
> --- a/drivers/block/loop.c
> +++ b/drivers/block/loop.c
> @@ -1300,12 +1300,13 @@ loop_get_status_old(struct loop_device *lo, struct loop_info __user *arg) {
>  static int
>  loop_get_status64(struct loop_device *lo, struct loop_info64 __user *arg) {
>         struct loop_info64 info64;
> -       int err = 0;
> +       int err;
>
> -       if (!arg)
> -               err = -EINVAL;
> -       if (!err)
> -               err = loop_get_status(lo, &info64);
> +       if (!arg) {
> +               mutex_unlock(&lo->lo_ctl_mutex);
> +               return -EINVAL;
> +       }
> +       err = loop_get_status(lo, &info64);
>         if (!err && copy_to_user(arg, &info64, sizeof(info64)))
>                 err = -EFAULT;
>
>
> I'll test it and send it up when I get into the office.


Was this ever submitted? Or some other fix for this?

The bug is still open, but last happened 289 days ago:
https://syzkaller.appspot.com/bug?id=608144371e7fc2cb6285b9ed871fb1eb817a61ce

But it also has 10 duplicates, some of which happened much more recently.
If a fix was submitted, but Reported-by tag wasn't added this open bug
can now mask lots of other new bugs.

Re: INFO: task hung in lo_ioctl

[Tetsuo Handa]

On 2019/01/20 3:56, Dmitry Vyukov wrote:
>> Seems easy enough to fix without resorting to __mutex_owner() (untested):
>>
>>
>> diff --git a/drivers/block/loop.c b/drivers/block/loop.c
>> index 264abaaff662..cee258d12a1e 100644
>> --- a/drivers/block/loop.c
>> +++ b/drivers/block/loop.c
>> @@ -1300,12 +1300,13 @@ loop_get_status_old(struct loop_device *lo, struct loop_info __user *arg) {
>>  static int
>>  loop_get_status64(struct loop_device *lo, struct loop_info64 __user *arg) {
>>         struct loop_info64 info64;
>> -       int err = 0;
>> +       int err;
>>
>> -       if (!arg)
>> -               err = -EINVAL;
>> -       if (!err)
>> -               err = loop_get_status(lo, &info64);
>> +       if (!arg) {
>> +               mutex_unlock(&lo->lo_ctl_mutex);
>> +               return -EINVAL;
>> +       }
>> +       err = loop_get_status(lo, &info64);
>>         if (!err && copy_to_user(arg, &info64, sizeof(info64)))
>>                 err = -EFAULT;
>>
>>
>> I'll test it and send it up when I get into the office.
> 
> 
> Was this ever submitted? Or some other fix for this?
> 
> The bug is still open, but last happened 289 days ago:
> https://syzkaller.appspot.com/bug?id=608144371e7fc2cb6285b9ed871fb1eb817a61ce
> 
> But it also has 10 duplicates, some of which happened much more recently.
> If a fix was submitted, but Reported-by tag wasn't added this open bug
> can now mask lots of other new bugs.
> 

The commit for this specific patch is bdac616db9bbadb9 ("loop: fix LOOP_GET_STATUS
lock imbalance"). But the root cause of these hung tasks would be fixed by a series
containing commit 1dded9acf6dc9a34 ("Avoid circular locking dependency between
loop_ctl_mutex and bd_mutex") or commit 04906b2f542c2362 ("blockdev: Fix livelocks
on loop device"). We were not aware of these bugs when you marked these reports as
duplicates on 2017/12/12. You can undup them and fix them if you want.

Re: INFO: task hung in lo_ioctl

[Dmitry Vyukov]

On Sun, Jan 20, 2019 at 3:36 AM Tetsuo Handa
<penguin-kernel@i-love.sakura.ne.jp> wrote:
>
> On 2019/01/20 3:56, Dmitry Vyukov wrote:
> >> Seems easy enough to fix without resorting to __mutex_owner() (untested):
> >>
> >>
> >> diff --git a/drivers/block/loop.c b/drivers/block/loop.c
> >> index 264abaaff662..cee258d12a1e 100644
> >> --- a/drivers/block/loop.c
> >> +++ b/drivers/block/loop.c
> >> @@ -1300,12 +1300,13 @@ loop_get_status_old(struct loop_device *lo, struct loop_info __user *arg) {
> >>  static int
> >>  loop_get_status64(struct loop_device *lo, struct loop_info64 __user *arg) {
> >>         struct loop_info64 info64;
> >> -       int err = 0;
> >> +       int err;
> >>
> >> -       if (!arg)
> >> -               err = -EINVAL;
> >> -       if (!err)
> >> -               err = loop_get_status(lo, &info64);
> >> +       if (!arg) {
> >> +               mutex_unlock(&lo->lo_ctl_mutex);
> >> +               return -EINVAL;
> >> +       }
> >> +       err = loop_get_status(lo, &info64);
> >>         if (!err && copy_to_user(arg, &info64, sizeof(info64)))
> >>                 err = -EFAULT;
> >>
> >>
> >> I'll test it and send it up when I get into the office.
> >
> >
> > Was this ever submitted? Or some other fix for this?
> >
> > The bug is still open, but last happened 289 days ago:
> > https://syzkaller.appspot.com/bug?id=608144371e7fc2cb6285b9ed871fb1eb817a61ce
> >
> > But it also has 10 duplicates, some of which happened much more recently.
> > If a fix was submitted, but Reported-by tag wasn't added this open bug
> > can now mask lots of other new bugs.
> >
>
> The commit for this specific patch is bdac616db9bbadb9 ("loop: fix LOOP_GET_STATUS
> lock imbalance"). But the root cause of these hung tasks would be fixed by a series
> containing commit 1dded9acf6dc9a34 ("Avoid circular locking dependency between
> loop_ctl_mutex and bd_mutex") or commit 04906b2f542c2362 ("blockdev: Fix livelocks
> on loop device"). We were not aware of these bugs when you marked these reports as
> duplicates on 2017/12/12. You can undup them and fix them if you want.

OK, let's just do then:

#syz fix: blockdev: Fix livelocks on loop device