* [PATCH] block: loop: fix deadlock between open and remove
@ 2021-06-04 0:04 Ming Lei
2021-06-04 6:57 ` Christoph Hellwig
0 siblings, 1 reply; 7+ messages in thread
From: Ming Lei @ 2021-06-04 0:04 UTC (permalink / raw)
To: Jens Axboe, Christoph Hellwig
Cc: linux-block, Ming Lei, Colin Ian King, Pavel Tatashin
Commit c76f48eb5c08 ("block: take bd_mutex around delete_partitions in
del_gendisk") adds disk->part0->bd_mutex in del_gendisk(), this way
causes the following AB/BA deadlock between removing loop and opening
loop:
1) loop_control_ioctl(LOOP_CTL_REMOVE)
- mutex_lock(&loop_ctl_mutex)
- mutex_lock(&disk->part0->bd_mutex) //del_gendisk
2) open look device
- mutex_lock(&disk->part0->bd_mutex) //blkdev_get_by_dev
- mutex_lock(&loop_ctl_mutex) //lo_open() <- __blkdev_get
Fixes the issue by not holding loop_ctl_mutex in lo_open(), and cover
the protection on bdev->bd_disk->private_data via disk->part0->bd_mutex.
Reported-by: Colin Ian King <colin.king@canonical.com>
Fixes: c76f48eb5c08 ("block: take bd_mutex around delete_partitions in del_gendisk")
Cc: Pavel Tatashin <pasha.tatashin@soleen.com>
Signed-off-by: Ming Lei <ming.lei@redhat.com>
---
drivers/block/loop.c | 31 +++++++++++++++----------------
1 file changed, 15 insertions(+), 16 deletions(-)
diff --git a/drivers/block/loop.c b/drivers/block/loop.c
index d58d68f3c7cd..b03d8f4c1cdf 100644
--- a/drivers/block/loop.c
+++ b/drivers/block/loop.c
@@ -1883,20 +1883,14 @@ static int lo_open(struct block_device *bdev, fmode_t mode)
int err;
/*
- * take loop_ctl_mutex to protect lo pointer from race with
- * loop_control_ioctl(LOOP_CTL_REMOVE), however, to reduce contention
- * release it prior to updating lo->lo_refcnt.
+ * both ->private_data and ->lo_refcnt are covered by disk's
+ * open_mutex, so race between open and remove can be avoided
*/
- err = mutex_lock_killable(&loop_ctl_mutex);
- if (err)
- return err;
lo = bdev->bd_disk->private_data;
- if (!lo) {
- mutex_unlock(&loop_ctl_mutex);
+ if (!lo)
return -ENXIO;
- }
+
err = mutex_lock_killable(&lo->lo_mutex);
- mutex_unlock(&loop_ctl_mutex);
if (err)
return err;
atomic_inc(&lo->lo_refcnt);
@@ -2272,21 +2266,26 @@ static long loop_control_ioctl(struct file *file, unsigned int cmd,
ret = loop_lookup(&lo, parm);
if (ret < 0)
break;
- ret = mutex_lock_killable(&lo->lo_mutex);
+ /* cover removing vs. opening loop device */
+ ret = mutex_lock_killable(&lo->lo_disk->part0->bd_mutex);
if (ret)
break;
- if (lo->lo_state != Lo_unbound) {
- ret = -EBUSY;
- mutex_unlock(&lo->lo_mutex);
+ ret = mutex_lock_killable(&lo->lo_mutex);
+ if (ret) {
+ mutex_unlock(&lo->lo_disk->part0->bd_mutex);
break;
}
- if (atomic_read(&lo->lo_refcnt) > 0) {
+ if (lo->lo_state != Lo_unbound ||
+ atomic_read(&lo->lo_refcnt) > 0) {
ret = -EBUSY;
mutex_unlock(&lo->lo_mutex);
+ mutex_unlock(&lo->lo_disk->part0->bd_mutex);
break;
}
- lo->lo_disk->private_data = NULL;
mutex_unlock(&lo->lo_mutex);
+ lo->lo_disk->private_data = NULL;
+ mutex_unlock(&lo->lo_disk->part0->bd_mutex);
+
idr_remove(&loop_index_idr, lo->lo_number);
loop_remove(lo);
break;
--
2.29.2
^ permalink raw reply related [flat|nested] 7+ messages in thread* Re: [PATCH] block: loop: fix deadlock between open and remove
2021-06-04 0:04 [PATCH] block: loop: fix deadlock between open and remove Ming Lei
@ 2021-06-04 6:57 ` Christoph Hellwig
2021-06-04 8:56 ` Ming Lei
0 siblings, 1 reply; 7+ messages in thread
From: Christoph Hellwig @ 2021-06-04 6:57 UTC (permalink / raw)
To: Ming Lei
Cc: Jens Axboe, Christoph Hellwig, linux-block, Colin Ian King,
Pavel Tatashin
This looks good, but I think we can do simple by just adding add new
Lo_deleting state. Completely untested as I'm at the tail end of a vacation
with a broken laptop:
diff --git a/drivers/block/loop.c b/drivers/block/loop.c
index d58d68f3c7cd..76e12f3482a9 100644
--- a/drivers/block/loop.c
+++ b/drivers/block/loop.c
@@ -1879,29 +1879,18 @@ static int lo_compat_ioctl(struct block_device *bdev, fmode_t mode,
static int lo_open(struct block_device *bdev, fmode_t mode)
{
- struct loop_device *lo;
+ struct loop_device *lo = bdev->bd_disk->private_data;
int err;
- /*
- * take loop_ctl_mutex to protect lo pointer from race with
- * loop_control_ioctl(LOOP_CTL_REMOVE), however, to reduce contention
- * release it prior to updating lo->lo_refcnt.
- */
- err = mutex_lock_killable(&loop_ctl_mutex);
- if (err)
- return err;
- lo = bdev->bd_disk->private_data;
- if (!lo) {
- mutex_unlock(&loop_ctl_mutex);
- return -ENXIO;
- }
err = mutex_lock_killable(&lo->lo_mutex);
- mutex_unlock(&loop_ctl_mutex);
if (err)
return err;
- atomic_inc(&lo->lo_refcnt);
+ if (lo->lo_state == Lo_deleting)
+ err = -ENXIO;
+ else
+ atomic_inc(&lo->lo_refcnt);
mutex_unlock(&lo->lo_mutex);
- return 0;
+ return err;
}
static void lo_release(struct gendisk *disk, fmode_t mode)
@@ -2285,7 +2274,7 @@ static long loop_control_ioctl(struct file *file, unsigned int cmd,
mutex_unlock(&lo->lo_mutex);
break;
}
- lo->lo_disk->private_data = NULL;
+ lo->lo_state = Lo_deleting;
mutex_unlock(&lo->lo_mutex);
idr_remove(&loop_index_idr, lo->lo_number);
loop_remove(lo);
diff --git a/drivers/block/loop.h b/drivers/block/loop.h
index a3c04f310672..5beb959b94d3 100644
--- a/drivers/block/loop.h
+++ b/drivers/block/loop.h
@@ -22,6 +22,7 @@ enum {
Lo_unbound,
Lo_bound,
Lo_rundown,
+ Lo_deleting,
};
struct loop_func_table;
^ permalink raw reply related [flat|nested] 7+ messages in thread* Re: [PATCH] block: loop: fix deadlock between open and remove
2021-06-04 6:57 ` Christoph Hellwig
@ 2021-06-04 8:56 ` Ming Lei
0 siblings, 0 replies; 7+ messages in thread
From: Ming Lei @ 2021-06-04 8:56 UTC (permalink / raw)
To: Christoph Hellwig; +Cc: Jens Axboe, linux-block, Colin Ian King, Pavel Tatashin
On Fri, Jun 04, 2021 at 08:57:33AM +0200, Christoph Hellwig wrote:
> This looks good, but I think we can do simple by just adding add new
> Lo_deleting state. Completely untested as I'm at the tail end of a vacation
> with a broken laptop:
Indeed, adding one deleting state is simpler to kill the lock of loop_ctl_mutex
in lo_open():
Reviewed-by: Ming Lei <ming.lei@redhat.com>
--
Ming
^ permalink raw reply [flat|nested] 7+ messages in thread
* [PATCH] block: loop: fix deadlock between open and remove
@ 2021-06-05 14:09 Christoph Hellwig
2021-06-05 15:00 ` Colin Ian King
` (2 more replies)
0 siblings, 3 replies; 7+ messages in thread
From: Christoph Hellwig @ 2021-06-05 14:09 UTC (permalink / raw)
To: axboe; +Cc: ming.lei, pasha.tatashin, linux-block, Colin Ian King
Commit c76f48eb5c08 ("block: take bd_mutex around delete_partitions in
del_gendisk") adds disk->part0->bd_mutex in del_gendisk(), this way
causes the following AB/BA deadlock between removing loop and opening
loop:
1) loop_control_ioctl(LOOP_CTL_REMOVE)
-> mutex_lock(&loop_ctl_mutex)
-> del_gendisk
-> mutex_lock(&disk->part0->bd_mutex)
2) blkdev_get_by_dev
-> mutex_lock(&disk->part0->bd_mutex)
-> lo_open
-> mutex_lock(&loop_ctl_mutex)
Add a new Lo_deleting state to remove the need for clearing
->private_data and thus holding loop_ctl_mutex in the ioctl
LOOP_CTL_REMOVE path.
Based on an analysis and earlier patch from
Ming Lei <ming.lei@redhat.com>.
Reported-by: Colin Ian King <colin.king@canonical.com>
Fixes: c76f48eb5c08 ("block: take bd_mutex around delete_partitions in del_gendisk")
Signed-off-by: Christoph Hellwig <hch@lst.de>
---
drivers/block/loop.c | 25 +++++++------------------
drivers/block/loop.h | 1 +
2 files changed, 8 insertions(+), 18 deletions(-)
diff --git a/drivers/block/loop.c b/drivers/block/loop.c
index d58d68f3c7cd..76e12f3482a9 100644
--- a/drivers/block/loop.c
+++ b/drivers/block/loop.c
@@ -1879,29 +1879,18 @@ static int lo_compat_ioctl(struct block_device *bdev, fmode_t mode,
static int lo_open(struct block_device *bdev, fmode_t mode)
{
- struct loop_device *lo;
+ struct loop_device *lo = bdev->bd_disk->private_data;
int err;
- /*
- * take loop_ctl_mutex to protect lo pointer from race with
- * loop_control_ioctl(LOOP_CTL_REMOVE), however, to reduce contention
- * release it prior to updating lo->lo_refcnt.
- */
- err = mutex_lock_killable(&loop_ctl_mutex);
- if (err)
- return err;
- lo = bdev->bd_disk->private_data;
- if (!lo) {
- mutex_unlock(&loop_ctl_mutex);
- return -ENXIO;
- }
err = mutex_lock_killable(&lo->lo_mutex);
- mutex_unlock(&loop_ctl_mutex);
if (err)
return err;
- atomic_inc(&lo->lo_refcnt);
+ if (lo->lo_state == Lo_deleting)
+ err = -ENXIO;
+ else
+ atomic_inc(&lo->lo_refcnt);
mutex_unlock(&lo->lo_mutex);
- return 0;
+ return err;
}
static void lo_release(struct gendisk *disk, fmode_t mode)
@@ -2285,7 +2274,7 @@ static long loop_control_ioctl(struct file *file, unsigned int cmd,
mutex_unlock(&lo->lo_mutex);
break;
}
- lo->lo_disk->private_data = NULL;
+ lo->lo_state = Lo_deleting;
mutex_unlock(&lo->lo_mutex);
idr_remove(&loop_index_idr, lo->lo_number);
loop_remove(lo);
diff --git a/drivers/block/loop.h b/drivers/block/loop.h
index a3c04f310672..5beb959b94d3 100644
--- a/drivers/block/loop.h
+++ b/drivers/block/loop.h
@@ -22,6 +22,7 @@ enum {
Lo_unbound,
Lo_bound,
Lo_rundown,
+ Lo_deleting,
};
struct loop_func_table;
--
2.30.2
^ permalink raw reply related [flat|nested] 7+ messages in thread* Re: [PATCH] block: loop: fix deadlock between open and remove
2021-06-05 14:09 Christoph Hellwig
@ 2021-06-05 15:00 ` Colin Ian King
2021-06-07 0:11 ` Ming Lei
2021-06-11 17:51 ` Jens Axboe
2 siblings, 0 replies; 7+ messages in thread
From: Colin Ian King @ 2021-06-05 15:00 UTC (permalink / raw)
To: Christoph Hellwig, axboe; +Cc: ming.lei, pasha.tatashin, linux-block
On 05/06/2021 15:09, Christoph Hellwig wrote:
> Commit c76f48eb5c08 ("block: take bd_mutex around delete_partitions in
> del_gendisk") adds disk->part0->bd_mutex in del_gendisk(), this way
> causes the following AB/BA deadlock between removing loop and opening
> loop:
>
> 1) loop_control_ioctl(LOOP_CTL_REMOVE)
> -> mutex_lock(&loop_ctl_mutex)
> -> del_gendisk
> -> mutex_lock(&disk->part0->bd_mutex)
>
> 2) blkdev_get_by_dev
> -> mutex_lock(&disk->part0->bd_mutex)
> -> lo_open
> -> mutex_lock(&loop_ctl_mutex)
>
> Add a new Lo_deleting state to remove the need for clearing
> ->private_data and thus holding loop_ctl_mutex in the ioctl
> LOOP_CTL_REMOVE path.
>
> Based on an analysis and earlier patch from
> Ming Lei <ming.lei@redhat.com>.
>
> Reported-by: Colin Ian King <colin.king@canonical.com>
> Fixes: c76f48eb5c08 ("block: take bd_mutex around delete_partitions in del_gendisk")
> Signed-off-by: Christoph Hellwig <hch@lst.de>
> ---
> drivers/block/loop.c | 25 +++++++------------------
> drivers/block/loop.h | 1 +
> 2 files changed, 8 insertions(+), 18 deletions(-)
>
> diff --git a/drivers/block/loop.c b/drivers/block/loop.c
> index d58d68f3c7cd..76e12f3482a9 100644
> --- a/drivers/block/loop.c
> +++ b/drivers/block/loop.c
> @@ -1879,29 +1879,18 @@ static int lo_compat_ioctl(struct block_device *bdev, fmode_t mode,
>
> static int lo_open(struct block_device *bdev, fmode_t mode)
> {
> - struct loop_device *lo;
> + struct loop_device *lo = bdev->bd_disk->private_data;
> int err;
>
> - /*
> - * take loop_ctl_mutex to protect lo pointer from race with
> - * loop_control_ioctl(LOOP_CTL_REMOVE), however, to reduce contention
> - * release it prior to updating lo->lo_refcnt.
> - */
> - err = mutex_lock_killable(&loop_ctl_mutex);
> - if (err)
> - return err;
> - lo = bdev->bd_disk->private_data;
> - if (!lo) {
> - mutex_unlock(&loop_ctl_mutex);
> - return -ENXIO;
> - }
> err = mutex_lock_killable(&lo->lo_mutex);
> - mutex_unlock(&loop_ctl_mutex);
> if (err)
> return err;
> - atomic_inc(&lo->lo_refcnt);
> + if (lo->lo_state == Lo_deleting)
> + err = -ENXIO;
> + else
> + atomic_inc(&lo->lo_refcnt);
> mutex_unlock(&lo->lo_mutex);
> - return 0;
> + return err;
> }
>
> static void lo_release(struct gendisk *disk, fmode_t mode)
> @@ -2285,7 +2274,7 @@ static long loop_control_ioctl(struct file *file, unsigned int cmd,
> mutex_unlock(&lo->lo_mutex);
> break;
> }
> - lo->lo_disk->private_data = NULL;
> + lo->lo_state = Lo_deleting;
> mutex_unlock(&lo->lo_mutex);
> idr_remove(&loop_index_idr, lo->lo_number);
> loop_remove(lo);
> diff --git a/drivers/block/loop.h b/drivers/block/loop.h
> index a3c04f310672..5beb959b94d3 100644
> --- a/drivers/block/loop.h
> +++ b/drivers/block/loop.h
> @@ -22,6 +22,7 @@ enum {
> Lo_unbound,
> Lo_bound,
> Lo_rundown,
> + Lo_deleting,
> };
>
> struct loop_func_table;
>
Tested and no longer hangs with the stress-ng loop test. Thank you for
the speedy turnaround on the fix.
Tested-by: Colin Ian King <colin.king@canonical.com>
Colin
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: [PATCH] block: loop: fix deadlock between open and remove
2021-06-05 14:09 Christoph Hellwig
2021-06-05 15:00 ` Colin Ian King
@ 2021-06-07 0:11 ` Ming Lei
2021-06-11 17:51 ` Jens Axboe
2 siblings, 0 replies; 7+ messages in thread
From: Ming Lei @ 2021-06-07 0:11 UTC (permalink / raw)
To: Christoph Hellwig; +Cc: axboe, pasha.tatashin, linux-block, Colin Ian King
On Sat, Jun 05, 2021 at 05:09:50PM +0300, Christoph Hellwig wrote:
> Commit c76f48eb5c08 ("block: take bd_mutex around delete_partitions in
> del_gendisk") adds disk->part0->bd_mutex in del_gendisk(), this way
> causes the following AB/BA deadlock between removing loop and opening
> loop:
>
> 1) loop_control_ioctl(LOOP_CTL_REMOVE)
> -> mutex_lock(&loop_ctl_mutex)
> -> del_gendisk
> -> mutex_lock(&disk->part0->bd_mutex)
>
> 2) blkdev_get_by_dev
> -> mutex_lock(&disk->part0->bd_mutex)
> -> lo_open
> -> mutex_lock(&loop_ctl_mutex)
>
> Add a new Lo_deleting state to remove the need for clearing
> ->private_data and thus holding loop_ctl_mutex in the ioctl
> LOOP_CTL_REMOVE path.
>
> Based on an analysis and earlier patch from
> Ming Lei <ming.lei@redhat.com>.
>
> Reported-by: Colin Ian King <colin.king@canonical.com>
> Fixes: c76f48eb5c08 ("block: take bd_mutex around delete_partitions in del_gendisk")
> Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Ming Lei <ming.lei@redhat.com>
--
Ming
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: [PATCH] block: loop: fix deadlock between open and remove
2021-06-05 14:09 Christoph Hellwig
2021-06-05 15:00 ` Colin Ian King
2021-06-07 0:11 ` Ming Lei
@ 2021-06-11 17:51 ` Jens Axboe
2 siblings, 0 replies; 7+ messages in thread
From: Jens Axboe @ 2021-06-11 17:51 UTC (permalink / raw)
To: Christoph Hellwig; +Cc: ming.lei, pasha.tatashin, linux-block, Colin Ian King
On 6/5/21 8:09 AM, Christoph Hellwig wrote:
> Commit c76f48eb5c08 ("block: take bd_mutex around delete_partitions in
> del_gendisk") adds disk->part0->bd_mutex in del_gendisk(), this way
> causes the following AB/BA deadlock between removing loop and opening
> loop:
>
> 1) loop_control_ioctl(LOOP_CTL_REMOVE)
> -> mutex_lock(&loop_ctl_mutex)
> -> del_gendisk
> -> mutex_lock(&disk->part0->bd_mutex)
>
> 2) blkdev_get_by_dev
> -> mutex_lock(&disk->part0->bd_mutex)
> -> lo_open
> -> mutex_lock(&loop_ctl_mutex)
>
> Add a new Lo_deleting state to remove the need for clearing
> ->private_data and thus holding loop_ctl_mutex in the ioctl
> LOOP_CTL_REMOVE path.
>
> Based on an analysis and earlier patch from
> Ming Lei <ming.lei@redhat.com>.
Applied, thanks.
--
Jens Axboe
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2021-06-11 17:51 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-06-04 0:04 [PATCH] block: loop: fix deadlock between open and remove Ming Lei
2021-06-04 6:57 ` Christoph Hellwig
2021-06-04 8:56 ` Ming Lei
2021-06-05 14:09 Christoph Hellwig
2021-06-05 15:00 ` Colin Ian King
2021-06-07 0:11 ` Ming Lei
2021-06-11 17:51 ` Jens Axboe
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).