* [PATCH v3 1/6] dm: fix bio splitting and its bio completion order for regular IO
2020-09-22 2:32 [PATCH v3 0/6] dm: fix then improve bio splitting Mike Snitzer
@ 2020-09-22 2:32 ` Mike Snitzer
2020-09-22 2:32 ` [PATCH v3 2/6] dm: fix comment in dm_process_bio() Mike Snitzer
` (5 subsequent siblings)
6 siblings, 0 replies; 10+ messages in thread
From: Mike Snitzer @ 2020-09-22 2:32 UTC (permalink / raw)
To: Jens Axboe; +Cc: Ming Lei, Vijayendra Suman, dm-devel, linux-block
dm_queue_split() is removed because __split_and_process_bio() _must_
handle splitting bios to ensure proper bio submission and completion
ordering as a bio is split.
Otherwise, multiple recursive calls to ->submit_bio will cause multiple
split bios to be allocated from the same ->bio_split mempool at the same
time. This would result in deadlock in low memory conditions because no
progress could be made (only one bio is available in ->bio_split
mempool).
This fix has been verified to still fix the loss of performance, due
to excess splitting, that commit 120c9257f5f1 provided.
Fixes: 120c9257f5f1 ("Revert "dm: always call blk_queue_split() in dm_process_bio()"")
Cc: stable@vger.kernel.org # 5.0+, requires custom backport due to 5.9 changes
Reported-by: Ming Lei <ming.lei@redhat.com>
Signed-off-by: Mike Snitzer <snitzer@redhat.com>
---
drivers/md/dm.c | 23 ++---------------------
1 file changed, 2 insertions(+), 21 deletions(-)
diff --git a/drivers/md/dm.c b/drivers/md/dm.c
index 4a40df8af7d3..d948cd522431 100644
--- a/drivers/md/dm.c
+++ b/drivers/md/dm.c
@@ -1724,23 +1724,6 @@ static blk_qc_t __process_bio(struct mapped_device *md, struct dm_table *map,
return ret;
}
-static void dm_queue_split(struct mapped_device *md, struct dm_target *ti, struct bio **bio)
-{
- unsigned len, sector_count;
-
- sector_count = bio_sectors(*bio);
- len = min_t(sector_t, max_io_len((*bio)->bi_iter.bi_sector, ti), sector_count);
-
- if (sector_count > len) {
- struct bio *split = bio_split(*bio, len, GFP_NOIO, &md->queue->bio_split);
-
- bio_chain(split, *bio);
- trace_block_split(md->queue, split, (*bio)->bi_iter.bi_sector);
- submit_bio_noacct(*bio);
- *bio = split;
- }
-}
-
static blk_qc_t dm_process_bio(struct mapped_device *md,
struct dm_table *map, struct bio *bio)
{
@@ -1768,14 +1751,12 @@ static blk_qc_t dm_process_bio(struct mapped_device *md,
if (current->bio_list) {
if (is_abnormal_io(bio))
blk_queue_split(&bio);
- else
- dm_queue_split(md, ti, &bio);
+ /* regular IO is split by __split_and_process_bio */
}
if (dm_get_md_type(md) == DM_TYPE_NVME_BIO_BASED)
return __process_bio(md, map, bio, ti);
- else
- return __split_and_process_bio(md, map, bio);
+ return __split_and_process_bio(md, map, bio);
}
static blk_qc_t dm_submit_bio(struct bio *bio)
--
2.15.0
^ permalink raw reply related [flat|nested] 10+ messages in thread* [PATCH v3 2/6] dm: fix comment in dm_process_bio()
2020-09-22 2:32 [PATCH v3 0/6] dm: fix then improve bio splitting Mike Snitzer
2020-09-22 2:32 ` [PATCH v3 1/6] dm: fix bio splitting and its bio completion order for regular IO Mike Snitzer
@ 2020-09-22 2:32 ` Mike Snitzer
2020-09-22 2:32 ` [PATCH v3 3/6] block: use lcm_not_zero() when stacking chunk_sectors Mike Snitzer
` (4 subsequent siblings)
6 siblings, 0 replies; 10+ messages in thread
From: Mike Snitzer @ 2020-09-22 2:32 UTC (permalink / raw)
To: Jens Axboe; +Cc: Ming Lei, Vijayendra Suman, dm-devel, linux-block
Refer to the correct function (->submit_bio instead of ->queue_bio).
Also, add details about why using blk_queue_split() isn't needed for
dm_wq_work()'s call to dm_process_bio().
Fixes: c62b37d96b6eb ("block: move ->make_request_fn to struct block_device_operations")
Signed-off-by: Mike Snitzer <snitzer@redhat.com>
---
drivers/md/dm.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/drivers/md/dm.c b/drivers/md/dm.c
index d948cd522431..6ed05ca65a0f 100644
--- a/drivers/md/dm.c
+++ b/drivers/md/dm.c
@@ -1744,9 +1744,11 @@ static blk_qc_t dm_process_bio(struct mapped_device *md,
}
/*
- * If in ->queue_bio we need to use blk_queue_split(), otherwise
+ * If in ->submit_bio we need to use blk_queue_split(), otherwise
* queue_limits for abnormal requests (e.g. discard, writesame, etc)
* won't be imposed.
+ * If called from dm_wq_work() for deferred bio processing, bio
+ * was already handled by following code with previous ->submit_bio.
*/
if (current->bio_list) {
if (is_abnormal_io(bio))
--
2.15.0
^ permalink raw reply related [flat|nested] 10+ messages in thread* [PATCH v3 3/6] block: use lcm_not_zero() when stacking chunk_sectors
2020-09-22 2:32 [PATCH v3 0/6] dm: fix then improve bio splitting Mike Snitzer
2020-09-22 2:32 ` [PATCH v3 1/6] dm: fix bio splitting and its bio completion order for regular IO Mike Snitzer
2020-09-22 2:32 ` [PATCH v3 2/6] dm: fix comment in dm_process_bio() Mike Snitzer
@ 2020-09-22 2:32 ` Mike Snitzer
2020-09-22 19:28 ` Martin K. Petersen
2020-09-22 2:32 ` [PATCH v3 4/6] block: allow 'chunk_sectors' to be non-power-of-2 Mike Snitzer
` (3 subsequent siblings)
6 siblings, 1 reply; 10+ messages in thread
From: Mike Snitzer @ 2020-09-22 2:32 UTC (permalink / raw)
To: Jens Axboe; +Cc: Ming Lei, Vijayendra Suman, dm-devel, linux-block
Like 'io_opt', blk_stack_limits() should stack 'chunk_sectors' using
lcm_not_zero() rather than min_not_zero() -- otherwise the final
'chunk_sectors' could result in sub-optimal alignment of IO to
component devices in the IO stack.
Also, if 'chunk_sectors' isn't a multiple of 'physical_block_size'
then it is a bug in the driver and the device should be flagged as
'misaligned'.
Signed-off-by: Mike Snitzer <snitzer@redhat.com>
Reviewed-by: Ming Lei <ming.lei@redhat.com>
---
block/blk-settings.c | 12 ++++++++----
1 file changed, 8 insertions(+), 4 deletions(-)
diff --git a/block/blk-settings.c b/block/blk-settings.c
index 76a7e03bcd6c..b2e1a929a6db 100644
--- a/block/blk-settings.c
+++ b/block/blk-settings.c
@@ -534,6 +534,7 @@ int blk_stack_limits(struct queue_limits *t, struct queue_limits *b,
t->io_min = max(t->io_min, b->io_min);
t->io_opt = lcm_not_zero(t->io_opt, b->io_opt);
+ t->chunk_sectors = lcm_not_zero(t->chunk_sectors, b->chunk_sectors);
/* Physical block size a multiple of the logical block size? */
if (t->physical_block_size & (t->logical_block_size - 1)) {
@@ -556,6 +557,13 @@ int blk_stack_limits(struct queue_limits *t, struct queue_limits *b,
ret = -1;
}
+ /* chunk_sectors a multiple of the physical block size? */
+ if ((t->chunk_sectors << 9) & (t->physical_block_size - 1)) {
+ t->chunk_sectors = 0;
+ t->misaligned = 1;
+ ret = -1;
+ }
+
t->raid_partial_stripes_expensive =
max(t->raid_partial_stripes_expensive,
b->raid_partial_stripes_expensive);
@@ -594,10 +602,6 @@ int blk_stack_limits(struct queue_limits *t, struct queue_limits *b,
t->discard_granularity;
}
- if (b->chunk_sectors)
- t->chunk_sectors = min_not_zero(t->chunk_sectors,
- b->chunk_sectors);
-
t->zoned = max(t->zoned, b->zoned);
return ret;
}
--
2.15.0
^ permalink raw reply related [flat|nested] 10+ messages in thread* Re: [PATCH v3 3/6] block: use lcm_not_zero() when stacking chunk_sectors
2020-09-22 2:32 ` [PATCH v3 3/6] block: use lcm_not_zero() when stacking chunk_sectors Mike Snitzer
@ 2020-09-22 19:28 ` Martin K. Petersen
0 siblings, 0 replies; 10+ messages in thread
From: Martin K. Petersen @ 2020-09-22 19:28 UTC (permalink / raw)
To: Mike Snitzer
Cc: Jens Axboe, Ming Lei, Vijayendra Suman, dm-devel, linux-block
Mike,
> Like 'io_opt', blk_stack_limits() should stack 'chunk_sectors' using
> lcm_not_zero() rather than min_not_zero() -- otherwise the final
> 'chunk_sectors' could result in sub-optimal alignment of IO to
> component devices in the IO stack.
>
> Also, if 'chunk_sectors' isn't a multiple of 'physical_block_size'
> then it is a bug in the driver and the device should be flagged as
> 'misaligned'.
Looks good.
Reviewed-by: Martin K. Petersen <martin.petersen@oracle.com>
--
Martin K. Petersen Oracle Linux Engineering
^ permalink raw reply [flat|nested] 10+ messages in thread
* [PATCH v3 4/6] block: allow 'chunk_sectors' to be non-power-of-2
2020-09-22 2:32 [PATCH v3 0/6] dm: fix then improve bio splitting Mike Snitzer
` (2 preceding siblings ...)
2020-09-22 2:32 ` [PATCH v3 3/6] block: use lcm_not_zero() when stacking chunk_sectors Mike Snitzer
@ 2020-09-22 2:32 ` Mike Snitzer
2020-09-22 19:30 ` Martin K. Petersen
2020-09-22 2:32 ` [PATCH v3 5/6] dm table: stack 'chunk_sectors' limit to account for target-specific splitting Mike Snitzer
` (2 subsequent siblings)
6 siblings, 1 reply; 10+ messages in thread
From: Mike Snitzer @ 2020-09-22 2:32 UTC (permalink / raw)
To: Jens Axboe; +Cc: Ming Lei, Vijayendra Suman, dm-devel, linux-block
It is possible, albeit more unlikely, for a block device to have a non
power-of-2 for chunk_sectors (e.g. 10+2 RAID6 with 128K chunk_sectors,
which results in a full-stripe size of 1280K. This causes the RAID6's
io_opt to be advertised as 1280K, and a stacked device _could_ then be
made to use a blocksize, aka chunk_sectors, that matches non power-of-2
io_opt of underlying RAID6 -- resulting in stacked device's
chunk_sectors being a non power-of-2).
Update blk_queue_chunk_sectors() and blk_max_size_offset() to
accommodate drivers that need a non power-of-2 chunk_sectors.
Signed-off-by: Mike Snitzer <snitzer@redhat.com>
Reviewed-by: Ming Lei <ming.lei@redhat.com>
---
block/blk-settings.c | 10 ++++------
include/linux/blkdev.h | 12 +++++++++---
2 files changed, 13 insertions(+), 9 deletions(-)
diff --git a/block/blk-settings.c b/block/blk-settings.c
index b2e1a929a6db..5ea3de48afba 100644
--- a/block/blk-settings.c
+++ b/block/blk-settings.c
@@ -172,15 +172,13 @@ EXPORT_SYMBOL(blk_queue_max_hw_sectors);
*
* Description:
* If a driver doesn't want IOs to cross a given chunk size, it can set
- * this limit and prevent merging across chunks. Note that the chunk size
- * must currently be a power-of-2 in sectors. Also note that the block
- * layer must accept a page worth of data at any offset. So if the
- * crossing of chunks is a hard limitation in the driver, it must still be
- * prepared to split single page bios.
+ * this limit and prevent merging across chunks. Note that the block layer
+ * must accept a page worth of data at any offset. So if the crossing of
+ * chunks is a hard limitation in the driver, it must still be prepared
+ * to split single page bios.
**/
void blk_queue_chunk_sectors(struct request_queue *q, unsigned int chunk_sectors)
{
- BUG_ON(!is_power_of_2(chunk_sectors));
q->limits.chunk_sectors = chunk_sectors;
}
EXPORT_SYMBOL(blk_queue_chunk_sectors);
diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h
index bb5636cc17b9..51d98a595943 100644
--- a/include/linux/blkdev.h
+++ b/include/linux/blkdev.h
@@ -1059,11 +1059,17 @@ static inline unsigned int blk_queue_get_max_sectors(struct request_queue *q,
static inline unsigned int blk_max_size_offset(struct request_queue *q,
sector_t offset)
{
- if (!q->limits.chunk_sectors)
+ unsigned int chunk_sectors = q->limits.chunk_sectors;
+
+ if (!chunk_sectors)
return q->limits.max_sectors;
- return min(q->limits.max_sectors, (unsigned int)(q->limits.chunk_sectors -
- (offset & (q->limits.chunk_sectors - 1))));
+ if (likely(is_power_of_2(chunk_sectors)))
+ chunk_sectors -= offset & (chunk_sectors - 1);
+ else
+ chunk_sectors -= sector_div(offset, chunk_sectors);
+
+ return min(q->limits.max_sectors, chunk_sectors);
}
static inline unsigned int blk_rq_get_max_sectors(struct request *rq,
--
2.15.0
^ permalink raw reply related [flat|nested] 10+ messages in thread* Re: [PATCH v3 4/6] block: allow 'chunk_sectors' to be non-power-of-2
2020-09-22 2:32 ` [PATCH v3 4/6] block: allow 'chunk_sectors' to be non-power-of-2 Mike Snitzer
@ 2020-09-22 19:30 ` Martin K. Petersen
0 siblings, 0 replies; 10+ messages in thread
From: Martin K. Petersen @ 2020-09-22 19:30 UTC (permalink / raw)
To: Mike Snitzer
Cc: Jens Axboe, Ming Lei, Vijayendra Suman, dm-devel, linux-block
Mike,
> It is possible, albeit more unlikely, for a block device to have a non
> power-of-2 for chunk_sectors (e.g. 10+2 RAID6 with 128K chunk_sectors,
> which results in a full-stripe size of 1280K. This causes the RAID6's
> io_opt to be advertised as 1280K, and a stacked device _could_ then be
> made to use a blocksize, aka chunk_sectors, that matches non
> power-of-2 io_opt of underlying RAID6 -- resulting in stacked device's
> chunk_sectors being a non power-of-2).
>
> Update blk_queue_chunk_sectors() and blk_max_size_offset() to
> accommodate drivers that need a non power-of-2 chunk_sectors.
Reviewed-by: Martin K. Petersen <martin.petersen@oracle.com>
--
Martin K. Petersen Oracle Linux Engineering
^ permalink raw reply [flat|nested] 10+ messages in thread
* [PATCH v3 5/6] dm table: stack 'chunk_sectors' limit to account for target-specific splitting
2020-09-22 2:32 [PATCH v3 0/6] dm: fix then improve bio splitting Mike Snitzer
` (3 preceding siblings ...)
2020-09-22 2:32 ` [PATCH v3 4/6] block: allow 'chunk_sectors' to be non-power-of-2 Mike Snitzer
@ 2020-09-22 2:32 ` Mike Snitzer
2020-09-22 2:32 ` [PATCH v3 6/6] dm: change max_io_len() to use blk_max_size_offset() Mike Snitzer
2020-09-23 16:38 ` [PATCH v3 0/6] dm: fix then improve bio splitting Jens Axboe
6 siblings, 0 replies; 10+ messages in thread
From: Mike Snitzer @ 2020-09-22 2:32 UTC (permalink / raw)
To: Jens Axboe; +Cc: Ming Lei, Vijayendra Suman, dm-devel, linux-block
If target set ti->max_io_len it must be used when stacking
DM device's queue_limits to establish a 'chunk_sectors' that is
compatible with the IO stack.
By using lcm_not_zero() care is taken to avoid blindly overriding the
chunk_sectors limit stacked up by blk_stack_limits().
Signed-off-by: Mike Snitzer <snitzer@redhat.com>
---
drivers/md/dm-table.c | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/drivers/md/dm-table.c b/drivers/md/dm-table.c
index 229f461e7def..3f4e7c7912a2 100644
--- a/drivers/md/dm-table.c
+++ b/drivers/md/dm-table.c
@@ -18,6 +18,7 @@
#include <linux/mutex.h>
#include <linux/delay.h>
#include <linux/atomic.h>
+#include <linux/lcm.h>
#include <linux/blk-mq.h>
#include <linux/mount.h>
#include <linux/dax.h>
@@ -1506,6 +1507,10 @@ int dm_calculate_queue_limits(struct dm_table *table,
zone_sectors = ti_limits.chunk_sectors;
}
+ /* Stack chunk_sectors if target-specific splitting is required */
+ if (ti->max_io_len)
+ ti_limits.chunk_sectors = lcm_not_zero(ti->max_io_len,
+ ti_limits.chunk_sectors);
/* Set I/O hints portion of queue limits */
if (ti->type->io_hints)
ti->type->io_hints(ti, &ti_limits);
--
2.15.0
^ permalink raw reply related [flat|nested] 10+ messages in thread* [PATCH v3 6/6] dm: change max_io_len() to use blk_max_size_offset()
2020-09-22 2:32 [PATCH v3 0/6] dm: fix then improve bio splitting Mike Snitzer
` (4 preceding siblings ...)
2020-09-22 2:32 ` [PATCH v3 5/6] dm table: stack 'chunk_sectors' limit to account for target-specific splitting Mike Snitzer
@ 2020-09-22 2:32 ` Mike Snitzer
2020-09-23 16:38 ` [PATCH v3 0/6] dm: fix then improve bio splitting Jens Axboe
6 siblings, 0 replies; 10+ messages in thread
From: Mike Snitzer @ 2020-09-22 2:32 UTC (permalink / raw)
To: Jens Axboe; +Cc: Ming Lei, Vijayendra Suman, dm-devel, linux-block
Using blk_max_size_offset() enables DM core's splitting to impose
ti->max_io_len (via q->limits.chunk_sectors) and also fallback to
respecting q->limits.max_sectors if chunk_sectors isn't set.
Signed-off-by: Mike Snitzer <snitzer@redhat.com>
---
drivers/md/dm.c | 20 ++++++++------------
1 file changed, 8 insertions(+), 12 deletions(-)
diff --git a/drivers/md/dm.c b/drivers/md/dm.c
index 6ed05ca65a0f..3982012b1309 100644
--- a/drivers/md/dm.c
+++ b/drivers/md/dm.c
@@ -1051,22 +1051,18 @@ static sector_t max_io_len_target_boundary(sector_t sector, struct dm_target *ti
static sector_t max_io_len(sector_t sector, struct dm_target *ti)
{
sector_t len = max_io_len_target_boundary(sector, ti);
- sector_t offset, max_len;
+ sector_t max_len;
/*
* Does the target need to split even further?
+ * - q->limits.chunk_sectors reflects ti->max_io_len so
+ * blk_max_size_offset() provides required splitting.
+ * - blk_max_size_offset() also respects q->limits.max_sectors
*/
- if (ti->max_io_len) {
- offset = dm_target_offset(ti, sector);
- if (unlikely(ti->max_io_len & (ti->max_io_len - 1)))
- max_len = sector_div(offset, ti->max_io_len);
- else
- max_len = offset & (ti->max_io_len - 1);
- max_len = ti->max_io_len - max_len;
-
- if (len > max_len)
- len = max_len;
- }
+ max_len = blk_max_size_offset(dm_table_get_md(ti->table)->queue,
+ dm_target_offset(ti, sector));
+ if (len > max_len)
+ len = max_len;
return len;
}
--
2.15.0
^ permalink raw reply related [flat|nested] 10+ messages in thread* Re: [PATCH v3 0/6] dm: fix then improve bio splitting
2020-09-22 2:32 [PATCH v3 0/6] dm: fix then improve bio splitting Mike Snitzer
` (5 preceding siblings ...)
2020-09-22 2:32 ` [PATCH v3 6/6] dm: change max_io_len() to use blk_max_size_offset() Mike Snitzer
@ 2020-09-23 16:38 ` Jens Axboe
6 siblings, 0 replies; 10+ messages in thread
From: Jens Axboe @ 2020-09-23 16:38 UTC (permalink / raw)
To: Mike Snitzer; +Cc: Ming Lei, Vijayendra Suman, dm-devel, linux-block
On 9/21/20 8:32 PM, Mike Snitzer wrote:
> Hi,
>
> Patches 1 and 2 are queued for me to send to Linus later this week.
>
> Patches 3 and 4 are block core and should get picked up for 5.10.
> Jens, please pick them up. I revised the header for patch 4 to give
> better context for use-case where non power-of-2 chunk_sectors
> occurs. Patch 4 enables DM to switch to using blk_max_size_offset() in
> Patch 6.
>
> Patches 5 and 6 just show how DM will be enhanced for 5.10 once
> patches 3 and 4 land in the block tree.
Applied 3-4 for 5.10, thanks.
--
Jens Axboe
^ permalink raw reply [flat|nested] 10+ messages in thread