[RFC PATCH V3 0/1] block: fix I/O errors in BLKRRPART

[RFC PATCH V3 0/1] block: fix I/O errors in BLKRRPART

[Minwoo Im]

Hello,

  This patch fixes I/O errors during BLKRRPART ioctl() behavior right
after format operation that changed logical block size of the block
device with a same file descriptor opened.

Testcase:

  The following testcase is a case of NVMe namespace with the following
conditions:

  - Current LBA format is lbaf=0 (512 bytes logical block size)
  - LBA Format(lbaf=1) has 4096 bytes logical block size

  # Format block device logical block size 512B to 4096B
  nvme format /dev/nvme0n1 --lbaf=1 --force

  This will cause I/O errors because BLKRRPART ioctl() happened right after
the format command with same file descriptor opened in application
(e.g., nvme-cli) like:

  fd = open("/dev/nvme0n1", O_RDONLY);

  nvme_format(fd, ...);
  if (ioctl(fd, BLKRRPART) < 0)
	...

Errors:

  We can see the Read command with Number of LBA(NLB) 0xffff(65535) which
was under-flowed because BLKRRPART operation requested request size based
on i_blkbits of the block device which is 9 via buffer_head.

  [dmesg-snip]                                                                                                                              
    [   10.771740] blk_update_request: operation not supported error, dev nvme0n1, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
    [   10.780262] Buffer I/O error on dev nvme0n1, logical block 0, async page read

  [event-snip]                                                                                                                                                                                           
    kworker/0:1H-56      [000] ....   913.456922: nvme_setup_cmd: nvme0: disk=nvme0n1, qid=1, cmdid=216, nsid=1, flags=0x0, meta=0x0, cmd=(nvme_cmd_read slba=0, len=65535, ctrl=0x0, dsmgmt=0, reftag=0)
     ksoftirqd/0-9       [000] .Ns.   916.566351: nvme_complete_rq: nvme0: disk=nvme0n1, qid=1, cmdid=216, res=0x0, retries=0, flags=0x0, status=0x4002

  The patch below fixes the I/O errors by rejecting I/O requests from the
block layer with setting a flag to gendisk until the file descriptor
re-opened to be updated by __blkdev_get().  This is based on the previous
discussion [1].

Since V2:
  - Cover letter with testcase and error logs attached. Removed un-related
    changes: empty line. (Chaitanya, [2])
  - Put blkdev with blkdev_put_no_open().

Since V1:
  - Updated patch to reject I/O rather than updating i_blkbits of the
    block device's inode directly from driver. (Christoph, [1])

[1] https://lore.kernel.org/linux-nvme/20201223183143.GB13354@localhost.localdomain/T/#t
[2] https://lore.kernel.org/linux-nvme/20201230140504.GB7917@localhost.localdomain/T/#t

Thanks,

Minwoo Im (1):
  block: reject I/O for same fd if block size changed

 block/blk-settings.c    |  8 ++++++++
 block/partitions/core.c | 11 +++++++++++
 fs/block_dev.c          |  6 ++++++
 include/linux/genhd.h   |  1 +
 4 files changed, 26 insertions(+)

-- 
2.17.1

[RFC PATCH V3 1/1] block: reject I/O for same fd if block size changed

[Minwoo Im]

This patch fixes I/O errors during BLKRRPART ioctl() behavior right
after format operation that changed logical block size of the block
device with a same file descriptor opened.

This issue can be easily reproduced with a single format command in case
of NVMe (logical block size 512B to 4096B).

	nvme format /dev/nvme0n1 --lbaf=1 --force

This is because the application, nvme-cli format subcommand issues an
admin command followed by BLKRRPART ioctl to re-read partition
information without closing the file descriptor.  If file descriptor
stays opened, __blkdev_get() will not be invoked at all even logical
block size has been changed.

It will cause I/O errors with invalid Read operations during the
BLKRRPART ioctl due to i_blkbits mismatch. The invalid operations in
BLKRRPART happens with under-flowed Number of LBA(NLB) values
0xffff(65535) because i_blkbits is still set to 9 even the logical block
size has been updated to 4096.  The BLKRRPART will lead buffer_head to
hold 512B data which is less than the logical lock size of the block
device.

The root cause, which is because i_blkbits of inode of the block device
is not updated, can be solved easily by re-opening file descriptor
again from application.  But, that's just for application's business
and kernel should reject invalid Read operations during the BLKRRPART
ioctl.

This patch rejects I/O from the path of add_partitions() to avoid
issuing invalid Read operations to device.  It also sets a flag to
gendisk in blk_queue_logical_block_size to minimize caller-side updates.

Signed-off-by: Minwoo Im <minwoo.im.dev@gmail.com>
---
 block/blk-settings.c    |  8 ++++++++
 block/partitions/core.c | 11 +++++++++++
 fs/block_dev.c          |  6 ++++++
 include/linux/genhd.h   |  1 +
 4 files changed, 26 insertions(+)

diff --git a/block/blk-settings.c b/block/blk-settings.c
index 43990b1d148b..84136ea4e2a4 100644
--- a/block/blk-settings.c
+++ b/block/blk-settings.c
@@ -328,6 +328,14 @@ EXPORT_SYMBOL(blk_queue_max_segment_size);
 void blk_queue_logical_block_size(struct request_queue *q, unsigned int size)
 {
 	struct queue_limits *limits = &q->limits;
+	struct block_device *bdev;
+
+	if (q->backing_dev_info && q->backing_dev_info->owner &&
+			limits->logical_block_size != size) {
+		bdev = blkdev_get_no_open(q->backing_dev_info->owner->devt);
+		bdev->bd_disk->flags |= GENHD_FL_BLOCK_SIZE_CHANGED;
+		blkdev_put_no_open(bdev);
+	}
 
 	limits->logical_block_size = size;
 
diff --git a/block/partitions/core.c b/block/partitions/core.c
index e7d776db803b..5a0330c1b6f9 100644
--- a/block/partitions/core.c
+++ b/block/partitions/core.c
@@ -618,6 +618,17 @@ int blk_add_partitions(struct gendisk *disk, struct block_device *bdev)
 	if (!disk_part_scan_enabled(disk))
 		return 0;
 
+	/*
+	 * Reject to check partition information if block size has been changed
+	 * in the runtime.  If block size of a block device has been changed,
+	 * the file descriptor should be opened agian to update the blkbits.
+	 */
+	if (disk->flags & GENHD_FL_BLOCK_SIZE_CHANGED) {
+		pr_warn("%s: rejecting checking partition. fd should be opened again.\n",
+				disk->disk_name);
+		return -EBADFD;
+	}
+
 	state = check_partition(disk, bdev);
 	if (!state)
 		return 0;
diff --git a/fs/block_dev.c b/fs/block_dev.c
index 9293045e128c..c996de3d6084 100644
--- a/fs/block_dev.c
+++ b/fs/block_dev.c
@@ -131,6 +131,12 @@ EXPORT_SYMBOL(truncate_bdev_range);
 static void set_init_blocksize(struct block_device *bdev)
 {
 	bdev->bd_inode->i_blkbits = blksize_bits(bdev_logical_block_size(bdev));
+
+	/*
+	 * Allow I/O commands for this block device.  We can say that this
+	 * block device has been set to a proper block size.
+	 */
+	bdev->bd_disk->flags &= ~GENHD_FL_BLOCK_SIZE_CHANGED;
 }
 
 int set_blocksize(struct block_device *bdev, int size)
diff --git a/include/linux/genhd.h b/include/linux/genhd.h
index 809aaa32d53c..0e0e24917003 100644
--- a/include/linux/genhd.h
+++ b/include/linux/genhd.h
@@ -103,6 +103,7 @@ struct partition_meta_info {
 #define GENHD_FL_BLOCK_EVENTS_ON_EXCL_WRITE	0x0100
 #define GENHD_FL_NO_PART_SCAN			0x0200
 #define GENHD_FL_HIDDEN				0x0400
+#define GENHD_FL_BLOCK_SIZE_CHANGED		0x0800
 
 enum {
 	DISK_EVENT_MEDIA_CHANGE			= 1 << 0, /* media changed */
-- 
2.17.1

Re: [RFC PATCH V3 1/1] block: reject I/O for same fd if block size changed

[Christoph Hellwig]

On Mon, Jan 04, 2021 at 10:06:59PM +0900, Minwoo Im wrote:
> +	if (q->backing_dev_info && q->backing_dev_info->owner &&
> +			limits->logical_block_size != size) {
> +		bdev = blkdev_get_no_open(q->backing_dev_info->owner->devt);
> +		bdev->bd_disk->flags |= GENHD_FL_BLOCK_SIZE_CHANGED;
> +		blkdev_put_no_open(bdev);
> +	}

We really need the backpointer from the queue to the gendisk I've wanted
to add for a while.  Can we at least restrict this to a live gendisk?

Also I think the size change flag should go into the ->state field and
use the atomic bitops helpers to avoid concurrency problems.

Re: [RFC PATCH V3 1/1] block: reject I/O for same fd if block size changed

[Christoph Hellwig]

On Mon, Jan 04, 2021 at 06:11:08PM +0100, Christoph Hellwig wrote:
> On Mon, Jan 04, 2021 at 10:06:59PM +0900, Minwoo Im wrote:
> > +	if (q->backing_dev_info && q->backing_dev_info->owner &&
> > +			limits->logical_block_size != size) {
> > +		bdev = blkdev_get_no_open(q->backing_dev_info->owner->devt);
> > +		bdev->bd_disk->flags |= GENHD_FL_BLOCK_SIZE_CHANGED;
> > +		blkdev_put_no_open(bdev);
> > +	}
> 
> We really need the backpointer from the queue to the gendisk I've wanted
> to add for a while.  Can we at least restrict this to a live gendisk?

Alternatively we could make this request_queue QUEUE* flag for now.

Re: [RFC PATCH V3 1/1] block: reject I/O for same fd if block size changed

[Minwoo Im]

Hello Christoph,

Thanks for your review.

On 21-01-04 18:11:41, Christoph Hellwig wrote:
> On Mon, Jan 04, 2021 at 06:11:08PM +0100, Christoph Hellwig wrote:
> > On Mon, Jan 04, 2021 at 10:06:59PM +0900, Minwoo Im wrote:
> > > +	if (q->backing_dev_info && q->backing_dev_info->owner &&
> > > +			limits->logical_block_size != size) {
> > > +		bdev = blkdev_get_no_open(q->backing_dev_info->owner->devt);
> > > +		bdev->bd_disk->flags |= GENHD_FL_BLOCK_SIZE_CHANGED;
> > > +		blkdev_put_no_open(bdev);
> > > +	}
> > 
> > We really need the backpointer from the queue to the gendisk I've wanted
> > to add for a while.  Can we at least restrict this to a live gendisk?

It was a point that I really would like to ask by RFC whether we can
have backpointer to the gendisk from the request_queue.  And I'd like to
have it to simplify this routine and for future usages also.

I will restrict this one by checking GENHD_FL_UP flag from the gendisk
for the next patch.

> 
> Alternatively we could make this request_queue QUEUE* flag for now.

As this patch rejects I/O from the block layer partition code, can we
have this flag in gendisk rather than request_queue ?

Thanks,

Re: [RFC PATCH V3 1/1] block: reject I/O for same fd if block size changed

[Christoph Hellwig]

On Tue, Jan 05, 2021 at 10:04:56AM +0900, Minwoo Im wrote:
> It was a point that I really would like to ask by RFC whether we can
> have backpointer to the gendisk from the request_queue.  And I'd like to
> have it to simplify this routine and for future usages also.

I think it is the right thing to do, at least mid-term, although I
don't want to enforce the burden on you right now.

> I will restrict this one by checking GENHD_FL_UP flag from the gendisk
> for the next patch.
> 
> > 
> > Alternatively we could make this request_queue QUEUE* flag for now.
> 
> As this patch rejects I/O from the block layer partition code, can we
> have this flag in gendisk rather than request_queue ?

For now we can as the request_queue is required.  I have some plans to
clean up this area, but just using a request_queue flag for now is
probably the simplest, even if it means more work for me later.

Re: [RFC PATCH V3 1/1] block: reject I/O for same fd if block size changed

[Minwoo Im]

Hello,

On 21-01-05 08:50:09, Christoph Hellwig wrote:
> On Tue, Jan 05, 2021 at 10:04:56AM +0900, Minwoo Im wrote:
> > It was a point that I really would like to ask by RFC whether we can
> > have backpointer to the gendisk from the request_queue.  And I'd like to
> > have it to simplify this routine and for future usages also.
> 
> I think it is the right thing to do, at least mid-term, although I
> don't want to enforce the burden on you right now.
> 
> > I will restrict this one by checking GENHD_FL_UP flag from the gendisk
> > for the next patch.
> > 
> > > 
> > > Alternatively we could make this request_queue QUEUE* flag for now.
> > 
> > As this patch rejects I/O from the block layer partition code, can we
> > have this flag in gendisk rather than request_queue ?
> 
> For now we can as the request_queue is required.  I have some plans to
> clean up this area, but just using a request_queue flag for now is
> probably the simplest, even if it means more work for me later.

Please let me prepare the next quick fix for this issue with request_queue
flag.

Thanks!