Re: [PATCH] bcache: make stripe_size configurable and persistent for hardware raid5/6

From: Eric Wheeler <bcache@lists.ewheeler.net>
To: "Martin K. Petersen" <martin.petersen@oracle.com>
Cc: Coly Li <colyli@suse.de>,
	linux-block@vger.kernel.org, Jonathan Corbet <corbet@lwn.net>,
	Kent Overstreet <kent.overstreet@gmail.com>,
	"open list:DOCUMENTATION" <linux-doc@vger.kernel.org>,
	open list <linux-kernel@vger.kernel.org>,
	"open list:BCACHE (BLOCK LAYER CACHE)"
	<linux-bcache@vger.kernel.org>
Subject: Re: [PATCH] bcache: make stripe_size configurable and persistent for hardware raid5/6
Date: Fri, 7 Jan 2022 20:54:16 -0800 (PST)	[thread overview]
Message-ID: <c9abd220-6b7f-9299-48a1-a16d64981734@ewheeler.net> (raw)
In-Reply-To: <yq15yqvw1f0.fsf@ca-mkp.ca.oracle.com>

On Fri, 7 Jan 2022, Martin K. Petersen wrote:
> Eric,
> 
> > Even new new RAID controlers that _do_ provide `io_opt` still do _not_ 
> > indicate partial_stripes_expensive (which is an mdraid feature, but Martin 
> > please correct me if I'm wrong here).
> 
> partial_stripes_expensive is a bcache thing, I am not sure why it needs
> a separate flag. It is implied, although I guess one could argue that
> RAID0 is a special case since partial writes are not as painful as with
> parity RAID.

I'm guessing bcache used did some optimization for 
queue->limits.raid_partial_stripes_expensive because md raid5 code sets 
this flag.  At least when using Linux md as the RAID5 implementation it 
gets configured automatically:
   raid5.c:       mddev->queue->limits.raid_partial_stripes_expensive = 1;

https://elixir.bootlin.com/linux/latest/source/drivers/md/raid5.c#L7729

Interestingly only bcache uses it, but md does set it.

> The SCSI spec states that submitting an I/O that is smaller than io_min
> "may incur delays in processing the command". And similarly, submitting
> a command larger than io_opt "may incur delays in processing the
> command".
> 
> IOW, the spec says "don't write less than an aligned multiple of the
> stripe chunk size" and "don't write more than an aligned full
> stripe". That leaves "aligned multiples of the stripe chunk size but
> less than the full stripe width" unaccounted for. And I guess that's
> what the bcache flag is trying to capture.

Maybe any time io_opt is provided then partial_stripes_expensive should be 
flagged too and any code to the contrary should be removed?

Question: Does anyone have a reason to keep partial_stripes_expensive in 
the kernel at all?

> SCSI doesn't go into details about RAID levels and other implementation
> details which is why the wording is deliberately vague. But obviously
> the expectation is that partial stripe writes are slower than full.
> 
> In my book any component in the stack that sees either io_min or io_opt
> should try very hard to send I/Os that are aligned multiples of those
> values. I am not opposed to letting users manually twiddle the
> settings. But I do think that we should aim for the stack doing the
> right thing when it sees io_opt reported on a device.

Agreed, thanks for the feedback!

-Eric

> 
> -- 
> Martin K. Petersen	Oracle Linux Engineering
> 