From mboxrd@z Thu Jan 1 00:00:00 1970 From: Mark Nelson Subject: Re: block layer API for file system creation - when to use multidisk mode Date: Fri, 30 Nov 2018 12:05:46 -0600 Message-ID: <4125f92c-25f6-3397-19bb-61ce71c615c0@redhat.com> References: <20181004222952.GV31060@dastard> <67627995-714c-5c38-a796-32b503de7d13@sandeen.net> <20181005232710.GH12041@dastard> <20181006232037.GB18095@dastard> <36bc3f17-e7d1-ce8b-2088-36ff5d7b1e8b@sandeen.net> <0290ec9f-ab2b-7c1b-faaf-409d72f99e5f@gmail.com> <20181129214851.GU6311@dastard> <39031e68-3936-b5e1-bcb6-6fdecc5988c1@gmail.com> <20181130022510.GW6311@dastard> <3da04164-a89f-f4c0-1529-eab12b3226e1@gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8"; Format="flowed" Content-Transfer-Encoding: base64 Return-path: In-Reply-To: <3da04164-a89f-f4c0-1529-eab12b3226e1@gmail.com> Content-Language: en-US List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: dm-devel-bounces@redhat.com Errors-To: dm-devel-bounces@redhat.com To: Ric Wheeler , Dave Chinner Cc: Jens Axboe , Eric Sandeen , Mike Snitzer , "linux-scsi@vger.kernel.org" , Eric Sandeen , xfs , IDE/ATA development list , device-mapper development , Ilya Dryomov List-Id: linux-ide@vger.kernel.org Ck9uIDExLzMwLzE4IDEyOjAwIFBNLCBSaWMgV2hlZWxlciB3cm90ZToKPiBPbiAxMS8zMC8xOCA3 OjU1IEFNLCBEYXZlIENoaW5uZXIgd3JvdGU6Cj4+IE9uIFRodSwgTm92IDI5LCAyMDE4IGF0IDA2 OjUzOjE0UE0gLTA1MDAsIFJpYyBXaGVlbGVyIHdyb3RlOgo+Pj4gT24gMTEvMjkvMTggNDo0OCBQ TSwgRGF2ZSBDaGlubmVyIHdyb3RlOgo+Pj4+IE9uIFRodSwgTm92IDI5LCAyMDE4IGF0IDA4OjUz OjM5QU0gLTA1MDAsIFJpYyBXaGVlbGVyIHdyb3RlOgo+Pj4+PiBPbiAxMC82LzE4IDg6MTQgUE0s IEVyaWMgU2FuZGVlbiB3cm90ZToKPj4+Pj4+IE9uIDEwLzYvMTggNjoyMCBQTSwgRGF2ZSBDaGlu bmVyIHdyb3RlOgo+Pj4+Pj4+PiBDYW4geW91IGdpdmUgYW4gZXhhbXBsZSBvZiBhIHVzZSBjYXNl IHRoYXQgd291bGQgYmUgbmVnYXRpdmVseSAKPj4+Pj4+Pj4gYWZmZWN0ZWQKPj4+Pj4+Pj4gaWYg dGhpcyBoZXVyaXN0aWMgd2FzIHN3aXRjaGVkIGZyb20gInN1bml0IiB0byAic3VuaXQgPCBzd2lk dGgiPwo+Pj4+Pj4+IEFueSB0aW1lIHlvdSBvbmx5IGtub3cgYSBzaW5nbGUgYWxpZ25tZW50IGNo YXJhY3RlcmlzdGljIG9mIHRoZQo+Pj4+Pj4+IHVuZGVybHlpbmcgbXVsdGktZGlzayBzdG9yYWdl LiBlLmcuIGhhcmR3YXJlIFJBSUQwLzUvNiB0aGF0IHNldHMKPj4+Pj4+PiBpb21pbiA9IGlvb3B0 LCBtdWx0aS1sZXZlbCBSQUlEIGNvbnN0cnVjdHMgd2hlcmUgb25seSB0aGUgbGFyZ2VzdAo+Pj4+ Pj4+IGFsaWdubWVudCByZXF1aXJlbWVudCBpcyBleHBvc2VkLCBSQUlEMSBkZXZpY2VzIGV4cG9z aW5nIHRoZWlyIAo+Pj4+Pj4+IGNodW5rCj4+Pj4+Pj4gc2l6ZSwgcmVtb3RlIHJlcGxpY2F0aW9u IGNodW5rIGFsaWdubWVudCAoYmVjYXVzZSByZW1vdGUgcmVwLiBpcwo+Pj4+Pj4+IHNsb3cgYW5k IHNvIHdlIG5lZWQgbW9yZSBjb25jdXJyZW5jeSB0byBrZWVwIHRoZSBwaXBlbGluZSBmdWxsKSwK Pj4+Pj4+PiBldGMuCj4+Pj4+PiBTbyB0aGUgdGw7ZHIgaGVyZSBpcyAiZ2l2ZW4gYW55IGlvbWlu ID4gNTEyLCB3ZSBzaG91bGQgaW5mZXIgbG93IAo+Pj4+Pj4gc2Vlawo+Pj4+Pj4gbGF0ZW5jeSBh bmQgcGFyYWxsZWxpc20gYW5kIGFkanVzdCBnZW9tZXRyeSBhY2NvcmRpbmdseT8iCj4+Pj4+Pgo+ Pj4+Pj4gLUVyaWMKPj4+Pj4gQ2hpbWluZyBpbiBsYXRlIGhlcmUsIGJ1dCBJIGRvIHRoaW5rIHRo YXQgZXZlcnkgZGVjYWRlIG9yIHR3byAobm8KPj4+Pj4gZGlzcmVzcGVjdCB0byB4ZnMhKSwgaXQg aXMgd29ydGggaGF2aW5nIGEgc2Vjb25kIGxvb2sgYXQgaG93IHRoZQo+Pj4+PiBzdG9yYWdlIGhh cyBjaGFuZ2VkIHVuZGVyIHVzLgo+Pj4+Pgo+Pj4+PiBUaGUgd29ya2xvYWQgdGhhdCBoYXMgbG90 cyBvZiBmaWxlIHN5c3RlbXMgcG91bmRpbmcgb24gYSBzaGFyZWQKPj4+Pj4gZGV2aWNlIGZvciBl eGFtcGxlIGlzIG9uZSB3YXkgdG8gbGF5IG91dCBjb250YWluZXIgc3RvcmFnZS4KPj4+PiBUaGUg cHJvYmxlbSBpcyB0aGF0IGRlZmF1bHRzIGNhbid0IGNhdGVyIGZvciBldmVyeSB1c2UgY2FzZS4K Pj4+PiBBbmQgaW4gdGhpcyBjYXNlLCB3ZSd2ZSBnb3Qgbm90aGluZyB0byB0ZWxsIHVzIHRoYXQg dGhpcyBpcwo+Pj4+IGFnZ3JlZ2F0ZWQvc2hhcmVkIHN0b3JhZ2UgcmF0aGVyIHRoYW4gInRoZSBm aWxleXN0ZW0gb3ducyB0aGUKPj4+PiBlbnRpcmUgZGV2aWNlIi4KPj4+Pgo+Pj4+PiBObyBhcmd1 bWVudCBhYm91dCBkb2N1bWVudGluZyBob3cgdG8gZml4IHRoaXMgd2l0aCBjb21tYW5kIGxpbmUK Pj4+Pj4gdHdlYWtzIGZvciBub3csIGJ1dCBtYXliZSB0aGlzIHdvdWxkIGJlIGEgZ29vZCB0b3Bp YyBmb3IgdGhlIG5leHQKPj4+Pj4gTFNGL01NIHNoYXJlZCB0cmFjayBvZiBmaWxlICYgc3RvcmFn ZSBwZW9wbGUgdG8gZGViYXRlPwo+Pj4+IERvdWJ0IGl0IC0gdGhpcyBpcyByZWFsbHkgb25seSBh biBYRlMgcHJvYmxlbSBhdCB0aGlzIHBvaW50Lgo+Pj4+Cj4+Pj4gaS5lLiBpZiB3ZSBjYW4ndCBp bmZlciB3aGF0IHRoZSB1c2VyIHdhbnRzIGZyb20gZXhpc3RpbmcKPj4+PiBpbmZvcm1hdGlvbiwg dGhlbiBJIGRvbid0IHNlZSBob3cgdGhlIHN0b3JhZ2UgaXMgZ29pbmcgdG8gYmUgYWJsZSB0bwo+ Pj4+IHRlbGwgdXMgYW55dGhpbmcgZGlmZmVyZW50LCBlaXRoZXIuwqAgaS5lLiBzb21ld2hlcmUg aW4gdGhlIHN0YWNrIHRoZQo+Pj4+IHVzZXIgaXMgZ29pbmcgdG8gaGF2ZSB0byB0ZWxsIHRoZSBi bG9jayBkZXZpY2UgdGhhdCB0aGlzIGlzCj4+Pj4gYWdncmVnYXRlZCBzdG9yYWdlLgo+Pj4+Cj4+ Pj4gQnV0IGV2ZW4gdGhlbiwgaWYgaXQncyBhZ2dyZWdhdGVkIHNvbGlkIHN0YXRlIHN0b3JhZ2Us IHdlIHN0aWxsIHdhbnQKPj4+PiB0byBtYWtlIHVzZSBvZiB0aGUgY29uY3VyZW5jeSBvbiBpbmNy ZWFzZWQgQUcgY291bnQgYmVjYXVzZSB0aGVyZSBpcwo+Pj4+IG5vIHNlZWsgcGVuYWx0eSBsaWtl IHNwaW5uaW5nIGRyaXZlcyBlbmQgdXAgd2l0aC4gT3IgaWYgdGhlCj4+Pj4gYWdncmVnYXRlZCBz dG9yYWdlIGlzIHRoaW5seSBwcm92aXNpb25lZCwgdGhlIEFHIGNvdW50IG9mIGZpbGVzeXN0ZW0K Pj4+PiBqdXN0IGRvZXNuJ3QgbWF0dGVyIGJlY2F1c2UgdGhlIElPIGlzIGdvaW5nIHRvIGJlIG1h c3NpdmVseQo+Pj4+IHJhbmRvbWlzZWQgKGkuZSB0YWtlIHJhbmRvbSBzZWVrIHBlbmFsdGllcykg YnkgdGhlIHRoaW5wIGxheW91dC4KPj4+Pgo+Pj4+IFNvIHRoZXJlJ3MgcmVhbGx5IG5vIGdvb2Qg d2F5IG9mICJndWVzc2luZyIgd2hldGhlciBhZ2dyZWdhdGVkCj4+Pj4gc3RvcmFnZSBzaG91bGQg b3Igc2hvdWxkbid0IHVzZSBlbGV2YXRlZCBBRyBjb3VudHMgZXZlbiBpZiB0aGUKPj4+PiBzdG9y YWdlIHNheXMgInRoaXMgaXMgYWdncmVnYXRlZCBzdG9yYWdlIi4gVGhlIHVzZXIgc3RpbGwgaGFz IHRvCj4+Pj4gZ2l2ZSB1cyBzb21lIGtpbmQgb2YgZXhwbGljdCBoaW50IGFib3V0IGhvdyB0aGUg ZmlsZXN5c3RlbSBzaG91bGQKPj4+PiBiZSBjb25maWd1cmVkLgo+Pj4+Cj4+Pj4gV2hhdCB3ZSBu ZWVkIGlzIGZvciBhIHNvbGlkLCByZWxpYWJsZSBkZXRlY3Rpb24gaHVlcmlzdGljIHRvIGJlCj4+ Pj4gc3VnZ2VzdGVkIGJ5IHRoZSBwZW9wbGUgdGhhdCBuZWVkIHRoaXMgZnVuY3Rpb25hbGl0eSBi ZWZvcmUgdGhlcmUncwo+Pj4+IGFueXRoaW5nIHdlIGNhbiB0YWxrIGFib3V0Lgo+Pj4gSSB0aGlu ayB0aGF0IGlzIGV4YWN0bHkgdGhlIGtpbmQgb2YgZGlzY3Vzc2lvbiB0aGF0IHRoZSBzaGFyZWQK Pj4+IGZpbGUvc3RvcmFnZSB0cmFjayBpcyBnb29kIGZvci4KPj4gWWVzLCBidXQgd2h5IG9uIGVh cnRoIGRvIHdlIG5lZWQgdG8gd2FpdCA2IG1vbnRocyB0byBoYXZlIHRoYXQKPj4gY29udmVyc2F0 aW9uLiBTdGFydCBpdCBub3cuLi4KPgo+Cj4gU3VyZSwgdGhhdCBpcyBkZWZpbml0ZWx5IGEgZ29v ZCBpZGVhIC0gYWRkZWQgaW4gc29tZSBvZiB0aGUgc3RvcmFnZSAKPiBsaXN0cyB0byB0aGlzIHJl cGx5LiBObyBwZXJmZWN0IGFsbCBlbmNvbXBhc3NpbmcgYmxvY2sgbGF5ZXIgbGlzdCB0aGF0IAo+ IEkga25vdyBvZi4KPgo+Cj4+Cj4+PiBPdGhlciBmaWxlIHN5c3RlbXMgYWxzbyBuZWVkIHRvCj4+ PiBhY2NvbW1vZGF0ZS9wcm9iZSBiZWhpbmQgdGhlIGZpY3RpdGlvdXMgdmlzaWJsZSBzdG9yYWdl IGRldmljZQo+Pj4gbGF5ZXIuLi4gU3BlY2lmaWNhbGx5LCBpcyB0aGVyZSBzb21ldGhpbmcgd2Ug Y2FuIGFkZCBwZXIgYmxvY2sKPj4+IGRldmljZSB0byBoZWxwIGhlcmU/IE51bWJlciBvZiBpbmRl cGVuZGVudCBkZXZpY2VzCj4+IFRoYXQncyBob3cgbWtmcy54ZnMgdXNlZCB0byBkbyBzdHJpcGUg dW5pdC9zdHJpcGUgd2lkdGggY2FsY3VsYXRpb25zCj4+IGF1dG9tYXRpY2FsbHkgb24gTUQgZGV2 aWNlcyBiYWNrIGluIHRoZSAyMDAwcy4gV2UgZ290IHJpZCBvZiB0aGF0Cj4+IGZvciBtb3JlIGdl bmVyYWx5IGFwcGxpY2FibGUgY29uZmlndXJhdGlvbiBpbmZvcm1hdGlvbiBzdWNoIGFzCj4+IG1p bmltdW0vb3B0aW1hbCBJTyBzaXplcyBzbyB3ZSBjb3VsZCBleHBvc2UgZXF1aXZhbGVudCBhbGln bm1lbnQKPj4gaW5mb3JtYXRpb24gZnJvbSBsb3RzIG9mIGRpZmZlcmVudCB0eXBlcyBvZiBzdG9y YWdlIGRldmljZS4uLi4KPj4KPj4+IG9yIGEgbWFwIG9mCj4+PiB0aG9zZSByZWdpb25zPwo+PiBO b3Qgc3VyZSB3aGF0IHRoaXMgbWVhbnMgb3IgaG93IHdlJ2QgdXNlIGl0Lgo+Pgo+PiBDaGVlcnMs Cj4+Cj4+IERhdmUuCj4KPiBXaGF0IEkgd2FzIHRoaW5raW5nIG9mIHdhcyBhIHdheSBvZiBnaXZp bmcgdXAgYSBnb29kIG91dGxpbmUgb2YgaG93IAo+IG1hbnkgaW5kZXBlbmRlbnQgcmVnaW9ucyB0 aGF0IGFyZSBiZWhpbmQgb25lICJ2aXJ0dWFsIiBibG9jayBkZXZpY2UgCj4gbGlrZSBhIGNlcGgg cmJkIG9yIGRldmljZSBtYXBwZXIgZGV2aWNlLiBNeSBhc3N1bXB0aW9uIGlzIHRoYXQgd2UgYXJl IAo+IHRyeWluZyB0byBsYXkgZG93biAoYXQgbGVhc3Qgb25lKSBhbGxvY2F0aW9uIGdyb3VwIHBl ciByZWdpb24uCj4KPiBXaGF0IHdlIG5lZWQgdG8gb3B0aW1pemUgZm9yIGluY2x1ZGVzOgo+Cj4g wqDCoMKgICogaG93IG1hbnkgaW5kZXBlbmRlbnQgcmVnaW9ucyBhcmUgdGhlcmU/Cj4KPiDCoMKg wqAgKiB3aGF0IGFyZSB0aGUgYm91bmRhcmllcyBvZiB0aG9zZSByZWdpb25zPwo+Cj4gwqDCoMKg ICogb3B0aW1hbCBJTyBzaXplL2FsaWdubWVudC9ldGMKPgo+IFNvbWUgb2YgdGhhdCB3ZSBoYXZl LCBidXQgdGhlIGN1cnJlbnQgYXNzdW1wdGlvbnMgZG9uJ3Qgd29yayB3ZWxsIGZvciAKPiBhbGwg ZGV2aWNlIHR5cGVzLgo+Cj4gUmVnYXJkcywKPgo+IFJpYwo+Ckkgd29uJ3QgY29tbWVudCBvbiB0 aGUgZGV0YWlscyBhcyB0aGVyZSBhcmUgb3RoZXJzIGhlcmUgdGhhdCBhcmUgZmFyIAptb3JlIGtu b3dsZWRnZWFibGUgdGhhbiBJIGFtLCBidXQgYXQgYSBoaWdoIGxldmVsIEkgdGhpbmsgeW91ciBp ZGVhIGlzIAphYnNvbHV0ZWx5IGZhbnRhc3RpYyBmcm9tIHRoZSBzdGFuZHBvaW50IG9mIG1ha2lu ZyB0aGlzIGRlY2lzaW9uIHByb2Nlc3MgCm1vcmUgZXhwbGljaXQuCgoKTWFyawoKLS0KZG0tZGV2 ZWwgbWFpbGluZyBsaXN0CmRtLWRldmVsQHJlZGhhdC5jb20KaHR0cHM6Ly93d3cucmVkaGF0LmNv bS9tYWlsbWFuL2xpc3RpbmZvL2RtLWRldmVs From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mx1.redhat.com ([209.132.183.28]:37621 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725754AbeLAFQB (ORCPT ); Sat, 1 Dec 2018 00:16:01 -0500 Subject: Re: block layer API for file system creation - when to use multidisk mode References: <20181004222952.GV31060@dastard> <67627995-714c-5c38-a796-32b503de7d13@sandeen.net> <20181005232710.GH12041@dastard> <20181006232037.GB18095@dastard> <36bc3f17-e7d1-ce8b-2088-36ff5d7b1e8b@sandeen.net> <0290ec9f-ab2b-7c1b-faaf-409d72f99e5f@gmail.com> <20181129214851.GU6311@dastard> <39031e68-3936-b5e1-bcb6-6fdecc5988c1@gmail.com> <20181130022510.GW6311@dastard> <3da04164-a89f-f4c0-1529-eab12b3226e1@gmail.com> From: Mark Nelson Message-ID: <4125f92c-25f6-3397-19bb-61ce71c615c0@redhat.com> Date: Fri, 30 Nov 2018 12:05:46 -0600 MIME-Version: 1.0 In-Reply-To: <3da04164-a89f-f4c0-1529-eab12b3226e1@gmail.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US Sender: linux-xfs-owner@vger.kernel.org List-ID: List-Id: xfs To: Ric Wheeler , Dave Chinner Cc: Eric Sandeen , Ilya Dryomov , xfs , Eric Sandeen , Mike Snitzer , "linux-scsi@vger.kernel.org" , IDE/ATA development list , device-mapper development , Jens Axboe On 11/30/18 12:00 PM, Ric Wheeler wrote: > On 11/30/18 7:55 AM, Dave Chinner wrote: >> On Thu, Nov 29, 2018 at 06:53:14PM -0500, Ric Wheeler wrote: >>> On 11/29/18 4:48 PM, Dave Chinner wrote: >>>> On Thu, Nov 29, 2018 at 08:53:39AM -0500, Ric Wheeler wrote: >>>>> On 10/6/18 8:14 PM, Eric Sandeen wrote: >>>>>> On 10/6/18 6:20 PM, Dave Chinner wrote: >>>>>>>> Can you give an example of a use case that would be negatively >>>>>>>> affected >>>>>>>> if this heuristic was switched from "sunit" to "sunit < swidth"? >>>>>>> Any time you only know a single alignment characteristic of the >>>>>>> underlying multi-disk storage. e.g. hardware RAID0/5/6 that sets >>>>>>> iomin = ioopt, multi-level RAID constructs where only the largest >>>>>>> alignment requirement is exposed, RAID1 devices exposing their >>>>>>> chunk >>>>>>> size, remote replication chunk alignment (because remote rep. is >>>>>>> slow and so we need more concurrency to keep the pipeline full), >>>>>>> etc. >>>>>> So the tl;dr here is "given any iomin > 512, we should infer low >>>>>> seek >>>>>> latency and parallelism and adjust geometry accordingly?" >>>>>> >>>>>> -Eric >>>>> Chiming in late here, but I do think that every decade or two (no >>>>> disrespect to xfs!), it is worth having a second look at how the >>>>> storage has changed under us. >>>>> >>>>> The workload that has lots of file systems pounding on a shared >>>>> device for example is one way to lay out container storage. >>>> The problem is that defaults can't cater for every use case. >>>> And in this case, we've got nothing to tell us that this is >>>> aggregated/shared storage rather than "the fileystem owns the >>>> entire device". >>>> >>>>> No argument about documenting how to fix this with command line >>>>> tweaks for now, but maybe this would be a good topic for the next >>>>> LSF/MM shared track of file & storage people to debate? >>>> Doubt it - this is really only an XFS problem at this point. >>>> >>>> i.e. if we can't infer what the user wants from existing >>>> information, then I don't see how the storage is going to be able to >>>> tell us anything different, either.  i.e. somewhere in the stack the >>>> user is going to have to tell the block device that this is >>>> aggregated storage. >>>> >>>> But even then, if it's aggregated solid state storage, we still want >>>> to make use of the concurency on increased AG count because there is >>>> no seek penalty like spinning drives end up with. Or if the >>>> aggregated storage is thinly provisioned, the AG count of filesystem >>>> just doesn't matter because the IO is going to be massively >>>> randomised (i.e take random seek penalties) by the thinp layout. >>>> >>>> So there's really no good way of "guessing" whether aggregated >>>> storage should or shouldn't use elevated AG counts even if the >>>> storage says "this is aggregated storage". The user still has to >>>> give us some kind of explict hint about how the filesystem should >>>> be configured. >>>> >>>> What we need is for a solid, reliable detection hueristic to be >>>> suggested by the people that need this functionality before there's >>>> anything we can talk about. >>> I think that is exactly the kind of discussion that the shared >>> file/storage track is good for. >> Yes, but why on earth do we need to wait 6 months to have that >> conversation. Start it now... > > > Sure, that is definitely a good idea - added in some of the storage > lists to this reply. No perfect all encompassing block layer list that > I know of. > > >> >>> Other file systems also need to >>> accommodate/probe behind the fictitious visible storage device >>> layer... Specifically, is there something we can add per block >>> device to help here? Number of independent devices >> That's how mkfs.xfs used to do stripe unit/stripe width calculations >> automatically on MD devices back in the 2000s. We got rid of that >> for more generaly applicable configuration information such as >> minimum/optimal IO sizes so we could expose equivalent alignment >> information from lots of different types of storage device.... >> >>> or a map of >>> those regions? >> Not sure what this means or how we'd use it. >> >> Cheers, >> >> Dave. > > What I was thinking of was a way of giving up a good outline of how > many independent regions that are behind one "virtual" block device > like a ceph rbd or device mapper device. My assumption is that we are > trying to lay down (at least one) allocation group per region. > > What we need to optimize for includes: > >     * how many independent regions are there? > >     * what are the boundaries of those regions? > >     * optimal IO size/alignment/etc > > Some of that we have, but the current assumptions don't work well for > all device types. > > Regards, > > Ric > I won't comment on the details as there are others here that are far more knowledgeable than I am, but at a high level I think your idea is absolutely fantastic from the standpoint of making this decision process more explicit. Mark