From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: From: Bart Van Assche To: "snitzer@redhat.com" CC: "dm-devel@redhat.com" , "hch@infradead.org" , "linux-kernel@vger.kernel.org" , "linux-block@vger.kernel.org" , "osandov@fb.com" , "ming.lei@redhat.com" , "axboe@kernel.dk" Subject: Re: [RFC PATCH] blk-mq: fixup RESTART when queue becomes idle Date: Thu, 18 Jan 2018 17:20:57 +0000 Message-ID: <1516296056.2676.23.camel@wdc.com> References: <20180118024124.8079-1-ming.lei@redhat.com> <20180118170353.GB19734@redhat.com> In-Reply-To: <20180118170353.GB19734@redhat.com> Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 List-ID: T24gVGh1LCAyMDE4LTAxLTE4IGF0IDEyOjAzIC0wNTAwLCBNaWtlIFNuaXR6ZXIgd3JvdGU6DQo+ IE9uIFRodSwgSmFuIDE4IDIwMTggYXQgMTE6NTBhbSAtMDUwMCwNCj4gQmFydCBWYW4gQXNzY2hl IDxiYXJ0LnZhbmFzc2NoZUB3ZGMuY29tPiB3cm90ZToNCj4gPiBNeSBjb21tZW50cyBhYm91dCB0 aGUgYWJvdmUgYXJlIGFzIGZvbGxvd3M6DQo+ID4gLSBJdCBjYW4gdGFrZSB1cCB0byBxLT5ycV90 aW1lb3V0IGppZmZpZXMgYWZ0ZXIgYSAucXVldWVfcnEoKQ0KPiA+ICAgaW1wbGVtZW50YXRpb24g cmV0dXJuZWQgQkxLX1NUU19SRVNPVVJDRSBiZWZvcmUgYmxrX21xX3RpbWVvdXRfd29yaygpDQo+ ID4gICBnZXRzIGNhbGxlZC4gSG93ZXZlciwgaXQgY2FuIGhhcHBlbiB0aGF0IG9ubHkgYSBmZXcg bWlsbGlzZWNvbmRzIGFmdGVyDQo+ID4gICAucXVldWVfcnEoKSByZXR1cm5lZCBCTEtfU1RTX1JF U09VUkNFIHRoYXQgdGhlIGNvbmRpdGlvbiB0aGF0IGNhdXNlZA0KPiA+ICAgaXQgdG8gcmV0dXJu IEJMS19TVFNfUkVTT1VSQ0UgZ2V0cyBjbGVhcmVkLiBTbyB0aGUgYWJvdmUgYXBwcm9hY2ggY2Fu DQo+ID4gICByZXN1bHQgaW4gbG9uZyBkZWxheXMgZHVyaW5nIHdoaWNoIGl0IHdpbGwgc2VlbSBs aWtlIHRoZSBxdWV1ZSBnb3QNCj4gPiAgIHN0dWNrLiBBZGRpdGlvbmFsbHksIEkgdGhpbmsgdGhh dCB0aGUgYmxvY2sgZHJpdmVyIHNob3VsZCBkZWNpZGUgaG93DQo+ID4gICBsb25nIGl0IHRha2Vz IGJlZm9yZSBhIHF1ZXVlIGlzIHJlcnVuIGFuZCBub3QgdGhlIGJsb2NrIGxheWVyIGNvcmUuDQo+ IA0KPiBTbyBjb25maWd1cmUgcS0+cnFfdGltZW91dCB0byBiZSBzaG9ydGVyPyAgV2hpY2ggaXMg Y29uZmlndXJhYmxlIHRob3VnaA0KPiBibGtfbXFfdGFnX3NldCdzICd0aW1lb3V0JyBtZW1iZXIu ICBJdCBhcHBhcmVudGx5IGRlZmF1bHRzIHRvIDMwICogSFouDQo+IA0KPiBUaGF0IGlzIHRoZSBw cm9ibGVtIHdpdGggdGltZW91dHMsIHRoZXJlIGlzIGdlbmVyYWxseSBubyBvbmUgc2l6ZSBmaXRz DQo+IGFsbC4NCg0KU29ycnkgYnV0IEkgdGhpbmsgdGhhdCB3b3VsZCBiZSB3cm9uZy4gVGhlIGRl bGF5IGFmdGVyIHdoaWNoIGEgcXVldWUgaXMgcmVydW4NCnNob3VsZCBub3QgYmUgY291cGxlZCB0 byB0aGUgcmVxdWVzdCB0aW1lb3V0LiBUaGVzZSB0d28gc2hvdWxkIGJlIGluZGVwZW5kZW50Lg0K DQo+ID4gLSBUaGUgbG9ja3VwIHRoYXQgSSByZXBvcnRlZCBvbmx5IG9jY3VycyB3aXRoIHRoZSBk bSBkcml2ZXIgYnV0IG5vdCBhbnkNCj4gPiAgIG90aGVyIGJsb2NrIGRyaXZlci4gU28gd2h5IHRv IG1vZGlmeSB0aGUgYmxvY2sgbGF5ZXIgY29yZSBzaW5jZSB0aGlzDQo+ID4gICBjYW4gYmUgZml4 ZWQgYnkgbW9kaWZ5aW5nIHRoZSBkbSBkcml2ZXI/DQo+IA0KPiBIYXJkIHRvIGtub3cgaXQgaXMg b25seSBETSdzIGJsay1tcSB0aGF0IGlzIGltcGFjdGVkLiAgVGhhdCBpcyB0aGUgb25seQ0KPiBi bGstbXEgZHJpdmVyIHRoYXQgeW91J3JlIHRlc3RpbmcgbGlrZSB0aGlzICh0aGF0IGlzIGFsc28g YWJsZSB0byBoYW5kbGUNCj4gZmF1bHRzLCBldGMpLg0KDQpUaGF0J3Mgbm90IGNvcnJlY3QuIEkn bSBhbHNvIHRlc3RpbmcgdGhlIFNDU0kgY29yZSwgd2hpY2ggaXMgb25lIG9mIHRoZSBtb3N0DQpj b21wbGljYXRlZCBibG9jayBkcml2ZXJzLg0KDQo+ID4gLSBBIG11Y2ggc2ltcGxlciBmaXggYW5k IGEgZml4IHRoYXQgaXMga25vd24gdG8gd29yayBleGlzdHMsIG5hbWVseQ0KPiA+ICAgaW5zZXJ0 aW5nIGEgYmxrX21xX2RlbGF5X3J1bl9od19xdWV1ZSgpIGNhbGwgaW4gdGhlIGRtIGRyaXZlci4N Cj4gDQo+IEJlY2F1c2UgeW91ciAibXVjaCBzaW1wbGVyIiBmaXggYWN0aXZlbHkgaHVydHMgcGVy Zm9ybWFuY2UsIGFzIGlzDQo+IGRldGFpbGVkIGluIHRoaXMgaGVhZGVyOg0KPiBodHRwczovL2dp dC5rZXJuZWwub3JnL3B1Yi9zY20vbGludXgva2VybmVsL2dpdC9kZXZpY2UtbWFwcGVyL2xpbnV4 LWRtLmdpdC9jb21taXQvP2g9ZG0tNC4xNiZpZD1lYzNlYWY5YTY3MzEwNmY2NjYwNjg5NmFlZDZk ZGQyMDE4MGIwMmVjDQoNCldlIGFyZSBjbG9zZSB0byB0aGUgc3RhcnQgb2YgdGhlIG1lcmdlIHdp bmRvdyBzbyBJIHRoaW5rIGl0J3MgYmV0dGVyIHRvIGZhbGwNCmJhY2sgdG8gYW4gb2xkIGFwcHJv YWNoIHRoYXQgaXMga25vd24gdG8gd29yayB0aGFuIHRvIGtlZXAgYSBuZXcgYXBwcm9hY2gNCnRo YXQgaXMga25vd24gbm90IHRvIHdvcmsuIEFkZGl0aW9uYWxseSwgdGhlIHBlcmZvcm1hbmNlIGlz c3VlIHlvdSByZWZlcnJlZA0KdG8gb25seSBhZmZlY3RzIElPUFMgYW5kIGJhbmR3aWR0aCBtb3Jl IHRoYW4gMSUgd2l0aCB0aGUgbHBmYyBkcml2ZXIgYW5kIHRoYXQNCmlzIGJlY2F1c2UgdGhlIHF1 ZXVlIGRlcHRoIGl0IHN1cHBvcnRzIGlzIG11Y2ggbG93ZXIgdGhhbiBmb3Igb3RoZXIgU0NTSSBI QkFzLA0KbmFtZWx5IDMgaW5zdGVhZCBvZiA2NC4NCg0KVGhhbmtzLA0KDQpCYXJ0Lg== From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755082AbeARRVF (ORCPT ); Thu, 18 Jan 2018 12:21:05 -0500 Received: from esa6.hgst.iphmx.com ([216.71.154.45]:41423 "EHLO esa6.hgst.iphmx.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753228AbeARRVB (ORCPT ); Thu, 18 Jan 2018 12:21:01 -0500 X-IronPort-AV: E=Sophos;i="5.46,378,1511798400"; d="scan'208";a="69871375" From: Bart Van Assche To: "snitzer@redhat.com" CC: "dm-devel@redhat.com" , "hch@infradead.org" , "linux-kernel@vger.kernel.org" , "linux-block@vger.kernel.org" , "osandov@fb.com" , "ming.lei@redhat.com" , "axboe@kernel.dk" Subject: Re: [RFC PATCH] blk-mq: fixup RESTART when queue becomes idle Thread-Topic: [RFC PATCH] blk-mq: fixup RESTART when queue becomes idle Thread-Index: AQHTkAXlp7DN5JBZKUeyLfttG5XpOaN52OmAgAADroCAAATDAA== Date: Thu, 18 Jan 2018 17:20:57 +0000 Message-ID: <1516296056.2676.23.camel@wdc.com> References: <20180118024124.8079-1-ming.lei@redhat.com> <20180118170353.GB19734@redhat.com> In-Reply-To: <20180118170353.GB19734@redhat.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=Bart.VanAssche@wdc.com; x-originating-ip: [199.255.44.172] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1;CY1PR0401MB0971;7:CCZMvBXUcjsd/JxOFqzgFQdGmtY5rTokurdPDbfP0KQFZoAUCZMQnPscTOFIpO7q5qNkmCxVHHYxFbeFyKUUXWs9zSIcSgMTA228VtkYxF8ynzM5HsAuSqX1ebccBrOLtfoIRSUebmNTh/OpEKbZe1v/n1+gqkcC0bp4l6w8TLvLFKcGjn04wR7ltw5iQpc+BFhPQU7tLi2lSzYEPxPYY57bAkm5ex5Q7IYamtuCN1h+LrRFt2rOuhWyuGTUIahq;20:RZSRbiXAHOJTzcq//QFh2SGLXCceD84bfel4FaJBtJsfyM/hXZTEPR9o+lftZvF1O/oSDbqTWnsuh4t7SKzcehAlcFQk7T/5PJP88bLMA7u6dDKO7vC0X/qVj8XgAMI9KIrgcPFXM4wCCsv6BgPWq6SB01TAdwwnE8FcyzC2m4M= x-ms-exchange-antispam-srfa-diagnostics: SSOS; x-ms-office365-filtering-ht: Tenant x-ms-office365-filtering-correlation-id: b9af751f-e11b-4dab-51a9-08d55e97d67c x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:(7020095)(4652020)(48565401081)(5600026)(4604075)(3008032)(2017052603307)(7153060)(7193020);SRVR:CY1PR0401MB0971; x-ms-traffictypediagnostic: CY1PR0401MB0971: wdcipoutbound: EOP-TRUE x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(84791874153150); x-exchange-antispam-report-cfa-test: BCL:0;PCL:0;RULEID:(2401047)(5005006)(8121501046)(93006095)(93001095)(3231023)(2400064)(944501161)(10201501046)(3002001)(6055026)(6041268)(20161123564045)(201703131423095)(201703061421075)(20161123560045)(20161123558120)(20161123562045)(6042181)(6072148)(201708071742011);SRVR:CY1PR0401MB0971;BCL:0;PCL:0;RULEID:(100000803101)(100110400095);SRVR:CY1PR0401MB0971; x-forefront-prvs: 05568D1FF7 x-forefront-antispam-report: SFV:NSPM;SFS:(10019020)(366004)(377424004)(199004)(189003)(51444003)(2950100002)(6916009)(7736002)(6246003)(2906002)(3280700002)(5660300001)(68736007)(305945005)(59450400001)(102836004)(97736004)(4326008)(54906003)(66066001)(77096007)(53936002)(86362001)(6486002)(6436002)(72206003)(14454004)(5640700003)(2351001)(26005)(105586002)(106356001)(2900100001)(508600001)(25786009)(8936002)(6306002)(76176011)(229853002)(99286004)(966005)(6506007)(3660700001)(36756003)(1730700003)(81156014)(3846002)(6512007)(103116003)(81166006)(2501003)(8676002)(6116002);DIR:OUT;SFP:1102;SCL:1;SRVR:CY1PR0401MB0971;H:CY1PR0401MB1536.namprd04.prod.outlook.com;FPR:;SPF:None;PTR:InfoNoRecords;A:1;MX:1;LANG:en; x-microsoft-antispam-message-info: JDgsruNXbZ68z72wDL17PDX7ryrSzFTy6t9P8oJI484AOfVhIXx9d5dQ2zKpMdqOhB9HQt2chqZoA4U3V2hjDg== spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="utf-8" Content-ID: <2B12C166E6858A4BB4E6A3473F5F72B8@namprd04.prod.outlook.com> MIME-Version: 1.0 X-OriginatorOrg: wdc.com X-MS-Exchange-CrossTenant-Network-Message-Id: b9af751f-e11b-4dab-51a9-08d55e97d67c X-MS-Exchange-CrossTenant-originalarrivaltime: 18 Jan 2018 17:20:57.3556 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: b61c8803-16f3-4c35-9b17-6f65f441df86 X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY1PR0401MB0971 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from base64 to 8bit by mail.home.local id w0IHLB7d027666 On Thu, 2018-01-18 at 12:03 -0500, Mike Snitzer wrote: > On Thu, Jan 18 2018 at 11:50am -0500, > Bart Van Assche wrote: > > My comments about the above are as follows: > > - It can take up to q->rq_timeout jiffies after a .queue_rq() > > implementation returned BLK_STS_RESOURCE before blk_mq_timeout_work() > > gets called. However, it can happen that only a few milliseconds after > > .queue_rq() returned BLK_STS_RESOURCE that the condition that caused > > it to return BLK_STS_RESOURCE gets cleared. So the above approach can > > result in long delays during which it will seem like the queue got > > stuck. Additionally, I think that the block driver should decide how > > long it takes before a queue is rerun and not the block layer core. > > So configure q->rq_timeout to be shorter? Which is configurable though > blk_mq_tag_set's 'timeout' member. It apparently defaults to 30 * HZ. > > That is the problem with timeouts, there is generally no one size fits > all. Sorry but I think that would be wrong. The delay after which a queue is rerun should not be coupled to the request timeout. These two should be independent. > > - The lockup that I reported only occurs with the dm driver but not any > > other block driver. So why to modify the block layer core since this > > can be fixed by modifying the dm driver? > > Hard to know it is only DM's blk-mq that is impacted. That is the only > blk-mq driver that you're testing like this (that is also able to handle > faults, etc). That's not correct. I'm also testing the SCSI core, which is one of the most complicated block drivers. > > - A much simpler fix and a fix that is known to work exists, namely > > inserting a blk_mq_delay_run_hw_queue() call in the dm driver. > > Because your "much simpler" fix actively hurts performance, as is > detailed in this header: > https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=dm-4.16&id=ec3eaf9a673106f66606896aed6ddd20180b02ec We are close to the start of the merge window so I think it's better to fall back to an old approach that is known to work than to keep a new approach that is known not to work. Additionally, the performance issue you referred to only affects IOPS and bandwidth more than 1% with the lpfc driver and that is because the queue depth it supports is much lower than for other SCSI HBAs, namely 3 instead of 64. Thanks, Bart.