From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: From: Bart Van Assche To: "ming.lei@redhat.com" CC: "dm-devel@redhat.com" , "hch@infradead.org" , "linux-kernel@vger.kernel.org" , "linux-block@vger.kernel.org" , "osandov@fb.com" , "axboe@kernel.dk" , "snitzer@redhat.com" Subject: Re: [RFC PATCH] blk-mq: fixup RESTART when queue becomes idle Date: Fri, 19 Jan 2018 19:47:42 +0000 Message-ID: <1516391260.2968.27.camel@wdc.com> References: <20180118024124.8079-1-ming.lei@redhat.com> <20180118170353.GB19734@redhat.com> <1516296056.2676.23.camel@wdc.com> <20180118183039.GA20121@redhat.com> <1516301278.2676.35.camel@wdc.com> <20180119023212.GA25413@ming.t460p> <1516338585.4077.1.camel@wdc.com> <20180119073423.GC25369@ming.t460p> In-Reply-To: <20180119073423.GC25369@ming.t460p> Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 List-ID: T24gRnJpLCAyMDE4LTAxLTE5IGF0IDE1OjM0ICswODAwLCBNaW5nIExlaSB3cm90ZToNCj4gQ291 bGQgeW91IGV4cGxhaW4gYSBiaXQgd2hlbiBTQ1NJIHRhcmdldCByZXBsaWVzIHdpdGggQlVTWSB2 ZXJ5IG9mdGVuPw0KPiANCj4gSW5zaWRlIGluaXRpYXRvciwgd2UgaGF2ZSBsaW1pdGVkIHRoZSBt YXggcGVyLUxVTiByZXF1ZXN0cyBhbmQgcGVyLWhvc3QNCj4gcmVxdWVzdHMgYWxyZWFkeSBiZWZv cmUgY2FsbGluZyAucXVldWVfcnEoKS4NCg0KVGhhdCdzIGNvcnJlY3QuIEhvd2V2ZXIsIHdoZW4g YSBTQ1NJIGluaXRpYXRvciBhbmQgdGFyZ2V0IHN5c3RlbSBhcmUNCmNvbW11bmljYXRpbmcgd2l0 aCBlYWNoIG90aGVyIHRoZXJlIGlzIG5vIGd1YXJhbnRlZSB0aGF0IGluaXRpYXRvciBhbmQgdGFy Z2V0DQpxdWV1ZSBkZXB0aCBoYXZlIGJlZW4gdHVuZWQgcHJvcGVybHkuIElmIHRoZSBpbml0aWF0 b3IgcXVldWUgZGVwdGggYW5kIHRoZQ0KbnVtYmVyIG9mIHJlcXVlc3RzIHRoYXQgY2FuIGJlIGlu IGZsaWdodCBhY2NvcmRpbmcgdG8gdGhlIG5ldHdvcmsgcHJvdG9jb2wNCmFyZSBib3RoIGxhcmdl ciB0aGFuIHRoZSB0YXJnZXQgcXVldWUgZGVwdGggYW5kIGlmIHRoZSB0YXJnZXQgc3lzdGVtIHVz ZXMNCnJlbGF0aXZlbHkgc2xvdyBzdG9yYWdlIChlLmcuIGhhcmRkaXNrcykgdGhlbiBpdCBjYW4g aGFwcGVuIHRoYXQgdGhlIHRhcmdldA0KcmVwbGllcyB3aXRoIEJVU1kgb2Z0ZW4uDQoNClRoZSBM aW51eCBpU0NTSSBpbml0aWF0b3IgbGltaXRzIE1heE91dHN0YW5kaW5nUjJUICh0aGUgbnVtYmVy IG9mIHJlcXVlc3RzDQphbiBpbml0aWF0b3IgbWF5IHNlbnQgd2l0aG91dCBoYXZpbmcgcmVjZWl2 ZWQgYW4gYW5zd2VyIGZyb20gdGhlIHRhcmdldCkgdG8NCm9uZSBzbyBJIGRvbid0IHRoaW5rIHRo aXMgY2FuIGhhcHBlbiB3aGVuIHVzaW5nIGlTQ1NJL1RDUC4NCg0KV2l0aCB0aGUgU1JQIGluaXRp YXRvciBob3dldmVyIHRoZSBtYXhpbXVtIHJlcXVlc3RzIHRoYXQgY2FuIGJlIGluIGZsaWdodA0K YmV0d2VlbiBpbml0aWF0b3IgYW5kIHRhcmdldCBkZXBlbmRzIG9uIHRoZSBudW1iZXIgb2YgY3Jl ZGl0cyB0aGF0IHdlcmUNCm5lZ290aWF0ZWQgZHVyaW5nIGxvZ2luIGJldHdlZW4gaW5pdGlhdG9y IGFuZCB0YXJnZXQuIFNvbWUgdGltZSBhZ28gSSBtb2RpZmllZA0KdGhlIFNSUCBpbml0aWF0b3Ig c3VjaCB0aGF0IGl0IGxpbWl0cyB0aGUgaW5pdGlhdG9yIHF1ZXVlIGRlcHRoIHRvIHRoZSBudW1i ZXINCm9mIFNSUCBjcmVkaXRzIG1pbnVzIG9uZSAoZm9yIHRhc2sgbWFuYWdlbWVudCkuIFRoYXQg cmVzdWx0ZWQgaW4gYSBwZXJmb3JtYW5jZQ0KaW1wcm92ZW1lbnQgZHVlIHRvIGZld2VyIEJVU1kg Y29uZGl0aW9ucyBhdCB0aGUgaW5pdGlhdG9yIHNpZGUgKHNlZSBhbHNvIGNvbW1pdA0KN2FkZTQw MGFiYTlhICgiSUIvc3JwOiBSZWR1Y2UgbnVtYmVyIG9mIEJVU1kgY29uZGl0aW9ucyIpKS4gQW5v dGhlciBwYXRjaCBmb3INCnRoZSBTQ1NUIFNSUCB0YXJnZXQgZHJpdmVyIGxpbWl0ZWQgdGhlIG51 bWJlciBvZiBTUlAgY3JlZGl0cyB0byB0aGUgcXVldWUgZGVwdGgNCm9mIHRoZSBibG9jayBkZXZp Y2UgYXQgdGhlIHRhcmdldCBzaWRlLiBJJ20gcmVmZXJyaW5nIHRvIHRoZSBmb2xsb3dpbmcgY29k ZToNCmNoLT5ycV9zaXplID0gbWluKE1BWF9TUlBUX1JRX1NJWkUsIHNjc3RfZ2V0X21heF9sdW5f Y29tbWFuZHMoTlVMTCwgMCkpIChJIGhhdmUNCm5vdCB5ZXQgaGFkIHRoZSB0aW1lIHRvIHBvcnQg dGhpcyBjaGFuZ2UgdG8gTElPKS4NCg0KV2l0aG91dCBzdWNoIHR1bmluZyBhY3Jvc3MgaW5pdGlh dG9yIGFuZCB0YXJnZXQgaXQgY2FuIGhhcHBlbiBvZnRlbiB0aGF0IHRoZQ0KdGFyZ2V0IHN5c3Rl bSBzZW5kcyB0aGUgcmVwbHkgIkJVU1kiIGJhY2sgdG8gdGhlIGluaXRpYXRvci4gSSB0aGluayB0 aGF0J3Mgd2h5DQp0aGVyZSBpcyBjb2RlIGluIHRoZSBTQ1NJIGNvcmUgdG8gYXV0b21hdGljYWxs eSBhZGp1c3QgdGhlIGluaXRpYXRvciBxdWV1ZQ0KZGVwdGggaWYgdGhlICJCVVNZIiBjb25kaXRp b24gaXMgZW5jb3VudGVyZWQgZnJlcXVlbnRseS4gU2VlIGFsc28NCnNjc2lfdHJhY2tfcXVldWVf ZnVsbCgpLg0KDQpCYXJ0Lg== From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1756208AbeASTrw (ORCPT ); Fri, 19 Jan 2018 14:47:52 -0500 Received: from esa6.hgst.iphmx.com ([216.71.154.45]:59152 "EHLO esa6.hgst.iphmx.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755932AbeASTrp (ORCPT ); Fri, 19 Jan 2018 14:47:45 -0500 X-IronPort-AV: E=Sophos;i="5.46,382,1511798400"; d="scan'208";a="69989640" From: Bart Van Assche To: "ming.lei@redhat.com" CC: "dm-devel@redhat.com" , "hch@infradead.org" , "linux-kernel@vger.kernel.org" , "linux-block@vger.kernel.org" , "osandov@fb.com" , "axboe@kernel.dk" , "snitzer@redhat.com" Subject: Re: [RFC PATCH] blk-mq: fixup RESTART when queue becomes idle Thread-Topic: [RFC PATCH] blk-mq: fixup RESTART when queue becomes idle Thread-Index: AQHTkAXlp7DN5JBZKUeyLfttG5XpOaN52OmAgAADroCAAATDAIAAE3uAgAAE1gCAABc0gIAAaoKAgAAsBICAAChpgIAAzOAA Date: Fri, 19 Jan 2018 19:47:42 +0000 Message-ID: <1516391260.2968.27.camel@wdc.com> References: <20180118024124.8079-1-ming.lei@redhat.com> <20180118170353.GB19734@redhat.com> <1516296056.2676.23.camel@wdc.com> <20180118183039.GA20121@redhat.com> <1516301278.2676.35.camel@wdc.com> <20180119023212.GA25413@ming.t460p> <1516338585.4077.1.camel@wdc.com> <20180119073423.GC25369@ming.t460p> In-Reply-To: <20180119073423.GC25369@ming.t460p> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=Bart.VanAssche@wdc.com; x-originating-ip: [199.255.44.172] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1;CY1PR0401MB1691;7:hH4EAmbCKaGQOFAoj+j0pwWkXdSXS+6Xvh9Btpz6qikCpPanPggljS8zAa10x7xK4ZUq7vzkxiusZNNeP0xWQCPYnfOImZjIMh/D+et614rQC3Spv9w9SKmehEFnP6bWvorRmRQLZ8vusYfhvztnhF197jVhNSm6inDD+GxhCODkaISFDGV29S8zzUIXiPX0mGP7gBLRBi0aKWSXk2J2pmyeJDLwVBXDbx5V4xyqWYsOIQuxl+S+trTMzOOWwHvN;20:i10mmAnUP/2pG8prAHncglBmDxehTsR2TNEp45hDUHhtu4IZzociXzPdzlizWjaoibXB7Prlgr32RV/lRraCfh5+LW0g5cLT+L1513G2mPlolAZoimfPFD/rinIct7S4ZLscMg8b1ob/+T93w16/XRmLtC3/CW92caCrUghw2C0= x-ms-exchange-antispam-srfa-diagnostics: SSOS; x-ms-office365-filtering-ht: Tenant x-ms-office365-filtering-correlation-id: cdfb4191-92e8-4513-75d8-08d55f758189 x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:(7020095)(4652020)(48565401081)(5600026)(4604075)(3008032)(2017052603307)(7153060)(7193020);SRVR:CY1PR0401MB1691; x-ms-traffictypediagnostic: CY1PR0401MB1691: wdcipoutbound: EOP-TRUE x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(17755550239193); x-exchange-antispam-report-cfa-test: BCL:0;PCL:0;RULEID:(6040498)(2401047)(5005006)(8121501046)(10201501046)(93006095)(93001095)(3231023)(2400081)(944501161)(3002001)(6055026)(6041285)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123558120)(20161123564045)(20161123562045)(20161123560045)(6072148)(201708071742011);SRVR:CY1PR0401MB1691;BCL:0;PCL:0;RULEID:(100000803101)(100110400095);SRVR:CY1PR0401MB1691; x-forefront-prvs: 0557CBAD84 x-forefront-antispam-report: SFV:NSPM;SFS:(10019020)(346002)(39380400002)(376002)(39860400002)(396003)(366004)(51444003)(377424004)(199004)(189003)(53936002)(6246003)(102836004)(6506007)(6436002)(14454004)(93886005)(6486002)(6512007)(54906003)(316002)(5640700003)(36756003)(25786009)(103116003)(478600001)(4326008)(72206003)(229853002)(106356001)(2501003)(2906002)(97736004)(8936002)(66066001)(6116002)(3846002)(81156014)(81166006)(68736007)(2900100001)(305945005)(3280700002)(105586002)(3660700001)(59450400001)(76176011)(86362001)(6916009)(2351001)(2950100002)(5660300001)(7736002)(77096007)(99286004)(8676002)(26005);DIR:OUT;SFP:1102;SCL:1;SRVR:CY1PR0401MB1691;H:CY1PR0401MB1536.namprd04.prod.outlook.com;FPR:;SPF:None;PTR:InfoNoRecords;A:1;MX:1;LANG:en; x-microsoft-antispam-message-info: Cn0mJujgVLYrzekNqRqQUnS3oOEyVRHuX2YzRw9hFcdoWlWG67IVZsk8r0pPVyqE7pzv4zmh89Y0xDHeBSbCaw== spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="utf-8" Content-ID: <45B7B73428ED1F44BF70190FC576D038@namprd04.prod.outlook.com> MIME-Version: 1.0 X-OriginatorOrg: wdc.com X-MS-Exchange-CrossTenant-Network-Message-Id: cdfb4191-92e8-4513-75d8-08d55f758189 X-MS-Exchange-CrossTenant-originalarrivaltime: 19 Jan 2018 19:47:43.0468 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: b61c8803-16f3-4c35-9b17-6f65f441df86 X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY1PR0401MB1691 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from base64 to 8bit by mail.home.local id w0JJlulR004628 On Fri, 2018-01-19 at 15:34 +0800, Ming Lei wrote: > Could you explain a bit when SCSI target replies with BUSY very often? > > Inside initiator, we have limited the max per-LUN requests and per-host > requests already before calling .queue_rq(). That's correct. However, when a SCSI initiator and target system are communicating with each other there is no guarantee that initiator and target queue depth have been tuned properly. If the initiator queue depth and the number of requests that can be in flight according to the network protocol are both larger than the target queue depth and if the target system uses relatively slow storage (e.g. harddisks) then it can happen that the target replies with BUSY often. The Linux iSCSI initiator limits MaxOutstandingR2T (the number of requests an initiator may sent without having received an answer from the target) to one so I don't think this can happen when using iSCSI/TCP. With the SRP initiator however the maximum requests that can be in flight between initiator and target depends on the number of credits that were negotiated during login between initiator and target. Some time ago I modified the SRP initiator such that it limits the initiator queue depth to the number of SRP credits minus one (for task management). That resulted in a performance improvement due to fewer BUSY conditions at the initiator side (see also commit 7ade400aba9a ("IB/srp: Reduce number of BUSY conditions")). Another patch for the SCST SRP target driver limited the number of SRP credits to the queue depth of the block device at the target side. I'm referring to the following code: ch->rq_size = min(MAX_SRPT_RQ_SIZE, scst_get_max_lun_commands(NULL, 0)) (I have not yet had the time to port this change to LIO). Without such tuning across initiator and target it can happen often that the target system sends the reply "BUSY" back to the initiator. I think that's why there is code in the SCSI core to automatically adjust the initiator queue depth if the "BUSY" condition is encountered frequently. See also scsi_track_queue_full(). Bart.