From: Bart Van Assche <Bart.VanAssche@wdc.com>
To: "ming.lei@redhat.com" <ming.lei@redhat.com>
Cc: "dm-devel@redhat.com" <dm-devel@redhat.com>,
"hch@infradead.org" <hch@infradead.org>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
"linux-block@vger.kernel.org" <linux-block@vger.kernel.org>,
"osandov@fb.com" <osandov@fb.com>,
"axboe@kernel.dk" <axboe@kernel.dk>,
"snitzer@redhat.com" <snitzer@redhat.com>
Subject: Re: [RFC PATCH] blk-mq: fixup RESTART when queue becomes idle
Date: Fri, 19 Jan 2018 19:47:42 +0000 [thread overview]
Message-ID: <1516391260.2968.27.camel@wdc.com> (raw)
In-Reply-To: <20180119073423.GC25369@ming.t460p>
T24gRnJpLCAyMDE4LTAxLTE5IGF0IDE1OjM0ICswODAwLCBNaW5nIExlaSB3cm90ZToNCj4gQ291
bGQgeW91IGV4cGxhaW4gYSBiaXQgd2hlbiBTQ1NJIHRhcmdldCByZXBsaWVzIHdpdGggQlVTWSB2
ZXJ5IG9mdGVuPw0KPiANCj4gSW5zaWRlIGluaXRpYXRvciwgd2UgaGF2ZSBsaW1pdGVkIHRoZSBt
YXggcGVyLUxVTiByZXF1ZXN0cyBhbmQgcGVyLWhvc3QNCj4gcmVxdWVzdHMgYWxyZWFkeSBiZWZv
cmUgY2FsbGluZyAucXVldWVfcnEoKS4NCg0KVGhhdCdzIGNvcnJlY3QuIEhvd2V2ZXIsIHdoZW4g
YSBTQ1NJIGluaXRpYXRvciBhbmQgdGFyZ2V0IHN5c3RlbSBhcmUNCmNvbW11bmljYXRpbmcgd2l0
aCBlYWNoIG90aGVyIHRoZXJlIGlzIG5vIGd1YXJhbnRlZSB0aGF0IGluaXRpYXRvciBhbmQgdGFy
Z2V0DQpxdWV1ZSBkZXB0aCBoYXZlIGJlZW4gdHVuZWQgcHJvcGVybHkuIElmIHRoZSBpbml0aWF0
b3IgcXVldWUgZGVwdGggYW5kIHRoZQ0KbnVtYmVyIG9mIHJlcXVlc3RzIHRoYXQgY2FuIGJlIGlu
IGZsaWdodCBhY2NvcmRpbmcgdG8gdGhlIG5ldHdvcmsgcHJvdG9jb2wNCmFyZSBib3RoIGxhcmdl
ciB0aGFuIHRoZSB0YXJnZXQgcXVldWUgZGVwdGggYW5kIGlmIHRoZSB0YXJnZXQgc3lzdGVtIHVz
ZXMNCnJlbGF0aXZlbHkgc2xvdyBzdG9yYWdlIChlLmcuIGhhcmRkaXNrcykgdGhlbiBpdCBjYW4g
aGFwcGVuIHRoYXQgdGhlIHRhcmdldA0KcmVwbGllcyB3aXRoIEJVU1kgb2Z0ZW4uDQoNClRoZSBM
aW51eCBpU0NTSSBpbml0aWF0b3IgbGltaXRzIE1heE91dHN0YW5kaW5nUjJUICh0aGUgbnVtYmVy
IG9mIHJlcXVlc3RzDQphbiBpbml0aWF0b3IgbWF5IHNlbnQgd2l0aG91dCBoYXZpbmcgcmVjZWl2
ZWQgYW4gYW5zd2VyIGZyb20gdGhlIHRhcmdldCkgdG8NCm9uZSBzbyBJIGRvbid0IHRoaW5rIHRo
aXMgY2FuIGhhcHBlbiB3aGVuIHVzaW5nIGlTQ1NJL1RDUC4NCg0KV2l0aCB0aGUgU1JQIGluaXRp
YXRvciBob3dldmVyIHRoZSBtYXhpbXVtIHJlcXVlc3RzIHRoYXQgY2FuIGJlIGluIGZsaWdodA0K
YmV0d2VlbiBpbml0aWF0b3IgYW5kIHRhcmdldCBkZXBlbmRzIG9uIHRoZSBudW1iZXIgb2YgY3Jl
ZGl0cyB0aGF0IHdlcmUNCm5lZ290aWF0ZWQgZHVyaW5nIGxvZ2luIGJldHdlZW4gaW5pdGlhdG9y
IGFuZCB0YXJnZXQuIFNvbWUgdGltZSBhZ28gSSBtb2RpZmllZA0KdGhlIFNSUCBpbml0aWF0b3Ig
c3VjaCB0aGF0IGl0IGxpbWl0cyB0aGUgaW5pdGlhdG9yIHF1ZXVlIGRlcHRoIHRvIHRoZSBudW1i
ZXINCm9mIFNSUCBjcmVkaXRzIG1pbnVzIG9uZSAoZm9yIHRhc2sgbWFuYWdlbWVudCkuIFRoYXQg
cmVzdWx0ZWQgaW4gYSBwZXJmb3JtYW5jZQ0KaW1wcm92ZW1lbnQgZHVlIHRvIGZld2VyIEJVU1kg
Y29uZGl0aW9ucyBhdCB0aGUgaW5pdGlhdG9yIHNpZGUgKHNlZSBhbHNvIGNvbW1pdA0KN2FkZTQw
MGFiYTlhICgiSUIvc3JwOiBSZWR1Y2UgbnVtYmVyIG9mIEJVU1kgY29uZGl0aW9ucyIpKS4gQW5v
dGhlciBwYXRjaCBmb3INCnRoZSBTQ1NUIFNSUCB0YXJnZXQgZHJpdmVyIGxpbWl0ZWQgdGhlIG51
bWJlciBvZiBTUlAgY3JlZGl0cyB0byB0aGUgcXVldWUgZGVwdGgNCm9mIHRoZSBibG9jayBkZXZp
Y2UgYXQgdGhlIHRhcmdldCBzaWRlLiBJJ20gcmVmZXJyaW5nIHRvIHRoZSBmb2xsb3dpbmcgY29k
ZToNCmNoLT5ycV9zaXplID0gbWluKE1BWF9TUlBUX1JRX1NJWkUsIHNjc3RfZ2V0X21heF9sdW5f
Y29tbWFuZHMoTlVMTCwgMCkpIChJIGhhdmUNCm5vdCB5ZXQgaGFkIHRoZSB0aW1lIHRvIHBvcnQg
dGhpcyBjaGFuZ2UgdG8gTElPKS4NCg0KV2l0aG91dCBzdWNoIHR1bmluZyBhY3Jvc3MgaW5pdGlh
dG9yIGFuZCB0YXJnZXQgaXQgY2FuIGhhcHBlbiBvZnRlbiB0aGF0IHRoZQ0KdGFyZ2V0IHN5c3Rl
bSBzZW5kcyB0aGUgcmVwbHkgIkJVU1kiIGJhY2sgdG8gdGhlIGluaXRpYXRvci4gSSB0aGluayB0
aGF0J3Mgd2h5DQp0aGVyZSBpcyBjb2RlIGluIHRoZSBTQ1NJIGNvcmUgdG8gYXV0b21hdGljYWxs
eSBhZGp1c3QgdGhlIGluaXRpYXRvciBxdWV1ZQ0KZGVwdGggaWYgdGhlICJCVVNZIiBjb25kaXRp
b24gaXMgZW5jb3VudGVyZWQgZnJlcXVlbnRseS4gU2VlIGFsc28NCnNjc2lfdHJhY2tfcXVldWVf
ZnVsbCgpLg0KDQpCYXJ0Lg==
WARNING: multiple messages have this Message-ID (diff)
From: Bart Van Assche <Bart.VanAssche@wdc.com>
To: "ming.lei@redhat.com" <ming.lei@redhat.com>
Cc: "dm-devel@redhat.com" <dm-devel@redhat.com>,
"hch@infradead.org" <hch@infradead.org>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
"linux-block@vger.kernel.org" <linux-block@vger.kernel.org>,
"osandov@fb.com" <osandov@fb.com>,
"axboe@kernel.dk" <axboe@kernel.dk>,
"snitzer@redhat.com" <snitzer@redhat.com>
Subject: Re: [RFC PATCH] blk-mq: fixup RESTART when queue becomes idle
Date: Fri, 19 Jan 2018 19:47:42 +0000 [thread overview]
Message-ID: <1516391260.2968.27.camel@wdc.com> (raw)
In-Reply-To: <20180119073423.GC25369@ming.t460p>
On Fri, 2018-01-19 at 15:34 +0800, Ming Lei wrote:
> Could you explain a bit when SCSI target replies with BUSY very often?
>
> Inside initiator, we have limited the max per-LUN requests and per-host
> requests already before calling .queue_rq().
That's correct. However, when a SCSI initiator and target system are
communicating with each other there is no guarantee that initiator and target
queue depth have been tuned properly. If the initiator queue depth and the
number of requests that can be in flight according to the network protocol
are both larger than the target queue depth and if the target system uses
relatively slow storage (e.g. harddisks) then it can happen that the target
replies with BUSY often.
The Linux iSCSI initiator limits MaxOutstandingR2T (the number of requests
an initiator may sent without having received an answer from the target) to
one so I don't think this can happen when using iSCSI/TCP.
With the SRP initiator however the maximum requests that can be in flight
between initiator and target depends on the number of credits that were
negotiated during login between initiator and target. Some time ago I modified
the SRP initiator such that it limits the initiator queue depth to the number
of SRP credits minus one (for task management). That resulted in a performance
improvement due to fewer BUSY conditions at the initiator side (see also commit
7ade400aba9a ("IB/srp: Reduce number of BUSY conditions")). Another patch for
the SCST SRP target driver limited the number of SRP credits to the queue depth
of the block device at the target side. I'm referring to the following code:
ch->rq_size = min(MAX_SRPT_RQ_SIZE, scst_get_max_lun_commands(NULL, 0)) (I have
not yet had the time to port this change to LIO).
Without such tuning across initiator and target it can happen often that the
target system sends the reply "BUSY" back to the initiator. I think that's why
there is code in the SCSI core to automatically adjust the initiator queue
depth if the "BUSY" condition is encountered frequently. See also
scsi_track_queue_full().
Bart.
next prev parent reply other threads:[~2018-01-19 19:47 UTC|newest]
Thread overview: 80+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-01-18 2:41 [RFC PATCH] blk-mq: fixup RESTART when queue becomes idle Ming Lei
2018-01-18 16:50 ` Bart Van Assche
2018-01-18 17:03 ` Mike Snitzer
2018-01-18 17:03 ` Mike Snitzer
2018-01-18 17:20 ` Bart Van Assche
2018-01-18 17:20 ` Bart Van Assche
2018-01-18 18:30 ` Mike Snitzer
2018-01-18 18:47 ` Bart Van Assche
2018-01-18 18:47 ` Bart Van Assche
2018-01-18 20:11 ` Jens Axboe
2018-01-18 20:11 ` Jens Axboe
2018-01-18 20:48 ` Mike Snitzer
2018-01-18 20:58 ` Bart Van Assche
2018-01-18 20:58 ` Bart Van Assche
2018-01-18 21:23 ` Mike Snitzer
2018-01-18 21:23 ` Mike Snitzer
2018-01-18 21:37 ` Laurence Oberman
2018-01-18 21:39 ` [dm-devel] " Bart Van Assche
2018-01-18 21:39 ` Bart Van Assche
2018-01-18 21:45 ` Laurence Oberman
2018-01-18 21:45 ` Laurence Oberman
2018-01-18 22:01 ` Mike Snitzer
2018-01-18 22:18 ` Laurence Oberman
2018-01-18 22:20 ` Laurence Oberman
2018-01-18 22:20 ` Laurence Oberman
2018-01-18 22:24 ` Bart Van Assche
2018-01-18 22:24 ` Bart Van Assche
2018-01-18 22:35 ` Laurence Oberman
2018-01-18 22:39 ` Jens Axboe
2018-01-18 22:55 ` Bart Van Assche
2018-01-18 22:55 ` Bart Van Assche
2018-01-18 22:20 ` Bart Van Assche
2018-01-18 22:20 ` Bart Van Assche
2018-01-23 9:22 ` [PATCH] block: neutralize blk_insert_cloned_request IO stall regression (was: Re: [RFC PATCH] blk-mq: fixup RESTART when queue becomes idle) Mike Snitzer
2018-01-23 10:53 ` Ming Lei
2018-01-23 12:15 ` Mike Snitzer
2018-01-23 12:17 ` Ming Lei
2018-01-23 12:43 ` Mike Snitzer
2018-01-23 16:43 ` [PATCH] " Bart Van Assche
2018-01-23 16:43 ` Bart Van Assche
2018-01-19 2:32 ` [RFC PATCH] blk-mq: fixup RESTART when queue becomes idle Ming Lei
2018-01-19 4:02 ` Jens Axboe
2018-01-19 7:26 ` Ming Lei
2018-01-19 15:20 ` Bart Van Assche
2018-01-19 15:20 ` Bart Van Assche
2018-01-19 15:25 ` Jens Axboe
2018-01-19 15:33 ` Ming Lei
2018-01-19 16:06 ` Bart Van Assche
2018-01-19 16:06 ` Bart Van Assche
2018-01-19 15:24 ` Jens Axboe
2018-01-19 15:40 ` Ming Lei
2018-01-19 15:40 ` Ming Lei
2018-01-19 15:48 ` Jens Axboe
2018-01-19 16:05 ` Ming Lei
2018-01-19 16:19 ` Jens Axboe
2018-01-19 16:26 ` Ming Lei
2018-01-19 16:27 ` Jens Axboe
2018-01-19 16:37 ` Ming Lei
2018-01-19 16:41 ` Jens Axboe
2018-01-19 16:41 ` Jens Axboe
2018-01-19 16:47 ` Mike Snitzer
2018-01-19 16:52 ` Jens Axboe
2018-01-19 17:05 ` Ming Lei
2018-01-19 17:09 ` Jens Axboe
2018-01-19 17:20 ` Ming Lei
2018-01-19 17:38 ` Jens Axboe
2018-01-19 18:24 ` Ming Lei
2018-01-19 18:24 ` Ming Lei
2018-01-19 18:33 ` Mike Snitzer
2018-01-19 23:52 ` Ming Lei
2018-01-20 4:27 ` Jens Axboe
2018-01-19 16:13 ` Mike Snitzer
2018-01-19 16:23 ` Jens Axboe
2018-01-19 23:57 ` Ming Lei
2018-01-29 22:37 ` Bart Van Assche
2018-01-19 5:09 ` Bart Van Assche
2018-01-19 5:09 ` Bart Van Assche
2018-01-19 7:34 ` Ming Lei
2018-01-19 19:47 ` Bart Van Assche [this message]
2018-01-19 19:47 ` Bart Van Assche
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1516391260.2968.27.camel@wdc.com \
--to=bart.vanassche@wdc.com \
--cc=axboe@kernel.dk \
--cc=dm-devel@redhat.com \
--cc=hch@infradead.org \
--cc=linux-block@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=ming.lei@redhat.com \
--cc=osandov@fb.com \
--cc=snitzer@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.