From: Bart Van Assche <Bart.VanAssche@wdc.com>
To: "snitzer@redhat.com" <snitzer@redhat.com>
Cc: "dm-devel@redhat.com" <dm-devel@redhat.com>,
"hch@infradead.org" <hch@infradead.org>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
"linux-block@vger.kernel.org" <linux-block@vger.kernel.org>,
"osandov@fb.com" <osandov@fb.com>,
"ming.lei@redhat.com" <ming.lei@redhat.com>,
"axboe@kernel.dk" <axboe@kernel.dk>
Subject: Re: [RFC PATCH] blk-mq: fixup RESTART when queue becomes idle
Date: Thu, 18 Jan 2018 17:20:57 +0000 [thread overview]
Message-ID: <1516296056.2676.23.camel@wdc.com> (raw)
In-Reply-To: <20180118170353.GB19734@redhat.com>
T24gVGh1LCAyMDE4LTAxLTE4IGF0IDEyOjAzIC0wNTAwLCBNaWtlIFNuaXR6ZXIgd3JvdGU6DQo+
IE9uIFRodSwgSmFuIDE4IDIwMTggYXQgMTE6NTBhbSAtMDUwMCwNCj4gQmFydCBWYW4gQXNzY2hl
IDxiYXJ0LnZhbmFzc2NoZUB3ZGMuY29tPiB3cm90ZToNCj4gPiBNeSBjb21tZW50cyBhYm91dCB0
aGUgYWJvdmUgYXJlIGFzIGZvbGxvd3M6DQo+ID4gLSBJdCBjYW4gdGFrZSB1cCB0byBxLT5ycV90
aW1lb3V0IGppZmZpZXMgYWZ0ZXIgYSAucXVldWVfcnEoKQ0KPiA+ICAgaW1wbGVtZW50YXRpb24g
cmV0dXJuZWQgQkxLX1NUU19SRVNPVVJDRSBiZWZvcmUgYmxrX21xX3RpbWVvdXRfd29yaygpDQo+
ID4gICBnZXRzIGNhbGxlZC4gSG93ZXZlciwgaXQgY2FuIGhhcHBlbiB0aGF0IG9ubHkgYSBmZXcg
bWlsbGlzZWNvbmRzIGFmdGVyDQo+ID4gICAucXVldWVfcnEoKSByZXR1cm5lZCBCTEtfU1RTX1JF
U09VUkNFIHRoYXQgdGhlIGNvbmRpdGlvbiB0aGF0IGNhdXNlZA0KPiA+ICAgaXQgdG8gcmV0dXJu
IEJMS19TVFNfUkVTT1VSQ0UgZ2V0cyBjbGVhcmVkLiBTbyB0aGUgYWJvdmUgYXBwcm9hY2ggY2Fu
DQo+ID4gICByZXN1bHQgaW4gbG9uZyBkZWxheXMgZHVyaW5nIHdoaWNoIGl0IHdpbGwgc2VlbSBs
aWtlIHRoZSBxdWV1ZSBnb3QNCj4gPiAgIHN0dWNrLiBBZGRpdGlvbmFsbHksIEkgdGhpbmsgdGhh
dCB0aGUgYmxvY2sgZHJpdmVyIHNob3VsZCBkZWNpZGUgaG93DQo+ID4gICBsb25nIGl0IHRha2Vz
IGJlZm9yZSBhIHF1ZXVlIGlzIHJlcnVuIGFuZCBub3QgdGhlIGJsb2NrIGxheWVyIGNvcmUuDQo+
IA0KPiBTbyBjb25maWd1cmUgcS0+cnFfdGltZW91dCB0byBiZSBzaG9ydGVyPyAgV2hpY2ggaXMg
Y29uZmlndXJhYmxlIHRob3VnaA0KPiBibGtfbXFfdGFnX3NldCdzICd0aW1lb3V0JyBtZW1iZXIu
ICBJdCBhcHBhcmVudGx5IGRlZmF1bHRzIHRvIDMwICogSFouDQo+IA0KPiBUaGF0IGlzIHRoZSBw
cm9ibGVtIHdpdGggdGltZW91dHMsIHRoZXJlIGlzIGdlbmVyYWxseSBubyBvbmUgc2l6ZSBmaXRz
DQo+IGFsbC4NCg0KU29ycnkgYnV0IEkgdGhpbmsgdGhhdCB3b3VsZCBiZSB3cm9uZy4gVGhlIGRl
bGF5IGFmdGVyIHdoaWNoIGEgcXVldWUgaXMgcmVydW4NCnNob3VsZCBub3QgYmUgY291cGxlZCB0
byB0aGUgcmVxdWVzdCB0aW1lb3V0LiBUaGVzZSB0d28gc2hvdWxkIGJlIGluZGVwZW5kZW50Lg0K
DQo+ID4gLSBUaGUgbG9ja3VwIHRoYXQgSSByZXBvcnRlZCBvbmx5IG9jY3VycyB3aXRoIHRoZSBk
bSBkcml2ZXIgYnV0IG5vdCBhbnkNCj4gPiAgIG90aGVyIGJsb2NrIGRyaXZlci4gU28gd2h5IHRv
IG1vZGlmeSB0aGUgYmxvY2sgbGF5ZXIgY29yZSBzaW5jZSB0aGlzDQo+ID4gICBjYW4gYmUgZml4
ZWQgYnkgbW9kaWZ5aW5nIHRoZSBkbSBkcml2ZXI/DQo+IA0KPiBIYXJkIHRvIGtub3cgaXQgaXMg
b25seSBETSdzIGJsay1tcSB0aGF0IGlzIGltcGFjdGVkLiAgVGhhdCBpcyB0aGUgb25seQ0KPiBi
bGstbXEgZHJpdmVyIHRoYXQgeW91J3JlIHRlc3RpbmcgbGlrZSB0aGlzICh0aGF0IGlzIGFsc28g
YWJsZSB0byBoYW5kbGUNCj4gZmF1bHRzLCBldGMpLg0KDQpUaGF0J3Mgbm90IGNvcnJlY3QuIEkn
bSBhbHNvIHRlc3RpbmcgdGhlIFNDU0kgY29yZSwgd2hpY2ggaXMgb25lIG9mIHRoZSBtb3N0DQpj
b21wbGljYXRlZCBibG9jayBkcml2ZXJzLg0KDQo+ID4gLSBBIG11Y2ggc2ltcGxlciBmaXggYW5k
IGEgZml4IHRoYXQgaXMga25vd24gdG8gd29yayBleGlzdHMsIG5hbWVseQ0KPiA+ICAgaW5zZXJ0
aW5nIGEgYmxrX21xX2RlbGF5X3J1bl9od19xdWV1ZSgpIGNhbGwgaW4gdGhlIGRtIGRyaXZlci4N
Cj4gDQo+IEJlY2F1c2UgeW91ciAibXVjaCBzaW1wbGVyIiBmaXggYWN0aXZlbHkgaHVydHMgcGVy
Zm9ybWFuY2UsIGFzIGlzDQo+IGRldGFpbGVkIGluIHRoaXMgaGVhZGVyOg0KPiBodHRwczovL2dp
dC5rZXJuZWwub3JnL3B1Yi9zY20vbGludXgva2VybmVsL2dpdC9kZXZpY2UtbWFwcGVyL2xpbnV4
LWRtLmdpdC9jb21taXQvP2g9ZG0tNC4xNiZpZD1lYzNlYWY5YTY3MzEwNmY2NjYwNjg5NmFlZDZk
ZGQyMDE4MGIwMmVjDQoNCldlIGFyZSBjbG9zZSB0byB0aGUgc3RhcnQgb2YgdGhlIG1lcmdlIHdp
bmRvdyBzbyBJIHRoaW5rIGl0J3MgYmV0dGVyIHRvIGZhbGwNCmJhY2sgdG8gYW4gb2xkIGFwcHJv
YWNoIHRoYXQgaXMga25vd24gdG8gd29yayB0aGFuIHRvIGtlZXAgYSBuZXcgYXBwcm9hY2gNCnRo
YXQgaXMga25vd24gbm90IHRvIHdvcmsuIEFkZGl0aW9uYWxseSwgdGhlIHBlcmZvcm1hbmNlIGlz
c3VlIHlvdSByZWZlcnJlZA0KdG8gb25seSBhZmZlY3RzIElPUFMgYW5kIGJhbmR3aWR0aCBtb3Jl
IHRoYW4gMSUgd2l0aCB0aGUgbHBmYyBkcml2ZXIgYW5kIHRoYXQNCmlzIGJlY2F1c2UgdGhlIHF1
ZXVlIGRlcHRoIGl0IHN1cHBvcnRzIGlzIG11Y2ggbG93ZXIgdGhhbiBmb3Igb3RoZXIgU0NTSSBI
QkFzLA0KbmFtZWx5IDMgaW5zdGVhZCBvZiA2NC4NCg0KVGhhbmtzLA0KDQpCYXJ0Lg==
WARNING: multiple messages have this Message-ID (diff)
From: Bart Van Assche <Bart.VanAssche@wdc.com>
To: "snitzer@redhat.com" <snitzer@redhat.com>
Cc: "dm-devel@redhat.com" <dm-devel@redhat.com>,
"hch@infradead.org" <hch@infradead.org>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
"linux-block@vger.kernel.org" <linux-block@vger.kernel.org>,
"osandov@fb.com" <osandov@fb.com>,
"ming.lei@redhat.com" <ming.lei@redhat.com>,
"axboe@kernel.dk" <axboe@kernel.dk>
Subject: Re: [RFC PATCH] blk-mq: fixup RESTART when queue becomes idle
Date: Thu, 18 Jan 2018 17:20:57 +0000 [thread overview]
Message-ID: <1516296056.2676.23.camel@wdc.com> (raw)
In-Reply-To: <20180118170353.GB19734@redhat.com>
On Thu, 2018-01-18 at 12:03 -0500, Mike Snitzer wrote:
> On Thu, Jan 18 2018 at 11:50am -0500,
> Bart Van Assche <bart.vanassche@wdc.com> wrote:
> > My comments about the above are as follows:
> > - It can take up to q->rq_timeout jiffies after a .queue_rq()
> > implementation returned BLK_STS_RESOURCE before blk_mq_timeout_work()
> > gets called. However, it can happen that only a few milliseconds after
> > .queue_rq() returned BLK_STS_RESOURCE that the condition that caused
> > it to return BLK_STS_RESOURCE gets cleared. So the above approach can
> > result in long delays during which it will seem like the queue got
> > stuck. Additionally, I think that the block driver should decide how
> > long it takes before a queue is rerun and not the block layer core.
>
> So configure q->rq_timeout to be shorter? Which is configurable though
> blk_mq_tag_set's 'timeout' member. It apparently defaults to 30 * HZ.
>
> That is the problem with timeouts, there is generally no one size fits
> all.
Sorry but I think that would be wrong. The delay after which a queue is rerun
should not be coupled to the request timeout. These two should be independent.
> > - The lockup that I reported only occurs with the dm driver but not any
> > other block driver. So why to modify the block layer core since this
> > can be fixed by modifying the dm driver?
>
> Hard to know it is only DM's blk-mq that is impacted. That is the only
> blk-mq driver that you're testing like this (that is also able to handle
> faults, etc).
That's not correct. I'm also testing the SCSI core, which is one of the most
complicated block drivers.
> > - A much simpler fix and a fix that is known to work exists, namely
> > inserting a blk_mq_delay_run_hw_queue() call in the dm driver.
>
> Because your "much simpler" fix actively hurts performance, as is
> detailed in this header:
> https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=dm-4.16&id=ec3eaf9a673106f66606896aed6ddd20180b02ec
We are close to the start of the merge window so I think it's better to fall
back to an old approach that is known to work than to keep a new approach
that is known not to work. Additionally, the performance issue you referred
to only affects IOPS and bandwidth more than 1% with the lpfc driver and that
is because the queue depth it supports is much lower than for other SCSI HBAs,
namely 3 instead of 64.
Thanks,
Bart.
next prev parent reply other threads:[~2018-01-18 17:20 UTC|newest]
Thread overview: 80+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-01-18 2:41 [RFC PATCH] blk-mq: fixup RESTART when queue becomes idle Ming Lei
2018-01-18 16:50 ` Bart Van Assche
2018-01-18 17:03 ` Mike Snitzer
2018-01-18 17:03 ` Mike Snitzer
2018-01-18 17:20 ` Bart Van Assche [this message]
2018-01-18 17:20 ` Bart Van Assche
2018-01-18 18:30 ` Mike Snitzer
2018-01-18 18:47 ` Bart Van Assche
2018-01-18 18:47 ` Bart Van Assche
2018-01-18 20:11 ` Jens Axboe
2018-01-18 20:11 ` Jens Axboe
2018-01-18 20:48 ` Mike Snitzer
2018-01-18 20:58 ` Bart Van Assche
2018-01-18 20:58 ` Bart Van Assche
2018-01-18 21:23 ` Mike Snitzer
2018-01-18 21:23 ` Mike Snitzer
2018-01-18 21:37 ` Laurence Oberman
2018-01-18 21:39 ` [dm-devel] " Bart Van Assche
2018-01-18 21:39 ` Bart Van Assche
2018-01-18 21:45 ` Laurence Oberman
2018-01-18 21:45 ` Laurence Oberman
2018-01-18 22:01 ` Mike Snitzer
2018-01-18 22:18 ` Laurence Oberman
2018-01-18 22:20 ` Laurence Oberman
2018-01-18 22:20 ` Laurence Oberman
2018-01-18 22:24 ` Bart Van Assche
2018-01-18 22:24 ` Bart Van Assche
2018-01-18 22:35 ` Laurence Oberman
2018-01-18 22:39 ` Jens Axboe
2018-01-18 22:55 ` Bart Van Assche
2018-01-18 22:55 ` Bart Van Assche
2018-01-18 22:20 ` Bart Van Assche
2018-01-18 22:20 ` Bart Van Assche
2018-01-23 9:22 ` [PATCH] block: neutralize blk_insert_cloned_request IO stall regression (was: Re: [RFC PATCH] blk-mq: fixup RESTART when queue becomes idle) Mike Snitzer
2018-01-23 10:53 ` Ming Lei
2018-01-23 12:15 ` Mike Snitzer
2018-01-23 12:17 ` Ming Lei
2018-01-23 12:43 ` Mike Snitzer
2018-01-23 16:43 ` [PATCH] " Bart Van Assche
2018-01-23 16:43 ` Bart Van Assche
2018-01-19 2:32 ` [RFC PATCH] blk-mq: fixup RESTART when queue becomes idle Ming Lei
2018-01-19 4:02 ` Jens Axboe
2018-01-19 7:26 ` Ming Lei
2018-01-19 15:20 ` Bart Van Assche
2018-01-19 15:20 ` Bart Van Assche
2018-01-19 15:25 ` Jens Axboe
2018-01-19 15:33 ` Ming Lei
2018-01-19 16:06 ` Bart Van Assche
2018-01-19 16:06 ` Bart Van Assche
2018-01-19 15:24 ` Jens Axboe
2018-01-19 15:40 ` Ming Lei
2018-01-19 15:40 ` Ming Lei
2018-01-19 15:48 ` Jens Axboe
2018-01-19 16:05 ` Ming Lei
2018-01-19 16:19 ` Jens Axboe
2018-01-19 16:26 ` Ming Lei
2018-01-19 16:27 ` Jens Axboe
2018-01-19 16:37 ` Ming Lei
2018-01-19 16:41 ` Jens Axboe
2018-01-19 16:41 ` Jens Axboe
2018-01-19 16:47 ` Mike Snitzer
2018-01-19 16:52 ` Jens Axboe
2018-01-19 17:05 ` Ming Lei
2018-01-19 17:09 ` Jens Axboe
2018-01-19 17:20 ` Ming Lei
2018-01-19 17:38 ` Jens Axboe
2018-01-19 18:24 ` Ming Lei
2018-01-19 18:24 ` Ming Lei
2018-01-19 18:33 ` Mike Snitzer
2018-01-19 23:52 ` Ming Lei
2018-01-20 4:27 ` Jens Axboe
2018-01-19 16:13 ` Mike Snitzer
2018-01-19 16:23 ` Jens Axboe
2018-01-19 23:57 ` Ming Lei
2018-01-29 22:37 ` Bart Van Assche
2018-01-19 5:09 ` Bart Van Assche
2018-01-19 5:09 ` Bart Van Assche
2018-01-19 7:34 ` Ming Lei
2018-01-19 19:47 ` Bart Van Assche
2018-01-19 19:47 ` Bart Van Assche
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1516296056.2676.23.camel@wdc.com \
--to=bart.vanassche@wdc.com \
--cc=axboe@kernel.dk \
--cc=dm-devel@redhat.com \
--cc=hch@infradead.org \
--cc=linux-block@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=ming.lei@redhat.com \
--cc=osandov@fb.com \
--cc=snitzer@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.