From: Bart Van Assche <Bart.VanAssche@wdc.com>
To: "snitzer@redhat.com" <snitzer@redhat.com>
Cc: "dm-devel@redhat.com" <dm-devel@redhat.com>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
"hch@infradead.org" <hch@infradead.org>,
"linux-block@vger.kernel.org" <linux-block@vger.kernel.org>,
"osandov@fb.com" <osandov@fb.com>,
"axboe@kernel.dk" <axboe@kernel.dk>,
"ming.lei@redhat.com" <ming.lei@redhat.com>
Subject: Re: [RFC PATCH] blk-mq: fixup RESTART when queue becomes idle
Date: Thu, 18 Jan 2018 22:20:13 +0000 [thread overview]
Message-ID: <1516314012.2676.76.camel@wdc.com> (raw)
In-Reply-To: <20180118220132.GA20860@redhat.com>
T24gVGh1LCAyMDE4LTAxLTE4IGF0IDE3OjAxIC0wNTAwLCBNaWtlIFNuaXR6ZXIgd3JvdGU6DQo+
IEFuZCB5ZXQgTGF1cmVuY2UgY2Fubm90IHJlcHJvZHVjZSBhbnkgc3VjaCBsb2NrdXBzIHdpdGgg
eW91ciB0ZXN0Li4uDQoNCkhtbSAuLi4gbWF5YmUgSSBtaXN1bmRlcnN0b29kIExhdXJlbmNlIGJ1
dCBJIGRvbid0IHRoaW5rIHRoYXQgTGF1cmVuY2UgaGFzDQphbHJlYWR5IHN1Y2NlZWRlZCBhdCBy
dW5uaW5nIGFuIHVubW9kaWZpZWQgdmVyc2lvbiBvZiBteSB0ZXN0cy4gSW4gb25lIG9mIHRoZQ0K
ZS1tYWlscyBMYXVyZW5jZSBzZW50IG1lIHRoaXMgbW9ybmluZyBJIHJlYWQgdGhhdCBoZSBtb2Rp
ZmllZCB0aGVzZSBzY3JpcHRzDQp0byBnZXQgcGFzdCBhIGtlcm5lbCBtb2R1bGUgdW5sb2FkIGZh
aWx1cmUgdGhhdCB3YXMgcmVwb3J0ZWQgd2hpbGUgc3RhcnRpbmcNCnRoZXNlIHRlc3RzLiBTbyB0
aGUgbmV4dCBzdGVwIGlzIHRvIGNoZWNrIHdoaWNoIGNoYW5nZXMgd2VyZSBtYWRlIHRvIHRoZSB0
ZXN0DQpzY3JpcHRzIGFuZCBhbHNvIHdoZXRoZXIgdGhlIHRlc3QgcmVzdWx0cyBhcmUgc3RpbGwg
dmFsaWQuDQoNCj4gQXJlIHlvdSBhYnNvbHV0ZWx5IGNlcnRhaW4gdGhpcyBwYXRjaCBkb2Vzbid0
IGhlbHAgeW91Pw0KPiBodHRwczovL3BhdGNod29yay5rZXJuZWwub3JnL3BhdGNoLzEwMTc0MDM3
Lw0KPiANCj4gSWYgaXQgZG9lc24ndCB0aGVuIHRoYXQgaXMgYWN0dWFsbHkgdmVyeSB1c2VmdWwg
dG8ga25vdy4NCg0KVGhlIGZpcnN0IEkgdHJpZWQgdGhpcyBtb3JuaW5nIGlzIHRvIHJ1biB0aGUg
c3JwLXRlc3Qgc29mdHdhcmUgYWdhaW5zdCBhIG1lcmdlDQpvZiBKZW5zJyBmb3ItbmV4dCBicmFu
Y2ggYW5kIHlvdXIgZG0tNC4xNiBicmFuY2guIFNpbmNlIEkgbm90aWNlZCB0aGF0IHRoZSBkbQ0K
cXVldWUgbG9ja2VkIHVwIEkgcmVpbnNlcnRlZCBhIGJsa19tcV9kZWxheV9ydW5faHdfcXVldWUo
KSBjYWxsIGluIHRoZSBkbSBjb2RlLg0KU2luY2UgZXZlbiB0aGF0IHdhcyBub3Qgc3VmZmljaWVu
dCBJIHRyaWVkIHRvIGtpY2sgdGhlIHF1ZXVlcyB2aWEgZGVidWdmcyAoZm9yDQpzIGluIC9zeXMv
a2VybmVsL2RlYnVnL2Jsb2NrLyovc3RhdGU7IGRvIGVjaG8ga2ljayA+JHM7IGRvbmUpLiBTaW5j
ZSB0aGF0IHdhcw0Kbm90IHN1ZmZpY2llbnQgdG8gcmVzb2x2ZSB0aGUgcXVldWUgc3RhbGwgSSBy
ZXZlcnRlZCB0aGUgZm9sbG93aW5nIHRyZWUgcGF0Y2hlcw0KdGhhdCBhcmUgaW4gSmVucycgdHJl
ZToNCiogImJsay1tcTogaW1wcm92ZSBETSdzIGJsay1tcSBJTyBtZXJnaW5nIHZpYSBibGtfaW5z
ZXJ0X2Nsb25lZF9yZXF1ZXN0IGZlZWRiYWNrIg0KKiAiYmxrLW1xLXNjaGVkOiByZW1vdmUgdW51
c2VkICdjYW5fYmxvY2snIGFyZyBmcm9tIGJsa19tcV9zY2hlZF9pbnNlcnRfcmVxdWVzdCINCiog
ImJsay1tcTogZG9uJ3QgZGlzcGF0Y2ggcmVxdWVzdCBpbiBibGtfbXFfcmVxdWVzdF9kaXJlY3Rf
aXNzdWUgaWYgcXVldWUgaXMgYnVzeSINCg0KT25seSBhZnRlciBJIGhhZCBkb25lIHRoaXMgdGhl
IHNycC10ZXN0IHNvZnR3YXJlIHJhbiBhZ2FpbiB3aXRob3V0IHRyaWdnZXJpbmcNCmRtIHF1ZXVl
IGxvY2t1cHMuIFNvcnJ5IGJ1dCBJIGhhdmUgbm90IHlldCBoYWQgdGhlIHRpbWUgdG8gdGVzdCBw
YXRjaCAiW1JGQ10NCmJsay1tcTogZml4dXAgUkVTVEFSVCB3aGVuIHF1ZXVlIGJlY29tZXMgaWRs
ZSIuDQoNCj4gUGxlYXNlIGp1c3QgZm9jdXMgb24gaGVscGluZyBMYXVyZW5jZSBnZXQgaGlzIHZl
cnkgY2FwYWJsZSB0ZXN0YmVkIHRvDQo+IHJlcHJvZHVjZSB0aGlzIGlzc3VlLiAgT25jZSB3ZSBj
YW4gcmVwcm9kdWNlIHRoZXNlICJ1bmtpbGxhYmxlIiAic3RhbGxzIg0KPiBpbi1ob3VzZSBpdCds
bCBiZSBfbXVjaF8gZWFzaWVyIHRvIGFuYWx5emUgYW5kIGZpeC4NCg0KT0ssIEkgd2lsbCB3b3Jr
IHdpdGggTGF1cmVuY2Ugb24gdGhpcy4gTWF5YmUgTGF1cmVuY2UgYW5kIEkgc2hvdWxkIHdvcmsg
b24gdGhpcw0KYmVmb3JlIGFuYWx5emluZyB0aGUgbG9ja3VwIHRoYXQgd2FzIG1lbnRpb25lZCBh
Ym92ZSBmdXJ0aGVyPw0KDQpCYXJ0Lg==
WARNING: multiple messages have this Message-ID (diff)
From: Bart Van Assche <Bart.VanAssche@wdc.com>
To: "snitzer@redhat.com" <snitzer@redhat.com>
Cc: "dm-devel@redhat.com" <dm-devel@redhat.com>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
"hch@infradead.org" <hch@infradead.org>,
"linux-block@vger.kernel.org" <linux-block@vger.kernel.org>,
"osandov@fb.com" <osandov@fb.com>,
"axboe@kernel.dk" <axboe@kernel.dk>,
"ming.lei@redhat.com" <ming.lei@redhat.com>
Subject: Re: [RFC PATCH] blk-mq: fixup RESTART when queue becomes idle
Date: Thu, 18 Jan 2018 22:20:13 +0000 [thread overview]
Message-ID: <1516314012.2676.76.camel@wdc.com> (raw)
In-Reply-To: <20180118220132.GA20860@redhat.com>
On Thu, 2018-01-18 at 17:01 -0500, Mike Snitzer wrote:
> And yet Laurence cannot reproduce any such lockups with your test...
Hmm ... maybe I misunderstood Laurence but I don't think that Laurence has
already succeeded at running an unmodified version of my tests. In one of the
e-mails Laurence sent me this morning I read that he modified these scripts
to get past a kernel module unload failure that was reported while starting
these tests. So the next step is to check which changes were made to the test
scripts and also whether the test results are still valid.
> Are you absolutely certain this patch doesn't help you?
> https://patchwork.kernel.org/patch/10174037/
>
> If it doesn't then that is actually very useful to know.
The first I tried this morning is to run the srp-test software against a merge
of Jens' for-next branch and your dm-4.16 branch. Since I noticed that the dm
queue locked up I reinserted a blk_mq_delay_run_hw_queue() call in the dm code.
Since even that was not sufficient I tried to kick the queues via debugfs (for
s in /sys/kernel/debug/block/*/state; do echo kick >$s; done). Since that was
not sufficient to resolve the queue stall I reverted the following tree patches
that are in Jens' tree:
* "blk-mq: improve DM's blk-mq IO merging via blk_insert_cloned_request feedback"
* "blk-mq-sched: remove unused 'can_block' arg from blk_mq_sched_insert_request"
* "blk-mq: don't dispatch request in blk_mq_request_direct_issue if queue is busy"
Only after I had done this the srp-test software ran again without triggering
dm queue lockups. Sorry but I have not yet had the time to test patch "[RFC]
blk-mq: fixup RESTART when queue becomes idle".
> Please just focus on helping Laurence get his very capable testbed to
> reproduce this issue. Once we can reproduce these "unkillable" "stalls"
> in-house it'll be _much_ easier to analyze and fix.
OK, I will work with Laurence on this. Maybe Laurence and I should work on this
before analyzing the lockup that was mentioned above further?
Bart.
next prev parent reply other threads:[~2018-01-18 22:20 UTC|newest]
Thread overview: 80+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-01-18 2:41 [RFC PATCH] blk-mq: fixup RESTART when queue becomes idle Ming Lei
2018-01-18 16:50 ` Bart Van Assche
2018-01-18 17:03 ` Mike Snitzer
2018-01-18 17:03 ` Mike Snitzer
2018-01-18 17:20 ` Bart Van Assche
2018-01-18 17:20 ` Bart Van Assche
2018-01-18 18:30 ` Mike Snitzer
2018-01-18 18:47 ` Bart Van Assche
2018-01-18 18:47 ` Bart Van Assche
2018-01-18 20:11 ` Jens Axboe
2018-01-18 20:11 ` Jens Axboe
2018-01-18 20:48 ` Mike Snitzer
2018-01-18 20:58 ` Bart Van Assche
2018-01-18 20:58 ` Bart Van Assche
2018-01-18 21:23 ` Mike Snitzer
2018-01-18 21:23 ` Mike Snitzer
2018-01-18 21:37 ` Laurence Oberman
2018-01-18 21:39 ` [dm-devel] " Bart Van Assche
2018-01-18 21:39 ` Bart Van Assche
2018-01-18 21:45 ` Laurence Oberman
2018-01-18 21:45 ` Laurence Oberman
2018-01-18 22:01 ` Mike Snitzer
2018-01-18 22:18 ` Laurence Oberman
2018-01-18 22:20 ` Laurence Oberman
2018-01-18 22:20 ` Laurence Oberman
2018-01-18 22:24 ` Bart Van Assche
2018-01-18 22:24 ` Bart Van Assche
2018-01-18 22:35 ` Laurence Oberman
2018-01-18 22:39 ` Jens Axboe
2018-01-18 22:55 ` Bart Van Assche
2018-01-18 22:55 ` Bart Van Assche
2018-01-18 22:20 ` Bart Van Assche [this message]
2018-01-18 22:20 ` Bart Van Assche
2018-01-23 9:22 ` [PATCH] block: neutralize blk_insert_cloned_request IO stall regression (was: Re: [RFC PATCH] blk-mq: fixup RESTART when queue becomes idle) Mike Snitzer
2018-01-23 10:53 ` Ming Lei
2018-01-23 12:15 ` Mike Snitzer
2018-01-23 12:17 ` Ming Lei
2018-01-23 12:43 ` Mike Snitzer
2018-01-23 16:43 ` [PATCH] " Bart Van Assche
2018-01-23 16:43 ` Bart Van Assche
2018-01-19 2:32 ` [RFC PATCH] blk-mq: fixup RESTART when queue becomes idle Ming Lei
2018-01-19 4:02 ` Jens Axboe
2018-01-19 7:26 ` Ming Lei
2018-01-19 15:20 ` Bart Van Assche
2018-01-19 15:20 ` Bart Van Assche
2018-01-19 15:25 ` Jens Axboe
2018-01-19 15:33 ` Ming Lei
2018-01-19 16:06 ` Bart Van Assche
2018-01-19 16:06 ` Bart Van Assche
2018-01-19 15:24 ` Jens Axboe
2018-01-19 15:40 ` Ming Lei
2018-01-19 15:40 ` Ming Lei
2018-01-19 15:48 ` Jens Axboe
2018-01-19 16:05 ` Ming Lei
2018-01-19 16:19 ` Jens Axboe
2018-01-19 16:26 ` Ming Lei
2018-01-19 16:27 ` Jens Axboe
2018-01-19 16:37 ` Ming Lei
2018-01-19 16:41 ` Jens Axboe
2018-01-19 16:41 ` Jens Axboe
2018-01-19 16:47 ` Mike Snitzer
2018-01-19 16:52 ` Jens Axboe
2018-01-19 17:05 ` Ming Lei
2018-01-19 17:09 ` Jens Axboe
2018-01-19 17:20 ` Ming Lei
2018-01-19 17:38 ` Jens Axboe
2018-01-19 18:24 ` Ming Lei
2018-01-19 18:24 ` Ming Lei
2018-01-19 18:33 ` Mike Snitzer
2018-01-19 23:52 ` Ming Lei
2018-01-20 4:27 ` Jens Axboe
2018-01-19 16:13 ` Mike Snitzer
2018-01-19 16:23 ` Jens Axboe
2018-01-19 23:57 ` Ming Lei
2018-01-29 22:37 ` Bart Van Assche
2018-01-19 5:09 ` Bart Van Assche
2018-01-19 5:09 ` Bart Van Assche
2018-01-19 7:34 ` Ming Lei
2018-01-19 19:47 ` Bart Van Assche
2018-01-19 19:47 ` Bart Van Assche
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1516314012.2676.76.camel@wdc.com \
--to=bart.vanassche@wdc.com \
--cc=axboe@kernel.dk \
--cc=dm-devel@redhat.com \
--cc=hch@infradead.org \
--cc=linux-block@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=ming.lei@redhat.com \
--cc=osandov@fb.com \
--cc=snitzer@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.