From: Bart Van Assche <Bart.VanAssche@wdc.com>
To: "dm-devel@redhat.com" <dm-devel@redhat.com>,
"linux-scsi@vger.kernel.org" <linux-scsi@vger.kernel.org>,
"linux-block@vger.kernel.org" <linux-block@vger.kernel.org>,
"ming.lei@redhat.com" <ming.lei@redhat.com>
Cc: "loberman@redhat.com" <loberman@redhat.com>
Subject: Re: [v4.13-rc BUG] system lockup when running big buffered write(4M) to IB SRP via mpath
Date: Tue, 8 Aug 2017 23:10:45 +0000 [thread overview]
Message-ID: <1502233843.2686.4.camel@wdc.com> (raw)
In-Reply-To: <20170808141715.GB22763@ming.t460p>
T24gVHVlLCAyMDE3LTA4LTA4IGF0IDIyOjE3ICswODAwLCBNaW5nIExlaSB3cm90ZToNCj4gTGF1
cmVuY2UgYW5kIEkgc2VlIGEgc3lzdGVtIGxvY2t1cCBpc3N1ZSB3aGVuIHJ1bm5pbmcgY29uY3Vy
cmVudA0KPiBiaWcgYnVmZmVyZWQgd3JpdGUoNE0gYnl0ZXMpIHRvIElCIFNSUCBvbiB2NC4xMy1y
YzMuDQo+IFsgLi4uIF0gDQo+IAkjY2F0IGhhbW1lcl93cml0ZS5zaA0KPiAJIyEvYmluL2Jhc2gN
Cj4gCXdoaWxlIHRydWU7IGRvDQo+IAkJZGQgaWY9L2Rldi96ZXJvIG9mPS9kZXYvbWFwcGVyLyQx
IGJzPTQwOTZrIGNvdW50PTgwMCAJDQo+ICAgICAgIGRvbmUNCg0KSGVsbG8gTGF1cmVuY2UsDQoN
CklzIHlvdXIgZ29hbCBwZXJoYXBzIHRvIHNpbXVsYXRlIGEgREROIHdvcmtsb2FkPyBJbiB0aGF0
IGNhc2UgSSB0aGluayB5b3UNCm5lZWQgb2ZsYWc9ZGlyZWN0IHRvIHRoZSBkZCBhcmd1bWVudCBs
aXN0IHN1Y2ggdGhhdCB0aGUgcGFnZSBjYWNoZSB3cml0ZWJhY2sNCmNvZGUgZG9lcyBub3QgYWx0
ZXIgdGhlIHNpemUgb2YgdGhlIHdyaXRlIHJlcXVlc3RzLiBBbnl3YXksIHRoaXMgdGVzdCBzaG91
bGQNCm5vdCB0cmlnZ2VyIGEgbG9ja3VwLiBDYW4geW91IGNoZWNrIHdoZXRoZXIgdGhlIHBhdGNo
IGJlbG93IG1ha2VzIHRoZSBzb2Z0DQpsb2NrdXAgY29tcGxhaW50cyBkaXNhcHBlYXIgKHdpdGhv
dXQgY2hhbmdpbmcgdGhlIGhhbW1lcl93cml0ZS5zaCB0ZXN0DQpzY3JpcHQpPw0KDQpUaGFua3Ms
DQoNCkJhcnQuDQotLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tDQpbUEFUQ0hdIGJsb2NrOiBNYWtlIGJsa19t
cV9kZWxheV9raWNrX3JlcXVldWVfbGlzdCgpIHJlcnVuIHRoZSBxdWV1ZSBhdCBhDQpxdWlldCB0
aW1lDQoNCkRyaXZlcnMgbGlrZSBkbS1tcGF0aCByZXF1ZXVlIHJlcXVlc3RzIGlmIG5vIHBhdGhz
IGFyZSBhdmFpbGFibGUgYW5kDQppZiBjb25maWd1cmVkIHRvIGRvIHNvLiBJZiB0aGUgcXVldWUg
ZGVwdGggaXMgc3VmZmljaWVudGx5IGhpZ2ggYW5kDQp0aGUgcXVldWUgcmVydW5uaW5nIGRlbGF5
IHN1ZmZpY2llbnRseSBzaG9ydCB0aGVuIC5yZXF1ZXVlX3dvcmsgY2FuDQpiZSBxdWV1ZWQgc28g
b2Z0ZW4gdGhhdCBvdGhlciB3b3JrIGl0ZW1zIHF1ZXVlZCBvbiB0aGUgc2FtZSB3b3JrIHF1ZXVl
DQpkbyBub3QgZ2V0IGV4ZWN1dGVkLiBBdm9pZCB0aGF0IHRoaXMgaGFwcGVucyBieSBvbmx5IHJl
cnVubmluZyB0aGUNCnF1ZXVlIGFmdGVyIG5vIGJsa19tcV9kZWxheV9raWNrX3JlcXVldWVfbGlz
dCgpIGNhbGxzIGhhdmUgb2NjdXJyZWQNCmR1cmluZyBAbXNlY3MgbWlsbGlzZWNvbmRzLiBTaW5j
ZSB0aGUgZGV2aWNlIG1hcHBlciBjb3JlIGlzIHRoZSBvbmx5DQp1c2VyIG9mIGJsa19tcV9kZWxh
eV9raWNrX3JlcXVldWVfbGlzdCgpLCBtb2RpZnkgdGhlIGltcGxlbWVudGF0aW9uDQpvZiB0aGlz
IGZ1bmN0aW9uIGluc3RlYWQgb2YgY3JlYXRpbmcgYSBuZXcgZnVuY3Rpb24uDQoNClNpZ25lZC1v
ZmYtYnk6IEJhcnQgVmFuIEFzc2NoZSA8YmFydC52YW5hc3NjaGVAd2RjLmNvbT4NCi0tLQ0KIGJs
b2NrL2Jsay1tcS5jIHwgNCArKy0tDQogMSBmaWxlIGNoYW5nZWQsIDIgaW5zZXJ0aW9ucygrKSwg
MiBkZWxldGlvbnMoLSkNCg0KZGlmZiAtLWdpdCBhL2Jsb2NrL2Jsay1tcS5jIGIvYmxvY2svYmxr
LW1xLmMNCmluZGV4IDA0MWY3YjdmYTBkNi4uOGJmZWEzNmU5MmY5IDEwMDY0NA0KLS0tIGEvYmxv
Y2svYmxrLW1xLmMNCisrKyBiL2Jsb2NrL2Jsay1tcS5jDQpAQCAtNjc5LDggKzY3OSw4IEBAIEVY
UE9SVF9TWU1CT0woYmxrX21xX2tpY2tfcmVxdWV1ZV9saXN0KTsNCiB2b2lkIGJsa19tcV9kZWxh
eV9raWNrX3JlcXVldWVfbGlzdChzdHJ1Y3QgcmVxdWVzdF9xdWV1ZSAqcSwNCiAJCQkJICAgIHVu
c2lnbmVkIGxvbmcgbXNlY3MpDQogew0KLQlrYmxvY2tkX3NjaGVkdWxlX2RlbGF5ZWRfd29yaygm
cS0+cmVxdWV1ZV93b3JrLA0KLQkJCQkgICAgICBtc2Vjc190b19qaWZmaWVzKG1zZWNzKSk7DQor
CWtibG9ja2RfbW9kX2RlbGF5ZWRfd29ya19vbihXT1JLX0NQVV9VTkJPVU5ELCAmcS0+cmVxdWV1
ZV93b3JrLA0KKwkJCQkgICAgbXNlY3NfdG9famlmZmllcyhtc2VjcykpOw0KIH0NCiBFWFBPUlRf
U1lNQk9MKGJsa19tcV9kZWxheV9raWNrX3JlcXVldWVfbGlzdCk7DQo=
WARNING: multiple messages have this Message-ID (diff)
From: Bart Van Assche <Bart.VanAssche@wdc.com>
To: "dm-devel@redhat.com" <dm-devel@redhat.com>,
"linux-scsi@vger.kernel.org" <linux-scsi@vger.kernel.org>,
"linux-block@vger.kernel.org" <linux-block@vger.kernel.org>,
"ming.lei@redhat.com" <ming.lei@redhat.com>
Cc: "loberman@redhat.com" <loberman@redhat.com>
Subject: Re: [v4.13-rc BUG] system lockup when running big buffered write(4M) to IB SRP via mpath
Date: Tue, 8 Aug 2017 23:10:45 +0000 [thread overview]
Message-ID: <1502233843.2686.4.camel@wdc.com> (raw)
In-Reply-To: <20170808141715.GB22763@ming.t460p>
On Tue, 2017-08-08 at 22:17 +0800, Ming Lei wrote:
> Laurence and I see a system lockup issue when running concurrent
> big buffered write(4M bytes) to IB SRP on v4.13-rc3.
> [ ... ]
> #cat hammer_write.sh
> #!/bin/bash
> while true; do
> dd if=/dev/zero of=/dev/mapper/$1 bs=4096k count=800
> done
Hello Laurence,
Is your goal perhaps to simulate a DDN workload? In that case I think you
need oflag=direct to the dd argument list such that the page cache writeback
code does not alter the size of the write requests. Anyway, this test should
not trigger a lockup. Can you check whether the patch below makes the soft
lockup complaints disappear (without changing the hammer_write.sh test
script)?
Thanks,
Bart.
----------------------------------------------------------------------------
[PATCH] block: Make blk_mq_delay_kick_requeue_list() rerun the queue at a
quiet time
Drivers like dm-mpath requeue requests if no paths are available and
if configured to do so. If the queue depth is sufficiently high and
the queue rerunning delay sufficiently short then .requeue_work can
be queued so often that other work items queued on the same work queue
do not get executed. Avoid that this happens by only rerunning the
queue after no blk_mq_delay_kick_requeue_list() calls have occurred
during @msecs milliseconds. Since the device mapper core is the only
user of blk_mq_delay_kick_requeue_list(), modify the implementation
of this function instead of creating a new function.
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
---
block/blk-mq.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/block/blk-mq.c b/block/blk-mq.c
index 041f7b7fa0d6..8bfea36e92f9 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -679,8 +679,8 @@ EXPORT_SYMBOL(blk_mq_kick_requeue_list);
void blk_mq_delay_kick_requeue_list(struct request_queue *q,
unsigned long msecs)
{
- kblockd_schedule_delayed_work(&q->requeue_work,
- msecs_to_jiffies(msecs));
+ kblockd_mod_delayed_work_on(WORK_CPU_UNBOUND, &q->requeue_work,
+ msecs_to_jiffies(msecs));
}
EXPORT_SYMBOL(blk_mq_delay_kick_requeue_list);
next prev parent reply other threads:[~2017-08-08 23:10 UTC|newest]
Thread overview: 16+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-08-08 14:17 [v4.13-rc BUG] system lockup when running big buffered write(4M) to IB SRP via mpath Ming Lei
2017-08-08 14:17 ` Ming Lei
2017-08-08 23:10 ` Bart Van Assche [this message]
2017-08-08 23:10 ` Bart Van Assche
2017-08-09 0:11 ` Laurence Oberman
2017-08-09 2:28 ` Laurence Oberman
2017-08-09 16:43 ` Laurence Oberman
2017-08-09 17:10 ` Bart Van Assche
2017-08-09 17:10 ` Bart Van Assche
2017-08-23 11:35 ` Ming Lei
2017-08-23 12:46 ` Ming Lei
2017-08-23 12:46 ` Ming Lei
2017-08-23 15:12 ` Bart Van Assche
2017-08-23 15:12 ` Bart Van Assche
2017-08-23 16:03 ` Ming Lei
2017-08-23 23:51 ` Mike Snitzer
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1502233843.2686.4.camel@wdc.com \
--to=bart.vanassche@wdc.com \
--cc=dm-devel@redhat.com \
--cc=linux-block@vger.kernel.org \
--cc=linux-scsi@vger.kernel.org \
--cc=loberman@redhat.com \
--cc=ming.lei@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.