* v3.15 dm-mpath regression: cable pull test causes I/O hang
@ 2014-06-27 13:02 Bart Van Assche
2014-06-27 13:33 ` Mike Snitzer
0 siblings, 1 reply; 19+ messages in thread
From: Bart Van Assche @ 2014-06-27 13:02 UTC (permalink / raw)
To: Mike Snitzer, Hannes Reinecke; +Cc: device-mapper development
Hello,
While running a cable pull simulation test with dm_multipath on top of
the SRP initiator driver I noticed that after a few iterations I/O locks
up instead of dm_multipath processing the path failure properly (see also
below for a call trace). At least kernel versions 3.15 and 3.16-rc2 are
vulnerable. This issue does not occur with kernel 3.14. I have tried to
bisect this but gave up when I noticed that I/O locked up completely with
a kernel built from git commit ID e809917735ebf1b9a56c24e877ce0d320baee2ec
(dm mpath: push back requests instead of queueing). But with the bisect I
have been able to narrow down this issue to one of the patches in "Merge
tag 'dm-3.15-changes' of git://git.kernel.org/pub/scm/linux/kernel/git/
device-mapper/linux-dm". Does anyone have a suggestion how to analyze this
further or how to fix this ?
Thanks,
Bart.
systemd-udevd D ffff880831b58000 0 9926 356 0x00000006
ffff8807f8bb79b8 0000000000000002 ffff880831b58000 ffff8807f8bb7fd8
00000000000131c0 00000000000131c0 ffff88083b490000 ffff88085fc53ad0
ffff88085ff6bf38 ffff8807f8bb7a40 0000000000000002 ffffffff81135bd0
Call Trace:
[<ffffffff814bba8d>] io_schedule+0x9d/0x130
[<ffffffff81135bde>] sleep_on_page+0xe/0x20
[<ffffffff814bc0d8>] __wait_on_bit_lock+0x48/0xb0
[<ffffffff81135cea>] __lock_page+0x6a/0x70
[<ffffffff811471df>] truncate_inode_pages_range+0x3ff/0x690
[<ffffffff81147485>] truncate_inode_pages+0x15/0x20
[<ffffffff811d2f85>] kill_bdev+0x35/0x40
[<ffffffff811d4509>] __blkdev_put+0x69/0x1b0
[<ffffffff811d4fb0>] blkdev_put+0x50/0x160
[<ffffffff811d5175>] blkdev_close+0x25/0x30
[<ffffffff81199eda>] __fput+0xea/0x1f0
[<ffffffff8119a02e>] ____fput+0xe/0x10
[<ffffffff81074d9c>] task_work_run+0xac/0xe0
[<ffffffff8104ff37>] do_exit+0x2c7/0xc60
[<ffffffff81051c7c>] do_group_exit+0x4c/0xc0
[<ffffffff81064261>] get_signal_to_deliver+0x2e1/0x940
[<ffffffff81002528>] do_signal+0x48/0x630
[<ffffffff81002b81>] do_notify_resume+0x71/0xc0
[<ffffffff814c1918>] int_signal+0x12/0x17
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: v3.15 dm-mpath regression: cable pull test causes I/O hang
2014-06-27 13:02 v3.15 dm-mpath regression: cable pull test causes I/O hang Bart Van Assche
@ 2014-06-27 13:33 ` Mike Snitzer
2014-06-27 14:18 ` Bart Van Assche
2014-07-02 22:02 ` Mike Snitzer
0 siblings, 2 replies; 19+ messages in thread
From: Mike Snitzer @ 2014-06-27 13:33 UTC (permalink / raw)
To: Bart Van Assche; +Cc: Jun'ichi Nomura, device-mapper development
On Fri, Jun 27 2014 at 9:02am -0400,
Bart Van Assche <bvanassche@acm.org> wrote:
> Hello,
>
> While running a cable pull simulation test with dm_multipath on top of
> the SRP initiator driver I noticed that after a few iterations I/O locks
> up instead of dm_multipath processing the path failure properly (see also
> below for a call trace). At least kernel versions 3.15 and 3.16-rc2 are
> vulnerable. This issue does not occur with kernel 3.14. I have tried to
> bisect this but gave up when I noticed that I/O locked up completely with
> a kernel built from git commit ID e809917735ebf1b9a56c24e877ce0d320baee2ec
> (dm mpath: push back requests instead of queueing). But with the bisect I
> have been able to narrow down this issue to one of the patches in "Merge
> tag 'dm-3.15-changes' of git://git.kernel.org/pub/scm/linux/kernel/git/
> device-mapper/linux-dm". Does anyone have a suggestion how to analyze this
> further or how to fix this ?
>
> Thanks,
>
> Bart.
>
> systemd-udevd D ffff880831b58000 0 9926 356 0x00000006
> ffff8807f8bb79b8 0000000000000002 ffff880831b58000 ffff8807f8bb7fd8
> 00000000000131c0 00000000000131c0 ffff88083b490000 ffff88085fc53ad0
> ffff88085ff6bf38 ffff8807f8bb7a40 0000000000000002 ffffffff81135bd0
> Call Trace:
> [<ffffffff814bba8d>] io_schedule+0x9d/0x130
> [<ffffffff81135bde>] sleep_on_page+0xe/0x20
> [<ffffffff814bc0d8>] __wait_on_bit_lock+0x48/0xb0
> [<ffffffff81135cea>] __lock_page+0x6a/0x70
> [<ffffffff811471df>] truncate_inode_pages_range+0x3ff/0x690
> [<ffffffff81147485>] truncate_inode_pages+0x15/0x20
> [<ffffffff811d2f85>] kill_bdev+0x35/0x40
> [<ffffffff811d4509>] __blkdev_put+0x69/0x1b0
> [<ffffffff811d4fb0>] blkdev_put+0x50/0x160
> [<ffffffff811d5175>] blkdev_close+0x25/0x30
> [<ffffffff81199eda>] __fput+0xea/0x1f0
> [<ffffffff8119a02e>] ____fput+0xe/0x10
> [<ffffffff81074d9c>] task_work_run+0xac/0xe0
> [<ffffffff8104ff37>] do_exit+0x2c7/0xc60
> [<ffffffff81051c7c>] do_group_exit+0x4c/0xc0
> [<ffffffff81064261>] get_signal_to_deliver+0x2e1/0x940
> [<ffffffff81002528>] do_signal+0x48/0x630
> [<ffffffff81002b81>] do_notify_resume+0x71/0xc0
> [<ffffffff814c1918>] int_signal+0x12/0x17
(we've seen sync on last close cause problems when the block device
isn't reachable).
Any other threads that look suspect in output from?:
echo t > /proc/sysrq-trigger
Can you provide your dmsetup table output for the relevant mpath device?
Are you using queue_if_no_path? Also, AFAIK you don't use
multipath-tools, but if by some chance you do please provide your
multipath.conf. I'll attempt to reproduce.
But I'm almost tempted to just revert _all_ of 3.15's dm-mpath changes,
and only reintroduce them once they can pass your testing. I'd like to
avoid that, so Hannes and/or Jun'ichi, it is time for looking at this
seriously.. any help would be very appreciated. For starters, have you
guys done cable pull tests with > 3.15 ?
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: v3.15 dm-mpath regression: cable pull test causes I/O hang
2014-06-27 13:33 ` Mike Snitzer
@ 2014-06-27 14:18 ` Bart Van Assche
2014-07-02 22:02 ` Mike Snitzer
1 sibling, 0 replies; 19+ messages in thread
From: Bart Van Assche @ 2014-06-27 14:18 UTC (permalink / raw)
To: Mike Snitzer; +Cc: Jun'ichi Nomura, device-mapper development
[-- Attachment #1: Type: text/plain, Size: 891 bytes --]
On 06/27/14 15:33, Mike Snitzer wrote:
> (we've seen sync on last close cause problems when the block device
> isn't reachable).
>
> Any other threads that look suspect in output from?:
> echo t > /proc/sysrq-trigger
I have attached the echo w > /proc/sysrq-trigger output since that's the
only output I had saved. I have been analyzing the SysRq-t output but I
haven't found any additional clue in that output.
> Can you provide your dmsetup table output for the relevant mpath device?
> Are you using queue_if_no_path? Also, AFAIK you don't use
> multipath-tools, but if by some chance you do please provide your
> multipath.conf. I'll attempt to reproduce.
The test I ran was a fio data verification test on top of a dm device.
multipathd was running during this test. Is multipath.conf sufficient ?
I have attached my multipath configuration file to this e-mail.
Thanks,
Bart.
[-- Attachment #2: multipath-lockup-v3.16-rc2.txt.xz --]
[-- Type: application/x-xz, Size: 19228 bytes --]
[-- Attachment #3: multipath.conf --]
[-- Type: text/plain, Size: 470 bytes --]
defaults {
polling_interval 1
queue_without_daemon no
features "3 queue_if_no_path pg_init_retries 50"
path_grouping_policy group_by_prio
}
blacklist {
device {
vendor "ATA"
product "*"
}
}
devices {
device {
vendor FUSIONIO
product "*"
features "3 queue_if_no_path pg_init_retries 50"
#hardware_handler "1 alua"
#prio alua
#failback followover
path_checker tur
path_selector "queue-length 0"
#path_selector "round-robin 0"
}
}
[-- Attachment #4: Type: text/plain, Size: 0 bytes --]
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: v3.15 dm-mpath regression: cable pull test causes I/O hang
2014-06-27 13:33 ` Mike Snitzer
2014-06-27 14:18 ` Bart Van Assche
@ 2014-07-02 22:02 ` Mike Snitzer
2014-07-03 5:43 ` Hannes Reinecke
2014-07-03 13:56 ` Bart Van Assche
1 sibling, 2 replies; 19+ messages in thread
From: Mike Snitzer @ 2014-07-02 22:02 UTC (permalink / raw)
To: Bart Van Assche; +Cc: Jun'ichi Nomura, device-mapper development
On Fri, Jun 27 2014 at 9:33am -0400,
Mike Snitzer <snitzer@redhat.com> wrote:
> On Fri, Jun 27 2014 at 9:02am -0400,
> Bart Van Assche <bvanassche@acm.org> wrote:
>
> > Hello,
> >
> > While running a cable pull simulation test with dm_multipath on top of
> > the SRP initiator driver I noticed that after a few iterations I/O locks
> > up instead of dm_multipath processing the path failure properly (see also
> > below for a call trace). At least kernel versions 3.15 and 3.16-rc2 are
> > vulnerable. This issue does not occur with kernel 3.14. I have tried to
> > bisect this but gave up when I noticed that I/O locked up completely with
> > a kernel built from git commit ID e809917735ebf1b9a56c24e877ce0d320baee2ec
> > (dm mpath: push back requests instead of queueing). But with the bisect I
> > have been able to narrow down this issue to one of the patches in "Merge
> > tag 'dm-3.15-changes' of git://git.kernel.org/pub/scm/linux/kernel/git/
> > device-mapper/linux-dm". Does anyone have a suggestion how to analyze this
> > further or how to fix this ?
I still don't have a _known_ fix for your issue but I reviewed commit
e809917735ebf1b9a56c24e877ce0d320baee2ec closer and identified what
looks to be a regression in logic for multipath_busy, it now calls
!pg_ready() instead of directly checking pg_init_in_progress. I think
this is needed (Hannes, what do you think?):
diff --git a/drivers/md/dm-mpath.c b/drivers/md/dm-mpath.c
index 3f6fd9d..561ead6 100644
--- a/drivers/md/dm-mpath.c
+++ b/drivers/md/dm-mpath.c
@@ -373,7 +373,7 @@ static int __must_push_back(struct multipath *m)
dm_noflush_suspending(m->ti)));
}
-#define pg_ready(m) (!(m)->queue_io && !(m)->pg_init_required)
+#define pg_ready(m) (!(m)->queue_io && !(m)->pg_init_required && !(m)->pg_init_in_progress)
/*
* Map cloned requests
^ permalink raw reply related [flat|nested] 19+ messages in thread
* Re: v3.15 dm-mpath regression: cable pull test causes I/O hang
2014-07-02 22:02 ` Mike Snitzer
@ 2014-07-03 5:43 ` Hannes Reinecke
2014-07-03 13:56 ` Bart Van Assche
1 sibling, 0 replies; 19+ messages in thread
From: Hannes Reinecke @ 2014-07-03 5:43 UTC (permalink / raw)
To: Mike Snitzer, Bart Van Assche
Cc: Jun'ichi Nomura, device-mapper development
On 07/03/2014 12:02 AM, Mike Snitzer wrote:
> On Fri, Jun 27 2014 at 9:33am -0400,
> Mike Snitzer <snitzer@redhat.com> wrote:
>
>> On Fri, Jun 27 2014 at 9:02am -0400,
>> Bart Van Assche <bvanassche@acm.org> wrote:
>>
>>> Hello,
>>>
>>> While running a cable pull simulation test with dm_multipath on top of
>>> the SRP initiator driver I noticed that after a few iterations I/O locks
>>> up instead of dm_multipath processing the path failure properly (see also
>>> below for a call trace). At least kernel versions 3.15 and 3.16-rc2 are
>>> vulnerable. This issue does not occur with kernel 3.14. I have tried to
>>> bisect this but gave up when I noticed that I/O locked up completely with
>>> a kernel built from git commit ID e809917735ebf1b9a56c24e877ce0d320baee2ec
>>> (dm mpath: push back requests instead of queueing). But with the bisect I
>>> have been able to narrow down this issue to one of the patches in "Merge
>>> tag 'dm-3.15-changes' of git://git.kernel.org/pub/scm/linux/kernel/git/
>>> device-mapper/linux-dm". Does anyone have a suggestion how to analyze this
>>> further or how to fix this ?
>
> I still don't have a _known_ fix for your issue but I reviewed commit
> e809917735ebf1b9a56c24e877ce0d320baee2ec closer and identified what
> looks to be a regression in logic for multipath_busy, it now calls
> !pg_ready() instead of directly checking pg_init_in_progress. I think
> this is needed (Hannes, what do you think?):
>
> diff --git a/drivers/md/dm-mpath.c b/drivers/md/dm-mpath.c
> index 3f6fd9d..561ead6 100644
> --- a/drivers/md/dm-mpath.c
> +++ b/drivers/md/dm-mpath.c
> @@ -373,7 +373,7 @@ static int __must_push_back(struct multipath *m)
> dm_noflush_suspending(m->ti)));
> }
>
> -#define pg_ready(m) (!(m)->queue_io && !(m)->pg_init_required)
> +#define pg_ready(m) (!(m)->queue_io && !(m)->pg_init_required && !(m)->pg_init_in_progress)
>
> /*
> * Map cloned requests
>
D'oh. You're correct. Can you try with this patch?
Cheers,
Hannes
--
Dr. Hannes Reinecke zSeries & Storage
hare@suse.de +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: v3.15 dm-mpath regression: cable pull test causes I/O hang
2014-07-02 22:02 ` Mike Snitzer
2014-07-03 5:43 ` Hannes Reinecke
@ 2014-07-03 13:56 ` Bart Van Assche
2014-07-03 13:58 ` Hannes Reinecke
2014-07-03 14:05 ` Mike Snitzer
1 sibling, 2 replies; 19+ messages in thread
From: Bart Van Assche @ 2014-07-03 13:56 UTC (permalink / raw)
To: Mike Snitzer; +Cc: Jun'ichi Nomura, device-mapper development
On 07/03/14 00:02, Mike Snitzer wrote:
> On Fri, Jun 27 2014 at 9:33am -0400,
> Mike Snitzer <snitzer@redhat.com> wrote:
>
>> On Fri, Jun 27 2014 at 9:02am -0400,
>> Bart Van Assche <bvanassche@acm.org> wrote:
>>
>>> Hello,
>>>
>>> While running a cable pull simulation test with dm_multipath on top of
>>> the SRP initiator driver I noticed that after a few iterations I/O locks
>>> up instead of dm_multipath processing the path failure properly (see also
>>> below for a call trace). At least kernel versions 3.15 and 3.16-rc2 are
>>> vulnerable. This issue does not occur with kernel 3.14. I have tried to
>>> bisect this but gave up when I noticed that I/O locked up completely with
>>> a kernel built from git commit ID e809917735ebf1b9a56c24e877ce0d320baee2ec
>>> (dm mpath: push back requests instead of queueing). But with the bisect I
>>> have been able to narrow down this issue to one of the patches in "Merge
>>> tag 'dm-3.15-changes' of git://git.kernel.org/pub/scm/linux/kernel/git/
>>> device-mapper/linux-dm". Does anyone have a suggestion how to analyze this
>>> further or how to fix this ?
>
> I still don't have a _known_ fix for your issue but I reviewed commit
> e809917735ebf1b9a56c24e877ce0d320baee2ec closer and identified what
> looks to be a regression in logic for multipath_busy, it now calls
> !pg_ready() instead of directly checking pg_init_in_progress. I think
> this is needed (Hannes, what do you think?):
>
> diff --git a/drivers/md/dm-mpath.c b/drivers/md/dm-mpath.c
> index 3f6fd9d..561ead6 100644
> --- a/drivers/md/dm-mpath.c
> +++ b/drivers/md/dm-mpath.c
> @@ -373,7 +373,7 @@ static int __must_push_back(struct multipath *m)
> dm_noflush_suspending(m->ti)));
> }
>
> -#define pg_ready(m) (!(m)->queue_io && !(m)->pg_init_required)
> +#define pg_ready(m) (!(m)->queue_io && !(m)->pg_init_required && !(m)->pg_init_in_progress)
>
> /*
> * Map cloned requests
Hello Mike,
Sorry but even with this patch applied and additionally with commit IDs
86d56134f1b6 ("kobject: Make support for uevent_helper optional") and
bcccff93af35 ("kobject: don't block for each kobject_uevent") reverted
my multipath test still hangs after a few iterations. I also reran the
same test with kernel 3.14.3 and it is still running after 30 iterations.
Bart.
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: v3.15 dm-mpath regression: cable pull test causes I/O hang
2014-07-03 13:56 ` Bart Van Assche
@ 2014-07-03 13:58 ` Hannes Reinecke
2014-07-03 14:05 ` Mike Snitzer
1 sibling, 0 replies; 19+ messages in thread
From: Hannes Reinecke @ 2014-07-03 13:58 UTC (permalink / raw)
To: Bart Van Assche, Mike Snitzer
Cc: Jun'ichi Nomura, device-mapper development
On 07/03/2014 03:56 PM, Bart Van Assche wrote:
> On 07/03/14 00:02, Mike Snitzer wrote:
>> On Fri, Jun 27 2014 at 9:33am -0400,
>> Mike Snitzer <snitzer@redhat.com> wrote:
>>
>>> On Fri, Jun 27 2014 at 9:02am -0400,
>>> Bart Van Assche <bvanassche@acm.org> wrote:
>>>
>>>> Hello,
>>>>
>>>> While running a cable pull simulation test with dm_multipath on top of
>>>> the SRP initiator driver I noticed that after a few iterations I/O locks
>>>> up instead of dm_multipath processing the path failure properly (see also
>>>> below for a call trace). At least kernel versions 3.15 and 3.16-rc2 are
>>>> vulnerable. This issue does not occur with kernel 3.14. I have tried to
>>>> bisect this but gave up when I noticed that I/O locked up completely with
>>>> a kernel built from git commit ID e809917735ebf1b9a56c24e877ce0d320baee2ec
>>>> (dm mpath: push back requests instead of queueing). But with the bisect I
>>>> have been able to narrow down this issue to one of the patches in "Merge
>>>> tag 'dm-3.15-changes' of git://git.kernel.org/pub/scm/linux/kernel/git/
>>>> device-mapper/linux-dm". Does anyone have a suggestion how to analyze this
>>>> further or how to fix this ?
>>
>> I still don't have a _known_ fix for your issue but I reviewed commit
>> e809917735ebf1b9a56c24e877ce0d320baee2ec closer and identified what
>> looks to be a regression in logic for multipath_busy, it now calls
>> !pg_ready() instead of directly checking pg_init_in_progress. I think
>> this is needed (Hannes, what do you think?):
>>
>> diff --git a/drivers/md/dm-mpath.c b/drivers/md/dm-mpath.c
>> index 3f6fd9d..561ead6 100644
>> --- a/drivers/md/dm-mpath.c
>> +++ b/drivers/md/dm-mpath.c
>> @@ -373,7 +373,7 @@ static int __must_push_back(struct multipath *m)
>> dm_noflush_suspending(m->ti)));
>> }
>>
>> -#define pg_ready(m) (!(m)->queue_io && !(m)->pg_init_required)
>> +#define pg_ready(m) (!(m)->queue_io && !(m)->pg_init_required && !(m)->pg_init_in_progress)
>>
>> /*
>> * Map cloned requests
>
> Hello Mike,
>
> Sorry but even with this patch applied and additionally with commit IDs
> 86d56134f1b6 ("kobject: Make support for uevent_helper optional") and
> bcccff93af35 ("kobject: don't block for each kobject_uevent") reverted
> my multipath test still hangs after a few iterations. I also reran the
> same test with kernel 3.14.3 and it is still running after 30 iterations.
>
Hmm. Would've been too easy.
Sigh.
Cheers,
Hannes
--
Dr. Hannes Reinecke zSeries & Storage
hare@suse.de +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: v3.15 dm-mpath regression: cable pull test causes I/O hang
2014-07-03 13:56 ` Bart Van Assche
2014-07-03 13:58 ` Hannes Reinecke
@ 2014-07-03 14:05 ` Mike Snitzer
2014-07-03 14:15 ` Hannes Reinecke
2014-07-03 14:34 ` Bart Van Assche
1 sibling, 2 replies; 19+ messages in thread
From: Mike Snitzer @ 2014-07-03 14:05 UTC (permalink / raw)
To: Bart Van Assche; +Cc: Jun'ichi Nomura, device-mapper development
On Thu, Jul 03 2014 at 9:56am -0400,
Bart Van Assche <bvanassche@acm.org> wrote:
> On 07/03/14 00:02, Mike Snitzer wrote:
> > On Fri, Jun 27 2014 at 9:33am -0400,
> > Mike Snitzer <snitzer@redhat.com> wrote:
> >
> >> On Fri, Jun 27 2014 at 9:02am -0400,
> >> Bart Van Assche <bvanassche@acm.org> wrote:
> >>
> >>> Hello,
> >>>
> >>> While running a cable pull simulation test with dm_multipath on top of
> >>> the SRP initiator driver I noticed that after a few iterations I/O locks
> >>> up instead of dm_multipath processing the path failure properly (see also
> >>> below for a call trace). At least kernel versions 3.15 and 3.16-rc2 are
> >>> vulnerable. This issue does not occur with kernel 3.14. I have tried to
> >>> bisect this but gave up when I noticed that I/O locked up completely with
> >>> a kernel built from git commit ID e809917735ebf1b9a56c24e877ce0d320baee2ec
> >>> (dm mpath: push back requests instead of queueing). But with the bisect I
> >>> have been able to narrow down this issue to one of the patches in "Merge
> >>> tag 'dm-3.15-changes' of git://git.kernel.org/pub/scm/linux/kernel/git/
> >>> device-mapper/linux-dm". Does anyone have a suggestion how to analyze this
> >>> further or how to fix this ?
> >
> > I still don't have a _known_ fix for your issue but I reviewed commit
> > e809917735ebf1b9a56c24e877ce0d320baee2ec closer and identified what
> > looks to be a regression in logic for multipath_busy, it now calls
> > !pg_ready() instead of directly checking pg_init_in_progress. I think
> > this is needed (Hannes, what do you think?):
> >
> > diff --git a/drivers/md/dm-mpath.c b/drivers/md/dm-mpath.c
> > index 3f6fd9d..561ead6 100644
> > --- a/drivers/md/dm-mpath.c
> > +++ b/drivers/md/dm-mpath.c
> > @@ -373,7 +373,7 @@ static int __must_push_back(struct multipath *m)
> > dm_noflush_suspending(m->ti)));
> > }
> >
> > -#define pg_ready(m) (!(m)->queue_io && !(m)->pg_init_required)
> > +#define pg_ready(m) (!(m)->queue_io && !(m)->pg_init_required && !(m)->pg_init_in_progress)
> >
> > /*
> > * Map cloned requests
>
> Hello Mike,
>
> Sorry but even with this patch applied and additionally with commit IDs
> 86d56134f1b6 ("kobject: Make support for uevent_helper optional") and
> bcccff93af35 ("kobject: don't block for each kobject_uevent") reverted
> my multipath test still hangs after a few iterations. I also reran the
> same test with kernel 3.14.3 and it is still running after 30 iterations.
OK, thanks for testing though! I still think the patch is needed.
You are using queue_if_no_path, do you see hangs due to paths not being
restored after the "cable" is restored? Any errors in the multipathd
userspace logging? Or abnormal errors in kernel? Basically I'm looking
for some other clue besides the hung task timeout spew.
How easy would it be to replicate your testbed? Is it uniquely FIO hw
dependent? How are you simulating the cable pull tests?
I'd love to setup a testbed that would enable me to chase this more
interactively rather than punting to you for testing.
Hannes, do you have a testbed for heavy cable pull testing? Are you
able to replicate these hangs?
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: v3.15 dm-mpath regression: cable pull test causes I/O hang
2014-07-03 14:05 ` Mike Snitzer
@ 2014-07-03 14:15 ` Hannes Reinecke
2014-07-03 14:18 ` Mike Snitzer
2014-07-03 14:34 ` Bart Van Assche
1 sibling, 1 reply; 19+ messages in thread
From: Hannes Reinecke @ 2014-07-03 14:15 UTC (permalink / raw)
To: Mike Snitzer, Bart Van Assche
Cc: Jun'ichi Nomura, device-mapper development
On 07/03/2014 04:05 PM, Mike Snitzer wrote:
> On Thu, Jul 03 2014 at 9:56am -0400,
> Bart Van Assche <bvanassche@acm.org> wrote:
>
>> On 07/03/14 00:02, Mike Snitzer wrote:
>>> On Fri, Jun 27 2014 at 9:33am -0400,
>>> Mike Snitzer <snitzer@redhat.com> wrote:
>>>
>>>> On Fri, Jun 27 2014 at 9:02am -0400,
>>>> Bart Van Assche <bvanassche@acm.org> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> While running a cable pull simulation test with dm_multipath on top of
>>>>> the SRP initiator driver I noticed that after a few iterations I/O locks
>>>>> up instead of dm_multipath processing the path failure properly (see also
>>>>> below for a call trace). At least kernel versions 3.15 and 3.16-rc2 are
>>>>> vulnerable. This issue does not occur with kernel 3.14. I have tried to
>>>>> bisect this but gave up when I noticed that I/O locked up completely with
>>>>> a kernel built from git commit ID e809917735ebf1b9a56c24e877ce0d320baee2ec
>>>>> (dm mpath: push back requests instead of queueing). But with the bisect I
>>>>> have been able to narrow down this issue to one of the patches in "Merge
>>>>> tag 'dm-3.15-changes' of git://git.kernel.org/pub/scm/linux/kernel/git/
>>>>> device-mapper/linux-dm". Does anyone have a suggestion how to analyze this
>>>>> further or how to fix this ?
>>>
>>> I still don't have a _known_ fix for your issue but I reviewed commit
>>> e809917735ebf1b9a56c24e877ce0d320baee2ec closer and identified what
>>> looks to be a regression in logic for multipath_busy, it now calls
>>> !pg_ready() instead of directly checking pg_init_in_progress. I think
>>> this is needed (Hannes, what do you think?):
>>>
>>> diff --git a/drivers/md/dm-mpath.c b/drivers/md/dm-mpath.c
>>> index 3f6fd9d..561ead6 100644
>>> --- a/drivers/md/dm-mpath.c
>>> +++ b/drivers/md/dm-mpath.c
>>> @@ -373,7 +373,7 @@ static int __must_push_back(struct multipath *m)
>>> dm_noflush_suspending(m->ti)));
>>> }
>>>
>>> -#define pg_ready(m) (!(m)->queue_io && !(m)->pg_init_required)
>>> +#define pg_ready(m) (!(m)->queue_io && !(m)->pg_init_required && !(m)->pg_init_in_progress)
>>>
>>> /*
>>> * Map cloned requests
>>
>> Hello Mike,
>>
>> Sorry but even with this patch applied and additionally with commit IDs
>> 86d56134f1b6 ("kobject: Make support for uevent_helper optional") and
>> bcccff93af35 ("kobject: don't block for each kobject_uevent") reverted
>> my multipath test still hangs after a few iterations. I also reran the
>> same test with kernel 3.14.3 and it is still running after 30 iterations.
>
> OK, thanks for testing though! I still think the patch is needed.
>
> You are using queue_if_no_path, do you see hangs due to paths not being
> restored after the "cable" is restored? Any errors in the multipathd
> userspace logging? Or abnormal errors in kernel? Basically I'm looking
> for some other clue besides the hung task timeout spew.
>
> How easy would it be to replicate your testbed? Is it uniquely FIO hw
> dependent? How are you simulating the cable pull tests?
>
> I'd love to setup a testbed that would enable me to chase this more
> interactively rather than punting to you for testing.
>
> Hannes, do you have a testbed for heavy cable pull testing? Are you
> able to replicate these hangs?
>
Yes, I do. But sadly I've been tied up with polishing up SLES12
(release deadline is looming nearer) and for some inexplicable
reason management seems to find releasing a product more important
than working on mainline issue ...
But I hope to find some time soonish (ie start of next week) to work
on this; it's the very next thing on my to-do list.
Cheers,
Hannes
--
Dr. Hannes Reinecke zSeries & Storage
hare@suse.de +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: v3.15 dm-mpath regression: cable pull test causes I/O hang
2014-07-03 14:15 ` Hannes Reinecke
@ 2014-07-03 14:18 ` Mike Snitzer
0 siblings, 0 replies; 19+ messages in thread
From: Mike Snitzer @ 2014-07-03 14:18 UTC (permalink / raw)
To: Hannes Reinecke
Cc: Jun'ichi Nomura, device-mapper development, Bart Van Assche
On Thu, Jul 03 2014 at 10:15am -0400,
Hannes Reinecke <hare@suse.de> wrote:
> On 07/03/2014 04:05 PM, Mike Snitzer wrote:
> >
> >Hannes, do you have a testbed for heavy cable pull testing? Are you
> >able to replicate these hangs?
> >
> Yes, I do. But sadly I've been tied up with polishing up SLES12
> (release deadline is looming nearer) and for some inexplicable
> reason management seems to find releasing a product more important
> than working on mainline issue ...
OK, so long as you didn't include these dm-mpath changes.. but if you
did then you could kill 2 birds with one stone ;)
> But I hope to find some time soonish (ie start of next week) to work
> on this; it's the very next thing on my to-do list.
OK, I really appreciate it, thanks.
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: v3.15 dm-mpath regression: cable pull test causes I/O hang
2014-07-03 14:05 ` Mike Snitzer
2014-07-03 14:15 ` Hannes Reinecke
@ 2014-07-03 14:34 ` Bart Van Assche
2014-07-03 15:00 ` Mike Snitzer
2014-07-04 3:10 ` Junichi Nomura
1 sibling, 2 replies; 19+ messages in thread
From: Bart Van Assche @ 2014-07-03 14:34 UTC (permalink / raw)
To: Mike Snitzer; +Cc: Jun'ichi Nomura, device-mapper development
On 07/03/14 16:05, Mike Snitzer wrote:
> How easy would it be to replicate your testbed? Is it uniquely FIO hw
> dependent? How are you simulating the cable pull tests?
>
> I'd love to setup a testbed that would enable me to chase this more
> interactively rather than punting to you for testing.
Hello Mike,
The only nonstandard hardware that is required to run my test is a pair
of InfiniBand HCA's and an IB cable to connect these back-to-back. The
test I ran is as follows:
* Let an SRP initiator log in to an SRP target system.
* Start multipathd and srpd.
* Start a fio data integrity test on the initiator system on top of
/dev/dm-0.
* From the target system simulate a cable pull by disabling IB traffic
via the ibportstate command.
* After a random delay, unload and reload SCST and the IB stack. This
makes the IB ports operational again.
* After a random delay, repeat the previous two steps.
If you want I can send you the scripts I use to run this test and also
the instructions that are necessary to build and install the SCST SRP
target driver.
Bart.
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: v3.15 dm-mpath regression: cable pull test causes I/O hang
2014-07-03 14:34 ` Bart Van Assche
@ 2014-07-03 15:00 ` Mike Snitzer
2014-07-07 13:28 ` Bart Van Assche
2014-07-04 3:10 ` Junichi Nomura
1 sibling, 1 reply; 19+ messages in thread
From: Mike Snitzer @ 2014-07-03 15:00 UTC (permalink / raw)
To: Bart Van Assche; +Cc: Jun'ichi Nomura, device-mapper development
On Thu, Jul 03 2014 at 10:34am -0400,
Bart Van Assche <bvanassche@acm.org> wrote:
> On 07/03/14 16:05, Mike Snitzer wrote:
> > How easy would it be to replicate your testbed? Is it uniquely FIO hw
> > dependent? How are you simulating the cable pull tests?
> >
> > I'd love to setup a testbed that would enable me to chase this more
> > interactively rather than punting to you for testing.
>
> Hello Mike,
>
> The only nonstandard hardware that is required to run my test is a pair
> of InfiniBand HCA's and an IB cable to connect these back-to-back. The
> test I ran is as follows:
> * Let an SRP initiator log in to an SRP target system.
> * Start multipathd and srpd.
> * Start a fio data integrity test on the initiator system on top of
> /dev/dm-0.
> * From the target system simulate a cable pull by disabling IB traffic
> via the ibportstate command.
> * After a random delay, unload and reload SCST and the IB stack. This
> makes the IB ports operational again.
> * After a random delay, repeat the previous two steps.
I'll work on getting some IB cards. But I _should_ be able to achieve
the same using iSCSI right?
> If you want I can send you the scripts I use to run this test and also
> the instructions that are necessary to build and install the SCST SRP
> target driver.
Please do, thanks!
Also, Red Hat has a rather extensive battery of dm-mpath FC cable pull
tests for RHEL but these dm-mpath changes haven't been included in any
RHEL yet.. I _could_ port these upstream changes to a test RHEL7 kernel
just to leverage the RHEL-based testing.
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: v3.15 dm-mpath regression: cable pull test causes I/O hang
2014-07-03 14:34 ` Bart Van Assche
2014-07-03 15:00 ` Mike Snitzer
@ 2014-07-04 3:10 ` Junichi Nomura
2014-07-07 13:40 ` Bart Van Assche
1 sibling, 1 reply; 19+ messages in thread
From: Junichi Nomura @ 2014-07-04 3:10 UTC (permalink / raw)
To: device-mapper development, Bart Van Assche; +Cc: Mike Snitzer
On 07/03/14 23:34, Bart Van Assche wrote:
> On 07/03/14 16:05, Mike Snitzer wrote:
> The only nonstandard hardware that is required to run my test is a pair
> of InfiniBand HCA's and an IB cable to connect these back-to-back. The
> test I ran is as follows:
> * Let an SRP initiator log in to an SRP target system.
> * Start multipathd and srpd.
> * Start a fio data integrity test on the initiator system on top of
> /dev/dm-0.
> * From the target system simulate a cable pull by disabling IB traffic
> via the ibportstate command.
> * After a random delay, unload and reload SCST and the IB stack. This
> makes the IB ports operational again.
> * After a random delay, repeat the previous two steps.
>
> If you want I can send you the scripts I use to run this test and also
> the instructions that are necessary to build and install the SCST SRP
> target driver.
Hi Bart,
could you collect output of 'dmsetup table' on the test machine?
My particular interest is whether pg_init is involved or not
since Hannes's patch made major surgery on pg_init processing.
Also, output of the followings while the problem is happening might help:
dmsetup status
dmsetup info -c
cat /proc/diskstat
multipathd -k"show paths"
multipathd -k"show maps status"
multipathd -k"show maps stats"
--
Jun'ichi Nomura, NEC Corporation
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: v3.15 dm-mpath regression: cable pull test causes I/O hang
2014-07-03 15:00 ` Mike Snitzer
@ 2014-07-07 13:28 ` Bart Van Assche
0 siblings, 0 replies; 19+ messages in thread
From: Bart Van Assche @ 2014-07-07 13:28 UTC (permalink / raw)
To: Mike Snitzer; +Cc: Jun'ichi Nomura, device-mapper development
[-- Attachment #1: Type: text/plain, Size: 2763 bytes --]
On 07/03/14 17:00, Mike Snitzer wrote:
> On Thu, Jul 03 2014 at 10:34am -0400,
> Bart Van Assche <bvanassche@acm.org> wrote:
>
>> On 07/03/14 16:05, Mike Snitzer wrote:
>>> How easy would it be to replicate your testbed? Is it uniquely FIO hw
>>> dependent? How are you simulating the cable pull tests?
>>>
>>> I'd love to setup a testbed that would enable me to chase this more
>>> interactively rather than punting to you for testing.
>>
>> Hello Mike,
>>
>> The only nonstandard hardware that is required to run my test is a pair
>> of InfiniBand HCA's and an IB cable to connect these back-to-back. The
>> test I ran is as follows:
>> * Let an SRP initiator log in to an SRP target system.
>> * Start multipathd and srpd.
>> * Start a fio data integrity test on the initiator system on top of
>> /dev/dm-0.
>> * From the target system simulate a cable pull by disabling IB traffic
>> via the ibportstate command.
>> * After a random delay, unload and reload SCST and the IB stack. This
>> makes the IB ports operational again.
>> * After a random delay, repeat the previous two steps.
>
> I'll work on getting some IB cards. But I _should_ be able to achieve
> the same using iSCSI right?
I'm not sure. There are differences between the SRP and iSCSI initiator
that could matter here, e.g. that the SRP initiator triggers
scsi_remove_host() some time after a path failure occurred but the iSCSI
initiator not. So far I have not yet been able to trigger this issue
with the iSCSI initiator with replacement_timeout = 1 and by using the
following loop to simulate path failures: while true; do iptables -A
INPUT -p tcp --destination-port 3260 -j DROP; sleep 10; iptables -D
INPUT -p tcp --destination-port 3260 -j DROP; sleep 10; done
>> If you want I can send you the scripts I use to run this test and also
>> the instructions that are necessary to build and install the SCST SRP
>> target driver.
>
> Please do, thanks!
The test I run at the initiator side is as follows:
# modprobe ib_srp
# systemctl restart srpd
# systemctl start multipathd
# mkfs.ext4 -FO ^has_journal /dev/dm-0
# umount /mnt; fsck /dev/dm-0 && mount /dev/dm-0 /mnt && rm -f
/mnt/test* && fio --verify=md5 --rw=randwrite --size=10M --bs=4K
--iodepth=64 --sync=1 --direct=1 --ioengine=libaio --directory=/mnt
--name=test --thread --numjobs=1 --loops=$((10**9))
The script I run at the target side is as follows (should also be
possible with the upstream SRP target driver instead of SCST):
* Download, build and install SCST.
* Create a configuration file (/etc/scst.conf) in which /dev/ram0 is
exported via the vdisk_blockio driver.
* Start SCST.
* Run the attached toggle-ib-port-loop script e.g. as follows:
initiator=${initiator_host_name} toggle-ib-port-loop
Bart.
[-- Attachment #2: toggle-ib-port-loop --]
[-- Type: text/plain, Size: 1473 bytes --]
#!/bin/bash
# How to start this test.
# On the initiator system, run:
# ~bart/bin/reload-srp-initiator
# /etc/init.d/srpd start
# mkfs.ext4 -O ^has_journal /dev/sdb
# /etc/init.d/multipathd start
# umount /mnt; mount /dev/dm-0 /mnt && rm -f /mnt/test* && ~bart/bin/fio-stress-test-6 /mnt 16
# On the target system, run:
# initiator=antec ~bart/software/tools/toggle-ib-port-loop
function port_guid() {
local gid guid
gid="$(</sys/class/infiniband/mlx4_0/ports/$1/gids/0)" || return $?
guid="${gid#fe80:0000:0000:0000}"
echo "0x${guid//:/}"
}
if [ -z "${initiator}" ]; then
echo "Error: variable \${initiator} has not been set"
exit 1
fi
guid1="$(port_guid 1)"
guid2="$(port_guid 2)"
set -x
/etc/init.d/srpd stop
while true; do
ssh ${initiator} ibportstate -G "$guid1" 1 disable
ssh ${initiator} ibportstate -G "$guid2" 2 disable
sleep $((RANDOM*150/32767))
/etc/init.d/scst stop
/etc/init.d/opensmd stop
/etc/init.d/openibd stop
for m in mlx4_en mlx4_ib mlx4_core; do
modprobe -r $m
done
/etc/init.d/openibd start
/etc/init.d/opensmd start
umount /dev/sr1
ibstat |
sed -n 's/^[[:blank:]]*Port GUID: 0x\(..\)\(..\)\(..\)....\(..\)\(..\)\(..\)/00:\2:\3:\4:\5:\6/p' |
while read a; do
p="$(cd /sys/class/net && grep -lw $a */address)"
if [ -n "$p" ]; then
ifup "$(dirname $p)"
fi
done
/etc/init.d/scst restart
sleep $((30 + RANDOM*30/32767))
done
[-- Attachment #3: Type: text/plain, Size: 0 bytes --]
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: v3.15 dm-mpath regression: cable pull test causes I/O hang
2014-07-04 3:10 ` Junichi Nomura
@ 2014-07-07 13:40 ` Bart Van Assche
2014-07-08 0:55 ` Junichi Nomura
0 siblings, 1 reply; 19+ messages in thread
From: Bart Van Assche @ 2014-07-07 13:40 UTC (permalink / raw)
To: Junichi Nomura, device-mapper development; +Cc: Mike Snitzer
[-- Attachment #1: Type: text/plain, Size: 654 bytes --]
On 07/04/14 05:10, Junichi Nomura wrote:
> could you collect output of 'dmsetup table' on the test machine?
> My particular interest is whether pg_init is involved or not
> since Hannes's patch made major surgery on pg_init processing.
>
> Also, output of the followings while the problem is happening might help:
> dmsetup status
> dmsetup info -c
> cat /proc/diskstat
> multipathd -k"show paths"
> multipathd -k"show maps status"
> multipathd -k"show maps stats"
Hello Junichi,
Thanks for looking into this issue. I have attached the requested
information to this e-mail for a test run with kernel 3.16-rc4 at the
initiator side.
Bart.
[-- Attachment #2: multipath-status.txt --]
[-- Type: text/plain, Size: 34727 bytes --]
# dmsetup table
26133373139386639: 0 5368709120 multipath 3 queue_if_no_path pg_init_retries 50 0 1 1 service-time 0 2 2 65:16 1000 1 65:192 1000 1
26230643330303865: 0 5368709120 multipath 3 queue_if_no_path pg_init_retries 50 0 1 1 service-time 0 2 2 65:48 1000 1 65:208 1000 1
23834333461383137: 0 5368709120 multipath 3 queue_if_no_path pg_init_retries 50 0 1 1 service-time 0 2 2 8:208 1000 1 65:128 1000 1
26361626563396362: 0 5368709120 multipath 3 queue_if_no_path pg_init_retries 50 0 1 1 service-time 0 2 2 65:80 1000 1 65:224 1000 1
26363633765623538: 0 256000 multipath 3 queue_if_no_path pg_init_retries 50 0 1 1 service-time 0 2 2 8:32 1000 1 8:128 1000 1
23832346438613834: 0 256000 multipath 3 queue_if_no_path pg_init_retries 50 0 1 1 service-time 0 2 2 8:96 1000 1 65:0 1000 1
26638323034336331: 0 256000 multipath 3 queue_if_no_path pg_init_retries 50 0 1 1 service-time 0 2 2 8:80 1000 1 8:224 1000 1
23937393633303630: 0 5368709120 multipath 3 queue_if_no_path pg_init_retries 50 0 1 1 service-time 0 2 2 8:240 1000 1 65:160 1000 1
23931656631326633: 0 256000 multipath 3 queue_if_no_path pg_init_retries 50 0 1 1 service-time 0 2 2 8:112 1000 1 65:32 1000 1
26665353936313532: 0 5368709120 multipath 3 queue_if_no_path pg_init_retries 50 0 1 1 service-time 0 2 2 65:176 1000 1 66:16 1000 1
26135303862613661: 0 256000 multipath 3 queue_if_no_path pg_init_retries 50 0 1 1 service-time 0 2 2 8:144 1000 1 65:64 1000 1
26439316335316263: 0 5368709120 multipath 3 queue_if_no_path pg_init_retries 50 0 1 1 service-time 0 2 2 65:112 1000 1 65:240 1000 1
26564666266393235: 0 5368709120 multipath 3 queue_if_no_path pg_init_retries 50 0 1 1 service-time 0 2 2 65:144 1000 1 66:0 1000 1
26562383264626236: 0 256000 multipath 3 queue_if_no_path pg_init_retries 50 0 1 1 service-time 0 2 2 8:64 1000 1 8:192 1000 1
26236616132323164: 0 256000 multipath 3 queue_if_no_path pg_init_retries 50 0 1 1 service-time 0 2 2 8:176 1000 1 65:96 1000 1
26466363537333266: 0 256000 multipath 3 queue_if_no_path pg_init_retries 50 0 1 1 service-time 0 2 2 8:48 1000 1 8:160 1000 1
# dmsetup status
26133373139386639: 0 5368709120 multipath 2 0 0 0 1 1 A 0 2 2 65:16 A 0 0 1 65:192 A 0 0 1
26230643330303865: 0 5368709120 multipath 2 0 0 0 1 1 A 0 2 2 65:48 A 0 0 1 65:208 A 0 0 1
23834333461383137: 0 5368709120 multipath 2 1 0 0 1 1 E 0 2 2 8:208 A 0 0 1 65:128 A 0 0 1
26361626563396362: 0 5368709120 multipath 2 1 0 0 1 1 E 0 2 2 65:80 A 0 0 1 65:224 A 0 0 1
26363633765623538: 0 256000 multipath 2 0 0 0 1 1 A 0 2 2 8:32 A 0 0 1 8:128 A 0 0 1
23832346438613834: 0 256000 multipath 2 1 0 0 1 1 E 0 2 2 8:96 A 0 0 1 65:0 A 0 0 1
26638323034336331: 0 256000 multipath 2 1 0 0 1 1 E 0 2 2 8:80 A 0 0 1 8:224 A 0 0 1
23937393633303630: 0 5368709120 multipath 2 0 0 0 1 1 A 0 2 2 8:240 A 0 0 1 65:160 A 0 0 1
23931656631326633: 0 256000 multipath 2 1 0 0 1 1 E 0 2 2 8:112 A 0 0 1 65:32 A 0 0 1
26665353936313532: 0 5368709120 multipath 2 0 0 0 1 1 A 0 2 2 65:176 A 0 0 1 66:16 A 0 0 1
26135303862613661: 0 256000 multipath 2 0 0 0 1 1 A 0 2 2 8:144 A 0 0 1 65:64 A 0 0 1
26439316335316263: 0 5368709120 multipath 2 0 0 0 1 1 A 0 2 2 65:112 A 0 0 1 65:240 A 0 0 1
26564666266393235: 0 5368709120 multipath 2 0 0 0 1 1 A 0 2 2 65:144 A 0 0 1 66:0 A 0 0 1
26562383264626236: 0 256000 multipath 2 1 0 0 1 1 E 0 2 2 8:64 A 0 0 1 8:192 A 0 0 1
26236616132323164: 0 256000 multipath 2 0 0 0 1 1 A 0 2 2 8:176 A 0 0 1 65:96 A 0 0 1
26466363537333266: 0 256000 multipath 2 1 0 0 1 1 E 0 2 2 8:48 A 0 0 1 8:160 A 0 0 1
# dmsetup info -c
Name Maj Min Stat Open Targ Event UUID
26133373139386639 253 10 L--w 0 1 0 mpath-26133373139386639
26230643330303865 253 11 L--w 0 1 0 mpath-26230643330303865
23834333461383137 253 9 L--w 1 1 0 mpath-23834333461383137
26361626563396362 253 13 L--w 1 1 0 mpath-26361626563396362
26363633765623538 253 0 L--w 1 1 5 mpath-26363633765623538
23832346438613834 253 4 L--w 1 1 0 mpath-23832346438613834
26638323034336331 253 3 L--w 1 1 0 mpath-26638323034336331
23937393633303630 253 8 L--w 0 1 0 mpath-23937393633303630
23931656631326633 253 5 L--w 1 1 0 mpath-23931656631326633
26665353936313532 253 15 L--w 0 1 0 mpath-26665353936313532
26135303862613661 253 6 L--w 0 1 0 mpath-26135303862613661
26439316335316263 253 12 L--w 0 1 0 mpath-26439316335316263
26564666266393235 253 14 L--w 0 1 0 mpath-26564666266393235
26562383264626236 253 2 L--w 1 1 0 mpath-26562383264626236
26236616132323164 253 7 L--w 0 1 0 mpath-26236616132323164
26466363537333266 253 1 L--w 1 1 0 mpath-26466363537333266
# cat /proc/diskstats
8 0 sda 63494 687 2082482 45080 11530 6225 362768 690500 0 23530 734810
8 1 sda1 170 0 1360 190 0 0 0 0 0 190 190
8 2 sda2 63143 687 2079674 44620 11509 6225 362768 690490 0 23500 734330
8 16 sdb 374 6 3040 460 0 0 0 0 0 460 460
8 17 sdb1 180 0 1440 190 0 0 0 0 0 190 190
11 0 sr0 0 0 0 0 0 0 0 0 0 0 0
253 0 dm-0 283565 1406 2268002 22758620 295490 134 2560844 532030 0 671930 23290480
8 32 sdc 1489 0 11912 30130 1168 0 9320 1020 0 1640 31150
8 48 sdd 362 0 2896 1310 0 0 0 0 0 1310 1310
8 64 sde 347 0 2776 1180 0 0 0 0 0 1170 1180
8 80 sdf 346 0 2768 910 0 0 0 0 0 900 910
8 96 sdg 389 0 3112 1020 0 0 0 0 0 1020 1020
8 112 sdh 417 0 3336 1080 0 0 0 0 0 1070 1080
8 128 sdi 234 0 1872 450 1132 0 9050 1590 0 2030 2040
8 144 sdj 422 0 3400 1050 0 0 0 0 0 1040 1050
8 160 sdk 234 0 1872 520 0 0 0 0 0 520 520
8 176 sdl 421 0 3392 1030 0 0 0 0 0 1020 1030
8 192 sdm 234 0 1872 510 0 0 0 0 0 510 510
8 208 sdn 369 0 2952 880 0 0 0 0 0 880 880
8 224 sdo 234 0 1872 500 0 0 0 0 0 500 500
253 1 dm-1 127 0 1016 1070 0 0 0 0 1 278360 278820
8 240 sdp 425 0 3424 890 0 0 0 0 0 880 890
65 0 sdq 234 0 1872 500 0 0 0 0 0 500 500
65 16 sdr 424 0 3416 910 0 0 0 0 0 910 910
253 2 dm-2 112 0 896 1050 0 0 0 0 1 278240 278710
65 32 sds 233 0 1864 590 0 0 0 0 0 590 590
65 48 sdt 426 0 3432 860 0 0 0 0 0 850 860
65 64 sdu 421 0 3392 830 0 0 0 0 0 830 830
253 3 dm-3 111 0 888 1120 0 0 0 0 1 278170 278700
65 96 sdw 422 0 3400 910 0 0 0 0 0 910 910
65 80 sdv 404 0 3232 900 0 0 0 0 0 900 900
253 4 dm-4 154 0 1232 1310 0 0 0 0 1 278110 278720
65 112 sdx 425 0 3424 960 0 0 0 0 0 960 960
65 128 sdy 235 0 1880 570 0 0 0 0 0 570 570
253 5 dm-5 183 0 1464 1410 0 0 0 0 1 278050 278700
65 144 sdz 425 0 3424 820 0 0 0 0 0 820 820
65 160 sdaa 425 0 3424 830 0 0 0 0 0 830 830
253 6 dm-6 374 6 3040 1750 0 0 0 0 0 940 1750
65 176 sdab 426 0 3432 820 0 0 0 0 0 820 820
65 192 sdac 428 0 3448 830 0 0 0 0 0 830 830
65 208 sdad 425 0 3424 740 0 0 0 0 0 740 740
253 7 dm-7 374 6 3040 1620 0 0 0 0 0 880 1620
65 224 sdae 236 0 1888 500 0 0 0 0 0 500 500
65 240 sdaf 427 0 3440 640 0 0 0 0 0 640 640
66 0 sdag 427 0 3440 530 0 0 0 0 0 530 530
66 16 sdah 426 0 3432 540 0 0 0 0 0 540 540
253 8 dm-8 378 6 3072 1190 0 0 0 0 0 640 1190
253 9 dm-9 134 0 1072 520 0 0 0 0 1 277420 277640
253 10 dm-10 378 6 3072 1180 0 0 0 0 0 660 1180
253 11 dm-11 378 6 3072 1000 0 0 0 0 0 530 1000
253 12 dm-12 378 6 3072 990 0 0 0 0 0 560 990
253 13 dm-13 168 0 1344 500 0 0 0 0 1 276920 277150
253 14 dm-14 378 6 3072 910 0 0 0 0 0 480 910
253 15 dm-15 378 6 3072 950 0 0 0 0 0 500 950
# multipathd -k"show paths"
hcil dev dev_t pri dm_st chk_st dev_st next_check
15:0:0:0 sdc 8:32 1 active ready running XX........ 1/4
15:0:0:1 sdd 8:48 1 active ready running XX........ 1/4
15:0:0:2 sde 8:64 1 active ready running XX........ 1/4
15:0:0:3 sdf 8:80 1 active ready running XX........ 1/4
15:0:0:4 sdg 8:96 1 active ready running XX........ 1/4
15:0:0:5 sdh 8:112 1 active ready running XX........ 1/4
15:0:0:6 sdj 8:144 1 active ready running XX........ 1/4
16:0:0:0 sdi 8:128 1 active ready running XX........ 1/4
15:0:0:7 sdl 8:176 1 active ready running XX........ 1/4
16:0:0:1 sdk 8:160 1 active ready running XX........ 1/4
16:0:0:2 sdm 8:192 1 active ready running XX........ 1/4
16:0:0:3 sdo 8:224 1 active ready running XX........ 1/4
15:0:0:11 sdp 8:240 1 active ready running XX........ 1/4
15:0:0:10 sdn 8:208 1 active ready running XX........ 1/4
16:0:0:4 sdq 65:0 1 active ready running XX........ 1/4
15:0:0:12 sdr 65:16 1 active ready running XX........ 1/4
16:0:0:5 sds 65:32 1 active ready running XX........ 1/4
16:0:0:6 sdu 65:64 1 active ready running XX........ 1/4
15:0:0:13 sdt 65:48 1 active ready running XX........ 1/4
16:0:0:10 sdy 65:128 1 active ready running XX........ 1/4
16:0:0:7 sdw 65:96 1 active ready running XX........ 1/4
15:0:0:15 sdx 65:112 1 active ready running XX........ 1/4
15:0:0:14 sdv 65:80 1 active ready running XX........ 1/4
15:0:0:16 sdz 65:144 1 active ready running XX........ 1/4
15:0:0:17 sdab 65:176 1 active ready running XX........ 1/4
16:0:0:12 sdac 65:192 1 active ready running XX........ 1/4
16:0:0:11 sdaa 65:160 1 active ready running XX........ 1/4
16:0:0:13 sdad 65:208 1 active ready running XX........ 1/4
16:0:0:14 sdae 65:224 1 active ready running XX........ 1/4
16:0:0:15 sdaf 65:240 1 active ready running XX........ 1/4
16:0:0:17 sdah 66:16 1 active ready running XX........ 1/4
16:0:0:16 sdag 66:0 1 active ready running XX........ 1/4
# multipathd -k"show maps status"
name failback queueing paths dm-st write_prot
26363633765623538 - on 2 active rw
26466363537333266 - on 2 active rw
26562383264626236 - on 2 active rw
26638323034336331 - on 2 active rw
23832346438613834 - on 2 active rw
23931656631326633 - on 2 active rw
26135303862613661 - on 2 active rw
26236616132323164 - on 2 active rw
23937393633303630 - on 2 active rw
23834333461383137 - on 2 active rw
26133373139386639 - on 2 active rw
26230643330303865 - on 2 active rw
26439316335316263 - on 2 active rw
26361626563396362 - on 2 active rw
26564666266393235 - on 2 active rw
26665353936313532 - on 2 active rw
# multipathd -k"show maps stats"
name path_faults switch_grp map_loads total_q_time q_timeouts
26363633765623538 5 0 9 0 0
26466363537333266 0 0 2 0 0
26562383264626236 0 0 2 0 0
26638323034336331 0 0 2 0 0
23832346438613834 0 0 2 0 0
23931656631326633 0 0 2 0 0
26135303862613661 0 0 2 0 0
26236616132323164 0 0 2 0 0
23937393633303630 0 0 2 0 0
23834333461383137 0 0 2 0 0
26133373139386639 0 0 2 0 0
26230643330303865 0 0 2 0 0
26439316335316263 0 0 2 0 0
26361626563396362 0 0 2 0 0
26564666266393235 0 0 2 0 0
26665353936313532 0 0 2 0 0
SysRq : Show Blocked State
task PC stack pid father
systemd-udevd D ffff88083b6f8000 0 8283 358 0x00000000
ffff8807c69f3b30 0000000000000002 ffff88083b6f8000 ffff8807c69f3fd8
00000000000131c0 00000000000131c0 ffffffff81a154c0 ffff8807d7578698
ffff8807d75786a0 0000000000000246 ffff88083b6f8000 0000000000000000
Call Trace:
[<ffffffff814bc3a3>] schedule_preempt_disabled+0x33/0x80
[<ffffffff814bd3dc>] mutex_lock_nested+0x15c/0x3d0
[<ffffffff811d4b56>] ? __blkdev_get+0x66/0x4c0
[<ffffffff811d4b56>] __blkdev_get+0x66/0x4c0
[<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50
[<ffffffff811d5195>] blkdev_get+0x1e5/0x380
[<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50
[<ffffffff814c1091>] ? _raw_spin_unlock+0x31/0x50
[<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50
[<ffffffff811d53db>] blkdev_open+0x5b/0x80
[<ffffffff81195fc2>] do_dentry_open.isra.15+0x1e2/0x320
[<ffffffff81196210>] finish_open+0x30/0x40
[<ffffffff811a82cd>] do_last.isra.61+0xa3d/0x1150
[<ffffffff811a4725>] ? path_init+0x5d5/0x750
[<ffffffff810a6900>] ? trace_hardirqs_on_caller+0xc0/0x1c0
[<ffffffff811a8a58>] ? path_openat+0x78/0x620
[<ffffffff811a8a97>] path_openat+0xb7/0x620
[<ffffffff811a967a>] do_filp_open+0x3a/0x90
[<ffffffff814c1091>] ? _raw_spin_unlock+0x31/0x50
[<ffffffff811b8a1c>] ? __alloc_fd+0xac/0x1b0
[<ffffffff81197a5e>] do_sys_open+0x12e/0x210
[<ffffffff8101209d>] ? syscall_trace_enter+0xfd/0x200
[<ffffffff81197b5e>] SyS_open+0x1e/0x20
[<ffffffff814c1ea2>] tracesys+0xd0/0xd5
systemd-udevd D ffff88083b322230 0 8317 358 0x00000000
ffff8807cfa27b30 0000000000000002 ffff88083b322230 ffff8807cfa27fd8
00000000000131c0 00000000000131c0 ffff88083b490000 ffff88083ec1a098
ffff88083ec1a0a0 0000000000000246 ffff88083b322230 0000000000000000
Call Trace:
[<ffffffff814bc3a3>] schedule_preempt_disabled+0x33/0x80
[<ffffffff814bd3dc>] mutex_lock_nested+0x15c/0x3d0
[<ffffffff811d4b56>] ? __blkdev_get+0x66/0x4c0
[<ffffffff811d4b56>] __blkdev_get+0x66/0x4c0
[<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50
[<ffffffff811d5195>] blkdev_get+0x1e5/0x380
[<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50
[<ffffffff814c1091>] ? _raw_spin_unlock+0x31/0x50
[<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50
[<ffffffff811d53db>] blkdev_open+0x5b/0x80
[<ffffffff81195fc2>] do_dentry_open.isra.15+0x1e2/0x320
[<ffffffff81196210>] finish_open+0x30/0x40
[<ffffffff811a82cd>] do_last.isra.61+0xa3d/0x1150
[<ffffffff811a4725>] ? path_init+0x5d5/0x750
[<ffffffff810a6900>] ? trace_hardirqs_on_caller+0xc0/0x1c0
[<ffffffff811a8a58>] ? path_openat+0x78/0x620
[<ffffffff811a8a97>] path_openat+0xb7/0x620
[<ffffffff811a967a>] do_filp_open+0x3a/0x90
[<ffffffff814c1091>] ? _raw_spin_unlock+0x31/0x50
[<ffffffff811b8a1c>] ? __alloc_fd+0xac/0x1b0
[<ffffffff81197a5e>] do_sys_open+0x12e/0x210
[<ffffffff8101209d>] ? syscall_trace_enter+0xfd/0x200
[<ffffffff81197b5e>] SyS_open+0x1e/0x20
[<ffffffff814c1ea2>] tracesys+0xd0/0xd5
systemd-udevd D ffff88083b00a230 0 8329 358 0x00000000
ffff8807ee1b3b30 0000000000000002 ffff88083b00a230 ffff8807ee1b3fd8
00000000000131c0 00000000000131c0 ffff88083baea230 ffff88083ec1b418
ffff88083ec1b420 0000000000000246 ffff88083b00a230 0000000000000000
Call Trace:
[<ffffffff814bc3a3>] schedule_preempt_disabled+0x33/0x80
[<ffffffff814bd3dc>] mutex_lock_nested+0x15c/0x3d0
[<ffffffff811d4b56>] ? __blkdev_get+0x66/0x4c0
[<ffffffff811d4b56>] __blkdev_get+0x66/0x4c0
[<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50
[<ffffffff811d5195>] blkdev_get+0x1e5/0x380
[<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50
[<ffffffff814c1091>] ? _raw_spin_unlock+0x31/0x50
[<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50
[<ffffffff811d53db>] blkdev_open+0x5b/0x80
[<ffffffff81195fc2>] do_dentry_open.isra.15+0x1e2/0x320
[<ffffffff81196210>] finish_open+0x30/0x40
[<ffffffff811a82cd>] do_last.isra.61+0xa3d/0x1150
[<ffffffff811a4725>] ? path_init+0x5d5/0x750
[<ffffffff810a6900>] ? trace_hardirqs_on_caller+0xc0/0x1c0
[<ffffffff811a8a58>] ? path_openat+0x78/0x620
[<ffffffff811a8a97>] path_openat+0xb7/0x620
[<ffffffff811a967a>] do_filp_open+0x3a/0x90
[<ffffffff814c1091>] ? _raw_spin_unlock+0x31/0x50
[<ffffffff811b8a1c>] ? __alloc_fd+0xac/0x1b0
[<ffffffff81197a5e>] do_sys_open+0x12e/0x210
[<ffffffff8101209d>] ? syscall_trace_enter+0xfd/0x200
[<ffffffff81197b5e>] SyS_open+0x1e/0x20
[<ffffffff814c1ea2>] tracesys+0xd0/0xd5
systemd-udevd D ffff8807d1290000 0 8330 358 0x00000000
ffff8807cee23b30 0000000000000002 ffff8807d1290000 ffff8807cee23fd8
00000000000131c0 00000000000131c0 ffff88083b492230 ffff8807d77a6e98
ffff8807d77a6ea0 0000000000000246 ffff8807d1290000 0000000000000000
Call Trace:
[<ffffffff814bc3a3>] schedule_preempt_disabled+0x33/0x80
[<ffffffff814bd3dc>] mutex_lock_nested+0x15c/0x3d0
[<ffffffff811d4b56>] ? __blkdev_get+0x66/0x4c0
[<ffffffff811d4b56>] __blkdev_get+0x66/0x4c0
[<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50
[<ffffffff811d5195>] blkdev_get+0x1e5/0x380
[<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50
[<ffffffff814c1091>] ? _raw_spin_unlock+0x31/0x50
[<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50
[<ffffffff811d53db>] blkdev_open+0x5b/0x80
[<ffffffff81195fc2>] do_dentry_open.isra.15+0x1e2/0x320
[<ffffffff81196210>] finish_open+0x30/0x40
[<ffffffff811a82cd>] do_last.isra.61+0xa3d/0x1150
[<ffffffff811a4725>] ? path_init+0x5d5/0x750
[<ffffffff810a6900>] ? trace_hardirqs_on_caller+0xc0/0x1c0
[<ffffffff811a8a58>] ? path_openat+0x78/0x620
[<ffffffff811a8a97>] path_openat+0xb7/0x620
[<ffffffff811a967a>] do_filp_open+0x3a/0x90
[<ffffffff814c1091>] ? _raw_spin_unlock+0x31/0x50
[<ffffffff811b8a1c>] ? __alloc_fd+0xac/0x1b0
[<ffffffff81197a5e>] do_sys_open+0x12e/0x210
[<ffffffff8101209d>] ? syscall_trace_enter+0xfd/0x200
[<ffffffff81197b5e>] SyS_open+0x1e/0x20
[<ffffffff814c1ea2>] tracesys+0xd0/0xd5
systemd-udevd D ffff8807d168c460 0 8334 358 0x00000000
ffff8807d13bfb30 0000000000000002 ffff8807d168c460 ffff8807d13bffd8
00000000000131c0 00000000000131c0 ffff88083b422230 ffff8807d77a5498
ffff8807d77a54a0 0000000000000246 ffff8807d168c460 0000000000000000
Call Trace:
[<ffffffff814bc3a3>] schedule_preempt_disabled+0x33/0x80
[<ffffffff814bd3dc>] mutex_lock_nested+0x15c/0x3d0
[<ffffffff811d4b56>] ? __blkdev_get+0x66/0x4c0
[<ffffffff811d4b56>] __blkdev_get+0x66/0x4c0
[<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50
[<ffffffff811d5195>] blkdev_get+0x1e5/0x380
[<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50
[<ffffffff814c1091>] ? _raw_spin_unlock+0x31/0x50
[<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50
[<ffffffff811d53db>] blkdev_open+0x5b/0x80
[<ffffffff81195fc2>] do_dentry_open.isra.15+0x1e2/0x320
[<ffffffff81196210>] finish_open+0x30/0x40
[<ffffffff811a82cd>] do_last.isra.61+0xa3d/0x1150
[<ffffffff811a4725>] ? path_init+0x5d5/0x750
[<ffffffff810a6900>] ? trace_hardirqs_on_caller+0xc0/0x1c0
[<ffffffff811a8a58>] ? path_openat+0x78/0x620
[<ffffffff811a8a97>] path_openat+0xb7/0x620
[<ffffffff811a967a>] do_filp_open+0x3a/0x90
[<ffffffff814c1091>] ? _raw_spin_unlock+0x31/0x50
[<ffffffff811b8a1c>] ? __alloc_fd+0xac/0x1b0
[<ffffffff81197a5e>] do_sys_open+0x12e/0x210
[<ffffffff8101209d>] ? syscall_trace_enter+0xfd/0x200
[<ffffffff81197b5e>] SyS_open+0x1e/0x20
[<ffffffff814c1ea2>] tracesys+0xd0/0xd5
systemd-udevd D ffff8807d1688000 0 8335 358 0x00000000
ffff8807cf5f3b30 0000000000000002 ffff8807d1688000 ffff8807cf5f3fd8
00000000000131c0 00000000000131c0 ffff88083b494460 ffff88083ec1a718
ffff88083ec1a720 0000000000000246 ffff8807d1688000 0000000000000000
Call Trace:
[<ffffffff814bc3a3>] schedule_preempt_disabled+0x33/0x80
[<ffffffff814bd3dc>] mutex_lock_nested+0x15c/0x3d0
[<ffffffff811d4b56>] ? __blkdev_get+0x66/0x4c0
[<ffffffff811d4b56>] __blkdev_get+0x66/0x4c0
[<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50
[<ffffffff811d5195>] blkdev_get+0x1e5/0x380
[<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50
[<ffffffff814c1091>] ? _raw_spin_unlock+0x31/0x50
[<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50
[<ffffffff811d53db>] blkdev_open+0x5b/0x80
[<ffffffff81195fc2>] do_dentry_open.isra.15+0x1e2/0x320
[<ffffffff81196210>] finish_open+0x30/0x40
[<ffffffff811a82cd>] do_last.isra.61+0xa3d/0x1150
[<ffffffff811a4725>] ? path_init+0x5d5/0x750
[<ffffffff810a6900>] ? trace_hardirqs_on_caller+0xc0/0x1c0
[<ffffffff811a8a58>] ? path_openat+0x78/0x620
[<ffffffff811a8a97>] path_openat+0xb7/0x620
[<ffffffff811a967a>] do_filp_open+0x3a/0x90
[<ffffffff814c1091>] ? _raw_spin_unlock+0x31/0x50
[<ffffffff811b8a1c>] ? __alloc_fd+0xac/0x1b0
[<ffffffff81197a5e>] do_sys_open+0x12e/0x210
[<ffffffff8101209d>] ? syscall_trace_enter+0xfd/0x200
[<ffffffff81197b5e>] SyS_open+0x1e/0x20
[<ffffffff814c1ea2>] tracesys+0xd0/0xd5
systemd-udevd D ffff88082df38000 0 8341 358 0x00000000
ffff8807d1763b30 0000000000000002 ffff88082df38000 ffff8807d1763fd8
00000000000131c0 00000000000131c0 ffff88083b422230 ffff8807d757e818
ffff8807d757e820 0000000000000246 ffff88082df38000 0000000000000000
Call Trace:
[<ffffffff814bc3a3>] schedule_preempt_disabled+0x33/0x80
[<ffffffff814bd3dc>] mutex_lock_nested+0x15c/0x3d0
[<ffffffff811d4b56>] ? __blkdev_get+0x66/0x4c0
[<ffffffff811d4b56>] __blkdev_get+0x66/0x4c0
[<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50
[<ffffffff811d5195>] blkdev_get+0x1e5/0x380
[<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50
[<ffffffff814c1091>] ? _raw_spin_unlock+0x31/0x50
[<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50
[<ffffffff811d53db>] blkdev_open+0x5b/0x80
[<ffffffff81195fc2>] do_dentry_open.isra.15+0x1e2/0x320
[<ffffffff81196210>] finish_open+0x30/0x40
[<ffffffff811a82cd>] do_last.isra.61+0xa3d/0x1150
[<ffffffff811a4725>] ? path_init+0x5d5/0x750
[<ffffffff810a6900>] ? trace_hardirqs_on_caller+0xc0/0x1c0
[<ffffffff811a8a58>] ? path_openat+0x78/0x620
[<ffffffff811a8a97>] path_openat+0xb7/0x620
[<ffffffff811a967a>] do_filp_open+0x3a/0x90
[<ffffffff814c1091>] ? _raw_spin_unlock+0x31/0x50
[<ffffffff811b8a1c>] ? __alloc_fd+0xac/0x1b0
[<ffffffff81197a5e>] do_sys_open+0x12e/0x210
[<ffffffff8101209d>] ? syscall_trace_enter+0xfd/0x200
[<ffffffff81197b5e>] SyS_open+0x1e/0x20
[<ffffffff814c1ea2>] tracesys+0xd0/0xd5
blkid D ffff8807d3110000 0 8422 1 0x00000006
ffff8807d59879b8 0000000000000002 ffff8807d3110000 ffff8807d5987fd8
00000000000131c0 00000000000131c0 ffff88083b4c4460 ffff88085fd33ad0
ffff88085ff83038 ffff8807d5987a40 0000000000000002 ffffffff81135db0
Call Trace:
[<ffffffff81135db0>] ? wait_on_page_read+0x60/0x60
[<ffffffff814bc0ed>] io_schedule+0x9d/0x130
[<ffffffff81135dbe>] sleep_on_page+0xe/0x20
[<ffffffff814bc738>] __wait_on_bit_lock+0x48/0xb0
[<ffffffff81135eca>] __lock_page+0x6a/0x70
[<ffffffff8109c8e0>] ? autoremove_wake_function+0x40/0x40
[<ffffffff811474cf>] truncate_inode_pages_range+0x3ff/0x690
[<ffffffff81147775>] truncate_inode_pages+0x15/0x20
[<ffffffff811d3425>] kill_bdev+0x35/0x40
[<ffffffff811d49a9>] __blkdev_put+0x69/0x1b0
[<ffffffff811d5450>] blkdev_put+0x50/0x160
[<ffffffff811d5615>] blkdev_close+0x25/0x30
[<ffffffff8119a37a>] __fput+0xea/0x1f0
[<ffffffff8119a4ce>] ____fput+0xe/0x10
[<ffffffff81074ddc>] task_work_run+0xac/0xe0
[<ffffffff8104ff67>] do_exit+0x2c7/0xc60
[<ffffffff81064091>] ? get_signal_to_deliver+0xd1/0x940
[<ffffffff814c118c>] ? _raw_spin_unlock_irq+0x2c/0x60
[<ffffffff81051cac>] do_group_exit+0x4c/0xc0
[<ffffffff810642a1>] get_signal_to_deliver+0x2e1/0x940
[<ffffffff81002528>] do_signal+0x48/0x630
[<ffffffff811d3f77>] ? blkdev_read_iter+0x37/0x40
[<ffffffff8119809e>] ? new_sync_read+0x7e/0xb0
[<ffffffff8127f0d7>] ? debug_smp_processor_id+0x17/0x20
[<ffffffff810ba5d6>] ? rcu_eqs_exit_common.isra.43+0x26/0xe0
[<ffffffff8113539a>] ? context_tracking_user_exit+0x5a/0x1a0
[<ffffffff810a693d>] ? trace_hardirqs_on_caller+0xfd/0x1c0
[<ffffffff81002b81>] do_notify_resume+0x71/0xc0
[<ffffffff814c1f98>] int_signal+0x12/0x17
blkid D ffff8807d3632230 0 8458 1 0x00000006
ffff8807d360f9b8 0000000000000002 ffff8807d3632230 ffff8807d360ffd8
00000000000131c0 00000000000131c0 ffff88083b492230 ffff88085fc93ad0
ffff88085ff7fa38 ffff8807d360fa40 0000000000000002 ffffffff81135db0
Call Trace:
[<ffffffff81135db0>] ? wait_on_page_read+0x60/0x60
[<ffffffff814bc0ed>] io_schedule+0x9d/0x130
[<ffffffff81135dbe>] sleep_on_page+0xe/0x20
[<ffffffff814bc738>] __wait_on_bit_lock+0x48/0xb0
[<ffffffff81135eca>] __lock_page+0x6a/0x70
[<ffffffff8109c8e0>] ? autoremove_wake_function+0x40/0x40
[<ffffffff811474cf>] truncate_inode_pages_range+0x3ff/0x690
[<ffffffff81147775>] truncate_inode_pages+0x15/0x20
[<ffffffff811d3425>] kill_bdev+0x35/0x40
[<ffffffff811d49a9>] __blkdev_put+0x69/0x1b0
[<ffffffff811d5450>] blkdev_put+0x50/0x160
[<ffffffff811d5615>] blkdev_close+0x25/0x30
[<ffffffff8119a37a>] __fput+0xea/0x1f0
[<ffffffff8119a4ce>] ____fput+0xe/0x10
[<ffffffff81074ddc>] task_work_run+0xac/0xe0
[<ffffffff8104ff67>] do_exit+0x2c7/0xc60
[<ffffffff81064091>] ? get_signal_to_deliver+0xd1/0x940
[<ffffffff814c118c>] ? _raw_spin_unlock_irq+0x2c/0x60
[<ffffffff81051cac>] do_group_exit+0x4c/0xc0
[<ffffffff810642a1>] get_signal_to_deliver+0x2e1/0x940
[<ffffffff81002528>] do_signal+0x48/0x630
[<ffffffff811d3f77>] ? blkdev_read_iter+0x37/0x40
[<ffffffff8119809e>] ? new_sync_read+0x7e/0xb0
[<ffffffff8127f0d7>] ? debug_smp_processor_id+0x17/0x20
[<ffffffff810ba5d6>] ? rcu_eqs_exit_common.isra.43+0x26/0xe0
[<ffffffff8113539a>] ? context_tracking_user_exit+0x5a/0x1a0
[<ffffffff810a693d>] ? trace_hardirqs_on_caller+0xfd/0x1c0
[<ffffffff81002b81>] do_notify_resume+0x71/0xc0
[<ffffffff814c1f98>] int_signal+0x12/0x17
blkid D ffff8807d4432230 0 8467 1 0x00000006
ffff8807d4daf9b8 0000000000000002 ffff8807d4432230 ffff8807d4daffd8
00000000000131c0 00000000000131c0 ffff88083b4c2230 ffff88085fd53ad0
ffff88085ffa4338 ffff8807d4dafa40 0000000000000002 ffffffff81135db0
Call Trace:
[<ffffffff81135db0>] ? wait_on_page_read+0x60/0x60
[<ffffffff814bc0ed>] io_schedule+0x9d/0x130
[<ffffffff81135dbe>] sleep_on_page+0xe/0x20
[<ffffffff814bc738>] __wait_on_bit_lock+0x48/0xb0
[<ffffffff81135eca>] __lock_page+0x6a/0x70
[<ffffffff8109c8e0>] ? autoremove_wake_function+0x40/0x40
[<ffffffff811474cf>] truncate_inode_pages_range+0x3ff/0x690
[<ffffffff81147775>] truncate_inode_pages+0x15/0x20
[<ffffffff811d3425>] kill_bdev+0x35/0x40
[<ffffffff811d49a9>] __blkdev_put+0x69/0x1b0
[<ffffffff811d5450>] blkdev_put+0x50/0x160
[<ffffffff811d5615>] blkdev_close+0x25/0x30
[<ffffffff8119a37a>] __fput+0xea/0x1f0
[<ffffffff8119a4ce>] ____fput+0xe/0x10
[<ffffffff81074ddc>] task_work_run+0xac/0xe0
[<ffffffff8104ff67>] do_exit+0x2c7/0xc60
[<ffffffff81064091>] ? get_signal_to_deliver+0xd1/0x940
[<ffffffff814c118c>] ? _raw_spin_unlock_irq+0x2c/0x60
[<ffffffff81051cac>] do_group_exit+0x4c/0xc0
[<ffffffff810642a1>] get_signal_to_deliver+0x2e1/0x940
[<ffffffff81002528>] do_signal+0x48/0x630
[<ffffffff811d3f77>] ? blkdev_read_iter+0x37/0x40
[<ffffffff8119809e>] ? new_sync_read+0x7e/0xb0
[<ffffffff8127f0d7>] ? debug_smp_processor_id+0x17/0x20
[<ffffffff810ba5d6>] ? rcu_eqs_exit_common.isra.43+0x26/0xe0
[<ffffffff8113539a>] ? context_tracking_user_exit+0x5a/0x1a0
[<ffffffff810a693d>] ? trace_hardirqs_on_caller+0xfd/0x1c0
[<ffffffff81002b81>] do_notify_resume+0x71/0xc0
[<ffffffff814c1f98>] int_signal+0x12/0x17
blkid D ffff8807d275a230 0 8472 1 0x00000006
ffff8807d27bb9b8 0000000000000002 ffff8807d275a230 ffff8807d27bbfd8
00000000000131c0 00000000000131c0 ffff8807d1290000 ffff88085fc93ad0
ffff88085ff87838 ffff8807d27bba40 0000000000000002 ffffffff81135db0
Call Trace:
[<ffffffff81135db0>] ? wait_on_page_read+0x60/0x60
[<ffffffff814bc0ed>] io_schedule+0x9d/0x130
[<ffffffff81135dbe>] sleep_on_page+0xe/0x20
[<ffffffff814bc738>] __wait_on_bit_lock+0x48/0xb0
[<ffffffff81135eca>] __lock_page+0x6a/0x70
[<ffffffff8109c8e0>] ? autoremove_wake_function+0x40/0x40
[<ffffffff811474cf>] truncate_inode_pages_range+0x3ff/0x690
[<ffffffff81147775>] truncate_inode_pages+0x15/0x20
[<ffffffff811d3425>] kill_bdev+0x35/0x40
[<ffffffff811d49a9>] __blkdev_put+0x69/0x1b0
[<ffffffff811d5450>] blkdev_put+0x50/0x160
[<ffffffff811d5615>] blkdev_close+0x25/0x30
[<ffffffff8119a37a>] __fput+0xea/0x1f0
[<ffffffff8119a4ce>] ____fput+0xe/0x10
[<ffffffff81074ddc>] task_work_run+0xac/0xe0
[<ffffffff8104ff67>] do_exit+0x2c7/0xc60
[<ffffffff81064091>] ? get_signal_to_deliver+0xd1/0x940
[<ffffffff814c118c>] ? _raw_spin_unlock_irq+0x2c/0x60
[<ffffffff81051cac>] do_group_exit+0x4c/0xc0
[<ffffffff810642a1>] get_signal_to_deliver+0x2e1/0x940
[<ffffffff81002528>] do_signal+0x48/0x630
[<ffffffff811d3f77>] ? blkdev_read_iter+0x37/0x40
[<ffffffff8119809e>] ? new_sync_read+0x7e/0xb0
[<ffffffff8127f0d7>] ? debug_smp_processor_id+0x17/0x20
[<ffffffff810ba5d6>] ? rcu_eqs_exit_common.isra.43+0x26/0xe0
[<ffffffff8113539a>] ? context_tracking_user_exit+0x5a/0x1a0
[<ffffffff810a693d>] ? trace_hardirqs_on_caller+0xfd/0x1c0
[<ffffffff81002b81>] do_notify_resume+0x71/0xc0
[<ffffffff814c1f98>] int_signal+0x12/0x17
blkid D ffff8807c6b10000 0 8482 1 0x00000006
ffff8807d4dbf9b8 0000000000000002 ffff8807c6b10000 ffff8807d4dbffd8
00000000000131c0 00000000000131c0 ffff88082df38000 ffff88085fd73ad0
ffff88085ff74638 ffff8807d4dbfa40 0000000000000002 ffffffff81135db0
Call Trace:
[<ffffffff81135db0>] ? wait_on_page_read+0x60/0x60
[<ffffffff814bc0ed>] io_schedule+0x9d/0x130
[<ffffffff81135dbe>] sleep_on_page+0xe/0x20
[<ffffffff814bc738>] __wait_on_bit_lock+0x48/0xb0
[<ffffffff81135eca>] __lock_page+0x6a/0x70
[<ffffffff8109c8e0>] ? autoremove_wake_function+0x40/0x40
[<ffffffff811474cf>] truncate_inode_pages_range+0x3ff/0x690
[<ffffffff81147775>] truncate_inode_pages+0x15/0x20
[<ffffffff811d3425>] kill_bdev+0x35/0x40
[<ffffffff811d49a9>] __blkdev_put+0x69/0x1b0
[<ffffffff811d5450>] blkdev_put+0x50/0x160
[<ffffffff811d5615>] blkdev_close+0x25/0x30
[<ffffffff8119a37a>] __fput+0xea/0x1f0
[<ffffffff8119a4ce>] ____fput+0xe/0x10
[<ffffffff81074ddc>] task_work_run+0xac/0xe0
[<ffffffff8104ff67>] do_exit+0x2c7/0xc60
[<ffffffff81064091>] ? get_signal_to_deliver+0xd1/0x940
[<ffffffff814c118c>] ? _raw_spin_unlock_irq+0x2c/0x60
[<ffffffff81051cac>] do_group_exit+0x4c/0xc0
[<ffffffff810642a1>] get_signal_to_deliver+0x2e1/0x940
[<ffffffff81002528>] do_signal+0x48/0x630
[<ffffffff811d3f77>] ? blkdev_read_iter+0x37/0x40
[<ffffffff8119809e>] ? new_sync_read+0x7e/0xb0
[<ffffffff8127f0d7>] ? debug_smp_processor_id+0x17/0x20
[<ffffffff810ba5d6>] ? rcu_eqs_exit_common.isra.43+0x26/0xe0
[<ffffffff8113539a>] ? context_tracking_user_exit+0x5a/0x1a0
[<ffffffff810a693d>] ? trace_hardirqs_on_caller+0xfd/0x1c0
[<ffffffff81002b81>] do_notify_resume+0x71/0xc0
[<ffffffff814c1f98>] int_signal+0x12/0x17
blkid D ffff8807c6b88000 0 8549 1 0x00000006
ffff8807c6bb39b8 0000000000000002 ffff8807c6b88000 ffff8807c6bb3fd8
00000000000131c0 00000000000131c0 ffff8807d168c460 ffff88085fc33ad0
ffff88085ff88a38 ffff8807c6bb3a40 0000000000000002 ffffffff81135db0
Call Trace:
[<ffffffff81135db0>] ? wait_on_page_read+0x60/0x60
[<ffffffff814bc0ed>] io_schedule+0x9d/0x130
[<ffffffff81135dbe>] sleep_on_page+0xe/0x20
[<ffffffff814bc738>] __wait_on_bit_lock+0x48/0xb0
[<ffffffff81135eca>] __lock_page+0x6a/0x70
[<ffffffff8109c8e0>] ? autoremove_wake_function+0x40/0x40
[<ffffffff811474cf>] truncate_inode_pages_range+0x3ff/0x690
[<ffffffff81147775>] truncate_inode_pages+0x15/0x20
[<ffffffff811d3425>] kill_bdev+0x35/0x40
[<ffffffff811d49a9>] __blkdev_put+0x69/0x1b0
[<ffffffff811d5450>] blkdev_put+0x50/0x160
[<ffffffff811d5615>] blkdev_close+0x25/0x30
[<ffffffff8119a37a>] __fput+0xea/0x1f0
[<ffffffff8119a4ce>] ____fput+0xe/0x10
[<ffffffff81074ddc>] task_work_run+0xac/0xe0
[<ffffffff8104ff67>] do_exit+0x2c7/0xc60
[<ffffffff81064091>] ? get_signal_to_deliver+0xd1/0x940
[<ffffffff814c118c>] ? _raw_spin_unlock_irq+0x2c/0x60
[<ffffffff81051cac>] do_group_exit+0x4c/0xc0
[<ffffffff810642a1>] get_signal_to_deliver+0x2e1/0x940
[<ffffffff81002528>] do_signal+0x48/0x630
[<ffffffff811d3f77>] ? blkdev_read_iter+0x37/0x40
[<ffffffff8119809e>] ? new_sync_read+0x7e/0xb0
[<ffffffff8127f0d7>] ? debug_smp_processor_id+0x17/0x20
[<ffffffff810ba5d6>] ? rcu_eqs_exit_common.isra.43+0x26/0xe0
[<ffffffff8113539a>] ? context_tracking_user_exit+0x5a/0x1a0
[<ffffffff810a693d>] ? trace_hardirqs_on_caller+0xfd/0x1c0
[<ffffffff81002b81>] do_notify_resume+0x71/0xc0
[<ffffffff814c1f98>] int_signal+0x12/0x17
blkid D ffff8807c7268000 0 8615 1 0x00000006
ffff8807d43b39b8 0000000000000002 ffff8807c7268000 ffff8807d43b3fd8
00000000000131c0 00000000000131c0 ffff88083b4a8000 ffff88085fcb3ad0
ffff88085ffb0038 ffff8807d43b3a40 0000000000000002 ffffffff81135db0
Call Trace:
[<ffffffff81135db0>] ? wait_on_page_read+0x60/0x60
[<ffffffff814bc0ed>] io_schedule+0x9d/0x130
[<ffffffff81135dbe>] sleep_on_page+0xe/0x20
[<ffffffff814bc738>] __wait_on_bit_lock+0x48/0xb0
[<ffffffff81135eca>] __lock_page+0x6a/0x70
[<ffffffff8109c8e0>] ? autoremove_wake_function+0x40/0x40
[<ffffffff811474cf>] truncate_inode_pages_range+0x3ff/0x690
[<ffffffff81147775>] truncate_inode_pages+0x15/0x20
[<ffffffff811d3425>] kill_bdev+0x35/0x40
[<ffffffff811d49a9>] __blkdev_put+0x69/0x1b0
[<ffffffff811d5450>] blkdev_put+0x50/0x160
[<ffffffff811d5615>] blkdev_close+0x25/0x30
[<ffffffff8119a37a>] __fput+0xea/0x1f0
[<ffffffff8119a4ce>] ____fput+0xe/0x10
[<ffffffff81074ddc>] task_work_run+0xac/0xe0
[<ffffffff8104ff67>] do_exit+0x2c7/0xc60
[<ffffffff81064091>] ? get_signal_to_deliver+0xd1/0x940
[<ffffffff814c118c>] ? _raw_spin_unlock_irq+0x2c/0x60
[<ffffffff81051cac>] do_group_exit+0x4c/0xc0
[<ffffffff810642a1>] get_signal_to_deliver+0x2e1/0x940
[<ffffffff81002528>] do_signal+0x48/0x630
[<ffffffff811d3f77>] ? blkdev_read_iter+0x37/0x40
[<ffffffff8119809e>] ? new_sync_read+0x7e/0xb0
[<ffffffff8127f0d7>] ? debug_smp_processor_id+0x17/0x20
[<ffffffff810ba5d6>] ? rcu_eqs_exit_common.isra.43+0x26/0xe0
[<ffffffff8113539a>] ? context_tracking_user_exit+0x5a/0x1a0
[<ffffffff810a693d>] ? trace_hardirqs_on_caller+0xfd/0x1c0
[<ffffffff81002b81>] do_notify_resume+0x71/0xc0
[<ffffffff814c1f98>] int_signal+0x12/0x17
[-- Attachment #3: Type: text/plain, Size: 0 bytes --]
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: v3.15 dm-mpath regression: cable pull test causes I/O hang
2014-07-07 13:40 ` Bart Van Assche
@ 2014-07-08 0:55 ` Junichi Nomura
2014-07-08 9:43 ` Bart Van Assche
2014-07-08 16:33 ` Mike Snitzer
0 siblings, 2 replies; 19+ messages in thread
From: Junichi Nomura @ 2014-07-08 0:55 UTC (permalink / raw)
To: Bart Van Assche, device-mapper development; +Cc: Mike Snitzer
On 07/07/14 22:40, Bart Van Assche wrote:
> Thanks for looking into this issue. I have attached the requested
> information to this e-mail for a test run with kernel 3.16-rc4 at the
> initiator side.
Thank you, Bart. The information is helpful.
From "dmsetup table" output, hardware handler is not used
in your setup. So pg_init is not involved.
> # cat /proc/diskstats
..
> 253 1 dm-1 127 0 1016 1070 0 0 0 0 1 278360 278820
In-flight IO remains.
> # dmsetup info -c
> Name Maj Min Stat Open Targ Event UUID
...
> 26466363537333266 253 1 L--w 1 1 0 mpath-26466363537333266
Typical reason of remainig IO is device staying as suspended.
But the device state is ok here.
> # dmsetup status
...
> 26466363537333266: 0 256000 multipath 2 1 0 0 1 1 E 0 2 2 8:48 A 0 0 1 8:160 A 0 0 1
Single path group with both paths being active on dm-1.
But the path group is not active.
I suspect what's happening here is nobody clears m->queue_io:
multipath_busy() returns busy when queue_io=1 while clearing queue_io
needs __choose_pgpath(), which won't be called if multipath_busy() is true.
I think if you run 'sg_inq /dev/dm-1' for example in this case,
the device will start working again.
Since ioctl is not affected by multipath_busy(), somehow the problem
was hidden in many cases by udev activities, for example.
Attached patch should fix the problem.
Could you give it a try?
-
Jun'ichi Nomura, NEC Corporation
pg_ready() checks the current state of the multipath and may return
false even if a new IO is needed to change the state.
OTOH, if multipath_busy() returns busy, a new IO will not be sent
to multipath target and the state change won't happen. That results
in lock up.
The intent of multipath_busy() is to avoid unnecessary cycles of
dequeue + request_fn + requeue if it is known that multipath device
will requeue.
Such situation would be:
- path group is being activated
- there is no path and the multipath is setup to requeue if no path
This patch should fix the problem introduced as a part of this commit:
commit e809917735ebf1b9a56c24e877ce0d320baee2ec
dm mpath: push back requests instead of queueing
diff --git a/drivers/md/dm-mpath.c b/drivers/md/dm-mpath.c
index ebfa411..d58343e 100644
--- a/drivers/md/dm-mpath.c
+++ b/drivers/md/dm-mpath.c
@@ -1620,8 +1620,9 @@ static int multipath_busy(struct dm_target *ti)
spin_lock_irqsave(&m->lock, flags);
- /* pg_init in progress, requeue until done */
- if (!pg_ready(m)) {
+ /* pg_init in progress or no paths available */
+ if (m->pg_init_in_progress ||
+ (!m->nr_valid_paths && m->queue_if_no_path)) {
busy = 1;
goto out;
}
^ permalink raw reply related [flat|nested] 19+ messages in thread
* Re: v3.15 dm-mpath regression: cable pull test causes I/O hang
2014-07-08 0:55 ` Junichi Nomura
@ 2014-07-08 9:43 ` Bart Van Assche
2014-07-08 16:33 ` Mike Snitzer
1 sibling, 0 replies; 19+ messages in thread
From: Bart Van Assche @ 2014-07-08 9:43 UTC (permalink / raw)
To: Junichi Nomura; +Cc: device-mapper development, Mike Snitzer
On 07/08/14 02:55, Junichi Nomura wrote:
> pg_ready() checks the current state of the multipath and may return
> false even if a new IO is needed to change the state.
>
> OTOH, if multipath_busy() returns busy, a new IO will not be sent
> to multipath target and the state change won't happen. That results
> in lock up.
>
> The intent of multipath_busy() is to avoid unnecessary cycles of
> dequeue + request_fn + requeue if it is known that multipath device
> will requeue.
>
> Such situation would be:
> - path group is being activated
> - there is no path and the multipath is setup to requeue if no path
>
> This patch should fix the problem introduced as a part of this commit:
> commit e809917735ebf1b9a56c24e877ce0d320baee2ec
> dm mpath: push back requests instead of queueing
>
> diff --git a/drivers/md/dm-mpath.c b/drivers/md/dm-mpath.c
> index ebfa411..d58343e 100644
> --- a/drivers/md/dm-mpath.c
> +++ b/drivers/md/dm-mpath.c
> @@ -1620,8 +1620,9 @@ static int multipath_busy(struct dm_target *ti)
>
> spin_lock_irqsave(&m->lock, flags);
>
> - /* pg_init in progress, requeue until done */
> - if (!pg_ready(m)) {
> + /* pg_init in progress or no paths available */
> + if (m->pg_init_in_progress ||
> + (!m->nr_valid_paths && m->queue_if_no_path)) {
> busy = 1;
> goto out;
> }
>
This patch seems to fix the issue reported at the start of this thread -
with this patch applied my test passes.
Thanks !
Bart.
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: v3.15 dm-mpath regression: cable pull test causes I/O hang
2014-07-08 0:55 ` Junichi Nomura
2014-07-08 9:43 ` Bart Van Assche
@ 2014-07-08 16:33 ` Mike Snitzer
2014-07-08 23:24 ` Junichi Nomura
1 sibling, 1 reply; 19+ messages in thread
From: Mike Snitzer @ 2014-07-08 16:33 UTC (permalink / raw)
To: Junichi Nomura; +Cc: device-mapper development, Bart Van Assche
On Mon, Jul 07 2014 at 8:55pm -0400,
Junichi Nomura <j-nomura@ce.jp.nec.com> wrote:
> On 07/07/14 22:40, Bart Van Assche wrote:
> > Thanks for looking into this issue. I have attached the requested
> > information to this e-mail for a test run with kernel 3.16-rc4 at the
> > initiator side.
>
> Thank you, Bart. The information is helpful.
>
> >From "dmsetup table" output, hardware handler is not used
> in your setup. So pg_init is not involved.
>
> > # cat /proc/diskstats
> ..
> > 253 1 dm-1 127 0 1016 1070 0 0 0 0 1 278360 278820
>
> In-flight IO remains.
>
> > # dmsetup info -c
> > Name Maj Min Stat Open Targ Event UUID
> ...
> > 26466363537333266 253 1 L--w 1 1 0 mpath-26466363537333266
>
> Typical reason of remainig IO is device staying as suspended.
> But the device state is ok here.
>
> > # dmsetup status
> ...
> > 26466363537333266: 0 256000 multipath 2 1 0 0 1 1 E 0 2 2 8:48 A 0 0 1 8:160 A 0 0 1
>
> Single path group with both paths being active on dm-1.
> But the path group is not active.
>
> I suspect what's happening here is nobody clears m->queue_io:
> multipath_busy() returns busy when queue_io=1 while clearing queue_io
> needs __choose_pgpath(), which won't be called if multipath_busy() is true.
>
> I think if you run 'sg_inq /dev/dm-1' for example in this case,
> the device will start working again.
> Since ioctl is not affected by multipath_busy(), somehow the problem
> was hidden in many cases by udev activities, for example.
>
> Attached patch should fix the problem.
> Could you give it a try?
>
> -
> Jun'ichi Nomura, NEC Corporation
>
>
> pg_ready() checks the current state of the multipath and may return
> false even if a new IO is needed to change the state.
>
> OTOH, if multipath_busy() returns busy, a new IO will not be sent
> to multipath target and the state change won't happen. That results
> in lock up.
>
> The intent of multipath_busy() is to avoid unnecessary cycles of
> dequeue + request_fn + requeue if it is known that multipath device
> will requeue.
>
> Such situation would be:
> - path group is being activated
> - there is no path and the multipath is setup to requeue if no path
>
> This patch should fix the problem introduced as a part of this commit:
> commit e809917735ebf1b9a56c24e877ce0d320baee2ec
> dm mpath: push back requests instead of queueing
Thanks Jun'ichi! I knew multipath_busy() wasn't quite right ;)
I've merged your fix with a revised header, if you see anything wrong
with the header please feel free to let me know! See:
https://git.kernel.org/cgit/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=for-next&id=75c76c45b76e53b7c2f025d30e7e308bfe331004
This will likely go to Linus for 3.16-rc5 by this Friday.
BTW, I also staged a related cleanup for 3.17, see:
https://git.kernel.org/cgit/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=for-next&id=2122e57c913cd842e5061137a7093bd06e728677
Thanks again,
Mike
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: v3.15 dm-mpath regression: cable pull test causes I/O hang
2014-07-08 16:33 ` Mike Snitzer
@ 2014-07-08 23:24 ` Junichi Nomura
0 siblings, 0 replies; 19+ messages in thread
From: Junichi Nomura @ 2014-07-08 23:24 UTC (permalink / raw)
To: device-mapper development, Mike Snitzer; +Cc: Bart Van Assche
On 07/09/14 01:33, Mike Snitzer wrote:
> I've merged your fix with a revised header, if you see anything wrong
> with the header please feel free to let me know! See:
> https://git.kernel.org/cgit/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=for-next&id=75c76c45b76e53b7c2f025d30e7e308bfe331004
>
> This will likely go to Linus for 3.16-rc5 by this Friday.
Thank you, Mike. I have no problem with the header.
> BTW, I also staged a related cleanup for 3.17, see:
> https://git.kernel.org/cgit/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=for-next&id=2122e57c913cd842e5061137a7093bd06e728677
Yeah, I'm ok with that.
--
Jun'ichi Nomura, NEC Corporation
^ permalink raw reply [flat|nested] 19+ messages in thread
end of thread, other threads:[~2014-07-08 23:24 UTC | newest]
Thread overview: 19+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2014-06-27 13:02 v3.15 dm-mpath regression: cable pull test causes I/O hang Bart Van Assche
2014-06-27 13:33 ` Mike Snitzer
2014-06-27 14:18 ` Bart Van Assche
2014-07-02 22:02 ` Mike Snitzer
2014-07-03 5:43 ` Hannes Reinecke
2014-07-03 13:56 ` Bart Van Assche
2014-07-03 13:58 ` Hannes Reinecke
2014-07-03 14:05 ` Mike Snitzer
2014-07-03 14:15 ` Hannes Reinecke
2014-07-03 14:18 ` Mike Snitzer
2014-07-03 14:34 ` Bart Van Assche
2014-07-03 15:00 ` Mike Snitzer
2014-07-07 13:28 ` Bart Van Assche
2014-07-04 3:10 ` Junichi Nomura
2014-07-07 13:40 ` Bart Van Assche
2014-07-08 0:55 ` Junichi Nomura
2014-07-08 9:43 ` Bart Van Assche
2014-07-08 16:33 ` Mike Snitzer
2014-07-08 23:24 ` Junichi Nomura
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.