xfstests generic/347 seems unhappy on todays block for-next tree

xfstests generic/347 seems unhappy on todays block for-next tree

[Christoph Hellwig]

This test is described as:

# Test very basic thin device usage, exhaustion, and growth


And seems to hang like this:

generic/347 74s ... [  701.658856] run fstests generic/347 at 2018-12-10 21:05:23
[  701.845672] XFS (nvme0n1): Mounting V5 Filesystem
[  702.048278] XFS (nvme0n1): Ending clean mount
[  702.184992] XFS (nvme1n1): Unmounting Filesystem
[  702.453064] device-mapper: thin: Data device (dm-1) discard unsupported: Disabling discard passdown.
[  847.533607] INFO: task dmsetup:28965 blocked for more than 120 seconds.
[  847.536715]       Not tainted 4.20.0-rc6+ #4
[  847.538857] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  847.542613] dmsetup         D    0 28965  28695 0x00000000
[  847.545405] Call Trace:
[  847.548500]  ? __schedule+0x2e6/0xb00
[  847.550331]  schedule+0x27/0x70
[  847.551977]  io_schedule+0x11/0x40
[  847.553991]  dm_wait_for_completion+0xbe/0xd0
[  847.556151]  ? wait_woken+0x80/0x80
[  847.558165]  __dm_suspend+0x90/0x1e0
[  847.559888]  dm_internal_suspend_noflush+0x90/0xe0
[  847.562380]  pool_presuspend+0x41/0x60
[  847.564626]  suspend_targets+0x45/0xa0
[  847.566783]  __dm_suspend+0xe0/0x1e0
[  847.568454]  ? table_clear+0xb0/0xb0
[  847.570289]  dm_suspend+0xd3/0x110
[  847.572218]  ? up_read+0x17/0x90
[  847.573960]  dev_suspend+0x148/0x250
[  847.575671]  ctl_ioctl+0x1a7/0x3a0
[  847.577413]  ? _raw_spin_lock_irqsave_nested+0x10/0x40
[  847.579895]  dm_ctl_ioctl+0x5/0x10
[  847.581580]  do_vfs_ioctl+0xa0/0x6a0
[  847.583299]  ksys_ioctl+0x5b/0x90
[  847.584739]  __x64_sys_ioctl+0x11/0x20
[  847.586701]  do_syscall_64+0x4b/0x180
[  847.588679]  entry_SYSCALL_64_after_hwframe+0x49/0xbe
[  847.590450] RIP: 0033:0x7f3b400a2f07
[  847.591709] Code: Bad RIP value.
[  847.592688] RSP: 002b:00007ffc29d632f8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[  847.595091] RAX: ffffffffffffffda RBX: 000055fbf64e0b00 RCX: 00007f3b400a2f07
[  847.597344] RDX: 000055fbf64e0b00 RSI: 00000000c138fd06 RDI: 0000000000000003
[  847.599027] RBP: 000000000000000c R08: 00007f3b405bc648 R09: 00007ffc29d63160
[  847.600660] R10: 00007f3b405bbb53 R11: 0000000000000246 R12: 000055fbf64e0b30
[  847.602441] R13: 00007f3b405bbb53 R14: 000055fbf64e0940 R15: 0000000000000000
[  847.604404] 
[  847.604404] Showing all locks held in the system:
[  847.605655] 1 lock held by khungtaskd/620:
[  847.606542]  #0: 00000000253d4c96 (rcu_read_lock){....}, at: debug_show_all_locks+0x15/0x175
[  847.608204] 1 lock held by in:imklog/3712:
[  847.608979]  #0: 00000000af869ccd (&f->f_pos_lock){+.+.}, at: __fdget_pos+0x3f/0x50
[  847.610629] 2 locks held by dmsetup/28965:
[  847.611410]  #0: 00000000e234e2ee (&md->suspend_lock/1){+.+.}, at: dm_suspend+0x22/0x110
[  847.613066]  #1: 00000000756de3cd (&md->suspend_lock#2){+.+.}, at: dm_internal_suspend_noflush+0xc/0xe0
[  847.614989] 
[  847.615323] =============================================
[  847.615323] 

Re: xfstests generic/347 seems unhappy on todays block for-next tree

[Jens Axboe]

On 12/10/18 2:37 PM, Christoph Hellwig wrote:
> This test is described as:
> 
> # Test very basic thin device usage, exhaustion, and growth

Does that tree have:

commit c616cbee97aed4bc6178f148a7240206dcdb85a6
Author: Jens Axboe <axboe@kernel.dk>
Date:   Thu Dec 6 22:17:44 2018 -0700

    blk-mq: punt failed direct issue to dispatch list

?

-- 
Jens Axboe

Re: xfstests generic/347 seems unhappy on todays block for-next tree

[Christoph Hellwig]

On Mon, Dec 10, 2018 at 02:39:39PM -0700, Jens Axboe wrote:
> On 12/10/18 2:37 PM, Christoph Hellwig wrote:
> > This test is described as:
> > 
> > # Test very basic thin device usage, exhaustion, and growth
> 
> Does that tree have:
> 
> commit c616cbee97aed4bc6178f148a7240206dcdb85a6
> Author: Jens Axboe <axboe@kernel.dk>
> Date:   Thu Dec 6 22:17:44 2018 -0700
> 
>     blk-mq: punt failed direct issue to dispatch list
> 
> ?

yes.

The latest commit is 6f75723190d88e1319bea623bfe0292bf3917965

Re: xfstests generic/347 seems unhappy on todays block for-next tree

[Jens Axboe]

On 12/10/18 2:53 PM, Christoph Hellwig wrote:
> On Mon, Dec 10, 2018 at 02:39:39PM -0700, Jens Axboe wrote:
>> On 12/10/18 2:37 PM, Christoph Hellwig wrote:
>>> This test is described as:
>>>
>>> # Test very basic thin device usage, exhaustion, and growth
>>
>> Does that tree have:
>>
>> commit c616cbee97aed4bc6178f148a7240206dcdb85a6
>> Author: Jens Axboe <axboe@kernel.dk>
>> Date:   Thu Dec 6 22:17:44 2018 -0700
>>
>>     blk-mq: punt failed direct issue to dispatch list
>>
>> ?
> 
> yes.
> 
> The latest commit is 6f75723190d88e1319bea623bfe0292bf3917965

Reproduces here, guessing it's the inflight counters... Trying without.

-- 
Jens Axboe

Re: xfstests generic/347 seems unhappy on todays block for-next tree

[Jens Axboe]

On 12/10/18 3:00 PM, Jens Axboe wrote:
> On 12/10/18 2:53 PM, Christoph Hellwig wrote:
>> On Mon, Dec 10, 2018 at 02:39:39PM -0700, Jens Axboe wrote:
>>> On 12/10/18 2:37 PM, Christoph Hellwig wrote:
>>>> This test is described as:
>>>>
>>>> # Test very basic thin device usage, exhaustion, and growth
>>>
>>> Does that tree have:
>>>
>>> commit c616cbee97aed4bc6178f148a7240206dcdb85a6
>>> Author: Jens Axboe <axboe@kernel.dk>
>>> Date:   Thu Dec 6 22:17:44 2018 -0700
>>>
>>>     blk-mq: punt failed direct issue to dispatch list
>>>
>>> ?
>>
>> yes.
>>
>> The latest commit is 6f75723190d88e1319bea623bfe0292bf3917965
> 
> Reproduces here, guessing it's the inflight counters... Trying without.

Yep, works without the inflight changes. Deferring to Mike to sort
this one out.

-- 
Jens Axboe

Re: xfstests generic/347 seems unhappy on todays block for-next tree

[Jens Axboe]

On 12/10/18 3:09 PM, Jens Axboe wrote:
> On 12/10/18 3:00 PM, Jens Axboe wrote:
>> On 12/10/18 2:53 PM, Christoph Hellwig wrote:
>>> On Mon, Dec 10, 2018 at 02:39:39PM -0700, Jens Axboe wrote:
>>>> On 12/10/18 2:37 PM, Christoph Hellwig wrote:
>>>>> This test is described as:
>>>>>
>>>>> # Test very basic thin device usage, exhaustion, and growth
>>>>
>>>> Does that tree have:
>>>>
>>>> commit c616cbee97aed4bc6178f148a7240206dcdb85a6
>>>> Author: Jens Axboe <axboe@kernel.dk>
>>>> Date:   Thu Dec 6 22:17:44 2018 -0700
>>>>
>>>>     blk-mq: punt failed direct issue to dispatch list
>>>>
>>>> ?
>>>
>>> yes.
>>>
>>> The latest commit is 6f75723190d88e1319bea623bfe0292bf3917965
>>
>> Reproduces here, guessing it's the inflight counters... Trying without.
> 
> Yep, works without the inflight changes. Deferring to Mike to sort
> this one out.

I think this should work much better...


diff --git a/drivers/md/dm.c b/drivers/md/dm.c
index 70568f8b6c53..1389a467ab63 100644
--- a/drivers/md/dm.c
+++ b/drivers/md/dm.c
@@ -650,14 +650,14 @@ static bool md_in_flight(struct mapped_device *md)
 {
 	int cpu;
 	struct hd_struct *part = &dm_disk(md)->part0;
+	long sum = 0;
 
 	for_each_possible_cpu(cpu) {
-		if (part_stat_local_read_cpu(part, in_flight[0], cpu) ||
-		    part_stat_local_read_cpu(part, in_flight[1], cpu))
-			return true;
+		sum += part_stat_local_read_cpu(part, in_flight[0], cpu);
+		sum += part_stat_local_read_cpu(part, in_flight[1], cpu);
 	}
 
-	return false;
+	return sum != 0;
 }
 
 static void start_io_acct(struct dm_io *io)

-- 
Jens Axboe

Re: xfstests generic/347 seems unhappy on todays block for-next tree

[Jens Axboe]

On 12/10/18 3:36 PM, Jens Axboe wrote:
> On 12/10/18 3:09 PM, Jens Axboe wrote:
>> On 12/10/18 3:00 PM, Jens Axboe wrote:
>>> On 12/10/18 2:53 PM, Christoph Hellwig wrote:
>>>> On Mon, Dec 10, 2018 at 02:39:39PM -0700, Jens Axboe wrote:
>>>>> On 12/10/18 2:37 PM, Christoph Hellwig wrote:
>>>>>> This test is described as:
>>>>>>
>>>>>> # Test very basic thin device usage, exhaustion, and growth
>>>>>
>>>>> Does that tree have:
>>>>>
>>>>> commit c616cbee97aed4bc6178f148a7240206dcdb85a6
>>>>> Author: Jens Axboe <axboe@kernel.dk>
>>>>> Date:   Thu Dec 6 22:17:44 2018 -0700
>>>>>
>>>>>     blk-mq: punt failed direct issue to dispatch list
>>>>>
>>>>> ?
>>>>
>>>> yes.
>>>>
>>>> The latest commit is 6f75723190d88e1319bea623bfe0292bf3917965
>>>
>>> Reproduces here, guessing it's the inflight counters... Trying without.
>>
>> Yep, works without the inflight changes. Deferring to Mike to sort
>> this one out.
> 
> I think this should work much better...

We can improve upon that, we don't need to ever read the inflight
counter from IO completion. Testing this one now.


diff --git a/drivers/md/dm.c b/drivers/md/dm.c
index 70568f8b6c53..79ad4b3d215c 100644
--- a/drivers/md/dm.c
+++ b/drivers/md/dm.c
@@ -650,14 +650,14 @@ static bool md_in_flight(struct mapped_device *md)
 {
 	int cpu;
 	struct hd_struct *part = &dm_disk(md)->part0;
+	long sum = 0;
 
 	for_each_possible_cpu(cpu) {
-		if (part_stat_local_read_cpu(part, in_flight[0], cpu) ||
-		    part_stat_local_read_cpu(part, in_flight[1], cpu))
-			return true;
+		sum += part_stat_local_read_cpu(part, in_flight[0], cpu);
+		sum += part_stat_local_read_cpu(part, in_flight[1], cpu);
 	}
 
-	return false;
+	return sum != 0;
 }
 
 static void start_io_acct(struct dm_io *io)
@@ -691,10 +691,8 @@ static void end_io_acct(struct dm_io *io)
 				    true, duration, &io->stats_aux);
 
 	/* nudge anyone waiting on suspend queue */
-	if (unlikely(waitqueue_active(&md->wait))) {
-		if (!md_in_flight(md))
-			wake_up(&md->wait);
-	}
+	if (unlikely(waitqueue_active(&md->wait)))
+		wake_up(&md->wait);
 }
 
 /*

-- 
Jens Axboe