BUG: Hung task timeouts in for-4.10/dio

BUG: Hung task timeouts in for-4.10/dio

[Logan Gunthorpe]

[-- Attachment #1: Type: text/plain, Size: 856 bytes --]

Hi guys,

We were looking at testing the new IO polling improvements and we built
a kernel from the 'for-4.10/dio' (64ead7d) branch in linux-block.
However this branch seems to cause hung tasks when booted. Most
noticeably, dhclient seems to always hang as it tries to read from it's
leases file, and that means networking does not work on the computers we
tested on. Other tasks seemed to hang occasionally and randomly.

We tested on two machines with radically different hardware but both
running Debian Jessie.  (One is a dual-socket server system with the
root FS on an HDD and the other is an off the shelf commodity
motherboard with root on an SSD.)

We performed a bisect to find the culprit commit to be:

[b685d3d65ac791406e0dfd8779cc9b3707fea5a3] block: treat REQ_FUA and
REQ_PREFLUSH as synchronous

I've attached a bisect log.

Thanks,

Logan

[-- Attachment #2: block-bisect.log --]
[-- Type: text/x-log, Size: 1346 bytes --]

git bisect start
# good: [1001354ca34179f3db924eb66672442a173147dc] Linux 4.9-rc1
git bisect good 1001354ca34179f3db924eb66672442a173147dc
# bad: [64ead7d24b34ed40e85577e9be8bd203835e57c4] blk-mq: make the polling code adaptive
git bisect bad 64ead7d24b34ed40e85577e9be8bd203835e57c4
# bad: [70fd76140a6cb63262bd47b68d57b42e889c10ee] block,fs: use REQ_* flags directly
git bisect bad 70fd76140a6cb63262bd47b68d57b42e889c10ee
# good: [2552e3f878c2b43b41d7728a328821d8220c28da] blk-mq: get rid of confusing blk_map_ctx structure
git bisect good 2552e3f878c2b43b41d7728a328821d8220c28da
# good: [aa39ebd404423e62f74cfd3e27e9ffe7e38b2a25] cfq-iosched: use op_is_sync instead of opencoding it
git bisect good aa39ebd404423e62f74cfd3e27e9ffe7e38b2a25
# good: [67f055c798c72c49ee0c844eae0cd6e9c83b1b16] btrfs: use op_is_sync to check for synchronous requests
git bisect good 67f055c798c72c49ee0c844eae0cd6e9c83b1b16
# bad: [b685d3d65ac791406e0dfd8779cc9b3707fea5a3] block: treat REQ_FUA and REQ_PREFLUSH as synchronous
git bisect bad b685d3d65ac791406e0dfd8779cc9b3707fea5a3
# good: [6f6b29171a192e84b666c816e49d2175afbbb09f] block: don't use REQ_SYNC in the READ_SYNC definition
git bisect good 6f6b29171a192e84b666c816e49d2175afbbb09f
# first bad commit: [b685d3d65ac791406e0dfd8779cc9b3707fea5a3] block: treat REQ_FUA and REQ_PREFLUSH as synchronous

Re: BUG: Hung task timeouts in for-4.10/dio

[Jens Axboe]

On 11/08/2016 11:16 AM, Logan Gunthorpe wrote:
> Hi guys,
>
> We were looking at testing the new IO polling improvements and we built
> a kernel from the 'for-4.10/dio' (64ead7d) branch in linux-block.
> However this branch seems to cause hung tasks when booted. Most
> noticeably, dhclient seems to always hang as it tries to read from it's
> leases file, and that means networking does not work on the computers we
> tested on. Other tasks seemed to hang occasionally and randomly.
>
> We tested on two machines with radically different hardware but both
> running Debian Jessie.  (One is a dual-socket server system with the
> root FS on an HDD and the other is an off the shelf commodity
> motherboard with root on an SSD.)
>
> We performed a bisect to find the culprit commit to be:
>
> [b685d3d65ac791406e0dfd8779cc9b3707fea5a3] block: treat REQ_FUA and
> REQ_PREFLUSH as synchronous
>
> I've attached a bisect log.

I don't think that's right. The version you ran has a bug in the stats
code. Please update to the current for-4.10/dio branch (82a78cd682bf)
and I think you'll have more luck.

-- 
Jens Axboe

Re: BUG: Hung task timeouts in for-4.10/dio

[Logan Gunthorpe]

Hi Jens,

Thanks for the quick reply. I just built 82a78cd and I'm seeing the same
problem as reported.

Logan

On 08/11/16 11:21 AM, Jens Axboe wrote:
> On 11/08/2016 11:16 AM, Logan Gunthorpe wrote:
>> Hi guys,
>>
>> We were looking at testing the new IO polling improvements and we built
>> a kernel from the 'for-4.10/dio' (64ead7d) branch in linux-block.
>> However this branch seems to cause hung tasks when booted. Most
>> noticeably, dhclient seems to always hang as it tries to read from it's
>> leases file, and that means networking does not work on the computers we
>> tested on. Other tasks seemed to hang occasionally and randomly.
>>
>> We tested on two machines with radically different hardware but both
>> running Debian Jessie.  (One is a dual-socket server system with the
>> root FS on an HDD and the other is an off the shelf commodity
>> motherboard with root on an SSD.)
>>
>> We performed a bisect to find the culprit commit to be:
>>
>> [b685d3d65ac791406e0dfd8779cc9b3707fea5a3] block: treat REQ_FUA and
>> REQ_PREFLUSH as synchronous
>>
>> I've attached a bisect log.
> 
> I don't think that's right. The version you ran has a bug in the stats
> code. Please update to the current for-4.10/dio branch (82a78cd682bf)
> and I think you'll have more luck.
> 

Re: BUG: Hung task timeouts in for-4.10/dio

[Jens Axboe]

On 11/08/2016 11:59 AM, Logan Gunthorpe wrote:
> Hi Jens,
>
> Thanks for the quick reply. I just built 82a78cd and I'm seeing the same
> problem as reported.

Hmm, very odd. Does it work if you run that branch and revert the commit 
you bisected as troublesome, b685d3d65ac7?

-- 
Jens Axboe

Re: BUG: Hung task timeouts in for-4.10/dio

[Logan Gunthorpe]

Hey,

I haven't check 82a78cd, but when I tried reverting the commit in
yesterdays version there were conflicts, as a subsequent patch removed
the defines that the specific patch operated on.

Logan


On 08/11/16 12:01 PM, Jens Axboe wrote:
> On 11/08/2016 11:59 AM, Logan Gunthorpe wrote:
>> Hi Jens,
>>
>> Thanks for the quick reply. I just built 82a78cd and I'm seeing the same
>> problem as reported.
> 
> Hmm, very odd. Does it work if you run that branch and revert the commit
> you bisected as troublesome, b685d3d65ac7?
> 

Re: BUG: Hung task timeouts in for-4.10/dio

[Jens Axboe]

On 11/08/2016 12:03 PM, Logan Gunthorpe wrote:
> Hey,
>
> I haven't check 82a78cd, but when I tried reverting the commit in
> yesterdays version there were conflicts, as a subsequent patch removed
> the defines that the specific patch operated on.

Can you try and boot for-4.10/block instead?

-- 
Jens Axboe

Re: BUG: Hung task timeouts in for-4.10/dio

[Logan Gunthorpe]

Hey,

On 08/11/16 12:19 PM, Jens Axboe wrote:
> Can you try and boot for-4.10/block instead?

Yup. I'm seeing the same issue with that branch too. (b57d74a)

Thanks,

Logan

Re: BUG: Hung task timeouts in for-4.10/dio

[Jens Axboe]

On 11/08/2016 12:47 PM, Logan Gunthorpe wrote:
> Hey,
>
> On 08/11/16 12:19 PM, Jens Axboe wrote:
>> Can you try and boot for-4.10/block instead?
>
> Yup. I'm seeing the same issue with that branch too. (b57d74a)

OK, let's get some more info on this setup then, so we can get to the
bottom of that. Can you send a dmesg from a working boot? What file
systems are you using? Etc.

-- 
Jens Axboe

Re: BUG: Hung task timeouts in for-4.10/dio

[Jens Axboe]

On 11/08/2016 12:55 PM, Logan Gunthorpe wrote:
> Hey,
>
> I've attached the output of dmesg from a working boot and the output of
> mount.
>
> Pretty much all the file systems are ext4. We have some experimental
> nvme devices in this system which I did try removing to eliminate that
> possibility.
>
> Let me know if you need anything else.

You're using dm, that might be related. Mike, have you tried booting
for-4.10/block and checking if dm works fine?

-- 
Jens Axboe

Re: BUG: Hung task timeouts in for-4.10/dio

[Mike Snitzer]

On Tue, Nov 08 2016 at  3:21pm -0500,
Jens Axboe <axboe@fb.com> wrote:

> On 11/08/2016 12:55 PM, Logan Gunthorpe wrote:
> >Hey,
> >
> >I've attached the output of dmesg from a working boot and the output of
> >mount.
> >
> >Pretty much all the file systems are ext4. We have some experimental
> >nvme devices in this system which I did try removing to eliminate that
> >possibility.
> >
> >Let me know if you need anything else.

I'm not seeing Logan's above message (with dmesg) on linux-block
archives.  So I cannot yet appreciate which DM targets are in play.

> You're using dm, that might be related. Mike, have you tried booting
> for-4.10/block and checking if dm works fine?

I haven't tried for-4.10/block yet.  But I should be able to try it this
week.

Re: BUG: Hung task timeouts in for-4.10/dio

[Damien Le Moal]

Jens,

On 11/9/16 05:21, Jens Axboe wrote:
> On 11/08/2016 12:55 PM, Logan Gunthorpe wrote:
>> Hey,
>>
>> I've attached the output of dmesg from a working boot and the output of
>> mount.
>>
>> Pretty much all the file systems are ext4. We have some experimental
>> nvme devices in this system which I did try removing to eliminate that
>> possibility.
>>
>> Let me know if you need anything else.
> 
> You're using dm, that might be related. Mike, have you tried booting
> for-4.10/block and checking if dm works fine?

Using yesterday's tree, I experienced similar problems with
for-4.10/block without using dm (using ext4 on top of SSDs): random
tasks hung, starting from boot, with the machine eventually completely
freezing.

I did not dig into the problem a lot. I just looked at task stack traces
(echo t > /proc/sysrq-trigger) and noticed that hung tasks are waiting
for requests. Ex:

[   55.356418] plymouthd       D ffffffff81671758     0   353      1
0x00000000
[   55.356419]  ffff8807fbf1ec00 0000000000000000 ffff8807fba6d500
ffff8807fba3b600
[   55.356420]  ffff88081fb97900 ffff8807f04079a8 ffffffff81671758
000000000000158f
[   55.356421]  0000000000000000 ffff8807f3373800 ffff8807fba3b600
ffff88081fb97900
[   55.356421] Call Trace:
[   55.356421]  [<ffffffff81671758>] ? __schedule+0x178/0x650
[   55.356422]  [<ffffffff81671c70>] schedule+0x40/0x90
[   55.356423]  [<ffffffff816749d1>] schedule_timeout+0x2b1/0x3e0
[   55.356424]  [<ffffffff8115419d>] ? mempool_alloc_slab+0x1d/0x30
[   55.356425]  [<ffffffff810e0971>] ? ktime_get+0x41/0xb0
[   55.356426]  [<ffffffff81671574>] io_schedule_timeout+0xa4/0x110
[   55.356427]  [<ffffffff8130ee2b>] get_request+0x3fb/0x7d0
[   55.356428]  [<ffffffff8120fd83>] ? __find_get_block+0xf3/0x180
[   55.356429]  [<ffffffff810be260>] ? wait_woken+0x90/0x90
[   55.356431]  [<ffffffff813117cb>] blk_queue_bio+0xfb/0x3c0
[   55.356432]  [<ffffffff8130fb90>] generic_make_request+0xd0/0x180
[   55.356433]  [<ffffffff8130fcac>] submit_bio+0x6c/0x130
[   55.356436]  [<ffffffff81270f08>] ext4_io_submit+0x38/0x50
[   55.356437]  [<ffffffff8126c241>] ext4_writepages+0x561/0xdb0
[   55.356439]  [<ffffffff811601e1>] do_writepages+0x21/0x30
[   55.356440]  [<ffffffff811520aa>] __filemap_fdatawrite_range+0xaa/0xf0
[   55.356440]  [<ffffffff811524df>] ? __generic_file_write_iter+0x14f/0x1d0
[   55.356441]  [<ffffffff8115213c>] filemap_flush+0x1c/0x20
[   55.356442]  [<ffffffff812698bc>] ext4_alloc_da_blocks+0x2c/0x80
[   55.356443]  [<ffffffff81262268>] ext4_release_file+0x78/0xc0
[   55.356446]  [<ffffffff811db2a9>] __fput+0xb9/0x200
[   55.356447]  [<ffffffff811db42e>] ____fput+0xe/0x10
[   55.356449]  [<ffffffff81097bf5>] task_work_run+0x85/0xb0
[   55.356450]  [<ffffffff810016a7>] exit_to_usermode_loop+0x97/0xa0
[   55.356451]  [<ffffffff810019e3>] syscall_return_slowpath+0x53/0x60
[   55.356452]  [<ffffffff8167605f>] entry_SYSCALL_64_fastpath+0x92/0x94

I needed the ZBC code so I detached the head back to 5f2808f and
everything then worked fine. I will try to bisect.

Best regards.

-- 
Damien Le Moal, Ph.D.
Sr. Manager, System Software Research Group,
Western Digital Corporation
Damien.LeMoal@wdc.com
(+81) 0466-98-3593 (ext. 513593)
1 kirihara-cho, Fujisawa,
Kanagawa, 252-0888 Japan
www.wdc.com, www.hgst.com

Re: BUG: Hung task timeouts in for-4.10/dio

[Christoph Hellwig]

Ok, sounds like I'm really the one to blame.  I'll see if I can
find a reproducer.  Damien, or you using device mapper on that
system?

Re: BUG: Hung task timeouts in for-4.10/dio

[Damien Le Moal]

On 11/9/16 10:09, Christoph Hellwig wrote:
> Ok, sounds like I'm really the one to blame.  I'll see if I can
> find a reproducer.  Damien, or you using device mapper on that
> system?

No LVM/md/dm used on boot. Mount is direct to the block device (SSDs
with ext4). The devices are simple SSDs, so no polling involved.

The hangs suspiciously look like they are either background write or
flush. So I was wondering if it is indeed related to FLUSH/FUA as Logan
suggested or the background write stuff, rather than the direct-IO
optimization & polling.

Will try again/bisect to see if I can get more info.

Cheers.

-- 
Damien Le Moal, Ph.D.
Sr. Manager, System Software Research Group,
Western Digital Corporation
Damien.LeMoal@wdc.com
(+81) 0466-98-3593 (ext. 513593)
1 kirihara-cho, Fujisawa,
Kanagawa, 252-0888 Japan
www.wdc.com, www.hgst.com

Re: BUG: Hung task timeouts in for-4.10/dio

[Jens Axboe]

On 11/08/2016 06:25 PM, Damien Le Moal wrote:
>
> On 11/9/16 10:09, Christoph Hellwig wrote:
>> Ok, sounds like I'm really the one to blame.  I'll see if I can
>> find a reproducer.  Damien, or you using device mapper on that
>> system?
>
> No LVM/md/dm used on boot. Mount is direct to the block device (SSDs
> with ext4). The devices are simple SSDs, so no polling involved.
>
> The hangs suspiciously look like they are either background write or
> flush. So I was wondering if it is indeed related to FLUSH/FUA as Logan
> suggested or the background write stuff, rather than the direct-IO
> optimization & polling.

The background write stuff is not in either of those branches, plus the
backtrace would have looked different. Yours is showing us waiting for a
request. I don't think it's the direct-io or polling code, it looks like
a generic issue.

> Will try again/bisect to see if I can get more info.

Maybe try and revert the one that Logan pointed his finger at, if that
is doable.

-- 
Jens Axboe

Re: BUG: Hung task timeouts in for-4.10/dio

[Jens Axboe]

On 11/08/2016 06:28 PM, Jens Axboe wrote:
> On 11/08/2016 06:25 PM, Damien Le Moal wrote:
>>
>> On 11/9/16 10:09, Christoph Hellwig wrote:
>>> Ok, sounds like I'm really the one to blame.  I'll see if I can
>>> find a reproducer.  Damien, or you using device mapper on that
>>> system?
>>
>> No LVM/md/dm used on boot. Mount is direct to the block device (SSDs
>> with ext4). The devices are simple SSDs, so no polling involved.
>>
>> The hangs suspiciously look like they are either background write or
>> flush. So I was wondering if it is indeed related to FLUSH/FUA as Logan
>> suggested or the background write stuff, rather than the direct-IO
>> optimization & polling.
>
> The background write stuff is not in either of those branches, plus the
> backtrace would have looked different. Yours is showing us waiting for a
> request. I don't think it's the direct-io or polling code, it looks like
> a generic issue.
>
>> Will try again/bisect to see if I can get more info.
>
> Maybe try and revert the one that Logan pointed his finger at, if that
> is doable.

It smells like an accounting error. One thing that I don't like with the
current scheme is the implicit knowledge that certain flags imply sync
as well. If we clear any of those flags, then we screw up accounting at
the end.

Does this make a difference?


diff --git a/block/blk-flush.c b/block/blk-flush.c
index c486b7aa62ee..d70983e28115 100644
--- a/block/blk-flush.c
+++ b/block/blk-flush.c
@@ -395,6 +395,8 @@ void blk_insert_flush(struct request *rq)
  	if (!(fflags & (1UL << QUEUE_FLAG_FUA)))
  		rq->cmd_flags &= ~REQ_FUA;

+	rq->cmd_flags |= REQ_SYNC;
+
  	/*
  	 * An empty flush handed down from a stacking driver may
  	 * translate into nothing if the underlying device does not

-- 
Jens Axboe

Re: BUG: Hung task timeouts in for-4.10/dio

[Christoph Hellwig]

On Tue, Nov 08, 2016 at 07:02:52PM -0700, Jens Axboe wrote:
> It smells like an accounting error. One thing that I don't like with the
> current scheme is the implicit knowledge that certain flags imply sync
> as well. If we clear any of those flags, then we screw up accounting at
> the end.
>
> Does this make a difference?

That looks interesting.  In the meantime I reproduced a similar
hang, but only half-way through an xfstests run with a non-mq device.
I'll see how far I can narrow it down and will give your patch a try as
well.

Re: BUG: Hung task timeouts in for-4.10/dio

[Jens Axboe]

On 11/08/2016 07:05 PM, Christoph Hellwig wrote:
> On Tue, Nov 08, 2016 at 07:02:52PM -0700, Jens Axboe wrote:
>> It smells like an accounting error. One thing that I don't like with the
>> current scheme is the implicit knowledge that certain flags imply sync
>> as well. If we clear any of those flags, then we screw up accounting at
>> the end.
>>
>> Does this make a difference?
>
> That looks interesting.  In the meantime I reproduced a similar
> hang, but only half-way through an xfstests run with a non-mq device.
> I'll see how far I can narrow it down and will give your patch a try as
> well.

It'd only trigger on non-mq, and the symptoms (and the bisect) point at
this being an accounting issue. Damien/Logan, would be great you could
try the debug patch I sent.

My non-mq drive on my test box has write through caching, which probably
explains why I haven't seen the issue.

-- 
Jens Axboe

Re: BUG: Hung task timeouts in for-4.10/dio

[Damien Le Moal]

Jens,

On 11/9/16 11:02, Jens Axboe wrote:
> It smells like an accounting error. One thing that I don't like with the
> current scheme is the implicit knowledge that certain flags imply sync
> as well. If we clear any of those flags, then we screw up accounting at
> the end.
> 
> Does this make a difference?
> 
> 
> diff --git a/block/blk-flush.c b/block/blk-flush.c
> index c486b7aa62ee..d70983e28115 100644
> --- a/block/blk-flush.c
> +++ b/block/blk-flush.c
> @@ -395,6 +395,8 @@ void blk_insert_flush(struct request *rq)
>   	if (!(fflags & (1UL << QUEUE_FLAG_FUA)))
>   		rq->cmd_flags &= ~REQ_FUA;
> 
> +	rq->cmd_flags |= REQ_SYNC;
> +
>   	/*
>   	 * An empty flush handed down from a stacking driver may
>   	 * translate into nothing if the underlying device does not

That worked. Still seeing hangs/failures at boot with the latest
for-4.10/block (networkmanager failing, no login shell, etc) but with
the above patch, I get a clean boot and login with network working.
Once booted, simple tests on ext4 and xfs do not show any problem, but
that was only very light testing.

Best regards.

-- 
Damien Le Moal, Ph.D.
Sr. Manager, System Software Research Group,
Western Digital Corporation
Damien.LeMoal@wdc.com
(+81) 0466-98-3593 (ext. 513593)
1 kirihara-cho, Fujisawa,
Kanagawa, 252-0888 Japan
www.wdc.com, www.hgst.com

Re: BUG: Hung task timeouts in for-4.10/dio

[Jens Axboe]

On 11/08/2016 07:17 PM, Damien Le Moal wrote:
>
> Jens,
>
> On 11/9/16 11:02, Jens Axboe wrote:
>> It smells like an accounting error. One thing that I don't like with the
>> current scheme is the implicit knowledge that certain flags imply sync
>> as well. If we clear any of those flags, then we screw up accounting at
>> the end.
>>
>> Does this make a difference?
>>
>>
>> diff --git a/block/blk-flush.c b/block/blk-flush.c
>> index c486b7aa62ee..d70983e28115 100644
>> --- a/block/blk-flush.c
>> +++ b/block/blk-flush.c
>> @@ -395,6 +395,8 @@ void blk_insert_flush(struct request *rq)
>>   	if (!(fflags & (1UL << QUEUE_FLAG_FUA)))
>>   		rq->cmd_flags &= ~REQ_FUA;
>>
>> +	rq->cmd_flags |= REQ_SYNC;
>> +
>>   	/*
>>   	 * An empty flush handed down from a stacking driver may
>>   	 * translate into nothing if the underlying device does not
>
> That worked. Still seeing hangs/failures at boot with the latest
> for-4.10/block (networkmanager failing, no login shell, etc) but with
> the above patch, I get a clean boot and login with network working.
> Once booted, simple tests on ext4 and xfs do not show any problem, but
> that was only very light testing.

Great, thanks for testing! So that validates the theory. I'll get
something committed with a comment. It's not the prettiest patch, but
it'll do for now.

-- 
Jens Axboe

Re: BUG: Hung task timeouts in for-4.10/dio

[Christoph Hellwig]

On Tue, Nov 08, 2016 at 07:38:29PM -0700, Jens Axboe wrote:
> Great, thanks for testing! So that validates the theory. I'll get
> something committed with a comment. It's not the prettiest patch, but
> it'll do for now.

Should we revert the change instead (logically, not with an actual
git revert as that would break) and require and add REQ_SYNC to
all users instead?

Re: BUG: Hung task timeouts in for-4.10/dio

[Jens Axboe]

On 11/08/2016 07:40 PM, Christoph Hellwig wrote:
> On Tue, Nov 08, 2016 at 07:38:29PM -0700, Jens Axboe wrote:
>> Great, thanks for testing! So that validates the theory. I'll get
>> something committed with a comment. It's not the prettiest patch, but
>> it'll do for now.
>
> Should we revert the change instead (logically, not with an actual
> git revert as that would break) and require and add REQ_SYNC to
> all users instead?

I just committed the work-around. But yes, let's have a logical revert
and require that REQ_SYNC be set for REQ_FUA|REQ_PREFLUSH to avoid it
being this fragile.

-- 
Jens Axboe

Re: BUG: Hung task timeouts in for-4.10/dio

[Damien Le Moal]

Jens,

On 11/9/16 11:45, Jens Axboe wrote:
> I just committed the work-around. But yes, let's have a logical revert
> and require that REQ_SYNC be set for REQ_FUA|REQ_PREFLUSH to avoid it
> being this fragile.

Great ! Thank you.

-- =

Damien Le Moal, Ph.D.
Sr. Manager, System Software Research Group,
Western Digital Corporation
Damien.LeMoal@wdc.com
(+81) 0466-98-3593 (ext. 513593)
1 kirihara-cho, Fujisawa,
Kanagawa, 252-0888 Japan
www.wdc.com, www.hgst.com
Western Digital Corporation (and its subsidiaries) E-mail Confidentiality N=
otice & Disclaimer:

This e-mail and any files transmitted with it may contain confidential or l=
egally privileged information of WDC and/or its affiliates, and are intende=
d solely for the use of the individual or entity to which they are addresse=
d. If you are not the intended recipient, any disclosure, copying, distribu=
tion or any action taken or omitted to be taken in reliance on it, is prohi=
bited. If you have received this e-mail in error, please notify the sender =
immediately and delete the e-mail in its entirety from your system.

Re: BUG: Hung task timeouts in for-4.10/dio

[Logan Gunthorpe]

Hey,

I just tested with the latest for-4.10/block branch and it looks like it
fixed our problem.

Thanks!

Logan

On 08/11/16 07:55 PM, Damien Le Moal wrote:
> 
> Jens,
> 
> On 11/9/16 11:45, Jens Axboe wrote:
>> I just committed the work-around. But yes, let's have a logical revert
>> and require that REQ_SYNC be set for REQ_FUA|REQ_PREFLUSH to avoid it
>> being this fragile.
> 
> Great ! Thank you.
>