[PATCH] block: fix potential IO hang when turning off io_poll

[PATCH] block: fix potential IO hang when turning off io_poll

[Jeffle Xu]

QUEUE_FLAG_POLL flag will be cleared when turning off 'io_poll', while
at that moment there may be IOs stuck in hw queue uncompleted. The
following polling routine won't help reap these IOs, since blk_poll()
will return immediately because of cleared QUEUE_FLAG_POLL flag. Thus
these IOs will hang until they finnaly time out. The hang out can be
observed by 'fio --engine=io_uring iodepth=1', while turning off
'io_poll' at the same time.

To fix this, freeze and flush the request queue first when turning off
'io_poll'.

Signed-off-by: Jeffle Xu <jefflexu@linux.alibaba.com>
---
 block/blk-sysfs.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/block/blk-sysfs.c b/block/blk-sysfs.c
index b513f1683af0..10d74741c002 100644
--- a/block/blk-sysfs.c
+++ b/block/blk-sysfs.c
@@ -430,8 +430,11 @@ static ssize_t queue_poll_store(struct request_queue *q, const char *page,
 
 	if (poll_on)
 		blk_queue_flag_set(QUEUE_FLAG_POLL, q);
-	else
+	else {
+		blk_mq_freeze_queue(q);
 		blk_queue_flag_clear(QUEUE_FLAG_POLL, q);
+		blk_mq_unfreeze_queue(q);
+	}
 
 	return ret;
 }
-- 
2.27.0

Re: [PATCH] block: fix potential IO hang when turning off io_poll

[Joseph Qi]

On 2/22/21 2:54 PM, Jeffle Xu wrote:
> QUEUE_FLAG_POLL flag will be cleared when turning off 'io_poll', while
> at that moment there may be IOs stuck in hw queue uncompleted. The
> following polling routine won't help reap these IOs, since blk_poll()
> will return immediately because of cleared QUEUE_FLAG_POLL flag. Thus
> these IOs will hang until they finnaly time out. The hang out can be
> observed by 'fio --engine=io_uring iodepth=1', while turning off
> 'io_poll' at the same time.
> 
> To fix this, freeze and flush the request queue first when turning off
> 'io_poll'.
> 
> Signed-off-by: Jeffle Xu <jefflexu@linux.alibaba.com>
> ---
>  block/blk-sysfs.c | 5 ++++-
>  1 file changed, 4 insertions(+), 1 deletion(-)
> 
> diff --git a/block/blk-sysfs.c b/block/blk-sysfs.c
> index b513f1683af0..10d74741c002 100644
> --- a/block/blk-sysfs.c
> +++ b/block/blk-sysfs.c
> @@ -430,8 +430,11 @@ static ssize_t queue_poll_store(struct request_queue *q, const char *page,
>  
>  	if (poll_on)
>  		blk_queue_flag_set(QUEUE_FLAG_POLL, q);
> -	else
> +	else {
> +		blk_mq_freeze_queue(q);
>  		blk_queue_flag_clear(QUEUE_FLAG_POLL, q);
> +		blk_mq_unfreeze_queue(q);
> +	}
>  
>  	return ret;
>  }
> 

Better to place brace to 'if' as well.

Thanks,
Joseph

Re: [PATCH] block: fix potential IO hang when turning off io_poll

[JeffleXu]

On 2/22/21 3:12 PM, Joseph Qi wrote:
> 
> 
> On 2/22/21 2:54 PM, Jeffle Xu wrote:
>> QUEUE_FLAG_POLL flag will be cleared when turning off 'io_poll', while
>> at that moment there may be IOs stuck in hw queue uncompleted. The
>> following polling routine won't help reap these IOs, since blk_poll()
>> will return immediately because of cleared QUEUE_FLAG_POLL flag. Thus
>> these IOs will hang until they finnaly time out. The hang out can be
>> observed by 'fio --engine=io_uring iodepth=1', while turning off
>> 'io_poll' at the same time.
>>
>> To fix this, freeze and flush the request queue first when turning off
>> 'io_poll'.
>>
>> Signed-off-by: Jeffle Xu <jefflexu@linux.alibaba.com>
>> ---
>>  block/blk-sysfs.c | 5 ++++-
>>  1 file changed, 4 insertions(+), 1 deletion(-)
>>
>> diff --git a/block/blk-sysfs.c b/block/blk-sysfs.c
>> index b513f1683af0..10d74741c002 100644
>> --- a/block/blk-sysfs.c
>> +++ b/block/blk-sysfs.c
>> @@ -430,8 +430,11 @@ static ssize_t queue_poll_store(struct request_queue *q, const char *page,
>>  
>>  	if (poll_on)
>>  		blk_queue_flag_set(QUEUE_FLAG_POLL, q);
>> -	else
>> +	else {
>> +		blk_mq_freeze_queue(q);
>>  		blk_queue_flag_clear(QUEUE_FLAG_POLL, q);
>> +		blk_mq_unfreeze_queue(q);
>> +	}
>>  
>>  	return ret;
>>  }
>>
> 
> Better to place brace to 'if' as well.

Got it, thanks.


-- 
Thanks,
Jeffle

Re: [PATCH] block: fix potential IO hang when turning off io_poll

[Jens Axboe]

On 2/21/21 11:54 PM, Jeffle Xu wrote:
> QUEUE_FLAG_POLL flag will be cleared when turning off 'io_poll', while
> at that moment there may be IOs stuck in hw queue uncompleted. The
> following polling routine won't help reap these IOs, since blk_poll()
> will return immediately because of cleared QUEUE_FLAG_POLL flag. Thus
> these IOs will hang until they finnaly time out. The hang out can be
> observed by 'fio --engine=io_uring iodepth=1', while turning off
> 'io_poll' at the same time.
> 
> To fix this, freeze and flush the request queue first when turning off
> 'io_poll'.

Applied, thanks. Fixed up the braces.

-- 
Jens Axboe