Re: IO hang when cache do not have enough buckets on small SSD

[Coly Li <colyli@suse.de>]

On 5/26/21 1:58 PM, Jim Guo wrote:
>> What is the kernel version in your system? And where the kernel package
>> is from?
> I am using kernel version 4.19.142, I compile it from source code
> downloaded from kernel.org.
> 
>> Do you have a testing result on this idea?
> Sorry, the testing environment is not owned by me and I did not keep
> any testing result currently. I will test for this later in my own
> testing environment.
> 

OK. And I would suggest to start your work on upstream bcache code.

Coly Li


> Coly Li <colyli@suse.de> 于2021年5月17日周一 下午7:53写道：
> 
>>
>> On 5/17/21 11:54 AM, Jim Guo wrote:
>>> Hello, Mr. Li.
>>> Recently I was experiencing frequent io hang when testing with fio
>>> with 4K random write. Fio iops dropped  to 0 for about 20 seconds
>>> every several minutes.
>>> After some debugging, I discovered that it is the incremental gc that
>>> cause this problem.
>>> My cache disk is relatively small (375GiB with 4K block size and 512K
>>> bucket size), backing hdds are 4 x 1 TiB. I cannot reproduce this on
>>> another environment with bigger cache disk.
>>> When running 4K random write fio bench, the buckets are consumed  very
>>> quickly and soon it has to invalidate some bucket (this happens quite
>>> often). Since the cache disk is small, a lot of write io will soon
>>> reach sectors_to_gc and trigger gc thread. Write io will also increase
>>> search_inflight, which cause gc thread to sleep for 100ms. This will
>>> cause gc procedure to execute for a long time, and invalidating bucket
>>> for the write io will wait for the whole gc procedure.
>>> After removing the 100ms sleep from the incremental gc patch,  the io
>>> never hang any more.
>>
>> What is the kernel version in your system? And where the kernel package
>> is from?
>>
>>
>>> I think for small ssd, sleeping for 100ms seems too long or maybe
>>> write io should not trigger gc thread to sleep for 100ms?
>>> Thank you very much.
>>>
>>
>> Do you have a testing result on this idea?
>>
>>
>> Thanks.
>>
>> Coly Li