Re: A lot of flush requests to the backing device

[Aleksei Zakharov <zakharov.a.g@yandex.ru>]

> [Sorry for the Spam detection ... ]
> 
> Hi Aleksei,
> 
> This is a very interesting finding, I understand that ceph blustore
> will issue fdatasync requests when it tries to flush data or metadata
> (via bluefs) to the OSD device. But I'm surprised to see so much
> pressure it can bring to the backing device.
> May I know how do you measure the number of flush requests to the
> backing device per second that is sent from the bcache with the
> REQ_PREFLUSH flag? (ftrace to some bcache tracepoint ?)
That was easy: the writeback rate was minimal and there were a lot
of write requests to the backing device in iostat -xtd 1 output and
bytes/s was too small for that number of writes. It was relatively old kernel,
so flushes were not separated in the block layer stats yet.

> 
> My understanding is that the bcache doesn't need to wait for the flush
> requests to be completed from the backing device in order to finish
> the write request, since it used a new bio "flush" for the backing
> device.
> So I don't think this will increase the fdatasync latency as long as
> the write can be performed in a writeback mode. It does increase the
> read latency if the read io missed the cache.
Hm, that might be truth for the reads, i'll do some experiments.
But, I don't see any reason to send flush request to the backing
device if there's nothing to flush.

> Or maybe I am missing something, let me know how did you observe the
> latency increasing from bcache layer , I would want to do some
> experiments as well?
I'll do some experiments and come back with more details on the
issue in a week! Already quit that job and don't work with ceph anymore,
but still thinking about this interesting issue.

> 
> Regards,
> Dongdong
> 
> On Fri, Nov 5, 2021 at 7:21 PM Aleksei Zakharov <zakharov.a.g@yandex.ru> wrote:
> 
>> Hi all,
>>
>> I've used bcache a lot for the last three years, mostly in writeback mode with ceph, and I faced a strange behavior. When there's a heavy write load on the bcache device with a lot of fsync()/fdatasync() requests, the bcache device issues a lot of flush requests to the backing device. If the writeback rate is low, then there might be hundreds of flush requests per second issued to the backing device.
>>
>> If the writeback rate growths, then latency of the flush requests increases. And latency of the bcache device increases as a result and the application experiences higher disk latency. So, this behavior of bcache slows the application in it's I/O requests when writeback rate becomes high.
>>
>> This workload pattern with a lot of fsync()/fdatasync() requests is a common for a latency-sensitive applications. And it seems that this bcache behavior slows down this type of workloads.
>>
>> As I understand, if a write request with REQ_PREFLUSH is issued to bcache device, then bcache issues new empty write request with REQ_PREFLUSH to the backing device. What is the purpose of this behavior? It looks like it might be eliminated for the better performance.
>>
>> --
>> Regards,
>> Aleksei Zakharov
>> alexzzz.ru
--
Regards,
Aleksei Zakharov
alexzzz.ru