Re: A lot of flush requests to the backing device

From: Dongdong Tao <dongdong.tao@canonical.com>
To: Aleksei Zakharov <zakharov.a.g@yandex.ru>
Cc: linux-bcache@vger.kernel.org
Subject: Re: A lot of flush requests to the backing device
Date: Mon, 8 Nov 2021 13:38:14 +0800	[thread overview]
Message-ID: <CAJS8hV+KdLA6c8c5OV=z_KmJN2TSWROR6k9Y6_qut4EavJ0=tA@mail.gmail.com> (raw)
In-Reply-To: <10612571636111279@vla5-f98fea902492.qloud-c.yandex.net>

[Sorry for the Spam detection ... ]

Hi Aleksei,

This is a very interesting finding, I understand that ceph blustore
will issue fdatasync requests when it tries to flush data or metadata
(via bluefs) to the OSD device. But I'm surprised to see so much
pressure it can bring to the backing device.
May I know how do you measure the number of flush requests to the
backing device per second that is sent from the bcache with the
REQ_PREFLUSH flag?  (ftrace to some bcache tracepoint ?)

My understanding is that the bcache doesn't need to wait for the flush
requests to be completed from the backing device in order to finish
the write request, since it used a new bio "flush" for the backing
device.
So I don't think this will increase the fdatasync latency as long as
the write can be performed in a writeback mode.  It does increase the
read latency if the read io missed the cache.
Or maybe I am missing something, let me know how did you observe the
latency increasing from bcache layer , I would want to do some
experiments as well?

Regards,
Dongdong

On Fri, Nov 5, 2021 at 7:21 PM Aleksei Zakharov <zakharov.a.g@yandex.ru> wrote:
>
> Hi all,
>
> I've used bcache a lot for the last three years, mostly in writeback mode with ceph, and I faced a strange behavior. When there's a heavy write load on the bcache device with a lot of fsync()/fdatasync() requests, the bcache device issues a lot of flush requests to the backing device. If the writeback rate is low, then there might be hundreds of flush requests per second issued to the backing device.
>
> If the writeback rate growths, then latency of the flush requests increases. And latency of the bcache device increases as a result and the application experiences higher disk latency. So, this behavior of bcache slows the application in it's I/O requests when writeback rate becomes high.
>
> This workload pattern with a lot of fsync()/fdatasync() requests is a common for a latency-sensitive applications. And it seems that this bcache behavior slows down this type of workloads.
>
> As I understand, if a write request with REQ_PREFLUSH is issued to bcache device, then bcache issues new empty write request with REQ_PREFLUSH to the backing device. What is the purpose of this behavior? It looks like it might be eliminated for the better performance.
>
> --
> Regards,
> Aleksei Zakharov
> alexzzz.ru