Throttling performance due to backing device latency

Throttling performance due to backing device latency

[benard_bsc]

I am currently using bcache in a configuration where 1 cache SSD is
caching IO for 3 backing HDDs. The cache mode is writeback and I have
configured bcache to disable seq cutoff and congestion tuning so all
write IO goes to the cache device:
echo writeback > /sys/block/bcache0/cache_mode
echo 0 > /sys/block/bcache0/bcache/sequential_cutoff
echo 0 > /sys/fs/bcache/<cache set>/congested_read_threshold_us
echo 0 > /sys/fs/bcache/<cache set>/congested_write_threshold_us
 
I have increased the minimum writeback rate so that the available space
for the dirty data never goes to 100%
echo 16384 > /sys/block/bcache0/bcache/writeback_rate_minimum
 
 
I have noticed that during times of higher throughput the performance
of fsynced writes (and other high IOPs workloads as well) seems to
suffer a disproportionate amount. The throughput is something that the
ssd should be able to handle easily and it still has enough cache to
buffer any writes there so I am confused as to why performance would
suffer to such a high degree. Are there any internal mechanisms that
perhaps measure latency to the backing devices and then throttle IO? If
so how would I go about tuning those?
 
Regards,
 
Benard

Re: Throttling performance due to backing device latency

[gius db]

Hi.

I'm (negatively) surprised by the non-reaction from the developers
(and everyone).

The point and the proposed solution is not esoteric, it doesn't
concern strange or complicated things, but something trivial and
evident (when using writeback, cache performance is lost unnecessarily
and without warning).

Okay, I'll be wrong, and anyway bcache for me, it's useful, it works
well, I can fix it.

Throttling performance due to backing device latency

[gius db]

Hi.

bcache does not have a good throttling strategy for dirty cache writeback.
(I personally think the available ones are wrong.)

I was forced to start writing a bash script to adjust writeback speed, so that:
- When the discs have medium/low usage, the speed is reduced to a minimum.
- When they have very low or zero usage, the speed is greatly increased.

In this way I get the two fundamental things:
- having the maximum dirty cache used  (when needed).
- having zero dirty cache when possible (to avoid data loss, to have
more dirty cache available).