Low hit ratio and cache usage

Low hit ratio and cache usage

[Santiago Castillo Oli]

Hi all!


I'm using bcache and I think I have a rather low hit ratio and cache 
occupation.


My setup is:

- Cache device: 82 GiB partition on a SSD drive. Bucket size=4M. The 
partition is aligned on a Gigabyte boundary.

- Backing device: 3.6 TiB partition on a HDD drive. There is 732 GiB of 
data usage on this partition. This 732 GiB are used by 9 qcow2 files 
assigned to 3 VMs running on the host.

- Neither the SDD nor HDD drives have another partitions in use.

- After 24 hours of use, according to priority_stats the cache is 75% 
Unused (63 GiB Unused - 19 GiB used), but...

- ... according to "smartctl -a" in those 24 hours "Writes to Flash" has 
increased in 160 GiB and "GB written from host" has increased in 90 GiB

- cache_hit_ratio is 10 %



- I'm using maximum bucket size (4M) trying to minimize write 
amplification. With this bucket size, "Writes to Flash" (160) to "GB 
written from host"(90) ratio is 1,78. Previously, some days ago, I was 
using default bucket size. The write amplification ratio then was 2,01.

- Isn't the cache_hit_ratio (10%) a bit low?

- Is it normal that, after 24 hours running, the cache occupation is 
that low (82-63 = 19GiB, 25%)  when the host has written 90 GiB to the 
cache device in the same period? I don´t understand why 90 GiB of data 
has been written to fill 19 GiB of cache.


Any ideas?


Thank you and regards.


-- 
___________________________________________________________

Re: Low hit ratio and cache usage

[Coly Li]

On 6/4/21 7:07 PM, Santiago Castillo Oli wrote:
> Hi all!
>
>
> I'm using bcache and I think I have a rather low hit ratio and cache
> occupation.
>
>
> My setup is:
>
> - Cache device: 82 GiB partition on a SSD drive. Bucket size=4M. The
> partition is aligned on a Gigabyte boundary.
>
> - Backing device: 3.6 TiB partition on a HDD drive. There is 732 GiB
> of data usage on this partition. This 732 GiB are used by 9 qcow2
> files assigned to 3 VMs running on the host.
>
> - Neither the SDD nor HDD drives have another partitions in use.
>
> - After 24 hours of use, according to priority_stats the cache is 75%
> Unused (63 GiB Unused - 19 GiB used), but...
>
> - ... according to "smartctl -a" in those 24 hours "Writes to Flash"
> has increased in 160 GiB and "GB written from host" has increased in
> 90 GiB
>
> - cache_hit_ratio is 10 %
>
>
>
> - I'm using maximum bucket size (4M) trying to minimize write
> amplification. With this bucket size, "Writes to Flash" (160) to "GB
> written from host"(90) ratio is 1,78. Previously, some days ago, I was
> using default bucket size. The write amplification ratio then was 2,01.
>
> - Isn't the cache_hit_ratio (10%) a bit low?
>
> - Is it normal that, after 24 hours running, the cache occupation is
> that low (82-63 = 19GiB, 25%)  when the host has written 90 GiB to the
> cache device in the same period? I don´t understand why 90 GiB of data
> has been written to fill 19 GiB of cache.
>
>
> Any ideas?
>
>
> Thank you and regards.
>
>

What is the kernel version and where do you have the kernel ?  And what
is the workload on your machine ?

Most of the read requests are missing, so they will read from backing
device and refilled into cache device as used-and-clean data. Once there
is no enough space to hold more read-cached data, garbage colleague may
retire the used-and-clean data very fast and make available room for new
refilling read data. The 19GB data might be existing data from last time gc.

Coly Li

Re: Low hit ratio and cache usage

[Santiago Castillo Oli]

Hi Pierre!


I see your point about Copy On Write, but in this scenario most of the 
data is only written once and read many times. I hoped bcache to perform 
better as a read cache. I´m afraid that bcache is only caching written 
(new and modified) blocks, not blocks already in backing device but not 
in cache device. Cache device was attached with most of data already 
resting on backing device.


What other setup would you say to be an optimal configuration to speed 
up VMs I/O using qcow2 files?


Thank you and regards


El 04/06/2021 a las 14:00, Pierre Juhen escribió:
>
> Hi !
>
> COW from qcow2 means Copy On Write.
>
> It means that a new block is written for each modification on an 
> existing block.
>
> Therefore, a "living" block is read only once, and the statistics are 
> not favorable keep the blocks in the cache.
>
> Only the "static" files (OS and frequently used program) benefit from 
> the cache.
>
> So I think that qcow2 and bcache might not be a optimal configuration.
>
> Any complement on this quick analysis ?
>
> Regards,
>
>
> Le 04/06/2021 à 13:07, Santiago Castillo Oli a écrit :
>> Hi all!
>>
>>
>> I'm using bcache and I think I have a rather low hit ratio and cache 
>> occupation.
>>
>>
>> My setup is:
>>
>> - Cache device: 82 GiB partition on a SSD drive. Bucket size=4M. The 
>> partition is aligned on a Gigabyte boundary.
>>
>> - Backing device: 3.6 TiB partition on a HDD drive. There is 732 GiB 
>> of data usage on this partition. This 732 GiB are used by 9 qcow2 
>> files assigned to 3 VMs running on the host.
>>
>> - Neither the SDD nor HDD drives have another partitions in use.
>>
>> - After 24 hours of use, according to priority_stats the cache is 75% 
>> Unused (63 GiB Unused - 19 GiB used), but...
>>
>> - ... according to "smartctl -a" in those 24 hours "Writes to Flash" 
>> has increased in 160 GiB and "GB written from host" has increased in 
>> 90 GiB
>>
>> - cache_hit_ratio is 10 %
>>
>>
>>
>> - I'm using maximum bucket size (4M) trying to minimize write 
>> amplification. With this bucket size, "Writes to Flash" (160) to "GB 
>> written from host"(90) ratio is 1,78. Previously, some days ago, I 
>> was using default bucket size. The write amplification ratio then was 
>> 2,01.
>>
>> - Isn't the cache_hit_ratio (10%) a bit low?
>>
>> - Is it normal that, after 24 hours running, the cache occupation is 
>> that low (82-63 = 19GiB, 25%)  when the host has written 90 GiB to 
>> the cache device in the same period? I don´t understand why 90 GiB of 
>> data has been written to fill 19 GiB of cache.
>>
>>
>> Any ideas?
>>
>>
>> Thank you and regards.
>>
>>
-- 
___________________________________________________________

Re: Low hit ratio and cache usage

[Santiago Castillo Oli]

Hi Coli!


El 04/06/2021 a las 14:05, Coly Li escribió:
> What is the kernel version and where do you have the kernel ?  And what
> is the workload on your machine ?

I'm using debian 10 with default debian kernel (4.19.0-16-amd64) in host 
and guests.

For virtualization I'm using KVM.


There is a host, where bcache is running. The filesystem over bcache 
device is ext4. In that filesystem there is only 9 qcow2 files user by 
three VM guests. Two VM are running small nextcloud instances, another 
one is running transmission (bittorrent) for feeding debian and other 
distro iso files (30 files - 60 GiB approx.)


> Most of the read requests are missing, so they will read from backing
> device and refilled into cache device as used-and-clean data. Once there
> is no enough space to hold more read-cached data, garbage colleague may
> retire the used-and-clean data very fast and make available room for new
> refilling read data. The 19GB data might be existing data from last time gc.

Is it possible to know GC last execution time?


Regards and thank you.


-- 

___________________________________________________________

Re: Low hit ratio and cache usage

[Coly Li]

On 6/4/21 8:35 PM, Santiago Castillo Oli wrote:
> Hi Coli!
>
>
> El 04/06/2021 a las 14:05, Coly Li escribió:
>> What is the kernel version and where do you have the kernel ?  And what
>> is the workload on your machine ?
>
> I'm using debian 10 with default debian kernel (4.19.0-16-amd64) in
> host and guests.
>
> For virtualization I'm using KVM.


The kernel version is too old. I strongly suggest to use 5.3+ kernel,
which most of obvious bugs were fixed.
Then let's see what will happen.

>
>
> There is a host, where bcache is running. The filesystem over bcache
> device is ext4. In that filesystem there is only 9 qcow2 files user by
> three VM guests. Two VM are running small nextcloud instances, another
> one is running transmission (bittorrent) for feeding debian and other
> distro iso files (30 files - 60 GiB approx.)
>
>
>> Most of the read requests are missing, so they will read from backing
>> device and refilled into cache device as used-and-clean data. Once there
>> is no enough space to hold more read-cached data, garbage colleague may
>> retire the used-and-clean data very fast and make available room for new
>> refilling read data. The 19GB data might be existing data from last
>> time gc.
>
> Is it possible to know GC last execution time?
>

See /sys/fs/bcache/<UUID>/internal/btree_gc_last_sec



Coly Li

Re: Low hit ratio and cache usage

[Kai Krakow]

Hello!

Am Fr., 4. Juni 2021 um 14:36 Uhr schrieb Santiago Castillo Oli
<scastillo@aragon.es>:
>
> Hi Coli!
>
>
> El 04/06/2021 a las 14:05, Coly Li escribió:
> > What is the kernel version and where do you have the kernel ?  And what
> > is the workload on your machine ?
>
> I'm using debian 10 with default debian kernel (4.19.0-16-amd64) in host
> and guests.
>
> For virtualization I'm using KVM.
>
>
> There is a host, where bcache is running. The filesystem over bcache
> device is ext4. In that filesystem there is only 9 qcow2 files user by
> three VM guests. Two VM are running small nextcloud instances, another
> one is running transmission (bittorrent) for feeding debian and other
> distro iso files (30 files - 60 GiB approx.)

Besides Coly recommending to use a newer kernel, I think there may be
some misunderstanding of how bcache works:

* bcache is mostly about reducing latency so it skips sequential
access, you should measure block access latency instead of throughput
or fill rate

* thus, it probably fills your cache very slowly if a lot of patterns
are sequential

* ext4 has a write journal which turns many random write patterns into
sequential write patterns, YMMV if you disable ordered data mode or
journalling

* qcow2 is copy-on-write: new blocks are appended, resulting in
possible write amplification in bcache, it also creates sequential
write patterns

* kvm/qemu probably use directio / uncached IO by default which may
bypass bcache or caching completely, you should try a different IO
mode in kvm (e.g. unsafe cached)


Regards,
Kai