Re: Large latency with bcache for Ceph OSD

["Norman.Kern" <norman.kern@gmx.com>]

On 2021/2/25 下午10:44, Coly Li wrote:
> On 2/25/21 9:00 PM, Norman.Kern wrote:
>> I made a test:
> BTW, what is the version of your kernel, and your bcache-tool, and which
> distribution is running ?
root@WXS0106:~# uname -a
Linux WXS0106 5.4.0-58-generic #64~18.04.1-Ubuntu SMP Wed Dec 9 17:11:11 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
root@WXS0106:~# cat /etc/os-release
NAME="Ubuntu"
VERSION="16.04 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
UBUNTU_CODENAME=xenial
root@WXS0106:~# dpkg -l bcache-tools
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                                  Version                 Architecture            Description
+++-=====================================-=======================-=======================-================================================================================
ii  bcache-tools                          1.0.8-2                 amd64                   bcache userspace tools

>
>> - Stop writing and wait for dirty data writen back
>>
>> $ lsblk
>> NAME                                                                                                   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
>> sdf                                                                                                      8:80   0   7.3T  0 disk
>> └─bcache0                                                                                              252:0    0   7.3T  0 disk
>>   └─ceph--32a481f9--313c--417e--aaf7--bdd74515fd86-osd--data--2f670929--3c8a--45dd--bcef--c60ce3ee08e1 253:1    0   7.3T  0 lvm 
>> sdd                                                                                                      8:48   0   7.3T  0 disk
>> sdb                                                                                                      8:16   0   7.3T  0 disk
>> sdk                                                                                                      8:160  0 893.8G  0 disk
>> └─bcache0                                                                                              252:0    0   7.3T  0 disk
>>   └─ceph--32a481f9--313c--417e--aaf7--bdd74515fd86-osd--data--2f670929--3c8a--45dd--bcef--c60ce3ee08e1 253:1    0   7.3T  0 lvm 
>> $ cat /sys/block/bcache0/bcache/dirty_data
>> 0.0k
>>
>> root@WXS0106:~# bcache-super-show /dev/sdf
>> sb.magic                ok
>> sb.first_sector         8 [match]
>> sb.csum                 71DA9CA968B4A625 [match]
>> sb.version              1 [backing device]
>>
>> dev.label               (empty)
>> dev.uuid                d07dc435-129d-477d-8378-a6af75199852
>> dev.sectors_per_block   8
>> dev.sectors_per_bucket  1024
>> dev.data.first_sector   16
>> dev.data.cache_mode     1 [writeback]
>> dev.data.cache_state    1 [clean]
>> cset.uuid               d87713c6-2e76-4a09-8517-d48306468659
>>
>> - check the available cache
>>
>> # cat /sys/fs/bcache/d87713c6-2e76-4a09-8517-d48306468659/cache_available_percent
>> 27
>>
> What is the content from
> /sys/fs/bcache/<cache-set-uuid>/cache0/priority_stats ? Can you past
> here too.
I forgot get the info before I triggered gc..., I think I can reproduce the problem. When I reproduce it, I will collect the infomation.
>
> There is no dirty blocks, but cache is occupied 78% buckets, if you are
> using 5.8+ kernel, then a gc is probably desired.
>
> You may try to trigger a gc by writing to
> sys/fs/bcache/<cache-set-uuid>/internal/trigger_gc
>
When all cache had written back, I triggered gc, it recalled.

root@WXS0106:~# cat /sys/block/bcache0/bcache/cache/cache_available_percent
30

root@WXS0106:~# echo 1 > /sys/block/bcache0/bcache/cache/internal/trigger_gc
root@WXS0106:~# cat /sys/block/bcache0/bcache/cache/cache_available_percent
97

Why must I trigger gc manually? Is not a default action of bcache-gc thread? And I found it can only work when all dirty data written back.

>> As the doc described:
>>
>> cache_available_percent
>>     Percentage of cache device which doesn’t contain dirty data, and could potentially be used for writeback. This doesn’t mean this space isn’t used for clean cached data; the unused statistic (in priority_stats) is typically much lower.
>> When all dirty data writen back,  why cache_available_percent was not 100?
>>
>> And when I start the write I/O, the new writen didn't replace the clean cache(it think the cache is diry now?), so it cause the hdd with large latency:
>>
>> ./bin/iosnoop -Q -d '8,80'
>>
>> <...>        73338  WS   8,80     3513701472   4096     217.69
>> <...>        73338  WS   8,80     3513759360   4096     448.80
>> <...>        73338  WS   8,80     3562211912   4096     511.69
>> <...>        73335  WS   8,80     3562212528   4096     505.08
>> <...>        73339  WS   8,80     3562213376   4096     501.19
>> <...>        73336  WS   8,80     3562213992   4096     511.16
>> <...>        73343  WS   8,80     3562214016   4096     511.74
>> <...>        73340  WS   8,80     3562214128   4096     512.95
>> <...>        73329  WS   8,80     3562214208   4096     510.48
>> <...>        73338  WS   8,80     3562214600   4096     518.64
>> <...>        73341  WS   8,80     3562214632   4096     519.09
>> <...>        73342  WS   8,80     3562214664   4096     518.28
>> <...>        73336  WS   8,80     3562214688   4096     519.27
>> <...>        73343  WS   8,80     3562214736   4096     528.31
>> <...>        73339  WS   8,80     3562214784   4096     530.13
>>
> I just wondering why gc thread does not run up ....
>
>
> Thanks.
>
> Coly Li
>