Re: Large latency with bcache for Ceph OSD

["Norman.Kern" <norman.kern@gmx.com>]

On 2021/2/24 下午4:52, Coly Li wrote:
> On 2/22/21 7:48 AM, Norman.Kern wrote:
>> Ping.
>>
>> I'm confused on the SYNC I/O on bcache. why SYNC I/O must be writen back
>> for persistent cache?  It can cause some latency.
>>
>> @Coly, can you give help me to explain why bcache handle O_SYNC like this.?
>>
>>
> Hmm, normally we won't observe the application issuing I/Os on backing
> device except for,
> - I/O bypass by SSD congestion
> - Sequential I/O request
> - Dirty buckets exceeds the cutoff threshold
> - Write through mode
>
> Do you set the write/read congestion threshold to 0 ?

Thanks for you reply.

I have set the threshold to zero, all configs:

#make-bcache -C -b 4m -w 4k --discard --cache_replacement_policy=lru /dev/sdm
#make-bcache -B --writeback -w 4KiB /dev/sdn --wipe-bcache
congested_read_threshold_us = 0
congested_write_threshold_us = 0

# I tried to set sequential_cutoff to 0, but it didn't solve it.

sequential_cutoff = 4194304
writeback_percent = 40
cache_mode = writeback

I renew the cluster， run for hours and reproduced the problem. I check the cache status:

root@WXS0106:/root/perf-tools# cat /sys/fs/bcache/d87713c6-2e76-4a09-8517-d48306468659/cache_available_percent
29
root@WXS0106:/root/perf-tools# cat /sys/fs/bcache/d87713c6-2e76-4a09-8517-d48306468659/internal/cutoff_writeback_sync
70
'Dirty buckets exceeds the cutoff threshold' caused the problem?  My configs  are wrong or other reasons?

>
> Coly Li
>
>> On 2021/2/18 下午3:56, Norman.Kern wrote:
>>> Hi guys,
>>>
>>> I am testing ceph with bcache, I found some I/O with O_SYNC writeback
>>> to HDD, which caused large latency on HDD, I trace the I/O with iosnoop:
>>>
>>> ./iosnoop  -Q -ts -d '8,192
>>>
>>> Tracing block I/O for 1 seconds (buffered)...
>>> STARTs          ENDs            COMM         PID    TYPE DEV
>>> BLOCK        BYTES     LATms
>>>
>>> 1809296.292350  1809296.319052  tp_osd_tp    22191  R    8,192
>>> 4578940240   16384     26.70
>>> 1809296.292330  1809296.320974  tp_osd_tp    22191  R    8,192
>>> 4577938704   16384     28.64
>>> 1809296.292614  1809296.323292  tp_osd_tp    22191  R    8,192
>>> 4600404304   16384     30.68
>>> 1809296.292353  1809296.325300  tp_osd_tp    22191  R    8,192
>>> 4578343088   16384     32.95
>>> 1809296.292340  1809296.328013  tp_osd_tp    22191  R    8,192
>>> 4578055472   16384     35.67
>>> 1809296.292606  1809296.330518  tp_osd_tp    22191  R    8,192
>>> 4578581648   16384     37.91
>>> 1809295.169266  1809296.334041  bstore_kv_fi 17266  WS   8,192
>>> 4244996360   4096    1164.78
>>> 1809296.292618  1809296.336349  tp_osd_tp    22191  R    8,192
>>> 4602631760   16384     43.73
>>> 1809296.292618  1809296.338812  tp_osd_tp    22191  R    8,192
>>> 4602632976   16384     46.19
>>> 1809296.030103  1809296.342780  tp_osd_tp    22180  WS   8,192
>>> 4741276048   131072   312.68
>>> 1809296.292347  1809296.345045  tp_osd_tp    22191  R    8,192
>>> 4609037872   16384     52.70
>>> 1809296.292620  1809296.345109  tp_osd_tp    22191  R    8,192
>>> 4609037904   16384     52.49
>>> 1809296.292612  1809296.347251  tp_osd_tp    22191  R    8,192
>>> 4578937616   16384     54.64
>>> 1809296.292621  1809296.351136  tp_osd_tp    22191  R    8,192
>>> 4612654992   16384     58.51
>>> 1809296.292341  1809296.353428  tp_osd_tp    22191  R    8,192
>>> 4578220656   16384     61.09
>>> 1809296.292342  1809296.353864  tp_osd_tp    22191  R    8,192
>>> 4578220880   16384     61.52
>>> 1809295.167650  1809296.358510  bstore_kv_fi 17266  WS   8,192
>>> 4923695960   4096    1190.86
>>> 1809296.292347  1809296.361885  tp_osd_tp    22191  R    8,192
>>> 4607437136   16384     69.54
>>> 1809296.029363  1809296.367313  tp_osd_tp    22180  WS   8,192
>>> 4739824400   98304    337.95
>>> 1809296.292349  1809296.370245  tp_osd_tp    22191  R    8,192
>>> 4591379888   16384     77.90
>>> 1809296.292348  1809296.376273  tp_osd_tp    22191  R    8,192
>>> 4591289552   16384     83.92
>>> 1809296.292353  1809296.378659  tp_osd_tp    22191  R    8,192
>>> 4578248656   16384     86.31
>>> 1809296.292619  1809296.384835  tp_osd_tp    22191  R    8,192
>>> 4617494160   65536     92.22
>>> 1809295.165451  1809296.393715  bstore_kv_fi 17266  WS   8,192
>>> 1355703120   4096    1228.26
>>> 1809295.168595  1809296.401560  bstore_kv_fi 17266  WS   8,192
>>> 1122200      4096    1232.96
>>> 1809295.165221  1809296.408018  bstore_kv_fi 17266  WS   8,192
>>> 960656       4096    1242.80
>>> 1809295.166737  1809296.411505  bstore_kv_fi 17266  WS   8,192
>>> 57682504     4096    1244.77
>>> 1809296.292352  1809296.418123  tp_osd_tp    22191  R    8,192
>>> 4579459056   32768    125.77
>>>
>>> I'm confused why write with O_SYNC must writeback on the backend
>>> storage device?  And when I used bcache for a time,
>>>
>>> the latency increased a lot.(The SSD is not very busy), There's some
>>> best practices on configuration?
>>>