From: Coly Li <colyli@suse.de>
To: "Norman.Kern" <norman.kern@gmx.com>
Cc: linux-block@vger.kernel.org, axboe@kernel.dk,
linux-bcache@vger.kernel.org
Subject: Re: Large latency with bcache for Ceph OSD
Date: Thu, 25 Feb 2021 22:44:05 +0800 [thread overview]
Message-ID: <96daa0bf-c8e1-a334-14cb-2d260aed5115@suse.de> (raw)
In-Reply-To: <cfe2746f-18a7-a768-ea72-901793a3133e@gmx.com>
On 2/25/21 9:00 PM, Norman.Kern wrote:
> I made a test:
BTW, what is the version of your kernel, and your bcache-tool, and which
distribution is running ?
>
> - Stop writing and wait for dirty data writen back
>
> $ lsblk
> NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
> sdf 8:80 0 7.3T 0 disk
> └─bcache0 252:0 0 7.3T 0 disk
> └─ceph--32a481f9--313c--417e--aaf7--bdd74515fd86-osd--data--2f670929--3c8a--45dd--bcef--c60ce3ee08e1 253:1 0 7.3T 0 lvm
> sdd 8:48 0 7.3T 0 disk
> sdb 8:16 0 7.3T 0 disk
> sdk 8:160 0 893.8G 0 disk
> └─bcache0 252:0 0 7.3T 0 disk
> └─ceph--32a481f9--313c--417e--aaf7--bdd74515fd86-osd--data--2f670929--3c8a--45dd--bcef--c60ce3ee08e1 253:1 0 7.3T 0 lvm
> $ cat /sys/block/bcache0/bcache/dirty_data
> 0.0k
>
> root@WXS0106:~# bcache-super-show /dev/sdf
> sb.magic ok
> sb.first_sector 8 [match]
> sb.csum 71DA9CA968B4A625 [match]
> sb.version 1 [backing device]
>
> dev.label (empty)
> dev.uuid d07dc435-129d-477d-8378-a6af75199852
> dev.sectors_per_block 8
> dev.sectors_per_bucket 1024
> dev.data.first_sector 16
> dev.data.cache_mode 1 [writeback]
> dev.data.cache_state 1 [clean]
> cset.uuid d87713c6-2e76-4a09-8517-d48306468659
>
> - check the available cache
>
> # cat /sys/fs/bcache/d87713c6-2e76-4a09-8517-d48306468659/cache_available_percent
> 27
>
What is the content from
/sys/fs/bcache/<cache-set-uuid>/cache0/priority_stats ? Can you past
here too.
There is no dirty blocks, but cache is occupied 78% buckets, if you are
using 5.8+ kernel, then a gc is probably desired.
You may try to trigger a gc by writing to
sys/fs/bcache/<cache-set-uuid>/internal/trigger_gc
>
> As the doc described:
>
> cache_available_percent
> Percentage of cache device which doesn’t contain dirty data, and could potentially be used for writeback. This doesn’t mean this space isn’t used for clean cached data; the unused statistic (in priority_stats) is typically much lower.
> When all dirty data writen back, why cache_available_percent was not 100?
>
> And when I start the write I/O, the new writen didn't replace the clean cache(it think the cache is diry now?), so it cause the hdd with large latency:
>
> ./bin/iosnoop -Q -d '8,80'
>
> <...> 73338 WS 8,80 3513701472 4096 217.69
> <...> 73338 WS 8,80 3513759360 4096 448.80
> <...> 73338 WS 8,80 3562211912 4096 511.69
> <...> 73335 WS 8,80 3562212528 4096 505.08
> <...> 73339 WS 8,80 3562213376 4096 501.19
> <...> 73336 WS 8,80 3562213992 4096 511.16
> <...> 73343 WS 8,80 3562214016 4096 511.74
> <...> 73340 WS 8,80 3562214128 4096 512.95
> <...> 73329 WS 8,80 3562214208 4096 510.48
> <...> 73338 WS 8,80 3562214600 4096 518.64
> <...> 73341 WS 8,80 3562214632 4096 519.09
> <...> 73342 WS 8,80 3562214664 4096 518.28
> <...> 73336 WS 8,80 3562214688 4096 519.27
> <...> 73343 WS 8,80 3562214736 4096 528.31
> <...> 73339 WS 8,80 3562214784 4096 530.13
>
I just wondering why gc thread does not run up ....
Thanks.
Coly Li
next prev parent reply other threads:[~2021-02-25 14:45 UTC|newest]
Thread overview: 11+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-02-18 7:56 Large latency with bcache for Ceph OSD Norman.Kern
2021-02-21 23:48 ` Norman.Kern
2021-02-24 8:52 ` Coly Li
2021-02-25 2:22 ` Norman.Kern
2021-02-25 2:23 ` Norman.Kern
2021-02-25 13:00 ` Norman.Kern
2021-02-25 14:44 ` Coly Li [this message]
2021-02-26 8:57 ` Norman.Kern
2021-02-26 9:54 ` Coly Li
2021-03-02 2:03 ` Norman.Kern
2021-03-02 5:30 ` Norman.Kern
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=96daa0bf-c8e1-a334-14cb-2d260aed5115@suse.de \
--to=colyli@suse.de \
--cc=axboe@kernel.dk \
--cc=linux-bcache@vger.kernel.org \
--cc=linux-block@vger.kernel.org \
--cc=norman.kern@gmx.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).