From: Coly Li <colyli@suse.de>
To: "Norman.Kern" <norman.kern@gmx.com>
Cc: linux-bcache@vger.kernel.org, linux-block@vger.kernel.org
Subject: Re: Large latency with bcache for Ceph OSD(new mail thread)
Date: Fri, 5 Mar 2021 18:03:11 +0800 [thread overview]
Message-ID: <1fa52fcb-1886-148f-2d55-02060dce7f93@suse.de> (raw)
In-Reply-To: <91cf3980-ed9d-a5f8-4f2d-d9a79b1cbed0@gmx.com>
On 3/5/21 5:00 PM, Norman.Kern wrote:
>
> On 2021/3/2 下午9:20, Coly Li wrote:
>> On 3/2/21 6:20 PM, Norman.Kern wrote:
>>> Sorry for creating a new mail thread(the origin is so long...)
>>>
>>>
>>> I made a test again and get more infomation:
>>>
>>> root@WXS0089:~# cat /sys/block/bcache0/bcache/dirty_data
>>> 0.0k
>>> root@WXS0089:~# lsblk /dev/sda
>>> NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
>>> sda 8:0 0 447.1G 0 disk
>>> `-bcache0 252:0 0 10.9T 0 disk
>>> root@WXS0089:~# cat /sys/block/sda/bcache/priority_stats
>>> Unused: 1%
>>> Clean: 29%
>>> Dirty: 70%
>>> Metadata: 0%
>>> Average: 49
>>> Sectors per Q: 29184768
>>> Quantiles: [1 2 3 5 6 8 9 11 13 14 16 19 21 23 26 29 32 36 39 43 48 53 59 65 73 83 94 109 129 156 203]
>>> root@WXS0089:~# cat /sys/fs/bcache/066319e1-8680-4b5b-adb8-49596319154b/internal/gc_after_writeback
>>> 1
>>> You have new mail in /var/mail/root
>>> root@WXS0089:~# cat /sys/fs/bcache/066319e1-8680-4b5b-adb8-49596319154b/cache_available_percent
>>> 28
>>>
>>> I read the source codes and found if cache_available_percent > 50, it should wakeup gc while doing writeback, but it seemed not work right.
>>>
>> If gc_after_writeback is enabled, and after it is enabled and the cache
>> usage > 50%, a tag BCH_DO_AUTO_GC will be set to c->gc_after_writeback.
>> Then when the writeback completed the gc thread will wake up in force.
>>
>> so the auto gc after writeback will be triggered when,
>> 1, the bcache device is in writeback mode
>> 2, gc_after_writeback set to 1
>> 3, After 2) done, the cache usage exceeds 50% threshold.
>> 4, writeback rate set to maximum rate when the bcache device is idle (no
>> regular I/O request)
>> 5, after the writeback accomplished, the gc thread will be waken up.
>>
>> But /sys/block/bcache0/bcache/dirty_data is 0.0k doesn't mean the
>> writeback is accomplished. It is possible the writeback thread still
>> goes through all btree keys for the last try even all the dirty data are
>> flushed. Therefore you should check whether the writeback thread is
>> still active before a conclusion is made that the writeback is completed.
>>
>> BTW, do you try a Linux v5.8+ kernel and see how things are ?
>
> I have test on 5.8.X, but it doesn't help. I test on the same config on another server(480G SSD + 8T HDD),
>
What do you mean on "doesn't help" ? Do you mean the force gc does not
trigger, or something else.
> it can't reproduce, this really made me confused. I will compare the configs and try to find out the diffs.
For which behavior that it don't reproduce ?
Thanks.
Coly Li
next prev parent reply other threads:[~2021-03-05 10:03 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-03-02 10:20 Large latency with bcache for Ceph OSD(new mail thread) Norman.Kern
2021-03-02 13:20 ` Coly Li
2021-03-03 3:25 ` Norman.Kern
2021-03-05 9:00 ` Norman.Kern
2021-03-05 10:03 ` Coly Li [this message]
2021-03-08 5:47 ` Norman.Kern
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1fa52fcb-1886-148f-2d55-02060dce7f93@suse.de \
--to=colyli@suse.de \
--cc=linux-bcache@vger.kernel.org \
--cc=linux-block@vger.kernel.org \
--cc=norman.kern@gmx.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).