Nearly all reads are bypassing the cache

Nearly all reads are bypassing the cache

[L. Karkkainen]

I have a bcache device that has been acting up lately. I believe it 
worked correctly until this week but now I cannot get any sequential 
reads to use the cache anymore. Writes appear to be cached correctly, 
but when reading back the same sectors the data comes from spinning rust 
instead. I tried recreating the cache device but that did not help.

# dd if=/dev/bcache0 of=/dev/null bs=1M count=32768
34359738368 bytes (34 GB, 32 GiB) copied, 67.976 s, 505 MB/s

# dd if=/dev/bcache0 of=/dev/null bs=1M count=32768
34359738368 bytes (34 GB, 32 GiB) copied, 74.5009 s, 461 MB/s

# dd if=/dev/md/CacheRaid of=/dev/null bs=1M count=32768
34359738368 bytes (34 GB, 32 GiB) copied, 7.4617 s, 4.6 GB/s

Every repeated read on the same data shows similar speeds to the first 
run. Accessing through a mounted btrfs shows similar results.
Expected: On repeated rounds speeds a few gigabytes per second, as per 
the cache device speed.

# bcache-super-show /dev/mapper/BigRaw
sb.version              4 [backing device]
dev.sectors_per_block   8
dev.sectors_per_bucket  2048
dev.data.first_sector   4096
dev.data.cache_mode     0 [writethrough]
dev.data.cache_state    1 [clean]

# bcache-super-show /dev/md/CacheRaid
sb.version              3 [cache device]
dev.sectors_per_block   8
dev.sectors_per_bucket  512
dev.cache.first_sector  512
dev.cache.cache_sectors 3369647616
dev.cache.total_sectors 3369648128
dev.cache.ordered       yes
dev.cache.discard       no
dev.cache.pos           0
dev.cache.replacement   0 [lru]

# cat /sys/block/bcache0/bcache/cache/cache0/priority_stats
Unused:     91%
Clean:      8%
Dirty:      0%
Metadata:   0%
Average:    127
Sectors per Q:  9959504
Quantiles:  [0 0 5 8 12 15 18 21 24 50 85 93 99 107 114 122 131 140 149 
159 169 180 191 203 215 228 241 254 269 281 309]

Expected: Clean at least 50 % and unused below 50 % if all the data read 
was cached (I read a much larger chunk than 32 GiB to test this).

/sys/block/bcache0/bcache/stats_total/bypassed 1.9G
/sys/block/bcache0/bcache/stats_total/cache_bypass_hits 57
/sys/block/bcache0/bcache/stats_total/cache_bypass_misses 890
/sys/block/bcache0/bcache/stats_total/cache_hit_ratio 3
/sys/block/bcache0/bcache/stats_total/cache_hits 15264
/sys/block/bcache0/bcache/stats_total/cache_miss_collisions 932
/sys/block/bcache0/bcache/stats_total/cache_misses 441927
/sys/block/bcache0/bcache/sequential_cutoff 0.0k
/sys/block/bcache0/bcache/cache/congested_read_threshold_us 0
/sys/block/bcache0/bcache/cache/congested_write_threshold_us 0

# uname -a
Linux xxx 5.14.8-zen1-1-zen #1 ZEN SMP PREEMPT Sun, 26 Sep 2021 19:36:16 
+0000 x86_64 GNU/Linux