[linux-lvm] dm-cache / lvmcache - no cleanup for dirty data in writeback/cleaner mode

[linux-lvm] dm-cache / lvmcache - no cleanup for dirty data in writeback/cleaner mode

[CoolCold]

Good day!

Main issue as I see it: dirty data size grows, cached data size
declines, no automagic flushing happens (or much slower than growing
up)
Sub issue: cleaner policy flushes data back to origin device very slowly.

  LVM version:     2.03.07(2) (2019-11-30)
  Kernel 5.4.0-109-generic #123-Ubuntu SMP Fri Apr 8 09:10:54 UTC 2022
x86_64 x86_64 x86_64 GNU/Linux

Sizes: 2TB origin/700GB cache device
  [cachedata_cpool]       vg0 Cwi---C--- 700.00g
         99.99  21.21           72.70
  [cachedata_cpool_cdata] vg0 Cwi-ao---- 700.00g
  [cachedata_cpool_cmeta] vg0 ewi-ao----  20.00m

Let's start from cleaner mode, as it what is happening on server right now:

root@TasteWP2:~# lvs -o+cache_policy,cache_settings
  LV       VG  Attr       LSize   Pool              Origin       Data%
 Meta%  Move Log Cpy%Sync Convert CachePolicy CacheSettings
  data     vg0 Cwi-aoC---   2.00t [cachedata_cpool] [data_corig] 99.99
 21.21           72.70            cleaner
migration_threshold=65536

and Cpy%Sync moving down very slowly - over 4 hours, it moved from
72.74% to 72.70% only.

The origin device (a couple of HDDs in mdadm RAID1) is almost idling,
5-15% busy based on iostat.

graphical representation from collectd/rrd - https://snipboard.io/WNdGhz.jpg

What I've tried to do: play around setting migration_threshhold, but
see no changes
lvchange --cachesettings 'migration_threshold=65536' vg0/data
lvchange --cachesettings 'migration_threshold=8192' vg0/data

 ---
I think this subissue leads to the main issue with dirty growing up,
not leaving space for "real" data.

My very limited knowledge, checking through history on github
https://github.com/torvalds/linux/commits/master/drivers/md/dm-cache-target.c
back to 2019 (time of kernel release)
have not provided any hints on fixes/changes added here, so I'm bit
lost. Help is appreciated.

-- 
Best regards,
[COOLCOLD-RIPN]

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/