[linux-lvm] Unable to un-cache logical volume when chunk size is over 1MiB

[linux-lvm] Unable to un-cache logical volume when chunk size is over 1MiB

[Ryan Launchbury]

[-- Attachment #1: Type: text/plain, Size: 3651 bytes --]

Hello,

I'm having a problem uncaching logical volumes when the cache data 
chunck size is over 1MiB.
The process I'm using to uncache is: lvconvert --uncache vg/lv


The issue occurs across multiple systems with different hardware and 
different versions of LVM.

Steps to reproduce:

 1. Create origin VG & LV
 2. Add cache device over 1TB to the origin VG
 3. Create the cache data lv:
    lvcreate -n cachedata -L 1770GB cached_vg /dev/nvme0n1
 4. Create the cache metadata lv:
    lvcreate -n cachemeta -L 1770MB cached_vg /dev/nvme0n1
 5. Convert to a cache pool:
    lvconvert --type cache-pool --cachemode writethrough --poolmetadata
    cached_vg/cachemeta cached_vg/cachedata
 6. Enable caching on the origin LVM:
    lvconvert --type cache --cachepool cached_vg/cachedata
    cached_vg/filestore01
 7. Write some data to the main LV so as the cache device is used:
    dd if=/dev/zero of=/mnt/filestore01/test.dat bs=1M count=10000
 8. Check the cache stats:
    lvs -a -o +cache_total_blocks,cache_used_blocks,cache_dirty_blocks
 9. Repeating step 8 over time will show that the dirty blocks are not
    being written back at all
10. Try to uncache the device:
    lvconvert --uncache cached_vg/filestore01
11. You will get a repeating message. This will loop indefinitely and
    not decrease or complete:
    Flushing x blocks for cache cached_vg/filestore01.

After testing multiple times, the issue seems to be tied to the chunk 
size selected in step 5. The LVM man page mentions that the chunk must 
be a multiple of 32KiB, however the next chunk size automatically 
assigned over 1MiB is usually 1.03MiB. With a chunk size of 1.03MiB or 
higher, the cache is not able to flush. Creating a cache device with a 
chunk size of 1MiB or less, the cache is flushable.

Now knowing how to avoid the issue, I just need to be able to safely 
un-cache systems with do have a cache that will not flush.

Details:

Version info from lvm version:

LVM version:     2.02.171(2)-RHEL7 (2017-05-03)
   Library version: 1.02.140-RHEL7 (2017-05-03)
   Driver version:  4.35.0
   Configuration:   ./configure --build=x86_64-redhat-linux-gnu 
--host=x86_64-redhat-linux-gnu --program-prefix= 
--disable-dependency-tracking --prefix=/usr --exec-prefix=/usr 
--bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc 
--datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64 
--libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/var/lib 
--mandir=/usr/share/man --infodir=/usr/share/info 
--with-default-dm-run-dir=/run --with-default-run-dir=/run/lvm 
--with-default-pid-dir=/run --with-default-locking-dir=/run/lock/lvm 
--with-usrlibdir=/usr/lib64 --enable-lvm1_fallback --enable-fsadm 
--with-pool=internal --enable-write_install --with-user= --with-group= 
--with-device-uid=0 --with-device-gid=6 --with-device-mode=0660 
--enable-pkgconfig --enable-applib --enable-cmdlib --enable-dmeventd 
--enable-blkid_wiping --enable-python2-bindings --with-cluster=internal 
--with-clvmd=corosync --enable-cmirrord 
--with-udevdir=/usr/lib/udev/rules.d --enable-udev_sync 
--with-thin=internal --enable-lvmetad --with-cache=internal 
--enable-lvmpolld --enable-lvmlockd-dlm --enable-lvmlockd-sanlock 
--enable-dmfilemapd

System info:
System 1,2,3:
- Dell R730XD server
- 12x disk in RAID 6 to onboard PERC/Megaraid controller

System 4:
-Dell R630 server
-60x Disk (6 luns) in RAID 6 to PCI megaraid controller

The systems are currently in production, so it's quite hard for me to 
change the configuration to enable logging.

Any assistance would be much appreciated! If any more info is needed 
please let me know.
Best regards,
Ryan


[-- Attachment #2: Type: text/html, Size: 4730 bytes --]

Re: [linux-lvm] Unable to un-cache logical volume when chunk size is over 1MiB

[Zdenek Kabelac]

Dne 20.6.2018 v 11:18 Ryan Launchbury napsal(a):
> Hello,
> 
> I'm having a problem uncaching logical volumes when the cache data chunck size 
> is over 1MiB.
> The process I'm using to uncache is: lvconvert --uncache vg/lv
> 
> 
> The issue occurs across multiple systems with different hardware and different 
> versions of LVM.
> 
> Steps to reproduce:
> 
>  1. Create origin VG & LV
>  2. Add cache device over 1TB to the origin VG
>  3. Create the cache data lv:
>     lvcreate -n cachedata -L 1770GB cached_vg /dev/nvme0n1
>  4. Create the cache metadata lv:
>     lvcreate -n cachemeta -L 1770MB cached_vg /dev/nvme0n1
>  5. Convert to a cache pool:
>     lvconvert --type cache-pool --cachemode writethrough --poolmetadata
>     cached_vg/cachemeta cached_vg/cachedata
>  6. Enable caching on the origin LVM:
>     lvconvert --type cache --cachepool cached_vg/cachedata cached_vg/filestore01
>  7. Write some data to the main LV so as the cache device is used:
>     dd if=/dev/zero of=/mnt/filestore01/test.dat bs=1M count=10000
>  8. Check the cache stats:
>     lvs -a -o +cache_total_blocks,cache_used_blocks,cache_dirty_blocks
>  9. Repeating step 8 over time will show that the dirty blocks are not being
>     written back at all
> 10. Try to uncache the device:
>     lvconvert --uncache cached_vg/filestore01
> 11. You will get a repeating message. This will loop indefinitely and not
>     decrease or complete:
>     Flushing x blocks for cache cached_vg/filestore01.
> 
> After testing multiple times, the issue seems to be tied to the chunk size 
> selected in step 5. The LVM man page mentions that the chunk must be a 
> multiple of 32KiB, however the next chunk size automatically assigned over 
> 1MiB is usually 1.03MiB. With a chunk size of 1.03MiB or higher, the cache is 
> not able to flush. Creating a cache device with a chunk size of 1MiB or less, 
> the cache is flushable.
> 
> Now knowing how to avoid the issue, I just need to be able to safely un-cache 
> systems with do have a cache that will not flush.
> 
> Details:
> 
> Version info from lvm version:
> 
> LVM version:     2.02.171(2)-RHEL7 (2017-05-03)
>    Library version: 1.02.140-RHEL7 (2017-05-03)
>    Driver version:  4.35.0

What is the kernel version and Linux distro in use ?

> 
> System info:
> System 1,2,3:
> - Dell R730XD server
> - 12x disk in RAID 6 to onboard PERC/Megaraid controller
> 
> System 4:
> -Dell R630 server
> -60x Disk (6 luns) in RAID 6 to PCI megaraid controller
> 
> The systems are currently in production, so it's quite hard for me to change 
> the configuration to enable logging.
> 
> Any assistance would be much appreciated! If any more info is needed please 
> let me know.

Hi

Aren't there any kernel write errors in your 'dmegs'.
LV becomes fragile if the associated devices with cache are having HW issues 
(disk read/write errors)

Zdenek

Re: [linux-lvm] Unable to un-cache logical volume when chunk size is over 1MiB

[Ryan Launchbury]

Hi Zdenek,

Kernel is: Linux 3.10.0-693.21.1.el7.x86_64
Distro is: Centos 7 - Linux release 7.4.1708


Zdenek Kabelac wrote on 20/06/2018 11:15:
> Dne 20.6.2018 v 11:18 Ryan Launchbury napsal(a):
>> Hello,
>>
>> I'm having a problem uncaching logical volumes when the cache data 
>> chunck size is over 1MiB.
>> The process I'm using to uncache is: lvconvert --uncache vg/lv
>>
>>
>> The issue occurs across multiple systems with different hardware and 
>> different versions of LVM.
>>
>> Steps to reproduce:
>>
>>  1. Create origin VG & LV
>>  2. Add cache device over 1TB to the origin VG
>>  3. Create the cache data lv:
>>     lvcreate -n cachedata -L 1770GB cached_vg /dev/nvme0n1
>>  4. Create the cache metadata lv:
>>     lvcreate -n cachemeta -L 1770MB cached_vg /dev/nvme0n1
>>  5. Convert to a cache pool:
>>     lvconvert --type cache-pool --cachemode writethrough --poolmetadata
>>     cached_vg/cachemeta cached_vg/cachedata
>>  6. Enable caching on the origin LVM:
>>     lvconvert --type cache --cachepool cached_vg/cachedata 
>> cached_vg/filestore01
>>  7. Write some data to the main LV so as the cache device is used:
>>     dd if=/dev/zero of=/mnt/filestore01/test.dat bs=1M count=10000
>>  8. Check the cache stats:
>>     lvs -a -o +cache_total_blocks,cache_used_blocks,cache_dirty_blocks
>>  9. Repeating step 8 over time will show that the dirty blocks are 
>> not being
>>     written back at all
>> 10. Try to uncache the device:
>>     lvconvert --uncache cached_vg/filestore01
>> 11. You will get a repeating message. This will loop indefinitely and 
>> not
>>     decrease or complete:
>>     Flushing x blocks for cache cached_vg/filestore01.
>>
>> After testing multiple times, the issue seems to be tied to the chunk 
>> size selected in step 5. The LVM man page mentions that the chunk 
>> must be a multiple of 32KiB, however the next chunk size 
>> automatically assigned over 1MiB is usually 1.03MiB. With a chunk 
>> size of 1.03MiB or higher, the cache is not able to flush. Creating a 
>> cache device with a chunk size of 1MiB or less, the cache is flushable.
>>
>> Now knowing how to avoid the issue, I just need to be able to safely 
>> un-cache systems with do have a cache that will not flush.
>>
>> Details:
>>
>> Version info from lvm version:
>>
>> LVM version:     2.02.171(2)-RHEL7 (2017-05-03)
>>    Library version: 1.02.140-RHEL7 (2017-05-03)
>>    Driver version:  4.35.0
>
> What is the kernel version and Linux distro in use ?
>
>>
>> System info:
>> System 1,2,3:
>> - Dell R730XD server
>> - 12x disk in RAID 6 to onboard PERC/Megaraid controller
>>
>> System 4:
>> -Dell R630 server
>> -60x Disk (6 luns) in RAID 6 to PCI megaraid controller
>>
>> The systems are currently in production, so it's quite hard for me to 
>> change the configuration to enable logging.
>>
>> Any assistance would be much appreciated! If any more info is needed 
>> please let me know.
>
> Hi
>
> Aren't there any kernel write errors in your 'dmegs'.
> LV becomes fragile if the associated devices with cache are having HW 
> issues (disk read/write errors)
>
> Zdenek

Nope, no write errors in /var/log/dmesg. The last log entry was at 
10.871493 and the system has been on for 61 days.

Best regards,
Ryan

Re: [linux-lvm] Unable to un-cache logical volume when chunk size is over 1MiB

[Gionatan Danti]

Il 20-06-2018 12:15 Zdenek Kabelac ha scritto:
> Hi
> 
> Aren't there any kernel write errors in your 'dmegs'.
> LV becomes fragile if the associated devices with cache are having HW
> issues (disk read/write errors)
> 
> Zdenek

Is that true even when using a writethrough cache mode?

-- 
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.danti@assyoma.it - info@assyoma.it
GPG public key ID: FF5F32A8

Re: [linux-lvm] Unable to un-cache logical volume when chunk size is over 1MiB

[Ryan Launchbury]

[-- Attachment #1: Type: text/plain, Size: 804 bytes --]

Hi Gionatan,

My development system is out on rental at the moment. I'll check that for
you as soon as I can.

Best regards,
Ryan

On Fri, Jun 22, 2018 at 7:13 PM, Gionatan Danti <g.danti@assyoma.it> wrote:

> Il 20-06-2018 12:15 Zdenek Kabelac ha scritto:
>
>> Hi
>>
>> Aren't there any kernel write errors in your 'dmegs'.
>> LV becomes fragile if the associated devices with cache are having HW
>> issues (disk read/write errors)
>>
>> Zdenek
>>
>
> Is that true even when using a writethrough cache mode?
>
> --
> Danti Gionatan
> Supporto Tecnico
> Assyoma S.r.l. - www.assyoma.it
> email: g.danti@assyoma.it - info@assyoma.it
> GPG public key ID: FF5F32A8
>



-- 
*Ryan Launchbury*

System Engineer
magenta broadcast



magenta.tv

mob: 07939 276 897
support: 020 8050 1920

office: 020 8050 1080

[-- Attachment #2: Type: text/html, Size: 2769 bytes --]

Re: [linux-lvm] Unable to un-cache logical volume when chunk size is over 1MiB

[Zdenek Kabelac]

Dne 22.6.2018 v 20:13 Gionatan Danti napsal(a):
> Il 20-06-2018 12:15 Zdenek Kabelac ha scritto:
>> Hi
>>
>> Aren't there any kernel write errors in your 'dmegs'.
>> LV becomes fragile if the associated devices with cache are having HW
>> issues (disk read/write errors)
>>
>> Zdenek
> 
> Is that true even when using a writethrough cache mode?

With writethrough - all writes are first committed on 'origin' disk - before 
they are ack back to writing apps - so cache can be through away anytime.

When cache will experience write error - it will become invalidate and will
need to be dropped - but this thing is not automated ATM - so admin
works is needed to handle this task.

Regards

Zdenek

Re: [linux-lvm] Unable to un-cache logical volume when chunk size is over 1MiB

[Gionatan Danti]

Il 22-06-2018 22:07 Zdenek Kabelac ha scritto:
> When cache will experience write error - it will become invalidate and 
> will
> need to be dropped - but this thing is not automated ATM - so admin
> works is needed to handle this task.

So, if a writethrough cache experience write errors but the 
administrator is not able to immediately intervene to drop the cache, 
what problem can arise? Stale reads? Slow performance?

What about cache *read* error? Is the read simply redirected to the 
underlying slow/main volume?

Thanks.

-- 
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.danti@assyoma.it - info@assyoma.it
GPG public key ID: FF5F32A8

Re: [linux-lvm] Unable to un-cache logical volume when chunk size is over 1MiB

[Ryan Launchbury]

[-- Attachment #1: Type: text/plain, Size: 1687 bytes --]

Hi Zdenek and Gionatan,

Thanks for your reply's.
Something else of note: on systems which are unable to flush the cache,
data is still being written to the origin LV somehow, because there is
200TB of data in the LV, but the cache is only 1.8TB, so somehow it is
working. However when running any commands to flush the cache, or uncache,
it seems unable to.

What sort of admin work needs to be done/can be done to force the flush and
remove the cache?
I've tried the cleaner policy, however, it doesn't seem to be flushing
anything.

In testing, forcibly removing the cache, via editing the LVM config file
has caused extensive XFS filesystem corruption, even when backing up the
metadata first and restoring after the cache device is missing. Any advice
on how to safely uncache the volume would be massively appreciated.

Please let me know if you need any more logs or data.
Best regards,
Ryan


On Sat, Jun 23, 2018 at 11:09 AM Gionatan Danti <g.danti@assyoma.it> wrote:

> Il 22-06-2018 22:07 Zdenek Kabelac ha scritto:
> > When cache will experience write error - it will become invalidate and
> > will
> > need to be dropped - but this thing is not automated ATM - so admin
> > works is needed to handle this task.
>
> So, if a writethrough cache experience write errors but the
> administrator is not able to immediately intervene to drop the cache,
> what problem can arise? Stale reads? Slow performance?
>
> What about cache *read* error? Is the read simply redirected to the
> underlying slow/main volume?
>
> Thanks.
>
> --
> Danti Gionatan
> Supporto Tecnico
> Assyoma S.r.l. - www.assyoma.it
> email: g.danti@assyoma.it - info@assyoma.it
> GPG public key ID: FF5F32A8
>

[-- Attachment #2: Type: text/html, Size: 2351 bytes --]

Re: [linux-lvm] Unable to un-cache logical volume when chunk size is over 1MiB

[Gionatan Danti]

Il 24-06-2018 21:18 Ryan Launchbury ha scritto:
> In testing, forcibly removing the cache, via editing the LVM config
> file has caused extensive XFS filesystem corruption, even when backing
> up the metadata first and restoring after the cache device is missing.
> Any advice on how to safely uncache the volume would be massively
> appreciated.

It is my understanding that a writethrough cache should *never* have any 
data that are not on the backing volumes already.
In other words, forcibly removing a writethough cache (ie: disconnetting 
the physical cache device) should not cause any harms to 
filesystem/data.

Can you show the output of "dmsetup table"?
Thanks.

-- 
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.danti@assyoma.it - info@assyoma.it
GPG public key ID: FF5F32A8

Re: [linux-lvm] Unable to un-cache logical volume when chunk size is over 1MiB

[Ryan Launchbury]

[-- Attachment #1: Type: text/plain, Size: 1027 bytes --]

Hi Gionatan,

The system with the issue is with  writeback cache mode enabled.

Best regards,

Ryan

On Mon, 25 Jun 2018, 18:19 Gionatan Danti, <g.danti@assyoma.it> wrote:

> Il 24-06-2018 21:18 Ryan Launchbury ha scritto:
> > In testing, forcibly removing the cache, via editing the LVM config
> > file has caused extensive XFS filesystem corruption, even when backing
> > up the metadata first and restoring after the cache device is missing.
> > Any advice on how to safely uncache the volume would be massively
> > appreciated.
>
> It is my understanding that a writethrough cache should *never* have any
> data that are not on the backing volumes already.
> In other words, forcibly removing a writethough cache (ie: disconnetting
> the physical cache device) should not cause any harms to
> filesystem/data.
>
> Can you show the output of "dmsetup table"?
> Thanks.
>
> --
> Danti Gionatan
> Supporto Tecnico
> Assyoma S.r.l. - www.assyoma.it
> email: g.danti@assyoma.it - info@assyoma.it
> GPG public key ID: FF5F32A8
>

[-- Attachment #2: Type: text/html, Size: 1753 bytes --]

Re: [linux-lvm] Unable to un-cache logical volume when chunk size is over 1MiB

[Gionatan Danti]

Il 25-06-2018 19:20 Ryan Launchbury ha scritto:
> Hi Gionatan,
> 
> The system with the issue is with  writeback cache mode enabled.
> 
> Best regards,
> 
> Ryan

Ah, I was under the impression that it was a writethough cache.
Sorry for the noise.

-- 
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.danti@assyoma.it - info@assyoma.it
GPG public key ID: FF5F32A8

Re: [linux-lvm] Unable to un-cache logical volume when chunk size is over 1MiB

[Ryan Launchbury]

[-- Attachment #1: Type: text/plain, Size: 457 bytes --]

Hi all,

Does anyone have any other ideas or potential workarounds for this issue?

Please let me know if you require more info.

Best regards,
Ryan



Gionatan Danti wrote on 25/06/2018 18:40:
> Il 25-06-2018 19:20 Ryan Launchbury ha scritto:
>> Hi Gionatan,
>>
>> The system with the issue is with  writeback cache mode enabled.
>>
>> Best regards,
>>
>> Ryan
>
> Ah, I was under the impression that it was a writethough cache.
> Sorry for the noise.
>


[-- Attachment #2: Type: text/html, Size: 958 bytes --]

Re: [linux-lvm] Unable to un-cache logical volume when chunk size is over 1MiB

[Douglas Paul]

On Wed, Jul 18, 2018 at 03:25:10PM +0100, Ryan Launchbury wrote:
> Does anyone have any other ideas or potential workarounds for this issue?
> 
> Please let me know if you require more info.

I didn't see this in the previous messages, but have you tried temporarily
remounting the filesystems read-only if you can't unmount them?

Also, earlier you mentioned looking at /var/log/dmesg for write errors.  On
my systems, that only contains a snapshot at boot. I have to run dmesg to
see the latest updates. It seems suspicious to have nothing in the dmesg
during 61 days past 10 seconds after system boot ...

-- 
Douglas Paul

Re: [linux-lvm] Unable to un-cache logical volume when chunk size is over 1MiB

[Zdenek Kabelac]

Dne 18. 07. 18 v 16:58 Douglas Paul napsal(a):
> On Wed, Jul 18, 2018 at 03:25:10PM +0100, Ryan Launchbury wrote:
>> Does anyone have any other ideas or potential workarounds for this issue?
>>
>> Please let me know if you require more info.
> 
> I didn't see this in the previous messages, but have you tried temporarily
> remounting the filesystems read-only if you can't unmount them?
> 
> Also, earlier you mentioned looking at /var/log/dmesg for write errors.  On
> my systems, that only contains a snapshot at boot. I have to run dmesg to
> see the latest updates. It seems suspicious to have nothing in the dmesg
> during 61 days past 10 seconds after system boot ...
> 

Hi

So this should help:

lvchange --cachesettings  migration_threshold=16384 vg/cacheLV

the problem was the usage of unexpectedly big chunk sizes,
that are used when cache-pool size is big (>TiB) and chunksizes
are  >= 1MiB.

Upstream has committed patches for automatic protection against too small
migration threshold.


Zdenek