From: Ryan Launchbury <ryan@magenta.tv>
To: Zdenek Kabelac <zkabelac@redhat.com>
Cc: LVM general discussion and development <linux-lvm@redhat.com>
Subject: Re: [linux-lvm] Unable to un-cache logical volume when chunk size is over 1MiB
Date: Wed, 20 Jun 2018 12:10:02 +0100 [thread overview]
Message-ID: <c454f732-5832-fe5e-60ed-96695da6927d@magenta.tv> (raw)
In-Reply-To: <6e53a6f7-9896-2905-92ad-6b9c36f565ab@redhat.com>
Hi Zdenek,
Kernel is: Linux 3.10.0-693.21.1.el7.x86_64
Distro is: Centos 7 - Linux release 7.4.1708
Zdenek Kabelac wrote on 20/06/2018 11:15:
> Dne 20.6.2018 v 11:18 Ryan Launchbury napsal(a):
>> Hello,
>>
>> I'm having a problem uncaching logical volumes when the cache data
>> chunck size is over 1MiB.
>> The process I'm using to uncache is: lvconvert --uncache vg/lv
>>
>>
>> The issue occurs across multiple systems with different hardware and
>> different versions of LVM.
>>
>> Steps to reproduce:
>>
>> 1. Create origin VG & LV
>> 2. Add cache device over 1TB to the origin VG
>> 3. Create the cache data lv:
>> lvcreate -n cachedata -L 1770GB cached_vg /dev/nvme0n1
>> 4. Create the cache metadata lv:
>> lvcreate -n cachemeta -L 1770MB cached_vg /dev/nvme0n1
>> 5. Convert to a cache pool:
>> lvconvert --type cache-pool --cachemode writethrough --poolmetadata
>> cached_vg/cachemeta cached_vg/cachedata
>> 6. Enable caching on the origin LVM:
>> lvconvert --type cache --cachepool cached_vg/cachedata
>> cached_vg/filestore01
>> 7. Write some data to the main LV so as the cache device is used:
>> dd if=/dev/zero of=/mnt/filestore01/test.dat bs=1M count=10000
>> 8. Check the cache stats:
>> lvs -a -o +cache_total_blocks,cache_used_blocks,cache_dirty_blocks
>> 9. Repeating step 8 over time will show that the dirty blocks are
>> not being
>> written back at all
>> 10. Try to uncache the device:
>> lvconvert --uncache cached_vg/filestore01
>> 11. You will get a repeating message. This will loop indefinitely and
>> not
>> decrease or complete:
>> Flushing x blocks for cache cached_vg/filestore01.
>>
>> After testing multiple times, the issue seems to be tied to the chunk
>> size selected in step 5. The LVM man page mentions that the chunk
>> must be a multiple of 32KiB, however the next chunk size
>> automatically assigned over 1MiB is usually 1.03MiB. With a chunk
>> size of 1.03MiB or higher, the cache is not able to flush. Creating a
>> cache device with a chunk size of 1MiB or less, the cache is flushable.
>>
>> Now knowing how to avoid the issue, I just need to be able to safely
>> un-cache systems with do have a cache that will not flush.
>>
>> Details:
>>
>> Version info from lvm version:
>>
>> LVM version: 2.02.171(2)-RHEL7 (2017-05-03)
>> Library version: 1.02.140-RHEL7 (2017-05-03)
>> Driver version: 4.35.0
>
> What is the kernel version and Linux distro in use ?
>
>>
>> System info:
>> System 1,2,3:
>> - Dell R730XD server
>> - 12x disk in RAID 6 to onboard PERC/Megaraid controller
>>
>> System 4:
>> -Dell R630 server
>> -60x Disk (6 luns) in RAID 6 to PCI megaraid controller
>>
>> The systems are currently in production, so it's quite hard for me to
>> change the configuration to enable logging.
>>
>> Any assistance would be much appreciated! If any more info is needed
>> please let me know.
>
> Hi
>
> Aren't there any kernel write errors in your 'dmegs'.
> LV becomes fragile if the associated devices with cache are having HW
> issues (disk read/write errors)
>
> Zdenek
Nope, no write errors in /var/log/dmesg. The last log entry was at
10.871493 and the system has been on for 61 days.
Best regards,
Ryan
next prev parent reply other threads:[~2018-06-20 11:10 UTC|newest]
Thread overview: 14+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-06-20 9:18 [linux-lvm] Unable to un-cache logical volume when chunk size is over 1MiB Ryan Launchbury
2018-06-20 10:15 ` Zdenek Kabelac
2018-06-20 11:10 ` Ryan Launchbury [this message]
2018-06-22 18:13 ` Gionatan Danti
2018-06-22 19:22 ` Ryan Launchbury
2018-06-22 20:07 ` Zdenek Kabelac
2018-06-23 10:09 ` Gionatan Danti
2018-06-24 19:18 ` Ryan Launchbury
2018-06-25 17:19 ` Gionatan Danti
2018-06-25 17:20 ` Ryan Launchbury
2018-06-25 17:40 ` Gionatan Danti
2018-07-18 14:25 ` Ryan Launchbury
2018-07-18 14:58 ` Douglas Paul
2019-01-21 20:19 ` Zdenek Kabelac
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=c454f732-5832-fe5e-60ed-96695da6927d@magenta.tv \
--to=ryan@magenta.tv \
--cc=linux-lvm@redhat.com \
--cc=zkabelac@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).