linux-lvm.redhat.com archive mirror
 help / color / mirror / Atom feed
From: Zdenek Kabelac <zkabelac@redhat.com>
To: LVM general discussion and development <linux-lvm@redhat.com>,
	Ryan Launchbury <ryan@magenta.tv>
Subject: Re: [linux-lvm] Unable to un-cache logical volume when chunk size is over 1MiB
Date: Wed, 20 Jun 2018 12:15:55 +0200	[thread overview]
Message-ID: <6e53a6f7-9896-2905-92ad-6b9c36f565ab@redhat.com> (raw)
In-Reply-To: <66e4dbc1-ca00-3abf-5100-d19f7439a281@magenta.tv>

Dne 20.6.2018 v 11:18 Ryan Launchbury napsal(a):
> Hello,
> 
> I'm having a problem uncaching logical volumes when the cache data chunck size 
> is over 1MiB.
> The process I'm using to uncache is: lvconvert --uncache vg/lv
> 
> 
> The issue occurs across multiple systems with different hardware and different 
> versions of LVM.
> 
> Steps to reproduce:
> 
>  1. Create origin VG & LV
>  2. Add cache device over 1TB to the origin VG
>  3. Create the cache data lv:
>     lvcreate -n cachedata -L 1770GB cached_vg /dev/nvme0n1
>  4. Create the cache metadata lv:
>     lvcreate -n cachemeta -L 1770MB cached_vg /dev/nvme0n1
>  5. Convert to a cache pool:
>     lvconvert --type cache-pool --cachemode writethrough --poolmetadata
>     cached_vg/cachemeta cached_vg/cachedata
>  6. Enable caching on the origin LVM:
>     lvconvert --type cache --cachepool cached_vg/cachedata cached_vg/filestore01
>  7. Write some data to the main LV so as the cache device is used:
>     dd if=/dev/zero of=/mnt/filestore01/test.dat bs=1M count=10000
>  8. Check the cache stats:
>     lvs -a -o +cache_total_blocks,cache_used_blocks,cache_dirty_blocks
>  9. Repeating step 8 over time will show that the dirty blocks are not being
>     written back at all
> 10. Try to uncache the device:
>     lvconvert --uncache cached_vg/filestore01
> 11. You will get a repeating message. This will loop indefinitely and not
>     decrease or complete:
>     Flushing x blocks for cache cached_vg/filestore01.
> 
> After testing multiple times, the issue seems to be tied to the chunk size 
> selected in step 5. The LVM man page mentions that the chunk must be a 
> multiple of 32KiB, however the next chunk size automatically assigned over 
> 1MiB is usually 1.03MiB. With a chunk size of 1.03MiB or higher, the cache is 
> not able to flush. Creating a cache device with a chunk size of 1MiB or less, 
> the cache is flushable.
> 
> Now knowing how to avoid the issue, I just need to be able to safely un-cache 
> systems with do have a cache that will not flush.
> 
> Details:
> 
> Version info from lvm version:
> 
> LVM version:     2.02.171(2)-RHEL7 (2017-05-03)
>    Library version: 1.02.140-RHEL7 (2017-05-03)
>    Driver version:  4.35.0

What is the kernel version and Linux distro in use ?

> 
> System info:
> System 1,2,3:
> - Dell R730XD server
> - 12x disk in RAID 6 to onboard PERC/Megaraid controller
> 
> System 4:
> -Dell R630 server
> -60x Disk (6 luns) in RAID 6 to PCI megaraid controller
> 
> The systems are currently in production, so it's quite hard for me to change 
> the configuration to enable logging.
> 
> Any assistance would be much appreciated! If any more info is needed please 
> let me know.

Hi

Aren't there any kernel write errors in your 'dmegs'.
LV becomes fragile if the associated devices with cache are having HW issues 
(disk read/write errors)

Zdenek

  reply	other threads:[~2018-06-20 10:15 UTC|newest]

Thread overview: 14+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-06-20  9:18 [linux-lvm] Unable to un-cache logical volume when chunk size is over 1MiB Ryan Launchbury
2018-06-20 10:15 ` Zdenek Kabelac [this message]
2018-06-20 11:10   ` Ryan Launchbury
2018-06-22 18:13   ` Gionatan Danti
2018-06-22 19:22     ` Ryan Launchbury
2018-06-22 20:07     ` Zdenek Kabelac
2018-06-23 10:09       ` Gionatan Danti
2018-06-24 19:18         ` Ryan Launchbury
2018-06-25 17:19           ` Gionatan Danti
2018-06-25 17:20             ` Ryan Launchbury
2018-06-25 17:40               ` Gionatan Danti
2018-07-18 14:25                 ` Ryan Launchbury
2018-07-18 14:58                   ` Douglas Paul
2019-01-21 20:19                     ` Zdenek Kabelac

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=6e53a6f7-9896-2905-92ad-6b9c36f565ab@redhat.com \
    --to=zkabelac@redhat.com \
    --cc=linux-lvm@redhat.com \
    --cc=ryan@magenta.tv \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).