Re: [linux-lvm] inconsistency between thin pool metadata mapped_blocks and lvs output

[John Hamilton <john.l.hamilton@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 5575 bytes --]

So it turns out simply running lvconvert --repair fixed the issue and lvs
is now reporting the correct utilization.

On Fri, May 11, 2018 at 12:09 PM John Hamilton <john.l.hamilton@gmail.com>
wrote:

> Thanks for the response.
>
> >Is this everything?
>
> Yes, that is everything in the metadata xml dump.  I just removed all of
> the *_mapping entries for brevity.  For the lvs output I removed other
> logical volumes that aren't related to this pool.
>
> >Is this a pool used by docker, which does not (did not) use LVM to
> manage thin-volumes?
>
> It's not docker, but it is an application called serviced that uses
> docker's library for managing the volumes
>
> >LVM just queries DM, and displays whatever that provides
>
> Yeah, it looks like dmsetup status output matches lvs:
>
> myvg-my--pool: 0 5242880000 thin-pool 70 207941/4145152 29018611/40960000 - rw discard_passdown queue_if_no_space -
> myvg-my--pool_tdata: 0 4194304000 <(419)%20430-4000> linear
> myvg-my--pool_tdata: 4194304000 <(419)%20430-4000> 1048576000 linear
> myvg-my--pool_tmeta: 0 33161216 linear
>
> >What is kernel/lvm version?
>
> # uname -r
> 3.10.0-693.21.1.el7.x86_64
>
> # lvm version
> LVM version:     2.02.171(2)-RHEL7 (2017-05-03)
> Library version: 1.02.140-RHEL7 (2017-05-03)
> Driver version:  4.35.0
> Configuration:   ./configure --build=x86_64-redhat-linux-gnu --host=x86_64-redhat-linux-gnu --program-prefix= --disable-dependency-tracking --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64 --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/var/lib --mandir=/usr/share/man --infodir=/usr/share/info --with-default-dm-run-dir=/run --with-default-run-dir=/run/lvm --with-default-pid-dir=/run --with-default-locking-dir=/run/lock/lvm --with-usrlibdir=/usr/lib64 --enable-lvm1_fallback --enable-fsadm --with-pool=internal --enable-write_install --with-user= --with-group= --with-device-uid=0 --with-device-gid=6 --with-device-mode=0660 --enable-pkgconfig --enable-applib --enable-cmdlib --enable-dmeventd --enable-blkid_wiping --enable-python2-bindings --with-cluster=internal --with-clvmd=corosync --enable-cmirrord --with-udevdir=/usr/lib/udev/rules.d --enable-udev_sync --with-thin=internal --enable-lvmetad --with-cache=internal --enable-lvmpolld --enable-lvmlockd-dlm --enable-lvmlockd-sanlock --enable-dmfilemapd
>
> >Is thin_check_executable configured in lvm.conf?
>
> Yes
>
> I also just found out that they apparently ran thin_check recently and got
> a message about a corrupt superblock, but didn't repair it.  They were
> still able to re-activate the pool though. We'll run a repair as soon as we
> get a chance and see if that fixes it.
>
> Thanks,
>
> John
>
> On Fri, May 11, 2018 at 3:54 AM Marian Csontos <mcsontos@redhat.com>
> wrote:
>
>> On 05/11/2018 10:21 AM, Joe Thornber wrote:
>> > On Thu, May 10, 2018 at 07:30:09PM +0000, John Hamilton wrote:
>> >> I saw something today that I don't understand and I'm hoping somebody
>> can
>> >> help.  We had a ~2.5TB thin pool that was showing 69% data utilization
>> in
>> >> lvs:
>> >>
>> >> # lvs -a
>> >>    LV                    VG       Attr       LSize  Pool Origin Data%
>> >> Meta%  Move Log Cpy%Sync Convert
>> >>    my-pool         myvg twi-aotz--  2.44t             69.04  4.90
>> >>    [my-pool_tdata] myvg Twi-ao----  2.44t
>> >>    [my-pool_tmeta] myvg ewi-ao---- 15.81g
>>
>> Is this everything? Is this a pool used by docker, which does not (did
>> not) use LVM to manage thin-volumes?
>>
>> >> However, when I dump the thin pool metadata and look at the
>> mapped_blocks
>> >> for the 2 devices in the pool, I can only account for about 950GB.
>> Here is
>> >> the superblock and device entries from the metadata xml.  There are no
>> >> other devices listed in the metadata:
>> >>
>> >> <superblock uuid="" time="34" transaction="68" flags="0" version="2"
>> >> data_block_size="128" nr_data_blocks="0">
>> >>    <device dev_id="1" mapped_blocks="258767" transaction="0"
>> >> creation_time="0" snap_time="14">
>> >>    <device dev_id="8" mapped_blocks="15616093" transaction="27"
>> >> creation_time="15" snap_time="34">
>> >>
>> >> That first device looks like it has about 16GB allocated to it and the
>> >> second device about 950GB.  So, I would expect lvs to show somewhere
>> >> between 950G-966G Is something wrong, or am I misunderstanding how to
>> read
>> >> the metadata dump?  Where is the other 700 or so GB that lvs is showing
>> >> used?
>> >
>> > The non zero snap_time suggests that you're using snapshots.  I which
>> case it
>> > could just be there is common data shared between volumes that is
>> getting counted
>> > more than once.
>> >
>> > You can confirm this using the thin_ls tool and specifying a format
>> line that
>> > includes EXCLUSIVE_BLOCKS, or SHARED_BLOCKS.  Lvm doesn't take shared
>> blocks into
>> > account because it has to scan all the metadata to calculate what's
>> shared.
>>
>> LVM just queries DM, and displays whatever that provides. You could see
>> that in `dmsetup status` output, there are two pairs of '/' separated
>> entries - first is metadata usage (USED_BLOCKS/ALL_BLOCKS), second data
>> usage (USED_CHUNKS/ALL_CHUNKS).
>>
>> So the error lies somewhere between dmsetup and kernel.
>>
>> What is kernel/lvm version?
>> Is thin_check_executable configured in lvm.conf?
>>
>> -- Martian
>>
>

[-- Attachment #2: Type: text/html, Size: 10142 bytes --]