[linux-lvm] cache_check --clear-needs-check-flag does not clear needs_check flag?

[linux-lvm] cache_check --clear-needs-check-flag does not clear needs_check flag?

[Dennis Schridde]

[-- Attachment #1.1: Type: text/plain, Size: 4362 bytes --]

Hello!

A cached logical volume of mine cannot be activated anymore:

$ sudo vgchange -ay
   device-mapper: reload ioctl on   (253:6) failed: Invalid argument
   0 logical volume(s) in volume group "vg_ernie" now active


dmesg logs:

device-mapper: cache: 253:6: unable to switch cache to write mode until
repaired.
device-mapper: cache: 253:6: switching cache to read-only mode
device-mapper: table: 253:6: cache: Unable to get write access to metadata,
please check/repair metadata.
device-mapper: ioctl: error adding target to table


The code in question seems to be: https://github.com/torvalds/linux/blob/v5.8/
drivers/md/dm-cache-target.c#L957-L964


Hence I set out to check the cache and, if it is clean, clear the needs_check
flag:

$ sudo lvchange -ay vg_ernie/lv_cache
Do you want to activate component LV in read-only mode? [y/n]: y
   Allowing activation of component LV.

$ sudo cache_check /dev/vg_ernie/lv_cache
examining superblock
examining mapping array
examining hint array
examining discard bitset

$ sudo cache_check --clear-needs-check-flag /dev/vg_ernie/lv_cache
examining superblock
examining mapping array
examining hint array
examining discard bitset

$ sudo lvchange -an vg_ernie/lv_cache


But the problem persists:

$ sudo vgchange -ay
   device-mapper: reload ioctl on   (253:6) failed: Invalid argument
   0 logical volume(s) in volume group "vg_ernie" now active


I tried again in read/write mode, in case that would make a difference /
silently fail:

$ sudo lvchange -ay vg_ernie/lv_cache
Do you want to activate component LV in read-only mode? [y/n]: n

$ sudo cache_check --clear-needs-check-flag /dev/vg_ernie/lv_cache
examining superblock
examining mapping array
examining hint array
examining discard bitset

$ sudo lvchange -an vg_ernie/lv_cache


With the same results:

$ sudo vgchange -ay
   device-mapper: reload ioctl on   (253:6) failed: Invalid argument
   0 logical volume(s) in volume group "vg_ernie" now active


A bit puzzling is that the status of the needs_check flag appears to be
"unknown":

$ sudo lvs -a -o +lv_check_needed
   LV                        VG          Attr          LSize    Pool
Origin                  Data%   Meta%   Move Log Cpy%Sync Convert CheckNeeded
   [lv_cache]            vg_ernie CRi-a-C--- 232.88g
unknown
   lv_system             vg_ernie Cwi---C---   <1.82t [lv_cache]
[lv_system_corig]
unknown
   [lv_system_corig] vg_ernie owi---C---   <1.82t
unknown


The live system I am running these commands from is a Fedora 33:

$ uname -a
Linux localhost-live 5.8.15-301.fc33.x86_64 #1 SMP Thu Oct 15 16:58:06 UTC
2020 x86_64 x86_64 x86_64 GNU/Linux

$ sudo lvm version
   LVM version:       2.03.10(2) (2020-08-09)
   Library version: 1.02.173 (2020-08-09)
   Driver version:   4.42.0
   Configuration:    ./configure --build=x86_64-redhat-linux-gnu --
host=x86_64-redhat-linux-gnu --program-prefix= --disable-dependency-tracking
--prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --
sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/
lib64 --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/var/lib
--mandir=/usr/share/man --infodir=/usr/share/info --with-default-dm-run-dir=/
run --with-default-run-dir=/run/lvm --with-default-pid-dir=/run --with-
default-locking-dir=/run/lock/lvm --with-usrlibdir=/usr/lib64 --enable-fsadm
--enable-write_install --with-user= --with-group= --with-device-uid=0 --with-
device-gid=6 --with-device-mode=0660 --enable-pkgconfig --enable-cmdlib --
enable-dmeventd --enable-blkid_wiping --with-cluster=internal --enable-
cmirrord --with-udevdir=/usr/lib/udev/rules.d --enable-udev_sync --with-
thin=internal --with-cache=internal --enable-lvmpolld --enable-lvmlockd-dlm --
enable-lvmlockd-dlmcontrol --enable-lvmlockd-sanlock --enable-dbus-service --
enable-notify-dbus --enable-dmfilemapd --with-writecache=internal --with-
vdo=internal --with-vdo-format=/usr/bin/vdoformat --disable-silent-rules


Is it possible that `cache_check --clear-needs-check-flag does not clear
needs_check flag` does not actually clear the needs_check flag in my case?


Any help to get the system back online (without data loss, if possible) is
appreciated!

Best regards,
Dennis

[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

[-- Attachment #2: Type: text/plain, Size: 197 bytes --]

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] cache_check --clear-needs-check-flag does not clear needs_check flag?

[Dennis Schridde]

[-- Attachment #1.1: Type: text/plain, Size: 881 bytes --]

On Sonntag, 6. Dezember 2020 22:01:37 CET Dennis Schridde wrote:
> [...]
> $ sudo lvchange -ay vg_ernie/lv_cache
> Do you want to activate component LV in read-only mode? [y/n]: y
>    Allowing activation of component LV.
> [...]
> Is it possible that `cache_check --clear-needs-check-flag does not clear
> needs_check flag` does not actually clear the needs_check flag in my case?

I suspect that this has something to do with the read only status of the LV...

What if the superblock gets changed, but is then not written back to the LV,
because it is read only, and failure to write it back was not treated as an
error?  I dd-ed lv_cache into a file, ran cache_check --clear-needs-check-flag
on it, and then tried to dd it back to the LV.  Sadly that results in a
permission denied error.  I was not able to figure out how to force writing it
back.

--Dennis

[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

[-- Attachment #2: Type: text/plain, Size: 197 bytes --]

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] cache_check --clear-needs-check-flag does not clear needs_check flag?

[Zdenek Kabelac]

Dne 06. 12. 20 v 22:01 Dennis Schridde napsal(a):
> Hello!
> 
> A cached logical volume of mine cannot be activated anymore:
> 
> $ sudo vgchange -ay
>     device-mapper: reload ioctl on   (253:6) failed: Invalid argument
>     0 logical volume(s) in volume group "vg_ernie" now active
> 
> 
> dmesg logs:
> 
> device-mapper: cache: 253:6: unable to switch cache to write mode until
> repaired.
> device-mapper: cache: 253:6: switching cache to read-only mode
> device-mapper: table: 253:6: cache: Unable to get write access to metadata,
> please check/repair metadata.
> device-mapper: ioctl: error adding target to table
> 
> 
> The code in question seems to be: https://github.com/torvalds/linux/blob/v5.8/
> drivers/md/dm-cache-target.c#L957-L964
> 
> 
> Hence I set out to check the cache and, if it is clean, clear the needs_check
> flag:
> 
> $ sudo lvchange -ay vg_ernie/lv_cache
> Do you want to activate component LV in read-only mode? [y/n]: y
>     Allowing activation of component LV.

As said - by component activation you will get 'read-only' volume
thus you cannot do 'in-place' changes this way.


> 
> A bit puzzling is that the status of the needs_check flag appears to be
> "unknown":
> 
> $ sudo lvs -a -o +lv_check_needed
>     LV                        VG          Attr          LSize    Pool
> Origin                  Data%   Meta%   Move Log Cpy%Sync Convert CheckNeeded
>     [lv_cache]            vg_ernie CRi-a-C--- 232.88g
> unknown
>     lv_system             vg_ernie Cwi---C---   <1.82t [lv_cache]
> [lv_system_corig]
> unknown
>     [lv_system_corig] vg_ernie owi---C---   <1.82t
> unknown
> 
> 
> The live system I am running these commands from is a Fedora 33:
> 
> $ uname -a
> Linux localhost-live 5.8.15-301.fc33.x86_64 #1 SMP Thu Oct 15 16:58:06 UTC
> 2020 x86_64 x86_64 x86_64 GNU/Linux
> 
> $ sudo lvm version
>     LVM version:       2.03.10(2) (2020-08-09)
>     Library version: 1.02.173 (2020-08-09)
>     Driver version:   4.42.0

It seems you are using  'cvol' instead of 'cpool' solution which has been 
evolving and you will need a newer lvm2 version.

If you want to using caching with the version of lvm2 you have - you will
need to use cpools  (which are a bit faster anyway).

Regards

Zdenek

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] cache_check --clear-needs-check-flag does not clear needs_check flag?

[Dennis Schridde]

[-- Attachment #1.1: Type: text/plain, Size: 1359 bytes --]

Hello Zdenek!

Thank you for your reply!

On Mittwoch, 9. Dezember 2020 12:05:13 CET Zdenek Kabelac wrote:
> Dne 06. 12. 20 v 22:01 Dennis Schridde napsal(a):
> > The live system I am running these commands from is a Fedora 33:
> >
> > $ uname -a
> > Linux localhost-live 5.8.15-301.fc33.x86_64 #1 SMP Thu Oct 15 16:58:06 UTC
> > 2020 x86_64 x86_64 x86_64 GNU/Linux
> >
> > $ sudo lvm version
> >
> >     LVM version:       2.03.10(2) (2020-08-09)
> >     Library version: 1.02.173 (2020-08-09)
> >     Driver version:   4.42.0
>
> It seems you are using  'cvol' instead of 'cpool' solution which has been
> evolving and you will need a newer lvm2 version.
>
> If you want to using caching with the version of lvm2 you have - you will
> need to use cpools  (which are a bit faster anyway).

2.03.10 is the latest released version of lvm2.  So I would have to use the
Git main branch instead?

A bit confusing is that the lvmcache(7) manpage of lvm2 2.03.10 suggests to
create a cachevol as the first option:
$ lvconvert --type cache --cachevol fast vg/main

Should this be adjusted, if cachevols are not yet fully supported in 2.03.10?

How would you proceed to repair this cache?  I can recreate it as a cachepool,
but first I need to gain back access to the data, i.e. get into a state where
I can --uncache.

--Dennis

[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

[-- Attachment #2: Type: text/plain, Size: 197 bytes --]

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] cache_check --clear-needs-check-flag does not clear needs_check flag?

[David Teigland]

On Wed, Dec 09, 2020 at 08:06:41PM +0100, Dennis Schridde wrote:
> How would you proceed to repair this cache?  I can recreate it as a cachepool,
> but first I need to gain back access to the data, i.e. get into a state where
> I can --uncache.

I know that cachevols do not yet work with lvconvert --repair, which uses
cache_repair, but this must be another case where cache_repair is still
needed.  If this is writethough mode you should be able to simply drop the
cache with lvconvert --splitcache and then recreate it.
Dave

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] cache_check --clear-needs-check-flag does not clear needs_check flag?

[Dennis Schridde]

[-- Attachment #1.1: Type: text/plain, Size: 956 bytes --]

On Mittwoch, 9. Dezember 2020 20:30:51 CET David Teigland wrote:
> On Wed, Dec 09, 2020 at 08:06:41PM +0100, Dennis Schridde wrote:
> > How would you proceed to repair this cache?  I can recreate it as a
> > cachepool, but first I need to gain back access to the data, i.e. get
> > into a state where I can --uncache.
>
> I know that cachevols do not yet work with lvconvert --repair, which uses
> cache_repair, but this must be another case where cache_repair is still
> needed.  If this is writethough mode you should be able to simply drop the
> cache with lvconvert --splitcache and then recreate it.

Sadly the cache is in writeback mode.

Is my understanding correct that the cache itself is fine?  It is just the
inability to clear the needs_check flag that is preventing me from using the
device?  Or did I miss to run some checks that could reveal a problem / did I
misinterpret the output of cache_check on lv_cache?

--Dennis

[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

[-- Attachment #2: Type: text/plain, Size: 197 bytes --]

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] cache_check --clear-needs-check-flag does not clear needs_check flag?

[Dennis Schridde]

[-- Attachment #1.1: Type: text/plain, Size: 748 bytes --]

On Mittwoch, 9. Dezember 2020 20:30:51 CET David Teigland wrote:
> On Wed, Dec 09, 2020 at 08:06:41PM +0100, Dennis Schridde wrote:
> > How would you proceed to repair this cache?  I can recreate it as a
> > cachepool, but first I need to gain back access to the data, i.e. get
> > into a state where I can --uncache.
>
> I know that cachevols do not yet work with lvconvert --repair, which uses
> cache_repair, but this must be another case where cache_repair is still
> needed.  If this is writethough mode you should be able to simply drop the
> cache with lvconvert --splitcache and then recreate it.

Would this be safe if cache_dump showed all blocks are clean (dirty=false),
even if the cache was in writeback mode?

--Dennis

[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

[-- Attachment #2: Type: text/plain, Size: 197 bytes --]

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] cache_check --clear-needs-check-flag does not clear needs_check flag?

[David Teigland]

On Wed, Dec 09, 2020 at 08:57:07PM +0100, Dennis Schridde wrote:
> On Mittwoch, 9. Dezember 2020 20:30:51 CET David Teigland wrote:
> > On Wed, Dec 09, 2020 at 08:06:41PM +0100, Dennis Schridde wrote:
> > > How would you proceed to repair this cache?  I can recreate it as a
> > > cachepool, but first I need to gain back access to the data, i.e. get
> > > into a state where I can --uncache.
> >
> > I know that cachevols do not yet work with lvconvert --repair, which uses
> > cache_repair, but this must be another case where cache_repair is still
> > needed.  If this is writethough mode you should be able to simply drop the
> > cache with lvconvert --splitcache and then recreate it.
> 
> Sadly the cache is in writeback mode.
> 
> Is my understanding correct that the cache itself is fine?  It is just the
> inability to clear the needs_check flag that is preventing me from using the
> device?  

I suspect the cache itself is fine and we're just missing a way to clear
that flag in the cache metadata.  I'm looking for a workaround for that
until we can sort out a complete fix.

Dave

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] cache_check --clear-needs-check-flag does not clear needs_check flag?

[David Teigland]

> I suspect the cache itself is fine and we're just missing a way to clear
> that flag in the cache metadata.  I'm looking for a workaround for that
> until we can sort out a complete fix.

The proper fix is here
https://sourceware.org/git/?p=lvm2.git;a=commit;h=9fe7aba251ff24c14277dfdf9be2d861a7699230

(A test for that using actual flagged metadata is still needed.)

Dave

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/