[linux-lvm] Removing lvmcache fails

[linux-lvm] Removing lvmcache fails

[Roy Sigurd Karlsbakk]

Hi all

I have an ssd hooked up to my system to do some caching of the raid-6 array. Now, that SSD isn't very new and has started developing some smart errors, so I thought I'd just remove it and find a nice recycle bin for it. However, this fails.

This is Debian Buster (latest) with kernel 5.4 and lvm 2.03.02(2)

Any idea how to get rid of this?

# lvconvert --uncache data/data
  Unknown feature in status: 8 2488/262144 128 819192/819200 57263288 35277148 19955332 9916661 0 1 1 3 metadata2 writethrough no_discard_passdown 2 migration_threshold 2048 smq 0 rw -
  Flushing 1 blocks for cache data/data.
  Unknown feature in status: 8 2484/262144 128 819192/819200 56391295 33200598 19684834 6317925 0 0 1 3 metadata2 writethrough no_discard_passdown 2 migration_threshold 2048 cleaner 0 rw -
  Flushing 1 blocks for cache data/data.
  Unknown feature in status: 8 2484/262144 128 819192/819200 56391295 33200598 19684835 6317925 0 0 1 3 metadata2 writethrough no_discard_passdown 2 migration_threshold 2048 cleaner 0 rw -
  Flushing 1 blocks for cache data/data.
  Unknown feature in status: 8 2484/262144 128 819192/819200 56391295 33200598 19684835 6317925 0 0 1 3 metadata2 writethrough no_discard_passdown 2 migration_threshold 2048 cleaner 0 rw -
  Flushing 1 blocks for cache data/data.
  Unknown feature in status: 8 2484/262144 128 819192/819200 56391295 33200598 19684835 6317925 0 0 1 3 metadata2 writethrough no_discard_passdown 2 migration_threshold 
(and so on)

# lvs -o+cache_mode data/data
  Unknown feature in status: 8 2484/262144 128 819195/819200 57263401 35277148 19955449 9916671 0 3 1 3 metadata2 writethrough no_discard_passdown 2 migration_threshold 2048 smq 0 rw -
  LV   VG   Attr       LSize  Pool     Origin       Data%  Meta%  Move Log Cpy%Sync Convert CacheMode
  data data Cwi-aoC--- 13,67t [_cache] [data_corig] 99,99  0,95            0,01             writethrough

Vennlig hilsen

roy
-- 
Roy Sigurd Karlsbakk
(+47) 98013356
http://blogg.karlsbakk.net/
GPG Public key: http://karlsbakk.net/roysigurdkarlsbakk.pubkey.txt
--
Hið góða skaltu í stein höggva, hið illa í snjó rita.

Re: [linux-lvm] Removing lvmcache fails

[Zdenek Kabelac]

Dne 21. 09. 20 v 10:51 Roy Sigurd Karlsbakk napsal(a):
> Hi all
> 
> I have an ssd hooked up to my system to do some caching of the raid-6 array. Now, that SSD isn't very new and has started developing some smart errors, so I thought I'd just remove it and find a nice recycle bin for it. However, this fails.

Hmm - cache isn't yet very easy to use once you start to have faulty devices 
in your device stack

> 
> This is Debian Buster (latest) with kernel 5.4 and lvm 2.03.02(2)
> 
> Any idea how to get rid of this?
> 
> # lvconvert --uncache data/data
>    Unknown feature in status: 8 2488/262144 128 819192/819200 57263288 35277148 19955332 9916661 0 1 1 3 metadata2 writethrough no_discard_passdown 2 migration_threshold 2048 smq 0 rw -


However since you have 'writethrough' cache mode - it should be possible to 
use  'lvconvert --uncache --force'

But I'm somehow confused how you can have any dirty blocks in this case ??

lvm2 2.03.02 version was somewhat experimental release - so I've recommend 
something newer  - which will also properly parse newer status output from
newer kernel dm-cache module.

Regards

Zdenek

Re: [linux-lvm] Removing lvmcache fails

[Roy Sigurd Karlsbakk]

> Dne 21. 09. 20 v 10:51 Roy Sigurd Karlsbakk napsal(a):
>> Hi all
>> 
>> I have an ssd hooked up to my system to do some caching of the raid-6 array.
>> Now, that SSD isn't very new and has started developing some smart errors, so I
>> thought I'd just remove it and find a nice recycle bin for it. However, this
>> fails.
> 
> Hmm - cache isn't yet very easy to use once you start to have faulty devices
> in your device stack

Well, that's why I want to replace it ;)

>> This is Debian Buster (latest) with kernel 5.4 and lvm 2.03.02(2)
>> 
>> Any idea how to get rid of this?
>> 
>> # lvconvert --uncache data/data
>>    Unknown feature in status: 8 2488/262144 128 819192/819200 57263288 35277148
>>    19955332 9916661 0 1 1 3 metadata2 writethrough no_discard_passdown 2
>>    migration_threshold 2048 smq 0 rw -
> 
> 
> However since you have 'writethrough' cache mode - it should be possible to
> use  'lvconvert --uncache --force'

# lvconvert --uncache --force data/data
  Unknown feature in status: 8 2484/262144 128 819198/819200 57470386 35323243 19987362 9940326 0 0 1 3 metadata2 writethrough no_discard_passdown 2 migration_threshold 2048 smq 0 rw -
  Flushing 1 blocks for cache data/data.
  Unknown feature in status: 8 2484/262144 128 819198/819200 57439248 35307947 19981793 9938992 0 0 1 3 metadata2 writethrough no_discard_passdown 2 migration_threshold 2048 cleaner 0 rw -
  Flushing 1 blocks for cache data/data.
  Unknown feature in status: 8 2484/262144 128 819198/819200 57439248 35307947 19981793 9938992 0 0 1 3 metadata2 writethrough no_discard_passdown 2 migration_threshold 2048 cleaner 0 rw -
(et cetera et cetera ad infinitum)

> But I'm somehow confused how you can have any dirty blocks in this case ??

So am I

> lvm2 2.03.02 version was somewhat experimental release - so I've recommend
> something newer  - which will also properly parse newer status output from
> newer kernel dm-cache module.

Well, that's weird - debian is usually rather conservative on its packages in a stable release. Are you sure?

I also updated to a backported kernel 5.7 without any change.

Vennlig hilsen

roy
-- 
Roy Sigurd Karlsbakk
(+47) 98013356
http://blogg.karlsbakk.net/
GPG Public key: http://karlsbakk.net/roysigurdkarlsbakk.pubkey.txt
--
Hið góða skaltu í stein höggva, hið illa í snjó rita.

Re: [linux-lvm] Removing lvmcache fails

[Zdenek Kabelac]

Dne 21. 09. 20 v 12:01 Roy Sigurd Karlsbakk napsal(a):
>> Dne 21. 09. 20 v 10:51 Roy Sigurd Karlsbakk napsal(a):
>>> Hi all
>>>

>> However since you have 'writethrough' cache mode - it should be possible to
>> use  'lvconvert --uncache --force'
> 
> # lvconvert --uncache --force data/data
>    Unknown feature in status: 8 2484/262144 128 819198/819200 57470386 35323243 19987362 9940326 0 0 1 3 metadata2 writethrough no_discard_passdown 2 migration_threshold 2048 smq 0 rw -
>    Flushing 1 blocks for cache data/data.
>    Unknown feature in status: 8 2484/262144 128 819198/819200 57439248 35307947 19981793 9938992 0 0 1 3 metadata2 writethrough no_discard_passdown 2 migration_threshold 2048 cleaner 0 rw -
>    Flushing 1 blocks for cache data/data.
>    Unknown feature in status: 8 2484/262144 128 819198/819200 57439248 35307947 19981793 9938992 0 0 1 3 metadata2 writethrough no_discard_passdown 2 migration_threshold 2048 cleaner 0 rw -
> (et cetera et cetera ad infinitum)
> 
>> But I'm somehow confused how you can have any dirty blocks in this case ??
> 
> So am I
> 
>> lvm2 2.03.02 version was somewhat experimental release - so I've recommend
>> something newer  - which will also properly parse newer status output from
>> newer kernel dm-cache module.
> 
> Well, that's weird - debian is usually rather conservative on its packages in a stable release. Are you sure?
> 
> I also updated to a backported kernel 5.7 without any change.


Hi

Please can you open upstream BZ here:

https://bugzilla.redhat.com/enter_bug.cgi?product=LVM%20and%20device-mapper

List all the info - i.e. package versions (yep even .debs), kernel versions
lvmdump  archive.

I can probably easily 'hand-make' lvm2 metadata for you for 'vgcfgrestore' - 
but I'd like to track this case and create some reproducer for this issue so
we can handle case were the cache cannot be cleared.

Regards

Zdenek

Re: [linux-lvm] Removing lvmcache fails

[Roy Sigurd Karlsbakk]

> Please can you open upstream BZ here:
> 
> https://bugzilla.redhat.com/enter_bug.cgi?product=LVM%20and%20device-mapper
> 
> List all the info - i.e. package versions (yep even .debs), kernel versions
> lvmdump  archive.
> 
> I can probably easily 'hand-make' lvm2 metadata for you for 'vgcfgrestore' -
> but I'd like to track this case and create some reproducer for this issue so
> we can handle case were the cache cannot be cleared.

https://bugzilla.redhat.com/show_bug.cgi?id=1881056

Here you go. Thanks for any help :)

PS: I love languages, I just wonder how you pronounce Zdenek ;)

Vennlig hilsen

roy
-- 
Roy Sigurd Karlsbakk
(+47) 98013356
http://blogg.karlsbakk.net/
GPG Public key: http://karlsbakk.net/roysigurdkarlsbakk.pubkey.txt
--
Hið góða skaltu í stein höggva, hið illa í snjó rita.