[linux-lvm] LVM cachepool inconsistency after power event

[linux-lvm] LVM cachepool inconsistency after power event

[Krzysztof Chojnowski]

Hello all,

I'm experimenting with lvmcache, trying to use NVMe disk to speed up
access to rotational disks. I was using the cachepool in a writeback
mode when a power failure occurred, this left the cache in a
inconsistent state. So far I managed to activate the underlying _corig
LV and copy the data from there, but I'm wondering if it's still
possible to repair such cache and if not how do I remove failed LVs to
reclaim the disk space.

This is the relevant portion of the lvs:
  tpg1-wdata                    vg0 Cwi---C--- 500,00g
[wdata_cachepool_cpool] [tpg1-wdata_corig]
  [tpg1-wdata_corig]            vg0 owi---C--- 500,00g
  [wdata_cachepool_cpool]       vg0 Cwi---C---  50,00g
  [wdata_cachepool_cpool_cdata] vg0 Cwi-------  50,00g
  [wdata_cachepool_cpool_cmeta] vg0 ewi-------  40,00m

Trying to activate the tpg1-wdata LV results in an error:
sudo lvchange -ay -v vg0/tpg1-wdata
  Activating logical volume vg0/tpg1-wdata.
  activation/volume_list configuration setting not defined: Checking
only host tags for vg0/tpg1-wdata.
  Creating vg0-wdata_cachepool_cpool_cdata
  Loading table for vg0-wdata_cachepool_cpool_cdata (253:17).
  Resuming vg0-wdata_cachepool_cpool_cdata (253:17).
  Creating vg0-wdata_cachepool_cpool_cmeta
  Loading table for vg0-wdata_cachepool_cpool_cmeta (253:18).
  Resuming vg0-wdata_cachepool_cpool_cmeta (253:18).
  Creating vg0-tpg1--wdata_corig
  Loading table for vg0-tpg1--wdata_corig (253:19).
  Resuming vg0-tpg1--wdata_corig (253:19).
  Executing: /usr/sbin/cache_check -q
/dev/mapper/vg0-wdata_cachepool_cpool_cmeta
  /usr/sbin/cache_check failed: 1
  Piping: /usr/sbin/cache_check -V
  Found version of /usr/sbin/cache_check 0.9.0 is better then requested 0.7.0.
  Check of pool vg0/wdata_cachepool_cpool failed (status:1). Manual
repair required!
  Removing vg0-tpg1--wdata_corig (253:19)
  Removing vg0-wdata_cachepool_cpool_cmeta (253:18)
  Removing vg0-wdata_cachepool_cpool_cdata (253:17)

I tried repairing the volume, but no change:
sudo lvconvert --repair -v vg0/tpg1-wdata
  activation/volume_list configuration setting not defined: Checking
only host tags for vg0/lvol6_pmspare.
  Creating vg0-lvol6_pmspare
  Loading table for vg0-lvol6_pmspare (253:17).
  Resuming vg0-lvol6_pmspare (253:17).
  activation/volume_list configuration setting not defined: Checking
only host tags for vg0/wdata_cachepool_cpool_cmeta.
  Creating vg0-wdata_cachepool_cpool_cmeta
  Loading table for vg0-wdata_cachepool_cpool_cmeta (253:18).
  Resuming vg0-wdata_cachepool_cpool_cmeta (253:18).
  Executing: /usr/sbin/cache_repair  -i
/dev/mapper/vg0-wdata_cachepool_cpool_cmeta -o
/dev/mapper/vg0-lvol6_pmspare
  Removing vg0-wdata_cachepool_cpool_cmeta (253:18)
  Removing vg0-lvol6_pmspare (253:17)
  Preparing pool metadata spare volume for Volume group vg0.
  Archiving volume group "vg0" metadata (seqno 51).
  Creating logical volume lvol7
  Creating volume group backup "/etc/lvm/backup/vg0" (seqno 52).
  Activating logical volume vg0/lvol7.
  activation/volume_list configuration setting not defined: Checking
only host tags for vg0/lvol7.
  Creating vg0-lvol7
  Loading table for vg0-lvol7 (253:17).
  Resuming vg0-lvol7 (253:17).
  Initializing 40,00 MiB of logical volume vg0/lvol7 with value 0.
  Temporary logical volume "lvol7" created.
  Removing vg0-lvol7 (253:17)
  Renaming lvol7 as pool metadata spare volume lvol7_pmspare.
  WARNING: If everything works, remove vg0/tpg1-wdata_meta1 volume.
  WARNING: Use pvmove command to move vg0/wdata_cachepool_cpool_cmeta
on the best fitting PV.

Trying to remove the cache volume also fails:
sudo lvremove -ff vg0/tpg1-wdata
  Check of pool vg0/wdata_cachepool_cpool failed (status:1). Manual
repair required!
  Failed to activate vg0/tpg1-wdata to flush cache.

Any help in resolving this is appreciated!
Thanks,

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] LVM cachepool inconsistency after power event

[Ming-Hung Tsai]

Hi,

Could you help run cache_check independently, to see the error messages? Thanks.

# sudo lvchange -ay -f vg0/wdata_cachepool_cpool_cmeta
# sudo cache_check /dev/mapper/vg0-wdata_cachepool_cpool_cmeta
# sudo lvchange -an -f vg0/wdata_cachepool_cpool_cmeta


Ming-Hung Tsai

On Tue, Oct 5, 2021 at 1:26 PM Krzysztof Chojnowski <frirajder@gmail.com> wrote:
> Trying to activate the tpg1-wdata LV results in an error:
> sudo lvchange -ay -v vg0/tpg1-wdata
>   Executing: /usr/sbin/cache_check -q
> /dev/mapper/vg0-wdata_cachepool_cpool_cmeta
>   /usr/sbin/cache_check failed: 1
>   Piping: /usr/sbin/cache_check -V
>   Found version of /usr/sbin/cache_check 0.9.0 is better then requested 0.7.0.
>   Check of pool vg0/wdata_cachepool_cpool failed (status:1). Manual
> repair required!

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] LVM cachepool inconsistency after power event

[Krzysztof Chojnowski]

On Tue, Oct 5, 2021 at 06:35 AM Ming-Hung Tsai <mingnus@gmail.com> wrote:
>Could you help run cache_check independently, to see the error messages? Thanks.
>
># sudo lvchange -ay -f vg0/wdata_cachepool_cpool_cmeta
># sudo cache_check /dev/mapper/vg0-wdata_cachepool_cpool_cmeta
># sudo lvchange -an -f vg0/wdata_cachepool_cpool_cmeta

Thanks for your help, this is what I get:
$ sudo lvchange -ay -f vg0/wdata_cachepool_cpool_cmeta
Do you want to activate component LV in read-only mode? [y/n]: y
  Allowing activation of component LV.
$ sudo cache_check /dev/mapper/vg0-wdata_cachepool_cpool_cmeta
examining superblock
examining mapping array
  missing mappings [1, 0]:
    missing blocks
examining hint array
  missing mappings [1, 0]:
    missing blocks
examining discard bitset
$ echo $?
1
$ sudo lvchange -an -f vg0/wdata_cachepool_cpool_cmeta
$ sudo lvchange -ay   vg0/tpg1-wdata
  Check of pool vg0/wdata_cachepool_cpool failed (status:1). Manual
repair required!

Regards,
Krzysztof Chojnowski

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] LVM cachepool inconsistency after power event

[Zdenek Kabelac]

Dne 05. 10. 21 v 13:34 Krzysztof Chojnowski napsal(a):
> On Tue, Oct 5, 2021 at 06:35 AM Ming-Hung Tsai <mingnus@gmail.com> wrote:
>> Could you help run cache_check independently, to see the error messages? Thanks.
>>
>> # sudo lvchange -ay -f vg0/wdata_cachepool_cpool_cmeta
>> # sudo cache_check /dev/mapper/vg0-wdata_cachepool_cpool_cmeta
>> # sudo lvchange -an -f vg0/wdata_cachepool_cpool_cmeta
> 
> Thanks for your help, this is what I get:
> $ sudo lvchange -ay -f vg0/wdata_cachepool_cpool_cmeta
> Do you want to activate component LV in read-only mode? [y/n]: y
>    Allowing activation of component LV.
> $ sudo cache_check /dev/mapper/vg0-wdata_cachepool_cpool_cmeta
> examining superblock
> examining mapping array
>    missing mappings [1, 0]:
>      missing blocks
> examining hint array
>    missing mappings [1, 0]:
>      missing blocks
> examining discard bitset
> $ echo $?
> 1
> $ sudo lvchange -an -f vg0/wdata_cachepool_cpool_cmeta
> $ sudo lvchange -ay   vg0/tpg1-wdata
>    Check of pool vg0/wdata_cachepool_cpool failed (status:1). Manual
> repair required!

Hello Krzystof

You need to repair your cache-pool metadata.

But before continuing with advices - what is the version of kernel lvm2 & your
device-mapper-persistent-data package (aka  'cache_check -V)

Component activation allows activation of your _cmeta LV - but only in 
read-only mode - so repair must go into a new LV.

Since cache_check reported  '1' exist code (as an error) - your metadata do 
require a fix.

lvconvert --repair should be able to handle this case - although likely 
without 'smart' placement' of fixed metadata (pvmove needed after metadata fix)

You can allocated your new metadata and easily cache_repair them.


Regards

Zdenek

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] LVM cachepool inconsistency after power event

[Ming Hung Tsai]

On Tue, Oct 5, 2021 at 11:54 PM Zdenek Kabelac <zkabelac@redhat.com> wrote:
>
> Dne 05. 10. 21 v 13:34 Krzysztof Chojnowski napsal(a):
> > $ sudo cache_check /dev/mapper/vg0-wdata_cachepool_cpool_cmeta
> > examining superblock
> > examining mapping array
> >    missing mappings [1, 0]:
> >      missing blocks
> > examining hint array
> >    missing mappings [1, 0]:
> >      missing blocks
> > examining discard bitset
>
> Hello Krzystof
>
> You need to repair your cache-pool metadata.
>
> But before continuing with advices - what is the version of kernel lvm2 & your
> device-mapper-persistent-data package (aka  'cache_check -V)
>
> Component activation allows activation of your _cmeta LV - but only in
> read-only mode - so repair must go into a new LV.
>
> Since cache_check reported  '1' exist code (as an error) - your metadata do
> require a fix.
>
> lvconvert --repair should be able to handle this case - although likely
> without 'smart' placement' of fixed metadata (pvmove needed after metadata fix)
>
> You can allocated your new metadata and easily cache_repair them.
>
>
> Regards
>
> Zdenek


I think it's a case that the superblock was not committed before the
power outage, that's why the error message shows zero in the third
line. Some fields or unwritten blocks might be lost.

cache_repair might not handle this case perfectly. If possible, you
could upload the "original" metadata somewhere for me to take a look,
by dd it into a binary file.

The "original" metadata should be named vg0/tpg1-wdata_meta0 or
something suffixed with "_meta0" since you've run lvconvert.

Also, as Zdenek mentioned, please help provide the LVM, dmpd, and
kernel versions.


Ming-Hung Tsai

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] LVM cachepool inconsistency after power event

[Krzysztof Chojnowski]

[-- Attachment #1.1: Type: text/plain, Size: 2630 bytes --]

On Tue, Oct 5, 2021 at 6:57 PM Ming Hung Tsai <mtsai@redhat.com> wrote:

> On Tue, Oct 5, 2021 at 11:54 PM Zdenek Kabelac <zkabelac@redhat.com>
> wrote:
> > But before continuing with advices - what is the version of kernel lvm2
> & your
> > device-mapper-persistent-data package (aka  'cache_check -V)
> >
>
These are the software versions:
$ /usr/sbin/cache_check -V
0.9.0
$ /usr/sbin/lvm version
  LVM version:     2.03.11(2) (2021-01-08)
  Library version: 1.02.175 (2021-01-08)
/dev/mapper/control: open failed: Permission denied
Failure to communicate with kernel device-mapper driver.
Incompatible libdevmapper 1.02.175 (2021-01-08) and kernel driver (unknown
version).
  Configuration:   ./configure --build=x86_64-linux-gnu --prefix=/usr
--includedir=${prefix}/include --mandir=${prefix}/share/man
--infodir=${prefix}/share/info --sysconfdir=/etc --localstatedir=/var
--disable-option-checking --disable-silent-rules
--libdir=${prefix}/lib/x86_64-linux-gnu --runstatedir=/run
--disable-maintainer-mode --disable-dependency-tracking
--libdir=/lib/x86_64-linux-gnu --sbindir=/sbin
--with-usrlibdir=/usr/lib/x86_64-linux-gnu --with-optimisation=-O2
--with-cache=internal --with-device-uid=0 --with-device-gid=6
--with-device-mode=0660 --with-default-pid-dir=/run
--with-default-run-dir=/run/lvm --with-default-locking-dir=/run/lock/lvm
--with-thin=internal --with-thin-check=/usr/sbin/thin_check
--with-thin-dump=/usr/sbin/thin_dump
--with-thin-repair=/usr/sbin/thin_repair --with-udev-prefix=/
--enable-applib --enable-blkid_wiping --enable-cmdlib --enable-dmeventd
--enable-editline --enable-lvmlockd-dlm --enable-lvmlockd-sanlock
--enable-lvmpolld --enable-notify-dbus --enable-pkgconfig
--enable-udev_rules --enable-udev_sync --disable-readline
$ uname -a
Linux archer 5.14.0-1-amd64 #1 SMP Debian 5.14.6-3 (2021-09-28) x86_64
GNU/Linux


> > lvconvert --repair should be able to handle this case - although likely
> > without 'smart' placement' of fixed metadata (pvmove needed after
> metadata fix)
>
I tried running lvconvert --repair earlier (please see the 1st of my mails)
and even trough the operation finished successfully (as in the $? == 0) I
couldn't activate the volume afterwards.


> cache_repair might not handle this case perfectly. If possible, you
> could upload the "original" metadata somewhere for me to take a look,
> by dd it into a binary file.
>
Unfortunately I deleted the original metadata volume when I was trying to
remove the cached volume.

So right now, I guess, the question is how do I remove the cached volume,
as the repair doesn't seem to be possible.

Regards,
Krzysztof Chojnowski

[-- Attachment #1.2: Type: text/html, Size: 3511 bytes --]

[-- Attachment #2: Type: text/plain, Size: 201 bytes --]

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] LVM cachepool inconsistency after power event

[Ming Hung Tsai]

On Thu, Oct 7, 2021 at 3:57 PM Krzysztof Chojnowski <frirajder@gmail.com> wrote:
>
> On Tue, Oct 5, 2021 at 6:57 PM Ming Hung Tsai <mtsai@redhat.com> wrote:
>>
>> On Tue, Oct 5, 2021 at 11:54 PM Zdenek Kabelac <zkabelac@redhat.com> wrote:
>>
>> > lvconvert --repair should be able to handle this case - although likely
>> > without 'smart' placement' of fixed metadata (pvmove needed after metadata fix)
>
> I tried running lvconvert --repair earlier (please see the 1st of my mails) and even trough the operation finished successfully (as in the $? == 0) I couldn't activate the volume afterwards.
>
>>
>> cache_repair might not handle this case perfectly. If possible, you
>> could upload the "original" metadata somewhere for me to take a look,
>> by dd it into a binary file.
>
> Unfortunately I deleted the original metadata volume when I was trying to remove the cached volume.
> So right now, I guess, the question is how do I remove the cached volume, as the repair doesn't seem to be possible.

Hi,

You could simply detach the cache:
sudo lvconvert vg0/tpg1-wdata --splitcache

Or alternatively, reinitialize the cache by swapping the metadata with
a blank one, which bypasses cache_check examination and then steps to
metadata formatting:
sudo lvcreate vg0 --size 40m --name cmeta_bak
sudo lvconvert vg0/wdata_cachepool_cpool --swapmetadata --poolmetadata cmeta_bak

then you should be able to activate the tpg1-wdata.

BTW, could you help provide the reproducing steps? I guess that you
performed a power cut soon after creating the cache. Thanks.


Ming-Hung Tsai

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] LVM cachepool inconsistency after power event

[Krzysztof Chojnowski]

On Fri, Oct 8, 2021 at 9:57 AM Ming Hung Tsai <mtsai@redhat.com> wrote:
>
> You could simply detach the cache:
> sudo lvconvert vg0/tpg1-wdata --splitcache

Unfortunately this one didn't work:
$ sudo lvconvert vg0/tpg1-wdata --splitcache
Check of pool vg0/wdata_cachepool_cpool failed (status:1). Manual
repair required!
Failed to activate vg0/tpg1-wdata to flush cache.

> Or alternatively, reinitialize the cache by swapping the metadata with
> a blank one, which bypasses cache_check examination and then steps to
> metadata formatting:
> sudo lvcreate vg0 --size 40m --name cmeta_bak
> sudo lvconvert vg0/wdata_cachepool_cpool --swapmetadata --poolmetadata cmeta_bak

Thanks a lot! This one did work and I was able to activate the volume.
I tried doing something like this earlier but I used wrong syntax (I
missed the --swapmetadata flag).

> BTW, could you help provide the reproducing steps? I guess that you
> performed a power cut soon after creating the cache. Thanks.

Unfortunately, I don't have clear reproducing steps. As you've guessed
correctly - the power event has happened shortly (the same day) after
I created the cached volume. IIRC the volume hasn't even been used at
all so the cache would be empty. If I manage to reproduce it I will
report back here.

Thanks for all your help!
Krzysztof Chojnowski

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] LVM cachepool inconsistency after power event

[Ming Hung Tsai]

On Mon, Oct 11, 2021 at 8:18 PM Krzysztof Chojnowski
<frirajder@gmail.com> wrote:
> Unfortunately, I don't have clear reproducing steps. As you've guessed
> correctly - the power event has happened shortly (the same day) after
> I created the cached volume. IIRC the volume hasn't even been used at
> all so the cache would be empty.

So the cache is clean without any data written? I thought that there
was dirty data since you mentioned you've copied data from _corig in
the first post, so now I just want to confirm my understanding.


Ming-Hung Tsai

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] LVM cachepool inconsistency after power event

[Krzysztof Chojnowski]

On Mon, Oct 11, 2021 at 6:30 PM Ming Hung Tsai <mtsai@redhat.com> wrote:
>
> On Mon, Oct 11, 2021 at 8:18 PM Krzysztof Chojnowski
> <frirajder@gmail.com> wrote:
> > Unfortunately, I don't have clear reproducing steps. As you've guessed
> > correctly - the power event has happened shortly (the same day) after
> > I created the cached volume. IIRC the volume hasn't even been used at
> > all so the cache would be empty.
>
> So the cache is clean without any data written? I thought that there
> was dirty data since you mentioned you've copied data from _corig in
> the first post, so now I just want to confirm my understanding.

The _corig volume has been used without caching for long time.
Recently I started experiment with the caching capability and I used
command similar to following to set it up:
https://wiki.archlinux.org/title/LVM#Create_cache

Regards,
Krzysztof Chojnowski

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

[linux-lvm] LVM cachepool inconsistency after power event

[Krzysztof Chojnowski]

Hello all,

I'm experimenting with lvmcache, trying to use NVMe disk to speed up
access to rotational disks. I was using the cachepool in a writeback
mode when a power failure occurred, this left the cache in a
inconsistent state. So far I managed to activate the underlying _corig
LV and copy the data from there, but I'm wondering if it's still
possible to repair such cache and if not how do I remove failed LVs to
reclaim the disk space.

This is the relevant portion of the lvs:
  tpg1-wdata                    vg0 Cwi---C--- 500,00g
[wdata_cachepool_cpool] [tpg1-wdata_corig]
  [tpg1-wdata_corig]            vg0 owi---C--- 500,00g
  [wdata_cachepool_cpool]       vg0 Cwi---C---  50,00g
  [wdata_cachepool_cpool_cdata] vg0 Cwi-------  50,00g
  [wdata_cachepool_cpool_cmeta] vg0 ewi-------  40,00m

Trying to activate the tpg1-wdata LV results in an error:
sudo lvchange -ay -v vg0/tpg1-wdata
  Activating logical volume vg0/tpg1-wdata.
  activation/volume_list configuration setting not defined: Checking
only host tags for vg0/tpg1-wdata.
  Creating vg0-wdata_cachepool_cpool_cdata
  Loading table for vg0-wdata_cachepool_cpool_cdata (253:17).
  Resuming vg0-wdata_cachepool_cpool_cdata (253:17).
  Creating vg0-wdata_cachepool_cpool_cmeta
  Loading table for vg0-wdata_cachepool_cpool_cmeta (253:18).
  Resuming vg0-wdata_cachepool_cpool_cmeta (253:18).
  Creating vg0-tpg1--wdata_corig
  Loading table for vg0-tpg1--wdata_corig (253:19).
  Resuming vg0-tpg1--wdata_corig (253:19).
  Executing: /usr/sbin/cache_check -q
/dev/mapper/vg0-wdata_cachepool_cpool_cmeta
  /usr/sbin/cache_check failed: 1
  Piping: /usr/sbin/cache_check -V
  Found version of /usr/sbin/cache_check 0.9.0 is better then requested 0.7.0.
  Check of pool vg0/wdata_cachepool_cpool failed (status:1). Manual
repair required!
  Removing vg0-tpg1--wdata_corig (253:19)
  Removing vg0-wdata_cachepool_cpool_cmeta (253:18)
  Removing vg0-wdata_cachepool_cpool_cdata (253:17)

I tried repairing the volume, but no change:
sudo lvconvert --repair -v vg0/tpg1-wdata
  activation/volume_list configuration setting not defined: Checking
only host tags for vg0/lvol6_pmspare.
  Creating vg0-lvol6_pmspare
  Loading table for vg0-lvol6_pmspare (253:17).
  Resuming vg0-lvol6_pmspare (253:17).
  activation/volume_list configuration setting not defined: Checking
only host tags for vg0/wdata_cachepool_cpool_cmeta.
  Creating vg0-wdata_cachepool_cpool_cmeta
  Loading table for vg0-wdata_cachepool_cpool_cmeta (253:18).
  Resuming vg0-wdata_cachepool_cpool_cmeta (253:18).
  Executing: /usr/sbin/cache_repair  -i
/dev/mapper/vg0-wdata_cachepool_cpool_cmeta -o
/dev/mapper/vg0-lvol6_pmspare
  Removing vg0-wdata_cachepool_cpool_cmeta (253:18)
  Removing vg0-lvol6_pmspare (253:17)
  Preparing pool metadata spare volume for Volume group vg0.
  Archiving volume group "vg0" metadata (seqno 51).
  Creating logical volume lvol7
  Creating volume group backup "/etc/lvm/backup/vg0" (seqno 52).
  Activating logical volume vg0/lvol7.
  activation/volume_list configuration setting not defined: Checking
only host tags for vg0/lvol7.
  Creating vg0-lvol7
  Loading table for vg0-lvol7 (253:17).
  Resuming vg0-lvol7 (253:17).
  Initializing 40,00 MiB of logical volume vg0/lvol7 with value 0.
  Temporary logical volume "lvol7" created.
  Removing vg0-lvol7 (253:17)
  Renaming lvol7 as pool metadata spare volume lvol7_pmspare.
  WARNING: If everything works, remove vg0/tpg1-wdata_meta1 volume.
  WARNING: Use pvmove command to move vg0/wdata_cachepool_cpool_cmeta
on the best fitting PV.

Trying to remove the cache volume also fails:
sudo lvremove -ff vg0/tpg1-wdata
  Check of pool vg0/wdata_cachepool_cpool failed (status:1). Manual
repair required!
  Failed to activate vg0/tpg1-wdata to flush cache.

Any help in resolving this is appreciated!
Thanks,

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/