linux-lvm.redhat.com archive mirror
 help / color / mirror / Atom feed
* [linux-lvm] Recovering from a failed pvmove
@ 2022-06-27  6:52 Roger James
  2022-06-28  8:30 ` Roger James
  2022-06-28 18:12 ` Roger Heflin
  0 siblings, 2 replies; 4+ messages in thread
From: Roger James @ 2022-06-27  6:52 UTC (permalink / raw)
  To: linux-lvm


[-- Attachment #1.1: Type: text/plain, Size: 4075 bytes --]

Hi,

I am struggling to recover from a failed pvmove. Unfortunately I only have 
a limited knowledge of lvm. I setup my lvm configuration many years ago.

I was trying to move a lv to a SSD using pvmove. Unfortunately my brand new 
SSD choose that moment to fail (never buy cheap SSDs, lesson learnt!").

This is the current status.

roger@dragon:~$ sudo pvs
  WARNING: Couldn't find device with uuid 
uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ.
  WARNING: VG wd is missing PV uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ (last 
written to [unknown]).
  PV VG Fmt Attr PSize PFree
  /dev/sda1 wd lvm2 a-- <465.76g 0
  /dev/sdb1 wd lvm2 a-- <465.76g <80.45g
  /dev/sdc2 wd lvm2 a-- 778.74g 278.74g
  /dev/sdd1 wd lvm2 a-- <465.76g 0
  [unknown] wd lvm2 a-m <784.49g 685.66g
roger@dragon:~$ sudo lvs
  WARNING: Couldn't find device with uuid 
uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ.
  WARNING: VG wd is missing PV uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ (last 
written to [unknown]).
  LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
  home wd -wi------- 1.46t
  root wd -wI-----p- <108.83g
  swap wd -wi------- 8.00g
  work wd -wi------- 200.00g
roger@dragon:~$ sudo vgs
  WARNING: Couldn't find device with uuid 
uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ.
  WARNING: VG wd is missing PV uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ (last 
written to [unknown]).
  VG #PV #LV #SN Attr VSize VFree
  wd 5 4 0 wz-pn- 2.89t 1.02t

This is a recap of what I have tried so far.

roger@dragon:~$ sudo pvmove --abort
  WARNING: Couldn't find device with uuid 
uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ.
  WARNING: VG wd is missing PV uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ (last 
written to [unknown]).
  LVM command executed by lvmpolld failed.
  For more information see lvmpolld messages in syslog or lvmpolld log file.
roger@dragon:~$ sudo vgreduce --removemissing wd
  WARNING: Couldn't find device with uuid 
uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ.
  WARNING: VG wd is missing PV uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ (last 
written to [unknown]).
  WARNING: Couldn't find device with uuid 
uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ.
  WARNING: Partial LV root needs to be repaired or removed.
  WARNING: Partial LV pvmove0 needs to be repaired or removed.
  There are still partial LVs in VG wd.
  To remove them unconditionally use: vgreduce --removemissing --force.
  To remove them unconditionally from mirror LVs use: vgreduce 
--removemissing --mirrorsonly --force.
  WARNING: Proceeding to remove empty missing PVs.
  WARNING: Couldn't find device with uuid 
uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ.
roger@dragon:~$ sudo lvchange -an wd/root
  WARNING: Couldn't find device with uuid 
uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ.
  WARNING: VG wd is missing PV uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ (last 
written to [unknown]).
roger@dragon:~$ sudo vgreduce --removemissing wd
  WARNING: Couldn't find device with uuid 
uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ.
  WARNING: VG wd is missing PV uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ (last 
written to [unknown]).
  WARNING: Couldn't find device with uuid 
uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ.
  WARNING: Partial LV root needs to be repaired or removed.
  WARNING: Partial LV pvmove0 needs to be repaired or removed.
  There are still partial LVs in VG wd.
  To remove them unconditionally use: vgreduce --removemissing --force.
  To remove them unconditionally from mirror LVs use: vgreduce 
--removemissing --mirrorsonly --force.
  WARNING: Proceeding to remove empty missing PVs.
  WARNING: Couldn't find device with uuid 
uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ.
roger@dragon:~$ sudo lvremove wd/pvmove0
  WARNING: Couldn't find device with uuid 
uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ.
  WARNING: VG wd is missing PV uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ (last 
written to [unknown]).
  WARNING: Couldn't find device with uuid 
uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ.
  Can't remove locked logical volume wd/pvmove0.

I am quite happy to loose the root lv, I just need the home and work lvs. 
What am I missing?

Help!

Roger

[-- Attachment #1.2: Type: text/html, Size: 6799 bytes --]

[-- Attachment #2: Type: text/plain, Size: 202 bytes --]

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [linux-lvm] Recovering from a failed pvmove
  2022-06-27  6:52 [linux-lvm] Recovering from a failed pvmove Roger James
@ 2022-06-28  8:30 ` Roger James
  2022-06-29 11:05   ` Roger Heflin
  2022-06-28 18:12 ` Roger Heflin
  1 sibling, 1 reply; 4+ messages in thread
From: Roger James @ 2022-06-28  8:30 UTC (permalink / raw)
  To: LVM general discussion and development


[-- Attachment #1.1: Type: text/plain, Size: 4803 bytes --]

I have now managed to fix the problem. I ran vgcfgbackup, then made a copy 
of the backup for safety purposes. I than hand edited the original backup 
to remove the missing pv (pv4), the root lv and the pvmove0 lv. I then than 
vgcfgrestore. Everything is working.

There must be a better way of doing this. Hand editing cfg files is not 
safe or sensible. What have I missed?

Roger

On 28 June 2022 07:38:48 Roger James <roger@beardandsandals.co.uk> wrote:
> Hi,
>
> I am struggling to recover from a failed pvmove. Unfortunately I only have 
> a limited knowledge of lvm. I setup my lvm configuration many years ago.
>
> I was trying to move a lv to a SSD using pvmove. Unfortunately my brand new 
> SSD choose that moment to fail (never buy cheap SSDs, lesson learnt!").
>
> This is the current status.
>
> roger@dragon:~$ sudo pvs
> WARNING: Couldn't find device with uuid uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ.
> WARNING: VG wd is missing PV uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ (last 
> written to [unknown]).
> PV VG Fmt Attr PSize PFree
> /dev/sda1 wd lvm2 a-- <465.76g 0
> /dev/sdb1 wd lvm2 a-- <465.76g <80.45g
> /dev/sdc2 wd lvm2 a-- 778.74g 278.74g
> /dev/sdd1 wd lvm2 a-- <465.76g 0
> [unknown] wd lvm2 a-m <784.49g 685.66g
> roger@dragon:~$ sudo lvs
> WARNING: Couldn't find device with uuid uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ.
> WARNING: VG wd is missing PV uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ (last 
> written to [unknown]).
> LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
> home wd -wi------- 1.46t
> root wd -wI-----p- <108.83g
> swap wd -wi------- 8.00g
> work wd -wi------- 200.00g
> roger@dragon:~$ sudo vgs
> WARNING: Couldn't find device with uuid uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ.
> WARNING: VG wd is missing PV uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ (last 
> written to [unknown]).
> VG #PV #LV #SN Attr VSize VFree
> wd 5 4 0 wz-pn- 2.89t 1.02t
>
> This is a recap of what I have tried so far.
>
> roger@dragon:~$ sudo pvmove --abort
> WARNING: Couldn't find device with uuid uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ.
> WARNING: VG wd is missing PV uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ (last 
> written to [unknown]).
> LVM command executed by lvmpolld failed.
> For more information see lvmpolld messages in syslog or lvmpolld log file.
> roger@dragon:~$ sudo vgreduce --removemissing wd
> WARNING: Couldn't find device with uuid uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ.
> WARNING: VG wd is missing PV uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ (last 
> written to [unknown]).
> WARNING: Couldn't find device with uuid uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ.
> WARNING: Partial LV root needs to be repaired or removed.
> WARNING: Partial LV pvmove0 needs to be repaired or removed.
> There are still partial LVs in VG wd.
> To remove them unconditionally use: vgreduce --removemissing --force.
> To remove them unconditionally from mirror LVs use: vgreduce 
> --removemissing --mirrorsonly --force.
> WARNING: Proceeding to remove empty missing PVs.
> WARNING: Couldn't find device with uuid uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ.
> roger@dragon:~$ sudo lvchange -an wd/root
> WARNING: Couldn't find device with uuid uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ.
> WARNING: VG wd is missing PV uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ (last 
> written to [unknown]).
> roger@dragon:~$ sudo vgreduce --removemissing wd
> WARNING: Couldn't find device with uuid uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ.
> WARNING: VG wd is missing PV uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ (last 
> written to [unknown]).
> WARNING: Couldn't find device with uuid uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ.
> WARNING: Partial LV root needs to be repaired or removed.
> WARNING: Partial LV pvmove0 needs to be repaired or removed.
> There are still partial LVs in VG wd.
> To remove them unconditionally use: vgreduce --removemissing --force.
> To remove them unconditionally from mirror LVs use: vgreduce 
> --removemissing --mirrorsonly --force.
> WARNING: Proceeding to remove empty missing PVs.
> WARNING: Couldn't find device with uuid uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ.
> roger@dragon:~$ sudo lvremove wd/pvmove0
> WARNING: Couldn't find device with uuid uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ.
> WARNING: VG wd is missing PV uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ (last 
> written to [unknown]).
> WARNING: Couldn't find device with uuid uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ.
> Can't remove locked logical volume wd/pvmove0.
>
> I am quite happy to loose the root lv, I just need the home and work lvs. 
> What am I missing?
>
> Help!
>
> Roger
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://listman.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/


[-- Attachment #1.2: Type: text/html, Size: 8397 bytes --]

[-- Attachment #2: Type: text/plain, Size: 202 bytes --]

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [linux-lvm] Recovering from a failed pvmove
  2022-06-27  6:52 [linux-lvm] Recovering from a failed pvmove Roger James
  2022-06-28  8:30 ` Roger James
@ 2022-06-28 18:12 ` Roger Heflin
  1 sibling, 0 replies; 4+ messages in thread
From: Roger Heflin @ 2022-06-28 18:12 UTC (permalink / raw)
  To: LVM general discussion and development


[-- Attachment #1.1: Type: text/plain, Size: 5646 bytes --]

For a case like this vgcfgrestore is probably the best option.  man
vgcfgrestore.

You need to see if you have archived vg copies that you can revert to
before the "add" of the pv that went bad.

The archives are typically  in /etc/lvm/archiive/<vgname>* on RedHat
derivative OSes, not sure if they are different(and/or configured to exist)
on other distributions.

grep -i before /etc/lvm/archive/<vgname>* and see which archive was made
before the initial pv addition.  vgcfgrestore -f <goodconfig> should work
but I usually have to adjust command line options to get it work when I
have used it to revert configs.  I think in that case it will find the vg
and pvid correctly.  No cleanup should need to be done so long as the other
device is completely gone.

And you will probably need to answer some prompts and warnings, and then
reboot the machine, and/or do this all under a livecd rescue boot.

What kind of cheap ssd were you using?  I have had really bad luck with
ones without RAM.  I RMA'ed one that failed in under a week and the new one
also failed in a very similar way in under a week.

On Tue, Jun 28, 2022 at 1:38 AM Roger James <roger@beardandsandals.co.uk>
wrote:

> Hi,
>
> I am struggling to recover from a failed pvmove. Unfortunately I only have
> a limited knowledge of lvm. I setup my lvm configuration many years ago.
>
> I was trying to move a lv to a SSD using pvmove. Unfortunately my brand
> new SSD choose that moment to fail (never buy cheap SSDs, lesson learnt!").
>
> This is the current status.
>
> roger@dragon:~$ sudo pvs
>   WARNING: Couldn't find device with uuid
> uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ.
>   WARNING: VG wd is missing PV uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ
> (last written to [unknown]).
>   PV VG Fmt Attr PSize PFree
>   /dev/sda1 wd lvm2 a-- <465.76g 0
>   /dev/sdb1 wd lvm2 a-- <465.76g <80.45g
>   /dev/sdc2 wd lvm2 a-- 778.74g 278.74g
>   /dev/sdd1 wd lvm2 a-- <465.76g 0
>   [unknown] wd lvm2 a-m <784.49g 685.66g
> roger@dragon:~$ sudo lvs
>   WARNING: Couldn't find device with uuid
> uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ.
>   WARNING: VG wd is missing PV uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ
> (last written to [unknown]).
>   LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
>   home wd -wi------- 1.46t
>   root wd -wI-----p- <108.83g
>
>   swap wd -wi------- 8.00g
>   work wd -wi------- 200.00g
> roger@dragon:~$ sudo vgs
>   WARNING: Couldn't find device with uuid
> uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ.
>   WARNING: VG wd is missing PV uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ
> (last written to [unknown]).
>   VG #PV #LV #SN Attr VSize VFree
>   wd 5 4 0 wz-pn- 2.89t 1.02t
>
> This is a recap of what I have tried so far.
>
> roger@dragon:~$ sudo pvmove --abort
>   WARNING: Couldn't find device with uuid
> uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ.
>   WARNING: VG wd is missing PV uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ
> (last written to [unknown]).
>   LVM command executed by lvmpolld failed.
>   For more information see lvmpolld messages in syslog or lvmpolld log
> file.
> roger@dragon:~$ sudo vgreduce --removemissing wd
>   WARNING: Couldn't find device with uuid
> uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ.
>   WARNING: VG wd is missing PV uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ
> (last written to [unknown]).
>   WARNING: Couldn't find device with uuid
> uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ.
>   WARNING: Partial LV root needs to be repaired or removed.
>   WARNING: Partial LV pvmove0 needs to be repaired or removed.
>   There are still partial LVs in VG wd.
>   To remove them unconditionally use: vgreduce --removemissing --force.
>   To remove them unconditionally from mirror LVs use: vgreduce
> --removemissing --mirrorsonly --force.
>   WARNING: Proceeding to remove empty missing PVs.
>   WARNING: Couldn't find device with uuid
> uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ.
> roger@dragon:~$ sudo lvchange -an wd/root
>   WARNING: Couldn't find device with uuid
> uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ.
>   WARNING: VG wd is missing PV uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ
> (last written to [unknown]).
> roger@dragon:~$ sudo vgreduce --removemissing wd
>   WARNING: Couldn't find device with uuid
> uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ.
>   WARNING: VG wd is missing PV uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ
> (last written to [unknown]).
>   WARNING: Couldn't find device with uuid
> uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ.
>   WARNING: Partial LV root needs to be repaired or removed.
>   WARNING: Partial LV pvmove0 needs to be repaired or removed.
>   There are still partial LVs in VG wd.
>   To remove them unconditionally use: vgreduce --removemissing --force.
>   To remove them unconditionally from mirror LVs use: vgreduce
> --removemissing --mirrorsonly --force.
>   WARNING: Proceeding to remove empty missing PVs.
>   WARNING: Couldn't find device with uuid
> uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ.
> roger@dragon:~$ sudo lvremove wd/pvmove0
>   WARNING: Couldn't find device with uuid
> uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ.
>   WARNING: VG wd is missing PV uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ
> (last written to [unknown]).
>   WARNING: Couldn't find device with uuid
> uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ.
>   Can't remove locked logical volume wd/pvmove0.
>
> I am quite happy to loose the root lv, I just need the home and work lvs.
> What am I missing?
>
> Help!
>
> Roger
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://listman.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>

[-- Attachment #1.2: Type: text/html, Size: 8295 bytes --]

[-- Attachment #2: Type: text/plain, Size: 202 bytes --]

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [linux-lvm] Recovering from a failed pvmove
  2022-06-28  8:30 ` Roger James
@ 2022-06-29 11:05   ` Roger Heflin
  0 siblings, 0 replies; 4+ messages in thread
From: Roger Heflin @ 2022-06-29 11:05 UTC (permalink / raw)
  To: LVM general discussion and development

Did your operating system have backups/archives of the old cfgs?

Fedora/Redhat keeps around 20 old copies going back typically several
months and so typically one of those can be used.

My archive dir looks like this for the roovg (note I do not do a lot
of vg work, typically
 ls -l /etc/lvm/archive/fedora*
-rw-------. 1 root root 3159 Oct 16  2021
/etc/lvm/archive/fedora_00031-1021957302.vg
-rw-------. 1 root root 2740 Oct 16  2021
/etc/lvm/archive/fedora_00032-1140767599.vg
-rw-------. 1 root root 2331 Oct 16  2021
/etc/lvm/archive/fedora_00033-104560841.vg
-rw-------. 1 root root 2084 Oct 16  2021
/etc/lvm/archive/fedora_00034-239698665.vg
-rw-------. 1 root root 2330 Oct 16  2021
/etc/lvm/archive/fedora_00035-1994061504.vg
-rw-------. 1 root root 1973 Oct 16  2021
/etc/lvm/archive/fedora_00036-974449793.vg
-rw-------. 1 root root 1997 Oct 16  2021
/etc/lvm/archive/fedora_00037-1503932417.vg
-rw-------. 1 root root 1997 Oct 16  2021
/etc/lvm/archive/fedora_00038-951442204.vg
-rw-------. 1 root root 1997 Oct 16  2021
/etc/lvm/archive/fedora_00039-989943813.vg
-rw-------. 1 root root 1968 Oct 16  2021
/etc/lvm/archive/fedora_00040-815563362.vg
-rw-------. 1 root root 1997 Oct 16  2021
/etc/lvm/archive/fedora_00041-1303737065.vg
-rw-------. 1 root root 1968 Oct 16  2021
/etc/lvm/archive/fedora_00042-714254626.vg
-rw-------. 1 root root 1976 Oct 16  2021
/etc/lvm/archive/fedora_00043-858775161.vg
-rw-------. 1 root root 2229 Oct 16  2021
/etc/lvm/archive/fedora_00044-1360584830.vg
-rw-------. 1 root root 2238 Oct 16  2021
/etc/lvm/archive/fedora_00045-1806472194.vg
-rw-------. 1 root root 2218 Oct 16  2021
/etc/lvm/archive/fedora_00046-519515389.vg
-rw-------. 1 root root 1992 Oct 16  2021
/etc/lvm/archive/fedora_00047-1997891375.vg
-rw-------. 1 root root 1988 Oct 16  2021
/etc/lvm/archive/fedora_00048-1666128451.vg
-rw-------. 1 root root 2396 Oct 18  2021
/etc/lvm/archive/fedora_00049-67607509.vg
-rw-------. 1 root root 2396 Oct 18  2021
/etc/lvm/archive/fedora_00050-1102265641.vg
-rw-------. 1 root root 2404 Oct 18  2021
/etc/lvm/archive/fedora_00051-243833122.vg
-rw-------. 1 root root 1738 Oct 18  2021
/etc/lvm/archive/fedora_00052-1619420890.vg
-rw-------. 1 root root 1744 Oct 18  2021
/etc/lvm/archive/fedora_00053-149551096.vg
-rw-------. 1 root root 1748 Nov 11  2021
/etc/lvm/archive/fedora_00054-1785934566.vg
-rw-------. 1 root root 1748 Nov 11  2021
/etc/lvm/archive/fedora_00055-1992369253.vg
-rw-------. 1 root root 1719 Nov 11  2021
/etc/lvm/archive/fedora_00056-26730175.vg
-rw-------. 1 root root 1748 Nov 11  2021
/etc/lvm/archive/fedora_00057-1989410182.vg
-rw-------. 1 root root 1719 Nov 11  2021
/etc/lvm/archive/fedora_00058-1590942582.vg
-rw-------. 1 root root 1748 Nov 11  2021
/etc/lvm/archive/fedora_00059-140743745.vg

I have only personally had to edit the lvm archive I needed to use  a
few times, and that has always been to remove the MISSING tag from
PV's that are no longer missing and the missing tag blocks the archive
from being used.

Most of the time I am correcting VG's I revert to the archive copy
that was taked before the bad steps were done.

There are options that can be put in /etc/lvm/lvm.conf and
/etc/lvm/lvmlocal.conf that can cause the archive's to get collected
if they are not being automatically collected, I am pretty sure I have
seen a few distributions that do not seem to collect the archives that
do typically give you a restore point to revert to (without editing
the file directly).


On Wed, Jun 29, 2022 at 2:32 AM Roger James <roger@beardandsandals.co.uk> wrote:
>
> I have now managed to fix the problem. I ran vgcfgbackup, then made a copy of the backup for safety purposes. I than hand edited the original backup to remove the missing pv (pv4), the root lv and the pvmove0 lv. I then than vgcfgrestore. Everything is working.
>
> There must be a better way of doing this. Hand editing cfg files is not safe or sensible. What have I missed?
>
> Roger
>
> On 28 June 2022 07:38:48 Roger James <roger@beardandsandals.co.uk> wrote:
>>
>> Hi,
>>
>> I am struggling to recover from a failed pvmove. Unfortunately I only have a limited knowledge of lvm. I setup my lvm configuration many years ago.
>>
>> I was trying to move a lv to a SSD using pvmove. Unfortunately my brand new SSD choose that moment to fail (never buy cheap SSDs, lesson learnt!").
>>
>> This is the current status.
>>
>> roger@dragon:~$ sudo pvs
>>   WARNING: Couldn't find device with uuid uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ.
>>   WARNING: VG wd is missing PV uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ (last written to [unknown]).
>>   PV VG Fmt Attr PSize PFree
>>   /dev/sda1 wd lvm2 a-- <465.76g 0
>>   /dev/sdb1 wd lvm2 a-- <465.76g <80.45g
>>   /dev/sdc2 wd lvm2 a-- 778.74g 278.74g
>>   /dev/sdd1 wd lvm2 a-- <465.76g 0
>>   [unknown] wd lvm2 a-m <784.49g 685.66g
>> roger@dragon:~$ sudo lvs
>>   WARNING: Couldn't find device with uuid uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ.
>>   WARNING: VG wd is missing PV uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ (last written to [unknown]).
>>   LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
>>   home wd -wi------- 1.46t
>>   root wd -wI-----p- <108.83g
>>   swap wd -wi------- 8.00g
>>   work wd -wi------- 200.00g
>> roger@dragon:~$ sudo vgs
>>   WARNING: Couldn't find device with uuid uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ.
>>   WARNING: VG wd is missing PV uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ (last written to [unknown]).
>>   VG #PV #LV #SN Attr VSize VFree
>>   wd 5 4 0 wz-pn- 2.89t 1.02t
>>
>> This is a recap of what I have tried so far.
>>
>> roger@dragon:~$ sudo pvmove --abort
>>   WARNING: Couldn't find device with uuid uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ.
>>   WARNING: VG wd is missing PV uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ (last written to [unknown]).
>>   LVM command executed by lvmpolld failed.
>>   For more information see lvmpolld messages in syslog or lvmpolld log file.
>> roger@dragon:~$ sudo vgreduce --removemissing wd
>>   WARNING: Couldn't find device with uuid uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ.
>>   WARNING: VG wd is missing PV uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ (last written to [unknown]).
>>   WARNING: Couldn't find device with uuid uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ.
>>   WARNING: Partial LV root needs to be repaired or removed.
>>   WARNING: Partial LV pvmove0 needs to be repaired or removed.
>>   There are still partial LVs in VG wd.
>>   To remove them unconditionally use: vgreduce --removemissing --force.
>>   To remove them unconditionally from mirror LVs use: vgreduce --removemissing --mirrorsonly --force.
>>   WARNING: Proceeding to remove empty missing PVs.
>>   WARNING: Couldn't find device with uuid uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ.
>> roger@dragon:~$ sudo lvchange -an wd/root
>>   WARNING: Couldn't find device with uuid uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ.
>>   WARNING: VG wd is missing PV uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ (last written to [unknown]).
>> roger@dragon:~$ sudo vgreduce --removemissing wd
>>   WARNING: Couldn't find device with uuid uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ.
>>   WARNING: VG wd is missing PV uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ (last written to [unknown]).
>>   WARNING: Couldn't find device with uuid uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ.
>>   WARNING: Partial LV root needs to be repaired or removed.
>>   WARNING: Partial LV pvmove0 needs to be repaired or removed.
>>   There are still partial LVs in VG wd.
>>   To remove them unconditionally use: vgreduce --removemissing --force.
>>   To remove them unconditionally from mirror LVs use: vgreduce --removemissing --mirrorsonly --force.
>>   WARNING: Proceeding to remove empty missing PVs.
>>   WARNING: Couldn't find device with uuid uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ.
>> roger@dragon:~$ sudo lvremove wd/pvmove0
>>   WARNING: Couldn't find device with uuid uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ.
>>   WARNING: VG wd is missing PV uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ (last written to [unknown]).
>>   WARNING: Couldn't find device with uuid uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ.
>>   Can't remove locked logical volume wd/pvmove0.
>>
>> I am quite happy to loose the root lv, I just need the home and work lvs. What am I missing?
>>
>> Help!
>>
>> Roger
>>
>> _______________________________________________
>> linux-lvm mailing list
>> linux-lvm@redhat.com
>> https://listman.redhat.com/mailman/listinfo/linux-lvm
>> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>>
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://listman.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/


^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, other threads:[~2022-06-29 11:06 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-06-27  6:52 [linux-lvm] Recovering from a failed pvmove Roger James
2022-06-28  8:30 ` Roger James
2022-06-29 11:05   ` Roger Heflin
2022-06-28 18:12 ` Roger Heflin

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).