* [linux-lvm] Can't work normally after attaching disk volumes originally in a VG on another machine
@ 2018-03-23 8:30 Gang He
2018-03-23 9:04 ` Xen
2018-10-16 20:59 ` [linux-lvm] [lvm-devel] " Nir Soffer
0 siblings, 2 replies; 17+ messages in thread
From: Gang He @ 2018-03-23 8:30 UTC (permalink / raw)
To: linux-lvm, lvm-devel
Hello List,
Since I am new to LVM area, but I got a problem about LVM can't work normally after attaching disk volumes originally in a VG on another machine.
Then, I sent this problem to the list, to see if we can get a fix or there has been a fix.
The problem is very easy to reproduce,
1) create two virtual machines(tb0307-nd1, tb0307-nd2), create two virtual disks.
2) attach two virtual disks to VM1(tb0307-nd1).
tb0307-nd1:/# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sr0 11:0 1 4.3G 0 rom
vda 254:0 0 40G 0 disk
├─vda1 254:1 0 4G 0 part [SWAP]
├─vda2 254:2 0 23.6G 0 part /
└─vda3 254:3 0 12.4G 0 part /home
vdb 254:16 0 40G 0 disk
vdc 254:32 0 20G 0 disk <<= disk1
vdd 254:48 0 20G 0 disk <<= disk2
3) pvcreate two virtual disks, and add them to vg1.
tb0307-nd1:/# pvcreate /dev/vdc
Physical volume "/dev/vdc" successfully created.
tb0307-nd1:/# pvcreate /dev/vdd
Physical volume "/dev/vdd" successfully created.
tb0307-nd1:/# pvs
PV VG Fmt Attr PSize PFree
/dev/vdc lvm2 --- 20.00g 20.00g
/dev/vdd lvm2 --- 20.00g 20.00g
tb0307-nd1:/# vgcreate vg1 /dev/vdc /dev/vdd
4) disconnect two virtual disks from VM1, and attach disk1 to VM2(tb0307-nd2).
5) delete vg1, create vg2, and add disk1 to vg2.
tb0307-nd2:~ # pvs
WARNING: Device for PV t6gjHU-dtY4-th5O-A0mr-5OEC-R1ZK-TPQgIh not found or rejected by a filter.
PV VG Fmt Attr PSize PFree
/dev/vdc vg1 lvm2 a-- 20.00g 20.00g
[unknown] vg1 lvm2 a-m 20.00g 20.00g
tb0307-nd2:~ # pvremove /dev/vdc
WARNING: Device for PV t6gjHU-dtY4-th5O-A0mr-5OEC-R1ZK-TPQgIh not found or rejected by a filter.
PV /dev/vdc is used by VG vg1 so please use vgreduce first.
(If you are certain you need pvremove, then confirm by using --force twice.)
/dev/vdc: physical volume label not removed.
tb0307-nd2:~ # vgreduce --removemissing vg1
WARNING: Device for PV t6gjHU-dtY4-th5O-A0mr-5OEC-R1ZK-TPQgIh not found or rejected by a filter.
Wrote out consistent volume group vg1.
tb0307-nd2:~ # pvs
PV VG Fmt Attr PSize PFree
/dev/vdc vg1 lvm2 a-- 20.00g 20.00g
tb0307-nd2:~ # vgremove vg1
Volume group "vg1" successfully removed
tb0307-nd2:~ # vgcreate vg2 /dev/vdc
Volume group "vg2" successfully created
tb0307-nd2:~ # vgs
VG #PV #LV #SN Attr VSize VFree
vg2 1 0 0 wz--n- 20.00g 20.00g
tb0307-nd2:~ # pvs
PV VG Fmt Attr PSize PFree
/dev/vdc vg2 lvm2 a-- 20.00g 20.00g
6) attach disk2 to VM2(tb0307-nd2), the vg on VM2 looks abnormal.
tb0307-nd2:~ # pvs
WARNING: Device for PV JJOL4H-kc0j-jyTD-LDwl-71FZ-dHKM-YoFtNV not found or rejected by a filter.
PV VG Fmt Attr PSize PFree
/dev/vdc vg2 lvm2 a-- 20.00g 20.00g
/dev/vdd vg1 lvm2 a-- 20.00g 20.00g
[unknown] vg1 lvm2 a-m 20.00g 20.00g
tb0307-nd2:~ # vgs
WARNING: Device for PV JJOL4H-kc0j-jyTD-LDwl-71FZ-dHKM-YoFtNV not found or rejected by a filter.
VG #PV #LV #SN Attr VSize VFree
vg1 2 0 0 wz-pn- 39.99g 39.99g
vg2 1 0 0 wz--n- 20.00g 20.00g
7) reboot VM2, the result looks worse (vdc disk belongs to two vg).
tb0307-nd2:/mnt/shared # pvs
PV VG Fmt Attr PSize PFree
/dev/vdc vg1 lvm2 a-- 20.00g 0
/dev/vdc vg2 lvm2 a-- 20.00g 10.00g
/dev/vdd vg1 lvm2 a-- 20.00g 9.99g
It looks like LVM2 meta consistency problem? please help to take a look.
Thanks
Gang
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [linux-lvm] Can't work normally after attaching disk volumes originally in a VG on another machine
2018-03-23 8:30 [linux-lvm] Can't work normally after attaching disk volumes originally in a VG on another machine Gang He
@ 2018-03-23 9:04 ` Xen
2018-03-26 6:04 ` Gang He
2018-10-16 20:59 ` [linux-lvm] [lvm-devel] " Nir Soffer
1 sibling, 1 reply; 17+ messages in thread
From: Xen @ 2018-03-23 9:04 UTC (permalink / raw)
To: linux-lvm
Gang He schreef op 23-03-2018 9:30:
> 6) attach disk2 to VM2(tb0307-nd2), the vg on VM2 looks abnormal.
> tb0307-nd2:~ # pvs
> WARNING: Device for PV JJOL4H-kc0j-jyTD-LDwl-71FZ-dHKM-YoFtNV not
> found or rejected by a filter.
> PV VG Fmt Attr PSize PFree
> /dev/vdc vg2 lvm2 a-- 20.00g 20.00g
> /dev/vdd vg1 lvm2 a-- 20.00g 20.00g
> [unknown] vg1 lvm2 a-m 20.00g 20.00g
This is normal because /dev/vdd contains metadata for vg1 which includes
now missing disk /dev/vdc .... as the PV is no longer the same.
> tb0307-nd2:~ # vgs
> WARNING: Device for PV JJOL4H-kc0j-jyTD-LDwl-71FZ-dHKM-YoFtNV not
> found or rejected by a filter.
> VG #PV #LV #SN Attr VSize VFree
> vg1 2 0 0 wz-pn- 39.99g 39.99g
> vg2 1 0 0 wz--n- 20.00g 20.00g
This is normal because you haven't removed /dev/vdc from vg1 on
/dev/vdd, since it was detached while you operated on its vg.
> 7) reboot VM2, the result looks worse (vdc disk belongs to two vg).
> tb0307-nd2:/mnt/shared # pvs
> PV VG Fmt Attr PSize PFree
> /dev/vdc vg1 lvm2 a-- 20.00g 0
> /dev/vdc vg2 lvm2 a-- 20.00g 10.00g
> /dev/vdd vg1 lvm2 a-- 20.00g 9.99g
When you removed vdd when it was not attached, the VG1 metadata on vdd
was not altered. The metadata resides on both disks, so you had
inconsistent metadata between both disks because you operated on the
shared volume group while one device was missing.
You also did not recreate PV on /dev/vdc so it has the same UUID as when
it was part of VG1, this is why VG1 when VDD is booted will still try to
include /dev/vdc because it was never removed from the volume group on
VDD.
So the state of affairs is:
/dev/vdc contains volume group info for VG2 and includes only /dev/vdc
/dev/vdd contains volume group info for VG1, and includes both /dev/vdc
and /dev/vdd by UUID for its PV, however, it is a bug that it should
include /dev/vdc even though the VG UUID is now different (and the name
as well).
Regardless, from vdd's perspective /dev/vdc is still part of VG1.
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [linux-lvm] Can't work normally after attaching disk volumes originally in a VG on another machine
2018-03-23 9:04 ` Xen
@ 2018-03-26 6:04 ` Gang He
2018-03-26 10:17 ` Fran Garcia
` (3 more replies)
0 siblings, 4 replies; 17+ messages in thread
From: Gang He @ 2018-03-26 6:04 UTC (permalink / raw)
To: linux-lvm
Hi Xen,
>>>
> Gang He schreef op 23-03-2018 9:30:
>
>> 6) attach disk2 to VM2(tb0307-nd2), the vg on VM2 looks abnormal.
>> tb0307-nd2:~ # pvs
>> WARNING: Device for PV JJOL4H-kc0j-jyTD-LDwl-71FZ-dHKM-YoFtNV not
>> found or rejected by a filter.
>> PV VG Fmt Attr PSize PFree
>> /dev/vdc vg2 lvm2 a-- 20.00g 20.00g
>> /dev/vdd vg1 lvm2 a-- 20.00g 20.00g
>> [unknown] vg1 lvm2 a-m 20.00g 20.00g
>
> This is normal because /dev/vdd contains metadata for vg1 which includes
> now missing disk /dev/vdc .... as the PV is no longer the same.
>
>
>
>
>> tb0307-nd2:~ # vgs
>> WARNING: Device for PV JJOL4H-kc0j-jyTD-LDwl-71FZ-dHKM-YoFtNV not
>> found or rejected by a filter.
>> VG #PV #LV #SN Attr VSize VFree
>> vg1 2 0 0 wz-pn- 39.99g 39.99g
>> vg2 1 0 0 wz--n- 20.00g 20.00g
>
> This is normal because you haven't removed /dev/vdc from vg1 on
> /dev/vdd, since it was detached while you operated on its vg.
>
>
>> 7) reboot VM2, the result looks worse (vdc disk belongs to two vg).
>> tb0307-nd2:/mnt/shared # pvs
>> PV VG Fmt Attr PSize PFree
>> /dev/vdc vg1 lvm2 a-- 20.00g 0
>> /dev/vdc vg2 lvm2 a-- 20.00g 10.00g
>> /dev/vdd vg1 lvm2 a-- 20.00g 9.99g
>
> When you removed vdd when it was not attached, the VG1 metadata on vdd
> was not altered. The metadata resides on both disks, so you had
> inconsistent metadata between both disks because you operated on the
> shared volume group while one device was missing.
>
> You also did not recreate PV on /dev/vdc so it has the same UUID as when
> it was part of VG1, this is why VG1 when VDD is booted will still try to
> include /dev/vdc because it was never removed from the volume group on
> VDD.
>
> So the state of affairs is:
>
> /dev/vdc contains volume group info for VG2 and includes only /dev/vdc
>
> /dev/vdd contains volume group info for VG1, and includes both /dev/vdc
> and /dev/vdd by UUID for its PV, however, it is a bug that it should
> include /dev/vdc even though the VG UUID is now different (and the name
> as well).
It looks like each PV includes a copy meta data for VG, but if some PV has changed (e.g. removed, or moved to another VG),
the remained PV should have a method to check the integrity when each startup (activated?), to avoid such inconsistent problem automatically.
Thanks
Gang
>
> Regardless, from vdd's perspective /dev/vdc is still part of VG1.
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [linux-lvm] Can't work normally after attaching disk volumes originally in a VG on another machine
2018-03-26 6:04 ` Gang He
@ 2018-03-26 10:17 ` Fran Garcia
2018-03-26 10:23 ` Fran Garcia
` (2 subsequent siblings)
3 siblings, 0 replies; 17+ messages in thread
From: Fran Garcia @ 2018-03-26 10:17 UTC (permalink / raw)
To: LVM general discussion and development
On 26 March 2018 at 08:04, Gang He <ghe@suse.com> wrote:
>> Gang He schreef op 23-03-2018 9:30:
>>
>>> 6) attach disk2 to VM2(tb0307-nd2), the vg on VM2 looks abnormal.
>>> tb0307-nd2:~ # pvs
>>> WARNING: Device for PV JJOL4H-kc0j-jyTD-LDwl-71FZ-dHKM-YoFtNV not
>>> found or rejected by a filter.
>>> PV VG Fmt Attr PSize PFree
>>> /dev/vdc vg2 lvm2 a-- 20.00g 20.00g
>>> /dev/vdd vg1 lvm2 a-- 20.00g 20.00g
>>> [unknown] vg1 lvm2 a-m 20.00g 20.00g
>>
>> This is normal because /dev/vdd contains metadata for vg1 which includes
>> now missing disk /dev/vdc .... as the PV is no longer the same.
>
> It looks like each PV includes a copy meta data for VG, but if some PV has changed (e.g. removed, or moved to another VG),
> the remained PV should have a method to check the integrity when each startup (activated?), to avoid such inconsistent problem automatically.
Your workflow is strange. What are you trying to accomplish here?
Your steps in 5 should be:
vgreduce vg01 /dev/vdc /dev/vdc
pvremove /dev/vdc /dev/vdd
That way you ensure there's no leftover metadata in the PVs (specially
if you need to attach those disks to a different system)
Again a usecase to understand your workflow would be beneficial...
Cheers
Fran
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [linux-lvm] Can't work normally after attaching disk volumes originally in a VG on another machine
2018-03-26 6:04 ` Gang He
2018-03-26 10:17 ` Fran Garcia
@ 2018-03-26 10:23 ` Fran Garcia
2018-03-27 5:55 ` Gang He
2018-03-26 10:46 ` Zdenek Kabelac
2018-03-26 11:33 ` Xen
3 siblings, 1 reply; 17+ messages in thread
From: Fran Garcia @ 2018-03-26 10:23 UTC (permalink / raw)
To: LVM general discussion and development
On 26 March 2018 at 08:04, Gang He <ghe@suse.com> wrote:
> It looks like each PV includes a copy meta data for VG, but if some PV has changed (e.g. removed, or moved to another VG),
> the remained PV should have a method to check the integrity when each startup (activated?), to avoid such inconsistent problem automatically.
Your workflow is strange. What are you trying to accomplish here?
Your steps in 5 should be:
vgreduce vg01 /dev/vdc /dev/vdc
pvremove /dev/vdc /dev/vdd
That way you ensure there's no leftover metadata in the PVs (specially
if you need to attach those disks to a different system)
Again a usecase to understand your workflow would be beneficial...
Cheers
fran
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [linux-lvm] Can't work normally after attaching disk volumes originally in a VG on another machine
2018-03-26 6:04 ` Gang He
2018-03-26 10:17 ` Fran Garcia
2018-03-26 10:23 ` Fran Garcia
@ 2018-03-26 10:46 ` Zdenek Kabelac
2018-05-18 4:56 ` Gang He
2018-03-26 11:33 ` Xen
3 siblings, 1 reply; 17+ messages in thread
From: Zdenek Kabelac @ 2018-03-26 10:46 UTC (permalink / raw)
To: LVM general discussion and development, Gang He
Dne 26.3.2018 v 08:04 Gang He napsal(a):
> Hi Xen,
>
>
>>>>
>> Gang He schreef op 23-03-2018 9:30:
>>
>>> 6) attach disk2 to VM2(tb0307-nd2), the vg on VM2 looks abnormal.
>>> tb0307-nd2:~ # pvs
>>> WARNING: Device for PV JJOL4H-kc0j-jyTD-LDwl-71FZ-dHKM-YoFtNV not
>>> found or rejected by a filter.
>>> PV VG Fmt Attr PSize PFree
>>> /dev/vdc vg2 lvm2 a-- 20.00g 20.00g
>>> /dev/vdd vg1 lvm2 a-- 20.00g 20.00g
>>> [unknown] vg1 lvm2 a-m 20.00g 20.00g
>>
>> This is normal because /dev/vdd contains metadata for vg1 which includes
>> now missing disk /dev/vdc .... as the PV is no longer the same.
>>
>>
>>
>>
>>> tb0307-nd2:~ # vgs
>>> WARNING: Device for PV JJOL4H-kc0j-jyTD-LDwl-71FZ-dHKM-YoFtNV not
>>> found or rejected by a filter.
>>> VG #PV #LV #SN Attr VSize VFree
>>> vg1 2 0 0 wz-pn- 39.99g 39.99g
>>> vg2 1 0 0 wz--n- 20.00g 20.00g
>>
>> This is normal because you haven't removed /dev/vdc from vg1 on
>> /dev/vdd, since it was detached while you operated on its vg.
>>
>>
>>> 7) reboot VM2, the result looks worse (vdc disk belongs to two vg).
>>> tb0307-nd2:/mnt/shared # pvs
>>> PV VG Fmt Attr PSize PFree
>>> /dev/vdc vg1 lvm2 a-- 20.00g 0
>>> /dev/vdc vg2 lvm2 a-- 20.00g 10.00g
>>> /dev/vdd vg1 lvm2 a-- 20.00g 9.99g
>>
>> When you removed vdd when it was not attached, the VG1 metadata on vdd
>> was not altered. The metadata resides on both disks, so you had
>> inconsistent metadata between both disks because you operated on the
>> shared volume group while one device was missing.
>>
>> You also did not recreate PV on /dev/vdc so it has the same UUID as when
>> it was part of VG1, this is why VG1 when VDD is booted will still try to
>> include /dev/vdc because it was never removed from the volume group on
>> VDD.
>>
>> So the state of affairs is:
>>
>> /dev/vdc contains volume group info for VG2 and includes only /dev/vdc
>>
>> /dev/vdd contains volume group info for VG1, and includes both /dev/vdc
>> and /dev/vdd by UUID for its PV, however, it is a bug that it should
>> include /dev/vdc even though the VG UUID is now different (and the name
>> as well).
> It looks like each PV includes a copy meta data for VG, but if some PV has changed (e.g. removed, or moved to another VG),
> the remained PV should have a method to check the integrity when each startup (activated?), to avoid such inconsistent problem automatically.
>
>
Hi
I'm not really sure what are you trying to achieve - are you 'validating' that
you cannot foolish lvm2 too easily or something else ?
Simple case is when you have a VG on 2 PV disks. Both PV hold full metadata
for a VG. There are numerous other case - i.e. you can have 1000PVs in single
VG then any update of metadata would require to update 1000 disks - for this
case you can select lower number metadata copies - i.e. randomly or
user-selected PVs only hold VG metadata and rest of PV are without metadata.
The less metadata copies - the less secure it is, but update is faster...
There are no metadata for use stored in your filesystem - VG metadata are
always recorded in PV metadata area.
1.) So when you 1st. remove device and then you run 'pvremove' on this missing
disk, it's kind of pointless operation.
2.) lvm2 command will not let you 'easily' remove PV which is in use by some
LV in your VG
3.) lvm2 supports 2 commands:
'vgreduce --removemissing' (try to make consistent VG when PV is lost)
'vgextend --restoeremissing' (restore missing PV back into VG)
Regards
Zdenek
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [linux-lvm] Can't work normally after attaching disk volumes originally in a VG on another machine
2018-03-26 6:04 ` Gang He
` (2 preceding siblings ...)
2018-03-26 10:46 ` Zdenek Kabelac
@ 2018-03-26 11:33 ` Xen
3 siblings, 0 replies; 17+ messages in thread
From: Xen @ 2018-03-26 11:33 UTC (permalink / raw)
To: linux-lvm
Gang He schreef op 26-03-2018 8:04:
> It looks like each PV includes a copy meta data for VG, but if some PV
> has changed (e.g. removed, or moved to another VG),
> the remained PV should have a method to check the integrity when each
> startup (activated?), to avoid such inconsistent problem
> automatically.
Well I'm not really interested in "what should be", I just answered your
question.
Also automatic resolution of differences turns into an automatic (or
bidirectional) "sync" problem which generally depends on consistent time
AND an agreed-upon strategy for merging conflicts which I doubt is truly
a realistic proposal because you are talking about data loss here.
I'm sorry, I don't think (I'm not an LVM developer here) what you ask
for is theoretically [im]possible.
At best you could have a "best merge" strategy with all associated
risks, and I don't think you should even go that route, honestly.
It's more something for an individual administrator to solve with
appropriate tools.
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [linux-lvm] Can't work normally after attaching disk volumes originally in a VG on another machine
2018-03-26 10:23 ` Fran Garcia
@ 2018-03-27 5:55 ` Gang He
2018-03-27 8:32 ` Zdenek Kabelac
2018-03-27 9:12 ` Xen
0 siblings, 2 replies; 17+ messages in thread
From: Gang He @ 2018-03-27 5:55 UTC (permalink / raw)
To: linux-lvm
Hi Fran,
>>>
> On 26 March 2018 at 08:04, Gang He <ghe@suse.com> wrote:
>> It looks like each PV includes a copy meta data for VG, but if some PV has
> changed (e.g. removed, or moved to another VG),
>> the remained PV should have a method to check the integrity when each
> startup (activated?), to avoid such inconsistent problem automatically.
>
> Your workflow is strange. What are you trying to accomplish here?
I just reproduced a problem from the customer, since they did virtual disk migration from one virtual machine to another one.
According to your comments, this does not look like a LVM code problem,
the problem can be considered as LVM administer misoperation?
Thanks
Gang
>
> Your steps in 5 should be:
>
> vgreduce vg01 /dev/vdc /dev/vdc
> pvremove /dev/vdc /dev/vdd
>
> That way you ensure there's no leftover metadata in the PVs (specially
> if you need to attach those disks to a different system)
>
> Again a usecase to understand your workflow would be beneficial...
>
> Cheers
>
> fran
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [linux-lvm] Can't work normally after attaching disk volumes originally in a VG on another machine
2018-03-27 5:55 ` Gang He
@ 2018-03-27 8:32 ` Zdenek Kabelac
2018-03-28 2:09 ` Gang He
2018-03-27 9:12 ` Xen
1 sibling, 1 reply; 17+ messages in thread
From: Zdenek Kabelac @ 2018-03-27 8:32 UTC (permalink / raw)
To: LVM general discussion and development, Gang He
Dne 27.3.2018 v 07:55 Gang He napsal(a):
> Hi Fran,
>
>
>>>>
>> On 26 March 2018 at 08:04, Gang He <ghe@suse.com> wrote:
>>> It looks like each PV includes a copy meta data for VG, but if some PV has
>> changed (e.g. removed, or moved to another VG),
>>> the remained PV should have a method to check the integrity when each
>> startup (activated?), to avoid such inconsistent problem automatically.
>>
>> Your workflow is strange. What are you trying to accomplish here?
> I just reproduced a problem from the customer, since they did virtual disk migration from one virtual machine to another one.
> According to your comments, this does not look like a LVM code problem,
> the problem can be considered as LVM administer misoperation?
>
> Thanks
> Gang
Ahh, so welcome Eric's replacement :)
Yes - this use scenario was improper usage of lvm2 - and lvm2 has cached the
user before he could ruing his data any further...
Regards
Zdenek
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [linux-lvm] Can't work normally after attaching disk volumes originally in a VG on another machine
2018-03-27 5:55 ` Gang He
2018-03-27 8:32 ` Zdenek Kabelac
@ 2018-03-27 9:12 ` Xen
2018-03-27 10:22 ` Zdenek Kabelac
1 sibling, 1 reply; 17+ messages in thread
From: Xen @ 2018-03-27 9:12 UTC (permalink / raw)
To: linux-lvm
Gang He schreef op 27-03-2018 7:55:
> I just reproduced a problem from the customer, since they did virtual
> disk migration from one virtual machine to another one.
> According to your comments, this does not look like a LVM code problem,
> the problem can be considered as LVM administer misoperation?
Counterintuitively, you must remove the PV from the VG before you remove
the (physical) disk from the system.
Yes that is something you can often forget doing, but as it stands
resolving the situation often becomes a lot harder when you do it in
reverse.
Ie. removing the disk first and then removing the PV from the VG is a
lot harder.
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [linux-lvm] Can't work normally after attaching disk volumes originally in a VG on another machine
2018-03-27 9:12 ` Xen
@ 2018-03-27 10:22 ` Zdenek Kabelac
2018-03-27 10:27 ` Xen
0 siblings, 1 reply; 17+ messages in thread
From: Zdenek Kabelac @ 2018-03-27 10:22 UTC (permalink / raw)
To: LVM general discussion and development, Xen
Dne 27.3.2018 v 11:12 Xen napsal(a):
> Gang He schreef op 27-03-2018 7:55:
>
>> I just reproduced a problem from the customer, since they did virtual
>> disk migration from one virtual machine� to another one.
>> According to your comments, this does not look like a LVM code problem,
>> the problem can be considered as LVM administer misoperation?
>
> Counterintuitively, you must remove the PV from the VG before you remove the
> (physical) disk from the system.
>
> Yes that is something you can often forget doing, but as it stands resolving
> the situation often becomes a lot harder when you do it in reverse.
>
> Ie. removing the disk first and then removing the PV from the VG is a lot harder.
>
IMO 'vgreduce --removemissing' doesn't look to me like a real rocket science.
Of course if the space on a missing PV has been in-use with some LVs - it gets
a bit more complex - but still as long as you don't care about data adding
'--force' option should usually help (although there can be some room for
improvement - but every coding takes its time - and there are not to many
hands working on this...)
Regards
Zdenek
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [linux-lvm] Can't work normally after attaching disk volumes originally in a VG on another machine
2018-03-27 10:22 ` Zdenek Kabelac
@ 2018-03-27 10:27 ` Xen
2018-03-27 22:17 ` Zdenek Kabelac
0 siblings, 1 reply; 17+ messages in thread
From: Xen @ 2018-03-27 10:27 UTC (permalink / raw)
To: linux-lvm
Zdenek Kabelac schreef op 27-03-2018 12:22:
> IMO 'vgreduce --removemissing' doesn't look to me like a real rocket
> science.
Yeah I don't wanna get into it.
--force didn't work very well when the missing PV was a cache PV as it
removed the entire cached volume including the origin.
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [linux-lvm] Can't work normally after attaching disk volumes originally in a VG on another machine
2018-03-27 10:27 ` Xen
@ 2018-03-27 22:17 ` Zdenek Kabelac
2018-03-28 10:08 ` Xen
0 siblings, 1 reply; 17+ messages in thread
From: Zdenek Kabelac @ 2018-03-27 22:17 UTC (permalink / raw)
To: LVM general discussion and development, Xen
Dne 27.3.2018 v 12:27 Xen napsal(a):
> Zdenek Kabelac schreef op 27-03-2018 12:22:
>
>> IMO 'vgreduce --removemissing'� doesn't look to me like a real rocket science.
>
> Yeah I don't wanna get into it.
>
> --force didn't work very well when the missing PV was a cache PV as it removed
> the entire cached volume including the origin.
Hi
This is why users do open BZ if they would like to see some enhancement.
Normally cache is integral part of a volume - so it's partially missing -
whole volume is considered to be garbage.
But in 'writethrough' mode there could be likely possible better recovery.
Of course this case needs usability without --force.
So please open RFE BZ for this case.
Regards
Zdenek
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [linux-lvm] Can't work normally after attaching disk volumes originally in a VG on another machine
2018-03-27 8:32 ` Zdenek Kabelac
@ 2018-03-28 2:09 ` Gang He
0 siblings, 0 replies; 17+ messages in thread
From: Gang He @ 2018-03-28 2:09 UTC (permalink / raw)
To: linux-lvm, zkabelac
Hi Zdenek,
>>>
> Dne 27.3.2018 v 07:55 Gang He napsal(a):
>> Hi Fran,
>>
>>
>>>>>
>>> On 26 March 2018 at 08:04, Gang He <ghe@suse.com> wrote:
>>>> It looks like each PV includes a copy meta data for VG, but if some PV has
>>> changed (e.g. removed, or moved to another VG),
>>>> the remained PV should have a method to check the integrity when each
>>> startup (activated?), to avoid such inconsistent problem automatically.
>>>
>>> Your workflow is strange. What are you trying to accomplish here?
>> I just reproduced a problem from the customer, since they did virtual disk
> migration from one virtual machine to another one.
>> According to your comments, this does not look like a LVM code problem,
>> the problem can be considered as LVM administer misoperation?
>>
>> Thanks
>> Gang
>
>
> Ahh, so welcome Eric's replacement :)
Yes, thank for your inputs.
>
> Yes - this use scenario was improper usage of lvm2 - and lvm2 has cached the
> user before he could ruing his data any further...
>
>
> Regards
>
>
> Zdenek
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [linux-lvm] Can't work normally after attaching disk volumes originally in a VG on another machine
2018-03-27 22:17 ` Zdenek Kabelac
@ 2018-03-28 10:08 ` Xen
0 siblings, 0 replies; 17+ messages in thread
From: Xen @ 2018-03-28 10:08 UTC (permalink / raw)
To: linux-lvm
Zdenek Kabelac schreef op 28-03-2018 0:17:
> Hi
>
> This is why users do open BZ if they would like to see some
> enhancement.
>
> Normally cache is integral part of a volume - so it's partially
> missing - whole volume is considered to be garbage.
>
> But in 'writethrough' mode there could be likely possible better
> recovery.
>
> Of course this case needs usability without --force.
>
> So please open RFE BZ for this case.
It goes into the mess I usually get myself into; if you "dd copy" the
disk containing the origin volume before uncaching it, and then go to
some live session where you only have the new backup copy, but you want
to clean up its LVM,
then you now must fix the VGs in isolation of the cache; I suppose this
is just the wrong order of doing things, but as part of a backup you
don't really want to uncache first, as that requires more work to get it
back to normal after.
So you end up in a situation where the new origin copy has a reference
to the cache disk --- all of this assumes writethrough mode --- and you
need to clear that reference.
However, you cannot, or should not, attach the cache disk again; it
might get effected, and you don't want that, you want it to remain in
its pristine state.
Therefore, you are now left with the task of removing the cache from the
VG, because you cannot actually run vgimportclone while the cache disk
is missing.
The obvious solution is to *also* clone the cache disk and then run
operations on the combined set, but this might not be possible.
Therefore, all that was left was:
vgreduce --remove-missing --force
cd /etc/lvm/archive
cp <latest backup> /etc/lvm/backup/<vg>
cd /etc/lvm/backup
vi <vg>
" remove cache PV, and change origin to regular linear volume, and add
" the visible tag
vgcfgrestore <vg>
# presto, origin is restored as regular volume without the cache
vgimportclone -i <vg> <bla>
# now have distinct volume group, VG UUID and PV UUID
So the problem is making dd backups of origin, perhaps dd backups should
be avoided, but for some purposes (such as system migration) file copies
are just
more work
in general, and can complicate things as well, for instance if there are
NTFS partitions or whatnot.
And disk images can be nice to have, in any case.
This was the use case basically.
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [linux-lvm] Can't work normally after attaching disk volumes originally in a VG on another machine
2018-03-26 10:46 ` Zdenek Kabelac
@ 2018-05-18 4:56 ` Gang He
0 siblings, 0 replies; 17+ messages in thread
From: Gang He @ 2018-05-18 4:56 UTC (permalink / raw)
To: linux-lvm
Hi Zdenek and Guys,
As you know, we prefer to consider this problem as mis-operation.
But, the customer want to know if we should add a warning in this case?
Any comments for this customer's request?
Thanks
Gang
>>> Zdenek Kabelac <zkabelac@redhat.com> 2018/3/26 18:46 >>>
Dne 26.3.2018 v 08:04 Gang He napsal(a):
> Hi Xen,
>
>
>>>>
>> Gang He schreef op 23-03-2018 9:30:
>>
>>> 6) attach disk2 to VM2(tb0307-nd2), the vg on VM2 looks abnormal.
>>> tb0307-nd2:~ # pvs
>>> WARNING: Device for PV JJOL4H-kc0j-jyTD-LDwl-71FZ-dHKM-YoFtNV not
>>> found or rejected by a filter.
>>> PV VG Fmt Attr PSize PFree
>>> /dev/vdc vg2 lvm2 a-- 20.00g 20.00g
>>> /dev/vdd vg1 lvm2 a-- 20.00g 20.00g
>>> [unknown] vg1 lvm2 a-m 20.00g 20.00g
>>
>> This is normal because /dev/vdd contains metadata for vg1 which includes
>> now missing disk /dev/vdc .... as the PV is no longer the same.
>>
>>
>>
>>
>>> tb0307-nd2:~ # vgs
>>> WARNING: Device for PV JJOL4H-kc0j-jyTD-LDwl-71FZ-dHKM-YoFtNV not
>>> found or rejected by a filter.
>>> VG #PV #LV #SN Attr VSize VFree
>>> vg1 2 0 0 wz-pn- 39.99g 39.99g
>>> vg2 1 0 0 wz--n- 20.00g 20.00g
>>
>> This is normal because you haven't removed /dev/vdc from vg1 on
>> /dev/vdd, since it was detached while you operated on its vg.
>>
>>
>>> 7) reboot VM2, the result looks worse (vdc disk belongs to two vg).
>>> tb0307-nd2:/mnt/shared # pvs
>>> PV VG Fmt Attr PSize PFree
>>> /dev/vdc vg1 lvm2 a-- 20.00g 0
>>> /dev/vdc vg2 lvm2 a-- 20.00g 10.00g
>>> /dev/vdd vg1 lvm2 a-- 20.00g 9.99g
>>
>> When you removed vdd when it was not attached, the VG1 metadata on vdd
>> was not altered. The metadata resides on both disks, so you had
>> inconsistent metadata between both disks because you operated on the
>> shared volume group while one device was missing.
>>
>> You also did not recreate PV on /dev/vdc so it has the same UUID as when
>> it was part of VG1, this is why VG1 when VDD is booted will still try to
>> include /dev/vdc because it was never removed from the volume group on
>> VDD.
>>
>> So the state of affairs is:
>>
>> /dev/vdc contains volume group info for VG2 and includes only /dev/vdc
>>
>> /dev/vdd contains volume group info for VG1, and includes both /dev/vdc
>> and /dev/vdd by UUID for its PV, however, it is a bug that it should
>> include /dev/vdc even though the VG UUID is now different (and the name
>> as well).
> It looks like each PV includes a copy meta data for VG, but if some PV has changed (e.g. removed, or moved to another VG),
> the remained PV should have a method to check the integrity when each startup (activated?), to avoid such inconsistent problem automatically.
>
>
Hi
I'm not really sure what are you trying to achieve - are you 'validating' that
you cannot foolish lvm2 too easily or something else ?
Simple case is when you have a VG on 2 PV disks. Both PV hold full metadata
for a VG. There are numerous other case - i.e. you can have 1000PVs in single
VG then any update of metadata would require to update 1000 disks - for this
case you can select lower number metadata copies - i.e. randomly or
user-selected PVs only hold VG metadata and rest of PV are without metadata.
The less metadata copies - the less secure it is, but update is faster...
There are no metadata for use stored in your filesystem - VG metadata are
always recorded in PV metadata area.
1.) So when you 1st. remove device and then you run 'pvremove' on this missing
disk, it's kind of pointless operation.
2.) lvm2 command will not let you 'easily' remove PV which is in use by some
LV in your VG
3.) lvm2 supports 2 commands:
'vgreduce --removemissing' (try to make consistent VG when PV is lost)
'vgextend --restoeremissing' (restore missing PV back into VG)
Regards
Zdenek
_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [linux-lvm] [lvm-devel] Can't work normally after attaching disk volumes originally in a VG on another machine
2018-03-23 8:30 [linux-lvm] Can't work normally after attaching disk volumes originally in a VG on another machine Gang He
2018-03-23 9:04 ` Xen
@ 2018-10-16 20:59 ` Nir Soffer
1 sibling, 0 replies; 17+ messages in thread
From: Nir Soffer @ 2018-10-16 20:59 UTC (permalink / raw)
To: lvm-devel; +Cc: linux-lvm
[-- Attachment #1: Type: text/plain, Size: 4421 bytes --]
On Fri, Mar 23, 2018 at 11:30 AM Gang He <ghe@suse.com> wrote:
> Hello List,
>
> Since I am new to LVM area, but I got a problem about LVM can't work
> normally after attaching disk volumes originally in a VG on another machine.
> Then, I sent this problem to the list, to see if we can get a fix or there
> has been a fix.
> The problem is very easy to reproduce,
> 1) create two virtual machines(tb0307-nd1, tb0307-nd2), create two virtual
> disks.
>
Do you mean 2 logical volumes on the host?
> 2) attach two virtual disks to VM1(tb0307-nd1).
>
Are you attaching the logical volumes created on the host to the VM?
> tb0307-nd1:/# lsblk
> NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
> sr0 11:0 1 4.3G 0 rom
> vda 254:0 0 40G 0 disk
> ├─vda1 254:1 0 4G 0 part [SWAP]
> ├─vda2 254:2 0 23.6G 0 part /
> └─vda3 254:3 0 12.4G 0 part /home
> vdb 254:16 0 40G 0 disk
> vdc 254:32 0 20G 0 disk <<= disk1
> vdd 254:48 0 20G 0 disk <<= disk2
> 3) pvcreate two virtual disks, and add them to vg1.
> tb0307-nd1:/# pvcreate /dev/vdc
> Physical volume "/dev/vdc" successfully created.
> tb0307-nd1:/# pvcreate /dev/vdd
> Physical volume "/dev/vdd" successfully created.
> tb0307-nd1:/# pvs
> PV VG Fmt Attr PSize PFree
> /dev/vdc lvm2 --- 20.00g 20.00g
> /dev/vdd lvm2 --- 20.00g 20.00g
> tb0307-nd1:/# vgcreate vg1 /dev/vdc /dev/vdd
> 4) disconnect two virtual disks from VM1, and attach disk1 to
> VM2(tb0307-nd2).
>
If /dev/vdc and /dev/vdd are logical volumes on the host, lvm on the host
just scanned these lvs and discovered the metadata created within the VM.
> 5) delete vg1, create vg2, and add disk1 to vg2.
> tb0307-nd2:~ # pvs
> WARNING: Device for PV t6gjHU-dtY4-th5O-A0mr-5OEC-R1ZK-TPQgIh not found
> or rejected by a filter.
> PV VG Fmt Attr PSize PFree
> /dev/vdc vg1 lvm2 a-- 20.00g 20.00g
> [unknown] vg1 lvm2 a-m 20.00g 20.00g
> tb0307-nd2:~ # pvremove /dev/vdc
> WARNING: Device for PV t6gjHU-dtY4-th5O-A0mr-5OEC-R1ZK-TPQgIh not found
> or rejected by a filter.
> PV /dev/vdc is used by VG vg1 so please use vgreduce first.
> (If you are certain you need pvremove, then confirm by using --force
> twice.)
> /dev/vdc: physical volume label not removed.
> tb0307-nd2:~ # vgreduce --removemissing vg1
> WARNING: Device for PV t6gjHU-dtY4-th5O-A0mr-5OEC-R1ZK-TPQgIh not found
> or rejected by a filter.
> Wrote out consistent volume group vg1.
> tb0307-nd2:~ # pvs
> PV VG Fmt Attr PSize PFree
> /dev/vdc vg1 lvm2 a-- 20.00g 20.00g
> tb0307-nd2:~ # vgremove vg1
> Volume group "vg1" successfully removed
> tb0307-nd2:~ # vgcreate vg2 /dev/vdc
> Volume group "vg2" successfully created
> tb0307-nd2:~ # vgs
> VG #PV #LV #SN Attr VSize VFree
> vg2 1 0 0 wz--n- 20.00g 20.00g
> tb0307-nd2:~ # pvs
> PV VG Fmt Attr PSize PFree
> /dev/vdc vg2 lvm2 a-- 20.00g 20.00g
>
> 6) attach disk2 to VM2(tb0307-nd2), the vg on VM2 looks abnormal.
> tb0307-nd2:~ # pvs
> WARNING: Device for PV JJOL4H-kc0j-jyTD-LDwl-71FZ-dHKM-YoFtNV not found
> or rejected by a filter.
> PV VG Fmt Attr PSize PFree
> /dev/vdc vg2 lvm2 a-- 20.00g 20.00g
> /dev/vdd vg1 lvm2 a-- 20.00g 20.00g
> [unknown] vg1 lvm2 a-m 20.00g 20.00g
> tb0307-nd2:~ # vgs
> WARNING: Device for PV JJOL4H-kc0j-jyTD-LDwl-71FZ-dHKM-YoFtNV not found
> or rejected by a filter.
> VG #PV #LV #SN Attr VSize VFree
> vg1 2 0 0 wz-pn- 39.99g 39.99g
> vg2 1 0 0 wz--n- 20.00g 20.00g
>
> 7) reboot VM2, the result looks worse (vdc disk belongs to two vg).
> tb0307-nd2:/mnt/shared # pvs
> PV VG Fmt Attr PSize PFree
> /dev/vdc vg1 lvm2 a-- 20.00g 0
> /dev/vdc vg2 lvm2 a-- 20.00g 10.00g
> /dev/vdd vg1 lvm2 a-- 20.00g 9.99g
>
>
> It looks like LVM2 meta consistency problem? please help to take a look.
>
If you want to attach logical volumes to VMs, you must make sure that these
volumes are not accessible to LVM on the host. A good way to achieve this is
to setup devices:filter in lvm.conf so LVM can access only the disks
required
by the host (e.g. /dev/sda2).
Check this for more info on this problem:
https://ovirt.org/blog/2017/12/lvm-configuration-the-easy-way/
Nir
[-- Attachment #2: Type: text/html, Size: 5748 bytes --]
^ permalink raw reply [flat|nested] 17+ messages in thread
end of thread, other threads:[~2018-10-16 20:59 UTC | newest]
Thread overview: 17+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-03-23 8:30 [linux-lvm] Can't work normally after attaching disk volumes originally in a VG on another machine Gang He
2018-03-23 9:04 ` Xen
2018-03-26 6:04 ` Gang He
2018-03-26 10:17 ` Fran Garcia
2018-03-26 10:23 ` Fran Garcia
2018-03-27 5:55 ` Gang He
2018-03-27 8:32 ` Zdenek Kabelac
2018-03-28 2:09 ` Gang He
2018-03-27 9:12 ` Xen
2018-03-27 10:22 ` Zdenek Kabelac
2018-03-27 10:27 ` Xen
2018-03-27 22:17 ` Zdenek Kabelac
2018-03-28 10:08 ` Xen
2018-03-26 10:46 ` Zdenek Kabelac
2018-05-18 4:56 ` Gang He
2018-03-26 11:33 ` Xen
2018-10-16 20:59 ` [linux-lvm] [lvm-devel] " Nir Soffer
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).