[linux-lvm] file-based locking initialization failed after pvresize

[linux-lvm] file-based locking initialization failed after pvresize

[Christian Lahti]

Hello all:

I have a VMWare virtual CentOS 6 server with the following disk layout:

/dev/sda 15GB
/dev/sdb 100GB

Standard CentOS LVM setup

on sda1
/dev/vg_centos6/lv_root mounted on /
/dev/vg_centos6/lv_swap

on sdb1
/dev/vg_centos6/lv_var mounted on /var

I was starting to run low on disk space for /var, so I shut down the 
machine gracefully, extended the disk in VMWare to 250G and rebooted in 
single user mode. I then ran the following two commands:

pvresize --setphysicalvolumesize 250G /dev/sdb1
lvresize -L250G /dev/vg_centos6/lv_var

I then rebooted expecting the /var to now be 250G instead of 100G. What 
I got was:

No such file or directory trying to open /dev/vg_centos6/lv_var

After dropping to a maintenance shell I could not run any pv* or lv* 
commands, always resulting in the error:

File-based locking initialization failed

After Googling forever I found some lv* and pv* commands take 
--ignorelockingfailure as a parameter, so now I can "see" the pv and lv 
information but I do not know how to recover this partition:

pvdisplay --ignorelockingfailure /dev/sdb1
File-based locking initialization failed
--- Physical volume ---
PV Name /dev/sdb1
VG Name vg_centos6
PV Size 250 GiB / not useable 3.00 MiB
Allocatable yes
PE Size 4.00 MiB
Total PE 63999
Free PE 25599
Allocated PE 38400

lvdisplay --ignorelockingfailure /dev/vg_centos6/lv_var
File-based locking initialization failed
--- Logical volume ---
LV Path /dev/vg_centos6/lv_var
LV Name lv_var
VG Name vg_centos6
LV Write Access read/write
LV Status suspended
# open 0
LV Size 150.00 GiB
Current LE 38400
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:2

I cannot mount this, I cannot fsck, /dev/mapper/vg_centos6/lv_var does 
not even get created. This is a critical server, can anyone please tell 
me how to reverse this or at least recover the data?

Thanks!

/Christian

[linux-lvm] file-based locking initialization failed after pvresize

[Christian Lahti]

Hello all:

I have a VMWare virtual CentOS 6 server with the following disk layout:

/dev/sda 15GB
/dev/sdb 100GB

Standard CentOS LVM setup

on sda1
/dev/vg_centos6/lv_root mounted on /
/dev/vg_centos6/lv_swap

on sdb1
/dev/vg_centos6/lv_var mounted on /var

I was starting to run low on disk space for /var, so I shut down the
machine gracefully, extended the disk in VMWare to 250G and rebooted in
single user mode. I then ran the following two commands:

pvresize --setphysicalvolumesize 250G /dev/sdb1
lvresize -L250G /dev/vg_centos6/lv_var

I then rebooted expecting the /var to now be 250G instead of 100G. What
I got was:

No such file or directory trying to open /dev/vg_centos6/lv_var

After dropping to a maintenance shell I could not run any pv* or lv*
commands, always resulting in the error:

File-based locking initialization failed

After Googling forever I found some lv* and pv* commands take
--ignorelockingfailure as a parameter, so now I can "see" the pv and lv
information but I do not know how to recover this partition:

pvdisplay --ignorelockingfailure /dev/sdb1
File-based locking initialization failed
--- Physical volume ---
PV Name /dev/sdb1
VG Name vg_centos6
PV Size 250 GiB / not useable 3.00 MiB
Allocatable yes
PE Size 4.00 MiB
Total PE 63999
Free PE 25599
Allocated PE 38400

lvdisplay --ignorelockingfailure /dev/vg_centos6/lv_var
File-based locking initialization failed
--- Logical volume ---
LV Path /dev/vg_centos6/lv_var
LV Name lv_var
VG Name vg_centos6
LV Write Access read/write
LV Status suspended
# open 0
LV Size 150.00 GiB
Current LE 38400
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:2

I cannot mount this, I cannot fsck, /dev/mapper/vg_centos6/lv_var does
not even get created. This is a critical server, can anyone please tell
me how to reverse this or at least recover the data?

Thanks!

/Christian

[linux-lvm] Fwd: file-based locking initialization failed after pvresize

[Christian Lahti]

[-- Attachment #1: Type: text/plain, Size: 1860 bytes --]

Sorry forgot to copy the list.

---------- Forwarded message ----------
From: Christian Lahti <clahti@gmail.com>
Date: Thu, May 30, 2013 at 8:36 PM
Subject: Re: [linux-lvm] file-based locking initialization failed after
pvresize
To: matthew patton <pattonme@yahoo.com>


Thank you for your reply, I realize that I must have done something rather
bone-headed and I am usually pretty clued.  That being said I am still in
the same pickle I was in to begin with, something constructive would be
fantastically appreciated.  If there were proper documentation *anywhere*
google-able I would not have bothered the mailing list.  I am pretty sure
the data must be in tact since this was done in single user mode and no
writes to the disk/volume have occurred since the two commands were run, I
have /etc/lvm in tact with information prior to and after the event.  If
someone would be so kind as to please let me know exactly what information
they might need, I will do my level best with minimum "back and forth" to
provide such information to get out of said pickle.  And I furthermore
promise not to do it again.

/Christian



On Thu, May 30, 2013 at 3:14 PM, matthew patton <pattonme@yahoo.com> wrote:

> What's with people and "extending" disks? DO NOT DO THAT!!!! But if you
> do, then you can't forget to do all the rest of the necessary steps. You
> have to fix the partition table. And you have to make sure your PVresize
> doesn't run off the end of the media. Is VMware's 250G really 250G when
> probed by PVscan? Doubt it very much.
>
> Next time don't partition disks for starters. And "grow" filesystems by
> ADDing disks and adding them to the volume group and then resize2fs. Just
> because Windows was written for dummies and lets you get away with what you
> did, doesn't mean you should attempt it in Linux if you don't know what
> you're doing.
>

[-- Attachment #2: Type: text/html, Size: 2603 bytes --]

Re: [linux-lvm] Fwd: file-based locking initialization failed after pvresize

[Zdenek Kabelac]

Dne 31.5.2013 05:37, Christian Lahti napsal(a):
> Sorry forgot to copy the list.
>
> ---------- Forwarded message ----------
> From: *Christian Lahti* <clahti@gmail.com <mailto:clahti@gmail.com>>
> Date: Thu, May 30, 2013 at 8:36 PM
> Subject: Re: [linux-lvm] file-based locking initialization failed after pvresize
> To: matthew patton <pattonme@yahoo.com <mailto:pattonme@yahoo.com>>
>
>
> Thank you for your reply, I realize that I must have done something rather
> bone-headed and I am usually pretty clued.  That being said I am still in the
> same pickle I was in to begin with, something constructive would be
> fantastically appreciated.  If there were proper documentation *anywhere*
> google-able I would not have bothered the mailing list.  I am pretty sure the
> data must be in tact since this was done in single user mode and no writes to
> the disk/volume have occurred since the two commands were run, I have /etc/lvm
> in tact with information prior to and after the event.  If someone would be so
> kind as to please let me know exactly what information they might need, I will
> do my level best with minimum "back and forth" to provide such information to
> get out of said pickle.  And I furthermore promise not to do it again.

I'm mostly convinced you've probably already made some irreparable damage to 
your disk data - since the story is usually - Data are intact - I've done only 
this and that :) - but anyway it's probably better trying to resolve this 
interactively on #lvm irc channel.

The easiest way is to restore to the previous version of your metadata,
just before you've started your pvresize command  (which must have been loudly 
objecting to your command line options)

Also next time - when the tool is giving you WARNINGS - and you have no idea,
what it means - it's not a sign to go forward, but to STOP immediately...

Regards

Zdenek

Re: [linux-lvm] Fwd: file-based locking initialization failed after pvresize

[Gabriel Barazer]

[-- Attachment #1: Type: text/plain, Size: 3426 bytes --]

Hi,

Did you try booting off another system (e.g. live usb ) and restoring the LVM meta data ? If your root filesystem is still bootable, you can find a backup of the LVM metadata in /etc/lvm/backup, and then start over your data resizing procedure. Look for the vgcfgrestore command.

The right procedure to resize a LV after growing a PV is :
- adjust the partition table on sdb with fdisk, the simple way is deleting the partition table and creating a new one with the main partition having the same start offset (_very_ important)
- THEN pvresize the device (no need to add any --setphysicalwhatever argument)
- then lvresize, but if you want to be safe always check that the size you set is larger than the current volume size otherwise you can damage your data pretty bad. A good way to do this is to add a "+" in the lvresize command : lvresize -L +100G /device adds 100G to volume.
- then run a filesystem resize command, depending on the filesystem used (xfs_growfs for XFS, resize2fs for ext4)

Anyway, If you are going to use full disks with LVM, it's always easier to not even bother creating a partition table : use the whole device with LVM (/dev/sdX) instead of a partition (/dev/sdX1).

HTH,

Gabriel


On May 31, 2013, at 5:37 AM, Christian Lahti <clahti@gmail.com> wrote:

> Sorry forgot to copy the list.
> 
> ---------- Forwarded message ----------
> From: Christian Lahti <clahti@gmail.com>
> Date: Thu, May 30, 2013 at 8:36 PM
> Subject: Re: [linux-lvm] file-based locking initialization failed after pvresize
> To: matthew patton <pattonme@yahoo.com>
> 
> 
> Thank you for your reply, I realize that I must have done something rather bone-headed and I am usually pretty clued.  That being said I am still in the same pickle I was in to begin with, something constructive would be fantastically appreciated.  If there were proper documentation *anywhere* google-able I would not have bothered the mailing list.  I am pretty sure the data must be in tact since this was done in single user mode and no writes to the disk/volume have occurred since the two commands were run, I have /etc/lvm in tact with information prior to and after the event.  If someone would be so kind as to please let me know exactly what information they might need, I will do my level best with minimum "back and forth" to provide such information to get out of said pickle.  And I furthermore promise not to do it again.
> 
> /Christian
> 
> 
> 
> On Thu, May 30, 2013 at 3:14 PM, matthew patton <pattonme@yahoo.com> wrote:
> What's with people and "extending" disks? DO NOT DO THAT!!!! But if you do, then you can't forget to do all the rest of the necessary steps. You have to fix the partition table. And you have to make sure your PVresize doesn't run off the end of the media. Is VMware's 250G really 250G when probed by PVscan? Doubt it very much.
> 
> Next time don't partition disks for starters. And "grow" filesystems by ADDing disks and adding them to the volume group and then resize2fs. Just because Windows was written for dummies and lets you get away with what you did, doesn't mean you should attempt it in Linux if you don't know what you're doing.
> 
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/


[-- Attachment #2: Type: text/html, Size: 4645 bytes --]

Re: [linux-lvm] file-based locking initialization failed after pvresize

[service hofman]

[-- Attachment #1: Type: text/plain, Size: 631 bytes --]

I am a linux expert

all the problems associated with the LVM
Linux systems are reliable.
LVM is not a problem, but your system and kernel
clean kernel.org occurs only in linux.
this system are:
kernel.org, debian, knopix,
and those that benefit from the source.

systems:
Ubuntu, CentOS, mac.OS, and all VirtualMachine type XEN.
not equipped with LVM
 only virtual vLVM
This is your fault,
because your system is part of the
IBM, AIX kernel

AIX kernel is not free, so dumb access to their source.

who would like to have a stable system

Contact service.hofman@gmail.com
https://plus.google.com/105547044162966304899

linux free

[-- Attachment #2: Type: text/html, Size: 919 bytes --]

Re: [linux-lvm] Fwd: file-based locking initialization failed after pvresize

[Zdenek Kabelac]

Dne 31.5.2013 11:04, Gabriel Barazer napsal(a):
> Hi,
>
> Did you try booting off another system (e.g. live usb ) and restoring the LVM
> meta data ? If your root filesystem is still bootable, you can find a backup
> of the LVM metadata in /etc/lvm/backup, and then start over your data resizing
> procedure. Look for the vgcfgrestore command.
>
> The right procedure to resize a LV after growing a PV is :
> - adjust the partition table on sdb with fdisk, the simple way is deleting the
> partition table and creating a new one with the main partition having the same
> start offset (_very_ important)
> - THEN pvresize the device (no need to add any --setphysicalwhatever argument)
> - then lvresize, but if you want to be safe always check that the size you set
> is larger than the current volume size otherwise you can damage your data
> pretty bad. A good way to do this is to add a "+" in the lvresize command :
> lvresize -L +100G /device adds 100G to volume.
> - then run a filesystem resize command, depending on the filesystem used
> (xfs_growfs for XFS, resize2fs for ext4)

For couple years lvresize has the option '-r'  so it knows how to resize 
common filesystems (via fsadm script) - so it keeps the right order of 
commands (very important in case you are reducing size)

Zdenek

Re: [linux-lvm] file-based locking initialization failed after pvresize

[Christian Lahti]

[-- Attachment #1: Type: text/plain, Size: 2784 bytes --]

I was assisted on the #lvm IRC channel by kabi_ (thank you), the 
solution for others who might find themselves in the same boat:

  * look in /etc/lvm/archive and find the file that has the information
    just prior to the destructive act
  * in this text file is a UUID for the volume that has been corrupted
  * boot into rescue disk (or single user mode if the corrupted disk is
    not your boot volume) and run the following
    pvcreate --config 'global{locking_type=0}' --uuid <UUID>
    --restorefile /etc/lvm/backup/<file containing previous info>
    /dev/<device>
    vgcfgrestore -f /etc/lvm/archive/<file containing previous info>
    <volume group name>

Thanks again!

/Christian

On 5/30/2013 11:48 AM, Christian Lahti wrote:
> Hello all:
>
> I have a VMWare virtual CentOS 6 server with the following disk layout:
>
> /dev/sda 15GB
> /dev/sdb 100GB
>
> Standard CentOS LVM setup
>
> on sda1
> /dev/vg_centos6/lv_root mounted on /
> /dev/vg_centos6/lv_swap
>
> on sdb1
> /dev/vg_centos6/lv_var mounted on /var
>
> I was starting to run low on disk space for /var, so I shut down the 
> machine gracefully, extended the disk in VMWare to 250G and rebooted 
> in single user mode. I then ran the following two commands:
>
> pvresize --setphysicalvolumesize 250G /dev/sdb1
> lvresize -L250G /dev/vg_centos6/lv_var
>
> I then rebooted expecting the /var to now be 250G instead of 100G. 
> What I got was:
>
> No such file or directory trying to open /dev/vg_centos6/lv_var
>
> After dropping to a maintenance shell I could not run any pv* or lv* 
> commands, always resulting in the error:
>
> File-based locking initialization failed
>
> After Googling forever I found some lv* and pv* commands take 
> --ignorelockingfailure as a parameter, so now I can "see" the pv and 
> lv information but I do not know how to recover this partition:
>
> pvdisplay --ignorelockingfailure /dev/sdb1
> File-based locking initialization failed
> --- Physical volume ---
> PV Name /dev/sdb1
> VG Name vg_centos6
> PV Size 250 GiB / not useable 3.00 MiB
> Allocatable yes
> PE Size 4.00 MiB
> Total PE 63999
> Free PE 25599
> Allocated PE 38400
>
> lvdisplay --ignorelockingfailure /dev/vg_centos6/lv_var
> File-based locking initialization failed
> --- Logical volume ---
> LV Path /dev/vg_centos6/lv_var
> LV Name lv_var
> VG Name vg_centos6
> LV Write Access read/write
> LV Status suspended
> # open 0
> LV Size 150.00 GiB
> Current LE 38400
> Segments 1
> Allocation inherit
> Read ahead sectors auto
> - currently set to 256
> Block device 253:2
>
> I cannot mount this, I cannot fsck, /dev/mapper/vg_centos6/lv_var does 
> not even get created. This is a critical server, can anyone please 
> tell me how to reverse this or at least recover the data?
>
> Thanks!
>
> /Christian


[-- Attachment #2: Type: text/html, Size: 4225 bytes --]