[linux-lvm] Dead device and cleaning up LVM

[linux-lvm] Dead device and cleaning up LVM

[Ray Van Dolson]

I have a volgroup that was created on top of an iSCSI mounted block
device (target was stgt on RHEL 5.3 and initiator is the default iSCSI
initiator in RHEL 4.7).

Through a goof the target went offline and I tried shutting down the
initiator software before disabling the volume group on the initiator
with vgchange -an.  Now I'm left with a bit of a mess including iSCSI
initiator modules that won't completely unload.  I'm trying to avoid a
reboot and we have been theorizing that we somehow need to tell LVM to
"let go" of the remaining ghosts of the iSCSI device before things can
go back to normal.

The /dev/sdXX device is gone, but how can I convice LVM to quit trying
to use /dev/dm-2?  Can I edit the /etc/lvm/cache/.cache file and remove
entries there?

I'm seeing the following stuff in dmesg:

  scsi7 (0:1): rejecting I/O to dead device

And all the various pvscan/vgscan/lvscan tools show something similar
to the following:

  /dev/dm-2: read failed after 0 of 4096 at 4398038056960: Input/output error
  /dev/dm-2: read failed after 0 of 4096 at 4398038114304: Input/output error
  /dev/dm-2: read failed after 0 of 4096 at 0: Input/output error
  /dev/dm-2: read failed after 0 of 4096 at 4096: Input/output error
  /dev/dm-2: read failed after 0 of 4096 at 0: Input/output error

Anyone know how I can remove this device both from the system and from
LVM safely?  I can reboot, but saving that as a last resort if
possible.

Also, any best practices or suggestions for using iSCSI backed block
devices as volgroup members would be appreciated.  I'm worried about
LVM not recognizing the volume group of physical volumes correctly on
reboots...

Thanks,
Ray

Re: [linux-lvm] Dead device and cleaning up LVM

[Bryn M. Reeves]

Ray Van Dolson wrote:
> Anyone know how I can remove this device both from the system and from
> LVM safely?  I can reboot, but saving that as a last resort if
> possible.

You don't mention the name of your VG/LV but you should be able to 
remove the active lvm devices with "dmsetup remove", e.g.:

dmsetup remove $VG-$LV

For each LV in each VG that was using the iSCSI devices.

> Also, any best practices or suggestions for using iSCSI backed block
> devices as volgroup members would be appreciated.  I'm worried about
> LVM not recognizing the volume group of physical volumes correctly on
> reboots...

You might need to label any file system mounts in /etc/fstab with 
"_netdev" and enable the netfs service to have the VGs automatically 
activated/mounted at boot time.

Regards,
Bryn.

Re: [linux-lvm] Dead device and cleaning up LVM

[Ray Van Dolson]

On Mon, Feb 09, 2009 at 09:45:46AM -0800, Bryn M. Reeves wrote:
> Ray Van Dolson wrote:
> > Anyone know how I can remove this device both from the system and from
> > LVM safely?  I can reboot, but saving that as a last resort if
> > possible.
> 
> You don't mention the name of your VG/LV but you should be able to 
> remove the active lvm devices with "dmsetup remove", e.g.:
> 
> dmsetup remove $VG-$LV
> 
> For each LV in each VG that was using the iSCSI devices.

Beautiful.  This did the trick!  I can now also unload the iSCSI kernel
modules correctly on the initiator.

Thanks,
Ray

Re: [linux-lvm] Dead device and cleaning up LVM

[Ray Van Dolson]

On Mon, Feb 09, 2009 at 09:58:04AM -0800, Ray Van Dolson wrote:
> On Mon, Feb 09, 2009 at 09:45:46AM -0800, Bryn M. Reeves wrote:
> > Ray Van Dolson wrote:
> > > Anyone know how I can remove this device both from the system and from
> > > LVM safely?  I can reboot, but saving that as a last resort if
> > > possible.
> > 
> > You don't mention the name of your VG/LV but you should be able to 
> > remove the active lvm devices with "dmsetup remove", e.g.:
> > 
> > dmsetup remove $VG-$LV
> > 
> > For each LV in each VG that was using the iSCSI devices.
> 
> Beautiful.  This did the trick!  I can now also unload the iSCSI kernel
> modules correctly on the initiator.
> 

Also, just an FYI in case anyone else runs across this thread... to get
my filesystem back (pvscan didn't detect the LVM at all):

  # pvcreate --uuid 3HpAVg-fRz0-HWXp-EFxK-khVO-IvN6-J4a6w1 /dev/sdb
  (Found the old UUID by examining the /etc/lvm/backup directory)
  # vgcfgrestore FTP1
  # vgchange -ay FTP1
  # fsck.ext3 /dev/FTP1/LogVol00
  # mount -a

Back in action!

Ray

Re: [linux-lvm] Dead device and cleaning up LVM

[Bryn M. Reeves]

Ray Van Dolson wrote:
> On Mon, Feb 09, 2009 at 09:58:04AM -0800, Ray Van Dolson wrote:
>> On Mon, Feb 09, 2009 at 09:45:46AM -0800, Bryn M. Reeves wrote:
>>> Ray Van Dolson wrote:
>>>> Anyone know how I can remove this device both from the system and from
>>>> LVM safely?  I can reboot, but saving that as a last resort if
>>>> possible.
>>> You don't mention the name of your VG/LV but you should be able to 
>>> remove the active lvm devices with "dmsetup remove", e.g.:
>>>
>>> dmsetup remove $VG-$LV
>>>
>>> For each LV in each VG that was using the iSCSI devices.
>> Beautiful.  This did the trick!  I can now also unload the iSCSI kernel
>> modules correctly on the initiator.
>>
> 
> Also, just an FYI in case anyone else runs across this thread... to get
> my filesystem back (pvscan didn't detect the LVM at all):
> 
>   # pvcreate --uuid 3HpAVg-fRz0-HWXp-EFxK-khVO-IvN6-J4a6w1 /dev/sdb

If you also specify "--restorefile=" and the path to the current 
metadata backup in /etc/lvm, pvcreate will ensure that the new PV can 
accommodate the VG you want to restore to it. It'll almost certainly 
always work without this if you always use the default options but by 
specifying --restorefile you can be completely sure.

>   (Found the old UUID by examining the /etc/lvm/backup directory)
>   # vgcfgrestore FTP1
>   # vgchange -ay FTP1
>   # fsck.ext3 /dev/FTP1/LogVol00
>   # mount -a
> 
> Back in action!

Glad to hear it!

Regards,
Bryn.

[linux-lvm] Re: Dead device and cleaning up LVM

[Stefan Monnier]

> You don't mention the name of your VG/LV but you should be able to remove
> the active lvm devices with "dmsetup remove", e.g.:

> dmsetup remove $VG-$LV

IIUC you can also use

   lvchange -an /dev/<VG>/<LV>

at leas that's what I do when an LVM volume becomes unavailable
(e.g., because the corresponding USB drive was disconnected).
I guess "vgchange -an" might work as well.


        Stefan