[linux-lvm] PV not found

[linux-lvm] PV not found

[Giuseppe Vacanti]

Hello,

after a reboot after a maintenance power interruption of a few hours, one of my LVs is not coming back (Ubuntu 18.04). The situation I am at is the following:

- the physical disk is an iSCSI and it is seen by the kernel as /dev/sdc

- lvmdiskscan -v sees the PV on /dev/sdc
  /dev/sdc              [      72.76 TiB] LVM physical volume

- blkid returns the UUID that I can also find in the LVM configuration
 /dev/sdc: UUID="fvUXXf-pVOF-EPnn-c8eg-tZ5S-iMVW-wsSFDy" TYPE="LVM2_member"

- this entry lacks a PARTUUID entry that other LVs on the system have. Is this a clue? I could not quite connect this piece of info with anything.

- pvscan does not report /dev/sdc

- pvdisplay does not seem to know about this PV
  pvdisplay /dev/sdc
  Failed to find physical volume "/dev/sdc"

Reading man pages and searching the web does not really bring me closer to understanding what is happening. This machine has its OS on LVs and those have all come back, and of course it is not the first time that I reboot. Probably I lack the correct terms to describe the problem so that I can find a solution. Anybody here on this list who can point me in the right direction?

Regards, Giuseppe

--
Giuseppe Vacanti

Re: [linux-lvm] PV not found

[John Stoffel]

Giuseppe> after a reboot after a maintenance power interruption of a
Giuseppe> few hours, one of my LVs is not coming back (Ubuntu
Giuseppe> 18.04). The situation I am at is the following:

Ouch, this isn't fun.  

Giuseppe> - the physical disk is an iSCSI and it is seen by the kernel as /dev/sdc

Does it have any partitions on it?  Is this the only iSCSI volume on
the system?  

Giuseppe> - lvmdiskscan -v sees the PV on /dev/sdc
Giuseppe>   /dev/sdc              [      72.76 TiB] LVM physical volume

Can you show the full details of all your LVMs please?  Do any of the
others have a conflict with this device?

Giuseppe> - blkid returns the UUID that I can also find in the LVM configuration
Giuseppe>  /dev/sdc: UUID="fvUXXf-pVOF-EPnn-c8eg-tZ5S-iMVW-wsSFDy" TYPE="LVM2_member"

That's good.  What about if you do:

       pvchange -v /dev/sdc

what happens then?  

Giuseppe> - this entry lacks a PARTUUID entry that other LVs on the
Giuseppe> system have. Is this a clue? I could not quite connect this
Giuseppe> piece of info with anything.

Can you show the full output and the differences?

Giuseppe> - pvscan does not report /dev/sdc

Not good.

Giuseppe> - pvdisplay does not seem to know about this PV
Giuseppe>   pvdisplay /dev/sdc
Giuseppe>   Failed to find physical volume "/dev/sdc"

Not good either. How about the 'pvchange' and/or 'vgchange -ay' as
well?  Maybe the iSCSI LUN is getting mapped after all the rest of the
LVM setup happens...

Giuseppe> Reading man pages and searching the web does not really
Giuseppe> bring me closer to understanding what is happening. This
Giuseppe> machine has its OS on LVs and those have all come back, and
Giuseppe> of course it is not the first time that I reboot. Probably I
Giuseppe> lack the correct terms to describe the problem so that I can
Giuseppe> find a solution. Anybody here on this list who can point me
Giuseppe> in the right direction?

By any chance did you put a partition table on there?  What system is
the iSCSI LUN being exported from?  Can you run the following iscsiadm
command and share the details?

 iscsiadm --mode discoverydb --type sendtargets --portal <IP> --discover

where <IP> is the address of the iSCSI server you're mounting from.
Did that server go down as well?

Good luck!
John

Re: [linux-lvm] PV not found

[Giuseppe Vacanti]

[-- Attachment #1: Type: text/plain, Size: 363 bytes --]

Hi John Thanks for your answer. On my system I cannot run pvchange -v /dev/sdc as the command expects some further instructions (No command with matching syntax recognised.) I have opened a question at the link below with some additional information https://unix.stackexchange.com/questions/489182/lvm-pv-missing-after-reboot Regards, Giuseppe -- Giuseppe Vacanti

[-- Attachment #2: Type: text/html, Size: 997 bytes --]

Re: [linux-lvm] PV not found

[Gionatan Danti]

Il 15-12-2018 18:59 Giuseppe Vacanti ha scritto:
> - pvscan does not report /dev/sdc
> 
> - pvdisplay does not seem to know about this PV
>   pvdisplay /dev/sdc
>   Failed to find physical volume "/dev/sdc"

Can you show the output of "lsblk" and "pvscan -vvv" ?
Thanks.

-- 
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.danti@assyoma.it - info@assyoma.it
GPG public key ID: FF5F32A8

Re: [linux-lvm] PV not found

[Giuseppe Vacanti]

[-- Attachment #1: Type: text/plain, Size: 887 bytes --]

Hello, I have found a solution to the problem, however the problem seems to come back after every reboot. I have been put on the good path by the remarks made on this list by John and Gionatan. The fix is the following: - convince myself that the device has the correct uuid, I used blkid and compared the result with the information in the lvm backup - run pvck -t on /dev/sdc. Nothing special, all seems OK - run vgscan --cache. This command brings back knowledge of the VG the PV belongs to - now pvs lists my PV again, but the LV is inactive - I activate the LV with lvchange - all is back, I can mount and use the LV After a reboot however the information about the VG is gone again, and I have to repeat the steps above to be able to use the LV. I have a running system, but I don't know what is happening. Where can I look to figure this out? Regards, Giuseppe -- Giuseppe Vacanti

[-- Attachment #2: Type: text/html, Size: 1580 bytes --]

Re: [linux-lvm] PV not found

[Roger Heflin]

Disable lvmetad (in systemctl and by editing lvm.conf to not use metad).

I have had lvmetad cause disks to not get detected on boot up on
several reboots. Restarting it will fix it sometimes and then doing a
vgchange -ay afterwards enabled the lvs.  It is easiest to just
disable it as I believe I have seen notes indicating lvmetad is being
moved away from.

On Wed, Jan 2, 2019 at 4:11 AM Giuseppe Vacanti <giuseppe@vacanti.org> wrote:
>
> Hello,
>
> I have found a solution to the problem, however the problem seems to come back after every reboot.
>
> I have been put on the good path by the remarks made on this list by John and Gionatan.
>
> The fix is the following:
>
> - convince myself that the device has the correct uuid, I used blkid and compared the result with the information in the lvm backup
> - run pvck -t on /dev/sdc. Nothing special, all seems OK
> - run vgscan --cache. This command brings back knowledge of the VG the PV belongs to
> - now pvs lists my PV again, but the LV is inactive
> - I activate the LV with lvchange
> - all is back, I can mount and use the LV
>
> After a reboot however the information about the VG is gone again, and I have to repeat the steps above to be able to use the LV.
>
> I have a running system, but I don't know what is happening. Where can I look to figure this out?
>
> Regards, Giuseppe
>
> --
> Giuseppe Vacanti
>
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/