[linux-lvm] Recovering "broken" disk ( 17th )

[linux-lvm] Recovering "broken" disk ( 17th )

[Brian McCullough]

Folks,

I have had a disk go bad on me, causing me to lose one PV.


I seem to have retrieved the partition using ddrescue, but it also seems
to be missing some label information, because pvscan doesn't see it.

Using hexdump, I see the string " LVM2 " at 0x1004, but nothing before
that.  The whole phrase is: 

0x01000  16 d6 8e db 20 4c 56 4d  32 20 78 5b 35 41 25 72


I find what appears to be an LVM2 configuration section at 0x1200, and
so I was able to read the UUID that this PV should have.


On another machine, I dumped a PV partition, and find "LABLEONE" at
0x200, with the same " LVM2 " at 0x01000.

I was concerned that my dump was offset, but the comparison to the
"good" one suggests that that isn't the problem, but just the missing
"LABLEONE" and related information at 0x0200.

If I do a "pvcreate --uuid xxxx" would this fix that recovered partition
so that pvscan and friends can work properly, and I can finally boot
that machine?


Thank you,
Brian


_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] Recovering "broken" disk ( 17th )

[Roger Heflin]

[-- Attachment #1.1: Type: text/plain, Size: 1965 bytes --]

You will need a lvm backup file for the pvcreate --uuid I believe (there
may be some option to get around needing the backup file).

That will put the header back on if you either have an lvm backup and/or
archive file, you might also need a vgcfgrestore afterwards depending on if
anything else is missing.

I have never done it, but it looks possible to make a lvm backup file by
reading it directly off the disk with dd, so that you will have a file that
pvcreate is ok with, that is if there is no way to force it without a
backup file.

But, this should get the pv back showing up with whatever sectors that you
successfully recovered.

On Sun, Oct 17, 2021 at 10:02 PM Brian McCullough <bdmc@bdmcc-us.com> wrote:

>
> Folks,
>
> I have had a disk go bad on me, causing me to lose one PV.
>
>
> I seem to have retrieved the partition using ddrescue, but it also seems
> to be missing some label information, because pvscan doesn't see it.
>
> Using hexdump, I see the string " LVM2 " at 0x1004, but nothing before
> that.  The whole phrase is:
>
> 0x01000  16 d6 8e db 20 4c 56 4d  32 20 78 5b 35 41 25 72
>
>
> I find what appears to be an LVM2 configuration section at 0x1200, and
> so I was able to read the UUID that this PV should have.
>
>
> On another machine, I dumped a PV partition, and find "LABLEONE" at
> 0x200, with the same " LVM2 " at 0x01000.
>
> I was concerned that my dump was offset, but the comparison to the
> "good" one suggests that that isn't the problem, but just the missing
> "LABLEONE" and related information at 0x0200.
>
> If I do a "pvcreate --uuid xxxx" would this fix that recovered partition
> so that pvscan and friends can work properly, and I can finally boot
> that machine?
>
>
> Thank you,
> Brian
>
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://listman.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>
>

[-- Attachment #1.2: Type: text/html, Size: 2778 bytes --]

[-- Attachment #2: Type: text/plain, Size: 201 bytes --]

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] Recovering "broken" disk ( 17th )

[Brian McCullough]

On Mon, Oct 18, 2021 at 09:49:44AM -0500, Roger Heflin wrote:
> You will need a lvm backup file for the pvcreate --uuid I believe (there
> may be some option to get around needing the backup file).
> 
> That will put the header back on if you either have an lvm backup and/or
> archive file, you might also need a vgcfgrestore afterwards depending on if
> anything else is missing.
> 
> I have never done it, but it looks possible to make a lvm backup file by
> reading it directly off the disk with dd, so that you will have a file that
> pvcreate is ok with, that is if there is no way to force it without a
> backup file.
> 
> But, this should get the pv back showing up with whatever sectors that you
> successfully recovered.

Thank you, Roger.

I might be able to do that on the original machine, which won't boot at
the moment because of this failure, but I have been using a working
machine so far to use ddrestore etc.

Yes, as you said, I was able to use dd to copy off the vgconfig data
into a file.  So you think that I might be able to use that as the
reference in pvcreate?  That makes some sense.

I started this out asking ( but not clearly ) whether the data on the
disk appeared to be in the correct place and whether what was there was
correct, but I think that I answered that for myself with the
comparison to the working PV partition.


Thanks again,
Brian

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] Recovering "broken" disk ( 17th )

[Roger Heflin]

[-- Attachment #1.1: Type: text/plain, Size: 2438 bytes --]

I would edit the vgconfig you dd'ed with an editor and make sure it looks
reasonable for what you think you had.

When you do the pvcreate --uuid it won't use anything except the uuid info
so the rest may not need to be exactly right, if you have to do a
vgcfgrestore to get it to read the rest of the info will be used.

I have seen some weird disk controller failures that appeared to zero out
the first bit of the disk (enough to get the partition table, grub, and the
pv header depending on where the first partition starts).

You will need to reinstall grub if this was the bootable disk, since there
were 384 bytes of grub in the sector with the partition table that you know
are missing.

On Tue, Oct 19, 2021 at 1:04 AM Brian McCullough <bdmc@buadh-brath.com>
wrote:

> On Mon, Oct 18, 2021 at 09:49:44AM -0500, Roger Heflin wrote:
> > You will need a lvm backup file for the pvcreate --uuid I believe (there
> > may be some option to get around needing the backup file).
> >
> > That will put the header back on if you either have an lvm backup and/or
> > archive file, you might also need a vgcfgrestore afterwards depending on
> if
> > anything else is missing.
> >
> > I have never done it, but it looks possible to make a lvm backup file by
> > reading it directly off the disk with dd, so that you will have a file
> that
> > pvcreate is ok with, that is if there is no way to force it without a
> > backup file.
> >
> > But, this should get the pv back showing up with whatever sectors that
> you
> > successfully recovered.
>
> Thank you, Roger.
>
> I might be able to do that on the original machine, which won't boot at
> the moment because of this failure, but I have been using a working
> machine so far to use ddrestore etc.
>
> Yes, as you said, I was able to use dd to copy off the vgconfig data
> into a file.  So you think that I might be able to use that as the
> reference in pvcreate?  That makes some sense.
>
> I started this out asking ( but not clearly ) whether the data on the
> disk appeared to be in the correct place and whether what was there was
> correct, but I think that I answered that for myself with the
> comparison to the working PV partition.
>
>
> Thanks again,
> Brian
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://listman.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>
>

[-- Attachment #1.2: Type: text/html, Size: 3226 bytes --]

[-- Attachment #2: Type: text/plain, Size: 201 bytes --]

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] Recovering "broken" disk ( 17th )

[Brian McCullough]

On Tue, Oct 19, 2021 at 05:06:37AM -0500, Roger Heflin wrote:
> I would edit the vgconfig you dd'ed with an editor and make sure it looks
> reasonable for what you think you had.

It turns out, comparing the information that I pulled off of the drive
with what I find in /etc/lvm/backup, that the first part of the vgconfig
information is missing.  As I said in one of my messages, the
information that I retrieved from the disk starts at 0x1200.  I don't
know whether that is correct or not.  It does not appear to be a proper
"backup" file, which I think it should be.

I rebooted ( partially ) the machine and copied the vgconfig backup file
from that, but am somewhat concerned, because I don't seem to be able to
match the UUIDs.  The one that I seem to see in the vgconfig data that I
pulled off of the drive vs what I got out of /etc/lvm/backup.  Maybe I
am just mis-reading it.  I will continue my research for a bit.




> When you do the pvcreate --uuid it won't use anything except the uuid info
> so the rest may not need to be exactly right, if you have to do a
> vgcfgrestore to get it to read the rest of the info will be used.

Oh, thank you.   I did see that things got somewhat different on the
target drive when I did "pvcreate --uuid --restorefile."  I got paranoid
when I saw that, and re-copied the ddrestore file back to the target
drive before I did anything else.   Should I do "pvcreate --uuid
--norestorefile," instead?  Then, once it is back in the machine, do the
pvscan and vgcfgrestore, and expect good things?



> I have seen some weird disk controller failures that appeared to zero out
> the first bit of the disk (enough to get the partition table, grub, and the
> pv header depending on where the first partition starts).

I APPEAR to have a partition table, containing an NTFS partition, an LVM
partiton ( the one that I am concentrating on ) and a Linux partion.  I
would have thought that it was all LVM, but my memory could easily be
wrong.



> You will need to reinstall grub if this was the bootable disk, since there
> were 384 bytes of grub in the sector with the partition table that you know
> are missing.

Fortunately, this is all data, nothing to do with the boot sequence,
except that the machine will not boot with the missing PV.



Thank you,
Brian

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] Recovering "broken" disk ( 17th )

[Roger Heflin]

[-- Attachment #1.1: Type: text/plain, Size: 3378 bytes --]

is the pv in the root device vg?  if not changing fstab to not mount the
missing  fs(es) should get it bootable.   I have a practice of putting
",nofail" on all non-root filesystems (ie defaults,nofail) since priority
#1 is getting the machine up and on the network after a reboot such that it
can be ssh'ed to and fixed as needed.

If it is not the root device then on the root device there should be
several prior archive copy in /etc/lvm/archive/<vgname>*, and maybe some
copies in /etc/lvm/backup.

In the vg backup file there will be a bunch of uuids, you want the specific
pv uuid and not the vg/lv uuids.  Each pv has a uuid and each lv has a uuid
and the vg has a uuid.

On Wed, Oct 20, 2021 at 8:39 AM Brian McCullough <bdmc@bdmcc-us.com> wrote:

> On Tue, Oct 19, 2021 at 05:06:37AM -0500, Roger Heflin wrote:
> > I would edit the vgconfig you dd'ed with an editor and make sure it looks
> > reasonable for what you think you had.
>
> It turns out, comparing the information that I pulled off of the drive
> with what I find in /etc/lvm/backup, that the first part of the vgconfig
> information is missing.  As I said in one of my messages, the
> information that I retrieved from the disk starts at 0x1200.  I don't
> know whether that is correct or not.  It does not appear to be a proper
> "backup" file, which I think it should be.
>
> I rebooted ( partially ) the machine and copied the vgconfig backup file
> from that, but am somewhat concerned, because I don't seem to be able to
> match the UUIDs.  The one that I seem to see in the vgconfig data that I
> pulled off of the drive vs what I got out of /etc/lvm/backup.  Maybe I
> am just mis-reading it.  I will continue my research for a bit.
>
>
>
>
> > When you do the pvcreate --uuid it won't use anything except the uuid
> info
> > so the rest may not need to be exactly right, if you have to do a
> > vgcfgrestore to get it to read the rest of the info will be used.
>
> Oh, thank you.   I did see that things got somewhat different on the
> target drive when I did "pvcreate --uuid --restorefile."  I got paranoid
> when I saw that, and re-copied the ddrestore file back to the target
> drive before I did anything else.   Should I do "pvcreate --uuid
> --norestorefile," instead?  Then, once it is back in the machine, do the
> pvscan and vgcfgrestore, and expect good things?
>
>
>
> > I have seen some weird disk controller failures that appeared to zero out
> > the first bit of the disk (enough to get the partition table, grub, and
> the
> > pv header depending on where the first partition starts).
>
> I APPEAR to have a partition table, containing an NTFS partition, an LVM
> partiton ( the one that I am concentrating on ) and a Linux partion.  I
> would have thought that it was all LVM, but my memory could easily be
> wrong.
>
>
>
> > You will need to reinstall grub if this was the bootable disk, since
> there
> > were 384 bytes of grub in the sector with the partition table that you
> know
> > are missing.
>
> Fortunately, this is all data, nothing to do with the boot sequence,
> except that the machine will not boot with the missing PV.
>
>
>
> Thank you,
> Brian
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://listman.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>
>

[-- Attachment #1.2: Type: text/html, Size: 4278 bytes --]

[-- Attachment #2: Type: text/plain, Size: 201 bytes --]

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] Recovering "broken" disk ( 17th )

[Roger Heflin]

[-- Attachment #1.1: Type: text/plain, Size: 3878 bytes --]

replying to my last email.

do the pvcreate -uuid and then do a pvs/lvs/vgs and see if the vg/lv's look
like that are there.   if so do a vgchange -ay <vgname> and then test
mounting the fs.

And if with the fs either commented out and/or ,nofail the normal os boots
up work from there as you should have backup files.

On Wed, Oct 20, 2021 at 1:01 PM Roger Heflin <rogerheflin@gmail.com> wrote:

> is the pv in the root device vg?  if not changing fstab to not mount the
> missing  fs(es) should get it bootable.   I have a practice of putting
> ",nofail" on all non-root filesystems (ie defaults,nofail) since priority
> #1 is getting the machine up and on the network after a reboot such that it
> can be ssh'ed to and fixed as needed.
>
> If it is not the root device then on the root device there should be
> several prior archive copy in /etc/lvm/archive/<vgname>*, and maybe some
> copies in /etc/lvm/backup.
>
> In the vg backup file there will be a bunch of uuids, you want the
> specific pv uuid and not the vg/lv uuids.  Each pv has a uuid and each lv
> has a uuid and the vg has a uuid.
>
> On Wed, Oct 20, 2021 at 8:39 AM Brian McCullough <bdmc@bdmcc-us.com>
> wrote:
>
>> On Tue, Oct 19, 2021 at 05:06:37AM -0500, Roger Heflin wrote:
>> > I would edit the vgconfig you dd'ed with an editor and make sure it
>> looks
>> > reasonable for what you think you had.
>>
>> It turns out, comparing the information that I pulled off of the drive
>> with what I find in /etc/lvm/backup, that the first part of the vgconfig
>> information is missing.  As I said in one of my messages, the
>> information that I retrieved from the disk starts at 0x1200.  I don't
>> know whether that is correct or not.  It does not appear to be a proper
>> "backup" file, which I think it should be.
>>
>> I rebooted ( partially ) the machine and copied the vgconfig backup file
>> from that, but am somewhat concerned, because I don't seem to be able to
>> match the UUIDs.  The one that I seem to see in the vgconfig data that I
>> pulled off of the drive vs what I got out of /etc/lvm/backup.  Maybe I
>> am just mis-reading it.  I will continue my research for a bit.
>>
>>
>>
>>
>> > When you do the pvcreate --uuid it won't use anything except the uuid
>> info
>> > so the rest may not need to be exactly right, if you have to do a
>> > vgcfgrestore to get it to read the rest of the info will be used.
>>
>> Oh, thank you.   I did see that things got somewhat different on the
>> target drive when I did "pvcreate --uuid --restorefile."  I got paranoid
>> when I saw that, and re-copied the ddrestore file back to the target
>> drive before I did anything else.   Should I do "pvcreate --uuid
>> --norestorefile," instead?  Then, once it is back in the machine, do the
>> pvscan and vgcfgrestore, and expect good things?
>>
>>
>>
>> > I have seen some weird disk controller failures that appeared to zero
>> out
>> > the first bit of the disk (enough to get the partition table, grub, and
>> the
>> > pv header depending on where the first partition starts).
>>
>> I APPEAR to have a partition table, containing an NTFS partition, an LVM
>> partiton ( the one that I am concentrating on ) and a Linux partion.  I
>> would have thought that it was all LVM, but my memory could easily be
>> wrong.
>>
>>
>>
>> > You will need to reinstall grub if this was the bootable disk, since
>> there
>> > were 384 bytes of grub in the sector with the partition table that you
>> know
>> > are missing.
>>
>> Fortunately, this is all data, nothing to do with the boot sequence,
>> except that the machine will not boot with the missing PV.
>>
>>
>>
>> Thank you,
>> Brian
>>
>> _______________________________________________
>> linux-lvm mailing list
>> linux-lvm@redhat.com
>> https://listman.redhat.com/mailman/listinfo/linux-lvm
>> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>>
>>

[-- Attachment #1.2: Type: text/html, Size: 5047 bytes --]

[-- Attachment #2: Type: text/plain, Size: 201 bytes --]

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] Recovering "broken" disk ( 17th )

[Brian McCullough]

On Wed, Oct 20, 2021 at 01:04:44PM -0500, Roger Heflin wrote:
> replying to my last email.

Thank you for your help, Roger.  I will answer with more detail when the
sun comes around to my side of the planet, but will say that this is a
"secondary" VG, so your suggestion of changing fstab is very attractive.


B

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/