* [linux-lvm] Recovering "broken" disk ( 17th ) @ 2021-10-18 2:17 Brian McCullough 2021-10-18 14:49 ` Roger Heflin 0 siblings, 1 reply; 8+ messages in thread From: Brian McCullough @ 2021-10-18 2:17 UTC (permalink / raw) To: LVM general discussion and development Folks, I have had a disk go bad on me, causing me to lose one PV. I seem to have retrieved the partition using ddrescue, but it also seems to be missing some label information, because pvscan doesn't see it. Using hexdump, I see the string " LVM2 " at 0x1004, but nothing before that. The whole phrase is: 0x01000 16 d6 8e db 20 4c 56 4d 32 20 78 5b 35 41 25 72 I find what appears to be an LVM2 configuration section at 0x1200, and so I was able to read the UUID that this PV should have. On another machine, I dumped a PV partition, and find "LABLEONE" at 0x200, with the same " LVM2 " at 0x01000. I was concerned that my dump was offset, but the comparison to the "good" one suggests that that isn't the problem, but just the missing "LABLEONE" and related information at 0x0200. If I do a "pvcreate --uuid xxxx" would this fix that recovered partition so that pvscan and friends can work properly, and I can finally boot that machine? Thank you, Brian _______________________________________________ linux-lvm mailing list linux-lvm@redhat.com https://listman.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [linux-lvm] Recovering "broken" disk ( 17th ) 2021-10-18 2:17 [linux-lvm] Recovering "broken" disk ( 17th ) Brian McCullough @ 2021-10-18 14:49 ` Roger Heflin 2021-10-18 19:45 ` Brian McCullough 0 siblings, 1 reply; 8+ messages in thread From: Roger Heflin @ 2021-10-18 14:49 UTC (permalink / raw) To: LVM general discussion and development [-- Attachment #1.1: Type: text/plain, Size: 1965 bytes --] You will need a lvm backup file for the pvcreate --uuid I believe (there may be some option to get around needing the backup file). That will put the header back on if you either have an lvm backup and/or archive file, you might also need a vgcfgrestore afterwards depending on if anything else is missing. I have never done it, but it looks possible to make a lvm backup file by reading it directly off the disk with dd, so that you will have a file that pvcreate is ok with, that is if there is no way to force it without a backup file. But, this should get the pv back showing up with whatever sectors that you successfully recovered. On Sun, Oct 17, 2021 at 10:02 PM Brian McCullough <bdmc@bdmcc-us.com> wrote: > > Folks, > > I have had a disk go bad on me, causing me to lose one PV. > > > I seem to have retrieved the partition using ddrescue, but it also seems > to be missing some label information, because pvscan doesn't see it. > > Using hexdump, I see the string " LVM2 " at 0x1004, but nothing before > that. The whole phrase is: > > 0x01000 16 d6 8e db 20 4c 56 4d 32 20 78 5b 35 41 25 72 > > > I find what appears to be an LVM2 configuration section at 0x1200, and > so I was able to read the UUID that this PV should have. > > > On another machine, I dumped a PV partition, and find "LABLEONE" at > 0x200, with the same " LVM2 " at 0x01000. > > I was concerned that my dump was offset, but the comparison to the > "good" one suggests that that isn't the problem, but just the missing > "LABLEONE" and related information at 0x0200. > > If I do a "pvcreate --uuid xxxx" would this fix that recovered partition > so that pvscan and friends can work properly, and I can finally boot > that machine? > > > Thank you, > Brian > > > _______________________________________________ > linux-lvm mailing list > linux-lvm@redhat.com > https://listman.redhat.com/mailman/listinfo/linux-lvm > read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ > > [-- Attachment #1.2: Type: text/html, Size: 2778 bytes --] [-- Attachment #2: Type: text/plain, Size: 201 bytes --] _______________________________________________ linux-lvm mailing list linux-lvm@redhat.com https://listman.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [linux-lvm] Recovering "broken" disk ( 17th ) 2021-10-18 14:49 ` Roger Heflin @ 2021-10-18 19:45 ` Brian McCullough 2021-10-19 10:06 ` Roger Heflin 0 siblings, 1 reply; 8+ messages in thread From: Brian McCullough @ 2021-10-18 19:45 UTC (permalink / raw) To: LVM general discussion and development On Mon, Oct 18, 2021 at 09:49:44AM -0500, Roger Heflin wrote: > You will need a lvm backup file for the pvcreate --uuid I believe (there > may be some option to get around needing the backup file). > > That will put the header back on if you either have an lvm backup and/or > archive file, you might also need a vgcfgrestore afterwards depending on if > anything else is missing. > > I have never done it, but it looks possible to make a lvm backup file by > reading it directly off the disk with dd, so that you will have a file that > pvcreate is ok with, that is if there is no way to force it without a > backup file. > > But, this should get the pv back showing up with whatever sectors that you > successfully recovered. Thank you, Roger. I might be able to do that on the original machine, which won't boot at the moment because of this failure, but I have been using a working machine so far to use ddrestore etc. Yes, as you said, I was able to use dd to copy off the vgconfig data into a file. So you think that I might be able to use that as the reference in pvcreate? That makes some sense. I started this out asking ( but not clearly ) whether the data on the disk appeared to be in the correct place and whether what was there was correct, but I think that I answered that for myself with the comparison to the working PV partition. Thanks again, Brian _______________________________________________ linux-lvm mailing list linux-lvm@redhat.com https://listman.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [linux-lvm] Recovering "broken" disk ( 17th ) 2021-10-18 19:45 ` Brian McCullough @ 2021-10-19 10:06 ` Roger Heflin 2021-10-20 13:38 ` Brian McCullough 0 siblings, 1 reply; 8+ messages in thread From: Roger Heflin @ 2021-10-19 10:06 UTC (permalink / raw) To: LVM general discussion and development [-- Attachment #1.1: Type: text/plain, Size: 2438 bytes --] I would edit the vgconfig you dd'ed with an editor and make sure it looks reasonable for what you think you had. When you do the pvcreate --uuid it won't use anything except the uuid info so the rest may not need to be exactly right, if you have to do a vgcfgrestore to get it to read the rest of the info will be used. I have seen some weird disk controller failures that appeared to zero out the first bit of the disk (enough to get the partition table, grub, and the pv header depending on where the first partition starts). You will need to reinstall grub if this was the bootable disk, since there were 384 bytes of grub in the sector with the partition table that you know are missing. On Tue, Oct 19, 2021 at 1:04 AM Brian McCullough <bdmc@buadh-brath.com> wrote: > On Mon, Oct 18, 2021 at 09:49:44AM -0500, Roger Heflin wrote: > > You will need a lvm backup file for the pvcreate --uuid I believe (there > > may be some option to get around needing the backup file). > > > > That will put the header back on if you either have an lvm backup and/or > > archive file, you might also need a vgcfgrestore afterwards depending on > if > > anything else is missing. > > > > I have never done it, but it looks possible to make a lvm backup file by > > reading it directly off the disk with dd, so that you will have a file > that > > pvcreate is ok with, that is if there is no way to force it without a > > backup file. > > > > But, this should get the pv back showing up with whatever sectors that > you > > successfully recovered. > > Thank you, Roger. > > I might be able to do that on the original machine, which won't boot at > the moment because of this failure, but I have been using a working > machine so far to use ddrestore etc. > > Yes, as you said, I was able to use dd to copy off the vgconfig data > into a file. So you think that I might be able to use that as the > reference in pvcreate? That makes some sense. > > I started this out asking ( but not clearly ) whether the data on the > disk appeared to be in the correct place and whether what was there was > correct, but I think that I answered that for myself with the > comparison to the working PV partition. > > > Thanks again, > Brian > > _______________________________________________ > linux-lvm mailing list > linux-lvm@redhat.com > https://listman.redhat.com/mailman/listinfo/linux-lvm > read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ > > [-- Attachment #1.2: Type: text/html, Size: 3226 bytes --] [-- Attachment #2: Type: text/plain, Size: 201 bytes --] _______________________________________________ linux-lvm mailing list linux-lvm@redhat.com https://listman.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [linux-lvm] Recovering "broken" disk ( 17th ) 2021-10-19 10:06 ` Roger Heflin @ 2021-10-20 13:38 ` Brian McCullough 2021-10-20 18:01 ` Roger Heflin 0 siblings, 1 reply; 8+ messages in thread From: Brian McCullough @ 2021-10-20 13:38 UTC (permalink / raw) To: LVM general discussion and development On Tue, Oct 19, 2021 at 05:06:37AM -0500, Roger Heflin wrote: > I would edit the vgconfig you dd'ed with an editor and make sure it looks > reasonable for what you think you had. It turns out, comparing the information that I pulled off of the drive with what I find in /etc/lvm/backup, that the first part of the vgconfig information is missing. As I said in one of my messages, the information that I retrieved from the disk starts at 0x1200. I don't know whether that is correct or not. It does not appear to be a proper "backup" file, which I think it should be. I rebooted ( partially ) the machine and copied the vgconfig backup file from that, but am somewhat concerned, because I don't seem to be able to match the UUIDs. The one that I seem to see in the vgconfig data that I pulled off of the drive vs what I got out of /etc/lvm/backup. Maybe I am just mis-reading it. I will continue my research for a bit. > When you do the pvcreate --uuid it won't use anything except the uuid info > so the rest may not need to be exactly right, if you have to do a > vgcfgrestore to get it to read the rest of the info will be used. Oh, thank you. I did see that things got somewhat different on the target drive when I did "pvcreate --uuid --restorefile." I got paranoid when I saw that, and re-copied the ddrestore file back to the target drive before I did anything else. Should I do "pvcreate --uuid --norestorefile," instead? Then, once it is back in the machine, do the pvscan and vgcfgrestore, and expect good things? > I have seen some weird disk controller failures that appeared to zero out > the first bit of the disk (enough to get the partition table, grub, and the > pv header depending on where the first partition starts). I APPEAR to have a partition table, containing an NTFS partition, an LVM partiton ( the one that I am concentrating on ) and a Linux partion. I would have thought that it was all LVM, but my memory could easily be wrong. > You will need to reinstall grub if this was the bootable disk, since there > were 384 bytes of grub in the sector with the partition table that you know > are missing. Fortunately, this is all data, nothing to do with the boot sequence, except that the machine will not boot with the missing PV. Thank you, Brian _______________________________________________ linux-lvm mailing list linux-lvm@redhat.com https://listman.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [linux-lvm] Recovering "broken" disk ( 17th ) 2021-10-20 13:38 ` Brian McCullough @ 2021-10-20 18:01 ` Roger Heflin 2021-10-20 18:04 ` Roger Heflin 0 siblings, 1 reply; 8+ messages in thread From: Roger Heflin @ 2021-10-20 18:01 UTC (permalink / raw) To: LVM general discussion and development [-- Attachment #1.1: Type: text/plain, Size: 3378 bytes --] is the pv in the root device vg? if not changing fstab to not mount the missing fs(es) should get it bootable. I have a practice of putting ",nofail" on all non-root filesystems (ie defaults,nofail) since priority #1 is getting the machine up and on the network after a reboot such that it can be ssh'ed to and fixed as needed. If it is not the root device then on the root device there should be several prior archive copy in /etc/lvm/archive/<vgname>*, and maybe some copies in /etc/lvm/backup. In the vg backup file there will be a bunch of uuids, you want the specific pv uuid and not the vg/lv uuids. Each pv has a uuid and each lv has a uuid and the vg has a uuid. On Wed, Oct 20, 2021 at 8:39 AM Brian McCullough <bdmc@bdmcc-us.com> wrote: > On Tue, Oct 19, 2021 at 05:06:37AM -0500, Roger Heflin wrote: > > I would edit the vgconfig you dd'ed with an editor and make sure it looks > > reasonable for what you think you had. > > It turns out, comparing the information that I pulled off of the drive > with what I find in /etc/lvm/backup, that the first part of the vgconfig > information is missing. As I said in one of my messages, the > information that I retrieved from the disk starts at 0x1200. I don't > know whether that is correct or not. It does not appear to be a proper > "backup" file, which I think it should be. > > I rebooted ( partially ) the machine and copied the vgconfig backup file > from that, but am somewhat concerned, because I don't seem to be able to > match the UUIDs. The one that I seem to see in the vgconfig data that I > pulled off of the drive vs what I got out of /etc/lvm/backup. Maybe I > am just mis-reading it. I will continue my research for a bit. > > > > > > When you do the pvcreate --uuid it won't use anything except the uuid > info > > so the rest may not need to be exactly right, if you have to do a > > vgcfgrestore to get it to read the rest of the info will be used. > > Oh, thank you. I did see that things got somewhat different on the > target drive when I did "pvcreate --uuid --restorefile." I got paranoid > when I saw that, and re-copied the ddrestore file back to the target > drive before I did anything else. Should I do "pvcreate --uuid > --norestorefile," instead? Then, once it is back in the machine, do the > pvscan and vgcfgrestore, and expect good things? > > > > > I have seen some weird disk controller failures that appeared to zero out > > the first bit of the disk (enough to get the partition table, grub, and > the > > pv header depending on where the first partition starts). > > I APPEAR to have a partition table, containing an NTFS partition, an LVM > partiton ( the one that I am concentrating on ) and a Linux partion. I > would have thought that it was all LVM, but my memory could easily be > wrong. > > > > > You will need to reinstall grub if this was the bootable disk, since > there > > were 384 bytes of grub in the sector with the partition table that you > know > > are missing. > > Fortunately, this is all data, nothing to do with the boot sequence, > except that the machine will not boot with the missing PV. > > > > Thank you, > Brian > > _______________________________________________ > linux-lvm mailing list > linux-lvm@redhat.com > https://listman.redhat.com/mailman/listinfo/linux-lvm > read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ > > [-- Attachment #1.2: Type: text/html, Size: 4278 bytes --] [-- Attachment #2: Type: text/plain, Size: 201 bytes --] _______________________________________________ linux-lvm mailing list linux-lvm@redhat.com https://listman.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [linux-lvm] Recovering "broken" disk ( 17th ) 2021-10-20 18:01 ` Roger Heflin @ 2021-10-20 18:04 ` Roger Heflin 2021-10-21 6:13 ` Brian McCullough 0 siblings, 1 reply; 8+ messages in thread From: Roger Heflin @ 2021-10-20 18:04 UTC (permalink / raw) To: LVM general discussion and development [-- Attachment #1.1: Type: text/plain, Size: 3878 bytes --] replying to my last email. do the pvcreate -uuid and then do a pvs/lvs/vgs and see if the vg/lv's look like that are there. if so do a vgchange -ay <vgname> and then test mounting the fs. And if with the fs either commented out and/or ,nofail the normal os boots up work from there as you should have backup files. On Wed, Oct 20, 2021 at 1:01 PM Roger Heflin <rogerheflin@gmail.com> wrote: > is the pv in the root device vg? if not changing fstab to not mount the > missing fs(es) should get it bootable. I have a practice of putting > ",nofail" on all non-root filesystems (ie defaults,nofail) since priority > #1 is getting the machine up and on the network after a reboot such that it > can be ssh'ed to and fixed as needed. > > If it is not the root device then on the root device there should be > several prior archive copy in /etc/lvm/archive/<vgname>*, and maybe some > copies in /etc/lvm/backup. > > In the vg backup file there will be a bunch of uuids, you want the > specific pv uuid and not the vg/lv uuids. Each pv has a uuid and each lv > has a uuid and the vg has a uuid. > > On Wed, Oct 20, 2021 at 8:39 AM Brian McCullough <bdmc@bdmcc-us.com> > wrote: > >> On Tue, Oct 19, 2021 at 05:06:37AM -0500, Roger Heflin wrote: >> > I would edit the vgconfig you dd'ed with an editor and make sure it >> looks >> > reasonable for what you think you had. >> >> It turns out, comparing the information that I pulled off of the drive >> with what I find in /etc/lvm/backup, that the first part of the vgconfig >> information is missing. As I said in one of my messages, the >> information that I retrieved from the disk starts at 0x1200. I don't >> know whether that is correct or not. It does not appear to be a proper >> "backup" file, which I think it should be. >> >> I rebooted ( partially ) the machine and copied the vgconfig backup file >> from that, but am somewhat concerned, because I don't seem to be able to >> match the UUIDs. The one that I seem to see in the vgconfig data that I >> pulled off of the drive vs what I got out of /etc/lvm/backup. Maybe I >> am just mis-reading it. I will continue my research for a bit. >> >> >> >> >> > When you do the pvcreate --uuid it won't use anything except the uuid >> info >> > so the rest may not need to be exactly right, if you have to do a >> > vgcfgrestore to get it to read the rest of the info will be used. >> >> Oh, thank you. I did see that things got somewhat different on the >> target drive when I did "pvcreate --uuid --restorefile." I got paranoid >> when I saw that, and re-copied the ddrestore file back to the target >> drive before I did anything else. Should I do "pvcreate --uuid >> --norestorefile," instead? Then, once it is back in the machine, do the >> pvscan and vgcfgrestore, and expect good things? >> >> >> >> > I have seen some weird disk controller failures that appeared to zero >> out >> > the first bit of the disk (enough to get the partition table, grub, and >> the >> > pv header depending on where the first partition starts). >> >> I APPEAR to have a partition table, containing an NTFS partition, an LVM >> partiton ( the one that I am concentrating on ) and a Linux partion. I >> would have thought that it was all LVM, but my memory could easily be >> wrong. >> >> >> >> > You will need to reinstall grub if this was the bootable disk, since >> there >> > were 384 bytes of grub in the sector with the partition table that you >> know >> > are missing. >> >> Fortunately, this is all data, nothing to do with the boot sequence, >> except that the machine will not boot with the missing PV. >> >> >> >> Thank you, >> Brian >> >> _______________________________________________ >> linux-lvm mailing list >> linux-lvm@redhat.com >> https://listman.redhat.com/mailman/listinfo/linux-lvm >> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ >> >> [-- Attachment #1.2: Type: text/html, Size: 5047 bytes --] [-- Attachment #2: Type: text/plain, Size: 201 bytes --] _______________________________________________ linux-lvm mailing list linux-lvm@redhat.com https://listman.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [linux-lvm] Recovering "broken" disk ( 17th ) 2021-10-20 18:04 ` Roger Heflin @ 2021-10-21 6:13 ` Brian McCullough 0 siblings, 0 replies; 8+ messages in thread From: Brian McCullough @ 2021-10-21 6:13 UTC (permalink / raw) To: LVM general discussion and development On Wed, Oct 20, 2021 at 01:04:44PM -0500, Roger Heflin wrote: > replying to my last email. Thank you for your help, Roger. I will answer with more detail when the sun comes around to my side of the planet, but will say that this is a "secondary" VG, so your suggestion of changing fstab is very attractive. B _______________________________________________ linux-lvm mailing list linux-lvm@redhat.com https://listman.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2021-10-21 6:13 UTC | newest] Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-10-18 2:17 [linux-lvm] Recovering "broken" disk ( 17th ) Brian McCullough 2021-10-18 14:49 ` Roger Heflin 2021-10-18 19:45 ` Brian McCullough 2021-10-19 10:06 ` Roger Heflin 2021-10-20 13:38 ` Brian McCullough 2021-10-20 18:01 ` Roger Heflin 2021-10-20 18:04 ` Roger Heflin 2021-10-21 6:13 ` Brian McCullough
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).