* [linux-lvm] Problem with partially activate logical volume @ 2022-07-25 14:48 Ken Bass 2022-07-26 8:50 ` Roger James 2022-07-27 19:56 ` Zdenek Kabelac 0 siblings, 2 replies; 9+ messages in thread From: Ken Bass @ 2022-07-25 14:48 UTC (permalink / raw) To: linux-lvm [-- Attachment #1.1: Type: text/plain, Size: 909 bytes --] (fwiw: I am new to this list, so please bear with me.) Background: I have a very large (20TB) logical volume consisting of 3 drives. One of those drives unexpectedloy died (isn't that always the case :-)). The drive that failed happened to be the last PV. So I am assuming that there is still 2/3 of the data still intact and, to some extent, recoverable. Although, apparently the ext4 fs is not recognised. I activated the LV partially (via -P). But running any utility on that (eg: dumpe2fs, e2fsck, ...) I get many of these in dmesg: "Buffer I/O error on dev dm-0, logical block xxxxxxx, async page read." The thing is, the xxxxxxx block is on the missing drive/pv. I have also tried some recovery software, but eventually get these same messages, and the data recovered is not really useful. Please help! How can I get passed that dmesg error, and move on. 14TB recovered is better than 0. TIA ken [-- Attachment #1.2: Type: text/html, Size: 1156 bytes --] [-- Attachment #2: Type: text/plain, Size: 202 bytes --] _______________________________________________ linux-lvm mailing list linux-lvm@redhat.com https://listman.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [linux-lvm] Problem with partially activate logical volume 2022-07-25 14:48 [linux-lvm] Problem with partially activate logical volume Ken Bass @ 2022-07-26 8:50 ` Roger James 2022-07-27 10:49 ` Roger Heflin 2022-07-27 19:56 ` Zdenek Kabelac 1 sibling, 1 reply; 9+ messages in thread From: Roger James @ 2022-07-26 8:50 UTC (permalink / raw) To: LVM general discussion and development [-- Attachment #1.1: Type: text/plain, Size: 1311 bytes --] Try https://www.linuxsysadmins.com/recover-a-deleted-physical-volume/?amp On 26 July 2022 09:16:32 Ken Bass <daytooner@gmail.com> wrote: > (fwiw: I am new to this list, so please bear with me.) > > Background: I have a very large (20TB) logical volume consisting of 3 > drives. One of those drives unexpectedloy died (isn't that always the case > :-)). The drive that failed happened to be the last PV. So I am assuming > that there is still 2/3 of the data still intact and, to some extent, > recoverable. Although, apparently the ext4 fs is not recognised. > > I activated the LV partially (via -P). But running any utility on that (eg: > dumpe2fs, e2fsck, ...) I get many of these in dmesg: > > "Buffer I/O error on dev dm-0, logical block xxxxxxx, async page read." > The thing is, the xxxxxxx block is on the missing drive/pv. > > I have also tried some recovery software, but eventually get these same > messages, and the data recovered is not really useful. > > Please help! How can I get passed that dmesg error, and move on. 14TB > recovered is better than 0. > > TIA > ken > > > _______________________________________________ > linux-lvm mailing list > linux-lvm@redhat.com > https://listman.redhat.com/mailman/listinfo/linux-lvm > read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ [-- Attachment #1.2: Type: text/html, Size: 2563 bytes --] [-- Attachment #2: Type: text/plain, Size: 202 bytes --] _______________________________________________ linux-lvm mailing list linux-lvm@redhat.com https://listman.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [linux-lvm] Problem with partially activate logical volume 2022-07-26 8:50 ` Roger James @ 2022-07-27 10:49 ` Roger Heflin 2022-07-27 11:26 ` Roger James 0 siblings, 1 reply; 9+ messages in thread From: Roger Heflin @ 2022-07-27 10:49 UTC (permalink / raw) To: LVM general discussion and development I don't believe that is going to work. His issue is that the filesystem is refusing to work because of the missing data. man debugfs It will let you manually look at the metadata and structures of the ext2/3/4 fs. You will likely need to use the "-c" option. It will be very manual and you should probably read up on the fs structure a bit. A data recovery company could get most of the data back, but they charge 5k-10k per TB, so likely close to 100k US$. And the issues will be that 1/3 of the metadata was on the missing disk, and some of the data was on the missing disk. I was able to do debugfs /dev/sda2 (my /boot) and do an ls and list out the files and then do a dump <filename> /tmp/junk.out and copy out that file. So the issue will be writing up a script to do lses and find all of the files and dump all of the files to someplace else. On Wed, Jul 27, 2022 at 2:39 AM Roger James <roger@beardandsandals.co.uk> wrote: > > Try https://www.linuxsysadmins.com/recover-a-deleted-physical-volume/?amp > > On 26 July 2022 09:16:32 Ken Bass <daytooner@gmail.com> wrote: >> >> (fwiw: I am new to this list, so please bear with me.) >> >> Background: I have a very large (20TB) logical volume consisting of 3 drives. One of those drives unexpectedloy died (isn't that always the case :-)). The drive that failed happened to be the last PV. So I am assuming that there is still 2/3 of the data still intact and, to some extent, recoverable. Although, apparently the ext4 fs is not recognised. >> >> I activated the LV partially (via -P). But running any utility on that (eg: dumpe2fs, e2fsck, ...) I get many of these in dmesg: >> >> "Buffer I/O error on dev dm-0, logical block xxxxxxx, async page read." The thing is, the xxxxxxx block is on the missing drive/pv. >> >> I have also tried some recovery software, but eventually get these same messages, and the data recovered is not really useful. >> >> Please help! How can I get passed that dmesg error, and move on. 14TB recovered is better than 0. >> >> TIA >> ken >> >> >> _______________________________________________ >> linux-lvm mailing list >> linux-lvm@redhat.com >> https://listman.redhat.com/mailman/listinfo/linux-lvm >> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ >> > > _______________________________________________ > linux-lvm mailing list > linux-lvm@redhat.com > https://listman.redhat.com/mailman/listinfo/linux-lvm > read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ _______________________________________________ linux-lvm mailing list linux-lvm@redhat.com https://listman.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [linux-lvm] Problem with partially activate logical volume 2022-07-27 10:49 ` Roger Heflin @ 2022-07-27 11:26 ` Roger James 2022-08-03 21:31 ` Ken Bass 0 siblings, 1 reply; 9+ messages in thread From: Roger James @ 2022-07-27 11:26 UTC (permalink / raw) To: LVM general discussion and development [-- Attachment #1.1: Type: text/plain, Size: 3150 bytes --] The procedure outlined should at least get you back to a state where the lv is consistent but with blank sectors where the data is missing. I would suggest using dd to make a backup partition image. Then you can either work on that or the original to mend the fs. On 27 July 2022 11:50:07 Roger Heflin <rogerheflin@gmail.com> wrote: > I don't believe that is going to work. > > His issue is that the filesystem is refusing to work because of the > missing data. > > man debugfs > > It will let you manually look at the metadata and structures of the > ext2/3/4 fs. You will likely need to use the "-c" option. > > It will be very manual and you should probably read up on the fs > structure a bit. > > A data recovery company could get most of the data back, but they > charge 5k-10k per TB, so likely close to 100k US$. > > And the issues will be that 1/3 of the metadata was on the missing > disk, and some of the data was on the missing disk. > > I was able to do debugfs /dev/sda2 (my /boot) and do an ls and list > out the files and then do a dump <filename> /tmp/junk.out and copy out > that file. > > So the issue will be writing up a script to do lses and find all of > the files and dump all of the files to someplace else. > > On Wed, Jul 27, 2022 at 2:39 AM Roger James <roger@beardandsandals.co.uk> > wrote: >> >> Try https://www.linuxsysadmins.com/recover-a-deleted-physical-volume/?amp >> >> On 26 July 2022 09:16:32 Ken Bass <daytooner@gmail.com> wrote: >>> >>> (fwiw: I am new to this list, so please bear with me.) >>> >>> Background: I have a very large (20TB) logical volume consisting of 3 >>> drives. One of those drives unexpectedloy died (isn't that always the case >>> :-)). The drive that failed happened to be the last PV. So I am assuming >>> that there is still 2/3 of the data still intact and, to some extent, >>> recoverable. Although, apparently the ext4 fs is not recognised. >>> >>> I activated the LV partially (via -P). But running any utility on that (eg: >>> dumpe2fs, e2fsck, ...) I get many of these in dmesg: >>> >>> "Buffer I/O error on dev dm-0, logical block xxxxxxx, async page read." >>> The thing is, the xxxxxxx block is on the missing drive/pv. >>> >>> I have also tried some recovery software, but eventually get these same >>> messages, and the data recovered is not really useful. >>> >>> Please help! How can I get passed that dmesg error, and move on. 14TB >>> recovered is better than 0. >>> >>> TIA >>> ken >>> >>> >>> _______________________________________________ >>> linux-lvm mailing list >>> linux-lvm@redhat.com >>> https://listman.redhat.com/mailman/listinfo/linux-lvm >>> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ >>> >> >> _______________________________________________ >> linux-lvm mailing list >> linux-lvm@redhat.com >> https://listman.redhat.com/mailman/listinfo/linux-lvm >> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ > > _______________________________________________ > linux-lvm mailing list > linux-lvm@redhat.com > https://listman.redhat.com/mailman/listinfo/linux-lvm > read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ [-- Attachment #1.2: Type: text/html, Size: 5357 bytes --] [-- Attachment #2: Type: text/plain, Size: 202 bytes --] _______________________________________________ linux-lvm mailing list linux-lvm@redhat.com https://listman.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [linux-lvm] Problem with partially activate logical volume 2022-07-27 11:26 ` Roger James @ 2022-08-03 21:31 ` Ken Bass 2022-08-04 11:07 ` Roger Heflin 2022-08-05 12:03 ` Zdenek Kabelac 0 siblings, 2 replies; 9+ messages in thread From: Ken Bass @ 2022-08-03 21:31 UTC (permalink / raw) To: LVM general discussion and development [-- Attachment #1.1: Type: text/plain, Size: 4717 bytes --] That's pretty much it. Whenever any app attempts to read a block from the missing drive, I get the "Buffer I/O error" message. So, even though my recovery apps can scan the LV, marking blocks on the last drive as missing/unknown/etc., they can't display any recovered data - which I know does exist. Looking at raw data from the apps' scans, I can see directory entries, as well as files. I'm sure the inodes and bitmaps are still there for some of these, I just can't really reverse engineer and follow them through. But isn't that what the apps are supposed to do? As for debugfs: pretty much the same issue: in order to use it, I need to open the fs. But that (in debugfs) fails as well. So it can't help much. Unless I'm missing something about debugfs. The one thing I haven't tried is to use vgreduce to remove the missing PV; but that will also remove the LV as well, which is why I haven't tried it yet. Sorry I haven't replied sooner, but it takes a long time (days) to clone, then scan 16Tb... So, please any suggestions are greatly appreciated, as well as needed. ken (I know: No backup; got burned; it hurts; and I will now always have backups. 'Nuf said.) On Thu, Jul 28, 2022 at 3:12 AM Roger James <roger@beardandsandals.co.uk> wrote: > The procedure outlined should at least get you back to a state where the > lv is consistent but with blank sectors where the data is missing. I would > suggest using dd to make a backup partition image. Then you can either work > on that or the original to mend the fs. > > On 27 July 2022 11:50:07 Roger Heflin <rogerheflin@gmail.com> wrote: > > I don't believe that is going to work. >> >> His issue is that the filesystem is refusing to work because of the >> missing data. >> >> man debugfs >> >> It will let you manually look at the metadata and structures of the >> ext2/3/4 fs. You will likely need to use the "-c" option. >> >> It will be very manual and you should probably read up on the fs >> structure a bit. >> >> A data recovery company could get most of the data back, but they >> charge 5k-10k per TB, so likely close to 100k US$. >> >> And the issues will be that 1/3 of the metadata was on the missing >> disk, and some of the data was on the missing disk. >> >> I was able to do debugfs /dev/sda2 (my /boot) and do an ls and list >> out the files and then do a dump <filename> /tmp/junk.out and copy out >> that file. >> >> So the issue will be writing up a script to do lses and find all of >> the files and dump all of the files to someplace else. >> >> On Wed, Jul 27, 2022 at 2:39 AM Roger James <roger@beardandsandals.co.uk> >> wrote: >> >>> >>> Try >>> https://www.linuxsysadmins.com/recover-a-deleted-physical-volume/?amp >>> >>> On 26 July 2022 09:16:32 Ken Bass <daytooner@gmail.com> wrote: >>> >>>> >>>> (fwiw: I am new to this list, so please bear with me.) >>>> >>>> Background: I have a very large (20TB) logical volume consisting of 3 >>>> drives. One of those drives unexpectedloy died (isn't that always the case >>>> :-)). The drive that failed happened to be the last PV. So I am assuming >>>> that there is still 2/3 of the data still intact and, to some extent, >>>> recoverable. Although, apparently the ext4 fs is not recognised. >>>> >>>> I activated the LV partially (via -P). But running any utility on that >>>> (eg: dumpe2fs, e2fsck, ...) I get many of these in dmesg: >>>> >>>> "Buffer I/O error on dev dm-0, logical block xxxxxxx, async page read." >>>> The thing is, the xxxxxxx block is on the missing drive/pv. >>>> >>>> I have also tried some recovery software, but eventually get these same >>>> messages, and the data recovered is not really useful. >>>> >>>> Please help! How can I get passed that dmesg error, and move on. 14TB >>>> recovered is better than 0. >>>> >>>> TIA >>>> ken >>>> >>>> >>>> _______________________________________________ >>>> linux-lvm mailing list >>>> linux-lvm@redhat.com >>>> https://listman.redhat.com/mailman/listinfo/linux-lvm >>>> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ >>>> >>>> >>> _______________________________________________ >>> linux-lvm mailing list >>> linux-lvm@redhat.com >>> https://listman.redhat.com/mailman/listinfo/linux-lvm >>> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ >>> >> >> _______________________________________________ >> linux-lvm mailing list >> linux-lvm@redhat.com >> https://listman.redhat.com/mailman/listinfo/linux-lvm >> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ >> > > _______________________________________________ > linux-lvm mailing list > linux-lvm@redhat.com > https://listman.redhat.com/mailman/listinfo/linux-lvm > read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ > [-- Attachment #1.2: Type: text/html, Size: 8463 bytes --] [-- Attachment #2: Type: text/plain, Size: 202 bytes --] _______________________________________________ linux-lvm mailing list linux-lvm@redhat.com https://listman.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [linux-lvm] Problem with partially activate logical volume 2022-08-03 21:31 ` Ken Bass @ 2022-08-04 11:07 ` Roger Heflin 2022-08-05 12:03 ` Zdenek Kabelac 1 sibling, 0 replies; 9+ messages in thread From: Roger Heflin @ 2022-08-04 11:07 UTC (permalink / raw) To: LVM general discussion and development fsck might be able to fix it enough that debugfs and/or mount works, but it may also eliminate all the data, but since you have a clone and no data it is probably worth a shot. fsck -f -y will either get you some data or not and make you have to clone from the copy if you have something else to try. Did you try the "-c" option on debug fs? On Thu, Aug 4, 2022 at 2:07 AM Ken Bass <daytooner@gmail.com> wrote: > > > That's pretty much it. Whenever any app attempts to read a block from the missing drive, I get the "Buffer I/O error" message. So, even though my recovery apps can scan the LV, marking blocks on the last drive as missing/unknown/etc., they can't display any recovered data - which I know does exist. Looking at raw data from the apps' scans, I can see directory entries, as well as files. I'm sure the inodes and bitmaps are still there for some of these, I just can't really reverse engineer and follow them through. But isn't that what the apps are supposed to do? > > As for debugfs: pretty much the same issue: in order to use it, I need to open the fs. But that (in debugfs) fails as well. So it can't help much. Unless I'm missing something about debugfs. > > The one thing I haven't tried is to use vgreduce to remove the missing PV; but that will also remove the LV as well, which is why I haven't tried it yet. > > Sorry I haven't replied sooner, but it takes a long time (days) to clone, then scan 16Tb... > > So, please any suggestions are greatly appreciated, as well as needed. > > ken > > (I know: No backup; got burned; it hurts; and I will now always have backups. 'Nuf said.) > > > On Thu, Jul 28, 2022 at 3:12 AM Roger James <roger@beardandsandals.co.uk> wrote: >> >> The procedure outlined should at least get you back to a state where the lv is consistent but with blank sectors where the data is missing. I would suggest using dd to make a backup partition image. Then you can either work on that or the original to mend the fs. >> >> On 27 July 2022 11:50:07 Roger Heflin <rogerheflin@gmail.com> wrote: >> >>> I don't believe that is going to work. >>> >>> His issue is that the filesystem is refusing to work because of the >>> missing data. >>> >>> man debugfs >>> >>> It will let you manually look at the metadata and structures of the >>> ext2/3/4 fs. You will likely need to use the "-c" option. >>> >>> It will be very manual and you should probably read up on the fs >>> structure a bit. >>> >>> A data recovery company could get most of the data back, but they >>> charge 5k-10k per TB, so likely close to 100k US$. >>> >>> And the issues will be that 1/3 of the metadata was on the missing >>> disk, and some of the data was on the missing disk. >>> >>> I was able to do debugfs /dev/sda2 (my /boot) and do an ls and list >>> out the files and then do a dump <filename> /tmp/junk.out and copy out >>> that file. >>> >>> So the issue will be writing up a script to do lses and find all of >>> the files and dump all of the files to someplace else. >>> >>> On Wed, Jul 27, 2022 at 2:39 AM Roger James <roger@beardandsandals.co.uk> wrote: >>>> >>>> >>>> Try https://www.linuxsysadmins.com/recover-a-deleted-physical-volume/?amp >>>> >>>> On 26 July 2022 09:16:32 Ken Bass <daytooner@gmail.com> wrote: >>>>> >>>>> >>>>> (fwiw: I am new to this list, so please bear with me.) >>>>> >>>>> Background: I have a very large (20TB) logical volume consisting of 3 drives. One of those drives unexpectedloy died (isn't that always the case :-)). The drive that failed happened to be the last PV. So I am assuming that there is still 2/3 of the data still intact and, to some extent, recoverable. Although, apparently the ext4 fs is not recognised. >>>>> >>>>> I activated the LV partially (via -P). But running any utility on that (eg: dumpe2fs, e2fsck, ...) I get many of these in dmesg: >>>>> >>>>> "Buffer I/O error on dev dm-0, logical block xxxxxxx, async page read." The thing is, the xxxxxxx block is on the missing drive/pv. >>>>> >>>>> I have also tried some recovery software, but eventually get these same messages, and the data recovered is not really useful. >>>>> >>>>> Please help! How can I get passed that dmesg error, and move on. 14TB recovered is better than 0. >>>>> >>>>> TIA >>>>> ken >>>>> >>>>> >>>>> _______________________________________________ >>>>> linux-lvm mailing list >>>>> linux-lvm@redhat.com >>>>> https://listman.redhat.com/mailman/listinfo/linux-lvm >>>>> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ >>>>> >>>> >>>> _______________________________________________ >>>> linux-lvm mailing list >>>> linux-lvm@redhat.com >>>> https://listman.redhat.com/mailman/listinfo/linux-lvm >>>> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ >>> >>> >>> _______________________________________________ >>> linux-lvm mailing list >>> linux-lvm@redhat.com >>> https://listman.redhat.com/mailman/listinfo/linux-lvm >>> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ >> >> >> _______________________________________________ >> linux-lvm mailing list >> linux-lvm@redhat.com >> https://listman.redhat.com/mailman/listinfo/linux-lvm >> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ > > _______________________________________________ > linux-lvm mailing list > linux-lvm@redhat.com > https://listman.redhat.com/mailman/listinfo/linux-lvm > read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ _______________________________________________ linux-lvm mailing list linux-lvm@redhat.com https://listman.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [linux-lvm] Problem with partially activate logical volume 2022-08-03 21:31 ` Ken Bass 2022-08-04 11:07 ` Roger Heflin @ 2022-08-05 12:03 ` Zdenek Kabelac 2022-08-05 12:42 ` Ken Bass 1 sibling, 1 reply; 9+ messages in thread From: Zdenek Kabelac @ 2022-08-05 12:03 UTC (permalink / raw) To: LVM general discussion and development, Ken Bass Dne 03. 08. 22 v 23:31 Ken Bass napsal(a): > > That's pretty much it. Whenever any app attempts to read a block from the > missing drive, I get the "Buffer I/O error" message. So, even though my > recovery apps can scan the LV, marking blocks on the last drive as > missing/unknown/etc., they can't display any recovered data - which I know > does exist. Looking at raw data from the apps' scans, I can see directory > entries, as well as files. I'm sure the inodes and bitmaps are still there for > some of these, I just can't really reverse engineer and follow them through. > But isn't that what the apps are supposed to do? As mentioned by my previous email you shall *NOT* fix the partially activated device in-place - this will not lead to good result. User should copy the content to some valid storage device with the same size as he tries to recover. You can 'partially' activate device with "zero" filler instead of "error" (see the lvm.conf setting: missing_stripe_filler="...") - this way you will just 'read' zero for missing parts. Your another 2nd. option is to 'correct' the VG by filling missing PV with a new one with preferable zeroed content - so you will not read 'random' garbage in places this new PV will fill the space after your missing PV. Although even in this case - I'd still run 'fsck' on the snapshot created on top of such LV to give you another chance of recovery if you will pick a wrong answer (since fsck might be 'quite' interactive when doing such large-scale repair) > Sorry I haven't replied sooner, but it takes a long time (days) to clone, then > scan 16Tb... > > So, please any suggestions are greatly appreciated, as well as needed. > > ken > > (I know: No backup; got burned; it hurts; and I will now always have backups. > 'Nuf said.) Before you run your 'fsck' create a snapshot of your newly created 'backup' and make all the repair actions in the snapshots. Once you are 'satisfied' with 'repaired' filesystem you can then 'merge' snapshot back to your origin and use it. Regards Zdenek _______________________________________________ linux-lvm mailing list linux-lvm@redhat.com https://listman.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [linux-lvm] Problem with partially activate logical volume 2022-08-05 12:03 ` Zdenek Kabelac @ 2022-08-05 12:42 ` Ken Bass 0 siblings, 0 replies; 9+ messages in thread From: Ken Bass @ 2022-08-05 12:42 UTC (permalink / raw) To: Zdenek Kabelac; +Cc: LVM general discussion and development [-- Attachment #1.1: Type: text/plain, Size: 2414 bytes --] Zdenek: thanks. That makes more sense. I will try after (re-)cloning. Give it a day or so. Ken On Aug 5, 2022, 7:03 AM, at 7:03 AM, Zdenek Kabelac <zdenek.kabelac@gmail.com> wrote: >Dne 03. 08. 22 v 23:31 Ken Bass napsal(a): >> >> That's pretty much it. Whenever any app attempts to read a block from >the >> missing drive, I get the "Buffer I/O error" message. So, even though >my >> recovery apps can scan the LV, marking blocks on the last drive as >> missing/unknown/etc., they can't display any recovered data - which I >know >> does exist. Looking at raw data from the apps' scans, I can see >directory >> entries, as well as files. I'm sure the inodes and bitmaps are still >there for >> some of these, I just can't really reverse engineer and follow them >through. >> But isn't that what the apps are supposed to do? > >As mentioned by my previous email you shall *NOT* fix the partially >activated >device in-place - this will not lead to good result. > >User should copy the content to some valid storage device with the same >size >as he tries to recover. > >You can 'partially' activate device with "zero" filler instead of >"error" >(see the lvm.conf setting: missing_stripe_filler="...") - this way >you >will just 'read' zero for missing parts. > >Your another 2nd. option is to 'correct' the VG by filling missing PV >with a >new one with preferable zeroed content - so you will not read 'random' >garbage >in places this new PV will fill the space after your missing PV. >Although even in this case - I'd still run 'fsck' on the snapshot >created on >top of such LV to give you another chance of recovery if you will pick >a wrong >answer (since fsck might be 'quite' interactive when doing such >large-scale >repair) > > >> Sorry I haven't replied sooner, but it takes a long time (days) to >clone, then >> scan 16Tb... >> >> So, please any suggestions are greatly appreciated, as well as >needed. >> >> ken >> >> (I know: No backup; got burned; it hurts; and I will now always have >backups. >> 'Nuf said.) > >Before you run your 'fsck' create a snapshot of your newly created >'backup' >and make all the repair actions in the snapshots. > >Once you are 'satisfied' with 'repaired' filesystem you can then >'merge' >snapshot back to your origin and use it. > >Regards > >Zdenek [-- Attachment #1.2: Type: text/html, Size: 3035 bytes --] [-- Attachment #2: Type: text/plain, Size: 202 bytes --] _______________________________________________ linux-lvm mailing list linux-lvm@redhat.com https://listman.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [linux-lvm] Problem with partially activate logical volume 2022-07-25 14:48 [linux-lvm] Problem with partially activate logical volume Ken Bass 2022-07-26 8:50 ` Roger James @ 2022-07-27 19:56 ` Zdenek Kabelac 1 sibling, 0 replies; 9+ messages in thread From: Zdenek Kabelac @ 2022-07-27 19:56 UTC (permalink / raw) To: LVM general discussion and development, Ken Bass Dne 25. 07. 22 v 16:48 Ken Bass napsal(a): > (fwiw: I am new to this list, so please bear with me.) > > Background: I have a very large (20TB) logical volume consisting of 3 drives. > One of those drives unexpectedloy died (isn't that always the case :-)). The > drive that failed happened to be the last PV. So I am assuming that there is > still 2/3 of the data still intact and, to some extent, recoverable. Although, > apparently the ext4 fs is not recognised. > > I activated the LV partially (via -P). But running any utility on that (eg: > dumpe2fs, e2fsck, ...) I get many of these in dmesg: > > "Buffer I/O error on dev dm-0, logical block xxxxxxx, async page read." The > thing is, the xxxxxxx block is on the missing drive/pv. > > I have also tried some recovery software, but eventually get these same > messages, and the data recovered is not really useful. > > Please help! How can I get passed that dmesg error, and move on. 14TB > recovered is better than 0. Loosing such a large portion of device is always going to be a BIG problem. Filesystem spreads metadata all over the place - ExtX is somewhat better then BTree based FS like XFS,BTRFS and may give you lot of your data back. But that's why people should never underestimate how important is to keep reasonable fresh backups of their data - otherwise sooner or later there comes lesson like this one. What you could try is to 'add' new PV to VG and use space for taking snapshot of your LV you want to repair - but this is somewhat complicated as you need to 'fix' VG first - which would ideally need to have some size of storage you just lost - but this gives you then fairly easy way forward. (one way to do this is to use even 'virtual' storage over the loop back device - but that's for likely skilled admin.) Lvm2 partial activation is designed to be used in a way to ACTIVATE LV - and copy the content to better/safe/secure location and there start to recover the storage. Repairing storage in-place is usually straight road to hell - as there could be numerous way of recovery to try - but if your 1st. try actually destroys data even more, you can't retry with different strategy. So depending on how much money and time you want to put into recovery of your data there are several different strategies possible - considering storage space is relatively 'cheap' if your are data are really important. Regards Zdenek _______________________________________________ linux-lvm mailing list linux-lvm@redhat.com https://listman.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2022-08-08 7:30 UTC | newest] Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2022-07-25 14:48 [linux-lvm] Problem with partially activate logical volume Ken Bass 2022-07-26 8:50 ` Roger James 2022-07-27 10:49 ` Roger Heflin 2022-07-27 11:26 ` Roger James 2022-08-03 21:31 ` Ken Bass 2022-08-04 11:07 ` Roger Heflin 2022-08-05 12:03 ` Zdenek Kabelac 2022-08-05 12:42 ` Ken Bass 2022-07-27 19:56 ` Zdenek Kabelac
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).