That's pretty much it. Whenever any app attempts to read a block from the
missing drive, I get the "Buffer I/O error" message. So, even though my
recovery apps can scan the LV, marking blocks on the last drive as
missing/unknown/etc., they can't display any recovered data - which I know
does exist. Looking at raw data from the apps' scans, I can see directory
entries, as well as files. I'm sure the inodes and bitmaps are still there
for some of these, I just can't really reverse engineer and follow them
through. But isn't that what the apps are supposed to do?

As for debugfs: pretty much the same issue: in order to use it, I need to
open the fs. But that (in debugfs) fails as well. So it can't help much.
Unless I'm missing something about debugfs.

The one thing I haven't tried is to use vgreduce to remove the missing PV;
but that will also remove the LV as well, which is why I haven't tried it
yet.

Sorry I haven't replied sooner, but it takes a long time (days) to clone,
then scan 16Tb...

So, please any suggestions are greatly appreciated, as well as needed.

ken

(I know: No backup; got burned; it hurts; and I will now always have
backups. 'Nuf said.)


On Thu, Jul 28, 2022 at 3:12 AM Roger James <roger@beardandsandals.co.uk>
wrote:

> The procedure outlined should at least get you back to a state where the
> lv is consistent but with blank sectors where the data is missing. I would
> suggest using dd to make a backup partition image. Then you can either work
> on that or the original to mend the fs.
>
> On 27 July 2022 11:50:07 Roger Heflin <rogerheflin@gmail.com> wrote:
>
> I don't believe that is going to work.
>>
>> His issue is that the filesystem is refusing to work because of the
>> missing data.
>>
>> man debugfs
>>
>> It will let you manually look at the metadata and structures of the
>> ext2/3/4 fs.  You will likely need to use the "-c" option.
>>
>> It will be very manual and you should probably read up on the fs
>> structure a bit.
>>
>> A data recovery company could get most of the data back, but they
>> charge 5k-10k per TB, so likely close to 100k US$.
>>
>> And the issues will be that 1/3 of the metadata was on the missing
>> disk, and some of the data was on the missing disk.
>>
>> I was able to do debugfs /dev/sda2  (my /boot) and do an ls and list
>> out the files and then do a dump <filename> /tmp/junk.out and copy out
>> that file.
>>
>> So the issue will be writing up a script to do lses and find all of
>> the files and dump all of the files to someplace else.
>>
>> On Wed, Jul 27, 2022 at 2:39 AM Roger James <roger@beardandsandals.co.uk>
>> wrote:
>>
>>>
>>> Try
>>> https://www.linuxsysadmins.com/recover-a-deleted-physical-volume/?amp
>>>
>>> On 26 July 2022 09:16:32 Ken Bass <daytooner@gmail.com> wrote:
>>>
>>>>
>>>> (fwiw: I am new to this list, so please bear with me.)
>>>>
>>>> Background: I have a very large (20TB) logical volume consisting of 3
>>>> drives. One of those drives unexpectedloy died (isn't that always the case
>>>> :-)). The drive that failed happened to be the last PV. So I am assuming
>>>> that there is still 2/3 of the data still intact and, to some extent,
>>>> recoverable. Although, apparently the ext4 fs is not recognised.
>>>>
>>>> I activated the LV partially (via -P). But running any utility on that
>>>> (eg: dumpe2fs, e2fsck, ...) I get many of these  in dmesg:
>>>>
>>>> "Buffer I/O error on dev dm-0, logical block xxxxxxx, async page read."
>>>>  The thing is, the xxxxxxx block is on the missing drive/pv.
>>>>
>>>> I have also tried some recovery software, but eventually get these same
>>>> messages, and the data recovered is not really useful.
>>>>
>>>> Please help! How can I get passed that dmesg error, and move on. 14TB
>>>> recovered is better than 0.
>>>>
>>>> TIA
>>>> ken
>>>>
>>>>
>>>> _______________________________________________
>>>> linux-lvm mailing list
>>>> linux-lvm@redhat.com
>>>> https://listman.redhat.com/mailman/listinfo/linux-lvm
>>>> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>>>>
>>>>
>>> _______________________________________________
>>> linux-lvm mailing list
>>> linux-lvm@redhat.com
>>> https://listman.redhat.com/mailman/listinfo/linux-lvm
>>> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>>>
>>
>> _______________________________________________
>> linux-lvm mailing list
>> linux-lvm@redhat.com
>> https://listman.redhat.com/mailman/listinfo/linux-lvm
>> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>>
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://listman.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>