[linux-lvm] Problem with partially activate logical volume

[linux-lvm] Problem with partially activate logical volume

[Ken Bass]

[-- Attachment #1.1: Type: text/plain, Size: 909 bytes --]

(fwiw: I am new to this list, so please bear with me.)

Background: I have a very large (20TB) logical volume consisting of 3
drives. One of those drives unexpectedloy died (isn't that always the case
:-)). The drive that failed happened to be the last PV. So I am assuming
that there is still 2/3 of the data still intact and, to some extent,
recoverable. Although, apparently the ext4 fs is not recognised.

I activated the LV partially (via -P). But running any utility on that (eg:
dumpe2fs, e2fsck, ...) I get many of these  in dmesg:

"Buffer I/O error on dev dm-0, logical block xxxxxxx, async page read."
The thing is, the xxxxxxx block is on the missing drive/pv.

I have also tried some recovery software, but eventually get these same
messages, and the data recovered is not really useful.

Please help! How can I get passed that dmesg error, and move on. 14TB
recovered is better than 0.

TIA
ken

[-- Attachment #1.2: Type: text/html, Size: 1156 bytes --]

[-- Attachment #2: Type: text/plain, Size: 202 bytes --]

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] Problem with partially activate logical volume

[Roger James]

[-- Attachment #1.1: Type: text/plain, Size: 1311 bytes --]

Try https://www.linuxsysadmins.com/recover-a-deleted-physical-volume/?amp

On 26 July 2022 09:16:32 Ken Bass <daytooner@gmail.com> wrote:
> (fwiw: I am new to this list, so please bear with me.)
>
> Background: I have a very large (20TB) logical volume consisting of 3 
> drives. One of those drives unexpectedloy died (isn't that always the case 
> :-)). The drive that failed happened to be the last PV. So I am assuming 
> that there is still 2/3 of the data still intact and, to some extent, 
> recoverable. Although, apparently the ext4 fs is not recognised.
>
> I activated the LV partially (via -P). But running any utility on that (eg: 
> dumpe2fs, e2fsck, ...) I get many of these  in dmesg:
>
> "Buffer I/O error on dev dm-0, logical block xxxxxxx, async page read."  
> The thing is, the xxxxxxx block is on the missing drive/pv.
>
> I have also tried some recovery software, but eventually get these same 
> messages, and the data recovered is not really useful.
>
> Please help! How can I get passed that dmesg error, and move on. 14TB 
> recovered is better than 0.
>
> TIA
> ken
>
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://listman.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/


[-- Attachment #1.2: Type: text/html, Size: 2563 bytes --]

[-- Attachment #2: Type: text/plain, Size: 202 bytes --]

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] Problem with partially activate logical volume

[Roger Heflin]

I don't believe that is going to work.

His issue is that the filesystem is refusing to work because of the
missing data.

man debugfs

It will let you manually look at the metadata and structures of the
ext2/3/4 fs.  You will likely need to use the "-c" option.

It will be very manual and you should probably read up on the fs
structure a bit.

A data recovery company could get most of the data back, but they
charge 5k-10k per TB, so likely close to 100k US$.

And the issues will be that 1/3 of the metadata was on the missing
disk, and some of the data was on the missing disk.

I was able to do debugfs /dev/sda2  (my /boot) and do an ls and list
out the files and then do a dump <filename> /tmp/junk.out and copy out
that file.

So the issue will be writing up a script to do lses and find all of
the files and dump all of the files to someplace else.

On Wed, Jul 27, 2022 at 2:39 AM Roger James <roger@beardandsandals.co.uk> wrote:
>
> Try https://www.linuxsysadmins.com/recover-a-deleted-physical-volume/?amp
>
> On 26 July 2022 09:16:32 Ken Bass <daytooner@gmail.com> wrote:
>>
>> (fwiw: I am new to this list, so please bear with me.)
>>
>> Background: I have a very large (20TB) logical volume consisting of 3 drives. One of those drives unexpectedloy died (isn't that always the case :-)). The drive that failed happened to be the last PV. So I am assuming that there is still 2/3 of the data still intact and, to some extent, recoverable. Although, apparently the ext4 fs is not recognised.
>>
>> I activated the LV partially (via -P). But running any utility on that (eg: dumpe2fs, e2fsck, ...) I get many of these  in dmesg:
>>
>> "Buffer I/O error on dev dm-0, logical block xxxxxxx, async page read."  The thing is, the xxxxxxx block is on the missing drive/pv.
>>
>> I have also tried some recovery software, but eventually get these same messages, and the data recovered is not really useful.
>>
>> Please help! How can I get passed that dmesg error, and move on. 14TB recovered is better than 0.
>>
>> TIA
>> ken
>>
>>
>> _______________________________________________
>> linux-lvm mailing list
>> linux-lvm@redhat.com
>> https://listman.redhat.com/mailman/listinfo/linux-lvm
>> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>>
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://listman.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] Problem with partially activate logical volume

[Roger James]

[-- Attachment #1.1: Type: text/plain, Size: 3150 bytes --]

The procedure outlined should at least get you back to a state where the lv 
is consistent but with blank sectors where the data is missing. I would 
suggest using dd to make a backup partition image. Then you can either work 
on that or the original to mend the fs.

On 27 July 2022 11:50:07 Roger Heflin <rogerheflin@gmail.com> wrote:

> I don't believe that is going to work.
>
> His issue is that the filesystem is refusing to work because of the
> missing data.
>
> man debugfs
>
> It will let you manually look at the metadata and structures of the
> ext2/3/4 fs.  You will likely need to use the "-c" option.
>
> It will be very manual and you should probably read up on the fs
> structure a bit.
>
> A data recovery company could get most of the data back, but they
> charge 5k-10k per TB, so likely close to 100k US$.
>
> And the issues will be that 1/3 of the metadata was on the missing
> disk, and some of the data was on the missing disk.
>
> I was able to do debugfs /dev/sda2  (my /boot) and do an ls and list
> out the files and then do a dump <filename> /tmp/junk.out and copy out
> that file.
>
> So the issue will be writing up a script to do lses and find all of
> the files and dump all of the files to someplace else.
>
> On Wed, Jul 27, 2022 at 2:39 AM Roger James <roger@beardandsandals.co.uk> 
> wrote:
>>
>> Try https://www.linuxsysadmins.com/recover-a-deleted-physical-volume/?amp
>>
>> On 26 July 2022 09:16:32 Ken Bass <daytooner@gmail.com> wrote:
>>>
>>> (fwiw: I am new to this list, so please bear with me.)
>>>
>>> Background: I have a very large (20TB) logical volume consisting of 3 
>>> drives. One of those drives unexpectedloy died (isn't that always the case 
>>> :-)). The drive that failed happened to be the last PV. So I am assuming 
>>> that there is still 2/3 of the data still intact and, to some extent, 
>>> recoverable. Although, apparently the ext4 fs is not recognised.
>>>
>>> I activated the LV partially (via -P). But running any utility on that (eg: 
>>> dumpe2fs, e2fsck, ...) I get many of these  in dmesg:
>>>
>>> "Buffer I/O error on dev dm-0, logical block xxxxxxx, async page read."  
>>> The thing is, the xxxxxxx block is on the missing drive/pv.
>>>
>>> I have also tried some recovery software, but eventually get these same 
>>> messages, and the data recovered is not really useful.
>>>
>>> Please help! How can I get passed that dmesg error, and move on. 14TB 
>>> recovered is better than 0.
>>>
>>> TIA
>>> ken
>>>
>>>
>>> _______________________________________________
>>> linux-lvm mailing list
>>> linux-lvm@redhat.com
>>> https://listman.redhat.com/mailman/listinfo/linux-lvm
>>> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>>>
>>
>> _______________________________________________
>> linux-lvm mailing list
>> linux-lvm@redhat.com
>> https://listman.redhat.com/mailman/listinfo/linux-lvm
>> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://listman.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/


[-- Attachment #1.2: Type: text/html, Size: 5357 bytes --]

[-- Attachment #2: Type: text/plain, Size: 202 bytes --]

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] Problem with partially activate logical volume

[Zdenek Kabelac]

Dne 25. 07. 22 v 16:48 Ken Bass napsal(a):
> (fwiw: I am new to this list, so please bear with me.)
> 
> Background: I have a very large (20TB) logical volume consisting of 3 drives. 
> One of those drives unexpectedloy died (isn't that always the case :-)). The 
> drive that failed happened to be the last PV. So I am assuming that there is 
> still 2/3 of the data still intact and, to some extent, recoverable. Although, 
> apparently the ext4 fs is not recognised.
> 
> I activated the LV partially (via -P). But running any utility on that (eg: 
> dumpe2fs, e2fsck, ...) I get many of these  in dmesg:
> 
> "Buffer I/O error on dev dm-0, logical block xxxxxxx, async page read."  The 
> thing is, the xxxxxxx block is on the missing drive/pv.
> 
> I have also tried some recovery software, but eventually get these same 
> messages, and the data recovered is not really useful.
> 
> Please help! How can I get passed that dmesg error, and move on. 14TB 
> recovered is better than 0.


Loosing such a large portion of device is always going to be a BIG problem.
Filesystem spreads metadata all over the place -  ExtX is somewhat better then 
BTree based FS like  XFS,BTRFS and may give you lot of your data back.

But that's why people should never underestimate how important is to keep 
reasonable fresh backups of their data - otherwise sooner or later there comes 
lesson like this one.

What you could try is to 'add' new PV to VG and use space for taking snapshot 
of your LV you want to repair - but this is somewhat complicated as you need 
to 'fix' VG first - which would ideally need to have some size of storage you 
just lost - but this gives you then fairly easy way forward.
(one way to do this is to use even 'virtual' storage over the loop back device 
- but that's for likely skilled admin.)

Lvm2 partial activation is designed to be used in a way to ACTIVATE LV - and 
copy the content to better/safe/secure location and there start to recover the 
storage.

Repairing storage in-place is usually straight road to hell - as there could 
be numerous way of recovery to try - but if your 1st. try actually destroys 
data even more, you can't retry with different strategy.

So depending on how much money and time you want to put into recovery of your 
data there are several different strategies possible  - considering storage 
space is relatively 'cheap' if your are data are really important.

Regards

Zdenek

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] Problem with partially activate logical volume

[Ken Bass]

[-- Attachment #1.1: Type: text/plain, Size: 4717 bytes --]

That's pretty much it. Whenever any app attempts to read a block from the
missing drive, I get the "Buffer I/O error" message. So, even though my
recovery apps can scan the LV, marking blocks on the last drive as
missing/unknown/etc., they can't display any recovered data - which I know
does exist. Looking at raw data from the apps' scans, I can see directory
entries, as well as files. I'm sure the inodes and bitmaps are still there
for some of these, I just can't really reverse engineer and follow them
through. But isn't that what the apps are supposed to do?

As for debugfs: pretty much the same issue: in order to use it, I need to
open the fs. But that (in debugfs) fails as well. So it can't help much.
Unless I'm missing something about debugfs.

The one thing I haven't tried is to use vgreduce to remove the missing PV;
but that will also remove the LV as well, which is why I haven't tried it
yet.

Sorry I haven't replied sooner, but it takes a long time (days) to clone,
then scan 16Tb...

So, please any suggestions are greatly appreciated, as well as needed.

ken

(I know: No backup; got burned; it hurts; and I will now always have
backups. 'Nuf said.)


On Thu, Jul 28, 2022 at 3:12 AM Roger James <roger@beardandsandals.co.uk>
wrote:

> The procedure outlined should at least get you back to a state where the
> lv is consistent but with blank sectors where the data is missing. I would
> suggest using dd to make a backup partition image. Then you can either work
> on that or the original to mend the fs.
>
> On 27 July 2022 11:50:07 Roger Heflin <rogerheflin@gmail.com> wrote:
>
> I don't believe that is going to work.
>>
>> His issue is that the filesystem is refusing to work because of the
>> missing data.
>>
>> man debugfs
>>
>> It will let you manually look at the metadata and structures of the
>> ext2/3/4 fs.  You will likely need to use the "-c" option.
>>
>> It will be very manual and you should probably read up on the fs
>> structure a bit.
>>
>> A data recovery company could get most of the data back, but they
>> charge 5k-10k per TB, so likely close to 100k US$.
>>
>> And the issues will be that 1/3 of the metadata was on the missing
>> disk, and some of the data was on the missing disk.
>>
>> I was able to do debugfs /dev/sda2  (my /boot) and do an ls and list
>> out the files and then do a dump <filename> /tmp/junk.out and copy out
>> that file.
>>
>> So the issue will be writing up a script to do lses and find all of
>> the files and dump all of the files to someplace else.
>>
>> On Wed, Jul 27, 2022 at 2:39 AM Roger James <roger@beardandsandals.co.uk>
>> wrote:
>>
>>>
>>> Try
>>> https://www.linuxsysadmins.com/recover-a-deleted-physical-volume/?amp
>>>
>>> On 26 July 2022 09:16:32 Ken Bass <daytooner@gmail.com> wrote:
>>>
>>>>
>>>> (fwiw: I am new to this list, so please bear with me.)
>>>>
>>>> Background: I have a very large (20TB) logical volume consisting of 3
>>>> drives. One of those drives unexpectedloy died (isn't that always the case
>>>> :-)). The drive that failed happened to be the last PV. So I am assuming
>>>> that there is still 2/3 of the data still intact and, to some extent,
>>>> recoverable. Although, apparently the ext4 fs is not recognised.
>>>>
>>>> I activated the LV partially (via -P). But running any utility on that
>>>> (eg: dumpe2fs, e2fsck, ...) I get many of these  in dmesg:
>>>>
>>>> "Buffer I/O error on dev dm-0, logical block xxxxxxx, async page read."
>>>>  The thing is, the xxxxxxx block is on the missing drive/pv.
>>>>
>>>> I have also tried some recovery software, but eventually get these same
>>>> messages, and the data recovered is not really useful.
>>>>
>>>> Please help! How can I get passed that dmesg error, and move on. 14TB
>>>> recovered is better than 0.
>>>>
>>>> TIA
>>>> ken
>>>>
>>>>
>>>> _______________________________________________
>>>> linux-lvm mailing list
>>>> linux-lvm@redhat.com
>>>> https://listman.redhat.com/mailman/listinfo/linux-lvm
>>>> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>>>>
>>>>
>>> _______________________________________________
>>> linux-lvm mailing list
>>> linux-lvm@redhat.com
>>> https://listman.redhat.com/mailman/listinfo/linux-lvm
>>> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>>>
>>
>> _______________________________________________
>> linux-lvm mailing list
>> linux-lvm@redhat.com
>> https://listman.redhat.com/mailman/listinfo/linux-lvm
>> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>>
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://listman.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>

[-- Attachment #1.2: Type: text/html, Size: 8463 bytes --]

[-- Attachment #2: Type: text/plain, Size: 202 bytes --]

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] Problem with partially activate logical volume

[Roger Heflin]

fsck might be able to fix it enough that debugfs and/or mount works,
but it may also eliminate all the data, but since you have a clone and
no data it is probably worth a shot.

fsck -f -y   will either get you some data or not and make you have to
clone from the copy if you have something else to  try.

Did you try the "-c" option on debug fs?

On Thu, Aug 4, 2022 at 2:07 AM Ken Bass <daytooner@gmail.com> wrote:
>
>
> That's pretty much it. Whenever any app attempts to read a block from the missing drive, I get the "Buffer I/O error" message. So, even though my recovery apps can scan the LV, marking blocks on the last drive as missing/unknown/etc., they can't display any recovered data - which I know does exist. Looking at raw data from the apps' scans, I can see directory entries, as well as files. I'm sure the inodes and bitmaps are still there for some of these, I just can't really reverse engineer and follow them through. But isn't that what the apps are supposed to do?
>
> As for debugfs: pretty much the same issue: in order to use it, I need to open the fs. But that (in debugfs) fails as well. So it can't help much. Unless I'm missing something about debugfs.
>
> The one thing I haven't tried is to use vgreduce to remove the missing PV; but that will also remove the LV as well, which is why I haven't tried it yet.
>
> Sorry I haven't replied sooner, but it takes a long time (days) to clone, then scan 16Tb...
>
> So, please any suggestions are greatly appreciated, as well as needed.
>
> ken
>
> (I know: No backup; got burned; it hurts; and I will now always have backups. 'Nuf said.)
>
>
> On Thu, Jul 28, 2022 at 3:12 AM Roger James <roger@beardandsandals.co.uk> wrote:
>>
>> The procedure outlined should at least get you back to a state where the lv is consistent but with blank sectors where the data is missing. I would suggest using dd to make a backup partition image. Then you can either work on that or the original to mend the fs.
>>
>> On 27 July 2022 11:50:07 Roger Heflin <rogerheflin@gmail.com> wrote:
>>
>>> I don't believe that is going to work.
>>>
>>> His issue is that the filesystem is refusing to work because of the
>>> missing data.
>>>
>>> man debugfs
>>>
>>> It will let you manually look at the metadata and structures of the
>>> ext2/3/4 fs.  You will likely need to use the "-c" option.
>>>
>>> It will be very manual and you should probably read up on the fs
>>> structure a bit.
>>>
>>> A data recovery company could get most of the data back, but they
>>> charge 5k-10k per TB, so likely close to 100k US$.
>>>
>>> And the issues will be that 1/3 of the metadata was on the missing
>>> disk, and some of the data was on the missing disk.
>>>
>>> I was able to do debugfs /dev/sda2  (my /boot) and do an ls and list
>>> out the files and then do a dump <filename> /tmp/junk.out and copy out
>>> that file.
>>>
>>> So the issue will be writing up a script to do lses and find all of
>>> the files and dump all of the files to someplace else.
>>>
>>> On Wed, Jul 27, 2022 at 2:39 AM Roger James <roger@beardandsandals.co.uk> wrote:
>>>>
>>>>
>>>> Try https://www.linuxsysadmins.com/recover-a-deleted-physical-volume/?amp
>>>>
>>>> On 26 July 2022 09:16:32 Ken Bass <daytooner@gmail.com> wrote:
>>>>>
>>>>>
>>>>> (fwiw: I am new to this list, so please bear with me.)
>>>>>
>>>>> Background: I have a very large (20TB) logical volume consisting of 3 drives. One of those drives unexpectedloy died (isn't that always the case :-)). The drive that failed happened to be the last PV. So I am assuming that there is still 2/3 of the data still intact and, to some extent, recoverable. Although, apparently the ext4 fs is not recognised.
>>>>>
>>>>> I activated the LV partially (via -P). But running any utility on that (eg: dumpe2fs, e2fsck, ...) I get many of these  in dmesg:
>>>>>
>>>>> "Buffer I/O error on dev dm-0, logical block xxxxxxx, async page read."  The thing is, the xxxxxxx block is on the missing drive/pv.
>>>>>
>>>>> I have also tried some recovery software, but eventually get these same messages, and the data recovered is not really useful.
>>>>>
>>>>> Please help! How can I get passed that dmesg error, and move on. 14TB recovered is better than 0.
>>>>>
>>>>> TIA
>>>>> ken
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> linux-lvm mailing list
>>>>> linux-lvm@redhat.com
>>>>> https://listman.redhat.com/mailman/listinfo/linux-lvm
>>>>> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>>>>>
>>>>
>>>> _______________________________________________
>>>> linux-lvm mailing list
>>>> linux-lvm@redhat.com
>>>> https://listman.redhat.com/mailman/listinfo/linux-lvm
>>>> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>>>
>>>
>>> _______________________________________________
>>> linux-lvm mailing list
>>> linux-lvm@redhat.com
>>> https://listman.redhat.com/mailman/listinfo/linux-lvm
>>> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>>
>>
>> _______________________________________________
>> linux-lvm mailing list
>> linux-lvm@redhat.com
>> https://listman.redhat.com/mailman/listinfo/linux-lvm
>> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://listman.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] Problem with partially activate logical volume

[Zdenek Kabelac]

Dne 03. 08. 22 v 23:31 Ken Bass napsal(a):
> 
> That's pretty much it. Whenever any app attempts to read a block from the 
> missing drive, I get the "Buffer I/O error" message. So, even though my 
> recovery apps can scan the LV, marking blocks on the last drive as 
> missing/unknown/etc., they can't display any recovered data - which I know 
> does exist. Looking at raw data from the apps' scans, I can see directory 
> entries, as well as files. I'm sure the inodes and bitmaps are still there for 
> some of these, I just can't really reverse engineer and follow them through. 
> But isn't that what the apps are supposed to do?

As mentioned by my previous email you shall *NOT* fix the partially activated 
device in-place - this will not lead to good result.

User should copy the content to some valid storage device with the same size 
as he tries to recover.

You can 'partially' activate device with  "zero"  filler instead of "error" 
(see the  lvm.conf setting:     missing_stripe_filler="...") - this way you 
will just 'read' zero for missing parts.

Your another 2nd. option is to 'correct' the VG by filling  missing PV with a 
new one with preferable zeroed content - so you will not read 'random' garbage 
in places this new PV will fill the space after your missing PV.
Although even in this case - I'd still run  'fsck' on the snapshot created on 
top of such LV to give you another chance of recovery if you will pick a wrong 
answer  (since fsck might be 'quite' interactive when doing such large-scale 
repair)


> Sorry I haven't replied sooner, but it takes a long time (days) to clone, then 
> scan 16Tb...
> 
> So, please any suggestions are greatly appreciated, as well as needed.
> 
> ken
> 
> (I know: No backup; got burned; it hurts; and I will now always have backups. 
> 'Nuf said.)

Before you run your 'fsck' create a snapshot of your newly created 'backup' 
and make all the repair actions in the snapshots.

Once you are 'satisfied' with 'repaired'  filesystem you can then 'merge' 
snapshot back to your origin and use it.

Regards

Zdenek

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] Problem with partially activate logical volume

[Ken Bass]

[-- Attachment #1.1: Type: text/plain, Size: 2414 bytes --]

Zdenek: thanks. That makes more sense. I will try after (re-)cloning. Give it  a day or so. 

Ken 


On Aug 5, 2022, 7:03 AM, at 7:03 AM, Zdenek Kabelac <zdenek.kabelac@gmail.com> wrote:
>Dne 03. 08. 22 v 23:31 Ken Bass napsal(a):
>> 
>> That's pretty much it. Whenever any app attempts to read a block from
>the 
>> missing drive, I get the "Buffer I/O error" message. So, even though
>my 
>> recovery apps can scan the LV, marking blocks on the last drive as 
>> missing/unknown/etc., they can't display any recovered data - which I
>know 
>> does exist. Looking at raw data from the apps' scans, I can see
>directory 
>> entries, as well as files. I'm sure the inodes and bitmaps are still
>there for 
>> some of these, I just can't really reverse engineer and follow them
>through. 
>> But isn't that what the apps are supposed to do?
>
>As mentioned by my previous email you shall *NOT* fix the partially
>activated 
>device in-place - this will not lead to good result.
>
>User should copy the content to some valid storage device with the same
>size 
>as he tries to recover.
>
>You can 'partially' activate device with  "zero"  filler instead of
>"error" 
>(see the  lvm.conf setting:     missing_stripe_filler="...") - this way
>you 
>will just 'read' zero for missing parts.
>
>Your another 2nd. option is to 'correct' the VG by filling  missing PV
>with a 
>new one with preferable zeroed content - so you will not read 'random'
>garbage 
>in places this new PV will fill the space after your missing PV.
>Although even in this case - I'd still run  'fsck' on the snapshot
>created on 
>top of such LV to give you another chance of recovery if you will pick
>a wrong 
>answer  (since fsck might be 'quite' interactive when doing such
>large-scale 
>repair)
>
>
>> Sorry I haven't replied sooner, but it takes a long time (days) to
>clone, then 
>> scan 16Tb...
>> 
>> So, please any suggestions are greatly appreciated, as well as
>needed.
>> 
>> ken
>> 
>> (I know: No backup; got burned; it hurts; and I will now always have
>backups. 
>> 'Nuf said.)
>
>Before you run your 'fsck' create a snapshot of your newly created
>'backup' 
>and make all the repair actions in the snapshots.
>
>Once you are 'satisfied' with 'repaired'  filesystem you can then
>'merge' 
>snapshot back to your origin and use it.
>
>Regards
>
>Zdenek

[-- Attachment #1.2: Type: text/html, Size: 3035 bytes --]

[-- Attachment #2: Type: text/plain, Size: 202 bytes --]

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/