linux-ext4.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* ext4 recovery
@ 2020-01-14 16:03 James Courtier-Dutton
  2020-01-14 20:35 ` Andreas Dilger
  2020-01-14 21:25 ` Theodore Y. Ts'o
  0 siblings, 2 replies; 6+ messages in thread
From: James Courtier-Dutton @ 2020-01-14 16:03 UTC (permalink / raw)
  To: linux-ext4

Hi,

Say I started with 1 disk using LVM with an ext4 partition.
I then added another disk. Added it to the LVM group, expanded the
ext4 partition to then fill 2 disks.
I then added another disk. Added it to the LVM group, expanded the
ext4 partition to then fill 3 disks.

One of the disk has now failed.
Are there any tools available for ext4 that could help recover this?
Note, I am a single user, not a company, so there is no money to give
to a data recovery company, so I wish to try myself.

Is there a mount option that will mount an ext4 partition even if
there are lots of sector errors, or a missing disk. So, at least some
of the files will be recoverable.
Or a tool that will scan the disk for super blocks, then find inodes
and then find the sector location of the files, even if they don't
have filenames.

Kind Regards

James

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: ext4 recovery
  2020-01-14 16:03 ext4 recovery James Courtier-Dutton
@ 2020-01-14 20:35 ` Andreas Dilger
  2020-01-14 21:25 ` Theodore Y. Ts'o
  1 sibling, 0 replies; 6+ messages in thread
From: Andreas Dilger @ 2020-01-14 20:35 UTC (permalink / raw)
  To: James Courtier-Dutton; +Cc: linux-ext4

[-- Attachment #1: Type: text/plain, Size: 1955 bytes --]

On Jan 14, 2020, at 9:03 AM, James Courtier-Dutton <james.dutton@gmail.com> wrote:
> 
> Hi,
> 
> Say I started with 1 disk using LVM with an ext4 partition.
> I then added another disk. Added it to the LVM group, expanded the
> ext4 partition to then fill 2 disks.
> I then added another disk. Added it to the LVM group, expanded the
> ext4 partition to then fill 3 disks.
> 
> One of the disk has now failed.
> Are there any tools available for ext4 that could help recover this?
> Note, I am a single user, not a company, so there is no money to give
> to a data recovery company, so I wish to try myself.
> 
> Is there a mount option that will mount an ext4 partition even if
> there are lots of sector errors, or a missing disk. So, at least some
> of the files will be recoverable.
> Or a tool that will scan the disk for super blocks, then find inodes
> and then find the sector location of the files, even if they don't
> have filenames.

The first thing I would do is get new disk(s) to make a full backup of
the existing disks, so that whatever you do for recovery doesn't cause
further data loss.

There are two approaches to recovering the data:
- add in a replacement disk and re-add the missing LVs that fill in the
  lost space of the failed disk, then run e2fsck to rebuild the metadata
  for the missing disk.  I'm not sure how easy/hard this is with LVM.
- use a block-level scanner to recover files on disk, which is useful for
  e.g. .jpg images, or other files with good magic signatures and internal
  data integrity checks.

You didn't mention whether it was the first, second, or third disk that
failed.  If it is the third disk, that may not even need to be replaced,
since you could set the filesystem size to be the end of the second disk
and then e2fsck will repair the bad block/inode references.  If it is the
first or second disk then it is more complex.


Cheers, Andreas






[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 873 bytes --]

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: ext4 recovery
  2020-01-14 16:03 ext4 recovery James Courtier-Dutton
  2020-01-14 20:35 ` Andreas Dilger
@ 2020-01-14 21:25 ` Theodore Y. Ts'o
  2020-01-20 22:06   ` James Courtier-Dutton
  1 sibling, 1 reply; 6+ messages in thread
From: Theodore Y. Ts'o @ 2020-01-14 21:25 UTC (permalink / raw)
  To: James Courtier-Dutton; +Cc: linux-ext4

On Tue, Jan 14, 2020 at 04:03:53PM +0000, James Courtier-Dutton wrote:
> 
> Say I started with 1 disk using LVM with an ext4 partition.
> I then added another disk. Added it to the LVM group, expanded the
> ext4 partition to then fill 2 disks.
> I then added another disk. Added it to the LVM group, expanded the
> ext4 partition to then fill 3 disks.

Where you using RAID 0, or some more advanced RAID level?

> One of the disk has now failed.

How has it failed?  It is dead dead dead?  Or are there a large number
of sector errors?

> Are there any tools available for ext4 that could help recover this?
> Note, I am a single user, not a company, so there is no money to give
> to a data recovery company, so I wish to try myself.

How valuable is your data?  The first thing I would recommend, if your
data is worth it (and only you can make that decision) is to create a
new RAID set (using larger disks if that helps reduce the price) so
you can make an block-level image backup using the dd_rescue program.

If you can, then run e2fsck on the backup copy, and then see what you
can recover from that image set.  This will save time (how much is
your time worth?) and perhaps increase the amount of data you can
recover (how much is your data worth?).

					- Ted

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: ext4 recovery
  2020-01-14 21:25 ` Theodore Y. Ts'o
@ 2020-01-20 22:06   ` James Courtier-Dutton
  2020-01-21  2:20     ` Andreas Dilger
  0 siblings, 1 reply; 6+ messages in thread
From: James Courtier-Dutton @ 2020-01-20 22:06 UTC (permalink / raw)
  To: Theodore Y. Ts'o, Andreas Dilger; +Cc: linux-ext4

On Tue, 14 Jan 2020 at 21:25, Theodore Y. Ts'o <tytso@mit.edu> wrote:
>
> On Tue, Jan 14, 2020 at 04:03:53PM +0000, James Courtier-Dutton wrote:
> >
> > Say I started with 1 disk using LVM with an ext4 partition.
> > I then added another disk. Added it to the LVM group, expanded the
> > ext4 partition to then fill 2 disks.
> > I then added another disk. Added it to the LVM group, expanded the
> > ext4 partition to then fill 3 disks.
>
> Where you using RAID 0, or some more advanced RAID level?
>
> > One of the disk has now failed.
>
> How has it failed?  It is dead dead dead?  Or are there a large number
> of sector errors?
>
> > Are there any tools available for ext4 that could help recover this?
> > Note, I am a single user, not a company, so there is no money to give
> > to a data recovery company, so I wish to try myself.
>
> How valuable is your data?  The first thing I would recommend, if your
> data is worth it (and only you can make that decision) is to create a
> new RAID set (using larger disks if that helps reduce the price) so
> you can make an block-level image backup using the dd_rescue program.
>
> If you can, then run e2fsck on the backup copy, and then see what you
> can recover from that image set.  This will save time (how much is
> your time worth?) and perhaps increase the amount of data you can
> recover (how much is your data worth?).
>
>                                         - Ted

Hi,

Thank you all for the help.
I have made some progress.
Disk 1 is 100% OK.
Disk 2 did dd_rescue and recovered about 30% of the sectors.
Disk 3 is 100% OK.

I have made images of all of that. LVM worked out what order they went
in, and ext4 even let me mount the LVM volume.
I have managed to recover a lot of the data, so thank you for the hints.
What I would like to do now is find out which files were on Disk 2.
I have found the fiemap  IOCTL that gives me the logical sectors for
the file itself.
How do I convert the value from the IOCTL into a physical sector, or
at least an offset into the LVM volume?
I can then work at that sector 0 to A is on Disk 1,  A to B is on Disk
2, and B to C is on Disk 3.

For example, this is how it is done on btrfs. What is the equivalent on ext4?

$ sudo filefrag -v UEFI_Spec_2_8_final.pdf
Filesystem type is: 9123683e
File size of UEFI_Spec_2_8_final.pdf is 18586279 (4538 blocks of 4096 bytes)
 ext:     logical_offset:        physical_offset: length:   expected: flags:
   0:        0..    4537:    9798022..   9802559:   4538:             last,eof
UEFI_Spec_2_8_final.pdf: 1 extent found
$ sudo btrfs-map-logical -l $[9798022*4096] /dev/nvme0n1p7
mirror 1 logical 40132698112 physical 31475654656 device /dev/nvme0n1p7
$ sudo dd if=/dev/nvme0n1p7 bs=1
skip=31475654656 count=64 2>/dev/null | hexdump -C
00000000  25 50 44 46 2d 31 2e 36  0d 25 e2 e3 cf d3 0d 0a  |%PDF-1.6.%......|
00000010  33 30 35 35 31 39 20 30  20 6f 62 6a 0d 3c 3c 2f  |305519 0 obj.<</|
00000020  46 69 6c 74 65 72 2f 46  6c 61 74 65 44 65 63 6f  |Filter/FlateDeco|
00000030  64 65 2f 46 69 72 73 74  20 31 31 37 39 2f 4c 65  |de/First 1179/Le|
00000040


Kind Regards

James

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: ext4 recovery
  2020-01-20 22:06   ` James Courtier-Dutton
@ 2020-01-21  2:20     ` Andreas Dilger
  2020-04-05 16:10       ` James Courtier-Dutton
  0 siblings, 1 reply; 6+ messages in thread
From: Andreas Dilger @ 2020-01-21  2:20 UTC (permalink / raw)
  To: James Courtier-Dutton; +Cc: Theodore Y. Ts'o, linux-ext4

You can use "filefrag -k" to get units of KB instead of blocks for the physical offsets of the files, and "lvdisplay -m /dev/XXX" to show the mapping of LVs to disks (not sure if you can get it to print different units, or just LE number and you gave to multiply by the LE size).  Hopefully the LV was layer out linearly and is just a matter of computing "> X and < Y" to determine the affected range. 

Note that it is likely that some files which were on the second disk will not show anything from filefrag since the inodes were zeroed out by dd_rescue. Some files on the second disk that _do_ show blocks that were allocated on that disk may just have zeroes for the data. 

Cheers, Andreas

> On Jan 20, 2020, at 15:07, James Courtier-Dutton <james.dutton@gmail.com> wrote:
> 
> On Tue, 14 Jan 2020 at 21:25, Theodore Y. Ts'o <tytso@mit.edu> wrote:
>> 
>>> On Tue, Jan 14, 2020 at 04:03:53PM +0000, James Courtier-Dutton wrote:
>>> 
>>> Say I started with 1 disk using LVM with an ext4 partition.
>>> I then added another disk. Added it to the LVM group, expanded the
>>> ext4 partition to then fill 2 disks.
>>> I then added another disk. Added it to the LVM group, expanded the
>>> ext4 partition to then fill 3 disks.
>> 
>> Where you using RAID 0, or some more advanced RAID level?
>> 
>>> One of the disk has now failed.
>> 
>> How has it failed?  It is dead dead dead?  Or are there a large number
>> of sector errors?
>> 
>>> Are there any tools available for ext4 that could help recover this?
>>> Note, I am a single user, not a company, so there is no money to give
>>> to a data recovery company, so I wish to try myself.
>> 
>> How valuable is your data?  The first thing I would recommend, if your
>> data is worth it (and only you can make that decision) is to create a
>> new RAID set (using larger disks if that helps reduce the price) so
>> you can make an block-level image backup using the dd_rescue program.
>> 
>> If you can, then run e2fsck on the backup copy, and then see what you
>> can recover from that image set.  This will save time (how much is
>> your time worth?) and perhaps increase the amount of data you can
>> recover (how much is your data worth?).
>> 
>>                                        - Ted
> 
> Hi,
> 
> Thank you all for the help.
> I have made some progress.
> Disk 1 is 100% OK.
> Disk 2 did dd_rescue and recovered about 30% of the sectors.
> Disk 3 is 100% OK.
> 
> I have made images of all of that. LVM worked out what order they went
> in, and ext4 even let me mount the LVM volume.
> I have managed to recover a lot of the data, so thank you for the hints.
> What I would like to do now is find out which files were on Disk 2.
> I have found the fiemap  IOCTL that gives me the logical sectors for
> the file itself.
> How do I convert the value from the IOCTL into a physical sector, or
> at least an offset into the LVM volume?
> I can then work at that sector 0 to A is on Disk 1,  A to B is on Disk
> 2, and B to C is on Disk 3.
> 
> For example, this is how it is done on btrfs. What is the equivalent on ext4?
> 
> $ sudo filefrag -v UEFI_Spec_2_8_final.pdf
> Filesystem type is: 9123683e
> File size of UEFI_Spec_2_8_final.pdf is 18586279 (4538 blocks of 4096 bytes)
> ext:     logical_offset:        physical_offset: length:   expected: flags:
>   0:        0..    4537:    9798022..   9802559:   4538:             last,eof
> UEFI_Spec_2_8_final.pdf: 1 extent found
> $ sudo btrfs-map-logical -l $[9798022*4096] /dev/nvme0n1p7
> mirror 1 logical 40132698112 physical 31475654656 device /dev/nvme0n1p7
> $ sudo dd if=/dev/nvme0n1p7 bs=1
> skip=31475654656 count=64 2>/dev/null | hexdump -C
> 00000000  25 50 44 46 2d 31 2e 36  0d 25 e2 e3 cf d3 0d 0a  |%PDF-1.6.%......|
> 00000010  33 30 35 35 31 39 20 30  20 6f 62 6a 0d 3c 3c 2f  |305519 0 obj.<</|
> 00000020  46 69 6c 74 65 72 2f 46  6c 61 74 65 44 65 63 6f  |Filter/FlateDeco|
> 00000030  64 65 2f 46 69 72 73 74  20 31 31 37 39 2f 4c 65  |de/First 1179/Le|
> 00000040
> 
> 
> Kind Regards
> 
> James

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: ext4 recovery
  2020-01-21  2:20     ` Andreas Dilger
@ 2020-04-05 16:10       ` James Courtier-Dutton
  0 siblings, 0 replies; 6+ messages in thread
From: James Courtier-Dutton @ 2020-04-05 16:10 UTC (permalink / raw)
  To: Andreas Dilger; +Cc: Theodore Y. Ts'o, linux-ext4

HI,

Just an update. I managed to recover all the data I needed to. Many
thanks to everyone who responded.
Ext4 is the greatest file system!!!

Kind Regards

James



On Tue, 21 Jan 2020 at 02:20, Andreas Dilger <adilger@dilger.ca> wrote:
>
> You can use "filefrag -k" to get units of KB instead of blocks for the physical offsets of the files, and "lvdisplay -m /dev/XXX" to show the mapping of LVs to disks (not sure if you can get it to print different units, or just LE number and you gave to multiply by the LE size).  Hopefully the LV was layer out linearly and is just a matter of computing "> X and < Y" to determine the affected range.
>
> Note that it is likely that some files which were on the second disk will not show anything from filefrag since the inodes were zeroed out by dd_rescue. Some files on the second disk that _do_ show blocks that were allocated on that disk may just have zeroes for the data.
>
> Cheers, Andreas
>
> > On Jan 20, 2020, at 15:07, James Courtier-Dutton <james.dutton@gmail.com> wrote:
> >
> > On Tue, 14 Jan 2020 at 21:25, Theodore Y. Ts'o <tytso@mit.edu> wrote:
> >>
> >>> On Tue, Jan 14, 2020 at 04:03:53PM +0000, James Courtier-Dutton wrote:
> >>>
> >>> Say I started with 1 disk using LVM with an ext4 partition.
> >>> I then added another disk. Added it to the LVM group, expanded the
> >>> ext4 partition to then fill 2 disks.
> >>> I then added another disk. Added it to the LVM group, expanded the
> >>> ext4 partition to then fill 3 disks.
> >>
> >> Where you using RAID 0, or some more advanced RAID level?
> >>
> >>> One of the disk has now failed.
> >>
> >> How has it failed?  It is dead dead dead?  Or are there a large number
> >> of sector errors?
> >>
> >>> Are there any tools available for ext4 that could help recover this?
> >>> Note, I am a single user, not a company, so there is no money to give
> >>> to a data recovery company, so I wish to try myself.
> >>
> >> How valuable is your data?  The first thing I would recommend, if your
> >> data is worth it (and only you can make that decision) is to create a
> >> new RAID set (using larger disks if that helps reduce the price) so
> >> you can make an block-level image backup using the dd_rescue program.
> >>
> >> If you can, then run e2fsck on the backup copy, and then see what you
> >> can recover from that image set.  This will save time (how much is
> >> your time worth?) and perhaps increase the amount of data you can
> >> recover (how much is your data worth?).
> >>
> >>                                        - Ted
> >
> > Hi,
> >
> > Thank you all for the help.
> > I have made some progress.
> > Disk 1 is 100% OK.
> > Disk 2 did dd_rescue and recovered about 30% of the sectors.
> > Disk 3 is 100% OK.
> >
> > I have made images of all of that. LVM worked out what order they went
> > in, and ext4 even let me mount the LVM volume.
> > I have managed to recover a lot of the data, so thank you for the hints.
> > What I would like to do now is find out which files were on Disk 2.
> > I have found the fiemap  IOCTL that gives me the logical sectors for
> > the file itself.
> > How do I convert the value from the IOCTL into a physical sector, or
> > at least an offset into the LVM volume?
> > I can then work at that sector 0 to A is on Disk 1,  A to B is on Disk
> > 2, and B to C is on Disk 3.
> >
> > For example, this is how it is done on btrfs. What is the equivalent on ext4?
> >
> > $ sudo filefrag -v UEFI_Spec_2_8_final.pdf
> > Filesystem type is: 9123683e
> > File size of UEFI_Spec_2_8_final.pdf is 18586279 (4538 blocks of 4096 bytes)
> > ext:     logical_offset:        physical_offset: length:   expected: flags:
> >   0:        0..    4537:    9798022..   9802559:   4538:             last,eof
> > UEFI_Spec_2_8_final.pdf: 1 extent found
> > $ sudo btrfs-map-logical -l $[9798022*4096] /dev/nvme0n1p7
> > mirror 1 logical 40132698112 physical 31475654656 device /dev/nvme0n1p7
> > $ sudo dd if=/dev/nvme0n1p7 bs=1
> > skip=31475654656 count=64 2>/dev/null | hexdump -C
> > 00000000  25 50 44 46 2d 31 2e 36  0d 25 e2 e3 cf d3 0d 0a  |%PDF-1.6.%......|
> > 00000010  33 30 35 35 31 39 20 30  20 6f 62 6a 0d 3c 3c 2f  |305519 0 obj.<</|
> > 00000020  46 69 6c 74 65 72 2f 46  6c 61 74 65 44 65 63 6f  |Filter/FlateDeco|
> > 00000030  64 65 2f 46 69 72 73 74  20 31 31 37 39 2f 4c 65  |de/First 1179/Le|
> > 00000040
> >
> >
> > Kind Regards
> >
> > James

^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2020-04-05 16:11 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-01-14 16:03 ext4 recovery James Courtier-Dutton
2020-01-14 20:35 ` Andreas Dilger
2020-01-14 21:25 ` Theodore Y. Ts'o
2020-01-20 22:06   ` James Courtier-Dutton
2020-01-21  2:20     ` Andreas Dilger
2020-04-05 16:10       ` James Courtier-Dutton

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).