* ext4 recovery @ 2020-01-14 16:03 James Courtier-Dutton 2020-01-14 20:35 ` Andreas Dilger 2020-01-14 21:25 ` Theodore Y. Ts'o 0 siblings, 2 replies; 6+ messages in thread From: James Courtier-Dutton @ 2020-01-14 16:03 UTC (permalink / raw) To: linux-ext4 Hi, Say I started with 1 disk using LVM with an ext4 partition. I then added another disk. Added it to the LVM group, expanded the ext4 partition to then fill 2 disks. I then added another disk. Added it to the LVM group, expanded the ext4 partition to then fill 3 disks. One of the disk has now failed. Are there any tools available for ext4 that could help recover this? Note, I am a single user, not a company, so there is no money to give to a data recovery company, so I wish to try myself. Is there a mount option that will mount an ext4 partition even if there are lots of sector errors, or a missing disk. So, at least some of the files will be recoverable. Or a tool that will scan the disk for super blocks, then find inodes and then find the sector location of the files, even if they don't have filenames. Kind Regards James ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: ext4 recovery 2020-01-14 16:03 ext4 recovery James Courtier-Dutton @ 2020-01-14 20:35 ` Andreas Dilger 2020-01-14 21:25 ` Theodore Y. Ts'o 1 sibling, 0 replies; 6+ messages in thread From: Andreas Dilger @ 2020-01-14 20:35 UTC (permalink / raw) To: James Courtier-Dutton; +Cc: linux-ext4 [-- Attachment #1: Type: text/plain, Size: 1955 bytes --] On Jan 14, 2020, at 9:03 AM, James Courtier-Dutton <james.dutton@gmail.com> wrote: > > Hi, > > Say I started with 1 disk using LVM with an ext4 partition. > I then added another disk. Added it to the LVM group, expanded the > ext4 partition to then fill 2 disks. > I then added another disk. Added it to the LVM group, expanded the > ext4 partition to then fill 3 disks. > > One of the disk has now failed. > Are there any tools available for ext4 that could help recover this? > Note, I am a single user, not a company, so there is no money to give > to a data recovery company, so I wish to try myself. > > Is there a mount option that will mount an ext4 partition even if > there are lots of sector errors, or a missing disk. So, at least some > of the files will be recoverable. > Or a tool that will scan the disk for super blocks, then find inodes > and then find the sector location of the files, even if they don't > have filenames. The first thing I would do is get new disk(s) to make a full backup of the existing disks, so that whatever you do for recovery doesn't cause further data loss. There are two approaches to recovering the data: - add in a replacement disk and re-add the missing LVs that fill in the lost space of the failed disk, then run e2fsck to rebuild the metadata for the missing disk. I'm not sure how easy/hard this is with LVM. - use a block-level scanner to recover files on disk, which is useful for e.g. .jpg images, or other files with good magic signatures and internal data integrity checks. You didn't mention whether it was the first, second, or third disk that failed. If it is the third disk, that may not even need to be replaced, since you could set the filesystem size to be the end of the second disk and then e2fsck will repair the bad block/inode references. If it is the first or second disk then it is more complex. Cheers, Andreas [-- Attachment #2: Message signed with OpenPGP --] [-- Type: application/pgp-signature, Size: 873 bytes --] ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: ext4 recovery 2020-01-14 16:03 ext4 recovery James Courtier-Dutton 2020-01-14 20:35 ` Andreas Dilger @ 2020-01-14 21:25 ` Theodore Y. Ts'o 2020-01-20 22:06 ` James Courtier-Dutton 1 sibling, 1 reply; 6+ messages in thread From: Theodore Y. Ts'o @ 2020-01-14 21:25 UTC (permalink / raw) To: James Courtier-Dutton; +Cc: linux-ext4 On Tue, Jan 14, 2020 at 04:03:53PM +0000, James Courtier-Dutton wrote: > > Say I started with 1 disk using LVM with an ext4 partition. > I then added another disk. Added it to the LVM group, expanded the > ext4 partition to then fill 2 disks. > I then added another disk. Added it to the LVM group, expanded the > ext4 partition to then fill 3 disks. Where you using RAID 0, or some more advanced RAID level? > One of the disk has now failed. How has it failed? It is dead dead dead? Or are there a large number of sector errors? > Are there any tools available for ext4 that could help recover this? > Note, I am a single user, not a company, so there is no money to give > to a data recovery company, so I wish to try myself. How valuable is your data? The first thing I would recommend, if your data is worth it (and only you can make that decision) is to create a new RAID set (using larger disks if that helps reduce the price) so you can make an block-level image backup using the dd_rescue program. If you can, then run e2fsck on the backup copy, and then see what you can recover from that image set. This will save time (how much is your time worth?) and perhaps increase the amount of data you can recover (how much is your data worth?). - Ted ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: ext4 recovery 2020-01-14 21:25 ` Theodore Y. Ts'o @ 2020-01-20 22:06 ` James Courtier-Dutton 2020-01-21 2:20 ` Andreas Dilger 0 siblings, 1 reply; 6+ messages in thread From: James Courtier-Dutton @ 2020-01-20 22:06 UTC (permalink / raw) To: Theodore Y. Ts'o, Andreas Dilger; +Cc: linux-ext4 On Tue, 14 Jan 2020 at 21:25, Theodore Y. Ts'o <tytso@mit.edu> wrote: > > On Tue, Jan 14, 2020 at 04:03:53PM +0000, James Courtier-Dutton wrote: > > > > Say I started with 1 disk using LVM with an ext4 partition. > > I then added another disk. Added it to the LVM group, expanded the > > ext4 partition to then fill 2 disks. > > I then added another disk. Added it to the LVM group, expanded the > > ext4 partition to then fill 3 disks. > > Where you using RAID 0, or some more advanced RAID level? > > > One of the disk has now failed. > > How has it failed? It is dead dead dead? Or are there a large number > of sector errors? > > > Are there any tools available for ext4 that could help recover this? > > Note, I am a single user, not a company, so there is no money to give > > to a data recovery company, so I wish to try myself. > > How valuable is your data? The first thing I would recommend, if your > data is worth it (and only you can make that decision) is to create a > new RAID set (using larger disks if that helps reduce the price) so > you can make an block-level image backup using the dd_rescue program. > > If you can, then run e2fsck on the backup copy, and then see what you > can recover from that image set. This will save time (how much is > your time worth?) and perhaps increase the amount of data you can > recover (how much is your data worth?). > > - Ted Hi, Thank you all for the help. I have made some progress. Disk 1 is 100% OK. Disk 2 did dd_rescue and recovered about 30% of the sectors. Disk 3 is 100% OK. I have made images of all of that. LVM worked out what order they went in, and ext4 even let me mount the LVM volume. I have managed to recover a lot of the data, so thank you for the hints. What I would like to do now is find out which files were on Disk 2. I have found the fiemap IOCTL that gives me the logical sectors for the file itself. How do I convert the value from the IOCTL into a physical sector, or at least an offset into the LVM volume? I can then work at that sector 0 to A is on Disk 1, A to B is on Disk 2, and B to C is on Disk 3. For example, this is how it is done on btrfs. What is the equivalent on ext4? $ sudo filefrag -v UEFI_Spec_2_8_final.pdf Filesystem type is: 9123683e File size of UEFI_Spec_2_8_final.pdf is 18586279 (4538 blocks of 4096 bytes) ext: logical_offset: physical_offset: length: expected: flags: 0: 0.. 4537: 9798022.. 9802559: 4538: last,eof UEFI_Spec_2_8_final.pdf: 1 extent found $ sudo btrfs-map-logical -l $[9798022*4096] /dev/nvme0n1p7 mirror 1 logical 40132698112 physical 31475654656 device /dev/nvme0n1p7 $ sudo dd if=/dev/nvme0n1p7 bs=1 skip=31475654656 count=64 2>/dev/null | hexdump -C 00000000 25 50 44 46 2d 31 2e 36 0d 25 e2 e3 cf d3 0d 0a |%PDF-1.6.%......| 00000010 33 30 35 35 31 39 20 30 20 6f 62 6a 0d 3c 3c 2f |305519 0 obj.<</| 00000020 46 69 6c 74 65 72 2f 46 6c 61 74 65 44 65 63 6f |Filter/FlateDeco| 00000030 64 65 2f 46 69 72 73 74 20 31 31 37 39 2f 4c 65 |de/First 1179/Le| 00000040 Kind Regards James ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: ext4 recovery 2020-01-20 22:06 ` James Courtier-Dutton @ 2020-01-21 2:20 ` Andreas Dilger 2020-04-05 16:10 ` James Courtier-Dutton 0 siblings, 1 reply; 6+ messages in thread From: Andreas Dilger @ 2020-01-21 2:20 UTC (permalink / raw) To: James Courtier-Dutton; +Cc: Theodore Y. Ts'o, linux-ext4 You can use "filefrag -k" to get units of KB instead of blocks for the physical offsets of the files, and "lvdisplay -m /dev/XXX" to show the mapping of LVs to disks (not sure if you can get it to print different units, or just LE number and you gave to multiply by the LE size). Hopefully the LV was layer out linearly and is just a matter of computing "> X and < Y" to determine the affected range. Note that it is likely that some files which were on the second disk will not show anything from filefrag since the inodes were zeroed out by dd_rescue. Some files on the second disk that _do_ show blocks that were allocated on that disk may just have zeroes for the data. Cheers, Andreas > On Jan 20, 2020, at 15:07, James Courtier-Dutton <james.dutton@gmail.com> wrote: > > On Tue, 14 Jan 2020 at 21:25, Theodore Y. Ts'o <tytso@mit.edu> wrote: >> >>> On Tue, Jan 14, 2020 at 04:03:53PM +0000, James Courtier-Dutton wrote: >>> >>> Say I started with 1 disk using LVM with an ext4 partition. >>> I then added another disk. Added it to the LVM group, expanded the >>> ext4 partition to then fill 2 disks. >>> I then added another disk. Added it to the LVM group, expanded the >>> ext4 partition to then fill 3 disks. >> >> Where you using RAID 0, or some more advanced RAID level? >> >>> One of the disk has now failed. >> >> How has it failed? It is dead dead dead? Or are there a large number >> of sector errors? >> >>> Are there any tools available for ext4 that could help recover this? >>> Note, I am a single user, not a company, so there is no money to give >>> to a data recovery company, so I wish to try myself. >> >> How valuable is your data? The first thing I would recommend, if your >> data is worth it (and only you can make that decision) is to create a >> new RAID set (using larger disks if that helps reduce the price) so >> you can make an block-level image backup using the dd_rescue program. >> >> If you can, then run e2fsck on the backup copy, and then see what you >> can recover from that image set. This will save time (how much is >> your time worth?) and perhaps increase the amount of data you can >> recover (how much is your data worth?). >> >> - Ted > > Hi, > > Thank you all for the help. > I have made some progress. > Disk 1 is 100% OK. > Disk 2 did dd_rescue and recovered about 30% of the sectors. > Disk 3 is 100% OK. > > I have made images of all of that. LVM worked out what order they went > in, and ext4 even let me mount the LVM volume. > I have managed to recover a lot of the data, so thank you for the hints. > What I would like to do now is find out which files were on Disk 2. > I have found the fiemap IOCTL that gives me the logical sectors for > the file itself. > How do I convert the value from the IOCTL into a physical sector, or > at least an offset into the LVM volume? > I can then work at that sector 0 to A is on Disk 1, A to B is on Disk > 2, and B to C is on Disk 3. > > For example, this is how it is done on btrfs. What is the equivalent on ext4? > > $ sudo filefrag -v UEFI_Spec_2_8_final.pdf > Filesystem type is: 9123683e > File size of UEFI_Spec_2_8_final.pdf is 18586279 (4538 blocks of 4096 bytes) > ext: logical_offset: physical_offset: length: expected: flags: > 0: 0.. 4537: 9798022.. 9802559: 4538: last,eof > UEFI_Spec_2_8_final.pdf: 1 extent found > $ sudo btrfs-map-logical -l $[9798022*4096] /dev/nvme0n1p7 > mirror 1 logical 40132698112 physical 31475654656 device /dev/nvme0n1p7 > $ sudo dd if=/dev/nvme0n1p7 bs=1 > skip=31475654656 count=64 2>/dev/null | hexdump -C > 00000000 25 50 44 46 2d 31 2e 36 0d 25 e2 e3 cf d3 0d 0a |%PDF-1.6.%......| > 00000010 33 30 35 35 31 39 20 30 20 6f 62 6a 0d 3c 3c 2f |305519 0 obj.<</| > 00000020 46 69 6c 74 65 72 2f 46 6c 61 74 65 44 65 63 6f |Filter/FlateDeco| > 00000030 64 65 2f 46 69 72 73 74 20 31 31 37 39 2f 4c 65 |de/First 1179/Le| > 00000040 > > > Kind Regards > > James ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: ext4 recovery 2020-01-21 2:20 ` Andreas Dilger @ 2020-04-05 16:10 ` James Courtier-Dutton 0 siblings, 0 replies; 6+ messages in thread From: James Courtier-Dutton @ 2020-04-05 16:10 UTC (permalink / raw) To: Andreas Dilger; +Cc: Theodore Y. Ts'o, linux-ext4 HI, Just an update. I managed to recover all the data I needed to. Many thanks to everyone who responded. Ext4 is the greatest file system!!! Kind Regards James On Tue, 21 Jan 2020 at 02:20, Andreas Dilger <adilger@dilger.ca> wrote: > > You can use "filefrag -k" to get units of KB instead of blocks for the physical offsets of the files, and "lvdisplay -m /dev/XXX" to show the mapping of LVs to disks (not sure if you can get it to print different units, or just LE number and you gave to multiply by the LE size). Hopefully the LV was layer out linearly and is just a matter of computing "> X and < Y" to determine the affected range. > > Note that it is likely that some files which were on the second disk will not show anything from filefrag since the inodes were zeroed out by dd_rescue. Some files on the second disk that _do_ show blocks that were allocated on that disk may just have zeroes for the data. > > Cheers, Andreas > > > On Jan 20, 2020, at 15:07, James Courtier-Dutton <james.dutton@gmail.com> wrote: > > > > On Tue, 14 Jan 2020 at 21:25, Theodore Y. Ts'o <tytso@mit.edu> wrote: > >> > >>> On Tue, Jan 14, 2020 at 04:03:53PM +0000, James Courtier-Dutton wrote: > >>> > >>> Say I started with 1 disk using LVM with an ext4 partition. > >>> I then added another disk. Added it to the LVM group, expanded the > >>> ext4 partition to then fill 2 disks. > >>> I then added another disk. Added it to the LVM group, expanded the > >>> ext4 partition to then fill 3 disks. > >> > >> Where you using RAID 0, or some more advanced RAID level? > >> > >>> One of the disk has now failed. > >> > >> How has it failed? It is dead dead dead? Or are there a large number > >> of sector errors? > >> > >>> Are there any tools available for ext4 that could help recover this? > >>> Note, I am a single user, not a company, so there is no money to give > >>> to a data recovery company, so I wish to try myself. > >> > >> How valuable is your data? The first thing I would recommend, if your > >> data is worth it (and only you can make that decision) is to create a > >> new RAID set (using larger disks if that helps reduce the price) so > >> you can make an block-level image backup using the dd_rescue program. > >> > >> If you can, then run e2fsck on the backup copy, and then see what you > >> can recover from that image set. This will save time (how much is > >> your time worth?) and perhaps increase the amount of data you can > >> recover (how much is your data worth?). > >> > >> - Ted > > > > Hi, > > > > Thank you all for the help. > > I have made some progress. > > Disk 1 is 100% OK. > > Disk 2 did dd_rescue and recovered about 30% of the sectors. > > Disk 3 is 100% OK. > > > > I have made images of all of that. LVM worked out what order they went > > in, and ext4 even let me mount the LVM volume. > > I have managed to recover a lot of the data, so thank you for the hints. > > What I would like to do now is find out which files were on Disk 2. > > I have found the fiemap IOCTL that gives me the logical sectors for > > the file itself. > > How do I convert the value from the IOCTL into a physical sector, or > > at least an offset into the LVM volume? > > I can then work at that sector 0 to A is on Disk 1, A to B is on Disk > > 2, and B to C is on Disk 3. > > > > For example, this is how it is done on btrfs. What is the equivalent on ext4? > > > > $ sudo filefrag -v UEFI_Spec_2_8_final.pdf > > Filesystem type is: 9123683e > > File size of UEFI_Spec_2_8_final.pdf is 18586279 (4538 blocks of 4096 bytes) > > ext: logical_offset: physical_offset: length: expected: flags: > > 0: 0.. 4537: 9798022.. 9802559: 4538: last,eof > > UEFI_Spec_2_8_final.pdf: 1 extent found > > $ sudo btrfs-map-logical -l $[9798022*4096] /dev/nvme0n1p7 > > mirror 1 logical 40132698112 physical 31475654656 device /dev/nvme0n1p7 > > $ sudo dd if=/dev/nvme0n1p7 bs=1 > > skip=31475654656 count=64 2>/dev/null | hexdump -C > > 00000000 25 50 44 46 2d 31 2e 36 0d 25 e2 e3 cf d3 0d 0a |%PDF-1.6.%......| > > 00000010 33 30 35 35 31 39 20 30 20 6f 62 6a 0d 3c 3c 2f |305519 0 obj.<</| > > 00000020 46 69 6c 74 65 72 2f 46 6c 61 74 65 44 65 63 6f |Filter/FlateDeco| > > 00000030 64 65 2f 46 69 72 73 74 20 31 31 37 39 2f 4c 65 |de/First 1179/Le| > > 00000040 > > > > > > Kind Regards > > > > James ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2020-04-05 16:11 UTC | newest] Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2020-01-14 16:03 ext4 recovery James Courtier-Dutton 2020-01-14 20:35 ` Andreas Dilger 2020-01-14 21:25 ` Theodore Y. Ts'o 2020-01-20 22:06 ` James Courtier-Dutton 2020-01-21 2:20 ` Andreas Dilger 2020-04-05 16:10 ` James Courtier-Dutton
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).