recovering data from a corrupt ext3?

recovering data from a corrupt ext3?

[Dan Stromberg]

I have an ext3, under lvm3, on a Fedora Core 3 system, that:

1) Is corrupt

2) Contains data I want back, including some financial records, some
opensource applications I was developing, my list of URL's for conversion
from HTML to palmdoc, and so on.

I've tried the usual: booting up, fsck'ing, fsck'ing with alternative
superblocks, running e2salvage, running e2extract...  But none are
providing satisfaction.  The fsck's give "invalid superblock" or "invalid
argument", running e2salvage apparently OOM'd with the message
"Terminated", and e2extract just listed a huge number of 0 length files. 
Also, when I boot from a Fedora Core 3 cdrom and let it try to mount my
filesystems, it 

When I boot up into
the FC3 rescue CD and let it try to find my fedora install, it gets really
confused.  More specifically, it says:

Searching for Fedora Core installations...

        0%              install exited abnormally -- received signal 15
                                kernel panic - not syncing: Out of
memory and no killable processes


If I remove "quiet" and add "single" to my boot options, I get:

EXT3-fs: INFO: recovery required on readonly filesystem.
EXT3-fs: write access will be enabled during recovery.
kjournald starting.  Commit interval 5 seconds
EXT3-fs: dm-0: orphan cleanup on readonly fs

...and there it hangs.




My questions are:

1) Is there a better tool for ext3 data recovery?

2) If there isn't, is there a document that provides an overview of the
ext3 on-disk filesystem structure, so that I might write a tool for doing
the recovery?  (I wrote one once for the atari 800 floppy disk filesystem).

3) Given that this ext3 is under LVM2, and that I haven't rearranged disk
blocks within LVM2 (say, by mucking with pv's, vg's or lv's), is it then
reasonably safe to conclude that the blocks of my ext3 are contiguous, as
they would be if they weren't under LVM2?

On the subject of question #2 immediately above, I'm primarily interested
in:

1) What kind of alignment assumptions can I make about the various data
structures?  EG, if I can assume that a bunch of directory entries always
start on a 512 byte boundary, that'll speed up directory entry hunting
considerably.

2) What are the relationships between the on-disk datastructures?  A tree
diagram that indicates 1-n, n-1, 1-1 and n-n relationships would probably
be really useful.  I believe I mostly grok the superblock, inode and
directory entry datastructures, but the block_group and fragment
structures are a relative unknown to me.  I am familiar with the idea of
"cylinder groups" though - is that basically what the "block_group" stuff
is about?

3) Are there any on-disk data structures that are strictly (or
mostly) contiguous? Or are the various pieces distributed throughout the
filesystem to minimize track to track seeking?  Or are none of the
datastructures contiguous, aside from some of the disk blocks belonging to
individual files?

Thanks!

Re: recovering data from a corrupt ext3?

[Alan Cox]

On Iau, 2004-12-16 at 18:20, Dan Stromberg wrote:
> 1) Is there a better tool for ext3 data recovery?

You can use e2fsdebug to poke around a bit and to follow inodes. You
might want to start lower down the stack however and check the LVM
claims to be the right size for the file system and any partition data
looks right.

> 2) If there isn't, is there a document that provides an overview of the
> ext3 on-disk filesystem structure, so that I might write a tool for doing
> the recovery?  (I wrote one once for the atari 800 floppy disk filesystem).

This is the stuff e2fsdebug essentially knows about - ext3 is ext2 with
journal so both are very similar.

> 1) What kind of alignment assumptions can I make about the various data
> structures?  EG, if I can assume that a bunch of directory entries always
> start on a 512 byte boundary, that'll speed up directory entry hunting
> considerably.

Superblocks give you all the basic layout information. The disk is split
into cylinder groups which in turn are split into inode and data zones.
Lookups are essentially

	directory entry -> inode number -> inode block

	inode block -> data (first few blocks direct, next indirected once,
then twice ..)


The superblocks will at least be disk block aligned so a good starting
point might be to find the superblocks on the disk itself without LVM in
the way. That should tell you the start/end/layout of the fs and you may
even be able to then pull those blocks into a file and loopback mount
the copy

Re: recovering data from a corrupt ext3?

[Dan Stromberg]

On Thu, 16 Dec 2004 19:50:59 +0000, Alan Cox wrote:

This is what worked for me:

On FC3's rescue disk:

1) Do startup network interfaces
2) Don't try to automatically mount the filesystems - not even readonly
3) lvm vgchange --ignorelockingfailure -P -a y
4) fdisk -l, and guess which partition is which based on size: the small one was /boot, and the large one was /
5) mkdir /mnt/boot
6) mount /dev/hda1 /mnt/boot
7) Look up the device node for the root filesystem in /mnt/boot/grub/grub.conf
8) A first tentative step, to see if things are working: fsck -n /dev/VolGroup00/LogVol00
9) Dive in: fsck -f -y /dev/VolGroup00/LogVol00
10) Wait a while...  Be patient.  Don't interrupt it
11) Reboot