XFS, empty files after a crash

XFS, empty files after a crash

[kfx]

Hi,

After a crash, a lot of files on a xfs file system report an empty size 
with "ls -a" but not with "du".
xfs_check and xfs_repair don't report any problem to repair (although an 
internal log is used) :(
So if I understand correctly, those empty files actually use space on 
the disk but are incorrectly reported as empty by the system.
Is there a way to copy back the used blocks to recover the files ?

Example for a previously not empty file:
# ls -l myfile
-rw-r--r-- 1 me me 0 2012-01-24 15:29 myfile

# du  myfile
27460    myfile

# du --apparent-size myfile
0    myfile

# xfs_bmap myfile
myfile: no extents

System: Ubuntu 11.04  2.6.38-8-server x86_64

xfs_info /dev/sdc1
meta-data=/dev/sdc1              isize=256    agcount=10, 
agsize=268435328 blks
          =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=2441215488, imaxpct=5
          =                       sunit=128    swidth=640 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=4096   blocks=521728, version=2
          =                       sectsz=512   sunit=8 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

Thank you

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS, empty files after a crash

[Peter Grandi]

> Hi, After a crash, a lot of files on a xfs file system report
> an empty size with "ls -a" but not with "du". [ ... ]

It is a FAQ:

  http://xfs.org/index.php/XFS_FAQ#Q:_Why_do_I_see_binary_NULLS_in_some_files_after_recovery_when_I_unplugged_the_power.3F

Usually it is a good idea to do a web search before reporting a
problem.

> Is there a way to copy back the used blocks to recover the
> files ?

No, because your applications told XFS that the content of those
blocks could be thrown away in case of crash.

It is your responsibility to ensure that your applications use
'fsync' or at least 'fsyncdata' when data should be stored
permanently before a crash occurs.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS, empty files after a crash

[Stan Hoeppner]

On 2/21/2012 6:01 AM, kfx wrote:

> After a crash, a lot of files on a xfs file system report an empty size
> with "ls -a" but not with "du".
> xfs_check and xfs_repair don't report any problem to repair (although an
> internal log is used) :(
> So if I understand correctly, those empty files actually use space on
> the disk but are incorrectly reported as empty by the system.
> Is there a way to copy back the used blocks to recover the files ?

What was the nature of the crash?  Kernel panic?  Power failure?  Other?

Do you have have backups of previously existing files that are now shown
as zero size?  If so delete and restore, maybe just overwrite.

For new files that were in the process of being created you're simply
out of luck.  The metadata may have hit the journal but the file data in
buffer cache was lost.

In memory write buffering/ordering techniques are what give modern
filesystems their write throughput performance.  Unfortunately this
comes at a cost, when systems crash.  As Peter mentioned, one way to
mitigate this is coding apps to sync their writes.  This obviously has a
performance impact.  Another is possibly to mount filesystems with the
sync option, which will obviously severely impact write performance.
Both will prevent or at least lessen the carnage you've seen as a result
of this crash.  You may not be able to live the performance of the
latter, though.

-- 
Stan

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS, empty files after a crash

[Eric Sandeen]

On 2/21/12 10:25 AM, Peter Grandi wrote:
>> Hi, After a crash, a lot of files on a xfs file system report
>> an empty size with "ls -a" but not with "du". [ ... ]
> 
> It is a FAQ:
> 
>   http://xfs.org/index.php/XFS_FAQ#Q:_Why_do_I_see_binary_NULLS_in_some_files_after_recovery_when_I_unplugged_the_power.3F
> 
> Usually it is a good idea to do a web search before reporting a
> problem.

He seems to be reporting a different problem.

In the FAQ case above, one would get a file but "it has is a size but no extents " - i.e. ls -l would show 1M, but du would show 0, and no extents are allocated.

In this case, he has no size, blocks are reported used, but no extents are allocated.  That actually seems like a new/odd case.

>> Is there a way to copy back the used blocks to recover the
>> files ?
> 
> No, because your applications told XFS that the content of those
> blocks could be thrown away in case of crash.

> It is your responsibility to ensure that your applications use
> 'fsync' or at least 'fsyncdata' when data should be stored
> permanently before a crash occurs.

I'd agree that there is likely nothing to recover; there are no extents allocated.  I'm not sure why du is reporting space used though.

-Eric

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS, empty files after a crash

[Nathaniel W. Turner]

On 02/21/2012 10:18 PM, Eric Sandeen wrote:
> I'd agree that there is likely nothing to recover; there are no extents 
> allocated. I'm not sure why du is reporting space used though.

I have noticed that xfs_bmap (which uses the ioctl bmap interface) will not 
report extents after EOF.  There probably *are* extents allocated to this file, 
but they are not being reported by xfs_bmap.

kfx, try getting the inode number of the file (via stat or ls -i) and then 
doing something like this:

xfs_db -r $DEV -c "inode $INO" -c "bmap"

This will dump the entire extent map, and I suspect you'll see at least one extent.

nate


_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS, empty files after a crash

[kadafax]

> kfx, try getting the inode number of the file (via stat or ls -i) and 
> then doing something like this:
>
> xfs_db -r $DEV -c "inode $INO" -c "bmap" 


# xfs_db -r /dev/sdc1 -c "inode 114748" -c "bmap"
data offset 0 startblock 1881705728 (7/2657536) count 6460 flag 0

# xfs_db -r /dev/sdc1 -c "inode 114754" -c "bmap"
data offset 0 startblock 1077794560 (4/4052736) count 6582 flag 0


_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS, empty files after a crash

[Nathaniel W. Turner]

On 02/23/2012 03:07 PM, kadafax@gmail.com wrote:
>
>> kfx, try getting the inode number of the file (via stat or ls -i) and then 
>> doing something like this:
>>
>> xfs_db -r $DEV -c "inode $INO" -c "bmap" 
>
>
> # xfs_db -r /dev/sdc1 -c "inode 114748" -c "bmap"
> data offset 0 startblock 1881705728 (7/2657536) count 6460 flag 0
>
> # xfs_db -r /dev/sdc1 -c "inode 114754" -c "bmap"
> data offset 0 startblock 1077794560 (4/4052736) count 6582 flag 0

If you want to see what's behind those data extents (which are probably 
partially written), you could do something along these lines:

# Determine the AG size
agblocks=$(xfs_db -r /dev/sdc1 -c sb -c p | grep ^agblocks | sed 's/.* = //')

# Copy the extent in the first file, which consists of 6460 blocks (~26MB)
# in AG 7 starting at AG-relative block 2657536:
dd if=/dev/sdc1 bs=4096 skip=$(($agblocks * 7 + 2657536)) count=6460 of=./blob
# examine ./blob
...

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS, empty files after a crash

[kadafax]

Le 23/02/12 23:15, Nathaniel W. Turner a écrit :
> On 02/23/2012 03:07 PM, kadafax@gmail.com wrote:
>>
>>> kfx, try getting the inode number of the file (via stat or ls -i) 
>>> and then doing something like this:
>>>
>>> xfs_db -r $DEV -c "inode $INO" -c "bmap" 
>>
>>
>> # xfs_db -r /dev/sdc1 -c "inode 114748" -c "bmap"
>> data offset 0 startblock 1881705728 (7/2657536) count 6460 flag 0
>>
>> # xfs_db -r /dev/sdc1 -c "inode 114754" -c "bmap"
>> data offset 0 startblock 1077794560 (4/4052736) count 6582 flag 0
>
> If you want to see what's behind those data extents (which are 
> probably partially written), you could do something along these lines:
>
> # Determine the AG size
> agblocks=$(xfs_db -r /dev/sdc1 -c sb -c p | grep ^agblocks | sed 's/.* 
> = //')
>
> # Copy the extent in the first file, which consists of 6460 blocks 
> (~26MB)
> # in AG 7 starting at AG-relative block 2657536:
> dd if=/dev/sdc1 bs=4096 skip=$(($agblocks * 7 + 2657536)) count=6460 
> of=./blob
> # examine ./blob

Awesome, it's working for each file with non-null data extents.

I guess that for inode for which the command "xfs_db -r $DEV -c "inode 
$INO" -c "bmap"" returns a null result there is no hope ?

Anyway, thanks a lot Nathaniel !

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS, empty files after a crash

[Dave Chinner]

On Tue, Feb 21, 2012 at 01:01:43PM +0100, kfx wrote:
> Hi,
> 
> After a crash, a lot of files on a xfs file system report an empty
> size with "ls -a" but not with "du".
> xfs_check and xfs_repair don't report any problem to repair
> (although an internal log is used) :(
> So if I understand correctly, those empty files actually use space
> on the disk but are incorrectly reported as empty by the system.
> Is there a way to copy back the used blocks to recover the files ?
> 
> Example for a previously not empty file:
> # ls -l myfile
> -rw-r--r-- 1 me me 0 2012-01-24 15:29 myfile
> 
> # du  myfile
> 27460    myfile
> 
> # du --apparent-size myfile
> 0    myfile
> 
> # xfs_bmap myfile
> myfile: no extents

That doesn't seem right:

$ ls -l foobar
-rw------- 1 root root 0 Feb 27 11:54 foobar
$ du foobar 
1024    foobar
$ du --apparent-size foobar 
0       foobar
$ xfs_bmap foobar
foobar:
	0: [0..2047]: 255169872..255171919

So xfs_bmap can and does report extents beyond EOF.

du uses the fstat(2) syscall to get the block count from the inode,
so it's seeing an inode with a block count but no extents. Can you
dump the inode via xfs_db like so:

$ ls -i foobar
604424233 foobar
$ sudo xfs_db -r -c "inode 604424233" -c p /dev/md0
core.magic = 0x494e
core.mode = 0100600
core.version = 2
.....

And post the output?

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS, empty files after a crash

[kadafax]

Le 27/02/12 02:04, Dave Chinner a écrit :
> On Tue, Feb 21, 2012 at 01:01:43PM +0100, kfx wrote:
>> # du  myfile
>> 27460    myfile
>>
>> # du --apparent-size myfile
>> 0    myfile
>>
>> # xfs_bmap myfile
>> myfile: no extents
> That doesn't seem right:
>
> $ ls -l foobar
> -rw------- 1 root root 0 Feb 27 11:54 foobar
> $ du foobar
> 1024    foobar
> $ du --apparent-size foobar
> 0       foobar
> $ xfs_bmap foobar
> foobar:
> 	0: [0..2047]: 255169872..255171919
>
> So xfs_bmap can and does report extents beyond EOF.
>
> du uses the fstat(2) syscall to get the block count from the inode,
> so it's seeing an inode with a block count but no extents. Can you
> dump the inode via xfs_db like so:
>
> $ ls -i foobar
> 604424233 foobar
> $ sudo xfs_db -r -c "inode 604424233" -c p /dev/md0
> core.magic = 0x494e
> core.mode = 0100600
> core.version = 2
> .....
>
> And post the output?
[root@server 1_out]# du foobar
25840    foobar
[root@server 1_out]# du --apparent-size foobar
0    foobar
[root@server 1_out]# xfs_bmap foobar
foobar: no extents
[root@server 1_out]# ls -i foobar
114748 foobar
[root@server 1_out]# xfs_db -r -c "inode 114748" -c p /dev/sdc1
core.magic = 0x494e
core.mode = 0100644
core.version = 2
core.format = 2 (extents)
core.nlinkv2 = 1
core.onlink = 0
core.projid = 0
core.uid = 12488
core.gid = 12488
core.flushiter = 0
core.atime.sec = Tue Jan 24 14:50:52 2012
core.atime.nsec = 609667096
core.mtime.sec = Tue Jan 24 14:50:53 2012
core.mtime.nsec = 579672930
core.ctime.sec = Tue Jan 24 14:50:53 2012
core.ctime.nsec = 579672930
core.size = 0
core.nblocks = 6460
core.extsize = 0
core.nextents = 1
core.naextents = 0
core.forkoff = 0
core.aformat = 2 (extents)
core.dmevmask = 0
core.dmstate = 0
core.newrtbm = 0
core.prealloc = 0
core.realtime = 0
core.immutable = 0
core.append = 0
core.sync = 0
core.noatime = 0
core.nodump = 0
core.rtinherit = 0
core.projinherit = 0
core.nosymlinks = 0
core.extsz = 0
core.extszinherit = 0
core.nodefrag = 0
core.filestream = 0
core.gen = 1924597489
next_unlinked = null
u.bmx[0] = [startoff,startblock,blockcount,extentflag] 
0:[0,1881705728,6460,0]

After using the Nathaniel's advices and dd'ing the file:
[root@server recover]# ls -alh foobar
-rw-r--r-- 1 root root 26M Feb 24 10:39 /recover/foobar




_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS, empty files after a crash

[Dave Chinner]

On Mon, Feb 27, 2012 at 09:43:42AM +0100, kadafax@gmail.com wrote:
> Le 27/02/12 02:04, Dave Chinner a écrit :
> >On Tue, Feb 21, 2012 at 01:01:43PM +0100, kfx wrote:
> >># du  myfile
> >>27460    myfile
> >>
> >># du --apparent-size myfile
> >>0    myfile
> >>
> >># xfs_bmap myfile
> >>myfile: no extents
> >That doesn't seem right:
> >
> >$ ls -l foobar
> >-rw------- 1 root root 0 Feb 27 11:54 foobar
> >$ du foobar
> >1024    foobar
> >$ du --apparent-size foobar
> >0       foobar
> >$ xfs_bmap foobar
> >foobar:
> >	0: [0..2047]: 255169872..255171919
> >
> >So xfs_bmap can and does report extents beyond EOF.
> >
> >du uses the fstat(2) syscall to get the block count from the inode,
> >so it's seeing an inode with a block count but no extents. Can you
> >dump the inode via xfs_db like so:
.....
> [root@server 1_out]# xfs_db -r -c "inode 114748" -c p /dev/sdc1
> core.magic = 0x494e
> core.mode = 0100644
> core.version = 2
> core.format = 2 (extents)
.....
> core.size = 0
> core.nblocks = 6460
> core.extsize = 0
> core.nextents = 1
.....
> core.prealloc = 0
....
> core.extsz = 0
> core.extszinherit = 0
....
> u.bmx[0] = [startoff,startblock,blockcount,extentflag]
> 0:[0,1881705728,6460,0]

Ok, so there's an extent there, but xfs_bmap is not showing it
because it isn't a preallocation but rather a normal extent. What
this indicates is that the inode size was not updated on disk after
the data write had occurred.

Given you are running 2.6.38 (the unbuntu kernel), this is likely
caused by a known bug introduced in 2.6.37 by VFS changes outside
XFS to prevent sync livelocks. That has been fixed in current 3.0.x
stable kernels and more recent 3.2+ kernels.  I'd suggest upgrading
your kernel.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs