All of lore.kernel.org
 help / color / mirror / Atom feed
* xfs filesystem corruption with kernel 2.6.37
@ 2012-10-25 13:45 Kamal Dasu
  2012-10-25 22:47 ` Dave Chinner
  0 siblings, 1 reply; 9+ messages in thread
From: Kamal Dasu @ 2012-10-25 13:45 UTC (permalink / raw)
  To: linux-xfs

I am trying to understand how to get out of a failing  mount
gracefully to be able to repair the disk.
(dev/sda2 is log parition and dev/sda/3 is used as  rt subvolume partition)
I have a corrupted disk that is used for recording video constantly.
Using xfs on 2.6.37 kernel with all the knwn rt subvolume patches, the
mount appears to hang, however it gets  into an infinite loop in
xfs_itruncate_finish():xfs_inode.c.  :

#4  0x801bd180 in xfs_bunmapi (tp=0xe83476cf, ip=0x80a386cf,
bno=4278648832, len=72057594054705152, flags=0, nexts=33554432,
firstblock=0xb0bbcdcf, flist=0xb8bbcdcf, done=0xa8bbcdcf)
    at fs/xfs/xfs_bmap.c:5266
#5  0x801dd3f8 in xfs_itruncate_finish (tp=0x1cbccdcf, ip=0x80a386cf,
new_size=<optimized out>, fork=0, sync=16777216) at
fs/xfs/xfs_inode.c:1585                        <=== never gets done
..
..

with  "CONFIG_XFS_DEBUG=y" I get the following assertion:

Assertion failed: prev.br_state == XFS_EXT_NORM, file:
fs/xfs/xfs_bmap.c, line: 5192
 ...
Call Trace:
[<80234c04>] assfail+0x28/0x2c
[<801cb57c>] xfs_bunmapi+0x1234/0x144c
[<801f6540>] xfs_itruncate_finish+0x3e8/0x7f4
[<8021deb4>] xfs_inactive+0x47c/0x4f0
[<800dcd64>] evict+0x28/0xd0
[<800dd310>] iput+0x19c/0x2d8
[<8020e27c>] xlog_recover_process_one_iunlink+0x150/0x198
[<8020e36c>] xlog_recover_process_iunlinks+0xa8/0x108
[<8020f3f8>] xlog_recover_finish+0x58/0x110
[<80213944>] xfs_mountfs+0x478/0x69c
[<80232ae8>] xfs_fs_fill_super+0x1dc/0x304
[<800c5fe8>] mount_bdev+0x21c/0x258
[<8022ff64>] xfs_fs_mount+0x18/0x24
[<800c4860>] vfs_kern_mount+0x64/0x1b8
[<800c4a08>] do_kern_mount+0x44/0x120
[<800e3f08>] do_mount+0x1b0/0x7cc
[<800e49d0>] sys_mount+0x84/0xf0
[<80011ebc>] stack_done+0x20/0x40

xfs_check, xfs_repair

# xfs_check /dev/sda2
ERROR: The filesystem has valuable metadata changes in a log which needs to
be replayed.  Mount the filesystem to replay the log, and unmount it before
re-running xfs_check.  If you are unable to mount the filesystem, then use
the xfs_repair -L option to destroy the log and attempt a repair.
Note that destroying the log may cause corruption -- please attempt a mount
of the filesystem before doing this.

sh-3.1# xfs_repair -n /dev/sda2 -r /dev/sda3
Phase 1 - find and verify superblock...
Phase 2 - using internal log
        - scan filesystem freespace and inode maps...
agi unlinked bucket 33 is 743329 in ag 2 (inode=34297761)
sb_icount 5184, counted 39040
sb_ifree 1315, counted 86
sb_fdblocks 3836812, counted 3644217
        - found root inode chunk
Phase 3 - for each AG...
        - scan (but don't clear) agi unlinked lists...
        - process known inodes and perform inode discovery...
        - agno = 0
inode 6776 - bad rt extent start block number 3518437212496384, offset 4216781
bad data fork in inode 6776
would have cleared inode 6776
        - agno = 1
771a3500: Badness in key lookup (length)
bp=(bno 16107312, len 16384 bytes) key=(bno 16107312, len 8192 bytes)
        - agno = 2
bad nblocks 5120 for inode 33701135, would reset to 4096
inode 34297761 - bad rt extent start block number 2392537303836672,
offset 6627188
bad data fork in inode 34297761
would have cleared inode 34297761
        - agno = 3
        - process newly discovered inodes...
Phase 4 - check for duplicate blocks...
        - setting up duplicate extent list...
        - check for inodes claiming duplicate blocks...
        - agno = 0
inode 6776 - bad rt extent start block number 3518437212496384, offset 4216781
bad data fork in inode 6776
would have cleared inode 6776
        - agno = 1
        - agno = 2
entry "0000000000754974720" at block 0 offset 1488 in directory inode
33700909 references free inode 6776
        would clear inode number in entry at offset 1488...
bad nblocks 5120 for inode 33701135, would reset to 4096
inode 34297761 - bad rt extent start block number 2392537303836672,
offset 6627188
bad data fork in inode 34297761
would have cleared inode 34297761
        - agno = 3
No modify flag set, skipping phase 5
Phase 6 - check inode connectivity...
        - traversing filesystem ...
entry "0000000000754974720" in directory inode 33700909 points to free
inode 6776, would junk entry
bad hash table for directory inode 33700909 (no data entry): would rebuild
        - traversal finished ...
        - moving disconnected inodes to lost+found ...
Phase 7 - verify link counts...
No modify flag set, skipping filesystem flush and exiting.
#

Currently would like to know how to gracefully get out of this
situation with error returned to mount so that we can repair the disk
also if there is something that can be done to avoid this situation in
the first place.

Thanks
Kamal

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: xfs filesystem corruption with kernel 2.6.37
  2012-10-25 13:45 xfs filesystem corruption with kernel 2.6.37 Kamal Dasu
@ 2012-10-25 22:47 ` Dave Chinner
  2012-11-01 19:30   ` Kamal Dasu
  0 siblings, 1 reply; 9+ messages in thread
From: Dave Chinner @ 2012-10-25 22:47 UTC (permalink / raw)
  To: Kamal Dasu; +Cc: linux-xfs

On Thu, Oct 25, 2012 at 09:45:10AM -0400, Kamal Dasu wrote:
> I am trying to understand how to get out of a failing  mount
> gracefully to be able to repair the disk.
> (dev/sda2 is log parition and dev/sda/3 is used as  rt subvolume partition)
> I have a corrupted disk that is used for recording video constantly.
> Using xfs on 2.6.37 kernel with all the knwn rt subvolume patches, the
> mount appears to hang, however it gets  into an infinite loop in
> xfs_itruncate_finish():xfs_inode.c.  :
> 
> #4  0x801bd180 in xfs_bunmapi (tp=0xe83476cf, ip=0x80a386cf,
> bno=4278648832, len=72057594054705152, flags=0, nexts=33554432,
                      ^^^^^^^^^^^^^^^^^

That length looks suspiciously like there are bit errors somewhere,
as bit 24 of both upper and lower 32 bits is set. i.e
len = 0x100000001000000

Do you really have a file that long (~68PB)?

Well, you can't - you're on a 32 bit box, which limits maximum file
size to 16TB. That tends to imply corruption as the cause.

> firstblock=0xb0bbcdcf, flist=0xb8bbcdcf, done=0xa8bbcdcf)
>     at fs/xfs/xfs_bmap.c:5266
> #5  0x801dd3f8 in xfs_itruncate_finish (tp=0x1cbccdcf, ip=0x80a386cf,
> new_size=<optimized out>, fork=0, sync=16777216) at
> fs/xfs/xfs_inode.c:1585                        <=== never gets done

because it been given an invalid length, and I'd say that 2.6.37
doesn't check it properly.

> ..
> ..
> 
> with  "CONFIG_XFS_DEBUG=y" I get the following assertion:
> 
> Assertion failed: prev.br_state == XFS_EXT_NORM, file:
> fs/xfs/xfs_bmap.c, line: 5192

Yup, that's pretty clear indication of a corrupted extent record.

>  ...
> Call Trace:
> [<80234c04>] assfail+0x28/0x2c
> [<801cb57c>] xfs_bunmapi+0x1234/0x144c
> [<801f6540>] xfs_itruncate_finish+0x3e8/0x7f4
> [<8021deb4>] xfs_inactive+0x47c/0x4f0
> [<800dcd64>] evict+0x28/0xd0
> [<800dd310>] iput+0x19c/0x2d8
> [<8020e27c>] xlog_recover_process_one_iunlink+0x150/0x198
> [<8020e36c>] xlog_recover_process_iunlinks+0xa8/0x108
> [<8020f3f8>] xlog_recover_finish+0x58/0x110
> [<80213944>] xfs_mountfs+0x478/0x69c
> [<80232ae8>] xfs_fs_fill_super+0x1dc/0x304
> [<800c5fe8>] mount_bdev+0x21c/0x258
> [<8022ff64>] xfs_fs_mount+0x18/0x24
> [<800c4860>] vfs_kern_mount+0x64/0x1b8
> [<800c4a08>] do_kern_mount+0x44/0x120
> [<800e3f08>] do_mount+0x1b0/0x7cc
> [<800e49d0>] sys_mount+0x84/0xf0
> [<80011ebc>] stack_done+0x20/0x40
> 
> xfs_check, xfs_repair
> 
> # xfs_check /dev/sda2
> ERROR: The filesystem has valuable metadata changes in a log which needs to
> be replayed.  Mount the filesystem to replay the log, and unmount it before
> re-running xfs_check.  If you are unable to mount the filesystem, then use
> the xfs_repair -L option to destroy the log and attempt a repair.
> Note that destroying the log may cause corruption -- please attempt a mount
> of the filesystem before doing this.
> 
> sh-3.1# xfs_repair -n /dev/sda2 -r /dev/sda3
> Phase 1 - find and verify superblock...
> Phase 2 - using internal log
>         - scan filesystem freespace and inode maps...
> agi unlinked bucket 33 is 743329 in ag 2 (inode=34297761)

an open but unlinked file when the system last crashed.

> sb_icount 5184, counted 39040
> sb_ifree 1315, counted 86
> sb_fdblocks 3836812, counted 3644217
>         - found root inode chunk
> Phase 3 - for each AG...
>         - scan (but don't clear) agi unlinked lists...
>         - process known inodes and perform inode discovery...
>         - agno = 0
> inode 6776 - bad rt extent start block number 3518437212496384, offset 4216781
						0xC800000372200, 
	
Yup, there's stray values in the upper 32 bits of the block number.

> bad data fork in inode 6776
> would have cleared inode 6776
>         - agno = 1
> 771a3500: Badness in key lookup (length)
> bp=(bno 16107312, len 16384 bytes) key=(bno 16107312, len 8192 bytes)
>         - agno = 2
> bad nblocks 5120 for inode 33701135, would reset to 4096
> inode 34297761 - bad rt extent start block number 2392537303836672,
						0x88000001B6800

That's the open, unlinked file at the time the system crashed. That
may be where your problems are coming from. The RT is mostly
untested, and we sure as anything don't do any crash resiliency or
recovery testing on it, so there's a good chance there are bugs in
it that might show up in situations like this....

> Currently would like to know how to gracefully get out of this
> situation with error returned to mount so that we can repair the disk

You need to detect extents with invalid lengths in them and trigger
a corruption-based filesystem shutdown.

> also if there is something that can be done to avoid this situation in
> the first place.

Track down where those stray upper bits in the block numbers are
coming from, and you'll have your answer.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: xfs filesystem corruption with kernel 2.6.37
  2012-10-25 22:47 ` Dave Chinner
@ 2012-11-01 19:30   ` Kamal Dasu
  2012-11-02  1:27     ` Dave Chinner
  0 siblings, 1 reply; 9+ messages in thread
From: Kamal Dasu @ 2012-11-01 19:30 UTC (permalink / raw)
  To: xfs


Dave,

Thanks for you reply.

I am trying to act on the hints you gave me but I still have a few
questions.

On Thu, Oct 25, 2012 at 6:47 PM, Dave Chinner <david@fromorbit.com> wrote:
> On Thu, Oct 25, 2012 at 09:45:10AM -0400, Kamal Dasu wrote:
>> with  "CONFIG_XFS_DEBUG=y" I get the following assertion:
>>
>> Assertion failed: prev.br_state == XFS_EXT_NORM, file:
>> fs/xfs/xfs_bmap.c, line: 5192
>
> Yup, that's pretty clear indication of a corrupted extent record.
>

What is the best way to prevent  transactions that record bad
extent length and block numbers.

>> would have cleared inode 6776
>>         - agno = 1
>> 771a3500: Badness in key lookup (length)
>> bp=(bno 16107312, len 16384 bytes) key=(bno 16107312, len 8192 bytes)
>>         - agno = 2
>> bad nblocks 5120 for inode 33701135, would reset to 4096
>> inode 34297761 - bad rt extent start block number 2392537303836672,
>                                                 0x88000001B6800
>
> That's the open, unlinked file at the time the system crashed. That
> may be where your problems are coming from. The RT is mostly
> untested, and we sure as anything don't do any crash resiliency or
> recovery testing on it, so there's a good chance there are bugs in
> it that might show up in situations like this....
>
> You need to detect extents with invalid lengths in them and trigger
> a corruption-based filesystem shutdown.
>

Looked at the log during one of the filesystem shutdown when the
I/O error occurs. is this an indication of already corrupted log due to
corrupted in-memory metadata structures?.
===
attempt to access beyond end of device
sda2: rw=0, want=33792081130943048, limit=31471329
I/O error in filesystem ("sda2") meta-data dev sda2 block
0x780db80007f240       ("xfs_trans_read_buf") error 5 buf count 4096
xfs_force_shutdown(sda2,0x1) called from line 395 of file
fs/xfs/xfs_trans_buf.c.  Return address = 0x801f4f88
Filesystem "sda2": I/O Error Detected.  Shutting down filesystem: sda2
Please umount the filesystem, and rectify the problem(s)
====

However the log is already corrupted. So is there a check on a write
to the log ?.

>> also if there is something that can be done to avoid this situation in
>> the first place.
>
> Track down where those stray upper bits in the block numbers are
> coming from, and you'll have your answer.
>

Have not been able to track this down yet. But could it be a possible memory
corruption, leading to the in-memory metadata to get corrupted.

On a similar occurrence of this issue on recovery after a reboot seems
to always go through the evict path

Filesystem "sda2": XFS internal error xfs_trans_cancel at line 1815
of file fs/xfs/xfs_trans.c.  Caller 0x801f8524

Call Trace:
[<80439d2c>] dump_stack+0x8/0x34
[<801f3bec>] xfs_trans_cancel+0x10c/0x128
[<801f8524>] xfs_inactive+0x2fc/0x450
[<800dcd54>] evict+0x28/0xd0
[<800dd300>] iput+0x19c/0x2d8
[<801e5bcc>] xlog_recover_process_one_iunlink+0xec/0x130
[<801e7b60>] xlog_recover_process_iunlinks.clone.25+0xa8/0x108
[<801eb360>] xlog_recover_finish+0x40/0x100
[<801eedd8>] xfs_mountfs+0x434/0x654
..
.
Filesystem "sda2": Corruption of in-memory data detected.  Shutting
down filesystem: sda2

-- 
View this message in context: http://old.nabble.com/xfs-filesystem-corruption-with-kernel-2.6.37-tp34601185p34630253.html
Sent from the Xfs - General mailing list archive at Nabble.com.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: xfs filesystem corruption with kernel 2.6.37
  2012-11-01 19:30   ` Kamal Dasu
@ 2012-11-02  1:27     ` Dave Chinner
  2012-11-02 16:34       ` Kamal Dasu
  0 siblings, 1 reply; 9+ messages in thread
From: Dave Chinner @ 2012-11-02  1:27 UTC (permalink / raw)
  To: Kamal Dasu; +Cc: xfs

On Thu, Nov 01, 2012 at 12:30:13PM -0700, Kamal Dasu wrote:
> 
> Dave,
> 
> Thanks for you reply.
> 
> I am trying to act on the hints you gave me but I still have a few
> questions.
> 
> On Thu, Oct 25, 2012 at 6:47 PM, Dave Chinner <david@fromorbit.com> wrote:
> > On Thu, Oct 25, 2012 at 09:45:10AM -0400, Kamal Dasu wrote:
> >> with  "CONFIG_XFS_DEBUG=y" I get the following assertion:
> >>
> >> Assertion failed: prev.br_state == XFS_EXT_NORM, file:
> >> fs/xfs/xfs_bmap.c, line: 5192
> >
> > Yup, that's pretty clear indication of a corrupted extent record.
> >
> 
> What is the best way to prevent  transactions that record bad
> extent length and block numbers.

That should never occur - there are already checks in place to
prevent that. However, the log must be treated as potentially
corrupt during recovery, so when freeing extents on recovered files
we might be walking corrupt extents. xfs_bunmapi() is the place that
should be checking that the extent being freed is of a sane length.
(Just like xfs_bmapi checks to ensure the extent allocated is of
sane length).


> > That's the open, unlinked file at the time the system crashed. That
> > may be where your problems are coming from. The RT is mostly
> > untested, and we sure as anything don't do any crash resiliency or
> > recovery testing on it, so there's a good chance there are bugs in
> > it that might show up in situations like this....
> >
> > You need to detect extents with invalid lengths in them and trigger
> > a corruption-based filesystem shutdown.
> 
> Looked at the log during one of the filesystem shutdown when the
> I/O error occurs. is this an indication of already corrupted log due to
> corrupted in-memory metadata structures?.
> ===
> attempt to access beyond end of device
> sda2: rw=0, want=33792081130943048, limit=31471329
> I/O error in filesystem ("sda2") meta-data dev sda2 block
> 0x780db80007f240       ("xfs_trans_read_buf") error 5 buf count 4096
> xfs_force_shutdown(sda2,0x1) called from line 395 of file
> fs/xfs/xfs_trans_buf.c.  Return address = 0x801f4f88
> Filesystem "sda2": I/O Error Detected.  Shutting down filesystem: sda2
> Please umount the filesystem, and rectify the problem(s)

The I/O error is what triggered the shutdown. The transaction tried
to read a metadata block beyond EOF.

> However the log is already corrupted. So is there a check on a write
> to the log ?.

No, the above check caught the corruption as soon as it was found.
You need to walk back from this event to find where the corruption
was introduced.

> >> also if there is something that can be done to avoid this situation in
> >> the first place.
> >
> > Track down where those stray upper bits in the block numbers are
> > coming from, and you'll have your answer.
> >
> 
> Have not been able to track this down yet. But could it be a possible memory
> corruption, leading to the in-memory metadata to get corrupted.

Yes, that is a possible cause that lead to a bad block number being
written to disk.

> On a similar occurrence of this issue on recovery after a reboot seems
> to always go through the evict path
> 
> Filesystem "sda2": XFS internal error xfs_trans_cancel at line 1815
> of file fs/xfs/xfs_trans.c.  Caller 0x801f8524
> 
> Call Trace:
> [<80439d2c>] dump_stack+0x8/0x34
> [<801f3bec>] xfs_trans_cancel+0x10c/0x128
> [<801f8524>] xfs_inactive+0x2fc/0x450
> [<800dcd54>] evict+0x28/0xd0
> [<800dd300>] iput+0x19c/0x2d8
> [<801e5bcc>] xlog_recover_process_one_iunlink+0xec/0x130
> [<801e7b60>] xlog_recover_process_iunlinks.clone.25+0xa8/0x108
> [<801eb360>] xlog_recover_finish+0x40/0x100
> [<801eedd8>] xfs_mountfs+0x434/0x654

That's where it is processing files that were unlinked but still
referenced at the time of the crash. We already know that these are
the corrupted files...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: xfs filesystem corruption with kernel 2.6.37
  2012-11-02  1:27     ` Dave Chinner
@ 2012-11-02 16:34       ` Kamal Dasu
  2012-11-02 22:55         ` Dave Chinner
  0 siblings, 1 reply; 9+ messages in thread
From: Kamal Dasu @ 2012-11-02 16:34 UTC (permalink / raw)
  To: xfs


Dave,

I see the following. On two different systems with the same want= value.
Does not seem like a random corruption.

>attempt to access beyond end of device
> sda2: rw=0, want=33792081130943048, limit=3147132

> > Track down where those stray upper bits in the block numbers are
> > coming from, and you'll have your answer. 

Where would be the best place to put this check. 

Also on a XFS DEBUG all the asserts seem to be in the unlink (delete path).


Kamal

-- 
View this message in context: http://old.nabble.com/xfs-filesystem-corruption-with-kernel-2.6.37-tp34601185p34633803.html
Sent from the Xfs - General mailing list archive at Nabble.com.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: xfs filesystem corruption with kernel 2.6.37
  2012-11-02 16:34       ` Kamal Dasu
@ 2012-11-02 22:55         ` Dave Chinner
  2012-11-03  1:57           ` Kamal Dasu
  0 siblings, 1 reply; 9+ messages in thread
From: Dave Chinner @ 2012-11-02 22:55 UTC (permalink / raw)
  To: Kamal Dasu; +Cc: xfs

On Fri, Nov 02, 2012 at 09:34:32AM -0700, Kamal Dasu wrote:
> 
> Dave,
> 
> I see the following. On two different systems with the same want= value.
> Does not seem like a random corruption.
> 
> >attempt to access beyond end of device
> > sda2: rw=0, want=33792081130943048, limit=3147132

0x780dbd80007f248

Once again it's corruption in the upper 32 bits of the 64 bit number.
Those bits should be zero. Perhaps looking at all the trace events
from recovery might give you a closer approximation of where the bad
extent records is found....

> > > Track down where those stray upper bits in the block numbers are
> > > coming from, and you'll have your answer. 
> 
> Where would be the best place to put this check. 

I don't know - you're just just going to have to put cheks
everywhere - from when the extent record is first read from disk, to
where it is modified to where it is backed back into disk format.

> Also on a XFS DEBUG all the asserts seem to be in the unlink (delete path).

No surprise, the failures found when removing open-but-unlinked
files after a crash. i.e. during unlink in the final stage of
recover.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: xfs filesystem corruption with kernel 2.6.37
  2012-11-02 22:55         ` Dave Chinner
@ 2012-11-03  1:57           ` Kamal Dasu
  2012-11-03 22:25             ` Dave Chinner
  0 siblings, 1 reply; 9+ messages in thread
From: Kamal Dasu @ 2012-11-03  1:57 UTC (permalink / raw)
  To: Dave Chinner; +Cc: xfs

Dave,


> 
> 0x780dbd80007f248
> 
> Once again it's corruption in the upper 32 bits of the 64 bit number.
> Those bits should be zero. Perhaps looking at all the trace events
> from recovery might give you a closer approximation of where the bad
> extent records is found....

I have been trying to use the xfs_db xfs_logprint, how ever xfs_logprint bails out when it finds an error. 
How do I look at trace events ?.


Kamal

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: xfs filesystem corruption with kernel 2.6.37
  2012-11-03  1:57           ` Kamal Dasu
@ 2012-11-03 22:25             ` Dave Chinner
  2012-11-09 21:18               ` Kamal Dasu
  0 siblings, 1 reply; 9+ messages in thread
From: Dave Chinner @ 2012-11-03 22:25 UTC (permalink / raw)
  To: Kamal Dasu; +Cc: xfs

On Fri, Nov 02, 2012 at 09:57:27PM -0400, Kamal Dasu wrote:
> Dave,
> 
> 
> > 
> > 0x780dbd80007f248
> > 
> > Once again it's corruption in the upper 32 bits of the 64 bit number.
> > Those bits should be zero. Perhaps looking at all the trace events
> > from recovery might give you a closer approximation of where the bad
> > extent records is found....
> 
> I have been trying to use the xfs_db xfs_logprint, how ever
> xfs_logprint bails out when it finds an error.

$ man xfs_logprint
....
	-c     Attempt to continue when an error is detected.

> How do I look at
> trace events ?.

Documentation/trace/events.txt

Or better yet, use trace-cmd as suggested in the FAQ:

http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: xfs filesystem corruption with kernel 2.6.37
  2012-11-03 22:25             ` Dave Chinner
@ 2012-11-09 21:18               ` Kamal Dasu
  0 siblings, 0 replies; 9+ messages in thread
From: Kamal Dasu @ 2012-11-09 21:18 UTC (permalink / raw)
  To: xfs


Dave,

On more analysis of the corrupt disks seems exhibit a problem with duplicate
rt inode extents.

On mount this would be the assert:
 prev.br_state == XFS_EXT_NORM assert with Anita disk)
Starting XFS recovery on filesystem: sda2 (logdev: internal)
Assertion failed: del.br_startblock == prev.br_startblock +
prev.br_blockcount, file: fs/xfs/xfs_bmap.c, line: 5201

# xfs_repair –n /dev/sda2 –r /dev/sda3
..
..
Phase 2 - using internal log
        - scan filesystem freespace and inode maps...
agi unlinked bucket 38 is 138790 in ag 3 (inode=50470438)
..
..
Phase 4 - check for duplicate blocks...
        - setting up duplicate extent list...
        - check for inodes claiming duplicate blocks...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
data fork in rt ino 50470438 claims dup rt extent,off - 1024, start -
12963072, count 160
bad data fork in inode 50470438
would have cleared inode 50470438

sh-3.1# xfs_db  /dev/sda2
xfs_db> inode 50470438
xfs_db> bmap
data offset 0 startblock 11024384 (10/538624) count 256 flag 0
data offset 1024 startblock 12963072 (12/380160) count 160 flag 1
data offset 1184 startblock 12963232 (12/380320) count 1120 flag 0
data offset 2304 startblock 12951808 (12/368896) count 1024 flag 0
data offset 3328 startblock 12824064 (12/241152) count 768 flag 0
data offset 6356034 startblock 2392537314989568 (2281701388/366080) count
928 flag 0

Similar problem in context of xfs_repair was reported with respect to
realtime extents on 2.6.37. Seems like unaligned extent size can cause
sharing of the  unwritten free part of a written  with the next allocated
unwritten adjecent extent .  During runtime (with XFS_DEBUG enabled) we do
notice that the same extent is being unlinked multiple times and an assert
for.

Assertion failed: fsb != NULLFSBLOCK, file: fs/xfs/xfs_rtalloc.c, line: 875
 
I am wondering if this can cause corruption when it happens during runtime
without XFS_DEBUG build.

I was looking at a similar thread where you have commented regarding a
possible fix in the kernel bug duplicate extent which overlap due unaligned
rt extent size calculation.

http://oss.sgi.com/archives/xfs/2012-09/msg00292.html

Thanks
Kamal
-- 
View this message in context: http://old.nabble.com/xfs-filesystem-corruption-with-kernel-2.6.37-tp34601185p34662440.html
Sent from the Xfs - General mailing list archive at Nabble.com.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 9+ messages in thread

end of thread, other threads:[~2012-11-09 21:16 UTC | newest]

Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2012-10-25 13:45 xfs filesystem corruption with kernel 2.6.37 Kamal Dasu
2012-10-25 22:47 ` Dave Chinner
2012-11-01 19:30   ` Kamal Dasu
2012-11-02  1:27     ` Dave Chinner
2012-11-02 16:34       ` Kamal Dasu
2012-11-02 22:55         ` Dave Chinner
2012-11-03  1:57           ` Kamal Dasu
2012-11-03 22:25             ` Dave Chinner
2012-11-09 21:18               ` Kamal Dasu

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.