File System Corruption - Internal error xfs_dir3_data_reada_verify

File System Corruption - Internal error xfs_dir3_data_reada_verify

[Richard Neuboeck]

[-- Attachment #1.1.1: Type: text/plain, Size: 1184 bytes --]

Hi,

for some time now our storage machine using XFS stops the file
system due to some reason I don't seem to have found so far. In this
process the file system gets corrupted and the attached trace log is
shown. After xfs_repair is run it's running again for an always
changing amount of time. In general it fails within a few hours or
days. There are no relevant log messages before the entries shown
below and no immediate actions that lead to this condition. So far
my experiments (Ubuntu upgrade from 10.04 to 14.04, different kernel
versions, changes to the hypervisor) didn't show any lasting effects
(positive or negative). If any one could shed some light on what XFS
is trying to tell me it would be highly appreciated.

I've found the mention of 'xfs_dir3_data_reada_verify' in the
mailing list but didn't find a solution that was applicable.

Thanks in advance
Richard


Operating System: Ubuntu 14.04 x86_64
Kernel: 3.13.0-32-generic
Machine: Virtual, gets an LVM device (/dev/vdb)

The system hosting the virtual machine is Ubuntu lucid (10.04LTS)
x86_64. There are no log messages related to this incident on this
machine.

-- 
/dev/null

[-- Attachment #1.1.2: xfs_trace.log --]
[-- Type: text/plain, Size: 9122 bytes --]

Aug 13 10:11:16 huddle kernel: [ 5247.325984] ffff880113855000: 20 73 63 6f 70 65 2e 52 65 73 65 74 48 69 73 74   scope.ResetHist
Aug 13 10:11:16 huddle kernel: [ 5247.326066] ffff880113855010: 6f 72 79 28 29 3b 0a 20 20 20 20 20 20 20 20 20  ory();.         
Aug 13 10:11:16 huddle kernel: [ 5247.326107] ffff880113855020: 20 20 20 20 20 20 20 68 61 6e 64 6c 65 2e 52 65         handle.Re
Aug 13 10:11:16 huddle kernel: [ 5247.326147] ffff880113855030: 73 65 74 28 29 3b 0a 20 20 20 20 20 20 20 20 20  set();.         
Aug 13 10:11:16 huddle kernel: [ 5247.326196] XFS (vdb): Internal error xfs_dir3_data_reada_verify at line 258 of file /build/buildd/linux-3.13.0/fs/xfs/xfs_dir2_data.c.  Caller 0xffffffffa006b6c5
Aug 13 10:11:16 huddle kernel: [ 5247.326276] CPU: 0 PID: 341 Comm: kworker/0:1H Not tainted 3.13.0-32-generic #57-Ubuntu
Aug 13 10:11:16 huddle kernel: [ 5247.326278] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2007
Aug 13 10:11:16 huddle kernel: [ 5247.326352] Workqueue: xfslogd xfs_buf_iodone_work [xfs]
Aug 13 10:11:16 huddle kernel: [ 5247.326357]  0000000000000001 ffff880211f97d88 ffffffff8171bcb4 ffff8802125bd000
Aug 13 10:11:16 huddle kernel: [ 5247.326361]  ffff880211f97da0 ffffffffa006e53b ffffffffa006b6c5 ffff880211f97dd8
Aug 13 10:11:16 huddle kernel: [ 5247.326363]  ffffffffa006e595 0000010211f97dc8 ffff880101aa8000 ffff880101aa8000
Aug 13 10:11:16 huddle kernel: [ 5247.326365] Call Trace:
Aug 13 10:11:16 huddle kernel: [ 5247.326373]  [<ffffffff8171bcb4>] dump_stack+0x45/0x56
Aug 13 10:11:16 huddle kernel: [ 5247.326385]  [<ffffffffa006e53b>] xfs_error_report+0x3b/0x40 [xfs]
Aug 13 10:11:16 huddle kernel: [ 5247.326395]  [<ffffffffa006b6c5>] ? xfs_buf_iodone_work+0x85/0xf0 [xfs]
Aug 13 10:11:16 huddle kernel: [ 5247.326406]  [<ffffffffa006e595>] xfs_corruption_error+0x55/0x80 [xfs]
Aug 13 10:11:16 huddle kernel: [ 5247.326442]  [<ffffffffa00a9012>] xfs_dir3_data_reada_verify+0x62/0xc0 [xfs]
Aug 13 10:11:16 huddle kernel: [ 5247.326451]  [<ffffffffa006b6c5>] ? xfs_buf_iodone_work+0x85/0xf0 [xfs]
Aug 13 10:11:16 huddle kernel: [ 5247.326460]  [<ffffffffa006b6c5>] xfs_buf_iodone_work+0x85/0xf0 [xfs]
Aug 13 10:11:16 huddle kernel: [ 5247.326465]  [<ffffffff810838f2>] process_one_work+0x182/0x450
Aug 13 10:11:16 huddle kernel: [ 5247.326478]  [<ffffffff810846e1>] worker_thread+0x121/0x410
Aug 13 10:11:16 huddle kernel: [ 5247.326481]  [<ffffffff810845c0>] ? rescuer_thread+0x430/0x430
Aug 13 10:11:16 huddle kernel: [ 5247.326484]  [<ffffffff8108b3d2>] kthread+0xd2/0xf0
Aug 13 10:11:16 huddle kernel: [ 5247.326486]  [<ffffffff8108b300>] ? kthread_create_on_node+0x1d0/0x1d0
Aug 13 10:11:16 huddle kernel: [ 5247.326490]  [<ffffffff8172c5bc>] ret_from_fork+0x7c/0xb0
Aug 13 10:11:16 huddle kernel: [ 5247.326492]  [<ffffffff8108b300>] ? kthread_create_on_node+0x1d0/0x1d0
Aug 13 10:11:16 huddle kernel: [ 5247.326494] XFS (vdb): Corruption detected. Unmount and run xfs_repair
Aug 13 10:11:16 huddle kernel: [ 5247.326532] XFS (vdb): metadata I/O error: block 0x160003e488 ("xfs_trans_read_buf_map") error 117 numblks 8
Aug 13 10:11:16 huddle kernel: [ 5247.326789] ffff880113855000: 20 73 63 6f 70 65 2e 52 65 73 65 74 48 69 73 74   scope.ResetHist
Aug 13 10:11:16 huddle kernel: [ 5247.326831] ffff880113855010: 6f 72 79 28 29 3b 0a 20 20 20 20 20 20 20 20 20  ory();.         
Aug 13 10:11:16 huddle kernel: [ 5247.326870] ffff880113855020: 20 20 20 20 20 20 20 68 61 6e 64 6c 65 2e 52 65         handle.Re
Aug 13 10:11:16 huddle kernel: [ 5247.326909] ffff880113855030: 73 65 74 28 29 3b 0a 20 20 20 20 20 20 20 20 20  set();.         
Aug 13 10:11:16 huddle kernel: [ 5247.326949] XFS (vdb): Internal error xfs_dir3_data_read_verify at line 274 of file /build/buildd/linux-3.13.0/fs/xfs/xfs_dir2_data.c.  Caller 0xffffffffa006b6c5
Aug 13 10:11:16 huddle kernel: [ 5247.327029] CPU: 0 PID: 341 Comm: kworker/0:1H Not tainted 3.13.0-32-generic #57-Ubuntu
Aug 13 10:11:16 huddle kernel: [ 5247.327031] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2007
Aug 13 10:11:16 huddle kernel: [ 5247.327042] Workqueue: xfslogd xfs_buf_iodone_work [xfs]
Aug 13 10:11:16 huddle kernel: [ 5247.327044]  0000000000000001 ffff880211f97d68 ffffffff8171bcb4 ffff8802125bd000
Aug 13 10:11:16 huddle kernel: [ 5247.327045]  ffff880211f97d80 ffffffffa006e53b ffffffffa006b6c5 ffff880211f97db8
Aug 13 10:11:16 huddle kernel: [ 5247.327047]  ffffffffa006e595 0000011211f97db0 ffff880036ab1300 ffff8802125bd000
Aug 13 10:11:16 huddle kernel: [ 5247.327049] Call Trace:
Aug 13 10:11:16 huddle kernel: [ 5247.327052]  [<ffffffff8171bcb4>] dump_stack+0x45/0x56
Aug 13 10:11:16 huddle kernel: [ 5247.327061]  [<ffffffffa006e53b>] xfs_error_report+0x3b/0x40 [xfs]
Aug 13 10:11:16 huddle kernel: [ 5247.327070]  [<ffffffffa006b6c5>] ? xfs_buf_iodone_work+0x85/0xf0 [xfs]
Aug 13 10:11:16 huddle kernel: [ 5247.327079]  [<ffffffffa006e595>] xfs_corruption_error+0x55/0x80 [xfs]
Aug 13 10:11:16 huddle kernel: [ 5247.327092]  [<ffffffffa00a8f2d>] xfs_dir3_data_read_verify+0x6d/0xf0 [xfs]
Aug 13 10:11:16 huddle kernel: [ 5247.327101]  [<ffffffffa006b6c5>] ? xfs_buf_iodone_work+0x85/0xf0 [xfs]
Aug 13 10:11:16 huddle kernel: [ 5247.327104]  [<ffffffff81097508>] ? finish_task_switch+0x128/0x170
Aug 13 10:11:16 huddle kernel: [ 5247.327113]  [<ffffffffa006b6c5>] xfs_buf_iodone_work+0x85/0xf0 [xfs]
Aug 13 10:11:16 huddle kernel: [ 5247.327115]  [<ffffffff810838f2>] process_one_work+0x182/0x450
Aug 13 10:11:16 huddle kernel: [ 5247.327117]  [<ffffffff810846e1>] worker_thread+0x121/0x410
Aug 13 10:11:16 huddle kernel: [ 5247.327119]  [<ffffffff810845c0>] ? rescuer_thread+0x430/0x430
Aug 13 10:11:16 huddle kernel: [ 5247.327121]  [<ffffffff8108b3d2>] kthread+0xd2/0xf0
Aug 13 10:11:16 huddle kernel: [ 5247.327122]  [<ffffffff8108b300>] ? kthread_create_on_node+0x1d0/0x1d0
Aug 13 10:11:16 huddle kernel: [ 5247.327124]  [<ffffffff8172c5bc>] ret_from_fork+0x7c/0xb0
Aug 13 10:11:16 huddle kernel: [ 5247.327126]  [<ffffffff8108b300>] ? kthread_create_on_node+0x1d0/0x1d0
Aug 13 10:11:16 huddle kernel: [ 5247.327127] XFS (vdb): Corruption detected. Unmount and run xfs_repair
Aug 13 10:11:16 huddle kernel: [ 5247.327164] XFS (vdb): metadata I/O error: block 0x160003e488 ("xfs_trans_read_buf_map") error 117 numblks 8
Aug 13 10:11:21 huddle kernel: [ 5252.482540] XFS: Internal error XFS_WANT_CORRUPTED_GOTO at line 1602 of file /build/buildd/linux-3.13.0/fs/xfs/xfs_alloc.c.  Caller 0xffffffffa0088485
Aug 13 10:11:21 huddle kernel: [ 5252.482644] CPU: 0 PID: 2167 Comm: rm Not tainted 3.13.0-32-generic #57-Ubuntu
Aug 13 10:11:21 huddle kernel: [ 5252.482646] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2007
Aug 13 10:11:21 huddle kernel: [ 5252.482648]  0000000000fef248 ffff88010d125b88 ffffffff8171bcb4 0000000000000004
Aug 13 10:11:21 huddle kernel: [ 5252.482651]  ffff88010d125ba0 ffffffffa006e53b ffffffffa0088485 ffff88010d125c38
Aug 13 10:11:21 huddle kernel: [ 5252.482653]  ffffffffa0086f19 ffff8802125bd000 ffff880033fff780 ffff880211f9b9c0
Aug 13 10:11:21 huddle kernel: [ 5252.482655] Call Trace:
Aug 13 10:11:21 huddle kernel: [ 5252.482664]  [<ffffffff8171bcb4>] dump_stack+0x45/0x56
Aug 13 10:11:21 huddle kernel: [ 5252.482694]  [<ffffffffa006e53b>] xfs_error_report+0x3b/0x40 [xfs]
Aug 13 10:11:21 huddle kernel: [ 5252.482706]  [<ffffffffa0088485>] ? xfs_free_extent+0xc5/0xf0 [xfs]
Aug 13 10:11:21 huddle kernel: [ 5252.482718]  [<ffffffffa0086f19>] xfs_free_ag_extent+0x1e9/0x710 [xfs]
Aug 13 10:11:21 huddle kernel: [ 5252.482730]  [<ffffffffa0088485>] xfs_free_extent+0xc5/0xf0 [xfs]
Aug 13 10:11:21 huddle kernel: [ 5252.482748]  [<ffffffffa00680df>] xfs_bmap_finish+0x13f/0x190 [xfs]
Aug 13 10:11:21 huddle kernel: [ 5252.482766]  [<ffffffffa00b6acd>] xfs_itruncate_extents+0x16d/0x2a0 [xfs]
Aug 13 10:11:21 huddle kernel: [ 5252.482781]  [<ffffffffa00b6c8d>] xfs_inactive_truncate+0x8d/0x120 [xfs]
Aug 13 10:11:21 huddle kernel: [ 5252.482812]  [<ffffffffa00b7438>] xfs_inactive+0x138/0x160 [xfs]
Aug 13 10:11:21 huddle kernel: [ 5252.482844]  [<ffffffffa007fa50>] xfs_fs_evict_inode+0x80/0xc0 [xfs]
Aug 13 10:11:21 huddle kernel: [ 5252.482848]  [<ffffffff811d7d70>] evict+0xb0/0x1b0
Aug 13 10:11:21 huddle kernel: [ 5252.482850]  [<ffffffff811d8585>] iput+0xf5/0x180
Aug 13 10:11:21 huddle kernel: [ 5252.482852]  [<ffffffff811ccf8e>] do_unlinkat+0x18e/0x2b0
Aug 13 10:11:21 huddle kernel: [ 5252.482857]  [<ffffffff81020d45>] ? syscall_trace_enter+0x145/0x250
Aug 13 10:11:21 huddle kernel: [ 5252.482859]  [<ffffffff811cdeab>] SyS_unlinkat+0x1b/0x40
Aug 13 10:11:21 huddle kernel: [ 5252.482863]  [<ffffffff8172c87f>] tracesys+0xe1/0xe6
Aug 13 10:11:21 huddle kernel: [ 5252.482872] XFS (vdb): xfs_do_force_shutdown(0x8) called from line 138 of file /build/buildd/linux-3.13.0/fs/xfs/xfs_bmap_util.c.  Return address = 0xffffffffa0068118
Aug 13 10:11:21 huddle kernel: [ 5252.483790] XFS (vdb): Corruption of in-memory data detected.  Shutting down filesystem
Aug 13 10:11:21 huddle kernel: [ 5252.483830] XFS (vdb): Please umount the filesystem and rectify the problem(s)

[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: File System Corruption - Internal error xfs_dir3_data_reada_verify

[Dave Chinner]

On Wed, Aug 13, 2014 at 11:42:26AM +0200, Richard Neuboeck wrote:
> Hi,
> 
> for some time now our storage machine using XFS stops the file
> system due to some reason I don't seem to have found so far. In this
> process the file system gets corrupted and the attached trace log is
> shown.

What's the workload the VM runs?

> After xfs_repair is run it's running again for an always
> changing amount of time. 

What errors does xfs_repair correct? Can you post the output of a
repair run that corrects the issue.

> In general it fails within a few hours or
> days. There are no relevant log messages before the entries shown
> below and no immediate actions that lead to this condition. So far
> my experiments (Ubuntu upgrade from 10.04 to 14.04, different kernel
> versions, changes to the hypervisor) didn't show any lasting effects
> (positive or negative). If any one could shed some light on what XFS
> is trying to tell me it would be highly appreciated.

The directory is trying to read a block of data that does not
contain directory data. i.e. the directory has somehow been
corrupted. The block contains file data, but that's about all
I can tell you right now.

> I've found the mention of 'xfs_dir3_data_reada_verify' in the
> mailing list but didn't find a solution that was applicable.

It's just checking the block read from disk.

However, that's not the only error that is occurring:

> [ 5247.327164] XFS (vdb): metadata I/O error: block 0x160003e488 ("xfs_trans_read_buf_map") error 117 numblks 8
> [ 5252.482540] XFS: Internal error XFS_WANT_CORRUPTED_GOTO at line 1602 of file /build/buildd/linux-3.13.0/fs/xfs/xfs_alloc.c.  Caller 0xffffffffa0088485

There are corrupted free space btrees. In this case, the by-bno tree
has been found to be inconsistent. So there's something corrupting
more than just the directory.

SO, more information needed. Lets start with:

http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F

and the output of xfs_repair. Also, a metadump image of the
filesystem before you run repair would be helpful. And finally, the
configuration of the block devices the VM is using (i.e. virtio,
cache=?, etc). Describing the physical storage the VM is using might
also be helpful - it could be host based corruption, not guest based
corruption that is occurring...

Cheers,

Dave.

> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs


-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs