* md raid5 blocked in get_active_stripe after disk failure
@ 2011-01-12 13:04 David Flynn
2011-01-12 14:06 ` David Flynn
2011-01-12 20:27 ` NeilBrown
0 siblings, 2 replies; 6+ messages in thread
From: David Flynn @ 2011-01-12 13:04 UTC (permalink / raw)
To: linux-raid
I have a fileserver that mounts an md raid5 array with 20 active disks,
holding an xfs filesystem which is then exported via nfs. (system is
a twin-quadcore intel, 64bit)
I've witnessed a failure which resulted in a reproducible lockup of the
array after one of the array components suffered a read error.
The order of events was roughly:
1) disk fails read:
<snip mny errors>
[103364.894588] sd 3:4:0:0: [sdt] Add. Sense: Unrecovered read error - auto reallocate failed
[103364.949216] sd 3:4:0:0: [sdt] CDB: Read(10): 28 00 0b d3 95 3f 00 01 00 00
[103364.969589] end_request: I/O error, dev sdt, sector 198415713
[103364.969619] ata4: EH complete
2) nfs clients report hung tasks
3) file server reports hung tasks
(re-reviewing this in the cold light of the morning, makes me wonder
if the cause of the problem isn't in pid 2777, eitherway it is too early
to thrink straight)
[log is included at end of message]
4) an hour passes, i'm alerted to the issue
5) top reports 30-40% cpu utilization in iowait, the rest idle
6) remembering an old strip_cache issue, i note that:
/sys/class/block/md1/md/stripe_cache_active = 0
/sys/class/block/md1/md/stripe_cache_size = 256
doubling stripe_cache_size has no effect.
7) mark sdt1 as faulty:
[109593.732485] md/raid:md1: Disk failure on sdt1, disabling device.
[109593.732487] <1>md/raid:md1: Operation continuing on 19 devices.
8) try to remove sdt1:
[110152.939464] md: cannot remove active disk sdt1 from md1 ...
9) attempting to run perf top blocks.
10) use fdisk to partition another disk, (write out partition table is ok)
but fdisk blocks while requesting a partition table scan. although
using hdparm -z directly on the disk succeeds.
11) can't reboot, can't sync, can't unmount file systems, etc.,
eventually powercycle the machine
When the machine reboots, restarts the array (was dirty, so rebuilds),
remounts the filesystems and nfs comes back up and the clients retry their
requests, the disk error occurred again and md locked up in the same way.
Any thoughts are greatly appreciated.
Kind regards,
..david
== kernel log from (3) above ==
[103561.064060] INFO: task md1_raid5:2371 blocked for more than 120 seconds.
[103561.109844] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[103561.162479] md1_raid5 D 0000000101892965 0 2371 2 0x00000000
[103561.209434] ffff880426984af0 0000000000000046 ffff88042ae12fb0 ffff880000000000
[103561.259613] ffffffff81632020 00000000000154c0 00000000000154c0 00000000000154c0
[103561.309815] ffff880425123fd8 00000000000154c0 ffff880426984af0 ffff880425123fd8
[103561.359995] Call Trace:
[103561.380375] [<ffffffffa026b133>] ? get_active_stripe+0xfe/0x55b [raid456]
[103561.427278] [<ffffffff8103f760>] ? default_wake_function+0x0/0xf
[103561.469536] [<ffffffffa026cf92>] ? raid5d+0x149/0x3f2 [raid456]
[103561.511297] [<ffffffffa0228010>] ? md_thread+0xf2/0x110 [md_mod]
[103561.553580] [<ffffffff8105b995>] ? autoremove_wake_function+0x0/0x2a
[103561.597882] [<ffffffffa0227f1e>] ? md_thread+0x0/0x110 [md_mod]
[103561.639553] [<ffffffff8105b541>] ? kthread+0x75/0x7d
[103561.675472] [<ffffffff810097e4>] ? kernel_thread_helper+0x4/0x10
[103561.717604] [<ffffffff8105b4cc>] ? kthread+0x0/0x7d
[103561.752937] [<ffffffff810097e0>] ? kernel_thread_helper+0x0/0x10
[103561.795009] INFO: task flush-9:1:2491 blocked for more than 120 seconds.
[103561.840753] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[103561.893356] flush-9:1 D 0000000101892965 0 2491 2 0x00000000
[103561.940355] ffff880426b206d0 0000000000000046 ffff88042486d70c ffffffff00000000
[103561.990544] ffff88042e0da8e0 00000000000154c0 00000000000154c0 00000000000154c0
[103562.040754] ffff88042486dfd8 00000000000154c0 ffff880426b206d0 ffff88042486dfd8
[103562.090973] Call Trace:
[103562.111360] [<ffffffffa026b133>] ? get_active_stripe+0xfe/0x55b [raid456]
[103562.158352] [<ffffffff8103f760>] ? default_wake_function+0x0/0xf
[103562.200605] [<ffffffffa026eafa>] ? make_request+0x1dd/0x403 [raid456]
[103562.245486] [<ffffffff8105b995>] ? autoremove_wake_function+0x0/0x2a
[103562.289870] [<ffffffffa02267ca>] ? md_make_request+0xd0/0x1d4 [md_mod]
[103562.335319] [<ffffffff8117583c>] ? generic_make_request+0x27a/0x2f0
[103562.379181] [<ffffffff810b09dd>] ? mempool_alloc+0x53/0x106
[103562.418843] [<ffffffff81175983>] ? submit_bio+0xd1/0xed
[103562.456376] [<ffffffff8118b006>] ? radix_tree_tag_set+0x64/0xc8
[103562.498025] [<ffffffff8110b97c>] ? __bio_add_page+0x156/0x1f1
[103562.538646] [<ffffffffa05aa640>] ? xfs_submit_ioend_bio+0x53/0x5f [xfs]
[103562.584444] [<ffffffffa05aa702>] ? xfs_submit_ioend+0xb6/0xd8 [xfs]
[103562.628140] [<ffffffffa05ab62a>] ? xfs_page_state_convert+0x4bd/0x50e [xfs]
[103562.676040] [<ffffffffa05ab843>] ? xfs_vm_writepage+0xef/0x12c [xfs]
[103562.720299] [<ffffffff810b548a>] ? __writepage+0xa/0x21
[103562.757774] [<ffffffff810b6504>] ? write_cache_pages+0x1d8/0x2ec
[103562.799928] [<ffffffff810b5480>] ? __writepage+0x0/0x21
[103562.837380] [<ffffffff81102cd4>] ? writeback_single_inode+0xd1/0x2e5
[103562.881556] [<ffffffff8110323e>] ? writeback_sb_inodes+0x136/0x204
[103562.924674] [<ffffffff811038e3>] ? writeback_inodes_wb+0x127/0x139
[103562.967663] [<ffffffff81103a83>] ? wb_writeback+0x18e/0x208
[103563.006933] [<ffffffff81103c30>] ? wb_do_writeback+0x133/0x149
[103563.047721] [<ffffffff81103c80>] ? bdi_writeback_task+0x3a/0x112
[103563.089528] [<ffffffff8105b8bb>] ? bit_waitqueue+0x14/0xa1
[103563.128129] [<ffffffff810c1f4f>] ? bdi_start_fn+0x0/0xc3
[103563.165632] [<ffffffff810c1fb2>] ? bdi_start_fn+0x63/0xc3
[103563.165644] [<ffffffff810c1f4f>] ? bdi_start_fn+0x0/0xc3
[103563.165656] [<ffffffff8105b541>] ? kthread+0x75/0x7d
[103563.165667] [<ffffffff810097e4>] ? kernel_thread_helper+0x4/0x10
[103563.165679] [<ffffffff8105b4cc>] ? kthread+0x0/0x7d
[103563.165690] [<ffffffff810097e0>] ? kernel_thread_helper+0x0/0x10
[103563.165708] INFO: task xfssyncd/md1:2777 blocked for more than 120 seconds.
[103563.165715] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[103563.165724] xfssyncd/md1 D ffff88042b1c3680 0 2777 2 0x00000000
[103563.165735] ffff88042b1c3680 0000000000000046 ffff88042a4ed000 ffffffffa002b0e0
[103563.165747] ffff88042e121b40 00000000000154c0 00000000000154c0 00000000000154c0
[103563.165758] ffff880424e61fd8 00000000000154c0 ffff88042b1c3680 ffff880424e61fd8
[103563.165770] Call Trace:
[103563.165797] [<ffffffffa002b0e0>] ? scsi_setup_blk_pc_cmnd+0x39/0xff [scsi_mod]
[103563.165823] [<ffffffffa01a426c>] ? __ata_scsi_queuecmd+0x17f/0x1d6 [libata]
[103563.165838] [<ffffffff8100e7d9>] ? sched_clock+0x5/0x8
[103563.165850] [<ffffffffa0269520>] ? raid5_quiesce+0xe3/0x19b [raid456]
[103563.165864] [<ffffffff8103f760>] ? default_wake_function+0x0/0xf
[103563.165877] [<ffffffffa026e94e>] ? make_request+0x31/0x403 [raid456]
[103563.165893] [<ffffffffa02267ca>] ? md_make_request+0xd0/0x1d4 [md_mod]
[103563.165906] [<ffffffff8117583c>] ? generic_make_request+0x27a/0x2f0
[103563.165919] [<ffffffff810b09dd>] ? mempool_alloc+0x53/0x106
[103563.165931] [<ffffffff81175983>] ? submit_bio+0xd1/0xed
[103563.165959] [<ffffffffa05ac4d2>] ? _xfs_buf_ioapply+0x212/0x24f [xfs]
[103563.165987] [<ffffffffa0597ace>] ? xlog_bdstrat+0x3a/0x3e [xfs]
[103563.166015] [<ffffffffa05ad430>] ? xfs_buf_iorequest+0x75/0x8e [xfs]
[103563.166043] [<ffffffffa0597ace>] ? xlog_bdstrat+0x3a/0x3e [xfs]
[103563.166070] [<ffffffffa0597ce1>] ? xlog_sync+0x20f/0x326 [xfs]
[103563.166098] [<ffffffffa05986d4>] ? _xfs_log_force+0xd9/0x196 [xfs]
[103563.166111] [<ffffffff81050771>] ? process_timeout+0x0/0xa
[103563.166138] [<ffffffffa0598915>] ? xfs_log_force+0xb/0x2a [xfs]
[103563.166167] [<ffffffffa05b537f>] ? xfs_sync_worker+0x14/0x60 [xfs]
[103563.166195] [<ffffffffa05b4c2e>] ? xfssyncd+0x155/0x1d1 [xfs]
[103563.166223] [<ffffffffa05b4ad9>] ? xfssyncd+0x0/0x1d1 [xfs]
[103563.166235] [<ffffffff8105b541>] ? kthread+0x75/0x7d
[103563.166246] [<ffffffff810097e4>] ? kernel_thread_helper+0x4/0x10
[103563.166258] [<ffffffff8105b4cc>] ? kthread+0x0/0x7d
[103563.166269] [<ffffffff810097e0>] ? kernel_thread_helper+0x0/0x10
[103563.166284] INFO: task nfsd:3494 blocked for more than 120 seconds.
[103563.166292] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[103563.166302] nfsd D 000000010189190f 0 3494 2 0x00000000
[103563.166314] ffff88042ae14420 0000000000000046 ffff880200001000 0000000000000000
[103563.166325] ffff88042e0d8da0 00000000000154c0 00000000000154c0 00000000000154c0
[103563.166337] ffff880420a03fd8 00000000000154c0 ffff88042ae14420 ffff880420a03fd8
[103563.166348] Call Trace:
[103563.166354] [<ffffffff8100e806>] ? read_tsc+0x5/0x16
[103563.166365] [<ffffffff810aefe5>] ? sync_page+0x0/0x45
[103563.166378] [<ffffffff813059f0>] ? io_schedule+0x6b/0xaa
[103563.166390] [<ffffffff810af026>] ? sync_page+0x41/0x45
[103563.166399] [<ffffffff81305e3b>] ? __wait_on_bit_lock+0x3c/0x85
[103563.166411] [<ffffffff810aefd2>] ? __lock_page+0x5d/0x63
[103563.166422] [<ffffffff8105b9bf>] ? wake_bit_function+0x0/0x2e
[103563.166435] [<ffffffff811067e1>] ? __generic_file_splice_read+0x26c/0x420
[103563.166450] [<ffffffff8124f695>] ? memcpy_toiovec+0x38/0x67
[103563.166463] [<ffffffff81306561>] ? down_read+0x9/0x19
[103563.166488] [<ffffffffa058f66a>] ? xfs_ilock+0x5c/0x97 [xfs]
[103563.166514] [<ffffffffa058ff1c>] ? xfs_iget+0x51f/0x58b [xfs]
[103563.166527] [<ffffffff812491ca>] ? release_sock+0x19/0xf3
[103563.166539] [<ffffffff813070ce>] ? common_interrupt+0xe/0x13
[103563.166552] [<ffffffff81104eca>] ? spd_release_page+0x0/0x14
[103563.166563] [<ffffffff811069d0>] ? generic_file_splice_read+0x3b/0x63
[103563.166592] [<ffffffffa05ae4cd>] ? xfs_file_splice_read+0xe4/0x12e [xfs]
[103563.166606] [<ffffffff81105019>] ? splice_direct_to_actor+0xba/0x17b
[103563.166626] [<ffffffffa061d851>] ? nfsd_direct_splice_actor+0x0/0x13 [nfsd]
[103563.166644] [<ffffffffa061d748>] ? nfsd_vfs_read+0x242/0x34b [nfsd]
[103563.166661] [<ffffffffa061de4d>] ? nfsd_read+0xa5/0xc4 [nfsd]
[103563.166679] [<ffffffffa0624583>] ? nfsd3_proc_read+0xd6/0x114 [nfsd]
[103563.166695] [<ffffffffa0619334>] ? nfsd_dispatch+0xdf/0x1b5 [nfsd]
[103563.166721] [<ffffffffa02a7596>] ? svc_process+0x40d/0x60b [sunrpc]
[103563.166737] [<ffffffffa0619851>] ? nfsd+0xe8/0x131 [nfsd]
[103563.166751] [<ffffffffa0619769>] ? nfsd+0x0/0x131 [nfsd]
[103563.166761] [<ffffffff8105b541>] ? kthread+0x75/0x7d
[103563.166772] [<ffffffff810097e4>] ? kernel_thread_helper+0x4/0x10
[103563.166784] [<ffffffff8105b4cc>] ? kthread+0x0/0x7d
[103563.166794] [<ffffffff810097e0>] ? kernel_thread_helper+0x0/0x10
[103563.166807] INFO: task nfsd:3495 blocked for more than 120 seconds.
[103563.166815] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[103563.166824] nfsd D 0000000101892965 0 3495 2 0x00000000
[103563.166834] ffff88042ae173d0 0000000000000046 ffff880420b0578c ffffffff00000000
[103563.166846] ffff88042e0d8da0 00000000000154c0 00000000000154c0 00000000000154c0
[103563.166857] ffff880420b05fd8 00000000000154c0 ffff88042ae173d0 ffff880420b05fd8
[103563.166868] Call Trace:
[103563.166875] [<ffffffffa026b133>] ? get_active_stripe+0xfe/0x55b [raid456]
[103563.166889] [<ffffffff8103f760>] ? default_wake_function+0x0/0xf
[103563.166902] [<ffffffffa026eafa>] ? make_request+0x1dd/0x403 [raid456]
[103563.166915] [<ffffffff8105b995>] ? autoremove_wake_function+0x0/0x2a
[103563.166931] [<ffffffffa02267ca>] ? md_make_request+0xd0/0x1d4 [md_mod]
[103563.166944] [<ffffffff8117583c>] ? generic_make_request+0x27a/0x2f0
[103563.166957] [<ffffffff813070ce>] ? common_interrupt+0xe/0x13
[103563.166969] [<ffffffff81175983>] ? submit_bio+0xd1/0xed
[103563.166979] [<ffffffff8118b006>] ? radix_tree_tag_set+0x64/0xc8
[103563.166991] [<ffffffff8110b97c>] ? __bio_add_page+0x156/0x1f1
[103563.167019] [<ffffffffa05aa640>] ? xfs_submit_ioend_bio+0x53/0x5f [xfs]
[103563.167048] [<ffffffffa05aa702>] ? xfs_submit_ioend+0xb6/0xd8 [xfs]
[103563.167077] [<ffffffffa05ab62a>] ? xfs_page_state_convert+0x4bd/0x50e [xfs]
[103563.167105] [<ffffffffa058ff1c>] ? xfs_iget+0x51f/0x58b [xfs]
[103563.167133] [<ffffffffa05ab843>] ? xfs_vm_writepage+0xef/0x12c [xfs]
[103563.167146] [<ffffffff810b548a>] ? __writepage+0xa/0x21
[103563.167157] [<ffffffff810b6504>] ? write_cache_pages+0x1d8/0x2ec
[103563.167167] [<ffffffff810b5480>] ? __writepage+0x0/0x21
[103563.167183] [<ffffffffa0621ae5>] ? nfsd_setuser+0x1a2/0x1e5 [nfsd]
[103563.167199] [<ffffffffa061bff4>] ? nfsd_setuser_and_check_port+0x76/0x92 [nfsd]
[103563.167214] [<ffffffff810af4d2>] ? __filemap_fdatawrite_range+0x48/0x4d
[103563.167227] [<ffffffff810af4fe>] ? filemap_write_and_wait_range+0x27/0x50
[103563.167240] [<ffffffff81106b66>] ? vfs_fsync_range+0x2f/0x79
[103563.167256] [<ffffffffa061dc99>] ? nfsd_commit+0x67/0x9a [nfsd]
[103563.167269] [<ffffffff810dfe2f>] ? virt_to_head_page+0x9/0x2a
[103563.167286] [<ffffffffa0623871>] ? nfsd3_proc_commit+0x86/0x92 [nfsd]
[103563.167302] [<ffffffffa0619334>] ? nfsd_dispatch+0xdf/0x1b5 [nfsd]
[103563.167322] [<ffffffffa02a7596>] ? svc_process+0x40d/0x60b [sunrpc]
[103563.167338] [<ffffffffa0619851>] ? nfsd+0xe8/0x131 [nfsd]
[103563.167352] [<ffffffffa0619769>] ? nfsd+0x0/0x131 [nfsd]
[103563.167364] [<ffffffff8105b541>] ? kthread+0x75/0x7d
[103563.167375] [<ffffffff810097e4>] ? kernel_thread_helper+0x4/0x10
[103563.167387] [<ffffffff8105b4cc>] ? kthread+0x0/0x7d
[103563.167398] [<ffffffff810097e0>] ? kernel_thread_helper+0x0/0x10
[103563.167409] INFO: task nfsd:3496 blocked for more than 120 seconds.
[103563.167417] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[103563.167427] nfsd D 0000000101892957 0 3496 2 0x00000000
[103563.167437] ffff88042ae12fb0 0000000000000046 ffff88042ae12fe8 0000000000000000
[103563.167449] ffff88042e06b680 00000000000154c0 00000000000154c0 00000000000154c0
[103563.167460] ffff88042041dfd8 00000000000154c0 ffff88042ae12fb0 ffff88042041dfd8
[103563.167471] Call Trace:
[103563.167477] [<ffffffffa026b133>] ? get_active_stripe+0xfe/0x55b [raid456]
[103563.167491] [<ffffffff8103f760>] ? default_wake_function+0x0/0xf
[103563.167505] [<ffffffffa026eafa>] ? make_request+0x1dd/0x403 [raid456]
[103563.167518] [<ffffffff8105b995>] ? autoremove_wake_function+0x0/0x2a
[103563.167533] [<ffffffffa02267ca>] ? md_make_request+0xd0/0x1d4 [md_mod]
[103563.167547] [<ffffffff8117583c>] ? generic_make_request+0x27a/0x2f0
[103563.167560] [<ffffffff810b09dd>] ? mempool_alloc+0x53/0x106
[103563.167572] [<ffffffff81175983>] ? submit_bio+0xd1/0xed
[103563.167582] [<ffffffff8110b97c>] ? __bio_add_page+0x156/0x1f1
[103563.167609] [<ffffffffa05aa640>] ? xfs_submit_ioend_bio+0x53/0x5f [xfs]
[103563.167638] [<ffffffffa05aa702>] ? xfs_submit_ioend+0xb6/0xd8 [xfs]
[103563.167667] [<ffffffffa05ab62a>] ? xfs_page_state_convert+0x4bd/0x50e [xfs]
[103563.167692] [<ffffffffa058ff1c>] ? xfs_iget+0x51f/0x58b [xfs]
[103563.167720] [<ffffffffa05ab843>] ? xfs_vm_writepage+0xef/0x12c [xfs]
[103563.167733] [<ffffffff810b548a>] ? __writepage+0xa/0x21
[103563.167744] [<ffffffff810b6504>] ? write_cache_pages+0x1d8/0x2ec
[103563.167756] [<ffffffff810b5480>] ? __writepage+0x0/0x21
[103563.167770] [<ffffffffa0621ae5>] ? nfsd_setuser+0x1a2/0x1e5 [nfsd]
[103563.167786] [<ffffffffa061bff4>] ? nfsd_setuser_and_check_port+0x76/0x92 [nfsd]
[103563.167801] [<ffffffff810af4d2>] ? __filemap_fdatawrite_range+0x48/0x4d
[103563.167814] [<ffffffff810af4fe>] ? filemap_write_and_wait_range+0x27/0x50
[103563.167828] [<ffffffff81106b66>] ? vfs_fsync_range+0x2f/0x79
[103563.167843] [<ffffffffa061dc99>] ? nfsd_commit+0x67/0x9a [nfsd]
[103563.167856] [<ffffffff810dfe2f>] ? virt_to_head_page+0x9/0x2a
[103563.167872] [<ffffffffa0623871>] ? nfsd3_proc_commit+0x86/0x92 [nfsd]
[103563.167888] [<ffffffffa0619334>] ? nfsd_dispatch+0xdf/0x1b5 [nfsd]
[103563.167914] [<ffffffffa02a7596>] ? svc_process+0x40d/0x60b [sunrpc]
[103563.167939] [<ffffffffa0619851>] ? nfsd+0xe8/0x131 [nfsd]
[103563.167953] [<ffffffffa0619769>] ? nfsd+0x0/0x131 [nfsd]
[103563.167964] [<ffffffff8105b541>] ? kthread+0x75/0x7d
[103563.167976] [<ffffffff810097e4>] ? kernel_thread_helper+0x4/0x10
[103563.167988] [<ffffffff8105b4cc>] ? kthread+0x0/0x7d
[103563.167999] [<ffffffff810097e0>] ? kernel_thread_helper+0x0/0x10
[103563.168011] INFO: task nfsd:3497 blocked for more than 120 seconds.
[103563.168019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[103563.168028] nfsd D 0000000101891d57 0 3497 2 0x00000000
[103563.168039] ffff88042ae14af0 0000000000000046 ffff880177966600 0000000000000000
[103563.168050] ffff88042e0da8e0 00000000000154c0 00000000000154c0 00000000000154c0
[103563.168062] ffff880420517fd8 00000000000154c0 ffff88042ae14af0 ffff880420517fd8
[103563.168073] Call Trace:
[103563.168080] [<ffffffffa026e8a9>] ? chunk_aligned_read+0x1be/0x232 [raid456]
[103563.168094] [<ffffffff8103f760>] ? default_wake_function+0x0/0xf
[103563.168107] [<ffffffffa022508c>] ? md_write_start+0x13f/0x144 [md_mod]
[103563.168120] [<ffffffffa026e995>] ? make_request+0x78/0x403 [raid456]
[103563.168136] [<ffffffffa02267ca>] ? md_make_request+0xd0/0x1d4 [md_mod]
[103563.168149] [<ffffffff8117583c>] ? generic_make_request+0x27a/0x2f0
[103563.168162] [<ffffffff8110bbca>] ? bvec_alloc_bs+0xa5/0xc0
[103563.168174] [<ffffffff81175983>] ? submit_bio+0xd1/0xed
[103563.168185] [<ffffffff8110bd13>] ? bio_alloc+0x10/0x1f
[103563.168196] [<ffffffff8110f562>] ? mpage_bio_submit+0x22/0x26
[103563.168208] [<ffffffff8110f9ff>] ? do_mpage_readpage+0x316/0x42f
[103563.168236] [<ffffffffa05aa4f7>] ? xfs_get_blocks+0x0/0x14 [xfs]
[103563.168264] [<ffffffffa05aa4f7>] ? xfs_get_blocks+0x0/0x14 [xfs]
[103563.168277] [<ffffffff810af286>] ? add_to_page_cache_locked+0x71/0xbb
[103563.168306] [<ffffffffa05aa4f7>] ? xfs_get_blocks+0x0/0x14 [xfs]
[103563.168318] [<ffffffff8110fc58>] ? mpage_readpages+0xce/0x112
[103563.168346] [<ffffffffa05aa4f7>] ? xfs_get_blocks+0x0/0x14 [xfs]
[103563.168358] [<ffffffff81305929>] ? schedule+0x4d4/0x530
[103563.168370] [<ffffffff810b6876>] ? __do_page_cache_readahead+0x117/0x1b3
[103563.168382] [<ffffffff810b692e>] ? ra_submit+0x1c/0x20
[103563.168393] [<ffffffff8110668b>] ? __generic_file_splice_read+0x116/0x420
[103563.168407] [<ffffffff81306561>] ? down_read+0x9/0x19
[103563.168432] [<ffffffffa058f66a>] ? xfs_ilock+0x5c/0x97 [xfs]
[103563.168458] [<ffffffffa058ff1c>] ? xfs_iget+0x51f/0x58b [xfs]
[103563.168470] [<ffffffff812491ca>] ? release_sock+0x19/0xf3
[103563.168481] [<ffffffff810b2dfc>] ? free_pcppages_bulk+0x13d/0x150
[103563.168494] [<ffffffff81104eca>] ? spd_release_page+0x0/0x14
[103563.168506] [<ffffffff811069d0>] ? generic_file_splice_read+0x3b/0x63
[103563.168534] [<ffffffffa05ae4cd>] ? xfs_file_splice_read+0xe4/0x12e [xfs]
[103563.168549] [<ffffffff81105019>] ? splice_direct_to_actor+0xba/0x17b
[103563.168565] [<ffffffffa061d851>] ? nfsd_direct_splice_actor+0x0/0x13 [nfsd]
[103563.168582] [<ffffffffa061d748>] ? nfsd_vfs_read+0x242/0x34b [nfsd]
[103563.168599] [<ffffffffa061de4d>] ? nfsd_read+0xa5/0xc4 [nfsd]
[103563.168616] [<ffffffffa0624583>] ? nfsd3_proc_read+0xd6/0x114 [nfsd]
[103563.168631] [<ffffffffa0619334>] ? nfsd_dispatch+0xdf/0x1b5 [nfsd]
[103563.168651] [<ffffffffa02a7596>] ? svc_process+0x40d/0x60b [sunrpc]
[103563.168667] [<ffffffffa0619851>] ? nfsd+0xe8/0x131 [nfsd]
[103563.168681] [<ffffffffa0619769>] ? nfsd+0x0/0x131 [nfsd]
[103563.168692] [<ffffffff8105b541>] ? kthread+0x75/0x7d
[103563.168703] [<ffffffff810097e4>] ? kernel_thread_helper+0x4/0x10
[103563.168716] [<ffffffff8105b4cc>] ? kthread+0x0/0x7d
[103563.168726] [<ffffffff810097e0>] ? kernel_thread_helper+0x0/0x10
[103563.168738] INFO: task nfsd:3498 blocked for more than 120 seconds.
[103563.168757] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[103563.168767] nfsd D 0000000101891ce5 0 3498 2 0x00000000
[103563.168778] ffff88042ae15f60 0000000000000046 ffff88042023b780 0000000000000000
[103563.168789] ffff88042e0d8da0 00000000000154c0 00000000000154c0 00000000000154c0
[103563.168801] ffff880420611fd8 00000000000154c0 ffff88042ae15f60 ffff880420611fd8
[103563.168812] Call Trace:
[103563.168818] [<ffffffffa026e8a9>] ? chunk_aligned_read+0x1be/0x232 [raid456]
[103563.168832] [<ffffffff8103f760>] ? default_wake_function+0x0/0xf
[103563.168846] [<ffffffffa026e995>] ? make_request+0x78/0x403 [raid456]
[103563.168862] [<ffffffffa02267ca>] ? md_make_request+0xd0/0x1d4 [md_mod]
[103563.168875] [<ffffffff8117583c>] ? generic_make_request+0x27a/0x2f0
[103563.168888] [<ffffffff8110bbca>] ? bvec_alloc_bs+0xa5/0xc0
[103563.168899] [<ffffffff81175983>] ? submit_bio+0xd1/0xed
[103563.168909] [<ffffffff8110bd13>] ? bio_alloc+0x10/0x1f
[103563.168919] [<ffffffff8110f562>] ? mpage_bio_submit+0x22/0x26
[103563.168931] [<ffffffff8110f9ff>] ? do_mpage_readpage+0x316/0x42f
[103563.168959] [<ffffffffa05aa4f7>] ? xfs_get_blocks+0x0/0x14 [xfs]
[103563.168972] [<ffffffff810b72dc>] ? ____pagevec_lru_add+0x12d/0x143
[103563.169000] [<ffffffffa05aa4f7>] ? xfs_get_blocks+0x0/0x14 [xfs]
[103563.169028] [<ffffffffa05aa4f7>] ? xfs_get_blocks+0x0/0x14 [xfs]
[103563.169040] [<ffffffff8110fc58>] ? mpage_readpages+0xce/0x112
[103563.169068] [<ffffffffa05aa4f7>] ? xfs_get_blocks+0x0/0x14 [xfs]
[103563.169081] [<ffffffff81305929>] ? schedule+0x4d4/0x530
[103563.169092] [<ffffffff810b6876>] ? __do_page_cache_readahead+0x117/0x1b3
[103563.169105] [<ffffffff810b692e>] ? ra_submit+0x1c/0x20
[103563.169116] [<ffffffff8110668b>] ? __generic_file_splice_read+0x116/0x420
[103563.169141] [<ffffffffa0572f5b>] ? xfs_bmap_search_multi_extents+0x85/0xdb [xfs]
[103563.169156] [<ffffffff8124f695>] ? memcpy_toiovec+0x38/0x67
[103563.169169] [<ffffffff81306561>] ? down_read+0x9/0x19
[103563.169194] [<ffffffffa058f66a>] ? xfs_ilock+0x5c/0x97 [xfs]
[103563.169220] [<ffffffffa058ff1c>] ? xfs_iget+0x51f/0x58b [xfs]
[103563.169232] [<ffffffff812491ca>] ? release_sock+0x19/0xf3
[103563.169257] [<ffffffffa058f4d0>] ? xfs_iunlock+0x5b/0xb7 [xfs]
[103563.169270] [<ffffffff81104eca>] ? spd_release_page+0x0/0x14
[103563.169272] [<ffffffff811069d0>] ? generic_file_splice_read+0x3b/0x63
[103563.169298] [<ffffffffa05ae4cd>] ? xfs_file_splice_read+0xe4/0x12e [xfs]
[103563.169313] [<ffffffff81105019>] ? splice_direct_to_actor+0xba/0x17b
[103563.169329] [<ffffffffa061d851>] ? nfsd_direct_splice_actor+0x0/0x13 [nfsd]
[103563.169347] [<ffffffffa061d748>] ? nfsd_vfs_read+0x242/0x34b [nfsd]
[103563.169363] [<ffffffffa061de4d>] ? nfsd_read+0xa5/0xc4 [nfsd]
[103563.169378] [<ffffffffa0624583>] ? nfsd3_proc_read+0xd6/0x114 [nfsd]
[103563.169394] [<ffffffffa0619334>] ? nfsd_dispatch+0xdf/0x1b5 [nfsd]
[103563.169414] [<ffffffffa02a7596>] ? svc_process+0x40d/0x60b [sunrpc]
[103563.169430] [<ffffffffa0619851>] ? nfsd+0xe8/0x131 [nfsd]
[103563.169444] [<ffffffffa0619769>] ? nfsd+0x0/0x131 [nfsd]
[103563.169455] [<ffffffff8105b541>] ? kthread+0x75/0x7d
[103563.169466] [<ffffffff810097e4>] ? kernel_thread_helper+0x4/0x10
[103563.169479] [<ffffffff8105b4cc>] ? kthread+0x0/0x7d
[103563.169489] [<ffffffff810097e0>] ? kernel_thread_helper+0x0/0x10
[103563.169501] INFO: task nfsd:3499 blocked for more than 120 seconds.
[103563.169510] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[103563.169519] nfsd D 0000000101892025 0 3499 2 0x00000000
[103563.169531] ffff88042ae15890 0000000000000046 ffff8804258dad80 0000000000000000
[103563.169544] ffff88042e0dc420 00000000000154c0 00000000000154c0 00000000000154c0
[103563.169555] ffff880420713fd8 00000000000154c0 ffff88042ae15890 ffff880420713fd8
[103563.169567] Call Trace:
[103563.169574] [<ffffffffa026e8a9>] ? chunk_aligned_read+0x1be/0x232 [raid456]
[103563.169588] [<ffffffff8103f760>] ? default_wake_function+0x0/0xf
[103563.169602] [<ffffffffa026e995>] ? make_request+0x78/0x403 [raid456]
[103563.169615] [<ffffffff8103a650>] ? find_busiest_group+0x296/0x848
[103563.169630] [<ffffffffa02267ca>] ? md_make_request+0xd0/0x1d4 [md_mod]
[103563.169657] [<ffffffffa058f4d0>] ? xfs_iunlock+0x5b/0xb7 [xfs]
[103563.169669] [<ffffffff8117583c>] ? generic_make_request+0x27a/0x2f0
[103563.169680] [<ffffffff81175983>] ? submit_bio+0xd1/0xed
[103563.169691] [<ffffffff8110f562>] ? mpage_bio_submit+0x22/0x26
[103563.169703] [<ffffffff8110f9ff>] ? do_mpage_readpage+0x316/0x42f
[103563.169717] [<ffffffff81036c0d>] ? update_curr+0xb1/0x159
[103563.169744] [<ffffffffa05aa4f7>] ? xfs_get_blocks+0x0/0x14 [xfs]
[103563.169772] [<ffffffffa05aa4f7>] ? xfs_get_blocks+0x0/0x14 [xfs]
[103563.169784] [<ffffffff810af286>] ? add_to_page_cache_locked+0x71/0xbb
[103563.169813] [<ffffffffa05aa4f7>] ? xfs_get_blocks+0x0/0x14 [xfs]
[103563.169825] [<ffffffff8110fc58>] ? mpage_readpages+0xce/0x112
[103563.169853] [<ffffffffa05aa4f7>] ? xfs_get_blocks+0x0/0x14 [xfs]
[103563.169865] [<ffffffff81305929>] ? schedule+0x4d4/0x530
[103563.169877] [<ffffffff810b6876>] ? __do_page_cache_readahead+0x117/0x1b3
[103563.169891] [<ffffffff810b692e>] ? ra_submit+0x1c/0x20
[103563.169902] [<ffffffff8110668b>] ? __generic_file_splice_read+0x116/0x420
[103563.169917] [<ffffffff81292523>] ? tcp_write_xmit+0x445/0x8fc
[103563.169929] [<ffffffff81306561>] ? down_read+0x9/0x19
[103563.169954] [<ffffffffa058f66a>] ? xfs_ilock+0x5c/0x97 [xfs]
[103563.169980] [<ffffffffa058ff1c>] ? xfs_iget+0x51f/0x58b [xfs]
[103563.169990] [<ffffffff812491ca>] ? release_sock+0x19/0xf3
[103563.170002] [<ffffffff810b2dfc>] ? free_pcppages_bulk+0x13d/0x150
[103563.170015] [<ffffffff81104eca>] ? spd_release_page+0x0/0x14
[103563.170027] [<ffffffff811069d0>] ? generic_file_splice_read+0x3b/0x63
[103563.170056] [<ffffffffa05ae4cd>] ? xfs_file_splice_read+0xe4/0x12e [xfs]
[103563.170070] [<ffffffff81105019>] ? splice_direct_to_actor+0xba/0x17b
[103563.170086] [<ffffffffa061d851>] ? nfsd_direct_splice_actor+0x0/0x13 [nfsd]
[103563.170103] [<ffffffffa061d748>] ? nfsd_vfs_read+0x242/0x34b [nfsd]
[103563.170121] [<ffffffffa061de4d>] ? nfsd_read+0xa5/0xc4 [nfsd]
[103563.170137] [<ffffffffa0624583>] ? nfsd3_proc_read+0xd6/0x114 [nfsd]
[103563.170153] [<ffffffffa0619334>] ? nfsd_dispatch+0xdf/0x1b5 [nfsd]
[103563.170172] [<ffffffffa02a7596>] ? svc_process+0x40d/0x60b [sunrpc]
[103563.170188] [<ffffffffa0619851>] ? nfsd+0xe8/0x131 [nfsd]
[103563.170201] [<ffffffffa0619769>] ? nfsd+0x0/0x131 [nfsd]
[103563.170212] [<ffffffff8105b541>] ? kthread+0x75/0x7d
[103563.170224] [<ffffffff810097e4>] ? kernel_thread_helper+0x4/0x10
[103563.170236] [<ffffffff8105b4cc>] ? kthread+0x0/0x7d
[103563.170246] [<ffffffff810097e0>] ? kernel_thread_helper+0x0/0x10
[103563.170258] INFO: task nfsd:3500 blocked for more than 120 seconds.
[103563.170267] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[103563.170277] nfsd D 000000010189190d 0 3500 2 0x00000000
[103563.170288] ffff88042aa506d0 0000000000000046 ffff880426984af0 0000000000000000
[103563.170300] ffff88042e0da8e0 00000000000154c0 00000000000154c0 00000000000154c0
[103563.170310] ffff880420035fd8 00000000000154c0 ffff88042aa506d0 ffff880420035fd8
[103563.170322] Call Trace:
[103563.170327] [<ffffffff8100e806>] ? read_tsc+0x5/0x16
[103563.170338] [<ffffffff810aefe5>] ? sync_page+0x0/0x45
[103563.170349] [<ffffffff813059f0>] ? io_schedule+0x6b/0xaa
[103563.170361] [<ffffffff810af026>] ? sync_page+0x41/0x45
[103563.170371] [<ffffffff81305e3b>] ? __wait_on_bit_lock+0x3c/0x85
[103563.170384] [<ffffffff810aefd2>] ? __lock_page+0x5d/0x63
[103563.170395] [<ffffffff8105b9bf>] ? wake_bit_function+0x0/0x2e
[103563.170407] [<ffffffff811067e1>] ? __generic_file_splice_read+0x26c/0x420
[103563.170433] [<ffffffffa0572f5b>] ? xfs_bmap_search_multi_extents+0x85/0xdb [xfs]
[103563.170447] [<ffffffff8124f695>] ? memcpy_toiovec+0x38/0x67
[103563.170459] [<ffffffff81306561>] ? down_read+0x9/0x19
[103563.170484] [<ffffffffa058f66a>] ? xfs_ilock+0x5c/0x97 [xfs]
[103563.170510] [<ffffffffa058ff1c>] ? xfs_iget+0x51f/0x58b [xfs]
[103563.170522] [<ffffffff812491ca>] ? release_sock+0x19/0xf3
[103563.170548] [<ffffffffa058f4d0>] ? xfs_iunlock+0x5b/0xb7 [xfs]
[103563.170560] [<ffffffff81104eca>] ? spd_release_page+0x0/0x14
[103563.170572] [<ffffffff811069d0>] ? generic_file_splice_read+0x3b/0x63
[103563.170601] [<ffffffffa05ae4cd>] ? xfs_file_splice_read+0xe4/0x12e [xfs]
[103563.170615] [<ffffffff81105019>] ? splice_direct_to_actor+0xba/0x17b
[103563.170631] [<ffffffffa061d851>] ? nfsd_direct_splice_actor+0x0/0x13 [nfsd]
[103563.170649] [<ffffffffa061d748>] ? nfsd_vfs_read+0x242/0x34b [nfsd]
[103563.170665] [<ffffffffa061de4d>] ? nfsd_read+0xa5/0xc4 [nfsd]
[103563.170682] [<ffffffffa0624583>] ? nfsd3_proc_read+0xd6/0x114 [nfsd]
[103563.170698] [<ffffffffa0619334>] ? nfsd_dispatch+0xdf/0x1b5 [nfsd]
[103563.170719] [<ffffffffa02a7596>] ? svc_process+0x40d/0x60b [sunrpc]
[103563.170734] [<ffffffffa0619851>] ? nfsd+0xe8/0x131 [nfsd]
[103563.170748] [<ffffffffa0619769>] ? nfsd+0x0/0x131 [nfsd]
[103563.170759] [<ffffffff8105b541>] ? kthread+0x75/0x7d
[103563.170770] [<ffffffff810097e4>] ? kernel_thread_helper+0x4/0x10
[103563.170782] [<ffffffff8105b4cc>] ? kthread+0x0/0x7d
[103563.170793] [<ffffffff810097e0>] ? kernel_thread_helper+0x0/0x10
== full error from (1) ==
[103350.849291] ata4.04: exception Emask 0x0 SAct 0x92f SErr 0x0 action 0x6
[103350.892975] ata4.04: irq_stat 0x00020002, device error via D2H FIS
[103350.934022] ata4.04: failed command: READ FPDMA QUEUED
[103350.968878] ata4.04: cmd 60/80:00:3f:97:d3/00:00:0b:00:00/40 tag 0 ncq 65536 in
[103350.968879] res 41/40:00:61:95:d3/00:01:0b:00:00/00 Emask 0x9 (media error)
[103351.067418] ata4.04: status: { DRDY ERR }
[103351.095674] ata4.04: error: { UNC }
[103351.120802] ata4.04: failed command: READ FPDMA QUEUED
[103351.155855] ata4.04: cmd 60/00:08:3f:95:d3/01:00:0b:00:00/40 tag 1 ncq 131072 in
[103351.155857] res 41/40:00:61:95:d3/00:01:0b:00:00/00 Emask 0x409 (media error) <F>
[103351.258408] ata4.04: status: { DRDY ERR }
[103351.286897] ata4.04: error: { UNC }
[103351.312300] ata4.04: failed command: READ FPDMA QUEUED
[103351.347597] ata4.04: cmd 60/80:10:3f:98:d3/00:00:0b:00:00/40 tag 2 ncq 65536 in
[103351.347598] res 60/80:10:3f:98:d3/00:00:0b:00:00/40 Emask 0x10 (ATA bus error)
[103351.448427] ata4.04: status: { DRDY DF }
[103351.476575] ata4.04: error: { ICRC }
[103351.502633] ata4.04: failed command: READ FPDMA QUEUED
[103351.538054] ata4.04: cmd 60/80:18:bf:97:d3/00:00:0b:00:00/40 tag 3 ncq 65536 in
[103351.538055] res 60/80:18:bf:97:d3/00:00:0b:00:00/40 Emask 0x10 (ATA bus error)
[103351.639136] ata4.04: status: { DRDY DF }
[103351.667451] ata4.04: error: { ICRC }
[103351.693624] ata4.04: failed command: READ FPDMA QUEUED
[103351.729201] ata4.04: cmd 60/00:28:3f:96:d3/01:00:0b:00:00/40 tag 5 ncq 131072 in
[103351.729202] res 5a/2a:04:04:00:00/00:00:00:50:5a/00 Emask 0x2 (HSM violation)
[103351.830689] ata4.04: status: { DRDY DRQ }
[103351.859707] ata4.04: failed command: READ FPDMA QUEUED
[103351.895529] ata4.04: cmd 60/80:40:bf:98:d3/00:00:0b:00:00/40 tag 8 ncq 65536 in
[103351.895530] res 60/80:40:bf:98:d3/00:00:0b:00:00/40 Emask 0x10 (ATA bus error)
[103351.997385] ata4.04: status: { DRDY DF }
[103352.026059] ata4.04: error: { ICRC }
[103352.052553] ata4.04: failed command: READ FPDMA QUEUED
[103352.088421] ata4.04: cmd 60/00:58:3f:99:d3/01:00:0b:00:00/40 tag 11 ncq 131072 in
[103352.088422] res 60/00:58:3f:99:d3/00:00:0b:00:00/40 Emask 0x1 (device error)
[103352.190450] ata4.04: status: { DRDY DF }
[103352.219323] ata4.04: hard resetting link
[103352.584379] ata4.04: SATA link up 3.0 Gbps (SStatus 123 SControl 320)
[103352.631003] ata4.04: configured for UDMA/100
[103352.661948] ata4: EH complete
[103354.630686] ata4.04: exception Emask 0x0 SAct 0x620 SErr 0x0 action 0x6
[103354.675663] ata4.04: irq_stat 0x00060002, device error via SDB FIS
[103354.718067] ata4.04: failed command: READ FPDMA QUEUED
[103354.754258] ata4.04: cmd 60/80:28:bf:98:d3/00:00:0b:00:00/40 tag 5 ncq 65536 in
[103354.754259] res 5a/2a:04:04:00:00/00:00:00:50:5a/00 Emask 0x2 (HSM violation)
[103354.856506] ata4.04: status: { DRDY DRQ }
[103354.886119] ata4.04: failed command: READ FPDMA QUEUED
[103354.922522] ata4.04: cmd 60/00:48:3f:95:d3/01:00:0b:00:00/40 tag 9 ncq 131072 in
[103354.922523] res 41/40:00:61:95:d3/00:01:0b:00:00/00 Emask 0x409 (media error) <F>
[103355.027674] ata4.04: status: { DRDY ERR }
[103355.057331] ata4.04: error: { UNC }
[103355.083824] ata4.04: failed command: READ FPDMA QUEUED
[103355.083830] ata4.04: cmd 60/80:50:3f:97:d3/00:00:0b:00:00/40 tag 10 ncq 65536 in
[103355.083833] res 5a/2a:04:04:00:00/00:00:00:a0:5a/00 Emask 0x2 (HSM violation)
[103355.083837] ata4.04: status: { DRDY DRQ }
[103355.083907] ata4.04: hard resetting link
[103355.432335] ata4.04: SATA link up 3.0 Gbps (SStatus 123 SControl 320)
[103355.479370] ata4.04: configured for UDMA/100
[103355.510657] ata4: EH complete
[103357.495976] ata4.04: exception Emask 0x0 SAct 0x2 SErr 0x0 action 0x0
[103357.540217] ata4.04: irq_stat 0x00060002, device error via SDB FIS
[103357.582910] ata4.04: failed command: READ FPDMA QUEUED
[103357.619344] ata4.04: cmd 60/00:08:3f:95:d3/01:00:0b:00:00/40 tag 1 ncq 131072 in
[103357.619345] res 41/40:00:61:95:d3/00:01:0b:00:00/00 Emask 0x409 (media error) <F>
[103357.724580] ata4.04: status: { DRDY ERR }
[103357.754332] ata4.04: error: { UNC }
[103357.783799] ata4.04: configured for UDMA/100
[103357.815172] ata4: EH complete
[103359.786533] ata4.04: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x0
[103359.830796] ata4.04: irq_stat 0x00060002, device error via SDB FIS
[103359.873539] ata4.04: failed command: READ FPDMA QUEUED
[103359.910035] ata4.04: cmd 60/00:00:3f:95:d3/01:00:0b:00:00/40 tag 0 ncq 131072 in
[103359.910037] res 41/40:00:61:95:d3/00:01:0b:00:00/00 Emask 0x409 (media error) <F>
[103359.910040] ata4.04: status: { DRDY ERR }
[103359.910041] ata4.04: error: { UNC }
[103360.018086] ata4.04: configured for UDMA/100
[103360.071645] ata4: EH complete
[103362.052078] ata4.04: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x0
[103362.096263] ata4.04: irq_stat 0x00060002, device error via SDB FIS
[103362.138914] ata4.04: failed command: READ FPDMA QUEUED
[103362.175334] ata4.04: cmd 60/00:00:3f:95:d3/01:00:0b:00:00/40 tag 0 ncq 131072 in
[103362.175336] res 41/40:00:61:95:d3/00:01:0b:00:00/00 Emask 0x409 (media error) <F>
[103362.280506] ata4.04: status: { DRDY ERR }
[103362.310179] ata4.04: error: { UNC }
[103362.339586] ata4.04: configured for UDMA/100
[103362.370840] ata4: EH complete
[103364.334349] ata4.04: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x0
[103364.378582] ata4.04: irq_stat 0x00060002, device error via SDB FIS
[103364.421199] ata4.04: failed command: READ FPDMA QUEUED
[103364.457555] ata4.04: cmd 60/00:00:3f:95:d3/01:00:0b:00:00/40 tag 0 ncq 131072 in
[103364.457556] res 41/40:00:61:95:d3/00:01:0b:00:00/00 Emask 0x409 (media error) <F>
[103364.562587] ata4.04: status: { DRDY ERR }
[103364.592185] ata4.04: error: { UNC }
[103364.621387] ata4.04: configured for UDMA/100
[103364.652557] sd 3:4:0:0: [sdt] Unhandled sense code
[103364.686712] sd 3:4:0:0: [sdt] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[103364.734972] sd 3:4:0:0: [sdt] Sense Key : Medium Error [current] [descriptor]
[103364.783292] Descriptor sense data with sense descriptors (in hex):
[103364.825802] 72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00
[103364.869717] 0b d3 95 61
[103364.894588] sd 3:4:0:0: [sdt] Add. Sense: Unrecovered read error - auto reallocate failed
[103364.949216] sd 3:4:0:0: [sdt] CDB: Read(10): 28 00 0b d3 95 3f 00 01 00 00
[103364.969589] end_request: I/O error, dev sdt, sector 198415713
[103364.969619] ata4: EH complete
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: md raid5 blocked in get_active_stripe after disk failure
2011-01-12 13:04 md raid5 blocked in get_active_stripe after disk failure David Flynn
@ 2011-01-12 14:06 ` David Flynn
2011-01-12 20:33 ` NeilBrown
2011-01-12 20:27 ` NeilBrown
1 sibling, 1 reply; 6+ messages in thread
From: David Flynn @ 2011-01-12 14:06 UTC (permalink / raw)
To: linux-raid
* David Flynn (davidf@rd.bbc.co.uk) wrote:
> I have a fileserver that mounts an md raid5 array with 20 active disks,
> holding an xfs filesystem which is then exported via nfs. (system is
> a twin-quadcore intel, 64bit)
I ought to mention, this is running 2.6.35.8
..david
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: md raid5 blocked in get_active_stripe after disk failure
2011-01-12 13:04 md raid5 blocked in get_active_stripe after disk failure David Flynn
2011-01-12 14:06 ` David Flynn
@ 2011-01-12 20:27 ` NeilBrown
1 sibling, 0 replies; 6+ messages in thread
From: NeilBrown @ 2011-01-12 20:27 UTC (permalink / raw)
To: David Flynn; +Cc: linux-raid
On Wed, 12 Jan 2011 13:04:20 +0000 David Flynn <davidf@rd.bbc.co.uk> wrote:
> I have a fileserver that mounts an md raid5 array with 20 active disks,
> holding an xfs filesystem which is then exported via nfs. (system is
> a twin-quadcore intel, 64bit)
>
> I've witnessed a failure which resulted in a reproducible lockup of the
> array after one of the array components suffered a read error.
>
> The order of events was roughly:
> 1) disk fails read:
> <snip mny errors>
> [103364.894588] sd 3:4:0:0: [sdt] Add. Sense: Unrecovered read error - auto reallocate failed
> [103364.949216] sd 3:4:0:0: [sdt] CDB: Read(10): 28 00 0b d3 95 3f 00 01 00 00
> [103364.969589] end_request: I/O error, dev sdt, sector 198415713
> [103364.969619] ata4: EH complete
> 2) nfs clients report hung tasks
> 3) file server reports hung tasks
> (re-reviewing this in the cold light of the morning, makes me wonder
> if the cause of the problem isn't in pid 2777, eitherway it is too early
> to thrink straight)
> [log is included at end of message]
>
> 4) an hour passes, i'm alerted to the issue
> 5) top reports 30-40% cpu utilization in iowait, the rest idle
> 6) remembering an old strip_cache issue, i note that:
> /sys/class/block/md1/md/stripe_cache_active = 0
> /sys/class/block/md1/md/stripe_cache_size = 256
> doubling stripe_cache_size has no effect.
> 7) mark sdt1 as faulty:
> [109593.732485] md/raid:md1: Disk failure on sdt1, disabling device.
> [109593.732487] <1>md/raid:md1: Operation continuing on 19 devices.
> 8) try to remove sdt1:
> [110152.939464] md: cannot remove active disk sdt1 from md1 ...
> 9) attempting to run perf top blocks.
> 10) use fdisk to partition another disk, (write out partition table is ok)
> but fdisk blocks while requesting a partition table scan. although
> using hdparm -z directly on the disk succeeds.
> 11) can't reboot, can't sync, can't unmount file systems, etc.,
> eventually powercycle the machine
>
> When the machine reboots, restarts the array (was dirty, so rebuilds),
> remounts the filesystems and nfs comes back up and the clients retry their
> requests, the disk error occurred again and md locked up in the same way.
>
> Any thoughts are greatly appreciated.
It was a READ error, so md/raid5 should try to find the correct data to
over-write with.
So either md is locking up while trying to do that, to the underlying block
device didn't return an error properly to md.
The stack traces show that pid 2777 is trying to handle a barrier request
from XFS and so is trying to flush all other request. That stops new
requests from being submitted which explains all the other pids.
But I don't know why raid5_quiesce is not completing. Either it is still
waiting on something from a device, or it has become confused somehow.
What kernel version are you running?
NeilBrown
>
> Kind regards,
>
> ..david
>
>
> == kernel log from (3) above ==
> [103561.064060] INFO: task md1_raid5:2371 blocked for more than 120 seconds.
> [103561.109844] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [103561.162479] md1_raid5 D 0000000101892965 0 2371 2 0x00000000
> [103561.209434] ffff880426984af0 0000000000000046 ffff88042ae12fb0 ffff880000000000
> [103561.259613] ffffffff81632020 00000000000154c0 00000000000154c0 00000000000154c0
> [103561.309815] ffff880425123fd8 00000000000154c0 ffff880426984af0 ffff880425123fd8
> [103561.359995] Call Trace:
> [103561.380375] [<ffffffffa026b133>] ? get_active_stripe+0xfe/0x55b [raid456]
> [103561.427278] [<ffffffff8103f760>] ? default_wake_function+0x0/0xf
> [103561.469536] [<ffffffffa026cf92>] ? raid5d+0x149/0x3f2 [raid456]
> [103561.511297] [<ffffffffa0228010>] ? md_thread+0xf2/0x110 [md_mod]
> [103561.553580] [<ffffffff8105b995>] ? autoremove_wake_function+0x0/0x2a
> [103561.597882] [<ffffffffa0227f1e>] ? md_thread+0x0/0x110 [md_mod]
> [103561.639553] [<ffffffff8105b541>] ? kthread+0x75/0x7d
> [103561.675472] [<ffffffff810097e4>] ? kernel_thread_helper+0x4/0x10
> [103561.717604] [<ffffffff8105b4cc>] ? kthread+0x0/0x7d
> [103561.752937] [<ffffffff810097e0>] ? kernel_thread_helper+0x0/0x10
> [103561.795009] INFO: task flush-9:1:2491 blocked for more than 120 seconds.
> [103561.840753] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [103561.893356] flush-9:1 D 0000000101892965 0 2491 2 0x00000000
> [103561.940355] ffff880426b206d0 0000000000000046 ffff88042486d70c ffffffff00000000
> [103561.990544] ffff88042e0da8e0 00000000000154c0 00000000000154c0 00000000000154c0
> [103562.040754] ffff88042486dfd8 00000000000154c0 ffff880426b206d0 ffff88042486dfd8
> [103562.090973] Call Trace:
> [103562.111360] [<ffffffffa026b133>] ? get_active_stripe+0xfe/0x55b [raid456]
> [103562.158352] [<ffffffff8103f760>] ? default_wake_function+0x0/0xf
> [103562.200605] [<ffffffffa026eafa>] ? make_request+0x1dd/0x403 [raid456]
> [103562.245486] [<ffffffff8105b995>] ? autoremove_wake_function+0x0/0x2a
> [103562.289870] [<ffffffffa02267ca>] ? md_make_request+0xd0/0x1d4 [md_mod]
> [103562.335319] [<ffffffff8117583c>] ? generic_make_request+0x27a/0x2f0
> [103562.379181] [<ffffffff810b09dd>] ? mempool_alloc+0x53/0x106
> [103562.418843] [<ffffffff81175983>] ? submit_bio+0xd1/0xed
> [103562.456376] [<ffffffff8118b006>] ? radix_tree_tag_set+0x64/0xc8
> [103562.498025] [<ffffffff8110b97c>] ? __bio_add_page+0x156/0x1f1
> [103562.538646] [<ffffffffa05aa640>] ? xfs_submit_ioend_bio+0x53/0x5f [xfs]
> [103562.584444] [<ffffffffa05aa702>] ? xfs_submit_ioend+0xb6/0xd8 [xfs]
> [103562.628140] [<ffffffffa05ab62a>] ? xfs_page_state_convert+0x4bd/0x50e [xfs]
> [103562.676040] [<ffffffffa05ab843>] ? xfs_vm_writepage+0xef/0x12c [xfs]
> [103562.720299] [<ffffffff810b548a>] ? __writepage+0xa/0x21
> [103562.757774] [<ffffffff810b6504>] ? write_cache_pages+0x1d8/0x2ec
> [103562.799928] [<ffffffff810b5480>] ? __writepage+0x0/0x21
> [103562.837380] [<ffffffff81102cd4>] ? writeback_single_inode+0xd1/0x2e5
> [103562.881556] [<ffffffff8110323e>] ? writeback_sb_inodes+0x136/0x204
> [103562.924674] [<ffffffff811038e3>] ? writeback_inodes_wb+0x127/0x139
> [103562.967663] [<ffffffff81103a83>] ? wb_writeback+0x18e/0x208
> [103563.006933] [<ffffffff81103c30>] ? wb_do_writeback+0x133/0x149
> [103563.047721] [<ffffffff81103c80>] ? bdi_writeback_task+0x3a/0x112
> [103563.089528] [<ffffffff8105b8bb>] ? bit_waitqueue+0x14/0xa1
> [103563.128129] [<ffffffff810c1f4f>] ? bdi_start_fn+0x0/0xc3
> [103563.165632] [<ffffffff810c1fb2>] ? bdi_start_fn+0x63/0xc3
> [103563.165644] [<ffffffff810c1f4f>] ? bdi_start_fn+0x0/0xc3
> [103563.165656] [<ffffffff8105b541>] ? kthread+0x75/0x7d
> [103563.165667] [<ffffffff810097e4>] ? kernel_thread_helper+0x4/0x10
> [103563.165679] [<ffffffff8105b4cc>] ? kthread+0x0/0x7d
> [103563.165690] [<ffffffff810097e0>] ? kernel_thread_helper+0x0/0x10
> [103563.165708] INFO: task xfssyncd/md1:2777 blocked for more than 120 seconds.
> [103563.165715] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [103563.165724] xfssyncd/md1 D ffff88042b1c3680 0 2777 2 0x00000000
> [103563.165735] ffff88042b1c3680 0000000000000046 ffff88042a4ed000 ffffffffa002b0e0
> [103563.165747] ffff88042e121b40 00000000000154c0 00000000000154c0 00000000000154c0
> [103563.165758] ffff880424e61fd8 00000000000154c0 ffff88042b1c3680 ffff880424e61fd8
> [103563.165770] Call Trace:
> [103563.165797] [<ffffffffa002b0e0>] ? scsi_setup_blk_pc_cmnd+0x39/0xff [scsi_mod]
> [103563.165823] [<ffffffffa01a426c>] ? __ata_scsi_queuecmd+0x17f/0x1d6 [libata]
> [103563.165838] [<ffffffff8100e7d9>] ? sched_clock+0x5/0x8
> [103563.165850] [<ffffffffa0269520>] ? raid5_quiesce+0xe3/0x19b [raid456]
> [103563.165864] [<ffffffff8103f760>] ? default_wake_function+0x0/0xf
> [103563.165877] [<ffffffffa026e94e>] ? make_request+0x31/0x403 [raid456]
> [103563.165893] [<ffffffffa02267ca>] ? md_make_request+0xd0/0x1d4 [md_mod]
> [103563.165906] [<ffffffff8117583c>] ? generic_make_request+0x27a/0x2f0
> [103563.165919] [<ffffffff810b09dd>] ? mempool_alloc+0x53/0x106
> [103563.165931] [<ffffffff81175983>] ? submit_bio+0xd1/0xed
> [103563.165959] [<ffffffffa05ac4d2>] ? _xfs_buf_ioapply+0x212/0x24f [xfs]
> [103563.165987] [<ffffffffa0597ace>] ? xlog_bdstrat+0x3a/0x3e [xfs]
> [103563.166015] [<ffffffffa05ad430>] ? xfs_buf_iorequest+0x75/0x8e [xfs]
> [103563.166043] [<ffffffffa0597ace>] ? xlog_bdstrat+0x3a/0x3e [xfs]
> [103563.166070] [<ffffffffa0597ce1>] ? xlog_sync+0x20f/0x326 [xfs]
> [103563.166098] [<ffffffffa05986d4>] ? _xfs_log_force+0xd9/0x196 [xfs]
> [103563.166111] [<ffffffff81050771>] ? process_timeout+0x0/0xa
> [103563.166138] [<ffffffffa0598915>] ? xfs_log_force+0xb/0x2a [xfs]
> [103563.166167] [<ffffffffa05b537f>] ? xfs_sync_worker+0x14/0x60 [xfs]
> [103563.166195] [<ffffffffa05b4c2e>] ? xfssyncd+0x155/0x1d1 [xfs]
> [103563.166223] [<ffffffffa05b4ad9>] ? xfssyncd+0x0/0x1d1 [xfs]
> [103563.166235] [<ffffffff8105b541>] ? kthread+0x75/0x7d
> [103563.166246] [<ffffffff810097e4>] ? kernel_thread_helper+0x4/0x10
> [103563.166258] [<ffffffff8105b4cc>] ? kthread+0x0/0x7d
> [103563.166269] [<ffffffff810097e0>] ? kernel_thread_helper+0x0/0x10
> [103563.166284] INFO: task nfsd:3494 blocked for more than 120 seconds.
> [103563.166292] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [103563.166302] nfsd D 000000010189190f 0 3494 2 0x00000000
> [103563.166314] ffff88042ae14420 0000000000000046 ffff880200001000 0000000000000000
> [103563.166325] ffff88042e0d8da0 00000000000154c0 00000000000154c0 00000000000154c0
> [103563.166337] ffff880420a03fd8 00000000000154c0 ffff88042ae14420 ffff880420a03fd8
> [103563.166348] Call Trace:
> [103563.166354] [<ffffffff8100e806>] ? read_tsc+0x5/0x16
> [103563.166365] [<ffffffff810aefe5>] ? sync_page+0x0/0x45
> [103563.166378] [<ffffffff813059f0>] ? io_schedule+0x6b/0xaa
> [103563.166390] [<ffffffff810af026>] ? sync_page+0x41/0x45
> [103563.166399] [<ffffffff81305e3b>] ? __wait_on_bit_lock+0x3c/0x85
> [103563.166411] [<ffffffff810aefd2>] ? __lock_page+0x5d/0x63
> [103563.166422] [<ffffffff8105b9bf>] ? wake_bit_function+0x0/0x2e
> [103563.166435] [<ffffffff811067e1>] ? __generic_file_splice_read+0x26c/0x420
> [103563.166450] [<ffffffff8124f695>] ? memcpy_toiovec+0x38/0x67
> [103563.166463] [<ffffffff81306561>] ? down_read+0x9/0x19
> [103563.166488] [<ffffffffa058f66a>] ? xfs_ilock+0x5c/0x97 [xfs]
> [103563.166514] [<ffffffffa058ff1c>] ? xfs_iget+0x51f/0x58b [xfs]
> [103563.166527] [<ffffffff812491ca>] ? release_sock+0x19/0xf3
> [103563.166539] [<ffffffff813070ce>] ? common_interrupt+0xe/0x13
> [103563.166552] [<ffffffff81104eca>] ? spd_release_page+0x0/0x14
> [103563.166563] [<ffffffff811069d0>] ? generic_file_splice_read+0x3b/0x63
> [103563.166592] [<ffffffffa05ae4cd>] ? xfs_file_splice_read+0xe4/0x12e [xfs]
> [103563.166606] [<ffffffff81105019>] ? splice_direct_to_actor+0xba/0x17b
> [103563.166626] [<ffffffffa061d851>] ? nfsd_direct_splice_actor+0x0/0x13 [nfsd]
> [103563.166644] [<ffffffffa061d748>] ? nfsd_vfs_read+0x242/0x34b [nfsd]
> [103563.166661] [<ffffffffa061de4d>] ? nfsd_read+0xa5/0xc4 [nfsd]
> [103563.166679] [<ffffffffa0624583>] ? nfsd3_proc_read+0xd6/0x114 [nfsd]
> [103563.166695] [<ffffffffa0619334>] ? nfsd_dispatch+0xdf/0x1b5 [nfsd]
> [103563.166721] [<ffffffffa02a7596>] ? svc_process+0x40d/0x60b [sunrpc]
> [103563.166737] [<ffffffffa0619851>] ? nfsd+0xe8/0x131 [nfsd]
> [103563.166751] [<ffffffffa0619769>] ? nfsd+0x0/0x131 [nfsd]
> [103563.166761] [<ffffffff8105b541>] ? kthread+0x75/0x7d
> [103563.166772] [<ffffffff810097e4>] ? kernel_thread_helper+0x4/0x10
> [103563.166784] [<ffffffff8105b4cc>] ? kthread+0x0/0x7d
> [103563.166794] [<ffffffff810097e0>] ? kernel_thread_helper+0x0/0x10
> [103563.166807] INFO: task nfsd:3495 blocked for more than 120 seconds.
> [103563.166815] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [103563.166824] nfsd D 0000000101892965 0 3495 2 0x00000000
> [103563.166834] ffff88042ae173d0 0000000000000046 ffff880420b0578c ffffffff00000000
> [103563.166846] ffff88042e0d8da0 00000000000154c0 00000000000154c0 00000000000154c0
> [103563.166857] ffff880420b05fd8 00000000000154c0 ffff88042ae173d0 ffff880420b05fd8
> [103563.166868] Call Trace:
> [103563.166875] [<ffffffffa026b133>] ? get_active_stripe+0xfe/0x55b [raid456]
> [103563.166889] [<ffffffff8103f760>] ? default_wake_function+0x0/0xf
> [103563.166902] [<ffffffffa026eafa>] ? make_request+0x1dd/0x403 [raid456]
> [103563.166915] [<ffffffff8105b995>] ? autoremove_wake_function+0x0/0x2a
> [103563.166931] [<ffffffffa02267ca>] ? md_make_request+0xd0/0x1d4 [md_mod]
> [103563.166944] [<ffffffff8117583c>] ? generic_make_request+0x27a/0x2f0
> [103563.166957] [<ffffffff813070ce>] ? common_interrupt+0xe/0x13
> [103563.166969] [<ffffffff81175983>] ? submit_bio+0xd1/0xed
> [103563.166979] [<ffffffff8118b006>] ? radix_tree_tag_set+0x64/0xc8
> [103563.166991] [<ffffffff8110b97c>] ? __bio_add_page+0x156/0x1f1
> [103563.167019] [<ffffffffa05aa640>] ? xfs_submit_ioend_bio+0x53/0x5f [xfs]
> [103563.167048] [<ffffffffa05aa702>] ? xfs_submit_ioend+0xb6/0xd8 [xfs]
> [103563.167077] [<ffffffffa05ab62a>] ? xfs_page_state_convert+0x4bd/0x50e [xfs]
> [103563.167105] [<ffffffffa058ff1c>] ? xfs_iget+0x51f/0x58b [xfs]
> [103563.167133] [<ffffffffa05ab843>] ? xfs_vm_writepage+0xef/0x12c [xfs]
> [103563.167146] [<ffffffff810b548a>] ? __writepage+0xa/0x21
> [103563.167157] [<ffffffff810b6504>] ? write_cache_pages+0x1d8/0x2ec
> [103563.167167] [<ffffffff810b5480>] ? __writepage+0x0/0x21
> [103563.167183] [<ffffffffa0621ae5>] ? nfsd_setuser+0x1a2/0x1e5 [nfsd]
> [103563.167199] [<ffffffffa061bff4>] ? nfsd_setuser_and_check_port+0x76/0x92 [nfsd]
> [103563.167214] [<ffffffff810af4d2>] ? __filemap_fdatawrite_range+0x48/0x4d
> [103563.167227] [<ffffffff810af4fe>] ? filemap_write_and_wait_range+0x27/0x50
> [103563.167240] [<ffffffff81106b66>] ? vfs_fsync_range+0x2f/0x79
> [103563.167256] [<ffffffffa061dc99>] ? nfsd_commit+0x67/0x9a [nfsd]
> [103563.167269] [<ffffffff810dfe2f>] ? virt_to_head_page+0x9/0x2a
> [103563.167286] [<ffffffffa0623871>] ? nfsd3_proc_commit+0x86/0x92 [nfsd]
> [103563.167302] [<ffffffffa0619334>] ? nfsd_dispatch+0xdf/0x1b5 [nfsd]
> [103563.167322] [<ffffffffa02a7596>] ? svc_process+0x40d/0x60b [sunrpc]
> [103563.167338] [<ffffffffa0619851>] ? nfsd+0xe8/0x131 [nfsd]
> [103563.167352] [<ffffffffa0619769>] ? nfsd+0x0/0x131 [nfsd]
> [103563.167364] [<ffffffff8105b541>] ? kthread+0x75/0x7d
> [103563.167375] [<ffffffff810097e4>] ? kernel_thread_helper+0x4/0x10
> [103563.167387] [<ffffffff8105b4cc>] ? kthread+0x0/0x7d
> [103563.167398] [<ffffffff810097e0>] ? kernel_thread_helper+0x0/0x10
> [103563.167409] INFO: task nfsd:3496 blocked for more than 120 seconds.
> [103563.167417] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [103563.167427] nfsd D 0000000101892957 0 3496 2 0x00000000
> [103563.167437] ffff88042ae12fb0 0000000000000046 ffff88042ae12fe8 0000000000000000
> [103563.167449] ffff88042e06b680 00000000000154c0 00000000000154c0 00000000000154c0
> [103563.167460] ffff88042041dfd8 00000000000154c0 ffff88042ae12fb0 ffff88042041dfd8
> [103563.167471] Call Trace:
> [103563.167477] [<ffffffffa026b133>] ? get_active_stripe+0xfe/0x55b [raid456]
> [103563.167491] [<ffffffff8103f760>] ? default_wake_function+0x0/0xf
> [103563.167505] [<ffffffffa026eafa>] ? make_request+0x1dd/0x403 [raid456]
> [103563.167518] [<ffffffff8105b995>] ? autoremove_wake_function+0x0/0x2a
> [103563.167533] [<ffffffffa02267ca>] ? md_make_request+0xd0/0x1d4 [md_mod]
> [103563.167547] [<ffffffff8117583c>] ? generic_make_request+0x27a/0x2f0
> [103563.167560] [<ffffffff810b09dd>] ? mempool_alloc+0x53/0x106
> [103563.167572] [<ffffffff81175983>] ? submit_bio+0xd1/0xed
> [103563.167582] [<ffffffff8110b97c>] ? __bio_add_page+0x156/0x1f1
> [103563.167609] [<ffffffffa05aa640>] ? xfs_submit_ioend_bio+0x53/0x5f [xfs]
> [103563.167638] [<ffffffffa05aa702>] ? xfs_submit_ioend+0xb6/0xd8 [xfs]
> [103563.167667] [<ffffffffa05ab62a>] ? xfs_page_state_convert+0x4bd/0x50e [xfs]
> [103563.167692] [<ffffffffa058ff1c>] ? xfs_iget+0x51f/0x58b [xfs]
> [103563.167720] [<ffffffffa05ab843>] ? xfs_vm_writepage+0xef/0x12c [xfs]
> [103563.167733] [<ffffffff810b548a>] ? __writepage+0xa/0x21
> [103563.167744] [<ffffffff810b6504>] ? write_cache_pages+0x1d8/0x2ec
> [103563.167756] [<ffffffff810b5480>] ? __writepage+0x0/0x21
> [103563.167770] [<ffffffffa0621ae5>] ? nfsd_setuser+0x1a2/0x1e5 [nfsd]
> [103563.167786] [<ffffffffa061bff4>] ? nfsd_setuser_and_check_port+0x76/0x92 [nfsd]
> [103563.167801] [<ffffffff810af4d2>] ? __filemap_fdatawrite_range+0x48/0x4d
> [103563.167814] [<ffffffff810af4fe>] ? filemap_write_and_wait_range+0x27/0x50
> [103563.167828] [<ffffffff81106b66>] ? vfs_fsync_range+0x2f/0x79
> [103563.167843] [<ffffffffa061dc99>] ? nfsd_commit+0x67/0x9a [nfsd]
> [103563.167856] [<ffffffff810dfe2f>] ? virt_to_head_page+0x9/0x2a
> [103563.167872] [<ffffffffa0623871>] ? nfsd3_proc_commit+0x86/0x92 [nfsd]
> [103563.167888] [<ffffffffa0619334>] ? nfsd_dispatch+0xdf/0x1b5 [nfsd]
> [103563.167914] [<ffffffffa02a7596>] ? svc_process+0x40d/0x60b [sunrpc]
> [103563.167939] [<ffffffffa0619851>] ? nfsd+0xe8/0x131 [nfsd]
> [103563.167953] [<ffffffffa0619769>] ? nfsd+0x0/0x131 [nfsd]
> [103563.167964] [<ffffffff8105b541>] ? kthread+0x75/0x7d
> [103563.167976] [<ffffffff810097e4>] ? kernel_thread_helper+0x4/0x10
> [103563.167988] [<ffffffff8105b4cc>] ? kthread+0x0/0x7d
> [103563.167999] [<ffffffff810097e0>] ? kernel_thread_helper+0x0/0x10
> [103563.168011] INFO: task nfsd:3497 blocked for more than 120 seconds.
> [103563.168019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [103563.168028] nfsd D 0000000101891d57 0 3497 2 0x00000000
> [103563.168039] ffff88042ae14af0 0000000000000046 ffff880177966600 0000000000000000
> [103563.168050] ffff88042e0da8e0 00000000000154c0 00000000000154c0 00000000000154c0
> [103563.168062] ffff880420517fd8 00000000000154c0 ffff88042ae14af0 ffff880420517fd8
> [103563.168073] Call Trace:
> [103563.168080] [<ffffffffa026e8a9>] ? chunk_aligned_read+0x1be/0x232 [raid456]
> [103563.168094] [<ffffffff8103f760>] ? default_wake_function+0x0/0xf
> [103563.168107] [<ffffffffa022508c>] ? md_write_start+0x13f/0x144 [md_mod]
> [103563.168120] [<ffffffffa026e995>] ? make_request+0x78/0x403 [raid456]
> [103563.168136] [<ffffffffa02267ca>] ? md_make_request+0xd0/0x1d4 [md_mod]
> [103563.168149] [<ffffffff8117583c>] ? generic_make_request+0x27a/0x2f0
> [103563.168162] [<ffffffff8110bbca>] ? bvec_alloc_bs+0xa5/0xc0
> [103563.168174] [<ffffffff81175983>] ? submit_bio+0xd1/0xed
> [103563.168185] [<ffffffff8110bd13>] ? bio_alloc+0x10/0x1f
> [103563.168196] [<ffffffff8110f562>] ? mpage_bio_submit+0x22/0x26
> [103563.168208] [<ffffffff8110f9ff>] ? do_mpage_readpage+0x316/0x42f
> [103563.168236] [<ffffffffa05aa4f7>] ? xfs_get_blocks+0x0/0x14 [xfs]
> [103563.168264] [<ffffffffa05aa4f7>] ? xfs_get_blocks+0x0/0x14 [xfs]
> [103563.168277] [<ffffffff810af286>] ? add_to_page_cache_locked+0x71/0xbb
> [103563.168306] [<ffffffffa05aa4f7>] ? xfs_get_blocks+0x0/0x14 [xfs]
> [103563.168318] [<ffffffff8110fc58>] ? mpage_readpages+0xce/0x112
> [103563.168346] [<ffffffffa05aa4f7>] ? xfs_get_blocks+0x0/0x14 [xfs]
> [103563.168358] [<ffffffff81305929>] ? schedule+0x4d4/0x530
> [103563.168370] [<ffffffff810b6876>] ? __do_page_cache_readahead+0x117/0x1b3
> [103563.168382] [<ffffffff810b692e>] ? ra_submit+0x1c/0x20
> [103563.168393] [<ffffffff8110668b>] ? __generic_file_splice_read+0x116/0x420
> [103563.168407] [<ffffffff81306561>] ? down_read+0x9/0x19
> [103563.168432] [<ffffffffa058f66a>] ? xfs_ilock+0x5c/0x97 [xfs]
> [103563.168458] [<ffffffffa058ff1c>] ? xfs_iget+0x51f/0x58b [xfs]
> [103563.168470] [<ffffffff812491ca>] ? release_sock+0x19/0xf3
> [103563.168481] [<ffffffff810b2dfc>] ? free_pcppages_bulk+0x13d/0x150
> [103563.168494] [<ffffffff81104eca>] ? spd_release_page+0x0/0x14
> [103563.168506] [<ffffffff811069d0>] ? generic_file_splice_read+0x3b/0x63
> [103563.168534] [<ffffffffa05ae4cd>] ? xfs_file_splice_read+0xe4/0x12e [xfs]
> [103563.168549] [<ffffffff81105019>] ? splice_direct_to_actor+0xba/0x17b
> [103563.168565] [<ffffffffa061d851>] ? nfsd_direct_splice_actor+0x0/0x13 [nfsd]
> [103563.168582] [<ffffffffa061d748>] ? nfsd_vfs_read+0x242/0x34b [nfsd]
> [103563.168599] [<ffffffffa061de4d>] ? nfsd_read+0xa5/0xc4 [nfsd]
> [103563.168616] [<ffffffffa0624583>] ? nfsd3_proc_read+0xd6/0x114 [nfsd]
> [103563.168631] [<ffffffffa0619334>] ? nfsd_dispatch+0xdf/0x1b5 [nfsd]
> [103563.168651] [<ffffffffa02a7596>] ? svc_process+0x40d/0x60b [sunrpc]
> [103563.168667] [<ffffffffa0619851>] ? nfsd+0xe8/0x131 [nfsd]
> [103563.168681] [<ffffffffa0619769>] ? nfsd+0x0/0x131 [nfsd]
> [103563.168692] [<ffffffff8105b541>] ? kthread+0x75/0x7d
> [103563.168703] [<ffffffff810097e4>] ? kernel_thread_helper+0x4/0x10
> [103563.168716] [<ffffffff8105b4cc>] ? kthread+0x0/0x7d
> [103563.168726] [<ffffffff810097e0>] ? kernel_thread_helper+0x0/0x10
> [103563.168738] INFO: task nfsd:3498 blocked for more than 120 seconds.
> [103563.168757] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [103563.168767] nfsd D 0000000101891ce5 0 3498 2 0x00000000
> [103563.168778] ffff88042ae15f60 0000000000000046 ffff88042023b780 0000000000000000
> [103563.168789] ffff88042e0d8da0 00000000000154c0 00000000000154c0 00000000000154c0
> [103563.168801] ffff880420611fd8 00000000000154c0 ffff88042ae15f60 ffff880420611fd8
> [103563.168812] Call Trace:
> [103563.168818] [<ffffffffa026e8a9>] ? chunk_aligned_read+0x1be/0x232 [raid456]
> [103563.168832] [<ffffffff8103f760>] ? default_wake_function+0x0/0xf
> [103563.168846] [<ffffffffa026e995>] ? make_request+0x78/0x403 [raid456]
> [103563.168862] [<ffffffffa02267ca>] ? md_make_request+0xd0/0x1d4 [md_mod]
> [103563.168875] [<ffffffff8117583c>] ? generic_make_request+0x27a/0x2f0
> [103563.168888] [<ffffffff8110bbca>] ? bvec_alloc_bs+0xa5/0xc0
> [103563.168899] [<ffffffff81175983>] ? submit_bio+0xd1/0xed
> [103563.168909] [<ffffffff8110bd13>] ? bio_alloc+0x10/0x1f
> [103563.168919] [<ffffffff8110f562>] ? mpage_bio_submit+0x22/0x26
> [103563.168931] [<ffffffff8110f9ff>] ? do_mpage_readpage+0x316/0x42f
> [103563.168959] [<ffffffffa05aa4f7>] ? xfs_get_blocks+0x0/0x14 [xfs]
> [103563.168972] [<ffffffff810b72dc>] ? ____pagevec_lru_add+0x12d/0x143
> [103563.169000] [<ffffffffa05aa4f7>] ? xfs_get_blocks+0x0/0x14 [xfs]
> [103563.169028] [<ffffffffa05aa4f7>] ? xfs_get_blocks+0x0/0x14 [xfs]
> [103563.169040] [<ffffffff8110fc58>] ? mpage_readpages+0xce/0x112
> [103563.169068] [<ffffffffa05aa4f7>] ? xfs_get_blocks+0x0/0x14 [xfs]
> [103563.169081] [<ffffffff81305929>] ? schedule+0x4d4/0x530
> [103563.169092] [<ffffffff810b6876>] ? __do_page_cache_readahead+0x117/0x1b3
> [103563.169105] [<ffffffff810b692e>] ? ra_submit+0x1c/0x20
> [103563.169116] [<ffffffff8110668b>] ? __generic_file_splice_read+0x116/0x420
> [103563.169141] [<ffffffffa0572f5b>] ? xfs_bmap_search_multi_extents+0x85/0xdb [xfs]
> [103563.169156] [<ffffffff8124f695>] ? memcpy_toiovec+0x38/0x67
> [103563.169169] [<ffffffff81306561>] ? down_read+0x9/0x19
> [103563.169194] [<ffffffffa058f66a>] ? xfs_ilock+0x5c/0x97 [xfs]
> [103563.169220] [<ffffffffa058ff1c>] ? xfs_iget+0x51f/0x58b [xfs]
> [103563.169232] [<ffffffff812491ca>] ? release_sock+0x19/0xf3
> [103563.169257] [<ffffffffa058f4d0>] ? xfs_iunlock+0x5b/0xb7 [xfs]
> [103563.169270] [<ffffffff81104eca>] ? spd_release_page+0x0/0x14
> [103563.169272] [<ffffffff811069d0>] ? generic_file_splice_read+0x3b/0x63
> [103563.169298] [<ffffffffa05ae4cd>] ? xfs_file_splice_read+0xe4/0x12e [xfs]
> [103563.169313] [<ffffffff81105019>] ? splice_direct_to_actor+0xba/0x17b
> [103563.169329] [<ffffffffa061d851>] ? nfsd_direct_splice_actor+0x0/0x13 [nfsd]
> [103563.169347] [<ffffffffa061d748>] ? nfsd_vfs_read+0x242/0x34b [nfsd]
> [103563.169363] [<ffffffffa061de4d>] ? nfsd_read+0xa5/0xc4 [nfsd]
> [103563.169378] [<ffffffffa0624583>] ? nfsd3_proc_read+0xd6/0x114 [nfsd]
> [103563.169394] [<ffffffffa0619334>] ? nfsd_dispatch+0xdf/0x1b5 [nfsd]
> [103563.169414] [<ffffffffa02a7596>] ? svc_process+0x40d/0x60b [sunrpc]
> [103563.169430] [<ffffffffa0619851>] ? nfsd+0xe8/0x131 [nfsd]
> [103563.169444] [<ffffffffa0619769>] ? nfsd+0x0/0x131 [nfsd]
> [103563.169455] [<ffffffff8105b541>] ? kthread+0x75/0x7d
> [103563.169466] [<ffffffff810097e4>] ? kernel_thread_helper+0x4/0x10
> [103563.169479] [<ffffffff8105b4cc>] ? kthread+0x0/0x7d
> [103563.169489] [<ffffffff810097e0>] ? kernel_thread_helper+0x0/0x10
> [103563.169501] INFO: task nfsd:3499 blocked for more than 120 seconds.
> [103563.169510] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [103563.169519] nfsd D 0000000101892025 0 3499 2 0x00000000
> [103563.169531] ffff88042ae15890 0000000000000046 ffff8804258dad80 0000000000000000
> [103563.169544] ffff88042e0dc420 00000000000154c0 00000000000154c0 00000000000154c0
> [103563.169555] ffff880420713fd8 00000000000154c0 ffff88042ae15890 ffff880420713fd8
> [103563.169567] Call Trace:
> [103563.169574] [<ffffffffa026e8a9>] ? chunk_aligned_read+0x1be/0x232 [raid456]
> [103563.169588] [<ffffffff8103f760>] ? default_wake_function+0x0/0xf
> [103563.169602] [<ffffffffa026e995>] ? make_request+0x78/0x403 [raid456]
> [103563.169615] [<ffffffff8103a650>] ? find_busiest_group+0x296/0x848
> [103563.169630] [<ffffffffa02267ca>] ? md_make_request+0xd0/0x1d4 [md_mod]
> [103563.169657] [<ffffffffa058f4d0>] ? xfs_iunlock+0x5b/0xb7 [xfs]
> [103563.169669] [<ffffffff8117583c>] ? generic_make_request+0x27a/0x2f0
> [103563.169680] [<ffffffff81175983>] ? submit_bio+0xd1/0xed
> [103563.169691] [<ffffffff8110f562>] ? mpage_bio_submit+0x22/0x26
> [103563.169703] [<ffffffff8110f9ff>] ? do_mpage_readpage+0x316/0x42f
> [103563.169717] [<ffffffff81036c0d>] ? update_curr+0xb1/0x159
> [103563.169744] [<ffffffffa05aa4f7>] ? xfs_get_blocks+0x0/0x14 [xfs]
> [103563.169772] [<ffffffffa05aa4f7>] ? xfs_get_blocks+0x0/0x14 [xfs]
> [103563.169784] [<ffffffff810af286>] ? add_to_page_cache_locked+0x71/0xbb
> [103563.169813] [<ffffffffa05aa4f7>] ? xfs_get_blocks+0x0/0x14 [xfs]
> [103563.169825] [<ffffffff8110fc58>] ? mpage_readpages+0xce/0x112
> [103563.169853] [<ffffffffa05aa4f7>] ? xfs_get_blocks+0x0/0x14 [xfs]
> [103563.169865] [<ffffffff81305929>] ? schedule+0x4d4/0x530
> [103563.169877] [<ffffffff810b6876>] ? __do_page_cache_readahead+0x117/0x1b3
> [103563.169891] [<ffffffff810b692e>] ? ra_submit+0x1c/0x20
> [103563.169902] [<ffffffff8110668b>] ? __generic_file_splice_read+0x116/0x420
> [103563.169917] [<ffffffff81292523>] ? tcp_write_xmit+0x445/0x8fc
> [103563.169929] [<ffffffff81306561>] ? down_read+0x9/0x19
> [103563.169954] [<ffffffffa058f66a>] ? xfs_ilock+0x5c/0x97 [xfs]
> [103563.169980] [<ffffffffa058ff1c>] ? xfs_iget+0x51f/0x58b [xfs]
> [103563.169990] [<ffffffff812491ca>] ? release_sock+0x19/0xf3
> [103563.170002] [<ffffffff810b2dfc>] ? free_pcppages_bulk+0x13d/0x150
> [103563.170015] [<ffffffff81104eca>] ? spd_release_page+0x0/0x14
> [103563.170027] [<ffffffff811069d0>] ? generic_file_splice_read+0x3b/0x63
> [103563.170056] [<ffffffffa05ae4cd>] ? xfs_file_splice_read+0xe4/0x12e [xfs]
> [103563.170070] [<ffffffff81105019>] ? splice_direct_to_actor+0xba/0x17b
> [103563.170086] [<ffffffffa061d851>] ? nfsd_direct_splice_actor+0x0/0x13 [nfsd]
> [103563.170103] [<ffffffffa061d748>] ? nfsd_vfs_read+0x242/0x34b [nfsd]
> [103563.170121] [<ffffffffa061de4d>] ? nfsd_read+0xa5/0xc4 [nfsd]
> [103563.170137] [<ffffffffa0624583>] ? nfsd3_proc_read+0xd6/0x114 [nfsd]
> [103563.170153] [<ffffffffa0619334>] ? nfsd_dispatch+0xdf/0x1b5 [nfsd]
> [103563.170172] [<ffffffffa02a7596>] ? svc_process+0x40d/0x60b [sunrpc]
> [103563.170188] [<ffffffffa0619851>] ? nfsd+0xe8/0x131 [nfsd]
> [103563.170201] [<ffffffffa0619769>] ? nfsd+0x0/0x131 [nfsd]
> [103563.170212] [<ffffffff8105b541>] ? kthread+0x75/0x7d
> [103563.170224] [<ffffffff810097e4>] ? kernel_thread_helper+0x4/0x10
> [103563.170236] [<ffffffff8105b4cc>] ? kthread+0x0/0x7d
> [103563.170246] [<ffffffff810097e0>] ? kernel_thread_helper+0x0/0x10
> [103563.170258] INFO: task nfsd:3500 blocked for more than 120 seconds.
> [103563.170267] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [103563.170277] nfsd D 000000010189190d 0 3500 2 0x00000000
> [103563.170288] ffff88042aa506d0 0000000000000046 ffff880426984af0 0000000000000000
> [103563.170300] ffff88042e0da8e0 00000000000154c0 00000000000154c0 00000000000154c0
> [103563.170310] ffff880420035fd8 00000000000154c0 ffff88042aa506d0 ffff880420035fd8
> [103563.170322] Call Trace:
> [103563.170327] [<ffffffff8100e806>] ? read_tsc+0x5/0x16
> [103563.170338] [<ffffffff810aefe5>] ? sync_page+0x0/0x45
> [103563.170349] [<ffffffff813059f0>] ? io_schedule+0x6b/0xaa
> [103563.170361] [<ffffffff810af026>] ? sync_page+0x41/0x45
> [103563.170371] [<ffffffff81305e3b>] ? __wait_on_bit_lock+0x3c/0x85
> [103563.170384] [<ffffffff810aefd2>] ? __lock_page+0x5d/0x63
> [103563.170395] [<ffffffff8105b9bf>] ? wake_bit_function+0x0/0x2e
> [103563.170407] [<ffffffff811067e1>] ? __generic_file_splice_read+0x26c/0x420
> [103563.170433] [<ffffffffa0572f5b>] ? xfs_bmap_search_multi_extents+0x85/0xdb [xfs]
> [103563.170447] [<ffffffff8124f695>] ? memcpy_toiovec+0x38/0x67
> [103563.170459] [<ffffffff81306561>] ? down_read+0x9/0x19
> [103563.170484] [<ffffffffa058f66a>] ? xfs_ilock+0x5c/0x97 [xfs]
> [103563.170510] [<ffffffffa058ff1c>] ? xfs_iget+0x51f/0x58b [xfs]
> [103563.170522] [<ffffffff812491ca>] ? release_sock+0x19/0xf3
> [103563.170548] [<ffffffffa058f4d0>] ? xfs_iunlock+0x5b/0xb7 [xfs]
> [103563.170560] [<ffffffff81104eca>] ? spd_release_page+0x0/0x14
> [103563.170572] [<ffffffff811069d0>] ? generic_file_splice_read+0x3b/0x63
> [103563.170601] [<ffffffffa05ae4cd>] ? xfs_file_splice_read+0xe4/0x12e [xfs]
> [103563.170615] [<ffffffff81105019>] ? splice_direct_to_actor+0xba/0x17b
> [103563.170631] [<ffffffffa061d851>] ? nfsd_direct_splice_actor+0x0/0x13 [nfsd]
> [103563.170649] [<ffffffffa061d748>] ? nfsd_vfs_read+0x242/0x34b [nfsd]
> [103563.170665] [<ffffffffa061de4d>] ? nfsd_read+0xa5/0xc4 [nfsd]
> [103563.170682] [<ffffffffa0624583>] ? nfsd3_proc_read+0xd6/0x114 [nfsd]
> [103563.170698] [<ffffffffa0619334>] ? nfsd_dispatch+0xdf/0x1b5 [nfsd]
> [103563.170719] [<ffffffffa02a7596>] ? svc_process+0x40d/0x60b [sunrpc]
> [103563.170734] [<ffffffffa0619851>] ? nfsd+0xe8/0x131 [nfsd]
> [103563.170748] [<ffffffffa0619769>] ? nfsd+0x0/0x131 [nfsd]
> [103563.170759] [<ffffffff8105b541>] ? kthread+0x75/0x7d
> [103563.170770] [<ffffffff810097e4>] ? kernel_thread_helper+0x4/0x10
> [103563.170782] [<ffffffff8105b4cc>] ? kthread+0x0/0x7d
> [103563.170793] [<ffffffff810097e0>] ? kernel_thread_helper+0x0/0x10
>
>
> == full error from (1) ==
> [103350.849291] ata4.04: exception Emask 0x0 SAct 0x92f SErr 0x0 action 0x6
> [103350.892975] ata4.04: irq_stat 0x00020002, device error via D2H FIS
> [103350.934022] ata4.04: failed command: READ FPDMA QUEUED
> [103350.968878] ata4.04: cmd 60/80:00:3f:97:d3/00:00:0b:00:00/40 tag 0 ncq 65536 in
> [103350.968879] res 41/40:00:61:95:d3/00:01:0b:00:00/00 Emask 0x9 (media error)
> [103351.067418] ata4.04: status: { DRDY ERR }
> [103351.095674] ata4.04: error: { UNC }
> [103351.120802] ata4.04: failed command: READ FPDMA QUEUED
> [103351.155855] ata4.04: cmd 60/00:08:3f:95:d3/01:00:0b:00:00/40 tag 1 ncq 131072 in
> [103351.155857] res 41/40:00:61:95:d3/00:01:0b:00:00/00 Emask 0x409 (media error) <F>
> [103351.258408] ata4.04: status: { DRDY ERR }
> [103351.286897] ata4.04: error: { UNC }
> [103351.312300] ata4.04: failed command: READ FPDMA QUEUED
> [103351.347597] ata4.04: cmd 60/80:10:3f:98:d3/00:00:0b:00:00/40 tag 2 ncq 65536 in
> [103351.347598] res 60/80:10:3f:98:d3/00:00:0b:00:00/40 Emask 0x10 (ATA bus error)
> [103351.448427] ata4.04: status: { DRDY DF }
> [103351.476575] ata4.04: error: { ICRC }
> [103351.502633] ata4.04: failed command: READ FPDMA QUEUED
> [103351.538054] ata4.04: cmd 60/80:18:bf:97:d3/00:00:0b:00:00/40 tag 3 ncq 65536 in
> [103351.538055] res 60/80:18:bf:97:d3/00:00:0b:00:00/40 Emask 0x10 (ATA bus error)
> [103351.639136] ata4.04: status: { DRDY DF }
> [103351.667451] ata4.04: error: { ICRC }
> [103351.693624] ata4.04: failed command: READ FPDMA QUEUED
> [103351.729201] ata4.04: cmd 60/00:28:3f:96:d3/01:00:0b:00:00/40 tag 5 ncq 131072 in
> [103351.729202] res 5a/2a:04:04:00:00/00:00:00:50:5a/00 Emask 0x2 (HSM violation)
> [103351.830689] ata4.04: status: { DRDY DRQ }
> [103351.859707] ata4.04: failed command: READ FPDMA QUEUED
> [103351.895529] ata4.04: cmd 60/80:40:bf:98:d3/00:00:0b:00:00/40 tag 8 ncq 65536 in
> [103351.895530] res 60/80:40:bf:98:d3/00:00:0b:00:00/40 Emask 0x10 (ATA bus error)
> [103351.997385] ata4.04: status: { DRDY DF }
> [103352.026059] ata4.04: error: { ICRC }
> [103352.052553] ata4.04: failed command: READ FPDMA QUEUED
> [103352.088421] ata4.04: cmd 60/00:58:3f:99:d3/01:00:0b:00:00/40 tag 11 ncq 131072 in
> [103352.088422] res 60/00:58:3f:99:d3/00:00:0b:00:00/40 Emask 0x1 (device error)
> [103352.190450] ata4.04: status: { DRDY DF }
> [103352.219323] ata4.04: hard resetting link
> [103352.584379] ata4.04: SATA link up 3.0 Gbps (SStatus 123 SControl 320)
> [103352.631003] ata4.04: configured for UDMA/100
> [103352.661948] ata4: EH complete
> [103354.630686] ata4.04: exception Emask 0x0 SAct 0x620 SErr 0x0 action 0x6
> [103354.675663] ata4.04: irq_stat 0x00060002, device error via SDB FIS
> [103354.718067] ata4.04: failed command: READ FPDMA QUEUED
> [103354.754258] ata4.04: cmd 60/80:28:bf:98:d3/00:00:0b:00:00/40 tag 5 ncq 65536 in
> [103354.754259] res 5a/2a:04:04:00:00/00:00:00:50:5a/00 Emask 0x2 (HSM violation)
> [103354.856506] ata4.04: status: { DRDY DRQ }
> [103354.886119] ata4.04: failed command: READ FPDMA QUEUED
> [103354.922522] ata4.04: cmd 60/00:48:3f:95:d3/01:00:0b:00:00/40 tag 9 ncq 131072 in
> [103354.922523] res 41/40:00:61:95:d3/00:01:0b:00:00/00 Emask 0x409 (media error) <F>
> [103355.027674] ata4.04: status: { DRDY ERR }
> [103355.057331] ata4.04: error: { UNC }
> [103355.083824] ata4.04: failed command: READ FPDMA QUEUED
> [103355.083830] ata4.04: cmd 60/80:50:3f:97:d3/00:00:0b:00:00/40 tag 10 ncq 65536 in
> [103355.083833] res 5a/2a:04:04:00:00/00:00:00:a0:5a/00 Emask 0x2 (HSM violation)
> [103355.083837] ata4.04: status: { DRDY DRQ }
> [103355.083907] ata4.04: hard resetting link
> [103355.432335] ata4.04: SATA link up 3.0 Gbps (SStatus 123 SControl 320)
> [103355.479370] ata4.04: configured for UDMA/100
> [103355.510657] ata4: EH complete
> [103357.495976] ata4.04: exception Emask 0x0 SAct 0x2 SErr 0x0 action 0x0
> [103357.540217] ata4.04: irq_stat 0x00060002, device error via SDB FIS
> [103357.582910] ata4.04: failed command: READ FPDMA QUEUED
> [103357.619344] ata4.04: cmd 60/00:08:3f:95:d3/01:00:0b:00:00/40 tag 1 ncq 131072 in
> [103357.619345] res 41/40:00:61:95:d3/00:01:0b:00:00/00 Emask 0x409 (media error) <F>
> [103357.724580] ata4.04: status: { DRDY ERR }
> [103357.754332] ata4.04: error: { UNC }
> [103357.783799] ata4.04: configured for UDMA/100
> [103357.815172] ata4: EH complete
> [103359.786533] ata4.04: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x0
> [103359.830796] ata4.04: irq_stat 0x00060002, device error via SDB FIS
> [103359.873539] ata4.04: failed command: READ FPDMA QUEUED
> [103359.910035] ata4.04: cmd 60/00:00:3f:95:d3/01:00:0b:00:00/40 tag 0 ncq 131072 in
> [103359.910037] res 41/40:00:61:95:d3/00:01:0b:00:00/00 Emask 0x409 (media error) <F>
> [103359.910040] ata4.04: status: { DRDY ERR }
> [103359.910041] ata4.04: error: { UNC }
> [103360.018086] ata4.04: configured for UDMA/100
> [103360.071645] ata4: EH complete
> [103362.052078] ata4.04: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x0
> [103362.096263] ata4.04: irq_stat 0x00060002, device error via SDB FIS
> [103362.138914] ata4.04: failed command: READ FPDMA QUEUED
> [103362.175334] ata4.04: cmd 60/00:00:3f:95:d3/01:00:0b:00:00/40 tag 0 ncq 131072 in
> [103362.175336] res 41/40:00:61:95:d3/00:01:0b:00:00/00 Emask 0x409 (media error) <F>
> [103362.280506] ata4.04: status: { DRDY ERR }
> [103362.310179] ata4.04: error: { UNC }
> [103362.339586] ata4.04: configured for UDMA/100
> [103362.370840] ata4: EH complete
> [103364.334349] ata4.04: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x0
> [103364.378582] ata4.04: irq_stat 0x00060002, device error via SDB FIS
> [103364.421199] ata4.04: failed command: READ FPDMA QUEUED
> [103364.457555] ata4.04: cmd 60/00:00:3f:95:d3/01:00:0b:00:00/40 tag 0 ncq 131072 in
> [103364.457556] res 41/40:00:61:95:d3/00:01:0b:00:00/00 Emask 0x409 (media error) <F>
> [103364.562587] ata4.04: status: { DRDY ERR }
> [103364.592185] ata4.04: error: { UNC }
> [103364.621387] ata4.04: configured for UDMA/100
> [103364.652557] sd 3:4:0:0: [sdt] Unhandled sense code
> [103364.686712] sd 3:4:0:0: [sdt] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [103364.734972] sd 3:4:0:0: [sdt] Sense Key : Medium Error [current] [descriptor]
> [103364.783292] Descriptor sense data with sense descriptors (in hex):
> [103364.825802] 72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00
> [103364.869717] 0b d3 95 61
> [103364.894588] sd 3:4:0:0: [sdt] Add. Sense: Unrecovered read error - auto reallocate failed
> [103364.949216] sd 3:4:0:0: [sdt] CDB: Read(10): 28 00 0b d3 95 3f 00 01 00 00
> [103364.969589] end_request: I/O error, dev sdt, sector 198415713
> [103364.969619] ata4: EH complete
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: md raid5 blocked in get_active_stripe after disk failure
2011-01-12 14:06 ` David Flynn
@ 2011-01-12 20:33 ` NeilBrown
2011-01-19 13:45 ` David Flynn
2011-02-12 0:41 ` David Flynn
0 siblings, 2 replies; 6+ messages in thread
From: NeilBrown @ 2011-01-12 20:33 UTC (permalink / raw)
To: David Flynn; +Cc: linux-raid
On Wed, 12 Jan 2011 14:06:32 +0000 David Flynn <davidf@rd.bbc.co.uk> wrote:
> * David Flynn (davidf@rd.bbc.co.uk) wrote:
> > I have a fileserver that mounts an md raid5 array with 20 active disks,
> > holding an xfs filesystem which is then exported via nfs. (system is
> > a twin-quadcore intel, 64bit)
>
> I ought to mention, this is running 2.6.35.8
>
Ah... you had already answer that, sorry.
There have been no changes in md since then which could affect your problem,
so if it in an md problem it is still there.
If this is easily reproducible (which I think you suggested it is), I would
put a printk in raid5_end_read_request where it sets "retry = 1" to report
"trying to fix a failed read".
If you then get a hang but don't get that message, it is a device driver
problem. If you do get that message it is probably an md/raid5 problem.
NeilBrown
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: md raid5 blocked in get_active_stripe after disk failure
2011-01-12 20:33 ` NeilBrown
@ 2011-01-19 13:45 ` David Flynn
2011-02-12 0:41 ` David Flynn
1 sibling, 0 replies; 6+ messages in thread
From: David Flynn @ 2011-01-19 13:45 UTC (permalink / raw)
To: NeilBrown; +Cc: David Flynn, linux-raid
* NeilBrown (neilb@suse.de) wrote:
> If this is easily reproducible (which I think you suggested it is), I would
> put a printk in raid5_end_read_request where it sets "retry = 1" to report
> "trying to fix a failed read".
>
> If you then get a hang but don't get that message, it is a device driver
> problem. If you do get that message it is probably an md/raid5 problem.
Thanks for the quick reply.
At the time it was easily reproducible (after rebooting as soon as the
nfs clients retried their requests). Unfortunately, I had to remove the
disk from the array, and will not be able to test the above until next
month.
Kind regards,
..david
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: md raid5 blocked in get_active_stripe after disk failure
2011-01-12 20:33 ` NeilBrown
2011-01-19 13:45 ` David Flynn
@ 2011-02-12 0:41 ` David Flynn
1 sibling, 0 replies; 6+ messages in thread
From: David Flynn @ 2011-02-12 0:41 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
On Thu, Jan 13, 2011 at 07:33:31AM +1100, NeilBrown wrote:
> If this is easily reproducible (which I think you suggested it is), I would
> put a printk in raid5_end_read_request where it sets "retry = 1" to report
> "trying to fix a failed read".
>
> If you then get a hang but don't get that message, it is a device driver
> problem. If you do get that message it is probably an md/raid6 problem.
I was able to reproduce again (without specific interaction from me), and
with the following patch applied:
diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
index 96c6902..eff0a0a 100644
--- a/drivers/md/raid5.c
+++ b/drivers/md/raid5.c
@@ -1618,8 +1618,12 @@ static void raid5_end_read_request(struct bio * bi, int error)
printk(KERN_WARNING
"md/raid:%s: Too many read errors, failing device %s.\n",
mdname(conf->mddev), bdn);
- else
+ else {
+ printk(KERN_WARNING
+ "md/raid:%s: trying to fix a failed read (%s)\n",
+ mdname(conf->mddev), bdn);
retry = 1;
+ }
if (retry)
set_bit(R5_ReadError, &sh->dev[i].flags);
else {
the added printk wasn't hit (see below). access to the underlying
block device [sdj] was possible, although anything that touches the
xfs filesystem on md0 blocks. i didn't try poking md0 directly this
time. subsequent dd'ing from the faulty sector on sdj eventually
caused the disk to succeed in reading, although this didn't help
anything else.
this was repeated twice:
- the first time didn't have the prink mod. pulling the disk
physically, ata spots it, but md doesn't. re-inserting
doesn't help.
- the second did have the printk mod (and also started on resync)
the failure is logged below:
kind regards,
..david
[23184.612855] end_request: I/O error, dev sdj, sector 1463405046
[23184.652751] ata2: EH complete
[23404.788116] INFO: task md1_raid5:2312 blocked for more than 120 seconds.
[23404.833357] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[23404.885415] md1_raid5 D 0000000100574c4b 0 2312 2 0x00000000
[23404.901082] ffff880426600da0 0000000000000046 0000000000000007 ffffffff00000000
[23404.901085] ffff88042e0ddf60 00000000000154c0 00000000000154c0 00000000000154c0
[23404.901087] ffff880426651fd8 00000000000154c0 ffff880426600da0 ffff880426651fd8
[23404.901089] Call Trace:
[23404.901099] [<ffffffffa01de156>] ? get_active_stripe+0xfe/0x55b [raid456]
[23404.901105] [<ffffffff8103f760>] ? default_wake_function+0x0/0xf
[23404.901108] [<ffffffffa01dffb5>] ? raid5d+0x149/0x3f2 [raid456]
[23404.901118] [<ffffffffa0278010>] ? md_thread+0xf2/0x110 [md_mod]
[23404.901122] [<ffffffff8105b995>] ? autoremove_wake_function+0x0/0x2a
[23404.901126] [<ffffffffa0277f1e>] ? md_thread+0x0/0x110 [md_mod]
[23404.901129] [<ffffffff8105b541>] ? kthread+0x75/0x7d
[23404.901133] [<ffffffff810097e4>] ? kernel_thread_helper+0x4/0x10
[23404.901135] [<ffffffff8105b4cc>] ? kthread+0x0/0x7d
[23404.901137] [<ffffffff810097e0>] ? kernel_thread_helper+0x0/0x10
[23404.901140] INFO: task md1_resync:2313 blocked for more than 120 seconds.
[23404.901141] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[23404.901142] md1_resync D 0000000100573f32 0 2313 2 0x00000000
[23404.901145] ffff88042783a210 0000000000000046 0000000000000286 ffff880400000000
[23404.901147] ffff88042e0ddf60 00000000000154c0 00000000000154c0 00000000000154c0
[23404.901149] ffff880426653fd8 00000000000154c0 ffff88042783a210 ffff880426653fd8
[23404.901151] Call Trace:
[23404.901154] [<ffffffffa01e103f>] ? sync_request+0x16a/0x2d1 [raid456]
[23404.901157] [<ffffffff8105b995>] ? autoremove_wake_function+0x0/0x2a
[23404.901162] [<ffffffffa02778f0>] ? md_do_sync+0x712/0xb0d [md_mod]
[23404.901167] [<ffffffffa0278010>] ? md_thread+0xf2/0x110 [md_mod]
[23404.901171] [<ffffffffa0277f1e>] ? md_thread+0x0/0x110 [md_mod]
[23404.901174] [<ffffffff8105b541>] ? kthread+0x75/0x7d
[23404.901176] [<ffffffff810097e4>] ? kernel_thread_helper+0x4/0x10
[23404.901178] [<ffffffff8105b4cc>] ? kthread+0x0/0x7d
[23404.901181] [<ffffffff810097e0>] ? kernel_thread_helper+0x0/0x10
[23404.901183] INFO: task flush-9:1:2341 blocked for more than 120 seconds.
[23404.901184] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[23404.901185] flush-9:1 D 0000000100573f32 0 2341 2 0x00000000
[23404.901188] ffff880427839470 0000000000000046 ffff8804252c5998 ffff880400000000
[23404.901190] ffff88042e0da8e0 00000000000154c0 00000000000154c0 00000000000154c0
[23404.901192] ffff8804252c5fd8 00000000000154c0 ffff880427839470 ffff8804252c5fd8
[23404.901194] Call Trace:
[23404.901199] [<ffffffff81305d07>] ? schedule_timeout+0x2d/0xd7
[23404.901202] [<ffffffff81306645>] ? __down_common+0x92/0xe3
[23404.901233] [<ffffffffa057a3bd>] ? xfs_getsb+0x25/0x32 [xfs]
[23404.901236] [<ffffffff8105f54c>] ? down+0x27/0x37
[23404.901255] [<ffffffffa05877ab>] ? xfs_buf_lock+0x73/0xaa [xfs]
[23404.901272] [<ffffffffa057a3bd>] ? xfs_getsb+0x25/0x32 [xfs]
[23404.901291] [<ffffffffa057feda>] ? xfs_trans_getsb+0x63/0xb2 [xfs]
[23404.901309] [<ffffffffa057dfcd>] ? xfs_trans_apply_sb_deltas+0x12/0x1da [xfs]
[23404.901328] [<ffffffffa057e9a8>] ? _xfs_trans_commit+0x50/0x221 [xfs]
[23404.901345] [<ffffffffa0570794>] ? xfs_iomap_write_allocate+0x270/0x36c [xfs]
[23404.901362] [<ffffffffa056a4f0>] ? xfs_iunlock+0x7b/0xb7 [xfs]
[23404.901380] [<ffffffffa0585356>] ? xfs_map_blocks+0x37/0x3e [xfs]
[23404.901398] [<ffffffffa05863d5>] ? xfs_page_state_convert+0x268/0x50e [xfs]
[23404.901417] [<ffffffffa0586843>] ? xfs_vm_writepage+0xef/0x12c [xfs]
[23404.901420] [<ffffffff810b548a>] ? __writepage+0xa/0x21
[23404.901422] [<ffffffff810b6504>] ? write_cache_pages+0x1d8/0x2ec
[23404.901425] [<ffffffff810b5480>] ? __writepage+0x0/0x21
[23404.901429] [<ffffffff81102cd4>] ? writeback_single_inode+0xd1/0x2e5
[23404.901432] [<ffffffff8110323e>] ? writeback_sb_inodes+0x136/0x204
[23404.901434] [<ffffffff811038e3>] ? writeback_inodes_wb+0x127/0x139
[23404.901437] [<ffffffff81103a83>] ? wb_writeback+0x18e/0x208
[23404.901439] [<ffffffff81103c30>] ? wb_do_writeback+0x133/0x149
[23404.901442] [<ffffffff81103c80>] ? bdi_writeback_task+0x3a/0x112
[23404.901444] [<ffffffff8105b8bb>] ? bit_waitqueue+0x14/0xa1
[23404.901447] [<ffffffff810c1f4f>] ? bdi_start_fn+0x0/0xc3
[23404.901449] [<ffffffff810c1fb2>] ? bdi_start_fn+0x63/0xc3
[23404.901451] [<ffffffff810c1f4f>] ? bdi_start_fn+0x0/0xc3
[23404.901453] [<ffffffff8105b541>] ? kthread+0x75/0x7d
[23404.901456] [<ffffffff810097e4>] ? kernel_thread_helper+0x4/0x10
[23404.901458] [<ffffffff8105b4cc>] ? kthread+0x0/0x7d
[23404.901460] [<ffffffff810097e0>] ? kernel_thread_helper+0x0/0x10
[23404.901468] INFO: task xfssyncd/md1:2676 blocked for more than 120 seconds.
[23404.901469] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[23404.901470] xfssyncd/md1 D 0000000100573f32 0 2676 2 0x00000000
[23404.901473] ffff88042a71afb0 0000000000000046 0000000000000086 ffffffff00000000
[23404.901475] ffff88042e06ed00 00000000000154c0 00000000000154c0 00000000000154c0
[23404.901477] ffff88042b57ffd8 00000000000154c0 ffff88042a71afb0 ffff88042b57ffd8
[23404.901479] Call Trace:
[23404.901482] [<ffffffffa01dc520>] ? raid5_quiesce+0xe3/0x19b [raid456]
[23404.901485] [<ffffffff8100e7d9>] ? sched_clock+0x5/0x8
[23404.901487] [<ffffffff8103f760>] ? default_wake_function+0x0/0xf
[23404.901491] [<ffffffffa01e1971>] ? make_request+0x31/0x403 [raid456]
[23404.901508] [<ffffffffa0573f35>] ? xlog_write+0x430/0x46e [xfs]
[23404.901513] [<ffffffffa02767ca>] ? md_make_request+0xd0/0x1d4 [md_mod]
[23404.901530] [<ffffffffa057542f>] ? xlog_commit_record+0x92/0xc2 [xfs]
[23404.901548] [<ffffffffa0580333>] ? xfs_trans_unlock_chunk+0x7d/0xc1 [xfs]
[23404.901552] [<ffffffff8117583c>] ? generic_make_request+0x27a/0x2f0
[23404.901556] [<ffffffff810b09dd>] ? mempool_alloc+0x53/0x106
[23404.901558] [<ffffffff81175983>] ? submit_bio+0xd1/0xed
[23404.901576] [<ffffffffa05874d2>] ? _xfs_buf_ioapply+0x212/0x24f [xfs]
[23404.901593] [<ffffffffa0572ace>] ? xlog_bdstrat+0x3a/0x3e [xfs]
[23404.901611] [<ffffffffa0588430>] ? xfs_buf_iorequest+0x75/0x8e [xfs]
[23404.901628] [<ffffffffa0572ace>] ? xlog_bdstrat+0x3a/0x3e [xfs]
[23404.901645] [<ffffffffa0572ce1>] ? xlog_sync+0x20f/0x326 [xfs]
[23404.901663] [<ffffffffa05736d4>] ? _xfs_log_force+0xd9/0x196 [xfs]
[23404.901680] [<ffffffffa0573915>] ? xfs_log_force+0xb/0x2a [xfs]
[23404.901698] [<ffffffffa058fd5d>] ? xfs_commit_dummy_trans+0xb3/0xc2 [xfs]
[23404.901717] [<ffffffffa05903ac>] ? xfs_sync_worker+0x41/0x60 [xfs]
[23404.901735] [<ffffffffa058fc2e>] ? xfssyncd+0x155/0x1d1 [xfs]
[23404.901753] [<ffffffffa058fad9>] ? xfssyncd+0x0/0x1d1 [xfs]
[23404.901755] [<ffffffff8105b541>] ? kthread+0x75/0x7d
[23404.901757] [<ffffffff810097e4>] ? kernel_thread_helper+0x4/0x10
[23404.901760] [<ffffffff8105b4cc>] ? kthread+0x0/0x7d
[23404.901762] [<ffffffff810097e0>] ? kernel_thread_helper+0x0/0x10
[23404.901767] INFO: task nfsd:3388 blocked for more than 120 seconds.
[23404.901768] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[23404.901770] nfsd D 00000001005740cc 0 3388 2 0x00000000
[23404.901772] ffff88042a71a210 0000000000000046 ffff880100001000 0000000000000000
[23404.901775] ffff88042e0da8e0 00000000000154c0 00000000000154c0 00000000000154c0
[23404.901777] ffff88042128dfd8 00000000000154c0 ffff88042a71a210 ffff88042128dfd8
[23404.901779] Call Trace:
[23404.901781] [<ffffffff8100e806>] ? read_tsc+0x5/0x16
[23404.901784] [<ffffffff810aefe5>] ? sync_page+0x0/0x45
[23404.901786] [<ffffffff813059f0>] ? io_schedule+0x6b/0xaa
[23404.901788] [<ffffffff810af026>] ? sync_page+0x41/0x45
[23404.901791] [<ffffffff81305e3b>] ? __wait_on_bit_lock+0x3c/0x85
[23404.901793] [<ffffffff810aefd2>] ? __lock_page+0x5d/0x63
[23404.901795] [<ffffffff8105b9bf>] ? wake_bit_function+0x0/0x2e
[23404.901798] [<ffffffff811067e1>] ? __generic_file_splice_read+0x26c/0x420
[23404.901802] [<ffffffff81255ec9>] ? dev_queue_xmit+0x42d/0x45b
[23404.901805] [<ffffffff8127e914>] ? ip_queue_xmit+0x2dc/0x33c
[23404.901807] [<ffffffff81306561>] ? down_read+0x9/0x19
[23404.901824] [<ffffffffa056a66a>] ? xfs_ilock+0x5c/0x97 [xfs]
[23404.901840] [<ffffffffa056af1c>] ? xfs_iget+0x51f/0x58b [xfs]
[23404.901843] [<ffffffff812491ca>] ? release_sock+0x19/0xf3
[23404.901846] [<ffffffff810b2dfc>] ? free_pcppages_bulk+0x13d/0x150
[23404.901849] [<ffffffff81104eca>] ? spd_release_page+0x0/0x14
[23404.901851] [<ffffffff811069d0>] ? generic_file_splice_read+0x3b/0x63
[23404.901869] [<ffffffffa05894cd>] ? xfs_file_splice_read+0xe4/0x12e [xfs]
[23404.901872] [<ffffffff81105019>] ? splice_direct_to_actor+0xba/0x17b
[23404.901881] [<ffffffffa05d4851>] ? nfsd_direct_splice_actor+0x0/0x13 [nfsd]
[23404.901887] [<ffffffffa05d4748>] ? nfsd_vfs_read+0x242/0x34b [nfsd]
[23404.901894] [<ffffffffa05d4e4d>] ? nfsd_read+0xa5/0xc4 [nfsd]
[23404.901901] [<ffffffffa05db583>] ? nfsd3_proc_read+0xd6/0x114 [nfsd]
[23404.901907] [<ffffffffa05d0334>] ? nfsd_dispatch+0xdf/0x1b5 [nfsd]
[23404.901929] [<ffffffffa02e0596>] ? svc_process+0x40d/0x60b [sunrpc]
[23404.901934] [<ffffffffa05d0851>] ? nfsd+0xe8/0x131 [nfsd]
[23404.901939] [<ffffffffa05d0769>] ? nfsd+0x0/0x131 [nfsd]
[23404.901941] [<ffffffff8105b541>] ? kthread+0x75/0x7d
[23404.901943] [<ffffffff810097e4>] ? kernel_thread_helper+0x4/0x10
[23404.901946] [<ffffffff8105b4cc>] ? kthread+0x0/0x7d
[23404.901948] [<ffffffff810097e0>] ? kernel_thread_helper+0x0/0x10
[23404.901950] INFO: task nfsd:3389 blocked for more than 120 seconds.
[23404.901951] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[23404.901953] nfsd D 0000000100573eb1 0 3389 2 0x00000000
[23404.901955] ffff88042a71e630 0000000000000046 0000000100000000 ffff880400000000
[23404.901958] ffff88042e06b680 00000000000154c0 00000000000154c0 00000000000154c0
[23404.901960] ffff880421387fd8 00000000000154c0 ffff88042a71e630 ffff880421387fd8
[23404.901962] Call Trace:
[23404.901964] [<ffffffff8100e806>] ? read_tsc+0x5/0x16
[23404.901966] [<ffffffff810aefe5>] ? sync_page+0x0/0x45
[23404.901969] [<ffffffff813059f0>] ? io_schedule+0x6b/0xaa
[23404.901971] [<ffffffff810af026>] ? sync_page+0x41/0x45
[23404.901973] [<ffffffff81305f39>] ? __wait_on_bit+0x3e/0x6f
[23404.901975] [<ffffffff810af161>] ? wait_on_page_bit+0x6a/0x70
[23404.901978] [<ffffffff8105b9bf>] ? wake_bit_function+0x0/0x2e
[23404.901980] [<ffffffff810b6de4>] ? pagevec_lookup_tag+0x18/0x1f
[23404.901982] [<ffffffff810af3b5>] ? filemap_fdatawait_range+0x96/0x14a
[23404.901985] [<ffffffff810af4d2>] ? __filemap_fdatawrite_range+0x48/0x4d
[23404.901988] [<ffffffff810af514>] ? filemap_write_and_wait_range+0x3d/0x50
[23404.902006] [<ffffffffa058a5d1>] ? xfs_file_aio_write+0x6ab/0x762 [xfs]
[23404.902023] [<ffffffffa056af1c>] ? xfs_iget+0x51f/0x58b [xfs]
[23404.902026] [<ffffffff810e0afe>] ? __slab_free+0x1c/0x250
[23404.902044] [<ffffffffa0589f26>] ? xfs_file_aio_write+0x0/0x762 [xfs]
[23404.902047] [<ffffffff810e9324>] ? do_sync_readv_writev+0x9a/0xd5
[23404.902063] [<ffffffffa056a4f0>] ? xfs_iunlock+0x7b/0xb7 [xfs]
[23404.902066] [<ffffffff810e0f8b>] ? __slab_alloc+0xf9/0x3eb
[23404.902068] [<ffffffff810e90d5>] ? copy_from_user+0x18/0x30
[23404.902071] [<ffffffff810e9e43>] ? rw_copy_check_uvector+0x6a/0xe1
[23404.902073] [<ffffffff810e9f5e>] ? do_readv_writev+0xa4/0x118
[23404.902080] [<ffffffffa05d8ae5>] ? nfsd_setuser+0x1a2/0x1e5 [nfsd]
[23404.902085] [<ffffffffa05d2ff4>] ? nfsd_setuser_and_check_port+0x76/0x92 [nfsd]
[23404.902091] [<ffffffffa05d4300>] ? nfsd_vfs_write+0x11b/0x321 [nfsd]
[23404.902093] [<ffffffff810e81f8>] ? __dentry_open+0x184/0x28f
[23404.902099] [<ffffffffa05d4d87>] ? nfsd_write+0xbb/0xdc [nfsd]
[23404.902106] [<ffffffffa05d8e8a>] ? nfsd_cache_lookup+0x26e/0x37d [nfsd]
[23404.902113] [<ffffffffa05db492>] ? nfsd3_proc_write+0xcf/0xea [nfsd]
[23404.902118] [<ffffffffa05d0334>] ? nfsd_dispatch+0xdf/0x1b5 [nfsd]
[23404.902128] [<ffffffffa02e0596>] ? svc_process+0x40d/0x60b [sunrpc]
[23404.902133] [<ffffffffa05d0851>] ? nfsd+0xe8/0x131 [nfsd]
[23404.902138] [<ffffffffa05d0769>] ? nfsd+0x0/0x131 [nfsd]
[23404.902140] [<ffffffff8105b541>] ? kthread+0x75/0x7d
[23404.902143] [<ffffffff810097e4>] ? kernel_thread_helper+0x4/0x10
[23404.902145] [<ffffffff8105b4cc>] ? kthread+0x0/0x7d
[23404.902147] [<ffffffff810097e0>] ? kernel_thread_helper+0x0/0x10
[25525.716038] SysRq : Show Blocked State
[25525.719829] task PC stack pid father
[25525.719829] md1_raid5 D 0000000100574c4b 0 2312 2 0x00000000
[25525.719829] ffff880426600da0 0000000000000046 0000000000000007 ffffffff00000000
[25525.719829] ffff88042e0ddf60 00000000000154c0 00000000000154c0 00000000000154c0
[25525.916295] ffff880426651fd8 00000000000154c0 ffff880426600da0 ffff880426651fd8
[25525.916295] Call Trace:
[25525.916295] [<ffffffffa01de156>] ? get_active_stripe+0xfe/0x55b [raid456]
[25525.916295] [<ffffffff8103f760>] ? default_wake_function+0x0/0xf
[25525.916295] [<ffffffffa01dffb5>] ? raid5d+0x149/0x3f2 [raid456]
[25525.916295] [<ffffffffa0278010>] ? md_thread+0xf2/0x110 [md_mod]
[25525.916295] [<ffffffff8105b995>] ? autoremove_wake_function+0x0/0x2a
[25525.916295] [<ffffffffa0277f1e>] ? md_thread+0x0/0x110 [md_mod]
[25525.916295] [<ffffffff8105b541>] ? kthread+0x75/0x7d
[25525.916295] [<ffffffff810097e4>] ? kernel_thread_helper+0x4/0x10
[25525.916295] [<ffffffff8105b4cc>] ? kthread+0x0/0x7d
[25525.916295] [<ffffffff810097e0>] ? kernel_thread_helper+0x0/0x10
[25525.916295] md1_resync D 0000000100573f32 0 2313 2 0x00000000
[25525.916295] ffff88042783a210 0000000000000046 0000000000000286 ffff880400000000
[25525.916295] ffff88042e0ddf60 00000000000154c0 00000000000154c0 00000000000154c0
[25525.916295] ffff880426653fd8 00000000000154c0 ffff88042783a210 ffff880426653fd8
[25525.916295] Call Trace:
[25525.916295] [<ffffffffa01e103f>] ? sync_request+0x16a/0x2d1 [raid456]
[25525.916295] [<ffffffff8105b995>] ? autoremove_wake_function+0x0/0x2a
[25525.916295] [<ffffffffa02778f0>] ? md_do_sync+0x712/0xb0d [md_mod]
[25525.916295] [<ffffffffa0278010>] ? md_thread+0xf2/0x110 [md_mod]
[25525.916295] [<ffffffffa0277f1e>] ? md_thread+0x0/0x110 [md_mod]
[25525.916295] [<ffffffff8105b541>] ? kthread+0x75/0x7d
[25525.916295] [<ffffffff810097e4>] ? kernel_thread_helper+0x4/0x10
[25525.916295] [<ffffffff8105b4cc>] ? kthread+0x0/0x7d
[25525.916295] [<ffffffff810097e0>] ? kernel_thread_helper+0x0/0x10
[25525.916295] flush-9:1 D 0000000100573f32 0 2341 2 0x00000000
[25525.916295] ffff880427839470 0000000000000046 ffff8804252c5998 ffff880400000000
[25525.916295] ffff88042e0da8e0 00000000000154c0 00000000000154c0 00000000000154c0
[25525.916295] ffff8804252c5fd8 00000000000154c0 ffff880427839470 ffff8804252c5fd8
[25525.916295] Call Trace:
[25525.916295] [<ffffffff81305d07>] ? schedule_timeout+0x2d/0xd7
[25525.916295] [<ffffffff81306645>] ? __down_common+0x92/0xe3
[25525.916295] [<ffffffffa057a3bd>] ? xfs_getsb+0x25/0x32 [xfs]
[25525.916295] [<ffffffff8105f54c>] ? down+0x27/0x37
[25525.916295] [<ffffffffa05877ab>] ? xfs_buf_lock+0x73/0xaa [xfs]
[25525.916295] [<ffffffffa057a3bd>] ? xfs_getsb+0x25/0x32 [xfs]
[25525.916295] [<ffffffffa057feda>] ? xfs_trans_getsb+0x63/0xb2 [xfs]
[25525.916295] [<ffffffffa057dfcd>] ? xfs_trans_apply_sb_deltas+0x12/0x1da [xfs]
[25525.916295] [<ffffffffa057e9a8>] ? _xfs_trans_commit+0x50/0x221 [xfs]
[25525.916295] [<ffffffffa0570794>] ? xfs_iomap_write_allocate+0x270/0x36c [xfs]
[25525.916295] [<ffffffffa056a4f0>] ? xfs_iunlock+0x7b/0xb7 [xfs]
[25525.916295] [<ffffffffa0585356>] ? xfs_map_blocks+0x37/0x3e [xfs]
[25525.916295] [<ffffffffa05863d5>] ? xfs_page_state_convert+0x268/0x50e [xfs]
[25525.916295] [<ffffffffa0586843>] ? xfs_vm_writepage+0xef/0x12c [xfs]
[25525.916295] [<ffffffff810b548a>] ? __writepage+0xa/0x21
[25525.916295] [<ffffffff810b6504>] ? write_cache_pages+0x1d8/0x2ec
[25525.916295] [<ffffffff810b5480>] ? __writepage+0x0/0x21
[25525.916295] [<ffffffff81102cd4>] ? writeback_single_inode+0xd1/0x2e5
[25525.916295] [<ffffffff8110323e>] ? writeback_sb_inodes+0x136/0x204
[25525.916295] [<ffffffff811038e3>] ? writeback_inodes_wb+0x127/0x139
[25525.916295] [<ffffffff81103a83>] ? wb_writeback+0x18e/0x208
[25525.916295] [<ffffffff81103c30>] ? wb_do_writeback+0x133/0x149
[25525.916295] [<ffffffff81103c80>] ? bdi_writeback_task+0x3a/0x112
[25525.916295] [<ffffffff8105b8bb>] ? bit_waitqueue+0x14/0xa1
[25525.916295] [<ffffffff810c1f4f>] ? bdi_start_fn+0x0/0xc3
[25525.916295] [<ffffffff810c1fb2>] ? bdi_start_fn+0x63/0xc3
[25525.916295] [<ffffffff810c1f4f>] ? bdi_start_fn+0x0/0xc3
[25525.916295] [<ffffffff8105b541>] ? kthread+0x75/0x7d
[25525.916295] [<ffffffff810097e4>] ? kernel_thread_helper+0x4/0x10
[25525.916295] [<ffffffff8105b4cc>] ? kthread+0x0/0x7d
[25525.916295] [<ffffffff810097e0>] ? kernel_thread_helper+0x0/0x10
[25525.916295] xfssyncd/md1 D 0000000100573f32 0 2676 2 0x00000000
[25525.916295] ffff88042a71afb0 0000000000000046 0000000000000086 ffffffff00000000
[25525.916295] ffff88042e06ed00 00000000000154c0 00000000000154c0 00000000000154c0
[25525.916295] ffff88042b57ffd8 00000000000154c0 ffff88042a71afb0 ffff88042b57ffd8
[25525.916295] Call Trace:
[25525.916295] [<ffffffffa01dc520>] ? raid5_quiesce+0xe3/0x19b [raid456]
[25525.916295] [<ffffffff8100e7d9>] ? sched_clock+0x5/0x8
[25525.916295] [<ffffffff8103f760>] ? default_wake_function+0x0/0xf
[25525.916295] [<ffffffffa01e1971>] ? make_request+0x31/0x403 [raid456]
[25525.916295] [<ffffffffa0573f35>] ? xlog_write+0x430/0x46e [xfs]
[25525.916295] [<ffffffffa02767ca>] ? md_make_request+0xd0/0x1d4 [md_mod]
[25525.916295] [<ffffffffa057542f>] ? xlog_commit_record+0x92/0xc2 [xfs]
[25525.916295] [<ffffffffa0580333>] ? xfs_trans_unlock_chunk+0x7d/0xc1 [xfs]
[25525.916295] [<ffffffff8117583c>] ? generic_make_request+0x27a/0x2f0
[25525.916295] [<ffffffff810b09dd>] ? mempool_alloc+0x53/0x106
[25525.916295] [<ffffffff81175983>] ? submit_bio+0xd1/0xed
[25525.916295] [<ffffffffa05874d2>] ? _xfs_buf_ioapply+0x212/0x24f [xfs]
[25525.916295] [<ffffffffa0572ace>] ? xlog_bdstrat+0x3a/0x3e [xfs]
[25525.916295] [<ffffffffa0588430>] ? xfs_buf_iorequest+0x75/0x8e [xfs]
[25525.916295] [<ffffffffa0572ace>] ? xlog_bdstrat+0x3a/0x3e [xfs]
[25525.916295] [<ffffffffa0572ce1>] ? xlog_sync+0x20f/0x326 [xfs]
[25525.916295] [<ffffffffa05736d4>] ? _xfs_log_force+0xd9/0x196 [xfs]
[25525.916295] [<ffffffffa0573915>] ? xfs_log_force+0xb/0x2a [xfs]
[25525.916295] [<ffffffffa058fd5d>] ? xfs_commit_dummy_trans+0xb3/0xc2 [xfs]
[25525.916295] [<ffffffffa05903ac>] ? xfs_sync_worker+0x41/0x60 [xfs]
[25525.916295] [<ffffffffa058fc2e>] ? xfssyncd+0x155/0x1d1 [xfs]
[25525.916295] [<ffffffffa058fad9>] ? xfssyncd+0x0/0x1d1 [xfs]
[25525.916295] [<ffffffff8105b541>] ? kthread+0x75/0x7d
[25525.916295] [<ffffffff810097e4>] ? kernel_thread_helper+0x4/0x10
[25525.916295] [<ffffffff8105b4cc>] ? kthread+0x0/0x7d
[25525.916295] [<ffffffff810097e0>] ? kernel_thread_helper+0x0/0x10
[25525.916295] nfsd D 00000001005740cc 0 3388 2 0x00000000
[25525.916295] ffff88042a71a210 0000000000000046 ffff880100001000 0000000000000000
[25525.916295] ffff88042e0da8e0 00000000000154c0 00000000000154c0 00000000000154c0
[25525.916295] ffff88042128dfd8 00000000000154c0 ffff88042a71a210 ffff88042128dfd8
[25525.916295] Call Trace:
[25525.916295] [<ffffffff8100e806>] ? read_tsc+0x5/0x16
[25525.916295] [<ffffffff810aefe5>] ? sync_page+0x0/0x45
[25525.916295] [<ffffffff813059f0>] ? io_schedule+0x6b/0xaa
[25525.916295] [<ffffffff810af026>] ? sync_page+0x41/0x45
[25525.916295] [<ffffffff81305e3b>] ? __wait_on_bit_lock+0x3c/0x85
[25525.916295] [<ffffffff810aefd2>] ? __lock_page+0x5d/0x63
[25525.916295] [<ffffffff8105b9bf>] ? wake_bit_function+0x0/0x2e
[25525.916295] [<ffffffff811067e1>] ? __generic_file_splice_read+0x26c/0x420
[25525.916295] [<ffffffff81255ec9>] ? dev_queue_xmit+0x42d/0x45b
[25525.916295] [<ffffffff8127e914>] ? ip_queue_xmit+0x2dc/0x33c
[25525.916295] [<ffffffff81306561>] ? down_read+0x9/0x19
[25525.916295] [<ffffffffa056a66a>] ? xfs_ilock+0x5c/0x97 [xfs]
[25525.916295] [<ffffffffa056af1c>] ? xfs_iget+0x51f/0x58b [xfs]
[25525.916295] [<ffffffff812491ca>] ? release_sock+0x19/0xf3
[25525.916295] [<ffffffff810b2dfc>] ? free_pcppages_bulk+0x13d/0x150
[25525.916295] [<ffffffff81104eca>] ? spd_release_page+0x0/0x14
[25525.916295] [<ffffffff811069d0>] ? generic_file_splice_read+0x3b/0x63
[25525.916295] [<ffffffffa05894cd>] ? xfs_file_splice_read+0xe4/0x12e [xfs]
[25525.916295] [<ffffffff81105019>] ? splice_direct_to_actor+0xba/0x17b
[25525.916295] [<ffffffffa05d4851>] ? nfsd_direct_splice_actor+0x0/0x13 [nfsd]
[25525.916295] [<ffffffffa05d4748>] ? nfsd_vfs_read+0x242/0x34b [nfsd]
[25525.916295] [<ffffffffa05d4e4d>] ? nfsd_read+0xa5/0xc4 [nfsd]
[25525.916295] [<ffffffffa05db583>] ? nfsd3_proc_read+0xd6/0x114 [nfsd]
[25525.916295] [<ffffffffa05d0334>] ? nfsd_dispatch+0xdf/0x1b5 [nfsd]
[25525.916295] [<ffffffffa02e0596>] ? svc_process+0x40d/0x60b [sunrpc]
[25525.916295] [<ffffffffa05d0851>] ? nfsd+0xe8/0x131 [nfsd]
[25525.916295] [<ffffffffa05d0769>] ? nfsd+0x0/0x131 [nfsd]
[25525.916295] [<ffffffff8105b541>] ? kthread+0x75/0x7d
[25525.916295] [<ffffffff810097e4>] ? kernel_thread_helper+0x4/0x10
[25525.916295] [<ffffffff8105b4cc>] ? kthread+0x0/0x7d
[25525.916295] [<ffffffff810097e0>] ? kernel_thread_helper+0x0/0x10
[25525.916295] nfsd D 0000000100573eb1 0 3389 2 0x00000000
[25525.916295] ffff88042a71e630 0000000000000046 0000000100000000 ffff880400000000
[25525.916295] ffff88042e06b680 00000000000154c0 00000000000154c0 00000000000154c0
[25525.916295] ffff880421387fd8 00000000000154c0 ffff88042a71e630 ffff880421387fd8
[25525.916295] Call Trace:
[25525.916295] [<ffffffff8100e806>] ? read_tsc+0x5/0x16
[25525.916295] [<ffffffff810aefe5>] ? sync_page+0x0/0x45
[25525.916295] [<ffffffff813059f0>] ? io_schedule+0x6b/0xaa
[25525.916295] [<ffffffff810af026>] ? sync_page+0x41/0x45
[25525.916295] [<ffffffff81305f39>] ? __wait_on_bit+0x3e/0x6f
[25525.916295] [<ffffffff810af161>] ? wait_on_page_bit+0x6a/0x70
[25525.916295] [<ffffffff8105b9bf>] ? wake_bit_function+0x0/0x2e
[25525.916295] [<ffffffff810b6de4>] ? pagevec_lookup_tag+0x18/0x1f
[25525.916295] [<ffffffff810af3b5>] ? filemap_fdatawait_range+0x96/0x14a
[25525.916295] [<ffffffff810af4d2>] ? __filemap_fdatawrite_range+0x48/0x4d
[25525.916295] [<ffffffff810af514>] ? filemap_write_and_wait_range+0x3d/0x50
[25525.916295] [<ffffffffa058a5d1>] ? xfs_file_aio_write+0x6ab/0x762 [xfs]
[25525.916295] [<ffffffffa056af1c>] ? xfs_iget+0x51f/0x58b [xfs]
[25525.916295] [<ffffffff810e0afe>] ? __slab_free+0x1c/0x250
[25525.916295] [<ffffffffa0589f26>] ? xfs_file_aio_write+0x0/0x762 [xfs]
[25525.916295] [<ffffffff810e9324>] ? do_sync_readv_writev+0x9a/0xd5
[25525.916295] [<ffffffffa056a4f0>] ? xfs_iunlock+0x7b/0xb7 [xfs]
[25525.916295] [<ffffffff810e0f8b>] ? __slab_alloc+0xf9/0x3eb
[25525.916295] [<ffffffff810e90d5>] ? copy_from_user+0x18/0x30
[25525.916295] [<ffffffff810e9e43>] ? rw_copy_check_uvector+0x6a/0xe1
[25525.916295] [<ffffffff810e9f5e>] ? do_readv_writev+0xa4/0x118
[25525.916295] [<ffffffffa05d8ae5>] ? nfsd_setuser+0x1a2/0x1e5 [nfsd]
[25525.916295] [<ffffffffa05d2ff4>] ? nfsd_setuser_and_check_port+0x76/0x92 [nfsd]
[25525.916295] [<ffffffffa05d4300>] ? nfsd_vfs_write+0x11b/0x321 [nfsd]
[25525.916295] [<ffffffff810e81f8>] ? __dentry_open+0x184/0x28f
[25525.916295] [<ffffffffa05d4d87>] ? nfsd_write+0xbb/0xdc [nfsd]
[25525.916295] [<ffffffffa05d8e8a>] ? nfsd_cache_lookup+0x26e/0x37d [nfsd]
[25525.916295] [<ffffffffa05db492>] ? nfsd3_proc_write+0xcf/0xea [nfsd]
[25525.916295] [<ffffffffa05d0334>] ? nfsd_dispatch+0xdf/0x1b5 [nfsd]
[25525.916295] [<ffffffffa02e0596>] ? svc_process+0x40d/0x60b [sunrpc]
[25525.916295] [<ffffffffa05d0851>] ? nfsd+0xe8/0x131 [nfsd]
[25525.916295] [<ffffffffa05d0769>] ? nfsd+0x0/0x131 [nfsd]
[25525.916295] [<ffffffff8105b541>] ? kthread+0x75/0x7d
[25525.916295] [<ffffffff810097e4>] ? kernel_thread_helper+0x4/0x10
[25525.916295] [<ffffffff8105b4cc>] ? kthread+0x0/0x7d
[25525.916295] [<ffffffff810097e0>] ? kernel_thread_helper+0x0/0x10
[25525.916295] nfsd D 0000000100574314 0 3390 2 0x00000000
[25525.916295] ffff88042a71f3d0 0000000000000046 00000000000002c9 ffff880400000000
[25525.916295] ffffffff81632020 00000000000154c0 00000000000154c0 00000000000154c0
[25525.916295] ffff880420c81fd8 00000000000154c0 ffff88042a71f3d0 ffff880420c81fd8
[25525.916295] Call Trace:
[25525.916295] [<ffffffff81306bdd>] ? rwsem_down_failed_common+0x97/0xcb
[25525.916295] [<ffffffff81306c58>] ? rwsem_down_read_failed+0x22/0x2b
[25525.916295] [<ffffffff8118f714>] ? call_rwsem_down_read_failed+0x14/0x30
[25525.916295] [<ffffffff8130656f>] ? down_read+0x17/0x19
[25525.916295] [<ffffffffa056a66a>] ? xfs_ilock+0x5c/0x97 [xfs]
[25525.916295] [<ffffffffa056acd9>] ? xfs_iget+0x2dc/0x58b [xfs]
[25525.916295] [<ffffffff810e0afe>] ? __slab_free+0x1c/0x250
[25525.916295] [<ffffffff812491ca>] ? release_sock+0x19/0xf3
[25525.916295] [<ffffffffa05d2eb4>] ? nfsd_acceptable+0x0/0xca [nfsd]
[25525.916295] [<ffffffffa05890d0>] ? xfs_nfs_get_inode+0x34/0x83 [xfs]
[25525.916295] [<ffffffffa02e858b>] ? cache_check+0x3f/0x29c [sunrpc]
[25525.916295] [<ffffffffa05891b9>] ? xfs_fs_fh_to_dentry+0x67/0x71 [xfs]
[25525.916295] [<ffffffffa01a2589>] ? exportfs_decode_fh+0x46/0x215 [exportfs]
[25525.916295] [<ffffffffa02e858b>] ? cache_check+0x3f/0x29c [sunrpc]
[25525.916295] [<ffffffffa05d7687>] ? exp_find_key+0x6a/0x7d [nfsd]
[25525.916295] [<ffffffff812457b6>] ? sock_recvmsg+0x86/0x9e
[25525.916295] [<ffffffffa05d32b1>] ? fh_verify+0x2a1/0x4ec [nfsd]
[25525.916295] [<ffffffffa05d495e>] ? nfsd_open+0x1a/0x13b [nfsd]
[25525.916295] [<ffffffffa05d4d5a>] ? nfsd_write+0x8e/0xdc [nfsd]
[25525.916295] [<ffffffff810e0afe>] ? __slab_free+0x1c/0x250
[25525.916295] [<ffffffffa05d8e8a>] ? nfsd_cache_lookup+0x26e/0x37d [nfsd]
[25525.916295] [<ffffffffa05db492>] ? nfsd3_proc_write+0xcf/0xea [nfsd]
[25525.916295] [<ffffffffa05d0334>] ? nfsd_dispatch+0xdf/0x1b5 [nfsd]
[25525.916295] [<ffffffffa02e0596>] ? svc_process+0x40d/0x60b [sunrpc]
[25525.916295] [<ffffffffa05d0851>] ? nfsd+0xe8/0x131 [nfsd]
[25525.916295] [<ffffffffa05d0769>] ? nfsd+0x0/0x131 [nfsd]
[25525.916295] [<ffffffff8105b541>] ? kthread+0x75/0x7d
[25525.916295] [<ffffffff810097e4>] ? kernel_thread_helper+0x4/0x10
[25525.916295] [<ffffffff8105b4cc>] ? kthread+0x0/0x7d
[25525.916295] [<ffffffff810097e0>] ? kernel_thread_helper+0x0/0x10
[25525.916295] nfsd D 0000000100573eb1 0 3391 2 0x00000000
[25525.916295] ffff88042a71caf0 0000000000000046 ffff88042a443000 ffffffff00000000
[25525.916295] ffff88042e0d8da0 00000000000154c0 00000000000154c0 00000000000154c0
[25525.916295] ffff880420d99fd8 00000000000154c0 ffff88042a71caf0 ffff880420d99fd8
[25525.916295] Call Trace:
[25525.916295] [<ffffffff8100e806>] ? read_tsc+0x5/0x16
[25525.916295] [<ffffffff810aefe5>] ? sync_page+0x0/0x45
[25525.916295] [<ffffffff813059f0>] ? io_schedule+0x6b/0xaa
[25525.916295] [<ffffffff810af026>] ? sync_page+0x41/0x45
[25525.916295] [<ffffffff81305f39>] ? __wait_on_bit+0x3e/0x6f
[25525.916295] [<ffffffff810af161>] ? wait_on_page_bit+0x6a/0x70
[25525.916295] [<ffffffff8105b9bf>] ? wake_bit_function+0x0/0x2e
[25525.916295] [<ffffffff810b6de4>] ? pagevec_lookup_tag+0x18/0x1f
[25525.916295] [<ffffffff810af3b5>] ? filemap_fdatawait_range+0x96/0x14a
[25525.916295] [<ffffffff810af4d2>] ? __filemap_fdatawrite_range+0x48/0x4d
[25525.916295] [<ffffffff810af514>] ? filemap_write_and_wait_range+0x3d/0x50
[25525.916295] [<ffffffffa058a5d1>] ? xfs_file_aio_write+0x6ab/0x762 [xfs]
[25525.916295] [<ffffffffa056af1c>] ? xfs_iget+0x51f/0x58b [xfs]
[25525.916295] [<ffffffff810e0afe>] ? __slab_free+0x1c/0x250
[25525.916295] [<ffffffffa0589f26>] ? xfs_file_aio_write+0x0/0x762 [xfs]
[25525.916295] [<ffffffff810e9324>] ? do_sync_readv_writev+0x9a/0xd5
[25525.916295] [<ffffffffa056a4f0>] ? xfs_iunlock+0x7b/0xb7 [xfs]
[25525.916295] [<ffffffffa01a2602>] ? exportfs_decode_fh+0xbf/0x215 [exportfs]
[25525.916295] [<ffffffff810e90d5>] ? copy_from_user+0x18/0x30
[25525.916295] [<ffffffff810e9e43>] ? rw_copy_check_uvector+0x6a/0xe1
[25525.916295] [<ffffffff810e9f5e>] ? do_readv_writev+0xa4/0x118
[25525.916295] [<ffffffffa05d8ae5>] ? nfsd_setuser+0x1a2/0x1e5 [nfsd]
[25525.916295] [<ffffffffa05d2ff4>] ? nfsd_setuser_and_check_port+0x76/0x92 [nfsd]
[25525.916295] [<ffffffffa05d4300>] ? nfsd_vfs_write+0x11b/0x321 [nfsd]
[25525.916295] [<ffffffff810e81f8>] ? __dentry_open+0x184/0x28f
[25525.916295] [<ffffffffa05d4d87>] ? nfsd_write+0xbb/0xdc [nfsd]
[25525.916295] [<ffffffffa05d8e8a>] ? nfsd_cache_lookup+0x26e/0x37d [nfsd]
[25525.916295] [<ffffffffa05db492>] ? nfsd3_proc_write+0xcf/0xea [nfsd]
[25525.916295] [<ffffffffa05d0334>] ? nfsd_dispatch+0xdf/0x1b5 [nfsd]
[25525.916295] [<ffffffffa02e0596>] ? svc_process+0x40d/0x60b [sunrpc]
[25525.916295] [<ffffffffa05d0851>] ? nfsd+0xe8/0x131 [nfsd]
[25525.916295] [<ffffffffa05d0769>] ? nfsd+0x0/0x131 [nfsd]
[25525.916295] [<ffffffff8105b541>] ? kthread+0x75/0x7d
[25525.916295] [<ffffffff810097e4>] ? kernel_thread_helper+0x4/0x10
[25525.916295] [<ffffffff8105b4cc>] ? kthread+0x0/0x7d
[25525.916295] [<ffffffff810097e0>] ? kernel_thread_helper+0x0/0x10
[25525.916295] nfsd D 00000001005740cf 0 3392 2 0x00000000
[25525.916295] ffff88042a71d890 0000000000000046 ffff880420e93678 ffff880420bdb930
[25525.916295] ffffffff81632020 00000000000154c0 00000000000154c0 00000000000154c0
[25525.916295] ffff880420e93fd8 00000000000154c0 ffff88042a71d890 ffff880420e93fd8
[25525.916295] Call Trace:
[25525.916295] [<ffffffff810dfe2f>] ? virt_to_head_page+0x9/0x2a
[25525.916295] [<ffffffff81305d07>] ? schedule_timeout+0x2d/0xd7
[25525.916295] [<ffffffff81306645>] ? __down_common+0x92/0xe3
[25525.916295] [<ffffffffa057a3bd>] ? xfs_getsb+0x25/0x32 [xfs]
[25525.916295] [<ffffffff8105f54c>] ? down+0x27/0x37
[25525.916295] [<ffffffffa05877ab>] ? xfs_buf_lock+0x73/0xaa [xfs]
[25525.916295] [<ffffffffa057a3bd>] ? xfs_getsb+0x25/0x32 [xfs]
[25525.916295] [<ffffffffa057feda>] ? xfs_trans_getsb+0x63/0xb2 [xfs]
[25525.916295] [<ffffffffa057dfcd>] ? xfs_trans_apply_sb_deltas+0x12/0x1da [xfs]
[25525.916295] [<ffffffffa057e9a8>] ? _xfs_trans_commit+0x50/0x221 [xfs]
[25525.916295] [<ffffffffa056b179>] ? xfs_iext_get_ext+0x40/0x66 [xfs]
[25525.916295] [<ffffffffa0570794>] ? xfs_iomap_write_allocate+0x270/0x36c [xfs]
[25525.916295] [<ffffffffa056a4f0>] ? xfs_iunlock+0x7b/0xb7 [xfs]
[25525.916295] [<ffffffffa0585356>] ? xfs_map_blocks+0x37/0x3e [xfs]
[25525.916295] [<ffffffffa05863d5>] ? xfs_page_state_convert+0x268/0x50e [xfs]
[25525.916295] [<ffffffffa0586843>] ? xfs_vm_writepage+0xef/0x12c [xfs]
[25525.916295] [<ffffffff810b548a>] ? __writepage+0xa/0x21
[25525.916295] [<ffffffff810b6504>] ? write_cache_pages+0x1d8/0x2ec
[25525.916295] [<ffffffff810b5480>] ? __writepage+0x0/0x21
[25525.916295] [<ffffffff81108dc5>] ? block_write_end+0x4e/0x57
[25525.916295] [<ffffffff81108e1a>] ? generic_write_end+0x4c/0x6d
[25525.916295] [<ffffffff810af4d2>] ? __filemap_fdatawrite_range+0x48/0x4d
[25525.916295] [<ffffffff810af4fe>] ? filemap_write_and_wait_range+0x27/0x50
[25525.916295] [<ffffffffa058a5d1>] ? xfs_file_aio_write+0x6ab/0x762 [xfs]
[25525.916295] [<ffffffffa056af1c>] ? xfs_iget+0x51f/0x58b [xfs]
[25525.916295] [<ffffffff810e0b47>] ? __slab_free+0x65/0x250
[25525.916295] [<ffffffffa0589f26>] ? xfs_file_aio_write+0x0/0x762 [xfs]
[25525.916295] [<ffffffff810e9324>] ? do_sync_readv_writev+0x9a/0xd5
[25525.916295] [<ffffffffa056a4f0>] ? xfs_iunlock+0x7b/0xb7 [xfs]
[25525.916295] [<ffffffff810615c7>] ? groups_alloc+0x35/0xb4
[25525.916295] [<ffffffff810e90d5>] ? copy_from_user+0x18/0x30
[25525.916295] [<ffffffff810e9e43>] ? rw_copy_check_uvector+0x6a/0xe1
[25525.916295] [<ffffffff810e9f5e>] ? do_readv_writev+0xa4/0x118
[25525.916295] [<ffffffffa05d8ae5>] ? nfsd_setuser+0x1a2/0x1e5 [nfsd]
[25525.916295] [<ffffffff810e09d3>] ? get_partial_node+0x15/0x82
[25525.916295] [<ffffffff810e0f8b>] ? __slab_alloc+0xf9/0x3eb
[25525.916295] [<ffffffff810ea823>] ? get_empty_filp+0x70/0x12f
[25525.916295] [<ffffffffa05d4300>] ? nfsd_vfs_write+0x11b/0x321 [nfsd]
[25525.916295] [<ffffffff810e81f8>] ? __dentry_open+0x184/0x28f
[25525.916295] [<ffffffffa05d4d87>] ? nfsd_write+0xbb/0xdc [nfsd]
[25525.916295] [<ffffffffa05d8e8a>] ? nfsd_cache_lookup+0x26e/0x37d [nfsd]
[25525.916295] [<ffffffffa05db492>] ? nfsd3_proc_write+0xcf/0xea [nfsd]
[25525.916295] [<ffffffffa05d0334>] ? nfsd_dispatch+0xdf/0x1b5 [nfsd]
[25525.916295] [<ffffffffa02e0596>] ? svc_process+0x40d/0x60b [sunrpc]
[25525.916295] [<ffffffffa05d0851>] ? nfsd+0xe8/0x131 [nfsd]
[25525.916295] [<ffffffffa05d0769>] ? nfsd+0x0/0x131 [nfsd]
[25525.916295] [<ffffffff8105b541>] ? kthread+0x75/0x7d
[25525.916295] [<ffffffff810097e4>] ? kernel_thread_helper+0x4/0x10
[25525.916295] [<ffffffff8105b4cc>] ? kthread+0x0/0x7d
[25525.916295] [<ffffffff810097e0>] ? kernel_thread_helper+0x0/0x10
[25525.916295] nfsd D 0000000100573eb1 0 3393 2 0x00000000
[25525.916295] ffff88042a71df60 0000000000000046 ffff88042a442000 ffffffff00000000
[25525.916295] ffffffff81632020 00000000000154c0 00000000000154c0 00000000000154c0
[25525.916295] ffff880420f8dfd8 00000000000154c0 ffff88042a71df60 ffff880420f8dfd8
[25525.916295] Call Trace:
[25525.916295] [<ffffffff8100e806>] ? read_tsc+0x5/0x16
[25525.916295] [<ffffffff810aefe5>] ? sync_page+0x0/0x45
[25525.916295] [<ffffffff813059f0>] ? io_schedule+0x6b/0xaa
[25525.916295] [<ffffffff810af026>] ? sync_page+0x41/0x45
[25525.916295] [<ffffffff81305f39>] ? __wait_on_bit+0x3e/0x6f
[25525.916295] [<ffffffff810af161>] ? wait_on_page_bit+0x6a/0x70
[25525.916295] [<ffffffff8105b9bf>] ? wake_bit_function+0x0/0x2e
[25525.916295] [<ffffffff810b6de4>] ? pagevec_lookup_tag+0x18/0x1f
[25525.916295] [<ffffffff810af3b5>] ? filemap_fdatawait_range+0x96/0x14a
[25525.916295] [<ffffffff810af4d2>] ? __filemap_fdatawrite_range+0x48/0x4d
[25525.916295] [<ffffffff810af514>] ? filemap_write_and_wait_range+0x3d/0x50
[25525.916295] [<ffffffffa058a5d1>] ? xfs_file_aio_write+0x6ab/0x762 [xfs]
[25525.916295] [<ffffffffa056af1c>] ? xfs_iget+0x51f/0x58b [xfs]
[25525.916295] [<ffffffff810e0afe>] ? __slab_free+0x1c/0x250
[25525.916295] [<ffffffffa0589f26>] ? xfs_file_aio_write+0x0/0x762 [xfs]
[25525.916295] [<ffffffff810e9324>] ? do_sync_readv_writev+0x9a/0xd5
[25525.916295] [<ffffffffa056a4f0>] ? xfs_iunlock+0x7b/0xb7 [xfs]
[25525.916295] [<ffffffffa01a2602>] ? exportfs_decode_fh+0xbf/0x215 [exportfs]
[25525.916295] [<ffffffff810e90d5>] ? copy_from_user+0x18/0x30
[25525.916295] [<ffffffff810e9e43>] ? rw_copy_check_uvector+0x6a/0xe1
[25525.916295] [<ffffffff810e9f5e>] ? do_readv_writev+0xa4/0x118
[25525.916295] [<ffffffffa05d8ae5>] ? nfsd_setuser+0x1a2/0x1e5 [nfsd]
[25525.916295] [<ffffffffa05d2ff4>] ? nfsd_setuser_and_check_port+0x76/0x92 [nfsd]
[25525.916295] [<ffffffffa05d4300>] ? nfsd_vfs_write+0x11b/0x321 [nfsd]
[25525.916295] [<ffffffff810e81f8>] ? __dentry_open+0x184/0x28f
[25525.916295] [<ffffffffa05d4d87>] ? nfsd_write+0xbb/0xdc [nfsd]
[25525.916295] [<ffffffffa05d8e8a>] ? nfsd_cache_lookup+0x26e/0x37d [nfsd]
[25525.916295] [<ffffffffa05db492>] ? nfsd3_proc_write+0xcf/0xea [nfsd]
[25525.916295] [<ffffffffa05d0334>] ? nfsd_dispatch+0xdf/0x1b5 [nfsd]
[25525.916295] [<ffffffffa02e0596>] ? svc_process+0x40d/0x60b [sunrpc]
[25525.916295] [<ffffffffa05d0851>] ? nfsd+0xe8/0x131 [nfsd]
[25525.916295] [<ffffffffa05d0769>] ? nfsd+0x0/0x131 [nfsd]
[25525.916295] [<ffffffff8105b541>] ? kthread+0x75/0x7d
[25525.916295] [<ffffffff810097e4>] ? kernel_thread_helper+0x4/0x10
[25525.916295] [<ffffffff8105b4cc>] ? kthread+0x0/0x7d
[25525.916295] [<ffffffff810097e0>] ? kernel_thread_helper+0x0/0x10
[25525.916295] nfsd D 0000000100573eb1 0 3394 2 0x00000000
[25525.916295] ffff88042a719b40 0000000000000046 ffff88042a443800 ffffffff00000000
[25525.916295] ffff88042e06ed00 00000000000154c0 00000000000154c0 00000000000154c0
[25525.916295] ffff8804208affd8 00000000000154c0 ffff88042a719b40 ffff8804208affd8
[25525.916295] Call Trace:
[25525.916295] [<ffffffff8100e806>] ? read_tsc+0x5/0x16
[25525.916295] [<ffffffff810aefe5>] ? sync_page+0x0/0x45
[25525.916295] [<ffffffff813059f0>] ? io_schedule+0x6b/0xaa
[25525.916295] [<ffffffff810af026>] ? sync_page+0x41/0x45
[25525.916295] [<ffffffff81305f39>] ? __wait_on_bit+0x3e/0x6f
[25525.916295] [<ffffffff810af161>] ? wait_on_page_bit+0x6a/0x70
[25525.916295] [<ffffffff8105b9bf>] ? wake_bit_function+0x0/0x2e
[25525.916295] [<ffffffff810b6de4>] ? pagevec_lookup_tag+0x18/0x1f
[25525.916295] [<ffffffff810af3b5>] ? filemap_fdatawait_range+0x96/0x14a
[25525.916295] [<ffffffff810af4d2>] ? __filemap_fdatawrite_range+0x48/0x4d
[25525.916295] [<ffffffff810af514>] ? filemap_write_and_wait_range+0x3d/0x50
[25525.916295] [<ffffffffa058a5d1>] ? xfs_file_aio_write+0x6ab/0x762 [xfs]
[25525.916295] [<ffffffff8118bd35>] ? rwsem_wake+0x38/0x3f
[25525.916295] [<ffffffff8118f767>] ? call_rwsem_wake+0x17/0x30
[25525.916295] [<ffffffffa0589f26>] ? xfs_file_aio_write+0x0/0x762 [xfs]
[25525.916295] [<ffffffff810e9324>] ? do_sync_readv_writev+0x9a/0xd5
[25525.916295] [<ffffffffa056a4f0>] ? xfs_iunlock+0x7b/0xb7 [xfs]
[25525.916295] [<ffffffff810615c7>] ? groups_alloc+0x35/0xb4
[25525.916295] [<ffffffff810e90d5>] ? copy_from_user+0x18/0x30
[25525.916295] [<ffffffff810e9e43>] ? rw_copy_check_uvector+0x6a/0xe1
[25525.916295] [<ffffffff810e9f5e>] ? do_readv_writev+0xa4/0x118
[25525.916295] [<ffffffffa05d8ae5>] ? nfsd_setuser+0x1a2/0x1e5 [nfsd]
[25525.916295] [<ffffffffa05d2ff4>] ? nfsd_setuser_and_check_port+0x76/0x92 [nfsd]
[25525.916295] [<ffffffffa05d4300>] ? nfsd_vfs_write+0x11b/0x321 [nfsd]
[25525.916295] [<ffffffff810e81f8>] ? __dentry_open+0x184/0x28f
[25525.916295] [<ffffffffa05d4d87>] ? nfsd_write+0xbb/0xdc [nfsd]
[25525.916295] [<ffffffffa05d8e8a>] ? nfsd_cache_lookup+0x26e/0x37d [nfsd]
[25525.916295] [<ffffffffa05db492>] ? nfsd3_proc_write+0xcf/0xea [nfsd]
[25525.916295] [<ffffffffa05d0334>] ? nfsd_dispatch+0xdf/0x1b5 [nfsd]
[25525.916295] [<ffffffffa02e0596>] ? svc_process+0x40d/0x60b [sunrpc]
[25525.916295] [<ffffffffa05d0851>] ? nfsd+0xe8/0x131 [nfsd]
[25525.916295] [<ffffffffa05d0769>] ? nfsd+0x0/0x131 [nfsd]
[25525.916295] [<ffffffff8105b541>] ? kthread+0x75/0x7d
[25525.916295] [<ffffffff810097e4>] ? kernel_thread_helper+0x4/0x10
[25525.916295] [<ffffffff8105b4cc>] ? kthread+0x0/0x7d
[25525.916295] [<ffffffff810097e0>] ? kernel_thread_helper+0x0/0x10
[25525.916295] nfsd D 0000000100573ec5 0 3395 2 0x00000000
[25525.916295] ffff88042a718da0 0000000000000046 0000000000000000 0000020c00000000
[25525.916295] ffff88042e06ed00 00000000000154c0 00000000000154c0 00000000000154c0
[25525.916295] ffff8804209a9fd8 00000000000154c0 ffff88042a718da0 ffff8804209a9fd8
[25525.916295] Call Trace:
[25525.916295] [<ffffffffa01dc520>] ? raid5_quiesce+0xe3/0x19b [raid456]
[25525.916295] [<ffffffff8103f760>] ? default_wake_function+0x0/0xf
[25525.916295] [<ffffffffa01e1971>] ? make_request+0x31/0x403 [raid456]
[25525.916295] [<ffffffffa057fae4>] ? _xfs_trans_bjoin+0x3b/0x47 [xfs]
[25525.916295] [<ffffffffa057f961>] ? xfs_trans_log_buf+0x8f/0xb5 [xfs]
[25525.916295] [<ffffffffa05570b9>] ? xfs_btree_log_block+0x51/0x7b [xfs]
[25525.916295] [<ffffffffa02767ca>] ? md_make_request+0xd0/0x1d4 [md_mod]
[25525.916295] [<ffffffff8117583c>] ? generic_make_request+0x27a/0x2f0
[25525.916295] [<ffffffff810b09dd>] ? mempool_alloc+0x53/0x106
[25525.916295] [<ffffffff81175983>] ? submit_bio+0xd1/0xed
[25525.916295] [<ffffffffa05874d2>] ? _xfs_buf_ioapply+0x212/0x24f [xfs]
[25525.916295] [<ffffffffa0572ace>] ? xlog_bdstrat+0x3a/0x3e [xfs]
[25525.916295] [<ffffffffa0588430>] ? xfs_buf_iorequest+0x75/0x8e [xfs]
[25525.916295] [<ffffffffa0572ace>] ? xlog_bdstrat+0x3a/0x3e [xfs]
[25525.916295] [<ffffffffa0572ce1>] ? xlog_sync+0x20f/0x326 [xfs]
[25525.916295] [<ffffffffa0573e1a>] ? xlog_write+0x315/0x46e [xfs]
[25525.916295] [<ffffffffa0573fc2>] ? xfs_log_write+0x4f/0x7a [xfs]
[25525.916295] [<ffffffffa057e814>] ? xfs_trans_commit_iclog+0x146/0x28a [xfs]
[25525.916295] [<ffffffff8105f533>] ? down+0xe/0x37
[25525.916295] [<ffffffffa057fae4>] ? _xfs_trans_bjoin+0x3b/0x47 [xfs]
[25525.916295] [<ffffffffa057f961>] ? xfs_trans_log_buf+0x8f/0xb5 [xfs]
[25525.916295] [<ffffffffa057ea32>] ? _xfs_trans_commit+0xda/0x221 [xfs]
[25525.916295] [<ffffffffa0570794>] ? xfs_iomap_write_allocate+0x270/0x36c [xfs]
[25525.916295] [<ffffffffa056a4f0>] ? xfs_iunlock+0x7b/0xb7 [xfs]
[25525.916295] [<ffffffffa0585356>] ? xfs_map_blocks+0x37/0x3e [xfs]
[25525.916295] [<ffffffffa05863d5>] ? xfs_page_state_convert+0x268/0x50e [xfs]
[25525.916295] [<ffffffffa0586843>] ? xfs_vm_writepage+0xef/0x12c [xfs]
[25525.916295] [<ffffffff810b548a>] ? __writepage+0xa/0x21
[25525.916295] [<ffffffff810b6504>] ? write_cache_pages+0x1d8/0x2ec
[25525.916295] [<ffffffff810b5480>] ? __writepage+0x0/0x21
[25525.916295] [<ffffffff810af4d2>] ? __filemap_fdatawrite_range+0x48/0x4d
[25525.916295] [<ffffffff810af4fe>] ? filemap_write_and_wait_range+0x27/0x50
[25525.916295] [<ffffffff81106b66>] ? vfs_fsync_rang
^ permalink raw reply related [flat|nested] 6+ messages in thread
end of thread, other threads:[~2011-02-12 0:41 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2011-01-12 13:04 md raid5 blocked in get_active_stripe after disk failure David Flynn
2011-01-12 14:06 ` David Flynn
2011-01-12 20:33 ` NeilBrown
2011-01-19 13:45 ` David Flynn
2011-02-12 0:41 ` David Flynn
2011-01-12 20:27 ` NeilBrown
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.