* mdadm hang when one subdev error(raid1)
@ 2015-04-17 7:23 席智勇
2015-04-17 9:56 ` hui jiao
0 siblings, 1 reply; 3+ messages in thread
From: 席智勇 @ 2015-04-17 7:23 UTC (permalink / raw)
To: linux-raid
hi all:
I create some raid1-device by mdadm, when one subdev error, all mdadm related operation just hang there, process state was D.
The backgroud is a physical disk was error, so a subdev which is part of the error disk created by device mapper must be errorred, then i did the command 'mdadm --fail' to fail the subdev from the md device, I found it not responsable, just hang there、I tryed 'mdadm --remove', even 'mdadm -D', all hang there. Later, I found not just mdadm operation hang on the problem md device, all mdadm operation on the machine connot be excute.
I wana find out what's the problem is, is it a bug of raid when disk error occur， or a problem of my system, because when i found the mdadm hang, the errored disk(/dev/sdp)just missing from my system, I said the disk was error judging from the error log in raid card log.
Can anyone give me a help？
thanks.
uname -a :Linux **-***-***-** 3.10.45-****-amd64 #1 SMP Tue Jul 1 01:52:20 UTC 2014 x86_64 GNU/Linux
mdadm --version:mdadm - v3.2.5 - 18th May 2012
kern.log:
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561205] kvm D ffff88407f313f40 0 11581 1 0x00000000
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561209] ffff88356e846080 0000000000000082 0000000000000092 ffff881fe2d6a080
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561215] 0000000000013f40 ffff882849dfdfd8 ffff882849dfdfd8 ffff88356e846080
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561219] ffffffff8139958c ffff881cac478000 ffff882849dfdcb0 ffff881cac478290
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561224] Call Trace:
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561233] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561266] [<ffffffffa02efff0>] ? md_write_start+0x131/0x147 [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561271] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561275] [<ffffffffa05c2762>] ? make_request+0x37/0xa63 [raid1]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561284] [<ffffffffa02f4b7e>] ? md_make_request+0xee/0x1df [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561295] [<ffffffffa0006902>] ? dm_request+0x150/0x163 [dm_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561300] [<ffffffff811b3efc>] ? generic_make_request+0x96/0xd5
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561304] [<ffffffff811b4c79>] ? submit_bio+0x10a/0x13b
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561307] [<ffffffff811b6b0a>] ? blkdev_issue_flush+0x86/0xc4
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561311] [<ffffffff8113b298>] ? blkdev_fsync+0x2b/0x37
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561315] [<ffffffff81134a43>] ? do_fsync+0x2b/0x50
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561318] [<ffffffff81134c4f>] ? SyS_fdatasync+0xb/0xf
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561356] [<ffffffff8139ea69>] ? system_call_fastpath+0x16/0x1b
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561367] INFO: task md52_raid1:39767 blocked for more than 120 seconds.
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.563976] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.566971] md52_raid1 D ffff88407f233f40 0 39767 2 0x00000000
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.566975] ffff883e60fa9810 0000000000000046 ffff883e60fa9810 ffff881fe2d620c0
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.566981] 0000000000013f40 ffff883d61523fd8 ffff883d61523fd8 ffff883e60fa9810
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.566985] ffffffff8139958c ffff883d61523c60 ffff881cac478000 ffff881cac478290
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.566990] Call Trace:
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.566995] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567007] [<ffffffffa02f4d65>] ? md_super_wait+0x69/0x7f [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567011] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567019] [<ffffffffa02f5131>] ? md_update_sb+0x3b6/0x4b8 [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567023] [<ffffffff813995cb>] ? _raw_spin_lock_irqsave+0x14/0x35
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567030] [<ffffffffa02f59ee>] ? md_check_recovery+0x1c6/0x3d1 [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567034] [<ffffffffa05c31cc>] ? raid1d+0x3e/0xb22 [raid1]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567038] [<ffffffff813988db>] ? __schedule+0x4e7/0x53d
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567042] [<ffffffff813978a3>] ? schedule_timeout+0x2c/0x123
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567045] [<ffffffff813995cb>] ? _raw_spin_lock_irqsave+0x14/0x35
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567048] [<ffffffff813995cb>] ? _raw_spin_lock_irqsave+0x14/0x35
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567088] [<ffffffffa02f02ed>] ? md_thread+0x114/0x132 [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567092] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567100] [<ffffffffa02f01d9>] ? signal_pending+0x10/0x10 [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567107] [<ffffffffa02f01d9>] ? signal_pending+0x10/0x10 [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567110] [<ffffffff81059295>] ? kthread+0x81/0x89
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567113] [<ffffffff81059214>] ? __kthread_parkme+0x5d/0x5d
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567117] [<ffffffff8139e9bc>] ? ret_from_fork+0x7c/0xb0
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567120] [<ffffffff81059214>] ? __kthread_parkme+0x5d/0x5d
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567125] INFO: task kvm:19762 blocked for more than 120 seconds.
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.570192] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573492] kvm D ffff88407f273f40 0 19762 1 0x00000000
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573496] ffff883fc90f97d0 0000000000000082 0000000000011200 ffff881fe2d64040
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573502] 0000000000013f40 ffff882aad01dfd8 ffff882aad01dfd8 ffff883fc90f97d0
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573507] ffffffff8139958c ffff881fb5475800 ffff882aad01dcb0 ffff881fb5475a90
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573512] Call Trace:
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573517] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573529] [<ffffffffa02efff0>] ? md_write_start+0x131/0x147 [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573545] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573549] [<ffffffffa05c2762>] ? make_request+0x37/0xa63 [raid1]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573557] [<ffffffffa000679f>] ? __split_and_process_bio+0x40d/0x420 [dm_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573561] [<ffffffff81100cbd>] ? ____cache_alloc+0x25d/0x293
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573569] [<ffffffffa02f4b7e>] ? md_make_request+0xee/0x1df [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573576] [<ffffffffa0006902>] ? dm_request+0x150/0x163 [dm_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573580] [<ffffffff811b3efc>] ? generic_make_request+0x96/0xd5
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573583] [<ffffffff811b4c79>] ? submit_bio+0x10a/0x13b
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573586] [<ffffffff811b6b0a>] ? blkdev_issue_flush+0x86/0xc4
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573590] [<ffffffff8113b298>] ? blkdev_fsync+0x2b/0x37
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573593] [<ffffffff81134a43>] ? do_fsync+0x2b/0x50
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573596] [<ffffffff81134c4f>] ? SyS_fdatasync+0xb/0xf
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573600] [<ffffffff8139ea69>] ? system_call_fastpath+0x16/0x1b
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573603] INFO: task kvm:9147 blocked for more than 120 seconds.
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.576936] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580590] kvm D ffff88407f3f3f40 0 9147 1 0x00000000
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580594] ffff883fd9bfd080 0000000000000082 0000000000000096 ffff881fe2db5040
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580631] 0000000000013f40 ffff882da06a1fd8 ffff882da06a1fd8 ffff883fd9bfd080
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580635] ffffffff8139958c ffff881fb5475800 ffff882da06a1820 ffff881fb5475a90
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580648] Call Trace:
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580653] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580666] [<ffffffffa02efff0>] ? md_write_start+0x131/0x147 [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580670] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580674] [<ffffffffa05c2762>] ? make_request+0x37/0xa63 [raid1]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580678] [<ffffffff81399449>] ? _raw_read_lock_irqsave+0x21/0x2a
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580685] [<ffffffffa000679f>] ? __split_and_process_bio+0x40d/0x420 [dm_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580689] [<ffffffff81102005>] ? kmem_cache_alloc+0xe1/0x154
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580697] [<ffffffffa02f4b7e>] ? md_make_request+0xee/0x1df [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580704] [<ffffffffa0006902>] ? dm_request+0x150/0x163 [dm_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580708] [<ffffffff811b3efc>] ? generic_make_request+0x96/0xd5
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580712] [<ffffffff811b4c79>] ? submit_bio+0x10a/0x13b
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580716] [<ffffffff8113ce89>] ? dio_bio_submit+0x68/0x88
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580719] [<ffffffff8113dc35>] ? do_blockdev_direct_IO+0x957/0xae8
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580723] [<ffffffff8113aef1>] ? I_BDEV+0x8/0x8
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580752] [<ffffffff810c800a>] ? generic_file_direct_write+0xe3/0x14a
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580755] [<ffffffff810c818c>] ? __generic_file_aio_write+0x11b/0x1ff
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580759] [<ffffffff8113b617>] ? blkdev_aio_write+0x44/0x93
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580763] [<ffffffff811120b3>] ? do_sync_write+0x55/0x7c
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580767] [<ffffffff81112ab0>] ? vfs_write+0x9d/0x103
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580770] [<ffffffff81112eb9>] ? SyS_pwrite64+0x61/0x87
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580773] [<ffffffff8139ea69>] ? system_call_fastpath+0x16/0x1b
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580778] INFO: task md42_raid1:36156 blocked for more than 120 seconds.
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.584560] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588685] md42_raid1 D ffff88207fa33f40 0 36156 2 0x00000000
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588690] ffff881d0b324080 0000000000000046 ffff881d0b324080 ffff881fe2d62810
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588695] 0000000000013f40 ffff881fe1157fd8 ffff881fe1157fd8 ffff881d0b324080
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588699] ffffffff8139958c ffff881fe1157c60 ffff881fb5475800 ffff881fb5475a90
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588703] Call Trace:
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588710] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588734] [<ffffffffa02f4d65>] ? md_super_wait+0x69/0x7f [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588738] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588745] [<ffffffffa02f5131>] ? md_update_sb+0x3b6/0x4b8 [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588750] [<ffffffff8100c02f>] ? load_TLS+0x7/0xa
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588758] [<ffffffffa02f59ee>] ? md_check_recovery+0x1c6/0x3d1 [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588761] [<ffffffffa05c31cc>] ? raid1d+0x3e/0xb22 [raid1]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588766] [<ffffffff81049389>] ? lock_timer_base.isra.35+0x23/0x48
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588769] [<ffffffff810490d4>] ? detach_if_pending+0x18/0x6c
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588772] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588775] [<ffffffff810494ae>] ? try_to_del_timer_sync+0x4e/0x59
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588778] [<ffffffff810494e0>] ? del_timer_sync+0x27/0x44
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588781] [<ffffffff8139796c>] ? schedule_timeout+0xf5/0x123
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588784] [<ffffffff813995cb>] ? _raw_spin_lock_irqsave+0x14/0x35
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588787] [<ffffffff813995cb>] ? _raw_spin_lock_irqsave+0x14/0x35
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588794] [<ffffffffa02f02ed>] ? md_thread+0x114/0x132 [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588797] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588804] [<ffffffffa02f01d9>] ? signal_pending+0x10/0x10 [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588810] [<ffffffffa02f01d9>] ? signal_pending+0x10/0x10 [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588813] [<ffffffff81059295>] ? kthread+0x81/0x89
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588816] [<ffffffff81059214>] ? __kthread_parkme+0x5d/0x5d
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588819] [<ffffffff8139e9bc>] ? ret_from_fork+0x7c/0xb0
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588822] [<ffffffff81059214>] ? __kthread_parkme+0x5d/0x5d
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588834] INFO: task kvm:14262 blocked for more than 120 seconds.
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.592943] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597362] kvm D ffff88207fbb3f40 0 14262 1 0x00000000
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597365] ffff88361cc91080 0000000000000082 0000000000000092 ffff881fe2db3810
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597371] 0000000000013f40 ffff882849fd9fd8 ffff882849fd9fd8 ffff88361cc91080
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597374] ffffffff8139958c ffff883fe1297000 ffff882849fd9cb0 ffff883fe1297290
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597378] Call Trace:
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597383] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597400] [<ffffffffa02efff0>] ? md_write_start+0x131/0x147 [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597403] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597407] [<ffffffffa05c2762>] ? make_request+0x37/0xa63 [raid1]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597415] [<ffffffffa02f4b7e>] ? md_make_request+0xee/0x1df [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597422] [<ffffffffa0006902>] ? dm_request+0x150/0x163 [dm_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597426] [<ffffffff811b3efc>] ? generic_make_request+0x96/0xd5
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597430] [<ffffffff811b4c79>] ? submit_bio+0x10a/0x13b
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597433] [<ffffffff811b6b0a>] ? blkdev_issue_flush+0x86/0xc4
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597438] [<ffffffff8113b298>] ? blkdev_fsync+0x2b/0x37
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597442] [<ffffffff81134a43>] ? do_fsync+0x2b/0x50
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597445] [<ffffffff81134c4f>] ? SyS_fdatasync+0xb/0xf
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597449] [<ffffffff8139ea69>] ? system_call_fastpath+0x16/0x1b
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597452] INFO: task kvm:14266 blocked for more than 120 seconds.
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.602000] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606850] kvm D ffff88207fa53f40 0 14266 1 0x00000000
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606854] ffff883e6f7a3810 0000000000000082 0000000000000096 ffff881fe2d63850
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606860] 0000000000013f40 ffff883c8fc49fd8 ffff883c8fc49fd8 ffff883e6f7a3810
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606865] ffffffff8139958c ffff881b5fa52800 ffff883c8fc49700 ffff881b5fa52a90
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606870] Call Trace:
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606877] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606897] [<ffffffffa02efff0>] ? md_write_start+0x131/0x147 [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606901] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606906] [<ffffffffa05c2762>] ? make_request+0x37/0xa63 [raid1]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606909] [<ffffffff81399449>] ? _raw_read_lock_irqsave+0x21/0x2a
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606917] [<ffffffffa000679f>] ? __split_and_process_bio+0x40d/0x420 [dm_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606925] [<ffffffffa02f4b7e>] ? md_make_request+0xee/0x1df [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606932] [<ffffffffa0006902>] ? dm_request+0x150/0x163 [dm_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606936] [<ffffffff811b3efc>] ? generic_make_request+0x96/0xd5
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606940] [<ffffffff811b4c79>] ? submit_bio+0x10a/0x13b
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606944] [<ffffffff8113ce89>] ? dio_bio_submit+0x68/0x88
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606947] [<ffffffff8113d115>] ? dio_send_cur_page+0x7d/0xa8
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606951] [<ffffffff8113d1e8>] ? submit_page_section+0xa8/0x112
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606954] [<ffffffff8113da99>] ? do_blockdev_direct_IO+0x7bb/0xae8
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606958] [<ffffffff8113aef1>] ? I_BDEV+0x8/0x8
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606961] [<ffffffff8113b0e8>] ? blkdev_direct_IO+0x4e/0x53
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606964] [<ffffffff8113aef1>] ? I_BDEV+0x8/0x8
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606969] [<ffffffff810c800a>] ? generic_file_direct_write+0xe3/0x14a
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606972] [<ffffffff810c818c>] ? __generic_file_aio_write+0x11b/0x1ff
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606976] [<ffffffff8113b617>] ? blkdev_aio_write+0x44/0x93
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606980] [<ffffffff81112038>] ? do_sync_readv_writev+0x50/0x76
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606983] [<ffffffff8113b5d3>] ? bd_may_claim+0x2c/0x2c
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606987] [<ffffffff811130a6>] ? do_readv_writev+0xbf/0x135
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606990] [<ffffffff8113b5d3>] ? bd_may_claim+0x2c/0x2c
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606993] [<ffffffff8111205e>] ? do_sync_readv_writev+0x76/0x76
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606997] [<ffffffff81126eee>] ? fget_light+0x6b/0x7c
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.607000] [<ffffffff81111fbb>] ? fdget+0xe/0x17
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.607004] [<ffffffff811133fe>] ? SyS_pwritev+0x65/0xb0
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.607007] [<ffffffff8139ea69>] ? system_call_fastpath+0x16/0x1b
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.607010] INFO: task kvm:14306 blocked for more than 120 seconds.
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.612040] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617441] kvm D ffff88207fa53f40 0 14306 1 0x00000000
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617445] ffff880eedbaa080 0000000000000082 0000000000011200 ffff881fe2d63850
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617451] 0000000000013f40 ffff881d8e5edfd8 ffff881d8e5edfd8 ffff880eedbaa080
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617451] 0000000000013f40 ffff881d8e5edfd8 ffff881d8e5edfd8 ffff880eedbaa080
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617456] ffffffff8139958c ffff881b5fa52800 ffff881d8e5ed820 ffff881b5fa52a90
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617460] Call Trace:
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617467] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617486] [<ffffffffa02efff0>] ? md_write_start+0x131/0x147 [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617490] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617509] [<ffffffffa05c2762>] ? make_request+0x37/0xa63 [raid1]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617513] [<ffffffff81399449>] ? _raw_read_lock_irqsave+0x21/0x2a
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617531] [<ffffffffa000679f>] ? __split_and_process_bio+0x40d/0x420 [dm_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617536] [<ffffffff81102005>] ? kmem_cache_alloc+0xe1/0x154
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617552] [<ffffffffa02f4b7e>] ? md_make_request+0xee/0x1df [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617559] [<ffffffffa0006902>] ? dm_request+0x150/0x163 [dm_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617563] [<ffffffff811b3efc>] ? generic_make_request+0x96/0xd5
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617573] [<ffffffff811b4c79>] ? submit_bio+0x10a/0x13b
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617577] [<ffffffff8113ce89>] ? dio_bio_submit+0x68/0x88
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617580] [<ffffffff8113dc35>] ? do_blockdev_direct_IO+0x957/0xae8
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617584] [<ffffffff8113aef1>] ? I_BDEV+0x8/0x8
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617587] [<ffffffff8113b0e8>] ? blkdev_direct_IO+0x4e/0x53
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617590] [<ffffffff8113aef1>] ? I_BDEV+0x8/0x8
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617594] [<ffffffff810c800a>] ? generic_file_direct_write+0xe3/0x14a
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617597] [<ffffffff810c818c>] ? __generic_file_aio_write+0x11b/0x1ff
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617601] [<ffffffff8113b617>] ? blkdev_aio_write+0x44/0x93
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617604] [<ffffffff811120b3>] ? do_sync_write+0x55/0x7c
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617608] [<ffffffff81112ab0>] ? vfs_write+0x9d/0x103
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617611] [<ffffffff81112eb9>] ? SyS_pwrite64+0x61/0x87
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617614] [<ffffffff8139ea69>] ? system_call_fastpath+0x16/0x1b
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617629] INFO: task md71_raid1:14478 blocked for more than 120 seconds.
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.622944] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628476] md71_raid1 D ffff88207fad3f40 0 14478 2 0x00000000
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628480] ffff883ed1a09040 0000000000000046 ffff883ed1a09040 ffff881fe2d68850
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628486] 0000000000013f40 ffff883fbd0b7fd8 ffff883fbd0b7fd8 ffff883ed1a09040
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628490] ffffffff8139958c ffff883fbd0b7c60 ffff881b5fa52800 ffff881b5fa52a90
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628498] Call Trace:
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628504] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628522] [<ffffffffa02f4d65>] ? md_super_wait+0x69/0x7f [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628526] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628534] [<ffffffffa02f5131>] ? md_update_sb+0x3b6/0x4b8 [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628542] [<ffffffffa02f59ee>] ? md_check_recovery+0x1c6/0x3d1 [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628546] [<ffffffffa05c31cc>] ? raid1d+0x3e/0xb22 [raid1]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628550] [<ffffffff813988db>] ? __schedule+0x4e7/0x53d
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628553] [<ffffffff813978a3>] ? schedule_timeout+0x2c/0x123
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628557] [<ffffffff813995cb>] ? _raw_spin_lock_irqsave+0x14/0x35
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628560] [<ffffffff813995cb>] ? _raw_spin_lock_irqsave+0x14/0x35
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628568] [<ffffffffa02f02ed>] ? md_thread+0x114/0x132 [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628571] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628578] [<ffffffffa02f01d9>] ? signal_pending+0x10/0x10 [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628585] [<ffffffffa02f01d9>] ? signal_pending+0x10/0x10 [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628588] [<ffffffff81059295>] ? kthread+0x81/0x89
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628591] [<ffffffff81059214>] ? __kthread_parkme+0x5d/0x5d
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628594] [<ffffffff8139e9bc>] ? ret_from_fork+0x7c/0xb0
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628597] [<ffffffff81059214>] ? __kthread_parkme+0x5d/0x5d
-------------
Zhiyong Xi
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: mdadm hang when one subdev error(raid1)
2015-04-17 7:23 mdadm hang when one subdev error(raid1) 席智勇
@ 2015-04-17 9:56 ` hui jiao
2015-04-19 0:06 ` Reply:Re: " 席智勇
0 siblings, 1 reply; 3+ messages in thread
From: hui jiao @ 2015-04-17 9:56 UTC (permalink / raw)
To: 席智勇; +Cc: linux-raid
The md42 is trying to update on-disk superblock, but the operation
doesn't finish. During this peroid, the configuration lock is holded
by md42_raid1 thread, all other operations which need the lock will
wait, such as -D --fail --remove.
what's the status of the subdev now? is it suspended?
check and resume it:
[root@node0 ~]# cat /sys/block/dm-0/dm/suspended
1
[root@node0 ~]# dmsetup resume /dev/dm-0
[root@node0 ~]# cat /sys/block/dm-0/dm/suspended
0
On Fri, Apr 17, 2015 at 3:23 PM, 席智勇 <xizhiyong18@163.com> wrote:
> hi all:
>
> I create some raid1-device by mdadm, when one subdev error, all mdadm related operation just hang there, process state was D.
> The backgroud is a physical disk was error, so a subdev which is part of the error disk created by device mapper must be errorred, then i did the command 'mdadm --fail' to fail the subdev from the md device, I found it not responsable, just hang there、I tryed 'mdadm --remove', even 'mdadm -D', all hang there. Later, I found not just mdadm operation hang on the problem md device, all mdadm operation on the machine connot be excute.
> I wana find out what's the problem is, is it a bug of raid when disk error occur， or a problem of my system, because when i found the mdadm hang, the errored disk(/dev/sdp)just missing from my system, I said the disk was error judging from the error log in raid card log.
> Can anyone give me a help？
> thanks.
>
> uname -a :Linux **-***-***-** 3.10.45-****-amd64 #1 SMP Tue Jul 1 01:52:20 UTC 2014 x86_64 GNU/Linux
> mdadm --version:mdadm - v3.2.5 - 18th May 2012
> kern.log:
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561205] kvm D ffff88407f313f40 0 11581 1 0x00000000
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561209] ffff88356e846080 0000000000000082 0000000000000092 ffff881fe2d6a080
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561215] 0000000000013f40 ffff882849dfdfd8 ffff882849dfdfd8 ffff88356e846080
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561219] ffffffff8139958c ffff881cac478000 ffff882849dfdcb0 ffff881cac478290
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561224] Call Trace:
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561233] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561266] [<ffffffffa02efff0>] ? md_write_start+0x131/0x147 [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561271] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561275] [<ffffffffa05c2762>] ? make_request+0x37/0xa63 [raid1]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561284] [<ffffffffa02f4b7e>] ? md_make_request+0xee/0x1df [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561295] [<ffffffffa0006902>] ? dm_request+0x150/0x163 [dm_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561300] [<ffffffff811b3efc>] ? generic_make_request+0x96/0xd5
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561304] [<ffffffff811b4c79>] ? submit_bio+0x10a/0x13b
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561307] [<ffffffff811b6b0a>] ? blkdev_issue_flush+0x86/0xc4
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561311] [<ffffffff8113b298>] ? blkdev_fsync+0x2b/0x37
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561315] [<ffffffff81134a43>] ? do_fsync+0x2b/0x50
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561318] [<ffffffff81134c4f>] ? SyS_fdatasync+0xb/0xf
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561356] [<ffffffff8139ea69>] ? system_call_fastpath+0x16/0x1b
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561367] INFO: task md52_raid1:39767 blocked for more than 120 seconds.
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.563976] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.566971] md52_raid1 D ffff88407f233f40 0 39767 2 0x00000000
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.566975] ffff883e60fa9810 0000000000000046 ffff883e60fa9810 ffff881fe2d620c0
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.566981] 0000000000013f40 ffff883d61523fd8 ffff883d61523fd8 ffff883e60fa9810
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.566985] ffffffff8139958c ffff883d61523c60 ffff881cac478000 ffff881cac478290
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.566990] Call Trace:
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.566995] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567007] [<ffffffffa02f4d65>] ? md_super_wait+0x69/0x7f [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567011] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567019] [<ffffffffa02f5131>] ? md_update_sb+0x3b6/0x4b8 [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567023] [<ffffffff813995cb>] ? _raw_spin_lock_irqsave+0x14/0x35
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567030] [<ffffffffa02f59ee>] ? md_check_recovery+0x1c6/0x3d1 [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567034] [<ffffffffa05c31cc>] ? raid1d+0x3e/0xb22 [raid1]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567038] [<ffffffff813988db>] ? __schedule+0x4e7/0x53d
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567042] [<ffffffff813978a3>] ? schedule_timeout+0x2c/0x123
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567045] [<ffffffff813995cb>] ? _raw_spin_lock_irqsave+0x14/0x35
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567048] [<ffffffff813995cb>] ? _raw_spin_lock_irqsave+0x14/0x35
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567088] [<ffffffffa02f02ed>] ? md_thread+0x114/0x132 [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567092] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567100] [<ffffffffa02f01d9>] ? signal_pending+0x10/0x10 [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567107] [<ffffffffa02f01d9>] ? signal_pending+0x10/0x10 [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567110] [<ffffffff81059295>] ? kthread+0x81/0x89
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567113] [<ffffffff81059214>] ? __kthread_parkme+0x5d/0x5d
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567117] [<ffffffff8139e9bc>] ? ret_from_fork+0x7c/0xb0
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567120] [<ffffffff81059214>] ? __kthread_parkme+0x5d/0x5d
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567125] INFO: task kvm:19762 blocked for more than 120 seconds.
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.570192] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573492] kvm D ffff88407f273f40 0 19762 1 0x00000000
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573496] ffff883fc90f97d0 0000000000000082 0000000000011200 ffff881fe2d64040
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573502] 0000000000013f40 ffff882aad01dfd8 ffff882aad01dfd8 ffff883fc90f97d0
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573507] ffffffff8139958c ffff881fb5475800 ffff882aad01dcb0 ffff881fb5475a90
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573512] Call Trace:
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573517] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573529] [<ffffffffa02efff0>] ? md_write_start+0x131/0x147 [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573545] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573549] [<ffffffffa05c2762>] ? make_request+0x37/0xa63 [raid1]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573557] [<ffffffffa000679f>] ? __split_and_process_bio+0x40d/0x420 [dm_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573561] [<ffffffff81100cbd>] ? ____cache_alloc+0x25d/0x293
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573569] [<ffffffffa02f4b7e>] ? md_make_request+0xee/0x1df [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573576] [<ffffffffa0006902>] ? dm_request+0x150/0x163 [dm_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573580] [<ffffffff811b3efc>] ? generic_make_request+0x96/0xd5
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573583] [<ffffffff811b4c79>] ? submit_bio+0x10a/0x13b
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573586] [<ffffffff811b6b0a>] ? blkdev_issue_flush+0x86/0xc4
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573590] [<ffffffff8113b298>] ? blkdev_fsync+0x2b/0x37
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573593] [<ffffffff81134a43>] ? do_fsync+0x2b/0x50
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573596] [<ffffffff81134c4f>] ? SyS_fdatasync+0xb/0xf
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573600] [<ffffffff8139ea69>] ? system_call_fastpath+0x16/0x1b
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573603] INFO: task kvm:9147 blocked for more than 120 seconds.
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.576936] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580590] kvm D ffff88407f3f3f40 0 9147 1 0x00000000
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580594] ffff883fd9bfd080 0000000000000082 0000000000000096 ffff881fe2db5040
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580631] 0000000000013f40 ffff882da06a1fd8 ffff882da06a1fd8 ffff883fd9bfd080
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580635] ffffffff8139958c ffff881fb5475800 ffff882da06a1820 ffff881fb5475a90
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580648] Call Trace:
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580653] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580666] [<ffffffffa02efff0>] ? md_write_start+0x131/0x147 [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580670] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580674] [<ffffffffa05c2762>] ? make_request+0x37/0xa63 [raid1]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580678] [<ffffffff81399449>] ? _raw_read_lock_irqsave+0x21/0x2a
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580685] [<ffffffffa000679f>] ? __split_and_process_bio+0x40d/0x420 [dm_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580689] [<ffffffff81102005>] ? kmem_cache_alloc+0xe1/0x154
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580697] [<ffffffffa02f4b7e>] ? md_make_request+0xee/0x1df [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580704] [<ffffffffa0006902>] ? dm_request+0x150/0x163 [dm_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580708] [<ffffffff811b3efc>] ? generic_make_request+0x96/0xd5
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580712] [<ffffffff811b4c79>] ? submit_bio+0x10a/0x13b
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580716] [<ffffffff8113ce89>] ? dio_bio_submit+0x68/0x88
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580719] [<ffffffff8113dc35>] ? do_blockdev_direct_IO+0x957/0xae8
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580723] [<ffffffff8113aef1>] ? I_BDEV+0x8/0x8
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580752] [<ffffffff810c800a>] ? generic_file_direct_write+0xe3/0x14a
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580755] [<ffffffff810c818c>] ? __generic_file_aio_write+0x11b/0x1ff
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580759] [<ffffffff8113b617>] ? blkdev_aio_write+0x44/0x93
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580763] [<ffffffff811120b3>] ? do_sync_write+0x55/0x7c
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580767] [<ffffffff81112ab0>] ? vfs_write+0x9d/0x103
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580770] [<ffffffff81112eb9>] ? SyS_pwrite64+0x61/0x87
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580773] [<ffffffff8139ea69>] ? system_call_fastpath+0x16/0x1b
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580778] INFO: task md42_raid1:36156 blocked for more than 120 seconds.
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.584560] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588685] md42_raid1 D ffff88207fa33f40 0 36156 2 0x00000000
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588690] ffff881d0b324080 0000000000000046 ffff881d0b324080 ffff881fe2d62810
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588695] 0000000000013f40 ffff881fe1157fd8 ffff881fe1157fd8 ffff881d0b324080
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588699] ffffffff8139958c ffff881fe1157c60 ffff881fb5475800 ffff881fb5475a90
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588703] Call Trace:
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588710] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588734] [<ffffffffa02f4d65>] ? md_super_wait+0x69/0x7f [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588738] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588745] [<ffffffffa02f5131>] ? md_update_sb+0x3b6/0x4b8 [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588750] [<ffffffff8100c02f>] ? load_TLS+0x7/0xa
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588758] [<ffffffffa02f59ee>] ? md_check_recovery+0x1c6/0x3d1 [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588761] [<ffffffffa05c31cc>] ? raid1d+0x3e/0xb22 [raid1]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588766] [<ffffffff81049389>] ? lock_timer_base.isra.35+0x23/0x48
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588769] [<ffffffff810490d4>] ? detach_if_pending+0x18/0x6c
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588772] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588775] [<ffffffff810494ae>] ? try_to_del_timer_sync+0x4e/0x59
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588778] [<ffffffff810494e0>] ? del_timer_sync+0x27/0x44
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588781] [<ffffffff8139796c>] ? schedule_timeout+0xf5/0x123
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588784] [<ffffffff813995cb>] ? _raw_spin_lock_irqsave+0x14/0x35
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588787] [<ffffffff813995cb>] ? _raw_spin_lock_irqsave+0x14/0x35
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588794] [<ffffffffa02f02ed>] ? md_thread+0x114/0x132 [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588797] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588804] [<ffffffffa02f01d9>] ? signal_pending+0x10/0x10 [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588810] [<ffffffffa02f01d9>] ? signal_pending+0x10/0x10 [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588813] [<ffffffff81059295>] ? kthread+0x81/0x89
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588816] [<ffffffff81059214>] ? __kthread_parkme+0x5d/0x5d
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588819] [<ffffffff8139e9bc>] ? ret_from_fork+0x7c/0xb0
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588822] [<ffffffff81059214>] ? __kthread_parkme+0x5d/0x5d
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588834] INFO: task kvm:14262 blocked for more than 120 seconds.
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.592943] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597362] kvm D ffff88207fbb3f40 0 14262 1 0x00000000
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597365] ffff88361cc91080 0000000000000082 0000000000000092 ffff881fe2db3810
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597371] 0000000000013f40 ffff882849fd9fd8 ffff882849fd9fd8 ffff88361cc91080
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597374] ffffffff8139958c ffff883fe1297000 ffff882849fd9cb0 ffff883fe1297290
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597378] Call Trace:
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597383] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597400] [<ffffffffa02efff0>] ? md_write_start+0x131/0x147 [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597403] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597407] [<ffffffffa05c2762>] ? make_request+0x37/0xa63 [raid1]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597415] [<ffffffffa02f4b7e>] ? md_make_request+0xee/0x1df [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597422] [<ffffffffa0006902>] ? dm_request+0x150/0x163 [dm_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597426] [<ffffffff811b3efc>] ? generic_make_request+0x96/0xd5
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597430] [<ffffffff811b4c79>] ? submit_bio+0x10a/0x13b
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597433] [<ffffffff811b6b0a>] ? blkdev_issue_flush+0x86/0xc4
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597438] [<ffffffff8113b298>] ? blkdev_fsync+0x2b/0x37
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597442] [<ffffffff81134a43>] ? do_fsync+0x2b/0x50
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597445] [<ffffffff81134c4f>] ? SyS_fdatasync+0xb/0xf
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597449] [<ffffffff8139ea69>] ? system_call_fastpath+0x16/0x1b
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597452] INFO: task kvm:14266 blocked for more than 120 seconds.
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.602000] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606850] kvm D ffff88207fa53f40 0 14266 1 0x00000000
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606854] ffff883e6f7a3810 0000000000000082 0000000000000096 ffff881fe2d63850
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606860] 0000000000013f40 ffff883c8fc49fd8 ffff883c8fc49fd8 ffff883e6f7a3810
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606865] ffffffff8139958c ffff881b5fa52800 ffff883c8fc49700 ffff881b5fa52a90
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606870] Call Trace:
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606877] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606897] [<ffffffffa02efff0>] ? md_write_start+0x131/0x147 [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606901] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606906] [<ffffffffa05c2762>] ? make_request+0x37/0xa63 [raid1]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606909] [<ffffffff81399449>] ? _raw_read_lock_irqsave+0x21/0x2a
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606917] [<ffffffffa000679f>] ? __split_and_process_bio+0x40d/0x420 [dm_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606925] [<ffffffffa02f4b7e>] ? md_make_request+0xee/0x1df [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606932] [<ffffffffa0006902>] ? dm_request+0x150/0x163 [dm_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606936] [<ffffffff811b3efc>] ? generic_make_request+0x96/0xd5
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606940] [<ffffffff811b4c79>] ? submit_bio+0x10a/0x13b
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606944] [<ffffffff8113ce89>] ? dio_bio_submit+0x68/0x88
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606947] [<ffffffff8113d115>] ? dio_send_cur_page+0x7d/0xa8
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606951] [<ffffffff8113d1e8>] ? submit_page_section+0xa8/0x112
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606954] [<ffffffff8113da99>] ? do_blockdev_direct_IO+0x7bb/0xae8
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606958] [<ffffffff8113aef1>] ? I_BDEV+0x8/0x8
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606961] [<ffffffff8113b0e8>] ? blkdev_direct_IO+0x4e/0x53
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606964] [<ffffffff8113aef1>] ? I_BDEV+0x8/0x8
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606969] [<ffffffff810c800a>] ? generic_file_direct_write+0xe3/0x14a
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606972] [<ffffffff810c818c>] ? __generic_file_aio_write+0x11b/0x1ff
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606976] [<ffffffff8113b617>] ? blkdev_aio_write+0x44/0x93
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606980] [<ffffffff81112038>] ? do_sync_readv_writev+0x50/0x76
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606983] [<ffffffff8113b5d3>] ? bd_may_claim+0x2c/0x2c
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606987] [<ffffffff811130a6>] ? do_readv_writev+0xbf/0x135
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606990] [<ffffffff8113b5d3>] ? bd_may_claim+0x2c/0x2c
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606993] [<ffffffff8111205e>] ? do_sync_readv_writev+0x76/0x76
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606997] [<ffffffff81126eee>] ? fget_light+0x6b/0x7c
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.607000] [<ffffffff81111fbb>] ? fdget+0xe/0x17
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.607004] [<ffffffff811133fe>] ? SyS_pwritev+0x65/0xb0
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.607007] [<ffffffff8139ea69>] ? system_call_fastpath+0x16/0x1b
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.607010] INFO: task kvm:14306 blocked for more than 120 seconds.
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.612040] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617441] kvm D ffff88207fa53f40 0 14306 1 0x00000000
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617445] ffff880eedbaa080 0000000000000082 0000000000011200 ffff881fe2d63850
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617451] 0000000000013f40 ffff881d8e5edfd8 ffff881d8e5edfd8 ffff880eedbaa080
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617451] 0000000000013f40 ffff881d8e5edfd8 ffff881d8e5edfd8 ffff880eedbaa080
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617456] ffffffff8139958c ffff881b5fa52800 ffff881d8e5ed820 ffff881b5fa52a90
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617460] Call Trace:
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617467] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617486] [<ffffffffa02efff0>] ? md_write_start+0x131/0x147 [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617490] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617509] [<ffffffffa05c2762>] ? make_request+0x37/0xa63 [raid1]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617513] [<ffffffff81399449>] ? _raw_read_lock_irqsave+0x21/0x2a
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617531] [<ffffffffa000679f>] ? __split_and_process_bio+0x40d/0x420 [dm_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617536] [<ffffffff81102005>] ? kmem_cache_alloc+0xe1/0x154
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617552] [<ffffffffa02f4b7e>] ? md_make_request+0xee/0x1df [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617559] [<ffffffffa0006902>] ? dm_request+0x150/0x163 [dm_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617563] [<ffffffff811b3efc>] ? generic_make_request+0x96/0xd5
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617573] [<ffffffff811b4c79>] ? submit_bio+0x10a/0x13b
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617577] [<ffffffff8113ce89>] ? dio_bio_submit+0x68/0x88
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617580] [<ffffffff8113dc35>] ? do_blockdev_direct_IO+0x957/0xae8
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617584] [<ffffffff8113aef1>] ? I_BDEV+0x8/0x8
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617587] [<ffffffff8113b0e8>] ? blkdev_direct_IO+0x4e/0x53
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617590] [<ffffffff8113aef1>] ? I_BDEV+0x8/0x8
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617594] [<ffffffff810c800a>] ? generic_file_direct_write+0xe3/0x14a
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617597] [<ffffffff810c818c>] ? __generic_file_aio_write+0x11b/0x1ff
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617601] [<ffffffff8113b617>] ? blkdev_aio_write+0x44/0x93
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617604] [<ffffffff811120b3>] ? do_sync_write+0x55/0x7c
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617608] [<ffffffff81112ab0>] ? vfs_write+0x9d/0x103
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617611] [<ffffffff81112eb9>] ? SyS_pwrite64+0x61/0x87
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617614] [<ffffffff8139ea69>] ? system_call_fastpath+0x16/0x1b
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617629] INFO: task md71_raid1:14478 blocked for more than 120 seconds.
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.622944] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628476] md71_raid1 D ffff88207fad3f40 0 14478 2 0x00000000
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628480] ffff883ed1a09040 0000000000000046 ffff883ed1a09040 ffff881fe2d68850
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628486] 0000000000013f40 ffff883fbd0b7fd8 ffff883fbd0b7fd8 ffff883ed1a09040
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628490] ffffffff8139958c ffff883fbd0b7c60 ffff881b5fa52800 ffff881b5fa52a90
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628498] Call Trace:
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628504] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628522] [<ffffffffa02f4d65>] ? md_super_wait+0x69/0x7f [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628526] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628534] [<ffffffffa02f5131>] ? md_update_sb+0x3b6/0x4b8 [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628542] [<ffffffffa02f59ee>] ? md_check_recovery+0x1c6/0x3d1 [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628546] [<ffffffffa05c31cc>] ? raid1d+0x3e/0xb22 [raid1]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628550] [<ffffffff813988db>] ? __schedule+0x4e7/0x53d
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628553] [<ffffffff813978a3>] ? schedule_timeout+0x2c/0x123
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628557] [<ffffffff813995cb>] ? _raw_spin_lock_irqsave+0x14/0x35
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628560] [<ffffffff813995cb>] ? _raw_spin_lock_irqsave+0x14/0x35
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628568] [<ffffffffa02f02ed>] ? md_thread+0x114/0x132 [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628571] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628578] [<ffffffffa02f01d9>] ? signal_pending+0x10/0x10 [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628585] [<ffffffffa02f01d9>] ? signal_pending+0x10/0x10 [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628588] [<ffffffff81059295>] ? kthread+0x81/0x89
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628591] [<ffffffff81059214>] ? __kthread_parkme+0x5d/0x5d
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628594] [<ffffffff8139e9bc>] ? ret_from_fork+0x7c/0xb0
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628597] [<ffffffff81059214>] ? __kthread_parkme+0x5d/0x5d
>
>
> -------------
>
> Zhiyong Xi
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 3+ messages in thread
* Reply:Re: mdadm hang when one subdev error(raid1)
2015-04-17 9:56 ` hui jiao
@ 2015-04-19 0:06 ` 席智勇
0 siblings, 0 replies; 3+ messages in thread
From: 席智勇 @ 2015-04-19 0:06 UTC (permalink / raw)
To: hui jiao; +Cc: linux-raid
thanks for your reply.
in my system, the subdev was a device exported by iscsi through net. as you said the device was suspended by unknown resion.(in fact, the exported disk was errorred.)
what i do not understand is why the io did not timeout, if it was writing info on superblock while the device cannot response it.
now i think the io should have timeout at iscsi level.but it did not, in this situation，mdadm can did nothing except waiting?
At 2015-04-17 17:56:55, "hui jiao" <simonjiaoh@gmail.com> wrote:
>The md42 is trying to update on-disk superblock, but the operation
>doesn't finish. During this peroid, the configuration lock is holded
>by md42_raid1 thread, all other operations which need the lock will
>wait, such as -D --fail --remove.
>
>what's the status of the subdev now? is it suspended?
>check and resume it:
>[root@node0 ~]# cat /sys/block/dm-0/dm/suspended
>1
>[root@node0 ~]# dmsetup resume /dev/dm-0
>[root@node0 ~]# cat /sys/block/dm-0/dm/suspended
>0
>
>On Fri, Apr 17, 2015 at 3:23 PM, 席智勇 <xizhiyong18@163.com> wrote:
>> hi all:
>>
>> I create some raid1-device by mdadm, when one subdev error, all mdadm related operation just hang there, process state was D.
>> The backgroud is a physical disk was error, so a subdev which is part of the error disk created by device mapper must be errorred, then i did the command 'mdadm --fail' to fail the subdev from the md device, I found it not responsable, just hang there、I tryed 'mdadm --remove', even 'mdadm -D', all hang there. Later, I found not just mdadm operation hang on the problem md device, all mdadm operation on the machine connot be excute.
>> I wana find out what's the problem is, is it a bug of raid when disk error occur， or a problem of my system, because when i found the mdadm hang, the errored disk(/dev/sdp)just missing from my system, I said the disk was error judging from the error log in raid card log.
>> Can anyone give me a help？
>> thanks.
>>
>> uname -a :Linux **-***-***-** 3.10.45-****-amd64 #1 SMP Tue Jul 1 01:52:20 UTC 2014 x86_64 GNU/Linux
>> mdadm --version:mdadm - v3.2.5 - 18th May 2012
>> kern.log:
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561205] kvm D ffff88407f313f40 0 11581 1 0x00000000
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561209] ffff88356e846080 0000000000000082 0000000000000092 ffff881fe2d6a080
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561215] 0000000000013f40 ffff882849dfdfd8 ffff882849dfdfd8 ffff88356e846080
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561219] ffffffff8139958c ffff881cac478000 ffff882849dfdcb0 ffff881cac478290
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561224] Call Trace:
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561233] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561266] [<ffffffffa02efff0>] ? md_write_start+0x131/0x147 [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561271] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561275] [<ffffffffa05c2762>] ? make_request+0x37/0xa63 [raid1]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561284] [<ffffffffa02f4b7e>] ? md_make_request+0xee/0x1df [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561295] [<ffffffffa0006902>] ? dm_request+0x150/0x163 [dm_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561300] [<ffffffff811b3efc>] ? generic_make_request+0x96/0xd5
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561304] [<ffffffff811b4c79>] ? submit_bio+0x10a/0x13b
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561307] [<ffffffff811b6b0a>] ? blkdev_issue_flush+0x86/0xc4
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561311] [<ffffffff8113b298>] ? blkdev_fsync+0x2b/0x37
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561315] [<ffffffff81134a43>] ? do_fsync+0x2b/0x50
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561318] [<ffffffff81134c4f>] ? SyS_fdatasync+0xb/0xf
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561356] [<ffffffff8139ea69>] ? system_call_fastpath+0x16/0x1b
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561367] INFO: task md52_raid1:39767 blocked for more than 120 seconds.
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.563976] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.566971] md52_raid1 D ffff88407f233f40 0 39767 2 0x00000000
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.566975] ffff883e60fa9810 0000000000000046 ffff883e60fa9810 ffff881fe2d620c0
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.566981] 0000000000013f40 ffff883d61523fd8 ffff883d61523fd8 ffff883e60fa9810
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.566985] ffffffff8139958c ffff883d61523c60 ffff881cac478000 ffff881cac478290
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.566990] Call Trace:
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.566995] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567007] [<ffffffffa02f4d65>] ? md_super_wait+0x69/0x7f [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567011] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567019] [<ffffffffa02f5131>] ? md_update_sb+0x3b6/0x4b8 [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567023] [<ffffffff813995cb>] ? _raw_spin_lock_irqsave+0x14/0x35
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567030] [<ffffffffa02f59ee>] ? md_check_recovery+0x1c6/0x3d1 [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567034] [<ffffffffa05c31cc>] ? raid1d+0x3e/0xb22 [raid1]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567038] [<ffffffff813988db>] ? __schedule+0x4e7/0x53d
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567042] [<ffffffff813978a3>] ? schedule_timeout+0x2c/0x123
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567045] [<ffffffff813995cb>] ? _raw_spin_lock_irqsave+0x14/0x35
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567048] [<ffffffff813995cb>] ? _raw_spin_lock_irqsave+0x14/0x35
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567088] [<ffffffffa02f02ed>] ? md_thread+0x114/0x132 [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567092] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567100] [<ffffffffa02f01d9>] ? signal_pending+0x10/0x10 [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567107] [<ffffffffa02f01d9>] ? signal_pending+0x10/0x10 [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567110] [<ffffffff81059295>] ? kthread+0x81/0x89
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567113] [<ffffffff81059214>] ? __kthread_parkme+0x5d/0x5d
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567117] [<ffffffff8139e9bc>] ? ret_from_fork+0x7c/0xb0
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567120] [<ffffffff81059214>] ? __kthread_parkme+0x5d/0x5d
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567125] INFO: task kvm:19762 blocked for more than 120 seconds.
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.570192] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573492] kvm D ffff88407f273f40 0 19762 1 0x00000000
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573496] ffff883fc90f97d0 0000000000000082 0000000000011200 ffff881fe2d64040
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573502] 0000000000013f40 ffff882aad01dfd8 ffff882aad01dfd8 ffff883fc90f97d0
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573507] ffffffff8139958c ffff881fb5475800 ffff882aad01dcb0 ffff881fb5475a90
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573512] Call Trace:
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573517] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573529] [<ffffffffa02efff0>] ? md_write_start+0x131/0x147 [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573545] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573549] [<ffffffffa05c2762>] ? make_request+0x37/0xa63 [raid1]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573557] [<ffffffffa000679f>] ? __split_and_process_bio+0x40d/0x420 [dm_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573561] [<ffffffff81100cbd>] ? ____cache_alloc+0x25d/0x293
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573569] [<ffffffffa02f4b7e>] ? md_make_request+0xee/0x1df [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573576] [<ffffffffa0006902>] ? dm_request+0x150/0x163 [dm_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573580] [<ffffffff811b3efc>] ? generic_make_request+0x96/0xd5
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573583] [<ffffffff811b4c79>] ? submit_bio+0x10a/0x13b
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573586] [<ffffffff811b6b0a>] ? blkdev_issue_flush+0x86/0xc4
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573590] [<ffffffff8113b298>] ? blkdev_fsync+0x2b/0x37
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573593] [<ffffffff81134a43>] ? do_fsync+0x2b/0x50
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573596] [<ffffffff81134c4f>] ? SyS_fdatasync+0xb/0xf
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573600] [<ffffffff8139ea69>] ? system_call_fastpath+0x16/0x1b
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573603] INFO: task kvm:9147 blocked for more than 120 seconds.
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.576936] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580590] kvm D ffff88407f3f3f40 0 9147 1 0x00000000
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580594] ffff883fd9bfd080 0000000000000082 0000000000000096 ffff881fe2db5040
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580631] 0000000000013f40 ffff882da06a1fd8 ffff882da06a1fd8 ffff883fd9bfd080
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580635] ffffffff8139958c ffff881fb5475800 ffff882da06a1820 ffff881fb5475a90
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580648] Call Trace:
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580653] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580666] [<ffffffffa02efff0>] ? md_write_start+0x131/0x147 [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580670] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580674] [<ffffffffa05c2762>] ? make_request+0x37/0xa63 [raid1]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580678] [<ffffffff81399449>] ? _raw_read_lock_irqsave+0x21/0x2a
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580685] [<ffffffffa000679f>] ? __split_and_process_bio+0x40d/0x420 [dm_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580689] [<ffffffff81102005>] ? kmem_cache_alloc+0xe1/0x154
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580697] [<ffffffffa02f4b7e>] ? md_make_request+0xee/0x1df [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580704] [<ffffffffa0006902>] ? dm_request+0x150/0x163 [dm_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580708] [<ffffffff811b3efc>] ? generic_make_request+0x96/0xd5
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580712] [<ffffffff811b4c79>] ? submit_bio+0x10a/0x13b
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580716] [<ffffffff8113ce89>] ? dio_bio_submit+0x68/0x88
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580719] [<ffffffff8113dc35>] ? do_blockdev_direct_IO+0x957/0xae8
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580723] [<ffffffff8113aef1>] ? I_BDEV+0x8/0x8
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580752] [<ffffffff810c800a>] ? generic_file_direct_write+0xe3/0x14a
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580755] [<ffffffff810c818c>] ? __generic_file_aio_write+0x11b/0x1ff
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580759] [<ffffffff8113b617>] ? blkdev_aio_write+0x44/0x93
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580763] [<ffffffff811120b3>] ? do_sync_write+0x55/0x7c
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580767] [<ffffffff81112ab0>] ? vfs_write+0x9d/0x103
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580770] [<ffffffff81112eb9>] ? SyS_pwrite64+0x61/0x87
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580773] [<ffffffff8139ea69>] ? system_call_fastpath+0x16/0x1b
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580778] INFO: task md42_raid1:36156 blocked for more than 120 seconds.
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.584560] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588685] md42_raid1 D ffff88207fa33f40 0 36156 2 0x00000000
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588690] ffff881d0b324080 0000000000000046 ffff881d0b324080 ffff881fe2d62810
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588695] 0000000000013f40 ffff881fe1157fd8 ffff881fe1157fd8 ffff881d0b324080
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588699] ffffffff8139958c ffff881fe1157c60 ffff881fb5475800 ffff881fb5475a90
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588703] Call Trace:
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588710] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588734] [<ffffffffa02f4d65>] ? md_super_wait+0x69/0x7f [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588738] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588745] [<ffffffffa02f5131>] ? md_update_sb+0x3b6/0x4b8 [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588750] [<ffffffff8100c02f>] ? load_TLS+0x7/0xa
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588758] [<ffffffffa02f59ee>] ? md_check_recovery+0x1c6/0x3d1 [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588761] [<ffffffffa05c31cc>] ? raid1d+0x3e/0xb22 [raid1]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588766] [<ffffffff81049389>] ? lock_timer_base.isra.35+0x23/0x48
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588769] [<ffffffff810490d4>] ? detach_if_pending+0x18/0x6c
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588772] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588775] [<ffffffff810494ae>] ? try_to_del_timer_sync+0x4e/0x59
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588778] [<ffffffff810494e0>] ? del_timer_sync+0x27/0x44
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588781] [<ffffffff8139796c>] ? schedule_timeout+0xf5/0x123
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588784] [<ffffffff813995cb>] ? _raw_spin_lock_irqsave+0x14/0x35
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588787] [<ffffffff813995cb>] ? _raw_spin_lock_irqsave+0x14/0x35
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588794] [<ffffffffa02f02ed>] ? md_thread+0x114/0x132 [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588797] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588804] [<ffffffffa02f01d9>] ? signal_pending+0x10/0x10 [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588810] [<ffffffffa02f01d9>] ? signal_pending+0x10/0x10 [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588813] [<ffffffff81059295>] ? kthread+0x81/0x89
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588816] [<ffffffff81059214>] ? __kthread_parkme+0x5d/0x5d
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588819] [<ffffffff8139e9bc>] ? ret_from_fork+0x7c/0xb0
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588822] [<ffffffff81059214>] ? __kthread_parkme+0x5d/0x5d
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588834] INFO: task kvm:14262 blocked for more than 120 seconds.
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.592943] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597362] kvm D ffff88207fbb3f40 0 14262 1 0x00000000
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597365] ffff88361cc91080 0000000000000082 0000000000000092 ffff881fe2db3810
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597371] 0000000000013f40 ffff882849fd9fd8 ffff882849fd9fd8 ffff88361cc91080
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597374] ffffffff8139958c ffff883fe1297000 ffff882849fd9cb0 ffff883fe1297290
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597378] Call Trace:
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597383] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597400] [<ffffffffa02efff0>] ? md_write_start+0x131/0x147 [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597403] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597407] [<ffffffffa05c2762>] ? make_request+0x37/0xa63 [raid1]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597415] [<ffffffffa02f4b7e>] ? md_make_request+0xee/0x1df [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597422] [<ffffffffa0006902>] ? dm_request+0x150/0x163 [dm_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597426] [<ffffffff811b3efc>] ? generic_make_request+0x96/0xd5
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597430] [<ffffffff811b4c79>] ? submit_bio+0x10a/0x13b
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597433] [<ffffffff811b6b0a>] ? blkdev_issue_flush+0x86/0xc4
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597438] [<ffffffff8113b298>] ? blkdev_fsync+0x2b/0x37
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597442] [<ffffffff81134a43>] ? do_fsync+0x2b/0x50
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597445] [<ffffffff81134c4f>] ? SyS_fdatasync+0xb/0xf
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597449] [<ffffffff8139ea69>] ? system_call_fastpath+0x16/0x1b
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597452] INFO: task kvm:14266 blocked for more than 120 seconds.
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.602000] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606850] kvm D ffff88207fa53f40 0 14266 1 0x00000000
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606854] ffff883e6f7a3810 0000000000000082 0000000000000096 ffff881fe2d63850
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606860] 0000000000013f40 ffff883c8fc49fd8 ffff883c8fc49fd8 ffff883e6f7a3810
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606865] ffffffff8139958c ffff881b5fa52800 ffff883c8fc49700 ffff881b5fa52a90
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606870] Call Trace:
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606877] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606897] [<ffffffffa02efff0>] ? md_write_start+0x131/0x147 [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606901] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606906] [<ffffffffa05c2762>] ? make_request+0x37/0xa63 [raid1]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606909] [<ffffffff81399449>] ? _raw_read_lock_irqsave+0x21/0x2a
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606917] [<ffffffffa000679f>] ? __split_and_process_bio+0x40d/0x420 [dm_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606925] [<ffffffffa02f4b7e>] ? md_make_request+0xee/0x1df [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606932] [<ffffffffa0006902>] ? dm_request+0x150/0x163 [dm_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606936] [<ffffffff811b3efc>] ? generic_make_request+0x96/0xd5
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606940] [<ffffffff811b4c79>] ? submit_bio+0x10a/0x13b
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606944] [<ffffffff8113ce89>] ? dio_bio_submit+0x68/0x88
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606947] [<ffffffff8113d115>] ? dio_send_cur_page+0x7d/0xa8
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606951] [<ffffffff8113d1e8>] ? submit_page_section+0xa8/0x112
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606954] [<ffffffff8113da99>] ? do_blockdev_direct_IO+0x7bb/0xae8
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606958] [<ffffffff8113aef1>] ? I_BDEV+0x8/0x8
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606961] [<ffffffff8113b0e8>] ? blkdev_direct_IO+0x4e/0x53
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606964] [<ffffffff8113aef1>] ? I_BDEV+0x8/0x8
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606969] [<ffffffff810c800a>] ? generic_file_direct_write+0xe3/0x14a
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606972] [<ffffffff810c818c>] ? __generic_file_aio_write+0x11b/0x1ff
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606976] [<ffffffff8113b617>] ? blkdev_aio_write+0x44/0x93
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606980] [<ffffffff81112038>] ? do_sync_readv_writev+0x50/0x76
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606983] [<ffffffff8113b5d3>] ? bd_may_claim+0x2c/0x2c
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606987] [<ffffffff811130a6>] ? do_readv_writev+0xbf/0x135
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606990] [<ffffffff8113b5d3>] ? bd_may_claim+0x2c/0x2c
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606993] [<ffffffff8111205e>] ? do_sync_readv_writev+0x76/0x76
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606997] [<ffffffff81126eee>] ? fget_light+0x6b/0x7c
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.607000] [<ffffffff81111fbb>] ? fdget+0xe/0x17
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.607004] [<ffffffff811133fe>] ? SyS_pwritev+0x65/0xb0
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.607007] [<ffffffff8139ea69>] ? system_call_fastpath+0x16/0x1b
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.607010] INFO: task kvm:14306 blocked for more than 120 seconds.
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.612040] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617441] kvm D ffff88207fa53f40 0 14306 1 0x00000000
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617445] ffff880eedbaa080 0000000000000082 0000000000011200 ffff881fe2d63850
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617451] 0000000000013f40 ffff881d8e5edfd8 ffff881d8e5edfd8 ffff880eedbaa080
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617451] 0000000000013f40 ffff881d8e5edfd8 ffff881d8e5edfd8 ffff880eedbaa080
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617456] ffffffff8139958c ffff881b5fa52800 ffff881d8e5ed820 ffff881b5fa52a90
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617460] Call Trace:
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617467] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617486] [<ffffffffa02efff0>] ? md_write_start+0x131/0x147 [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617490] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617509] [<ffffffffa05c2762>] ? make_request+0x37/0xa63 [raid1]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617513] [<ffffffff81399449>] ? _raw_read_lock_irqsave+0x21/0x2a
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617531] [<ffffffffa000679f>] ? __split_and_process_bio+0x40d/0x420 [dm_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617536] [<ffffffff81102005>] ? kmem_cache_alloc+0xe1/0x154
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617552] [<ffffffffa02f4b7e>] ? md_make_request+0xee/0x1df [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617559] [<ffffffffa0006902>] ? dm_request+0x150/0x163 [dm_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617563] [<ffffffff811b3efc>] ? generic_make_request+0x96/0xd5
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617573] [<ffffffff811b4c79>] ? submit_bio+0x10a/0x13b
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617577] [<ffffffff8113ce89>] ? dio_bio_submit+0x68/0x88
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617580] [<ffffffff8113dc35>] ? do_blockdev_direct_IO+0x957/0xae8
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617584] [<ffffffff8113aef1>] ? I_BDEV+0x8/0x8
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617587] [<ffffffff8113b0e8>] ? blkdev_direct_IO+0x4e/0x53
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617590] [<ffffffff8113aef1>] ? I_BDEV+0x8/0x8
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617594] [<ffffffff810c800a>] ? generic_file_direct_write+0xe3/0x14a
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617597] [<ffffffff810c818c>] ? __generic_file_aio_write+0x11b/0x1ff
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617601] [<ffffffff8113b617>] ? blkdev_aio_write+0x44/0x93
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617604] [<ffffffff811120b3>] ? do_sync_write+0x55/0x7c
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617608] [<ffffffff81112ab0>] ? vfs_write+0x9d/0x103
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617611] [<ffffffff81112eb9>] ? SyS_pwrite64+0x61/0x87
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617614] [<ffffffff8139ea69>] ? system_call_fastpath+0x16/0x1b
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617629] INFO: task md71_raid1:14478 blocked for more than 120 seconds.
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.622944] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628476] md71_raid1 D ffff88207fad3f40 0 14478 2 0x00000000
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628480] ffff883ed1a09040 0000000000000046 ffff883ed1a09040 ffff881fe2d68850
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628486] 0000000000013f40 ffff883fbd0b7fd8 ffff883fbd0b7fd8 ffff883ed1a09040
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628490] ffffffff8139958c ffff883fbd0b7c60 ffff881b5fa52800 ffff881b5fa52a90
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628498] Call Trace:
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628504] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628522] [<ffffffffa02f4d65>] ? md_super_wait+0x69/0x7f [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628526] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628534] [<ffffffffa02f5131>] ? md_update_sb+0x3b6/0x4b8 [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628542] [<ffffffffa02f59ee>] ? md_check_recovery+0x1c6/0x3d1 [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628546] [<ffffffffa05c31cc>] ? raid1d+0x3e/0xb22 [raid1]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628550] [<ffffffff813988db>] ? __schedule+0x4e7/0x53d
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628553] [<ffffffff813978a3>] ? schedule_timeout+0x2c/0x123
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628557] [<ffffffff813995cb>] ? _raw_spin_lock_irqsave+0x14/0x35
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628560] [<ffffffff813995cb>] ? _raw_spin_lock_irqsave+0x14/0x35
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628568] [<ffffffffa02f02ed>] ? md_thread+0x114/0x132 [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628571] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628578] [<ffffffffa02f01d9>] ? signal_pending+0x10/0x10 [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628585] [<ffffffffa02f01d9>] ? signal_pending+0x10/0x10 [md_mod]
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628588] [<ffffffff81059295>] ? kthread+0x81/0x89
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628591] [<ffffffff81059214>] ? __kthread_parkme+0x5d/0x5d
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628594] [<ffffffff8139e9bc>] ? ret_from_fork+0x7c/0xb0
>> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628597] [<ffffffff81059214>] ? __kthread_parkme+0x5d/0x5d
>>
>>
>> -------------
>>
>> Zhiyong Xi
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2015-04-19 0:06 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-04-17 7:23 mdadm hang when one subdev error(raid1) 席智勇
2015-04-17 9:56 ` hui jiao
2015-04-19 0:06 ` Reply:Re: " 席智勇
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.