* Kernel 3.7.9 rcu_sched detected stalls on CPUs/tasks:
@ 2013-03-15 21:57 Urban Loesch
0 siblings, 0 replies; only message in thread
From: Urban Loesch @ 2013-03-15 21:57 UTC (permalink / raw)
To: linux-kernel; +Cc: bind
Hi,
I'm running a new DELL PER420 with 2x Intel Xeon E5-2440 2,40 Ghz and
16GB of RAM with Kernel 3.7.9 and Linux vserver patch vs2.3.5.6 from
http://linux-vserver.org. Uptime was about 14 days, 8 hours.
Today evening the system has become not responsive and the netconsole
got me the following log entries. I had to hard reset the machine to got
it to work again. There was no output on the screen, too.
Logs entries I got:
2013-03-15 19:34:01 [1239160.944985] INFO: rcu_sched detected stalls on
CPUs/tasks:
2013-03-15 19:34:01
2013-03-15 19:34:01 [1239160.951430] 0: (2 GPs behind)
idle=d99/140000000000001/0 drain=0 . timer not pending
2013-03-15 19:34:01 [1239160.960413]
2013-03-15 19:34:01 (detected by 6, t=6003 jiffies)
2013-03-15 19:34:01 [1239160.965429] Pid: 0, comm: swapper/6 Not tainted
3.7.9-vs2.3.5.6-rol-em64t-tg3+sdfix #3
2013-03-15 19:34:01 [1239160.974516] Call Trace:
2013-03-15 19:34:01 [1239160.977470] <IRQ>
2013-03-15 19:34:01 [<ffffffff810f8e15>] rcu_pending+0x535/0x550
2013-03-15 19:34:01 [1239160.984439] [<ffffffff810facc2>]
rcu_check_callbacks+0xe2/0x180
2013-03-15 19:34:01 [1239160.991373] [<ffffffff810663b8>]
update_process_times+0x48/0x90
2013-03-15 19:34:01 [1239160.998307] [<ffffffff810bb4e3>]
tick_sched_timer+0x83/0xd0
2013-03-15 19:34:01 [1239161.004844] [<ffffffff8107d479>]
__run_hrtimer+0x79/0x1d0
2013-03-15 19:34:01 [1239161.011195] [<ffffffff810bb460>] ?
tick_nohz_handler+0xf0/0xf0
2013-03-15 19:34:01 [1239161.019993] [<ffffffff8107dd97>]
hrtimer_interrupt+0xf7/0x230
2013-03-15 19:34:01 [1239161.026732] [<ffffffff816b3279>]
smp_apic_timer_interrupt+0x69/0x99
2013-03-15 19:34:01 [1239161.034053] [<ffffffff816b230a>]
apic_timer_interrupt+0x6a/0x70
2013-03-15 19:34:01 [1239161.040985] <EOI>
2013-03-15 19:34:01 [<ffffffff8107d937>] ?
__hrtimer_start_range_ns+0x1b7/0x400
2013-03-15 19:34:01 [1239161.049435] [<ffffffff813d521d>] ?
intel_idle+0xed/0x150
2013-03-15 19:34:01 [1239161.055687] [<ffffffff813d51fe>] ?
intel_idle+0xce/0x150
2013-03-15 19:34:01 [1239161.061940] [<ffffffff8156b559>]
cpuidle_enter+0x19/0x20
2013-03-15 19:34:01 [1239161.068192] [<ffffffff8156bc22>]
cpuidle_idle_call+0xa2/0x1b0
2013-03-15 19:34:01 [1239161.074924] [<ffffffff8101d02f>]
cpu_idle+0xaf/0x120
2013-03-15 19:34:01 [1239161.080791] [<ffffffff81695c9d>]
start_secondary+0x1d4/0x1db
2013-03-15 19:33:58 kernel: [1239160.944985] INFO: rcu_sched detected
stalls on CPUs/tasks:
2013-03-15 19:33:58 kernel: [1239160.951430] 0: (2 GPs behind)
idle=d99/140000000000001/0 drain=0 . timer not pending
2013-03-15 19:33:58 kernel: [1239160.960413] (detected by 6, t=6003
jiffies)
2013-03-15 19:33:58 kernel: [1239160.965429] Pid: 0, comm: swapper/6 Not
tainted 3.7.9-vs2.3.5.6-rol-em64t-tg3+sdfix #3
2013-03-15 19:33:58 kernel: [1239160.974516] Call Trace:
2013-03-15 19:33:58 kernel: [1239160.977470] <IRQ>
[<ffffffff810f8e15>] rcu_pending+0x535/0x550
2013-03-15 19:33:58 kernel: [1239160.984439] [<ffffffff810facc2>]
rcu_check_callbacks+0xe2/0x180
2013-03-15 19:33:58 kernel: [1239160.991373] [<ffffffff810663b8>]
update_process_times+0x48/0x90
2013-03-15 19:33:58 kernel: [1239160.998307] [<ffffffff810bb4e3>]
tick_sched_timer+0x83/0xd0
2013-03-15 19:33:58 kernel: [1239161.004844] [<ffffffff8107d479>]
__run_hrtimer+0x79/0x1d0
2013-03-15 19:33:58 kernel: [1239161.011195] [<ffffffff810bb460>] ?
tick_nohz_handler+0xf0/0xf0
2013-03-15 19:33:58 kernel: [1239161.019993] [<ffffffff8107dd97>]
hrtimer_interrupt+0xf7/0x230
2013-03-15 19:33:58 kernel: [1239161.026732] [<ffffffff816b3279>]
smp_apic_timer_interrupt+0x69/0x99
2013-03-15 19:33:58 kernel: [1239161.034053] [<ffffffff816b230a>]
apic_timer_interrupt+0x6a/0x70
2013-03-15 19:37:01 [1239340.713569] INFO: rcu_sched detected stalls on
CPUs/tasks:
2013-03-15 19:37:01
2013-03-15 19:37:01 [1239340.719931] 0: (2 GPs behind)
idle=d99/140000000000001/0 drain=0 . timer not pending
2013-03-15 19:37:01 [1239340.728915]
2013-03-15 19:37:01 (detected by 6, t=24008 jiffies)
2013-03-15 19:37:01 [1239340.734016] Pid: 0, comm: swapper/6 Not tainted
3.7.9-vs2.3.5.6-rol-em64t-tg3+sdfix #3
2013-03-15 19:37:01 [1239340.743103] Call Trace:
2013-03-15 19:37:01 [1239340.746057] <IRQ>
2013-03-15 19:37:01 [<ffffffff810f8e15>] rcu_pending+0x535/0x550
2013-03-15 19:37:01 [1239340.753023] [<ffffffff810facc2>]
rcu_check_callbacks+0xe2/0x180
2013-03-15 19:37:01 [1239340.759956] [<ffffffff810663b8>]
update_process_times+0x48/0x90
2013-03-15 19:37:01 [1239340.766888] [<ffffffff810bb4e3>]
tick_sched_timer+0x83/0xd0
2013-03-15 19:37:01 [1239340.773431] [<ffffffff8107d479>]
__run_hrtimer+0x79/0x1d0
2013-03-15 19:37:01 [1239340.779781] [<ffffffff810bb460>] ?
tick_nohz_handler+0xf0/0xf0
2013-03-15 19:37:01 [1239340.786615] [<ffffffff8107dd97>]
hrtimer_interrupt+0xf7/0x230
2013-03-15 19:37:01 [1239340.793353] [<ffffffff816b3279>]
smp_apic_timer_interrupt+0x69/0x99
2013-03-15 19:37:01 [1239340.800673] [<ffffffff816b230a>]
apic_timer_interrupt+0x6a/0x70
2013-03-15 19:37:01 [1239340.807604] <EOI>
2013-03-15 19:37:01 [<ffffffff8107d937>] ?
__hrtimer_start_range_ns+0x1b7/0x400
2013-03-15 19:37:01 [1239340.816045] [<ffffffff813d521d>] ?
intel_idle+0xed/0x150
2013-03-15 19:37:01 [1239340.822297] [<ffffffff813d51fe>] ?
intel_idle+0xce/0x150
2013-03-15 19:37:01 [1239340.828550] [<ffffffff8156b559>]
cpuidle_enter+0x19/0x20
2013-03-15 19:37:01 [1239340.834801] [<ffffffff8156bc22>]
cpuidle_idle_call+0xa2/0x1b0
2013-03-15 19:37:01 [1239340.841539] [<ffffffff8101d02f>]
cpu_idle+0xaf/0x120
2013-03-15 19:37:01 [1239340.847397] [<ffffffff81695c9d>]
start_secondary+0x1d4/0x1db
2013-03-15 19:37:03 [1239342.520795] INFO: task jbd2/sda2-8:416 blocked
for more than 120 seconds.
2013-03-15 19:37:03 [1239342.528610] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2013-03-15 19:37:03 [1239342.537608] jbd2/sda2-8 D
2013-03-15 19:37:03 ffff88042fc53c80
2013-03-15 19:37:03 0 416 2 0x00000000
2013-03-15 19:37:03 [1239342.545759] ffff880221ab1ad8
2013-03-15 19:37:03 0000000000000046
2013-03-15 19:37:03 ffff880221de9720
2013-03-15 19:37:03 ffff880221ab1fd8
2013-03-15 19:37:03
2013-03-15 19:37:03 [1239342.554330] ffff880221ab1fd8
2013-03-15 19:37:03 ffff880221ab1fd8
2013-03-15 19:37:03 ffff8802248cdc80
2013-03-15 19:37:03 ffff880221de9720
2013-03-15 19:37:03
2013-03-15 19:37:03 [1239342.562892] ffff880221ab1aa8
2013-03-15 19:37:03 ffff880221de9720
2013-03-15 19:37:03 ffff88042fc544e8
2013-03-15 19:37:03 0000000000000002
2013-03-15 19:37:03
2013-03-15 19:37:03 [1239342.571465] Call Trace:
2013-03-15 19:37:03 [1239342.574426] [<ffffffff81132440>] ?
__lock_page+0x70/0x70
2013-03-15 19:37:03 [1239342.580684] [<ffffffff816a80c9>]
schedule+0x29/0x70
2013-03-15 19:37:03 [1239342.586449] [<ffffffff816a819f>]
io_schedule+0x8f/0xd0
2013-03-15 19:37:03 [1239342.592510] [<ffffffff8113244e>]
sleep_on_page+0xe/0x20
2013-03-15 19:37:03 [1239342.598653] [<ffffffff816a6870>]
__wait_on_bit+0x60/0x90
2013-03-15 19:37:03 [1239342.604907] [<ffffffff81132b81>] ?
find_get_pages_tag+0xd1/0x180
2013-03-15 19:37:03 [1239342.611937] [<ffffffff81132580>]
wait_on_page_bit+0x80/0x90
2013-03-15 19:37:03 [1239342.618484] [<ffffffff81079a00>] ?
autoremove_wake_function+0x40/0x40
2013-03-15 19:37:03 [1239342.625998] [<ffffffff81132951>]
filemap_fdatawait_range+0x101/0x190
2013-03-15 19:37:03 [1239342.633422] [<ffffffff8135ba49>] ?
submit_bio+0x79/0x160
2013-03-15 19:37:03 [1239342.639676] [<ffffffff811c6cd5>] ?
bio_alloc_bioset+0x65/0x120
2013-03-15 19:37:03 [1239342.646512] [<ffffffff81132a07>]
filemap_fdatawait+0x27/0x30
2013-03-15 19:37:03 [1239342.653160] [<ffffffff8128b77c>]
jbd2_journal_commit_transaction+0xa9c/0x19d0
2013-03-15 19:37:03 [1239342.661475] [<ffffffff81042af9>] ?
default_spin_lock_flags+0x9/0x10
2013-03-15 19:37:03 [1239342.668792] [<ffffffff8129113d>]
kjournald2+0xbd/0x240
2013-03-15 19:37:03 [1239342.674852] [<ffffffff810799c0>] ?
finish_wait+0x80/0x80
2013-03-15 19:37:03 [1239342.681099] [<ffffffff81291080>] ?
jbd2_journal_release_jbd_inode+0x130/0x130
2013-03-15 19:37:03 [1239342.689407] [<ffffffff81079070>] kthread+0xc0/0xd0
2013-03-15 19:37:03 [1239342.695080] [<ffffffff81078fb0>] ?
kthread_create_on_node+0x130/0x130
2013-03-15 19:37:03 [1239342.702596] [<ffffffff816b17ac>]
ret_from_fork+0x7c/0xb0
2013-03-15 19:37:03 [1239342.708846] [<ffffffff81078fb0>] ?
kthread_create_on_node+0x130/0x130
2013-03-15 19:37:03 [1239342.716361] INFO: task jbd2/sda3-8:1930 blocked
for more than 120 seconds.
2013-03-15 19:37:03 [1239342.724259] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2013-03-15 19:37:03 [1239342.733247] jbd2/sda3-8 D
2013-03-15 19:37:03 ffff880227c33c80
2013-03-15 19:37:03 0 1930 2 0x00000000
2013-03-15 19:37:03 [1239342.741394] ffff8802234afad8
2013-03-15 19:37:03 0000000000000046
2013-03-15 19:37:03 ffff8802234b2e40
2013-03-15 19:37:03 ffff8802234affd8
2013-03-15 19:37:03
2013-03-15 19:37:03 [1239342.749960] ffff8802234affd8
2013-03-15 19:37:03 ffff8802234affd8
2013-03-15 19:37:03 ffff8802248c9720
2013-03-15 19:37:03 ffff8802234b2e40
2013-03-15 19:37:03
2013-03-15 19:37:03 [1239342.758527] ffff8802234afaa8
2013-03-15 19:37:03 ffff8802234b2e40
2013-03-15 19:37:03 ffff880227c344e8
2013-03-15 19:37:03 0000000000000002
2013-03-15 19:37:03
2013-03-15 19:37:03 [1239342.767097] Call Trace:
2013-03-15 19:37:03 [1239342.770049] [<ffffffff81132440>] ?
__lock_page+0x70/0x70
2013-03-15 19:37:03 [1239342.776303] [<ffffffff816a80c9>]
schedule+0x29/0x70
2013-03-15 19:37:03 [1239342.782074] [<ffffffff816a819f>]
io_schedule+0x8f/0xd0
2013-03-15 19:37:03 [1239342.788131] [<ffffffff8113244e>]
sleep_on_page+0xe/0x20
2013-03-15 19:37:03 [1239342.794288] [<ffffffff816a6870>]
__wait_on_bit+0x60/0x90
2013-03-15 19:37:03 [1239342.800534] [<ffffffff81132b81>] ?
find_get_pages_tag+0xd1/0x180
2013-03-15 19:37:03 [1239342.807553] [<ffffffff81132580>]
wait_on_page_bit+0x80/0x90
2013-03-15 19:37:03 [1239342.814098] [<ffffffff81079a00>] ?
autoremove_wake_function+0x40/0x40
2013-03-15 19:37:03 [1239342.821612] [<ffffffff81132951>]
filemap_fdatawait_range+0x101/0x190
2013-03-15 19:37:03 [1239342.829027] [<ffffffff8135ba49>] ?
submit_bio+0x79/0x160
2013-03-15 19:37:03 [1239342.835282] [<ffffffff811c6cd5>] ?
bio_alloc_bioset+0x65/0x120
2013-03-15 19:37:03 [1239342.842108] [<ffffffff81132a07>]
filemap_fdatawait+0x27/0x30
2013-03-15 19:37:03 [1239342.848748] [<ffffffff8128b77c>]
jbd2_journal_commit_transaction+0xa9c/0x19d0
2013-03-15 19:37:03 [1239342.857059] [<ffffffff81042af9>] ?
default_spin_lock_flags+0x9/0x10
2013-03-15 19:37:03 [1239342.864377] [<ffffffff8129113d>]
kjournald2+0xbd/0x240
2013-03-15 19:37:03 [1239342.870438] [<ffffffff810799c0>] ?
finish_wait+0x80/0x80
2013-03-15 19:37:03 [1239342.876698] [<ffffffff81291080>] ?
jbd2_journal_release_jbd_inode+0x130/0x130
2013-03-15 19:37:03 [1239342.885013] [<ffffffff81079070>] kthread+0xc0/0xd0
2013-03-15 19:37:03 [1239342.890687] [<ffffffff81078fb0>] ?
kthread_create_on_node+0x130/0x130
2013-03-15 19:37:03 [1239342.898190] [<ffffffff816b17ac>]
ret_from_fork+0x7c/0xb0
2013-03-15 19:37:04 [1239342.904445] [<ffffffff81078fb0>] ?
kthread_create_on_node+0x130/0x130
2013-03-15 19:37:04 [1239342.911962] INFO: task flush-8:0:3588 blocked
for more than 120 seconds.
2013-03-15 19:37:04 [1239342.919664] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2013-03-15 19:37:04 [1239342.928652] flush-8:0 D
2013-03-15 19:37:04 ffff88042fc53c80
2013-03-15 19:37:04 0 3588 2 0x00000000
2013-03-15 19:37:04 [1239342.936798] ffff880221fcf698
2013-03-15 19:37:04 0000000000000046
2013-03-15 19:37:04 ffff8802233f5c80
2013-03-15 19:37:04 ffff880221fcffd8
2013-03-15 19:37:04
2013-03-15 19:37:04 [1239342.945366] ffff880221fcffd8
2013-03-15 19:37:04 ffff880221fcffd8
2013-03-15 19:37:04 ffff8802248cdc80
2013-03-15 19:37:04 ffff8802233f5c80
2013-03-15 19:37:04
2013-03-15 19:37:04 [1239342.953916] ffff880221fcf6a8
2013-03-15 19:37:04 ffff8801ed90daf8
2013-03-15 19:37:04 ffff8803db412070
2013-03-15 19:37:04 ffff880423779ca8
2013-03-15 19:37:04
2013-03-15 19:37:04 [1239342.962483] Call Trace:
2013-03-15 19:37:04 [1239342.965434] [<ffffffff816a80c9>]
schedule+0x29/0x70
2013-03-15 19:37:04 [1239342.971206] [<ffffffff8128a1dd>]
do_get_write_access+0x2cd/0x4d0
2013-03-15 19:37:04 [1239342.978235] [<ffffffff8137e53e>] ?
radix_tree_lookup_slot+0xe/0x10
2013-03-15 19:37:04 [1239342.985458] [<ffffffff81079a00>] ?
autoremove_wake_function+0x40/0x40
2013-03-15 19:37:04 [1239342.992976] [<ffffffff8128a520>]
jbd2_journal_get_write_access+0x30/0x50
2013-03-15 19:37:04 [1239343.000785] [<ffffffff8126c31d>]
__ext4_journal_get_write_access+0x3d/0x80
2013-03-15 19:37:04 [1239343.008786] [<ffffffff812354b5>] ?
ext4_read_block_bitmap+0x35/0x60
2013-03-15 19:37:04 [1239343.016106] [<ffffffff8127279b>]
ext4_mb_mark_diskspace_used+0x7b/0x4d0
2013-03-15 19:37:04 [1239343.023877] [<ffffffff8117e81f>] ?
__kmalloc+0xaf/0x180
2013-03-15 19:37:04 [1239343.030043] [<ffffffff8126d7c6>] ?
ext4_mb_initialize_context+0x86/0x1b0
2013-03-15 19:37:04 [1239343.037843] [<ffffffff81273f5b>]
ext4_mb_new_blocks+0x29b/0x550
2013-03-15 19:37:04 [1239343.044778] [<ffffffff8126a97f>]
ext4_ext_map_blocks+0xc4f/0x1610
2013-03-15 19:37:04 [1239343.051906] [<ffffffff8123c035>]
ext4_map_blocks+0x1d5/0x270
2013-03-15 19:37:04 [1239343.058541] [<ffffffff8124045d>]
mpage_da_map_and_submit+0x15d/0x4b0
2013-03-15 19:37:04 [1239343.065956] [<ffffffff81241017>] ?
ext4_da_writepages+0x317/0x5e0
2013-03-15 19:37:04 [1239343.073083] [<ffffffff81241063>]
ext4_da_writepages+0x363/0x5e0
2013-03-15 19:37:04 [1239343.080019] [<ffffffff8113e3be>]
do_writepages+0x1e/0x40
2013-03-15 19:37:04 [1239343.086269] [<ffffffff811b977b>]
__writeback_single_inode+0x3b/0x160
2013-03-15 19:37:04 [1239343.093685] [<ffffffff811ba7d0>]
writeback_sb_inodes+0x1a0/0x390
2013-03-15 19:37:04 [1239343.100714] [<ffffffff811baa5f>]
__writeback_inodes_wb+0x9f/0xd0
2013-03-15 19:37:04 [1239343.107730] [<ffffffff811bacdb>]
wb_writeback+0x24b/0x2e0
2013-03-15 19:37:04 [1239343.114080] [<ffffffff811a9b22>] ?
get_nr_inodes+0x52/0x70
2013-03-15 19:37:04 [1239343.120528] [<ffffffff811bbf76>]
wb_do_writeback+0x1f6/0x200
2013-03-15 19:37:04 [1239343.127161] [<ffffffff811bc00b>]
bdi_writeback_thread+0x8b/0x230
2013-03-15 19:37:04 [1239343.134187] [<ffffffff811bbf80>] ?
wb_do_writeback+0x200/0x200
2013-03-15 19:37:04 [1239343.141023] [<ffffffff81079070>] kthread+0xc0/0xd0
2013-03-15 19:37:04 [1239343.146680] [<ffffffff81078fb0>] ?
kthread_create_on_node+0x130/0x130
2013-03-15 19:37:04 [1239343.154184] [<ffffffff816b17ac>]
ret_from_fork+0x7c/0xb0
2013-03-15 19:37:04 [1239343.160428] [<ffffffff81078fb0>] ?
kthread_create_on_node+0x130/0x130
2013-03-15 19:37:04 [1239343.167940] INFO: task tlsmgr:6983 blocked for
more than 120 seconds.
2013-03-15 19:37:04 [1239343.175352] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2013-03-15 19:37:04 [1239343.184336] tlsmgr D
2013-03-15 19:37:04 ffff88042fc13c80
2013-03-15 19:37:04 0 6983 6611 0x00000000
2013-03-15 19:37:04 [1239343.192482] ffff8804230e3a38
2013-03-15 19:37:04 0000000000000082
2013-03-15 19:37:04 ffff880423be5c80
2013-03-15 19:37:04 ffff8804230e3fd8
2013-03-15 19:37:04
2013-03-15 19:37:04 [1239343.201038] ffff8804230e3fd8
2013-03-15 19:37:04 ffff8804230e3fd8
2013-03-15 19:37:04 ffff8802248c8000
2013-03-15 19:37:04 ffff880423be5c80
2013-03-15 19:37:04
2013-03-15 19:37:04 [1239343.209601] ffff8804230e3a48
2013-03-15 19:37:04 ffff880329d98618
2013-03-15 19:37:04 ffff88030b96da10
2013-03-15 19:37:04 ffff880423492b28
2013-03-15 19:37:04
2013-03-15 19:37:04 [1239343.218165] Call Trace:
2013-03-15 19:37:04 [1239343.221119] [<ffffffff816a80c9>]
schedule+0x29/0x70
2013-03-15 19:37:04 [1239343.226874] [<ffffffff8128a1dd>]
do_get_write_access+0x2cd/0x4d0
2013-03-15 19:37:04 [1239343.233903] [<ffffffff81079a00>] ?
autoremove_wake_function+0x40/0x40
2013-03-15 19:37:04 [1239343.241406] [<ffffffff8128a520>]
jbd2_journal_get_write_access+0x30/0x50
2013-03-15 19:37:04 [1239343.249206] [<ffffffff8126c31d>]
__ext4_journal_get_write_access+0x3d/0x80
2013-03-15 19:37:04 [1239343.257204] [<ffffffff8123f178>]
ext4_reserve_inode_write+0x78/0xa0
2013-03-15 19:37:04 [1239343.264522] [<ffffffff812417ec>] ?
ext4_dirty_inode+0x3c/0x60
2013-03-15 19:37:04 [1239343.271261] [<ffffffff8123f1ea>]
ext4_mark_inode_dirty+0x4a/0x200
2013-03-15 19:37:04 [1239343.278373] [<ffffffff812417ec>]
ext4_dirty_inode+0x3c/0x60
2013-03-15 19:37:04 [1239343.284916] [<ffffffff811ba433>]
__mark_inode_dirty+0x43/0x240
2013-03-15 19:37:04 [1239343.291751] [<ffffffff811a9ddb>]
update_time+0xab/0xe0
2013-03-15 19:37:04 [1239343.297807] [<ffffffff811a9ffa>]
touch_atime+0xfa/0x140
2013-03-15 19:37:04 [1239343.303965] [<ffffffff8113445e>]
generic_file_aio_read+0x4ae/0x6c0
2013-03-15 19:37:04 [1239343.311191] [<ffffffff8118f487>]
do_sync_read+0xa7/0xe0
2013-03-15 19:37:04 [1239343.317341] [<ffffffff8118fba9>]
vfs_read+0xa9/0x180
2013-03-15 19:37:04 [1239343.323208] [<ffffffff8118fcd2>]
sys_read+0x52/0xa0
2013-03-15 19:37:04 [1239343.328973] [<ffffffff816b1859>]
system_call_fastpath+0x16/0x1b
2013-03-15 19:39:04 [1239463.141965] INFO: task jbd2/sda3-8:1930 blocked
for more than 120 seconds.
2013-03-15 19:39:04 [1239463.149880] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2013-03-15 19:39:04 [1239463.158882] jbd2/sda3-8 D
2013-03-15 19:39:04 ffff880227c33c80
2013-03-15 19:39:04 0 1930 2 0x00000000
2013-03-15 19:39:04 [1239463.169000] ffff8802234afad8
2013-03-15 19:39:04 0000000000000046
2013-03-15 19:39:04 ffff8802234b2e40
2013-03-15 19:39:04 ffff8802234affd8
2013-03-15 19:39:04
2013-03-15 19:39:04 [1239463.177569] ffff8802234affd8
2013-03-15 19:39:04 ffff8802234affd8
2013-03-15 19:39:04 ffff8802248c9720
2013-03-15 19:39:04 ffff8802234b2e40
2013-03-15 19:39:04
2013-03-15 19:39:04 [1239463.186137] ffff8802234afaa8
2013-03-15 19:39:04 ffff8802234b2e40
2013-03-15 19:39:04 ffff880227c344e8
2013-03-15 19:39:04 0000000000000002
2013-03-15 19:39:04
2013-03-15 19:39:04 [1239463.194705] Call Trace:
2013-03-15 19:39:04 [1239463.197661] [<ffffffff81132440>] ?
__lock_page+0x70/0x70
2013-03-15 19:39:04 [1239463.203918] [<ffffffff816a80c9>]
schedule+0x29/0x70
2013-03-15 19:39:04 [1239463.209683] [<ffffffff816a819f>]
io_schedule+0x8f/0xd0
2013-03-15 19:39:04 [1239463.215744] [<ffffffff8113244e>]
sleep_on_page+0xe/0x20
2013-03-15 19:39:04 [1239463.221910] [<ffffffff816a6870>]
__wait_on_bit+0x60/0x90
2013-03-15 19:39:04 [1239463.228160] [<ffffffff81132b81>] ?
find_get_pages_tag+0xd1/0x180
2013-03-15 19:39:04 [1239463.235189] [<ffffffff81132580>]
wait_on_page_bit+0x80/0x90
2013-03-15 19:39:04 [1239463.241731] [<ffffffff81079a00>] ?
autoremove_wake_function+0x40/0x40
2013-03-15 19:39:04 [1239463.249245] [<ffffffff81132951>]
filemap_fdatawait_range+0x101/0x190
2013-03-15 19:39:04 [1239463.256665] [<ffffffff8135ba49>] ?
submit_bio+0x79/0x160
2013-03-15 19:39:04 [1239463.262921] [<ffffffff811c6cd5>] ?
bio_alloc_bioset+0x65/0x120
2013-03-15 19:39:04 [1239463.269753] [<ffffffff81132a07>]
filemap_fdatawait+0x27/0x30
2013-03-15 19:39:04 [1239463.276404] [<ffffffff8128b77c>]
jbd2_journal_commit_transaction+0xa9c/0x19d0
2013-03-15 19:39:04 [1239463.284721] [<ffffffff81042af9>] ?
default_spin_lock_flags+0x9/0x10
2013-03-15 19:39:04 [1239463.292043] [<ffffffff8129113d>]
kjournald2+0xbd/0x240
2013-03-15 19:39:04 [1239463.298099] [<ffffffff810799c0>] ?
finish_wait+0x80/0x80
2013-03-15 19:39:04 [1239463.304352] [<ffffffff81291080>] ?
jbd2_journal_release_jbd_inode+0x130/0x130
2013-03-15 19:39:04 [1239463.312665] [<ffffffff81079070>] kthread+0xc0/0xd0
2013-03-15 19:39:04 [1239463.318332] [<ffffffff81078fb0>] ?
kthread_create_on_node+0x130/0x130
2013-03-15 19:39:04 [1239463.325848] [<ffffffff816b17ac>]
ret_from_fork+0x7c/0xb0
2013-03-15 19:39:04 [1239463.332102] [<ffffffff81078fb0>] ?
kthread_create_on_node+0x130/0x130
2013-03-15 19:39:04 [1239463.339604] INFO: task rs:main Q:Reg:3217
blocked for more than 120 seconds.
2013-03-15 19:39:04 [1239463.347719] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2013-03-15 19:39:04 [1239463.356706] rs:main Q:Reg D
2013-03-15 19:39:04 ffff88042fc33c80
2013-03-15 19:39:04 0 3217 1 0x00000000
2013-03-15 19:39:04 [1239463.364851] ffff880221e1b948
2013-03-15 19:39:04 0000000000000082
2013-03-15 19:39:04 ffff880223fe5c80
2013-03-15 19:39:04 ffff880221e1bfd8
2013-03-15 19:39:04
2013-03-15 19:39:04 [1239463.373422] ffff880221e1bfd8
2013-03-15 19:39:04 ffff880221e1bfd8
2013-03-15 19:39:04 ffff8802248cae40
2013-03-15 19:39:04 ffff880223fe5c80
2013-03-15 19:39:04
2013-03-15 19:39:04 [1239463.381992] ffff880221e1b958
2013-03-15 19:39:04 ffff8801b1080410
2013-03-15 19:39:04 ffff880340f3c000
2013-03-15 19:39:04 ffff88042370fb70
2013-03-15 19:39:04
2013-03-15 19:39:04 [1239463.390548] Call Trace:
2013-03-15 19:39:04 [1239463.393505] [<ffffffff816a80c9>]
schedule+0x29/0x70
2013-03-15 19:39:04 [1239463.399271] [<ffffffff8128a1dd>]
do_get_write_access+0x2cd/0x4d0
2013-03-15 19:39:04 [1239463.406311] [<ffffffff816a92ae>] ?
_raw_spin_lock_irqsave+0x2e/0x40
2013-03-15 19:39:04 [1239463.413640] [<ffffffff81079a00>] ?
autoremove_wake_function+0x40/0x40
2013-03-15 19:39:04 [1239463.421152] [<ffffffff8128a520>]
jbd2_journal_get_write_access+0x30/0x50
2013-03-15 19:39:04 [1239463.428959] [<ffffffff8126c31d>]
__ext4_journal_get_write_access+0x3d/0x80
2013-03-15 19:39:04 [1239463.436961] [<ffffffff8123f178>]
ext4_reserve_inode_write+0x78/0xa0
2013-03-15 19:39:04 [1239463.444281] [<ffffffff812417ec>] ?
ext4_dirty_inode+0x3c/0x60
2013-03-15 19:39:04 [1239463.451017] [<ffffffff8123f1ea>]
ext4_mark_inode_dirty+0x4a/0x200
2013-03-15 19:39:04 [1239463.458146] [<ffffffff812417ec>]
ext4_dirty_inode+0x3c/0x60
2013-03-15 19:39:04 [1239463.464692] [<ffffffff811ba433>]
__mark_inode_dirty+0x43/0x240
2013-03-15 19:39:04 [1239463.471531] [<ffffffff811a9ddb>]
update_time+0xab/0xe0
2013-03-15 19:39:04 [1239463.477585] [<ffffffff811a9eab>]
file_update_time+0x9b/0xf0
2013-03-15 19:39:04 [1239463.484130] [<ffffffff81133c70>]
__generic_file_aio_write+0x190/0x3d0
2013-03-15 19:39:04 [1239463.491640] [<ffffffff8108e0f8>] ?
__enqueue_entity+0x78/0x80
2013-03-15 19:39:04 [1239463.499614] [<ffffffff81133f2f>]
generic_file_aio_write+0x7f/0x100
2013-03-15 19:39:04 [1239463.506845] [<ffffffff81236d39>]
ext4_file_write+0xa9/0x490
2013-03-15 19:39:04 [1239463.513391] [<ffffffff8108aeb0>] ?
wake_up_state+0x10/0x20
2013-03-15 19:39:04 [1239463.519848] [<ffffffff810bcff6>] ?
wake_futex+0x66/0x80
2013-03-15 19:39:04 [1239463.526006] [<ffffffff810bd11b>] ?
futex_wake+0x10b/0x130
2013-03-15 19:39:04 [1239463.532359] [<ffffffff8118f3a7>]
do_sync_write+0xa7/0xe0
2013-03-15 19:39:04 [1239463.538611] [<ffffffff8118fa2c>]
vfs_write+0xac/0x180
2013-03-15 19:39:04 [1239463.544575] [<ffffffff8118fd72>]
sys_write+0x52/0xa0
2013-03-15 19:39:04 [1239463.550428] [<ffffffff816b1859>]
system_call_fastpath+0x16/0x1b
2013-03-15 19:39:04 [1239463.557361] INFO: task flush-8:0:3588 blocked
for more than 120 seconds.
2013-03-15 19:39:04 [1239463.565068] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2013-03-15 19:39:04 [1239463.574049] flush-8:0 D
2013-03-15 19:39:04 ffff88042fc53c80
2013-03-15 19:39:04 0 3588 2 0x00000000
2013-03-15 19:39:04 [1239463.582190] ffff880221fcf698
2013-03-15 19:39:04 0000000000000046
2013-03-15 19:39:04 ffff8802233f5c80
2013-03-15 19:39:04 ffff880221fcffd8
2013-03-15 19:39:04
2013-03-15 19:39:04 [1239463.590757] ffff880221fcffd8
2013-03-15 19:39:04 ffff880221fcffd8
2013-03-15 19:39:04 ffff8802248cdc80
2013-03-15 19:39:04 ffff8802233f5c80
2013-03-15 19:39:04
2013-03-15 19:39:04 [1239463.599317] ffff880221fcf6a8
2013-03-15 19:39:04 ffff8801ed90daf8
2013-03-15 19:39:04 ffff8803db412070
2013-03-15 19:39:04 ffff880423779ca8
2013-03-15 19:39:04
2013-03-15 19:39:04 [1239463.607890] Call Trace:
2013-03-15 19:39:04 [1239463.610842] [<ffffffff816a80c9>]
schedule+0x29/0x70
2013-03-15 19:39:04 [1239463.616613] [<ffffffff8128a1dd>]
do_get_write_access+0x2cd/0x4d0
2013-03-15 19:39:04 [1239463.623644] [<ffffffff8137e53e>] ?
radix_tree_lookup_slot+0xe/0x10
2013-03-15 19:39:04 [1239463.630866] [<ffffffff81079a00>] ?
autoremove_wake_function+0x40/0x40
2013-03-15 19:39:04 [1239463.638385] [<ffffffff8128a520>]
jbd2_journal_get_write_access+0x30/0x50
2013-03-15 19:39:04 [1239463.646192] [<ffffffff8126c31d>]
__ext4_journal_get_write_access+0x3d/0x80
2013-03-15 19:39:04 [1239463.654193] [<ffffffff812354b5>] ?
ext4_read_block_bitmap+0x35/0x60
2013-03-15 19:39:04 [1239463.661519] [<ffffffff8127279b>]
ext4_mb_mark_diskspace_used+0x7b/0x4d0
2013-03-15 19:39:04 [1239463.669225] [<ffffffff8117e81f>] ?
__kmalloc+0xaf/0x180
2013-03-15 19:39:04 [1239463.675376] [<ffffffff8126d7c6>] ?
ext4_mb_initialize_context+0x86/0x1b0
2013-03-15 19:39:04 [1239463.683184] [<ffffffff81273f5b>]
ext4_mb_new_blocks+0x29b/0x550
2013-03-15 19:39:04 [1239463.690113] [<ffffffff8126a97f>]
ext4_ext_map_blocks+0xc4f/0x1610
2013-03-15 19:39:04 [1239463.697242] [<ffffffff8123c035>]
ext4_map_blocks+0x1d5/0x270
2013-03-15 19:39:04 [1239463.703887] [<ffffffff8124045d>]
mpage_da_map_and_submit+0x15d/0x4b0
2013-03-15 19:39:04 [1239463.711310] [<ffffffff81241017>] ?
ext4_da_writepages+0x317/0x5e0
2013-03-15 19:39:05 [1239463.718435] [<ffffffff81241063>]
ext4_da_writepages+0x363/0x5e0
2013-03-15 19:39:05 [1239463.725371] [<ffffffff8113e3be>]
do_writepages+0x1e/0x40
2013-03-15 19:39:05 [1239463.731628] [<ffffffff811b977b>]
__writeback_single_inode+0x3b/0x160
2013-03-15 19:39:05 [1239463.739043] [<ffffffff811ba7d0>]
writeback_sb_inodes+0x1a0/0x390
2013-03-15 19:39:05 [1239463.746075] [<ffffffff811baa5f>]
__writeback_inodes_wb+0x9f/0xd0
2013-03-15 19:39:05 [1239463.753109] [<ffffffff811bacdb>]
wb_writeback+0x24b/0x2e0
2013-03-15 19:39:05 [1239463.759455] [<ffffffff811a9b22>] ?
get_nr_inodes+0x52/0x70
2013-03-15 19:39:05 [1239463.765900] [<ffffffff811bbf76>]
wb_do_writeback+0x1f6/0x200
2013-03-15 19:39:05 [1239463.772539] [<ffffffff811bc00b>]
bdi_writeback_thread+0x8b/0x230
2013-03-15 19:39:05 [1239463.779562] [<ffffffff811bbf80>] ?
wb_do_writeback+0x200/0x200
2013-03-15 19:39:05 [1239463.786394] [<ffffffff81079070>] kthread+0xc0/0xd0
2013-03-15 19:39:05 [1239463.792061] [<ffffffff81078fb0>] ?
kthread_create_on_node+0x130/0x130
2013-03-15 19:39:05 [1239463.799569] [<ffffffff816b17ac>]
ret_from_fork+0x7c/0xb0
2013-03-15 19:39:05 [1239463.805818] [<ffffffff81078fb0>] ?
kthread_create_on_node+0x130/0x130
2013-03-15 19:39:05 [1239463.813330] INFO: task tlsmgr:6983 blocked for
more than 120 seconds.
2013-03-15 19:39:05 [1239463.820738] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2013-03-15 19:39:05 [1239463.829726] tlsmgr D
2013-03-15 19:39:05 ffff88042fc13c80
2013-03-15 19:39:05 0 6983 6611 0x00000000
2013-03-15 19:39:05 [1239463.837917] ffff8804230e3a38
2013-03-15 19:39:05 0000000000000082
2013-03-15 19:39:05 ffff880423be5c80
2013-03-15 19:39:05 ffff8804230e3fd8
2013-03-15 19:39:05
2013-03-15 19:39:05 [1239463.846468] ffff8804230e3fd8
2013-03-15 19:39:05 ffff8804230e3fd8
2013-03-15 19:39:05 ffff8802248c8000
2013-03-15 19:39:05 ffff880423be5c80
2013-03-15 19:39:05
2013-03-15 19:39:05 [1239463.855021] ffff8804230e3a48
2013-03-15 19:39:05 ffff880329d98618
2013-03-15 19:39:05 ffff88030b96da10
2013-03-15 19:39:05 ffff880423492b28
2013-03-15 19:39:05
2013-03-15 19:39:05 [1239463.863572] Call Trace:
2013-03-15 19:39:05 [1239463.866520] [<ffffffff816a80c9>]
schedule+0x29/0x70
2013-03-15 19:39:05 [1239463.872286] [<ffffffff8128a1dd>]
do_get_write_access+0x2cd/0x4d0
2013-03-15 19:39:05 [1239463.879309] [<ffffffff81079a00>] ?
autoremove_wake_function+0x40/0x40
2013-03-15 19:39:05 [1239463.886819] [<ffffffff8128a520>]
jbd2_journal_get_write_access+0x30/0x50
2013-03-15 19:39:05 [1239463.894621] [<ffffffff8126c31d>]
__ext4_journal_get_write_access+0x3d/0x80
2013-03-15 19:39:05 [1239463.902616] [<ffffffff8123f178>]
ext4_reserve_inode_write+0x78/0xa0
2013-03-15 19:39:05 [1239463.909929] [<ffffffff812417ec>] ?
ext4_dirty_inode+0x3c/0x60
2013-03-15 19:39:05 [1239463.916662] [<ffffffff8123f1ea>]
ext4_mark_inode_dirty+0x4a/0x200
2013-03-15 19:39:05 [1239463.923783] [<ffffffff812417ec>]
ext4_dirty_inode+0x3c/0x60
2013-03-15 19:39:05 [1239463.930322] [<ffffffff811ba433>]
__mark_inode_dirty+0x43/0x240
2013-03-15 19:39:05 [1239463.937152] [<ffffffff811a9ddb>]
update_time+0xab/0xe0
2013-03-15 19:39:05 [1239463.943208] [<ffffffff811a9ffa>]
touch_atime+0xfa/0x140
2013-03-15 19:39:05 [1239463.949356] [<ffffffff8113445e>]
generic_file_aio_read+0x4ae/0x6c0
2013-03-15 19:39:05 [1239463.956576] [<ffffffff8118f487>]
do_sync_read+0xa7/0xe0
2013-03-15 19:39:05 [1239463.962729] [<ffffffff8118fba9>]
vfs_read+0xa9/0x180
2013-03-15 19:39:05 [1239463.968588] [<ffffffff8118fcd2>]
sys_read+0x52/0xa0
2013-03-15 19:39:05 [1239463.974343] [<ffffffff816b1859>]
system_call_fastpath+0x16/0x1b
2013-03-15 19:40:01 [1239520.482156] INFO: rcu_sched detected stalls on
CPUs/tasks:
2013-03-15 19:40:01
2013-03-15 19:40:01 [1239520.488530] 0: (2 GPs behind)
idle=d99/140000000000001/0 drain=0 . timer not pending
2013-03-15 19:40:01 [1239520.497520]
2013-03-15 19:40:01 (detected by 2, t=42013 jiffies)
2013-03-15 19:40:01 [1239520.502629] Pid: 0, comm: swapper/2 Not tainted
3.7.9-vs2.3.5.6-rol-em64t-tg3+sdfix #3
2013-03-15 19:40:01 [1239520.511717] Call Trace:
2013-03-15 19:40:01 [1239520.514670] <IRQ>
2013-03-15 19:40:01 [<ffffffff810f8e15>] rcu_pending+0x535/0x550
2013-03-15 19:40:01 [1239520.521636] [<ffffffff810facc2>]
rcu_check_callbacks+0xe2/0x180
2013-03-15 19:40:01 [1239520.528570] [<ffffffff810663b8>]
update_process_times+0x48/0x90
2013-03-15 19:40:01 [1239520.535502] [<ffffffff810bb4e3>]
tick_sched_timer+0x83/0xd0
2013-03-15 19:40:01 [1239520.542039] [<ffffffff8107d479>]
__run_hrtimer+0x79/0x1d0
2013-03-15 19:40:01 [1239520.548379] [<ffffffff810bb460>] ?
tick_nohz_handler+0xf0/0xf0
2013-03-15 19:40:01 [1239520.555213] [<ffffffff8107dd97>]
hrtimer_interrupt+0xf7/0x230
2013-03-15 19:40:01 [1239520.561951] [<ffffffff816b3279>]
smp_apic_timer_interrupt+0x69/0x99
2013-03-15 19:40:01 [1239520.569265] [<ffffffff816b230a>]
apic_timer_interrupt+0x6a/0x70
2013-03-15 19:40:01 [1239520.576195] <EOI>
2013-03-15 19:40:01 [<ffffffff813d521d>] ? intel_idle+0xed/0x150
2013-03-15 19:40:01 [1239520.583159] [<ffffffff813d51fe>] ?
intel_idle+0xce/0x150
2013-03-15 19:40:01 [1239520.589413] [<ffffffff8156b559>]
cpuidle_enter+0x19/0x20
2013-03-15 19:40:01 [1239520.595665] [<ffffffff8156bc22>]
cpuidle_idle_call+0xa2/0x1b0
2013-03-15 19:40:01 [1239520.602397] [<ffffffff8101d02f>]
cpu_idle+0xaf/0x120
2013-03-15 19:40:01 [1239520.608262] [<ffffffff81695c9d>]
start_secondary+0x1d4/0x1db
2013-03-15 19:41:05 [1239583.793109] INFO: task jbd2/sda2-8:416 blocked
for more than 120 seconds.
2013-03-15 19:41:05 [1239583.800926] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2013-03-15 19:41:05 [1239583.809923] jbd2/sda2-8 D
2013-03-15 19:41:05 ffff88042fc53c80
2013-03-15 19:41:05 0 416 2 0x00000000
2013-03-15 19:41:05 [1239583.818072] ffff880221ab1ad8
2013-03-15 19:41:05 0000000000000046
2013-03-15 19:41:05 ffff880221de9720
2013-03-15 19:41:05 ffff880221ab1fd8
2013-03-15 19:41:05
2013-03-15 19:41:05 [1239583.826641] ffff880221ab1fd8
2013-03-15 19:41:05 ffff880221ab1fd8
2013-03-15 19:41:05 ffff8802248cdc80
2013-03-15 19:41:05 ffff880221de9720
2013-03-15 19:41:05
2013-03-15 19:41:05 [1239583.835203] ffff880221ab1aa8
2013-03-15 19:41:05 ffff880221de9720
2013-03-15 19:41:05 ffff88042fc544e8
2013-03-15 19:41:05 0000000000000002
2013-03-15 19:41:05
2013-03-15 19:41:05 [1239583.843765] Call Trace:
2013-03-15 19:41:05 [1239583.846718] [<ffffffff81132440>] ?
__lock_page+0x70/0x70
2013-03-15 19:41:05 [1239583.852983] [<ffffffff816a80c9>]
schedule+0x29/0x70
2013-03-15 19:41:05 [1239583.858748] [<ffffffff816a819f>]
io_schedule+0x8f/0xd0
2013-03-15 19:41:05 [1239583.864809] [<ffffffff8113244e>]
sleep_on_page+0xe/0x20
2013-03-15 19:41:05 [1239583.870963] [<ffffffff816a6870>]
__wait_on_bit+0x60/0x90
2013-03-15 19:41:05 [1239583.877218] [<ffffffff81132b81>] ?
find_get_pages_tag+0xd1/0x180
2013-03-15 19:41:05 [1239583.884249] [<ffffffff81132580>]
wait_on_page_bit+0x80/0x90
2013-03-15 19:41:05 [1239583.890791] [<ffffffff81079a00>] ?
autoremove_wake_function+0x40/0x40
2013-03-15 19:41:05 [1239583.898306] [<ffffffff81132951>]
filemap_fdatawait_range+0x101/0x190
2013-03-15 19:41:05 [1239583.907691] [<ffffffff8135ba49>] ?
submit_bio+0x79/0x160
2013-03-15 19:41:05 [1239583.913946] [<ffffffff811c6cd5>] ?
bio_alloc_bioset+0x65/0x120
2013-03-15 19:41:05 [1239583.920776] [<ffffffff81132a07>]
filemap_fdatawait+0x27/0x30
2013-03-15 19:41:05 [1239583.927419] [<ffffffff8128b77c>]
jbd2_journal_commit_transaction+0xa9c/0x19d0
2013-03-15 19:41:05 [1239583.935733] [<ffffffff81042af9>] ?
default_spin_lock_flags+0x9/0x10
2013-03-15 19:41:05 [1239583.943055] [<ffffffff8129113d>]
kjournald2+0xbd/0x240
2013-03-15 19:41:05 [1239583.949110] [<ffffffff810799c0>] ?
finish_wait+0x80/0x80
2013-03-15 19:41:05 [1239583.955364] [<ffffffff81291080>] ?
jbd2_journal_release_jbd_inode+0x130/0x130
2013-03-15 19:41:05 [1239583.963676] [<ffffffff81079070>] kthread+0xc0/0xd0
2013-03-15 19:41:05 [1239583.969345] [<ffffffff81078fb0>] ?
kthread_create_on_node+0x130/0x130
2013-03-15 19:41:05 [1239583.976861] [<ffffffff816b17ac>]
ret_from_fork+0x7c/0xb0
2013-03-15 19:41:05 [1239583.983115] [<ffffffff81078fb0>] ?
kthread_create_on_node+0x130/0x130
2013-03-15 19:41:05 [1239583.990627] INFO: task jbd2/sda3-8:1930 blocked
for more than 120 seconds.
2013-03-15 19:41:05 [1239583.998526] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2013-03-15 19:41:05 [1239584.007514] jbd2/sda3-8 D
2013-03-15 19:41:05 ffff880227c33c80
2013-03-15 19:41:05 0 1930 2 0x00000000
2013-03-15 19:41:05 [1239584.015662] ffff8802234afad8
2013-03-15 19:41:05 0000000000000046
2013-03-15 19:41:05 ffff8802234b2e40
2013-03-15 19:41:05 ffff8802234affd8
2013-03-15 19:41:05
2013-03-15 19:41:05 [1239584.024223] ffff8802234affd8
2013-03-15 19:41:05 ffff8802234affd8
2013-03-15 19:41:05 ffff8802248c9720
2013-03-15 19:41:05 ffff8802234b2e40
2013-03-15 19:41:05
2013-03-15 19:41:05 [1239584.032796] ffff8802234afaa8
2013-03-15 19:41:05 ffff8802234b2e40
2013-03-15 19:41:05 ffff880227c344e8
2013-03-15 19:41:05 0000000000000002
2013-03-15 19:41:05
2013-03-15 19:41:05 [1239584.041354] Call Trace:
2013-03-15 19:41:05 [1239584.044311] [<ffffffff81132440>] ?
__lock_page+0x70/0x70
2013-03-15 19:41:05 [1239584.050561] [<ffffffff816a80c9>]
schedule+0x29/0x70
2013-03-15 19:41:05 [1239584.056332] [<ffffffff816a819f>]
io_schedule+0x8f/0xd0
2013-03-15 19:41:05 [1239584.062386] [<ffffffff8113244e>]
sleep_on_page+0xe/0x20
2013-03-15 19:41:05 [1239584.068541] [<ffffffff816a6870>]
__wait_on_bit+0x60/0x90
2013-03-15 19:41:05 [1239584.074795] [<ffffffff81132b81>] ?
find_get_pages_tag+0xd1/0x180
2013-03-15 19:41:05 [1239584.081820] [<ffffffff81132580>]
wait_on_page_bit+0x80/0x90
2013-03-15 19:41:05 [1239584.088366] [<ffffffff81079a00>] ?
autoremove_wake_function+0x40/0x40
2013-03-15 19:41:05 [1239584.095880] [<ffffffff81132951>]
filemap_fdatawait_range+0x101/0x190
2013-03-15 19:41:05 [1239584.103299] [<ffffffff8135ba49>] ?
submit_bio+0x79/0x160
2013-03-15 19:41:05 [1239584.109549] [<ffffffff811c6cd5>] ?
bio_alloc_bioset+0x65/0x120
2013-03-15 19:41:05 [1239584.116385] [<ffffffff81132a07>]
filemap_fdatawait+0x27/0x30
2013-03-15 19:41:05 [1239584.123028] [<ffffffff8128b77c>]
jbd2_journal_commit_transaction+0xa9c/0x19d0
2013-03-15 19:41:05 [1239584.131336] [<ffffffff81042af9>] ?
default_spin_lock_flags+0x9/0x10
2013-03-15 19:41:05 [1239584.138656] [<ffffffff8129113d>]
kjournald2+0xbd/0x240
2013-03-15 19:41:05 [1239584.144717] [<ffffffff810799c0>] ?
finish_wait+0x80/0x80
2013-03-15 19:41:05 [1239584.150958] [<ffffffff81291080>] ?
jbd2_journal_release_jbd_inode+0x130/0x130
2013-03-15 19:41:05 [1239584.159269] [<ffffffff81079070>] kthread+0xc0/0xd0
2013-03-15 19:41:05 [1239584.164943] [<ffffffff81078fb0>] ?
kthread_create_on_node+0x130/0x130
2013-03-15 19:41:05 [1239584.172467] [<ffffffff816b17ac>]
ret_from_fork+0x7c/0xb0
2013-03-15 19:41:05 [1239584.178716] [<ffffffff81078fb0>] ?
kthread_create_on_node+0x130/0x130
2013-03-15 19:43:01 [1239700.250748] INFO: rcu_sched detected stalls on
CPUs/tasks:
2013-03-15 19:43:01
2013-03-15 19:43:01 [1239700.257116] 0: (2 GPs behind)
idle=d99/140000000000001/0 drain=0 . timer not pending
2013-03-15 19:43:01 [1239700.266100]
2013-03-15 19:43:01 (detected by 2, t=60018 jiffies)
2013-03-15 19:43:01 [1239700.271202] Pid: 0, comm: swapper/2 Not tainted
3.7.9-vs2.3.5.6-rol-em64t-tg3+sdfix #3
2013-03-15 19:43:01 [1239700.280288] Call Trace:
2013-03-15 19:43:01 [1239700.283242] <IRQ>
2013-03-15 19:43:01 [<ffffffff810f8e15>] rcu_pending+0x535/0x550
2013-03-15 19:43:01 [1239700.290208] [<ffffffff810facc2>]
rcu_check_callbacks+0xe2/0x180
2013-03-15 19:43:01 [1239700.297141] [<ffffffff810663b8>]
update_process_times+0x48/0x90
2013-03-15 19:43:01 [1239700.304073] [<ffffffff810bb4e3>]
tick_sched_timer+0x83/0xd0
2013-03-15 19:43:01 [1239700.310610] [<ffffffff8107d479>]
__run_hrtimer+0x79/0x1d0
2013-03-15 19:43:01 [1239700.316956] [<ffffffff810bb460>] ?
tick_nohz_handler+0xf0/0xf0
2013-03-15 19:43:01 [1239700.323791] [<ffffffff8107dd97>]
hrtimer_interrupt+0xf7/0x230
2013-03-15 19:43:01 [1239700.330529] [<ffffffff816b3279>]
smp_apic_timer_interrupt+0x69/0x99
2013-03-15 19:43:01 [1239700.337851] [<ffffffff816b230a>]
apic_timer_interrupt+0x6a/0x70
2013-03-15 19:43:01 [1239700.344781] <EOI>
2013-03-15 19:43:02 [<ffffffff813d521d>] ? intel_idle+0xed/0x150
2013-03-15 19:43:02 [1239700.351743] [<ffffffff813d51fe>] ?
intel_idle+0xce/0x150
2013-03-15 19:43:02 [1239700.357997] [<ffffffff8156b559>]
cpuidle_enter+0x19/0x20
2013-03-15 19:43:02 [1239700.364249] [<ffffffff8156bc22>]
cpuidle_idle_call+0xa2/0x1b0
2013-03-15 19:43:02 [1239700.370980] [<ffffffff8101d02f>]
cpu_idle+0xaf/0x120
2013-03-15 19:43:02 [1239700.376846] [<ffffffff81695c9d>]
start_secondary+0x1d4/0x1db
2013-03-15 19:46:01 [1239880.019328] INFO: rcu_sched detected stalls on
CPUs/tasks:
2013-03-15 19:46:01
2013-03-15 19:46:01 [1239880.025693] 0: (2 GPs behind)
idle=d99/140000000000001/0 drain=0 . timer not pending
2013-03-15 19:46:01 [1239880.034683]
2013-03-15 19:46:01 (detected by 2, t=78023 jiffies)
2013-03-15 19:46:01 [1239880.039784] Pid: 0, comm: swapper/2 Not tainted
3.7.9-vs2.3.5.6-rol-em64t-tg3+sdfix #3
2013-03-15 19:46:01 [1239880.048870] Call Trace:
2013-03-15 19:46:01 [1239880.051824] <IRQ>
2013-03-15 19:46:01 [<ffffffff810f8e15>] rcu_pending+0x535/0x550
2013-03-15 19:46:01 [1239880.058788] [<ffffffff810facc2>]
rcu_check_callbacks+0xe2/0x180
2013-03-15 19:46:02 [1239880.065722] [<ffffffff810663b8>]
update_process_times+0x48/0x90
2013-03-15 19:46:02 [1239880.072655] [<ffffffff810bb4e3>]
tick_sched_timer+0x83/0xd0
2013-03-15 19:46:02 [1239880.079200] [<ffffffff8107d479>]
__run_hrtimer+0x79/0x1d0
2013-03-15 19:46:02 [1239880.085540] [<ffffffff810bb460>] ?
tick_nohz_handler+0xf0/0xf0
2013-03-15 19:46:02 [1239880.092376] [<ffffffff8107dd97>]
hrtimer_interrupt+0xf7/0x230
2013-03-15 19:46:02 [1239880.099115] [<ffffffff816b3279>]
smp_apic_timer_interrupt+0x69/0x99
2013-03-15 19:46:02 [1239880.106435] [<ffffffff816b230a>]
apic_timer_interrupt+0x6a/0x70
2013-03-15 19:46:02 [1239880.113365] <EOI>
2013-03-15 19:46:02 [<ffffffff813d521d>] ? intel_idle+0xed/0x150
2013-03-15 19:46:02 [1239880.120328] [<ffffffff813d51fe>] ?
intel_idle+0xce/0x150
2013-03-15 19:46:02 [1239880.126581] [<ffffffff8156b559>]
cpuidle_enter+0x19/0x20
2013-03-15 19:46:02 [1239880.132839] [<ffffffff8156bc22>]
cpuidle_idle_call+0xa2/0x1b0
2013-03-15 19:46:02 [1239880.139569] [<ffffffff8101d02f>]
cpu_idle+0xaf/0x120
2013-03-15 19:46:02 [1239880.145428] [<ffffffff81695c9d>]
start_secondary+0x1d4/0x1db
2013-03-15 19:49:02 [1240059.787917] INFO: rcu_sched detected stalls on
CPUs/tasks:
2013-03-15 19:49:02
2013-03-15 19:49:02 [1240059.794279] 0: (2 GPs behind)
idle=d99/140000000000001/0 drain=0 . timer not pending
2013-03-15 19:49:02 [1240059.803260]
2013-03-15 19:49:02 (detected by 2, t=96028 jiffies)
2013-03-15 19:49:02 [1240059.808354] Pid: 0, comm: swapper/2 Not tainted
3.7.9-vs2.3.5.6-rol-em64t-tg3+sdfix #3
2013-03-15 19:49:02 [1240059.817435] Call Trace:
2013-03-15 19:49:02 [1240059.820380] <IRQ>
2013-03-15 19:49:02 [<ffffffff810f8e15>] rcu_pending+0x535/0x550
2013-03-15 19:49:02 [1240059.827330] [<ffffffff810facc2>]
rcu_check_callbacks+0xe2/0x180
2013-03-15 19:49:02 [1240059.834255] [<ffffffff810663b8>]
update_process_times+0x48/0x90
2013-03-15 19:49:02 [1240059.841179] [<ffffffff810bb4e3>]
tick_sched_timer+0x83/0xd0
2013-03-15 19:49:02 [1240059.847718] [<ffffffff8107d479>]
__run_hrtimer+0x79/0x1d0
2013-03-15 19:49:02 [1240059.854060] [<ffffffff810bb460>] ?
tick_nohz_handler+0xf0/0xf0
2013-03-15 19:49:02 [1240059.860887] [<ffffffff8107dd97>]
hrtimer_interrupt+0xf7/0x230
2013-03-15 19:49:02 [1240059.867618] [<ffffffff816b3279>]
smp_apic_timer_interrupt+0x69/0x99
2013-03-15 19:49:02 [1240059.874997] [<ffffffff816b230a>]
apic_timer_interrupt+0x6a/0x70
2013-03-15 19:49:02 [1240059.881920] <EOI>
2013-03-15 19:49:02 [<ffffffff813d521d>] ? intel_idle+0xed/0x150
2013-03-15 19:49:02 [1240059.888867] [<ffffffff813d51fe>] ?
intel_idle+0xce/0x150
2013-03-15 19:49:02 [1240059.895112] [<ffffffff8156b559>]
cpuidle_enter+0x19/0x20
2013-03-15 19:49:02 [1240059.901358] [<ffffffff8156bc22>]
cpuidle_idle_call+0xa2/0x1b0
2013-03-15 19:49:02 [1240059.908088] [<ffffffff8101d02f>]
cpu_idle+0xaf/0x120
2013-03-15 19:49:02 [1240059.913948] [<ffffffff81695c9d>]
start_secondary+0x1d4/0x1db
2013-03-15 19:52:02 [1240239.556500] INFO: rcu_sched detected stalls on
CPUs/tasks:
2013-03-15 19:52:02
2013-03-15 19:52:02 [1240239.562862] 0: (2 GPs behind)
idle=d99/140000000000001/0 drain=0 . timer not pending
2013-03-15 19:52:02 [1240239.571844]
2013-03-15 19:52:02 (detected by 2, t=114032 jiffies)
2013-03-15 19:52:02 [1240239.577042] Pid: 0, comm: swapper/2 Not tainted
3.7.9-vs2.3.5.6-rol-em64t-tg3+sdfix #3
2013-03-15 19:52:02 [1240239.586127] Call Trace:
2013-03-15 19:52:02 [1240239.589074] <IRQ>
2013-03-15 19:52:02 [<ffffffff810f8e15>] rcu_pending+0x535/0x550
2013-03-15 19:52:02 [1240239.596025] [<ffffffff810facc2>]
rcu_check_callbacks+0xe2/0x180
2013-03-15 19:52:02 [1240239.603044] [<ffffffff810663b8>]
update_process_times+0x48/0x90
2013-03-15 19:52:02 [1240239.609970] [<ffffffff810bb4e3>]
tick_sched_timer+0x83/0xd0
2013-03-15 19:52:02 [1240239.616511] [<ffffffff8107d479>]
__run_hrtimer+0x79/0x1d0
2013-03-15 19:52:02 [1240239.622854] [<ffffffff810bb460>] ?
tick_nohz_handler+0xf0/0xf0
2013-03-15 19:52:02 [1240239.629674] [<ffffffff8107dd97>]
hrtimer_interrupt+0xf7/0x230
2013-03-15 19:52:02 [1240239.636413] [<ffffffff816b3279>]
smp_apic_timer_interrupt+0x69/0x99
2013-03-15 19:52:02 [1240239.643719] [<ffffffff816b230a>]
apic_timer_interrupt+0x6a/0x70
2013-03-15 19:52:02 [1240239.650641] <EOI>
2013-03-15 19:52:02 [<ffffffff813d521d>] ? intel_idle+0xed/0x150
2013-03-15 19:52:02 [1240239.657589] [<ffffffff813d51fe>] ?
intel_idle+0xce/0x150
2013-03-15 19:52:02 [1240239.663834] [<ffffffff8156b559>]
cpuidle_enter+0x19/0x20
2013-03-15 19:52:02 [1240239.670080] [<ffffffff8156bc22>]
cpuidle_idle_call+0xa2/0x1b0
2013-03-15 19:52:02 [1240239.676801] [<ffffffff8101d02f>]
cpu_idle+0xaf/0x120
2013-03-15 19:52:02 [1240239.682660] [<ffffffff81695c9d>]
start_secondary+0x1d4/0x1db
2013-03-15 19:55:02 [1240419.325086] INFO: rcu_sched detected stalls on
CPUs/tasks:
2013-03-15 19:55:02
2013-03-15 19:55:02 [1240419.331445] 0: (2 GPs behind)
idle=d99/140000000000001/0 drain=0 . timer not pending
2013-03-15 19:55:02 [1240419.340428]
2013-03-15 19:55:02 (detected by 2, t=132038 jiffies)
2013-03-15 19:55:02 [1240419.345620] Pid: 0, comm: swapper/2 Not tainted
3.7.9-vs2.3.5.6-rol-em64t-tg3+sdfix #3
2013-03-15 19:55:02 [1240419.354700] Call Trace:
2013-03-15 19:55:02 [1240419.357647] <IRQ>
2013-03-15 19:55:02 [<ffffffff810f8e15>] rcu_pending+0x535/0x550
2013-03-15 19:55:02 [1240419.364597] [<ffffffff810facc2>]
rcu_check_callbacks+0xe2/0x180
2013-03-15 19:55:02 [1240419.371521] [<ffffffff810663b8>]
update_process_times+0x48/0x90
2013-03-15 19:55:02 [1240419.378446] [<ffffffff810bb4e3>]
tick_sched_timer+0x83/0xd0
2013-03-15 19:55:02 [1240419.384982] [<ffffffff8107d479>]
__run_hrtimer+0x79/0x1d0
2013-03-15 19:55:02 [1240419.391324] [<ffffffff810bb460>] ?
tick_nohz_handler+0xf0/0xf0
2013-03-15 19:55:02 [1240419.398156] [<ffffffff8107dd97>]
hrtimer_interrupt+0xf7/0x230
2013-03-15 19:55:02 [1240419.404888] [<ffffffff816b3279>]
smp_apic_timer_interrupt+0x69/0x99
2013-03-15 19:55:02 [1240419.412201] [<ffffffff816b230a>]
apic_timer_interrupt+0x6a/0x70
2013-03-15 19:55:02 [1240419.419123] <EOI>
2013-03-15 19:55:02 [<ffffffff813d521d>] ? intel_idle+0xed/0x150
2013-03-15 19:55:02 [1240419.426072] [<ffffffff813d51fe>] ?
intel_idle+0xce/0x150
2013-03-15 19:55:02 [1240419.432317] [<ffffffff8156b559>]
cpuidle_enter+0x19/0x20
2013-03-15 19:55:02 [1240419.438562] [<ffffffff8156bc22>]
cpuidle_idle_call+0xa2/0x1b0
2013-03-15 19:55:02 [1240419.445294] [<ffffffff8101d02f>]
cpu_idle+0xaf/0x120
2013-03-15 19:55:02 [1240419.451151] [<ffffffff81695c9d>]
start_secondary+0x1d4/0x1db
Have you any idea what this could be?
Please CC me, as I'm not a member of the LKML.
Many thanks and regards
Urban Loesch
^ permalink raw reply [flat|nested] only message in thread
only message in thread, other threads:[~2013-03-15 21:57 UTC | newest]
Thread overview: (only message) (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2013-03-15 21:57 Kernel 3.7.9 rcu_sched detected stalls on CPUs/tasks: Urban Loesch
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.