From: Urban Loesch <bind@enas.net>
To: linux-kernel@vger.kernel.org
Cc: bind@enas.net
Subject: Kernel 3.7.9 rcu_sched detected stalls on CPUs/tasks:
Date: Fri, 15 Mar 2013 22:57:06 +0100 [thread overview]
Message-ID: <51439932.6060209@enas.net> (raw)
Hi,
I'm running a new DELL PER420 with 2x Intel Xeon E5-2440 2,40 Ghz and
16GB of RAM with Kernel 3.7.9 and Linux vserver patch vs2.3.5.6 from
http://linux-vserver.org. Uptime was about 14 days, 8 hours.
Today evening the system has become not responsive and the netconsole
got me the following log entries. I had to hard reset the machine to got
it to work again. There was no output on the screen, too.
Logs entries I got:
2013-03-15 19:34:01 [1239160.944985] INFO: rcu_sched detected stalls on
CPUs/tasks:
2013-03-15 19:34:01
2013-03-15 19:34:01 [1239160.951430] 0: (2 GPs behind)
idle=d99/140000000000001/0 drain=0 . timer not pending
2013-03-15 19:34:01 [1239160.960413]
2013-03-15 19:34:01 (detected by 6, t=6003 jiffies)
2013-03-15 19:34:01 [1239160.965429] Pid: 0, comm: swapper/6 Not tainted
3.7.9-vs2.3.5.6-rol-em64t-tg3+sdfix #3
2013-03-15 19:34:01 [1239160.974516] Call Trace:
2013-03-15 19:34:01 [1239160.977470] <IRQ>
2013-03-15 19:34:01 [<ffffffff810f8e15>] rcu_pending+0x535/0x550
2013-03-15 19:34:01 [1239160.984439] [<ffffffff810facc2>]
rcu_check_callbacks+0xe2/0x180
2013-03-15 19:34:01 [1239160.991373] [<ffffffff810663b8>]
update_process_times+0x48/0x90
2013-03-15 19:34:01 [1239160.998307] [<ffffffff810bb4e3>]
tick_sched_timer+0x83/0xd0
2013-03-15 19:34:01 [1239161.004844] [<ffffffff8107d479>]
__run_hrtimer+0x79/0x1d0
2013-03-15 19:34:01 [1239161.011195] [<ffffffff810bb460>] ?
tick_nohz_handler+0xf0/0xf0
2013-03-15 19:34:01 [1239161.019993] [<ffffffff8107dd97>]
hrtimer_interrupt+0xf7/0x230
2013-03-15 19:34:01 [1239161.026732] [<ffffffff816b3279>]
smp_apic_timer_interrupt+0x69/0x99
2013-03-15 19:34:01 [1239161.034053] [<ffffffff816b230a>]
apic_timer_interrupt+0x6a/0x70
2013-03-15 19:34:01 [1239161.040985] <EOI>
2013-03-15 19:34:01 [<ffffffff8107d937>] ?
__hrtimer_start_range_ns+0x1b7/0x400
2013-03-15 19:34:01 [1239161.049435] [<ffffffff813d521d>] ?
intel_idle+0xed/0x150
2013-03-15 19:34:01 [1239161.055687] [<ffffffff813d51fe>] ?
intel_idle+0xce/0x150
2013-03-15 19:34:01 [1239161.061940] [<ffffffff8156b559>]
cpuidle_enter+0x19/0x20
2013-03-15 19:34:01 [1239161.068192] [<ffffffff8156bc22>]
cpuidle_idle_call+0xa2/0x1b0
2013-03-15 19:34:01 [1239161.074924] [<ffffffff8101d02f>]
cpu_idle+0xaf/0x120
2013-03-15 19:34:01 [1239161.080791] [<ffffffff81695c9d>]
start_secondary+0x1d4/0x1db
2013-03-15 19:33:58 kernel: [1239160.944985] INFO: rcu_sched detected
stalls on CPUs/tasks:
2013-03-15 19:33:58 kernel: [1239160.951430] 0: (2 GPs behind)
idle=d99/140000000000001/0 drain=0 . timer not pending
2013-03-15 19:33:58 kernel: [1239160.960413] (detected by 6, t=6003
jiffies)
2013-03-15 19:33:58 kernel: [1239160.965429] Pid: 0, comm: swapper/6 Not
tainted 3.7.9-vs2.3.5.6-rol-em64t-tg3+sdfix #3
2013-03-15 19:33:58 kernel: [1239160.974516] Call Trace:
2013-03-15 19:33:58 kernel: [1239160.977470] <IRQ>
[<ffffffff810f8e15>] rcu_pending+0x535/0x550
2013-03-15 19:33:58 kernel: [1239160.984439] [<ffffffff810facc2>]
rcu_check_callbacks+0xe2/0x180
2013-03-15 19:33:58 kernel: [1239160.991373] [<ffffffff810663b8>]
update_process_times+0x48/0x90
2013-03-15 19:33:58 kernel: [1239160.998307] [<ffffffff810bb4e3>]
tick_sched_timer+0x83/0xd0
2013-03-15 19:33:58 kernel: [1239161.004844] [<ffffffff8107d479>]
__run_hrtimer+0x79/0x1d0
2013-03-15 19:33:58 kernel: [1239161.011195] [<ffffffff810bb460>] ?
tick_nohz_handler+0xf0/0xf0
2013-03-15 19:33:58 kernel: [1239161.019993] [<ffffffff8107dd97>]
hrtimer_interrupt+0xf7/0x230
2013-03-15 19:33:58 kernel: [1239161.026732] [<ffffffff816b3279>]
smp_apic_timer_interrupt+0x69/0x99
2013-03-15 19:33:58 kernel: [1239161.034053] [<ffffffff816b230a>]
apic_timer_interrupt+0x6a/0x70
2013-03-15 19:37:01 [1239340.713569] INFO: rcu_sched detected stalls on
CPUs/tasks:
2013-03-15 19:37:01
2013-03-15 19:37:01 [1239340.719931] 0: (2 GPs behind)
idle=d99/140000000000001/0 drain=0 . timer not pending
2013-03-15 19:37:01 [1239340.728915]
2013-03-15 19:37:01 (detected by 6, t=24008 jiffies)
2013-03-15 19:37:01 [1239340.734016] Pid: 0, comm: swapper/6 Not tainted
3.7.9-vs2.3.5.6-rol-em64t-tg3+sdfix #3
2013-03-15 19:37:01 [1239340.743103] Call Trace:
2013-03-15 19:37:01 [1239340.746057] <IRQ>
2013-03-15 19:37:01 [<ffffffff810f8e15>] rcu_pending+0x535/0x550
2013-03-15 19:37:01 [1239340.753023] [<ffffffff810facc2>]
rcu_check_callbacks+0xe2/0x180
2013-03-15 19:37:01 [1239340.759956] [<ffffffff810663b8>]
update_process_times+0x48/0x90
2013-03-15 19:37:01 [1239340.766888] [<ffffffff810bb4e3>]
tick_sched_timer+0x83/0xd0
2013-03-15 19:37:01 [1239340.773431] [<ffffffff8107d479>]
__run_hrtimer+0x79/0x1d0
2013-03-15 19:37:01 [1239340.779781] [<ffffffff810bb460>] ?
tick_nohz_handler+0xf0/0xf0
2013-03-15 19:37:01 [1239340.786615] [<ffffffff8107dd97>]
hrtimer_interrupt+0xf7/0x230
2013-03-15 19:37:01 [1239340.793353] [<ffffffff816b3279>]
smp_apic_timer_interrupt+0x69/0x99
2013-03-15 19:37:01 [1239340.800673] [<ffffffff816b230a>]
apic_timer_interrupt+0x6a/0x70
2013-03-15 19:37:01 [1239340.807604] <EOI>
2013-03-15 19:37:01 [<ffffffff8107d937>] ?
__hrtimer_start_range_ns+0x1b7/0x400
2013-03-15 19:37:01 [1239340.816045] [<ffffffff813d521d>] ?
intel_idle+0xed/0x150
2013-03-15 19:37:01 [1239340.822297] [<ffffffff813d51fe>] ?
intel_idle+0xce/0x150
2013-03-15 19:37:01 [1239340.828550] [<ffffffff8156b559>]
cpuidle_enter+0x19/0x20
2013-03-15 19:37:01 [1239340.834801] [<ffffffff8156bc22>]
cpuidle_idle_call+0xa2/0x1b0
2013-03-15 19:37:01 [1239340.841539] [<ffffffff8101d02f>]
cpu_idle+0xaf/0x120
2013-03-15 19:37:01 [1239340.847397] [<ffffffff81695c9d>]
start_secondary+0x1d4/0x1db
2013-03-15 19:37:03 [1239342.520795] INFO: task jbd2/sda2-8:416 blocked
for more than 120 seconds.
2013-03-15 19:37:03 [1239342.528610] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2013-03-15 19:37:03 [1239342.537608] jbd2/sda2-8 D
2013-03-15 19:37:03 ffff88042fc53c80
2013-03-15 19:37:03 0 416 2 0x00000000
2013-03-15 19:37:03 [1239342.545759] ffff880221ab1ad8
2013-03-15 19:37:03 0000000000000046
2013-03-15 19:37:03 ffff880221de9720
2013-03-15 19:37:03 ffff880221ab1fd8
2013-03-15 19:37:03
2013-03-15 19:37:03 [1239342.554330] ffff880221ab1fd8
2013-03-15 19:37:03 ffff880221ab1fd8
2013-03-15 19:37:03 ffff8802248cdc80
2013-03-15 19:37:03 ffff880221de9720
2013-03-15 19:37:03
2013-03-15 19:37:03 [1239342.562892] ffff880221ab1aa8
2013-03-15 19:37:03 ffff880221de9720
2013-03-15 19:37:03 ffff88042fc544e8
2013-03-15 19:37:03 0000000000000002
2013-03-15 19:37:03
2013-03-15 19:37:03 [1239342.571465] Call Trace:
2013-03-15 19:37:03 [1239342.574426] [<ffffffff81132440>] ?
__lock_page+0x70/0x70
2013-03-15 19:37:03 [1239342.580684] [<ffffffff816a80c9>]
schedule+0x29/0x70
2013-03-15 19:37:03 [1239342.586449] [<ffffffff816a819f>]
io_schedule+0x8f/0xd0
2013-03-15 19:37:03 [1239342.592510] [<ffffffff8113244e>]
sleep_on_page+0xe/0x20
2013-03-15 19:37:03 [1239342.598653] [<ffffffff816a6870>]
__wait_on_bit+0x60/0x90
2013-03-15 19:37:03 [1239342.604907] [<ffffffff81132b81>] ?
find_get_pages_tag+0xd1/0x180
2013-03-15 19:37:03 [1239342.611937] [<ffffffff81132580>]
wait_on_page_bit+0x80/0x90
2013-03-15 19:37:03 [1239342.618484] [<ffffffff81079a00>] ?
autoremove_wake_function+0x40/0x40
2013-03-15 19:37:03 [1239342.625998] [<ffffffff81132951>]
filemap_fdatawait_range+0x101/0x190
2013-03-15 19:37:03 [1239342.633422] [<ffffffff8135ba49>] ?
submit_bio+0x79/0x160
2013-03-15 19:37:03 [1239342.639676] [<ffffffff811c6cd5>] ?
bio_alloc_bioset+0x65/0x120
2013-03-15 19:37:03 [1239342.646512] [<ffffffff81132a07>]
filemap_fdatawait+0x27/0x30
2013-03-15 19:37:03 [1239342.653160] [<ffffffff8128b77c>]
jbd2_journal_commit_transaction+0xa9c/0x19d0
2013-03-15 19:37:03 [1239342.661475] [<ffffffff81042af9>] ?
default_spin_lock_flags+0x9/0x10
2013-03-15 19:37:03 [1239342.668792] [<ffffffff8129113d>]
kjournald2+0xbd/0x240
2013-03-15 19:37:03 [1239342.674852] [<ffffffff810799c0>] ?
finish_wait+0x80/0x80
2013-03-15 19:37:03 [1239342.681099] [<ffffffff81291080>] ?
jbd2_journal_release_jbd_inode+0x130/0x130
2013-03-15 19:37:03 [1239342.689407] [<ffffffff81079070>] kthread+0xc0/0xd0
2013-03-15 19:37:03 [1239342.695080] [<ffffffff81078fb0>] ?
kthread_create_on_node+0x130/0x130
2013-03-15 19:37:03 [1239342.702596] [<ffffffff816b17ac>]
ret_from_fork+0x7c/0xb0
2013-03-15 19:37:03 [1239342.708846] [<ffffffff81078fb0>] ?
kthread_create_on_node+0x130/0x130
2013-03-15 19:37:03 [1239342.716361] INFO: task jbd2/sda3-8:1930 blocked
for more than 120 seconds.
2013-03-15 19:37:03 [1239342.724259] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2013-03-15 19:37:03 [1239342.733247] jbd2/sda3-8 D
2013-03-15 19:37:03 ffff880227c33c80
2013-03-15 19:37:03 0 1930 2 0x00000000
2013-03-15 19:37:03 [1239342.741394] ffff8802234afad8
2013-03-15 19:37:03 0000000000000046
2013-03-15 19:37:03 ffff8802234b2e40
2013-03-15 19:37:03 ffff8802234affd8
2013-03-15 19:37:03
2013-03-15 19:37:03 [1239342.749960] ffff8802234affd8
2013-03-15 19:37:03 ffff8802234affd8
2013-03-15 19:37:03 ffff8802248c9720
2013-03-15 19:37:03 ffff8802234b2e40
2013-03-15 19:37:03
2013-03-15 19:37:03 [1239342.758527] ffff8802234afaa8
2013-03-15 19:37:03 ffff8802234b2e40
2013-03-15 19:37:03 ffff880227c344e8
2013-03-15 19:37:03 0000000000000002
2013-03-15 19:37:03
2013-03-15 19:37:03 [1239342.767097] Call Trace:
2013-03-15 19:37:03 [1239342.770049] [<ffffffff81132440>] ?
__lock_page+0x70/0x70
2013-03-15 19:37:03 [1239342.776303] [<ffffffff816a80c9>]
schedule+0x29/0x70
2013-03-15 19:37:03 [1239342.782074] [<ffffffff816a819f>]
io_schedule+0x8f/0xd0
2013-03-15 19:37:03 [1239342.788131] [<ffffffff8113244e>]
sleep_on_page+0xe/0x20
2013-03-15 19:37:03 [1239342.794288] [<ffffffff816a6870>]
__wait_on_bit+0x60/0x90
2013-03-15 19:37:03 [1239342.800534] [<ffffffff81132b81>] ?
find_get_pages_tag+0xd1/0x180
2013-03-15 19:37:03 [1239342.807553] [<ffffffff81132580>]
wait_on_page_bit+0x80/0x90
2013-03-15 19:37:03 [1239342.814098] [<ffffffff81079a00>] ?
autoremove_wake_function+0x40/0x40
2013-03-15 19:37:03 [1239342.821612] [<ffffffff81132951>]
filemap_fdatawait_range+0x101/0x190
2013-03-15 19:37:03 [1239342.829027] [<ffffffff8135ba49>] ?
submit_bio+0x79/0x160
2013-03-15 19:37:03 [1239342.835282] [<ffffffff811c6cd5>] ?
bio_alloc_bioset+0x65/0x120
2013-03-15 19:37:03 [1239342.842108] [<ffffffff81132a07>]
filemap_fdatawait+0x27/0x30
2013-03-15 19:37:03 [1239342.848748] [<ffffffff8128b77c>]
jbd2_journal_commit_transaction+0xa9c/0x19d0
2013-03-15 19:37:03 [1239342.857059] [<ffffffff81042af9>] ?
default_spin_lock_flags+0x9/0x10
2013-03-15 19:37:03 [1239342.864377] [<ffffffff8129113d>]
kjournald2+0xbd/0x240
2013-03-15 19:37:03 [1239342.870438] [<ffffffff810799c0>] ?
finish_wait+0x80/0x80
2013-03-15 19:37:03 [1239342.876698] [<ffffffff81291080>] ?
jbd2_journal_release_jbd_inode+0x130/0x130
2013-03-15 19:37:03 [1239342.885013] [<ffffffff81079070>] kthread+0xc0/0xd0
2013-03-15 19:37:03 [1239342.890687] [<ffffffff81078fb0>] ?
kthread_create_on_node+0x130/0x130
2013-03-15 19:37:03 [1239342.898190] [<ffffffff816b17ac>]
ret_from_fork+0x7c/0xb0
2013-03-15 19:37:04 [1239342.904445] [<ffffffff81078fb0>] ?
kthread_create_on_node+0x130/0x130
2013-03-15 19:37:04 [1239342.911962] INFO: task flush-8:0:3588 blocked
for more than 120 seconds.
2013-03-15 19:37:04 [1239342.919664] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2013-03-15 19:37:04 [1239342.928652] flush-8:0 D
2013-03-15 19:37:04 ffff88042fc53c80
2013-03-15 19:37:04 0 3588 2 0x00000000
2013-03-15 19:37:04 [1239342.936798] ffff880221fcf698
2013-03-15 19:37:04 0000000000000046
2013-03-15 19:37:04 ffff8802233f5c80
2013-03-15 19:37:04 ffff880221fcffd8
2013-03-15 19:37:04
2013-03-15 19:37:04 [1239342.945366] ffff880221fcffd8
2013-03-15 19:37:04 ffff880221fcffd8
2013-03-15 19:37:04 ffff8802248cdc80
2013-03-15 19:37:04 ffff8802233f5c80
2013-03-15 19:37:04
2013-03-15 19:37:04 [1239342.953916] ffff880221fcf6a8
2013-03-15 19:37:04 ffff8801ed90daf8
2013-03-15 19:37:04 ffff8803db412070
2013-03-15 19:37:04 ffff880423779ca8
2013-03-15 19:37:04
2013-03-15 19:37:04 [1239342.962483] Call Trace:
2013-03-15 19:37:04 [1239342.965434] [<ffffffff816a80c9>]
schedule+0x29/0x70
2013-03-15 19:37:04 [1239342.971206] [<ffffffff8128a1dd>]
do_get_write_access+0x2cd/0x4d0
2013-03-15 19:37:04 [1239342.978235] [<ffffffff8137e53e>] ?
radix_tree_lookup_slot+0xe/0x10
2013-03-15 19:37:04 [1239342.985458] [<ffffffff81079a00>] ?
autoremove_wake_function+0x40/0x40
2013-03-15 19:37:04 [1239342.992976] [<ffffffff8128a520>]
jbd2_journal_get_write_access+0x30/0x50
2013-03-15 19:37:04 [1239343.000785] [<ffffffff8126c31d>]
__ext4_journal_get_write_access+0x3d/0x80
2013-03-15 19:37:04 [1239343.008786] [<ffffffff812354b5>] ?
ext4_read_block_bitmap+0x35/0x60
2013-03-15 19:37:04 [1239343.016106] [<ffffffff8127279b>]
ext4_mb_mark_diskspace_used+0x7b/0x4d0
2013-03-15 19:37:04 [1239343.023877] [<ffffffff8117e81f>] ?
__kmalloc+0xaf/0x180
2013-03-15 19:37:04 [1239343.030043] [<ffffffff8126d7c6>] ?
ext4_mb_initialize_context+0x86/0x1b0
2013-03-15 19:37:04 [1239343.037843] [<ffffffff81273f5b>]
ext4_mb_new_blocks+0x29b/0x550
2013-03-15 19:37:04 [1239343.044778] [<ffffffff8126a97f>]
ext4_ext_map_blocks+0xc4f/0x1610
2013-03-15 19:37:04 [1239343.051906] [<ffffffff8123c035>]
ext4_map_blocks+0x1d5/0x270
2013-03-15 19:37:04 [1239343.058541] [<ffffffff8124045d>]
mpage_da_map_and_submit+0x15d/0x4b0
2013-03-15 19:37:04 [1239343.065956] [<ffffffff81241017>] ?
ext4_da_writepages+0x317/0x5e0
2013-03-15 19:37:04 [1239343.073083] [<ffffffff81241063>]
ext4_da_writepages+0x363/0x5e0
2013-03-15 19:37:04 [1239343.080019] [<ffffffff8113e3be>]
do_writepages+0x1e/0x40
2013-03-15 19:37:04 [1239343.086269] [<ffffffff811b977b>]
__writeback_single_inode+0x3b/0x160
2013-03-15 19:37:04 [1239343.093685] [<ffffffff811ba7d0>]
writeback_sb_inodes+0x1a0/0x390
2013-03-15 19:37:04 [1239343.100714] [<ffffffff811baa5f>]
__writeback_inodes_wb+0x9f/0xd0
2013-03-15 19:37:04 [1239343.107730] [<ffffffff811bacdb>]
wb_writeback+0x24b/0x2e0
2013-03-15 19:37:04 [1239343.114080] [<ffffffff811a9b22>] ?
get_nr_inodes+0x52/0x70
2013-03-15 19:37:04 [1239343.120528] [<ffffffff811bbf76>]
wb_do_writeback+0x1f6/0x200
2013-03-15 19:37:04 [1239343.127161] [<ffffffff811bc00b>]
bdi_writeback_thread+0x8b/0x230
2013-03-15 19:37:04 [1239343.134187] [<ffffffff811bbf80>] ?
wb_do_writeback+0x200/0x200
2013-03-15 19:37:04 [1239343.141023] [<ffffffff81079070>] kthread+0xc0/0xd0
2013-03-15 19:37:04 [1239343.146680] [<ffffffff81078fb0>] ?
kthread_create_on_node+0x130/0x130
2013-03-15 19:37:04 [1239343.154184] [<ffffffff816b17ac>]
ret_from_fork+0x7c/0xb0
2013-03-15 19:37:04 [1239343.160428] [<ffffffff81078fb0>] ?
kthread_create_on_node+0x130/0x130
2013-03-15 19:37:04 [1239343.167940] INFO: task tlsmgr:6983 blocked for
more than 120 seconds.
2013-03-15 19:37:04 [1239343.175352] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2013-03-15 19:37:04 [1239343.184336] tlsmgr D
2013-03-15 19:37:04 ffff88042fc13c80
2013-03-15 19:37:04 0 6983 6611 0x00000000
2013-03-15 19:37:04 [1239343.192482] ffff8804230e3a38
2013-03-15 19:37:04 0000000000000082
2013-03-15 19:37:04 ffff880423be5c80
2013-03-15 19:37:04 ffff8804230e3fd8
2013-03-15 19:37:04
2013-03-15 19:37:04 [1239343.201038] ffff8804230e3fd8
2013-03-15 19:37:04 ffff8804230e3fd8
2013-03-15 19:37:04 ffff8802248c8000
2013-03-15 19:37:04 ffff880423be5c80
2013-03-15 19:37:04
2013-03-15 19:37:04 [1239343.209601] ffff8804230e3a48
2013-03-15 19:37:04 ffff880329d98618
2013-03-15 19:37:04 ffff88030b96da10
2013-03-15 19:37:04 ffff880423492b28
2013-03-15 19:37:04
2013-03-15 19:37:04 [1239343.218165] Call Trace:
2013-03-15 19:37:04 [1239343.221119] [<ffffffff816a80c9>]
schedule+0x29/0x70
2013-03-15 19:37:04 [1239343.226874] [<ffffffff8128a1dd>]
do_get_write_access+0x2cd/0x4d0
2013-03-15 19:37:04 [1239343.233903] [<ffffffff81079a00>] ?
autoremove_wake_function+0x40/0x40
2013-03-15 19:37:04 [1239343.241406] [<ffffffff8128a520>]
jbd2_journal_get_write_access+0x30/0x50
2013-03-15 19:37:04 [1239343.249206] [<ffffffff8126c31d>]
__ext4_journal_get_write_access+0x3d/0x80
2013-03-15 19:37:04 [1239343.257204] [<ffffffff8123f178>]
ext4_reserve_inode_write+0x78/0xa0
2013-03-15 19:37:04 [1239343.264522] [<ffffffff812417ec>] ?
ext4_dirty_inode+0x3c/0x60
2013-03-15 19:37:04 [1239343.271261] [<ffffffff8123f1ea>]
ext4_mark_inode_dirty+0x4a/0x200
2013-03-15 19:37:04 [1239343.278373] [<ffffffff812417ec>]
ext4_dirty_inode+0x3c/0x60
2013-03-15 19:37:04 [1239343.284916] [<ffffffff811ba433>]
__mark_inode_dirty+0x43/0x240
2013-03-15 19:37:04 [1239343.291751] [<ffffffff811a9ddb>]
update_time+0xab/0xe0
2013-03-15 19:37:04 [1239343.297807] [<ffffffff811a9ffa>]
touch_atime+0xfa/0x140
2013-03-15 19:37:04 [1239343.303965] [<ffffffff8113445e>]
generic_file_aio_read+0x4ae/0x6c0
2013-03-15 19:37:04 [1239343.311191] [<ffffffff8118f487>]
do_sync_read+0xa7/0xe0
2013-03-15 19:37:04 [1239343.317341] [<ffffffff8118fba9>]
vfs_read+0xa9/0x180
2013-03-15 19:37:04 [1239343.323208] [<ffffffff8118fcd2>]
sys_read+0x52/0xa0
2013-03-15 19:37:04 [1239343.328973] [<ffffffff816b1859>]
system_call_fastpath+0x16/0x1b
2013-03-15 19:39:04 [1239463.141965] INFO: task jbd2/sda3-8:1930 blocked
for more than 120 seconds.
2013-03-15 19:39:04 [1239463.149880] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2013-03-15 19:39:04 [1239463.158882] jbd2/sda3-8 D
2013-03-15 19:39:04 ffff880227c33c80
2013-03-15 19:39:04 0 1930 2 0x00000000
2013-03-15 19:39:04 [1239463.169000] ffff8802234afad8
2013-03-15 19:39:04 0000000000000046
2013-03-15 19:39:04 ffff8802234b2e40
2013-03-15 19:39:04 ffff8802234affd8
2013-03-15 19:39:04
2013-03-15 19:39:04 [1239463.177569] ffff8802234affd8
2013-03-15 19:39:04 ffff8802234affd8
2013-03-15 19:39:04 ffff8802248c9720
2013-03-15 19:39:04 ffff8802234b2e40
2013-03-15 19:39:04
2013-03-15 19:39:04 [1239463.186137] ffff8802234afaa8
2013-03-15 19:39:04 ffff8802234b2e40
2013-03-15 19:39:04 ffff880227c344e8
2013-03-15 19:39:04 0000000000000002
2013-03-15 19:39:04
2013-03-15 19:39:04 [1239463.194705] Call Trace:
2013-03-15 19:39:04 [1239463.197661] [<ffffffff81132440>] ?
__lock_page+0x70/0x70
2013-03-15 19:39:04 [1239463.203918] [<ffffffff816a80c9>]
schedule+0x29/0x70
2013-03-15 19:39:04 [1239463.209683] [<ffffffff816a819f>]
io_schedule+0x8f/0xd0
2013-03-15 19:39:04 [1239463.215744] [<ffffffff8113244e>]
sleep_on_page+0xe/0x20
2013-03-15 19:39:04 [1239463.221910] [<ffffffff816a6870>]
__wait_on_bit+0x60/0x90
2013-03-15 19:39:04 [1239463.228160] [<ffffffff81132b81>] ?
find_get_pages_tag+0xd1/0x180
2013-03-15 19:39:04 [1239463.235189] [<ffffffff81132580>]
wait_on_page_bit+0x80/0x90
2013-03-15 19:39:04 [1239463.241731] [<ffffffff81079a00>] ?
autoremove_wake_function+0x40/0x40
2013-03-15 19:39:04 [1239463.249245] [<ffffffff81132951>]
filemap_fdatawait_range+0x101/0x190
2013-03-15 19:39:04 [1239463.256665] [<ffffffff8135ba49>] ?
submit_bio+0x79/0x160
2013-03-15 19:39:04 [1239463.262921] [<ffffffff811c6cd5>] ?
bio_alloc_bioset+0x65/0x120
2013-03-15 19:39:04 [1239463.269753] [<ffffffff81132a07>]
filemap_fdatawait+0x27/0x30
2013-03-15 19:39:04 [1239463.276404] [<ffffffff8128b77c>]
jbd2_journal_commit_transaction+0xa9c/0x19d0
2013-03-15 19:39:04 [1239463.284721] [<ffffffff81042af9>] ?
default_spin_lock_flags+0x9/0x10
2013-03-15 19:39:04 [1239463.292043] [<ffffffff8129113d>]
kjournald2+0xbd/0x240
2013-03-15 19:39:04 [1239463.298099] [<ffffffff810799c0>] ?
finish_wait+0x80/0x80
2013-03-15 19:39:04 [1239463.304352] [<ffffffff81291080>] ?
jbd2_journal_release_jbd_inode+0x130/0x130
2013-03-15 19:39:04 [1239463.312665] [<ffffffff81079070>] kthread+0xc0/0xd0
2013-03-15 19:39:04 [1239463.318332] [<ffffffff81078fb0>] ?
kthread_create_on_node+0x130/0x130
2013-03-15 19:39:04 [1239463.325848] [<ffffffff816b17ac>]
ret_from_fork+0x7c/0xb0
2013-03-15 19:39:04 [1239463.332102] [<ffffffff81078fb0>] ?
kthread_create_on_node+0x130/0x130
2013-03-15 19:39:04 [1239463.339604] INFO: task rs:main Q:Reg:3217
blocked for more than 120 seconds.
2013-03-15 19:39:04 [1239463.347719] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2013-03-15 19:39:04 [1239463.356706] rs:main Q:Reg D
2013-03-15 19:39:04 ffff88042fc33c80
2013-03-15 19:39:04 0 3217 1 0x00000000
2013-03-15 19:39:04 [1239463.364851] ffff880221e1b948
2013-03-15 19:39:04 0000000000000082
2013-03-15 19:39:04 ffff880223fe5c80
2013-03-15 19:39:04 ffff880221e1bfd8
2013-03-15 19:39:04
2013-03-15 19:39:04 [1239463.373422] ffff880221e1bfd8
2013-03-15 19:39:04 ffff880221e1bfd8
2013-03-15 19:39:04 ffff8802248cae40
2013-03-15 19:39:04 ffff880223fe5c80
2013-03-15 19:39:04
2013-03-15 19:39:04 [1239463.381992] ffff880221e1b958
2013-03-15 19:39:04 ffff8801b1080410
2013-03-15 19:39:04 ffff880340f3c000
2013-03-15 19:39:04 ffff88042370fb70
2013-03-15 19:39:04
2013-03-15 19:39:04 [1239463.390548] Call Trace:
2013-03-15 19:39:04 [1239463.393505] [<ffffffff816a80c9>]
schedule+0x29/0x70
2013-03-15 19:39:04 [1239463.399271] [<ffffffff8128a1dd>]
do_get_write_access+0x2cd/0x4d0
2013-03-15 19:39:04 [1239463.406311] [<ffffffff816a92ae>] ?
_raw_spin_lock_irqsave+0x2e/0x40
2013-03-15 19:39:04 [1239463.413640] [<ffffffff81079a00>] ?
autoremove_wake_function+0x40/0x40
2013-03-15 19:39:04 [1239463.421152] [<ffffffff8128a520>]
jbd2_journal_get_write_access+0x30/0x50
2013-03-15 19:39:04 [1239463.428959] [<ffffffff8126c31d>]
__ext4_journal_get_write_access+0x3d/0x80
2013-03-15 19:39:04 [1239463.436961] [<ffffffff8123f178>]
ext4_reserve_inode_write+0x78/0xa0
2013-03-15 19:39:04 [1239463.444281] [<ffffffff812417ec>] ?
ext4_dirty_inode+0x3c/0x60
2013-03-15 19:39:04 [1239463.451017] [<ffffffff8123f1ea>]
ext4_mark_inode_dirty+0x4a/0x200
2013-03-15 19:39:04 [1239463.458146] [<ffffffff812417ec>]
ext4_dirty_inode+0x3c/0x60
2013-03-15 19:39:04 [1239463.464692] [<ffffffff811ba433>]
__mark_inode_dirty+0x43/0x240
2013-03-15 19:39:04 [1239463.471531] [<ffffffff811a9ddb>]
update_time+0xab/0xe0
2013-03-15 19:39:04 [1239463.477585] [<ffffffff811a9eab>]
file_update_time+0x9b/0xf0
2013-03-15 19:39:04 [1239463.484130] [<ffffffff81133c70>]
__generic_file_aio_write+0x190/0x3d0
2013-03-15 19:39:04 [1239463.491640] [<ffffffff8108e0f8>] ?
__enqueue_entity+0x78/0x80
2013-03-15 19:39:04 [1239463.499614] [<ffffffff81133f2f>]
generic_file_aio_write+0x7f/0x100
2013-03-15 19:39:04 [1239463.506845] [<ffffffff81236d39>]
ext4_file_write+0xa9/0x490
2013-03-15 19:39:04 [1239463.513391] [<ffffffff8108aeb0>] ?
wake_up_state+0x10/0x20
2013-03-15 19:39:04 [1239463.519848] [<ffffffff810bcff6>] ?
wake_futex+0x66/0x80
2013-03-15 19:39:04 [1239463.526006] [<ffffffff810bd11b>] ?
futex_wake+0x10b/0x130
2013-03-15 19:39:04 [1239463.532359] [<ffffffff8118f3a7>]
do_sync_write+0xa7/0xe0
2013-03-15 19:39:04 [1239463.538611] [<ffffffff8118fa2c>]
vfs_write+0xac/0x180
2013-03-15 19:39:04 [1239463.544575] [<ffffffff8118fd72>]
sys_write+0x52/0xa0
2013-03-15 19:39:04 [1239463.550428] [<ffffffff816b1859>]
system_call_fastpath+0x16/0x1b
2013-03-15 19:39:04 [1239463.557361] INFO: task flush-8:0:3588 blocked
for more than 120 seconds.
2013-03-15 19:39:04 [1239463.565068] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2013-03-15 19:39:04 [1239463.574049] flush-8:0 D
2013-03-15 19:39:04 ffff88042fc53c80
2013-03-15 19:39:04 0 3588 2 0x00000000
2013-03-15 19:39:04 [1239463.582190] ffff880221fcf698
2013-03-15 19:39:04 0000000000000046
2013-03-15 19:39:04 ffff8802233f5c80
2013-03-15 19:39:04 ffff880221fcffd8
2013-03-15 19:39:04
2013-03-15 19:39:04 [1239463.590757] ffff880221fcffd8
2013-03-15 19:39:04 ffff880221fcffd8
2013-03-15 19:39:04 ffff8802248cdc80
2013-03-15 19:39:04 ffff8802233f5c80
2013-03-15 19:39:04
2013-03-15 19:39:04 [1239463.599317] ffff880221fcf6a8
2013-03-15 19:39:04 ffff8801ed90daf8
2013-03-15 19:39:04 ffff8803db412070
2013-03-15 19:39:04 ffff880423779ca8
2013-03-15 19:39:04
2013-03-15 19:39:04 [1239463.607890] Call Trace:
2013-03-15 19:39:04 [1239463.610842] [<ffffffff816a80c9>]
schedule+0x29/0x70
2013-03-15 19:39:04 [1239463.616613] [<ffffffff8128a1dd>]
do_get_write_access+0x2cd/0x4d0
2013-03-15 19:39:04 [1239463.623644] [<ffffffff8137e53e>] ?
radix_tree_lookup_slot+0xe/0x10
2013-03-15 19:39:04 [1239463.630866] [<ffffffff81079a00>] ?
autoremove_wake_function+0x40/0x40
2013-03-15 19:39:04 [1239463.638385] [<ffffffff8128a520>]
jbd2_journal_get_write_access+0x30/0x50
2013-03-15 19:39:04 [1239463.646192] [<ffffffff8126c31d>]
__ext4_journal_get_write_access+0x3d/0x80
2013-03-15 19:39:04 [1239463.654193] [<ffffffff812354b5>] ?
ext4_read_block_bitmap+0x35/0x60
2013-03-15 19:39:04 [1239463.661519] [<ffffffff8127279b>]
ext4_mb_mark_diskspace_used+0x7b/0x4d0
2013-03-15 19:39:04 [1239463.669225] [<ffffffff8117e81f>] ?
__kmalloc+0xaf/0x180
2013-03-15 19:39:04 [1239463.675376] [<ffffffff8126d7c6>] ?
ext4_mb_initialize_context+0x86/0x1b0
2013-03-15 19:39:04 [1239463.683184] [<ffffffff81273f5b>]
ext4_mb_new_blocks+0x29b/0x550
2013-03-15 19:39:04 [1239463.690113] [<ffffffff8126a97f>]
ext4_ext_map_blocks+0xc4f/0x1610
2013-03-15 19:39:04 [1239463.697242] [<ffffffff8123c035>]
ext4_map_blocks+0x1d5/0x270
2013-03-15 19:39:04 [1239463.703887] [<ffffffff8124045d>]
mpage_da_map_and_submit+0x15d/0x4b0
2013-03-15 19:39:04 [1239463.711310] [<ffffffff81241017>] ?
ext4_da_writepages+0x317/0x5e0
2013-03-15 19:39:05 [1239463.718435] [<ffffffff81241063>]
ext4_da_writepages+0x363/0x5e0
2013-03-15 19:39:05 [1239463.725371] [<ffffffff8113e3be>]
do_writepages+0x1e/0x40
2013-03-15 19:39:05 [1239463.731628] [<ffffffff811b977b>]
__writeback_single_inode+0x3b/0x160
2013-03-15 19:39:05 [1239463.739043] [<ffffffff811ba7d0>]
writeback_sb_inodes+0x1a0/0x390
2013-03-15 19:39:05 [1239463.746075] [<ffffffff811baa5f>]
__writeback_inodes_wb+0x9f/0xd0
2013-03-15 19:39:05 [1239463.753109] [<ffffffff811bacdb>]
wb_writeback+0x24b/0x2e0
2013-03-15 19:39:05 [1239463.759455] [<ffffffff811a9b22>] ?
get_nr_inodes+0x52/0x70
2013-03-15 19:39:05 [1239463.765900] [<ffffffff811bbf76>]
wb_do_writeback+0x1f6/0x200
2013-03-15 19:39:05 [1239463.772539] [<ffffffff811bc00b>]
bdi_writeback_thread+0x8b/0x230
2013-03-15 19:39:05 [1239463.779562] [<ffffffff811bbf80>] ?
wb_do_writeback+0x200/0x200
2013-03-15 19:39:05 [1239463.786394] [<ffffffff81079070>] kthread+0xc0/0xd0
2013-03-15 19:39:05 [1239463.792061] [<ffffffff81078fb0>] ?
kthread_create_on_node+0x130/0x130
2013-03-15 19:39:05 [1239463.799569] [<ffffffff816b17ac>]
ret_from_fork+0x7c/0xb0
2013-03-15 19:39:05 [1239463.805818] [<ffffffff81078fb0>] ?
kthread_create_on_node+0x130/0x130
2013-03-15 19:39:05 [1239463.813330] INFO: task tlsmgr:6983 blocked for
more than 120 seconds.
2013-03-15 19:39:05 [1239463.820738] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2013-03-15 19:39:05 [1239463.829726] tlsmgr D
2013-03-15 19:39:05 ffff88042fc13c80
2013-03-15 19:39:05 0 6983 6611 0x00000000
2013-03-15 19:39:05 [1239463.837917] ffff8804230e3a38
2013-03-15 19:39:05 0000000000000082
2013-03-15 19:39:05 ffff880423be5c80
2013-03-15 19:39:05 ffff8804230e3fd8
2013-03-15 19:39:05
2013-03-15 19:39:05 [1239463.846468] ffff8804230e3fd8
2013-03-15 19:39:05 ffff8804230e3fd8
2013-03-15 19:39:05 ffff8802248c8000
2013-03-15 19:39:05 ffff880423be5c80
2013-03-15 19:39:05
2013-03-15 19:39:05 [1239463.855021] ffff8804230e3a48
2013-03-15 19:39:05 ffff880329d98618
2013-03-15 19:39:05 ffff88030b96da10
2013-03-15 19:39:05 ffff880423492b28
2013-03-15 19:39:05
2013-03-15 19:39:05 [1239463.863572] Call Trace:
2013-03-15 19:39:05 [1239463.866520] [<ffffffff816a80c9>]
schedule+0x29/0x70
2013-03-15 19:39:05 [1239463.872286] [<ffffffff8128a1dd>]
do_get_write_access+0x2cd/0x4d0
2013-03-15 19:39:05 [1239463.879309] [<ffffffff81079a00>] ?
autoremove_wake_function+0x40/0x40
2013-03-15 19:39:05 [1239463.886819] [<ffffffff8128a520>]
jbd2_journal_get_write_access+0x30/0x50
2013-03-15 19:39:05 [1239463.894621] [<ffffffff8126c31d>]
__ext4_journal_get_write_access+0x3d/0x80
2013-03-15 19:39:05 [1239463.902616] [<ffffffff8123f178>]
ext4_reserve_inode_write+0x78/0xa0
2013-03-15 19:39:05 [1239463.909929] [<ffffffff812417ec>] ?
ext4_dirty_inode+0x3c/0x60
2013-03-15 19:39:05 [1239463.916662] [<ffffffff8123f1ea>]
ext4_mark_inode_dirty+0x4a/0x200
2013-03-15 19:39:05 [1239463.923783] [<ffffffff812417ec>]
ext4_dirty_inode+0x3c/0x60
2013-03-15 19:39:05 [1239463.930322] [<ffffffff811ba433>]
__mark_inode_dirty+0x43/0x240
2013-03-15 19:39:05 [1239463.937152] [<ffffffff811a9ddb>]
update_time+0xab/0xe0
2013-03-15 19:39:05 [1239463.943208] [<ffffffff811a9ffa>]
touch_atime+0xfa/0x140
2013-03-15 19:39:05 [1239463.949356] [<ffffffff8113445e>]
generic_file_aio_read+0x4ae/0x6c0
2013-03-15 19:39:05 [1239463.956576] [<ffffffff8118f487>]
do_sync_read+0xa7/0xe0
2013-03-15 19:39:05 [1239463.962729] [<ffffffff8118fba9>]
vfs_read+0xa9/0x180
2013-03-15 19:39:05 [1239463.968588] [<ffffffff8118fcd2>]
sys_read+0x52/0xa0
2013-03-15 19:39:05 [1239463.974343] [<ffffffff816b1859>]
system_call_fastpath+0x16/0x1b
2013-03-15 19:40:01 [1239520.482156] INFO: rcu_sched detected stalls on
CPUs/tasks:
2013-03-15 19:40:01
2013-03-15 19:40:01 [1239520.488530] 0: (2 GPs behind)
idle=d99/140000000000001/0 drain=0 . timer not pending
2013-03-15 19:40:01 [1239520.497520]
2013-03-15 19:40:01 (detected by 2, t=42013 jiffies)
2013-03-15 19:40:01 [1239520.502629] Pid: 0, comm: swapper/2 Not tainted
3.7.9-vs2.3.5.6-rol-em64t-tg3+sdfix #3
2013-03-15 19:40:01 [1239520.511717] Call Trace:
2013-03-15 19:40:01 [1239520.514670] <IRQ>
2013-03-15 19:40:01 [<ffffffff810f8e15>] rcu_pending+0x535/0x550
2013-03-15 19:40:01 [1239520.521636] [<ffffffff810facc2>]
rcu_check_callbacks+0xe2/0x180
2013-03-15 19:40:01 [1239520.528570] [<ffffffff810663b8>]
update_process_times+0x48/0x90
2013-03-15 19:40:01 [1239520.535502] [<ffffffff810bb4e3>]
tick_sched_timer+0x83/0xd0
2013-03-15 19:40:01 [1239520.542039] [<ffffffff8107d479>]
__run_hrtimer+0x79/0x1d0
2013-03-15 19:40:01 [1239520.548379] [<ffffffff810bb460>] ?
tick_nohz_handler+0xf0/0xf0
2013-03-15 19:40:01 [1239520.555213] [<ffffffff8107dd97>]
hrtimer_interrupt+0xf7/0x230
2013-03-15 19:40:01 [1239520.561951] [<ffffffff816b3279>]
smp_apic_timer_interrupt+0x69/0x99
2013-03-15 19:40:01 [1239520.569265] [<ffffffff816b230a>]
apic_timer_interrupt+0x6a/0x70
2013-03-15 19:40:01 [1239520.576195] <EOI>
2013-03-15 19:40:01 [<ffffffff813d521d>] ? intel_idle+0xed/0x150
2013-03-15 19:40:01 [1239520.583159] [<ffffffff813d51fe>] ?
intel_idle+0xce/0x150
2013-03-15 19:40:01 [1239520.589413] [<ffffffff8156b559>]
cpuidle_enter+0x19/0x20
2013-03-15 19:40:01 [1239520.595665] [<ffffffff8156bc22>]
cpuidle_idle_call+0xa2/0x1b0
2013-03-15 19:40:01 [1239520.602397] [<ffffffff8101d02f>]
cpu_idle+0xaf/0x120
2013-03-15 19:40:01 [1239520.608262] [<ffffffff81695c9d>]
start_secondary+0x1d4/0x1db
2013-03-15 19:41:05 [1239583.793109] INFO: task jbd2/sda2-8:416 blocked
for more than 120 seconds.
2013-03-15 19:41:05 [1239583.800926] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2013-03-15 19:41:05 [1239583.809923] jbd2/sda2-8 D
2013-03-15 19:41:05 ffff88042fc53c80
2013-03-15 19:41:05 0 416 2 0x00000000
2013-03-15 19:41:05 [1239583.818072] ffff880221ab1ad8
2013-03-15 19:41:05 0000000000000046
2013-03-15 19:41:05 ffff880221de9720
2013-03-15 19:41:05 ffff880221ab1fd8
2013-03-15 19:41:05
2013-03-15 19:41:05 [1239583.826641] ffff880221ab1fd8
2013-03-15 19:41:05 ffff880221ab1fd8
2013-03-15 19:41:05 ffff8802248cdc80
2013-03-15 19:41:05 ffff880221de9720
2013-03-15 19:41:05
2013-03-15 19:41:05 [1239583.835203] ffff880221ab1aa8
2013-03-15 19:41:05 ffff880221de9720
2013-03-15 19:41:05 ffff88042fc544e8
2013-03-15 19:41:05 0000000000000002
2013-03-15 19:41:05
2013-03-15 19:41:05 [1239583.843765] Call Trace:
2013-03-15 19:41:05 [1239583.846718] [<ffffffff81132440>] ?
__lock_page+0x70/0x70
2013-03-15 19:41:05 [1239583.852983] [<ffffffff816a80c9>]
schedule+0x29/0x70
2013-03-15 19:41:05 [1239583.858748] [<ffffffff816a819f>]
io_schedule+0x8f/0xd0
2013-03-15 19:41:05 [1239583.864809] [<ffffffff8113244e>]
sleep_on_page+0xe/0x20
2013-03-15 19:41:05 [1239583.870963] [<ffffffff816a6870>]
__wait_on_bit+0x60/0x90
2013-03-15 19:41:05 [1239583.877218] [<ffffffff81132b81>] ?
find_get_pages_tag+0xd1/0x180
2013-03-15 19:41:05 [1239583.884249] [<ffffffff81132580>]
wait_on_page_bit+0x80/0x90
2013-03-15 19:41:05 [1239583.890791] [<ffffffff81079a00>] ?
autoremove_wake_function+0x40/0x40
2013-03-15 19:41:05 [1239583.898306] [<ffffffff81132951>]
filemap_fdatawait_range+0x101/0x190
2013-03-15 19:41:05 [1239583.907691] [<ffffffff8135ba49>] ?
submit_bio+0x79/0x160
2013-03-15 19:41:05 [1239583.913946] [<ffffffff811c6cd5>] ?
bio_alloc_bioset+0x65/0x120
2013-03-15 19:41:05 [1239583.920776] [<ffffffff81132a07>]
filemap_fdatawait+0x27/0x30
2013-03-15 19:41:05 [1239583.927419] [<ffffffff8128b77c>]
jbd2_journal_commit_transaction+0xa9c/0x19d0
2013-03-15 19:41:05 [1239583.935733] [<ffffffff81042af9>] ?
default_spin_lock_flags+0x9/0x10
2013-03-15 19:41:05 [1239583.943055] [<ffffffff8129113d>]
kjournald2+0xbd/0x240
2013-03-15 19:41:05 [1239583.949110] [<ffffffff810799c0>] ?
finish_wait+0x80/0x80
2013-03-15 19:41:05 [1239583.955364] [<ffffffff81291080>] ?
jbd2_journal_release_jbd_inode+0x130/0x130
2013-03-15 19:41:05 [1239583.963676] [<ffffffff81079070>] kthread+0xc0/0xd0
2013-03-15 19:41:05 [1239583.969345] [<ffffffff81078fb0>] ?
kthread_create_on_node+0x130/0x130
2013-03-15 19:41:05 [1239583.976861] [<ffffffff816b17ac>]
ret_from_fork+0x7c/0xb0
2013-03-15 19:41:05 [1239583.983115] [<ffffffff81078fb0>] ?
kthread_create_on_node+0x130/0x130
2013-03-15 19:41:05 [1239583.990627] INFO: task jbd2/sda3-8:1930 blocked
for more than 120 seconds.
2013-03-15 19:41:05 [1239583.998526] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2013-03-15 19:41:05 [1239584.007514] jbd2/sda3-8 D
2013-03-15 19:41:05 ffff880227c33c80
2013-03-15 19:41:05 0 1930 2 0x00000000
2013-03-15 19:41:05 [1239584.015662] ffff8802234afad8
2013-03-15 19:41:05 0000000000000046
2013-03-15 19:41:05 ffff8802234b2e40
2013-03-15 19:41:05 ffff8802234affd8
2013-03-15 19:41:05
2013-03-15 19:41:05 [1239584.024223] ffff8802234affd8
2013-03-15 19:41:05 ffff8802234affd8
2013-03-15 19:41:05 ffff8802248c9720
2013-03-15 19:41:05 ffff8802234b2e40
2013-03-15 19:41:05
2013-03-15 19:41:05 [1239584.032796] ffff8802234afaa8
2013-03-15 19:41:05 ffff8802234b2e40
2013-03-15 19:41:05 ffff880227c344e8
2013-03-15 19:41:05 0000000000000002
2013-03-15 19:41:05
2013-03-15 19:41:05 [1239584.041354] Call Trace:
2013-03-15 19:41:05 [1239584.044311] [<ffffffff81132440>] ?
__lock_page+0x70/0x70
2013-03-15 19:41:05 [1239584.050561] [<ffffffff816a80c9>]
schedule+0x29/0x70
2013-03-15 19:41:05 [1239584.056332] [<ffffffff816a819f>]
io_schedule+0x8f/0xd0
2013-03-15 19:41:05 [1239584.062386] [<ffffffff8113244e>]
sleep_on_page+0xe/0x20
2013-03-15 19:41:05 [1239584.068541] [<ffffffff816a6870>]
__wait_on_bit+0x60/0x90
2013-03-15 19:41:05 [1239584.074795] [<ffffffff81132b81>] ?
find_get_pages_tag+0xd1/0x180
2013-03-15 19:41:05 [1239584.081820] [<ffffffff81132580>]
wait_on_page_bit+0x80/0x90
2013-03-15 19:41:05 [1239584.088366] [<ffffffff81079a00>] ?
autoremove_wake_function+0x40/0x40
2013-03-15 19:41:05 [1239584.095880] [<ffffffff81132951>]
filemap_fdatawait_range+0x101/0x190
2013-03-15 19:41:05 [1239584.103299] [<ffffffff8135ba49>] ?
submit_bio+0x79/0x160
2013-03-15 19:41:05 [1239584.109549] [<ffffffff811c6cd5>] ?
bio_alloc_bioset+0x65/0x120
2013-03-15 19:41:05 [1239584.116385] [<ffffffff81132a07>]
filemap_fdatawait+0x27/0x30
2013-03-15 19:41:05 [1239584.123028] [<ffffffff8128b77c>]
jbd2_journal_commit_transaction+0xa9c/0x19d0
2013-03-15 19:41:05 [1239584.131336] [<ffffffff81042af9>] ?
default_spin_lock_flags+0x9/0x10
2013-03-15 19:41:05 [1239584.138656] [<ffffffff8129113d>]
kjournald2+0xbd/0x240
2013-03-15 19:41:05 [1239584.144717] [<ffffffff810799c0>] ?
finish_wait+0x80/0x80
2013-03-15 19:41:05 [1239584.150958] [<ffffffff81291080>] ?
jbd2_journal_release_jbd_inode+0x130/0x130
2013-03-15 19:41:05 [1239584.159269] [<ffffffff81079070>] kthread+0xc0/0xd0
2013-03-15 19:41:05 [1239584.164943] [<ffffffff81078fb0>] ?
kthread_create_on_node+0x130/0x130
2013-03-15 19:41:05 [1239584.172467] [<ffffffff816b17ac>]
ret_from_fork+0x7c/0xb0
2013-03-15 19:41:05 [1239584.178716] [<ffffffff81078fb0>] ?
kthread_create_on_node+0x130/0x130
2013-03-15 19:43:01 [1239700.250748] INFO: rcu_sched detected stalls on
CPUs/tasks:
2013-03-15 19:43:01
2013-03-15 19:43:01 [1239700.257116] 0: (2 GPs behind)
idle=d99/140000000000001/0 drain=0 . timer not pending
2013-03-15 19:43:01 [1239700.266100]
2013-03-15 19:43:01 (detected by 2, t=60018 jiffies)
2013-03-15 19:43:01 [1239700.271202] Pid: 0, comm: swapper/2 Not tainted
3.7.9-vs2.3.5.6-rol-em64t-tg3+sdfix #3
2013-03-15 19:43:01 [1239700.280288] Call Trace:
2013-03-15 19:43:01 [1239700.283242] <IRQ>
2013-03-15 19:43:01 [<ffffffff810f8e15>] rcu_pending+0x535/0x550
2013-03-15 19:43:01 [1239700.290208] [<ffffffff810facc2>]
rcu_check_callbacks+0xe2/0x180
2013-03-15 19:43:01 [1239700.297141] [<ffffffff810663b8>]
update_process_times+0x48/0x90
2013-03-15 19:43:01 [1239700.304073] [<ffffffff810bb4e3>]
tick_sched_timer+0x83/0xd0
2013-03-15 19:43:01 [1239700.310610] [<ffffffff8107d479>]
__run_hrtimer+0x79/0x1d0
2013-03-15 19:43:01 [1239700.316956] [<ffffffff810bb460>] ?
tick_nohz_handler+0xf0/0xf0
2013-03-15 19:43:01 [1239700.323791] [<ffffffff8107dd97>]
hrtimer_interrupt+0xf7/0x230
2013-03-15 19:43:01 [1239700.330529] [<ffffffff816b3279>]
smp_apic_timer_interrupt+0x69/0x99
2013-03-15 19:43:01 [1239700.337851] [<ffffffff816b230a>]
apic_timer_interrupt+0x6a/0x70
2013-03-15 19:43:01 [1239700.344781] <EOI>
2013-03-15 19:43:02 [<ffffffff813d521d>] ? intel_idle+0xed/0x150
2013-03-15 19:43:02 [1239700.351743] [<ffffffff813d51fe>] ?
intel_idle+0xce/0x150
2013-03-15 19:43:02 [1239700.357997] [<ffffffff8156b559>]
cpuidle_enter+0x19/0x20
2013-03-15 19:43:02 [1239700.364249] [<ffffffff8156bc22>]
cpuidle_idle_call+0xa2/0x1b0
2013-03-15 19:43:02 [1239700.370980] [<ffffffff8101d02f>]
cpu_idle+0xaf/0x120
2013-03-15 19:43:02 [1239700.376846] [<ffffffff81695c9d>]
start_secondary+0x1d4/0x1db
2013-03-15 19:46:01 [1239880.019328] INFO: rcu_sched detected stalls on
CPUs/tasks:
2013-03-15 19:46:01
2013-03-15 19:46:01 [1239880.025693] 0: (2 GPs behind)
idle=d99/140000000000001/0 drain=0 . timer not pending
2013-03-15 19:46:01 [1239880.034683]
2013-03-15 19:46:01 (detected by 2, t=78023 jiffies)
2013-03-15 19:46:01 [1239880.039784] Pid: 0, comm: swapper/2 Not tainted
3.7.9-vs2.3.5.6-rol-em64t-tg3+sdfix #3
2013-03-15 19:46:01 [1239880.048870] Call Trace:
2013-03-15 19:46:01 [1239880.051824] <IRQ>
2013-03-15 19:46:01 [<ffffffff810f8e15>] rcu_pending+0x535/0x550
2013-03-15 19:46:01 [1239880.058788] [<ffffffff810facc2>]
rcu_check_callbacks+0xe2/0x180
2013-03-15 19:46:02 [1239880.065722] [<ffffffff810663b8>]
update_process_times+0x48/0x90
2013-03-15 19:46:02 [1239880.072655] [<ffffffff810bb4e3>]
tick_sched_timer+0x83/0xd0
2013-03-15 19:46:02 [1239880.079200] [<ffffffff8107d479>]
__run_hrtimer+0x79/0x1d0
2013-03-15 19:46:02 [1239880.085540] [<ffffffff810bb460>] ?
tick_nohz_handler+0xf0/0xf0
2013-03-15 19:46:02 [1239880.092376] [<ffffffff8107dd97>]
hrtimer_interrupt+0xf7/0x230
2013-03-15 19:46:02 [1239880.099115] [<ffffffff816b3279>]
smp_apic_timer_interrupt+0x69/0x99
2013-03-15 19:46:02 [1239880.106435] [<ffffffff816b230a>]
apic_timer_interrupt+0x6a/0x70
2013-03-15 19:46:02 [1239880.113365] <EOI>
2013-03-15 19:46:02 [<ffffffff813d521d>] ? intel_idle+0xed/0x150
2013-03-15 19:46:02 [1239880.120328] [<ffffffff813d51fe>] ?
intel_idle+0xce/0x150
2013-03-15 19:46:02 [1239880.126581] [<ffffffff8156b559>]
cpuidle_enter+0x19/0x20
2013-03-15 19:46:02 [1239880.132839] [<ffffffff8156bc22>]
cpuidle_idle_call+0xa2/0x1b0
2013-03-15 19:46:02 [1239880.139569] [<ffffffff8101d02f>]
cpu_idle+0xaf/0x120
2013-03-15 19:46:02 [1239880.145428] [<ffffffff81695c9d>]
start_secondary+0x1d4/0x1db
2013-03-15 19:49:02 [1240059.787917] INFO: rcu_sched detected stalls on
CPUs/tasks:
2013-03-15 19:49:02
2013-03-15 19:49:02 [1240059.794279] 0: (2 GPs behind)
idle=d99/140000000000001/0 drain=0 . timer not pending
2013-03-15 19:49:02 [1240059.803260]
2013-03-15 19:49:02 (detected by 2, t=96028 jiffies)
2013-03-15 19:49:02 [1240059.808354] Pid: 0, comm: swapper/2 Not tainted
3.7.9-vs2.3.5.6-rol-em64t-tg3+sdfix #3
2013-03-15 19:49:02 [1240059.817435] Call Trace:
2013-03-15 19:49:02 [1240059.820380] <IRQ>
2013-03-15 19:49:02 [<ffffffff810f8e15>] rcu_pending+0x535/0x550
2013-03-15 19:49:02 [1240059.827330] [<ffffffff810facc2>]
rcu_check_callbacks+0xe2/0x180
2013-03-15 19:49:02 [1240059.834255] [<ffffffff810663b8>]
update_process_times+0x48/0x90
2013-03-15 19:49:02 [1240059.841179] [<ffffffff810bb4e3>]
tick_sched_timer+0x83/0xd0
2013-03-15 19:49:02 [1240059.847718] [<ffffffff8107d479>]
__run_hrtimer+0x79/0x1d0
2013-03-15 19:49:02 [1240059.854060] [<ffffffff810bb460>] ?
tick_nohz_handler+0xf0/0xf0
2013-03-15 19:49:02 [1240059.860887] [<ffffffff8107dd97>]
hrtimer_interrupt+0xf7/0x230
2013-03-15 19:49:02 [1240059.867618] [<ffffffff816b3279>]
smp_apic_timer_interrupt+0x69/0x99
2013-03-15 19:49:02 [1240059.874997] [<ffffffff816b230a>]
apic_timer_interrupt+0x6a/0x70
2013-03-15 19:49:02 [1240059.881920] <EOI>
2013-03-15 19:49:02 [<ffffffff813d521d>] ? intel_idle+0xed/0x150
2013-03-15 19:49:02 [1240059.888867] [<ffffffff813d51fe>] ?
intel_idle+0xce/0x150
2013-03-15 19:49:02 [1240059.895112] [<ffffffff8156b559>]
cpuidle_enter+0x19/0x20
2013-03-15 19:49:02 [1240059.901358] [<ffffffff8156bc22>]
cpuidle_idle_call+0xa2/0x1b0
2013-03-15 19:49:02 [1240059.908088] [<ffffffff8101d02f>]
cpu_idle+0xaf/0x120
2013-03-15 19:49:02 [1240059.913948] [<ffffffff81695c9d>]
start_secondary+0x1d4/0x1db
2013-03-15 19:52:02 [1240239.556500] INFO: rcu_sched detected stalls on
CPUs/tasks:
2013-03-15 19:52:02
2013-03-15 19:52:02 [1240239.562862] 0: (2 GPs behind)
idle=d99/140000000000001/0 drain=0 . timer not pending
2013-03-15 19:52:02 [1240239.571844]
2013-03-15 19:52:02 (detected by 2, t=114032 jiffies)
2013-03-15 19:52:02 [1240239.577042] Pid: 0, comm: swapper/2 Not tainted
3.7.9-vs2.3.5.6-rol-em64t-tg3+sdfix #3
2013-03-15 19:52:02 [1240239.586127] Call Trace:
2013-03-15 19:52:02 [1240239.589074] <IRQ>
2013-03-15 19:52:02 [<ffffffff810f8e15>] rcu_pending+0x535/0x550
2013-03-15 19:52:02 [1240239.596025] [<ffffffff810facc2>]
rcu_check_callbacks+0xe2/0x180
2013-03-15 19:52:02 [1240239.603044] [<ffffffff810663b8>]
update_process_times+0x48/0x90
2013-03-15 19:52:02 [1240239.609970] [<ffffffff810bb4e3>]
tick_sched_timer+0x83/0xd0
2013-03-15 19:52:02 [1240239.616511] [<ffffffff8107d479>]
__run_hrtimer+0x79/0x1d0
2013-03-15 19:52:02 [1240239.622854] [<ffffffff810bb460>] ?
tick_nohz_handler+0xf0/0xf0
2013-03-15 19:52:02 [1240239.629674] [<ffffffff8107dd97>]
hrtimer_interrupt+0xf7/0x230
2013-03-15 19:52:02 [1240239.636413] [<ffffffff816b3279>]
smp_apic_timer_interrupt+0x69/0x99
2013-03-15 19:52:02 [1240239.643719] [<ffffffff816b230a>]
apic_timer_interrupt+0x6a/0x70
2013-03-15 19:52:02 [1240239.650641] <EOI>
2013-03-15 19:52:02 [<ffffffff813d521d>] ? intel_idle+0xed/0x150
2013-03-15 19:52:02 [1240239.657589] [<ffffffff813d51fe>] ?
intel_idle+0xce/0x150
2013-03-15 19:52:02 [1240239.663834] [<ffffffff8156b559>]
cpuidle_enter+0x19/0x20
2013-03-15 19:52:02 [1240239.670080] [<ffffffff8156bc22>]
cpuidle_idle_call+0xa2/0x1b0
2013-03-15 19:52:02 [1240239.676801] [<ffffffff8101d02f>]
cpu_idle+0xaf/0x120
2013-03-15 19:52:02 [1240239.682660] [<ffffffff81695c9d>]
start_secondary+0x1d4/0x1db
2013-03-15 19:55:02 [1240419.325086] INFO: rcu_sched detected stalls on
CPUs/tasks:
2013-03-15 19:55:02
2013-03-15 19:55:02 [1240419.331445] 0: (2 GPs behind)
idle=d99/140000000000001/0 drain=0 . timer not pending
2013-03-15 19:55:02 [1240419.340428]
2013-03-15 19:55:02 (detected by 2, t=132038 jiffies)
2013-03-15 19:55:02 [1240419.345620] Pid: 0, comm: swapper/2 Not tainted
3.7.9-vs2.3.5.6-rol-em64t-tg3+sdfix #3
2013-03-15 19:55:02 [1240419.354700] Call Trace:
2013-03-15 19:55:02 [1240419.357647] <IRQ>
2013-03-15 19:55:02 [<ffffffff810f8e15>] rcu_pending+0x535/0x550
2013-03-15 19:55:02 [1240419.364597] [<ffffffff810facc2>]
rcu_check_callbacks+0xe2/0x180
2013-03-15 19:55:02 [1240419.371521] [<ffffffff810663b8>]
update_process_times+0x48/0x90
2013-03-15 19:55:02 [1240419.378446] [<ffffffff810bb4e3>]
tick_sched_timer+0x83/0xd0
2013-03-15 19:55:02 [1240419.384982] [<ffffffff8107d479>]
__run_hrtimer+0x79/0x1d0
2013-03-15 19:55:02 [1240419.391324] [<ffffffff810bb460>] ?
tick_nohz_handler+0xf0/0xf0
2013-03-15 19:55:02 [1240419.398156] [<ffffffff8107dd97>]
hrtimer_interrupt+0xf7/0x230
2013-03-15 19:55:02 [1240419.404888] [<ffffffff816b3279>]
smp_apic_timer_interrupt+0x69/0x99
2013-03-15 19:55:02 [1240419.412201] [<ffffffff816b230a>]
apic_timer_interrupt+0x6a/0x70
2013-03-15 19:55:02 [1240419.419123] <EOI>
2013-03-15 19:55:02 [<ffffffff813d521d>] ? intel_idle+0xed/0x150
2013-03-15 19:55:02 [1240419.426072] [<ffffffff813d51fe>] ?
intel_idle+0xce/0x150
2013-03-15 19:55:02 [1240419.432317] [<ffffffff8156b559>]
cpuidle_enter+0x19/0x20
2013-03-15 19:55:02 [1240419.438562] [<ffffffff8156bc22>]
cpuidle_idle_call+0xa2/0x1b0
2013-03-15 19:55:02 [1240419.445294] [<ffffffff8101d02f>]
cpu_idle+0xaf/0x120
2013-03-15 19:55:02 [1240419.451151] [<ffffffff81695c9d>]
start_secondary+0x1d4/0x1db
Have you any idea what this could be?
Please CC me, as I'm not a member of the LKML.
Many thanks and regards
Urban Loesch
reply other threads:[~2013-03-15 21:57 UTC|newest]
Thread overview: [no followups] expand[flat|nested] mbox.gz Atom feed
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=51439932.6060209@enas.net \
--to=bind@enas.net \
--cc=linux-kernel@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.