* sync-Regression in 2.6.28.2?
@ 2009-01-27 9:35 Ralf Hildebrandt
2009-01-27 21:09 ` Federico Cuello
0 siblings, 1 reply; 14+ messages in thread
From: Ralf Hildebrandt @ 2009-01-27 9:35 UTC (permalink / raw)
To: linux-kernel; +Cc: Udo Wolter, cf
I recently installed 2.6.28.2 on our postfix/dovecot-based
mailboxserver. Previously, 2.6.28 and 2.6.28.1 have been running there
without a hitch.
Now with 2.6.28.2 I had two major lockups: All writes to the users'
Maildirs (on ext4) would stall, the load would rise, "sync" would never
return.
I had to "reboot -f -n" to get the machine back. All hanging processes
were unkillable, even with kill -9.
There was NOTHING in dmesg, nothing in the log either. No indication
of filesystem errors or anything.
Lookin at the
http://www.eu.kernel.org/pub/linux/kernel/v2.6/ChangeLog-2.6.28.2
I see changes to
fs: sys_sync fix
fs: sync_sb_inodes fix
fs: remove WB_SYNC_HOLD
Since the mailserver uses sync() extensively (both dovecot and
postfix), maybe that's the culprit?
Note: I'm really just guessing here!
Back to 2.6.28.1 for now.
--
Ralf Hildebrandt Ralf.Hildebrandt@charite.de
Charite - Universitätsmedizin Berlin Tel. +49 (0)30-450 570-155
Geschäftsbereich IT | Abt. Netzwerk Fax. +49 (0)30-450 570-962
Hindenburgdamm 30 | 12200 Berlin
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: sync-Regression in 2.6.28.2?
2009-01-27 9:35 sync-Regression in 2.6.28.2? Ralf Hildebrandt
@ 2009-01-27 21:09 ` Federico Cuello
2009-02-03 1:09 ` Nick Piggin
0 siblings, 1 reply; 14+ messages in thread
From: Federico Cuello @ 2009-01-27 21:09 UTC (permalink / raw)
To: linux-kernel, Udo Wolter, cf
Ralf Hildebrandt escribió:
> I recently installed 2.6.28.2 on our postfix/dovecot-based
> mailboxserver. Previously, 2.6.28 and 2.6.28.1 have been running there
> without a hitch.
>
> Now with 2.6.28.2 I had two major lockups: All writes to the users'
> Maildirs (on ext4) would stall, the load would rise, "sync" would never
> return.
>
> I had to "reboot -f -n" to get the machine back. All hanging processes
> were unkillable, even with kill -9.
> [...]
The same is happening to me, but I have some logs taken with sysrq.
Here is my vmstat output:
procs -----------memory---------- ---swap-- -----io---- -system--
----cpu----
r b swpd free buff cache si so bi bo in cs us sy
id wa
0 2 99028 46536 26356 1569400 0 0 0 0 1400 437 0
3 0 96
0 2 99028 46536 26356 1569400 0 0 0 0 1344 355 0
6 0 94
0 2 99028 46536 26356 1569400 0 0 0 0 1373 387 0
0 0 100
0 2 99028 46536 26364 1569400 0 0 0 12 1403 384 1
0 0 99
0 2 99028 46536 26364 1569400 0 0 0 0 1370 378 0
0 0 100
0 2 99028 46536 26364 1569400 0 0 0 0 1351 346 0
0 0 100
0 2 99028 46536 26364 1569400 0 0 0 0 1395 412 0
0 0 100
0 2 99028 46536 26364 1569400 0 0 0 0 1349 332 0
0 0 100
0 2 99028 46536 26368 1569400 0 0 4 0 1407 387 0
0 0 100
Notice the 100% iowait.
I also managed to reproduce it doing a rsync from one partition to a USB
drive. After the lockup I can't read any file from the source partition,
but the other partitions can be accessed normally.
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: sync-Regression in 2.6.28.2?
2009-01-27 21:09 ` Federico Cuello
@ 2009-02-03 1:09 ` Nick Piggin
2009-02-03 19:51 ` Federico Cuello
0 siblings, 1 reply; 14+ messages in thread
From: Nick Piggin @ 2009-02-03 1:09 UTC (permalink / raw)
To: Federico Cuello; +Cc: linux-kernel, Udo Wolter, cf
On Wednesday 28 January 2009 08:09:10 Federico Cuello wrote:
> Ralf Hildebrandt escribió:
> > I recently installed 2.6.28.2 on our postfix/dovecot-based
> > mailboxserver. Previously, 2.6.28 and 2.6.28.1 have been running there
> > without a hitch.
> >
> > Now with 2.6.28.2 I had two major lockups: All writes to the users'
> > Maildirs (on ext4) would stall, the load would rise, "sync" would never
> > return.
> >
> > I had to "reboot -f -n" to get the machine back. All hanging processes
> > were unkillable, even with kill -9.
> > [...]
>
> The same is happening to me, but I have some logs taken with sysrq.
>
> Here is my vmstat output:
>
> procs -----------memory---------- ---swap-- -----io---- -system--
> ----cpu----
> r b swpd free buff cache si so bi bo in cs us sy
> id wa
> 0 2 99028 46536 26356 1569400 0 0 0 0 1400 437 0
> 3 0 96
> 0 2 99028 46536 26356 1569400 0 0 0 0 1344 355 0
> 6 0 94
> 0 2 99028 46536 26356 1569400 0 0 0 0 1373 387 0
> 0 0 100
> 0 2 99028 46536 26364 1569400 0 0 0 12 1403 384 1
> 0 0 99
> 0 2 99028 46536 26364 1569400 0 0 0 0 1370 378 0
> 0 0 100
> 0 2 99028 46536 26364 1569400 0 0 0 0 1351 346 0
> 0 0 100
> 0 2 99028 46536 26364 1569400 0 0 0 0 1395 412 0
> 0 0 100
> 0 2 99028 46536 26364 1569400 0 0 0 0 1349 332 0
> 0 0 100
> 0 2 99028 46536 26368 1569400 0 0 4 0 1407 387 0
> 0 0 100
>
> Notice the 100% iowait.
>
> I also managed to reproduce it doing a rsync from one partition to a USB
> drive. After the lockup I can't read any file from the source partition,
> but the other partitions can be accessed normally.
Hm, thanks for reporting, can you guys get a sysrq+W trace when the system
reaches this state?
Thanks,
Nick
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: sync-Regression in 2.6.28.2?
2009-02-03 1:09 ` Nick Piggin
@ 2009-02-03 19:51 ` Federico Cuello
2009-02-04 6:17 ` Nick Piggin
0 siblings, 1 reply; 14+ messages in thread
From: Federico Cuello @ 2009-02-03 19:51 UTC (permalink / raw)
To: linux-kernel
Nick Piggin wrote:
> On Wednesday 28 January 2009 08:09:10 Federico Cuello wrote:
>
>> Ralf Hildebrandt escribió:
>>
>>> I recently installed 2.6.28.2 on our postfix/dovecot-based
>>> mailboxserver. Previously, 2.6.28 and 2.6.28.1 have been running there
>>> without a hitch.
>>>
>>> Now with 2.6.28.2 I had two major lockups: All writes to the users'
>>> Maildirs (on ext4) would stall, the load would rise, "sync" would never
>>> return.
>>>
>>> I had to "reboot -f -n" to get the machine back. All hanging processes
>>> were unkillable, even with kill -9.
>>> [...]
>>>
>> The same is happening to me, but I have some logs taken with sysrq.
>>
>> Here is my vmstat output:
>>
>> procs -----------memory---------- ---swap-- -----io---- -system--
>> ----cpu----
>> r b swpd free buff cache si so bi bo in cs us sy
>> id wa
>> 0 2 99028 46536 26356 1569400 0 0 0 0 1400 437 0
>> 3 0 96
>> 0 2 99028 46536 26356 1569400 0 0 0 0 1344 355 0
>> 6 0 94
>> 0 2 99028 46536 26356 1569400 0 0 0 0 1373 387 0
>> 0 0 100
>> 0 2 99028 46536 26364 1569400 0 0 0 12 1403 384 1
>> 0 0 99
>> 0 2 99028 46536 26364 1569400 0 0 0 0 1370 378 0
>> 0 0 100
>> 0 2 99028 46536 26364 1569400 0 0 0 0 1351 346 0
>> 0 0 100
>> 0 2 99028 46536 26364 1569400 0 0 0 0 1395 412 0
>> 0 0 100
>> 0 2 99028 46536 26364 1569400 0 0 0 0 1349 332 0
>> 0 0 100
>> 0 2 99028 46536 26368 1569400 0 0 4 0 1407 387 0
>> 0 0 100
>>
>> Notice the 100% iowait.
>>
>> I also managed to reproduce it doing a rsync from one partition to a USB
>> drive. After the lockup I can't read any file from the source partition,
>> but the other partitions can be accessed normally.
>>
>
> Hm, thanks for reporting, can you guys get a sysrq+W trace when the system
> reaches this state?
Here it is, if you need something else please ask:
Jan 26 23:44:38 kaoz [10117.763822] SysRq : Show Memory
Jan 26 23:44:38 kaoz [10117.765692] Mem-Info:
Jan 26 23:44:38 kaoz [10117.767141] DMA per-cpu:
Jan 26 23:44:38 kaoz [10117.769184] CPU 0: hi: 0, btch: 1 usd: 0
Jan 26 23:44:38 kaoz [10117.769184] CPU 1: hi: 0, btch: 1 usd: 0
Jan 26 23:44:38 kaoz [10117.769184] DMA32 per-cpu:
Jan 26 23:44:38 kaoz [10117.769184] CPU 0: hi: 186, btch: 31 usd: 42
Jan 26 23:44:38 kaoz [10117.769184] CPU 1: hi: 186, btch: 31 usd: 31
Jan 26 23:44:38 kaoz [10117.769184] Active_anon:24752 active_file:11041
inactive_anon:10001
Jan 26 23:44:38 kaoz [10117.769184] inactive_file:396899 unevictable:0
dirty:70 writeback:0 unstable:0
Jan 26 23:44:38 kaoz [10117.769184] free:43547 slab:15243 mapped:7057
pagetables:1711 bounce:0
Jan 26 23:44:38 kaoz [10117.769184] DMA free:8092kB min:28kB low:32kB
high:40kB active_anon:0kB inactive_anon:12kB active_file:2360kB
inactive_file:1324kB unevictable:0kB present:11100kB pages_scanned:0
all_unreclaimable? no
Jan 26 23:44:38 kaoz [10117.785214] lowmem_reserve[]: 0 2004 2004 2004
Jan 26 23:44:38 kaoz [10117.785214] DMA32 free:165928kB min:5712kB
low:7140kB high:8568kB active_anon:99008kB inactive_anon:39992kB
active_file:41804kB inactive_file:1586396kB unevictable:0kB
present:2052128kB pages_scanned:0 all_unreclaimable? no
Jan 26 23:44:38 kaoz [10117.785214] lowmem_reserve[]: 0 0 0 0
Jan 26 23:44:38 kaoz [10117.785214] DMA: 15*4kB 24*8kB 20*16kB 21*32kB
19*64kB 10*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 8092kB
Jan 26 23:44:38 kaoz [10117.785214] DMA32: 5284*4kB 5258*8kB 995*16kB
486*32kB 336*64kB 127*128kB 40*256kB 19*512kB 9*1024kB 2*2048kB 0*4096kB
= 165712kB
Jan 26 23:44:38 kaoz [10117.785214] 409973 total pagecache pages
Jan 26 23:44:38 kaoz [10117.785214] 1917 pages in swap cache
Jan 26 23:44:38 kaoz [10117.785214] Swap cache stats: add 38607, delete
36690, find 9693/11511
Jan 26 23:44:38 kaoz [10117.785214] Free swap = 1037804kB
Jan 26 23:44:38 kaoz [10117.785214] Total swap = 1048568kB
Jan 26 23:44:38 kaoz [10117.785214] 524240 pages RAM
Jan 26 23:44:38 kaoz [10117.785214] 9080 pages reserved
Jan 26 23:44:38 kaoz [10117.785214] 192696 pages shared
Jan 26 23:44:38 kaoz [10117.785214] 286249 pages non-shared
Jan 26 23:44:38 kaoz [10123.434993] SysRq : Emergency Remount R/O
Jan 26 23:44:38 kaoz [10127.036454] SysRq : HELP : loglevel0-8 reBoot
tErm Full kIll saK aLlcpus showMem Nice powerOff showPc
show-all-timers(Q) unRaw Sync showTasks Unmount shoW-blocked-tasks
Jan 26 23:44:38 kaoz [10136.390066] SysRq : Show Blocked State
Jan 26 23:44:38 kaoz [10136.391430] task PC
stack pid father
Jan 26 23:44:38 kaoz [10136.392801] kjournald2 D ffff880001014c20
0 4635 2
Jan 26 23:44:38 kaoz [10136.393391] ffff88007e5e7d50 0000000000000046
ffff8800581e9d00 ffff88007e5e38a0
Jan 26 23:44:38 kaoz [10136.393391] ffffffff805de000 ffff88007ea6ad80
ffffffff80544340 ffff88007ea6afe8
Jan 26 23:44:38 kaoz [10136.393391] 0000000000000000 000000010023bf96
ffff88007ea6afe8 ffff88007c5df090
Jan 26 23:44:38 kaoz [10136.393391] Call Trace:
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8022e140>]
update_curr+0xd0/0x140
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8031d561>]
jbd2_journal_commit_transaction+0x171/0x1240
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80474ef7>]
thread_return+0x3d/0x5e6
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80251d80>]
autoremove_wake_function+0x0/0x30
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80246204>]
lock_timer_base+0x34/0x70
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80246294>]
try_to_del_timer_sync+0x54/0x60
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80322548>]
kjournald2+0xd8/0x230
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80251d80>]
autoremove_wake_function+0x0/0x30
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80322470>]
kjournald2+0x0/0x230
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802519b7>] kthread+0x47/0x80
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802386c7>]
schedule_tail+0x27/0x70
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8020c7e9>] child_rip+0xa/0x11
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80251970>] kthread+0x0/0x80
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8020c7df>] child_rip+0x0/0x11
Jan 26 23:44:38 kaoz [10136.393391] quasselcore D ffff88000100ad98
0 6054 1
Jan 26 23:44:38 kaoz [10136.393391] ffff88006d2e5bb8 0000000000000082
ffff88007c5dec00 ffffffff80496fd0
Jan 26 23:44:38 kaoz [10136.393391] ffffffff805de000 ffff88007c52b8e0
ffff88005597c9f0 ffff88007c52bb48
Jan 26 23:44:38 kaoz [10136.393391] 0000000100000004 000000010023b9b0
ffff88007c52bb48 ffffffff802fa6a1
Jan 26 23:44:38 kaoz [10136.393391] Call Trace:
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802fa6a1>]
ext4_da_write_end+0x141/0x280
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffffa008c6cc>]
dm_table_unplug_all+0x3c/0x90 [dm_mod]
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8026f940>] sync_page+0x0/0x70
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff804754d7>]
io_schedule+0x37/0x50
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8026f975>]
sync_page+0x35/0x70
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8047585a>]
__wait_on_bit_lock+0x4a/0x80
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8026f91e>]
__lock_page+0x5e/0x70
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80251db0>]
wake_bit_function+0x0/0x30
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80276d1d>]
write_cache_pages+0x1ed/0x3d0
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802fbea0>]
__mpage_da_writepage+0x0/0x170
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802fbc57>]
ext4_da_writepages+0x237/0x3b0
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80251d80>]
autoremove_wake_function+0x0/0x30
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802fd1b0>]
ext4_da_get_block_write+0x0/0x160
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80276f50>]
do_writepages+0x20/0x40
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8027077d>]
__filemap_fdatawrite_range+0x4d/0x60
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802c12c5>] do_fsync+0x45/0xe0
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802c138e>]
__do_fsync+0x2e/0x50
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8020b8cb>]
system_call_fastpath+0x16/0x1b
Jan 26 23:44:38 kaoz [10136.393391] klauncher D ffff880001020c20
0 7760 1
Jan 26 23:44:38 kaoz [10136.393391] ffff880060b65cf0 0000000000000086
ffffffff8026f7ad 0000000000000000
Jan 26 23:44:38 kaoz [10136.393391] ffffffff805de000 ffff88005b8cc9f0
ffff88007f862220 ffff88005b8ccc58
Jan 26 23:44:38 kaoz [10136.393391] 00000001805e19e0 00000001002ce821
ffff88005b8ccc58 ffff880060b65cc8
Jan 26 23:44:38 kaoz [10136.393391] Call Trace:
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8026f7ad>]
find_get_page+0x8d/0xc0
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8022f783>]
__wake_up+0x43/0x70
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80475c25>]
__mutex_lock_slowpath+0x75/0xc0
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80475a5a>]
mutex_lock+0xa/0x10
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80271b4f>]
generic_file_aio_write+0x4f/0xe0
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802f67e8>]
ext4_file_write+0x58/0x170
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8029f69b>]
do_sync_write+0xdb/0x120
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80227320>]
do_page_fault+0x300/0x970
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80251d80>]
autoremove_wake_function+0x0/0x30
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802476c0>]
do_sigaction+0x130/0x1c0
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802a00bb>]
vfs_write+0xcb/0x170
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802a0263>]
sys_write+0x53/0xa0
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8020b8cb>]
system_call_fastpath+0x16/0x1b
Jan 26 23:44:38 kaoz [10136.393391] kded4 D ffff880001020c20
0 7762 1
Jan 26 23:44:38 kaoz [10136.393391] ffff880060999cf0 0000000000000086
ffffffff8026f7ad 0000000000000000
Jan 26 23:44:38 kaoz [10136.393391] ffffffff805de000 ffff88005b8b8000
ffff88007f862220 ffff88005b8b8268
Jan 26 23:44:38 kaoz [10136.393391] 00000001805e19e0 00000001002c4165
ffff88005b8b8268 ffff880060999e08
Jan 26 23:44:38 kaoz [10136.393391] Call Trace:
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8026f7ad>]
find_get_page+0x8d/0xc0
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80475c25>]
__mutex_lock_slowpath+0x75/0xc0
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80475a5a>]
mutex_lock+0xa/0x10
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80271b4f>]
generic_file_aio_write+0x4f/0xe0
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802f67e8>]
ext4_file_write+0x58/0x170
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8029f69b>]
do_sync_write+0xdb/0x120
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80227320>]
do_page_fault+0x300/0x970
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80251d80>]
autoremove_wake_function+0x0/0x30
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8028eda5>]
free_pages_and_swap_cache+0x85/0xb0
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802867a0>]
unmap_region+0x180/0x1a0
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802a00bb>]
vfs_write+0xcb/0x170
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802a0263>]
sys_write+0x53/0xa0
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8020b8cb>]
system_call_fastpath+0x16/0x1b
Jan 26 23:44:38 kaoz [10136.393391] ksmserver D ffff8800583319c8
0 7773 1
Jan 26 23:44:38 kaoz [10136.393391] ffff88005bb99cf0 0000000000000086
ffffffff8026f7ad 0000000000000000
Jan 26 23:44:38 kaoz [10136.393391] ffffffff805de000 ffff88005b8bb8e0
ffff88007c52e0b0 ffff88005b8bbb48
Jan 26 23:44:38 kaoz [10136.393391] 00000001805e19e0 ffffe200006b6fe0
ffff88005b8bbb48 ffff88005bb99cc8
Jan 26 23:44:38 kaoz [10136.393391] Call Trace:
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8026f7ad>]
find_get_page+0x8d/0xc0
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8022f783>]
__wake_up+0x43/0x70
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80475c25>]
__mutex_lock_slowpath+0x75/0xc0
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80475a5a>]
mutex_lock+0xa/0x10
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80271b4f>]
generic_file_aio_write+0x4f/0xe0
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802f67e8>]
ext4_file_write+0x58/0x170
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8029f69b>]
do_sync_write+0xdb/0x120
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80227320>]
do_page_fault+0x300/0x970
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8022f783>]
__wake_up+0x43/0x70
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80251d80>]
autoremove_wake_function+0x0/0x30
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80251c60>]
bit_waitqueue+0x10/0xc0
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802b0673>] d_kill+0x53/0x70
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802a00bb>]
vfs_write+0xcb/0x170
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802a0263>]
sys_write+0x53/0xa0
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8020b8cb>]
system_call_fastpath+0x16/0x1b
Jan 26 23:44:38 kaoz [10136.393391] kwin D ffff8800583319c8
0 7775 7773
Jan 26 23:44:38 kaoz [10136.393391] ffff88005ba6bcf0 0000000000000082
ffffffff8026f7ad 0000000000000000
Jan 26 23:44:38 kaoz [10136.393391] ffffffff805de000 ffff88005b8bf1c0
ffff88007c52e660 ffff88005b8bf428
Jan 26 23:44:38 kaoz [10136.393391] 00000001805e19e0 ffffe200008c8618
ffff88005b8bf428 0000000000000000
Jan 26 23:44:38 kaoz [10136.393391] Call Trace:
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8026f7ad>]
find_get_page+0x8d/0xc0
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80475c25>]
__mutex_lock_slowpath+0x75/0xc0
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80475a5a>]
mutex_lock+0xa/0x10
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80271b4f>]
generic_file_aio_write+0x4f/0xe0
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802f67e8>]
ext4_file_write+0x58/0x170
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8029f69b>]
do_sync_write+0xdb/0x120
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80227320>]
do_page_fault+0x300/0x970
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8022f783>]
__wake_up+0x43/0x70
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80251d80>]
autoremove_wake_function+0x0/0x30
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802a00bb>]
vfs_write+0xcb/0x170
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802a0263>]
sys_write+0x53/0xa0
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8020b8cb>]
system_call_fastpath+0x16/0x1b
Jan 26 23:44:38 kaoz [10136.393391] nepomukserver D ffff8800583319c8
0 7778 1
Jan 26 23:44:38 kaoz [10136.393391] ffff88007c0bfcf0 0000000000000082
ffffffff8026f7ad 0000000000000000
Jan 26 23:44:38 kaoz [10136.393391] ffffffff805de000 ffff88005b8b9110
ffff88007c52e660 ffff88005b8b9378
Jan 26 23:44:38 kaoz [10136.393391] 00000001805e19e0 ffffe200005c0370
ffff88005b8b9378 0000000000000000
Jan 26 23:44:38 kaoz [10136.393391] Call Trace:
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8026f7ad>]
find_get_page+0x8d/0xc0
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80475c25>]
__mutex_lock_slowpath+0x75/0xc0
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80475a5a>]
mutex_lock+0xa/0x10
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80271b4f>]
generic_file_aio_write+0x4f/0xe0
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802f67e8>]
ext4_file_write+0x58/0x170
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8029f69b>]
do_sync_write+0xdb/0x120
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80227320>]
do_page_fault+0x300/0x970
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80251d80>]
autoremove_wake_function+0x0/0x30
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802a00bb>]
vfs_write+0xcb/0x170
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802a0263>]
sys_write+0x53/0xa0
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8020b8cb>]
system_call_fastpath+0x16/0x1b
Jan 26 23:44:38 kaoz [10136.393391] plasma D ffff880001020c20
0 7781 1
Jan 26 23:44:38 kaoz [10136.393391] ffff88005b8ddcf0 0000000000000082
ffffffff8026f7ad 0000000000000000
Jan 26 23:44:38 kaoz [10136.393391] ffffffff805de000 ffff88005b8be660
ffff88007f862220 ffff88005b8be8c8
Jan 26 23:44:38 kaoz [10136.393391] 00000001805e19e0 00000001002c24a1
ffff88005b8be8c8 ffff88005b8dde08
Jan 26 23:44:38 kaoz [10136.393391] Call Trace:
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8026f7ad>]
find_get_page+0x8d/0xc0
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80475c25>]
__mutex_lock_slowpath+0x75/0xc0
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80475a5a>]
mutex_lock+0xa/0x10
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80271b4f>]
generic_file_aio_write+0x4f/0xe0
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802f67e8>]
ext4_file_write+0x58/0x170
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8029f69b>]
do_sync_write+0xdb/0x120
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80227320>]
do_page_fault+0x300/0x970
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80251d80>]
autoremove_wake_function+0x0/0x30
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802a3350>]
vfs_lstat_fd+0x20/0x60
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802a00bb>]
vfs_write+0xcb/0x170
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802a0263>]
sys_write+0x53/0xa0
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8020b8cb>]
system_call_fastpath+0x16/0x1b
Jan 26 23:44:38 kaoz [10136.393391] mysqld D 0000000000000002
0 7820 7804
Jan 26 23:44:38 kaoz [10136.393391] ffff880063a39b58 0000000000000082
ffff880063a39ad8 ffffffff80236cbb
Jan 26 23:44:38 kaoz [10136.393391] ffffffff805de000 ffff88005b8cb8e0
ffff88005597f770 ffff88005b8cbb48
Jan 26 23:44:38 kaoz [10136.393391] 0000000163a39b98 ffffffff80474ef7
ffff88005b8cbb48 0000000000000000
Jan 26 23:44:38 kaoz [10136.393391] Call Trace:
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80236cbb>]
finish_task_switch+0x2b/0xe0
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80474ef7>]
thread_return+0x3d/0x5e6
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8031cebb>]
start_this_handle+0x33b/0x450
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80251d80>]
autoremove_wake_function+0x0/0x30
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8031d1a6>]
jbd2_journal_start+0x96/0xe0
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802f9bc6>]
ext4_dirty_inode+0x36/0xb0
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802be294>]
__mark_inode_dirty+0x34/0x1d0
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802b33a0>]
file_update_time+0x110/0x130
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80270fd6>]
__generic_file_aio_write_nolock+0x246/0x470
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80280382>]
__do_fault+0x202/0x4c0
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80271b64>]
generic_file_aio_write+0x64/0xe0
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802f67e8>]
ext4_file_write+0x58/0x170
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8029f69b>]
do_sync_write+0xdb/0x120
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80227320>]
do_page_fault+0x300/0x970
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80251d80>]
autoremove_wake_function+0x0/0x30
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80223b50>]
flat_send_IPI_mask+0x0/0x60
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802a00bb>]
vfs_write+0xcb/0x170
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802a0263>]
sys_write+0x53/0xa0
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8020b8cb>]
system_call_fastpath+0x16/0x1b
Jan 26 23:44:38 kaoz [10136.393391] pdflush D ffff880001014c20
0 9198 2
Jan 26 23:44:38 kaoz [10136.393391] ffff88003c311a20 0000000000000046
ffff88007ea00000 0000000000000000
Jan 26 23:44:38 kaoz [10136.393391] ffffffff805de000 ffff88005597d550
ffffffff80544340 ffff88005597d7b8
Jan 26 23:44:38 kaoz [10136.393391] 0000000000000000 000000010023b9b6
ffff88005597d7b8 ffff88005591d900
Jan 26 23:44:38 kaoz [10136.393391] Call Trace:
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8032f3c7>]
generic_make_request+0x207/0x340
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffffa008c6cc>]
dm_table_unplug_all+0x3c/0x90 [dm_mod]
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8026f940>] sync_page+0x0/0x70
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff804754d7>]
io_schedule+0x37/0x50
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8026f975>]
sync_page+0x35/0x70
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8047585a>]
__wait_on_bit_lock+0x4a/0x80
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8026f91e>]
__lock_page+0x5e/0x70
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80251db0>]
wake_bit_function+0x0/0x30
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802786ca>]
pagevec_lookup_tag+0x1a/0x30
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80276d1d>]
write_cache_pages+0x1ed/0x3d0
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802fbea0>]
__mpage_da_writepage+0x0/0x170
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8031bb5f>]
jbd2_journal_stop+0x17f/0x200
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802fbc57>]
ext4_da_writepages+0x237/0x3b0
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8022e6a7>]
dequeue_task+0x57/0x70
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80236cbb>]
finish_task_switch+0x2b/0xe0
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802fd1b0>]
ext4_da_get_block_write+0x0/0x160
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80276f50>]
do_writepages+0x20/0x40
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802bd526>]
__writeback_single_inode+0xa6/0x470
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802bdeb0>]
generic_sync_sb_inodes+0x370/0x4d0
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802be1cd>]
writeback_inodes+0x4d/0xe0
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802775e3>]
background_writeout+0xb3/0xf0
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80277d1b>]
pdflush+0x15b/0x240
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80277530>]
background_writeout+0x0/0xf0
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80277bc0>] pdflush+0x0/0x240
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802519b7>] kthread+0x47/0x80
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802386c7>]
schedule_tail+0x27/0x70
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8020c7e9>] child_rip+0xa/0x11
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80251970>] kthread+0x0/0x80
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8020c7df>] child_rip+0x0/0x11
Jan 26 23:44:38 kaoz [10136.393391] krunner_lock D 0000000000000002
0 9309 1
Jan 26 23:44:38 kaoz [10136.393391] ffff88005b843b58 0000000000000082
ffff88005b843ad8 ffffffff8022e140
Jan 26 23:44:38 kaoz [10136.393391] ffffffff805de000 ffff88006d3eec10
ffff88007c52a7d0 ffff88006d3eee78
Jan 26 23:44:38 kaoz [10136.393391] 000000015b843b08 ffffffff8022f9f8
ffff88006d3eee78 ffff8800010252f0
Jan 26 23:44:38 kaoz [10136.393391] Call Trace:
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8022e140>]
update_curr+0xd0/0x140
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8022f9f8>]
dequeue_entity+0x18/0x190
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8022f958>]
set_next_entity+0x38/0x80
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8031cebb>]
start_this_handle+0x33b/0x450
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80251d80>]
autoremove_wake_function+0x0/0x30
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8031d1a6>]
jbd2_journal_start+0x96/0xe0
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802f9bc6>]
ext4_dirty_inode+0x36/0xb0
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802be294>]
__mark_inode_dirty+0x34/0x1d0
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802b33a0>]
file_update_time+0x110/0x130
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80270fd6>]
__generic_file_aio_write_nolock+0x246/0x470
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80280382>]
__do_fault+0x202/0x4c0
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80271b64>]
generic_file_aio_write+0x64/0xe0
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802f67e8>]
ext4_file_write+0x58/0x170
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8029f69b>]
do_sync_write+0xdb/0x120
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80227320>]
do_page_fault+0x300/0x970
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80251d80>]
autoremove_wake_function+0x0/0x30
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8028eda5>]
free_pages_and_swap_cache+0x85/0xb0
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802867a0>]
unmap_region+0x180/0x1a0
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802a00bb>]
vfs_write+0xcb/0x170
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802a0263>]
sys_write+0x53/0xa0
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8020b8cb>]
system_call_fastpath+0x16/0x1b
Jan 26 23:44:38 kaoz [10136.393391] krunner_lock D ffff8800583319c8
0 9310 9309
Jan 26 23:44:38 kaoz [10136.393391] ffff88007fa9bcf0 0000000000000082
ffffffff8026f7ad 0000000000000000
Jan 26 23:44:38 kaoz [10136.393391] ffffffff805de000 ffff88006d3ee660
ffff8800708d9c70 ffff88006d3ee8c8
Jan 26 23:44:38 kaoz [10136.393391] 00000001805e19e0 ffffe200010d3fb0
ffff88006d3ee8c8 ffff88007fa9bed8
Jan 26 23:44:38 kaoz [10136.393391] Call Trace:
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8026f7ad>]
find_get_page+0x8d/0xc0
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80475c25>]
__mutex_lock_slowpath+0x75/0xc0
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80475a5a>]
mutex_lock+0xa/0x10
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80271b4f>]
generic_file_aio_write+0x4f/0xe0
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802f67e8>]
ext4_file_write+0x58/0x170
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8029f69b>]
do_sync_write+0xdb/0x120
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80227320>]
do_page_fault+0x300/0x970
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80251d80>]
autoremove_wake_function+0x0/0x30
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8024940f>]
kill_pid_info+0x4f/0x80
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802496c2>]
sys_kill+0xa2/0x1c0
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802a00bb>]
vfs_write+0xcb/0x170
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802a0263>]
sys_write+0x53/0xa0
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8020b8cb>]
system_call_fastpath+0x16/0x1b
Jan 26 23:44:38 kaoz [10136.393391] kblankscrn.ks D ffff880001020c20
0 9313 9310
Jan 26 23:44:38 kaoz [10136.393391] ffff88004c291cf0 0000000000000082
ffffe200010d3fb0 ffffffff805e19e0
Jan 26 23:44:38 kaoz [10136.393391] ffffffff805de000 ffff88006d3eb8e0
ffff88007f862220 ffff88006d3ebb48
Jan 26 23:44:38 kaoz [10136.393391] 0000000180279245 000000010024fa44
ffff88006d3ebb48 ffffffff80570008
Jan 26 23:44:38 kaoz [10136.393391] Call Trace:
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80475c25>]
__mutex_lock_slowpath+0x75/0xc0
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80475a5a>]
mutex_lock+0xa/0x10
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80271b4f>]
generic_file_aio_write+0x4f/0xe0
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802f67e8>]
ext4_file_write+0x58/0x170
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8029f69b>]
do_sync_write+0xdb/0x120
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80227320>]
do_page_fault+0x300/0x970
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80251d80>]
autoremove_wake_function+0x0/0x30
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80286726>]
unmap_region+0x106/0x1a0
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802a00bb>]
vfs_write+0xcb/0x170
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802a0263>]
sys_write+0x53/0xa0
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8020b8cb>]
system_call_fastpath+0x16/0x1b
Jan 26 23:44:38 kaoz [10136.393391] less D ffff880001014c20
0 9633 1
Jan 26 23:44:38 kaoz [10136.393391] ffff88003ba2bec0 0000000000000086
ffff88003ba2be68 0000000000008000
Jan 26 23:44:38 kaoz [10136.393391] ffffffff805de000 ffff8800708d96c0
ffffffff80544340 ffff8800708d9928
Jan 26 23:44:38 kaoz [10136.393391] 0000000000000001 0000000100282d66
ffff8800708d9928 000000000000fe03
Jan 26 23:44:38 kaoz [10136.393391] Call Trace:
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80475c25>]
__mutex_lock_slowpath+0x75/0xc0
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80475a5a>]
mutex_lock+0xa/0x10
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802a06ff>]
generic_file_llseek+0x2f/0x70
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8029f31e>]
sys_lseek+0x7e/0x90
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8029da9b>]
do_sys_open+0xeb/0x110
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8020b8cb>]
system_call_fastpath+0x16/0x1b
Jan 26 23:44:38 kaoz [10136.393391] pdflush D ffffffff80552468
0 9675 2
Jan 26 23:44:38 kaoz [10136.393391] ffff880041ac9e38 0000000000000046
ffffffff80270658 ffff880041ac9dc0
Jan 26 23:44:38 kaoz [10136.393391] ffffffff805de000 ffff88005b8bbe90
ffff88007f863330 ffff88005b8bc0f8
Jan 26 23:44:38 kaoz [10136.393391] 000000017f85ff00 ffff880041ac9de0
ffff88005b8bc0f8 ffffffff805e5b80
Jan 26 23:44:38 kaoz [10136.393391] Call Trace:
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80270658>]
wait_on_page_writeback_range+0x78/0x130
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80276f50>]
do_writepages+0x20/0x40
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80475c25>]
__mutex_lock_slowpath+0x75/0xc0
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80475a5a>]
mutex_lock+0xa/0x10
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802a15c3>]
sync_filesystems+0x13/0x130
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802c145e>] do_sync+0x1e/0x60
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80277d1b>]
pdflush+0x15b/0x240
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802c1440>] do_sync+0x0/0x60
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80277bc0>] pdflush+0x0/0x240
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802519b7>] kthread+0x47/0x80
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802386c7>]
schedule_tail+0x27/0x70
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8020c7e9>] child_rip+0xa/0x11
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80251970>] kthread+0x0/0x80
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8020c7df>] child_rip+0x0/0x11
Jan 26 23:44:38 kaoz [10136.393391] shutdown D ffff880001020c20
0 10071 1
Jan 26 23:44:38 kaoz [10136.393391] ffff88003ac5de68 0000000000000086
ffff88003ac5de78 ffffffff80474ef7
Jan 26 23:44:38 kaoz [10136.393391] ffffffff805de000 ffff88005b8bad80
ffff88007f862220 ffff88005b8bafe8
Jan 26 23:44:38 kaoz [10136.393391] 000000017f862220 00000001002c243e
ffff88005b8bafe8 00000001002c2435
Jan 26 23:44:38 kaoz [10136.393391] Call Trace:
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80474ef7>]
thread_return+0x3d/0x5e6
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8031cebb>]
start_this_handle+0x33b/0x450
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80251d80>]
autoremove_wake_function+0x0/0x30
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8031d1a6>]
jbd2_journal_start+0x96/0xe0
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8031d1fe>]
jbd2_journal_force_commit+0xe/0x30
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802a1687>]
sync_filesystems+0xd7/0x130
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802c1465>] do_sync+0x25/0x60
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802c14ae>] sys_sync+0xe/0x20
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8020b8cb>]
system_call_fastpath+0x16/0x1b
Jan 26 23:44:38 kaoz [10136.393391] kdm D ffff880001014c20
0 10137 6943
Jan 26 23:44:38 kaoz [10136.393391] ffff88005b953cd8 0000000000000086
ffffffff80570008 ffff880062e76a28
Jan 26 23:44:38 kaoz [10136.393391] ffffffff805de000 ffff88005bbf5b00
ffffffff80544340 ffff88005bbf5d68
Jan 26 23:44:38 kaoz [10136.393391] 0000000000000012 00000001002ce9e5
ffff88005bbf5d68 ffffffff805e19e0
Jan 26 23:44:38 kaoz [10136.393391] Call Trace:
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802b2557>]
__d_lookup+0xc7/0x140
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8031cebb>]
start_this_handle+0x33b/0x450
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80251d80>]
autoremove_wake_function+0x0/0x30
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8031d1a6>]
jbd2_journal_start+0x96/0xe0
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80301cf5>]
ext4_create+0x85/0x120
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802a904e>]
vfs_create+0xae/0xf0
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802ac836>]
do_filp_open+0x856/0x960
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8022e140>]
update_curr+0xd0/0x140
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802a33c3>]
vfs_stat_fd+0x23/0x60
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802b5c42>]
alloc_fd+0x122/0x150
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8029da30>]
do_sys_open+0x80/0x110
Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8020b8cb>]
system_call_fastpath+0x16/0x1b
Jan 26 23:44:38 kaoz [10141.340218] SysRq : Show Blocked State
Jan 26 23:44:38 kaoz [10141.341232] task PC
stack pid father
Jan 26 23:44:38 kaoz [10141.342274] kjournald2 D ffff880001014c20
0 4635 2
Jan 26 23:44:38 kaoz [10141.343331] ffff88007e5e7d50 0000000000000046
ffff8800581e9d00 ffff88007e5e38a0
Jan 26 23:44:38 kaoz [10141.343544] ffffffff805de000 ffff88007ea6ad80
ffffffff80544340 ffff88007ea6afe8
Jan 26 23:44:38 kaoz [10141.343544] 0000000000000000 000000010023bf96
ffff88007ea6afe8 ffff88007c5df090
Jan 26 23:44:38 kaoz [10141.343544] Call Trace:
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8022e140>]
update_curr+0xd0/0x140
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8031d561>]
jbd2_journal_commit_transaction+0x171/0x1240
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80474ef7>]
thread_return+0x3d/0x5e6
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80251d80>]
autoremove_wake_function+0x0/0x30
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80246204>]
lock_timer_base+0x34/0x70
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80246294>]
try_to_del_timer_sync+0x54/0x60
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80322548>]
kjournald2+0xd8/0x230
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80251d80>]
autoremove_wake_function+0x0/0x30
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80322470>]
kjournald2+0x0/0x230
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802519b7>] kthread+0x47/0x80
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802386c7>]
schedule_tail+0x27/0x70
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8020c7e9>] child_rip+0xa/0x11
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80251970>] kthread+0x0/0x80
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8020c7df>] child_rip+0x0/0x11
Jan 26 23:44:38 kaoz [10141.343544] quasselcore D ffff88000100ad98
0 6054 1
Jan 26 23:44:38 kaoz [10141.343544] ffff88006d2e5bb8 0000000000000082
ffff88007c5dec00 ffffffff80496fd0
Jan 26 23:44:38 kaoz [10141.343544] ffffffff805de000 ffff88007c52b8e0
ffff88005597c9f0 ffff88007c52bb48
Jan 26 23:44:38 kaoz [10141.343544] 0000000100000004 000000010023b9b0
ffff88007c52bb48 ffffffff802fa6a1
Jan 26 23:44:38 kaoz [10141.343544] Call Trace:
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802fa6a1>]
ext4_da_write_end+0x141/0x280
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffffa008c6cc>]
dm_table_unplug_all+0x3c/0x90 [dm_mod]
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8026f940>] sync_page+0x0/0x70
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff804754d7>]
io_schedule+0x37/0x50
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8026f975>]
sync_page+0x35/0x70
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8047585a>]
__wait_on_bit_lock+0x4a/0x80
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8026f91e>]
__lock_page+0x5e/0x70
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80251db0>]
wake_bit_function+0x0/0x30
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80276d1d>]
write_cache_pages+0x1ed/0x3d0
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802fbea0>]
__mpage_da_writepage+0x0/0x170
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802fbc57>]
ext4_da_writepages+0x237/0x3b0
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80251d80>]
autoremove_wake_function+0x0/0x30
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802fd1b0>]
ext4_da_get_block_write+0x0/0x160
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80276f50>]
do_writepages+0x20/0x40
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8027077d>]
__filemap_fdatawrite_range+0x4d/0x60
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802c12c5>] do_fsync+0x45/0xe0
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802c138e>]
__do_fsync+0x2e/0x50
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8020b8cb>]
system_call_fastpath+0x16/0x1b
Jan 26 23:44:38 kaoz [10141.343544] klauncher D ffff880001020c20
0 7760 1
Jan 26 23:44:38 kaoz [10141.343544] ffff880060b65cf0 0000000000000086
ffffffff8026f7ad 0000000000000000
Jan 26 23:44:38 kaoz [10141.343544] ffffffff805de000 ffff88005b8cc9f0
ffff88007f862220 ffff88005b8ccc58
Jan 26 23:44:38 kaoz [10141.343544] 00000001805e19e0 00000001002ce821
ffff88005b8ccc58 ffff880060b65cc8
Jan 26 23:44:38 kaoz [10141.343544] Call Trace:
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8026f7ad>]
find_get_page+0x8d/0xc0
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8022f783>]
__wake_up+0x43/0x70
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80475c25>]
__mutex_lock_slowpath+0x75/0xc0
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80475a5a>]
mutex_lock+0xa/0x10
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80271b4f>]
generic_file_aio_write+0x4f/0xe0
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802f67e8>]
ext4_file_write+0x58/0x170
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8029f69b>]
do_sync_write+0xdb/0x120
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80227320>]
do_page_fault+0x300/0x970
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80251d80>]
autoremove_wake_function+0x0/0x30
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802476c0>]
do_sigaction+0x130/0x1c0
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802a00bb>]
vfs_write+0xcb/0x170
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802a0263>]
sys_write+0x53/0xa0
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8020b8cb>]
system_call_fastpath+0x16/0x1b
Jan 26 23:44:38 kaoz [10141.343544] kded4 D ffff880001020c20
0 7762 1
Jan 26 23:44:38 kaoz [10141.343544] ffff880060999cf0 0000000000000086
ffffffff8026f7ad 0000000000000000
Jan 26 23:44:38 kaoz [10141.343544] ffffffff805de000 ffff88005b8b8000
ffff88007f862220 ffff88005b8b8268
Jan 26 23:44:38 kaoz [10141.343544] 00000001805e19e0 00000001002c4165
ffff88005b8b8268 ffff880060999e08
Jan 26 23:44:38 kaoz [10141.343544] Call Trace:
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8026f7ad>]
find_get_page+0x8d/0xc0
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80475c25>]
__mutex_lock_slowpath+0x75/0xc0
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80475a5a>]
mutex_lock+0xa/0x10
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80271b4f>]
generic_file_aio_write+0x4f/0xe0
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802f67e8>]
ext4_file_write+0x58/0x170
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8029f69b>]
do_sync_write+0xdb/0x120
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80227320>]
do_page_fault+0x300/0x970
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80251d80>]
autoremove_wake_function+0x0/0x30
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8028eda5>]
free_pages_and_swap_cache+0x85/0xb0
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802867a0>]
unmap_region+0x180/0x1a0
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802a00bb>]
vfs_write+0xcb/0x170
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802a0263>]
sys_write+0x53/0xa0
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8020b8cb>]
system_call_fastpath+0x16/0x1b
Jan 26 23:44:38 kaoz [10141.343544] ksmserver D ffff8800583319c8
0 7773 1
Jan 26 23:44:38 kaoz [10141.343544] ffff88005bb99cf0 0000000000000086
ffffffff8026f7ad 0000000000000000
Jan 26 23:44:38 kaoz [10141.343544] ffffffff805de000 ffff88005b8bb8e0
ffff88007c52e0b0 ffff88005b8bbb48
Jan 26 23:44:38 kaoz [10141.343544] 00000001805e19e0 ffffe200006b6fe0
ffff88005b8bbb48 ffff88005bb99cc8
Jan 26 23:44:38 kaoz [10141.343544] Call Trace:
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8026f7ad>]
find_get_page+0x8d/0xc0
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8022f783>]
__wake_up+0x43/0x70
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80475c25>]
__mutex_lock_slowpath+0x75/0xc0
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80475a5a>]
mutex_lock+0xa/0x10
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80271b4f>]
generic_file_aio_write+0x4f/0xe0
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802f67e8>]
ext4_file_write+0x58/0x170
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8029f69b>]
do_sync_write+0xdb/0x120
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80227320>]
do_page_fault+0x300/0x970
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8022f783>]
__wake_up+0x43/0x70
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80251d80>]
autoremove_wake_function+0x0/0x30
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80251c60>]
bit_waitqueue+0x10/0xc0
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802b0673>] d_kill+0x53/0x70
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802a00bb>]
vfs_write+0xcb/0x170
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802a0263>]
sys_write+0x53/0xa0
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8020b8cb>]
system_call_fastpath+0x16/0x1b
Jan 26 23:44:38 kaoz [10141.343544] kwin D ffff8800583319c8
0 7775 7773
Jan 26 23:44:38 kaoz [10141.343544] ffff88005ba6bcf0 0000000000000082
ffffffff8026f7ad 0000000000000000
Jan 26 23:44:38 kaoz [10141.343544] ffffffff805de000 ffff88005b8bf1c0
ffff88007c52e660 ffff88005b8bf428
Jan 26 23:44:38 kaoz [10141.343544] 00000001805e19e0 ffffe200008c8618
ffff88005b8bf428 0000000000000000
Jan 26 23:44:38 kaoz [10141.343544] Call Trace:
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8026f7ad>]
find_get_page+0x8d/0xc0
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80475c25>]
__mutex_lock_slowpath+0x75/0xc0
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80475a5a>]
mutex_lock+0xa/0x10
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80271b4f>]
generic_file_aio_write+0x4f/0xe0
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802f67e8>]
ext4_file_write+0x58/0x170
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8029f69b>]
do_sync_write+0xdb/0x120
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80227320>]
do_page_fault+0x300/0x970
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8022f783>]
__wake_up+0x43/0x70
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80251d80>]
autoremove_wake_function+0x0/0x30
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802a00bb>]
vfs_write+0xcb/0x170
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802a0263>]
sys_write+0x53/0xa0
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8020b8cb>]
system_call_fastpath+0x16/0x1b
Jan 26 23:44:38 kaoz [10141.343544] nepomukserver D ffff8800583319c8
0 7778 1
Jan 26 23:44:38 kaoz [10141.343544] ffff88007c0bfcf0 0000000000000082
ffffffff8026f7ad 0000000000000000
Jan 26 23:44:38 kaoz [10141.343544] ffffffff805de000 ffff88005b8b9110
ffff88007c52e660 ffff88005b8b9378
Jan 26 23:44:38 kaoz [10141.343544] 00000001805e19e0 ffffe200005c0370
ffff88005b8b9378 0000000000000000
Jan 26 23:44:38 kaoz [10141.343544] Call Trace:
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8026f7ad>]
find_get_page+0x8d/0xc0
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80475c25>]
__mutex_lock_slowpath+0x75/0xc0
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80475a5a>]
mutex_lock+0xa/0x10
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80271b4f>]
generic_file_aio_write+0x4f/0xe0
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802f67e8>]
ext4_file_write+0x58/0x170
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8029f69b>]
do_sync_write+0xdb/0x120
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80227320>]
do_page_fault+0x300/0x970
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80251d80>]
autoremove_wake_function+0x0/0x30
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802a00bb>]
vfs_write+0xcb/0x170
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802a0263>]
sys_write+0x53/0xa0
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8020b8cb>]
system_call_fastpath+0x16/0x1b
Jan 26 23:44:38 kaoz [10141.343544] plasma D ffff880001020c20
0 7781 1
Jan 26 23:44:38 kaoz [10141.343544] ffff88005b8ddcf0 0000000000000082
ffffffff8026f7ad 0000000000000000
Jan 26 23:44:38 kaoz [10141.343544] ffffffff805de000 ffff88005b8be660
ffff88007f862220 ffff88005b8be8c8
Jan 26 23:44:38 kaoz [10141.343544] 00000001805e19e0 00000001002c24a1
ffff88005b8be8c8 ffff88005b8dde08
Jan 26 23:44:38 kaoz [10141.343544] Call Trace:
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8026f7ad>]
find_get_page+0x8d/0xc0
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80475c25>]
__mutex_lock_slowpath+0x75/0xc0
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80475a5a>]
mutex_lock+0xa/0x10
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80271b4f>]
generic_file_aio_write+0x4f/0xe0
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802f67e8>]
ext4_file_write+0x58/0x170
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8029f69b>]
do_sync_write+0xdb/0x120
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80227320>]
do_page_fault+0x300/0x970
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80251d80>]
autoremove_wake_function+0x0/0x30
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802a3350>]
vfs_lstat_fd+0x20/0x60
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802a00bb>]
vfs_write+0xcb/0x170
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802a0263>]
sys_write+0x53/0xa0
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8020b8cb>]
system_call_fastpath+0x16/0x1b
Jan 26 23:44:38 kaoz [10141.343544] mysqld D 0000000000000002
0 7820 7804
Jan 26 23:44:38 kaoz [10141.343544] ffff880063a39b58 0000000000000082
ffff880063a39ad8 ffffffff80236cbb
Jan 26 23:44:38 kaoz [10141.343544] ffffffff805de000 ffff88005b8cb8e0
ffff88005597f770 ffff88005b8cbb48
Jan 26 23:44:38 kaoz [10141.343544] 0000000163a39b98 ffffffff80474ef7
ffff88005b8cbb48 0000000000000000
Jan 26 23:44:38 kaoz [10141.343544] Call Trace:
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80236cbb>]
finish_task_switch+0x2b/0xe0
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80474ef7>]
thread_return+0x3d/0x5e6
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8031cebb>]
start_this_handle+0x33b/0x450
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80251d80>]
autoremove_wake_function+0x0/0x30
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8031d1a6>]
jbd2_journal_start+0x96/0xe0
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802f9bc6>]
ext4_dirty_inode+0x36/0xb0
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802be294>]
__mark_inode_dirty+0x34/0x1d0
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802b33a0>]
file_update_time+0x110/0x130
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80270fd6>]
__generic_file_aio_write_nolock+0x246/0x470
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80280382>]
__do_fault+0x202/0x4c0
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80271b64>]
generic_file_aio_write+0x64/0xe0
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802f67e8>]
ext4_file_write+0x58/0x170
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8029f69b>]
do_sync_write+0xdb/0x120
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80227320>]
do_page_fault+0x300/0x970
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80251d80>]
autoremove_wake_function+0x0/0x30
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80223b50>]
flat_send_IPI_mask+0x0/0x60
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802a00bb>]
vfs_write+0xcb/0x170
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802a0263>]
sys_write+0x53/0xa0
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8020b8cb>]
system_call_fastpath+0x16/0x1b
Jan 26 23:44:38 kaoz [10141.343544] pdflush D ffff880001014c20
0 9198 2
Jan 26 23:44:38 kaoz [10141.343544] ffff88003c311a20 0000000000000046
ffff88007ea00000 0000000000000000
Jan 26 23:44:38 kaoz [10141.343544] ffffffff805de000 ffff88005597d550
ffffffff80544340 ffff88005597d7b8
Jan 26 23:44:38 kaoz [10141.343544] 0000000000000000 000000010023b9b6
ffff88005597d7b8 ffff88005591d900
Jan 26 23:44:38 kaoz [10141.343544] Call Trace:
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8032f3c7>]
generic_make_request+0x207/0x340
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffffa008c6cc>]
dm_table_unplug_all+0x3c/0x90 [dm_mod]
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8026f940>] sync_page+0x0/0x70
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff804754d7>]
io_schedule+0x37/0x50
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8026f975>]
sync_page+0x35/0x70
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8047585a>]
__wait_on_bit_lock+0x4a/0x80
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8026f91e>]
__lock_page+0x5e/0x70
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80251db0>]
wake_bit_function+0x0/0x30
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802786ca>]
pagevec_lookup_tag+0x1a/0x30
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80276d1d>]
write_cache_pages+0x1ed/0x3d0
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802fbea0>]
__mpage_da_writepage+0x0/0x170
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8031bb5f>]
jbd2_journal_stop+0x17f/0x200
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802fbc57>]
ext4_da_writepages+0x237/0x3b0
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8022e6a7>]
dequeue_task+0x57/0x70
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80236cbb>]
finish_task_switch+0x2b/0xe0
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802fd1b0>]
ext4_da_get_block_write+0x0/0x160
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80276f50>]
do_writepages+0x20/0x40
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802bd526>]
__writeback_single_inode+0xa6/0x470
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802bdeb0>]
generic_sync_sb_inodes+0x370/0x4d0
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802be1cd>]
writeback_inodes+0x4d/0xe0
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802775e3>]
background_writeout+0xb3/0xf0
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80277d1b>]
pdflush+0x15b/0x240
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80277530>]
background_writeout+0x0/0xf0
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80277bc0>] pdflush+0x0/0x240
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802519b7>] kthread+0x47/0x80
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802386c7>]
schedule_tail+0x27/0x70
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8020c7e9>] child_rip+0xa/0x11
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80251970>] kthread+0x0/0x80
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8020c7df>] child_rip+0x0/0x11
Jan 26 23:44:38 kaoz [10141.343544] krunner_lock D 0000000000000002
0 9309 1
Jan 26 23:44:38 kaoz [10141.343544] ffff88005b843b58 0000000000000082
ffff88005b843ad8 ffffffff8022e140
Jan 26 23:44:38 kaoz [10141.343544] ffffffff805de000 ffff88006d3eec10
ffff88007c52a7d0 ffff88006d3eee78
Jan 26 23:44:38 kaoz [10141.343544] 000000015b843b08 ffffffff8022f9f8
ffff88006d3eee78 ffff8800010252f0
Jan 26 23:44:38 kaoz [10141.343544] Call Trace:
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8022e140>]
update_curr+0xd0/0x140
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8022f9f8>]
dequeue_entity+0x18/0x190
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8022f958>]
set_next_entity+0x38/0x80
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8031cebb>]
start_this_handle+0x33b/0x450
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80251d80>]
autoremove_wake_function+0x0/0x30
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8031d1a6>]
jbd2_journal_start+0x96/0xe0
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802f9bc6>]
ext4_dirty_inode+0x36/0xb0
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802be294>]
__mark_inode_dirty+0x34/0x1d0
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802b33a0>]
file_update_time+0x110/0x130
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80270fd6>]
__generic_file_aio_write_nolock+0x246/0x470
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80280382>]
__do_fault+0x202/0x4c0
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80271b64>]
generic_file_aio_write+0x64/0xe0
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802f67e8>]
ext4_file_write+0x58/0x170
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8029f69b>]
do_sync_write+0xdb/0x120
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80227320>]
do_page_fault+0x300/0x970
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80251d80>]
autoremove_wake_function+0x0/0x30
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8028eda5>]
free_pages_and_swap_cache+0x85/0xb0
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802867a0>]
unmap_region+0x180/0x1a0
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802a00bb>]
vfs_write+0xcb/0x170
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802a0263>]
sys_write+0x53/0xa0
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8020b8cb>]
system_call_fastpath+0x16/0x1b
Jan 26 23:44:38 kaoz [10141.343544] krunner_lock D ffff8800583319c8
0 9310 9309
Jan 26 23:44:38 kaoz [10141.343544] ffff88007fa9bcf0 0000000000000082
ffffffff8026f7ad 0000000000000000
Jan 26 23:44:38 kaoz [10141.343544] ffffffff805de000 ffff88006d3ee660
ffff8800708d9c70 ffff88006d3ee8c8
Jan 26 23:44:38 kaoz [10141.343544] 00000001805e19e0 ffffe200010d3fb0
ffff88006d3ee8c8 ffff88007fa9bed8
Jan 26 23:44:38 kaoz [10141.343544] Call Trace:
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8026f7ad>]
find_get_page+0x8d/0xc0
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80475c25>]
__mutex_lock_slowpath+0x75/0xc0
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80475a5a>]
mutex_lock+0xa/0x10
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80271b4f>]
generic_file_aio_write+0x4f/0xe0
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802f67e8>]
ext4_file_write+0x58/0x170
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8029f69b>]
do_sync_write+0xdb/0x120
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80227320>]
do_page_fault+0x300/0x970
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80251d80>]
autoremove_wake_function+0x0/0x30
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8024940f>]
kill_pid_info+0x4f/0x80
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802496c2>]
sys_kill+0xa2/0x1c0
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802a00bb>]
vfs_write+0xcb/0x170
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802a0263>]
sys_write+0x53/0xa0
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8020b8cb>]
system_call_fastpath+0x16/0x1b
Jan 26 23:44:38 kaoz [10141.343544] kblankscrn.ks D ffff880001020c20
0 9313 9310
Jan 26 23:44:38 kaoz [10141.343544] ffff88004c291cf0 0000000000000082
ffffe200010d3fb0 ffffffff805e19e0
Jan 26 23:44:38 kaoz [10141.343544] ffffffff805de000 ffff88006d3eb8e0
ffff88007f862220 ffff88006d3ebb48
Jan 26 23:44:38 kaoz [10141.343544] 0000000180279245 000000010024fa44
ffff88006d3ebb48 ffffffff80570008
Jan 26 23:44:38 kaoz [10141.343544] Call Trace:
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80475c25>]
__mutex_lock_slowpath+0x75/0xc0
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80475a5a>]
mutex_lock+0xa/0x10
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80271b4f>]
generic_file_aio_write+0x4f/0xe0
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802f67e8>]
ext4_file_write+0x58/0x170
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8029f69b>]
do_sync_write+0xdb/0x120
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80227320>]
do_page_fault+0x300/0x970
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80251d80>]
autoremove_wake_function+0x0/0x30
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80286726>]
unmap_region+0x106/0x1a0
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802a00bb>]
vfs_write+0xcb/0x170
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802a0263>]
sys_write+0x53/0xa0
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8020b8cb>]
system_call_fastpath+0x16/0x1b
Jan 26 23:44:38 kaoz [10141.343544] less D ffff880001014c20
0 9633 1
Jan 26 23:44:38 kaoz [10141.343544] ffff88003ba2bec0 0000000000000086
ffff88003ba2be68 0000000000008000
Jan 26 23:44:38 kaoz [10141.343544] ffffffff805de000 ffff8800708d96c0
ffffffff80544340 ffff8800708d9928
Jan 26 23:44:38 kaoz [10141.343544] 0000000000000001 0000000100282d66
ffff8800708d9928 000000000000fe03
Jan 26 23:44:38 kaoz [10141.343544] Call Trace:
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80475c25>]
__mutex_lock_slowpath+0x75/0xc0
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80475a5a>]
mutex_lock+0xa/0x10
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802a06ff>]
generic_file_llseek+0x2f/0x70
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8029f31e>]
sys_lseek+0x7e/0x90
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8029da9b>]
do_sys_open+0xeb/0x110
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8020b8cb>]
system_call_fastpath+0x16/0x1b
Jan 26 23:44:38 kaoz [10141.343544] pdflush D ffffffff80552468
0 9675 2
Jan 26 23:44:38 kaoz [10141.343544] ffff880041ac9e38 0000000000000046
ffffffff80270658 ffff880041ac9dc0
Jan 26 23:44:38 kaoz [10141.343544] ffffffff805de000 ffff88005b8bbe90
ffff88007f863330 ffff88005b8bc0f8
Jan 26 23:44:38 kaoz [10141.343544] 000000017f85ff00 ffff880041ac9de0
ffff88005b8bc0f8 ffffffff805e5b80
Jan 26 23:44:38 kaoz [10141.343544] Call Trace:
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80270658>]
wait_on_page_writeback_range+0x78/0x130
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80276f50>]
do_writepages+0x20/0x40
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80475c25>]
__mutex_lock_slowpath+0x75/0xc0
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80475a5a>]
mutex_lock+0xa/0x10
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802a15c3>]
sync_filesystems+0x13/0x130
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802c145e>] do_sync+0x1e/0x60
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80277d1b>]
pdflush+0x15b/0x240
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802c1440>] do_sync+0x0/0x60
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80277bc0>] pdflush+0x0/0x240
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802519b7>] kthread+0x47/0x80
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802386c7>]
schedule_tail+0x27/0x70
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8020c7e9>] child_rip+0xa/0x11
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80251970>] kthread+0x0/0x80
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8020c7df>] child_rip+0x0/0x11
Jan 26 23:44:38 kaoz [10141.343544] shutdown D ffff880001020c20
0 10071 1
Jan 26 23:44:38 kaoz [10141.343544] ffff88003ac5de68 0000000000000086
ffff88003ac5de78 ffffffff80474ef7
Jan 26 23:44:38 kaoz [10141.343544] ffffffff805de000 ffff88005b8bad80
ffff88007f862220 ffff88005b8bafe8
Jan 26 23:44:38 kaoz [10141.343544] 000000017f862220 00000001002c243e
ffff88005b8bafe8 00000001002c2435
Jan 26 23:44:38 kaoz [10141.343544] Call Trace:
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80474ef7>]
thread_return+0x3d/0x5e6
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8031cebb>]
start_this_handle+0x33b/0x450
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80251d80>]
autoremove_wake_function+0x0/0x30
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8031d1a6>]
jbd2_journal_start+0x96/0xe0
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8031d1fe>]
jbd2_journal_force_commit+0xe/0x30
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802a1687>]
sync_filesystems+0xd7/0x130
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802c1465>] do_sync+0x25/0x60
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802c14ae>] sys_sync+0xe/0x20
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8020b8cb>]
system_call_fastpath+0x16/0x1b
Jan 26 23:44:38 kaoz [10141.343544] kdm D ffff880001014c20
0 10137 6943
Jan 26 23:44:38 kaoz [10141.343544] ffff88005b953cd8 0000000000000086
ffffffff80570008 ffff880062e76a28
Jan 26 23:44:38 kaoz [10141.343544] ffffffff805de000 ffff88005bbf5b00
ffffffff80544340 ffff88005bbf5d68
Jan 26 23:44:38 kaoz [10141.343544] 0000000000000012 00000001002ce9e5
ffff88005bbf5d68 ffffffff805e19e0
Jan 26 23:44:38 kaoz [10141.343544] Call Trace:
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802b2557>]
__d_lookup+0xc7/0x140
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8031cebb>]
start_this_handle+0x33b/0x450
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80251d80>]
autoremove_wake_function+0x0/0x30
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8031d1a6>]
jbd2_journal_start+0x96/0xe0
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80301cf5>]
ext4_create+0x85/0x120
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802a904e>]
vfs_create+0xae/0xf0
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802ac836>]
do_filp_open+0x856/0x960
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8022e140>]
update_curr+0xd0/0x140
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802a33c3>]
vfs_stat_fd+0x23/0x60
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802b5c42>]
alloc_fd+0x122/0x150
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8029da30>]
do_sys_open+0x80/0x110
Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8020b8cb>]
system_call_fastpath+0x16/0x1b
Jan 26 23:44:38 kaoz [10155.129152] SysRq : HELP : loglevel0-8 reBoot
tErm Full kIll saK aLlcpus showMem Nice powerOff showPc
show-all-timers(Q) unRaw Sync showTasks Unmount shoW-blocked-tasks
Jan 26 23:44:38 kaoz [10161.699932] SysRq : Show clockevent devices &
pending hrtimers (no others)
Jan 26 23:44:38 kaoz [10161.701338] Timer List Version: v0.4
Jan 26 23:44:38 kaoz [10161.702747] HRTIMER_MAX_CLOCK_BASES: 2
Jan 26 23:44:38 kaoz [10161.703258] now at 10161701339810 nsecs
Jan 26 23:44:38 kaoz [10161.703258]
Jan 26 23:44:38 kaoz [10161.703258] cpu: 0
Jan 26 23:44:38 kaoz [10161.703258] clock 0:
Jan 26 23:44:38 kaoz [10161.703258] .base: ffff880001015428
Jan 26 23:44:38 kaoz [10161.703258] .index: 0
Jan 26 23:44:38 kaoz [10161.703258] .resolution: 1 nsecs
Jan 26 23:44:38 kaoz [10161.703258] .get_time: ktime_get_real
Jan 26 23:44:38 kaoz [10161.703258] .offset: 1233010481853079954 nsecs
Jan 26 23:44:38 kaoz [10161.703258] active timers:
Jan 26 23:44:38 kaoz [10161.703258] clock 1:
Jan 26 23:44:38 kaoz [10161.703258] .base: ffff880001015468
Jan 26 23:44:38 kaoz [10161.703258] .index: 1
Jan 26 23:44:38 kaoz [10161.703258] .resolution: 1 nsecs
Jan 26 23:44:38 kaoz [10161.703258] .get_time: ktime_get
Jan 26 23:44:38 kaoz [10161.703258] .offset: 0 nsecs
Jan 26 23:44:38 kaoz [10161.703258] active timers:
Jan 26 23:44:38 kaoz [10161.703258] #0: <ffff880001015500>,
tick_sched_timer, S:01
Jan 26 23:44:38 kaoz [10161.703258] # expires at
10162345650432-10162345650432 nsecs [in 644310622 to 644310622 nsecs]
Jan 26 23:44:38 kaoz [10161.703258] #1: <ffff88005bb220e8>, it_real_fn,
S:01
Jan 26 23:44:38 kaoz [10161.703258] # expires at
32702039507086-32702039507086 nsecs [in 22540338167276 to 22540338167276
nsecs]
Jan 26 23:44:38 kaoz [10161.703258] .expires_next : 10162345650432 nsecs
Jan 26 23:44:38 kaoz [10161.703258] .hres_active : 1
Jan 26 23:44:38 kaoz [10161.703258] .nr_events : 1715455
Jan 26 23:44:38 kaoz [10161.703258] .nohz_mode : 2
Jan 26 23:44:38 kaoz [10161.703258] .idle_tick : 10161348983865 nsecs
Jan 26 23:44:38 kaoz [10161.703258] .tick_stopped : 1
Jan 26 23:44:38 kaoz [10161.703258] .idle_jiffies : 4297925700
Jan 26 23:44:38 kaoz [10161.703258] .idle_calls : 2284355
Jan 26 23:44:38 kaoz [10161.703258] .idle_sleeps : 164609
Jan 26 23:44:38 kaoz [10161.703258] .idle_entrytime : 10161737675287 nsecs
Jan 26 23:44:38 kaoz [10161.703258] .idle_waketime : 10161737671047 nsecs
Jan 26 23:44:38 kaoz [10161.703258] .idle_exittime : 10161345670207 nsecs
Jan 26 23:44:38 kaoz [10161.703258] .idle_sleeptime : 8728154262894 nsecs
Jan 26 23:44:38 kaoz [10161.703258] .last_jiffies : 4297925817
Jan 26 23:44:38 kaoz [10161.703258] .next_jiffies : 4297926000
Jan 26 23:44:38 kaoz [10161.703258] .idle_expires : 10162345650432 nsecs
Jan 26 23:44:38 kaoz [10161.703258] jiffies: 4297925817
Jan 26 23:44:38 kaoz [10161.703258]
Jan 26 23:44:38 kaoz [10161.703258] cpu: 1
Jan 26 23:44:38 kaoz [10161.703258] clock 0:
Jan 26 23:44:38 kaoz [10161.703258] .base: ffff880001021428
Jan 26 23:44:38 kaoz [10161.703258] .index: 0
Jan 26 23:44:38 kaoz [10161.703258] .resolution: 1 nsecs
Jan 26 23:44:38 kaoz [10161.703258] .get_time: ktime_get_real
Jan 26 23:44:38 kaoz [10161.703258] .offset: 1233010481853079954 nsecs
Jan 26 23:44:38 kaoz [10161.703258] active timers:
Jan 26 23:44:38 kaoz [10161.703258] clock 1:
Jan 26 23:44:38 kaoz [10161.703258] .base: ffff880001021468
Jan 26 23:44:38 kaoz [10161.703258] .index: 1
Jan 26 23:44:38 kaoz [10161.703258] .resolution: 1 nsecs
Jan 26 23:44:38 kaoz [10161.703258] .get_time: ktime_get
Jan 26 23:44:38 kaoz [10161.703258] .offset: 0 nsecs
Jan 26 23:44:38 kaoz [10161.703258] active timers:
Jan 26 23:44:38 kaoz [10161.703258] #0: <ffff88007f85d9a8>,
hrtimer_wakeup, S:01
Jan 26 23:44:38 kaoz [10161.703258] # expires at
10161860916333-10161865916331 nsecs [in 159576523 to 164576521 nsecs]
Jan 26 23:44:38 kaoz [10161.703258] #1: <ffff880001021500>,
tick_sched_timer, S:01
Jan 26 23:44:38 kaoz [10161.703258] # expires at
10161878983812-10161878983812 nsecs [in 177644002 to 177644002 nsecs]
Jan 26 23:44:38 kaoz [10161.703258] .expires_next : 10161865916331 nsecs
Jan 26 23:44:38 kaoz [10161.703258] .hres_active : 1
Jan 26 23:44:38 kaoz [10161.703258] .nr_events : 2322638
Jan 26 23:44:38 kaoz [10161.703258] .nohz_mode : 2
Jan 26 23:44:38 kaoz [10161.703258] .idle_tick : 10161683150498 nsecs
Jan 26 23:44:38 kaoz [10161.703258] .tick_stopped : 1
Jan 26 23:44:38 kaoz [10161.703258] .idle_jiffies : 4297925800
Jan 26 23:44:38 kaoz [10161.703258] .idle_calls : 13381452
Jan 26 23:44:38 kaoz [10161.703258] .idle_sleeps : 5700675
Jan 26 23:44:38 kaoz [10161.703258] .idle_entrytime : 10161699594129 nsecs
Jan 26 23:44:38 kaoz [10161.703258] .idle_waketime : 10161699926354 nsecs
Jan 26 23:44:38 kaoz [10161.703258] .idle_exittime : 10161676129546 nsecs
Jan 26 23:44:38 kaoz [10161.703258] .idle_sleeptime : 7638933463214 nsecs
Jan 26 23:44:38 kaoz [10161.703258] .last_jiffies : 4297925806
Jan 26 23:44:38 kaoz [10161.703258] .next_jiffies : 4297925860
Jan 26 23:44:38 kaoz [10161.703258] .idle_expires : 10161878983812 nsecs
Jan 26 23:44:38 kaoz [10161.703258] jiffies: 4297925826
Jan 26 23:44:38 kaoz [10161.703258]
Jan 26 23:44:38 kaoz [10161.703258]
Jan 26 23:44:38 kaoz [10161.703258] Tick Device: mode: 1
Jan 26 23:44:38 kaoz [10161.703258] Broadcast device
Jan 26 23:44:38 kaoz [10161.703258] Clock Event Device: hpet
Jan 26 23:44:38 kaoz [10161.703258] max_delta_ns: 85899346200
Jan 26 23:44:38 kaoz [10161.703258] min_delta_ns: 5000
Jan 26 23:44:38 kaoz [10161.703258] mult: 107374182
Jan 26 23:44:38 kaoz [10161.703258] shift: 32
Jan 26 23:44:38 kaoz [10161.703258] mode: 3
Jan 26 23:44:38 kaoz [10161.703258] next_event: 9223372036854775807
nsecs
Jan 26 23:44:38 kaoz [10161.703258] set_next_event: hpet_legacy_next_event
Jan 26 23:44:38 kaoz [10161.703258] set_mode: hpet_legacy_set_mode
Jan 26 23:44:38 kaoz [10161.703258] event_handler:
tick_handle_oneshot_broadcast
Jan 26 23:44:38 kaoz [10161.703258] tick_broadcast_mask: 00000000
Jan 26 23:44:38 kaoz [10161.703258] tick_broadcast_oneshot_mask: 00000000
Jan 26 23:44:38 kaoz [10161.703258]
Jan 26 23:44:38 kaoz [10161.703258]
Jan 26 23:44:38 kaoz [10161.703258] Tick Device: mode: 1
Jan 26 23:44:38 kaoz [10161.703258] Per CPU device: 0
Jan 26 23:44:38 kaoz [10161.703258] Clock Event Device: lapic
Jan 26 23:44:38 kaoz [10161.703258] max_delta_ns: 671049102
Jan 26 23:44:38 kaoz [10161.703258] min_delta_ns: 1199
Jan 26 23:44:38 kaoz [10161.703258] mult: 53690248
Jan 26 23:44:38 kaoz [10161.703258] shift: 32
Jan 26 23:44:38 kaoz [10161.703258] mode: 3
Jan 26 23:44:38 kaoz [10161.703258] next_event: 10162345650432 nsecs
Jan 26 23:44:38 kaoz [10161.703258] set_next_event: lapic_next_event
Jan 26 23:44:38 kaoz [10161.703258] set_mode: lapic_timer_setup
Jan 26 23:44:38 kaoz [10161.703258] event_handler: hrtimer_interrupt
Jan 26 23:44:38 kaoz [10161.703258]
Jan 26 23:44:38 kaoz [10161.703258] Tick Device: mode: 1
Jan 26 23:44:38 kaoz [10161.703258] Per CPU device: 1
Jan 26 23:44:38 kaoz [10161.703258] Clock Event Device: lapic
Jan 26 23:44:38 kaoz [10161.703258] max_delta_ns: 671049102
Jan 26 23:44:38 kaoz [10161.703258] min_delta_ns: 1199
Jan 26 23:44:38 kaoz [10161.703258] mult: 53690248
Jan 26 23:44:38 kaoz [10161.703258] shift: 32
Jan 26 23:44:38 kaoz [10161.703258] mode: 3
Jan 26 23:44:38 kaoz [10161.703258] next_event: 10161865916331 nsecs
Jan 26 23:44:38 kaoz [10161.703258] set_next_event: lapic_next_event
Jan 26 23:44:38 kaoz [10161.703258] set_mode: lapic_timer_setup
Jan 26 23:44:38 kaoz [10161.703258] event_handler: hrtimer_interrupt
Jan 26 23:44:38 kaoz [10161.703258]
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: sync-Regression in 2.6.28.2?
2009-02-03 19:51 ` Federico Cuello
@ 2009-02-04 6:17 ` Nick Piggin
2009-02-04 17:31 ` Federico Cuello
0 siblings, 1 reply; 14+ messages in thread
From: Nick Piggin @ 2009-02-04 6:17 UTC (permalink / raw)
To: Federico Cuello, Ralf Hildebrandt, Artem Bityutskiy; +Cc: linux-kernel
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="utf-8", Size: 4237 bytes --]
On Wednesday 04 February 2009 06:51:16 Federico Cuello wrote:> Nick Piggin wrote:> > On Wednesday 28 January 2009 08:09:10 Federico Cuello wrote:> >> Ralf Hildebrandt escribió:> >>> I recently installed 2.6.28.2 on our postfix/dovecot-based> >>> mailboxserver. Previously, 2.6.28 and 2.6.28.1 have been running there> >>> without a hitch.> >>>> >>> Now with 2.6.28.2S I had two major lockups: All writes to the users'> >>> Maildirs (on ext4) would stall, the load would rise, "sync" would never> >>> return.> >>>> >>> I had to "reboot -f -n" to get the machine back. All hanging processes> >>> were unkillable, even with kill -9.> >>> [...]> >>> >> The same is happening to me, but I have some logs taken with sysrq.[...]> >> 0 2 99028 46536 26368 1569400 0 0 4 0 1407 387 0> >> 0 0 100> >>> >> Notice the 100% iowait.> >>> >> I also managed to reproduce it doing a rsync from one partition to a USB> >> drive. After the lockup I can't read any file from the source partition,> >> but the other partitions can be accessed normally.> >> > Hm, thanks for reporting, can you guys get a sysrq+W trace when the> > system reaches this state?>> Here it is, if you need something else please ask:
Thanks, could you reply-to-all when replying to retain ccs please?
Common theme is ext4, which uses no_nrwrite_index_update, and I introduceda bug in there which could possibly cause ext4 to go into a loop...
Would it be possible if you can test the following patch?
Thanks,Nick
---
Commit 05fe478dd04e02fa230c305ab9b5616669821dd3 introduced some@wbc->nr_to_write breakage. Here is the change from the commit:
>--- a/mm/page-writeback.c>+++ b/mm/page-writeback.c>@@ -963,8 +963,10 @@ retry:> }> }>> if (--nr_to_write <= 0)> done = 1;> if (wbc->sync_mode == WB_SYNC_NONE) {> if (--wbc->nr_to_write <= 0)> done = 1;> }> if (wbc->nonblocking && bdi_write_congested(bdi)) {> wbc->encountered_congestion = 1;> done = 1> }
It makes the following changes:1. Decrement wbc->nr_to_write instead of nr_to_write2. Decrement wbc->nr_to_write _only_ if wbc->sync_mode == WB_SYNC_NONE3. If synced nr_to_write pages, stop only if if wbc->sync_mode ==WB_SYNC_NONE, otherwise keep going.
However, according to the commit message, the intention was toonly make change 3. Change 1 is a bug. Change 2 does not seem to benecessary, and it breaks UBIFS expectations, so if needed, itshould be done separately later. And change 2 does not seem tobe documented in the commit message.
This patch does the following:1. Undo changes 1 and 22. Add a comment explaining change 3 (it very useful to have comments in_code_, not only in the commit).
Signed-off-by: Artem Bityutskiy <Artem.Bityutskiy@nokia.com>Acked-by: Nick Piggin <npiggin@suse.de>Cc: Andrew Morton <akpm@linux-foundation.org>--- mm/page-writeback.c | 21 +++++++++++++++------ 1 files changed, 15 insertions(+), 6 deletions(-)
diff --git a/mm/page-writeback.c b/mm/page-writeback.cindex b493db7..dc32dae 100644--- a/mm/page-writeback.c+++ b/mm/page-writeback.c@@ -1051,13 +1051,22 @@ continue_unlock: } } - if (wbc->sync_mode == WB_SYNC_NONE) {- wbc->nr_to_write--;- if (wbc->nr_to_write <= 0) {- done = 1;- break;- }+ if (nr_to_write > 0)+ nr_to_write--;+ else if (wbc->sync_mode == WB_SYNC_NONE) {+ /*+ * We stop writing back only if we are not+ * doing integrity sync. In case of integrity+ * sync we have to keep going because someone+ * may be concurrently dirtying pages, and we+ * might have synced a lot of newly appeared+ * dirty pages, but have not synced all of the+ * old dirty pages.+ */+ done = 1;+ break; }+ if (wbc->nonblocking && bdi_write_congested(bdi)) { wbc->encountered_congestion = 1; done = 1;\0ÿôèº{.nÇ+‰·Ÿ®‰†+%ŠËÿ±éݶ\x17¥Šwÿº{.nÇ+‰·¥Š{±þG«éÿŠ{ayº\x1dʇڙë,j\a¢f£¢·hšïêÿ‘êçz_è®\x03(階ŠÝ¢j"ú\x1a¶^[m§ÿÿ¾\a«þG«éÿ¢¸?™¨èÚ&£ø§~á¶iO•æ¬z·švØ^\x14\x04\x1a¶^[m§ÿÿÃ\fÿ¶ìÿ¢¸?–I¥
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: sync-Regression in 2.6.28.2?
2009-02-04 6:17 ` Nick Piggin
@ 2009-02-04 17:31 ` Federico Cuello
2009-02-05 3:25 ` Nick Piggin
2009-02-05 10:19 ` Ralf Hildebrandt
0 siblings, 2 replies; 14+ messages in thread
From: Federico Cuello @ 2009-02-04 17:31 UTC (permalink / raw)
To: Nick Piggin; +Cc: Ralf Hildebrandt, Artem Bityutskiy, linux-kernel
Nick Piggin wrote:
> [...]
> Thanks, could you reply-to-all when replying to retain ccs please?
>
> Common theme is ext4, which uses no_nrwrite_index_update, and I introduced
> a bug in there which could possibly cause ext4 to go into a loop...
>
> Would it be possible if you can test the following patch?
>
I'll test it as soon as I get home.
Meanwhile, I think the new patch may be slightly wrong. If I understand
correctly PageWriteback(page) is called before nr_to_write is tested for
being > 0 and then decremented if true, but "done" is not set to 1
until the next iteration. So another call to PageWriteback(page) while
take place and then "done" will be set to true (if wbc->sync_mode ==
WB_SYNC_NONE).
If nr_to_write == 1 at the beginning of the loop then two pages will be
written.
I think the test condition should something like:
if (--nr_to_write <= 0 && wbc->sync_mode == WB_SYNC_NONE) {
done = 1;
break;
}
.
.
> Signed-off-by: Artem Bityutskiy <Artem.Bityutskiy@nokia.com>
> Acked-by: Nick Piggin <npiggin@suse.de>
> Cc: Andrew Morton <akpm@linux-foundation.org>
> ---
> mm/page-writeback.c | 21 +++++++++++++++------
> 1 files changed, 15 insertions(+), 6 deletions(-)
>
> diff --git a/mm/page-writeback.c b/mm/page-writeback.c
> index b493db7..dc32dae 100644
> --- a/mm/page-writeback.c
> +++ b/mm/page-writeback.c
> @@ -1051,13 +1051,22 @@ continue_unlock:
> }
> }
>
> - if (wbc->sync_mode == WB_SYNC_NONE) {
> - wbc->nr_to_write--;
> - if (wbc->nr_to_write <= 0) {
> - done = 1;
> - break;
> - }
> + if (nr_to_write > 0)
> + nr_to_write--;
> + else if (wbc->sync_mode == WB_SYNC_NONE) {
> + /*
> + * We stop writing back only if we are not
> + * doing integrity sync. In case of integrity
> + * sync we have to keep going because someone
> + * may be concurrently dirtying pages, and we
> + * might have synced a lot of newly appeared
> + * dirty pages, but have not synced all of the
> + * old dirty pages.
> + */
> + done = 1;
> + break;
> }
> +
> if (wbc->nonblocking && bdi_write_congested(bdi)) {
> wbc->encountered_congestion = 1;
> done = 1;
>
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: sync-Regression in 2.6.28.2?
2009-02-04 17:31 ` Federico Cuello
@ 2009-02-05 3:25 ` Nick Piggin
2009-02-05 11:54 ` Federico Cuello
2009-02-05 10:19 ` Ralf Hildebrandt
1 sibling, 1 reply; 14+ messages in thread
From: Nick Piggin @ 2009-02-05 3:25 UTC (permalink / raw)
To: Federico Cuello; +Cc: Ralf Hildebrandt, Artem Bityutskiy, linux-kernel
On Thursday 05 February 2009 04:31:00 Federico Cuello wrote:
> Nick Piggin wrote:
> > [...]
> > Thanks, could you reply-to-all when replying to retain ccs please?
> >
> > Common theme is ext4, which uses no_nrwrite_index_update, and I
> > introduced a bug in there which could possibly cause ext4 to go into a
> > loop...
> >
> > Would it be possible if you can test the following patch?
>
> I'll test it as soon as I get home.
Thanks.
> Meanwhile, I think the new patch may be slightly wrong. If I understand
> correctly PageWriteback(page) is called before nr_to_write is tested for
> being > 0 and then decremented if true, but "done" is not set to 1
> until the next iteration. So another call to PageWriteback(page) while
> take place and then "done" will be set to true (if wbc->sync_mode ==
> WB_SYNC_NONE).
>
> If nr_to_write == 1 at the beginning of the loop then two pages will be
> written.
>
> I think the test condition should something like:
>
> if (--nr_to_write <= 0 && wbc->sync_mode == WB_SYNC_NONE) {
> done = 1;
> break;
> }
I think you're quite right. Good catch. We probably want to prevent
nr_to_write from going -ve, though.
I think something like this
if (nr_to_write > 0)
nr_to_write--;
if (!nr_to_write && wbc->sync_mode == WB_SYNC_NONE) {
...
Would you care to send a patch?
Thanks,
Nick
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: sync-Regression in 2.6.28.2?
2009-02-04 17:31 ` Federico Cuello
2009-02-05 3:25 ` Nick Piggin
@ 2009-02-05 10:19 ` Ralf Hildebrandt
1 sibling, 0 replies; 14+ messages in thread
From: Ralf Hildebrandt @ 2009-02-05 10:19 UTC (permalink / raw)
To: Federico Cuello; +Cc: Nick Piggin, Artem Bityutskiy, linux-kernel
* Federico Cuello <fedux@lugmen.org.ar>:
> Nick Piggin wrote:
> > [...]
> > Thanks, could you reply-to-all when replying to retain ccs please?
> >
> > Common theme is ext4, which uses no_nrwrite_index_update, and I introduced
> > a bug in there which could possibly cause ext4 to go into a loop...
> >
> > Would it be possible if you can test the following patch?
> >
>
> I'll test it as soon as I get home.
>
> Meanwhile, I think the new patch may be slightly wrong. If I understand
> correctly PageWriteback(page) is called before nr_to_write is tested for
> being > 0 and then decremented if true, but "done" is not set to 1
> until the next iteration. So another call to PageWriteback(page) while
> take place and then "done" will be set to true (if wbc->sync_mode ==
> WB_SYNC_NONE).
>
> If nr_to_write == 1 at the beginning of the loop then two pages will be
> written.
>
> I think the test condition should something like:
>
> if (--nr_to_write <= 0 && wbc->sync_mode == WB_SYNC_NONE) {
> done = 1;
> break;
> }
>
> .
> .
>
> > Signed-off-by: Artem Bityutskiy <Artem.Bityutskiy@nokia.com>
> > Acked-by: Nick Piggin <npiggin@suse.de>
> > Cc: Andrew Morton <akpm@linux-foundation.org>
> > ---
> > mm/page-writeback.c | 21 +++++++++++++++------
> > 1 files changed, 15 insertions(+), 6 deletions(-)
> >
> > diff --git a/mm/page-writeback.c b/mm/page-writeback.c
> > index b493db7..dc32dae 100644
> > --- a/mm/page-writeback.c
> > +++ b/mm/page-writeback.c
> > @@ -1051,13 +1051,22 @@ continue_unlock:
> > }
> > }
> >
> > - if (wbc->sync_mode == WB_SYNC_NONE) {
> > - wbc->nr_to_write--;
> > - if (wbc->nr_to_write <= 0) {
> > - done = 1;
> > - break;
> > - }
> > + if (nr_to_write > 0)
> > + nr_to_write--;
> > + else if (wbc->sync_mode == WB_SYNC_NONE) {
> > + /*
> > + * We stop writing back only if we are not
> > + * doing integrity sync. In case of integrity
> > + * sync we have to keep going because someone
> > + * may be concurrently dirtying pages, and we
> > + * might have synced a lot of newly appeared
> > + * dirty pages, but have not synced all of the
> > + * old dirty pages.
> > + */
> > + done = 1;
> > + break;
> > }
> > +
> > if (wbc->nonblocking && bdi_write_congested(bdi)) {
> > wbc->encountered_congestion = 1;
> > done = 1;
> >
>
While I didn't test the patch, I can confirm that the problem is also
in 2.6.28.3
--
Ralf Hildebrandt Ralf.Hildebrandt@charite.de
Charite - Universitätsmedizin Berlin Tel. +49 (0)30-450 570-155
Geschäftsbereich IT | Abt. Netzwerk Fax. +49 (0)30-450 570-962
Hindenburgdamm 30 | 12200 Berlin
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: sync-Regression in 2.6.28.2?
2009-02-05 3:25 ` Nick Piggin
@ 2009-02-05 11:54 ` Federico Cuello
2009-02-09 13:45 ` Nick Piggin
0 siblings, 1 reply; 14+ messages in thread
From: Federico Cuello @ 2009-02-05 11:54 UTC (permalink / raw)
To: Nick Piggin; +Cc: Ralf Hildebrandt, Artem Bityutskiy, linux-kernel
Nick Piggin wrote:
> On Thursday 05 February 2009 04:31:00 Federico Cuello wrote:
>
>> Nick Piggin wrote:
>>
>>> [...]
>>> Thanks, could you reply-to-all when replying to retain ccs please?
>>>
>>> Common theme is ext4, which uses no_nrwrite_index_update, and I
>>> introduced a bug in there which could possibly cause ext4 to go into a
>>> loop...
>>>
>>> Would it be possible if you can test the following patch?
>>>
>> I'll test it as soon as I get home.
>>
>
> Thanks.
>
>
>
>> Meanwhile, I think the new patch may be slightly wrong. If I understand
>> correctly PageWriteback(page) is called before nr_to_write is tested for
>> being > 0 and then decremented if true, but "done" is not set to 1
>> until the next iteration. So another call to PageWriteback(page) while
>> take place and then "done" will be set to true (if wbc->sync_mode ==
>> WB_SYNC_NONE).
>>
>> If nr_to_write == 1 at the beginning of the loop then two pages will be
>> written.
>>
>> I think the test condition should something like:
>>
>> if (--nr_to_write <= 0 && wbc->sync_mode == WB_SYNC_NONE) {
>> done = 1;
>> break;
>> }
>>
>
> I think you're quite right. Good catch. We probably want to prevent
> nr_to_write from going -ve, though.
>
> I think something like this
>
> if (nr_to_write > 0)
> nr_to_write--;
> if (!nr_to_write && wbc->sync_mode == WB_SYNC_NONE) {
> ...
>
> Would you care to send a patch?
>
Ok, after extensive testing I haven't been able to solve the problem.
I'm posting below the patch I used. I tried 3 different patches with one
successful test run with the one you sent me. I don't know if it was
just a coincidence as I had no time to test it again.
Now, with the patch below, it stalls with 50% IO-wait (dual core, one
core at 100%). Perhaps the patch is part of the solution, I don't know.
I also have the sysrq-W logs and I'm also posting them below.
Thanks,
diff --git a/mm/page-writeback.c b/mm/page-writeback.c
index 08d2b96..9e2ae50 100644
--- a/mm/page-writeback.c
+++ b/mm/page-writeback.c
@@ -981,13 +981,23 @@ continue_unlock:
}
}
- if (wbc->sync_mode == WB_SYNC_NONE) {
- wbc->nr_to_write--;
- if (wbc->nr_to_write <= 0) {
+ if (nr_to_write > 0) {
+ nr_to_write--;
+ if (nr_to_write == 0 && wbc->sync_mode
== WB_SYNC_NONE) {
+ /*
+ * We stop writing back only if
we are not
+ * doing integrity sync. In case
of integrity
+ * sync we have to keep going
because someone
+ * may be concurrently dirtying
pages, and we
+ * might have synced a lot of
newly appeared
+ * dirty pages, but have not
synced all of the
+ * old dirty pages.
+ */
done = 1;
break;
}
}
+
if (wbc->nonblocking && bdi_write_congested(bdi)) {
wbc->encountered_congestion = 1;
done = 1;
Feb 5 03:15:19 kaoz [ 1953.633858] SysRq : Show Blocked State
Feb 5 03:15:19 kaoz [ 1953.634882] task PC stack pid father
Feb 5 03:15:19 kaoz [ 1953.634896] kjournald2 D ffff880001020c20 0 4713 2
Feb 5 03:15:19 kaoz [ 1953.634904] ffff88007e4edd50 0000000000000046 ffff8800582e2900 ffff88007e64a840
Feb 5 03:15:19 kaoz [ 1953.634912] ffffffff805de000 ffff88007eb4ec10 ffff88007f862220 ffff88007eb4ee78
Feb 5 03:15:19 kaoz [ 1953.634919] 0000000100000000 0000000100057f3b ffff88007eb4ee78 ffff88007edc0090
Feb 5 03:15:19 kaoz [ 1953.634926] Call Trace:
Feb 5 03:15:19 kaoz [ 1953.634938] [<ffffffff8022e110>] update_curr+0xd0/0x140
Feb 5 03:15:19 kaoz [ 1953.634949] [<ffffffff8031d5e1>] jbd2_journal_commit_transaction+0x171/0x1240
Feb 5 03:15:19 kaoz [ 1953.634957] [<ffffffff8023048f>] dequeue_task_fair+0x3f/0xc0
Feb 5 03:15:19 kaoz [ 1953.634963] [<ffffffff8022e677>] dequeue_task+0x57/0x70
Feb 5 03:15:19 kaoz [ 1953.634972] [<ffffffff80474d27>] thread_return+0x3d/0x5e6
Feb 5 03:15:19 kaoz [ 1953.634979] [<ffffffff80251d20>] autoremove_wake_function+0x0/0x30
Feb 5 03:15:19 kaoz [ 1953.634986] [<ffffffff802461a4>] lock_timer_base+0x34/0x70
Feb 5 03:15:19 kaoz [ 1953.634991] [<ffffffff80246234>] try_to_del_timer_sync+0x54/0x60
Feb 5 03:15:19 kaoz [ 1953.634998] [<ffffffff80322598>] kjournald2+0xd8/0x230
Feb 5 03:15:19 kaoz [ 1953.635004] [<ffffffff80251d20>] autoremove_wake_function+0x0/0x30
Feb 5 03:15:19 kaoz [ 1953.635009] [<ffffffff803224c0>] kjournald2+0x0/0x230
Feb 5 03:15:19 kaoz [ 1953.635014] [<ffffffff80251957>] kthread+0x47/0x80
Feb 5 03:15:19 kaoz [ 1953.635020] [<ffffffff80238697>] schedule_tail+0x27/0x70
Feb 5 03:15:19 kaoz [ 1953.635026] [<ffffffff8020c7e9>] child_rip+0xa/0x11
Feb 5 03:15:19 kaoz [ 1953.635031] [<ffffffff80251910>] kthread+0x0/0x80
Feb 5 03:15:19 kaoz [ 1953.635036] [<ffffffff8020c7df>] child_rip+0x0/0x11
Feb 5 03:15:19 kaoz [ 1953.635068] kdeinit4 D ffff88007c760870 0 6084 1
Feb 5 03:15:19 kaoz [ 1953.635075] ffff88006e83bcf0 0000000000000086 ffffffff8026f77d 0000000000000000
Feb 5 03:15:19 kaoz [ 1953.635082] ffffffff805de000 ffff880070ae4440 ffff880070ae3330 ffff880070ae46a8
Feb 5 03:15:19 kaoz [ 1953.635089] 00000001805e19e0 ffffe20001a79688 ffff880070ae46a8 0000000000000000
Feb 5 03:15:19 kaoz [ 1953.635095] Call Trace:
Feb 5 03:15:19 kaoz [ 1953.635103] [<ffffffff8026f77d>] find_get_page+0x8d/0xc0
Feb 5 03:15:19 kaoz [ 1953.635109] [<ffffffff80475a55>] __mutex_lock_slowpath+0x75/0xc0
Feb 5 03:15:19 kaoz [ 1953.635114] [<ffffffff8047588a>] mutex_lock+0xa/0x10
Feb 5 03:15:19 kaoz [ 1953.635119] [<ffffffff80271b1f>] generic_file_aio_write+0x4f/0xe0
Feb 5 03:15:19 kaoz [ 1953.635126] [<ffffffff802f6848>] ext4_file_write+0x58/0x170
Feb 5 03:15:19 kaoz [ 1953.635134] [<ffffffff8029f66b>] do_sync_write+0xdb/0x120
Feb 5 03:15:19 kaoz [ 1953.635142] [<ffffffff80227210>] do_page_fault+0x300/0x970
Feb 5 03:15:19 kaoz [ 1953.635148] [<ffffffff80251d20>] autoremove_wake_function+0x0/0x30
Feb 5 03:15:19 kaoz [ 1953.635155] [<ffffffff802a008b>] vfs_write+0xcb/0x170
Feb 5 03:15:19 kaoz [ 1953.635161] [<ffffffff802a0233>] sys_write+0x53/0xa0
Feb 5 03:15:19 kaoz [ 1953.635166] [<ffffffff8020b8cb>] system_call_fastpath+0x16/0x1b
Feb 5 03:15:19 kaoz [ 1953.635173] quasselcore D ffff880001006b08 0 6204 1
Feb 5 03:15:19 kaoz [ 1953.635179] ffff88006ae09bb8 0000000000000086 ffff88007e91d800 ffffffff80496fd0
Feb 5 03:15:19 kaoz [ 1953.635186] ffffffff805de000 ffff88006ad685b0 ffff880055a605b0 ffff88006ad68818
Feb 5 03:15:19 kaoz [ 1953.635192] 0000000100000004 0000000000000000 ffff88006ad68818 ffffffff802fa711
Feb 5 03:15:19 kaoz [ 1953.635199] Call Trace:
Feb 5 03:15:19 kaoz [ 1953.635205] [<ffffffff802fa711>] ext4_da_write_end+0x141/0x280
Feb 5 03:15:19 kaoz [ 1953.635259] [<ffffffffa008c6cc>] dm_table_unplug_all+0x3c/0x90 [dm_mod]
Feb 5 03:15:19 kaoz [ 1953.635266] [<ffffffff8026f910>] sync_page+0x0/0x70
Feb 5 03:15:19 kaoz [ 1953.635273] [<ffffffff80475307>] io_schedule+0x37/0x50
Feb 5 03:15:19 kaoz [ 1953.635279] [<ffffffff8026f945>] sync_page+0x35/0x70
Feb 5 03:15:19 kaoz [ 1953.635283] [<ffffffff8047568a>] __wait_on_bit_lock+0x4a/0x80
Feb 5 03:15:19 kaoz [ 1953.635289] [<ffffffff8026f8ee>] __lock_page+0x5e/0x70
Feb 5 03:15:19 kaoz [ 1953.635295] [<ffffffff80251d50>] wake_bit_function+0x0/0x30
Feb 5 03:15:19 kaoz [ 1953.635302] [<ffffffff80276cdd>] write_cache_pages+0x1ed/0x3d0
Feb 5 03:15:19 kaoz [ 1953.635308] [<ffffffff802fbf10>] __mpage_da_writepage+0x0/0x170
Feb 5 03:15:19 kaoz [ 1953.635315] [<ffffffff802fbcc7>] ext4_da_writepages+0x237/0x3b0
Feb 5 03:15:19 kaoz [ 1953.635321] [<ffffffff80251d20>] autoremove_wake_function+0x0/0x30
Feb 5 03:15:19 kaoz [ 1953.635328] [<ffffffff802fd240>] ext4_da_get_block_write+0x0/0x160
Feb 5 03:15:19 kaoz [ 1953.635333] [<ffffffff80276f10>] do_writepages+0x20/0x40
Feb 5 03:15:19 kaoz [ 1953.635338] [<ffffffff8027074d>] __filemap_fdatawrite_range+0x4d/0x60
Feb 5 03:15:19 kaoz [ 1953.635345] [<ffffffff802c12b5>] do_fsync+0x45/0xe0
Feb 5 03:15:19 kaoz [ 1953.635350] [<ffffffff802c137e>] __do_fsync+0x2e/0x50
Feb 5 03:15:19 kaoz [ 1953.635355] [<ffffffff8020b8cb>] system_call_fastpath+0x16/0x1b
Feb 5 03:15:19 kaoz [ 1953.635361] kwin D ffff88007c760870 0 6307 6283
Feb 5 03:15:19 kaoz [ 1953.635366] ffff88006af0fcf0 0000000000000082 ffffffff8026f77d 0000000000000000
Feb 5 03:15:19 kaoz [ 1953.635373] ffffffff805de000 ffff88007e7ea7d0 ffff8800709d96c0 ffff88007e7eaa38
Feb 5 03:15:19 kaoz [ 1953.635380] 00000000805e19e0 ffffe20000664a20 ffff88007e7eaa38 ffff88006af0fcc8
Feb 5 03:15:19 kaoz [ 1953.635386] Call Trace:
Feb 5 03:15:19 kaoz [ 1953.635392] [<ffffffff8026f77d>] find_get_page+0x8d/0xc0
Feb 5 03:15:19 kaoz [ 1953.635398] [<ffffffff8022f753>] __wake_up+0x43/0x70
Feb 5 03:15:19 kaoz [ 1953.635403] [<ffffffff80475a55>] __mutex_lock_slowpath+0x75/0xc0
Feb 5 03:15:19 kaoz [ 1953.635408] [<ffffffff8047588a>] mutex_lock+0xa/0x10
Feb 5 03:15:19 kaoz [ 1953.635413] [<ffffffff80271b1f>] generic_file_aio_write+0x4f/0xe0
Feb 5 03:15:19 kaoz [ 1953.635419] [<ffffffff802f6848>] ext4_file_write+0x58/0x170
Feb 5 03:15:19 kaoz [ 1953.635425] [<ffffffff8029f66b>] do_sync_write+0xdb/0x120
Feb 5 03:15:19 kaoz [ 1953.635431] [<ffffffff80227210>] do_page_fault+0x300/0x970
Feb 5 03:15:19 kaoz [ 1953.635437] [<ffffffff80251d20>] autoremove_wake_function+0x0/0x30
Feb 5 03:15:19 kaoz [ 1953.635443] [<ffffffff803fc4d6>] sock_ioctl+0x56/0x260
Feb 5 03:15:19 kaoz [ 1953.635450] [<ffffffff802ad76f>] vfs_ioctl+0x2f/0xa0
Feb 5 03:15:19 kaoz [ 1953.635456] [<ffffffff802a008b>] vfs_write+0xcb/0x170
Feb 5 03:15:19 kaoz [ 1953.635462] [<ffffffff802a0233>] sys_write+0x53/0xa0
Feb 5 03:15:19 kaoz [ 1953.635467] [<ffffffff8020b8cb>] system_call_fastpath+0x16/0x1b
Feb 5 03:15:19 kaoz [ 1953.635479] mysqld D 0000000000000002 0 6688 6674
Feb 5 03:15:19 kaoz [ 1953.635485] ffff88005c58db58 0000000000000086 ffff88005c58dad8 ffffffff80230285
Feb 5 03:15:19 kaoz [ 1953.635491] ffffffff805de000 ffff88007f95ec10 ffff8800636ca7d0 ffff88007f95ee78
Feb 5 03:15:19 kaoz [ 1953.635498] 0000000100000001 ffff880070bdd840 ffff88007f95ee78 ffffffff8023384b
Feb 5 03:15:19 kaoz [ 1953.635504] Call Trace:
Feb 5 03:15:19 kaoz [ 1953.635510] [<ffffffff80230285>] enqueue_entity+0xf5/0x160
Feb 5 03:15:19 kaoz [ 1953.635517] [<ffffffff8023384b>] check_preempt_wakeup+0x15b/0x180
Feb 5 03:15:19 kaoz [ 1953.635522] [<ffffffff802356ae>] try_to_wake_up+0xee/0x1a0
Feb 5 03:15:19 kaoz [ 1953.635528] [<ffffffff8031cf3b>] start_this_handle+0x33b/0x450
Feb 5 03:15:19 kaoz [ 1953.635534] [<ffffffff80251d20>] autoremove_wake_function+0x0/0x30
Feb 5 03:15:19 kaoz [ 1953.635541] [<ffffffff8031d226>] jbd2_journal_start+0x96/0xe0
Feb 5 03:15:19 kaoz [ 1953.635547] [<ffffffff802f9c36>] ext4_dirty_inode+0x36/0xb0
Feb 5 03:15:19 kaoz [ 1953.635552] [<ffffffff802be284>] __mark_inode_dirty+0x34/0x1d0
Feb 5 03:15:19 kaoz [ 1953.635558] [<ffffffff802b3380>] file_update_time+0x110/0x130
Feb 5 03:15:19 kaoz [ 1953.635564] [<ffffffff80270fa6>] __generic_file_aio_write_nolock+0x246/0x470
Feb 5 03:15:19 kaoz [ 1953.635569] [<ffffffff80271b34>] generic_file_aio_write+0x64/0xe0
Feb 5 03:15:19 kaoz [ 1953.635575] [<ffffffff802f6848>] ext4_file_write+0x58/0x170
Feb 5 03:15:19 kaoz [ 1953.635582] [<ffffffff8029f66b>] do_sync_write+0xdb/0x120
Feb 5 03:15:19 kaoz [ 1953.635588] [<ffffffff80227210>] do_page_fault+0x300/0x970
Feb 5 03:15:19 kaoz [ 1953.635594] [<ffffffff80251d20>] autoremove_wake_function+0x0/0x30
Feb 5 03:15:19 kaoz [ 1953.635600] [<ffffffff802a008b>] vfs_write+0xcb/0x170
Feb 5 03:15:19 kaoz [ 1953.635606] [<ffffffff802a0233>] sys_write+0x53/0xa0
Feb 5 03:15:19 kaoz [ 1953.635611] [<ffffffff8020b8cb>] system_call_fastpath+0x16/0x1b
Feb 5 03:15:19 kaoz [ 1953.635616] konsole D ffff88007c760870 0 6722 1
Feb 5 03:15:19 kaoz [ 1953.635622] ffff88006ae49cf0 0000000000000082 ffffffff8026f77d 0000000000000000
Feb 5 03:15:19 kaoz [ 1953.635628] ffffffff805de000 ffff88005c594fa0 ffff88007c0b2220 ffff88005c595208
Feb 5 03:15:19 kaoz [ 1953.635635] 00000001805e19e0 ffffe20001a79688 ffff88005c595208 0000000000000000
Feb 5 03:15:19 kaoz [ 1953.635641] Call Trace:
Feb 5 03:15:19 kaoz [ 1953.635647] [<ffffffff8026f77d>] find_get_page+0x8d/0xc0
Feb 5 03:15:19 kaoz [ 1953.635652] [<ffffffff80475a55>] __mutex_lock_slowpath+0x75/0xc0
Feb 5 03:15:19 kaoz [ 1953.635657] [<ffffffff8047588a>] mutex_lock+0xa/0x10
Feb 5 03:15:19 kaoz [ 1953.635662] [<ffffffff80271b1f>] generic_file_aio_write+0x4f/0xe0
Feb 5 03:15:19 kaoz [ 1953.635668] [<ffffffff802f6848>] ext4_file_write+0x58/0x170
Feb 5 03:15:19 kaoz [ 1953.635674] [<ffffffff8029f66b>] do_sync_write+0xdb/0x120
Feb 5 03:15:19 kaoz [ 1953.635680] [<ffffffff80227210>] do_page_fault+0x300/0x970
Feb 5 03:15:19 kaoz [ 1953.635686] [<ffffffff80251d20>] autoremove_wake_function+0x0/0x30
Feb 5 03:15:19 kaoz [ 1953.635692] [<ffffffff802ad76f>] vfs_ioctl+0x2f/0xa0
Feb 5 03:15:19 kaoz [ 1953.635698] [<ffffffff802a008b>] vfs_write+0xcb/0x170
Feb 5 03:15:19 kaoz [ 1953.635704] [<ffffffff802a0233>] sys_write+0x53/0xa0
Feb 5 03:15:19 kaoz [ 1953.635709] [<ffffffff8020b8cb>] system_call_fastpath+0x16/0x1b
Feb 5 03:15:19 kaoz [ 1953.635715] pdflush D ffff880001003178 0 7292 2
Feb 5 03:15:19 kaoz [ 1953.635720] ffff88007c09ba20 0000000000000046 ffff880024e2b870 ffff88007ebd32d0
Feb 5 03:15:19 kaoz [ 1953.635727] ffffffff805de000 ffff880055a638e0 ffff88007ea7c9f0 ffff880055a63b48
Feb 5 03:15:19 kaoz [ 1953.635734] 000000017c09b9c0 ffff880024e2b870 ffff880055a63b48 ffff88007edd1d80
Feb 5 03:15:19 kaoz [ 1953.635740] Call Trace:
Feb 5 03:15:19 kaoz [ 1953.635746] [<ffffffff8032f437>] generic_make_request+0x207/0x340
Feb 5 03:15:19 kaoz [ 1953.635767] [<ffffffffa008c6cc>] dm_table_unplug_all+0x3c/0x90 [dm_mod]
Feb 5 03:15:19 kaoz [ 1953.635775] [<ffffffff80341b27>] radix_tree_gang_lookup_slot+0x77/0xc0
Feb 5 03:15:19 kaoz [ 1953.635781] [<ffffffff8026f910>] sync_page+0x0/0x70
Feb 5 03:15:19 kaoz [ 1953.635788] [<ffffffff80475307>] io_schedule+0x37/0x50
Feb 5 03:15:19 kaoz [ 1953.635794] [<ffffffff8026f945>] sync_page+0x35/0x70
Feb 5 03:15:19 kaoz [ 1953.635798] [<ffffffff8047568a>] __wait_on_bit_lock+0x4a/0x80
Feb 5 03:15:19 kaoz [ 1953.635805] [<ffffffff8026f8ee>] __lock_page+0x5e/0x70
Feb 5 03:15:19 kaoz [ 1953.635810] [<ffffffff80251d50>] wake_bit_function+0x0/0x30
Feb 5 03:15:19 kaoz [ 1953.635816] [<ffffffff8027867a>] pagevec_lookup_tag+0x1a/0x30
Feb 5 03:15:19 kaoz [ 1953.635822] [<ffffffff80276cdd>] write_cache_pages+0x1ed/0x3d0
Feb 5 03:15:19 kaoz [ 1953.635828] [<ffffffff802fbf10>] __mpage_da_writepage+0x0/0x170
Feb 5 03:15:19 kaoz [ 1953.635835] [<ffffffff8031bbdf>] jbd2_journal_stop+0x17f/0x200
Feb 5 03:15:19 kaoz [ 1953.635842] [<ffffffff802fbcc7>] ext4_da_writepages+0x237/0x3b0
Feb 5 03:15:19 kaoz [ 1953.635848] [<ffffffff80476f6c>] _spin_unlock_irq+0xc/0x30
Feb 5 03:15:19 kaoz [ 1953.635853] [<ffffffff80236c8b>] finish_task_switch+0x2b/0xe0
Feb 5 03:15:19 kaoz [ 1953.635860] [<ffffffff802fd240>] ext4_da_get_block_write+0x0/0x160
Feb 5 03:15:19 kaoz [ 1953.635865] [<ffffffff80276f10>] do_writepages+0x20/0x40
Feb 5 03:15:19 kaoz [ 1953.635871] [<ffffffff802bd516>] __writeback_single_inode+0xa6/0x470
Feb 5 03:15:19 kaoz [ 1953.635876] [<ffffffff802bdea0>] generic_sync_sb_inodes+0x370/0x4d0
Feb 5 03:15:19 kaoz [ 1953.635882] [<ffffffff802be1bd>] writeback_inodes+0x4d/0xe0
Feb 5 03:15:19 kaoz [ 1953.635887] [<ffffffff8027759d>] background_writeout+0xad/0xe0
Feb 5 03:15:19 kaoz [ 1953.635893] [<ffffffff80277ccb>] pdflush+0x15b/0x240
Feb 5 03:15:19 kaoz [ 1953.635898] [<ffffffff802774f0>] background_writeout+0x0/0xe0
Feb 5 03:15:19 kaoz [ 1953.635904] [<ffffffff80277b70>] pdflush+0x0/0x240
Feb 5 03:15:19 kaoz [ 1953.635909] [<ffffffff80251957>] kthread+0x47/0x80
Feb 5 03:15:19 kaoz [ 1953.635914] [<ffffffff80238697>] schedule_tail+0x27/0x70
Feb 5 03:15:19 kaoz [ 1953.635919] [<ffffffff8020c7e9>] child_rip+0xa/0x11
Feb 5 03:15:19 kaoz [ 1953.635924] [<ffffffff80251910>] kthread+0x0/0x80
Feb 5 03:15:19 kaoz [ 1953.635929] [<ffffffff8020c7df>] child_rip+0x0/0x11
Feb 5 03:15:19 kaoz [ 1953.635932] krunner_lock D ffff88007c760870 0 7306 6670
Feb 5 03:15:19 kaoz [ 1953.635938] ffff880063e77cf0 0000000000000082 ffffffff8026f77d 0000000000000000
Feb 5 03:15:19 kaoz [ 1953.635945] ffffffff805de000 ffff88000395b330 ffff880055a63e90 ffff88000395b598
Feb 5 03:15:19 kaoz [ 1953.635952] 00000001805e19e0 ffffe200000b0d88 ffff88000395b598 0000000000000086
Feb 5 03:15:19 kaoz [ 1953.635958] Call Trace:
Feb 5 03:15:19 kaoz [ 1953.635964] [<ffffffff8026f77d>] find_get_page+0x8d/0xc0
Feb 5 03:15:19 kaoz [ 1953.635969] [<ffffffff80475a55>] __mutex_lock_slowpath+0x75/0xc0
Feb 5 03:15:19 kaoz [ 1953.635974] [<ffffffff8047588a>] mutex_lock+0xa/0x10
Feb 5 03:15:19 kaoz [ 1953.635979] [<ffffffff80271b1f>] generic_file_aio_write+0x4f/0xe0
Feb 5 03:15:19 kaoz [ 1953.635985] [<ffffffff802f6848>] ext4_file_write+0x58/0x170
Feb 5 03:15:19 kaoz [ 1953.635991] [<ffffffff8029f66b>] do_sync_write+0xdb/0x120
Feb 5 03:15:19 kaoz [ 1953.635997] [<ffffffff80227210>] do_page_fault+0x300/0x970
Feb 5 03:15:19 kaoz [ 1953.636003] [<ffffffff80251d20>] autoremove_wake_function+0x0/0x30
Feb 5 03:15:19 kaoz [ 1953.636010] [<ffffffff802493af>] kill_pid_info+0x4f/0x80
Feb 5 03:15:19 kaoz [ 1953.636016] [<ffffffff802a008b>] vfs_write+0xcb/0x170
Feb 5 03:15:19 kaoz [ 1953.636022] [<ffffffff802a0233>] sys_write+0x53/0xa0
Feb 5 03:15:19 kaoz [ 1953.636027] [<ffffffff8020b8cb>] system_call_fastpath+0x16/0x1b
Feb 5 03:15:19 kaoz [ 1953.636031] krunner_lock D ffff88007c760870 0 7307 7306
Feb 5 03:15:19 kaoz [ 1953.636037] ffff88006364bcf0 0000000000000082 ffffe200000b0d88 ffffffff805e19e0
Feb 5 03:15:19 kaoz [ 1953.636044] ffffffff805de000 ffff880055a63e90 ffff88007c0b2220 ffff880055a640f8
Feb 5 03:15:19 kaoz [ 1953.636050] 00000001802791f5 ffffe200000b0d88 ffff880055a640f8 0000000000000000
Feb 5 03:15:19 kaoz [ 1953.636057] Call Trace:
Feb 5 03:15:19 kaoz [ 1953.636062] [<ffffffff80475a55>] __mutex_lock_slowpath+0x75/0xc0
Feb 5 03:15:19 kaoz [ 1953.636066] [<ffffffff8047588a>] mutex_lock+0xa/0x10
Feb 5 03:15:19 kaoz [ 1953.636071] [<ffffffff80271b1f>] generic_file_aio_write+0x4f/0xe0
Feb 5 03:15:19 kaoz [ 1953.636077] [<ffffffff802f6848>] ext4_file_write+0x58/0x170
Feb 5 03:15:19 kaoz [ 1953.636084] [<ffffffff8029f66b>] do_sync_write+0xdb/0x120
Feb 5 03:15:19 kaoz [ 1953.636090] [<ffffffff80227210>] do_page_fault+0x300/0x970
Feb 5 03:15:19 kaoz [ 1953.636095] [<ffffffff80251d20>] autoremove_wake_function+0x0/0x30
Feb 5 03:15:19 kaoz [ 1953.636101] [<ffffffff802493af>] kill_pid_info+0x4f/0x80
Feb 5 03:15:19 kaoz [ 1953.636108] [<ffffffff802a008b>] vfs_write+0xcb/0x170
Feb 5 03:15:19 kaoz [ 1953.636113] [<ffffffff802a0233>] sys_write+0x53/0xa0
Feb 5 03:15:19 kaoz [ 1953.636119] [<ffffffff8020b8cb>] system_call_fastpath+0x16/0x1b
Feb 5 03:15:19 kaoz [ 1953.636122] kblankscrn.ks D ffff880001020c20 0 7309 7306
Feb 5 03:15:19 kaoz [ 1953.636128] ffff8800635e3cf0 0000000000000086 ffffffff8026f77d 0000000000000000
Feb 5 03:15:19 kaoz [ 1953.636135] ffffffff805de000 ffff8800039585b0 ffff88007f862220 ffff880003958818
Feb 5 03:15:19 kaoz [ 1953.636141] 00000001805e19e0 000000010005efb2 ffff880003958818 ffffffff8056ffc8
Feb 5 03:15:19 kaoz [ 1953.636148] Call Trace:
Feb 5 03:15:19 kaoz [ 1953.636154] [<ffffffff8026f77d>] find_get_page+0x8d/0xc0
Feb 5 03:15:19 kaoz [ 1953.636159] [<ffffffff80475a55>] __mutex_lock_slowpath+0x75/0xc0
Feb 5 03:15:19 kaoz [ 1953.636164] [<ffffffff8047588a>] mutex_lock+0xa/0x10
Feb 5 03:15:19 kaoz [ 1953.636169] [<ffffffff80271b1f>] generic_file_aio_write+0x4f/0xe0
Feb 5 03:15:19 kaoz [ 1953.636175] [<ffffffff802f6848>] ext4_file_write+0x58/0x170
Feb 5 03:15:19 kaoz [ 1953.636181] [<ffffffff8029f66b>] do_sync_write+0xdb/0x120
Feb 5 03:15:19 kaoz [ 1953.636187] [<ffffffff80227210>] do_page_fault+0x300/0x970
Feb 5 03:15:19 kaoz [ 1953.636193] [<ffffffff80251d20>] autoremove_wake_function+0x0/0x30
Feb 5 03:15:19 kaoz [ 1953.636200] [<ffffffff802866e6>] unmap_region+0x106/0x1a0
Feb 5 03:15:19 kaoz [ 1953.636206] [<ffffffff802a008b>] vfs_write+0xcb/0x170
Feb 5 03:15:19 kaoz [ 1953.636212] [<ffffffff802a0233>] sys_write+0x53/0xa0
Feb 5 03:15:19 kaoz [ 1953.636217] [<ffffffff8020b8cb>] system_call_fastpath+0x16/0x1b
Feb 5 03:15:19 kaoz [ 1953.636221] kblankscrn.ks D 0000000000000002 0 7311 7307
Feb 5 03:15:19 kaoz [ 1953.636227] ffff880042a2bb58 0000000000000086 ffff880024edb150 0000000000b29b6f
Feb 5 03:15:19 kaoz [ 1953.636233] ffffffff805de000 ffff88000395b8e0 ffff8800039585b0 ffff88000395bb48
Feb 5 03:15:19 kaoz [ 1953.636240] 000000010395b8e0 000300018022f9c8 ffff88000395bb48 ffff880070bdd900
Feb 5 03:15:19 kaoz [ 1953.636246] Call Trace:
Feb 5 03:15:19 kaoz [ 1953.636252] [<ffffffff8022f928>] set_next_entity+0x38/0x80
Feb 5 03:15:19 kaoz [ 1953.636259] [<ffffffff8031cf3b>] start_this_handle+0x33b/0x450
Feb 5 03:15:19 kaoz [ 1953.636265] [<ffffffff80251d20>] autoremove_wake_function+0x0/0x30
Feb 5 03:15:19 kaoz [ 1953.636271] [<ffffffff8031d226>] jbd2_journal_start+0x96/0xe0
Feb 5 03:15:19 kaoz [ 1953.636277] [<ffffffff802f9c36>] ext4_dirty_inode+0x36/0xb0
Feb 5 03:15:19 kaoz [ 1953.636282] [<ffffffff802be284>] __mark_inode_dirty+0x34/0x1d0
Feb 5 03:15:19 kaoz [ 1953.636288] [<ffffffff802b3380>] file_update_time+0x110/0x130
Feb 5 03:15:19 kaoz [ 1953.636293] [<ffffffff80270fa6>] __generic_file_aio_write_nolock+0x246/0x470
Feb 5 03:15:19 kaoz [ 1953.636299] [<ffffffff80280332>] __do_fault+0x202/0x4c0
Feb 5 03:15:19 kaoz [ 1953.636304] [<ffffffff80271b34>] generic_file_aio_write+0x64/0xe0
Feb 5 03:15:19 kaoz [ 1953.636310] [<ffffffff802f6848>] ext4_file_write+0x58/0x170
Feb 5 03:15:19 kaoz [ 1953.636316] [<ffffffff8029f66b>] do_sync_write+0xdb/0x120
Feb 5 03:15:19 kaoz [ 1953.636322] [<ffffffff80227210>] do_page_fault+0x300/0x970
Feb 5 03:15:19 kaoz [ 1953.636328] [<ffffffff80251d20>] autoremove_wake_function+0x0/0x30
Feb 5 03:15:19 kaoz [ 1953.636333] [<ffffffff802866e6>] unmap_region+0x106/0x1a0
Feb 5 03:15:19 kaoz [ 1953.636340] [<ffffffff802a008b>] vfs_write+0xcb/0x170
Feb 5 03:15:19 kaoz [ 1953.636346] [<ffffffff802a0233>] sys_write+0x53/0xa0
Feb 5 03:15:19 kaoz [ 1953.636351] [<ffffffff8020b8cb>] system_call_fastpath+0x16/0x1b
Feb 5 03:15:19 kaoz [ 1953.636355] updatedb D ffff880001020c20 0 7511 7506
Feb 5 03:15:19 kaoz [ 1953.636361] ffff8800372fdbf8 0000000000000086 ffff88007ea30000 0000000000000000
Feb 5 03:15:19 kaoz [ 1953.636367] ffffffff805de000 ffff88000395e660 ffff88007f862220 ffff88000395e8c8
Feb 5 03:15:19 kaoz [ 1953.636374] 000000017fa93a50 00000001000791d8 ffff88000395e8c8 ffff88007eb544f0
Feb 5 03:15:19 kaoz [ 1953.636381] Call Trace:
Feb 5 03:15:19 kaoz [ 1953.636386] [<ffffffff802c3880>] sync_buffer+0x0/0x50
Feb 5 03:15:19 kaoz [ 1953.636392] [<ffffffff80330564>] generic_unplug_device+0x24/0x40
Feb 5 03:15:19 kaoz [ 1953.636412] [<ffffffffa008c6cc>] dm_table_unplug_all+0x3c/0x90 [dm_mod]
Feb 5 03:15:19 kaoz [ 1953.636418] [<ffffffff802c3880>] sync_buffer+0x0/0x50
Feb 5 03:15:19 kaoz [ 1953.636424] [<ffffffff80475307>] io_schedule+0x37/0x50
Feb 5 03:15:19 kaoz [ 1953.636429] [<ffffffff802c38bd>] sync_buffer+0x3d/0x50
Feb 5 03:15:19 kaoz [ 1953.636433] [<ffffffff804757b0>] __wait_on_bit+0x50/0x80
Feb 5 03:15:19 kaoz [ 1953.636439] [<ffffffff802c3880>] sync_buffer+0x0/0x50
Feb 5 03:15:19 kaoz [ 1953.636444] [<ffffffff80475859>] out_of_line_wait_on_bit+0x79/0xa0
Feb 5 03:15:19 kaoz [ 1953.636449] [<ffffffff80251d50>] wake_bit_function+0x0/0x30
Feb 5 03:15:19 kaoz [ 1953.636455] [<ffffffff802fd5ba>] ext4_bread+0x5a/0x80
Feb 5 03:15:19 kaoz [ 1953.636461] [<ffffffff803032b3>] htree_dirblock_to_tree+0x33/0x1b0
Feb 5 03:15:19 kaoz [ 1953.636466] [<ffffffff803035a2>] ext4_htree_fill_tree+0x172/0x210
Feb 5 03:15:19 kaoz [ 1953.636472] [<ffffffff802aaf4e>] do_path_lookup+0x9e/0x1b0
Feb 5 03:15:19 kaoz [ 1953.636477] [<ffffffff802a0805>] file_move+0x25/0x70
Feb 5 03:15:19 kaoz [ 1953.636483] [<ffffffff802f6492>] ext4_readdir+0x4a2/0x600
Feb 5 03:15:19 kaoz [ 1953.636489] [<ffffffff802ae0b0>] filldir+0x0/0xe0
Feb 5 03:15:19 kaoz [ 1953.636494] [<ffffffff8029829f>] add_partial+0x1f/0x80
Feb 5 03:15:19 kaoz [ 1953.636500] [<ffffffff802a34c4>] cp_new_stat+0xe4/0x100
Feb 5 03:15:19 kaoz [ 1953.636506] [<ffffffff802ae0b0>] filldir+0x0/0xe0
Feb 5 03:15:19 kaoz [ 1953.636511] [<ffffffff802ae320>] vfs_readdir+0xb0/0xd0
Feb 5 03:15:19 kaoz [ 1953.636518] [<ffffffff802ae4b4>] sys_getdents+0x84/0xe0
Feb 5 03:15:19 kaoz [ 1953.636524] [<ffffffff8029e6ca>] sys_fchdir+0x4a/0x90
Feb 5 03:15:19 kaoz [ 1953.636529] [<ffffffff8020b8cb>] system_call_fastpath+0x16/0x1b
^ permalink raw reply related [flat|nested] 14+ messages in thread
* Re: sync-Regression in 2.6.28.2?
2009-02-05 11:54 ` Federico Cuello
@ 2009-02-09 13:45 ` Nick Piggin
2009-02-09 13:49 ` Ralf Hildebrandt
0 siblings, 1 reply; 14+ messages in thread
From: Nick Piggin @ 2009-02-09 13:45 UTC (permalink / raw)
To: Federico Cuello; +Cc: Ralf Hildebrandt, Artem Bityutskiy, linux-kernel
On Thursday 05 February 2009 22:54:16 Federico Cuello wrote:
> Nick Piggin wrote:
> > On Thursday 05 February 2009 04:31:00 Federico Cuello wrote:
> >> Nick Piggin wrote:
> >>> [...]
> >>> Thanks, could you reply-to-all when replying to retain ccs please?
> >>>
> >>> Common theme is ext4, which uses no_nrwrite_index_update, and I
> >>> introduced a bug in there which could possibly cause ext4 to go into a
> >>> loop...
> >>>
> >>> Would it be possible if you can test the following patch?
> >>
> >> I'll test it as soon as I get home.
> >
> > Thanks.
> >
> >> Meanwhile, I think the new patch may be slightly wrong. If I understand
> >> correctly PageWriteback(page) is called before nr_to_write is tested for
> >> being > 0 and then decremented if true, but "done" is not set to 1
> >> until the next iteration. So another call to PageWriteback(page) while
> >> take place and then "done" will be set to true (if wbc->sync_mode ==
> >> WB_SYNC_NONE).
> >>
> >> If nr_to_write == 1 at the beginning of the loop then two pages will be
> >> written.
> >>
> >> I think the test condition should something like:
> >>
> >> if (--nr_to_write <= 0 && wbc->sync_mode == WB_SYNC_NONE) {
> >> done = 1;
> >> break;
> >> }
> >
> > I think you're quite right. Good catch. We probably want to prevent
> > nr_to_write from going -ve, though.
> >
> > I think something like this
> >
> > if (nr_to_write > 0)
> > nr_to_write--;
> > if (!nr_to_write && wbc->sync_mode == WB_SYNC_NONE) {
> > ...
> >
> > Would you care to send a patch?
>
> Ok, after extensive testing I haven't been able to solve the problem.
>
> I'm posting below the patch I used. I tried 3 different patches with one
> successful test run with the one you sent me. I don't know if it was
> just a coincidence as I had no time to test it again.
>
> Now, with the patch below, it stalls with 50% IO-wait (dual core, one
> core at 100%). Perhaps the patch is part of the solution, I don't know.
>
> I also have the sysrq-W logs and I'm also posting them below.
>
> Thanks,
>
>
>
> diff --git a/mm/page-writeback.c b/mm/page-writeback.c
> index 08d2b96..9e2ae50 100644
> --- a/mm/page-writeback.c
> +++ b/mm/page-writeback.c
> @@ -981,13 +981,23 @@ continue_unlock:
> }
> }
>
> - if (wbc->sync_mode == WB_SYNC_NONE) {
> - wbc->nr_to_write--;
> - if (wbc->nr_to_write <= 0) {
> + if (nr_to_write > 0) {
> + nr_to_write--;
> + if (nr_to_write == 0 && wbc->sync_mode
> == WB_SYNC_NONE) {
> + /*
> + * We stop writing back only if
> we are not
> + * doing integrity sync. In case
> of integrity
> + * sync we have to keep going
> because someone
> + * may be concurrently dirtying
> pages, and we
> + * might have synced a lot of
> newly appeared
> + * dirty pages, but have not
> synced all of the
> + * old dirty pages.
> + */
> done = 1;
> break;
> }
> }
> +
> if (wbc->nonblocking && bdi_write_congested(bdi)) {
> wbc->encountered_congestion = 1;
> done = 1;
This patch seems good to me. If you would care to add a changelog and
Signed-off-by: line, then we could get it merged?
I am not too sure about this bug. I have reproduced a strange hang with
ext4 (which does include sys_sync and write_cache_pages traces), and
also turned up a lockdep report. Also, we haven't seen any reports of
this problem on other filesystems. So it could be an ext4 bug.
Your traces also have lots of tasks hung waiting for page lock. It is
possible that wakeups get lost, which is fixed by this commit in
mainline
777c6c5f1f6e757ae49ecca2ed72d6b1f523c007
Which might also be your bug.
Any chance you can test this patch (as well as the existing patches
you are using to fix write_cache_pages?).
Thanks,
Nick
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: sync-Regression in 2.6.28.2?
2009-02-09 13:45 ` Nick Piggin
@ 2009-02-09 13:49 ` Ralf Hildebrandt
2009-02-15 13:42 ` Ralf Hildebrandt
0 siblings, 1 reply; 14+ messages in thread
From: Ralf Hildebrandt @ 2009-02-09 13:49 UTC (permalink / raw)
To: Nick Piggin; +Cc: Federico Cuello, Artem Bityutskiy, linux-kernel
> This patch seems good to me. If you would care to add a changelog and
> Signed-off-by: line, then we could get it merged?
>
> I am not too sure about this bug. I have reproduced a strange hang with
> ext4 (which does include sys_sync and write_cache_pages traces), and
> also turned up a lockdep report. Also, we haven't seen any reports of
> this problem on other filesystems. So it could be an ext4 bug.
>
> Your traces also have lots of tasks hung waiting for page lock. It is
> possible that wakeups get lost, which is fixed by this commit in
> mainline
> 777c6c5f1f6e757ae49ecca2ed72d6b1f523c007
>
> Which might also be your bug.
>
>
> Any chance you can test this patch (as well as the existing patches
> you are using to fix write_cache_pages?).
I could test 2.6.28.4
--
Ralf Hildebrandt Ralf.Hildebrandt@charite.de
Charite - Universitätsmedizin Berlin Tel. +49 (0)30-450 570-155
Geschäftsbereich IT | Abt. Netzwerk Fax. +49 (0)30-450 570-962
Hindenburgdamm 30 | 12200 Berlin
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: sync-Regression in 2.6.28.2?
2009-02-09 13:49 ` Ralf Hildebrandt
@ 2009-02-15 13:42 ` Ralf Hildebrandt
2009-02-17 4:17 ` Nick Piggin
0 siblings, 1 reply; 14+ messages in thread
From: Ralf Hildebrandt @ 2009-02-15 13:42 UTC (permalink / raw)
To: linux-kernel; +Cc: Nick Piggin, Federico Cuello, Artem Bityutskiy
* Ralf Hildebrandt <Ralf.Hildebrandt@charite.de>:
> > This patch seems good to me. If you would care to add a changelog and
> > Signed-off-by: line, then we could get it merged?
> >
> > I am not too sure about this bug. I have reproduced a strange hang with
> > ext4 (which does include sys_sync and write_cache_pages traces), and
> > also turned up a lockdep report. Also, we haven't seen any reports of
> > this problem on other filesystems. So it could be an ext4 bug.
> >
> > Your traces also have lots of tasks hung waiting for page lock. It is
> > possible that wakeups get lost, which is fixed by this commit in
> > mainline
> > 777c6c5f1f6e757ae49ecca2ed72d6b1f523c007
> >
> > Which might also be your bug.
> >
> >
> > Any chance you can test this patch (as well as the existing patches
> > you are using to fix write_cache_pages?).
>
> I could test 2.6.28.4
Still there in 2.6.28.5 :(
--
Ralf Hildebrandt Ralf.Hildebrandt@charite.de
Charite - Universitätsmedizin Berlin Tel. +49 (0)30-450 570-155
Geschäftsbereich IT | Abt. Netzwerk Fax. +49 (0)30-450 570-962
Hindenburgdamm 30 | 12200 Berlin
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: sync-Regression in 2.6.28.2?
2009-02-15 13:42 ` Ralf Hildebrandt
@ 2009-02-17 4:17 ` Nick Piggin
2009-02-17 15:15 ` Ralf Hildebrandt
0 siblings, 1 reply; 14+ messages in thread
From: Nick Piggin @ 2009-02-17 4:17 UTC (permalink / raw)
To: Ralf Hildebrandt, stable, linux-ext4
Cc: linux-kernel, Federico Cuello, Artem Bityutskiy
On Monday 16 February 2009 00:42:35 Ralf Hildebrandt wrote:
> * Ralf Hildebrandt <Ralf.Hildebrandt@charite.de>:
> > > This patch seems good to me. If you would care to add a changelog and
> > > Signed-off-by: line, then we could get it merged?
> > >
> > > I am not too sure about this bug. I have reproduced a strange hang with
> > > ext4 (which does include sys_sync and write_cache_pages traces), and
> > > also turned up a lockdep report. Also, we haven't seen any reports of
> > > this problem on other filesystems. So it could be an ext4 bug.
> > >
> > > Your traces also have lots of tasks hung waiting for page lock. It is
> > > possible that wakeups get lost, which is fixed by this commit in
> > > mainline
> > > 777c6c5f1f6e757ae49ecca2ed72d6b1f523c007
> > >
> > > Which might also be your bug.
> > >
> > >
> > > Any chance you can test this patch (as well as the existing patches
> > > you are using to fix write_cache_pages?).
> >
> > I could test 2.6.28.4
>
> Still there in 2.6.28.5 :(
I believe it is basically a deadlock in ext4 due to the cyclic writeback
change.
http://git.kernel.org/?p=linux/kernel/git/tytso/ext4.git;a=commitdiff;h=2acf2c261b823d9d9ed954f348b97620297a36b5
ext4 guys, is this correct? Is there a minimal fix that can be made for
.stable kernels?
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: sync-Regression in 2.6.28.2?
2009-02-17 4:17 ` Nick Piggin
@ 2009-02-17 15:15 ` Ralf Hildebrandt
0 siblings, 0 replies; 14+ messages in thread
From: Ralf Hildebrandt @ 2009-02-17 15:15 UTC (permalink / raw)
To: linux-kernel
* Nick Piggin <nickpiggin@yahoo.com.au>:
> > Still there in 2.6.28.5 :(
>
> I believe it is basically a deadlock in ext4 due to the cyclic writeback
> change.
That could be. It happens during time of big write activity here.
> http://git.kernel.org/?p=linux/kernel/git/tytso/ext4.git;a=commitdiff;h=2acf2c261b823d9d9ed954f348b97620297a36b5
>
> ext4 guys, is this correct? Is there a minimal fix that can be made for
> .stable kernels?
>
--
Ralf Hildebrandt Ralf.Hildebrandt@charite.de
Charite - Universitätsmedizin Berlin Tel. +49 (0)30-450 570-155
Geschäftsbereich IT | Abt. Netzwerk Fax. +49 (0)30-450 570-962
Hindenburgdamm 30 | 12200 Berlin
^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2009-02-17 15:15 UTC | newest]
Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2009-01-27 9:35 sync-Regression in 2.6.28.2? Ralf Hildebrandt
2009-01-27 21:09 ` Federico Cuello
2009-02-03 1:09 ` Nick Piggin
2009-02-03 19:51 ` Federico Cuello
2009-02-04 6:17 ` Nick Piggin
2009-02-04 17:31 ` Federico Cuello
2009-02-05 3:25 ` Nick Piggin
2009-02-05 11:54 ` Federico Cuello
2009-02-09 13:45 ` Nick Piggin
2009-02-09 13:49 ` Ralf Hildebrandt
2009-02-15 13:42 ` Ralf Hildebrandt
2009-02-17 4:17 ` Nick Piggin
2009-02-17 15:15 ` Ralf Hildebrandt
2009-02-05 10:19 ` Ralf Hildebrandt
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).