kernel thread "flush-254:12" eats 100% CPU

kernel thread "flush-254:12" eats 100% CPU

[Marcin M]

Hello!
I hope i'm writing to correct place (but i suspect g.l.kernel list could 
be proper also) :)
I'm observing situation like in subject. From time to time 
"flush-254:12" takes 100% of cpu for a couple of minutes. Then 
everything becomes ok.
This problem appears on my two different boxes.

box A)
bare metal, i686, hardened-kernel-2.6.{37-38}
on this box kernel thread never stops to do 100% of cpu. I've to reboot 
box (using sysrq, because i could't even umount partition associated 
with this thread)
My workarround for it: i changed ext4 to xfs on dm device and problem 
disappeared

box B)
xen, full virtualization, x86_64, kernels: 
hardened-kernel-{2.6.39,3.0.3,3.0.4} and 3.1.0-rc4-git2
problem is as i described above, flush took one cpu for a couple minutes 
and then everythings works correctly.

And again, 254-12 device is ext4 filesystem.
In both cases it was used lvm. At box B, additionaly i'm using dmcrypt. 
Box B is configured in this way: sda->lvm2->dmcrypt->filesystem.

How can i help to debug this problem? Or maybe it's fixed already?

Regards,
Marcin.

Re: kernel thread "flush-254:12" eats 100% CPU

[Marcin M]

Suplement.
High cpu load appears in random time interval.
All filesystems which has this problem was created with smaller than
default block size. I've got a few other fs with small black size but
thread flush-254 always appear on the same device.

Re: kernel thread "flush-254:12" eats 100% CPU

[Tao Ma]

On 09/07/2011 11:46 PM, Marcin M wrote:
> Hello!
> I hope i'm writing to correct place (but i suspect g.l.kernel list could
> be proper also) :)
> I'm observing situation like in subject. From time to time
> "flush-254:12" takes 100% of cpu for a couple of minutes. Then
> everything becomes ok.
> This problem appears on my two different boxes.
> 
> box A)
> bare metal, i686, hardened-kernel-2.6.{37-38}
> on this box kernel thread never stops to do 100% of cpu. I've to reboot
> box (using sysrq, because i could't even umount partition associated
> with this thread)
> My workarround for it: i changed ext4 to xfs on dm device and problem
> disappeared
So do you have the backstrace for it? Just wonder what it is doing.
echo t > /proc/sysrq-trigger should tell us what the flusher is doing at
that time.

Thanks
Tao
> 
> box B)
> xen, full virtualization, x86_64, kernels:
> hardened-kernel-{2.6.39,3.0.3,3.0.4} and 3.1.0-rc4-git2
> problem is as i described above, flush took one cpu for a couple minutes
> and then everythings works correctly.
> 
> And again, 254-12 device is ext4 filesystem.
> In both cases it was used lvm. At box B, additionaly i'm using dmcrypt.
> Box B is configured in this way: sda->lvm2->dmcrypt->filesystem.
> 
> How can i help to debug this problem? Or maybe it's fixed already?
> 
> Regards,
> Marcin.
> 
> -- 
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: kernel thread "flush-254:12" eats 100% CPU

[Marcin M]

W dniu 08.09.2011 07:15, Tao Ma pisze:
> So do you have the backstrace for it? Just wonder what it is doing.
> echo t>  /proc/sysrq-trigger should tell us what the flusher is doing at
> that time.

I don't have backstrace, if you tellme how to get it then i'll do it:)
I did echo t>.... when flush was too active, here it is:

[70946.742426] flush-254:12    R  running task        0 27198      2 
0x00000000
[70946.742426]  ffff88001ea5bde0 0000000000000046 0000000000000265 
ffff88001ead56a0
[70946.742426]  000000000000fe00 ffff88000f515fd8 0000000000004000 
ffff88000f514000
[70946.742426]  ffff88000f515fd8 000000000000fe00 ffff88001ea5bde0 
000000000000fe00
[70946.742426] Call Trace:
[70946.742426]  [<ffffffff8119d2c3>] ? 
radix_tree_gang_lookup_tag_slot+0x83/0xe0
[70946.742426]  [<ffffffff81083a43>] ? find_get_pages_tag+0x33/0x130
[70946.742426]  [<ffffffff8102ed7d>] ? __cond_resched+0x1d/0x30
[70946.742426]  [<ffffffff812fb50f>] ? _cond_resched+0x2f/0x40
[70946.742426]  [<ffffffff811353ef>] ? write_cache_pages_da+0x32f/0x380
[70946.742426]  [<ffffffff8115d7f5>] ? jbd2__journal_start+0xd5/0x120
[70946.742426]  [<ffffffff8113f358>] ? ext4_journal_start_sb+0x68/0x170
[70946.742426]  [<ffffffff8113567e>] ? ext4_da_writepages+0x23e/0x440
[70946.742426]  [<ffffffff810ee72e>] ? writeback_single_inode+0x11e/0x2c0
[70946.742426]  [<ffffffff810eecb0>] ? writeback_sb_inodes+0x1c0/0x280
[70946.742426]  [<ffffffff810ef17e>] ? __writeback_inodes_wb+0x8e/0xc0
[70946.742426]  [<ffffffff810ef383>] ? wb_writeback+0x1d3/0x1e0
[70946.742426]  [<ffffffff810e4791>] ? get_nr_dirty_inodes+0x51/0x80
[70946.742426]  [<ffffffff810efbfe>] ? wb_do_writeback+0x1ce/0x1e0
[70946.742426]  [<ffffffff810efc92>] ? bdi_writeback_thread+0x82/0x150
[70946.742426]  [<ffffffff810efc10>] ? wb_do_writeback+0x1e0/0x1e0
[70946.742426]  [<ffffffff810efc10>] ? wb_do_writeback+0x1e0/0x1e0
[70946.742426]  [<ffffffff81050a46>] ? kthread+0x96/0xa0
[70946.742426]  [<ffffffff812fe974>] ? kernel_thread_helper+0x4/0x10
[70946.742426]  [<ffffffff810509b0>] ? kthread_worker_fn+0x120/0x120
[70946.742426]  [<ffffffff812fe970>] ? gs_change+0xb/0xb


Thanks.

Re: kernel thread "flush-254:12" eats 100% CPU

[Marcin M]

Hello,
Is it something that i can do to help with this problem?
Regards.