Re: Ext4 flush blocked

Re: Ext4 flush blocked

[Felipe Wilhelms Damasio - Taghos]

    Hi Again,

Em 14/07/2011 22:19, Felipe Wilhelms Damasio - Taghos escreveu:
>      After a while using (about an hour) I get a lot of:
>
> INFO: task flush-8:16:6650 blocked for more than 120 seconds.

    I also saw this:

INFO: task jbd2/sdb1-8:1672 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
jbd2/sdb1-8   D 0000000000000004     0  1672      2 0x00000000
 ffff88042e171aa0 0000000000000046 0000000000012500 ffff88042e171fd8
 ffff88042e171fd8 ffff88042c2eadf0 0000000000012500 0000000000012500
 0000000000012500 ffff88042c2eadf0 ffff88042e0d0690 ffff88042c2eb098
Call Trace:
 [<ffffffff8151c837>] io_schedule+0x7b/0xc2
 [<ffffffff8108fceb>] sync_page+0x41/0x45
 [<ffffffff8151cc1a>] __wait_on_bit+0x47/0x79
 [<ffffffff8108fcaa>] ? sync_page+0x0/0x45
 [<ffffffff8108fea1>] wait_on_page_bit+0x6e/0x75
 [<ffffffff8104ed1c>] ? wake_bit_function+0x0/0x2a
 [<ffffffff81097a36>] ? pagevec_lookup_tag+0x22/0x2b
 [<ffffffff810965cb>] write_cache_pages+0x1d5/0x2e9
 [<ffffffff81095ee0>] ? __writepage+0x0/0x2f
 [<ffffffff810966fe>] generic_writepages+0x1f/0x25
 [<ffffffff8114251d>] jbd2_journal_commit_transaction+0x35b/0x10b2
 [<ffffffff81035288>] ? get_parent_ip+0x11/0x41
 [<ffffffff811468bf>] kjournald2+0xd2/0x22b
 [<ffffffff8104ece4>] ? autoremove_wake_function+0x0/0x38
 [<ffffffff8151dbdf>] ? _raw_spin_unlock_irqrestore+0x2c/0x37
 [<ffffffff811467ed>] ? kjournald2+0x0/0x22b
 [<ffffffff8104e892>] kthread+0x7d/0x85
 [<ffffffff810037d4>] kernel_thread_helper+0x4/0x10
 [<ffffffff8104e815>] ? kthread+0x0/0x85
 [<ffffffff810037d0>] ? kernel_thread_helper+0x0/0x10


    Don't know if this helps, though.

    Is there something I can try to fix this?

    Cheers,

-- 
Felipe Wilhelms Damasio
TAGHOS - Tecnologia
Rua Prof. Alvaro Alvim, 211
Porto Alegre - RS - (51) 3239-3180
www.taghos.com.br <http://www.taghos.com.br/>

Re: Ext4 flush blocked

[Tao Ma]

Hi Felipe,
On 07/15/2011 09:27 AM, Felipe Wilhelms Damasio - Taghos wrote:
>     Hi Again,
> 
> Em 14/07/2011 22:19, Felipe Wilhelms Damasio - Taghos escreveu:
>>      After a while using (about an hour) I get a lot of:
>>
>> INFO: task flush-8:16:6650 blocked for more than 120 seconds.
> 
>     I also saw this:
> 
> INFO: task jbd2/sdb1-8:1672 blocked for more than 120 seconds.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> jbd2/sdb1-8   D 0000000000000004     0  1672      2 0x00000000
>  ffff88042e171aa0 0000000000000046 0000000000012500 ffff88042e171fd8
>  ffff88042e171fd8 ffff88042c2eadf0 0000000000012500 0000000000012500
>  0000000000012500 ffff88042c2eadf0 ffff88042e0d0690 ffff88042c2eb098
> Call Trace:
>  [<ffffffff8151c837>] io_schedule+0x7b/0xc2
>  [<ffffffff8108fceb>] sync_page+0x41/0x45
>  [<ffffffff8151cc1a>] __wait_on_bit+0x47/0x79
>  [<ffffffff8108fcaa>] ? sync_page+0x0/0x45
>  [<ffffffff8108fea1>] wait_on_page_bit+0x6e/0x75
>  [<ffffffff8104ed1c>] ? wake_bit_function+0x0/0x2a
>  [<ffffffff81097a36>] ? pagevec_lookup_tag+0x22/0x2b
>  [<ffffffff810965cb>] write_cache_pages+0x1d5/0x2e9
>  [<ffffffff81095ee0>] ? __writepage+0x0/0x2f
>  [<ffffffff810966fe>] generic_writepages+0x1f/0x25
>  [<ffffffff8114251d>] jbd2_journal_commit_transaction+0x35b/0x10b2
>  [<ffffffff81035288>] ? get_parent_ip+0x11/0x41
>  [<ffffffff811468bf>] kjournald2+0xd2/0x22b
>  [<ffffffff8104ece4>] ? autoremove_wake_function+0x0/0x38
>  [<ffffffff8151dbdf>] ? _raw_spin_unlock_irqrestore+0x2c/0x37
>  [<ffffffff811467ed>] ? kjournald2+0x0/0x22b
>  [<ffffffff8104e892>] kthread+0x7d/0x85
>  [<ffffffff810037d4>] kernel_thread_helper+0x4/0x10
>  [<ffffffff8104e815>] ? kthread+0x0/0x85
>  [<ffffffff810037d0>] ? kernel_thread_helper+0x0/0x10
> 
> 
>     Don't know if this helps, though.
> 
>     Is there something I can try to fix this?
could you please try to change io scheduler to 'deadline'?
We have met with some similar warnings and changing to 'deadline' makes
ext4 happy.

Thanks
Tao

Re: Ext4 flush blocked

[Felipe Wilhelms Damasio - Taghos]

    Hi Tao,

Em 15/07/2011 00:31, Tao Ma escreveu:
> Is there something I can try to fix this?
> could you please try to change io scheduler to 'deadline'?
> We have met with some similar warnings and changing to 'deadline' makes
> ext4 happy.
    The symptoms were away after changing the system to use deadline
scheduler, thanks.

    Cheers,

-- 
Felipe Wilhelms Damasio
TAGHOS - Tecnologia
Rua Prof. Alvaro Alvim, 211
Porto Alegre - RS - (51) 3239-3180
www.taghos.com.br <http://www.taghos.com.br/>

Ext4 flush blocked

[Felipe Wilhelms Damasio - Taghos]

    Hi,

    I'm using a mmap-intensive file server on a Dell Machine with 2.6.35.13.

    The partition is a RAID-0 mounted with ext4 and noatime.

    After a while using (about an hour) I get a lot of:

INFO: task flush-8:16:6650 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
flush-8:16    D 0000000000000000     0  6650      2 0x00000000
 ffff880400b0d980 0000000000000046 0000000000012500 ffff880400b0dfd8
 ffff880400b0dfd8 ffff880418b34830 0000000000012500 0000000000012500
 0000000000012500 ffff880418b34830 ffffffff81a11020 ffff880418b34ad8
Call Trace:
 [<ffffffff8151c837>] io_schedule+0x7b/0xc2
 [<ffffffff8108fceb>] sync_page+0x41/0x45
 [<ffffffff8151cb15>] __wait_on_bit_lock+0x45/0x8c
 [<ffffffff8108fcaa>] ? sync_page+0x0/0x45
 [<ffffffff8108fc96>] __lock_page+0x63/0x6a
 [<ffffffff8104ed1c>] ? wake_bit_function+0x0/0x2a
 [<ffffffff8108fd88>] ? unlock_page+0x22/0x27
 [<ffffffff81122f43>] ext4_da_writepages+0x516/0x8e1
 [<ffffffff8103233f>] ? find_busiest_group+0x2e9/0x900
 [<ffffffff81096720>] do_writepages+0x1c/0x25
 [<ffffffff810df4d6>] writeback_single_inode+0xe8/0x329
 [<ffffffff810dfaa9>] writeback_sb_inodes+0x14e/0x225
 [<ffffffff810e02e1>] writeback_inodes_wb+0x146/0x156
 [<ffffffff810e04a1>] wb_writeback+0x1b0/0x232
 [<ffffffff81035288>] ? get_parent_ip+0x11/0x41
 [<ffffffff810e065c>] wb_do_writeback+0x139/0x14f
 [<ffffffff810e06b0>] bdi_writeback_task+0x3e/0x112
 [<ffffffff8104ec02>] ? bit_waitqueue+0x12/0xa3
 [<ffffffff810a1800>] ? bdi_start_fn+0x0/0xd2
 [<ffffffff810a1871>] bdi_start_fn+0x71/0xd2
 [<ffffffff810a1800>] ? bdi_start_fn+0x0/0xd2
 [<ffffffff8104e892>] kthread+0x7d/0x85
 [<ffffffff810037d4>] kernel_thread_helper+0x4/0x10
 [<ffffffff8104e815>] ? kthread+0x0/0x85
 [<ffffffff810037d0>] ? kernel_thread_helper+0x0/0x10


    The hardware is:

02:00.0 SCSI storage controller: LSI Logic / Symbios Logic SAS1068E
PCI-Express Fusion-MPT SAS (rev 08)

    The machine is RAID-0 with 2 450GB SAS 15K RPM hard drives.

sd 0:2:1:0: [sdb] 1755840512 512-byte logical blocks: (898 GB/837 GiB)
sd 0:2:1:0: [sdb] Write Protect is off
sd 0:2:1:0: [sdb] Mode Sense: 1f 00 00 08
scsi 0:0:32:0: Attached scsi generic sg0 type 13
sd 0:2:1:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support
DPO or FUA
sd 0:2:0:0: Attached scsi generic sg1 type 0
sd 0:2:1:0: Attached scsi generic sg2 type 0

    Is there any other info I can provide you to help track this bug down?

    Cheers,


-- 
Felipe Wilhelms Damasio

TAGHOS - Tecnologia
Rua Prof. Alvaro Alvim, 211
Porto Alegre - RS - (51) 3239-3180
www.taghos.com.br <http://www.taghos.com.br/>