Kernel BUG in 2.6.23.9

Kernel BUG in 2.6.23.9

[Jerry Geis]

I am using centos 5.1, 2.6.23.9 kernel. Machine works great all day. Then sometime through the night the kernel dies,
the trace is below.

My machine is an AMD 6400+ X2, seagate 500G SATA, Nvidia 7100, 4 GIG RAM.
---------------------------
Linux version 2.6.23.9 (root@devcentos5x64.msgnet.com) (gcc version 4.1.2 20070626 (Red Hat 4.1.2-14)) #1 SMP Mon Dec 3 10:20:13 EST 2007


If some fields are empty or look unusual you may have an old version.
Compare to the current minimal requirements in Documentation/Changes.
 
Linux devcentos5x64.msgnet.com 2.6.23.9 #1 SMP Mon Dec 3 10:20:13 EST 2007 x86_64 x86_64 x86_64 GNU/Linux
 
Gnu C                  4.1.2
Gnu make               3.81
binutils               2.17.50.0.6-5.el5
util-linux             2.13-pre7
mount                  2.13-pre7
module-init-tools      3.3-pre2
e2fsprogs              1.39
pcmciautils            014
quota-tools            3.13.
PPP                    2.4.4
isdn4k-utils           3.9
Linux C Library        2.5
Dynamic linker (ldd)   2.5
Procps                 3.2.7
Net-tools              1.60
Kbd                    1.12
oprofile               0.9.2
Sh-utils               5.97
udev                   095
wireless-tools         28
Modules Loaded         zttranscode nfsd exportfs lockd nfs_acl auth_rpcgss autofs4 hidp rfcomm l2cap bluetooth sunrpc wcusb wctdm wcfxo wcte11xp wct1xxp wcte12xp wct4xxp tor2 nf_conntrack_netbios_ns ipt_REJECT nf_conntrack_ipv4 xt_state nf_conntrack nfnetlink iptable_filter ip_tables ip6t_REJECT xt_tcpudp ip6table_filter ip6_tables x_tables cpufreq_ondemand dm_mirror dm_multipath dm_mod video output sbs battery ac ipv6 lp shpchp snd_hda_intel snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd soundcore snd_page_alloc nvidia sr_mod wctdm24xxp i2c_nforce2 i2c_core cdrom zaptel crc_ccitt forcedeth usb_storage button sg parport_pc parport serio_raw sata_nv libata sd_mod scsi_mod raid1 ext3 jbd ehci_hcd ohci_hcd uhci_hcd
[root@devcentos5x64 linux-2.6.23.9]# 



Dec  8 22:23:57 devcentos5x64 kernel: Assertion failure in journal_dirty_data() at fs/jbd/transaction.c:983: "jh->b_transaction == journal->j_committing_transaction"
Dec  8 22:23:57 devcentos5x64 kernel: ------------[ cut here ]------------
Dec  8 22:23:57 devcentos5x64 kernel: kernel BUG at fs/jbd/transaction.c:983!
Dec  8 22:23:57 devcentos5x64 kernel: invalid opcode: 0000 [1] SMP 
Dec  8 22:23:57 devcentos5x64 kernel: CPU 1 
Dec  8 22:23:57 devcentos5x64 kernel: Modules linked in: zttranscode nfsd exportfs lockd nfs_acl auth_rpcgss autofs4 hidp rfcomm l2cap bluetooth sunrpc wcusb wctdm wcfxo wcte11xp wct1xxp wcte12xp wct4xxp tor2 nf_conntrack_netbios_ns ipt_REJECT nf_conntrack_ipv4 xt_state nf_conntrack nfnetlink iptable_filter ip_tables ip6t_REJECT xt_tcpudp ip6table_filter ip6_tables x_tables cpufreq_ondemand dm_mirror dm_multipath dm_mod video output sbs battery ac ipv6 lp shpchp snd_hda_intel snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss snd_pcm nvidia(P) snd_timer wctdm24xxp snd zaptel soundcore sr_mod forcedeth sg parport_pc snd_page_alloc crc_ccitt serio_raw cdrom parport i2c_nforce2 usb_storage i2c_core button sata_nv libata sd_mod scsi_mod raid1 ext3 jbd ehci_hcd ohci_hcd uhci_hcd
Dec  8 22:23:57 devcentos5x64 kernel: Pid: 5028, comm: rsync Tainted: P        2.6.23.9 #1
Dec  8 22:23:57 devcentos5x64 kernel: RIP: 0010:[<ffffffff88034a3b>]  [<ffffffff88034a3b>] :jbd:journal_dirty_data+0xad/0x1ed
Dec  8 22:23:57 devcentos5x64 kernel: RSP: 0018:ffff810095edd7e8  EFLAGS: 00010292
Dec  8 22:23:57 devcentos5x64 kernel: RAX: 0000000000000083 RBX: ffff81008e1f50c8 RCX: ffffffff81344c28
Dec  8 22:23:57 devcentos5x64 kernel: RDX: 0000000100000000 RSI: 0000000000000096 RDI: ffffffff81344c20
Dec  8 22:23:57 devcentos5x64 kernel: RBP: ffff81009b42fdf4 R08: ffffffff81344c28 R09: 00000002ffffffff
Dec  8 22:23:57 devcentos5x64 kernel: R10: 0000000000000046 R11: ffffffff813b8000 R12: ffff8100c082b348
Dec  8 22:23:58 devcentos5x64 kernel: R13: ffff81010ce3cc00 R14: 0000000000000000 R15: 0000000000000000
Dec  8 22:23:58 devcentos5x64 kernel: FS:  00002ae6d4b2b560(0000) GS:ffff81011fa9a6c0(0000) knlGS:00000000f7f009a0
Dec  8 22:23:58 devcentos5x64 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Dec  8 22:23:58 devcentos5x64 kernel: CR2: 00002aea6b0d30a8 CR3: 00000000959f0000 CR4: 00000000000006e0
Dec  8 22:23:58 devcentos5x64 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Dec  8 22:23:58 devcentos5x64 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Dec  8 22:23:58 devcentos5x64 kernel: Process rsync (pid: 5028, threadinfo ffff810095edc000, task ffff8100ccc78000)
Dec  8 22:23:58 devcentos5x64 kernel: Stack:  0000000000001000 ffff8100c082b348 ffff81008e1f50c8 ffff81008e1f50c8
Dec  8 22:23:58 devcentos5x64 kernel:  ffff81008e1f50c8 ffffffff8804ff8f 0000000000001000 0000000000001000
Dec  8 22:23:58 devcentos5x64 kernel:  0000000000001000 ffffffff8804f437 ffffffff8804ffb4 0000000000001000
Dec  8 22:23:58 devcentos5x64 kernel: Call Trace:
Dec  8 22:23:58 devcentos5x64 kernel:  [<ffffffff8804ff8f>] :ext3:ext3_journal_dirty_data+0xf/0x34
Dec  8 22:23:58 devcentos5x64 kernel:  [<ffffffff8804f437>] :ext3:walk_page_buffers+0x65/0x8b
Dec  8 22:23:58 devcentos5x64 kernel:  [<ffffffff8804ffb4>] :ext3:journal_dirty_data_fn+0x0/0xe
Dec  8 22:23:58 devcentos5x64 kernel:  [<ffffffff880521da>] :ext3:ext3_ordered_writepage+0x108/0x18c
Dec  8 22:23:58 devcentos5x64 kernel:  [<ffffffff8106bf9a>] __writepage+0xa/0x23
Dec  8 22:23:58 devcentos5x64 kernel:  [<ffffffff8106c4b8>] write_cache_pages+0x17b/0x2aa
Dec  8 22:23:58 devcentos5x64 kernel:  [<ffffffff8106bf90>] __writepage+0x0/0x23
Dec  8 22:23:58 devcentos5x64 kernel:  [<ffffffff8106c62a>] do_writepages+0x27/0x2d
Dec  8 22:23:58 devcentos5x64 kernel:  [<ffffffff810a8749>] __writeback_single_inode+0x1ba/0x332
Dec  8 22:23:58 devcentos5x64 kernel:  [<ffffffff8805b7ea>] :ext3:__ext3_journal_dirty_metadata+0x1e/0x46
Dec  8 22:23:58 devcentos5x64 kernel:  [<ffffffff810a8c00>] sync_sb_inodes+0x1a3/0x261
Dec  8 22:23:58 devcentos5x64 kernel:  [<ffffffff810a9030>] writeback_inodes+0x82/0xd8
Dec  8 22:23:58 devcentos5x64 kernel:  [<ffffffff8106c869>] balance_dirty_pages_ratelimited_nr+0x114/0x1f4
Dec  8 22:23:58 devcentos5x64 kernel:  [<ffffffff81068168>] generic_file_buffered_write+0x62d/0x751
Dec  8 22:23:58 devcentos5x64 kernel:  [<ffffffff810685c8>] __generic_file_aio_write_nolock+0x33c/0x3a6
Dec  8 22:23:58 devcentos5x64 kernel:  [<ffffffff811ec26b>] sock_aio_read+0x10c/0x124
Dec  8 22:23:58 devcentos5x64 kernel:  [<ffffffff81068693>] generic_file_aio_write+0x61/0xc1
Dec  8 22:23:58 devcentos5x64 kernel:  [<ffffffff8804e1ae>] :ext3:ext3_file_write+0x16/0x94
Dec  8 22:23:58 devcentos5x64 kernel:  [<ffffffff8108d950>] do_sync_write+0xc9/0x10c
Dec  8 22:23:58 devcentos5x64 kernel:  [<ffffffff8104912d>] autoremove_wake_function+0x0/0x2e
Dec  8 22:23:58 devcentos5x64 kernel:  [<ffffffff8108e118>] vfs_write+0xce/0x157
Dec  8 22:23:58 devcentos5x64 kernel:  [<ffffffff8108e676>] sys_write+0x45/0x6e
Dec  8 22:23:58 devcentos5x64 kernel:  [<ffffffff8100bdfc>] tracesys+0xdc/0xe1
Dec  8 22:23:58 devcentos5x64 kernel: 
Dec  8 22:23:58 devcentos5x64 kernel: 
Dec  8 22:23:58 devcentos5x64 kernel: Code: 0f 0b eb fe 8b 45 0c 83 f8 01 76 09 83 f8 08 0f 85 eb 00 00 
Dec  8 22:23:58 devcentos5x64 kernel: RIP  [<ffffffff88034a3b>] :jbd:journal_dirty_data+0xad/0x1ed
Dec  8 22:23:58 devcentos5x64 kernel:  RSP <ffff810095edd7e8>

Re: Kernel BUG in 2.6.23.9

[Arjan van de Ven]

On Sun, 09 Dec 2007 14:49:16 -0500
Jerry Geis <geisj@pagestation.com> wrote:

> I am using centos 5.1, 2.6.23.9 kernel. Machine works great all day.
> Then sometime through the night the kernel dies, the trace is below.

is this when a 3D screensaver is active?
Probably worth running without the proprietary nvidia driver.... it
also tends to get more people to pay attention :)


In addition to this suspicion; you're the first one to report this oops
since at least a year as shown in
http://www.kerneloops.org/guilty.php?guilty=journal_dirty_data&version=2.6.23-release&start=1540096&end=1572863

so it might well be something weird on your side (like the binary
driver or some heat problem created by having a 3D screensaver)

-- 
If you want to reach me at my work email, use arjan@linux.intel.com
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org

Re: Kernel BUG in 2.6.23.9

[Tarkan Erimer]

Arjan van de Ven wrote:
> On Sun, 09 Dec 2007 14:49:16 -0500
> Jerry Geis <geisj@pagestation.com> wrote:
>
>   
>> I am using centos 5.1, 2.6.23.9 kernel. Machine works great all day.
>> Then sometime through the night the kernel dies, the trace is below.
>>     
>
> is this when a 3D screensaver is active?
> Probably worth running without the proprietary nvidia driver.... it
> also tends to get more people to pay attention :)
>
>
> In addition to this suspicion; you're the first one to report this oops
> since at least a year as shown in
> http://www.kerneloops.org/guilty.php?guilty=journal_dirty_data&version=2.6.23-release&start=1540096&end=1572863
>
> so it might well be something weird on your side (like the binary
> driver or some heat problem created by having a 3D screensaver)
>
>   
Hi,

For a while, I have the same behavior. I tested under 2.6.23.9, 
2.6.24-rc3 and rc4. So all of these kernels have the same issue. Also, I 
don't use any proprietary driver. My video card is Intel-945 and I use 
the "intel" driver. I use 3D screensavers as well. These kernels die 
after some long idles like 1 or 2 days after. My distribution is Kubuntu 
7.10 and with Kubuntu kernel this behavior did not happened. So, how to 
track this interesting problem ? Any ideas welcomed :-)

Cheers,

Tarkan Erimer

Re: Kernel BUG in 2.6.23.9

[Jerry Geis]

Tarkan Erimer wrote:
> Arjan van de Ven wrote:
>> On Sun, 09 Dec 2007 14:49:16 -0500
>> Jerry Geis <geisj@pagestation.com> wrote:
>>
>>  
>>> I am using centos 5.1, 2.6.23.9 kernel. Machine works great all day.
>>> Then sometime through the night the kernel dies, the trace is below.
>>>     
>>
>> is this when a 3D screensaver is active?
>> Probably worth running without the proprietary nvidia driver.... it
>> also tends to get more people to pay attention :)
>>
>>
>> In addition to this suspicion; you're the first one to report this oops
>> since at least a year as shown in
>> http://www.kerneloops.org/guilty.php?guilty=journal_dirty_data&version=2.6.23-release&start=1540096&end=1572863 
>>
>>
>> so it might well be something weird on your side (like the binary
>> driver or some heat problem created by having a 3D screensaver)
>>
>>   
> Hi,
>
> For a while, I have the same behavior. I tested under 2.6.23.9, 
> 2.6.24-rc3 and rc4. So all of these kernels have the same issue. Also, 
> I don't use any proprietary driver. My video card is Intel-945 and I 
> use the "intel" driver. I use 3D screensavers as well. These kernels 
> die after some long idles like 1 or 2 days after. My distribution is 
> Kubuntu 7.10 and with Kubuntu kernel this behavior did not happened. 
> So, how to track this interesting problem ? Any ideas welcomed :-)
>
> Cheers,
>
> Tarkan Erimer
>
>
>
>
>
I am not running a 3D screen saver. I was just using blank screen.
However, I did turn off the screen saver and last night my machine did 
not crash.

I am still using the NVIDIA driver.
All I changed was the screen saver.

Any ideas on that?

Jerry

Re: Kernel BUG in 2.6.23.9

[Tarkan Erimer]

Arjan van de Ven wrote:
> On Sun, 09 Dec 2007 14:49:16 -0500
> Jerry Geis <geisj@pagestation.com> wrote:
>
>  
>> I am using centos 5.1, 2.6.23.9 kernel. Machine works great all day.
>> Then sometime through the night the kernel dies, the trace is below.
>>     
>
> is this when a 3D screensaver is active?
> Probably worth running without the proprietary nvidia driver.... it
> also tends to get more people to pay attention :)
>
>
> In addition to this suspicion; you're the first one to report this oops
> since at least a year as shown in
> http://www.kerneloops.org/guilty.php?guilty=journal_dirty_data&version=2.6.23-release&start=1540096&end=1572863 
>
>
> so it might well be something weird on your side (like the binary
> driver or some heat problem created by having a 3D screensaver)
>
>   
Hi,

For a while, I have the same behavior. I tested under 2.6.23.9, 
2.6.24-rc3 and rc4. So all of these kernels have the same issue. Also, I 
don't use any proprietary driver. My video card is Intel-945 and I use 
the "intel" driver. I use 3D screensavers as well. These kernels die 
after some long idles like 1 or 2 days after. My distribution is Kubuntu 
7.10 and with Kubuntu kernel this behavior did not happened. So, how to 
track this interesting problem ? Any ideas welcomed :-)

Cheers,

Tarkan Erimer

Re: Kernel BUG in 2.6.23.9

[Arjan van de Ven]

On Mon, 10 Dec 2007 11:14:47 +0200
Tarkan Erimer <tarkan@netone.net.tr> wrote:
   
> Hi,
> 
> For a while, I have the same behavior. I tested under 2.6.23.9, 
> 2.6.24-rc3 and rc4. So all of these kernels have the same issue.
> Also, I don't use any proprietary driver. My video card is Intel-945
> and I use the "intel" driver. I use 3D screensavers as well. These
> kernels die after some long idles like 1 or 2 days after. My
> distribution is Kubuntu 7.10 and with Kubuntu kernel this behavior
> did not happened. So, how to track this interesting problem ? Any
> ideas welcomed :-)
> 


well... try to get the oops message out; for many causes of crash that's a good way to find out what happened...
(not always, but it allows us to find patterns if nothing else)


-- 
If you want to reach me at my work email, use arjan@linux.intel.com
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org

Re: Kernel BUG in 2.6.23.9

[Tarkan Erimer]

Arjan van de Ven wrote:
> well... try to get the oops message out; for many causes of crash that's a good way to find out what happened...
> (not always, but it allows us to find patterns if nothing else)
>
>   
Unfortunately, I couldn't get the oops message. Because when this 
behavior happens, the machine hardly freezes and the SysRq combinations 
do not work. Also, I checked the logs (message,syslog,kern.log etc.) to 
see something suspicious. Unfortunately, there was none. Just one thing 
I noticed  when investigating on the logs: the kernel is doing its 
regular jobs. Like cron etc. So, the kernel is not really dieing. Next 
time, I will try to ssh to my machine to check what's going on, when 
this behavior occurred.

Re: Kernel BUG in 2.6.23.9

[Jerry Geis]

Tarkan Erimer wrote:
> Arjan van de Ven wrote:
>> well... try to get the oops message out; for many causes of crash 
>> that's a good way to find out what happened...
>> (not always, but it allows us to find patterns if nothing else)
>>
>>   
> Unfortunately, I couldn't get the oops message. Because when this 
> behavior happens, the machine hardly freezes and the SysRq 
> combinations do not work. Also, I checked the logs 
> (message,syslog,kern.log etc.) to see something suspicious. 
> Unfortunately, there was none. Just one thing I noticed  when 
> investigating on the logs: the kernel is doing its regular jobs. Like 
> cron etc. So, the kernel is not really dieing. Next time, I will try 
> to ssh to my machine to check what's going on, when this behavior 
> occurred.
>
I agree, the kernel is not completely dead. I could ping my box but 
nothing else.

This was the /var/log/messages I had:

Dec  8 22:23:57 devcentos5x64 kernel: Assertion failure in journal_dirty_data() at fs/jbd/transaction.c:983: "jh->b_transaction == journal->j_committing_transaction"
Dec  8 22:23:57 devcentos5x64 kernel: ------------[ cut here ]------------
Dec  8 22:23:57 devcentos5x64 kernel: kernel BUG at fs/jbd/transaction.c:983!
Dec  8 22:23:57 devcentos5x64 kernel: invalid opcode: 0000 [1] SMP 
Dec  8 22:23:57 devcentos5x64 kernel: CPU 1 
Dec  8 22:23:57 devcentos5x64 kernel: Modules linked in: zttranscode nfsd exportfs lockd nfs_acl auth_rpcgss autofs4 hidp rfcomm l2cap bluetooth sunrpc wcusb wctdm wcfxo wcte11xp wct1xxp wcte12xp wct4xxp tor2 nf_conntrack_netbios_ns ipt_REJECT nf_conntrack_ipv4 xt_state nf_conntrack nfnetlink iptable_filter ip_tables ip6t_REJECT xt_tcpudp ip6table_filter ip6_tables x_tables cpufreq_ondemand dm_mirror dm_multipath dm_mod video output sbs battery ac ipv6 lp shpchp snd_hda_intel snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss snd_pcm nvidia(P) snd_timer wctdm24xxp snd zaptel soundcore sr_mod forcedeth sg parport_pc snd_page_alloc crc_ccitt serio_raw cdrom parport i2c_nforce2 usb_storage i2c_core button sata_nv libata sd_mod scsi_mod raid1 ext3 jbd ehci_hcd ohci_hcd uhci_hcd
Dec  8 22:23:57 devcentos5x64 kernel: Pid: 5028, comm: rsync Tainted: P        2.6.23.9 #1
Dec  8 22:23:57 devcentos5x64 kernel: RIP: 0010:[<ffffffff88034a3b>]  [<ffffffff88034a3b>] :jbd:journal_dirty_data+0xad/0x1ed
Dec  8 22:23:57 devcentos5x64 kernel: RSP: 0018:ffff810095edd7e8  EFLAGS: 00010292
Dec  8 22:23:57 devcentos5x64 kernel: RAX: 0000000000000083 RBX: ffff81008e1f50c8 RCX: ffffffff81344c28
Dec  8 22:23:57 devcentos5x64 kernel: RDX: 0000000100000000 RSI: 0000000000000096 RDI: ffffffff81344c20
Dec  8 22:23:57 devcentos5x64 kernel: RBP: ffff81009b42fdf4 R08: ffffffff81344c28 R09: 00000002ffffffff
Dec  8 22:23:57 devcentos5x64 kernel: R10: 0000000000000046 R11: ffffffff813b8000 R12: ffff8100c082b348
Dec  8 22:23:58 devcentos5x64 kernel: R13: ffff81010ce3cc00 R14: 0000000000000000 R15: 0000000000000000
Dec  8 22:23:58 devcentos5x64 kernel: FS:  00002ae6d4b2b560(0000) GS:ffff81011fa9a6c0(0000) knlGS:00000000f7f009a0
Dec  8 22:23:58 devcentos5x64 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Dec  8 22:23:58 devcentos5x64 kernel: CR2: 00002aea6b0d30a8 CR3: 00000000959f0000 CR4: 00000000000006e0
Dec  8 22:23:58 devcentos5x64 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Dec  8 22:23:58 devcentos5x64 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Dec  8 22:23:58 devcentos5x64 kernel: Process rsync (pid: 5028, threadinfo ffff810095edc000, task ffff8100ccc78000)
Dec  8 22:23:58 devcentos5x64 kernel: Stack:  0000000000001000 ffff8100c082b348 ffff81008e1f50c8 ffff81008e1f50c8
Dec  8 22:23:58 devcentos5x64 kernel:  ffff81008e1f50c8 ffffffff8804ff8f 0000000000001000 0000000000001000
Dec  8 22:23:58 devcentos5x64 kernel:  0000000000001000 ffffffff8804f437 ffffffff8804ffb4 0000000000001000
Dec  8 22:23:58 devcentos5x64 kernel: Call Trace:
Dec  8 22:23:58 devcentos5x64 kernel:  [<ffffffff8804ff8f>] :ext3:ext3_journal_dirty_data+0xf/0x34
Dec  8 22:23:58 devcentos5x64 kernel:  [<ffffffff8804f437>] :ext3:walk_page_buffers+0x65/0x8b
Dec  8 22:23:58 devcentos5x64 kernel:  [<ffffffff8804ffb4>] :ext3:journal_dirty_data_fn+0x0/0xe
Dec  8 22:23:58 devcentos5x64 kernel:  [<ffffffff880521da>] :ext3:ext3_ordered_writepage+0x108/0x18c
Dec  8 22:23:58 devcentos5x64 kernel:  [<ffffffff8106bf9a>] __writepage+0xa/0x23
Dec  8 22:23:58 devcentos5x64 kernel:  [<ffffffff8106c4b8>] write_cache_pages+0x17b/0x2aa
Dec  8 22:23:58 devcentos5x64 kernel:  [<ffffffff8106bf90>] __writepage+0x0/0x23
Dec  8 22:23:58 devcentos5x64 kernel:  [<ffffffff8106c62a>] do_writepages+0x27/0x2d
Dec  8 22:23:58 devcentos5x64 kernel:  [<ffffffff810a8749>] __writeback_single_inode+0x1ba/0x332
Dec  8 22:23:58 devcentos5x64 kernel:  [<ffffffff8805b7ea>] :ext3:__ext3_journal_dirty_metadata+0x1e/0x46
Dec  8 22:23:58 devcentos5x64 kernel:  [<ffffffff810a8c00>] sync_sb_inodes+0x1a3/0x261
Dec  8 22:23:58 devcentos5x64 kernel:  [<ffffffff810a9030>] writeback_inodes+0x82/0xd8
Dec  8 22:23:58 devcentos5x64 kernel:  [<ffffffff8106c869>] balance_dirty_pages_ratelimited_nr+0x114/0x1f4
Dec  8 22:23:58 devcentos5x64 kernel:  [<ffffffff81068168>] generic_file_buffered_write+0x62d/0x751
Dec  8 22:23:58 devcentos5x64 kernel:  [<ffffffff810685c8>] __generic_file_aio_write_nolock+0x33c/0x3a6
Dec  8 22:23:58 devcentos5x64 kernel:  [<ffffffff811ec26b>] sock_aio_read+0x10c/0x124
Dec  8 22:23:58 devcentos5x64 kernel:  [<ffffffff81068693>] generic_file_aio_write+0x61/0xc1
Dec  8 22:23:58 devcentos5x64 kernel:  [<ffffffff8804e1ae>] :ext3:ext3_file_write+0x16/0x94
Dec  8 22:23:58 devcentos5x64 kernel:  [<ffffffff8108d950>] do_sync_write+0xc9/0x10c
Dec  8 22:23:58 devcentos5x64 kernel:  [<ffffffff8104912d>] autoremove_wake_function+0x0/0x2e
Dec  8 22:23:58 devcentos5x64 kernel:  [<ffffffff8108e118>] vfs_write+0xce/0x157
Dec  8 22:23:58 devcentos5x64 kernel:  [<ffffffff8108e676>] sys_write+0x45/0x6e
Dec  8 22:23:58 devcentos5x64 kernel:  [<ffffffff8100bdfc>] tracesys+0xdc/0xe1
Dec  8 22:23:58 devcentos5x64 kernel: 
Dec  8 22:23:58 devcentos5x64 kernel: 
Dec  8 22:23:58 devcentos5x64 kernel: Code: 0f 0b eb fe 8b 45 0c 83 f8 01 76 09 83 f8 08 0f 85 eb 00 00 
Dec  8 22:23:58 devcentos5x64 kernel: RIP  [<ffffffff88034a3b>] :jbd:journal_dirty_data+0xad/0x1ed
Dec  8 22:23:58 devcentos5x64 kernel:  RSP <ffff810095edd7e8>