* Ext4 Kernel Crash
@ 2012-01-29 17:50 Nanakos Chrysostomos
2012-01-29 22:03 ` Jesper Juhl
2012-01-30 17:26 ` Ted Ts'o
0 siblings, 2 replies; 5+ messages in thread
From: Nanakos Chrysostomos @ 2012-01-29 17:50 UTC (permalink / raw)
To: linux-kernel
Dear Linux Kernel Developers,
I am writing to report a kernel or ext4 bug.
I have two servers populated with 2 LUNs from the main storage facility,
each server has one LUN which is about 1.2TB. On top of that I am
utilizing glusterfs and samba to mount the 2.4TB created space to a
Windows server and other Linux machines (via glusterfs native client).
Yesterday I tried to copy to the new storage glusterfs space about 788GB
and in the morning I got from both machines the attached kernel messages.
I have to mention that the first machine runs samba and mounts the
allocated glusterfs space with the help of FUSE glusterfs driver. The
second machine doesn't mount/export anything and simply runs the glusterfs
service.
I have to also add that similar problems have happened to another system
which is used as a TIF to JP2 converter. The conversion path starts from
the conversion of a pdf file to png and then to jp2. As you can imagine
this process produces hundreds of thousands of files. Similarly the
machine crashes and I get similar kernel messages concerning ext4. I can't
give you more information on that because the machine crashes, I have to
take a photo and send it back to you next time.
What do you believe about the kernel messages? Is this a ext4 bug when
thousands of files are created? How can I help resolve that?
Please Cc me because I am not in the list.
With kind regards,
Chris.
PS: Are there any ext4 kernel optimization's for avoiding that or
narrowing down the problem if this is not so obvious?
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
glusterfsd D 0000000000000000 0 2070 1 0x00000000
ffff88003daa5c68 0000000000000082 0000000000000000 000000010000000b
ffff88003daa5d18 ffff88003daa5c98 ffff88003daa5bf8 ffffffff81090957
ffff88003a912638 ffff88003daa5fd8 000000000000f4e8 ffff88003a912638
Call Trace:
[<ffffffff81090957>] ? bit_waitqueue+0x17/0xd0
[<ffffffffa00de09d>] do_get_write_access+0x29d/0x520 [jbd2]
[<ffffffff81090ad0>] ? wake_bit_function+0x0/0x50
[<ffffffffa013293b>] ? __ext4_handle_dirty_metadata+0x7b/0x100 [ext4]
[<ffffffffa00de471>] jbd2_journal_get_write_access+0x31/0x50 [jbd2]
[<ffffffffa0132b78>] __ext4_journal_get_write_access+0x38/0x80 [ext4]
[<ffffffffa0114fb9>] ext4_orphan_add+0xb9/0x1f0 [ext4]
[<ffffffffa01146b9>] ? ext4_delete_entry+0xf9/0x110 [ext4]
[<ffffffffa0116a8b>] ext4_unlink+0x29b/0x2b0 [ext4]
[<ffffffff81183edf>] vfs_unlink+0x9f/0xe0
[<ffffffff81182c2a>] ? lookup_hash+0x3a/0x50
[<ffffffff81186463>] do_unlinkat+0x183/0x1c0
[<ffffffff81183601>] ? path_put+0x31/0x40
[<ffffffff8119ad11>] ? sys_lgetxattr+0x61/0x80
[<ffffffff811864b6>] sys_unlink+0x16/0x20
[<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
INFO: task glusterfsd:2322 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
glusterfsd D 0000000000000000 0 2322 1 0x00000000
ffff88003c70dc48 0000000000000082 0000000000000000 00000001810949db
0000000000000000 0000000000000000 ffff88003c70dcb0 0000000000000000
ffff88003b18baf8 ffff88003c70dfd8 000000000000f4e8 ffff88003b18baf8
Call Trace:
[<ffffffffa00de09d>] do_get_write_access+0x29d/0x520 [jbd2]
[<ffffffff81090ad0>] ? wake_bit_function+0x0/0x50
[<ffffffffa00de471>] jbd2_journal_get_write_access+0x31/0x50 [jbd2]
[<ffffffffa0132b78>] __ext4_journal_get_write_access+0x38/0x80 [ext4]
[<ffffffffa010e253>] ext4_reserve_inode_write+0x73/0xa0 [ext4]
[<ffffffffa0107158>] ? call_filldir+0x78/0xe0 [ext4]
[<ffffffffa010e2cc>] ext4_mark_inode_dirty+0x4c/0x1d0 [ext4]
[<ffffffffa010e5c0>] ext4_dirty_inode+0x40/0x60 [ext4]
[<ffffffff8119fceb>] __mark_inode_dirty+0x3b/0x160
[<ffffffff8119040d>] touch_atime+0x12d/0x170
[<ffffffff8118a290>] ? filldir+0x0/0xe0
[<ffffffff8118a526>] vfs_readdir+0xd6/0xe0
[<ffffffff8118a699>] sys_getdents+0x89/0xf0
[<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
INFO: task smbd:2585 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
smbd D 0000000000000000 0 2585 1998 0x00000000
ffff88003b299bb8 0000000000000082 000000000000007a 0000000000000286
ffff88003b23b8c0 000000000000007e ffff88003b299b98 ffff88003b23b8c0
ffff88003a913ab8 ffff88003b299fd8 000000000000f4e8 ffff88003a913ab8
Call Trace:
[<ffffffff814ee01e>] __mutex_lock_slowpath+0x13e/0x180
[<ffffffff8118d11f>] ? d_free+0x3f/0x60
[<ffffffff814edebb>] mutex_lock+0x2b/0x50
[<ffffffff811848af>] do_lookup+0xef/0x1e0
[<ffffffff811851d4>] __link_path_walk+0x734/0x1030
[<ffffffff81183795>] ? path_to_nameidata+0x45/0x60
[<ffffffff81185d5a>] path_walk+0x6a/0xe0
[<ffffffff81185f2b>] do_path_lookup+0x5b/0xa0
[<ffffffff81186b97>] user_path_at+0x57/0xa0
[<ffffffff81042c24>] ? __do_page_fault+0x1e4/0x480
[<ffffffff81182fc5>] ? putname+0x35/0x50
[<ffffffff81186ba2>] ? user_path_at+0x62/0xa0
[<ffffffff8117b786>] vfs_fstatat+0x46/0x80
[<ffffffff81195630>] ? mntput_no_expire+0x30/0x110
[<ffffffff8117b8eb>] vfs_stat+0x1b/0x20
[<ffffffff8117b914>] sys_newstat+0x24/0x50
[<ffffffff814f23fe>] ? do_page_fault+0x3e/0xa0
[<ffffffff814ef7b5>] ? page_fault+0x25/0x30
[<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
INFO: task smbd:2586 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
smbd D 0000000000000000 0 2586 1998 0x00000000
ffff88003b1e7bb8 0000000000000082 0000000000000088 0000000000000286
ffff88003b619380 000000000000008c ffff88003b1e7b98 ffff88003b619380
ffff880037a59af8 ffff88003b1e7fd8 000000000000f4e8 ffff880037a59af8
Call Trace:
[<ffffffff8141ccde>] ? release_sock+0xce/0xe0
[<ffffffff814ee01e>] __mutex_lock_slowpath+0x13e/0x180
[<ffffffff814edebb>] mutex_lock+0x2b/0x50
[<ffffffff811848af>] do_lookup+0xef/0x1e0
[<ffffffff811851d4>] __link_path_walk+0x734/0x1030
[<ffffffff81183795>] ? path_to_nameidata+0x45/0x60
[<ffffffff81185d5a>] path_walk+0x6a/0xe0
[<ffffffff81185f2b>] do_path_lookup+0x5b/0xa0
[<ffffffff81186b97>] user_path_at+0x57/0xa0
[<ffffffff81042c24>] ? __do_page_fault+0x1e4/0x480
[<ffffffff81182fc5>] ? putname+0x35/0x50
[<ffffffff81186ba2>] ? user_path_at+0x62/0xa0
[<ffffffff8117b786>] vfs_fstatat+0x46/0x80
[<ffffffff81195630>] ? mntput_no_expire+0x30/0x110
[<ffffffff8117b8eb>] vfs_stat+0x1b/0x20
[<ffffffff8117b914>] sys_newstat+0x24/0x50
[<ffffffff814f23fe>] ? do_page_fault+0x3e/0xa0
[<ffffffff814ef7b5>] ? page_fault+0x25/0x30
[<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
INFO: task smbd:2587 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
smbd D 0000000000000001 0 2587 1998 0x00000000
ffff88003d5efbb8 0000000000000086 0000000000000000 0000000000000286
ffff880037047cc0 000000000000007e ffff88003d5efb98 ffff880037047cc0
ffff88003b04f038 ffff88003d5effd8 000000000000f4e8 ffff88003b04f038
Call Trace:
[<ffffffff814ee01e>] __mutex_lock_slowpath+0x13e/0x180
[<ffffffff814edebb>] mutex_lock+0x2b/0x50
[<ffffffff811848af>] do_lookup+0xef/0x1e0
[<ffffffff811851d4>] __link_path_walk+0x734/0x1030
[<ffffffff81183795>] ? path_to_nameidata+0x45/0x60
[<ffffffff81185d5a>] path_walk+0x6a/0xe0
[<ffffffff81185f2b>] do_path_lookup+0x5b/0xa0
[<ffffffff81186b97>] user_path_at+0x57/0xa0
[<ffffffff81042c24>] ? __do_page_fault+0x1e4/0x480
[<ffffffff81182fc5>] ? putname+0x35/0x50
[<ffffffff81186ba2>] ? user_path_at+0x62/0xa0
[<ffffffff8117b786>] vfs_fstatat+0x46/0x80
[<ffffffff81195630>] ? mntput_no_expire+0x30/0x110
[<ffffffff8117b8eb>] vfs_stat+0x1b/0x20
[<ffffffff8117b914>] sys_newstat+0x24/0x50
[<ffffffff814f23fe>] ? do_page_fault+0x3e/0xa0
[<ffffffff814ef7b5>] ? page_fault+0x25/0x30
[<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
INFO: task smbd:2588 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
smbd D 0000000000000001 0 2588 1998 0x00000000
ffff88003b093bb8 0000000000000082 0000000000000000 0000000000000286
ffff88003a9e69c0 000000000000007e ffff88003b093b98 ffff88003a9e69c0
ffff8800376f9b38 ffff88003b093fd8 000000000000f4e8 ffff8800376f9b38
Call Trace:
[<ffffffff814ee01e>] __mutex_lock_slowpath+0x13e/0x180
[<ffffffff814edebb>] mutex_lock+0x2b/0x50
[<ffffffff811848af>] do_lookup+0xef/0x1e0
[<ffffffff811851d4>] __link_path_walk+0x734/0x1030
[<ffffffff81183795>] ? path_to_nameidata+0x45/0x60
[<ffffffff81185d5a>] path_walk+0x6a/0xe0
[<ffffffff81185f2b>] do_path_lookup+0x5b/0xa0
[<ffffffff81186b97>] user_path_at+0x57/0xa0
[<ffffffff81042c24>] ? __do_page_fault+0x1e4/0x480
[<ffffffff81182fc5>] ? putname+0x35/0x50
[<ffffffff81186ba2>] ? user_path_at+0x62/0xa0
[<ffffffff8117b786>] vfs_fstatat+0x46/0x80
[<ffffffff81195630>] ? mntput_no_expire+0x30/0x110
[<ffffffff8117b8eb>] vfs_stat+0x1b/0x20
[<ffffffff8117b914>] sys_newstat+0x24/0x50
[<ffffffff814f23fe>] ? do_page_fault+0x3e/0xa0
[<ffffffff814ef7b5>] ? page_fault+0x25/0x30
[<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
INFO: task smbd:2589 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
smbd D 0000000000000001 0 2589 1998 0x00000000
ffff8800374c1bb8 0000000000000086 0000000000000088 0000000000000286
ffff880037bf50c0 000000000000008c ffff8800374c1b98 ffff880037bf50c0
ffff88003b18a678 ffff8800374c1fd8 000000000000f4e8 ffff88003b18a678
Call Trace:
[<ffffffff8141ccde>] ? release_sock+0xce/0xe0
[<ffffffff814ee01e>] __mutex_lock_slowpath+0x13e/0x180
[<ffffffff814edebb>] mutex_lock+0x2b/0x50
[<ffffffff811848af>] do_lookup+0xef/0x1e0
[<ffffffff811851d4>] __link_path_walk+0x734/0x1030
[<ffffffff81183795>] ? path_to_nameidata+0x45/0x60
[<ffffffff81185d5a>] path_walk+0x6a/0xe0
[<ffffffff81185f2b>] do_path_lookup+0x5b/0xa0
[<ffffffff81186b97>] user_path_at+0x57/0xa0
[<ffffffff81042c24>] ? __do_page_fault+0x1e4/0x480
[<ffffffff81182fc5>] ? putname+0x35/0x50
[<ffffffff81186ba2>] ? user_path_at+0x62/0xa0
[<ffffffff8117b786>] vfs_fstatat+0x46/0x80
[<ffffffff81195630>] ? mntput_no_expire+0x30/0x110
[<ffffffff8117b8eb>] vfs_stat+0x1b/0x20
[<ffffffff8117b914>] sys_newstat+0x24/0x50
[<ffffffff814f23fe>] ? do_page_fault+0x3e/0xa0
[<ffffffff814ef7b5>] ? page_fault+0x25/0x30
[<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
INFO: task glusterfsd:2322 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
glusterfsd D 0000000000000001 0 2322 1 0x00000000
ffff88003c70da28 0000000000000082 ffff880000020f58 ffff88003b8615a8
ffff88003c70d9a8 ffffffff81467515 ffff88003b8615a8 ffff88003b8615a8
ffff88003b18baf8 ffff88003c70dfd8 000000000000f4e8 ffff88003b18baf8
Call Trace:
[<ffffffff81467515>] ? ip_local_out+0x25/0x30
[<ffffffff81090d7e>] ? prepare_to_wait+0x4e/0x80
[<ffffffffa00de09d>] do_get_write_access+0x29d/0x520 [jbd2]
[<ffffffff81090ad0>] ? wake_bit_function+0x0/0x50
[<ffffffffa00de471>] jbd2_journal_get_write_access+0x31/0x50 [jbd2]
[<ffffffffa0132b78>] __ext4_journal_get_write_access+0x38/0x80 [ext4]
[<ffffffffa010e253>] ext4_reserve_inode_write+0x73/0xa0 [ext4]
[<ffffffffa010e2cc>] ext4_mark_inode_dirty+0x4c/0x1d0 [ext4]
[<ffffffffa0141ccd>] ? ext4_init_security+0x2d/0x296 [ext4]
[<ffffffffa012d73e>] ext4_ext_tree_init+0x3e/0x60 [ext4]
[<ffffffffa010a500>] ext4_new_inode+0x1150/0x1190 [ext4]
[<ffffffffa0117c90>] ext4_create+0xc0/0x150 [ext4]
[<ffffffff811822a3>] ? generic_permission+0x23/0xb0
[<ffffffff81183c84>] vfs_create+0xb4/0xe0
[<ffffffff8118788f>] do_filp_open+0xb2f/0xd60
[<ffffffff81193482>] ? alloc_fd+0x92/0x160
[<ffffffff81173a49>] do_sys_open+0x69/0x140
[<ffffffff8119ad11>] ? sys_lgetxattr+0x61/0x80
[<ffffffff81173b60>] sys_open+0x20/0x30
[<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
INFO: task smbd:2614 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
smbd D 0000000000000001 0 2614 1998 0x00000000
ffff88003c63dbb8 0000000000000086 0000000000000000 0000000000000286
ffff88003b6014c0 000000000000007c ffff88003c63db98 ffff88003b6014c0
ffff88003a913ab8 ffff88003c63dfd8 000000000000f4e8 ffff88003a913ab8
Call Trace:
[<ffffffff814ee01e>] __mutex_lock_slowpath+0x13e/0x180
[<ffffffff8118e957>] ? __d_lookup+0xa7/0x150
[<ffffffff814edebb>] mutex_lock+0x2b/0x50
[<ffffffff811848af>] do_lookup+0xef/0x1e0
[<ffffffff811851d4>] __link_path_walk+0x734/0x1030
[<ffffffff81183795>] ? path_to_nameidata+0x45/0x60
[<ffffffff81185d5a>] path_walk+0x6a/0xe0
[<ffffffff81185f2b>] do_path_lookup+0x5b/0xa0
[<ffffffff81186b97>] user_path_at+0x57/0xa0
[<ffffffff81042c24>] ? __do_page_fault+0x1e4/0x480
[<ffffffff81182fc5>] ? putname+0x35/0x50
[<ffffffff81186ba2>] ? user_path_at+0x62/0xa0
[<ffffffff8117b786>] vfs_fstatat+0x46/0x80
[<ffffffff81195630>] ? mntput_no_expire+0x30/0x110
[<ffffffff8117b8eb>] vfs_stat+0x1b/0x20
[<ffffffff8117b914>] sys_newstat+0x24/0x50
[<ffffffff814f23fe>] ? do_page_fault+0x3e/0xa0
[<ffffffff814ef7b5>] ? page_fault+0x25/0x30
[<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
INFO: task smbd:2615 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
smbd D 0000000000000001 0 2615 1998 0x00000000
ffff88003d9bbbb8 0000000000000086 0000000000000000 0000000000000286
ffff88003720f580 000000000000008a ffff88003d9bbb98 ffff88003720f580
ffff880037a59af8 ffff88003d9bbfd8 000000000000f4e8 ffff880037a59af8
Call Trace:
[<ffffffff814ee01e>] __mutex_lock_slowpath+0x13e/0x180
[<ffffffff814edebb>] mutex_lock+0x2b/0x50
[<ffffffff811848af>] do_lookup+0xef/0x1e0
[<ffffffff811851d4>] __link_path_walk+0x734/0x1030
[<ffffffff81183795>] ? path_to_nameidata+0x45/0x60
[<ffffffff81185d5a>] path_walk+0x6a/0xe0
[<ffffffff81185f2b>] do_path_lookup+0x5b/0xa0
[<ffffffff81186b97>] user_path_at+0x57/0xa0
[<ffffffff81042c24>] ? __do_page_fault+0x1e4/0x480
[<ffffffff81182fc5>] ? putname+0x35/0x50
[<ffffffff81186ba2>] ? user_path_at+0x62/0xa0
[<ffffffff8117b786>] vfs_fstatat+0x46/0x80
[<ffffffff81195630>] ? mntput_no_expire+0x30/0x110
[<ffffffff8117b8eb>] vfs_stat+0x1b/0x20
[<ffffffff8117b914>] sys_newstat+0x24/0x50
[<ffffffff814f23fe>] ? do_page_fault+0x3e/0xa0
[<ffffffff814ef7b5>] ? page_fault+0x25/0x30
[<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
[root@gluster00 ~]#
[root@gluster00 ~]# mount
/dev/sda3 on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
/dev/sda1 on /boot type ext4 (rw)
/dev/sdb1 on /GLUSTER00 type ext4 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
gluster00:/mm-volume on /mmvolume type fuse.glusterfs (rw,allow_other,default_permissions,max_read=131072)
[root@gluster00 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 18G 2.0G 15G 13% /
tmpfs 499M 0 499M 0% /dev/shm
/dev/sda1 248M 103M 133M 44% /boot
/dev/sdb1 960G 200M 911G 1% /GLUSTER00
gluster00:/mm-volume 2.4T 397M 2.2T 1% /mmvolume
[root@gluster00 ~]# gluster -V
glusterfs 3.2.5 built on Dec 7 2011 18:41:39
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GlusterFS under the terms of the GNU General Public License.
[root@gluster00 ~]# uname -a
Linux gluster00.prod 2.6.32-220.4.1.el6.x86_64 #1 SMP Tue Jan 24 02:13:44 GMT 2012 x86_64 x86_64 x86_64 GNU/Linux
[root@gluster00 ~]# smbd -V
Version 3.5.10-114.el6
THE MACHINE RUNS CENTOS 6.2
INFO: task glusterfsd:1878 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
glusterfsd D 0000000000000000 0 1878 1 0x00000000
ffff88003b683c68 0000000000000086 0000000000000000 000000010000000b
ffff88003b683d18 ffff88003b683c98 ffff88003b683bf8 ffffffff81090957
ffff880037a9e6b8 ffff88003b683fd8 000000000000f4e8 ffff880037a9e6b8
Call Trace:
[<ffffffff81090957>] ? bit_waitqueue+0x17/0xd0
[<ffffffffa00de09d>] do_get_write_access+0x29d/0x520 [jbd2]
[<ffffffff81090ad0>] ? wake_bit_function+0x0/0x50
[<ffffffffa013293b>] ? __ext4_handle_dirty_metadata+0x7b/0x100 [ext4]
[<ffffffffa00de471>] jbd2_journal_get_write_access+0x31/0x50 [jbd2]
[<ffffffffa0132b78>] __ext4_journal_get_write_access+0x38/0x80 [ext4]
[<ffffffffa0114fb9>] ext4_orphan_add+0xb9/0x1f0 [ext4]
[<ffffffffa01146b9>] ? ext4_delete_entry+0xf9/0x110 [ext4]
[<ffffffffa0116a8b>] ext4_unlink+0x29b/0x2b0 [ext4]
[<ffffffff81183edf>] vfs_unlink+0x9f/0xe0
[<ffffffff81182c2a>] ? lookup_hash+0x3a/0x50
[<ffffffff81186463>] do_unlinkat+0x183/0x1c0
[<ffffffff81183601>] ? path_put+0x31/0x40
[<ffffffff8119ad11>] ? sys_lgetxattr+0x61/0x80
[<ffffffff811864b6>] sys_unlink+0x16/0x20
[<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
INFO: task glusterfsd:1878 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
glusterfsd D 0000000000000000 0 1878 1 0x00000000
ffff88003b683c68 0000000000000086 0000000000000000 000000010000000b
ffff88003b683d18 ffff88003b683c98 ffff88003b683bf8 ffffffff81090957
ffff880037a9e6b8 ffff88003b683fd8 000000000000f4e8 ffff880037a9e6b8
Call Trace:
[<ffffffff81090957>] ? bit_waitqueue+0x17/0xd0
[<ffffffffa00de09d>] do_get_write_access+0x29d/0x520 [jbd2]
[<ffffffff81090ad0>] ? wake_bit_function+0x0/0x50
[<ffffffffa013293b>] ? __ext4_handle_dirty_metadata+0x7b/0x100 [ext4]
[<ffffffffa00de471>] jbd2_journal_get_write_access+0x31/0x50 [jbd2]
[<ffffffffa0132b78>] __ext4_journal_get_write_access+0x38/0x80 [ext4]
[<ffffffffa0114fb9>] ext4_orphan_add+0xb9/0x1f0 [ext4]
[<ffffffffa01146b9>] ? ext4_delete_entry+0xf9/0x110 [ext4]
[<ffffffffa0116a8b>] ext4_unlink+0x29b/0x2b0 [ext4]
[<ffffffff81183edf>] vfs_unlink+0x9f/0xe0
[<ffffffff81182c2a>] ? lookup_hash+0x3a/0x50
[<ffffffff81186463>] do_unlinkat+0x183/0x1c0
[<ffffffff81183601>] ? path_put+0x31/0x40
[<ffffffff8119ad11>] ? sys_lgetxattr+0x61/0x80
[<ffffffff811864b6>] sys_unlink+0x16/0x20
[<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
INFO: task glusterfsd:1878 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
glusterfsd D 0000000000000000 0 1878 1 0x00000000
ffff88003b683c68 0000000000000086 0000000000000000 000000010000000b
ffff88003b683d18 ffff88003b683c98 ffff88003b683bf8 ffffffff81090957
ffff880037a9e6b8 ffff88003b683fd8 000000000000f4e8 ffff880037a9e6b8
Call Trace:
[<ffffffff81090957>] ? bit_waitqueue+0x17/0xd0
[<ffffffffa00de09d>] do_get_write_access+0x29d/0x520 [jbd2]
[<ffffffff81090ad0>] ? wake_bit_function+0x0/0x50
[<ffffffffa013293b>] ? __ext4_handle_dirty_metadata+0x7b/0x100 [ext4]
[<ffffffffa00de471>] jbd2_journal_get_write_access+0x31/0x50 [jbd2]
[<ffffffffa0132b78>] __ext4_journal_get_write_access+0x38/0x80 [ext4]
[<ffffffffa0114fb9>] ext4_orphan_add+0xb9/0x1f0 [ext4]
[<ffffffffa01146b9>] ? ext4_delete_entry+0xf9/0x110 [ext4]
[<ffffffffa0116a8b>] ext4_unlink+0x29b/0x2b0 [ext4]
[<ffffffff81183edf>] vfs_unlink+0x9f/0xe0
[<ffffffff81182c2a>] ? lookup_hash+0x3a/0x50
[<ffffffff81186463>] do_unlinkat+0x183/0x1c0
[<ffffffff81183601>] ? path_put+0x31/0x40
[<ffffffff8119ad11>] ? sys_lgetxattr+0x61/0x80
[<ffffffff811864b6>] sys_unlink+0x16/0x20
[<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
INFO: task glusterfsd:1878 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
glusterfsd D 0000000000000000 0 1878 1 0x00000000
ffff88003b683c68 0000000000000086 0000000000000000 000000010000000b
ffff88003b683d18 ffff88003b683c98 ffff88003b683bf8 ffffffff81090957
ffff880037a9e6b8 ffff88003b683fd8 000000000000f4e8 ffff880037a9e6b8
Call Trace:
[<ffffffff81090957>] ? bit_waitqueue+0x17/0xd0
[<ffffffffa00de09d>] do_get_write_access+0x29d/0x520 [jbd2]
[<ffffffff81090ad0>] ? wake_bit_function+0x0/0x50
[<ffffffffa013293b>] ? __ext4_handle_dirty_metadata+0x7b/0x100 [ext4]
[<ffffffffa00de471>] jbd2_journal_get_write_access+0x31/0x50 [jbd2]
[<ffffffffa0132b78>] __ext4_journal_get_write_access+0x38/0x80 [ext4]
[<ffffffffa0114fb9>] ext4_orphan_add+0xb9/0x1f0 [ext4]
[<ffffffffa01146b9>] ? ext4_delete_entry+0xf9/0x110 [ext4]
[<ffffffffa0116a8b>] ext4_unlink+0x29b/0x2b0 [ext4]
[<ffffffff81183edf>] vfs_unlink+0x9f/0xe0
[<ffffffff81182c2a>] ? lookup_hash+0x3a/0x50
[<ffffffff81186463>] do_unlinkat+0x183/0x1c0
[<ffffffff81183601>] ? path_put+0x31/0x40
[<ffffffff8119ad11>] ? sys_lgetxattr+0x61/0x80
[<ffffffff811864b6>] sys_unlink+0x16/0x20
[<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
INFO: task glusterfsd:1754 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
glusterfsd D 0000000000000000 0 1754 1 0x00000000
ffff88003b479c48 0000000000000086 ffff880000000001 000000013712d580
0000000000000000 ffff88003f810080 ffff88003faacc00 ffff88003f8224c0
ffff88003712db38 ffff88003b479fd8 000000000000f4e8 ffff88003712db38
Call Trace:
[<ffffffff81090d7e>] ? prepare_to_wait+0x4e/0x80
[<ffffffffa00de09d>] do_get_write_access+0x29d/0x520 [jbd2]
[<ffffffff81090ad0>] ? wake_bit_function+0x0/0x50
[<ffffffffa00de471>] jbd2_journal_get_write_access+0x31/0x50 [jbd2]
[<ffffffffa0132b78>] __ext4_journal_get_write_access+0x38/0x80 [ext4]
[<ffffffffa010e253>] ext4_reserve_inode_write+0x73/0xa0 [ext4]
[<ffffffffa0107158>] ? call_filldir+0x78/0xe0 [ext4]
[<ffffffffa010e2cc>] ext4_mark_inode_dirty+0x4c/0x1d0 [ext4]
[<ffffffffa010e5c0>] ext4_dirty_inode+0x40/0x60 [ext4]
[<ffffffff8119fceb>] __mark_inode_dirty+0x3b/0x160
[<ffffffff8119040d>] touch_atime+0x12d/0x170
[<ffffffff8118a290>] ? filldir+0x0/0xe0
[<ffffffff8118a526>] vfs_readdir+0xd6/0xe0
[<ffffffff8118a699>] sys_getdents+0x89/0xf0
[<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
INFO: task glusterfsd:1754 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
glusterfsd D 0000000000000000 0 1754 1 0x00000000
ffff88003b479c48 0000000000000086 ffff880000000001 000000013712d580
0000000000000000 ffff88003f810080 ffff88003faacc00 ffff88003f8224c0
ffff88003712db38 ffff88003b479fd8 000000000000f4e8 ffff88003712db38
Call Trace:
[<ffffffff81090d7e>] ? prepare_to_wait+0x4e/0x80
[<ffffffffa00de09d>] do_get_write_access+0x29d/0x520 [jbd2]
[<ffffffff81090ad0>] ? wake_bit_function+0x0/0x50
[<ffffffffa00de471>] jbd2_journal_get_write_access+0x31/0x50 [jbd2]
[<ffffffffa0132b78>] __ext4_journal_get_write_access+0x38/0x80 [ext4]
[<ffffffffa010e253>] ext4_reserve_inode_write+0x73/0xa0 [ext4]
[<ffffffffa0107158>] ? call_filldir+0x78/0xe0 [ext4]
[<ffffffffa010e2cc>] ext4_mark_inode_dirty+0x4c/0x1d0 [ext4]
[<ffffffffa010e5c0>] ext4_dirty_inode+0x40/0x60 [ext4]
[<ffffffff8119fceb>] __mark_inode_dirty+0x3b/0x160
[<ffffffff8119040d>] touch_atime+0x12d/0x170
[<ffffffff8118a290>] ? filldir+0x0/0xe0
[<ffffffff8118a526>] vfs_readdir+0xd6/0xe0
[<ffffffff8118a699>] sys_getdents+0x89/0xf0
[<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
INFO: task glusterfsd:1754 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
glusterfsd D 0000000000000000 0 1754 1 0x00000000
ffff88003b479c48 0000000000000086 ffff880000000001 000000013712d580
0000000000000000 ffff88003f810080 ffff88003faacc00 ffff88003f8224c0
ffff88003712db38 ffff88003b479fd8 000000000000f4e8 ffff88003712db38
Call Trace:
[<ffffffff81090d7e>] ? prepare_to_wait+0x4e/0x80
[<ffffffffa00de09d>] do_get_write_access+0x29d/0x520 [jbd2]
[<ffffffff81090ad0>] ? wake_bit_function+0x0/0x50
[<ffffffffa00de471>] jbd2_journal_get_write_access+0x31/0x50 [jbd2]
[<ffffffffa0132b78>] __ext4_journal_get_write_access+0x38/0x80 [ext4]
[<ffffffffa010e253>] ext4_reserve_inode_write+0x73/0xa0 [ext4]
[<ffffffffa0107158>] ? call_filldir+0x78/0xe0 [ext4]
[<ffffffffa010e2cc>] ext4_mark_inode_dirty+0x4c/0x1d0 [ext4]
[<ffffffffa010e5c0>] ext4_dirty_inode+0x40/0x60 [ext4]
[<ffffffff8119fceb>] __mark_inode_dirty+0x3b/0x160
[<ffffffff8119040d>] touch_atime+0x12d/0x170
[<ffffffff8118a290>] ? filldir+0x0/0xe0
[<ffffffff8118a526>] vfs_readdir+0xd6/0xe0
[<ffffffff8118a699>] sys_getdents+0x89/0xf0
[<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
INFO: task glusterfsd:1878 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
glusterfsd D 0000000000000000 0 1878 1 0x00000000
ffff88003b683c48 0000000000000086 0000000000000000 00000001810949db
0000000000000000 0000000000000000 ffff88003b683cb0 0000000000000000
ffff880037a9e6b8 ffff88003b683fd8 000000000000f4e8 ffff880037a9e6b8
Call Trace:
[<ffffffffa00de09d>] do_get_write_access+0x29d/0x520 [jbd2]
[<ffffffff81090ad0>] ? wake_bit_function+0x0/0x50
[<ffffffffa00de471>] jbd2_journal_get_write_access+0x31/0x50 [jbd2]
[<ffffffffa0132b78>] __ext4_journal_get_write_access+0x38/0x80 [ext4]
[<ffffffffa010e253>] ext4_reserve_inode_write+0x73/0xa0 [ext4]
[<ffffffffa0107158>] ? call_filldir+0x78/0xe0 [ext4]
[<ffffffffa010e2cc>] ext4_mark_inode_dirty+0x4c/0x1d0 [ext4]
[<ffffffffa010e5c0>] ext4_dirty_inode+0x40/0x60 [ext4]
[<ffffffff8119fceb>] __mark_inode_dirty+0x3b/0x160
[<ffffffff8119040d>] touch_atime+0x12d/0x170
[<ffffffff8118a290>] ? filldir+0x0/0xe0
[<ffffffff8118a526>] vfs_readdir+0xd6/0xe0
[<ffffffff8118a699>] sys_getdents+0x89/0xf0
[<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
INFO: task glusterfsd:1878 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
glusterfsd D 0000000000000000 0 1878 1 0x00000000
ffff88003b683c48 0000000000000086 0000000000000000 00000001810949db
0000000000000000 0000000000000000 ffff88003b683cb0 0000000000000000
ffff880037a9e6b8 ffff88003b683fd8 000000000000f4e8 ffff880037a9e6b8
Call Trace:
[<ffffffffa00de09d>] do_get_write_access+0x29d/0x520 [jbd2]
[<ffffffff81090ad0>] ? wake_bit_function+0x0/0x50
[<ffffffffa00de471>] jbd2_journal_get_write_access+0x31/0x50 [jbd2]
[<ffffffffa0132b78>] __ext4_journal_get_write_access+0x38/0x80 [ext4]
[<ffffffffa010e253>] ext4_reserve_inode_write+0x73/0xa0 [ext4]
[<ffffffffa0107158>] ? call_filldir+0x78/0xe0 [ext4]
[<ffffffffa010e2cc>] ext4_mark_inode_dirty+0x4c/0x1d0 [ext4]
[<ffffffffa010e5c0>] ext4_dirty_inode+0x40/0x60 [ext4]
[<ffffffff8119fceb>] __mark_inode_dirty+0x3b/0x160
[<ffffffff8119040d>] touch_atime+0x12d/0x170
[<ffffffff8118a290>] ? filldir+0x0/0xe0
[<ffffffff8118a526>] vfs_readdir+0xd6/0xe0
[<ffffffff8118a699>] sys_getdents+0x89/0xf0
[<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
INFO: task glusterfsd:1878 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
glusterfsd D 0000000000000000 0 1878 1 0x00000000
ffff88003b683c48 0000000000000086 0000000000000000 00000001810949db
0000000000000000 0000000000000000 ffff88003b683cb0 0000000000000000
ffff880037a9e6b8 ffff88003b683fd8 000000000000f4e8 ffff880037a9e6b8
Call Trace:
[<ffffffffa00de09d>] do_get_write_access+0x29d/0x520 [jbd2]
[<ffffffff81090ad0>] ? wake_bit_function+0x0/0x50
[<ffffffffa00de471>] jbd2_journal_get_write_access+0x31/0x50 [jbd2]
[<ffffffffa0132b78>] __ext4_journal_get_write_access+0x38/0x80 [ext4]
[<ffffffffa010e253>] ext4_reserve_inode_write+0x73/0xa0 [ext4]
[<ffffffffa0107158>] ? call_filldir+0x78/0xe0 [ext4]
[<ffffffffa010e2cc>] ext4_mark_inode_dirty+0x4c/0x1d0 [ext4]
[<ffffffffa010e5c0>] ext4_dirty_inode+0x40/0x60 [ext4]
[<ffffffff8119fceb>] __mark_inode_dirty+0x3b/0x160
[<ffffffff8119040d>] touch_atime+0x12d/0x170
[<ffffffff8118a290>] ? filldir+0x0/0xe0
[<ffffffff8118a526>] vfs_readdir+0xd6/0xe0
[<ffffffff8118a699>] sys_getdents+0x89/0xf0
[<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
[root@gluster01 ~]#
[root@gluster01 ~]# mount
/dev/sda3 on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
/dev/sda1 on /boot type ext4 (rw)
/dev/sdb1 on /GLUSTER01 type ext4 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
[root@gluster01 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 18G 1.8G 15G 11% /
tmpfs 499M 0 499M 0% /dev/shm
/dev/sda1 248M 103M 133M 44% /boot
/dev/sdb1 1.4T 198M 1.4T 1% /GLUSTER01
[root@gluster01 ~]# gluster -V
glusterfs 3.2.5 built on Dec 7 2011 18:41:39
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GlusterFS under the terms of the GNU General Public License.
[root@gluster01 ~]# uname -a
Linux gluster01.prod 2.6.32-220.4.1.el6.x86_64 #1 SMP Tue Jan 24 02:13:44 GMT 2012 x86_64 x86_64 x86_64 GNU/Linux
THE MACHINE RUNS CENTOS 6.2
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Ext4 Kernel Crash
2012-01-29 17:50 Ext4 Kernel Crash Nanakos Chrysostomos
@ 2012-01-29 22:03 ` Jesper Juhl
2012-01-31 8:34 ` Nanakos V. Chrysostomos
2012-01-30 17:26 ` Ted Ts'o
1 sibling, 1 reply; 5+ messages in thread
From: Jesper Juhl @ 2012-01-29 22:03 UTC (permalink / raw)
To: Nanakos Chrysostomos
Cc: linux-kernel, linux-ext4, Andreas Dilger, Theodore Ts'o
On Sun, 29 Jan 2012, Nanakos Chrysostomos wrote:
> Dear Linux Kernel Developers,
> I am writing to report a kernel or ext4 bug.
> I have two servers populated with 2 LUNs from the main storage facility,
> each server has one LUN which is about 1.2TB. On top of that I am
> utilizing glusterfs and samba to mount the 2.4TB created space to a
> Windows server and other Linux machines (via glusterfs native client).
> Yesterday I tried to copy to the new storage glusterfs space about 788GB
> and in the morning I got from both machines the attached kernel messages.
> I have to mention that the first machine runs samba and mounts the
> allocated glusterfs space with the help of FUSE glusterfs driver. The
> second machine doesn't mount/export anything and simply runs the glusterfs
> service.
> I have to also add that similar problems have happened to another system
> which is used as a TIF to JP2 converter. The conversion path starts from
> the conversion of a pdf file to png and then to jp2. As you can imagine
> this process produces hundreds of thousands of files. Similarly the
> machine crashes and I get similar kernel messages concerning ext4. I can't
> give you more information on that because the machine crashes, I have to
> take a photo and send it back to you next time.
> What do you believe about the kernel messages? Is this a ext4 bug when
> thousands of files are created? How can I help resolve that?
>
> Please Cc me because I am not in the list.
>
> With kind regards,
> Chris.
>
1. Adding ext4 people to CC.
2. Took a quick look (without knowing much about ext4, mind you), and I
noticed something that is, probably unrelated, but just in case it is not:
In ext4_new_inode(), at the 'got:' label we have this code:
[...]
got:
/* We may have to initialize the block bitmap if it isn't already */
if (EXT4_HAS_RO_COMPAT_FEATURE(sb, EXT4_FEATURE_RO_COMPAT_GDT_CSUM) &&
gdp->bg_flags & cpu_to_le16(EXT4_BG_BLOCK_UNINIT)) {
struct buffer_head *block_bitmap_bh;
block_bitmap_bh = ext4_read_block_bitmap(sb, group);
BUFFER_TRACE(block_bitmap_bh, "get block bitmap access");
err = ext4_journal_get_write_access(handle, block_bitmap_bh);
if (err) {
brelse(block_bitmap_bh);
goto fail;
}
BUFFER_TRACE(block_bitmap_bh, "dirty block bitmap");
err = ext4_handle_dirty_metadata(handle, NULL, block_bitmap_bh);
[...]
In the above, ext4_read_block_bitmap() may return NULL. If it does, then
the calls to ext4_journal_get_write_access() and
ext4_handle_dirty_metadata() are not safe, since both those functions may
call other functions that may end up dereferencing their argument (the
ext4_read_block_bitmap() return value).
Whether or not that will ever actually happen and whether or not it's
related to your problem, I don't know.
But I thought I'd share the info.
>
> PS: Are there any ext4 kernel optimization's for avoiding that or
> narrowing down the problem if this is not so obvious?
>
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> glusterfsd D 0000000000000000 0 2070 1 0x00000000
> ffff88003daa5c68 0000000000000082 0000000000000000 000000010000000b
> ffff88003daa5d18 ffff88003daa5c98 ffff88003daa5bf8 ffffffff81090957
> ffff88003a912638 ffff88003daa5fd8 000000000000f4e8 ffff88003a912638
> Call Trace:
> [<ffffffff81090957>] ? bit_waitqueue+0x17/0xd0
> [<ffffffffa00de09d>] do_get_write_access+0x29d/0x520 [jbd2]
> [<ffffffff81090ad0>] ? wake_bit_function+0x0/0x50
> [<ffffffffa013293b>] ? __ext4_handle_dirty_metadata+0x7b/0x100 [ext4]
> [<ffffffffa00de471>] jbd2_journal_get_write_access+0x31/0x50 [jbd2]
> [<ffffffffa0132b78>] __ext4_journal_get_write_access+0x38/0x80 [ext4]
> [<ffffffffa0114fb9>] ext4_orphan_add+0xb9/0x1f0 [ext4]
> [<ffffffffa01146b9>] ? ext4_delete_entry+0xf9/0x110 [ext4]
> [<ffffffffa0116a8b>] ext4_unlink+0x29b/0x2b0 [ext4]
> [<ffffffff81183edf>] vfs_unlink+0x9f/0xe0
> [<ffffffff81182c2a>] ? lookup_hash+0x3a/0x50
> [<ffffffff81186463>] do_unlinkat+0x183/0x1c0
> [<ffffffff81183601>] ? path_put+0x31/0x40
> [<ffffffff8119ad11>] ? sys_lgetxattr+0x61/0x80
> [<ffffffff811864b6>] sys_unlink+0x16/0x20
> [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
> INFO: task glusterfsd:2322 blocked for more than 120 seconds.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> glusterfsd D 0000000000000000 0 2322 1 0x00000000
> ffff88003c70dc48 0000000000000082 0000000000000000 00000001810949db
> 0000000000000000 0000000000000000 ffff88003c70dcb0 0000000000000000
> ffff88003b18baf8 ffff88003c70dfd8 000000000000f4e8 ffff88003b18baf8
> Call Trace:
> [<ffffffffa00de09d>] do_get_write_access+0x29d/0x520 [jbd2]
> [<ffffffff81090ad0>] ? wake_bit_function+0x0/0x50
> [<ffffffffa00de471>] jbd2_journal_get_write_access+0x31/0x50 [jbd2]
> [<ffffffffa0132b78>] __ext4_journal_get_write_access+0x38/0x80 [ext4]
> [<ffffffffa010e253>] ext4_reserve_inode_write+0x73/0xa0 [ext4]
> [<ffffffffa0107158>] ? call_filldir+0x78/0xe0 [ext4]
> [<ffffffffa010e2cc>] ext4_mark_inode_dirty+0x4c/0x1d0 [ext4]
> [<ffffffffa010e5c0>] ext4_dirty_inode+0x40/0x60 [ext4]
> [<ffffffff8119fceb>] __mark_inode_dirty+0x3b/0x160
> [<ffffffff8119040d>] touch_atime+0x12d/0x170
> [<ffffffff8118a290>] ? filldir+0x0/0xe0
> [<ffffffff8118a526>] vfs_readdir+0xd6/0xe0
> [<ffffffff8118a699>] sys_getdents+0x89/0xf0
> [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
> INFO: task smbd:2585 blocked for more than 120 seconds.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> smbd D 0000000000000000 0 2585 1998 0x00000000
> ffff88003b299bb8 0000000000000082 000000000000007a 0000000000000286
> ffff88003b23b8c0 000000000000007e ffff88003b299b98 ffff88003b23b8c0
> ffff88003a913ab8 ffff88003b299fd8 000000000000f4e8 ffff88003a913ab8
> Call Trace:
> [<ffffffff814ee01e>] __mutex_lock_slowpath+0x13e/0x180
> [<ffffffff8118d11f>] ? d_free+0x3f/0x60
> [<ffffffff814edebb>] mutex_lock+0x2b/0x50
> [<ffffffff811848af>] do_lookup+0xef/0x1e0
> [<ffffffff811851d4>] __link_path_walk+0x734/0x1030
> [<ffffffff81183795>] ? path_to_nameidata+0x45/0x60
> [<ffffffff81185d5a>] path_walk+0x6a/0xe0
> [<ffffffff81185f2b>] do_path_lookup+0x5b/0xa0
> [<ffffffff81186b97>] user_path_at+0x57/0xa0
> [<ffffffff81042c24>] ? __do_page_fault+0x1e4/0x480
> [<ffffffff81182fc5>] ? putname+0x35/0x50
> [<ffffffff81186ba2>] ? user_path_at+0x62/0xa0
> [<ffffffff8117b786>] vfs_fstatat+0x46/0x80
> [<ffffffff81195630>] ? mntput_no_expire+0x30/0x110
> [<ffffffff8117b8eb>] vfs_stat+0x1b/0x20
> [<ffffffff8117b914>] sys_newstat+0x24/0x50
> [<ffffffff814f23fe>] ? do_page_fault+0x3e/0xa0
> [<ffffffff814ef7b5>] ? page_fault+0x25/0x30
> [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
> INFO: task smbd:2586 blocked for more than 120 seconds.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> smbd D 0000000000000000 0 2586 1998 0x00000000
> ffff88003b1e7bb8 0000000000000082 0000000000000088 0000000000000286
> ffff88003b619380 000000000000008c ffff88003b1e7b98 ffff88003b619380
> ffff880037a59af8 ffff88003b1e7fd8 000000000000f4e8 ffff880037a59af8
> Call Trace:
> [<ffffffff8141ccde>] ? release_sock+0xce/0xe0
> [<ffffffff814ee01e>] __mutex_lock_slowpath+0x13e/0x180
> [<ffffffff814edebb>] mutex_lock+0x2b/0x50
> [<ffffffff811848af>] do_lookup+0xef/0x1e0
> [<ffffffff811851d4>] __link_path_walk+0x734/0x1030
> [<ffffffff81183795>] ? path_to_nameidata+0x45/0x60
> [<ffffffff81185d5a>] path_walk+0x6a/0xe0
> [<ffffffff81185f2b>] do_path_lookup+0x5b/0xa0
> [<ffffffff81186b97>] user_path_at+0x57/0xa0
> [<ffffffff81042c24>] ? __do_page_fault+0x1e4/0x480
> [<ffffffff81182fc5>] ? putname+0x35/0x50
> [<ffffffff81186ba2>] ? user_path_at+0x62/0xa0
> [<ffffffff8117b786>] vfs_fstatat+0x46/0x80
> [<ffffffff81195630>] ? mntput_no_expire+0x30/0x110
> [<ffffffff8117b8eb>] vfs_stat+0x1b/0x20
> [<ffffffff8117b914>] sys_newstat+0x24/0x50
> [<ffffffff814f23fe>] ? do_page_fault+0x3e/0xa0
> [<ffffffff814ef7b5>] ? page_fault+0x25/0x30
> [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
> INFO: task smbd:2587 blocked for more than 120 seconds.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> smbd D 0000000000000001 0 2587 1998 0x00000000
> ffff88003d5efbb8 0000000000000086 0000000000000000 0000000000000286
> ffff880037047cc0 000000000000007e ffff88003d5efb98 ffff880037047cc0
> ffff88003b04f038 ffff88003d5effd8 000000000000f4e8 ffff88003b04f038
> Call Trace:
> [<ffffffff814ee01e>] __mutex_lock_slowpath+0x13e/0x180
> [<ffffffff814edebb>] mutex_lock+0x2b/0x50
> [<ffffffff811848af>] do_lookup+0xef/0x1e0
> [<ffffffff811851d4>] __link_path_walk+0x734/0x1030
> [<ffffffff81183795>] ? path_to_nameidata+0x45/0x60
> [<ffffffff81185d5a>] path_walk+0x6a/0xe0
> [<ffffffff81185f2b>] do_path_lookup+0x5b/0xa0
> [<ffffffff81186b97>] user_path_at+0x57/0xa0
> [<ffffffff81042c24>] ? __do_page_fault+0x1e4/0x480
> [<ffffffff81182fc5>] ? putname+0x35/0x50
> [<ffffffff81186ba2>] ? user_path_at+0x62/0xa0
> [<ffffffff8117b786>] vfs_fstatat+0x46/0x80
> [<ffffffff81195630>] ? mntput_no_expire+0x30/0x110
> [<ffffffff8117b8eb>] vfs_stat+0x1b/0x20
> [<ffffffff8117b914>] sys_newstat+0x24/0x50
> [<ffffffff814f23fe>] ? do_page_fault+0x3e/0xa0
> [<ffffffff814ef7b5>] ? page_fault+0x25/0x30
> [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
> INFO: task smbd:2588 blocked for more than 120 seconds.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> smbd D 0000000000000001 0 2588 1998 0x00000000
> ffff88003b093bb8 0000000000000082 0000000000000000 0000000000000286
> ffff88003a9e69c0 000000000000007e ffff88003b093b98 ffff88003a9e69c0
> ffff8800376f9b38 ffff88003b093fd8 000000000000f4e8 ffff8800376f9b38
> Call Trace:
> [<ffffffff814ee01e>] __mutex_lock_slowpath+0x13e/0x180
> [<ffffffff814edebb>] mutex_lock+0x2b/0x50
> [<ffffffff811848af>] do_lookup+0xef/0x1e0
> [<ffffffff811851d4>] __link_path_walk+0x734/0x1030
> [<ffffffff81183795>] ? path_to_nameidata+0x45/0x60
> [<ffffffff81185d5a>] path_walk+0x6a/0xe0
> [<ffffffff81185f2b>] do_path_lookup+0x5b/0xa0
> [<ffffffff81186b97>] user_path_at+0x57/0xa0
> [<ffffffff81042c24>] ? __do_page_fault+0x1e4/0x480
> [<ffffffff81182fc5>] ? putname+0x35/0x50
> [<ffffffff81186ba2>] ? user_path_at+0x62/0xa0
> [<ffffffff8117b786>] vfs_fstatat+0x46/0x80
> [<ffffffff81195630>] ? mntput_no_expire+0x30/0x110
> [<ffffffff8117b8eb>] vfs_stat+0x1b/0x20
> [<ffffffff8117b914>] sys_newstat+0x24/0x50
> [<ffffffff814f23fe>] ? do_page_fault+0x3e/0xa0
> [<ffffffff814ef7b5>] ? page_fault+0x25/0x30
> [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
> INFO: task smbd:2589 blocked for more than 120 seconds.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> smbd D 0000000000000001 0 2589 1998 0x00000000
> ffff8800374c1bb8 0000000000000086 0000000000000088 0000000000000286
> ffff880037bf50c0 000000000000008c ffff8800374c1b98 ffff880037bf50c0
> ffff88003b18a678 ffff8800374c1fd8 000000000000f4e8 ffff88003b18a678
> Call Trace:
> [<ffffffff8141ccde>] ? release_sock+0xce/0xe0
> [<ffffffff814ee01e>] __mutex_lock_slowpath+0x13e/0x180
> [<ffffffff814edebb>] mutex_lock+0x2b/0x50
> [<ffffffff811848af>] do_lookup+0xef/0x1e0
> [<ffffffff811851d4>] __link_path_walk+0x734/0x1030
> [<ffffffff81183795>] ? path_to_nameidata+0x45/0x60
> [<ffffffff81185d5a>] path_walk+0x6a/0xe0
> [<ffffffff81185f2b>] do_path_lookup+0x5b/0xa0
> [<ffffffff81186b97>] user_path_at+0x57/0xa0
> [<ffffffff81042c24>] ? __do_page_fault+0x1e4/0x480
> [<ffffffff81182fc5>] ? putname+0x35/0x50
> [<ffffffff81186ba2>] ? user_path_at+0x62/0xa0
> [<ffffffff8117b786>] vfs_fstatat+0x46/0x80
> [<ffffffff81195630>] ? mntput_no_expire+0x30/0x110
> [<ffffffff8117b8eb>] vfs_stat+0x1b/0x20
> [<ffffffff8117b914>] sys_newstat+0x24/0x50
> [<ffffffff814f23fe>] ? do_page_fault+0x3e/0xa0
> [<ffffffff814ef7b5>] ? page_fault+0x25/0x30
> [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
> INFO: task glusterfsd:2322 blocked for more than 120 seconds.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> glusterfsd D 0000000000000001 0 2322 1 0x00000000
> ffff88003c70da28 0000000000000082 ffff880000020f58 ffff88003b8615a8
> ffff88003c70d9a8 ffffffff81467515 ffff88003b8615a8 ffff88003b8615a8
> ffff88003b18baf8 ffff88003c70dfd8 000000000000f4e8 ffff88003b18baf8
> Call Trace:
> [<ffffffff81467515>] ? ip_local_out+0x25/0x30
> [<ffffffff81090d7e>] ? prepare_to_wait+0x4e/0x80
> [<ffffffffa00de09d>] do_get_write_access+0x29d/0x520 [jbd2]
> [<ffffffff81090ad0>] ? wake_bit_function+0x0/0x50
> [<ffffffffa00de471>] jbd2_journal_get_write_access+0x31/0x50 [jbd2]
> [<ffffffffa0132b78>] __ext4_journal_get_write_access+0x38/0x80 [ext4]
> [<ffffffffa010e253>] ext4_reserve_inode_write+0x73/0xa0 [ext4]
> [<ffffffffa010e2cc>] ext4_mark_inode_dirty+0x4c/0x1d0 [ext4]
> [<ffffffffa0141ccd>] ? ext4_init_security+0x2d/0x296 [ext4]
> [<ffffffffa012d73e>] ext4_ext_tree_init+0x3e/0x60 [ext4]
> [<ffffffffa010a500>] ext4_new_inode+0x1150/0x1190 [ext4]
> [<ffffffffa0117c90>] ext4_create+0xc0/0x150 [ext4]
> [<ffffffff811822a3>] ? generic_permission+0x23/0xb0
> [<ffffffff81183c84>] vfs_create+0xb4/0xe0
> [<ffffffff8118788f>] do_filp_open+0xb2f/0xd60
> [<ffffffff81193482>] ? alloc_fd+0x92/0x160
> [<ffffffff81173a49>] do_sys_open+0x69/0x140
> [<ffffffff8119ad11>] ? sys_lgetxattr+0x61/0x80
> [<ffffffff81173b60>] sys_open+0x20/0x30
> [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
> INFO: task smbd:2614 blocked for more than 120 seconds.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> smbd D 0000000000000001 0 2614 1998 0x00000000
> ffff88003c63dbb8 0000000000000086 0000000000000000 0000000000000286
> ffff88003b6014c0 000000000000007c ffff88003c63db98 ffff88003b6014c0
> ffff88003a913ab8 ffff88003c63dfd8 000000000000f4e8 ffff88003a913ab8
> Call Trace:
> [<ffffffff814ee01e>] __mutex_lock_slowpath+0x13e/0x180
> [<ffffffff8118e957>] ? __d_lookup+0xa7/0x150
> [<ffffffff814edebb>] mutex_lock+0x2b/0x50
> [<ffffffff811848af>] do_lookup+0xef/0x1e0
> [<ffffffff811851d4>] __link_path_walk+0x734/0x1030
> [<ffffffff81183795>] ? path_to_nameidata+0x45/0x60
> [<ffffffff81185d5a>] path_walk+0x6a/0xe0
> [<ffffffff81185f2b>] do_path_lookup+0x5b/0xa0
> [<ffffffff81186b97>] user_path_at+0x57/0xa0
> [<ffffffff81042c24>] ? __do_page_fault+0x1e4/0x480
> [<ffffffff81182fc5>] ? putname+0x35/0x50
> [<ffffffff81186ba2>] ? user_path_at+0x62/0xa0
> [<ffffffff8117b786>] vfs_fstatat+0x46/0x80
> [<ffffffff81195630>] ? mntput_no_expire+0x30/0x110
> [<ffffffff8117b8eb>] vfs_stat+0x1b/0x20
> [<ffffffff8117b914>] sys_newstat+0x24/0x50
> [<ffffffff814f23fe>] ? do_page_fault+0x3e/0xa0
> [<ffffffff814ef7b5>] ? page_fault+0x25/0x30
> [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
> INFO: task smbd:2615 blocked for more than 120 seconds.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> smbd D 0000000000000001 0 2615 1998 0x00000000
> ffff88003d9bbbb8 0000000000000086 0000000000000000 0000000000000286
> ffff88003720f580 000000000000008a ffff88003d9bbb98 ffff88003720f580
> ffff880037a59af8 ffff88003d9bbfd8 000000000000f4e8 ffff880037a59af8
> Call Trace:
> [<ffffffff814ee01e>] __mutex_lock_slowpath+0x13e/0x180
> [<ffffffff814edebb>] mutex_lock+0x2b/0x50
> [<ffffffff811848af>] do_lookup+0xef/0x1e0
> [<ffffffff811851d4>] __link_path_walk+0x734/0x1030
> [<ffffffff81183795>] ? path_to_nameidata+0x45/0x60
> [<ffffffff81185d5a>] path_walk+0x6a/0xe0
> [<ffffffff81185f2b>] do_path_lookup+0x5b/0xa0
> [<ffffffff81186b97>] user_path_at+0x57/0xa0
> [<ffffffff81042c24>] ? __do_page_fault+0x1e4/0x480
> [<ffffffff81182fc5>] ? putname+0x35/0x50
> [<ffffffff81186ba2>] ? user_path_at+0x62/0xa0
> [<ffffffff8117b786>] vfs_fstatat+0x46/0x80
> [<ffffffff81195630>] ? mntput_no_expire+0x30/0x110
> [<ffffffff8117b8eb>] vfs_stat+0x1b/0x20
> [<ffffffff8117b914>] sys_newstat+0x24/0x50
> [<ffffffff814f23fe>] ? do_page_fault+0x3e/0xa0
> [<ffffffff814ef7b5>] ? page_fault+0x25/0x30
> [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
> [root@gluster00 ~]#
>
>
> [root@gluster00 ~]# mount
> /dev/sda3 on / type ext4 (rw)
> proc on /proc type proc (rw)
> sysfs on /sys type sysfs (rw)
> devpts on /dev/pts type devpts (rw,gid=5,mode=620)
> tmpfs on /dev/shm type tmpfs (rw)
> /dev/sda1 on /boot type ext4 (rw)
> /dev/sdb1 on /GLUSTER00 type ext4 (rw)
> none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
> sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
> gluster00:/mm-volume on /mmvolume type fuse.glusterfs (rw,allow_other,default_permissions,max_read=131072)
>
> [root@gluster00 ~]# df -h
> Filesystem Size Used Avail Use% Mounted on
> /dev/sda3 18G 2.0G 15G 13% /
> tmpfs 499M 0 499M 0% /dev/shm
> /dev/sda1 248M 103M 133M 44% /boot
> /dev/sdb1 960G 200M 911G 1% /GLUSTER00
> gluster00:/mm-volume 2.4T 397M 2.2T 1% /mmvolume
>
>
> [root@gluster00 ~]# gluster -V
> glusterfs 3.2.5 built on Dec 7 2011 18:41:39
> Repository revision: git://git.gluster.com/glusterfs.git
> Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com>
> GlusterFS comes with ABSOLUTELY NO WARRANTY.
> You may redistribute copies of GlusterFS under the terms of the GNU General Public License.
>
> [root@gluster00 ~]# uname -a
> Linux gluster00.prod 2.6.32-220.4.1.el6.x86_64 #1 SMP Tue Jan 24 02:13:44 GMT 2012 x86_64 x86_64 x86_64 GNU/Linux
>
> [root@gluster00 ~]# smbd -V
> Version 3.5.10-114.el6
>
>
> THE MACHINE RUNS CENTOS 6.2
>
> INFO: task glusterfsd:1878 blocked for more than 120 seconds.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> glusterfsd D 0000000000000000 0 1878 1 0x00000000
> ffff88003b683c68 0000000000000086 0000000000000000 000000010000000b
> ffff88003b683d18 ffff88003b683c98 ffff88003b683bf8 ffffffff81090957
> ffff880037a9e6b8 ffff88003b683fd8 000000000000f4e8 ffff880037a9e6b8
> Call Trace:
> [<ffffffff81090957>] ? bit_waitqueue+0x17/0xd0
> [<ffffffffa00de09d>] do_get_write_access+0x29d/0x520 [jbd2]
> [<ffffffff81090ad0>] ? wake_bit_function+0x0/0x50
> [<ffffffffa013293b>] ? __ext4_handle_dirty_metadata+0x7b/0x100 [ext4]
> [<ffffffffa00de471>] jbd2_journal_get_write_access+0x31/0x50 [jbd2]
> [<ffffffffa0132b78>] __ext4_journal_get_write_access+0x38/0x80 [ext4]
> [<ffffffffa0114fb9>] ext4_orphan_add+0xb9/0x1f0 [ext4]
> [<ffffffffa01146b9>] ? ext4_delete_entry+0xf9/0x110 [ext4]
> [<ffffffffa0116a8b>] ext4_unlink+0x29b/0x2b0 [ext4]
> [<ffffffff81183edf>] vfs_unlink+0x9f/0xe0
> [<ffffffff81182c2a>] ? lookup_hash+0x3a/0x50
> [<ffffffff81186463>] do_unlinkat+0x183/0x1c0
> [<ffffffff81183601>] ? path_put+0x31/0x40
> [<ffffffff8119ad11>] ? sys_lgetxattr+0x61/0x80
> [<ffffffff811864b6>] sys_unlink+0x16/0x20
> [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
> INFO: task glusterfsd:1878 blocked for more than 120 seconds.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> glusterfsd D 0000000000000000 0 1878 1 0x00000000
> ffff88003b683c68 0000000000000086 0000000000000000 000000010000000b
> ffff88003b683d18 ffff88003b683c98 ffff88003b683bf8 ffffffff81090957
> ffff880037a9e6b8 ffff88003b683fd8 000000000000f4e8 ffff880037a9e6b8
> Call Trace:
> [<ffffffff81090957>] ? bit_waitqueue+0x17/0xd0
> [<ffffffffa00de09d>] do_get_write_access+0x29d/0x520 [jbd2]
> [<ffffffff81090ad0>] ? wake_bit_function+0x0/0x50
> [<ffffffffa013293b>] ? __ext4_handle_dirty_metadata+0x7b/0x100 [ext4]
> [<ffffffffa00de471>] jbd2_journal_get_write_access+0x31/0x50 [jbd2]
> [<ffffffffa0132b78>] __ext4_journal_get_write_access+0x38/0x80 [ext4]
> [<ffffffffa0114fb9>] ext4_orphan_add+0xb9/0x1f0 [ext4]
> [<ffffffffa01146b9>] ? ext4_delete_entry+0xf9/0x110 [ext4]
> [<ffffffffa0116a8b>] ext4_unlink+0x29b/0x2b0 [ext4]
> [<ffffffff81183edf>] vfs_unlink+0x9f/0xe0
> [<ffffffff81182c2a>] ? lookup_hash+0x3a/0x50
> [<ffffffff81186463>] do_unlinkat+0x183/0x1c0
> [<ffffffff81183601>] ? path_put+0x31/0x40
> [<ffffffff8119ad11>] ? sys_lgetxattr+0x61/0x80
> [<ffffffff811864b6>] sys_unlink+0x16/0x20
> [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
> INFO: task glusterfsd:1878 blocked for more than 120 seconds.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> glusterfsd D 0000000000000000 0 1878 1 0x00000000
> ffff88003b683c68 0000000000000086 0000000000000000 000000010000000b
> ffff88003b683d18 ffff88003b683c98 ffff88003b683bf8 ffffffff81090957
> ffff880037a9e6b8 ffff88003b683fd8 000000000000f4e8 ffff880037a9e6b8
> Call Trace:
> [<ffffffff81090957>] ? bit_waitqueue+0x17/0xd0
> [<ffffffffa00de09d>] do_get_write_access+0x29d/0x520 [jbd2]
> [<ffffffff81090ad0>] ? wake_bit_function+0x0/0x50
> [<ffffffffa013293b>] ? __ext4_handle_dirty_metadata+0x7b/0x100 [ext4]
> [<ffffffffa00de471>] jbd2_journal_get_write_access+0x31/0x50 [jbd2]
> [<ffffffffa0132b78>] __ext4_journal_get_write_access+0x38/0x80 [ext4]
> [<ffffffffa0114fb9>] ext4_orphan_add+0xb9/0x1f0 [ext4]
> [<ffffffffa01146b9>] ? ext4_delete_entry+0xf9/0x110 [ext4]
> [<ffffffffa0116a8b>] ext4_unlink+0x29b/0x2b0 [ext4]
> [<ffffffff81183edf>] vfs_unlink+0x9f/0xe0
> [<ffffffff81182c2a>] ? lookup_hash+0x3a/0x50
> [<ffffffff81186463>] do_unlinkat+0x183/0x1c0
> [<ffffffff81183601>] ? path_put+0x31/0x40
> [<ffffffff8119ad11>] ? sys_lgetxattr+0x61/0x80
> [<ffffffff811864b6>] sys_unlink+0x16/0x20
> [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
> INFO: task glusterfsd:1878 blocked for more than 120 seconds.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> glusterfsd D 0000000000000000 0 1878 1 0x00000000
> ffff88003b683c68 0000000000000086 0000000000000000 000000010000000b
> ffff88003b683d18 ffff88003b683c98 ffff88003b683bf8 ffffffff81090957
> ffff880037a9e6b8 ffff88003b683fd8 000000000000f4e8 ffff880037a9e6b8
> Call Trace:
> [<ffffffff81090957>] ? bit_waitqueue+0x17/0xd0
> [<ffffffffa00de09d>] do_get_write_access+0x29d/0x520 [jbd2]
> [<ffffffff81090ad0>] ? wake_bit_function+0x0/0x50
> [<ffffffffa013293b>] ? __ext4_handle_dirty_metadata+0x7b/0x100 [ext4]
> [<ffffffffa00de471>] jbd2_journal_get_write_access+0x31/0x50 [jbd2]
> [<ffffffffa0132b78>] __ext4_journal_get_write_access+0x38/0x80 [ext4]
> [<ffffffffa0114fb9>] ext4_orphan_add+0xb9/0x1f0 [ext4]
> [<ffffffffa01146b9>] ? ext4_delete_entry+0xf9/0x110 [ext4]
> [<ffffffffa0116a8b>] ext4_unlink+0x29b/0x2b0 [ext4]
> [<ffffffff81183edf>] vfs_unlink+0x9f/0xe0
> [<ffffffff81182c2a>] ? lookup_hash+0x3a/0x50
> [<ffffffff81186463>] do_unlinkat+0x183/0x1c0
> [<ffffffff81183601>] ? path_put+0x31/0x40
> [<ffffffff8119ad11>] ? sys_lgetxattr+0x61/0x80
> [<ffffffff811864b6>] sys_unlink+0x16/0x20
> [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
> INFO: task glusterfsd:1754 blocked for more than 120 seconds.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> glusterfsd D 0000000000000000 0 1754 1 0x00000000
> ffff88003b479c48 0000000000000086 ffff880000000001 000000013712d580
> 0000000000000000 ffff88003f810080 ffff88003faacc00 ffff88003f8224c0
> ffff88003712db38 ffff88003b479fd8 000000000000f4e8 ffff88003712db38
> Call Trace:
> [<ffffffff81090d7e>] ? prepare_to_wait+0x4e/0x80
> [<ffffffffa00de09d>] do_get_write_access+0x29d/0x520 [jbd2]
> [<ffffffff81090ad0>] ? wake_bit_function+0x0/0x50
> [<ffffffffa00de471>] jbd2_journal_get_write_access+0x31/0x50 [jbd2]
> [<ffffffffa0132b78>] __ext4_journal_get_write_access+0x38/0x80 [ext4]
> [<ffffffffa010e253>] ext4_reserve_inode_write+0x73/0xa0 [ext4]
> [<ffffffffa0107158>] ? call_filldir+0x78/0xe0 [ext4]
> [<ffffffffa010e2cc>] ext4_mark_inode_dirty+0x4c/0x1d0 [ext4]
> [<ffffffffa010e5c0>] ext4_dirty_inode+0x40/0x60 [ext4]
> [<ffffffff8119fceb>] __mark_inode_dirty+0x3b/0x160
> [<ffffffff8119040d>] touch_atime+0x12d/0x170
> [<ffffffff8118a290>] ? filldir+0x0/0xe0
> [<ffffffff8118a526>] vfs_readdir+0xd6/0xe0
> [<ffffffff8118a699>] sys_getdents+0x89/0xf0
> [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
> INFO: task glusterfsd:1754 blocked for more than 120 seconds.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> glusterfsd D 0000000000000000 0 1754 1 0x00000000
> ffff88003b479c48 0000000000000086 ffff880000000001 000000013712d580
> 0000000000000000 ffff88003f810080 ffff88003faacc00 ffff88003f8224c0
> ffff88003712db38 ffff88003b479fd8 000000000000f4e8 ffff88003712db38
> Call Trace:
> [<ffffffff81090d7e>] ? prepare_to_wait+0x4e/0x80
> [<ffffffffa00de09d>] do_get_write_access+0x29d/0x520 [jbd2]
> [<ffffffff81090ad0>] ? wake_bit_function+0x0/0x50
> [<ffffffffa00de471>] jbd2_journal_get_write_access+0x31/0x50 [jbd2]
> [<ffffffffa0132b78>] __ext4_journal_get_write_access+0x38/0x80 [ext4]
> [<ffffffffa010e253>] ext4_reserve_inode_write+0x73/0xa0 [ext4]
> [<ffffffffa0107158>] ? call_filldir+0x78/0xe0 [ext4]
> [<ffffffffa010e2cc>] ext4_mark_inode_dirty+0x4c/0x1d0 [ext4]
> [<ffffffffa010e5c0>] ext4_dirty_inode+0x40/0x60 [ext4]
> [<ffffffff8119fceb>] __mark_inode_dirty+0x3b/0x160
> [<ffffffff8119040d>] touch_atime+0x12d/0x170
> [<ffffffff8118a290>] ? filldir+0x0/0xe0
> [<ffffffff8118a526>] vfs_readdir+0xd6/0xe0
> [<ffffffff8118a699>] sys_getdents+0x89/0xf0
> [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
> INFO: task glusterfsd:1754 blocked for more than 120 seconds.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> glusterfsd D 0000000000000000 0 1754 1 0x00000000
> ffff88003b479c48 0000000000000086 ffff880000000001 000000013712d580
> 0000000000000000 ffff88003f810080 ffff88003faacc00 ffff88003f8224c0
> ffff88003712db38 ffff88003b479fd8 000000000000f4e8 ffff88003712db38
> Call Trace:
> [<ffffffff81090d7e>] ? prepare_to_wait+0x4e/0x80
> [<ffffffffa00de09d>] do_get_write_access+0x29d/0x520 [jbd2]
> [<ffffffff81090ad0>] ? wake_bit_function+0x0/0x50
> [<ffffffffa00de471>] jbd2_journal_get_write_access+0x31/0x50 [jbd2]
> [<ffffffffa0132b78>] __ext4_journal_get_write_access+0x38/0x80 [ext4]
> [<ffffffffa010e253>] ext4_reserve_inode_write+0x73/0xa0 [ext4]
> [<ffffffffa0107158>] ? call_filldir+0x78/0xe0 [ext4]
> [<ffffffffa010e2cc>] ext4_mark_inode_dirty+0x4c/0x1d0 [ext4]
> [<ffffffffa010e5c0>] ext4_dirty_inode+0x40/0x60 [ext4]
> [<ffffffff8119fceb>] __mark_inode_dirty+0x3b/0x160
> [<ffffffff8119040d>] touch_atime+0x12d/0x170
> [<ffffffff8118a290>] ? filldir+0x0/0xe0
> [<ffffffff8118a526>] vfs_readdir+0xd6/0xe0
> [<ffffffff8118a699>] sys_getdents+0x89/0xf0
> [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
> INFO: task glusterfsd:1878 blocked for more than 120 seconds.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> glusterfsd D 0000000000000000 0 1878 1 0x00000000
> ffff88003b683c48 0000000000000086 0000000000000000 00000001810949db
> 0000000000000000 0000000000000000 ffff88003b683cb0 0000000000000000
> ffff880037a9e6b8 ffff88003b683fd8 000000000000f4e8 ffff880037a9e6b8
> Call Trace:
> [<ffffffffa00de09d>] do_get_write_access+0x29d/0x520 [jbd2]
> [<ffffffff81090ad0>] ? wake_bit_function+0x0/0x50
> [<ffffffffa00de471>] jbd2_journal_get_write_access+0x31/0x50 [jbd2]
> [<ffffffffa0132b78>] __ext4_journal_get_write_access+0x38/0x80 [ext4]
> [<ffffffffa010e253>] ext4_reserve_inode_write+0x73/0xa0 [ext4]
> [<ffffffffa0107158>] ? call_filldir+0x78/0xe0 [ext4]
> [<ffffffffa010e2cc>] ext4_mark_inode_dirty+0x4c/0x1d0 [ext4]
> [<ffffffffa010e5c0>] ext4_dirty_inode+0x40/0x60 [ext4]
> [<ffffffff8119fceb>] __mark_inode_dirty+0x3b/0x160
> [<ffffffff8119040d>] touch_atime+0x12d/0x170
> [<ffffffff8118a290>] ? filldir+0x0/0xe0
> [<ffffffff8118a526>] vfs_readdir+0xd6/0xe0
> [<ffffffff8118a699>] sys_getdents+0x89/0xf0
> [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
> INFO: task glusterfsd:1878 blocked for more than 120 seconds.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> glusterfsd D 0000000000000000 0 1878 1 0x00000000
> ffff88003b683c48 0000000000000086 0000000000000000 00000001810949db
> 0000000000000000 0000000000000000 ffff88003b683cb0 0000000000000000
> ffff880037a9e6b8 ffff88003b683fd8 000000000000f4e8 ffff880037a9e6b8
> Call Trace:
> [<ffffffffa00de09d>] do_get_write_access+0x29d/0x520 [jbd2]
> [<ffffffff81090ad0>] ? wake_bit_function+0x0/0x50
> [<ffffffffa00de471>] jbd2_journal_get_write_access+0x31/0x50 [jbd2]
> [<ffffffffa0132b78>] __ext4_journal_get_write_access+0x38/0x80 [ext4]
> [<ffffffffa010e253>] ext4_reserve_inode_write+0x73/0xa0 [ext4]
> [<ffffffffa0107158>] ? call_filldir+0x78/0xe0 [ext4]
> [<ffffffffa010e2cc>] ext4_mark_inode_dirty+0x4c/0x1d0 [ext4]
> [<ffffffffa010e5c0>] ext4_dirty_inode+0x40/0x60 [ext4]
> [<ffffffff8119fceb>] __mark_inode_dirty+0x3b/0x160
> [<ffffffff8119040d>] touch_atime+0x12d/0x170
> [<ffffffff8118a290>] ? filldir+0x0/0xe0
> [<ffffffff8118a526>] vfs_readdir+0xd6/0xe0
> [<ffffffff8118a699>] sys_getdents+0x89/0xf0
> [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
> INFO: task glusterfsd:1878 blocked for more than 120 seconds.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> glusterfsd D 0000000000000000 0 1878 1 0x00000000
> ffff88003b683c48 0000000000000086 0000000000000000 00000001810949db
> 0000000000000000 0000000000000000 ffff88003b683cb0 0000000000000000
> ffff880037a9e6b8 ffff88003b683fd8 000000000000f4e8 ffff880037a9e6b8
> Call Trace:
> [<ffffffffa00de09d>] do_get_write_access+0x29d/0x520 [jbd2]
> [<ffffffff81090ad0>] ? wake_bit_function+0x0/0x50
> [<ffffffffa00de471>] jbd2_journal_get_write_access+0x31/0x50 [jbd2]
> [<ffffffffa0132b78>] __ext4_journal_get_write_access+0x38/0x80 [ext4]
> [<ffffffffa010e253>] ext4_reserve_inode_write+0x73/0xa0 [ext4]
> [<ffffffffa0107158>] ? call_filldir+0x78/0xe0 [ext4]
> [<ffffffffa010e2cc>] ext4_mark_inode_dirty+0x4c/0x1d0 [ext4]
> [<ffffffffa010e5c0>] ext4_dirty_inode+0x40/0x60 [ext4]
> [<ffffffff8119fceb>] __mark_inode_dirty+0x3b/0x160
> [<ffffffff8119040d>] touch_atime+0x12d/0x170
> [<ffffffff8118a290>] ? filldir+0x0/0xe0
> [<ffffffff8118a526>] vfs_readdir+0xd6/0xe0
> [<ffffffff8118a699>] sys_getdents+0x89/0xf0
> [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
> [root@gluster01 ~]#
>
>
> [root@gluster01 ~]# mount
> /dev/sda3 on / type ext4 (rw)
> proc on /proc type proc (rw)
> sysfs on /sys type sysfs (rw)
> devpts on /dev/pts type devpts (rw,gid=5,mode=620)
> tmpfs on /dev/shm type tmpfs (rw)
> /dev/sda1 on /boot type ext4 (rw)
> /dev/sdb1 on /GLUSTER01 type ext4 (rw)
> none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
> sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
>
> [root@gluster01 ~]# df -h
> Filesystem Size Used Avail Use% Mounted on
> /dev/sda3 18G 1.8G 15G 11% /
> tmpfs 499M 0 499M 0% /dev/shm
> /dev/sda1 248M 103M 133M 44% /boot
> /dev/sdb1 1.4T 198M 1.4T 1% /GLUSTER01
>
> [root@gluster01 ~]# gluster -V
> glusterfs 3.2.5 built on Dec 7 2011 18:41:39
> Repository revision: git://git.gluster.com/glusterfs.git
> Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com>
> GlusterFS comes with ABSOLUTELY NO WARRANTY.
> You may redistribute copies of GlusterFS under the terms of the GNU General Public License.
>
> [root@gluster01 ~]# uname -a
> Linux gluster01.prod 2.6.32-220.4.1.el6.x86_64 #1 SMP Tue Jan 24 02:13:44 GMT 2012 x86_64 x86_64 x86_64 GNU/Linux
>
>
> THE MACHINE RUNS CENTOS 6.2
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>
--
Jesper Juhl <jj@chaosbits.net> http://www.chaosbits.net/
Don't top-post http://www.catb.org/jargon/html/T/top-post.html
Plain text mails only, please.
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Ext4 Kernel Crash
2012-01-29 17:50 Ext4 Kernel Crash Nanakos Chrysostomos
2012-01-29 22:03 ` Jesper Juhl
@ 2012-01-30 17:26 ` Ted Ts'o
1 sibling, 0 replies; 5+ messages in thread
From: Ted Ts'o @ 2012-01-30 17:26 UTC (permalink / raw)
To: Nanakos Chrysostomos; +Cc: linux-kernel, linux-ext4
On Sun, Jan 29, 2012 at 07:50:57PM +0200, Nanakos Chrysostomos wrote:
> Dear Linux Kernel Developers,
> I am writing to report a kernel or ext4 bug.
Hi Nanakos,
My apologies for not responding to the message that you sent to me
personally three days ago, but I was on vacation in Hawaii; the best
place to send queries about ext4 file systems is to use the
linux-ext4@vger.kernel.org mailing list.
One thing that I would immediately suggest is to see whether you can
reproduce this bug using a modern kernel. You are currently using a
RHEL6-derived kernel (2.6.32-220.4.1.el6.x86_64). The ext4 file
system in such a kernel is quite different from what is in upstream,
and there were a large number of bugs that have been fixed in ext4
since 2.6.32. Many of these fixes were backported into the RHEL6
kernel, but I'm not an expert on which fixes have been backported into
RHEL6 and which haven't. (Especially since Red Hat doesn't release
broken-out patches for their RHEL kernels....)
My understanding is that Gluster doesn't require any kernel-side
patches, so it should be OK to run Gluster on a v3.2 kernel.
Alternatively, if you are a paid customer of RHEL6, I'd suggest
sending a support request to Red Hat.
Best regards,
- Ted
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Ext4 Kernel Crash
2012-01-29 22:03 ` Jesper Juhl
@ 2012-01-31 8:34 ` Nanakos V. Chrysostomos
0 siblings, 0 replies; 5+ messages in thread
From: Nanakos V. Chrysostomos @ 2012-01-31 8:34 UTC (permalink / raw)
To: Jesper Juhl; +Cc: linux-kernel, linux-ext4, Andreas Dilger, Theodore Ts'o
Jesper thanks very much for your answer.
Is there anyone else that wants to elaborate a bit more?
Kind regards,
Chris.
> On Sun, 29 Jan 2012, Nanakos Chrysostomos wrote:
>
>> Dear Linux Kernel Developers,
>> I am writing to report a kernel or ext4 bug.
>> I have two servers populated with 2 LUNs from the main storage facility,
>> each server has one LUN which is about 1.2TB. On top of that I am
>> utilizing glusterfs and samba to mount the 2.4TB created space to a
>> Windows server and other Linux machines (via glusterfs native client).
>> Yesterday I tried to copy to the new storage glusterfs space about 788GB
>> and in the morning I got from both machines the attached kernel
>> messages.
>> I have to mention that the first machine runs samba and mounts the
>> allocated glusterfs space with the help of FUSE glusterfs driver. The
>> second machine doesn't mount/export anything and simply runs the
>> glusterfs
>> service.
>> I have to also add that similar problems have happened to another system
>> which is used as a TIF to JP2 converter. The conversion path starts from
>> the conversion of a pdf file to png and then to jp2. As you can imagine
>> this process produces hundreds of thousands of files. Similarly the
>> machine crashes and I get similar kernel messages concerning ext4. I
>> can't
>> give you more information on that because the machine crashes, I have to
>> take a photo and send it back to you next time.
>> What do you believe about the kernel messages? Is this a ext4 bug when
>> thousands of files are created? How can I help resolve that?
>>
>> Please Cc me because I am not in the list.
>>
>> With kind regards,
>> Chris.
>>
>
> 1. Adding ext4 people to CC.
>
> 2. Took a quick look (without knowing much about ext4, mind you), and I
> noticed something that is, probably unrelated, but just in case it is not:
>
> In ext4_new_inode(), at the 'got:' label we have this code:
>
> [...]
> got:
> /* We may have to initialize the block bitmap if it isn't already
> */
> if (EXT4_HAS_RO_COMPAT_FEATURE(sb,
> EXT4_FEATURE_RO_COMPAT_GDT_CSUM) &&
> gdp->bg_flags & cpu_to_le16(EXT4_BG_BLOCK_UNINIT)) {
> struct buffer_head *block_bitmap_bh;
>
> block_bitmap_bh = ext4_read_block_bitmap(sb, group);
> BUFFER_TRACE(block_bitmap_bh, "get block bitmap access");
> err = ext4_journal_get_write_access(handle,
> block_bitmap_bh);
> if (err) {
> brelse(block_bitmap_bh);
> goto fail;
> }
>
> BUFFER_TRACE(block_bitmap_bh, "dirty block bitmap");
> err = ext4_handle_dirty_metadata(handle, NULL,
> block_bitmap_bh);
> [...]
>
> In the above, ext4_read_block_bitmap() may return NULL. If it does, then
> the calls to ext4_journal_get_write_access() and
> ext4_handle_dirty_metadata() are not safe, since both those functions may
> call other functions that may end up dereferencing their argument (the
> ext4_read_block_bitmap() return value).
>
> Whether or not that will ever actually happen and whether or not it's
> related to your problem, I don't know.
> But I thought I'd share the info.
>
>>
>> PS: Are there any ext4 kernel optimization's for avoiding that or
>> narrowing down the problem if this is not so obvious?
>>
>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
>> message.
>> glusterfsd D 0000000000000000 0 2070 1 0x00000000
>> ffff88003daa5c68 0000000000000082 0000000000000000 000000010000000b
>> ffff88003daa5d18 ffff88003daa5c98 ffff88003daa5bf8 ffffffff81090957
>> ffff88003a912638 ffff88003daa5fd8 000000000000f4e8 ffff88003a912638
>> Call Trace:
>> [<ffffffff81090957>] ? bit_waitqueue+0x17/0xd0
>> [<ffffffffa00de09d>] do_get_write_access+0x29d/0x520 [jbd2]
>> [<ffffffff81090ad0>] ? wake_bit_function+0x0/0x50
>> [<ffffffffa013293b>] ? __ext4_handle_dirty_metadata+0x7b/0x100 [ext4]
>> [<ffffffffa00de471>] jbd2_journal_get_write_access+0x31/0x50 [jbd2]
>> [<ffffffffa0132b78>] __ext4_journal_get_write_access+0x38/0x80 [ext4]
>> [<ffffffffa0114fb9>] ext4_orphan_add+0xb9/0x1f0 [ext4]
>> [<ffffffffa01146b9>] ? ext4_delete_entry+0xf9/0x110 [ext4]
>> [<ffffffffa0116a8b>] ext4_unlink+0x29b/0x2b0 [ext4]
>> [<ffffffff81183edf>] vfs_unlink+0x9f/0xe0
>> [<ffffffff81182c2a>] ? lookup_hash+0x3a/0x50
>> [<ffffffff81186463>] do_unlinkat+0x183/0x1c0
>> [<ffffffff81183601>] ? path_put+0x31/0x40
>> [<ffffffff8119ad11>] ? sys_lgetxattr+0x61/0x80
>> [<ffffffff811864b6>] sys_unlink+0x16/0x20
>> [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
>> INFO: task glusterfsd:2322 blocked for more than 120 seconds.
>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
>> message.
>> glusterfsd D 0000000000000000 0 2322 1 0x00000000
>> ffff88003c70dc48 0000000000000082 0000000000000000 00000001810949db
>> 0000000000000000 0000000000000000 ffff88003c70dcb0 0000000000000000
>> ffff88003b18baf8 ffff88003c70dfd8 000000000000f4e8 ffff88003b18baf8
>> Call Trace:
>> [<ffffffffa00de09d>] do_get_write_access+0x29d/0x520 [jbd2]
>> [<ffffffff81090ad0>] ? wake_bit_function+0x0/0x50
>> [<ffffffffa00de471>] jbd2_journal_get_write_access+0x31/0x50 [jbd2]
>> [<ffffffffa0132b78>] __ext4_journal_get_write_access+0x38/0x80 [ext4]
>> [<ffffffffa010e253>] ext4_reserve_inode_write+0x73/0xa0 [ext4]
>> [<ffffffffa0107158>] ? call_filldir+0x78/0xe0 [ext4]
>> [<ffffffffa010e2cc>] ext4_mark_inode_dirty+0x4c/0x1d0 [ext4]
>> [<ffffffffa010e5c0>] ext4_dirty_inode+0x40/0x60 [ext4]
>> [<ffffffff8119fceb>] __mark_inode_dirty+0x3b/0x160
>> [<ffffffff8119040d>] touch_atime+0x12d/0x170
>> [<ffffffff8118a290>] ? filldir+0x0/0xe0
>> [<ffffffff8118a526>] vfs_readdir+0xd6/0xe0
>> [<ffffffff8118a699>] sys_getdents+0x89/0xf0
>> [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
>> INFO: task smbd:2585 blocked for more than 120 seconds.
>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
>> message.
>> smbd D 0000000000000000 0 2585 1998 0x00000000
>> ffff88003b299bb8 0000000000000082 000000000000007a 0000000000000286
>> ffff88003b23b8c0 000000000000007e ffff88003b299b98 ffff88003b23b8c0
>> ffff88003a913ab8 ffff88003b299fd8 000000000000f4e8 ffff88003a913ab8
>> Call Trace:
>> [<ffffffff814ee01e>] __mutex_lock_slowpath+0x13e/0x180
>> [<ffffffff8118d11f>] ? d_free+0x3f/0x60
>> [<ffffffff814edebb>] mutex_lock+0x2b/0x50
>> [<ffffffff811848af>] do_lookup+0xef/0x1e0
>> [<ffffffff811851d4>] __link_path_walk+0x734/0x1030
>> [<ffffffff81183795>] ? path_to_nameidata+0x45/0x60
>> [<ffffffff81185d5a>] path_walk+0x6a/0xe0
>> [<ffffffff81185f2b>] do_path_lookup+0x5b/0xa0
>> [<ffffffff81186b97>] user_path_at+0x57/0xa0
>> [<ffffffff81042c24>] ? __do_page_fault+0x1e4/0x480
>> [<ffffffff81182fc5>] ? putname+0x35/0x50
>> [<ffffffff81186ba2>] ? user_path_at+0x62/0xa0
>> [<ffffffff8117b786>] vfs_fstatat+0x46/0x80
>> [<ffffffff81195630>] ? mntput_no_expire+0x30/0x110
>> [<ffffffff8117b8eb>] vfs_stat+0x1b/0x20
>> [<ffffffff8117b914>] sys_newstat+0x24/0x50
>> [<ffffffff814f23fe>] ? do_page_fault+0x3e/0xa0
>> [<ffffffff814ef7b5>] ? page_fault+0x25/0x30
>> [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
>> INFO: task smbd:2586 blocked for more than 120 seconds.
>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
>> message.
>> smbd D 0000000000000000 0 2586 1998 0x00000000
>> ffff88003b1e7bb8 0000000000000082 0000000000000088 0000000000000286
>> ffff88003b619380 000000000000008c ffff88003b1e7b98 ffff88003b619380
>> ffff880037a59af8 ffff88003b1e7fd8 000000000000f4e8 ffff880037a59af8
>> Call Trace:
>> [<ffffffff8141ccde>] ? release_sock+0xce/0xe0
>> [<ffffffff814ee01e>] __mutex_lock_slowpath+0x13e/0x180
>> [<ffffffff814edebb>] mutex_lock+0x2b/0x50
>> [<ffffffff811848af>] do_lookup+0xef/0x1e0
>> [<ffffffff811851d4>] __link_path_walk+0x734/0x1030
>> [<ffffffff81183795>] ? path_to_nameidata+0x45/0x60
>> [<ffffffff81185d5a>] path_walk+0x6a/0xe0
>> [<ffffffff81185f2b>] do_path_lookup+0x5b/0xa0
>> [<ffffffff81186b97>] user_path_at+0x57/0xa0
>> [<ffffffff81042c24>] ? __do_page_fault+0x1e4/0x480
>> [<ffffffff81182fc5>] ? putname+0x35/0x50
>> [<ffffffff81186ba2>] ? user_path_at+0x62/0xa0
>> [<ffffffff8117b786>] vfs_fstatat+0x46/0x80
>> [<ffffffff81195630>] ? mntput_no_expire+0x30/0x110
>> [<ffffffff8117b8eb>] vfs_stat+0x1b/0x20
>> [<ffffffff8117b914>] sys_newstat+0x24/0x50
>> [<ffffffff814f23fe>] ? do_page_fault+0x3e/0xa0
>> [<ffffffff814ef7b5>] ? page_fault+0x25/0x30
>> [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
>> INFO: task smbd:2587 blocked for more than 120 seconds.
>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
>> message.
>> smbd D 0000000000000001 0 2587 1998 0x00000000
>> ffff88003d5efbb8 0000000000000086 0000000000000000 0000000000000286
>> ffff880037047cc0 000000000000007e ffff88003d5efb98 ffff880037047cc0
>> ffff88003b04f038 ffff88003d5effd8 000000000000f4e8 ffff88003b04f038
>> Call Trace:
>> [<ffffffff814ee01e>] __mutex_lock_slowpath+0x13e/0x180
>> [<ffffffff814edebb>] mutex_lock+0x2b/0x50
>> [<ffffffff811848af>] do_lookup+0xef/0x1e0
>> [<ffffffff811851d4>] __link_path_walk+0x734/0x1030
>> [<ffffffff81183795>] ? path_to_nameidata+0x45/0x60
>> [<ffffffff81185d5a>] path_walk+0x6a/0xe0
>> [<ffffffff81185f2b>] do_path_lookup+0x5b/0xa0
>> [<ffffffff81186b97>] user_path_at+0x57/0xa0
>> [<ffffffff81042c24>] ? __do_page_fault+0x1e4/0x480
>> [<ffffffff81182fc5>] ? putname+0x35/0x50
>> [<ffffffff81186ba2>] ? user_path_at+0x62/0xa0
>> [<ffffffff8117b786>] vfs_fstatat+0x46/0x80
>> [<ffffffff81195630>] ? mntput_no_expire+0x30/0x110
>> [<ffffffff8117b8eb>] vfs_stat+0x1b/0x20
>> [<ffffffff8117b914>] sys_newstat+0x24/0x50
>> [<ffffffff814f23fe>] ? do_page_fault+0x3e/0xa0
>> [<ffffffff814ef7b5>] ? page_fault+0x25/0x30
>> [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
>> INFO: task smbd:2588 blocked for more than 120 seconds.
>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
>> message.
>> smbd D 0000000000000001 0 2588 1998 0x00000000
>> ffff88003b093bb8 0000000000000082 0000000000000000 0000000000000286
>> ffff88003a9e69c0 000000000000007e ffff88003b093b98 ffff88003a9e69c0
>> ffff8800376f9b38 ffff88003b093fd8 000000000000f4e8 ffff8800376f9b38
>> Call Trace:
>> [<ffffffff814ee01e>] __mutex_lock_slowpath+0x13e/0x180
>> [<ffffffff814edebb>] mutex_lock+0x2b/0x50
>> [<ffffffff811848af>] do_lookup+0xef/0x1e0
>> [<ffffffff811851d4>] __link_path_walk+0x734/0x1030
>> [<ffffffff81183795>] ? path_to_nameidata+0x45/0x60
>> [<ffffffff81185d5a>] path_walk+0x6a/0xe0
>> [<ffffffff81185f2b>] do_path_lookup+0x5b/0xa0
>> [<ffffffff81186b97>] user_path_at+0x57/0xa0
>> [<ffffffff81042c24>] ? __do_page_fault+0x1e4/0x480
>> [<ffffffff81182fc5>] ? putname+0x35/0x50
>> [<ffffffff81186ba2>] ? user_path_at+0x62/0xa0
>> [<ffffffff8117b786>] vfs_fstatat+0x46/0x80
>> [<ffffffff81195630>] ? mntput_no_expire+0x30/0x110
>> [<ffffffff8117b8eb>] vfs_stat+0x1b/0x20
>> [<ffffffff8117b914>] sys_newstat+0x24/0x50
>> [<ffffffff814f23fe>] ? do_page_fault+0x3e/0xa0
>> [<ffffffff814ef7b5>] ? page_fault+0x25/0x30
>> [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
>> INFO: task smbd:2589 blocked for more than 120 seconds.
>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
>> message.
>> smbd D 0000000000000001 0 2589 1998 0x00000000
>> ffff8800374c1bb8 0000000000000086 0000000000000088 0000000000000286
>> ffff880037bf50c0 000000000000008c ffff8800374c1b98 ffff880037bf50c0
>> ffff88003b18a678 ffff8800374c1fd8 000000000000f4e8 ffff88003b18a678
>> Call Trace:
>> [<ffffffff8141ccde>] ? release_sock+0xce/0xe0
>> [<ffffffff814ee01e>] __mutex_lock_slowpath+0x13e/0x180
>> [<ffffffff814edebb>] mutex_lock+0x2b/0x50
>> [<ffffffff811848af>] do_lookup+0xef/0x1e0
>> [<ffffffff811851d4>] __link_path_walk+0x734/0x1030
>> [<ffffffff81183795>] ? path_to_nameidata+0x45/0x60
>> [<ffffffff81185d5a>] path_walk+0x6a/0xe0
>> [<ffffffff81185f2b>] do_path_lookup+0x5b/0xa0
>> [<ffffffff81186b97>] user_path_at+0x57/0xa0
>> [<ffffffff81042c24>] ? __do_page_fault+0x1e4/0x480
>> [<ffffffff81182fc5>] ? putname+0x35/0x50
>> [<ffffffff81186ba2>] ? user_path_at+0x62/0xa0
>> [<ffffffff8117b786>] vfs_fstatat+0x46/0x80
>> [<ffffffff81195630>] ? mntput_no_expire+0x30/0x110
>> [<ffffffff8117b8eb>] vfs_stat+0x1b/0x20
>> [<ffffffff8117b914>] sys_newstat+0x24/0x50
>> [<ffffffff814f23fe>] ? do_page_fault+0x3e/0xa0
>> [<ffffffff814ef7b5>] ? page_fault+0x25/0x30
>> [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
>> INFO: task glusterfsd:2322 blocked for more than 120 seconds.
>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
>> message.
>> glusterfsd D 0000000000000001 0 2322 1 0x00000000
>> ffff88003c70da28 0000000000000082 ffff880000020f58 ffff88003b8615a8
>> ffff88003c70d9a8 ffffffff81467515 ffff88003b8615a8 ffff88003b8615a8
>> ffff88003b18baf8 ffff88003c70dfd8 000000000000f4e8 ffff88003b18baf8
>> Call Trace:
>> [<ffffffff81467515>] ? ip_local_out+0x25/0x30
>> [<ffffffff81090d7e>] ? prepare_to_wait+0x4e/0x80
>> [<ffffffffa00de09d>] do_get_write_access+0x29d/0x520 [jbd2]
>> [<ffffffff81090ad0>] ? wake_bit_function+0x0/0x50
>> [<ffffffffa00de471>] jbd2_journal_get_write_access+0x31/0x50 [jbd2]
>> [<ffffffffa0132b78>] __ext4_journal_get_write_access+0x38/0x80 [ext4]
>> [<ffffffffa010e253>] ext4_reserve_inode_write+0x73/0xa0 [ext4]
>> [<ffffffffa010e2cc>] ext4_mark_inode_dirty+0x4c/0x1d0 [ext4]
>> [<ffffffffa0141ccd>] ? ext4_init_security+0x2d/0x296 [ext4]
>> [<ffffffffa012d73e>] ext4_ext_tree_init+0x3e/0x60 [ext4]
>> [<ffffffffa010a500>] ext4_new_inode+0x1150/0x1190 [ext4]
>> [<ffffffffa0117c90>] ext4_create+0xc0/0x150 [ext4]
>> [<ffffffff811822a3>] ? generic_permission+0x23/0xb0
>> [<ffffffff81183c84>] vfs_create+0xb4/0xe0
>> [<ffffffff8118788f>] do_filp_open+0xb2f/0xd60
>> [<ffffffff81193482>] ? alloc_fd+0x92/0x160
>> [<ffffffff81173a49>] do_sys_open+0x69/0x140
>> [<ffffffff8119ad11>] ? sys_lgetxattr+0x61/0x80
>> [<ffffffff81173b60>] sys_open+0x20/0x30
>> [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
>> INFO: task smbd:2614 blocked for more than 120 seconds.
>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
>> message.
>> smbd D 0000000000000001 0 2614 1998 0x00000000
>> ffff88003c63dbb8 0000000000000086 0000000000000000 0000000000000286
>> ffff88003b6014c0 000000000000007c ffff88003c63db98 ffff88003b6014c0
>> ffff88003a913ab8 ffff88003c63dfd8 000000000000f4e8 ffff88003a913ab8
>> Call Trace:
>> [<ffffffff814ee01e>] __mutex_lock_slowpath+0x13e/0x180
>> [<ffffffff8118e957>] ? __d_lookup+0xa7/0x150
>> [<ffffffff814edebb>] mutex_lock+0x2b/0x50
>> [<ffffffff811848af>] do_lookup+0xef/0x1e0
>> [<ffffffff811851d4>] __link_path_walk+0x734/0x1030
>> [<ffffffff81183795>] ? path_to_nameidata+0x45/0x60
>> [<ffffffff81185d5a>] path_walk+0x6a/0xe0
>> [<ffffffff81185f2b>] do_path_lookup+0x5b/0xa0
>> [<ffffffff81186b97>] user_path_at+0x57/0xa0
>> [<ffffffff81042c24>] ? __do_page_fault+0x1e4/0x480
>> [<ffffffff81182fc5>] ? putname+0x35/0x50
>> [<ffffffff81186ba2>] ? user_path_at+0x62/0xa0
>> [<ffffffff8117b786>] vfs_fstatat+0x46/0x80
>> [<ffffffff81195630>] ? mntput_no_expire+0x30/0x110
>> [<ffffffff8117b8eb>] vfs_stat+0x1b/0x20
>> [<ffffffff8117b914>] sys_newstat+0x24/0x50
>> [<ffffffff814f23fe>] ? do_page_fault+0x3e/0xa0
>> [<ffffffff814ef7b5>] ? page_fault+0x25/0x30
>> [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
>> INFO: task smbd:2615 blocked for more than 120 seconds.
>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
>> message.
>> smbd D 0000000000000001 0 2615 1998 0x00000000
>> ffff88003d9bbbb8 0000000000000086 0000000000000000 0000000000000286
>> ffff88003720f580 000000000000008a ffff88003d9bbb98 ffff88003720f580
>> ffff880037a59af8 ffff88003d9bbfd8 000000000000f4e8 ffff880037a59af8
>> Call Trace:
>> [<ffffffff814ee01e>] __mutex_lock_slowpath+0x13e/0x180
>> [<ffffffff814edebb>] mutex_lock+0x2b/0x50
>> [<ffffffff811848af>] do_lookup+0xef/0x1e0
>> [<ffffffff811851d4>] __link_path_walk+0x734/0x1030
>> [<ffffffff81183795>] ? path_to_nameidata+0x45/0x60
>> [<ffffffff81185d5a>] path_walk+0x6a/0xe0
>> [<ffffffff81185f2b>] do_path_lookup+0x5b/0xa0
>> [<ffffffff81186b97>] user_path_at+0x57/0xa0
>> [<ffffffff81042c24>] ? __do_page_fault+0x1e4/0x480
>> [<ffffffff81182fc5>] ? putname+0x35/0x50
>> [<ffffffff81186ba2>] ? user_path_at+0x62/0xa0
>> [<ffffffff8117b786>] vfs_fstatat+0x46/0x80
>> [<ffffffff81195630>] ? mntput_no_expire+0x30/0x110
>> [<ffffffff8117b8eb>] vfs_stat+0x1b/0x20
>> [<ffffffff8117b914>] sys_newstat+0x24/0x50
>> [<ffffffff814f23fe>] ? do_page_fault+0x3e/0xa0
>> [<ffffffff814ef7b5>] ? page_fault+0x25/0x30
>> [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
>> [root@gluster00 ~]#
>>
>>
>> [root@gluster00 ~]# mount
>> /dev/sda3 on / type ext4 (rw)
>> proc on /proc type proc (rw)
>> sysfs on /sys type sysfs (rw)
>> devpts on /dev/pts type devpts (rw,gid=5,mode=620)
>> tmpfs on /dev/shm type tmpfs (rw)
>> /dev/sda1 on /boot type ext4 (rw)
>> /dev/sdb1 on /GLUSTER00 type ext4 (rw)
>> none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
>> sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
>> gluster00:/mm-volume on /mmvolume type fuse.glusterfs
>> (rw,allow_other,default_permissions,max_read=131072)
>>
>> [root@gluster00 ~]# df -h
>> Filesystem Size Used Avail Use% Mounted on
>> /dev/sda3 18G 2.0G 15G 13% /
>> tmpfs 499M 0 499M 0% /dev/shm
>> /dev/sda1 248M 103M 133M 44% /boot
>> /dev/sdb1 960G 200M 911G 1% /GLUSTER00
>> gluster00:/mm-volume 2.4T 397M 2.2T 1% /mmvolume
>>
>>
>> [root@gluster00 ~]# gluster -V
>> glusterfs 3.2.5 built on Dec 7 2011 18:41:39
>> Repository revision: git://git.gluster.com/glusterfs.git
>> Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com>
>> GlusterFS comes with ABSOLUTELY NO WARRANTY.
>> You may redistribute copies of GlusterFS under the terms of the GNU
>> General Public License.
>>
>> [root@gluster00 ~]# uname -a
>> Linux gluster00.prod 2.6.32-220.4.1.el6.x86_64 #1 SMP Tue Jan 24
>> 02:13:44 GMT 2012 x86_64 x86_64 x86_64 GNU/Linux
>>
>> [root@gluster00 ~]# smbd -V
>> Version 3.5.10-114.el6
>>
>>
>> THE MACHINE RUNS CENTOS 6.2
>>
>> INFO: task glusterfsd:1878 blocked for more than 120 seconds.
>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
>> message.
>> glusterfsd D 0000000000000000 0 1878 1 0x00000000
>> ffff88003b683c68 0000000000000086 0000000000000000 000000010000000b
>> ffff88003b683d18 ffff88003b683c98 ffff88003b683bf8 ffffffff81090957
>> ffff880037a9e6b8 ffff88003b683fd8 000000000000f4e8 ffff880037a9e6b8
>> Call Trace:
>> [<ffffffff81090957>] ? bit_waitqueue+0x17/0xd0
>> [<ffffffffa00de09d>] do_get_write_access+0x29d/0x520 [jbd2]
>> [<ffffffff81090ad0>] ? wake_bit_function+0x0/0x50
>> [<ffffffffa013293b>] ? __ext4_handle_dirty_metadata+0x7b/0x100 [ext4]
>> [<ffffffffa00de471>] jbd2_journal_get_write_access+0x31/0x50 [jbd2]
>> [<ffffffffa0132b78>] __ext4_journal_get_write_access+0x38/0x80 [ext4]
>> [<ffffffffa0114fb9>] ext4_orphan_add+0xb9/0x1f0 [ext4]
>> [<ffffffffa01146b9>] ? ext4_delete_entry+0xf9/0x110 [ext4]
>> [<ffffffffa0116a8b>] ext4_unlink+0x29b/0x2b0 [ext4]
>> [<ffffffff81183edf>] vfs_unlink+0x9f/0xe0
>> [<ffffffff81182c2a>] ? lookup_hash+0x3a/0x50
>> [<ffffffff81186463>] do_unlinkat+0x183/0x1c0
>> [<ffffffff81183601>] ? path_put+0x31/0x40
>> [<ffffffff8119ad11>] ? sys_lgetxattr+0x61/0x80
>> [<ffffffff811864b6>] sys_unlink+0x16/0x20
>> [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
>> INFO: task glusterfsd:1878 blocked for more than 120 seconds.
>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
>> message.
>> glusterfsd D 0000000000000000 0 1878 1 0x00000000
>> ffff88003b683c68 0000000000000086 0000000000000000 000000010000000b
>> ffff88003b683d18 ffff88003b683c98 ffff88003b683bf8 ffffffff81090957
>> ffff880037a9e6b8 ffff88003b683fd8 000000000000f4e8 ffff880037a9e6b8
>> Call Trace:
>> [<ffffffff81090957>] ? bit_waitqueue+0x17/0xd0
>> [<ffffffffa00de09d>] do_get_write_access+0x29d/0x520 [jbd2]
>> [<ffffffff81090ad0>] ? wake_bit_function+0x0/0x50
>> [<ffffffffa013293b>] ? __ext4_handle_dirty_metadata+0x7b/0x100 [ext4]
>> [<ffffffffa00de471>] jbd2_journal_get_write_access+0x31/0x50 [jbd2]
>> [<ffffffffa0132b78>] __ext4_journal_get_write_access+0x38/0x80 [ext4]
>> [<ffffffffa0114fb9>] ext4_orphan_add+0xb9/0x1f0 [ext4]
>> [<ffffffffa01146b9>] ? ext4_delete_entry+0xf9/0x110 [ext4]
>> [<ffffffffa0116a8b>] ext4_unlink+0x29b/0x2b0 [ext4]
>> [<ffffffff81183edf>] vfs_unlink+0x9f/0xe0
>> [<ffffffff81182c2a>] ? lookup_hash+0x3a/0x50
>> [<ffffffff81186463>] do_unlinkat+0x183/0x1c0
>> [<ffffffff81183601>] ? path_put+0x31/0x40
>> [<ffffffff8119ad11>] ? sys_lgetxattr+0x61/0x80
>> [<ffffffff811864b6>] sys_unlink+0x16/0x20
>> [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
>> INFO: task glusterfsd:1878 blocked for more than 120 seconds.
>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
>> message.
>> glusterfsd D 0000000000000000 0 1878 1 0x00000000
>> ffff88003b683c68 0000000000000086 0000000000000000 000000010000000b
>> ffff88003b683d18 ffff88003b683c98 ffff88003b683bf8 ffffffff81090957
>> ffff880037a9e6b8 ffff88003b683fd8 000000000000f4e8 ffff880037a9e6b8
>> Call Trace:
>> [<ffffffff81090957>] ? bit_waitqueue+0x17/0xd0
>> [<ffffffffa00de09d>] do_get_write_access+0x29d/0x520 [jbd2]
>> [<ffffffff81090ad0>] ? wake_bit_function+0x0/0x50
>> [<ffffffffa013293b>] ? __ext4_handle_dirty_metadata+0x7b/0x100 [ext4]
>> [<ffffffffa00de471>] jbd2_journal_get_write_access+0x31/0x50 [jbd2]
>> [<ffffffffa0132b78>] __ext4_journal_get_write_access+0x38/0x80 [ext4]
>> [<ffffffffa0114fb9>] ext4_orphan_add+0xb9/0x1f0 [ext4]
>> [<ffffffffa01146b9>] ? ext4_delete_entry+0xf9/0x110 [ext4]
>> [<ffffffffa0116a8b>] ext4_unlink+0x29b/0x2b0 [ext4]
>> [<ffffffff81183edf>] vfs_unlink+0x9f/0xe0
>> [<ffffffff81182c2a>] ? lookup_hash+0x3a/0x50
>> [<ffffffff81186463>] do_unlinkat+0x183/0x1c0
>> [<ffffffff81183601>] ? path_put+0x31/0x40
>> [<ffffffff8119ad11>] ? sys_lgetxattr+0x61/0x80
>> [<ffffffff811864b6>] sys_unlink+0x16/0x20
>> [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
>> INFO: task glusterfsd:1878 blocked for more than 120 seconds.
>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
>> message.
>> glusterfsd D 0000000000000000 0 1878 1 0x00000000
>> ffff88003b683c68 0000000000000086 0000000000000000 000000010000000b
>> ffff88003b683d18 ffff88003b683c98 ffff88003b683bf8 ffffffff81090957
>> ffff880037a9e6b8 ffff88003b683fd8 000000000000f4e8 ffff880037a9e6b8
>> Call Trace:
>> [<ffffffff81090957>] ? bit_waitqueue+0x17/0xd0
>> [<ffffffffa00de09d>] do_get_write_access+0x29d/0x520 [jbd2]
>> [<ffffffff81090ad0>] ? wake_bit_function+0x0/0x50
>> [<ffffffffa013293b>] ? __ext4_handle_dirty_metadata+0x7b/0x100 [ext4]
>> [<ffffffffa00de471>] jbd2_journal_get_write_access+0x31/0x50 [jbd2]
>> [<ffffffffa0132b78>] __ext4_journal_get_write_access+0x38/0x80 [ext4]
>> [<ffffffffa0114fb9>] ext4_orphan_add+0xb9/0x1f0 [ext4]
>> [<ffffffffa01146b9>] ? ext4_delete_entry+0xf9/0x110 [ext4]
>> [<ffffffffa0116a8b>] ext4_unlink+0x29b/0x2b0 [ext4]
>> [<ffffffff81183edf>] vfs_unlink+0x9f/0xe0
>> [<ffffffff81182c2a>] ? lookup_hash+0x3a/0x50
>> [<ffffffff81186463>] do_unlinkat+0x183/0x1c0
>> [<ffffffff81183601>] ? path_put+0x31/0x40
>> [<ffffffff8119ad11>] ? sys_lgetxattr+0x61/0x80
>> [<ffffffff811864b6>] sys_unlink+0x16/0x20
>> [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
>> INFO: task glusterfsd:1754 blocked for more than 120 seconds.
>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
>> message.
>> glusterfsd D 0000000000000000 0 1754 1 0x00000000
>> ffff88003b479c48 0000000000000086 ffff880000000001 000000013712d580
>> 0000000000000000 ffff88003f810080 ffff88003faacc00 ffff88003f8224c0
>> ffff88003712db38 ffff88003b479fd8 000000000000f4e8 ffff88003712db38
>> Call Trace:
>> [<ffffffff81090d7e>] ? prepare_to_wait+0x4e/0x80
>> [<ffffffffa00de09d>] do_get_write_access+0x29d/0x520 [jbd2]
>> [<ffffffff81090ad0>] ? wake_bit_function+0x0/0x50
>> [<ffffffffa00de471>] jbd2_journal_get_write_access+0x31/0x50 [jbd2]
>> [<ffffffffa0132b78>] __ext4_journal_get_write_access+0x38/0x80 [ext4]
>> [<ffffffffa010e253>] ext4_reserve_inode_write+0x73/0xa0 [ext4]
>> [<ffffffffa0107158>] ? call_filldir+0x78/0xe0 [ext4]
>> [<ffffffffa010e2cc>] ext4_mark_inode_dirty+0x4c/0x1d0 [ext4]
>> [<ffffffffa010e5c0>] ext4_dirty_inode+0x40/0x60 [ext4]
>> [<ffffffff8119fceb>] __mark_inode_dirty+0x3b/0x160
>> [<ffffffff8119040d>] touch_atime+0x12d/0x170
>> [<ffffffff8118a290>] ? filldir+0x0/0xe0
>> [<ffffffff8118a526>] vfs_readdir+0xd6/0xe0
>> [<ffffffff8118a699>] sys_getdents+0x89/0xf0
>> [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
>> INFO: task glusterfsd:1754 blocked for more than 120 seconds.
>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
>> message.
>> glusterfsd D 0000000000000000 0 1754 1 0x00000000
>> ffff88003b479c48 0000000000000086 ffff880000000001 000000013712d580
>> 0000000000000000 ffff88003f810080 ffff88003faacc00 ffff88003f8224c0
>> ffff88003712db38 ffff88003b479fd8 000000000000f4e8 ffff88003712db38
>> Call Trace:
>> [<ffffffff81090d7e>] ? prepare_to_wait+0x4e/0x80
>> [<ffffffffa00de09d>] do_get_write_access+0x29d/0x520 [jbd2]
>> [<ffffffff81090ad0>] ? wake_bit_function+0x0/0x50
>> [<ffffffffa00de471>] jbd2_journal_get_write_access+0x31/0x50 [jbd2]
>> [<ffffffffa0132b78>] __ext4_journal_get_write_access+0x38/0x80 [ext4]
>> [<ffffffffa010e253>] ext4_reserve_inode_write+0x73/0xa0 [ext4]
>> [<ffffffffa0107158>] ? call_filldir+0x78/0xe0 [ext4]
>> [<ffffffffa010e2cc>] ext4_mark_inode_dirty+0x4c/0x1d0 [ext4]
>> [<ffffffffa010e5c0>] ext4_dirty_inode+0x40/0x60 [ext4]
>> [<ffffffff8119fceb>] __mark_inode_dirty+0x3b/0x160
>> [<ffffffff8119040d>] touch_atime+0x12d/0x170
>> [<ffffffff8118a290>] ? filldir+0x0/0xe0
>> [<ffffffff8118a526>] vfs_readdir+0xd6/0xe0
>> [<ffffffff8118a699>] sys_getdents+0x89/0xf0
>> [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
>> INFO: task glusterfsd:1754 blocked for more than 120 seconds.
>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
>> message.
>> glusterfsd D 0000000000000000 0 1754 1 0x00000000
>> ffff88003b479c48 0000000000000086 ffff880000000001 000000013712d580
>> 0000000000000000 ffff88003f810080 ffff88003faacc00 ffff88003f8224c0
>> ffff88003712db38 ffff88003b479fd8 000000000000f4e8 ffff88003712db38
>> Call Trace:
>> [<ffffffff81090d7e>] ? prepare_to_wait+0x4e/0x80
>> [<ffffffffa00de09d>] do_get_write_access+0x29d/0x520 [jbd2]
>> [<ffffffff81090ad0>] ? wake_bit_function+0x0/0x50
>> [<ffffffffa00de471>] jbd2_journal_get_write_access+0x31/0x50 [jbd2]
>> [<ffffffffa0132b78>] __ext4_journal_get_write_access+0x38/0x80 [ext4]
>> [<ffffffffa010e253>] ext4_reserve_inode_write+0x73/0xa0 [ext4]
>> [<ffffffffa0107158>] ? call_filldir+0x78/0xe0 [ext4]
>> [<ffffffffa010e2cc>] ext4_mark_inode_dirty+0x4c/0x1d0 [ext4]
>> [<ffffffffa010e5c0>] ext4_dirty_inode+0x40/0x60 [ext4]
>> [<ffffffff8119fceb>] __mark_inode_dirty+0x3b/0x160
>> [<ffffffff8119040d>] touch_atime+0x12d/0x170
>> [<ffffffff8118a290>] ? filldir+0x0/0xe0
>> [<ffffffff8118a526>] vfs_readdir+0xd6/0xe0
>> [<ffffffff8118a699>] sys_getdents+0x89/0xf0
>> [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
>> INFO: task glusterfsd:1878 blocked for more than 120 seconds.
>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
>> message.
>> glusterfsd D 0000000000000000 0 1878 1 0x00000000
>> ffff88003b683c48 0000000000000086 0000000000000000 00000001810949db
>> 0000000000000000 0000000000000000 ffff88003b683cb0 0000000000000000
>> ffff880037a9e6b8 ffff88003b683fd8 000000000000f4e8 ffff880037a9e6b8
>> Call Trace:
>> [<ffffffffa00de09d>] do_get_write_access+0x29d/0x520 [jbd2]
>> [<ffffffff81090ad0>] ? wake_bit_function+0x0/0x50
>> [<ffffffffa00de471>] jbd2_journal_get_write_access+0x31/0x50 [jbd2]
>> [<ffffffffa0132b78>] __ext4_journal_get_write_access+0x38/0x80 [ext4]
>> [<ffffffffa010e253>] ext4_reserve_inode_write+0x73/0xa0 [ext4]
>> [<ffffffffa0107158>] ? call_filldir+0x78/0xe0 [ext4]
>> [<ffffffffa010e2cc>] ext4_mark_inode_dirty+0x4c/0x1d0 [ext4]
>> [<ffffffffa010e5c0>] ext4_dirty_inode+0x40/0x60 [ext4]
>> [<ffffffff8119fceb>] __mark_inode_dirty+0x3b/0x160
>> [<ffffffff8119040d>] touch_atime+0x12d/0x170
>> [<ffffffff8118a290>] ? filldir+0x0/0xe0
>> [<ffffffff8118a526>] vfs_readdir+0xd6/0xe0
>> [<ffffffff8118a699>] sys_getdents+0x89/0xf0
>> [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
>> INFO: task glusterfsd:1878 blocked for more than 120 seconds.
>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
>> message.
>> glusterfsd D 0000000000000000 0 1878 1 0x00000000
>> ffff88003b683c48 0000000000000086 0000000000000000 00000001810949db
>> 0000000000000000 0000000000000000 ffff88003b683cb0 0000000000000000
>> ffff880037a9e6b8 ffff88003b683fd8 000000000000f4e8 ffff880037a9e6b8
>> Call Trace:
>> [<ffffffffa00de09d>] do_get_write_access+0x29d/0x520 [jbd2]
>> [<ffffffff81090ad0>] ? wake_bit_function+0x0/0x50
>> [<ffffffffa00de471>] jbd2_journal_get_write_access+0x31/0x50 [jbd2]
>> [<ffffffffa0132b78>] __ext4_journal_get_write_access+0x38/0x80 [ext4]
>> [<ffffffffa010e253>] ext4_reserve_inode_write+0x73/0xa0 [ext4]
>> [<ffffffffa0107158>] ? call_filldir+0x78/0xe0 [ext4]
>> [<ffffffffa010e2cc>] ext4_mark_inode_dirty+0x4c/0x1d0 [ext4]
>> [<ffffffffa010e5c0>] ext4_dirty_inode+0x40/0x60 [ext4]
>> [<ffffffff8119fceb>] __mark_inode_dirty+0x3b/0x160
>> [<ffffffff8119040d>] touch_atime+0x12d/0x170
>> [<ffffffff8118a290>] ? filldir+0x0/0xe0
>> [<ffffffff8118a526>] vfs_readdir+0xd6/0xe0
>> [<ffffffff8118a699>] sys_getdents+0x89/0xf0
>> [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
>> INFO: task glusterfsd:1878 blocked for more than 120 seconds.
>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
>> message.
>> glusterfsd D 0000000000000000 0 1878 1 0x00000000
>> ffff88003b683c48 0000000000000086 0000000000000000 00000001810949db
>> 0000000000000000 0000000000000000 ffff88003b683cb0 0000000000000000
>> ffff880037a9e6b8 ffff88003b683fd8 000000000000f4e8 ffff880037a9e6b8
>> Call Trace:
>> [<ffffffffa00de09d>] do_get_write_access+0x29d/0x520 [jbd2]
>> [<ffffffff81090ad0>] ? wake_bit_function+0x0/0x50
>> [<ffffffffa00de471>] jbd2_journal_get_write_access+0x31/0x50 [jbd2]
>> [<ffffffffa0132b78>] __ext4_journal_get_write_access+0x38/0x80 [ext4]
>> [<ffffffffa010e253>] ext4_reserve_inode_write+0x73/0xa0 [ext4]
>> [<ffffffffa0107158>] ? call_filldir+0x78/0xe0 [ext4]
>> [<ffffffffa010e2cc>] ext4_mark_inode_dirty+0x4c/0x1d0 [ext4]
>> [<ffffffffa010e5c0>] ext4_dirty_inode+0x40/0x60 [ext4]
>> [<ffffffff8119fceb>] __mark_inode_dirty+0x3b/0x160
>> [<ffffffff8119040d>] touch_atime+0x12d/0x170
>> [<ffffffff8118a290>] ? filldir+0x0/0xe0
>> [<ffffffff8118a526>] vfs_readdir+0xd6/0xe0
>> [<ffffffff8118a699>] sys_getdents+0x89/0xf0
>> [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
>> [root@gluster01 ~]#
>>
>>
>> [root@gluster01 ~]# mount
>> /dev/sda3 on / type ext4 (rw)
>> proc on /proc type proc (rw)
>> sysfs on /sys type sysfs (rw)
>> devpts on /dev/pts type devpts (rw,gid=5,mode=620)
>> tmpfs on /dev/shm type tmpfs (rw)
>> /dev/sda1 on /boot type ext4 (rw)
>> /dev/sdb1 on /GLUSTER01 type ext4 (rw)
>> none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
>> sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
>>
>> [root@gluster01 ~]# df -h
>> Filesystem Size Used Avail Use% Mounted on
>> /dev/sda3 18G 1.8G 15G 11% /
>> tmpfs 499M 0 499M 0% /dev/shm
>> /dev/sda1 248M 103M 133M 44% /boot
>> /dev/sdb1 1.4T 198M 1.4T 1% /GLUSTER01
>>
>> [root@gluster01 ~]# gluster -V
>> glusterfs 3.2.5 built on Dec 7 2011 18:41:39
>> Repository revision: git://git.gluster.com/glusterfs.git
>> Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com>
>> GlusterFS comes with ABSOLUTELY NO WARRANTY.
>> You may redistribute copies of GlusterFS under the terms of the GNU
>> General Public License.
>>
>> [root@gluster01 ~]# uname -a
>> Linux gluster01.prod 2.6.32-220.4.1.el6.x86_64 #1 SMP Tue Jan 24
>> 02:13:44 GMT 2012 x86_64 x86_64 x86_64 GNU/Linux
>>
>>
>> THE MACHINE RUNS CENTOS 6.2
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-kernel"
>> in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>> Please read the FAQ at http://www.tux.org/lkml/
>>
>
> --
> Jesper Juhl <jj@chaosbits.net> http://www.chaosbits.net/
> Don't top-post http://www.catb.org/jargon/html/T/top-post.html
> Plain text mails only, please.
>
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Ext4 Kernel Crash
@ 2012-02-26 12:18 Nanakos Chrysostomos
0 siblings, 0 replies; 5+ messages in thread
From: Nanakos Chrysostomos @ 2012-02-26 12:18 UTC (permalink / raw)
To: tytso; +Cc: linux-ext4, linux-kernel
Hi Ted,
I tried a newer kernel and for the moment there are no problems.
I am testing our systems this weekend and the new week. I will
let you know the results.
Kind regards,
Chris
On Mon, Jan 30, 2012 at 12:26:59PM -0500, Ted Ts'o wrote:
> On Sun, Jan 29, 2012 at 07:50:57PM +0200, Nanakos Chrysostomos wrote:
> > Dear Linux Kernel Developers,
> > I am writing to report a kernel or ext4 bug.
>
> Hi Nanakos,
>
> My apologies for not responding to the message that you sent to me
> personally three days ago, but I was on vacation in Hawaii; the best
> place to send queries about ext4 file systems is to use the
> linux-ext4@vger.kernel.org mailing list.
>
> One thing that I would immediately suggest is to see whether you can
> reproduce this bug using a modern kernel. You are currently using a
> RHEL6-derived kernel (2.6.32-220.4.1.el6.x86_64). The ext4 file
> system in such a kernel is quite different from what is in upstream,
> and there were a large number of bugs that have been fixed in ext4
> since 2.6.32. Many of these fixes were backported into the RHEL6
> kernel, but I'm not an expert on which fixes have been backported into
> RHEL6 and which haven't. (Especially since Red Hat doesn't release
> broken-out patches for their RHEL kernels....)
>
> My understanding is that Gluster doesn't require any kernel-side
> patches, so it should be OK to run Gluster on a v3.2 kernel.
> Alternatively, if you are a paid customer of RHEL6, I'd suggest
> sending a support request to Red Hat.
>
> Best regards,
>
> - Ted
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2012-02-26 12:18 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2012-01-29 17:50 Ext4 Kernel Crash Nanakos Chrysostomos
2012-01-29 22:03 ` Jesper Juhl
2012-01-31 8:34 ` Nanakos V. Chrysostomos
2012-01-30 17:26 ` Ted Ts'o
2012-02-26 12:18 Nanakos Chrysostomos
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).