* Ceph hard lock Hammer 9.2
@ 2015-06-22 20:45 Barclay Jameson
2015-06-23 11:54 ` Gregory Farnum
0 siblings, 1 reply; 5+ messages in thread
From: Barclay Jameson @ 2015-06-22 20:45 UTC (permalink / raw)
To: ceph-devel
Has anyone seen this?
Jun 22 15:09:27 node kernel: Call Trace:
Jun 22 15:09:27 node kernel: [<ffffffff816803ee>] schedule+0x3e/0x90
Jun 22 15:09:27 node kernel: [<ffffffff8168062e>]
schedule_preempt_disabled+0xe/0x10
Jun 22 15:09:27 node kernel: [<ffffffff81681ce3>]
__mutex_lock_slowpath+0x93/0x100
Jun 22 15:09:27 node kernel: [<ffffffffa060def8>] ?
__cap_is_valid+0x58/0x70 [ceph]
Jun 22 15:09:27 node kernel: [<ffffffff81681d73>] mutex_lock+0x23/0x40
Jun 22 15:09:27 node kernel: [<ffffffffa0610f2d>]
ceph_check_caps+0x38d/0x780 [ceph]
Jun 22 15:09:27 node kernel: [<ffffffff812f5a9b>] ?
__radix_tree_delete_node+0x7b/0x130
Jun 22 15:09:27 node kernel: [<ffffffffa0612637>]
ceph_put_wrbuffer_cap_refs+0xf7/0x240 [ceph]
Jun 22 15:09:27 node kernel: [<ffffffffa060b170>]
writepages_finish+0x200/0x290 [ceph]
Jun 22 15:09:27 node kernel: [<ffffffffa05e2731>]
handle_reply+0x4f1/0x640 [libceph]
Jun 22 15:09:27 node kernel: [<ffffffffa05e3065>] dispatch+0x85/0xa0 [libceph]
Jun 22 15:09:27 node kernel: [<ffffffffa05d7ceb>]
process_message+0xab/0xd0 [libceph]
Jun 22 15:09:27 node kernel: [<ffffffffa05db052>] try_read+0x2d2/0x430 [libceph]
Jun 22 15:09:27 node kernel: [<ffffffffa05db7e8>] con_work+0x78/0x220 [libceph]
Jun 22 15:09:27 node kernel: [<ffffffff8108c475>] process_one_work+0x145/0x460
Jun 22 15:09:27 node kernel: [<ffffffff8108c8b2>] worker_thread+0x122/0x420
Jun 22 15:09:27 node kernel: [<ffffffff8167fdb8>] ? __schedule+0x398/0x840
Jun 22 15:09:27 node kernel: [<ffffffff8108c790>] ? process_one_work+0x460/0x460
Jun 22 15:09:27 node kernel: [<ffffffff8108c790>] ? process_one_work+0x460/0x460
Jun 22 15:09:27 node kernel: [<ffffffff8109170e>] kthread+0xce/0xf0
Jun 22 15:09:27 node kernel: [<ffffffff81091640>] ?
kthread_freezable_should_stop+0x70/0x70
Jun 22 15:09:27 node kernel: [<ffffffff81683dd8>] ret_from_fork+0x58/0x90
Jun 22 15:09:27 node kernel: [<ffffffff81091640>] ?
kthread_freezable_should_stop+0x70/0x70
Jun 22 15:11:27 node kernel: INFO: task kworker/2:1:40 blocked for
more than 120 seconds.
Jun 22 15:11:27 node kernel: Tainted: G I
4.0.4-1.el6.elrepo.x86_64 #1
Jun 22 15:11:27 node kernel: "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun 22 15:11:27 node kernel: kworker/2:1 D ffff881ff279f7f8 0
40 2 0x00000000
Jun 22 15:11:27 node kernel: Workqueue: ceph-msgr con_work [libceph]
Jun 22 15:11:27 node kernel: ffff881ff279f7f8 ffff881ff261c010
ffff881ff2b67050 ffff88207fd95270
Jun 22 15:11:27 node kernel: ffff881ff279c010 ffff88207fd15200
7fffffffffffffff 0000000000000002
Jun 22 15:11:27 node kernel: ffffffff81680ae0 ffff881ff279f818
ffffffff816803ee ffffffff810ae63b
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Ceph hard lock Hammer 9.2
2015-06-22 20:45 Ceph hard lock Hammer 9.2 Barclay Jameson
@ 2015-06-23 11:54 ` Gregory Farnum
2015-06-23 14:25 ` Barclay Jameson
0 siblings, 1 reply; 5+ messages in thread
From: Gregory Farnum @ 2015-06-23 11:54 UTC (permalink / raw)
To: Barclay Jameson; +Cc: ceph-devel, 严正
On Mon, Jun 22, 2015 at 9:45 PM, Barclay Jameson
<almightybeeij@gmail.com> wrote:
> Has anyone seen this?
Can you describe the kernel you're using, the workload you were
running, the Ceph cluster you're running against, etc?
>
> Jun 22 15:09:27 node kernel: Call Trace:
> Jun 22 15:09:27 node kernel: [<ffffffff816803ee>] schedule+0x3e/0x90
> Jun 22 15:09:27 node kernel: [<ffffffff8168062e>]
> schedule_preempt_disabled+0xe/0x10
> Jun 22 15:09:27 node kernel: [<ffffffff81681ce3>]
> __mutex_lock_slowpath+0x93/0x100
> Jun 22 15:09:27 node kernel: [<ffffffffa060def8>] ?
> __cap_is_valid+0x58/0x70 [ceph]
> Jun 22 15:09:27 node kernel: [<ffffffff81681d73>] mutex_lock+0x23/0x40
> Jun 22 15:09:27 node kernel: [<ffffffffa0610f2d>]
> ceph_check_caps+0x38d/0x780 [ceph]
> Jun 22 15:09:27 node kernel: [<ffffffff812f5a9b>] ?
> __radix_tree_delete_node+0x7b/0x130
> Jun 22 15:09:27 node kernel: [<ffffffffa0612637>]
> ceph_put_wrbuffer_cap_refs+0xf7/0x240 [ceph]
> Jun 22 15:09:27 node kernel: [<ffffffffa060b170>]
> writepages_finish+0x200/0x290 [ceph]
> Jun 22 15:09:27 node kernel: [<ffffffffa05e2731>]
> handle_reply+0x4f1/0x640 [libceph]
> Jun 22 15:09:27 node kernel: [<ffffffffa05e3065>] dispatch+0x85/0xa0 [libceph]
> Jun 22 15:09:27 node kernel: [<ffffffffa05d7ceb>]
> process_message+0xab/0xd0 [libceph]
> Jun 22 15:09:27 node kernel: [<ffffffffa05db052>] try_read+0x2d2/0x430 [libceph]
> Jun 22 15:09:27 node kernel: [<ffffffffa05db7e8>] con_work+0x78/0x220 [libceph]
> Jun 22 15:09:27 node kernel: [<ffffffff8108c475>] process_one_work+0x145/0x460
> Jun 22 15:09:27 node kernel: [<ffffffff8108c8b2>] worker_thread+0x122/0x420
> Jun 22 15:09:27 node kernel: [<ffffffff8167fdb8>] ? __schedule+0x398/0x840
> Jun 22 15:09:27 node kernel: [<ffffffff8108c790>] ? process_one_work+0x460/0x460
> Jun 22 15:09:27 node kernel: [<ffffffff8108c790>] ? process_one_work+0x460/0x460
> Jun 22 15:09:27 node kernel: [<ffffffff8109170e>] kthread+0xce/0xf0
> Jun 22 15:09:27 node kernel: [<ffffffff81091640>] ?
> kthread_freezable_should_stop+0x70/0x70
> Jun 22 15:09:27 node kernel: [<ffffffff81683dd8>] ret_from_fork+0x58/0x90
> Jun 22 15:09:27 node kernel: [<ffffffff81091640>] ?
> kthread_freezable_should_stop+0x70/0x70
> Jun 22 15:11:27 node kernel: INFO: task kworker/2:1:40 blocked for
> more than 120 seconds.
> Jun 22 15:11:27 node kernel: Tainted: G I
> 4.0.4-1.el6.elrepo.x86_64 #1
> Jun 22 15:11:27 node kernel: "echo 0 >
> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Jun 22 15:11:27 node kernel: kworker/2:1 D ffff881ff279f7f8 0
> 40 2 0x00000000
> Jun 22 15:11:27 node kernel: Workqueue: ceph-msgr con_work [libceph]
> Jun 22 15:11:27 node kernel: ffff881ff279f7f8 ffff881ff261c010
> ffff881ff2b67050 ffff88207fd95270
> Jun 22 15:11:27 node kernel: ffff881ff279c010 ffff88207fd15200
> 7fffffffffffffff 0000000000000002
> Jun 22 15:11:27 node kernel: ffffffff81680ae0 ffff881ff279f818
> ffffffff816803ee ffffffff810ae63b
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Ceph hard lock Hammer 9.2
2015-06-23 11:54 ` Gregory Farnum
@ 2015-06-23 14:25 ` Barclay Jameson
2015-06-24 13:48 ` Yan, Zheng
0 siblings, 1 reply; 5+ messages in thread
From: Barclay Jameson @ 2015-06-23 14:25 UTC (permalink / raw)
To: Gregory Farnum; +Cc: ceph-devel, 严正
Sure,
I guess it's actually a soft kernel lock since it's only the
filesystem that is hung with high IO wait.
The kernel is 4.0.4-1.el6.elrepo.x86.
The Ceph version is 0.94.2 (Sorry about the confusion I missed a 4
when I typed in the subject line).
I was testing copying 100,000 files from directory (dir1) to
(dir1-`hostname`) on three septate hosts.
2 of the hosts completed the job and the third one hung with the stack
trace in /var/log/messages.
On Tue, Jun 23, 2015 at 6:54 AM, Gregory Farnum <greg@gregs42.com> wrote:
> On Mon, Jun 22, 2015 at 9:45 PM, Barclay Jameson
> <almightybeeij@gmail.com> wrote:
>> Has anyone seen this?
>
> Can you describe the kernel you're using, the workload you were
> running, the Ceph cluster you're running against, etc?
>
>>
>> Jun 22 15:09:27 node kernel: Call Trace:
>> Jun 22 15:09:27 node kernel: [<ffffffff816803ee>] schedule+0x3e/0x90
>> Jun 22 15:09:27 node kernel: [<ffffffff8168062e>]
>> schedule_preempt_disabled+0xe/0x10
>> Jun 22 15:09:27 node kernel: [<ffffffff81681ce3>]
>> __mutex_lock_slowpath+0x93/0x100
>> Jun 22 15:09:27 node kernel: [<ffffffffa060def8>] ?
>> __cap_is_valid+0x58/0x70 [ceph]
>> Jun 22 15:09:27 node kernel: [<ffffffff81681d73>] mutex_lock+0x23/0x40
>> Jun 22 15:09:27 node kernel: [<ffffffffa0610f2d>]
>> ceph_check_caps+0x38d/0x780 [ceph]
>> Jun 22 15:09:27 node kernel: [<ffffffff812f5a9b>] ?
>> __radix_tree_delete_node+0x7b/0x130
>> Jun 22 15:09:27 node kernel: [<ffffffffa0612637>]
>> ceph_put_wrbuffer_cap_refs+0xf7/0x240 [ceph]
>> Jun 22 15:09:27 node kernel: [<ffffffffa060b170>]
>> writepages_finish+0x200/0x290 [ceph]
>> Jun 22 15:09:27 node kernel: [<ffffffffa05e2731>]
>> handle_reply+0x4f1/0x640 [libceph]
>> Jun 22 15:09:27 node kernel: [<ffffffffa05e3065>] dispatch+0x85/0xa0 [libceph]
>> Jun 22 15:09:27 node kernel: [<ffffffffa05d7ceb>]
>> process_message+0xab/0xd0 [libceph]
>> Jun 22 15:09:27 node kernel: [<ffffffffa05db052>] try_read+0x2d2/0x430 [libceph]
>> Jun 22 15:09:27 node kernel: [<ffffffffa05db7e8>] con_work+0x78/0x220 [libceph]
>> Jun 22 15:09:27 node kernel: [<ffffffff8108c475>] process_one_work+0x145/0x460
>> Jun 22 15:09:27 node kernel: [<ffffffff8108c8b2>] worker_thread+0x122/0x420
>> Jun 22 15:09:27 node kernel: [<ffffffff8167fdb8>] ? __schedule+0x398/0x840
>> Jun 22 15:09:27 node kernel: [<ffffffff8108c790>] ? process_one_work+0x460/0x460
>> Jun 22 15:09:27 node kernel: [<ffffffff8108c790>] ? process_one_work+0x460/0x460
>> Jun 22 15:09:27 node kernel: [<ffffffff8109170e>] kthread+0xce/0xf0
>> Jun 22 15:09:27 node kernel: [<ffffffff81091640>] ?
>> kthread_freezable_should_stop+0x70/0x70
>> Jun 22 15:09:27 node kernel: [<ffffffff81683dd8>] ret_from_fork+0x58/0x90
>> Jun 22 15:09:27 node kernel: [<ffffffff81091640>] ?
>> kthread_freezable_should_stop+0x70/0x70
>> Jun 22 15:11:27 node kernel: INFO: task kworker/2:1:40 blocked for
>> more than 120 seconds.
>> Jun 22 15:11:27 node kernel: Tainted: G I
>> 4.0.4-1.el6.elrepo.x86_64 #1
>> Jun 22 15:11:27 node kernel: "echo 0 >
>> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>> Jun 22 15:11:27 node kernel: kworker/2:1 D ffff881ff279f7f8 0
>> 40 2 0x00000000
>> Jun 22 15:11:27 node kernel: Workqueue: ceph-msgr con_work [libceph]
>> Jun 22 15:11:27 node kernel: ffff881ff279f7f8 ffff881ff261c010
>> ffff881ff2b67050 ffff88207fd95270
>> Jun 22 15:11:27 node kernel: ffff881ff279c010 ffff88207fd15200
>> 7fffffffffffffff 0000000000000002
>> Jun 22 15:11:27 node kernel: ffffffff81680ae0 ffff881ff279f818
>> ffffffff816803ee ffffffff810ae63b
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Ceph hard lock Hammer 9.2
2015-06-23 14:25 ` Barclay Jameson
@ 2015-06-24 13:48 ` Yan, Zheng
2015-06-25 14:11 ` Barclay Jameson
0 siblings, 1 reply; 5+ messages in thread
From: Yan, Zheng @ 2015-06-24 13:48 UTC (permalink / raw)
To: Barclay Jameson; +Cc: Gregory Farnum, ceph-devel, 严正
Could you please run "echo 1 > /proc/sys/kernel/sysrq; echo t >
/proc/sysrq-trigger" when this warning happens again. then send the
kernel message to us.
Regards
Yan, Zheng
On Tue, Jun 23, 2015 at 10:25 PM, Barclay Jameson
<almightybeeij@gmail.com> wrote:
> Sure,
> I guess it's actually a soft kernel lock since it's only the
> filesystem that is hung with high IO wait.
> The kernel is 4.0.4-1.el6.elrepo.x86.
> The Ceph version is 0.94.2 (Sorry about the confusion I missed a 4
> when I typed in the subject line).
> I was testing copying 100,000 files from directory (dir1) to
> (dir1-`hostname`) on three septate hosts.
> 2 of the hosts completed the job and the third one hung with the stack
> trace in /var/log/messages.
>
> On Tue, Jun 23, 2015 at 6:54 AM, Gregory Farnum <greg@gregs42.com> wrote:
>> On Mon, Jun 22, 2015 at 9:45 PM, Barclay Jameson
>> <almightybeeij@gmail.com> wrote:
>>> Has anyone seen this?
>>
>> Can you describe the kernel you're using, the workload you were
>> running, the Ceph cluster you're running against, etc?
>>
>>>
>>> Jun 22 15:09:27 node kernel: Call Trace:
>>> Jun 22 15:09:27 node kernel: [<ffffffff816803ee>] schedule+0x3e/0x90
>>> Jun 22 15:09:27 node kernel: [<ffffffff8168062e>]
>>> schedule_preempt_disabled+0xe/0x10
>>> Jun 22 15:09:27 node kernel: [<ffffffff81681ce3>]
>>> __mutex_lock_slowpath+0x93/0x100
>>> Jun 22 15:09:27 node kernel: [<ffffffffa060def8>] ?
>>> __cap_is_valid+0x58/0x70 [ceph]
>>> Jun 22 15:09:27 node kernel: [<ffffffff81681d73>] mutex_lock+0x23/0x40
>>> Jun 22 15:09:27 node kernel: [<ffffffffa0610f2d>]
>>> ceph_check_caps+0x38d/0x780 [ceph]
>>> Jun 22 15:09:27 node kernel: [<ffffffff812f5a9b>] ?
>>> __radix_tree_delete_node+0x7b/0x130
>>> Jun 22 15:09:27 node kernel: [<ffffffffa0612637>]
>>> ceph_put_wrbuffer_cap_refs+0xf7/0x240 [ceph]
>>> Jun 22 15:09:27 node kernel: [<ffffffffa060b170>]
>>> writepages_finish+0x200/0x290 [ceph]
>>> Jun 22 15:09:27 node kernel: [<ffffffffa05e2731>]
>>> handle_reply+0x4f1/0x640 [libceph]
>>> Jun 22 15:09:27 node kernel: [<ffffffffa05e3065>] dispatch+0x85/0xa0 [libceph]
>>> Jun 22 15:09:27 node kernel: [<ffffffffa05d7ceb>]
>>> process_message+0xab/0xd0 [libceph]
>>> Jun 22 15:09:27 node kernel: [<ffffffffa05db052>] try_read+0x2d2/0x430 [libceph]
>>> Jun 22 15:09:27 node kernel: [<ffffffffa05db7e8>] con_work+0x78/0x220 [libceph]
>>> Jun 22 15:09:27 node kernel: [<ffffffff8108c475>] process_one_work+0x145/0x460
>>> Jun 22 15:09:27 node kernel: [<ffffffff8108c8b2>] worker_thread+0x122/0x420
>>> Jun 22 15:09:27 node kernel: [<ffffffff8167fdb8>] ? __schedule+0x398/0x840
>>> Jun 22 15:09:27 node kernel: [<ffffffff8108c790>] ? process_one_work+0x460/0x460
>>> Jun 22 15:09:27 node kernel: [<ffffffff8108c790>] ? process_one_work+0x460/0x460
>>> Jun 22 15:09:27 node kernel: [<ffffffff8109170e>] kthread+0xce/0xf0
>>> Jun 22 15:09:27 node kernel: [<ffffffff81091640>] ?
>>> kthread_freezable_should_stop+0x70/0x70
>>> Jun 22 15:09:27 node kernel: [<ffffffff81683dd8>] ret_from_fork+0x58/0x90
>>> Jun 22 15:09:27 node kernel: [<ffffffff81091640>] ?
>>> kthread_freezable_should_stop+0x70/0x70
>>> Jun 22 15:11:27 node kernel: INFO: task kworker/2:1:40 blocked for
>>> more than 120 seconds.
>>> Jun 22 15:11:27 node kernel: Tainted: G I
>>> 4.0.4-1.el6.elrepo.x86_64 #1
>>> Jun 22 15:11:27 node kernel: "echo 0 >
>>> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>> Jun 22 15:11:27 node kernel: kworker/2:1 D ffff881ff279f7f8 0
>>> 40 2 0x00000000
>>> Jun 22 15:11:27 node kernel: Workqueue: ceph-msgr con_work [libceph]
>>> Jun 22 15:11:27 node kernel: ffff881ff279f7f8 ffff881ff261c010
>>> ffff881ff2b67050 ffff88207fd95270
>>> Jun 22 15:11:27 node kernel: ffff881ff279c010 ffff88207fd15200
>>> 7fffffffffffffff 0000000000000002
>>> Jun 22 15:11:27 node kernel: ffffffff81680ae0 ffff881ff279f818
>>> ffffffff816803ee ffffffff810ae63b
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Ceph hard lock Hammer 9.2
2015-06-24 13:48 ` Yan, Zheng
@ 2015-06-25 14:11 ` Barclay Jameson
0 siblings, 0 replies; 5+ messages in thread
From: Barclay Jameson @ 2015-06-25 14:11 UTC (permalink / raw)
To: Yan, Zheng; +Cc: Gregory Farnum, ceph-devel, 严正
If I get it to happen again I will send you the kernel message.
Thanks again Zheng!
On Wed, Jun 24, 2015 at 8:48 AM, Yan, Zheng <ukernel@gmail.com> wrote:
> Could you please run "echo 1 > /proc/sys/kernel/sysrq; echo t >
> /proc/sysrq-trigger" when this warning happens again. then send the
> kernel message to us.
>
> Regards
> Yan, Zheng
>
> On Tue, Jun 23, 2015 at 10:25 PM, Barclay Jameson
> <almightybeeij@gmail.com> wrote:
>> Sure,
>> I guess it's actually a soft kernel lock since it's only the
>> filesystem that is hung with high IO wait.
>> The kernel is 4.0.4-1.el6.elrepo.x86.
>> The Ceph version is 0.94.2 (Sorry about the confusion I missed a 4
>> when I typed in the subject line).
>> I was testing copying 100,000 files from directory (dir1) to
>> (dir1-`hostname`) on three septate hosts.
>> 2 of the hosts completed the job and the third one hung with the stack
>> trace in /var/log/messages.
>>
>> On Tue, Jun 23, 2015 at 6:54 AM, Gregory Farnum <greg@gregs42.com> wrote:
>>> On Mon, Jun 22, 2015 at 9:45 PM, Barclay Jameson
>>> <almightybeeij@gmail.com> wrote:
>>>> Has anyone seen this?
>>>
>>> Can you describe the kernel you're using, the workload you were
>>> running, the Ceph cluster you're running against, etc?
>>>
>>>>
>>>> Jun 22 15:09:27 node kernel: Call Trace:
>>>> Jun 22 15:09:27 node kernel: [<ffffffff816803ee>] schedule+0x3e/0x90
>>>> Jun 22 15:09:27 node kernel: [<ffffffff8168062e>]
>>>> schedule_preempt_disabled+0xe/0x10
>>>> Jun 22 15:09:27 node kernel: [<ffffffff81681ce3>]
>>>> __mutex_lock_slowpath+0x93/0x100
>>>> Jun 22 15:09:27 node kernel: [<ffffffffa060def8>] ?
>>>> __cap_is_valid+0x58/0x70 [ceph]
>>>> Jun 22 15:09:27 node kernel: [<ffffffff81681d73>] mutex_lock+0x23/0x40
>>>> Jun 22 15:09:27 node kernel: [<ffffffffa0610f2d>]
>>>> ceph_check_caps+0x38d/0x780 [ceph]
>>>> Jun 22 15:09:27 node kernel: [<ffffffff812f5a9b>] ?
>>>> __radix_tree_delete_node+0x7b/0x130
>>>> Jun 22 15:09:27 node kernel: [<ffffffffa0612637>]
>>>> ceph_put_wrbuffer_cap_refs+0xf7/0x240 [ceph]
>>>> Jun 22 15:09:27 node kernel: [<ffffffffa060b170>]
>>>> writepages_finish+0x200/0x290 [ceph]
>>>> Jun 22 15:09:27 node kernel: [<ffffffffa05e2731>]
>>>> handle_reply+0x4f1/0x640 [libceph]
>>>> Jun 22 15:09:27 node kernel: [<ffffffffa05e3065>] dispatch+0x85/0xa0 [libceph]
>>>> Jun 22 15:09:27 node kernel: [<ffffffffa05d7ceb>]
>>>> process_message+0xab/0xd0 [libceph]
>>>> Jun 22 15:09:27 node kernel: [<ffffffffa05db052>] try_read+0x2d2/0x430 [libceph]
>>>> Jun 22 15:09:27 node kernel: [<ffffffffa05db7e8>] con_work+0x78/0x220 [libceph]
>>>> Jun 22 15:09:27 node kernel: [<ffffffff8108c475>] process_one_work+0x145/0x460
>>>> Jun 22 15:09:27 node kernel: [<ffffffff8108c8b2>] worker_thread+0x122/0x420
>>>> Jun 22 15:09:27 node kernel: [<ffffffff8167fdb8>] ? __schedule+0x398/0x840
>>>> Jun 22 15:09:27 node kernel: [<ffffffff8108c790>] ? process_one_work+0x460/0x460
>>>> Jun 22 15:09:27 node kernel: [<ffffffff8108c790>] ? process_one_work+0x460/0x460
>>>> Jun 22 15:09:27 node kernel: [<ffffffff8109170e>] kthread+0xce/0xf0
>>>> Jun 22 15:09:27 node kernel: [<ffffffff81091640>] ?
>>>> kthread_freezable_should_stop+0x70/0x70
>>>> Jun 22 15:09:27 node kernel: [<ffffffff81683dd8>] ret_from_fork+0x58/0x90
>>>> Jun 22 15:09:27 node kernel: [<ffffffff81091640>] ?
>>>> kthread_freezable_should_stop+0x70/0x70
>>>> Jun 22 15:11:27 node kernel: INFO: task kworker/2:1:40 blocked for
>>>> more than 120 seconds.
>>>> Jun 22 15:11:27 node kernel: Tainted: G I
>>>> 4.0.4-1.el6.elrepo.x86_64 #1
>>>> Jun 22 15:11:27 node kernel: "echo 0 >
>>>> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>>> Jun 22 15:11:27 node kernel: kworker/2:1 D ffff881ff279f7f8 0
>>>> 40 2 0x00000000
>>>> Jun 22 15:11:27 node kernel: Workqueue: ceph-msgr con_work [libceph]
>>>> Jun 22 15:11:27 node kernel: ffff881ff279f7f8 ffff881ff261c010
>>>> ffff881ff2b67050 ffff88207fd95270
>>>> Jun 22 15:11:27 node kernel: ffff881ff279c010 ffff88207fd15200
>>>> 7fffffffffffffff 0000000000000002
>>>> Jun 22 15:11:27 node kernel: ffffffff81680ae0 ffff881ff279f818
>>>> ffffffff816803ee ffffffff810ae63b
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2015-06-25 14:11 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-06-22 20:45 Ceph hard lock Hammer 9.2 Barclay Jameson
2015-06-23 11:54 ` Gregory Farnum
2015-06-23 14:25 ` Barclay Jameson
2015-06-24 13:48 ` Yan, Zheng
2015-06-25 14:11 ` Barclay Jameson
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.