WARN_ON(!rt_rq->rt_nr_running) question?

WARN_ON(!rt_rq->rt_nr_running) question?

[raja gobi]

[-- Attachment #1: Type: text/plain, Size: 877 bytes --]


Hi all,

This is a question about sched_rt.c. I am hoping this is the correct mailing list for such questions. If not, could somebody direct me to the correct place?

I am using 2.6.25.16 on an MX27 (ARM)

While running a process with 3 threads using RT scheduler (priorities at RT, -93 and -87), get  the WARN_ON()

WARN_ON(!rt_rq->rt_nr_running) 

I also see

WARN_ON(rt_se_prio(rt_se) < rt_rq->highest_prio);

Both
these warnings are very rare. This happens after running the
application for several hours (> 6 hours) to a small number of days
(1 or 2 days)

CONFIG_SMP is not enabled
CONFIG_RT_GROUP_SCHED is enabled

As it happens very rarely, it has been hard to track it down. Does anybody know of any issues in this area in 2.6.25.16 and if yes, what is the best way to address this?

I am attaching the dump_stack() output.

Thanks much for your help,
-raja.


      

[-- Attachment #2: warn_on_slowpath.txt --]
[-- Type: text/plain, Size: 11806 bytes --]

------------[ cut here ]------------
WARNING: at kernel/sched_rt.c:306 dequeue_rt_entity+0x204/0x21c()
Modules linked in: ddriver
[<c0022524>] (dump_stack+0x0/0x14) from [<c0034294>] (warn_on_slowpath+0x4c/0x60)
[<c0034248>] (warn_on_slowpath+0x0/0x60) from [<c002c4e4>] (dequeue_rt_entity+0x204/0x21c)
 r6:c02c3d90 r5:00000000 r4:c02c3998
[<c002c2e0>] (dequeue_rt_entity+0x0/0x21c) from [<c002c540>] (dequeue_rt_stack+0x44/0x50)
 r6:c7c4a8c0 r5:00000fad r4:c7c4aa18
[<c002c4fc>] (dequeue_rt_stack+0x0/0x50) from [<c002c910>] (enqueue_task_rt+0x28/0x48)
 r4:c7c4aa18
[<c002c8e8>] (enqueue_task_rt+0x0/0x48) from [<c002b6d4>] (enqueue_task+0x44/0x58)
 r4:122934e0
[<c002b690>] (enqueue_task+0x0/0x58) from [<c002b714>] (activate_task+0x2c/0x3c)
 r6:c02c3d18 r5:c7c4a8c0 r4:c02c3d18
[<c002b6e8>] (activate_task+0x0/0x3c) from [<c002fee0>] (try_to_wake_up+0x108/0x130)
 r5:c7c4a8c0 r4:00000001
[<c002fdd8>] (try_to_wake_up+0x0/0x130) from [<c002ff1c>] (default_wake_function+0x14/0x18)
 r9:00000000 r8:00000000 r7:00000000 r6:bf018bcc r5:00000000
r4:c7fe5ea8
[<c002ff08>] (default_wake_function+0x0/0x18) from [<c004b4ec>] (autoremove_wake_function+0x18/0x40)
[<c004b4d4>] (autoremove_wake_function+0x0/0x40) from [<c002b2f0>] (__wake_up_common+0x58/0x88)
 r4:bf018bc0
[<c002b298>] (__wake_up_common+0x0/0x88) from [<c002dd3c>] (__wake_up+0x4c/0x7c)
[<c002dcf0>] (__wake_up+0x0/0x7c) from [<bf002f48>] (ddrivermbox_event_reader+0x1f0/0x3b4 [ddriver])
 r8:00000000 r7:00000000 r6:00000001 r5:00000001 r4:bf018ae8
[<bf002d58>] (ddriver_event_reader+0x0/0x3b4 [ddriver]) from [<c0046fd8>] (run_workqueue+0xd8/0x1a0)
[<c0046f00>] (run_workqueue+0x0/0x1a0) from [<c0047bf8>] (worker_thread+0x78/0xd0)
 r8:00000000 r7:00000000 r6:c7c27fac r5:c7c019e8 r4:c7c019e0
[<c0047b80>] (worker_thread+0x0/0xd0) from [<c004b120>] (kthread+0x50/0x7c)
 r6:00000000 r5:c0047b80 r4:c7c019e0
[<c004b0d0>] (kthread+0x0/0x7c) from [<c00381bc>] (do_exit+0x0/0x6c0)
 r5:00000000 r4:00000000
---[ end trace 84135610baf6d9d3 ]---
------------[ cut here ]------------
WARNING: at kernel/sched_rt.c:312 dequeue_rt_entity+0x1a8/0x21c()
Modules linked in: ddriver
[<c0022524>] (dump_stack+0x0/0x14) from [<c0034294>] (warn_on_slowpath+0x4c/0x60)
[<c0034248>] (warn_on_slowpath+0x0/0x60) from [<c002c488>] (dequeue_rt_entity+0x1a8/0x21c)
 r6:c02c3d90 r5:ffffffff r4:c02c3998
[<c002c2e0>] (dequeue_rt_entity+0x0/0x21c) from [<c002c540>] (dequeue_rt_stack+0x44/0x50)
 r6:c7c4a8c0 r5:00000fad r4:c7c4aa18
[<c002c4fc>] (dequeue_rt_stack+0x0/0x50) from [<c002c910>] (enqueue_task_rt+0x28/0x48)
 r4:c7c4aa18
[<c002c8e8>] (enqueue_task_rt+0x0/0x48) from [<c002b6d4>] (enqueue_task+0x44/0x58)
 r4:122934e0
[<c002b690>] (enqueue_task+0x0/0x58) from [<c002b714>] (activate_task+0x2c/0x3c)
 r6:c02c3d18 r5:c7c4a8c0 r4:c02c3d18
[<c002b6e8>] (activate_task+0x0/0x3c) from [<c002fee0>] (try_to_wake_up+0x108/0x130)
 r5:c7c4a8c0 r4:00000001
[<c002fdd8>] (try_to_wake_up+0x0/0x130) from [<c002ff1c>] (default_wake_function+0x14/0x18)
 r9:00000000 r8:00000000 r7:00000000 r6:bf018bcc r5:00000000
r4:c7fe5ea8
[<c002ff08>] (default_wake_function+0x0/0x18) from [<c004b4ec>] (autoremove_wake_function+0x18/0x40)
[<c004b4d4>] (autoremove_wake_function+0x0/0x40) from [<c002b2f0>] (__wake_up_common+0x58/0x88)
 r4:bf018bc0
[<c002b298>] (__wake_up_common+0x0/0x88) from [<c002dd3c>] (__wake_up+0x4c/0x7c)
[<c002dcf0>] (__wake_up+0x0/0x7c) from [<bf002f48>] (ddrivermbox_event_reader+0x1f0/0x3b4 [ddriver])
 r8:00000000 r7:00000000 r6:00000001 r5:00000001 r4:bf018ae8
[<bf002d58>] (ddrivermbox_event_reader+0x0/0x3b4 [ddriver]) from [<c0046fd8>] (run_workqueue+0xd8/0x1a0)
[<c0046f00>] (run_workqueue+0x0/0x1a0) from [<c0047bf8>] (worker_thread+0x78/0xd0)
 r8:00000000 r7:00000000 r6:c7c27fac r5:c7c019e8 r4:c7c019e0
[<c0047b80>] (worker_thread+0x0/0xd0) from [<c004b120>] (kthread+0x50/0x7c)
 r6:00000000 r5:c0047b80 r4:c7c019e0
[<c004b0d0>] (kthread+0x0/0x7c) from [<c00381bc>] (do_exit+0x0/0x6c0)
 r5:00000000 r4:00000000
---[ end trace 84135610baf6d9d3 ]---
------------[ cut here ]------------
WARNING: at kernel/sched_rt.c:306 dequeue_rt_entity+0x204/0x21c()
Modules linked in: ddriver
[<c0022524>] (dump_stack+0x0/0x14) from [<c0034294>] (warn_on_slowpath+0x4c/0x60)
[<c0034248>] (warn_on_slowpath+0x0/0x60) from [<c002c4e4>] (dequeue_rt_entity+0x204/0x21c)
 r6:c02c3d90 r5:00000000 r4:c02c3998
[<c002c2e0>] (dequeue_rt_entity+0x0/0x21c) from [<c002c540>] (dequeue_rt_stack+0x44/0x50)
 r6:c7cac540 r5:00000fad r4:c7cac698
[<c002c4fc>] (dequeue_rt_stack+0x0/0x50) from [<c002c910>] (enqueue_task_rt+0x28/0x48)
 r4:c7cac698
[<c002c8e8>] (enqueue_task_rt+0x0/0x48) from [<c002b6d4>] (enqueue_task+0x44/0x58)
 r4:12c1cb60
[<c002b690>] (enqueue_task+0x0/0x58) from [<c002b714>] (activate_task+0x2c/0x3c)
 r6:c02c3d18 r5:c7cac540 r4:c02c3d18
[<c002b6e8>] (activate_task+0x0/0x3c) from [<c002fee0>] (try_to_wake_up+0x108/0x130)
 r5:c7cac540 r4:0000000f
[<c002fdd8>] (try_to_wake_up+0x0/0x130) from [<c002ff50>] (wake_up_process+0x18/0x1c)
 r9:7fffffff r8:ffffffff r7:00005810 r6:c02b39e8 r5:c004ed58
r4:00000000
[<c002ff38>] (wake_up_process+0x0/0x1c) from [<c004ed84>] (hrtimer_wakeup+0x2c/0x34)
[<c004ed58>] (hrtimer_wakeup+0x0/0x34) from [<c004e968>] (__run_hrtimer+0xc8/0xe0)
 r4:c7feff18
[<c004e8a0>] (__run_hrtimer+0x0/0xe0) from [<c004f6d0>] (hrtimer_interrupt+0x15c/0x1fc)
 r6:082527d8 r5:c02b39e8 r4:c7feff18
[<c004f574>] (hrtimer_interrupt+0x0/0x1fc) from [<c0028d48>] (mxc_timer_interrupt+0x28/0x38)
[<c0028d20>] (mxc_timer_interrupt+0x0/0x38) from [<c005ecf0>] (handle_IRQ_event+0x44/0x84)
[<c005ecac>] (handle_IRQ_event+0x0/0x84) from [<c0060ba8>] (handle_level_irq+0xa8/0x158)
 r7:c02b2170 r6:0000001a r5:c7c26000 r4:c02b4510
[<c0060b00>] (handle_level_irq+0x0/0x158) from [<c001e048>] (__exception_text_start+0x48/0x60)
 r7:c7c27ec8 r6:c02d9c04 r5:c02b4510 r4:0000001a
[<c001e000>] (__exception_text_start+0x0/0x60) from [<c021d2a4>] (__irq_svc+0x44/0x6c)
Exception stack(0xc7c27e20 to 0xc7c27e68)
7e20: 00000032 c7d19420 c7c26000 60000013 c7d19420 c7c26000 00000032 c7d19420
7e40: 00000001 c7c26000 00000000 c7c27e84 c7c27e88 c7c27e68 c0060ba8 c005ecd8
7e60: 60000013 ffffffff
 r7:00010003 r6:001a0000 r5:d4040000 r4:ffffffff
[<c005ecac>] (handle_IRQ_event+0x0/0x84) from [<c0060ba8>] (handle_level_irq+0xa8/0x158)
 r7:c7d19420 r6:00000032 r5:c7c26000 r4:c02b4a50
[<c0060b00>] (handle_level_irq+0x0/0x158) from [<c001e048>] (__exception_text_start+0x48/0x60)
 r7:00000000 r6:c02d9c04 r5:c02b4a50 r4:00000032
[<c001e000>] (__exception_text_start+0x0/0x60) from [<c021d2a4>] (__irq_svc+0x44/0x6c)
Exception stack(0xc7c27ec8 to 0xc7c27f10)
7ec0:                   00000001 c7fe5eb4 bf018bcc bf018bcc bf018bcc 00000001
7ee0: 00000000 00000000 00000013 00000017 00000000 c7c27f3c bf018bc0 c7c27f10
7f00: c002ff04 c002dd40 60000013 ffffffff
 r7:00000002 r6:00320000 r5:d4040000 r4:ffffffff
[<c002dcf0>] (__wake_up+0x0/0x7c) from [<bf002f48>] (ddrivermbox_event_reader+0x1f0/0x3b4 [ddriver])
 r8:00000000 r7:00000000 r6:00000001 r5:00000001 r4:bf018ae8
[<bf002d58>] (ddrivermbox_event_reader+0x0/0x3b4 [ddriver]) from [<c0046fd8>] (run_workqueue+0xd8/0x1a0)
[<c0046f00>] (run_workqueue+0x0/0x1a0) from [<c0047bf8>] (worker_thread+0x78/0xd0)
 r8:00000000 r7:00000000 r6:c7c27fac r5:c7c019e8 r4:c7c019e0
[<c0047b80>] (worker_thread+0x0/0xd0) from [<c004b120>] (kthread+0x50/0x7c)
 r6:00000000 r5:c0047b80 r4:c7c019e0
[<c004b0d0>] (kthread+0x0/0x7c) from [<c00381bc>] (do_exit+0x0/0x6c0)
 r5:00000000 r4:00000000
---[ end trace 84135610baf6d9d3 ]---
------------[ cut here ]------------
WARNING: at kernel/sched_rt.c:306 dequeue_rt_entity+0x204/0x21c()
Modules linked in: ddriver
[<c0022524>] (dump_stack+0x0/0x14) from [<c0034294>] (warn_on_slowpath+0x4c/0x60)
[<c0034248>] (warn_on_slowpath+0x0/0x60) from [<c002c4e4>] (dequeue_rt_entity+0x204/0x21c)
 r6:c02c3d90 r5:00000000 r4:c02c3998
[<c002c2e0>] (dequeue_rt_entity+0x0/0x21c) from [<c002c540>] (dequeue_rt_stack+0x44/0x50)
 r6:c7c4b940 r5:00000fad r4:c7c4ba98
[<c002c4fc>] (dequeue_rt_stack+0x0/0x50) from [<c002c910>] (enqueue_task_rt+0x28/0x48)
 r4:c7c4ba98
[<c002c8e8>] (enqueue_task_rt+0x0/0x48) from [<c002b6d4>] (enqueue_task+0x44/0x58)
 r4:279e63e4
[<c002b690>] (enqueue_task+0x0/0x58) from [<c002b714>] (activate_task+0x2c/0x3c)
 r6:c02c3d18 r5:c7c4b940 r4:c02c3d18
[<c002b6e8>] (activate_task+0x0/0x3c) from [<c002fee0>] (try_to_wake_up+0x108/0x130)
 r5:c7c4b940 r4:00000003
[<c002fdd8>] (try_to_wake_up+0x0/0x130) from [<c002ff1c>] (default_wake_function+0x14/0x18)
 r9:00000000 r8:00000000 r7:00000000 r6:c7fcde64 r5:00000000
r4:c7fcde58
[<c002ff08>] (default_wake_function+0x0/0x18) from [<c002b2f0>] (__wake_up_common+0x58/0x88)
[<c002b298>] (__wake_up_common+0x0/0x88) from [<c002dd3c>] (__wake_up+0x4c/0x7c)
[<c002dcf0>] (__wake_up+0x0/0x7c) from [<c00573e0>] (wake_futex+0x4c/0x5c)
 r8:ffffffff r7:00000001 r6:c7e63bb4 r5:c02d7460 r4:c7fcde50
[<c0057394>] (wake_futex+0x0/0x5c) from [<c00577c4>] (futex_wake+0xf0/0x140)
 r4:c02d745c
[<c00576d4>] (futex_wake+0x0/0x140) from [<c005856c>] (do_futex+0xb4/0x9ac)
[<c00584b8>] (do_futex+0x0/0x9ac) from [<c0058f28>] (sys_futex+0xc4/0x150)
[<c0058e64>] (sys_futex+0x0/0x150) from [<c001e5a0>] (ret_fast_syscall+0x0/0x2c)
---[ end trace 84135610baf6d9d3 ]---
------------[ cut here ]------------
WARNING: at kernel/sched_rt.c:306 dequeue_rt_entity+0x204/0x21c()
Modules linked in: ddriver
[<c0022524>] (dump_stack+0x0/0x14) from [<c0034294>] (warn_on_slowpath+0x4c/0x60)
[<c0034248>] (warn_on_slowpath+0x0/0x60) from [<c002c4e4>] (dequeue_rt_entity+0x204/0x21c)
 r6:c02c3d90 r5:00000000 r4:c02c3998
[<c002c2e0>] (dequeue_rt_entity+0x0/0x21c) from [<c002c540>] (dequeue_rt_stack+0x44/0x50)
 r6:c7c4b520 r5:00000fad r4:c7c4b678
[<c002c4fc>] (dequeue_rt_stack+0x0/0x50) from [<c002c910>] (enqueue_task_rt+0x28/0x48)
 r4:c7c4b678
[<c002c8e8>] (enqueue_task_rt+0x0/0x48) from [<c002b6d4>] (enqueue_task+0x44/0x58)
 r4:3553ef49
[<c002b690>] (enqueue_task+0x0/0x58) from [<c002b714>] (activate_task+0x2c/0x3c)
 r6:c02c3d18 r5:c7c4b520 r4:c02c3d18
[<c002b6e8>] (activate_task+0x0/0x3c) from [<c002fee0>] (try_to_wake_up+0x108/0x130)
 r5:c7c4b520 r4:0000000f
[<c002fdd8>] (try_to_wake_up+0x0/0x130) from [<c002ff50>] (wake_up_process+0x18/0x1c)
 r9:7fffffff r8:ffffffff r7:00005812 r6:c02b39e8 r5:c004ed58
r4:00000000
[<c002ff38>] (wake_up_process+0x0/0x1c) from [<c004ed84>] (hrtimer_wakeup+0x2c/0x34)
[<c004ed58>] (hrtimer_wakeup+0x0/0x34) from [<c004e968>] (__run_hrtimer+0xc8/0xe0)
 r4:c7fe9e00
[<c004e8a0>] (__run_hrtimer+0x0/0xe0) from [<c004f6d0>] (hrtimer_interrupt+0x15c/0x1fc)
 r6:2e545ff4 r5:c02b39e8 r4:c7fe9e00
[<c004f574>] (hrtimer_interrupt+0x0/0x1fc) from [<c0028d48>] (mxc_timer_interrupt+0x28/0x38)
[<c0028d20>] (mxc_timer_interrupt+0x0/0x38) from [<c005ecf0>] (handle_IRQ_event+0x44/0x84)
[<c005ecac>] (handle_IRQ_event+0x0/0x84) from [<c0060ba8>] (handle_level_irq+0xa8/0x158)
 r7:c02b2170 r6:0000001a r5:c02aa000 r4:c02b4510
[<c0060b00>] (handle_level_irq+0x0/0x158) from [<c001e048>] (__exception_text_start+0x48/0x60)
 r7:00000000 r6:c02d9c04 r5:c02b4510 r4:0000001a
[<c001e000>] (__exception_text_start+0x0/0x60) from [<c021d2a4>] (__irq_svc+0x44/0x6c)
Exception stack(0xc02abf40 to 0xc02abf88)
bf40: 00000000 0005317f 0005217f 60000013 c001f7e8 c02aa000 c02c2b08 c02add78
bf60: a001a95c 41069264 a001a928 c02abf94 600000d3 c02abf88 c001f82c c001f838
bf80: 60000013 ffffffff
 r7:00000002 r6:001a0000 r5:d4040000 r4:ffffffff
[<c001f7e8>] (default_idle+0x0/0x58) from [<c001f698>] (cpu_idle+0x4c/0x7c)
[<c001f64c>] (cpu_idle+0x0/0x7c) from [<c021a348>] (rest_init+0x64/0x74)
 r6:c001bdbc r5:c02c2ac4 r4:c02d8184
[<c021a2e4>] (rest_init+0x0/0x74) from [<c0008bc8>] (start_kernel+0x200/0x288)
[<c00089c8>] (start_kernel+0x0/0x288) from [<a0008034>] (0xa0008034)
---[ end trace 84135610baf6d9d3 ]---
aaw-11-1-5:/var/volatile/log/ol #

Re: WARN_ON(!rt_rq->rt_nr_running) question?

[Uwe Kleine-König]

Hello,

On Tue, Feb 10, 2009 at 07:17:57PM -0800, raja gobi wrote:
> This is a question about sched_rt.c. I am hoping this is the correct
> mailing list for such questions. If not, could somebody direct me to
> the correct place?
> 
> I am using 2.6.25.16 on an MX27 (ARM)
If you're not running an -rt kernel, probably the linux-arm-kernel ML is
the right place to ask.  Or if you think this problem is not ARM
specific, use LKML.  And probably you should Cc: Ingo Molnar.

Best regards
Uwe

-- 
Pengutronix e.K.                              | Uwe Kleine-König            |
Industrial Linux Solutions                    | http://www.pengutronix.de/  |
--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html