* PREEMPT_RT with Full Tickless
@ 2015-12-11 11:29 GeHao Kang
2015-12-11 14:36 ` Mike Galbraith
0 siblings, 1 reply; 5+ messages in thread
From: GeHao Kang @ 2015-12-11 11:29 UTC (permalink / raw)
To: tglx, peterz, jason.low2; +Cc: linux-rt-users, Mike Galbraith
Hi Thomas, Peter, and Jason,
When I enable CONFIG_PREEMPT_RT_FULL and CONFIG_NO_HZ_FULL
in Linux Kernel v4.1.12-rt12, I find the tick on the nohz_full cpu never stops.
Whenever the nohz_full cpu wants to stop its tick, it would use
get_next_timer_interrupt() to get the time of next timer and check
whether this time is more than one jiffy after current time.
However, get_next_timer_interrupt() always returns now+1 in PREEMPT_RT
and this cpu has no chance of stopping its tick. Please reference
kernel/time/timer.c: get_next_timer_interrupt().
#ifdef CONFIG_PREEMPT_RT_FULL
/*
* On PREEMPT_RT we cannot sleep here. As a result we can't take
* the base lock to check when the next timer is pending and so
* we assume the next jiffy.
*/
return now + 1;
#endif
Does this mean it is a restriction on the current development of PREEMPT_RT?
Thanks,
- Kang
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: PREEMPT_RT with Full Tickless
2015-12-11 11:29 PREEMPT_RT with Full Tickless GeHao Kang
@ 2015-12-11 14:36 ` Mike Galbraith
2015-12-11 17:50 ` Sebastian Andrzej Siewior
0 siblings, 1 reply; 5+ messages in thread
From: Mike Galbraith @ 2015-12-11 14:36 UTC (permalink / raw)
To: GeHao Kang, tglx, peterz, jason.low2; +Cc: linux-rt-users
On Fri, 2015-12-11 at 19:29 +0800, GeHao Kang wrote:
> Hi Thomas, Peter, and Jason,
>
> When I enable CONFIG_PREEMPT_RT_FULL and CONFIG_NO_HZ_FULL
> in Linux Kernel v4.1.12-rt12, I find the tick on the nohz_full cpu
> never stops.
Because the workarounds that let it work have been dropped. I recall
Thomas mentioning that, but can't find the message in my rubble pile.
-Mike
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: PREEMPT_RT with Full Tickless
2015-12-11 14:36 ` Mike Galbraith
@ 2015-12-11 17:50 ` Sebastian Andrzej Siewior
2015-12-12 20:11 ` Gary Robertson
0 siblings, 1 reply; 5+ messages in thread
From: Sebastian Andrzej Siewior @ 2015-12-11 17:50 UTC (permalink / raw)
To: Mike Galbraith; +Cc: GeHao Kang, tglx, peterz, jason.low2, linux-rt-users
* Mike Galbraith | 2015-12-11 15:36:17 [+0100]:
>On Fri, 2015-12-11 at 19:29 +0800, GeHao Kang wrote:
>> When I enable CONFIG_PREEMPT_RT_FULL and CONFIG_NO_HZ_FULL
>> in Linux Kernel v4.1.12-rt12, I find the tick on the nohz_full cpu
>> never stops.
>
>Because the workarounds that let it work have been dropped. I recall
>Thomas mentioning that, but can't find the message in my rubble pile.
Right. This was the case on the first v4.0 or v4.1 release. The longterm
plan is to bring it back once the timers have been re-done so we don't
need any hacks.
> -Mike
Sebastian
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: PREEMPT_RT with Full Tickless
2015-12-11 17:50 ` Sebastian Andrzej Siewior
@ 2015-12-12 20:11 ` Gary Robertson
2015-12-13 4:19 ` Mike Galbraith
0 siblings, 1 reply; 5+ messages in thread
From: Gary Robertson @ 2015-12-12 20:11 UTC (permalink / raw)
To: Sebastian Andrzej Siewior
Cc: Mike Galbraith, GeHao Kang, Thomas Gleixner, Peter Zijlstra,
jason.low2, RT
On Fri, Dec 11, 2015 at 11:50 AM, Sebastian Andrzej Siewior
<bigeasy@linutronix.de> wrote:
> * Mike Galbraith | 2015-12-11 15:36:17 [+0100]:
>
>>On Fri, 2015-12-11 at 19:29 +0800, GeHao Kang wrote:
>>> When I enable CONFIG_PREEMPT_RT_FULL and CONFIG_NO_HZ_FULL
>>> in Linux Kernel v4.1.12-rt12, I find the tick on the nohz_full cpu
>>> never stops.
>>
Also - since in RT the tick timer interrupt results in a schedulable
thread running to handle the software interrupt work stimulated by the
tick - upon exit from the interrupt handler there are always at least
2 runnable threads in the CPU running the tick if there was a runnable
task interrupted by the tick event. Therefore the scheduler is still
needed on that CPU, the CPU is ineligible to enter full tickless mode,
and another tick timer event is scheduled. Below is an ftrace snippet
from an isolation test of an Linaro Networking Group kernel with the
RT patch applied. It is a v4.1.10 kernel running on a dual-core ARM
Cortex A15 armv7a platform and compiled with
CONFIG_NO_HZ_FULL_ALL=y... so CPU 1 is designated for full tickless
operation when possible. The cpuhog process is affined to the
isolated CPU and runs a tight loop with no system calls so it is
always runnable. The snippet below shows two tick cycles but this
condition persists indefinitely.
# tracer: function
#
# entries-in-buffer/entries-written: 34219/132331 #P:2
#
# _-----=> irqs-off
# / _----=> need-resched
# |/ _-----=> need-resched_lazy
# || / _---=> hardirq/softirq
# ||| / _--=> preempt-depth
# |||| /_--=> preempt-lazy-depth
# ||||| _-=> migrate-disable
# ||||| / delay
# TASK-PID CPU# |||||| TIMESTAMP FUNCTION
# | | | |||||| | |
cpuhog-1800 [001] dn..1.. 1103.211867: tick_stop:
success=no msg=more than 1 task in runqueue
cpuhog-1800 [001] dn..3.. 1103.211870: sched_stat_runtime:
comm=cpuhog pid=1800 runtime=12083 [ns] vruntime=90040150190 [ns]
cpuhog-1800 [001] d...3.. 1103.211872: sched_switch:
prev_comm=cpuhog prev_pid=1800 prev_prio=120 prev_state=R ==>
next_comm=ksoftirqd/1 next_pid=18 next_prio=98
ksoftirqd/1-18 [001] ....111 1103.211880: softirq_entry:
vec=1 [action=TIMER]
ksoftirqd/1-18 [001] ....111 1103.211883: softirq_exit: vec=1
[action=TIMER]
ksoftirqd/1-18 [001] d...3.. 1103.211889: sched_stat_wait:
comm=cpuhog pid=1800 delay=37834 [ns]
ksoftirqd/1-18 [001] d...3.. 1103.211891: sched_switch:
prev_comm=ksoftirqd/1 prev_pid=18 prev_prio=98 prev_state=S ==>
next_comm=cpuhog next_pid=1800 next_prio=120
cpuhog-1800 [001] d..h1.. 1103.216830: irq_handler_entry:
irq=49 name=arch_timer
cpuhog-1800 [001] d..h2.. 1103.216832: hrtimer_cancel:
hrtimer=ee7a5c98
cpuhog-1800 [001] d..h1.. 1103.216834:
hrtimer_expire_entry: hrtimer=ee7a5c98 function=tick_sched_timer
now=1103195005858
cpuhog-1800 [001] d..h2.. 1103.216836: sched_stat_runtime:
comm=cpuhog pid=1800 runtime=4949166 [ns] vruntime=90045099356 [ns]
cpuhog-1800 [001] d..h1.. 1103.216839: softirq_raise:
vec=1 [action=TIMER]
cpuhog-1800 [001] d..h1.. 1103.216841:
hrtimer_expire_exit: hrtimer=ee7a5c98
cpuhog-1800 [001] d..h2.. 1103.216843: hrtimer_start:
hrtimer=ee7a5c98 function=tick_sched_timer expires=1103200000000
softexpires=1103200000000
cpuhog-1800 [001] d..h1.. 1103.216845: irq_handler_exit:
irq=49 ret=handled
cpuhog-1800 [001] d...3.. 1103.216849:
tick_nohz_full_kick_cpu <-enqueue_top_rt_rq
cpuhog-1800 [001] dn..3.. 1103.216853: sched_wakeup:
comm=ksoftirqd/1 pid=18 prio=98 success=1 target_cpu=001
cpuhog-1800 [001] dn..1.. 1103.216855: tick_nohz_irq_exit <-irq_exit
cpuhog-1800 [001] dn..1.. 1103.216856: tick_stop:
success=no msg=more than 1 task in runqueue
cpuhog-1800 [001] dn..1.. 1103.216862: tick_nohz_irq_exit <-irq_exit
cpuhog-1800 [001] dn..1.. 1103.216863: tick_stop:
success=no msg=more than 1 task in runqueue
cpuhog-1800 [001] dn..3.. 1103.216866: sched_stat_runtime:
comm=cpuhog pid=1800 runtime=12209 [ns] vruntime=90045111565 [ns]
cpuhog-1800 [001] d...3.. 1103.216868: sched_switch:
prev_comm=cpuhog prev_pid=1800 prev_prio=120 prev_state=R ==>
next_comm=ksoftirqd/1 next_pid=18 next_prio=98
ksoftirqd/1-18 [001] ....111 1103.216876: softirq_entry:
vec=1 [action=TIMER]
ksoftirqd/1-18 [001] ....111 1103.216879: softirq_exit: vec=1
[action=TIMER]
ksoftirqd/1-18 [001] d...3.. 1103.216885: sched_stat_wait:
comm=cpuhog pid=1800 delay=34791 [ns]
ksoftirqd/1-18 [001] d...3.. 1103.216887: sched_switch:
prev_comm=ksoftirqd/1 prev_pid=18 prev_prio=98 prev_state=S ==>
next_comm=cpuhog next_pid=1800 next_prio=120
cpuhog-1800 [001] d..h1.. 1103.221830: irq_handler_entry:
irq=49 name=arch_timer
cpuhog-1800 [001] d..h2.. 1103.221832: hrtimer_cancel:
hrtimer=ee7a5c98
cpuhog-1800 [001] d..h1.. 1103.221834:
hrtimer_expire_entry: hrtimer=ee7a5c98 function=tick_sched_timer
now=1103200005650
cpuhog-1800 [001] d..h2.. 1103.221836: sched_stat_runtime:
comm=cpuhog pid=1800 runtime=4952959 [ns] vruntime=90050064524 [ns]
cpuhog-1800 [001] d..h1.. 1103.221839: softirq_raise:
vec=1 [action=TIMER]
cpuhog-1800 [001] d..h1.. 1103.221841:
hrtimer_expire_exit: hrtimer=ee7a5c98
cpuhog-1800 [001] d..h2.. 1103.221843: hrtimer_start:
hrtimer=ee7a5c98 function=tick_sched_timer expires=1103205000000
softexpires=1103205000000
cpuhog-1800 [001] d..h1.. 1103.221845: irq_handler_exit:
irq=49 ret=handled
cpuhog-1800 [001] d...3.. 1103.221849:
tick_nohz_full_kick_cpu <-enqueue_top_rt_rq
cpuhog-1800 [001] dn..3.. 1103.221852: sched_wakeup:
comm=ksoftirqd/1 pid=18 prio=98 success=1 target_cpu=001
cpuhog-1800 [001] dn..1.. 1103.221855: tick_nohz_irq_exit <-irq_exit
cpuhog-1800 [001] dn..1.. 1103.221856: tick_stop:
success=no msg=more than 1 task in runqueue
cpuhog-1800 [001] dn..1.. 1103.221862: tick_nohz_irq_exit <-irq_exit
cpuhog-1800 [001] dn..1.. 1103.221863: tick_stop:
success=no msg=more than 1 task in runqueue
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: PREEMPT_RT with Full Tickless
2015-12-12 20:11 ` Gary Robertson
@ 2015-12-13 4:19 ` Mike Galbraith
0 siblings, 0 replies; 5+ messages in thread
From: Mike Galbraith @ 2015-12-13 4:19 UTC (permalink / raw)
To: Gary Robertson, Sebastian Andrzej Siewior
Cc: GeHao Kang, Thomas Gleixner, Peter Zijlstra, jason.low2, RT
On Sat, 2015-12-12 at 14:11 -0600, Gary Robertson wrote:
> On Fri, Dec 11, 2015 at 11:50 AM, Sebastian Andrzej Siewior
> <bigeasy@linutronix.de> wrote:
> > * Mike Galbraith | 2015-12-11 15:36:17 [+0100]:
> >
> > > On Fri, 2015-12-11 at 19:29 +0800, GeHao Kang wrote:
> > > > When I enable CONFIG_PREEMPT_RT_FULL and CONFIG_NO_HZ_FULL
> > > > in Linux Kernel v4.1.12-rt12, I find the tick on the nohz_full
> > > > cpu
> > > > never stops.
> > >
>
>
> Also - since in RT the tick timer interrupt results in a schedulable
> thread running to handle the software interrupt work stimulated by
> the
> tick - upon exit from the interrupt handler there are always at least
> 2 runnable threads in the CPU running the tick if there was a
> runnable
> task interrupted by the tick event.
Yup, handling that was one of the hacks that was dropped. If you look
at what all was dropped in one 'let's con NOHZ_FULL into working'
patch, you'd think twice about submitting it for review.
-Mike
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2015-12-13 4:19 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-12-11 11:29 PREEMPT_RT with Full Tickless GeHao Kang
2015-12-11 14:36 ` Mike Galbraith
2015-12-11 17:50 ` Sebastian Andrzej Siewior
2015-12-12 20:11 ` Gary Robertson
2015-12-13 4:19 ` Mike Galbraith
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.