[rt-preempt / powerpc] Looks like tick_sched_timer() is a major latency source

[rt-preempt / powerpc] Looks like tick_sched_timer() is a major latency source

[Nikita V. Youshchenko]

Hi

I was trying to get minimum possible in-kernel latencies on a freescale 
8360 CPU based board (powerpc arch). There is a project where <30 us worst 
case latency is required, and question was - is it possible to use 
linux+preempt-rt (which is preferred for a number of reason) or not.

I used 2.6.29.5-rt21 kernel, which, for my best knowledge, was the 
last "release" of preempt-rt at the moment when I started.

I used 8360's Generic Timer Module to both to generate interrupts and to 
measure lanencies. GTM timers continue to count after firing an interrupt, 
so latency may be measured my reading current timer value and comparing 
that with the value at which interrupt was generated. I believe this 
method is both simple and accurate.

The result is:

- average latency up to IRQF_NODELAY interrupt header is 2-3 us, even on a 
system with high i/o and cpu load. To generate load, I used a flood ping 
and several 'while true; do true; done' cpu-eaters.

- but worst-case latency is BAD. I occasionally got >50 us even on idle 
system.

I tried hard to identify the latency source, and at some moment discovered 
tick_sched_timer() from kernel/time/tick-sched.c. This routine is called 
from timer interrupt with hardware interrupts disabled, and may execute 
for 50 us and more.

Could someone please comment on this?
Is it possible to move (part of) tick_sched_timer() call tree out of 
hardware-interrupts-disabled context without breaking things?

Nikita

Please CC: reply to my e-mail address. Thanks.

Re: [rt-preempt / powerpc] Looks like tick_sched_timer() is a major latency source

[Thomas Gleixner]

On Tue, 23 Jun 2009, Nikita V. Youshchenko wrote:
> I tried hard to identify the latency source, and at some moment discovered 
> tick_sched_timer() from kernel/time/tick-sched.c. This routine is called 
> from timer interrupt with hardware interrupts disabled, and may execute 
> for 50 us and more.
> 
> Could someone please comment on this?

Did you have CONFIG_NOHZ enabled ? If yes, please disable.

> Is it possible to move (part of) tick_sched_timer() call tree out of 
> hardware-interrupts-disabled context without breaking things?

Yes, we can move things out and split the irq disabled regions. One
particular thing which can be split out is the time update, which we
already had in the timer softirq at some point, but we moved it back
as it showed problems with NTP stability. It's one of the things which
can be revisited. The whole call chain needs some investigation, but
that's not in my main focus now.

Thanks,

	tglx

Re: [rt-preempt / powerpc] Looks like tick_sched_timer() is a major latency source

[Nikita V. Youshchenko]

> > I tried hard to identify the latency source, and at some moment
> > discovered tick_sched_timer() from kernel/time/tick-sched.c. This
> > routine is called from timer interrupt with hardware interrupts
> > disabled, and may execute for 50 us and more.
> >
> > Could someone please comment on this?
>
> Did you have CONFIG_NOHZ enabled ? If yes, please disable.

I have disabled CONFIG_NOHZ long ago, since it really influences latency.

> > Is it possible to move (part of) tick_sched_timer() call tree out of
> > hardware-interrupts-disabled context without breaking things?
>
> Yes, we can move things out and split the irq disabled regions. One
> particular thing which can be split out is the time update, which we
> already had in the timer softirq at some point, but we moved it back
> as it showed problems with NTP stability. It's one of the things which
> can be revisited. The whole call chain needs some investigation, but
> that's not in my main focus now.

Is that code with time update split out available anywhere?

Nikita