high latency introduced by hrtimer_interrupt

high latency introduced by hrtimer_interrupt

[yosi yarchi]

hi

I'm using linux-rt, 5.4.41, at91 sam9x35 (single core).

AFAIK, the hrtimer_interrupt must run at HW context, and could not be 
threaded.
On my system, hrtimer_interrupt processing takes ~100us in average, 
~40us in best case and ~200us in worst case. when adding the overhead of 
the re-scheduling that follow each interrupt, we get to worst case 
overhead of ~250us for the hrtimer_interrupt.
This worst case and the huge variance are too high for some of my 
threaded interrupts, which have to record timestamps of HW events with 
precision < 100us.

Digging in documentation and mailing lists, I didn't find any solution...

Any idea on how to overcome this problem?

With best regards
Yosi Yarchi

Re: high latency introduced by hrtimer_interrupt

[Sebastian Andrzej Siewior]

On 2022-05-12 08:13:58 [+0300], yosi yarchi wrote:
> hi
Hi,

> I'm using linux-rt, 5.4.41, at91 sam9x35 (single core).
> 
> AFAIK, the hrtimer_interrupt must run at HW context, and could not be
> threaded.
> On my system, hrtimer_interrupt processing takes ~100us in average, ~40us in
> best case and ~200us in worst case. when adding the overhead of the
> re-scheduling that follow each interrupt, we get to worst case overhead of
> ~250us for the hrtimer_interrupt.
> This worst case and the huge variance are too high for some of my threaded
> interrupts, which have to record timestamps of HW events with precision <
> 100us.
> 
> Digging in documentation and mailing lists, I didn't find any solution...
> 
> Any idea on how to overcome this problem?

So the hrtimer runs in hardirq context for the important timer like the
system tick or a wake up of RT tasks. It can run at lower priority
(threaded) if it is a timer for non-RT tasks. Nevertheless even the
"threaded" hrtimer needs a wake up from the hard irq context. 
You could enable additional events to figure out what exactly is the
hrtimer doing (processing a specific callback, waking a task, …).

Either there is a timer doing something nasty, leading to these long
runs of the callback _or_ your CPU is simply slow and this is the best
you can get.

> With best regards
> Yosi Yarchi

Sebastian

Questions related to nano_sleep()/hrtimer

[manty kuma]

I went through the source code for hrtimers and understood at a decent
level about how they are working. However, I have the following
questions.

- What is the clock source for HRTimers and what is the frequency of
this clock device?
- with timer subsystem CONFIG_HZ can go to a max of 1000 meaning at
the maximum only 1 ms latency can be reliably established. I
understand that hrtimers are not using CONFIG_HZ but i am just curious
as to what their clock source is and how 1 ns precision is achieved?
- I am using a RT kernel and I see that interrupt handlers are
executed as threaded-irqs. Is it possible to configure the priority of
the interrupt handler hrtimer_interrupt()? for a FF process, the
wakeup() is called from interrupt context(called by threaded irq)
which is actually having lesser priority than my higher priority
process. If possible I would like to change the priority of the
threaded_irq  process that handles timer interrupts. I believe this
way sleep() will not take longer than expected.(I am debugging issues
where even my FF process with prio 110 sometimes fails to wake up back
in time)

Thank you in advance!

Regards,
Manty

Re: Questions related to nano_sleep()/hrtimer

[Thomas Gleixner]

Manty,

On Tue, Jun 29 2021 at 13:54, manty kuma wrote:
> - What is the clock source for HRTimers and what is the frequency of
> this clock device?

Depends on your hardware.

> - with timer subsystem CONFIG_HZ can go to a max of 1000 meaning at
> the maximum only 1 ms latency can be reliably established. I
> understand that hrtimers are not using CONFIG_HZ but i am just curious
> as to what their clock source is and how 1 ns precision is achieved?

1ns is the theoretical precision and depending on the frequency of the
clock source (for reading the time) and the frequency of the clock event
device (for arming the next event) the resulting precision might be >
1ns, but all calculations are at the 1ns level.

> - I am using a RT kernel and I see that interrupt handlers are
> executed as threaded-irqs. Is it possible to configure the priority of
> the interrupt handler hrtimer_interrupt()? for a FF process, the

The timer interrupt which ends up in hrtimer_interrupt() is never
threaded.

> wakeup() is called from interrupt context(called by threaded irq)

I seriously doubt that. If at all then the wakeup() happens from
ksoftirqd().

> which is actually having lesser priority than my higher priority
> process. If possible I would like to change the priority of the
> threaded_irq  process that handles timer interrupts. I believe this
> way sleep() will not take longer than expected.(I am debugging issues
> where even my FF process with prio 110 sometimes fails to wake up back
> in time)

Prio 110? FIFO maximum priority for user space is 99.

If you talk about the kernel internal priority view, then 110 is _not_
FIFO.

Thanks,

        tglx

high latency introduced by hrtimer_interrupt

[yosi yarchi]

hi


I'm using linux-rt, 5.4.41, at91 sam9x35 (single core).


AFAIK, the hrtimer_interrupt must run at HW context, and could not be 
threaded.
On my system, hrtimer_interrupt processing takes ~100us in average, 
~40us in best case and ~200us in worst case. when adding the overhead of 
the re-scheduling that follow each interrupt, we get to worst case 
overhead of ~250us for the hrtimer_interrupt.
This worst case and the huge variance are too high for some of my 
threaded interrupts, which have to record timestamps of HW events with 
precision < 100us.

Digging in documentation and mailing lists, I didn't find any solution...


Any idea on how to overcome this problem?


With best regards
Yosi Yarchi