timerfd performances

timerfd performances

[Sebastien Laveze]

Hi,

Since the removal of ktimersoftd which happened in 6f6ba7715a91
("softirq: rework"), I wonder what is the best approach when using
timer APIs relying on softirqs for wake-up.

My application polls on a timerfd, and I used to have good and stable
performance (< 100us) by setting a high priority to the appropriate
ktimersoftd. Now that the timer handling is handled in the unique
ksoftirqd I'm having increased latencies since (per my understanding)
the ksoftirqd is handling other things (in my use case, networking).

When I say increased latencies, with sufficient load (~450K fps UDP
traffic) it can be over a millisecond... And yet, I haven't added other
load handled in ksoftd like tasklets or IOs.

I tried raising the ksoftd prio over threaded interrupts, that helps a
bit but latencies are still high and setting ksoftd to a realtime
priority doesn't seem a viable option anyway.

I guess some tuning could also be done such as tuning the NAPI budget
but this doesn't seem very robust nor generic.

In the end, it seems my most viable option at the moment is to move to
clock_nanosleep() but it implies radical changes in my application
architecture as it's not as scalable and flexible as timerfd.

Any hints on how to "secure" the wake-up latency of timer fds or what
is expected in future works would be greatly appreciated !

Thanks,
Seb

Re: timerfd performances

[Sebastian Andrzej Siewior]

On 2021-01-18 17:10:15 [+0100], Sebastien Laveze wrote:
> Hi,
Hi,

> In the end, it seems my most viable option at the moment is to move to
> clock_nanosleep() but it implies radical changes in my application
> architecture as it's not as scalable and flexible as timerfd.
> 
> Any hints on how to "secure" the wake-up latency of timer fds or what
> is expected in future works would be greatly appreciated !

Splitting the timer-wakeups back into ktimersoftd would look like
helping. But note that all timer wake ups would happen there - even the
accumulated case where we have to wake 100 tasks at once. This would
then probably ask for split on SCHED_OTHER vs RT tasks like it is done
for clock_nanosleep().

> Thanks,
> Seb

Sebastian

Re: timerfd performances

[Sebastien Laveze]

On Fri, 2021-01-22 at 13:30 +0100, Sebastian Andrzej Siewior wrote:
> Splitting the timer-wakeups back into ktimersoftd would look like
> helping. But note that all timer wake ups would happen there - even the
> accumulated case where we have to wake 100 tasks at once. This would
> then probably ask for split on SCHED_OTHER vs RT tasks like it is done
> for clock_nanosleep().
I didn't know there was a difference in the wakeup mechanism when using
clock_nanosleep if the task was RT or not but that makes sense indeed.
(just looking at __hrtimer_init_sleeper)

So basically, if re-introduced, the ktimersoftd (with a RT priority by
default) would only handle the wakeup of RT tasks ? Other non-RT
POSIX/timerfd users would still be woken-up by ksoftirqd ?

> 
> Sebastian
Thx,
Seb

Re: timerfd performances

[Sebastian Andrzej Siewior]

On 2021-01-22 14:23:06 [+0100], Sebastien Laveze wrote:
> On Fri, 2021-01-22 at 13:30 +0100, Sebastian Andrzej Siewior wrote:
> > Splitting the timer-wakeups back into ktimersoftd would look like
> > helping. But note that all timer wake ups would happen there - even the
> > accumulated case where we have to wake 100 tasks at once. This would
> > then probably ask for split on SCHED_OTHER vs RT tasks like it is done
> > for clock_nanosleep().
> I didn't know there was a difference in the wakeup mechanism when using
> clock_nanosleep if the task was RT or not but that makes sense indeed.
> (just looking at __hrtimer_init_sleeper)
> 
> So basically, if re-introduced, the ktimersoftd (with a RT priority by
> default) would only handle the wakeup of RT tasks ? Other non-RT
> POSIX/timerfd users would still be woken-up by ksoftirqd ?

On top of my head, yes. There might be a better way. Using one thread at
prio 1 would mix the wakeup of all threads with various priorities. This
is not a problem for clock_nanosleep() because it can be woken up
directly from the hardirq.

> Thx,
> Seb

Sebastian