Re: 3.14-rt ARM performance regression?

[Gratian Crisan <gratian.crisan@ni.com>]

To add to Josh's post I have posted some of the data captured during the 
investigation at: https://github.com/gratian/tests

More details available in-line below.

linux-rt-users-owner@vger.kernel.org wrote on 01/23/2015 08:03:41 PM:
> Subject: 3.14-rt ARM performance regression?
> 
> Hey folks-
> 
> We've recently undertaken an upgrade of our kernel from 3.2-rt to
> 3.14-rt, and have run into a performance regression on our ARM boards.
> We're still in the process of trying to isolate what we can, but
> hopefully someone's already run into this and has a solution or might
> have some useful debugging ideas.
> 
<snip>
> We suspected something was up with time accounting, as since 3.2, 
> Zynq gained a
> clock driver, and shifted to using the arm_global_timer driver as it's
> clocksource.  We've compared register dumps of the clocks, cache, and 
timers
> between kernels, and the hardware appears to be configured the same.

The register dumps from the 3.2-rt and 3.14-rt kernel runs are available 
at: https://github.com/gratian/tests/tree/master/register-dumps
In order to make sense of it you will need the Xilinx, Zynq-7000 technical 
reference manual available at: 
http://www.xilinx.com/support/documentation/user_guides/ug585-Zynq-7000-TRM.pdf

> It also
> seems that the runtimes of identical code paths appear to run slower in
> 3.14-rt, as observed by the function tracer and the local ftrace clock; 
we're
> looking to better characterize this.
> 
> We did, however, construct a test to validate via an external clock that
> clock_nanosleep() was sleeping for as long as it says it was by toggling 
a
> GPIO, sleeping for a small period of time, and toggling again, and 
validating
> via a scope that the duration matched.

Test and results available at: 
https://github.com/gratian/tests/tree/master/clock-validation
 
> The toolchain is the same for both kernels (gcc 4.7.2).
> 
> We also brought up 3.14-rt on a BeagleBone Black (also ARM) and compared 
it's
> performance to a 3.8-rt build (bringing up 3.2-rt would require a bit 
more
> effort).  We observed a ~30% degradation on this platform as well.
> 
> If anyone has any ideas, please let us know!  Otherwise, we'll follow up 
with
> anything else we discover.
> 

One of the investigation paths we took is profiling hrtimer_interrupt().

In order to provide a load a simple timer stress test was used: 
https://github.com/gratian/tests/blob/master/timer-stress/timer-stress.c
that in essence starts a large number of non-RT threads that are doing 
clock_nanosleep() calls with a random interval of up to 1ms.

Plotting the CPU cycle counts for hrtimer_interrupt() in 3.14-rt vs. 
3.2-rt appears to show a slowdown of ~12us.
See screenshots under: 
https://github.com/gratian/tests/tree/master/hrtimer_interrupt-profiling

Digging deeper the worst offender when the max is reached seems to be one 
of the callbacks invoked from hrtimer_interrupt.
More specifically the code path seems to be 
hrtimer_interrupt()->tick_sched_timer()->tick_sched_handle()->update_process_times().
I am still profiling this code path trying to pinpoint the source of the 
3.14-rt slowdown in update_process_times().

Ideas/suggestions welcomed.

Thanks,
        Gratian