runaway latency detection

runaway latency detection

[T. Smith]

Trying to isolate PREEMPT_RT (full preemption) runaway latency conditions seen when using the cyclictest on the 4.1.8 SMP PREEMPT_RT kernel.  Seen average latency of 3uS, but runaway or outlier max latency random spikes seem indeterminate.  Noted 72uS with one run of the cyclictest for example. Is this a known issue, the cause understood, and is there a patch for this?   This is on ARM8 Cortex-A53 four core. Running the cyclictest on an isolated core 2 or 3 shows same behavior as do other platforms.  No Trusted OS configured.

The goal is to isolate causes of indeterminism when using the PREEMPT_RT kernel configuration with full preemption and to characterize latency and jitter using ftrace, any recommendations on catching random latency spikes using ftrace or profiling when running the cyclictest?

Re: runaway latency detection

[Steven Rostedt]

On August 20, 2016 3:04:43 PM EDT, Tracy Smith 


>Using ftace it is difficult to catch the latency since it can occur at
>any
>point during the cyclictest. It is an uncomfortable level of
>indeterminism
>without knowing the cause.
>
>Is there a way to trace random latency jumps on a single core using
>trace,
>or any other suggestion to isolate it? I know the functions and tasks
>running on the core using ftrace, but can't catch the random latency
>jumps.

Have you tried using the -b option of cyclictest?  It stops the tracing when it detects a latency. It shows what happened up to that point. I can discuss more about what to do later but I'll be traveling to LinuxCon tomorrow.

-- Steve


>
>Thx, Tracy
>On Aug 20, 2016 12:54 PM, "Nikolay Borisov" <n.borisov.lkml@gmail.com>
>wrote:
>
>>
>>
>> On 20.08.2016 20:03, T. Smith wrote:
>> > The goal is to isolate causes of indeterminism when using the
>PREEMPT_RT
>> kernel configuration with full preemption and to characterize latency
>and
>> jitter using ftrace, any recommendations?
>>
>> What hardware is it ? If it's x86 it's entirely possible you are
>seeing
>> SMI. Perhaps you should look into disabling those if you are on x86?
>>

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: runaway latency detection

[Nikolay Borisov]

On 20.08.2016 20:03, T. Smith wrote:
> The goal is to isolate causes of indeterminism when using the PREEMPT_RT kernel configuration with full preemption and to characterize latency and jitter using ftrace, any recommendations?

What hardware is it ? If it's x86 it's entirely possible you are seeing
SMI. Perhaps you should look into disabling those if you are on x86?

runaway latency detection

[T. Smith]

Trying to isolate PREEMPT_RT (full preemption) runaway latency conditions seen when using the cyclictest on the 4.1.8 SMP PREEMPT_RT kernel.  Seen average latency of 3uS, but runaway or outlier max latency seems indeterminate.  Noted 72uS with one run of the cyclictest for example. Is this a known issue, the cause understood, and is there a patch for this?  

The goal is to isolate causes of indeterminism when using the PREEMPT_RT kernel configuration with full preemption and to characterize latency and jitter using ftrace, any recommendations?