Variance, Standard Deviation, Skewness and Kurtosis for cyclictest results?

Variance, Standard Deviation, Skewness and Kurtosis for cyclictest results?

[rolf.freitag]

Hi,

many programs, e. g. ping also print the standard deviation of the measured times, so i would like get the standard deviation(s) also from cyclictest.
It's easy to also calculate Variance, Skewness and Kurtosis in the same pass: https://www.johndcook.com/blog/skewness_kurtosis/
Has someone planned to implement this in cyclictest?

Regards,

Rolf

Re: Variance, Standard Deviation, Skewness and Kurtosis for cyclictest results?

[Sebastian Andrzej Siewior]

On 2017-06-24 00:38:49 [+0200], rolf.freitag@email.de wrote:
> Hi,
Hi,

> many programs, e. g. ping also print the standard deviation of the measured times, so i would like get the standard deviation(s) also from cyclictest.
> It's easy to also calculate Variance, Skewness and Kurtosis in the same pass: https://www.johndcook.com/blog/skewness_kurtosis/
> Has someone planned to implement this in cyclictest?

cyclictest prints min/avg/max. min and avg are nice to look at. The
important part is the max value. If you have avg at 30us and max at
500us then you know, that you can't meet your deadline for a cycle of
400us.

> Regards,
> 
> Rolf

Sebastian

RE: Variance, Standard Deviation, Skewness and Kurtosis for cyclictest results?

[Piotr Gregor]

Hi Sebastian,

I think Rolf understands that but he is simply interested in deviation anyway.
I can agree deviation gives some more insight into nature of latency observed even if it is clear
then max peak is what determines real-timeness of the setup. One may be interested in distribution
of latency - you may have two setups with same average and max peak while there is much less
meaningful peaks on one than on the other.

cheers,
Piotr

-----Original Message-----
From: linux-rt-users-owner@vger.kernel.org [mailto:linux-rt-users-owner@vger.kernel.org] On Behalf Of Sebastian Andrzej Siewior
Sent: 26 June 2017 15:31
To: rolf.freitag@email.de
Cc: r t <linux-rt-users@vger.kernel.org>
Subject: Re: Variance, Standard Deviation, Skewness and Kurtosis for cyclictest results?

On 2017-06-24 00:38:49 [+0200], rolf.freitag@email.de wrote:
> Hi,
Hi,

> many programs, e. g. ping also print the standard deviation of the measured times, so i would like get the standard deviation(s) also from cyclictest.
> It's easy to also calculate Variance, Skewness and Kurtosis in the 
> same pass: https://www.johndcook.com/blog/skewness_kurtosis/
> Has someone planned to implement this in cyclictest?

cyclictest prints min/avg/max. min and avg are nice to look at. The important part is the max value. If you have avg at 30us and max at 500us then you know, that you can't meet your deadline for a cycle of 400us.

> Regards,
> 
> Rolf

Sebastian
--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Variance, Standard Deviation, Skewness and Kurtosis for cyclictest results?

[Sebastian Andrzej Siewior]

On 2017-06-26 15:59:20 [+0000], Piotr Gregor wrote:
> Hi Sebastian,
Hi Piotr,

> I think Rolf understands that but he is simply interested in deviation anyway.
Sure. I just tried to clarify why we have what we have and I can't
remember patches floating around implementing something else. I am not
saying no to any other "average" implementation.

That said, there is the latency-histogram output and people feed those
numbers into gnuplot. So there is this.

> cheers,
> Piotr

Sebastian

RE: Variance, Standard Deviation, Skewness and Kurtosis for cyclictest results?

[rolf.freitag]

Hi,

yes, at voltages the mean is the DC part, the standard deviation is the AC part and usually no one asks for the peak values.
And these values (mean, stddev.) can also be used to characterize soft real-time behavior.

Regards,

Rolf

 

>Gesendet: Montag, 26. Juni 2017 um 17:59 Uhr
>Von: "Piotr Gregor" <pgregor@quadrant-systems.co.uk>
>An: "Sebastian Andrzej Siewior" <bigeasy@linutronix.de>, "rolf.freitag@email.de" <rolf.freitag@email.de>
>Cc: "r t" <linux-rt-users@vger.kernel.org>
>Betreff: RE: Variance, Standard Deviation, Skewness and Kurtosis for cyclictest results?
>Hi Sebastian,
>
>I think Rolf understands that but he is simply interested in deviation anyway.
>I can agree deviation gives some more insight into nature of latency observed even if it is clear
>then max peak is what determines real-timeness of the setup. One may be interested in distribution
>of latency - you may have two setups with same average and max peak while there is much less
>meaningful peaks on one than on the other.
>
>cheers,
>Piotr
>
>-----Original Message-----
>From: linux-rt-users-owner@vger.kernel.org [mailto:linux-rt-users-owner@vger.kernel.org] On Behalf Of Sebastian Andrzej Siewior
>Sent: 26 June 2017 15:31
>To: rolf.freitag@email.de
>Cc: r t <linux-rt-users@vger.kernel.org>
>Subject: Re: Variance, Standard Deviation, Skewness and Kurtosis for cyclictest results?
>
>On 2017-06-24 00:38:49 [+0200], rolf.freitag@email.de wrote:
>> Hi,
>Hi,
>
>> many programs, e. g. ping also print the standard deviation of the measured times, so i would like get the standard deviation(s) also from cyclictest.
>> It's easy to also calculate Variance, Skewness and Kurtosis in the
>> same pass: https://www.johndcook.com/blog/skewness_kurtosis/
>> Has someone planned to implement this in cyclictest?
>
>cyclictest prints min/avg/max. min and avg are nice to look at. The important part is the max value. If you have avg at 30us and max at 500us then you know, that you can't meet your deadline for a cycle of 400us.
>
>> Regards,
>>
>> Rolf
>
>Sebastian
>--
>To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html[http://vger.kernel.org/majordomo->info.html]

Re: Variance, Standard Deviation, Skewness and Kurtosis for cyclictest results?

[Nicholas Mc Guire]

On Mon, Jun 26, 2017 at 03:59:20PM +0000, Piotr Gregor wrote:
> Hi Sebastian,
> 
> I think Rolf understands that but he is simply interested in deviation anyway.
> I can agree deviation gives some more insight into nature of latency observed even if it is clear
> then max peak is what determines real-timeness of the setup. One may be interested in distribution
> of latency - you may have two setups with same average and max peak while there is much less
> meaningful peaks on one than on the other.
>
you have to be careful here - any statistics estimation of the maximum
with e.g. asymtotic extreemvalue distribution is only valid if the data
is in fact iid values. The problem is that it is not assured that you actually
have a distribution (implying a stochastic process as source) but
the max can be systematic problem e.g. SMIs or other HW effects that
are do not follow a distribution at all - for estimation of extreemvalues
the most important property is that the tail characteristic is resonably
constant - any systematic effects could mess that up. So before trying 
to use any statistics you need to verify that you actually have a stochastic
process at the core - and if you want to use simple metrics that apply if
normal-distribution can be assumed you need to verify this assumptions first
rt-systems are rarely (if ever) normally distributed.

That ping is printing standard-deviations is a bit funny as network times
need not be clean distributions at all (and by no means stable over time) and
the tail characteristics of ping are dependent on systematic effects (e.g.
bandwidth trhotling of providers etc.) so I would question the validity of
such exercises - just producing numbers without checking precoditions is a
well known method of missusing statistics. Even a ping in the local network 
is multimodal and calculating a std-dev on it is quite meaningless.
as .

For some rt-systems it is resonably to do statistical estimations of means
and extreem values based on measurement sets like you find in the QA-Farm
but you can not do that with a single data set - bascially you treat the
data sets as samples and then you can derive predictions for the system
maximum based on the distribution of the local maxima of each of the data
sets (say 2h measurements each or so).

Off-topic side-note: If you do have a statiscially deterministic system 
(stable distribution, iid assumption holds, homoscedasticity, etc.) at the 
root - then you can in fact compensate jitter by replicaion over cores. So
depending on what you want to achieve with the system a well behaved 
distribution with a max of 500us can actually provide guarantees of 
400us response-time if you replicate the task and let the "winner" continue 
and the "looser" (task-replica) go to sleep again without performing any 
actions.

thx!
hofrat

RE: Variance, Standard Deviation, Skewness and Kurtosis for cyclictest results?

[Piotr Gregor]

Hi Nicholas,

I think Rolf is not talking about estimating of extreme values but calculating a simple measure, standard deviation.
You can always calculate in-sample deviation given a set of samples and it will give some additional insight into the nature of observed phenomena.
You can also apply tests of robustness and/or calculate all the statistical hypothesis if you want to.
Rolf is likely talking about sane approach of having simple in-sample deviation calculated though.

I agree with Sebastian that histogram does the job if you have histogram,
but assuming you want to have some script comparing results you need
this picture to be quantified, so producing deviation may be the way to go.

cheers,
Piotr 

-----Original Message-----
From: Nicholas Mc Guire [mailto:der.herr@hofr.at] 
Sent: 27 June 2017 09:19
To: Piotr Gregor <pgregor@quadrant-systems.co.uk>
Cc: Sebastian Andrzej Siewior <bigeasy@linutronix.de>; rolf.freitag@email.de; r t <linux-rt-users@vger.kernel.org>
Subject: Re: Variance, Standard Deviation, Skewness and Kurtosis for cyclictest results?

On Mon, Jun 26, 2017 at 03:59:20PM +0000, Piotr Gregor wrote:
> Hi Sebastian,
> 
> I think Rolf understands that but he is simply interested in deviation anyway.
> I can agree deviation gives some more insight into nature of latency 
> observed even if it is clear then max peak is what determines 
> real-timeness of the setup. One may be interested in distribution of 
> latency - you may have two setups with same average and max peak while there is much less meaningful peaks on one than on the other.
>
you have to be careful here - any statistics estimation of the maximum with e.g. asymtotic extreemvalue distribution is only valid if the data is in fact iid values. The problem is that it is not assured that you actually have a distribution (implying a stochastic process as source) but the max can be systematic problem e.g. SMIs or other HW effects that are do not follow a distribution at all - for estimation of extreemvalues the most important property is that the tail characteristic is resonably constant - any systematic effects could mess that up. So before trying to use any statistics you need to verify that you actually have a stochastic process at the core - and if you want to use simple metrics that apply if normal-distribution can be assumed you need to verify this assumptions first rt-systems are rarely (if ever) normally distributed.

That ping is printing standard-deviations is a bit funny as network times need not be clean distributions at all (and by no means stable over time) and the tail characteristics of ping are dependent on systematic effects (e.g.
bandwidth trhotling of providers etc.) so I would question the validity of such exercises - just producing numbers without checking precoditions is a well known method of missusing statistics. Even a ping in the local network is multimodal and calculating a std-dev on it is quite meaningless.
as .

For some rt-systems it is resonably to do statistical estimations of means and extreem values based on measurement sets like you find in the QA-Farm but you can not do that with a single data set - bascially you treat the data sets as samples and then you can derive predictions for the system maximum based on the distribution of the local maxima of each of the data sets (say 2h measurements each or so).

Off-topic side-note: If you do have a statiscially deterministic system (stable distribution, iid assumption holds, homoscedasticity, etc.) at the root - then you can in fact compensate jitter by replicaion over cores. So depending on what you want to achieve with the system a well behaved distribution with a max of 500us can actually provide guarantees of 400us response-time if you replicate the task and let the "winner" continue and the "looser" (task-replica) go to sleep again without performing any actions.

thx!
hofrat

Re: Variance, Standard Deviation, Skewness and Kurtosis for cyclictest results?

[Nicholas Mc Guire]

On Tue, Jun 27, 2017 at 10:33:53AM +0000, Piotr Gregor wrote:
> Hi Nicholas,
> 
> I think Rolf is not talking about estimating of extreme values but calculating a simple measure, standard deviation.
> You can always calculate in-sample deviation given a set of samples and it will give some additional insight into the nature of observed phenomena.
> You can also apply tests of robustness and/or calculate all the statistical hypothesis if you want to.
> Rolf is likely talking about sane approach of having simple in-sample deviation calculated though.

You can always calculate in-sample deviations - but are they meaningufl ?
You are making assumptions on the distribution that may or may not hold
and that is why you can not do "simplly in-sample deviations" they are
meaningless as a metric.

> 
> I agree with Sebastian that histogram does the job if you have histogram,
> but assuming you want to have some script comparing results you need
> this picture to be quantified, so producing deviation may be the way to go.

unfortnuaately you can not do that - it would be nice - but its not possible
you can do this but you are just generating numeric noise - there is no meaning
of a standard deviation if you underlying process is not a single process (
or the summation of well-behaved stochastic processes) but a set of 
independent stochastic processes that is producing cumulative effects.

If you want to do statistic trending (thats what it seems you are trying to)
then you need a model that faithfuly approximates the underlying process that
is "emitting" the data you are looking at - you can not simply make assumptions
of normality.

If you want to compare performance trends for RT then fire up R - build a 
model for an asymptotic extreem value distribution (probably type II Frechet
distributions for maximum would be the most suitable one). Its not impossible
to automate - but its not going to be simple ither - your steps are basically
 1) identify the distribution in the current data set.
 2) verify that your data sets are more or less homogenous.
 3) compare if the new data set fits the trend of previous data sets by
    looking at appropriate extreemalue distributions.

Just generating in-sample numbers that have no meaning is not going to help.

thx!
hofrat

RE: Variance, Standard Deviation, Skewness and Kurtosis for cyclictest results?

[Piotr Gregor]

Hi Nicholas,

-----Original Message-----
From: Nicholas Mc Guire [mailto:der.herr@hofr.at] 
Sent: 27 June 2017 12:39
To: Piotr Gregor <pgregor@quadrant-systems.co.uk>
Cc: Sebastian Andrzej Siewior <bigeasy@linutronix.de>; rolf.freitag@email.de; r t <linux-rt-users@vger.kernel.org>
Subject: Re: Variance, Standard Deviation, Skewness and Kurtosis for cyclictest results?

"Just generating in-sample numbers that have no meaning is not going to help."
... unless this is exactly what you need.

cheers,
Piotr