[LTP] [PATCH 0/3] lib/tst_net.sh: calc median instead of mean in tst_netload()

[LTP] [PATCH 0/3] lib/tst_net.sh: calc median instead of mean in tst_netload()

[Alexey Kodanev]

This series of patches is intended to improve the reliability
of results in network tests that using tst_netload().

After TST_NETLOAD_RUN_COUNT was added, we get several results (5 by
default). The first patch allows all of them to be displayed along
with the final result. It helps to debug test performance failures.

Further patches replace the mean calculation of such data set with
the median one. It allows to exclude extreme points from the final
result.

Alexey Kodanev (3):
  lib/tst_net.sh: print all netstress results in tst_netload()
  lib: add tst_get_median helper binary for use in script tests
  lib/tst_net.sh: calc median instead of mean in tst_netload()

 testcases/lib/.gitignore       |  1 +
 testcases/lib/Makefile         |  3 ++-
 testcases/lib/tst_get_median.c | 37 ++++++++++++++++++++++++++++++++++
 testcases/lib/tst_net.sh       | 10 ++++-----
 4 files changed, 45 insertions(+), 6 deletions(-)
 create mode 100644 testcases/lib/tst_get_median.c

-- 
2.25.1

[LTP] [PATCH 1/3] lib/tst_net.sh: print all netstress results in tst_netload()

[Alexey Kodanev]

This helps during debugging when you need to see from which
dataset the final result was obtained (mean).

Signed-off-by: Alexey Kodanev <alexey.kodanev@oracle.com>
---
 testcases/lib/tst_net.sh | 14 +++++++++-----
 1 file changed, 9 insertions(+), 5 deletions(-)

diff --git a/testcases/lib/tst_net.sh b/testcases/lib/tst_net.sh
index ef9354903..f1a498306 100644
--- a/testcases/lib/tst_net.sh
+++ b/testcases/lib/tst_net.sh
@@ -710,7 +710,7 @@ tst_netload()
 	tst_rhost_run -c "pkill -9 netstress\$"
 	rm -f tst_netload.log
 
-	local res=0
+	local results
 	local passed=0
 
 	for i in $(seq 1 $run_cnt); do
@@ -751,7 +751,7 @@ tst_netload()
 		[ ! -f $rfile ] && \
 			tst_netload_brk TFAIL "can't read $rfile"
 
-		res="$((res + $(cat $rfile)))"
+		results="$results $(cat $rfile)"
 		passed=$((passed + 1))
 	done
 
@@ -761,10 +761,14 @@ tst_netload()
 		tst_netload_brk TFAIL "expected '$expect_res' but ret: '$ret'"
 	fi
 
-	res=$((res / $passed))
-	echo "$res" > $rfile
+	local mean res_sum
+	for r in $results; do
+		res_sum="$((res_sum + r))"
+	done
+	mean=$((res_sum / passed))
+	echo "$mean" > $rfile
 
-	tst_res_ TPASS "netstress passed, mean time '$res' ms"
+	tst_res_ TPASS "netstress passed, mean time $mean ms, data:$results"
 
 	return $ret
 }
-- 
2.25.1

[LTP] [PATCH 2/3] lib: add tst_get_median helper binary for use in script tests

[Alexey Kodanev]

/tst_get_median 10 11 300 8 9 14
output: 10

Signed-off-by: Alexey Kodanev <alexey.kodanev@oracle.com>
---
 testcases/lib/.gitignore       |  1 +
 testcases/lib/Makefile         |  3 ++-
 testcases/lib/tst_get_median.c | 37 ++++++++++++++++++++++++++++++++++
 3 files changed, 40 insertions(+), 1 deletion(-)
 create mode 100644 testcases/lib/tst_get_median.c

diff --git a/testcases/lib/.gitignore b/testcases/lib/.gitignore
index 52f99dc45..bc299b6ee 100644
--- a/testcases/lib/.gitignore
+++ b/testcases/lib/.gitignore
@@ -2,6 +2,7 @@
 /tst_checkpoint
 /tst_device
 /tst_getconf
+/tst_get_median
 /tst_get_unused_port
 /tst_kvcmp
 /tst_net_iface_prefix
diff --git a/testcases/lib/Makefile b/testcases/lib/Makefile
index 4616e24c0..f77da0d56 100644
--- a/testcases/lib/Makefile
+++ b/testcases/lib/Makefile
@@ -28,6 +28,7 @@ INSTALL_TARGETS		:= *.sh
 
 MAKE_TARGETS		:= tst_sleep tst_random tst_checkpoint tst_rod tst_kvcmp\
 			   tst_device tst_net_iface_prefix tst_net_ip_prefix tst_net_vars\
-			   tst_getconf tst_supported_fs tst_check_drivers tst_get_unused_port
+			   tst_getconf tst_supported_fs tst_check_drivers tst_get_unused_port\
+			   tst_get_median
 
 include $(top_srcdir)/include/mk/generic_leaf_target.mk
diff --git a/testcases/lib/tst_get_median.c b/testcases/lib/tst_get_median.c
new file mode 100644
index 000000000..5246f12e0
--- /dev/null
+++ b/testcases/lib/tst_get_median.c
@@ -0,0 +1,37 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/* Copyright (c) 2021 Oracle and/or its affiliates. All Rights Reserved. */
+
+#include <stdio.h>
+#include <stdlib.h>
+
+static int cmp(const void *a, const void *b)
+{
+   return (*(int *)a - *(int *)b);
+}
+
+int main(int argc, const char *argv[])
+{
+	const size_t size = argc - 1;
+
+	if (!size) {
+		fprintf(stderr, "Please provide a numeric list\n");
+		return 1;
+	}
+	if (size == 1) {
+		printf("%d", atoi(argv[1]));
+		return 0;
+	}
+
+	int arr[size];
+	size_t i;
+
+	for (i = 0; i < size; ++i)
+		arr[i] = atoi(argv[i + 1]);
+
+	qsort(arr, size, sizeof(arr[0]), cmp);
+
+	const size_t size2 = size / 2;
+	printf("%d", (size & 1) ? arr[size2] : ((arr[size2 - 1] + arr[size2]) / 2));
+
+	return 0;
+}
-- 
2.25.1

[LTP] [PATCH 3/3] lib/tst_net.sh: calc median instead of mean in tst_netload()

[Alexey Kodanev]

Sometimes the tests can get the following results on a test network:

gre01   1 TINFO: run server 'netstress -D ltp_v0...
gre01   1 TINFO: run client 'netstress -l -D ltp_v0... 5 times
gre01   1 TPASS: netstress passed, mean time 4633 ms, data: 128 22627 134 142 137
...
vxlan03 1 TINFO: run server 'netstress -D ltp_v0...
vxlan03 1 TINFO: run client 'netstress -l -D ltp_v0... 5 times
vxlan03 1 TPASS: netstress passed, mean time 4584 ms, data: 142 140 146 145 22350

One unsuccessful run can have a huge impact on the final result,
when using the mean time with such data.

A more suitable solution for short runs would be to obtain a median
time that can remove all outliers. This will lead to more consistent
performance test results. For example, instead of the above runs, we
would get this:

gre01   1 TPASS: netstress passed, median time 137 ms, data: 128 22627 134 142 137
vxlan03 1 TPASS: netstress passed, median time 145 ms, data: 142 140 146 145 22350

Signed-off-by: Alexey Kodanev <alexey.kodanev@oracle.com>
---
 testcases/lib/tst_net.sh | 10 +++-------
 1 file changed, 3 insertions(+), 7 deletions(-)

diff --git a/testcases/lib/tst_net.sh b/testcases/lib/tst_net.sh
index f1a498306..ca21fe326 100644
--- a/testcases/lib/tst_net.sh
+++ b/testcases/lib/tst_net.sh
@@ -761,14 +761,10 @@ tst_netload()
 		tst_netload_brk TFAIL "expected '$expect_res' but ret: '$ret'"
 	fi
 
-	local mean res_sum
-	for r in $results; do
-		res_sum="$((res_sum + r))"
-	done
-	mean=$((res_sum / passed))
-	echo "$mean" > $rfile
+	local median=$(tst_get_median $results)
+	echo "$median" > $rfile
 
-	tst_res_ TPASS "netstress passed, mean time $mean ms, data:$results"
+	tst_res_ TPASS "netstress passed, median time $median ms, data:$results"
 
 	return $ret
 }
-- 
2.25.1

[LTP] [PATCH 1/3] lib/tst_net.sh: print all netstress results in tst_netload()

[Petr Vorel]

Hi Alexey,

Reviewed-by: Petr Vorel <pvorel@suse.cz>

...
> -	res=$((res / $passed))
> -	echo "$res" > $rfile
> +	local mean res_sum
r should be also local.
> +	for r in $results; do
> +		res_sum="$((res_sum + r))"
> +	done
> +	mean=$((res_sum / passed))
> +	echo "$mean" > $rfile

> -	tst_res_ TPASS "netstress passed, mean time '$res' ms"
> +	tst_res_ TPASS "netstress passed, mean time $mean ms, data:$results"

Kind regards,
Petr

[LTP] [PATCH 2/3] lib: add tst_get_median helper binary for use in script tests

[Petr Vorel]

Hi Alexey,

> /tst_get_median 10 11 300 8 9 14
> output: 10

> Signed-off-by: Alexey Kodanev <alexey.kodanev@oracle.com>

Reviewed-by: Petr Vorel <pvorel@suse.cz>
Thanks, very good idea!

Kind regards,
Petr

[LTP] [PATCH 3/3] lib/tst_net.sh: calc median instead of mean in tst_netload()

[Petr Vorel]

Hi Alexey,

> Sometimes the tests can get the following results on a test network:

> gre01   1 TINFO: run server 'netstress -D ltp_v0...
> gre01   1 TINFO: run client 'netstress -l -D ltp_v0... 5 times
> gre01   1 TPASS: netstress passed, mean time 4633 ms, data: 128 22627 134 142 137
> ...
> vxlan03 1 TINFO: run server 'netstress -D ltp_v0...
> vxlan03 1 TINFO: run client 'netstress -l -D ltp_v0... 5 times
> vxlan03 1 TPASS: netstress passed, mean time 4584 ms, data: 142 140 146 145 22350

> One unsuccessful run can have a huge impact on the final result,
> when using the mean time with such data.

> A more suitable solution for short runs would be to obtain a median
> time that can remove all outliers. This will lead to more consistent
> performance test results. For example, instead of the above runs, we
> would get this:

> gre01   1 TPASS: netstress passed, median time 137 ms, data: 128 22627 134 142 137
> vxlan03 1 TPASS: netstress passed, median time 145 ms, data: 142 140 146 145 22350
Reviewed-by: Petr Vorel <pvorel@suse.cz>

Kind regards,
Petr

[LTP] [PATCH 0/3] lib/tst_net.sh: calc median instead of mean in tst_netload()

[Petr Vorel]

Hi Alexey,

> This series of patches is intended to improve the reliability
> of results in network tests that using tst_netload().

> After TST_NETLOAD_RUN_COUNT was added, we get several results (5 by
> default). The first patch allows all of them to be displayed along
> with the final result. It helps to debug test performance failures.

> Further patches replace the mean calculation of such data set with
> the median one. It allows to exclude extreme points from the final
> result.

Thanks for improving stability of these tests!
I've also noticed often failures in net.features and some of ipsec tests,
this should help.

FYI as TODO I'd like some of net.features have also as functional tests
(ignoring the performance).

Kind regards,
Petr

[LTP] [PATCH 0/3] lib/tst_net.sh: calc median instead of mean in tst_netload()

[Alexey Kodanev]

On 05.02.2021 13:21, Petr Vorel wrote:
> Hi Alexey,
> 
>> This series of patches is intended to improve the reliability
>> of results in network tests that using tst_netload().
> 
>> After TST_NETLOAD_RUN_COUNT was added, we get several results (5 by
>> default). The first patch allows all of them to be displayed along
>> with the final result. It helps to debug test performance failures.
> 
>> Further patches replace the mean calculation of such data set with
>> the median one. It allows to exclude extreme points from the final
>> result.
> 
> Thanks for improving stability of these tests!
> I've also noticed often failures in net.features and some of ipsec tests,
> this should help.
>
> FYI as TODO I'd like some of net.features have also as functional tests
> (ignoring the performance).
> 

Merged, thanks for review Petr!