[PATCH] app/test: reduced duration of red_autotest

[PATCH] app/test: reduced duration of red_autotest

[Tomasz Kantecki]

'red_autotest' changed to run only functional tests without test #4 which was
taking ~53 seconds. 'red_autotest' takes ~12[s] now.
'red_perf' has been added to run performance tests only (~26[s]).
'red_all' has been added to run all functional tests (including #4) and
perfromance tests. This reflects current 'red_autotest' behavior (~92[s]).
---
 app/test/test_red.c | 75 ++++++++++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 65 insertions(+), 10 deletions(-)

diff --git a/app/test/test_red.c b/app/test/test_red.c
index 81c9d67..a0a942c 100644
--- a/app/test/test_red.c
+++ b/app/test/test_red.c
@@ -1747,6 +1747,16 @@ struct tests func_tests[] = {
 	{ &ovfl_test1_config, ovfl_test1 },
 };
 
+struct tests func_tests_quick[] = {
+	{ &func_test1_config, func_test1 },
+	{ &func_test2_config, func_test2 },
+	{ &func_test3_config, func_test3 },
+	/* no test 4 as it takes a lot of time */
+	{ &func_test5_config, func_test5 },
+	{ &func_test6_config, func_test6 },
+	{ &ovfl_test1_config, ovfl_test1 },
+};
+
 struct tests perf_tests[] = {
 	{ &perf1_test1_config, perf1_test },
 	{ &perf1_test2_config, perf1_test },
@@ -1850,27 +1860,60 @@ test_invalid_parameters(void)
 	return 0;
 }
 
+static void
+show_stats(const uint32_t num_tests, const uint32_t num_pass)
+{
+	if (num_pass == num_tests)
+		printf("[total: %u, pass: %u]\n", num_tests, num_pass);
+	else
+		printf("[total: %u, pass: %u, fail: %u]\n", num_tests, num_pass,
+		       num_tests - num_pass);
+}
+
+static int
+tell_the_result(const uint32_t num_tests, const uint32_t num_pass)
+{
+	return (num_pass == num_tests) ? 0 : 1;
+}
+
 static int
 test_red(void)
 {
 	uint32_t num_tests = 0;
 	uint32_t num_pass = 0;
-	int ret = 0;
 
 	if (test_invalid_parameters() < 0)
 		return -1;
+	run_tests(func_tests_quick, RTE_DIM(func_tests_quick),
+		  &num_tests, &num_pass);
+	show_stats(num_tests, num_pass);
+	return tell_the_result(num_tests, num_pass);
+}
+
+static int
+test_red_perf(void)
+{
+	uint32_t num_tests = 0;
+	uint32_t num_pass = 0;
 
-	run_tests(func_tests, RTE_DIM(func_tests), &num_tests, &num_pass);
 	run_tests(perf_tests, RTE_DIM(perf_tests), &num_tests, &num_pass);
+	show_stats(num_tests, num_pass);
+	return tell_the_result(num_tests, num_pass);
+}
 
-	if (num_pass == num_tests) {
-		printf("[total: %u, pass: %u]\n", num_tests, num_pass);
-		ret = 0;
-	} else {
-		printf("[total: %u, pass: %u, fail: %u]\n", num_tests, num_pass, num_tests - num_pass);
-		ret = -1;
-	}
-	return ret;
+static int
+test_red_all(void)
+{
+	uint32_t num_tests = 0;
+	uint32_t num_pass = 0;
+
+	if (test_invalid_parameters() < 0)
+		return -1;
+
+	run_tests(func_tests, RTE_DIM(func_tests), &num_tests, &num_pass);
+	run_tests(perf_tests, RTE_DIM(perf_tests), &num_tests, &num_pass);
+	show_stats(num_tests, num_pass);
+	return tell_the_result(num_tests, num_pass);
 }
 
 static struct test_command red_cmd = {
@@ -1878,3 +1921,15 @@ static struct test_command red_cmd = {
 	.callback = test_red,
 };
 REGISTER_TEST_COMMAND(red_cmd);
+
+static struct test_command red_cmd_perf = {
+	.command = "red_perf",
+	.callback = test_red_perf,
+};
+REGISTER_TEST_COMMAND(red_cmd_perf);
+
+static struct test_command red_cmd_all = {
+	.command = "red_all",
+	.callback = test_red_all,
+};
+REGISTER_TEST_COMMAND(red_cmd_all);
-- 
1.9.3

--------------------------------------------------------------
Intel Research and Development Ireland Limited
Registered in Ireland
Registered Office: Collinstown Industrial Park, Leixlip, County Kildare
Registered Number: 308263


This e-mail and any attachments may contain confidential material for the sole
use of the intended recipient(s). Any review or distribution by others is
strictly prohibited. If you are not the intended recipient, please contact the
sender and delete all copies.

Re: [PATCH] app/test: reduced duration of red_autotest

[Thomas Monjalon]

Hi Tomasz,

Thanks for splitting tests.

2016-06-07 14:49, Tomasz Kantecki:
> 'red_autotest' changed to run only functional tests without test #4 which was
> taking ~53 seconds. 'red_autotest' takes ~12[s] now.

12s is long for a functional test. We are used to have tests of less than 1s.
Do you think this magnitude order can be achieved for red tests?

Re: [PATCH] app/test: reduced duration of red_autotest

[Kantecki, Tomasz]

Hi Thomas,

> 12s is long for a functional test. We are used to have tests of less than 1s.
> Do you think this magnitude order can be achieved for red tests?

This is a real challenge :) Let me take another look into it.

Regards,
Tomasz
--------------------------------------------------------------
Intel Research and Development Ireland Limited
Registered in Ireland
Registered Office: Collinstown Industrial Park, Leixlip, County Kildare
Registered Number: 308263


This e-mail and any attachments may contain confidential material for the sole
use of the intended recipient(s). Any review or distribution by others is
strictly prohibited. If you are not the intended recipient, please contact the
sender and delete all copies.

Re: [PATCH] app/test: reduced duration of red_autotest

[Kantecki, Tomasz]

Hi Thomas,
 
> > 12s is long for a functional test. We are used to have tests of less than 1s.
> > Do you think this magnitude order can be achieved for red tests?
> 
> This is a real challenge :) Let me take another look into it.

I had a 2nd look into this test suite and reducing time for functional tests is doable but labor intensive. Let me explain:
RED/WRED algorithm is time based (packet queue events in time). For tracking time the test uses TSC (x86 time stamp counter). This could be replaced with some fake time stamp in functional tests and result in significant reduction of execution time. It needs couple of days to carefully replace TSC with a fake time stamp solution without breaking test logic.

Alternatively, one or two more test cases can be removed from the fast_test. A few experiments with this option give me test duration at 5[s] level. I guess this too much off from the 1[s] target.

Regards,
Tomasz


--------------------------------------------------------------
Intel Research and Development Ireland Limited
Registered in Ireland
Registered Office: Collinstown Industrial Park, Leixlip, County Kildare
Registered Number: 308263


This e-mail and any attachments may contain confidential material for the sole
use of the intended recipient(s). Any review or distribution by others is
strictly prohibited. If you are not the intended recipient, please contact the
sender and delete all copies.

Re: [PATCH] app/test: reduced duration of red_autotest

[Thomas Monjalon]

2016-06-08 15:56, Kantecki, Tomasz:
> Hi Thomas,
>  
> > > 12s is long for a functional test. We are used to have tests of less than 1s.
> > > Do you think this magnitude order can be achieved for red tests?
> > 
> > This is a real challenge :) Let me take another look into it.
> 
> I had a 2nd look into this test suite and reducing time for functional tests is doable but labor intensive. Let me explain:
> RED/WRED algorithm is time based (packet queue events in time). For tracking time the test uses TSC (x86 time stamp counter). This could be replaced with some fake time stamp in functional tests and result in significant reduction of execution time. It needs couple of days to carefully replace TSC with a fake time stamp solution without breaking test logic.

Sorry, I haven't dived into the code, but I don't understand how we get
some seconds delays in a code for fast processing on nano-second CPU.
Are you sure you cannot have the same result with less iterations and
shorter timeouts?

[PATCH v2] app/test: reduced duration of red_autotest

[Tomasz Kantecki]

These changes don't break the tests on my systems and reduce
execution time to ~2[s]. I tried "faking" CPU clock frequency but
in some cases it leads to intermittent test fails.
Tomasz

'red_autotest' changed to run only functional tests without test #4 which was
taking ~53 seconds. 'red_autotest' takes ~2[s] now.
'red_perf' has been added to run performance tests only).
'red_all' has been added to run all functional tests (including #4) and
perfromance tests. This reflects current 'red_autotest' behavior.

Other changes:
- machine TSC clock frequency detection takes place only once now.
- timeouts and number of iterations in functional tests have been reduced
  in order to shorten test duration.
---
 app/test/test_red.c | 89 ++++++++++++++++++++++++++++++++++++++++++++---------
 1 file changed, 74 insertions(+), 15 deletions(-)

diff --git a/app/test/test_red.c b/app/test/test_red.c
index 81c9d67..2384c55 100644
--- a/app/test/test_red.c
+++ b/app/test/test_red.c
@@ -57,6 +57,7 @@
 #define MSEC_PER_SEC           1000      /**< Milli-seconds per second */
 #define USEC_PER_MSEC          1000      /**< Micro-seconds per milli-second */
 #define USEC_PER_SEC           1000000   /**< Micro-seconds per second */
+#define NSEC_PER_SEC           (USEC_PER_SEC * 1000) /**< Nano-seconds per second */
 
 /**< structures for testing rte_red performance and function */
 struct test_rte_red_config {        /**< Test structure for RTE_RED config */
@@ -280,12 +281,15 @@ static uint64_t get_machclk_freq(void)
 	uint64_t start = 0;
 	uint64_t end = 0;
 	uint64_t diff = 0;
-	uint64_t clk_freq_hz = 0;
+	static uint64_t clk_freq_hz;
 	struct timespec tv_start = {0, 0}, tv_end = {0, 0};
 	struct timespec req = {0, 0};
 
-	req.tv_sec = 1;
-	req.tv_nsec = 0;
+	if (clk_freq_hz != 0)
+		return clk_freq_hz;
+
+	req.tv_sec = 0;
+	req.tv_nsec = NSEC_PER_SEC / 4;
 
 	clock_gettime(CLOCK_REALTIME, &tv_start);
 	start = rte_rdtsc();
@@ -435,8 +439,8 @@ static struct test_queue ft_tqueue = {
 };
 
 static struct test_var ft_tvar = {
-	.wait_usec = 250000,
-	.num_iterations = 20,
+	.wait_usec = 10000,
+	.num_iterations = 5,
 	.num_ops = 10000,
 	.clk_freq = 0,
 	.dropped = ft_dropped,
@@ -1747,6 +1751,16 @@ struct tests func_tests[] = {
 	{ &ovfl_test1_config, ovfl_test1 },
 };
 
+struct tests func_tests_quick[] = {
+	{ &func_test1_config, func_test1 },
+	{ &func_test2_config, func_test2 },
+	{ &func_test3_config, func_test3 },
+	/* no test 4 as it takes a lot of time */
+	{ &func_test5_config, func_test5 },
+	{ &func_test6_config, func_test6 },
+	{ &ovfl_test1_config, ovfl_test1 },
+};
+
 struct tests perf_tests[] = {
 	{ &perf1_test1_config, perf1_test },
 	{ &perf1_test2_config, perf1_test },
@@ -1850,27 +1864,60 @@ test_invalid_parameters(void)
 	return 0;
 }
 
+static void
+show_stats(const uint32_t num_tests, const uint32_t num_pass)
+{
+	if (num_pass == num_tests)
+		printf("[total: %u, pass: %u]\n", num_tests, num_pass);
+	else
+		printf("[total: %u, pass: %u, fail: %u]\n", num_tests, num_pass,
+		       num_tests - num_pass);
+}
+
+static int
+tell_the_result(const uint32_t num_tests, const uint32_t num_pass)
+{
+	return (num_pass == num_tests) ? 0 : 1;
+}
+
 static int
 test_red(void)
 {
 	uint32_t num_tests = 0;
 	uint32_t num_pass = 0;
-	int ret = 0;
 
 	if (test_invalid_parameters() < 0)
 		return -1;
+	run_tests(func_tests_quick, RTE_DIM(func_tests_quick),
+		  &num_tests, &num_pass);
+	show_stats(num_tests, num_pass);
+	return tell_the_result(num_tests, num_pass);
+}
+
+static int
+test_red_perf(void)
+{
+	uint32_t num_tests = 0;
+	uint32_t num_pass = 0;
 
-	run_tests(func_tests, RTE_DIM(func_tests), &num_tests, &num_pass);
 	run_tests(perf_tests, RTE_DIM(perf_tests), &num_tests, &num_pass);
+	show_stats(num_tests, num_pass);
+	return tell_the_result(num_tests, num_pass);
+}
 
-	if (num_pass == num_tests) {
-		printf("[total: %u, pass: %u]\n", num_tests, num_pass);
-		ret = 0;
-	} else {
-		printf("[total: %u, pass: %u, fail: %u]\n", num_tests, num_pass, num_tests - num_pass);
-		ret = -1;
-	}
-	return ret;
+static int
+test_red_all(void)
+{
+	uint32_t num_tests = 0;
+	uint32_t num_pass = 0;
+
+	if (test_invalid_parameters() < 0)
+		return -1;
+
+	run_tests(func_tests, RTE_DIM(func_tests), &num_tests, &num_pass);
+	run_tests(perf_tests, RTE_DIM(perf_tests), &num_tests, &num_pass);
+	show_stats(num_tests, num_pass);
+	return tell_the_result(num_tests, num_pass);
 }
 
 static struct test_command red_cmd = {
@@ -1878,3 +1925,15 @@ static struct test_command red_cmd = {
 	.callback = test_red,
 };
 REGISTER_TEST_COMMAND(red_cmd);
+
+static struct test_command red_cmd_perf = {
+	.command = "red_perf",
+	.callback = test_red_perf,
+};
+REGISTER_TEST_COMMAND(red_cmd_perf);
+
+static struct test_command red_cmd_all = {
+	.command = "red_all",
+	.callback = test_red_all,
+};
+REGISTER_TEST_COMMAND(red_cmd_all);
-- 
1.9.3

--------------------------------------------------------------
Intel Research and Development Ireland Limited
Registered in Ireland
Registered Office: Collinstown Industrial Park, Leixlip, County Kildare
Registered Number: 308263


This e-mail and any attachments may contain confidential material for the sole
use of the intended recipient(s). Any review or distribution by others is
strictly prohibited. If you are not the intended recipient, please contact the
sender and delete all copies.

[PATCH v3] app/test: reduced duration of red_autotest

[Tomasz Kantecki]

'red_autotest' changed to run only functional tests without test #4 which was
taking ~53 seconds. 'red_autotest' takes ~2[s] now.
'red_perf' has been added to run performance tests only).
'red_all' has been added to run all functional tests (including #4) and
perfromance tests. This reflects current 'red_autotest' behavior.

Other changes:
- machine TSC clock frequency detection takes place only once now.
- timeouts and number of iterations in functional tests have been reduced
  in order to shorten test duration.

Signed-off-by: Tomasz Kantecki <tomasz.kantecki@intel.com>
---
 app/test/test_red.c | 89 ++++++++++++++++++++++++++++++++++++++++++++---------
 1 file changed, 74 insertions(+), 15 deletions(-)

diff --git a/app/test/test_red.c b/app/test/test_red.c
index 81c9d67..2384c55 100644
--- a/app/test/test_red.c
+++ b/app/test/test_red.c
@@ -57,6 +57,7 @@
 #define MSEC_PER_SEC           1000      /**< Milli-seconds per second */
 #define USEC_PER_MSEC          1000      /**< Micro-seconds per milli-second */
 #define USEC_PER_SEC           1000000   /**< Micro-seconds per second */
+#define NSEC_PER_SEC           (USEC_PER_SEC * 1000) /**< Nano-seconds per second */
 
 /**< structures for testing rte_red performance and function */
 struct test_rte_red_config {        /**< Test structure for RTE_RED config */
@@ -280,12 +281,15 @@ static uint64_t get_machclk_freq(void)
 	uint64_t start = 0;
 	uint64_t end = 0;
 	uint64_t diff = 0;
-	uint64_t clk_freq_hz = 0;
+	static uint64_t clk_freq_hz;
 	struct timespec tv_start = {0, 0}, tv_end = {0, 0};
 	struct timespec req = {0, 0};
 
-	req.tv_sec = 1;
-	req.tv_nsec = 0;
+	if (clk_freq_hz != 0)
+		return clk_freq_hz;
+
+	req.tv_sec = 0;
+	req.tv_nsec = NSEC_PER_SEC / 4;
 
 	clock_gettime(CLOCK_REALTIME, &tv_start);
 	start = rte_rdtsc();
@@ -435,8 +439,8 @@ static struct test_queue ft_tqueue = {
 };
 
 static struct test_var ft_tvar = {
-	.wait_usec = 250000,
-	.num_iterations = 20,
+	.wait_usec = 10000,
+	.num_iterations = 5,
 	.num_ops = 10000,
 	.clk_freq = 0,
 	.dropped = ft_dropped,
@@ -1747,6 +1751,16 @@ struct tests func_tests[] = {
 	{ &ovfl_test1_config, ovfl_test1 },
 };
 
+struct tests func_tests_quick[] = {
+	{ &func_test1_config, func_test1 },
+	{ &func_test2_config, func_test2 },
+	{ &func_test3_config, func_test3 },
+	/* no test 4 as it takes a lot of time */
+	{ &func_test5_config, func_test5 },
+	{ &func_test6_config, func_test6 },
+	{ &ovfl_test1_config, ovfl_test1 },
+};
+
 struct tests perf_tests[] = {
 	{ &perf1_test1_config, perf1_test },
 	{ &perf1_test2_config, perf1_test },
@@ -1850,27 +1864,60 @@ test_invalid_parameters(void)
 	return 0;
 }
 
+static void
+show_stats(const uint32_t num_tests, const uint32_t num_pass)
+{
+	if (num_pass == num_tests)
+		printf("[total: %u, pass: %u]\n", num_tests, num_pass);
+	else
+		printf("[total: %u, pass: %u, fail: %u]\n", num_tests, num_pass,
+		       num_tests - num_pass);
+}
+
+static int
+tell_the_result(const uint32_t num_tests, const uint32_t num_pass)
+{
+	return (num_pass == num_tests) ? 0 : 1;
+}
+
 static int
 test_red(void)
 {
 	uint32_t num_tests = 0;
 	uint32_t num_pass = 0;
-	int ret = 0;
 
 	if (test_invalid_parameters() < 0)
 		return -1;
+	run_tests(func_tests_quick, RTE_DIM(func_tests_quick),
+		  &num_tests, &num_pass);
+	show_stats(num_tests, num_pass);
+	return tell_the_result(num_tests, num_pass);
+}
+
+static int
+test_red_perf(void)
+{
+	uint32_t num_tests = 0;
+	uint32_t num_pass = 0;
 
-	run_tests(func_tests, RTE_DIM(func_tests), &num_tests, &num_pass);
 	run_tests(perf_tests, RTE_DIM(perf_tests), &num_tests, &num_pass);
+	show_stats(num_tests, num_pass);
+	return tell_the_result(num_tests, num_pass);
+}
 
-	if (num_pass == num_tests) {
-		printf("[total: %u, pass: %u]\n", num_tests, num_pass);
-		ret = 0;
-	} else {
-		printf("[total: %u, pass: %u, fail: %u]\n", num_tests, num_pass, num_tests - num_pass);
-		ret = -1;
-	}
-	return ret;
+static int
+test_red_all(void)
+{
+	uint32_t num_tests = 0;
+	uint32_t num_pass = 0;
+
+	if (test_invalid_parameters() < 0)
+		return -1;
+
+	run_tests(func_tests, RTE_DIM(func_tests), &num_tests, &num_pass);
+	run_tests(perf_tests, RTE_DIM(perf_tests), &num_tests, &num_pass);
+	show_stats(num_tests, num_pass);
+	return tell_the_result(num_tests, num_pass);
 }
 
 static struct test_command red_cmd = {
@@ -1878,3 +1925,15 @@ static struct test_command red_cmd = {
 	.callback = test_red,
 };
 REGISTER_TEST_COMMAND(red_cmd);
+
+static struct test_command red_cmd_perf = {
+	.command = "red_perf",
+	.callback = test_red_perf,
+};
+REGISTER_TEST_COMMAND(red_cmd_perf);
+
+static struct test_command red_cmd_all = {
+	.command = "red_all",
+	.callback = test_red_all,
+};
+REGISTER_TEST_COMMAND(red_cmd_all);
-- 
1.9.3

--------------------------------------------------------------
Intel Research and Development Ireland Limited
Registered in Ireland
Registered Office: Collinstown Industrial Park, Leixlip, County Kildare
Registered Number: 308263


This e-mail and any attachments may contain confidential material for the sole
use of the intended recipient(s). Any review or distribution by others is
strictly prohibited. If you are not the intended recipient, please contact the
sender and delete all copies.

Re: [PATCH v3] app/test: reduced duration of red_autotest

[Thomas Monjalon]

2016-06-13 11:26, Tomasz Kantecki:
> 'red_autotest' changed to run only functional tests without test #4 which was
> taking ~53 seconds. 'red_autotest' takes ~2[s] now.
> 'red_perf' has been added to run performance tests only).
> 'red_all' has been added to run all functional tests (including #4) and
> perfromance tests. This reflects current 'red_autotest' behavior.
> 
> Other changes:
> - machine TSC clock frequency detection takes place only once now.
> - timeouts and number of iterations in functional tests have been reduced
>   in order to shorten test duration.
> 
> Signed-off-by: Tomasz Kantecki <tomasz.kantecki@intel.com>

Applied, thanks