[PATCH 0/2] add support for "unsupported testcases" in test framework

[PATCH 0/2] add support for "unsupported testcases" in test framework

[Jerin Jacob]

This patchset adds support for marking "unsupported" test case
if the test case returns -ENOTSUP.
The change was driven by the need that captured in the following link

http://dpdk.org/ml/archives/dev/2017-March/059950.html

Jerin Jacob (2):
  test: store only the test case name
  test: add a new unit test case status

 test/test/test.c | 24 +++++++++++++++---------
 test/test/test.h | 23 +++++++++--------------
 2 files changed, 24 insertions(+), 23 deletions(-)

-- 
2.5.5

[PATCH 1/2] test: store only the test case name

[Jerin Jacob]

Store only the test case name in unit test case structure.The actor who
renders the test status can add appropriate test status. This enables
adding the new test case status without storing the additional
information in the unit test case structure.

Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
---
 test/test/test.c | 14 ++++++--------
 test/test/test.h | 23 +++++++++--------------
 2 files changed, 15 insertions(+), 22 deletions(-)

diff --git a/test/test/test.c b/test/test/test.c
index cd0e784..a86dc86 100644
--- a/test/test/test.c
+++ b/test/test/test.c
@@ -158,6 +158,7 @@ unit_test_suite_runner(struct unit_test_suite *suite)
 {
 	int test_success;
 	unsigned total = 0, executed = 0, skipped = 0, succeeded = 0, failed = 0;
+	const char *status;
 
 	if (suite->suite_name) {
 		printf(" + ------------------------------------------------------- +\n");
@@ -201,15 +202,12 @@ unit_test_suite_runner(struct unit_test_suite *suite)
 			suite->unit_test_cases[total].teardown();
 
 		if (test_success == TEST_SUCCESS)
-			printf(" + TestCase [%2d] : %s\n", total,
-					suite->unit_test_cases[total].success_msg ?
-					suite->unit_test_cases[total].success_msg :
-					"passed");
+			status = "succeeded";
 		else
-			printf(" + TestCase [%2d] : %s\n", total,
-					suite->unit_test_cases[total].fail_msg ?
-					suite->unit_test_cases[total].fail_msg :
-					"failed");
+			status = "failed";
+
+		printf(" + TestCase [%2d] : %s %s\n", total,
+				suite->unit_test_cases[total].name, status);
 
 		total++;
 	}
diff --git a/test/test/test.h b/test/test/test.h
index 82831f4..9a979d3 100644
--- a/test/test/test.h
+++ b/test/test/test.h
@@ -185,29 +185,24 @@ struct unit_test_case {
 	int (*setup)(void);
 	void (*teardown)(void);
 	int (*testcase)(void);
-	const char *success_msg;
-	const char *fail_msg;
+	const char *name;
 	unsigned enabled;
 };
 
-#define TEST_CASE(fn) { NULL, NULL, fn, #fn " succeeded", #fn " failed", 1 }
+#define TEST_CASE(fn) { NULL, NULL, fn, #fn, 1 }
 
-#define TEST_CASE_NAMED(name, fn) { NULL, NULL, fn, name " succeeded", \
-		name " failed", 1 }
+#define TEST_CASE_NAMED(name, fn) { NULL, NULL, fn, name, 1 }
 
-#define TEST_CASE_ST(setup, teardown, testcase)         \
-		{ setup, teardown, testcase, #testcase " succeeded",    \
-		#testcase " failed ", 1 }
+#define TEST_CASE_ST(setup, teardown, testcase) \
+		{ setup, teardown, testcase, #testcase, 1 }
 
 
-#define TEST_CASE_DISABLED(fn) { NULL, NULL, fn, #fn " succeeded", \
-	#fn " failed", 0 }
+#define TEST_CASE_DISABLED(fn) { NULL, NULL, fn, #fn, 0 }
 
-#define TEST_CASE_ST_DISABLED(setup, teardown, testcase)         \
-		{ setup, teardown, testcase, #testcase " succeeded",    \
-		#testcase " failed ", 0 }
+#define TEST_CASE_ST_DISABLED(setup, teardown, testcase) \
+		{ setup, teardown, testcase, #testcase, 0 }
 
-#define TEST_CASES_END() { NULL, NULL, NULL, NULL, NULL, 0 }
+#define TEST_CASES_END() { NULL, NULL, NULL, NULL, 0 }
 
 #if RTE_LOG_LEVEL >= RTE_LOG_DEBUG
 #define TEST_HEXDUMP(file, title, buf, len) rte_hexdump(file, title, buf, len)
-- 
2.5.5

[PATCH 2/2] test: add a new unit test case status

[Jerin Jacob]

Add a new unit test case status called "unsupported".
This is useful in marking a test case "unsupported" if testcase
returns -ENOTSUP at runtime.

Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
---
 test/test/test.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/test/test/test.c b/test/test/test.c
index a86dc86..c561eb5 100644
--- a/test/test/test.c
+++ b/test/test/test.c
@@ -157,7 +157,8 @@ int
 unit_test_suite_runner(struct unit_test_suite *suite)
 {
 	int test_success;
-	unsigned total = 0, executed = 0, skipped = 0, succeeded = 0, failed = 0;
+	unsigned int total = 0, executed = 0, skipped = 0;
+	unsigned int succeeded = 0, failed = 0, unsupported = 0;
 	const char *status;
 
 	if (suite->suite_name) {
@@ -191,8 +192,12 @@ unit_test_suite_runner(struct unit_test_suite *suite)
 			test_success = suite->unit_test_cases[total].testcase();
 			if (test_success == TEST_SUCCESS)
 				succeeded++;
+			else if (test_success == -ENOTSUP)
+				unsupported++;
 			else
 				failed++;
+		} else if (test_success == -ENOTSUP) {
+			unsupported++;
 		} else {
 			failed++;
 		}
@@ -203,6 +208,8 @@ unit_test_suite_runner(struct unit_test_suite *suite)
 
 		if (test_success == TEST_SUCCESS)
 			status = "succeeded";
+		else if (test_success == -ENOTSUP)
+			status = "unsupported";
 		else
 			status = "failed";
 
@@ -224,6 +231,7 @@ unit_test_suite_runner(struct unit_test_suite *suite)
 	printf(" + Tests Total :       %2d\n", total);
 	printf(" + Tests Skipped :     %2d\n", skipped);
 	printf(" + Tests Executed :    %2d\n", executed);
+	printf(" + Tests Unsupported:  %2d\n", unsupported);
 	printf(" + Tests Passed :      %2d\n", succeeded);
 	printf(" + Tests Failed :      %2d\n", failed);
 	printf(" + ------------------------------------------------------- +\n");
-- 
2.5.5

Re: [PATCH 1/2] test: store only the test case name

[Van Haaren, Harry]

> From: Jerin Jacob [mailto:jerin.jacob@caviumnetworks.com]
> Sent: Sunday, March 19, 2017 1:38 PM
> To: dev@dpdk.org
> Cc: thomas.monjalon@6wind.com; Van Haaren, Harry <harry.van.haaren@intel.com>; Jerin
> Jacob <jerin.jacob@caviumnetworks.com>
> Subject: [dpdk-dev] [PATCH 1/2] test: store only the test case name
> 
> Store only the test case name in unit test case structure.The actor who
> renders the test status can add appropriate test status. This enables
> adding the new test case status without storing the additional
> information in the unit test case structure.
> 
> Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
> ---
>  test/test/test.c | 14 ++++++--------
>  test/test/test.h | 23 +++++++++--------------
>  2 files changed, 15 insertions(+), 22 deletions(-)
> 
> diff --git a/test/test/test.c b/test/test/test.c


It looks like the patch got corrupted somehow - the above line should be  app/test/test.c, not test/test/test.c
Applying fails here, but with a s/test/app in the right places, the patches (this and next) both apply cleanly, and work well.
This handles the event/sw timeout_ticks() unsupported function as should so,

Tested-by: Harry van Haaren <harry.van.haaren@intel.com>


With the above apply issue resolved, also

Acked-by: Harry van Haaren <harry.van.haaren@intel.com>

Re: [PATCH 2/2] test: add a new unit test case status

[Van Haaren, Harry]

> From: Jerin Jacob [mailto:jerin.jacob@caviumnetworks.com]
> Sent: Sunday, March 19, 2017 1:38 PM
> To: dev@dpdk.org
> Cc: thomas.monjalon@6wind.com; Van Haaren, Harry <harry.van.haaren@intel.com>; Jerin
> Jacob <jerin.jacob@caviumnetworks.com>
> Subject: [dpdk-dev] [PATCH 2/2] test: add a new unit test case status
> 
> Add a new unit test case status called "unsupported".
> This is useful in marking a test case "unsupported" if testcase
> returns -ENOTSUP at runtime.
> 
> Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>


Same   test/test/test.c   instead of   app/test/test.c  issue as patch 1/2, but apart from that

Acked-by: Harry van Haaren <harry.van.haaren@intel.com>


> ---
>  test/test/test.c | 10 +++++++++-
>  1 file changed, 9 insertions(+), 1 deletion(-)
> 
> diff --git a/test/test/test.c b/test/test/test.c
> index a86dc86..c561eb5 100644
> --- a/test/test/test.c
> +++ b/test/test/test.c
> @@ -157,7 +157,8 @@ int
>  unit_test_suite_runner(struct unit_test_suite *suite)
>  {
>  	int test_success;
> -	unsigned total = 0, executed = 0, skipped = 0, succeeded = 0, failed = 0;
> +	unsigned int total = 0, executed = 0, skipped = 0;
> +	unsigned int succeeded = 0, failed = 0, unsupported = 0;
>  	const char *status;
> 
>  	if (suite->suite_name) {
> @@ -191,8 +192,12 @@ unit_test_suite_runner(struct unit_test_suite *suite)
>  			test_success = suite->unit_test_cases[total].testcase();
>  			if (test_success == TEST_SUCCESS)
>  				succeeded++;
> +			else if (test_success == -ENOTSUP)
> +				unsupported++;
>  			else
>  				failed++;
> +		} else if (test_success == -ENOTSUP) {
> +			unsupported++;
>  		} else {
>  			failed++;
>  		}
> @@ -203,6 +208,8 @@ unit_test_suite_runner(struct unit_test_suite *suite)
> 
>  		if (test_success == TEST_SUCCESS)
>  			status = "succeeded";
> +		else if (test_success == -ENOTSUP)
> +			status = "unsupported";
>  		else
>  			status = "failed";
> 
> @@ -224,6 +231,7 @@ unit_test_suite_runner(struct unit_test_suite *suite)
>  	printf(" + Tests Total :       %2d\n", total);
>  	printf(" + Tests Skipped :     %2d\n", skipped);
>  	printf(" + Tests Executed :    %2d\n", executed);
> +	printf(" + Tests Unsupported:  %2d\n", unsupported);
>  	printf(" + Tests Passed :      %2d\n", succeeded);
>  	printf(" + Tests Failed :      %2d\n", failed);
>  	printf(" + ------------------------------------------------------- +\n");
> --
> 2.5.5

Re: [PATCH 1/2] test: store only the test case name

[Jerin Jacob]

On Mon, Mar 20, 2017 at 09:58:04AM +0000, Van Haaren, Harry wrote:
> > From: Jerin Jacob [mailto:jerin.jacob@caviumnetworks.com]
> > Sent: Sunday, March 19, 2017 1:38 PM
> > To: dev@dpdk.org
> > Cc: thomas.monjalon@6wind.com; Van Haaren, Harry <harry.van.haaren@intel.com>; Jerin
> > Jacob <jerin.jacob@caviumnetworks.com>
> > Subject: [dpdk-dev] [PATCH 1/2] test: store only the test case name
> > 
> > Store only the test case name in unit test case structure.The actor who
> > renders the test status can add appropriate test status. This enables
> > adding the new test case status without storing the additional
> > information in the unit test case structure.
> > 
> > Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
> > ---
> >  test/test/test.c | 14 ++++++--------
> >  test/test/test.h | 23 +++++++++--------------
> >  2 files changed, 15 insertions(+), 22 deletions(-)
> > 
> > diff --git a/test/test/test.c b/test/test/test.c
> 
> 
> It looks like the patch got corrupted somehow - the above line should be  app/test/test.c, not test/test/test.c
> Applying fails here, but with a s/test/app in the right places, the patches (this and next) both apply cleanly, and work well.
> This handles the event/sw timeout_ticks() unsupported function as should so,
> 
> Tested-by: Harry van Haaren <harry.van.haaren@intel.com>
> 
> 
> With the above apply issue resolved, also

Recently there was a rework in test directory. Now test/test points to old app/test
in dpdk-master.So no issues.

http://dpdk.org/browse/dpdk/tree/test/test

> 
> Acked-by: Harry van Haaren <harry.van.haaren@intel.com>

Re: [PATCH 1/2] test: store only the test case name

[Van Haaren, Harry]

> From: Jerin Jacob [mailto:jerin.jacob@caviumnetworks.com]


> > It looks like the patch got corrupted somehow - the above line should be
> app/test/test.c, not test/test/test.c
> > Applying fails here, but with a s/test/app in the right places, the patches (this and
> next) both apply cleanly, and work well.
> > This handles the event/sw timeout_ticks() unsupported function as should so,
> >
> > Tested-by: Harry van Haaren <harry.van.haaren@intel.com>
> >
> >
> > With the above apply issue resolved, also
> 
> Recently there was a rework in test directory. Now test/test points to old app/test
> in dpdk-master.So no issues.
> 
> http://dpdk.org/browse/dpdk/tree/test/test


Ah, I'm behind on master - apologies, should have done a git pull.

In that case, just

Acked-by: Harry van Haaren <harry.van.haaren@intel.com>


Thanks for taking the lead on this patchset!

Re: [PATCH 0/2] add support for "unsupported testcases" in test framework

[Thomas Monjalon]

2017-03-19 19:07, Jerin Jacob:
> This patchset adds support for marking "unsupported" test case
> if the test case returns -ENOTSUP.
> The change was driven by the need that captured in the following link
> 
> http://dpdk.org/ml/archives/dev/2017-March/059950.html
> 
> Jerin Jacob (2):
>   test: store only the test case name
>   test: add a new unit test case status

Applied, thanks