[igt-dev] [PATCH i-g-t 00/11] runner: Don't report subtests that have dynamic subtests

[igt-dev] [PATCH i-g-t 00/11] runner: Don't report subtests that have dynamic subtests

[Petri Latvala]

A feature requested by Martin before full-on conversion to the use of
dynamic subtests can begin from cibuglog's perspective.

If we have dynamic subtests to report, don't also report the subtest
that contains them.

For example, if we have test "some-test" with subtest "some-subtest"
with dynamic subtests "foo" and "bar", we only report
igt@some-test@some-subtest@foo and igt@some-test@some-subtest@bar,
avoiding igt@some-test@some-subtest altogether.

Parsing dynamic subtests has been changed so that the first dynamic
subtest includes all output from the beginning of the subtest so we
get the setup messages *somewhere*. Same for the last dynamic subtest,
we include all output to the end of the subtest output.

Note that this only changes what gets into results.json; Running the
test yourself still gives you all the output.


Petri Latvala (11):
  runner/resultgen: Extract igt-version field handling to a helper
  runner/resultgen: Extract finding begin/end lines for a subtest to a
    helper
  runner/resultgen: Extract finding begin/end pointers for test output
    to helpers
  runner/resultgen: Hoist handling of dynamic subtest output to a helper
  runner/resultgen: Extrude dynamic subtest result texts
  runner/resultgen: Add support for extra validation hook in
    find_matches()
  runner/resultgen: Make subtest result line finding more robust
  runner/json_tests: Adapt to dynamic subtest result parsing
  runner/resultgen: Don't report subtest result if it has dynamic
    subtests
  runner/json_tests: Adapt to no longer reporting subtests with dynamic
    subtests
  runner/json_tests: Add test for parsing dynamic subtests with same
    name

 .../0/dmesg.txt                               |  10 +
 .../0/err.txt                                 |  10 +
 .../0/journal.txt                             |   3 +
 .../0/out.txt                                 |  11 +
 .../README.txt                                |   2 +
 .../endtime.txt                               |   1 +
 .../joblist.txt                               |   1 +
 .../metadata.txt                              |  12 +
 .../reference.json                            |  84 +++
 .../starttime.txt                             |   1 +
 .../uname.txt                                 |   1 +
 .../dynamic-subtests/reference.json           |  70 +--
 runner/resultgen.c                            | 528 +++++++++++-------
 runner/runner_json_tests.c                    |   1 +
 14 files changed, 475 insertions(+), 260 deletions(-)
 create mode 100644 runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/0/dmesg.txt
 create mode 100644 runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/0/err.txt
 create mode 100644 runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/0/journal.txt
 create mode 100644 runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/0/out.txt
 create mode 100644 runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/README.txt
 create mode 100644 runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/endtime.txt
 create mode 100644 runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/joblist.txt
 create mode 100644 runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/metadata.txt
 create mode 100644 runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/reference.json
 create mode 100644 runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/starttime.txt
 create mode 100644 runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/uname.txt

-- 
2.19.1

_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev

[igt-dev] [PATCH i-g-t 01/11] runner/resultgen: Extract igt-version field handling to a helper

[Petri Latvala]

Signed-off-by: Petri Latvala <petri.latvala@intel.com>
---
 runner/resultgen.c | 27 ++++++++++++++-------------
 1 file changed, 14 insertions(+), 13 deletions(-)

diff --git a/runner/resultgen.c b/runner/resultgen.c
index aa438000..31bc0bb1 100644
--- a/runner/resultgen.c
+++ b/runner/resultgen.c
@@ -365,6 +365,17 @@ static void free_matches(struct matches *matches)
 	free(matches->items);
 }
 
+static void add_igt_version(struct json_object *testobj,
+			    const char *igt_version,
+			    size_t igt_version_len)
+{
+	if (igt_version)
+		json_object_object_add(testobj, "igt-version",
+				       json_object_new_string_len(igt_version,
+								  igt_version_len));
+
+}
+
 static bool fill_from_output(int fd, const char *binary, const char *key,
 			     struct subtest_list *subtests,
 			     struct json_object *tests)
@@ -420,10 +431,7 @@ static bool fill_from_output(int fd, const char *binary, const char *key,
 
 		json_object_object_add(current_test, key,
 				       json_object_new_string_len(buf, statbuf.st_size));
-		if (igt_version)
-			json_object_object_add(current_test, "igt-version",
-					       json_object_new_string_len(igt_version,
-									  igt_version_len));
+		add_igt_version(current_test, igt_version, igt_version_len);
 
 		return true;
 	}
@@ -522,11 +530,7 @@ static bool fill_from_output(int fd, const char *binary, const char *key,
 		json_object_object_add(current_test, key,
 				       json_object_new_string_len(beg, end - beg));
 
-		if (igt_version) {
-			json_object_object_add(current_test, "igt-version",
-					       json_object_new_string_len(igt_version,
-									  igt_version_len));
-		}
+		add_igt_version(current_test, igt_version, igt_version_len);
 
 		if (!json_object_object_get_ex(current_test, "result", NULL)) {
 			parse_subtest_result(subtests->subs[i].name,
@@ -608,10 +612,7 @@ static bool fill_from_output(int fd, const char *binary, const char *key,
 
 				json_object_object_add(current_dynamic_test, key,
 						       json_object_new_string_len(dynbeg, dynend - dynbeg));
-				if (igt_version)
-					json_object_object_add(current_dynamic_test, "igt-version",
-							       json_object_new_string_len(igt_version,
-											  igt_version_len));
+				add_igt_version(current_dynamic_test, igt_version, igt_version_len);
 
 				if (!json_object_object_get_ex(current_dynamic_test, "result", NULL)) {
 					const char *dynresulttext;
-- 
2.19.1

_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev

[igt-dev] [PATCH i-g-t 02/11] runner/resultgen: Extract finding begin/end lines for a subtest to a helper

[Petri Latvala]

Signed-off-by: Petri Latvala <petri.latvala@intel.com>
---
 runner/resultgen.c | 99 ++++++++++++++++++++++++----------------------
 1 file changed, 51 insertions(+), 48 deletions(-)

diff --git a/runner/resultgen.c b/runner/resultgen.c
index 31bc0bb1..be058327 100644
--- a/runner/resultgen.c
+++ b/runner/resultgen.c
@@ -376,6 +376,54 @@ static void add_igt_version(struct json_object *testobj,
 
 }
 
+static int find_subtest_idx_limited(struct matches matches,
+				    const char *bufend,
+				    const char *linekey,
+				    const char *pattern,
+				    const char *subtest_name,
+				    int first,
+				    int last)
+{
+	char *full_line;
+	int line_len;
+	int k;
+
+	/*
+	 * The pattern is a string literal in all call-sites, and we
+	 * don't want to disable this warning globally
+	 */
+#pragma GCC diagnostic push
+#pragma GCC diagnostic ignored "-Wformat-nonliteral"
+	line_len = asprintf(&full_line, pattern, linekey, subtest_name);
+#pragma GCC diagnostic pop
+
+	if (line_len < 0)
+		return -1;
+
+	for (k = first; k < last; k++)
+		if (matches.items[k].what == linekey &&
+		    !memcmp(matches.items[k].where,
+			    full_line,
+			    min(line_len, bufend - matches.items[k].where)))
+			break;
+
+	free(full_line);
+
+	if (k == last)
+		k = -1;
+
+	return k;
+}
+
+static int find_subtest_idx(struct matches matches,
+			    const char *bufend,
+			    const char *linekey,
+			    const char *pattern,
+			    const char *subtest_name)
+{
+	return find_subtest_idx_limited(matches, bufend, linekey, pattern, subtest_name, 0, matches.size);
+}
+
 static bool fill_from_output(int fd, const char *binary, const char *key,
 			     struct subtest_list *subtests,
 			     struct json_object *tests)
@@ -439,44 +487,16 @@ static bool fill_from_output(int fd, const char *binary, const char *key,
 	matches = find_matches(buf, bufend, needles);
 
 	for (i = 0; i < subtests->size; i++) {
-		char *this_sub_begin, *this_sub_result;
 		int begin_idx = -1, result_idx = -1;
 		const char *resulttext;
 		const char *beg, *end;
 		double time;
-		int begin_len;
-		int result_len;
 
 		generate_piglit_name(binary, subtests->subs[i].name, piglit_name, sizeof(piglit_name));
 		current_test = get_or_create_json_object(tests, piglit_name);
 
-		begin_len = asprintf(&this_sub_begin, "%s%s\n", STARTING_SUBTEST, subtests->subs[i].name);
-		result_len = asprintf(&this_sub_result, "%s%s: ", SUBTEST_RESULT, subtests->subs[i].name);
-
-		if (begin_len < 0 || result_len < 0) {
-			fprintf(stderr, "Failure generating strings\n");
-			return false;
-		}
-
-		for (k = 0; k < matches.size; k++) {
-			if (matches.items[k].what == STARTING_SUBTEST &&
-			    !memcmp(matches.items[k].where,
-				    this_sub_begin,
-				    min(begin_len, bufend - matches.items[k].where))) {
-				beg = matches.items[k].where;
-				begin_idx = k;
-			}
-			if (matches.items[k].what == SUBTEST_RESULT &&
-			    !memcmp(matches.items[k].where,
-				    this_sub_result,
-				    min(result_len, bufend - matches.items[k].where))) {
-				end = matches.items[k].where;
-				result_idx = k;
-			}
-		}
-
-		free(this_sub_begin);
-		free(this_sub_result);
+		begin_idx = find_subtest_idx(matches, bufend, STARTING_SUBTEST, "%s%s\n", subtests->subs[i].name);
+		result_idx = find_subtest_idx(matches, bufend, SUBTEST_RESULT, "%s%s: ", subtests->subs[i].name);
 
 		if (begin_idx < 0 && result_idx < 0) {
 			/* No output at all */
@@ -562,31 +582,14 @@ static bool fill_from_output(int fd, const char *binary, const char *key,
 				int dyn_result_idx = -1;
 				char dynamic_name[256];
 				char dynamic_piglit_name[256];
-				char *this_dyn_result;
-				int dyn_result_len;
 				const char *dynbeg, *dynend;
-				int n;
 
 				if (sscanf(matches.items[k].where + strlen(STARTING_DYNAMIC_SUBTEST), "%s", dynamic_name) != 1) {
 					/* Cannot parse name, just ignore this one */
 					continue;
 				}
 
-				dyn_result_len = asprintf(&this_dyn_result, "%s%s: ", DYNAMIC_SUBTEST_RESULT, dynamic_name);
-				if (dyn_result_len < 0)
-					continue;
-
-				for (n = k + 1; n < result_idx; n++) {
-					if (matches.items[n].what == DYNAMIC_SUBTEST_RESULT &&
-					    !memcmp(matches.items[n].where,
-						    this_dyn_result,
-						    min(dyn_result_len, bufend - matches.items[n].where))) {
-						dyn_result_idx = n;
-						break;
-					}
-				}
-
-				free(this_dyn_result);
+				dyn_result_idx = find_subtest_idx_limited(matches, end, DYNAMIC_SUBTEST_RESULT, "%s%s: ", dynamic_name, k, result_idx);
 
 				if (k == 0)
 					dynbeg = beg;
-- 
2.19.1

_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev

[igt-dev] [PATCH i-g-t 03/11] runner/resultgen: Extract finding begin/end pointers for test output to helpers

[Petri Latvala]

Signed-off-by: Petri Latvala <petri.latvala@intel.com>
---
 runner/resultgen.c | 211 +++++++++++++++++++++++----------------------
 1 file changed, 109 insertions(+), 102 deletions(-)

diff --git a/runner/resultgen.c b/runner/resultgen.c
index be058327..23bb3331 100644
--- a/runner/resultgen.c
+++ b/runner/resultgen.c
@@ -424,6 +424,71 @@ static int find_subtest_idx(struct matches matches,
 	return find_subtest_idx_limited(matches, bufend, linekey, pattern, subtest_name, 0, matches.size);
 }
 
+static const char *find_subtest_begin_limit(struct matches matches,
+					    int begin_idx,
+					    int result_idx,
+					    const char *buf,
+					    const char *bufend)
+{
+	/* No matching output at all, include everything */
+	if (begin_idx < 0 && result_idx < 0)
+		return buf;
+
+	if (begin_idx < 0) {
+		/*
+		 * Subtest didn't start, but we have the
+		 * result. Probably because an igt_fixture
+		 * made it fail/skip.
+		 *
+		 * We go backwards one match from the result match,
+		 * and start from the next line.
+		 */
+		if (result_idx > 0)
+			return next_line(matches.items[result_idx - 1].where, bufend);
+		else
+			return buf;
+	}
+
+	/* Include all non-special output before the beginning line. */
+	if (begin_idx == 0)
+		return buf;
+
+	return next_line(matches.items[begin_idx - 1].where, bufend);
+}
+
+static const char *find_subtest_end_limit(struct matches matches,
+					  int begin_idx,
+					  int result_idx,
+					  const char *buf,
+					  const char *bufend)
+{
+	int k;
+
+	/* No matching output at all, include everything */
+	if (begin_idx < 0 && result_idx < 0)
+		return bufend;
+
+	if (result_idx < 0) {
+		/*
+		 * Incomplete result. Include all output up to the
+		 * next starting subtest, or the result of one.
+		 */
+		for (k = begin_idx + 1; k < matches.size; k++) {
+			if (matches.items[k].what == STARTING_SUBTEST ||
+			    matches.items[k].what == SUBTEST_RESULT)
+				return matches.items[k].where;
+		}
+
+		return bufend;
+	}
+
+	/* Include all non-special output to the next match, whatever it is. */
+	if (result_idx < matches.size - 1)
+		return matches.items[result_idx + 1].where;
+
+	return bufend;
+}
+
 static bool fill_from_output(int fd, const char *binary, const char *key,
 			     struct subtest_list *subtests,
 			     struct json_object *tests)
@@ -498,54 +563,8 @@ static bool fill_from_output(int fd, const char *binary, const char *key,
 		begin_idx = find_subtest_idx(matches, bufend, STARTING_SUBTEST, "%s%s\n", subtests->subs[i].name);
 		result_idx = find_subtest_idx(matches, bufend, SUBTEST_RESULT, "%s%s: ", subtests->subs[i].name);
 
-		if (begin_idx < 0 && result_idx < 0) {
-			/* No output at all */
-			beg = buf;
-			end = bufend;
-		} else if (begin_idx < 0) {
-			/*
-			 * Subtest didn't start, but we have the
-			 * result. Probably because an igt_fixture
-			 * made it fail/skip.
-			 *
-			 * Start the output right after the previous
-			 * subtest result/start.
-			 */
-			if (result_idx > 0)
-				beg = next_line(matches.items[result_idx - 1].where, bufend);
-			else
-				beg = buf;
-		} else {
-			/* Stretch output beginning backwards */
-			if (begin_idx == 0)
-				beg = buf;
-			else
-				beg = next_line(matches.items[begin_idx - 1].where, bufend);
-		}
-
-		if (result_idx < 0 && begin_idx < 0) {
-			/* No output at all: Already handled above */
-		} else if (result_idx < 0) {
-			/*
-			 * Incomplete result. Include output up to the
-			 * next starting subtest or result.
-			 */
-			for (k = begin_idx + 1; k < matches.size; k++) {
-				if (matches.items[k].what == STARTING_SUBTEST ||
-				    matches.items[k].what == SUBTEST_RESULT)
-					break;
-			}
-			if (k < matches.size)
-				end = matches.items[k].where;
-			else
-				end = bufend;
-		} else {
-			/* Stretch output to the next starting subtest or result. */
-			if (result_idx < matches.size - 1)
-				end = matches.items[result_idx + 1].where;
-			else
-				end = bufend;
-		}
+		beg = find_subtest_begin_limit(matches, begin_idx, result_idx, buf, bufend);
+		end = find_subtest_end_limit(matches, begin_idx, result_idx, buf, bufend);
 
 		json_object_object_add(current_test, key,
 				       json_object_new_string_len(beg, end - beg));
@@ -567,68 +586,56 @@ static bool fill_from_output(int fd, const char *binary, const char *key,
 		 * the subtest output.
 		 */
 		if (result_idx < 0) {
-			/* If the subtest itself is incomplete, stop at the next start of a subtest */
+			/* If the subtest itself is incomplete, stop at the next start/end of a subtest */
 			for (result_idx = begin_idx + 1;
 			     result_idx < matches.size;
 			     result_idx++) {
-				if (matches.items[result_idx].what == STARTING_SUBTEST)
+				if (matches.items[result_idx].what == STARTING_SUBTEST ||
+				    matches.items[result_idx].what == SUBTEST_RESULT)
 					break;
 			}
 		}
 
 		for (k = begin_idx + 1; k < result_idx; k++) {
-			if (matches.items[k].what == STARTING_DYNAMIC_SUBTEST) {
-				struct json_object *current_dynamic_test = NULL;
-				int dyn_result_idx = -1;
-				char dynamic_name[256];
-				char dynamic_piglit_name[256];
-				const char *dynbeg, *dynend;
-
-				if (sscanf(matches.items[k].where + strlen(STARTING_DYNAMIC_SUBTEST), "%s", dynamic_name) != 1) {
-					/* Cannot parse name, just ignore this one */
-					continue;
-				}
-
-				dyn_result_idx = find_subtest_idx_limited(matches, end, DYNAMIC_SUBTEST_RESULT, "%s%s: ", dynamic_name, k, result_idx);
-
-				if (k == 0)
-					dynbeg = beg;
-				else
-					dynbeg = next_line(matches.items[k - 1].where, end);
-
-				if (dyn_result_idx < 0) {
-					if (k < matches.size - 1)
-						dynend = matches.items[k + 1].where;
-					else
-						dynend = end;
-				} else {
-					if (dyn_result_idx < matches.size - 1)
-						dynend = matches.items[dyn_result_idx + 1].where;
-					else
-						dynend = end;
-				}
-
-				generate_piglit_name_for_dynamic(piglit_name, dynamic_name, dynamic_piglit_name, sizeof(dynamic_piglit_name));
-
-				add_dynamic_subtest(&subtests->subs[i], strdup(dynamic_name));
-				current_dynamic_test = get_or_create_json_object(tests, dynamic_piglit_name);
-
-				json_object_object_add(current_dynamic_test, key,
-						       json_object_new_string_len(dynbeg, dynend - dynbeg));
-				add_igt_version(current_dynamic_test, igt_version, igt_version_len);
-
-				if (!json_object_object_get_ex(current_dynamic_test, "result", NULL)) {
-					const char *dynresulttext;
-					double dyntime;
-
-					parse_subtest_result(dynamic_name,
-							     DYNAMIC_SUBTEST_RESULT,
-							     &dynresulttext, &dyntime,
-							     dyn_result_idx < 0 ? NULL : matches.items[dyn_result_idx].where,
-							     dynend);
-					set_result(current_dynamic_test, dynresulttext);
-					set_runtime(current_dynamic_test, dyntime);
-				}
+			struct json_object *current_dynamic_test = NULL;
+			int dyn_result_idx = -1;
+			char dynamic_name[256];
+			char dynamic_piglit_name[256];
+			const char *dynbeg, *dynend;
+
+			if (matches.items[k].what != STARTING_DYNAMIC_SUBTEST)
+				continue;
+
+			if (sscanf(matches.items[k].where + strlen(STARTING_DYNAMIC_SUBTEST), "%s", dynamic_name) != 1) {
+				/* Cannot parse name, just ignore this one */
+				continue;
+			}
+
+			dyn_result_idx = find_subtest_idx_limited(matches, end, DYNAMIC_SUBTEST_RESULT, "%s%s: ", dynamic_name, k, result_idx);
+
+			dynbeg = find_subtest_begin_limit(matches, k, dyn_result_idx, beg, end);
+			dynend = find_subtest_end_limit(matches, k, dyn_result_idx, beg, end);
+
+			generate_piglit_name_for_dynamic(piglit_name, dynamic_name, dynamic_piglit_name, sizeof(dynamic_piglit_name));
+
+			add_dynamic_subtest(&subtests->subs[i], strdup(dynamic_name));
+			current_dynamic_test = get_or_create_json_object(tests, dynamic_piglit_name);
+
+			json_object_object_add(current_dynamic_test, key,
+					       json_object_new_string_len(dynbeg, dynend - dynbeg));
+			add_igt_version(current_dynamic_test, igt_version, igt_version_len);
+
+			if (!json_object_object_get_ex(current_dynamic_test, "result", NULL)) {
+				const char *dynresulttext;
+				double dyntime;
+
+				parse_subtest_result(dynamic_name,
+						     DYNAMIC_SUBTEST_RESULT,
+						     &dynresulttext, &dyntime,
+						     dyn_result_idx < 0 ? NULL : matches.items[dyn_result_idx].where,
+						     dynend);
+				set_result(current_dynamic_test, dynresulttext);
+				set_runtime(current_dynamic_test, dyntime);
 			}
 		}
 	}
-- 
2.19.1

_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev

[igt-dev] [PATCH i-g-t 04/11] runner/resultgen: Hoist handling of dynamic subtest output to a helper

[Petri Latvala]

Signed-off-by: Petri Latvala <petri.latvala@intel.com>
---
 runner/resultgen.c | 136 ++++++++++++++++++++++++++-------------------
 1 file changed, 78 insertions(+), 58 deletions(-)

diff --git a/runner/resultgen.c b/runner/resultgen.c
index 23bb3331..8f4bbe98 100644
--- a/runner/resultgen.c
+++ b/runner/resultgen.c
@@ -489,6 +489,75 @@ static const char *find_subtest_end_limit(struct matches matches,
 	return bufend;
 }
 
+static void process_dynamic_subtest_output(const char *piglit_name,
+					   const char *igt_version,
+					   size_t igt_version_len,
+					   struct matches matches,
+					   int begin_idx,
+					   int result_idx,
+					   const char *beg,
+					   const char *end,
+					   const char *key,
+					   struct json_object *tests,
+					   struct subtest *subtest)
+{
+	size_t k;
+
+	if (result_idx < 0) {
+		/* If the subtest itself is incomplete, stop at the next start/end of a subtest */
+		for (result_idx = begin_idx + 1;
+		     result_idx < matches.size;
+		     result_idx++) {
+			if (matches.items[result_idx].what == STARTING_SUBTEST ||
+			    matches.items[result_idx].what == SUBTEST_RESULT)
+				break;
+		}
+	}
+
+	for (k = begin_idx + 1; k < result_idx; k++) {
+		struct json_object *current_dynamic_test = NULL;
+		int dyn_result_idx = -1;
+		char dynamic_name[256];
+		char dynamic_piglit_name[256];
+		const char *dynbeg, *dynend;
+
+		if (matches.items[k].what != STARTING_DYNAMIC_SUBTEST)
+			continue;
+
+		if (sscanf(matches.items[k].where + strlen(STARTING_DYNAMIC_SUBTEST), "%s", dynamic_name) != 1) {
+			/* Cannot parse name, just ignore this one */
+			continue;
+		}
+
+		dyn_result_idx = find_subtest_idx_limited(matches, end, DYNAMIC_SUBTEST_RESULT, "%s%s: ", dynamic_name, k, result_idx);
+
+		dynbeg = find_subtest_begin_limit(matches, k, dyn_result_idx, beg, end);
+		dynend = find_subtest_end_limit(matches, k, dyn_result_idx, beg, end);
+
+		generate_piglit_name_for_dynamic(piglit_name, dynamic_name, dynamic_piglit_name, sizeof(dynamic_piglit_name));
+
+		add_dynamic_subtest(subtest, strdup(dynamic_name));
+		current_dynamic_test = get_or_create_json_object(tests, dynamic_piglit_name);
+
+		json_object_object_add(current_dynamic_test, key,
+				       json_object_new_string_len(dynbeg, dynend - dynbeg));
+		add_igt_version(current_dynamic_test, igt_version, igt_version_len);
+
+		if (!json_object_object_get_ex(current_dynamic_test, "result", NULL)) {
+			const char *dynresulttext;
+			double dyntime;
+
+			parse_subtest_result(dynamic_name,
+					     DYNAMIC_SUBTEST_RESULT,
+					     &dynresulttext, &dyntime,
+					     dyn_result_idx < 0 ? NULL : matches.items[dyn_result_idx].where,
+					     dynend);
+			set_result(current_dynamic_test, dynresulttext);
+			set_runtime(current_dynamic_test, dyntime);
+		}
+	}
+}
+
 static bool fill_from_output(int fd, const char *binary, const char *key,
 			     struct subtest_list *subtests,
 			     struct json_object *tests)
@@ -507,7 +576,7 @@ static bool fill_from_output(int fd, const char *binary, const char *key,
 		NULL
 	};
 	struct matches matches = {};
-	size_t i, k;
+	size_t i;
 
 	if (fstat(fd, &statbuf))
 		return false;
@@ -581,63 +650,14 @@ static bool fill_from_output(int fd, const char *binary, const char *key,
 			set_runtime(current_test, time);
 		}
 
-		/*
-		 * Look for dynamic subtests: If any, they are within
-		 * the subtest output.
-		 */
-		if (result_idx < 0) {
-			/* If the subtest itself is incomplete, stop at the next start/end of a subtest */
-			for (result_idx = begin_idx + 1;
-			     result_idx < matches.size;
-			     result_idx++) {
-				if (matches.items[result_idx].what == STARTING_SUBTEST ||
-				    matches.items[result_idx].what == SUBTEST_RESULT)
-					break;
-			}
-		}
-
-		for (k = begin_idx + 1; k < result_idx; k++) {
-			struct json_object *current_dynamic_test = NULL;
-			int dyn_result_idx = -1;
-			char dynamic_name[256];
-			char dynamic_piglit_name[256];
-			const char *dynbeg, *dynend;
-
-			if (matches.items[k].what != STARTING_DYNAMIC_SUBTEST)
-				continue;
-
-			if (sscanf(matches.items[k].where + strlen(STARTING_DYNAMIC_SUBTEST), "%s", dynamic_name) != 1) {
-				/* Cannot parse name, just ignore this one */
-				continue;
-			}
-
-			dyn_result_idx = find_subtest_idx_limited(matches, end, DYNAMIC_SUBTEST_RESULT, "%s%s: ", dynamic_name, k, result_idx);
-
-			dynbeg = find_subtest_begin_limit(matches, k, dyn_result_idx, beg, end);
-			dynend = find_subtest_end_limit(matches, k, dyn_result_idx, beg, end);
-
-			generate_piglit_name_for_dynamic(piglit_name, dynamic_name, dynamic_piglit_name, sizeof(dynamic_piglit_name));
-
-			add_dynamic_subtest(&subtests->subs[i], strdup(dynamic_name));
-			current_dynamic_test = get_or_create_json_object(tests, dynamic_piglit_name);
-
-			json_object_object_add(current_dynamic_test, key,
-					       json_object_new_string_len(dynbeg, dynend - dynbeg));
-			add_igt_version(current_dynamic_test, igt_version, igt_version_len);
-
-			if (!json_object_object_get_ex(current_dynamic_test, "result", NULL)) {
-				const char *dynresulttext;
-				double dyntime;
-
-				parse_subtest_result(dynamic_name,
-						     DYNAMIC_SUBTEST_RESULT,
-						     &dynresulttext, &dyntime,
-						     dyn_result_idx < 0 ? NULL : matches.items[dyn_result_idx].where,
-						     dynend);
-				set_result(current_dynamic_test, dynresulttext);
-				set_runtime(current_dynamic_test, dyntime);
-			}
-		}
+		process_dynamic_subtest_output(piglit_name,
+					       igt_version, igt_version_len,
+					       matches,
+					       begin_idx, result_idx,
+					       beg, end,
+					       key,
+					       tests,
+					       &subtests->subs[i]);
 	}
 
 	free_matches(&matches);
-- 
2.19.1

_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev

[igt-dev] [PATCH i-g-t 05/11] runner/resultgen: Extrude dynamic subtest result texts

[Petri Latvala]

The first dynamic part now starts its reported output at the beginning
of its subtest's output, and the last dynamic part ends its reported
output at the end of its subtest's output.

Signed-off-by: Petri Latvala <petri.latvala@intel.com>
---
 runner/resultgen.c | 64 ++++++++++++++++++++++++++++++----------------
 1 file changed, 42 insertions(+), 22 deletions(-)

diff --git a/runner/resultgen.c b/runner/resultgen.c
index 8f4bbe98..e139e3fc 100644
--- a/runner/resultgen.c
+++ b/runner/resultgen.c
@@ -424,17 +424,18 @@ static int find_subtest_idx(struct matches matches,
 	return find_subtest_idx_limited(matches, bufend, linekey, pattern, subtest_name, 0, matches.size);
 }
 
-static const char *find_subtest_begin_limit(struct matches matches,
-					    int begin_idx,
-					    int result_idx,
-					    const char *buf,
-					    const char *bufend)
+static const char *find_subtest_begin_limit_limited(struct matches matches,
+						    int begin_idx,
+						    int result_idx,
+						    const char *buf,
+						    const char *bufend,
+						    int first_idx)
 {
 	/* No matching output at all, include everything */
-	if (begin_idx < 0 && result_idx < 0)
+	if (begin_idx < first_idx && result_idx < first_idx)
 		return buf;
 
-	if (begin_idx < 0) {
+	if (begin_idx < first_idx) {
 		/*
 		 * Subtest didn't start, but we have the
 		 * result. Probably because an igt_fixture
@@ -443,37 +444,48 @@ static const char *find_subtest_begin_limit(struct matches matches,
 		 * We go backwards one match from the result match,
 		 * and start from the next line.
 		 */
-		if (result_idx > 0)
+		if (result_idx > first_idx)
 			return next_line(matches.items[result_idx - 1].where, bufend);
 		else
 			return buf;
 	}
 
 	/* Include all non-special output before the beginning line. */
-	if (begin_idx == 0)
+	if (begin_idx <= first_idx)
 		return buf;
 
 	return next_line(matches.items[begin_idx - 1].where, bufend);
 }
 
-static const char *find_subtest_end_limit(struct matches matches,
-					  int begin_idx,
-					  int result_idx,
-					  const char *buf,
-					  const char *bufend)
+static const char *find_subtest_begin_limit(struct matches matches,
+					    int begin_idx,
+					    int result_idx,
+					    const char *buf,
+					    const char *bufend)
+{
+	return find_subtest_begin_limit_limited(matches, begin_idx, result_idx, buf, bufend, 0);
+}
+
+static const char *find_subtest_end_limit_limited(struct matches matches,
+						  int begin_idx,
+						  int result_idx,
+						  const char *buf,
+						  const char *bufend,
+						  int first_idx,
+						  int last_idx)
 {
 	int k;
 
 	/* No matching output at all, include everything */
-	if (begin_idx < 0 && result_idx < 0)
+	if (begin_idx < first_idx && result_idx < first_idx)
 		return bufend;
 
-	if (result_idx < 0) {
+	if (result_idx < first_idx) {
 		/*
 		 * Incomplete result. Include all output up to the
 		 * next starting subtest, or the result of one.
 		 */
-		for (k = begin_idx + 1; k < matches.size; k++) {
+		for (k = begin_idx + 1; k < last_idx; k++) {
 			if (matches.items[k].what == STARTING_SUBTEST ||
 			    matches.items[k].what == SUBTEST_RESULT)
 				return matches.items[k].where;
@@ -483,12 +495,21 @@ static const char *find_subtest_end_limit(struct matches matches,
 	}
 
 	/* Include all non-special output to the next match, whatever it is. */
-	if (result_idx < matches.size - 1)
+	if (result_idx < last_idx - 1)
 		return matches.items[result_idx + 1].where;
 
 	return bufend;
 }
 
+static const char *find_subtest_end_limit(struct matches matches,
+					  int begin_idx,
+					  int result_idx,
+					  const char *buf,
+					  const char *bufend)
+{
+	return find_subtest_end_limit_limited(matches, begin_idx, result_idx, buf, bufend, 0, matches.size);
+}
+
 static void process_dynamic_subtest_output(const char *piglit_name,
 					   const char *igt_version,
 					   size_t igt_version_len,
@@ -531,8 +552,8 @@ static void process_dynamic_subtest_output(const char *piglit_name,
 
 		dyn_result_idx = find_subtest_idx_limited(matches, end, DYNAMIC_SUBTEST_RESULT, "%s%s: ", dynamic_name, k, result_idx);
 
-		dynbeg = find_subtest_begin_limit(matches, k, dyn_result_idx, beg, end);
-		dynend = find_subtest_end_limit(matches, k, dyn_result_idx, beg, end);
+		dynbeg = find_subtest_begin_limit_limited(matches, k, dyn_result_idx, beg, end, begin_idx + 1);
+		dynend = find_subtest_end_limit_limited(matches, k, dyn_result_idx, beg, end, begin_idx + 1, result_idx);
 
 		generate_piglit_name_for_dynamic(piglit_name, dynamic_name, dynamic_piglit_name, sizeof(dynamic_piglit_name));
 
@@ -931,8 +952,7 @@ static bool fill_from_dmesg(int fd,
 			}
 		}
 		append_line(&dmesg, &dmesglen, formatted);
-		if (current_test != NULL)
-			append_line(&dynamic_dmesg, &dynamic_dmesg_len, formatted);
+		append_line(&dynamic_dmesg, &dynamic_dmesg_len, formatted);
 		free(formatted);
 	}
 	free(line);
-- 
2.19.1

_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev

[igt-dev] [PATCH i-g-t 06/11] runner/resultgen: Add support for extra validation hook in find_matches()

[Petri Latvala]

Signed-off-by: Petri Latvala <petri.latvala@intel.com>
---
 runner/resultgen.c | 43 +++++++++++++++++++++++++------------------
 1 file changed, 25 insertions(+), 18 deletions(-)

diff --git a/runner/resultgen.c b/runner/resultgen.c
index e139e3fc..b01e0a48 100644
--- a/runner/resultgen.c
+++ b/runner/resultgen.c
@@ -324,6 +324,12 @@ struct matches
 	size_t size;
 };
 
+struct match_needle
+{
+	const char *str;
+	bool (*validate)(const char *needle, const char *line, const char *bufend);
+};
+
 static void match_add(struct matches *matches, const char *where, const char *what)
 {
 	struct match_item newitem = { where, what };
@@ -334,19 +340,20 @@ static void match_add(struct matches *matches, const char *where, const char *wh
 }
 
 static struct matches find_matches(const char *buf, const char *bufend,
-				   const char **needles)
+				   const struct match_needle *needles)
 {
 	struct matches ret = {};
 
 	while (buf < bufend) {
-		const char **needle;
+		const struct match_needle *needle;
 
-		for (needle = needles; *needle; needle++) {
-			if (bufend - buf < strlen(*needle))
+		for (needle = needles; needle->str; needle++) {
+			if (bufend - buf < strlen(needle->str))
 				continue;
 
-			if (!memcmp(buf, *needle, strlen(*needle))) {
-				match_add(&ret, buf, *needle);
+			if (!memcmp(buf, needle->str, strlen(needle->str)) &&
+			    (!needle->validate || needle->validate(needle->str, buf, bufend))) {
+				match_add(&ret, buf, needle->str);
 				goto end_find;
 			}
 		}
@@ -589,12 +596,12 @@ static bool fill_from_output(int fd, const char *binary, const char *key,
 	char *igt_version = NULL;
 	size_t igt_version_len = 0;
 	struct json_object *current_test = NULL;
-	const char *needles[] = {
-		STARTING_SUBTEST,
-		SUBTEST_RESULT,
-		STARTING_DYNAMIC_SUBTEST,
-		DYNAMIC_SUBTEST_RESULT,
-		NULL
+	struct match_needle needles[] = {
+		{ STARTING_SUBTEST, NULL },
+		{ SUBTEST_RESULT, NULL },
+		{ STARTING_DYNAMIC_SUBTEST, NULL },
+		{ DYNAMIC_SUBTEST_RESULT, NULL },
+		{ NULL, NULL },
 	};
 	struct matches matches = {};
 	size_t i;
@@ -1110,12 +1117,12 @@ static void fill_from_journal(int fd,
 
 static bool stderr_contains_warnings(const char *beg, const char *end)
 {
-	const char *needles[] = {
-		STARTING_SUBTEST,
-		SUBTEST_RESULT,
-		STARTING_DYNAMIC_SUBTEST,
-		DYNAMIC_SUBTEST_RESULT,
-		NULL
+	struct match_needle needles[] = {
+		{ STARTING_SUBTEST, NULL },
+		{ SUBTEST_RESULT, NULL },
+		{ STARTING_DYNAMIC_SUBTEST, NULL },
+		{ DYNAMIC_SUBTEST_RESULT, NULL },
+		{ NULL, NULL },
 	};
 	struct matches matches;
 	size_t i = 0;
-- 
2.19.1

_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev

[igt-dev] [PATCH i-g-t 07/11] runner/resultgen: Make subtest result line finding more robust

[Petri Latvala]

On an assertion failure, the string "Subtest xyz failed" is
printed. Make sure we don't match that for SUBTEST_RESULT, or the
equivalent for dynamic subtests.

Parsing the results already explicitly searched for the proper result
line, the difference is when we delimit output up to "the next line of
interest".

Signed-off-by: Petri Latvala <petri.latvala@intel.com>
---
 runner/resultgen.c | 41 +++++++++++++++++++++++++++++++++++++----
 1 file changed, 37 insertions(+), 4 deletions(-)

diff --git a/runner/resultgen.c b/runner/resultgen.c
index b01e0a48..64b899ca 100644
--- a/runner/resultgen.c
+++ b/runner/resultgen.c
@@ -367,6 +367,39 @@ static struct matches find_matches(const char *buf, const char *bufend,
 	return ret;
 }
 
+static bool valid_char_for_subtest_name(char x)
+{
+	return x == '-' || x == '_' || isalnum(x);
+}
+
+static bool is_subtest_result_line(const char *needle, const char *line, const char *bufend)
+{
+	line += strlen(needle);
+
+	/*
+	 * At this point of the string we're expecting:
+	 * - The subtest name (one or more of a-z, A-Z, 0-9, '-' and '_')
+	 * - The characters ':' and ' '
+	 *
+	 * If we find all those, allow parsing this line as [dynamic]
+	 * subtest result.
+	 */
+
+	if (!valid_char_for_subtest_name(*line))
+		return false;
+
+	while (line < bufend && valid_char_for_subtest_name(*line))
+		line++;
+
+	if (line >= bufend || *line++ != ':')
+		return false;
+
+	if (line >= bufend || *line++ != ' ')
+		return false;
+
+	return true;
+}
+
 static void free_matches(struct matches *matches)
 {
 	free(matches->items);
@@ -598,9 +631,9 @@ static bool fill_from_output(int fd, const char *binary, const char *key,
 	struct json_object *current_test = NULL;
 	struct match_needle needles[] = {
 		{ STARTING_SUBTEST, NULL },
-		{ SUBTEST_RESULT, NULL },
+		{ SUBTEST_RESULT, is_subtest_result_line },
 		{ STARTING_DYNAMIC_SUBTEST, NULL },
-		{ DYNAMIC_SUBTEST_RESULT, NULL },
+		{ DYNAMIC_SUBTEST_RESULT, is_subtest_result_line },
 		{ NULL, NULL },
 	};
 	struct matches matches = {};
@@ -1119,9 +1152,9 @@ static bool stderr_contains_warnings(const char *beg, const char *end)
 {
 	struct match_needle needles[] = {
 		{ STARTING_SUBTEST, NULL },
-		{ SUBTEST_RESULT, NULL },
+		{ SUBTEST_RESULT, is_subtest_result_line },
 		{ STARTING_DYNAMIC_SUBTEST, NULL },
-		{ DYNAMIC_SUBTEST_RESULT, NULL },
+		{ DYNAMIC_SUBTEST_RESULT, is_subtest_result_line },
 		{ NULL, NULL },
 	};
 	struct matches matches;
-- 
2.19.1

_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev

[igt-dev] [PATCH i-g-t 08/11] runner/json_tests: Adapt to dynamic subtest result parsing

[Petri Latvala]

Dynamic subtests now include more output, change the dynamic subtest
parsing test accordingly.

Signed-off-by: Petri Latvala <petri.latvala@intel.com>
---
 .../dynamic-subtests/reference.json           | 20 +++++++++----------
 1 file changed, 10 insertions(+), 10 deletions(-)

diff --git a/runner/json_tests_data/dynamic-subtests/reference.json b/runner/json_tests_data/dynamic-subtests/reference.json
index 274173f7..24a1fe8c 100644
--- a/runner/json_tests_data/dynamic-subtests/reference.json
+++ b/runner/json_tests_data/dynamic-subtests/reference.json
@@ -22,7 +22,7 @@
       "dmesg":"<6> [23426155.175691] Console: switching to colour dummy device 80x25\n<6> [23426155.175708] [IGT] dynamic: executing\n<6> [23426155.184875] [IGT] dynamic: starting subtest debug-log-checking\n<6> [23426155.184895] [IGT] dynamic: starting dynamic subtest this-is-dynamic-1\n<6> [23426155.240164] [IGT] dynamic: starting dynamic subtest this-is-dynamic-2\n<6> [23426155.293846] [IGT] dynamic: exiting, ret=98\n<6> [23426155.294003] Console: switching to colour frame buffer device 240x75\n"
     },
     "igt@dynamic@debug-log-checking@this-is-dynamic-1":{
-      "out":"Starting dynamic subtest: this-is-dynamic-1\nStack trace:\n  #0 ..\/lib\/igt_core.c:1607 __igt_fail_assert()\n  #1 ..\/runner\/testdata\/dynamic.c:11 __real_main3()\n  #2 ..\/runner\/testdata\/dynamic.c:3 main()\n  #3 ..\/csu\/libc-start.c:342 __libc_start_main()\n  #4 [_start+0x2a]\nDynamic subtest this-is-dynamic-1: FAIL (0.055s)\n",
+      "out":"IGT-Version: 1.23-g9e957acd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)\nStarting subtest: debug-log-checking\nStarting dynamic subtest: this-is-dynamic-1\nStack trace:\n  #0 ..\/lib\/igt_core.c:1607 __igt_fail_assert()\n  #1 ..\/runner\/testdata\/dynamic.c:11 __real_main3()\n  #2 ..\/runner\/testdata\/dynamic.c:3 main()\n  #3 ..\/csu\/libc-start.c:342 __libc_start_main()\n  #4 [_start+0x2a]\nDynamic subtest this-is-dynamic-1: FAIL (0.055s)\n",
       "igt-version":"IGT-Version: 1.23-g9e957acd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)",
       "result":"fail",
       "time":{
@@ -30,11 +30,11 @@
         "start":0,
         "end":0.055
       },
-      "err":"Starting dynamic subtest: this-is-dynamic-1\n(dynamic:20904) CRITICAL: Test assertion failure function __real_main3, file ..\/runner\/testdata\/dynamic.c:8:\n(dynamic:20904) CRITICAL: Failed assertion: false\nDynamic subtest this-is-dynamic-1 failed.\n**** DEBUG ****\n(dynamic:20904) DEBUG: This print is from 1\n(dynamic:20904) CRITICAL: Test assertion failure function __real_main3, file ..\/runner\/testdata\/dynamic.c:8:\n(dynamic:20904) CRITICAL: Failed assertion: false\n(dynamic:20904) igt_core-INFO: Stack trace:\n(dynamic:20904) igt_core-INFO:   #0 ..\/lib\/igt_core.c:1607 __igt_fail_assert()\n(dynamic:20904) igt_core-INFO:   #1 ..\/runner\/testdata\/dynamic.c:11 __real_main3()\n(dynamic:20904) igt_core-INFO:   #2 ..\/runner\/testdata\/dynamic.c:3 main()\n(dynamic:20904) igt_core-INFO:   #3 ..\/csu\/libc-start.c:342 __libc_start_main()\n(dynamic:20904) igt_core-INFO:   #4 [_start+0x2a]\n****  END  ****\nDynamic subtest this-is-dynamic-1: FAIL (0.055s)\n",
-      "dmesg":"<6> [23426155.184875] [IGT] dynamic: starting subtest debug-log-checking\n<6> [23426155.184895] [IGT] dynamic: starting dynamic subtest this-is-dynamic-1\n"
+      "err":"Starting subtest: debug-log-checking\nStarting dynamic subtest: this-is-dynamic-1\n(dynamic:20904) CRITICAL: Test assertion failure function __real_main3, file ..\/runner\/testdata\/dynamic.c:8:\n(dynamic:20904) CRITICAL: Failed assertion: false\nDynamic subtest this-is-dynamic-1 failed.\n**** DEBUG ****\n(dynamic:20904) DEBUG: This print is from 1\n(dynamic:20904) CRITICAL: Test assertion failure function __real_main3, file ..\/runner\/testdata\/dynamic.c:8:\n(dynamic:20904) CRITICAL: Failed assertion: false\n(dynamic:20904) igt_core-INFO: Stack trace:\n(dynamic:20904) igt_core-INFO:   #0 ..\/lib\/igt_core.c:1607 __igt_fail_assert()\n(dynamic:20904) igt_core-INFO:   #1 ..\/runner\/testdata\/dynamic.c:11 __real_main3()\n(dynamic:20904) igt_core-INFO:   #2 ..\/runner\/testdata\/dynamic.c:3 main()\n(dynamic:20904) igt_core-INFO:   #3 ..\/csu\/libc-start.c:342 __libc_start_main()\n(dynamic:20904) igt_core-INFO:   #4 [_start+0x2a]\n****  END  ****\nDynamic subtest this-is-dynamic-1: FAIL (0.055s)\n",
+      "dmesg":"<6> [23426155.175691] Console: switching to colour dummy device 80x25\n<6> [23426155.175708] [IGT] dynamic: executing\n<6> [23426155.184875] [IGT] dynamic: starting subtest debug-log-checking\n<6> [23426155.184895] [IGT] dynamic: starting dynamic subtest this-is-dynamic-1\n"
     },
     "igt@dynamic@debug-log-checking@this-is-dynamic-2":{
-      "out":"Starting dynamic subtest: this-is-dynamic-2\nStack trace:\n  #0 ..\/lib\/igt_core.c:1607 __igt_fail_assert()\n  #1 ..\/runner\/testdata\/dynamic.c:5 __real_main3()\n  #2 ..\/runner\/testdata\/dynamic.c:3 main()\n  #3 ..\/csu\/libc-start.c:342 __libc_start_main()\n  #4 [_start+0x2a]\nDynamic subtest this-is-dynamic-2: FAIL (0.054s)\n",
+      "out":"Starting dynamic subtest: this-is-dynamic-2\nStack trace:\n  #0 ..\/lib\/igt_core.c:1607 __igt_fail_assert()\n  #1 ..\/runner\/testdata\/dynamic.c:5 __real_main3()\n  #2 ..\/runner\/testdata\/dynamic.c:3 main()\n  #3 ..\/csu\/libc-start.c:342 __libc_start_main()\n  #4 [_start+0x2a]\nDynamic subtest this-is-dynamic-2: FAIL (0.054s)\nSubtest debug-log-checking: FAIL (0.109s)\n",
       "igt-version":"IGT-Version: 1.23-g9e957acd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)",
       "result":"fail",
       "time":{
@@ -42,7 +42,7 @@
         "start":0,
         "end":0.054
       },
-      "err":"Starting dynamic subtest: this-is-dynamic-2\n(dynamic:20904) CRITICAL: Test assertion failure function __real_main3, file ..\/runner\/testdata\/dynamic.c:13:\n(dynamic:20904) CRITICAL: Failed assertion: false\nDynamic subtest this-is-dynamic-2 failed.\n**** DEBUG ****\n(dynamic:20904) DEBUG: This print is from 2\n(dynamic:20904) CRITICAL: Test assertion failure function __real_main3, file ..\/runner\/testdata\/dynamic.c:13:\n(dynamic:20904) CRITICAL: Failed assertion: false\n(dynamic:20904) igt_core-INFO: Stack trace:\n(dynamic:20904) igt_core-INFO:   #0 ..\/lib\/igt_core.c:1607 __igt_fail_assert()\n(dynamic:20904) igt_core-INFO:   #1 ..\/runner\/testdata\/dynamic.c:5 __real_main3()\n(dynamic:20904) igt_core-INFO:   #2 ..\/runner\/testdata\/dynamic.c:3 main()\n(dynamic:20904) igt_core-INFO:   #3 ..\/csu\/libc-start.c:342 __libc_start_main()\n(dynamic:20904) igt_core-INFO:   #4 [_start+0x2a]\n****  END  ****\nDynamic subtest this-is-dynamic-2: FAIL (0.054s)\n",
+      "err":"Starting dynamic subtest: this-is-dynamic-2\n(dynamic:20904) CRITICAL: Test assertion failure function __real_main3, file ..\/runner\/testdata\/dynamic.c:13:\n(dynamic:20904) CRITICAL: Failed assertion: false\nDynamic subtest this-is-dynamic-2 failed.\n**** DEBUG ****\n(dynamic:20904) DEBUG: This print is from 2\n(dynamic:20904) CRITICAL: Test assertion failure function __real_main3, file ..\/runner\/testdata\/dynamic.c:13:\n(dynamic:20904) CRITICAL: Failed assertion: false\n(dynamic:20904) igt_core-INFO: Stack trace:\n(dynamic:20904) igt_core-INFO:   #0 ..\/lib\/igt_core.c:1607 __igt_fail_assert()\n(dynamic:20904) igt_core-INFO:   #1 ..\/runner\/testdata\/dynamic.c:5 __real_main3()\n(dynamic:20904) igt_core-INFO:   #2 ..\/runner\/testdata\/dynamic.c:3 main()\n(dynamic:20904) igt_core-INFO:   #3 ..\/csu\/libc-start.c:342 __libc_start_main()\n(dynamic:20904) igt_core-INFO:   #4 [_start+0x2a]\n****  END  ****\nDynamic subtest this-is-dynamic-2: FAIL (0.054s)\nSubtest debug-log-checking failed.\nNo log.\nSubtest debug-log-checking: FAIL (0.109s)\n",
       "dmesg":"<6> [23426155.240164] [IGT] dynamic: starting dynamic subtest this-is-dynamic-2\n<6> [23426155.293846] [IGT] dynamic: exiting, ret=98\n<6> [23426155.294003] Console: switching to colour frame buffer device 240x75\n"
     },
     "igt@dynamic@empty-container":{
@@ -70,7 +70,7 @@
       "dmesg":"<6> [23426155.175691] Console: switching to colour dummy device 80x25\n<6> [23426155.175708] [IGT] dynamic: executing\n<6> [23426155.184875] [IGT] dynamic: starting subtest normal\n<6> [23426155.184895] Dmesg output for normal\n<6> [23426155.184895] [IGT] dynamic: starting dynamic subtest normal-dynamic-subtest\n"
     },
     "igt@dynamic@normal@normal-dynamic-subtest":{
-      "out":"Starting dynamic subtest: normal-dynamic-subtest\nDynamic subtest normal-dynamic-subtest: SUCCESS (0.055s)\n",
+      "out":"IGT-Version: 1.23-g9e957acd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)\nStarting subtest: normal\nStarting dynamic subtest: normal-dynamic-subtest\nDynamic subtest normal-dynamic-subtest: SUCCESS (0.055s)\nSubtest normal: SUCCESS (0.100s)\n",
       "igt-version":"IGT-Version: 1.23-g9e957acd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)",
       "result":"pass",
       "time":{
@@ -78,8 +78,8 @@
         "start":0,
         "end":0.055
       },
-      "err":"Starting dynamic subtest: normal-dynamic-subtest\nDynamic subtest normal-dynamic-subtest: SUCCESS (0.055s)\n",
-      "dmesg":"<6> [23426155.184875] [IGT] dynamic: starting subtest normal\n<6> [23426155.184895] Dmesg output for normal\n<6> [23426155.184895] [IGT] dynamic: starting dynamic subtest normal-dynamic-subtest\n"
+      "err":"Starting subtest: normal\nStarting dynamic subtest: normal-dynamic-subtest\nDynamic subtest normal-dynamic-subtest: SUCCESS (0.055s)\nSubtest normal: SUCCESS (0.100s)\n",
+      "dmesg":"<6> [23426155.175691] Console: switching to colour dummy device 80x25\n<6> [23426155.175708] [IGT] dynamic: executing\n<6> [23426155.184875] [IGT] dynamic: starting subtest normal\n<6> [23426155.184895] Dmesg output for normal\n<6> [23426155.184895] [IGT] dynamic: starting dynamic subtest normal-dynamic-subtest\n"
     },
     "igt@dynamic@incomplete":{
       "out":"Starting subtest: incomplete\nStarting dynamic subtest: this-is-incomplete\nThis is some output\n",
@@ -94,7 +94,7 @@
       "dmesg":"<6> [23426155.184875] [IGT] dynamic: starting subtest incomplete\n<6> [23426155.184895] [IGT] dynamic: starting dynamic subtest this-is-incomplete\n<6> [23426155.184895] Dmesg output for incomplete\n"
     },
     "igt@dynamic@incomplete@this-is-incomplete":{
-      "out":"Starting dynamic subtest: this-is-incomplete\nThis is some output\n",
+      "out":"Starting subtest: incomplete\nStarting dynamic subtest: this-is-incomplete\nThis is some output\n",
       "igt-version":"IGT-Version: 1.23-g9e957acd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)",
       "result":"incomplete",
       "time":{
@@ -102,7 +102,7 @@
         "start":0,
         "end":0
       },
-      "err":"Starting dynamic subtest: this-is-incomplete\n",
+      "err":"Starting subtest: incomplete\nStarting dynamic subtest: this-is-incomplete\n",
       "dmesg":"<6> [23426155.184875] [IGT] dynamic: starting subtest incomplete\n<6> [23426155.184895] [IGT] dynamic: starting dynamic subtest this-is-incomplete\n<6> [23426155.184895] Dmesg output for incomplete\n"
     },
     "igt@dynamic@resume":{
-- 
2.19.1

_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev

[igt-dev] [PATCH i-g-t 09/11] runner/resultgen: Don't report subtest result if it has dynamic subtests

[Petri Latvala]

Signed-off-by: Petri Latvala <petri.latvala@intel.com>
---
 runner/resultgen.c | 43 +++++++++++++++++++++++++++++++++----------
 1 file changed, 33 insertions(+), 10 deletions(-)

diff --git a/runner/resultgen.c b/runner/resultgen.c
index 64b899ca..2e15331e 100644
--- a/runner/resultgen.c
+++ b/runner/resultgen.c
@@ -1148,6 +1148,22 @@ static void fill_from_journal(int fd,
 	fclose(f);
 }
 
+static void prune_subtests_with_dynamic_subtests(const char *binary,
+						 struct subtest_list *subtests,
+						 struct json_object *tests)
+{
+	char piglit_name[256];
+	size_t i;
+
+	for (i = 0; i < subtests->size; i++) {
+		if (subtests->subs[i].dynamic_size) {
+			generate_piglit_name(binary, subtests->subs[i].name, piglit_name, sizeof(piglit_name));
+			json_object_object_del(tests, piglit_name);
+		}
+
+	}
+}
+
 static bool stderr_contains_warnings(const char *beg, const char *end)
 {
 	struct match_needle needles[] = {
@@ -1218,8 +1234,10 @@ static void override_results(char *binary,
 
 	for (i = 0; i < subtests->size; i++) {
 		generate_piglit_name(binary, subtests->subs[i].name, piglit_name, sizeof(piglit_name));
-		obj = get_or_create_json_object(tests, piglit_name);
-		override_result_single(obj);
+		if (subtests->subs[i].dynamic_size == 0) {
+			obj = get_or_create_json_object(tests, piglit_name);
+			override_result_single(obj);
+		}
 
 		for (k = 0; k < subtests->subs[i].dynamic_size; k++) {
 			generate_piglit_name_for_dynamic(piglit_name, subtests->subs[i].dynamic_names[k],
@@ -1300,15 +1318,18 @@ static void add_to_totals(const char *binary,
 
 	for (i = 0; i < subtests->size; i++) {
 		generate_piglit_name(binary, subtests->subs[i].name, piglit_name, sizeof(piglit_name));
-		test = get_or_create_json_object(results->tests, piglit_name);
-		if (!json_object_object_get_ex(test, "result", &resultobj)) {
-			fprintf(stderr, "Warning: No results set for %s\n", piglit_name);
-			return;
+
+		if (subtests->subs[i].dynamic_size == 0) {
+			test = get_or_create_json_object(results->tests, piglit_name);
+			if (!json_object_object_get_ex(test, "result", &resultobj)) {
+				fprintf(stderr, "Warning: No results set for %s\n", piglit_name);
+				return;
+			}
+			result = json_object_get_string(resultobj);
+			add_result_to_totals(emptystrtotal, result);
+			add_result_to_totals(roottotal, result);
+			add_result_to_totals(binarytotal, result);
 		}
-		result = json_object_get_string(resultobj);
-		add_result_to_totals(emptystrtotal, result);
-		add_result_to_totals(roottotal, result);
-		add_result_to_totals(binarytotal, result);
 
 		for (k = 0; k < subtests->subs[i].dynamic_size; k++) {
 			generate_piglit_name_for_dynamic(piglit_name, subtests->subs[i].dynamic_names[k],
@@ -1355,6 +1376,8 @@ static bool parse_test_directory(int dirfd,
 		goto parse_output_end;
 	}
 
+	prune_subtests_with_dynamic_subtests(entry->binary, &subtests, results->tests);
+
 	override_results(entry->binary, &subtests, results->tests);
 	add_to_totals(entry->binary, &subtests, results);
 
-- 
2.19.1

_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev

[igt-dev] [PATCH i-g-t 10/11] runner/json_tests: Adapt to no longer reporting subtests with dynamic subtests

[Petri Latvala]

Signed-off-by: Petri Latvala <petri.latvala@intel.com>
---
 .../dynamic-subtests/reference.json           | 54 ++++---------------
 1 file changed, 9 insertions(+), 45 deletions(-)

diff --git a/runner/json_tests_data/dynamic-subtests/reference.json b/runner/json_tests_data/dynamic-subtests/reference.json
index 24a1fe8c..ca8f6cd7 100644
--- a/runner/json_tests_data/dynamic-subtests/reference.json
+++ b/runner/json_tests_data/dynamic-subtests/reference.json
@@ -9,18 +9,6 @@
     "end":1560163492.4104891
   },
   "tests":{
-    "igt@dynamic@debug-log-checking":{
-      "out":"IGT-Version: 1.23-g9e957acd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)\nStarting subtest: debug-log-checking\nStarting dynamic subtest: this-is-dynamic-1\nStack trace:\n  #0 ..\/lib\/igt_core.c:1607 __igt_fail_assert()\n  #1 ..\/runner\/testdata\/dynamic.c:11 __real_main3()\n  #2 ..\/runner\/testdata\/dynamic.c:3 main()\n  #3 ..\/csu\/libc-start.c:342 __libc_start_main()\n  #4 [_start+0x2a]\nDynamic subtest this-is-dynamic-1: FAIL (0.055s)\nStarting dynamic subtest: this-is-dynamic-2\nStack trace:\n  #0 ..\/lib\/igt_core.c:1607 __igt_fail_assert()\n  #1 ..\/runner\/testdata\/dynamic.c:5 __real_main3()\n  #2 ..\/runner\/testdata\/dynamic.c:3 main()\n  #3 ..\/csu\/libc-start.c:342 __libc_start_main()\n  #4 [_start+0x2a]\nDynamic subtest this-is-dynamic-2: FAIL (0.054s)\nSubtest debug-log-checking: FAIL (0.109s)\n",
-      "igt-version":"IGT-Version: 1.23-g9e957acd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)",
-      "result":"fail",
-      "time":{
-        "__type__":"TimeAttribute",
-        "start":0,
-        "end":0.109
-      },
-      "err":"Starting subtest: debug-log-checking\nStarting dynamic subtest: this-is-dynamic-1\n(dynamic:20904) CRITICAL: Test assertion failure function __real_main3, file ..\/runner\/testdata\/dynamic.c:8:\n(dynamic:20904) CRITICAL: Failed assertion: false\nDynamic subtest this-is-dynamic-1 failed.\n**** DEBUG ****\n(dynamic:20904) DEBUG: This print is from 1\n(dynamic:20904) CRITICAL: Test assertion failure function __real_main3, file ..\/runner\/testdata\/dynamic.c:8:\n(dynamic:20904) CRITICAL: Failed assertion: false\n(dynamic:20904) igt_core-INFO: Stack trace:\n(dynamic:20904) igt_core-INFO:   #0 ..\/lib\/igt_core.c:1607 __igt_fail_assert()\n(dynamic:20904) igt_core-INFO:   #1 ..\/runner\/testdata\/dynamic.c:11 __real_main3()\n(dynamic:20904) igt_core-INFO:   #2 ..\/runner\/testdata\/dynamic.c:3 main()\n(dynamic:20904) igt_core-INFO:   #3 ..\/csu\/libc-start.c:342 __libc_start_main()\n(dynamic:20904) igt_core-INFO:   #4 [_start+0x2a]\n****  END  ****\nDynamic subtest this-is-dynamic-1: FAIL (0.055s)\nStarting dynamic subtest: this-is-dynamic-2\n(dynamic:20904) CRITICAL: Test assertion failure function __real_main3, file ..\/runner\/testdata\/dynamic.c:13:\n(dynamic:20904) CRITICAL: Failed assertion: false\nDynamic subtest this-is-dynamic-2 failed.\n**** DEBUG ****\n(dynamic:20904) DEBUG: This print is from 2\n(dynamic:20904) CRITICAL: Test assertion failure function __real_main3, file ..\/runner\/testdata\/dynamic.c:13:\n(dynamic:20904) CRITICAL: Failed assertion: false\n(dynamic:20904) igt_core-INFO: Stack trace:\n(dynamic:20904) igt_core-INFO:   #0 ..\/lib\/igt_core.c:1607 __igt_fail_assert()\n(dynamic:20904) igt_core-INFO:   #1 ..\/runner\/testdata\/dynamic.c:5 __real_main3()\n(dynamic:20904) igt_core-INFO:   #2 ..\/runner\/testdata\/dynamic.c:3 main()\n(dynamic:20904) igt_core-INFO:   #3 ..\/csu\/libc-start.c:342 __libc_start_main()\n(dynamic:20904) igt_core-INFO:   #4 [_start+0x2a]\n****  END  ****\nDynamic subtest this-is-dynamic-2: FAIL (0.054s)\nSubtest debug-log-checking failed.\nNo log.\nSubtest debug-log-checking: FAIL (0.109s)\n",
-      "dmesg":"<6> [23426155.175691] Console: switching to colour dummy device 80x25\n<6> [23426155.175708] [IGT] dynamic: executing\n<6> [23426155.184875] [IGT] dynamic: starting subtest debug-log-checking\n<6> [23426155.184895] [IGT] dynamic: starting dynamic subtest this-is-dynamic-1\n<6> [23426155.240164] [IGT] dynamic: starting dynamic subtest this-is-dynamic-2\n<6> [23426155.293846] [IGT] dynamic: exiting, ret=98\n<6> [23426155.294003] Console: switching to colour frame buffer device 240x75\n"
-    },
     "igt@dynamic@debug-log-checking@this-is-dynamic-1":{
       "out":"IGT-Version: 1.23-g9e957acd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)\nStarting subtest: debug-log-checking\nStarting dynamic subtest: this-is-dynamic-1\nStack trace:\n  #0 ..\/lib\/igt_core.c:1607 __igt_fail_assert()\n  #1 ..\/runner\/testdata\/dynamic.c:11 __real_main3()\n  #2 ..\/runner\/testdata\/dynamic.c:3 main()\n  #3 ..\/csu\/libc-start.c:342 __libc_start_main()\n  #4 [_start+0x2a]\nDynamic subtest this-is-dynamic-1: FAIL (0.055s)\n",
       "igt-version":"IGT-Version: 1.23-g9e957acd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)",
@@ -57,18 +45,6 @@
       "err":"Starting subtest: empty-container\nSubtest empty-container: SKIP (0.000s)\n",
       "dmesg":"<6> [23426155.304955] Console: switching to colour dummy device 80x25\n<6> [23426155.304968] [IGT] dynamic: executing\n<6> [23426155.308644] [IGT] dynamic: starting subtest empty-container\n<6> [23426155.308671] [IGT] dynamic: exiting, ret=77\n<6> [23426155.308822] Console: switching to colour frame buffer device 240x75\n"
     },
-    "igt@dynamic@normal":{
-      "out":"IGT-Version: 1.23-g9e957acd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)\nStarting subtest: normal\nStarting dynamic subtest: normal-dynamic-subtest\nDynamic subtest normal-dynamic-subtest: SUCCESS (0.055s)\nSubtest normal: SUCCESS (0.100s)\n",
-      "igt-version":"IGT-Version: 1.23-g9e957acd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)",
-      "result":"pass",
-      "time":{
-        "__type__":"TimeAttribute",
-        "start":0,
-        "end":0.100
-      },
-      "err":"Starting subtest: normal\nStarting dynamic subtest: normal-dynamic-subtest\nDynamic subtest normal-dynamic-subtest: SUCCESS (0.055s)\nSubtest normal: SUCCESS (0.100s)\n",
-      "dmesg":"<6> [23426155.175691] Console: switching to colour dummy device 80x25\n<6> [23426155.175708] [IGT] dynamic: executing\n<6> [23426155.184875] [IGT] dynamic: starting subtest normal\n<6> [23426155.184895] Dmesg output for normal\n<6> [23426155.184895] [IGT] dynamic: starting dynamic subtest normal-dynamic-subtest\n"
-    },
     "igt@dynamic@normal@normal-dynamic-subtest":{
       "out":"IGT-Version: 1.23-g9e957acd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)\nStarting subtest: normal\nStarting dynamic subtest: normal-dynamic-subtest\nDynamic subtest normal-dynamic-subtest: SUCCESS (0.055s)\nSubtest normal: SUCCESS (0.100s)\n",
       "igt-version":"IGT-Version: 1.23-g9e957acd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)",
@@ -81,18 +57,6 @@
       "err":"Starting subtest: normal\nStarting dynamic subtest: normal-dynamic-subtest\nDynamic subtest normal-dynamic-subtest: SUCCESS (0.055s)\nSubtest normal: SUCCESS (0.100s)\n",
       "dmesg":"<6> [23426155.175691] Console: switching to colour dummy device 80x25\n<6> [23426155.175708] [IGT] dynamic: executing\n<6> [23426155.184875] [IGT] dynamic: starting subtest normal\n<6> [23426155.184895] Dmesg output for normal\n<6> [23426155.184895] [IGT] dynamic: starting dynamic subtest normal-dynamic-subtest\n"
     },
-    "igt@dynamic@incomplete":{
-      "out":"Starting subtest: incomplete\nStarting dynamic subtest: this-is-incomplete\nThis is some output\n",
-      "igt-version":"IGT-Version: 1.23-g9e957acd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)",
-      "result":"incomplete",
-      "time":{
-        "__type__":"TimeAttribute",
-        "start":0,
-        "end":0
-      },
-      "err":"Starting subtest: incomplete\nStarting dynamic subtest: this-is-incomplete\n",
-      "dmesg":"<6> [23426155.184875] [IGT] dynamic: starting subtest incomplete\n<6> [23426155.184895] [IGT] dynamic: starting dynamic subtest this-is-incomplete\n<6> [23426155.184895] Dmesg output for incomplete\n"
-    },
     "igt@dynamic@incomplete@this-is-incomplete":{
       "out":"Starting subtest: incomplete\nStarting dynamic subtest: this-is-incomplete\nThis is some output\n",
       "igt-version":"IGT-Version: 1.23-g9e957acd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)",
@@ -121,38 +85,38 @@
   "totals":{
     "":{
       "crash":0,
-      "pass":3,
+      "pass":2,
       "dmesg-fail":0,
       "dmesg-warn":0,
       "skip":1,
-      "incomplete":2,
+      "incomplete":1,
       "timeout":0,
       "notrun":0,
-      "fail":3,
+      "fail":2,
       "warn":0
     },
     "root":{
       "crash":0,
-      "pass":3,
+      "pass":2,
       "dmesg-fail":0,
       "dmesg-warn":0,
       "skip":1,
-      "incomplete":2,
+      "incomplete":1,
       "timeout":0,
       "notrun":0,
-      "fail":3,
+      "fail":2,
       "warn":0
     },
     "igt@dynamic":{
       "crash":0,
-      "pass":3,
+      "pass":2,
       "dmesg-fail":0,
       "dmesg-warn":0,
       "skip":1,
-      "incomplete":2,
+      "incomplete":1,
       "timeout":0,
       "notrun":0,
-      "fail":3,
+      "fail":2,
       "warn":0
     }
   },
-- 
2.19.1

_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev

[igt-dev] [PATCH i-g-t 11/11] runner/json_tests: Add test for parsing dynamic subtests with same name

[Petri Latvala]

Multiple different subtests can have a dynamic subtest with the same
name. Add a test to make sure we correctly delimit the output parsing.

Signed-off-by: Petri Latvala <petri.latvala@intel.com>
---
 .../0/dmesg.txt                               | 10 +++
 .../0/err.txt                                 | 10 +++
 .../0/journal.txt                             |  3 +
 .../0/out.txt                                 | 11 +++
 .../README.txt                                |  2 +
 .../endtime.txt                               |  1 +
 .../joblist.txt                               |  1 +
 .../metadata.txt                              | 12 +++
 .../reference.json                            | 84 +++++++++++++++++++
 .../starttime.txt                             |  1 +
 .../uname.txt                                 |  1 +
 runner/runner_json_tests.c                    |  1 +
 12 files changed, 137 insertions(+)
 create mode 100644 runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/0/dmesg.txt
 create mode 100644 runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/0/err.txt
 create mode 100644 runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/0/journal.txt
 create mode 100644 runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/0/out.txt
 create mode 100644 runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/README.txt
 create mode 100644 runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/endtime.txt
 create mode 100644 runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/joblist.txt
 create mode 100644 runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/metadata.txt
 create mode 100644 runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/reference.json
 create mode 100644 runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/starttime.txt
 create mode 100644 runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/uname.txt

diff --git a/runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/0/dmesg.txt b/runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/0/dmesg.txt
new file mode 100644
index 00000000..59741e5f
--- /dev/null
+++ b/runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/0/dmesg.txt
@@ -0,0 +1,10 @@
+6,1157,23426155175691,-;Console: switching to colour dummy device 80x25
+14,1158,23426155175708,-;[IGT] dynamic: executing
+14,1159,23426155184875,-;[IGT] dynamic: starting subtest subtest-one
+14,1160,23426155184895,-;[IGT] dynamic: starting dynamic subtest this-name-is-shared
+14,1160,23426155184895,-;This dmesg output is from subtest-one/this-name-is-shared
+14,1159,23426155184875,-;[IGT] dynamic: starting subtest subtest-two
+14,1160,23426155184895,-;[IGT] dynamic: starting dynamic subtest this-name-is-shared
+14,1160,23426155184895,-;This dmesg output is from subtest-two/this-name-is-shared
+14,1162,23426155293846,-;[IGT] dynamic: exiting, ret=0
+6,1163,23426155294003,-;Console: switching to colour frame buffer device 240x75
diff --git a/runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/0/err.txt b/runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/0/err.txt
new file mode 100644
index 00000000..3d551819
--- /dev/null
+++ b/runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/0/err.txt
@@ -0,0 +1,10 @@
+Starting subtest: subtest-one
+Starting dynamic subtest: this-name-is-shared
+This stderr output is from subtest-one/this-name-is-shared
+Dynamic subtest this-name-is-shared: SUCCESS (0.050s)
+Subtest subtest-one: SUCCESS (0.050s)
+Starting subtest: subtest-two
+Starting dynamic subtest: this-name-is-shared
+This stderr output is from subtest-two/this-name-is-shared
+Dynamic subtest this-name-is-shared: SUCCESS (0.050s)
+Subtest subtest-two: SUCCESS (0.050s)
diff --git a/runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/0/journal.txt b/runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/0/journal.txt
new file mode 100644
index 00000000..3af2b007
--- /dev/null
+++ b/runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/0/journal.txt
@@ -0,0 +1,3 @@
+subtest-one
+subtest-two
+exit:0 (0.100s)
diff --git a/runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/0/out.txt b/runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/0/out.txt
new file mode 100644
index 00000000..e025c096
--- /dev/null
+++ b/runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/0/out.txt
@@ -0,0 +1,11 @@
+IGT-Version: 1.23-g9e957acd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)
+Starting subtest: subtest-one
+Starting dynamic subtest: this-name-is-shared
+This stdout output is from subtest-one/this-name-is-shared
+Dynamic subtest this-name-is-shared: SUCCESS (0.050s)
+Subtest subtest-one: SUCCESS (0.050s)
+Starting subtest: subtest-two
+Starting dynamic subtest: this-name-is-shared
+This stdout output is from subtest-two/this-name-is-shared
+Dynamic subtest this-name-is-shared: SUCCESS (0.050s)
+Subtest subtest-two: SUCCESS (0.050s)
diff --git a/runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/README.txt b/runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/README.txt
new file mode 100644
index 00000000..e141077d
--- /dev/null
+++ b/runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/README.txt
@@ -0,0 +1,2 @@
+Parsing of dynamic subtest output is tricky if multiple subtests use
+the same name for a dynamic subtest.
diff --git a/runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/endtime.txt b/runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/endtime.txt
new file mode 100644
index 00000000..5c7608b5
--- /dev/null
+++ b/runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/endtime.txt
@@ -0,0 +1 @@
+1560163492.410489
diff --git a/runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/joblist.txt b/runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/joblist.txt
new file mode 100644
index 00000000..9296b9ed
--- /dev/null
+++ b/runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/joblist.txt
@@ -0,0 +1 @@
+dynamic subtest-one,subtest-two
diff --git a/runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/metadata.txt b/runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/metadata.txt
new file mode 100644
index 00000000..8da1aee2
--- /dev/null
+++ b/runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/metadata.txt
@@ -0,0 +1,12 @@
+abort_mask : 0
+name : dynamic-subtest-name-in-multiple-subtests
+dry_run : 0
+sync : 0
+log_level : 0
+overwrite : 0
+multiple_mode : 1
+inactivity_timeout : 0
+use_watchdog : 0
+piglit_style_dmesg : 0
+test_root : /path/does/not/exist
+results_path : /path/does/not/exist
diff --git a/runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/reference.json b/runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/reference.json
new file mode 100644
index 00000000..370fce4d
--- /dev/null
+++ b/runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/reference.json
@@ -0,0 +1,84 @@
+{
+  "__type__":"TestrunResult",
+  "results_version":10,
+  "name":"dynamic-subtest-name-in-multiple-subtests",
+  "uname":"Linux hostname 4.18.0-1-amd64 #1 SMP Debian 4.18.6-1 (2018-09-06) x86_64",
+  "time_elapsed":{
+    "__type__":"TimeAttribute",
+    "start":1560163492.266377,
+    "end":1560163492.4104891
+  },
+  "tests":{
+    "igt@dynamic@subtest-one@this-name-is-shared":{
+      "out":"IGT-Version: 1.23-g9e957acd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)\nStarting subtest: subtest-one\nStarting dynamic subtest: this-name-is-shared\nThis stdout output is from subtest-one\/this-name-is-shared\nDynamic subtest this-name-is-shared: SUCCESS (0.050s)\nSubtest subtest-one: SUCCESS (0.050s)\n",
+      "igt-version":"IGT-Version: 1.23-g9e957acd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)",
+      "result":"warn",
+      "time":{
+        "__type__":"TimeAttribute",
+        "start":0,
+        "end":0.050000000000000003
+      },
+      "err":"Starting subtest: subtest-one\nStarting dynamic subtest: this-name-is-shared\nThis stderr output is from subtest-one\/this-name-is-shared\nDynamic subtest this-name-is-shared: SUCCESS (0.050s)\nSubtest subtest-one: SUCCESS (0.050s)\n",
+      "dmesg":"<6> [23426155.175691] Console: switching to colour dummy device 80x25\n<6> [23426155.175708] [IGT] dynamic: executing\n<6> [23426155.184875] [IGT] dynamic: starting subtest subtest-one\n<6> [23426155.184895] [IGT] dynamic: starting dynamic subtest this-name-is-shared\n<6> [23426155.184895] This dmesg output is from subtest-one\/this-name-is-shared\n"
+    },
+    "igt@dynamic@subtest-two@this-name-is-shared":{
+      "out":"Starting subtest: subtest-two\nStarting dynamic subtest: this-name-is-shared\nThis stdout output is from subtest-two\/this-name-is-shared\nDynamic subtest this-name-is-shared: SUCCESS (0.050s)\nSubtest subtest-two: SUCCESS (0.050s)\n",
+      "igt-version":"IGT-Version: 1.23-g9e957acd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)",
+      "result":"warn",
+      "time":{
+        "__type__":"TimeAttribute",
+        "start":0,
+        "end":0.050000000000000003
+      },
+      "err":"Starting subtest: subtest-two\nStarting dynamic subtest: this-name-is-shared\nThis stderr output is from subtest-two\/this-name-is-shared\nDynamic subtest this-name-is-shared: SUCCESS (0.050s)\nSubtest subtest-two: SUCCESS (0.050s)\n",
+      "dmesg":"<6> [23426155.184875] [IGT] dynamic: starting subtest subtest-two\n<6> [23426155.184895] [IGT] dynamic: starting dynamic subtest this-name-is-shared\n<6> [23426155.184895] This dmesg output is from subtest-two\/this-name-is-shared\n<6> [23426155.293846] [IGT] dynamic: exiting, ret=0\n<6> [23426155.294003] Console: switching to colour frame buffer device 240x75\n"
+    }
+  },
+  "totals":{
+    "":{
+      "crash":0,
+      "pass":0,
+      "dmesg-fail":0,
+      "dmesg-warn":0,
+      "skip":0,
+      "incomplete":0,
+      "timeout":0,
+      "notrun":0,
+      "fail":0,
+      "warn":2
+    },
+    "root":{
+      "crash":0,
+      "pass":0,
+      "dmesg-fail":0,
+      "dmesg-warn":0,
+      "skip":0,
+      "incomplete":0,
+      "timeout":0,
+      "notrun":0,
+      "fail":0,
+      "warn":2
+    },
+    "igt@dynamic":{
+      "crash":0,
+      "pass":0,
+      "dmesg-fail":0,
+      "dmesg-warn":0,
+      "skip":0,
+      "incomplete":0,
+      "timeout":0,
+      "notrun":0,
+      "fail":0,
+      "warn":2
+    }
+  },
+  "runtimes":{
+    "igt@dynamic":{
+      "time":{
+        "__type__":"TimeAttribute",
+        "start":0,
+        "end":0.10000000000000001
+      }
+    }
+  }
+}
diff --git a/runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/starttime.txt b/runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/starttime.txt
new file mode 100644
index 00000000..b30b32ee
--- /dev/null
+++ b/runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/starttime.txt
@@ -0,0 +1 @@
+1560163492.266377
diff --git a/runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/uname.txt b/runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/uname.txt
new file mode 100644
index 00000000..a7aef6f7
--- /dev/null
+++ b/runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/uname.txt
@@ -0,0 +1 @@
+Linux hostname 4.18.0-1-amd64 #1 SMP Debian 4.18.6-1 (2018-09-06) x86_64
diff --git a/runner/runner_json_tests.c b/runner/runner_json_tests.c
index 3443bd2b..21e87a02 100644
--- a/runner/runner_json_tests.c
+++ b/runner/runner_json_tests.c
@@ -163,6 +163,7 @@ static const char *dirnames[] = {
 	"dmesg-warn-level-piglit-style",
 	"dmesg-warn-level-one-piglit-style",
 	"dynamic-subtests",
+	"dynamic-subtest-name-in-multiple-subtests",
 };
 
 igt_main
-- 
2.19.1

_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev

[igt-dev] ✓ Fi.CI.BAT: success for runner: Don't report subtests that have dynamic subtests

[Patchwork]

== Series Details ==

Series: runner: Don't report subtests that have dynamic subtests
URL   : https://patchwork.freedesktop.org/series/70286/
State : success

== Summary ==

CI Bug Log - changes from CI_DRM_7460 -> IGTPW_3794
====================================================

Summary
-------

  **SUCCESS**

  No regressions found.

  External URL: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/index.html

Known issues
------------

  Here are the changes found in IGTPW_3794 that come from known issues:

### IGT changes ###

#### Issues hit ####

  * igt@i915_pm_rpm@module-reload:
    - fi-skl-6770hq:      [PASS][1] -> [FAIL][2] ([i915#178])
   [1]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/fi-skl-6770hq/igt@i915_pm_rpm@module-reload.html
   [2]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/fi-skl-6770hq/igt@i915_pm_rpm@module-reload.html

  * igt@i915_selftest@live_blt:
    - fi-hsw-4770:        [PASS][3] -> [DMESG-FAIL][4] ([i915#563])
   [3]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/fi-hsw-4770/igt@i915_selftest@live_blt.html
   [4]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/fi-hsw-4770/igt@i915_selftest@live_blt.html

  
#### Possible fixes ####

  * igt@i915_module_load@reload-with-fault-injection:
    - {fi-kbl-7560u}:     [INCOMPLETE][5] ([i915#243] / [i915#609]) -> [PASS][6]
   [5]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/fi-kbl-7560u/igt@i915_module_load@reload-with-fault-injection.html
   [6]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/fi-kbl-7560u/igt@i915_module_load@reload-with-fault-injection.html

  * igt@kms_frontbuffer_tracking@basic:
    - fi-icl-u2:          [FAIL][7] ([i915#49]) -> [PASS][8]
   [7]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/fi-icl-u2/igt@kms_frontbuffer_tracking@basic.html
   [8]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/fi-icl-u2/igt@kms_frontbuffer_tracking@basic.html

  
#### Warnings ####

  * igt@kms_cursor_legacy@basic-busy-flip-before-cursor-legacy:
    - fi-kbl-x1275:       [DMESG-WARN][9] ([i915#62] / [i915#92]) -> [DMESG-WARN][10] ([i915#62] / [i915#92] / [i915#95]) +4 similar issues
   [9]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/fi-kbl-x1275/igt@kms_cursor_legacy@basic-busy-flip-before-cursor-legacy.html
   [10]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/fi-kbl-x1275/igt@kms_cursor_legacy@basic-busy-flip-before-cursor-legacy.html

  * igt@kms_cursor_legacy@basic-flip-after-cursor-atomic:
    - fi-kbl-x1275:       [DMESG-WARN][11] ([i915#62] / [i915#92] / [i915#95]) -> [DMESG-WARN][12] ([i915#62] / [i915#92]) +2 similar issues
   [11]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/fi-kbl-x1275/igt@kms_cursor_legacy@basic-flip-after-cursor-atomic.html
   [12]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/fi-kbl-x1275/igt@kms_cursor_legacy@basic-flip-after-cursor-atomic.html

  
  {name}: This element is suppressed. This means it is ignored when computing
          the status of the difference (SUCCESS, WARNING, or FAILURE).

  [i915#178]: https://gitlab.freedesktop.org/drm/intel/issues/178
  [i915#243]: https://gitlab.freedesktop.org/drm/intel/issues/243
  [i915#49]: https://gitlab.freedesktop.org/drm/intel/issues/49
  [i915#563]: https://gitlab.freedesktop.org/drm/intel/issues/563
  [i915#609]: https://gitlab.freedesktop.org/drm/intel/issues/609
  [i915#62]: https://gitlab.freedesktop.org/drm/intel/issues/62
  [i915#92]: https://gitlab.freedesktop.org/drm/intel/issues/92
  [i915#95]: https://gitlab.freedesktop.org/drm/intel/issues/95


Participating hosts (50 -> 46)
------------------------------

  Additional (1): fi-tgl-y 
  Missing    (5): fi-ilk-m540 fi-hsw-4200u fi-bsw-cyan fi-ctg-p8600 fi-byt-clapper 


Build changes
-------------

  * CI: CI-20190529 -> None
  * IGT: IGT_5320 -> IGTPW_3794

  CI-20190529: 20190529
  CI_DRM_7460: 9cd639929b6ad1efb35e90ee862cf37a9b6f6348 @ git://anongit.freedesktop.org/gfx-ci/linux
  IGTPW_3794: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/index.html
  IGT_5320: 06833ffeaf7621c3fef097166539bbe999a33e9c @ git://anongit.freedesktop.org/xorg/app/intel-gpu-tools

== Logs ==

For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/index.html
_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev

[igt-dev] ✓ Fi.CI.IGT: success for runner: Don't report subtests that have dynamic subtests

[Patchwork]

== Series Details ==

Series: runner: Don't report subtests that have dynamic subtests
URL   : https://patchwork.freedesktop.org/series/70286/
State : success

== Summary ==

CI Bug Log - changes from CI_DRM_7460_full -> IGTPW_3794_full
====================================================

Summary
-------

  **SUCCESS**

  No regressions found.

  External URL: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/index.html

Known issues
------------

  Here are the changes found in IGTPW_3794_full that come from known issues:

### IGT changes ###

#### Issues hit ####

  * igt@gem_ctx_persistence@vcs1-cleanup:
    - shard-iclb:         [PASS][1] -> [SKIP][2] ([fdo#109276] / [fdo#112080]) +1 similar issue
   [1]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-iclb2/igt@gem_ctx_persistence@vcs1-cleanup.html
   [2]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-iclb3/igt@gem_ctx_persistence@vcs1-cleanup.html

  * igt@gem_exec_parallel@bcs0-contexts:
    - shard-hsw:          [PASS][3] -> [FAIL][4] ([i915#676]) +1 similar issue
   [3]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-hsw5/igt@gem_exec_parallel@bcs0-contexts.html
   [4]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-hsw4/igt@gem_exec_parallel@bcs0-contexts.html

  * igt@gem_exec_schedule@independent-bsd1:
    - shard-iclb:         [PASS][5] -> [SKIP][6] ([fdo#109276]) +1 similar issue
   [5]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-iclb4/igt@gem_exec_schedule@independent-bsd1.html
   [6]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-iclb5/igt@gem_exec_schedule@independent-bsd1.html

  * igt@gem_exec_schedule@preempt-queue-contexts-chain-bsd:
    - shard-iclb:         [PASS][7] -> [SKIP][8] ([fdo#112146])
   [7]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-iclb5/igt@gem_exec_schedule@preempt-queue-contexts-chain-bsd.html
   [8]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-iclb2/igt@gem_exec_schedule@preempt-queue-contexts-chain-bsd.html

  * igt@gem_exec_schedule@smoketest-all:
    - shard-tglb:         [PASS][9] -> [INCOMPLETE][10] ([i915#463])
   [9]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-tglb1/igt@gem_exec_schedule@smoketest-all.html
   [10]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-tglb4/igt@gem_exec_schedule@smoketest-all.html

  * igt@gem_userptr_blits@map-fixed-invalidate-busy:
    - shard-snb:          [PASS][11] -> [DMESG-WARN][12] ([fdo#111870])
   [11]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-snb6/igt@gem_userptr_blits@map-fixed-invalidate-busy.html
   [12]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-snb7/igt@gem_userptr_blits@map-fixed-invalidate-busy.html

  * igt@i915_pm_dc@dc6-psr:
    - shard-iclb:         [PASS][13] -> [FAIL][14] ([i915#454])
   [13]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-iclb2/igt@i915_pm_dc@dc6-psr.html
   [14]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-iclb3/igt@i915_pm_dc@dc6-psr.html

  * igt@kms_cursor_crc@pipe-a-cursor-256x85-onscreen:
    - shard-hsw:          [PASS][15] -> [DMESG-WARN][16] ([IGT#6])
   [15]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-hsw4/igt@kms_cursor_crc@pipe-a-cursor-256x85-onscreen.html
   [16]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-hsw6/igt@kms_cursor_crc@pipe-a-cursor-256x85-onscreen.html

  * igt@kms_cursor_crc@pipe-b-cursor-suspend:
    - shard-apl:          [PASS][17] -> [DMESG-FAIL][18] ([i915#180] / [i915#54])
   [17]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-apl1/igt@kms_cursor_crc@pipe-b-cursor-suspend.html
   [18]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-apl6/igt@kms_cursor_crc@pipe-b-cursor-suspend.html
    - shard-glk:          [PASS][19] -> [FAIL][20] ([i915#54])
   [19]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-glk4/igt@kms_cursor_crc@pipe-b-cursor-suspend.html
   [20]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-glk6/igt@kms_cursor_crc@pipe-b-cursor-suspend.html

  * igt@kms_draw_crc@draw-method-rgb565-render-xtiled:
    - shard-iclb:         [PASS][21] -> [INCOMPLETE][22] ([i915#140])
   [21]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-iclb5/igt@kms_draw_crc@draw-method-rgb565-render-xtiled.html
   [22]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-iclb4/igt@kms_draw_crc@draw-method-rgb565-render-xtiled.html

  * igt@kms_flip@2x-flip-vs-expired-vblank:
    - shard-glk:          [PASS][23] -> [FAIL][24] ([i915#79])
   [23]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-glk6/igt@kms_flip@2x-flip-vs-expired-vblank.html
   [24]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-glk4/igt@kms_flip@2x-flip-vs-expired-vblank.html

  * igt@kms_frontbuffer_tracking@fbc-1p-offscren-pri-shrfb-draw-blt:
    - shard-iclb:         [PASS][25] -> [FAIL][26] ([i915#49])
   [25]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-iclb2/igt@kms_frontbuffer_tracking@fbc-1p-offscren-pri-shrfb-draw-blt.html
   [26]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-iclb8/igt@kms_frontbuffer_tracking@fbc-1p-offscren-pri-shrfb-draw-blt.html

  * igt@kms_frontbuffer_tracking@fbc-1p-primscrn-indfb-plflip-blt:
    - shard-tglb:         [PASS][27] -> [FAIL][28] ([i915#49])
   [27]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-tglb7/igt@kms_frontbuffer_tracking@fbc-1p-primscrn-indfb-plflip-blt.html
   [28]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-tglb9/igt@kms_frontbuffer_tracking@fbc-1p-primscrn-indfb-plflip-blt.html

  * igt@kms_frontbuffer_tracking@fbc-1p-rte:
    - shard-iclb:         [PASS][29] -> [FAIL][30] ([fdo#110378] / [i915#49])
   [29]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-iclb5/igt@kms_frontbuffer_tracking@fbc-1p-rte.html
   [30]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-iclb5/igt@kms_frontbuffer_tracking@fbc-1p-rte.html

  * igt@kms_frontbuffer_tracking@fbcpsr-1p-primscrn-spr-indfb-draw-mmap-wc:
    - shard-tglb:         [PASS][31] -> [INCOMPLETE][32] ([i915#474])
   [31]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-tglb2/igt@kms_frontbuffer_tracking@fbcpsr-1p-primscrn-spr-indfb-draw-mmap-wc.html
   [32]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-tglb3/igt@kms_frontbuffer_tracking@fbcpsr-1p-primscrn-spr-indfb-draw-mmap-wc.html

  * igt@kms_plane@plane-panning-bottom-right-suspend-pipe-a-planes:
    - shard-tglb:         [PASS][33] -> [INCOMPLETE][34] ([i915#456] / [i915#460])
   [33]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-tglb6/igt@kms_plane@plane-panning-bottom-right-suspend-pipe-a-planes.html
   [34]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-tglb1/igt@kms_plane@plane-panning-bottom-right-suspend-pipe-a-planes.html

  * igt@kms_vblank@pipe-a-ts-continuation-suspend:
    - shard-kbl:          [PASS][35] -> [DMESG-WARN][36] ([i915#180]) +7 similar issues
   [35]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-kbl4/igt@kms_vblank@pipe-a-ts-continuation-suspend.html
   [36]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-kbl2/igt@kms_vblank@pipe-a-ts-continuation-suspend.html

  
#### Possible fixes ####

  * igt@gem_ctx_isolation@rcs0-s3:
    - shard-kbl:          [DMESG-WARN][37] ([i915#180]) -> [PASS][38] +2 similar issues
   [37]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-kbl4/igt@gem_ctx_isolation@rcs0-s3.html
   [38]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-kbl6/igt@gem_ctx_isolation@rcs0-s3.html

  * igt@gem_ctx_isolation@vcs1-s3:
    - shard-iclb:         [SKIP][39] ([fdo#109276] / [fdo#112080]) -> [PASS][40]
   [39]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-iclb5/igt@gem_ctx_isolation@vcs1-s3.html
   [40]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-iclb2/igt@gem_ctx_isolation@vcs1-s3.html

  * igt@gem_exec_parallel@vcs0-fds:
    - shard-hsw:          [FAIL][41] ([i915#676]) -> [PASS][42]
   [41]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-hsw4/igt@gem_exec_parallel@vcs0-fds.html
   [42]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-hsw7/igt@gem_exec_parallel@vcs0-fds.html

  * igt@gem_exec_schedule@in-order-bsd2:
    - shard-iclb:         [SKIP][43] ([fdo#109276]) -> [PASS][44] +4 similar issues
   [43]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-iclb6/igt@gem_exec_schedule@in-order-bsd2.html
   [44]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-iclb4/igt@gem_exec_schedule@in-order-bsd2.html

  * igt@gem_exec_schedule@reorder-wide-bsd:
    - shard-iclb:         [SKIP][45] ([fdo#112146]) -> [PASS][46] +1 similar issue
   [45]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-iclb4/igt@gem_exec_schedule@reorder-wide-bsd.html
   [46]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-iclb5/igt@gem_exec_schedule@reorder-wide-bsd.html

  * igt@gem_ppgtt@flink-and-close-vma-leak:
    - shard-apl:          [FAIL][47] ([i915#644]) -> [PASS][48]
   [47]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-apl6/igt@gem_ppgtt@flink-and-close-vma-leak.html
   [48]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-apl4/igt@gem_ppgtt@flink-and-close-vma-leak.html

  * igt@gem_userptr_blits@sync-unmap-cycles:
    - shard-snb:          [DMESG-WARN][49] ([fdo#111870]) -> [PASS][50] +1 similar issue
   [49]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-snb4/igt@gem_userptr_blits@sync-unmap-cycles.html
   [50]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-snb6/igt@gem_userptr_blits@sync-unmap-cycles.html

  * {igt@i915_pm_dc@dc3co-vpb-simulation}:
    - shard-tglb:         [SKIP][51] ([i915#657]) -> [PASS][52]
   [51]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-tglb2/igt@i915_pm_dc@dc3co-vpb-simulation.html
   [52]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-tglb1/igt@i915_pm_dc@dc3co-vpb-simulation.html

  * igt@i915_selftest@live_blt:
    - shard-hsw:          [DMESG-FAIL][53] ([i915#683]) -> [PASS][54]
   [53]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-hsw7/igt@i915_selftest@live_blt.html
   [54]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-hsw4/igt@i915_selftest@live_blt.html

  * igt@kms_color@pipe-a-ctm-green-to-red:
    - shard-kbl:          [FAIL][55] ([i915#129]) -> [PASS][56]
   [55]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-kbl2/igt@kms_color@pipe-a-ctm-green-to-red.html
   [56]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-kbl4/igt@kms_color@pipe-a-ctm-green-to-red.html
    - shard-apl:          [FAIL][57] ([i915#129]) -> [PASS][58]
   [57]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-apl2/igt@kms_color@pipe-a-ctm-green-to-red.html
   [58]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-apl6/igt@kms_color@pipe-a-ctm-green-to-red.html

  * igt@kms_frontbuffer_tracking@fbcpsr-indfb-scaledprimary:
    - shard-tglb:         [FAIL][59] ([i915#49]) -> [PASS][60]
   [59]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-tglb2/igt@kms_frontbuffer_tracking@fbcpsr-indfb-scaledprimary.html
   [60]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-tglb2/igt@kms_frontbuffer_tracking@fbcpsr-indfb-scaledprimary.html

  * igt@kms_plane@pixel-format-pipe-a-planes:
    - shard-kbl:          [INCOMPLETE][61] ([fdo#103665]) -> [PASS][62] +1 similar issue
   [61]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-kbl3/igt@kms_plane@pixel-format-pipe-a-planes.html
   [62]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-kbl2/igt@kms_plane@pixel-format-pipe-a-planes.html

  * igt@kms_psr@psr2_dpms:
    - shard-iclb:         [SKIP][63] ([fdo#109441]) -> [PASS][64] +1 similar issue
   [63]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-iclb5/igt@kms_psr@psr2_dpms.html
   [64]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-iclb2/igt@kms_psr@psr2_dpms.html

  * igt@kms_vblank@pipe-b-ts-continuation-suspend:
    - shard-apl:          [DMESG-WARN][65] ([i915#180]) -> [PASS][66] +1 similar issue
   [65]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-apl4/igt@kms_vblank@pipe-b-ts-continuation-suspend.html
   [66]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-apl6/igt@kms_vblank@pipe-b-ts-continuation-suspend.html

  * {igt@perf_pmu@frequency-idle}:
    - shard-hsw:          [FAIL][67] ([i915#675]) -> [PASS][68]
   [67]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-hsw4/igt@perf_pmu@frequency-idle.html
   [68]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-hsw5/igt@perf_pmu@frequency-idle.html

  
#### Warnings ####

  * igt@gem_exec_parallel@contexts:
    - shard-hsw:          [FAIL][69] ([i915#676]) -> [TIMEOUT][70] ([i915#676])
   [69]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-hsw1/igt@gem_exec_parallel@contexts.html
   [70]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-hsw5/igt@gem_exec_parallel@contexts.html

  * igt@gem_workarounds@suspend-resume-context:
    - shard-kbl:          [INCOMPLETE][71] ([fdo#103665]) -> [DMESG-WARN][72] ([i915#180])
   [71]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-kbl3/igt@gem_workarounds@suspend-resume-context.html
   [72]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-kbl6/igt@gem_workarounds@suspend-resume-context.html

  * igt@kms_cursor_crc@pipe-b-cursor-suspend:
    - shard-kbl:          [DMESG-WARN][73] ([i915#180]) -> [FAIL][74] ([i915#54])
   [73]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-kbl2/igt@kms_cursor_crc@pipe-b-cursor-suspend.html
   [74]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-kbl7/igt@kms_cursor_crc@pipe-b-cursor-suspend.html

  * igt@kms_flip@flip-vs-suspend-interruptible:
    - shard-kbl:          [DMESG-WARN][75] ([i915#180] / [i915#391]) -> [DMESG-WARN][76] ([i915#180])
   [75]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-kbl7/igt@kms_flip@flip-vs-suspend-interruptible.html
   [76]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-kbl1/igt@kms_flip@flip-vs-suspend-interruptible.html

  
  {name}: This element is suppressed. This means it is ignored when computing
          the status of the difference (SUCCESS, WARNING, or FAILURE).

  [IGT#6]: https://gitlab.freedesktop.org/drm/igt-gpu-tools/issues/6
  [fdo#103665]: https://bugs.freedesktop.org/show_bug.cgi?id=103665
  [fdo#109276]: https://bugs.freedesktop.org/show_bug.cgi?id=109276
  [fdo#109441]: https://bugs.freedesktop.org/show_bug.cgi?id=109441
  [fdo#110378]: https://bugs.freedesktop.org/show_bug.cgi?id=110378
  [fdo#111870]: https://bugs.freedesktop.org/show_bug.cgi?id=111870
  [fdo#112080]: https://bugs.freedesktop.org/show_bug.cgi?id=112080
  [fdo#112146]: https://bugs.freedesktop.org/show_bug.cgi?id=112146
  [i915#129]: https://gitlab.freedesktop.org/drm/intel/issues/129
  [i915#140]: https://gitlab.freedesktop.org/drm/intel/issues/140
  [i915#180]: https://gitlab.freedesktop.org/drm/intel/issues/180
  [i915#391]: https://gitlab.freedesktop.org/drm/intel/issues/391
  [i915#454]: https://gitlab.freedesktop.org/drm/intel/issues/454
  [i915#456]: https://gitlab.freedesktop.org/drm/intel/issues/456
  [i915#460]: https://gitlab.freedesktop.org/drm/intel/issues/460
  [i915#463]: https://gitlab.freedesktop.org/drm/intel/issues/463
  [i915#474]: https://gitlab.freedesktop.org/drm/intel/issues/474
  [i915#49]: https://gitlab.freedesktop.org/drm/intel/issues/49
  [i915#54]: https://gitlab.freedesktop.org/drm/intel/issues/54
  [i915#644]: https://gitlab.freedesktop.org/drm/intel/issues/644
  [i915#657]: https://gitlab.freedesktop.org/drm/intel/issues/657
  [i915#675]: https://gitlab.freedesktop.org/drm/intel/issues/675
  [i915#676]: https://gitlab.freedesktop.org/drm/intel/issues/676
  [i915#683]: https://gitlab.freedesktop.org/drm/intel/issues/683
  [i915#79]: https://gitlab.freedesktop.org/drm/intel/issues/79


Participating hosts (11 -> 8)
------------------------------

  Missing    (3): pig-skl-6260u pig-glk-j5005 pig-hsw-4770r 


Build changes
-------------

  * CI: CI-20190529 -> None
  * IGT: IGT_5320 -> IGTPW_3794
  * Piglit: piglit_4509 -> None

  CI-20190529: 20190529
  CI_DRM_7460: 9cd639929b6ad1efb35e90ee862cf37a9b6f6348 @ git://anongit.freedesktop.org/gfx-ci/linux
  IGTPW_3794: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/index.html
  IGT_5320: 06833ffeaf7621c3fef097166539bbe999a33e9c @ git://anongit.freedesktop.org/xorg/app/intel-gpu-tools
  piglit_4509: fdc5a4ca11124ab8413c7988896eec4c97336694 @ git://anongit.freedesktop.org/piglit

== Logs ==

For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/index.html
_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev

Re: [igt-dev] [PATCH i-g-t 02/11] runner/resultgen: Extract finding begin/end lines for a subtest to a helper

[Arkadiusz Hiler]

On Mon, Dec 02, 2019 at 03:01:00PM +0200, Petri Latvala wrote:
> Signed-off-by: Petri Latvala <petri.latvala@intel.com>
> ---
>  runner/resultgen.c | 99 ++++++++++++++++++++++++----------------------
>  1 file changed, 51 insertions(+), 48 deletions(-)
> 
> diff --git a/runner/resultgen.c b/runner/resultgen.c
> index 31bc0bb1..be058327 100644
> --- a/runner/resultgen.c
> +++ b/runner/resultgen.c
> @@ -376,6 +376,54 @@ static void add_igt_version(struct json_object *testobj,
>  
>  }
>  
> +static int find_subtest_idx_limited(struct matches matches,
> +				    const char *bufend,
> +				    const char *linekey,
> +				    const char *pattern,
> +				    const char *subtest_name,
> +				    int first,
> +				    int last)
> +{
> +	char *full_line;
> +	int line_len;
> +	int k;
> +
> +	/*
> +	 * The pattern is a string literal in all call-sites, and we
> +	 * don't want to disable this warning globally
> +	 */
> +#pragma GCC diagnostic push
> +#pragma GCC diagnostic ignored "-Wformat-nonliteral"
> +	line_len = asprintf(&full_line, pattern, linekey, subtest_name);
> +#pragma GCC diagnostic pop

I guess that you didn't go for:

__attribute__((format(printf, 5, 6)))
static int find_subtest_idx_limited(struct matches matches,
				    int first,
				    int last,
				    const char *bufend,
				    const char *pattern,
				    ...)

because linekey is needed for comparison few lines below this and
extracting it with va_start()/va_arg()/va_end() is equally ugly.

How about using an enum for selecting the desired one from a set of
locally defined patterns instead of passing it as char* with all this
pragma wizardry.

-- 
Cheers,
Arek
_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev

Re: [igt-dev] [PATCH i-g-t 02/11] runner/resultgen: Extract finding begin/end lines for a subtest to a helper

[Arkadiusz Hiler]

On Tue, Dec 03, 2019 at 03:33:45PM +0200, Arkadiusz Hiler wrote:
> On Mon, Dec 02, 2019 at 03:01:00PM +0200, Petri Latvala wrote:
> > Signed-off-by: Petri Latvala <petri.latvala@intel.com>
> > ---
> >  runner/resultgen.c | 99 ++++++++++++++++++++++++----------------------
> >  1 file changed, 51 insertions(+), 48 deletions(-)
> > 
> > diff --git a/runner/resultgen.c b/runner/resultgen.c
> > index 31bc0bb1..be058327 100644
> > --- a/runner/resultgen.c
> > +++ b/runner/resultgen.c
> > @@ -376,6 +376,54 @@ static void add_igt_version(struct json_object *testobj,
> >  
> >  }
> >  
> > +static int find_subtest_idx_limited(struct matches matches,
> > +				    const char *bufend,
> > +				    const char *linekey,
> > +				    const char *pattern,
> > +				    const char *subtest_name,
> > +				    int first,
> > +				    int last)
> > +{
> > +	char *full_line;
> > +	int line_len;
> > +	int k;
> > +
> > +	/*
> > +	 * The pattern is a string literal in all call-sites, and we
> > +	 * don't want to disable this warning globally
> > +	 */
> > +#pragma GCC diagnostic push
> > +#pragma GCC diagnostic ignored "-Wformat-nonliteral"
> > +	line_len = asprintf(&full_line, pattern, linekey, subtest_name);
> > +#pragma GCC diagnostic pop
> 
> I guess that you didn't go for:
> 
> __attribute__((format(printf, 5, 6)))
> static int find_subtest_idx_limited(struct matches matches,
> 				    int first,
> 				    int last,
> 				    const char *bufend,
> 				    const char *pattern,
> 				    ...)
> 
> because linekey is needed for comparison few lines below this and
> extracting it with va_start()/va_arg()/va_end() is equally ugly.
> 
> How about using an enum for selecting the desired one from a set of
> locally defined patterns instead of passing it as char* with all this
> pragma wizardry.

Other than the comment above the series is:
Reviewed-by: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev