All of lore.kernel.org
 help / color / mirror / Atom feed
* [LTP] [PATCH 0/2] Add support for a test tags
@ 2019-10-14 11:25 Cyril Hrubis
  2019-10-14 11:25 ` [LTP] [PATCH 1/2] lib: Add support for " Cyril Hrubis
  2019-10-14 11:25 ` [LTP] [PATCH 2/2] testcases/kernel: Add linux git and CVE tags Cyril Hrubis
  0 siblings, 2 replies; 17+ messages in thread
From: Cyril Hrubis @ 2019-10-14 11:25 UTC (permalink / raw)
  To: ltp

This is a newest version of the test tags, first patch adds a support
for test tags to a test library, the second one annotates tests with
tags. Not everything is annotated at the moment but fair amount of tests
is included.

These patches are also a stepping stone for the test metadata project
that will allow us to do much more than this. You can have a look at:

https://github.com/metan-ucw/ltp/tree/master/docparse

Also if you run a make in the docparse directory it will generate test
metadata along with a single html page with nice test metadata
description.

While the whole project is work-in-progress, the test tags are finished
and may add a value even without the rest of the changes, see the first
commit for details.

Cyril Hrubis (2):
  lib: Add support for test tags
  testcases/kernel: Add tags

 include/tst_test.h                            | 10 +++
 lib/tst_test.c                                | 77 +++++++++++++++++--
 testcases/cve/cve-2014-0196.c                 |  5 ++
 testcases/cve/cve-2015-3290.c                 |  5 ++
 testcases/cve/cve-2016-10044.c                |  5 ++
 testcases/cve/cve-2016-7042.c                 |  5 ++
 testcases/cve/cve-2016-7117.c                 |  5 ++
 testcases/cve/cve-2017-16939.c                |  5 ++
 testcases/cve/cve-2017-17052.c                |  5 ++
 testcases/cve/cve-2017-17053.c                |  5 ++
 testcases/cve/cve-2017-2618.c                 |  5 ++
 testcases/cve/cve-2017-2671.c                 |  5 ++
 testcases/cve/meltdown.c                      |  6 +-
 testcases/cve/stack_clash.c                   |  4 +
 testcases/kernel/crypto/af_alg01.c            |  5 ++
 testcases/kernel/crypto/af_alg02.c            |  5 ++
 testcases/kernel/crypto/af_alg03.c            |  4 +
 testcases/kernel/crypto/af_alg04.c            |  4 +
 testcases/kernel/crypto/af_alg05.c            |  8 +-
 testcases/kernel/crypto/af_alg06.c            |  4 +
 testcases/kernel/crypto/crypto_user01.c       |  6 ++
 testcases/kernel/crypto/crypto_user02.c       |  4 +
 testcases/kernel/crypto/pcrypt_aead01.c       |  5 ++
 testcases/kernel/sound/snd_timer01.c          |  6 ++
 testcases/kernel/syscalls/accept/accept02.c   |  5 ++
 testcases/kernel/syscalls/acct/acct02.c       |  4 +
 testcases/kernel/syscalls/add_key/add_key02.c |  5 ++
 testcases/kernel/syscalls/bpf/bpf_prog02.c    |  4 +
 .../syscalls/futex/futex_cmp_requeue02.c      |  5 ++
 testcases/kernel/syscalls/keyctl/keyctl02.c   |  4 +
 testcases/kernel/syscalls/keyctl/keyctl04.c   |  5 ++
 testcases/kernel/syscalls/keyctl/keyctl05.c   |  4 +
 .../kernel/syscalls/move_pages/move_pages12.c |  6 ++
 .../kernel/syscalls/realpath/realpath01.c     |  4 +
 .../syscalls/request_key/request_key05.c      |  4 +
 testcases/kernel/syscalls/sbrk/sbrk03.c       |  4 +
 testcases/kernel/syscalls/sendto/sendto02.c   |  4 +
 .../kernel/syscalls/setrlimit/setrlimit06.c   |  4 +
 38 files changed, 251 insertions(+), 9 deletions(-)

-- 
2.21.0


^ permalink raw reply	[flat|nested] 17+ messages in thread

* [LTP] [PATCH 1/2] lib: Add support for test tags
  2019-10-14 11:25 [LTP] [PATCH 0/2] Add support for a test tags Cyril Hrubis
@ 2019-10-14 11:25 ` Cyril Hrubis
  2019-10-15  5:12   ` Li Wang
  2019-10-16 13:30   ` Petr Vorel
  2019-10-14 11:25 ` [LTP] [PATCH 2/2] testcases/kernel: Add linux git and CVE tags Cyril Hrubis
  1 sibling, 2 replies; 17+ messages in thread
From: Cyril Hrubis @ 2019-10-14 11:25 UTC (permalink / raw)
  To: ltp

The newly introduced test tags are generic name-value pairs that can
hold test metadata, the intended use for now is to store kernel commit
hashes for kernel reproducers as well as CVE ids. The mechanism is
however choosen to be very generic so that it's easy to add basically
any information later on.

As it is the main purpose is to print hints for a test failures. If a
test that has been written as a kernel reproducer fails it prints nice
URL pointing to a kernel commit that may be missing.

Example output:

--------------------------------------------------------------------------
tst_test.c:1145: INFO: Timeout per run is 0h 05m 00s
add_key02.c:98: FAIL: unexpected error with key type 'asymmetric': EINVAL
add_key02.c:98: FAIL: unexpected error with key type 'cifs.idmap': EINVAL
add_key02.c:98: FAIL: unexpected error with key type 'cifs.spnego': EINVAL
add_key02.c:98: FAIL: unexpected error with key type 'pkcs7_test': EINVAL
add_key02.c:98: FAIL: unexpected error with key type 'rxrpc': EINVAL
add_key02.c:98: FAIL: unexpected error with key type 'rxrpc_s': EINVAL
add_key02.c:98: FAIL: unexpected error with key type 'user': EINVAL
add_key02.c:98: FAIL: unexpected error with key type 'logon': EINVAL

HINT: This is a regression test for linux kernel, see commit:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=5649645d725c

HINT: This test also tests for CVE-2017-15274, see:

https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017-15274

Summary:
passed   0
failed   8
skipped  0
warnings 0
--------------------------------------------------------------------------

Signed-off-by: Cyril Hrubis <chrubis@suse.cz>
---
 include/tst_test.h | 10 ++++++
 lib/tst_test.c     | 77 +++++++++++++++++++++++++++++++++++++++++-----
 2 files changed, 80 insertions(+), 7 deletions(-)

diff --git a/include/tst_test.h b/include/tst_test.h
index 84acf2c59..4a51b6d16 100644
--- a/include/tst_test.h
+++ b/include/tst_test.h
@@ -108,6 +108,11 @@ int tst_parse_int(const char *str, int *val, int min, int max);
 int tst_parse_long(const char *str, long *val, long min, long max);
 int tst_parse_float(const char *str, float *val, float min, float max);
 
+struct tst_tag {
+	const char *name;
+	const char *value;
+};
+
 extern unsigned int tst_variant;
 
 struct tst_test {
@@ -212,6 +217,11 @@ struct tst_test {
 	 * NULL-terminated array of capability settings
 	 */
 	struct tst_cap *caps;
+
+	/*
+	 * {NULL, NULL} terminated array of tags.
+	 */
+	const struct tst_tag *tags;
 };
 
 /*
diff --git a/lib/tst_test.c b/lib/tst_test.c
index 6239acf89..2129f38cb 100644
--- a/lib/tst_test.c
+++ b/lib/tst_test.c
@@ -31,6 +31,9 @@
 #include "old_device.h"
 #include "old_tmpdir.h"
 
+#define LINUX_GIT_URL "https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id="
+#define CVE_DB_URL "https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-"
+
 struct tst_test *tst_test;
 
 static const char *tid;
@@ -414,6 +417,9 @@ static void print_help(void)
 {
 	unsigned int i;
 
+	fprintf(stderr, "Options\n");
+	fprintf(stderr, "-------\n\n");
+
 	for (i = 0; i < ARRAY_SIZE(options); i++)
 		fprintf(stderr, "%s\n", options[i].help);
 
@@ -424,6 +430,27 @@ static void print_help(void)
 		fprintf(stderr, "%s\n", tst_test->options[i].help);
 }
 
+static void print_test_info(void)
+{
+	unsigned int i;
+	const struct tst_tag *tags = tst_test->tags;
+
+	printf("\nTags\n");
+	printf("----\n\n");
+
+	if (tags) {
+		for (i = 0; tags[i].name; i++) {
+			if (!strcmp(tags[i].name, "CVE"))
+				printf(CVE_DB_URL "%s\n", tags[i].value);
+			else if (!strcmp(tags[i].name, "linux-git"))
+				printf(LINUX_GIT_URL "%s\n", tags[i].value);
+			else
+				printf("%s: %s\n", tags[i].name, tags[i].value);
+			printf("\n");
+		}
+	}
+}
+
 static void check_option_collision(void)
 {
 	unsigned int i, j;
@@ -499,6 +526,7 @@ static void parse_opts(int argc, char *argv[])
 		break;
 		case 'h':
 			print_help();
+			print_test_info();
 			exit(0);
 		case 'i':
 			iterations = atoi(optarg);
@@ -584,26 +612,61 @@ int tst_parse_float(const char *str, float *val, float min, float max)
 	return 0;
 }
 
+static void print_colored(const char *str)
+{
+	if (tst_color_enabled(STDOUT_FILENO))
+		printf("%s%s%s", ANSI_COLOR_YELLOW, str, ANSI_COLOR_RESET);
+	else
+		printf("%s", str);
+}
+
+static void print_failure_hints(void)
+{
+	unsigned int i;
+	const struct tst_tag *tags = tst_test->tags;
+
+	if (!tags)
+		return;
+
+	for (i = 0; tags[i].name; i++) {
+		if (!strcmp(tags[i].name, "linux-git")) {
+			printf("\n");
+			print_colored("HINT: ");
+			printf("This is a regression test for linux kernel, see commit:\n\n"
+			       LINUX_GIT_URL "%s\n", tags[i].value);
+		}
+
+		if (!strcmp(tags[i].name, "CVE")) {
+			printf("\n");
+			print_colored("HINT: ");
+			printf("This test also tests for CVE-%s, see:\n\n"
+			       CVE_DB_URL "%s\n", tags[i].value, tags[i].value);
+		}
+	}
+}
+
 static void do_exit(int ret)
 {
 	if (results) {
-		printf("\nSummary:\n");
-		printf("passed   %d\n", results->passed);
-		printf("failed   %d\n", results->failed);
-		printf("skipped  %d\n", results->skipped);
-		printf("warnings %d\n", results->warnings);
-
 		if (results->passed && ret == TCONF)
 			ret = 0;
 
-		if (results->failed)
+		if (results->failed) {
 			ret |= TFAIL;
+			print_failure_hints();
+		}
 
 		if (results->skipped && !results->passed)
 			ret |= TCONF;
 
 		if (results->warnings)
 			ret |= TWARN;
+
+		printf("\nSummary:\n");
+		printf("passed   %d\n", results->passed);
+		printf("failed   %d\n", results->failed);
+		printf("skipped  %d\n", results->skipped);
+		printf("warnings %d\n", results->warnings);
 	}
 
 	do_cleanup();
-- 
2.21.0


^ permalink raw reply related	[flat|nested] 17+ messages in thread

* [LTP] [PATCH 2/2] testcases/kernel: Add linux git and CVE tags
  2019-10-14 11:25 [LTP] [PATCH 0/2] Add support for a test tags Cyril Hrubis
  2019-10-14 11:25 ` [LTP] [PATCH 1/2] lib: Add support for " Cyril Hrubis
@ 2019-10-14 11:25 ` Cyril Hrubis
  2019-10-15  4:54   ` Li Wang
  1 sibling, 1 reply; 17+ messages in thread
From: Cyril Hrubis @ 2019-10-14 11:25 UTC (permalink / raw)
  To: ltp

Not all tests are annotated at the moment but the majority has been
finished for now.

Signed-off-by: Cyril Hrubis <chrubis@suse.cz>
---
 testcases/cve/cve-2014-0196.c                         | 5 +++++
 testcases/cve/cve-2015-3290.c                         | 5 +++++
 testcases/cve/cve-2016-10044.c                        | 5 +++++
 testcases/cve/cve-2016-7042.c                         | 5 +++++
 testcases/cve/cve-2016-7117.c                         | 5 +++++
 testcases/cve/cve-2017-16939.c                        | 5 +++++
 testcases/cve/cve-2017-17052.c                        | 5 +++++
 testcases/cve/cve-2017-17053.c                        | 5 +++++
 testcases/cve/cve-2017-2618.c                         | 5 +++++
 testcases/cve/cve-2017-2671.c                         | 5 +++++
 testcases/cve/meltdown.c                              | 6 +++++-
 testcases/cve/stack_clash.c                           | 4 ++++
 testcases/kernel/crypto/af_alg01.c                    | 5 +++++
 testcases/kernel/crypto/af_alg02.c                    | 5 +++++
 testcases/kernel/crypto/af_alg03.c                    | 4 ++++
 testcases/kernel/crypto/af_alg04.c                    | 4 ++++
 testcases/kernel/crypto/af_alg05.c                    | 8 +++++++-
 testcases/kernel/crypto/af_alg06.c                    | 4 ++++
 testcases/kernel/crypto/crypto_user01.c               | 6 ++++++
 testcases/kernel/crypto/crypto_user02.c               | 4 ++++
 testcases/kernel/crypto/pcrypt_aead01.c               | 5 +++++
 testcases/kernel/sound/snd_timer01.c                  | 6 ++++++
 testcases/kernel/syscalls/accept/accept02.c           | 5 +++++
 testcases/kernel/syscalls/acct/acct02.c               | 4 ++++
 testcases/kernel/syscalls/add_key/add_key02.c         | 5 +++++
 testcases/kernel/syscalls/bpf/bpf_prog02.c            | 4 ++++
 testcases/kernel/syscalls/futex/futex_cmp_requeue02.c | 5 +++++
 testcases/kernel/syscalls/keyctl/keyctl02.c           | 4 ++++
 testcases/kernel/syscalls/keyctl/keyctl04.c           | 5 +++++
 testcases/kernel/syscalls/keyctl/keyctl05.c           | 4 ++++
 testcases/kernel/syscalls/move_pages/move_pages12.c   | 6 ++++++
 testcases/kernel/syscalls/realpath/realpath01.c       | 4 ++++
 testcases/kernel/syscalls/request_key/request_key05.c | 4 ++++
 testcases/kernel/syscalls/sbrk/sbrk03.c               | 4 ++++
 testcases/kernel/syscalls/sendto/sendto02.c           | 4 ++++
 testcases/kernel/syscalls/setrlimit/setrlimit06.c     | 4 ++++
 36 files changed, 171 insertions(+), 2 deletions(-)

diff --git a/testcases/cve/cve-2014-0196.c b/testcases/cve/cve-2014-0196.c
index 6997fffe4..012cbb7cd 100644
--- a/testcases/cve/cve-2014-0196.c
+++ b/testcases/cve/cve-2014-0196.c
@@ -141,4 +141,9 @@ static struct tst_test test = {
 	.setup = setup,
 	.cleanup = cleanup,
 	.test_all = run,
+	.tags = (const struct tst_tag[]) {
+		{"linux-git", "4291086b1f08"},
+		{"CVE", "2014-0196"},
+		{}
+	}
 };
diff --git a/testcases/cve/cve-2015-3290.c b/testcases/cve/cve-2015-3290.c
index fe630e4af..6c4fd57ac 100644
--- a/testcases/cve/cve-2015-3290.c
+++ b/testcases/cve/cve-2015-3290.c
@@ -469,6 +469,11 @@ static struct tst_test test = {
 	.setup = setup,
 	.timeout = TIMEOUT,
 	.test_all = run,
+	.tags = (const struct tst_tag[]) {
+		{"linux-git", "9b6e6a8334d5"},
+		{"CVE", "2015-3290"},
+		{}
+	}
 };
 
 #else /* HAVE_PERF_EVENT_ATTR && (defined(__x86_64__) || defined(__i386__)) */
diff --git a/testcases/cve/cve-2016-10044.c b/testcases/cve/cve-2016-10044.c
index c2e79df31..7519676a1 100644
--- a/testcases/cve/cve-2016-10044.c
+++ b/testcases/cve/cve-2016-10044.c
@@ -62,4 +62,9 @@ static struct tst_test test = {
 	.test_all = run,
 	.cleanup = cleanup,
 	.min_kver = "2.6.8",
+	.tags = (const struct tst_tag[]) {
+		{"linux-git", "22f6b4d34fcf"},
+		{"CVE", "2016-10044"},
+		{}
+	}
 };
diff --git a/testcases/cve/cve-2016-7042.c b/testcases/cve/cve-2016-7042.c
index 35950adc7..4434265dd 100644
--- a/testcases/cve/cve-2016-7042.c
+++ b/testcases/cve/cve-2016-7042.c
@@ -71,4 +71,9 @@ static struct tst_test test = {
 	.setup = setup,
 	.cleanup = cleanup,
 	.test_all = do_test,
+	.tags = (const struct tst_tag[]) {
+		{"linux-git", "03dab869b7b2"},
+		{"CVE", "2016-7042"},
+		{}
+	}
 };
diff --git a/testcases/cve/cve-2016-7117.c b/testcases/cve/cve-2016-7117.c
index 5d9f842aa..dca002924 100644
--- a/testcases/cve/cve-2016-7117.c
+++ b/testcases/cve/cve-2016-7117.c
@@ -150,4 +150,9 @@ static struct tst_test test = {
 	.setup = setup,
 	.cleanup = cleanup,
 	.min_kver = "2.6.33",
+	.tags = (const struct tst_tag[]) {
+		{"linux-git", "a2e2725541fa"},
+		{"CVE", "2016-7117"},
+		{}
+	}
 };
diff --git a/testcases/cve/cve-2017-16939.c b/testcases/cve/cve-2017-16939.c
index 4ecc86337..9ab1eb2c3 100644
--- a/testcases/cve/cve-2017-16939.c
+++ b/testcases/cve/cve-2017-16939.c
@@ -77,4 +77,9 @@ static void run(void)
 static struct tst_test test = {
 	.setup = setup,
 	.test_all = run,
+	.tags = (const struct tst_tag[]) {
+		{"linux-git", "1137b5e2529a"},
+		{"CVE", "2017-16939"},
+		{}
+	}
 };
diff --git a/testcases/cve/cve-2017-17052.c b/testcases/cve/cve-2017-17052.c
index 81123fce8..d89c62eb6 100644
--- a/testcases/cve/cve-2017-17052.c
+++ b/testcases/cve/cve-2017-17052.c
@@ -111,4 +111,9 @@ static struct tst_test test = {
 	.cleanup = cleanup,
 	.setup = setup,
 	.test_all = run,
+	.tags = (const struct tst_tag[]) {
+		{"linux-git", "2b7e8665b4ff"},
+		{"CVE", "2017-17052"},
+		{}
+	}
 };
diff --git a/testcases/cve/cve-2017-17053.c b/testcases/cve/cve-2017-17053.c
index 646ec11cc..08a08211d 100644
--- a/testcases/cve/cve-2017-17053.c
+++ b/testcases/cve/cve-2017-17053.c
@@ -154,6 +154,11 @@ static struct tst_test test = {
 	.setup = setup,
 	.cleanup = cleanup,
 	.test_all = run,
+	.tags = (const struct tst_tag[]) {
+		{"linux-git", "ccd5b3235180"},
+		{"CVE", "2017-17053"},
+		{}
+	}
 };
 
 #else
diff --git a/testcases/cve/cve-2017-2618.c b/testcases/cve/cve-2017-2618.c
index cf018d401..4ab3cfb8f 100644
--- a/testcases/cve/cve-2017-2618.c
+++ b/testcases/cve/cve-2017-2618.c
@@ -51,4 +51,9 @@ static struct tst_test test = {
 	.forks_child = 1,
 	.setup = setup,
 	.test_all = do_test,
+	.tags = (const struct tst_tag[]) {
+		{"linux-git", "0c461cb727d1"},
+		{"CVE", "2017-2618"},
+		{}
+	}
 };
diff --git a/testcases/cve/cve-2017-2671.c b/testcases/cve/cve-2017-2671.c
index b92c1d94b..e72795d15 100644
--- a/testcases/cve/cve-2017-2671.c
+++ b/testcases/cve/cve-2017-2671.c
@@ -109,4 +109,9 @@ static struct tst_test test = {
 	.test_all = run,
 	.cleanup = cleanup,
 	.needs_root = 1,
+	.tags = (const struct tst_tag[]) {
+		{"linux-git", "43a6684519ab"},
+		{"CVE", "2017-2671"},
+		{}
+	}
 };
diff --git a/testcases/cve/meltdown.c b/testcases/cve/meltdown.c
index 863f1a9e4..db00e1811 100644
--- a/testcases/cve/meltdown.c
+++ b/testcases/cve/meltdown.c
@@ -377,7 +377,11 @@ static struct tst_test test = {
 	.setup = setup,
 	.test_all = run,
 	.cleanup = cleanup,
-	.min_kver = "2.6.32"
+	.min_kver = "2.6.32",
+	.tags = (const struct tst_tag[]) {
+		{"CVE", "2017-5754"},
+		{}
+	}
 };
 
 #else /* #if defined(__x86_64__) || defined(__i386__) */
diff --git a/testcases/cve/stack_clash.c b/testcases/cve/stack_clash.c
index 3ea3ec491..ecbf5f6da 100644
--- a/testcases/cve/stack_clash.c
+++ b/testcases/cve/stack_clash.c
@@ -250,4 +250,8 @@ static struct tst_test test = {
 	.needs_root = 1,
 	.setup = setup,
 	.test_all = stack_clash_test,
+	.tags = (const struct tst_tag[]) {
+		{"CVE", "2017-1000364"},
+		{}
+	}
 };
diff --git a/testcases/kernel/crypto/af_alg01.c b/testcases/kernel/crypto/af_alg01.c
index bb6e01738..d3e823e6c 100644
--- a/testcases/kernel/crypto/af_alg01.c
+++ b/testcases/kernel/crypto/af_alg01.c
@@ -76,4 +76,9 @@ static void do_test(unsigned int i)
 static struct tst_test test = {
 	.test = do_test,
 	.tcnt = ARRAY_SIZE(hash_algs),
+	.tags = (const struct tst_tag[]) {
+		{"linux-git", "af3ff8045bbf"},
+		{"CVE", "2017-17806"},
+		{}
+	}
 };
diff --git a/testcases/kernel/crypto/af_alg02.c b/testcases/kernel/crypto/af_alg02.c
index 08087ecac..fab0010c9 100644
--- a/testcases/kernel/crypto/af_alg02.c
+++ b/testcases/kernel/crypto/af_alg02.c
@@ -59,4 +59,9 @@ static struct tst_test test = {
 	.test_all = run,
 	.timeout = 20,
 	.needs_checkpoints = 1,
+	.tags = (const struct tst_tag[]) {
+		{"linux-git", "ecaaab564978"},
+		{"CVE", "2017-17805"},
+		{}
+	}
 };
diff --git a/testcases/kernel/crypto/af_alg03.c b/testcases/kernel/crypto/af_alg03.c
index 240c52835..5f214e48b 100644
--- a/testcases/kernel/crypto/af_alg03.c
+++ b/testcases/kernel/crypto/af_alg03.c
@@ -28,4 +28,8 @@ static void run(void)
 
 static struct tst_test test = {
 	.test_all = run,
+	.tags = (const struct tst_tag[]) {
+		{"linux-git", "e57121d08c38"},
+		{}
+	}
 };
diff --git a/testcases/kernel/crypto/af_alg04.c b/testcases/kernel/crypto/af_alg04.c
index 48239e020..df049048a 100644
--- a/testcases/kernel/crypto/af_alg04.c
+++ b/testcases/kernel/crypto/af_alg04.c
@@ -55,4 +55,8 @@ static void run(void)
 static struct tst_test test = {
 	.test_all = run,
 	.forks_child = 1,
+	.tags = (const struct tst_tag[]) {
+		{"linux-git", "bb2964810233"},
+		{}
+	}
 };
diff --git a/testcases/kernel/crypto/af_alg05.c b/testcases/kernel/crypto/af_alg05.c
index 964adffcc..df2b03546 100644
--- a/testcases/kernel/crypto/af_alg05.c
+++ b/testcases/kernel/crypto/af_alg05.c
@@ -11,7 +11,7 @@
  * boundary.  Based on the reproducer from the commit message.  Note that this
  * issue only reproduces on certain architectures, such as arm and arm64.
  *
- * On some older kernel without commit 160544075f2 ("crypto: scatterwalk - Hide
+ * On some older kernel without commit 160544075f2a ("crypto: scatterwalk - Hide
  * PageSlab call to optimise away flush_dcache_page") , it doesn't use
  * ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE macro. It can crash on all architectures.
  * Without skcipher walk interface, it is also a regresstion test for commit
@@ -46,4 +46,10 @@ static void run(void)
 
 static struct tst_test test = {
 	.test_all = run,
+	.tags = (const struct tst_tag[]) {
+		{"linux-git", "8088d3dd4d7c"},
+		{"linux-git", "160544075f2a"},
+		{"linux-git", "0868def3e410"},
+		{}
+	}
 };
diff --git a/testcases/kernel/crypto/af_alg06.c b/testcases/kernel/crypto/af_alg06.c
index 586d91b1d..6acdf8532 100644
--- a/testcases/kernel/crypto/af_alg06.c
+++ b/testcases/kernel/crypto/af_alg06.c
@@ -48,4 +48,8 @@ static void run(void)
 
 static struct tst_test test = {
 	.test_all = run,
+	.tags = (const struct tst_tag[]) {
+		{"linux-git", "8f9c46934848"},
+		{}
+	}
 };
diff --git a/testcases/kernel/crypto/crypto_user01.c b/testcases/kernel/crypto/crypto_user01.c
index 163d7a2dd..47bf9f0d2 100644
--- a/testcases/kernel/crypto/crypto_user01.c
+++ b/testcases/kernel/crypto/crypto_user01.c
@@ -192,4 +192,10 @@ static struct tst_test test = {
 	.setup = setup,
 	.test_all = run,
 	.cleanup = cleanup,
+	.tags = (const struct tst_tag[]) {
+		{"linux-git", "f43f39958beb"},
+		{"CVE", "2013-2547"},
+		{"CVE", "2018-19854"},
+		{}
+	}
 };
diff --git a/testcases/kernel/crypto/crypto_user02.c b/testcases/kernel/crypto/crypto_user02.c
index 6090b5a4e..384e344f2 100644
--- a/testcases/kernel/crypto/crypto_user02.c
+++ b/testcases/kernel/crypto/crypto_user02.c
@@ -105,4 +105,8 @@ static struct tst_test test = {
 	.cleanup = cleanup,
 	.needs_root = 1,
 	.forks_child = 1,
+	.tags = (const struct tst_tag[]) {
+		{"linux-git", "21d4120ec6f5"},
+		{}
+	}
 };
diff --git a/testcases/kernel/crypto/pcrypt_aead01.c b/testcases/kernel/crypto/pcrypt_aead01.c
index d9c27f69d..42ae259fb 100644
--- a/testcases/kernel/crypto/pcrypt_aead01.c
+++ b/testcases/kernel/crypto/pcrypt_aead01.c
@@ -69,4 +69,9 @@ static struct tst_test test = {
 	.test_all = run,
 	.cleanup = cleanup,
 	.needs_root = 1,
+	.tags = (const struct tst_tag[]) {
+		{"linux-git", "d76c68109f37"},
+		{"CVE", "2017-5754"},
+		{}
+	}
 };
diff --git a/testcases/kernel/sound/snd_timer01.c b/testcases/kernel/sound/snd_timer01.c
index f62696659..ebac6e1c6 100644
--- a/testcases/kernel/sound/snd_timer01.c
+++ b/testcases/kernel/sound/snd_timer01.c
@@ -140,4 +140,10 @@ static struct tst_test test = {
 	.test_all = run,
 	.setup = setup,
 	.cleanup = cleanup,
+	.tags = (const struct tst_tag[]) {
+		{"linux-git", "d11662f4f798"},
+		{"linux-git", "ba3021b2c79b"},
+		{"CVE", "2017-1000380"},
+		{NULL, NULL}
+	}
 };
diff --git a/testcases/kernel/syscalls/accept/accept02.c b/testcases/kernel/syscalls/accept/accept02.c
index 1a0f625c9..576b91ce8 100644
--- a/testcases/kernel/syscalls/accept/accept02.c
+++ b/testcases/kernel/syscalls/accept/accept02.c
@@ -142,4 +142,9 @@ static struct tst_test test = {
 	.setup = setup,
 	.cleanup = cleanup,
 	.needs_checkpoints = 1,
+	.tags = (const struct tst_tag[]) {
+		{"CVE", "2017-8890"},
+		{"linux-git", "657831ff"},
+		{NULL, NULL},
+	}
 };
diff --git a/testcases/kernel/syscalls/acct/acct02.c b/testcases/kernel/syscalls/acct/acct02.c
index 4d95aafec..890568b08 100644
--- a/testcases/kernel/syscalls/acct/acct02.c
+++ b/testcases/kernel/syscalls/acct/acct02.c
@@ -263,4 +263,8 @@ static struct tst_test test = {
 	.cleanup = cleanup,
 	.needs_tmpdir = 1,
 	.needs_root = 1,
+	.tags = (const struct tst_tag[]) {
+		{"linux-git", "4d9570158b626"},
+		{}
+	}
 };
diff --git a/testcases/kernel/syscalls/add_key/add_key02.c b/testcases/kernel/syscalls/add_key/add_key02.c
index 4a5cf054a..159234f67 100644
--- a/testcases/kernel/syscalls/add_key/add_key02.c
+++ b/testcases/kernel/syscalls/add_key/add_key02.c
@@ -83,4 +83,9 @@ static void verify_add_key(unsigned int i)
 static struct tst_test test = {
 	.tcnt = ARRAY_SIZE(tcases),
 	.test = verify_add_key,
+	.tags = (const struct tst_tag[]) {
+		{"linux-git", "5649645d725c"},
+		{"CVE", "2017-15274"},
+		{}
+	}
 };
diff --git a/testcases/kernel/syscalls/bpf/bpf_prog02.c b/testcases/kernel/syscalls/bpf/bpf_prog02.c
index fb8f3f781..47f313a6d 100644
--- a/testcases/kernel/syscalls/bpf/bpf_prog02.c
+++ b/testcases/kernel/syscalls/bpf/bpf_prog02.c
@@ -190,5 +190,9 @@ static struct tst_test test = {
 		{&attr, .size = sizeof(*attr)},
 		{&msg, .size = sizeof(MSG)},
 		{},
+	},
+	.tags = (const struct tst_tag[]) {
+		{"linux-git", "3612af783cf5"},
+		{}
 	}
 };
diff --git a/testcases/kernel/syscalls/futex/futex_cmp_requeue02.c b/testcases/kernel/syscalls/futex/futex_cmp_requeue02.c
index 3ea19428c..228d8cff6 100644
--- a/testcases/kernel/syscalls/futex/futex_cmp_requeue02.c
+++ b/testcases/kernel/syscalls/futex/futex_cmp_requeue02.c
@@ -73,4 +73,9 @@ static struct tst_test test = {
 	.cleanup = cleanup,
 	.test = verify_futex_cmp_requeue,
 	.tcnt = ARRAY_SIZE(tcases),
+	.tags = (const struct tst_tag[]) {
+		{"CVE", "2018-6927"},
+		{"linux-git", "fbe0e839d1e2"},
+		{}
+	}
 };
diff --git a/testcases/kernel/syscalls/keyctl/keyctl02.c b/testcases/kernel/syscalls/keyctl/keyctl02.c
index b034275d2..f4ee03f33 100644
--- a/testcases/kernel/syscalls/keyctl/keyctl02.c
+++ b/testcases/kernel/syscalls/keyctl/keyctl02.c
@@ -136,4 +136,8 @@ static struct tst_test test = {
 	.setup = setup,
 	.cleanup = cleanup,
 	.test_all = do_test,
+	.tags = (const struct tst_tag[]) {
+		{"linux-git", "b4a1b4f5047e"},
+		{}
+	}
 };
diff --git a/testcases/kernel/syscalls/keyctl/keyctl04.c b/testcases/kernel/syscalls/keyctl/keyctl04.c
index d30d25b04..1fed23ca6 100644
--- a/testcases/kernel/syscalls/keyctl/keyctl04.c
+++ b/testcases/kernel/syscalls/keyctl/keyctl04.c
@@ -43,4 +43,9 @@ static void do_test(void)
 
 static struct tst_test test = {
 	.test_all = do_test,
+	.tags = (const struct tst_tag[]) {
+		{"CVE", "2017-7472"},
+		{"linux-git", "c9f838d104fe"},
+		{}
+	}
 };
diff --git a/testcases/kernel/syscalls/keyctl/keyctl05.c b/testcases/kernel/syscalls/keyctl/keyctl05.c
index 653b838f5..c592eb49e 100644
--- a/testcases/kernel/syscalls/keyctl/keyctl05.c
+++ b/testcases/kernel/syscalls/keyctl/keyctl05.c
@@ -207,4 +207,8 @@ static struct tst_test test = {
 	.tcnt = 3,
 	.test = do_test,
 	.forks_child = 1,
+	.tags = (const struct tst_tag[]) {
+		{"linux-git", "63a0b0509e70"},
+		{}
+	}
 };
diff --git a/testcases/kernel/syscalls/move_pages/move_pages12.c b/testcases/kernel/syscalls/move_pages/move_pages12.c
index c2927fb86..c906acbe3 100644
--- a/testcases/kernel/syscalls/move_pages/move_pages12.c
+++ b/testcases/kernel/syscalls/move_pages/move_pages12.c
@@ -330,6 +330,12 @@ static struct tst_test test = {
 	.cleanup = cleanup,
 	.test = do_test,
 	.tcnt = ARRAY_SIZE(tcases),
+	.tags = (const struct tst_tag[]) {
+		{"linux-git", "e66f17ff7177"},
+		{"linux-git", "c9d398fa2378"},
+		{"linux-git", "4643d67e8cb0"},
+		{}
+	}
 };
 
 #else
diff --git a/testcases/kernel/syscalls/realpath/realpath01.c b/testcases/kernel/syscalls/realpath/realpath01.c
index fc2af2a1c..c0381e9cb 100644
--- a/testcases/kernel/syscalls/realpath/realpath01.c
+++ b/testcases/kernel/syscalls/realpath/realpath01.c
@@ -41,4 +41,8 @@ static struct tst_test test = {
 	.setup = setup,
 	.needs_root = 1,
 	.needs_tmpdir = 1,
+	.tags = (const struct tst_tag[]) {
+		{"CVE", "2018-1000001"},
+		{}
+	}
 };
diff --git a/testcases/kernel/syscalls/request_key/request_key05.c b/testcases/kernel/syscalls/request_key/request_key05.c
index 129cc0507..a17d1d058 100644
--- a/testcases/kernel/syscalls/request_key/request_key05.c
+++ b/testcases/kernel/syscalls/request_key/request_key05.c
@@ -31,4 +31,8 @@ static void run(void)
 
 static struct tst_test test = {
 	.test_all = run,
+	.tags = (const struct tst_tag[]) {
+		{"CVE", "2017-6951"},
+		{}
+	}
 };
diff --git a/testcases/kernel/syscalls/sbrk/sbrk03.c b/testcases/kernel/syscalls/sbrk/sbrk03.c
index 761a01c9d..80d2020ff 100644
--- a/testcases/kernel/syscalls/sbrk/sbrk03.c
+++ b/testcases/kernel/syscalls/sbrk/sbrk03.c
@@ -66,4 +66,8 @@ static void sbrk_test(void)
 
 static struct tst_test test = {
 	.test_all = sbrk_test,
+	.tags = (const struct tst_tag[]) {
+		{"linux-git", "473a06572fcd"},
+		{}
+	}
 };
diff --git a/testcases/kernel/syscalls/sendto/sendto02.c b/testcases/kernel/syscalls/sendto/sendto02.c
index 5c9f00125..743101733 100644
--- a/testcases/kernel/syscalls/sendto/sendto02.c
+++ b/testcases/kernel/syscalls/sendto/sendto02.c
@@ -77,4 +77,8 @@ static struct tst_test test = {
 	.setup = setup,
 	.cleanup = cleanup,
 	.test_all = verify_sendto,
+	.tags = (const struct tst_tag[]) {
+		{"linux-git", "6e51fe757259"},
+		{}
+	}
 };
diff --git a/testcases/kernel/syscalls/setrlimit/setrlimit06.c b/testcases/kernel/syscalls/setrlimit/setrlimit06.c
index 87e6b0e32..726b26841 100644
--- a/testcases/kernel/syscalls/setrlimit/setrlimit06.c
+++ b/testcases/kernel/syscalls/setrlimit/setrlimit06.c
@@ -114,4 +114,8 @@ static struct tst_test test = {
 	.setup = setup,
 	.cleanup = cleanup,
 	.forks_child = 1,
+	.tags = (const struct tst_tag[]) {
+		{"linux-git", "c3bca5d450b62"},
+		{}
+	}
 };
-- 
2.21.0


^ permalink raw reply related	[flat|nested] 17+ messages in thread

* [LTP] [PATCH 2/2] testcases/kernel: Add linux git and CVE tags
  2019-10-14 11:25 ` [LTP] [PATCH 2/2] testcases/kernel: Add linux git and CVE tags Cyril Hrubis
@ 2019-10-15  4:54   ` Li Wang
  2019-10-15  9:52     ` Cyril Hrubis
  0 siblings, 1 reply; 17+ messages in thread
From: Li Wang @ 2019-10-15  4:54 UTC (permalink / raw)
  To: ltp

Cyril Hrubis <chrubis@suse.cz> wrote:


>
> diff --git a/testcases/cve/cve-2014-0196.c b/testcases/cve/cve-2014-0196.c
> index 6997fffe4..012cbb7cd 100644
> --- a/testcases/cve/cve-2014-0196.c
> +++ b/testcases/cve/cve-2014-0196.c
> @@ -141,4 +141,9 @@ static struct tst_test test = {
>         .setup = setup,
>         .cleanup = cleanup,
>         .test_all = run,
> +       .tags = (const struct tst_tag[]) {
> +               {"linux-git", "4291086b1f08"},
> +               {"CVE", "2014-0196"},
> +               {}
>

Some of the tests use {NULL, NULL} to terminate tag array but some not,
maybe we need to keep consistent in usage.

-- 
Regards,
Li Wang
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linux.it/pipermail/ltp/attachments/20191015/bc68b025/attachment.htm>

^ permalink raw reply	[flat|nested] 17+ messages in thread

* [LTP] [PATCH 1/2] lib: Add support for test tags
  2019-10-14 11:25 ` [LTP] [PATCH 1/2] lib: Add support for " Cyril Hrubis
@ 2019-10-15  5:12   ` Li Wang
  2019-10-15  7:42     ` Richard Palethorpe
  2019-10-15  9:54     ` Cyril Hrubis
  2019-10-16 13:30   ` Petr Vorel
  1 sibling, 2 replies; 17+ messages in thread
From: Li Wang @ 2019-10-15  5:12 UTC (permalink / raw)
  To: ltp

On Mon, Oct 14, 2019 at 7:25 PM Cyril Hrubis <chrubis@suse.cz> wrote:

> The newly introduced test tags are generic name-value pairs that can
> hold test metadata, the intended use for now is to store kernel commit
> hashes for kernel reproducers as well as CVE ids. The mechanism is
> however choosen to be very generic so that it's easy to add basically
> any information later on.
>
> As it is the main purpose is to print hints for a test failures. If a
> test that has been written as a kernel reproducer fails it prints nice
> URL pointing to a kernel commit that may be missing.
>

Nice to see this feature patchset achieved, it looks quite clear&good to
me. Just have some tiny comments below.

>
> Example output:
>
> --------------------------------------------------------------------------
> tst_test.c:1145: INFO: Timeout per run is 0h 05m 00s
> add_key02.c:98: FAIL: unexpected error with key type 'asymmetric': EINVAL
> add_key02.c:98: FAIL: unexpected error with key type 'cifs.idmap': EINVAL
> add_key02.c:98: FAIL: unexpected error with key type 'cifs.spnego': EINVAL
> add_key02.c:98: FAIL: unexpected error with key type 'pkcs7_test': EINVAL
> add_key02.c:98: FAIL: unexpected error with key type 'rxrpc': EINVAL
> add_key02.c:98: FAIL: unexpected error with key type 'rxrpc_s': EINVAL
> add_key02.c:98: FAIL: unexpected error with key type 'user': EINVAL
> add_key02.c:98: FAIL: unexpected error with key type 'logon': EINVAL
>
> HINT: This is a regression test for linux kernel, see commit:
>
>
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=5649645d725c
>
> HINT: This test also tests for CVE-2017-15274, see:
>
> https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017-15274
>
> Summary:
> passed   0
> failed   8
> skipped  0
> warnings 0
> --------------------------------------------------------------------------
>
> Signed-off-by: Cyril Hrubis <chrubis@suse.cz>
> ---
>  include/tst_test.h | 10 ++++++
>  lib/tst_test.c     | 77 +++++++++++++++++++++++++++++++++++++++++-----
>  2 files changed, 80 insertions(+), 7 deletions(-)
>
> diff --git a/include/tst_test.h b/include/tst_test.h
> index 84acf2c59..4a51b6d16 100644
> --- a/include/tst_test.h
> +++ b/include/tst_test.h
> @@ -108,6 +108,11 @@ int tst_parse_int(const char *str, int *val, int min,
> int max);
>  int tst_parse_long(const char *str, long *val, long min, long max);
>  int tst_parse_float(const char *str, float *val, float min, float max);
>
> +struct tst_tag {
> +       const char *name;
> +       const char *value;
> +};
> +
>  extern unsigned int tst_variant;
>
>  struct tst_test {
> @@ -212,6 +217,11 @@ struct tst_test {
>          * NULL-terminated array of capability settings
>          */
>         struct tst_cap *caps;
> +
> +       /*
> +        * {NULL, NULL} terminated array of tags.
> +        */
> +       const struct tst_tag *tags;
>  };
>
>  /*
> diff --git a/lib/tst_test.c b/lib/tst_test.c
> index 6239acf89..2129f38cb 100644
> --- a/lib/tst_test.c
> +++ b/lib/tst_test.c
> @@ -31,6 +31,9 @@
>  #include "old_device.h"
>  #include "old_tmpdir.h"
>
> +#define LINUX_GIT_URL "
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=
> "
> +#define CVE_DB_URL "https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-"
> +
>  struct tst_test *tst_test;
>
>  static const char *tid;
> @@ -414,6 +417,9 @@ static void print_help(void)
>  {
>         unsigned int i;
>
> +       fprintf(stderr, "Options\n");
> +       fprintf(stderr, "-------\n\n");
> +
>         for (i = 0; i < ARRAY_SIZE(options); i++)
>                 fprintf(stderr, "%s\n", options[i].help);
>
> @@ -424,6 +430,27 @@ static void print_help(void)
>                 fprintf(stderr, "%s\n", tst_test->options[i].help);
>  }
>
> +static void print_test_info(void)
>

print_test_info sounds like general information for the test, maybe
print_tags() is a better/precise name?

> +{
> +       unsigned int i;
> +       const struct tst_tag *tags = tst_test->tags;
> +
> +       printf("\nTags\n");
> +       printf("----\n\n");
> +
> +       if (tags) {
> +               for (i = 0; tags[i].name; i++) {
> +                       if (!strcmp(tags[i].name, "CVE"))
> +                               printf(CVE_DB_URL "%s\n", tags[i].value);
> +                       else if (!strcmp(tags[i].name, "linux-git"))
> +                               printf(LINUX_GIT_URL "%s\n",
> tags[i].value);
> +                       else
> +                               printf("%s: %s\n", tags[i].name,
> tags[i].value);
> +                       printf("\n");
> +               }
> +       }
> +}
> +
>  static void check_option_collision(void)
>  {
>         unsigned int i, j;
> @@ -499,6 +526,7 @@ static void parse_opts(int argc, char *argv[])
>                 break;
>                 case 'h':
>                         print_help();
> +                       print_test_info();
>                         exit(0);
>                 case 'i':
>                         iterations = atoi(optarg);
> @@ -584,26 +612,61 @@ int tst_parse_float(const char *str, float *val,
> float min, float max)
>         return 0;
>  }
>
> +static void print_colored(const char *str)
> +{
> +       if (tst_color_enabled(STDOUT_FILENO))
> +               printf("%s%s%s", ANSI_COLOR_YELLOW, str, ANSI_COLOR_RESET);
> +       else
> +               printf("%s", str);
> +}
> +
> +static void print_failure_hints(void)
> +{
> +       unsigned int i;
> +       const struct tst_tag *tags = tst_test->tags;
> +
> +       if (!tags)
> +               return;
> +
> +       for (i = 0; tags[i].name; i++) {
> +               if (!strcmp(tags[i].name, "linux-git")) {
> +                       printf("\n");
> +                       print_colored("HINT: ");

+                       printf("This is a regression test for linux kernel,
> see commit:\n\n"
> +                              LINUX_GIT_URL "%s\n", tags[i].value);
>

This sentence 'HINT: This is a ...' will be printed many times if there are
many commits in tags, I prefer to see only once in front of these
linux-kernel links.


> +               }
> +
> +               if (!strcmp(tags[i].name, "CVE")) {
> +                       printf("\n");
> +                       print_colored("HINT: ");
> +                       printf("This test also tests for CVE-%s, see:\n\n"
> +                              CVE_DB_URL "%s\n", tags[i].value,
> tags[i].value);
>

Here as well.

-- 
Regards,
Li Wang
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linux.it/pipermail/ltp/attachments/20191015/df8ab998/attachment.htm>

^ permalink raw reply	[flat|nested] 17+ messages in thread

* [LTP] [PATCH 1/2] lib: Add support for test tags
  2019-10-15  5:12   ` Li Wang
@ 2019-10-15  7:42     ` Richard Palethorpe
  2019-10-15  9:57       ` Cyril Hrubis
  2019-10-15  9:54     ` Cyril Hrubis
  1 sibling, 1 reply; 17+ messages in thread
From: Richard Palethorpe @ 2019-10-15  7:42 UTC (permalink / raw)
  To: ltp

Hello,

Li Wang <liwang@redhat.com> writes:

>>  }
>>
>> +static void print_colored(const char *str)
>> +{
>> +       if (tst_color_enabled(STDOUT_FILENO))
>> +               printf("%s%s%s", ANSI_COLOR_YELLOW, str, ANSI_COLOR_RESET);
>> +       else
>> +               printf("%s", str);
>> +}
>> +
>> +static void print_failure_hints(void)
>> +{
>> +       unsigned int i;
>> +       const struct tst_tag *tags = tst_test->tags;
>> +
>> +       if (!tags)
>> +               return;
>> +
>> +       for (i = 0; tags[i].name; i++) {
>> +               if (!strcmp(tags[i].name, "linux-git")) {
>> +                       printf("\n");
>> +                       print_colored("HINT: ");
>
> +                       printf("This is a regression test for linux kernel,
>> see commit:\n\n"

It may be better just to print "Associated Linux kernel commit: ..." and
"Associated bug ID: CVE-..." in the loop. Then we can avoid any
discussions about what type of test it is.

Also, a test description tag could be added if necessary.

Maybe it would also be a good idea to link back to the test source code
on github? Possibly this info could be injected at build time?

>> +                              LINUX_GIT_URL "%s\n", tags[i].value);
>>
>
> This sentence 'HINT: This is a ...' will be printed many times if there are
> many commits in tags, I prefer to see only once in front of these
> linux-kernel links.
>
>
>> +               }
>> +
>> +               if (!strcmp(tags[i].name, "CVE")) {
>> +                       printf("\n");
>> +                       print_colored("HINT: ");
>> +                       printf("This test also tests for CVE-%s, see:\n\n"
>> +                              CVE_DB_URL "%s\n", tags[i].value,
>> tags[i].value);
>>
>
> Here as well.
>
> -- 
> Regards,
> Li Wang


-- 
Thank you,
Richard.

^ permalink raw reply	[flat|nested] 17+ messages in thread

* [LTP] [PATCH 2/2] testcases/kernel: Add linux git and CVE tags
  2019-10-15  4:54   ` Li Wang
@ 2019-10-15  9:52     ` Cyril Hrubis
  2019-10-16 13:21       ` Petr Vorel
  0 siblings, 1 reply; 17+ messages in thread
From: Cyril Hrubis @ 2019-10-15  9:52 UTC (permalink / raw)
  To: ltp

Hi!
> > diff --git a/testcases/cve/cve-2014-0196.c b/testcases/cve/cve-2014-0196.c
> > index 6997fffe4..012cbb7cd 100644
> > --- a/testcases/cve/cve-2014-0196.c
> > +++ b/testcases/cve/cve-2014-0196.c
> > @@ -141,4 +141,9 @@ static struct tst_test test = {
> >         .setup = setup,
> >         .cleanup = cleanup,
> >         .test_all = run,
> > +       .tags = (const struct tst_tag[]) {
> > +               {"linux-git", "4291086b1f08"},
> > +               {"CVE", "2014-0196"},
> > +               {}
> >
> 
> Some of the tests use {NULL, NULL} to terminate tag array but some not,
> maybe we need to keep consistent in usage.

I guess I missed some when I was converting {NULL, NULL} to just {}. I
will fix that in v2.

-- 
Cyril Hrubis
chrubis@suse.cz

^ permalink raw reply	[flat|nested] 17+ messages in thread

* [LTP] [PATCH 1/2] lib: Add support for test tags
  2019-10-15  5:12   ` Li Wang
  2019-10-15  7:42     ` Richard Palethorpe
@ 2019-10-15  9:54     ` Cyril Hrubis
  1 sibling, 0 replies; 17+ messages in thread
From: Cyril Hrubis @ 2019-10-15  9:54 UTC (permalink / raw)
  To: ltp

Hi!
> > +static void print_test_info(void)
> >
> 
> print_test_info sounds like general information for the test, maybe
> print_tags() is a better/precise name?

I named it as such in a case that we will add anything else there, but I
do not have a strong opinion, I can change that if you insist.

> > +static void print_failure_hints(void)
> > +{
> > +       unsigned int i;
> > +       const struct tst_tag *tags = tst_test->tags;
> > +
> > +       if (!tags)
> > +               return;
> > +
> > +       for (i = 0; tags[i].name; i++) {
> > +               if (!strcmp(tags[i].name, "linux-git")) {
> > +                       printf("\n");
> > +                       print_colored("HINT: ");
> 
> +                       printf("This is a regression test for linux kernel,
> > see commit:\n\n"
> > +                              LINUX_GIT_URL "%s\n", tags[i].value);
> >
> 
> This sentence 'HINT: This is a ...' will be printed many times if there are
> many commits in tags, I prefer to see only once in front of these
> linux-kernel links.

Good catch, I will do something about this.

-- 
Cyril Hrubis
chrubis@suse.cz

^ permalink raw reply	[flat|nested] 17+ messages in thread

* [LTP] [PATCH 1/2] lib: Add support for test tags
  2019-10-15  7:42     ` Richard Palethorpe
@ 2019-10-15  9:57       ` Cyril Hrubis
  0 siblings, 0 replies; 17+ messages in thread
From: Cyril Hrubis @ 2019-10-15  9:57 UTC (permalink / raw)
  To: ltp

Hi!
> It may be better just to print "Associated Linux kernel commit: ..." and
> "Associated bug ID: CVE-..." in the loop. Then we can avoid any
> discussions about what type of test it is.

I will try that.

> Also, a test description tag could be added if necessary.

My plan is actually to keep the description in the top level comment but
parse that by the docparse tool, then we will have a browseable test
documentation that could be put to the web and we can point to that
here.

> Maybe it would also be a good idea to link back to the test source code
> on github? Possibly this info could be injected at build time?

I tried to inject things at a build time and it wasn't pretty, so I
would like to avoid that.

-- 
Cyril Hrubis
chrubis@suse.cz

^ permalink raw reply	[flat|nested] 17+ messages in thread

* [LTP] [PATCH 2/2] testcases/kernel: Add linux git and CVE tags
  2019-10-15  9:52     ` Cyril Hrubis
@ 2019-10-16 13:21       ` Petr Vorel
  2019-10-17  3:22         ` Li Wang
  2019-10-17  8:38         ` Jan Stancek
  0 siblings, 2 replies; 17+ messages in thread
From: Petr Vorel @ 2019-10-16 13:21 UTC (permalink / raw)
  To: ltp

Hi,

> > > +       .tags = (const struct tst_tag[]) {
> > > +               {"linux-git", "4291086b1f08"},
> > > +               {"CVE", "2014-0196"},
> > > +               {}

> > Some of the tests use {NULL, NULL} to terminate tag array but some not,
> > maybe we need to keep consistent in usage.

> I guess I missed some when I was converting {NULL, NULL} to just {}. I
> will fix that in v2.
+1

Nice work, thanks :).
Reviewed-by: Petr Vorel <pvorel@suse.cz>

nit: I don't that much blank lines, I'd prefer more compact output:

OPTIONS
-h       Prints this help
-i n     Execute test n times
-I x     Execute test for n seconds
-C ARG   Run child process with ARG arguments (used internally)

TAGS
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=0c461cb727d1
https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017-2618

Kind regards,
Petr

^ permalink raw reply	[flat|nested] 17+ messages in thread

* [LTP] [PATCH 1/2] lib: Add support for test tags
  2019-10-14 11:25 ` [LTP] [PATCH 1/2] lib: Add support for " Cyril Hrubis
  2019-10-15  5:12   ` Li Wang
@ 2019-10-16 13:30   ` Petr Vorel
  2019-10-16 14:46     ` Cyril Hrubis
  1 sibling, 1 reply; 17+ messages in thread
From: Petr Vorel @ 2019-10-16 13:30 UTC (permalink / raw)
  To: ltp

Hi,

> +	if (tags) {
> +		for (i = 0; tags[i].name; i++) {
> +			if (!strcmp(tags[i].name, "CVE"))
> +				printf(CVE_DB_URL "%s\n", tags[i].value);
> +			else if (!strcmp(tags[i].name, "linux-git"))
Not sure if it's worth of defining some enums instead of "CVE", "linux-git"
(would catch typos).

> +				printf(LINUX_GIT_URL "%s\n", tags[i].value);

> +			else
> +				printf("%s: %s\n", tags[i].name, tags[i].value);
> +			printf("\n");

Kind regards,
Petr

^ permalink raw reply	[flat|nested] 17+ messages in thread

* [LTP] [PATCH 1/2] lib: Add support for test tags
  2019-10-16 13:30   ` Petr Vorel
@ 2019-10-16 14:46     ` Cyril Hrubis
  2019-10-16 19:43       ` Petr Vorel
  0 siblings, 1 reply; 17+ messages in thread
From: Cyril Hrubis @ 2019-10-16 14:46 UTC (permalink / raw)
  To: ltp

Hi!
> > +	if (tags) {
> > +		for (i = 0; tags[i].name; i++) {
> > +			if (!strcmp(tags[i].name, "CVE"))
> > +				printf(CVE_DB_URL "%s\n", tags[i].value);
> > +			else if (!strcmp(tags[i].name, "linux-git"))
> Not sure if it's worth of defining some enums instead of "CVE", "linux-git"
> (would catch typos).

I wanted to avoid enums because they are not flexible enough. The plan
here is that docparse would do sanity checks on test metadata and fail
the compilation if it founds typos there. With the checks in the
docparse tool we can easily check for tags that looks like typos, i.e.
check the levenshtein distance from all known tags and print a nice
looking error message...

-- 
Cyril Hrubis
chrubis@suse.cz

^ permalink raw reply	[flat|nested] 17+ messages in thread

* [LTP] [PATCH 1/2] lib: Add support for test tags
  2019-10-16 14:46     ` Cyril Hrubis
@ 2019-10-16 19:43       ` Petr Vorel
  0 siblings, 0 replies; 17+ messages in thread
From: Petr Vorel @ 2019-10-16 19:43 UTC (permalink / raw)
  To: ltp

Hi Cyril,

> > > +	if (tags) {
> > > +		for (i = 0; tags[i].name; i++) {
> > > +			if (!strcmp(tags[i].name, "CVE"))
> > > +				printf(CVE_DB_URL "%s\n", tags[i].value);
> > > +			else if (!strcmp(tags[i].name, "linux-git"))
> > Not sure if it's worth of defining some enums instead of "CVE", "linux-git"
> > (would catch typos).

> I wanted to avoid enums because they are not flexible enough. The plan
> here is that docparse would do sanity checks on test metadata and fail
> the compilation if it founds typos there. With the checks in the
> docparse tool we can easily check for tags that looks like typos, i.e.
> check the levenshtein distance from all known tags and print a nice
> looking error message...
OK, sounds good :).

Kind regards,
Petr

^ permalink raw reply	[flat|nested] 17+ messages in thread

* [LTP] [PATCH 2/2] testcases/kernel: Add linux git and CVE tags
  2019-10-16 13:21       ` Petr Vorel
@ 2019-10-17  3:22         ` Li Wang
  2019-10-17  8:38         ` Jan Stancek
  1 sibling, 0 replies; 17+ messages in thread
From: Li Wang @ 2019-10-17  3:22 UTC (permalink / raw)
  To: ltp

Petr Vorel <pvorel@suse.cz> wrote:

...
> nit: I don't that much blank lines, I'd prefer more compact output:
> ...
> TAGS
>
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=0c461cb727d1
> https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017-2618
>
>
The explanation(HINT: ...) is needed, otherwise, it will misleading someone
not familiar with LTP I think.

-- 
Regards,
Li Wang
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linux.it/pipermail/ltp/attachments/20191017/3f153d07/attachment.htm>

^ permalink raw reply	[flat|nested] 17+ messages in thread

* [LTP] [PATCH 2/2] testcases/kernel: Add linux git and CVE tags
  2019-10-16 13:21       ` Petr Vorel
  2019-10-17  3:22         ` Li Wang
@ 2019-10-17  8:38         ` Jan Stancek
  1 sibling, 0 replies; 17+ messages in thread
From: Jan Stancek @ 2019-10-17  8:38 UTC (permalink / raw)
  To: ltp



----- Original Message -----
> nit: I don't that much blank lines, I'd prefer more compact output:
...
> 
> TAGS
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=0c461cb727d1
> https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017-2618

+1, at least drop some blank lines. With more tags output seems unnecessarily scattered.

^ permalink raw reply	[flat|nested] 17+ messages in thread

* [LTP] [PATCH 0/2] Add support for a test tags
  2019-10-14 11:23 [LTP] [PATCH 0/2] Add support for a test tags Cyril Hrubis
@ 2019-10-14 11:25 ` Cyril Hrubis
  0 siblings, 0 replies; 17+ messages in thread
From: Cyril Hrubis @ 2019-10-14 11:25 UTC (permalink / raw)
  To: ltp

Hi!
Please ignore this patchset, I will resend it from my work email
address.

-- 
Cyril Hrubis
chrubis@suse.cz

^ permalink raw reply	[flat|nested] 17+ messages in thread

* [LTP] [PATCH 0/2] Add support for a test tags
@ 2019-10-14 11:23 Cyril Hrubis
  2019-10-14 11:25 ` Cyril Hrubis
  0 siblings, 1 reply; 17+ messages in thread
From: Cyril Hrubis @ 2019-10-14 11:23 UTC (permalink / raw)
  To: ltp

From: Cyril Hrubis <chrubis@suse.cz>

This is a newest version of the test tags, first patch adds a support
for test tags to a test library, the second one annotates tests with
tags. Not everything is annotated at the moment but fair amount of tests
is included.

These patches are also a stepping stone for the test metadata project
that will allow us to do much more than this. You can have a look at:

https://github.com/metan-ucw/ltp/tree/master/docparse

Also if you run a make in the docparse directory it will generate test
metadata along with a single html page with nice test metadata
description.

While the whole project is work-in-progress, the test tags are finished
and may add a value even without the rest of the changes, see the first
commit for details.

Cyril Hrubis (2):
  lib: Add support for test tags
  testcases/kernel: Add tags

 include/tst_test.h                            | 10 +++
 lib/tst_test.c                                | 77 +++++++++++++++++--
 testcases/cve/cve-2014-0196.c                 |  5 ++
 testcases/cve/cve-2015-3290.c                 |  5 ++
 testcases/cve/cve-2016-10044.c                |  5 ++
 testcases/cve/cve-2016-7042.c                 |  5 ++
 testcases/cve/cve-2016-7117.c                 |  5 ++
 testcases/cve/cve-2017-16939.c                |  5 ++
 testcases/cve/cve-2017-17052.c                |  5 ++
 testcases/cve/cve-2017-17053.c                |  5 ++
 testcases/cve/cve-2017-2618.c                 |  5 ++
 testcases/cve/cve-2017-2671.c                 |  5 ++
 testcases/cve/meltdown.c                      |  6 +-
 testcases/cve/stack_clash.c                   |  4 +
 testcases/kernel/crypto/af_alg01.c            |  5 ++
 testcases/kernel/crypto/af_alg02.c            |  5 ++
 testcases/kernel/crypto/af_alg03.c            |  4 +
 testcases/kernel/crypto/af_alg04.c            |  4 +
 testcases/kernel/crypto/af_alg05.c            |  8 +-
 testcases/kernel/crypto/af_alg06.c            |  4 +
 testcases/kernel/crypto/crypto_user01.c       |  6 ++
 testcases/kernel/crypto/crypto_user02.c       |  4 +
 testcases/kernel/crypto/pcrypt_aead01.c       |  5 ++
 testcases/kernel/sound/snd_timer01.c          |  6 ++
 testcases/kernel/syscalls/accept/accept02.c   |  5 ++
 testcases/kernel/syscalls/acct/acct02.c       |  4 +
 testcases/kernel/syscalls/add_key/add_key02.c |  5 ++
 testcases/kernel/syscalls/bpf/bpf_prog02.c    |  4 +
 .../syscalls/futex/futex_cmp_requeue02.c      |  5 ++
 testcases/kernel/syscalls/keyctl/keyctl02.c   |  4 +
 testcases/kernel/syscalls/keyctl/keyctl04.c   |  5 ++
 testcases/kernel/syscalls/keyctl/keyctl05.c   |  4 +
 .../kernel/syscalls/move_pages/move_pages12.c |  6 ++
 .../kernel/syscalls/realpath/realpath01.c     |  4 +
 .../syscalls/request_key/request_key05.c      |  4 +
 testcases/kernel/syscalls/sbrk/sbrk03.c       |  4 +
 testcases/kernel/syscalls/sendto/sendto02.c   |  4 +
 .../kernel/syscalls/setrlimit/setrlimit06.c   |  4 +
 38 files changed, 251 insertions(+), 9 deletions(-)

-- 
2.21.0


^ permalink raw reply	[flat|nested] 17+ messages in thread

end of thread, other threads:[~2019-10-17  8:38 UTC | newest]

Thread overview: 17+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-10-14 11:25 [LTP] [PATCH 0/2] Add support for a test tags Cyril Hrubis
2019-10-14 11:25 ` [LTP] [PATCH 1/2] lib: Add support for " Cyril Hrubis
2019-10-15  5:12   ` Li Wang
2019-10-15  7:42     ` Richard Palethorpe
2019-10-15  9:57       ` Cyril Hrubis
2019-10-15  9:54     ` Cyril Hrubis
2019-10-16 13:30   ` Petr Vorel
2019-10-16 14:46     ` Cyril Hrubis
2019-10-16 19:43       ` Petr Vorel
2019-10-14 11:25 ` [LTP] [PATCH 2/2] testcases/kernel: Add linux git and CVE tags Cyril Hrubis
2019-10-15  4:54   ` Li Wang
2019-10-15  9:52     ` Cyril Hrubis
2019-10-16 13:21       ` Petr Vorel
2019-10-17  3:22         ` Li Wang
2019-10-17  8:38         ` Jan Stancek
  -- strict thread matches above, loose matches on Subject: below --
2019-10-14 11:23 [LTP] [PATCH 0/2] Add support for a test tags Cyril Hrubis
2019-10-14 11:25 ` Cyril Hrubis

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.