Patch series to add to and imporve tests for CoreSight

Patch series to add to and imporve tests for CoreSight

[carsten.haitzler]

This improves how perf test handles finding shell scripts to run as
part of the testing allowing sub-directories, skipping files that
are not intentional shell script tests and then adds a set of tests
that improve what we cover for Arm CoreSight testing. The goal of
these tests is to expand the scenarios and data traced and examined
to track quality of trace data and improvements to that over time.

Eventually it'd be neater to break up the current Arm CoreSight tests
into smaller focused tests that share the same infrastructure, but
this here would be a first step.

[PATCH 01/14] perf test: Refactor shell tests allowing subdirs

[carsten.haitzler]

From: "Carsten Haitzler (Rasterman)" <raster@rasterman.com>

This is a prelude to adding more tests to shell tests and in order to
support putting those tests into subdirectories, I need to change the
test code that scans/finds and runs them.

To support subdirs I have to recurse so it's time to refactor the code to
allow this and centralize the shell script finding into one location and
only one single scan that builds a list of all the found tests in memory
instead of it being duplicated in 3 places.

This code also optimizes things like knowing the max width of desciption
strings (as we can do that while we scan instead of a whole new pass
of opening files). It also more cleanly filters scripts to see only
*.sh files thus skipping random other files in directories like *~
backup files, other random junk/data files that may appear and the
scripts must be executable to make the cut (this ensures the script
lib dir is not seen as scripts to run). This avoids perf test running
previous older versions of test scripts that are editor backup files
as well as skipping perf.data files that may appear and so on.

Signed-off-by: Carsten Haitzler <carsten.haitzler@arm.com>
---
 tools/perf/tests/Build               |   1 +
 tools/perf/tests/builtin-test-list.c | 201 +++++++++++++++++++++++++++
 tools/perf/tests/builtin-test-list.h |  12 ++
 tools/perf/tests/builtin-test.c      | 152 +++-----------------
 4 files changed, 232 insertions(+), 134 deletions(-)
 create mode 100644 tools/perf/tests/builtin-test-list.c
 create mode 100644 tools/perf/tests/builtin-test-list.h

diff --git a/tools/perf/tests/Build b/tools/perf/tests/Build
index af2b37ef7c70..2064a640facb 100644
--- a/tools/perf/tests/Build
+++ b/tools/perf/tests/Build
@@ -1,6 +1,7 @@
 # SPDX-License-Identifier: GPL-2.0
 
 perf-y += builtin-test.o
+perf-y += builtin-test-list.o
 perf-y += parse-events.o
 perf-y += dso-data.o
 perf-y += attr.o
diff --git a/tools/perf/tests/builtin-test-list.c b/tools/perf/tests/builtin-test-list.c
new file mode 100644
index 000000000000..1e60088c1005
--- /dev/null
+++ b/tools/perf/tests/builtin-test-list.c
@@ -0,0 +1,201 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#include <fcntl.h>
+#include <errno.h>
+#include <unistd.h>
+#include <string.h>
+#include <stdlib.h>
+#include <sys/types.h>
+#include <dirent.h>
+#include <sys/wait.h>
+#include <sys/stat.h>
+#include "builtin.h"
+#include "hist.h"
+#include "intlist.h"
+#include "tests.h"
+#include "debug.h"
+#include "color.h"
+#include <subcmd/parse-options.h>
+#include "string2.h"
+#include "symbol.h"
+#include "util/rlimit.h"
+#include <linux/kernel.h>
+#include <linux/string.h>
+#include <subcmd/exec-cmd.h>
+#include <linux/zalloc.h>
+
+#include "builtin-test-list.h"
+
+#include <linux/ctype.h>
+
+/* As this is a singleton built once for the run of the process, there is
+ * no value in trying to free it and just let it stay around until process
+ * exits when it's cleaned up. */
+static size_t files_num = 0;
+static struct script_file *files = NULL;
+static int files_max_width = 0;
+
+static const char *shell_tests__dir(char *path, size_t size)
+{
+	const char *devel_dirs[] = { "./tools/perf/tests", "./tests", };
+	char *exec_path;
+	unsigned int i;
+
+	for (i = 0; i < ARRAY_SIZE(devel_dirs); ++i) {
+		struct stat st;
+
+		if (!lstat(devel_dirs[i], &st)) {
+			scnprintf(path, size, "%s/shell", devel_dirs[i]);
+			if (!lstat(devel_dirs[i], &st))
+				return path;
+		}
+	}
+
+	/* Then installed path. */
+	exec_path = get_argv_exec_path();
+	scnprintf(path, size, "%s/tests/shell", exec_path);
+	free(exec_path);
+	return path;
+}
+
+static const char *shell_test__description(char *description, size_t size,
+                                           const char *path, const char *name)
+{
+	FILE *fp;
+	char filename[PATH_MAX];
+	int ch;
+
+	path__join(filename, sizeof(filename), path, name);
+	fp = fopen(filename, "r");
+	if (!fp)
+		return NULL;
+
+	/* Skip first line - should be #!/bin/sh Shebang */
+	do {
+		ch = fgetc(fp);
+	} while (ch != EOF && ch != '\n');
+
+	description = fgets(description, size, fp);
+	fclose(fp);
+
+	/* Assume first char on line is omment everything after that desc */
+	return description ? strim(description + 1) : NULL;
+}
+
+static bool is_shell_script(const char *path)
+{ /* is this full file path a shell script */
+	const char *ext;
+
+	ext = strrchr(path, '.');
+	if (!ext)
+		return false;
+	if (!strcmp(ext, ".sh")) { /* Has .sh extension */
+		if (access(path, R_OK | X_OK) == 0) /* Is executable */
+			return true;
+	}
+	return false;
+}
+
+static bool is_test_script(const char *path, const char *name)
+{ /* Is this file in this dir a shell script (for test purposes) */
+	char filename[PATH_MAX];
+
+	path__join(filename, sizeof(filename), path, name);
+	if (!is_shell_script(filename)) return false;
+	return true;
+}
+
+static char *strdup_check(const char *str)
+{ /* Duplicate a string and fall over and die if we run out of memory */
+	char *newstr;
+
+	newstr = strdup(str);
+	if (!newstr) {
+		pr_err("Out of memory while duplicating test script string\n");
+		abort();
+	}
+	return newstr;
+}
+
+static void append_script(const char *dir, const char *file, const char *desc)
+{
+	struct script_file *files_tmp;
+	size_t files_num_tmp;
+	int width;
+
+	files_num_tmp = files_num + 1;
+	if (files_num_tmp < 1) {
+		pr_err("Too many script files\n");
+		abort();
+	}
+	/* Realloc is good enough, though we could realloc by chunks, not that
+	 * anyone will ever measure performance here */
+	files_tmp = realloc(files,
+			    (files_num_tmp + 1) * sizeof(struct script_file));
+	if (files_tmp == NULL) {
+		pr_err("Out of memory while building test list\n");
+		abort();
+	}
+	/* Add file to end and NULL terminate the struct array */
+	files = files_tmp;
+	files_num = files_num_tmp;
+	files[files_num - 1].dir = strdup_check(dir);
+	files[files_num - 1].file = strdup_check(file);
+	files[files_num - 1].desc = strdup_check(desc);
+	files[files_num].dir = NULL;
+	files[files_num].file = NULL;
+	files[files_num].desc = NULL;
+
+	width = strlen(desc); /* Track max width of desc */
+	if (width > files_max_width)
+		files_max_width = width;
+}
+
+static void append_scripts_in_dir(const char *path)
+{
+	struct dirent **entlist;
+	struct dirent *ent;
+	int n_dirs, i;
+	char filename[PATH_MAX];
+
+	/* List files, sorted by alpha */
+	n_dirs = scandir(path, &entlist, NULL, alphasort);
+	if (n_dirs == -1)
+		return;
+	for (i = 0; i < n_dirs && (ent = entlist[i]); i++) {
+		if (ent->d_name[0] == '.') continue; /* Skip hidden files */
+		if (is_test_script(path, ent->d_name)) { /* It's a test */
+			char bf[256];
+			const char *desc = shell_test__description
+				(bf, sizeof(bf), path, ent->d_name);
+
+			if (desc) /* It has a desc line - valid script */
+				append_script(path, ent->d_name, desc);
+		} else if (is_directory(path, ent)) { /* Scan the subdir */
+			path__join(filename, sizeof(filename),
+				   path, ent->d_name);
+			append_scripts_in_dir(filename);
+		}
+	}
+	for (i = 0; i < n_dirs; i++) /* Clean up */
+		zfree(&entlist[i]);
+	free(entlist);
+}
+
+const struct script_file *list_script_files(void)
+{
+	char path_dir[PATH_MAX];
+	const char *path;
+
+	if (files) return files; /* Singleton - we already know our list */
+
+	path = shell_tests__dir(path_dir, sizeof(path_dir)); /* Walk  dir */
+	append_scripts_in_dir(path);
+
+	return files;
+}
+
+int list_script_max_width(void)
+{
+	list_script_files(); /* Ensure we have scanned all scriptd */
+	return files_max_width;
+}
diff --git a/tools/perf/tests/builtin-test-list.h b/tools/perf/tests/builtin-test-list.h
new file mode 100644
index 000000000000..eb81f3aa6683
--- /dev/null
+++ b/tools/perf/tests/builtin-test-list.h
@@ -0,0 +1,12 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+struct script_file {
+	char *dir;
+	char *file;
+	char *desc;
+};
+
+/* List available script tests to run - singleton - never freed */
+const struct script_file *list_script_files(void);
+/* Get maximum width of description string */
+int list_script_max_width(void);
diff --git a/tools/perf/tests/builtin-test.c b/tools/perf/tests/builtin-test.c
index 81cf241cd109..7122eae1d98d 100644
--- a/tools/perf/tests/builtin-test.c
+++ b/tools/perf/tests/builtin-test.c
@@ -28,6 +28,8 @@
 #include <subcmd/exec-cmd.h>
 #include <linux/zalloc.h>
 
+#include "builtin-test-list.h"
+
 static bool dont_fork;
 
 struct test_suite *__weak arch_tests[] = {
@@ -274,91 +276,6 @@ static int test_and_print(struct test_suite *t, int subtest)
 	return err;
 }
 
-static const char *shell_test__description(char *description, size_t size,
-					   const char *path, const char *name)
-{
-	FILE *fp;
-	char filename[PATH_MAX];
-	int ch;
-
-	path__join(filename, sizeof(filename), path, name);
-	fp = fopen(filename, "r");
-	if (!fp)
-		return NULL;
-
-	/* Skip shebang */
-	do {
-		ch = fgetc(fp);
-	} while (ch != EOF && ch != '\n');
-
-	description = fgets(description, size, fp);
-	fclose(fp);
-
-	return description ? strim(description + 1) : NULL;
-}
-
-#define for_each_shell_test(entlist, nr, base, ent)	                \
-	for (int __i = 0; __i < nr && (ent = entlist[__i]); __i++)	\
-		if (!is_directory(base, ent) && \
-			is_executable_file(base, ent) && \
-			ent->d_name[0] != '.')
-
-static const char *shell_tests__dir(char *path, size_t size)
-{
-	const char *devel_dirs[] = { "./tools/perf/tests", "./tests", };
-        char *exec_path;
-	unsigned int i;
-
-	for (i = 0; i < ARRAY_SIZE(devel_dirs); ++i) {
-		struct stat st;
-		if (!lstat(devel_dirs[i], &st)) {
-			scnprintf(path, size, "%s/shell", devel_dirs[i]);
-			if (!lstat(devel_dirs[i], &st))
-				return path;
-		}
-	}
-
-        /* Then installed path. */
-        exec_path = get_argv_exec_path();
-        scnprintf(path, size, "%s/tests/shell", exec_path);
-	free(exec_path);
-	return path;
-}
-
-static int shell_tests__max_desc_width(void)
-{
-	struct dirent **entlist;
-	struct dirent *ent;
-	int n_dirs, e;
-	char path_dir[PATH_MAX];
-	const char *path = shell_tests__dir(path_dir, sizeof(path_dir));
-	int width = 0;
-
-	if (path == NULL)
-		return -1;
-
-	n_dirs = scandir(path, &entlist, NULL, alphasort);
-	if (n_dirs == -1)
-		return -1;
-
-	for_each_shell_test(entlist, n_dirs, path, ent) {
-		char bf[256];
-		const char *desc = shell_test__description(bf, sizeof(bf), path, ent->d_name);
-
-		if (desc) {
-			int len = strlen(desc);
-
-			if (width < len)
-				width = len;
-		}
-	}
-
-	for (e = 0; e < n_dirs; e++)
-		zfree(&entlist[e]);
-	free(entlist);
-	return width;
-}
-
 struct shell_test {
 	const char *dir;
 	const char *file;
@@ -385,33 +302,17 @@ static int shell_test__run(struct test_suite *test, int subdir __maybe_unused)
 static int run_shell_tests(int argc, const char *argv[], int i, int width,
 				struct intlist *skiplist)
 {
-	struct dirent **entlist;
-	struct dirent *ent;
-	int n_dirs, e;
-	char path_dir[PATH_MAX];
-	struct shell_test st = {
-		.dir = shell_tests__dir(path_dir, sizeof(path_dir)),
-	};
-
-	if (st.dir == NULL)
-		return -1;
+	struct shell_test st;
+	const struct script_file *files, *file;
 
-	n_dirs = scandir(st.dir, &entlist, NULL, alphasort);
-	if (n_dirs == -1) {
-		pr_err("failed to open shell test directory: %s\n",
-			st.dir);
-		return -1;
-	}
-
-	for_each_shell_test(entlist, n_dirs, st.dir, ent) {
+	files = list_script_files();
+	if (!files)
+		return 0;
+	for (file = files; file->dir; file++) {
 		int curr = i++;
-		char desc[256];
 		struct test_case test_cases[] = {
 			{
-				.desc = shell_test__description(desc,
-								sizeof(desc),
-								st.dir,
-								ent->d_name),
+				.desc = file->desc,
 				.run_case = shell_test__run,
 			},
 			{ .name = NULL, }
@@ -421,12 +322,13 @@ static int run_shell_tests(int argc, const char *argv[], int i, int width,
 			.test_cases = test_cases,
 			.priv = &st,
 		};
+		st.dir = file->dir;
 
 		if (test_suite.desc == NULL ||
 		    !perf_test__matches(test_suite.desc, curr, argc, argv))
 			continue;
 
-		st.file = ent->d_name;
+		st.file = file->file;
 		pr_info("%3d: %-*s:", i, width, test_suite.desc);
 
 		if (intlist__find(skiplist, i)) {
@@ -436,10 +338,6 @@ static int run_shell_tests(int argc, const char *argv[], int i, int width,
 
 		test_and_print(&test_suite, 0);
 	}
-
-	for (e = 0; e < n_dirs; e++)
-		zfree(&entlist[e]);
-	free(entlist);
 	return 0;
 }
 
@@ -448,7 +346,7 @@ static int __cmd_test(int argc, const char *argv[], struct intlist *skiplist)
 	struct test_suite *t;
 	unsigned int j, k;
 	int i = 0;
-	int width = shell_tests__max_desc_width();
+	int width = list_script_max_width();
 
 	for_each_test(j, k, t) {
 		int len = strlen(test_description(t, -1));
@@ -529,36 +427,22 @@ static int __cmd_test(int argc, const char *argv[], struct intlist *skiplist)
 
 static int perf_test__list_shell(int argc, const char **argv, int i)
 {
-	struct dirent **entlist;
-	struct dirent *ent;
-	int n_dirs, e;
-	char path_dir[PATH_MAX];
-	const char *path = shell_tests__dir(path_dir, sizeof(path_dir));
-
-	if (path == NULL)
-		return -1;
+	const struct script_file *files, *file;
 
-	n_dirs = scandir(path, &entlist, NULL, alphasort);
-	if (n_dirs == -1)
-		return -1;
-
-	for_each_shell_test(entlist, n_dirs, path, ent) {
+	files = list_script_files();
+	if (!files)
+		return 0;
+	for (file = files; file->dir; file++) {
 		int curr = i++;
-		char bf[256];
 		struct test_suite t = {
-			.desc = shell_test__description(bf, sizeof(bf), path, ent->d_name),
+			.desc = file->desc
 		};
 
 		if (!perf_test__matches(t.desc, curr, argc, argv))
 			continue;
 
 		pr_info("%3d: %s\n", i, t.desc);
-
 	}
-
-	for (e = 0; e < n_dirs; e++)
-		zfree(&entlist[e]);
-	free(entlist);
 	return 0;
 }
 
-- 
2.32.0

[PATCH 02/14] perf test: Add CoreSight shell lib shared code for future tests

[carsten.haitzler]

From: "Carsten Haitzler (Rasterman)" <raster@rasterman.com>

This adds a library of shell "code" to be shared and used by future
tests that target quality testing for Arm CoreSight support in perf
and the Linux kernel.

Signed-off-by: Carsten Haitzler <carsten.haitzler@arm.com>
---
 tools/perf/tests/shell/lib/coresight.sh | 122 ++++++++++++++++++++++++
 1 file changed, 122 insertions(+)
 create mode 100644 tools/perf/tests/shell/lib/coresight.sh

diff --git a/tools/perf/tests/shell/lib/coresight.sh b/tools/perf/tests/shell/lib/coresight.sh
new file mode 100644
index 000000000000..1d6c90c3b8c1
--- /dev/null
+++ b/tools/perf/tests/shell/lib/coresight.sh
@@ -0,0 +1,122 @@
+# SPDX-License-Identifier: GPL-2.0
+# Carsten Haitzler <carsten.haitzler@arm.com>, 2021
+
+# This is sourced from a driver script so no need for #!/bin... etc. at the
+# top - the assumption below is that it runs as part of sourcing after the
+# test sets up some basic env vars to say what it is.
+
+# perf record options for the perf tests to use
+PERFRECMEM="-m ,16M"
+PERFRECOPT="$PERFRECMEM -e cs_etm//u"
+
+TOOLS=$(dirname $0)
+DIR="$TOOLS/$TEST"
+BIN="$DIR/$TEST"
+# If the test tool/binary does not exist and is executable then skip the test
+if ! test -x "$BIN"; then exit 2; fi
+DATD="."
+# If the data dir env is set then make the data dir use that instead of ./
+if test -n "$PERF_TEST_CORESIGHT_DATADIR"; then
+	DATD="$PERF_TEST_CORESIGHT_DATADIR";
+fi
+# If the stat dir env is set then make the data dir use that instead of ./
+STATD="."
+if test -n "$PERF_TEST_CORESIGHT_STATDIR"; then
+	STATD="$PERF_TEST_CORESIGHT_STATDIR";
+fi
+
+# Called if the test fails - error code 2
+err() {
+	echo "$1"
+	exit 1
+}
+
+# Check that some statistics from our perf
+check_val_min() {
+	STATF="$4"
+	if test "$2" -lt "$3"; then
+		echo ", FAILED" >> "$STATF"
+		err "Sanity check number of $1 is too low ($2 < $3)"
+	fi
+}
+
+perf_dump_aux_verify() {
+	# Some basic checking that the AUX chunk contains some sensible data
+	# to see that we are recording something and at least a minimum
+	# amount of it. We should almost always see F3 atoms in just about
+	# anything but certainly we will see some trace info and async atom
+	# chunks.
+	DUMP="$DATD/perf-tmp-aux-dump.txt"
+	perf report --stdio --dump -i "$1" | \
+		grep -o -e I_ATOM_F3 -e I_ASYNC -e I_TRACE_INFO > "$DUMP"
+	# Simply count how many of these atoms we find to see that we are
+	# producing a reasonable amount of data - exact checks are not sane
+	# as this is a lossy  process where we may lose some blocks and the
+	# compiler may produce different code depending on the compiler and
+	# optimization options, so this is rough  just to see if we're
+	# either missing almost all the data or all of it
+	ATOM_F3_NUM=`grep I_ATOM_F3 "$DUMP" | wc -l`
+	ATOM_ASYNC_NUM=`grep I_ASYNC "$DUMP" | wc -l`
+	ATOM_TRACE_INFO_NUM=`grep I_TRACE_INFO "$DUMP" | wc -l`
+	rm -f "$DUMP"
+
+	# Arguments provide minimums for a pass
+	CHECK_F3_MIN="$2"
+	CHECK_ASYNC_MIN="$3"
+	CHECK_TRACE_INFO_MIN="$4"
+
+	# Write out statistics, so over time you can track results to see if
+	# there is a pattern - for example we have less "noisy" results that
+	# produce more consistent amounts of data each run, to see if over
+	# time any techinques to  minimize data loss are having an effect or
+	# not
+	STATF="$STATD/stats-$TEST-$DATV.csv"
+	if ! test -f "$STATF"; then
+		echo "ATOM F3 Count, Minimum, ATOM ASYNC Count, Minimum, TRACE INFO Count, Minimum" > "$STATF"
+	fi
+	echo -n "$ATOM_F3_NUM, $CHECK_F3_MIN, $ATOM_ASYNC_NUM, $CHECK_ASYNC_MIN, $ATOM_TRACE_INFO_NUM, $CHECK_TRACE_INFO_MIN" >> "$STATF"
+
+	# Actually check to see if we passed or failed.
+	check_val_min "ATOM_F3" "$ATOM_F3_NUM" "$CHECK_F3_MIN" "$STATF"
+	check_val_min "ASYNC" "$ATOM_ASYNC_NUM" "$CHECK_ASYNC_MIN" "$STATF"
+	check_val_min "TRACE_INFO" "$ATOM_TRACE_INFO_NUM" "$CHECK_TRACE_INFO_MIN" "$STATF"
+	echo ", Ok" >> "$STATF"
+}
+
+perf_dump_aux_tid_verify() {
+	# Specifically crafted test will produce a list of Tread ID's to
+	# stdout that need to be checked to  see that they have had trace
+	# info collected in AUX blocks in the perf data. This will go
+	# through all the TID's that are listed as CID=0xabcdef and see
+	# that all the Thread IDs the test tool reports are  in the perf
+	# data AUX chunks
+
+	# The TID test tools will print a TID per stdout line that are being
+	# tested
+	TIDS=`cat "$2"`
+	# Scan the perf report to find the TIDs that are actually CID in hex
+	# and build a list of the ones found
+	FOUND_TIDS=`perf report --stdio --dump -i "$1" | \
+			grep -o "CID=0x[0-9a-z]\+" | sed 's/CID=//g' | \
+			uniq | sort | uniq`
+
+	# Iterate over the list of TIDs that the test says it has and find
+	# them in the TIDs found in the perf report
+	MISSING=""
+	for TID2 in $TIDS; do
+		FOUND=""
+		for TIDHEX in $FOUND_TIDS; do
+			TID=`printf "%i" $TIDHEX`
+			if test "$TID" -eq "$TID2"; then
+				FOUND="y"
+				break
+			fi
+		done
+		if test -z "$FOUND"; then
+			MISSING="$MISSING $TID"
+		fi
+	done
+	if test -n "$MISSING"; then
+		err "Thread IDs $MISSING not found in perf AUX data"
+	fi
+}
-- 
2.32.0

[PATCH 03/14] perf test: Add build infra for perf test tools for CoreSight tests

[carsten.haitzler]

From: "Carsten Haitzler (Rasterman)" <raster@rasterman.com>

This adds the initial build infrastructure (makefiles maintainers
information) for adding follow-on tests for CoreSight.

Signed-off-by: Carsten Haitzler <carsten.haitzler@arm.com>
---
 MAINTAINERS                                   |  1 +
 tools/perf/Makefile.perf                      | 18 ++++++++++---
 tools/perf/tests/shell/coresight/Makefile     | 26 +++++++++++++++++++
 .../tests/shell/coresight/Makefile.miniconfig | 24 +++++++++++++++++
 4 files changed, 66 insertions(+), 3 deletions(-)
 create mode 100644 tools/perf/tests/shell/coresight/Makefile
 create mode 100644 tools/perf/tests/shell/coresight/Makefile.miniconfig

diff --git a/MAINTAINERS b/MAINTAINERS
index 171563d8dc14..87e4ac463429 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1989,6 +1989,7 @@ F:	drivers/hwtracing/coresight/*
 F:	include/dt-bindings/arm/coresight-cti-dt.h
 F:	include/linux/coresight*
 F:	samples/coresight/*
+F:	tools/perf/tests/shell/coresight/*
 F:	tools/perf/arch/arm/util/auxtrace.c
 F:	tools/perf/arch/arm/util/cs-etm.c
 F:	tools/perf/arch/arm/util/cs-etm.h
diff --git a/tools/perf/Makefile.perf b/tools/perf/Makefile.perf
index 8f738e11356d..edb621ace2e2 100644
--- a/tools/perf/Makefile.perf
+++ b/tools/perf/Makefile.perf
@@ -629,7 +629,15 @@ sync_file_range_tbls := $(srctree)/tools/perf/trace/beauty/sync_file_range.sh
 $(sync_file_range_arrays): $(linux_uapi_dir)/fs.h $(sync_file_range_tbls)
 	$(Q)$(SHELL) '$(sync_file_range_tbls)' $(linux_uapi_dir) > $@
 
-all: shell_compatibility_test $(ALL_PROGRAMS) $(LANG_BINDINGS) $(OTHER_PROGRAMS)
+TESTS_CORESIGHT_DIR := $(srctree)/tools/perf/tests/shell/coresight
+
+tests-coresight-targets: FORCE
+	$(Q)$(MAKE) -C $(TESTS_CORESIGHT_DIR)
+
+tests-coresight-targets-clean:
+	$(Q)$(MAKE) -C $(TESTS_CORESIGHT_DIR) clean
+
+all: shell_compatibility_test $(ALL_PROGRAMS) $(LANG_BINDINGS) $(OTHER_PROGRAMS) tests-coresight-targets
 
 # Create python binding output directory if not already present
 _dummy := $(shell [ -d '$(OUTPUT)python' ] || mkdir -p '$(OUTPUT)python')
@@ -1015,7 +1023,10 @@ install-tests: all install-gtk
 		$(INSTALL) -d -m 755 '$(DESTDIR_SQ)$(perfexec_instdir_SQ)/tests/shell'; \
 		$(INSTALL) tests/shell/*.sh '$(DESTDIR_SQ)$(perfexec_instdir_SQ)/tests/shell'; \
 		$(INSTALL) -d -m 755 '$(DESTDIR_SQ)$(perfexec_instdir_SQ)/tests/shell/lib'; \
-		$(INSTALL) tests/shell/lib/*.sh '$(DESTDIR_SQ)$(perfexec_instdir_SQ)/tests/shell/lib'
+		$(INSTALL) tests/shell/lib/*.sh '$(DESTDIR_SQ)$(perfexec_instdir_SQ)/tests/shell/lib'; \
+		$(INSTALL) -d -m 755 '$(DESTDIR_SQ)$(perfexec_instdir_SQ)/tests/shell/coresight'; \
+		$(INSTALL) tests/shell/coresight/*.sh '$(DESTDIR_SQ)$(perfexec_instdir_SQ)/tests/shell/coresight'
+	$(Q)$(MAKE) -C tests/shell/coresight install-tests
 
 install-bin: install-tools install-tests install-traceevent-plugins
 
@@ -1085,7 +1096,7 @@ endif # BUILD_BPF_SKEL
 bpf-skel-clean:
 	$(call QUIET_CLEAN, bpf-skel) $(RM) -r $(SKEL_TMP_OUT) $(SKELETONS)
 
-clean:: $(LIBTRACEEVENT)-clean $(LIBAPI)-clean $(LIBBPF)-clean $(LIBSUBCMD)-clean $(LIBPERF)-clean fixdep-clean python-clean bpf-skel-clean
+clean:: $(LIBTRACEEVENT)-clean $(LIBAPI)-clean $(LIBBPF)-clean $(LIBSUBCMD)-clean $(LIBPERF)-clean fixdep-clean python-clean bpf-skel-clean tests-coresight-targets-clean
 	$(call QUIET_CLEAN, core-objs)  $(RM) $(LIBPERF_A) $(OUTPUT)perf-archive $(OUTPUT)perf-iostat $(LANG_BINDINGS)
 	$(Q)find $(or $(OUTPUT),.) -name '*.o' -delete -o -name '\.*.cmd' -delete -o -name '\.*.d' -delete
 	$(Q)$(RM) $(OUTPUT).config-detected
@@ -1143,5 +1154,6 @@ FORCE:
 .PHONY: shell_compatibility_test please_set_SHELL_PATH_to_a_more_modern_shell
 .PHONY: .FORCE-PERF-VERSION-FILE TAGS tags cscope FORCE prepare
 .PHONY: libtraceevent_plugins archheaders
+.PHONY: $(TESTS_CORESIGHT_TARGETS)
 
 endif # force_fixdep
diff --git a/tools/perf/tests/shell/coresight/Makefile b/tools/perf/tests/shell/coresight/Makefile
new file mode 100644
index 000000000000..3b816bb4ced3
--- /dev/null
+++ b/tools/perf/tests/shell/coresight/Makefile
@@ -0,0 +1,26 @@
+# SPDX-License-Identifier: GPL-2.0-only
+# Carsten Haitzler <carsten.haitzler@arm.com>, 2021
+include ../../../../../tools/scripts/Makefile.include
+include ../../../../../tools/scripts/Makefile.arch
+include ../../../../../tools/scripts/utilities.mak
+
+SUBDIRS =
+
+all: $(SUBDIRS)
+$(SUBDIRS):
+	$(Q)$(MAKE) -C $@
+
+INSTALLDIRS = $(SUBDIRS:%=install-%)
+
+install-tests: $(INSTALLDIRS)
+$(INSTALLDIRS):
+	$(Q)$(MAKE) -C $(@:install-%=%) install-tests
+
+CLEANDIRS = $(SUBDIRS:%=clean-%)
+
+clean: $(CLEANDIRS)
+$(CLEANDIRS):
+	$(Q)$(MAKE) -C $(@:clean-%=%) clean >/dev/null
+
+.PHONY: all clean $(SUBDIRS) $(CLEANDIRS) $(INSTALLDIRS)
+
diff --git a/tools/perf/tests/shell/coresight/Makefile.miniconfig b/tools/perf/tests/shell/coresight/Makefile.miniconfig
new file mode 100644
index 000000000000..a65482d769ab
--- /dev/null
+++ b/tools/perf/tests/shell/coresight/Makefile.miniconfig
@@ -0,0 +1,24 @@
+# SPDX-License-Identifier: GPL-2.0-only
+# Carsten Haitzler <carsten.haitzler@arm.com>, 2021
+
+ifndef DESTDIR
+prefix ?= $(HOME)
+endif
+
+DESTDIR_SQ = $(subst ','\'',$(DESTDIR))
+perfexecdir = libexec/perf-core
+perfexec_instdir = $(perfexecdir)
+
+ifneq ($(filter /%,$(firstword $(perfexecdir))),)
+perfexec_instdir = $(perfexecdir)
+else
+perfexec_instdir = $(prefix)/$(perfexecdir)
+endif
+
+perfexec_instdir_SQ = $(subst ','\'',$(perfexec_instdir))
+INSTALL = install
+INSTDIR_SUB = tests/shell/coresight
+
+include ../../../../../scripts/Makefile.include
+include ../../../../../scripts/Makefile.arch
+include ../../../../../scripts/utilities.mak
-- 
2.32.0

[PATCH 04/14] perf test: Add asm pureloop test tool

[carsten.haitzler]

From: "Carsten Haitzler (Rasterman)" <raster@rasterman.com>

Add test tool to be driven by further test scripts. This tool is pure
arm64 ASM with no libc usage to ensure it is the same exact
binary/code every time so it can also be re-used for many uses. It
just loops for a given fixed number of loops.

Signed-off-by: Carsten Haitzler <carsten.haitzler@arm.com>
---
 tools/perf/tests/shell/coresight/Makefile     |  3 +-
 .../shell/coresight/asm_pure_loop/.gitignore  |  1 +
 .../shell/coresight/asm_pure_loop/Makefile    | 34 +++++++++++++++++++
 .../coresight/asm_pure_loop/asm_pure_loop.S   | 28 +++++++++++++++
 4 files changed, 65 insertions(+), 1 deletion(-)
 create mode 100644 tools/perf/tests/shell/coresight/asm_pure_loop/.gitignore
 create mode 100644 tools/perf/tests/shell/coresight/asm_pure_loop/Makefile
 create mode 100644 tools/perf/tests/shell/coresight/asm_pure_loop/asm_pure_loop.S

diff --git a/tools/perf/tests/shell/coresight/Makefile b/tools/perf/tests/shell/coresight/Makefile
index 3b816bb4ced3..d4f868d55773 100644
--- a/tools/perf/tests/shell/coresight/Makefile
+++ b/tools/perf/tests/shell/coresight/Makefile
@@ -4,7 +4,8 @@ include ../../../../../tools/scripts/Makefile.include
 include ../../../../../tools/scripts/Makefile.arch
 include ../../../../../tools/scripts/utilities.mak
 
-SUBDIRS =
+SUBDIRS = \
+	asm_pure_loop
 
 all: $(SUBDIRS)
 $(SUBDIRS):
diff --git a/tools/perf/tests/shell/coresight/asm_pure_loop/.gitignore b/tools/perf/tests/shell/coresight/asm_pure_loop/.gitignore
new file mode 100644
index 000000000000..468673ac32e8
--- /dev/null
+++ b/tools/perf/tests/shell/coresight/asm_pure_loop/.gitignore
@@ -0,0 +1 @@
+asm_pure_loop
diff --git a/tools/perf/tests/shell/coresight/asm_pure_loop/Makefile b/tools/perf/tests/shell/coresight/asm_pure_loop/Makefile
new file mode 100644
index 000000000000..206849e92bc9
--- /dev/null
+++ b/tools/perf/tests/shell/coresight/asm_pure_loop/Makefile
@@ -0,0 +1,34 @@
+# SPDX-License-Identifier: GPL-2.0
+# Carsten Haitzler <carsten.haitzler@arm.com>, 2021
+
+include ../Makefile.miniconfig
+
+# Binary to produce
+BIN=asm_pure_loop
+# Any linking/libraries needed for the binary - empty if none needed
+LIB=
+
+all: $(BIN)
+
+$(BIN): $(BIN).S
+ifdef CORESIGHT
+ifeq ($(ARCH),arm64)
+# Build line - this is raw asm with no libc to have an always exact binary
+	$(Q)$(CC) $(BIN).S -nostdlib -static -o $(BIN) $(LIB)
+endif
+endif
+
+install-tests: all
+ifdef CORESIGHT
+ifeq ($(ARCH),arm64)
+# Install the test tool in the right place
+	$(call QUIET_INSTALL, tests) \
+		$(INSTALL) -d -m 755 '$(DESTDIR_SQ)$(perfexec_instdir_SQ)/$(INSTDIR_SUB)/$(BIN)'; \
+		$(INSTALL) $(BIN) '$(DESTDIR_SQ)$(perfexec_instdir_SQ)/$(INSTDIR_SUB)/$(BIN)/$(BIN)'
+endif
+endif
+
+clean:
+	$(Q)$(RM) -f $(BIN)
+
+.PHONY: all clean install-tests
diff --git a/tools/perf/tests/shell/coresight/asm_pure_loop/asm_pure_loop.S b/tools/perf/tests/shell/coresight/asm_pure_loop/asm_pure_loop.S
new file mode 100644
index 000000000000..75cf084a927d
--- /dev/null
+++ b/tools/perf/tests/shell/coresight/asm_pure_loop/asm_pure_loop.S
@@ -0,0 +1,28 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Tamas Zsoldos <tamas.zsoldos@arm.com>, 2021 */
+
+.globl _start
+_start:
+	mov	x0, 0x0000ffff
+	mov	x1, xzr
+loop:
+	nop
+	nop
+	cbnz	x1, noskip
+	nop
+	nop
+	adrp	x2, skip
+	add 	x2, x2, :lo12:skip
+	br	x2
+	nop
+	nop
+noskip:
+	nop
+	nop
+skip:
+	sub	x0, x0, 1
+	cbnz	x0, loop
+
+	mov	x0, #0
+	mov	x8, #93 // __NR_exit syscall
+	svc	#0
-- 
2.32.0

[PATCH 05/14] perf test: Add asm pureloop test shell script

[carsten.haitzler]

From: "Carsten Haitzler (Rasterman)" <raster@rasterman.com>

Add a script to drive the asm pureloop test for arm64/CoreSight that
gathers data so it passes a minimum bar for amount and quality of
content that we extract from the kernel's perf support.

Signed-off-by: Carsten Haitzler <carsten.haitzler@arm.com>
---
 .../tests/shell/coresight/asm_pure_loop.sh     | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)
 create mode 100755 tools/perf/tests/shell/coresight/asm_pure_loop.sh

diff --git a/tools/perf/tests/shell/coresight/asm_pure_loop.sh b/tools/perf/tests/shell/coresight/asm_pure_loop.sh
new file mode 100755
index 000000000000..569e9d46162b
--- /dev/null
+++ b/tools/perf/tests/shell/coresight/asm_pure_loop.sh
@@ -0,0 +1,18 @@
+#!/bin/sh -e
+# CoreSight / ASM Pure Loop
+
+# SPDX-License-Identifier: GPL-2.0
+# Carsten Haitzler <carsten.haitzler@arm.com>, 2021
+
+TEST="asm_pure_loop"
+. $(dirname $0)/../lib/coresight.sh
+ARGS=""
+DATV="out"
+DATA="$DATD/perf-$TEST-$DATV.data"
+
+perf record $PERFRECOPT -o "$DATA" "$BIN" $ARGS
+
+perf_dump_aux_verify "$DATA" 10 10 10
+
+err=$?
+exit $err
-- 
2.32.0

[PATCH 06/14] perf test: Add git ignore for perf data generated by the CoreSight tests

[carsten.haitzler]

From: "Carsten Haitzler (Rasterman)" <raster@rasterman.com>

Ignore perf output data files generated by perf tests for cleaner
git status.

Signed-off-by: Carsten Haitzler <carsten.haitzler@arm.com>
---
 tools/perf/.gitignore | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tools/perf/.gitignore b/tools/perf/.gitignore
index 4b9c71faa01a..faa23b5d32f5 100644
--- a/tools/perf/.gitignore
+++ b/tools/perf/.gitignore
@@ -15,8 +15,8 @@ perf*.1
 perf*.xml
 perf*.html
 common-cmds.h
-perf.data
-perf.data.old
+perf*.data
+perf*.data.old
 output.svg
 perf-archive
 perf-iostat
-- 
2.32.0

[PATCH 07/14] perf test: Add memcpy thread test tool

[carsten.haitzler]

From: "Carsten Haitzler (Rasterman)" <raster@rasterman.com>

Add test tool to be driven by further test scripts. This is a simple C
based memcpy with threads test to drive from scripts.

Signed-off-by: Carsten Haitzler <carsten.haitzler@arm.com>
---
 tools/perf/tests/shell/coresight/Makefile     |  3 +-
 .../shell/coresight/memcpy_thread/.gitignore  |  1 +
 .../shell/coresight/memcpy_thread/Makefile    | 33 ++++++++
 .../coresight/memcpy_thread/memcpy_thread.c   | 79 +++++++++++++++++++
 4 files changed, 115 insertions(+), 1 deletion(-)
 create mode 100644 tools/perf/tests/shell/coresight/memcpy_thread/.gitignore
 create mode 100644 tools/perf/tests/shell/coresight/memcpy_thread/Makefile
 create mode 100644 tools/perf/tests/shell/coresight/memcpy_thread/memcpy_thread.c

diff --git a/tools/perf/tests/shell/coresight/Makefile b/tools/perf/tests/shell/coresight/Makefile
index d4f868d55773..561c807022ec 100644
--- a/tools/perf/tests/shell/coresight/Makefile
+++ b/tools/perf/tests/shell/coresight/Makefile
@@ -5,7 +5,8 @@ include ../../../../../tools/scripts/Makefile.arch
 include ../../../../../tools/scripts/utilities.mak
 
 SUBDIRS = \
-	asm_pure_loop
+	asm_pure_loop \
+	memcpy_thread
 
 all: $(SUBDIRS)
 $(SUBDIRS):
diff --git a/tools/perf/tests/shell/coresight/memcpy_thread/.gitignore b/tools/perf/tests/shell/coresight/memcpy_thread/.gitignore
new file mode 100644
index 000000000000..f8217e56091e
--- /dev/null
+++ b/tools/perf/tests/shell/coresight/memcpy_thread/.gitignore
@@ -0,0 +1 @@
+memcpy_thread
diff --git a/tools/perf/tests/shell/coresight/memcpy_thread/Makefile b/tools/perf/tests/shell/coresight/memcpy_thread/Makefile
new file mode 100644
index 000000000000..2db637eb2c26
--- /dev/null
+++ b/tools/perf/tests/shell/coresight/memcpy_thread/Makefile
@@ -0,0 +1,33 @@
+# SPDX-License-Identifier: GPL-2.0
+# Carsten Haitzler <carsten.haitzler@arm.com>, 2021
+include ../Makefile.miniconfig
+
+# Binary to produce
+BIN=memcpy_thread
+# Any linking/libraries needed for the binary - empty if none needed
+LIB=-pthread
+
+all: $(BIN)
+
+$(BIN): $(BIN).c
+ifdef CORESIGHT
+ifeq ($(ARCH),arm64)
+# Build line
+	$(Q)$(CC) $(BIN).c -o $(BIN) $(LIB)
+endif
+endif
+
+install-tests: all
+ifdef CORESIGHT
+ifeq ($(ARCH),arm64)
+# Install the test tool in the right place
+	$(call QUIET_INSTALL, tests) \
+		$(INSTALL) -d -m 755 '$(DESTDIR_SQ)$(perfexec_instdir_SQ)/$(INSTDIR_SUB)/$(BIN)'; \
+		$(INSTALL) $(BIN) '$(DESTDIR_SQ)$(perfexec_instdir_SQ)/$(INSTDIR_SUB)/$(BIN)/$(BIN)'
+endif
+endif
+
+clean:
+	$(Q)$(RM) -f $(BIN)
+
+.PHONY: all clean install-tests
diff --git a/tools/perf/tests/shell/coresight/memcpy_thread/memcpy_thread.c b/tools/perf/tests/shell/coresight/memcpy_thread/memcpy_thread.c
new file mode 100644
index 000000000000..a7e169d1bf64
--- /dev/null
+++ b/tools/perf/tests/shell/coresight/memcpy_thread/memcpy_thread.c
@@ -0,0 +1,79 @@
+// SPDX-License-Identifier: GPL-2.0
+// Carsten Haitzler <carsten.haitzler@arm.com>, 2021
+#include <stdio.h>
+#include <stdlib.h>
+#include <unistd.h>
+#include <string.h>
+#include <pthread.h>
+
+struct args {
+	unsigned long loops;
+	unsigned long size;
+	pthread_t th;
+	void *ret;
+};
+
+static void *thrfn(void *arg)
+{
+	struct args *a = arg;
+	unsigned long i, len = a->loops;
+	unsigned char *src, *dst;
+
+	src = malloc(a->size * 1024);
+	dst = malloc(a->size * 1024);
+	if ((!src) || (!dst)) {
+		printf("ERR: Can't allocate memory\n");
+		exit(1);
+	}
+	for (i = 0; i < len; i++)
+		memcpy(dst, src, a->size * 1024);
+}
+
+static pthread_t new_thr(void *(*fn) (void *arg), void *arg)
+{
+	pthread_t t;
+	pthread_attr_t attr;
+
+	pthread_attr_init(&attr);
+	pthread_create(&t, &attr, fn, arg);
+	return t;
+}
+
+int main(int argc, char **argv)
+{
+	unsigned long i, len, size, thr;
+	pthread_t threads[256];
+	struct args args[256];
+	long long v;
+
+	if (argc < 4) {
+		printf("ERR: %s [copysize Kb] [numthreads] [numloops (hundreds)]\n", argv[0]);
+		exit(1);
+	}
+
+	v = atoll(argv[1]);
+	if ((v < 1) || (v > (1024 * 1024))) {
+		printf("ERR: max memory 1GB (1048576 KB)\n");
+		exit(1);
+	}
+	size = v;
+	thr = atol(argv[2]);
+	if ((thr < 1) || (thr > 256)) {
+		printf("ERR: threads 1-256\n");
+		exit(1);
+	}
+	v = atoll(argv[3]);
+	if ((v < 1) || (v > 40000000000ll)) {
+		printf("ERR: loops 1-40000000000 (hundreds)\n");
+		exit(1);
+	}
+	len = v * 100;
+	for (i = 0; i < thr; i++) {
+		args[i].loops = len;
+		args[i].size = size;
+		args[i].th = new_thr(thrfn, &(args[i]));
+	}
+	for (i = 0; i < thr; i++)
+		pthread_join(args[i].th, &(args[i].ret));
+	return 0;
+}
-- 
2.32.0

[PATCH 08/14] perf test: Add memcpy thread test shell script

[carsten.haitzler]

From: "Carsten Haitzler (Rasterman)" <raster@rasterman.com>

Add a script to drive the threaded memcpy test that gathers data so
it passes a minimum bar for amount and quality of content that we
extract from the kernel's perf support.

Signed-off-by: Carsten Haitzler <carsten.haitzler@arm.com>
---
 .../shell/coresight/memcpy_thread_16k_10.sh    | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)
 create mode 100755 tools/perf/tests/shell/coresight/memcpy_thread_16k_10.sh

diff --git a/tools/perf/tests/shell/coresight/memcpy_thread_16k_10.sh b/tools/perf/tests/shell/coresight/memcpy_thread_16k_10.sh
new file mode 100755
index 000000000000..d21ba8545938
--- /dev/null
+++ b/tools/perf/tests/shell/coresight/memcpy_thread_16k_10.sh
@@ -0,0 +1,18 @@
+#!/bin/sh -e
+# CoreSight / Memcpy 16k 10 Threads
+
+# SPDX-License-Identifier: GPL-2.0
+# Carsten Haitzler <carsten.haitzler@arm.com>, 2021
+
+TEST="memcpy_thread"
+. $(dirname $0)/../lib/coresight.sh
+ARGS="16 10 1"
+DATV="16k_10"
+DATA="$DATD/perf-$TEST-$DATV.data"
+
+perf record $PERFRECOPT -o "$DATA" "$BIN" $ARGS
+
+perf_dump_aux_verify "$DATA" 10 10 10
+
+err=$?
+exit $err
-- 
2.32.0

Re: [PATCH 08/14] perf test: Add memcpy thread test shell script

[James Clark]

On 01/07/2022 13:07, carsten.haitzler@foss.arm.com wrote:
> From: "Carsten Haitzler (Rasterman)" <raster@rasterman.com>
> 
> Add a script to drive the threaded memcpy test that gathers data so
> it passes a minimum bar for amount and quality of content that we
> extract from the kernel's perf support.
> 

On this one I get a failure about 1/50 times on N1SDP (I ran it about 150
times and saw 3 failures so it's quite consistent). Usually it records
about a 1.4MB file with one aux record. But when it fails the file is
only 20K and has one small aux record:

   0 0 0x1a10 [0x30]: PERF_RECORD_AUXTRACE size: 0x1820  offset: 0  ref: 0x1c23126d7ff3d2ab  idx: 3  tid: 682799  cpu: 3

Nothing was dropped, and the load on the system wasn't any different
to when it passes. So I'm not sure if this is a real coresight bug
or that the test is flaky. There was a bug in SPE before where
threads weren't followed after forking, but only very rarely. It feels
a bit like that.

It could also be some contention issue because 10 threads are launched
but the machine only has 4 cores.

The failure message from the test looks like this:

   77: CoreSight / Memcpy 16k 10 Threads                               :
   --- start ---
   Couldn't synthesize bpf events.
   [ perf record: Woken up 1 times to write data ]
   [ perf record: Captured and wrote 0.012 MB ./perf-memcpy_thread-16k_10.data ]
   Sanity check number of ASYNC is too low (3 < 10)
    ---- end ----
   CoreSight / Memcpy 16k 10 Threads: FAILED!

I didn't see this issue on any of the other tests. Sometimes very small
files were made if I loaded the system, but the tests still passed.

Thanks
James

> Signed-off-by: Carsten Haitzler <carsten.haitzler@arm.com>
> ---
>  .../shell/coresight/memcpy_thread_16k_10.sh    | 18 ++++++++++++++++++
>  1 file changed, 18 insertions(+)
>  create mode 100755 tools/perf/tests/shell/coresight/memcpy_thread_16k_10.sh
> 
> diff --git a/tools/perf/tests/shell/coresight/memcpy_thread_16k_10.sh b/tools/perf/tests/shell/coresight/memcpy_thread_16k_10.sh
> new file mode 100755
> index 000000000000..d21ba8545938
> --- /dev/null
> +++ b/tools/perf/tests/shell/coresight/memcpy_thread_16k_10.sh
> @@ -0,0 +1,18 @@
> +#!/bin/sh -e
> +# CoreSight / Memcpy 16k 10 Threads
> +
> +# SPDX-License-Identifier: GPL-2.0
> +# Carsten Haitzler <carsten.haitzler@arm.com>, 2021
> +
> +TEST="memcpy_thread"
> +. $(dirname $0)/../lib/coresight.sh
> +ARGS="16 10 1"
> +DATV="16k_10"
> +DATA="$DATD/perf-$TEST-$DATV.data"
> +
> +perf record $PERFRECOPT -o "$DATA" "$BIN" $ARGS
> +
> +perf_dump_aux_verify "$DATA" 10 10 10
> +
> +err=$?
> +exit $err

Re: [PATCH 08/14] perf test: Add memcpy thread test shell script

[James Clark]

On 05/07/2022 15:25, James Clark wrote:
> 
> 
> On 01/07/2022 13:07, carsten.haitzler@foss.arm.com wrote:
>> From: "Carsten Haitzler (Rasterman)" <raster@rasterman.com>
>>
>> Add a script to drive the threaded memcpy test that gathers data so
>> it passes a minimum bar for amount and quality of content that we
>> extract from the kernel's perf support.
>>
> 
> On this one I get a failure about 1/50 times on N1SDP (I ran it about 150
> times and saw 3 failures so it's quite consistent). Usually it records
> about a 1.4MB file with one aux record. But when it fails the file is
> only 20K and has one small aux record:
> 
>    0 0 0x1a10 [0x30]: PERF_RECORD_AUXTRACE size: 0x1820  offset: 0  ref: 0x1c23126d7ff3d2ab  idx: 3  tid: 682799  cpu: 3
> 
> Nothing was dropped, and the load on the system wasn't any different
> to when it passes. So I'm not sure if this is a real coresight bug
> or that the test is flaky. There was a bug in SPE before where
> threads weren't followed after forking, but only very rarely. It feels
> a bit like that.
> 
> It could also be some contention issue because 10 threads are launched
> but the machine only has 4 cores.
> 
> The failure message from the test looks like this:
> 
>    77: CoreSight / Memcpy 16k 10 Threads                               :
>    --- start ---
>    Couldn't synthesize bpf events.
>    [ perf record: Woken up 1 times to write data ]
>    [ perf record: Captured and wrote 0.012 MB ./perf-memcpy_thread-16k_10.data ]
>    Sanity check number of ASYNC is too low (3 < 10)
>     ---- end ----
>    CoreSight / Memcpy 16k 10 Threads: FAILED!
> 
> I didn't see this issue on any of the other tests. Sometimes very small
> files were made if I loaded the system, but the tests still passed.

Spoke too soon, same thing on another test with an unloaded system. It's
just a bit more rare:

  80: CoreSight / Unroll Loop Thread 10                               :
  --- start ---
  Couldn't synthesize bpf events.
  [ perf record: Woken up 1 times to write data ]
  [ perf record: Captured and wrote 0.041 MB ./perf-unroll_loop_thread-10.data ]
  Sanity check number of ASYNC is too low (6 < 10)
  ---- end ----
  CoreSight / Unroll Loop Thread 10: FAILED!

> 
> Thanks
> James
> 
>> Signed-off-by: Carsten Haitzler <carsten.haitzler@arm.com>
>> ---
>>  .../shell/coresight/memcpy_thread_16k_10.sh    | 18 ++++++++++++++++++
>>  1 file changed, 18 insertions(+)
>>  create mode 100755 tools/perf/tests/shell/coresight/memcpy_thread_16k_10.sh
>>
>> diff --git a/tools/perf/tests/shell/coresight/memcpy_thread_16k_10.sh b/tools/perf/tests/shell/coresight/memcpy_thread_16k_10.sh
>> new file mode 100755
>> index 000000000000..d21ba8545938
>> --- /dev/null
>> +++ b/tools/perf/tests/shell/coresight/memcpy_thread_16k_10.sh
>> @@ -0,0 +1,18 @@
>> +#!/bin/sh -e
>> +# CoreSight / Memcpy 16k 10 Threads
>> +
>> +# SPDX-License-Identifier: GPL-2.0
>> +# Carsten Haitzler <carsten.haitzler@arm.com>, 2021
>> +
>> +TEST="memcpy_thread"
>> +. $(dirname $0)/../lib/coresight.sh
>> +ARGS="16 10 1"
>> +DATV="16k_10"
>> +DATA="$DATD/perf-$TEST-$DATV.data"
>> +
>> +perf record $PERFRECOPT -o "$DATA" "$BIN" $ARGS
>> +
>> +perf_dump_aux_verify "$DATA" 10 10 10
>> +
>> +err=$?
>> +exit $err
> _______________________________________________
> CoreSight mailing list -- coresight@lists.linaro.org
> To unsubscribe send an email to coresight-leave@lists.linaro.org

Re: [PATCH 08/14] perf test: Add memcpy thread test shell script

[Carsten Haitzler]

On 7/5/22 15:25, James Clark wrote:
> 
> 
> On 01/07/2022 13:07, carsten.haitzler@foss.arm.com wrote:
>> From: "Carsten Haitzler (Rasterman)" <raster@rasterman.com>
>>
>> Add a script to drive the threaded memcpy test that gathers data so
>> it passes a minimum bar for amount and quality of content that we
>> extract from the kernel's perf support.
>>
> 
> On this one I get a failure about 1/50 times on N1SDP (I ran it about 150

I also see inconsistent results. The whole point of these tests is to 
point this out and provide data to track it and then lead eventually to 
improvements/fixes. A failing test is probably good - it found a 
problem. Perf test for me has lots of failures so I'm taking the 
position that failures are OK normally in perf test as long as you know 
what those failures are and why.

> times and saw 3 failures so it's quite consistent). Usually it records
> about a 1.4MB file with one aux record. But when it fails the file is
> only 20K and has one small aux record:
> 
>     0 0 0x1a10 [0x30]: PERF_RECORD_AUXTRACE size: 0x1820  offset: 0  ref: 0x1c23126d7ff3d2ab  idx: 3  tid: 682799  cpu: 3
> 
> Nothing was dropped, and the load on the system wasn't any different
> to when it passes. So I'm not sure if this is a real coresight bug
> or that the test is flaky. There was a bug in SPE before where

The binary is the same with the same content running the same perf 
command every time. Workload doesn't change. The perf data captured does 
change. It sometimes captures so little it fails even the low pass bar 
given in the test.

> threads weren't followed after forking, but only very rarely. It feels
> a bit like that.

That ... would be a "CoreSight" bug though I think, not the test.

> It could also be some contention issue because 10 threads are launched
> but the machine only has 4 cores.

We still should be capturing data reliably (in theory). If you have 10 
threads on a 4 core machine it'll take longer to run for the same 
workload as the threads will have to share the same cores, but this 
should still result in decent data collection as the cores switch 
between threads. That's the point.

> The failure message from the test looks like this:
> 
>     77: CoreSight / Memcpy 16k 10 Threads                               :
>     --- start ---
>     Couldn't synthesize bpf events.
>     [ perf record: Woken up 1 times to write data ]
>     [ perf record: Captured and wrote 0.012 MB ./perf-memcpy_thread-16k_10.data ]
>     Sanity check number of ASYNC is too low (3 < 10)
>      ---- end ----
>     CoreSight / Memcpy 16k 10 Threads: FAILED!
> 
> I didn't see this issue on any of the other tests. Sometimes very small
> files were made if I loaded the system, but the tests still passed.

For me the "Check TID" tests fails very often... but as I said - the 
point here is to find issues and ensure they are reported in results. 
The test even track the results over time/many runs in the csv files so 
you get a good idea of consistency and even how it may statistically 
change over time matching that up to changes in the kernel.

Unless of course you think it's acceptable that sometimes perf record + 
CoreSight will output essentially no data (your 20k example). :)

> Thanks
> James
> 
>> Signed-off-by: Carsten Haitzler <carsten.haitzler@arm.com>
>> ---
>>   .../shell/coresight/memcpy_thread_16k_10.sh    | 18 ++++++++++++++++++
>>   1 file changed, 18 insertions(+)
>>   create mode 100755 tools/perf/tests/shell/coresight/memcpy_thread_16k_10.sh
>>
>> diff --git a/tools/perf/tests/shell/coresight/memcpy_thread_16k_10.sh b/tools/perf/tests/shell/coresight/memcpy_thread_16k_10.sh
>> new file mode 100755
>> index 000000000000..d21ba8545938
>> --- /dev/null
>> +++ b/tools/perf/tests/shell/coresight/memcpy_thread_16k_10.sh
>> @@ -0,0 +1,18 @@
>> +#!/bin/sh -e
>> +# CoreSight / Memcpy 16k 10 Threads
>> +
>> +# SPDX-License-Identifier: GPL-2.0
>> +# Carsten Haitzler <carsten.haitzler@arm.com>, 2021
>> +
>> +TEST="memcpy_thread"
>> +. $(dirname $0)/../lib/coresight.sh
>> +ARGS="16 10 1"
>> +DATV="16k_10"
>> +DATA="$DATD/perf-$TEST-$DATV.data"
>> +
>> +perf record $PERFRECOPT -o "$DATA" "$BIN" $ARGS
>> +
>> +perf_dump_aux_verify "$DATA" 10 10 10
>> +
>> +err=$?
>> +exit $err

[PATCH 09/14] perf test: Add thread loop test tool

[carsten.haitzler]

From: "Carsten Haitzler (Rasterman)" <raster@rasterman.com>

Add test tool to be driven by further test scripts. This is a simple C
based loop with threads test to drive from scripts that can output TIDs
for tracking/checking.

Signed-off-by: Carsten Haitzler <carsten.haitzler@arm.com>
---
 tools/perf/tests/shell/coresight/Makefile     |  3 +-
 .../shell/coresight/thread_loop/.gitignore    |  1 +
 .../shell/coresight/thread_loop/Makefile      | 33 +++++++
 .../shell/coresight/thread_loop/thread_loop.c | 86 +++++++++++++++++++
 4 files changed, 122 insertions(+), 1 deletion(-)
 create mode 100644 tools/perf/tests/shell/coresight/thread_loop/.gitignore
 create mode 100644 tools/perf/tests/shell/coresight/thread_loop/Makefile
 create mode 100644 tools/perf/tests/shell/coresight/thread_loop/thread_loop.c

diff --git a/tools/perf/tests/shell/coresight/Makefile b/tools/perf/tests/shell/coresight/Makefile
index 561c807022ec..004974a71fb8 100644
--- a/tools/perf/tests/shell/coresight/Makefile
+++ b/tools/perf/tests/shell/coresight/Makefile
@@ -6,7 +6,8 @@ include ../../../../../tools/scripts/utilities.mak
 
 SUBDIRS = \
 	asm_pure_loop \
-	memcpy_thread
+	memcpy_thread \
+	thread_loop
 
 all: $(SUBDIRS)
 $(SUBDIRS):
diff --git a/tools/perf/tests/shell/coresight/thread_loop/.gitignore b/tools/perf/tests/shell/coresight/thread_loop/.gitignore
new file mode 100644
index 000000000000..6d4c33eaa9e8
--- /dev/null
+++ b/tools/perf/tests/shell/coresight/thread_loop/.gitignore
@@ -0,0 +1 @@
+thread_loop
diff --git a/tools/perf/tests/shell/coresight/thread_loop/Makefile b/tools/perf/tests/shell/coresight/thread_loop/Makefile
new file mode 100644
index 000000000000..ea846c038e7a
--- /dev/null
+++ b/tools/perf/tests/shell/coresight/thread_loop/Makefile
@@ -0,0 +1,33 @@
+# SPDX-License-Identifier: GPL-2.0
+# Carsten Haitzler <carsten.haitzler@arm.com>, 2021
+include ../Makefile.miniconfig
+
+# Binary to produce
+BIN=thread_loop
+# Any linking/libraries needed for the binary - empty if none needed
+LIB=-pthread
+
+all: $(BIN)
+
+$(BIN): $(BIN).c
+ifdef CORESIGHT
+ifeq ($(ARCH),arm64)
+# Build line
+	$(Q)$(CC) $(BIN).c -o $(BIN) $(LIB)
+endif
+endif
+
+install-tests: all
+ifdef CORESIGHT
+ifeq ($(ARCH),arm64)
+# Install the test tool in the right place
+	$(call QUIET_INSTALL, tests) \
+		$(INSTALL) -d -m 755 '$(DESTDIR_SQ)$(perfexec_instdir_SQ)/$(INSTDIR_SUB)/$(BIN)'; \
+		$(INSTALL) $(BIN) '$(DESTDIR_SQ)$(perfexec_instdir_SQ)/$(INSTDIR_SUB)/$(BIN)/$(BIN)'
+endif
+endif
+
+clean:
+	$(Q)$(RM) -f $(BIN)
+
+.PHONY: all clean install-tests
diff --git a/tools/perf/tests/shell/coresight/thread_loop/thread_loop.c b/tools/perf/tests/shell/coresight/thread_loop/thread_loop.c
new file mode 100644
index 000000000000..c0158fac7d0b
--- /dev/null
+++ b/tools/perf/tests/shell/coresight/thread_loop/thread_loop.c
@@ -0,0 +1,86 @@
+// SPDX-License-Identifier: GPL-2.0
+// Carsten Haitzler <carsten.haitzler@arm.com>, 2021
+
+// define this for gettid()
+#define _GNU_SOURCE
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <unistd.h>
+#include <string.h>
+#include <pthread.h>
+#include <sys/syscall.h>
+#ifndef SYS_gettid
+// gettid is 178 on arm64
+# define SYS_gettid 178
+#endif
+#define gettid() syscall(SYS_gettid)
+
+struct args {
+	unsigned int loops;
+	pthread_t th;
+	void *ret;
+};
+
+static void *thrfn(void *arg)
+{
+	struct args *a = arg;
+	int i = 0, len = a->loops;
+
+	if (getenv("SHOW_TID")) {
+		unsigned long long tid = gettid();
+
+		printf("%llu\n", tid);
+	}
+	asm volatile(
+		"loop:\n"
+		"add %[i], %[i], #1\n"
+		"cmp %[i], %[len]\n"
+		"blt loop\n"
+		: /* out */
+		: /* in */ [i] "r" (i), [len] "r" (len)
+		: /* clobber */
+	);
+	return (void *)(long)i;
+}
+
+static pthread_t new_thr(void *(*fn) (void *arg), void *arg)
+{
+	pthread_t t;
+	pthread_attr_t attr;
+
+	pthread_attr_init(&attr);
+	pthread_create(&t, &attr, fn, arg);
+	return t;
+}
+
+int main(int argc, char **argv)
+{
+	unsigned int i, len, thr;
+	pthread_t threads[256];
+	struct args args[256];
+
+	if (argc < 3) {
+		printf("ERR: %s [numthreads] [numloops (millions)]\n", argv[0]);
+		exit(1);
+	}
+
+	thr = atoi(argv[1]);
+	if ((thr < 1) || (thr > 256)) {
+		printf("ERR: threads 1-256\n");
+		exit(1);
+	}
+	len = atoi(argv[2]);
+	if ((len < 1) || (len > 4000)) {
+		printf("ERR: max loops 4000 (millions)\n");
+		exit(1);
+	}
+	len *= 1000000;
+	for (i = 0; i < thr; i++) {
+		args[i].loops = len;
+		args[i].th = new_thr(thrfn, &(args[i]));
+	}
+	for (i = 0; i < thr; i++)
+		pthread_join(args[i].th, &(args[i].ret));
+	return 0;
+}
-- 
2.32.0

[PATCH 10/14] perf test: Add thread loop test shell scripts

[carsten.haitzler]

From: "Carsten Haitzler (Rasterman)" <raster@rasterman.com>

Add a script to drive the thread loop test that gathers data so
it passes a minimum bar (in this case do we get any perf context data
for every thread).

Signed-off-by: Carsten Haitzler <carsten.haitzler@arm.com>
---
 .../coresight/thread_loop_check_tid_10.sh     | 19 +++++++++++++++++++
 .../coresight/thread_loop_check_tid_2.sh      | 19 +++++++++++++++++++
 2 files changed, 38 insertions(+)
 create mode 100755 tools/perf/tests/shell/coresight/thread_loop_check_tid_10.sh
 create mode 100755 tools/perf/tests/shell/coresight/thread_loop_check_tid_2.sh

diff --git a/tools/perf/tests/shell/coresight/thread_loop_check_tid_10.sh b/tools/perf/tests/shell/coresight/thread_loop_check_tid_10.sh
new file mode 100755
index 000000000000..7c13636fc778
--- /dev/null
+++ b/tools/perf/tests/shell/coresight/thread_loop_check_tid_10.sh
@@ -0,0 +1,19 @@
+#!/bin/sh -e
+# CoreSight / Thread Loop 10 Threads - Check TID
+
+# SPDX-License-Identifier: GPL-2.0
+# Carsten Haitzler <carsten.haitzler@arm.com>, 2021
+
+TEST="thread_loop"
+. $(dirname $0)/../lib/coresight.sh
+ARGS="10 1"
+DATV="check-tid-10th"
+DATA="$DATD/perf-$TEST-$DATV.data"
+STDO="$DATD/perf-$TEST-$DATV.stdout"
+
+SHOW_TID=1 perf record -s $PERFRECOPT -o "$DATA" "$BIN" $ARGS > $STDO
+
+perf_dump_aux_tid_verify "$DATA" "$STDO"
+
+err=$?
+exit $err
diff --git a/tools/perf/tests/shell/coresight/thread_loop_check_tid_2.sh b/tools/perf/tests/shell/coresight/thread_loop_check_tid_2.sh
new file mode 100755
index 000000000000..a067145af43c
--- /dev/null
+++ b/tools/perf/tests/shell/coresight/thread_loop_check_tid_2.sh
@@ -0,0 +1,19 @@
+#!/bin/sh -e
+# CoreSight / Thread Loop 2 Threads - Check TID
+
+# SPDX-License-Identifier: GPL-2.0
+# Carsten Haitzler <carsten.haitzler@arm.com>, 2021
+
+TEST="thread_loop"
+. $(dirname $0)/../lib/coresight.sh
+ARGS="2 20"
+DATV="check-tid-2th"
+DATA="$DATD/perf-$TEST-$DATV.data"
+STDO="$DATD/perf-$TEST-$DATV.stdout"
+
+SHOW_TID=1 perf record -s $PERFRECOPT -o "$DATA" "$BIN" $ARGS > $STDO
+
+perf_dump_aux_tid_verify "$DATA" "$STDO"
+
+err=$?
+exit $err
-- 
2.32.0

Re: [PATCH 10/14] perf test: Add thread loop test shell scripts

[James Clark]

On 01/07/2022 13:07, carsten.haitzler@foss.arm.com wrote:
> From: "Carsten Haitzler (Rasterman)" <raster@rasterman.com>
> 
> Add a script to drive the thread loop test that gathers data so
> it passes a minimum bar (in this case do we get any perf context data
> for every thread).
> 
> Signed-off-by: Carsten Haitzler <carsten.haitzler@arm.com>

Hi Carsten,

I checked this on N1SDP and I get failures in both threads tests. This is
because it's looking for "CID=..." when in my output threads are shown as
"VMID=...":

    Idx:628048; ID:10;	I_ADDR_CTXT_L_64IS0 : Address & Context, Long, 64 bit, IS0.; Addr=0x0000AAAAE3BF0B18; Ctxt: AArch64,EL0, NS; VMID=0xa588c;

I think with a change to the grep it should work.

Thanks
James

> ---
>  .../coresight/thread_loop_check_tid_10.sh     | 19 +++++++++++++++++++
>  .../coresight/thread_loop_check_tid_2.sh      | 19 +++++++++++++++++++
>  2 files changed, 38 insertions(+)
>  create mode 100755 tools/perf/tests/shell/coresight/thread_loop_check_tid_10.sh
>  create mode 100755 tools/perf/tests/shell/coresight/thread_loop_check_tid_2.sh
> 
> diff --git a/tools/perf/tests/shell/coresight/thread_loop_check_tid_10.sh b/tools/perf/tests/shell/coresight/thread_loop_check_tid_10.sh
> new file mode 100755
> index 000000000000..7c13636fc778
> --- /dev/null
> +++ b/tools/perf/tests/shell/coresight/thread_loop_check_tid_10.sh
> @@ -0,0 +1,19 @@
> +#!/bin/sh -e
> +# CoreSight / Thread Loop 10 Threads - Check TID
> +
> +# SPDX-License-Identifier: GPL-2.0
> +# Carsten Haitzler <carsten.haitzler@arm.com>, 2021
> +
> +TEST="thread_loop"
> +. $(dirname $0)/../lib/coresight.sh
> +ARGS="10 1"
> +DATV="check-tid-10th"
> +DATA="$DATD/perf-$TEST-$DATV.data"
> +STDO="$DATD/perf-$TEST-$DATV.stdout"
> +
> +SHOW_TID=1 perf record -s $PERFRECOPT -o "$DATA" "$BIN" $ARGS > $STDO
> +
> +perf_dump_aux_tid_verify "$DATA" "$STDO"
> +
> +err=$?
> +exit $err
> diff --git a/tools/perf/tests/shell/coresight/thread_loop_check_tid_2.sh b/tools/perf/tests/shell/coresight/thread_loop_check_tid_2.sh
> new file mode 100755
> index 000000000000..a067145af43c
> --- /dev/null
> +++ b/tools/perf/tests/shell/coresight/thread_loop_check_tid_2.sh
> @@ -0,0 +1,19 @@
> +#!/bin/sh -e
> +# CoreSight / Thread Loop 2 Threads - Check TID
> +
> +# SPDX-License-Identifier: GPL-2.0
> +# Carsten Haitzler <carsten.haitzler@arm.com>, 2021
> +
> +TEST="thread_loop"
> +. $(dirname $0)/../lib/coresight.sh
> +ARGS="2 20"
> +DATV="check-tid-2th"
> +DATA="$DATD/perf-$TEST-$DATV.data"
> +STDO="$DATD/perf-$TEST-$DATV.stdout"
> +
> +SHOW_TID=1 perf record -s $PERFRECOPT -o "$DATA" "$BIN" $ARGS > $STDO
> +
> +perf_dump_aux_tid_verify "$DATA" "$STDO"
> +
> +err=$?
> +exit $err

Re: [PATCH 10/14] perf test: Add thread loop test shell scripts

[Carsten Haitzler]

On 7/5/22 14:53, James Clark wrote:
> 
> 
> On 01/07/2022 13:07, carsten.haitzler@foss.arm.com wrote:
>> From: "Carsten Haitzler (Rasterman)" <raster@rasterman.com>
>>
>> Add a script to drive the thread loop test that gathers data so
>> it passes a minimum bar (in this case do we get any perf context data
>> for every thread).
>>
>> Signed-off-by: Carsten Haitzler <carsten.haitzler@arm.com>
> 
> Hi Carsten,
> 
> I checked this on N1SDP and I get failures in both threads tests. This is
> because it's looking for "CID=..." when in my output threads are shown as
> "VMID=...":
> 
>      Idx:628048; ID:10;	I_ADDR_CTXT_L_64IS0 : Address & Context, Long, 64 bit, IS0.; Addr=0x0000AAAAE3BF0B18; Ctxt: AArch64,EL0, NS; VMID=0xa588c;
> 
> I think with a change to the grep it should work.

Errrr... I get no VMID= ... it's all

Idx:563008; ID:12;	I_ADDR_CTXT_L_64IS0 : Address & Context, Long, 64 
bit, IS0.; Addr=0x0000AAAAE4B00A60; Ctxt: AArch64,EL0, NS; CID=0x00004aff;

are you using containers or something? because:

             if(context.updated_c)
             {
                 oss << "CID=0x" << std::hex << std::setfill('0') << 
std::setw(8) << context.ctxtID << "; ";
             }
             if(context.updated_v)
             {
                 oss << "VMID=0x" << std::hex << std::setfill('0') << 
std::setw(4) << context.VMID << "; ";
             }

I'm running without any containers etc. - bare metal. Haven't bothered 
with any VM stuff.

In OpenOCD the CID should be the the pid/thread id. It seems to not be 
the same thing as VMID. I haven't traced this beyond here as to exactly 
what this represents though my first reaction is "This is extra VM info 
and not the PID/TID being looked for". OpenOCD is full of tests with log 
dumps that produce CID and VMID:

Idx:1676; ID:10;        I_ADDR_CTXT_L_64IS0 : Address & Context, Long, 
64 bit, IS0.; Addr=0xFFFFFFC000096A00; Ctxt: AArch64,EL1, NS; 
CID=0x00000000; VMID=0x0000;

A quick git grep CID= in OpenCD will show them all. My understanding is 
CID is the thread/process ID and thus the test/check "Do we get reported 
data from all threads? - anything?".

I don't think using VMID is right. The fact you are missing a CID is an 
issue though...

> Thanks
> James
> 
>> ---
>>   .../coresight/thread_loop_check_tid_10.sh     | 19 +++++++++++++++++++
>>   .../coresight/thread_loop_check_tid_2.sh      | 19 +++++++++++++++++++
>>   2 files changed, 38 insertions(+)
>>   create mode 100755 tools/perf/tests/shell/coresight/thread_loop_check_tid_10.sh
>>   create mode 100755 tools/perf/tests/shell/coresight/thread_loop_check_tid_2.sh
>>
>> diff --git a/tools/perf/tests/shell/coresight/thread_loop_check_tid_10.sh b/tools/perf/tests/shell/coresight/thread_loop_check_tid_10.sh
>> new file mode 100755
>> index 000000000000..7c13636fc778
>> --- /dev/null
>> +++ b/tools/perf/tests/shell/coresight/thread_loop_check_tid_10.sh
>> @@ -0,0 +1,19 @@
>> +#!/bin/sh -e
>> +# CoreSight / Thread Loop 10 Threads - Check TID
>> +
>> +# SPDX-License-Identifier: GPL-2.0
>> +# Carsten Haitzler <carsten.haitzler@arm.com>, 2021
>> +
>> +TEST="thread_loop"
>> +. $(dirname $0)/../lib/coresight.sh
>> +ARGS="10 1"
>> +DATV="check-tid-10th"
>> +DATA="$DATD/perf-$TEST-$DATV.data"
>> +STDO="$DATD/perf-$TEST-$DATV.stdout"
>> +
>> +SHOW_TID=1 perf record -s $PERFRECOPT -o "$DATA" "$BIN" $ARGS > $STDO
>> +
>> +perf_dump_aux_tid_verify "$DATA" "$STDO"
>> +
>> +err=$?
>> +exit $err
>> diff --git a/tools/perf/tests/shell/coresight/thread_loop_check_tid_2.sh b/tools/perf/tests/shell/coresight/thread_loop_check_tid_2.sh
>> new file mode 100755
>> index 000000000000..a067145af43c
>> --- /dev/null
>> +++ b/tools/perf/tests/shell/coresight/thread_loop_check_tid_2.sh
>> @@ -0,0 +1,19 @@
>> +#!/bin/sh -e
>> +# CoreSight / Thread Loop 2 Threads - Check TID
>> +
>> +# SPDX-License-Identifier: GPL-2.0
>> +# Carsten Haitzler <carsten.haitzler@arm.com>, 2021
>> +
>> +TEST="thread_loop"
>> +. $(dirname $0)/../lib/coresight.sh
>> +ARGS="2 20"
>> +DATV="check-tid-2th"
>> +DATA="$DATD/perf-$TEST-$DATV.data"
>> +STDO="$DATD/perf-$TEST-$DATV.stdout"
>> +
>> +SHOW_TID=1 perf record -s $PERFRECOPT -o "$DATA" "$BIN" $ARGS > $STDO
>> +
>> +perf_dump_aux_tid_verify "$DATA" "$STDO"
>> +
>> +err=$?
>> +exit $err

Re: [PATCH 10/14] perf test: Add thread loop test shell scripts

[Mike Leach]

Hi,

On Fri, 8 Jul 2022 at 10:22, Carsten Haitzler
<carsten.haitzler@foss.arm.com> wrote:
>
>
>
> On 7/5/22 14:53, James Clark wrote:
> >
> >
> > On 01/07/2022 13:07, carsten.haitzler@foss.arm.com wrote:
> >> From: "Carsten Haitzler (Rasterman)" <raster@rasterman.com>
> >>
> >> Add a script to drive the thread loop test that gathers data so
> >> it passes a minimum bar (in this case do we get any perf context data
> >> for every thread).
> >>
> >> Signed-off-by: Carsten Haitzler <carsten.haitzler@arm.com>
> >
> > Hi Carsten,
> >
> > I checked this on N1SDP and I get failures in both threads tests. This is
> > because it's looking for "CID=..." when in my output threads are shown as
> > "VMID=...":
> >
> >      Idx:628048; ID:10;       I_ADDR_CTXT_L_64IS0 : Address & Context, Long, 64 bit, IS0.; Addr=0x0000AAAAE3BF0B18; Ctxt: AArch64,EL0, NS; VMID=0xa588c;
> >
> > I think with a change to the grep it should work.
>
> Errrr... I get no VMID= ... it's all
>
> Idx:563008; ID:12;      I_ADDR_CTXT_L_64IS0 : Address & Context, Long, 64
> bit, IS0.; Addr=0x0000AAAAE4B00A60; Ctxt: AArch64,EL0, NS; CID=0x00004aff;
>
> are you using containers or something? because:
>
>              if(context.updated_c)
>              {
>                  oss << "CID=0x" << std::hex << std::setfill('0') <<
> std::setw(8) << context.ctxtID << "; ";
>              }
>              if(context.updated_v)
>              {
>                  oss << "VMID=0x" << std::hex << std::setfill('0') <<
> std::setw(4) << context.VMID << "; ";
>              }
>
> I'm running without any containers etc. - bare metal. Haven't bothered
> with any VM stuff.
>
> In OpenOCD the CID should be the the pid/thread id. It seems to not be
> the same thing as VMID. I haven't traced this beyond here as to exactly
> what this represents though my first reaction is "This is extra VM info
> and not the PID/TID being looked for". OpenOCD is full of tests with log
> dumps that produce CID and VMID:
>
> Idx:1676; ID:10;        I_ADDR_CTXT_L_64IS0 : Address & Context, Long,
> 64 bit, IS0.; Addr=0xFFFFFFC000096A00; Ctxt: AArch64,EL1, NS;
> CID=0x00000000; VMID=0x0000;
>
> A quick git grep CID= in OpenCD will show them all. My understanding is
> CID is the thread/process ID and thus the test/check "Do we get reported
> data from all threads? - anything?".
>
> I don't think using VMID is right. The fact you are missing a CID is an
> issue though...
>

The register used for linux TID trace is dependent on the EL of the kernel.
EL1 => CONTEXT_IDR_EL1
EL2 => CONTEXT_IDR_EL2.

By design, the trace hardware traces CONTEXT_IDR_EL2 as the VMID packet.

So, depending on your kernel build, TID can validly be traced as CID or VMID

Regards

Mike

> > Thanks
> > James
> >
> >> ---
> >>   .../coresight/thread_loop_check_tid_10.sh     | 19 +++++++++++++++++++
> >>   .../coresight/thread_loop_check_tid_2.sh      | 19 +++++++++++++++++++
> >>   2 files changed, 38 insertions(+)
> >>   create mode 100755 tools/perf/tests/shell/coresight/thread_loop_check_tid_10.sh
> >>   create mode 100755 tools/perf/tests/shell/coresight/thread_loop_check_tid_2.sh
> >>
> >> diff --git a/tools/perf/tests/shell/coresight/thread_loop_check_tid_10.sh b/tools/perf/tests/shell/coresight/thread_loop_check_tid_10.sh
> >> new file mode 100755
> >> index 000000000000..7c13636fc778
> >> --- /dev/null
> >> +++ b/tools/perf/tests/shell/coresight/thread_loop_check_tid_10.sh
> >> @@ -0,0 +1,19 @@
> >> +#!/bin/sh -e
> >> +# CoreSight / Thread Loop 10 Threads - Check TID
> >> +
> >> +# SPDX-License-Identifier: GPL-2.0
> >> +# Carsten Haitzler <carsten.haitzler@arm.com>, 2021
> >> +
> >> +TEST="thread_loop"
> >> +. $(dirname $0)/../lib/coresight.sh
> >> +ARGS="10 1"
> >> +DATV="check-tid-10th"
> >> +DATA="$DATD/perf-$TEST-$DATV.data"
> >> +STDO="$DATD/perf-$TEST-$DATV.stdout"
> >> +
> >> +SHOW_TID=1 perf record -s $PERFRECOPT -o "$DATA" "$BIN" $ARGS > $STDO
> >> +
> >> +perf_dump_aux_tid_verify "$DATA" "$STDO"
> >> +
> >> +err=$?
> >> +exit $err
> >> diff --git a/tools/perf/tests/shell/coresight/thread_loop_check_tid_2.sh b/tools/perf/tests/shell/coresight/thread_loop_check_tid_2.sh
> >> new file mode 100755
> >> index 000000000000..a067145af43c
> >> --- /dev/null
> >> +++ b/tools/perf/tests/shell/coresight/thread_loop_check_tid_2.sh
> >> @@ -0,0 +1,19 @@
> >> +#!/bin/sh -e
> >> +# CoreSight / Thread Loop 2 Threads - Check TID
> >> +
> >> +# SPDX-License-Identifier: GPL-2.0
> >> +# Carsten Haitzler <carsten.haitzler@arm.com>, 2021
> >> +
> >> +TEST="thread_loop"
> >> +. $(dirname $0)/../lib/coresight.sh
> >> +ARGS="2 20"
> >> +DATV="check-tid-2th"
> >> +DATA="$DATD/perf-$TEST-$DATV.data"
> >> +STDO="$DATD/perf-$TEST-$DATV.stdout"
> >> +
> >> +SHOW_TID=1 perf record -s $PERFRECOPT -o "$DATA" "$BIN" $ARGS > $STDO
> >> +
> >> +perf_dump_aux_tid_verify "$DATA" "$STDO"
> >> +
> >> +err=$?
> >> +exit $err



-- 
Mike Leach
Principal Engineer, ARM Ltd.
Manchester Design Centre. UK

Re: [PATCH 10/14] perf test: Add thread loop test shell scripts

[Carsten Haitzler]

On 7/8/22 11:27, Mike Leach wrote:
> Hi,
> 
> On Fri, 8 Jul 2022 at 10:22, Carsten Haitzler
> <carsten.haitzler@foss.arm.com> wrote:
>>
>>
>>
>> On 7/5/22 14:53, James Clark wrote:
>>>
>>>
>>> On 01/07/2022 13:07, carsten.haitzler@foss.arm.com wrote:
>>>> From: "Carsten Haitzler (Rasterman)" <raster@rasterman.com>
>>>>
>>>> Add a script to drive the thread loop test that gathers data so
>>>> it passes a minimum bar (in this case do we get any perf context data
>>>> for every thread).
>>>>
>>>> Signed-off-by: Carsten Haitzler <carsten.haitzler@arm.com>
>>>
>>> Hi Carsten,
>>>
>>> I checked this on N1SDP and I get failures in both threads tests. This is
>>> because it's looking for "CID=..." when in my output threads are shown as
>>> "VMID=...":
>>>
>>>       Idx:628048; ID:10;       I_ADDR_CTXT_L_64IS0 : Address & Context, Long, 64 bit, IS0.; Addr=0x0000AAAAE3BF0B18; Ctxt: AArch64,EL0, NS; VMID=0xa588c;
>>>
>>> I think with a change to the grep it should work.
>>
>> Errrr... I get no VMID= ... it's all
>>
>> Idx:563008; ID:12;      I_ADDR_CTXT_L_64IS0 : Address & Context, Long, 64
>> bit, IS0.; Addr=0x0000AAAAE4B00A60; Ctxt: AArch64,EL0, NS; CID=0x00004aff;
>>
>> are you using containers or something? because:
>>
>>               if(context.updated_c)
>>               {
>>                   oss << "CID=0x" << std::hex << std::setfill('0') <<
>> std::setw(8) << context.ctxtID << "; ";
>>               }
>>               if(context.updated_v)
>>               {
>>                   oss << "VMID=0x" << std::hex << std::setfill('0') <<
>> std::setw(4) << context.VMID << "; ";
>>               }
>>
>> I'm running without any containers etc. - bare metal. Haven't bothered
>> with any VM stuff.
>>
>> In OpenOCD the CID should be the the pid/thread id. It seems to not be
>> the same thing as VMID. I haven't traced this beyond here as to exactly
>> what this represents though my first reaction is "This is extra VM info
>> and not the PID/TID being looked for". OpenOCD is full of tests with log
>> dumps that produce CID and VMID:
>>
>> Idx:1676; ID:10;        I_ADDR_CTXT_L_64IS0 : Address & Context, Long,
>> 64 bit, IS0.; Addr=0xFFFFFFC000096A00; Ctxt: AArch64,EL1, NS;
>> CID=0x00000000; VMID=0x0000;
>>
>> A quick git grep CID= in OpenCD will show them all. My understanding is
>> CID is the thread/process ID and thus the test/check "Do we get reported
>> data from all threads? - anything?".
>>
>> I don't think using VMID is right. The fact you are missing a CID is an
>> issue though...
>>
> 
> The register used for linux TID trace is dependent on the EL of the kernel.
> EL1 => CONTEXT_IDR_EL1
> EL2 => CONTEXT_IDR_EL2.
> 
> By design, the trace hardware traces CONTEXT_IDR_EL2 as the VMID packet.
> 
> So, depending on your kernel build, TID can validly be traced as CID or VMID

Ahhh I haven't encountered that. So basically look for CID=xxx OR 
VMID=xxx if no CID=xxx is there.

> Regards
> 
> Mike
> 
>>> Thanks
>>> James
>>>
>>>> ---
>>>>    .../coresight/thread_loop_check_tid_10.sh     | 19 +++++++++++++++++++
>>>>    .../coresight/thread_loop_check_tid_2.sh      | 19 +++++++++++++++++++
>>>>    2 files changed, 38 insertions(+)
>>>>    create mode 100755 tools/perf/tests/shell/coresight/thread_loop_check_tid_10.sh
>>>>    create mode 100755 tools/perf/tests/shell/coresight/thread_loop_check_tid_2.sh
>>>>
>>>> diff --git a/tools/perf/tests/shell/coresight/thread_loop_check_tid_10.sh b/tools/perf/tests/shell/coresight/thread_loop_check_tid_10.sh
>>>> new file mode 100755
>>>> index 000000000000..7c13636fc778
>>>> --- /dev/null
>>>> +++ b/tools/perf/tests/shell/coresight/thread_loop_check_tid_10.sh
>>>> @@ -0,0 +1,19 @@
>>>> +#!/bin/sh -e
>>>> +# CoreSight / Thread Loop 10 Threads - Check TID
>>>> +
>>>> +# SPDX-License-Identifier: GPL-2.0
>>>> +# Carsten Haitzler <carsten.haitzler@arm.com>, 2021
>>>> +
>>>> +TEST="thread_loop"
>>>> +. $(dirname $0)/../lib/coresight.sh
>>>> +ARGS="10 1"
>>>> +DATV="check-tid-10th"
>>>> +DATA="$DATD/perf-$TEST-$DATV.data"
>>>> +STDO="$DATD/perf-$TEST-$DATV.stdout"
>>>> +
>>>> +SHOW_TID=1 perf record -s $PERFRECOPT -o "$DATA" "$BIN" $ARGS > $STDO
>>>> +
>>>> +perf_dump_aux_tid_verify "$DATA" "$STDO"
>>>> +
>>>> +err=$?
>>>> +exit $err
>>>> diff --git a/tools/perf/tests/shell/coresight/thread_loop_check_tid_2.sh b/tools/perf/tests/shell/coresight/thread_loop_check_tid_2.sh
>>>> new file mode 100755
>>>> index 000000000000..a067145af43c
>>>> --- /dev/null
>>>> +++ b/tools/perf/tests/shell/coresight/thread_loop_check_tid_2.sh
>>>> @@ -0,0 +1,19 @@
>>>> +#!/bin/sh -e
>>>> +# CoreSight / Thread Loop 2 Threads - Check TID
>>>> +
>>>> +# SPDX-License-Identifier: GPL-2.0
>>>> +# Carsten Haitzler <carsten.haitzler@arm.com>, 2021
>>>> +
>>>> +TEST="thread_loop"
>>>> +. $(dirname $0)/../lib/coresight.sh
>>>> +ARGS="2 20"
>>>> +DATV="check-tid-2th"
>>>> +DATA="$DATD/perf-$TEST-$DATV.data"
>>>> +STDO="$DATD/perf-$TEST-$DATV.stdout"
>>>> +
>>>> +SHOW_TID=1 perf record -s $PERFRECOPT -o "$DATA" "$BIN" $ARGS > $STDO
>>>> +
>>>> +perf_dump_aux_tid_verify "$DATA" "$STDO"
>>>> +
>>>> +err=$?
>>>> +exit $err
> 
> 
> 

[PATCH 11/14] perf test: Add unroll thread test tool

[carsten.haitzler]

From: "Carsten Haitzler (Rasterman)" <raster@rasterman.com>

Add test tool to be driven by further test scripts. This is a simple C
based test that is for arm64 with some inline ASM to manually unroll a
lot of code to have a very long sequence of commands.

Signed-off-by: Carsten Haitzler <carsten.haitzler@arm.com>
---
 tools/perf/tests/shell/coresight/Makefile     |  3 +-
 .../coresight/unroll_loop_thread/.gitignore   |  1 +
 .../coresight/unroll_loop_thread/Makefile     | 33 +++++++++
 .../unroll_loop_thread/unroll_loop_thread.c   | 74 +++++++++++++++++++
 4 files changed, 110 insertions(+), 1 deletion(-)
 create mode 100644 tools/perf/tests/shell/coresight/unroll_loop_thread/.gitignore
 create mode 100644 tools/perf/tests/shell/coresight/unroll_loop_thread/Makefile
 create mode 100644 tools/perf/tests/shell/coresight/unroll_loop_thread/unroll_loop_thread.c

diff --git a/tools/perf/tests/shell/coresight/Makefile b/tools/perf/tests/shell/coresight/Makefile
index 004974a71fb8..3b2b876cd9e2 100644
--- a/tools/perf/tests/shell/coresight/Makefile
+++ b/tools/perf/tests/shell/coresight/Makefile
@@ -7,7 +7,8 @@ include ../../../../../tools/scripts/utilities.mak
 SUBDIRS = \
 	asm_pure_loop \
 	memcpy_thread \
-	thread_loop
+	thread_loop \
+	unroll_loop_thread
 
 all: $(SUBDIRS)
 $(SUBDIRS):
diff --git a/tools/perf/tests/shell/coresight/unroll_loop_thread/.gitignore b/tools/perf/tests/shell/coresight/unroll_loop_thread/.gitignore
new file mode 100644
index 000000000000..2cb4e996dbf3
--- /dev/null
+++ b/tools/perf/tests/shell/coresight/unroll_loop_thread/.gitignore
@@ -0,0 +1 @@
+unroll_loop_thread
diff --git a/tools/perf/tests/shell/coresight/unroll_loop_thread/Makefile b/tools/perf/tests/shell/coresight/unroll_loop_thread/Makefile
new file mode 100644
index 000000000000..6264c4e3abd1
--- /dev/null
+++ b/tools/perf/tests/shell/coresight/unroll_loop_thread/Makefile
@@ -0,0 +1,33 @@
+# SPDX-License-Identifier: GPL-2.0
+# Carsten Haitzler <carsten.haitzler@arm.com>, 2021
+include ../Makefile.miniconfig
+
+# Binary to produce
+BIN=unroll_loop_thread
+# Any linking/libraries needed for the binary - empty if none needed
+LIB=-pthread
+
+all: $(BIN)
+
+$(BIN): $(BIN).c
+ifdef CORESIGHT
+ifeq ($(ARCH),arm64)
+# Build line
+	$(Q)$(CC) $(BIN).c -o $(BIN) $(LIB)
+endif
+endif
+
+install-tests: all
+ifdef CORESIGHT
+ifeq ($(ARCH),arm64)
+# Install the test tool in the right place
+	$(call QUIET_INSTALL, tests) \
+		$(INSTALL) -d -m 755 '$(DESTDIR_SQ)$(perfexec_instdir_SQ)/$(INSTDIR_SUB)/$(BIN)'; \
+		$(INSTALL) $(BIN) '$(DESTDIR_SQ)$(perfexec_instdir_SQ)/$(INSTDIR_SUB)/$(BIN)/$(BIN)'
+endif
+endif
+
+clean:
+	$(Q)$(RM) -f $(BIN)
+
+.PHONY: all clean install-tests
diff --git a/tools/perf/tests/shell/coresight/unroll_loop_thread/unroll_loop_thread.c b/tools/perf/tests/shell/coresight/unroll_loop_thread/unroll_loop_thread.c
new file mode 100644
index 000000000000..cb9d22c7dfb9
--- /dev/null
+++ b/tools/perf/tests/shell/coresight/unroll_loop_thread/unroll_loop_thread.c
@@ -0,0 +1,74 @@
+// SPDX-License-Identifier: GPL-2.0
+// Carsten Haitzler <carsten.haitzler@arm.com>, 2021
+#include <stdio.h>
+#include <stdlib.h>
+#include <unistd.h>
+#include <string.h>
+#include <pthread.h>
+
+struct args {
+	pthread_t th;
+	unsigned int in, out;
+	void *ret;
+};
+
+static void *thrfn(void *arg)
+{
+	struct args *a = arg;
+	unsigned int i, in = a->in;
+
+	for (i = 0; i < 10000; i++) {
+		asm volatile (
+// force an unroll of thia add instruction so we can test long runs of code
+#define SNIP1 "add %[in], %[in], #1\n"
+// 10
+#define SNIP2 SNIP1 SNIP1 SNIP1 SNIP1 SNIP1 SNIP1 SNIP1 SNIP1 SNIP1 SNIP1
+// 100
+#define SNIP3 SNIP2 SNIP2 SNIP2 SNIP2 SNIP2 SNIP2 SNIP2 SNIP2 SNIP2 SNIP2
+// 1000
+#define SNIP4 SNIP3 SNIP3 SNIP3 SNIP3 SNIP3 SNIP3 SNIP3 SNIP3 SNIP3 SNIP3
+// 10000
+#define SNIP5 SNIP4 SNIP4 SNIP4 SNIP4 SNIP4 SNIP4 SNIP4 SNIP4 SNIP4 SNIP4
+// 100000
+			SNIP5 SNIP5 SNIP5 SNIP5 SNIP5 SNIP5 SNIP5 SNIP5 SNIP5 SNIP5
+			: /* out */
+			: /* in */ [in] "r" (in)
+			: /* clobber */
+		);
+	}
+}
+
+static pthread_t new_thr(void *(*fn) (void *arg), void *arg)
+{
+	pthread_t t;
+	pthread_attr_t attr;
+
+	pthread_attr_init(&attr);
+	pthread_create(&t, &attr, fn, arg);
+	return t;
+}
+
+int main(int argc, char **argv)
+{
+	unsigned int i, thr;
+	pthread_t threads[256];
+	struct args args[256];
+
+	if (argc < 2) {
+		printf("ERR: %s [numthreads]\n", argv[0]);
+		exit(1);
+	}
+
+	thr = atoi(argv[1]);
+	if ((thr > 256) || (thr < 1)) {
+		printf("ERR: threads 1-256\n");
+		exit(1);
+	}
+	for (i = 0; i < thr; i++) {
+		args[i].in = rand();
+		args[i].th = new_thr(thrfn, &(args[i]));
+	}
+	for (i = 0; i < thr; i++)
+		pthread_join(args[i].th, &(args[i].ret));
+	return 0;
+}
-- 
2.32.0

[PATCH 12/14] perf test: Add unroll thread test shell script

[carsten.haitzler]

From: "Carsten Haitzler (Rasterman)" <raster@rasterman.com>

This adds scripts to drive the unroll thread tests to compare perf
output against a minimum bar of content/quality.

Signed-off-by: Carsten Haitzler <carsten.haitzler@arm.com>
---
 .../shell/coresight/unroll_loop_thread_10.sh   | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)
 create mode 100755 tools/perf/tests/shell/coresight/unroll_loop_thread_10.sh

diff --git a/tools/perf/tests/shell/coresight/unroll_loop_thread_10.sh b/tools/perf/tests/shell/coresight/unroll_loop_thread_10.sh
new file mode 100755
index 000000000000..f48c85230b15
--- /dev/null
+++ b/tools/perf/tests/shell/coresight/unroll_loop_thread_10.sh
@@ -0,0 +1,18 @@
+#!/bin/sh -e
+# CoreSight / Unroll Loop Thread 10
+
+# SPDX-License-Identifier: GPL-2.0
+# Carsten Haitzler <carsten.haitzler@arm.com>, 2021
+
+TEST="unroll_loop_thread"
+. $(dirname $0)/../lib/coresight.sh
+ARGS="10"
+DATV="10"
+DATA="$DATD/perf-$TEST-$DATV.data"
+
+perf record $PERFRECOPT -o "$DATA" "$BIN" $ARGS
+
+perf_dump_aux_verify "$DATA" 10 10 10
+
+err=$?
+exit $err
-- 
2.32.0

[PATCH 13/14] perf test: Add git ignore for tmp and output files of CoreSight tests

[carsten.haitzler]

From: "Carsten Haitzler (Rasterman)" <raster@rasterman.com>

Ignore other output files of the new CoreSight tests so they don't
fill git status with noise we don't need or want.

Signed-off-by: Carsten Haitzler <carsten.haitzler@arm.com>
---
 tools/perf/.gitignore | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/tools/perf/.gitignore b/tools/perf/.gitignore
index faa23b5d32f5..a653311d9693 100644
--- a/tools/perf/.gitignore
+++ b/tools/perf/.gitignore
@@ -22,6 +22,7 @@ perf-archive
 perf-iostat
 tags
 TAGS
+stats-*.csv
 cscope*
 config.mak
 config.mak.autogen
@@ -29,6 +30,7 @@ config.mak.autogen
 *-flex.*
 *.pyc
 *.pyo
+*.stdout
 .config-detected
 util/intel-pt-decoder/inat-tables.c
 arch/*/include/generated/
-- 
2.32.0

[PATCH 14/14] perf test: Add relevant documentation about CoreSight testing

[carsten.haitzler]

From: "Carsten Haitzler (Rasterman)" <raster@rasterman.com>

This adds/improves documentation helping people get started with
CoreSight and perf as well as describing the testing and how it works.

Signed-off-by: Carsten Haitzler <carsten.haitzler@arm.com>
---
 .../trace/coresight/coresight-perf.rst        | 160 ++++++++++++++++++
 tools/perf/Documentation/arm-coresight.txt    |   5 +
 2 files changed, 165 insertions(+)
 create mode 100644 Documentation/trace/coresight/coresight-perf.rst
 create mode 100644 tools/perf/Documentation/arm-coresight.txt

diff --git a/Documentation/trace/coresight/coresight-perf.rst b/Documentation/trace/coresight/coresight-perf.rst
new file mode 100644
index 000000000000..de25082447dd
--- /dev/null
+++ b/Documentation/trace/coresight/coresight-perf.rst
@@ -0,0 +1,160 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+================
+CoreSight - Perf
+================
+
+    :Author:   Carsten Haitzler <carsten.haitzler@arm.com>
+    :Date:     June 29th, 2022
+
+Perf is able to locally access CoreSight trace data and store it to the
+output perf data files. This data can then be later decoded to give the
+instructions that were traced for debugging or profiling purposes. You
+can log such data with a perf record command like:
+
+    perf record -e cs_etm//u testbinary
+
+This would run some test binary (testbinary) until it exits and record
+a perf.data trace file. That file would have AUX sections if CoreSight
+is working correctly. You can dump the content of this file as
+readable text with a command like:
+
+    perf report --stdio --dump -i perf.data
+
+You should find some sections of this file have AUX data blocks like:
+
+    0x1e78 [0x30]: PERF_RECORD_AUXTRACE size: 0x11dd0  offset: 0  ref: 0x1b614fc1061b0ad1  idx: 0  tid: 531230  cpu: -1
+
+    . ... CoreSight ETM Trace data: size 73168 bytes
+            Idx:0; ID:10;   I_ASYNC : Alignment Synchronisation.
+              Idx:12; ID:10;  I_TRACE_INFO : Trace Info.; INFO=0x0 { CC.0 }
+              Idx:17; ID:10;  I_ADDR_L_64IS0 : Address, Long, 64 bit, IS0.; Addr=0x0000000000000000;
+              Idx:26; ID:10;  I_TRACE_ON : Trace On.
+              Idx:27; ID:10;  I_ADDR_CTXT_L_64IS0 : Address & Context, Long, 64 bit, IS0.; Addr=0x0000FFFFB6069140; Ctxt: AArch64,EL0, NS;
+              Idx:38; ID:10;  I_ATOM_F6 : Atom format 6.; EEEEEEEEEEEEEEEEEEEEEEEE
+              Idx:39; ID:10;  I_ATOM_F6 : Atom format 6.; EEEEEEEEEEEEEEEEEEEEEEEE
+              Idx:40; ID:10;  I_ATOM_F6 : Atom format 6.; EEEEEEEEEEEEEEEEEEEEEEEE
+              Idx:41; ID:10;  I_ATOM_F6 : Atom format 6.; EEEEEEEEEEEN
+              ...
+
+If you see these above, then your system is tracing CoreSight data
+correctly.
+
+To compile perf with CoreSight support in the tools/perf directory do
+
+    make CORESIGHT=1
+
+This requires OpenCSD to build. You may install distribution packages
+for the support such as libopencsd and libopencsd-dev or download it
+and build yourself. Upstream OpenCSD is located at:
+
+  https://github.com/Linaro/OpenCSD
+
+For complete information on building perf with CoreSight support and
+more extensive usage look at:
+
+  https://github.com/Linaro/OpenCSD/blob/master/HOWTO.md
+
+
+Kernel CoreSight Support
+------------------------
+
+You will also want CoreSight support enabled in your kernel config.
+Ensure it is enabled with:
+
+    CONFIG_CORESIGHT=y
+
+There are various other CoreSight options you probably also want
+enabled like:
+
+    CONFIG_CORESIGHT_LINKS_AND_SINKS=y
+    CONFIG_CORESIGHT_LINK_AND_SINK_TMC=y
+    CONFIG_CORESIGHT_CATU=y
+    CONFIG_CORESIGHT_SINK_TPIU=y
+    CONFIG_CORESIGHT_SINK_ETBV10=y
+    CONFIG_CORESIGHT_SOURCE_ETM4X=y
+    CONFIG_CORESIGHT_STM=y
+    CONFIG_CORESIGHT_CPU_DEBUG=y
+    CONFIG_CORESIGHT_CTI=y
+    CONFIG_CORESIGHT_CTI_INTEGRATION_REGS=y
+
+Please refer to the kernel configuration help for more information.
+
+Perf test - Verify kernel and userspace perf CoreSight work
+-----------------------------------------------------------
+
+When you run perf test, it will do a lot of self tests. Some of those
+tests will cover CoreSight (only if enabled and on ARM64). You
+generally would run perf test from the tools/perf directory in the
+kernel tree. Some tests will check some internal perf support like:
+
+    Check Arm CoreSight trace data recording and synthesized samples
+    Check Arm SPE trace data recording and synthesized samples
+
+Some others will actually use perf record and some test binaries that
+are in tests/shell/coresight and will collect traces to ensure a
+minimum level of functionality is met. The scripts that launch these
+tests are in the same directory. These will all look like:
+
+    CoreSight / ASM Pure Loop
+    CoreSight / Memcpy 16k 10 Threads
+    CoreSight / Thread Loop 10 Threads - Check TID
+    ...
+
+These perf record tests will not run if the tool binaries do not exist
+in tests/shell/coresight/*/ and will be skipped. If you do not have
+CoreSight support in hardware then either do not build perf with
+CoreSight support or remove these binaries in order to not have these
+tests fail and have them skip instead.
+
+These tests will log historical results in the current working
+directory (e.g. tools/perf) and will be named stats-*.csv like:
+
+    stats-asm_pure_loop-out.csv
+    stats-memcpy_thread-16k_10.csv
+    ...
+
+These statistic files log some aspects of the AUX data sections in
+the perf data output counting some numbers of certain encodings (a
+good way to know that it's working in a very simple way). One problem
+with CoreSight is that given a large enough amount of data needing to
+be logged, some of it can be lost due to the processor not waking up
+in time to read out all the data from buffers etc.. You will notice
+that the amount of data collected can vary a lot per run of perf test.
+If you wish to see how this changes over time, simply run perf test
+multiple times and all these csv files will have more and more data
+appended to it that you can later examine, graph and otherwise use to
+figure out if things have become worse or better.
+
+This means sometimes these tests fail as they don't capture all the
+data needed. This is about tracking quality and amount of data
+produced over time and to see when changes to the Linux kernel improve
+quality of traces.
+
+Be aware that some of these tests take quite a while to run, specifically
+in processing the perf data file and dumping contents to then examine what
+is inside.
+
+You can change where these csv logs are stored by setting the
+PERF_TEST_CORESIGHT_STATDIR environment variable before running perf
+test like:
+
+    export PERF_TEST_CORESIGHT_STATDIR=/var/tmp
+    perf test
+
+They will also store resulting perf output data in the current
+directory for later inspection like:
+
+    perf-asm_pure_loop-out.data
+    perf-memcpy_thread-16k_10.data
+    ...
+
+You can alter where the perf data files are stored by setting the
+PERF_TEST_CORESIGHT_DATADIR environment variable such as:
+
+    PERF_TEST_CORESIGHT_DATADIR=/var/tmp
+    perf test
+
+You may wish to set these above environment variables if you whish to
+keep the output of tests outside of the current working directory for
+longer term storage and examination.
diff --git a/tools/perf/Documentation/arm-coresight.txt b/tools/perf/Documentation/arm-coresight.txt
new file mode 100644
index 000000000000..f94743a4d161
--- /dev/null
+++ b/tools/perf/Documentation/arm-coresight.txt
@@ -0,0 +1,5 @@
+Arm CoreSight Support
+=====================
+
+Please see docuentation in the central CoreSight location in the
+kernel tree under Documentation/trace/coresight
-- 
2.32.0

Re: [PATCH 14/14] perf test: Add relevant documentation about CoreSight testing

[Bagas Sanjaya]

On Fri, Jul 01, 2022 at 01:08:03PM +0100, carsten.haitzler@foss.arm.com wrote:
> From: "Carsten Haitzler (Rasterman)" <raster@rasterman.com>
> 

Hi Carsten,

This doc patch can be improved, see below.

> This adds/improves documentation helping people get started with
> CoreSight and perf as well as describing the testing and how it works.
> 

Use imperative mood instead of descriptive one for patch description.

> Signed-off-by: Carsten Haitzler <carsten.haitzler@arm.com>
> ---
>  .../trace/coresight/coresight-perf.rst        | 160 ++++++++++++++++++
>  tools/perf/Documentation/arm-coresight.txt    |   5 +
>  2 files changed, 165 insertions(+)
>  create mode 100644 Documentation/trace/coresight/coresight-perf.rst
>  create mode 100644 tools/perf/Documentation/arm-coresight.txt
> 
> diff --git a/Documentation/trace/coresight/coresight-perf.rst b/Documentation/trace/coresight/coresight-perf.rst
> new file mode 100644
> index 000000000000..de25082447dd
> --- /dev/null
> +++ b/Documentation/trace/coresight/coresight-perf.rst
> @@ -0,0 +1,160 @@
> +.. SPDX-License-Identifier: GPL-2.0
> +
> +================
> +CoreSight - Perf
> +================
> +
> +    :Author:   Carsten Haitzler <carsten.haitzler@arm.com>
> +    :Date:     June 29th, 2022
> +
> +Perf is able to locally access CoreSight trace data and store it to the
> +output perf data files. This data can then be later decoded to give the
> +instructions that were traced for debugging or profiling purposes. You
> +can log such data with a perf record command like:
> +
> +    perf record -e cs_etm//u testbinary
> +

Use literal code block.

> +This would run some test binary (testbinary) until it exits and record
> +a perf.data trace file. That file would have AUX sections if CoreSight
> +is working correctly. You can dump the content of this file as
> +readable text with a command like:
> +
> +    perf report --stdio --dump -i perf.data
> +

Same as above.

> +You should find some sections of this file have AUX data blocks like:
> +
> +    0x1e78 [0x30]: PERF_RECORD_AUXTRACE size: 0x11dd0  offset: 0  ref: 0x1b614fc1061b0ad1  idx: 0  tid: 531230  cpu: -1
> +
> +    . ... CoreSight ETM Trace data: size 73168 bytes
> +            Idx:0; ID:10;   I_ASYNC : Alignment Synchronisation.
> +              Idx:12; ID:10;  I_TRACE_INFO : Trace Info.; INFO=0x0 { CC.0 }
> +              Idx:17; ID:10;  I_ADDR_L_64IS0 : Address, Long, 64 bit, IS0.; Addr=0x0000000000000000;
> +              Idx:26; ID:10;  I_TRACE_ON : Trace On.
> +              Idx:27; ID:10;  I_ADDR_CTXT_L_64IS0 : Address & Context, Long, 64 bit, IS0.; Addr=0x0000FFFFB6069140; Ctxt: AArch64,EL0, NS;
> +              Idx:38; ID:10;  I_ATOM_F6 : Atom format 6.; EEEEEEEEEEEEEEEEEEEEEEEE
> +              Idx:39; ID:10;  I_ATOM_F6 : Atom format 6.; EEEEEEEEEEEEEEEEEEEEEEEE
> +              Idx:40; ID:10;  I_ATOM_F6 : Atom format 6.; EEEEEEEEEEEEEEEEEEEEEEEE
> +              Idx:41; ID:10;  I_ATOM_F6 : Atom format 6.; EEEEEEEEEEEN
> +              ...
> +

Here too.

> +If you see these above, then your system is tracing CoreSight data
> +correctly.
> +
> +To compile perf with CoreSight support in the tools/perf directory do
> +
> +    make CORESIGHT=1
> +

Here too again.

> +This requires OpenCSD to build. You may install distribution packages
> +for the support such as libopencsd and libopencsd-dev or download it
> +and build yourself. Upstream OpenCSD is located at:
> +
> +  https://github.com/Linaro/OpenCSD
> +
> +For complete information on building perf with CoreSight support and
> +more extensive usage look at:
> +
> +  https://github.com/Linaro/OpenCSD/blob/master/HOWTO.md
> +
> +
> +Kernel CoreSight Support
> +------------------------
> +
> +You will also want CoreSight support enabled in your kernel config.
> +Ensure it is enabled with:
> +
> +    CONFIG_CORESIGHT=y
> +
> +There are various other CoreSight options you probably also want
> +enabled like:
> +
> +    CONFIG_CORESIGHT_LINKS_AND_SINKS=y
> +    CONFIG_CORESIGHT_LINK_AND_SINK_TMC=y
> +    CONFIG_CORESIGHT_CATU=y
> +    CONFIG_CORESIGHT_SINK_TPIU=y
> +    CONFIG_CORESIGHT_SINK_ETBV10=y
> +    CONFIG_CORESIGHT_SOURCE_ETM4X=y
> +    CONFIG_CORESIGHT_STM=y
> +    CONFIG_CORESIGHT_CPU_DEBUG=y
> +    CONFIG_CORESIGHT_CTI=y
> +    CONFIG_CORESIGHT_CTI_INTEGRATION_REGS=y
> +

Same as above again.

> +Please refer to the kernel configuration help for more information.
> +
> +Perf test - Verify kernel and userspace perf CoreSight work
> +-----------------------------------------------------------
> +
> +When you run perf test, it will do a lot of self tests. Some of those
> +tests will cover CoreSight (only if enabled and on ARM64). You
> +generally would run perf test from the tools/perf directory in the
> +kernel tree. Some tests will check some internal perf support like:
> +
> +    Check Arm CoreSight trace data recording and synthesized samples
> +    Check Arm SPE trace data recording and synthesized samples
> +

Use bullet lists.

> +Some others will actually use perf record and some test binaries that
> +are in tests/shell/coresight and will collect traces to ensure a
> +minimum level of functionality is met. The scripts that launch these
> +tests are in the same directory. These will all look like:
> +
> +    CoreSight / ASM Pure Loop
> +    CoreSight / Memcpy 16k 10 Threads
> +    CoreSight / Thread Loop 10 Threads - Check TID
> +    ...
> +

Same as above.

> +These perf record tests will not run if the tool binaries do not exist
> +in tests/shell/coresight/*/ and will be skipped. If you do not have
> +CoreSight support in hardware then either do not build perf with
> +CoreSight support or remove these binaries in order to not have these
> +tests fail and have them skip instead.
> +
> +These tests will log historical results in the current working
> +directory (e.g. tools/perf) and will be named stats-*.csv like:
> +
> +    stats-asm_pure_loop-out.csv
> +    stats-memcpy_thread-16k_10.csv
> +    ...
> +

These above causes htmldocs warning (unescaped wildcard), so I have to apply
the fixup:

---- >8 ----

diff --git a/Documentation/trace/coresight/coresight-perf.rst b/Documentation/trace/coresight/coresight-perf.rst
index de25082447dd50..a25fcda5c37c55 100644
--- a/Documentation/trace/coresight/coresight-perf.rst
+++ b/Documentation/trace/coresight/coresight-perf.rst
@@ -102,13 +102,13 @@ tests are in the same directory. These will all look like:
     ...
 
 These perf record tests will not run if the tool binaries do not exist
-in tests/shell/coresight/*/ and will be skipped. If you do not have
+in tests/shell/coresight/\*/ and will be skipped. If you do not have
 CoreSight support in hardware then either do not build perf with
 CoreSight support or remove these binaries in order to not have these
 tests fail and have them skip instead.
 
 These tests will log historical results in the current working
-directory (e.g. tools/perf) and will be named stats-*.csv like:
+directory (e.g. tools/perf) and will be named stats-\*.csv like:
 
     stats-asm_pure_loop-out.csv
     stats-memcpy_thread-16k_10.csv

---- >8 ----

Also, the output list above could be inside code block (since these
are output).

> +These statistic files log some aspects of the AUX data sections in
> +the perf data output counting some numbers of certain encodings (a
> +good way to know that it's working in a very simple way). One problem
> +with CoreSight is that given a large enough amount of data needing to
> +be logged, some of it can be lost due to the processor not waking up
> +in time to read out all the data from buffers etc.. You will notice
> +that the amount of data collected can vary a lot per run of perf test.
> +If you wish to see how this changes over time, simply run perf test
> +multiple times and all these csv files will have more and more data
> +appended to it that you can later examine, graph and otherwise use to
> +figure out if things have become worse or better.
> +
> +This means sometimes these tests fail as they don't capture all the
> +data needed. This is about tracking quality and amount of data
> +produced over time and to see when changes to the Linux kernel improve
> +quality of traces.
> +
> +Be aware that some of these tests take quite a while to run, specifically
> +in processing the perf data file and dumping contents to then examine what
> +is inside.
> +
> +You can change where these csv logs are stored by setting the
> +PERF_TEST_CORESIGHT_STATDIR environment variable before running perf
> +test like:
> +
> +    export PERF_TEST_CORESIGHT_STATDIR=/var/tmp
> +    perf test
> +
> +They will also store resulting perf output data in the current
> +directory for later inspection like:
> +
> +    perf-asm_pure_loop-out.data
> +    perf-memcpy_thread-16k_10.data
> +    ...
> +
> +You can alter where the perf data files are stored by setting the
> +PERF_TEST_CORESIGHT_DATADIR environment variable such as:
> +
> +    PERF_TEST_CORESIGHT_DATADIR=/var/tmp
> +    perf test
> +

Use code block.

> +You may wish to set these above environment variables if you whish to
> +keep the output of tests outside of the current working directory for
> +longer term storage and examination.
> diff --git a/tools/perf/Documentation/arm-coresight.txt b/tools/perf/Documentation/arm-coresight.txt
> new file mode 100644
> index 000000000000..f94743a4d161
> --- /dev/null
> +++ b/tools/perf/Documentation/arm-coresight.txt
> @@ -0,0 +1,5 @@
> +Arm CoreSight Support
> +=====================
> +
> +Please see docuentation in the central CoreSight location in the
> +kernel tree under Documentation/trace/coresight

s/ducuentation/documentation/

So here's the improv:

---- >8 ----

diff --git a/Documentation/trace/coresight/coresight-perf.rst b/Documentation/trace/coresight/coresight-perf.rst
index a25fcda5c37c55..0dd4689a699ecd 100644
--- a/Documentation/trace/coresight/coresight-perf.rst
+++ b/Documentation/trace/coresight/coresight-perf.rst
@@ -10,37 +10,37 @@ CoreSight - Perf
 Perf is able to locally access CoreSight trace data and store it to the
 output perf data files. This data can then be later decoded to give the
 instructions that were traced for debugging or profiling purposes. You
-can log such data with a perf record command like:
+can log such data with a perf record command like::
 
-    perf record -e cs_etm//u testbinary
+   perf record -e cs_etm//u testbinary
 
 This would run some test binary (testbinary) until it exits and record
 a perf.data trace file. That file would have AUX sections if CoreSight
 is working correctly. You can dump the content of this file as
-readable text with a command like:
+readable text with a command like::
 
-    perf report --stdio --dump -i perf.data
+   perf report --stdio --dump -i perf.data
 
-You should find some sections of this file have AUX data blocks like:
+You should find some sections of this file have AUX data blocks like::
 
-    0x1e78 [0x30]: PERF_RECORD_AUXTRACE size: 0x11dd0  offset: 0  ref: 0x1b614fc1061b0ad1  idx: 0  tid: 531230  cpu: -1
+   0x1e78 [0x30]: PERF_RECORD_AUXTRACE size: 0x11dd0  offset: 0  ref: 0x1b614fc1061b0ad1  idx: 0  tid: 531230  cpu: -1
 
-    . ... CoreSight ETM Trace data: size 73168 bytes
-            Idx:0; ID:10;   I_ASYNC : Alignment Synchronisation.
-              Idx:12; ID:10;  I_TRACE_INFO : Trace Info.; INFO=0x0 { CC.0 }
-              Idx:17; ID:10;  I_ADDR_L_64IS0 : Address, Long, 64 bit, IS0.; Addr=0x0000000000000000;
-              Idx:26; ID:10;  I_TRACE_ON : Trace On.
-              Idx:27; ID:10;  I_ADDR_CTXT_L_64IS0 : Address & Context, Long, 64 bit, IS0.; Addr=0x0000FFFFB6069140; Ctxt: AArch64,EL0, NS;
-              Idx:38; ID:10;  I_ATOM_F6 : Atom format 6.; EEEEEEEEEEEEEEEEEEEEEEEE
-              Idx:39; ID:10;  I_ATOM_F6 : Atom format 6.; EEEEEEEEEEEEEEEEEEEEEEEE
-              Idx:40; ID:10;  I_ATOM_F6 : Atom format 6.; EEEEEEEEEEEEEEEEEEEEEEEE
-              Idx:41; ID:10;  I_ATOM_F6 : Atom format 6.; EEEEEEEEEEEN
-              ...
+   . ... CoreSight ETM Trace data: size 73168 bytes
+           Idx:0; ID:10;   I_ASYNC : Alignment Synchronisation.
+             Idx:12; ID:10;  I_TRACE_INFO : Trace Info.; INFO=0x0 { CC.0 }
+             Idx:17; ID:10;  I_ADDR_L_64IS0 : Address, Long, 64 bit, IS0.; Addr=0x0000000000000000;
+             Idx:26; ID:10;  I_TRACE_ON : Trace On.
+             Idx:27; ID:10;  I_ADDR_CTXT_L_64IS0 : Address & Context, Long, 64 bit, IS0.; Addr=0x0000FFFFB6069140; Ctxt: AArch64,EL0, NS;
+             Idx:38; ID:10;  I_ATOM_F6 : Atom format 6.; EEEEEEEEEEEEEEEEEEEEEEEE
+             Idx:39; ID:10;  I_ATOM_F6 : Atom format 6.; EEEEEEEEEEEEEEEEEEEEEEEE
+             Idx:40; ID:10;  I_ATOM_F6 : Atom format 6.; EEEEEEEEEEEEEEEEEEEEEEEE
+             Idx:41; ID:10;  I_ATOM_F6 : Atom format 6.; EEEEEEEEEEEN
+             ...
 
 If you see these above, then your system is tracing CoreSight data
 correctly.
 
-To compile perf with CoreSight support in the tools/perf directory do
+To compile perf with CoreSight support in the tools/perf directory do::
 
     make CORESIGHT=1
 
@@ -60,23 +60,23 @@ Kernel CoreSight Support
 ------------------------
 
 You will also want CoreSight support enabled in your kernel config.
-Ensure it is enabled with:
+Ensure it is enabled with::
 
-    CONFIG_CORESIGHT=y
+   CONFIG_CORESIGHT=y
 
 There are various other CoreSight options you probably also want
-enabled like:
+enabled like::
 
-    CONFIG_CORESIGHT_LINKS_AND_SINKS=y
-    CONFIG_CORESIGHT_LINK_AND_SINK_TMC=y
-    CONFIG_CORESIGHT_CATU=y
-    CONFIG_CORESIGHT_SINK_TPIU=y
-    CONFIG_CORESIGHT_SINK_ETBV10=y
-    CONFIG_CORESIGHT_SOURCE_ETM4X=y
-    CONFIG_CORESIGHT_STM=y
-    CONFIG_CORESIGHT_CPU_DEBUG=y
-    CONFIG_CORESIGHT_CTI=y
-    CONFIG_CORESIGHT_CTI_INTEGRATION_REGS=y
+   CONFIG_CORESIGHT_LINKS_AND_SINKS=y
+   CONFIG_CORESIGHT_LINK_AND_SINK_TMC=y
+   CONFIG_CORESIGHT_CATU=y
+   CONFIG_CORESIGHT_SINK_TPIU=y
+   CONFIG_CORESIGHT_SINK_ETBV10=y
+   CONFIG_CORESIGHT_SOURCE_ETM4X=y
+   CONFIG_CORESIGHT_STM=y
+   CONFIG_CORESIGHT_CPU_DEBUG=y
+   CONFIG_CORESIGHT_CTI=y
+   CONFIG_CORESIGHT_CTI_INTEGRATION_REGS=y
 
 Please refer to the kernel configuration help for more information.
 
@@ -88,18 +88,18 @@ tests will cover CoreSight (only if enabled and on ARM64). You
 generally would run perf test from the tools/perf directory in the
 kernel tree. Some tests will check some internal perf support like:
 
-    Check Arm CoreSight trace data recording and synthesized samples
-    Check Arm SPE trace data recording and synthesized samples
+* Check Arm CoreSight trace data recording and synthesized samples
+* Check Arm SPE trace data recording and synthesized samples
 
 Some others will actually use perf record and some test binaries that
 are in tests/shell/coresight and will collect traces to ensure a
 minimum level of functionality is met. The scripts that launch these
 tests are in the same directory. These will all look like:
 
-    CoreSight / ASM Pure Loop
-    CoreSight / Memcpy 16k 10 Threads
-    CoreSight / Thread Loop 10 Threads - Check TID
-    ...
+* CoreSight / ASM Pure Loop
+* CoreSight / Memcpy 16k 10 Threads
+* CoreSight / Thread Loop 10 Threads - Check TID
+* etc.
 
 These perf record tests will not run if the tool binaries do not exist
 in tests/shell/coresight/\*/ and will be skipped. If you do not have
@@ -108,11 +108,11 @@ CoreSight support or remove these binaries in order to not have these
 tests fail and have them skip instead.
 
 These tests will log historical results in the current working
-directory (e.g. tools/perf) and will be named stats-\*.csv like:
+directory (e.g. tools/perf) and will be named stats-\*.csv like::
 
-    stats-asm_pure_loop-out.csv
-    stats-memcpy_thread-16k_10.csv
-    ...
+   stats-asm_pure_loop-out.csv
+   stats-memcpy_thread-16k_10.csv
+   ...
 
 These statistic files log some aspects of the AUX data sections in
 the perf data output counting some numbers of certain encodings (a
@@ -137,23 +137,23 @@ is inside.
 
 You can change where these csv logs are stored by setting the
 PERF_TEST_CORESIGHT_STATDIR environment variable before running perf
-test like:
+test like::
 
-    export PERF_TEST_CORESIGHT_STATDIR=/var/tmp
-    perf test
+   export PERF_TEST_CORESIGHT_STATDIR=/var/tmp
+   perf test
 
 They will also store resulting perf output data in the current
-directory for later inspection like:
+directory for later inspection like::
 
-    perf-asm_pure_loop-out.data
-    perf-memcpy_thread-16k_10.data
-    ...
+   perf-asm_pure_loop-out.data
+   perf-memcpy_thread-16k_10.data
+   ...
 
 You can alter where the perf data files are stored by setting the
-PERF_TEST_CORESIGHT_DATADIR environment variable such as:
+PERF_TEST_CORESIGHT_DATADIR environment variable such as::
 
-    PERF_TEST_CORESIGHT_DATADIR=/var/tmp
-    perf test
+   PERF_TEST_CORESIGHT_DATADIR=/var/tmp
+   perf test
 
 You may wish to set these above environment variables if you whish to
 keep the output of tests outside of the current working directory for
diff --git a/tools/perf/Documentation/arm-coresight.txt b/tools/perf/Documentation/arm-coresight.txt
index f94743a4d161f2..c117fc50a2a956 100644
--- a/tools/perf/Documentation/arm-coresight.txt
+++ b/tools/perf/Documentation/arm-coresight.txt
@@ -1,5 +1,5 @@
 Arm CoreSight Support
 =====================
 
-Please see docuentation in the central CoreSight location in the
-kernel tree under Documentation/trace/coresight
+For full documentation, see Documentation/trace/coresight/coresight-perf.rst
+in the kernel tree.

---- >8 ----

Note: since this is documentation patch, don't forget to Cc linux-doc list.
I add it for you.

Thanks.

-- 
An old man doll... just what I always wanted! - Clara

Re: [PATCH 14/14] perf test: Add relevant documentation about CoreSight testing

[Carsten Haitzler]

On 7/2/22 04:02, Bagas Sanjaya wrote:
> On Fri, Jul 01, 2022 at 01:08:03PM +0100, carsten.haitzler@foss.arm.com wrote:
>> From: "Carsten Haitzler (Rasterman)" <raster@rasterman.com>
>>
> 
> Hi Carsten,
> 
> This doc patch can be improved, see below.

I'll look at addressing the below - with some exceptions as they are not 
what you think they are.

>> This adds/improves documentation helping people get started with
>> CoreSight and perf as well as describing the testing and how it works.
>>
> 
> Use imperative mood instead of descriptive one for patch description.
> 
>> Signed-off-by: Carsten Haitzler <carsten.haitzler@arm.com>
>> ---
>>   .../trace/coresight/coresight-perf.rst        | 160 ++++++++++++++++++
>>   tools/perf/Documentation/arm-coresight.txt    |   5 +
>>   2 files changed, 165 insertions(+)
>>   create mode 100644 Documentation/trace/coresight/coresight-perf.rst
>>   create mode 100644 tools/perf/Documentation/arm-coresight.txt
>>
>> diff --git a/Documentation/trace/coresight/coresight-perf.rst b/Documentation/trace/coresight/coresight-perf.rst
>> new file mode 100644
>> index 000000000000..de25082447dd
>> --- /dev/null
>> +++ b/Documentation/trace/coresight/coresight-perf.rst
>> @@ -0,0 +1,160 @@
>> +.. SPDX-License-Identifier: GPL-2.0
>> +
>> +================
>> +CoreSight - Perf
>> +================
>> +
>> +    :Author:   Carsten Haitzler <carsten.haitzler@arm.com>
>> +    :Date:     June 29th, 2022
>> +
>> +Perf is able to locally access CoreSight trace data and store it to the
>> +output perf data files. This data can then be later decoded to give the
>> +instructions that were traced for debugging or profiling purposes. You
>> +can log such data with a perf record command like:
>> +
>> +    perf record -e cs_etm//u testbinary
>> +
> 
> Use literal code block.
> 
>> +This would run some test binary (testbinary) until it exits and record
>> +a perf.data trace file. That file would have AUX sections if CoreSight
>> +is working correctly. You can dump the content of this file as
>> +readable text with a command like:
>> +
>> +    perf report --stdio --dump -i perf.data
>> +
> 
> Same as above.
> 
>> +You should find some sections of this file have AUX data blocks like:
>> +
>> +    0x1e78 [0x30]: PERF_RECORD_AUXTRACE size: 0x11dd0  offset: 0  ref: 0x1b614fc1061b0ad1  idx: 0  tid: 531230  cpu: -1
>> +
>> +    . ... CoreSight ETM Trace data: size 73168 bytes
>> +            Idx:0; ID:10;   I_ASYNC : Alignment Synchronisation.
>> +              Idx:12; ID:10;  I_TRACE_INFO : Trace Info.; INFO=0x0 { CC.0 }
>> +              Idx:17; ID:10;  I_ADDR_L_64IS0 : Address, Long, 64 bit, IS0.; Addr=0x0000000000000000;
>> +              Idx:26; ID:10;  I_TRACE_ON : Trace On.
>> +              Idx:27; ID:10;  I_ADDR_CTXT_L_64IS0 : Address & Context, Long, 64 bit, IS0.; Addr=0x0000FFFFB6069140; Ctxt: AArch64,EL0, NS;
>> +              Idx:38; ID:10;  I_ATOM_F6 : Atom format 6.; EEEEEEEEEEEEEEEEEEEEEEEE
>> +              Idx:39; ID:10;  I_ATOM_F6 : Atom format 6.; EEEEEEEEEEEEEEEEEEEEEEEE
>> +              Idx:40; ID:10;  I_ATOM_F6 : Atom format 6.; EEEEEEEEEEEEEEEEEEEEEEEE
>> +              Idx:41; ID:10;  I_ATOM_F6 : Atom format 6.; EEEEEEEEEEEN
>> +              ...
>> +
> 
> Here too.
> 
>> +If you see these above, then your system is tracing CoreSight data
>> +correctly.
>> +
>> +To compile perf with CoreSight support in the tools/perf directory do
>> +
>> +    make CORESIGHT=1
>> +
> 
> Here too again.
> 
>> +This requires OpenCSD to build. You may install distribution packages
>> +for the support such as libopencsd and libopencsd-dev or download it
>> +and build yourself. Upstream OpenCSD is located at:
>> +
>> +  https://github.com/Linaro/OpenCSD
>> +
>> +For complete information on building perf with CoreSight support and
>> +more extensive usage look at:
>> +
>> +  https://github.com/Linaro/OpenCSD/blob/master/HOWTO.md
>> +
>> +
>> +Kernel CoreSight Support
>> +------------------------
>> +
>> +You will also want CoreSight support enabled in your kernel config.
>> +Ensure it is enabled with:
>> +
>> +    CONFIG_CORESIGHT=y
>> +
>> +There are various other CoreSight options you probably also want
>> +enabled like:
>> +
>> +    CONFIG_CORESIGHT_LINKS_AND_SINKS=y
>> +    CONFIG_CORESIGHT_LINK_AND_SINK_TMC=y
>> +    CONFIG_CORESIGHT_CATU=y
>> +    CONFIG_CORESIGHT_SINK_TPIU=y
>> +    CONFIG_CORESIGHT_SINK_ETBV10=y
>> +    CONFIG_CORESIGHT_SOURCE_ETM4X=y
>> +    CONFIG_CORESIGHT_STM=y
>> +    CONFIG_CORESIGHT_CPU_DEBUG=y
>> +    CONFIG_CORESIGHT_CTI=y
>> +    CONFIG_CORESIGHT_CTI_INTEGRATION_REGS=y
>> +
> 
> Same as above again.
> 
>> +Please refer to the kernel configuration help for more information.
>> +
>> +Perf test - Verify kernel and userspace perf CoreSight work
>> +-----------------------------------------------------------
>> +
>> +When you run perf test, it will do a lot of self tests. Some of those
>> +tests will cover CoreSight (only if enabled and on ARM64). You
>> +generally would run perf test from the tools/perf directory in the
>> +kernel tree. Some tests will check some internal perf support like:
>> +
>> +    Check Arm CoreSight trace data recording and synthesized samples
>> +    Check Arm SPE trace data recording and synthesized samples
>> +
> 
> Use bullet lists.

Actually this would be a code block - it is literally the stdout from 
perf test (just 2 lines of it for those tests).

>> +Some others will actually use perf record and some test binaries that
>> +are in tests/shell/coresight and will collect traces to ensure a
>> +minimum level of functionality is met. The scripts that launch these
>> +tests are in the same directory. These will all look like:
>> +
>> +    CoreSight / ASM Pure Loop
>> +    CoreSight / Memcpy 16k 10 Threads
>> +    CoreSight / Thread Loop 10 Threads - Check TID
>> +    ...
>> +
> 
> Same as above.

This too - a code block.

>> +These perf record tests will not run if the tool binaries do not exist
>> +in tests/shell/coresight/*/ and will be skipped. If you do not have
>> +CoreSight support in hardware then either do not build perf with
>> +CoreSight support or remove these binaries in order to not have these
>> +tests fail and have them skip instead.
>> +
>> +These tests will log historical results in the current working
>> +directory (e.g. tools/perf) and will be named stats-*.csv like:
>> +
>> +    stats-asm_pure_loop-out.csv
>> +    stats-memcpy_thread-16k_10.csv
>> +    ...
>> +
> 
> These above causes htmldocs warning (unescaped wildcard), so I have to apply
> the fixup:
> 
> ---- >8 ----
> 
> diff --git a/Documentation/trace/coresight/coresight-perf.rst b/Documentation/trace/coresight/coresight-perf.rst
> index de25082447dd50..a25fcda5c37c55 100644
> --- a/Documentation/trace/coresight/coresight-perf.rst
> +++ b/Documentation/trace/coresight/coresight-perf.rst
> @@ -102,13 +102,13 @@ tests are in the same directory. These will all look like:
>       ...
>   
>   These perf record tests will not run if the tool binaries do not exist
> -in tests/shell/coresight/*/ and will be skipped. If you do not have
> +in tests/shell/coresight/\*/ and will be skipped. If you do not have
>   CoreSight support in hardware then either do not build perf with
>   CoreSight support or remove these binaries in order to not have these
>   tests fail and have them skip instead.
>   
>   These tests will log historical results in the current working
> -directory (e.g. tools/perf) and will be named stats-*.csv like:
> +directory (e.g. tools/perf) and will be named stats-\*.csv like:
>   
>       stats-asm_pure_loop-out.csv
>       stats-memcpy_thread-16k_10.csv
> 
> ---- >8 ----
> 
> Also, the output list above could be inside code block (since these
> are output).

Yup.

>> +These statistic files log some aspects of the AUX data sections in
>> +the perf data output counting some numbers of certain encodings (a
>> +good way to know that it's working in a very simple way). One problem
>> +with CoreSight is that given a large enough amount of data needing to
>> +be logged, some of it can be lost due to the processor not waking up
>> +in time to read out all the data from buffers etc.. You will notice
>> +that the amount of data collected can vary a lot per run of perf test.
>> +If you wish to see how this changes over time, simply run perf test
>> +multiple times and all these csv files will have more and more data
>> +appended to it that you can later examine, graph and otherwise use to
>> +figure out if things have become worse or better.
>> +
>> +This means sometimes these tests fail as they don't capture all the
>> +data needed. This is about tracking quality and amount of data
>> +produced over time and to see when changes to the Linux kernel improve
>> +quality of traces.
>> +
>> +Be aware that some of these tests take quite a while to run, specifically
>> +in processing the perf data file and dumping contents to then examine what
>> +is inside.
>> +
>> +You can change where these csv logs are stored by setting the
>> +PERF_TEST_CORESIGHT_STATDIR environment variable before running perf
>> +test like:
>> +
>> +    export PERF_TEST_CORESIGHT_STATDIR=/var/tmp
>> +    perf test
>> +
>> +They will also store resulting perf output data in the current
>> +directory for later inspection like:
>> +
>> +    perf-asm_pure_loop-out.data
>> +    perf-memcpy_thread-16k_10.data
>> +    ...
>> +
>> +You can alter where the perf data files are stored by setting the
>> +PERF_TEST_CORESIGHT_DATADIR environment variable such as:
>> +
>> +    PERF_TEST_CORESIGHT_DATADIR=/var/tmp
>> +    perf test
>> +
> 
> Use code block.
> 
>> +You may wish to set these above environment variables if you whish to
>> +keep the output of tests outside of the current working directory for
>> +longer term storage and examination.
>> diff --git a/tools/perf/Documentation/arm-coresight.txt b/tools/perf/Documentation/arm-coresight.txt
>> new file mode 100644
>> index 000000000000..f94743a4d161
>> --- /dev/null
>> +++ b/tools/perf/Documentation/arm-coresight.txt
>> @@ -0,0 +1,5 @@
>> +Arm CoreSight Support
>> +=====================
>> +
>> +Please see docuentation in the central CoreSight location in the
>> +kernel tree under Documentation/trace/coresight
> 
> s/ducuentation/documentation/
> 
> So here's the improv:
> 
> ---- >8 ----
> 
> diff --git a/Documentation/trace/coresight/coresight-perf.rst b/Documentation/trace/coresight/coresight-perf.rst
> index a25fcda5c37c55..0dd4689a699ecd 100644
> --- a/Documentation/trace/coresight/coresight-perf.rst
> +++ b/Documentation/trace/coresight/coresight-perf.rst
> @@ -10,37 +10,37 @@ CoreSight - Perf
>   Perf is able to locally access CoreSight trace data and store it to the
>   output perf data files. This data can then be later decoded to give the
>   instructions that were traced for debugging or profiling purposes. You
> -can log such data with a perf record command like:
> +can log such data with a perf record command like::
>   
> -    perf record -e cs_etm//u testbinary
> +   perf record -e cs_etm//u testbinary
>   
>   This would run some test binary (testbinary) until it exits and record
>   a perf.data trace file. That file would have AUX sections if CoreSight
>   is working correctly. You can dump the content of this file as
> -readable text with a command like:
> +readable text with a command like::
>   
> -    perf report --stdio --dump -i perf.data
> +   perf report --stdio --dump -i perf.data
>   
> -You should find some sections of this file have AUX data blocks like:
> +You should find some sections of this file have AUX data blocks like::
>   
> -    0x1e78 [0x30]: PERF_RECORD_AUXTRACE size: 0x11dd0  offset: 0  ref: 0x1b614fc1061b0ad1  idx: 0  tid: 531230  cpu: -1
> +   0x1e78 [0x30]: PERF_RECORD_AUXTRACE size: 0x11dd0  offset: 0  ref: 0x1b614fc1061b0ad1  idx: 0  tid: 531230  cpu: -1
>   
> -    . ... CoreSight ETM Trace data: size 73168 bytes
> -            Idx:0; ID:10;   I_ASYNC : Alignment Synchronisation.
> -              Idx:12; ID:10;  I_TRACE_INFO : Trace Info.; INFO=0x0 { CC.0 }
> -              Idx:17; ID:10;  I_ADDR_L_64IS0 : Address, Long, 64 bit, IS0.; Addr=0x0000000000000000;
> -              Idx:26; ID:10;  I_TRACE_ON : Trace On.
> -              Idx:27; ID:10;  I_ADDR_CTXT_L_64IS0 : Address & Context, Long, 64 bit, IS0.; Addr=0x0000FFFFB6069140; Ctxt: AArch64,EL0, NS;
> -              Idx:38; ID:10;  I_ATOM_F6 : Atom format 6.; EEEEEEEEEEEEEEEEEEEEEEEE
> -              Idx:39; ID:10;  I_ATOM_F6 : Atom format 6.; EEEEEEEEEEEEEEEEEEEEEEEE
> -              Idx:40; ID:10;  I_ATOM_F6 : Atom format 6.; EEEEEEEEEEEEEEEEEEEEEEEE
> -              Idx:41; ID:10;  I_ATOM_F6 : Atom format 6.; EEEEEEEEEEEN
> -              ...
> +   . ... CoreSight ETM Trace data: size 73168 bytes
> +           Idx:0; ID:10;   I_ASYNC : Alignment Synchronisation.
> +             Idx:12; ID:10;  I_TRACE_INFO : Trace Info.; INFO=0x0 { CC.0 }
> +             Idx:17; ID:10;  I_ADDR_L_64IS0 : Address, Long, 64 bit, IS0.; Addr=0x0000000000000000;
> +             Idx:26; ID:10;  I_TRACE_ON : Trace On.
> +             Idx:27; ID:10;  I_ADDR_CTXT_L_64IS0 : Address & Context, Long, 64 bit, IS0.; Addr=0x0000FFFFB6069140; Ctxt: AArch64,EL0, NS;
> +             Idx:38; ID:10;  I_ATOM_F6 : Atom format 6.; EEEEEEEEEEEEEEEEEEEEEEEE
> +             Idx:39; ID:10;  I_ATOM_F6 : Atom format 6.; EEEEEEEEEEEEEEEEEEEEEEEE
> +             Idx:40; ID:10;  I_ATOM_F6 : Atom format 6.; EEEEEEEEEEEEEEEEEEEEEEEE
> +             Idx:41; ID:10;  I_ATOM_F6 : Atom format 6.; EEEEEEEEEEEN
> +             ...
>   
>   If you see these above, then your system is tracing CoreSight data
>   correctly.
>   
> -To compile perf with CoreSight support in the tools/perf directory do
> +To compile perf with CoreSight support in the tools/perf directory do::
>   
>       make CORESIGHT=1
>   
> @@ -60,23 +60,23 @@ Kernel CoreSight Support
>   ------------------------
>   
>   You will also want CoreSight support enabled in your kernel config.
> -Ensure it is enabled with:
> +Ensure it is enabled with::
>   
> -    CONFIG_CORESIGHT=y
> +   CONFIG_CORESIGHT=y
>   
>   There are various other CoreSight options you probably also want
> -enabled like:
> +enabled like::
>   
> -    CONFIG_CORESIGHT_LINKS_AND_SINKS=y
> -    CONFIG_CORESIGHT_LINK_AND_SINK_TMC=y
> -    CONFIG_CORESIGHT_CATU=y
> -    CONFIG_CORESIGHT_SINK_TPIU=y
> -    CONFIG_CORESIGHT_SINK_ETBV10=y
> -    CONFIG_CORESIGHT_SOURCE_ETM4X=y
> -    CONFIG_CORESIGHT_STM=y
> -    CONFIG_CORESIGHT_CPU_DEBUG=y
> -    CONFIG_CORESIGHT_CTI=y
> -    CONFIG_CORESIGHT_CTI_INTEGRATION_REGS=y
> +   CONFIG_CORESIGHT_LINKS_AND_SINKS=y
> +   CONFIG_CORESIGHT_LINK_AND_SINK_TMC=y
> +   CONFIG_CORESIGHT_CATU=y
> +   CONFIG_CORESIGHT_SINK_TPIU=y
> +   CONFIG_CORESIGHT_SINK_ETBV10=y
> +   CONFIG_CORESIGHT_SOURCE_ETM4X=y
> +   CONFIG_CORESIGHT_STM=y
> +   CONFIG_CORESIGHT_CPU_DEBUG=y
> +   CONFIG_CORESIGHT_CTI=y
> +   CONFIG_CORESIGHT_CTI_INTEGRATION_REGS=y
>   
>   Please refer to the kernel configuration help for more information.
>   
> @@ -88,18 +88,18 @@ tests will cover CoreSight (only if enabled and on ARM64). You
>   generally would run perf test from the tools/perf directory in the
>   kernel tree. Some tests will check some internal perf support like:
>   
> -    Check Arm CoreSight trace data recording and synthesized samples
> -    Check Arm SPE trace data recording and synthesized samples
> +* Check Arm CoreSight trace data recording and synthesized samples
> +* Check Arm SPE trace data recording and synthesized samples
>   
>   Some others will actually use perf record and some test binaries that
>   are in tests/shell/coresight and will collect traces to ensure a
>   minimum level of functionality is met. The scripts that launch these
>   tests are in the same directory. These will all look like:
>   
> -    CoreSight / ASM Pure Loop
> -    CoreSight / Memcpy 16k 10 Threads
> -    CoreSight / Thread Loop 10 Threads - Check TID
> -    ...
> +* CoreSight / ASM Pure Loop
> +* CoreSight / Memcpy 16k 10 Threads
> +* CoreSight / Thread Loop 10 Threads - Check TID
> +* etc.
>   
>   These perf record tests will not run if the tool binaries do not exist
>   in tests/shell/coresight/\*/ and will be skipped. If you do not have
> @@ -108,11 +108,11 @@ CoreSight support or remove these binaries in order to not have these
>   tests fail and have them skip instead.
>   
>   These tests will log historical results in the current working
> -directory (e.g. tools/perf) and will be named stats-\*.csv like:
> +directory (e.g. tools/perf) and will be named stats-\*.csv like::
>   
> -    stats-asm_pure_loop-out.csv
> -    stats-memcpy_thread-16k_10.csv
> -    ...
> +   stats-asm_pure_loop-out.csv
> +   stats-memcpy_thread-16k_10.csv
> +   ...
>   
>   These statistic files log some aspects of the AUX data sections in
>   the perf data output counting some numbers of certain encodings (a
> @@ -137,23 +137,23 @@ is inside.
>   
>   You can change where these csv logs are stored by setting the
>   PERF_TEST_CORESIGHT_STATDIR environment variable before running perf
> -test like:
> +test like::
>   
> -    export PERF_TEST_CORESIGHT_STATDIR=/var/tmp
> -    perf test
> +   export PERF_TEST_CORESIGHT_STATDIR=/var/tmp
> +   perf test
>   
>   They will also store resulting perf output data in the current
> -directory for later inspection like:
> +directory for later inspection like::
>   
> -    perf-asm_pure_loop-out.data
> -    perf-memcpy_thread-16k_10.data
> -    ...
> +   perf-asm_pure_loop-out.data
> +   perf-memcpy_thread-16k_10.data
> +   ...
>   
>   You can alter where the perf data files are stored by setting the
> -PERF_TEST_CORESIGHT_DATADIR environment variable such as:
> +PERF_TEST_CORESIGHT_DATADIR environment variable such as::
>   
> -    PERF_TEST_CORESIGHT_DATADIR=/var/tmp
> -    perf test
> +   PERF_TEST_CORESIGHT_DATADIR=/var/tmp
> +   perf test
>   
>   You may wish to set these above environment variables if you whish to
>   keep the output of tests outside of the current working directory for
> diff --git a/tools/perf/Documentation/arm-coresight.txt b/tools/perf/Documentation/arm-coresight.txt
> index f94743a4d161f2..c117fc50a2a956 100644
> --- a/tools/perf/Documentation/arm-coresight.txt
> +++ b/tools/perf/Documentation/arm-coresight.txt
> @@ -1,5 +1,5 @@
>   Arm CoreSight Support
>   =====================
>   
> -Please see docuentation in the central CoreSight location in the
> -kernel tree under Documentation/trace/coresight
> +For full documentation, see Documentation/trace/coresight/coresight-perf.rst
> +in the kernel tree.
> 
> ---- >8 ----
> 
> Note: since this is documentation patch, don't forget to Cc linux-doc list.
> I add it for you.

I'll split this out to a separate stand-alone patch from this series. I 
put docs in the place I was told to by the perf coresight maintainers, 
but that now complicates everyone to send the patch series to and so 
only the docs are relevant to the doc mailing list, so I'll split it off.

Re: [PATCH 14/14] perf test: Add relevant documentation about CoreSight testing

[Mike Leach]

Hi Carsten

On Fri, 8 Jul 2022 at 10:27, Carsten Haitzler
<carsten.haitzler@foss.arm.com> wrote:
>
>
>
> On 7/2/22 04:02, Bagas Sanjaya wrote:
> > On Fri, Jul 01, 2022 at 01:08:03PM +0100, carsten.haitzler@foss.arm.com wrote:
> >> From: "Carsten Haitzler (Rasterman)" <raster@rasterman.com>
> >>
> >
> > Hi Carsten,
> >
> > This doc patch can be improved, see below.
>
> I'll look at addressing the below - with some exceptions as they are not
> what you think they are.
>
> >> This adds/improves documentation helping people get started with
> >> CoreSight and perf as well as describing the testing and how it works.
> >>
> >
> > Use imperative mood instead of descriptive one for patch description.
> >
> >> Signed-off-by: Carsten Haitzler <carsten.haitzler@arm.com>
> >> ---
> >>   .../trace/coresight/coresight-perf.rst        | 160 ++++++++++++++++++
> >>   tools/perf/Documentation/arm-coresight.txt    |   5 +
> >>   2 files changed, 165 insertions(+)
> >>   create mode 100644 Documentation/trace/coresight/coresight-perf.rst
> >>   create mode 100644 tools/perf/Documentation/arm-coresight.txt
> >>
> >> diff --git a/Documentation/trace/coresight/coresight-perf.rst b/Documentation/trace/coresight/coresight-perf.rst
> >> new file mode 100644
> >> index 000000000000..de25082447dd
> >> --- /dev/null
> >> +++ b/Documentation/trace/coresight/coresight-perf.rst
> >> @@ -0,0 +1,160 @@
> >> +.. SPDX-License-Identifier: GPL-2.0
> >> +
> >> +================
> >> +CoreSight - Perf
> >> +================
> >> +
> >> +    :Author:   Carsten Haitzler <carsten.haitzler@arm.com>
> >> +    :Date:     June 29th, 2022
> >> +
> >> +Perf is able to locally access CoreSight trace data and store it to the
> >> +output perf data files. This data can then be later decoded to give the
> >> +instructions that were traced for debugging or profiling purposes. You
> >> +can log such data with a perf record command like:
> >> +
> >> +    perf record -e cs_etm//u testbinary
> >> +
> >
> > Use literal code block.
> >
> >> +This would run some test binary (testbinary) until it exits and record
> >> +a perf.data trace file. That file would have AUX sections if CoreSight
> >> +is working correctly. You can dump the content of this file as
> >> +readable text with a command like:
> >> +
> >> +    perf report --stdio --dump -i perf.data
> >> +
> >
> > Same as above.
> >
> >> +You should find some sections of this file have AUX data blocks like:
> >> +
> >> +    0x1e78 [0x30]: PERF_RECORD_AUXTRACE size: 0x11dd0  offset: 0  ref: 0x1b614fc1061b0ad1  idx: 0  tid: 531230  cpu: -1
> >> +
> >> +    . ... CoreSight ETM Trace data: size 73168 bytes
> >> +            Idx:0; ID:10;   I_ASYNC : Alignment Synchronisation.
> >> +              Idx:12; ID:10;  I_TRACE_INFO : Trace Info.; INFO=0x0 { CC.0 }
> >> +              Idx:17; ID:10;  I_ADDR_L_64IS0 : Address, Long, 64 bit, IS0.; Addr=0x0000000000000000;
> >> +              Idx:26; ID:10;  I_TRACE_ON : Trace On.
> >> +              Idx:27; ID:10;  I_ADDR_CTXT_L_64IS0 : Address & Context, Long, 64 bit, IS0.; Addr=0x0000FFFFB6069140; Ctxt: AArch64,EL0, NS;
> >> +              Idx:38; ID:10;  I_ATOM_F6 : Atom format 6.; EEEEEEEEEEEEEEEEEEEEEEEE
> >> +              Idx:39; ID:10;  I_ATOM_F6 : Atom format 6.; EEEEEEEEEEEEEEEEEEEEEEEE
> >> +              Idx:40; ID:10;  I_ATOM_F6 : Atom format 6.; EEEEEEEEEEEEEEEEEEEEEEEE
> >> +              Idx:41; ID:10;  I_ATOM_F6 : Atom format 6.; EEEEEEEEEEEN
> >> +              ...
> >> +
> >
> > Here too.
> >
> >> +If you see these above, then your system is tracing CoreSight data
> >> +correctly.
> >> +
> >> +To compile perf with CoreSight support in the tools/perf directory do
> >> +
> >> +    make CORESIGHT=1
> >> +
> >
> > Here too again.
> >
> >> +This requires OpenCSD to build. You may install distribution packages
> >> +for the support such as libopencsd and libopencsd-dev or download it
> >> +and build yourself. Upstream OpenCSD is located at:
> >> +
> >> +  https://github.com/Linaro/OpenCSD
> >> +
> >> +For complete information on building perf with CoreSight support and
> >> +more extensive usage look at:
> >> +
> >> +  https://github.com/Linaro/OpenCSD/blob/master/HOWTO.md
> >> +
> >> +
> >> +Kernel CoreSight Support
> >> +------------------------
> >> +
> >> +You will also want CoreSight support enabled in your kernel config.
> >> +Ensure it is enabled with:
> >> +
> >> +    CONFIG_CORESIGHT=y
> >> +
> >> +There are various other CoreSight options you probably also want
> >> +enabled like:
> >> +
> >> +    CONFIG_CORESIGHT_LINKS_AND_SINKS=y
> >> +    CONFIG_CORESIGHT_LINK_AND_SINK_TMC=y
> >> +    CONFIG_CORESIGHT_CATU=y
> >> +    CONFIG_CORESIGHT_SINK_TPIU=y
> >> +    CONFIG_CORESIGHT_SINK_ETBV10=y
> >> +    CONFIG_CORESIGHT_SOURCE_ETM4X=y
> >> +    CONFIG_CORESIGHT_STM=y
> >> +    CONFIG_CORESIGHT_CPU_DEBUG=y
> >> +    CONFIG_CORESIGHT_CTI=y
> >> +    CONFIG_CORESIGHT_CTI_INTEGRATION_REGS=y
> >> +
> >
> > Same as above again.
> >
> >> +Please refer to the kernel configuration help for more information.
> >> +
> >> +Perf test - Verify kernel and userspace perf CoreSight work
> >> +-----------------------------------------------------------
> >> +
> >> +When you run perf test, it will do a lot of self tests. Some of those
> >> +tests will cover CoreSight (only if enabled and on ARM64). You
> >> +generally would run perf test from the tools/perf directory in the
> >> +kernel tree. Some tests will check some internal perf support like:
> >> +
> >> +    Check Arm CoreSight trace data recording and synthesized samples
> >> +    Check Arm SPE trace data recording and synthesized samples
> >> +
> >
> > Use bullet lists.
>
> Actually this would be a code block - it is literally the stdout from
> perf test (just 2 lines of it for those tests).
>
> >> +Some others will actually use perf record and some test binaries that
> >> +are in tests/shell/coresight and will collect traces to ensure a
> >> +minimum level of functionality is met. The scripts that launch these
> >> +tests are in the same directory. These will all look like:
> >> +
> >> +    CoreSight / ASM Pure Loop
> >> +    CoreSight / Memcpy 16k 10 Threads
> >> +    CoreSight / Thread Loop 10 Threads - Check TID
> >> +    ...
> >> +
> >
> > Same as above.
>
> This too - a code block.
>
> >> +These perf record tests will not run if the tool binaries do not exist
> >> +in tests/shell/coresight/*/ and will be skipped. If you do not have
> >> +CoreSight support in hardware then either do not build perf with
> >> +CoreSight support or remove these binaries in order to not have these
> >> +tests fail and have them skip instead.
> >> +
> >> +These tests will log historical results in the current working
> >> +directory (e.g. tools/perf) and will be named stats-*.csv like:
> >> +
> >> +    stats-asm_pure_loop-out.csv
> >> +    stats-memcpy_thread-16k_10.csv
> >> +    ...
> >> +
> >
> > These above causes htmldocs warning (unescaped wildcard), so I have to apply
> > the fixup:
> >
> > ---- >8 ----
> >
> > diff --git a/Documentation/trace/coresight/coresight-perf.rst b/Documentation/trace/coresight/coresight-perf.rst
> > index de25082447dd50..a25fcda5c37c55 100644
> > --- a/Documentation/trace/coresight/coresight-perf.rst
> > +++ b/Documentation/trace/coresight/coresight-perf.rst
> > @@ -102,13 +102,13 @@ tests are in the same directory. These will all look like:
> >       ...
> >
> >   These perf record tests will not run if the tool binaries do not exist
> > -in tests/shell/coresight/*/ and will be skipped. If you do not have
> > +in tests/shell/coresight/\*/ and will be skipped. If you do not have
> >   CoreSight support in hardware then either do not build perf with
> >   CoreSight support or remove these binaries in order to not have these
> >   tests fail and have them skip instead.
> >
> >   These tests will log historical results in the current working
> > -directory (e.g. tools/perf) and will be named stats-*.csv like:
> > +directory (e.g. tools/perf) and will be named stats-\*.csv like:
> >
> >       stats-asm_pure_loop-out.csv
> >       stats-memcpy_thread-16k_10.csv
> >
> > ---- >8 ----
> >
> > Also, the output list above could be inside code block (since these
> > are output).
>
> Yup.
>
> >> +These statistic files log some aspects of the AUX data sections in
> >> +the perf data output counting some numbers of certain encodings (a
> >> +good way to know that it's working in a very simple way). One problem
> >> +with CoreSight is that given a large enough amount of data needing to
> >> +be logged, some of it can be lost due to the processor not waking up
> >> +in time to read out all the data from buffers etc.. You will notice
> >> +that the amount of data collected can vary a lot per run of perf test.
> >> +If you wish to see how this changes over time, simply run perf test
> >> +multiple times and all these csv files will have more and more data
> >> +appended to it that you can later examine, graph and otherwise use to
> >> +figure out if things have become worse or better.
> >> +
> >> +This means sometimes these tests fail as they don't capture all the
> >> +data needed. This is about tracking quality and amount of data
> >> +produced over time and to see when changes to the Linux kernel improve
> >> +quality of traces.
> >> +
> >> +Be aware that some of these tests take quite a while to run, specifically
> >> +in processing the perf data file and dumping contents to then examine what
> >> +is inside.
> >> +
> >> +You can change where these csv logs are stored by setting the
> >> +PERF_TEST_CORESIGHT_STATDIR environment variable before running perf
> >> +test like:
> >> +
> >> +    export PERF_TEST_CORESIGHT_STATDIR=/var/tmp
> >> +    perf test
> >> +
> >> +They will also store resulting perf output data in the current
> >> +directory for later inspection like:
> >> +
> >> +    perf-asm_pure_loop-out.data
> >> +    perf-memcpy_thread-16k_10.data
> >> +    ...
> >> +
> >> +You can alter where the perf data files are stored by setting the
> >> +PERF_TEST_CORESIGHT_DATADIR environment variable such as:
> >> +
> >> +    PERF_TEST_CORESIGHT_DATADIR=/var/tmp
> >> +    perf test
> >> +
> >
> > Use code block.
> >
> >> +You may wish to set these above environment variables if you whish to
> >> +keep the output of tests outside of the current working directory for
> >> +longer term storage and examination.
> >> diff --git a/tools/perf/Documentation/arm-coresight.txt b/tools/perf/Documentation/arm-coresight.txt
> >> new file mode 100644
> >> index 000000000000..f94743a4d161
> >> --- /dev/null
> >> +++ b/tools/perf/Documentation/arm-coresight.txt
> >> @@ -0,0 +1,5 @@
> >> +Arm CoreSight Support
> >> +=====================
> >> +
> >> +Please see docuentation in the central CoreSight location in the
> >> +kernel tree under Documentation/trace/coresight
> >
> > s/ducuentation/documentation/
> >
> > So here's the improv:
> >
> > ---- >8 ----
> >
> > diff --git a/Documentation/trace/coresight/coresight-perf.rst b/Documentation/trace/coresight/coresight-perf.rst
> > index a25fcda5c37c55..0dd4689a699ecd 100644
> > --- a/Documentation/trace/coresight/coresight-perf.rst
> > +++ b/Documentation/trace/coresight/coresight-perf.rst
> > @@ -10,37 +10,37 @@ CoreSight - Perf
> >   Perf is able to locally access CoreSight trace data and store it to the
> >   output perf data files. This data can then be later decoded to give the
> >   instructions that were traced for debugging or profiling purposes. You
> > -can log such data with a perf record command like:
> > +can log such data with a perf record command like::
> >
> > -    perf record -e cs_etm//u testbinary
> > +   perf record -e cs_etm//u testbinary
> >
> >   This would run some test binary (testbinary) until it exits and record
> >   a perf.data trace file. That file would have AUX sections if CoreSight
> >   is working correctly. You can dump the content of this file as
> > -readable text with a command like:
> > +readable text with a command like::
> >
> > -    perf report --stdio --dump -i perf.data
> > +   perf report --stdio --dump -i perf.data
> >
> > -You should find some sections of this file have AUX data blocks like:
> > +You should find some sections of this file have AUX data blocks like::
> >
> > -    0x1e78 [0x30]: PERF_RECORD_AUXTRACE size: 0x11dd0  offset: 0  ref: 0x1b614fc1061b0ad1  idx: 0  tid: 531230  cpu: -1
> > +   0x1e78 [0x30]: PERF_RECORD_AUXTRACE size: 0x11dd0  offset: 0  ref: 0x1b614fc1061b0ad1  idx: 0  tid: 531230  cpu: -1
> >
> > -    . ... CoreSight ETM Trace data: size 73168 bytes
> > -            Idx:0; ID:10;   I_ASYNC : Alignment Synchronisation.
> > -              Idx:12; ID:10;  I_TRACE_INFO : Trace Info.; INFO=0x0 { CC.0 }
> > -              Idx:17; ID:10;  I_ADDR_L_64IS0 : Address, Long, 64 bit, IS0.; Addr=0x0000000000000000;
> > -              Idx:26; ID:10;  I_TRACE_ON : Trace On.
> > -              Idx:27; ID:10;  I_ADDR_CTXT_L_64IS0 : Address & Context, Long, 64 bit, IS0.; Addr=0x0000FFFFB6069140; Ctxt: AArch64,EL0, NS;
> > -              Idx:38; ID:10;  I_ATOM_F6 : Atom format 6.; EEEEEEEEEEEEEEEEEEEEEEEE
> > -              Idx:39; ID:10;  I_ATOM_F6 : Atom format 6.; EEEEEEEEEEEEEEEEEEEEEEEE
> > -              Idx:40; ID:10;  I_ATOM_F6 : Atom format 6.; EEEEEEEEEEEEEEEEEEEEEEEE
> > -              Idx:41; ID:10;  I_ATOM_F6 : Atom format 6.; EEEEEEEEEEEN
> > -              ...
> > +   . ... CoreSight ETM Trace data: size 73168 bytes
> > +           Idx:0; ID:10;   I_ASYNC : Alignment Synchronisation.
> > +             Idx:12; ID:10;  I_TRACE_INFO : Trace Info.; INFO=0x0 { CC.0 }
> > +             Idx:17; ID:10;  I_ADDR_L_64IS0 : Address, Long, 64 bit, IS0.; Addr=0x0000000000000000;
> > +             Idx:26; ID:10;  I_TRACE_ON : Trace On.
> > +             Idx:27; ID:10;  I_ADDR_CTXT_L_64IS0 : Address & Context, Long, 64 bit, IS0.; Addr=0x0000FFFFB6069140; Ctxt: AArch64,EL0, NS;
> > +             Idx:38; ID:10;  I_ATOM_F6 : Atom format 6.; EEEEEEEEEEEEEEEEEEEEEEEE
> > +             Idx:39; ID:10;  I_ATOM_F6 : Atom format 6.; EEEEEEEEEEEEEEEEEEEEEEEE
> > +             Idx:40; ID:10;  I_ATOM_F6 : Atom format 6.; EEEEEEEEEEEEEEEEEEEEEEEE
> > +             Idx:41; ID:10;  I_ATOM_F6 : Atom format 6.; EEEEEEEEEEEN
> > +             ...
> >
> >   If you see these above, then your system is tracing CoreSight data
> >   correctly.
> >
> > -To compile perf with CoreSight support in the tools/perf directory do
> > +To compile perf with CoreSight support in the tools/perf directory do::
> >
> >       make CORESIGHT=1
> >
> > @@ -60,23 +60,23 @@ Kernel CoreSight Support
> >   ------------------------
> >
> >   You will also want CoreSight support enabled in your kernel config.
> > -Ensure it is enabled with:
> > +Ensure it is enabled with::
> >
> > -    CONFIG_CORESIGHT=y
> > +   CONFIG_CORESIGHT=y
> >
> >   There are various other CoreSight options you probably also want
> > -enabled like:
> > +enabled like::
> >
> > -    CONFIG_CORESIGHT_LINKS_AND_SINKS=y
> > -    CONFIG_CORESIGHT_LINK_AND_SINK_TMC=y
> > -    CONFIG_CORESIGHT_CATU=y
> > -    CONFIG_CORESIGHT_SINK_TPIU=y
> > -    CONFIG_CORESIGHT_SINK_ETBV10=y
> > -    CONFIG_CORESIGHT_SOURCE_ETM4X=y
> > -    CONFIG_CORESIGHT_STM=y
> > -    CONFIG_CORESIGHT_CPU_DEBUG=y
> > -    CONFIG_CORESIGHT_CTI=y
> > -    CONFIG_CORESIGHT_CTI_INTEGRATION_REGS=y
> > +   CONFIG_CORESIGHT_LINKS_AND_SINKS=y
> > +   CONFIG_CORESIGHT_LINK_AND_SINK_TMC=y
> > +   CONFIG_CORESIGHT_CATU=y
> > +   CONFIG_CORESIGHT_SINK_TPIU=y
> > +   CONFIG_CORESIGHT_SINK_ETBV10=y
> > +   CONFIG_CORESIGHT_SOURCE_ETM4X=y
> > +   CONFIG_CORESIGHT_STM=y
> > +   CONFIG_CORESIGHT_CPU_DEBUG=y
> > +   CONFIG_CORESIGHT_CTI=y
> > +   CONFIG_CORESIGHT_CTI_INTEGRATION_REGS=y
> >
> >   Please refer to the kernel configuration help for more information.
> >
> > @@ -88,18 +88,18 @@ tests will cover CoreSight (only if enabled and on ARM64). You
> >   generally would run perf test from the tools/perf directory in the
> >   kernel tree. Some tests will check some internal perf support like:
> >
> > -    Check Arm CoreSight trace data recording and synthesized samples
> > -    Check Arm SPE trace data recording and synthesized samples
> > +* Check Arm CoreSight trace data recording and synthesized samples
> > +* Check Arm SPE trace data recording and synthesized samples
> >
> >   Some others will actually use perf record and some test binaries that
> >   are in tests/shell/coresight and will collect traces to ensure a
> >   minimum level of functionality is met. The scripts that launch these
> >   tests are in the same directory. These will all look like:
> >
> > -    CoreSight / ASM Pure Loop
> > -    CoreSight / Memcpy 16k 10 Threads
> > -    CoreSight / Thread Loop 10 Threads - Check TID
> > -    ...
> > +* CoreSight / ASM Pure Loop
> > +* CoreSight / Memcpy 16k 10 Threads
> > +* CoreSight / Thread Loop 10 Threads - Check TID
> > +* etc.
> >
> >   These perf record tests will not run if the tool binaries do not exist
> >   in tests/shell/coresight/\*/ and will be skipped. If you do not have
> > @@ -108,11 +108,11 @@ CoreSight support or remove these binaries in order to not have these
> >   tests fail and have them skip instead.
> >
> >   These tests will log historical results in the current working
> > -directory (e.g. tools/perf) and will be named stats-\*.csv like:
> > +directory (e.g. tools/perf) and will be named stats-\*.csv like::
> >
> > -    stats-asm_pure_loop-out.csv
> > -    stats-memcpy_thread-16k_10.csv
> > -    ...
> > +   stats-asm_pure_loop-out.csv
> > +   stats-memcpy_thread-16k_10.csv
> > +   ...
> >
> >   These statistic files log some aspects of the AUX data sections in
> >   the perf data output counting some numbers of certain encodings (a
> > @@ -137,23 +137,23 @@ is inside.
> >
> >   You can change where these csv logs are stored by setting the
> >   PERF_TEST_CORESIGHT_STATDIR environment variable before running perf
> > -test like:
> > +test like::
> >
> > -    export PERF_TEST_CORESIGHT_STATDIR=/var/tmp
> > -    perf test
> > +   export PERF_TEST_CORESIGHT_STATDIR=/var/tmp
> > +   perf test
> >
> >   They will also store resulting perf output data in the current
> > -directory for later inspection like:
> > +directory for later inspection like::
> >
> > -    perf-asm_pure_loop-out.data
> > -    perf-memcpy_thread-16k_10.data
> > -    ...
> > +   perf-asm_pure_loop-out.data
> > +   perf-memcpy_thread-16k_10.data
> > +   ...
> >
> >   You can alter where the perf data files are stored by setting the
> > -PERF_TEST_CORESIGHT_DATADIR environment variable such as:
> > +PERF_TEST_CORESIGHT_DATADIR environment variable such as::
> >
> > -    PERF_TEST_CORESIGHT_DATADIR=/var/tmp
> > -    perf test
> > +   PERF_TEST_CORESIGHT_DATADIR=/var/tmp
> > +   perf test
> >
> >   You may wish to set these above environment variables if you whish to
> >   keep the output of tests outside of the current working directory for
> > diff --git a/tools/perf/Documentation/arm-coresight.txt b/tools/perf/Documentation/arm-coresight.txt
> > index f94743a4d161f2..c117fc50a2a956 100644
> > --- a/tools/perf/Documentation/arm-coresight.txt
> > +++ b/tools/perf/Documentation/arm-coresight.txt
> > @@ -1,5 +1,5 @@
> >   Arm CoreSight Support
> >   =====================
> >
> > -Please see docuentation in the central CoreSight location in the
> > -kernel tree under Documentation/trace/coresight
> > +For full documentation, see Documentation/trace/coresight/coresight-perf.rst
> > +in the kernel tree.
> >
> > ---- >8 ----
> >
> > Note: since this is documentation patch, don't forget to Cc linux-doc list.
> > I add it for you.
>
> I'll split this out to a separate stand-alone patch from this series. I
> put docs in the place I was told to by the perf coresight maintainers,
> but that now complicates everyone to send the patch series to and so
> only the docs are relevant to the doc mailing list, so I'll split it off.

Please keep the documentation with this patch set - both for
continuity sake and to make sure that the docs are available to the
reviewers.

The easiest way to ensure that just this patch goes to the doc list is to put a
Cc: linux-doc@vger.kernel.org
above your Signed-off-by in the patch description.

git send-email will then automatically read the Cc: list for the patch
and send to the doc list.

Regards

Mike

-- 
Mike Leach
Principal Engineer, ARM Ltd.
Manchester Design Centre. UK

Re: [PATCH 14/14] perf test: Add relevant documentation about CoreSight testing

[kernel test robot]

Hi,

Thank you for the patch! Perhaps something to improve:

[auto build test WARNING on acme/perf/core]
[also build test WARNING on tip/perf/core linus/master v5.19-rc5 next-20220705]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]

url:    https://github.com/intel-lab-lkp/linux/commits/carsten-haitzler-foss-arm-com/perf-test-Refactor-shell-tests-allowing-subdirs/20220701-210837
base:   https://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git perf/core
reproduce: make htmldocs

If you fix the issue, kindly add following tag where applicable
Reported-by: kernel test robot <lkp@intel.com>

All warnings (new ones prefixed by >>):

>> Documentation/trace/coresight/coresight-perf.rst:104: WARNING: Inline emphasis start-string without end-string.

vim +104 Documentation/trace/coresight/coresight-perf.rst

    98	
    99	    CoreSight / ASM Pure Loop
   100	    CoreSight / Memcpy 16k 10 Threads
   101	    CoreSight / Thread Loop 10 Threads - Check TID
   102	    ...
   103	
 > 104	These perf record tests will not run if the tool binaries do not exist
   105	in tests/shell/coresight/*/ and will be skipped. If you do not have
   106	CoreSight support in hardware then either do not build perf with
   107	CoreSight support or remove these binaries in order to not have these
   108	tests fail and have them skip instead.
   109	

-- 
0-DAY CI Kernel Test Service
https://01.org/lkp