[PATCH 0/8] perf buildid-cache: Add clean command

[PATCH 0/8] perf buildid-cache: Add clean command

[Jiri Olsa]

hi,
adding 'perf buildid-cache clean' command to allow removal
files from '~/.debug` cache plus other fixes.

Basically the clean command allows to display/remove cache
files/sizes like:

Display cache files older than 3 days:
  $ perf buildid-cache clean 3d

Remove cache files older than 3 days:
  $ perf buildid-cache clean 3d -r
    
Total cache removal:
  $ perf buildid-cache clean -r

Remove and display items bigger than 200M
  $ perf buildid-cache clean -r -a 200M


Also available in here:
  git://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git
  perf/core_buildid_cache_clean

thanks,
jirka


Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Corey Ashford <cjashfor@linux.vnet.ibm.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Stephane Eranian <eranian@google.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
Jiri Olsa (8):
      perf tools: Use single strcmp call instead of two
      perf tools: Remove extra debugdir variables
      perf tools: Add --buildid-dir option to set cache directory
      perf buildid-cache: Add clean command
      perf buildid-cache: Add automated tests
      perf buildid cache: Fix -a segfault
      perf buildid-cache: Fix kallsyms removal
      perf buildid-cache: Try to remove empty directories

 tools/perf/Documentation/perf-buildid-cache.txt |  59 +++++++++++
 tools/perf/Documentation/perf.txt               |   4 +
 tools/perf/Makefile.perf                        |   7 +-
 tools/perf/builtin-buildid-cache.c              | 501 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++--
 tools/perf/perf.c                               |  14 ++-
 tools/perf/tests/buildid-cache.c                |  88 ++++++++++++++++
 tools/perf/tests/buildid-cache.sh               |  60 +++++++++++
 tools/perf/tests/builtin-test.c                 |   8 ++
 tools/perf/tests/tests.h                        |   1 +
 tools/perf/util/build-id.c                      |  53 ++++++++--
 tools/perf/util/config.c                        |  10 +-
 tools/perf/util/util.h                          |   2 +-
 12 files changed, 781 insertions(+), 26 deletions(-)
 create mode 100644 tools/perf/tests/buildid-cache.c
 create mode 100755 tools/perf/tests/buildid-cache.sh

[PATCH 1/8] perf tools: Use single strcmp call instead of two

[Jiri Olsa]

There's no need to use 2 strcmp calls, one is enough.

Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Corey Ashford <cjashfor@linux.vnet.ibm.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Stephane Eranian <eranian@google.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 tools/perf/util/config.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tools/perf/util/config.c b/tools/perf/util/config.c
index 57ff826f150b..c802236b2bfe 100644
--- a/tools/perf/util/config.c
+++ b/tools/perf/util/config.c
@@ -522,7 +522,7 @@ static int buildid_dir_command_config(const char *var, const char *value,
 	const char *v;
 
 	/* same dir for all commands */
-	if (!prefixcmp(var, "buildid.") && !strcmp(var + 8, "dir")) {
+	if (!strcmp(var, "buildid.dir")) {
 		v = perf_config_dirname(var, value);
 		if (!v)
 			return -1;
-- 
1.9.3

[tip:perf/urgent] perf tools: Use single strcmp call instead of two

[tip-bot for Jiri Olsa]

Commit-ID:  cfd31d85bb42b96449157bd57c638dc779070753
Gitweb:     http://git.kernel.org/tip/cfd31d85bb42b96449157bd57c638dc779070753
Author:     Jiri Olsa <jolsa@kernel.org>
AuthorDate: Mon, 1 Dec 2014 20:06:22 +0100
Committer:  Arnaldo Carvalho de Melo <acme@redhat.com>
CommitDate: Tue, 9 Dec 2014 09:14:33 -0300

perf tools: Use single strcmp call instead of two

There's no need to use 2 strcmp calls, one is enough.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Cc: Corey Ashford <cjashfor@linux.vnet.ibm.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Stephane Eranian <eranian@google.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Link: http://lkml.kernel.org/r/1417460789-13874-2-git-send-email-jolsa@kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
 tools/perf/util/config.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tools/perf/util/config.c b/tools/perf/util/config.c
index 57ff826..c802236 100644
--- a/tools/perf/util/config.c
+++ b/tools/perf/util/config.c
@@ -522,7 +522,7 @@ static int buildid_dir_command_config(const char *var, const char *value,
 	const char *v;
 
 	/* same dir for all commands */
-	if (!prefixcmp(var, "buildid.") && !strcmp(var + 8, "dir")) {
+	if (!strcmp(var, "buildid.dir")) {
 		v = perf_config_dirname(var, value);
 		if (!v)
 			return -1;

[PATCH 2/8] perf tools: Remove extra debugdir variables

[Jiri Olsa]

There's no need to copy over the buildid_dir into separate
variable with no change.

This is leftover from commit:
  45de34bbe3e1 perf buildid: add perfconfig option to specify buildid cache dir

that added global buildid_dir variable that holds cache
directory, but did not cleanup the debugdir copies.

Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Corey Ashford <cjashfor@linux.vnet.ibm.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Stephane Eranian <eranian@google.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 tools/perf/builtin-buildid-cache.c | 11 ++++-------
 tools/perf/util/build-id.c         |  9 +++------
 2 files changed, 7 insertions(+), 13 deletions(-)

diff --git a/tools/perf/builtin-buildid-cache.c b/tools/perf/builtin-buildid-cache.c
index 70385756da63..29f24c071bc6 100644
--- a/tools/perf/builtin-buildid-cache.c
+++ b/tools/perf/builtin-buildid-cache.c
@@ -285,7 +285,6 @@ int cmd_buildid_cache(int argc, const char **argv,
 	struct str_node *pos;
 	int ret = 0;
 	bool force = false;
-	char debugdir[PATH_MAX];
 	char const *add_name_list_str = NULL,
 		   *remove_name_list_str = NULL,
 		   *missing_filename = NULL,
@@ -335,13 +334,11 @@ int cmd_buildid_cache(int argc, const char **argv,
 
 	setup_pager();
 
-	snprintf(debugdir, sizeof(debugdir), "%s", buildid_dir);
-
 	if (add_name_list_str) {
 		list = strlist__new(true, add_name_list_str);
 		if (list) {
 			strlist__for_each(pos, list)
-				if (build_id_cache__add_file(pos->s, debugdir)) {
+				if (build_id_cache__add_file(pos->s, buildid_dir)) {
 					if (errno == EEXIST) {
 						pr_debug("%s already in the cache\n",
 							 pos->s);
@@ -359,7 +356,7 @@ int cmd_buildid_cache(int argc, const char **argv,
 		list = strlist__new(true, remove_name_list_str);
 		if (list) {
 			strlist__for_each(pos, list)
-				if (build_id_cache__remove_file(pos->s, debugdir)) {
+				if (build_id_cache__remove_file(pos->s, buildid_dir)) {
 					if (errno == ENOENT) {
 						pr_debug("%s wasn't in the cache\n",
 							 pos->s);
@@ -380,7 +377,7 @@ int cmd_buildid_cache(int argc, const char **argv,
 		list = strlist__new(true, update_name_list_str);
 		if (list) {
 			strlist__for_each(pos, list)
-				if (build_id_cache__update_file(pos->s, debugdir)) {
+				if (build_id_cache__update_file(pos->s, buildid_dir)) {
 					if (errno == ENOENT) {
 						pr_debug("%s wasn't in the cache\n",
 							 pos->s);
@@ -395,7 +392,7 @@ int cmd_buildid_cache(int argc, const char **argv,
 	}
 
 	if (kcore_filename &&
-	    build_id_cache__add_kcore(kcore_filename, debugdir, force))
+	    build_id_cache__add_kcore(kcore_filename, buildid_dir, force))
 		pr_warning("Couldn't add %s\n", kcore_filename);
 
 out:
diff --git a/tools/perf/util/build-id.c b/tools/perf/util/build-id.c
index e8d79e5bfaf7..0c72680a977f 100644
--- a/tools/perf/util/build-id.c
+++ b/tools/perf/util/build-id.c
@@ -410,21 +410,18 @@ int perf_session__cache_build_ids(struct perf_session *session)
 {
 	struct rb_node *nd;
 	int ret;
-	char debugdir[PATH_MAX];
 
 	if (no_buildid_cache)
 		return 0;
 
-	snprintf(debugdir, sizeof(debugdir), "%s", buildid_dir);
-
-	if (mkdir(debugdir, 0755) != 0 && errno != EEXIST)
+	if (mkdir(buildid_dir, 0755) != 0 && errno != EEXIST)
 		return -1;
 
-	ret = machine__cache_build_ids(&session->machines.host, debugdir);
+	ret = machine__cache_build_ids(&session->machines.host, buildid_dir);
 
 	for (nd = rb_first(&session->machines.guests); nd; nd = rb_next(nd)) {
 		struct machine *pos = rb_entry(nd, struct machine, rb_node);
-		ret |= machine__cache_build_ids(pos, debugdir);
+		ret |= machine__cache_build_ids(pos, buildid_dir);
 	}
 	return ret ? -1 : 0;
 }
-- 
1.9.3

[tip:perf/urgent] perf buildid-cache: Remove extra debugdir variables

[tip-bot for Jiri Olsa]

Commit-ID:  498922adf1173ddeebd155f82646d4a9d518d606
Gitweb:     http://git.kernel.org/tip/498922adf1173ddeebd155f82646d4a9d518d606
Author:     Jiri Olsa <jolsa@kernel.org>
AuthorDate: Mon, 1 Dec 2014 20:06:23 +0100
Committer:  Arnaldo Carvalho de Melo <acme@redhat.com>
CommitDate: Tue, 9 Dec 2014 09:14:34 -0300

perf buildid-cache: Remove extra debugdir variables

There's no need to copy over the buildid_dir into separate variable with
no change.

This is leftover from commit:
  45de34bbe3e1 perf buildid: add perfconfig option to specify buildid cache dir

that added global buildid_dir variable that holds cache directory, but
did not cleanup the debugdir copies.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Cc: Corey Ashford <cjashfor@linux.vnet.ibm.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Stephane Eranian <eranian@google.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Link: http://lkml.kernel.org/r/1417460789-13874-3-git-send-email-jolsa@kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
 tools/perf/builtin-buildid-cache.c | 11 ++++-------
 tools/perf/util/build-id.c         |  9 +++------
 2 files changed, 7 insertions(+), 13 deletions(-)

diff --git a/tools/perf/builtin-buildid-cache.c b/tools/perf/builtin-buildid-cache.c
index 7038575..29f24c0 100644
--- a/tools/perf/builtin-buildid-cache.c
+++ b/tools/perf/builtin-buildid-cache.c
@@ -285,7 +285,6 @@ int cmd_buildid_cache(int argc, const char **argv,
 	struct str_node *pos;
 	int ret = 0;
 	bool force = false;
-	char debugdir[PATH_MAX];
 	char const *add_name_list_str = NULL,
 		   *remove_name_list_str = NULL,
 		   *missing_filename = NULL,
@@ -335,13 +334,11 @@ int cmd_buildid_cache(int argc, const char **argv,
 
 	setup_pager();
 
-	snprintf(debugdir, sizeof(debugdir), "%s", buildid_dir);
-
 	if (add_name_list_str) {
 		list = strlist__new(true, add_name_list_str);
 		if (list) {
 			strlist__for_each(pos, list)
-				if (build_id_cache__add_file(pos->s, debugdir)) {
+				if (build_id_cache__add_file(pos->s, buildid_dir)) {
 					if (errno == EEXIST) {
 						pr_debug("%s already in the cache\n",
 							 pos->s);
@@ -359,7 +356,7 @@ int cmd_buildid_cache(int argc, const char **argv,
 		list = strlist__new(true, remove_name_list_str);
 		if (list) {
 			strlist__for_each(pos, list)
-				if (build_id_cache__remove_file(pos->s, debugdir)) {
+				if (build_id_cache__remove_file(pos->s, buildid_dir)) {
 					if (errno == ENOENT) {
 						pr_debug("%s wasn't in the cache\n",
 							 pos->s);
@@ -380,7 +377,7 @@ int cmd_buildid_cache(int argc, const char **argv,
 		list = strlist__new(true, update_name_list_str);
 		if (list) {
 			strlist__for_each(pos, list)
-				if (build_id_cache__update_file(pos->s, debugdir)) {
+				if (build_id_cache__update_file(pos->s, buildid_dir)) {
 					if (errno == ENOENT) {
 						pr_debug("%s wasn't in the cache\n",
 							 pos->s);
@@ -395,7 +392,7 @@ int cmd_buildid_cache(int argc, const char **argv,
 	}
 
 	if (kcore_filename &&
-	    build_id_cache__add_kcore(kcore_filename, debugdir, force))
+	    build_id_cache__add_kcore(kcore_filename, buildid_dir, force))
 		pr_warning("Couldn't add %s\n", kcore_filename);
 
 out:
diff --git a/tools/perf/util/build-id.c b/tools/perf/util/build-id.c
index e8d79e5..0c72680 100644
--- a/tools/perf/util/build-id.c
+++ b/tools/perf/util/build-id.c
@@ -410,21 +410,18 @@ int perf_session__cache_build_ids(struct perf_session *session)
 {
 	struct rb_node *nd;
 	int ret;
-	char debugdir[PATH_MAX];
 
 	if (no_buildid_cache)
 		return 0;
 
-	snprintf(debugdir, sizeof(debugdir), "%s", buildid_dir);
-
-	if (mkdir(debugdir, 0755) != 0 && errno != EEXIST)
+	if (mkdir(buildid_dir, 0755) != 0 && errno != EEXIST)
 		return -1;
 
-	ret = machine__cache_build_ids(&session->machines.host, debugdir);
+	ret = machine__cache_build_ids(&session->machines.host, buildid_dir);
 
 	for (nd = rb_first(&session->machines.guests); nd; nd = rb_next(nd)) {
 		struct machine *pos = rb_entry(nd, struct machine, rb_node);
-		ret |= machine__cache_build_ids(pos, debugdir);
+		ret |= machine__cache_build_ids(pos, buildid_dir);
 	}
 	return ret ? -1 : 0;
 }

[PATCH 3/8] perf tools: Add --buildid-dir option to set cache directory

[Jiri Olsa]

Adding --buildid-dir to be able to set specific cache
directory. It's going to be handy for buildid tests
coming in shortly.

Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Corey Ashford <cjashfor@linux.vnet.ibm.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Stephane Eranian <eranian@google.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 tools/perf/Documentation/perf.txt |  4 ++++
 tools/perf/perf.c                 | 14 ++++++++++++--
 tools/perf/util/config.c          |  8 +++++---
 tools/perf/util/util.h            |  2 +-
 4 files changed, 22 insertions(+), 6 deletions(-)

diff --git a/tools/perf/Documentation/perf.txt b/tools/perf/Documentation/perf.txt
index d240bb2e5b22..1e8e400b4493 100644
--- a/tools/perf/Documentation/perf.txt
+++ b/tools/perf/Documentation/perf.txt
@@ -18,6 +18,10 @@ OPTIONS
 	  --debug verbose   # sets verbose = 1
 	  --debug verbose=2 # sets verbose = 2
 
+--buildid-dir::
+	Setup buildid cache directory. It has higher priority than
+	buildid.dir config file option.
+
 DESCRIPTION
 -----------
 Performance counters for Linux are a new kernel-based subsystem
diff --git a/tools/perf/perf.c b/tools/perf/perf.c
index 452a8474d29d..3700a7faca6c 100644
--- a/tools/perf/perf.c
+++ b/tools/perf/perf.c
@@ -200,6 +200,16 @@ static int handle_options(const char ***argv, int *argc, int *envchanged)
 				*envchanged = 1;
 			(*argv)++;
 			(*argc)--;
+		} else if (!strcmp(cmd, "--buildid-dir")) {
+			if (*argc < 2) {
+				fprintf(stderr, "No directory given for --buildid-dir.\n");
+				usage(perf_usage_string);
+			}
+			set_buildid_dir((*argv)[1]);
+			if (envchanged)
+				*envchanged = 1;
+			(*argv)++;
+			(*argc)--;
 		} else if (!prefixcmp(cmd, CMD_DEBUGFS_DIR)) {
 			perf_debugfs_set_path(cmd + strlen(CMD_DEBUGFS_DIR));
 			fprintf(stderr, "dir: %s\n", debugfs_mountpoint);
@@ -499,7 +509,7 @@ int main(int argc, const char **argv)
 	}
 	if (!prefixcmp(cmd, "trace")) {
 #ifdef HAVE_LIBAUDIT_SUPPORT
-		set_buildid_dir();
+		set_buildid_dir(NULL);
 		setup_path();
 		argv[0] = "trace";
 		return cmd_trace(argc, argv, NULL);
@@ -514,7 +524,7 @@ int main(int argc, const char **argv)
 	argc--;
 	handle_options(&argv, &argc, NULL);
 	commit_pager_choice();
-	set_buildid_dir();
+	set_buildid_dir(NULL);
 
 	if (argc > 0) {
 		if (!prefixcmp(argv[0], "--"))
diff --git a/tools/perf/util/config.c b/tools/perf/util/config.c
index c802236b2bfe..e18f653cd7db 100644
--- a/tools/perf/util/config.c
+++ b/tools/perf/util/config.c
@@ -539,12 +539,14 @@ static void check_buildid_dir_config(void)
 	perf_config(buildid_dir_command_config, &c);
 }
 
-void set_buildid_dir(void)
+void set_buildid_dir(const char *dir)
 {
-	buildid_dir[0] = '\0';
+	if (dir)
+		scnprintf(buildid_dir, MAXPATHLEN-1, "%s", dir);
 
 	/* try config file */
-	check_buildid_dir_config();
+	if (buildid_dir[0] == '\0')
+		check_buildid_dir_config();
 
 	/* default to $HOME/.debug */
 	if (buildid_dir[0] == '\0') {
diff --git a/tools/perf/util/util.h b/tools/perf/util/util.h
index 419bee030f83..abc445ee4f60 100644
--- a/tools/perf/util/util.h
+++ b/tools/perf/util/util.h
@@ -153,7 +153,7 @@ extern void warning(const char *err, ...) __attribute__((format (printf, 1, 2)))
 extern void set_die_routine(void (*routine)(const char *err, va_list params) NORETURN);
 
 extern int prefixcmp(const char *str, const char *prefix);
-extern void set_buildid_dir(void);
+extern void set_buildid_dir(const char *dir);
 
 static inline const char *skip_prefix(const char *str, const char *prefix)
 {
-- 
1.9.3

Re: [PATCH 3/8] perf tools: Add --buildid-dir option to set cache directory

[Arnaldo Carvalho de Melo]

Em Mon, Dec 01, 2014 at 08:06:24PM +0100, Jiri Olsa escreveu:
> Adding --buildid-dir to be able to set specific cache
> directory. It's going to be handy for buildid tests
> coming in shortly.

Thanks, applied.

[tip:perf/urgent] perf tools: Add --buildid-dir option to set cache directory

[tip-bot for Jiri Olsa]

Commit-ID:  99ce8e9fce99147f865cda8a8e471900518c9a49
Gitweb:     http://git.kernel.org/tip/99ce8e9fce99147f865cda8a8e471900518c9a49
Author:     Jiri Olsa <jolsa@kernel.org>
AuthorDate: Mon, 1 Dec 2014 20:06:24 +0100
Committer:  Arnaldo Carvalho de Melo <acme@redhat.com>
CommitDate: Tue, 9 Dec 2014 09:14:35 -0300

perf tools: Add --buildid-dir option to set cache directory

Adding --buildid-dir to be able to set specific cache directory. It's
going to be handy for buildid tests coming in shortly.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Cc: Corey Ashford <cjashfor@linux.vnet.ibm.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Stephane Eranian <eranian@google.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Link: http://lkml.kernel.org/r/1417460789-13874-4-git-send-email-jolsa@kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
 tools/perf/Documentation/perf.txt |  4 ++++
 tools/perf/perf.c                 | 14 ++++++++++++--
 tools/perf/util/config.c          |  8 +++++---
 tools/perf/util/util.h            |  2 +-
 4 files changed, 22 insertions(+), 6 deletions(-)

diff --git a/tools/perf/Documentation/perf.txt b/tools/perf/Documentation/perf.txt
index d240bb2..1e8e400 100644
--- a/tools/perf/Documentation/perf.txt
+++ b/tools/perf/Documentation/perf.txt
@@ -18,6 +18,10 @@ OPTIONS
 	  --debug verbose   # sets verbose = 1
 	  --debug verbose=2 # sets verbose = 2
 
+--buildid-dir::
+	Setup buildid cache directory. It has higher priority than
+	buildid.dir config file option.
+
 DESCRIPTION
 -----------
 Performance counters for Linux are a new kernel-based subsystem
diff --git a/tools/perf/perf.c b/tools/perf/perf.c
index 452a847..3700a7f 100644
--- a/tools/perf/perf.c
+++ b/tools/perf/perf.c
@@ -200,6 +200,16 @@ static int handle_options(const char ***argv, int *argc, int *envchanged)
 				*envchanged = 1;
 			(*argv)++;
 			(*argc)--;
+		} else if (!strcmp(cmd, "--buildid-dir")) {
+			if (*argc < 2) {
+				fprintf(stderr, "No directory given for --buildid-dir.\n");
+				usage(perf_usage_string);
+			}
+			set_buildid_dir((*argv)[1]);
+			if (envchanged)
+				*envchanged = 1;
+			(*argv)++;
+			(*argc)--;
 		} else if (!prefixcmp(cmd, CMD_DEBUGFS_DIR)) {
 			perf_debugfs_set_path(cmd + strlen(CMD_DEBUGFS_DIR));
 			fprintf(stderr, "dir: %s\n", debugfs_mountpoint);
@@ -499,7 +509,7 @@ int main(int argc, const char **argv)
 	}
 	if (!prefixcmp(cmd, "trace")) {
 #ifdef HAVE_LIBAUDIT_SUPPORT
-		set_buildid_dir();
+		set_buildid_dir(NULL);
 		setup_path();
 		argv[0] = "trace";
 		return cmd_trace(argc, argv, NULL);
@@ -514,7 +524,7 @@ int main(int argc, const char **argv)
 	argc--;
 	handle_options(&argv, &argc, NULL);
 	commit_pager_choice();
-	set_buildid_dir();
+	set_buildid_dir(NULL);
 
 	if (argc > 0) {
 		if (!prefixcmp(argv[0], "--"))
diff --git a/tools/perf/util/config.c b/tools/perf/util/config.c
index c802236..e18f653 100644
--- a/tools/perf/util/config.c
+++ b/tools/perf/util/config.c
@@ -539,12 +539,14 @@ static void check_buildid_dir_config(void)
 	perf_config(buildid_dir_command_config, &c);
 }
 
-void set_buildid_dir(void)
+void set_buildid_dir(const char *dir)
 {
-	buildid_dir[0] = '\0';
+	if (dir)
+		scnprintf(buildid_dir, MAXPATHLEN-1, "%s", dir);
 
 	/* try config file */
-	check_buildid_dir_config();
+	if (buildid_dir[0] == '\0')
+		check_buildid_dir_config();
 
 	/* default to $HOME/.debug */
 	if (buildid_dir[0] == '\0') {
diff --git a/tools/perf/util/util.h b/tools/perf/util/util.h
index 419bee0..abc445e 100644
--- a/tools/perf/util/util.h
+++ b/tools/perf/util/util.h
@@ -153,7 +153,7 @@ extern void warning(const char *err, ...) __attribute__((format (printf, 1, 2)))
 extern void set_die_routine(void (*routine)(const char *err, va_list params) NORETURN);
 
 extern int prefixcmp(const char *str, const char *prefix);
-extern void set_buildid_dir(void);
+extern void set_buildid_dir(const char *dir);
 
 static inline const char *skip_prefix(const char *str, const char *prefix)
 {

[PATCH 4/8] perf buildid-cache: Add clean command

[Jiri Olsa]

The 'perf buildid-cache clean' command provides means to clean the cache
directory. Unless the '-r' option is specified it displays the contents
(with size) of the cache. You can also specify time or size limit as
a cache filer (see CLEAN LIMIT section).

User can specify size or time limit as a filter to display cache files.
The syntax of the limit is, time:
  $ perf buildid-cache clean 1d  # displays files older than 1 day
  $ perf buildid-cache clean 4w  # displays files older than 4 weeks
  $ perf buildid-cache clean 2m  # displays files older than 2 months
  $ perf buildid-cache clean 1y  # displays files older than 1 year

or size:
  $ perf buildid-cache clean 100B  # displays files with size >= 100 B
  $ perf buildid-cache clean 4K    # displays files with size >= 4 KB
  $ perf buildid-cache clean 2M    # displays files with size >= 2 MB
  $ perf buildid-cache clean 1G    # displays files with size >= 1 GB

Few examples:

Display cache files older than 3 days and sort them by time:
  $ perf buildid-cache clean --time 3d

Total cache removal:
  $ perf buildid-cache clean -r

Remove items older than 2 weeks
  $ perf buildid-cache clean -r 2w

Remove and display items bigger than 200M
  $ perf buildid-cache clean -r -a 200M

Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Corey Ashford <cjashfor@linux.vnet.ibm.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Stephane Eranian <eranian@google.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 tools/perf/Documentation/perf-buildid-cache.txt |  59 +++
 tools/perf/builtin-buildid-cache.c              | 454 +++++++++++++++++++++++-
 2 files changed, 512 insertions(+), 1 deletion(-)

diff --git a/tools/perf/Documentation/perf-buildid-cache.txt b/tools/perf/Documentation/perf-buildid-cache.txt
index fd77d81ea748..dc605d4ee9e7 100644
--- a/tools/perf/Documentation/perf-buildid-cache.txt
+++ b/tools/perf/Documentation/perf-buildid-cache.txt
@@ -9,6 +9,7 @@ SYNOPSIS
 --------
 [verse]
 'perf buildid-cache <options>'
+'perf buildid-cache <options> clean [<options>] [limit]
 
 DESCRIPTION
 -----------
@@ -16,6 +17,11 @@ This command manages the build-id cache. It can add and remove files to/from
 the cache. In the future it should as well purge older entries, set upper
 limits for the space used by the cache, etc.
 
+The 'perf buildid-cache clean' command provides means to clean the cache
+directory. Unless the '-r' option is specified it displays the contents
+(with size) of the cache. You can also specify time or size limit as
+a cache filer (see CLEAN LIMIT section).
+
 OPTIONS
 -------
 -a::
@@ -48,6 +54,59 @@ OPTIONS
 --verbose::
 	Be more verbose.
 
+
+CLEAN OPTIONS
+-------------
+-a::
+--all::
+	Display each cache file separately.
+
+-r::
+--remove::
+	Remove all files displayed.
+
+--size::
+	Sort files by size (default).
+
+--time::
+	Sort files by time.
+
+-v::
+--verbose::
+	Be more verbose.
+
+
+CLEAN LIMIT
+-----------
+User can specify size or time limit as a filter to display cache files.
+The syntax of the limit is, time:
+  $ perf buildid-cache clean 1d  # displays files older than 1 day
+  $ perf buildid-cache clean 4w  # displays files older than 4 weeks
+  $ perf buildid-cache clean 2m  # displays files older than 2 months
+  $ perf buildid-cache clean 1y  # displays files older than 1 year
+
+or size:
+  $ perf buildid-cache clean 100B # displays files with size >= 100 B
+  $ perf buildid-cache clean 4K   # displays files with size >= 4 KB
+  $ perf buildid-cache clean 2M   # displays files with size >= 2 MB
+  $ perf buildid-cache clean 1G   # displays files with size >= 1 GB
+
+
+EXAMPLES
+--------
+Display cache files older than 3 days and sort them by time:
+$ perf buildid-cache clean --time 3d
+
+Total cache removal:
+$ perf buildid-cache clean -r
+
+Remove items older than 2 weeks
+$ perf buildid-cache clean -r 2w
+
+Remove and display items bigger than 200M
+$ perf buildid-cache clean -r -a 200M
+
+
 SEE ALSO
 --------
 linkperf:perf-record[1], linkperf:perf-report[1], linkperf:perf-buildid-list[1]
diff --git a/tools/perf/builtin-buildid-cache.c b/tools/perf/builtin-buildid-cache.c
index 29f24c071bc6..184955ec8a83 100644
--- a/tools/perf/builtin-buildid-cache.c
+++ b/tools/perf/builtin-buildid-cache.c
@@ -8,9 +8,13 @@
  */
 #include <sys/types.h>
 #include <sys/time.h>
+#include <asm/bug.h>
+#include <linux/rbtree.h>
 #include <time.h>
 #include <dirent.h>
 #include <unistd.h>
+#include <ftw.h>
+#include <time.h>
 #include "builtin.h"
 #include "perf.h"
 #include "util/cache.h"
@@ -278,6 +282,450 @@ static int build_id_cache__update_file(const char *filename,
 	return err;
 }
 
+enum cache_sort {
+	CACHE_SORT__NONE,
+	CACHE_SORT__SIZE,
+	CACHE_SORT__TIME,
+};
+
+enum cache_disp {
+	CACHE_DISP__NONE,
+	CACHE_DISP__ALL,
+};
+
+enum cache_limit {
+	CACHE_LIMIT__NONE,
+	CACHE_LIMIT__SIZE,
+	CACHE_LIMIT__TIME,
+};
+
+enum cache_remove {
+	CACHE_REMOVE__NONE,
+	CACHE_REMOVE__SINGLE,
+	CACHE_REMOVE__TOTAL,
+};
+
+struct cache_file {
+	char		*path;
+	u64		 size;
+	time_t		 time;
+	struct rb_node	 rb_node;
+};
+
+static struct rb_root cache_files;
+static struct cache_file *cache_total;
+
+static enum cache_sort cache_sort     = CACHE_SORT__NONE;
+static enum cache_disp cache_disp     = CACHE_DISP__NONE;
+static enum cache_limit cache_limit   = CACHE_LIMIT__NONE;
+static enum cache_remove cache_remove = CACHE_REMOVE__NONE;
+
+static time_t cache_limit__time;
+static u64    cache_limit__size;
+
+static struct cache_file*
+cache_file__alloc(const char *path, const struct stat *st)
+{
+	struct cache_file *file = zalloc(sizeof(*file));
+
+	if (file) {
+		file->path = strdup(path);
+		file->size = st ? st->st_size : 0;
+		file->time = st ? st->st_atime : 0;
+		RB_CLEAR_NODE(&file->rb_node);
+	}
+	return file;
+}
+
+static void cache_file__release(struct cache_file *file)
+{
+	free(file->path);
+	free(file);
+}
+
+static int cmp_u64(u64 a, u64 b)
+{
+	return a > b ? -1 : a == b ? 0 : 1;
+}
+
+static int cache_file__cmp(struct cache_file *a, struct cache_file *b)
+{
+	switch (cache_sort) {
+	case CACHE_SORT__SIZE:
+		return cmp_u64(a->size, b->size);
+	case CACHE_SORT__TIME:
+		return cmp_u64((u64) a->time, (u64) b->time);
+	case CACHE_SORT__NONE:
+	default:
+		pr_err("internal cache_sort bug\n");
+	}
+	return 0;
+}
+
+static void cache_files__add(struct cache_file *file)
+{
+	struct rb_node **p = &cache_files.rb_node;
+	struct rb_node *parent = NULL;
+	struct cache_file *n;
+
+	while (*p != NULL) {
+		parent = *p;
+		n = rb_entry(parent, struct cache_file, rb_node);
+		if (cache_file__cmp(n, file) >= 0)
+			p = &(*p)->rb_left;
+		else
+			p = &(*p)->rb_right;
+	}
+
+	rb_link_node(&file->rb_node, parent, p);
+	rb_insert_color(&file->rb_node, &cache_files);
+}
+
+typedef int (walk_cb_t)(struct cache_file *file, void *data);
+
+static int cache_files__walk(walk_cb_t cb, void *data)
+{
+	struct rb_node *nd;
+	int ret = 0;
+
+	for (nd = rb_first(&cache_files); !ret && nd; nd = rb_next(nd)) {
+		struct cache_file *n;
+
+		n = rb_entry(nd, struct cache_file, rb_node);
+		ret = cb(n, data);
+	}
+
+	return ret;
+}
+
+static int size_snprintf(u64 size, char *buf, int sz)
+{
+	struct {
+		int div;
+		const char *str;
+	} suffix[] = {
+		{ .str = "B", .div = 1 },
+		{ .str = "K", .div = 1024 },
+		{ .str = "M", .div = 1024*1024 },
+		{ .str = "G", .div = 1024*1024*1024 },
+	};
+	unsigned i;
+
+	for (i = 0; i < ARRAY_SIZE(suffix); i++) {
+		if (size / suffix[i].div < 1)
+			break;
+	}
+
+	i--;
+	return scnprintf(buf, sz, "%.1f%s",
+			 (double) (size / suffix[i].div), suffix[i].str);
+}
+
+static int date_snprintf(time_t t, char *buf, int sz)
+{
+	struct tm tm;
+
+	localtime_r(&t, &tm);
+	return strftime(buf, sz, "%b %d", &tm);
+}
+
+static int cache_file__fprintf(FILE *out, struct cache_file *file)
+{
+	char size_buf[100];
+	char date_buf[100];
+	int ret = 0;
+
+	if (cache_remove != CACHE_REMOVE__NONE)
+		ret += fprintf(out, "Removed ");
+
+	size_snprintf(file->size, size_buf, 100);
+	date_snprintf(file->time, date_buf, 100);
+	return ret + fprintf(out, "%10s  %6s  %s\n", size_buf, date_buf, file->path);
+}
+
+static int cache_file__process(struct cache_file *file, void *data)
+{
+	FILE *out = data;
+	int ret = 0;
+
+	if (cache_remove != CACHE_REMOVE__NONE)
+		ret = build_id_cache__remove_file(file->path, buildid_dir);
+
+	if (cache_disp == CACHE_DISP__ALL)
+		cache_file__fprintf(out, file);
+
+	return ret;
+}
+
+/*
+ * We want to go through each file only if we remove or
+ * display single files.
+ */
+static bool want_post_process(void)
+{
+	return (cache_remove == CACHE_REMOVE__SINGLE) ||
+	       (cache_disp == CACHE_DISP__ALL);
+}
+
+static int cache_files__process(FILE *out)
+{
+	int ret = 0;
+
+	/* Display total as first file/line. */
+	cache_file__fprintf(out, cache_total);
+
+	if (want_post_process())
+		ret = cache_files__walk(cache_file__process, out);
+
+	return ret;
+}
+
+static bool is_in_limit(const struct stat *st)
+{
+	bool in_limit = true;
+
+	switch (cache_limit) {
+	case CACHE_LIMIT__TIME:
+		in_limit = st->st_atime <= cache_limit__time;
+		break;
+	case CACHE_LIMIT__SIZE:
+		in_limit = (u64) st->st_size >= cache_limit__size;
+		break;
+	case CACHE_LIMIT__NONE:
+	default:
+		break;
+	};
+
+	return in_limit;
+}
+
+static int remove_file(const char *fpath, const struct stat *st)
+{
+	int ret;
+
+	if (S_ISDIR(st->st_mode))
+		ret = rmdir(fpath);
+	else
+		ret = unlink(fpath);
+
+	if (ret)
+		perror("failed to remove cache file");
+
+	return ret;
+}
+
+static int nftw_cb(const char *fpath, const struct stat *st,
+		   int typeflag __maybe_unused, struct FTW *ftwbuf)
+{
+	/* Do not touch the '.debug' directory itself. */
+	if (!ftwbuf->level)
+		return 0;
+
+	/*
+	 * Total cache wipe out handled right here. We try
+	 * to remove everything despite the possible removal
+	 * failures.
+	 */
+	if (cache_remove == CACHE_REMOVE__TOTAL) {
+		cache_total->size += st->st_size;
+
+		/* Ignore failure, remove as much as we can. */
+		remove_file(fpath, st);
+		return 0;
+	}
+
+	if (!is_in_limit(st))
+		return 0;
+
+	/* Sorting only regular files. */
+	if (want_post_process() && S_ISREG(st->st_mode)) {
+		struct cache_file *file;
+
+		file = cache_file__alloc(fpath, st);
+		if (!file)
+			return -1;
+
+		cache_files__add(file);
+	}
+
+	cache_total->size += st->st_size;
+	return 0;
+}
+
+static int cache_files__alloc(void)
+{
+	int flags = FTW_PHYS;
+	struct stat st;
+
+	if (stat(buildid_dir, &st)) {
+		pr_err("Failed to stat buildid directory %s.", buildid_dir);
+		return -1;
+	}
+
+	cache_total = cache_file__alloc(buildid_dir, &st);
+	if (!cache_total)
+		return -1;
+
+	/*
+	 * If we're going to remove all the files, switch the walk
+	 * files order to get inner directories/files first.  This
+	 * way we can remove them immediately.
+	 */
+	if (cache_remove == CACHE_REMOVE__TOTAL)
+		flags |= FTW_DEPTH;
+
+	return nftw(buildid_dir, nftw_cb, 0, flags);
+}
+
+static int cache_file__remove(struct cache_file *file,
+			      void *data __maybe_unused)
+{
+	rb_erase(&file->rb_node, &cache_files);
+	cache_file__release(file);
+	return 0;
+}
+
+static void cache_files__release(void)
+{
+	cache_files__walk(cache_file__remove, NULL);
+	cache_file__release(cache_total);
+}
+
+static int setup_limit(char *limit)
+{
+	struct suffix {
+		char s;
+		long m;
+	};
+	struct suffix suffix_time[] = {
+		{ .s = 'd', .m =   1*24*60*60 },
+		{ .s = 'w', .m =   7*24*60*60 },
+		{ .s = 'm', .m =  30*24*60*60 },
+		{ .s = 'y', .m = 365*24*60*60 },
+	};
+	struct suffix suffix_size[] = {
+		{ .s = 'B', .m = 1 },
+		{ .s = 'K', .m = 1*1024 },
+		{ .s = 'M', .m = 1*1024*1024 },
+		{ .s = 'G', .m = 1*1024*1024*1024 },
+	};
+	char *suffix;
+	long val;
+	unsigned i;
+
+	if (strlen(limit) < 2)
+		return -1;
+
+	val = strtol(limit, &suffix, 10);
+	if (!suffix)
+		return -1;
+
+	if (strlen(suffix) != 1)
+		return -1;
+
+	for (i = 0; i < ARRAY_SIZE(suffix_time); i++) {
+		char buf[100];
+
+		if (suffix_time[i].s != suffix[0])
+			continue;
+
+		val *= -1 * suffix_time[i].m;
+		val += time(0);
+		cache_limit__time = val;
+		cache_limit = CACHE_LIMIT__TIME;
+
+		date_snprintf(cache_limit__time, buf, sizeof(buf));
+		pr_debug("time limit: %s\n", buf);
+		return 0;
+	}
+
+	for (i = 0; i < ARRAY_SIZE(suffix_size); i++) {
+		char buf[100];
+
+		if (suffix_size[i].s != suffix[0])
+			continue;
+
+		val *= suffix_size[i].m;
+		cache_limit__size = val;
+		cache_limit = CACHE_LIMIT__SIZE;
+
+		size_snprintf(cache_limit__size, buf, sizeof(buf));
+		pr_debug("size limit: %s\n", buf);
+		return 0;
+	}
+
+	return -1;
+}
+
+static int cmd_buildid_cache_clean(int argc, const char **argv)
+{
+	const struct option buildid_cache_clean_options[] = {
+	OPT_SET_UINT(0, "size", &cache_sort, "sort by size", CACHE_SORT__SIZE),
+	OPT_SET_UINT(0, "time", &cache_sort, "sort by time", CACHE_SORT__TIME),
+	OPT_SET_UINT('a', "all", &cache_disp, "display all files",
+		     CACHE_DISP__ALL),
+	OPT_SET_UINT('r', "remove", &cache_remove, "display all files",
+		     CACHE_REMOVE__SINGLE),
+	OPT_INCR('v', "verbose", &verbose, "be more verbose"),
+	OPT_END(),
+	};
+	const char * const buildid_cache_clean_usage[] = {
+		"perf buildid-cache clean [<options>]",
+		NULL,
+	};
+	int ret;
+
+	argc = parse_options(argc, argv, buildid_cache_clean_options,
+			     buildid_cache_clean_usage, 0);
+
+	/* Check if user specified a limit. */
+	if (argc) {
+		char *limit = (char *) argv[0];
+
+		if (argc != 1 || setup_limit(limit)) {
+			pr_err("Failed: unsupported limit '%s'\n", limit);
+			return -1;
+		}
+	}
+
+	/* Full removal is handled separately. */
+	if ((cache_remove == CACHE_REMOVE__SINGLE) &&
+	    (cache_limit  == CACHE_LIMIT__NONE)    &&
+	    (cache_disp   == CACHE_DISP__NONE)  &&
+	    (cache_sort   == CACHE_SORT__NONE))
+		cache_remove = CACHE_REMOVE__TOTAL;
+
+	/*
+	 * Sort by size by default and display all entries in case
+	 * --size or --time option is specified.
+	 */
+	if (cache_sort == CACHE_SORT__NONE)
+		cache_sort = CACHE_SORT__SIZE;
+	else
+		cache_disp = CACHE_DISP__ALL;
+
+	if (cache_remove == CACHE_REMOVE__NONE)
+		pr_warning("(mock mode, run with '-r' to actually remove data)\n");
+
+	ret = cache_files__alloc();
+	if (!ret)
+		cache_files__process(stderr);
+
+	cache_files__release();
+	return ret;
+}
+
+static int process_subcmd(int argc, const char **argv)
+{
+	const char *cmd = argv[0];
+
+	if (!strcmp(cmd, "clean"))
+		return cmd_buildid_cache_clean(argc, argv);
+
+	pr_err("Failed: unknown sub command '%s'\n", cmd);
+	return -EINVAL;
+}
+
 int cmd_buildid_cache(int argc, const char **argv,
 		      const char *prefix __maybe_unused)
 {
@@ -318,7 +766,8 @@ int cmd_buildid_cache(int argc, const char **argv,
 	};
 
 	argc = parse_options(argc, argv, buildid_cache_options,
-			     buildid_cache_usage, 0);
+			     buildid_cache_usage,
+			     PARSE_OPT_STOP_AT_NON_OPTION);
 
 	if (missing_filename) {
 		file.path = missing_filename;
@@ -399,5 +848,8 @@ out:
 	if (session)
 		perf_session__delete(session);
 
+	if (!ret && argc)
+		ret = process_subcmd(argc, argv);
+
 	return ret;
 }
-- 
1.9.3

[PATCH 5/8] perf buildid-cache: Add automated tests

[Jiri Olsa]

Adding automated test for buildid-cache command/processing.

Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Corey Ashford <cjashfor@linux.vnet.ibm.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Stephane Eranian <eranian@google.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 tools/perf/Makefile.perf          |  7 +++-
 tools/perf/tests/buildid-cache.c  | 88 +++++++++++++++++++++++++++++++++++++++
 tools/perf/tests/buildid-cache.sh | 60 ++++++++++++++++++++++++++
 tools/perf/tests/builtin-test.c   |  8 ++++
 tools/perf/tests/tests.h          |  1 +
 5 files changed, 163 insertions(+), 1 deletion(-)
 create mode 100644 tools/perf/tests/buildid-cache.c
 create mode 100755 tools/perf/tests/buildid-cache.sh

diff --git a/tools/perf/Makefile.perf b/tools/perf/Makefile.perf
index 478efa9b2364..28a8f64ab49c 100644
--- a/tools/perf/Makefile.perf
+++ b/tools/perf/Makefile.perf
@@ -447,6 +447,7 @@ endif
 LIB_OBJS += $(OUTPUT)tests/mmap-thread-lookup.o
 LIB_OBJS += $(OUTPUT)tests/thread-mg-share.o
 LIB_OBJS += $(OUTPUT)tests/switch-tracking.o
+LIB_OBJS += $(OUTPUT)tests/buildid-cache.o
 
 BUILTIN_OBJS += $(OUTPUT)builtin-annotate.o
 BUILTIN_OBJS += $(OUTPUT)builtin-bench.o
@@ -699,6 +700,9 @@ $(OUTPUT)tests/attr.o: tests/attr.c $(OUTPUT)PERF-CFLAGS
 		'-DBINDIR="$(bindir_SQ)"' -DPYTHON='"$(PYTHON_WORD)"' \
 		$<
 
+$(OUTPUT)tests/buildid-cache.o: tests/buildid-cache.c $(OUTPUT)PERF-CFLAGS
+	$(QUIET_CC)$(CC) -o $@ -c $(CFLAGS) '-DBINDIR="$(bindir_SQ)"' $<
+
 $(OUTPUT)tests/python-use.o: tests/python-use.c $(OUTPUT)PERF-CFLAGS
 	$(QUIET_CC)$(CC) -o $@ -c $(CFLAGS) \
 		-DPYTHONPATH='"$(OUTPUT)python"' \
@@ -944,7 +948,8 @@ endif
 		$(INSTALL) -d -m 755 '$(DESTDIR_SQ)$(perfexec_instdir_SQ)/tests'; \
 		$(INSTALL) tests/attr.py '$(DESTDIR_SQ)$(perfexec_instdir_SQ)/tests'; \
 		$(INSTALL) -d -m 755 '$(DESTDIR_SQ)$(perfexec_instdir_SQ)/tests/attr'; \
-		$(INSTALL) tests/attr/* '$(DESTDIR_SQ)$(perfexec_instdir_SQ)/tests/attr'
+		$(INSTALL) tests/attr/* '$(DESTDIR_SQ)$(perfexec_instdir_SQ)/tests/attr'; \
+		$(INSTALL) tests/buildid-cache.sh '$(DESTDIR_SQ)$(perfexec_instdir_SQ)/tests'
 
 install: install-bin try-install-man install-traceevent-plugins
 
diff --git a/tools/perf/tests/buildid-cache.c b/tools/perf/tests/buildid-cache.c
new file mode 100644
index 000000000000..52dfd2765527
--- /dev/null
+++ b/tools/perf/tests/buildid-cache.c
@@ -0,0 +1,88 @@
+#include <api/fs/fs.h>
+#include "tests.h"
+#include "symbol.h"
+#include "build-id.h"
+#include "debug.h"
+#include "exec_cmd.h"
+
+static int add_kernel(void)
+{
+	char path[PATH_MAX];
+	u8 build_id[BUILD_ID_SIZE];
+	char sbuild_id[BUILD_ID_SIZE * 2 + 1];
+	int ret;
+
+	sprintf(path, "%s/kernel/notes", sysfs__mountpoint());
+
+	ret = sysfs__read_build_id(path, build_id, sizeof(build_id));
+	TEST_ASSERT_VAL("failed to get kernel buildid", !ret);
+
+	build_id__sprintf(build_id, sizeof(build_id), sbuild_id);
+
+	return build_id_cache__add_s(sbuild_id, buildid_dir,
+				     "[kernel.kallsyms]", true, false);
+}
+
+static int __run_script(const char *script, const char *perf, char *cache)
+{
+	char cmd[PATH_MAX * 3 + 5];
+
+	scnprintf(cmd, sizeof(cmd), "%s/buildid-cache.sh %s %s %d",
+		  script, perf, cache, verbose);
+	return system(cmd);
+}
+
+static int run_script(char *cache)
+{
+	struct stat st;
+	char path_perf[PATH_MAX];
+	char path_script[PATH_MAX];
+
+	/* First try development tree tests. */
+	if (!lstat("./tests", &st))
+		return __run_script("./tests", "./perf", cache);
+
+	/* Then installed path. */
+	snprintf(path_script, PATH_MAX, "%s/tests", perf_exec_path());
+	snprintf(path_perf, PATH_MAX, "%s/perf", BINDIR);
+
+	if (!lstat(path_script, &st) && !lstat(path_perf, &st))
+		return __run_script(path_script, path_perf, cache);
+
+	fprintf(stderr, " (omitted)");
+	return 0;
+}
+
+static int __test__buildid_cache(char *cache)
+{
+	set_buildid_dir(cache);
+
+	/*
+	 * Adding [kernel.kallsyms] entry, because we will test
+	 * its removal via perf buildid-cache clean in the
+	 * script part.
+	 * NOTE it's not possible to add [kernel.kallsyms] entry
+	 * by script at the moment.
+	 */
+	TEST_ASSERT_VAL("failed to add [kernel.kallsyms] buildid",
+			!add_kernel());
+
+	TEST_ASSERT_VAL("script failed", !run_script(cache));
+	return 0;
+}
+
+int test__buildid_cache(void)
+{
+	char cache[50];
+
+	/*
+	 * The directory removal is done within
+	 * __test__buildid_cache function.
+	 */
+	snprintf(cache, sizeof(cache), "/tmp/perf-XXXXXX");
+	TEST_ASSERT_VAL("failed to make temp directory", mkdtemp(cache));
+
+	pr_debug("buildid cache directory: %s\n", cache);
+
+	return __test__buildid_cache(cache);
+}
diff --git a/tools/perf/tests/buildid-cache.sh b/tools/perf/tests/buildid-cache.sh
new file mode 100755
index 000000000000..fbd92c278c2a
--- /dev/null
+++ b/tools/perf/tests/buildid-cache.sh
@@ -0,0 +1,60 @@
+#!/bin/sh
+
+perf=$1
+cache=$2
+verbose=$3
+
+function pr_debug
+{
+	if [ "$verbose" -gt "0" ]; then
+		echo "$@"
+	fi
+}
+
+function run_perf
+{
+	$perf --no-pager --buildid-dir $cache $@
+}
+
+# Remove prepared '[kernel.kallsyms]' via 1 byte clean limit
+run_perf buildid-cache clean -a -r 1B 2>/dev/null
+dir="$cache/[kernel.kallsyms]"
+if [ "$(ls -A $cache)" ]; then
+	pr_debug "Failed to remove [kernel.kallsyms] cache files"
+	exit 1
+fi
+
+# add perf binary
+run_perf buildid-cache -a $perf 2>/dev/null
+dir_perf=$cache/`realpath $perf`
+if [ ! -d $dir_perf ]; then
+	pr_debug "Failed to add perf binary into cache"
+	exit 1
+fi
+
+# remove perf binary
+run_perf buildid-cache -r $perf 2>/dev/null
+if [ "$(ls -A $cache)" ]; then
+	pr_debug "Failed to remove perf binary from cache"
+	exit 1
+fi
+
+# add perf binary
+run_perf buildid-cache -a $perf 2>/dev/null
+# add sh binary
+run_perf buildid-cache -a `realpath /bin/sh` 2>/dev/null
+dir_sh=$cache/`realpath /bin/sh`
+if [ ! -d $dir_perf -o ! -d $dir_sh ]; then
+	pr_debug "Failed to add perf/sh binary into cache"
+	exit 1
+fi
+
+# clean all
+run_perf buildid-cache clean -r 2>/dev/null
+if [ "$(ls -A $cache)" ]; then
+	pr_debug "Failed to cleanup the cache"
+	exit 1
+fi
+
+# last command, $cache directory should be empty
+rmdir $cache
diff --git a/tools/perf/tests/builtin-test.c b/tools/perf/tests/builtin-test.c
index 4b7d9ab0f049..5cbe55634181 100644
--- a/tools/perf/tests/builtin-test.c
+++ b/tools/perf/tests/builtin-test.c
@@ -167,6 +167,14 @@ static struct test {
 		.func = test__fdarray__add,
 	},
 	{
+		.desc = "Add fd to a fdarray, making it autogrow",
+		.func = test__fdarray__add,
+	},
+	{
+		.desc = "Test buildid cache",
+		.func = test__buildid_cache,
+	},
+	{
 		.func = NULL,
 	},
 };
diff --git a/tools/perf/tests/tests.h b/tools/perf/tests/tests.h
index 00e776a87a9c..190e3df7431f 100644
--- a/tools/perf/tests/tests.h
+++ b/tools/perf/tests/tests.h
@@ -51,6 +51,7 @@ int test__hists_cumulate(void);
 int test__switch_tracking(void);
 int test__fdarray__filter(void);
 int test__fdarray__add(void);
+int test__buildid_cache(void);
 
 #if defined(__x86_64__) || defined(__i386__) || defined(__arm__)
 #ifdef HAVE_DWARF_UNWIND_SUPPORT
-- 
1.9.3

Re: [PATCH 5/8] perf buildid-cache: Add automated tests

[Arnaldo Carvalho de Melo]

Em Mon, Dec 01, 2014 at 08:06:26PM +0100, Jiri Olsa escreveu:
> Adding automated test for buildid-cache command/processing.

Does it only work if one is in the tools/perf/ directory? Why not remove
that './' from './perf' and ditch that fallback mechanism (BINDIR).

If all you want to make sure is that the binary used is the one just
built and not the system one, then you can make sure that in your PATH
the directory with the development one is comes first, right?

I.e. no need for that fallbacking mechanism, etc.

- Arnaldo
 
> Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
> Cc: Corey Ashford <cjashfor@linux.vnet.ibm.com>
> Cc: David Ahern <dsahern@gmail.com>
> Cc: Frederic Weisbecker <fweisbec@gmail.com>
> Cc: Ingo Molnar <mingo@kernel.org>
> Cc: Namhyung Kim <namhyung@kernel.org>
> Cc: Paul Mackerras <paulus@samba.org>
> Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
> Cc: Stephane Eranian <eranian@google.com>
> Cc: Steven Rostedt <rostedt@goodmis.org>
> Signed-off-by: Jiri Olsa <jolsa@kernel.org>
> ---
>  tools/perf/Makefile.perf          |  7 +++-
>  tools/perf/tests/buildid-cache.c  | 88 +++++++++++++++++++++++++++++++++++++++
>  tools/perf/tests/buildid-cache.sh | 60 ++++++++++++++++++++++++++
>  tools/perf/tests/builtin-test.c   |  8 ++++
>  tools/perf/tests/tests.h          |  1 +
>  5 files changed, 163 insertions(+), 1 deletion(-)
>  create mode 100644 tools/perf/tests/buildid-cache.c
>  create mode 100755 tools/perf/tests/buildid-cache.sh
> 
> diff --git a/tools/perf/Makefile.perf b/tools/perf/Makefile.perf
> index 478efa9b2364..28a8f64ab49c 100644
> --- a/tools/perf/Makefile.perf
> +++ b/tools/perf/Makefile.perf
> @@ -447,6 +447,7 @@ endif
>  LIB_OBJS += $(OUTPUT)tests/mmap-thread-lookup.o
>  LIB_OBJS += $(OUTPUT)tests/thread-mg-share.o
>  LIB_OBJS += $(OUTPUT)tests/switch-tracking.o
> +LIB_OBJS += $(OUTPUT)tests/buildid-cache.o
>  
>  BUILTIN_OBJS += $(OUTPUT)builtin-annotate.o
>  BUILTIN_OBJS += $(OUTPUT)builtin-bench.o
> @@ -699,6 +700,9 @@ $(OUTPUT)tests/attr.o: tests/attr.c $(OUTPUT)PERF-CFLAGS
>  		'-DBINDIR="$(bindir_SQ)"' -DPYTHON='"$(PYTHON_WORD)"' \
>  		$<
>  
> +$(OUTPUT)tests/buildid-cache.o: tests/buildid-cache.c $(OUTPUT)PERF-CFLAGS
> +	$(QUIET_CC)$(CC) -o $@ -c $(CFLAGS) '-DBINDIR="$(bindir_SQ)"' $<
> +
>  $(OUTPUT)tests/python-use.o: tests/python-use.c $(OUTPUT)PERF-CFLAGS
>  	$(QUIET_CC)$(CC) -o $@ -c $(CFLAGS) \
>  		-DPYTHONPATH='"$(OUTPUT)python"' \
> @@ -944,7 +948,8 @@ endif
>  		$(INSTALL) -d -m 755 '$(DESTDIR_SQ)$(perfexec_instdir_SQ)/tests'; \
>  		$(INSTALL) tests/attr.py '$(DESTDIR_SQ)$(perfexec_instdir_SQ)/tests'; \
>  		$(INSTALL) -d -m 755 '$(DESTDIR_SQ)$(perfexec_instdir_SQ)/tests/attr'; \
> -		$(INSTALL) tests/attr/* '$(DESTDIR_SQ)$(perfexec_instdir_SQ)/tests/attr'
> +		$(INSTALL) tests/attr/* '$(DESTDIR_SQ)$(perfexec_instdir_SQ)/tests/attr'; \
> +		$(INSTALL) tests/buildid-cache.sh '$(DESTDIR_SQ)$(perfexec_instdir_SQ)/tests'
>  
>  install: install-bin try-install-man install-traceevent-plugins
>  
> diff --git a/tools/perf/tests/buildid-cache.c b/tools/perf/tests/buildid-cache.c
> new file mode 100644
> index 000000000000..52dfd2765527
> --- /dev/null
> +++ b/tools/perf/tests/buildid-cache.c
> @@ -0,0 +1,88 @@
> +#include <api/fs/fs.h>
> +#include "tests.h"
> +#include "symbol.h"
> +#include "build-id.h"
> +#include "debug.h"
> +#include "exec_cmd.h"
> +
> +static int add_kernel(void)
> +{
> +	char path[PATH_MAX];
> +	u8 build_id[BUILD_ID_SIZE];
> +	char sbuild_id[BUILD_ID_SIZE * 2 + 1];
> +	int ret;
> +
> +	sprintf(path, "%s/kernel/notes", sysfs__mountpoint());
> +
> +	ret = sysfs__read_build_id(path, build_id, sizeof(build_id));
> +	TEST_ASSERT_VAL("failed to get kernel buildid", !ret);
> +
> +	build_id__sprintf(build_id, sizeof(build_id), sbuild_id);
> +
> +	return build_id_cache__add_s(sbuild_id, buildid_dir,
> +				     "[kernel.kallsyms]", true, false);
> +}
> +
> +static int __run_script(const char *script, const char *perf, char *cache)
> +{
> +	char cmd[PATH_MAX * 3 + 5];
> +
> +	scnprintf(cmd, sizeof(cmd), "%s/buildid-cache.sh %s %s %d",
> +		  script, perf, cache, verbose);
> +	return system(cmd);
> +}
> +
> +static int run_script(char *cache)
> +{
> +	struct stat st;
> +	char path_perf[PATH_MAX];
> +	char path_script[PATH_MAX];
> +
> +	/* First try development tree tests. */
> +	if (!lstat("./tests", &st))
> +		return __run_script("./tests", "./perf", cache);
> +
> +	/* Then installed path. */
> +	snprintf(path_script, PATH_MAX, "%s/tests", perf_exec_path());
> +	snprintf(path_perf, PATH_MAX, "%s/perf", BINDIR);
> +
> +	if (!lstat(path_script, &st) && !lstat(path_perf, &st))
> +		return __run_script(path_script, path_perf, cache);
> +
> +	fprintf(stderr, " (omitted)");
> +	return 0;
> +}
> +
> +static int __test__buildid_cache(char *cache)
> +{
> +	set_buildid_dir(cache);
> +
> +	/*
> +	 * Adding [kernel.kallsyms] entry, because we will test
> +	 * its removal via perf buildid-cache clean in the
> +	 * script part.
> +	 * NOTE it's not possible to add [kernel.kallsyms] entry
> +	 * by script at the moment.
> +	 */
> +	TEST_ASSERT_VAL("failed to add [kernel.kallsyms] buildid",
> +			!add_kernel());
> +
> +	TEST_ASSERT_VAL("script failed", !run_script(cache));
> +	return 0;
> +}
> +
> +int test__buildid_cache(void)
> +{
> +	char cache[50];
> +
> +	/*
> +	 * The directory removal is done within
> +	 * __test__buildid_cache function.
> +	 */
> +	snprintf(cache, sizeof(cache), "/tmp/perf-XXXXXX");
> +	TEST_ASSERT_VAL("failed to make temp directory", mkdtemp(cache));
> +
> +	pr_debug("buildid cache directory: %s\n", cache);
> +
> +	return __test__buildid_cache(cache);
> +}
> diff --git a/tools/perf/tests/buildid-cache.sh b/tools/perf/tests/buildid-cache.sh
> new file mode 100755
> index 000000000000..fbd92c278c2a
> --- /dev/null
> +++ b/tools/perf/tests/buildid-cache.sh
> @@ -0,0 +1,60 @@
> +#!/bin/sh
> +
> +perf=$1
> +cache=$2
> +verbose=$3
> +
> +function pr_debug
> +{
> +	if [ "$verbose" -gt "0" ]; then
> +		echo "$@"
> +	fi
> +}
> +
> +function run_perf
> +{
> +	$perf --no-pager --buildid-dir $cache $@
> +}
> +
> +# Remove prepared '[kernel.kallsyms]' via 1 byte clean limit
> +run_perf buildid-cache clean -a -r 1B 2>/dev/null
> +dir="$cache/[kernel.kallsyms]"
> +if [ "$(ls -A $cache)" ]; then
> +	pr_debug "Failed to remove [kernel.kallsyms] cache files"
> +	exit 1
> +fi
> +
> +# add perf binary
> +run_perf buildid-cache -a $perf 2>/dev/null
> +dir_perf=$cache/`realpath $perf`
> +if [ ! -d $dir_perf ]; then
> +	pr_debug "Failed to add perf binary into cache"
> +	exit 1
> +fi
> +
> +# remove perf binary
> +run_perf buildid-cache -r $perf 2>/dev/null
> +if [ "$(ls -A $cache)" ]; then
> +	pr_debug "Failed to remove perf binary from cache"
> +	exit 1
> +fi
> +
> +# add perf binary
> +run_perf buildid-cache -a $perf 2>/dev/null
> +# add sh binary
> +run_perf buildid-cache -a `realpath /bin/sh` 2>/dev/null
> +dir_sh=$cache/`realpath /bin/sh`
> +if [ ! -d $dir_perf -o ! -d $dir_sh ]; then
> +	pr_debug "Failed to add perf/sh binary into cache"
> +	exit 1
> +fi
> +
> +# clean all
> +run_perf buildid-cache clean -r 2>/dev/null
> +if [ "$(ls -A $cache)" ]; then
> +	pr_debug "Failed to cleanup the cache"
> +	exit 1
> +fi
> +
> +# last command, $cache directory should be empty
> +rmdir $cache
> diff --git a/tools/perf/tests/builtin-test.c b/tools/perf/tests/builtin-test.c
> index 4b7d9ab0f049..5cbe55634181 100644
> --- a/tools/perf/tests/builtin-test.c
> +++ b/tools/perf/tests/builtin-test.c
> @@ -167,6 +167,14 @@ static struct test {
>  		.func = test__fdarray__add,
>  	},
>  	{
> +		.desc = "Add fd to a fdarray, making it autogrow",
> +		.func = test__fdarray__add,
> +	},
> +	{
> +		.desc = "Test buildid cache",
> +		.func = test__buildid_cache,
> +	},
> +	{
>  		.func = NULL,
>  	},
>  };
> diff --git a/tools/perf/tests/tests.h b/tools/perf/tests/tests.h
> index 00e776a87a9c..190e3df7431f 100644
> --- a/tools/perf/tests/tests.h
> +++ b/tools/perf/tests/tests.h
> @@ -51,6 +51,7 @@ int test__hists_cumulate(void);
>  int test__switch_tracking(void);
>  int test__fdarray__filter(void);
>  int test__fdarray__add(void);
> +int test__buildid_cache(void);
>  
>  #if defined(__x86_64__) || defined(__i386__) || defined(__arm__)
>  #ifdef HAVE_DWARF_UNWIND_SUPPORT
> -- 
> 1.9.3

Re: [PATCH 5/8] perf buildid-cache: Add automated tests

[Jiri Olsa]

On Fri, Dec 05, 2014 at 01:30:14PM -0200, Arnaldo Carvalho de Melo wrote:
> Em Mon, Dec 01, 2014 at 08:06:26PM +0100, Jiri Olsa escreveu:
> > Adding automated test for buildid-cache command/processing.
> 
> Does it only work if one is in the tools/perf/ directory? Why not remove
> that './' from './perf' and ditch that fallback mechanism (BINDIR).
> 
> If all you want to make sure is that the binary used is the one just
> built and not the system one, then you can make sure that in your PATH
> the directory with the development one is comes first, right?
> 
> I.e. no need for that fallbacking mechanism, etc.

well it's done the same way 'perf test attr' is done,
because both needs external script.. I did it only to
fit my needs.. from my POV I'm either in the:

  1) - $SRC/tools/perf - developing perf
or
  2) - anywhere else - not developing perf ;-)

in case 1) I need perf to run the in tree version
of the script and
in case 2) it runs the installed version of the script

jirka

> 
> - Arnaldo
>  
> > Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
> > Cc: Corey Ashford <cjashfor@linux.vnet.ibm.com>
> > Cc: David Ahern <dsahern@gmail.com>
> > Cc: Frederic Weisbecker <fweisbec@gmail.com>
> > Cc: Ingo Molnar <mingo@kernel.org>
> > Cc: Namhyung Kim <namhyung@kernel.org>
> > Cc: Paul Mackerras <paulus@samba.org>
> > Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
> > Cc: Stephane Eranian <eranian@google.com>
> > Cc: Steven Rostedt <rostedt@goodmis.org>
> > Signed-off-by: Jiri Olsa <jolsa@kernel.org>
> > ---
> >  tools/perf/Makefile.perf          |  7 +++-
> >  tools/perf/tests/buildid-cache.c  | 88 +++++++++++++++++++++++++++++++++++++++
> >  tools/perf/tests/buildid-cache.sh | 60 ++++++++++++++++++++++++++
> >  tools/perf/tests/builtin-test.c   |  8 ++++
> >  tools/perf/tests/tests.h          |  1 +
> >  5 files changed, 163 insertions(+), 1 deletion(-)
> >  create mode 100644 tools/perf/tests/buildid-cache.c
> >  create mode 100755 tools/perf/tests/buildid-cache.sh
> > 
> > diff --git a/tools/perf/Makefile.perf b/tools/perf/Makefile.perf
> > index 478efa9b2364..28a8f64ab49c 100644
> > --- a/tools/perf/Makefile.perf
> > +++ b/tools/perf/Makefile.perf
> > @@ -447,6 +447,7 @@ endif
> >  LIB_OBJS += $(OUTPUT)tests/mmap-thread-lookup.o
> >  LIB_OBJS += $(OUTPUT)tests/thread-mg-share.o
> >  LIB_OBJS += $(OUTPUT)tests/switch-tracking.o
> > +LIB_OBJS += $(OUTPUT)tests/buildid-cache.o
> >  
> >  BUILTIN_OBJS += $(OUTPUT)builtin-annotate.o
> >  BUILTIN_OBJS += $(OUTPUT)builtin-bench.o
> > @@ -699,6 +700,9 @@ $(OUTPUT)tests/attr.o: tests/attr.c $(OUTPUT)PERF-CFLAGS
> >  		'-DBINDIR="$(bindir_SQ)"' -DPYTHON='"$(PYTHON_WORD)"' \
> >  		$<
> >  
> > +$(OUTPUT)tests/buildid-cache.o: tests/buildid-cache.c $(OUTPUT)PERF-CFLAGS
> > +	$(QUIET_CC)$(CC) -o $@ -c $(CFLAGS) '-DBINDIR="$(bindir_SQ)"' $<
> > +
> >  $(OUTPUT)tests/python-use.o: tests/python-use.c $(OUTPUT)PERF-CFLAGS
> >  	$(QUIET_CC)$(CC) -o $@ -c $(CFLAGS) \
> >  		-DPYTHONPATH='"$(OUTPUT)python"' \
> > @@ -944,7 +948,8 @@ endif
> >  		$(INSTALL) -d -m 755 '$(DESTDIR_SQ)$(perfexec_instdir_SQ)/tests'; \
> >  		$(INSTALL) tests/attr.py '$(DESTDIR_SQ)$(perfexec_instdir_SQ)/tests'; \
> >  		$(INSTALL) -d -m 755 '$(DESTDIR_SQ)$(perfexec_instdir_SQ)/tests/attr'; \
> > -		$(INSTALL) tests/attr/* '$(DESTDIR_SQ)$(perfexec_instdir_SQ)/tests/attr'
> > +		$(INSTALL) tests/attr/* '$(DESTDIR_SQ)$(perfexec_instdir_SQ)/tests/attr'; \
> > +		$(INSTALL) tests/buildid-cache.sh '$(DESTDIR_SQ)$(perfexec_instdir_SQ)/tests'
> >  
> >  install: install-bin try-install-man install-traceevent-plugins
> >  
> > diff --git a/tools/perf/tests/buildid-cache.c b/tools/perf/tests/buildid-cache.c
> > new file mode 100644
> > index 000000000000..52dfd2765527
> > --- /dev/null
> > +++ b/tools/perf/tests/buildid-cache.c
> > @@ -0,0 +1,88 @@
> > +#include <api/fs/fs.h>
> > +#include "tests.h"
> > +#include "symbol.h"
> > +#include "build-id.h"
> > +#include "debug.h"
> > +#include "exec_cmd.h"
> > +
> > +static int add_kernel(void)
> > +{
> > +	char path[PATH_MAX];
> > +	u8 build_id[BUILD_ID_SIZE];
> > +	char sbuild_id[BUILD_ID_SIZE * 2 + 1];
> > +	int ret;
> > +
> > +	sprintf(path, "%s/kernel/notes", sysfs__mountpoint());
> > +
> > +	ret = sysfs__read_build_id(path, build_id, sizeof(build_id));
> > +	TEST_ASSERT_VAL("failed to get kernel buildid", !ret);
> > +
> > +	build_id__sprintf(build_id, sizeof(build_id), sbuild_id);
> > +
> > +	return build_id_cache__add_s(sbuild_id, buildid_dir,
> > +				     "[kernel.kallsyms]", true, false);
> > +}
> > +
> > +static int __run_script(const char *script, const char *perf, char *cache)
> > +{
> > +	char cmd[PATH_MAX * 3 + 5];
> > +
> > +	scnprintf(cmd, sizeof(cmd), "%s/buildid-cache.sh %s %s %d",
> > +		  script, perf, cache, verbose);
> > +	return system(cmd);
> > +}
> > +
> > +static int run_script(char *cache)
> > +{
> > +	struct stat st;
> > +	char path_perf[PATH_MAX];
> > +	char path_script[PATH_MAX];
> > +
> > +	/* First try development tree tests. */
> > +	if (!lstat("./tests", &st))
> > +		return __run_script("./tests", "./perf", cache);
> > +
> > +	/* Then installed path. */
> > +	snprintf(path_script, PATH_MAX, "%s/tests", perf_exec_path());
> > +	snprintf(path_perf, PATH_MAX, "%s/perf", BINDIR);
> > +
> > +	if (!lstat(path_script, &st) && !lstat(path_perf, &st))
> > +		return __run_script(path_script, path_perf, cache);
> > +
> > +	fprintf(stderr, " (omitted)");
> > +	return 0;
> > +}
> > +
> > +static int __test__buildid_cache(char *cache)
> > +{
> > +	set_buildid_dir(cache);
> > +
> > +	/*
> > +	 * Adding [kernel.kallsyms] entry, because we will test
> > +	 * its removal via perf buildid-cache clean in the
> > +	 * script part.
> > +	 * NOTE it's not possible to add [kernel.kallsyms] entry
> > +	 * by script at the moment.
> > +	 */
> > +	TEST_ASSERT_VAL("failed to add [kernel.kallsyms] buildid",
> > +			!add_kernel());
> > +
> > +	TEST_ASSERT_VAL("script failed", !run_script(cache));
> > +	return 0;
> > +}
> > +
> > +int test__buildid_cache(void)
> > +{
> > +	char cache[50];
> > +
> > +	/*
> > +	 * The directory removal is done within
> > +	 * __test__buildid_cache function.
> > +	 */
> > +	snprintf(cache, sizeof(cache), "/tmp/perf-XXXXXX");
> > +	TEST_ASSERT_VAL("failed to make temp directory", mkdtemp(cache));
> > +
> > +	pr_debug("buildid cache directory: %s\n", cache);
> > +
> > +	return __test__buildid_cache(cache);
> > +}
> > diff --git a/tools/perf/tests/buildid-cache.sh b/tools/perf/tests/buildid-cache.sh
> > new file mode 100755
> > index 000000000000..fbd92c278c2a
> > --- /dev/null
> > +++ b/tools/perf/tests/buildid-cache.sh
> > @@ -0,0 +1,60 @@
> > +#!/bin/sh
> > +
> > +perf=$1
> > +cache=$2
> > +verbose=$3
> > +
> > +function pr_debug
> > +{
> > +	if [ "$verbose" -gt "0" ]; then
> > +		echo "$@"
> > +	fi
> > +}
> > +
> > +function run_perf
> > +{
> > +	$perf --no-pager --buildid-dir $cache $@
> > +}
> > +
> > +# Remove prepared '[kernel.kallsyms]' via 1 byte clean limit
> > +run_perf buildid-cache clean -a -r 1B 2>/dev/null
> > +dir="$cache/[kernel.kallsyms]"
> > +if [ "$(ls -A $cache)" ]; then
> > +	pr_debug "Failed to remove [kernel.kallsyms] cache files"
> > +	exit 1
> > +fi
> > +
> > +# add perf binary
> > +run_perf buildid-cache -a $perf 2>/dev/null
> > +dir_perf=$cache/`realpath $perf`
> > +if [ ! -d $dir_perf ]; then
> > +	pr_debug "Failed to add perf binary into cache"
> > +	exit 1
> > +fi
> > +
> > +# remove perf binary
> > +run_perf buildid-cache -r $perf 2>/dev/null
> > +if [ "$(ls -A $cache)" ]; then
> > +	pr_debug "Failed to remove perf binary from cache"
> > +	exit 1
> > +fi
> > +
> > +# add perf binary
> > +run_perf buildid-cache -a $perf 2>/dev/null
> > +# add sh binary
> > +run_perf buildid-cache -a `realpath /bin/sh` 2>/dev/null
> > +dir_sh=$cache/`realpath /bin/sh`
> > +if [ ! -d $dir_perf -o ! -d $dir_sh ]; then
> > +	pr_debug "Failed to add perf/sh binary into cache"
> > +	exit 1
> > +fi
> > +
> > +# clean all
> > +run_perf buildid-cache clean -r 2>/dev/null
> > +if [ "$(ls -A $cache)" ]; then
> > +	pr_debug "Failed to cleanup the cache"
> > +	exit 1
> > +fi
> > +
> > +# last command, $cache directory should be empty
> > +rmdir $cache
> > diff --git a/tools/perf/tests/builtin-test.c b/tools/perf/tests/builtin-test.c
> > index 4b7d9ab0f049..5cbe55634181 100644
> > --- a/tools/perf/tests/builtin-test.c
> > +++ b/tools/perf/tests/builtin-test.c
> > @@ -167,6 +167,14 @@ static struct test {
> >  		.func = test__fdarray__add,
> >  	},
> >  	{
> > +		.desc = "Add fd to a fdarray, making it autogrow",
> > +		.func = test__fdarray__add,
> > +	},
> > +	{
> > +		.desc = "Test buildid cache",
> > +		.func = test__buildid_cache,
> > +	},
> > +	{
> >  		.func = NULL,
> >  	},
> >  };
> > diff --git a/tools/perf/tests/tests.h b/tools/perf/tests/tests.h
> > index 00e776a87a9c..190e3df7431f 100644
> > --- a/tools/perf/tests/tests.h
> > +++ b/tools/perf/tests/tests.h
> > @@ -51,6 +51,7 @@ int test__hists_cumulate(void);
> >  int test__switch_tracking(void);
> >  int test__fdarray__filter(void);
> >  int test__fdarray__add(void);
> > +int test__buildid_cache(void);
> >  
> >  #if defined(__x86_64__) || defined(__i386__) || defined(__arm__)
> >  #ifdef HAVE_DWARF_UNWIND_SUPPORT
> > -- 
> > 1.9.3

[PATCH 6/8] perf buildid cache: Fix -a segfault

[Jiri Olsa]

The kcore_filename is uninitialized and trash value could
trigger build_id_cache__add_kcore function ending up with
segfault.

Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Corey Ashford <cjashfor@linux.vnet.ibm.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Stephane Eranian <eranian@google.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 tools/perf/builtin-buildid-cache.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tools/perf/builtin-buildid-cache.c b/tools/perf/builtin-buildid-cache.c
index 184955ec8a83..7a3504a1b494 100644
--- a/tools/perf/builtin-buildid-cache.c
+++ b/tools/perf/builtin-buildid-cache.c
@@ -737,7 +737,7 @@ int cmd_buildid_cache(int argc, const char **argv,
 		   *remove_name_list_str = NULL,
 		   *missing_filename = NULL,
 		   *update_name_list_str = NULL,
-		   *kcore_filename;
+		   *kcore_filename = NULL;
 	char sbuf[STRERR_BUFSIZE];
 
 	struct perf_data_file file = {
-- 
1.9.3

Re: [PATCH 6/8] perf buildid cache: Fix -a segfault

[Arnaldo Carvalho de Melo]

Em Mon, Dec 01, 2014 at 08:06:27PM +0100, Jiri Olsa escreveu:
> The kcore_filename is uninitialized and trash value could
> trigger build_id_cache__add_kcore function ending up with
> segfault.

Applied
 
> Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
> Cc: Corey Ashford <cjashfor@linux.vnet.ibm.com>
> Cc: David Ahern <dsahern@gmail.com>
> Cc: Frederic Weisbecker <fweisbec@gmail.com>
> Cc: Ingo Molnar <mingo@kernel.org>
> Cc: Namhyung Kim <namhyung@kernel.org>
> Cc: Paul Mackerras <paulus@samba.org>
> Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
> Cc: Stephane Eranian <eranian@google.com>
> Cc: Steven Rostedt <rostedt@goodmis.org>
> Signed-off-by: Jiri Olsa <jolsa@kernel.org>
> ---
>  tools/perf/builtin-buildid-cache.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/tools/perf/builtin-buildid-cache.c b/tools/perf/builtin-buildid-cache.c
> index 184955ec8a83..7a3504a1b494 100644
> --- a/tools/perf/builtin-buildid-cache.c
> +++ b/tools/perf/builtin-buildid-cache.c
> @@ -737,7 +737,7 @@ int cmd_buildid_cache(int argc, const char **argv,
>  		   *remove_name_list_str = NULL,
>  		   *missing_filename = NULL,
>  		   *update_name_list_str = NULL,
> -		   *kcore_filename;
> +		   *kcore_filename = NULL;
>  	char sbuf[STRERR_BUFSIZE];
>  
>  	struct perf_data_file file = {
> -- 
> 1.9.3

[tip:perf/urgent] perf buildid cache: Fix -a segfault related to kcore handling

[tip-bot for Jiri Olsa]

Commit-ID:  eec5a688f426d6fe4097feb0b916ad41803d2ebb
Gitweb:     http://git.kernel.org/tip/eec5a688f426d6fe4097feb0b916ad41803d2ebb
Author:     Jiri Olsa <jolsa@kernel.org>
AuthorDate: Mon, 1 Dec 2014 20:06:27 +0100
Committer:  Arnaldo Carvalho de Melo <acme@redhat.com>
CommitDate: Tue, 9 Dec 2014 09:14:34 -0300

perf buildid cache: Fix -a segfault related to kcore handling

The kcore_filename is uninitialized and trash value could trigger
build_id_cache__add_kcore function ending up with segfault.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Corey Ashford <cjashfor@linux.vnet.ibm.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Stephane Eranian <eranian@google.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Link: http://lkml.kernel.org/r/1417460789-13874-7-git-send-email-jolsa@kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
 tools/perf/builtin-buildid-cache.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tools/perf/builtin-buildid-cache.c b/tools/perf/builtin-buildid-cache.c
index 29f24c0..77d5cae 100644
--- a/tools/perf/builtin-buildid-cache.c
+++ b/tools/perf/builtin-buildid-cache.c
@@ -289,7 +289,7 @@ int cmd_buildid_cache(int argc, const char **argv,
 		   *remove_name_list_str = NULL,
 		   *missing_filename = NULL,
 		   *update_name_list_str = NULL,
-		   *kcore_filename;
+		   *kcore_filename = NULL;
 	char sbuf[STRERR_BUFSIZE];
 
 	struct perf_data_file file = {

[PATCH 7/8] perf buildid-cache: Fix kallsyms removal

[Jiri Olsa]

Add support to remove '[kernel.kallsyms]' file. Currently
the removal fails because buildid read fails (obviously on
kallsyms file).

However the file name itself holds the buildid which is
used in the .build-id link. Adding check to detect the
'[kernel.kallsyms]' removal and using the filename as
the buildid.

Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Corey Ashford <cjashfor@linux.vnet.ibm.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Stephane Eranian <eranian@google.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 tools/perf/builtin-buildid-cache.c | 34 ++++++++++++++++++++++++++++++++--
 1 file changed, 32 insertions(+), 2 deletions(-)

diff --git a/tools/perf/builtin-buildid-cache.c b/tools/perf/builtin-buildid-cache.c
index 7a3504a1b494..5e076b8d5164 100644
--- a/tools/perf/builtin-buildid-cache.c
+++ b/tools/perf/builtin-buildid-cache.c
@@ -207,6 +207,25 @@ static int build_id_cache__add_file(const char *filename, const char *debugdir)
 	return err;
 }
 
+/*
+ * Takes basename from @filename argument and
+ * copy that into @buf.
+ */
+static int scnprintf_base(char *buf, int size, const char *filename)
+{
+	char *base = strrchr(filename, '/');
+
+	if (!base++)
+		return -1;
+
+	return scnprintf(buf, size, base, strlen(base));
+}
+
+static bool is_kallsyms_file(const char *filename)
+{
+	return strstr(filename, "kernel.kallsyms");
+}
+
 static int build_id_cache__remove_file(const char *filename,
 				       const char *debugdir)
 {
@@ -217,10 +236,21 @@ static int build_id_cache__remove_file(const char *filename,
 
 	if (filename__read_build_id(filename, &build_id, sizeof(build_id)) < 0) {
 		pr_debug("Couldn't read a build-id in %s\n", filename);
-		return -1;
+
+		if (!is_kallsyms_file(filename))
+			return -1;
+
+		pr_debug("Detected [kernel.kallsyms] file, trying basename as buildid.\n");
+
+		if (scnprintf_base(sbuild_id, sizeof(sbuild_id),
+					     filename) < 0) {
+			pr_debug("failed to get build-id\n");
+			return -1;
+		}
+	} else {
+		build_id__sprintf(build_id, sizeof(build_id), sbuild_id);
 	}
 
-	build_id__sprintf(build_id, sizeof(build_id), sbuild_id);
 	err = build_id_cache__remove_s(sbuild_id, debugdir);
 	if (verbose)
 		pr_info("Removing %s %s: %s\n", sbuild_id, filename,
-- 
1.9.3

[PATCH 8/8] perf buildid-cache: Try to remove empty directories

[Jiri Olsa]

The cache removal routine only removes files leaving
directories untouched. Adding code that tries to
remove directories up to the 'buildid_dir' level.

Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Corey Ashford <cjashfor@linux.vnet.ibm.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Stephane Eranian <eranian@google.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 tools/perf/util/build-id.c | 44 +++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 43 insertions(+), 1 deletion(-)

diff --git a/tools/perf/util/build-id.c b/tools/perf/util/build-id.c
index 0c72680a977f..c5dc066db6e9 100644
--- a/tools/perf/util/build-id.c
+++ b/tools/perf/util/build-id.c
@@ -329,11 +329,41 @@ static int build_id_cache__add_b(const u8 *build_id, size_t build_id_size,
 				     is_kallsyms, is_vdso);
 }
 
+/*
+ * Tries to remove directory of the file given in
+ * the @file parameter. It might change the string
+ * pointed to by the @file pointer.
+ */
+static void try_remove_dir(char *file, const char *debugdir)
+{
+	char *dir = dirname(file);
+
+	while (dir) {
+		char *slash;
+
+		if (!strcmp(dir, debugdir))
+			return;
+
+		pr_debug("try_remove_dir %s\n", dir);
+
+		/* We dont care if it failes, we tried.. */
+		if (rmdir(dir))
+			return;
+
+		slash = strrchr(dir, '/');
+		if (!slash)
+			return;
+
+		*slash = 0;
+	}
+}
+
 int build_id_cache__remove_s(const char *sbuild_id, const char *debugdir)
 {
 	const size_t size = PATH_MAX;
 	char *filename = zalloc(size),
-	     *linkname = zalloc(size);
+	     *linkname = zalloc(size),
+	     *build_id_link = NULL;
 	int err = -1;
 
 	if (filename == NULL || linkname == NULL)
@@ -351,6 +381,10 @@ int build_id_cache__remove_s(const char *sbuild_id, const char *debugdir)
 	if (unlink(linkname))
 		goto out_free;
 
+	build_id_link = strdup(linkname);
+	if (!build_id_link)
+		goto out_free;
+
 	/*
 	 * Since the link is relative, we must make it absolute:
 	 */
@@ -360,10 +394,18 @@ int build_id_cache__remove_s(const char *sbuild_id, const char *debugdir)
 	if (unlink(linkname))
 		goto out_free;
 
+	/*
+	 * The order matters here, because linkname path
+	 * is based on build_id_link path.
+	 */
+	try_remove_dir(linkname, debugdir);
+	try_remove_dir(build_id_link, debugdir);
+
 	err = 0;
 out_free:
 	free(filename);
 	free(linkname);
+	free(build_id_link);
 	return err;
 }
 
-- 
1.9.3

Re: [PATCH 0/8] perf buildid-cache: Add clean command

[Arnaldo Carvalho de Melo]

Em Mon, Dec 01, 2014 at 08:06:21PM +0100, Jiri Olsa escreveu:
> hi,
> adding 'perf buildid-cache clean' command to allow removal
> files from '~/.debug` cache plus other fixes.
> 
> Basically the clean command allows to display/remove cache
> files/sizes like:
> 
> Display cache files older than 3 days:
>   $ perf buildid-cache clean 3d

This is confusing, I went on parsing and the above command meant: clean
the files that are 3 days or older.

Right now we have:

 usage: perf buildid-cache [<options>]

    -a, --add <file list>
                          file(s) to add
    -k, --kcore <file>    kcore file to add
    -r, --remove <file list>
                          file(s) to remove
    -M, --missing <file>  to find missing build ids in the cache
    -f, --force           don't complain, do it
    -u, --update <file list>
                          file(s) to update
    -v, --verbose         be more verbose

[acme@sandy wb]$ 

If we keep the way it is, I think it would be clearer to add a -l/--list
command and that would:

  perf buildid-cache --list  # list all the files in the cache
  perf buildid-cache --list 3d  # list 3 days old files
  perf buildid-cache --list +3d  # list files 3 days or more older
  perf buildid-cache --list -3d  # list files up to 3 days old

And all these would print the sum of the matching files, i.e. the space
those files use in the filesystem.
 
> Remove cache files older than 3 days:
>   $ perf buildid-cache clean 3d -r

  perf buildid-cache --remove [-+]Nd # Same semantics as above

>     
> Total cache removal:
>   $ perf buildid-cache clean -r

 perf buildid-cache --remove '*'
 
> Remove and display items bigger than 200M
>   $ perf buildid-cache clean -r -a 200M

But yeah, perhaps we could switch to having subcommands and add:

  perf buildid-cache [add|remove|update|list]

What do you think?

Also in this process, we should rename buildid-cache to a shorter form,
'cache', i.e. 'perf cache', or we can leave this for later, for Masami's
patchkit we discussed recently.

- Arnaldo
 
> 
> Also available in here:
>   git://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git
>   perf/core_buildid_cache_clean
> 
> thanks,
> jirka
> 
> 
> Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
> Cc: Corey Ashford <cjashfor@linux.vnet.ibm.com>
> Cc: David Ahern <dsahern@gmail.com>
> Cc: Frederic Weisbecker <fweisbec@gmail.com>
> Cc: Ingo Molnar <mingo@kernel.org>
> Cc: Namhyung Kim <namhyung@kernel.org>
> Cc: Paul Mackerras <paulus@samba.org>
> Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
> Cc: Stephane Eranian <eranian@google.com>
> Cc: Steven Rostedt <rostedt@goodmis.org>
> Signed-off-by: Jiri Olsa <jolsa@kernel.org>
> ---
> Jiri Olsa (8):
>       perf tools: Use single strcmp call instead of two
>       perf tools: Remove extra debugdir variables
>       perf tools: Add --buildid-dir option to set cache directory
>       perf buildid-cache: Add clean command
>       perf buildid-cache: Add automated tests
>       perf buildid cache: Fix -a segfault
>       perf buildid-cache: Fix kallsyms removal
>       perf buildid-cache: Try to remove empty directories
> 
>  tools/perf/Documentation/perf-buildid-cache.txt |  59 +++++++++++
>  tools/perf/Documentation/perf.txt               |   4 +
>  tools/perf/Makefile.perf                        |   7 +-
>  tools/perf/builtin-buildid-cache.c              | 501 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++--
>  tools/perf/perf.c                               |  14 ++-
>  tools/perf/tests/buildid-cache.c                |  88 ++++++++++++++++
>  tools/perf/tests/buildid-cache.sh               |  60 +++++++++++
>  tools/perf/tests/builtin-test.c                 |   8 ++
>  tools/perf/tests/tests.h                        |   1 +
>  tools/perf/util/build-id.c                      |  53 ++++++++--
>  tools/perf/util/config.c                        |  10 +-
>  tools/perf/util/util.h                          |   2 +-
>  12 files changed, 781 insertions(+), 26 deletions(-)
>  create mode 100644 tools/perf/tests/buildid-cache.c
>  create mode 100755 tools/perf/tests/buildid-cache.sh

Re: [PATCH 0/8] perf buildid-cache: Add clean command

[Jiri Olsa]

On Mon, Dec 01, 2014 at 08:29:19PM -0200, Arnaldo Carvalho de Melo wrote:

SNIP

> [acme@sandy wb]$ 
> 
> If we keep the way it is, I think it would be clearer to add a -l/--list
> command and that would:
> 
>   perf buildid-cache --list  # list all the files in the cache
>   perf buildid-cache --list 3d  # list 3 days old files
>   perf buildid-cache --list +3d  # list files 3 days or more older
>   perf buildid-cache --list -3d  # list files up to 3 days old

ok, I was up to old files (and big files) removal only.. not sure
the other way will be actually needed.. but it's easy enought to
add

> 
> And all these would print the sum of the matching files, i.e. the space
> those files use in the filesystem.
>  
> > Remove cache files older than 3 days:
> >   $ perf buildid-cache clean 3d -r
> 
>   perf buildid-cache --remove [-+]Nd # Same semantics as above
> 
> >     
> > Total cache removal:
> >   $ perf buildid-cache clean -r
> 
>  perf buildid-cache --remove '*'

hum, maybe '--removal all' might be better

>  
> > Remove and display items bigger than 200M
> >   $ perf buildid-cache clean -r -a 200M
> 
> But yeah, perhaps we could switch to having subcommands and add:
> 
>   perf buildid-cache [add|remove|update|list]
> 
> What do you think?

yep, those commands seem nicer.. 

let's see of there're other comments and I'll come up with
new interface after..

thanks,
jirka

Re: [PATCH 0/8] perf buildid-cache: Add clean command

[Arnaldo Carvalho de Melo]

Em Tue, Dec 02, 2014 at 01:25:18PM +0100, Jiri Olsa escreveu:
> On Mon, Dec 01, 2014 at 08:29:19PM -0200, Arnaldo Carvalho de Melo wrote:
 
> SNIP
 
> > [acme@sandy wb]$ 

> > If we keep the way it is, I think it would be clearer to add a -l/--list
> > command and that would:

> >   perf buildid-cache --list  # list all the files in the cache
> >   perf buildid-cache --list 3d  # list 3 days old files
> >   perf buildid-cache --list +3d  # list files 3 days or more older
> >   perf buildid-cache --list -3d  # list files up to 3 days old
 
> ok, I was up to old files (and big files) removal only.. not sure
> the other way will be actually needed.. but it's easy enought to
> add

Right, I was thinking something along the lines of checking how much
space would be needed if I wanted to keep X days of cache, not how much
space I would free if I purged files older than Y days.

 
> > And all these would print the sum of the matching files, i.e. the space
> > those files use in the filesystem.

> > > Remove cache files older than 3 days:
> > >   $ perf buildid-cache clean 3d -r

> >   perf buildid-cache --remove [-+]Nd # Same semantics as above

> > >     
> > > Total cache removal:
> > >   $ perf buildid-cache clean -r

> >  perf buildid-cache --remove '*'
> 
> hum, maybe '--removal all' might be better

Humm, lemme see how ccache does this operation... That would be an
existing cache management workflow that we could reuse:


[acme@zoo linux]$ ccache --help
Usage:
  ccache [options]
  ccache compiler [compiler options]
  compiler [compiler options]          (via symbolic link)

Options:
    -c, --cleanup        delete old files and recalculate size counters
                         (normally not needed as this is done automatically)
    -C, --clear          clear the cache completely
    -F, --max-files=N    set maximum number of files in cache to N (use 0 for
                         no limit)
    -M, --max-size=SIZE  set maximum size of cache to SIZE (use 0 for no
                         limit; available suffixes: G, M and K; default
                         suffix: G)
    -s, --show-stats     show statistics summary
    -z, --zero-stats     zero statistics counters

    -h, --help           print this help text
    -V, --version        print version and copyright information

  See also <http://ccache.samba.org>.
  [acme@zoo linux]$ 

So, there you go, the 'clear' keywoard being used for doing what you
called 'removeall' and I called --remove '*'. Perhaps asking if the user
really wants to do that while, just like with the time limited 'remove'
operations showing how much space it will release (albeit that could
take some time to calculate...)?
 
> >  
> > > Remove and display items bigger than 200M
> > >   $ perf buildid-cache clean -r -a 200M

> > But yeah, perhaps we could switch to having subcommands and add:

> >   perf buildid-cache [add|remove|update|list]
 
> > What do you think?
 
> yep, those commands seem nicer.. 
 
> let's see of there're other comments and I'll come up with
> new interface after..

Ok!

- Arnaldo

Re: [PATCH 0/8] perf buildid-cache: Add clean command

[Namhyung Kim]

Hi Arnaldo and Jiri,

On Tue, Dec 2, 2014 at 11:10 PM, Arnaldo Carvalho de Melo
<acme@redhat.com> wrote:
> Em Tue, Dec 02, 2014 at 01:25:18PM +0100, Jiri Olsa escreveu:
>> On Mon, Dec 01, 2014 at 08:29:19PM -0200, Arnaldo Carvalho de Melo wrote:
>
>> SNIP
>
>> > [acme@sandy wb]$
>
>> > If we keep the way it is, I think it would be clearer to add a -l/--list
>> > command and that would:
>
>> >   perf buildid-cache --list  # list all the files in the cache
>> >   perf buildid-cache --list 3d  # list 3 days old files
>> >   perf buildid-cache --list +3d  # list files 3 days or more older
>> >   perf buildid-cache --list -3d  # list files up to 3 days old
>
>> ok, I was up to old files (and big files) removal only.. not sure
>> the other way will be actually needed.. but it's easy enought to
>> add
>
> Right, I was thinking something along the lines of checking how much
> space would be needed if I wanted to keep X days of cache, not how much
> space I would free if I purged files older than Y days.

Btw, which timestamp do you use for the checking?  It'd be great if we
can use atime but some system might use the noatime..


>
>
>> > And all these would print the sum of the matching files, i.e. the space
>> > those files use in the filesystem.
>
>> > > Remove cache files older than 3 days:
>> > >   $ perf buildid-cache clean 3d -r
>
>> >   perf buildid-cache --remove [-+]Nd # Same semantics as above
>
>> > >
>> > > Total cache removal:
>> > >   $ perf buildid-cache clean -r
>
>> >  perf buildid-cache --remove '*'
>>
>> hum, maybe '--removal all' might be better
>
> Humm, lemme see how ccache does this operation... That would be an
> existing cache management workflow that we could reuse:
>
>
> [acme@zoo linux]$ ccache --help
> Usage:
>   ccache [options]
>   ccache compiler [compiler options]
>   compiler [compiler options]          (via symbolic link)
>
> Options:
>     -c, --cleanup        delete old files and recalculate size counters
>                          (normally not needed as this is done automatically)
>     -C, --clear          clear the cache completely
>     -F, --max-files=N    set maximum number of files in cache to N (use 0 for
>                          no limit)
>     -M, --max-size=SIZE  set maximum size of cache to SIZE (use 0 for no
>                          limit; available suffixes: G, M and K; default
>                          suffix: G)
>     -s, --show-stats     show statistics summary
>     -z, --zero-stats     zero statistics counters
>
>     -h, --help           print this help text
>     -V, --version        print version and copyright information
>
>   See also <http://ccache.samba.org>.
>   [acme@zoo linux]$
>
> So, there you go, the 'clear' keywoard being used for doing what you
> called 'removeall' and I called --remove '*'. Perhaps asking if the user
> really wants to do that while, just like with the time limited 'remove'
> operations showing how much space it will release (albeit that could
> take some time to calculate...)?

Yeah, I think we also need to have the stats option/command.  And some
kind of cache management policies too - like max size and max days? -
so that it can trigger auto cleanup.. ;-)


>
>> >
>> > > Remove and display items bigger than 200M
>> > >   $ perf buildid-cache clean -r -a 200M
>
>> > But yeah, perhaps we could switch to having subcommands and add:
>
>> >   perf buildid-cache [add|remove|update|list]
>
>> > What do you think?
>
>> yep, those commands seem nicer..
>
>> let's see of there're other comments and I'll come up with
>> new interface after..

I also like the subcommand style..  What about git-like format to
specify time range?

perf buildid-cache list --since '3 days ago'
perf buildidi-cache remove --before '1 month ago'


Thanks,
Namhyung

Re: [PATCH 0/8] perf buildid-cache: Add clean command

[Jiri Olsa]

On Wed, Dec 03, 2014 at 05:13:06PM +0900, Namhyung Kim wrote:
> Hi Arnaldo and Jiri,
> 
> On Tue, Dec 2, 2014 at 11:10 PM, Arnaldo Carvalho de Melo
> <acme@redhat.com> wrote:
> > Em Tue, Dec 02, 2014 at 01:25:18PM +0100, Jiri Olsa escreveu:
> >> On Mon, Dec 01, 2014 at 08:29:19PM -0200, Arnaldo Carvalho de Melo wrote:
> >
> >> SNIP
> >
> >> > [acme@sandy wb]$
> >
> >> > If we keep the way it is, I think it would be clearer to add a -l/--list
> >> > command and that would:
> >
> >> >   perf buildid-cache --list  # list all the files in the cache
> >> >   perf buildid-cache --list 3d  # list 3 days old files
> >> >   perf buildid-cache --list +3d  # list files 3 days or more older
> >> >   perf buildid-cache --list -3d  # list files up to 3 days old
> >
> >> ok, I was up to old files (and big files) removal only.. not sure
> >> the other way will be actually needed.. but it's easy enought to
> >> add
> >
> > Right, I was thinking something along the lines of checking how much
> > space would be needed if I wanted to keep X days of cache, not how much
> > space I would free if I purged files older than Y days.
> 
> Btw, which timestamp do you use for the checking?  It'd be great if we
> can use atime but some system might use the noatime..

I used atime.. will need to check, I wasnt aware of this

SNIP

> >
> >> >
> >> > > Remove and display items bigger than 200M
> >> > >   $ perf buildid-cache clean -r -a 200M
> >
> >> > But yeah, perhaps we could switch to having subcommands and add:
> >
> >> >   perf buildid-cache [add|remove|update|list]
> >
> >> > What do you think?
> >
> >> yep, those commands seem nicer..
> >
> >> let's see of there're other comments and I'll come up with
> >> new interface after..
> 
> I also like the subcommand style..  What about git-like format to
> specify time range?
> 
> perf buildid-cache list --since '3 days ago'
> perf buildidi-cache remove --before '1 month ago'

yea, thats more readable, I'll check git sources.. hopefully they use
this in some steal-able way ;-)

jirka