* [PATCH 01/10] trace-cmd: Add APIs for library initialization and free
2021-04-22 7:38 [POC PATCH 00/10] Add trace file compression Tzvetomir Stoyanov (VMware)
@ 2021-04-22 7:38 ` Tzvetomir Stoyanov (VMware)
2021-04-22 7:38 ` [PATCH 02/10] trace-cmd library: Add support for compression algorithms Tzvetomir Stoyanov (VMware)
` (9 subsequent siblings)
10 siblings, 0 replies; 14+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-04-22 7:38 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
The trace-cmd library has no APIs for initialization and free of the
whole library. Added these new APIs:
tracecmd_lib_init()
tracecmd_lib_free()
and call them in trace-cmd main function.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
lib/trace-cmd/include/private/trace-cmd-private.h | 3 +++
lib/trace-cmd/trace-util.c | 9 +++++++++
tracecmd/trace-cmd.c | 11 ++++++++---
3 files changed, 20 insertions(+), 3 deletions(-)
diff --git a/lib/trace-cmd/include/private/trace-cmd-private.h b/lib/trace-cmd/include/private/trace-cmd-private.h
index def01b68..5d35b197 100644
--- a/lib/trace-cmd/include/private/trace-cmd-private.h
+++ b/lib/trace-cmd/include/private/trace-cmd-private.h
@@ -29,6 +29,9 @@
struct tep_plugin_list *trace_load_plugins(struct tep_handle *tep, int flags);
+int tracecmd_lib_init(void);
+void tracecmd_lib_free(void);
+
int *tracecmd_add_id(int *list, int id, int len);
enum {
diff --git a/lib/trace-cmd/trace-util.c b/lib/trace-cmd/trace-util.c
index a6fc64c8..3ef10eae 100644
--- a/lib/trace-cmd/trace-util.c
+++ b/lib/trace-cmd/trace-util.c
@@ -589,3 +589,12 @@ bool tracecmd_is_version_supported(unsigned int version)
return true;
return false;
}
+
+int tracecmd_lib_init(void)
+{
+ return 0;
+}
+
+void tracecmd_lib_free(void)
+{
+}
diff --git a/tracecmd/trace-cmd.c b/tracecmd/trace-cmd.c
index 7376c5a5..aad8e0a7 100644
--- a/tracecmd/trace-cmd.c
+++ b/tracecmd/trace-cmd.c
@@ -116,15 +116,20 @@ int main (int argc, char **argv)
if (argc < 2)
trace_usage(argc, argv);
+ tracecmd_lib_init();
+
for (i = 0; i < ARRAY_SIZE(commands); ++i) {
if (strcmp(argv[1], commands[i].name) == 0 ){
commands[i].run(argc, argv);
- goto out;
+ break;
}
}
+ tracecmd_lib_free();
+
/* No valid command found, show help */
- trace_usage(argc, argv);
-out:
+ if (i == ARRAY_SIZE(commands))
+ trace_usage(argc, argv);
+
exit(0);
}
--
2.30.2
^ permalink raw reply related [flat|nested] 14+ messages in thread* [PATCH 02/10] trace-cmd library: Add support for compression algorithms
2021-04-22 7:38 [POC PATCH 00/10] Add trace file compression Tzvetomir Stoyanov (VMware)
2021-04-22 7:38 ` [PATCH 01/10] trace-cmd: Add APIs for library initialization and free Tzvetomir Stoyanov (VMware)
@ 2021-04-22 7:38 ` Tzvetomir Stoyanov (VMware)
2021-04-22 7:38 ` [PATCH 03/10] trace-cmd library: Compress part of the trace file Tzvetomir Stoyanov (VMware)
` (8 subsequent siblings)
10 siblings, 0 replies; 14+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-04-22 7:38 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
Added infrastructure to trace-cmd library for compression. Introduced
various new APIs to work with this new functionality:
tracecmd_compress_proto_get()
tracecmd_compress_protos_get()
tracecmd_compress_proto_get_name()
tracecmd_compress_proto_select()
tracecmd_compress_proto_register()
tracecmd_compress_data()
tracecmd_uncompress_data()
tracecmd_get_compress_size()
The compression algorithms are not part of this patch.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
lib/trace-cmd/Makefile | 1 +
.../include/private/trace-cmd-private.h | 22 ++
lib/trace-cmd/include/trace-cmd-local.h | 2 +
lib/trace-cmd/trace-compress.c | 287 ++++++++++++++++++
lib/trace-cmd/trace-util.c | 3 +
5 files changed, 315 insertions(+)
create mode 100644 lib/trace-cmd/trace-compress.c
diff --git a/lib/trace-cmd/Makefile b/lib/trace-cmd/Makefile
index 17600318..bab4322d 100644
--- a/lib/trace-cmd/Makefile
+++ b/lib/trace-cmd/Makefile
@@ -25,6 +25,7 @@ ifeq ($(VSOCK_DEFINED), 1)
OBJS += trace-timesync-ptp.o
OBJS += trace-timesync-kvm.o
endif
+OBJS += trace-compress.o
# Additional util objects
OBJS += trace-blk-hack.o
diff --git a/lib/trace-cmd/include/private/trace-cmd-private.h b/lib/trace-cmd/include/private/trace-cmd-private.h
index 5d35b197..c562fc5d 100644
--- a/lib/trace-cmd/include/private/trace-cmd-private.h
+++ b/lib/trace-cmd/include/private/trace-cmd-private.h
@@ -471,6 +471,28 @@ void tracecmd_tsync_free(struct tracecmd_time_sync *tsync);
int tracecmd_write_guest_time_shift(struct tracecmd_output *handle,
struct tracecmd_time_sync *tsync);
+/* --- Compression --- */
+struct tracecmd_compress_proto;
+struct tracecmd_compress_proto *tracecmd_compress_proto_get(const char *name, const char *version);
+struct tracecmd_compress_proto *tracecmd_compress_proto_select(void);
+int tracecmd_compress_proto_register(const char *name, const char *version, int weight,
+ int (*compress)(char *in, unsigned int in_bytes,
+ char *out, unsigned int *out_bytes),
+ int (*uncompress)(char *in, unsigned int in_bytes,
+ char *out, unsigned int *out_bytes),
+ unsigned int (*comress_size)(unsigned int bytes),
+ bool (*is_supported)(const char *name, const char *version));
+int tracecmd_compress_protos_get(char ***names, char ***versions);
+int tracecmd_compress_proto_get_name(struct tracecmd_compress_proto *proto,
+ const char **name, const char **version);
+int tracecmd_compress_data(struct tracecmd_compress_proto *proto,
+ char *in, unsigned int in_bytes,
+ char *out, unsigned int *out_bytes);
+int tracecmd_uncompress_data(struct tracecmd_compress_proto *proto,
+ char *in, unsigned int in_bytes,
+ char *out, unsigned int *out_bytes);
+unsigned int tracecmd_get_compress_size(struct tracecmd_compress_proto *proto, unsigned int bytes);
+
/* --- Plugin handling --- */
extern struct tep_plugin_option trace_ftrace_options[];
diff --git a/lib/trace-cmd/include/trace-cmd-local.h b/lib/trace-cmd/include/trace-cmd-local.h
index 5578f00d..c8c51ff0 100644
--- a/lib/trace-cmd/include/trace-cmd-local.h
+++ b/lib/trace-cmd/include/trace-cmd-local.h
@@ -30,5 +30,7 @@ void tracecmd_fatal(const char *fmt, ...);
#endif
#endif
+int tracecmd_compress_init(void);
+void tracecmd_compress_free(void);
#endif /* _TRACE_CMD_LOCAL_H */
diff --git a/lib/trace-cmd/trace-compress.c b/lib/trace-cmd/trace-compress.c
new file mode 100644
index 00000000..c59c8bdb
--- /dev/null
+++ b/lib/trace-cmd/trace-compress.c
@@ -0,0 +1,287 @@
+// SPDX-License-Identifier: LGPL-2.1
+/*
+ * Copyright (C) 2021, VMware, Tzvetomir Stoyanov tz.stoyanov@gmail.com>
+ *
+ */
+#include <stdlib.h>
+#include <sys/time.h>
+
+#include "trace-cmd-private.h"
+#include "trace-cmd-local.h"
+
+struct tracecmd_compress_proto {
+ struct tracecmd_compress_proto *next;
+ char *proto_name;
+ char *proto_version;
+ int weight;
+
+ int (*comress_block)(char *in, unsigned int in_bytes,
+ char *out, unsigned int *out_bytes);
+ int (*uncompress_block)(char *in, unsigned int in_bytes,
+ char *out, unsigned int *out_bytes);
+ unsigned int (*comress_size)(unsigned int bytes);
+ bool (*is_supported)(const char *name, const char *version);
+};
+
+static struct tracecmd_compress_proto *proto_list;
+
+/**
+ * tracecmd_compress_init - initialize the library with available compression algorithms
+ *
+ * Returns 0. If no compression algorithms are available, a warning is printed.
+ */
+int tracecmd_compress_init(void)
+{
+ struct timeval time;
+ int count = 0;
+
+ gettimeofday(&time, NULL);
+ srand((time.tv_sec * 1000) + (time.tv_usec / 1000));
+
+/* toDo: add here initialization of compression protocols */
+
+ if (!count)
+ tracecmd_warning("No compression support available");
+
+ return 0;
+}
+
+/**
+ * tracecmd_compress_proto_get - get handler to compression algorithm from name and version
+ * @name: name of the compression algorithm.
+ * @version: version of the compression algorithm.
+ *
+ * Returns a handler to compression algorithm, or NULL if algorithm with given name and version is
+ * not found.
+ */
+struct tracecmd_compress_proto *tracecmd_compress_proto_get(const char *name, const char *version)
+{
+ struct tracecmd_compress_proto *proto = proto_list;
+
+ if (!name)
+ return NULL;
+
+ while (proto) {
+ if (proto->is_supported && proto->is_supported(name, version))
+ return proto;
+ proto = proto->next;
+ }
+ return NULL;
+}
+
+/**
+ * tracecmd_compress_proto_select - select compression algorithm
+ *
+ * The algorithm with the lowest weight is selected.
+ * Returns a handler to compression algorithm, or NULL if no algorithms are available.
+ */
+struct tracecmd_compress_proto *tracecmd_compress_proto_select(void)
+{
+ struct tracecmd_compress_proto *proto = proto_list;
+ struct tracecmd_compress_proto *selected = NULL;
+
+ while (proto) {
+ if (!selected || selected->weight > proto->weight)
+ selected = proto;
+ proto = proto->next;
+ }
+
+ return selected;
+}
+
+/**
+ * tracecmd_compress_proto_get_name - get name and version of compression algorithm
+ * @proto: handler to compression algorithm.
+ * @name: return, name of the compression algorithm.
+ * @version: return, version of the compression algorithm.
+ *
+ * Returns 0 on success, or -1 in case of an error. If 0 is returned, the name and version of the
+ * algorithm are stored in @name and @version. The returned strings must *not* be freed.
+ */
+int tracecmd_compress_proto_get_name(struct tracecmd_compress_proto *proto,
+ const char **name, const char **version)
+{
+ if (!proto)
+ return -1;
+ if (name)
+ *name = proto->proto_name;
+ if (version)
+ *version = proto->proto_version;
+ return 0;
+}
+
+/**
+ * tracecmd_compress_proto_register - register a new compression algorithm
+ * @name: name of the compression algorithm.
+ * @version: version of the compression algorithm.
+ * @weight: weight of the compression algorithm, lower is better.
+ * @compress: compression hook, called to compress a memory block.
+ * @uncompress: uncompression hook, called to uncompress a memory block.
+ * @comress_size: hook, called to get the required minimum size of the buffer for compression
+ * given number of bytes.
+ * @is_supported: check hook, called to check if compression with given name and version is
+ * supported by this plugin.
+ *
+ * Returns 0 on success, or -1 in case of an error. If algorithm with given name and version is
+ * already registered, -1 is returned.
+ */
+int tracecmd_compress_proto_register(const char *name, const char *version, int weight,
+ int (*compress)(char *in, unsigned int in_bytes,
+ char *out, unsigned int *out_bytes),
+ int (*uncompress)(char *in, unsigned int in_bytes,
+ char *out, unsigned int *out_bytes),
+ unsigned int (*comress_size)(unsigned int bytes),
+ bool (*is_supported)(const char *name, const char *version))
+{
+ struct tracecmd_compress_proto *new;
+
+ if (!name || !compress || !uncompress)
+ return -1;
+ if (tracecmd_compress_proto_get(name, version))
+ return -1;
+
+ new = calloc(1, sizeof(*new));
+ if (!new)
+ return -1;
+
+ new->proto_name = strdup(name);
+ if (!new->proto_name)
+ goto error;
+ new->proto_version = strdup(version);
+ if (!new->proto_version)
+ goto error;
+ new->comress_block = compress;
+ new->uncompress_block = uncompress;
+ new->comress_size = comress_size;
+ new->is_supported = is_supported;
+ new->weight = weight;
+ new->next = proto_list;
+ proto_list = new;
+ return 0;
+
+error:
+ free(new->proto_name);
+ free(new->proto_version);
+ free(new);
+ return -1;
+}
+
+/**
+ * tracecmd_compress_free - free the library resources, related to available compression algorithms
+ *
+ */
+void tracecmd_compress_free(void)
+{
+ struct tracecmd_compress_proto *proto = proto_list;
+ struct tracecmd_compress_proto *del;
+
+ while (proto) {
+ del = proto;
+ proto = proto->next;
+ free(del->proto_name);
+ free(del->proto_version);
+ free(del);
+ }
+ proto_list = NULL;
+}
+
+/**
+ * tracecmd_compress_data - compress a memory block
+ * @proto: handle to compression algorithm that will be used to compress the data.
+ * @in: pointer to the data that will be compressed.
+ * @in_bytes: number of bytes in @in.
+ * @out: pre-allocated memory to store the compressed data. It must be at least the size returned
+ * by tracecmd_get_compress_size().
+ * @out_bytes: size of the allocated memory in @out. On successful return, @out_bytes is updated
+ * with the size of the compressed data.
+ *
+ * Returns 0 on success, or -1 in case of an error.
+ */
+int tracecmd_compress_data(struct tracecmd_compress_proto *proto,
+ char *in, unsigned int in_bytes,
+ char *out, unsigned int *out_bytes)
+{
+ if (proto && proto->comress_block)
+ return proto->comress_block(in, in_bytes, out, out_bytes);
+ return -1;
+}
+
+/**
+ * tracecmd_uncompress_data - uncompress a memory block
+ * @proto: handle to compression algorithm that will be used to uncompress the data.
+ * @in: pointer to the data that will be uncompressed.
+ * @in_bytes: number of bytes in @in.
+ * @out: pre-allocated memory to store the uncompressed data.
+ * @out_bytes: size of the allocated memory in @out. On successful return, @out_bytes is updated
+ * with the size of the uncompressed data.
+ *
+ * Returns 0 on success, or -1 in case of an error.
+ */
+int tracecmd_uncompress_data(struct tracecmd_compress_proto *proto,
+ char *in, unsigned int in_bytes,
+ char *out, unsigned int *out_bytes)
+{
+ if (proto && proto->uncompress_block)
+ return proto->uncompress_block(in, in_bytes, out, out_bytes);
+ return -1;
+}
+
+/**
+ * tracecmd_get_compress_size - get the buffer size, required to compress given data
+ * @proto: handle to compression algorithm that will be used to compress the data.
+ * @bytes: number of bytes that will be compressed.
+ *
+ * Returns the required size of the buffer for compression of @bytes, or 0 in case of an error
+ */
+unsigned int tracecmd_get_compress_size(struct tracecmd_compress_proto *proto, unsigned int bytes)
+{
+ if (proto && proto->comress_size)
+ return proto->comress_size(bytes);
+ return -1;
+}
+
+
+/**
+ * tracecmd_compress_protos_get - get a list of all supported compression algorithms and versions
+ * @names: return, array with names of all supported compression algorithms
+ * @versions: return, array with versions of all supported compression algorithms
+ *
+ * On success, the size of @names and @versions arrays is returned. Those arrays are allocated by
+ * the API and must be freed with free() by the caller. Both arrays are with same size, each name
+ * from @names corresponds to a version from @versions.
+ * On error -1 is returned and @names and @versions arrays are not allocated.
+ */
+int tracecmd_compress_protos_get(char ***names, char ***versions)
+{
+ struct tracecmd_compress_proto *proto = proto_list;
+ char **n = NULL;
+ char **v = NULL;
+ int c, i;
+
+ for (c = 0; proto; proto = proto->next)
+ ;
+
+ if (c < 1)
+ return c;
+
+ n = calloc(c, sizeof(char *));
+ if (!n)
+ goto error;
+ v = calloc(c, sizeof(char *));
+ if (!n)
+ goto error;
+
+ for (i = 0; proto; proto = proto->next) {
+ n[i] = proto->proto_name;
+ v[i] = proto->proto_version;
+ }
+
+ names = &n;
+ versions = &v;
+ return c;
+
+error:
+ free(n);
+ free(v);
+ return -1;
+}
diff --git a/lib/trace-cmd/trace-util.c b/lib/trace-cmd/trace-util.c
index 3ef10eae..a42499fe 100644
--- a/lib/trace-cmd/trace-util.c
+++ b/lib/trace-cmd/trace-util.c
@@ -592,9 +592,12 @@ bool tracecmd_is_version_supported(unsigned int version)
int tracecmd_lib_init(void)
{
+
+ tracecmd_compress_init();
return 0;
}
void tracecmd_lib_free(void)
{
+ tracecmd_compress_free();
}
--
2.30.2
^ permalink raw reply related [flat|nested] 14+ messages in thread* [PATCH 03/10] trace-cmd library: Compress part of the trace file
2021-04-22 7:38 [POC PATCH 00/10] Add trace file compression Tzvetomir Stoyanov (VMware)
2021-04-22 7:38 ` [PATCH 01/10] trace-cmd: Add APIs for library initialization and free Tzvetomir Stoyanov (VMware)
2021-04-22 7:38 ` [PATCH 02/10] trace-cmd library: Add support for compression algorithms Tzvetomir Stoyanov (VMware)
@ 2021-04-22 7:38 ` Tzvetomir Stoyanov (VMware)
2021-04-22 7:38 ` [PATCH 04/10] trace-cmd library: Read compressed " Tzvetomir Stoyanov (VMware)
` (7 subsequent siblings)
10 siblings, 0 replies; 14+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-04-22 7:38 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
Compress some methadata of the trace.dat file. If there is compression
support, chose file version 7 and compress these parts of the file:
- ftrace events format
- format of recorded events
- information of the mapping of function addresses to the function names
- trace_printk() format strings
- information of the mapping a PID to a process name
A new compression header is added in the file, right after the page size
information.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
lib/trace-cmd/trace-output.c | 190 ++++++++++++++++++++++++++++++-----
1 file changed, 167 insertions(+), 23 deletions(-)
diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c
index edff7961..9d603cfe 100644
--- a/lib/trace-cmd/trace-output.c
+++ b/lib/trace-cmd/trace-output.c
@@ -58,6 +58,13 @@ struct tracecmd_output {
bool quiet;
unsigned long file_state;
unsigned long file_version;
+
+ bool compress;
+ unsigned int compress_size;
+ unsigned int commpress_capacity;
+ char *compress_buffer;
+ struct tracecmd_compress_proto *compress_proto;
+
struct list_head options;
struct tracecmd_msg_handle *msg_handle;
char *trace_clock;
@@ -75,9 +82,31 @@ struct list_event_system {
char *name;
};
+static stsize_t write_compress(struct tracecmd_output *handle, const void *data, tsize_t size)
+{
+ char *buf;
+ int extend;
+
+ if (handle->commpress_capacity < handle->compress_size + size) {
+ extend = (handle->compress_size + size) - handle->commpress_capacity;
+ extend = extend < BUFSIZ ? BUFSIZ : extend;
+ buf = realloc(handle->compress_buffer, handle->commpress_capacity + extend);
+ if (!buf)
+ return -ENOMEM;
+ handle->compress_buffer = buf;
+ handle->commpress_capacity += extend;
+ }
+ memcpy(&handle->compress_buffer[handle->compress_size], data, size);
+ handle->compress_size += size;
+ return 0;
+}
+
static stsize_t
do_write_check(struct tracecmd_output *handle, const void *data, tsize_t size)
{
+ if (handle->compress)
+ return write_compress(handle, data, size);
+
if (handle->msg_handle)
return tracecmd_msg_data_send(handle->msg_handle, data, size);
@@ -109,6 +138,70 @@ static unsigned long long convert_endian_8(struct tracecmd_output *handle,
return tep_read_number(handle->pevent, &val, 8);
}
+static stsize_t write_compress_data(struct tracecmd_output *handle)
+{
+ unsigned int size;
+ char *buf;
+ int endian4;
+ int ret;
+
+ size = tracecmd_get_compress_size(handle->compress_proto, handle->compress_size);
+ buf = malloc(size);
+ if (!buf)
+ return -ENOMEM;
+
+ ret = tracecmd_compress_data(handle->compress_proto, handle->compress_buffer,
+ handle->compress_size, buf, &size);
+ if (ret < 0)
+ goto out;
+ /* Write compressed data size */
+ endian4 = convert_endian_4(handle, size);
+ ret = do_write_check(handle, &endian4, 4);
+ if (ret < 0)
+ goto out;
+ /* Write uncompressed data size */
+ endian4 = convert_endian_4(handle, handle->compress_size);
+ ret = do_write_check(handle, &endian4, 4);
+ if (ret < 0)
+ goto out;
+ /* Write uncompressed the data */
+ ret = do_write_check(handle, buf, size);
+out:
+ free(buf);
+ return ret;
+}
+
+static inline void compression_reset(struct tracecmd_output *handle)
+{
+ if (handle->file_version < 7 || !handle->compress_proto)
+ return;
+ free(handle->compress_buffer);
+ handle->compress_buffer = NULL;
+ handle->commpress_capacity = 0;
+ handle->compress_size = 0;
+}
+
+static inline int compression_start(struct tracecmd_output *handle)
+{
+ if (handle->file_version < 7 || !handle->compress_proto)
+ return 0;
+ compression_reset(handle);
+ handle->compress = true;
+ return 0;
+}
+
+static inline int compression_end(struct tracecmd_output *handle)
+{
+ int ret;
+
+ if (handle->file_version < 7 || !handle->compress_proto)
+ return 0;
+ handle->compress = false;
+ ret = write_compress_data(handle);
+ compression_reset(handle);
+ return ret;
+}
+
/**
* tracecmd_set_quiet - Set if to print output to the screen
* @quiet: If non zero, print no output to the screen
@@ -659,13 +752,17 @@ static int read_ftrace_files(struct tracecmd_output *handle)
}
create_event_list_item(handle, &systems, &list);
-
+ compression_start(handle);
ret = copy_event_system(handle, systems);
+ if (!ret)
+ ret = compression_end(handle);
+ else
+ compression_reset(handle);
free_list_events(systems);
- handle->file_state = TRACECMD_FILE_FTRACE_EVENTS;
-
+ if (!ret)
+ handle->file_state = TRACECMD_FILE_FTRACE_EVENTS;
return ret;
}
@@ -715,6 +812,7 @@ static int read_event_files(struct tracecmd_output *handle,
for (slist = systems; slist; slist = slist->next)
count++;
+ compression_start(handle);
ret = -1;
endian4 = convert_endian_4(handle, count);
if (do_write_check(handle, &endian4, 4))
@@ -730,8 +828,14 @@ static int read_event_files(struct tracecmd_output *handle,
ret = copy_event_system(handle, slist);
}
- handle->file_state = TRACECMD_FILE_ALL_EVENTS;
+ if (!ret)
+ ret = compression_end(handle);
out_free:
+ if (!ret)
+ handle->file_state = TRACECMD_FILE_ALL_EVENTS;
+ else
+ compression_reset(handle);
+
free_list_events(systems);
return ret;
@@ -793,20 +897,20 @@ static int read_proc_kallsyms(struct tracecmd_output *handle,
if (kallsyms)
path = kallsyms;
-
+ compression_start(handle);
ret = stat(path, &st);
if (ret < 0) {
/* not found */
size = 0;
endian4 = convert_endian_4(handle, size);
- if (do_write_check(handle, &endian4, 4))
- return -1;
- return 0;
+ ret = do_write_check(handle, &endian4, 4);
+ goto out;
}
size = get_size(path);
endian4 = convert_endian_4(handle, size);
- if (do_write_check(handle, &endian4, 4))
- return -1;
+ ret = do_write_check(handle, &endian4, 4);
+ if (ret)
+ goto out;
set_proc_kptr_restrict(0);
check_size = copy_file(handle, path);
@@ -814,13 +918,18 @@ static int read_proc_kallsyms(struct tracecmd_output *handle,
errno = EINVAL;
tracecmd_warning("error in size of file '%s'", path);
set_proc_kptr_restrict(1);
- return -1;
+ ret = -1;
+ goto out;
}
set_proc_kptr_restrict(1);
- handle->file_state = TRACECMD_FILE_KALLSYMS;
-
- return 0;
+ ret = compression_end(handle);
+out:
+ if (!ret)
+ handle->file_state = TRACECMD_FILE_KALLSYMS;
+ else
+ compression_reset(handle);
+ return ret;
}
static int read_ftrace_printk(struct tracecmd_output *handle)
@@ -840,6 +949,7 @@ static int read_ftrace_printk(struct tracecmd_output *handle)
if (!path)
return -1;
+ compression_start(handle);
ret = stat(path, &st);
if (ret < 0) {
/* not found */
@@ -861,11 +971,14 @@ static int read_ftrace_printk(struct tracecmd_output *handle)
}
out:
- handle->file_state = TRACECMD_FILE_PRINTK;
put_tracing_file(path);
+ if (compression_end(handle))
+ return -1;
+ handle->file_state = TRACECMD_FILE_PRINTK;
return 0;
fail:
put_tracing_file(path);
+ compression_reset(handle);
return -1;
}
@@ -911,16 +1024,40 @@ out_free:
static int select_file_version(struct tracecmd_output *handle, struct tracecmd_input *ihandle,
unsigned long file_version)
{
- if (ihandle)
+ if (ihandle) {
handle->file_version = tracecmd_get_file_version(ihandle);
- else if (file_version > 0)
+ } else if (file_version > 0) {
handle->file_version = file_version;
- else
- handle->file_version = FILE_VERSION;
+ } else {
+ handle->compress_proto = tracecmd_compress_proto_select();
+ if (handle->compress_proto)
+ handle->file_version = 7;
+ else
+ handle->file_version = 6;
+ }
return 0;
}
+static int write_compression_header(struct tracecmd_output *handle)
+{
+ const char *name = NULL;
+ const char *ver = NULL;
+ char *buf;
+ int ret;
+
+ ret = tracecmd_compress_proto_get_name(handle->compress_proto, &name, &ver);
+ if (ret < 0 || !name || !ver)
+ return -1;
+ ret = asprintf(&buf, "%s %s", name, ver);
+ if (ret < 0)
+ return -1;
+ ret = do_write_check(handle, buf, strlen(buf) + 1);
+
+ free(buf);
+ return ret;
+}
+
static struct tracecmd_output *
create_file_fd(int fd, struct tracecmd_input *ihandle,
const char *tracing_dir,
@@ -996,7 +1133,8 @@ create_file_fd(int fd, struct tracecmd_input *ihandle,
if (do_write_check(handle, &endian4, 4))
goto out_free;
handle->file_state = TRACECMD_FILE_INIT;
-
+ if (handle->file_version >= 7 && write_compression_header(handle))
+ goto out_free;
if (ihandle)
return handle;
@@ -1329,11 +1467,17 @@ int tracecmd_write_cmdlines(struct tracecmd_output *handle)
handle->file_state);
return ret;
}
+ compression_start(handle);
+
ret = save_tracing_file_data(handle, "saved_cmdlines");
- if (ret < 0)
+ if (ret < 0) {
+ compression_reset(handle);
return ret;
- handle->file_state = TRACECMD_FILE_CMD_LINES;
- return 0;
+ }
+ ret = compression_end(handle);
+ if (!ret)
+ handle->file_state = TRACECMD_FILE_CMD_LINES;
+ return ret;
}
struct tracecmd_output *tracecmd_create_file_latency(const char *output_file, int cpus,
--
2.30.2
^ permalink raw reply related [flat|nested] 14+ messages in thread* [PATCH 04/10] trace-cmd library: Read compressed trace file
2021-04-22 7:38 [POC PATCH 00/10] Add trace file compression Tzvetomir Stoyanov (VMware)
` (2 preceding siblings ...)
2021-04-22 7:38 ` [PATCH 03/10] trace-cmd library: Compress part of the trace file Tzvetomir Stoyanov (VMware)
@ 2021-04-22 7:38 ` Tzvetomir Stoyanov (VMware)
2021-04-22 7:38 ` [PATCH 05/10] trace-cmd library: Add new API to get compression of input handler Tzvetomir Stoyanov (VMware)
` (6 subsequent siblings)
10 siblings, 0 replies; 14+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-04-22 7:38 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
If a trace file version 7 is detected, read compresses sections of it.
Read the compression header, check if the comporession algorithm and
version, used to compress the file, are supported and use it to
uncompress these secions from the file:
- ftrace events format
- format of recorded events
- information of the mapping of function addresses to the function names
- trace_printk() format strings
- information of the mapping a PID to a process name
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
lib/trace-cmd/trace-input.c | 293 +++++++++++++++++++++++++++---------
1 file changed, 221 insertions(+), 72 deletions(-)
diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c
index b2a03ab8..b96b0192 100644
--- a/lib/trace-cmd/trace-input.c
+++ b/lib/trace-cmd/trace-input.c
@@ -135,6 +135,11 @@ struct tracecmd_input {
long long ts_offset;
struct tsc2nsec tsc_calc;
+ struct tracecmd_compress_proto *z_proto;
+ char *z_buffer;
+ unsigned long z_buffer_size;
+ unsigned long z_pointer;
+
struct host_trace_info host;
double ts2secs;
char * cpustats;
@@ -254,12 +259,52 @@ static ssize_t do_read(struct tracecmd_input *handle, void *data, size_t size)
return tot;
}
+static ssize_t do_read_compressed(struct tracecmd_input *handle, void *data, size_t size)
+{
+ int ret;
+ int s;
+
+ if (handle->z_buffer) {
+ if (handle->z_pointer + size > handle->z_buffer_size)
+ s = handle->z_buffer_size - handle->z_pointer;
+ else
+ s = size;
+ memcpy(data, handle->z_buffer + handle->z_pointer, s);
+ handle->z_pointer += s;
+ ret = s;
+ } else {
+ ret = do_read(handle, data, size);
+ }
+
+ return ret;
+}
+
+static int do_lseek(struct tracecmd_input *handle, int offset, int whence)
+{
+ int ret;
+
+ if (handle->z_buffer) {
+ if (whence != SEEK_CUR)
+ return -1;
+ if (offset < 0 && handle->z_pointer < (-offset))
+ return -1;
+ if (offset > 0 && handle->z_pointer + offset > handle->z_buffer_size)
+ return -1;
+ handle->z_pointer += offset;
+ ret = 0;
+ } else {
+ ret = lseek(handle->fd, offset, whence);
+ }
+
+ return ret;
+}
+
static ssize_t
do_read_check(struct tracecmd_input *handle, void *data, size_t size)
{
ssize_t ret;
- ret = do_read(handle, data, size);
+ ret = do_read_compressed(handle, data, size);
if (ret < 0)
return ret;
if (ret != size)
@@ -277,7 +322,7 @@ static char *read_string(struct tracecmd_input *handle)
ssize_t r;
for (;;) {
- r = do_read(handle, buf, BUFSIZ);
+ r = do_read_compressed(handle, buf, BUFSIZ);
if (r < 0)
goto fail;
if (!r)
@@ -306,7 +351,7 @@ static char *read_string(struct tracecmd_input *handle)
}
/* move the file descriptor to the end of the string */
- r = lseek(handle->fd, -(r - (i+1)), SEEK_CUR);
+ r = do_lseek(handle, -(r - (i+1)), SEEK_CUR);
if (r < 0)
goto fail;
@@ -358,6 +403,54 @@ static int read8(struct tracecmd_input *handle, unsigned long long *size)
return 0;
}
+static void uncompress_reset(struct tracecmd_input *handle)
+{
+ free(handle->z_buffer);
+ handle->z_buffer = NULL;
+ handle->z_buffer_size = 0;
+ handle->z_pointer = 0;
+}
+
+static int uncompress_block(struct tracecmd_input *handle)
+{
+ unsigned int s_compressed;
+ unsigned int s_uncompressed;
+ char *bytes = NULL;
+ int ret;
+
+ if (handle->file_version < 7 || !handle->z_proto)
+ return 0;
+ uncompress_reset(handle);
+
+ if (read4(handle, &s_compressed) < 0)
+ return -1;
+ if (read4(handle, &s_uncompressed) < 0)
+ return -1;
+
+ handle->z_buffer = malloc(s_uncompressed);
+ if (!handle->z_buffer)
+ return -1;
+ bytes = malloc(s_compressed);
+ if (!bytes)
+ goto error;
+
+ if (do_read(handle, bytes, s_compressed) != s_compressed)
+ goto error;
+ ret = tracecmd_uncompress_data(handle->z_proto, bytes, s_compressed,
+ handle->z_buffer, &s_uncompressed);
+ if (ret)
+ goto error;
+ free(bytes);
+ handle->z_buffer_size = s_uncompressed;
+ handle->z_pointer = 0;
+ return 0;
+error:
+ uncompress_reset(handle);
+ free(bytes);
+ return -1;
+
+}
+
static int read_header_files(struct tracecmd_input *handle)
{
struct tep_handle *pevent = handle->pevent;
@@ -601,34 +694,40 @@ static int read_ftrace_files(struct tracecmd_input *handle, const char *regex)
}
}
- if (read4(handle, &count) < 0)
+ if (uncompress_block(handle))
return -1;
+ ret = read4(handle, &count);
+ if (ret < 0)
+ goto out;
+
for (i = 0; i < count; i++) {
- if (read8(handle, &size) < 0)
- return -1;
+ ret = read8(handle, &size);
+ if (ret < 0)
+ goto out;
ret = read_ftrace_file(handle, size, print_all, ereg);
if (ret < 0)
- return -1;
+ goto out;
}
handle->event_files_start =
lseek64(handle->fd, 0, SEEK_CUR);
+ handle->file_state = TRACECMD_FILE_FTRACE_EVENTS;
+ ret = 0;
+out:
if (sreg) {
regfree(sreg);
regfree(ereg);
}
-
- handle->file_state = TRACECMD_FILE_FTRACE_EVENTS;
-
- return 0;
+ uncompress_reset(handle);
+ return ret;
}
static int read_event_files(struct tracecmd_input *handle, const char *regex)
{
unsigned long long size;
- char *system;
+ char *system = NULL;
regex_t spreg;
regex_t epreg;
regex_t *sreg = NULL;
@@ -653,13 +752,19 @@ static int read_event_files(struct tracecmd_input *handle, const char *regex)
return -1;
}
- if (read4(handle, &systems) < 0)
+ if (uncompress_block(handle))
return -1;
+ ret = read4(handle, &systems);
+ if (ret < 0)
+ goto out;
+
for (i = 0; i < systems; i++) {
system = read_string(handle);
- if (!system)
- return -1;
+ if (!system) {
+ ret = -1;
+ goto out;
+ }
sys_printed = 0;
print_all = 0;
@@ -686,103 +791,117 @@ static int read_event_files(struct tracecmd_input *handle, const char *regex)
}
}
- if (read4(handle, &count) < 0)
- goto failed;
+ ret = read4(handle, &count);
+ if (ret < 0)
+ goto out;
for (x=0; x < count; x++) {
- if (read8(handle, &size) < 0)
- goto failed;
+ ret = read8(handle, &size);
+ if (ret < 0)
+ goto out;
ret = read_event_file(handle, system, size,
print_all, &sys_printed,
reg);
if (ret < 0)
- goto failed;
+ goto out;
}
free(system);
- }
-
- if (sreg) {
- regfree(sreg);
- regfree(ereg);
+ system = NULL;
}
handle->file_state = TRACECMD_FILE_ALL_EVENTS;
-
- return 0;
-
- failed:
+ ret = 0;
+ out:
+ uncompress_reset(handle);
if (sreg) {
regfree(sreg);
regfree(ereg);
}
free(system);
- return -1;
+ return ret;
}
static int read_proc_kallsyms(struct tracecmd_input *handle)
{
- struct tep_handle *pevent = handle->pevent;
+ struct tep_handle *tep = handle->pevent;
unsigned int size;
- char *buf;
+ char *buf = NULL;
+ int ret;
if (handle->file_state >= TRACECMD_FILE_KALLSYMS)
return 0;
- if (read4(handle, &size) < 0)
+ if (uncompress_block(handle))
return -1;
- if (!size)
- return 0; /* OK? */
- buf = malloc(size+1);
- if (!buf)
- return -1;
- if (do_read_check(handle, buf, size)){
- free(buf);
- return -1;
+ ret = read4(handle, &size);
+ if (ret < 0)
+ goto out;
+ if (!size) {
+ handle->file_state = TRACECMD_FILE_KALLSYMS;
+ goto out; /* OK? */
}
- buf[size] = 0;
-
- tep_parse_kallsyms(pevent, buf);
- free(buf);
+ buf = malloc(size+1);
+ if (!buf) {
+ ret = -1;
+ goto out;
+ }
+ ret = do_read_check(handle, buf, size);
+ if (ret < 0)
+ goto out;
+ buf[size] = 0;
+ tep_parse_kallsyms(tep, buf);
handle->file_state = TRACECMD_FILE_KALLSYMS;
-
- return 0;
+ ret = 0;
+out:
+ free(buf);
+ uncompress_reset(handle);
+ return ret;
}
static int read_ftrace_printk(struct tracecmd_input *handle)
{
unsigned int size;
- char *buf;
+ char *buf = NULL;
+ int ret;
if (handle->file_state >= TRACECMD_FILE_PRINTK)
return 0;
- if (read4(handle, &size) < 0)
+ if (uncompress_block(handle))
return -1;
- if (!size)
- return 0; /* OK? */
+
+ ret = read4(handle, &size);
+ if (ret < 0)
+ goto out;
+ if (!size) {
+ handle->file_state = TRACECMD_FILE_PRINTK;
+ goto out; /* OK? */
+ }
buf = malloc(size + 1);
- if (!buf)
- return -1;
- if (do_read_check(handle, buf, size)) {
- free(buf);
- return -1;
+ if (!buf) {
+ ret = -1;
+ goto out;
}
+ ret = do_read_check(handle, buf, size);
+ if (ret < 0)
+ goto out;
buf[size] = 0;
tep_parse_printk_formats(handle->pevent, buf);
-
- free(buf);
-
handle->file_state = TRACECMD_FILE_PRINTK;
+ ret = 0;
- return 0;
+out:
+ free(buf);
+ uncompress_reset(handle);
+ return ret;
}
static int read_and_parse_cmdlines(struct tracecmd_input *handle);
@@ -2962,20 +3081,30 @@ static int read_and_parse_cmdlines(struct tracecmd_input *handle)
{
struct tep_handle *pevent = handle->pevent;
unsigned long long size;
- char *cmdlines;
+ char *cmdlines = NULL;
+ int ret;
if (handle->file_state >= TRACECMD_FILE_CMD_LINES)
return 0;
- if (read_data_and_size(handle, &cmdlines, &size) < 0)
+ if (uncompress_block(handle))
return -1;
+
+ ret = read_data_and_size(handle, &cmdlines, &size);
+ if (ret < 0)
+ goto out;
+ if (!size) {
+ handle->file_state = TRACECMD_FILE_CMD_LINES;
+ goto out;
+ }
cmdlines[size] = 0;
tep_parse_saved_cmdlines(pevent, cmdlines);
- free(cmdlines);
-
handle->file_state = TRACECMD_FILE_CMD_LINES;
-
- return 0;
+ ret = 0;
+out:
+ free(cmdlines);
+ uncompress_reset(handle);
+ return ret;
}
static void extract_trace_clock(struct tracecmd_input *handle, char *line)
@@ -3256,7 +3385,8 @@ struct tracecmd_input *tracecmd_alloc_fd(int fd, int flags)
struct tracecmd_input *handle;
char test[] = TRACECMD_MAGIC;
unsigned int page_size;
- char *version;
+ char *str = NULL;
+ char *zver;
char buf[BUFSIZ];
unsigned long ver;
@@ -3279,11 +3409,11 @@ struct tracecmd_input *tracecmd_alloc_fd(int fd, int flags)
if (memcmp(buf, "tracing", 7) != 0)
goto failed_read;
- version = read_string(handle);
- if (!version)
+ str = read_string(handle);
+ if (!str)
goto failed_read;
- pr_stat("version = %s\n", version);
- ver = strtol(version, NULL, 10);
+ pr_stat("version = %s\n", str);
+ ver = strtol(str, NULL, 10);
if (!ver && errno)
goto failed_read;
if (!tracecmd_is_version_supported(ver)) {
@@ -3291,7 +3421,8 @@ struct tracecmd_input *tracecmd_alloc_fd(int fd, int flags)
goto failed_read;
}
handle->file_version = ver;
- free(version);
+ free(str);
+ str = NULL;
if (do_read_check(handle, buf, 1))
goto failed_read;
@@ -3325,11 +3456,29 @@ struct tracecmd_input *tracecmd_alloc_fd(int fd, int flags)
handle->header_files_start =
lseek64(handle->fd, handle->header_files_start, SEEK_SET);
+ if (handle->file_version >= 7) {
+ str = read_string(handle);
+ if (!str)
+ goto failed_read;
+ zver = strchr(str, ' ');
+ if (!zver)
+ goto failed_read;
+ *zver = '\0';
+ handle->z_proto = tracecmd_compress_proto_get(str, zver + 1);
+ if (!handle->z_proto) {
+ tracecmd_warning("Unsupported file compression %s %s", str, zver + 1);
+ goto failed_read;
+ }
+ free(str);
+ str = NULL;
+ }
+
handle->file_state = TRACECMD_FILE_INIT;
return handle;
failed_read:
+ free(str);
free(handle);
return NULL;
--
2.30.2
^ permalink raw reply related [flat|nested] 14+ messages in thread* [PATCH 05/10] trace-cmd library: Add new API to get compression of input handler
2021-04-22 7:38 [POC PATCH 00/10] Add trace file compression Tzvetomir Stoyanov (VMware)
` (3 preceding siblings ...)
2021-04-22 7:38 ` [PATCH 04/10] trace-cmd library: Read compressed " Tzvetomir Stoyanov (VMware)
@ 2021-04-22 7:38 ` Tzvetomir Stoyanov (VMware)
2021-04-22 7:38 ` [PATCH 06/10] trace-cmd library: Inherit compression algorithm from input file Tzvetomir Stoyanov (VMware)
` (5 subsequent siblings)
10 siblings, 0 replies; 14+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-04-22 7:38 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
The new API returns a handler to compression algorithm, used to compress
the trace file asociated with given input file handler:
tracecmd_get_file_comperssion()
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
lib/trace-cmd/include/private/trace-cmd-private.h | 2 ++
lib/trace-cmd/trace-input.c | 11 +++++++++++
2 files changed, 13 insertions(+)
diff --git a/lib/trace-cmd/include/private/trace-cmd-private.h b/lib/trace-cmd/include/private/trace-cmd-private.h
index c562fc5d..4d2116ab 100644
--- a/lib/trace-cmd/include/private/trace-cmd-private.h
+++ b/lib/trace-cmd/include/private/trace-cmd-private.h
@@ -236,6 +236,8 @@ unsigned long long
tracecmd_get_cursor(struct tracecmd_input *handle, int cpu);
unsigned long tracecmd_get_file_version(struct tracecmd_input *handle);
+struct tracecmd_compress_proto *
+tracecmd_get_file_comperssion(struct tracecmd_input *handle);
int tracecmd_ftrace_overrides(struct tracecmd_input *handle, struct tracecmd_ftrace *finfo);
bool tracecmd_get_use_trace_clock(struct tracecmd_input *handle);
diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c
index b96b0192..ad5af5fa 100644
--- a/lib/trace-cmd/trace-input.c
+++ b/lib/trace-cmd/trace-input.c
@@ -4192,6 +4192,17 @@ unsigned long tracecmd_get_file_version(struct tracecmd_input *handle)
return handle->file_version;
}
+/**
+ * tracecmd_get_file_comperssion - return a handler to compression protocol,
+ * used to compress the trace file
+ * @handle: input handle for the trace.dat file
+ */
+struct tracecmd_compress_proto *
+tracecmd_get_file_comperssion(struct tracecmd_input *handle)
+{
+ return handle->z_proto;
+}
+
/**
* tracecmd_get_use_trace_clock - return use_trace_clock
* @handle: input handle for the trace.dat file
--
2.30.2
^ permalink raw reply related [flat|nested] 14+ messages in thread* [PATCH 06/10] trace-cmd library: Inherit compression algorithm from input file
2021-04-22 7:38 [POC PATCH 00/10] Add trace file compression Tzvetomir Stoyanov (VMware)
` (4 preceding siblings ...)
2021-04-22 7:38 ` [PATCH 05/10] trace-cmd library: Add new API to get compression of input handler Tzvetomir Stoyanov (VMware)
@ 2021-04-22 7:38 ` Tzvetomir Stoyanov (VMware)
2021-04-22 7:38 ` [PATCH 07/10] trace-cmd library: Extend the create file APIs to support different compression Tzvetomir Stoyanov (VMware)
` (4 subsequent siblings)
10 siblings, 0 replies; 14+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-04-22 7:38 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
When a new trace file output handler is allocated, based on given trace
file input handler - use the same compression algorithm.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
lib/trace-cmd/trace-output.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c
index 9d603cfe..75cf143d 100644
--- a/lib/trace-cmd/trace-output.c
+++ b/lib/trace-cmd/trace-output.c
@@ -1026,6 +1026,7 @@ static int select_file_version(struct tracecmd_output *handle, struct tracecmd_i
{
if (ihandle) {
handle->file_version = tracecmd_get_file_version(ihandle);
+ handle->compress_proto = tracecmd_get_file_comperssion(ihandle);
} else if (file_version > 0) {
handle->file_version = file_version;
} else {
@@ -1769,6 +1770,7 @@ struct tracecmd_output *tracecmd_get_output_handle_fd(int fd)
tep_ref(handle->pevent);
handle->page_size = tracecmd_page_size(ihandle);
handle->file_version = tracecmd_get_file_version(ihandle);
+ handle->compress_proto = tracecmd_get_file_comperssion(ihandle);
list_head_init(&handle->options);
tracecmd_close(ihandle);
--
2.30.2
^ permalink raw reply related [flat|nested] 14+ messages in thread* [PATCH 07/10] trace-cmd library: Extend the create file APIs to support different compression
2021-04-22 7:38 [POC PATCH 00/10] Add trace file compression Tzvetomir Stoyanov (VMware)
` (5 preceding siblings ...)
2021-04-22 7:38 ` [PATCH 06/10] trace-cmd library: Inherit compression algorithm from input file Tzvetomir Stoyanov (VMware)
@ 2021-04-22 7:38 ` Tzvetomir Stoyanov (VMware)
2021-04-22 7:39 ` [PATCH 08/10] trace-cmd record: Add new parameter --compression Tzvetomir Stoyanov (VMware)
` (3 subsequent siblings)
10 siblings, 0 replies; 14+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-04-22 7:38 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
Added additional parameter for file compression to all trace-cmd library
APIs for creating a new trace file. The caller could specify what
compression algorithm should be used when creating a new trace file:
tracecmd_create_file_latency
tracecmd_create_init_file_glob
tracecmd_create_init_fd_glob
tracecmd_create_init_fd_msg
tracecmd_create_init_file
tracecmd_create_init_file_override
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
.../include/private/trace-cmd-private.h | 15 ++--
lib/trace-cmd/trace-output.c | 80 ++++++++++++++-----
tracecmd/trace-record.c | 10 +--
tracecmd/trace-restore.c | 4 +-
4 files changed, 74 insertions(+), 35 deletions(-)
diff --git a/lib/trace-cmd/include/private/trace-cmd-private.h b/lib/trace-cmd/include/private/trace-cmd-private.h
index 4d2116ab..244e6b4f 100644
--- a/lib/trace-cmd/include/private/trace-cmd-private.h
+++ b/lib/trace-cmd/include/private/trace-cmd-private.h
@@ -272,24 +272,27 @@ struct tracecmd_option;
struct tracecmd_msg_handle;
struct tracecmd_output *tracecmd_create_file_latency(const char *output_file, int cpus,
- unsigned long file_version);
+ unsigned long file_version,
+ const char *compression);
struct tracecmd_output *
tracecmd_create_init_file_glob(const char *output_file, struct tracecmd_event_list *list,
- unsigned long file_version);
+ unsigned long file_version, const char *compression);
struct tracecmd_output *tracecmd_create_init_fd(int fd);
struct tracecmd_output *
tracecmd_create_init_fd_glob(int fd, struct tracecmd_event_list *list,
- unsigned long file_version);
+ unsigned long file_version, const char *compression);
struct tracecmd_output *
tracecmd_create_init_fd_msg(struct tracecmd_msg_handle *msg_handle,
struct tracecmd_event_list *list,
- unsigned long file_version);
+ unsigned long file_version, const char *compression);
struct tracecmd_output *tracecmd_create_init_file(const char *output_file,
- unsigned long file_version);
+ unsigned long file_version,
+ const char *compression);
struct tracecmd_output *tracecmd_create_init_file_override(const char *output_file,
const char *tracing_dir,
const char *kallsyms,
- unsigned long file_version);
+ unsigned long file_version,
+ const char *compression);
struct tracecmd_option *tracecmd_add_option(struct tracecmd_output *handle,
unsigned short id, int size,
const void *data);
diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c
index 75cf143d..5fc7e778 100644
--- a/lib/trace-cmd/trace-output.c
+++ b/lib/trace-cmd/trace-output.c
@@ -1021,15 +1021,40 @@ out_free:
return ret;
}
-static int select_file_version(struct tracecmd_output *handle, struct tracecmd_input *ihandle,
- unsigned long file_version)
+static int select_file_version(struct tracecmd_output *handle,
+ struct tracecmd_input *ihandle,
+ unsigned long file_version, const char *compression)
{
if (ihandle) {
handle->file_version = tracecmd_get_file_version(ihandle);
handle->compress_proto = tracecmd_get_file_comperssion(ihandle);
- } else if (file_version > 0) {
+ return 0;
+ }
+
+ if (file_version > 0)
handle->file_version = file_version;
- } else {
+ if (compression) {
+ if (handle->file_version && handle->file_version < 7) {
+ tracecmd_warning("Compression is not supported with trace file version %d",
+ handle->file_version);
+ return -1;
+ }
+ handle->compress_proto = tracecmd_compress_proto_get(compression, NULL);
+ if (!handle->compress_proto) {
+ tracecmd_warning("Compression algorithm %s is not supported", compression);
+ return -1;
+ }
+ if (!handle->file_version)
+ handle->file_version = 7;
+ } else if (handle->file_version >= 7) {
+ handle->compress_proto = tracecmd_compress_proto_select();
+ if (!handle->compress_proto) {
+ tracecmd_warning("Compression is not supported");
+ return -1;
+ }
+ }
+
+ if (!handle->file_version) {
handle->compress_proto = tracecmd_compress_proto_select();
if (handle->compress_proto)
handle->file_version = 7;
@@ -1065,7 +1090,8 @@ create_file_fd(int fd, struct tracecmd_input *ihandle,
const char *kallsyms,
struct tracecmd_event_list *list,
struct tracecmd_msg_handle *msg_handle,
- unsigned long file_version)
+ unsigned long file_version,
+ const char *compression)
{
struct tracecmd_output *handle;
struct tep_handle *pevent;
@@ -1086,7 +1112,7 @@ create_file_fd(int fd, struct tracecmd_input *ihandle,
handle->msg_handle = msg_handle;
- if (select_file_version(handle, ihandle, file_version))
+ if (select_file_version(handle, ihandle, file_version, compression))
goto out_free;
list_head_init(&handle->options);
@@ -1166,7 +1192,7 @@ static struct tracecmd_output *create_file(const char *output_file,
const char *tracing_dir,
const char *kallsyms,
struct tracecmd_event_list *list,
- unsigned long file_version)
+ unsigned long file_version, const char *compression)
{
struct tracecmd_output *handle;
int fd;
@@ -1175,7 +1201,7 @@ static struct tracecmd_output *create_file(const char *output_file,
if (fd < 0)
return NULL;
- handle = create_file_fd(fd, ihandle, tracing_dir, kallsyms, list, NULL, file_version);
+ handle = create_file_fd(fd, ihandle, tracing_dir, kallsyms, list, NULL, file_version, compression);
if (!handle) {
close(fd);
unlink(output_file);
@@ -1482,14 +1508,15 @@ int tracecmd_write_cmdlines(struct tracecmd_output *handle)
}
struct tracecmd_output *tracecmd_create_file_latency(const char *output_file, int cpus,
- unsigned long file_version)
+ unsigned long file_version,
+ const char *compression)
{
struct tracecmd_output *handle;
char *path;
int ret;
handle = create_file(output_file, NULL, NULL, NULL,
- &all_event_list, file_version);
+ &all_event_list, file_version, compression);
if (!handle)
return NULL;
@@ -1785,46 +1812,48 @@ struct tracecmd_output *tracecmd_get_output_handle_fd(int fd)
struct tracecmd_output *tracecmd_create_init_fd(int fd)
{
- return create_file_fd(fd, NULL, NULL, NULL, &all_event_list, NULL, 0);
+ return create_file_fd(fd, NULL, NULL, NULL, &all_event_list, NULL, 0, NULL);
}
struct tracecmd_output *
tracecmd_create_init_fd_msg(struct tracecmd_msg_handle *msg_handle,
struct tracecmd_event_list *list,
- unsigned long file_version)
+ unsigned long file_version, const char *compression)
{
return create_file_fd(msg_handle->fd, NULL, NULL, NULL,
- list, msg_handle, file_version);
+ list, msg_handle, file_version, compression);
}
struct tracecmd_output *
tracecmd_create_init_fd_glob(int fd, struct tracecmd_event_list *list,
- unsigned long file_version)
+ unsigned long file_version, const char *compression)
{
- return create_file_fd(fd, NULL, NULL, NULL, list, NULL, file_version);
+ return create_file_fd(fd, NULL, NULL, NULL, list, NULL, file_version, compression);
}
struct tracecmd_output *
tracecmd_create_init_file_glob(const char *output_file, struct tracecmd_event_list *list,
- unsigned long file_version)
+ unsigned long file_version, const char *compression)
{
- return create_file(output_file, NULL, NULL, NULL, list, file_version);
+ return create_file(output_file, NULL, NULL, NULL, list, file_version, compression);
}
struct tracecmd_output *tracecmd_create_init_file(const char *output_file,
- unsigned long file_version)
+ unsigned long file_version,
+ const char *compression)
{
return create_file(output_file, NULL, NULL, NULL,
- &all_event_list, file_version);
+ &all_event_list, file_version, compression);
}
struct tracecmd_output *tracecmd_create_init_file_override(const char *output_file,
const char *tracing_dir,
const char *kallsyms,
- unsigned long file_version)
+ unsigned long file_version,
+ const char *compression)
{
return create_file(output_file, NULL, tracing_dir, kallsyms,
- &all_event_list, file_version);
+ &all_event_list, file_version, compression);
}
/**
@@ -1839,10 +1868,17 @@ struct tracecmd_output *tracecmd_create_init_file_override(const char *output_fi
struct tracecmd_output *tracecmd_copy(struct tracecmd_input *ihandle,
const char *file)
{
+ struct tracecmd_compress_proto *compr;
struct tracecmd_output *handle;
+ const char *compr_name = NULL;
+
+
+ compr = tracecmd_get_file_comperssion(ihandle);
+ if (compr)
+ tracecmd_compress_proto_get_name(compr, &compr_name, NULL);
handle = create_file(file, ihandle, NULL, NULL, &all_event_list,
- tracecmd_get_file_version(ihandle));
+ tracecmd_get_file_version(ihandle), compr_name);
if (!handle)
return NULL;
diff --git a/tracecmd/trace-record.c b/tracecmd/trace-record.c
index f95db0e4..b8126eae 100644
--- a/tracecmd/trace-record.c
+++ b/tracecmd/trace-record.c
@@ -3647,7 +3647,7 @@ setup_connection(struct buffer_instance *instance, struct common_record_context
/* Now create the handle through this socket */
if (msg_handle->version == V3_PROTOCOL) {
network_handle = tracecmd_create_init_fd_msg(msg_handle, listed_events,
- ctx->file_version);
+ ctx->file_version, NULL);
if (!network_handle)
goto error;
tracecmd_set_quiet(network_handle, quiet);
@@ -3666,7 +3666,7 @@ setup_connection(struct buffer_instance *instance, struct common_record_context
goto error;
} else {
network_handle = tracecmd_create_init_fd_glob(msg_handle->fd, listed_events,
- ctx->file_version);
+ ctx->file_version, NULL);
if (!network_handle)
goto error;
tracecmd_set_quiet(network_handle, quiet);
@@ -3854,7 +3854,7 @@ static void setup_agent(struct buffer_instance *instance,
struct tracecmd_output *network_handle;
network_handle = tracecmd_create_init_fd_msg(instance->msg_handle, listed_events,
- ctx->file_version);
+ ctx->file_version, NULL);
add_options(network_handle, ctx);
tracecmd_write_cmdlines(network_handle);
tracecmd_write_cpus(network_handle, instance->cpu_count);
@@ -4247,7 +4247,7 @@ static void record_data(struct common_record_context *ctx)
if (latency) {
handle = tracecmd_create_file_latency(ctx->output, local_cpu_count,
- ctx->file_version);
+ ctx->file_version, NULL);
tracecmd_set_quiet(handle, quiet);
} else {
if (!local_cpu_count)
@@ -4279,7 +4279,7 @@ static void record_data(struct common_record_context *ctx)
}
handle = tracecmd_create_init_file_glob(ctx->output, listed_events,
- ctx->file_version);
+ ctx->file_version, NULL);
if (!handle)
die("Error creating output file");
tracecmd_set_quiet(handle, quiet);
diff --git a/tracecmd/trace-restore.c b/tracecmd/trace-restore.c
index f2b15434..4b6a0c12 100644
--- a/tracecmd/trace-restore.c
+++ b/tracecmd/trace-restore.c
@@ -91,7 +91,7 @@ void trace_restore (int argc, char **argv)
}
handle = tracecmd_create_init_file_override(output, tracing_dir,
- kallsyms, 0);
+ kallsyms, 0, NULL);
if (!handle)
die("Unabled to create output file %s", output);
if (tracecmd_write_cmdlines(handle) < 0)
@@ -128,7 +128,7 @@ void trace_restore (int argc, char **argv)
handle = tracecmd_copy(ihandle, output);
tracecmd_close(ihandle);
} else
- handle = tracecmd_create_init_file(output, 0);
+ handle = tracecmd_create_init_file(output, 0, NULL);
if (!handle)
die("error writing to %s", output);
--
2.30.2
^ permalink raw reply related [flat|nested] 14+ messages in thread* [PATCH 08/10] trace-cmd record: Add new parameter --compression
2021-04-22 7:38 [POC PATCH 00/10] Add trace file compression Tzvetomir Stoyanov (VMware)
` (6 preceding siblings ...)
2021-04-22 7:38 ` [PATCH 07/10] trace-cmd library: Extend the create file APIs to support different compression Tzvetomir Stoyanov (VMware)
@ 2021-04-22 7:39 ` Tzvetomir Stoyanov (VMware)
2021-04-22 7:39 ` [PATCH 09/10] trace-cmd dump: Add support for trace files version 7 Tzvetomir Stoyanov (VMware)
` (2 subsequent siblings)
10 siblings, 0 replies; 14+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-04-22 7:39 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
Added a new optional parameter to "trace-cmd record", can be used to
select the desired compression algorithm for the trace output file.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
tracecmd/trace-record.c | 18 +++++++++++++-----
tracecmd/trace-usage.c | 1 +
2 files changed, 14 insertions(+), 5 deletions(-)
diff --git a/tracecmd/trace-record.c b/tracecmd/trace-record.c
index b8126eae..9b01ee05 100644
--- a/tracecmd/trace-record.c
+++ b/tracecmd/trace-record.c
@@ -199,6 +199,7 @@ struct common_record_context {
char *date2ts;
char *user;
const char *clock;
+ const char *compression;
unsigned long file_version;
struct tsc_nsec tsc2nsec;
int data_flags;
@@ -3647,7 +3648,7 @@ setup_connection(struct buffer_instance *instance, struct common_record_context
/* Now create the handle through this socket */
if (msg_handle->version == V3_PROTOCOL) {
network_handle = tracecmd_create_init_fd_msg(msg_handle, listed_events,
- ctx->file_version, NULL);
+ ctx->file_version, ctx->compression);
if (!network_handle)
goto error;
tracecmd_set_quiet(network_handle, quiet);
@@ -3666,7 +3667,7 @@ setup_connection(struct buffer_instance *instance, struct common_record_context
goto error;
} else {
network_handle = tracecmd_create_init_fd_glob(msg_handle->fd, listed_events,
- ctx->file_version, NULL);
+ ctx->file_version, ctx->compression);
if (!network_handle)
goto error;
tracecmd_set_quiet(network_handle, quiet);
@@ -3854,7 +3855,7 @@ static void setup_agent(struct buffer_instance *instance,
struct tracecmd_output *network_handle;
network_handle = tracecmd_create_init_fd_msg(instance->msg_handle, listed_events,
- ctx->file_version, NULL);
+ ctx->file_version, ctx->compression);
add_options(network_handle, ctx);
tracecmd_write_cmdlines(network_handle);
tracecmd_write_cpus(network_handle, instance->cpu_count);
@@ -4247,7 +4248,7 @@ static void record_data(struct common_record_context *ctx)
if (latency) {
handle = tracecmd_create_file_latency(ctx->output, local_cpu_count,
- ctx->file_version, NULL);
+ ctx->file_version, ctx->compression);
tracecmd_set_quiet(handle, quiet);
} else {
if (!local_cpu_count)
@@ -4279,7 +4280,7 @@ static void record_data(struct common_record_context *ctx)
}
handle = tracecmd_create_init_file_glob(ctx->output, listed_events,
- ctx->file_version, NULL);
+ ctx->file_version, ctx->compression);
if (!handle)
die("Error creating output file");
tracecmd_set_quiet(handle, quiet);
@@ -5505,6 +5506,7 @@ void init_top_instance(void)
}
enum {
+ OPT_comporession = 238,
OPT_file_version = 239,
OPT_tsc2nsec = 240,
OPT_fork = 241,
@@ -5941,6 +5943,7 @@ static void parse_record_options(int argc,
{"fork", no_argument, NULL, OPT_fork},
{"tsc2nsec", no_argument, NULL, OPT_tsc2nsec},
{"file-version", required_argument, NULL, OPT_file_version},
+ {"compression", required_argument, NULL, OPT_comporession},
{NULL, 0, NULL, 0}
};
@@ -6367,6 +6370,11 @@ static void parse_record_options(int argc,
if (!tracecmd_is_version_supported(ctx->file_version))
die("File version %d is not supported", ctx->file_version);
break;
+ case OPT_comporession:
+ ctx->compression = strdup(optarg);
+ if (!tracecmd_compress_proto_get(ctx->compression, NULL))
+ die("Compression algorithm %s is not supported", ctx->compression);
+ break;
case OPT_quiet:
case 'q':
quiet = true;
diff --git a/tracecmd/trace-usage.c b/tracecmd/trace-usage.c
index e5b54114..f2383834 100644
--- a/tracecmd/trace-usage.c
+++ b/tracecmd/trace-usage.c
@@ -69,6 +69,7 @@ static struct usage_help usage_help[] = {
" If 0 is specified, no loop is performed - timestamps offset is calculated only twice,"
" at the beginnig and at the end of the trace\n"
" --file-version select the desired version of the trace output file\n"
+ " --compression select the desired compression algorithm for the trace output file\n"
},
{
"set",
--
2.30.2
^ permalink raw reply related [flat|nested] 14+ messages in thread* [PATCH 09/10] trace-cmd dump: Add support for trace files version 7
2021-04-22 7:38 [POC PATCH 00/10] Add trace file compression Tzvetomir Stoyanov (VMware)
` (7 preceding siblings ...)
2021-04-22 7:39 ` [PATCH 08/10] trace-cmd record: Add new parameter --compression Tzvetomir Stoyanov (VMware)
@ 2021-04-22 7:39 ` Tzvetomir Stoyanov (VMware)
2021-04-22 7:39 ` [PATCH 10/10] trace-cmd library: Add support for zlib compression library Tzvetomir Stoyanov (VMware)
2021-04-29 1:35 ` [POC PATCH 00/10] Add trace file compression Steven Rostedt
10 siblings, 0 replies; 14+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-04-22 7:39 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
Added support to read and dump information of trace files version 7.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
tracecmd/trace-dump.c | 164 ++++++++++++++++++++++++++++++++++++------
1 file changed, 144 insertions(+), 20 deletions(-)
diff --git a/tracecmd/trace-dump.c b/tracecmd/trace-dump.c
index aa48eaad..2c65a1d8 100644
--- a/tracecmd/trace-dump.c
+++ b/tracecmd/trace-dump.c
@@ -26,6 +26,12 @@
static struct tep_handle *tep;
static unsigned int trace_cpus;
static int has_clock;
+static unsigned long file_version;
+
+static struct tracecmd_compress_proto *z_handle;
+static char *z_buffer;
+static unsigned long z_buffer_size;
+static unsigned long z_pointer;
enum dump_items {
SUMMARY = (1 << 0),
@@ -52,46 +58,72 @@ enum dump_items verbosity;
tracecmd_plog(fmt, ##__VA_ARGS__); \
} while (0)
-static int read_file_string(int fd, char *dst, int len)
+static int read_fd(int fd, char *dst, int len)
{
size_t size = 0;
int r;
do {
- r = read(fd, dst+size, 1);
+ r = read(fd, dst+size, len);
if (r > 0) {
- size++;
- len--;
+ size += r;
+ len -= r;
} else
break;
- if (!dst[size - 1])
- break;
- } while (r > 0 && len);
+ } while (r > 0);
- if (!size || dst[size - 1])
+ if (len)
return -1;
- return 0;
+ return size;
}
-static int read_file_bytes(int fd, char *dst, int len)
+
+static int read_compressed(int fd, char *dst, int len)
+{
+ int ret;
+
+ if (z_buffer) {
+ if (z_pointer + len > z_buffer_size)
+ return -1;
+ memcpy(dst, z_buffer + z_pointer, len);
+ z_pointer += len;
+ ret = len;
+ } else {
+ ret = read_fd(fd, dst, len);
+ }
+
+ return ret;
+}
+
+static int read_file_string(int fd, char *dst, int len)
{
size_t size = 0;
int r;
do {
- r = read(fd, dst+size, len);
+ r = read_compressed(fd, dst+size, 1);
if (r > 0) {
- size += r;
- len -= r;
+ size++;
+ len--;
} else
break;
- } while (r > 0);
+ if (!dst[size - 1])
+ break;
+ } while (r > 0 && len);
- if (len)
+ if (!size || dst[size - 1])
return -1;
return 0;
}
+static int read_file_bytes(int fd, char *dst, int len)
+{
+ int ret;
+
+ ret = read_compressed(fd, dst, len);
+ return ret < 0 ? ret : 0;
+}
+
static void read_dump_string(int fd, int size, enum dump_items id)
{
char buf[DUMP_SIZE];
@@ -146,7 +178,6 @@ static void dump_initial_format(int fd)
char magic[] = TRACECMD_MAGIC;
char buf[DUMP_SIZE];
int val4;
- unsigned long ver;
do_print(SUMMARY, "\t[Initial format]\n");
@@ -168,11 +199,11 @@ static void dump_initial_format(int fd)
die("no version string");
do_print(SUMMARY, "\t\t%s\t[Version]\n", buf);
- ver = strtol(buf, NULL, 10);
- if (!ver && errno)
+ file_version = strtol(buf, NULL, 10);
+ if (!file_version && errno)
die("Invalid file version string %s", buf);
- if (!tracecmd_is_version_supported(ver))
- die("Unsupported file version %lu", ver);
+ if (!tracecmd_is_version_supported(file_version))
+ die("Unsupported file version %lu", file_version);
/* get file endianness*/
if (read_file_bytes(fd, buf, 1))
@@ -192,6 +223,29 @@ static void dump_initial_format(int fd)
do_print(SUMMARY, "\t\t%d\t[Page size, bytes]\n", val4);
}
+static void dump_compress(int fd)
+{
+ char compr[DUMP_SIZE];
+ char *ver = NULL;
+
+ if (file_version < 7)
+ return;
+
+ /* get compression header */
+ if (read_file_string(fd, compr, DUMP_SIZE))
+ die("no compression header");
+ ver = strchr(compr, ' ');
+ if (!ver)
+ die("no compression version");
+ *ver = '\0';
+ do_print((SUMMARY), "\t\t%s\t[Compression algorithm]\n", compr);
+ do_print((SUMMARY), "\t\t%s\t[Compression version]\n", ver + 1);
+
+ z_handle = tracecmd_compress_proto_get(compr, ver + 1);
+ if (!z_handle)
+ die("cannot uncomress the file");
+}
+
static void dump_header_page(int fd)
{
unsigned long long size;
@@ -234,11 +288,60 @@ static void dump_header_event(int fd)
read_dump_string(fd, size, HEAD_EVENT);
}
+static void uncompress_reset(void)
+{
+ free(z_buffer);
+ z_buffer = NULL;
+ z_buffer_size = 0;
+ z_pointer = 0;
+}
+
+static int uncompress_block(int fd)
+{
+ unsigned int s_compressed;
+ unsigned int s_uncompressed;
+ char *bytes = NULL;
+ int ret;
+
+ if (file_version < 7 || !z_handle)
+ return 0;
+ uncompress_reset();
+ if (read_file_number(fd, &s_compressed, 4))
+ return -1;
+ if (read_file_number(fd, &s_uncompressed, 4))
+ return -1;
+
+ z_buffer = malloc(s_uncompressed);
+ if (!z_buffer)
+ return -1;
+ bytes = malloc(s_compressed);
+ if (!bytes)
+ goto error;
+
+ if (read_fd(fd, bytes, s_compressed) < 0)
+ goto error;
+ ret = tracecmd_uncompress_data(z_handle, bytes, s_compressed,
+ z_buffer, &s_uncompressed);
+ if (ret)
+ goto error;
+ free(bytes);
+ z_buffer_size = s_uncompressed;
+ z_pointer = 0;
+ return 0;
+error:
+ uncompress_reset();
+ free(bytes);
+ return -1;
+
+}
+
static void dump_ftrace_events_format(int fd)
{
unsigned long long size;
unsigned int count;
+ if (uncompress_block(fd))
+ die("cannot uncopress file block");
do_print((SUMMARY | FTRACE_FORMAT), "\t[Ftrace format, ");
if (read_file_number(fd, &count, 4))
die("cannot read the count of the ftrace events");
@@ -251,6 +354,7 @@ static void dump_ftrace_events_format(int fd)
read_dump_string(fd, size, FTRACE_FORMAT);
count--;
}
+ uncompress_reset();
}
static void dump_events_format(int fd)
@@ -262,6 +366,9 @@ static void dump_events_format(int fd)
do_print((SUMMARY | EVENT_FORMAT | EVENT_SYSTEMS), "\t[Events format, ");
+ if (uncompress_block(fd))
+ die("cannot uncopress file block");
+
if (read_file_number(fd, &systems, 4))
die("cannot read the count of the event systems");
@@ -284,6 +391,7 @@ static void dump_events_format(int fd)
}
systems--;
}
+ uncompress_reset();
}
static void dump_kallsyms(int fd)
@@ -292,12 +400,17 @@ static void dump_kallsyms(int fd)
do_print((SUMMARY | KALLSYMS), "\t[Kallsyms, ");
+ if (uncompress_block(fd))
+ die("cannot uncopress file block");
+
if (read_file_number(fd, &size, 4))
die("cannot read the size of the kallsyms");
do_print((SUMMARY | KALLSYMS), "%d bytes]\n", size);
read_dump_string(fd, size, KALLSYMS);
+
+ uncompress_reset();
}
static void dump_printk(int fd)
@@ -306,12 +419,17 @@ static void dump_printk(int fd)
do_print((SUMMARY | TRACE_PRINTK), "\t[Trace printk, ");
+ if (uncompress_block(fd))
+ die("cannot uncopress file block");
+
if (read_file_number(fd, &size, 4))
die("cannot read the size of the trace printk");
do_print((SUMMARY | TRACE_PRINTK), "%d bytes]\n", size);
read_dump_string(fd, size, TRACE_PRINTK);
+
+ uncompress_reset();
}
static void dump_cmdlines(int fd)
@@ -320,12 +438,17 @@ static void dump_cmdlines(int fd)
do_print((SUMMARY | CMDLINES), "\t[Saved command lines, ");
+ if (uncompress_block(fd))
+ die("cannot uncopress file block");
+
if (read_file_number(fd, &size, 8))
die("cannot read the size of the saved command lines");
do_print((SUMMARY | CMDLINES), "%d bytes]\n", size);
read_dump_string(fd, size, CMDLINES);
+
+ uncompress_reset();
}
static void dump_cpus_count(int fd)
@@ -669,6 +792,7 @@ static void dump_file(const char *file)
do_print(SUMMARY, "\n Tracing meta data in file %s:\n", file);
dump_initial_format(fd);
+ dump_compress(fd);
dump_header_page(fd);
dump_header_event(fd);
dump_ftrace_events_format(fd);
--
2.30.2
^ permalink raw reply related [flat|nested] 14+ messages in thread* [PATCH 10/10] trace-cmd library: Add support for zlib compression library
2021-04-22 7:38 [POC PATCH 00/10] Add trace file compression Tzvetomir Stoyanov (VMware)
` (8 preceding siblings ...)
2021-04-22 7:39 ` [PATCH 09/10] trace-cmd dump: Add support for trace files version 7 Tzvetomir Stoyanov (VMware)
@ 2021-04-22 7:39 ` Tzvetomir Stoyanov (VMware)
2021-04-29 1:35 ` [POC PATCH 00/10] Add trace file compression Steven Rostedt
10 siblings, 0 replies; 14+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-04-22 7:39 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
If libz is available, use that library to provide compression support
for the trace file compression. The library is detected runtime.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
lib/trace-cmd/Makefile | 10 ++
lib/trace-cmd/include/trace-cmd-local.h | 5 +
lib/trace-cmd/trace-compress-zlib.c | 172 ++++++++++++++++++++++++
lib/trace-cmd/trace-compress.c | 6 +-
4 files changed, 191 insertions(+), 2 deletions(-)
create mode 100644 lib/trace-cmd/trace-compress-zlib.c
diff --git a/lib/trace-cmd/Makefile b/lib/trace-cmd/Makefile
index bab4322d..83ba7016 100644
--- a/lib/trace-cmd/Makefile
+++ b/lib/trace-cmd/Makefile
@@ -7,6 +7,13 @@ ldir:=$(src)/lib/trace-cmd
DEFAULT_TARGET = $(LIBTRACECMD_STATIC)
+pound := \#
+ZLIB_INSTALLED := $(shell if (printf "$(pound)include <zlib.h>\n void main(){deflateInit(NULL, Z_BEST_COMPRESSION);}" | $(CC) -o /dev/null -x c - -lz >/dev/null 2>&1) ; then echo 1; else echo 0 ; fi)
+ifeq ($(ZLIB_INSTALLED), 1)
+CFLAGS += -DHAVE_ZLIB
+$(info Have zlib compression support)
+endif
+
OBJS =
OBJS += trace-hash.o
OBJS += trace-hooks.o
@@ -26,6 +33,9 @@ OBJS += trace-timesync-ptp.o
OBJS += trace-timesync-kvm.o
endif
OBJS += trace-compress.o
+ifeq ($(ZLIB_INSTALLED), 1)
+OBJS += trace-compress-zlib.o
+endif
# Additional util objects
OBJS += trace-blk-hack.o
diff --git a/lib/trace-cmd/include/trace-cmd-local.h b/lib/trace-cmd/include/trace-cmd-local.h
index c8c51ff0..86d91caf 100644
--- a/lib/trace-cmd/include/trace-cmd-local.h
+++ b/lib/trace-cmd/include/trace-cmd-local.h
@@ -30,6 +30,11 @@ void tracecmd_fatal(const char *fmt, ...);
#endif
#endif
+#ifdef HAVE_ZLIB
+int tracecmd_zlib_init(void);
+void tracecmd_zlib_free(void);
+#endif
+
int tracecmd_compress_init(void);
void tracecmd_compress_free(void);
diff --git a/lib/trace-cmd/trace-compress-zlib.c b/lib/trace-cmd/trace-compress-zlib.c
new file mode 100644
index 00000000..3208d57b
--- /dev/null
+++ b/lib/trace-cmd/trace-compress-zlib.c
@@ -0,0 +1,172 @@
+// SPDX-License-Identifier: LGPL-2.1
+/*
+ * Copyright (C) 2021, VMware, Tzvetomir Stoyanov tz.stoyanov@gmail.com>
+ *
+ */
+#include <stdlib.h>
+#include <dlfcn.h>
+#include <zlib.h>
+#include <errno.h>
+
+#include "trace-cmd-private.h"
+
+#define __ZLIB_NAME "zlib"
+#define __ZLIB_WEIGTH 10
+#define __ZLIB_FILE "libz.so"
+#define ZLIB_FUNC_COMPRESS "compress2"
+#define ZLIB_FUNC_DECOMOPRESS "uncompress"
+#define ZLIB_FUNC_SIZE "compressBound"
+#define ZLIB_FUNC_VERSION "zlibVersion"
+
+static void *zlib_handle;
+static int (*_lib_compress)(unsigned char *out, unsigned long *out_bytes,
+ unsigned char *in, unsigned long in_bytes, int level);
+static int (*_libz_decompress)(unsigned char *out, unsigned long *out_bytes,
+ unsigned char *in, unsigned long in_bytes);
+static unsigned long (*_libz_compress_bound)(unsigned long in_bytes);
+static const char *(*_libz_version)(void);
+
+static int zlib_compress(char *in, unsigned int in_bytes,
+ char *out, unsigned int *out_bytes)
+{
+ unsigned long out_size = *out_bytes;
+ int ret;
+
+ if (!_lib_compress)
+ return -1;
+
+ ret = _lib_compress((unsigned char *)out, &out_size,
+ (unsigned char *)in, (unsigned long)in_bytes, Z_BEST_COMPRESSION);
+ *out_bytes = out_size;
+ errno = 0;
+ switch (ret) {
+ case Z_OK:
+ return 0;
+ case Z_BUF_ERROR:
+ errno = -ENOBUFS;
+ break;
+ case Z_MEM_ERROR:
+ errno = -ENOMEM;
+ break;
+ case Z_STREAM_ERROR:
+ errno = -EINVAL;
+ break;
+ default:
+ errno = -EFAULT;
+ break;
+ }
+
+ return -1;
+}
+
+static int zlib_decompress(char *in, unsigned int in_bytes,
+ char *out, unsigned int *out_bytes)
+{
+ unsigned long out_size = *out_bytes;
+ int ret;
+
+ if (!_libz_decompress)
+ return -1;
+
+ ret = _libz_decompress((unsigned char *)out, &out_size,
+ (unsigned char *)in, (unsigned long)in_bytes);
+ *out_bytes = out_size;
+ errno = 0;
+ switch (ret) {
+ case Z_OK:
+ return 0;
+ case Z_BUF_ERROR:
+ errno = -ENOBUFS;
+ break;
+ case Z_MEM_ERROR:
+ errno = -ENOMEM;
+ break;
+ case Z_DATA_ERROR:
+ errno = -EINVAL;
+ break;
+ default:
+ errno = -EFAULT;
+ break;
+ }
+
+ return -1;
+}
+
+static unsigned int zlib_compress_bound(unsigned int in_bytes)
+{
+ if (!_libz_compress_bound)
+ return 0;
+ return _libz_compress_bound(in_bytes);
+}
+
+static bool zlib_is_supported(const char *name, const char *version)
+{
+ const char *zver;
+
+ if (!name)
+ return false;
+ if (strlen(name) != strlen(__ZLIB_NAME) || strcmp(name, __ZLIB_NAME))
+ return false;
+
+ if (!version)
+ return true;
+
+ if (!_libz_version)
+ return false;
+ zver = _libz_version();
+ if (!zver)
+ return false;
+
+ /* Compare the major version number */
+ if (atoi(version) <= atoi(zver))
+ return true;
+
+ return false;
+}
+
+int tracecmd_zlib_init(void)
+{
+ if (zlib_handle)
+ return 0;
+
+ zlib_handle = dlopen(__ZLIB_FILE, RTLD_NOW | RTLD_GLOBAL);
+ if (!zlib_handle)
+ return -1;
+ _lib_compress = dlsym(zlib_handle, ZLIB_FUNC_COMPRESS);
+ if (!_lib_compress)
+ goto error;
+ _libz_decompress = dlsym(zlib_handle, ZLIB_FUNC_DECOMOPRESS);
+ if (!_libz_decompress)
+ goto error;
+ _libz_compress_bound = dlsym(zlib_handle, ZLIB_FUNC_SIZE);
+ if (!_libz_compress_bound)
+ goto error;
+ _libz_version = dlsym(zlib_handle, ZLIB_FUNC_VERSION);
+ if (!_libz_version)
+ goto error;
+
+ return tracecmd_compress_proto_register(__ZLIB_NAME, _libz_version(), __ZLIB_WEIGTH,
+ zlib_compress, zlib_decompress,
+ zlib_compress_bound, zlib_is_supported);
+
+error:
+ _lib_compress = NULL;
+ _libz_decompress = NULL;
+ _libz_version = NULL;
+ dlclose(zlib_handle);
+ zlib_handle = NULL;
+ return -1;
+}
+
+void tracecmd_zlib_free(void)
+{
+ _lib_compress = NULL;
+ _libz_decompress = NULL;
+ _libz_version = NULL;
+
+ if (zlib_handle) {
+ dlclose(zlib_handle);
+ zlib_handle = NULL;
+ }
+
+}
diff --git a/lib/trace-cmd/trace-compress.c b/lib/trace-cmd/trace-compress.c
index c59c8bdb..560b9d0d 100644
--- a/lib/trace-cmd/trace-compress.c
+++ b/lib/trace-cmd/trace-compress.c
@@ -38,8 +38,10 @@ int tracecmd_compress_init(void)
gettimeofday(&time, NULL);
srand((time.tv_sec * 1000) + (time.tv_usec / 1000));
-/* toDo: add here initialization of compression protocols */
-
+#ifdef HAVE_ZLIB
+ if (!tracecmd_zlib_init())
+ count++;
+#endif
if (!count)
tracecmd_warning("No compression support available");
--
2.30.2
^ permalink raw reply related [flat|nested] 14+ messages in thread* Re: [POC PATCH 00/10] Add trace file compression
2021-04-22 7:38 [POC PATCH 00/10] Add trace file compression Tzvetomir Stoyanov (VMware)
` (9 preceding siblings ...)
2021-04-22 7:39 ` [PATCH 10/10] trace-cmd library: Add support for zlib compression library Tzvetomir Stoyanov (VMware)
@ 2021-04-29 1:35 ` Steven Rostedt
2021-04-29 3:51 ` Tzvetomir Stoyanov
10 siblings, 1 reply; 14+ messages in thread
From: Steven Rostedt @ 2021-04-29 1:35 UTC (permalink / raw)
To: Tzvetomir Stoyanov (VMware); +Cc: linux-trace-devel
On Thu, 22 Apr 2021 10:38:52 +0300
"Tzvetomir Stoyanov (VMware)" <tz.stoyanov@gmail.com> wrote:
> A new PoC implementation for adding a compression to trace file version 7.
> A basic infrastructure for compression support is added into the trace-cmd
> library. The zlib is used for compression, but more libraries and algorithms
> can be added. Only part of the trace file metadata is compressed:
> - ftrace events format
> - format of recorded events
> - information about the mapping of function addresses to the function names
> - trace_printk() format strings
> - information of the mapping a PID to a process name
>
> Note: not all trace-cmd commands are tested with these changes. These are
> verified to work with compressed trace files:
> trace-cmd record
> trace-cmd report
> trace-cmd dump
>
> Todo list, in order to have complete trace file compression:
> - Compress trace file options.
> - Compress trace data.
> - Add support for more compression libraries.
> - Add new trace-cmd command to convert trace files from version 6 to
> version 7 and vise versa.
> - Add compression support to all trace-cmd commands.
> - Update trace-cmd documentation.
>
> This patch set depends on "[PATCH 0/6] Bump trace file version"
> and should be applied on top of it.
> https://lore.kernel.org/linux-trace-devel/20210422071718.483383-1-tz.stoyanov@gmail.com/
>
> Tzvetomir Stoyanov (VMware) (10):
> trace-cmd: Add APIs for library initialization and free
> trace-cmd library: Add support for compression algorithms
> trace-cmd library: Compress part of the trace file
> trace-cmd library: Read compressed trace file
> trace-cmd library: Add new API to get compression of input handler
> trace-cmd library: Inherit compression algorithm from input file
> trace-cmd library: Extend the create file APIs to support different
> compression
> trace-cmd record: Add new parameter --compression
> trace-cmd dump: Add support for trace files version 7
> trace-cmd library: Add support for zlib compression library
I'm not able to apply these patches as I'm getting conflicts from your
previous series that bumps the version number, as if what you sent to
the mailing list was not what you have in your git repo.
Can you please resend the version update patch series.
-- Steve
>
> lib/trace-cmd/Makefile | 11 +
> .../include/private/trace-cmd-private.h | 42 ++-
> lib/trace-cmd/include/trace-cmd-local.h | 7 +
> lib/trace-cmd/trace-compress-zlib.c | 172 ++++++++++
> lib/trace-cmd/trace-compress.c | 289 +++++++++++++++++
> lib/trace-cmd/trace-input.c | 304 +++++++++++++-----
> lib/trace-cmd/trace-output.c | 268 ++++++++++++---
> lib/trace-cmd/trace-util.c | 12 +
> tracecmd/trace-cmd.c | 11 +-
> tracecmd/trace-dump.c | 164 ++++++++--
> tracecmd/trace-record.c | 18 +-
> tracecmd/trace-restore.c | 4 +-
> tracecmd/trace-usage.c | 1 +
> 13 files changed, 1152 insertions(+), 151 deletions(-)
> create mode 100644 lib/trace-cmd/trace-compress-zlib.c
> create mode 100644 lib/trace-cmd/trace-compress.c
>
^ permalink raw reply [flat|nested] 14+ messages in thread* Re: [POC PATCH 00/10] Add trace file compression
2021-04-29 1:35 ` [POC PATCH 00/10] Add trace file compression Steven Rostedt
@ 2021-04-29 3:51 ` Tzvetomir Stoyanov
2021-04-29 13:20 ` Steven Rostedt
0 siblings, 1 reply; 14+ messages in thread
From: Tzvetomir Stoyanov @ 2021-04-29 3:51 UTC (permalink / raw)
To: Steven Rostedt; +Cc: Linux Trace Devel
On Thu, Apr 29, 2021 at 4:35 AM Steven Rostedt <rostedt@goodmis.org> wrote:
>
> On Thu, 22 Apr 2021 10:38:52 +0300
> "Tzvetomir Stoyanov (VMware)" <tz.stoyanov@gmail.com> wrote:
>
> > A new PoC implementation for adding a compression to trace file version 7.
> > A basic infrastructure for compression support is added into the trace-cmd
> > library. The zlib is used for compression, but more libraries and algorithms
> > can be added. Only part of the trace file metadata is compressed:
> > - ftrace events format
> > - format of recorded events
> > - information about the mapping of function addresses to the function names
> > - trace_printk() format strings
> > - information of the mapping a PID to a process name
> >
> > Note: not all trace-cmd commands are tested with these changes. These are
> > verified to work with compressed trace files:
> > trace-cmd record
> > trace-cmd report
> > trace-cmd dump
> >
> > Todo list, in order to have complete trace file compression:
> > - Compress trace file options.
> > - Compress trace data.
> > - Add support for more compression libraries.
> > - Add new trace-cmd command to convert trace files from version 6 to
> > version 7 and vise versa.
> > - Add compression support to all trace-cmd commands.
> > - Update trace-cmd documentation.
> >
> > This patch set depends on "[PATCH 0/6] Bump trace file version"
> > and should be applied on top of it.
> > https://lore.kernel.org/linux-trace-devel/20210422071718.483383-1-tz.stoyanov@gmail.com/
> >
> > Tzvetomir Stoyanov (VMware) (10):
> > trace-cmd: Add APIs for library initialization and free
> > trace-cmd library: Add support for compression algorithms
> > trace-cmd library: Compress part of the trace file
> > trace-cmd library: Read compressed trace file
> > trace-cmd library: Add new API to get compression of input handler
> > trace-cmd library: Inherit compression algorithm from input file
> > trace-cmd library: Extend the create file APIs to support different
> > compression
> > trace-cmd record: Add new parameter --compression
> > trace-cmd dump: Add support for trace files version 7
> > trace-cmd library: Add support for zlib compression library
>
> I'm not able to apply these patches as I'm getting conflicts from your
> previous series that bumps the version number, as if what you sent to
> the mailing list was not what you have in your git repo.
>
> Can you please resend the version update patch series.
The order of applying the changes should be:
1. [PATCH v2] trace-cmd: Check if file version is supported
https://lore.kernel.org/linux-trace-devel/20210419093156.200099-1-tz.stoyanov@gmail.com/
2. [PATCH 0/6] Bump trace file version
https://lore.kernel.org/linux-trace-devel/20210422071718.483383-1-tz.stoyanov@gmail.com/
3. [POC PATCH 00/10] Add trace file compression
https://lore.kernel.org/linux-trace-devel/20210422073902.484953-1-tz.stoyanov@gmail.com/
I wrote these dependencies in the cover-letters. BTW, I do not see
(2) in patchwork, only in the mailing list.
I'll send v2 of "Bump trace file version" including (1) in the patchset.
>
> -- Steve
>
>
> >
> > lib/trace-cmd/Makefile | 11 +
> > .../include/private/trace-cmd-private.h | 42 ++-
> > lib/trace-cmd/include/trace-cmd-local.h | 7 +
> > lib/trace-cmd/trace-compress-zlib.c | 172 ++++++++++
> > lib/trace-cmd/trace-compress.c | 289 +++++++++++++++++
> > lib/trace-cmd/trace-input.c | 304 +++++++++++++-----
> > lib/trace-cmd/trace-output.c | 268 ++++++++++++---
> > lib/trace-cmd/trace-util.c | 12 +
> > tracecmd/trace-cmd.c | 11 +-
> > tracecmd/trace-dump.c | 164 ++++++++--
> > tracecmd/trace-record.c | 18 +-
> > tracecmd/trace-restore.c | 4 +-
> > tracecmd/trace-usage.c | 1 +
> > 13 files changed, 1152 insertions(+), 151 deletions(-)
> > create mode 100644 lib/trace-cmd/trace-compress-zlib.c
> > create mode 100644 lib/trace-cmd/trace-compress.c
> >
>
--
Tzvetomir (Ceco) Stoyanov
VMware Open Source Technology Center
^ permalink raw reply [flat|nested] 14+ messages in thread