linux-trace-devel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: "Tzvetomir Stoyanov (VMware)" <tz.stoyanov@gmail.com>
To: rostedt@goodmis.org
Cc: linux-trace-devel@vger.kernel.org
Subject: [PATCH v6 19/45] trace-cmd: Move buffers description outside of options
Date: Mon, 14 Jun 2021 10:50:03 +0300	[thread overview]
Message-ID: <20210614075029.598048-20-tz.stoyanov@gmail.com> (raw)
In-Reply-To: <20210614075029.598048-1-tz.stoyanov@gmail.com>

When a new buffer data are written in the trace file, the file offset is
stored in the buffer description. As the buffer description is part of
the trace options, which are already written in the file, the offset field
in this option is updated with the new value. This causes a serious
problem when option section of the file is compressed. Updating a value
in already compressed section may change the size of this section, which
can make the calculated offset of the buffer trace data invalid. To
solve that problem and to enable the compression of the option section
of the file, the structure of the trace file is changed:
 - In trace file v7, the description of the buffers is moved to a
   separate section in the trace file medatata. It is stored after the
   CPU count section and before the option section. The format of this
   new buffers description is:
    4 bytes - count of the buffers, stored in the file. The top buffer
      is not counted.
    For each buffer:
     8 bytes, offset of the trace data for this buffer in the file.
     null-terminated string, name of the buffer.
   The new section is mandatory, in case of no buffers - the count 0 is
   stored in the first 4 bytes.
 - In trace file v6, the description of the buffers is not changed.

These changes are needed for compression of the options sections from
the trace file.

Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
 .../include/private/trace-cmd-private.h       |  1 +
 lib/trace-cmd/include/trace-cmd-local.h       |  3 +
 lib/trace-cmd/trace-input.c                   | 40 +++++++++
 lib/trace-cmd/trace-output.c                  | 85 ++++++++++++++++++-
 tracecmd/trace-dump.c                         | 24 ++++++
 tracecmd/trace-listen.c                       |  3 +
 tracecmd/trace-record.c                       |  4 +
 7 files changed, 156 insertions(+), 4 deletions(-)

diff --git a/lib/trace-cmd/include/private/trace-cmd-private.h b/lib/trace-cmd/include/private/trace-cmd-private.h
index cbb578ec..faa8c533 100644
--- a/lib/trace-cmd/include/private/trace-cmd-private.h
+++ b/lib/trace-cmd/include/private/trace-cmd-private.h
@@ -112,6 +112,7 @@ enum tracecmd_file_states {
 	TRACECMD_FILE_PRINTK,
 	TRACECMD_FILE_CMD_LINES,
 	TRACECMD_FILE_CPU_COUNT,
+	TRACECMD_FILE_BUFERS,
 	TRACECMD_FILE_OPTIONS,
 	TRACECMD_FILE_CPU_LATENCY,
 	TRACECMD_FILE_CPU_FLYRECORD,
diff --git a/lib/trace-cmd/include/trace-cmd-local.h b/lib/trace-cmd/include/trace-cmd-local.h
index 40a6a40d..2f421103 100644
--- a/lib/trace-cmd/include/trace-cmd-local.h
+++ b/lib/trace-cmd/include/trace-cmd-local.h
@@ -46,4 +46,7 @@ void out_compression_reset(struct tracecmd_output *handle);
 void in_uncompress_reset(struct tracecmd_input *handle);
 int in_uncompress_block(struct tracecmd_input *handle);
 
+int write_buffers_description_v7(struct tracecmd_output *handle);
+int write_buffers_description_v6(struct tracecmd_output *handle);
+
 #endif /* _TRACE_CMD_LOCAL_H */
diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c
index 0b17b58d..e953909f 100644
--- a/lib/trace-cmd/trace-input.c
+++ b/lib/trace-cmd/trace-input.c
@@ -881,6 +881,39 @@ static int read_cpus(struct tracecmd_input *handle)
 	return 0;
 }
 
+static int read_buffers(struct tracecmd_input *handle)
+{
+	struct input_buffer_instance *buffer;
+	unsigned long long offset;
+	unsigned int count;
+	int i;
+
+	if (handle->file_version < 7 || handle->file_state >= TRACECMD_FILE_BUFERS)
+		return 0;
+
+	if (read4(handle, &count) < 0)
+		return -1;
+	i = handle->nr_buffers;
+	handle->nr_buffers += count;
+	handle->buffers = realloc(handle->buffers,
+				  sizeof(*handle->buffers) * handle->nr_buffers);
+	if (!handle->buffers)
+		return -1;
+
+	for (; i < handle->nr_buffers; i++) {
+		buffer = &handle->buffers[i];
+		if (read8(handle, &offset) < 0)
+			return -1;
+		buffer->offset = offset;
+		buffer->name = read_string(handle);
+		if (!buffer->name)
+			return -1;
+	}
+
+	handle->file_state = TRACECMD_FILE_BUFERS;
+	return 0;
+}
+
 /**
  * tracecmd_read_headers - read the header information from trace.dat
  * @handle: input handle for the trace.dat file
@@ -955,6 +988,13 @@ int tracecmd_read_headers(struct tracecmd_input *handle,
 	if (state <= handle->file_state)
 		return 0;
 
+	if (read_buffers(handle) < 0)
+		return -1;
+
+	if (state <= handle->file_state)
+		return 0;
+
+
 	if (read_options_type(handle) < 0)
 		return -1;
 
diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c
index 8f8ca164..7830f643 100644
--- a/lib/trace-cmd/trace-output.c
+++ b/lib/trace-cmd/trace-output.c
@@ -376,6 +376,17 @@ static int check_out_state(struct tracecmd_output *handle, int new_state)
 	if (!handle)
 		return -1;
 
+	if (handle->file_version < 7) {
+		switch (new_state) {
+		case TRACECMD_FILE_BUFERS:
+			tracecmd_warning("State %d is not supported in trace file version %d",
+					 new_state, handle->file_version);
+			return -1;
+		default:
+			break;
+		}
+	}
+
 	switch (new_state) {
 	case TRACECMD_FILE_HEADERS:
 	case TRACECMD_FILE_FTRACE_EVENTS:
@@ -384,10 +395,16 @@ static int check_out_state(struct tracecmd_output *handle, int new_state)
 	case TRACECMD_FILE_PRINTK:
 	case TRACECMD_FILE_CMD_LINES:
 	case TRACECMD_FILE_CPU_COUNT:
-	case TRACECMD_FILE_OPTIONS:
+	case TRACECMD_FILE_BUFERS:
 		if (handle->file_state == (new_state - 1))
 			return 0;
 		break;
+	case TRACECMD_FILE_OPTIONS:
+		if (handle->file_version < 7 && handle->file_state == TRACECMD_FILE_CPU_COUNT)
+			return 0;
+		if (handle->file_version >= 7 && handle->file_state == TRACECMD_FILE_BUFERS)
+			return 0;
+		break;
 	case TRACECMD_FILE_CPU_LATENCY:
 	case TRACECMD_FILE_CPU_FLYRECORD:
 		if (handle->file_state == TRACECMD_FILE_OPTIONS)
@@ -1433,7 +1450,7 @@ int tracecmd_add_buffer_description(struct tracecmd_output *handle, const char *
 	return 0;
 }
 
-int tracecmd_write_buffers_description(struct tracecmd_output *handle)
+__hidden int write_buffers_description_v6(struct tracecmd_output *handle)
 {
 	struct tracecmd_option *option;
 	struct tracecmd_buffer *buf;
@@ -1447,6 +1464,63 @@ int tracecmd_write_buffers_description(struct tracecmd_output *handle)
 	return 0;
 }
 
+__hidden int write_buffers_description_v7(struct tracecmd_output *handle)
+{
+	struct tracecmd_buffer *buf;
+	char *data = NULL;
+	int count = 0;
+	int size;
+	int ret;
+
+	if (!check_out_state(handle, TRACECMD_FILE_BUFERS)) {
+		tracecmd_warning("Cannot write buffers descriptions");
+		return -1;
+	}
+
+	list_for_each_entry(buf, &handle->buffers, list) {
+		count++;
+	}
+
+	count = convert_endian_4(handle, count);
+	ret = do_write_check(handle, &count, 4);
+	if (ret < 0)
+		return ret;
+
+	list_for_each_entry(buf, &handle->buffers, list) {
+		size = 8 + strlen(buf->name) + 1;
+		data = malloc(size);
+		if (!data)
+			goto error;
+		*(tsize_t *)data = 0;
+		strcpy(data + 8, buf->name);
+		/* Save the data location in case it needs to be updated */
+		buf->offset = do_lseek(handle, 0, SEEK_CUR);
+		if (do_write_check(handle, data, size))
+			goto error;
+
+		/*
+		 * In case a buffer instance has different number of CPUs as the
+		 * local machine.
+		 */
+		if (buf->cpus)
+			tracecmd_add_option(handle, TRACECMD_OPTION_CPUCOUNT,
+					    sizeof(int), &buf->cpus);
+	}
+
+	handle->file_state = TRACECMD_FILE_BUFERS;
+	return 0;
+error:
+	return -1;
+}
+
+int tracecmd_write_buffers_description(struct tracecmd_output *handle)
+{
+	if (handle->file_version >= 7)
+		return write_buffers_description_v7(handle);
+
+	return write_buffers_description_v6(handle);
+}
+
 static tsize_t get_buffer_file_offset(struct tracecmd_output *handle, const char *name)
 {
 	struct tracecmd_buffer *buf;
@@ -1505,7 +1579,8 @@ struct tracecmd_output *tracecmd_create_file_latency(const char *output_file, in
 
 	if (tracecmd_write_cpus(handle, cpus) < 0)
 		goto out_free;
-
+	if (tracecmd_write_buffers_description(handle) < 0)
+		goto out_free;
 	if (tracecmd_write_options(handle) < 0)
 		goto out_free;
 
@@ -1698,7 +1773,9 @@ int tracecmd_append_cpu_data(struct tracecmd_output *handle,
 	ret = tracecmd_write_cpus(handle, cpus);
 	if (ret)
 		return ret;
-
+	ret = tracecmd_write_buffers_description(handle);
+	if (ret)
+		return ret;
 	ret = tracecmd_write_options(handle);
 	if (ret)
 		return ret;
diff --git a/tracecmd/trace-dump.c b/tracecmd/trace-dump.c
index ed82104b..56acb01d 100644
--- a/tracecmd/trace-dump.c
+++ b/tracecmd/trace-dump.c
@@ -420,6 +420,29 @@ static void dump_cpus_count(int fd)
 	do_print(SUMMARY, "\t%d [CPUs with tracing data]\n", trace_cpus);
 }
 
+static void dump_buffers(int fd)
+{
+	unsigned long long offset;
+	char name[DUMP_SIZE];
+	int count;
+
+	if (file_version < 7)
+		return;
+
+	if (read_file_number(fd, &count, 4))
+		die("cannot read the buffers count");
+	do_print(SUMMARY, "\t%d [buffers with tracing data]\n", count);
+
+	while (count) {
+		if (read_file_number(fd, &offset, 8))
+			die("cannot read the buffer offset");
+		if (read_file_string(fd, name, DUMP_SIZE))
+			die("cannot read the buffer name");
+		do_print(SUMMARY, "\t\t 0x%llx %s[data offset, buffer name]\n", offset, name);
+		count--;
+	}
+}
+
 static void dump_option_string(int fd, int size, char *desc)
 {
 	do_print(OPTIONS, "\t\t[Option %s, %d bytes]\n", desc, size);
@@ -762,6 +785,7 @@ static void dump_file(const char *file)
 	dump_printk(fd);
 	dump_cmdlines(fd);
 	dump_cpus_count(fd);
+	dump_buffers(fd);
 	dump_therest(fd);
 
 	tep_free(tep);
diff --git a/tracecmd/trace-listen.c b/tracecmd/trace-listen.c
index 0ae1c948..62debdb6 100644
--- a/tracecmd/trace-listen.c
+++ b/tracecmd/trace-listen.c
@@ -604,6 +604,9 @@ static int put_together_file(int cpus, int ofd, const char *node,
 
 	if (write_options) {
 		ret = tracecmd_write_cpus(handle, cpus);
+		if (ret)
+			goto out;
+		ret = tracecmd_write_buffers_description(handle);
 		if (ret)
 			goto out;
 		ret = tracecmd_write_options(handle);
diff --git a/tracecmd/trace-record.c b/tracecmd/trace-record.c
index eff6f2f0..2a379d09 100644
--- a/tracecmd/trace-record.c
+++ b/tracecmd/trace-record.c
@@ -3711,6 +3711,9 @@ setup_connection(struct buffer_instance *instance, struct common_record_context
 		if (ret)
 			goto error;
 		ret = tracecmd_write_cpus(network_handle, instance->cpu_count);
+		if (ret)
+			goto error;
+		ret = tracecmd_write_buffers_description(network_handle);
 		if (ret)
 			goto error;
 		ret = tracecmd_write_options(network_handle);
@@ -4074,6 +4077,7 @@ static void setup_agent(struct buffer_instance *instance,
 	add_options(network_handle, ctx);
 	tracecmd_write_cmdlines(network_handle);
 	tracecmd_write_cpus(network_handle, instance->cpu_count);
+	tracecmd_write_buffers_description(network_handle);
 	tracecmd_write_options(network_handle);
 	tracecmd_msg_finish_sending_data(instance->msg_handle);
 	instance->network_handle = network_handle;
-- 
2.31.1


  parent reply	other threads:[~2021-06-14  7:52 UTC|newest]

Thread overview: 64+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-06-14  7:49 [PATCH v6 00/45] Add trace file compression Tzvetomir Stoyanov (VMware)
2021-06-14  7:49 ` [PATCH v6 01/45] trace-cmd library: Remove unused private APIs for creating trace files Tzvetomir Stoyanov (VMware)
2021-06-14  7:49 ` [PATCH v6 02/45] trace-cmd library: Remove unused API tracecmd_update_option Tzvetomir Stoyanov (VMware)
2021-06-14  7:49 ` [PATCH v6 03/45] trace-cmd: Check if file version is supported Tzvetomir Stoyanov (VMware)
2021-06-21 22:27   ` Steven Rostedt
2021-06-21 22:36     ` Steven Rostedt
2021-06-14  7:49 ` [PATCH v6 04/45] trace-cmd library: Add new API to get file version of input handler Tzvetomir Stoyanov (VMware)
2021-06-14  7:49 ` [PATCH v6 05/45] trace-cmd library: Select the file version when writing trace file Tzvetomir Stoyanov (VMware)
2021-06-14  7:49 ` [PATCH v6 06/45] trace-cmd: Add APIs for library initialization and free Tzvetomir Stoyanov (VMware)
2021-06-14  7:49 ` [PATCH v6 07/45] trace-cmd library: Add support for compression algorithms Tzvetomir Stoyanov (VMware)
2021-06-14  7:49 ` [PATCH v6 08/45] trace-cmd list: Show supported " Tzvetomir Stoyanov (VMware)
2021-06-14  7:49 ` [PATCH v6 09/45] trace-cmd library: Bump the trace file version to 7 Tzvetomir Stoyanov (VMware)
2021-06-14  7:49 ` [PATCH v6 10/45] trace-cmd library: Compress part of the trace file Tzvetomir Stoyanov (VMware)
2021-06-14  7:49 ` [PATCH v6 11/45] trace-cmd library: Read compressed " Tzvetomir Stoyanov (VMware)
2021-06-14  7:49 ` [PATCH v6 12/45] trace-cmd library: Add new API to get compression of input handler Tzvetomir Stoyanov (VMware)
2021-06-14  7:49 ` [PATCH v6 13/45] trace-cmd library: Inherit compression algorithm from input file Tzvetomir Stoyanov (VMware)
2021-06-14  7:49 ` [PATCH v6 14/45] trace-cmd library: Extend the create file APIs to support different compression Tzvetomir Stoyanov (VMware)
2021-06-14  7:49 ` [PATCH v6 15/45] trace-cmd record: Add new parameter --compression Tzvetomir Stoyanov (VMware)
2021-06-14  7:50 ` [PATCH v6 16/45] trace-cmd dump: Add support for trace files version 7 Tzvetomir Stoyanov (VMware)
2021-06-14  7:50 ` [PATCH v6 17/45] trace-cmd library: Add support for zlib compression library Tzvetomir Stoyanov (VMware)
2021-06-22  1:26   ` Steven Rostedt
2021-06-22 10:29     ` Tzvetomir Stoyanov
2021-06-22 13:31       ` Steven Rostedt
2021-06-14  7:50 ` [PATCH v6 18/45] trace-cmd library: Hide the logic for updating buffer offset Tzvetomir Stoyanov (VMware)
2021-06-22  2:10   ` Steven Rostedt
2021-06-14  7:50 ` Tzvetomir Stoyanov (VMware) [this message]
2021-06-21 23:07   ` [PATCH v6 19/45] trace-cmd: Move buffers description outside of options Steven Rostedt
2021-06-14  7:50 ` [PATCH v6 20/45] trace-cmd library: Track the offset in the option section in the trace file Tzvetomir Stoyanov (VMware)
2021-06-14  7:50 ` [PATCH v6 21/45] trace-cmd library: Add compression of the option section of " Tzvetomir Stoyanov (VMware)
2021-06-21 23:10   ` Steven Rostedt
2021-06-22 10:43     ` Tzvetomir Stoyanov
2021-06-14  7:50 ` [PATCH v6 22/45] trace-cmd library: Refactor the logic for writing trace data in the file Tzvetomir Stoyanov (VMware)
2021-06-21 23:12   ` Steven Rostedt
2021-06-14  7:50 ` [PATCH v6 23/45] trace-cmd library: Add APIs for read and write compressed data in chunks Tzvetomir Stoyanov (VMware)
2021-06-14  7:50 ` [PATCH v6 24/45] trace-cmd: Compress trace data Tzvetomir Stoyanov (VMware)
2021-06-14  7:50 ` [PATCH v6 25/45] trace-cmd: Read compressed " Tzvetomir Stoyanov (VMware)
2021-06-21 23:23   ` Steven Rostedt
2021-06-22 10:50     ` Tzvetomir Stoyanov
2021-06-22 13:51       ` Steven Rostedt
2021-06-14  7:50 ` [PATCH v6 26/45] trace-cmd library: Compress latency " Tzvetomir Stoyanov (VMware)
2021-06-14  7:50 ` [PATCH v6 27/45] trace-cmd: Read compressed " Tzvetomir Stoyanov (VMware)
2021-06-14  7:50 ` [PATCH v6 28/45] trace-cmd library: Reuse within the library the function that checks file state Tzvetomir Stoyanov (VMware)
2021-06-14  7:50 ` [PATCH v6 29/45] trace-cmd library: Make tracecmd_copy_headers() to work with output handler Tzvetomir Stoyanov (VMware)
2021-06-14  7:50 ` [PATCH v6 30/45] trace-cmd: Do not use trace file compression with streams Tzvetomir Stoyanov (VMware)
2021-06-14  7:50 ` [PATCH v6 31/45] trace-cmd library: Add new API to get file version of output handler Tzvetomir Stoyanov (VMware)
2021-06-14  7:50 ` [PATCH v6 32/45] trace-cmd: Add file state parameter to tracecmd_copy Tzvetomir Stoyanov (VMware)
2021-06-14  7:50 ` [PATCH v6 33/45] trace-cmd: Copy CPU count in tracecmd_copy Tzvetomir Stoyanov (VMware)
2021-06-21 23:27   ` Steven Rostedt
2021-06-14  7:50 ` [PATCH v6 34/45] trace-cmd: Copy buffers description " Tzvetomir Stoyanov (VMware)
2021-06-14  7:50 ` [PATCH v6 35/45] trace-cmd: Copy options " Tzvetomir Stoyanov (VMware)
2021-06-14  7:50 ` [PATCH v6 36/45] trace-cmd library: Refactor the logic for writing CPU trace data Tzvetomir Stoyanov (VMware)
2021-06-14  7:50 ` [PATCH v6 37/45] trace-cmd library: Refactor the logic for writing CPU instance " Tzvetomir Stoyanov (VMware)
2021-06-14  7:50 ` [PATCH v6 38/45] trace-cmd: Copy trace data in tracecmd_copy Tzvetomir Stoyanov (VMware)
2021-06-14  7:50 ` [PATCH v6 39/45] trace-cmd: Add compression parameter to tracecmd_copy Tzvetomir Stoyanov (VMware)
2021-06-14  7:50 ` [PATCH v6 40/45] trace-cmd: Add new command "trace-cmd convert" Tzvetomir Stoyanov (VMware)
2021-06-14  7:50 ` [PATCH v6 41/45] trace-cmd record: Update man page Tzvetomir Stoyanov (VMware)
2021-06-14  7:50 ` [PATCH v6 42/45] trace-cmd: Add convert " Tzvetomir Stoyanov (VMware)
2021-06-14  7:50 ` [PATCH v6 43/45] trace-cmd: Update bash completion Tzvetomir Stoyanov (VMware)
2021-06-14  7:50 ` [PATCH v6 44/45] trace-cmd list: Update the man page Tzvetomir Stoyanov (VMware)
2021-06-14  7:50 ` [PATCH v6 45/45] trace-cmd: Update trace.dat " Tzvetomir Stoyanov (VMware)
2021-06-22  0:37   ` Steven Rostedt
2021-06-22 11:05     ` Tzvetomir Stoyanov
2021-06-22 13:58       ` Steven Rostedt
2021-06-22  2:22 ` [PATCH v6 00/45] Add trace file compression Steven Rostedt

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20210614075029.598048-20-tz.stoyanov@gmail.com \
    --to=tz.stoyanov@gmail.com \
    --cc=linux-trace-devel@vger.kernel.org \
    --cc=rostedt@goodmis.org \
    --subject='Re: [PATCH v6 19/45] trace-cmd: Move buffers description outside of options' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).