linux-trace-devel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: "Tzvetomir Stoyanov (VMware)" <tz.stoyanov@gmail.com>
To: rostedt@goodmis.org
Cc: linux-trace-devel@vger.kernel.org
Subject: [PATCH v5 24/40] trace-cmd: Compress trace data
Date: Thu, 10 Jun 2021 14:34:10 +0300	[thread overview]
Message-ID: <20210610113426.257931-25-tz.stoyanov@gmail.com> (raw)
In-Reply-To: <20210610113426.257931-1-tz.stoyanov@gmail.com>

When creating a trace.dat file of version 7, compress the trace data.
The data is compressed in chunks. The chunk size is a compile time
parameter, set by default to 1MB.

Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
 lib/trace-cmd/trace-output.c | 50 +++++++++++++++++++++++++++++++-----
 1 file changed, 43 insertions(+), 7 deletions(-)

diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c
index 860be759..bd14452a 100644
--- a/lib/trace-cmd/trace-output.c
+++ b/lib/trace-cmd/trace-output.c
@@ -321,6 +321,41 @@ static tsize_t copy_file(struct tracecmd_output *handle,
 	return size;
 }
 
+#define COMPRESS_CHUNK_SIZE (1*1024*1024)
+static tsize_t copy_file_compress(struct tracecmd_output *handle,
+				  const char *file, unsigned long long *write_size)
+{
+	unsigned long long rsize = 0;
+	unsigned long long wsize = 0;
+	tsize_t size;
+	int ret;
+	int fd;
+
+	fd = open(file, O_RDONLY);
+	if (fd < 0) {
+		tracecmd_warning("Can't read '%s'", file);
+		return 0;
+	}
+
+	if (handle->file_version >= 7) {
+		ret = tracecmd_compress_copy_from(handle->compress, fd,
+						  COMPRESS_CHUNK_SIZE, &rsize, &wsize);
+		if (ret < 0) {
+			tracecmd_warning("Can't compress '%s'", file);
+			close(fd);
+			return 0;
+		}
+		size = rsize;
+		*write_size = wsize;
+	} else {
+		size = copy_file_fd(handle, fd);
+		*write_size = size;
+	}
+
+	close(fd);
+	return size;
+}
+
 /*
  * Finds the path to the debugfs/tracing
  * Allocates the string and stores it.
@@ -1692,8 +1727,9 @@ int tracecmd_write_cpu_data(struct tracecmd_output *handle,
 	struct data_file_write *data_files = NULL;
 	off64_t offset;
 	unsigned long long endian8;
+	unsigned long long read_size;
+	unsigned long long write_size;
 	char *clock = NULL;
-	off64_t check_size;
 	char *file;
 	struct stat st;
 	int ret;
@@ -1749,11 +1785,11 @@ int tracecmd_write_cpu_data(struct tracecmd_output *handle,
 			fprintf(stderr, "CPU%d data recorded at offset=0x%llx\n",
 				i, (unsigned long long) data_files[i].data_offset);
 
-		check_size = copy_file(handle, cpu_data_files[i]);
-		if (check_size != data_files[i].file_size) {
+		read_size = copy_file_compress(handle, cpu_data_files[i], &write_size);
+		if (read_size != data_files[i].file_size) {
 			errno = EINVAL;
 			tracecmd_warning("did not match size of %lld to %lld",
-					 check_size, data_files[i].file_size);
+					 read_size, data_files[i].file_size);
 			goto out_free;
 		}
 		/* Write the real CPU data offset inthe file */
@@ -1763,14 +1799,14 @@ int tracecmd_write_cpu_data(struct tracecmd_output *handle,
 			goto out_free;
 		/* Write the real CPU data size in the file */
 		offset = lseek64(handle->fd, data_files[i].soffset, SEEK_SET);
-		endian8 = convert_endian_8(handle, check_size);
+		endian8 = convert_endian_8(handle, write_size);
 		if (do_write_check(handle, &endian8, 8))
 			goto out_free;
-		offset = data_files[i].data_offset + check_size;
+		offset = data_files[i].data_offset + write_size;
 		offset = lseek64(handle->fd, offset, SEEK_SET);
 		if (!tracecmd_get_quiet(handle))
 			fprintf(stderr, "    %llu bytes in size\n",
-				(unsigned long long)check_size);
+				(unsigned long long)write_size);
 	}
 
 	if (lseek64(handle->fd, 0, SEEK_END) == (off64_t)-1)
-- 
2.31.1


  parent reply	other threads:[~2021-06-10 11:36 UTC|newest]

Thread overview: 44+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-06-10 11:33 [PATCH v5 00/40] Add trace file compression Tzvetomir Stoyanov (VMware)
2021-06-10 11:33 ` [PATCH v5 01/40] trace-cmd library: Remove unused private APIs for creating trace files Tzvetomir Stoyanov (VMware)
2021-06-10 11:33 ` [PATCH v5 02/40] trace-cmd library: Remove unused API tracecmd_update_option Tzvetomir Stoyanov (VMware)
2021-06-10 11:33 ` [PATCH v5 03/40] trace-cmd: Check if file version is supported Tzvetomir Stoyanov (VMware)
2021-06-10 11:33 ` [PATCH v5 04/40] trace-cmd library: Add new API to get file version of input handler Tzvetomir Stoyanov (VMware)
2021-06-10 11:33 ` [PATCH v5 05/40] trace-cmd library: Select the file version when writing trace file Tzvetomir Stoyanov (VMware)
2021-06-10 11:33 ` [PATCH v5 06/40] trace-cmd: Add APIs for library initialization and free Tzvetomir Stoyanov (VMware)
2021-06-22  1:27   ` Steven Rostedt
2021-06-22  6:41     ` Tzvetomir Stoyanov
2021-06-22 13:21       ` Steven Rostedt
2021-06-10 11:33 ` [PATCH v5 07/40] trace-cmd library: Add support for compression algorithms Tzvetomir Stoyanov (VMware)
2021-06-10 11:33 ` [PATCH v5 08/40] trace-cmd list: Show supported " Tzvetomir Stoyanov (VMware)
2021-06-10 11:33 ` [PATCH v5 09/40] trace-cmd library: Bump the trace file version to 7 Tzvetomir Stoyanov (VMware)
2021-06-10 11:33 ` [PATCH v5 10/40] trace-cmd library: Compress part of the trace file Tzvetomir Stoyanov (VMware)
2021-06-10 11:33 ` [PATCH v5 11/40] trace-cmd library: Read compressed " Tzvetomir Stoyanov (VMware)
2021-06-10 11:33 ` [PATCH v5 12/40] trace-cmd library: Add new API to get compression of input handler Tzvetomir Stoyanov (VMware)
2021-06-10 11:33 ` [PATCH v5 13/40] trace-cmd library: Inherit compression algorithm from input file Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 14/40] trace-cmd library: Extend the create file APIs to support different compression Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 15/40] trace-cmd record: Add new parameter --compression Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 16/40] trace-cmd dump: Add support for trace files version 7 Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 17/40] trace-cmd library: Add support for zlib compression library Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 18/40] trace-cmd library: Hide the logic for updating buffer offset Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 19/40] trace-cmd: Move buffers description outside of options Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 20/40] trace-cmd library: Track the offset in the option section in the trace file Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 21/40] trace-cmd library: Add compression of the option section of " Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 22/40] trace-cmd library: Refactor the logic for writing trace data in the file Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 23/40] trace-cmd library: Add APIs for read and write compressed data in chunks Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` Tzvetomir Stoyanov (VMware) [this message]
2021-06-10 11:34 ` [PATCH v5 25/40] trace-cmd: Read compressed trace data Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 26/40] trace-cmd library: Compress latency " Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 27/40] trace-cmd: Read compressed " Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 28/40] trace-cmd library: Reuse within the library the function that checks file state Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 29/40] trace-cmd library: Make tracecmd_copy_headers() to work with output handler Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 30/40] trace-cmd: Do not use trace file compression with streams Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 31/40] trace-cmd library: Add new API to get file version of output handler Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 32/40] trace-cmd: Add file state parameter to tracecmd_copy Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 33/40] trace-cmd: Copy CPU count in tracecmd_copy Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 34/40] trace-cmd: Copy buffers description " Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 35/40] trace-cmd: Copy options " Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 36/40] trace-cmd library: Refactor the logic for writing CPU trace data Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 37/40] trace-cmd library: Refactor the logic for writing CPU instance " Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 38/40] trace-cmd: Copy trace data in tracecmd_copy Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 39/40] trace-cmd: Add compression parameter to tracecmd_copy Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 40/40] trace-cmd: Add new command "trace-cmd convert" Tzvetomir Stoyanov (VMware)

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20210610113426.257931-25-tz.stoyanov@gmail.com \
    --to=tz.stoyanov@gmail.com \
    --cc=linux-trace-devel@vger.kernel.org \
    --cc=rostedt@goodmis.org \
    --subject='Re: [PATCH v5 24/40] trace-cmd: Compress trace data' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).