linux-trace-devel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: "Tzvetomir Stoyanov (VMware)" <tz.stoyanov@gmail.com>
To: rostedt@goodmis.org
Cc: linux-trace-devel@vger.kernel.org
Subject: [PATCH v5 23/40] trace-cmd library: Add APIs for read and write compressed data in chunks
Date: Thu, 10 Jun 2021 14:34:09 +0300	[thread overview]
Message-ID: <20210610113426.257931-24-tz.stoyanov@gmail.com> (raw)
In-Reply-To: <20210610113426.257931-1-tz.stoyanov@gmail.com>

New compression APIs are added for writing and reading compressed data
from / to files in chunks:
 tracecmd_compress_copy_from()
 tracecmd_uncompress_copy_to()
Format of the compressed data, used by these APIs, is:
 - 4 bytes, chunks count
   - for each chunk:
     - 4 bytes, size of compressed data in this chunk
     - 4 bytes, uncompressed size of the data in this chunk
     - data, bytes of <size of compressed data in this chunk>

Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
 .../include/private/trace-cmd-private.h       |   5 +-
 lib/trace-cmd/trace-compress.c                | 200 ++++++++++++++++++
 2 files changed, 204 insertions(+), 1 deletion(-)

diff --git a/lib/trace-cmd/include/private/trace-cmd-private.h b/lib/trace-cmd/include/private/trace-cmd-private.h
index 10089389..067ba34d 100644
--- a/lib/trace-cmd/include/private/trace-cmd-private.h
+++ b/lib/trace-cmd/include/private/trace-cmd-private.h
@@ -494,7 +494,10 @@ int tracecmd_compress_proto_register(const char *name, const char *version, int
 						       char *out, unsigned int *out_bytes),
 				     unsigned int (*comress_size)(unsigned int bytes),
 				     bool (*is_supported)(const char *name, const char *version));
-
+int tracecmd_compress_copy_from(struct tracecmd_compression *handle, int fd, int chunk_size,
+				unsigned long long *read_size, unsigned long long *write_size);
+int tracecmd_uncompress_copy_to(struct tracecmd_compression *handle, int fd,
+				unsigned long long *read_size, unsigned long long *write_size);
 /* --- Plugin handling --- */
 extern struct tep_plugin_option trace_ftrace_options[];
 
diff --git a/lib/trace-cmd/trace-compress.c b/lib/trace-cmd/trace-compress.c
index 2b329d00..4a4707c2 100644
--- a/lib/trace-cmd/trace-compress.c
+++ b/lib/trace-cmd/trace-compress.c
@@ -594,3 +594,203 @@ error:
 	free(v);
 	return -1;
 }
+
+/**
+ * tracecmd_compress_copy_from - Copy and compress data from a file
+ * @handle: compression handler
+ * @fd: file descriptor to uncompressed data
+ * @chunk_size: size of one compression chunk
+ * @read_size: in - max bytes to read from @fd, 0 to read till the EOF
+ *	       out - size of the uncompressed data read from @fd
+ * @write_size: return, size of the compressed data written into @handle
+ *
+ * This function reads uncompressed data from given @fd until EOF is reached, compresses
+ * the data using the @handle compression context and writes the compressed data into the fd
+ * associated with the @handle. The data is compressed on chunks with given @chunk_size size.
+ * The compressed data is written in the format:
+ *  - 4 bytes, chunks count
+ *  - for each chunk:
+ *    - 4 bytes, size of compressed data in this chunk
+ *    - 4 bytes, uncompressed size of the data in this chunk
+ *    - data, bytes of <size of compressed data in this chunk>
+ *
+ * On success 0 is returned, @read_size and @write_size are updated with the size of
+ * read and written data.
+ */
+int tracecmd_compress_copy_from(struct tracecmd_compression *handle, int fd, int chunk_size,
+				unsigned long long *read_size, unsigned long long *write_size)
+{
+	unsigned int chunks = 0;
+	unsigned int rsize = 0;
+	unsigned int rmax = 0;
+	unsigned int wsize = 0;
+	unsigned int csize;
+	unsigned int rchunk = 0;
+	unsigned int size;
+	unsigned int r;
+	off64_t offset;
+	char *buf_from;
+	char *buf_to;
+	int endian4;
+	int ret;
+
+	if (!handle || !handle->proto ||
+	    !handle->proto->comress_block || !handle->proto->comress_size)
+		return 0;
+	if (read_size)
+		rmax = *read_size;
+	csize = handle->proto->comress_size(chunk_size);
+	buf_from = malloc(chunk_size);
+	if (!buf_from)
+		return -1;
+	buf_to = malloc(csize);
+	if (!buf_to)
+		return -1;
+	/* save the initial offset and write 0 chunks */
+	offset = lseek64(handle->fd, 0, SEEK_CUR);
+	write_fd(handle->fd, &chunks, 4);
+
+	do {
+		if (rmax > 0 && (rmax - rsize) < chunk_size)
+			rchunk = (rmax - rsize);
+		else
+			rchunk = chunk_size;
+
+		r = read(fd, buf_from, rchunk);
+		if (r < 0 || (rmax > 0 && rsize >= rmax))
+			break;
+		rsize += r;
+		size = csize;
+		if (r > 0) {
+			ret = handle->proto->comress_block(buf_from, r, buf_to, &size);
+			if (ret < 0) {
+				if (errno == EINTR)
+					continue;
+				break;
+			}
+			/* Write compressed data size */
+			endian4 = tep_read_number(handle->tep, &size, 4);
+			ret = write_fd(handle->fd, &endian4, 4);
+			if (ret != 4)
+				break;
+			/* Write uncompressed data size */
+			endian4 = tep_read_number(handle->tep, &r, 4);
+			ret = write_fd(handle->fd, &endian4, 4);
+			if (ret != 4)
+				break;
+			/* Write the compressed data */
+			ret = write_fd(handle->fd, buf_to, size);
+			if (ret != size)
+				break;
+			/* data + compress header */
+			wsize += (size + 8);
+			chunks++;
+		}
+	} while (r > 0);
+	free(buf_from);
+	free(buf_to);
+	if (r)
+		return -1;
+	if (lseek64(handle->fd, offset, SEEK_SET) == (off_t)-1)
+		return -1;
+	endian4 = tep_read_number(handle->tep, &chunks, 4);
+	/* write chunks count*/
+	write_fd(handle->fd, &chunks, 4);
+	lseek64(handle->fd, offset, SEEK_SET);
+	if (lseek64(handle->fd, 0, SEEK_END) == (off_t)-1)
+		return -1;
+	if (read_size)
+		*read_size = rsize;
+	if (write_size)
+		*write_size = wsize;
+	return 0;
+}
+
+/**
+ * tracecmd_uncompress_copy_to - Uncompress data and copy to a file
+ * @handle: compression handler
+ * @fd: file descriptor to uncompressed data
+ * @read_size: return, size of the compressed data read from @handle
+ * @write_size: return, size of the uncompressed data written into @fd
+ *
+ * This function reads compressed data from the fd, associated with @handle, uncompresses it
+ * using the @handle compression context and writes the uncompressed data into the fd.
+ * The compressed data must be in the format:
+ *  - 4 bytes, chunks count
+ *  - for each chunk:
+ *    - 4 bytes, size of compressed data in this chunk
+ *    - 4 bytes, uncompressed size of the data in this chunk
+ *    - data, bytes of <size of compressed data in this chunk>
+ *
+ * On success 0 is returned, @read_size and @write_size are updated with the size of
+ * read and written data.
+ */
+int tracecmd_uncompress_copy_to(struct tracecmd_compression *handle, int fd,
+				unsigned long long *read_size, unsigned long long *write_size)
+{
+	unsigned int s_uncompressed;
+	unsigned int s_compressed;
+	unsigned int rsize = 0;
+	unsigned int wsize = 0;
+	char *bytes_out = NULL;
+	char *bytes_in = NULL;
+	int size_out;
+	int size_in;
+	int chunks;
+	char buf[4];
+	char *tmp;
+	int ret;
+
+	if (!handle || !handle->proto || !handle->proto->uncompress_block)
+		return -1;
+
+	if (read(handle->fd, buf, 4) != 4)
+		return -1;
+	chunks = tep_read_number(handle->tep, buf, 4);
+	rsize += 4;
+	while (chunks) {
+		if (read(handle->fd, buf, 4) != 4)
+			break;
+		s_compressed = tep_read_number(handle->tep, buf, 4);
+		rsize += 4;
+		if (read(handle->fd, buf, 4) != 4)
+			break;
+		s_uncompressed = tep_read_number(handle->tep, buf, 4);
+		rsize += 4;
+		if (!bytes_in || size_in < s_compressed) {
+			tmp = realloc(bytes_in, s_compressed);
+			if (!tmp)
+				break;
+			bytes_in = tmp;
+			size_in = s_compressed;
+		}
+
+		if (!bytes_out || size_out < s_uncompressed) {
+			tmp = realloc(bytes_out, s_uncompressed);
+			if (!tmp)
+				break;
+			bytes_out = tmp;
+			size_out = s_uncompressed;
+		}
+
+		if (read_fd(handle->fd, bytes_in, s_compressed) < 0)
+			break;
+		rsize += s_compressed;
+		ret = handle->proto->uncompress_block(bytes_in, s_compressed,
+						      bytes_out, &s_uncompressed);
+		if (ret)
+			break;
+		write_fd(fd, bytes_out, s_uncompressed);
+		wsize += s_uncompressed;
+		chunks--;
+	}
+	free(bytes_in);
+	free(bytes_out);
+	if (chunks)
+		return -1;
+	if (read_size)
+		*read_size = rsize;
+	if (write_size)
+		*write_size = wsize;
+	return 0;
+}
-- 
2.31.1


  parent reply	other threads:[~2021-06-10 11:35 UTC|newest]

Thread overview: 44+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-06-10 11:33 [PATCH v5 00/40] Add trace file compression Tzvetomir Stoyanov (VMware)
2021-06-10 11:33 ` [PATCH v5 01/40] trace-cmd library: Remove unused private APIs for creating trace files Tzvetomir Stoyanov (VMware)
2021-06-10 11:33 ` [PATCH v5 02/40] trace-cmd library: Remove unused API tracecmd_update_option Tzvetomir Stoyanov (VMware)
2021-06-10 11:33 ` [PATCH v5 03/40] trace-cmd: Check if file version is supported Tzvetomir Stoyanov (VMware)
2021-06-10 11:33 ` [PATCH v5 04/40] trace-cmd library: Add new API to get file version of input handler Tzvetomir Stoyanov (VMware)
2021-06-10 11:33 ` [PATCH v5 05/40] trace-cmd library: Select the file version when writing trace file Tzvetomir Stoyanov (VMware)
2021-06-10 11:33 ` [PATCH v5 06/40] trace-cmd: Add APIs for library initialization and free Tzvetomir Stoyanov (VMware)
2021-06-22  1:27   ` Steven Rostedt
2021-06-22  6:41     ` Tzvetomir Stoyanov
2021-06-22 13:21       ` Steven Rostedt
2021-06-10 11:33 ` [PATCH v5 07/40] trace-cmd library: Add support for compression algorithms Tzvetomir Stoyanov (VMware)
2021-06-10 11:33 ` [PATCH v5 08/40] trace-cmd list: Show supported " Tzvetomir Stoyanov (VMware)
2021-06-10 11:33 ` [PATCH v5 09/40] trace-cmd library: Bump the trace file version to 7 Tzvetomir Stoyanov (VMware)
2021-06-10 11:33 ` [PATCH v5 10/40] trace-cmd library: Compress part of the trace file Tzvetomir Stoyanov (VMware)
2021-06-10 11:33 ` [PATCH v5 11/40] trace-cmd library: Read compressed " Tzvetomir Stoyanov (VMware)
2021-06-10 11:33 ` [PATCH v5 12/40] trace-cmd library: Add new API to get compression of input handler Tzvetomir Stoyanov (VMware)
2021-06-10 11:33 ` [PATCH v5 13/40] trace-cmd library: Inherit compression algorithm from input file Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 14/40] trace-cmd library: Extend the create file APIs to support different compression Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 15/40] trace-cmd record: Add new parameter --compression Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 16/40] trace-cmd dump: Add support for trace files version 7 Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 17/40] trace-cmd library: Add support for zlib compression library Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 18/40] trace-cmd library: Hide the logic for updating buffer offset Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 19/40] trace-cmd: Move buffers description outside of options Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 20/40] trace-cmd library: Track the offset in the option section in the trace file Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 21/40] trace-cmd library: Add compression of the option section of " Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 22/40] trace-cmd library: Refactor the logic for writing trace data in the file Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` Tzvetomir Stoyanov (VMware) [this message]
2021-06-10 11:34 ` [PATCH v5 24/40] trace-cmd: Compress trace data Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 25/40] trace-cmd: Read compressed " Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 26/40] trace-cmd library: Compress latency " Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 27/40] trace-cmd: Read compressed " Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 28/40] trace-cmd library: Reuse within the library the function that checks file state Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 29/40] trace-cmd library: Make tracecmd_copy_headers() to work with output handler Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 30/40] trace-cmd: Do not use trace file compression with streams Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 31/40] trace-cmd library: Add new API to get file version of output handler Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 32/40] trace-cmd: Add file state parameter to tracecmd_copy Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 33/40] trace-cmd: Copy CPU count in tracecmd_copy Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 34/40] trace-cmd: Copy buffers description " Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 35/40] trace-cmd: Copy options " Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 36/40] trace-cmd library: Refactor the logic for writing CPU trace data Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 37/40] trace-cmd library: Refactor the logic for writing CPU instance " Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 38/40] trace-cmd: Copy trace data in tracecmd_copy Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 39/40] trace-cmd: Add compression parameter to tracecmd_copy Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 40/40] trace-cmd: Add new command "trace-cmd convert" Tzvetomir Stoyanov (VMware)

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20210610113426.257931-24-tz.stoyanov@gmail.com \
    --to=tz.stoyanov@gmail.com \
    --cc=linux-trace-devel@vger.kernel.org \
    --cc=rostedt@goodmis.org \
    --subject='Re: [PATCH v5 23/40] trace-cmd library: Add APIs for read and write compressed data in chunks' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).