linux-trace-devel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: "Tzvetomir Stoyanov (VMware)" <tz.stoyanov@gmail.com>
To: rostedt@goodmis.org
Cc: linux-trace-devel@vger.kernel.org
Subject: [PATCH v4 23/29] trace-cmd library: Add APIs for read and write compressed data in chunks
Date: Thu, 20 May 2021 06:19:53 +0300	[thread overview]
Message-ID: <20210520031959.346165-24-tz.stoyanov@gmail.com> (raw)
In-Reply-To: <20210520031959.346165-1-tz.stoyanov@gmail.com>

New compression APIs are added for writing and reading compressed data
from / to files in chunks:
 tracecmd_compress_copy_from()
 tracecmd_uncompress_copy_to()
Format of the compressed data, used by these APIs, is:
 - 4 bytes, chunks count
   - for each chunk:
     - 4 bytes, size of compressed data in this chunk
     - 4 bytes, uncompressed size of the data in this chunk
     - data, bytes of <size of compressed data in this chunk>

Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
 .../include/private/trace-cmd-private.h       |   5 +-
 lib/trace-cmd/trace-compress.c                | 191 ++++++++++++++++++
 2 files changed, 195 insertions(+), 1 deletion(-)

diff --git a/lib/trace-cmd/include/private/trace-cmd-private.h b/lib/trace-cmd/include/private/trace-cmd-private.h
index 2c28ee04..19dc9e05 100644
--- a/lib/trace-cmd/include/private/trace-cmd-private.h
+++ b/lib/trace-cmd/include/private/trace-cmd-private.h
@@ -493,7 +493,10 @@ int tracecmd_compress_proto_register(const char *name, const char *version, int
 						       char *out, unsigned int *out_bytes),
 				     unsigned int (*comress_size)(unsigned int bytes),
 				     bool (*is_supported)(const char *name, const char *version));
-
+int tracecmd_compress_copy_from(struct tracecmd_compression *handle, int fd, int chunk_size,
+				unsigned long long *read_size, unsigned long long *write_size);
+int tracecmd_uncompress_copy_to(struct tracecmd_compression *handle, int fd,
+				unsigned long long *read_size, unsigned long long *write_size);
 /* --- Plugin handling --- */
 extern struct tep_plugin_option trace_ftrace_options[];
 
diff --git a/lib/trace-cmd/trace-compress.c b/lib/trace-cmd/trace-compress.c
index 2b329d00..3a7accc2 100644
--- a/lib/trace-cmd/trace-compress.c
+++ b/lib/trace-cmd/trace-compress.c
@@ -594,3 +594,194 @@ error:
 	free(v);
 	return -1;
 }
+
+/**
+ * tracecmd_compress_copy_from - Copy and compress data from a file
+ * @handle: compression handler
+ * @fd: file descriptor to uncompressed data
+ * @chunk_size: size of one compression chunk
+ * @read_size: return, size of the uncompressed data read from @fd
+ * @write_size: return, size of the compressed data written into @handle
+ *
+ * This function reads uncompressed data from given @fd until EOF is reached, compresses
+ * the data using the @handle compression context and writes the compressed data into the fd
+ * associated with the @handle. The data is compressed on chunks with given @chunk_size size.
+ * The compressed data is written in the format:
+ *  - 4 bytes, chunks count
+ *  - for each chunk:
+ *    - 4 bytes, size of compressed data in this chunk
+ *    - 4 bytes, uncompressed size of the data in this chunk
+ *    - data, bytes of <size of compressed data in this chunk>
+ *
+ * On success 0 is returned, @read_size and @write_size are updated with the size of
+ * read and written data.
+ */
+int tracecmd_compress_copy_from(struct tracecmd_compression *handle, int fd, int chunk_size,
+				unsigned long long *read_size, unsigned long long *write_size)
+{
+	unsigned int chunks = 0;
+	unsigned int rsize = 0;
+	unsigned int wsize = 0;
+	unsigned int csize;
+	unsigned int size;
+	unsigned int r;
+	off64_t offset;
+	char *buf_from;
+	char *buf_to;
+	int endian4;
+	int ret;
+
+	if (!handle || !handle->proto ||
+	    !handle->proto->comress_block || !handle->proto->comress_size)
+		return 0;
+
+	csize = handle->proto->comress_size(chunk_size);
+	buf_from = malloc(chunk_size);
+	if (!buf_from)
+		return -1;
+	buf_to = malloc(csize);
+	if (!buf_to)
+		return -1;
+	/* save the initial offset and write 0 chunks */
+	offset = lseek64(handle->fd, 0, SEEK_CUR);
+	write_fd(handle->fd, &chunks, 4);
+
+	do {
+		r = read(fd, buf_from, chunk_size);
+		if (r < 0)
+			break;
+		rsize += r;
+		size = csize;
+		if (r > 0) {
+			ret = handle->proto->comress_block(buf_from, r, buf_to, &size);
+			if (ret < 0) {
+				if (errno == EINTR)
+					continue;
+				break;
+			}
+			/* Write compressed data size */
+			endian4 = tep_read_number(handle->tep, &size, 4);
+			ret = write_fd(handle->fd, &endian4, 4);
+			if (ret != 4)
+				break;
+			/* Write uncompressed data size */
+			endian4 = tep_read_number(handle->tep, &r, 4);
+			ret = write_fd(handle->fd, &endian4, 4);
+			if (ret != 4)
+				break;
+			/* Write the compressed data */
+			ret = write_fd(handle->fd, buf_to, size);
+			if (ret != size)
+				break;
+			/* data + compress header */
+			wsize += (size + 8);
+			chunks++;
+		}
+	} while (r > 0);
+	free(buf_from);
+	free(buf_to);
+	if (r)
+		return -1;
+	if (lseek64(handle->fd, offset, SEEK_SET) == (off_t)-1)
+		return -1;
+	endian4 = tep_read_number(handle->tep, &chunks, 4);
+	/* write chunks count*/
+	write_fd(handle->fd, &chunks, 4);
+	lseek64(handle->fd, offset, SEEK_SET);
+	if (lseek64(handle->fd, 0, SEEK_END) == (off_t)-1)
+		return -1;
+	if (read_size)
+		*read_size = rsize;
+	if (write_size)
+		*write_size = wsize;
+	return 0;
+}
+
+/**
+ * tracecmd_uncompress_copy_to - Uncompress data and copy to a file
+ * @handle: compression handler
+ * @fd: file descriptor to uncompressed data
+ * @read_size: return, size of the compressed data read from @handle
+ * @write_size: return, size of the uncompressed data written into @fd
+ *
+ * This function reads compressed data from the fd, associated with @handle, uncompresses it
+ * using the @handle compression context and writes the uncompressed data into the fd.
+ * The compressed data must be in the format:
+ *  - 4 bytes, chunks count
+ *  - for each chunk:
+ *    - 4 bytes, size of compressed data in this chunk
+ *    - 4 bytes, uncompressed size of the data in this chunk
+ *    - data, bytes of <size of compressed data in this chunk>
+ *
+ * On success 0 is returned, @read_size and @write_size are updated with the size of
+ * read and written data.
+ */
+int tracecmd_uncompress_copy_to(struct tracecmd_compression *handle, int fd,
+				unsigned long long *read_size, unsigned long long *write_size)
+{
+	unsigned int s_uncompressed;
+	unsigned int s_compressed;
+	unsigned int rsize = 0;
+	unsigned int wsize = 0;
+	char *bytes_out = NULL;
+	char *bytes_in = NULL;
+	int size_out;
+	int size_in;
+	int chunks;
+	char buf[4];
+	char *tmp;
+	int ret;
+
+	if (!handle || !handle->proto || !handle->proto->uncompress_block)
+		return -1;
+
+	if (read(handle->fd, buf, 4) != 4)
+		return -1;
+	chunks = tep_read_number(handle->tep, buf, 4);
+	rsize += 4;
+	while (chunks) {
+		if (read(handle->fd, buf, 4) != 4)
+			break;
+		s_compressed = tep_read_number(handle->tep, buf, 4);
+		rsize += 4;
+		if (read(handle->fd, buf, 4) != 4)
+			break;
+		s_uncompressed = tep_read_number(handle->tep, buf, 4);
+		rsize += 4;
+		if (!bytes_in || size_in < s_compressed) {
+			tmp = realloc(bytes_in, s_compressed);
+			if (!tmp)
+				break;
+			bytes_in = tmp;
+			size_in = s_compressed;
+		}
+
+		if (!bytes_out || size_out < s_uncompressed) {
+			tmp = realloc(bytes_out, s_uncompressed);
+			if (!tmp)
+				break;
+			bytes_out = tmp;
+			size_out = s_uncompressed;
+		}
+
+		if (read_fd(handle->fd, bytes_in, s_compressed) < 0)
+			break;
+		rsize += s_compressed;
+		ret = handle->proto->uncompress_block(bytes_in, s_compressed,
+						      bytes_out, &s_uncompressed);
+		if (ret)
+			break;
+		write_fd(fd, bytes_out, s_uncompressed);
+		wsize += s_uncompressed;
+		chunks--;
+	}
+	free(bytes_in);
+	free(bytes_out);
+	if (chunks)
+		return -1;
+	if (read_size)
+		*read_size = rsize;
+	if (write_size)
+		*write_size = wsize;
+	return 0;
+}
-- 
2.31.1


  parent reply	other threads:[~2021-05-20  3:20 UTC|newest]

Thread overview: 30+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-05-20  3:19 [PATCH v4 00/29] Add trace file compression Tzvetomir Stoyanov (VMware)
2021-05-20  3:19 ` [PATCH v4 01/29] trace-cmd library: Remove unused private APIs for creating trace files Tzvetomir Stoyanov (VMware)
2021-05-20  3:19 ` [PATCH v4 02/29] trace-cmd library: Remove unused API tracecmd_update_option Tzvetomir Stoyanov (VMware)
2021-05-20  3:19 ` [PATCH v4 03/29] trace-cmd: Check if file version is supported Tzvetomir Stoyanov (VMware)
2021-05-20  3:19 ` [PATCH v4 04/29] trace-cmd library: Add new API to get file version of input handler Tzvetomir Stoyanov (VMware)
2021-05-20  3:19 ` [PATCH v4 05/29] trace-cmd library: Select the file version when writing trace file Tzvetomir Stoyanov (VMware)
2021-05-20  3:19 ` [PATCH v4 06/29] trace-cmd: Add APIs for library initialization and free Tzvetomir Stoyanov (VMware)
2021-05-20  3:19 ` [PATCH v4 07/29] trace-cmd library: Add support for compression algorithms Tzvetomir Stoyanov (VMware)
2021-05-20  3:19 ` [PATCH v4 08/29] trace-cmd list: Show supported " Tzvetomir Stoyanov (VMware)
2021-05-20  3:19 ` [PATCH v4 09/29] trace-cmd library: Bump the trace file version to 7 Tzvetomir Stoyanov (VMware)
2021-05-20  3:19 ` [PATCH v4 10/29] trace-cmd library: Compress part of the trace file Tzvetomir Stoyanov (VMware)
2021-05-20  3:19 ` [PATCH v4 11/29] trace-cmd library: Read compressed " Tzvetomir Stoyanov (VMware)
2021-05-20  3:19 ` [PATCH v4 12/29] trace-cmd library: Add new API to get compression of input handler Tzvetomir Stoyanov (VMware)
2021-05-20  3:19 ` [PATCH v4 13/29] trace-cmd library: Inherit compression algorithm from input file Tzvetomir Stoyanov (VMware)
2021-05-20  3:19 ` [PATCH v4 14/29] trace-cmd library: Extend the create file APIs to support different compression Tzvetomir Stoyanov (VMware)
2021-05-20  3:19 ` [PATCH v4 15/29] trace-cmd record: Add new parameter --compression Tzvetomir Stoyanov (VMware)
2021-05-20  3:19 ` [PATCH v4 16/29] trace-cmd dump: Add support for trace files version 7 Tzvetomir Stoyanov (VMware)
2021-05-20  3:19 ` [PATCH v4 17/29] trace-cmd library: Add support for zlib compression library Tzvetomir Stoyanov (VMware)
2021-05-20  3:19 ` [PATCH v4 18/29] trace-cmd library: Hide the logic for updating buffer offset Tzvetomir Stoyanov (VMware)
2021-05-20  3:19 ` [PATCH v4 19/29] trace-cmd: Move buffers description outside of options Tzvetomir Stoyanov (VMware)
2021-05-20  3:19 ` [PATCH v4 20/29] trace-cmd library: Track the offset in the option section in the trace file Tzvetomir Stoyanov (VMware)
2021-05-20  3:19 ` [PATCH v4 21/29] trace-cmd library: Add compression of the option section of " Tzvetomir Stoyanov (VMware)
2021-05-20  3:19 ` [PATCH v4 22/29] trace-cmd library: Refactor the logic for writing trace data in the file Tzvetomir Stoyanov (VMware)
2021-05-20  3:19 ` Tzvetomir Stoyanov (VMware) [this message]
2021-05-20  3:19 ` [PATCH v4 24/29] trace-cmd: Compress trace data Tzvetomir Stoyanov (VMware)
2021-05-20  3:19 ` [PATCH v4 25/29] trace-cmd: Read compressed " Tzvetomir Stoyanov (VMware)
2021-05-20  3:19 ` [PATCH v4 26/29] trace-cmd library: Reuse within the library the function that checks file state Tzvetomir Stoyanov (VMware)
2021-05-20  3:19 ` [PATCH v4 27/29] trace-cmd library: New internal API to set file state of output handler Tzvetomir Stoyanov (VMware)
2021-05-20  3:19 ` [PATCH v4 28/29] trace-cmd library: Make tracecmd_copy_headers() to work with " Tzvetomir Stoyanov (VMware)
2021-05-20  3:19 ` [PATCH v4 29/29] trace-cmd: Do not use trace file compression with streams Tzvetomir Stoyanov (VMware)

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20210520031959.346165-24-tz.stoyanov@gmail.com \
    --to=tz.stoyanov@gmail.com \
    --cc=linux-trace-devel@vger.kernel.org \
    --cc=rostedt@goodmis.org \
    --subject='Re: [PATCH v4 23/29] trace-cmd library: Add APIs for read and write compressed data in chunks' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).