linux-trace-devel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: "Tzvetomir Stoyanov (VMware)" <tz.stoyanov@gmail.com>
To: rostedt@goodmis.org
Cc: linux-trace-devel@vger.kernel.org
Subject: [PATCH v4 00/29] Add trace file compression
Date: Thu, 20 May 2021 06:19:30 +0300	[thread overview]
Message-ID: <20210520031959.346165-1-tz.stoyanov@gmail.com> (raw)

A new PoC implementation, adding a compression to trace file.
A basic infrastructure for compression support is added into the trace-cmd
library. The zlib is used for compression, but more libraries and algorithms
can be added. Trace data is commpressed and part of the trace file metadata: 
 - ftrace events format
 - format of recorded events
 - information about the mapping of function addresses to the function names
 - trace_printk() format strings
 - information of the mapping a PID to a process name
 - options

Note: not all trace-cmd commands are tested with these changes. These are
verified to work with compressed trace files:
 trace-cmd record
 trace-cmd report
 trace-cmd dump
 trace-cmd agent
 trace-cmd profile
 trace-cmd split

Todo list, in order to have complete trace file compression:
 - Add support for more compression libraries.
 - Add new trace-cmd command to convert trace files from version 6 to
   version 7 and vise versa.
 - Test all trace-cmd commands with trace file v7.
 - Update trace-cmd documentation.

v4 changes:
 - Tested and fixed profile and split subcommands with compression files.
 - Bug fixes.

v3 changes:
 - Compress the trace data.
 - Added documentation to all compression APIs.
 - A few minor fixes.
v2 changes:
 - Refactored compression APIs.
 - Moved the trace buffers description out of the trace options section.
 - Added compression of "options" section of the trace file.
 - Updated "trace-cmd list" to show available compression algorithms.
 - Tested with host-guest tracing.
 - Merged with the patchset that bumps the trace file version.

Tzvetomir Stoyanov (VMware) (29):
  trace-cmd library: Remove unused private APIs for creating trace files
  trace-cmd library: Remove unused API tracecmd_update_option
  trace-cmd: Check if file version is supported
  trace-cmd library: Add new API to get file version of input handler
  trace-cmd library: Select the file version when writing trace file
  trace-cmd: Add APIs for library initialization and free
  trace-cmd library: Add support for compression algorithms
  trace-cmd list: Show supported compression algorithms
  trace-cmd library: Bump the trace file version to 7
  trace-cmd library: Compress part of the trace file
  trace-cmd library: Read compressed trace file
  trace-cmd library: Add new API to get compression of input handler
  trace-cmd library: Inherit compression algorithm from input file
  trace-cmd library: Extend the create file APIs to support different
    compression
  trace-cmd record: Add new parameter --compression
  trace-cmd dump: Add support for trace files version 7
  trace-cmd library: Add support for zlib compression library
  trace-cmd library: Hide the logic for updating buffer offset
  trace-cmd: Move buffers description outside of options
  trace-cmd library: Track the offset in the option section in the trace
    file
  trace-cmd library: Add compression of the option section of the trace
    file
  trace-cmd library: Refactor the logic for writing trace data in the
    file
  trace-cmd library: Add APIs for read and write compressed data in
    chunks
  trace-cmd: Compress trace data
  trace-cmd: Read compressed trace data
  trace-cmd library: Reuse within the library the function that checks
    file state.
  trace-cmd library: New internal API to set file state of output
    handler
  trace-cmd library: Make tracecmd_copy_headers() to work with output
    handler
  trace-cmd: Do not use trace file compression with streams

 lib/trace-cmd/Makefile                        |  11 +
 .../include/private/trace-cmd-private.h       |  79 +-
 lib/trace-cmd/include/trace-cmd-local.h       |  25 +-
 lib/trace-cmd/trace-compress-zlib.c           | 172 ++++
 lib/trace-cmd/trace-compress.c                | 787 ++++++++++++++++++
 lib/trace-cmd/trace-input.c                   | 711 +++++++++++-----
 lib/trace-cmd/trace-output.c                  | 738 ++++++++++------
 lib/trace-cmd/trace-util.c                    |  60 ++
 tracecmd/trace-cmd.c                          |  11 +-
 tracecmd/trace-dump.c                         | 162 +++-
 tracecmd/trace-list.c                         |  26 +
 tracecmd/trace-listen.c                       |   3 +
 tracecmd/trace-read.c                         |   8 +
 tracecmd/trace-record.c                       |  42 +-
 tracecmd/trace-restore.c                      |   4 +-
 tracecmd/trace-stream.c                       |   2 +-
 tracecmd/trace-usage.c                        |   6 +
 17 files changed, 2342 insertions(+), 505 deletions(-)
 create mode 100644 lib/trace-cmd/trace-compress-zlib.c
 create mode 100644 lib/trace-cmd/trace-compress.c

-- 
2.31.1


             reply	other threads:[~2021-05-20  3:20 UTC|newest]

Thread overview: 30+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-05-20  3:19 Tzvetomir Stoyanov (VMware) [this message]
2021-05-20  3:19 ` [PATCH v4 01/29] trace-cmd library: Remove unused private APIs for creating trace files Tzvetomir Stoyanov (VMware)
2021-05-20  3:19 ` [PATCH v4 02/29] trace-cmd library: Remove unused API tracecmd_update_option Tzvetomir Stoyanov (VMware)
2021-05-20  3:19 ` [PATCH v4 03/29] trace-cmd: Check if file version is supported Tzvetomir Stoyanov (VMware)
2021-05-20  3:19 ` [PATCH v4 04/29] trace-cmd library: Add new API to get file version of input handler Tzvetomir Stoyanov (VMware)
2021-05-20  3:19 ` [PATCH v4 05/29] trace-cmd library: Select the file version when writing trace file Tzvetomir Stoyanov (VMware)
2021-05-20  3:19 ` [PATCH v4 06/29] trace-cmd: Add APIs for library initialization and free Tzvetomir Stoyanov (VMware)
2021-05-20  3:19 ` [PATCH v4 07/29] trace-cmd library: Add support for compression algorithms Tzvetomir Stoyanov (VMware)
2021-05-20  3:19 ` [PATCH v4 08/29] trace-cmd list: Show supported " Tzvetomir Stoyanov (VMware)
2021-05-20  3:19 ` [PATCH v4 09/29] trace-cmd library: Bump the trace file version to 7 Tzvetomir Stoyanov (VMware)
2021-05-20  3:19 ` [PATCH v4 10/29] trace-cmd library: Compress part of the trace file Tzvetomir Stoyanov (VMware)
2021-05-20  3:19 ` [PATCH v4 11/29] trace-cmd library: Read compressed " Tzvetomir Stoyanov (VMware)
2021-05-20  3:19 ` [PATCH v4 12/29] trace-cmd library: Add new API to get compression of input handler Tzvetomir Stoyanov (VMware)
2021-05-20  3:19 ` [PATCH v4 13/29] trace-cmd library: Inherit compression algorithm from input file Tzvetomir Stoyanov (VMware)
2021-05-20  3:19 ` [PATCH v4 14/29] trace-cmd library: Extend the create file APIs to support different compression Tzvetomir Stoyanov (VMware)
2021-05-20  3:19 ` [PATCH v4 15/29] trace-cmd record: Add new parameter --compression Tzvetomir Stoyanov (VMware)
2021-05-20  3:19 ` [PATCH v4 16/29] trace-cmd dump: Add support for trace files version 7 Tzvetomir Stoyanov (VMware)
2021-05-20  3:19 ` [PATCH v4 17/29] trace-cmd library: Add support for zlib compression library Tzvetomir Stoyanov (VMware)
2021-05-20  3:19 ` [PATCH v4 18/29] trace-cmd library: Hide the logic for updating buffer offset Tzvetomir Stoyanov (VMware)
2021-05-20  3:19 ` [PATCH v4 19/29] trace-cmd: Move buffers description outside of options Tzvetomir Stoyanov (VMware)
2021-05-20  3:19 ` [PATCH v4 20/29] trace-cmd library: Track the offset in the option section in the trace file Tzvetomir Stoyanov (VMware)
2021-05-20  3:19 ` [PATCH v4 21/29] trace-cmd library: Add compression of the option section of " Tzvetomir Stoyanov (VMware)
2021-05-20  3:19 ` [PATCH v4 22/29] trace-cmd library: Refactor the logic for writing trace data in the file Tzvetomir Stoyanov (VMware)
2021-05-20  3:19 ` [PATCH v4 23/29] trace-cmd library: Add APIs for read and write compressed data in chunks Tzvetomir Stoyanov (VMware)
2021-05-20  3:19 ` [PATCH v4 24/29] trace-cmd: Compress trace data Tzvetomir Stoyanov (VMware)
2021-05-20  3:19 ` [PATCH v4 25/29] trace-cmd: Read compressed " Tzvetomir Stoyanov (VMware)
2021-05-20  3:19 ` [PATCH v4 26/29] trace-cmd library: Reuse within the library the function that checks file state Tzvetomir Stoyanov (VMware)
2021-05-20  3:19 ` [PATCH v4 27/29] trace-cmd library: New internal API to set file state of output handler Tzvetomir Stoyanov (VMware)
2021-05-20  3:19 ` [PATCH v4 28/29] trace-cmd library: Make tracecmd_copy_headers() to work with " Tzvetomir Stoyanov (VMware)
2021-05-20  3:19 ` [PATCH v4 29/29] trace-cmd: Do not use trace file compression with streams Tzvetomir Stoyanov (VMware)

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20210520031959.346165-1-tz.stoyanov@gmail.com \
    --to=tz.stoyanov@gmail.com \
    --cc=linux-trace-devel@vger.kernel.org \
    --cc=rostedt@goodmis.org \
    --subject='Re: [PATCH v4 00/29] Add trace file compression' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).