From: "Tzvetomir Stoyanov (VMware)" <tz.stoyanov@gmail.com>
To: rostedt@goodmis.org
Cc: linux-trace-devel@vger.kernel.org
Subject: [PATCH v5 00/40] Add trace file compression
Date: Thu, 10 Jun 2021 14:33:46 +0300 [thread overview]
Message-ID: <20210610113426.257931-1-tz.stoyanov@gmail.com> (raw)
A new PoC implementation, adding a compression to trace file.
A basic infrastructure for compression support is added into the trace-cmd
library. The zlib is used for compression, but more libraries and algorithms
can be added. Trace data is commpressed and part of the trace file metadata:
- ftrace events format
- format of recorded events
- information about the mapping of function addresses to the function names
- trace_printk() format strings
- information of the mapping a PID to a process name
- options
Note: not all trace-cmd commands are tested with these changes. These are
verified to work with compressed trace files:
trace-cmd record
trace-cmd report
trace-cmd dump
trace-cmd agent
trace-cmd profile
trace-cmd split
trace-cmd convert
Todo list, in order to have complete trace file compression:
- Add support for more compression libraries.
- Test all trace-cmd commands with trace file v7.
- Update trace-cmd documentation.
v5 changes:
- Comress the latency trace data.
- Add new "trace-cmd convert" subcommand.
- Fixed bug with empty CPUs in the trace file.
- A lot of minor bug fixes.
v4 changes:
- Tested and fixed profile and split subcommands with compression files.
- Bug fixes.
v3 changes:
- Compress the trace data.
- Added documentation to all compression APIs.
- A few minor fixes.
v2 changes:
- Refactored compression APIs.
- Moved the trace buffers description out of the trace options section.
- Added compression of "options" section of the trace file.
- Updated "trace-cmd list" to show available compression algorithms.
- Tested with host-guest tracing.
- Merged with the patchset that bumps the trace file version.
Tzvetomir Stoyanov (VMware) (40):
trace-cmd library: Remove unused private APIs for creating trace files
trace-cmd library: Remove unused API tracecmd_update_option
trace-cmd: Check if file version is supported
trace-cmd library: Add new API to get file version of input handler
trace-cmd library: Select the file version when writing trace file
trace-cmd: Add APIs for library initialization and free
trace-cmd library: Add support for compression algorithms
trace-cmd list: Show supported compression algorithms
trace-cmd library: Bump the trace file version to 7
trace-cmd library: Compress part of the trace file
trace-cmd library: Read compressed trace file
trace-cmd library: Add new API to get compression of input handler
trace-cmd library: Inherit compression algorithm from input file
trace-cmd library: Extend the create file APIs to support different
compression
trace-cmd record: Add new parameter --compression
trace-cmd dump: Add support for trace files version 7
trace-cmd library: Add support for zlib compression library
trace-cmd library: Hide the logic for updating buffer offset
trace-cmd: Move buffers description outside of options
trace-cmd library: Track the offset in the option section in the trace
file
trace-cmd library: Add compression of the option section of the trace
file
trace-cmd library: Refactor the logic for writing trace data in the
file
trace-cmd library: Add APIs for read and write compressed data in
chunks
trace-cmd: Compress trace data
trace-cmd: Read compressed trace data
trace-cmd library: Compress latency trace data
trace-cmd: Read compressed latency trace data
trace-cmd library: Reuse within the library the function that checks
file state.
trace-cmd library: Make tracecmd_copy_headers() to work with output
handler
trace-cmd: Do not use trace file compression with streams
trace-cmd library: Add new API to get file version of output handler
trace-cmd: Add file state parameter to tracecmd_copy
trace-cmd: Copy CPU count in tracecmd_copy
trace-cmd: Copy buffers description in tracecmd_copy
trace-cmd: Copy options in tracecmd_copy
trace-cmd library: Refactor the logic for writing CPU trace data
trace-cmd library: Refactor the logic for writing CPU instance trace
data
trace-cmd: Copy trace data in tracecmd_copy
trace-cmd: Add compression parameter to tracecmd_copy
trace-cmd: Add new command "trace-cmd convert"
lib/trace-cmd/Makefile | 11 +
.../include/private/trace-cmd-private.h | 92 +-
lib/trace-cmd/include/trace-cmd-local.h | 44 +-
lib/trace-cmd/trace-compress-zlib.c | 172 +++
lib/trace-cmd/trace-compress.c | 796 ++++++++++++++
lib/trace-cmd/trace-input.c | 999 ++++++++++++++----
lib/trace-cmd/trace-output.c | 877 ++++++++++-----
lib/trace-cmd/trace-util.c | 60 ++
tracecmd/Makefile | 1 +
tracecmd/include/trace-local.h | 2 +
tracecmd/trace-cmd.c | 12 +-
tracecmd/trace-convert.c | 93 ++
tracecmd/trace-dump.c | 162 ++-
tracecmd/trace-list.c | 26 +
tracecmd/trace-listen.c | 3 +
tracecmd/trace-read.c | 14 +-
tracecmd/trace-record.c | 42 +-
tracecmd/trace-restore.c | 6 +-
tracecmd/trace-split.c | 2 +-
tracecmd/trace-stream.c | 2 +-
tracecmd/trace-usage.c | 18 +
21 files changed, 2905 insertions(+), 529 deletions(-)
create mode 100644 lib/trace-cmd/trace-compress-zlib.c
create mode 100644 lib/trace-cmd/trace-compress.c
create mode 100644 tracecmd/trace-convert.c
--
2.31.1
next reply other threads:[~2021-06-10 11:35 UTC|newest]
Thread overview: 44+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-06-10 11:33 Tzvetomir Stoyanov (VMware) [this message]
2021-06-10 11:33 ` [PATCH v5 01/40] trace-cmd library: Remove unused private APIs for creating trace files Tzvetomir Stoyanov (VMware)
2021-06-10 11:33 ` [PATCH v5 02/40] trace-cmd library: Remove unused API tracecmd_update_option Tzvetomir Stoyanov (VMware)
2021-06-10 11:33 ` [PATCH v5 03/40] trace-cmd: Check if file version is supported Tzvetomir Stoyanov (VMware)
2021-06-10 11:33 ` [PATCH v5 04/40] trace-cmd library: Add new API to get file version of input handler Tzvetomir Stoyanov (VMware)
2021-06-10 11:33 ` [PATCH v5 05/40] trace-cmd library: Select the file version when writing trace file Tzvetomir Stoyanov (VMware)
2021-06-10 11:33 ` [PATCH v5 06/40] trace-cmd: Add APIs for library initialization and free Tzvetomir Stoyanov (VMware)
2021-06-22 1:27 ` Steven Rostedt
2021-06-22 6:41 ` Tzvetomir Stoyanov
2021-06-22 13:21 ` Steven Rostedt
2021-06-10 11:33 ` [PATCH v5 07/40] trace-cmd library: Add support for compression algorithms Tzvetomir Stoyanov (VMware)
2021-06-10 11:33 ` [PATCH v5 08/40] trace-cmd list: Show supported " Tzvetomir Stoyanov (VMware)
2021-06-10 11:33 ` [PATCH v5 09/40] trace-cmd library: Bump the trace file version to 7 Tzvetomir Stoyanov (VMware)
2021-06-10 11:33 ` [PATCH v5 10/40] trace-cmd library: Compress part of the trace file Tzvetomir Stoyanov (VMware)
2021-06-10 11:33 ` [PATCH v5 11/40] trace-cmd library: Read compressed " Tzvetomir Stoyanov (VMware)
2021-06-10 11:33 ` [PATCH v5 12/40] trace-cmd library: Add new API to get compression of input handler Tzvetomir Stoyanov (VMware)
2021-06-10 11:33 ` [PATCH v5 13/40] trace-cmd library: Inherit compression algorithm from input file Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 14/40] trace-cmd library: Extend the create file APIs to support different compression Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 15/40] trace-cmd record: Add new parameter --compression Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 16/40] trace-cmd dump: Add support for trace files version 7 Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 17/40] trace-cmd library: Add support for zlib compression library Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 18/40] trace-cmd library: Hide the logic for updating buffer offset Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 19/40] trace-cmd: Move buffers description outside of options Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 20/40] trace-cmd library: Track the offset in the option section in the trace file Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 21/40] trace-cmd library: Add compression of the option section of " Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 22/40] trace-cmd library: Refactor the logic for writing trace data in the file Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 23/40] trace-cmd library: Add APIs for read and write compressed data in chunks Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 24/40] trace-cmd: Compress trace data Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 25/40] trace-cmd: Read compressed " Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 26/40] trace-cmd library: Compress latency " Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 27/40] trace-cmd: Read compressed " Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 28/40] trace-cmd library: Reuse within the library the function that checks file state Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 29/40] trace-cmd library: Make tracecmd_copy_headers() to work with output handler Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 30/40] trace-cmd: Do not use trace file compression with streams Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 31/40] trace-cmd library: Add new API to get file version of output handler Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 32/40] trace-cmd: Add file state parameter to tracecmd_copy Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 33/40] trace-cmd: Copy CPU count in tracecmd_copy Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 34/40] trace-cmd: Copy buffers description " Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 35/40] trace-cmd: Copy options " Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 36/40] trace-cmd library: Refactor the logic for writing CPU trace data Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 37/40] trace-cmd library: Refactor the logic for writing CPU instance " Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 38/40] trace-cmd: Copy trace data in tracecmd_copy Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 39/40] trace-cmd: Add compression parameter to tracecmd_copy Tzvetomir Stoyanov (VMware)
2021-06-10 11:34 ` [PATCH v5 40/40] trace-cmd: Add new command "trace-cmd convert" Tzvetomir Stoyanov (VMware)
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210610113426.257931-1-tz.stoyanov@gmail.com \
--to=tz.stoyanov@gmail.com \
--cc=linux-trace-devel@vger.kernel.org \
--cc=rostedt@goodmis.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).