linux-trace-devel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Tzvetomir Stoyanov <tz.stoyanov@gmail.com>
To: Steven Rostedt <rostedt@goodmis.org>
Cc: Linux Trace Devel <linux-trace-devel@vger.kernel.org>
Subject: Re: [PATCH v6 25/45] trace-cmd: Read compressed trace data
Date: Tue, 22 Jun 2021 13:50:44 +0300	[thread overview]
Message-ID: <CAPpZLN4dHKxa2DztSurSow=PbsFSyuLgkifHt8yvJ6kL8pvu8Q@mail.gmail.com> (raw)
In-Reply-To: <20210621192310.43e68a92@gandalf.local.home>

On Tue, Jun 22, 2021 at 2:23 AM Steven Rostedt <rostedt@goodmis.org> wrote:
>
> On Mon, 14 Jun 2021 10:50:09 +0300
> "Tzvetomir Stoyanov (VMware)" <tz.stoyanov@gmail.com> wrote:
>
> > When reading a trace.dat file of version 7, uncompress the trace data.
> > The trace data for each CPU is uncompressed in a temporary file, located
> > in /tmp directory with prefix "trace_cpu_data".
>
> With large trace files, this will be an issue. Several systems setup the
> /tmp directory as a ramfs file system (that is, it is locate in ram, and
> not backed up on disk). If you have very large trace files, which you would
> if you are going to bother compressing them, by uncompressing them into
> /tmp, it could take up all the memory of the machine, or easily fill the
> /tmp limit.

There are a few possible approaches for solving that:
 - use the same directory where the input trace file is located
 - use an environment variable for user specified temp directory for these files
 - check if there is enough free space on the FS before uncompressing

>
> Simply uncompressing the entire trace data is not an option. The best we
> can do is to uncompress on a as needed basis. That would require having
> meta data that is stored to know what pages are compressed.
>
I can modify that logic to compress page by page, as the data is
loaded by pages. Or use some of the above approaches ?

> -- Steve



-- 
Tzvetomir (Ceco) Stoyanov
VMware Open Source Technology Center

  reply	other threads:[~2021-06-22 10:51 UTC|newest]

Thread overview: 64+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-06-14  7:49 [PATCH v6 00/45] Add trace file compression Tzvetomir Stoyanov (VMware)
2021-06-14  7:49 ` [PATCH v6 01/45] trace-cmd library: Remove unused private APIs for creating trace files Tzvetomir Stoyanov (VMware)
2021-06-14  7:49 ` [PATCH v6 02/45] trace-cmd library: Remove unused API tracecmd_update_option Tzvetomir Stoyanov (VMware)
2021-06-14  7:49 ` [PATCH v6 03/45] trace-cmd: Check if file version is supported Tzvetomir Stoyanov (VMware)
2021-06-21 22:27   ` Steven Rostedt
2021-06-21 22:36     ` Steven Rostedt
2021-06-14  7:49 ` [PATCH v6 04/45] trace-cmd library: Add new API to get file version of input handler Tzvetomir Stoyanov (VMware)
2021-06-14  7:49 ` [PATCH v6 05/45] trace-cmd library: Select the file version when writing trace file Tzvetomir Stoyanov (VMware)
2021-06-14  7:49 ` [PATCH v6 06/45] trace-cmd: Add APIs for library initialization and free Tzvetomir Stoyanov (VMware)
2021-06-14  7:49 ` [PATCH v6 07/45] trace-cmd library: Add support for compression algorithms Tzvetomir Stoyanov (VMware)
2021-06-14  7:49 ` [PATCH v6 08/45] trace-cmd list: Show supported " Tzvetomir Stoyanov (VMware)
2021-06-14  7:49 ` [PATCH v6 09/45] trace-cmd library: Bump the trace file version to 7 Tzvetomir Stoyanov (VMware)
2021-06-14  7:49 ` [PATCH v6 10/45] trace-cmd library: Compress part of the trace file Tzvetomir Stoyanov (VMware)
2021-06-14  7:49 ` [PATCH v6 11/45] trace-cmd library: Read compressed " Tzvetomir Stoyanov (VMware)
2021-06-14  7:49 ` [PATCH v6 12/45] trace-cmd library: Add new API to get compression of input handler Tzvetomir Stoyanov (VMware)
2021-06-14  7:49 ` [PATCH v6 13/45] trace-cmd library: Inherit compression algorithm from input file Tzvetomir Stoyanov (VMware)
2021-06-14  7:49 ` [PATCH v6 14/45] trace-cmd library: Extend the create file APIs to support different compression Tzvetomir Stoyanov (VMware)
2021-06-14  7:49 ` [PATCH v6 15/45] trace-cmd record: Add new parameter --compression Tzvetomir Stoyanov (VMware)
2021-06-14  7:50 ` [PATCH v6 16/45] trace-cmd dump: Add support for trace files version 7 Tzvetomir Stoyanov (VMware)
2021-06-14  7:50 ` [PATCH v6 17/45] trace-cmd library: Add support for zlib compression library Tzvetomir Stoyanov (VMware)
2021-06-22  1:26   ` Steven Rostedt
2021-06-22 10:29     ` Tzvetomir Stoyanov
2021-06-22 13:31       ` Steven Rostedt
2021-06-14  7:50 ` [PATCH v6 18/45] trace-cmd library: Hide the logic for updating buffer offset Tzvetomir Stoyanov (VMware)
2021-06-22  2:10   ` Steven Rostedt
2021-06-14  7:50 ` [PATCH v6 19/45] trace-cmd: Move buffers description outside of options Tzvetomir Stoyanov (VMware)
2021-06-21 23:07   ` Steven Rostedt
2021-06-14  7:50 ` [PATCH v6 20/45] trace-cmd library: Track the offset in the option section in the trace file Tzvetomir Stoyanov (VMware)
2021-06-14  7:50 ` [PATCH v6 21/45] trace-cmd library: Add compression of the option section of " Tzvetomir Stoyanov (VMware)
2021-06-21 23:10   ` Steven Rostedt
2021-06-22 10:43     ` Tzvetomir Stoyanov
2021-06-14  7:50 ` [PATCH v6 22/45] trace-cmd library: Refactor the logic for writing trace data in the file Tzvetomir Stoyanov (VMware)
2021-06-21 23:12   ` Steven Rostedt
2021-06-14  7:50 ` [PATCH v6 23/45] trace-cmd library: Add APIs for read and write compressed data in chunks Tzvetomir Stoyanov (VMware)
2021-06-14  7:50 ` [PATCH v6 24/45] trace-cmd: Compress trace data Tzvetomir Stoyanov (VMware)
2021-06-14  7:50 ` [PATCH v6 25/45] trace-cmd: Read compressed " Tzvetomir Stoyanov (VMware)
2021-06-21 23:23   ` Steven Rostedt
2021-06-22 10:50     ` Tzvetomir Stoyanov [this message]
2021-06-22 13:51       ` Steven Rostedt
2021-06-14  7:50 ` [PATCH v6 26/45] trace-cmd library: Compress latency " Tzvetomir Stoyanov (VMware)
2021-06-14  7:50 ` [PATCH v6 27/45] trace-cmd: Read compressed " Tzvetomir Stoyanov (VMware)
2021-06-14  7:50 ` [PATCH v6 28/45] trace-cmd library: Reuse within the library the function that checks file state Tzvetomir Stoyanov (VMware)
2021-06-14  7:50 ` [PATCH v6 29/45] trace-cmd library: Make tracecmd_copy_headers() to work with output handler Tzvetomir Stoyanov (VMware)
2021-06-14  7:50 ` [PATCH v6 30/45] trace-cmd: Do not use trace file compression with streams Tzvetomir Stoyanov (VMware)
2021-06-14  7:50 ` [PATCH v6 31/45] trace-cmd library: Add new API to get file version of output handler Tzvetomir Stoyanov (VMware)
2021-06-14  7:50 ` [PATCH v6 32/45] trace-cmd: Add file state parameter to tracecmd_copy Tzvetomir Stoyanov (VMware)
2021-06-14  7:50 ` [PATCH v6 33/45] trace-cmd: Copy CPU count in tracecmd_copy Tzvetomir Stoyanov (VMware)
2021-06-21 23:27   ` Steven Rostedt
2021-06-14  7:50 ` [PATCH v6 34/45] trace-cmd: Copy buffers description " Tzvetomir Stoyanov (VMware)
2021-06-14  7:50 ` [PATCH v6 35/45] trace-cmd: Copy options " Tzvetomir Stoyanov (VMware)
2021-06-14  7:50 ` [PATCH v6 36/45] trace-cmd library: Refactor the logic for writing CPU trace data Tzvetomir Stoyanov (VMware)
2021-06-14  7:50 ` [PATCH v6 37/45] trace-cmd library: Refactor the logic for writing CPU instance " Tzvetomir Stoyanov (VMware)
2021-06-14  7:50 ` [PATCH v6 38/45] trace-cmd: Copy trace data in tracecmd_copy Tzvetomir Stoyanov (VMware)
2021-06-14  7:50 ` [PATCH v6 39/45] trace-cmd: Add compression parameter to tracecmd_copy Tzvetomir Stoyanov (VMware)
2021-06-14  7:50 ` [PATCH v6 40/45] trace-cmd: Add new command "trace-cmd convert" Tzvetomir Stoyanov (VMware)
2021-06-14  7:50 ` [PATCH v6 41/45] trace-cmd record: Update man page Tzvetomir Stoyanov (VMware)
2021-06-14  7:50 ` [PATCH v6 42/45] trace-cmd: Add convert " Tzvetomir Stoyanov (VMware)
2021-06-14  7:50 ` [PATCH v6 43/45] trace-cmd: Update bash completion Tzvetomir Stoyanov (VMware)
2021-06-14  7:50 ` [PATCH v6 44/45] trace-cmd list: Update the man page Tzvetomir Stoyanov (VMware)
2021-06-14  7:50 ` [PATCH v6 45/45] trace-cmd: Update trace.dat " Tzvetomir Stoyanov (VMware)
2021-06-22  0:37   ` Steven Rostedt
2021-06-22 11:05     ` Tzvetomir Stoyanov
2021-06-22 13:58       ` Steven Rostedt
2021-06-22  2:22 ` [PATCH v6 00/45] Add trace file compression Steven Rostedt

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='CAPpZLN4dHKxa2DztSurSow=PbsFSyuLgkifHt8yvJ6kL8pvu8Q@mail.gmail.com' \
    --to=tz.stoyanov@gmail.com \
    --cc=linux-trace-devel@vger.kernel.org \
    --cc=rostedt@goodmis.org \
    --subject='Re: [PATCH v6 25/45] trace-cmd: Read compressed trace data' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).