Re: [RFC/PATCH 00/14] perf record: Add support to store data in directory

[Jiri Olsa <jolsa@redhat.com>]

On Mon, Feb 04, 2019 at 02:44:37PM -0800, Stephane Eranian wrote:
> Jiri,
> 
> While you're looking at the output format, I think it would be good
> time to simplify the code handling perf.data file.
> Today, perf record can emit in two formats: file mode or pipe mode.
> This adds complexity in the code and
> is error prone as the file mode path is tested more than the pipe mode
> path. We have run into multiple issues with
> the pipe mode in recent years. There is no real reason why we need to
> maintain two formats. If I recall, the pipe format
> was introduced because on pipes you cannot lseek to update the headers
> and therefore some of the information present as tables
> updated on the fly needed to be generated as pseudo records by the
> tool. I believe that the pipe format covers all the needs and could
> supersede the file mode format. That would simplify code in perf
> record and eliminate the risk of errors when new headers
> are introduced.

yep, I think we have almost all the features covered for pipe mode,
and we have all necessary events to describe events features

so with some effort we could switch off the superfluos file header
and use only events to describe events ;-) make sense, I'll check
on it

> 
> Related to your effort to make perf record multi-threaded, I was
> wondering how this would impact pipe mode.
> Could you explain?

there's no support for threaded pipe processing at the moment,
currently the data are stored in directory:

  $ ls -l perf.data
  total 344
  -rw-------. 1 jolsa jolsa 43864 Jan 20 22:26 data.0
  -rw-------. 1 jolsa jolsa 30464 Jan 20 22:26 data.1
  -rw-------. 1 jolsa jolsa 53816 Jan 20 22:26 data.2
  -rw-------. 1 jolsa jolsa 30368 Jan 20 22:26 data.3
  -rw-------. 1 jolsa jolsa 40088 Jan 20 22:26 data.4
  -rw-------. 1 jolsa jolsa 42592 Jan 20 22:26 data.5
  -rw-------. 1 jolsa jolsa 56136 Jan 20 22:26 data.6
  -rw-------. 1 jolsa jolsa 25992 Jan 20 22:26 data.7

  ^^^^ those are raw data files, contains only events with common header
        as stored by perf or kernel

  -rw-------. 1 jolsa jolsa  8832 Jan 20 22:26 header

  ^^^^ this one currently holds perf.data file header, describing events
       and features

if we switched to pipe mode by default we could omit the header file
and find a way to push the data streams through single file, like with
a new event describing the data stream.. we can have an option for that
so you could do something like:

  # perf record --threads --single-file -a ... | perf report -i -

however having single thread storing storing into single file without
any other processing is important on record side (for minimal overhead),
so I think we should keep creating the directory with data files also
for pipe mode to have minimal overhead

jirka