Re: [PATCH v6 0/4] perf: add support for profiling jitted code

[Brendan Gregg <brendan.d.gregg@gmail.com>]

G'Day Stephane,

On Mon, Mar 30, 2015 at 3:19 PM, Stephane Eranian <eranian@google.com> wrote:
[...]
> The current support only works when the runtime is monitored from
> start to finish: perf record java --agentpath:libpfmjvmti.so my_class.
>
> Once the run is completed, the jitdump file needs to be injected into
> the perf.data file. This is accomplished by using the perf inject command.
> This will also generate an ELF image for each jitted function. The
> inject MMAP records will point to those ELF images. The reasoning
> behind using ELF images is that it makes processing for perf report
> and annotate automatic and transparent. It also makes it easier to
> package and analyze on a remote machine.
[...]

This is really impressive work. Do we have an idea of the overhead for
running the java agent?

Today, I'm using perf-map-agent, loaded dynamically, to dump a
/tmp/perf*.map file as needed. My company has tens of thousands of
Linux instances running Java, but very few need profiling, and we
don't know which beforehand. So a snapshot-on-demand approach is
ideal. An always-on approach, well, we'd have to know the overhead (I
can build the agent and test...).

During a profile, our symbols can change, but it's not been a serious
problem. I can take a before and after snapshot of symbols
(/tmp/perf.map files), then highlight which have moved and are
therefore suspicious (not perfect, but ok). Yes, having a timestamped
symbol log would be most accurate, but I wouldn't pay a lot of
overhead for it. Is there a (horror) story behind why this was so
necessary?

Minor comment: I didn't notice an fsync() in the jvmti code. Can
symbol updates be buffered and not written out by the time perf is
reading them?

Again, this is impressive work. Just wanted to share comments from a
heavy user of perf with Java. You may already have other, better,
ideas for doing on-demand symbol dumps anyway.

Brendan