perf events over (net) console?

perf events over (net) console?

[Harald Gustafsson]

Hi,

We would like to monitor the perf events continuously on a remote
machine. Does it exist a solution (in the kernel) to direct the output
to a console or maybe even a netconsole? We would like to avoid a user
space application to transfer it, due to that the machine will be
running a test which will heavily load it and we want to avoid as many
unrelated user space tasks as possible. If not mainlined does anyone
have a patch for this?

Our purpose is to visualize task's CPU utilization and a few other
events like CPU frequency changes etc in realtime on the remote
machine.

Regards,
  Harald Gustafsson

Re: perf events over (net) console?

[Peter Zijlstra]

On Thu, 2010-09-09 at 13:06 +0200, Harald Gustafsson wrote:
> Hi,
> 
> We would like to monitor the perf events continuously on a remote
> machine. Does it exist a solution (in the kernel) to direct the output
> to a console or maybe even a netconsole? We would like to avoid a user
> space application to transfer it, due to that the machine will be
> running a test which will heavily load it and we want to avoid as many
> unrelated user space tasks as possible. If not mainlined does anyone
> have a patch for this?

No, and its a daft requirement.

You need a process context anyway to read the data and send it to
whatever place you want it.

Putting that in-kernel serves no purpose what so ever.

Re: perf events over (net) console?

[Frederic Weisbecker]

On Thu, Sep 09, 2010 at 01:19:01PM +0200, Peter Zijlstra wrote:
> On Thu, 2010-09-09 at 13:06 +0200, Harald Gustafsson wrote:
> > Hi,
> > 
> > We would like to monitor the perf events continuously on a remote
> > machine. Does it exist a solution (in the kernel) to direct the output
> > to a console or maybe even a netconsole? We would like to avoid a user
> > space application to transfer it, due to that the machine will be
> > running a test which will heavily load it and we want to avoid as many
> > unrelated user space tasks as possible. If not mainlined does anyone
> > have a patch for this?
> 
> No, and its a daft requirement.
> 
> You need a process context anyway to read the data and send it to
> whatever place you want it.
> 
> Putting that in-kernel serves no purpose what so ever.


But if we bring the splice support, that can be done with minimal userspace
noise. Plus that would work with the usual sockets but not limited to that.

trace-cmd does that to carry the traces over the network.

Re: perf events over (net) console?

[Peter Zijlstra]

On Thu, 2010-09-09 at 13:29 +0200, Frederic Weisbecker wrote:
> But if we bring the splice support,

Even simply write() from the mmap() buffer isn't terribly expensive. But
yeah, once we fix splice not to always copy that could be even better.

Splice() is on the todo list, but it needs significant user-interface
work and I'm rather busy..

Re: perf events over (net) console?

[Ingo Molnar]

* Frederic Weisbecker <fweisbec@gmail.com> wrote:

> On Thu, Sep 09, 2010 at 01:19:01PM +0200, Peter Zijlstra wrote:
> > On Thu, 2010-09-09 at 13:06 +0200, Harald Gustafsson wrote:
> > > Hi,
> > > 
> > > We would like to monitor the perf events continuously on a remote 
> > > machine. Does it exist a solution (in the kernel) to direct the 
> > > output to a console or maybe even a netconsole? We would like to 
> > > avoid a user space application to transfer it, due to that the 
> > > machine will be running a test which will heavily load it and we 
> > > want to avoid as many unrelated user space tasks as possible. If 
> > > not mainlined does anyone have a patch for this?
> > 
> > No, and its a daft requirement.
> > 
> > You need a process context anyway to read the data and send it to 
> > whatever place you want it.
> > 
> > Putting that in-kernel serves no purpose what so ever.
> 
> But if we bring the splice support, that can be done with minimal 
> userspace noise. Plus that would work with the usual sockets but not 
> limited to that.

Yes. If we can transform the data over the network without it touching 
disk, then that would be a sufficiently 'does not disturb other tasks' 
measurement method.

Thanks,

	Ingo

Re: perf events over (net) console?

[Frederic Weisbecker]

On Thu, Sep 09, 2010 at 01:35:42PM +0200, Peter Zijlstra wrote:
> On Thu, 2010-09-09 at 13:29 +0200, Frederic Weisbecker wrote:
> > But if we bring the splice support,
> 
> Even simply write() from the mmap() buffer isn't terribly expensive. But
> yeah, once we fix splice not to always copy that could be even better.
> 
> Splice() is on the todo list, but it needs significant user-interface
> work and I'm rather busy..


Yeah sure, that wasn't a citicism, just an idea on how that kind of
minimal userspace noise state can be achieved in the future :)

Re: perf events over (net) console?

[Harald Gustafsson]

2010/9/9 Ingo Molnar <mingo@elte.hu>:
>
> * Frederic Weisbecker <fweisbec@gmail.com> wrote:
>
>> On Thu, Sep 09, 2010 at 01:19:01PM +0200, Peter Zijlstra wrote:
>> > On Thu, 2010-09-09 at 13:06 +0200, Harald Gustafsson wrote:
>> > > Hi,
>> > >
>> > > We would like to monitor the perf events continuously on a remote
>> > > machine. Does it exist a solution (in the kernel) to direct the
>> > > output to a console or maybe even a netconsole? We would like to
>> > > avoid a user space application to transfer it, due to that the
>> > > machine will be running a test which will heavily load it and we
>> > > want to avoid as many unrelated user space tasks as possible. If
>> > > not mainlined does anyone have a patch for this?
>> >
>> > No, and its a daft requirement.

Sorry for being daft...

>> >
>> > You need a process context anyway to read the data and send it to
>> > whatever place you want it.
>> >
>> > Putting that in-kernel serves no purpose what so ever.
>>
>> But if we bring the splice support, that can be done with minimal
>> userspace noise. Plus that would work with the usual sockets but not
>> limited to that.
>
> Yes. If we can transform the data over the network without it touching
> disk, then that would be a sufficiently 'does not disturb other tasks'
> measurement method.

Thanks for the pointers to more information, and yes my thoughts was
more about avoiding the data copy then avoiding any processing context
at all.

Regards,

Re: perf events over (net) console?

[Peter Zijlstra]

On Thu, 2010-09-09 at 14:31 +0200, Harald Gustafsson wrote:

> Sorry for being daft...

No worries, I'm sure we all qualify at times ;-)

> >> >
> >> > You need a process context anyway to read the data and send it to
> >> > whatever place you want it.
> >> >
> >> > Putting that in-kernel serves no purpose what so ever.
> >>
> >> But if we bring the splice support, that can be done with minimal
> >> userspace noise. Plus that would work with the usual sockets but not
> >> limited to that.
> >
> > Yes. If we can transform the data over the network without it touching
> > disk, then that would be a sufficiently 'does not disturb other tasks'
> > measurement method.
> 
> Thanks for the pointers to more information, and yes my thoughts was
> more about avoiding the data copy then avoiding any processing context
> at all.

Right, currently you get a single copy with mmap() + write(), once we
manage to fix splice() and actually provide perf-splice() you'd be able
to do zero-copy.

Re: perf events over (net) console?

[Ingo Molnar]

* Peter Zijlstra <peterz@infradead.org> wrote:

> On Thu, 2010-09-09 at 14:31 +0200, Harald Gustafsson wrote:
> 
> > Sorry for being daft...
> 
> No worries, I'm sure we all qualify at times ;-)
> 
> > >> >
> > >> > You need a process context anyway to read the data and send it to
> > >> > whatever place you want it.
> > >> >
> > >> > Putting that in-kernel serves no purpose what so ever.
> > >>
> > >> But if we bring the splice support, that can be done with minimal
> > >> userspace noise. Plus that would work with the usual sockets but not
> > >> limited to that.
> > >
> > > Yes. If we can transform the data over the network without it touching
> > > disk, then that would be a sufficiently 'does not disturb other tasks'
> > > measurement method.
> > 
> > Thanks for the pointers to more information, and yes my thoughts was 
> > more about avoiding the data copy then avoiding any processing 
> > context at all.
> 
> Right, currently you get a single copy with mmap() + write(), once we 
> manage to fix splice() and actually provide perf-splice() you'd be 
> able to do zero-copy.

I think it would also be very useful to have some sort of tooling help 
for "low overhead/impact perf recording".

If there's a 'target box' and a different 'host box' (where most of 
development is done, etc.), then there might not be any NFS connection 
set up to make zero-copy file transfer easy. Doing it over ssh would add 
overhead.

One possible workflow would be to run this on the target/remote box:

  perf remote

And as long as that command is running there, it could be used from the 
development box (over a trusted local network), using something like:

  perf --remote <hostname> record sleep 60
  perf --remote <hostname> stat -a sleep 1

these would all do the measurements on the remote box, and the resulting 
perf.data would be created on the desktop box. Communication would be 
done via some well-known port.

etc.

An alternative implementation would be to drive this on the assumption 
that an ssh connection can be established with the target box - but 
followup high-volume data transfer would be done over an ordinary TCP 
connection.

I.e. the workflow would be even simpler, something like:

  perf --remote user@hostname record sleep 60
  perf --remote user@hostname stat -a sleep 1

Internally it would work by executing those commands on the remote box 
via ssh, and redirecting the output via a TCP connection. (some other 
details might be needed as well for splice to be usable in such a setup)

Would anyone be interested in having (and implementing ;-) this?

	Ingo

Re: perf events over (net) console?

[Frederic Weisbecker]

On Thu, Sep 09, 2010 at 03:07:23PM +0200, Ingo Molnar wrote:
> 
> * Peter Zijlstra <peterz@infradead.org> wrote:
> 
> > On Thu, 2010-09-09 at 14:31 +0200, Harald Gustafsson wrote:
> > 
> > > Sorry for being daft...
> > 
> > No worries, I'm sure we all qualify at times ;-)
> > 
> > > >> >
> > > >> > You need a process context anyway to read the data and send it to
> > > >> > whatever place you want it.
> > > >> >
> > > >> > Putting that in-kernel serves no purpose what so ever.
> > > >>
> > > >> But if we bring the splice support, that can be done with minimal
> > > >> userspace noise. Plus that would work with the usual sockets but not
> > > >> limited to that.
> > > >
> > > > Yes. If we can transform the data over the network without it touching
> > > > disk, then that would be a sufficiently 'does not disturb other tasks'
> > > > measurement method.
> > > 
> > > Thanks for the pointers to more information, and yes my thoughts was 
> > > more about avoiding the data copy then avoiding any processing 
> > > context at all.
> > 
> > Right, currently you get a single copy with mmap() + write(), once we 
> > manage to fix splice() and actually provide perf-splice() you'd be 
> > able to do zero-copy.
> 
> I think it would also be very useful to have some sort of tooling help 
> for "low overhead/impact perf recording".
> 
> If there's a 'target box' and a different 'host box' (where most of 
> development is done, etc.), then there might not be any NFS connection 
> set up to make zero-copy file transfer easy. Doing it over ssh would add 
> overhead.
> 
> One possible workflow would be to run this on the target/remote box:
> 
>   perf remote
> 
> And as long as that command is running there, it could be used from the 
> development box (over a trusted local network), using something like:
> 
>   perf --remote <hostname> record sleep 60
>   perf --remote <hostname> stat -a sleep 1
> 
> these would all do the measurements on the remote box, and the resulting 
> perf.data would be created on the desktop box. Communication would be 
> done via some well-known port.
> 
> etc.
> 
> An alternative implementation would be to drive this on the assumption 
> that an ssh connection can be established with the target box - but 
> followup high-volume data transfer would be done over an ordinary TCP 
> connection.
> 
> I.e. the workflow would be even simpler, something like:
> 
>   perf --remote user@hostname record sleep 60
>   perf --remote user@hostname stat -a sleep 1
> 
> Internally it would work by executing those commands on the remote box 
> via ssh, and redirecting the output via a TCP connection. (some other 
> details might be needed as well for splice to be usable in such a setup)
> 
> Would anyone be interested in having (and implementing ;-) this?
> 
> 	Ingo



In the beginning this could wrap into perf record - | perf pipe | netcat
and so, until we get the splice support.

Re: perf events over (net) console?

[Arnaldo Carvalho de Melo]

Em Thu, Sep 09, 2010 at 03:21:00PM +0200, Frederic Weisbecker escreveu:
> In the beginning this could wrap into perf record - | perf pipe | netcat
> and so, until we get the splice support.

Isn't this what Tom Zanussi's live mode, that we have already special
casing 'perf record -' does?

Look at these tools/perf/builtin-record.c excerpts:

static int __cmd_record(int argc, const char **argv)
{
<SNIP>
        if (!strcmp(output_name, "-"))
                pipe_output = 1;
<SNIP>
        if (pipe_output)
                output = STDOUT_FILENO;
        else
                output = open(output_name, flags, S_IRUSR | S_IWUSR);
<SNIP>
	if (pipe_output) {
                err = perf_header__write_pipe(output);
<SNIP>
        if (pipe_output) {
                err = event__synthesize_attrs(&session->header,
                                              process_synthesized_event,
                                              session)
<SNIP>
}

Harald, can you please take a look at the comment for these commits:

commit 454c407ec17a0c63e4023ac0877d687945a7df4a
Author: Tom Zanussi <tzanussi@gmail.com>
Date:   Sat May 1 01:41:20 2010 -0500

    perf: add perf-inject builtin

commit 529870e37473a9fc609078f03cc5b4148cf06a87
Author: Tom Zanussi <tzanussi@gmail.com>
Date:   Thu Apr 1 23:59:16 2010 -0500

    perf record: Introduce special handling for pipe output

commit 8dc58101f2c838355d44402aa77646649d10dbec
Author: Tom Zanussi <tzanussi@gmail.com>
Date:   Thu Apr 1 23:59:15 2010 -0500

    perf: Add pipe-specific header read/write and event processing code

Using live mode you can stream to a perf.data for later analisys or do
it live, piping it to trace scripts, etc.

Doing it using splice, talking directly to an in kernel mini-server, etc
are all optimizations that we should think about, but if you can try
using the live mode and see if it is ok, that would be good.

A mini binary with just record can come in handy if you like and should
be rather easy to build, please let me know if you think it is a good
idea and if you need help, lemme know.

- Arnaldo

Re: perf events over (net) console?

[Harald Gustafsson]

2010/9/9 Arnaldo Carvalho de Melo <acme@ghostprotocols.net>:
> Isn't this what Tom Zanussi's live mode, that we have already special
> casing 'perf record -' does?

> Harald, can you please take a look at the comment for these commits:
>
> commit 454c407ec17a0c63e4023ac0877d687945a7df4a
> Author: Tom Zanussi <tzanussi@gmail.com>
> Date:   Sat May 1 01:41:20 2010 -0500
>
>    perf: add perf-inject builtin
>
> commit 529870e37473a9fc609078f03cc5b4148cf06a87
> Author: Tom Zanussi <tzanussi@gmail.com>
> Date:   Thu Apr 1 23:59:16 2010 -0500
>
>    perf record: Introduce special handling for pipe output
>
> commit 8dc58101f2c838355d44402aa77646649d10dbec
> Author: Tom Zanussi <tzanussi@gmail.com>
> Date:   Thu Apr 1 23:59:15 2010 -0500
>
>    perf: Add pipe-specific header read/write and event processing code
>
> Using live mode you can stream to a perf.data for later analisys or do
> it live, piping it to trace scripts, etc.

I took a quick look and it seems promising, thanks for the info, I'll
do some tests tomorrow and get back with some results.

/Harald

> Doing it using splice, talking directly to an in kernel mini-server, etc
> are all optimizations that we should think about, but if you can try
> using the live mode and see if it is ok, that would be good.
>
> A mini binary with just record can come in handy if you like and should
> be rather easy to build, please let me know if you think it is a good
> idea and if you need help, lemme know.
>
> - Arnaldo
>