lttng-dev.lists.lttng.org archive mirror
 help / color / mirror / Atom feed
From: "熊毓华 via lttng-dev" <lttng-dev@lists.lttng.org>
To: "Jonathan Rajotte-Julien" <jonathan.rajotte-julien@efficios.com>,
	 "mathieu.desnoyers@efficios.com"
	<mathieu.desnoyers@efficios.com>,
	 "lttng-dev@lists.lttng.org" <lttng-dev@lists.lttng.org>
Subject: Re: [lttng-dev] Some confusion about cpu usage of the lttng-consumerd process
Date: Sat, 28 Nov 2020 14:49:28 +0800 (GMT+08:00)	[thread overview]
Message-ID: <2cade259.4f3f.1760d9d1b84.Coremail.xiongyuhua@zju.edu.cn> (raw)
In-Reply-To: <828264314.65604.1606493063962.JavaMail.zimbra@efficios.com>


[-- Attachment #1.1: Type: text/plain, Size: 6284 bytes --]

Hi,

I put my test scripts in the attachment.

You can just run the script directly, create the trace session with the "--live" option on the 8core server,

then you will find the cpu usage of the lttng-consumerd process reached 10% or more.




About the streaming mode of lttng,I did the test before, it worked well.

When I create the trace session with "lttng create my-session --output=/tmp/my-kernel-trace", or with "lttng create my-session --set-url=net://ip",

the number of CPU seems not affect the cpu usage with lttng-consumerd.

It seems that only live-mode will be affected.




thanks,

yuhua


-----原始邮件-----
发件人:"Jonathan Rajotte-Julien" <jonathan.rajotte-julien@efficios.com>
发送时间:2020-11-28 00:04:23 (星期六)
收件人: "熊毓华" <xiongyuhua@zju.edu.cn>
抄送: lttng-dev <lttng-dev@lists.lttng.org>
主题: Re: [lttng-dev] Some confusion about cpu usage of the lttng-consumerd process




From: "熊毓华" <xiongyuhua@zju.edu.cn>
To: "Jonathan Rajotte-Julien" <jonathan.rajotte-julien@efficios.com>, "lttng-dev" <lttng-dev@lists.lttng.org>
Sent: Friday, November 27, 2020 10:32:07 AM
Subject: Re: Re: [lttng-dev] Some confusion about cpu usage of the lttng-consumerd process

Hi,Dear.


Side note, you can remove the "Dear" here. ;)




The test script was used to generate some common fileIO,netIO events.


Please provide a complete code repository if possible. So that we can at least have a baseline for reproduction.




On all servers, the monitoring strategy I set up when I start lttng is the same, monitoring all fileIO, netIO and some related system calls. 
The following table records the amount of events generated by the test script per minute, and one babeltrace record represents one event.




For some reason the image does not load here. Please provide a text based alternative for this figure.






The unit of the number is every ten thousand events per minute. And the number were read out after parsing by babeltrace.
In addition, the server1 is 1core4G, server2 is 2core8G, server3 is 4core16G, server4 and server5 are 8core16G.

It can be seen that the average amount of data generated per minute on all servers is roughly the same.However, the CPU usage of the lttng-consumerd process behave differently on server4 and server5, as I mentioned in my last email.


In addition, the usage of cpu is recorded using the "top" command.




My test concluded that, while the same number of events collected, lttng-consumerd process need to consume more cpu on the 8-core server.

I want to know why is this and what else information do you need?



Well we also want to know why! You will understand that albeit we develop lttng we do not always have a quick and easy answer to all problems. Performance related problem are always tricky.
And we also have to keep in mind that we do not necessarily optimize for low-cpu usage on the lttng-consumerd side. 


We have to take a look at what "work" scale with the number of CPU on the lttng-consumerd side. One such thing is the live timer which is fired on an interval (default is 1s (1000000us)).


You could test this hypothesis by streaming the trace instead of using the live feature.


lttng create --set-url ....


Cheers



Looking forward to your reply.
thanks,
yuhua.


> -----原始邮件-----
> 发件人: "Jonathan Rajotte-Julien" <jonathan.rajotte-julien@efficios.com>
> 发送时间: 2020-11-27 22:05:48 (星期五)
> 收件人: "熊毓华" <xiongyuhua@zju.edu.cn>
> 抄送: lttng-dev@lists.lttng.org
> 主题: Re: [lttng-dev] Some confusion about cpu usage of the lttng-consumerd process
> 
> Hi,
> 
> On Fri, Nov 27, 2020 at 02:39:28PM +0800, 熊毓华 via lttng-dev wrote:
> > Hi,dear.
> > 
> > I have been using lttng to monitor my server these days,but I found something interesting.
> > 
> > The cpu usage of lttng varies with the number of cpu cores of the server.
> 
> Which is a bit expected since more CPU means more "data" source from the point
> of view of lttng hence more "work" overall.
> 
> > 
> > On the server, I create a tracing session in live mode, using "lttng create my-session --live". 
> > 
> > Then,I Start the babeltrace2 and configure it to connect to the relay daemon,using "--input-format=lttng-live" mode.
> > 
> > I used 5 cloud servers,1core4G 2core8G 4core16G 8core16G 8core16G.
> > 
> > And,the same test script was executed above to provide the same workload.
> 
> We would need the test script to have some context here of the workload.
> 
> > 
> > As we all know,lttng has 5 processes,
> > 
> > 1.lttng-runas    --daemonize
> > 
> > 2.lttng-runas      -k --consumerd-cmd-sock /var/run/lttng/kconsumerd/command --consumerd-err-sock /var/run/lttng/kconsumerd/error --group tracing
> 
> Based on this you are performing kernel tracing.
> 
> > 
> > 3.lttng-sessiond --daemonize
> > 
> > 4.lttng-relayd -L tcp://localhost:5344
> > 
> > 5.lttng-consumerd  -k --consumerd-cmd-sock /var/run/lttng/kconsumerd/command --consumerd-err-sock /var/run/lttng/kconsumerd/error --group tracing
> > 
> > 
> > The CPU usage of the first four processes is below 2% on the 5 servers,but the lttng-consumerd process is different.
> > 
> > On 1-core、2-core、4-core servers,the CPU usage of the lttng-consumerd process is below 2%.
> 
> How is the cpu usage measured here?
> 
> > 
> > But on two 8-core machines, the cpu usage of the lttng-consumerd process reached 10% or more.
> 
> Consumerd is responsible of "fetching" data from the ring buffers and "saving"
> it either locally (trace on disk) or remotely (streaming/live session). CPU usage
> should be a bit correlated with the event production rate. Did you have a look at the
> number of events generated for a similar interval?
> 
> > And,the cpu usage of the babeltrace process is not much different,just the cpu usage of the lttng-consumerd process varies with the number of cpu cores of the server.
> > 
> > Why it is like this?How should this phenomenon be analyzed?
> > 
> > Looking forward to your reply.
> > 
> > thanks,
> > yuhua
> > 
> > _______________________________________________
> > lttng-dev mailing list
> > lttng-dev@lists.lttng.org
> > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
> 
> 
> -- 
> Jonathan Rajotte-Julien
> EfficiOS




[-- Attachment #1.2: Type: text/html, Size: 13777 bytes --]

[-- Attachment #2: test script.rar --]
[-- Type: application/octet-stream, Size: 1120 bytes --]

[-- Attachment #3: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

  parent reply	other threads:[~2020-11-28  6:49 UTC|newest]

Thread overview: 7+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-11-27  6:39 [lttng-dev] Some confusion about cpu usage of the lttng-consumerd process 熊毓华 via lttng-dev
2020-11-27 14:05 ` Jonathan Rajotte-Julien via lttng-dev
2020-11-27 15:32   ` 熊毓华 via lttng-dev
2020-11-27 16:04     ` Jonathan Rajotte-Julien via lttng-dev
2020-11-27 17:11       ` Mathieu Desnoyers via lttng-dev
2020-11-28  6:49       ` 熊毓华 via lttng-dev [this message]
2020-11-30 14:24         ` Mathieu Desnoyers via lttng-dev

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=2cade259.4f3f.1760d9d1b84.Coremail.xiongyuhua@zju.edu.cn \
    --to=lttng-dev@lists.lttng.org \
    --cc=jonathan.rajotte-julien@efficios.com \
    --cc=mathieu.desnoyers@efficios.com \
    --cc=xiongyuhua@zju.edu.cn \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).