Linux-Trace-Users Archive on lore.kernel.org
 help / color / Atom feed
* How to find out which process cause a strange increasing cpu load on a certain core?
@ 2019-06-18 11:40 Naruto Nguyen
  2019-06-18 13:02 ` Steven Rostedt
  0 siblings, 1 reply; 3+ messages in thread
From: Naruto Nguyen @ 2019-06-18 11:40 UTC (permalink / raw)
  To: linux-trace-users

Hello everyone,

I have a kvm hypervisor and setup some VMs running OpenSUSE kernel
4.0. After some reboot, I saw that sometime there is a VM that a
certain core has higher %usr and %sys than other cores. The system is
in ide state. I have some applications running on this VM, but there
is no issue in other VMs.

If I enable ftrace in kernel or load any debug jprobe, the load is
back to normal. The pidstat command output does not show the high load
on one core like the "top" command. Could you please let me know if it
is the problem of "top' report higher usage? Any commands that can
help to troubleshoot this issue? I tried "perf" but most of the time
the perf output does not show any function that cause the increase as
well? Not sure, if the malfunction of top or anything else can add
more work load?

Thanks,
Brs,
Naruto

^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: How to find out which process cause a strange increasing cpu load on a certain core?
  2019-06-18 11:40 How to find out which process cause a strange increasing cpu load on a certain core? Naruto Nguyen
@ 2019-06-18 13:02 ` Steven Rostedt
  2019-06-18 16:28   ` Naruto Nguyen
  0 siblings, 1 reply; 3+ messages in thread
From: Steven Rostedt @ 2019-06-18 13:02 UTC (permalink / raw)
  To: Naruto Nguyen; +Cc: linux-trace-users

On Tue, 18 Jun 2019 18:40:13 +0700
Naruto Nguyen <narutonguyen2018@gmail.com> wrote:

> Hello everyone,
> 
> I have a kvm hypervisor and setup some VMs running OpenSUSE kernel
> 4.0. After some reboot, I saw that sometime there is a VM that a
> certain core has higher %usr and %sys than other cores. The system is
> in ide state. I have some applications running on this VM, but there
> is no issue in other VMs.

How much higher? And is it always an issue with the same core on the
same VM?

> 
> If I enable ftrace in kernel or load any debug jprobe, the load is

I'm assuming by "ftrace" you mean the full function tracer?

 # echo function > /sys/kernel/tracing/current_tracer

> back to normal. The pidstat command output does not show the high load
> on one core like the "top" command. Could you please let me know if it
> is the problem of "top' report higher usage? Any commands that can
> help to troubleshoot this issue? I tried "perf" but most of the time
> the perf output does not show any function that cause the increase as
> well? Not sure, if the malfunction of top or anything else can add
> more work load?

I would suggest just tracing sched switches and interrupts. That
shouldn't affect the load, as it is very low weight.

 # cd /sys/kernel/tracing
 # echo 1 > events/sched/sched_waking/enable
 # echo 1 > events/sched/sched_switch/enable
 # echo 1 > events/irq/enable

And then monitor that.

-- Steve

^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: How to find out which process cause a strange increasing cpu load on a certain core?
  2019-06-18 13:02 ` Steven Rostedt
@ 2019-06-18 16:28   ` Naruto Nguyen
  0 siblings, 0 replies; 3+ messages in thread
From: Naruto Nguyen @ 2019-06-18 16:28 UTC (permalink / raw)
  To: Steven Rostedt; +Cc: linux-trace-users

Hi Steven,

Thanks a lot for your reply.

It’s higher about 16% usr, 34% sys in top output but as I observe
pidstat has no any process consume this high cpu usage. After reboot
it can happen on a certain core on a certain VM, not fixed to any
core.

For ftrace, I mean function profile tracer by
echo 1 > /sys/kernel/debug/tracing/function_profile_enabled

I will try your suggestion.

Thanks again,
Brs,
Naruto

On Tue, 18 Jun 2019 at 20:02, Steven Rostedt <rostedt@goodmis.org> wrote:
>
> On Tue, 18 Jun 2019 18:40:13 +0700
> Naruto Nguyen <narutonguyen2018@gmail.com> wrote:
>
> > Hello everyone,
> >
> > I have a kvm hypervisor and setup some VMs running OpenSUSE kernel
> > 4.0. After some reboot, I saw that sometime there is a VM that a
> > certain core has higher %usr and %sys than other cores. The system is
> > in ide state. I have some applications running on this VM, but there
> > is no issue in other VMs.
>
> How much higher? And is it always an issue with the same core on the
> same VM?
>
> >
> > If I enable ftrace in kernel or load any debug jprobe, the load is
>
> I'm assuming by "ftrace" you mean the full function tracer?
>
>  # echo function > /sys/kernel/tracing/current_tracer
>
> > back to normal. The pidstat command output does not show the high load
> > on one core like the "top" command. Could you please let me know if it
> > is the problem of "top' report higher usage? Any commands that can
> > help to troubleshoot this issue? I tried "perf" but most of the time
> > the perf output does not show any function that cause the increase as
> > well? Not sure, if the malfunction of top or anything else can add
> > more work load?
>
> I would suggest just tracing sched switches and interrupts. That
> shouldn't affect the load, as it is very low weight.
>
>  # cd /sys/kernel/tracing
>  # echo 1 > events/sched/sched_waking/enable
>  # echo 1 > events/sched/sched_switch/enable
>  # echo 1 > events/irq/enable
>
> And then monitor that.
>
> -- Steve

^ permalink raw reply	[flat|nested] 3+ messages in thread

end of thread, back to index

Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-06-18 11:40 How to find out which process cause a strange increasing cpu load on a certain core? Naruto Nguyen
2019-06-18 13:02 ` Steven Rostedt
2019-06-18 16:28   ` Naruto Nguyen

Linux-Trace-Users Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-trace-users/0 linux-trace-users/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-trace-users linux-trace-users/ https://lore.kernel.org/linux-trace-users \
		linux-trace-users@vger.kernel.org
	public-inbox-index linux-trace-users

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-trace-users


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git