How to find out which process cause a strange increasing cpu load on a certain core?

How to find out which process cause a strange increasing cpu load on a certain core?

[Naruto Nguyen]

Hello everyone,

I have a kvm hypervisor and setup some VMs running OpenSUSE kernel
4.0. After some reboot, I saw that sometime there is a VM that a
certain core has higher %usr and %sys than other cores. The system is
in ide state. I have some applications running on this VM, but there
is no issue in other VMs.

If I enable ftrace in kernel or load any debug jprobe, the load is
back to normal. The pidstat command output does not show the high load
on one core like the "top" command. Could you please let me know if it
is the problem of "top' report higher usage? Any commands that can
help to troubleshoot this issue? I tried "perf" but most of the time
the perf output does not show any function that cause the increase as
well? Not sure, if the malfunction of top or anything else can add
more work load?

Thanks,
Brs,
Naruto

Re: How to find out which process cause a strange increasing cpu load on a certain core?

[Steven Rostedt]

On Tue, 18 Jun 2019 18:40:13 +0700
Naruto Nguyen <narutonguyen2018@gmail.com> wrote:

> Hello everyone,
> 
> I have a kvm hypervisor and setup some VMs running OpenSUSE kernel
> 4.0. After some reboot, I saw that sometime there is a VM that a
> certain core has higher %usr and %sys than other cores. The system is
> in ide state. I have some applications running on this VM, but there
> is no issue in other VMs.

How much higher? And is it always an issue with the same core on the
same VM?

> 
> If I enable ftrace in kernel or load any debug jprobe, the load is

I'm assuming by "ftrace" you mean the full function tracer?

 # echo function > /sys/kernel/tracing/current_tracer

> back to normal. The pidstat command output does not show the high load
> on one core like the "top" command. Could you please let me know if it
> is the problem of "top' report higher usage? Any commands that can
> help to troubleshoot this issue? I tried "perf" but most of the time
> the perf output does not show any function that cause the increase as
> well? Not sure, if the malfunction of top or anything else can add
> more work load?

I would suggest just tracing sched switches and interrupts. That
shouldn't affect the load, as it is very low weight.

 # cd /sys/kernel/tracing
 # echo 1 > events/sched/sched_waking/enable
 # echo 1 > events/sched/sched_switch/enable
 # echo 1 > events/irq/enable

And then monitor that.

-- Steve

Re: How to find out which process cause a strange increasing cpu load on a certain core?

[Naruto Nguyen]

Hi Steven,

Thanks a lot for your reply.

It’s higher about 16% usr, 34% sys in top output but as I observe
pidstat has no any process consume this high cpu usage. After reboot
it can happen on a certain core on a certain VM, not fixed to any
core.

For ftrace, I mean function profile tracer by
echo 1 > /sys/kernel/debug/tracing/function_profile_enabled

I will try your suggestion.

Thanks again,
Brs,
Naruto

On Tue, 18 Jun 2019 at 20:02, Steven Rostedt <rostedt@goodmis.org> wrote:
>
> On Tue, 18 Jun 2019 18:40:13 +0700
> Naruto Nguyen <narutonguyen2018@gmail.com> wrote:
>
> > Hello everyone,
> >
> > I have a kvm hypervisor and setup some VMs running OpenSUSE kernel
> > 4.0. After some reboot, I saw that sometime there is a VM that a
> > certain core has higher %usr and %sys than other cores. The system is
> > in ide state. I have some applications running on this VM, but there
> > is no issue in other VMs.
>
> How much higher? And is it always an issue with the same core on the
> same VM?
>
> >
> > If I enable ftrace in kernel or load any debug jprobe, the load is
>
> I'm assuming by "ftrace" you mean the full function tracer?
>
>  # echo function > /sys/kernel/tracing/current_tracer
>
> > back to normal. The pidstat command output does not show the high load
> > on one core like the "top" command. Could you please let me know if it
> > is the problem of "top' report higher usage? Any commands that can
> > help to troubleshoot this issue? I tried "perf" but most of the time
> > the perf output does not show any function that cause the increase as
> > well? Not sure, if the malfunction of top or anything else can add
> > more work load?
>
> I would suggest just tracing sched switches and interrupts. That
> shouldn't affect the load, as it is very low weight.
>
>  # cd /sys/kernel/tracing
>  # echo 1 > events/sched/sched_waking/enable
>  # echo 1 > events/sched/sched_switch/enable
>  # echo 1 > events/irq/enable
>
> And then monitor that.
>
> -- Steve