Ask for help about cpufreq issue

Ask for help about cpufreq issue

[Andy Tang]

Hi Viresh,

Cpufreq failed to work on our board. CPU frequency keeps at the lowest frequency whatever governor I choose.

Hardware platform: ls1043ardb board (4 cores arm64 arch)
Software: linux kernel v5.4.3 + ubuntu 1804.

Test log:
root@localhost:~# echo performance > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
root@localhost:~# cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq
500000
root@localhost:~# cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_frequencies
1600000 1000000 800000 500000

All the parameters you may need:
root@localhost:~# for entry in `ls /sys/devices/system/cpu/cpufreq/policy0/` ; do echo $entry;  cat  /sys/devices/system/cpu/cpufreq/policy0/$entry; done
affected_cpus
0 1 2 3
cpuinfo_cur_freq
500000
cpuinfo_max_freq
1600000
cpuinfo_min_freq
500000
cpuinfo_transition_latency
31
related_cpus
0 1 2 3
scaling_available_frequencies
1600000 1000000 800000 500000
scaling_available_governors
userspace ondemand performance schedutil
scaling_cur_freq
500000
scaling_driver
qoriq_cpufreq
scaling_governor
performance
scaling_max_freq
500000
scaling_min_freq
500000
scaling_setspeed
<unsupported>
stats
cat: /sys/devices/system/cpu/cpufreq/policy0/stats: Is a directory


Could you please instruct me how to investigate this issue?

BR,
Andy

Re: Ask for help about cpufreq issue

[Viresh Kumar]

On 07-02-20, 05:09, Andy Tang wrote:
> Hi Viresh,
> 
> Cpufreq failed to work on our board. CPU frequency keeps at the lowest frequency whatever governor I choose.

Have you tried performance governor ? Freq should move to highest freq
there for sure.

How do you check the frequency ? Without significant load you won't
see the freq movement.

Also have a look at the policy0/stats/ directory. It will show the
details of freq switching properly, include the transitions that have
happened.

> Hardware platform: ls1043ardb board (4 cores arm64 arch)
> Software: linux kernel v5.4.3 + ubuntu 1804.
> 
> Test log:
> root@localhost:~# echo performance > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
> root@localhost:~# cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq
> 500000
> root@localhost:~# cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_frequencies
> 1600000 1000000 800000 500000
> 
> All the parameters you may need:
> root@localhost:~# for entry in `ls /sys/devices/system/cpu/cpufreq/policy0/` ; do echo $entry;  cat  /sys/devices/system/cpu/cpufreq/policy0/$entry; done
> affected_cpus
> 0 1 2 3
> cpuinfo_cur_freq
> 500000
> cpuinfo_max_freq
> 1600000
> cpuinfo_min_freq
> 500000
> cpuinfo_transition_latency
> 31
> related_cpus
> 0 1 2 3
> scaling_available_frequencies
> 1600000 1000000 800000 500000
> scaling_available_governors
> userspace ondemand performance schedutil
> scaling_cur_freq
> 500000
> scaling_driver
> qoriq_cpufreq
> scaling_governor
> performance
> scaling_max_freq
> 500000
> scaling_min_freq
> 500000
> scaling_setspeed
> <unsupported>
> stats
> cat: /sys/devices/system/cpu/cpufreq/policy0/stats: Is a directory
> 
> 
> Could you please instruct me how to investigate this issue?
> 
> BR,
> Andy

-- 
viresh

RE: [EXT] Re: Ask for help about cpufreq issue

[Andy Tang]

Hi Viresh,

I have tried performance governor, cpu frequency is at the lowest frequency.
I also tried ondemand governor with high cpu load, cpu frequency doesn't increase too.

root@localhost:~# cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
performance
root@localhost:~# cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_frequencies
1600000 1000000 800000 500000
root@localhost:~#
root@localhost:~# cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq
500000

root@localhost:~# cat /sys/devices/system/cpu/cpufreq/policy0/stats/*
cat: /sys/devices/system/cpu/cpufreq/policy0/stats/reset: Permission denied
1600000 214
1000000 91
800000 82
500000 480088
240
   From  :    To
         :   1600000   1000000    800000    500000
  1600000:         0        15        10        10
  1000000:        21         0        22        18
   800000:         6        23         0        44
   500000:         7        23        41         0
BR,
Andy

> -----Original Message-----
> From: Viresh Kumar <viresh.kumar@linaro.org>
> Sent: 2020年2月7日 13:23
> To: Andy Tang <andy.tang@nxp.com>
> Cc: rjw@rjwysocki.net; linux-pm@vger.kernel.org
> Subject: [EXT] Re: Ask for help about cpufreq issue
> 
> Caution: EXT Email
> 
> On 07-02-20, 05:09, Andy Tang wrote:
> > Hi Viresh,
> >
> > Cpufreq failed to work on our board. CPU frequency keeps at the lowest
> frequency whatever governor I choose.
> 
> Have you tried performance governor ? Freq should move to highest freq
> there for sure.
> 
> How do you check the frequency ? Without significant load you won't see the
> freq movement.
> 
> Also have a look at the policy0/stats/ directory. It will show the details of freq
> switching properly, include the transitions that have happened.
> 
> > Hardware platform: ls1043ardb board (4 cores arm64 arch)
> > Software: linux kernel v5.4.3 + ubuntu 1804.
> >
> > Test log:
> > root@localhost:~# echo performance >
> > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
> > root@localhost:~# cat
> > /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq
> > 500000
> > root@localhost:~# cat
> > /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_frequencies
> > 1600000 1000000 800000 500000
> >
> > All the parameters you may need:
> > root@localhost:~# for entry in `ls
> > /sys/devices/system/cpu/cpufreq/policy0/` ; do echo $entry;  cat
> > /sys/devices/system/cpu/cpufreq/policy0/$entry; done affected_cpus
> > 0 1 2 3
> > cpuinfo_cur_freq
> > 500000
> > cpuinfo_max_freq
> > 1600000
> > cpuinfo_min_freq
> > 500000
> > cpuinfo_transition_latency
> > 31
> > related_cpus
> > 0 1 2 3
> > scaling_available_frequencies
> > 1600000 1000000 800000 500000
> > scaling_available_governors
> > userspace ondemand performance schedutil scaling_cur_freq
> > 500000
> > scaling_driver
> > qoriq_cpufreq
> > scaling_governor
> > performance
> > scaling_max_freq
> > 500000
> > scaling_min_freq
> > 500000
> > scaling_setspeed
> > <unsupported>
> > stats
> > cat: /sys/devices/system/cpu/cpufreq/policy0/stats: Is a directory
> >
> >
> > Could you please instruct me how to investigate this issue?
> >
> > BR,
> > Andy
> 
> --
> viresh

Re: [EXT] Re: Ask for help about cpufreq issue

[Viresh Kumar]

On 07-02-20, 06:03, Andy Tang wrote:
> Hi Viresh,
> 
> I have tried performance governor, cpu frequency is at the lowest frequency.
> I also tried ondemand governor with high cpu load, cpu frequency doesn't increase too.
> 
> root@localhost:~# cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
> performance
> root@localhost:~# cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_frequencies
> 1600000 1000000 800000 500000
> root@localhost:~#
> root@localhost:~# cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq
> 500000
> 
> root@localhost:~# cat /sys/devices/system/cpu/cpufreq/policy0/stats/*
> cat: /sys/devices/system/cpu/cpufreq/policy0/stats/reset: Permission denied
> 1600000 214
> 1000000 91
> 800000 82
> 500000 480088
> 240
>    From  :    To
>          :   1600000   1000000    800000    500000
>   1600000:         0        15        10        10
>   1000000:        21         0        22        18
>    800000:         6        23         0        44
>    500000:         7        23        41         0

See, frequencies are getting changed according to this table (at least
from cpufreq's point of view), which means that cpufreq transitions
are passing just fine.

I just had a look at the driver and may have some idea on what's going
on. The cpufreq driver doesn't program the frequencies, but instead it
just switches the parent clk for changing the frequency. And when you
try to read the frequency, all we do is read the frequency from the
clk pointer using clk_get_rate().

So, here are the things which can go wrong, in order of how I would
like to debug them.

- The frequency of all parent clocks is set to 500 MHz by someone and
  so clk_get_rate() always report that despite changes to the stats
  table.

- clk_set_parent() isn't setting the parent correctly.

Try putting some print messages to the driver to see if target_index()
routine is able to set the parent clk properly or not.

-- 
viresh

RE: [EXT] Re: Ask for help about cpufreq issue

[Andy Tang]

Hi Viresh,

I checked the clk part, no issue was found.
I did found a odd thing in cpufreq directory.

# cat scaling_available_frequencies
1600000 1000000 800000 500000
# cat scaling_max_freq
500000

Why there is a 1600Mhz frequency available, but the max freq is only 500M?
How the max freq is set?

Thanks,
Andy
> -----Original Message-----
> From: Viresh Kumar <viresh.kumar@linaro.org>
> Sent: 2020年2月7日 15:06
> To: Andy Tang <andy.tang@nxp.com>
> Cc: rjw@rjwysocki.net; linux-pm@vger.kernel.org
> Subject: Re: [EXT] Re: Ask for help about cpufreq issue
> 
> Caution: EXT Email
> 
> On 07-02-20, 06:03, Andy Tang wrote:
> > Hi Viresh,
> >
> > I have tried performance governor, cpu frequency is at the lowest
> frequency.
> > I also tried ondemand governor with high cpu load, cpu frequency doesn't
> increase too.
> >
> > root@localhost:~# cat
> > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
> > performance
> > root@localhost:~# cat
> > /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_frequencies
> > 1600000 1000000 800000 500000
> > root@localhost:~#
> > root@localhost:~# cat
> > /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq
> > 500000
> >
> > root@localhost:~# cat /sys/devices/system/cpu/cpufreq/policy0/stats/*
> > cat: /sys/devices/system/cpu/cpufreq/policy0/stats/reset: Permission
> > denied
> > 1600000 214
> > 1000000 91
> > 800000 82
> > 500000 480088
> > 240
> >    From  :    To
> >          :   1600000   1000000    800000    500000
> >   1600000:         0        15        10        10
> >   1000000:        21         0        22        18
> >    800000:         6        23         0        44
> >    500000:         7        23        41         0
> 
> See, frequencies are getting changed according to this table (at least from
> cpufreq's point of view), which means that cpufreq transitions are passing
> just fine.
> 
> I just had a look at the driver and may have some idea on what's going on.
> The cpufreq driver doesn't program the frequencies, but instead it just
> switches the parent clk for changing the frequency. And when you try to read
> the frequency, all we do is read the frequency from the clk pointer using
> clk_get_rate().
> 
> So, here are the things which can go wrong, in order of how I would like to
> debug them.
> 
> - The frequency of all parent clocks is set to 500 MHz by someone and
>   so clk_get_rate() always report that despite changes to the stats
>   table.
> 
> - clk_set_parent() isn't setting the parent correctly.
> 
> Try putting some print messages to the driver to see if target_index() routine
> is able to set the parent clk properly or not.
> 
> --
> viresh

Re: [EXT] Re: Ask for help about cpufreq issue

[Viresh Kumar]

On 11-02-20, 04:20, Andy Tang wrote:
> Hi Viresh,
> 
> I checked the clk part, no issue was found.
> I did found a odd thing in cpufreq directory.
> 
> # cat scaling_available_frequencies
> 1600000 1000000 800000 500000
> # cat scaling_max_freq
> 500000

Ahh, sorry about that as I must have noticed it earlier.

> Why there is a 1600Mhz frequency available, but the max freq is only 500M?
> How the max freq is set?

Hmm, I am not sure why this happened. Can you try to write 1600000 to
scaling_max_freq? This will fix it if the userspace is responsible for
this limitation and you need to find which part of your userspace is
causing it.

Else this is coming from within kernel and you need to track calls to
dev_pm_qos_add_request() and check all calls related to freq qos
constraints.

-- 
viresh

RE: [EXT] Re: Ask for help about cpufreq issue

[Andy Tang]

Thanks for your instruction. I guess it was caused by QOS.
I try to change scaling_max_freq. it can be written, but have no effect.

I am going to track to dev_pm_qos_add_request() as you suggested.
Before that, I am also wondering what the factor it could be to limit the cpu max frequency?

BR,
Andy

> -----Original Message-----
> From: Viresh Kumar <viresh.kumar@linaro.org>
> Sent: 2020年2月11日 13:53
> To: Andy Tang <andy.tang@nxp.com>
> Cc: rjw@rjwysocki.net; linux-pm@vger.kernel.org
> Subject: Re: [EXT] Re: Ask for help about cpufreq issue
> 
> Caution: EXT Email
> 
> On 11-02-20, 04:20, Andy Tang wrote:
> > Hi Viresh,
> >
> > I checked the clk part, no issue was found.
> > I did found a odd thing in cpufreq directory.
> >
> > # cat scaling_available_frequencies
> > 1600000 1000000 800000 500000
> > # cat scaling_max_freq
> > 500000
> 
> Ahh, sorry about that as I must have noticed it earlier.
> 
> > Why there is a 1600Mhz frequency available, but the max freq is only
> 500M?
> > How the max freq is set?
> 
> Hmm, I am not sure why this happened. Can you try to write 1600000 to
> scaling_max_freq? This will fix it if the userspace is responsible for this
> limitation and you need to find which part of your userspace is causing it.
> 
> Else this is coming from within kernel and you need to track calls to
> dev_pm_qos_add_request() and check all calls related to freq qos
> constraints.
> 
> --
> viresh

Re: [EXT] Re: Ask for help about cpufreq issue

[Viresh Kumar]

On 11-02-20, 06:23, Andy Tang wrote:
> I am going to track to dev_pm_qos_add_request() as you suggested.
> Before that, I am also wondering what the factor it could be to limit the cpu max frequency?

One of them is thermal, but there can be others as well. Just do
following search in your kernel source code:

git grep freq_qos_add_request

This will list all the sources of this constraint.

-- 
viresh

RE: [EXT] Re: Ask for help about cpufreq issue

[Andy Tang]

Thank you very much. I confirmed it was caused by QOS.
To be specific, it was caused by TMU.

BR,
Andy

> -----Original Message-----
> From: Viresh Kumar <viresh.kumar@linaro.org>
> Sent: 2020年2月11日 14:35
> To: Andy Tang <andy.tang@nxp.com>
> Cc: rjw@rjwysocki.net; linux-pm@vger.kernel.org
> Subject: Re: [EXT] Re: Ask for help about cpufreq issue
> 
> Caution: EXT Email
> 
> On 11-02-20, 06:23, Andy Tang wrote:
> > I am going to track to dev_pm_qos_add_request() as you suggested.
> > Before that, I am also wondering what the factor it could be to limit the cpu
> max frequency?
> 
> One of them is thermal, but there can be others as well. Just do following
> search in your kernel source code:
> 
> git grep freq_qos_add_request
> 
> This will list all the sources of this constraint.
> 
> --
> viresh

RE: [EXT] Re: Ask for help about cpufreq issue

[Andy Tang]

Hi Viresh,

I have another cpufreq issue on our lx2160a platform which has 16 cores and 8 clusters.

Cpufreq works well for ondemand, performance governor. 
But for conservative governor, even though cpu usage is almost zero, but some cpus are not
work on the lowest frequency sometimes. It keeps bouncing from the lowest frequency
to the other higher frequencies. 
The defaut values are used for the tunable for conservative governor.
In summary, why cpu doesn't work at the lowest frequency even the cpu load is almost zero?

root@lx2160ardb:/sys/devices/system/cpu# cat cpufreq/conservative/ 
down_threshold        freq_step             ignore_nice_load      sampling_down_factor  sampling_rate         up_threshold          
root@lx2160ardb:/sys/devices/system/cpu# cat cpufreq/conservative/*
20
5
0
1
8000
80

root@lx2160ardb:/sys/devices/system/cpu# cat cpu[0-9]/cpufreq/scaling_governor
conservative
conservative
conservative
conservative
conservative
conservative
conservative
conservative
conservative
conservative

root@lx2160ardb:/sys/devices/system/cpu# cat cpu[0-9]/cpufreq/scaling_cur_freq
1000000
1000000
1000000
1000000
1000000
1000000
2000000		/// this is not working at the lowest frequency
2000000
700000
700000

Could you please instruct me how to debug this issue?
Thanks,

BR,
Andy
> -----Original Message-----
> From: Viresh Kumar <viresh.kumar@linaro.org>
> Sent: 2020年2月11日 14:35
> To: Andy Tang <andy.tang@nxp.com>
> Cc: rjw@rjwysocki.net; linux-pm@vger.kernel.org
> Subject: Re: [EXT] Re: Ask for help about cpufreq issue
> 
> Caution: EXT Email
> 
> On 11-02-20, 06:23, Andy Tang wrote:
> > I am going to track to dev_pm_qos_add_request() as you suggested.
> > Before that, I am also wondering what the factor it could be to limit the cpu
> max frequency?
> 
> One of them is thermal, but there can be others as well. Just do following
> search in your kernel source code:
> 
> git grep freq_qos_add_request
> 
> This will list all the sources of this constraint.
> 
> --
> viresh

Re: [EXT] Re: Ask for help about cpufreq issue

[Viresh Kumar]

On 12-02-20, 10:11, Andy Tang wrote:
> Hi Viresh,
> 
> I have another cpufreq issue on our lx2160a platform which has 16 cores and 8 clusters.
> 
> Cpufreq works well for ondemand, performance governor. 
> But for conservative governor, even though cpu usage is almost zero, but some cpus are not
> work on the lowest frequency sometimes. It keeps bouncing from the lowest frequency
> to the other higher frequencies. 
> The defaut values are used for the tunable for conservative governor.
> In summary, why cpu doesn't work at the lowest frequency even the cpu load is almost zero?
> 
> root@lx2160ardb:/sys/devices/system/cpu# cat cpufreq/conservative/ 
> down_threshold        freq_step             ignore_nice_load      sampling_down_factor  sampling_rate         up_threshold          
> root@lx2160ardb:/sys/devices/system/cpu# cat cpufreq/conservative/*
> 20

Try increasing down_threshold to a value of 50 (i.e. we reduce
frequency if the load is below 50%) and see if it makes a difference.

I believe something is running on the CPU, that you are unaware of.
Try doing tracing to see what's going on all CPUs.

-- 
viresh

RE: [EXT] Re: Ask for help about cpufreq issue

[Andy Tang]

Hi Viresh,

> -----Original Message-----
> From: Viresh Kumar <viresh.kumar@linaro.org>
> Sent: 2020年2月12日 19:51
> To: Andy Tang <andy.tang@nxp.com>
> Cc: rjw@rjwysocki.net; linux-pm@vger.kernel.org
> Subject: Re: [EXT] Re: Ask for help about cpufreq issue
> 
> Caution: EXT Email
> 
> On 12-02-20, 10:11, Andy Tang wrote:
> > Hi Viresh,
> >
> > I have another cpufreq issue on our lx2160a platform which has 16 cores
> and 8 clusters.
> >
> > Cpufreq works well for ondemand, performance governor.
> > But for conservative governor, even though cpu usage is almost zero,
> > but some cpus are not work on the lowest frequency sometimes. It keeps
> > bouncing from the lowest frequency to the other higher frequencies.
> > The defaut values are used for the tunable for conservative governor.
> > In summary, why cpu doesn't work at the lowest frequency even the cpu
> load is almost zero?
> >
> > root@lx2160ardb:/sys/devices/system/cpu# cat cpufreq/conservative/
> > down_threshold        freq_step             ignore_nice_load
> sampling_down_factor  sampling_rate         up_threshold
> > root@lx2160ardb:/sys/devices/system/cpu# cat cpufreq/conservative/*
> > 20
> 
> Try increasing down_threshold to a value of 50 (i.e. we reduce frequency if
> the load is below 50%) and see if it makes a difference.
I tried down_threshold = 50, it is better. But some cpus still don't work at the lowest frequency sometimes.

> 
> I believe something is running on the CPU, that you are unaware of.
> Try doing tracing to see what's going on all CPUs.
I keep monitoring cpu usage for several minutes, by top and mpstat tool, and found
most of the cpus are 100% idle, once in a while one of the cpus increases the load to about 18%.

Technically if the cpu usage is not more than up_threshold(80), cpu won't increase its frequency and keep
at the lowest frequency when cpu is at the lowest frequency and conservative governor is used. 
So I can't understand why cpu frequency increased?

Thanks,
Andy
> 
> --
> viresh

Re: [EXT] Re: Ask for help about cpufreq issue

[Viresh Kumar]

On 13-02-20, 08:18, Andy Tang wrote:
> I keep monitoring cpu usage for several minutes, by top and mpstat tool, and found
> most of the cpus are 100% idle, once in a while one of the cpus increases the load to about 18%.

I don't think top will be a good way for doing this test as we are
talking about load on a CPU in interval in milliseconds. Please check
with tracing and see what's keeping the CPU busy..

> Technically if the cpu usage is not more than up_threshold(80), cpu won't increase its frequency and keep
> at the lowest frequency when cpu is at the lowest frequency and conservative governor is used. 

I agree. Which means that there are some spikes of work getting
scheduled on CPUs. The load will increase slowly and will also
decrease slowly (step-by-step) with conservative governor.

> So I can't understand why cpu frequency increased?

Tracing or debugging cs_dbs_update() in conservative governor is the
only way out I would suggest.

-- 
viresh

RE: [EXT] Re: Ask for help about cpufreq issue

[Andy Tang]

Hi Viresh,

> -----Original Message-----
> From: Viresh Kumar <viresh.kumar@linaro.org>
> Sent: 2020年2月13日 16:48
> To: Andy Tang <andy.tang@nxp.com>
> Cc: rjw@rjwysocki.net; linux-pm@vger.kernel.org
> Subject: Re: [EXT] Re: Ask for help about cpufreq issue
> 
> I agree. Which means that there are some spikes of work getting scheduled
> on CPUs. The load will increase slowly and will also decrease slowly
> (step-by-step) with conservative governor.
> 
> > So I can't understand why cpu frequency increased?
> 
> Tracing or debugging cs_dbs_update() in conservative governor is the only
> way out I would suggest.
What's the *TRACING" here referring to? Could you please explain more about it?
I have no idea what it is.

Thanks,
Andy
> 
> --
> viresh

Re: [EXT] Re: Ask for help about cpufreq issue

[Viresh Kumar]

On 13-02-20, 09:12, Andy Tang wrote:
> Hi Viresh,
> 
> > -----Original Message-----
> > From: Viresh Kumar <viresh.kumar@linaro.org>
> > Sent: 2020年2月13日 16:48
> > To: Andy Tang <andy.tang@nxp.com>
> > Cc: rjw@rjwysocki.net; linux-pm@vger.kernel.org
> > Subject: Re: [EXT] Re: Ask for help about cpufreq issue
> > 
> > I agree. Which means that there are some spikes of work getting scheduled
> > on CPUs. The load will increase slowly and will also decrease slowly
> > (step-by-step) with conservative governor.
> > 
> > > So I can't understand why cpu frequency increased?
> > 
> > Tracing or debugging cs_dbs_update() in conservative governor is the only
> > way out I would suggest.
> What's the *TRACING" here referring to? Could you please explain more about it?
> I have no idea what it is.

Sorry about that, I was talking about the in-kernel function tracer
here.

https://www.kernel.org/doc/Documentation/trace/ftrace.txt

-- 
viresh

RE: [EXT] Re: Ask for help about cpufreq issue

[Andy Tang]

Appreciate your help.

Thanks,
Andy


> -----Original Message-----
> From: Viresh Kumar <viresh.kumar@linaro.org>
> Sent: 2020年2月13日 17:16
> To: Andy Tang <andy.tang@nxp.com>
> Cc: rjw@rjwysocki.net; linux-pm@vger.kernel.org
> Subject: Re: [EXT] Re: Ask for help about cpufreq issue
> 
> Caution: EXT Email
> 
> On 13-02-20, 09:12, Andy Tang wrote:
> > Hi Viresh,
> >
> > > -----Original Message-----
> > > From: Viresh Kumar <viresh.kumar@linaro.org>
> > > Sent: 2020年2月13日 16:48
> > > To: Andy Tang <andy.tang@nxp.com>
> > > Cc: rjw@rjwysocki.net; linux-pm@vger.kernel.org
> > > Subject: Re: [EXT] Re: Ask for help about cpufreq issue
> > >
> > > I agree. Which means that there are some spikes of work getting
> > > scheduled on CPUs. The load will increase slowly and will also
> > > decrease slowly
> > > (step-by-step) with conservative governor.
> > >
> > > > So I can't understand why cpu frequency increased?
> > >
> > > Tracing or debugging cs_dbs_update() in conservative governor is the
> > > only way out I would suggest.
> > What's the *TRACING" here referring to? Could you please explain more
> about it?
> > I have no idea what it is.
> 
> Sorry about that, I was talking about the in-kernel function tracer here.
> 
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.
> kernel.org%2Fdoc%2FDocumentation%2Ftrace%2Fftrace.txt&amp;data=02%
> 7C01%7Candy.tang%40nxp.com%7Ce9d217e1e493426ff8a708d7b0654c77%
> 7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C63717182142741422
> 1&amp;sdata=HXTASVHLewwjZhX3W9dna9wD87arEgK95snQP7UNZgc%3D
> &amp;reserved=0
> 
> --
> viresh

RE: [EXT] Re: Ask for help about cpufreq issue

[Andy Tang]

Hi Viresh,

> -----Original Message-----
> From: Viresh Kumar <viresh.kumar@linaro.org>
> Sent: 2020年2月13日 16:48
> To: Andy Tang <andy.tang@nxp.com>
> Cc: rjw@rjwysocki.net; linux-pm@vger.kernel.org
> Subject: Re: [EXT] Re: Ask for help about cpufreq issue
> 
 
> On 13-02-20, 08:18, Andy Tang wrote:
> > I keep monitoring cpu usage for several minutes, by top and mpstat
> > tool, and found most of the cpus are 100% idle, once in a while one of the
> cpus increases the load to about 18%.
> 
> I don't think top will be a good way for doing this test as we are talking about
> load on a CPU in interval in milliseconds. Please check with tracing and see
> what's keeping the CPU busy..
> 
> > Technically if the cpu usage is not more than up_threshold(80), cpu
> > won't increase its frequency and keep at the lowest frequency when cpu is
> at the lowest frequency and conservative governor is used.
> 
> I agree. Which means that there are some spikes of work getting scheduled
> on CPUs. The load will increase slowly and will also decrease slowly
> (step-by-step) with conservative governor.
> 
> > So I can't understand why cpu frequency increased?
> 
> Tracing or debugging cs_dbs_update() in conservative governor is the only
> way out I would suggest.
From cs_dbs_update() function, I did see that sometimes the LOAD is at 100% usage.
There is almost no application running. That must be some spikes of working running.
Unfortunately I can't track them by cs_dbs_update() or ftrace tool.
I doubt the ftrace can be used this way. Do you have any other suggestions? Maybe another tools?

BR,
Andy
> 
> --
> viresh

Re: [EXT] Re: Ask for help about cpufreq issue

[Viresh Kumar]

On 14-02-20, 10:00, Andy Tang wrote:
> From cs_dbs_update() function, I did see that sometimes the LOAD is at 100% usage.
> There is almost no application running. That must be some spikes of working running.
> Unfortunately I can't track them by cs_dbs_update() or ftrace tool.
> I doubt the ftrace can be used this way. Do you have any other suggestions? Maybe another tools?

Not sure if you used ftrace correctly, do following steps:

Enable following options in kernel config and rebuild your image.
- CONFIG_FTRACE
- CONFIG_FUNCTION_TRACER

and then run this script at runtime.


-------------------------8<-------------------------
#!/bin/bash
# Locals
BUFFER=40000
FILE="dft"

# What to trace ??
#EVENTS=" -e sched -e power -e workqueue -e irq"
EVENTS="all"

echo "Starting trace"

/usr/bin/trace-cmd start -b $BUFFER $EVENTS

#You can run some test here that you want to trace.
sleep 5

/usr/bin/trace-cmd stop

cat /sys/kernel/debug/tracing/trace > $FILE.txt
/usr/bin/trace-cmd extract -o $FILE.dat

echo "Saved trace in: "$FILE
sync
-------------------------8<-------------------------

The output file here (dft.dat) can be opened with kernel-shark tool.

Your testing platform's userspace must have trace-cmd command for this to work.

-- 
viresh