kernelnewbies.kernelnewbies.org archive mirror
 help / color / mirror / Atom feed
* cpu-freq: running the perf increases the data rate?
@ 2020-08-27 16:55 Subhashini Rao Beerisetty
  2020-08-28 12:34 ` Artem Bityutskiy
  0 siblings, 1 reply; 3+ messages in thread
From: Subhashini Rao Beerisetty @ 2020-08-27 16:55 UTC (permalink / raw)
  To: linux-pm, kernelnewbies, LKML

 [ Please keep me in CC as I'm not subscribed to the list]

Hi all,

I have an application which finds the data rate over the PCIe
interface. I’m getting the lesser data rate in one of my Linux X86
systems.
When I change the scaling_governor from "powersave" to "performance"
mode for each CPU, then there is slight improvement in the PCIe data
rate.
Parallely I started profiling the workload with perf. Whenever I start
running the profile command “perf stat -a -d -p <PID>” surprisingly
the application resulted in excellent data rate over PCIe, but when I
kill the perf command again PCIe data rate drops. I am really confused
about this behavior.Any clues from this behaviour?


Also I noticed my system not having the 'cpuinfo_cur_freq' sys file.
Is that okay?
cat: /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_cur_freq: No such
file or directory


-- 
Thanks,

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: cpu-freq: running the perf increases the data rate?
  2020-08-27 16:55 cpu-freq: running the perf increases the data rate? Subhashini Rao Beerisetty
@ 2020-08-28 12:34 ` Artem Bityutskiy
  2020-08-31 16:22   ` Subhashini Rao Beerisetty
  0 siblings, 1 reply; 3+ messages in thread
From: Artem Bityutskiy @ 2020-08-28 12:34 UTC (permalink / raw)
  To: Subhashini Rao Beerisetty, linux-pm, kernelnewbies, LKML

On Thu, 2020-08-27 at 22:25 +0530, Subhashini Rao Beerisetty wrote:
> I have an application which finds the data rate over the PCIe
> interface. I’m getting the lesser data rate in one of my Linux X86
> systems.

Some more description, may be? Do you have a PCIe device reading one
RAM buffer and then writing to another RAM buffer? Or does it generate
dome data and writes them to a RAM buffer? Presumably it uses DMA? How
much is the CPU involved into the process? Are we talking about
transferring few kilobytes or gigabytes?

> When I change the scaling_governor from "powersave" to "performance"
> mode for each CPU, then there is slight improvement in the PCIe data
> rate.

Definitely this makes your CPU(s) run at max speed, but depending on
platform and settings, this may also affect C-states. Are the CPU(s)
generally idle while you measure, or busy (involved into the test)? You
could run 'turbostat' while measuring the bandwidth, to get some CPU
statistics (e.g., do C-states happen during the PCI test, how busy are
the CPUs).

> Parallely I started profiling the workload with perf. Whenever I start
> running the profile command “perf stat -a -d -p <PID>” surprisingly
> the application resulted in excellent data rate over PCIe, but when I
> kill the perf command again PCIe data rate drops. I am really confused
> about this behavior.Any clues from this behaviour?

Well, one possible reason that comes to mind - you get rid of C-states
when you rung perf, and this increases the PCI bandwidth. You can just
try disabling C-states (there are sysfs knobs) and check it out.
Turbostat could be useful to check for this (with and without perf, run
'turbostat sleep 10' or something like this (measure for 10 seconds in
this example), do this while running your PCI test.

But I am really just guessing here, I do not know enough about your
test and the system (e.g., "a Linux x86" system can be so many things,
like Intel or AMD server or a mobile device)...



_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: cpu-freq: running the perf increases the data rate?
  2020-08-28 12:34 ` Artem Bityutskiy
@ 2020-08-31 16:22   ` Subhashini Rao Beerisetty
  0 siblings, 0 replies; 3+ messages in thread
From: Subhashini Rao Beerisetty @ 2020-08-31 16:22 UTC (permalink / raw)
  To: dedekind1; +Cc: kernelnewbies, LKML, linux-pm

[-- Attachment #1: Type: text/plain, Size: 3332 bytes --]

On Fri, Aug 28, 2020 at 6:04 PM Artem Bityutskiy <dedekind1@gmail.com> wrote:
>
> On Thu, 2020-08-27 at 22:25 +0530, Subhashini Rao Beerisetty wrote:
> > I have an application which finds the data rate over the PCIe
> > interface. I’m getting the lesser data rate in one of my Linux X86
> > systems.
>
> Some more description, may be? Do you have a PCIe device reading one
> RAM buffer and then writing to another RAM buffer? Or does it generate
> dome data and writes them to a RAM buffer? Presumably it uses DMA? How
> much is the CPU involved into the process? Are we talking about
> transferring few kilobytes or gigabytes?
Thanks a lot for your help and reply.
Regarding hardware setup, Xilinx PCIe FPGA endpoint is connected to
HOST CPU via PCIe bus.
Xilinx PCIe FPGA endpoint has the DMA_REF block and it provides a
mechanism to DMA transfer data at the maximum rate between host CPU
memory and a FIFO in the DMA-REF block.
The host software sets up some data in it’s memory, it then transfers
the data to the DMA-REF’s FIFO and then reads it back into a different
location in its host memory. This is repeated in a loop.  There is a
register in the DMA-REF block that gives an indication of transfer
speed.


>
> > When I change the scaling_governor from "powersave" to "performance"
> > mode for each CPU, then there is slight improvement in the PCIe data
> > rate.
>
> Definitely this makes your CPU(s) run at max speed, but depending on
> platform and settings, this may also affect C-states. Are the CPU(s)
> generally idle while you measure, or busy (involved into the test)? You
> could run 'turbostat' while measuring the bandwidth, to get some CPU
> statistics (e.g., do C-states happen during the PCI test, how busy are
> the CPUs).
>
> > Parallely I started profiling the workload with perf. Whenever I start
> > running the profile command “perf stat -a -d -p <PID>” surprisingly
> > the application resulted in excellent data rate over PCIe, but when I
> > kill the perf command again PCIe data rate drops. I am really confused
> > about this behavior.Any clues from this behaviour?
>
> Well, one possible reason that comes to mind - you get rid of C-states
> when you rung perf, and this increases the PCI bandwidth. You can just
> try disabling C-states (there are sysfs knobs) and check it out.
> Turbostat could be useful to check for this (with and without perf, run
> 'turbostat sleep 10' or something like this (measure for 10 seconds in
> this example), do this while running your PCI test.
Disabling the C-states improved the throughput a lot, thanks a lot for
pointing this out. Could you please give some more explanation on how
disabling C-states improved the throughput?
As you suggested I collected and attached the turbostat log with and
without perf while running the PCIe test.
In my system, only 'performance\powersave' are listed in
scaling_available_governors. Rest other governors
"userspace\ondemand\schedutil" are not listed in available_goverors.
What might be the reason for this?

>
> But I am really just guessing here, I do not know enough about your
> test and the system (e.g., "a Linux x86" system can be so many things,
> like Intel or AMD server or a mobile device)…
It's an Intel Atom processor.
>
>

[-- Attachment #2: trubostat_with_perf.txt --]
[-- Type: text/plain, Size: 2633 bytes --]

turbostat version 17.06.23 - Len Brown <lenb@kernel.org>
CPUID(0): GenuineIntel 11 CPUID levels; family:model:stepping 0x6:37:9 (6:55:9)
CPUID(1): SSE3 MONITOR - EIST TM2 TSC MSR ACPI-TM TM
CPUID(6): APERF, No-TURBO, DTS, No-PTM, No-HWP, No-HWPnotify, No-HWPwindow, No-HWPepp, No-HWPpkg, EPB
cpu2: MSR_IA32_MISC_ENABLE: 0x00850089 (TCC EIST No-MWAIT PREFETCH TURBO)
CPUID(7): No-SGX
SLM BCLK: 83.3 Mhz
cpu2: MSR_CC6_DEMOTION_POLICY_CONFIG: 0x00000000 (DISable-CC6-Demotion)
cpu2: MSR_MC6_DEMOTION_POLICY_CONFIG: 0x00000000 (DISable-MC6-Demotion)
RAPL: 4581 sec. Joule Counter Range, at 30 Watts
cpu2: MSR_PLATFORM_INFO: 0x60000001700
6 * 83.3 = 499.8 MHz max efficiency frequency
23 * 83.3 = 1915.9 MHz base frequency
cpu2: MSR_IA32_POWER_CTL: 0x00000000 (C1E auto-promotion: DISabled)
cpu2: MSR_ATOM_CORE_RATIOS: 0x00170602
2 * 83.3 = 166.6 MHz minimum operating frequency
6 * 83.3 = 499.8 MHz low frequency mode (LFM)
23 * 83.3 = 1915.9 MHz base frequency
cpu2: MSR_ATOM_CORE_TURBO_RATIOS: 0x17171717
23 * 83.3 = 1915.9 MHz max turbo 4 active cores
23 * 83.3 = 1915.9 MHz max turbo 3 active cores
23 * 83.3 = 1915.9 MHz max turbo 2 active cores
23 * 83.3 = 1915.9 MHz max turbo 1 active core
cpu2: MSR_PKG_CST_CONFIG_CONTROL: 0x0017000f (UNlocked: pkg-cstate-limit=15: pc7)
cpu2: POLL: CPUIDLE CORE POLL IDLE
cpu2: C1: MWAIT 0x00
cpu2: C6N: MWAIT 0x58
cpu2: C6S: MWAIT 0x52
cpu2: cpufreq driver: intel_pstate
cpu2: cpufreq governor: performance
cpufreq intel_pstate no_turbo: 1
cpu0: MSR_IA32_ENERGY_PERF_BIAS: 0x00000006 (balanced)
cpu0: MSR_RAPL_POWER_UNIT: 0x00000505 (0.031250 Watts, 0.000032 Joules, 0.000977 sec.)
cpu0: MSR_PKG_POWER_LIMIT: 0x003880fa (UNlocked)
cpu0: PKG Limit #1: ENabled (7.812500 Watts, 262144.000000 sec, clamp DISabled)
cpu0: PKG Limit #2: DISabled (0.000000 Watts, 0.000977* sec, clamp DISabled)
cpu0: MSR_PP0_POWER_LIMIT: 0x00020000 (UNlocked)
cpu0: Cores Limit: DISabled (0.000000 Watts, 0.001953 sec, clamp DISabled)
cpu0: MSR_IA32_TEMPERATURE_TARGET: 0x006e0000 (110 C)
40.004417 sec
Core	CPU	Avg_MHz	Busy%	Bzy_MHz	TSC_MHz	IRQ	SMI	C1	C6N	C6S	C1%	C6N%	C6S%	CPU%c1	CPU%c6	Mod%c6	CoreTmp	GFX%rc6	Pkg%pc6	PkgWatt	CorWatt
-	-	748	39.33	1901	1917	458560	0	1566280	1889	7775	43.07	0.29	18.19	42.23	17.85	13.56	29	100.00	11.80	1.14	0.95
0	0	643	33.89	1896	1917	33647	0	405339	492	1950	48.44	0.31	18.28	47.57	17.69	13.32	28	100.00	11.80	1.14	0.95
1	1	1064	55.78	1906	1917	358521	0	449875	25	1910	27.37	0.03	17.83	26.41	17.34	13.32	29
2	2	647	34.14	1895	1917	33821	0	359155	517	1940	48.21	0.29	18.20	47.43	17.86	13.81	28
3	3	638	33.52	1902	1917	32571	0	351911	855	1975	48.27	0.54	18.47	47.51	18.51	13.81	28

[-- Attachment #3: trubostat_without_perf.txt --]
[-- Type: text/plain, Size: 2642 bytes --]

turbostat version 17.06.23 - Len Brown <lenb@kernel.org>
CPUID(0): GenuineIntel 11 CPUID levels; family:model:stepping 0x6:37:9 (6:55:9)
CPUID(1): SSE3 MONITOR - EIST TM2 TSC MSR ACPI-TM TM
CPUID(6): APERF, No-TURBO, DTS, No-PTM, No-HWP, No-HWPnotify, No-HWPwindow, No-HWPepp, No-HWPpkg, EPB
cpu2: MSR_IA32_MISC_ENABLE: 0x00850089 (TCC EIST No-MWAIT PREFETCH TURBO)
CPUID(7): No-SGX
SLM BCLK: 83.3 Mhz
cpu2: MSR_CC6_DEMOTION_POLICY_CONFIG: 0x00000000 (DISable-CC6-Demotion)
cpu2: MSR_MC6_DEMOTION_POLICY_CONFIG: 0x00000000 (DISable-MC6-Demotion)
RAPL: 4581 sec. Joule Counter Range, at 30 Watts
cpu2: MSR_PLATFORM_INFO: 0x60000001700
6 * 83.3 = 499.8 MHz max efficiency frequency
23 * 83.3 = 1915.9 MHz base frequency
cpu2: MSR_IA32_POWER_CTL: 0x00000000 (C1E auto-promotion: DISabled)
cpu2: MSR_ATOM_CORE_RATIOS: 0x00170602
2 * 83.3 = 166.6 MHz minimum operating frequency
6 * 83.3 = 499.8 MHz low frequency mode (LFM)
23 * 83.3 = 1915.9 MHz base frequency
cpu2: MSR_ATOM_CORE_TURBO_RATIOS: 0x17171717
23 * 83.3 = 1915.9 MHz max turbo 4 active cores
23 * 83.3 = 1915.9 MHz max turbo 3 active cores
23 * 83.3 = 1915.9 MHz max turbo 2 active cores
23 * 83.3 = 1915.9 MHz max turbo 1 active core
cpu2: MSR_PKG_CST_CONFIG_CONTROL: 0x0017000f (UNlocked: pkg-cstate-limit=15: pc7)
cpu2: POLL: CPUIDLE CORE POLL IDLE
cpu2: C1: MWAIT 0x00
cpu2: C6N: MWAIT 0x58
cpu2: C6S: MWAIT 0x52
cpu2: cpufreq driver: intel_pstate
cpu2: cpufreq governor: performance
cpufreq intel_pstate no_turbo: 1
cpu0: MSR_IA32_ENERGY_PERF_BIAS: 0x00000006 (balanced)
cpu0: MSR_RAPL_POWER_UNIT: 0x00000505 (0.031250 Watts, 0.000032 Joules, 0.000977 sec.)
cpu0: MSR_PKG_POWER_LIMIT: 0x003880fa (UNlocked)
cpu0: PKG Limit #1: ENabled (7.812500 Watts, 262144.000000 sec, clamp DISabled)
cpu0: PKG Limit #2: DISabled (0.000000 Watts, 0.000977* sec, clamp DISabled)
cpu0: MSR_PP0_POWER_LIMIT: 0x00020000 (UNlocked)
cpu0: Cores Limit: DISabled (0.000000 Watts, 0.001953 sec, clamp DISabled)
cpu0: MSR_IA32_TEMPERATURE_TARGET: 0x006e0000 (110 C)
40.003807 sec
Core	CPU	Avg_MHz	Busy%	Bzy_MHz	TSC_MHz	IRQ	SMI	C1	C6N	C6S	C1%	C6N%	C6S%	CPU%c1	CPU%c6	Mod%c6	CoreTmp	GFX%rc6	Pkg%pc6	PkgWatt	CorWatt
-	-	522	27.23	1915	1917	285074	0	784132	86336	168235	31.17	13.41	28.89	30.75	26.97	3.75	30	100.00	3.60	1.12	0.94
0	0	412	21.50	1915	1917	11229	0	155840	13229	69746	24.68	7.95	46.57	24.35	35.28	3.80	29	100.00	3.60	1.12	0.94
1	1	764	39.91	1914	1917	242649	0	269925	25179	32418	26.95	13.44	20.52	26.38	19.26	3.80	29
2	2	476	24.82	1915	1917	16549	0	224377	27839	1269	48.85	21.52	5.44	48.36	19.04	3.70	30
3	3	435	22.70	1915	1917	14647	0	133990	20089	64802	24.19	10.74	43.05	23.90	34.28	3.70	30

[-- Attachment #4: Type: text/plain, Size: 170 bytes --]

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

^ permalink raw reply	[flat|nested] 3+ messages in thread

end of thread, other threads:[~2020-09-05  2:36 UTC | newest]

Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-08-27 16:55 cpu-freq: running the perf increases the data rate? Subhashini Rao Beerisetty
2020-08-28 12:34 ` Artem Bityutskiy
2020-08-31 16:22   ` Subhashini Rao Beerisetty

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).