linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* Optimize perf stat for large number of events/cpus
@ 2019-11-07 18:16 Andi Kleen
  2019-11-07 18:16 ` [PATCH v5 01/13] perf pmu: Use file system cache to optimize sysfs access Andi Kleen
                   ` (12 more replies)
  0 siblings, 13 replies; 29+ messages in thread
From: Andi Kleen @ 2019-11-07 18:16 UTC (permalink / raw)
  To: jolsa; +Cc: acme, linux-kernel

[v5: Address review feedback. Split patches.]

This patch kit optimizes perf stat for a large number of events 
on systems with many CPUs and PMUs.

Some profiling shows that the most overhead is doing IPIs to
all the target CPUs. We can optimize this by using sched_setaffinity
to set the affinity to a target CPU once and then doing
the perf operation for all events on that CPU. This requires
some restructuring, but cuts the set up time quite a bit.

In theory we could go further by parallelizing these setups
too, but that would be much more complicated and for now just batching it
per CPU seems to be sufficient. At some point with many more cores 
parallelization or a better bulk perf setup API might be needed though.

In addition perf does a lot of redundant /sys accesses with
many PMUs, which can be also expensve. This is also optimized.

On a large test case (>700 events with many weak groups) on a 94 CPU
system I go from

real	0m8.607s
user	0m0.550s
sys	0m8.041s

to 

real	0m3.269s
user	0m0.760s
sys	0m1.694s

so shaving ~6 seconds of system time, at slightly more cost
in perf stat itself. On a 4 socket system with the savings
are more dramatic:

real	0m15.641s
user	0m0.873s
sys	0m14.729s

to 

real	0m4.493s
user	0m1.578s
sys	0m2.444s

so 11s difference in the user visible set up time.

Also available in 

git://git.kernel.org/pub/scm/linux/kernel/git/ak/linux-misc perf/stat-scale-8

v1: Initial post.
v2: Rebase. Fix some minor issues.
v3: Rebase. Address review feedback. Fix one minor issue
v4: Modified based on review feedback. Now it maintains
all_cpus per evlist. There is still a need for cpu_index iteration
to get the correct index for indexing the file descriptors.
Fix bug with unsorted cpu maps, now they are always sorted.
Some cleanups and refactoring.
v5: Split patches. Redo loop iteration again. Fix cpu map
merging for uncore. Remove duplicates from cpumaps. Add unit
tests.

-Andi


^ permalink raw reply	[flat|nested] 29+ messages in thread

end of thread, other threads:[~2019-11-12  0:41 UTC | newest]

Thread overview: 29+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-11-07 18:16 Optimize perf stat for large number of events/cpus Andi Kleen
2019-11-07 18:16 ` [PATCH v5 01/13] perf pmu: Use file system cache to optimize sysfs access Andi Kleen
2019-11-07 18:16 ` [PATCH v5 02/13] perf affinity: Add infrastructure to save/restore affinity Andi Kleen
2019-11-07 18:16 ` [PATCH v5 03/13] perf cpumap: Maintain cpumaps ordered and without dups Andi Kleen
2019-11-07 18:16 ` [PATCH v5 04/13] perf evlist: Maintain evlist->all_cpus Andi Kleen
2019-11-11 13:31   ` Jiri Olsa
2019-11-07 18:16 ` [PATCH v5 05/13] perf evsel: Add iterator to iterate over events ordered by CPU Andi Kleen
2019-11-07 18:16 ` [PATCH v5 06/13] perf evsel: Add functions to close evsel on a CPU Andi Kleen
2019-11-07 18:16 ` [PATCH v5 07/13] perf stat: Use affinity for closing file descriptors Andi Kleen
2019-11-11 13:30   ` Jiri Olsa
2019-11-11 16:56     ` Andi Kleen
2019-11-07 18:16 ` [PATCH v5 08/13] perf stat: Factor out open error handling Andi Kleen
2019-11-07 18:16 ` [PATCH v5 09/13] perf evsel: Support opening on a specific CPU Andi Kleen
2019-11-11 13:30   ` Jiri Olsa
2019-11-12  0:41     ` Andi Kleen
2019-11-07 18:16 ` [PATCH v5 10/13] perf stat: Use affinity for opening events Andi Kleen
2019-11-11 13:30   ` Jiri Olsa
2019-11-11 13:30   ` Jiri Olsa
2019-11-11 13:31   ` Jiri Olsa
2019-11-11 17:02     ` Andi Kleen
2019-11-07 18:16 ` [PATCH v5 11/13] perf stat: Use affinity for reading Andi Kleen
2019-11-11 13:31   ` Jiri Olsa
2019-11-07 18:16 ` [PATCH v5 12/13] perf evsel: Add functions to enable/disable for a specific CPU Andi Kleen
2019-11-11 13:30   ` Jiri Olsa
2019-11-07 18:16 ` [PATCH v5 13/13] perf stat: Use affinity for enabling/disabling events Andi Kleen
2019-11-11 14:04   ` Jiri Olsa
2019-11-11 16:50     ` Andi Kleen
2019-11-11 20:06       ` Jiri Olsa
2019-11-11 23:31         ` Andi Kleen

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).