On Mittwoch, 1. Juni 2016 18:49:01 CEST M Kelly wrote:
> Arnaldo Carvalho de Melo <acme <at> kernel.org> writes:
> > No need for DWARF if you want to know what leads to the spinlock just up
> > to the syscall layer or if your userland components have frame pointers.
> 
> Hi,
> 
> ok, I rebuilt our app using -fno-omit-frame-pointer
> and did the perf record -g and when looking at the
> report output I now see:
> 
>     40.90%  thorslave_mytho  [kernel.kallsyms]                   [k]
> _spin_lock
> 
>             --- _spin_lock
> 
>                |--49.11%-- futex_wake
>                |
>                |          do_futex
>                |          sys_futex
>                |          system_call_fastpath
>                |          __lll_unlock_wake
>                |          
>                |          |--99.81%-- CKeyCursor::gtEqual(char const*,
>                |          |char*, bool)
>                |          |
>                |          |          _ZN16CKeyLevelManager7_lookupEbj.constp
>                |          |          rop.100

<snip>

You are running an older perf, I recommend you rerun the report step and pass 
`-g graph`, the old default of `-g fractal` outputs relative-to-parent numbers 
which is often misleading.

The above already shows that you are suffering from lock contention issues, 
and where from (i.e. CKeyCursor::gtEqual). As the above only shows CPU time 
spent, I can only recommend to additionally look at sleep times, e.g. via 

https://perf.wiki.kernel.org/index.php/Tutorial#Profiling_sleep_times

You can easily put the above commands into a shell helper file, I did this:

https://github.com/milianw/shell-helpers/blob/master/perf-sleep-record

HTH
-- 
Milian Wolff | milian.wolff@kdab.com | Software Engineer
KDAB (Deutschland) GmbH&Co KG, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt Experts