Kernel Lock Analysis

Kernel Lock Analysis

[aleix]

[-- Attachment #1: Type: text/plain, Size: 2160 bytes --]

Hello List! This is my first message here :)

I would like to share my experience playing with Kernel lock
instrumentation, LTTng and Trace Compass which might be useful for
others when analyzing applications behavior.

Currently, the Linux kernel features a set of lock tracepoints which
track lock acquisition and contention [1]:

 - lock_acquire
 - lock_acquired
 - lock_contended
 - lock_release

However, the current version of LTTng does not instrument this
tracepoints (the code is in the source code but commented [2]). The
guys on LTTng IRC helped me enabling them again by just uncommenting
the code.

Thanks to these tracepoints, I could create a couple of Trace Compass
views (also attached in this mail) to easily track the kernel lock
contention, which clearly showed me what was going on. Please, see the
attached screenshot for an example. The application under analysis is
a parallel cholesky benchmark run on a server with 56 CPUs. I was
trying to figure out why almost all application threads became blocked
at some point as seen in the screenshot. The lock view showed that
there was a huge contention on the mm->mmap_sem lock when all threads
tried to allocate memory by calling mmap(), mmprotect() and triggered
page faults when data is written on the recently mmapped memory.

Hence, what I would like to point out is how useful it has been for me
to enable the LTTng lock tracepoints. I think it would be great if
they could be added back again into mainland. If this can be done, I
think it woulde make sense to propose the Trace Compass people to
include a kernel lock view.

It looks like the lock events are by far the most frequent events,
quickly filling LTTng buffers. However, they are only generated if the
kernel is compiled with CONFIG_LOCK_STAT so this should not annoy the
unaware user.

I hope my experience is of help! Thanks a lot for your work! :D

 [1] See Documentation/locking/lockstat.txt on the Linux Kernel source
     for more information.
 [2] As Compudj pointed out, the reason is found in this conversation:
     https://lists.lttng.org/pipermail/lttng-dev/2012-December/019256.html


http://bsc.es/disclaimer

[-- Attachment #2: cholesky-lock-analysis.png --]
[-- Type: image/png, Size: 150163 bytes --]

[-- Attachment #3: lock_contention_analysis.xml --]
[-- Type: application/xml, Size: 4595 bytes --]

[-- Attachment #4: per_lock_analysis.xml --]
[-- Type: application/xml, Size: 2989 bytes --]

[-- Attachment #5: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: Kernel Lock Analysis

[Jonathan Rajotte-Julien]

Hi,

On Thu, Apr 05, 2018 at 11:17:57AM +0200, aleix wrote:
> Hello List! This is my first message here :)

Welcome!

> 
> I would like to share my experience playing with Kernel lock
> instrumentation, LTTng and Trace Compass which might be useful for
> others when analyzing applications behavior.
Before going further, you might want to repost this on the Tracecompass mailing
list:

https://accounts.eclipse.org/mailing-list/tracecompass-dev

They are most likely interested by all of this.

> 
> Currently, the Linux kernel features a set of lock tracepoints which
> track lock acquisition and contention [1]:
> 
>  - lock_acquire
>  - lock_acquired
>  - lock_contended
>  - lock_release
> 
> However, the current version of LTTng does not instrument this
> tracepoints (the code is in the source code but commented [2]). The
> guys on LTTng IRC helped me enabling them again by just uncommenting
> the code.
> ...
> 
> Hence, what I would like to point out is how useful it has been for me
> to enable the LTTng lock tracepoints. I think it would be great if
> they could be added back again into mainland.

Given your current use case, it make a compelling point toward spending some
time investigating the possibility of making those tracepoints accessible.

Would you mind creating a bug here [1] regarding this so we can track it in the
future?

[1] https://bugs.lttng.org/projects/lttng-modules/

> If this can be done, I
> think it woulde make sense to propose the Trace Compass people to
> include a kernel lock view.
> 
> It looks like the lock events are by far the most frequent events,
> quickly filling LTTng buffers. However, they are only generated if the
> kernel is compiled with CONFIG_LOCK_STAT so this should not annoy the
> unaware user.
> 
> I hope my experience is of help! Thanks a lot for your work! :D

Absolutely, we are always interested in how our tools are used in the wild.

Thanks for sharing.

> 
>  [1] See Documentation/locking/lockstat.txt on the Linux Kernel source
>      for more information.
>  [2] As Compudj pointed out, the reason is found in this conversation:
>      https://lists.lttng.org/pipermail/lttng-dev/2012-December/019256.html

-- 
Jonathan Rajotte-Julien
EfficiOS
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev