All of lore.kernel.org
 help / color / mirror / Atom feed
From: David Miller <davem@davemloft.net>
To: acme@kernel.org
Cc: linux-kernel@vger.kernel.org, kan.liang@intel.com
Subject: perf synthesized mmap timeouts
Date: Sat, 27 Oct 2018 21:41:02 -0700 (PDT)	[thread overview]
Message-ID: <20181027.214102.1558835285408950686.davem@davemloft.net> (raw)


If I understand the commit message for:

commit 8cc42de736b617827a4e7664fb8d7a325bc125bc
Author: Kan Liang <kan.liang@intel.com>
Date:   Thu Jan 18 13:26:32 2018 -0800

    perf top: Check the latency of perf_top__mmap_read()

properly, the problem is that a malicious or out of control
app can be doing endless mmaps causing perf to loop forever
processing the /proc/$PID/maps file.

But that is not what this commit is handling at all.

It is instead applying a large hammer which quits if it is taking a
long time to process the maps, not if the process's mmap list is
growing endlessly while we process it.

This triggers any time I run perf top on a fully loaded system making
perf less useful than it should be.

And it triggers simply because the perf synthesize threads have to
share the cpu with the workload already running.

So it takes more than half a second to process emacs's 527 maps when
the number of running processes is ~NCPUS?  Big deal.  We should let
it finish....

The tradeoff choosen here is really bad.

Guess what happens if you don't have maps for a given process?

What happens is that for every single sample we get within that range,
we get a completely unique histogram entry.

This means potentially millions and millions of histogram entries
where there should only be a few hundred.

This makes the histogram rbtree huge, and slow to process.

So not only is top unable to provide correct histogram output, it is
also running sluggishly.

A way to mitigate the actual problem would be to snapshot the maps
file into a large buffer, if possible.  We can get the full contents
faster than the process in question can make more maps.  At most we
will do one additional read at the end if they were able to sneak in
one new mmap during the initial read.

No timeout necessary.  We have the complete maps file, our processing
time is therefore bounded.

Thanks.

             reply	other threads:[~2018-10-28  4:41 UTC|newest]

Thread overview: 2+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-10-28  4:41 David Miller [this message]
2018-10-29 13:46 ` perf synthesized mmap timeouts Liang, Kan

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20181027.214102.1558835285408950686.davem@davemloft.net \
    --to=davem@davemloft.net \
    --cc=acme@kernel.org \
    --cc=kan.liang@intel.com \
    --cc=linux-kernel@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.