From: Alexey Budankov <alexey.budankov@linux.intel.com>
To: Mark Rutland <mark.rutland@arm.com>,
Vince Weaver <vincent.weaver@maine.edu>
Cc: Kyle Huey <me@kylehuey.com>, "Jin, Yao" <yao.jin@linux.intel.com>,
Ingo Molnar <mingo@kernel.org>,
"Peter Zijlstra (Intel)" <peterz@infradead.org>,
stable@vger.kernel.org,
Alexander Shishkin <alexander.shishkin@linux.intel.com>,
Arnaldo Carvalho de Melo <acme@redhat.com>,
Jiri Olsa <jolsa@redhat.com>,
Linus Torvalds <torvalds@linux-foundation.org>,
Namhyung Kim <namhyung@kernel.org>,
Stephane Eranian <eranian@google.com>,
Thomas Gleixner <tglx@linutronix.de>,
acme@kernel.org, jolsa@kernel.org, kan.liang@intel.com,
Will Deacon <will.deacon@arm.com>,
gregkh@linuxfoundation.org,
"Robert O'Callahan" <robert@ocallahan.org>,
open list <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH] perf/core: generate overflow signal when samples are dropped (WAS: Re: [REGRESSION] perf/core: PMU interrupts dropped if we entered the kernel in the "skid" region)
Date: Thu, 29 Jun 2017 11:25:41 +0300 [thread overview]
Message-ID: <20fc1c2f-bed7-a229-6fdf-4efa6f1f9cf2@linux.intel.com> (raw)
In-Reply-To: <9521d6c9-1177-3b73-2991-cb4be9eea681@linux.intel.com>
On 29.06.2017 11:13, Alexey Budankov wrote:
> Hi Folks,
>
> On 28.06.2017 16:07, Mark Rutland wrote:
>> On Wed, Jun 28, 2017 at 08:40:30AM -0400, Vince Weaver wrote:
>>> On Wed, 28 Jun 2017, Mark Rutland wrote:
>>>
>>>> On Wed, Jun 28, 2017 at 11:12:48AM +0100, Mark Rutland wrote:
>>>
>>>> Instead of bailing out early in perf_event_overflow, we can bail prior
>>>> to performing the actual sampling in __perf_event_output(). This avoids
>>>> the information leak, but preserves the generation of the signal.
>>>>
>>>> Since we don't place any sample data into the ring buffer, the signal is
>>>> arguably spurious. However, a userspace ringbuffer consumer can already
>>>> consume data prior to taking the associated signals, and therefore must
>>>> handle spurious signals to operate correctly. Thus, this signal
>>>> shouldn't be harmful.
>>>
>>> this could still break some of my perf_event validation tests.
>>>
>>> Ones that set up a sampling event for every 1M instructions, run for 100M
>>> instructions, and expect there to be 100 samples received.
>>
>> Is that test reliable today?
>>
>> I'd expect that at least on ARM it's not, given that events can be
>> counted imprecisely, and mode filters can be applied imprecisely. So you
>> might get fewer (or more) samples. I'd imagine similar is true on other
>> archtiectures.
>>
>> If sampling took long enough, the existing ratelimiting could come into
>> effect, too.
>>
>> Surely that already has some error margin?
>
> FYI.
>
>>From my recent experience and observation (on Intel Xeon Phi)
> wakeup_events_overflow and overflow_poll tests may fail if
> /proc/sys/kernel/perf_event_max_sample_rate is low enough:
>
> # cat /proc/sys/kernel/perf_event_max_sample_rate
> 6000
> # abudanko/perf_event_tests/tests/overflow/wakeup_events_overflow
> This tests wakeup event overflows.
> Testing with wakeup_events=1.
> Counts, using mmap buffer 0x7f707b0dc000
> POLL_IN : 10
> POLL_OUT: 0
> POLL_MSG: 0
> POLL_ERR: 0
> POLL_PRI: 0
> POLL_HUP: 0
> UNKNOWN : 0
> Testing wakeup events overflow... PASSED
> # abudanko/perf_event_tests/tests/overflow/overflow_poll
> This tests using poll() to catch overflow.
> Monitoring pid 131412 status 1407
> Child has stopped due to signal 5 (Trace/breakpoint trap)
> Continuing child
> Returned HUP!
> Counts, using mmap buffer 0x7fb3bfe04000
> POLL_IN : 10
> POLL_OUT: 0
> POLL_MSG: 0
> POLL_ERR: 0
> POLL_PRI: 0
> POLL_HUP: 1
> UNKNOWN : 0
> Testing catching overflow with poll()... PASSED
>
> # echo 1000 > /proc/sys/kernel/perf_event_max_sample_rate
> # abudanko/perf_event_tests/tests/overflow/overflow_poll
> This tests using poll() to catch overflow.
> Monitoring pid 131551 status 1407
> Child has stopped due to signal 5 (Trace/breakpoint trap)
> Continuing child
> Returned HUP!
> Counts, using mmap buffer 0x7f80532df000
> POLL_IN : 9
> POLL_OUT: 0
> POLL_MSG: 0
> POLL_ERR: 0
> POLL_PRI: 0
> POLL_HUP: 1
> UNKNOWN : 0
> Unexpected POLL_IN interrupt.
> Testing catching overflow with poll()... FAILED
> [root@nntpdsd52-210 ~]# abudanko/perf_event_tests/tests/overflow/overflow_poll
> This tests using poll() to catch overflow.
> Monitoring pid 131553 status 1407
> Child has stopped due to signal 5 (Trace/breakpoint trap)
> Continuing child
> Returned HUP!
> Counts, using mmap buffer 0x7f650952c000
> POLL_IN : 9
> POLL_OUT: 0
> POLL_MSG: 0
> POLL_ERR: 0
> POLL_PRI: 0
> POLL_HUP: 1
> UNKNOWN : 0
> Unexpected POLL_IN interrupt.
> Testing catching overflow with poll()... FAILED
More of the other test:
# echo 1000 > /proc/sys/kernel/perf_event_max_sample_rate
[ ~]# abudanko/perf_event_tests/tests/overflow/wakeup_events_overflow
This tests wakeup event overflows.
Testing with wakeup_events=1.
Counts, using mmap buffer 0x7fe65deff000
POLL_IN : 4
POLL_OUT: 0
POLL_MSG: 0
POLL_ERR: 0
POLL_PRI: 0
POLL_HUP: 0
UNKNOWN : 0
POLL_IN value 4, expected 10.
Testing wakeup events overflow... FAILED
[ ~]# abudanko/perf_event_tests/tests/overflow/wakeup_events_overflow
This tests wakeup event overflows.
Testing with wakeup_events=1.
Counts, using mmap buffer 0x7f818a732000
POLL_IN : 2
POLL_OUT: 0
POLL_MSG: 0
POLL_ERR: 0
POLL_PRI: 0
POLL_HUP: 0
UNKNOWN : 0
POLL_IN value 2, expected 10.
Testing wakeup events overflow... FAILED
[ ~]# abudanko/perf_event_tests/tests/overflow/wakeup_events_overflow
This tests wakeup event overflows.
Testing with wakeup_events=1.
Counts, using mmap buffer 0x7fb3675c5000
POLL_IN : 4
POLL_OUT: 0
POLL_MSG: 0
POLL_ERR: 0
POLL_PRI: 0
POLL_HUP: 0
UNKNOWN : 0
POLL_IN value 4, expected 10.
Testing wakeup events overflow... FAILED
>
>>
>>> If we're so worried about info leakage, can't we just zero-out the problem
>>> address (or randomize the kernel address) rather than just pretending the
>>> interrupt didn't happen?
>>
>> Making up zeroed or randomized data is going to confuse users. I can't
>> imagine that real users are going to want bogus samples that they have
>> to identify (somehow) in order to skip when processing the data.
>>
>> I can see merit in signalling "lost" samples to userspace, so long as
>> they're easily distinguished from real samples.
>>
>> One option is to fake up a sample using the user regs regardless, but
>> that's both fragile and surprising in other cases.
>>
>> Thanks,
>> Mark.
>>
>
> Thanks,
> Alexey
>
>
>
next prev parent reply other threads:[~2017-06-29 8:26 UTC|newest]
Thread overview: 37+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <CAP045Ap8cMx6mzSgcQ3n3bnh_8GJuCp7_KZe_5ZTCR_K6cPTLw@mail.gmail.com>
2017-06-28 1:01 ` [REGRESSION] perf/core: PMU interrupts dropped if we entered the kernel in the "skid" region Kyle Huey
2017-06-28 2:09 ` Jin, Yao
2017-06-28 4:51 ` Kyle Huey
2017-06-28 5:35 ` Jin, Yao
2017-06-28 7:30 ` Kyle Huey
2017-06-28 10:12 ` Mark Rutland
2017-06-28 10:56 ` [PATCH] perf/core: generate overflow signal when samples are dropped (WAS: Re: [REGRESSION] perf/core: PMU interrupts dropped if we entered the kernel in the "skid" region) Mark Rutland
2017-06-28 12:40 ` Vince Weaver
2017-06-28 13:07 ` Mark Rutland
2017-06-29 8:13 ` Alexey Budankov
2017-06-29 8:25 ` Alexey Budankov [this message]
2017-06-28 16:48 ` Kyle Huey
2017-06-28 17:49 ` Mark Rutland
2017-06-28 22:55 ` Kyle Huey
2017-06-29 0:27 ` Jin, Yao
2017-06-30 17:44 ` Kyle Huey
2017-07-04 9:03 ` Peter Zijlstra
2017-07-04 9:33 ` Mark Rutland
2017-07-04 9:46 ` Peter Zijlstra
2017-07-04 10:21 ` Mark Rutland
2017-07-06 5:07 ` Robert O'Callahan
2017-07-11 2:03 ` Kyle Huey
2017-07-11 9:03 ` Ingo Molnar
2017-07-11 13:07 ` Jin, Yao
2017-07-12 7:57 ` Ingo Molnar
2017-07-11 14:26 ` Mark Rutland
2017-07-11 15:32 ` Kyle Huey
2017-07-18 0:07 ` Kyle Huey
2017-06-29 8:12 ` Ingo Molnar
2017-07-04 9:06 ` Peter Zijlstra
2017-07-04 10:04 ` Ingo Molnar
2017-07-11 9:03 ` [tip:perf/urgent] Revert "perf/core: Drop kernel samples even though :u is specified" tip-bot for Ingo Molnar
2017-06-28 16:46 ` [REGRESSION] perf/core: PMU interrupts dropped if we entered the kernel in the "skid" region Kyle Huey
2017-06-28 17:19 ` Mark Rutland
2017-06-28 17:36 ` Kyle Huey
2017-06-28 17:52 ` Mark Rutland
2017-06-28 17:48 ` Robert O'Callahan
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20fc1c2f-bed7-a229-6fdf-4efa6f1f9cf2@linux.intel.com \
--to=alexey.budankov@linux.intel.com \
--cc=acme@kernel.org \
--cc=acme@redhat.com \
--cc=alexander.shishkin@linux.intel.com \
--cc=eranian@google.com \
--cc=gregkh@linuxfoundation.org \
--cc=jolsa@kernel.org \
--cc=jolsa@redhat.com \
--cc=kan.liang@intel.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mark.rutland@arm.com \
--cc=me@kylehuey.com \
--cc=mingo@kernel.org \
--cc=namhyung@kernel.org \
--cc=peterz@infradead.org \
--cc=robert@ocallahan.org \
--cc=stable@vger.kernel.org \
--cc=tglx@linutronix.de \
--cc=torvalds@linux-foundation.org \
--cc=vincent.weaver@maine.edu \
--cc=will.deacon@arm.com \
--cc=yao.jin@linux.intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).