From: "Jin, Yao" <yao.jin@linux.intel.com>
To: Kyle Huey <me@kylehuey.com>
Cc: Ingo Molnar <mingo@kernel.org>,
"Peter Zijlstra (Intel)" <peterz@infradead.org>,
stable@vger.kernel.org,
Alexander Shishkin <alexander.shishkin@linux.intel.com>,
Arnaldo Carvalho de Melo <acme@redhat.com>,
Jiri Olsa <jolsa@redhat.com>,
Linus Torvalds <torvalds@linux-foundation.org>,
Namhyung Kim <namhyung@kernel.org>,
Stephane Eranian <eranian@google.com>,
Thomas Gleixner <tglx@linutronix.de>,
Vince Weaver <vincent.weaver@maine.edu>,
acme@kernel.org, jolsa@kernel.org, kan.liang@intel.com,
Mark Rutland <mark.rutland@arm.com>,
Will Deacon <will.deacon@arm.com>,
gregkh@linuxfoundation.org,
"Robert O'Callahan" <robert@ocallahan.org>,
open list <linux-kernel@vger.kernel.org>,
yao.jin@intel.com
Subject: Re: [REGRESSION] perf/core: PMU interrupts dropped if we entered the kernel in the "skid" region
Date: Wed, 28 Jun 2017 13:35:47 +0800 [thread overview]
Message-ID: <d66c1793-8498-b27b-8473-1685293550b4@linux.intel.com> (raw)
In-Reply-To: <CAP045AryPCUTO92S3hmrkag3D7NqgJM-hK82a7iKm9s-rWdn1w@mail.gmail.com>
Hi Kyle,
I understand your requirement. Sorry I don't know the detail of rr
debugger, but I guess if it just uses counter overflow to deliver a
signal, could it set the counter without "exclude_kernel"?
Another consideration is, I'm not sure if the kernel can accurately
realize that if the interrupt is to trigger sampling or just deliver a
signal. Userspace may know that but kernel may not.
Anyway let's listen to more comments from community.
Thanks
Jin Yao
On 6/28/2017 12:51 PM, Kyle Huey wrote:
> On Tue, Jun 27, 2017 at 7:09 PM, Jin, Yao <yao.jin@linux.intel.com> wrote:
>> Hi,
>>
>> In theory, the PMI interrupts in skid region should be dropped, right?
> No, why would they be dropped?
>
> My understanding of the situation is as follows:
>
> There is some time, call it t_0, where the hardware counter overflows.
> The PMU triggers an interrupt, but this is not instantaneous. Call
> the time when the interrupt is actually delivered t_1. Then t_1 - t_0
> is the "skid".
>
> Note that if the counter is `exclude_kernel`, then at t_0 the CPU
> *must* be running a userspace program. But by t_1, the CPU may be
> doing something else. Your patch changed things so that if at t_1 the
> CPU is in the kernel, then the interrupt is discarded. But rr has
> programmed the counter to deliver a signal on overflow (via F_SETSIG
> on the fd returned by perf_event_open). This change results in the
> signal never being delivered, because the interrupt was ignored.
> (More accurately, the signal is delivered the *next* time the counter
> overflows, which is far past where we wanted to inject our
> asynchronous event into our tracee.
>
> It seems to me that it might be reasonable to ignore the interrupt if
> the purpose of the interrupt is to trigger sampling of the CPUs
> register state. But if the interrupt will trigger some other
> operation, such as a signal on an fd, then there's no reason to drop
> it.
>
>> For a userspace debugger, is it the only choice that relies on the *skid*
>> PMI interrupt?
> I don't understand this question, but hopefully the above clarified things.
>
> - Kyle
>
>> Thanks
>> Jin Yao
>>
>>
>> On 6/28/2017 9:01 AM, Kyle Huey wrote:
>>> Sent again with LKML CCd, sorry for the noise.
>>>
>>> - Kyle
>>>
>>> On Tue, Jun 27, 2017 at 5:38 PM, Kyle Huey <me@kylehuey.com> wrote:
>>>> cc1582c231ea introduced a regression in v4.12.0-rc5, and appears to be
>>>> a candidate for backporting to stable branches.
>>>>
>>>> rr, a userspace record and replay debugger[0], uses the PMU interrupt
>>>> to stop a program during replay to inject asynchronous events such as
>>>> signals. We are counting retired conditional branches in userspace
>>>> only. This changeset causes the kernel to drop interrupts on the
>>>> floor if, during the PMU interrupt's "skid" region, the CPU enters
>>>> kernel mode for whatever reason. When replaying traces of complex
>>>> programs such as Firefox, we intermittently fail to deliver
>>>> asynchronous events on time, leading the replay to diverge from the
>>>> recorded state.
>>>>
>>>> It seems like this change should, at a bare minimum, be limited to
>>>> counters that actually perform sampling of register state when the
>>>> interrupt fires. In our case, with the retired conditional branches
>>>> counter restricted to counting userspace events only, it makes no
>>>> difference that the PMU interrupt happened to be delivered in the
>>>> kernel.
>>>>
>>>> As this makes rr unusable on complex applications and cannot be
>>>> efficiently worked around, we would appreciate this being addressed
>>>> before 4.12 is finalized, and the regression not being introduced to
>>>> stable branches.
>>>>
>>>> Thanks,
>>>>
>>>> - Kyle
>>>>
>>>> [0] http://rr-project.org/
>>
next prev parent reply other threads:[~2017-06-28 5:35 UTC|newest]
Thread overview: 37+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <CAP045Ap8cMx6mzSgcQ3n3bnh_8GJuCp7_KZe_5ZTCR_K6cPTLw@mail.gmail.com>
2017-06-28 1:01 ` [REGRESSION] perf/core: PMU interrupts dropped if we entered the kernel in the "skid" region Kyle Huey
2017-06-28 2:09 ` Jin, Yao
2017-06-28 4:51 ` Kyle Huey
2017-06-28 5:35 ` Jin, Yao [this message]
2017-06-28 7:30 ` Kyle Huey
2017-06-28 10:12 ` Mark Rutland
2017-06-28 10:56 ` [PATCH] perf/core: generate overflow signal when samples are dropped (WAS: Re: [REGRESSION] perf/core: PMU interrupts dropped if we entered the kernel in the "skid" region) Mark Rutland
2017-06-28 12:40 ` Vince Weaver
2017-06-28 13:07 ` Mark Rutland
2017-06-29 8:13 ` Alexey Budankov
2017-06-29 8:25 ` Alexey Budankov
2017-06-28 16:48 ` Kyle Huey
2017-06-28 17:49 ` Mark Rutland
2017-06-28 22:55 ` Kyle Huey
2017-06-29 0:27 ` Jin, Yao
2017-06-30 17:44 ` Kyle Huey
2017-07-04 9:03 ` Peter Zijlstra
2017-07-04 9:33 ` Mark Rutland
2017-07-04 9:46 ` Peter Zijlstra
2017-07-04 10:21 ` Mark Rutland
2017-07-06 5:07 ` Robert O'Callahan
2017-07-11 2:03 ` Kyle Huey
2017-07-11 9:03 ` Ingo Molnar
2017-07-11 13:07 ` Jin, Yao
2017-07-12 7:57 ` Ingo Molnar
2017-07-11 14:26 ` Mark Rutland
2017-07-11 15:32 ` Kyle Huey
2017-07-18 0:07 ` Kyle Huey
2017-06-29 8:12 ` Ingo Molnar
2017-07-04 9:06 ` Peter Zijlstra
2017-07-04 10:04 ` Ingo Molnar
2017-07-11 9:03 ` [tip:perf/urgent] Revert "perf/core: Drop kernel samples even though :u is specified" tip-bot for Ingo Molnar
2017-06-28 16:46 ` [REGRESSION] perf/core: PMU interrupts dropped if we entered the kernel in the "skid" region Kyle Huey
2017-06-28 17:19 ` Mark Rutland
2017-06-28 17:36 ` Kyle Huey
2017-06-28 17:52 ` Mark Rutland
2017-06-28 17:48 ` Robert O'Callahan
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=d66c1793-8498-b27b-8473-1685293550b4@linux.intel.com \
--to=yao.jin@linux.intel.com \
--cc=acme@kernel.org \
--cc=acme@redhat.com \
--cc=alexander.shishkin@linux.intel.com \
--cc=eranian@google.com \
--cc=gregkh@linuxfoundation.org \
--cc=jolsa@kernel.org \
--cc=jolsa@redhat.com \
--cc=kan.liang@intel.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mark.rutland@arm.com \
--cc=me@kylehuey.com \
--cc=mingo@kernel.org \
--cc=namhyung@kernel.org \
--cc=peterz@infradead.org \
--cc=robert@ocallahan.org \
--cc=stable@vger.kernel.org \
--cc=tglx@linutronix.de \
--cc=torvalds@linux-foundation.org \
--cc=vincent.weaver@maine.edu \
--cc=will.deacon@arm.com \
--cc=yao.jin@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).