From: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
To: rostedt <rostedt@goodmis.org>
Cc: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>,
linux-kernel <linux-kernel@vger.kernel.org>,
Peter Zijlstra <peterz@infradead.org>
Subject: Re: Possible use of RCU while in extended QS: idle vs RCU read-side in interrupt vs rcu_eqs_exit
Date: Thu, 10 Jan 2019 12:25:13 -0500 (EST) [thread overview]
Message-ID: <1083900143.1198.1547141113001.JavaMail.zimbra@efficios.com> (raw)
In-Reply-To: <600900741.1177.1547140315581.JavaMail.zimbra@efficios.com>
----- On Jan 10, 2019, at 9:11 AM, Mathieu Desnoyers mathieu.desnoyers@efficios.com wrote:
> ----- On Jan 10, 2019, at 8:44 AM, Mathieu Desnoyers
> mathieu.desnoyers@efficios.com wrote:
>
>> ----- On Jan 10, 2019, at 8:08 AM, rostedt rostedt@goodmis.org wrote:
>>
>>> On Wed, 9 Jan 2019 20:38:51 -0500 (EST)
>>> Mathieu Desnoyers <mathieu.desnoyers@efficios.com> wrote:
>>>
>>>> Hi Paul,
>>>>
>>>> I've had a user report that trace_sched_waking() appears to be
>>>> invoked while !rcu_is_watching() in some situation, so I started
>>>> digging into the scheduler idle code.
>>>
>>> I'm wondering if this isn't a bug. Do you have the backtrace for where
>>> trace_sched_waking() was called without rcu watching?
>>
>> I strongly suspect a bug as well. I'm awaiting a reproducer from the
>> user whom reported this issue so I can add a WARN_ON_ONCE(!rcu_is_watching())
>> in the scheduler code near trace_sched_waking() and gather a backtrace.
>>
>> It still has to be confirmed, but I suspect this have been triggered
>> within a HyperV guest. It may therefore be related to a virtualized environment.
>>
>> I'll try to ask more specifically on which environment this was encountered.
>
> So it ends up it happens directly on hardware on a Linux laptop. Here is
> the stacktrace:
>
> vmlinux!try_to_wake_up
> vmlinux!default_wake_function
> vmlinux!pollwake
> vmlinux!__wake_up_common
> vmlinux!__wake_up_common_lock
> vmlinux!__wake_up
> vmlinux!perf_event_wakeup
> vmlinux!perf_pending_event
> vmlinux!irq_work_run_list
> vmlinux!irq_work_run
> vmlinux!smp_irq_work_iterrupt
> vmlinux!irq_work_interrupt
> vmlinux!finish_task_switch
> vmlinux!__schedule
> vmlinux!schedule_idle
> vmlinux!do_idle
> vmlinux!cpu_startup_entry
> vmlinux!start_secondary
> vmlinux!secondary_startup_64
>
> Does it raise any red flag ?
Based on this backtrace, I think I start to get a better understanding
of the situation.
The initial problem reported to me was that ftrace was showing some sched_waking
events in its trace that were missed by perf.
I presumed this was because of the !rcu_is_watching() check, but I think I was
wrong.
This backtrace seems to show that perf is itself triggering a sched_waking()
event. It there is probably a check that discards nested events within perf,
which would discard this from perf traces, but ftrace (and lttng) would trace
it just fine.
Thoughts ?
Thanks,
Mathieu
>
> Thanks,
>
> Mathieu
>
>>
>> Thanks,
>>
>> Mathieu
>>
>>>
>>> -- Steve
>>>
>>>>
>>>> It appears that interrupts are re-enabled before rcu_eqs_exit() is
>>>> invoked when exiting idle code from the scheduler.
>>>>
>>>> I wonder what happens if an interrupt handler (including scheduler code)
>>>> happens to issue a RCU read-side critical section before rcu_eqs_exit()
>>>> is called ? Is there some code on interrupt entry that ensures rcu eqs
>>>> state is exited in such scenario ?
>>>>
>>>> Thanks,
>>>>
>>>> Mathieu
>>
>> --
>> Mathieu Desnoyers
>> EfficiOS Inc.
>> http://www.efficios.com
>
> --
> Mathieu Desnoyers
> EfficiOS Inc.
> http://www.efficios.com
--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
next prev parent reply other threads:[~2019-01-10 17:25 UTC|newest]
Thread overview: 14+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-01-10 1:38 Possible use of RCU while in extended QS: idle vs RCU read-side in interrupt vs rcu_eqs_exit Mathieu Desnoyers
2019-01-10 4:13 ` Paul E. McKenney
2019-01-10 6:30 ` Mathieu Desnoyers
2019-01-10 14:19 ` Paul E. McKenney
2019-01-10 16:08 ` Steven Rostedt
2019-01-10 16:44 ` Mathieu Desnoyers
2019-01-10 17:11 ` Mathieu Desnoyers
2019-01-10 17:23 ` Steven Rostedt
2019-01-10 17:25 ` Mathieu Desnoyers [this message]
2019-01-10 17:45 ` Perf: event wakeup discards sched_waking events Mathieu Desnoyers
2019-01-14 13:09 ` Peter Zijlstra
2019-01-14 21:36 ` Mathieu Desnoyers
2019-01-14 22:04 ` Steven Rostedt
2019-01-14 22:31 ` Mathieu Desnoyers
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1083900143.1198.1547141113001.JavaMail.zimbra@efficios.com \
--to=mathieu.desnoyers@efficios.com \
--cc=linux-kernel@vger.kernel.org \
--cc=paulmck@linux.vnet.ibm.com \
--cc=peterz@infradead.org \
--cc=rostedt@goodmis.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).