From: Steven Rostedt <rostedt@goodmis.org>
To: Allende Imanol <iallende@ikerlan.es>
Cc: "linux-trace-devel@vger.kernel.org"
<linux-trace-devel@vger.kernel.org>,
"mingo@kernel.org" <mingo@kernel.org>
Subject: Re: Deadlock with FTrace in child process
Date: Thu, 24 Oct 2019 21:18:50 -0400 [thread overview]
Message-ID: <20191024211850.613279c6@gandalf.local.home> (raw)
In-Reply-To: <DB8PR03MB589769D8EFD0DCE8AEC6D283B6650@DB8PR03MB5897.eurprd03.prod.outlook.com>
On Fri, 25 Oct 2019 00:44:56 +0000
Allende Imanol <iallende@ikerlan.es> wrote:
> > On Sat, 19 Oct 2019 07:50:53 +0000
> > Allende Imanol <iallende@ikerlan.es> wrote:
> >
> > > I am using Ftrace in a Ultra96 (arm64) 4.19.75-cip11 and it seems that
> > > I have a deadlock situation. I have a simple application executed as a
> > > child process from a python script. The child process reads from
> > > /dev/urandom and writes in /dev/zero. Ftrace is set only for this child
> > > process.
> >
> > Are you saying that ftrace is calling the blocked task?
> >
> > If you don't run ftrace, the task never gets stuck?
>
> Without FTrace I did not get to achieve the same stuck scenario.
>
OK, so you see this with function graph tracer enabled. The lockup is
that its stuck on a fast user-space mutex (futex). As function graph
does nothing to touch user space, but it does greatly effect timings
(function-graph is the slowest of the tracers), I'm guessing you have a
race that causes a deadlock in your application somewhere, but without
function graph tracing enabled, the race window is too small to trigger
it.
I've hit bugs like this before. Ftrace isn't the cause, it's the
instigator to help the real bug show its face.
Try limiting what gets traced, by either entering in functions in the
set_ftrace_notrace file, or entering in functions in the
set_ftrace_filter file. If you put in just a few functions in the
set_ftrace_filter file and you still see an issue, then you have a case
that it is the function graph tracer.
Start looking at the trace, and adding the most common functions used
into the set_ftrace_notrace file and see if the bug goes away.
Really, nothing in ftrace should cause an hang, except the fact that
you application has a race condition in it that ftrace makes the race
window big enough to trigger.
-- Steve
prev parent reply other threads:[~2019-10-25 1:18 UTC|newest]
Thread overview: 4+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-10-19 7:50 Deadlock with FTrace in child process Allende Imanol
2019-10-21 14:41 ` Steven Rostedt
2019-10-25 0:44 ` Allende Imanol
2019-10-25 1:18 ` Steven Rostedt [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20191024211850.613279c6@gandalf.local.home \
--to=rostedt@goodmis.org \
--cc=iallende@ikerlan.es \
--cc=linux-trace-devel@vger.kernel.org \
--cc=mingo@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).