Linux-Trace-Devel Archive on lore.kernel.org
 help / color / Atom feed
From: Steven Rostedt <rostedt@goodmis.org>
To: Allende Imanol <iallende@ikerlan.es>
Cc: "linux-trace-devel@vger.kernel.org" 
	<linux-trace-devel@vger.kernel.org>,
	"mingo@kernel.org" <mingo@kernel.org>
Subject: Re: Deadlock with FTrace in child process
Date: Thu, 24 Oct 2019 21:18:50 -0400
Message-ID: <20191024211850.613279c6@gandalf.local.home> (raw)
In-Reply-To: <DB8PR03MB589769D8EFD0DCE8AEC6D283B6650@DB8PR03MB5897.eurprd03.prod.outlook.com>

On Fri, 25 Oct 2019 00:44:56 +0000
Allende Imanol <iallende@ikerlan.es> wrote:

> > On Sat, 19 Oct 2019 07:50:53 +0000
> > Allende Imanol <iallende@ikerlan.es> wrote:
> >   
> > > I am using Ftrace in a Ultra96 (arm64) 4.19.75-cip11 and it seems that
> > > I have a deadlock situation. I have a simple application executed as a
> > > child process from a python script. The child process reads from
> > > /dev/urandom and writes in /dev/zero. Ftrace is set only for this child
> > > process.  
> > 
> > Are you saying that ftrace is calling the blocked task?
> > 
> > If you don't run ftrace, the task never gets stuck?  
> 
> Without FTrace I did not get to achieve the same stuck scenario.
> 

OK, so you see this with function graph tracer enabled. The lockup is
that its stuck on a fast user-space mutex (futex). As function graph
does nothing to touch user space, but it does greatly effect timings
(function-graph is the slowest of the tracers), I'm guessing you have a
race that causes a deadlock in your application somewhere, but without
function graph tracing enabled, the race window is too small to trigger
it.

I've hit bugs like this before. Ftrace isn't the cause, it's the
instigator to help the real bug show its face.

Try limiting what gets traced, by either entering in functions in the
set_ftrace_notrace file, or entering in functions in the
set_ftrace_filter file. If you put in just a few functions in the
set_ftrace_filter file and you still see an issue, then you have a case
that it is the function graph tracer.

Start looking at the trace, and adding the most common functions used
into the set_ftrace_notrace file and see if the bug goes away.

Really, nothing in ftrace should cause an hang, except the fact that
you application has a race condition in it that ftrace makes the race
window big enough to trigger.

-- Steve

      reply index

Thread overview: 4+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-10-19  7:50 Allende Imanol
2019-10-21 14:41 ` Steven Rostedt
2019-10-25  0:44   ` Allende Imanol
2019-10-25  1:18     ` Steven Rostedt [this message]

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20191024211850.613279c6@gandalf.local.home \
    --to=rostedt@goodmis.org \
    --cc=iallende@ikerlan.es \
    --cc=linux-trace-devel@vger.kernel.org \
    --cc=mingo@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-Trace-Devel Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-trace-devel/0 linux-trace-devel/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-trace-devel linux-trace-devel/ https://lore.kernel.org/linux-trace-devel \
		linux-trace-devel@vger.kernel.org
	public-inbox-index linux-trace-devel

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-trace-devel


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git