From: Pekka Paalanen <pq@iki.fi>
To: Steven Rostedt <rostedt@goodmis.org>
Cc: "Frédéric Weisbecker" <fweisbec@gmail.com>,
"Ingo Molnar" <mingo@elte.hu>,
"Linux Kernel" <linux-kernel@vger.kernel.org>
Subject: Re: Tracing/ftrace: trouble with trace_entries and trace_pipe
Date: Tue, 16 Sep 2008 21:01:14 +0300 [thread overview]
Message-ID: <20080916210114.7679a0dc@daedalus.pq.iki.fi> (raw)
In-Reply-To: <alpine.DEB.1.10.0809151556260.22620@gandalf.stny.rr.com>
On Mon, 15 Sep 2008 17:14:34 -0400 (EDT)
Steven Rostedt <rostedt@goodmis.org> wrote:
>
> On Mon, 15 Sep 2008, Pekka Paalanen wrote:
>
> > Hi Steven and others,
> >
> > first a minor bug: it seems the units of /debug/tracing/trace_entries
> > is different for read and write. This is confusing for the users, since
> > I can't say "if you have problems, double it". If I read from it
> > something like 16422, then write back 16422, next I read 214. I can't
> > recall the exact numbers, but the difference really is two orders of
> > magnitude. I have 1 GB RAM in this box, so that shouldn't be an issue.
>
> You write to it the same number that you read from and it returned
> something different?? That is indeed a bug, since it should definitely
> detect that. Is this linux-tip? I'll have to play with it to make sure
> nothing broke it recently.
>
> I just tried the latest mainline, and it seemed to work there.
Yes, it returned a very different number. This is Ingo's tip/master,
sorry for not being explicit. Checked-out on Sunday.
Echoing the following numbers to trace_entries triggers it:
200, 64, 640, 16422, 16422, 16422, 164220...
so the ridiculously small number 64 does something bad. After 640,
I read back something less than 200. Each following write increases
that number by 46.
16422 is the default before writing anything.
That is even stranger than I thought. :-o
I was going to test the buffer overflow with mmiotrace.
> > My other problem is with trace_pipe. It is again making 'cat' quit too
> > early. The condition triggered is
> > if (!tracer_enabled && iter->pos) {
> > in tracing_read_pipe(), and it is followed by triggering
> > /* stop when tracing is finished */
> > if (trace_empty(iter)) {
> > and then sret=0, so read returns 0 and 'cat' exits.
> >
> > Now, I am trying my mmiotrace marker patches, but as far as I can tell,
> > nothing I modified is the reason for this. I didn't yet explicitly test
> > for it, though. I'll send these patches after I hear from Frederic.
> >
> > The cat-quit problem is not a constant state. After boot, I could play
> > with my markers and testmmiotrace without cat quitting. Then something
> > happens, and cat starts the quitting behaviour, and won't get to normal
> > by disabling and enabling mmiotrace.
> >
> > I have a couple of wild guesses of what might be related:
> > - ring buffer wrap-around
> > - ring buffer overflow (at first try I hit these, the second try
> > after putting debug-pr_info's in place I don't hit this)
> > - ring buffer resize (after playing with trace_entries, cat-quit
> > problem was present, though it might have been present before)
> >
> > After viewing the git history, I have some more guesses, mainly
> > related to setting tracer_enabled to 0.
> > - commit 2b1bce1787700768cbc87c8509851c6f49d252dc
> > I don't see where tracer_enabled would be set to 1, when
> > mmiotrace is enabled. It used to default to 1 and mmiotrace was happy.
> > - __tracing_open() sets it to 0 (not called for the pipe)
> > - tracing_release() sets it to 1
> > - tracing_ctrl_write() toggles it
> > - tracing_read_pipe() tests it
> > - tracer_alloc_buffers() uses it
> > And other tracers seem to use it a lot.
> >
> > Mmiotrace does use the tracer::ctrl_update hook, and allow/disallow
> > calls to __trace_mmiotrace_{rw,map}() via enabling/disabling the whole
> > mmiotrace core. Is this not enough, or is it inappropriate?
> >
> > It seems tracer_enabled is used by the trace framework itself to
> > enable/disable... what? Hmm, maybe nothing I care about.
> >
> > Should mmiotrace simply do
> > tracer_enabled = 1;
> > in mmio_trace_init()?
> >
> > Should mmiotrace test tracer_enabled, and if so, when?
>
> No, tracer_enabled is something that is internal to the tracer
> infrastructure.
>
> if you read from either tracing/trace or tracing/latency_trace it will
> disable the tracer while you dump. But you should not be doing that. The
> trace_pipe should not disable that either.
Ok, so the problem is probably the commit I mentioned. It makes the
no_tracer tracer to set tracer_enabled to 0, and I can't find where
it would be set to 1 for mmiotrace. And this interferes with
tracing_read_pipe(), making it quit when iter->pos is non-zero.
See no_trace_init() in trace.c. According to this, the cat-quit
occurs when the buffer gets empty after first data, but this isn't
totally in agreement with what I recall from my experiments. And it
does happen also on other times than injecting markers.
So either it is wrong to check tracer_enabled in tracing_read_pipe(),
or no_trace_init() should not touch it.
Steven, what do you think?
--
Pekka Paalanen
http://www.iki.fi/pq/
next prev parent reply other threads:[~2008-09-16 18:01 UTC|newest]
Thread overview: 36+ messages / expand[flat|nested] mbox.gz Atom feed top
2008-08-24 21:42 [Patch] Tracing/ftrace: Adds a marker to allow user comments Frédéric Weisbecker
2008-08-25 8:44 ` Ingo Molnar
2008-08-25 13:59 ` Frédéric Weisbecker
2008-08-25 16:38 ` Steven Rostedt
2008-08-25 17:52 ` Frédéric Weisbecker
2008-08-27 9:59 ` Frédéric Weisbecker
2008-08-27 18:21 ` Pekka Paalanen
2008-08-28 9:44 ` Ingo Molnar
2008-08-28 16:03 ` Frédéric Weisbecker
2008-08-28 10:04 ` Frédéric Weisbecker
2008-08-28 18:42 ` Pekka Paalanen
2008-09-04 16:20 ` Frédéric Weisbecker
2008-09-04 17:30 ` Pekka Paalanen
2008-09-04 18:11 ` Steven Rostedt
2008-09-06 11:39 ` Frédéric Weisbecker
2008-09-06 13:49 ` Pekka Paalanen
2008-09-15 19:47 ` Tracing/ftrace: trouble with trace_entries and trace_pipe Pekka Paalanen
2008-09-15 21:14 ` Steven Rostedt
2008-09-16 18:01 ` Pekka Paalanen [this message]
2008-09-17 12:41 ` Frédéric Weisbecker
2008-09-17 17:36 ` Pekka Paalanen
2008-09-16 18:54 ` [PATCH 1/7] x86 mmiotrace: fix a rare memory leak Pekka Paalanen
2008-09-16 18:56 ` [PATCH 2/7] ftrace: move mmiotrace functions out of trace.c Pekka Paalanen
2008-09-16 18:58 ` [PATCH 3/7] ftrace: add trace_vprintk() Pekka Paalanen
2008-09-16 20:06 ` Steven Rostedt
2008-09-17 11:42 ` Ingo Molnar
2008-09-16 19:00 ` [PATCH 4/7] x86 mmiotrace: implement mmiotrace_printk() Pekka Paalanen
2008-09-16 19:02 ` [PATCH 5/7] mmiotrace: handle TRACE_PRINT entries Pekka Paalanen
2008-09-16 20:11 ` Steven Rostedt
2008-09-16 21:24 ` Pekka Paalanen
2008-09-16 19:03 ` [PATCH 6/7] mmiotrace: remove left-over marker cruft Pekka Paalanen
2008-09-16 19:06 ` [PATCH 7/7] ftrace: inject markers via trace_marker file Pekka Paalanen
2008-09-16 8:57 ` Tracing/ftrace: trouble with trace_entries and trace_pipe Frédéric Weisbecker
2008-09-07 14:11 ` [Patch] Tracing/ftrace: Adds a marker to allow user comments Pekka Paalanen
2008-09-07 17:29 ` Steven Rostedt
2008-09-08 17:19 ` Pekka Paalanen
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20080916210114.7679a0dc@daedalus.pq.iki.fi \
--to=pq@iki.fi \
--cc=fweisbec@gmail.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@elte.hu \
--cc=rostedt@goodmis.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).