From mboxrd@z Thu Jan 1 00:00:00 1970 From: Amit Margalit Subject: Re: Question about filtering bytecode Date: Mon, 22 Jul 2013 09:14:47 +0300 Message-ID: References: Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============0216331006==" Return-path: Received: from e06smtp13.uk.ibm.com ([195.75.94.109]) by ltt.polymtl.ca with esmtp (Exim 4.72) (envelope-from ) id 1V19Om-0003Xr-Hm for lttng-dev@lists.lttng.org; Mon, 22 Jul 2013 02:15:14 -0400 Received: from /spool/local by e06smtp13.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Mon, 22 Jul 2013 07:10:23 +0100 Received: from b06cxnps4074.portsmouth.uk.ibm.com (d06relay11.portsmouth.uk.ibm.com [9.149.109.196]) by d06dlp01.portsmouth.uk.ibm.com (Postfix) with ESMTP id A56D917D8059 for ; Mon, 22 Jul 2013 07:16:38 +0100 (BST) Received: from d06av12.portsmouth.uk.ibm.com (d06av12.portsmouth.uk.ibm.com [9.149.37.247]) by b06cxnps4074.portsmouth.uk.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id r6M6Elqf27132108 for ; Mon, 22 Jul 2013 06:14:47 GMT Received: from d06av12.portsmouth.uk.ibm.com (localhost [127.0.0.1]) by d06av12.portsmouth.uk.ibm.com (8.14.4/8.14.4/NCO v10.0 AVout) with ESMTP id r6M6EwKw005866 for ; Mon, 22 Jul 2013 00:14:58 -0600 In-Reply-To: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: lttng-dev-bounces@lists.lttng.org To: =?ISO-8859-1?Q?J=E9r=E9mie_Galarneau?= Cc: lttng-dev@lists.lttng.org, jeremie.galarneau@gmail.com List-Id: lttng-dev@lists.lttng.org This is a multipart message in MIME format. --===============0216331006== Content-Type: multipart/alternative; boundary="=_alternative 00223F3BC2257BB0_=" This is a multipart message in MIME format. --=_alternative 00223F3BC2257BB0_= Content-Type: text/plain; charset="ISO-8859-1" Content-Transfer-Encoding: quoted-printable Thanks for the explanations. I currently feel that the effort to get such a thing working is not going=20 to materialize in the timeframe required for my current work, and=20 therefore, regrettably, I will have to work around this somehow. I am=20 currently thinking of using a combination of Python and Cython to generate = a wrapper for the babeltrace iterator. We'll see. Thanks, Amit Margalit IBM XIV - Storage Reinvented XIV-NAS Development Team Tel. 03-689-7774 Fax. 03-689-7230 From: J=E9r=E9mie Galarneau To: Amit Margalit/Israel/IBM@IBMIL Cc: lttng-dev@lists.lttng.org Date: 07/21/2013 10:07 PM Subject: Re: [lttng-dev] Question about filtering bytecode Sent by: jeremie.galarneau@gmail.com On Sun, Jul 21, 2013 at 1:51 AM, Amit Margalit wrote: > Hi, > > I was thinking more about a programmer's guide to using this mechanism. > Unfortunately there is no programmer's guide on filters, but feel free to ask about specifics. > What I'd like to do is to have Babeltrace filter events it is showing me > from an existing trace. > > Consider the following example: > > There are 3 types of tracepoints being captured (without filtering): > > A debug log message (occasionally) which includes a "debug=5Flevel" value > (1..10 with 1 being highest importance). > Function entry (via gcc -finstrument-functions or other=20 instrumentation). > Function exit (similarly). > > During a 1-hour period, I have 2-3 GB of trace data captured. > The trace includes events from 4 different processes. > > > If I use Babeltrace to read the trace I will get an enormously big=20 result in > text (~30-40GB at least). > > I would like to be able to do something like this: > > $ babeltrace --filter '$ctx.procname =3D=3D "smbd" || debug=5Flevel < 3' > /local/lttng-traces/ust/uid/2140/64-bit > > This would then let me see any events from the "smbd" process (since i=20 added > procname to the context using lttng add-context) or debug messages from=20 all > processes where the debug level is 1 or 2. > That would be a neat feature! There are currently no plans to support filtering in Babeltrace, but that contribution would be welcome. lttng-ust's filtering code is licensed under LGPL v2.1 so it could be library-fied and used by the ctf-text plug-in. In the meantime, I guess you could work around this limitation with a combination of AWK and shell scripting... > Regarding linking - I am referring to the function > lttng=5Fenabler=5Fevent=5Flink=5Fbytecode(), called from > lttng-ust/liblttng-ust/lttng-events.c around line 628. > Thanks for clarifying! Keep in mind that the filter's byte code is generated by the lttng binary without knowing the events' declarations. Thus, the byte code must go through a linking stage not completely unlike what is done by a regular linker. One of the objectives of the linking stage is setting the appropriate offsets to load an event's fields. The byte code is also "specialized" by resolving each field's type and making sure the appropriate implementation of each operator is used. For instance, the FILTER=5FOP=5FEQ (check for equality) op-code is not implemented the same way for integers, doubles and strings (see lttng=5Ffilter=5Fspecialize=5Fbytecode in lttng-filter-specialize.c). You may want to have a look at =5Flttng=5Ffilter=5Fevent=5Flink=5Fbytecode (lttng-ust/liblttng-ust/lttng-filter.c, line 318) to see how each step of the linking process is implemented. Regards, J=E9r=E9mie > Thanks, > > Amit Margalit > IBM XIV - Storage Reinvented > XIV-NAS Development Team > Tel. 03-689-7774 > Fax. 03-689-7230 > > > > From: J=E9r=E9mie Galarneau > To: Amit Margalit/Israel/IBM@IBMIL > Cc: lttng-dev@lists.lttng.org > Date: 07/18/2013 07:10 PM > Subject: Re: [lttng-dev] Question about filtering bytecode > Sent by: jeremie.galarneau@gmail.com > =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F > > > > Hi Amit, > > You may find information on the Babeltrace API in babeltrace/doc/API.txt > The filtering mechanism used by the user space tracer is unrelated to > Babeltrace. > > As for the sequence of events, this is how I understand it. > > When invoking the "lttng" binary with --enable-event and a filter > option, a byte code is generated from the filtering rules > (lttng-tools/src/lib/lttng-ctl/lttng-ctl.c:704). This byte code is > then handed off to the session-daemon. > > Then, when an instrumented application is launched and connects to the > session-daemon, it receives the byte code while its events are being > enabled. To answer your question directly, that byte code will be > executed by the tracee when the event is "triggered" > (lttng-ust/liblttng-ust/lttng-filter-interpreter.c:175). > > I'm not sure I understand your question about filter/event linking. A > filter has to be linked to an event since its rules (expressions) are > applied on that event's values. Could you clarify if that's not what > you meant? > > Regards, > J=E9r=E9mie > > On Thu, Jul 18, 2013 at 3:12 AM, Amit Margalit wrote: >> Hi, >> >> I'm considering whether I can write a wrapper for the libbabeltrace ctf >> iterator which provides a filtered "next()" method. >> >> I've started reading the code in the hopes of understanding how to do >> this, >> and yet, I'd really appreciate a link to some ordered documentation, or = - >> any other kind of explanation. >> >> For example, one thing I still don't understand is the exact sequence,=20 and >> where the bytecode is actually executed? What is the meaning and=20 purpose >> of >> "linking" the filter to an event. >> >> Thanks, >> >> Amit Margalit >> IBM XIV - Storage Reinvented >> XIV-NAS Development Team >> Tel. 03-689-7774 >> Fax. 03-689-7230 >> =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F >> lttng-dev mailing list >> lttng-dev@lists.lttng.org >> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev >> > > > > -- > J=E9r=E9mie Galarneau > EfficiOS Inc. > http://www.efficios.com > > --=20 J=E9r=E9mie Galarneau EfficiOS Inc. http://www.efficios.com --=_alternative 00223F3BC2257BB0_= Content-Type: text/html; charset="ISO-8859-1" Content-Transfer-Encoding: quoted-printable Thanks for the explanations.

I currently feel that the effort to get such a thing working is not going to materialize in the timeframe requi= red for my current work, and therefore, regrettably, I will have to work around this somehow. I am currently thinking of using a combination of Python and Cython to generate a wrapper for the babeltrace iterator. We'll see.

Thanks,

Amit Margalit
IBM XIV - Storage Reinvented
XIV-NAS Development Team
Tel. 03-689-7774
Fax. 03-689-7230



From:     =    J=E9r=E9mie Galarneau <j= eremie.galarneau@efficios.com>
To:     &n= bsp;  Amit Margalit/Israel/IBM@IB= MIL
Cc:     &n= bsp;  lttng-dev@lists.lttng.org
Date:     =    07/21/2013 10:07 PM
Subject:   &nbs= p;    Re: [lttng-dev] Question about filtering bytecode
Sent by:   &nbs= p;    jeremie.galarneau@gm= ail.com



On Sun, Jul 21, 2013 at 1:51 AM, Amit Margalit <A= MITM@il.ibm.com> wrote:
> Hi,
>
> I was thinking more about a programmer's guide to using this mechanism= .
>

Unfortunately there is no programmer's guide on filters, but feel free
to ask about specifics.

> What I'd like to do is to have Babeltrace filter events it is showing me
> from an existing trace.
>
> Consider the following example:
>
> There are 3 types of tracepoints being captured (without filtering): >
> A debug log message (occasionally) which includes a "debug=5Fleve= l" value
> (1..10 with 1 being highest importance).
> Function entry (via gcc -finstrument-functions or other instrumentatio= n).
> Function exit (similarly).
>
> During a 1-hour period, I have 2-3 GB of trace data captured.
> The trace includes events from 4 different processes.
>
>
> If I use Babeltrace to read the trace I will get an enormously big result in
> text (~30-40GB at least).
>
> I would like to be able to do something like this:
>
> $ babeltrace --filter '$ctx.procname =3D=3D "smbd" || debug= =5Flevel < 3'
> /local/lttng-traces/ust/uid/2140/64-bit
>
> This would then let me see any events from the "smbd" process (since i added
> procname to the context using lttng add-context) or debug messages from all
> processes where the debug level is 1 or 2.
>

That would be a neat feature! There are currently no plans to support
filtering in Babeltrace, but that contribution would be welcome.
lttng-ust's filtering code is licensed under LGPL v2.1 so it could be
library-fied and used by the ctf-text plug-in. In the meantime, I
guess you could work around this limitation with a combination of AWK
and shell scripting...

> Regarding linking - I am referring to the function
> lttng=5Fenabler=5Fevent=5Flink=5Fbytecode(), called from
> lttng-ust/liblttng-ust/lttng-events.c around line 628.
>

Thanks for clarifying!

Keep in mind that the filter's byte code is generated by the lttng
binary without knowing the events' declarations. Thus, the byte code
must go through a linking stage not completely unlike what is done by
a regular linker. One of the objectives of the linking stage is
setting the appropriate offsets to load an event's fields. The byte
code is also "specialized" by resolving each field's type and making
sure the appropriate implementation of each operator is used. For
instance, the FILTER=5FOP=5FEQ (check for equality) op-code is not
implemented the same way for integers, doubles and strings (see
lttng=5Ffilter=5Fspecialize=5Fbytecode in lttng-filter-specialize.c).

You may want to have a look at =5Flttng=5Ffilter=5Fevent=5Flink=5Fbytecode<= br> (lttng-ust/liblttng-ust/lttng-filter.c, line 318) to see how each step
of the linking process is implemented.

Regards,
J=E9r=E9mie

> Thanks,
>
> Amit Margalit
> IBM XIV - Storage Reinvented
> XIV-NAS Development Team
> Tel. 03-689-7774
> Fax. 03-689-7230
>
>
>
> From:        J=E9r=E9mie Galarneau <jeremie.gal= arneau@efficios.com>
> To:        Amit Margalit/Israel/IBM@IBMIL
> Cc:        lttng-dev@lists.lttng.org
> Date:        07/18/2013 07:10 PM
> Subject:        Re: [lttng-dev] Question about filtering bytecode
> Sent by:        jeremie.galarneau@gmail.com
> =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F
>
>
>
> Hi Amit,
>
> You may find information on the Babeltrace API in babeltrace/doc/API.t= xt
> The filtering mechanism used by the user space tracer is unrelated to
> Babeltrace.
>
> As for the sequence of events, this is how I understand it.
>
> When invoking the "lttng" binary with --enable-event and a filter
> option, a byte code is generated from the filtering rules
> (lttng-tools/src/lib/lttng-ctl/lttng-ctl.c:704). This byte code is
> then handed off to the session-daemon.
>
> Then, when an instrumented application is launched and connects to the
> session-daemon, it receives the byte code while its events are being > enabled. To answer your question directly, that byte code will be
> executed by the tracee when the event is "triggered"
> (lttng-ust/liblttng-ust/lttng-filter-interpreter.c:175).
>
> I'm not sure I understand your question about filter/event linking. A
> filter has to be linked to an event since its rules (expressions) are
> applied on that event's values. Could you clarify if that's not what > you meant?
>
> Regards,
> J=E9r=E9mie
>
> On Thu, Jul 18, 2013 at 3:12 AM, Amit Margalit <AMITM@il.ibm.com> wrote:
>> Hi,
>>
>> I'm considering whether I can write a wrapper for the libbabeltrace ctf
>> iterator which provides a filtered "next()" method.
>>
>> I've started reading the code in the hopes of understanding how to do
>> this,
>> and yet, I'd really appreciate a link to some ordered documentatio= n, or -
>> any other kind of explanation.
>>
>> For example, one thing I still don't understand is the exact seque= nce, and
>> where the bytecode is actually executed? What is the meaning and purpose
>> of
>> "linking" the filter to an event.
>>
>> Thanks,
>>
>> Amit Margalit
>> IBM XIV - Storage Reinvented
>> XIV-NAS Development Team
>> Tel. 03-689-7774
>> Fax. 03-689-7230
>> =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=
>> lttng-dev mailing list
>> lttng-dev@lists.lttng.org
>> http://lists.lttng.org/cgi-bin/mailman/l= istinfo/lttng-dev
>>
>
>
>
> --
> J=E9r=E9mie Galarneau
> EfficiOS Inc.
> http= ://www.efficios.com
>
>



--
J=E9r=E9mie Galarneau
EfficiOS Inc.
http://ww= w.efficios.com
--=_alternative 00223F3BC2257BB0_=-- --===============0216331006== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ lttng-dev mailing list lttng-dev@lists.lttng.org http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev --===============0216331006==--