Re: Proposal: Make lttng-analyses part of lttng-tools

Re: Proposal: Make lttng-analyses part of lttng-tools

[Jonathan Rajotte-Julien]

Hi Geneviève,

On Tue, Jul 17, 2018 at 02:34:54PM -0400, Geneviève Bastien wrote:
> Hi all,
> 
> 
> Here's an idea to improve first impression of lttng and hopefully help
> the UX of the toolchain:
> 
> 
> tl;dr; I propose to make lttng-analyses part of lttng-tools to provide
> some basic recording/analysis facilities and make them the blessed way
> of getting started with lttng, so that end users can quickly obtain
> meaningful results after installing lttng.

As previously said in person and on IRC, I'm 100% in favour of putting an
emphasis on lttng-analyses. Still, I'm not sure that making the lttng-analyses
project part of the lttng-tools source tree is the way to go.

> 
> Here follows my argumentation. I'll focus on kernel tracing.
> 
> 
> Facts:
> 
> * People don't read documentation, at least not from the beginning.

Yup...

> 
> * First impressions stick

Pretty sure some people still thinks that you need to recompile your kernel for
kernel tracing to work. For those skimming, lttng-modules DOES NOT
requires a kernel compilation.

> 
> * Potential users trying to evaluate a tool will typically spend a few
> minutes/hours getting that first impression by reading an as
> minimalistic as possible readme file.

Yup

> 
> 
> Right now, with lttng, the first thing people are invited to do for
> kernel tracing, is to enable all the events (-a -k), which results in a
> number of lost events and a lot of noise when trying to analyze the

If I remember correctly, at some time, the default subbuffer size was bumped at the package
(Ubuntu) level. There is a commit in tools that bump the default size and
should fix the lost events stuff.

    commit c9eb1ddcd4af613f4e10938ab957f2a4a945db61
    Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
    Date:   Fri Mar 31 10:04:20 2017 -0400

        Bump default kernel, and UST per-uid/per-pid buffer size

        LTTng with current default buffer size often lead to discarded events,
        which is not something we want as a first user impression.

        The choice of default buffer size were made conservatively around 2010.
        Since then, the memory available on typical systems has increased, and
        so has the amount of instrumentation available. As an example, the
        mid-2010 Macbook Pro had 2GB ram. The current 2017 Macbook specification
        states 8GB ram, for a 4-fold installed memory size increase.

        Increase the kernel tracer buffer size from:
        4 x 256kB per core
        to:
        4 x 1MB per core

        Increase the UST tracer per-uid buffer size from:
        4 x 128kB per core
        to
        4 x 512kB per core

        Increase the UST tracer per-pid buffer size from:
        4 x 4kB per core
        to
        4 x 16kB per core


> results, and then 'lttng view' to display the babeltrace output! (see
> the Usage section in the lttng-modules readme page here:
> https://github.com/lttng/lttng-modules)

The Usage section of modules does not seems to instruct user to do "enable -k
-a" but I get your point.

> 
> That is not imho a good first impressions. Other tracers like ftrace or
> perf have a more per-event approach to tracing, their beginner's

The doc here seems to have a good basic primer indicating that the user could
choose to enable all events if necessary and the different projects for
analysis (Following section).

https://lttng.org/docs/v2.10/#doc-tracing-the-linux-kernel

> documentation showing how to enable 1 or 2 tracepoints. But to get
> interesting results, we usually need more than that and listing them
> individually can be cumbersome.
> Looking at my outbox, I sent my lttng
> script to enable events to get the critical path to at least 15 people

I'm not sure it is the responsibility of the tracer to define "pertinent" events.
The critical analysis is mostly a Trace Compass thing. The Trace Compass project
might want to have clearer event requirements (matching to lttng events name or
other tracer if possible) for their analysis.

> in the last years... It would be easy to have a predefined script for
> that purpose. Or many scripts (or profiles?) to enable events for
> various interesting analyses types.

Scripts could be done (lttng-analyses), not sure profiles at the tracer level make sense.

I think lttng-analyses already provides the essential part of a possible
solution. It would be a matter wrapping the lttng-analyses-record command and
the analyses scripts.

> 
> By comparison, take another kernel tracing tool: ebpf / bcc
> (https://github.com/iovisor/bcc). Installation is easy, then the readme
> points to a number of tools and example scripts of all kind. Running
> them after installation is easy and gives instant results. That makes
> one happy and feel good about bcc! Because having to write an ebpf
> script would have many users run for their sanity! ;-) But since many
> use cases are already covered by easily available scripts, people adopt
> the tool, then a few months later, when they need more, some may get to
> actually read the script, modify it, make them own.
> 
> lttng should aim for that kind of ease of use for beginners, have them
> reach that good feeling in a few minutes trial. And lttng-analyses
> already provides such scripts to record and analyze traces, and it
> should really be put forward and improved to give the user a more
> seamless first experience. I'd suggest to include it with lttng-tools,
> in a tools/ directory, as it would be one less repo to fork or one less

I would be more inclined to have it as a "recommended" package of
lttng-modules (since only kernel analyses are provided)
packages for package manager supporting this kind of feature (e.g apt).

Installing from source should not be the "go to" way of installing lttng for
beginners or people with limited involvement in the development of the project.

> package to install and it's a clear sign that it goes together with
> lttng. Keeping it as a separate project makes it appear as a side-dish,
> while it should be the center-plate!

I agree on center-plate stuff, not so much on how :).

I'm eager to get some feedback from Mathieu and Jérémie on all this.

Cheers.

-- 
Jonathan Rajotte-Julien
EfficiOS
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: Proposal: Make lttng-analyses part of lttng-tools

[Geneviève Bastien]

Hi Jonathan,


Thanks for your feedback, I'm happy we agree on the need to get to
analyses more easily. The unspoken bit of my email is that I am willing
to work on that right now and I can [help] maintain those scripts.

Our perspective at Poly is on the whole toolchain of trace collection,
visualization and analysis, of which lttng is an important part. So from
my point of view, the scripts could include all necessary tracing info
to get to view and analyze the traces with, say, Trace Compass, not just
Julien's lami scripts. I'll detail this idea below.

Also, to give more context, we are preparing a tutorial on trace
collection and visualization for the next LISA conference in October and
having simple one-liner scripts (that is available in a package) to get
traces would be much sexier than many big lines that people would need
to copy-paste manually.


On 2018-07-17 04:47 PM, Jonathan Rajotte-Julien wrote:
>
>> * First impressions stick
> Pretty sure some people still thinks that you need to recompile your kernel for
> kernel tracing to work. For those skimming, lttng-modules DOES NOT
> requires a kernel compilation.
Exactly what I was thinking of, even though you and I never knew that
era, it still sticks to the project! :p
>
>
>> That is not imho a good first impressions. Other tracers like ftrace or
>> perf have a more per-event approach to tracing, their beginner's
> The doc here seems to have a good basic primer indicating that the user could
> choose to enable all events if necessary and the different projects for
> analysis (Following section).
>
> https://lttng.org/docs/v2.10/#doc-tracing-the-linux-kernel
That is, when one actually reaches that part of the doc... which people
don't read remember ;-)
>
>> documentation showing how to enable 1 or 2 tracepoints. But to get
>> interesting results, we usually need more than that and listing them
>> individually can be cumbersome.
>> Looking at my outbox, I sent my lttng
>> script to enable events to get the critical path to at least 15 people
> I'm not sure it is the responsibility of the tracer to define "pertinent" events.
> The critical analysis is mostly a Trace Compass thing. The Trace Compass project
> might want to have clearer event requirements (matching to lttng events name or
> other tracer if possible) for their analysis.
It's not the responsibility of the tracer at all. But it could be the
responsiblity of tracing helper script. Having it in Trace Compass means
the use case would be to start tracing with Trace Compass. But the use
case we have in mind here is flight-recorder mode + snapshots on various
server machines when something occurs, then gather traces and if
required, open them in Trace Compass and maybe look at the critical
path. So tracing would be controlled by helper scripts.

The profiles I was mentioning are also at the script level. For
instance, one would start recording
./lttng-record [-p <profile_name>]

where we could have predefined profiles, for instance

* 'kernel_base' which would include all kernel events necessary for
critical path
* 'kernel_sched' for sched_switches only (to get thread status, CPU
times, etc for another userspace (not just UST) trace)
* 'kernel_memory' for memory events and other required for memory analysis
* 'kernel_kvm' ....(which will eventually include userspace stuff for
statedumps, etc)

You see the point.

>
>> in the last years... It would be easy to have a predefined script for
>> that purpose. Or many scripts (or profiles?) to enable events for
>> various interesting analyses types.
> Scripts could be done (lttng-analyses), not sure profiles at the tracer level make sense.
>
> I think lttng-analyses already provides the essential part of a possible
> solution. It would be a matter wrapping the lttng-analyses-record command and
> the analyses scripts.
I Agree, see above
>
>
> I would be more inclined to have it as a "recommended" package of
> lttng-modules (since only kernel analyses are provided)
for now
> packages for package manager supporting this kind of feature (e.g apt).
>
> Installing from source should not be the "go to" way of installing lttng for
> beginners or people with limited involvement in the development of the project.
You guys have any stats on the "go to" way of lttng users? Stats on
package downloads for different distros
>
>> package to install and it's a clear sign that it goes together with
>> lttng. Keeping it as a separate project makes it appear as a side-dish,
>> while it should be the center-plate!
> I agree on center-plate stuff, not so much on how :).
Maybe for now, we can keep it in a separate package, as long as it
becomes part of the new "go to" way for starters.
>
> I'm eager to get some feedback from Mathieu and Jérémie on all this.
So am I!

Cheers,
Geneviève
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: Proposal: Make lttng-analyses part of lttng-tools

[Philippe Proulx]

On Wed, Jul 18, 2018 at 11:37 AM Geneviève Bastien
<gbastien@versatic.net> wrote:
>
> > I'm eager to get some feedback from Mathieu and Jérémie on all this.
> So am I!

Adding mine even if you're not eager to get it ;-).

As someone who knows a few parts of the LTTng analyses project, it is
still considered experimental, prototypal, in other words not ready at
all to be integrated within any stable project. It is with good reason
that its version is still 0.x.

LTTng analyses helps us prove that such a tool can be useful (we
received good feedback in general), and it's a great project to quickly
develop a tracing analysis, but it's lacking thorough documentation
(outside its README file), tests, and by the nature of Python is very
slow and will probably remain like this forever. LTTng analyses also
does not handle exceptional scenarios very well.

Also, do not forget that this project has hard dependencies on Python
and Babeltrace. What would this mean, on the packaging level, when
making it part of LTTng-tools, which has a very limited set of
dependencies on purpose?

I believe our mid/long-term goal is to:

1. Release Babeltrace 2 ASAP.
2. Optimize Babeltrace 2 for the typical CTF input with stream muxing
   scenario.
3. Create/transfer LTTng analyses as Babeltrace 2 filter or sink
   components.
4. Encourage people to use Babeltrace 2 to analyse LTTng traces.

That's not to say I disagree with the highlighted problem: our efforts
should be directed towards offering a streamlined experience to the
user, from instrumenting to tracing to analyzing.

By the way, there are still a few professional, rigorous individuals who
read the documentation, thank God ;-).

Phil

>
> Cheers,
> Geneviève
> _______________________________________________
> lttng-dev mailing list
> lttng-dev@lists.lttng.org
> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: Proposal: Make lttng-analyses part of lttng-tools

[Geneviève Bastien]

On 2018-07-18 12:11 PM, Philippe Proulx wrote:
> On Wed, Jul 18, 2018 at 11:37 AM Geneviève Bastien
> <gbastien@versatic.net> wrote:
>>> I'm eager to get some feedback from Mathieu and Jérémie on all this.
>> So am I!
> Adding mine even if you're not eager to get it ;-).
I'm eager to hear anyone's opinion! And I think from the 2 opinions I
got so far, analyses will not be part of tools. I'll take that as
granted and consider all future comments of mine for lttng-analyses repo
alone...
>
> As someone who knows a few parts of the LTTng analyses project, it is
> still considered experimental, prototypal, in other words not ready at
> all to be integrated within any stable project. It is with good reason
> that its version is still 0.x.
>
> LTTng analyses helps us prove that such a tool can be useful (we
> received good feedback in general), and it's a great project to quickly
> develop a tracing analysis, but it's lacking thorough documentation
> (outside its README file), tests, and by the nature of Python is very
> slow and will probably remain like this forever. LTTng analyses also
> does not handle exceptional scenarios very well.
>
> Also, do not forget that this project has hard dependencies on Python
> and Babeltrace. What would this mean, on the packaging level, when
> making it part of LTTng-tools, which has a very limited set of
> dependencies on purpose?
Interesting remark: You highlight that lttng-"analyses" package has 2
purposes: trace collection and trace analysis.

Trace collection can only be bash, does not require any other
dependencies than lttng and that needs to be available on all machines
to trace (with this mechanism)

Trace analyses use babeltrace python bindings and needs to be run on the
"analysis" machine, which may not even be one that is traced.

2 very different use cases... Should they be split in 2 packages? Could
the trace collection part of analyses be moved to tools, leaving only
the analyses in analyses? (oh I lied up there, I did mention merging
with lttng-tools again ;-)
>
> I believe our mid/long-term goal is to:
>
> 1. Release Babeltrace 2 ASAP.
> 2. Optimize Babeltrace 2 for the typical CTF input with stream muxing
>    scenario.
> 3. Create/transfer LTTng analyses as Babeltrace 2 filter or sink
>    components.
> 4. Encourage people to use Babeltrace 2 to analyse LTTng traces.
Indeed our goals at Poly are different. But lttng being an open source
project, I hope our goals can be reconciled for the good of the project :)

Here are my short/mid-term goals:

1- Make trace collection with lttng for newcomers a painless process,
easy to document and explain, to get fast results (whether with
lttng-analyses or Trace Compass)
2- Develop and document methods to collect traces in an OpenStack/ceph
cloud, in conjunction with monitoring tools.

In the short term, I'm more interested in the trace collection part of
lttng-analyses (or tools?), and that, in the [very!] short term, I'm
ready to bring to 1.0.
>
> That's not to say I disagree with the highlighted problem: our efforts
> should be directed towards offering a streamlined experience to the
> user, from instrumenting to tracing to analyzing.
>
> By the way, there are still a few professional, rigorous individuals who
> read the documentation, thank God ;-).
By the number of times I've been told to read the documentation, I'm
definitely not one of them :p But yeah! Every time, the answer I was
looking for _was_ in the doc! The person who wrote it really knew his
stuff ;-)

Geneviève

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: Proposal: Make lttng-analyses part of lttng-tools

[Jonathan Rajotte-Julien]

On Wed, Jul 18, 2018 at 12:11:03PM -0400, Philippe Proulx wrote:
> On Wed, Jul 18, 2018 at 11:37 AM Geneviève Bastien
> <gbastien@versatic.net> wrote:
> >
> > > I'm eager to get some feedback from Mathieu and Jérémie on all this.
> > So am I!
> 
> Adding mine even if you're not eager to get it ;-).

Well, you will understand that some opinions have more weight than others and
that at the end of the day someone will have to decide something. :)

> 
> As someone who knows a few parts of the LTTng analyses project, it is
> still considered experimental, prototypal, in other words not ready at

It was deemed good enough to be in our Latest Stable PPA.

> all to be integrated within any stable project. It is with good reason
> that its version is still 0.x.
> 
> LTTng analyses helps us prove that such a tool can be useful (we
> received good feedback in general), and it's a great project to quickly

But does people use it? Were they "exposed" to it? Sure the doc mention it but
never uses it. That is normal since the doc is geared toward how to use lttng to
perform tracing. It is not geared toward "what can we get from a (kernel)
trace?". TBH, I'm quite good with that.

> develop a tracing analysis, but it's lacking thorough documentation

> (outside its README file),

It already have more doc than 98% of the utilities I use everyday.

> tests, and by the nature of Python is very
> slow and will probably remain like this forever. LTTng analyses also
> does not handle exceptional scenarios very well.
> 
> Also, do not forget that this project has hard dependencies on Python
> and Babeltrace. What would this mean, on the packaging level, when
> making it part of LTTng-tools, which has a very limited set of
> dependencies on purpose?

This is why "baking it" into the lttng-tools project/package make little sense
and leaving dependency management to the package managers would make sense.

Simply a matter of "do we install it by default or not?".

> 
> I believe our mid/long-term goal is to:
> 
> 1. Release Babeltrace 2 ASAP.

Babeltrace 2 was on the verge of being release 2 years ago if not more. I know that both you
and Jérémie are working quite hard on that and that should be imminent. Still, I
do not think that efforts of putting lttng-analyses in the spot light ,even if it is
considered "experimental", should be put on the Babeltrace 2 backburner.
Even more so if help is coming from a long time member of the community (tahini).

> 2. Optimize Babeltrace 2 for the typical CTF input with stream muxing
>    scenario.
> 3. Create/transfer LTTng analyses as Babeltrace 2 filter or sink
>    components.
> 4. Encourage people to use Babeltrace 2 to analyse LTTng traces.
> 
> That's not to say I disagree with the highlighted problem: our efforts
> should be directed towards offering a streamlined experience to the
> user, from instrumenting to tracing to analyzing.

> 
> By the way, there are still a few professional, rigorous individuals who
> read the documentation, thank God ;-).

Yeah, the doc is very nice. But sometime people need to see "magic" before the
"I want to know how" feeling kicks-in.

IMHO Geneviève's goals can be done outside of the
lttng-analyses project anyway.

>>> 1- Make trace collection with lttng for newcomers a painless process,
>>> easy to document and explain, to get fast results (whether with
>>> lttng-analyses or Trace Compass)
>>> 2- Develop and document methods to collect traces in an OpenStack/ceph
>>> cloud, in conjunction with monitoring tools.

Cheers

-- 
Jonathan Rajotte-Julien
EfficiOS
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: Proposal: Make lttng-analyses part of lttng-tools

[Jérémie Galarneau]

On 18 July 2018 at 12:48, Geneviève Bastien <gbastien@versatic.net> wrote:
> On 2018-07-18 12:11 PM, Philippe Proulx wrote:
>> On Wed, Jul 18, 2018 at 11:37 AM Geneviève Bastien
>> <gbastien@versatic.net> wrote:
>>>> I'm eager to get some feedback from Mathieu and Jérémie on all this.
>>> So am I!
>> Adding mine even if you're not eager to get it ;-).
> I'm eager to hear anyone's opinion! And I think from the 2 opinions I
> got so far, analyses will not be part of tools. I'll take that as
> granted and consider all future comments of mine for lttng-analyses repo
> alone...
>>
>> As someone who knows a few parts of the LTTng analyses project, it is
>> still considered experimental, prototypal, in other words not ready at
>> all to be integrated within any stable project. It is with good reason
>> that its version is still 0.x.
>>
>> LTTng analyses helps us prove that such a tool can be useful (we
>> received good feedback in general), and it's a great project to quickly
>> develop a tracing analysis, but it's lacking thorough documentation
>> (outside its README file), tests, and by the nature of Python is very
>> slow and will probably remain like this forever. LTTng analyses also
>> does not handle exceptional scenarios very well.
>>
>> Also, do not forget that this project has hard dependencies on Python
>> and Babeltrace. What would this mean, on the packaging level, when
>> making it part of LTTng-tools, which has a very limited set of
>> dependencies on purpose?
> Interesting remark: You highlight that lttng-"analyses" package has 2
> purposes: trace collection and trace analysis.
>
> Trace collection can only be bash, does not require any other
> dependencies than lttng and that needs to be available on all machines
> to trace (with this mechanism)
>
> Trace analyses use babeltrace python bindings and needs to be run on the
> "analysis" machine, which may not even be one that is traced.
>
> 2 very different use cases... Should they be split in 2 packages? Could
> the trace collection part of analyses be moved to tools, leaving only
> the analyses in analyses? (oh I lied up there, I did mention merging
> with lttng-tools again ;-)

There is always the possibility of shipping session configurations along
with lttng-tools and/or scripts if the configuration needs to be "smart".

That's something that can be improved right now, we just need to
identify presets that make sense. Is that a good first step?

>>
>> I believe our mid/long-term goal is to:
>>
>> 1. Release Babeltrace 2 ASAP.
>> 2. Optimize Babeltrace 2 for the typical CTF input with stream muxing
>>    scenario.
>> 3. Create/transfer LTTng analyses as Babeltrace 2 filter or sink
>>    components.
>> 4. Encourage people to use Babeltrace 2 to analyse LTTng traces.
> Indeed our goals at Poly are different. But lttng being an open source
> project, I hope our goals can be reconciled for the good of the project :)
>
> Here are my short/mid-term goals:
>
> 1- Make trace collection with lttng for newcomers a painless process,
> easy to document and explain, to get fast results (whether with
> lttng-analyses or Trace Compass)
> 2- Develop and document methods to collect traces in an OpenStack/ceph
> cloud, in conjunction with monitoring tools.
>

I share Philippe's goals, but I also understand you want to improve the
situation in the short term.

Without integrating lttng-analyses in lttng-tools, I think Jonathan has
a good point in terms of marking lttng-analyses as a recommended
package or making it part of the 'lttng' metapackages (if applicable).

Packagers (Michael?) may want to comment here as I guess this
is a distro-specific issue.

> In the short term, I'm more interested in the trace collection part of
> lttng-analyses (or tools?), and that, in the [very!] short term, I'm
> ready to bring to 1.0.

I'm a lot more comfortable shipping this than the analysis part.

I don't mind shipping analysis tools as part of lttng-tools at some
point, but as Philippe mentioned, they would need tests,
documentation and maintenance on par with the rest of the project.

>>
>> That's not to say I disagree with the highlighted problem: our efforts
>> should be directed towards offering a streamlined experience to the
>> user, from instrumenting to tracing to analyzing.
>>
>> By the way, there are still a few professional, rigorous individuals who
>> read the documentation, thank God ;-).
> By the number of times I've been told to read the documentation, I'm
> definitely not one of them :p But yeah! Every time, the answer I was
> looking for _was_ in the doc! The person who wrote it really knew his
> stuff ;-)
>

There's a need for comprehensive documentation which, as a project,
I feel we have addressed. However, there's also a need for tutorials and
guides, which are lacking right now.

I'm glad you are choosing to invest time there.

Thanks,
Jérémie

> Geneviève
>
> _______________________________________________
> lttng-dev mailing list
> lttng-dev@lists.lttng.org
> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev



-- 
Jérémie Galarneau
EfficiOS Inc.
http://www.efficios.com
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: Proposal: Make lttng-analyses part of lttng-tools

[Genevieve Bastien]

On 2018-07-18 03:35 PM, Jérémie Galarneau wrote:
> On 18 July 2018 at 12:48, Geneviève Bastien <gbastien@versatic.net> wrote:
>> On 2018-07-18 12:11 PM, Philippe Proulx wrote:
>>> On Wed, Jul 18, 2018 at 11:37 AM Geneviève Bastien
>>> <gbastien@versatic.net> wrote:
>>>>> I'm eager to get some feedback from Mathieu and Jérémie on all this.
>>>> So am I!
>>> Adding mine even if you're not eager to get it ;-).
>> I'm eager to hear anyone's opinion! And I think from the 2 opinions I
>> got so far, analyses will not be part of tools. I'll take that as
>> granted and consider all future comments of mine for lttng-analyses repo
>> alone...
>>> As someone who knows a few parts of the LTTng analyses project, it is
>>> still considered experimental, prototypal, in other words not ready at
>>> all to be integrated within any stable project. It is with good reason
>>> that its version is still 0.x.
>>>
>>> LTTng analyses helps us prove that such a tool can be useful (we
>>> received good feedback in general), and it's a great project to quickly
>>> develop a tracing analysis, but it's lacking thorough documentation
>>> (outside its README file), tests, and by the nature of Python is very
>>> slow and will probably remain like this forever. LTTng analyses also
>>> does not handle exceptional scenarios very well.
>>>
>>> Also, do not forget that this project has hard dependencies on Python
>>> and Babeltrace. What would this mean, on the packaging level, when
>>> making it part of LTTng-tools, which has a very limited set of
>>> dependencies on purpose?
>> Interesting remark: You highlight that lttng-"analyses" package has 2
>> purposes: trace collection and trace analysis.
>>
>> Trace collection can only be bash, does not require any other
>> dependencies than lttng and that needs to be available on all machines
>> to trace (with this mechanism)
>>
>> Trace analyses use babeltrace python bindings and needs to be run on the
>> "analysis" machine, which may not even be one that is traced.
>>
>> 2 very different use cases... Should they be split in 2 packages? Could
>> the trace collection part of analyses be moved to tools, leaving only
>> the analyses in analyses? (oh I lied up there, I did mention merging
>> with lttng-tools again ;-)
> There is always the possibility of shipping session configurations along
> with lttng-tools and/or scripts if the configuration needs to be "smart".
>
> That's something that can be improved right now, we just need to
> identify presets that make sense. Is that a good first step?
I thought from previous conversations that session configurations (what
you save and load in lttng) is much more complex than just the traced
events and is really machine-specific, so the workflow is typically for
a user to configure his session, then save it so he can load it after.
Are you saying that right now, it would be possible to just define
presets of events, leaving the rest of the sessions configuration as
package defaults?

But if it's easily done, then yes, it's definitely a good first step!

>
>>> I believe our mid/long-term goal is to:
>>>
>>> 1. Release Babeltrace 2 ASAP.
>>> 2. Optimize Babeltrace 2 for the typical CTF input with stream muxing
>>>    scenario.
>>> 3. Create/transfer LTTng analyses as Babeltrace 2 filter or sink
>>>    components.
>>> 4. Encourage people to use Babeltrace 2 to analyse LTTng traces.
>> Indeed our goals at Poly are different. But lttng being an open source
>> project, I hope our goals can be reconciled for the good of the project :)
>>
>> Here are my short/mid-term goals:
>>
>> 1- Make trace collection with lttng for newcomers a painless process,
>> easy to document and explain, to get fast results (whether with
>> lttng-analyses or Trace Compass)
>> 2- Develop and document methods to collect traces in an OpenStack/ceph
>> cloud, in conjunction with monitoring tools.
>>
> I share Philippe's goals, but I also understand you want to improve the
> situation in the short term.
>
> Without integrating lttng-analyses in lttng-tools, I think Jonathan has
> a good point in terms of marking lttng-analyses as a recommended
> package or making it part of the 'lttng' metapackages (if applicable).
>
> Packagers (Michael?) may want to comment here as I guess this
> is a distro-specific issue.
>
>> In the short term, I'm more interested in the trace collection part of
>> lttng-analyses (or tools?), and that, in the [very!] short term, I'm
>> ready to bring to 1.0.
> I'm a lot more comfortable shipping this than the analysis part.
#define "shipping this"

Do you mean to say your would be open to have trace collection scripts
ship with lttng-tools? So basically, to move and modify the
lttng-analysis-record script from analyses to tools, leaving
lttng-analyses as a suggested package to tools and modules.

#define "comfortable"

For the next release? :p And you're using present tense, not
conditional, that's good!

Another option would be to have trace collection as another
package/project, so that collection and analysis are 2 different things.
But if so, I'd like for this repo to be official and somewhat blessed by
the project, to use in tutorial and getting started documentation, and
not my own shady repo. It can eventually also contain scripts to manage
tracing sessions (configure, start, stop, snapshot) on multiple
machines, then gather those traces together for later analysis.

Anyway, it seems that there's a script out there that needs a home
downtown lttng and it's making buying offers here and there. Will it go
in tools, evolve in lttng-analyses or get a brand new place? We can
discuss that once the said script has reached maturity. But we all seem
to agree on the need for it and that it has a place in the lttng
project. Which I'm happy with right now.

Also, in case we decide to split trace collection and analyses somehow,
Francis Giraldeau had also done a trace recording script as part of
workload-kit[1]. That one is in python and may be more along the lines
of tracing profiles (for events) and including ust traces. That may also
be a good starting point. Everybody has python these days, right?

[1] https://github.com/giraldeau/workload-kit/blob/master/utils/lttng-simple

Thanks for all the feedback and comments,

Geneviève

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: Proposal: Make lttng-analyses part of lttng-tools

[Jérémie Galarneau]

On 18 July 2018 at 21:51, Genevieve Bastien <gbastien@versatic.net> wrote:
>
>
> On 2018-07-18 03:35 PM, Jérémie Galarneau wrote:
>> On 18 July 2018 at 12:48, Geneviève Bastien <gbastien@versatic.net> wrote:
>>> On 2018-07-18 12:11 PM, Philippe Proulx wrote:
>>>> On Wed, Jul 18, 2018 at 11:37 AM Geneviève Bastien
>>>> <gbastien@versatic.net> wrote:
>>>>>> I'm eager to get some feedback from Mathieu and Jérémie on all this.
>>>>> So am I!
>>>> Adding mine even if you're not eager to get it ;-).
>>> I'm eager to hear anyone's opinion! And I think from the 2 opinions I
>>> got so far, analyses will not be part of tools. I'll take that as
>>> granted and consider all future comments of mine for lttng-analyses repo
>>> alone...
>>>> As someone who knows a few parts of the LTTng analyses project, it is
>>>> still considered experimental, prototypal, in other words not ready at
>>>> all to be integrated within any stable project. It is with good reason
>>>> that its version is still 0.x.
>>>>
>>>> LTTng analyses helps us prove that such a tool can be useful (we
>>>> received good feedback in general), and it's a great project to quickly
>>>> develop a tracing analysis, but it's lacking thorough documentation
>>>> (outside its README file), tests, and by the nature of Python is very
>>>> slow and will probably remain like this forever. LTTng analyses also
>>>> does not handle exceptional scenarios very well.
>>>>
>>>> Also, do not forget that this project has hard dependencies on Python
>>>> and Babeltrace. What would this mean, on the packaging level, when
>>>> making it part of LTTng-tools, which has a very limited set of
>>>> dependencies on purpose?
>>> Interesting remark: You highlight that lttng-"analyses" package has 2
>>> purposes: trace collection and trace analysis.
>>>
>>> Trace collection can only be bash, does not require any other
>>> dependencies than lttng and that needs to be available on all machines
>>> to trace (with this mechanism)
>>>
>>> Trace analyses use babeltrace python bindings and needs to be run on the
>>> "analysis" machine, which may not even be one that is traced.
>>>
>>> 2 very different use cases... Should they be split in 2 packages? Could
>>> the trace collection part of analyses be moved to tools, leaving only
>>> the analyses in analyses? (oh I lied up there, I did mention merging
>>> with lttng-tools again ;-)
>> There is always the possibility of shipping session configurations along
>> with lttng-tools and/or scripts if the configuration needs to be "smart".
>>
>> That's something that can be improved right now, we just need to
>> identify presets that make sense. Is that a good first step?
> I thought from previous conversations that session configurations (what
> you save and load in lttng) is much more complex than just the traced
> events and is really machine-specific, so the workflow is typically for
> a user to configure his session, then save it so he can load it after.
> Are you saying that right now, it would be possible to just define
> presets of events, leaving the rest of the sessions configuration as
> package defaults?
>
> But if it's easily done, then yes, it's definitely a good first step!
>

Session configurations encompass the session's complete
configuration: session type, sub-buffer size/count, enabled events,
etc.

They are a good fit if you want to provide a couple of basic profiles.
For instance, setting up a session in snapshot mode with all disk
IO-related events enabled would be achievable with a session
configuration.

However, if we want to allow the profiles to be customized, a
stand-alone tool is a better option.


>>
>>>> I believe our mid/long-term goal is to:
>>>>
>>>> 1. Release Babeltrace 2 ASAP.
>>>> 2. Optimize Babeltrace 2 for the typical CTF input with stream muxing
>>>>    scenario.
>>>> 3. Create/transfer LTTng analyses as Babeltrace 2 filter or sink
>>>>    components.
>>>> 4. Encourage people to use Babeltrace 2 to analyse LTTng traces.
>>> Indeed our goals at Poly are different. But lttng being an open source
>>> project, I hope our goals can be reconciled for the good of the project :)
>>>
>>> Here are my short/mid-term goals:
>>>
>>> 1- Make trace collection with lttng for newcomers a painless process,
>>> easy to document and explain, to get fast results (whether with
>>> lttng-analyses or Trace Compass)
>>> 2- Develop and document methods to collect traces in an OpenStack/ceph
>>> cloud, in conjunction with monitoring tools.
>>>
>> I share Philippe's goals, but I also understand you want to improve the
>> situation in the short term.
>>
>> Without integrating lttng-analyses in lttng-tools, I think Jonathan has
>> a good point in terms of marking lttng-analyses as a recommended
>> package or making it part of the 'lttng' metapackages (if applicable).
>>
>> Packagers (Michael?) may want to comment here as I guess this
>> is a distro-specific issue.
>>
>>> In the short term, I'm more interested in the trace collection part of
>>> lttng-analyses (or tools?), and that, in the [very!] short term, I'm
>>> ready to bring to 1.0.
>> I'm a lot more comfortable shipping this than the analysis part.
> #define "shipping this"

Including collection scripts into lttng-tools and having them be part
of its default install.

>
> Do you mean to say your would be open to have trace collection scripts
> ship with lttng-tools? So basically, to move and modify the
> lttng-analysis-record script from analyses to tools, leaving
> lttng-analyses as a suggested package to tools and modules.
>
> #define "comfortable"
>
> For the next release? :p And you're using present tense, not
> conditional, that's good!

There are things that we can get away with in a 0.x project
(stability issues, API breaks, etc.) that I feel would not be
acceptable in lttng-tools.

At some point, it could make sense to merge this tool in
lttng-tools. But in the short term, I think it would stifle its
development and frustrate everyone involved.

Repos and project names are cheap, so let's start this effort
as a separate project and see where that leads us.

>
> Another option would be to have trace collection as another
> package/project, so that collection and analysis are 2 different things.
> But if so, I'd like for this repo to be official and somewhat blessed by
> the project, to use in tutorial and getting started documentation, and
> not my own shady repo. It can eventually also contain scripts to manage
> tracing sessions (configure, start, stop, snapshot) on multiple
> machines, then gather those traces together for later analysis.

I agree. I think this tool should evolve on its own timeline rather than
being bound to LTTng releases, especially early on.

As for the "blessing" part, I'll be glad to point people to any tool that
is useful, actively maintained, documented, and tested.

However, If its just going to be a script that is dumped on GitHub,
I personally won't.

>
> Anyway, it seems that there's a script out there that needs a home
> downtown lttng and it's making buying offers here and there. Will it go
> in tools, evolve in lttng-analyses or get a brand new place? We can
> discuss that once the said script has reached maturity. But we all seem
> to agree on the need for it and that it has a place in the lttng
> project. Which I'm happy with right now.

It see a fit under the LTTng umbrela. It's a shame the lttng-tools name
is already taken, but there's definitely room for an lttng-utils/helpers
project.

>
> Also, in case we decide to split trace collection and analyses somehow,
> Francis Giraldeau had also done a trace recording script as part of
> workload-kit[1]. That one is in python and may be more along the lines
> of tracing profiles (for events) and including ust traces. That may also
> be a good starting point. Everybody has python these days, right?

Python is is a good fit for this kind of work, but choosing the
language is the prerogative of the implementer :-)

Thanks,
Jérémie

>
> [1] https://github.com/giraldeau/workload-kit/blob/master/utils/lttng-simple
>
> Thanks for all the feedback and comments,
>
> Geneviève
>



-- 
Jérémie Galarneau
EfficiOS Inc.
http://www.efficios.com
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: RFC: lttng-utils to help record traces

[Genevieve Bastien]

Hi all,

As a followup to the email thread title "Proposal: Make lttng-analyses
part of lttng-tools", here's a first effort towards a helper script to
record lttng traces:

https://github.com/tahini/lttng-utils

Play with it, let me know what you think. It's aimed initially at people
starting with lttng, to help them easily get traces that can be analyzed
without having to enable all events or know what to enable exactly. But
it can also be customized and used by expert users I think.

Thanks,
Geneviève

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev