Re: Workflow to view old and current trace data

Re: Workflow to view old and current trace data

[Mathieu Desnoyers]

----- On Nov 8, 2019, at 3:49 AM, Benjamin Poirier benjamin.poirier@gmail.com wrote:

> Hi, long time no see!
> 
> Consider a system that's recording trace data continuously from a
> userspace application. A problem is noticed and I'd like to investigate
> it. What workflow would allow me to view past trace data to analyze the
> cause of the problem and view current (live) trace data while working on
> fixing the problem, all the while keeping a continuous record of events
> (ie. not stopping the trace)?
> 
> I thought of the following approaches but both seem to have
> disadvantages:
> 1)
> Have one continuous tracing session in normal mode. When a problem is
> noticed, use lttng-rotate to be able to read old trace data. Start a
> second tracing session in live mode with the same event rules to view
> live trace data. IIUC, there's no option to prevent lttng-relayd from
> writing the traces to disk, so we end up writing two sets of identical
> traces to disk during the time we want to look at live trace data.
> 2)
> Have one tracing session running in live mode all the time. When a
> problem is noticed, use the viewer on whatever lttng-relayd has flushed
> to disk to access old trace data (session rotation is not available).
> Furthermore, this workflow has the disadvantage that the trace data is
> going through lttng-relayd all the time (which I guess is less efficient
> than writing it from lttng-consumerd) merely to support the rare case
> where data needs to be seen live while analyzing a problem.
> 
> Is there some better approach that I didn't think about? What's your
> recommendation to support this workflow?
> 
> I'm comparing lttng to the situation where the userspace application is
> writing its logs directly to a file. In that case, it's simple to read
> the file to access old and "live" data. Of course I realize that lttng
> shines in other areas where simple file writing does not ;)

Why do you need the "live" mode at all ? What do you intend to do with the
live stream that cannot be done through notifiers invoked on rotation
completion and use size-based or timer-based rotations ?

Thanks,

Mathieu


> 
> Thank you,
> -Benjamin
> _______________________________________________
> lttng-dev mailing list
> lttng-dev@lists.lttng.org
> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

Workflow to view old and current trace data

[Benjamin Poirier]

Hi, long time no see!

Consider a system that's recording trace data continuously from a
userspace application. A problem is noticed and I'd like to investigate
it. What workflow would allow me to view past trace data to analyze the
cause of the problem and view current (live) trace data while working on
fixing the problem, all the while keeping a continuous record of events
(ie. not stopping the trace)?

I thought of the following approaches but both seem to have
disadvantages:
1)
Have one continuous tracing session in normal mode. When a problem is
noticed, use lttng-rotate to be able to read old trace data. Start a
second tracing session in live mode with the same event rules to view
live trace data. IIUC, there's no option to prevent lttng-relayd from
writing the traces to disk, so we end up writing two sets of identical
traces to disk during the time we want to look at live trace data.
2)
Have one tracing session running in live mode all the time. When a
problem is noticed, use the viewer on whatever lttng-relayd has flushed
to disk to access old trace data (session rotation is not available).
Furthermore, this workflow has the disadvantage that the trace data is
going through lttng-relayd all the time (which I guess is less efficient
than writing it from lttng-consumerd) merely to support the rare case
where data needs to be seen live while analyzing a problem.

Is there some better approach that I didn't think about? What's your
recommendation to support this workflow?

I'm comparing lttng to the situation where the userspace application is
writing its logs directly to a file. In that case, it's simple to read
the file to access old and "live" data. Of course I realize that lttng
shines in other areas where simple file writing does not ;)

Thank you,
-Benjamin