Re: A question on protocols

Re: A question on protocols

[Jonathan Rajotte-Julien]

Hi Patrick,

On Fri, Nov 16, 2018 at 09:49:52AM +0100, mrx wrote:
> Hi,
> 
> I have a need to collect LTTng live-traces from systems with very limited
> RAM
> and flash resources. This tracing will be running continuesly for months
> monitoring our systems. The only way for me to transport those CTF records
> somewhere else is via HTTP proxy. LTTng doesn't seem to have support for
> sending
> over proxies at all. So I think I really have a challenge a head of me, if
> this
> is at all possible.

VPN through http proxy. Better alternative would be to speak with your sysadmin
and see what you can do. Keep in mind that the protocol between relayd and
consumer is in no way "secure".

> 
> The plan is to write my own relayd from which I can then stream the
> received CTF
> records + metadata to where I can analyze them. For this to work I need
> documentation on the protocol between consumerd and relayd. I cannot find
> the
> documentation for this, where can I find it?

The source code.

> 
> Do you think this is a viable solution?

Doubt it. But we never know.

> 
> Once I receive the data where I have the possiblity to analyze it. Then I'm
> not
> sure if I'm required to write everything to the file system to be able to
> analyze the data. How would I then rotate the logs on disc so I can clean
> up?

The 2.11 release will include a new feature for session rotation.

See this presentation [1] from Jérémie Galarneau explaining how the session rotation
can be used.

[1] https://events.linuxfoundation.org/wp-content/uploads/2017/12/Fine-grained-Distributed-Application-Monitoring-Using-LTTng-J%C3%A9r%C3%A9mie-Galarneau-EfficiOS.pdf


> The best for me would be if I didn't have to go via disc at all I think.
> 
> Are there any others working on similar solution, if so, how are they
> solving this?
> How would you recommend I solve this?

I would go for session rotation via relayd (not in live mode) with a daemon
watching for ready-to-consume chunks. You can adjust for the granularity
you need at the target level.

This could be done close to the target then compress the trace chunks and send them
over http(s) to the monitoring pipeline.

> 
> The reason the current relayd doesn't work for me is two-fold:
> 1. I cannot get relayd to not write down the trace to disc. Can you control
> this at all for live tracing?

What is the real reason for not writing to disk on the relayd side?

> 2. I cannot find the documentation for the relayd <-> viewer protocol.
> Where can I find it?

Source code. The initial design proposal is under doc/ in the lttng-tools tree.

> 
> It might be that storing the traces on disc is a pre-requisite for serving a
> viewer properly. Perhaps it's just something required by relayd based on
> how it works internally. I don't know.

You can look at relayd as a specialized ftp server. The user is responsible in
managing the lifetime of the traces generated. The live protocol simply allow a
viewer to see the trace as it get received/stored.

Nothing prevent you to output on a tmpfs (ramdisk) if hitting the disk is such a
problem.

Keep in mind that trace production is normally much more quicker than trace
reading/analysis. A buffering scheme is mostly always necessary.

You can also use the --trace-file-size and --trace-file-count to limit the disk
footprint of each live session. Make sure to have enough buffer for live
reading if still using live.

Cheers
-- 
Jonathan Rajotte-Julien
EfficiOS

Re: A question on protocols

[mrx]

[-- Attachment #1.1: Type: text/plain, Size: 5173 bytes --]

On Fri, Nov 16, 2018 at 6:16 PM Jonathan Rajotte-Julien <
jonathan.rajotte-julien@efficios.com> wrote:

> Hi Patrick,
>
> On Fri, Nov 16, 2018 at 09:49:52AM +0100, mrx wrote:
> > Hi,
> >
> > I have a need to collect LTTng live-traces from systems with very limited
> > RAM
> > and flash resources. This tracing will be running continuesly for months
> > monitoring our systems. The only way for me to transport those CTF
> records
> > somewhere else is via HTTP proxy. LTTng doesn't seem to have support for
> > sending
> > over proxies at all. So I think I really have a challenge a head of me,
> if
> > this
> > is at all possible.
>
> VPN through http proxy. Better alternative would be to speak with your
> sysadmin
> and see what you can do. Keep in mind that the protocol between relayd and
> consumer is in no way "secure".
>

The security between consumerd and relayd isn't an issue as we'd be forced
to keep both on the device. This because we have no way of transporting the
consumerd <-> relayd communication over the HTTP proxy, which is our only
choice. We have already talked to the team in charge of the network paths.


>
> >
> > The plan is to write my own relayd from which I can then stream the
> > received CTF
> > records + metadata to where I can analyze them. For this to work I need
> > documentation on the protocol between consumerd and relayd. I cannot find
> > the
> > documentation for this, where can I find it?
>
> The source code.
>
> >
> > Do you think this is a viable solution?
>
> Doubt it. But we never know.
>
> >
> > Once I receive the data where I have the possiblity to analyze it. Then
> I'm
> > not
> > sure if I'm required to write everything to the file system to be able to
> > analyze the data. How would I then rotate the logs on disc so I can clean
> > up?
>
> The 2.11 release will include a new feature for session rotation.
>
> See this presentation [1] from Jérémie Galarneau explaining how the
> session rotation
> can be used.
>
> [1]
> https://events.linuxfoundation.org/wp-content/uploads/2017/12/Fine-grained-Distributed-Application-Monitoring-Using-LTTng-J%C3%A9r%C3%A9mie-Galarneau-EfficiOS.pdf


I'll for sure look into this.


>
>
>
> > The best for me would be if I didn't have to go via disc at all I think.
> >
> > Are there any others working on similar solution, if so, how are they
> > solving this?
> > How would you recommend I solve this?
>
> I would go for session rotation via relayd (not in live mode) with a daemon
> watching for ready-to-consume chunks. You can adjust for the granularity
> you need at the target level.
>
> This could be done close to the target then compress the trace chunks and
> send them
> over http(s) to the monitoring pipeline.
>
> >
> > The reason the current relayd doesn't work for me is two-fold:
> > 1. I cannot get relayd to not write down the trace to disc. Can you
> control
> > this at all for live tracing?
>
> What is the real reason for not writing to disk on the relayd side?
>

The relayd side have to be on the device. The device has very limited
amounts of free flash. Even if there were flash available it would wear
down the flash faster than what we'd like.


>
> > 2. I cannot find the documentation for the relayd <-> viewer protocol.
> > Where can I find it?
>
> Source code. The initial design proposal is under doc/ in the lttng-tools
> tree.
>

Thanks!


>
> >
> > It might be that storing the traces on disc is a pre-requisite for
> serving a
> > viewer properly. Perhaps it's just something required by relayd based on
> > how it works internally. I don't know.
>
> You can look at relayd as a specialized ftp server. The user is
> responsible in
> managing the lifetime of the traces generated. The live protocol simply
> allow a
> viewer to see the trace as it get received/stored.
>

That helps in visualizing the different components for me, thanks for that!

It also clarifies some of the problems we face with using relayd as a
solution to our problem.


>
> Nothing prevent you to output on a tmpfs (ramdisk) if hitting the disk is
> such a
> problem.
>

A RAM-based disc might have been a solution if we would have had any
mentionable amounts of RAM to spare for this. Unfortunately we don't have
that.


>
> Keep in mind that trace production is normally much more quicker than trace
> reading/analysis. A buffering scheme is mostly always necessary.
>

Very valid point, I'll for sure keep that in mind.

This speaks strongly against replacing relayd with something homebrewed, as
was the original plan. We'd probably not be able to shuffle the data fast
enough over the network to keep up with the pace.


>
> You can also use the --trace-file-size and --trace-file-count to limit the
> disk
> footprint of each live session. Make sure to have enough buffer for live
> reading if still using live.
>

This isn't an option as our disc is flash based and we'd like to limit the
wear due to collecting metrics.

Thanks for your insights, much appreciated.

// Patrik

[-- Attachment #1.2: Type: text/html, Size: 7145 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: A question on protocols

[Jonathan Rajotte-Julien]

Hi Patrick,

> The security between consumerd and relayd isn't an issue as we'd be forced
> to keep both on the device. This because we have no way of transporting the
> consumerd <-> relayd communication over the HTTP proxy, which is our only
> choice. We have already talked to the team in charge of the network paths.

In that case using lttng-relayd yield no advantage unless you are able to have
a device running only lttng-relayd and gathering data from other devices and
then sending the data via http.


> 
> The relayd side have to be on the device. The device has very limited
> amounts of free flash. Even if there were flash available it would wear
> down the flash faster than what we'd like.

Not sure I follow here. The whole point of lttng-relayd is to be off the device
being traced and to relay trace data using network only (skipping disk).

> A RAM-based disc might have been a solution if we would have had any
> mentionable amounts of RAM to spare for this. Unfortunately we don't have
> that.

In all cases you will need some RAM somewhere if using LTTng (and I think any
other monitoring/tracing tools) since we need to allocate tracing buffer.

> >
> > Keep in mind that trace production is normally much more quicker than trace
> > reading/analysis. A buffering scheme is mostly always necessary.
> >
> 
> Very valid point, I'll for sure keep that in mind.
> 
> This speaks strongly against replacing relayd with something homebrewed, as
> was the original plan. We'd probably not be able to shuffle the data fast
> enough over the network to keep up with the pace.

If you are unable to "reserve" RAM/network bandwidth and/or use FLASH this is
slowly becoming a "you can't have your cake and eat it too" situation. This must
be frustrating...

> >
> > You can also use the --trace-file-size and --trace-file-count to limit the
> > disk
> > footprint of each live session. Make sure to have enough buffer for live
> > reading if still using live.
> >
> 
> This isn't an option as our disc is flash based and we'd like to limit the
> wear due to collecting metrics.

I was under the impression that the lttng-relayd process was on another device.
I misunderstood the situation.

Cheers

-- 
Jonathan Rajotte-Julien
EfficiOS

Re: A question on protocols

[mrx]

[-- Attachment #1.1: Type: text/plain, Size: 4034 bytes --]

On Mon, Nov 19, 2018 at 7:46 PM Jonathan Rajotte-Julien <
jonathan.rajotte-julien@efficios.com> wrote:

> Hi Patrick,
>
> > The security between consumerd and relayd isn't an issue as we'd be
> forced
> > to keep both on the device. This because we have no way of transporting
> the
> > consumerd <-> relayd communication over the HTTP proxy, which is our only
> > choice. We have already talked to the team in charge of the network
> paths.
>
> In that case using lttng-relayd yield no advantage unless you are able to
> have
> a device running only lttng-relayd and gathering data from other devices
> and
> then sending the data via http.
>

Unfortunately, we're don't have any external device that can run the
lttng-relayd component.


>
>
> >
> > The relayd side have to be on the device. The device has very limited
> > amounts of free flash. Even if there were flash available it would wear
> > down the flash faster than what we'd like.
>
> Not sure I follow here. The whole point of lttng-relayd is to be off the
> device
> being traced and to relay trace data using network only (skipping disk).
>

Sure, so given this I cannot use relayd for our purposes. The only other
option would be to set LTTng to local tracing, which we don't have flash
for. My hope was that we could somehow stream the CTF records somewhere
else, over that HTTP proxy, ot be analyzed. As you explained earlier, the
relayd can be viewed as a FTP server of sorts. So that will most likely not
work.


>
> > A RAM-based disc might have been a solution if we would have had any
> > mentionable amounts of RAM to spare for this. Unfortunately we don't have
> > that.
>
> In all cases you will need some RAM somewhere if using LTTng (and I think
> any
> other monitoring/tracing tools) since we need to allocate tracing buffer.
>

We have RAM for the tracing buffer, no issues there. The tracing buffer is
just a temporary medium until you get the trace on disc, possibly via
relayd. Correct?

I interpretted your RAM disc suggestion as a place where I could store the
trace instead of using flash.


>
> > >
> > > Keep in mind that trace production is normally much more quicker than
> trace
> > > reading/analysis. A buffering scheme is mostly always necessary.
> > >
> >
> > Very valid point, I'll for sure keep that in mind.
> >
> > This speaks strongly against replacing relayd with something homebrewed,
> as
> > was the original plan. We'd probably not be able to shuffle the data fast
> > enough over the network to keep up with the pace.
>
> If you are unable to "reserve" RAM/network bandwidth and/or use FLASH this
> is
> slowly becoming a "you can't have your cake and eat it too" situation.
> This must
> be frustrating...
>
> > >
> > > You can also use the --trace-file-size and --trace-file-count to limit
> the
> > > disk
> > > footprint of each live session. Make sure to have enough buffer for
> live
> > > reading if still using live.
> > >
> >
> > This isn't an option as our disc is flash based and we'd like to limit
> the
> > wear due to collecting metrics.
>
> I was under the impression that the lttng-relayd process was on another
> device.
> I misunderstood the situation.
>

Perhaps i could have been more clear on the situation as well.

We have LTTng tracing capabilites on our devices. During development we can
use LTTng on the device and relayd on our dev machines and everything works
great.

In production, the situation is different.

The only means we have of transporting anything from the device is a HTTP
proxy. We're trying to find a way where we can keep a conitnuous LTTng
trace on the device, tunnel it through the HTTP proxy and analyze the data
remotely.

The devices are in the embedded realm, limited amounts of RAM, CPU, and
flash. This limits what we can do on the device when it comes to storage
for example.

Hopefully I've made our situation a bit more clear.

Based on our discussion so far, I have to say that LTTng is starting to
look less and less as a viable way forward for us.

Regards,
Patrik

[-- Attachment #1.2: Type: text/html, Size: 5333 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: A question on protocols

[Jonathan Rajotte-Julien]

Hi Patrick,

> Sure, so given this I cannot use relayd for our purposes. The only other
> option would be to set LTTng to local tracing, which we don't have flash
> for. My hope was that we could somehow stream the CTF records somewhere
> else, over that HTTP proxy, ot be analyzed. As you explained earlier, the
> relayd can be viewed as a FTP server of sorts. So that will most likely not
> work.

A possibility would be to spare some RAM (not easy I know), mount a tmpfs,
configure the session to output to the tmpfs, use session rotation based on
size, send the data, remove the data, rinse and repeat. Now the question is what
is the amount of tracing data generated and the amount of RAM available.

> We have RAM for the tracing buffer, no issues there. The tracing buffer is
> just a temporary medium until you get the trace on disc, possibly via
> relayd. Correct?

Yes.

The consumerd is responsible for fetching the data from the buffers, then it
output it locally or send it over the network to lttng-relayd.

You can take a look at the interaction here:

https://lttng.org/docs/v2.10/#doc-plumbing

> 
> I interpretted your RAM disc suggestion as a place where I could store the
> trace instead of using flash.

This was the correct interpretation.

> 
> Perhaps i could have been more clear on the situation as well.
> 
> We have LTTng tracing capabilites on our devices. During development we can
> use LTTng on the device and relayd on our dev machines and everything works
> great.

Glad to hear it.

> 
> In production, the situation is different.
> 
> The only means we have of transporting anything from the device is a HTTP
> proxy. We're trying to find a way where we can keep a conitnuous LTTng
> trace on the device, tunnel it through the HTTP proxy and analyze the data
> remotely.
> 
> The devices are in the embedded realm, limited amounts of RAM, CPU, and
> flash. This limits what we can do on the device when it comes to storage
> for example.
> 
> Hopefully I've made our situation a bit more clear.
> 
> Based on our discussion so far, I have to say that LTTng is starting to
> look less and less as a viable way forward for us.

It would be quite disappointing if you would not be able to reuse the work done
for instrumenting your applications.

Cheers

-- 
Jonathan Rajotte-Julien
EfficiOS

Re: A question on protocols

[mrx]

[-- Attachment #1.1: Type: text/plain, Size: 3226 bytes --]

On Tue, Nov 20, 2018 at 6:16 PM Jonathan Rajotte-Julien <
jonathan.rajotte-julien@efficios.com> wrote:

> Hi Patrick,
>
> > Sure, so given this I cannot use relayd for our purposes. The only other
> > option would be to set LTTng to local tracing, which we don't have flash
> > for. My hope was that we could somehow stream the CTF records somewhere
> > else, over that HTTP proxy, ot be analyzed. As you explained earlier, the
> > relayd can be viewed as a FTP server of sorts. So that will most likely
> not
> > work.
>
> A possibility would be to spare some RAM (not easy I know), mount a tmpfs,
> configure the session to output to the tmpfs, use session rotation based on
> size, send the data, remove the data, rinse and repeat. Now the question
> is what
> is the amount of tracing data generated and the amount of RAM available.
>
> > We have RAM for the tracing buffer, no issues there. The tracing buffer
> is
> > just a temporary medium until you get the trace on disc, possibly via
> > relayd. Correct?
>
> Yes.
>
> The consumerd is responsible for fetching the data from the buffers, then
> it
> output it locally or send it over the network to lttng-relayd.
>
> You can take a look at the interaction here:
>
> https://lttng.org/docs/v2.10/#doc-plumbing


Is the contents of the RAM buffer already proper CTF records ready to be
consumed, or is there some transformation that needs to take place as well?
If so, can i find that in some library or is it part of the consumerd code
base?

If i were to read the ring buffer instead of consumerd, then i would
perhaps be in a situation where I could stream the ring buffer to another
machine over that HTTP proxy? That way I wouldn't need any RAM disc or
anything, the data is already stored.

The consumerd logic that reads the ring buffer, is that part of some
library i could take advantage of?


>
>
> >
> > I interpretted your RAM disc suggestion as a place where I could store
> the
> > trace instead of using flash.
>
> This was the correct interpretation.
>
> >
> > Perhaps i could have been more clear on the situation as well.
> >
> > We have LTTng tracing capabilites on our devices. During development we
> can
> > use LTTng on the device and relayd on our dev machines and everything
> works
> > great.
>
> Glad to hear it.
>

We're very pleased with LTTng.


>
> >
> > In production, the situation is different.
> >
> > The only means we have of transporting anything from the device is a HTTP
> > proxy. We're trying to find a way where we can keep a conitnuous LTTng
> > trace on the device, tunnel it through the HTTP proxy and analyze the
> data
> > remotely.
> >
> > The devices are in the embedded realm, limited amounts of RAM, CPU, and
> > flash. This limits what we can do on the device when it comes to storage
> > for example.
> >
> > Hopefully I've made our situation a bit more clear.
> >
> > Based on our discussion so far, I have to say that LTTng is starting to
> > look less and less as a viable way forward for us.
>
> It would be quite disappointing if you would not be able to reuse the work
> done
> for instrumenting your applications.
>

Quite so, which is why we're trying so hard to find a viable way forward.

Regards,
Patrik

[-- Attachment #1.2: Type: text/html, Size: 4306 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

A question on protocols

[mrx]

[-- Attachment #1.1: Type: text/plain, Size: 1761 bytes --]

Hi,

I have a need to collect LTTng live-traces from systems with very limited
RAM
and flash resources. This tracing will be running continuesly for months
monitoring our systems. The only way for me to transport those CTF records
somewhere else is via HTTP proxy. LTTng doesn't seem to have support for
sending
over proxies at all. So I think I really have a challenge a head of me, if
this
is at all possible.

The plan is to write my own relayd from which I can then stream the
received CTF
records + metadata to where I can analyze them. For this to work I need
documentation on the protocol between consumerd and relayd. I cannot find
the
documentation for this, where can I find it?

Do you think this is a viable solution?

Once I receive the data where I have the possiblity to analyze it. Then I'm
not
sure if I'm required to write everything to the file system to be able to
analyze the data. How would I then rotate the logs on disc so I can clean
up?
The best for me would be if I didn't have to go via disc at all I think.

Are there any others working on similar solution, if so, how are they
solving this?
How would you recommend I solve this?

The reason the current relayd doesn't work for me is two-fold:
1. I cannot get relayd to not write down the trace to disc. Can you control
this
   at all for live tracing?
2. I cannot find the documentation for the relayd <-> viewer protocol.
Where can
   I find it?

It might be that storing the traces on disc is a pre-requisite for serving a
viewer properly. Perhaps it's just something required by relayd based on
how it
works internally. I don't know.

Do you have any suggestions on how to solve this?

Any insights and/or feedback would be appreciated. Thanks a lot in advance.

Patrik Iselind

[-- Attachment #1.2: Type: text/html, Size: 1960 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev