Re: introduction & usage of LTTNG in machinekit

Re: introduction & usage of LTTNG in machinekit

[Michel Dagenais]

> I am involved in machinekit.io, a portable realtime application for motion control
> machinekit runs on several kernels - starting from vanilla (no time
> guarantees), RT-PREEMPT, Xenomai, and RTAI; on x86, amd64 and ARM platforms.

I am actually using LinuxCNC on a small CNC mill and would love to get that running directly on PREEMPT-RT instead of Xenomai (modified Sieg SX3 mill, cncfusion conversion kit, servo motors on X, Y and Z axis, mesa FPGA and H bridge boards for control).

> As some of the processing is highly time sensitive, I'm looking into
> integrating LTTNG into the machinekit build so we have options beyond
> logging and timestamping manually. For instance, currently some
> transcendental math functions on ARM overrun the time budget.
> 
> I've started integrating the build support and am running into my first
> problems. So, that's the context of my following questions - thanks in
> advance for guiding me through any blunders.

This is an excellent application for LTTng. My group has been looking for good use cases of real-time applications. We have a significant expertise with Linux for real-time and PREEMPT-RT. We could easily help you with the machinekit tracing: kernel and user-space instrumentation, specific views and analysis for this application... 

Re: introduction & usage of LTTNG in machinekit

[Michael Haberler]

Michel,


> Am 27.03.2015 um 19:35 schrieb Michel Dagenais <michel.dagenais@polymtl.ca>:
> 
> 
>> I am involved in machinekit.io, a portable realtime application for motion control
>> machinekit runs on several kernels - starting from vanilla (no time
>> guarantees), RT-PREEMPT, Xenomai, and RTAI; on x86, amd64 and ARM platforms.
> 
> I am actually using LinuxCNC on a small CNC mill and would love to get that running directly on PREEMPT-RT instead of Xenomai (modified Sieg SX3 mill, cncfusion conversion kit, servo motors on X, Y and Z axis, mesa FPGA and H bridge boards for control).

machinekit is a fork of LinuxCNC and does all that stock, we've moved well beyond the original project's capabilities. Runs on x86, amd64 and arm7+; the beaglebones are quite popular around here, especially with the 3d printer users.

See http://www.machinekit.io/docs/packages-debian/ for packages, maybe join the mailing list.

> 
>> As some of the processing is highly time sensitive, I'm looking into
>> integrating LTTNG into the machinekit build so we have options beyond
>> logging and timestamping manually. For instance, currently some
>> transcendental math functions on ARM overrun the time budget.
>> 
>> I've started integrating the build support and am running into my first
>> problems. So, that's the context of my following questions - thanks in
>> advance for guiding me through any blunders.
> 
> This is an excellent application for LTTng. My group has been looking for good use cases of real-time applications. We have a significant expertise with Linux for real-time and PREEMPT-RT. We could easily help you with the machinekit tracing: kernel and user-space instrumentation, specific views and analysis for this application... 

Yes, I have the impression this could be a very good match. Thanks! right now I'm climbing the learning curve and integrating basic examples into the machinekit repo so others can follow and explore. No showstoppers so far.

well, my first round of questions is about to hit the list..


thanks! drop me a line if you need help using machinekit.

- Michael

Re: introduction & usage of LTTNG in machinekit

[Michael Haberler]

Michel,

> Am 27.03.2015 um 19:35 schrieb Michel Dagenais <michel.dagenais@polymtl.ca>:
>> 
>> I've started integrating the build support and am running into my first
>> problems. So, that's the context of my following questions - thanks in
>> advance for guiding me through any blunders.
> 
> This is an excellent application for LTTng. My group has been looking for good use cases of real-time applications. We have a significant expertise with Linux for real-time and PREEMPT-RT. We could easily help you with the machinekit tracing: kernel and user-space instrumentation, specific views and analysis for this application... 

wrote the first UST tracepoints, got them to work just fine - no surprises (congratulations on the manual and general usability!). I'm building them into a shared library for now.

this I'm unsure how to deal with:

- is there a complete example out-of-tree kernel module instrumented for LTTng? I worked through Steve Rostedt's sillymod (http://lwn.net/Articles/379903 ff) but am fuzzy on "Adding the LTTng adaptation layer" - is the example from the manual available in toto somewhere?

- much of the machinekit RT code (HAL library and components) can be compiled as userland shared objects (Posix, RT-PREEMPP, Xenomai threads) or kernel modules (RTAI, and the deprecated Xenomai kernel API), sitting ontop of an abstracted realtime API ("RTAPI"). Ideally the tracepoints would work unchanged except for the different context. The manual naturally assumes an either-or context. Am I out on a limb here ;-? How would I bring in the tracepoint definitions in a kernel context - as a separate module maybe or linked into each using module?

- we cannot assume that lttng is available when building machinekit packages at least for now, meaning we likely need shim macro definitions and possibly fake header files to adjust for the lacking lttng files. Any examples I can follow?

thanks!

- Michael

Re: introduction & usage of LTTNG in machinekit

[Michael Haberler]

Michel,

> Am 29.03.2015 um 16:31 schrieb Michel Dagenais <michel.dagenais@polymtl.ca>:
> 
> 
>> - is there a complete example out-of-tree kernel module instrumented for
>> LTTng? I worked through Steve Rostedt's sillymod
>> (http://lwn.net/Articles/379903 ff) but am fuzzy on "Adding the LTTng
>> adaptation layer" - is the example from the manual available in toto
>> somewhere?
> 
> A few people in my group have done so. I will check on Monday to get sample code for you.

on re-reading, I am almost there, but I'd appreciate an example anyway

> 
>> - much of the machinekit RT code (HAL library and components) can be compiled
>> as userland shared objects (Posix, RT-PREEMPP, Xenomai threads) or kernel
>> modules (RTAI, and the deprecated Xenomai kernel API), sitting ontop of an
>> abstracted realtime API ("RTAPI"). Ideally the tracepoints would work
>> unchanged except for the different context. The manual naturally assumes an
>> either-or context. Am I out on a limb here ;-? How would I bring in the
>> tracepoint definitions in a kernel context - as a separate module maybe or
>> linked into each using module?
> 
> I dont believe that we have examples of code that compile in both kernel and UST contexts. This can certainly be handled with conditional compilation. 

right, I need to wrap my head around the maze of #defines and #includes ;) but it looks possible

> 
>> - we cannot assume that lttng is available when building machinekit packages
>> at least for now, meaning we likely need shim macro definitions and possibly
>> fake header files to adjust for the lacking lttng files. Any examples I can
>> follow?
> 
> MariaDB and others can target different tracers (LTTng, DTrace, none)... Again, I will try to get you good sample code early next week.

I thought I might not be the first one to do this


semi-related because I was bitten by it: the Xenomai configs referenced here: http://www.xenomai.org/pipermail/xenomai/2013-January/027272.html turn CONFIG_FTRACE off, citing performance reasons. I think we'll turn it back on in our kernels, cant be that much extra overhead.


thanks in advance!

- Michael

Re: introduction & usage of LTTNG in machinekit

[Julien Desfossez]

Hi,

On 15-03-29 11:17 AM, Michael Haberler wrote:
> Michel,
> 
>> Am 29.03.2015 um 16:31 schrieb Michel Dagenais <michel.dagenais@polymtl.ca>:
>>
>>
>>> - is there a complete example out-of-tree kernel module instrumented for
>>> LTTng? I worked through Steve Rostedt's sillymod
>>> (http://lwn.net/Articles/379903 ff) but am fuzzy on "Adding the LTTng
>>> adaptation layer" - is the example from the manual available in toto
>>> somewhere?
>>
>> A few people in my group have done so. I will check on Monday to get sample code for you.
> 
> on re-reading, I am almost there, but I'd appreciate an example anyway

I have a complete example of instrumenting an out-of-tree module in the
latency_tracker and the patch for lttng-modules (master) in the "extras"
folder:
https://github.com/jdesfossez/latency_tracker

It might be very slow due to access the page to the DDoS in progress, if
you need it quickly, I'll send you a tar.

Julien

Re: introduction & usage of LTTNG in machinekit

[Michael Haberler]

Julien -

thanks a lot - cloned, and quickly at it!

bets regards

Michael

> Am 29.03.2015 um 17:47 schrieb Julien Desfossez <jdesfossez@efficios.com>:
> 
> Hi,
> 
> On 15-03-29 11:17 AM, Michael Haberler wrote:
>> Michel,
>> 
>>> Am 29.03.2015 um 16:31 schrieb Michel Dagenais <michel.dagenais@polymtl.ca>:
>>> 
>>> 
>>>> - is there a complete example out-of-tree kernel module instrumented for
>>>> LTTng? I worked through Steve Rostedt's sillymod
>>>> (http://lwn.net/Articles/379903 ff) but am fuzzy on "Adding the LTTng
>>>> adaptation layer" - is the example from the manual available in toto
>>>> somewhere?
>>> 
>>> A few people in my group have done so. I will check on Monday to get sample code for you.
>> 
>> on re-reading, I am almost there, but I'd appreciate an example anyway
> 
> I have a complete example of instrumenting an out-of-tree module in the
> latency_tracker and the patch for lttng-modules (master) in the "extras"
> folder:
> https://github.com/jdesfossez/latency_tracker
> 
> It might be very slow due to access the page to the DDoS in progress, if
> you need it quickly, I'll send you a tar.
> 
> Julien

Re: introduction & usage of LTTNG in machinekit

[Michel Dagenais]

> >> - is there a complete example out-of-tree kernel module instrumented for LTTng? 
> > A few people in my group have done so. I will check on Monday to get sample
> > code for you.
> on re-reading, I am almost there, but I'd appreciate an example anyway

Julien has sent an example. Now that you will be getting real-time traces, someone in my group, Mathieu Côté, is working specifically on views to help diagnose real-time problems. He will be in a position to help you and can even improve the current TraceCompass views if you have specific "real-time programming" "use cases" not currently well covered.  

> >> - much of the machinekit RT code (HAL library and components) can be compiled
> >> as userland shared objects (Posix, RT-PREEMPP, Xenomai threads) or kernel
> >> modules (RTAI, and the deprecated Xenomai kernel API), sitting ontop of an
> >> abstracted realtime API ("RTAPI"). Ideally the tracepoints would work
> >> unchanged except for the different context.

I checked with Mathieu Desnoyers about LTTng and Xenomai. He has not looked at it. Karim Yaghmour developed the original LTT while working in my group and adapted it to Xenomai. Later on, I believe that someone else within Xenomai switched Xenomai from LTT to LTTng but this has not necessarily been maintained. LTTng is built to work even in NMI conext and thus relies on very few kernel services. It mostly uses RCU critical sections: disabling preemption between read lock and read unlock, and calling synchronize_rcu.

> > MariaDB and others can target different tracers (LTTng, DTrace, none)...
> > Again, I will try to get you good sample code early next week.

QEMU is another example of project which interfaces to LTTng but can have other or no tracer as well.

> semi-related because I was bitten by it: the Xenomai configs referenced here:
> http://www.xenomai.org/pipermail/xenomai/2013-January/027272.html turn
> CONFIG_FTRACE off, citing performance reasons. I think we'll turn it back on
> in our kernels, cant be that much extra overhead.

If the option is enabled but tracing is not active, there are simply a few bytes of nops added at the beginning of each function. The processor pipeline is indeed extremely efficient at skipping those and the impact is barely measurable.

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: introduction & usage of LTTNG in machinekit

[Michel Dagenais]

How are you progressing to add LTTng instrumentation to the various layers of MachineKit? 

I will have a student this Summer in our lab looking at the use of LTTng on embedded platforms for real-time applications such as MachineKit. He will insure that LTTng is easy to use and deploy on these popular platforms (Raspberry Pi, BeagleBone Black, Parallela), and well documented. He will also look at specific analysis and views that would be useful for such platforms and applications and look at the possibility of tracing the peripheral processors such as the BBB PRU as well. He is in CC and could help you by getting your feedback and proposing adjustments and improvements to the LTTng toolchain as a result.

----- Mail original -----
> Michel,
> 
> > Am 29.03.2015 um 16:31 schrieb Michel Dagenais
> > <michel.dagenais@polymtl.ca>:
> > 
> > 
> >> - is there a complete example out-of-tree kernel module instrumented for
> >> LTTng? I worked through Steve Rostedt's sillymod
> >> (http://lwn.net/Articles/379903 ff) but am fuzzy on "Adding the LTTng
> >> adaptation layer" - is the example from the manual available in toto
> >> somewhere?
> > 
> > A few people in my group have done so. I will check on Monday to get sample
> > code for you.
> 
> on re-reading, I am almost there, but I'd appreciate an example anyway
> 
> > 
> >> - much of the machinekit RT code (HAL library and components) can be
> >> compiled
> >> as userland shared objects (Posix, RT-PREEMPP, Xenomai threads) or kernel
> >> modules (RTAI, and the deprecated Xenomai kernel API), sitting ontop of an
> >> abstracted realtime API ("RTAPI"). Ideally the tracepoints would work
> >> unchanged except for the different context. The manual naturally assumes
> >> an
> >> either-or context. Am I out on a limb here ;-? How would I bring in the
> >> tracepoint definitions in a kernel context - as a separate module maybe or
> >> linked into each using module?
> > 
> > I dont believe that we have examples of code that compile in both kernel
> > and UST contexts. This can certainly be handled with conditional
> > compilation.
> 
> right, I need to wrap my head around the maze of #defines and #includes ;)
> but it looks possible
> 
> > 
> >> - we cannot assume that lttng is available when building machinekit
> >> packages
> >> at least for now, meaning we likely need shim macro definitions and
> >> possibly
> >> fake header files to adjust for the lacking lttng files. Any examples I
> >> can
> >> follow?
> > 
> > MariaDB and others can target different tracers (LTTng, DTrace, none)...
> > Again, I will try to get you good sample code early next week.
> 
> I thought I might not be the first one to do this
> 
> 
> semi-related because I was bitten by it: the Xenomai configs referenced here:
> http://www.xenomai.org/pipermail/xenomai/2013-January/027272.html turn
> CONFIG_FTRACE off, citing performance reasons. I think we'll turn it back on
> in our kernels, cant be that much extra overhead.
> 
> 
> thanks in advance!
> 
> - Michael
> 
> 
> 
> 

Re: introduction & usage of LTTNG in machinekit

[Michael Haberler]

Hello Michel, Rafael,

> Am 26.04.2015 um 15:31 schrieb Michel Dagenais <michel.dagenais@polymtl.ca>:
> 
> How are you progressing to add LTTng instrumentation to the various layers of MachineKit? 

I've worked my way through examples to get traces, and I have added autoconf and build support to the machinekit repo so it can be used.

That said, the folks which have timing problems have not caught on yet.. it looks some more guidance, examples and a bit of a machinekit-specific writeup is needed, competing with the other 247 priority projects on my desk ;)

So in principle it's a go as far as the (more important) userland threads flavors (xenomai, rt-preempt) are concerned; RTAI/in-kernel instrumentation would be nice to have to round it out but given the prospects for RTAI's future this might be a bit academic an exercise.
 
> I will have a student this Summer in our lab looking at the use of LTTng on embedded platforms for real-time applications such as MachineKit. He will insure that LTTng is easy to use and deploy on these popular platforms (Raspberry Pi, BeagleBone Black, Parallela), and well documented. He will also look at specific analysis and views that would be useful for such platforms and applications and look at the possibility of tracing the peripheral processors such as the BBB PRU as well. He is in CC and could help you by getting your feedback and proposing adjustments and improvements to the LTTng toolchain as a result.

Well, I would be glad to get Rafael interested in instrumenting machinekit and I am certain we'd be able to coach him through the learning curve so any work actually makes a difference; I would love to see some critical components be instrumented and actually measured on several platforms, and embedded ARM's are becoming more important all the time.

As far as machinekit as a proving ground for LTTNG goes, it is certainly a nontrivial 'customer' - meaning I dont exclude some problems and roadblocks downstream. But the upside for LTTNG I see - if it works here, it should work just about anywhere.


Let me add a question out of the blue: I've read the literature on observing time and distributed systems and am aware of the theoretical issues, as well as clock synchronisation. Still - has any thought been given to merge traces of several systems based on synchronisation points (known causality), well-synchronized  clocks, or other forms of hardware synchronisation?

thanks for caring!

best regards

Michael Haberler

> 
> ----- Mail original -----
>> Michel,
>> 
>>> Am 29.03.2015 um 16:31 schrieb Michel Dagenais
>>> <michel.dagenais@polymtl.ca>:
>>> 
>>> 
>>>> - is there a complete example out-of-tree kernel module instrumented for
>>>> LTTng? I worked through Steve Rostedt's sillymod
>>>> (http://lwn.net/Articles/379903 ff) but am fuzzy on "Adding the LTTng
>>>> adaptation layer" - is the example from the manual available in toto
>>>> somewhere?
>>> 
>>> A few people in my group have done so. I will check on Monday to get sample
>>> code for you.
>> 
>> on re-reading, I am almost there, but I'd appreciate an example anyway
>> 
>>> 
>>>> - much of the machinekit RT code (HAL library and components) can be
>>>> compiled
>>>> as userland shared objects (Posix, RT-PREEMPP, Xenomai threads) or kernel
>>>> modules (RTAI, and the deprecated Xenomai kernel API), sitting ontop of an
>>>> abstracted realtime API ("RTAPI"). Ideally the tracepoints would work
>>>> unchanged except for the different context. The manual naturally assumes
>>>> an
>>>> either-or context. Am I out on a limb here ;-? How would I bring in the
>>>> tracepoint definitions in a kernel context - as a separate module maybe or
>>>> linked into each using module?
>>> 
>>> I dont believe that we have examples of code that compile in both kernel
>>> and UST contexts. This can certainly be handled with conditional
>>> compilation.
>> 
>> right, I need to wrap my head around the maze of #defines and #includes ;)
>> but it looks possible
>> 
>>> 
>>>> - we cannot assume that lttng is available when building machinekit
>>>> packages
>>>> at least for now, meaning we likely need shim macro definitions and
>>>> possibly
>>>> fake header files to adjust for the lacking lttng files. Any examples I
>>>> can
>>>> follow?
>>> 
>>> MariaDB and others can target different tracers (LTTng, DTrace, none)...
>>> Again, I will try to get you good sample code early next week.
>> 
>> I thought I might not be the first one to do this
>> 
>> 
>> semi-related because I was bitten by it: the Xenomai configs referenced here:
>> http://www.xenomai.org/pipermail/xenomai/2013-January/027272.html turn
>> CONFIG_FTRACE off, citing performance reasons. I think we'll turn it back on
>> in our kernels, cant be that much extra overhead.
>> 
>> 
>> thanks in advance!
>> 
>> - Michael
>> 
>> 
>> 
>> 

Re: introduction & usage of LTTNG in machinekit

[Michel Dagenais]

> That said, the folks which have timing problems have not caught on yet.. it
> looks some more guidance, examples and a bit of a machinekit-specific
> writeup is needed, competing with the other 247 priority projects on my desk
> ;)

If we can reproduce easily the problems, bad latency on Linux Prempt-RT in a specific configuration, we can take care of tracing and diagnosing the problems. Otherwise, the users will have to activate the tracing themselves on their setup and we can guide them for the diagnosis.

> Let me add a question out of the blue: I've read the literature on observing
> time and distributed systems and am aware of the theoretical issues, as well
> as clock synchronisation. Still - has any thought been given to merge traces
> of several systems based on synchronisation points (known causality),
> well-synchronized  clocks, or other forms of hardware synchronisation?

I suppose that you mean something like this which we offer in TraceCompass:

http://archive.eclipse.org/tracecompass/doc/org.eclipse.tracecompass.doc.user/Trace-synchronization.html

We have two use cases supported already, distributed systems exchanging packets through TCP and physical / virtual machines communicating through vmenter / exit and tracepoints on both sides of those transitions. This framework can easily be extended to other cases. We could in the future make it available declaratively (define event types and unique identifiers that should be used to match causality-related events).

Re: introduction & usage of LTTNG in machinekit

[Michael Haberler]

> Am 27.04.2015 um 14:30 schrieb Michel Dagenais <michel.dagenais@polymtl.ca>:
> 
> 
>> That said, the folks which have timing problems have not caught on yet.. it
>> looks some more guidance, examples and a bit of a machinekit-specific
>> writeup is needed, competing with the other 247 priority projects on my desk
>> ;)
> 
> If we can reproduce easily the problems, bad latency on Linux Prempt-RT in a specific configuration, we can take care of tracing and diagnosing the problems. Otherwise, the users will have to activate the tracing themselves on their setup and we can guide them for the diagnosis. 

The problem at hand has these characteristics:

- it happens only on low-end ARM platforms
- the code is part of the trajectory planner, which involves fp and transcendental functions
- the code is executed from a period Xenomai thread and once in a while execution time exceeds the cycle time budget
- no interrupts, system call, or Xenomai API call is involved, so kernel or ipipe tracing will not help to narrow it down

my idea was to insert trace points in the tp code and one triggered on 'time window exceeded' so one could look backwards in history to determine which span of code used excessive time; does this sound reasonable?

Alex (in cc) has a test case which reproducibly triggers the problem

> 
>> Let me add a question out of the blue: I've read the literature on observing
>> time and distributed systems and am aware of the theoretical issues, as well
>> as clock synchronisation. Still - has any thought been given to merge traces
>> of several systems based on synchronisation points (known causality),
>> well-synchronized  clocks, or other forms of hardware synchronisation?
> 
> I suppose that you mean something like this which we offer in TraceCompass:
> 
> http://archive.eclipse.org/tracecompass/doc/org.eclipse.tracecompass.doc.user/Trace-synchronization.html
> 
> We have two use cases supported already, distributed systems exchanging packets through TCP and physical / virtual machines communicating through vmenter / exit and tracepoints on both sides of those transitions. This framework can easily be extended to other cases. We could in the future make it available declaratively (define event types and unique identifiers that should be used to match causality-related events).

That's pretty impressive. I'll try to reproduce an example and think through how this could fit into what we're doing.

Background is - we're getting more into supporting fieldbus-connected components, some of which might run our code, so distributed tracing would become an option

with CANbus in particular, an open question is if driving a remote motion controller is within the range of bus throughput; ballpark - no, but there are some drives which have quite intelligent features and it might work. If not, it would help to know where we loose out, and up to which rates it would still work.

It would good to have such a tool at hand, and the developers aware of it, and how to use it. I could actually see a use case for mixed software tracing and hardware-visible events, like from a logic analyzer.

The other area where I could find use for such a capability is UI/server interaction (could it be there might be much wider interest for this than just me?)

Those is a more 'researchy' applications than the one above though. That one we need to nail.

- Michael

Re: introduction & usage of LTTNG in machinekit

[Michel Dagenais]

> The problem at hand has these characteristics:
> 
> - it happens only on low-end ARM platforms
> - the code is part of the trajectory planner, which involves fp and
> transcendental functions
> - the code is executed from a period Xenomai thread and once in a while
> execution time exceeds the cycle time budget
> - no interrupts, system call, or Xenomai API call is involved, so kernel or
> ipipe tracing will not help to narrow it down

I suppose that the time period is in the order of 100us or more?

> my idea was to insert trace points in the tp code and one triggered on 'time
> window exceeded' so one could look backwards in history to determine which
> span of code used excessive time; does this sound reasonable?

In most cases the surprises come from interrupts and competition for resources. If everything is within the tp code, indeed internal tracepoints will help. If you want to be more fancy, there are a few more complicated options. You could trace in "flight recorder" mode, without recording to disk, and trigger a snapshot of the circular buffer content only when a cycle time budget overflow is detected. You could also have a timer, armed for the time window limit, that triggers a stack dump on the interrupted task when fired.

> Alex (in cc) has a test case which reproducibly triggers the problem

Rafaël will start next week. That would indeed be an interesting problem given that it is reproducible.


> That's pretty impressive. I'll try to reproduce an example and think through
> how this could fit into what we're doing.
> 
> Background is - we're getting more into supporting fieldbus-connected
> components, some of which might run our code, so distributed tracing would
> become an option
...
> The other area where I could find use for such a capability is UI/server
> interaction (could it be there might be much wider interest for this than
> just me?)

Indeed, as long as you can record a trace on each node with local timestamps, and have event pairs with a causal relationship in each direction, it should work. You bring the traces together for analysis and synchronize them using these event pairs.

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: introduction & usage of LTTNG in machinekit

[Michel Dagenais]

[-- Attachment #1.1: Type: text/plain, Size: 2307 bytes --]


> That said, the folks which have timing problems have not caught on yet.. it 
> looks some more guidance, examples and a bit of a machinekit-specific 
> writeup is needed, competing with the other 247 priority projects on my desk 
> ;) 

Many people interested in MachineKit wonder about the control platform to select, from the most inexpensive (Raspberry PI) with more latency issues to the more expensive (Intel with FPGA card), with many interesting alternatives in between. The big question is always the real-time response for a typical challenging target such as 5 axis servo + spindle (3 phase variable frequency). We have the ability with LTTng to diagnose such real-time problems and thus to properly tune and then evaluate these different solutions. At the same time, it provides a challenging testbed for the LTTng toolchain. Your comments and suggestions on the hardware setup to test are most welcome! 

Here is my current list of popular and interesting hardware that we would like to try out and properly assess this Summer: 

- Raspberry PI 2, low end and most inexpensive. 

- BeagleBone Black and eventually BeagleBoard X15, slightly more expensive but very interesting for control applications with PRUs, PWM and eQEP. 

- Combined ARM and FPGA with Xilinx chips like Zedboards. The Adapteva Parallela board comes with a Xilinx Z-7010 or Z-7020. You get a dual-core A9, and FPGA for custom logic for encoder inputs and PWM or step generation. 

- Intel board (Atom or better, suggestions for model welcome) with parallel port I/O or any of the available FPGA I/O cards. 

Indeed, while some of the platforms may work without additional hardware (BeagleBone or Zedboard), an FPGA board provides a robust and high performance solution. I have a MesaNet 5I20 but the PCI bus is becoming legacy. Many newer models are offered using high speed data connections which are very convenient but not normally relied upon for real-time work (USB and Ethernet). However, given their high speed and with a dedicated link, it may be workable. Again an excellent testbed for the LTTng toolchain and some hardware measurements. We could test: 

- Mesanet 6I24 FPGA PCIe card with 72 GPIO (baseline for low latency) 

- Mesanet 7I80HD FPGA Ethernet with 72GPIO 

- Mesanet 7I61 FPGA USB with 96 GPIO 


[-- Attachment #1.2: Type: text/html, Size: 2722 bytes --]

[-- Attachment #2: Type: text/plain, Size: 155 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

introduction & usage of LTTNG in machinekit

[Michael Haberler]

Hello,

before I start asking questions - a short intro:

I learned about lttng by David Goulet's talk at FOSDEM 2013

I am involved in machinekit.io, a portable realtime application for motion control
machinekit runs on several kernels - starting from vanilla (no time guarantees), RT-PREEMPT, Xenomai, and RTAI; on x86, amd64 and ARM platforms.

As some of the processing is highly time sensitive, I'm looking into integrating LTTNG into the machinekit build so we have options beyond logging and timestamping manually. For instance, currently some transcendental math functions on ARM overrun the time budget.

I've started integrating the build support and am running into my first problems. So, that's the context of my following questions - thanks in advance for guiding me through any blunders.

looking forward using lttng productively.

Michael

ps: in case you're interested - a talk about the portable realtime API: http://static.mah.priv.at/public/portable-realtime-API-talk/