[Xenomai] Altera Cyclone V

[Xenomai] Altera Cyclone V

[Lowell Gilbert]

Hi. 

I'm starting out trying to get Xenomai running on the ARM cores in the
"golden reference design" for the Cyclone V SOC from Altera. So far, I'm
still working on getting a kernel booting with the ipipe support, and I
could use a hand. If anyone's already worked with this system, or if I
definitely can't get under 10-microsecond latency with core affinity
keeping my second core clear of non-real-time processing, I'd appreciate
a word.

I'm trying to do this on a 3.10 Linux kernel, because going back much
further will give me problems with backporting some of the peripheral
support. I may try 3.8 to see what happens, but I really can't go any
earlier than 3.6 in any case.

This chip contains a Cortex A9 dual-core ARM system. (It's really not
practical to do anything other than having Linux run on both cores; if I
could run bare-metal on one, I would.) I figured it shouldn't be too
different to use the TSC than on other Cortex A9 chips.

What I'm hitting at the moment is that gt_setup() is getting called, 
but twd_timer_setup() hasn't run yet, so the frequency isn't set yet. 
Both functions are in __cpuinit, and the file would be used the same for
other chips containing the Cortex A9, so I'm confused about what might
be wrong.

Any pointers would be appreciated.

Thanks.

Be well.
        Lowell Gilbert

Re: [Xenomai] Altera Cyclone V

[Gilles Chanteperdrix]

On 01/07/2014 08:52 PM, Lowell Gilbert wrote:
> What I'm hitting at the moment is that gt_setup() is getting called, 
> but twd_timer_setup() hasn't run yet, so the frequency isn't set yet. 
> Both functions are in __cpuinit, and the file would be used the same for
> other chips containing the Cortex A9, so I'm confused about what might
> be wrong.

gt_setup is supposed to be called after twd_local_timer_common_register,
which calls twd_get_clock, which gets the timer frequency from the twd
clock node. If your SOC does not have a clock node for the twd clock,
then you should add one, because the calibration method yields unprecise
results, which may result in "early shots", meaning too many interrupts,
if the GT is not used as TSC.

> 
> Any pointers would be appreciated.

See:
https://www.xenomai.org/index.php/I-pipe-core:ArmPorting
In particular, about this specific issue:

https://www.xenomai.org/index.php/I-pipe-core:ArmPorting#The_Cortex-A9_case

Regards.

-- 
                                                                Gilles.

Re: [Xenomai] Altera Cyclone V

[Lowell Gilbert]

Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org> writes:

> On 01/07/2014 08:52 PM, Lowell Gilbert wrote:
>> What I'm hitting at the moment is that gt_setup() is getting called, 
>> but twd_timer_setup() hasn't run yet, so the frequency isn't set yet. 
>> Both functions are in __cpuinit, and the file would be used the same for
>> other chips containing the Cortex A9, so I'm confused about what might
>> be wrong.
>
> gt_setup is supposed to be called after twd_local_timer_common_register,
> which calls twd_get_clock, which gets the timer frequency from the twd
> clock node. If your SOC does not have a clock node for the twd clock,
> then you should add one, because the calibration method yields unprecise
> results, which may result in "early shots", meaning too many interrupts,
> if the GT is not used as TSC.

But I do want the GT to be my TSC, no?

>> 
>> Any pointers would be appreciated.
>
> See:
> https://www.xenomai.org/index.php/I-pipe-core:ArmPorting
> In particular, about this specific issue:
>
> https://www.xenomai.org/index.php/I-pipe-core:ArmPorting#The_Cortex-A9_case

Ah; I had read that as saying that the clock node would be part of the
general A9 support. It should be the same for all chips using a Cortex
A9, no? 

Come to think of it, this is similar to my surprise at having to write
my own __ipipe_mach_update_tsc(), which surprised me. I'm clearly still
missing something, although I've been back over the documentation to try
to figure out what. 

Thanks again,
             Lowell Gilbert

Re: [Xenomai] Altera Cyclone V

[Gilles Chanteperdrix]

On 01/08/2014 11:45 PM, Lowell Gilbert wrote:
> Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org> writes:
> 
>> On 01/07/2014 08:52 PM, Lowell Gilbert wrote:
>>> What I'm hitting at the moment is that gt_setup() is getting called, 
>>> but twd_timer_setup() hasn't run yet, so the frequency isn't set yet. 
>>> Both functions are in __cpuinit, and the file would be used the same for
>>> other chips containing the Cortex A9, so I'm confused about what might
>>> be wrong.
>>
>> gt_setup is supposed to be called after twd_local_timer_common_register,
>> which calls twd_get_clock, which gets the timer frequency from the twd
>> clock node. If your SOC does not have a clock node for the twd clock,
>> then you should add one, because the calibration method yields unprecise
>> results, which may result in "early shots", meaning too many interrupts,
>> if the GT is not used as TSC.
> 
> But I do want the GT to be my TSC, no?

Yes, but that is not always possible, not all cortex A9 have a GT.
And as I said the calibration is imprecise, so, even if you do not get 
early shots, you will still have the Xenomai clock drifting fast from 
the Linux clock (if it is syncrhonized with ntp for instance).

> 
>>>
>>> Any pointers would be appreciated.
>>
>> See:
>> https://www.xenomai.org/index.php/I-pipe-core:ArmPorting
>> In particular, about this specific issue:
>>
>> https://www.xenomai.org/index.php/I-pipe-core:ArmPorting#The_Cortex-A9_case
> 
> Ah; I had read that as saying that the clock node would be part of the
> general A9 support. It should be the same for all chips using a Cortex
> A9, no? 

No, every ARM SOC has its own clocks implementation. This is currently 
changing with the switch to device tree, but not all SOCs are 
converted, at least not all of them were last time I checked. Anyway, 
if your SOC uses device tree, adding a clock may simply be a matter of 
adding it to the device tree.

> 
> Come to think of it, this is similar to my surprise at having to write
> my own __ipipe_mach_update_tsc(), which surprised me. I'm clearly still

No, you do not have to write your own __ipipe_mach_update_tsc(). See:
https://www.xenomai.org/index.php/I-pipe-core:ArmPorting#High_resolution_counter


-- 
                                                                Gilles.

Re: [Xenomai] Altera Cyclone V

[Lowell Gilbert]

Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org> writes:

> No, you do not have to write your own __ipipe_mach_update_tsc(). See:
> https://www.xenomai.org/index.php/I-pipe-core:ArmPorting#High_resolution_counter

I appreciate the help, but I still don't see an implementation
of __ipipe_mach_update_tsc() that I could have used.
Where was I supposed to get it from?

Re: [Xenomai] Altera Cyclone V

[Gilles Chanteperdrix]

On 01/09/2014 07:06 PM, Lowell Gilbert wrote:
> Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org> writes:
> 
>> No, you do not have to write your own __ipipe_mach_update_tsc(). See:
>> https://www.xenomai.org/index.php/I-pipe-core:ArmPorting#High_resolution_counter
> 
> I appreciate the help, but I still don't see an implementation
> of __ipipe_mach_update_tsc() that I could have used.
> Where was I supposed to get it from?

Simply select IPIPE_ARM_KUSER_TSC and everything gets taken care of
automatically. Since I suppose you read the documentation and did that,
I guess you have a problem. Could you explain us exactly what problem
you have?


-- 
                                                                Gilles.

Re: [Xenomai] Altera Cyclone V

[Lowell Gilbert]

Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org> writes:

> No, every ARM SOC has its own clocks implementation. This is currently 
> changing with the switch to device tree, but not all SOCs are 
> converted, at least not all of them were last time I checked. Anyway, 
> if your SOC uses device tree, adding a clock may simply be a matter of 
> adding it to the device tree.

It was already in the device tree. What I had missed was that the
frequency wasn't being set in the device tree, because there's a clock
manager block separate from the Cortex A9 block, so it wasn't inheriting
the master clock. I set it separately, but I'd prefer to tie it to the
clock manager numbers if I can figure out a way to do that.

I'm now puzzling through interrupt mapping. The BSP doesn't have a
handle_irq function, and I'm not sure whether I need to write one or
not.

Be well.

Re: [Xenomai] Altera Cyclone V

[Charles Steinkuehler]

On 1/7/2014 1:52 PM, Lowell Gilbert wrote:
> Hi. 
> 
> I'm starting out trying to get Xenomai running on the ARM cores in the
> "golden reference design" for the Cyclone V SOC from Altera. So far, I'm
> still working on getting a kernel booting with the ipipe support, and I
> could use a hand. If anyone's already worked with this system, or if I
> definitely can't get under 10-microsecond latency with core affinity
> keeping my second core clear of non-real-time processing, I'd appreciate
> a word.
> 
> I'm trying to do this on a 3.10 Linux kernel, because going back much
> further will give me problems with backporting some of the peripheral
> support. I may try 3.8 to see what happens, but I really can't go any
> earlier than 3.6 in any case.

I'd follow Gilles' advice for tracking down issues.  I'm working with
this SoC for my day job, and Xenomai on the A8 (TI AM3359 on a
BeagleBone) for my 3D printing hobby.  I design hardware and can't be
much help with the hard-core kernel coding, but I have a couple of dev
boards and am willing to do some compiling and testing if it would help.

If you're trying to stay under 10 uS latency, you may need to
instantiate a soft core CPU of some sort in the FPGA fabric.  The ARM A
series cores aren't really designed for low IRQ latency, but I haven't
seen numbers from a dual-core A9 part yet.  The single-core A8 on the
BeagleBone is good for about 25 uS typical and 80 uS or so worst case
latency.  I'm sure the dual-core A9 will be better, but you're needing
almost an order of magnitude improvement which seems unlikely (but
perhaps not impossible).

-- 
Charles Steinkuehler
charles@steinkuehler.net

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 261 bytes
Desc: OpenPGP digital signature
URL: <http://www.xenomai.org/pipermail/xenomai/attachments/20140107/e03ccb7f/attachment.sig>

Re: [Xenomai] Altera Cyclone V

[Gilles Chanteperdrix]

On 01/07/2014 11:19 PM, Charles Steinkuehler wrote:
> The single-core A8 on the
> BeagleBone is good for about 25 uS typical and 80 uS or so worst case
> latency.

That is really high. On a 720MHz OMAP3, with the latency test running
with a 100us period, I typically get latencies close to 40us (under
dohell load). Granted I do not run many functionalities of the SOC
(typically, not the graphic processor), but I would not expect latencies
to get so high. Is there any chance you could trigger a trace with the
I-pipe tracer?

-- 
                                                                Gilles.

Re: [Xenomai] Altera Cyclone V

[Charles Steinkuehler]

On 1/7/2014 4:47 PM, Gilles Chanteperdrix wrote:
> On 01/07/2014 11:19 PM, Charles Steinkuehler wrote:
>> The single-core A8 on the
>> BeagleBone is good for about 25 uS typical and 80 uS or so worst case
>> latency.
> 
> That is really high. On a 720MHz OMAP3, with the latency test running
> with a 100us period, I typically get latencies close to 40us (under
> dohell load). Granted I do not run many functionalities of the SOC
> (typically, not the graphic processor), but I would not expect latencies
> to get so high. Is there any chance you could trigger a trace with the
> I-pipe tracer?

I can try...give me a while to sort through the I-pipe:Tracer wiki page
(or are there better instructions?).

This may also have improved somewhat...my numbers are from a year or so
ago when everything first started working.  I haven't been regression
testing the latest kernels.  <blush>  I know there have been significant
improvements on both the Xenomai side and with the BeagleBone kernel
patches/drivers.

-- 
Charles Steinkuehler
charles@steinkuehler.net

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 261 bytes
Desc: OpenPGP digital signature
URL: <http://www.xenomai.org/pipermail/xenomai/attachments/20140107/a2c5d972/attachment.sig>

Re: [Xenomai] Altera Cyclone V

[Gilles Chanteperdrix]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 01/07/2014 11:55 PM, Charles Steinkuehler wrote:
> On 1/7/2014 4:47 PM, Gilles Chanteperdrix wrote:
>> On 01/07/2014 11:19 PM, Charles Steinkuehler wrote:
>>> The single-core A8 on the BeagleBone is good for about 25 uS
>>> typical and 80 uS or so worst case latency.
>> 
>> That is really high. On a 720MHz OMAP3, with the latency test
>> running with a 100us period, I typically get latencies close to
>> 40us (under dohell load). Granted I do not run many
>> functionalities of the SOC (typically, not the graphic
>> processor), but I would not expect latencies to get so high. Is
>> there any chance you could trigger a trace with the I-pipe
>> tracer?
> 
> I can try...give me a while to sort through the I-pipe:Tracer wiki
> page (or are there better instructions?).

Enable I-pipe tracer in kernel configuration, especially
IPIPE_TRACE_MCOUNT, and IPIPE_TRACE_VMALLOC.

When the system has booted, do:

echo 1 > /proc/ipipe/trace/enable
echo 1000 > /proc/ipipe/trace/back_trace_points

Then launch latency -p 200 -f

Run the load you usually run (for instance dohell).

When the test is finished, do
cat /proc/ipipe/trace/frozen > /tmp/frozen.txt

then post the resulting frozen.txt to this list.

Expect the latency found by the test to be double the latency you get
without the I-pipe tracer.

> 
> This may also have improved somewhat...my numbers are from a year
> or so ago when everything first started working.  I haven't been
> regression testing the latest kernels.  <blush>  I know there have
> been significant improvements on both the Xenomai side and with the
> BeagleBone kernel patches/drivers.

Actually, OMAP3 latency has been around 40us since 2.6.35. 2.6.33 even
was at 35us. Note that it is the latency at 10kHz, if you are running
with a lower frequency, you will get higher latencies.

Also note that there are some options which improve latency, such as:
compiling kernel and user-space using thumb2 (but in that case do not
forget to enable ARM_ERRATA_430973)
optimize for size
disable CONFIG_PREEMPT
enable stack unwinding
disable CONFIG_SMP (obviously, but it may be enabled by default if you
are using omap2plus_defconfig, since it also compiles for omap4)
disable unlocked context switch (xenomai option)
disable root thread priority coupling (xenomai option)


- -- 
                                                                Gilles.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with Icedove - http://www.enigmail.net/

iD8DBQFSzIoYGpcgE6m/fboRAhyRAJ48P6kxKSPh2IeNP0xwwh8cr35KhwCfQo20
cMlOFxFAV7dVNKaPWi6+XW0=
=hMP+
-----END PGP SIGNATURE-----

Re: [Xenomai] Altera Cyclone V

[Charles Steinkuehler]

On 1/7/2014 5:13 PM, Gilles Chanteperdrix wrote:
> On 01/07/2014 11:55 PM, Charles Steinkuehler wrote:
>> On 1/7/2014 4:47 PM, Gilles Chanteperdrix wrote:
>>> On 01/07/2014 11:19 PM, Charles Steinkuehler wrote:
>>>> The single-core A8 on the BeagleBone is good for about 25 uS
>>>> typical and 80 uS or so worst case latency.
>>>
>>> That is really high. On a 720MHz OMAP3, with the latency test
>>> running with a 100us period, I typically get latencies close to
>>> 40us (under dohell load). Granted I do not run many
>>> functionalities of the SOC (typically, not the graphic
>>> processor), but I would not expect latencies to get so high. Is
>>> there any chance you could trigger a trace with the I-pipe
>>> tracer?
> 
>> I can try...give me a while to sort through the I-pipe:Tracer wiki
>> page (or are there better instructions?).
> 
> Enable I-pipe tracer in kernel configuration, especially
> IPIPE_TRACE_MCOUNT, and IPIPE_TRACE_VMALLOC.
> 
> When the system has booted, do:

<snip>

Thanks for the details!  My to-do list just got longer...  :)

I'll do some testing (on the BeagleBone) and post the results, but it
might take a couple of days.

FYI, the GPU is currently disabled on the BeagleBone, so no mysterious
latency from that source (hopefully).  I need to migrate to a 3.12
kernel to get the GPU working (and it doesn't currently support X11,
only framebuffer applications), which is part of why I'm interested in
Xenomai on the 3.10 or newer kernel for the Cyclone-V SoC.

-- 
Charles Steinkuehler
charles@steinkuehler.net

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 261 bytes
Desc: OpenPGP digital signature
URL: <http://www.xenomai.org/pipermail/xenomai/attachments/20140107/99bb4e74/attachment.sig>

Re: [Xenomai] Altera Cyclone V

[Charles Steinkuehler]

On 1/7/2014 6:51 PM, Charles Steinkuehler wrote:
> 
> Thanks for the details!  My to-do list just got longer...  :)
> 
> I'll do some testing (on the BeagleBone) and post the results, but it
> might take a couple of days.

I did a quick check with the BeagleBone kernels I've already built (no
IPipe debugging) and I'm getting much better results than the last time
I tested.  Typical latency is around 20 uS, and worst case is about 45
uS (although I obviously didn't run overnight!).

I've got about half of the suggested kernel settings "correct" already,
but the others need to be tweaked (including disabling SMP, thanks for
the hint!).  I tried using all the recommended settings, but the kernel
build failed when linking:

  LD      init/built-in.o
arch/arm/kernel/built-in.o: In function `__irq_usr':
:(.kprobes.text+0x3c2): relocation truncated to fit: R_ARM_THM_JUMP19
against symbol `__ipipe_ret_to_user_irqs_disabled' defined in .text
section in arch/arm/kernel/built-in.o
arch/arm/kernel/built-in.o: In function `ret_from_exception':
:(.kprobes.text+0x600): relocation truncated to fit: R_ARM_THM_JUMP19
against symbol `__ipipe_ret_to_user_irqs_disabled' defined in .text
section in arch/arm/kernel/built-in.o
make: *** [vmlinux] Error 1

The link error goes away if I disable CONFIG_THUMB2_KERNEL, which wasn't
set in the original kernel config.

I'll report details when I get some debug kernels tested and capture the
requested data.

-- 
Charles Steinkuehler
charles@steinkuehler.net

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 261 bytes
Desc: OpenPGP digital signature
URL: <http://www.xenomai.org/pipermail/xenomai/attachments/20140107/d19016a8/attachment.sig>

Re: [Xenomai] Altera Cyclone V

[Gilles Chanteperdrix]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 01/08/2014 05:04 AM, Charles Steinkuehler wrote:
> On 1/7/2014 6:51 PM, Charles Steinkuehler wrote:
>> 
>> Thanks for the details!  My to-do list just got longer...  :)
>> 
>> I'll do some testing (on the BeagleBone) and post the results,
>> but it might take a couple of days.
> 
> I did a quick check with the BeagleBone kernels I've already built
> (no IPipe debugging) and I'm getting much better results than the
> last time I tested.  Typical latency is around 20 uS, and worst
> case is about 45 uS (although I obviously didn't run overnight!).
> 
> I've got about half of the suggested kernel settings "correct"
> already, but the others need to be tweaked (including disabling
> SMP, thanks for the hint!).  I tried using all the recommended
> settings, but the kernel build failed when linking:
> 
> LD      init/built-in.o arch/arm/kernel/built-in.o: In function
> `__irq_usr': :(.kprobes.text+0x3c2): relocation truncated to fit:
> R_ARM_THM_JUMP19 against symbol `__ipipe_ret_to_user_irqs_disabled'
> defined in .text section in arch/arm/kernel/built-in.o 
> arch/arm/kernel/built-in.o: In function `ret_from_exception': 
> :(.kprobes.text+0x600): relocation truncated to fit:
> R_ARM_THM_JUMP19 against symbol `__ipipe_ret_to_user_irqs_disabled'
> defined in .text section in arch/arm/kernel/built-in.o make: ***
> [vmlinux] Error 1
> 
> The link error goes away if I disable CONFIG_THUMB2_KERNEL, which
> wasn't set in the original kernel config.

Have you tried enabling workaround for broken gas?
CONFIG_THUMB2_AVOID_R_ARM_THM_JUMP11?

Also note that you may want to use a toolchain with hard floating
points. I use linaro toolchains for instance.


- -- 
                                                                Gilles.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with Icedove - http://www.enigmail.net/

iD8DBQFSzQqhGpcgE6m/fboRAjqcAJ9RZpIOpB/btilqK7VdeXZEsgpJOQCdEBYg
4nGKdWB78EzLvKIDKwqpvs4=
=kE9J
-----END PGP SIGNATURE-----

Re: [Xenomai] Altera Cyclone V

[Gilles Chanteperdrix]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 01/08/2014 05:04 AM, Charles Steinkuehler wrote:
> On 1/7/2014 6:51 PM, Charles Steinkuehler wrote:
>> 
>> Thanks for the details!  My to-do list just got longer...  :)
>> 
>> I'll do some testing (on the BeagleBone) and post the results,
>> but it might take a couple of days.
> 
> I did a quick check with the BeagleBone kernels I've already built
> (no IPipe debugging) and I'm getting much better results than the
> last time I tested.  Typical latency is around 20 uS, and worst
> case is about 45 uS (although I obviously didn't run overnight!).
> 
> I've got about half of the suggested kernel settings "correct"
> already, but the others need to be tweaked (including disabling
> SMP, thanks for the hint!).  I tried using all the recommended
> settings, but the kernel build failed when linking:
> 
> LD      init/built-in.o arch/arm/kernel/built-in.o: In function
> `__irq_usr': :(.kprobes.text+0x3c2): relocation truncated to fit:
> R_ARM_THM_JUMP19 against symbol `__ipipe_ret_to_user_irqs_disabled'
> defined in .text section in arch/arm/kernel/built-in.o 
> arch/arm/kernel/built-in.o: In function `ret_from_exception': 
> :(.kprobes.text+0x600): relocation truncated to fit:
> R_ARM_THM_JUMP19 against symbol `__ipipe_ret_to_user_irqs_disabled'
> defined in .text section in arch/arm/kernel/built-in.o make: ***
> [vmlinux] Error 1
> 
> The link error goes away if I disable CONFIG_THUMB2_KERNEL, which
> wasn't set in the original kernel config.
> 
> I'll report details when I get some debug kernels tested and
> capture the requested data.
> 
Also, something to be checked is whether the "PIC muting" is enabled
for the beaglebone. PIC muting is already implemented for the OMAP3
interrupt controller and GPIOS, you simply need to check that the
initialization function (omap3_pic_muter_register) gets called in the
BeagleBone processor initialization.

- -- 
                                                                Gilles.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with Icedove - http://www.enigmail.net/

iD8DBQFSzRMsGpcgE6m/fboRAmP6AJ46enpi6Yy9+cJ3iP1FBvLGOKjQIQCggg8y
pcHgitqDiBjDR0lxmjDhbdE=
=4/vB
-----END PGP SIGNATURE-----

[Xenomai] i-pipe tracer on in production kernels? (was Re: Altera Cyclone V)

[Michael Haberler]

http://www.xenomai.org/index.php/I-pipe:Tracer describes the trace API, which could be useful to track down issues

Q: does enabling the tracer incur significant overhead if compiled in but unused, or is it reasonable to leave it on in a production kernel?

if the former, we might have to build/make available a second i-pipe tracer enabled kernel to track down issues 'in the field'; if the latter, it'd be less build/distribution chores

thanks in advance,

Michael



Am 08.01.2014 um 01:51 schrieb Charles Steinkuehler <charles@steinkuehler.net>:

> On 1/7/2014 5:13 PM, Gilles Chanteperdrix wrote:
>> On 01/07/2014 11:55 PM, Charles Steinkuehler wrote:
>>> On 1/7/2014 4:47 PM, Gilles Chanteperdrix wrote:
>>>> On 01/07/2014 11:19 PM, Charles Steinkuehler wrote:
>>>>> The single-core A8 on the BeagleBone is good for about 25 uS
>>>>> typical and 80 uS or so worst case latency.
>>>> 
>>>> That is really high. On a 720MHz OMAP3, with the latency test
>>>> running with a 100us period, I typically get latencies close to
>>>> 40us (under dohell load). Granted I do not run many
>>>> functionalities of the SOC (typically, not the graphic
>>>> processor), but I would not expect latencies to get so high. Is
>>>> there any chance you could trigger a trace with the I-pipe
>>>> tracer?
>> 
>>> I can try...give me a while to sort through the I-pipe:Tracer wiki
>>> page (or are there better instructions?).
>> 
>> Enable I-pipe tracer in kernel configuration, especially
>> IPIPE_TRACE_MCOUNT, and IPIPE_TRACE_VMALLOC.
>> 
>> When the system has booted, do:
> 
> <snip>
> 
> Thanks for the details!  My to-do list just got longer...  :)
> 
> I'll do some testing (on the BeagleBone) and post the results, but it
> might take a couple of days.
> 
> FYI, the GPU is currently disabled on the BeagleBone, so no mysterious
> latency from that source (hopefully).  I need to migrate to a 3.12
> kernel to get the GPU working (and it doesn't currently support X11,
> only framebuffer applications), which is part of why I'm interested in
> Xenomai on the 3.10 or newer kernel for the Cyclone-V SoC.
> 
> -- 
> Charles Steinkuehler
> charles@steinkuehler.net
> 
> -------------- next part --------------
> A non-text attachment was scrubbed...
> Name: signature.asc
> Type: application/pgp-signature
> Size: 261 bytes
> Desc: OpenPGP digital signature
> URL: <http://www.xenomai.org/pipermail/xenomai/attachments/20140107/99bb4e74/attachment.sig>
> _______________________________________________
> Xenomai mailing list
> Xenomai@xenomai.org
> http://www.xenomai.org/mailman/listinfo/xenomai

Re: [Xenomai] i-pipe tracer on in production kernels? (was Re: Altera Cyclone V)

[Gilles Chanteperdrix]

On 01/11/2014 11:22 AM, Michael Haberler wrote:
> 
> http://www.xenomai.org/index.php/I-pipe:Tracer describes the trace
> API, which could be useful to track down issues
> 
> Q: does enabling the tracer incur significant overhead if compiled in
> but unused, or is it reasonable to leave it on in a production
> kernel?

You should run some tests to answer that question: measure the latency
with tracer compiled-in but not enabled, and with tracer completely
disabled. I would expect to see a difference between the two
configurations, but I do not know if it would be very significant.

-- 
                                                                Gilles.

Re: [Xenomai] i-pipe tracer on in production kernels? (was Re: Altera Cyclone V)

[Gilles Chanteperdrix]

On 01/11/2014 11:22 AM, Michael Haberler wrote:
> 
> http://www.xenomai.org/index.php/I-pipe:Tracer describes the trace
> API, which could be useful to track down issues
> 
> Q: does enabling the tracer incur significant overhead if compiled in
> but unused, or is it reasonable to leave it on in a production
> kernel?
> 
> if the former, we might have to build/make available a second i-pipe
> tracer enabled kernel to track down issues 'in the field'; if the
> latter, it'd be less build/distribution chores

Normally, with DYNAMIC_FTRACE, there should be a way to reduce the
overhead. Unfortunately, the I-pipe tracer disables DYNAMIC_FTRACE,
because at some point in the past, it did not work (with the I-pipe
tracer) on ARM. We could re-enable DYNAMIC_FTRACE and try and understand
why it does not work (if it still does not work), but that means you
would have to add patches on top of the I-pipe patch, because all I-pipe
patches released for some time, forbid DYNAMIC_FTRACE with the I-pipe
tracer, at least on ARM.

-- 
                                                                Gilles.

Re: [Xenomai] i-pipe tracer on in production kernels? (was Re: Altera Cyclone V)

[Philippe Gerum]

On 01/11/2014 11:22 AM, Michael Haberler wrote:
>
> http://www.xenomai.org/index.php/I-pipe:Tracer describes the trace API, which could be useful to track down issues
>
> Q: does enabling the tracer incur significant overhead if compiled in but unused, or is it reasonable to leave it on in a production kernel?
>
> if the former, we might have to build/make available a second i-pipe tracer enabled kernel to track down issues 'in the field'; if the latter, it'd be less build/distribution chores
>

The I-pipe tracer is based on the ftrace infrastructure, so tracepoints 
are left out when the tracer is disabled. If you can live with the 
minimal overhead ftrace adds, then leaving the I-pipe tracer code built 
in may be an option.

> thanks in advance,
>
> Michael
>
>
>
> Am 08.01.2014 um 01:51 schrieb Charles Steinkuehler <charles@steinkuehler.net>:
>
>> On 1/7/2014 5:13 PM, Gilles Chanteperdrix wrote:
>>> On 01/07/2014 11:55 PM, Charles Steinkuehler wrote:
>>>> On 1/7/2014 4:47 PM, Gilles Chanteperdrix wrote:
>>>>> On 01/07/2014 11:19 PM, Charles Steinkuehler wrote:
>>>>>> The single-core A8 on the BeagleBone is good for about 25 uS
>>>>>> typical and 80 uS or so worst case latency.
>>>>>
>>>>> That is really high. On a 720MHz OMAP3, with the latency test
>>>>> running with a 100us period, I typically get latencies close to
>>>>> 40us (under dohell load). Granted I do not run many
>>>>> functionalities of the SOC (typically, not the graphic
>>>>> processor), but I would not expect latencies to get so high. Is
>>>>> there any chance you could trigger a trace with the I-pipe
>>>>> tracer?
>>>
>>>> I can try...give me a while to sort through the I-pipe:Tracer wiki
>>>> page (or are there better instructions?).
>>>
>>> Enable I-pipe tracer in kernel configuration, especially
>>> IPIPE_TRACE_MCOUNT, and IPIPE_TRACE_VMALLOC.
>>>
>>> When the system has booted, do:
>>
>> <snip>
>>
>> Thanks for the details!  My to-do list just got longer...  :)
>>
>> I'll do some testing (on the BeagleBone) and post the results, but it
>> might take a couple of days.
>>
>> FYI, the GPU is currently disabled on the BeagleBone, so no mysterious
>> latency from that source (hopefully).  I need to migrate to a 3.12
>> kernel to get the GPU working (and it doesn't currently support X11,
>> only framebuffer applications), which is part of why I'm interested in
>> Xenomai on the 3.10 or newer kernel for the Cyclone-V SoC.
>>
>> --
>> Charles Steinkuehler
>> charles@steinkuehler.net
>>
>> -------------- next part --------------
>> A non-text attachment was scrubbed...
>> Name: signature.asc
>> Type: application/pgp-signature
>> Size: 261 bytes
>> Desc: OpenPGP digital signature
>> URL: <http://www.xenomai.org/pipermail/xenomai/attachments/20140107/99bb4e74/attachment.sig>
>> _______________________________________________
>> Xenomai mailing list
>> Xenomai@xenomai.org
>> http://www.xenomai.org/mailman/listinfo/xenomai
>
>
> _______________________________________________
> Xenomai mailing list
> Xenomai@xenomai.org
> http://www.xenomai.org/mailman/listinfo/xenomai
>


-- 
Philippe.

Re: [Xenomai] i-pipe tracer on in production kernels? (was Re: Altera Cyclone V)

[Gilles Chanteperdrix]

On 01/11/2014 07:20 PM, Philippe Gerum wrote:
> On 01/11/2014 11:22 AM, Michael Haberler wrote:
>>
>> http://www.xenomai.org/index.php/I-pipe:Tracer describes the trace API, which could be useful to track down issues
>>
>> Q: does enabling the tracer incur significant overhead if compiled in but unused, or is it reasonable to leave it on in a production kernel?
>>
>> if the former, we might have to build/make available a second i-pipe tracer enabled kernel to track down issues 'in the field'; if the latter, it'd be less build/distribution chores
>>
> 
> The I-pipe tracer is based on the ftrace infrastructure, so tracepoints 
> are left out when the tracer is disabled. If you can live with the 
> minimal overhead ftrace adds, then leaving the I-pipe tracer code built 
> in may be an option.

That is less true without DYNAMIC_FTRACE than with DYNAMIC_FTRACE.


-- 
                                                                Gilles.

Re: [Xenomai] i-pipe tracer on in production kernels? (was Re: Altera Cyclone V)

[Philippe Gerum]

On 01/11/2014 07:27 PM, Gilles Chanteperdrix wrote:
> On 01/11/2014 07:20 PM, Philippe Gerum wrote:
>> On 01/11/2014 11:22 AM, Michael Haberler wrote:
>>>
>>> http://www.xenomai.org/index.php/I-pipe:Tracer describes the trace API, which could be useful to track down issues
>>>
>>> Q: does enabling the tracer incur significant overhead if compiled in but unused, or is it reasonable to leave it on in a production kernel?
>>>
>>> if the former, we might have to build/make available a second i-pipe tracer enabled kernel to track down issues 'in the field'; if the latter, it'd be less build/distribution chores
>>>
>>
>> The I-pipe tracer is based on the ftrace infrastructure, so tracepoints
>> are left out when the tracer is disabled. If you can live with the
>> minimal overhead ftrace adds, then leaving the I-pipe tracer code built
>> in may be an option.
>
> That is less true without DYNAMIC_FTRACE than with DYNAMIC_FTRACE.
>
>

This is about 10% overhead on low-end 52xx, 5-7% on am335x. I did not check on typical x86 < 2 Ghz yet, but this should be around 3-5%. It is just a matter of how much is too much.

Regarding DYNAMIC_FTRACE, we have had this out for too many moons (2.6.2x...), and we need to reconsider this (this looks somewhat like an old writing from an ancient book, nobody really remembers the meaning of - I wonder if I was not the guy who switched this off, out of some nightmare debugging session).

Anyway, enabling it on am335x looks ok at first sight, and I suspect that the usual problem of synchronizing better and harder against the code patching loop is gone.

i.e.

root@android:/ # cat /proc/ipipe/trace/enable                                  
1
root@android:/ # /bin/latency [   93.709690] request_suspend_state: wakeup (0->0) at 93601080338 (2000-01-01 00:54:46.112646335 UTC)
[   93.760153] init: untracked pid 637 exited
[   93.764831] init: untracked pid 638 exited

== Sampling period: 1000 us
== Test mode: periodic user-mode task
== All results in microseconds
warming up...
RTT|  00:00:01  (periodic user-mode task, 1000 us period, priority 99)
RTH|----lat min|----lat avg|----lat max|-overrun|---msw|---lat best|--lat worst
RTD|     26.999|     45.833|     72.958|       0|     0|     26.999|     72.958
RTD|     25.749|     47.208|     71.499|       0|     0|     25.749|     72.958
RTD|     25.416|     47.124|     73.333|       0|     0|     25.416|     73.333
RTD|     25.916|     46.541|     72.499|       0|     0|     25.416|     73.333
RTD|     27.124|     47.583|     75.624|       0|     0|     25.416|     75.624
^C---|-----------|-----------|-----------|--------|------|-------------------------
RTS|     25.416|     46.833|     75.624|       0|     0|    00:00:05/00:00:05
root@android:/ # echo 0 > /proc/ipipe/trace/enable                             
root@android:/ # /bin/latency                                                  
== Sampling period: 1000 us
== Test mode: periodic user-mode task
== All results in microseconds
warming up...
RTT|  00:00:01  (periodic user-mode task, 1000 us period, priority 99)
RTH|----lat min|----lat avg|----lat max|-overrun|---msw|---lat best|--lat worst
RTD|     -3.209|     21.291|     57.499|       0|     0|     -3.209|     57.499
RTD|      0.000|     26.666|     53.208|       0|     0|     -3.209|     57.499
RTD|      2.291|     19.041|     57.958|       0|     0|     -3.209|     57.958
RTD|      0.999|     16.541|     56.624|       0|     0|     -3.209|     57.958
RTD|      2.333|     17.291|     51.958|       0|     0|     -3.209|     57.958
^C---|-----------|-----------|-----------|--------|------|-------------------------
RTS|     -3.209|     20.166|     57.958|       0|     0|    00:00:05/00:00:05
root@android:/ # [  116.008470] request_suspend_state: wakeup (0->0) at 115899865474 (2000-01-01 00:55:08.411433263 UTC)
[  116.027659] init: untracked pid 898 exited
[  116.039731] init: untracked pid 899 exited


-- 
Philippe.

Re: [Xenomai] i-pipe tracer on in production kernels? (was Re: Altera Cyclone V)

[Gilles Chanteperdrix]

On 01/11/2014 08:01 PM, Philippe Gerum wrote:
> On 01/11/2014 07:27 PM, Gilles Chanteperdrix wrote:
>> On 01/11/2014 07:20 PM, Philippe Gerum wrote:
>>> On 01/11/2014 11:22 AM, Michael Haberler wrote:
>>>> 
>>>> http://www.xenomai.org/index.php/I-pipe:Tracer describes the
>>>> trace API, which could be useful to track down issues
>>>> 
>>>> Q: does enabling the tracer incur significant overhead if
>>>> compiled in but unused, or is it reasonable to leave it on in a
>>>> production kernel?
>>>> 
>>>> if the former, we might have to build/make available a second
>>>> i-pipe tracer enabled kernel to track down issues 'in the
>>>> field'; if the latter, it'd be less build/distribution chores
>>>> 
>>> 
>>> The I-pipe tracer is based on the ftrace infrastructure, so
>>> tracepoints are left out when the tracer is disabled. If you can
>>> live with the minimal overhead ftrace adds, then leaving the
>>> I-pipe tracer code built in may be an option.
>> 
>> That is less true without DYNAMIC_FTRACE than with DYNAMIC_FTRACE.
>> 
>> 
> 
> This is about 10% overhead on low-end 52xx, 5-7% on am335x. I did not
> check on typical x86 < 2 Ghz yet, but this should be around 3-5%. It
> is just a matter of how much is too much.
> 
> Regarding DYNAMIC_FTRACE, we have had this out for too many moons
> (2.6.2x...), and we need to reconsider this (this looks somewhat like
> an old writing from an ancient book, nobody really remembers the
> meaning of - I wonder if I was not the guy who switched this off, out
> of some nightmare debugging session).

I believed it was me, but checking the logs, it seems to be due to this
commit:

https://git.xenomai.org/ipipe-gch.git/commit/kernel/trace/Kconfig?h=for-ipipe-2.6.29-arm&id=6e8b204b6ec92ca007ec099f2878c340f5b64b48

So, it seems it has been that way since the I-pipe tracer is based on
ftrace.

> 
> Anyway, enabling it on am335x looks ok at first sight, and I suspect
> that the usual problem of synchronizing better and harder against the
> code patching loop is gone.

I have tried on OMAP3 on my side, and I agree it seems to work. The code
patching is done within an ipipe_critical added by the I-pipe patch.

-- 
                                                                Gilles.

Re: [Xenomai] Altera Cyclone V

[Lowell Gilbert]

Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org> writes:

> On 01/07/2014 11:19 PM, Charles Steinkuehler wrote:
>> The single-core A8 on the
>> BeagleBone is good for about 25 uS typical and 80 uS or so worst case
>> latency.
>
> That is really high. On a 720MHz OMAP3, with the latency test running
> with a 100us period, I typically get latencies close to 40us (under
> dohell load). Granted I do not run many functionalities of the SOC
> (typically, not the graphic processor), but I would not expect latencies
> to get so high. Is there any chance you could trigger a trace with the
> I-pipe tracer?

That seems high even to me. I got into that range on the Cyclone V eval
board with PREEMPT_RT alone.