Re: [Xenomai] Performance impact after switching from 2.6.2.1 to 2.6.4

[Wolfgang Netbal <wolfgang.netbal@sigmatek.at>]

Am 2016-06-06 um 17:35 schrieb Gilles Chanteperdrix:
> On Mon, Jun 06, 2016 at 09:03:40AM +0200, Wolfgang Netbal wrote:
>>
>> Am 2016-06-02 um 10:23 schrieb Gilles Chanteperdrix:
>>> On Thu, Jun 02, 2016 at 10:15:41AM +0200, Wolfgang Netbal wrote:
>>>> Am 2016-06-01 um 16:12 schrieb Gilles Chanteperdrix:
>>>>> On Wed, Jun 01, 2016 at 03:52:06PM +0200, Wolfgang Netbal wrote:
>>>>>> Am 2016-05-31 um 16:16 schrieb Gilles Chanteperdrix:
>>>>>>> On Tue, May 31, 2016 at 04:09:07PM +0200, Wolfgang Netbal wrote:
>>>>>>>> Dear all,
>>>>>>>>
>>>>>>>> we have moved our application from "XENOMAI 2.6.2.1 + Linux 3.0.43" to
>>>>>>>> "XENOMAI 2.6.4. + Linux 3.10.53". Our target is an i.MX6DL. The system
>>>>>>>> is now up and running and works stable. Unfortunately we see a
>>>>>>>> difference in the performance. Our old combination (XENOMAI 2.6.2.1 +
>>>>>>>> Linux 3.0.43) was slightly faster.
>>>>>>>>
>>>>>>>> At the moment it looks like that XENOMAI 2.6.4 calls
>>>>>>>> xnpod_schedule_handler much more often then XENOMAI 2.6.2.1 in our old
>>>>>>>> system.  Every call of xnpod_schedule_handler interrupts our main
>>>>>>>> XENOMAI task with priority = 95.
As I wrote above, I get interrupts 1037 handled by rthal_apc_handler()
and 1038 handled by xnpod_schedule_handler() while my realtime task
is running on kernel 3.10.53 with Xenomai 2.6.4.
On kernel 3.0.43 with Xenomai 2.6.4 there are no interrupts, except the
once that are send by my board using GPIOs, but this virtual interrupts
are assigned to Xenomai and Linux as well but I didn't see a handler 
installed.
I'm pretty sure that these interrupts are slowing down my system, but
where do they come from ?
why didn't I see them on Kernel 3.0.43 with Xenomai 2.6.4 ?
how long do they need to process ?

Is there any dependecy in Xenomai between the kernel version and this 
virtual interrupts ?

>>>>>>>> I have compared the configuration of both XENOMAI versions but did not
>>>>>>>> found any difference. I checked the source code (new commits) but did
>>>>>>>> also not find a solution.
>>>>>>> Have you tried Xenomai 2.6.4 with Linux 3.0.43 ? In order to see
>>>>>>> whether it comes from the kernel update or the Xenomai udpate?
>>>>>> I've tried Linux 3.0.43 with Xenomai 2.6.4 an there is no difference to
>>>>>> Xenomai 2.6.2.1
>>>>>> Looks like there is an other reason than Xenomai.
>>>>> Ok, one thing to pay attention to on imx6 is the L2 cache write
>>>>> allocate policy. You want to disable L2 write allocate on imx6 to
>>>>> get low latencies. I do not know which patches exactly you are
>>>>> using, so it is difficult to check, but the kernel normally displays
>>>>> the value set in the L2 auxiliary configuration register, you can
>>>>> check in the datasheet if it means that L2 write allocate is
>>>>> disabled or not. And check if you get the same value with 3.0 and
>>>>> 3.10.
>>>> Thank you for this hint, I looked around in the kernel config, but cant
>>>> find
>>>> an option sounds like L2 write allocate.
>>>> The only option I found was CACHE_L2X0 and that is activated on both
>>>> kernels.
>>>> Do you have an idea whats the name of this configuration or where in the
>>>> kernel sources it should be located, so I can find out whats the name of
>>>> the
>>>> config flag by searching the sourcecode.
>>> I never talked about any kernel configuration option. I am talking
>>> checking the value passed to the L2 cache auxiliary configuration
>>> register, this is a hardware register. Also, as I said, the value
>>> passed to the L2 cache auxiliary register is printed by the kernel
>>> during boot.
>>>
>>>
>> Sorry Gilles,
>> I found the message in the kernel log, you are right they are different
>> Kernel 3.0.43 shows   l2x0: 16 ways, CACHE_ID 0x410000c8, AUX_CTRL
>> 0x02850000, Cache size: 524288 B
>> Kernel 3.10.53 shows l2x0: 16 ways, CACHE_ID 0x410000c8, AUX_CTRL
>> 0x32c50000, Cache size: 524288 B
>> Kernel 3.10.53 sets addidtional the bits 22 (Shared attribute override
>> enable), 28 (Data prefetch) and 29 (Instruction prefetch)
>> I used the same settings on Kernel 3.0.43 but the perfromance didn't
>> change, looks like this configurations didn't slow down my
>> system.
>>
>> What I have seen while searching the kernel config was that there are a
>> few errate that are activated as dependency in 3.10.53,
>> to be sure none of the errata is the source of my performance reduction
>> I activated them on 3.0.43 as well.
>> But again no difference to our default configuration.
>>
>> To avoid our application is running slower I created a shell-script
>> incrementing a variable
>> 10.000 times and measuring the runtime with time
>>
>> #!/bin/sh
>> var=0
>> while [  $var -lt $1 ]; do
>>       let var++
>> done
>>
>>   > time /mnt/drive-C/CpuTime.sh 10000
>>
>> On this test
>> Kernel 3.0.43 Xenomai 2.6.2.1  needs 480 ms
>> Kernel 3.10.53  Xenomai 2.6.4  needs 820ms
> If you run the same test several times on the same kernel, do you
> reliably always get the same duration?
Yes I run this test 10 times and the values are always nearly the same 
+-10ms
>> This differences are huge, an I'm not sure if I can trust this test
>> because we also use a different busybox,
>> and the difference using our application are between 2% and 3%
>> in the realtime task (Xenomaitask with priority 95)
>> Do you have an idea why this is that much slower ?
> I would not call a 2% or 3% difference "much slower", only
> measurement noise.
I see the difference of 2% or 3% when the realtime task takes only 20% 
of the cpu time
if I set the used cpu time to 90% I have a difference of 12%.
The percente I wrote are average values measured over 10.000 cycles.

On every measurement or test I run is kernel 3.10.53 with xenomai 2.6.4 
slower that
kernel 3.0.43 with xenomai 2.6.2.1
>> I also see differences when I use the xeno-test command to check the speed
>> Some of the operations are faster on newer Xenomai but a few are much
>> slower,
>> for example inlined llimd.
> The differences in the "arith" test are measurement noise. Chances
> are, if you run twice the arith test with the same kernel you are
> not going to find the same values.
>> With every test I run it looks like the issue is not located in Kernel
>> or Xenomai.
>> Do you know any speed issues on system libraries like libc or something
>> like that ?
> Stupid question: do the two kernels run the processor at the same
> speed? You could have a difference if one kernel runs it at 1GHz and
> the other at 800MHz for instance.
>
Yes the tow kernels are running the same processor.
One of the first things I checked was if they setting the same cpu 
frequence
and the RAM settings.
I also changed the microsdcards to avoid that I have a processor that is 
faster.
There are no differences.

Thank you
Wolfgang