2.6.33.7-rt29 PREEMPT_RT worse latency than PREEMPT_DESKTOP on AT91?

2.6.33.7-rt29 PREEMPT_RT worse latency than PREEMPT_DESKTOP on AT91?

[Agustin Ferrin Pozuelo]

Hello,

I am fine tuning the configuration for an ARM system derived from 
AT91SAM9263-EK.

My goal is to minimize latency, and I am using "cyclictest" from 
rt-tools v0.78 for measuring it.

I get consistently better latency with PREEMPT_DESKTOP over what I get 
with PREEMPT_RT. This is an example for a very simple test run, which 
reflects the overall results I am getting:

    ### PREEMPT-RT, HRT, no NO_HZ, no RTC, no USE_SLOW_CLOCK

    root@at91sam9263cpc:~# time cyclictest -i 700000 -r -p 80 -l 33
    Clock resolution: 0.000000001 (s.ns)
    policy: fifo: loadavg: 0.11 0.10 0.04 1/49 884

    T: 0 (  861) P:80 I:700000 C:     33 Min:    370 Act:  627 Avg:  437 Max:     627
    real	0m 23.51s
    user	0m 0.38s
    sys	0m 2.20s

    ### PREEMPT-DESKTOP (no RT), HRT, no NO_HZ, no USE_SLOW_CLOCK
    root@at91sam9263cpc:~# time cyclictest -i 700000 -r -p 80 -l 33
    Clock resolution: 0.000000001 (s.ns)
    policy: fifo: loadavg: 0.24 0.08 0.03 1/46 600

    T: 0 (  574) P:80 I:700000 C:     33 Min:    173 Act:  196 Avg:  222 Max:     378
    real	0m 23.66s
    user	0m 0.32s
    sys	0m 1.29s
       

That is, I get much better min, avg, and max latency with 
"PREEMPT_DESKTOP". How can this be?

Anyway, the results seem worse than the <150us worst case) obtained with 
2.6.24.7-rt27 by Remy Bohmer 
<https://rt.wiki.kernel.org/index.php/CONFIG_PREEMPT_RT_Patch#Platforms_Tested_and_in_Use_with_CONFIG_PREEMPT_RT>, 
though he was apparently/ not using 'Tickless and HRT' /because of CPU 
usage issues... But without those I can't get cyclictest to run properly!

My guess is that the RT patch adds code (checks, locks) to may critical 
sections that happens to be specially slow on ARM9...

Clues & comments welcome!
--Agustín.

-- 
[CG logo]

Agustín Ferrín Pozuelo

Embedded Systems Engineer

CG Power Systems Ireland Limited

Automation Systems Division

Herbert House, Harmony Row, Dublin 2, Ireland.

Phone: +353 1 4153700 Web: www.cgglobal.com <http://www.cgglobal.com>

Save the environment. Please print only if essential.



CG DISCLAIMER: This email contains confidential information. It is intended exclusively for the addressees. If you are not an addressee, you must not store, transmit or disclose its contents. Instead please notify the sender immediately; and permanently delete this e-mail from your computer systems. We have taken reasonable precautions to ensure that no viruses are present. However, you must check this email and the attachments, for viruses. We accept no liability whatsoever, for any detriment caused by any transmitted virus.

--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 2.6.33.7-rt29 PREEMPT_RT worse latency than PREEMPT_DESKTOP on AT91?

[Thomas Gleixner]

On Mon, 23 Aug 2010, Agustin Ferrin Pozuelo wrote:

> Hello,
> 
> I am fine tuning the configuration for an ARM system derived from
> AT91SAM9263-EK.
> 
> My goal is to minimize latency, and I am using "cyclictest" from rt-tools
> v0.78 for measuring it.
> 
> I get consistently better latency with PREEMPT_DESKTOP over what I get with
> PREEMPT_RT. This is an example for a very simple test run, which reflects the
> overall results I am getting:
> 
>    ### PREEMPT-RT, HRT, no NO_HZ, no RTC, no USE_SLOW_CLOCK
> 
>    root@at91sam9263cpc:~# time cyclictest -i 700000 -r -p 80 -l 33
>    Clock resolution: 0.000000001 (s.ns)
>    policy: fifo: loadavg: 0.11 0.10 0.04 1/49 884
> 
>    T: 0 (  861) P:80 I:700000 C:     33 Min:    370 Act:  627 Avg:  437 Max:
> 627
>    real	0m 23.51s
>    user	0m 0.38s
>    sys	0m 2.20s
> 
>    ### PREEMPT-DESKTOP (no RT), HRT, no NO_HZ, no USE_SLOW_CLOCK
>    root@at91sam9263cpc:~# time cyclictest -i 700000 -r -p 80 -l 33
>    Clock resolution: 0.000000001 (s.ns)
>    policy: fifo: loadavg: 0.24 0.08 0.03 1/46 600
> 
>    T: 0 (  574) P:80 I:700000 C:     33 Min:    173 Act:  196 Avg:  222 Max:
> 378
>    real	0m 23.66s
>    user	0m 0.32s
>    sys	0m 1.29s

33 loops are not really giving you any useful information. Also run
both tests with some background load and not on a fully idle system.

Thanks,

	tglx

Re: 2.6.33.7-rt29 PREEMPT_RT worse latency than PREEMPT_DESKTOP on AT91?

[Agustin Ferrin Pozuelo]

  On 27/08/10 11:35, Thomas Gleixner wrote:
> On Mon, 23 Aug 2010, Agustin Ferrin Pozuelo wrote:
>> Hello,
>>
>> I am fine tuning the configuration for an ARM system derived from
>> AT91SAM9263-EK.
>>
>> My goal is to minimize latency, and I am using "cyclictest" from rt-tools
>> v0.78 for measuring it.
>>
>> I get consistently better latency with PREEMPT_DESKTOP over what I get with
>> PREEMPT_RT. This is an example for a very simple test run, which reflects the
>> overall results I am getting:
>>
>>     ### PREEMPT-RT, HRT, no NO_HZ, no RTC, no USE_SLOW_CLOCK
>>
>>     root@at91sam9263cpc:~# time cyclictest -i 700000 -r -p 80 -l 33
>>     Clock resolution: 0.000000001 (s.ns)
>>     policy: fifo: loadavg: 0.11 0.10 0.04 1/49 884
>>
>>     T: 0 (  861) P:80 I:700000 C:     33 Min:    370 Act:  627 Avg:  437 Max:
>> 627
>>     real	0m 23.51s
>>     user	0m 0.38s
>>     sys	0m 2.20s
>>
>>     ### PREEMPT-DESKTOP (no RT), HRT, no NO_HZ, no USE_SLOW_CLOCK
>>     root@at91sam9263cpc:~# time cyclictest -i 700000 -r -p 80 -l 33
>>     Clock resolution: 0.000000001 (s.ns)
>>     policy: fifo: loadavg: 0.24 0.08 0.03 1/46 600
>>
>>     T: 0 (  574) P:80 I:700000 C:     33 Min:    173 Act:  196 Avg:  222 Max:
>> 378
>>     real	0m 23.66s
>>     user	0m 0.32s
>>     sys	0m 1.29s
> 33 loops are not really giving you any useful information. Also run
> both tests with some background load and not on a fully idle system.
Sorry, my test setup is not very comprehensive at the moment. I have 
some overnight/overweekend results at hand with millions of loops:

     PREEMPT-DESKTOP:
     root@at91sam9263cpc:~# time cyclictest -i 70000 -p 80 -h 700 -r
     Clock resolution: 0.000000001 (s.ns)
     policy: fifo: loadavg: 0.07 0.09 0.02 1/38 8770
     T: 0 (  980) P:80 I:70000 C:1328241 Min:    135 Act:  218 Avg:  204 
Max:     720

     PREEMPT-RT:
     root@at91sam9263cpc:~# time cyclictest -i 70000 -p 80 -h 700 -r
     Clock resolution: 0.000000001 (s.ns)
     policy: fifo: loadavg: 0.97 1.12 1.18 4/51 31290
     T: 0 (  675) P:80 I:70000 C:3688312 Min:  244    Act:  358 Avg:  
395 Max:     732

Min and avg latencies much worse in PREEMPT-RT on otherwise same setup. 
Worst case similar though.

System is not loaded but many interrupts happen because it is running it 
interactively through ssh+ethernet. (Had to force ethernet to 10Mbit to 
avoid frequent buffer underruns due to sam9263 bug).

I am assuming context switching is more expensive on PREEMPT-RT under 
ARM9, where it seems already a bit expensive.

--Agustín.
-- 
[CG logo]

Agustín Ferrín Pozuelo

Embedded Systems Engineer

CG Power Systems Ireland Limited

Automation Systems Division

Herbert House, Harmony Row, Dublin 2, Ireland.

Phone: +353 1 4153700 Web: www.cgglobal.com <http://www.cgglobal.com>

Save the environment. Please print only if essential.



CG DISCLAIMER: This email contains confidential information. It is intended exclusively for the addressees. If you are not an addressee, you must not store, transmit or disclose its contents. Instead please notify the sender immediately; and permanently delete this e-mail from your computer systems. We have taken reasonable precautions to ensure that no viruses are present. However, you must check this email and the attachments, for viruses. We accept no liability whatsoever, for any detriment caused by any transmitted virus.

--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 2.6.33.7-rt29 PREEMPT_RT worse latency than PREEMPT_DESKTOP on AT91?

[Thomas Gleixner]

On Wed, 15 Sep 2010, Agustin Ferrin Pozuelo wrote:
>  On 27/08/10 11:35, Thomas Gleixner wrote:
> > On Mon, 23 Aug 2010, Agustin Ferrin Pozuelo wrote:
> Sorry, my test setup is not very comprehensive at the moment. I have some
> overnight/overweekend results at hand with millions of loops:
> 
>     PREEMPT-DESKTOP:
>     root@at91sam9263cpc:~# time cyclictest -i 70000 -p 80 -h 700 -r

Oh, you are running cyclictest with a signal based timer and relative
time. Any reason for this ?

What happens if you change the command line to

   cyclictest -i 70000 -p 80 -n

> I am assuming context switching is more expensive on PREEMPT-RT under ARM9,
> where it seems already a bit expensive.

No, the context switch is equally expensive, but signal delivery might
be a bit more overhead on -RT

Thanks,

	tglx

Re: 2.6.33.7-rt29 PREEMPT_RT worse latency than PREEMPT_DESKTOP on AT91?

[Agustin Ferrin Pozuelo]

  On 15/09/10 17:08, Thomas Gleixner wrote:
> On Wed, 15 Sep 2010, Agustin Ferrin Pozuelo wrote:
>>   On 27/08/10 11:35, Thomas Gleixner wrote:
>>> On Mon, 23 Aug 2010, Agustin Ferrin Pozuelo wrote:
>> Sorry, my test setup is not very comprehensive at the moment. I have some
>> overnight/overweekend results at hand with millions of loops:
>>
>>      PREEMPT-DESKTOP:
>>      root@at91sam9263cpc:~# time cyclictest -i 70000 -p 80 -h 700 -r
> Oh, you are running cyclictest with a signal based timer and relative
> time. Any reason for this ?
Not sure about the "signal based timer", is there an alternative to it? 
(clock_nanosleep?)

The relative time setting "-r" I put there to prevent issues with some 
specific configurations, specially when the interval was smaller than 
max latency, in which case cyclictest would freeze or give unusable 
results. That was happening often before using the workaround for the 
9263 ethernet bug which induced long latencies (non RT at all).

> What happens if you change the command line to
>
>     cyclictest -i 70000 -p 80 -n
>
Interesting, "-n" setting (use clock_nanosleep) brings down the min and 
avg latencies for 35~40 us. (~70 when using "-r") on the PREEMPT-DESKTOP 
configuration:

     root@at91sam9263cpc:~# cyclictest -i 70000 -p 80 -l 1000
     Clock resolution: 0.000000001 (s.ns)
     policy: fifo: loadavg: 0.14 0.13 0.06 1/36 954
     T: 0 (  876) P:80 I:70000 C:   1000 Min:    107 Act:  135 Avg:  164 
Max:     262

     root@at91sam9263cpc:~# cyclictest -i 70000 -p 80 -l 1000 -n
     Clock resolution: 0.000000001 (s.ns)
     policy: fifo: loadavg: 0.14 0.12 0.05 1/37 874
     T: 0 (  795) P:80 I:70000 C:   1000 Min:     72 Act:  127 Avg:  125 
Max:     275

     root@at91sam9263cpc:~# cyclictest -i 70000 -p 80 -l 1000 -r
     Clock resolution: 0.000000001 (s.ns)
     policy: fifo: loadavg: 0.04 0.10 0.06 1/36 1035
     T: 0 (  956) P:80 I:70000 C:   1000 Min:    145 Act:  205 Avg:  203 
Max:     348

     root@at91sam9263cpc:~# cyclictest -i 70000 -p 80 -l 1000 -r -n
     Clock resolution: 0.000000001 (s.ns)
     policy: fifo: loadavg: 0.01 0.07 0.05 2/37 1115
     T: 0 ( 1037) P:80 I:70000 C:   1000 Min:     87 Act:  135 Avg:  150 
Max:     439

(Longer runs giving similar results)
Note that"-r" seems to lower CPU usage, is loadavg 0.14 == 14% or 0.0014%?
Right now I don't have the PREEMPT-RT version available for this test, 
should I expect big improvement there?

>> I am assuming context switching is more expensive on PREEMPT-RT under ARM9,
>> where it seems already a bit expensive.
> No, the context switch is equally expensive, but signal delivery might
> be a bit more overhead on -RT
Thanks for the clarification and tips!
Regards,
--Agustín.

-- 
[CG logo]

Agustín Ferrín Pozuelo

Embedded Systems Engineer

CG Power Systems Ireland Limited

Automation Systems Division

Herbert House, Harmony Row, Dublin 2, Ireland.

Phone: +353 1 4153700 Web: www.cgglobal.com <http://www.cgglobal.com>

Save the environment. Please print only if essential.



CG DISCLAIMER: This email contains confidential information. It is intended exclusively for the addressees. If you are not an addressee, you must not store, transmit or disclose its contents. Instead please notify the sender immediately; and permanently delete this e-mail from your computer systems. We have taken reasonable precautions to ensure that no viruses are present. However, you must check this email and the attachments, for viruses. We accept no liability whatsoever, for any detriment caused by any transmitted virus.

--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html