[Xenomai-core] problem in pthread_mutex_lock/unlock

[Xenomai-core] problem in pthread_mutex_lock/unlock

[Nero Fernandez]

[-- Attachment #1.1: Type: text/plain, Size: 1523 bytes --]

Hi,

Please find an archive attached, containing :
 - a program for testing context-switch-latency using posix-APIs
   for native linux kernel and xenomai-posix-skin (userspace).
 - Makefile to build it using xenomai

In brief, the attempt is to spawn a bunch of pthreads, which share pairs
of read-write pthread-mutexes in a circular fashion, as shown below, and
measure the time taken for to complete a (rel_mtx_0 - acq_mtx_n) by
pthread_0
( - threads try to acquire read mutexes and release write_mutexes)

        pthread_0:write_mtx      =   pthread_1:read_mtx
        pthread_1:write_mtx      =   pthread_2:read_mtx
        ......
        pthread_(n-1):write_mtx  =   pthread_1:write_mtx
        pthread_n:write_mtx      =   pthread_0:read_mtx

The program runs fine on native linux userspace (ARM, x86) , but the
in the xenomai-space, threads lock up at trying to acquire their respective
read_mutexes.

A similar attempt using message-queues runs fine.

Am running a version between 2.5.2 and 2.5.3.
Will try to check with 2.5.3 as well.

Following is my current xeno-config
# ./xeno-config
xeno-config --verbose
        --version="2.5.2"
        --cc="arm-linux-gcc"
        --arch="arm"
        --prefix="/opt/xeno_utils"
        --xeno-cflags="-I/opt/xeno_utils/include -D_GNU_SOURCE -D_REENTRANT
-Wall -pipe -D__XENO__"
        --xeno-ldflags="-L/opt/xeno_utils/lib -lxenomai -lpthread "
        --posix-cflags=""
        --posix-ldflags="-Wl,--wrap,pthread_create
-Wl,--wrap,pthread_setschedparam -Wl,--wrap,"

[-- Attachment #1.2: Type: text/html, Size: 3591 bytes --]

[-- Attachment #2: context_switch_latency.tgz --]
[-- Type: application/x-gzip, Size: 9807 bytes --]

Re: [Xenomai-core] problem in pthread_mutex_lock/unlock

[Gilles Chanteperdrix]

Nero Fernandez wrote:
> Hi,
> 
> Please find an archive attached, containing :
>  - a program for testing context-switch-latency using posix-APIs
>    for native linux kernel and xenomai-posix-skin (userspace).
>  - Makefile to build it using xenomai

Your program is very long to tell fast. But it seems you are using the
mutex as if they were recursive. Xenomai posix skin mutexes used to be
recursive by default, but no longer are.

Also note that your code does not check the return value of the posix
skin services, which is a really bad idea.

-- 
					    Gilles.

[Xenomai-core] Fwd: problem in pthread_mutex_lock/unlock

[Nero Fernandez]

[-- Attachment #1: Type: text/plain, Size: 1546 bytes --]

On Fri, Jun 18, 2010 at 7:42 PM, Gilles Chanteperdrix <
gilles.chanteperdrix@xenomai.org> wrote:

> Nero Fernandez wrote:
> > Hi,
> >
> > Please find an archive attached, containing :
> >  - a program for testing context-switch-latency using posix-APIs
> >    for native linux kernel and xenomai-posix-skin (userspace).
> >  - Makefile to build it using xenomai
>
> Your program is very long to tell fast. But it seems you are using the
> mutex as if they were recursive. Xenomai posix skin mutexes used to be
> recursive by default, but no longer are.
>
> Also note that your code does not check the return value of the posix
> skin services, which is a really bad idea.
>
> --
>                                             Gilles.
>

Thanks for the prompt response.

Could you explain  'recursive usage of mutex' a little further?
Are the xenomai pthread-mutexes very different in behaviour than regular
posix mutexes?

The major portions of the code are the following three methods:
 - main()
      creates resource, spawns and collects pthreads
 - rt_task_master()
      calibrates the dry-run time, controls the mutex-passing and
      reports the final timing. Measuring scheme is similar to lmbench
 - rt_task_slave()
      task for passing the mutex lock()-unlock() chain over.

The program allocates same number of mutexes as the number of processes,
so ideally only one thread would be active and each would its read and write

locks in the following fashion:
 - unlock and lock , in case of master
 - lock and unlock, in case of slave

[-- Attachment #2: Type: text/html, Size: 3821 bytes --]

Re: [Xenomai-core] Fwd: problem in pthread_mutex_lock/unlock

[Gilles Chanteperdrix]

Nero Fernandez wrote:
> 
> On Fri, Jun 18, 2010 at 7:42 PM, Gilles Chanteperdrix
> <gilles.chanteperdrix@xenomai.org
> <mailto:gilles.chanteperdrix@xenomai.org>> wrote:
> 
>     Nero Fernandez wrote:
>     > Hi,
>     >
>     > Please find an archive attached, containing :
>     >  - a program for testing context-switch-latency using posix-APIs
>     >    for native linux kernel and xenomai-posix-skin (userspace).
>     >  - Makefile to build it using xenomai
> 
>     Your program is very long to tell fast. But it seems you are using the
>     mutex as if they were recursive. Xenomai posix skin mutexes used to be
>     recursive by default, but no longer are.
> 
>     Also note that your code does not check the return value of the posix
>     skin services, which is a really bad idea.
> 
>     --
>                                                Gilles.
> 
> 
> Thanks for the prompt response.
> 
> Could you explain  'recursive usage of mutex' a little further?
> Are the xenomai pthread-mutexes very different in behaviour than regular
> posix mutexes?

The posix specification does not define the default type of a mutex. So,
 in short, the behaviour of a "regular posix mutex" is unspecified.
However, following the principle of least surprise, Xenomai chose, like
Linux, to use the "normal" type by default.

What is the type of a posix mutex is explained in many places, starting
with Xenomai API documentation. So, no, I will not repeat it here.


-- 
					    Gilles.

Re: [Xenomai-core] Fwd: problem in pthread_mutex_lock/unlock

[Gilles Chanteperdrix]

Gilles Chanteperdrix wrote:
> Nero Fernandez wrote:
>> On Fri, Jun 18, 2010 at 7:42 PM, Gilles Chanteperdrix
>> <gilles.chanteperdrix@xenomai.org
>> <mailto:gilles.chanteperdrix@xenomai.org>> wrote:
>>
>>     Nero Fernandez wrote:
>>     > Hi,
>>     >
>>     > Please find an archive attached, containing :
>>     >  - a program for testing context-switch-latency using posix-APIs
>>     >    for native linux kernel and xenomai-posix-skin (userspace).
>>     >  - Makefile to build it using xenomai
>>
>>     Your program is very long to tell fast. But it seems you are using the
>>     mutex as if they were recursive. Xenomai posix skin mutexes used to be
>>     recursive by default, but no longer are.
>>
>>     Also note that your code does not check the return value of the posix
>>     skin services, which is a really bad idea.
>>
>>     --
>>                                                Gilles.
>>
>>
>> Thanks for the prompt response.
>>
>> Could you explain  'recursive usage of mutex' a little further?
>> Are the xenomai pthread-mutexes very different in behaviour than regular
>> posix mutexes?
> 
> The posix specification does not define the default type of a mutex. So,
>  in short, the behaviour of a "regular posix mutex" is unspecified.
> However, following the principle of least surprise, Xenomai chose, like
> Linux, to use the "normal" type by default.
> 
> What is the type of a posix mutex is explained in many places, starting
> with Xenomai API documentation. So, no, I will not repeat it here.

Actually, that is not your problem. However, you do not check the return
value of posix services, which is a bad idea. And indeed, if you check
it you will find your error: a thread which does not own a mutex tries
to unlock it.

Sorry, mutex are not semaphore, this is invalid, and Xenomai returns an
error in such a case.

-- 
					    Gilles.

Re: [Xenomai-core] Fwd: problem in pthread_mutex_lock/unlock

[Nero Fernandez]

[-- Attachment #1.1: Type: text/plain, Size: 2414 bytes --]

Thanks for your response, Gilles.

i modified the code to use semaphore instead of mutex, which worked fine.
Attached is a compilation of some latency figures and system loading figures
(using lmbench)
that i obtained from my proprietary ARM-9 board, using Xenomai-2.5.2.

Any comments are welcome. TIY.


On Sat, Jun 19, 2010 at 1:15 AM, Gilles Chanteperdrix <
gilles.chanteperdrix@xenomai.org> wrote:

> Gilles Chanteperdrix wrote:
> > Nero Fernandez wrote:
> >> On Fri, Jun 18, 2010 at 7:42 PM, Gilles Chanteperdrix
> >> <gilles.chanteperdrix@xenomai.org
> >> <mailto:gilles.chanteperdrix@xenomai.org>> wrote:
> >>
> >>     Nero Fernandez wrote:
> >>     > Hi,
> >>     >
> >>     > Please find an archive attached, containing :
> >>     >  - a program for testing context-switch-latency using posix-APIs
> >>     >    for native linux kernel and xenomai-posix-skin (userspace).
> >>     >  - Makefile to build it using xenomai
> >>
> >>     Your program is very long to tell fast. But it seems you are using
> the
> >>     mutex as if they were recursive. Xenomai posix skin mutexes used to
> be
> >>     recursive by default, but no longer are.
> >>
> >>     Also note that your code does not check the return value of the
> posix
> >>     skin services, which is a really bad idea.
> >>
> >>     --
> >>                                                Gilles.
> >>
> >>
> >> Thanks for the prompt response.
> >>
> >> Could you explain  'recursive usage of mutex' a little further?
> >> Are the xenomai pthread-mutexes very different in behaviour than regular
> >> posix mutexes?
> >
> > The posix specification does not define the default type of a mutex. So,
> >  in short, the behaviour of a "regular posix mutex" is unspecified.
> > However, following the principle of least surprise, Xenomai chose, like
> > Linux, to use the "normal" type by default.
> >
> > What is the type of a posix mutex is explained in many places, starting
> > with Xenomai API documentation. So, no, I will not repeat it here.
>
> Actually, that is not your problem. However, you do not check the return
> value of posix services, which is a bad idea. And indeed, if you check
> it you will find your error: a thread which does not own a mutex tries
> to unlock it.
>
> Sorry, mutex are not semaphore, this is invalid, and Xenomai returns an
> error in such a case.
>
> --
>                                             Gilles.
>

[-- Attachment #1.2: Type: text/html, Size: 3290 bytes --]

[-- Attachment #2: xenomai_analysis.xls --]
[-- Type: application/vnd.ms-excel, Size: 107008 bytes --]

Re: [Xenomai-core] Fwd: problem in pthread_mutex_lock/unlock

[Philippe Gerum]

On Thu, 2010-06-24 at 02:15 +0530, Nero Fernandez wrote:
> Thanks for your response, Gilles.
> 
> i modified the code to use semaphore instead of mutex, which worked
> fine.
> Attached is a compilation of some latency figures and system loading
> figures (using lmbench)
> that i obtained from my proprietary ARM-9 board, using Xenomai-2.5.2.
> 
> Any comments are welcome. TIY.
> 

Yikes. Let me sum up what I understood from your intent:

- you are measuring lmbench test latencies, that is to say, you don't
measure the real-time core capabilities at all. Unless you crafted a
Xenomai-linked version of lmbench, you are basically testing regular
processes.

- you are benchmarking your own port of the interrupt pipeline over some
random, outdated vendor kernel (2.6.18-based Mvista 5.0 dates back to
2007, right?), albeit the original ARM port of such code is based on
mainline since day #1. Since the latest latency-saving features like
FCSE are available with Adeos patches on recent kernels, you are likely
looking at ancient light rays from a fossile galaxy (btw, this may
explain the incorrect results in the 0k context switch test - you don't
have FCSE enabled in your Adeos port, right?).

- instead of reporting figures from a real-time interrupt handler
actually connected to the Xenomai core, you hijacked the system timer
core to pile up your instrumentation on top of the original code you
were supposed to benchmark. If this helps, run /usr/xenomai/bin/latency
-t2 and you will get the real figures.

Quoting you, from your document:
"The intent for running these tests is to gauge the overhead of running
interrupt-virtualization and further running a (real-time co-kernel + 
interrupt virtualization) on an embedded-device."

I'm unsure that you clearly identified the functional layers. If you
don't measure the Xenomai core based on Xenomai activities, then you
don't measure the co-kernel overhead. Besides, trying to measure the
interrupt pipeline overhead via the lmbench micro-benchmarks makes no
sense.

> 
> On Sat, Jun 19, 2010 at 1:15 AM, Gilles Chanteperdrix
> <gilles.chanteperdrix@xenomai.org> wrote:
>         
>         Gilles Chanteperdrix wrote:
>         > Nero Fernandez wrote:
>         >> On Fri, Jun 18, 2010 at 7:42 PM, Gilles Chanteperdrix
>         >> <gilles.chanteperdrix@xenomai.org
>         >> <mailto:gilles.chanteperdrix@xenomai.org>> wrote:
>         >>
>         >>     Nero Fernandez wrote:
>         >>     > Hi,
>         >>     >
>         >>     > Please find an archive attached, containing :
>         >>     >  - a program for testing context-switch-latency using
>         posix-APIs
>         >>     >    for native linux kernel and xenomai-posix-skin
>         (userspace).
>         >>     >  - Makefile to build it using xenomai
>         >>
>         >>     Your program is very long to tell fast. But it seems
>         you are using the
>         >>     mutex as if they were recursive. Xenomai posix skin
>         mutexes used to be
>         >>     recursive by default, but no longer are.
>         >>
>         >>     Also note that your code does not check the return
>         value of the posix
>         >>     skin services, which is a really bad idea.
>         >>
>         >>     --
>         >>                                                Gilles.
>         >>
>         >>
>         >> Thanks for the prompt response.
>         >>
>         >> Could you explain  'recursive usage of mutex' a little
>         further?
>         >> Are the xenomai pthread-mutexes very different in behaviour
>         than regular
>         >> posix mutexes?
>         >
>         > The posix specification does not define the default type of
>         a mutex. So,
>         >  in short, the behaviour of a "regular posix mutex" is
>         unspecified.
>         > However, following the principle of least surprise, Xenomai
>         chose, like
>         > Linux, to use the "normal" type by default.
>         >
>         > What is the type of a posix mutex is explained in many
>         places, starting
>         > with Xenomai API documentation. So, no, I will not repeat it
>         here.
>         
>         
>         Actually, that is not your problem. However, you do not check
>         the return
>         value of posix services, which is a bad idea. And indeed, if
>         you check
>         it you will find your error: a thread which does not own a
>         mutex tries
>         to unlock it.
>         
>         Sorry, mutex are not semaphore, this is invalid, and Xenomai
>         returns an
>         error in such a case.
>         
>         --
>                                                    Gilles.
> 
> _______________________________________________
> Xenomai-core mailing list
> Xenomai-core@domain.hid
> https://mail.gna.org/listinfo/xenomai-core


-- 
Philippe.

Re: [Xenomai-core] Fwd: problem in pthread_mutex_lock/unlock

[Nero Fernandez]

[-- Attachment #1: Type: text/plain, Size: 6833 bytes --]

Thanks for your response, Philippe.

The concerns while the carrying out my experiments were to:

 - compare xenomai co-kernel overheads (timer and context switch latencies)
   in xenomai-space vs similar native-linux overheads. These are presented
in
   the first two sheets.

 - find out, how addition of xenomai, xenomai+adeos effects the native
kernel's
   performance. Here, lmbench was used on the native linux side to estimate
   the changes to standard linux services.

Regarding the additions of latency measurements in sys-timer handler, i
performed
a similar measurement from xnintr_clock_handler(), and the results were
similar
to ones reported from sys-timer handler in xenomai-enabled linux. While
trying to
make both these measurements, i tried to take care that delay-value logging
is
done at the end the handler routines,but the __ipipe_mach_tsc value is
recorded
at the beginning of the routine (a patch for this is included in the
worksheet itself)

Regarding the system, changing the kernel version would invalidate my
results
as the system is a released CE device and has no plans to upgrade the
kernel.
AFAIK, enabling FCSE would limit the number of concurrent processes, hence
becoming inviable in my scenario.
As far as the adeos patch is concerned, i took a recent one (2.6.32) and
back-ported
it to 2.6.18, so as not to lose out on any new Adeos-only upgrades. i
carried out the
back-port activity for two platforms,a qemu-based integrator platform (for
minimal functional validity) and my proprietary board.

However, i am new to this field and would like to correct things if i went
wrong anywhere.
Your comments and guidance would be much appreciated.






On Thu, Jun 24, 2010 at 3:30 AM, Philippe Gerum <rpm@xenomai.org> wrote:

> On Thu, 2010-06-24 at 02:15 +0530, Nero Fernandez wrote:
> > Thanks for your response, Gilles.
> >
> > i modified the code to use semaphore instead of mutex, which worked
> > fine.
> > Attached is a compilation of some latency figures and system loading
> > figures (using lmbench)
> > that i obtained from my proprietary ARM-9 board, using Xenomai-2.5.2.
> >
> > Any comments are welcome. TIY.
> >
>
> Yikes. Let me sum up what I understood from your intent:
>
> - you are measuring lmbench test latencies, that is to say, you don't
> measure the real-time core capabilities at all. Unless you crafted a
> Xenomai-linked version of lmbench, you are basically testing regular
> processes.
>
> - you are benchmarking your own port of the interrupt pipeline over some
> random, outdated vendor kernel (2.6.18-based Mvista 5.0 dates back to
> 2007, right?), albeit the original ARM port of such code is based on
> mainline since day #1. Since the latest latency-saving features like
> FCSE are available with Adeos patches on recent kernels, you are likely
> looking at ancient light rays from a fossile galaxy (btw, this may
> explain the incorrect results in the 0k context switch test - you don't
> have FCSE enabled in your Adeos port, right?).
>
> - instead of reporting figures from a real-time interrupt handler
> actually connected to the Xenomai core, you hijacked the system timer
> core to pile up your instrumentation on top of the original code you
> were supposed to benchmark. If this helps, run /usr/xenomai/bin/latency
> -t2 and you will get the real figures.
>
> Quoting you, from your document:
> "The intent for running these tests is to gauge the overhead of running
> interrupt-virtualization and further running a (real-time co-kernel +
> interrupt virtualization) on an embedded-device."
>
> I'm unsure that you clearly identified the functional layers. If you
> don't measure the Xenomai core based on Xenomai activities, then you
> don't measure the co-kernel overhead. Besides, trying to measure the
> interrupt pipeline overhead via the lmbench micro-benchmarks makes no
> sense.
>
> >
> > On Sat, Jun 19, 2010 at 1:15 AM, Gilles Chanteperdrix
> > <gilles.chanteperdrix@xenomai.org> wrote:
> >
> >         Gilles Chanteperdrix wrote:
> >         > Nero Fernandez wrote:
> >         >> On Fri, Jun 18, 2010 at 7:42 PM, Gilles Chanteperdrix
> >         >> <gilles.chanteperdrix@xenomai.org
> >         >> <mailto:gilles.chanteperdrix@xenomai.org>> wrote:
> >         >>
> >         >>     Nero Fernandez wrote:
> >         >>     > Hi,
> >         >>     >
> >         >>     > Please find an archive attached, containing :
> >         >>     >  - a program for testing context-switch-latency using
> >         posix-APIs
> >         >>     >    for native linux kernel and xenomai-posix-skin
> >         (userspace).
> >         >>     >  - Makefile to build it using xenomai
> >         >>
> >         >>     Your program is very long to tell fast. But it seems
> >         you are using the
> >         >>     mutex as if they were recursive. Xenomai posix skin
> >         mutexes used to be
> >         >>     recursive by default, but no longer are.
> >         >>
> >         >>     Also note that your code does not check the return
> >         value of the posix
> >         >>     skin services, which is a really bad idea.
> >         >>
> >         >>     --
> >         >>                                                Gilles.
> >         >>
> >         >>
> >         >> Thanks for the prompt response.
> >         >>
> >         >> Could you explain  'recursive usage of mutex' a little
> >         further?
> >         >> Are the xenomai pthread-mutexes very different in behaviour
> >         than regular
> >         >> posix mutexes?
> >         >
> >         > The posix specification does not define the default type of
> >         a mutex. So,
> >         >  in short, the behaviour of a "regular posix mutex" is
> >         unspecified.
> >         > However, following the principle of least surprise, Xenomai
> >         chose, like
> >         > Linux, to use the "normal" type by default.
> >         >
> >         > What is the type of a posix mutex is explained in many
> >         places, starting
> >         > with Xenomai API documentation. So, no, I will not repeat it
> >         here.
> >
> >
> >         Actually, that is not your problem. However, you do not check
> >         the return
> >         value of posix services, which is a bad idea. And indeed, if
> >         you check
> >         it you will find your error: a thread which does not own a
> >         mutex tries
> >         to unlock it.
> >
> >         Sorry, mutex are not semaphore, this is invalid, and Xenomai
> >         returns an
> >         error in such a case.
> >
> >         --
> >                                                    Gilles.
> >
> > _______________________________________________
> > Xenomai-core mailing list
> > Xenomai-core@domain.hid
> > https://mail.gna.org/listinfo/xenomai-core
>
>
> --
> Philippe.
>
>
>

[-- Attachment #2: Type: text/html, Size: 8551 bytes --]

Re: [Xenomai-core] Fwd: problem in pthread_mutex_lock/unlock

[Gilles Chanteperdrix]

Nero Fernandez wrote:
> Thanks for your response, Philippe.
> 
> The concerns while the carrying out my experiments were to:
> 
>  - compare xenomai co-kernel overheads (timer and context switch latencies)
>    in xenomai-space vs similar native-linux overheads. These are
> presented in
>    the first two sheets.

On what ARM system do you get these latency figures? I really doubt the
linux kernel has a bounded latency under 35us. Because:
- the preempt_rt people, which work on getting a bounded latency get
something around 200us on AT91, an ARM9;
- there would be no reason of the preempt_rt effort if the linux kernel
interrupt latency was already bounded.

So, I take it that you do your measurement without generating a load. We
do our measurements using the latency test, while generating a load for
several hours. And on the average ARM, we usually get an interrupt
latency around 50us.

Please add some load on the system, and do the measurments again. The
best source of load we have found so far is to load the LTP testsuite
while running the latency test.

If you tell me what ARM SOC, or at least what ARM architecture revision
you use (the ARM920T core is an armv4, and the ARM926EJS is an armv5, so
ARM 9 does not tell us much), I can provide you with the root filesystem
we use for our tests.

-- 
					    Gilles.

Re: [Xenomai-core] Fwd: problem in pthread_mutex_lock/unlock

[Nero Fernandez]

[-- Attachment #1: Type: text/plain, Size: 2318 bytes --]

Yes, the measurements are on no-load scenarios.
I will try to repeat my measurements with system-loads as you suggest.

Following is the cpu-info of my board:
----------------------------------------------------------
Processor       : ARM926EJ-S rev 5 (v5l)
BogoMIPS        : 131.48
Features        : swp half fastmult edsp java
CPU implementer : 0x41
CPU architecture: 5TEJ
CPU variant     : 0x0
CPU part        : 0x926
CPU revision    : 5
Cache type      : write-back
Cache clean     : cp15 c7 ops
Cache lockdown  : format C
Cache format    : Harvard
I size          : 16384
I assoc         : 4
I line length   : 32
I sets          : 128
D size          : 16384
D assoc         : 4
D line length   : 32
D sets          : 128
----------------------------------------------------------

On Thu, Jun 24, 2010 at 5:20 PM, Gilles Chanteperdrix <
gilles.chanteperdrix@xenomai.org> wrote:

> Nero Fernandez wrote:
> > Thanks for your response, Philippe.
> >
> > The concerns while the carrying out my experiments were to:
> >
> >  - compare xenomai co-kernel overheads (timer and context switch
> latencies)
> >    in xenomai-space vs similar native-linux overheads. These are
> > presented in
> >    the first two sheets.
>
> On what ARM system do you get these latency figures? I really doubt the
> linux kernel has a bounded latency under 35us. Because:
> - the preempt_rt people, which work on getting a bounded latency get
> something around 200us on AT91, an ARM9;
> - there would be no reason of the preempt_rt effort if the linux kernel
> interrupt latency was already bounded.
>
> So, I take it that you do your measurement without generating a load. We
> do our measurements using the latency test, while generating a load for
> several hours. And on the average ARM, we usually get an interrupt
> latency around 50us.
>
> Please add some load on the system, and do the measurments again. The
> best source of load we have found so far is to load the LTP testsuite
> while running the latency test.
>
> If you tell me what ARM SOC, or at least what ARM architecture revision
> you use (the ARM920T core is an armv4, and the ARM926EJS is an armv5, so
> ARM 9 does not tell us much), I can provide you with the root filesystem
> we use for our tests.
>
> --
>                                             Gilles.
>

[-- Attachment #2: Type: text/html, Size: 5000 bytes --]

Re: [Xenomai-core] Fwd: problem in pthread_mutex_lock/unlock

[Gilles Chanteperdrix]

Nero Fernandez wrote:
> 
> Yes, the measurements are on no-load scenarios.
> I will try to repeat my measurements with system-loads as you suggest.

You can find a working root filesystem image with Xenomai 2.5.3 compiled
here:
http://www.xenomai.org/~gch/pub/rootfs-arm926-ejs.tar.bz2

The root password is empty, the system launches a telnet daemon, so you
can log on the board via telnet.

To run the tests, launch in a first telnet session:
echo 0 > /proc/xenomai/latency
latency -T 2 -H
in a second telnet sesssion, launch:
dohell
When you see "Listening on any address 5566", run on the host:
netcat <target-name-or-IP> 5566

where <target-name-or-IP> is the name of your arm board in the host
/etc/hosts file or its IP address.

Now, you can let the system run as long as the latency test prints
message. When the dohell script is done, it will kill the latency test,
which will cause it to print the histogram values and exit.

-- 
					    Gilles.

Re: [Xenomai-core] Fwd: problem in pthread_mutex_lock/unlock

[Nero Fernandez]

[-- Attachment #1: Type: text/plain, Size: 1423 bytes --]

Thanks for the rootfs, Gilles.
Although i am unable to use it directly (i get 'illegal instruction'
error while running any application/busybox-applet), i will try
to construct a similar test-setup for further testing.

On Thu, Jun 24, 2010 at 7:44 PM, Gilles Chanteperdrix <
gilles.chanteperdrix@xenomai.org> wrote:

> Nero Fernandez wrote:
> >
> > Yes, the measurements are on no-load scenarios.
> > I will try to repeat my measurements with system-loads as you suggest.
>
> You can find a working root filesystem image with Xenomai 2.5.3 compiled
> here:
> http://www.xenomai.org/~gch/pub/rootfs-arm926-ejs.tar.bz2<http://www.xenomai.org/%7Egch/pub/rootfs-arm926-ejs.tar.bz2>
>
> The root password is empty, the system launches a telnet daemon, so you
> can log on the board via telnet.
>
> To run the tests, launch in a first telnet session:
> echo 0 > /proc/xenomai/latency
> latency -T 2 -H
> in a second telnet sesssion, launch:
> dohell
> When you see "Listening on any address 5566", run on the host:
> netcat <target-name-or-IP> 5566
>
> where <target-name-or-IP> is the name of your arm board in the host
> /etc/hosts file or its IP address.
>
> Now, you can let the system run as long as the latency test prints
> message. When the dohell script is done, it will kill the latency test,
> which will cause it to print the histogram values and exit.
>
> --
>                                             Gilles.
>

[-- Attachment #2: Type: text/html, Size: 1927 bytes --]

Re: [Xenomai-core] Fwd: problem in pthread_mutex_lock/unlock

[Gilles Chanteperdrix]

Nero Fernandez wrote:
> Thanks for the rootfs, Gilles.
> Although i am unable to use it directly (i get 'illegal instruction'
> error while running any application/busybox-applet), i will try
> to construct a similar test-setup for further testing.

It is an EABI rootfs. Is EABI enabled in your kernel configuration?


-- 
					    Gilles.

Re: [Xenomai-core] Fwd: problem in pthread_mutex_lock/unlock

[Gilles Chanteperdrix]

Nero Fernandez wrote:
> As far as the adeos patch is concerned, i took a recent one (2.6.32) and
> back-ported
> it to 2.6.18, so as not to lose out on any new Adeos-only upgrades.

There is no such thing as an Adeos patch for linux 2.6.32 on the ARM
platforme.

-- 
					    Gilles.

Re: [Xenomai-core] co-kernel benchmarking on arm926 (was: Fwd: problem in pthread_mutex_lock/unlock)

[Philippe Gerum]

On Thu, 2010-06-24 at 17:05 +0530, Nero Fernandez wrote:
> Thanks for your response, Philippe.
> 
> The concerns while the carrying out my experiments were to:
> 
>  - compare xenomai co-kernel overheads (timer and context switch
> latencies)
>    in xenomai-space vs similar native-linux overheads. These are
> presented in 
>    the first two sheets.
> 
>  - find out, how addition of xenomai, xenomai+adeos effects the native
> kernel's 
>    performance. Here, lmbench was used on the native linux side to
> estimate
>    the changes to standard linux services.

How can your reasonably estimate the overhead of co-kernel services
without running any co-kernel services? Interrupt pipelining is not a
co-kernel service. You do nothing with interrupt pipelining except
enabling co-kernel services to be implemented with real-time response
guarantee.

> 
> Regarding the additions of latency measurements in sys-timer handler,
> i performed 
> a similar measurement from xnintr_clock_handler(), and the results
> were similar 
> to ones reported from sys-timer handler in xenomai-enabled linux.

If your benchmark is about Xenomai, then at least make sure to provide
results for Xenomai services, used in a relevant application and
platform context. Pretending that you instrumented
xnintr_clock_handler() at some point and got some results, but
eventually decided to illustrate your benchmark with other "similar"
results obtained from a totally unrelated instrumentation code, does not
help considering the figures as relevant.

Btw, hooking xnintr_clock_handler() is not correct. Again, benchmarking
interrupt latency with Xenomai has to measure the entire code path, from
the moment the interrupt is taken by the CPU, until it is delivered to
the Xenomai service user. By instrumenting directly in
xnintr_clock_handler(), your test bypasses the Xenomai timer handling
code which delivers the timer tick to the user code, and the
rescheduling procedure as well, so your figures are optimistically wrong
for any normal use case based on real-time tasks.

>  While trying to 
> make both these measurements, i tried to take care that delay-value
> logging is 
> done at the end the handler routines,but the __ipipe_mach_tsc value is
> recorded 
> at the beginning of the routine (a patch for this is included in the
> worksheet itself)

This patch is hopelessly useless and misleading. Unless your intent is
to have your application directly embodied into low-level interrupt
handlers, you are not measuring the actual overhead.

Latency is not solely a matter of interrupt masking, but also a matter
of I/D cache misses, particularly on ARM - you have to traverse the
actual code until delivery to exhibit the latter.

This is exactly what the latency tests shipped with Xenomai are for:
- /usr/xenomai/bin/latency -t0/1/2
- /usr/xenomai/bin/klatency
- /usr/xenomai/bin/irqbench

If your system involves user-space tasks, then you should benchmark
user-space response time using latency [-t0]. If you plan to use
kernel-based tasks such as RTDM tasks, then latency -t1 and klatency
tests will provide correct results for your benchmark.
If you are interested only in interrupt latency, then latency -t2 will
help.

If you do think that those tests do not measure what you seem to be
interested in, then you may want to explain why on this list, so that we
eventually understand what you are after.

> 
> Regarding the system, changing the kernel version would invalidate my
> results
> as the system is a released CE device and has no plans to upgrade the
> kernel.

Ok. But that makes your benchmark 100% irrelevant with respect to
assessing the real performances of a decent co-kernel on your setup.

> AFAIK, enabling FCSE would limit the number of concurrent processes,
> hence
> becoming inviable in my scenario.

Ditto. Besides, FCSE as implemented in recent I-pipe patches has a
best-effort mode which lifts those limitations, at the expense of
voiding the latency guarantee, but on the average, that would still be
much better than always suffering the VIVT cache insanity without FCSE.

Quoting a previous mail of yours, regarding your target:
> Processor       : ARM926EJ-S rev 5 (v5l)

The latency hit induced by VIVT caching on arm926 is typically in the
180-200 us range under load in user-space, and 100-120 us in kernel
space. So, without FCSE, this would bite at each Xenomai __and__ linux
process context switch. Since your application requires that more than
95 processes be available in the system, you will likely get quite a few
switches in any given period of time, unless most of them always sleep,
of course.

Ok, so let me do some wild guesses here: you told us this is a CE-based
application; maybe it exists already? maybe it has to be put on steroïds
for gaining decent real-time guarantees it doesn't have yet? and perhaps
the design of that application involves many processes undergoing
periodic activities, so lots of context switches with address space
changes during normal operations?

And, you want that to run on arm926, with no FCSE, and likely not a huge
amount of RAM either, with more than 95 different address spaces? Don't
you think there might be a problem? If so, don't you think implementing
a benchmark based on those assumptions might be irrelevant at some
point?

> As far as the adeos patch is concerned, i took a recent one (2.6.32)

I guess you meant 2.6.33?

>  and back-ported
> it to 2.6.18, so as not to lose out on any new Adeos-only upgrades. i
> carried out the 
> back-port activity for two platforms,a qemu-based integrator platform
> (for 
> minimal functional validity) and my proprietary board.
> 
> However, i am new to this field and would like to correct things if i
> went wrong anywhere.
> Your comments and guidance would be much appreciated.
> 

Since you told us only very few details, it's quite difficult to help.
AFAICS, the only advice that would make sense here, can be expressed as
a question for you: are you really, 100% sure that your app would fit on
that hardware, even without any real-time requirement?

> 
> 
> 
> 
> 
> 
> 
> 
> On Thu, Jun 24, 2010 at 3:30 AM, Philippe Gerum <rpm@xenomai.org>
> wrote:
>         On Thu, 2010-06-24 at 02:15 +0530, Nero Fernandez wrote:
>         > Thanks for your response, Gilles.
>         >
>         > i modified the code to use semaphore instead of mutex, which
>         worked
>         > fine.
>         > Attached is a compilation of some latency figures and system
>         loading
>         > figures (using lmbench)
>         > that i obtained from my proprietary ARM-9 board, using
>         Xenomai-2.5.2.
>         >
>         > Any comments are welcome. TIY.
>         >
>         
>         
>         Yikes. Let me sum up what I understood from your intent:
>         
>         - you are measuring lmbench test latencies, that is to say,
>         you don't
>         measure the real-time core capabilities at all. Unless you
>         crafted a
>         Xenomai-linked version of lmbench, you are basically testing
>         regular
>         processes.
>         
>         - you are benchmarking your own port of the interrupt pipeline
>         over some
>         random, outdated vendor kernel (2.6.18-based Mvista 5.0 dates
>         back to
>         2007, right?), albeit the original ARM port of such code is
>         based on
>         mainline since day #1. Since the latest latency-saving
>         features like
>         FCSE are available with Adeos patches on recent kernels, you
>         are likely
>         looking at ancient light rays from a fossile galaxy (btw, this
>         may
>         explain the incorrect results in the 0k context switch test -
>         you don't
>         have FCSE enabled in your Adeos port, right?).
>         
>         - instead of reporting figures from a real-time interrupt
>         handler
>         actually connected to the Xenomai core, you hijacked the
>         system timer
>         core to pile up your instrumentation on top of the original
>         code you
>         were supposed to benchmark. If this helps,
>         run /usr/xenomai/bin/latency
>         -t2 and you will get the real figures.
>         
>         Quoting you, from your document:
>         "The intent for running these tests is to gauge the overhead
>         of running
>         interrupt-virtualization and further running a (real-time
>         co-kernel +
>         interrupt virtualization) on an embedded-device."
>         
>         I'm unsure that you clearly identified the functional layers.
>         If you
>         don't measure the Xenomai core based on Xenomai activities,
>         then you
>         don't measure the co-kernel overhead. Besides, trying to
>         measure the
>         interrupt pipeline overhead via the lmbench micro-benchmarks
>         makes no
>         sense.
>         
>         
>         >
>         > On Sat, Jun 19, 2010 at 1:15 AM, Gilles Chanteperdrix
>         > <gilles.chanteperdrix@xenomai.org> wrote:
>         >
>         >         Gilles Chanteperdrix wrote:
>         >         > Nero Fernandez wrote:
>         >         >> On Fri, Jun 18, 2010 at 7:42 PM, Gilles
>         Chanteperdrix
>         >         >> <gilles.chanteperdrix@xenomai.org
>         >         >> <mailto:gilles.chanteperdrix@xenomai.org>> wrote:
>         >         >>
>         >         >>     Nero Fernandez wrote:
>         >         >>     > Hi,
>         >         >>     >
>         >         >>     > Please find an archive attached,
>         containing :
>         >         >>     >  - a program for testing
>         context-switch-latency using
>         >         posix-APIs
>         >         >>     >    for native linux kernel and
>         xenomai-posix-skin
>         >         (userspace).
>         >         >>     >  - Makefile to build it using xenomai
>         >         >>
>         >         >>     Your program is very long to tell fast. But
>         it seems
>         >         you are using the
>         >         >>     mutex as if they were recursive. Xenomai
>         posix skin
>         >         mutexes used to be
>         >         >>     recursive by default, but no longer are.
>         >         >>
>         >         >>     Also note that your code does not check the
>         return
>         >         value of the posix
>         >         >>     skin services, which is a really bad idea.
>         >         >>
>         >         >>     --
>         >         >>
>          Gilles.
>         >         >>
>         >         >>
>         >         >> Thanks for the prompt response.
>         >         >>
>         >         >> Could you explain  'recursive usage of mutex' a
>         little
>         >         further?
>         >         >> Are the xenomai pthread-mutexes very different in
>         behaviour
>         >         than regular
>         >         >> posix mutexes?
>         >         >
>         >         > The posix specification does not define the
>         default type of
>         >         a mutex. So,
>         >         >  in short, the behaviour of a "regular posix
>         mutex" is
>         >         unspecified.
>         >         > However, following the principle of least
>         surprise, Xenomai
>         >         chose, like
>         >         > Linux, to use the "normal" type by default.
>         >         >
>         >         > What is the type of a posix mutex is explained in
>         many
>         >         places, starting
>         >         > with Xenomai API documentation. So, no, I will not
>         repeat it
>         >         here.
>         >
>         >
>         >         Actually, that is not your problem. However, you do
>         not check
>         >         the return
>         >         value of posix services, which is a bad idea. And
>         indeed, if
>         >         you check
>         >         it you will find your error: a thread which does not
>         own a
>         >         mutex tries
>         >         to unlock it.
>         >
>         >         Sorry, mutex are not semaphore, this is invalid, and
>         Xenomai
>         >         returns an
>         >         error in such a case.
>         >
>         >         --
>         >                                                    Gilles.
>         >
>         
>         > _______________________________________________
>         > Xenomai-core mailing list
>         > Xenomai-core@domain.hid
>         > https://mail.gna.org/listinfo/xenomai-core
>         
>         
>         
>         --
>         Philippe.
>         
>         
> 


-- 
Philippe.

Re: [Xenomai-core] co-kernel benchmarking on arm926 (was: Fwd: problem in pthread_mutex_lock/unlock)

[Nero Fernandez]

[-- Attachment #1: Type: text/plain, Size: 14617 bytes --]

On Fri, Jun 25, 2010 at 8:30 PM, Philippe Gerum <rpm@xenomai.org> wrote:

> On Thu, 2010-06-24 at 17:05 +0530, Nero Fernandez wrote:
> > Thanks for your response, Philippe.
> >
> > The concerns while the carrying out my experiments were to:
> >
> >  - compare xenomai co-kernel overheads (timer and context switch
> > latencies)
> >    in xenomai-space vs similar native-linux overheads. These are
> > presented in
> >    the first two sheets.
> >
> >  - find out, how addition of xenomai, xenomai+adeos effects the native
> > kernel's
> >    performance. Here, lmbench was used on the native linux side to
> > estimate
> >    the changes to standard linux services.
>
> How can your reasonably estimate the overhead of co-kernel services
> without running any co-kernel services? Interrupt pipelining is not a
> co-kernel service. You do nothing with interrupt pipelining except
> enabling co-kernel services to be implemented with real-time response
> guarantee.
>

Repeating myself, sheet 1 and 2 contain the results of running
co-kernel services(real-time pthread, message-queues, semaphores
and clock-nansleep) and making measurment regarding scheduling
and timer-base functionality provided by co-kernel via posix skin.

Same code was then built native posix, instead of  xenomai-posix skin
and similar measurements were taken for linux-scheduler and timerbase.
This is something that i cant do with xenomai's native test (use it for
native linux benchmarking).
The point here is to demostrate what kind of benefits may be drawn using
 xenomai-space without any code change.



>
> > Regarding the additions of latency measurements in sys-timer handler,
> > i performed
> > a similar measurement from xnintr_clock_handler(), and the results
> > were similar
> > to ones reported from sys-timer handler in xenomai-enabled linux.
>
> If your benchmark is about Xenomai, then at least make sure to provide
> results for Xenomai services, used in a relevant application and
> platform context. Pretending that you instrumented
> xnintr_clock_handler() at some point and got some results, but
> eventually decided to illustrate your benchmark with other "similar"
> results obtained from a totally unrelated instrumentation code, does not
> help considering the figures as relevant.
>
> Btw, hooking xnintr_clock_handler() is not correct. Again, benchmarking
> interrupt latency with Xenomai has to measure the entire code path, from
> the moment the interrupt is taken by the CPU, until it is delivered to
> the Xenomai service user. By instrumenting directly in
> xnintr_clock_handler(), your test bypasses the Xenomai timer handling
> code which delivers the timer tick to the user code, and the
> rescheduling procedure as well, so your figures are optimistically wrong
> for any normal use case based on real-time tasks.
>

Regarding hooking up a measurement-device in sys-timer itself, it serves
the benefit of observing the changes that xenomai's aperiodic handling
of system-timer brings. This measurement does not attempt to measure
the co-kernel services in any manner.


>

>  While trying to
> > make both these measurements, i tried to take care that delay-value
> > logging is
> > done at the end the handler routines,but the __ipipe_mach_tsc value is
> > recorded
> > at the beginning of the routine (a patch for this is included in the
> > worksheet itself)
>
> This patch is hopelessly useless and misleading. Unless your intent is
> to have your application directly embodied into low-level interrupt
> handlers, you are not measuring the actual overhead.
>
> Latency is not solely a matter of interrupt masking, but also a matter
> of I/D cache misses, particularly on ARM - you have to traverse the
> actual code until delivery to exhibit the latter.
>
> This is exactly what the latency tests shipped with Xenomai are for:
> - /usr/xenomai/bin/latency -t0/1/2
> - /usr/xenomai/bin/klatency
> - /usr/xenomai/bin/irqbench
>
> If your system involves user-space tasks, then you should benchmark
> user-space response time using latency [-t0]. If you plan to use
> kernel-based tasks such as RTDM tasks, then latency -t1 and klatency
> tests will provide correct results for your benchmark.
> If you are interested only in interrupt latency, then latency -t2 will
> help.
>
> If you do think that those tests do not measure what you seem to be
> interested in, then you may want to explain why on this list, so that we
> eventually understand what you are after.
>
> >
> > Regarding the system, changing the kernel version would invalidate my
> > results
> > as the system is a released CE device and has no plans to upgrade the
> > kernel.
>
> Ok. But that makes your benchmark 100% irrelevant with respect to
> assessing the real performances of a decent co-kernel on your setup.
>
> > AFAIK, enabling FCSE would limit the number of concurrent processes,
> > hence
> > becoming inviable in my scenario.
>
> Ditto. Besides, FCSE as implemented in recent I-pipe patches has a
> best-effort mode which lifts those limitations, at the expense of
> voiding the latency guarantee, but on the average, that would still be
> much better than always suffering the VIVT cache insanity without FCSE
>

Thanks for mentioning this. I will try to enable this option for
re-measurements.


> Quoting a previous mail of yours, regarding your target:
> > Processor       : ARM926EJ-S rev 5 (v5l)
>
> The latency hit induced by VIVT caching on arm926 is typically in the
> 180-200 us range under load in user-space, and 100-120 us in kernel
> space. So, without FCSE, this would bite at each Xenomai __and__ linux
> process context switch. Since your application requires that more than
> 95 processes be available in the system, you will likely get quite a few
> switches in any given period of time, unless most of them always sleep,
> of course.
>
> Ok, so let me do some wild guesses here: you told us this is a CE-based
> application; maybe it exists already? maybe it has to be put on steroïds
> for gaining decent real-time guarantees it doesn't have yet? and perhaps
> the design of that application involves many processes undergoing
> periodic activities, so lots of context switches with address space
> changes during normal operations?
>
> And, you want that to run on arm926, with no FCSE, and likely not a huge
> amount of RAM either, with more than 95 different address spaces? Don't
> you think there might be a problem? If so, don't you think implementing
> a benchmark based on those assumptions might be irrelevant at some
> point?
>
> > As far as the adeos patch is concerned, i took a recent one (2.6.32)
>
> I guess you meant 2.6.33?
>

Correction, 2.6.30.


> >  and back-ported
> > it to 2.6.18, so as not to lose out on any new Adeos-only upgrades. i
> > carried out the
> > back-port activity for two platforms,a qemu-based integrator platform
> > (for
> > minimal functional validity) and my proprietary board.
> >
> > However, i am new to this field and would like to correct things if i
> > went wrong anywhere.0
> > Your comments and guidance would be much appreciated.
> >
>
> Since you told us only very few details, it's quite difficult to help.
> AFAICS, the only advice that would make sense here, can be expressed as
> a question for you: are you really, 100% sure that your app would fit on
> that hardware, even without any real-time requirement?
>
> >
> >
> >
> >
> >
> >
> >
> >
> > On Thu, Jun 24, 2010 at 3:30 AM, Philippe Gerum <rpm@xenomai.org>
> > wrote:
> >         On Thu, 2010-06-24 at 02:15 +0530, Nero Fernandez wrote:
> >         > Thanks for your response, Gilles.
> >         >
> >         > i modified the code to use semaphore instead of mutex, which
> >         worked
> >         > fine.
> >         > Attached is a compilation of some latency figures and system
> >         loading
> >         > figures (using lmbench)
> >         > that i obtained from my proprietary ARM-9 board, using
> >         Xenomai-2.5.2.
> >         >
> >         > Any comments are welcome. TIY.
> >         >
> >
> >
> >         Yikes. Let me sum up what I understood from your intent:
> >
> >         - you are measuring lmbench test latencies, that is to say,
> >         you don't
> >         measure the real-time core capabilities at all. Unless you
> >         crafted a
> >         Xenomai-linked version of lmbench, you are basically testing
> >         regular
> >         processes.
> >
> >         - you are benchmarking your own port of the interrupt pipeline
> >         over some
> >         random, outdated vendor kernel (2.6.18-based Mvista 5.0 dates
> >         back to
> >         2007, right?), albeit the original ARM port of such code is
> >         based on
> >         mainline since day #1. Since the latest latency-saving
> >         features like
> >         FCSE are available with Adeos patches on recent kernels, you
> >         are likely
> >         looking at ancient light rays from a fossile galaxy (btw, this
> >         may
> >         explain the incorrect results in the 0k context switch test -
> >         you don't
> >         have FCSE enabled in your Adeos port, right?).
> >
> >         - instead of reporting figures from a real-time interrupt
> >         handler
> >         actually connected to the Xenomai core, you hijacked the
> >         system timer
> >         core to pile up your instrumentation on top of the original
> >         code you
> >         were supposed to benchmark. If this helps,
> >         run /usr/xenomai/bin/latency
> >         -t2 and you will get the real figures.
> >
> >         Quoting you, from your document:
> >         "The intent for running these tests is to gauge the overhead
> >         of running
> >         interrupt-virtualization and further running a (real-time
> >         co-kernel +
> >         interrupt virtualization) on an embedded-device."
> >
> >         I'm unsure that you clearly identified the functional layers.
> >         If you
> >         don't measure the Xenomai core based on Xenomai activities,
> >         then you
> >         don't measure the co-kernel overhead. Besides, trying to
> >         measure the
> >         interrupt pipeline overhead via the lmbench micro-benchmarks
> >         makes no
> >         sense.
> >
> >
> >         >
> >         > On Sat, Jun 19, 2010 at 1:15 AM, Gilles Chanteperdrix
> >         > <gilles.chanteperdrix@xenomai.org> wrote:
> >         >
> >         >         Gilles Chanteperdrix wrote:
> >         >         > Nero Fernandez wrote:
> >         >         >> On Fri, Jun 18, 2010 at 7:42 PM, Gilles
> >         Chanteperdrix
> >         >         >> <gilles.chanteperdrix@xenomai.org
> >         >         >> <mailto:gilles.chanteperdrix@xenomai.org>> wrote:
> >         >         >>
> >         >         >>     Nero Fernandez wrote:
> >         >         >>     > Hi,
> >         >         >>     >
> >         >         >>     > Please find an archive attached,
> >         containing :
> >         >         >>     >  - a program for testing
> >         context-switch-latency using
> >         >         posix-APIs
> >         >         >>     >    for native linux kernel and
> >         xenomai-posix-skin
> >         >         (userspace).
> >         >         >>     >  - Makefile to build it using xenomai
> >         >         >>
> >         >         >>     Your program is very long to tell fast. But
> >         it seems
> >         >         you are using the
> >         >         >>     mutex as if they were recursive. Xenomai
> >         posix skin
> >         >         mutexes used to be
> >         >         >>     recursive by default, but no longer are.
> >         >         >>
> >         >         >>     Also note that your code does not check the
> >         return
> >         >         value of the posix
> >         >         >>     skin services, which is a really bad idea.
> >         >         >>
> >         >         >>     --
> >         >         >>
> >          Gilles.
> >         >         >>
> >         >         >>
> >         >         >> Thanks for the prompt response.
> >         >         >>
> >         >         >> Could you explain  'recursive usage of mutex' a
> >         little
> >         >         further?
> >         >         >> Are the xenomai pthread-mutexes very different in
> >         behaviour
> >         >         than regular
> >         >         >> posix mutexes?
> >         >         >
> >         >         > The posix specification does not define the
> >         default type of
> >         >         a mutex. So,
> >         >         >  in short, the behaviour of a "regular posix
> >         mutex" is
> >         >         unspecified.
> >         >         > However, following the principle of least
> >         surprise, Xenomai
> >         >         chose, like
> >         >         > Linux, to use the "normal" type by default.
> >         >         >
> >         >         > What is the type of a posix mutex is explained in
> >         many
> >         >         places, starting
> >         >         > with Xenomai API documentation. So, no, I will not
> >         repeat it
> >         >         here.
> >         >
> >         >
> >         >         Actually, that is not your problem. However, you do
> >         not check
> >         >         the return
> >         >         value of posix services, which is a bad idea. And
> >         indeed, if
> >         >         you check
> >         >         it you will find your error: a thread which does not
> >         own a
> >         >         mutex tries
> >         >         to unlock it.
> >         >
> >         >         Sorry, mutex are not semaphore, this is invalid, and
> >         Xenomai
> >         >         returns an
> >         >         error in such a case.
> >         >
> >         >         --
> >         >                                                    Gilles.
> >         >
> >
> >         > _______________________________________________
> >         > Xenomai-core mailing list
> >         > Xenomai-core@domain.hid
> >         > https://mail.gna.org/listinfo/xenomai-core
> >
> >
> >
> >         --
> >         Philippe.
> >
> >
> >
>
>
> --
> Philippe.
>
>
>

[-- Attachment #2: Type: text/html, Size: 18153 bytes --]

Re: [Xenomai-core] co-kernel benchmarking on arm926 (was: Fwd: problem in pthread_mutex_lock/unlock)

[Philippe Gerum]

On Mon, 2010-06-28 at 23:20 +0530, Nero Fernandez wrote:
> 
> 
> On Fri, Jun 25, 2010 at 8:30 PM, Philippe Gerum <rpm@xenomai.org>
> wrote:
>         On Thu, 2010-06-24 at 17:05 +0530, Nero Fernandez wrote:
>         > Thanks for your response, Philippe.
>         >
>         > The concerns while the carrying out my experiments were to:
>         >
>         >  - compare xenomai co-kernel overheads (timer and context
>         switch
>         > latencies)
>         >    in xenomai-space vs similar native-linux overheads. These
>         are
>         > presented in
>         >    the first two sheets.
>         >
>         >  - find out, how addition of xenomai, xenomai+adeos effects
>         the native
>         > kernel's
>         >    performance. Here, lmbench was used on the native linux
>         side to
>         > estimate
>         >    the changes to standard linux services.
>         
>         How can your reasonably estimate the overhead of co-kernel
>         services
>         without running any co-kernel services? Interrupt pipelining
>         is not a
>         co-kernel service. You do nothing with interrupt pipelining
>         except
>         enabling co-kernel services to be implemented with real-time
>         response
>         guarantee.
>  
> Repeating myself, sheet 1 and 2 contain the results of running 
> co-kernel services(real-time pthread, message-queues, semaphores 
> and clock-nansleep) and making measurment regarding scheduling 
> and timer-base functionality provided by co-kernel via posix skin.
> 

Ok, thanks for rehashing. But since you sent a series of latency
benchmarks on worst-case latencies done with no load on a sub-optimal
FCSE-less kernel, I just wanted to be sure that we were now 100% on the
same page regarding the protocol for testing those picky things. But
this rehashing still does not answer my concerns, actually:

> Same code was then built native posix, instead of  xenomai-posix skin
> and similar measurements were taken for linux-scheduler and timerbase.
> This is something that i cant do with xenomai's native test (use it
> for
> native linux benchmarking). 
> The point here is to demostrate what kind of benefits may be drawn
> using
>  xenomai-space without any code change.

Please re-read my post, you told us:

>  - find out, how addition of xenomai, xenomai+adeos effects the native
>         > kernel's
>         >    performance. Here, lmbench was used on the native linux
>         side to
>         > estimate
>         >    the changes to standard linux services.
>         

>From your description, you are trying to measure the overhead of the
interrupt pipeline activity on an some native lmbench load for assessing
the Xenomai core impact on your system. If you are not doing that, then
no issue.

>         
>         
> 
> 
>          
>         >
>         > Regarding the additions of latency measurements in sys-timer
>         handler,
>         > i performed
>         > a similar measurement from xnintr_clock_handler(), and the
>         results
>         > were similar
>         > to ones reported from sys-timer handler in xenomai-enabled
>         linux.
>         
>         If your benchmark is about Xenomai, then at least make sure to
>         provide
>         results for Xenomai services, used in a relevant application
>         and
>         platform context. Pretending that you instrumented
>         xnintr_clock_handler() at some point and got some results, but
>         eventually decided to illustrate your benchmark with other
>         "similar"
>         results obtained from a totally unrelated instrumentation
>         code, does not
>         help considering the figures as relevant.
>         
>         Btw, hooking xnintr_clock_handler() is not correct. Again,
>         benchmarking
>         interrupt latency with Xenomai has to measure the entire code
>         path, from
>         the moment the interrupt is taken by the CPU, until it is
>         delivered to
>         the Xenomai service user. By instrumenting directly in
>         xnintr_clock_handler(), your test bypasses the Xenomai timer
>         handling
>         code which delivers the timer tick to the user code, and the
>         rescheduling procedure as well, so your figures are
>         optimistically wrong
>         for any normal use case based on real-time tasks.
>  
> Regarding hooking up a measurement-device in sys-timer itself, it
> serves
> the benefit of observing the changes that xenomai's aperiodic handling
> of system-timer brings. This measurement does not attempt to measure 
> the co-kernel services in any manner.
>  

Your instrumentation code in the system timer handling seems to be about
measuring tick latencies, so if latency-related drifts in serving timers
are not "the changes" you want to observe this way, what changes do you
intend to observe, given that aperiodic tick management is 100%
Xenomai's business (see nucleus/timer.c)? 

>          
>         >  While trying to
>         > make both these measurements, i tried to take care that
>         delay-value
>         > logging is
>         > done at the end the handler routines,but the
>         __ipipe_mach_tsc value is
>         > recorded
>         > at the beginning of the routine (a patch for this is
>         included in the
>         > worksheet itself)
>         
>         This patch is hopelessly useless and misleading. Unless your
>         intent is
>         to have your application directly embodied into low-level
>         interrupt
>         handlers, you are not measuring the actual overhead.
>         
>         Latency is not solely a matter of interrupt masking, but also
>         a matter
>         of I/D cache misses, particularly on ARM - you have to
>         traverse the
>         actual code until delivery to exhibit the latter.
>         
>         This is exactly what the latency tests shipped with Xenomai
>         are for:
>         - /usr/xenomai/bin/latency -t0/1/2
>         - /usr/xenomai/bin/klatency
>         - /usr/xenomai/bin/irqbench
>         
>         If your system involves user-space tasks, then you should
>         benchmark
>         user-space response time using latency [-t0]. If you plan to
>         use
>         kernel-based tasks such as RTDM tasks, then latency -t1 and
>         klatency
>         tests will provide correct results for your benchmark.
>         If you are interested only in interrupt latency, then latency
>         -t2 will
>         help.
>         
>         If you do think that those tests do not measure what you seem
>         to be
>         interested in, then you may want to explain why on this list,
>         so that we
>         eventually understand what you are after.
>         
>         >
>         > Regarding the system, changing the kernel version would
>         invalidate my
>         > results
>         > as the system is a released CE device and has no plans to
>         upgrade the
>         > kernel.
>         
>         Ok. But that makes your benchmark 100% irrelevant with respect
>         to
>         assessing the real performances of a decent co-kernel on your
>         setup.
>         
>         > AFAIK, enabling FCSE would limit the number of concurrent
>         processes,
>         > hence
>         > becoming inviable in my scenario.
>         
>         Ditto. Besides, FCSE as implemented in recent I-pipe patches
>         has a
>         best-effort mode which lifts those limitations, at the expense
>         of
>         voiding the latency guarantee, but on the average, that would
>         still be
>         much better than always suffering the VIVT cache insanity
>         without FCSE
> 
> Thanks for mentioning this. I will try to enable this option for
> re-measurements.
>  
> 
>         Quoting a previous mail of yours, regarding your target:
>         > Processor       : ARM926EJ-S rev 5 (v5l)
>         
>         The latency hit induced by VIVT caching on arm926 is typically
>         in the
>         180-200 us range under load in user-space, and 100-120 us in
>         kernel
>         space. So, without FCSE, this would bite at each Xenomai
>         __and__ linux
>         process context switch. Since your application requires that
>         more than
>         95 processes be available in the system, you will likely get
>         quite a few
>         switches in any given period of time, unless most of them
>         always sleep,
>         of course.
>         
>         Ok, so let me do some wild guesses here: you told us this is a
>         CE-based
>         application; maybe it exists already? maybe it has to be put
>         on steroïds
>         for gaining decent real-time guarantees it doesn't have yet?
>         and perhaps
>         the design of that application involves many processes
>         undergoing
>         periodic activities, so lots of context switches with address
>         space
>         changes during normal operations?
>         
>         And, you want that to run on arm926, with no FCSE, and likely
>         not a huge
>         amount of RAM either, with more than 95 different address
>         spaces? Don't
>         you think there might be a problem? If so, don't you think
>         implementing
>         a benchmark based on those assumptions might be irrelevant at
>         some
>         point?
>         
>         > As far as the adeos patch is concerned, i took a recent one
>         (2.6.32)
>         
>         I guess you meant 2.6.33?
>  
> Correction, 2.6.30.
> 
> 
>         
>         >  and back-ported
>         > it to 2.6.18, so as not to lose out on any new Adeos-only
>         upgrades. i
>         > carried out the
>         > back-port activity for two platforms,a qemu-based integrator
>         platform
>         > (for
>         > minimal functional validity) and my proprietary board.
>         >
>         > However, i am new to this field and would like to correct
>         things if i
>         > went wrong anywhere.0
>         > Your comments and guidance would be much appreciated.
>         >
>         
>         Since you told us only very few details, it's quite difficult
>         to help.
>         AFAICS, the only advice that would make sense here, can be
>         expressed as
>         a question for you: are you really, 100% sure that your app
>         would fit on
>         that hardware, even without any real-time requirement?
>         
>         >
>         >
>         >
>         >
>         >
>         >
>         >
>         >
>         > On Thu, Jun 24, 2010 at 3:30 AM, Philippe Gerum
>         <rpm@xenomai.org>
>         > wrote:
>         >         On Thu, 2010-06-24 at 02:15 +0530, Nero Fernandez
>         wrote:
>         >         > Thanks for your response, Gilles.
>         >         >
>         >         > i modified the code to use semaphore instead of
>         mutex, which
>         >         worked
>         >         > fine.
>         >         > Attached is a compilation of some latency figures
>         and system
>         >         loading
>         >         > figures (using lmbench)
>         >         > that i obtained from my proprietary ARM-9 board,
>         using
>         >         Xenomai-2.5.2.
>         >         >
>         >         > Any comments are welcome. TIY.
>         >         >
>         >
>         >
>         >         Yikes. Let me sum up what I understood from your
>         intent:
>         >
>         >         - you are measuring lmbench test latencies, that is
>         to say,
>         >         you don't
>         >         measure the real-time core capabilities at all.
>         Unless you
>         >         crafted a
>         >         Xenomai-linked version of lmbench, you are basically
>         testing
>         >         regular
>         >         processes.
>         >
>         >         - you are benchmarking your own port of the
>         interrupt pipeline
>         >         over some
>         >         random, outdated vendor kernel (2.6.18-based Mvista
>         5.0 dates
>         >         back to
>         >         2007, right?), albeit the original ARM port of such
>         code is
>         >         based on
>         >         mainline since day #1. Since the latest
>         latency-saving
>         >         features like
>         >         FCSE are available with Adeos patches on recent
>         kernels, you
>         >         are likely
>         >         looking at ancient light rays from a fossile galaxy
>         (btw, this
>         >         may
>         >         explain the incorrect results in the 0k context
>         switch test -
>         >         you don't
>         >         have FCSE enabled in your Adeos port, right?).
>         >
>         >         - instead of reporting figures from a real-time
>         interrupt
>         >         handler
>         >         actually connected to the Xenomai core, you hijacked
>         the
>         >         system timer
>         >         core to pile up your instrumentation on top of the
>         original
>         >         code you
>         >         were supposed to benchmark. If this helps,
>         >         run /usr/xenomai/bin/latency
>         >         -t2 and you will get the real figures.
>         >
>         >         Quoting you, from your document:
>         >         "The intent for running these tests is to gauge the
>         overhead
>         >         of running
>         >         interrupt-virtualization and further running a
>         (real-time
>         >         co-kernel +
>         >         interrupt virtualization) on an embedded-device."
>         >
>         >         I'm unsure that you clearly identified the
>         functional layers.
>         >         If you
>         >         don't measure the Xenomai core based on Xenomai
>         activities,
>         >         then you
>         >         don't measure the co-kernel overhead. Besides,
>         trying to
>         >         measure the
>         >         interrupt pipeline overhead via the lmbench
>         micro-benchmarks
>         >         makes no
>         >         sense.
>         >
>         >
>         >         >
>         >         > On Sat, Jun 19, 2010 at 1:15 AM, Gilles
>         Chanteperdrix
>         >         > <gilles.chanteperdrix@xenomai.org> wrote:
>         >         >
>         >         >         Gilles Chanteperdrix wrote:
>         >         >         > Nero Fernandez wrote:
>         >         >         >> On Fri, Jun 18, 2010 at 7:42 PM, Gilles
>         >         Chanteperdrix
>         >         >         >> <gilles.chanteperdrix@xenomai.org
>         >         >         >>
>         <mailto:gilles.chanteperdrix@xenomai.org>> wrote:
>         >         >         >>
>         >         >         >>     Nero Fernandez wrote:
>         >         >         >>     > Hi,
>         >         >         >>     >
>         >         >         >>     > Please find an archive attached,
>         >         containing :
>         >         >         >>     >  - a program for testing
>         >         context-switch-latency using
>         >         >         posix-APIs
>         >         >         >>     >    for native linux kernel and
>         >         xenomai-posix-skin
>         >         >         (userspace).
>         >         >         >>     >  - Makefile to build it using
>         xenomai
>         >         >         >>
>         >         >         >>     Your program is very long to tell
>         fast. But
>         >         it seems
>         >         >         you are using the
>         >         >         >>     mutex as if they were recursive.
>         Xenomai
>         >         posix skin
>         >         >         mutexes used to be
>         >         >         >>     recursive by default, but no longer
>         are.
>         >         >         >>
>         >         >         >>     Also note that your code does not
>         check the
>         >         return
>         >         >         value of the posix
>         >         >         >>     skin services, which is a really
>         bad idea.
>         >         >         >>
>         >         >         >>     --
>         >         >         >>
>         >          Gilles.
>         >         >         >>
>         >         >         >>
>         >         >         >> Thanks for the prompt response.
>         >         >         >>
>         >         >         >> Could you explain  'recursive usage of
>         mutex' a
>         >         little
>         >         >         further?
>         >         >         >> Are the xenomai pthread-mutexes very
>         different in
>         >         behaviour
>         >         >         than regular
>         >         >         >> posix mutexes?
>         >         >         >
>         >         >         > The posix specification does not define
>         the
>         >         default type of
>         >         >         a mutex. So,
>         >         >         >  in short, the behaviour of a "regular
>         posix
>         >         mutex" is
>         >         >         unspecified.
>         >         >         > However, following the principle of
>         least
>         >         surprise, Xenomai
>         >         >         chose, like
>         >         >         > Linux, to use the "normal" type by
>         default.
>         >         >         >
>         >         >         > What is the type of a posix mutex is
>         explained in
>         >         many
>         >         >         places, starting
>         >         >         > with Xenomai API documentation. So, no,
>         I will not
>         >         repeat it
>         >         >         here.
>         >         >
>         >         >
>         >         >         Actually, that is not your problem.
>         However, you do
>         >         not check
>         >         >         the return
>         >         >         value of posix services, which is a bad
>         idea. And
>         >         indeed, if
>         >         >         you check
>         >         >         it you will find your error: a thread
>         which does not
>         >         own a
>         >         >         mutex tries
>         >         >         to unlock it.
>         >         >
>         >         >         Sorry, mutex are not semaphore, this is
>         invalid, and
>         >         Xenomai
>         >         >         returns an
>         >         >         error in such a case.
>         >         >
>         >         >         --
>         >         >
>          Gilles.
>         >         >
>         >
>         >         > _______________________________________________
>         >         > Xenomai-core mailing list
>         >         > Xenomai-core@domain.hid
>         >         > https://mail.gna.org/listinfo/xenomai-core
>         >
>         >
>         >
>         >         --
>         >         Philippe.
>         >
>         >
>         >
>         
>         
>         --
>         Philippe.
>         
>         
> 


-- 
Philippe.

Re: [Xenomai-core] co-kernel benchmarking on arm926

[Gilles Chanteperdrix]

Nero Fernandez wrote:
> On Fri, Jun 25, 2010 at 8:30 PM, Philippe Gerum <rpm@xenomai.org> wrote:
> 
>> On Thu, 2010-06-24 at 17:05 +0530, Nero Fernandez wrote:
>>> Thanks for your response, Philippe.
>>>
>>> The concerns while the carrying out my experiments were to:
>>>
>>>  - compare xenomai co-kernel overheads (timer and context switch
>>> latencies)
>>>    in xenomai-space vs similar native-linux overheads. These are
>>> presented in
>>>    the first two sheets.
>>>
>>>  - find out, how addition of xenomai, xenomai+adeos effects the native
>>> kernel's
>>>    performance. Here, lmbench was used on the native linux side to
>>> estimate
>>>    the changes to standard linux services.
>> How can your reasonably estimate the overhead of co-kernel services
>> without running any co-kernel services? Interrupt pipelining is not a
>> co-kernel service. You do nothing with interrupt pipelining except
>> enabling co-kernel services to be implemented with real-time response
>> guarantee.
>>
> 
> Repeating myself, sheet 1 and 2 contain the results of running
> co-kernel services(real-time pthread, message-queues, semaphores
> and clock-nansleep) and making measurment regarding scheduling
> and timer-base functionality provided by co-kernel via posix skin.
> 
> Same code was then built native posix, instead of  xenomai-posix skin
> and similar measurements were taken for linux-scheduler and timerbase.
> This is something that i cant do with xenomai's native test (use it for
> native linux benchmarking).
> The point here is to demostrate what kind of benefits may be drawn using
>  xenomai-space without any code change.
> 
> 
> 
>>> Regarding the additions of latency measurements in sys-timer handler,
>>> i performed
>>> a similar measurement from xnintr_clock_handler(), and the results
>>> were similar
>>> to ones reported from sys-timer handler in xenomai-enabled linux.
>> If your benchmark is about Xenomai, then at least make sure to provide
>> results for Xenomai services, used in a relevant application and
>> platform context. Pretending that you instrumented
>> xnintr_clock_handler() at some point and got some results, but
>> eventually decided to illustrate your benchmark with other "similar"
>> results obtained from a totally unrelated instrumentation code, does not
>> help considering the figures as relevant.
>>
>> Btw, hooking xnintr_clock_handler() is not correct. Again, benchmarking
>> interrupt latency with Xenomai has to measure the entire code path, from
>> the moment the interrupt is taken by the CPU, until it is delivered to
>> the Xenomai service user. By instrumenting directly in
>> xnintr_clock_handler(), your test bypasses the Xenomai timer handling
>> code which delivers the timer tick to the user code, and the
>> rescheduling procedure as well, so your figures are optimistically wrong
>> for any normal use case based on real-time tasks.
>>
> 
> Regarding hooking up a measurement-device in sys-timer itself, it serves
> the benefit of observing the changes that xenomai's aperiodic handling
> of system-timer brings. This measurement does not attempt to measure
> the co-kernel services in any manner.
> 
> 
> 
>>  While trying to
>>> make both these measurements, i tried to take care that delay-value
>>> logging is
>>> done at the end the handler routines,but the __ipipe_mach_tsc value is
>>> recorded
>>> at the beginning of the routine (a patch for this is included in the
>>> worksheet itself)
>> This patch is hopelessly useless and misleading. Unless your intent is
>> to have your application directly embodied into low-level interrupt
>> handlers, you are not measuring the actual overhead.
>>
>> Latency is not solely a matter of interrupt masking, but also a matter
>> of I/D cache misses, particularly on ARM - you have to traverse the
>> actual code until delivery to exhibit the latter.
>>
>> This is exactly what the latency tests shipped with Xenomai are for:
>> - /usr/xenomai/bin/latency -t0/1/2
>> - /usr/xenomai/bin/klatency
>> - /usr/xenomai/bin/irqbench
>>
>> If your system involves user-space tasks, then you should benchmark
>> user-space response time using latency [-t0]. If you plan to use
>> kernel-based tasks such as RTDM tasks, then latency -t1 and klatency
>> tests will provide correct results for your benchmark.
>> If you are interested only in interrupt latency, then latency -t2 will
>> help.
>>
>> If you do think that those tests do not measure what you seem to be
>> interested in, then you may want to explain why on this list, so that we
>> eventually understand what you are after.
>>
>>> Regarding the system, changing the kernel version would invalidate my
>>> results
>>> as the system is a released CE device and has no plans to upgrade the
>>> kernel.
>> Ok. But that makes your benchmark 100% irrelevant with respect to
>> assessing the real performances of a decent co-kernel on your setup.
>>
>>> AFAIK, enabling FCSE would limit the number of concurrent processes,
>>> hence
>>> becoming inviable in my scenario.
>> Ditto. Besides, FCSE as implemented in recent I-pipe patches has a
>> best-effort mode which lifts those limitations, at the expense of
>> voiding the latency guarantee, but on the average, that would still be
>> much better than always suffering the VIVT cache insanity without FCSE
>>
> 
> Thanks for mentioning this. I will try to enable this option for
> re-measurements.
> 
> 
>> Quoting a previous mail of yours, regarding your target:
>>> Processor       : ARM926EJ-S rev 5 (v5l)
>> The latency hit induced by VIVT caching on arm926 is typically in the
>> 180-200 us range under load in user-space, and 100-120 us in kernel
>> space. So, without FCSE, this would bite at each Xenomai __and__ linux
>> process context switch. Since your application requires that more than
>> 95 processes be available in the system, you will likely get quite a few
>> switches in any given period of time, unless most of them always sleep,
>> of course.
>>
>> Ok, so let me do some wild guesses here: you told us this is a CE-based
>> application; maybe it exists already? maybe it has to be put on steroïds
>> for gaining decent real-time guarantees it doesn't have yet? and perhaps
>> the design of that application involves many processes undergoing
>> periodic activities, so lots of context switches with address space
>> changes during normal operations?
>>
>> And, you want that to run on arm926, with no FCSE, and likely not a huge
>> amount of RAM either, with more than 95 different address spaces? Don't
>> you think there might be a problem? If so, don't you think implementing
>> a benchmark based on those assumptions might be irrelevant at some
>> point?
>>
>>> As far as the adeos patch is concerned, i took a recent one (2.6.32)
>> I guess you meant 2.6.33?
>>
> 
> Correction, 2.6.30.

Ok. If you are interested in the FCSE code, you may want to use FCSE v4.
See the comparison on the hackbench test here:
http://sisyphus.hd.free.fr/~gilles/pub/fcse/hackbench-fcse-v4.png

I did not rebase the I-pipe patch for 2.6.30 on this new fcse, but you
can find it in the patches for 2.6.31 and 2.6.33. Or as standalone trees
in my adeos git tree:
http://git.xenomai.org/?p=ipipe-gch.git;a=summary

Also note that since we are in the re-hashing tonight, as Philippe told
you, 95 processes is actually a lot on a low-end ARM platform, so you
would better be sure that you really need more than 95 processes
(beware, we are talking processes here, memory spaces, not threads, a
process may have has many threads as it wants) before deciding not to
use the FCSE guaranteed mode. Thinking that the number of processes is
unlimited on a low-end/embedded ARM system is an error: it is limited by
the available ressources (RAM, CPU) on your system. The lower the
ressources, the lower the practical limit is, and I bet this practical
limit is much lower than you would like.

-- 
					    Gilles.