[Xenomai] Xenomai and normal linux application - Wait time with select() not reliable

[Xenomai] Xenomai and normal linux application - Wait time with select() not reliable

[Jens Köhler]

Hello all,

I have a normal linux application running. Application waits in
following loop normally about 8 ms. Delay time of 8ms is reliable.

do
   {
      remain(&remainTime);

      if ((remainTime.tv_sec != 0L) || (remainTime.tv_usec != 0L))
         result = select(0, NULL, NULL, NULL, &remainTime);
      else
         break;
   }
   while ((result == -1) && (errno == EINTR));

If there is running in parallel a Xenomai application the wait time
could be up to 40 ms accidentally. What could be the reason?

Thanks and regards,
Jens

Re: [Xenomai] Xenomai and normal linux application - Wait time with select() not reliable

[Gilles Chanteperdrix]

On 07/03/2012 03:30 PM, Jens Köhler wrote:
> Hello all,
> 
> I have a normal linux application running. Application waits in
> following loop normally about 8 ms. Delay time of 8ms is reliable.
> 
> do
>    {
>       remain(&remainTime);
> 
>       if ((remainTime.tv_sec != 0L) || (remainTime.tv_usec != 0L))
>          result = select(0, NULL, NULL, NULL, &remainTime);
>       else
>          break;
>    }
>    while ((result == -1) && (errno == EINTR));
> 
> If there is running in parallel a Xenomai application the wait time
> could be up to 40 ms accidentally. What could be the reason?

Are you sure the "select" service guarantees delays with the version of
the linux kernel you use? Is the thread running with SCHED_FIFO policy?
If yes, what happens if you replace with nanosleep?

-- 
                                                                Gilles.

Re: [Xenomai] Xenomai and normal linux application - Wait time with select() not reliable

[Jens Köhler]

Thank you for your response Gilles. I set thread to SCHED_FIFO policy
with chrt command, Prio 99. But there is no improvement. Also
nanosleep do not help. The kernel is 2.6.26. What could be reason that
kernel is not restart thread when time is over?

Jens

2012/7/3, Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>:
> On 07/03/2012 03:30 PM, Jens Köhler wrote:
>> Hello all,
>>
>> I have a normal linux application running. Application waits in
>> following loop normally about 8 ms. Delay time of 8ms is reliable.
>>
>> do
>>    {
>>       remain(&remainTime);
>>
>>       if ((remainTime.tv_sec != 0L) || (remainTime.tv_usec != 0L))
>>          result = select(0, NULL, NULL, NULL, &remainTime);
>>       else
>>          break;
>>    }
>>    while ((result == -1) && (errno == EINTR));
>>
>> If there is running in parallel a Xenomai application the wait time
>> could be up to 40 ms accidentally. What could be the reason?
>
> Are you sure the "select" service guarantees delays with the version of
> the linux kernel you use? Is the thread running with SCHED_FIFO policy?
> If yes, what happens if you replace with nanosleep?
>
> --
>                                                                 Gilles.
>
>

Re: [Xenomai] Xenomai and normal linux application - Wait time with select() not reliable

[Gilles Chanteperdrix]

On 07/04/2012 10:36 AM, Jens Köhler wrote:
> Thank you for your response Gilles. I set thread to SCHED_FIFO policy
> with chrt command, Prio 99. But there is no improvement. Also
> nanosleep do not help. The kernel is 2.6.26. What could be reason that
> kernel is not restart thread when time is over?

It could be that the xenomai activity, if any, is running for long
periods of time and causing linux ticks to be delayed. As usual, it
would help if you could provide a self-contained test-case allowing us
to reproduce the issue.

-- 
					    Gilles.

Re: [Xenomai] Xenomai and normal linux application - Wait time with select() not reliable

[Philippe Gerum]

On 07/04/2012 10:45 AM, Gilles Chanteperdrix wrote:
> On 07/04/2012 10:36 AM, Jens Köhler wrote:
>> Thank you for your response Gilles. I set thread to SCHED_FIFO policy
>> with chrt command, Prio 99. But there is no improvement. Also
>> nanosleep do not help. The kernel is 2.6.26. What could be reason that
>> kernel is not restart thread when time is over?
>
> It could be that the xenomai activity, if any, is running for long
> periods of time and causing linux ticks to be delayed. As usual, it
> would help if you could provide a self-contained test-case allowing us
> to reproduce the issue.
>

/proc/xenomai/stat as well, in case the rt activity changes mode a lot.

-- 
Philippe.

Re: [Xenomai] Xenomai and normal linux application - Wait time with select() not reliable

[Jens Köhler]

The Xenomai application runs every 2ms for about 150 us and sends/
receives an udp packet.Rtnet socket is used for udp communication.

/proc/xenomai/stat:

CPU	PID	MSW	CSW	PF	STAT		%CPU  NAME
  0	0	0	272362	0	00500080	93.8  ROOT
  0	0	0	13475	0	00000082	0.0  rtnet-stack
  0	0	0	1	0	00000082	0.0  rtnet-rtpc
  0	1843	1	11677	0	00300182	0.1  SgbmReceive
  0	1844	1	67501	0	00300184	4.7  SgbmSend
  0	1845	63111	63304	0	00300380	0.3  SgbmUserSpaceComm
  0	1846	1	134963	0	00300184	0.5  SgbmTimer
  0	0	0	67500	0	00000000	0.3  IRQ19: rteth0
  0	0	0	298425	0	00000000	0.3  IRQ233: [timer]

Application was ported from RTAI to Xenomai. With Rtai was never a
problem with select.

Jens


2012/7/4, Philippe Gerum <rpm@xenomai.org>:
> On 07/04/2012 10:45 AM, Gilles Chanteperdrix wrote:
>> On 07/04/2012 10:36 AM, Jens Köhler wrote:
>>> Thank you for your response Gilles. I set thread to SCHED_FIFO policy
>>> with chrt command, Prio 99. But there is no improvement. Also
>>> nanosleep do not help. The kernel is 2.6.26. What could be reason that
>>> kernel is not restart thread when time is over?
>>
>> It could be that the xenomai activity, if any, is running for long
>> periods of time and causing linux ticks to be delayed. As usual, it
>> would help if you could provide a self-contained test-case allowing us
>> to reproduce the issue.
>>
>
> /proc/xenomai/stat as well, in case the rt activity changes mode a lot.
>
> --
> Philippe.
>
>
>

Re: [Xenomai] Xenomai and normal linux application - Wait time with select() not reliable

[Gilles Chanteperdrix]

On 07/04/2012 01:45 PM, Jens Köhler wrote:
> The Xenomai application runs every 2ms for about 150 us and sends/
> receives an udp packet.Rtnet socket is used for udp communication.
> 
> /proc/xenomai/stat:
> 
> CPU	PID	MSW	CSW	PF	STAT		%CPU  NAME
>   0	0	0	272362	0	00500080	93.8  ROOT
>   0	0	0	13475	0	00000082	0.0  rtnet-stack
>   0	0	0	1	0	00000082	0.0  rtnet-rtpc
>   0	1843	1	11677	0	00300182	0.1  SgbmReceive
>   0	1844	1	67501	0	00300184	4.7  SgbmSend
>   0	1845	63111	63304	0	00300380	0.3  SgbmUserSpaceComm
>   0	1846	1	134963	0	00300184	0.5  SgbmTimer
>   0	0	0	67500	0	00000000	0.3  IRQ19: rteth0
>   0	0	0	298425	0	00000000	0.3  IRQ233: [timer]
> 
> Application was ported from RTAI to Xenomai. With Rtai was never a
> problem with select.

What version of linux, the I-pipe patch, and xenomai are you using ?

-- 
					    Gilles.

Re: [Xenomai] Xenomai and normal linux application - Wait time with select() not reliable

[Gilles Chanteperdrix]

On 07/04/2012 02:49 PM, Jens Köhler wrote:
> Linux: 2.6.26.
> Xenomai:	2.4.10
> adeos-i-pipe: 2.6.26.7

Last question: are you compiling for the posix skin?

-- 
					    Gilles.

Re: [Xenomai] Xenomai and normal linux application - Wait time with select() not reliable

[Jens Köhler]

Linux: 2.6.26.
Xenomai:        2.4.10
adeos-i-pipe: 2.6.26.7

Jens

Re: [Xenomai] Xenomai and normal linux application - Wait time with select() not reliable

[Jens Köhler]

No posix skin

Jens

2012/7/4, Jens Köhler <jenskxxx@googlemail.com>:
> Linux: 2.6.26.
> Xenomai:        2.4.10
> adeos-i-pipe: 2.6.26.7
>
> Jens
>

Re: [Xenomai] Xenomai and normal linux application - Wait time with select() not reliable

[Gilles Chanteperdrix]

On 07/04/2012 03:10 PM, Jens Köhler wrote:
> Linux: 2.6.26.
> Xenomai:        2.4.10
> adeos-i-pipe: 2.6.26.7

This is not the first time you ask this question, last time Jan asked
you to try a more recent version, this is still true.

But before you do that, I suggest you try to reduce the problem to its
simplest expression. For instance trying to run the latency test
concurrently with the select loop, measuring the select loop latency. If
the bug happens to trigger in these conditions, then you do not need to
port all the needed drivers on xenomai latest release to test it.

-- 
					    Gilles.

Re: [Xenomai] Xenomai and normal linux application - Wait time with select() not reliable

[Philippe Gerum]

On 07/04/2012 01:45 PM, Jens Köhler wrote:
> The Xenomai application runs every 2ms for about 150 us and sends/
> receives an udp packet.Rtnet socket is used for udp communication.
>
> /proc/xenomai/stat:
>
> CPU	PID	MSW	CSW	PF	STAT		%CPU  NAME
>    0	0	0	272362	0	00500080	93.8  ROOT
>    0	0	0	13475	0	00000082	0.0  rtnet-stack
>    0	0	0	1	0	00000082	0.0  rtnet-rtpc
>    0	1843	1	11677	0	00300182	0.1  SgbmReceive
>    0	1844	1	67501	0	00300184	4.7  SgbmSend
>    0	1845	63111	63304	0	00300380	0.3  SgbmUserSpaceComm

This thread seems to be switching modes a lot. Does it do this 
frequently? Each mode switch involves a task switch linux-wise.

>    0	1846	1	134963	0	00300184	0.5  SgbmTimer
>    0	0	0	67500	0	00000000	0.3  IRQ19: rteth0
>    0	0	0	298425	0	00000000	0.3  IRQ233: [timer]
>
> Application was ported from RTAI to Xenomai. With Rtai was never a
> problem with select.

Is it supposed to give us any hint?

>
> Jens
>
>
> 2012/7/4, Philippe Gerum <rpm@xenomai.org>:
>> On 07/04/2012 10:45 AM, Gilles Chanteperdrix wrote:
>>> On 07/04/2012 10:36 AM, Jens Köhler wrote:
>>>> Thank you for your response Gilles. I set thread to SCHED_FIFO policy
>>>> with chrt command, Prio 99. But there is no improvement. Also
>>>> nanosleep do not help. The kernel is 2.6.26. What could be reason that
>>>> kernel is not restart thread when time is over?
>>>
>>> It could be that the xenomai activity, if any, is running for long
>>> periods of time and causing linux ticks to be delayed. As usual, it
>>> would help if you could provide a self-contained test-case allowing us
>>> to reproduce the issue.
>>>
>>
>> /proc/xenomai/stat as well, in case the rt activity changes mode a lot.
>>
>> --
>> Philippe.
>>
>>
>>
>


-- 
Philippe.

Re: [Xenomai] Xenomai and normal linux application - Wait time with select() not reliable

[Jens Köhler]

Hello,

Now is Xenomai 2.6 installed. But I have sometimes still delays in
user space applications.

Delay: 0.003064
Delay: 0.003066
Delay: 0.003504
Delay: 0.003066
Delay: 0.003066
Delay: 0.003066
Delay: 0.003090
Delay: 0.003064
Delay: 0.003066
Delay: 0.003067
Delay: 0.053807 <- 53 ms instead 3ms
Delay: 0.003067
Delay: 0.003067

I found out that problem depends from a real time queue between two
Xenomai tasks.

retval = rt_queue_create(&RtMeasureData, "RtMeasureData", 1000 * 256,
Q_UNLIMITED, Q_FIFO | Q_SHARED);

A real time task writes measurement data every 2 ms to a rt_queue.
retval = rt_queue_write(&RtMeasureData, &FifoMeasureData, 256, Q_NORMAL);

Another Xenomai task reads all measured data from queue and makes them
available for user land applications. (Task is not real time, context
switches)

retval = rt_queue_read(&RtMeasureData, &FifoMeasureData, 256, TM_NONBLOCK);

When nothing is written to queue is there also never an unexpected delay.

Any ideas or recommendations to find reason for problem?

Jens

Re: [Xenomai] Xenomai and normal linux application - Wait time with select() not reliable

[Gilles Chanteperdrix]

On 11/19/2012 02:29 PM, Jens Köhler wrote:
> Hello,
> 
> Now is Xenomai 2.6 installed. But I have sometimes still delays in
> user space applications.
> 
> Delay: 0.003064
> Delay: 0.003066
> Delay: 0.003504
> Delay: 0.003066
> Delay: 0.003066
> Delay: 0.003066
> Delay: 0.003090
> Delay: 0.003064
> Delay: 0.003066
> Delay: 0.003067
> Delay: 0.053807 <- 53 ms instead 3ms
> Delay: 0.003067
> Delay: 0.003067
> 
> I found out that problem depends from a real time queue between two
> Xenomai tasks.
> 
> retval = rt_queue_create(&RtMeasureData, "RtMeasureData", 1000 * 256,
> Q_UNLIMITED, Q_FIFO | Q_SHARED);
> 
> A real time task writes measurement data every 2 ms to a rt_queue.
> retval = rt_queue_write(&RtMeasureData, &FifoMeasureData, 256, Q_NORMAL);
> 
> Another Xenomai task reads all measured data from queue and makes them
> available for user land applications. (Task is not real time, context
> switches)
> 
> retval = rt_queue_read(&RtMeasureData, &FifoMeasureData, 256, TM_NONBLOCK);
> 
> When nothing is written to queue is there also never an unexpected delay.
> 
> Any ideas or recommendations to find reason for problem?

Without seeing any code, there is strictly nothing we can do.
If you are able to reduce the code to a simple, self-contained test case
we can run anywhere, then we can have a look at it.
If not, then use the I-pipe tracer.

-- 
					    Gilles.

Re: [Xenomai] Xenomai and normal linux application - Wait time with select() not reliable

[Jens Köhler]

Hello,

I found following additionaly out:
One of the Xenomai tasks writes cyclically to a shared linux memory
(its not a Xenomai shared memory). Write and read access is controlled
by Linux semaphores. If shared memory writing is out-commented is
there never a delay problem.

I found following explanation about Xenomai in internet. Could it
reason for problem?

"If an interrupt arrives while a Xenomai thread executes in secondary
mode, it is not forwarded to Linux! The mechanism works as follows:
When a Xenomai thread is executed in secondary mode, the interrupt
shield domain is activated All Linux interrupts are stalled, until the
Xenomai thread completes execution. The interrupts will be served
again when Linux goes back to execute in background mode."

Best regards,
Jens


2012/11/19, Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>:
> On 11/19/2012 02:29 PM, Jens Köhler wrote:
>> Hello,
>>
>> Now is Xenomai 2.6 installed. But I have sometimes still delays in
>> user space applications.
>>
>> Delay: 0.003064
>> Delay: 0.003066
>> Delay: 0.003504
>> Delay: 0.003066
>> Delay: 0.003066
>> Delay: 0.003066
>> Delay: 0.003090
>> Delay: 0.003064
>> Delay: 0.003066
>> Delay: 0.003067
>> Delay: 0.053807 <- 53 ms instead 3ms
>> Delay: 0.003067
>> Delay: 0.003067
>>
>> I found out that problem depends from a real time queue between two
>> Xenomai tasks.
>>
>> retval = rt_queue_create(&RtMeasureData, "RtMeasureData", 1000 * 256,
>> Q_UNLIMITED, Q_FIFO | Q_SHARED);
>>
>> A real time task writes measurement data every 2 ms to a rt_queue.
>> retval = rt_queue_write(&RtMeasureData, &FifoMeasureData, 256, Q_NORMAL);
>>
>> Another Xenomai task reads all measured data from queue and makes them
>> available for user land applications. (Task is not real time, context
>> switches)
>>
>> retval = rt_queue_read(&RtMeasureData, &FifoMeasureData, 256,
>> TM_NONBLOCK);
>>
>> When nothing is written to queue is there also never an unexpected delay.
>>
>> Any ideas or recommendations to find reason for problem?
>
> Without seeing any code, there is strictly nothing we can do.
> If you are able to reduce the code to a simple, self-contained test case
> we can run anywhere, then we can have a look at it.
> If not, then use the I-pipe tracer.
>
> --
> 					    Gilles.
>

Re: [Xenomai] Xenomai and normal linux application - Wait time with select() not reliable

[Gilles Chanteperdrix]

On 12/03/2012 08:59 AM, Jens Köhler wrote:

> Hello,
> 
> I found following additionaly out:
> One of the Xenomai tasks writes cyclically to a shared linux memory
> (its not a Xenomai shared memory). Write and read access is controlled
> by Linux semaphores. If shared memory writing is out-commented is
> there never a delay problem.


If you expect a task using a Linux semaphore to have bounded latencies,
you misunderstood the way Xenomai works.

> 
> I found following explanation about Xenomai in internet. Could it
> reason for problem?
> 
> "If an interrupt arrives while a Xenomai thread executes in secondary
> mode, it is not forwarded to Linux! The mechanism works as follows:
> When a Xenomai thread is executed in secondary mode, the interrupt
> shield domain is activated All Linux interrupts are stalled, until the
> Xenomai thread completes execution. The interrupts will be served
> again when Linux goes back to execute in background mode."


The interrupt shield no longer exists.


-- 
                                                                Gilles.

Re: [Xenomai] Xenomai and normal linux application - Wait time with select() not reliable

[Jens Köhler]

Hi,

> If you expect a task using a Linux semaphore to have bounded latencies,
> you misunderstood the way Xenomai works.

I have not so a deep knowledge about linux semaphores and xenomai. A
linux semaphore controls access to a linux shared memory. The Xenomai
task is the only task which writes to shared memory. Several linux
applications reading from shared memory. The task of semaphore is to
prevent that during data reading the Xenomai task could write to
shared memory. I understand Xenomai has highest priority. What does
xenomai/ linux when xenomai task tries to write to shared memory, but
semaphore blocks writing?

best regards,
Jens

Re: [Xenomai] Xenomai and normal linux application - Wait time with select() not reliable

[Gilles Chanteperdrix]

On 01/21/2013 01:06 PM, Jens Köhler wrote:

> Hello Gilles,
> 
> I did not understand your latests statement. What do you mean with "If
> you expect a task using a Linux semaphore to have bounded latencies,
> you misunderstood the way Xenomai works."
> 
> Do you think it should work without problems, without delays, latencies?
> Do you know if someone has already used linux semaphores in a xenomai thread?
> What do you recommend to find reason and a solution?


Using a Linux semaphore causes a switch to secondary mode, which means
potentially unbounded latencies.

Please keep the list in CC.

-- 
                                                                Gilles.

Re: [Xenomai] Xenomai and normal linux application - Wait time with select() not reliable

[Gilles Chanteperdrix]

On 01/21/2013 03:42 PM, Jens Köhler wrote:

> Hi Gilles,
> 
>> Using a Linux semaphore causes a switch to secondary mode, which means
>> potentially unbounded latencies.
> 
> Yes, I have no problem with the secondary mode. I have only the
> problem that linux kernel not restarts sleeping task as fast as
> expected. Instead 3 ms -> 50 ms. And it happens only when a linux
> semaphore is in use. Do you have an idea why this happens and how to
> find reason?


If this issue is not due to xenomai, then the xenomai mailing list may
not be the best place to ask.

> 
> 
> 
>> Please keep the list in CC.
> What means CC ?


Carbon Copy

-- 
                                                                Gilles.