[Xenomai] Message Pipe services behaviour

[Xenomai] Message Pipe services behaviour

[Steve M. Robbins]

Hi,

I will apologize in advance that this question concerns Xenomai 2.5.6.  I do 
realize it is an old version, but that is what our vendor provides so that is 
what I am using.

A while back I was asking about replacing a linux FIFO-based queue used to 
send messages from a real time task to a linux process.  The recommendation 
[1] was that the Message Pipe Service would serve nicely.  I recently 
implemented it and the messages are flowing, but I have questions about some 
observed behaviours.

Our real time task runs at 250 Hz and produces a small number of messages 
(usually about 2, but up to 10) each iteration.  Each message is about 120 
bytes.  I use rt_pipe_write() with P_NORMAL.  On the linux side, the process 
does a loop with select() and read().  

Initially, messages are flowing, but after some time, rt_pipe_write() begins to 
return -ENOMEM intermittently but more and more often as time passes.  The 
first thing I tried was changing the poolsize of rt_pipe_create() from 0 to 
1000*sizeof(message_structure).  The problem remained.

Next, I put in code to count the number of calls to rt_pipe_write() and the 
number of reads().  There is a growing gap that leads me to believe that the 
reader is not keeping up with the writer.

In the old FIFO-based code, we had a suspicion of the similar problem.  Since 
the linux process is not real time, we expect multiple messages to be waiting.  
To avoid running through the select/read loop once for each message, our old 
FIFO-based code opened the file in nonblocking mode and did a read into a buffer 
that could hold up to 100 messages.  We generally would read 5-7 at a time, 
not 100, so this proved that the reader could keep up when reading in batches.

Using this same technique with the /dev/rtpN end of a Message Pipe never reads 
more than 1 message.  I tried both blocking and non-blocking modes but got the 
same result.  So this is my main question: knowing that there are likely many 
messages in the pipe, how do I read them in a batch?




The description about is simplified.  I'm not sure whether this matters, but 
I'll give some more context.  

In the actual code, we have three queues with messages of different sizes: 
approximately 120, 150, and 1500 bytes.  In each iteration, the real time task 
generates exactly one of the largest message,  generally about 2 of the 
smallest, and generally none of the middle size.  But the small and middle 
sizes can have anywhere from 0 to maybe 10 on any one iteration.  

The linux process has multiple threads.  The main thread uses a select() loop 
that waits for input on the small and medium sized queues as well as a timer 
and a UDP socket.  The loop code handles all those inputs as they occur.  Both 
the small and middle queue use the read-in-batches technique but both only 
read 1 message each pass.

The largest queue is handled by a different thread that also uses a select() 
loop, but the large queue is the only input handled.  The read-in-batch 
technique is not used here.

When I created the pipes with poolsize=0, all three queues would eventually 
begin to return -ENOMEM.  After switching to poolsize=1000*sizeof(message), it 
seems to be only the middle-sized message that fails to allocate.  That's not 
a conclusive observation, but if the other two are failing to allocate it is 
much less often than the middle one which eventally gets stuffed up and always 
fails.  This is quite surprising to me.

I will admit that I did look through the code for the pipe [2,3] but wasn't 
able to discover the answer to my first question.   Any tips gratefully 
appreciated!

Thanks,
-Steve


[1] http://www.xenomai.org/pipermail/xenomai/2014-July/031365.html
[2] 
http://git.xenomai.org/xenomai-2.5.git/tree/ksrc/skins/native/pipe.c?id=v2.5.6
[3] http://git.xenomai.org/xenomai-2.5.git/tree/ksrc/nucleus/pipe.c?id=v2.5.6

Re: [Xenomai] Message Pipe services behaviour

[Philippe Gerum]

On 10/29/2014 05:11 PM, Steve M. Robbins wrote:

> In the old FIFO-based code, we had a suspicion of the similar problem.  Since 
> the linux process is not real time, we expect multiple messages to be waiting.  
> To avoid running through the select/read loop once for each message, our old 
> FIFO-based code opened the file in nonblocking mode and did a read into a buffer 
> that could hold up to 100 messages.  We generally would read 5-7 at a time, 
> not 100, so this proved that the reader could keep up when reading in batches.
> 
> Using this same technique with the /dev/rtpN end of a Message Pipe never reads 
> more than 1 message.  I tried both blocking and non-blocking modes but got the 
> same result.  So this is my main question: knowing that there are likely many 
> messages in the pipe, how do I read them in a batch?
> 

Reading with O_NONBLOCK set on /dev/rtp should do the trick. If it does
not, this could mean that your rt side is filling up the pool too fast
compared to the scheduling opportunities for the nrt side. The latter
won't run until the rt activity becomes quiescent long enough.

You can track how many bytes are readable on the nrt side by using
ioctl(.., FIONREAD, &some_int), to make sure this value does increase
over time until ENOMEM is received. Otherwise there must be a leak of
consumed message buffers.

[snip]

> When I created the pipes with poolsize=0, all three queues would eventually 
> begin to return -ENOMEM.  After switching to poolsize=1000*sizeof(message), it 
> seems to be only the middle-sized message that fails to allocate.  That's not 
> a conclusive observation, but if the other two are failing to allocate it is 
> much less often than the middle one which eventally gets stuffed up and always 
> fails.  This is quite surprising to me.

poolsize=0 means to pull the buffer memory from the main Xenomai heap,
the size of which is given by CONFIG_XENO_OPT_SYS_HEAPSZ in your
Kconfig. Otherwise, some kernel memory is pulled from the regular linux
allocator for this purpose.

So if ENOMEM is received with poolsize=0, this means that a memory
shortage happens due to the pressure on the Xenomai main/system heap,
which is always bad news. I would recommend to use a local pool (i.e.
poolsz > 0) to prevent a consumption peak on the pipe from affecting the
whole Xenomai system.

You can read /proc/xenomai/heap while your test runs, to observe how
heaps behave.

Also, a memory leak was fixed in the 2.6 time frame, but it would bite
only when closing the file descriptor to the pipe. This said, you may
want to patch it in, to save some hair pulling sessions in the future:
http://git.xenomai.org/xenomai-2.6.git/commit/?id=ef8be4e4e58489479c2245a87b1dbd04d331dca9

-- 
Philippe.

Re: [Xenomai] Message Pipe services behaviour

[Steve M. Robbins]

On Wed, Oct 29, 2014 at 06:30:40PM +0100, Philippe Gerum wrote:
> On 10/29/2014 05:11 PM, Steve M. Robbins wrote:
> 
> > In the old FIFO-based code, we had a suspicion of the similar problem.  Since 
> > the linux process is not real time, we expect multiple messages to be waiting.  
> > To avoid running through the select/read loop once for each message, our old 
> > FIFO-based code opened the file in nonblocking mode and did a read into a buffer 
> > that could hold up to 100 messages.  We generally would read 5-7 at a time, 
> > not 100, so this proved that the reader could keep up when reading in batches.
> > 
> > Using this same technique with the /dev/rtpN end of a Message Pipe never reads 
> > more than 1 message.  I tried both blocking and non-blocking modes but got the 
> > same result.  So this is my main question: knowing that there are likely many 
> > messages in the pipe, how do I read them in a batch?
> > 
> 
> Reading with O_NONBLOCK set on /dev/rtp should do the trick.

Great!  I changed back to O_NONBLOCK (and verified on the running file
using /proc/pid/fdstats).  And I'm using a private pool for the pipe.


> If it does
> not, this could mean that your rt side is filling up the pool too fast
> compared to the scheduling opportunities for the nrt side. The latter
> won't run until the rt activity becomes quiescent long enough.
> 
> You can track how many bytes are readable on the nrt side by using
> ioctl(.., FIONREAD, &some_int), to make sure this value does increase
> over time until ENOMEM is received. Otherwise there must be a leak of
> consumed message buffers.

Good suggestion.  I am now tracking the number of messages available
using the ioctl() just prior to read.  In the most recent run, there
was an average of 83 messages available, but read() only obtained one.
That suggests to me that the nrt side is being scheduled often enough.


I can also see via /proc/xenomai/heap (thanks for the tip!) that the
heap does fill up, so ENOMEM is valid.  When the message rate slows
down, the nrt side is able to catch up and the heap size comes back
down (and ENOMEM goes away).

The previous kernel FIFO-based code actually uses the same code on the
nrt side so I have a bit of confidence in that.  With the FIFO code, I
can see from the ioctl() and read() that it does read all available
bytes.  I'm really puzzled why the rt_pipe is behaving differently.

-Steve
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 173 bytes
Desc: Digital signature
URL: <http://www.xenomai.org/pipermail/xenomai/attachments/20141031/be1ecbbd/attachment.sig>

Re: [Xenomai] Message Pipe services behaviour

[Philippe Gerum]

On 10/31/2014 09:00 PM, Steve M. Robbins wrote:
> On Wed, Oct 29, 2014 at 06:30:40PM +0100, Philippe Gerum wrote:
>> On 10/29/2014 05:11 PM, Steve M. Robbins wrote:
>>
>>> In the old FIFO-based code, we had a suspicion of the similar problem.  Since 
>>> the linux process is not real time, we expect multiple messages to be waiting.  
>>> To avoid running through the select/read loop once for each message, our old 
>>> FIFO-based code opened the file in nonblocking mode and did a read into a buffer 
>>> that could hold up to 100 messages.  We generally would read 5-7 at a time, 
>>> not 100, so this proved that the reader could keep up when reading in batches.
>>>
>>> Using this same technique with the /dev/rtpN end of a Message Pipe never reads 
>>> more than 1 message.  I tried both blocking and non-blocking modes but got the 
>>> same result.  So this is my main question: knowing that there are likely many 
>>> messages in the pipe, how do I read them in a batch?
>>>
>>
>> Reading with O_NONBLOCK set on /dev/rtp should do the trick.
> 
> Great!  I changed back to O_NONBLOCK (and verified on the running file
> using /proc/pid/fdstats).  And I'm using a private pool for the pipe.
> 
> 
>> If it does
>> not, this could mean that your rt side is filling up the pool too fast
>> compared to the scheduling opportunities for the nrt side. The latter
>> won't run until the rt activity becomes quiescent long enough.
>>
>> You can track how many bytes are readable on the nrt side by using
>> ioctl(.., FIONREAD, &some_int), to make sure this value does increase
>> over time until ENOMEM is received. Otherwise there must be a leak of
>> consumed message buffers.
> 
> Good suggestion.  I am now tracking the number of messages available
> using the ioctl() just prior to read.  In the most recent run, there
> was an average of 83 messages available, but read() only obtained one.
> That suggests to me that the nrt side is being scheduled often enough.
> 

FIONREAD returns the number of bytes pending read, not the count of
individual messages, so this would denote an average of 83 bytes pending
read.

> 
> I can also see via /proc/xenomai/heap (thanks for the tip!) that the
> heap does fill up, so ENOMEM is valid.  When the message rate slows
> down, the nrt side is able to catch up and the heap size comes back
> down (and ENOMEM goes away).
> 
> The previous kernel FIFO-based code actually uses the same code on the
> nrt side so I have a bit of confidence in that.  With the FIFO code, I
> can see from the ioctl() and read() that it does read all available
> bytes.  I'm really puzzled why the rt_pipe is behaving differently.
> 

The fact that the message pool fills up again once the data is consumed
on the nrt side seems to rule out a memory leak.

Perhaps the rt side sends a large burst of data once in a while causing
the overflow? In this case, you could not detect the issue looming from
the nrt side until it happens, since the rt side has higher priority
(i.e. rt would cause ENOMEM even before nrt had a chance to resume
execution).

This is a major difference with a regular FIFO, where the sender would
block upon congestion, until the reader consumes enough data for the
write op to complete. With rt-pipes, we don't allow the rt side to wait
for the nrt side by design, so the only possible outcome upon write
congestion from rt to nrt is ENOMEM, basically.

-- 
Philippe.

Re: [Xenomai] Message Pipe services behaviour

[Steve M. Robbins]

On November 1, 2014 10:54:39 AM Philippe Gerum wrote:
> On 10/31/2014 09:00 PM, Steve M. Robbins wrote:
> > On Wed, Oct 29, 2014 at 06:30:40PM +0100, Philippe Gerum wrote:
> >> On 10/29/2014 05:11 PM, Steve M. Robbins wrote:
> >>> In the old FIFO-based code, we had a suspicion of the similar problem. 
> >>> Since the linux process is not real time, we expect multiple messages
> >>> to be waiting. To avoid running through the select/read loop once for
> >>> each message, our old FIFO-based code opened the file in nonblocking
> >>> mode and did a read into a buffer that could hold up to 100 messages. 
> >>> We generally would read 5-7 at a time, not 100, so this proved that the
> >>> reader could keep up when reading in batches.
> >>> 
> >>> Using this same technique with the /dev/rtpN end of a Message Pipe never
> >>> reads more than 1 message.  I tried both blocking and non-blocking
> >>> modes but got the same result.  So this is my main question: knowing
> >>> that there are likely many messages in the pipe, how do I read them in
> >>> a batch?
> >> 
> >> Reading with O_NONBLOCK set on /dev/rtp should do the trick.
> > 
> > Great!  I changed back to O_NONBLOCK (and verified on the running file
> > using /proc/pid/fdstats).  And I'm using a private pool for the pipe.
> > 
> >> If it does
> >> not, this could mean that your rt side is filling up the pool too fast
> >> compared to the scheduling opportunities for the nrt side. The latter
> >> won't run until the rt activity becomes quiescent long enough.
> >> 
> >> You can track how many bytes are readable on the nrt side by using
> >> ioctl(.., FIONREAD, &some_int), to make sure this value does increase
> >> over time until ENOMEM is received. Otherwise there must be a leak of
> >> consumed message buffers.
> > 
> > Good suggestion.  I am now tracking the number of messages available
> > using the ioctl() just prior to read.  In the most recent run, there
> > was an average of 83 messages available, but read() only obtained one.
> > That suggests to me that the nrt side is being scheduled often enough.
> 
> FIONREAD returns the number of bytes pending read, [...]

Right.  I divided by the message size, so it is indeed 83 messages available.


> > I can also see via /proc/xenomai/heap (thanks for the tip!) that the
> > heap does fill up, so ENOMEM is valid.  When the message rate slows
> > down, the nrt side is able to catch up and the heap size comes back
> > down (and ENOMEM goes away).
> > 
> > The previous kernel FIFO-based code actually uses the same code on the
> > nrt side so I have a bit of confidence in that.  With the FIFO code, I
> > can see from the ioctl() and read() that it does read all available
> > bytes.  I'm really puzzled why the rt_pipe is behaving differently.
> 
> The fact that the message pool fills up again once the data is consumed
> on the nrt side seems to rule out a memory leak.
> 
> Perhaps the rt side sends a large burst of data once in a while causing
> the overflow? In this case, you could not detect the issue looming from
> the nrt side until it happens, since the rt side has higher priority
> (i.e. rt would cause ENOMEM even before nrt had a chance to resume
> execution).

OK, I understand the theory.  However, I don't believe that is my case.  The 
message queue is transporting fault information and through user actions I can 
set it up to send 2 messages per cycle and using the FIFO code, the nrt side 
is indeed reading 2 messages.  Using message queues, I can see that there are 
multiple messages outstanding but it reads only one.


I do appreciate all your help!  

I'm at a loss of how to proceed, but I'd like to hear from anyone else using 
message pipes whether they succeeded in reading from nrt side using O_NONBLOCK 
-- and what xenomai version was used.

Thanks,
-Steve

Re: [Xenomai] Message Pipe services behaviour

[Philippe Gerum]

On 11/02/2014 01:53 AM, Steve M. Robbins wrote:
> On November 1, 2014 10:54:39 AM Philippe Gerum wrote:

>> Perhaps the rt side sends a large burst of data once in a while causing
>> the overflow? In this case, you could not detect the issue looming from
>> the nrt side until it happens, since the rt side has higher priority
>> (i.e. rt would cause ENOMEM even before nrt had a chance to resume
>> execution).
> 
> OK, I understand the theory.  However, I don't believe that is my case.  The 
> message queue is transporting fault information and through user actions I can 
> set it up to send 2 messages per cycle and using the FIFO code, the nrt side 
> is indeed reading 2 messages.  Using message queues, I can see that there are 
> multiple messages outstanding but it reads only one.
> 

Ok, so if no message gets lost, if you can see FIONREAD increase until
ENOMEM is raised, and if throttling the rt side only delays the issue
without solving it, the only explanation would be that a notification of
input availability is lost by the poll handler backing select() in the
rt-pipe driver. That would cause the messages to pile up on the rt side,
with too few deliveries to nrt.

Only for the purpose of checking this, could you try read()ing the large
message queue directly, then in a second attempt, enabling SIGIO on the
rtp fildes to get the messages asynchronously, instead of sensing them
via select()? The idea would be to check whether getting rid of select()
improves the situation.

NOTE: I'm assuming that the rt side only uses rt_pipe_write(), and never
rt_pipe_stream() which does packet coalescence by design.

-- 
Philippe.

Re: [Xenomai] Message Pipe services behaviour

[dietmar.schindler]

> Von: Steve M. Robbins
> Gesendet: Mittwoch, 29. Oktober 2014 17:11
> ...
> To avoid running through the select/read loop once for each message, our old
> FIFO-based code opened the file in nonblocking mode and did a read into a buffer
> that could hold up to 100 messages.  We generally would read 5-7 at a time,
> not 100, so this proved that the reader could keep up when reading in batches.
>
> Using this same technique with the /dev/rtpN end of a Message Pipe never reads
> more than 1 message.  I tried both blocking and non-blocking modes but got the
> same result.  So this is my main question: knowing that there are likely many
> messages in the pipe, how do I read them in a batch?

As far as I can see, read() from a Xenomai message pipe by design never returns more than one message at a time.
--
Regards,
Dietmar Schindler
________________________________
manroland web systems GmbH -- Managing Director: Joern Gossé
Registered Office: Augsburg -- Trade Register: AG Augsburg -- HRB-No.: 26816 -- VAT: DE281389840

Confidentiality note:
This eMail and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you are not the intended recipient, you are hereby notified that any use or dissemination of this communication is strictly prohibited. If you have received this eMail in error, then please delete this eMail.

! Please consider your environmental responsibility before printing this eMail !
________________________________

Re: [Xenomai] Message Pipe services behaviour

[Philippe Gerum]

On 11/03/2014 09:54 AM, dietmar.schindler@manroland-web.com wrote:
>> Von: Steve M. Robbins
>> Gesendet: Mittwoch, 29. Oktober 2014 17:11
>> ...
>> To avoid running through the select/read loop once for each message, our old
>> FIFO-based code opened the file in nonblocking mode and did a read into a buffer
>> that could hold up to 100 messages.  We generally would read 5-7 at a time,
>> not 100, so this proved that the reader could keep up when reading in batches.
>>
>> Using this same technique with the /dev/rtpN end of a Message Pipe never reads
>> more than 1 message.  I tried both blocking and non-blocking modes but got the
>> same result.  So this is my main question: knowing that there are likely many
>> messages in the pipe, how do I read them in a batch?
> 
> As far as I can see, read() from a Xenomai message pipe by design never returns more than one message at a time.

Clearly, but from the description, it seems like pulling all messages in
a loop from a non-blocking file descriptor only returns a single
message, although more messages are pending read, according to FIONREAD.

> --
> Regards,
> Dietmar Schindler
> ________________________________
> manroland web systems GmbH -- Managing Director: Joern Gossé
> Registered Office: Augsburg -- Trade Register: AG Augsburg -- HRB-No.: 26816 -- VAT: DE281389840
> 
> Confidentiality note:
> This eMail and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you are not the intended recipient, you are hereby notified that any use or dissemination of this communication is strictly prohibited. If you have received this eMail in error, then please delete this eMail.
> 
> ! Please consider your environmental responsibility before printing this eMail !
> ________________________________
> _______________________________________________
> Xenomai mailing list
> Xenomai@xenomai.org
> http://www.xenomai.org/mailman/listinfo/xenomai
> 


-- 
Philippe.

Re: [Xenomai] Message Pipe services behaviour

[Steve M. Robbins]

On Mon, Nov 03, 2014 at 09:18:53AM +0100, Philippe Gerum wrote:
> On 11/02/2014 01:53 AM, Steve M. Robbins wrote:
> > On November 1, 2014 10:54:39 AM Philippe Gerum wrote:
> 
> >> Perhaps the rt side sends a large burst of data once in a while causing
> >> the overflow? In this case, you could not detect the issue looming from
> >> the nrt side until it happens, since the rt side has higher priority
> >> (i.e. rt would cause ENOMEM even before nrt had a chance to resume
> >> execution).
> > 
> > OK, I understand the theory.  However, I don't believe that is my case.  The 
> > message queue is transporting fault information and through user actions I can 
> > set it up to send 2 messages per cycle and using the FIFO code, the nrt side 
> > is indeed reading 2 messages.  Using message queues, I can see that there are 
> > multiple messages outstanding but it reads only one.
> > 
> 
> Ok, so if no message gets lost, if you can see FIONREAD increase until
> ENOMEM is raised, and if throttling the rt side only delays the issue
> without solving it, the only explanation would be that a notification of
> input availability is lost by the poll handler backing select() in the
> rt-pipe driver. That would cause the messages to pile up on the rt side,
> with too few deliveries to nrt.

I want to emphasize that FIONREAD is being done on the nrt side, after
the select() and just prior to the actual read() call.  Thus the nrt
side is seeing N > 1 messages available on the file descriptor, but
read() only reads 1, even though the fd is nonblocking and the read
buffer is sized for 100 messages.  So it seems to me there is somehow
a disconnect between the "available bytes" as seen by read() versus
FIONREAD.  I admit to being naive about the kernel but that seems
like something that should not be possible.


Given that, I don't understand what you're saying about notification
of input availability being lost.  It may be true that the select()
doesn't fire as often as it should, but the nrt side does wake up
prior to the queue being completely full and still will only read 1
message.  Are you saying that missing a notification would affect the
"available bytes" seen by read() on the nrt side?  But still the
FIONREAD would tell us the truth?



> Only for the purpose of checking this, could you try read()ing the large
> message queue directly, then in a second attempt, enabling SIGIO on the
> rtp fildes to get the messages asynchronously, instead of sensing them
> via select()? The idea would be to check whether getting rid of select()
> improves the situation.

I'll check into doing that.



> NOTE: I'm assuming that the rt side only uses rt_pipe_write(), and never
> rt_pipe_stream() which does packet coalescence by design.

Yes, that's correct.  The only calls the rt side makes are: rt_pipe_create()
and rt_pipe_write().

The only calls the nrt side makes is open(), ioctl(FIONREAD), and read().
I am presently opening with O_RDONLY | O_NONBLOCK.  I have also tried
O_RDWR | O_NONBLOCK and saw the same results.

Also: the nrt side is a "mostly pure" linux program that does NOT call
wrapped versions of select(), etc.  It does, however, link with a
vendor library that links with xenomai libs and the vendor library
does somehow make the thread in question appear as a xenomai task.
Apart from a couple of mode switches (presumably when intializing the
vendor lib), it runs exclusively in linux mode.

-Steve
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 173 bytes
Desc: Digital signature
URL: <http://www.xenomai.org/pipermail/xenomai/attachments/20141103/84a8c58f/attachment.sig>

Re: [Xenomai] Message Pipe services behaviour

[Philippe Gerum]

On 11/03/2014 04:59 PM, Steve M. Robbins wrote:
> On Mon, Nov 03, 2014 at 09:18:53AM +0100, Philippe Gerum wrote:
>> On 11/02/2014 01:53 AM, Steve M. Robbins wrote:
>>> On November 1, 2014 10:54:39 AM Philippe Gerum wrote:
>>
>>>> Perhaps the rt side sends a large burst of data once in a while causing
>>>> the overflow? In this case, you could not detect the issue looming from
>>>> the nrt side until it happens, since the rt side has higher priority
>>>> (i.e. rt would cause ENOMEM even before nrt had a chance to resume
>>>> execution).
>>>
>>> OK, I understand the theory.  However, I don't believe that is my case.  The 
>>> message queue is transporting fault information and through user actions I can 
>>> set it up to send 2 messages per cycle and using the FIFO code, the nrt side 
>>> is indeed reading 2 messages.  Using message queues, I can see that there are 
>>> multiple messages outstanding but it reads only one.
>>>
>>
>> Ok, so if no message gets lost, if you can see FIONREAD increase until
>> ENOMEM is raised, and if throttling the rt side only delays the issue
>> without solving it, the only explanation would be that a notification of
>> input availability is lost by the poll handler backing select() in the
>> rt-pipe driver. That would cause the messages to pile up on the rt side,
>> with too few deliveries to nrt.
> 
> I want to emphasize that FIONREAD is being done on the nrt side, after
> the select() and just prior to the actual read() call.  Thus the nrt
> side is seeing N > 1 messages available on the file descriptor, but
> read() only reads 1, even though the fd is nonblocking and the read
> buffer is sized for 100 messages.

Just to make sure we are on the same page, as Dietmar mentioned in an
earlier mail, read() as implemented by the rt-pipe driver only returns
_one_ message at a time, although > 1 could be pending. One has to
iterate over a read() loop using a non-blocking fd on the /dev/rtp
device to get them all. I'm likely beating a dead horse here this said.

  So it seems to me there is somehow
> a disconnect between the "available bytes" as seen by read() versus
> FIONREAD.  I admit to being naive about the kernel but that seems
> like something that should not be possible.
> 
> 
> Given that, I don't understand what you're saying about notification
> of input availability being lost.  It may be true that the select()
> doesn't fire as often as it should, but the nrt side does wake up
> prior to the queue being completely full and still will only read 1
> message.  Are you saying that missing a notification would affect the
> "available bytes" seen by read() on the nrt side?  But still the
> FIONREAD would tell us the truth?

FIONREAD value and select() events are not synchronized, FIONREAD is
maintained in the rt-send and nrt-read handlers. FIONREAD always tells
the truth. On the other hand, if for some reason a wake up event is
missed, select() would be wrong. I don't have any evidence that such bug
happens looking at the code, but would something go wrong there, then it
might explain the behaviour you observed.

-- 
Philippe.

Re: [Xenomai] Message Pipe services behaviour

[Steve M. Robbins]

On Mon, Nov 03, 2014 at 05:29:53PM +0100, Philippe Gerum wrote:

> Just to make sure we are on the same page, as Dietmar mentioned in an
> earlier mail, read() as implemented by the rt-pipe driver only returns
> _one_ message at a time, although > 1 could be pending. One has to
> iterate over a read() loop using a non-blocking fd on the /dev/rtp
> device to get them all.

Aha!  Thanks (and to Dietmar).  I completely missed this fact.  I'll
go re-write the read as a loop and see if it works better.  I should read
until the read() returns 0 (bytes read)?

Thanks,
-Steve




-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 173 bytes
Desc: Digital signature
URL: <http://www.xenomai.org/pipermail/xenomai/attachments/20141103/ee00a96d/attachment.sig>

Re: [Xenomai] Message Pipe services behaviour

[Philippe Gerum]

On 11/03/2014 07:37 PM, Steve M. Robbins wrote:
> On Mon, Nov 03, 2014 at 05:29:53PM +0100, Philippe Gerum wrote:
> 
>> Just to make sure we are on the same page, as Dietmar mentioned in an
>> earlier mail, read() as implemented by the rt-pipe driver only returns
>> _one_ message at a time, although > 1 could be pending. One has to
>> iterate over a read() loop using a non-blocking fd on the /dev/rtp
>> device to get them all.
> 
> Aha!  Thanks (and to Dietmar).  I completely missed this fact.  I'll
> go re-write the read as a loop and see if it works better.  I should read
> until the read() returns 0 (bytes read)?
> 

Until you get -1 with errno == EWOULDBLOCK. Zero would rather mean that
the other end of pipe has been disconnected, just like with a regular FIFO.

-- 
Philippe.

Re: [Xenomai] Message Pipe services behaviour

[Gilles Chanteperdrix]

On Mon, Nov 03, 2014 at 09:01:32PM +0100, Philippe Gerum wrote:
> On 11/03/2014 07:37 PM, Steve M. Robbins wrote:
> > On Mon, Nov 03, 2014 at 05:29:53PM +0100, Philippe Gerum wrote:
> > 
> >> Just to make sure we are on the same page, as Dietmar mentioned in an
> >> earlier mail, read() as implemented by the rt-pipe driver only returns
> >> _one_ message at a time, although > 1 could be pending. One has to
> >> iterate over a read() loop using a non-blocking fd on the /dev/rtp
> >> device to get them all.
> > 
> > Aha!  Thanks (and to Dietmar).  I completely missed this fact.  I'll
> > go re-write the read as a loop and see if it works better.  I should read
> > until the read() returns 0 (bytes read)?
> > 
> 
> Until you get -1 with errno == EWOULDBLOCK. Zero would rather mean that
> the other end of pipe has been disconnected, just like with a regular FIFO.
> 

Just a stupid question: should not calling select then read make as
long as there are some messages to read?

-- 
					    Gilles.

Re: [Xenomai] Message Pipe services behaviour

[Philippe Gerum]

On 11/03/2014 09:03 PM, Gilles Chanteperdrix wrote:
> On Mon, Nov 03, 2014 at 09:01:32PM +0100, Philippe Gerum wrote:
>> On 11/03/2014 07:37 PM, Steve M. Robbins wrote:
>>> On Mon, Nov 03, 2014 at 05:29:53PM +0100, Philippe Gerum wrote:
>>>
>>>> Just to make sure we are on the same page, as Dietmar mentioned in an
>>>> earlier mail, read() as implemented by the rt-pipe driver only returns
>>>> _one_ message at a time, although > 1 could be pending. One has to
>>>> iterate over a read() loop using a non-blocking fd on the /dev/rtp
>>>> device to get them all.
>>>
>>> Aha!  Thanks (and to Dietmar).  I completely missed this fact.  I'll
>>> go re-write the read as a loop and see if it works better.  I should read
>>> until the read() returns 0 (bytes read)?
>>>
>>
>> Until you get -1 with errno == EWOULDBLOCK. Zero would rather mean that
>> the other end of pipe has been disconnected, just like with a regular FIFO.
>>
> 
> Just a stupid question: should not calling select then read make as
> long as there are some messages to read?
> 

That would be the normal option, but Steve observes issues with this
pattern, we are trying to find out whether a loss of event notification
might be at work in the poll() handler basically. I still can't see it
reading the code though, but we cannot rule out any option at this stage.

-- 
Philippe.

Re: [Xenomai] Message Pipe services behaviour

[Steve M. Robbins]

On Mon, Nov 03, 2014 at 09:01:32PM +0100, Philippe Gerum wrote:
> On 11/03/2014 07:37 PM, Steve M. Robbins wrote:
> > On Mon, Nov 03, 2014 at 05:29:53PM +0100, Philippe Gerum wrote:
> > 
> >> Just to make sure we are on the same page, as Dietmar mentioned in an
> >> earlier mail, read() as implemented by the rt-pipe driver only returns
> >> _one_ message at a time, although > 1 could be pending. One has to
> >> iterate over a read() loop using a non-blocking fd on the /dev/rtp
> >> device to get them all.
> > 
> > Aha!  Thanks (and to Dietmar).  I completely missed this fact.  I'll
> > go re-write the read as a loop and see if it works better.  I should read
> > until the read() returns 0 (bytes read)?
> > 
> 
> Until you get -1 with errno == EWOULDBLOCK. Zero would rather mean that
> the other end of pipe has been disconnected, just like with a regular FIFO.

Yes, that seems to work.  The loop is reading about 2 messages on
average as expected and no more filling up the heap.

Thanks!
-Steve
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 173 bytes
Desc: Digital signature
URL: <http://www.xenomai.org/pipermail/xenomai/attachments/20141103/1e452982/attachment.sig>