RTnet default behavior

RTnet default behavior

[Johannes Holtz]

Hi ,

I inherited a project where the RTnet stack is loaded manually at boot 
time. To be specific. Instead of using the rtnet command line programs 
the rtnet.ko kernel object as well as the necessary drivers are loaded 
with insmod. The devices are then brought up with rtifconfig and rtroute.

The network consist of 7 PCs (all with the same boot time mechanism) 
with pre-defined IPs. The setup works and initializes in an 
asynchronized way (like a typical non-RT network).

I'm investigating some performance problems and I don't know whats is 
really going on since no config is used. How is determined which 
participant is acting as Master and how the slots are distributed? What 
are the default values?

Also: The RTmac readme says a slot "transmits a single packet of up to a 
specific maximum size". I like to know what that size is and what the 
behavior is when I have many much smaller datagrams queued. Will they be 
distributed over as many cycles? And also what happens if I send a 
datagram which is much bigger. Is there an MTU?

Cheers,

Johannes

Re: RTnet default behavior

[Jan Kiszka]

On 18.10.19 16:10, Johannes Holtz via Xenomai wrote:
> Hi ,
> 
> I inherited a project where the RTnet stack is loaded manually at boot
> time. To be specific. Instead of using the rtnet command line programs
> the rtnet.ko kernel object as well as the necessary drivers are loaded
> with insmod. The devices are then brought up with rtifconfig and rtroute.
> 
> The network consist of 7 PCs (all with the same boot time mechanism)
> with pre-defined IPs. The setup works and initializes in an
> asynchronized way (like a typical non-RT network).
> 
> I'm investigating some performance problems and I don't know whats is
> really going on since no config is used. How is determined which
> participant is acting as Master and how the slots are distributed? What
> are the default values?

You only mentioned rtifconfig and rtroute above, no rtcfg or tdmacfg.
Are you sure RTmac is involved?

> 
> Also: The RTmac readme says a slot "transmits a single packet of up to a
> specific maximum size". I like to know what that size is and what the
> behavior is when I have many much smaller datagrams queued. Will they be
> distributed over as many cycles? And also what happens if I send a
> datagram which is much bigger. Is there an MTU?

I'll be happy to explain, but we should first of all try to find out
what is configured here.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: RTnet default behavior

[Johannes Holtz]

Am 18.10.19 um 16:33 schrieb Jan Kiszka:
> On 18.10.19 16:10, Johannes Holtz via Xenomai wrote:
>> Hi ,
>>
>> I inherited a project where the RTnet stack is loaded manually at boot
>> time. To be specific. Instead of using the rtnet command line programs
>> the rtnet.ko kernel object as well as the necessary drivers are loaded
>> with insmod. The devices are then brought up with rtifconfig and rtroute.
>>
>> The network consist of 7 PCs (all with the same boot time mechanism)
>> with pre-defined IPs. The setup works and initializes in an
>> asynchronized way (like a typical non-RT network).
>>
>> I'm investigating some performance problems and I don't know whats is
>> really going on since no config is used. How is determined which
>> participant is acting as Master and how the slots are distributed? What
>> are the default values?
> You only mentioned rtifconfig and rtroute above, no rtcfg or tdmacfg.
> Are you sure RTmac is involved?

This is the list of kernel object that are loaded:

rtnet.ko

rtipv4.ko

rtpacket.ko

rtudp.ko

rt_e100e.ko

rt_8139too.ko

rt_r8169.ko

subsequently rtifconfig and rtroute

Nothing else is done. I don't know id that satisfies for rtmac. How can 
I check?

>> Also: The RTmac readme says a slot "transmits a single packet of up to a
>> specific maximum size". I like to know what that size is and what the
>> behavior is when I have many much smaller datagrams queued. Will they be
>> distributed over as many cycles? And also what happens if I send a
>> datagram which is much bigger. Is there an MTU?
> I'll be happy to explain, but we should first of all try to find out
> what is configured here.
>
> Jan
>

Re: RTnet default behavior

[Jan Kiszka]

On 18.10.19 16:47, Johannes Holtz via Xenomai wrote:
> Am 18.10.19 um 16:33 schrieb Jan Kiszka:
>> On 18.10.19 16:10, Johannes Holtz via Xenomai wrote:
>>> Hi ,
>>>
>>> I inherited a project where the RTnet stack is loaded manually at boot
>>> time. To be specific. Instead of using the rtnet command line programs
>>> the rtnet.ko kernel object as well as the necessary drivers are loaded
>>> with insmod. The devices are then brought up with rtifconfig and
>>> rtroute.
>>>
>>> The network consist of 7 PCs (all with the same boot time mechanism)
>>> with pre-defined IPs. The setup works and initializes in an
>>> asynchronized way (like a typical non-RT network).
>>>
>>> I'm investigating some performance problems and I don't know whats is
>>> really going on since no config is used. How is determined which
>>> participant is acting as Master and how the slots are distributed? What
>>> are the default values?
>> You only mentioned rtifconfig and rtroute above, no rtcfg or tdmacfg.
>> Are you sure RTmac is involved?
> 
> This is the list of kernel object that are loaded:
> 
> rtnet.ko
> 
> rtipv4.ko
> 
> rtpacket.ko
> 
> rtudp.ko
> 
> rt_e100e.ko
> 
> rt_8139too.ko
> 
> rt_r8169.ko
> 
> subsequently rtifconfig and rtroute
> 
> Nothing else is done. I don't know id that satisfies for rtmac. How can
> I check?

Nope, there would be rtmac.ko, tdma.ko, possibly also rtcfg.ko in
addition. And you would see the script calling tdmacfg at least.

So, nothing of RTmac should prevent packets from being sent or add size
restrictions on them.

Maybe you can specify your performance problems in more details?

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: RTnet default behavior

[Johannes Holtz]

Am 18.10.19 um 16:57 schrieb Jan Kiszka:
> On 18.10.19 16:47, Johannes Holtz via Xenomai wrote:
>> Am 18.10.19 um 16:33 schrieb Jan Kiszka:
>>> On 18.10.19 16:10, Johannes Holtz via Xenomai wrote:
>>>> Hi ,
>>>>
>>>> I inherited a project where the RTnet stack is loaded manually at boot
>>>> time. To be specific. Instead of using the rtnet command line programs
>>>> the rtnet.ko kernel object as well as the necessary drivers are loaded
>>>> with insmod. The devices are then brought up with rtifconfig and
>>>> rtroute.
>>>>
>>>> The network consist of 7 PCs (all with the same boot time mechanism)
>>>> with pre-defined IPs. The setup works and initializes in an
>>>> asynchronized way (like a typical non-RT network).
>>>>
>>>> I'm investigating some performance problems and I don't know whats is
>>>> really going on since no config is used. How is determined which
>>>> participant is acting as Master and how the slots are distributed? What
>>>> are the default values?
>>> You only mentioned rtifconfig and rtroute above, no rtcfg or tdmacfg.
>>> Are you sure RTmac is involved?
>> This is the list of kernel object that are loaded:
>>
>> rtnet.ko
>>
>> rtipv4.ko
>>
>> rtpacket.ko
>>
>> rtudp.ko
>>
>> rt_e100e.ko
>>
>> rt_8139too.ko
>>
>> rt_r8169.ko
>>
>> subsequently rtifconfig and rtroute
>>
>> Nothing else is done. I don't know id that satisfies for rtmac. How can
>> I check?
> Nope, there would be rtmac.ko, tdma.ko, possibly also rtcfg.ko in
> addition. And you would see the script calling tdmacfg at least.
So this means I basically run standard Ethernet with no RT benefits?
>
> So, nothing of RTmac should prevent packets from being sent or add size
> restrictions on them.
>
> Maybe you can specify your performance problems in more details?
>
> Jan
>
Well, In this 7 node network one node is receiving datagrams from all 
other 6 containing sensor data. Each datagram has a sequence number to 
recognize packet loss. Each node is running a RT SW with 1ms loop. In 
certain situations the receiver detects that some (between 1 and 20) 
datagrams where lost. In these situations the amount of sent datagrams 
is higher than usual.

At first I though this might be a throughput issue but even after I 
reduced the payload size drastically I still miss datagrams. The sender 
has no problems sending and my receive code always reads all available 
data from the queue.

/proc/rtnet/stats doesn't show any errors or drops .

I'm working on an example program to test this issue in detail.

Johannes

Re: RTnet default behavior

[Jan Kiszka]

On 18.10.19 17:13, Johannes Holtz via Xenomai wrote:
> Am 18.10.19 um 16:57 schrieb Jan Kiszka:
>> On 18.10.19 16:47, Johannes Holtz via Xenomai wrote:
>>> Am 18.10.19 um 16:33 schrieb Jan Kiszka:
>>>> On 18.10.19 16:10, Johannes Holtz via Xenomai wrote:
>>>>> Hi ,
>>>>>
>>>>> I inherited a project where the RTnet stack is loaded manually at boot
>>>>> time. To be specific. Instead of using the rtnet command line programs
>>>>> the rtnet.ko kernel object as well as the necessary drivers are loaded
>>>>> with insmod. The devices are then brought up with rtifconfig and
>>>>> rtroute.
>>>>>
>>>>> The network consist of 7 PCs (all with the same boot time mechanism)
>>>>> with pre-defined IPs. The setup works and initializes in an
>>>>> asynchronized way (like a typical non-RT network).
>>>>>
>>>>> I'm investigating some performance problems and I don't know whats is
>>>>> really going on since no config is used. How is determined which
>>>>> participant is acting as Master and how the slots are distributed?
>>>>> What
>>>>> are the default values?
>>>> You only mentioned rtifconfig and rtroute above, no rtcfg or tdmacfg.
>>>> Are you sure RTmac is involved?
>>> This is the list of kernel object that are loaded:
>>>
>>> rtnet.ko
>>>
>>> rtipv4.ko
>>>
>>> rtpacket.ko
>>>
>>> rtudp.ko
>>>
>>> rt_e100e.ko
>>>
>>> rt_8139too.ko
>>>
>>> rt_r8169.ko
>>>
>>> subsequently rtifconfig and rtroute
>>>
>>> Nothing else is done. I don't know id that satisfies for rtmac. How can
>>> I check?
>> Nope, there would be rtmac.ko, tdma.ko, possibly also rtcfg.ko in
>> addition. And you would see the script calling tdmacfg at least.
> So this means I basically run standard Ethernet with no RT benefits?

The TX and RX path remain deterministic. If your application behave well
and do not generate overload on the network that can cause switches or
recipients to drop packets, all can still be fine.

>>
>> So, nothing of RTmac should prevent packets from being sent or add size
>> restrictions on them.
>>
>> Maybe you can specify your performance problems in more details?
>>
>> Jan
>>
> Well, In this 7 node network one node is receiving datagrams from all
> other 6 containing sensor data. Each datagram has a sequence number to
> recognize packet loss. Each node is running a RT SW with 1ms loop. In
> certain situations the receiver detects that some (between 1 and 20)
> datagrams where lost. In these situations the amount of sent datagrams
> is higher than usual.

What else is on the network? Only those nodes, and they only sent RT
datagrams at a well-defined rate?

> 
> At first I though this might be a throughput issue but even after I
> reduced the payload size drastically I still miss datagrams. The sender
> has no problems sending and my receive code always reads all available
> data from the queue.
> 
> /proc/rtnet/stats doesn't show any errors or drops .
> 
> I'm working on an example program to test this issue in detail.
> 

OK.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: RTnet default behavior

[Johannes Holtz]

Am 18.10.19 um 17:36 schrieb Jan Kiszka:
> On 18.10.19 17:13, Johannes Holtz via Xenomai wrote:
>> Am 18.10.19 um 16:57 schrieb Jan Kiszka:
>>> On 18.10.19 16:47, Johannes Holtz via Xenomai wrote:
>>>> Am 18.10.19 um 16:33 schrieb Jan Kiszka:
>>>>> On 18.10.19 16:10, Johannes Holtz via Xenomai wrote:
>>>>>> Hi ,
>>>>>>
>>>>>> I inherited a project where the RTnet stack is loaded manually at boot
>>>>>> time. To be specific. Instead of using the rtnet command line programs
>>>>>> the rtnet.ko kernel object as well as the necessary drivers are loaded
>>>>>> with insmod. The devices are then brought up with rtifconfig and
>>>>>> rtroute.
>>>>>>
>>>>>> The network consist of 7 PCs (all with the same boot time mechanism)
>>>>>> with pre-defined IPs. The setup works and initializes in an
>>>>>> asynchronized way (like a typical non-RT network).
>>>>>>
>>>>>> I'm investigating some performance problems and I don't know whats is
>>>>>> really going on since no config is used. How is determined which
>>>>>> participant is acting as Master and how the slots are distributed?
>>>>>> What
>>>>>> are the default values?
>>>>> You only mentioned rtifconfig and rtroute above, no rtcfg or tdmacfg.
>>>>> Are you sure RTmac is involved?
>>>> This is the list of kernel object that are loaded:
>>>>
>>>> rtnet.ko
>>>>
>>>> rtipv4.ko
>>>>
>>>> rtpacket.ko
>>>>
>>>> rtudp.ko
>>>>
>>>> rt_e100e.ko
>>>>
>>>> rt_8139too.ko
>>>>
>>>> rt_r8169.ko
>>>>
>>>> subsequently rtifconfig and rtroute
>>>>
>>>> Nothing else is done. I don't know id that satisfies for rtmac. How can
>>>> I check?
>>> Nope, there would be rtmac.ko, tdma.ko, possibly also rtcfg.ko in
>>> addition. And you would see the script calling tdmacfg at least.
>> So this means I basically run standard Ethernet with no RT benefits?
> The TX and RX path remain deterministic. If your application behave well
> and do not generate overload on the network that can cause switches or
> recipients to drop packets, all can still be fine.

Since this is not synced (not TDMA) it is very well possible that 
different nodes interfere and datagrams collide.

What would be considered overload? Since the PCs are connected via 
unmanaged switches  I can't detect if they have any problems.

>
>>> So, nothing of RTmac should prevent packets from being sent or add size
>>> restrictions on them.
>>>
>>> Maybe you can specify your performance problems in more details?
>>>
>>> Jan
>>>
>> Well, In this 7 node network one node is receiving datagrams from all
>> other 6 containing sensor data. Each datagram has a sequence number to
>> recognize packet loss. Each node is running a RT SW with 1ms loop. In
>> certain situations the receiver detects that some (between 1 and 20)
>> datagrams where lost. In these situations the amount of sent datagrams
>> is higher than usual.
> What else is on the network? Only those nodes, and they only sent RT
> datagrams at a well-defined rate?

PC1,PC2 <--->Switch1<--->Switch2<---->PC3-7

That's the whole network. Both are un-managed industry switches.

PC2 is the receiver.

>
>> At first I though this might be a throughput issue but even after I
>> reduced the payload size drastically I still miss datagrams. The sender
>> has no problems sending and my receive code always reads all available
>> data from the queue.
>>
>> /proc/rtnet/stats doesn't show any errors or drops .
>>
>> I'm working on an example program to test this issue in detail.
>>
> OK.
>
> Jan
>

Re: RTnet default behavior

[Johannes Holtz]

Am 18.10.19 um 18:00 schrieb Johannes Holtz:
> Am 18.10.19 um 17:36 schrieb Jan Kiszka:
>> On 18.10.19 17:13, Johannes Holtz via Xenomai wrote:
>>> Am 18.10.19 um 16:57 schrieb Jan Kiszka:
>>>> On 18.10.19 16:47, Johannes Holtz via Xenomai wrote:
>>>>> Am 18.10.19 um 16:33 schrieb Jan Kiszka:
>>>>>> On 18.10.19 16:10, Johannes Holtz via Xenomai wrote:
>>>>>>> Hi ,
>>>>>>>
>>>>>>> I inherited a project where the RTnet stack is loaded manually 
>>>>>>> at boot
>>>>>>> time. To be specific. Instead of using the rtnet command line 
>>>>>>> programs
>>>>>>> the rtnet.ko kernel object as well as the necessary drivers are 
>>>>>>> loaded
>>>>>>> with insmod. The devices are then brought up with rtifconfig and
>>>>>>> rtroute.
>>>>>>>
>>>>>>> The network consist of 7 PCs (all with the same boot time 
>>>>>>> mechanism)
>>>>>>> with pre-defined IPs. The setup works and initializes in an
>>>>>>> asynchronized way (like a typical non-RT network).
>>>>>>>
>>>>>>> I'm investigating some performance problems and I don't know 
>>>>>>> whats is
>>>>>>> really going on since no config is used. How is determined which
>>>>>>> participant is acting as Master and how the slots are distributed?
>>>>>>> What
>>>>>>> are the default values?
>>>>>> You only mentioned rtifconfig and rtroute above, no rtcfg or 
>>>>>> tdmacfg.
>>>>>> Are you sure RTmac is involved?
>>>>> This is the list of kernel object that are loaded:
>>>>>
>>>>> rtnet.ko
>>>>>
>>>>> rtipv4.ko
>>>>>
>>>>> rtpacket.ko
>>>>>
>>>>> rtudp.ko
>>>>>
>>>>> rt_e100e.ko
>>>>>
>>>>> rt_8139too.ko
>>>>>
>>>>> rt_r8169.ko
>>>>>
>>>>> subsequently rtifconfig and rtroute
>>>>>
>>>>> Nothing else is done. I don't know id that satisfies for rtmac. 
>>>>> How can
>>>>> I check?
>>>> Nope, there would be rtmac.ko, tdma.ko, possibly also rtcfg.ko in
>>>> addition. And you would see the script calling tdmacfg at least.
>>> So this means I basically run standard Ethernet with no RT benefits?
>> The TX and RX path remain deterministic. If your application behave well
>> and do not generate overload on the network that can cause switches or
>> recipients to drop packets, all can still be fine.
>
> Since this is not synced (not TDMA) it is very well possible that 
> different nodes interfere and datagrams collide.
>
> What would be considered overload? Since the PCs are connected via 
> unmanaged switches  I can't detect if they have any problems.
>
>>
>>>> So, nothing of RTmac should prevent packets from being sent or add 
>>>> size
>>>> restrictions on them.
>>>>
>>>> Maybe you can specify your performance problems in more details?
>>>>
>>>> Jan
>>>>
>>> Well, In this 7 node network one node is receiving datagrams from all
>>> other 6 containing sensor data. Each datagram has a sequence number to
>>> recognize packet loss. Each node is running a RT SW with 1ms loop. In
>>> certain situations the receiver detects that some (between 1 and 20)
>>> datagrams where lost. In these situations the amount of sent datagrams
>>> is higher than usual.
>> What else is on the network? Only those nodes, and they only sent RT
>> datagrams at a well-defined rate?
>
> PC1,PC2 <--->Switch1<--->Switch2<---->PC3-7
>
> That's the whole network. Both are un-managed industry switches.
>
> PC2 is the receiver.
>
>>
>>> At first I though this might be a throughput issue but even after I
>>> reduced the payload size drastically I still miss datagrams. The sender
>>> has no problems sending and my receive code always reads all available
>>> data from the queue.
>>>
>>> /proc/rtnet/stats doesn't show any errors or drops .
>>>
>>> I'm working on an example program to test this issue in detail.
>>>
>> OK.
>>
>> Jan
>>
>
Ok, after a lot of testing I arrived at some sort of insight. I almost 
became crazy after all the kernel segfaults.

I wrote a simple program with a sender and receiver where the sender 
sends a configurable amount of datagrams per 1ms cycle. (attached)

First I noticed that I get problems when allocating a static buffer of 
several kB. So I started to play around with the stack size. If I 
increase the overall stack size I can allocate that buffer but still get 
buffer size problems on the sender side so I also increased the socket 
buffer size and as long as the increased socket buffer is smaller than 
the stack everything is OK and performance increases. If not you get 
said kernel issues. Unsurprisingly in hindsight.

Obviously the size of the datagrams is also important. The smaller they 
are the smaller the buffers can be.

Curious is that I wasn't able to reproduce the case where datagrams 
where missed.

So plugging my insight of the stack and buffer increases into my actual 
program the overall system performance increases. The amount of missed 
datagrams is reduced dramatically. I'm not sure why though.


Cheers,

Johannes

-------------- next part --------------
A non-text attachment was scrubbed...
Name: main.c
Type: text/x-csrc
Size: 8469 bytes
Desc: not available
URL: <http://xenomai.org/pipermail/xenomai/attachments/20191022/9ee34255/attachment.c>

Re: RTnet default behavior

[Jan Kiszka]

On 22.10.19 10:30, Johannes Holtz via Xenomai wrote:
> Am 18.10.19 um 18:00 schrieb Johannes Holtz:
>> Am 18.10.19 um 17:36 schrieb Jan Kiszka:
>>> On 18.10.19 17:13, Johannes Holtz via Xenomai wrote:
>>>> Am 18.10.19 um 16:57 schrieb Jan Kiszka:
>>>>> On 18.10.19 16:47, Johannes Holtz via Xenomai wrote:
>>>>>> Am 18.10.19 um 16:33 schrieb Jan Kiszka:
>>>>>>> On 18.10.19 16:10, Johannes Holtz via Xenomai wrote:
>>>>>>>> Hi ,
>>>>>>>>
>>>>>>>> I inherited a project where the RTnet stack is loaded manually
>>>>>>>> at boot
>>>>>>>> time. To be specific. Instead of using the rtnet command line
>>>>>>>> programs
>>>>>>>> the rtnet.ko kernel object as well as the necessary drivers are
>>>>>>>> loaded
>>>>>>>> with insmod. The devices are then brought up with rtifconfig and
>>>>>>>> rtroute.
>>>>>>>>
>>>>>>>> The network consist of 7 PCs (all with the same boot time
>>>>>>>> mechanism)
>>>>>>>> with pre-defined IPs. The setup works and initializes in an
>>>>>>>> asynchronized way (like a typical non-RT network).
>>>>>>>>
>>>>>>>> I'm investigating some performance problems and I don't know
>>>>>>>> whats is
>>>>>>>> really going on since no config is used. How is determined which
>>>>>>>> participant is acting as Master and how the slots are distributed?
>>>>>>>> What
>>>>>>>> are the default values?
>>>>>>> You only mentioned rtifconfig and rtroute above, no rtcfg or
>>>>>>> tdmacfg.
>>>>>>> Are you sure RTmac is involved?
>>>>>> This is the list of kernel object that are loaded:
>>>>>>
>>>>>> rtnet.ko
>>>>>>
>>>>>> rtipv4.ko
>>>>>>
>>>>>> rtpacket.ko
>>>>>>
>>>>>> rtudp.ko
>>>>>>
>>>>>> rt_e100e.ko
>>>>>>
>>>>>> rt_8139too.ko
>>>>>>
>>>>>> rt_r8169.ko
>>>>>>
>>>>>> subsequently rtifconfig and rtroute
>>>>>>
>>>>>> Nothing else is done. I don't know id that satisfies for rtmac.
>>>>>> How can
>>>>>> I check?
>>>>> Nope, there would be rtmac.ko, tdma.ko, possibly also rtcfg.ko in
>>>>> addition. And you would see the script calling tdmacfg at least.
>>>> So this means I basically run standard Ethernet with no RT benefits?
>>> The TX and RX path remain deterministic. If your application behave well
>>> and do not generate overload on the network that can cause switches or
>>> recipients to drop packets, all can still be fine.
>>
>> Since this is not synced (not TDMA) it is very well possible that
>> different nodes interfere and datagrams collide.
>>
>> What would be considered overload? Since the PCs are connected via
>> unmanaged switches  I can't detect if they have any problems.
>>
>>>
>>>>> So, nothing of RTmac should prevent packets from being sent or add
>>>>> size
>>>>> restrictions on them.
>>>>>
>>>>> Maybe you can specify your performance problems in more details?
>>>>>
>>>>> Jan
>>>>>
>>>> Well, In this 7 node network one node is receiving datagrams from all
>>>> other 6 containing sensor data. Each datagram has a sequence number to
>>>> recognize packet loss. Each node is running a RT SW with 1ms loop. In
>>>> certain situations the receiver detects that some (between 1 and 20)
>>>> datagrams where lost. In these situations the amount of sent datagrams
>>>> is higher than usual.
>>> What else is on the network? Only those nodes, and they only sent RT
>>> datagrams at a well-defined rate?
>>
>> PC1,PC2 <--->Switch1<--->Switch2<---->PC3-7
>>
>> That's the whole network. Both are un-managed industry switches.
>>
>> PC2 is the receiver.
>>
>>>
>>>> At first I though this might be a throughput issue but even after I
>>>> reduced the payload size drastically I still miss datagrams. The sender
>>>> has no problems sending and my receive code always reads all available
>>>> data from the queue.
>>>>
>>>> /proc/rtnet/stats doesn't show any errors or drops .
>>>>
>>>> I'm working on an example program to test this issue in detail.
>>>>
>>> OK.
>>>
>>> Jan
>>>
>>
> Ok, after a lot of testing I arrived at some sort of insight. I almost
> became crazy after all the kernel segfaults.
>
> I wrote a simple program with a sender and receiver where the sender
> sends a configurable amount of datagrams per 1ms cycle. (attached)
>
> First I noticed that I get problems when allocating a static buffer of
> several kB. So I started to play around with the stack size. If I
> increase the overall stack size I can allocate that buffer but still get
> buffer size problems on the sender side so I also increased the socket
> buffer size and as long as the increased socket buffer is smaller than
> the stack everything is OK and performance increases. If not you get
> said kernel issues. Unsurprisingly in hindsight.
>
> Obviously the size of the datagrams is also important. The smaller they
> are the smaller the buffers can be.
>
> Curious is that I wasn't able to reproduce the case where datagrams
> where missed.
>
> So plugging my insight of the stack and buffer increases into my actual
> program the overall system performance increases. The amount of missed
> datagrams is reduced dramatically. I'm not sure why though.

Did you already read xenomai/kernel/drivers/net/doc/README.pools? Also
xenomai/kernel/drivers/net/stack/include/rtskb.h may shed some light on
how buffer management works with RTnet. If one side runs out of them and
if it cannot push back (via TCP specifically), you will lose messages.

Jan