Big jitter issues every 5 seconds

Big jitter issues every 5 seconds

[Ivan Jiang]

Dear xenomai Oners:

 

       My name is Ivan and our company use xenomai 3.10 patched on Linux 4.19.94 worked on Daul core Cortex A7 800MHz CPU.

       We used this solution to build an EtherCAT Master controller.

       With the isolcpus=1 nohz=on nohz_full=1  rcu_nocbs=1 the user space latency waveforms is like below:

 

       Normally jitter is from -10~10uS every 1mS period,

       With a -30~30uS worst jitter every 200mS which made by our wifi AP enabled on our A7 Core board, this one could be disappeared if we turned off the WIFI.

       But the most unimaginable jitter is up to -55~55uS every 5S, which could not find any reason,  that we closed all the peripherals on the board except the EtherCAT.

       Whatever we turn on/off the L2 Cache or change the size of registry slots and size of system heap / private heap / shared heap.

       Our biggest problems are why WIFI could influent the Jitters and the biggest jitter every 5S is forbidden, we must get rid of them.

       Could you help us to analysis these questions.

       Sincerely we need your help.

 

       Thank you,

       BR.,

       Ivan

 

 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 319147 bytes
Desc: not available
URL: <http://xenomai.org/pipermail/xenomai/attachments/20210821/03a1463a/attachment.png>

Re: Big jitter issues every 5 seconds

[Jan Kiszka]

On 20.08.21 19:14, Ivan Jiang via Xenomai wrote:
> Dear xenomai Oners:
> 
>  
> 
>        My name is Ivan and our company use xenomai 3.10 patched on Linux 4.19.94 worked on Daul core Cortex A7 800MHz CPU.
> 
>        We used this solution to build an EtherCAT Master controller.
> 
>        With the isolcpus=1 nohz=on nohz_full=1  rcu_nocbs=1 the user space latency waveforms is like below:
> 
>  
> 
>        Normally jitter is from -10~10uS every 1mS period,
> 
>        With a -30~30uS worst jitter every 200mS which made by our wifi AP enabled on our A7 Core board, this one could be disappeared if we turned off the WIFI.
> 
>        But the most unimaginable jitter is up to -55~55uS every 5S, which could not find any reason,  that we closed all the peripherals on the board except the EtherCAT.
> 
>        Whatever we turn on/off the L2 Cache or change the size of registry slots and size of system heap / private heap / shared heap.
> 
>        Our biggest problems are why WIFI could influent the Jitters and the biggest jitter every 5S is forbidden, we must get rid of them.
> 
>        Could you help us to analysis these questions.
> 

I'm afraid, 55 µs is already quite good for such a low-end system, even
with core isolation measures.

The last-level caches are typically shared (maybe lstopo can visualize
their hierachy), and the RT workload will likely not fit into core-local
L1 caches. So you cannot avoid eventual spikes already from that
perspective.

In addition, there are likely contention points on the I/O path of your
SoC - only the SoC vendor may be able to tell you where. One you
apparently found already (access to wifi vs. access to Ethernet or
something else on the RT I/O path).

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

Re: Big jitter issues every 5 seconds

[Stéphane Ancelot]

Hi,

I understand that you must check max jitters in your system.

However, I have already designed ethercat master stacks.

By design, having 55us or more jitter when sending the sync frame  is 
not a problem for an Ethercat master system,  since the implementation 
of this bus permits it.

Regards,

S.Ancelot

Le 20/08/2021 à 19:14, Ivan Jiang via Xenomai a écrit :
> Dear xenomai Oners:
>
>   
>
>         My name is Ivan and our company use xenomai 3.10 patched on Linux 4.19.94 worked on Daul core Cortex A7 800MHz CPU.
>
>         We used this solution to build an EtherCAT Master controller.
>
>         With the isolcpus=1 nohz=on nohz_full=1  rcu_nocbs=1 the user space latency waveforms is like below:
>
>   
>
>         Normally jitter is from -10~10uS every 1mS period,
>
>         With a -30~30uS worst jitter every 200mS which made by our wifi AP enabled on our A7 Core board, this one could be disappeared if we turned off the WIFI.
>
>         But the most unimaginable jitter is up to -55~55uS every 5S, which could not find any reason,  that we closed all the peripherals on the board except the EtherCAT.
>
>         Whatever we turn on/off the L2 Cache or change the size of registry slots and size of system heap / private heap / shared heap.
>
>         Our biggest problems are why WIFI could influent the Jitters and the biggest jitter every 5S is forbidden, we must get rid of them.
>
>         Could you help us to analysis these questions.
>
>         Sincerely we need your help.
>
>   
>
>         Thank you,
>
>         BR.,
>
>         Ivan
>
>   
>
>   
>
> -------------- next part --------------
> A non-text attachment was scrubbed...
> Name: image001.png
> Type: image/png
> Size: 319147 bytes
> Desc: not available
> URL: <http://xenomai.org/pipermail/xenomai/attachments/20210821/03a1463a/attachment.png>

Re: Big jitter issues every 5 seconds

[Ivan Jiang]

Hi, S.Ancelot:

	That's true 55us or even more jitter will not influent with EtherCAT Master especially with DC, and we've already done with the robot control or motion control.
	But for some enhanced implementations the control period must set to 125uS or 250uS, and this big jitter must be a big problem.
	Therefore, we are starting to use ftrace to analysis if any unrealized apps in kernel.
	Will you be pleased to give me more suggestion?
	Thank you.

	BR.,
	Ivan.

在 2021/8/23 15:25，“Xenomai 代表 Stéphane Ancelot via Xenomai”<xenomai-bounces@xenomai.org 代表 xenomai@xenomai.org> 写入:

    Hi,

    I understand that you must check max jitters in your system.

    However, I have already designed ethercat master stacks.

    By design, having 55us or more jitter when sending the sync frame  is 
    not a problem for an Ethercat master system,  since the implementation 
    of this bus permits it.

    Regards,

    S.Ancelot

    Le 20/08/2021 à 19:14, Ivan Jiang via Xenomai a écrit :
    > Dear xenomai Oners:
    >
    >   
    >
    >         My name is Ivan and our company use xenomai 3.10 patched on Linux 4.19.94 worked on Daul core Cortex A7 800MHz CPU.
    >
    >         We used this solution to build an EtherCAT Master controller.
    >
    >         With the isolcpus=1 nohz=on nohz_full=1  rcu_nocbs=1 the user space latency waveforms is like below:
    >
    >   
    >
    >         Normally jitter is from -10~10uS every 1mS period,
    >
    >         With a -30~30uS worst jitter every 200mS which made by our wifi AP enabled on our A7 Core board, this one could be disappeared if we turned off the WIFI.
    >
    >         But the most unimaginable jitter is up to -55~55uS every 5S, which could not find any reason,  that we closed all the peripherals on the board except the EtherCAT.
    >
    >         Whatever we turn on/off the L2 Cache or change the size of registry slots and size of system heap / private heap / shared heap.
    >
    >         Our biggest problems are why WIFI could influent the Jitters and the biggest jitter every 5S is forbidden, we must get rid of them.
    >
    >         Could you help us to analysis these questions.
    >
    >         Sincerely we need your help.
    >
    >   
    >
    >         Thank you,
    >
    >         BR.,
    >
    >         Ivan
    >
    >   
    >
    >   
    >
    > -------------- next part --------------
    > A non-text attachment was scrubbed...
    > Name: image001.png
    > Type: image/png
    > Size: 319147 bytes
    > Desc: not available
    > URL: <http://xenomai.org/pipermail/xenomai/attachments/20210821/03a1463a/attachment.png>