Arm64 jitter

Arm64 jitter

[Pierre FICHEUX]

Hi all,

I’ve been using Xenomai-3.1/I-pipe for several months on Pi 3 B/B+ with
mainline kernel (4.19.128 and 4.19.144, arm). I had good results (about 30
µs jitter) with latency + dohell (or latency + hackbench).

Unfortunately, a new arm64 test on  5.4.72 or 5.4.107
(ipipe-core-5.4.107-arm64-1 tag) leads to a  70 µs jitter on the same
board.

I know I-pipe support is recent for arm64 but do you have any idea about
the problem? What are the results on other Arm64 boards such as iMX, or
OMAP ?


Please note, I removed the warnings about APM/ACPI/CPU freq/MIGRATION from
the kernel config.


Thx by advance


Regards

-- 

Pierre FICHEUX -/- CTO Smile ECS, France -\- pierre.ficheux@smile.fr
                             http://www.smile.fr
                             https://smile.eu/fr/offres/embarque-iot
I would love to change the world, but they won't give me the source code

Re: Arm64 jitter

[Pierre FICHEUX]

I mean Arm64 AND 5.x is recent !

Le lun. 29 mars 2021 à 11:20, Pierre FICHEUX <pierre.ficheux@smile.fr> a
écrit :

> Hi all,
>
> I’ve been using Xenomai-3.1/I-pipe for several months on Pi 3 B/B+ with
> mainline kernel (4.19.128 and 4.19.144, arm). I had good results (about 30
> µs jitter) with latency + dohell (or latency + hackbench).
>
> Unfortunately, a new arm64 test on  5.4.72 or 5.4.107
> (ipipe-core-5.4.107-arm64-1 tag) leads to a  70 µs jitter on the same
> board.
>
> I know I-pipe support is recent for arm64 but do you have any idea about
> the problem? What are the results on other Arm64 boards such as iMX, or
> OMAP ?
>
>
> Please note, I removed the warnings about APM/ACPI/CPU freq/MIGRATION from
> the kernel config.
>
>
> Thx by advance
>
>
> Regards
>
> --
>
> Pierre FICHEUX -/- CTO Smile ECS, France -\- pierre.ficheux@smile.fr
>                              http://www.smile.fr
>                              https://smile.eu/fr/offres/embarque-iot
> I would love to change the world, but they won't give me the source code
>
>

-- 

Pierre FICHEUX -/- CTO Smile ECS, France -\- pierre.ficheux@smile.fr
                             http://www.smile.fr
                             https://smile.eu/fr/offres/embarque-iot
I would love to change the world, but they won't give me the source code

Re: Arm64 jitter

[Greg Gallagher]

On Mon, Mar 29, 2021 at 5:22 AM Pierre FICHEUX via Xenomai <
xenomai@xenomai.org> wrote:

> I mean Arm64 AND 5.x is recent !
>
> Le lun. 29 mars 2021 à 11:20, Pierre FICHEUX <pierre.ficheux@smile.fr> a
> écrit :
>
> > Hi all,
> >
> > I’ve been using Xenomai-3.1/I-pipe for several months on Pi 3 B/B+ with
> > mainline kernel (4.19.128 and 4.19.144, arm). I had good results (about
> 30
> > µs jitter) with latency + dohell (or latency + hackbench).
> >
> > Unfortunately, a new arm64 test on  5.4.72 or 5.4.107
> > (ipipe-core-5.4.107-arm64-1 tag) leads to a  70 µs jitter on the same
> > board.
> >
> > I know I-pipe support is recent for arm64 but do you have any idea about
> > the problem? What are the results on other Arm64 boards such as iMX, or
> > OMAP ?
> >
> >
> > Please note, I removed the warnings about APM/ACPI/CPU freq/MIGRATION
> from
> > the kernel config.
> >
> >
> > Thx by advance
> >
> >
> > Regards
> >
> > --
> >
> > Pierre FICHEUX -/- CTO Smile ECS, France -\- pierre.ficheux@smile.fr
> >                              http://www.smile.fr
> >                              https://smile.eu/fr/offres/embarque-iot
> > I would love to change the world, but they won't give me the source code
> >
> >
>
> --
>
> Pierre FICHEUX -/- CTO Smile ECS, France -\- pierre.ficheux@smile.fr
>                              http://www.smile.fr
>                              https://smile.eu/fr/offres/embarque-iot
> I would love to change the world, but they won't give me the source code


I test with Ultra96 board and have seen higher then normal. I’m using
ftrace to track down the source of the latency.

-Greg

Re: Arm64 jitter

[Pierre FICHEUX]

Hi Greg,

Thks for the answer, so you're currently tracking the problem?

Le lun. 29 mars 2021 à 13:43, Greg Gallagher <greg@embeddedgreg.com> a
écrit :

>
>
> On Mon, Mar 29, 2021 at 5:22 AM Pierre FICHEUX via Xenomai <
> xenomai@xenomai.org> wrote:
>
>> I mean Arm64 AND 5.x is recent !
>>
>> Le lun. 29 mars 2021 à 11:20, Pierre FICHEUX <pierre.ficheux@smile.fr> a
>> écrit :
>>
>> > Hi all,
>> >
>> > I’ve been using Xenomai-3.1/I-pipe for several months on Pi 3 B/B+ with
>> > mainline kernel (4.19.128 and 4.19.144, arm). I had good results (about
>> 30
>> > µs jitter) with latency + dohell (or latency + hackbench).
>> >
>> > Unfortunately, a new arm64 test on  5.4.72 or 5.4.107
>> > (ipipe-core-5.4.107-arm64-1 tag) leads to a  70 µs jitter on the same
>> > board.
>> >
>> > I know I-pipe support is recent for arm64 but do you have any idea about
>> > the problem? What are the results on other Arm64 boards such as iMX, or
>> > OMAP ?
>> >
>> >
>> > Please note, I removed the warnings about APM/ACPI/CPU freq/MIGRATION
>> from
>> > the kernel config.
>> >
>> >
>> > Thx by advance
>> >
>> >
>> > Regards
>> >
>> > --
>> >
>> > Pierre FICHEUX -/- CTO Smile ECS, France -\- pierre.ficheux@smile.fr
>> >                              http://www.smile.fr
>> >                              https://smile.eu/fr/offres/embarque-iot
>> > I would love to change the world, but they won't give me the source code
>> >
>> >
>>
>> --
>>
>> Pierre FICHEUX -/- CTO Smile ECS, France -\- pierre.ficheux@smile.fr
>>                              http://www.smile.fr
>>                              https://smile.eu/fr/offres/embarque-iot
>> I would love to change the world, but they won't give me the source code
>
>
> I test with Ultra96 board and have seen higher then normal. I’m using
> ftrace to track down the source of the latency.
>
> -Greg
>


-- 

Pierre FICHEUX -/- CTO Smile ECS, France -\- pierre.ficheux@smile.fr
                             http://www.smile.fr
                             https://smile.eu/fr/offres/embarque-iot
I would love to change the world, but they won't give me the source code

Re: Arm64 jitter

[Greg Gallagher]

On Mon, Mar 29, 2021 at 8:06 AM Pierre FICHEUX <pierre.ficheux@smile.fr>
wrote:

> Hi Greg,
>
> Thks for the answer, so you're currently tracking the problem?
>
> Le lun. 29 mars 2021 à 13:43, Greg Gallagher <greg@embeddedgreg.com> a
> écrit :
>
>>
>>
>> On Mon, Mar 29, 2021 at 5:22 AM Pierre FICHEUX via Xenomai <
>> xenomai@xenomai.org> wrote:
>>
>>> I mean Arm64 AND 5.x is recent !
>>>
>>> Le lun. 29 mars 2021 à 11:20, Pierre FICHEUX <pierre.ficheux@smile.fr> a
>>> écrit :
>>>
>>> > Hi all,
>>> >
>>> > I’ve been using Xenomai-3.1/I-pipe for several months on Pi 3 B/B+ with
>>> > mainline kernel (4.19.128 and 4.19.144, arm). I had good results
>>> (about 30
>>> > µs jitter) with latency + dohell (or latency + hackbench).
>>> >
>>> > Unfortunately, a new arm64 test on  5.4.72 or 5.4.107
>>> > (ipipe-core-5.4.107-arm64-1 tag) leads to a  70 µs jitter on the same
>>> > board.
>>> >
>>> > I know I-pipe support is recent for arm64 but do you have any idea
>>> about
>>> > the problem? What are the results on other Arm64 boards such as iMX, or
>>> > OMAP ?
>>> >
>>> >
>>> > Please note, I removed the warnings about APM/ACPI/CPU freq/MIGRATION
>>> from
>>> > the kernel config.
>>> >
>>> >
>>> > Thx by advance
>>> >
>>> >
>>> > Regards
>>> >
>>> > --
>>> >
>>> > Pierre FICHEUX -/- CTO Smile ECS, France -\- pierre.ficheux@smile.fr
>>> >                              http://www.smile.fr
>>> >                              https://smile.eu/fr/offres/embarque-iot
>>> > I would love to change the world, but they won't give me the source
>>> code
>>> >
>>> >
>>>
>>> --
>>>
>>> Pierre FICHEUX -/- CTO Smile ECS, France -\- pierre.ficheux@smile.fr
>>>                              http://www.smile.fr
>>>                              https://smile.eu/fr/offres/embarque-iot
>>> I would love to change the world, but they won't give me the source code
>>
>>
>> I test with Ultra96 board and have seen higher then normal. I’m using
>> ftrace to track down the source of the latency.
>>
>> -Greg
>>
>
>
> --
>
> Pierre FICHEUX -/- CTO Smile ECS, France -\- pierre.ficheux@smile.fr
>                              http://www.smile.fr
>                              https://smile.eu/fr/offres/embarque-iot
> I would love to change the world, but they won't give me the source code
>
> Hi Pierre,
  I have a hang to look at in arm64 qemu and then I’m getting back to
looking at what is causing the latency. The ultra96 is showing higher then
normal latency.
   I’m hoping to have some answers by the end of the week. A updated 4.19
release is on its way as well, but Im seeing good latency numbers with that
release.

-Greg

>

Re: Arm64 jitter

[Pierre FICHEUX]

Thx a lot !

Le lun. 29 mars 2021 à 15:17, Greg Gallagher <greg@embeddedgreg.com> a
écrit :

>
>
> On Mon, Mar 29, 2021 at 8:06 AM Pierre FICHEUX <pierre.ficheux@smile.fr>
> wrote:
>
>> Hi Greg,
>>
>> Thks for the answer, so you're currently tracking the problem?
>>
>> Le lun. 29 mars 2021 à 13:43, Greg Gallagher <greg@embeddedgreg.com> a
>> écrit :
>>
>>>
>>>
>>> On Mon, Mar 29, 2021 at 5:22 AM Pierre FICHEUX via Xenomai <
>>> xenomai@xenomai.org> wrote:
>>>
>>>> I mean Arm64 AND 5.x is recent !
>>>>
>>>> Le lun. 29 mars 2021 à 11:20, Pierre FICHEUX <pierre.ficheux@smile.fr>
>>>> a
>>>> écrit :
>>>>
>>>> > Hi all,
>>>> >
>>>> > I’ve been using Xenomai-3.1/I-pipe for several months on Pi 3 B/B+
>>>> with
>>>> > mainline kernel (4.19.128 and 4.19.144, arm). I had good results
>>>> (about 30
>>>> > µs jitter) with latency + dohell (or latency + hackbench).
>>>> >
>>>> > Unfortunately, a new arm64 test on  5.4.72 or 5.4.107
>>>> > (ipipe-core-5.4.107-arm64-1 tag) leads to a  70 µs jitter on the same
>>>> > board.
>>>> >
>>>> > I know I-pipe support is recent for arm64 but do you have any idea
>>>> about
>>>> > the problem? What are the results on other Arm64 boards such as iMX,
>>>> or
>>>> > OMAP ?
>>>> >
>>>> >
>>>> > Please note, I removed the warnings about APM/ACPI/CPU freq/MIGRATION
>>>> from
>>>> > the kernel config.
>>>> >
>>>> >
>>>> > Thx by advance
>>>> >
>>>> >
>>>> > Regards
>>>> >
>>>> > --
>>>> >
>>>> > Pierre FICHEUX -/- CTO Smile ECS, France -\- pierre.ficheux@smile.fr
>>>> >                              http://www.smile.fr
>>>> >                              https://smile.eu/fr/offres/embarque-iot
>>>> > I would love to change the world, but they won't give me the source
>>>> code
>>>> >
>>>> >
>>>>
>>>> --
>>>>
>>>> Pierre FICHEUX -/- CTO Smile ECS, France -\- pierre.ficheux@smile.fr
>>>>                              http://www.smile.fr
>>>>                              https://smile.eu/fr/offres/embarque-iot
>>>> I would love to change the world, but they won't give me the source code
>>>
>>>
>>> I test with Ultra96 board and have seen higher then normal. I’m using
>>> ftrace to track down the source of the latency.
>>>
>>> -Greg
>>>
>>
>>
>> --
>>
>> Pierre FICHEUX -/- CTO Smile ECS, France -\- pierre.ficheux@smile.fr
>>                              http://www.smile.fr
>>                              https://smile.eu/fr/offres/embarque-iot
>> I would love to change the world, but they won't give me the source code
>>
>> Hi Pierre,
>   I have a hang to look at in arm64 qemu and then I’m getting back to
> looking at what is causing the latency. The ultra96 is showing higher then
> normal latency.
>    I’m hoping to have some answers by the end of the week. A updated 4.19
> release is on its way as well, but Im seeing good latency numbers with that
> release.
>
> -Greg
>
>>

-- 

Pierre FICHEUX -/- CTO Smile ECS, France -\- pierre.ficheux@smile.fr
                             http://www.smile.fr
                             https://smile.eu/fr/offres/embarque-iot
I would love to change the world, but they won't give me the source code

Re: Arm64 jitter

[Greg Gallagher]

On Mon, Mar 29, 2021 at 9:35 AM Pierre FICHEUX <pierre.ficheux@smile.fr>
wrote:

> Thx a lot !
>
> Le lun. 29 mars 2021 à 15:17, Greg Gallagher <greg@embeddedgreg.com> a
> écrit :
>
>>
>>
>> On Mon, Mar 29, 2021 at 8:06 AM Pierre FICHEUX <pierre.ficheux@smile.fr>
>> wrote:
>>
>>> Hi Greg,
>>>
>>> Thks for the answer, so you're currently tracking the problem?
>>>
>>> Le lun. 29 mars 2021 à 13:43, Greg Gallagher <greg@embeddedgreg.com> a
>>> écrit :
>>>
>>>>
>>>>
>>>> On Mon, Mar 29, 2021 at 5:22 AM Pierre FICHEUX via Xenomai <
>>>> xenomai@xenomai.org> wrote:
>>>>
>>>>> I mean Arm64 AND 5.x is recent !
>>>>>
>>>>> Le lun. 29 mars 2021 à 11:20, Pierre FICHEUX <pierre.ficheux@smile.fr>
>>>>> a
>>>>> écrit :
>>>>>
>>>>> > Hi all,
>>>>> >
>>>>> > I’ve been using Xenomai-3.1/I-pipe for several months on Pi 3 B/B+
>>>>> with
>>>>> > mainline kernel (4.19.128 and 4.19.144, arm). I had good results
>>>>> (about 30
>>>>> > µs jitter) with latency + dohell (or latency + hackbench).
>>>>> >
>>>>> > Unfortunately, a new arm64 test on  5.4.72 or 5.4.107
>>>>> > (ipipe-core-5.4.107-arm64-1 tag) leads to a  70 µs jitter on the same
>>>>> > board.
>>>>> >
>>>>> > I know I-pipe support is recent for arm64 but do you have any idea
>>>>> about
>>>>> > the problem? What are the results on other Arm64 boards such as iMX,
>>>>> or
>>>>> > OMAP ?
>>>>> >
>>>>> >
>>>>> > Please note, I removed the warnings about APM/ACPI/CPU
>>>>> freq/MIGRATION from
>>>>> > the kernel config.
>>>>> >
>>>>> >
>>>>> > Thx by advance
>>>>> >
>>>>> >
>>>>> > Regards
>>>>> >
>>>>> > --
>>>>> >
>>>>> > Pierre FICHEUX -/- CTO Smile ECS, France -\- pierre.ficheux@smile.fr
>>>>> >                              http://www.smile.fr
>>>>> >                              https://smile.eu/fr/offres/embarque-iot
>>>>> > I would love to change the world, but they won't give me the source
>>>>> code
>>>>> >
>>>>> >
>>>>>
>>>>> --
>>>>>
>>>>> Pierre FICHEUX -/- CTO Smile ECS, France -\- pierre.ficheux@smile.fr
>>>>>                              http://www.smile.fr
>>>>>                              https://smile.eu/fr/offres/embarque-iot
>>>>> I would love to change the world, but they won't give me the source
>>>>> code
>>>>
>>>>
>>>> I test with Ultra96 board and have seen higher then normal. I’m using
>>>> ftrace to track down the source of the latency.
>>>>
>>>> -Greg
>>>>
>>>
>>>
>>> --
>>>
>>> Pierre FICHEUX -/- CTO Smile ECS, France -\- pierre.ficheux@smile.fr
>>>                              http://www.smile.fr
>>>                              https://smile.eu/fr/offres/embarque-iot
>>> I would love to change the world, but they won't give me the source code
>>>
>>> Hi Pierre,
>>   I have a hang to look at in arm64 qemu and then I’m getting back to
>> looking at what is causing the latency. The ultra96 is showing higher then
>> normal latency.
>>    I’m hoping to have some answers by the end of the week. A updated 4.19
>> release is on its way as well, but Im seeing good latency numbers with that
>> release.
>>
>> -Greg
>>
>>>
>
> --
>
> Pierre FICHEUX -/- CTO Smile ECS, France -\- pierre.ficheux@smile.fr
>                              http://www.smile.fr
>                              https://smile.eu/fr/offres/embarque-iot
> I would love to change the world, but they won't give me the source code
>
>
Looks like this is affecting 4.19/5.4 arm and arm64.  Still investigating,
hopefully I'll have a root cause shortly.

-Greg

Re: Arm64 jitter

[Robert Berger]

Hi,

My comments are in-line:

On 02/04/2021 07:54, Greg Gallagher via Xenomai wrote:
>>
>>
> Looks like this is affecting 4.19/5.4 arm and arm64.  Still investigating,
> hopefully I'll have a root cause shortly.

With cyclictest, stress-ng and a loadavg of over 12 I see a jitter of 
approx. 55 usec on an arm32: i.mx6 quad.

Did not yet start testing on arm64.

> 
> -Greg
> 

Regards,

Robert

Re: Arm64 jitter

[Greg Gallagher]

On Sun, Apr 4, 2021 at 4:49 PM Robert Berger <xenomai.list@gmail.com> wrote:

> Hi,
>
> My comments are in-line:
>
> On 02/04/2021 07:54, Greg Gallagher via Xenomai wrote:
> >>
> >>
> > Looks like this is affecting 4.19/5.4 arm and arm64.  Still
> investigating,
> > hopefully I'll have a root cause shortly.
>
> With cyclictest, stress-ng and a loadavg of over 12 I see a jitter of
> approx. 55 usec on an arm32: i.mx6 quad.
>
> Did not yet start testing on arm64.
>
> >
> > -Greg
> >
>
> Regards,
>
> Robert
>
Hi,

I think I've tracked down a couple issues that could be causing this.  I'll
hopefully have an update this week with a fix.

Thanks for testing,

Greg