[I-PIPE][ARM64] Experimenting on Jetson Xavier

[I-PIPE][ARM64] Experimenting on Jetson Xavier

[Deniz Uğur]

Hello all,

I’m currently trying to run Xenomai on Jetson Xavier (arm64) (base kernel L4T 32.5). I had some successes and fails so far and I wanted to share my progress with you all. Maybe we can find a way to accomplish this experiment.

So far I’ve attempted to build mainline kernel versions 4.19.144, 5.4.72, and 5.7.0. I was only able to boot up 5.7.0 on the Jetson Xavier. I believe this is a good thing because if we can boot up mainline linux then there is a possibility that Xenomai can run as well. On the other hand, NVIDIA supplies us with 4.9.140 and 4.9.210 versions. I have tried to apply 4.9.55 I-pipe patch to 4.9.140 (L4T) and I’ve resolved almost all conflicts. I couldn’t fix entry.S file...

So right now, I’m thinking about two ways to accomplish this experiment. 
	- Bump I-pipe 5.4.72 to 5.7.0 and use it as mainline kernel on the target. This could work because Linux 5.7.0 already works without a problem.
	- Bump I-pipe 4.9.55 to L4T 4.9.140 and use it as NVIDIA’s forked kernel on the target. If we can solve the conflict on “entry.S” file then maybe this could work as well.

What do you guys think about these solutions?
Was this matter already tested by any of you?

I’m definitely open to suggestions as I’m not sure if there are any other barriers on this path to run Xenomai on Jetson Xavier.

Thank you.
Be safe.

-Deniz

Re: [I-PIPE][ARM64] Experimenting on Jetson Xavier

[Philippe Gerum]

Deniz Uğur via Xenomai <xenomai@xenomai.org> writes:

> Hello all,
>
> I’m currently trying to run Xenomai on Jetson Xavier (arm64) (base kernel L4T 32.5). I had some successes and fails so far and I wanted to share my progress with you all. Maybe we can find a way to accomplish this experiment.
>
> So far I’ve attempted to build mainline kernel versions 4.19.144, 5.4.72, and 5.7.0. I was only able to boot up 5.7.0 on the Jetson Xavier. I believe this is a good thing because if we can boot up mainline linux then there is a possibility that Xenomai can run as well. On the other hand, NVIDIA supplies us with 4.9.140 and 4.9.210 versions. I have tried to apply 4.9.55 I-pipe patch to 4.9.140 (L4T) and I’ve resolved almost all conflicts. I couldn’t fix entry.S file...
>
> So right now, I’m thinking about two ways to accomplish this experiment. 
> 	- Bump I-pipe 5.4.72 to 5.7.0 and use it as mainline kernel on the target. This could work because Linux 5.7.0 already works without a problem.
> 	- Bump I-pipe 4.9.55 to L4T 4.9.140 and use it as NVIDIA’s forked kernel on the target. If we can solve the conflict on “entry.S” file then maybe this could work as well.
>
> What do you guys think about these solutions?
> Was this matter already tested by any of you?
>
> I’m definitely open to suggestions as I’m not sure if there are any other barriers on this path to run Xenomai on Jetson Xavier.
>

Since v5.7 boots this SoC, then I guess any later kernel release would
do, including v5.10.

If so, then there may be another option, based not on using Xenomai 3.x
on top of the I-pipe, but on top of Dovetail [1] which is the successor
to the I-pipe (the latter is on its way out).

There is a work in progress rebasing Xenomai 3.x on top of Dovetail for
v5.10 [2]. The x86 and aarch32 ports are in a reasonably good shape at the
moment, aarch64 will be available later next week. I'm confident that
the result should be stable enough to experiment with it.

This said, Dovetail (like the I-pipe) never met the Jetson yet (as far
as I know), so there might be a couple of tweaks required. In the
meantime, you could immediately experiment with the EVL core [5], which
is an evolution of Xenomai's Cobalt core, also running on top of
Dovetail. The EVL core will be at the heart of Xenomai 4.x.

The EVL stable tree is based on v5.10 [3]. Any fix we might have to do
there for the Jetson would likely be part of Dovetail, so this could be
seamlessly inherited by [2].

In short, I would suggest to try booting the Jetson with [3] first,
reporting any issue to this list so we can fix them, then wait for [2]
to be available for arm64 in order to experiment with Xenomai, which
should happen fairly soon.

[1] https://evlproject.org/dovetail/
[2] https://lab.xenomai.org/xenomai-rpm.git/log/?h=for-upstream/dovetail
[3] https://git.evlproject.org/linux-evl.git/log/?h=evl/v5.10
[4] https://evlproject.org/core/build-steps/
[5] https://evlproject.org/core/

-- 
Philippe.

Re: [I-PIPE][ARM64] Experimenting on Jetson Xavier

[Deniz Uğur]

Hi Philippe,

Thank you for your reply. It’s so good to hear that Xenomai is evolving.
Really appreciate the work you guys are doing.

> I would suggest to try booting the Jetson with [3] first,
> reporting any issue to this list so we can fix them

I will try this ASAP and report my results under this thread.
I will be eagerly waiting for the arm64 version of Dovetail :)

> On 21 Feb 2021, at 19:29, Philippe Gerum <rpm@xenomai.org> wrote:
> 
> 
> Deniz Uğur via Xenomai <xenomai@xenomai.org <mailto:xenomai@xenomai.org>> writes:
> 
>> Hello all,
>> 
>> I’m currently trying to run Xenomai on Jetson Xavier (arm64) (base kernel L4T 32.5). I had some successes and fails so far and I wanted to share my progress with you all. Maybe we can find a way to accomplish this experiment.
>> 
>> So far I’ve attempted to build mainline kernel versions 4.19.144, 5.4.72, and 5.7.0. I was only able to boot up 5.7.0 on the Jetson Xavier. I believe this is a good thing because if we can boot up mainline linux then there is a possibility that Xenomai can run as well. On the other hand, NVIDIA supplies us with 4.9.140 and 4.9.210 versions. I have tried to apply 4.9.55 I-pipe patch to 4.9.140 (L4T) and I’ve resolved almost all conflicts. I couldn’t fix entry.S file...
>> 
>> So right now, I’m thinking about two ways to accomplish this experiment. 
>> 	- Bump I-pipe 5.4.72 to 5.7.0 and use it as mainline kernel on the target. This could work because Linux 5.7.0 already works without a problem.
>> 	- Bump I-pipe 4.9.55 to L4T 4.9.140 and use it as NVIDIA’s forked kernel on the target. If we can solve the conflict on “entry.S” file then maybe this could work as well.
>> 
>> What do you guys think about these solutions?
>> Was this matter already tested by any of you?
>> 
>> I’m definitely open to suggestions as I’m not sure if there are any other barriers on this path to run Xenomai on Jetson Xavier.
>> 
> 
> Since v5.7 boots this SoC, then I guess any later kernel release would
> do, including v5.10.
> 
> If so, then there may be another option, based not on using Xenomai 3.x
> on top of the I-pipe, but on top of Dovetail [1] which is the successor
> to the I-pipe (the latter is on its way out).
> 
> There is a work in progress rebasing Xenomai 3.x on top of Dovetail for
> v5.10 [2]. The x86 and aarch32 ports are in a reasonably good shape at the
> moment, aarch64 will be available later next week. I'm confident that
> the result should be stable enough to experiment with it.
> 
> This said, Dovetail (like the I-pipe) never met the Jetson yet (as far
> as I know), so there might be a couple of tweaks required. In the
> meantime, you could immediately experiment with the EVL core [5], which
> is an evolution of Xenomai's Cobalt core, also running on top of
> Dovetail. The EVL core will be at the heart of Xenomai 4.x.
> 
> The EVL stable tree is based on v5.10 [3]. Any fix we might have to do
> there for the Jetson would likely be part of Dovetail, so this could be
> seamlessly inherited by [2].
> 
> In short, I would suggest to try booting the Jetson with [3] first,
> reporting any issue to this list so we can fix them, then wait for [2]
> to be available for arm64 in order to experiment with Xenomai, which
> should happen fairly soon.
> 
> [1] https://evlproject.org/dovetail/ <https://evlproject.org/dovetail/>
> [2] https://lab.xenomai.org/xenomai-rpm.git/log/?h=for-upstream/dovetail <https://lab.xenomai.org/xenomai-rpm.git/log/?h=for-upstream/dovetail>
> [3] https://git.evlproject.org/linux-evl.git/log/?h=evl/v5.10 <https://git.evlproject.org/linux-evl.git/log/?h=evl/v5.10>
> [4] https://evlproject.org/core/build-steps/ <https://evlproject.org/core/build-steps/>
> [5] https://evlproject.org/core/ <https://evlproject.org/core/>
> 
> -- 
> Philippe.

Re: [I-PIPE][ARM64] Experimenting on Jetson Xavier

[Philippe Gerum]

Deniz Uğur <deniz343@gmail.com> writes:

> Hi Philippe,
>
> Thank you for your reply. It’s so good to hear that Xenomai is evolving.
> Really appreciate the work you guys are doing.
>
>  I would suggest to try booting the Jetson with [3] first,
>  reporting any issue to this list so we can fix them
>
> I will try this ASAP and report my results under this thread.
> I will be eagerly waiting for the arm64 version of Dovetail :)
>

Actually, Dovetail/arm64 has been alive and kicking for several years
now, EVL runs it today up to the latest v5.11. What is missing is the
Xenomai port to Dovetail/arm64, which is brewing. I'll keep you posted
when it can be tested (within a week, I believe).

-- 
Philippe.

Re: [I-PIPE][ARM64] Experimenting on Jetson Xavier

[Deniz Uğur]

Hi again,

So I’ve installed the evl/v5.10 on Jetson Xavier and after setting the scaling_governor to performance I got somewhat good results.

I’ve set the isolcpus=0 command line argument but latmus reports that CPU0 is not isolated. I’m not sure if that’s a bug or I’ve missed something.

Other than that I’m getting latencies between +-10us and worst latency is 73us. Even though I’ve ran latmus -t I’m still getting negative latencies. Isn’t that supposed to fix that?

evl test reports OK for everything except for scheduler-quota-accuracy and sched-tp-accuracy (no kernel support)

Are these latencies best I can get or are there anything else I can do to lower/stabilize the latencies?

Thanks.
Deniz.

> On 21 Feb 2021, at 21:14, Philippe Gerum <rpm@xenomai.org> wrote:
> 
> 
> Deniz Uğur <deniz343@gmail.com> writes:
> 
>> Hi Philippe,
>> 
>> Thank you for your reply. It’s so good to hear that Xenomai is evolving.
>> Really appreciate the work you guys are doing.
>> 
>> I would suggest to try booting the Jetson with [3] first,
>> reporting any issue to this list so we can fix them
>> 
>> I will try this ASAP and report my results under this thread.
>> I will be eagerly waiting for the arm64 version of Dovetail :)
>> 
> 
> Actually, Dovetail/arm64 has been alive and kicking for several years
> now, EVL runs it today up to the latest v5.11. What is missing is the
> Xenomai port to Dovetail/arm64, which is brewing. I'll keep you posted
> when it can be tested (within a week, I believe).
> 
> -- 
> Philippe.

Re: [I-PIPE][ARM64] Experimenting on Jetson Xavier

[Deniz Uğur]

Okay so, isolcpus=0 did not isolated the CPU#0 but
isolcpus=1 worked on CPU#1, and that isolated CPU#1
(confirmed by task set -cp 1).

I don’t know why but I got the isolation working so its not a problem which core gets isolated.

After the isolation, the results are remarkable.

No stress
	min: ~0.1us
	avg: ~0.1us
	max: 5-20us

Hectic stress
	min: ~5us
	avg: ~6us
	max: 15-20us (worst is 41us)

> On 22 Feb 2021, at 00:44, Deniz Uğur <deniz343@gmail.com> wrote:
> 
> Hi again,
> 
> So I’ve installed the evl/v5.10 on Jetson Xavier and after setting the scaling_governor to performance I got somewhat good results.
> 
> I’ve set the isolcpus=0 command line argument but latmus reports that CPU0 is not isolated. I’m not sure if that’s a bug or I’ve missed something.
> 
> Other than that I’m getting latencies between +-10us and worst latency is 73us. Even though I’ve ran latmus -t I’m still getting negative latencies. Isn’t that supposed to fix that?
> 
> evl test reports OK for everything except for scheduler-quota-accuracy and sched-tp-accuracy (no kernel support)
> 
> Are these latencies best I can get or are there anything else I can do to lower/stabilize the latencies?
> 
> Thanks.
> Deniz.
> 
>> On 21 Feb 2021, at 21:14, Philippe Gerum <rpm@xenomai.org> wrote:
>> 
>> 
>> Deniz Uğur <deniz343@gmail.com> writes:
>> 
>>> Hi Philippe,
>>> 
>>> Thank you for your reply. It’s so good to hear that Xenomai is evolving.
>>> Really appreciate the work you guys are doing.
>>> 
>>> I would suggest to try booting the Jetson with [3] first,
>>> reporting any issue to this list so we can fix them
>>> 
>>> I will try this ASAP and report my results under this thread.
>>> I will be eagerly waiting for the arm64 version of Dovetail :)
>>> 
>> 
>> Actually, Dovetail/arm64 has been alive and kicking for several years
>> now, EVL runs it today up to the latest v5.11. What is missing is the
>> Xenomai port to Dovetail/arm64, which is brewing. I'll keep you posted
>> when it can be tested (within a week, I believe).
>> 
>> -- 
>> Philippe.
> 

Re: [I-PIPE][ARM64] Experimenting on Jetson Xavier

[Philippe Gerum]

Deniz Uğur <deniz343@gmail.com> writes:

> Hi again,
>
> So I’ve installed the evl/v5.10 on Jetson Xavier and after setting the scaling_governor to performance I got somewhat good results.
>
> I’ve set the isolcpus=0 command line argument but latmus reports that CPU0 is not isolated. I’m not sure if that’s a bug or I’ve missed something.
>

You cannot isolate CPU0, this is a reserved one Linux-wise.

> Other than that I’m getting latencies between +-10us and worst latency
> is 73us.

You may want to run "evl check" for a quick verification of the running
kernel configuration wrt real-time. This reads /proc/config.gz on the
current system by default unless you pass it a .config file, so
CONFIG_IKCONFIG_PROC should be enabled if you want to check a running
kernel.

> Even though I’ve ran latmus -t I’m still getting negative latencies. Isn’t that supposed to fix that?
>

Yep, it should. This can happen with high performance platforms as the
tuner is fairly conservative when determining the typical jitter. What
is the smallest negative value you get?

> evl test reports OK for everything except for scheduler-quota-accuracy and sched-tp-accuracy (no kernel support)
>

You will need CONFIG_EVL_SCHED_QUOTA and CONFIG_EVL_SCHED_TP enabled in
your Kconfig to run these tests.

The exhaustive list of the core options can be found at
https://evlproject.org/core/build-steps/#core-kconfig.

> Are these latencies best I can get or are there anything else I can do to lower/stabilize the latencies?
>

Isolating the test CPU will certainly help. Next, a closer look to the
impact of CPU idling may help, such as measuring the latency with and
without idle=poll on the kernel boot line. As I hinted at earlier, evl
check would detect the kernel options which are known to increase the
latency figures. There might be more SoC-specific tweaks, I don't know
the Jetson enough to figure this out at the moment though.

-- 
Philippe.

Re: [I-PIPE][ARM64] Experimenting on Jetson Xavier

[Deniz Uğur]

Hi,

> I don't know
> the Jetson enough to figure this out at the moment though.

That’s more than okay, the EVL on Jetson works exceptionally good at the moment.
Anything more would be the cherry on top of it :)

I will take a look at CPU idling and “evl check” to be sure that I’ve done everything 
that I could do to lower the latency figures.

> What
> is the smallest negative value you get?

I’m getting around -0.5us after isolating the CPU#1 before that I was seeing around -7us.

> On 22 Feb 2021, at 11:39, Philippe Gerum <rpm@xenomai.org> wrote:
> 
> 
> Deniz Uğur <deniz343@gmail.com> writes:
> 
>> Hi again,
>> 
>> So I’ve installed the evl/v5.10 on Jetson Xavier and after setting the scaling_governor to performance I got somewhat good results.
>> 
>> I’ve set the isolcpus=0 command line argument but latmus reports that CPU0 is not isolated. I’m not sure if that’s a bug or I’ve missed something.
>> 
> 
> You cannot isolate CPU0, this is a reserved one Linux-wise.
> 
>> Other than that I’m getting latencies between +-10us and worst latency
>> is 73us.
> 
> You may want to run "evl check" for a quick verification of the running
> kernel configuration wrt real-time. This reads /proc/config.gz on the
> current system by default unless you pass it a .config file, so
> CONFIG_IKCONFIG_PROC should be enabled if you want to check a running
> kernel.
> 
>> Even though I’ve ran latmus -t I’m still getting negative latencies. Isn’t that supposed to fix that?
>> 
> 
> Yep, it should. This can happen with high performance platforms as the
> tuner is fairly conservative when determining the typical jitter. What
> is the smallest negative value you get?
> 
>> evl test reports OK for everything except for scheduler-quota-accuracy and sched-tp-accuracy (no kernel support)
>> 
> 
> You will need CONFIG_EVL_SCHED_QUOTA and CONFIG_EVL_SCHED_TP enabled in
> your Kconfig to run these tests.
> 
> The exhaustive list of the core options can be found at
> https://evlproject.org/core/build-steps/#core-kconfig.
> 
>> Are these latencies best I can get or are there anything else I can do to lower/stabilize the latencies?
>> 
> 
> Isolating the test CPU will certainly help. Next, a closer look to the
> impact of CPU idling may help, such as measuring the latency with and
> without idle=poll on the kernel boot line. As I hinted at earlier, evl
> check would detect the kernel options which are known to increase the
> latency figures. There might be more SoC-specific tweaks, I don't know
> the Jetson enough to figure this out at the moment though.
> 
> -- 
> Philippe.