[Xenomai-help] An answer to Steven's questions.

[Xenomai-help] An answer to Steven's questions.

[Gilles Chanteperdrix]

Hi Steven,

what matters in an RTOS, Xenomai being no exception, is the latency.
Actually, there are three latencies, the interrupt latency, the
kernel-space scheduling latency, and the user-space scheduling latency.

The true nature of these latencies for Xenomai is complex. It is
probably a mix between the longest interrupt masking section in both
Xenomai and Linux code and the indirect effect of cache, spin-locking on
SMP, TLB, BTB and of other hardware effects. Trying to compute this
latency mathematically is hopeless.

For this reason, Xenomai comes with a tool called "latency", which
allows you to measure these latencies given a load applied on the
system. For this tool to give meaningful results, you should apply a
realistic load on the system, to be sure that most paths in Xenomai and
Linux code are taken so that their masking section time, or their effect
on cache, TLB, BTB, etc... if any, is taken into account.

So, you should not focus on whether this or that part of Xenomai or
Linux code is preemptible or not, you would miss the big picture. What
you have to do, is run the latency test given a proper load applied on
the system. Then you will get a number, and you can rely on this number
(maybe adding some safety margin) as being the latency of Xenomai on
your system. Needless to say, Xenomai syscalls and process migrations
are already done by the printing thread in the latency test, so, there
is no need to try them in a separate load program.

Regards.

-- 
					    Gilles.

Re: [Xenomai-help] An answer to Steven's questions.

[Steven Seeger]

The problem is that latency on Xenomai (even with kernel based  
threads) is terrible on our board compared to RTAI and it's causing us  
issues. I'm starting to think that Xenomai on Geode is not appropriate  
for our application, but we have to salvage the work that's already  
been done. At this point I'm simply looking to minimize the problem.

We are seeing jitters of upwards of 70us when we have a 125us and a  
500us task. That's a huge amount and we can't deal with it. Hopefully  
the problem will be something else. It would be nice to have my other  
questions answered, though.

Steven

On Apr 22, 2009, at 2:44 PM, Gilles Chanteperdrix wrote:

>
> Hi Steven,
>
> what matters in an RTOS, Xenomai being no exception, is the latency.
> Actually, there are three latencies, the interrupt latency, the
> kernel-space scheduling latency, and the user-space scheduling  
> latency.
>
> The true nature of these latencies for Xenomai is complex. It is
> probably a mix between the longest interrupt masking section in both
> Xenomai and Linux code and the indirect effect of cache, spin- 
> locking on
> SMP, TLB, BTB and of other hardware effects. Trying to compute this
> latency mathematically is hopeless.
>
> For this reason, Xenomai comes with a tool called "latency", which
> allows you to measure these latencies given a load applied on the
> system. For this tool to give meaningful results, you should apply a
> realistic load on the system, to be sure that most paths in Xenomai  
> and
> Linux code are taken so that their masking section time, or their  
> effect
> on cache, TLB, BTB, etc... if any, is taken into account.
>
> So, you should not focus on whether this or that part of Xenomai or
> Linux code is preemptible or not, you would miss the big picture. What
> you have to do, is run the latency test given a proper load applied on
> the system. Then you will get a number, and you can rely on this  
> number
> (maybe adding some safety margin) as being the latency of Xenomai on
> your system. Needless to say, Xenomai syscalls and process migrations
> are already done by the printing thread in the latency test, so, there
> is no need to try them in a separate load program.
>
> Regards.
>
> -- 
> 					    Gilles.

Re: [Xenomai-help] An answer to Steven's questions.

[Gilles Chanteperdrix]

Steven Seeger wrote:
> The problem is that latency on Xenomai (even with kernel based  
> threads) is terrible on our board compared to RTAI and it's causing us  
> issues. I'm starting to think that Xenomai on Geode is not appropriate  
> for our application, but we have to salvage the work that's already  
> been done. At this point I'm simply looking to minimize the problem.
> 
> We are seeing jitters of upwards of 70us when we have a 125us and a  
> 500us task. That's a huge amount and we can't deal with it. Hopefully  
> the problem will be something else. It would be nice to have my other  
> questions answered, though.

You are repeating yourself, so, I am going to repeat myself. Just run
the latency test, if it does not have the same latency, then your
problem is in your application. It it has the same latency, then there
is a problem with Xenomai, and you should enable the I-pipe tracer and
send us the trace.

-- 
					    Gilles.

Re: [Xenomai-help] An answer to Steven's questions.

[Gilles Chanteperdrix]

Steven Seeger wrote:
> The problem is that latency on Xenomai (even with kernel based  
> threads) is terrible on our board compared to RTAI and it's causing us  
> issues. I'm starting to think that Xenomai on Geode is not appropriate  
> for our application, but we have to salvage the work that's already  
> been done. At this point I'm simply looking to minimize the problem.
> 
> We are seeing jitters of upwards of 70us when we have a 125us and a  
> 500us task. That's a huge amount and we can't deal with it. Hopefully  
> the problem will be something else. It would be nice to have my other  
> questions answered, though.

To give you some figures, the Geode you sent me has a latency around
40us under full load.

-- 
					    Gilles.

Re: [Xenomai-help] An answer to Steven's questions.

[Steven Seeger]

> You are repeating yourself, so, I am going to repeat myself. Just run
> the latency test, if it does not have the same latency, then your
> problem is in your application. It it has the same latency, then there
> is a problem with Xenomai, and you should enable the I-pipe tracer and
> send us the trace.

I see what you are saying. I believe our application has the same  
latency but I am not there to see the timing analysis, so I will have  
to take other peoples' word.

Steven

Re: [Xenomai-help] An answer to Steven's questions.

[Steven Seeger]

> To give you some figures, the Geode you sent me has a latency around
> 40us under full load.

That is the "new geode." The older GX1 is much worse.

Steven

Re: [Xenomai-help] An answer to Steven's questions.

[Philippe Gerum]

On Wed, 2009-04-22 at 17:17 -0400, Steven Seeger wrote:
> > You are repeating yourself, so, I am going to repeat myself. Just run
> > the latency test, if it does not have the same latency, then your
> > problem is in your application. It it has the same latency, then there
> > is a problem with Xenomai, and you should enable the I-pipe tracer and
> > send us the trace.
> 
> I see what you are saying. I believe our application has the same  
> latency but I am not there to see the timing analysis, so I will have  
> to take other peoples' word.

Sorry for being desperately clueless, and very probably hopelessly lost
in space, but I still have a dumb question for the guy in charge of this
problem, whoever he is (since you don't seem to be the one here):

Why can't we just have the output of a simple latency test on the
problematic GX1, using the bundled program we provide for that, and
copy&paste this text into a mail message, eventually directed to _this_
list?

We do _not_ know what the heck your application does, whether it has
problems or not. But, we could find out whether Xenomai has problems or
not on your particular hardware. You already confirmed that your
RTAI-based implementation differed totally from your Xenomai-based one,
so comparing won't help either. Btw, do you really think that other
users would settle for 3 digit latency figures on x86? Obviously not, so
what you see must be pathological. Therefore, the only question worth
discussing is: who is causing this?

Do you think we might have some clue regarding the way Xenomai works,
then answer the above question in case this is a Xenomai issue? If so,
then give us some actual data to examine the situation. Otherwise, it's
a total loss of time for everybody.

===================================================
EXECUTIVE SUMMARY to send to the boss in charge:
1) run /usr/xenomai/bin/latency for an hour with
   some process load (hackbench) and interrupt load.
2) copy&paste output to new e-mail
3) hit Send.
===================================================

We will do our best after this. No kidding. C'mon!

> 
> Steven
> 
> 
> _______________________________________________
> Xenomai-help mailing list
> Xenomai-help@domain.hid
> https://mail.gna.org/listinfo/xenomai-help
-- 
Philippe.

Re: [Xenomai-core] [Xenomai-help] An answer to Steven's questions.

[Steven Seeger]

> We do _not_ know what the heck your application does, whether it has
> problems or not. But, we could find out whether Xenomai has problems  
> or
> not on your particular hardware. You already confirmed that your
> RTAI-based implementation differed totally from your Xenomai-based  
> one,
> so comparing won't help either. Btw, do you really think that other
> users would settle for 3 digit latency figures on x86? Obviously  
> not, so
> what you see must be pathological. Therefore, the only question worth
> discussing is: who is causing this?

We have sent output from latency before, and the ones I have seen I  
think had a max of like 85us. We never ran it for an hour. We can do  
that with both latency -t1 and -t2.

It is probably our app that has problems, but I am trying to  
understand how the interactions work. The highest priority thread in  
our program does nothing but flip a bit and we're seeing that do weird  
things on a logic analyzer.

Nice angry email! :)

Steven

Re: [Xenomai-core] [Xenomai-help] An answer to Steven's questions.

[Philippe Gerum]

On Wed, 2009-04-22 at 19:08 -0400, Steven Seeger wrote:
> > We do _not_ know what the heck your application does, whether it has
> > problems or not. But, we could find out whether Xenomai has problems  
> > or
> > not on your particular hardware. You already confirmed that your
> > RTAI-based implementation differed totally from your Xenomai-based  
> > one,
> > so comparing won't help either. Btw, do you really think that other
> > users would settle for 3 digit latency figures on x86? Obviously  
> > not, so
> > what you see must be pathological. Therefore, the only question worth
> > discussing is: who is causing this?
> 
> We have sent output from latency before, and the ones I have seen I  
> think had a max of like 85us. We never ran it for an hour. We can do  
> that with both latency -t1 and -t2.

The latency figures we have for the GX1 were obtained when chasing the
TSC issue, so we are not going to base any reasoning on them. We need
fresh values from your very latest setup in use.

But yes, having -t0, -t1 and -t2 would make sense, because comparing
them would help understanding where the weak point might be, if the
issue is Xenomai related. If it is not, then at least we would not chase
wild gooses uselessly.

> 
> It is probably our app that has problems, but I am trying to  
> understand how the interactions work.

I understand that, but this is not the way to go; as Gilles pointed out,
speculating on various unrelated/obscure issues would make us miss the
big picture. We first have to find a stable ground to build our
reasoning on, that is decoupled from any application artefacts. Standard
latency tests do provide this information.

>  The highest priority thread in  
> our program does nothing but flip a bit and we're seeing that do weird  
> things on a logic analyzer.

This is step #2: as soon as we may rule out a general platform issue, we
may introduce additional application monitoring such as the I-pipe
tracer in the picture.

> 
> Nice angry email! :)
> 

I was not angry, just surprised that we already spent that much time on
discussing those issues in a scatterbrain way instead of following a
simple and consistent approach. It's easy to find out when I'm writing
angry e-mails actually, I'm using "damnit" as a punctuation mark.

> Steven
> 
-- 
Philippe.