[Xenomai-help] About HARD real time

[Xenomai-help] About HARD real time

[Antoine Nourry]

Hi,
I just wonder :
Why can we read everywhere that PREEMPT-RT patch offer HARD real time 
while in other papers we can read that it cannot ensure determinism  ?
(because it only seems to permit low latencies but not constant 
execution times, by the way why ? Is it because this patch still uses 
undeterministic parts of the kernel ? )
Considering this, how Xenomai 3 will be able to rely on PREEMPT-RT as 
real time enabling technology, allowing user to bypass ADEOS ?

Thanks for your work,
Antoine

Re: [Xenomai-help] About HARD real time

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 3157 bytes --]

...to answer this almost forgotten question:

Antoine Nourry wrote:
> Hi,
> I just wonder :
> Why can we read everywhere that PREEMPT-RT patch offer HARD real time 
> while in other papers we can read that it cannot ensure determinism  ?

It depends on which method you use to verify that some hard real-time
requirement is fulfilled. In practice, measurement under more or less
reasonable chosen load is the most common method for real-time systems
of a complexity like PREEMPT-RT or even much simpler ones. Only
dedicated and small systems, either with a simple RTOS or even OS-less,
are entirely formally verified today, using also formal models for their
hardware.

There is also a hybrid approach in between those extremes (specifically
useful if the hardware platform is too complex for modeling - like x86):
first develop models of your worst-case execution paths based on the
source code, taking also application parameters into account, then
trigger and measure them on the target hardware. But this only works if
the complexity of the analyzed real-time system is still manageable,
either by a human alone or with the help of tools.

PREEMPT-RT currently only fulfills this requirement if you apply
probabilities ("It is sufficiently unlikely that all these n nested
locks are contended at the same time with the maximum delay, thus we can
ignore this case.") AND very carefully design your application. This
applies to quite a few but not all real-time use cases.

> (because it only seems to permit low latencies but not constant 
> execution times, by the way why ? Is it because this patch still uses 
> undeterministic parts of the kernel ? )

By design not, but bugs/regressions can always exist, just as they have
popped up in Xenomai & Adeos/I-pipe before. When people measure extreme
latencies, they typically trigger such cases (or run their tests on
unsuited commodity hardware). Sadly, the actual reason for such
latencies are too rarely analyzed then, thus only the statement "X does
not provide real-time" makes it into publications.

As said above, the most challenging issue of PREEMPT-RT is that you have
to carefully design both drivers and applications to not leave the path
that can provide sufficiently reliable latencies. Just as with Xenomai,
only a subset of the kernel can be used for time critical code paths.
Moreover, there is still a noticeably higher risk to drive the system
into excessive latencies with non-RT load than you have with a
co-scheduled approach.

> Considering this, how Xenomai 3 will be able to rely on PREEMPT-RT as 
> real time enabling technology, allowing user to bypass ADEOS ?

The current roadmap is to let Xenomai 3 come in two flavors: one based
on PREEMPT-RT (Xenomai/Solo), the other (Xenomai/Duo) using Adeos/I-pipe
and a second scheduler like Xenomai 2 does, but supporting much less
legacy than the latter (no kernel 2.4, no in-kernel RT applications
etc.). So depending on your requirements and PREEMPT-RT's progress,
you'll be able to chose the fitting platform or even switch between them
without too much effort.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 257 bytes --]

Re: [Xenomai-help] About HARD real time... and USB

[Antoine Nourry]

> The current roadmap is to let Xenomai 3 come in two flavors: one based
> on PREEMPT-RT (Xenomai/Solo), the other (Xenomai/Duo) using Adeos/I-pipe
> and a second scheduler like Xenomai 2 does, but supporting much less
> legacy than the latter (no kernel 2.4, no in-kernel RT applications
> etc.). So depending on your requirements and PREEMPT-RT's progress,
> you'll be able to chose the fitting platform or even switch between them
> without too much effort.
>
> Jan
>
>   
Thanks again Jan.

Another point; for RT USB developers, do you think we could use Linux 
native USB stack with PREEMPT_RT in the same way as Xenomai / USB4RT ?

I'm pleased with this last solution but the USB4RT bug (need to send a 
command twice) is a real problem for performances (i'd like to increase 
my acquisition rate within a fixed cycle). My only approach to real time 
for the moment is USB driver, and in the Open world i cannot find what 
could be the "right" or "best" solution...

Re: [Xenomai-help] About HARD real time

[Philippe Gerum]

On Thu, 2009-05-21 at 18:22 +0200, Antoine Nourry wrote: 
> Hi,
> I just wonder :
> Why can we read everywhere that PREEMPT-RT patch offer HARD real time 
> while in other papers we can read that it cannot ensure determinism  ?
> (because it only seems to permit low latencies but not constant 
> execution times, by the way why ? Is it because this patch still uses 
> undeterministic parts of the kernel ? )

Nobody is having constant execution times anyway, native or co-kernel:
CPUs are stuffed with caches and all sort of artefacts that come with
them, particularly with the current trend of multi-core architectures.
Part of the system wants throughput, many interfaces and lots of drivers
(the plain non-RT linux activities) and the other part wants absolute
priority via immediate preemption (the rt layer), all competing for the
same hardware resources at some point, none caring for the neighbour.
E.g.

- run a latency test, with some load. Fire one test with a 10Khz period,
and another one later using a 100hz period. Maybe counter-intuitively to
some, you will get lower worst-case latency with the former, because the
system will have less time/opportunity to evict the latency test from
the caches due to the higher sampling frequency.

- run a latency test while a process is causing lots of page table entry
flushes (LTP's mm stress tests do this). If your architecture has a
software-assisted MMU management, like we may find on powerpc, the
relevant code in a classic kernel is likely to operate with interrupts
disabled while flushing an entire PTE range. If your platform is
SMP-enabled, that code will probably have to exclude other CPUs while
running as well. Enable HIGHMEM to use > 1Gb memory, and you may well
end up doing that for 1024 PTE in a row. This may induce a latency
penalty > 400 us, depending on the hw. And having > 1Gb memory available
on modern embedded hardware is not that uncommon, especially with
network processors.

Regardless of -rt, co-kernel, whatever, we all have those problems with
respect to execution time, and also unbounded latency issues, that we
ought to take care of on a case-by-case basis. But even if you carefully
prevent unbounded latencies by fixing the vanilla kernel code, the PTE
range flush issue just described does still introduce uncertainty with
respect to the execution time; this jitter is only made acceptable
because we bounded it to a known maximum value (e.g. < 15 us on a dual
core 8641D). Which we measured... by observation, because a GPOS+RTOS
mix is currently too complex for any formal verification.

Therefore, the fact that some acceptable results with respect to latency
were obtained on a given sw/hw combo does not mean anything regarding a
significantly different combination, regardless of the RT approach you
have been using to collect them. Typically:

- focus on x86, enable the wrong set of option pertaining to the ACPI
kitchen-sink, and/or face SMI events, and/or run an X-server that plays
silly games with the interrupt state from userland, and both -rt and
co-kernels will have serious latency issues.

- leave the common x86 world to run -rt on embeddded platforms, enable
non-mainstream drivers, or enable HIGHMEM on platforms with
software-assisted MMUs, and you may get some reasons to get busy fixing
latency spots in the future (not to speak of the ARM VIVT cache flushing
issue causing the latency figures to skyrocket when switching task
contexts on classic kernels -- guess why Gilles enabled FCSE is Adeos
patches recently).

This makes assertions like "<blah> is able to sustain <blip> us
worst-case latency. period." without specifying the exact
characteristics of the sw and hw involved in the test, a bit suspicious,
to say the least (read: this is reaching 97 over 100 on the bs'o'meter).

> Considering this, how Xenomai 3 will be able to rely on PREEMPT-RT as 
> real time enabling technology, allowing user to bypass ADEOS ?
> 
> 

If one admits that both native preemption and dual kernel approaches
have their own set of exclusive pros and cons, while sharing the same
initial challenges brought by a mixed GPOS+RTOS environment, that
question is a no-brainer: Xenomai should be able to rely on -rt enabled
systems where no unacceptable latencies are observed regardless of the
workload. This is a case-by-case matter, which depends on a given kernel
release running on a given hw combo with a given software set enabled.

For that reason, x3 will allow users to base their RT system over -rt or
the I-pipe, as they see fit, as the requirements mandate. When running
over -rt, Xenomai will only be in charge of providing real-time APIs;
when running over the I-pipe, it will also provide the real-time bedrock
that keeps latencies low, like it already does with earlier versions.

As Paul E. McKenney puts it:
"This is not to say that hard real time is undefined or useless.
Instead, 'hard real time' is the start of a conversation rather than a
complete requirement."
http://www.linuxjournal.com/article/9361

> 
> Thanks for your work,
> Antoine
> 
> 
> _______________________________________________
> Xenomai-help mailing list
> Xenomai-help@domain.hid
> https://mail.gna.org/listinfo/xenomai-help
-- 
Philippe.

Re: [Xenomai-help] About HARD real time... and USB

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 2184 bytes --]

Antoine Nourry wrote:
> 
>> The current roadmap is to let Xenomai 3 come in two flavors: one based
>> on PREEMPT-RT (Xenomai/Solo), the other (Xenomai/Duo) using Adeos/I-pipe
>> and a second scheduler like Xenomai 2 does, but supporting much less
>> legacy than the latter (no kernel 2.4, no in-kernel RT applications
>> etc.). So depending on your requirements and PREEMPT-RT's progress,
>> you'll be able to chose the fitting platform or even switch between them
>> without too much effort.
>>
>> Jan
>>
>>   
> Thanks again Jan.
> 
> Another point; for RT USB developers, do you think we could use Linux
> native USB stack with PREEMPT_RT in the same way as Xenomai / USB4RT ?

I haven't reviewed the code path of a typical urb in mainline recently,
but I bet the usual two issues exist:

 - Shared (soft-)IRQ and/or workqueue threads with uncritical code
   (either other USB drivers or even totally unrelated kernel
   subsystems). In order to achieve reasonable latencies, you need to
   boost their prios, but that can quickly cause priority inversion,
   thus other latencies.

 - No resource isolation, ie. dynamic buffer allocation which can hit
   temporary shortages or contentions, raising latencies or even
   letting some requests fail.

Moreover you may face troubles as mainline USB is (reasonably) concerned
about hotplug, thus may perform management jobs while you try to acquire
some data at high frequencies and react on the data in real-time.

> 
> I'm pleased with this last solution but the USB4RT bug (need to send a
> command twice) is a real problem for performances (i'd like to increase
> my acquisition rate within a fixed cycle). My only approach to real time
> for the moment is USB driver, and in the Open world i cannot find what
> could be the "right" or "best" solution...

It's a pity, this bug now exists for, hmm, 3 years? But as long as no
one feels the need to dive into it, understand and fix it, this won't
change. Maybe one could even analyze this inside QEMU/KVM, also looking
at the virtual host controller state when the problems shows up. But it
still takes someone to /do/ this.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 257 bytes --]

Re: [Xenomai-help] About HARD real time

[Martin Shepherd]

On Tue, 2 Jun 2009, Philippe Gerum wrote:
> For that reason, x3 will allow users to base their RT system over -rt or
> the I-pipe, as they see fit, as the requirements mandate. When running
> over -rt, Xenomai will only be in charge of providing real-time APIs;
> when running over the I-pipe, it will also provide the real-time bedrock
> that keeps latencies low, like it already does with earlier versions.

Is the following a feasible development approach at the current time?

1. Start by developing one's realtime drivers and applications using
    RTDM-native for device drivers, and standard POSIX realtime threads
    and timers for one's applications, all running under a PREEMPT_RT
    patched kernel. This presents a comfortable and familiar
    development environment, with lots of tools, and the ability to
    reduce realtime priorities while investigating a bug that
    would otherwise lock up the computer.

2. After doing most of the hard work of debugging, testing and
    commissioning in the above environment, set up a Xenomai patched
    kernel, and recompile one's RTDM drivers and applications under
    Xenomai-duo. Even on systems which performed well under PREEMPT_RT,
    doing this would eliminate the worry that some un-excersized
    latency-generating path in PREEMPT_RT wouldn't cause mysterious
    occasional missed deadlines.

3. If later one wants to add significant features, or debug any
    residual problems, go back to step 1 to do further development
    and/or debugging.

Provided that one didn't mix realtime and non-realtime threads in a
single application, am I correct that the only thing that one would
have to do, to go back and forth between steps 1 to 2, would be to use
different makefiles and appropriately patched kernels?

Is RTDM-native sufficiently mature to use under a PREEMPT_RT kernel?

In the future, when Xenomai 3 comes out, will there be any practical
difference between the environment described above in step-1, and
using the POSIX skin of Xenomai-solo?

Martin

Re: [Xenomai-help] About HARD real time

[Philippe Gerum]

On Tue, 2009-06-02 at 15:40 -0700, Martin Shepherd wrote:
> On Tue, 2 Jun 2009, Philippe Gerum wrote:
> > For that reason, x3 will allow users to base their RT system over -rt or
> > the I-pipe, as they see fit, as the requirements mandate. When running
> > over -rt, Xenomai will only be in charge of providing real-time APIs;
> > when running over the I-pipe, it will also provide the real-time bedrock
> > that keeps latencies low, like it already does with earlier versions.
> 
> Is the following a feasible development approach at the current time?
> 
> 1. Start by developing one's realtime drivers and applications using
>     RTDM-native for device drivers, and standard POSIX realtime threads
>     and timers for one's applications, all running under a PREEMPT_RT
>     patched kernel. This presents a comfortable and familiar
>     development environment, with lots of tools, and the ability to
>     reduce realtime priorities while investigating a bug that
>     would otherwise lock up the computer.
> 
> 2. After doing most of the hard work of debugging, testing and
>     commissioning in the above environment, set up a Xenomai patched
>     kernel, and recompile one's RTDM drivers and applications under
>     Xenomai-duo. Even on systems which performed well under PREEMPT_RT,
>     doing this would eliminate the worry that some un-excersized
>     latency-generating path in PREEMPT_RT wouldn't cause mysterious
>     occasional missed deadlines.
> 
> 3. If later one wants to add significant features, or debug any
>     residual problems, go back to step 1 to do further development
>     and/or debugging.
> 

Let's put this differently: if one wants to be able to run the same
application code either over a dual kernel system or a native preemption
core, the strictest requirement comes from the co-kernel side: one has
to carefully segregate real-time and non real-time APIs, so that a time
critical thread won't switch to non-deterministic mode inadvertently.
For instance, by calling common malloc() or whatever glibc/pthread
service which may issue plain linux syscalls under the hood.

Therefore, the strictest requirement would already been there, even
before you start coding your app on top of preempt-rt, since I'm sure
you would not want to have to amend it later too much when time has come
to move it on top of the dual kernel. For instance, assuming that POSIX
signals are always delivered within a given time frame so as to use it
as an async real-time IPC might be fine with preempt-rt, but this would
be obviously wrong in a dual kernel system for userland code (this might
change at some point, but we still have to work on this).

Then, one should have some incentive to start working on a native
preemption core first if that is not the final RT framework to support
the application, and you mentioned the availability of tools in this
respect. I'm not sure this would be enough of an incentive in most
cases, since most of the common tools can be used over Xenomai as well
(gdb, ltt, profiling come to mind). Granted, using e.g. gdb over a
Xenomai RT thread makes it non-RT while the debugger holds it for
breakpointing, s-stepping etc., but in such a case, you just could not
claim any RT requirement anyway. However, valgrind-based tools are not
currently usable over a dual kernel Xenomai framework, that's a fact,
and may represent an incentive to use native preemption first in some
cases.

To sum up, I'm unsure that you would gain that a significant edge from
working over a native preemption core before porting everything to your
co-kernel system, in most cases. In any case, would you do so, you would
still have to abide by the co-kernel design and API constraints in your
initial implementation over preempt-rt.

There is one exception I see to the above though, when you actually want
to develop an application system on a powerful workstation before
eventually embedding it into your final target. In such a case, you
could use the desktop running preempt-rt as a development platform that
provides correct real-time behavior to the arch-independent part of your
application system.

> Provided that one didn't mix realtime and non-realtime threads in a
> single application, am I correct that the only thing that one would
> have to do, to go back and forth between steps 1 to 2, would be to use
> different makefiles and appropriately patched kernels?

Should be ok, but as suggested earlier, the "only thing" part you
mention here is likely to be the most demanding one when you will be at
the drawing board.

> 
> Is RTDM-native sufficiently mature to use under a PREEMPT_RT kernel?
> 

The question is two-fold:

- is preempt-rt mature enough to run on your particular target
- is RTDM-native mature enough

AFAIR, I don't think there are strong preempt-rt deps in RTDM native,
even if it was designed with preempt-rt in mind.

> In the future, when Xenomai 3 comes out, will there be any practical
> difference between the environment described above in step-1, and
> using the POSIX skin of Xenomai-solo?
> 

The point is that Xenomai-solo brings no POSIX skin at all; when coding
on top of POSIX with preempt-rt, you would just use the standard glibc.
Xenomai-solo brings our RTOS emulators to native linux platforms, and
being one of those, preempt-rt aims at delivering the real-time
performance for the set of services which should be RT capable per the
POSIX spec. So, when preempt-rt delivers on the RT promise, -solo does
as well, but you don't need -solo to run plain POSIX apps.

Xenomai-solo is three things:

1) a migration tool for people currently running a traditional RTOS
(e.g. VxWorks), who want to move over a preempt-rt based system whenever
available and mature for their target hw.

2) a migration tool for people currently running a traditional RTOS
emulator over a co-kernel based Xenomai environment, who want to move
over a preempt-rt based system.

3) a reusable workbench for the Xenomai project on the road to 3.0, so
that we may have a single code base for each traditional RTOS emulator
(VxWorks, pSOS...) we provide, that works both over a co-kernel and
native preemption cores. For that, we need two sets of building blocks
composing our generic RTOS core: one that works in dual kernel mode
(i.e. our "nucleus" thing), and another one that is built over common
real-time services from the POSIX standard, i.e. the -solo base library.

Put this library on top of the Xenomai POSIX skin in dual kernel mode,
and you should be able to stack the existing -solo RTOS emulators over
it as well. Use the plain glibc to get the POSIX services instead, but
over a preempt-rt enabled kernel, and you should get timely behavior as
well for those emulators, in native linux mode.

Well, that's the plan. If things start to hit the crapper, we will
probably have to resort to a successful technique commonly used among
software vendors: issue v4, which was there to paper over a serious
mis-design in v3, which was rolled out to fix half of v2's bugs, which
explained why we should not have even bothered with implementing v1 in
the first place. This technique is usually called "Customer Advantage
Program".

> Martin
-- 
Philippe.