[Ksummit-discuss] [TECH TOPIC] Mainlining PREEMPT_RT

[Ksummit-discuss] [TECH TOPIC] Mainlining PREEMPT_RT

[Steven Rostedt]

Now that it was announced in Dublin last week that mainlining
PREEMPT_RT has funding, I would like to propose a tech topic to discuss
what is needed to be done in the near future.

Thomas Gleixner and his minions will be doing the brunt of the work,
but there's issues that need to be addressed with the rest of the
kernel. The various topics would include:

 CPU hotplug
 preempt_disable and local_irq_disable annotations
 new timer_wheel implementation
 simple waitqueues
 trylock boosting

Several of the Real Time developers will be present at kernel summit. I
think this would be a great opportunity to have them be able to sit
down together and discuss the remaining issues.

-- Steve

Re: [Ksummit-discuss] [TECH TOPIC] Mainlining PREEMPT_RT

[Josh Triplett]

On Tue, Oct 13, 2015 at 12:42:30PM -0400, Steven Rostedt wrote:
> 
> Now that it was announced in Dublin last week that mainlining
> PREEMPT_RT has funding, I would like to propose a tech topic to discuss
> what is needed to be done in the near future.
> 
> Thomas Gleixner and his minions will be doing the brunt of the work,
> but there's issues that need to be addressed with the rest of the
> kernel. The various topics would include:
> 
>  CPU hotplug
>  preempt_disable and local_irq_disable annotations
>  new timer_wheel implementation
>  simple waitqueues
>  trylock boosting
> 
> Several of the Real Time developers will be present at kernel summit. I
> think this would be a great opportunity to have them be able to sit
> down together and discuss the remaining issues.

I would love to discuss this at Kernel Summit.

- Josh Triplett

Re: [Ksummit-discuss] [TECH TOPIC] Mainlining PREEMPT_RT

[Luis R. Rodriguez]

On Wed, Oct 14, 2015 at 12:40:10AM +0200, Rafael J. Wysocki wrote:
> On Tuesday, October 13, 2015 12:42:30 PM Steven Rostedt wrote:
> > 
> > Now that it was announced in Dublin last week that mainlining
> > PREEMPT_RT has funding, I would like to propose a tech topic to discuss
> > what is needed to be done in the near future.
> > 
> > Thomas Gleixner and his minions will be doing the brunt of the work,
> > but there's issues that need to be addressed with the rest of the
> > kernel. The various topics would include:
> > 
> >  CPU hotplug
> >  preempt_disable and local_irq_disable annotations
> >  new timer_wheel implementation
> >  simple waitqueues
> >  trylock boosting
> > 
> > Several of the Real Time developers will be present at kernel summit. I
> > think this would be a great opportunity to have them be able to sit
> > down together and discuss the remaining issues.
> 
> Well, that's not limited to the RT developers even.
> 
> Good topic.

How much of this is the above topics Vs lack of good semantics in general, or
code which needs love / good design? I ask as we have a few 'semantic'
discussions, and I have a feeling there might be overlap.

  Luis

Re: [Ksummit-discuss] [TECH TOPIC] Mainlining PREEMPT_RT

[Steven Rostedt]

On Wed, 14 Oct 2015 00:19:41 +0200
"Luis R. Rodriguez" <mcgrof@suse.com> wrote:

> On Wed, Oct 14, 2015 at 12:40:10AM +0200, Rafael J. Wysocki wrote:
> > On Tuesday, October 13, 2015 12:42:30 PM Steven Rostedt wrote:
> > > 
> > > Now that it was announced in Dublin last week that mainlining
> > > PREEMPT_RT has funding, I would like to propose a tech topic to discuss
> > > what is needed to be done in the near future.
> > > 
> > > Thomas Gleixner and his minions will be doing the brunt of the work,
> > > but there's issues that need to be addressed with the rest of the
> > > kernel. The various topics would include:
> > > 
> > >  CPU hotplug
> > >  preempt_disable and local_irq_disable annotations
> > >  new timer_wheel implementation
> > >  simple waitqueues
> > >  trylock boosting
> > > 
> > > Several of the Real Time developers will be present at kernel summit. I
> > > think this would be a great opportunity to have them be able to sit
> > > down together and discuss the remaining issues.
> > 
> > Well, that's not limited to the RT developers even.
> > 
> > Good topic.
> 
> How much of this is the above topics Vs lack of good semantics in general, or
> code which needs love / good design? I ask as we have a few 'semantic'
> discussions, and I have a feeling there might be overlap.

Well, CPU hotplug needs a rewrite. Preempt disabling and irq disabling
needs some commentary on what they are actually protecting. The timer
wheel is showing its age. Simple waitqueues can help make the majority
of the cases less complex. But the trylock boosting is -rt only.

As always, when -rt goes into mainline, we try hard to make sure those
changes improve mainline. Thus, we change designs and clean up code as
-rt goes in. We've been doing that since 2004, and nobody took
notice ;-)  That is, people saw improvements and never knew those
improvements were solely to get -rt working better.

-- Steve

Re: [Ksummit-discuss] [TECH TOPIC] Mainlining PREEMPT_RT

[Rafael J. Wysocki]

On Tuesday, October 13, 2015 12:42:30 PM Steven Rostedt wrote:
> 
> Now that it was announced in Dublin last week that mainlining
> PREEMPT_RT has funding, I would like to propose a tech topic to discuss
> what is needed to be done in the near future.
> 
> Thomas Gleixner and his minions will be doing the brunt of the work,
> but there's issues that need to be addressed with the rest of the
> kernel. The various topics would include:
> 
>  CPU hotplug
>  preempt_disable and local_irq_disable annotations
>  new timer_wheel implementation
>  simple waitqueues
>  trylock boosting
> 
> Several of the Real Time developers will be present at kernel summit. I
> think this would be a great opportunity to have them be able to sit
> down together and discuss the remaining issues.

Well, that's not limited to the RT developers even.

Good topic.

Thanks,
Rafael

Re: [Ksummit-discuss] [TECH TOPIC] Mainlining PREEMPT_RT

[Luis R. Rodriguez]

On Tue, Oct 13, 2015 at 4:04 PM, Rafael J. Wysocki <rjw@rjwysocki.net> wrote:
> On Wednesday, October 14, 2015 12:19:41 AM Luis R. Rodriguez wrote:
>> On Wed, Oct 14, 2015 at 12:40:10AM +0200, Rafael J. Wysocki wrote:
>> > On Tuesday, October 13, 2015 12:42:30 PM Steven Rostedt wrote:
>> > >
>> > > Now that it was announced in Dublin last week that mainlining
>> > > PREEMPT_RT has funding, I would like to propose a tech topic to discuss
>> > > what is needed to be done in the near future.
>> > >
>> > > Thomas Gleixner and his minions will be doing the brunt of the work,
>> > > but there's issues that need to be addressed with the rest of the
>> > > kernel. The various topics would include:
>> > >
>> > >  CPU hotplug
>> > >  preempt_disable and local_irq_disable annotations
>> > >  new timer_wheel implementation
>> > >  simple waitqueues
>> > >  trylock boosting
>> > >
>> > > Several of the Real Time developers will be present at kernel summit. I
>> > > think this would be a great opportunity to have them be able to sit
>> > > down together and discuss the remaining issues.
>> >
>> > Well, that's not limited to the RT developers even.
>> >
>> > Good topic.
>>
>> How much of this is the above topics Vs lack of good semantics in general, or
>> code which needs love / good design? I ask as we have a few 'semantic'
>> discussions, and I have a feeling there might be overlap.
>
> At this time it is not particularly clear to me how much overlap there's going
> to be, if any.
>
> And if there is overlap, we should be able to notice that and avoid
> discussing the same thing twice, perhaps?

Sounds good.

 Luis

Re: [Ksummit-discuss] [TECH TOPIC] Mainlining PREEMPT_RT

[Luis R. Rodriguez]

On Tue, Oct 13, 2015 at 06:39:34PM -0400, Steven Rostedt wrote:
> On Wed, 14 Oct 2015 00:19:41 +0200
> "Luis R. Rodriguez" <mcgrof@suse.com> wrote:
> 
> > On Wed, Oct 14, 2015 at 12:40:10AM +0200, Rafael J. Wysocki wrote:
> > > On Tuesday, October 13, 2015 12:42:30 PM Steven Rostedt wrote:
> > > > 
> > > > Now that it was announced in Dublin last week that mainlining
> > > > PREEMPT_RT has funding, I would like to propose a tech topic to discuss
> > > > what is needed to be done in the near future.
> > > > 
> > > > Thomas Gleixner and his minions will be doing the brunt of the work,
> > > > but there's issues that need to be addressed with the rest of the
> > > > kernel. The various topics would include:
> > > > 
> > > >  CPU hotplug
> > > >  preempt_disable and local_irq_disable annotations
> > > >  new timer_wheel implementation
> > > >  simple waitqueues
> > > >  trylock boosting
> > > > 
> > > > Several of the Real Time developers will be present at kernel summit. I
> > > > think this would be a great opportunity to have them be able to sit
> > > > down together and discuss the remaining issues.
> > > 
> > > Well, that's not limited to the RT developers even.
> > > 
> > > Good topic.
> > 
> > How much of this is the above topics Vs lack of good semantics in general, or
> > code which needs love / good design? I ask as we have a few 'semantic'
> > discussions, and I have a feeling there might be overlap.
> 
> Well, CPU hotplug needs a rewrite. Preempt disabling and irq disabling
> needs some commentary on what they are actually protecting. The timer
> wheel is showing its age. Simple waitqueues can help make the majority
> of the cases less complex. But the trylock boosting is -rt only.
> 
> As always, when -rt goes into mainline, we try hard to make sure those
> changes improve mainline. Thus, we change designs and clean up code as
> -rt goes in. We've been doing that since 2004, and nobody took
> notice ;-)  That is, people saw improvements and never knew those
> improvements were solely to get -rt working better.

That's what I was alluding to -- what lessons have been learned here
that we have needed over time to make that happen, which to others
not in the know, perhaps have been transparent / nothing new other
than yet-another-kernel-feature / kernel-bell-whistle.

  Luis

Re: [Ksummit-discuss] [TECH TOPIC] Mainlining PREEMPT_RT

[Rafael J. Wysocki]

On Wednesday, October 14, 2015 12:19:41 AM Luis R. Rodriguez wrote:
> On Wed, Oct 14, 2015 at 12:40:10AM +0200, Rafael J. Wysocki wrote:
> > On Tuesday, October 13, 2015 12:42:30 PM Steven Rostedt wrote:
> > > 
> > > Now that it was announced in Dublin last week that mainlining
> > > PREEMPT_RT has funding, I would like to propose a tech topic to discuss
> > > what is needed to be done in the near future.
> > > 
> > > Thomas Gleixner and his minions will be doing the brunt of the work,
> > > but there's issues that need to be addressed with the rest of the
> > > kernel. The various topics would include:
> > > 
> > >  CPU hotplug
> > >  preempt_disable and local_irq_disable annotations
> > >  new timer_wheel implementation
> > >  simple waitqueues
> > >  trylock boosting
> > > 
> > > Several of the Real Time developers will be present at kernel summit. I
> > > think this would be a great opportunity to have them be able to sit
> > > down together and discuss the remaining issues.
> > 
> > Well, that's not limited to the RT developers even.
> > 
> > Good topic.
> 
> How much of this is the above topics Vs lack of good semantics in general, or
> code which needs love / good design? I ask as we have a few 'semantic'
> discussions, and I have a feeling there might be overlap.

At this time it is not particularly clear to me how much overlap there's going
to be, if any.

And if there is overlap, we should be able to notice that and avoid
discussing the same thing twice, perhaps?

Thanks,
Rafael

Re: [Ksummit-discuss] [TECH TOPIC] Mainlining PREEMPT_RT

[Linus Walleij]

On Tue, Oct 13, 2015 at 6:42 PM, Steven Rostedt <rostedt@goodmis.org> wrote:

> The various topics would include:
>
>  CPU hotplug
>  preempt_disable and local_irq_disable annotations
>  new timer_wheel implementation
>  simple waitqueues
>  trylock boosting

- Decide on a case-per-case basis whether to make all irqchips threaded.

We recently ran into this in the GPIO subsystem as Grygorii Strashko
is working on RT on the ARM OMAP. So I drafted a checklist:
http://marc.info/?l=linux-gpio&m=144382150627266&w=2

Feedback welcome. I don't know if I even got the details right in that
checklist. There are a few irqchips around:

$ git grep 'struct irq_chip ' arch/ drivers/ | wc -l
721

If this isn't done I guess that for example a sudden IRQ on a GPIO
pin can delay a deadline. Not good.

Yours,
Linus Walleij

Re: [Ksummit-discuss] [TECH TOPIC] Mainlining PREEMPT_RT

[Christoph Lameter]

On Tue, 13 Oct 2015, Steven Rostedt wrote:

>  preempt_disable and local_irq_disable annotations

Hrm... Would there be a way to do Realtime without PREEMPT / PREEMPT_RT
and such things? Doing this performance degradation is pretty significant
and as far as I know makes this unacceptable for many uses. CONFIG_PREEMPT
is already a problem.

Or change CONFIG_PREEMPT first to be less intrusive? Have defined points
where preemption can occur like CONFIG_PREEMPT_VOLUNTARY? Could we have
realtime based on that? If we can keep the time between voluntary
preemption short enough then this should do this as well. Plus we can
likely reduce kernel complexity because code can rely on  not being
rescheduled unless an explicit call was done.

Re: [Ksummit-discuss] [TECH TOPIC] Mainlining PREEMPT_RT

[Steven Rostedt]

On Wed, 14 Oct 2015 06:45:31 -0500 (CDT)
Christoph Lameter <cl@linux.com> wrote:

> On Tue, 13 Oct 2015, Steven Rostedt wrote:
> 
> >  preempt_disable and local_irq_disable annotations
> 
> Hrm... Would there be a way to do Realtime without PREEMPT / PREEMPT_RT
> and such things? Doing this performance degradation is pretty significant
> and as far as I know makes this unacceptable for many uses. CONFIG_PREEMPT
> is already a problem.

The only RT that this would be good for is for CPU isolation and running
tasks that don't ever interact with the kernel. This is a completely
different use case than what PREEMPT_RT is about, although, NO_HZ_FULL
would be beneficial to both scenarios.

> 
> Or change CONFIG_PREEMPT first to be less intrusive? Have defined points
> where preemption can occur like CONFIG_PREEMPT_VOLUNTARY? Could we have
> realtime based on that? If we can keep the time between voluntary
> preemption short enough then this should do this as well. Plus we can
> likely reduce kernel complexity because code can rely on  not being
> rescheduled unless an explicit call was done.

Again, what you want has nothing to do with PREEMPT_RT, and not what
I'm proposing here for a tech topic.

-- Steve

Re: [Ksummit-discuss] [TECH TOPIC] Mainlining PREEMPT_RT

[Christoph Lameter]

On Wed, 14 Oct 2015, Steven Rostedt wrote:

> > Hrm... Would there be a way to do Realtime without PREEMPT / PREEMPT_RT
> > and such things? Doing this performance degradation is pretty significant
> > and as far as I know makes this unacceptable for many uses. CONFIG_PREEMPT
> > is already a problem.
>
> The only RT that this would be good for is for CPU isolation and running
> tasks that don't ever interact with the kernel. This is a completely
> different use case than what PREEMPT_RT is about, although, NO_HZ_FULL
> would be beneficial to both scenarios.

Well I though that was the only feasable way to get to a good RT
implementation in the kernel. If you can free up the processor from OS
activities by shifting them around then there is less of a need to
sprinkle preempt calls all over the code.

> > Or change CONFIG_PREEMPT first to be less intrusive? Have defined points
> > where preemption can occur like CONFIG_PREEMPT_VOLUNTARY? Could we have
> > realtime based on that? If we can keep the time between voluntary
> > preemption short enough then this should do this as well. Plus we can
> > likely reduce kernel complexity because code can rely on  not being
> > rescheduled unless an explicit call was done.
>
> Again, what you want has nothing to do with PREEMPT_RT, and not what
> I'm proposing here for a tech topic.

PREEMPT is already a problem and it has acceptance problems. So why go
further down the rathole? Lets fix this so that it is good and can offer
the lowest latencies that the hardware supports.

PREEMPT_RT was developed at a time where we only had one or two hardware
threads. Now we we have lots and we can separate things out to allow
deterministic execution on a few cores. That will be much better than
forcing all the kernel code to be deterministric.

Re: [Ksummit-discuss] [TECH TOPIC] Mainlining PREEMPT_RT

[Steven Rostedt]

On Wed, 14 Oct 2015 09:49:33 -0500 (CDT)
Christoph Lameter <cl@linux.com> wrote:

> On Wed, 14 Oct 2015, Steven Rostedt wrote:
> 
> > > Hrm... Would there be a way to do Realtime without PREEMPT / PREEMPT_RT
> > > and such things? Doing this performance degradation is pretty significant
> > > and as far as I know makes this unacceptable for many uses. CONFIG_PREEMPT
> > > is already a problem.
> >
> > The only RT that this would be good for is for CPU isolation and running
> > tasks that don't ever interact with the kernel. This is a completely
> > different use case than what PREEMPT_RT is about, although, NO_HZ_FULL
> > would be beneficial to both scenarios.
> 
> Well I though that was the only feasable way to get to a good RT
> implementation in the kernel. If you can free up the processor from OS
> activities by shifting them around then there is less of a need to
> sprinkle preempt calls all over the code.

Your use case is for large servers doing high frequency trading.
PREEMPT_RT works for that too, but for when the applications actually
use the kernel. PREEMPT_RT is also used for embedded, where your use
case does not fit.

> 
> > > Or change CONFIG_PREEMPT first to be less intrusive? Have defined points
> > > where preemption can occur like CONFIG_PREEMPT_VOLUNTARY? Could we have
> > > realtime based on that? If we can keep the time between voluntary
> > > preemption short enough then this should do this as well. Plus we can
> > > likely reduce kernel complexity because code can rely on  not being
> > > rescheduled unless an explicit call was done.
> >
> > Again, what you want has nothing to do with PREEMPT_RT, and not what
> > I'm proposing here for a tech topic.
> 
> PREEMPT is already a problem and it has acceptance problems. So why go
> further down the rathole? Lets fix this so that it is good and can offer
> the lowest latencies that the hardware supports.

Christoph, there's other use cases than what you want. This proposal is
about PREEMPT_RT and there's a lot of use cases for that. Just because
it's not what you need doesn't negate its usefulness.


> 
> PREEMPT_RT was developed at a time where we only had one or two hardware
> threads. Now we we have lots and we can separate things out to allow
> deterministic execution on a few cores. That will be much better than
> forcing all the kernel code to be deterministric.

Christoph, feel free to ignore this thread and this topic. It's not for
you. But there's others out there that don't use 1000 CPUs and find
what we are doing quite useful.

-- Steve

Re: [Ksummit-discuss] [TECH TOPIC] Mainlining PREEMPT_RT

[Christoph Lameter]

On Wed, 14 Oct 2015, Steven Rostedt wrote:

> Your use case is for large servers doing high frequency trading.

The use case is for regular servers wanting to use the processor
without OS interference in order to guarantee as much as possible a fast
"realtime" response (that term is used in this context by numerous users
of NOHZ. May be questionable I know but people use it that way).

> PREEMPT_RT works for that too, but for when the applications actually
> use the kernel. PREEMPT_RT is also used for embedded, where your use
> case does not fit.

The work on NOHZ was started and is still pushed by Chris Metcalf who is
working on embedded systems.

> Christoph, there's other use cases than what you want. This proposal is
> about PREEMPT_RT and there's a lot of use cases for that. Just because
> it's not what you need doesn't negate its usefulness.

Not seen it yet. Just references to customers that I have never worked
with. I keep heareing that RT is not useful because its soft realtime and
not guaranteed to keep its time constraints. Lets implement a real "hard"
realtime facility. Its possible if the processor is free of the OS.

> Christoph, feel free to ignore this thread and this topic. It's not for
> you. But there's others out there that don't use 1000 CPUs and find
> what we are doing quite useful.

Have been working with regular business class servers with two socket for
the last 8 years. The 1000s of cpus was a decade ago when I worked for
SGI.

This is regarding a common use case and AFAICT preempt and preempt_rt miss
the mark by increasing kernel code complexity significantly and thus
slowing things (cache footprint!!!) down instead of doing the obvious
which would be an enhanced NOHZ approach where a cpu can dedicate its full
power to a realtime load that may have to be fast and deterministic as the
hardware allows.

Re: [Ksummit-discuss] [TECH TOPIC] Mainlining PREEMPT_RT

[Steven Rostedt]

On Wed, 14 Oct 2015 13:12:06 -0500 (CDT)
Christoph Lameter <cl@linux.com> wrote:

> On Wed, 14 Oct 2015, Steven Rostedt wrote:
> 
> > Your use case is for large servers doing high frequency trading.
> 
> The use case is for regular servers wanting to use the processor
> without OS interference in order to guarantee as much as possible a fast
> "realtime" response (that term is used in this context by numerous users

real-time does not mean real-fast.

> of NOHZ. May be questionable I know but people use it that way).
> 
> > PREEMPT_RT works for that too, but for when the applications actually
> > use the kernel. PREEMPT_RT is also used for embedded, where your use
> > case does not fit.
> 
> The work on NOHZ was started and is still pushed by Chris Metcalf who is
> working on embedded systems.

I never said that NO_HZ_FULL was not appropriate for embedded. But it's
also not real-time.

> 
> > Christoph, there's other use cases than what you want. This proposal is
> > about PREEMPT_RT and there's a lot of use cases for that. Just because
> > it's not what you need doesn't negate its usefulness.
> 
> Not seen it yet. Just references to customers that I have never worked
> with. I keep heareing that RT is not useful because its soft realtime and
> not guaranteed to keep its time constraints. Lets implement a real "hard"
> realtime facility. Its possible if the processor is free of the OS.

If PREEMPT_RT is not the way to go, then why did several companies just
invest into getting it mainlined?

> 
> > Christoph, feel free to ignore this thread and this topic. It's not for
> > you. But there's others out there that don't use 1000 CPUs and find
> > what we are doing quite useful.
> 
> Have been working with regular business class servers with two socket for
> the last 8 years. The 1000s of cpus was a decade ago when I worked for
> SGI.
> 
> This is regarding a common use case and AFAICT preempt and preempt_rt miss
> the mark by increasing kernel code complexity significantly and thus
> slowing things (cache footprint!!!) down instead of doing the obvious
> which would be an enhanced NOHZ approach where a cpu can dedicate its full
> power to a realtime load that may have to be fast and deterministic as the
> hardware allows.

Again, real-time does not mean real-fast. How do you handle priority
inversion? How do you handle outliers? Which the Linux kernel has many
huge outliers if things are not handled correctly, which PREEMPT_RT
solve (making it, by the way, a hard real time design).  If you can
accept a single outlier, then that's soft real-time, and you are
talking about something completely different.

Hard real-time is about worse case latency, not the average latency.
That's called "soft real-time"

-- Steve

Re: [Ksummit-discuss] [TECH TOPIC] Mainlining PREEMPT_RT

[Christoph Lameter]

On Wed, 14 Oct 2015, Steven Rostedt wrote:

> If PREEMPT_RT is not the way to go, then why did several companies just
> invest into getting it mainlined?

I guess they want "realtime" support. Better talk to them what they mean
by realtime. I have never seen a real use case for PREEMPT_RT. Instead a
lot of high level "realtime" talk and disappointment when they found out
what the real constraints and limitations are.

> Again, real-time does not mean real-fast. How do you handle priority
> inversion? How do you handle outliers? Which the Linux kernel has many
> huge outliers if things are not handled correctly, which PREEMPT_RT
> solve (making it, by the way, a hard real time design).  If you can
> accept a single outlier, then that's soft real-time, and you are
> talking about something completely different.

You could put the processing for the stuff that creates outliers on a
different cpu? Layout the processing in such a way to guarantee no
outliers on the "realtime" processors?

> Hard real-time is about worse case latency, not the average latency.
> That's called "soft real-time"

Of course. Thats why I suggested combining it with the NOHZ appraoch. Put
the pieces that can create outliers on cpus not used for "realtime"
threads. Ensure that what is running on a hard realtime cpu can complete
in a defined timed interval.

Realtime is about guarantees of a response in a certain time period. That
is why "soft realtime" is not making much sense. There is no guarantee
anymore. Just more overhead. OS processing slows down due to the
sprinkling of preempt code but the timing guarantees that one seeks can be
violated. Increasing the processing overhead may increase the number of
failures to process a given task in a time period that one wants
guaranteed. So better run without preempt in order to have a higher chance
to finish in time? That is at least what my industry has decided to do.

But we really wish we had a hard realtime capability if we must respond in
a certain time period. Sadly there is no solution in sight right now.

Re: [Ksummit-discuss] [TECH TOPIC] Mainlining PREEMPT_RT

[James Bottomley]

On Wed, 2015-10-14 at 13:56 -0500, Christoph Lameter wrote:
> On Wed, 14 Oct 2015, Steven Rostedt wrote:
> 
> > If PREEMPT_RT is not the way to go, then why did several companies just
> > invest into getting it mainlined?
> 
> I guess they want "realtime" support. Better talk to them what they mean
> by realtime. I have never seen a real use case for PREEMPT_RT. Instead a
> lot of high level "realtime" talk and disappointment when they found out
> what the real constraints and limitations are.
> 
> > Again, real-time does not mean real-fast. How do you handle priority
> > inversion? How do you handle outliers? Which the Linux kernel has many
> > huge outliers if things are not handled correctly, which PREEMPT_RT
> > solve (making it, by the way, a hard real time design).  If you can
> > accept a single outlier, then that's soft real-time, and you are
> > talking about something completely different.
> 
> You could put the processing for the stuff that creates outliers on a
> different cpu? Layout the processing in such a way to guarantee no
> outliers on the "realtime" processors?
> 
> > Hard real-time is about worse case latency, not the average latency.
> > That's called "soft real-time"
> 
> Of course. Thats why I suggested combining it with the NOHZ appraoch. Put
> the pieces that can create outliers on cpus not used for "realtime"
> threads. Ensure that what is running on a hard realtime cpu can complete
> in a defined timed interval.
> 
> Realtime is about guarantees of a response in a certain time period. That
> is why "soft realtime" is not making much sense. There is no guarantee
> anymore. Just more overhead. OS processing slows down due to the
> sprinkling of preempt code but the timing guarantees that one seeks can be
> violated. Increasing the processing overhead may increase the number of
> failures to process a given task in a time period that one wants
> guaranteed. So better run without preempt in order to have a higher chance
> to finish in time? That is at least what my industry has decided to do.
> 
> But we really wish we had a hard realtime capability if we must respond in
> a certain time period. Sadly there is no solution in sight right now.

Yes, there is: PREEMPT_RT is (or can be if configured correctly) hard
real time.  The point, as Steve told you several times, is to sacrifice
performance for determinism.  As long as the general system deadlines
still fit, then the solution works ... well as it happens: this covers
many use cases because we sacrifice microseconds for millisecond
deadlines (or things even further apart in timescales); there's a huge
number of industrial use cases in this range.

What you're saying is that sacrificing performance is unacceptable for
your deadlines, so you're not able to use this solution: fair enough, it
doesn't cover all possible use cases, just a majority.  However trying
to imply that it's not a solution because it doesn't solve your problem
is really not going to motivate people involved to want to help you.

James

Re: [Ksummit-discuss] [TECH TOPIC] Mainlining PREEMPT_RT

[Tim Bird]

On 10/14/2015 11:56 AM, Christoph Lameter wrote:
> On Wed, 14 Oct 2015, Steven Rostedt wrote:
> 
>> If PREEMPT_RT is not the way to go, then why did several companies just
>> invest into getting it mainlined?
> 
> I guess they want "realtime" support. Better talk to them what they mean
> by realtime. I have never seen a real use case for PREEMPT_RT. Instead a
> lot of high level "realtime" talk and disappointment when they found out
> what the real constraints and limitations are.
> 
>> Again, real-time does not mean real-fast. How do you handle priority
>> inversion? How do you handle outliers? Which the Linux kernel has many
>> huge outliers if things are not handled correctly, which PREEMPT_RT
>> solve (making it, by the way, a hard real time design).  If you can
>> accept a single outlier, then that's soft real-time, and you are
>> talking about something completely different.
> 
> You could put the processing for the stuff that creates outliers on a
> different cpu? Layout the processing in such a way to guarantee no
> outliers on the "realtime" processors?
> 
>> Hard real-time is about worse case latency, not the average latency.
>> That's called "soft real-time"
> 
> Of course. Thats why I suggested combining it with the NOHZ appraoch. Put
> the pieces that can create outliers on cpus not used for "realtime"
> threads. Ensure that what is running on a hard realtime cpu can complete
> in a defined timed interval.
> 
> Realtime is about guarantees of a response in a certain time period. That
> is why "soft realtime" is not making much sense. There is no guarantee
> anymore. Just more overhead. OS processing slows down due to the
> sprinkling of preempt code but the timing guarantees that one seeks can be
> violated. Increasing the processing overhead may increase the number of
> failures to process a given task in a time period that one wants
> guaranteed. So better run without preempt in order to have a higher chance
> to finish in time? That is at least what my industry has decided to do.

This debate is as old as the hills. I remember having it in 2000 with
various parties, when talking about what used to be called the "dual-kernel"
approach.

> But we really wish we had a hard realtime capability if we must respond in
> a certain time period. Sadly there is no solution in sight right now.

Check out Xenomai. (xenomai.org)  I think that's what many embedded folk use
if they find that RT-PREEMPT does not meet their needs.  Xenomai 3.0
allows you to use their system either on top of RT-PREEMPT or
on top of their Cobalt kernel sitting next to (or in front of,
depending on your perspective) the Linux kernel.

Sony used RT-PREEMPT in some products, and a dual-kernel in others,
and isolated CPUs running something else (usually uItron) in yet others.

Personally, I'd be happy to see RT-PREEMPT go in - there are certain use
cases for it.  I'd also like to see the NOHZ stuff go in as well.  It should
solve a set of problems for isolating RT tasks without sacrificing
performance on the non-RT CPUs.  And finally, I wouldn't mind putting
a non-Linux RT kernel like Cobalt (oh the horrors) into mainline as well.
Each of these approaches has their strengths and weaknesses.  Linux is,
after all, whatever we say it is - and it could easily have a small RT
micro-kernel as part of the source base as well.

AFAIK the patent issues that plagued the dual-kernel approach are now
behind us, so this might be a good area of investigation.

I've long felt that RT-PREEMPT was sucking the oxygen out of getting other,
technically valid, approaches to RT with Linux into mainline.
 -- Tim

Re: [Ksummit-discuss] [TECH TOPIC] Mainlining PREEMPT_RT

[Thomas Gleixner]

Christoph,

On Wed, 14 Oct 2015, Christoph Lameter wrote:

> On Wed, 14 Oct 2015, Steven Rostedt wrote:
> 
> > If PREEMPT_RT is not the way to go, then why did several companies just
> > invest into getting it mainlined?
> 
> I guess they want "realtime" support. Better talk to them what they mean
> by realtime. I have never seen a real use case for PREEMPT_RT. Instead a
> lot of high level "realtime" talk and disappointment when they found out
> what the real constraints and limitations are.

Just because you did not see any real use does not mean, there is no
real use. There are whole application classes out there, which use it
today. Look at the OSADL member roaster and figure out what those
companies are doing.

> You could put the processing for the stuff that creates outliers on a
> different cpu? Layout the processing in such a way to guarantee no
> outliers on the "realtime" processors?

Sure, you just add another server to it and be done. That's not what
people in the embedded space can do.

They need to run highly multi threaded applications on a small system
and still need the guarantee that they meet their dead lines. They
cannot afford the waste of a CPU busy spinning in order to process the
next event. Neither they have the number of cores which would be
needed nor they can afford the energy wasted by it.

> Realtime is about guarantees of a response in a certain time period. That
> is why "soft realtime" is not making much sense. There is no guarantee
> anymore. Just more overhead. OS processing slows down due to the
> sprinkling of preempt code but the timing guarantees that one seeks can be
> violated. Increasing the processing overhead may increase the number of
> failures to process a given task in a time period that one wants
> guaranteed. So better run without preempt in order to have a higher chance
> to finish in time? That is at least what my industry has decided to do.

Your industry is a total different application space. Please stop to
project your narrow view of the application space onto everybody.

And your interpretation of real time is just silly. 

The key of an RTOS is the deterministic guarantee for the time it
takes to enqueue a given task up to the completion of the task. The
design goal is not high throughput. The design goal is to achieve
latency guarantees. An RTOS sacrifices performance and throughput to
achieve that guarantees.

Once again: The definition of Real-Time is determinism, not speed.

> But we really wish we had a hard realtime capability if we must respond in
> a certain time period. Sadly there is no solution in sight right now.

You're design goal is completely different. Your design goal is
sacrifying a single CPU in order to have FAST response on an event and
high throughput at the same time. That has nothing to do with real
time. Go and read on OS theory to figure out why.

I'm really tired of your obnoxious claims, that the only way of doing
RT is what you think it might be.

I'm well aware that your use case exists. It was one of the reasons
why I started the NOHZ FULL effort in the first place. Please, fix
your NOHZ FULL isolation problem and stop imposing your completely
domain specific blinkerdness on everybody else.

Thanks,

	tglx

Re: [Ksummit-discuss] [TECH TOPIC] Mainlining PREEMPT_RT

[Steven Rostedt]

On Wed, 14 Oct 2015 12:30:21 -0700
Tim Bird <tim.bird@sonymobile.com> wrote:

REPLY ALL Tim! I didn't notice this email till I saw it as QotW on LWN.


> Check out Xenomai. (xenomai.org)  I think that's what many embedded folk use
> if they find that RT-PREEMPT does not meet their needs.  Xenomai 3.0
> allows you to use their system either on top of RT-PREEMPT or
> on top of their Cobalt kernel sitting next to (or in front of,
> depending on your perspective) the Linux kernel.
> 
> Sony used RT-PREEMPT in some products, and a dual-kernel in others,
> and isolated CPUs running something else (usually uItron) in yet others.
> 
> Personally, I'd be happy to see RT-PREEMPT go in - there are certain use

BTW, it's PREEMPT_RT not RT-PREEMPT.

> cases for it.  I'd also like to see the NOHZ stuff go in as well.  It should
> solve a set of problems for isolating RT tasks without sacrificing
> performance on the non-RT CPUs.  And finally, I wouldn't mind putting
> a non-Linux RT kernel like Cobalt (oh the horrors) into mainline as well.
> Each of these approaches has their strengths and weaknesses.  Linux is,
> after all, whatever we say it is - and it could easily have a small RT
> micro-kernel as part of the source base as well.
> 
> AFAIK the patent issues that plagued the dual-kernel approach are now
> behind us, so this might be a good area of investigation.
> 
> I've long felt that RT-PREEMPT was sucking the oxygen out of getting other,
> technically valid, approaches to RT with Linux into mainline.

I'm perfectly fine with working together with Xenomai. When you need a
few microsecond latencies, that may be the best tool. I don't think
they are mutually exclusive, but actually complement each other quite
nicely.

-- Steve

Re: [Ksummit-discuss] [TECH TOPIC] Mainlining PREEMPT_RT

[Thomas Gleixner]

On Wed, 14 Oct 2015, Tim Bird wrote:
> I've long felt that RT-PREEMPT was sucking the oxygen out of getting other,
> technically valid, approaches to RT with Linux into mainline.

I have to disagree. PREEMPT_RT does not block anything else, I'm
actively supporting the NOHZ FULL stuff from the very beginning.

So if people want to have xenomai or whatever in mainline, then they
need to sit down and do the work. I'm not in the way as long as it
makes technically sense and does not impose hard to overcome issues
vs. maintainence and flexibility.

Thanks,

	tglx

Re: [Ksummit-discuss] [TECH TOPIC] Mainlining PREEMPT_RT

[Christoph Lameter]

On Wed, 14 Oct 2015, Thomas Gleixner wrote:

> Sure, you just add another server to it and be done. That's not what
> people in the embedded space can do.

Why not? Seems that even small devices have multiple cores? I just got a
quad core cheapo phone for $50. Even the Intel Edison platform is dual
core. So it may be possible to have deterministic hard real time on one
core.

> They need to run highly multi threaded applications on a small system
> and still need the guarantee that they meet their dead lines. They
> cannot afford the waste of a CPU busy spinning in order to process the
> next event. Neither they have the number of cores which would be
> needed nor they can afford the energy wasted by it.

There could be deterministic wake up logic for real time cpus? If you can
push off most of the effort on another cpu its likely possible to make the
behavior much more deterministic for a cpu that does real time processing.
Why does spinning come up here?

> > violated. Increasing the processing overhead may increase the number of
> > failures to process a given task in a time period that one wants
> > guaranteed. So better run without preempt in order to have a higher chance
> > to finish in time? That is at least what my industry has decided to do.
>
> Your industry is a total different application space. Please stop to
> project your narrow view of the application space onto everybody.

This is actually what I heard from the Dronecode people and what I
mentioned to you before in Dublin. Stop trying to make silly arguments
that the cases I discuss are just special. These attempts to kill the
discussion make it questionable if we can actually discuss this at all.
The dronecode people went to a special RTOS on a special processor because
the "realtime" offered by preempt_rt was not really realtime. They needed
hard realtime.

> The key of an RTOS is the deterministic guarantee for the time it
> takes to enqueue a given task up to the completion of the task. The
> design goal is not high throughput. The design goal is to achieve
> latency guarantees. An RTOS sacrifices performance and throughput to
> achieve that guarantees.

Indeed please lets do that. That means hard real time. And lets limit the
sacrifices as much as possible.

> Once again: The definition of Real-Time is determinism, not speed.

Yes then please do deterministic latencies. No exceptions. No soft
realtime.

> You're design goal is completely different. Your design goal is
> sacrifying a single CPU in order to have FAST response on an event and
> high throughput at the same time. That has nothing to do with real
> time. Go and read on OS theory to figure out why.

???. Please stay on topic. Maybe you would have benefited from the
clases on OS design that I taught. But maybe reading comprehension would
be much more urgent.

> I'm well aware that your use case exists. It was one of the reasons
> why I started the NOHZ FULL effort in the first place. Please, fix
> your NOHZ FULL isolation problem and stop imposing your completely
> domain specific blinkerdness on everybody else.

I have no particular use case for my company in mind here. This is what I
hear when talking to various companies in a number of different areas of
computing that have tried preempt_rt.

Re: [Ksummit-discuss] [TECH TOPIC] Mainlining PREEMPT_RT

[Steven Rostedt]

On Thu, 15 Oct 2015 09:22:58 -0500 (CDT)
Christoph Lameter <cl@linux.com> wrote:

> I have no particular use case for my company in mind here. This is what I
> hear when talking to various companies in a number of different areas of
> computing that have tried preempt_rt.

Why are they telling you this and not us? Of course this may be people
that have different requirements.

Need a few microsecond latencies; Try Xenomai.
Need a few hundred microsecond latencies; PREEMPT_RT will do
Need no interference from the kernel; Use NO_HZ_FULL

Look, there's obviously users of PREEMPT_RT, so why are you trying to
kill it? If anything, PREEMPT_RT has made the mainline kernel cleaner.

Just because you don't see a use for it, doesn't mean one does not
exist.

-- Steve

Re: [Ksummit-discuss] [TECH TOPIC] Mainlining PREEMPT_RT

[Jan Kara]

On Thu 15-10-15 11:13:10, Steven Rostedt wrote:
> On Thu, 15 Oct 2015 09:22:58 -0500 (CDT)
> Christoph Lameter <cl@linux.com> wrote:
> 
> > I have no particular use case for my company in mind here. This is what I
> > hear when talking to various companies in a number of different areas of
> > computing that have tried preempt_rt.
> 
> Why are they telling you this and not us? Of course this may be people
> that have different requirements.
> 
> Need a few microsecond latencies; Try Xenomai.
> Need a few hundred microsecond latencies; PREEMPT_RT will do
> Need no interference from the kernel; Use NO_HZ_FULL
> 
> Look, there's obviously users of PREEMPT_RT, so why are you trying to
> kill it? If anything, PREEMPT_RT has made the mainline kernel cleaner.
> 
> Just because you don't see a use for it, doesn't mean one does not
> exist.

I know close to nothing about RT so maybe a stupid question: Do I
understand correctly that PREEMP_RT is there for use cases where processes
have latency requirements but also need to do syscalls as a part of their
work? Or is it that we cannot afford to isolate kernel on some CPUs because
it's too hard to estimate how much work it will need to do, it wastes too
much etc.? Or both?

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

Re: [Ksummit-discuss] [TECH TOPIC] Mainlining PREEMPT_RT

[Steven Rostedt]

On Thu, 15 Oct 2015 19:21:39 +0200
Jan Kara <jack@suse.cz> wrote:

> I know close to nothing about RT so maybe a stupid question: Do I
> understand correctly that PREEMP_RT is there for use cases where processes
> have latency requirements but also need to do syscalls as a part of their
> work? Or is it that we cannot afford to isolate kernel on some CPUs because
> it's too hard to estimate how much work it will need to do, it wastes too
> much etc.? Or both?

Yes, it's for processes that do syscalls, and also might share
resources with non-rt threads. Isolating RT tasks on their own CPUs
help, but what happens when you have more RT threads than CPUs?

The deadline scheduler is also useful here, as it can guarantee that
tasks get a specified amount of cpu time per period. PREEMPT_RT removes
priority inversion, and lets tasks preempt pretty much anywhere in the
kernel. Yes, if you have a single RT task with nothing else running on
a CPU, it doesn't care about preemption, because it has nothing that it
needs to preempt.

If you can afford to place all your RT tasks on their own CPUs, then
running either PREEMPT_NONE or PREEMPT_VOLUNTARY, and NO_HZ_FULL, is
probably all that you need, as long as you don't have your RT tasks
grabbing any mutex that could be held by a non RT task, or an RT task
that shares the CPU with other RT tasks.

-- Steve

Re: [Ksummit-discuss] [TECH TOPIC] Mainlining PREEMPT_RT

[Thomas Gleixner]

On Thu, 15 Oct 2015, Christoph Lameter wrote:
> On Wed, 14 Oct 2015, Thomas Gleixner wrote:
> 
> > Sure, you just add another server to it and be done. That's not what
> > people in the embedded space can do.
> 
> Why not? Seems that even small devices have multiple cores? I just got a
> quad core cheapo phone for $50. Even the Intel Edison platform is dual
> core. So it may be possible to have deterministic hard real time on one
> core.

That fits into all current use cases of Preempt-RT how?

Please come up with a functional concept for applications like PLCs
(Programmable Logic Controllers) where the end customer defines the
application and therefor the number of involved thread and their
particular deadline requirements. That would for a change start a
useful discussion instead of your 'spend a CPU' mantra.

> > They need to run highly multi threaded applications on a small system
> > and still need the guarantee that they meet their dead lines. They
> > cannot afford the waste of a CPU busy spinning in order to process the
> > next event. Neither they have the number of cores which would be
> > needed nor they can afford the energy wasted by it.
> 
> There could be deterministic wake up logic for real time cpus? If you can
> push off most of the effort on another cpu its likely possible to make the
> behavior much more deterministic for a cpu that does real time processing.
> Why does spinning come up here?

Because that is what you and others do to solve your particular
problem, but it does not work for a application like Motion Control,
where you have a hierarchical control mechanism running in different
threads with different priorities, different code execution times and
different deadline requirements.

> > Once again: The definition of Real-Time is determinism, not speed.
> 
> Yes then please do deterministic latencies. No exceptions. No soft
> realtime.

Preempt-RT is not about soft realtime. It gives you hard guarantees
versus your defined deadlines, but only when the deadline fits into
the limitations of the hardware and the OS.

Realtime is not about as fast as possible, it's about as fast as
specified. And that specified can be anything from nanoseconds to
seconds or even hours.

The point is, if you miss your deadline, no matter how big or small
that deadline is, it's going to be a violation of the guarantee.

So if your deadline requirements are in a range which cannot be
satisfied by PREEMPT-RT, then you need a different solution. Bare
metal busy looping or whatever.

> > You're design goal is completely different. Your design goal is
> > sacrifying a single CPU in order to have FAST response on an event and
> > high throughput at the same time. That has nothing to do with real
> > time. Go and read on OS theory to figure out why.
> 
> ???. Please stay on topic. Maybe you would have benefited from the
> clases on OS design that I taught. But maybe reading comprehension would
> be much more urgent.

Lets start to talk about the terms used.

Deadline: The maximum time between an event which triggers a
	  computation and the time when the computation has completed.

Latency:  There are two types of latency:

	  1) The delta between the desired time of starting a computation
	     and the actual start of the computation.

	  2) The delta between the theoretical execution time of a
	     computation and the actual execution time

So the mathematical definition of a real-time system is:

   Te:  Trigger time of the computation
   Td:  The relative from from Te to completion of the computation,
        aka deadline
   Tsl:	The delta between Te and actual start of the computation
   Trt:	The theoretical (optimal) runtime of the computation
   Trl:	The delta between Trt and the actual runtime of the computation

   Ergo:

   Te + Td < Te + Tsl + Trt + Trl

   Any OS which can guarantee that under all circumstances qualifies
   as a hard real time OS.
 
The hardware and the OS define the maximum values of

   Trt is defined by the optimal performance of the hardware when the
   computation runs completely undisturbed.

   Tsl and Trl are influenced by both hardware and OS. So we get
   maximum values for Tsl and Trl: Tsl_max and Trl_max.

So for a given computation you can compute the deadline guarantee of a
given system:

   Td < Tsl_max + Trt + Trl_max

The application specifies Td, i.e. the deadline. If your applications
deadline requirements do not fit into the deadline guarantee of the
given hardware/OS combination, then you need to look after a different
solution.

Preempt-RT has its limitations how small Tsl_max and Trl_max can
become. But within these limitations it gives you the guarantee that
you meet your deadline.

So for applications, and those are real world applications which use
Preempt-RT today, which can tolerate the slightly larger latencies,
but at the same time require the functionality of a full blown OS
including load balancing and scheduling of non-RT background work on
multicore systems, Preempt-RT is the appropriate choice.

For other applications, which aim for way lower Tsl_max and Trl_max,
NOHZ_FULL might be the solution if they can spend a core for a
particular computation.

There are certainly applications out there which need both. Preempt-RT
on one set of cores to handle the tasks with less demanding deadline
requirements and NOHZ_FULL for tasks which aim for extreme low Tsl_max
and Trl_max values.

There are other applications, which will need a microcontroller or a
FPGA assisting because neither Preempt-RT nor NOHZ_FULL nor any other
OS nor bare metal can meet their requirements due to the hardware
immanent latencies.

There is no one fits it all solution in the variety of use
cases.

Preempt-RT covers a large range of use cases throughout the industry,
but I'm happy to switch to a different approach if that approach can
handle the requirements which are handled by Preempt-RT today. To my
knowledge neither NOHZ_FULL nor Xenomai nor bare metal fits, but I'm
happy to be educated on that.

Thanks,

	tglx

Re: [Ksummit-discuss] [TECH TOPIC] Mainlining PREEMPT_RT

[Thomas Gleixner]

On Thu, 15 Oct 2015, Thomas Gleixner wrote:

> On Thu, 15 Oct 2015, Christoph Lameter wrote:
> > On Wed, 14 Oct 2015, Thomas Gleixner wrote:
> > 
> > > Sure, you just add another server to it and be done. That's not what
> > > people in the embedded space can do.
> > 
> > Why not? Seems that even small devices have multiple cores? I just got a
> > quad core cheapo phone for $50. Even the Intel Edison platform is dual
> > core. So it may be possible to have deterministic hard real time on one
> > core.
> 
> That fits into all current use cases of Preempt-RT how?
> 
> Please come up with a functional concept for applications like PLCs
> (Programmable Logic Controllers) where the end customer defines the
> application and therefor the number of involved thread and their
> particular deadline requirements. That would for a change start a
> useful discussion instead of your 'spend a CPU' mantra.
> 
> > > They need to run highly multi threaded applications on a small system
> > > and still need the guarantee that they meet their dead lines. They
> > > cannot afford the waste of a CPU busy spinning in order to process the
> > > next event. Neither they have the number of cores which would be
> > > needed nor they can afford the energy wasted by it.
> > 
> > There could be deterministic wake up logic for real time cpus? If you can
> > push off most of the effort on another cpu its likely possible to make the
> > behavior much more deterministic for a cpu that does real time processing.
> > Why does spinning come up here?
> 
> Because that is what you and others do to solve your particular
> problem, but it does not work for a application like Motion Control,
> where you have a hierarchical control mechanism running in different
> threads with different priorities, different code execution times and
> different deadline requirements.
> 
> > > Once again: The definition of Real-Time is determinism, not speed.
> > 
> > Yes then please do deterministic latencies. No exceptions. No soft
> > realtime.
> 
> Preempt-RT is not about soft realtime. It gives you hard guarantees
> versus your defined deadlines, but only when the deadline fits into
> the limitations of the hardware and the OS.
> 
> Realtime is not about as fast as possible, it's about as fast as
> specified. And that specified can be anything from nanoseconds to
> seconds or even hours.
> 
> The point is, if you miss your deadline, no matter how big or small
> that deadline is, it's going to be a violation of the guarantee.
> 
> So if your deadline requirements are in a range which cannot be
> satisfied by PREEMPT-RT, then you need a different solution. Bare
> metal busy looping or whatever.
> 
> > > You're design goal is completely different. Your design goal is
> > > sacrifying a single CPU in order to have FAST response on an event and
> > > high throughput at the same time. That has nothing to do with real
> > > time. Go and read on OS theory to figure out why.
> > 
> > ???. Please stay on topic. Maybe you would have benefited from the
> > clases on OS design that I taught. But maybe reading comprehension would
> > be much more urgent.
> 
> Lets start to talk about the terms used.
> 
> Deadline: The maximum time between an event which triggers a
> 	  computation and the time when the computation has completed.
> 
> Latency:  There are two types of latency:
> 
> 	  1) The delta between the desired time of starting a computation
> 	     and the actual start of the computation.
> 
> 	  2) The delta between the theoretical execution time of a
> 	     computation and the actual execution time
> 
> So the mathematical definition of a real-time system is:
> 
>    Te:  Trigger time of the computation
>    Td:  The relative from from Te to completion of the computation,
>         aka deadline
>    Tsl:	The delta between Te and actual start of the computation
>    Trt:	The theoretical (optimal) runtime of the computation
>    Trl:	The delta between Trt and the actual runtime of the computation
> 
>    Ergo:
> 
>    Te + Td < Te + Tsl + Trt + Trl

That of course wants to be

     Te + Td > Te + Tsl + Trt + Trl

> 
>    Any OS which can guarantee that under all circumstances qualifies
>    as a hard real time OS.
>  
> The hardware and the OS define the maximum values of
> 
>    Trt is defined by the optimal performance of the hardware when the
>    computation runs completely undisturbed.
> 
>    Tsl and Trl are influenced by both hardware and OS. So we get
>    maximum values for Tsl and Trl: Tsl_max and Trl_max.
> 
> So for a given computation you can compute the deadline guarantee of a
> given system:
> 
>    Td < Tsl_max + Trt + Trl_max

Ditto

     Td > Tsl_max + Trt + Trl_max

Time for bed. :)
 
> The application specifies Td, i.e. the deadline. If your applications
> deadline requirements do not fit into the deadline guarantee of the
> given hardware/OS combination, then you need to look after a different
> solution.
> 
> Preempt-RT has its limitations how small Tsl_max and Trl_max can
> become. But within these limitations it gives you the guarantee that
> you meet your deadline.
> 
> So for applications, and those are real world applications which use
> Preempt-RT today, which can tolerate the slightly larger latencies,
> but at the same time require the functionality of a full blown OS
> including load balancing and scheduling of non-RT background work on
> multicore systems, Preempt-RT is the appropriate choice.
> 
> For other applications, which aim for way lower Tsl_max and Trl_max,
> NOHZ_FULL might be the solution if they can spend a core for a
> particular computation.
> 
> There are certainly applications out there which need both. Preempt-RT
> on one set of cores to handle the tasks with less demanding deadline
> requirements and NOHZ_FULL for tasks which aim for extreme low Tsl_max
> and Trl_max values.
> 
> There are other applications, which will need a microcontroller or a
> FPGA assisting because neither Preempt-RT nor NOHZ_FULL nor any other
> OS nor bare metal can meet their requirements due to the hardware
> immanent latencies.
> 
> There is no one fits it all solution in the variety of use
> cases.
> 
> Preempt-RT covers a large range of use cases throughout the industry,
> but I'm happy to switch to a different approach if that approach can
> handle the requirements which are handled by Preempt-RT today. To my
> knowledge neither NOHZ_FULL nor Xenomai nor bare metal fits, but I'm
> happy to be educated on that.
> 
> Thanks,
> 
> 	tglx
> 
> _______________________________________________
> Ksummit-discuss mailing list
> Ksummit-discuss@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss
> 

Re: [Ksummit-discuss] [TECH TOPIC] Mainlining PREEMPT_RT

[Darren Hart]

On Tue, Oct 13, 2015 at 12:42:30PM -0400, Steven Rostedt wrote:
> 
> Now that it was announced in Dublin last week that mainlining
> PREEMPT_RT has funding, I would like to propose a tech topic to discuss
> what is needed to be done in the near future.
> 
> Thomas Gleixner and his minions will be doing the brunt of the work,
> but there's issues that need to be addressed with the rest of the
> kernel. The various topics would include:
> 
>  CPU hotplug
>  preempt_disable and local_irq_disable annotations
>  new timer_wheel implementation
>  simple waitqueues
>  trylock boosting
> 
> Several of the Real Time developers will be present at kernel summit. I
> think this would be a great opportunity to have them be able to sit
> down together and discuss the remaining issues.

I'm very interested in participating in this.

Is this a TECH topic or a CORE topic?

-- 
Darren Hart
Intel Open Source Technology Center

Re: [Ksummit-discuss] [TECH TOPIC] Mainlining PREEMPT_RT

[Darren Hart]

On Fri, Aug 05, 2016 at 03:32:38PM -0700, Darren Hart wrote:
> On Tue, Oct 13, 2015 at 12:42:30PM -0400, Steven Rostedt wrote:
> > 
> > Now that it was announced in Dublin last week that mainlining
> > PREEMPT_RT has funding, I would like to propose a tech topic to discuss
> > what is needed to be done in the near future.
> > 
> > Thomas Gleixner and his minions will be doing the brunt of the work,
> > but there's issues that need to be addressed with the rest of the
> > kernel. The various topics would include:
> > 
> >  CPU hotplug
> >  preempt_disable and local_irq_disable annotations
> >  new timer_wheel implementation
> >  simple waitqueues
> >  trylock boosting
> > 
> > Several of the Real Time developers will be present at kernel summit. I
> > think this would be a great opportunity to have them be able to sit
> > down together and discuss the remaining issues.
> 
> I'm very interested in participating in this.
> 
> Is this a TECH topic or a CORE topic?

Apologies, this was from 2015. I failed to recognize that when searching for a
PREEMPT_RT TECH TOPIC proposal before responding to Davidlohr's self nomination
suggesting this topic.

Shall we recycle this topic for 2016 as well?

-- 
Darren Hart
Intel Open Source Technology Center