Real-time micro-conference proposal for Linux Plumbers 2019

Real-time micro-conference proposal for Linux Plumbers 2019

[Daniel Bristot de Oliveira]

Hi all!

Last year we had a very fun and productive real-time micro-conference at the
Linux Plumbers Conference, so the idea is to repeat it this year!

Are you interested in participating? Do you have any suggestion of topic? Let us
know!

This is the current proposal of micro-conference, we can still edit it and add
your thoughts.

======================== PROPOSAL ==============================
Since 2004 a project has improved the Real-time and low-latency features for
Linux. This project has become know as PREEMPT_RT, formally the real-time patch.
Over the past decade, many parts of the PREEMPT RT became part of the official
Linux code base. Examples of what came from PREEMPT_RT include: Real-time
mutexes, high-resolution timers, lockdep, ftrace, RT scheduling, SCHED_DEADLINE,
RCU_PREEMPT, generic interrupts, priority inheritance futexes, threaded
interrupt handlers and more. The number of patches that needs integration has
been reduced on the last years, and the pieces left are now mature enough to
make its way into mainline Linux. This year could possibly be the year
PREEMPT_RT is merged (tm)!

In the final lap of this race, the last patches are on the way to be merged, but
there are still some pieces missing. When the merge occurs, the preempt-rt will
start to follow a new pace: the Linus one. So, it is possible to raise the
following discussions:

  1) The status of the merge, and how can we resolve the last issues that block
the merge;
  2) How can we improve the testing of the -rt, to follow the problems raised as
Linus tree advances;
  3) What's next?

Possible topics:
   - Status of the PREEMPT_RT Merge

   - Merge - what is missing and who can help?

   - How do we teach the rest of the kernel developers how not to break
     PREEMPT_RT?

   - Stable maintainers tools discussion & improvements.

   - Interrupt threads are RT and are not protected by the RT Throttling.
     How can we prevent interrupt thread starvation from a rogue RT task?

   - Improvements on full CPU isolation

   - Newer methods like proxy execution, hierarchical scheduler?

   - What tools can we add into tools/ that other kernel developers can use to
     test and learn about PREEMMPT_RT?

   - What tests can we add into tools/testing/selftests?

   - New tools for timing regression test, e.g. locking, overheads...

   - What kernel boot self-tests can be added?

   - Discuss various types of failures that can happen with PREEMPT_RT
     that normally would not happen in the vanilla kernel, e.g, with lockdep,
     preemption model.

I will suggest the continuation of the discussions of the topics I presented
last year, based in the results of the work I did this year (spoiler: I am doing
model verification in the kernel, and it is very efficient!).

Paul already told that he could talk about ongoing work to make RCU's forward
progress be more robust in overloaded cloud deployments. There will be some
connection to real-time response involving rcu_poll and the infamous RCU kthread
priority!

The continuation of the discussion of topics from last year's micro-conference,
including the development done during this (almost) year, are also welcome!

-- Daniel

Re: Real-time micro-conference proposal for Linux Plumbers 2019

[Sean V Kelley]

> On May 2, 2019, at 11:53 PM, Daniel Bristot de Oliveira <bristot@redhat.com> wrote:
> 
> Hi all!
> 
> Last year we had a very fun and productive real-time micro-conference at the
> Linux Plumbers Conference, so the idea is to repeat it this year!
> 
> Are you interested in participating? Do you have any suggestion of topic? Let us
> know!
> 
> This is the current proposal of micro-conference, we can still edit it and add
> your thoughts.
> 
> ======================== PROPOSAL ==============================
> Since 2004 a project has improved the Real-time and low-latency features for
> Linux. This project has become know as PREEMPT_RT, formally the real-time patch.
> Over the past decade, many parts of the PREEMPT RT became part of the official
> Linux code base. Examples of what came from PREEMPT_RT include: Real-time
> mutexes, high-resolution timers, lockdep, ftrace, RT scheduling, SCHED_DEADLINE,
> RCU_PREEMPT, generic interrupts, priority inheritance futexes, threaded
> interrupt handlers and more. The number of patches that needs integration has
> been reduced on the last years, and the pieces left are now mature enough to
> make its way into mainline Linux. This year could possibly be the year
> PREEMPT_RT is merged (tm)!
> 
> In the final lap of this race, the last patches are on the way to be merged, but
> there are still some pieces missing. When the merge occurs, the preempt-rt will
> start to follow a new pace: the Linus one. So, it is possible to raise the
> following discussions:
> 
>  1) The status of the merge, and how can we resolve the last issues that block
> the merge;
>  2) How can we improve the testing of the -rt, to follow the problems raised as
> Linus tree advances;
>  3) What's next?
> 
> Possible topics:
>   - Status of the PREEMPT_RT Merge
> 
>   - Merge - what is missing and who can help?
> 
>   - How do we teach the rest of the kernel developers how not to break
>     PREEMPT_RT?
> 
>   - Stable maintainers tools discussion & improvements.
> 
>   - Interrupt threads are RT and are not protected by the RT Throttling.
>     How can we prevent interrupt thread starvation from a rogue RT task?
> 
>   - Improvements on full CPU isolation
> 
>   - Newer methods like proxy execution, hierarchical scheduler?
> 
>   - What tools can we add into tools/ that other kernel developers can use to
>     test and learn about PREEMMPT_RT?
> 
>   - What tests can we add into tools/testing/selftests?
> 
>   - New tools for timing regression test, e.g. locking, overheads...
> 
>   - What kernel boot self-tests can be added?
> 
>   - Discuss various types of failures that can happen with PREEMPT_RT
>     that normally would not happen in the vanilla kernel, e.g, with lockdep,
>     preemption model.


I’d definitely be interested in many of these topics at LPC.  Hoping to make it.

Thanks,

Sean


> 
> I will suggest the continuation of the discussions of the topics I presented
> last year, based in the results of the work I did this year (spoiler: I am doing
> model verification in the kernel, and it is very efficient!).
> 
> Paul already told that he could talk about ongoing work to make RCU's forward
> progress be more robust in overloaded cloud deployments. There will be some
> connection to real-time response involving rcu_poll and the infamous RCU kthread
> priority!
> 
> The continuation of the discussion of topics from last year's micro-conference,
> including the development done during this (almost) year, are also welcome!
> 
> -- Daniel

Re: Real-time micro-conference proposal for Linux Plumbers 2019

[Stéphane Ancelot]

Hi guys,

That's a good job. Regarding this topic, I would be interested in 
knowing the future of network drivers in preempt RT.

At the moment, it is mandatory to hack any network driver, if you want 
to achieve realtime.

This would imply inhibit some layer functions of the driver (eg vlan, 
etc ...).

The NAPI model in network drivers is not really compatible with RT tasks.

To achieve this task , I would suggest network driver architecture must 
adopt an enhanced driver model .

This may be some work planning.

Regards,

S.Ancelot


Le 03/05/2019 à 08:53, Daniel Bristot de Oliveira a écrit :
> Hi all!
>
> Last year we had a very fun and productive real-time micro-conference at the
> Linux Plumbers Conference, so the idea is to repeat it this year!
>
> Are you interested in participating? Do you have any suggestion of topic? Let us
> know!
>
> This is the current proposal of micro-conference, we can still edit it and add
> your thoughts.
>
> ======================== PROPOSAL ==============================
> Since 2004 a project has improved the Real-time and low-latency features for
> Linux. This project has become know as PREEMPT_RT, formally the real-time patch.
> Over the past decade, many parts of the PREEMPT RT became part of the official
> Linux code base. Examples of what came from PREEMPT_RT include: Real-time
> mutexes, high-resolution timers, lockdep, ftrace, RT scheduling, SCHED_DEADLINE,
> RCU_PREEMPT, generic interrupts, priority inheritance futexes, threaded
> interrupt handlers and more. The number of patches that needs integration has
> been reduced on the last years, and the pieces left are now mature enough to
> make its way into mainline Linux. This year could possibly be the year
> PREEMPT_RT is merged (tm)!
>
> In the final lap of this race, the last patches are on the way to be merged, but
> there are still some pieces missing. When the merge occurs, the preempt-rt will
> start to follow a new pace: the Linus one. So, it is possible to raise the
> following discussions:
>
>    1) The status of the merge, and how can we resolve the last issues that block
> the merge;
>    2) How can we improve the testing of the -rt, to follow the problems raised as
> Linus tree advances;
>    3) What's next?
>
> Possible topics:
>     - Status of the PREEMPT_RT Merge
>
>     - Merge - what is missing and who can help?
>
>     - How do we teach the rest of the kernel developers how not to break
>       PREEMPT_RT?
>
>     - Stable maintainers tools discussion & improvements.
>
>     - Interrupt threads are RT and are not protected by the RT Throttling.
>       How can we prevent interrupt thread starvation from a rogue RT task?
>
>     - Improvements on full CPU isolation
>
>     - Newer methods like proxy execution, hierarchical scheduler?
>
>     - What tools can we add into tools/ that other kernel developers can use to
>       test and learn about PREEMMPT_RT?
>
>     - What tests can we add into tools/testing/selftests?
>
>     - New tools for timing regression test, e.g. locking, overheads...
>
>     - What kernel boot self-tests can be added?
>
>     - Discuss various types of failures that can happen with PREEMPT_RT
>       that normally would not happen in the vanilla kernel, e.g, with lockdep,
>       preemption model.
>
> I will suggest the continuation of the discussions of the topics I presented
> last year, based in the results of the work I did this year (spoiler: I am doing
> model verification in the kernel, and it is very efficient!).
>
> Paul already told that he could talk about ongoing work to make RCU's forward
> progress be more robust in overloaded cloud deployments. There will be some
> connection to real-time response involving rcu_poll and the infamous RCU kthread
> priority!
>
> The continuation of the discussion of topics from last year's micro-conference,
> including the development done during this (almost) year, are also welcome!
>
> -- Daniel

Re: Real-time micro-conference proposal for Linux Plumbers 2019

[Daniel Bristot de Oliveira]

Hi Stephane!

On 5/7/19 8:46 AM, Stéphane Ancelot wrote:
> Hi guys,
> 
> That's a good job. Regarding this topic, I would be interested in knowing the
> future of network drivers in preempt RT.
> 
> At the moment, it is mandatory to hack any network driver, if you want to
> achieve realtime.
> 
> This would imply inhibit some layer functions of the driver (eg vlan, etc ...).
> 
> The NAPI model in network drivers is not really compatible with RT tasks.
> 
> To achieve this task , I would suggest network driver architecture must adopt an
> enhanced driver model .
> 
> This may be some work planning.

Sorry for the delay, I am attending to an academic conference (these conferences
drains my brain out).

This is an interesting topic! Will you attend to the Plumbers? Do you have any
suggestions on how to change the NAPI model or how to change the driver
architecture to be more "real-time friendly?"

-- Daniel


> Regards,
> 
> S.Ancelot
> 
> 
> Le 03/05/2019 à 08:53, Daniel Bristot de Oliveira a écrit :
>> Hi all!
>>
>> Last year we had a very fun and productive real-time micro-conference at the
>> Linux Plumbers Conference, so the idea is to repeat it this year!
>>
>> Are you interested in participating? Do you have any suggestion of topic? Let us
>> know!
>>
>> This is the current proposal of micro-conference, we can still edit it and add
>> your thoughts.
>>
>> ======================== PROPOSAL ==============================
>> Since 2004 a project has improved the Real-time and low-latency features for
>> Linux. This project has become know as PREEMPT_RT, formally the real-time patch.
>> Over the past decade, many parts of the PREEMPT RT became part of the official
>> Linux code base. Examples of what came from PREEMPT_RT include: Real-time
>> mutexes, high-resolution timers, lockdep, ftrace, RT scheduling, SCHED_DEADLINE,
>> RCU_PREEMPT, generic interrupts, priority inheritance futexes, threaded
>> interrupt handlers and more. The number of patches that needs integration has
>> been reduced on the last years, and the pieces left are now mature enough to
>> make its way into mainline Linux. This year could possibly be the year
>> PREEMPT_RT is merged (tm)!
>>
>> In the final lap of this race, the last patches are on the way to be merged, but
>> there are still some pieces missing. When the merge occurs, the preempt-rt will
>> start to follow a new pace: the Linus one. So, it is possible to raise the
>> following discussions:
>>
>>    1) The status of the merge, and how can we resolve the last issues that block
>> the merge;
>>    2) How can we improve the testing of the -rt, to follow the problems raised as
>> Linus tree advances;
>>    3) What's next?
>>
>> Possible topics:
>>     - Status of the PREEMPT_RT Merge
>>
>>     - Merge - what is missing and who can help?
>>
>>     - How do we teach the rest of the kernel developers how not to break
>>       PREEMPT_RT?
>>
>>     - Stable maintainers tools discussion & improvements.
>>
>>     - Interrupt threads are RT and are not protected by the RT Throttling.
>>       How can we prevent interrupt thread starvation from a rogue RT task?
>>
>>     - Improvements on full CPU isolation
>>
>>     - Newer methods like proxy execution, hierarchical scheduler?
>>
>>     - What tools can we add into tools/ that other kernel developers can use to
>>       test and learn about PREEMMPT_RT?
>>
>>     - What tests can we add into tools/testing/selftests?
>>
>>     - New tools for timing regression test, e.g. locking, overheads...
>>
>>     - What kernel boot self-tests can be added?
>>
>>     - Discuss various types of failures that can happen with PREEMPT_RT
>>       that normally would not happen in the vanilla kernel, e.g, with lockdep,
>>       preemption model.
>>
>> I will suggest the continuation of the discussions of the topics I presented
>> last year, based in the results of the work I did this year (spoiler: I am doing
>> model verification in the kernel, and it is very efficient!).
>>
>> Paul already told that he could talk about ongoing work to make RCU's forward
>> progress be more robust in overloaded cloud deployments. There will be some
>> connection to real-time response involving rcu_poll and the infamous RCU kthread
>> priority!
>>
>> The continuation of the discussion of topics from last year's micro-conference,
>> including the development done during this (almost) year, are also welcome!
>>
>> -- Daniel

RE: Real-time micro-conference proposal for Linux Plumbers 2019

[Tiejun Chen]

Hi Daniel,

I hope we can discuss if-how Preempt-RT kernel can be tuned to benefit container/docker. It would involve any potential new interfaces, tools/utilities, etc, and how to make Preempt-RT work very well with some Linux secure features.  Actually I tried to discuss this last year but for some reasons I did attend real-time micro-conference 2018, unfortunately. I hope we can have such a topic and I'd like to work on that as well.

Thanks
Tiejun

> -----Original Message-----
> From: linux-rt-users-owner@vger.kernel.org <linux-rt-users-
> owner@vger.kernel.org> On Behalf Of Daniel Bristot de Oliveira
> Sent: Friday, May 3, 2019 2:53 PM
> To: linux-rt-users@vger.kernel.org
> Subject: Real-time micro-conference proposal for Linux Plumbers 2019
> 
> Hi all!
> 
> Last year we had a very fun and productive real-time micro-conference at the
> Linux Plumbers Conference, so the idea is to repeat it this year!
> 
> Are you interested in participating? Do you have any suggestion of topic? Let us
> know!
> 
> This is the current proposal of micro-conference, we can still edit it and add your
> thoughts.
> 
> ======================== PROPOSAL
> ============================== Since 2004 a project has improved the
> Real-time and low-latency features for Linux. This project has become know as
> PREEMPT_RT, formally the real-time patch.
> Over the past decade, many parts of the PREEMPT RT became part of the official
> Linux code base. Examples of what came from PREEMPT_RT include: Real-time
> mutexes, high-resolution timers, lockdep, ftrace, RT scheduling,
> SCHED_DEADLINE, RCU_PREEMPT, generic interrupts, priority inheritance futexes,
> threaded interrupt handlers and more. The number of patches that needs
> integration has been reduced on the last years, and the pieces left are now mature
> enough to make its way into mainline Linux. This year could possibly be the year
> PREEMPT_RT is merged (tm)!
> 
> In the final lap of this race, the last patches are on the way to be merged, but
> there are still some pieces missing. When the merge occurs, the preempt-rt will
> start to follow a new pace: the Linus one. So, it is possible to raise the following
> discussions:
> 
>   1) The status of the merge, and how can we resolve the last issues that block the
> merge;
>   2) How can we improve the testing of the -rt, to follow the problems raised as
> Linus tree advances;
>   3) What's next?
> 
> Possible topics:
>    - Status of the PREEMPT_RT Merge
> 
>    - Merge - what is missing and who can help?
> 
>    - How do we teach the rest of the kernel developers how not to break
>      PREEMPT_RT?
> 
>    - Stable maintainers tools discussion & improvements.
> 
>    - Interrupt threads are RT and are not protected by the RT Throttling.
>      How can we prevent interrupt thread starvation from a rogue RT task?
> 
>    - Improvements on full CPU isolation
> 
>    - Newer methods like proxy execution, hierarchical scheduler?
> 
>    - What tools can we add into tools/ that other kernel developers can use to
>      test and learn about PREEMMPT_RT?
> 
>    - What tests can we add into tools/testing/selftests?
> 
>    - New tools for timing regression test, e.g. locking, overheads...
> 
>    - What kernel boot self-tests can be added?
> 
>    - Discuss various types of failures that can happen with PREEMPT_RT
>      that normally would not happen in the vanilla kernel, e.g, with lockdep,
>      preemption model.
> 
> I will suggest the continuation of the discussions of the topics I presented last year,
> based in the results of the work I did this year (spoiler: I am doing model
> verification in the kernel, and it is very efficient!).
> 
> Paul already told that he could talk about ongoing work to make RCU's forward
> progress be more robust in overloaded cloud deployments. There will be some
> connection to real-time response involving rcu_poll and the infamous RCU
> kthread priority!
> 
> The continuation of the discussion of topics from last year's micro-conference,
> including the development done during this (almost) year, are also welcome!
> 
> -- Daniel

Re: Real-time micro-conference proposal for Linux Plumbers 2019

[Daniel Bristot de Oliveira]

Hi Tiejun!

On 5/8/19 6:37 AM, Tiejun Chen wrote:
> Hi Daniel,
> 
> I hope we can discuss if-how Preempt-RT kernel can be tuned to benefit container/docker. It would involve any potential new interfaces, tools/utilities, etc, and how to make Preempt-RT work very well with some Linux secure features.  Actually I tried to discuss this last year but for some reasons I did attend real-time micro-conference 2018, unfortunately. I hope we can have such a topic and I'd like to work on that as well.

Yeah, this is a hot topic! Are you planning to attend to Plumber this year? If
so, would you mind to prepare 3 or 4 slides introducing the problem to "warm-up"
the discussion at the MC?

Thanks!

-- Daniel

> 
> Thanks
> Tiejun

RE: Real-time micro-conference proposal for Linux Plumbers 2019

[Tiejun Chen]

Dear Daniel,

Not pretty sure if I will be there currently but I will try my best. I will work on the preparation of these topics.

Thanks
Tiejun

> -----Original Message-----
> From: Daniel Bristot de Oliveira <daniel@bristot.me>
> Sent: Thursday, May 9, 2019 12:56 AM
> To: Tiejun Chen <tiejunc@vmware.com>; Daniel Bristot de Oliveira
> <bristot@redhat.com>; linux-rt-users@vger.kernel.org
> Subject: Re: Real-time micro-conference proposal for Linux Plumbers 2019
> 
> Hi Tiejun!
> 
> On 5/8/19 6:37 AM, Tiejun Chen wrote:
> > Hi Daniel,
> >
> > I hope we can discuss if-how Preempt-RT kernel can be tuned to benefit
> container/docker. It would involve any potential new interfaces, tools/utilities, etc,
> and how to make Preempt-RT work very well with some Linux secure features.
> Actually I tried to discuss this last year but for some reasons I did attend real-time
> micro-conference 2018, unfortunately. I hope we can have such a topic and I'd
> like to work on that as well.
> 
> Yeah, this is a hot topic! Are you planning to attend to Plumber this year? If so,
> would you mind to prepare 3 or 4 slides introducing the problem to "warm-up"
> the discussion at the MC?
> 
> Thanks!
> 
> -- Daniel
> 
> >
> > Thanks
> > Tiejun

Re: Real-time micro-conference proposal for Linux Plumbers 2019

[Daniel Bristot de Oliveira]

On 5/14/19 3:26 PM, Tiejun Chen wrote:
> Dear Daniel,
> 
> Not pretty sure if I will be there currently but I will try my best. I will work on the preparation of these topics.

Good! Let me know if have any update!

Thanks!

RE: Real-time micro-conference proposal for Linux Plumbers 2019

[Tiejun Chen]

Daniel,

Have you submitted your proposal? I cannot find such a RT MC track through LPC 2019 website.

Thanks
Tiejun

Re: Real-time micro-conference proposal for Linux Plumbers 2019

[Daniel Bristot de Oliveira]

On 13/06/2019 04:57, Tiejun Chen wrote:
> Daniel,
> 
> Have you submitted your proposal? I cannot find such a RT MC track through LPC 2019 website.

Hi Tiejun

The list in the LPC website shows the MCs that were already approved. The RT MC
is still under evaluation.

I will share with you all the reply from LPC committee as soon as I receive it.

Thanks!

-- Daniel

> Thanks
> Tiejun
> 

Re: Real-time micro-conference proposal for Linux Plumbers 2019

[Daniel Bristot de Oliveira]

On 5/3/19 8:53 AM, Daniel Bristot de Oliveira wrote:
> Hi all!
> 

Hi all!

Good news from OSPM!

I just talked to Frederic Weisbecker, and he is proposing talking about the
softirq rework that we need - as a step to the -rt merge!

Dhaval Giani is also mentioning the possibility of talking about their needs
w.r.t proxy execution!

Anyone else with a good idea? stable-rt people??? :-)

-- Daniel

Re: Real-time micro-conference proposal for Linux Plumbers 2019

[Arnaldo Carvalho de Melo]

Em Tue, May 21, 2019 at 03:07:54PM +0200, Daniel Bristot de Oliveira escreveu:
> 
> 
> On 5/3/19 8:53 AM, Daniel Bristot de Oliveira wrote:
> > Hi all!
> > 
> 
> Hi all!
> 
> Good news from OSPM!
> 
> I just talked to Frederic Weisbecker, and he is proposing talking about the
> softirq rework that we need - as a step to the -rt merge!
> 
> Dhaval Giani is also mentioning the possibility of talking about their needs
> w.r.t proxy execution!
> 
> Anyone else with a good idea? stable-rt people??? :-)

Has anyone thought about how BPF interacts with CONFIG_PREEMPT_RT_FULL?

- Arnaldo

Re: Real-time micro-conference proposal for Linux Plumbers 2019

[Daniel Bristot de Oliveira]

On 5/22/19 3:05 AM, Arnaldo Carvalho de Melo wrote:
> Em Tue, May 21, 2019 at 03:07:54PM +0200, Daniel Bristot de Oliveira escreveu:
>>
>>
>> On 5/3/19 8:53 AM, Daniel Bristot de Oliveira wrote:
>>> Hi all!
>>>
>>
>> Hi all!
>>
>> Good news from OSPM!
>>
>> I just talked to Frederic Weisbecker, and he is proposing talking about the
>> softirq rework that we need - as a step to the -rt merge!
>>
>> Dhaval Giani is also mentioning the possibility of talking about their needs
>> w.r.t proxy execution!
>>
>> Anyone else with a good idea? stable-rt people??? :-)
> 
> Has anyone thought about how BPF interacts with CONFIG_PREEMPT_RT_FULL?

I've heard that there are some preempt_disable sections around BPF code, right?

That is interesting too! (as people are more and more prone to use BPF to do
_put a complex thing here_).

-- Daniel

Re: Real-time micro-conference proposal for Linux Plumbers 2019

[Daniel Bristot de Oliveira]

On 5/3/19 8:53 AM, Daniel Bristot de Oliveira wrote:
> Hi all!

More good news! Peter will do one slide about RT and core-sched! I updated the
proposal with all the suggestions!

Thanks
-- Daniel