SCHED_DEADLINE as user

SCHED_DEADLINE as user

[Tim Blechmann]

hi all,

i wonder, what's the preferred way to obtain SCHED_DEADLINE privileges
as non-root user?
for SCHED_RR/SCHED_FIFO i'm typically using pam_limits/limits.conf, but
i haven't found any resources on how SCHED_DEADLINE can be obtained ...

thanks in advance,
tim

Re: SCHED_DEADLINE as user

[Daniel Bristot de Oliveira]

On 08/15/2018 08:08 AM, Tim Blechmann wrote:
> hi all,
> 
> i wonder, what's the preferred way to obtain SCHED_DEADLINE privileges
> as non-root user?

Hi Tim,

It is still not possible to use sched deadline as a non-root user :-(.

Thanks
-- Daniel

> for SCHED_RR/SCHED_FIFO i'm typically using pam_limits/limits.conf, but
> i haven't found any resources on how SCHED_DEADLINE can be obtained ...
> 
> thanks in advance,
> tim
> 

Re: SCHED_DEADLINE as user

[Juri Lelli]

Hi Tim,

On 15/08/18 10:21, Daniel Bristot de Oliveira wrote:
> 
> 
> On 08/15/2018 08:08 AM, Tim Blechmann wrote:
> > hi all,
> > 
> > i wonder, what's the preferred way to obtain SCHED_DEADLINE privileges
> > as non-root user?
> 
> Hi Tim,
> 
> It is still not possible to use sched deadline as a non-root user :-(.

Indeed, but Tim, is this hindering you from experimenting/using
DEADLINE?

Asking because we might want to have a look at how to relax the current
privileges requirement.

Best,

- Juri

Re: SCHED_DEADLINE as user

[Tim Blechmann]

[-- Attachment #1.1: Type: text/plain, Size: 659 bytes --]

hi juri,

>>> i wonder, what's the preferred way to obtain SCHED_DEADLINE privileges
>>> as non-root user?
>>
>> Hi Tim,
>>
>> It is still not possible to use sched deadline as a non-root user :-(.
> 
> Indeed, but Tim, is this hindering you from experimenting/using
> DEADLINE?

it's a showstopper for using it in audio applications, which are running
as user.
audio applications typically use SCHED_FIFO. however they are
rate-monotonic by design, so i wanted to evaluate how SCHED_DEADLINE
behaves here.

> Asking because we might want to have a look at how to relax the current
> privileges requirement.

please :)

cheers,
tim


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 841 bytes --]

Re: SCHED_DEADLINE as user

[Juri Lelli]

On 17/08/18 18:08, Tim Blechmann wrote:
> hi juri,
> 
> >>> i wonder, what's the preferred way to obtain SCHED_DEADLINE privileges
> >>> as non-root user?
> >>
> >> Hi Tim,
> >>
> >> It is still not possible to use sched deadline as a non-root user :-(.
> > 
> > Indeed, but Tim, is this hindering you from experimenting/using
> > DEADLINE?
> 
> it's a showstopper for using it in audio applications, which are running
> as user.
> audio applications typically use SCHED_FIFO. however they are
> rate-monotonic by design, so i wanted to evaluate how SCHED_DEADLINE
> behaves here.

Right. Exactly what I started to think about lately as well.

> 
> > Asking because we might want to have a look at how to relax the current
> > privileges requirement.
> 
> please :)

Anything to try attracting new users. :-P

More seriously, could you please describe in a bit more details your
specific use case? Based on JACK I guess? Is it the application you are
interested into public available somewhere?

Thanks,

- Juri

Re: SCHED_DEADLINE as user

[Juri Lelli]

On 18/08/18 15:26, Tim Blechmann wrote:
> >> audio applications typically use SCHED_FIFO. however they are
> >> rate-monotonic by design, so i wanted to evaluate how SCHED_DEADLINE
> >> behaves here.
> > 
> > Right. Exactly what I started to think about lately as well.
> > 
> >>
> >>> Asking because we might want to have a look at how to relax the current
> >>> privileges requirement.
> >>
> >> please :)
> > 
> > Anything to try attracting new users. :-P
> > 
> > More seriously, could you please describe in a bit more details your
> > specific use case? Based on JACK I guess? Is it the application you are
> > interested into public available somewhere?
> 
> it could be based on JACK or natively on ALSA. furthermore the fun part
> starts when trying to distribute work to multiple threads: during the
> time slice of one audio callback (could be as low as 1.3ms), we need to
> wake helper threads and (busy?)wait to collect all the results. (while
> one of the applications is open source, it's not entirely
> straight-forward to use)
> 
> for me it's more an investigation if SCHED_DEADLINE is actually feasible
> in real-world audio application these days. tommaso used to have a
> research prototype [1], built directly into JACK. however currently i'd
> be more interested in using it in ALSA. but yeah, it's more an
> investigation as SCHED_FIFO does a decent job for my use case ...

I (almost) see. Thanks for the details.

> [1] http://retis.sssup.it/~tommaso/publications/LAC-2011.pdf

So, this was IIRC using AQuoSA hierarchical RT on EDF scheduler
implementation; which it was handy to easy the task of assigning CBS
server parameters to JACK clients (graph).

In your case - assuming you could already use DEADLINE for !root - do
you think you would need to use the same interface for the multiple
threads you are distributing work among? Or would you be fine with
assigning runtime/deadline for each one of those?

BTW, cc-ing Tommaso and Luca. Discussion started from
https://www.spinics.net/lists/linux-rt-users/msg19296.html

Re: SCHED_DEADLINE as user

[Tim Blechmann]

[-- Attachment #1.1: Type: text/plain, Size: 1712 bytes --]

> In your case - assuming you could already use DEADLINE for !root - do
> you think you would need to use the same interface for the multiple
> threads you are distributing work among? Or would you be fine with
> assigning runtime/deadline for each one of those?

that's a tricky question and depends on the use-case: for me it's "good
enough" to use something like:

main audio thread:
while running:
  snd_pcm_read  // blocks until data is delivered from ALSA
  wake helper(s)
  do work()
  sync_with_helper(s)
  snd_pcm_write // blocks until data is delivered to ALSA

alsa is basically driven by the hardware, which delivers/requests data
every ~1.3ms.


distributing the work is a little tricky: to avoid excessive scheduling
overhead, the worker threads are typically woken up (after snd_pcm_read)
and the result is collected where sync_with_helper() typically boils
down to busy waiting. in this case, the workers as rate-monotonic in a
similar manner as the main audio thread.

one could also use lock-free queues with semaphores to wake only as many
threads as needed for the graph topology (which can depend on user
input). in this case SCHED_FIFO sounds more suitable.

--

from a practical point of view: i'm not targeting a safety critical
system. one advantage i'm seeing of DEADLINE over FIFO/RR is that it's
easier to prevent lockups (e.g. when a user overloads the system). in
the linux audio world this is typically done by a watchdog thread. the
other part of the unix world (mach) is using time-constraint threads by
default for audio use cases. so i'd assume that DEADLINE would free me
from the need to spawn the watchdog thread ...

cheers,
tim


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 841 bytes --]

Re: SCHED_DEADLINE as user

[Daniel Bristot de Oliveira]

On 08/20/2018 01:54 PM, Tim Blechmann wrote:
>> In your case - assuming you could already use DEADLINE for !root - do
>> you think you would need to use the same interface for the multiple
>> threads you are distributing work among? Or would you be fine with
>> assigning runtime/deadline for each one of those?
> 
> that's a tricky question and depends on the use-case: for me it's "good
> enough" to use something like:
> 
> main audio thread:
> while running:
>   snd_pcm_read  // blocks until data is delivered from ALSA
>   wake helper(s)
>   do work()
>   sync_with_helper(s)
>   snd_pcm_write // blocks until data is delivered to ALSA
> 
> alsa is basically driven by the hardware, which delivers/requests data
> every ~1.3ms.

Hi Tim!

> 
> distributing the work is a little tricky: to avoid excessive scheduling
> overhead, the worker threads are typically woken up (after snd_pcm_read)
> and the result is collected where sync_with_helper() typically boils
> down to busy waiting. in this case, the workers as rate-monotonic in a
> similar manner as the main audio thread.

When you say "rate-monitonic", do you mean that the worker threads with
smaller period (high rate, so) have a higher fixed-priority? If so, what
are the other periods?

as sync_with_helper() does a busy-wait, the "main audio thread" has a
relative lower priority than the worker threads, right?

> one could also use lock-free queues with semaphores to wake only as many
> threads as needed for the graph topology (which can depend on user
> input). in this case SCHED_FIFO sounds more suitable.
> --
> 
> from a practical point of view: i'm not targeting a safety critical
> system. one advantage i'm seeing of DEADLINE over FIFO/RR is that it's
> easier to prevent lockups (e.g. when a user overloads the system). in
> the linux audio world this is typically done by a watchdog thread. the
> other part of the unix world (mach) is using time-constraint threads by
> default for audio use cases. so i'd assume that DEADLINE would free me
> from the need to spawn the watchdog thread ...

So, your main wish here is to avoid having a watchdog thread to
"throttle" misbehaving workload/setup?

For curiosity, what are the "time-constraints" setup used on mach?

-- Daniel

Re: SCHED_DEADLINE as user

[Tim Blechmann]

[-- Attachment #1.1: Type: text/plain, Size: 2863 bytes --]

>> main audio thread:
>> while running:
>>   snd_pcm_read  // blocks until data is delivered from ALSA
>>   wake helper(s)
>>   do work()
>>   sync_with_helper(s)
>>   snd_pcm_write // blocks until data is delivered to ALSA
>>
>> alsa is basically driven by the hardware, which delivers/requests data
>> every ~1.3ms.

>> distributing the work is a little tricky: to avoid excessive scheduling
>> overhead, the worker threads are typically woken up (after snd_pcm_read)
>> and the result is collected where sync_with_helper() typically boils
>> down to busy waiting. in this case, the workers as rate-monotonic in a
>> similar manner as the main audio thread.
> 
> When you say "rate-monitonic", do you mean that the worker threads with
> smaller period (high rate, so) have a higher fixed-priority? If so, what
> are the other periods?

the period will be the same period: i wake up the workers once per
period of the main thread, distribute the work and collect it at the
end. so "period" is basically the same, "runtime" a little lower.


> as sync_with_helper() does a busy-wait, the "main audio thread" has a
> relative lower priority than the worker threads, right?

typically all threads have the same priority and there will be at most N
threads for N physical CPU cores. so each thread will have a CPU to run
on and no thread will be preempted.


> So, your main wish here is to avoid having a watchdog thread to
> "throttle" misbehaving workload/setup?

mainly ... and nerdy research if SCHED_DEADLINE is useful for audio
applications ... ;)


> For curiosity, what are the "time-constraints" setup used on mach?

it's pretty similar to the SCHED_DEADLINE constraints:

from:
https://opensource.apple.com/source/xnu/xnu-124.7/osfmk/mach/thread_policy.h.auto.html

> /*
>  * THREAD_TIME_CONSTRAINT_POLICY:
>  *
>  * This scheduling mode is for threads which have real time
>  * constraints on their execution.
>  *
>  * Parameters:
>  *
>  * period: This is the nominal amount of time between separate
>  * processing arrivals, specified in absolute time units.  A
>  * value of 0 indicates that there is no inherent periodicity in
>  * the computation.
>  *
>  * computation: This is the nominal amount of computation
>  * time needed during a separate processing arrival, specified
>  * in absolute time units.
>  *
>  * constraint: This is the maximum amount of real time that
>  * may elapse from the start of a separate processing arrival
>  * to the end of computation for logically correct functioning,
>  * specified in absolute time units.  Must be (>= computation).
>  * Note that latency = (constraint - computation).
>  *
>  * preemptible: This indicates that the computation may be
>  * interrupted, subject to the constraint specified above.
>  */

cheers,
tim


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 841 bytes --]

Re: SCHED_DEADLINE as user

[Tommaso Cucinotta]

Hi all,

+cc-ing Alessio, who has been working on this just recently on Android (will tell more answering on a different msg).

On 20/08/2018 18:45, Tim Blechmann wrote:
> the mach API also has this notion of "period == 0", but apart from that it's quite similar. 

in terms of API, XNU time constrained threads look like DEADLINE, however in terms of kernel behavior/implementation 
behind that, it's something completely different: last time I had a glance at this, it was .... ehm .... probably just 
during that old Linux Audio Conference (LAC) back in 2011, where the take away was: 1) in the kernel there's nothing 
like the CBS or deadline-based scheduling, but just a heuristic boosting a bit more those tasks, accompanied by usual 
heuristics to de-boost in case the task becomes CPU-intensive (similar to CFS heuristics for interactive-vs-batch); 2) 
the only user of that API seemed to be some sound daemon on OS-X, that had an almost random value set as "runtime", 
accompanied by a comment "this just works" ;-P

But, it was long ago, if anyone has a pointer to what XNU does with time constrained threads in these days....

> apart from that, i'm only missing two features:
>
> * using SCHED_DEADLINE from user applications is kind of a showstopper
> for me

unprivileged tasks using DEADLINE remained a request for quite a long time -- that old single-processor implementation 
(AQuoSA) we used long ago with JACK had such a feature, that was implementing this model:

   http://retis.sssup.it/~tommaso/papers/rtas08.php

the criticality for DEADLINE is that those tasks preempt RT99 ones, so even with controlled runtime/deadline boundaries....
> * a nicer glibc wrapper, that allows me to use pthreads rather than
> using `syscall`: it's all pretty easy to obtain the `tid` for a calling
> thread (gettid), but it seems to be impossible to obtain the `tid` from
>   `pthread_t` without some hacks which are way too dirty to share in
> public ...

well-known "pain" as well, thanks.

     T.

-- 
Tommaso Cucinotta, Computer Engineering PhD
Associate Professor at the Real-Time Systems Laboratory (ReTiS)
Scuola Superiore Sant'Anna, Pisa, Italy
http://retis.sssup.it/people/tommaso

Re: SCHED_DEADLINE as user

[Tommaso Cucinotta]

[copying Alessio]

Hi,

On 20/08/2018 13:54, Tim Blechmann wrote:
> that's a tricky question and depends on the use-case: for me it's "good
> enough" to use something like:
>
> main audio thread:
> while running:
>    snd_pcm_read  // blocks until data is delivered from ALSA
>    wake helper(s)
>    do work()
>    sync_with_helper(s)
>    snd_pcm_write // blocks until data is delivered to ALSA
>
> alsa is basically driven by the hardware, which delivers/requests data
> every ~1.3ms.

this is exactly the use-case we considered in that old paper back in LAC 2011, and let me just add that there is more 
experimentation and details and results in Giacomo's MSc thesis:

   https://retis.sssup.it/?q=node/77

FYI, we restarted activities on this use-case just recently, considering a couple of further dimensions:

1) support for heterogeneous platforms with the aim of achieving a sound support for Arm big.LITTLE on Android, and 
apply it to the Android video & audio processing pipelines;

2) proper consideration of power switching capabilities of the platform (assuming the user would block the CPU frequency 
to its maximum is not really multimedia-friendly);

3) support for multi-threaded processing workflows on multi-cores, as typically needed by (high-performance) audio 
applications, either with a single multi-threaded audio processing client, or, e.g., with JACK, with a DAG of 
computations that can take advantage of underlying multi-core processing.

In this context, we've been playing also with the hierarchical extension to SCHED_DEADLINE we sent last year on LKML

   https://lkml.org/lkml/2017/3/31/658

I hope we can share some good write-up & experimental results pretty soon about some of the above...
> distributing the work is a little tricky: to avoid excessive scheduling
> overhead, the worker threads are typically woken up (after snd_pcm_read)
> and the result is collected where sync_with_helper() typically boils
> down to busy waiting. in this case, the workers as rate-monotonic in a
> similar manner as the main audio thread.
>
> one could also use lock-free queues with semaphores to wake only as many
> threads as needed for the graph topology (which can depend on user
> input). in this case SCHED_FIFO sounds more suitable.

proper handling of task dependencies & synchronization, and associated priority inversion issues, is also an aspect 
we've been and we're still working on, as when you mix DEADLINE tasks in the picture, it's a big mess.
> from a practical point of view: i'm not targeting a safety critical
> system. one advantage i'm seeing of DEADLINE over FIFO/RR is that it's
> easier to prevent lockups (e.g. when a user overloads the system). in
> the linux audio world this is typically done by a watchdog thread. the
> other part of the unix world (mach) is using time-constraint threads by
> default for audio use cases. so i'd assume that DEADLINE would free me
> from the need to spawn the watchdog thread ...

I'm not sure I'm getting this: AFAIU, from this viewpoint, what you could get
with DEADLINE, is very similar to what you can get already with RT throttling
on RT (RR/FIFO), setting up properly CPU cgroups & limits. Why do you think
DEADLINE would allow you to avoid a watchdog thread, that you need with
RT instead ?

Thanks,

     T.

-- 
Tommaso Cucinotta, Computer Engineering PhD
Associate Professor at the Real-Time Systems Laboratory (ReTiS)
Scuola Superiore Sant'Anna, Pisa, Italy
http://retis.sssup.it/people/tommaso

Re: SCHED_DEADLINE as user

[Juri Lelli]

On 21/08/18 00:45, Tim Blechmann wrote:

[...]

> apart from that, i'm only missing two features:
> 
> * using SCHED_DEADLINE from user applications is kind of a showstopper
> for me

Right. Would you be up for continuing experimenting with DEADLINE (while
we get our heads around removing this constraint) by - ugly hack warning
- removing the check at

https://elixir.bootlin.com/linux/latest/source/kernel/sched/core.c#L4216

?

I guess it would be good if you could provide even more feedback than
you already did, so that we can plan ahead what needs fixing/changing or
implementing.

BTW, thanks a lot already for this discussion! :-)

> * a nicer glibc wrapper, that allows me to use pthreads rather than
> using `syscall`: it's all pretty easy to obtain the `tid` for a calling
> thread (gettid), but it seems to be impossible to obtain the `tid` from
>  `pthread_t` without some hacks which are way too dirty to share in
> public ...

Indeed. We have been basically relying on

https://github.com/scheduler-tools/rt-app/tree/master/libdl

so far. But, I guess it's time - it has been time for a while - that we
reach glibc guys and try to understand why this is not yet all in the
standard libraries.

Re: SCHED_DEADLINE as user

[Tim Blechmann]

[-- Attachment #1.1: Type: text/plain, Size: 1977 bytes --]

>> alsa is basically driven by the hardware, which delivers/requests data
>> every ~1.3ms.
> 
> this is exactly the use-case we considered in that old paper back in LAC
> 2011, and let me just add that there is more experimentation and details
> and results in Giacomo's MSc thesis:
> 
>   https://retis.sssup.it/?q=node/77
> 
> FYI, we restarted activities on this use-case just recently, considering
> a couple of further dimensions:
> 
> 1) support for heterogeneous platforms with the aim of achieving a sound
> support for Arm big.LITTLE on Android, and apply it to the Android video
> & audio processing pipelines;
> 
> 2) proper consideration of power switching capabilities of the platform
> (assuming the user would block the CPU frequency to its maximum is not
> really multimedia-friendly);
> 
> 3) support for multi-threaded processing workflows on multi-cores, as
> typically needed by (high-performance) audio applications, either with a
> single multi-threaded audio processing client, or, e.g., with JACK, with
> a DAG of computations that can take advantage of underlying multi-core
> processing.
> 
> In this context, we've been playing also with the hierarchical extension
> to SCHED_DEADLINE we sent last year on LKML
> 
>   https://lkml.org/lkml/2017/3/31/658
> 
> I hope we can share some good write-up & experimental results pretty
> soon about some of the above...

very interesting! thanks for the links ...


> I'm not sure I'm getting this: AFAIU, from this viewpoint, what you
> could get
> with DEADLINE, is very similar to what you can get already with RT
> throttling
> on RT (RR/FIFO), setting up properly CPU cgroups & limits. Why do you think
> DEADLINE would allow you to avoid a watchdog thread, that you need with
> RT instead ?

interesting, wasn't aware of rt throttling ... (haven't done much linux
audio developments for almost 10 years). will give it another read ...

cheers,
tim


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 841 bytes --]

Re: SCHED_DEADLINE as user

[Juri Lelli]

On 21/08/18 09:38, Juri Lelli wrote:
> On 21/08/18 00:45, Tim Blechmann wrote:
> 
> [...]
> 
> > apart from that, i'm only missing two features:
> > 
> > * using SCHED_DEADLINE from user applications is kind of a showstopper
> > for me
> 
> Right. Would you be up for continuing experimenting with DEADLINE (while
> we get our heads around removing this constraint) by - ugly hack warning
> - removing the check at
> 
> https://elixir.bootlin.com/linux/latest/source/kernel/sched/core.c#L4216
> 
> ?
> 
> I guess it would be good if you could provide even more feedback than
> you already did, so that we can plan ahead what needs fixing/changing or
> implementing.
> 
> BTW, thanks a lot already for this discussion! :-)
> 
> > * a nicer glibc wrapper, that allows me to use pthreads rather than
> > using `syscall`: it's all pretty easy to obtain the `tid` for a calling
> > thread (gettid), but it seems to be impossible to obtain the `tid` from
> >  `pthread_t` without some hacks which are way too dirty to share in
> > public ...
> 
> Indeed. We have been basically relying on
> 
> https://github.com/scheduler-tools/rt-app/tree/master/libdl
> 
> so far. But, I guess it's time - it has been time for a while - that we
> reach glibc guys and try to understand why this is not yet all in the
> standard libraries.

FYI, I added both points to the list of known issues. Feel free to add
details.

https://github.com/jlelli/linux/issues

Best,

- Juri

Re: SCHED_DEADLINE as user

[luca abeni]

Hi Juri,

sorry for the late reply (I am just back from vacations), and thanks
for cc-ing me.

I see you opened a bug on github about this, so I am going to add more
details there, but basically I think that in order to use
SCHED_DEADLINE as non-root we need to:
1) Disable the boosting mechanism (not the inheritance, just the "soft
   enforcement behaviour" of tasks holding mutexes)
2) Implement some mechanism to limit the amount of dl bandwidth a user
   can allocate to itself (I think the cgroup-based approach we
   discussed some time ago should be OK... If I remember well, you even
   had a patch implementing it :)



				Luca

On Mon, 20 Aug 2018 10:54:20 +0200
Juri Lelli <juri.lelli@gmail.com> wrote:

> On 18/08/18 15:26, Tim Blechmann wrote:
> > >> audio applications typically use SCHED_FIFO. however they are
> > >> rate-monotonic by design, so i wanted to evaluate how
> > >> SCHED_DEADLINE behaves here.  
> > > 
> > > Right. Exactly what I started to think about lately as well.
> > >   
> > >>  
> > >>> Asking because we might want to have a look at how to relax the
> > >>> current privileges requirement.  
> > >>
> > >> please :)  
> > > 
> > > Anything to try attracting new users. :-P
> > > 
> > > More seriously, could you please describe in a bit more details
> > > your specific use case? Based on JACK I guess? Is it the
> > > application you are interested into public available somewhere?  
> > 
> > it could be based on JACK or natively on ALSA. furthermore the fun
> > part starts when trying to distribute work to multiple threads:
> > during the time slice of one audio callback (could be as low as
> > 1.3ms), we need to wake helper threads and (busy?)wait to collect
> > all the results. (while one of the applications is open source,
> > it's not entirely straight-forward to use)
> > 
> > for me it's more an investigation if SCHED_DEADLINE is actually
> > feasible in real-world audio application these days. tommaso used
> > to have a research prototype [1], built directly into JACK. however
> > currently i'd be more interested in using it in ALSA. but yeah,
> > it's more an investigation as SCHED_FIFO does a decent job for my
> > use case ...  
> 
> I (almost) see. Thanks for the details.
> 
> > [1] http://retis.sssup.it/~tommaso/publications/LAC-2011.pdf  
> 
> So, this was IIRC using AQuoSA hierarchical RT on EDF scheduler
> implementation; which it was handy to easy the task of assigning CBS
> server parameters to JACK clients (graph).
> 
> In your case - assuming you could already use DEADLINE for !root - do
> you think you would need to use the same interface for the multiple
> threads you are distributing work among? Or would you be fine with
> assigning runtime/deadline for each one of those?
> 
> BTW, cc-ing Tommaso and Luca. Discussion started from
> https://www.spinics.net/lists/linux-rt-users/msg19296.html

Re: SCHED_DEADLINE as user

[Juri Lelli]

Hi,

On 23/08/18 11:56, luca abeni wrote:
> Hi Juri,
> 
> sorry for the late reply (I am just back from vacations), and thanks
> for cc-ing me.

Welcome back! :-)

> I see you opened a bug on github about this, so I am going to add more
> details there,

I use github issues to keep track of things, but I guess mailing list
discussions are to be preferred though (so that more people can
potentially follow).

> but basically I think that in order to use SCHED_DEADLINE as non-root
> we need to:
> 1) Disable the boosting mechanism (not the inheritance, just the "soft
>    enforcement behaviour" of tasks holding mutexes)

But then what is a sane inheritance mechanism? Walk the chain and find
the next potential deadline to inherit for the current boosted (still
runtime enforced) task before throttling it? Not sure it's going to be
any easier than the proxy solution. :-/

> 2) Implement some mechanism to limit the amount of dl bandwidth a user
>    can allocate to itself (I think the cgroup-based approach we
>    discussed some time ago should be OK... If I remember well, you even
>    had a patch implementing it :)

I think most (all?) distributions today run with CONFIG_RT_GROUP_SCHED
disabled, we should also think about a solution that doesn't rely on
that interface. Maybe the existing system wide sched_rt_{period,
runtime}_us are enough?

Best,

- Juri

Re: SCHED_DEADLINE as user

[luca abeni]

Hi Juri,

On Thu, 23 Aug 2018 12:23:50 +0200
Juri Lelli <juri.lelli@gmail.com> wrote:
> > sorry for the late reply (I am just back from vacations), and thanks
> > for cc-ing me.  
> 
> Welcome back! :-)

Thanks :)


> > I see you opened a bug on github about this, so I am going to add
> > more details there,  
> 
> I use github issues to keep track of things, but I guess mailing list
> discussions are to be preferred though (so that more people can
> potentially follow).

Ok


> > but basically I think that in order to use SCHED_DEADLINE as
> > non-root we need to:
> > 1) Disable the boosting mechanism (not the inheritance, just the
> > "soft enforcement behaviour" of tasks holding mutexes)  
> 
> But then what is a sane inheritance mechanism?

In my understanding (please correct me if I am wrong), this is an
orthogonal issue: if I understand well, the issue preventing non-root
usage of SCHED_DEADLINE is that a task inheriting a dl entity is not
throttled (when the current runtime rrives to 0, the deadline is
postponed, but the task stays schedulable). So, I think that removing
this behaviour should allow to use SCHED_DEADLINE without starving
other tasks...

Then, there is the issue about the deadline and runtime to inherit. And
I agree that this is important (and the solution is not easy), but you
have this issue even if you use the current "dl_boosted" behaviour...
No?


> Walk the chain and find
> the next potential deadline to inherit for the current boosted (still
> runtime enforced) task before throttling it? Not sure it's going to be
> any easier than the proxy solution. :-/

Right; this is not easy... But I think it is not related with the issue
we are discussing (if I understand this email thread well). Yes, it has
to be fixed, but it does not prevent non-root usage. Or am I missing
something?


> > 2) Implement some mechanism to limit the amount of dl bandwidth a
> > user can allocate to itself (I think the cgroup-based approach we
> >    discussed some time ago should be OK... If I remember well, you
> > even had a patch implementing it :)  
> 
> I think most (all?) distributions today run with CONFIG_RT_GROUP_SCHED
> disabled, we should also think about a solution that doesn't rely on
> that interface.

I guess CONFIG_RT_GROUP_SCHED is often disabled because it ends up
changing the "traditional" SCHED_{RR/FIFO} behaviour. So, maybe the
solution is to have a different dl_{runtime,period} interface in
control groups (enabled by CONFIG_DL_GROUP_SCHED :).
CONFIG_DL_GROUP_SCHED does not change the scheduling behaviour, but
only the admission test... So, enabling it could be safer than enabling
CONFIG_RT_GROUP_SCHED.

> Maybe the existing system wide sched_rt_{period,
> runtime}_us are enough?

I do not know... A cgroup-based interface looks more powerful (and not
so difficult to implement... :), and would allow the sysadmin to decide
which users can use SCHED_DEADLINE, how much SCHED_DEADLINE bandwidth
can each user/group use, etc...


				Luca

Re: SCHED_DEADLINE as user

[Juri Lelli]

On 23/08/18 12:43, luca abeni wrote:
> On Thu, 23 Aug 2018 12:23:50 +0200
> Juri Lelli <juri.lelli@gmail.com> wrote:

[...]

> > But then what is a sane inheritance mechanism?
> 
> In my understanding (please correct me if I am wrong), this is an
> orthogonal issue: if I understand well, the issue preventing non-root
> usage of SCHED_DEADLINE is that a task inheriting a dl entity is not
> throttled (when the current runtime rrives to 0, the deadline is
> postponed, but the task stays schedulable). So, I think that removing
> this behaviour should allow to use SCHED_DEADLINE without starving
> other tasks...

Right, potential starvation would be gone...

> Then, there is the issue about the deadline and runtime to inherit. And
> I agree that this is important (and the solution is not easy), but you
> have this issue even if you use the current "dl_boosted" behaviour...
> No?

... but, while it's true that current inheritance has problems w/ o w/o
boosting behaviour, I wonder if the problem might be more painful for a
normal user that it's still under runtime enforcement. Disabling
enforcement seems to hide a bit the fact that we need proper
inheritance. :-(

> > Walk the chain and find
> > the next potential deadline to inherit for the current boosted (still
> > runtime enforced) task before throttling it? Not sure it's going to be
> > any easier than the proxy solution. :-/
> 
> Right; this is not easy... But I think it is not related with the issue
> we are discussing (if I understand this email thread well). Yes, it has
> to be fixed, but it does not prevent non-root usage. Or am I missing
> something?

It depends on how bad we think it's what I said above I guess.

> > > 2) Implement some mechanism to limit the amount of dl bandwidth a
> > > user can allocate to itself (I think the cgroup-based approach we
> > >    discussed some time ago should be OK... If I remember well, you
> > > even had a patch implementing it :)  
> > 
> > I think most (all?) distributions today run with CONFIG_RT_GROUP_SCHED
> > disabled, we should also think about a solution that doesn't rely on
> > that interface.
> 
> I guess CONFIG_RT_GROUP_SCHED is often disabled because it ends up
> changing the "traditional" SCHED_{RR/FIFO} behaviour. So, maybe the
> solution is to have a different dl_{runtime,period} interface in
> control groups (enabled by CONFIG_DL_GROUP_SCHED :).
> CONFIG_DL_GROUP_SCHED does not change the scheduling behaviour, but
> only the admission test... So, enabling it could be safer than enabling
> CONFIG_RT_GROUP_SCHED.

Not sure if adding yet another config switch is acceptable. FWIW, I'd
prefer not to add it, also looking back at all the problems RT_GROUP_
SCHED seems to pose/have posed, but if turns to be the only option..

> > Maybe the existing system wide sched_rt_{period,
> > runtime}_us are enough?
> 
> I do not know... A cgroup-based interface looks more powerful (and not
> so difficult to implement... :), and would allow the sysadmin to decide
> which users can use SCHED_DEADLINE, how much SCHED_DEADLINE bandwidth
> can each user/group use, etc...

How about extending PAM limits instead? It looks like it's what (e.g.)
audio users rely on already [1]. It is maybe possible to add dlruntime,
dlperiod, dldeadline parameters in there?

1 - https://fedoraproject.org/wiki/JACK_Audio_Connection_Kit#Running_Jack_in_Realtime_mode

Re: SCHED_DEADLINE as user

[luca abeni]

On Thu, 23 Aug 2018 14:37:39 +0200
Juri Lelli <juri.lelli@gmail.com> wrote:

> On 23/08/18 12:43, luca abeni wrote:
> > On Thu, 23 Aug 2018 12:23:50 +0200
> > Juri Lelli <juri.lelli@gmail.com> wrote:  
> 
> [...]
> 
> > > But then what is a sane inheritance mechanism?  
> > 
> > In my understanding (please correct me if I am wrong), this is an
> > orthogonal issue: if I understand well, the issue preventing
> > non-root usage of SCHED_DEADLINE is that a task inheriting a dl
> > entity is not throttled (when the current runtime rrives to 0, the
> > deadline is postponed, but the task stays schedulable). So, I think
> > that removing this behaviour should allow to use SCHED_DEADLINE
> > without starving other tasks...  
> 
> Right, potential starvation would be gone...
> 
> > Then, there is the issue about the deadline and runtime to inherit.
> > And I agree that this is important (and the solution is not easy),
> > but you have this issue even if you use the current "dl_boosted"
> > behaviour... No?  
> 
> ... but, while it's true that current inheritance has problems w/ o
> w/o boosting behaviour, I wonder if the problem might be more painful
> for a normal user that it's still under runtime enforcement.

I think that the only way to have an answer to this doubt is to try :)


> Disabling
> enforcement seems to hide a bit the fact that we need proper
> inheritance. :-(

My impression is that it hides the problem in some situations, but
the Murphy law ensures that the issue will appear when the quality of
the audio is more important :)


			Luca

Re: SCHED_DEADLINE as user

[luca abeni]

On Thu, 23 Aug 2018 14:37:39 +0200
Juri Lelli <juri.lelli@gmail.com> wrote:
[...]
> > > Maybe the existing system wide sched_rt_{period,
> > > runtime}_us are enough?  
> > 
> > I do not know... A cgroup-based interface looks more powerful (and
> > not so difficult to implement... :), and would allow the sysadmin
> > to decide which users can use SCHED_DEADLINE, how much
> > SCHED_DEADLINE bandwidth can each user/group use, etc...  
> 
> How about extending PAM limits instead? It looks like it's what (e.g.)
> audio users rely on already [1]. It is maybe possible to add
> dlruntime, dlperiod, dldeadline parameters in there?

Ok, I do not know much about these PAM limits... Where can I find more
information? What do we need to implement exactly, to add these
dlruntime, dlperiod, dldeadline parameters?


			Thanks,
				Luca

Re: SCHED_DEADLINE as user

[Juri Lelli]

On 23/08/18 15:00, luca abeni wrote:
> On Thu, 23 Aug 2018 14:37:39 +0200
> Juri Lelli <juri.lelli@gmail.com> wrote:
> [...]
> > > > Maybe the existing system wide sched_rt_{period,
> > > > runtime}_us are enough?  
> > > 
> > > I do not know... A cgroup-based interface looks more powerful (and
> > > not so difficult to implement... :), and would allow the sysadmin
> > > to decide which users can use SCHED_DEADLINE, how much
> > > SCHED_DEADLINE bandwidth can each user/group use, etc...  
> > 
> > How about extending PAM limits instead? It looks like it's what (e.g.)
> > audio users rely on already [1]. It is maybe possible to add
> > dlruntime, dlperiod, dldeadline parameters in there?
> 
> Ok, I do not know much about these PAM limits... Where can I find more
> information?

The link I shared already gives some information, man pam_limits and
limits.conf also.

> What do we need to implement exactly, to add these dlruntime,
> dlperiod, dldeadline parameters?

Not sure yet "exactly" :-), but I'd say something along these lines

https://elixir.bootlin.com/linux/latest/source/kernel/sched/core.c#L4198

Re: SCHED_DEADLINE as user

[luca abeni]

On Thu, 23 Aug 2018 15:09:57 +0200
Juri Lelli <juri.lelli@gmail.com> wrote:

> On 23/08/18 15:00, luca abeni wrote:
> > On Thu, 23 Aug 2018 14:37:39 +0200
> > Juri Lelli <juri.lelli@gmail.com> wrote:
> > [...]  
> > > > > Maybe the existing system wide sched_rt_{period,
> > > > > runtime}_us are enough?    
> > > > 
> > > > I do not know... A cgroup-based interface looks more powerful
> > > > (and not so difficult to implement... :), and would allow the
> > > > sysadmin to decide which users can use SCHED_DEADLINE, how much
> > > > SCHED_DEADLINE bandwidth can each user/group use, etc...    
> > > 
> > > How about extending PAM limits instead? It looks like it's what
> > > (e.g.) audio users rely on already [1]. It is maybe possible to
> > > add dlruntime, dlperiod, dldeadline parameters in there?  
> > 
> > Ok, I do not know much about these PAM limits... Where can I find
> > more information?  
> 
> The link I shared already gives some information, man pam_limits and
> limits.conf also.
> 
> > What do we need to implement exactly, to add these dlruntime,
> > dlperiod, dldeadline parameters?  
> 
> Not sure yet "exactly" :-), but I'd say something along these lines
> 
> https://elixir.bootlin.com/linux/latest/source/kernel/sched/core.c#L4198

Ok, thanks; I think I got it: we need to extend the "rlim" array (in
signal_struct) adding fields for "RLIMIT_DLRUNTIME" and friends
(actually, I think a "RLIMIT_UTILIZATION" field is also needed). And
then we need to add checks for these fields in __sched_setscheduler(),
etc...

Maybe I am misunderstanding this stuff, but the cgroup-based approach
looks simpler / more elegant :)



			Thanks,
				Luca

Re: SCHED_DEADLINE as user

[Gene Heskett]

On Thursday 23 August 2018 09:00:40 luca abeni wrote:

> On Thu, 23 Aug 2018 14:37:39 +0200
> Juri Lelli <juri.lelli@gmail.com> wrote:
> [...]
>
> > > > Maybe the existing system wide sched_rt_{period,
> > > > runtime}_us are enough?
> > >
> > > I do not know... A cgroup-based interface looks more powerful (and
> > > not so difficult to implement... :), and would allow the sysadmin
> > > to decide which users can use SCHED_DEADLINE, how much
> > > SCHED_DEADLINE bandwidth can each user/group use, etc...
> >
> > How about extending PAM limits instead? It looks like it's what
> > (e.g.) audio users rely on already [1]. It is maybe possible to add
> > dlruntime, dlperiod, dldeadline parameters in there?
>
> Ok, I do not know much about these PAM limits... Where can I find more
> information? What do we need to implement exactly, to add these
> dlruntime, dlperiod, dldeadline parameters?
>
>
> 			Thanks,
> 				Luca

Are you folks interested in input from a real world user? On who at the 
moment is building about half these kernels as you announce them on a 
pi-3b, the same pi-3b running 1500 lbs of a Sheldon 11"x36" lathe.

I am running linuxcnc on both x86 and armhf machinery, two lathes and two 
milling machines, all being moved by stepper motors.

Steppers, if the steps are being software generated, are very sensitive 
to timing wibbles in a stream of steps, with a 5 microsecond wobble in 
the timing haveing serious effects on the available torque by causing a 
speedup and slowdown.  This severely limits the speed at which the motor 
can be driven without stalling or losing steps, which in a stepper 
system is the equivalent of losing its home reference which=wrecked 
part. I have one machine still depending on software stepping. I could 
easily triple the speed of that machine by spending another 100 dollars 
to offload that to an fpga based interface card. As it is the base 
thread runs at about 28 microseconds. That also represents the 
granularity of a speed change. So it of late has been relegated to doing 
very precise work via EDM, which is slow anyway.

In short, linuxcnc, and its best rtai kernels can do pergaps 10"/minute 
whereas the fpga cards can move that same machine at 100 ipm. Any help 
in getting the latency down means we can do things faster.

LinuxCNC is also run as a common user, never as root.  Thats my $0.02 
from a users viewpoint.

-- 
Cheers, Gene Heskett
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page <http://geneslinuxbox.net:6309/gene>

Re: SCHED_DEADLINE as user

[Chris Friesen]

On 08/23/2018 07:45 AM, Gene Heskett wrote:

> LinuxCNC is also run as a common user, never as root.  Thats my $0.02
> from a users viewpoint.

Given that these are special-purpose machines, can I ask why you wouldn't run as 
root if it gave better performance?

Chris

Re: SCHED_DEADLINE as user

[Gene Heskett]

On Thursday 23 August 2018 12:01:41 Chris Friesen wrote:

> On 08/23/2018 07:45 AM, Gene Heskett wrote:
> > LinuxCNC is also run as a common user, never as root.  Thats my
> > $0.02 from a users viewpoint.
>
> Given that these are special-purpose machines, can I ask why you
> wouldn't run as root if it gave better performance?
>
> Chris

Its a rather bad habit, and if I get used to doing it on this machine, I 
might do it on one of the metal carvers, which has a tendency to mess 
with the user perms. Root or known user is all the same to  lcnc AFAIK. 
I have done it 3 or 4 times, but there is no detectable diff in 
performance. But sometime in the last few years it has now been trained 
to refuse to run as root.

The kernel argument isolcpus= comes in handy occasionally on really old 
but still multicore hdwe but has had little or no effect since about 
kernel 3.4.0 built for SMP and PREEMPT. We don't use  more than 2GB of 
d-ram for the same reason, the patches to give a 4GB capable address are 
also a performance penalty.

Currently running
Linux GO704 3.4-9-rtai-686-pae #1 SMP PREEMPT Debian 3.4.55-4linuxcnc 
i686 GNU/Linux, a 32 bit build because its faster at a context switch 
than the larger stack image the 64 bit build uses.

This machine, my "doitall" machine doesn't drive any of that sort of 
hardware, and is currently running:3.16.0-0.bpo.4-amd64 #1 SMP Debian 
3.16.39-1+deb8u1~bpo70+1 (2017-02-24) x86_64 GNU/Linux which will run 
the simulator so I can write cgode from a comfy chair.  Then export it 
with sshfs to the machine that will actually make the swarf, aka 
metallic sawdust.

Thanks Chris.

-- 
Cheers, Gene Heskett
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page <http://geneslinuxbox.net:6309/gene>