Split process across multiple schedulers?

Split process across multiple schedulers?

[Olsson John]

Hi!

I have tried reading the documentation for CGroups V1 and V2 and it seems that there is one usecase that we are interested in that *is* supported by CGroups V1 but not by CGroups V2. I really hope that I have overlooked something and you can tell med to RTFM. ;)

Assume that you want to have a virtual machine running on at least one core. Connected to the VM you need to also have a VMM (Virtual Machine Monitor) that acts as some sort of glue between KVM and the Guest in the VM. For instance Virtual Box on Linux uses QEmu as its VMM. Within the VM each virtual core is connected to a thread in the Host computer, and usually you want to have a 1:1 mapping between virtual core threads and physical cores.

Preferably for optimal performance you want to isolate the cores where the virtual core threads are running so nothing else interferes with them (besides kernel threads connected to IRQs that can't be moved from the isolated cores). The VMM is then running on another core that is not running a virtual core thread. CGroups is the preferred way of accomplishing this. :)

The virtual core threads need to have some parent process and the VMM process is the natural home of for these threads. My understanding of CGroups V1 is that it is possible to have one scheduler associated  there are use cases where you might want to have one kind of scheduler for the VMM process (for instance CFS) and another scheduler for the virtual core threads (for instance FIFO).

My conclusion after reading the documentation for CGroups V2 is that the above scenario is no longer possible to do. Or have I misunderstood something here?


A bit more detailed background after the general idea behind my question. Assume that the software running within the VM is an old Windows 98 game you have the license key for (for instance SimCity 3000). The developers of SimCity 3000 didn't anticipate that you would want to run the game on a computer that was several times faster than what existed at its release. Music plays fine, but when you try to scroll around when zoomed in using the cursor keys on the keyboard (or the mouse by placing the mouse pointer on the edge of the screen) the scrolling speed is inhumanly fast. Basically the game is no longer enjoyable to play. It runs too fast.

Trying to slow it down using a tool like cpulimit is way too coarse; the music is no longer possible to listen to as it is played in bursts. However if you instead have a specially designed scheduler that tries to execute a program with a configured "slow-down"-factor (each scheduling tick you ensure that it is executed only a fraction of that tick) then you could get it to run with a just about right speed. However you do not want to slow down VMM with the same scheduler and thus you want to be able to set different schedulers for different threads within a single process (assuming that they are running on different cores).

And yes, there are other applications for this besides running old games. ;)
 

Re: Split process across multiple schedulers?

[Michal Koutný]

Hello.

On Mon, Mar 14, 2022 at 03:19:56PM +0000, Olsson John <john.olsson-zlDGZTLOEuz3oGB3hsPCZA@public.gmane.org> wrote:
> I have tried reading the documentation for CGroups V1 and V2 and it
> seems that there is one usecase that we are interested in that *is*
> supported by CGroups V1 but not by CGroups V2.

Are you missing CONFIG_RT_GROUP_SCHED and v1 cpu controller's
cpu.rt_{runtime,period}_us? (Just asking, you didn't mention this
explicitly in your e-mail but it sounds so and it's a thing that's
indeed missing in v2.)

> My understanding of CGroups V1 is that it is possible to have one
> scheduler associated  there are use cases where you might want to have
> one kind of scheduler for the VMM process (for instance CFS) and
> another scheduler for the virtual core threads (for instance FIFO).

sched_setscheduler(2) applies to threads regardless of cgroup
membership, there's no change between v1 and v2.

(Without CONFIG_RT_GROUP_SCHED all RT threads are effectively in the
root cgroup.)

> My conclusion after reading the documentation for CGroups V2 is that
> the above scenario is no longer possible to do. Or have I
> misunderstood something here?

You may need to enable threaded mode on v2 (see cgroup.type) to
manipulate with individual threads across cgroups. (E.g. if you want to
use cpuset controller to pin/restrict individual threads.)

Regards,
Michal

RE: [EXTERNAL] Re: Split process across multiple schedulers?

[Olsson John]

Thank you for your pointers Michal! :)


> Are you missing CONFIG_RT_GROUP_SCHED and v1 cpu controller's
> cpu.rt_{runtime,period}_us? (Just asking, you didn't mention this
> explicitly in your e-mail but it sounds so and it's a thing that's
> indeed missing in v2.)

I'm not sure. I have to double check. If I'm indeed missing
CONFIG_RT_GROUP_SCHED I'll let you know. ;)


> sched_setscheduler(2) applies to threads regardless of cgroup
> membership, there's no change between v1 and v2.

If I'm understanding you correctly this effectively means that it is
possible to spread a process and its threads across multiple cgroups
that in turn may have different schedulers (and CPU affinity)
associated with them?


> (Without CONFIG_RT_GROUP_SCHED all RT threads are effectively in the
> root cgroup.)

Interesting! I have missed this little tidbit of information. This is
indeed very good to know!

A side effect of this is that in V2 you can't have an RT thread pinned
to a specific core that is evacuated, right? If you could do this it
would also be possible to remove the portion of the scheduling
interval that is left for non-RT threads in the cgroup config since
there would not be any other threads on this evacuated core. By doing
that you would eliminate jitter due to that otherwise the scheduler
would interrupt the RT thread and immediately re-schedule it
again. And thus you would theoretically get very good RT properties
(unless you make system calls).

Now one could argue that there is no point in having a pinned RT
thread on a single core that is evacuated, but that means that the
thread would be interrupted HZ times per second. If you instead used
FIFO scheduling (which handles RT threads only, right?) then you could
eliminate this noise. Or I am just showing off how little I understand
about scheduling in Linux. ;)


> You may need to enable threaded mode on v2 (see cgroup.type) to
> manipulate with individual threads across cgroups. (E.g. if you want
> to use cpuset controller to pin/restrict individual threads.)

I'll read up a bit more and try what you suggest. If we have
misunderstood each other I'll contact the mailing list again with more
details. :)

Re: [EXTERNAL] Re: Split process across multiple schedulers?

[Michal Koutný]

On Tue, Mar 15, 2022 at 08:19:26AM +0000, Olsson John <john.olsson-zlDGZTLOEuz3oGB3hsPCZA@public.gmane.org> wrote:
> If I'm understanding you correctly this effectively means that it is
> possible to spread a process and its threads across multiple cgroups
> that in turn may have different schedulers (and CPU affinity)
> associated with them?

Yes, the docs is here
https://www.kernel.org/doc/html/v5.17-rc8/admin-guide/cgroup-v2.html#threads

> > (Without CONFIG_RT_GROUP_SCHED all RT threads are effectively in the
> > root cgroup.)
> 
> Interesting! I have missed this little tidbit of information. This is
> indeed very good to know!

Maybe I should have added this applies from the POV of the cpu
controller in particular...

> A side effect of this is that in V2 you can't have an RT thread pinned
> to a specific core that is evacuated, right?

...it has no effect for grouping of cpuset controller (assuming both cpu
and cpuset are enabled in given subtree).

> If you could do this it would also be possible to remove the portion
> of the scheduling interval that is left for non-RT threads in the
> cgroup config since there would not be any other threads on this
> evacuated core.
> By doing that you would eliminate jitter due to that otherwise the
> scheduler would interrupt the RT thread and immediately re-schedule it
> again. And thus you would theoretically get very good RT properties
> (unless you make system calls).

Well, there are more jobs that can interfere with RT workload on a cpu
(see isolcpus= [1]) and there's some ongoing work to make these more
convenient via cpuset controller [2]. The currently working approach
would be to use isolcpus= cmdline to isolate the CPUs and use either
sched_setaffinity() or cpuset controller to place tasks on the reserved
CPUs (the cpuset approach is more strict as it may prevent unprivileged
threads to switch to another CPU). 

> If you instead used FIFO scheduling (which handles RT threads only,
> right?) then you could eliminate this noise. Or I am just showing off
> how little I understand about scheduling in Linux. ;)

(Actually when I take a step back and read your motivational example of
a legacy game in VM, I don't think FIFO (or another RT policy) is suited
for this case. Plain SCHED_OTHER and cpu controller's bandwidth
limitation could do just fine -- you can apply to a (threaded) cgroup
with chosen threads only.)

HTH,
Michal


[1] https://www.kernel.org/doc/html/v5.17-rc8/admin-guide/kernel-parameters.html?highlight=isolcpus
[2] https://lore.kernel.org/all/20211205183220.818872-1-longman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org/

RE: [EXTERNAL] Re: Split process across multiple schedulers?

[Olsson John]

> (Actually when I take a step back and read your motivational example
> of a legacy game in VM, I don't think FIFO (or another RT policy) is
> suited for this case. Plain SCHED_OTHER and cpu controller's
> bandwidth limitation could do just fine -- you can apply to a
> (threaded) cgroup with chosen threads only.)

As you might have already surmised it was a placeholder example to
give the general idea. I think it is time to add some more details. :)

Assume that you have an embedded system running some kind of software
with real time like properties. You want to develop and debug your
software locally on your high-end machine since it is much more
convenient. Alas the software runs way too fast due to the difference
in performance so you can't detect overruns etc.

Assume also that you have a special scheduler that behaves in a very
strange way compared to ordinary schedulers. Each scheduling tick it
waits a configurable time before it lets the scheduled task execute
its tick. This effectively means that the task is throttled and thus
executes slower than it normally would. By tuning the waiting time you
can tune the execution speed so comes close enough to what it is on
the target system.

Assume also that you have enough cores in your system so you can
dedicate one core for the VMM and one core for the virtual core thread
the Guest uses to execute the software. One way of implementing this
kind of scheduler would be to create a fork of the FIFO scheduler that
have this behavior.

This crazy(?) technique can of course be used for other things such as
running old DOS games or running an RTOS intended for embedded systems
as the Guest in the VM and so on. ;)


/John

-----Original Message-----
From: Michal Koutný <mkoutny-IBi9RG/b67k@public.gmane.org> 
Sent: den 15 mars 2022 11:36
To: Olsson John <john.olsson-zlDGZTLOEuz3oGB3hsPCZA@public.gmane.org>
Cc: cgroups-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
Subject: Re: [EXTERNAL] Re: Split process across multiple schedulers?

On Tue, Mar 15, 2022 at 08:19:26AM +0000, Olsson John <john.olsson-zlDGZTLOEuz3oGB3hsPCZA@public.gmane.org> wrote:
> If I'm understanding you correctly this effectively means that it is 
> possible to spread a process and its threads across multiple cgroups 
> that in turn may have different schedulers (and CPU affinity) 
> associated with them?

Yes, the docs is here
https://www.kernel.org/doc/html/v5.17-rc8/admin-guide/cgroup-v2.html#threads

> > (Without CONFIG_RT_GROUP_SCHED all RT threads are effectively in the 
> > root cgroup.)
> 
> Interesting! I have missed this little tidbit of information. This is 
> indeed very good to know!

Maybe I should have added this applies from the POV of the cpu controller in particular...

> A side effect of this is that in V2 you can't have an RT thread pinned 
> to a specific core that is evacuated, right?

...it has no effect for grouping of cpuset controller (assuming both cpu and cpuset are enabled in given subtree).

> If you could do this it would also be possible to remove the portion 
> of the scheduling interval that is left for non-RT threads in the 
> cgroup config since there would not be any other threads on this 
> evacuated core.
> By doing that you would eliminate jitter due to that otherwise the 
> scheduler would interrupt the RT thread and immediately re-schedule it 
> again. And thus you would theoretically get very good RT properties 
> (unless you make system calls).

Well, there are more jobs that can interfere with RT workload on a cpu (see isolcpus= [1]) and there's some ongoing work to make these more convenient via cpuset controller [2]. The currently working approach would be to use isolcpus= cmdline to isolate the CPUs and use either
sched_setaffinity() or cpuset controller to place tasks on the reserved CPUs (the cpuset approach is more strict as it may prevent unprivileged threads to switch to another CPU). 

> If you instead used FIFO scheduling (which handles RT threads only,
> right?) then you could eliminate this noise. Or I am just showing off 
> how little I understand about scheduling in Linux. ;)

(Actually when I take a step back and read your motivational example of a legacy game in VM, I don't think FIFO (or another RT policy) is suited for this case. Plain SCHED_OTHER and cpu controller's bandwidth limitation could do just fine -- you can apply to a (threaded) cgroup with chosen threads only.)

HTH,
Michal


[1] https://www.kernel.org/doc/html/v5.17-rc8/admin-guide/kernel-parameters.html?highlight=isolcpus
[2] https://lore.kernel.org/all/20211205183220.818872-1-longman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org/

Re: [EXTERNAL] Re: Split process across multiple schedulers?

[Michal Koutný]

On Tue, Mar 15, 2022 at 03:49:52PM +0000, Olsson John <john.olsson-zlDGZTLOEuz3oGB3hsPCZA@public.gmane.org> wrote:
> As you might have already surmised it was a placeholder example to
> give the general idea. I think it is time to add some more details. :)

Thanks for sharing the additional description.

> Assume that you have an embedded system running some kind of software
> with real time like properties. You want to develop and debug your
> software locally on your high-end machine since it is much more
> convenient. Alas the software runs way too fast due to the difference
> in performance so you can't detect overruns etc.

There are certainly more knowledgeable people than me to help with
debugging such environments.

> One way of implementing this kind of scheduler would be to create a
> fork of the FIFO scheduler that have this behavior.

I think you've complete info now how threads are handled with cgroups.

Good luck with the fork :-)


Michal

RE: [EXTERNAL] Re: Split process across multiple schedulers?

[Olsson John]

> I think you've complete info now how threads are handled with cgroups.

Thank you for your excellent support! :D


> Good luck with the fork :-)

It's basically already done as part of a Master Thesis. Now we'll
experiment with it and see how theory clashes with the real world. :)


/John

-----Original Message-----
From: Michal Koutný <mkoutny-IBi9RG/b67k@public.gmane.org> 
Sent: den 15 mars 2022 18:34
To: Olsson John <john.olsson-zlDGZTLOEuz3oGB3hsPCZA@public.gmane.org>
Cc: cgroups-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
Subject: Re: [EXTERNAL] Re: Split process across multiple schedulers?

On Tue, Mar 15, 2022 at 03:49:52PM +0000, Olsson John <john.olsson@saabgroup.com> wrote:
> As you might have already surmised it was a placeholder example to 
> give the general idea. I think it is time to add some more details. :)

Thanks for sharing the additional description.

> Assume that you have an embedded system running some kind of software 
> with real time like properties. You want to develop and debug your 
> software locally on your high-end machine since it is much more 
> convenient. Alas the software runs way too fast due to the difference 
> in performance so you can't detect overruns etc.

There are certainly more knowledgeable people than me to help with debugging such environments.

> One way of implementing this kind of scheduler would be to create a 
> fork of the FIFO scheduler that have this behavior.

I think you've complete info now how threads are handled with cgroups.

Good luck with the fork :-)


Michal

Re: Split process across multiple schedulers?

[Tejun Heo]

Hello,

On Mon, Mar 14, 2022 at 03:19:56PM +0000, Olsson John wrote:
> Preferably for optimal performance you want to isolate the cores where the virtual core threads are running so nothing else interferes with them (besides kernel threads connected to IRQs that can't be moved from the isolated cores). The VMM is then running on another core that is not running a virtual core thread. CGroups is the preferred way of accomplishing this. :)

I have a basic question. cgroup provides new capabilities through its
ability to hierarchically organize workloads on the system and distributes
resources across the hierarchy. If all one wants to do is affining specific
threads to specific CPUs or changing some other attributes of them, there's
nothing extra that cgroup provides compared to using plain per-task
interface. Is there some other eason why cgroup is the preferred way here?

Thanks.

-- 
tejun

RE: [EXTERNAL] Re: Split process across multiple schedulers?

[Olsson John]

> I have a basic question. cgroup provides new capabilities through
> its ability to hierarchically organize workloads on the system and
> distributes resources across the hierarchy. If all one wants to do
> is affining specific threads to specific CPUs or changing some other
> attributes of them, there's nothing extra that cgroup provides
> compared to using plain per-task interface. Is there some other
> eason why cgroup is the preferred way here?

Very good question! And yes, there actually are some reasons. :)

By using cgroup you can delegate the authority to configure a subset
of the hierarchy via rwx and user, group, others. By using the
per-task interface you have to be root, right?

Also, we want to separate the configuration of the threads from the
application as it need to be deployed in different hardware scenarios.

And we need to be able to easily replicate a configuration from one
machine to another machine.

We also need to configure other aspects that cgroup allows us to do
for the set of processes.

Since cgroup solves all of the above problems for us, why using
something else? :)

Re: [EXTERNAL] Re: Split process across multiple schedulers?

[Tejun Heo]

Hello,

On Wed, Mar 16, 2022 at 04:49:29PM +0000, Olsson John wrote:
> By using cgroup you can delegate the authority to configure a subset
> of the hierarchy via rwx and user, group, others. By using the
> per-task interface you have to be root, right?

Yeah, deligation can be useful. However, given that the configuration would
need some automation / scripting anyway, it shouldn't be too difficult to
work around.

> Also, we want to separate the configuration of the threads from the
> application as it need to be deployed in different hardware scenarios.

The thing is, to put different threads of a process into different cgroups,
one has to know which threads are doing what, which is the same knowledge
needed to configure per-thread attributes.

> And we need to be able to easily replicate a configuration from one
> machine to another machine.

Again, I'm not sure how needing to put different threads into different
cgroups is much different.

> We also need to configure other aspects that cgroup allows us to do
> for the set of processes.

This one, I agree. There are controller features which aren't available
through regular thread interface such as bandwidth throttling.

> Since cgroup solves all of the above problems for us, why using
> something else? :)

Yeah, mostly curious from cgroup design POV. It'd be nice to support use
cases like this well but we likely don't wanna twist anything for it.

Thanks.

-- 
tejun

RE: [EXTERNAL] Re: Split process across multiple schedulers?

[Olsson John]

> Yeah, mostly curious from cgroup design POV. It'd be nice to support
> use cases like this well but we likely don't wanna twist anything
> for it.

Ok. I'll give you some more details. :)

The building block we are discussing here (running a VM where VMM and
virtual core threads are scheduled using different schedulers and thus
effectively on different cores) is a building block in a larger
context. The use cases I have provided so far are still valid and
needed.

What you run in the VM is the OS plus application of some embedded
multicore system where the target hardware is way less competent than
what you are running the VM on. It will most likely even be using a
different CPU architecture so you cross compile to for instance x86.

Now put this building block in a simulator where other parts of the
larger system are connected to this VM. This simulator is used to
train persons who operate the real system and it is important it's
behavior closely mimics the actual real system. For instance response
times from the embedded system may not be too fast nor to too slow.


> Yeah, deligation can be useful. However, given that the
> configuration would need some automation / scripting anyway, it
> shouldn't be too difficult to work around.

Think of it like this. IT department installs the OS of the Host and
sets up cgroup node where a user group controls access to a
subtree. Developers belonging to this group are then allowed to
configure cgroup using abstract names for the nodes in the tree. The
software running on the server uses these abstract names and does not
need to know how they are configured. This means that when the
software is deployed in a different setting where you want another
behavior you just need to reconfigure the cgroup without needing to
recompile the software. And by relying on cgroups you also get less
source code to maintain (and that you might get wrong).


> The thing is, to put different threads of a process into different
> cgroups, one has to know which threads are doing what, which is the
> same knowledge needed to configure per-thread attributes.

Well, you might also have generic rules like for VM named "foo" you
put the VMM thread in "foo-vmm" and virtual core threads in
"foo-vct01" through "foo-vctnn". The actual mapping of these cgroups
to physical cores might differ from server to server. And you might
also select different schedulers and so on depending on server
configuration and the intention with the software installation. For
instance CI-loops that run batches where real-time behavior (matching
elapsed wall clock time in the simulation to actual system) is not
that important and you can fast-forward the simulation as you are
interested in functional tests and so on. This mean that the actual
software does not need to worry about such things.


/John