Short process stall after assigning it to a cgroup

Short process stall after assigning it to a cgroup

[Ronny Meeus]

Hello

I want to use cgroups to control my heavy cpuload consuming applications.
All apps are running in the realtime domain and I'm using kernel 4.9
and cgroup v1.

I created a small application that monitors the CPU load (by reading
the /proc filesystem) and when it enters a full load condition, I
dynamically start to put the high consuming processes into a cgroup
(which were created during system start-up). Each process will have
it's own cgroup created under the root-cgroup.

The budget I assign to the process is equal to the budget it has
consumed in the previous measurement interval (for example 5s). As
long as the load continues to be high, I start to gradually reduce the
budget of the cgroup until the system is idle enough.

This works reasonably well, but in some cases I see that a very high
load consuming application is stopped completely at the moment it is
put in a cgroup, although the budget allocated to it is correctly
calculated based on the load it consumed in my previous interval.

An example:
- cpu.rt_period_us = 1000000
- cpu.rt_runtime_us = 400000
I would assume that an application put in a cgroup with this
configuration can consume 40% of the CPU and it actually does. But
sometimes, immediately after the process assignment, it stops for a
short period (something like 1 or 2s) and then starts to consume 40%
again.

Is that expected behavior?

It looks like the "budget" it has consumed in the root-cgroup is taken
into account when it is moved to its own group and this results in the
stall.

Best regards,
Ronny

Re: Short process stall after assigning it to a cgroup

[Ronny Meeus]

Op ma 14 jun. 2021 om 17:29 schreef Ronny Meeus <ronny.meeus-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>:
>
> Hello
>
> I want to use cgroups to control my heavy cpuload consuming applications.
> All apps are running in the realtime domain and I'm using kernel 4.9
> and cgroup v1.
>
> I created a small application that monitors the CPU load (by reading
> the /proc filesystem) and when it enters a full load condition, I
> dynamically start to put the high consuming processes into a cgroup
> (which were created during system start-up). Each process will have
> it's own cgroup created under the root-cgroup.
>
> The budget I assign to the process is equal to the budget it has
> consumed in the previous measurement interval (for example 5s). As
> long as the load continues to be high, I start to gradually reduce the
> budget of the cgroup until the system is idle enough.
>
> This works reasonably well, but in some cases I see that a very high
> load consuming application is stopped completely at the moment it is
> put in a cgroup, although the budget allocated to it is correctly
> calculated based on the load it consumed in my previous interval.
>
> An example:
> - cpu.rt_period_us = 1000000
> - cpu.rt_runtime_us = 400000
> I would assume that an application put in a cgroup with this
> configuration can consume 40% of the CPU and it actually does. But
> sometimes, immediately after the process assignment, it stops for a
> short period (something like 1 or 2s) and then starts to consume 40%
> again.
>
> Is that expected behavior?
>
> It looks like the "budget" it has consumed in the root-cgroup is taken
> into account when it is moved to its own group and this results in the
> stall.
>
> Best regards,
> Ronny

Note that this is a dual-core CPU and the process that is doing the cgroup
updates is running in a single thread. So that other core will most
probably running the application that is moved into the child cgroup.

Re: Short process stall after assigning it to a cgroup

[Michal Koutný]

[-- Attachment #1: Type: text/plain, Size: 996 bytes --]

Hello Ronny.

On Mon, Jun 14, 2021 at 05:29:35PM +0200, Ronny Meeus <ronny.meeus-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> All apps are running in the realtime domain and I'm using kernel 4.9
> and cgroup v1. [...]  when it enters a full load condition [...]
> I start to gradually reduce the budget of the cgroup until the system
> is idle enough.

Has your application some RT requirements or is there other reason why
you use group RT allocations? (When your app seems to require all CPU
time, you decide to curb it. And it still fullfills RT requirements?)


> But sometimes, immediately after the process assignment, it stops for
> a short period (something like 1 or 2s) and then starts to consume 40%
> again.

What if you reduce cpu.rt_period_us (and cpu.rt_runtime_us
proportionally)? (Are the pauses shorter?) Is there any useful info in
/proc/$PID/stack during these periods?

> Is that expected behavior?

Someone with RT group schedulling knowledge may tell :-)

HTH,
Michal

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Short process stall after assigning it to a cgroup

[Ronny Meeus]

Op wo 23 jun. 2021 om 19:28 schreef Michal Koutn√Ω <mkoutny-IBi9RG/b67k@public.gmane.org>:
>
> Hello Ronny.
>
> On Mon, Jun 14, 2021 at 05:29:35PM +0200, Ronny Meeus <ronny.meeus@gmail.com> wrote:
> > All apps are running in the realtime domain and I'm using kernel 4.9
> > and cgroup v1. [...]  when it enters a full load condition [...]
> > I start to gradually reduce the budget of the cgroup until the system
> > is idle enough.
>
> Has your application some RT requirements or is there other reason why
> you use group RT allocations? (When your app seems to require all CPU
> time, you decide to curb it. And it still fullfills RT requirements?)
>

The application does not have strict RT requirements.
The main reason for using cgroups is to reduce the load of the high
consumer applications when the system is under high load so that also
lower prio apps can have a portion of the CPU.
We were working with fixed croups initially but this has the big
disadvantage that the unused budget configured in one group cannot be
used by another group and as such the processing power is basically
lost.

>
> > But sometimes, immediately after the process assignment, it stops for
> > a short period (something like 1 or 2s) and then starts to consume 40%
> > again.
>
> What if you reduce cpu.rt_period_us (and cpu.rt_runtime_us
> proportionally)? (Are the pauses shorter?) Is there any useful info in
> /proc/$PID/stack during these periods?
>

I tried to use shorter periods like 100ms instead of 1s but the
problem is still observed.
Using a proportionally reducing algo is more complex to implement and
I think would not solve the issue either.

About the stack: it is difficult to know from the SW when the issue
happens so dumping the stack is not easy I think but it is a good
idea.
I will certainly think about it.
To observe the system I use a spirent traffic generator which shows me
the number of processed packets in a nice graph. In this way it is
easy to see that there are short peaks when the system is not
returning any packets.

> > Is that expected behavior?
>
> Someone with RT group schedulling knowledge may tell :-)
>
> HTH,
> Michal

Re: Short process stall after assigning it to a cgroup

[Michal Koutný]

[-- Attachment #1: Type: text/plain, Size: 992 bytes --]

On Fri, Jun 25, 2021 at 09:32:59AM +0200, Ronny Meeus <ronny.meeus-Re5JQEeQqe8@public.gmane.orgm> wrote:
> The application does not have strict RT requirements.

Can you even use the normal non-RT scheduling policy? ...

> We were working with fixed croups initially but this has the big
> disadvantage that the unused budget configured in one group cannot be
> used by another group and as such the processing power is basically
> lost.

...then your problem may be solvable with mere weights to adjust priorities
of competitors. (Or if you need to stick with RT policies you can assign
different priorities on task basis, I wouldn't use RT groups for that.)

> About the stack: it is difficult to know from the SW when the issue
> happens so dumping the stack is not easy I think but it is a good
> idea.
> I will certainly think about it.

You may sample it periodically or start prior a migration to get more
insights what's causing the delay. 

Regards,
Michal


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Short process stall after assigning it to a cgroup

[Ronny Meeus]

Op vr 25 jun. 2021 om 10:52 schreef Michal Koutn√Ω <mkoutny-IBi9RG/b67k@public.gmane.org>:
>
> On Fri, Jun 25, 2021 at 09:32:59AM +0200, Ronny Meeus <ronny.meeus@gmail.com> wrote:
> > The application does not have strict RT requirements.
>
> Can you even use the normal non-RT scheduling policy? ...

No since we want certain functions (like packet processing) to have a
higher priority than for example management activities.

>
> > We were working with fixed croups initially but this has the big
> > disadvantage that the unused budget configured in one group cannot be
> > used by another group and as such the processing power is basically
> > lost.
>
> ...then your problem may be solvable with mere weights to adjust priorities
> of competitors. (Or if you need to stick with RT policies you can assign
> different priorities on task basis, I wouldn't use RT groups for that.)
>
> > About the stack: it is difficult to know from the SW when the issue
> > happens so dumping the stack is not easy I think but it is a good
> > idea.
> > I will certainly think about it.
>
> You may sample it periodically or start prior a migration to get more
> insights what's causing the delay.
>

I observed that when I create the cgroups dynamically (at the moment I
need them) and immediately assign the threads to it, the symptom seems
to be gone.
I'm going to observe the system some more to confirm that the problem
is really solved in this way.

Thanks Michal for the time you spent on this.

> Regards,
> Michal
>