cgroups and nice

cgroups and nice

[Marat Khalili]

I have a question as a cgroup cpu limits user: how does it interact with 
nice? Documentation creates the impression that, as long as number of 
processes demanding the cpu time exceeds number of available cores, time 
allocated will be proportional to configured cpu.shares. However, in 
practice I observe that group with niced processes significantly under 
perform.

For example, suppose on a 6-core box /cgroup/cpu/group1/cpu.shares is 
400, and /cgroup/cpu/group2/cpu.shares is 200.
1) If I run `stress -c 6` in both groups, I should see approximately 
400% of cpu time in group1 and 200% in group2 in top output, regardless 
of their relative nice value.
2) If I run `nice -n 19 stress -c 1` in cgroup1 and `stress -c 24` in 
group2, I should see at least 100% of cpu time in group1.

What I see is significantly less cpu time in group1 if group1 processes 
happen to have greater nice value, and especially if group2 have greater 
number of processes involved: cpu load of group1 in example 2 can be as 
low as 20%. It may create tensions among users in my case; how can this 
be avoided except by renicing all processes to the same value?

> $ uname -a
> Linux redacted 2.6.32-642.11.1.el6.x86_64 #1 SMP Fri Nov 18 19:25:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

> $ lsb_release -a
> LSB Version:	:base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch
> Distributor ID:	CentOS
> Description:	CentOS release 6.8 (Final)
> Release:	6.8
> Codename:	Final

(My apologies if I'm posting to incorrect list.)

--

With Best Regards,
Marat Khalili

Re: cgroups and nice

[Dhaval Giani]

[Resending because gmail doesn't understand when to go plaintext :-) ]
[Added a few other folks who might have something to say about it]

On Fri, Nov 25, 2016 at 9:34 AM, Marat Khalili <mkh@rqc.ru> wrote:
> I have a question as a cgroup cpu limits user: how does it interact with
> nice? Documentation creates the impression that, as long as number of
> processes demanding the cpu time exceeds number of available cores, time
> allocated will be proportional to configured cpu.shares. However, in
> practice I observe that group with niced processes significantly under
> perform.
>
> For example, suppose on a 6-core box /cgroup/cpu/group1/cpu.shares is 400,
> and /cgroup/cpu/group2/cpu.shares is 200.
> 1) If I run `stress -c 6` in both groups, I should see approximately 400% of
> cpu time in group1 and 200% in group2 in top output, regardless of their
> relative nice value.
> 2) If I run `nice -n 19 stress -c 1` in cgroup1 and `stress -c 24` in
> group2, I should see at least 100% of cpu time in group1.
>
> What I see is significantly less cpu time in group1 if group1 processes
> happen to have greater nice value, and especially if group2 have greater
> number of processes involved: cpu load of group1 in example 2 can be as low
> as 20%. It may create tensions among users in my case; how can this be
> avoided except by renicing all processes to the same value?
>
>> $ uname -a
>> Linux redacted 2.6.32-642.11.1.el6.x86_64 #1 SMP Fri Nov 18 19:25:05 UTC
>> 2016 x86_64 x86_64 x86_64 GNU/Linux
>

This is an old version of the kernel. Do you see the same behavior on
a newer version of the kernel? (4.8 is the latest stable kernel)

>
>> $ lsb_release -a
>> LSB Version:
>> :base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch
>> Distributor ID: CentOS
>> Description:    CentOS release 6.8 (Final)
>> Release:        6.8
>> Codename:       Final
>
>
> (My apologies if I'm posting to incorrect list.)
>
> --
>
> With Best Regards,
> Marat Khalili
> --

Thanks,
Dhaval

Re: cgroups and nice

[Dhaval Giani]

[Resending because gmail doesn't understand when to go plaintext :-) ]
[Added a few other folks who might have something to say about it]

On Fri, Nov 25, 2016 at 9:34 AM, Marat Khalili <mkh-CePww0+mnY4@public.gmane.org> wrote:
> I have a question as a cgroup cpu limits user: how does it interact with
> nice? Documentation creates the impression that, as long as number of
> processes demanding the cpu time exceeds number of available cores, time
> allocated will be proportional to configured cpu.shares. However, in
> practice I observe that group with niced processes significantly under
> perform.
>
> For example, suppose on a 6-core box /cgroup/cpu/group1/cpu.shares is 400,
> and /cgroup/cpu/group2/cpu.shares is 200.
> 1) If I run `stress -c 6` in both groups, I should see approximately 400% of
> cpu time in group1 and 200% in group2 in top output, regardless of their
> relative nice value.
> 2) If I run `nice -n 19 stress -c 1` in cgroup1 and `stress -c 24` in
> group2, I should see at least 100% of cpu time in group1.
>
> What I see is significantly less cpu time in group1 if group1 processes
> happen to have greater nice value, and especially if group2 have greater
> number of processes involved: cpu load of group1 in example 2 can be as low
> as 20%. It may create tensions among users in my case; how can this be
> avoided except by renicing all processes to the same value?
>
>> $ uname -a
>> Linux redacted 2.6.32-642.11.1.el6.x86_64 #1 SMP Fri Nov 18 19:25:05 UTC
>> 2016 x86_64 x86_64 x86_64 GNU/Linux
>

This is an old version of the kernel. Do you see the same behavior on
a newer version of the kernel? (4.8 is the latest stable kernel)

>
>> $ lsb_release -a
>> LSB Version:
>> :base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch
>> Distributor ID: CentOS
>> Description:    CentOS release 6.8 (Final)
>> Release:        6.8
>> Codename:       Final
>
>
> (My apologies if I'm posting to incorrect list.)
>
> --
>
> With Best Regards,
> Marat Khalili
> --

Thanks,
Dhaval

Re: cgroups and nice

[Marat Khalili]

On 29/11/16 00:13, Dhaval Giani wrote:
> This is an old version of the kernel. Do you see the same behavior on
> a newer version of the kernel? (4.8 is the latest stable kernel)
Sadly, RedHat is not very keen on updating their kernels. I did a quick 
experiment on Ubuntu box with kernel 4.4, and indeed it does not behave 
this strange way. So I'll write this off as a long fixed kernel bug, 
stick to auto-renicing in cron, and look forward to upgrading this 
CentOS machine some day. Thank you very much for your help.

--

With Best Regards,
Marat Khalili