cgroups: Assigning tasks to a cgroup slow and does not scale

cgroups: Assigning tasks to a cgroup slow and does not scale

[Martin Baum]

Hello,

assigning a task to new cgroup takes an unexpected high amount of time  
on my system - about 0.05s. Strangely it does it does not scale with  
more processes: When I run 10 processes that do nothing but change  
their cgroups each cgroup change takes about 0.5 seconds while CPU  
usage remains very low. So my system maxes out at 20 cgroup changes  
per second while cpu and disk usage remain minimal (according to top).  
Is this expected behaviour?

It is a problem for me: I run a daemon that forks and calls suid for  
each request. I want to assign each of these forks to a different  
cgroup. However this currently kills performance going down from >100  
req/sec to about 20 req/sec.


Here is my test case:

cat > test_cgroup.sh <<EOF
#!/bin/bash
mkdir /tmp/cg
mount -t cgroup none -o cpuacct /tmp/cg
mkdir /tmp/cg/test
while /bin/true; do
   time (/bin/echo \$\$ > /tmp/cg/test/tasks)
   time (/bin/echo \$\$ > /tmp/cg/tasks)
done;
EOF


Start test:

bash ./test_cgroup.sh &  <--- execute this 10 times and watch the time  
reports go up to 0.5sec

I already tested this with a different controller (cpuset): Same  
result. I tried different kernel configurations relating to cgroup -  
no success. I also tried 2.6.32.40 - same result.

What is confusing me most is that disk and cpu usage remains low  
during the test. Maybe this is related to locking?


My system:

Linux testserver 2.6.39.1 #1 SMP Sat Jun 4 09:21:40 CEST 2011 i686  
Intel(R) Core(TM)2 Duo CPU E7500 @ 2.93GHz GenuineIntel GNU/Linux


Kernel is vanilla source. Relevant part of .config:

$ grep -i cgroup .config
CONFIG_CGROUPS=y
# CONFIG_CGROUP_DEBUG is not set
# CONFIG_CGROUP_NS is not set
CONFIG_CGROUP_FREEZER=y
# CONFIG_CGROUP_DEVICE is not set
CONFIG_CGROUP_CPUACCT=y
# CONFIG_CGROUP_MEM_RES_CTLR is not set
# CONFIG_CGROUP_PERF is not set
# CONFIG_CGROUP_SCHED is not set
# CONFIG_BLK_CGROUP is not set



best regards,
Martin

Re: cgroups: Assigning tasks to a cgroup slow and does not scale

[Mike Galbraith]

On Sat, 2011-06-04 at 10:23 +0200, Martin Baum wrote:
> Hello,
> 
> assigning a task to new cgroup takes an unexpected high amount of time  
> on my system - about 0.05s. Strangely it does it does not scale with  
> more processes: When I run 10 processes that do nothing but change  
> their cgroups each cgroup change takes about 0.5 seconds while CPU  
> usage remains very low. So my system maxes out at 20 cgroup changes  
> per second while cpu and disk usage remain minimal (according to top).  
> Is this expected behaviour?

Attaching a task invokes synchronize_rcu() along the way, so yup, it's
the expected behavior.

> It is a problem for me: I run a daemon that forks and calls suid for  
> each request. I want to assign each of these forks to a different  
> cgroup. However this currently kills performance going down from >100  
> req/sec to about 20 req/sec.

You can work around it via static cgroups with an agent waiting inside
each one.  Tell your agent what to do, he forks off a worker.  You can
inherit a cgroup quickly, but moving between cgroups is very expensive,
as you noticed.

(right way is eventually stop exporting rcu grace periods to userland)

	-Mike