[RFC] How to handle the rules engine for cgroups

[Vivek Goyal <vgoyal@redhat.com>]

Hi,

While development is going on for cgroup and various controllers, we also
need a facility so that an admin/user can specify the group creation and
also specify the rules based on which tasks should be placed in respective
groups. Group creation part will be handled by libcg which is already
under development. We still need to tackle the issue of how to specify
the rules and how these rules are enforced (rules engine).

I have gathered few views, with regards to how rule engine can possibly be
implemented, I am listing these down.

Proposal 1
==========
Let user space daemon hanle all that. Daemon will open a netlink socket
and receive the notifications for various kernel events. Daemon will
also parse appropriate admin specified rules config file and place the
processes in right cgroup based on rules as and when events happen.

I have written a prototype user space program which does that. Program 
can be found here. Currently it is in very crude shape.

http://people.redhat.com/vgoyal/misc/rules-engine-daemon/user-id-based-namespaces.patch

Various people have raised two main issues with this approach.

- netlink is not a reliable protocol.
	- Messages can be dropped and one can loose message. That means a
	  newly forked process might never go into right group as meant.

- How to handle delays in rule exectuion?
	- For example, if an "exec" happens and by the time process is moved to
	 right group, it might have forked off few more processes or might
	 have done quite some amount of memory allocation which will be
   	 charged to the wring group. Or, newly exec process might get
 	 killed in existing cgroup because of lack of memory (despite the
	 fact that destination cgroup has sufficient memory).

Proposal 2
==========
Implement one or more kernel modules which will implement the rule engine.
User space program can parse the config files and pass it to module.
Kernel will be patched only on select points to look for the rules (as
provided by modules). Very minimal code running inside the kernel if there
are no rules loaded.

Concerns:
- Rules can become complex and we don't want to handle that complexity in
  kernel.

Pros:
- Reliable and precise movement of tasks in right cgroup based on rules.

Proposal 3
==========
How about if additional parameters can be passed to system calls and one
can pass destination cgroup as additional parameter. Probably something
like sys_indirect proposal. Maybe glibc can act as a wrapper to pass
additional parameter so that applications don't need any modifications.

Concerns:
========
- Looks like sys_indirect interface for passing extra flags was rejected.
- Requires extra work in glibc which can also involve parsing of rule
  files. :-(

Proposal 4
==========
Some vauge thoughts are there regarding how about kind of freezing the
process or thread upon fork, exec and unfreeze it once the thread has been
placed in right cgroup. 

Concerns:
========
- Requires reliable netlink protocol otherwise there is a possibility that
  a task never gets unfrozen.
- On what basis does one freeze a thread. There might not be any rules to
  process for that thread we will unnecessarily delay it.


Please provide your inputs regarding what's the best way to handle the
rules engine.

To me, letting the rules live in separate module/modules seems to be a
reasonable way to move forward which will provide reliable and timely
execution of rules and by making it modular, we can remove most of the
complexity from core kernel code.

Thanks
Vivek