[PATCH] CKRM: 0/10 Class Based Kernel Resource Management

[PATCH] CKRM: 0/10 Class Based Kernel Resource Management

[Gerrit Huizenga]

The following ten patches add the core of CKRM (Class Based Resource
Management) to Linux.  Current patches are against 2.6.10-rc2.  This
set of patches is essentailly a cleaned up version of what is
known on the ckrm-tech@lists.sourcerforge.net as the E16 code base.
As compared to E16, the patch breakout has been reorganized for easier
application to mainline with a number of stylistic cleanups more
in line with mainline kernel code.

The following patches include:

01-diff_ckrm_events:
	Base CKRM events, mods to existing kernel code

02-diff_delay_acct:
	More accurate accounting for CPU scheduling, IO scheduling

03-diff_ckrm_core:
	Main/core CKRM code, beginings of Resource Control Filesystem

04-diff_rcfs:
	Full directory suppport for rcfs

05-diff_taskclass:
	Task based management for CPU, memory and Disk I/O.

06-diff_sockclass:
	CKRM tracking for socket classes for inbound connection control,
	bandwidth control, etc.

07-diff_numtasks:
	Resource controller for number of tasks per class.

08-diff_listenaq:
	Resource Controller for prioritizing inbound connection
	requests.  Can control queue weights for multiple accept
	queues.

09-diff_rbce
	A very basic rules based classification engine for automatically
	adding tasks to classes.  Also includes an enhanced rules based
	classification engine with better per-process delay data and
	ability to better monitor class related activities.

10-diff_docs
	CKRM documentation.

Please send comments to ckrm-tech@lists.sourceforge.net

thanks,

gerrit

Re: [PATCH] CKRM: 0/10 Class Based Kernel Resource Management

[Andrew Morton]

Gerrit Huizenga <gh@us.ibm.com> wrote:
>
> The following ten patches add the core of CKRM (Class Based Resource
> Management) to Linux.

How useful is this code at present?  What are its limitations?  And what is
the plan for future enhancements?

Thanks.

Re: [PATCH] CKRM: 0/10 Class Based Kernel Resource Management

[Christoph Hellwig]

On Mon, Nov 29, 2004 at 10:44:49AM -0800, Gerrit Huizenga wrote:
> The following ten patches add the core of CKRM (Class Based Resource
> Management) to Linux.  Current patches are against 2.6.10-rc2.  This
> set of patches is essentailly a cleaned up version of what is
> known on the ckrm-tech@lists.sourcerforge.net as the E16 code base.
> As compared to E16, the patch breakout has been reorganized for easier
> application to mainline with a number of stylistic cleanups more
> in line with mainline kernel code.

And where's the people who wrote the code?  Are people at IBM really
all anxious cowards these days that can't submit their own code but have
to abuse a highlevel manager for it.

I must also say that I'm a bit disappointed by you, Gerrit.  Either you
haven't actually read the code or I vastly overrated your taste.

Re: [PATCH] CKRM: 0/10 Class Based Kernel Resource Management

[Gerrit Huizenga]

On Mon, 29 Nov 2004 12:23:58 PST, Andrew Morton wrote:
> Gerrit Huizenga <gh@us.ibm.com> wrote:
> >
> > The following ten patches add the core of CKRM (Class Based Resource
> > Management) to Linux.
> 
> How useful is this code at present?  What are its limitations?  And what is
> the plan for future enhancements?

This set of code alone allows for creation of classes which include
per-class resource accounting (including delay accounting), basic
task management for memory, CPU and disk IO, limited socket & listener
queue management for networking, and the related rules based infrastructure.

So, in short, it is a useful set of code to work with to demonstrate
real utility with CKRM.  However, this submission is not as full featured
as is being used by those on the ckrm-tech list, such as the PlanetLab
work.  There are also things in SLES9 that are more featureful than
this set although those will be worked into here in time.

It does not have the full memory management and scheduler support that
other versions do and I'm not yet convinced that those are ready to
submit.  Future enhancements will start with the cleanups as recommended
by lkml so far (thanks all ;-) followed by more work on the scheduler
and memory management side in the short term.  There are also ways
to hook in additional resource controllers for any exhaustible resource,
e.g. file handles. setrlimit style resources, etc.

Most of the next level of changes will build on these and are based
on work currently in progress on the ckrm-tech list.  However, this is
a stripped down set of code which is believed to be stable (tested on
IA32, x86-64, PPC64) with a variety of config options using both
standard regression suites (e.g. LTP, kernbench, the ckrm tests, etc.).

gerrit

Re: [PATCH] CKRM: 0/10 Class Based Kernel Resource Management

[Gerrit Huizenga]

On Mon, 29 Nov 2004 22:32:51 GMT, Christoph Hellwig wrote:
> On Mon, Nov 29, 2004 at 10:44:49AM -0800, Gerrit Huizenga wrote:
> > The following ten patches add the core of CKRM (Class Based Resource
> > Management) to Linux.  Current patches are against 2.6.10-rc2.  This
> > set of patches is essentailly a cleaned up version of what is
> > known on the ckrm-tech@lists.sourcerforge.net as the E16 code base.
> > As compared to E16, the patch breakout has been reorganized for easier
> > application to mainline with a number of stylistic cleanups more
> > in line with mainline kernel code.
> 
> And where's the people who wrote the code?  Are people at IBM really
> all anxious cowards these days that can't submit their own code but have
> to abuse a highlevel manager for it.
> 
> I must also say that I'm a bit disappointed by you, Gerrit.  Either you
> haven't actually read the code or I vastly overrated your taste.

LOL, Christoph, Christoph...  No, folks at IBM working on CKRM aren't
cowards, you'll see most of them on ckrm-tech or working with distros
or end users of the code or writing new code.  You've seen them at KS
and OLS.  Many are soaked up into a couple of other deliverables at
the moment but they'll be back.

We've needed some help in stabilizing and aggregating the code and so
we are using my copious free cycles.  And, no, silly Christoph, I'm no
manager and never have been.

Yes, I've read the code, yes I cleaned up a bunch of it.  But rather
than me sitting on it forever cleaning, I figured I'd shake it out and
get more eyes providing feedback which is where LKML is very handy.
Lots of eyes making sure that it is useful and minimally invasive never
hurts.  And stewing it around just ckrm-tech is great for functionality
but less great for converging and cleanup.  Hence, time to get this
into the wider eye of the community, including yours.

If you see needed cleanups, send me patches.  If they make sense, I'll
integrate.  If not, I'll go back to that training class on how to
deal with Christoph on LKML.  ;-)

gerrit

Re: [PATCH] CKRM: 0/10 Class Based Kernel Resource Management

[Greg KH]

On Mon, Nov 29, 2004 at 10:44:49AM -0800, Gerrit Huizenga wrote:
> 09-diff_rbce
> 	A very basic rules based classification engine for automatically
> 	adding tasks to classes.  Also includes an enhanced rules based
> 	classification engine with better per-process delay data and
> 	ability to better monitor class related activities.

This one didn't look like it made it to lkml.

Oh, and I stopped reviewing the other patches in the series, as the same
comments pretty much applied to them :(

thanks,

greg k-h

Re: [PATCH] CKRM: 0/10 Class Based Kernel Resource Management

[Gerrit Huizenga]

On Mon, 29 Nov 2004 18:43:21 PST, Greg KH wrote:
> On Mon, Nov 29, 2004 at 10:44:49AM -0800, Gerrit Huizenga wrote:
> > 09-diff_rbce
> > 	A very basic rules based classification engine for automatically
> > 	adding tasks to classes.  Also includes an enhanced rules based
> > 	classification engine with better per-process delay data and
> > 	ability to better monitor class related activities.
> 
> This one didn't look like it made it to lkml.
> 
> Oh, and I stopped reviewing the other patches in the series, as the same
> comments pretty much applied to them :(

Yeah, I checked marc earlier and didn't see it there.  I'm making
the suggested changes now, will resend the whole set when I apply
and test a bit.

gerrit

Re: [PATCH] CKRM: 0/10 Class Based Kernel Resource Management

[Greg KH]

On Mon, Nov 29, 2004 at 06:48:19PM -0800, Gerrit Huizenga wrote:
> 
> On Mon, 29 Nov 2004 18:43:21 PST, Greg KH wrote:
> > On Mon, Nov 29, 2004 at 10:44:49AM -0800, Gerrit Huizenga wrote:
> > > 09-diff_rbce
> > > 	A very basic rules based classification engine for automatically
> > > 	adding tasks to classes.  Also includes an enhanced rules based
> > > 	classification engine with better per-process delay data and
> > > 	ability to better monitor class related activities.
> > 
> > This one didn't look like it made it to lkml.
> > 
> > Oh, and I stopped reviewing the other patches in the series, as the same
> > comments pretty much applied to them :(
> 
> Yeah, I checked marc earlier and didn't see it there.  I'm making
> the suggested changes now, will resend the whole set when I apply
> and test a bit.

And the questions that I and others had about portions of the code?
Please address them in responses to the messages and don't expect us to
try to pick out if they are still present in the next round of patches
:)

thanks,

greg k-h

Re: [PATCH] CKRM: 0/10 Class Based Kernel Resource Management

[Gerrit Huizenga]

On Mon, 29 Nov 2004 20:07:53 PST, Greg KH wrote:
> On Mon, Nov 29, 2004 at 06:48:19PM -0800, Gerrit Huizenga wrote:
> > 
> > On Mon, 29 Nov 2004 18:43:21 PST, Greg KH wrote:
> > > On Mon, Nov 29, 2004 at 10:44:49AM -0800, Gerrit Huizenga wrote:
> > > > 09-diff_rbce
> > > > 	A very basic rules based classification engine for automatically
> > > > 	adding tasks to classes.  Also includes an enhanced rules based
> > > > 	classification engine with better per-process delay data and
> > > > 	ability to better monitor class related activities.
> > > 
> > > This one didn't look like it made it to lkml.
> > > 
> > > Oh, and I stopped reviewing the other patches in the series, as the same
> > > comments pretty much applied to them :(
> > 
> > Yeah, I checked marc earlier and didn't see it there.  I'm making
> > the suggested changes now, will resend the whole set when I apply
> > and test a bit.
> 
> And the questions that I and others had about portions of the code?
> Please address them in responses to the messages and don't expect us to
> try to pick out if they are still present in the next round of patches
> :)

Yeah, yeah, working on it.  Will generically apply a number of your
changes.  Just sent you a couple of questions about a couple of other
changes.

gerrit

Re: [PATCH] CKRM: 0/10 Class Based Kernel Resource Management

[Nikita Danilov]

Christoph Hellwig <hch@infradead.org> writes:

[...]

>
> And where's the people who wrote the code?  Are people at IBM really
> all anxious cowards these days that can't submit their own code but have
> to abuse a highlevel manager for it.

Once upon a time everybody and his dog (old dogs and puppies alike) at
namesys used to send patches directly to Linus. Which resulted in the
latter being confused in a maze of @namesys.com addresses all similar,
and so he asked Hans to send all patches to him from Hans' address.

Nikita.

RE: [ckrm-tech] Re: [PATCH] CKRM: 0/10 Class Based Kernel Resource Management

[Marc E. Fiuczynski]

Hi Andrew and Gerrit,

I integrated CKRM with the kernel used by PlanetLab (www.planet-lab.org),
and I believe we (PlanetLab) are the first to use CKRM in a production
setting.  Our kernel is deployed on roughly 100 machines worldwide and we
intend to upgrade all of our machines (roughly 400) over the next few weeks.
Our kernel uses linux-vservers to create rather thin "virtual machines" (for
the lack of a better name), but uses CKRM to provide for performance
isolation between each vserver.  The integration between CKRM and vservers
was easy!

PlanetLab is used by tons of researchers.  The software of each research is
placed into a vserver, and each PlanetLab machine typically has anywhere
from 20-40 actively running vservers running at a constant load of roughly
20.  Some of the services running on PlanetLab have been discussed on
Slashdot.

Gerrit mentioned that PlanetLab uses a more featureful version of CKRM.
This is true.  For each vserver we create a corresponding CKRM class, and
then use the rule-based classification engine (RBCE) to automatically
classify vserver processes to the appropriate CKRM class.  We are itching to
deploy the CKRM memory controller and IO controller, but unfortunately those
have not been ready for prime time.  For now, we've only deployed a variant
of CKRM's cpu scheduler.  We currently do not leverage the hierarchical
support provided by CKRM, but envision a use for it in the future.

Unlike the posted CKRM patchset, the CPU, IO, and Memory controller make
more invasive modifications to various kernel subsystems.  I suspect that
the CPU and IO controllers can be completely modularized into the pluggable
CPU and IO framework that Con and Jens posted earlier, if that's the
direction that mainline is heading.  The CKRM memory controller makes a few
choice modifications to mm/vmscan.c, which I suspect will rouse a fair
amount of dicussion on LKML when the day arrives.

Hope this helps.

Cheers,
Marc

ps. our kernel is based on FC2 1.521 (2.6.8.1) and is available via anon
cvs.
cvs -d :pserver:anon@cvs.planet-lab.org:/cvs co linux-2.6

> -----Original Message-----
> From: ckrm-tech-admin@lists.sourceforge.net
> [mailto:ckrm-tech-admin@lists.sourceforge.net]On Behalf Of Gerrit
> Huizenga
> Sent: Monday, November 29, 2004 5:34 PM
> To: Andrew Morton
> Cc: linux-kernel@vger.kernel.org; riel@redhat.com; mason@suse.com;
> ckrm-tech@lists.sourceforge.net
> Subject: [ckrm-tech] Re: [PATCH] CKRM: 0/10 Class Based Kernel Resource
> Management
>
>
>
> On Mon, 29 Nov 2004 12:23:58 PST, Andrew Morton wrote:
> > Gerrit Huizenga <gh@us.ibm.com> wrote:
> > >
> > > The following ten patches add the core of CKRM (Class Based Resource
> > > Management) to Linux.
> >
> > How useful is this code at present?  What are its limitations?
> And what is
> > the plan for future enhancements?
>
> This set of code alone allows for creation of classes which include
> per-class resource accounting (including delay accounting), basic
> task management for memory, CPU and disk IO, limited socket & listener
> queue management for networking, and the related rules based
> infrastructure.
>
> So, in short, it is a useful set of code to work with to demonstrate
> real utility with CKRM.  However, this submission is not as full featured
> as is being used by those on the ckrm-tech list, such as the PlanetLab
> work.  There are also things in SLES9 that are more featureful than
> this set although those will be worked into here in time.
>
> It does not have the full memory management and scheduler support that
> other versions do and I'm not yet convinced that those are ready to
> submit.  Future enhancements will start with the cleanups as recommended
> by lkml so far (thanks all ;-) followed by more work on the scheduler
> and memory management side in the short term.  There are also ways
> to hook in additional resource controllers for any exhaustible resource,
> e.g. file handles. setrlimit style resources, etc.
>
> Most of the next level of changes will build on these and are based
> on work currently in progress on the ckrm-tech list.  However, this is
> a stripped down set of code which is believed to be stable (tested on
> IA32, x86-64, PPC64) with a variety of config options using both
> standard regression suites (e.g. LTP, kernbench, the ckrm tests, etc.).
>
> gerrit

Re: [ckrm-tech] Re: [PATCH] CKRM: 0/10 Class Based Kernel Resource Management

[Andrew Morton]

"Marc E. Fiuczynski" <mef@CS.Princeton.EDU> wrote:
>
> I integrated CKRM with the kernel used by PlanetLab (www.planet-lab.org),
> and I believe we (PlanetLab) are the first to use CKRM in a production
> setting.
> ...
> Hope this helps.

It does, thanks.

A concern which I have about the CKRM implementation is that the patches
which have been sent out appear to be simply the "core" of CKRM, plus
minimally-intrusive hooks.  I have the impression that this core will not
be terribly useful to real-world users and that follow-on patches will be
required to add more functionality and to wire up more instrumentation and
control points.

I would not like to be in a situation where we merge the "core" patch, but
the as-yet-unseen follow-on patches which make CKRM useful and complete end
up creating a big unmaintainable mess.  We end up not wanting to go
forwards and being unable to go backwards.

IOW: I think we need to see a reasonably-close-to-final implementation of
CKRM before we can take it much further.

Re: [ckrm-tech] Re: [PATCH] CKRM: 0/10 Class Based Kernel Resource Management

[Gerrit Huizenga]

On Fri, 03 Dec 2004 16:40:34 PST, Andrew Morton wrote:
> "Marc E. Fiuczynski" <mef@CS.Princeton.EDU> wrote:
> >
> > I integrated CKRM with the kernel used by PlanetLab (www.planet-lab.org),
> > and I believe we (PlanetLab) are the first to use CKRM in a production
> > setting.
> > ...
> > Hope this helps.
> 
> It does, thanks.
> 
> A concern which I have about the CKRM implementation is that the patches
> which have been sent out appear to be simply the "core" of CKRM, plus
> minimally-intrusive hooks.  I have the impression that this core will not
> be terribly useful to real-world users and that follow-on patches will be
> required to add more functionality and to wire up more instrumentation and
> control points.
> 
> I would not like to be in a situation where we merge the "core" patch, but
> the as-yet-unseen follow-on patches which make CKRM useful and complete end
> up creating a big unmaintainable mess.  We end up not wanting to go
> forwards and being unable to go backwards.
> 
> IOW: I think we need to see a reasonably-close-to-final implementation of
> CKRM before we can take it much further.

Understood.  We do have a more complete set of patches floating around,
although most are ported to an existing distro rather than set for
current mainline adoption.  But if we can get general consensus on
the patches (once I finish the current round of cleanup and testing),
we do have work in memory management, IO scheduling, and even CPU
scheduling (the latter being the most debatable for mainline acceptance
given the rate of scheduler replacements in recent past) that are
being used today.

We can dump the current, raw distro patches or the rest of the e16
patch set from ckrm-tech on you although I believe they will need
some significant review/modification to be mainline acceptable yet.
One big problem is that these changes are somewhat hard to maintain as
distinct from mainline and yet remain relatively current.  There are
several developers working in distinct areas and each area moves at
its own pace.  Hence, I'd like to get to a more stable -mm compatible
core, and build up from there.  As we see that the entire set approaches
stability/utility, we can push from the core up through the working set
of resource controllers.

If getting you a set of patches for general concept review as based
on a current distro would help, just say the word.  However, getting
those up to current mainline, integrated with each other and fully
tested (while holding their development stable long enough to do that)
is the requirement, well, that will take us a fair bit longer.

Part of the goal of this posting was to start to stabilize a core
and improve on it, rather than try to deliver an entire project as a
moderately large set of changes as a fait accompli.  And, we are more
than willing to continue to tweak and tune this to be generally useful
to a wider audience, even though we have a set that works well for
some groups needing better workload management.

So, Andrew, can you clarify how much we need to put in your hands, how
well tested it needs to be and how clean and current the entire set needs
to be before this is ready for -mm testing?

gerrit

Re: [ckrm-tech] Re: [PATCH] CKRM: 0/10 Class Based Kernel Resource Management

[Andrew Morton]

Gerrit Huizenga <gh@us.ibm.com> wrote:
>
> So, Andrew, can you clarify how much we need to put in your hands, how
> well tested it needs to be and how clean and current the entire set needs
> to be before this is ready for -mm testing?

Well we can toss stuff into -mm any old time really.  Doing it too early
will cause rather a lot of difficulty and churn at both ends - working
against -mm can be an extra burden at times.

I'd say that it would be best to wait until the code is, in your opinion,
in a Linus-mergeable form.  Then after one lkml review round and any
subsequent rework we should be in good shape.