[PATCH 0/8] Domain Groups: Introduction

[PATCH 0/8] Domain Groups: Introduction

[Chris]

This patchset allows the user to define and manage groups of domains.
The patch augments the xm utility with the following commands:
grp-create, grp-shutdown, grp-destroy, grp-reboot, grp-pause,
grp-unpause, grp-save, grp-restore, grp-join, and grp-migrate.

A goal during development of group operations was to match support for
common domain operations: create, shutdown, destroy, reboot, pause,
unpause, save, restore, and migrate.  Their group-specific counterparts
do what you would expect, but operate on a group of domains instead of
on a single domain.

Groups are defined using a configuration file (similar to the domain
configuration file) that specifies member domains.  Groups have names,
IDs, and UUIDs, much like domains.  Group membership is dynamic; running
domains can be moved between groups using the 'xm grp-join' command.
Domains can be a member of one and only one group at any time.

We also made use of the Xen API, so third party developers should find
it feasible to drive group operations without relying solely on the xm
interface.

To achieve grouping functionality we chose to augment both the control
stack and the hypervisor.  Augmenting the hypervisor allows, among other
things, integration with a MAC framework (either the proposed XSM
framework or existing the ACM framework), which will produce a powerful
policy mechanism to define access control properties in terms of domain
groups.

Two default groups are defined: Group-0 and the Null-Group.  Domain-0
is, by default, the only domain in Group-0.  It is our hope that Group-0
will be a useful way to collect and isolate privileged domains,
particularly during the decomposition of domain 0.  The Null-Group is
used for domains that have not been explicitly put into a group.  Upon
integration with a MAC framework it would be possible to impose rules on
the Null-Group to make it a jail where no two members can communicate;
it should also be equally feasible to make the Null-Group a "don't care"
zone where any inter-domain interactions are allowed.

On the purely practical side, we believe group migration will be an
attractive tool allowing system administrators to move groups of domains
between machines.  Live migration is also supported.  For usage we
envision, for example, a group of interdependent database, web, and mail
servers, where migrating these domains as a group has obvious
operational advantages.  Also, for hosting service providers, it seems
useful to group domains belonging to a specific customer.

Future Work:

1. Currently lacking is support for grp-suspend/grp-resume.  Work is
underway on this feature.
2. Operation ordering: it is advantageous to guarantee the order of
group operations.  A practical example is to ensure that the group's
database server is always running before and after the group's web server.

Patch Application:
- Patches should apply cleanly to cs 14006
- Apply patches in any order.

----

 b/tools/examples/xmexample.grp                 |   20 +
 b/tools/libxc/xc_domain_group.c                |  100 +++++++
 b/tools/python/xen/xend/XendDomainGroup.py     |  345
+++++++++++++++++++++++++
 b/tools/python/xen/xend/XendDomainGroupInfo.py |  239 +++++++++++++++++
 b/tools/python/xen/xm/group.py                 |  274 +++++++++++++++++++
 b/xen/common/domgrp.c                          |  317
++++++++++++++++++++++
 b/xen/common/domgrpctl.c                       |  134 +++++++++
 b/xen/include/public/domgrpctl.h               |   86 ++++++
 b/xen/include/xen/domgrp.h                     |   36 ++
 tools/examples/Makefile                        |    1
 tools/libxc/Makefile                           |    1
 tools/libxc/xc_domain.c                        |    4
 tools/libxc/xc_private.h                       |   31 ++
 tools/libxc/xenctrl.h                          |   31 ++
 tools/python/xen/lowlevel/xc/xc.c              |  241 +++++++++++++++--
 tools/python/xen/xend/XendCheckpoint.py        |   49 +++
 tools/python/xen/xend/XendClient.py            |    1
 tools/python/xen/xend/XendConfig.py            |   25 +
 tools/python/xen/xend/XendConstants.py         |   11
 tools/python/xen/xend/XendDomain.py            |   10
 tools/python/xen/xend/XendDomainInfo.py        |   26 +
 tools/python/xen/xend/XendError.py             |    4
 tools/python/xen/xend/server/XMLRPCServer.py   |   34 ++
 tools/python/xen/xm/create.py                  |    8
 tools/python/xen/xm/main.py                    |  161 +++++++++++
 xen/arch/ia64/xen/xensetup.c                   |    7
 xen/arch/powerpc/powerpc64/hypercall_table.S   |    1
 xen/arch/powerpc/setup.c                       |    7
 xen/arch/x86/setup.c                           |    7
 xen/arch/x86/x86_32/entry.S                    |    2
 xen/arch/x86/x86_64/entry.S                    |    2
 xen/common/Makefile                            |    2
 xen/common/domain.c                            |    8
 xen/common/domctl.c                            |   11
 xen/include/public/domctl.h                    |    2
 xen/include/public/xen.h                       |    5
 xen/include/xen/hypercall.h                    |    5
 xen/include/xen/sched.h                        |   21 +
 38 files changed, 2220 insertions(+), 49 deletions(-)

Re: [PATCH 0/8] Domain Groups: Introduction

[Daniel P. Berrange]

On Tue, Feb 20, 2007 at 02:55:33PM -0500, Chris wrote:
> This patchset allows the user to define and manage groups of domains.
> The patch augments the xm utility with the following commands:
> grp-create, grp-shutdown, grp-destroy, grp-reboot, grp-pause,
> grp-unpause, grp-save, grp-restore, grp-join, and grp-migrate.
> 
> A goal during development of group operations was to match support for
> common domain operations: create, shutdown, destroy, reboot, pause,
> unpause, save, restore, and migrate.  Their group-specific counterparts
> do what you would expect, but operate on a group of domains instead of
> on a single domain.

What is the error handling policy ? eg, if 'save' fails will it just
skip over that domain and save the rest, or will it abort and restart
the ones it had saved upto that point, or just abort ? Likewise for
the other group operations

> 2. Operation ordering: it is advantageous to guarantee the order of
> group operations.  A practical example is to ensure that the group's
> database server is always running before and after the group's web server.

This somewhat ties into my question on error handling above, but also
raises the question of how do you know whether the DB server has completed
booting far enough to be able to start the web server. The latter seems a
pretty much impossible question to answer reliably unless you've got
some notification from the app inside the guest to say it is ready to
serve. 

Regards,
Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=| 

Re: [PATCH 0/8] Domain Groups: Introduction

[Chris]

Daniel P. Berrange wrote:
> On Tue, Feb 20, 2007 at 02:55:33PM -0500, Chris wrote:
>> A goal during development of group operations was to match support for
>> common domain operations: create, shutdown, destroy, reboot, pause,
>> unpause, save, restore, and migrate.  Their group-specific counterparts
>> do what you would expect, but operate on a group of domains instead of
>> on a single domain.
> 
> What is the error handling policy ? eg, if 'save' fails will it just
> skip over that domain and save the rest, or will it abort and restart
> the ones it had saved upto that point, or just abort ? Likewise for
> the other group operations

During save, any errors should throw an exception, which would halt the
grp-save -- quite possibly before all members are saved.  What happens
to the domain that failed the operation greatly depends on the error.
The failure may result in a domain that is left unscathed or defunct in
exactly the same way a failed operation on a non-grouped domain would.

The domains that were operated on successfully are not acted on in the
event of a failure in separate domain operation.  This was a design
choice.  Exceptions could instead be caught (and ignored), but I suspect
no matter what I did there would be someone who wants the opposite
behavior.  After an error the user can fall back to using non-group
commands (those all should still work on grouped domains) to either
restore the saved domains or whatever is appropriate for the situation.

Perhaps a configuration option for each operation could be added that
would allow the user to specify behavior in the event of a failure.

>> 2. Operation ordering: it is advantageous to guarantee the order of
>> group operations.  A practical example is to ensure that the group's
>> database server is always running before and after the group's web server.
> 
> This somewhat ties into my question on error handling above, but also
> raises the question of how do you know whether the DB server has completed
> booting far enough to be able to start the web server. The latter seems a
> pretty much impossible question to answer reliably unless you've got
> some notification from the app inside the guest to say it is ready to
> serve. 

I have spent just enough time thinking about that problem to realize it
was a big enough challenge to spin off into a separate project.  A first
step would be to provide the mechanism to impose order on basic domain
operations: create, shutdown, migrate, etc.  E.g., a mechanism to
provide a guarantee that domA is paused before domB.  From there
ordering could be extended with an app in the guest to provide
application specific readiness information.  I'm not saying this is the
way to go, but rather it's an important problem that needs some
discussion.  Thanks for bringing it up.  :)

The intent of this patchset is not to solve all domain grouping
problems.  Rather, it is a starting point designed to provide a base set
of functionality for domain groups.  Inclusion in the mainstream would
make life a lot easier and allow me time to work on improvements (like
the good ones you've suggested).

Cheers,
Chris

Re: [PATCH 0/8] Domain Groups: Introduction

[Keir Fraser]

On 20/2/07 19:55, "Chris" <hap10@tycho.ncsc.mil> wrote:

> On the purely practical side, we believe group migration will be an
> attractive tool allowing system administrators to move groups of domains
> between machines.  Live migration is also supported.  For usage we
> envision, for example, a group of interdependent database, web, and mail
> servers, where migrating these domains as a group has obvious
> operational advantages.  Also, for hosting service providers, it seems
> useful to group domains belonging to a specific customer.

I think we need to decide whether the benefits of supporting this concept
down to the hypervisor make it worth adding significant extra management
code at ring 0.

 -- Keir

RE: [PATCH 0/8] Domain Groups: Introduction

[Ian Pratt]

> On 20/2/07 19:55, "Chris" <hap10@tycho.ncsc.mil> wrote:
> 
> > On the purely practical side, we believe group migration will be an
> > attractive tool allowing system administrators to move groups of
domains
> > between machines.  Live migration is also supported.  For usage we
> > envision, for example, a group of interdependent database, web, and
mail
> > servers, where migrating these domains as a group has obvious
> > operational advantages.  Also, for hosting service providers, it
seems
> > useful to group domains belonging to a specific customer.
> 
> I think we need to decide whether the benefits of supporting this
concept
> down to the hypervisor make it worth adding significant extra
management
> code at ring 0.

It's certainly useful to be able to pause groups of domains e.g. when
debugging cluster filesystems etc. I think there were even patches for
this floating around some time back.

Ian

Re: [PATCH 0/8] Domain Groups: Introduction

[Keir Fraser]

On 20/2/07 23:01, "Ian Pratt" <m+Ian.Pratt@cl.cam.ac.uk> wrote:

>> I think we need to decide whether the benefits of supporting this
> concept
>> down to the hypervisor make it worth adding significant extra
> management
>> code at ring 0.
> 
> It's certainly useful to be able to pause groups of domains e.g. when
> debugging cluster filesystems etc.

That doesn't require Xen to be aware of the groups. If the domains need to
be paused as instantaneously as possible then a multicall of domctls may
well suffice.

There may be a more compelling argument for groups as a security
abstraction, but I think we need to stand back and work out the overall
story there with XSM and all.

 -- Keir

Re: [PATCH 0/8] Domain Groups: Introduction

[Chris]

Keir Fraser wrote:
> On 20/2/07 23:01, "Ian Pratt" <m+Ian.Pratt@cl.cam.ac.uk> wrote:
>> It's certainly useful to be able to pause groups of domains e.g. when
>> debugging cluster filesystems etc.
> 
> That doesn't require Xen to be aware of the groups. If the domains need to
> be paused as instantaneously as possible then a multicall of domctls may
> well suffice.

Using multicall is a different, but reasonable approach to operate on a
set of domains with a similar level of atomicity to what's currently in
the domgrps patch.  That said, I stopped well short of absolute
atomicity in the domgrps implementation to avoid being offensively
intrusive.  By extending the domgrps approach to work with the scheduler
it's possible to provide group operations with a level of atomicity not
achievable from the current mutlicall implementation.  However, deciding
what degree of atomicity is necessary falls into what I believe should
be a separate (but important) discussion.

> I think we need to decide whether the benefits of supporting
> this concept down to the hypervisor make it worth adding significant
> extra management code at ring 0.

This is the part of the discussion I most wanted to have.  Whatever
mechanism you use to operate on a set of domains, life is better when
the hypervisor is aware of the group abstraction.  With a group-aware
hypervisor there is a robustness gain because even if the entire control
stack falls over, group data can be re-populated from the hypervisor.
Also, having group data managed in the hypervisor provides a level of
separation between the group policy in the control stack and the
management mechanism in the VMM.  However, most interesting are the
opportunities gained with integration of XSM/ACM.  There are new
opportunities for isolation of domain groups, dom0 decomposition, and
more powerful primitives for the XSM policy language.  We can get into
more detail here if this high-level overview doesn't do enough to
justify the ~650 lines (including comments and whitespace) of additional
ring 0 code.

> There may be a more compelling argument for groups as a security
> abstraction, but I think we need to stand back and work out the overall
> story there with XSM and all.

Although both Domain Groups and XSM can stand on their own merits we've
been aiming for integration of two from the start.  The integration is
planned, but not yet started because we want to incorporate community
feedback on each project individually.  Improvements are in progress for
XSM and I'm certainly willing to discuss revisions for Domain Groups.

-Chris

Re: [PATCH 0/8] Domain Groups: Introduction

[Keir Fraser]

On 21/2/07 17:17, "Chris" <hap10@tycho.ncsc.mil> wrote:

> This is the part of the discussion I most wanted to have.  Whatever
> mechanism you use to operate on a set of domains, life is better when
> the hypervisor is aware of the group abstraction.  With a group-aware
> hypervisor there is a robustness gain because even if the entire control
> stack falls over, group data can be re-populated from the hypervisor.

There are other ways to store information in a way that persists across
daemon restarts. Like writing it to a file (perhaps via xenstored).

> Also, having group data managed in the hypervisor provides a level of
> separation between the group policy in the control stack and the
> management mechanism in the VMM.

A layered implementation, including separation of mechanism and policy, is
quite possible without putting the different layers at different privilege
levels.

> Although both Domain Groups and XSM can stand on their own merits

I'm not convinced! :-)

 -- Keir

Re: [PATCH 0/8] Domain Groups: Introduction

[Chris]

We chose to make hypervisor extensions not because it was the only or
even the least intrusive way to do operations on a set of domains, but
because it creates new opportunities to innovate that aren't possible
otherwise.  Although this patchset provides immediately useful tools,
its major contribution is a foundation on which future developments can
be made.

One of our future developments will be VMM access control frameworks
like XSM that have the ability to specify access control policy for
groups of domains instead of just on individual domains.  For us, this
greatly simplifies both policy development and analysis.

-Chris

Re: [PATCH 0/8] Domain Groups: Introduction

[Keir Fraser]

On 22/2/07 20:39, "Chris" <hap10@tycho.ncsc.mil> wrote:

> One of our future developments will be VMM access control frameworks
> like XSM that have the ability to specify access control policy for
> groups of domains instead of just on individual domains.  For us, this
> greatly simplifies both policy development and analysis.

There's nothing preventing you from individually and separately applying
group rules to all domains of a group. There's no reason that a single rule
at the policy-language level cannot correspond to multiple rules within the
hypervisor. From this point of view Domain Groups are potentially an
optimisation that may be worthwhile if observed usage of XSM indicates that
creating extra rules inside Xen is measurably costly in time or space.

 -- Keir