Re: [PATCH 0 of 3 v5/leftover] Automatic NUMA placement for xl

[Ian Campbell <Ian.Campbell@citrix.com>]

On Fri, 2012-07-20 at 12:43 +0100, Andre Przywara wrote:
> On 07/20/2012 01:07 PM, David Vrabel wrote:
> > On 16/07/12 18:13, Dario Faggioli wrote:
> >> Hello again,
> >>
> >> This is a new version (fixing a small bug) of this series:
> >> <patchbomb.1341932613@Solace>, which in turn was a repost of what remained
> >> un-committed of my NUMA placement series.
> >
> > Whilst I don't think this should prevent the merging of this series now,
> > I think there needs to be some sort of unit tests for the placement
> > algorithm before the 4.2 release.
> >
> > I think the tests should test a representative sample of common system
> > configurations, available memory and VM memory requirements.  I'd
> > suggest you'd be looking at 100s of test cases here for reasonable coverage.
> >
> > One method would be to start with various 'empty' systems and pile as
> > many differently sized VMs as will fit.  You may want both fixed test of
> > reproducible tests and random ones.
> 
> That sounds good. Though I don't have much automated testing experience 
> with Xen I could chime in with two things:
> 
> 1. If we focus on placement only, I have good experience with 
> ttylinux.iso. Those live distros can be killed easily at any time and 
> you just need one instance of the .iso file on the disk.
> 2. # xl vcpu-list | sed -e 1d | sort -n -k 7 | tr -s \  | cut -d\  -f7 | 
> uniq -c
> This gives the number of VCPUs per node (sort of ;-)

Ideally you wouldn't need a xen system at all for this, you just want a
database of input configurations (host NUMA setup, existing guest
layout) and hypothetical new guests and their mapping to the expected
output. You can then feed these offline to the algorithmn and validate
that the output is the expected one.

> > Some means of automatically verifying the quality of the result at each
> > stage would be essential.
> 
> This "automatically verifying the quality of the result" doesn't sound 
> trivial. If we knew this exactly, we could just code this into the 
> algorithm, right?

Well, just knowing something has changed is useful, even you can't
decide in an automated way if it is the before or after case which is
the good/best one.

Even if we just blindly store the current result today as the expected
one then when someone makes a tweak we can manually inspect the
differences in the output and say "yes, that seems sane" or "no, that's
mad". The need to do this will be encoded in the diff which makes the
change (since you'd have to patch the test suite...)

> Also it seems to depend on the setup. Maybe we just collect some test 
> output first and then try to come up with quality algorithms?
> 
> Regards,
> Andre.
> 
>