Re: [PATCH 0 of 3 v5/leftover] Automatic NUMA placement for xl

From: David Vrabel <dvrabel@cantab.net>
To: Andre Przywara <andre.przywara@amd.com>
Cc: Ian Campbell <Ian.Campbell@citrix.com>,
	Stefano Stabellini <Stefano.Stabellini@eu.citrix.com>,
	George Dunlap <george.dunlap@eu.citrix.com>,
	Juergen Gross <juergen.gross@ts.fujitsu.com>,
	Ian Jackson <Ian.Jackson@eu.citrix.com>,
	xen-devel <xen-devel@lists.xen.org>,
	Dario Faggioli <raistlin@linux.it>
Subject: Re: [PATCH 0 of 3 v5/leftover] Automatic NUMA placement for	xl
Date: Fri, 20 Jul 2012 13:14:03 +0100	[thread overview]
Message-ID: <50094B8B.2010300@cantab.net> (raw)
In-Reply-To: <5009446E.3000900@amd.com>

On 20/07/12 12:43, Andre Przywara wrote:
> On 07/20/2012 01:07 PM, David Vrabel wrote:
>> On 16/07/12 18:13, Dario Faggioli wrote:
>>> Hello again,
>>>
>>> This is a new version (fixing a small bug) of this series:
>>> <patchbomb.1341932613@Solace>, which in turn was a repost of what
>>> remained
>>> un-committed of my NUMA placement series.
>>
>> Whilst I don't think this should prevent the merging of this series now,
>> I think there needs to be some sort of unit tests for the placement
>> algorithm before the 4.2 release.
>>
>> I think the tests should test a representative sample of common system
>> configurations, available memory and VM memory requirements.  I'd
>> suggest you'd be looking at 100s of test cases here for reasonable
>> coverage.
>>
>> One method would be to start with various 'empty' systems and pile as
>> many differently sized VMs as will fit.  You may want both fixed test of
>> reproducible tests and random ones.
> 
> That sounds good. Though I don't have much automated testing experience
> with Xen I could chime in with two things:
> 
> 1. If we focus on placement only, I have good experience with
> ttylinux.iso. Those live distros can be killed easily at any time and
> you just need one instance of the .iso file on the disk.
> 2. # xl vcpu-list | sed -e 1d | sort -n -k 7 | tr -s \  | cut -d\  -f7 |
> uniq -c
> This gives the number of VCPUs per node (sort of ;-)

I'm talking about unit tests of the algorithm not system tests on real
hardware.  i.e., some sort of wrapper around the C functions that call
then with various sets of input data.

>> Some means of automatically verifying the quality of the result at each
>> stage would be essential.
> 
> This "automatically verifying the quality of the result" doesn't sound
> trivial. If we knew this exactly, we could just code this into the
> algorithm, right?

I think it is much easier to verify the result than generate the
solution.  The quality score is basically the percentage of memory that
ended up on the expected number of nodes -- this is easy to calculate.

If the algorithm is tweaked and the resulting quality score takes a nose
dive this would give a very quick indication that the tweak may be
broken.  Conversely, if the quality goes up, the tweak is more likely to
be good.

David