Re: [PATCH v4 21/21] xl: vNUMA support

[Ian Campbell <Ian.Campbell@citrix.com>]

On Fri, 2015-01-23 at 11:13 +0000, Wei Liu wrote:
> This patch includes configuration options parser and documentation.
> 
> Please find the hunk to xl.cfg.pod.5 for more information.
> 
> Signed-off-by: Wei Liu <wei.liu2@citrix.com>
> Cc: Ian Campbell <ian.campbell@citrix.com>
> Cc: Ian Jackson <ian.jackson@eu.citrix.com>
> Cc: Dario Faggioli <dario.faggioli@citrix.com>
> Cc: Elena Ufimtseva <ufimtseva@gmail.com>
> ---
> Changes in v2:
> 1. Make vnuma_vdistances mandatory.
> 2. Use nested list to specify vdistances.
> 3. Update documentation.
> ---
>  docs/man/xl.cfg.pod.5    |   39 ++++++++++++
>  tools/libxl/xl_cmdimpl.c |  147 ++++++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 186 insertions(+)
> 
> diff --git a/docs/man/xl.cfg.pod.5 b/docs/man/xl.cfg.pod.5
> index e2f91fc..b23bd6f 100644
> --- a/docs/man/xl.cfg.pod.5
> +++ b/docs/man/xl.cfg.pod.5
> @@ -266,6 +266,45 @@ it will crash.
>  
>  =back
>  
> +=head3 Virtual NUMA Memory Allocation
> +
> +=over 4
> +
> +=item B<vnuma_memory=[ NUMBER, NUMBER, ... ]>
> +
> +Specify the size of memory covered by each virtual NUMA node. The number of
> +elements in the list also implicitly defines the number of virtual NUMA nodes.
> +
> +The sum of all elements in this list should be equal to memory size specified
> +by B<maxmem=> in guest configuration file, or B<memory=> if B<maxmem=> is not
> +specified.

Could we make it permissible to omit memory= and have the toolstack do
the maths for us?

I suppose == maxmem is due to no vnuma vs. PoD? What about for PV
guests, is preballooning allowed there too?

> +
> +=item B<vnuma_vcpu_map=[ NUMBER, NUMBER, ... ]>
> +
> +Specifiy which virutal NUMA node a specific vcpu belongs to. The number of

"Specify" and "virtual".

> +elements in this list should be equal to B<maxvcpus=> in guest configuration
> +file, or B<vcpus=> if B<maxvcpus=> is not specified.

Again, can we relieve the user of these tedious sums?
> +=item B<vnuma_pnode_map=[ NUMBER, NUMBER, ... ]>
> +
> +Specifiy which physical NUMA node a specific virtual NUMA node maps to. The

"Specify" again.

> +number of elements in this list should be equal to the number of virtual
> +NUMA nodes defined in B<vnuma_memory=>.

Would it make sense to instead have a single array or e.g. "NODE:SIZE"
or something?

> +
> +=item B<vnuma_vdistance=[ [NUMBER, ..., NUMBER], [NUMBER, ..., NUMBER], ... ]>
> +
> +Two dimensional list to specify distances among nodes.
> +
> +The number of elements in the first dimension list equals the number of virtual
> +nodes. Each element in position B<i> is a list that specifies the distances
> +from node B<i> to other nodes.
> +
> +For example, for a guest with 2 virtual nodes, user can specify:
> +
> +  vnuma_vdistance = [ [10, 20], [20, 10] ]

Any guidance on how a user should choose these numbers?

Do we support a mode where something figures this out based on the
underlying distances between the pnode to which a vnode is assigned?

Would a user ever want/need to override such a mapping?

Ian.