From mboxrd@z Thu Jan  1 00:00:00 1970
From: Ian Campbell <Ian.Campbell@citrix.com>
Subject: Re: [PATCH v4 21/21] xl: vNUMA support
Date: Wed, 28 Jan 2015 16:46:05 +0000
Message-ID: <1422463565.5187.68.camel@citrix.com>
References: <1422011632-22018-1-git-send-email-wei.liu2@citrix.com>
	<1422011632-22018-22-git-send-email-wei.liu2@citrix.com>
Mime-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Return-path: <xen-devel-bounces@lists.xen.org>
In-Reply-To: <1422011632-22018-22-git-send-email-wei.liu2@citrix.com>
List-Unsubscribe: <http://lists.xen.org/cgi-bin/mailman/options/xen-devel>,
	<mailto:xen-devel-request@lists.xen.org?subject=unsubscribe>
List-Post: <mailto:xen-devel@lists.xen.org>
List-Help: <mailto:xen-devel-request@lists.xen.org?subject=help>
List-Subscribe: <http://lists.xen.org/cgi-bin/mailman/listinfo/xen-devel>,
	<mailto:xen-devel-request@lists.xen.org?subject=subscribe>
Sender: xen-devel-bounces@lists.xen.org
Errors-To: xen-devel-bounces@lists.xen.org
To: Wei Liu <wei.liu2@citrix.com>
Cc: ufimtseva@gmail.com, andrew.cooper3@citrix.com, dario.faggioli@citrix.com, ian.jackson@eu.citrix.com, xen-devel@lists.xen.org, JBeulich@suse.com
List-Id: xen-devel@lists.xenproject.org

On Fri, 2015-01-23 at 11:13 +0000, Wei Liu wrote:
> This patch includes configuration options parser and documentation.
> 
> Please find the hunk to xl.cfg.pod.5 for more information.
> 
> Signed-off-by: Wei Liu <wei.liu2@citrix.com>
> Cc: Ian Campbell <ian.campbell@citrix.com>
> Cc: Ian Jackson <ian.jackson@eu.citrix.com>
> Cc: Dario Faggioli <dario.faggioli@citrix.com>
> Cc: Elena Ufimtseva <ufimtseva@gmail.com>
> ---
> Changes in v2:
> 1. Make vnuma_vdistances mandatory.
> 2. Use nested list to specify vdistances.
> 3. Update documentation.
> ---
>  docs/man/xl.cfg.pod.5    |   39 ++++++++++++
>  tools/libxl/xl_cmdimpl.c |  147 ++++++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 186 insertions(+)
> 
> diff --git a/docs/man/xl.cfg.pod.5 b/docs/man/xl.cfg.pod.5
> index e2f91fc..b23bd6f 100644
> --- a/docs/man/xl.cfg.pod.5
> +++ b/docs/man/xl.cfg.pod.5
> @@ -266,6 +266,45 @@ it will crash.
>  
>  =back
>  
> +=head3 Virtual NUMA Memory Allocation
> +
> +=over 4
> +
> +=item B<vnuma_memory=[ NUMBER, NUMBER, ... ]>
> +
> +Specify the size of memory covered by each virtual NUMA node. The number of
> +elements in the list also implicitly defines the number of virtual NUMA nodes.
> +
> +The sum of all elements in this list should be equal to memory size specified
> +by B<maxmem=> in guest configuration file, or B<memory=> if B<maxmem=> is not
> +specified.

Could we make it permissible to omit memory= and have the toolstack do
the maths for us?

I suppose == maxmem is due to no vnuma vs. PoD? What about for PV
guests, is preballooning allowed there too?

> +
> +=item B<vnuma_vcpu_map=[ NUMBER, NUMBER, ... ]>
> +
> +Specifiy which virutal NUMA node a specific vcpu belongs to. The number of

"Specify" and "virtual".

> +elements in this list should be equal to B<maxvcpus=> in guest configuration
> +file, or B<vcpus=> if B<maxvcpus=> is not specified.

Again, can we relieve the user of these tedious sums?
> +=item B<vnuma_pnode_map=[ NUMBER, NUMBER, ... ]>
> +
> +Specifiy which physical NUMA node a specific virtual NUMA node maps to. The

"Specify" again.

> +number of elements in this list should be equal to the number of virtual
> +NUMA nodes defined in B<vnuma_memory=>.

Would it make sense to instead have a single array or e.g. "NODE:SIZE"
or something?

> +
> +=item B<vnuma_vdistance=[ [NUMBER, ..., NUMBER], [NUMBER, ..., NUMBER], ... ]>
> +
> +Two dimensional list to specify distances among nodes.
> +
> +The number of elements in the first dimension list equals the number of virtual
> +nodes. Each element in position B<i> is a list that specifies the distances
> +from node B<i> to other nodes.
> +
> +For example, for a guest with 2 virtual nodes, user can specify:
> +
> +  vnuma_vdistance = [ [10, 20], [20, 10] ]

Any guidance on how a user should choose these numbers?

Do we support a mode where something figures this out based on the
underlying distances between the pnode to which a vnode is assigned?

Would a user ever want/need to override such a mapping?

Ian.