[PATCH v12 0/5] Implement vcpu soft affinity for credit1 (toolstack side, leftover)

[PATCH v12 0/5] Implement vcpu soft affinity for credit1 (toolstack side, leftover)

[Dario Faggioli]

Here it comes another round! :-)

The only relevant comment on v11 was to keep the parsing and handling of the
string and list variants (for specifying affinities) separated, instead of
having them sharing the code. This is what this new version of the series does.

Ack-s are missing from patches 2 and 4.

A branch is available here:
 git://xenbits.xen.org/people/dariof/xen.git  numa/per-vcpu-affinity-v12
 http://xenbits.xen.org/gitweb/?p=people/dariof/xen.git;a=shortlog;h=refs/heads/numa/per-vcpu-affinity-v12

Thanks and Regards,
Dario

---

Dario Faggioli (5):
      xl: enable getting and setting soft affinity
      xl: move away from the use of cpumap for hard affinity
      xl: move the vcpu affinity parsing in a function
      libxl/xl: make it possible to specify soft-affinity in domain config file
      libxl: automatic NUMA placement affects soft affinity


 docs/man/xl.cfg.pod.5                |   54 ++++++--
 docs/man/xl.pod.1                    |   32 ++++-
 docs/misc/xl-numa-placement.markdown |   14 ++
 tools/libxl/libxl.h                  |    3 
 tools/libxl/libxl_create.c           |   15 ++
 tools/libxl/libxl_dom.c              |   69 ++++++++--
 tools/libxl/libxl_types.idl          |    1 
 tools/libxl/xl_cmdimpl.c             |  231 +++++++++++++++++++++++-----------
 tools/libxl/xl_cmdtable.c            |    2 
 9 files changed, 302 insertions(+), 119 deletions(-)

--
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

[PATCH v12 1/5] xl: enable getting and setting soft affinity

[Dario Faggioli]

Getting happens via `xl vcpu-list', which now looks like this:

 # xl vcpu-list -s
 Name       ID VCPU CPU State Time(s) Affinity (Hard / Soft)
 Domain-0   0     0  11  -b-     5.4  8-15 / all
 Domain-0   0     1  11  -b-     1.0  8-15 / all
 Domain-0   0    14  13  -b-     1.4  8-15 / all
 Domain-0   0    15   8  -b-     1.6  8-15 / all
 vm-test    3     0   4  -b-     2.5  0-12 / 0-7
 vm-test    3     1   0  -b-     3.2  0-12 / 0-7

Setting happens by specifying two pCPU masks to the `xl vcpu-pin'
command, the first one will be hard affinity, the second soft
affinity. If only one mask is specified, it is only hard affinity
that is affected. To change only soft affinity, '-' can be used
as the hard affinity mask parameter, and it will be left alone.

xl manual page is updated accordingly.

Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
Changes from v10:
 * strdup() inside vcpupin_parse() instead than outside. This
   is happening in the next patch in the series anyway, so do
   it here, which also fixes a compilation error (wiredly
   enough only) on some version of gcc, as reported during
   review.

Changes from v7:
 * 'const char *' used instead of #define/#undef for string
   constants, as requested during review;
 * more consistently use the 'hard' and 'soft' pointers rather
   than the actual cpumap variables, as suggested during review;
 * bitmap holding hard affinity renamed from 'cpumap' to
   'cpumap_hard'.

Changes from v6:
 * removed a remnant from previous iteration, when `vcpu-pin'
   was tacking a porameter;
 * tried to simplify parsing inside of `xl vcpu-pin', as suggested
   during review. Could not take the exact approach described in the
   emails, but still did my best for making this more clear and easy
   to read;
 * fixed a few typos and leftovers.

Changes from v5:
 * change command line interface for 'vcpu-pin', as suggested during
   review.

Changes from v4:
 * fix and rephrased the manual entry, as suggested during review;
 * more refactoring to remove some leftover special casing, as
   suggested during review.

Changes from v3:
 * fix typos in doc, rephrased the help message and changed
   the title of the column for hard/soft affinity, as suggested
   during review.

Changes from v2:
 * this patch folds what in v2 were patches 13 and 14;
 * `xl vcpu-pin' always shows both had and soft affinity,
   without the need of passing '-s'.
---
 docs/man/xl.pod.1         |   32 ++++++++++---
 tools/libxl/xl_cmdimpl.c  |  111 +++++++++++++++++++++++++++++----------------
 tools/libxl/xl_cmdtable.c |    2 -
 3 files changed, 96 insertions(+), 49 deletions(-)

diff --git a/docs/man/xl.pod.1 b/docs/man/xl.pod.1
index 30bd4bf..9d1c2a5 100644
--- a/docs/man/xl.pod.1
+++ b/docs/man/xl.pod.1
@@ -651,16 +651,32 @@ after B<vcpu-set>, go to B<SEE ALSO> section for information.
 Lists VCPU information for a specific domain.  If no domain is
 specified, VCPU information for all domains will be provided.
 
-=item B<vcpu-pin> I<domain-id> I<vcpu> I<cpus>
+=item B<vcpu-pin> I<domain-id> I<vcpu> I<cpus hard> I<cpus soft>
 
-Pins the VCPU to only run on the specific CPUs.  The keyword
-B<all> can be used to apply the I<cpus> list to all VCPUs in the
-domain.
+Set hard and soft affinity for a I<vcpu> of <domain-id>. Normally VCPUs
+can float between available CPUs whenever Xen deems a different run state
+is appropriate.
 
-Normally VCPUs can float between available CPUs whenever Xen deems a
-different run state is appropriate.  Pinning can be used to restrict
-this, by ensuring certain VCPUs can only run on certain physical
-CPUs.
+Hard affinity can be used to restrict this, by ensuring certain VCPUs
+can only run on certain physical CPUs. Soft affinity specifies a I<preferred>
+set of CPUs. Soft affinity needs special support in the scheduler, which is
+only provided in credit1.
+
+The keyword B<all> can be used to apply the hard and soft affinity masks to
+all the VCPUs in the domain. The symbol '-' can be used to leave either
+hard or soft affinity alone.
+
+For example:
+
+ xl vcpu-pin 0 3 - 6-9
+
+will set soft affinity for vCPU 3 of domain 0 to pCPUs 6,7,8 and 9,
+leaving its hard affinity untouched. On the othe hand:
+
+ xl vcpu-pin 0 3 3,4 6-9
+
+will set both hard and soft affinity, the former to pCPUs 3 and 4, the
+latter to pCPUs 6,7,8, and 9.
 
 =item B<vm-list>
 
diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
index 68df548..b32345b 100644
--- a/tools/libxl/xl_cmdimpl.c
+++ b/tools/libxl/xl_cmdimpl.c
@@ -653,17 +653,18 @@ static int update_cpumap_range(const char *str, libxl_bitmap *cpumap)
  * single cpus or as eintire NUMA nodes) and turns it into the
  * corresponding libxl_bitmap (in cpumap).
  */
-static int vcpupin_parse(char *cpu, libxl_bitmap *cpumap)
+static int vcpupin_parse(const char *cpu, libxl_bitmap *cpumap)
 {
-    char *ptr, *saveptr = NULL;
+    char *ptr, *saveptr = NULL, *buf = strdup(cpu);
     int rc = 0;
 
-    for (ptr = strtok_r(cpu, ",", &saveptr); ptr;
+    for (ptr = strtok_r(buf, ",", &saveptr); ptr;
          ptr = strtok_r(NULL, ",", &saveptr)) {
         rc = update_cpumap_range(ptr, cpumap);
         if (rc)
             break;
     }
+    free(buf);
 
     return rc;
 }
@@ -826,17 +827,14 @@ static void parse_config_data(const char *config_source,
         libxl_defbool_set(&b_info->numa_placement, false);
     }
     else if (!xlu_cfg_get_string (config, "cpus", &buf, 0)) {
-        char *buf2 = strdup(buf);
-
         if (libxl_cpu_bitmap_alloc(ctx, &b_info->cpumap, 0)) {
             fprintf(stderr, "Unable to allocate cpumap\n");
             exit(1);
         }
 
         libxl_bitmap_set_none(&b_info->cpumap);
-        if (vcpupin_parse(buf2, &b_info->cpumap))
+        if (vcpupin_parse(buf, &b_info->cpumap))
             exit(1);
-        free(buf2);
 
         libxl_defbool_set(&b_info->numa_placement, false);
     }
@@ -4480,8 +4478,10 @@ static void print_vcpuinfo(uint32_t tdomid,
     }
     /*      TIM */
     printf("%9.1f  ", ((float)vcpuinfo->vcpu_time / 1e9));
-    /* CPU AFFINITY */
+    /* CPU HARD AND SOFT AFFINITY */
     print_bitmap(vcpuinfo->cpumap.map, nr_cpus, stdout);
+    printf(" / ");
+    print_bitmap(vcpuinfo->cpumap_soft.map, nr_cpus, stdout);
     printf("\n");
 }
 
@@ -4516,7 +4516,8 @@ static void vcpulist(int argc, char **argv)
     }
 
     printf("%-32s %5s %5s %5s %5s %9s %s\n",
-           "Name", "ID", "VCPU", "CPU", "State", "Time(s)", "CPU Affinity");
+           "Name", "ID", "VCPU", "CPU", "State", "Time(s)",
+           "Affinity (Hard / Soft)");
     if (!argc) {
         if (!(dominfo = libxl_list_domain(ctx, &nb_domain))) {
             fprintf(stderr, "libxl_list_domain failed.\n");
@@ -4549,17 +4550,29 @@ int main_vcpulist(int argc, char **argv)
     return 0;
 }
 
-static int vcpupin(uint32_t domid, const char *vcpu, char *cpu)
+int main_vcpupin(int argc, char **argv)
 {
     libxl_vcpuinfo *vcpuinfo;
-    libxl_bitmap cpumap;
-
-    uint32_t vcpuid;
+    libxl_bitmap cpumap_hard, cpumap_soft;;
+    libxl_bitmap *soft = &cpumap_soft, *hard = &cpumap_hard;
+    uint32_t vcpuid, domid;
+    const char *vcpu, *hard_str, *soft_str;
     char *endptr;
-    int i, nb_cpu, nb_vcpu, rc = -1;
+    int opt, nb_cpu, nb_vcpu, rc = -1;
 
-    libxl_bitmap_init(&cpumap);
+    libxl_bitmap_init(&cpumap_hard);
+    libxl_bitmap_init(&cpumap_soft);
+
+    SWITCH_FOREACH_OPT(opt, "", NULL, "vcpu-pin", 3) {
+        /* No options */
+    }
 
+    domid = find_domain(argv[optind]);
+    vcpu = argv[optind+1];
+    hard_str = argv[optind+2];
+    soft_str = (argc > optind+3) ? argv[optind+3] : NULL;
+
+    /* Figure out with which vCPU we are dealing with */
     vcpuid = strtoul(vcpu, &endptr, 10);
     if (vcpu == endptr) {
         if (strcmp(vcpu, "all")) {
@@ -4569,10 +4582,35 @@ static int vcpupin(uint32_t domid, const char *vcpu, char *cpu)
         vcpuid = -1;
     }
 
-    if (libxl_cpu_bitmap_alloc(ctx, &cpumap, 0))
+    if (libxl_cpu_bitmap_alloc(ctx, &cpumap_hard, 0) ||
+        libxl_cpu_bitmap_alloc(ctx, &cpumap_soft, 0))
         goto out;
 
-    if (vcpupin_parse(cpu, &cpumap))
+    /*
+     * Syntax is: xl vcpu-pin <domid> <vcpu> <hard> <soft>
+     * We want to handle all the following cases ('-' means
+     * "leave it alone"):
+     *  xl vcpu-pin 0 3 3,4
+     *  xl vcpu-pin 0 3 3,4 -
+     *  xl vcpu-pin 0 3 - 6-9
+     *  xl vcpu-pin 0 3 3,4 6-9
+     */
+
+    /*
+     * Hard affinity is always present. However, if it's "-", all we need
+     * is passing a NULL pointer to the libxl_set_vcpuaffinity() call below.
+     */
+    if (!strcmp(hard_str, "-"))
+        hard = NULL;
+    else if (vcpupin_parse(hard_str, hard))
+        goto out;
+    /*
+     * Soft affinity is handled similarly. Only difference: we also want
+     * to pass NULL to libxl_set_vcpuaffinity() if it is not specified.
+     */
+    if (argc <= optind+3 || !strcmp(soft_str, "-"))
+        soft = NULL;
+    else if (vcpupin_parse(soft_str, soft))
         goto out;
 
     if (dryrun_only) {
@@ -4583,7 +4621,14 @@ static int vcpupin(uint32_t domid, const char *vcpu, char *cpu)
         }
 
         fprintf(stdout, "cpumap: ");
-        print_bitmap(cpumap.map, nb_cpu, stdout);
+        if (hard)
+            print_bitmap(hard->map, nb_cpu, stdout);
+        else
+            fprintf(stdout, "-");
+        if (soft) {
+            fprintf(stdout, " ");
+            print_bitmap(soft->map, nb_cpu, stdout);
+        }
         fprintf(stdout, "\n");
 
         if (ferror(stdout) || fflush(stdout)) {
@@ -4596,43 +4641,29 @@ static int vcpupin(uint32_t domid, const char *vcpu, char *cpu)
     }
 
     if (vcpuid != -1) {
-        if (libxl_set_vcpuaffinity(ctx, domid, vcpuid, &cpumap, NULL)) {
-            fprintf(stderr, "Could not set affinity for vcpu `%u'.\n", vcpuid);
+        if (libxl_set_vcpuaffinity(ctx, domid, vcpuid, hard, soft)) {
+            fprintf(stderr, "Could not set affinity for vcpu `%u'.\n",
+                    vcpuid);
             goto out;
         }
     }
     else {
-        if (!(vcpuinfo = libxl_list_vcpu(ctx, domid, &nb_vcpu, &i))) {
+        if (!(vcpuinfo = libxl_list_vcpu(ctx, domid, &nb_vcpu, &nb_cpu))) {
             fprintf(stderr, "libxl_list_vcpu failed.\n");
             goto out;
         }
-        for (i = 0; i < nb_vcpu; i++) {
-            if (libxl_set_vcpuaffinity(ctx, domid, vcpuinfo[i].vcpuid,
-                                       &cpumap, NULL)) {
-                fprintf(stderr, "libxl_set_vcpuaffinity failed"
-                                " on vcpu `%u'.\n", vcpuinfo[i].vcpuid);
-            }
-        }
+        if (libxl_set_vcpuaffinity_all(ctx, domid, nb_vcpu, hard, soft))
+            fprintf(stderr, "Could not set affinity.\n");
         libxl_vcpuinfo_list_free(vcpuinfo, nb_vcpu);
     }
 
     rc = 0;
  out:
-    libxl_bitmap_dispose(&cpumap);
+    libxl_bitmap_dispose(&cpumap_soft);
+    libxl_bitmap_dispose(&cpumap_hard);
     return rc;
 }
 
-int main_vcpupin(int argc, char **argv)
-{
-    int opt;
-
-    SWITCH_FOREACH_OPT(opt, "", NULL, "vcpu-pin", 3) {
-        /* No options */
-    }
-
-    return vcpupin(find_domain(argv[optind]), argv[optind+1] , argv[optind+2]);
-}
-
 static void vcpuset(uint32_t domid, const char* nr_vcpus, int check_host)
 {
     char *endptr;
diff --git a/tools/libxl/xl_cmdtable.c b/tools/libxl/xl_cmdtable.c
index 4279b9f..7b7fa92 100644
--- a/tools/libxl/xl_cmdtable.c
+++ b/tools/libxl/xl_cmdtable.c
@@ -218,7 +218,7 @@ struct cmd_spec cmd_table[] = {
     { "vcpu-pin",
       &main_vcpupin, 1, 1,
       "Set which CPUs a VCPU can use",
-      "<Domain> <VCPU|all> <CPUs|all>",
+      "<Domain> <VCPU|all> <Hard affinity|-|all> <Soft affinity|-|all>",
     },
     { "vcpu-set",
       &main_vcpuset, 0, 1,

[PATCH v12 2/5] xl: move away from the use of cpumap for hard affinity

[Dario Faggioli]

and start using the vcpu_hard_affinity array instead. This is done
as when, in a subsequent patch ("libxl/xl: make it possible to
specify soft-affinity in domain config file") we will become able
to deal with soft affinity, code can be shared.

This change also enables more advanced VCPU to PCPU (hard, for now)
affinity specification, in case a list is used, like:

      cpus = ["3-4", "2-6,^4"]

What it means is that VCPU 0 must be pinned to PCPU 3,4 and VCPU 1
to PCPUs 2,3,5,6 (before this change, cpus=[xx, yy] only supported
single values). Of course, the old (e.g., cpus=[2, 3]) syntax
continues to work.

Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
---
Changes from v11:
  * improved manpage, as requested during review;
  * do no unify the handling of string and list any longer, as
    requested during review.

Changes from v10:
  * changed the logic that checks whether we are dealing with a
    string or a list a bit. Basically, added a bool flag to store
    that, and this killed the need of having buf2 which on it
    turn needed to be 'spuriously' initialized on gcc >= 4.9.0.

Changes from v9:
 * new patch, basically containing the xl bits of what was the
   cpumap deprecation patch in v9.
---
 docs/man/xl.cfg.pod.5    |   12 ++++++++----
 tools/libxl/xl_cmdimpl.c |   31 +++++++++++++++++++++++--------
 2 files changed, 31 insertions(+), 12 deletions(-)

diff --git a/docs/man/xl.cfg.pod.5 b/docs/man/xl.cfg.pod.5
index ff9ea77..ffd94a8 100644
--- a/docs/man/xl.cfg.pod.5
+++ b/docs/man/xl.cfg.pod.5
@@ -143,11 +143,15 @@ Combining this with "all" is also possible, meaning "all,^nodes:1"
 results in all the vcpus of the guest running on all the cpus on the
 host, except for the cpus belonging to the host NUMA node 1.
 
-=item ["2", "3"] (or [2, 3])
+=item ["2", "3-8,^5"]
 
-To ask for specific vcpu mapping. That means (in this example), vcpu #0
-of the guest will run on cpu #2 of the host and vcpu #1 of the guest will
-run on cpu #3 of the host.
+To ask for specific vcpu mapping. That means (in this example), vcpu 0
+of the guest will run on cpu 2 of the host and vcpu 1 of the guest will
+run on cpus 3,4,6,7,8 of the host.
+
+More complex notation can be also used, exactly as described above. So
+"all,^5-8", or just "all", or "node:0,node:2,^9-11,18-20" are all legal,
+for each element of the list.
 
 =back
 
diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
index b32345b..d481681 100644
--- a/tools/libxl/xl_cmdimpl.c
+++ b/tools/libxl/xl_cmdimpl.c
@@ -808,16 +808,15 @@ static void parse_config_data(const char *config_source,
         b_info->vcpu_hard_affinity = xmalloc(num_cpus * sizeof(libxl_bitmap));
 
         while ((buf = xlu_cfg_get_listitem(cpus, j)) != NULL && j < num_cpus) {
-            i = atoi(buf);
-
             libxl_bitmap_init(&b_info->vcpu_hard_affinity[j]);
             if (libxl_cpu_bitmap_alloc(ctx,
                                        &b_info->vcpu_hard_affinity[j], 0)) {
                 fprintf(stderr, "Unable to allocate cpumap for vcpu %d\n", j);
                 exit(1);
             }
-            libxl_bitmap_set_none(&b_info->vcpu_hard_affinity[j]);
-            libxl_bitmap_set(&b_info->vcpu_hard_affinity[j], i);
+
+            if (vcpupin_parse(buf, &b_info->vcpu_hard_affinity[j]))
+                exit(1);
 
             j++;
         }
@@ -827,15 +826,31 @@ static void parse_config_data(const char *config_source,
         libxl_defbool_set(&b_info->numa_placement, false);
     }
     else if (!xlu_cfg_get_string (config, "cpus", &buf, 0)) {
-        if (libxl_cpu_bitmap_alloc(ctx, &b_info->cpumap, 0)) {
-            fprintf(stderr, "Unable to allocate cpumap\n");
+        b_info->vcpu_hard_affinity =
+            xmalloc(b_info->max_vcpus * sizeof(libxl_bitmap));
+
+        libxl_bitmap_init(&b_info->vcpu_hard_affinity[0]);
+        if (libxl_cpu_bitmap_alloc(ctx,
+                                   &b_info->vcpu_hard_affinity[0], 0)) {
+            fprintf(stderr, "Unable to allocate cpumap for vcpu 0\n");
             exit(1);
         }
 
-        libxl_bitmap_set_none(&b_info->cpumap);
-        if (vcpupin_parse(buf, &b_info->cpumap))
+        if (vcpupin_parse(buf, &b_info->vcpu_hard_affinity[0]))
             exit(1);
 
+        for (i = 1; i < b_info->max_vcpus; i++) {
+            libxl_bitmap_init(&b_info->vcpu_hard_affinity[i]);
+            if (libxl_cpu_bitmap_alloc(ctx,
+                                       &b_info->vcpu_hard_affinity[i], 0)) {
+                fprintf(stderr, "Unable to allocate cpumap for vcpu %d\n", i);
+                exit(1);
+            }
+            libxl_bitmap_copy(ctx, &b_info->vcpu_hard_affinity[i],
+                              &b_info->vcpu_hard_affinity[0]);
+        }
+        b_info->num_vcpu_hard_affinity = num_cpus;
+
         libxl_defbool_set(&b_info->numa_placement, false);
     }

[PATCH v12 3/5] xl: move the vcpu affinity parsing in a function

[Dario Faggioli]

so that such parsing code can be used for both hard and soft
affinity, the support for which is introduced in the next
change.

This is pure code motion, no functional change intended.

Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
 tools/libxl/xl_cmdimpl.c |  127 +++++++++++++++++++++++++---------------------
 1 file changed, 70 insertions(+), 57 deletions(-)

diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
index d481681..6718f88 100644
--- a/tools/libxl/xl_cmdimpl.c
+++ b/tools/libxl/xl_cmdimpl.c
@@ -691,6 +691,72 @@ static void parse_top_level_sdl_options(XLU_Config *config,
     xlu_cfg_replace_string (config, "xauthority", &sdl->xauthority, 0);
 }
 
+static void parse_vcpu_affinity(XLU_Config *config,
+                                libxl_domain_build_info *b_info)
+{
+    XLU_ConfigList *cpus;
+    const char *buf;
+    int num_cpus;
+
+    if (!xlu_cfg_get_list (config, "cpus", &cpus, &num_cpus, 1)) {
+        int j = 0;
+
+        /* Silently ignore values corresponding to non existing vcpus */
+        if (num_cpus > b_info->max_vcpus)
+            num_cpus = b_info->max_vcpus;
+
+        b_info->vcpu_hard_affinity = xmalloc(num_cpus * sizeof(libxl_bitmap));
+
+        while ((buf = xlu_cfg_get_listitem(cpus, j)) != NULL && j < num_cpus) {
+            libxl_bitmap_init(&b_info->vcpu_hard_affinity[j]);
+            if (libxl_cpu_bitmap_alloc(ctx,
+                                       &b_info->vcpu_hard_affinity[j], 0)) {
+                fprintf(stderr, "Unable to allocate cpumap for vcpu %d\n", j);
+                exit(1);
+            }
+
+            if (vcpupin_parse(buf, &b_info->vcpu_hard_affinity[j]))
+                exit(1);
+
+            j++;
+        }
+        b_info->num_vcpu_hard_affinity = num_cpus;
+
+        /* We have a list of cpumaps, disable automatic placement */
+        libxl_defbool_set(&b_info->numa_placement, false);
+    }
+    else if (!xlu_cfg_get_string (config, "cpus", &buf, 0)) {
+        int i;
+
+        b_info->vcpu_hard_affinity =
+            xmalloc(b_info->max_vcpus * sizeof(libxl_bitmap));
+
+        libxl_bitmap_init(&b_info->vcpu_hard_affinity[0]);
+        if (libxl_cpu_bitmap_alloc(ctx,
+                                   &b_info->vcpu_hard_affinity[0], 0)) {
+            fprintf(stderr, "Unable to allocate cpumap for vcpu 0\n");
+            exit(1);
+        }
+
+        if (vcpupin_parse(buf, &b_info->vcpu_hard_affinity[0]))
+            exit(1);
+
+        for (i = 1; i < b_info->max_vcpus; i++) {
+            libxl_bitmap_init(&b_info->vcpu_hard_affinity[i]);
+            if (libxl_cpu_bitmap_alloc(ctx,
+                                       &b_info->vcpu_hard_affinity[i], 0)) {
+                fprintf(stderr, "Unable to allocate cpumap for vcpu %d\n", i);
+                exit(1);
+            }
+            libxl_bitmap_copy(ctx, &b_info->vcpu_hard_affinity[i],
+                              &b_info->vcpu_hard_affinity[0]);
+        }
+        b_info->num_vcpu_hard_affinity = num_cpus;
+
+        libxl_defbool_set(&b_info->numa_placement, false);
+    }
+}
+
 static void parse_config_data(const char *config_source,
                               const char *config_data,
                               int config_len,
@@ -699,9 +765,9 @@ static void parse_config_data(const char *config_source,
     const char *buf;
     long l;
     XLU_Config *config;
-    XLU_ConfigList *cpus, *vbds, *nics, *pcis, *cvfbs, *cpuids, *vtpms;
+    XLU_ConfigList *vbds, *nics, *pcis, *cvfbs, *cpuids, *vtpms;
     XLU_ConfigList *ioports, *irqs, *iomem;
-    int num_ioports, num_irqs, num_iomem, num_cpus;
+    int num_ioports, num_irqs, num_iomem;
     int pci_power_mgmt = 0;
     int pci_msitranslate = 0;
     int pci_permissive = 0;
@@ -798,61 +864,8 @@ static void parse_config_data(const char *config_source,
     if (!xlu_cfg_get_long (config, "maxvcpus", &l, 0))
         b_info->max_vcpus = l;
 
-    if (!xlu_cfg_get_list (config, "cpus", &cpus, &num_cpus, 1)) {
-        int j = 0;
-
-        /* Silently ignore values corresponding to non existing vcpus */
-        if (num_cpus > b_info->max_vcpus)
-            num_cpus = b_info->max_vcpus;
-
-        b_info->vcpu_hard_affinity = xmalloc(num_cpus * sizeof(libxl_bitmap));
-
-        while ((buf = xlu_cfg_get_listitem(cpus, j)) != NULL && j < num_cpus) {
-            libxl_bitmap_init(&b_info->vcpu_hard_affinity[j]);
-            if (libxl_cpu_bitmap_alloc(ctx,
-                                       &b_info->vcpu_hard_affinity[j], 0)) {
-                fprintf(stderr, "Unable to allocate cpumap for vcpu %d\n", j);
-                exit(1);
-            }
-
-            if (vcpupin_parse(buf, &b_info->vcpu_hard_affinity[j]))
-                exit(1);
-
-            j++;
-        }
-        b_info->num_vcpu_hard_affinity = num_cpus;
-
-        /* We have a list of cpumaps, disable automatic placement */
-        libxl_defbool_set(&b_info->numa_placement, false);
-    }
-    else if (!xlu_cfg_get_string (config, "cpus", &buf, 0)) {
-        b_info->vcpu_hard_affinity =
-            xmalloc(b_info->max_vcpus * sizeof(libxl_bitmap));
-
-        libxl_bitmap_init(&b_info->vcpu_hard_affinity[0]);
-        if (libxl_cpu_bitmap_alloc(ctx,
-                                   &b_info->vcpu_hard_affinity[0], 0)) {
-            fprintf(stderr, "Unable to allocate cpumap for vcpu 0\n");
-            exit(1);
-        }
-
-        if (vcpupin_parse(buf, &b_info->vcpu_hard_affinity[0]))
-            exit(1);
-
-        for (i = 1; i < b_info->max_vcpus; i++) {
-            libxl_bitmap_init(&b_info->vcpu_hard_affinity[i]);
-            if (libxl_cpu_bitmap_alloc(ctx,
-                                       &b_info->vcpu_hard_affinity[i], 0)) {
-                fprintf(stderr, "Unable to allocate cpumap for vcpu %d\n", i);
-                exit(1);
-            }
-            libxl_bitmap_copy(ctx, &b_info->vcpu_hard_affinity[i],
-                              &b_info->vcpu_hard_affinity[0]);
-        }
-        b_info->num_vcpu_hard_affinity = num_cpus;
-
-        libxl_defbool_set(&b_info->numa_placement, false);
-    }
+    /* Figure out VCPU hard-affinity ("cpus" config option) */
+    parse_vcpu_affinity(config, b_info);
 
     if (!xlu_cfg_get_long (config, "memory", &l, 0)) {
         b_info->max_memkb = l * 1024;

[PATCH v12 4/5] libxl/xl: make it possible to specify soft-affinity in domain config file

[Dario Faggioli]

To do so, we add the vcpu_soft_affinity array to build_info, and
treat it much like vcpu_hard_affinity. The new config option is
called "cpus_soft".

Note that the vcpu_hard_affinity array, introduced in a previous
patch, and the vcpu_soft_affinity array, introduced here, share
the same LIBXL_HAVE_xxx macro, in libxl.h. That is called
LIBXL_HAVE_BUILDINFO_VCPU_AFFINITY_ARRAYS, and was introduced
together with vcpu_hard_affinity, but only inside a comment.
In this change, we uncomment, and hence properly define it.

In order to avoid having to issue separate calls to
libxl_set_vcpuaffinity() (one for hard affinity and one for soft
affinity) in libxl__build_pre(), in case the caller uses
b_info->cpumap (for the former) and b_info->vcpu_soft_affinity (for
the latter), we also set (again!) a new default for b_info->cpumap.
This allows, from this change on, to always and only deal with
b_info->vcpu_hard_affinity all around libxl internals.

Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
---
Changes from v10:
 * default for b_info->cpumap changed again. Since we are not
   formally deprecating it, it is not just a corner case that one
   would want to specify hard affinity via cpumap, and soft affinity
   via the vcpu_soft_affinity array. That would require two distinct
   calls to libxl_set_vcpuaffinity(), which is something we rather
   avoid. To do so, if cpumap is used, convert it to vcpu_hard_affinity
   in *_build_info_setdefault() and then forget about it!
   This solution came up already as a possibility during v9's review,
   but at the time I thought it were not worth, as we were deprecating
   cpumap anyway. Since we're not, that is now the best solution IMO.

Changes from v9:
 * patch reworked again, due to changes in the preceding ones
   in the series. The structure is similar, it's still based
   on adding some indirection, so that the same code can be
   used to pars and enact both hard and soft affinity, but
   the code did change, I'm afraid.

Changes from v8:
 * fix a type in the LIBXL_HAVE_xxx macro name.

Changes from v7:
 * WARNING: this patch underwent quite a fundamental rework,
   given it's now building on top of Wei's "push vcpu affinity
   to libxl" patch. That's why I think it should be re-reviewed
   almost from scratch (sorry! :-P), and that's why I did not
   add IanC's ack, although he provided it to the v7 version
   of it.

Changes from v6:
 * update and improve the changelog.

Changes from v4:
 * fix typos and rephrase docs, as suggested during review;
 * more refactoring, i.e., more addressing factor of potential
   common code, as requested during review.

Changes from v3:
 * fix typos and language issues in docs and comments, as
   suggested during review;
 * common code to soft and hard affinity parsing factored
   together, as requested uring review.

Changes from v2:
 * use the new libxl API. Although the implementation changed
   only a little bit, I removed IanJ's Acked-by, although I am
   here saying that he did provided it, as requested.
---
 docs/man/xl.cfg.pod.5       |   23 ++++++++++--
 tools/libxl/libxl.h         |    3 +-
 tools/libxl/libxl_create.c  |   15 ++++++++
 tools/libxl/libxl_dom.c     |   33 ++++++++++++-----
 tools/libxl/libxl_types.idl |    1 +
 tools/libxl/xl_cmdimpl.c    |   84 ++++++++++++++++++++++++++-----------------
 6 files changed, 111 insertions(+), 48 deletions(-)

diff --git a/docs/man/xl.cfg.pod.5 b/docs/man/xl.cfg.pod.5
index ffd94a8..5833054 100644
--- a/docs/man/xl.cfg.pod.5
+++ b/docs/man/xl.cfg.pod.5
@@ -156,19 +156,36 @@ for each element of the list.
 =back
 
 If this option is not specified, no vcpu to cpu pinning is established,
-and the vcpus of the guest can run on all the cpus of the host.
+and the vcpus of the guest can run on all the cpus of the host. If this
+option is specified, the intersection of the vcpu pinning mask, provided
+here, and the soft affinity mask, provided via B<cpus\_soft=> (if any),
+is utilized to compute the domain node-affinity, for driving memory
+allocations.
 
 If we are on a NUMA machine (i.e., if the host has more than one NUMA
 node) and this option is not specified, libxl automatically tries to
 place the guest on the least possible number of nodes. That, however,
 will not affect vcpu pinning, so the guest will still be able to run on
-all the cpus, it will just prefer the ones from the node it has been
-placed on. A heuristic approach is used for choosing the best node (or
+all the cpus. A heuristic approach is used for choosing the best node (or
 set of nodes), with the goals of maximizing performance for the guest
 and, at the same time, achieving efficient utilization of host cpus
 and memory. See F<docs/misc/xl-numa-placement.markdown> for more
 details.
 
+=item B<cpus_soft="CPU-LIST">
+
+Exactly as B<cpus=>, but specifies soft affinity, rather than pinning
+(hard affinity). When using the credit scheduler, this means what cpus
+the vcpus of the domain prefer.
+
+A C<CPU-LIST> is specified exactly as above, for B<cpus=>.
+
+If this option is not specified, the vcpus of the guest will not have
+any preference regarding on what cpu to run. If this option is specified,
+the intersection of the soft affinity mask, provided here, and the vcpu
+pinning, provided via B<cpus=> (if any), is utilized to compute the
+domain node-affinity, for driving memory allocations.
+
 =back
 
 =head3 CPU Scheduling
diff --git a/tools/libxl/libxl.h b/tools/libxl/libxl.h
index 5ae6532..bfeb3bc 100644
--- a/tools/libxl/libxl.h
+++ b/tools/libxl/libxl.h
@@ -380,8 +380,7 @@ typedef struct libxl__ctx libxl_ctx;
  * Each bitmap should be big enough to accommodate the maximum number of
  * PCPUs of the host.
  */
-/* to be uncommented when soft array added */
-/* #define LIBXL_HAVE_BUILDINFO_VCPU_AFFINITY_ARRAYS 1 */
+#define LIBXL_HAVE_BUILDINFO_VCPU_AFFINITY_ARRAYS 1
 
 /*
  * LIBXL_HAVE_BUILDINFO_USBDEVICE_LIST
diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
index 0686f96..13992b4 100644
--- a/tools/libxl/libxl_create.c
+++ b/tools/libxl/libxl_create.c
@@ -187,6 +187,21 @@ int libxl__domain_build_info_setdefault(libxl__gc *gc,
     } else if (b_info->avail_vcpus.size > HVM_MAX_VCPUS)
         return ERROR_FAIL;
 
+    /* In libxl internals, we want to deal with vcpu_hard_affinity only! */
+    if (b_info->cpumap.size && !b_info->num_vcpu_hard_affinity) {
+        int i;
+
+        b_info->vcpu_hard_affinity = libxl__calloc(gc, b_info->max_vcpus,
+                                                   sizeof(libxl_bitmap));
+        for (i = 0; i < b_info->max_vcpus; i++) {
+            if (libxl_cpu_bitmap_alloc(CTX, &b_info->vcpu_hard_affinity[i], 0))
+                return ERROR_FAIL;
+            libxl_bitmap_copy(CTX, &b_info->vcpu_hard_affinity[i],
+                              &b_info->cpumap);
+        }
+        b_info->num_vcpu_hard_affinity = b_info->max_vcpus;
+    }
+
     libxl_defbool_setdefault(&b_info->numa_placement, true);
 
     if (b_info->max_memkb == LIBXL_MEMKB_DEFAULT)
diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c
index 83eb29a..cfbd13d 100644
--- a/tools/libxl/libxl_dom.c
+++ b/tools/libxl/libxl_dom.c
@@ -272,21 +272,36 @@ int libxl__build_pre(libxl__gc *gc, uint32_t domid,
     if (info->nodemap.size)
         libxl_domain_set_nodeaffinity(ctx, domid, &info->nodemap);
     /* As mentioned in libxl.h, vcpu_hard_array takes precedence */
-    if (info->num_vcpu_hard_affinity) {
-        int i;
+    if (info->num_vcpu_hard_affinity || info->num_vcpu_soft_affinity) {
+        libxl_bitmap *hard_affinity, *soft_affinity;
+        int i, n_vcpus;
+
+        n_vcpus = info->num_vcpu_hard_affinity > info->num_vcpu_soft_affinity ?
+            info->num_vcpu_hard_affinity : info->num_vcpu_soft_affinity;
+
+        for (i = 0; i < n_vcpus; i++) {
+            /*
+             * Prepare hard and soft affinity pointers in a way that allows
+             * us to issue only one call to libxl_set_vcpuaffinity(), setting,
+             * for each vcpu, both hard and soft affinity "atomically".
+             */
+            hard_affinity = NULL;
+            if (info->num_vcpu_hard_affinity &&
+                i < info->num_vcpu_hard_affinity)
+                hard_affinity = &info->vcpu_hard_affinity[i];
+
+            soft_affinity = NULL;
+            if (info->num_vcpu_soft_affinity &&
+                i < info->num_vcpu_soft_affinity)
+                soft_affinity = &info->vcpu_soft_affinity[i];
 
-        for (i = 0; i < info->num_vcpu_hard_affinity; i++) {
             if (libxl_set_vcpuaffinity(ctx, domid, i,
-                                       &info->vcpu_hard_affinity[i],
-                                       NULL)) {
+                                       hard_affinity, soft_affinity)) {
                 LOG(ERROR, "setting affinity failed on vcpu `%d'", i);
                 return ERROR_FAIL;
             }
         }
-    } else if (info->cpumap.size)
-        libxl_set_vcpuaffinity_all(ctx, domid, info->max_vcpus,
-                                   &info->cpumap, NULL);
-
+    }
 
     if (xc_domain_setmaxmem(ctx->xch, domid, info->target_memkb +
         LIBXL_MAXMEM_CONSTANT) < 0) {
diff --git a/tools/libxl/libxl_types.idl b/tools/libxl/libxl_types.idl
index a412f9c..0b3496f 100644
--- a/tools/libxl/libxl_types.idl
+++ b/tools/libxl/libxl_types.idl
@@ -319,6 +319,7 @@ libxl_domain_build_info = Struct("domain_build_info",[
     ("cpumap",          libxl_bitmap),
     ("nodemap",         libxl_bitmap),
     ("vcpu_hard_affinity", Array(libxl_bitmap, "num_vcpu_hard_affinity")),
+    ("vcpu_soft_affinity", Array(libxl_bitmap, "num_vcpu_soft_affinity")),
     ("numa_placement",  libxl_defbool),
     ("tsc_mode",        libxl_tsc_mode),
     ("max_memkb",       MemKB),
diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
index 6718f88..e593e0f 100644
--- a/tools/libxl/xl_cmdimpl.c
+++ b/tools/libxl/xl_cmdimpl.c
@@ -691,67 +691,76 @@ static void parse_top_level_sdl_options(XLU_Config *config,
     xlu_cfg_replace_string (config, "xauthority", &sdl->xauthority, 0);
 }
 
-static void parse_vcpu_affinity(XLU_Config *config,
-                                libxl_domain_build_info *b_info)
+static void parse_vcpu_affinity(libxl_domain_build_info *b_info,
+                                XLU_ConfigList *cpus, const char *buf,
+                                int num_cpus, bool is_hard)
 {
-    XLU_ConfigList *cpus;
-    const char *buf;
-    int num_cpus;
+    libxl_bitmap *vcpu_affinity_array;
 
-    if (!xlu_cfg_get_list (config, "cpus", &cpus, &num_cpus, 1)) {
-        int j = 0;
+    /*
+     * If we are here, and buf is !NULL, we're dealing with a string. What
+     * we do in this case is parse it, and copy the result in _all_ (up to
+     * b_info->max_vcpus) the elements of the vcpu affinity array.
+     *
+     * If buf is NULL, we have a list, and what we do is putting in the
+     * i-eth element of the vcpu affinity array the result of the parsing
+     * of the i-eth entry of the list. If there are more vcpus than
+     * entries, it is fine to just not touch the last array elements.
+     */
 
-        /* Silently ignore values corresponding to non existing vcpus */
-        if (num_cpus > b_info->max_vcpus)
-            num_cpus = b_info->max_vcpus;
+    /* Silently ignore values corresponding to non existing vcpus */
+    if (buf || num_cpus > b_info->max_vcpus)
+        num_cpus = b_info->max_vcpus;
 
+    if (is_hard) {
+        b_info->num_vcpu_hard_affinity = num_cpus;
         b_info->vcpu_hard_affinity = xmalloc(num_cpus * sizeof(libxl_bitmap));
+        vcpu_affinity_array = b_info->vcpu_hard_affinity;
+    } else {
+        b_info->num_vcpu_soft_affinity = num_cpus;
+        b_info->vcpu_soft_affinity = xmalloc(num_cpus * sizeof(libxl_bitmap));
+        vcpu_affinity_array = b_info->vcpu_soft_affinity;
+    }
+
+    if (!buf) {
+        int j = 0;
 
         while ((buf = xlu_cfg_get_listitem(cpus, j)) != NULL && j < num_cpus) {
-            libxl_bitmap_init(&b_info->vcpu_hard_affinity[j]);
-            if (libxl_cpu_bitmap_alloc(ctx,
-                                       &b_info->vcpu_hard_affinity[j], 0)) {
+            libxl_bitmap_init(&vcpu_affinity_array[j]);
+            if (libxl_cpu_bitmap_alloc(ctx, &vcpu_affinity_array[j], 0)) {
                 fprintf(stderr, "Unable to allocate cpumap for vcpu %d\n", j);
                 exit(1);
             }
 
-            if (vcpupin_parse(buf, &b_info->vcpu_hard_affinity[j]))
+            if (vcpupin_parse(buf, &vcpu_affinity_array[j]))
                 exit(1);
 
             j++;
         }
-        b_info->num_vcpu_hard_affinity = num_cpus;
 
         /* We have a list of cpumaps, disable automatic placement */
         libxl_defbool_set(&b_info->numa_placement, false);
-    }
-    else if (!xlu_cfg_get_string (config, "cpus", &buf, 0)) {
+    } else {
         int i;
 
-        b_info->vcpu_hard_affinity =
-            xmalloc(b_info->max_vcpus * sizeof(libxl_bitmap));
-
-        libxl_bitmap_init(&b_info->vcpu_hard_affinity[0]);
-        if (libxl_cpu_bitmap_alloc(ctx,
-                                   &b_info->vcpu_hard_affinity[0], 0)) {
+        libxl_bitmap_init(&vcpu_affinity_array[0]);
+        if (libxl_cpu_bitmap_alloc(ctx, &vcpu_affinity_array[0], 0)) {
             fprintf(stderr, "Unable to allocate cpumap for vcpu 0\n");
             exit(1);
         }
 
-        if (vcpupin_parse(buf, &b_info->vcpu_hard_affinity[0]))
+        if (vcpupin_parse(buf, &vcpu_affinity_array[0]))
             exit(1);
 
         for (i = 1; i < b_info->max_vcpus; i++) {
-            libxl_bitmap_init(&b_info->vcpu_hard_affinity[i]);
-            if (libxl_cpu_bitmap_alloc(ctx,
-                                       &b_info->vcpu_hard_affinity[i], 0)) {
+            libxl_bitmap_init(&vcpu_affinity_array[i]);
+            if (libxl_cpu_bitmap_alloc(ctx, &vcpu_affinity_array[i], 0)) {
                 fprintf(stderr, "Unable to allocate cpumap for vcpu %d\n", i);
                 exit(1);
             }
-            libxl_bitmap_copy(ctx, &b_info->vcpu_hard_affinity[i],
-                              &b_info->vcpu_hard_affinity[0]);
+            libxl_bitmap_copy(ctx, &vcpu_affinity_array[i],
+                              &vcpu_affinity_array[0]);
         }
-        b_info->num_vcpu_hard_affinity = num_cpus;
 
         libxl_defbool_set(&b_info->numa_placement, false);
     }
@@ -765,9 +774,9 @@ static void parse_config_data(const char *config_source,
     const char *buf;
     long l;
     XLU_Config *config;
-    XLU_ConfigList *vbds, *nics, *pcis, *cvfbs, *cpuids, *vtpms;
+    XLU_ConfigList *cpus, *vbds, *nics, *pcis, *cvfbs, *cpuids, *vtpms;
     XLU_ConfigList *ioports, *irqs, *iomem;
-    int num_ioports, num_irqs, num_iomem;
+    int num_ioports, num_irqs, num_iomem, num_cpus;
     int pci_power_mgmt = 0;
     int pci_msitranslate = 0;
     int pci_permissive = 0;
@@ -864,8 +873,15 @@ static void parse_config_data(const char *config_source,
     if (!xlu_cfg_get_long (config, "maxvcpus", &l, 0))
         b_info->max_vcpus = l;
 
-    /* Figure out VCPU hard-affinity ("cpus" config option) */
-    parse_vcpu_affinity(config, b_info);
+    buf = NULL;
+    if (!xlu_cfg_get_list (config, "cpus", &cpus, &num_cpus, 1) ||
+        !xlu_cfg_get_string (config, "cpus", &buf, 0))
+        parse_vcpu_affinity(b_info, cpus, buf, num_cpus, /* is_hard */ true);
+
+    buf = NULL;
+    if (!xlu_cfg_get_list (config, "cpus_soft", &cpus, &num_cpus, 1) ||
+        !xlu_cfg_get_string (config, "cpus_soft", &buf, 0))
+        parse_vcpu_affinity(b_info, cpus, buf, num_cpus, false);
 
     if (!xlu_cfg_get_long (config, "memory", &l, 0)) {
         b_info->max_memkb = l * 1024;

[PATCH v12 5/5] libxl: automatic NUMA placement affects soft affinity

[Dario Faggioli]

vCPU soft affinity and NUMA-aware scheduling does not have
to be related. However, soft affinity is how NUMA-aware
scheduling is actually implemented, and therefore, by default,
the results of automatic NUMA placement (at VM creation time)
are also used to set the soft affinity of all the vCPUs of
the domain.

Of course, this only happens if automatic NUMA placement is
enabled and actually takes place (for instance, if the user
does not specify any hard and soft affiniy in the xl config
file).

This also takes care of the vice-versa, i.e., don't trigger
automatic placement if the config file specifies either an
hard (the check for which was already there) or a soft (the
check for which is introduced by this commit) affinity.

Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
Acked-by: George Dunlap <george.dunlap@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
Changes from v8:
 * fixed a trivial memory leak (of a libxl_bitmap) introduced
   in v8 itself.

Changes from v7:
 * the patch has been reworked, following the major changes
   in the preceding ones. However, such rework involves
   --I think-- only straightforward code transformation,
   so I'm actually retaining the Ack-s this time.

Changes from v3:
 * rephrase comments and docs, as suggestd during review.
---
 docs/man/xl.cfg.pod.5                |   21 ++++++++++----------
 docs/misc/xl-numa-placement.markdown |   14 +++++++++++--
 tools/libxl/libxl_dom.c              |   36 +++++++++++++++++++++++++++++++---
 3 files changed, 56 insertions(+), 15 deletions(-)

diff --git a/docs/man/xl.cfg.pod.5 b/docs/man/xl.cfg.pod.5
index 5833054..1e04eed 100644
--- a/docs/man/xl.cfg.pod.5
+++ b/docs/man/xl.cfg.pod.5
@@ -162,16 +162,6 @@ here, and the soft affinity mask, provided via B<cpus\_soft=> (if any),
 is utilized to compute the domain node-affinity, for driving memory
 allocations.
 
-If we are on a NUMA machine (i.e., if the host has more than one NUMA
-node) and this option is not specified, libxl automatically tries to
-place the guest on the least possible number of nodes. That, however,
-will not affect vcpu pinning, so the guest will still be able to run on
-all the cpus. A heuristic approach is used for choosing the best node (or
-set of nodes), with the goals of maximizing performance for the guest
-and, at the same time, achieving efficient utilization of host cpus
-and memory. See F<docs/misc/xl-numa-placement.markdown> for more
-details.
-
 =item B<cpus_soft="CPU-LIST">
 
 Exactly as B<cpus=>, but specifies soft affinity, rather than pinning
@@ -186,6 +176,17 @@ the intersection of the soft affinity mask, provided here, and the vcpu
 pinning, provided via B<cpus=> (if any), is utilized to compute the
 domain node-affinity, for driving memory allocations.
 
+If this option is not specified (and B<cpus=> is not specified either),
+libxl automatically tries to place the guest on the least possible
+number of nodes. A heuristic approach is used for choosing the best
+node (or set of nodes), with the goal of maximizing performance for
+the guest and, at the same time, achieving efficient utilization of
+host cpus and memory. In that case, the soft affinity of all the vcpus
+of the domain will be set to the pcpus belonging to the NUMA nodes
+chosen during placement.
+
+For more details, see F<docs/misc/xl-numa-placement.markdown>.
+
 =back
 
 =head3 CPU Scheduling
diff --git a/docs/misc/xl-numa-placement.markdown b/docs/misc/xl-numa-placement.markdown
index 9d64eae..f863492 100644
--- a/docs/misc/xl-numa-placement.markdown
+++ b/docs/misc/xl-numa-placement.markdown
@@ -126,10 +126,20 @@ or Xen won't be able to guarantee the locality for their memory accesses.
 That, of course, also mean the vCPUs of the domain will only be able to
 execute on those same pCPUs.
 
+It is is also possible to have a "cpus\_soft=" option in the xl config file,
+to specify the soft affinity for all the vCPUs of the domain. This affects
+the NUMA placement in the following way:
+
+ * if only "cpus\_soft=" is present, the VM's node-affinity will be equal
+   to the nodes to which the pCPUs in the soft affinity mask belong;
+ * if both "cpus\_soft=" and "cpus=" are present, the VM's node-affinity
+   will be equal to the nodes to which the pCPUs present both in hard and
+   soft affinity belong.
+
 ### Placing the guest automatically ###
 
-If no "cpus=" option is specified in the config file, libxl tries
-to figure out on its own on which node(s) the domain could fit best.
+If neither "cpus=" nor "cpus\_soft=" are present in the config file, libxl
+tries to figure out on its own on which node(s) the domain could fit best.
 If it finds one (some), the domain's node affinity get set to there,
 and both memory allocations and NUMA aware scheduling (for the credit
 scheduler and starting from Xen 4.3) will comply with it. Starting from
diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c
index cfbd13d..c944804 100644
--- a/tools/libxl/libxl_dom.c
+++ b/tools/libxl/libxl_dom.c
@@ -247,11 +247,20 @@ int libxl__build_pre(libxl__gc *gc, uint32_t domid,
      * updated accordingly; if it does not manage, info->nodemap is just left
      * alone. It is then the the subsequent call to
      * libxl_domain_set_nodeaffinity() that enacts the actual placement.
+     *
+     * As far as scheduling is concerned, we achieve NUMA-aware scheduling
+     * by having the results of placement affect the soft affinity of all
+     * the vcpus of the domain. Of course, we want that iff placement is
+     * enabled and actually happens, so we only change info->cpumap_soft to
+     * reflect the placement result if that is the case
      */
     if (libxl_defbool_val(info->numa_placement)) {
-        if (info->cpumap.size || info->num_vcpu_hard_affinity) {
+        libxl_bitmap cpumap_soft;
+
+        if (info->cpumap.size ||
+            info->num_vcpu_hard_affinity || info->num_vcpu_soft_affinity) {
             LOG(ERROR, "Can run NUMA placement only if no vcpu "
-                       "affinity is specified explicitly");
+                       "(hard or soft) affinity is specified explicitly");
             return ERROR_INVAL;
         }
         if (info->nodemap.size) {
@@ -265,9 +274,30 @@ int libxl__build_pre(libxl__gc *gc, uint32_t domid,
             return rc;
         libxl_bitmap_set_any(&info->nodemap);
 
-        rc = numa_place_domain(gc, domid, info);
+        rc = libxl_cpu_bitmap_alloc(ctx, &cpumap_soft, 0);
         if (rc)
             return rc;
+
+        rc = numa_place_domain(gc, domid, info);
+        if (rc) {
+            libxl_bitmap_dispose(&cpumap_soft);
+            return rc;
+        }
+
+        /*
+         * All we need to do now is converting the result of automatic
+         * placement from nodemap to cpumap, and then use such cpumap as
+         * the soft affinity for all the vcpus of the domain.
+         *
+         * When calling libxl_set_vcpuaffinity_all(), it is ok to use NULL
+         * as hard affinity, as we know we don't have one, or we won't be
+         * here.
+         */
+        libxl_nodemap_to_cpumap(ctx, &info->nodemap, &cpumap_soft);
+        libxl_set_vcpuaffinity_all(ctx, domid, info->max_vcpus,
+                                   NULL, &cpumap_soft);
+
+        libxl_bitmap_dispose(&cpumap_soft);
     }
     if (info->nodemap.size)
         libxl_domain_set_nodeaffinity(ctx, domid, &info->nodemap);

Re: [PATCH v12 2/5] xl: move away from the use of cpumap for hard affinity

[Ian Campbell]

On Tue, 2014-07-22 at 17:45 +0200, Dario Faggioli wrote:
[...]
> @@ -827,15 +826,31 @@ static void parse_config_data(const char *config_source,
>          libxl_defbool_set(&b_info->numa_placement, false);
>      }
>      else if (!xlu_cfg_get_string (config, "cpus", &buf, 0)) {
> -        if (libxl_cpu_bitmap_alloc(ctx, &b_info->cpumap, 0)) {
> -            fprintf(stderr, "Unable to allocate cpumap\n");
> +        b_info->vcpu_hard_affinity =
> +            xmalloc(b_info->max_vcpus * sizeof(libxl_bitmap));
> +
> +        libxl_bitmap_init(&b_info->vcpu_hard_affinity[0]);
> +        if (libxl_cpu_bitmap_alloc(ctx,
> +                                   &b_info->vcpu_hard_affinity[0], 0)) {
> +            fprintf(stderr, "Unable to allocate cpumap for vcpu 0\n");
>              exit(1);
>          }
>  
> -        libxl_bitmap_set_none(&b_info->cpumap);
> -        if (vcpupin_parse(buf, &b_info->cpumap))
> +        if (vcpupin_parse(buf, &b_info->vcpu_hard_affinity[0]))
>              exit(1);
>  
> +        for (i = 1; i < b_info->max_vcpus; i++) {
> +            libxl_bitmap_init(&b_info->vcpu_hard_affinity[i]);
> +            if (libxl_cpu_bitmap_alloc(ctx,
> +                                       &b_info->vcpu_hard_affinity[i], 0)) {
> +                fprintf(stderr, "Unable to allocate cpumap for vcpu %d\n", i);
> +                exit(1);
> +            }
> +            libxl_bitmap_copy(ctx, &b_info->vcpu_hard_affinity[i],
> +                              &b_info->vcpu_hard_affinity[0]);
> +        }
> +        b_info->num_vcpu_hard_affinity = num_cpus;

In all the other loops etc you use b_info->max_vcpus and AFAICT num_cpus
is only initialised if the xlu_cfg_get_list succeeded, which it can't
have done if we ended up in this else branch.

Other than looks good to me.

Ian.

Re: [PATCH v12 4/5] libxl/xl: make it possible to specify soft-affinity in domain config file

[Ian Campbell]

On Tue, 2014-07-22 at 17:46 +0200, Dario Faggioli wrote:
> To do so, we add the vcpu_soft_affinity array to build_info, and
> treat it much like vcpu_hard_affinity. The new config option is
> called "cpus_soft".
> 
> Note that the vcpu_hard_affinity array, introduced in a previous
> patch, and the vcpu_soft_affinity array, introduced here, share
> the same LIBXL_HAVE_xxx macro, in libxl.h. That is called
> LIBXL_HAVE_BUILDINFO_VCPU_AFFINITY_ARRAYS, and was introduced
> together with vcpu_hard_affinity, but only inside a comment.
> In this change, we uncomment, and hence properly define it.
> 
> In order to avoid having to issue separate calls to
> libxl_set_vcpuaffinity() (one for hard affinity and one for soft
> affinity) in libxl__build_pre(), in case the caller uses
> b_info->cpumap (for the former) and b_info->vcpu_soft_affinity (for
> the latter), we also set (again!) a new default for b_info->cpumap.
> This allows, from this change on, to always and only deal with
> b_info->vcpu_hard_affinity all around libxl internals.
> 
> Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>

I acked v11 and there is no "Changes from v11" so unless things did
actually changed, without re-reading the patch:
Acked-by: Ian Campbell <ian.campbell@citrix.com>