[PATCH v14] claim and its friends for allocating multiple self-ballooning guests.

[PATCH v14] claim and its friends for allocating multiple self-ballooning guests.

[Konrad Rzeszutek Wilk]

Changes since v13:
 - Addressed Ian Jacksons' comments - added extra docs, redid the parsing
   of claim_mode.
 - s/global_claim_mode/claim_mode/
 - Dropped xend patch
 - Dropped 'outstanding_pages' patches for libxc and libxl.
Changes since v12:
 - Addressed Ian Campbells' comments.

The patch (mmu: Introduce XENMEM_claim_pages (subop of memory ops) is already
in the hypervisor and described in details the problem/solution/alternative
solutions. This builds upon that new hypercall to expand the toolstack to
utilize it.

The patches follow the normal code-flow - the patch to implement the two 
hypercalls: XENMEM_claim_pages and XENMEM_get_outstanding_pages.

Then the patches to utilize them in the libxc. The hypercall's are only utilized
if the toolstack (libxl) sets the claim_mode to 1 (true).

Then the toolstack (libxl + xl) patches. They revolve around two different 
changes:
 1). Add 'claim_mode=0|1' global configuration value that determines
     whether the claim hypercall should be used as part of guest creation.
 2). As part of  'xl info' output how many pages are claimed by different guests.
     This is more of a diagnostic patch.

These patches are also visible at:

  git://xenbits.xen.org/people/konradwilk/xen.git claim.v14


 docs/man/xl.conf.pod.5         | 41 +++++++++++++++++++++++++++++++++++++++++
 docs/man/xl.pod.1              | 13 +++++++++++++
 tools/examples/xl.conf         |  6 ++++++
 tools/libxc/xc_dom.h           |  1 +
 tools/libxc/xc_dom_x86.c       | 12 ++++++++++++
 tools/libxc/xc_domain.c        | 30 ++++++++++++++++++++++++++++++
 tools/libxc/xc_hvm_build_x86.c | 23 +++++++++++++++++++----
 tools/libxc/xenctrl.h          |  6 ++++++
 tools/libxc/xenguest.h         |  2 ++
 tools/libxl/libxl.c            | 13 +++++++++++++
 tools/libxl/libxl.h            |  2 +-
 tools/libxl/libxl_create.c     |  2 ++
 tools/libxl/libxl_dom.c        |  3 ++-
 tools/libxl/libxl_types.idl    |  2 +-
 tools/libxl/xl.c               |  5 +++++
 tools/libxl/xl.h               |  1 +
 tools/libxl/xl_cmdimpl.c       | 26 ++++++++++++++++++++++++++
 17 files changed, 181 insertions(+), 7 deletions(-)


Dan Magenheimer (1):
      xc: use XENMEM_claim_pages hypercall during guest creation.

Konrad Rzeszutek Wilk (2):
      xl: Implement XENMEM_claim_pages support via 'claim_mode' global config
      xl: 'xl info' print outstanding claims if enabled (claim_mode=1 in xl.conf)

[PATCH 1/3] xc: use XENMEM_claim_pages hypercall during guest creation.

[Konrad Rzeszutek Wilk]

From: Dan Magenheimer <dan.magenheimer@oracle.com>

We add an extra parameter to the structures passed to the
PV routine (arch_setup_meminit) and HVM routine (setup_guest)
that determines whether the claim hypercall is to be done.

The contents of the 'claim_enabled' is defined as an 'int'
in case the hypercall expands in the future with extra
flags (for example for per-NUMA allocation). For right now
the proper values are: 0 to disable it or 1 to enable
it.

If the hypervisor does not support this function, the
xc_domain_claim_pages and xc_domain_get_outstanding_pages
will silently return 0 (and set errno to zero).

Signed-off-by: Dan Magenheimer <dan.magenheimer@oracle.com>
[v2: Updated per Ian's recommendations]
[v3: Added support for out-of-sync hypervisor]
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 tools/libxc/xc_dom.h           |  1 +
 tools/libxc/xc_dom_x86.c       | 12 ++++++++++++
 tools/libxc/xc_domain.c        | 30 ++++++++++++++++++++++++++++++
 tools/libxc/xc_hvm_build_x86.c | 23 +++++++++++++++++++----
 tools/libxc/xenctrl.h          |  6 ++++++
 tools/libxc/xenguest.h         |  2 ++
 6 files changed, 70 insertions(+), 4 deletions(-)

diff --git a/tools/libxc/xc_dom.h b/tools/libxc/xc_dom.h
index 779b9d4..ac36600 100644
--- a/tools/libxc/xc_dom.h
+++ b/tools/libxc/xc_dom.h
@@ -135,6 +135,7 @@ struct xc_dom_image {
     domid_t guest_domid;
     int8_t vhpt_size_log2; /* for IA64 */
     int8_t superpages;
+    int claim_enabled; /* 0 by default, 1 enables it */
     int shadow_enabled;
 
     int xen_version;
diff --git a/tools/libxc/xc_dom_x86.c b/tools/libxc/xc_dom_x86.c
index eb9ac07..d89526d 100644
--- a/tools/libxc/xc_dom_x86.c
+++ b/tools/libxc/xc_dom_x86.c
@@ -706,6 +706,13 @@ int arch_setup_meminit(struct xc_dom_image *dom)
     }
     else
     {
+        /* try to claim pages for early warning of insufficient memory avail */
+        if ( dom->claim_enabled ) {
+            rc = xc_domain_claim_pages(dom->xch, dom->guest_domid,
+                                       dom->total_pages);
+            if ( rc )
+                return rc;
+        }
         /* setup initial p2m */
         for ( pfn = 0; pfn < dom->total_pages; pfn++ )
             dom->p2m_host[pfn] = pfn;
@@ -722,6 +729,11 @@ int arch_setup_meminit(struct xc_dom_image *dom)
                 dom->xch, dom->guest_domid, allocsz,
                 0, 0, &dom->p2m_host[i]);
         }
+
+        /* Ensure no unclaimed pages are left unused.
+         * OK to call if hadn't done the earlier claim call. */
+        (void)xc_domain_claim_pages(dom->xch, dom->guest_domid,
+                                    0 /* cancels the claim */);
     }
 
     return rc;
diff --git a/tools/libxc/xc_domain.c b/tools/libxc/xc_domain.c
index 480ce91..299c907 100644
--- a/tools/libxc/xc_domain.c
+++ b/tools/libxc/xc_domain.c
@@ -775,6 +775,36 @@ int xc_domain_add_to_physmap(xc_interface *xch,
     return do_memory_op(xch, XENMEM_add_to_physmap, &xatp, sizeof(xatp));
 }
 
+int xc_domain_claim_pages(xc_interface *xch,
+                               uint32_t domid,
+                               unsigned long nr_pages)
+{
+    int err;
+    struct xen_memory_reservation reservation = {
+        .nr_extents   = nr_pages,
+        .extent_order = 0,
+        .mem_flags    = 0, /* no flags */
+        .domid        = domid
+    };
+
+    set_xen_guest_handle(reservation.extent_start, HYPERCALL_BUFFER_NULL);
+
+    err = do_memory_op(xch, XENMEM_claim_pages, &reservation, sizeof(reservation));
+    /* Ignore it if the hypervisor does not support the call. */
+    if (err == -1 && errno == ENOSYS)
+        err = errno = 0;
+    return err;
+}
+unsigned long xc_domain_get_outstanding_pages(xc_interface *xch)
+{
+    long ret = do_memory_op(xch, XENMEM_get_outstanding_pages, NULL, 0);
+
+    /* Ignore it if the hypervisor does not support the call. */
+    if (ret == -1 && errno == ENOSYS)
+        ret = errno = 0;
+    return ret;
+}
+
 int xc_domain_populate_physmap(xc_interface *xch,
                                uint32_t domid,
                                unsigned long nr_extents,
diff --git a/tools/libxc/xc_hvm_build_x86.c b/tools/libxc/xc_hvm_build_x86.c
index 3b5d777..ab33a7f 100644
--- a/tools/libxc/xc_hvm_build_x86.c
+++ b/tools/libxc/xc_hvm_build_x86.c
@@ -252,6 +252,7 @@ static int setup_guest(xc_interface *xch,
     unsigned long stat_normal_pages = 0, stat_2mb_pages = 0, 
         stat_1gb_pages = 0;
     int pod_mode = 0;
+    int claim_enabled = args->claim_enabled;
 
     if ( nr_pages > target_pages )
         pod_mode = XENMEMF_populate_on_demand;
@@ -329,6 +330,16 @@ static int setup_guest(xc_interface *xch,
         xch, dom, 0xa0, 0, pod_mode, &page_array[0x00]);
     cur_pages = 0xc0;
     stat_normal_pages = 0xc0;
+
+    /* try to claim pages for early warning of insufficient memory available */
+    if ( claim_enabled ) {
+        rc = xc_domain_claim_pages(xch, dom, nr_pages - cur_pages);
+        if ( rc != 0 )
+        {
+            PERROR("Could not allocate memory for HVM guest as we cannot claim memory!");
+            goto error_out;
+        }
+    }
     while ( (rc == 0) && (nr_pages > cur_pages) )
     {
         /* Clip count to maximum 1GB extent. */
@@ -506,12 +517,16 @@ static int setup_guest(xc_interface *xch,
         munmap(page0, PAGE_SIZE);
     }
 
-    free(page_array);
-    return 0;
-
+    rc = 0;
+    goto out;
  error_out:
+    rc = -1;
+ out:
+    /* ensure no unclaimed pages are left unused */
+    xc_domain_claim_pages(xch, dom, 0 /* cancels the claim */);
+
     free(page_array);
-    return -1;
+    return rc;
 }
 
 /* xc_hvm_build:
diff --git a/tools/libxc/xenctrl.h b/tools/libxc/xenctrl.h
index 32122fd..e695456 100644
--- a/tools/libxc/xenctrl.h
+++ b/tools/libxc/xenctrl.h
@@ -1129,6 +1129,12 @@ int xc_domain_populate_physmap_exact(xc_interface *xch,
                                      unsigned int mem_flags,
                                      xen_pfn_t *extent_start);
 
+int xc_domain_claim_pages(xc_interface *xch,
+                               uint32_t domid,
+                               unsigned long nr_pages);
+
+unsigned long xc_domain_get_outstanding_pages(xc_interface *xch);
+
 int xc_domain_memory_exchange_pages(xc_interface *xch,
                                     int domid,
                                     unsigned long nr_in_extents,
diff --git a/tools/libxc/xenguest.h b/tools/libxc/xenguest.h
index 7d4ac33..4714bd2 100644
--- a/tools/libxc/xenguest.h
+++ b/tools/libxc/xenguest.h
@@ -231,6 +231,8 @@ struct xc_hvm_build_args {
 
     /* Extra SMBIOS structures passed to HVMLOADER */
     struct xc_hvm_firmware_module smbios_module;
+    /* Whether to use claim hypercall (1 - enable, 0 - disable). */
+    int claim_enabled;
 };
 
 /**
-- 
1.8.0.2

[PATCH 2/3] xl: Implement XENMEM_claim_pages support via 'claim_mode' global config

[Konrad Rzeszutek Wilk]

The XENMEM_claim_pages hypercall operates per domain and it should be
used system wide. As such this patch introduces a global configuration
option 'claim_mode' that by default is disabled.

If this option is enabled then when a guest is created there will be an
guarantee that there is memory available for the guest. This is an
particularly acute problem on hosts with memory over-provisioned guests
that use tmem and have self-balloon enabled (which is the default option
for them). The self-balloon mechanism can deflate/inflate the balloon
quickly and the amount of free memory (which 'xl info' can show) is stale
the moment it is printed. When claim is enabled a reservation for the
amount of memory ('memory' in guest config) is set, which is then reduced
as the domain's memory is populated and eventually reaches zero.

If the reservation cannot be meet the guest creation fails immediately
instead of taking seconds/minutes (depending on the size of the guest)
while the guest is populated.

Note that to enable tmem type guests, one needs to provide 'tmem' on the
Xen hypervisor argument and as well on the Linux kernel command line.

There are two boolean options:

(0) No claim is made. Memory population during guest creation will be
attempted as normal and may fail due to memory exhaustion.

(1) Normal memory and freeable pool of ephemeral pages (tmem) is used when
calculating whether there is enough memory free to launch a guest.
This guarantees immediate feedback whether the guest can be launched due
to memory exhaustion (which can take a long time to find out if launching
massively huge guests) and in parallel.

[v1: Removed own claim_mode type, using just bool, improved docs, all per
Ian's suggestion]
[v2: Updated the comments]
[v3: Rebase on top 733b9c524dbc2bec318bfc3588ed1652455d30ec (xl: add vif.default.script)]
[v4: Fixed up comments]
[v5: s/global_claim_mode/claim_mode/]
[v6: Ian Jackson's feedback: use libxl_defbool, better comments, etc]
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 docs/man/xl.conf.pod.5      | 41 +++++++++++++++++++++++++++++++++++++++++
 tools/examples/xl.conf      |  6 ++++++
 tools/libxl/libxl.h         |  1 -
 tools/libxl/libxl_create.c  |  2 ++
 tools/libxl/libxl_dom.c     |  3 ++-
 tools/libxl/libxl_types.idl |  2 +-
 tools/libxl/xl.c            |  5 +++++
 tools/libxl/xl.h            |  1 +
 tools/libxl/xl_cmdimpl.c    |  2 ++
 9 files changed, 60 insertions(+), 3 deletions(-)

diff --git a/docs/man/xl.conf.pod.5 b/docs/man/xl.conf.pod.5
index 7b9fcac..08c7120 100644
--- a/docs/man/xl.conf.pod.5
+++ b/docs/man/xl.conf.pod.5
@@ -108,6 +108,47 @@ Configures the name of the first block device to be used for temporary
 block device allocations by the toolstack.
 The default choice is "xvda".
 
+=item B<claim_mode=BOOLEAN>
+
+If this option is enabled then when a guest is created there will be an
+guarantee that there is memory available for the guest. This is an
+particularly acute problem on hosts with memory over-provisioned guests
+that use tmem and have self-balloon enabled (which is the default
+option). The self-balloon mechanism can deflate/inflate the balloon
+quickly and the amount of free memory (which C<xl info> can show) is
+stale the moment it is printed. When claim is enabled a reservation for
+the amount of memory (see 'memory' in xl.conf(5)) is set, which is then
+reduced as the domain's memory is populated and eventually reaches zero.
+
+If the reservation cannot be meet the guest creation fails immediately
+instead of taking seconds/minutes (depending on the size of the guest)
+while the guest is populated.
+
+Note that to enable tmem type guests, one needs to provide C<tmem> on the
+Xen hypervisor argument and as well on the Linux kernel command line.
+
+Note that the claim call is not attempted if C<superpages> option is
+used in the guest config (see xl.cfg(5)).
+
+Default: C<0>
+
+=over 4
+
+=item C<0>
+
+No claim is made. Memory population during guest creation will be
+attempted as normal and may fail due to memory exhaustion.
+
+=item C<1>
+
+Normal memory and freeable pool of ephemeral pages (tmem) is used when
+calculating whether there is enough memory free to launch a guest.
+This guarantees immediate feedback whether the guest can be launched due
+to memory exhaustion (which can take a long time to find out if launching
+massively huge guests).
+
+=back
+
 =back
 
 =head1 SEE ALSO
diff --git a/tools/examples/xl.conf b/tools/examples/xl.conf
index b0caa32..f386bb9 100644
--- a/tools/examples/xl.conf
+++ b/tools/examples/xl.conf
@@ -26,3 +26,9 @@
 
 # default bridge device to use with vif-bridge hotplug scripts
 #vif.default.bridge="xenbr0"
+
+# Reserve a claim of memory when launching a guest. This guarantees immediate
+# feedback whether the guest can be launched due to memory exhaustion
+# (which can take a long time to find out if launching huge guests).
+# see xl.conf(5) for details.
+#claim_mode=0
diff --git a/tools/libxl/libxl.h b/tools/libxl/libxl.h
index 030aa86..c4ad58b 100644
--- a/tools/libxl/libxl.h
+++ b/tools/libxl/libxl.h
@@ -579,7 +579,6 @@ int libxl_wait_for_free_memory(libxl_ctx *ctx, uint32_t domid, uint32_t memory_k
 /* wait for the memory target of a domain to be reached */
 int libxl_wait_for_memory_target(libxl_ctx *ctx, uint32_t domid, int wait_secs);
 
-
 int libxl_vncviewer_exec(libxl_ctx *ctx, uint32_t domid, int autopass);
 int libxl_console_exec(libxl_ctx *ctx, uint32_t domid, int cons_num, libxl_console_type type);
 /* libxl_primary_console_exec finds the domid and console number
diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
index 30a4507..ae72f21 100644
--- a/tools/libxl/libxl_create.c
+++ b/tools/libxl/libxl_create.c
@@ -196,6 +196,8 @@ int libxl__domain_build_info_setdefault(libxl__gc *gc,
     if (b_info->target_memkb == LIBXL_MEMKB_DEFAULT)
         b_info->target_memkb = b_info->max_memkb;
 
+    libxl_defbool_setdefault(&b_info->claim_mode, false);
+
     libxl_defbool_setdefault(&b_info->localtime, false);
 
     libxl_defbool_setdefault(&b_info->disable_migrate, false);
diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c
index 2dd429f..92a6628 100644
--- a/tools/libxl/libxl_dom.c
+++ b/tools/libxl/libxl_dom.c
@@ -371,6 +371,7 @@ int libxl__build_pv(libxl__gc *gc, uint32_t domid,
     dom->console_domid = state->console_domid;
     dom->xenstore_evtchn = state->store_port;
     dom->xenstore_domid = state->store_domid;
+    dom->claim_enabled = libxl_defbool_val(info->claim_mode);
 
     if ( (ret = xc_dom_boot_xen_init(dom, ctx->xch, domid)) != 0 ) {
         LOGE(ERROR, "xc_dom_boot_xen_init failed");
@@ -605,7 +606,7 @@ int libxl__build_hvm(libxl__gc *gc, uint32_t domid,
      */
     args.mem_size = (uint64_t)(info->max_memkb - info->video_memkb) << 10;
     args.mem_target = (uint64_t)(info->target_memkb - info->video_memkb) << 10;
-
+    args.claim_enabled = libxl_defbool_val(info->claim_mode);
     if (libxl__domain_firmware(gc, info, &args)) {
         LOG(ERROR, "initializing domain firmware failed");
         goto out;
diff --git a/tools/libxl/libxl_types.idl b/tools/libxl/libxl_types.idl
index f3c212b..0f1f118 100644
--- a/tools/libxl/libxl_types.idl
+++ b/tools/libxl/libxl_types.idl
@@ -293,7 +293,7 @@ libxl_domain_build_info = Struct("domain_build_info",[
     ("ioports",          Array(libxl_ioport_range, "num_ioports")),
     ("irqs",             Array(uint32, "num_irqs")),
     ("iomem",            Array(libxl_iomem_range, "num_iomem")),
-
+    ("claim_mode",	     libxl_defbool),
     ("u", KeyedUnion(None, libxl_domain_type, "type",
                 [("hvm", Struct(None, [("firmware",         string),
                                        ("bios",             libxl_bios_type),
diff --git a/tools/libxl/xl.c b/tools/libxl/xl.c
index 4c598db..211facd 100644
--- a/tools/libxl/xl.c
+++ b/tools/libxl/xl.c
@@ -45,6 +45,7 @@ char *default_vifscript = NULL;
 char *default_bridge = NULL;
 char *default_gatewaydev = NULL;
 enum output_format default_output_format = OUTPUT_FORMAT_JSON;
+libxl_defbool claim_mode;
 
 static xentoollog_level minmsglevel = XTL_PROGRESS;
 
@@ -134,6 +135,10 @@ static void parse_global_config(const char *configfile,
     }
     if (!xlu_cfg_get_string (config, "blkdev_start", &buf, 0))
         blkdev_start = strdup(buf);
+
+    libxl_defbool_setdefault(&claim_mode, false);
+    (void)xlu_cfg_get_defbool (config, "claim_mode", &claim_mode, 0);
+
     xlu_cfg_destroy(config);
 }
 
diff --git a/tools/libxl/xl.h b/tools/libxl/xl.h
index b881f92..4c5e5d1 100644
--- a/tools/libxl/xl.h
+++ b/tools/libxl/xl.h
@@ -145,6 +145,7 @@ int xl_child_pid(xlchildnum); /* returns 0 if child struct is not in use */
 extern int autoballoon;
 extern int run_hotplug_scripts;
 extern int dryrun_only;
+extern libxl_defbool claim_mode;
 extern char *lockfile;
 extern char *default_vifscript;
 extern char *default_bridge;
diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
index 2d40f8f..c8b0a99 100644
--- a/tools/libxl/xl_cmdimpl.c
+++ b/tools/libxl/xl_cmdimpl.c
@@ -757,6 +757,8 @@ static void parse_config_data(const char *config_source,
     if (!xlu_cfg_get_long (config, "maxmem", &l, 0))
         b_info->max_memkb = l * 1024;
 
+    b_info->claim_mode = claim_mode;
+
     if (xlu_cfg_get_string (config, "on_poweroff", &buf, 0))
         buf = "destroy";
     if (!parse_action_on_shutdown(buf, &d_config->on_poweroff)) {
-- 
1.8.0.2

[PATCH 3/3] xl: 'xl info' print outstanding claims if enabled (claim_mode=1 in xl.conf)

[Konrad Rzeszutek Wilk]

This patch provides the value of the currently outstanding pages
claimed for all domains. This is a total global value that influences
the hypervisors' MM system.

When a claim call is done, a reservation for a specific amount of pages
is set and also a global value is incremented. This global value is then
reduced as the domain's memory is populated and eventually reaches zero.
The toolstack can also choose to set the domain's claim to zero which
cancels the reservation and decrements the global value by the amount
of claim that has not been satisfied.

If the reservation cannot be meet the guest creation fails immediately
instead of taking seconds or minutes (depending on the size of the guest)
while the toolstack populates memory.

See patch: "xl: Implement XENMEM_claim_pages support via 'claim_mode'
global config" for details on how it is implemented.

The value fluctuates quite often so the value is stale once it is provided
to the user-space.  However it is useful for diagnostic purposes.

It is only printed when the global "claim_mode" option in xl.conf(5)
is set to enabled (1). The 'man xl' shows the details of this item.

[v1: s/unclaimed/outstanding/]
[v2: Made libxl_get_claiminfo return just MemKB suggested by Ian Campbell]
[v3: Made libxl_get_claininfo return MemMB to conform to the other values printed]
[v4: Improvements suggested by Ian Jackson, also added docs to xl.pod.1]
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 docs/man/xl.pod.1        | 13 +++++++++++++
 tools/libxl/libxl.c      | 13 +++++++++++++
 tools/libxl/libxl.h      |  1 +
 tools/libxl/xl_cmdimpl.c | 24 ++++++++++++++++++++++++
 4 files changed, 51 insertions(+)

diff --git a/docs/man/xl.pod.1 b/docs/man/xl.pod.1
index a0e298e..98349ab 100644
--- a/docs/man/xl.pod.1
+++ b/docs/man/xl.pod.1
@@ -704,6 +704,7 @@ Sample output looks as follows:
  total_memory           : 6141
  free_memory            : 4274
  free_cpus              : 0
+ outstanding_claims     : 0
  xen_major              : 4
  xen_minor              : 2
  xen_extra              : -unstable
@@ -738,6 +739,18 @@ the feature bits returned by the cpuid command on x86 platforms.
 
 Available memory (in MB) not allocated to Xen, or any other domains.
 
+=item B<outstanding_claims>
+
+When a claim call is done (see L<xl.conf>) a reservation for a specific
+amount of pages is set and also a global value is incremented. This
+global value (outstanding_claims) is then reduced as the domain's memory
+is populated and eventually reaches zero. The toolstack can also choose
+to set the domain's claim to zero which cancels the reservation and
+decrements the global value by the amount of claim that has not been
+satisfied. Most of the time the value will be zero, but if you are
+launching multiple guests, and B<claim_mode> is enabled, this value
+can increase/decrease.
+
 =item B<xen_caps>
 
 The Xen version and architecture.  Architecture values can be one of:
diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
index 572c2c6..348284c 100644
--- a/tools/libxl/libxl.c
+++ b/tools/libxl/libxl.c
@@ -4057,6 +4057,19 @@ libxl_numainfo *libxl_get_numainfo(libxl_ctx *ctx, int *nr)
     return ret;
 }
 
+uint64_t libxl_get_claiminfo(libxl_ctx *ctx)
+{
+    long l;
+
+    l = xc_domain_get_outstanding_pages(ctx->xch);
+    if (l < 0) {
+        LIBXL__LOG_ERRNOVAL(ctx, LIBXL__LOG_WARNING, l, "xc_domain_get_outstanding_pages failed.");
+        return ERROR_FAIL;
+    }
+    /* In MB */
+    return (l >> 8);
+}
+
 const libxl_version_info* libxl_get_version_info(libxl_ctx *ctx)
 {
     union {
diff --git a/tools/libxl/libxl.h b/tools/libxl/libxl.h
index c4ad58b..5dab24b 100644
--- a/tools/libxl/libxl.h
+++ b/tools/libxl/libxl.h
@@ -579,6 +579,7 @@ int libxl_wait_for_free_memory(libxl_ctx *ctx, uint32_t domid, uint32_t memory_k
 /* wait for the memory target of a domain to be reached */
 int libxl_wait_for_memory_target(libxl_ctx *ctx, uint32_t domid, int wait_secs);
 
+uint64_t libxl_get_claiminfo(libxl_ctx *ctx);
 int libxl_vncviewer_exec(libxl_ctx *ctx, uint32_t domid, int autopass);
 int libxl_console_exec(libxl_ctx *ctx, uint32_t domid, int cons_num, libxl_console_type type);
 /* libxl_primary_console_exec finds the domid and console number
diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
index c8b0a99..1e97961 100644
--- a/tools/libxl/xl_cmdimpl.c
+++ b/tools/libxl/xl_cmdimpl.c
@@ -4650,6 +4650,29 @@ static void output_topologyinfo(void)
     return;
 }
 
+static void output_claim(void)
+{
+    long l;
+
+    /*
+     * Note that the xl.c (which calls us) has already read from the
+     * global configuration the 'claim_mode' value.
+     */
+    if (!libxl_defbool_val(claim_mode))
+        return;
+
+    l = libxl_get_claiminfo(ctx);
+    if (l < 0) {
+        fprintf(stderr, "libxl_get_claiminfo failed. errno: %d (%s)\n",
+                errno, strerror(errno));
+        return;
+    }
+
+    printf("outstanding_claims     : %ld\n", l);
+
+    return;
+}
+
 static void print_info(int numa)
 {
     output_nodeinfo();
@@ -4660,6 +4683,7 @@ static void print_info(int numa)
         output_topologyinfo();
         output_numainfo();
     }
+    output_claim();
 
     output_xeninfo();
 
-- 
1.8.0.2

Re: [PATCH 3/3] xl: 'xl info' print outstanding claims if enabled (claim_mode=1 in xl.conf)

[Ian Jackson]

Konrad Rzeszutek Wilk writes ("[Xen-devel] [PATCH 3/3] xl: 'xl info' print outstanding claims if enabled (claim_mode=1 in xl.conf)"):
> +=item B<outstanding_claims>
> +
> +When a claim call is done (see L<xl.conf>) a reservation for a specific
> +amount of pages is set and also a global value is incremented. This
> +global value (outstanding_claims) is then reduced as the domain's memory
> +is populated and eventually reaches zero. The toolstack can also choose
> +to set the domain's claim to zero which cancels the reservation and
> +decrements the global value by the amount of claim that has not been
> +satisfied. Most of the time the value will be zero, but if you are
> +launching multiple guests, and B<claim_mode> is enabled, this value
> +can increase/decrease.

OK, thanks, this is clear enough in general.

But you need to explain how the user can get the toolstack to set the
domain's claim to zero.

And you need to explain whether the memory counted in
outstanding_claims value is _also_ counted as free memory, or as
memory belonging to its respective domain(s).

> +uint64_t libxl_get_claiminfo(libxl_ctx *ctx)
> +{
> +    long l;
> +
> +    l = xc_domain_get_outstanding_pages(ctx->xch);
> +    if (l < 0) {
> +        LIBXL__LOG_ERRNOVAL(ctx, LIBXL__LOG_WARNING, l, "xc_domain_get_outstanding_pages failed.");

This needs wrapping to 75 columns or so.

Also, is there a way to find out which domain these claims belong to ?
I didn't see it in your previous patch.

Thanks,
Ian.

Re: [PATCH 3/3] xl: 'xl info' print outstanding claims if enabled (claim_mode=1 in xl.conf)

[Konrad Rzeszutek Wilk]

On Mon, Apr 08, 2013 at 05:41:07PM +0100, Ian Jackson wrote:
> Konrad Rzeszutek Wilk writes ("[Xen-devel] [PATCH 3/3] xl: 'xl info' print outstanding claims if enabled (claim_mode=1 in xl.conf)"):
> > +=item B<outstanding_claims>
> > +
> > +When a claim call is done (see L<xl.conf>) a reservation for a specific
> > +amount of pages is set and also a global value is incremented. This
> > +global value (outstanding_claims) is then reduced as the domain's memory
> > +is populated and eventually reaches zero. The toolstack can also choose
> > +to set the domain's claim to zero which cancels the reservation and
> > +decrements the global value by the amount of claim that has not been
> > +satisfied. Most of the time the value will be zero, but if you are
> > +launching multiple guests, and B<claim_mode> is enabled, this value
> > +can increase/decrease.
> 
> OK, thanks, this is clear enough in general.
> 
> But you need to explain how the user can get the toolstack to set the
> domain's claim to zero.

I can certainly include that. It does that once the guest has been started
(also the hypervisor sets it to zero as the pages get allocated for the guest).

> 
> And you need to explain whether the memory counted in
> outstanding_claims value is _also_ counted as free memory, or as
> memory belonging to its respective domain(s).

Sure thing.
> 
> > +uint64_t libxl_get_claiminfo(libxl_ctx *ctx)
> > +{
> > +    long l;
> > +
> > +    l = xc_domain_get_outstanding_pages(ctx->xch);
> > +    if (l < 0) {
> > +        LIBXL__LOG_ERRNOVAL(ctx, LIBXL__LOG_WARNING, l, "xc_domain_get_outstanding_pages failed.");
> 
> This needs wrapping to 75 columns or so.

OK. Do you prefer the "xc..." to be split up or on a seperate line?
> 
> Also, is there a way to find out which domain these claims belong to ?
> I didn't see it in your previous patch.

It is possible - the two patches that would have exported said information
to xcinfo were dropped as I was unsure where to display this. As in I could
alter 'xl list' or 'xl mem-list' to contain that information, but that would
change the syntax of the printout. I don't know if that is OK? 

Or I can introduce a new command 'xl claim-list' ? That would give that information?
> 
> Thanks,
> Ian.

Re: [PATCH 3/3] xl: 'xl info' print outstanding claims if enabled (claim_mode=1 in xl.conf)

[Ian Jackson]

Konrad Rzeszutek Wilk writes ("Re: [Xen-devel] [PATCH 3/3] xl: 'xl info' print outstanding claims if enabled (claim_mode=1 in xl.conf)"):
> On Mon, Apr 08, 2013 at 05:41:07PM +0100, Ian Jackson wrote:
> > But you need to explain how the user can get the toolstack to set the
> > domain's claim to zero.
> 
> I can certainly include that. It does that once the guest has been started
> (also the hypervisor sets it to zero as the pages get allocated for the guest).

Maybe I'm confused.  I took

     The toolstack can also choose
     to set the domain's claim to zero

to mean that a user might reasonably instruct the toolstack to set the
claim to zero.

If there is no facility to do that then I'm not sure why you would
mention setting the claim to zero in the documentation.

> > > +    l = xc_domain_get_outstanding_pages(ctx->xch);
> > > +    if (l < 0) {
> > > +        LIBXL__LOG_ERRNOVAL(ctx, LIBXL__LOG_WARNING, l, "xc_domain_get_outstanding_pages failed.");
> > 
> > This needs wrapping to 75 columns or so.
> 
> OK. Do you prefer the "xc..." to be split up or on a seperate line?

Either will do.

> > Also, is there a way to find out which domain these claims belong to ?
> > I didn't see it in your previous patch.
> 
> It is possible - the two patches that would have exported said
> information to xcinfo were dropped as I was unsure where to display
> this. As in I could alter 'xl list' or 'xl mem-list' to contain that
> information, but that would change the syntax of the printout. I
> don't know if that is OK?

I agree that adding stuff to those interfaces is a bit troublesome.
But maybe there could be an option to enable claim information ?

I guess by default the claimed-but-not-yet-taken memory should show up
as included in the domain's memory allocation ?  Since
taken+(claimed_but_not_taken) is a more stable value and one the user
can adjust.

> Or I can introduce a new command 'xl claim-list' ? That would give
> that information?

That would be fine too.  Whatever you think is best.

Ian.