[PATCH v3 0/3] libxl: memory management patches

[PATCH v3 0/3] libxl: memory management patches

[Daniel Kiper]

Hi,

Here are three small but important libxl/xl memory management patches:
  - libxl: xl mem-max et consortes must update static-max in xenstore too,
  - xl: Allow user to configure xl mem-set behavior,
  - xl: Improve xl documentation in regards to guest memory management.

Daniel

[PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too

[Daniel Kiper]

xl mem-max et consortes must update static-max in xenstore too.
Without this patch there is no chance to increase memory reservation
for given domain above memory limit defined in config file. It means
that memory hotplug is practicaly unusable without this patch.

Signed-off-by: Daniel Kiper <daniel.kiper@oracle.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad@kernel.org>
---
 tools/libxl/libxl.c |    3 +++
 1 file changed, 3 insertions(+)

diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
index 572c2c6..6f4cf09 100644
--- a/tools/libxl/libxl.c
+++ b/tools/libxl/libxl.c
@@ -3484,6 +3484,9 @@ int libxl_domain_setmaxmem(libxl_ctx *ctx, uint32_t domid, uint32_t max_memkb)
         goto out;
     }
 
+    libxl__xs_write(gc, XBT_NULL, libxl__sprintf(gc, "%s/memory/static-max",
+			    dompath), "%"PRIu32, max_memkb);
+
     rc = 0;
 out:
     GC_FREE;
-- 
1.7.10.4

[PATCH v3 2/3] xl: Allow user to configure xl mem-set behavior

[Daniel Kiper]

Add mem_set_enforce_limit option to xl.conf file and equivalent xl
command line option. Those two things give a chance to align xl mem-set
behavior to xm mem-set behavior. Default xl mem-set behavior is not changed.

v3 - suggestions/fixes:
   - add xl command line option
     (suggested by Ian Jackson).

Signed-off-by: Daniel Kiper <daniel.kiper@oracle.com>
---
 tools/libxl/xl.c          |    4 ++++
 tools/libxl/xl.h          |    1 +
 tools/libxl/xl_cmdimpl.c  |   16 ++++++++++------
 tools/libxl/xl_cmdtable.c |    3 ++-
 4 files changed, 17 insertions(+), 7 deletions(-)

diff --git a/tools/libxl/xl.c b/tools/libxl/xl.c
index 4c598db..5c445ca 100644
--- a/tools/libxl/xl.c
+++ b/tools/libxl/xl.c
@@ -38,6 +38,7 @@ xentoollog_logger_stdiostream *logger;
 int dryrun_only;
 int force_execution;
 int autoballoon = 1;
+int mem_set_enforce_limit = 1;
 char *blkdev_start;
 int run_hotplug_scripts = 1;
 char *lockfile;
@@ -72,6 +73,9 @@ static void parse_global_config(const char *configfile,
     if (!xlu_cfg_get_long (config, "autoballoon", &l, 0))
         autoballoon = l;
 
+    if (!xlu_cfg_get_long (config, "mem_set_enforce_limit", &l, 0))
+        mem_set_enforce_limit = l;
+
     if (!xlu_cfg_get_long (config, "run_hotplug_scripts", &l, 0))
         run_hotplug_scripts = l;
 
diff --git a/tools/libxl/xl.h b/tools/libxl/xl.h
index b881f92..321a0d0 100644
--- a/tools/libxl/xl.h
+++ b/tools/libxl/xl.h
@@ -143,6 +143,7 @@ int xl_child_pid(xlchildnum); /* returns 0 if child struct is not in use */
 
 /* global options */
 extern int autoballoon;
+extern int mem_set_enforce_limit;
 extern int run_hotplug_scripts;
 extern int dryrun_only;
 extern char *lockfile;
diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
index 2d40f8f..a0ff3e5 100644
--- a/tools/libxl/xl_cmdimpl.c
+++ b/tools/libxl/xl_cmdimpl.c
@@ -2495,7 +2495,7 @@ int main_memmax(int argc, char **argv)
     return 0;
 }
 
-static void set_memory_target(uint32_t domid, const char *mem)
+static void set_memory_target(uint32_t domid, const char *mem, int enforce_limit)
 {
     long long int memorykb;
 
@@ -2505,23 +2505,27 @@ static void set_memory_target(uint32_t domid, const char *mem)
         exit(3);
     }
 
-    libxl_set_memory_target(ctx, domid, memorykb, 0, /* enforce */ 1);
+    libxl_set_memory_target(ctx, domid, memorykb, 0, enforce_limit);
 }
 
 int main_memset(int argc, char **argv)
 {
     uint32_t domid;
-    int opt = 0;
+    int enforce_limit, opt = 0;
     const char *mem;
 
-    SWITCH_FOREACH_OPT(opt, "", NULL, "mem-set", 2) {
-        /* No options */
+    enforce_limit = mem_set_enforce_limit;
+
+    SWITCH_FOREACH_OPT(opt, "e:", NULL, "mem-set", 2) {
+    case 'e':
+        enforce_limit = atoi(optarg) ? 1 : 0;
+        break;
     }
 
     domid = find_domain(argv[optind]);
     mem = argv[optind + 1];
 
-    set_memory_target(domid, mem);
+    set_memory_target(domid, mem, enforce_limit);
     return 0;
 }
 
diff --git a/tools/libxl/xl_cmdtable.c b/tools/libxl/xl_cmdtable.c
index b4a87ca..5cfdb70 100644
--- a/tools/libxl/xl_cmdtable.c
+++ b/tools/libxl/xl_cmdtable.c
@@ -195,7 +195,8 @@ struct cmd_spec cmd_table[] = {
     { "mem-set",
       &main_memset, 0, 1,
       "Set the current memory usage for a domain",
-      "<Domain> <MemMB['b'[bytes]|'k'[KB]|'m'[MB]|'g'[GB]|'t'[TB]]>",
+      "[-e <0|1>] <Domain> <MemMB['b'[bytes]|'k'[KB]|'m'[MB]|'g'[GB]|'t'[TB]]>",
+      "-e <0|1> (Not)Enforce maximum memory allocation for a domain"
     },
     { "button-press",
       &main_button_press, 0, 1,
-- 
1.7.10.4

[PATCH v3 3/3] xl: Improve xl documentation in regards to guest memory management

[Daniel Kiper]

Improve xl documentation in regards to guest memory management.

Some improvements are suggested by Ian Jackson and Konrad Rzeszutek Wilk.

Signed-off-by: Daniel Kiper <daniel.kiper@oracle.com>
---
 docs/man/xl.conf.pod.5            |   15 +++++++++++++++
 docs/misc/libxl_memory.txt        |   38 +++++++++++++++++++++++--------------
 docs/misc/xenstore-paths.markdown |   15 +++++++--------
 tools/examples/xl.conf            |    4 ++++
 4 files changed, 50 insertions(+), 22 deletions(-)

diff --git a/docs/man/xl.conf.pod.5 b/docs/man/xl.conf.pod.5
index 7b9fcac..34d6a5d 100644
--- a/docs/man/xl.conf.pod.5
+++ b/docs/man/xl.conf.pod.5
@@ -55,6 +55,21 @@ reduce the amount of memory given to domain 0 by default.
 
 Default: C<1>
 
+=item B<mem_set_enforce_limit=BOOLEAN>
+
+If enabled, then C<xl mem-set> by default will set a memory allocation
+target and enforce the maximum memory allocation for a given domain.
+If disabled, then C<xl mem-set> by default will set a memory allocation
+target only for a given domain.
+
+The memory allocation target is a suggested amount of memory which
+should be allocated by a guest. It is nice when the guest follows this
+value but it may have its own idea about memory requirments. However,
+it could not allocate more memory than a ceiling known as the maximum
+memory allocation.
+
+Default: C<1>
+
 =item B<run_hotplug_scripts=BOOLEAN>
 
 If disabled hotplug scripts will be called from udev, as it used to
diff --git a/docs/misc/libxl_memory.txt b/docs/misc/libxl_memory.txt
index 253476d..f9b9a67 100644
--- a/docs/misc/libxl_memory.txt
+++ b/docs/misc/libxl_memory.txt
@@ -15,14 +15,19 @@
              |  |          |  |    |        | target  |          |            
              |  | guest    |  |    | build  |         |          |            
              |  |          |  |    | start  |         |          |            
-      static |  |          |  |    |        |         |          |            
-     maximum |  +----------+  |    +        +         +          +            
-             |  |          |  |                                               
-             |  |          |  |                                               
+             |  |          |  |    |        |         |          |            
+             |  +----------+  |    +        +         +          +            
+      static |  |          |  |                                               
+     maximum |  |          |  |                                               
              |  | balloon  |  | build                                         
              |  |          |  | maximum                                       
              |  |          |  |                                               
-             +  +----------+  +                                               
+             |  +----------+  +                                               
+             |  |          |
+             |  | memory   |
+             |  | hotplug  |
+             |  |          |
+             +  +----------+
                 
                 
     extra internal = LIBXL_MAXMEM_CONSTANT
@@ -31,8 +36,8 @@
     static maximum = libxl_domain_build_info.max_memkb
     video = libxl_domain_build_info.video_memkb
     build start = libxl_domain_build_info.target_memkb
-    libxl_domain_setmaxmem -> xen maximum
-    libxl_set_memory_target -> actual target
+    libxl_domain_setmaxmem -> xen maximum & static maximum
+    libxl_set_memory_target -> actual target & optionaly xen maximum
                 
                 
  === Domain memory breakdown: PV guests ==================================
@@ -49,22 +54,27 @@
              |  |          |  |    |        | actual  | maximum  | 
              |  | guest    |  |    | build  | target  |          |            
              |  |          |  |    | start  |         |          |            
-      static |  |          |  |    |        |         |          |            
-     maximum |  +----------+  |    +        +         +          +            
+             |  |          |  |    |        |         |          |            
+             |  +----------+  |    +        +         +          +            
              |  |          |  |                                               
-             |  |          |  |                                               
-             |  | balloon  |  | build                                         
+      static |  |          |  |                                               
+     maximum |  | balloon  |  | build                                         
              |  |          |  | maximum                                       
              |  |          |  |                                               
-             +  +----------+  +                                               
+             |  +----------+  +                                               
+             |  |          |
+             |  | memory   |
+             |  | hotplug  |
+             |  |          |
+             +  +----------+
                 
 
     extra internal = LIBXL_MAXMEM_CONSTANT
     extra external = LIBXL_PV_EXTRA_MEMORY
     static maximum = libxl_domain_build_info.max_memkb
     build start = libxl_domain_build_info.target_memkb
-    libxl_domain_setmaxmem -> xen maximum
-    libxl_set_memory_target -> actual target
+    libxl_domain_setmaxmem -> xen maximum & static maximum
+    libxl_set_memory_target -> actual target & optionaly xen maximum
 
 
  ========================================================================= */
diff --git a/docs/misc/xenstore-paths.markdown b/docs/misc/xenstore-paths.markdown
index 535830e..911d1ad 100644
--- a/docs/misc/xenstore-paths.markdown
+++ b/docs/misc/xenstore-paths.markdown
@@ -124,18 +124,17 @@ This protocol is not currently well documented.
 
 #### ~/memory/static-max = MEMKB []
 
-Specifies a static maximum amount memory which this domain should
-expect to be given. In the absence of in-guest memory hotplug support
-this set on domain boot and is usually the maximum amount of RAM which
-a guest can make use of. See [docs/misc/libxl_memory.txt][LIBXLMEM]
-for a description of how memory is accounted for in toolstacks using
-the libxl library.
+Specifies a static maximum amount of memory which this domain should
+expect to be given. It is usually the maximum amount of RAM which a guest
+can make use of. This value is set on the domain boot and can be changed by
+xl mem-max. See [docs/misc/libxl_memory.txt][LIBXLMEM] for a description
+of how the memory is accounted for in toolstacks using the libxl library.
 
 #### ~/memory/target = MEMKB []
 
 The current balloon target for the domain. The balloon driver within
-the guest is expected to make every effort to every effort use no more
-than this amount of RAM.
+the guest is expected to make every effort to use no more than this
+amount of RAM.
 
 #### ~/memory/videoram = MEMKB [HVM,INTERNAL]
 
diff --git a/tools/examples/xl.conf b/tools/examples/xl.conf
index b0caa32..c48caad 100644
--- a/tools/examples/xl.conf
+++ b/tools/examples/xl.conf
@@ -4,6 +4,10 @@
 # memory to create a domain
 #autoballoon=1
 
+# xl mem-set by default will set a memory allocation target and enforce
+# the maximum memory allocation for a given domain
+#mem_set_enforce_limit=1
+
 # full path of the lockfile used by xl during domain creation
 #lockfile="/var/lock/xl"
 
-- 
1.7.10.4

Re: [PATCH v3 3/3] xl: Improve xl documentation in regards to guest memory management

[Ian Jackson]

Daniel Kiper writes ("[Xen-devel] [PATCH v3 3/3] xl: Improve xl documentation in regards to guest memory management"):
> Improve xl documentation in regards to guest memory management.

Thanks.

> +=item B<mem_set_enforce_limit=BOOLEAN>
> +
> +If enabled, then C<xl mem-set> by default will set a memory allocation
> +target and enforce the maximum memory allocation for a given domain.
> +If disabled, then C<xl mem-set> by default will set a memory allocation
> +target only for a given domain.

I think you mean "for each domain" rather than "for a given domain",
and "will set only a memory allocation target for each domain" ?

> +The memory allocation target is a suggested amount of memory which
> +should be allocated by a guest. It is nice when the guest follows this
> +value but it may have its own idea about memory requirments. However,
> +it could not allocate more memory than a ceiling known as the maximum
> +memory allocation.

I would write "However the guest is prevented by Xen from allocating
more memory than a ceiling known as the "xen maximum" memory
allocation" or some such.

> --- a/docs/misc/libxl_memory.txt
> +++ b/docs/misc/libxl_memory.txt
> @@ -15,14 +15,19 @@
>               |  |          |  |    |        | target  |          | 
>               |  | guest    |  |    | build  |         |          | 
>               |  |          |  |    | start  |         |          | 
> -      static |  |          |  |    |        |         |          | 
> -     maximum |  +----------+  |    +        +         +          + 
> -             |  |          |  | 
> -             |  |          |  | 
> +             |  |          |  |    |        |         |          | 
> +             |  +----------+  |    +        +         +          + 
> +      static |  |          |  | 
> +     maximum |  |          |  | 
>               |  | balloon  |  | build 
>               |  |          |  | maximum 
>               |  |          |  | 
> -             +  +----------+  + 
> +             |  +----------+  + 
> +             |  |          |
> +             |  | memory   |
> +             |  | hotplug  |
> +             |  |          |
> +             +  +----------+

This change to the diagram implies that the target, and and the xen
maximum, never include any hotplug memory.  I don't think that can be
true.

Presumably hotplugged memory can be ballooned too.  So perhaps we need
another column ?

>  #### ~/memory/target = MEMKB []
>  
>  The current balloon target for the domain. The balloon driver within
> -the guest is expected to make every effort to every effort use no more
> -than this amount of RAM.
> +the guest is expected to make every effort to use no more than this
> +amount of RAM.

Good fix.

It might be worth adding:

  "Guests whose actual allocation exceeds the target may experience
   memory allocation failures."

Thanks,
Ian.

Re: [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too

[Ian Jackson]

Daniel Kiper writes ("[Xen-devel] [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too"):
> xl mem-max et consortes must update static-max in xenstore too.
> Without this patch there is no chance to increase memory reservation
> for given domain above memory limit defined in config file. It means
> that memory hotplug is practicaly unusable without this patch.

Doesn't this, together with your previous patch, conflate the "static
maximum" (ie, boot-time memory size which in the absence of memory
hotplug can never be exceeded), with the "xen maximum" (ie the
enforced memory limit) ?

I thought that the static-max xenstore key was used for the former.

Ian.

Re: [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too

[Daniel Kiper]

On Mon, Apr 08, 2013 at 05:21:27PM +0100, Ian Jackson wrote:
> Daniel Kiper writes ("[Xen-devel] [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too"):
> > xl mem-max et consortes must update static-max in xenstore too.
> > Without this patch there is no chance to increase memory reservation
> > for given domain above memory limit defined in config file. It means
> > that memory hotplug is practicaly unusable without this patch.
>
> Doesn't this, together with your previous patch, conflate the "static
> maximum" (ie, boot-time memory size which in the absence of memory
> hotplug can never be exceeded), with the "xen maximum" (ie the
> enforced memory limit) ?
>
> I thought that the static-max xenstore key was used for the former.

To some extent. However, static-max has always a bit larger value
than "xen maximum". xl uses static-max to enforce limits on guests
but it is just an info for guest itself. "xen maximum" is a kind
of hard limit which could not be exceeded and is enforced on guest
by Xen hypervisor.

Daniel

Re: [PATCH v3 3/3] xl: Improve xl documentation in regards to guest memory management

[Daniel Kiper]

On Mon, Apr 08, 2013 at 05:19:00PM +0100, Ian Jackson wrote:
> Daniel Kiper writes ("[Xen-devel] [PATCH v3 3/3] xl: Improve xl documentation in regards to guest memory management"):
> > Improve xl documentation in regards to guest memory management.
>
> Thanks.
>
> > +=item B<mem_set_enforce_limit=BOOLEAN>
> > +
> > +If enabled, then C<xl mem-set> by default will set a memory allocation
> > +target and enforce the maximum memory allocation for a given domain.
> > +If disabled, then C<xl mem-set> by default will set a memory allocation
> > +target only for a given domain.
>
> I think you mean "for each domain" rather than "for a given domain",
> and "will set only a memory allocation target for each domain" ?

Yes, thanks.

> > +The memory allocation target is a suggested amount of memory which
> > +should be allocated by a guest. It is nice when the guest follows this
> > +value but it may have its own idea about memory requirments. However,
> > +it could not allocate more memory than a ceiling known as the maximum
> > +memory allocation.
>
> I would write "However the guest is prevented by Xen from allocating
> more memory than a ceiling known as the "xen maximum" memory
> allocation" or some such.
>
> > --- a/docs/misc/libxl_memory.txt
> > +++ b/docs/misc/libxl_memory.txt
> > @@ -15,14 +15,19 @@
> >               |  |          |  |    |        | target  |          |
> >               |  | guest    |  |    | build  |         |          |
> >               |  |          |  |    | start  |         |          |
> > -      static |  |          |  |    |        |         |          |
> > -     maximum |  +----------+  |    +        +         +          +
> > -             |  |          |  |
> > -             |  |          |  |
> > +             |  |          |  |    |        |         |          |
> > +             |  +----------+  |    +        +         +          +
> > +      static |  |          |  |
> > +     maximum |  |          |  |
> >               |  | balloon  |  | build
> >               |  |          |  | maximum
> >               |  |          |  |
> > -             +  +----------+  +
> > +             |  +----------+  +
> > +             |  |          |
> > +             |  | memory   |
> > +             |  | hotplug  |
> > +             |  |          |
> > +             +  +----------+
>
> This change to the diagram implies that the target, and and the xen
> maximum, never include any hotplug memory.  I don't think that can be
> true.

Right, "xen maximum" should include hotplug memory. However, I think that
"actual target" should be split to "suggested target" and "actual target".
"suggested target" could be up to "static maximum". "actual target"
always include only video and guest memory (currently alloceted memory
by domain).

> Presumably hotplugged memory can be ballooned too.  So perhaps we need
> another column ?

I think that it is better to add comment that after
hotplugging memory could be ballooned down.

> >  #### ~/memory/target = MEMKB []
> >
> >  The current balloon target for the domain. The balloon driver within
> > -the guest is expected to make every effort to every effort use no more
> > -than this amount of RAM.
> > +the guest is expected to make every effort to use no more than this
> > +amount of RAM.
>
> Good fix.
>
> It might be worth adding:
>
>   "Guests whose actual allocation exceeds the target may experience
>    memory allocation failures."

... or "Guests which attempts to exceed the target may experience
memory allocation failures."

Daniel

Re: [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too

[Ian Jackson]

Daniel Kiper writes ("Re: [Xen-devel] [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too"):
> On Mon, Apr 08, 2013 at 05:21:27PM +0100, Ian Jackson wrote:
> > Doesn't this, together with your previous patch, conflate the "static
> > maximum" (ie, boot-time memory size which in the absence of memory
> > hotplug can never be exceeded), with the "xen maximum" (ie the
> > enforced memory limit) ?
> >
> > I thought that the static-max xenstore key was used for the former.
> 
> To some extent. However, static-max has always a bit larger value
> than "xen maximum". xl uses static-max to enforce limits on guests
> but it is just an info for guest itself. "xen maximum" is a kind
> of hard limit which could not be exceeded and is enforced on guest
> by Xen hypervisor.

The reason for xl using static-max is that in the absence of memory
hotplug, attempting to raise a guest above static-max will not work.
And this check takes effect in xl.

So that's not "enforcing a limit on guests".

Ian.

Re: [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too

[Daniel Kiper]

On Tue, Apr 09, 2013 at 02:10:53PM +0100, Ian Jackson wrote:
> Daniel Kiper writes ("Re: [Xen-devel] [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too"):
> > On Mon, Apr 08, 2013 at 05:21:27PM +0100, Ian Jackson wrote:
> > > Doesn't this, together with your previous patch, conflate the "static
> > > maximum" (ie, boot-time memory size which in the absence of memory
> > > hotplug can never be exceeded), with the "xen maximum" (ie the
> > > enforced memory limit) ?
> > >
> > > I thought that the static-max xenstore key was used for the former.
> >
> > To some extent. However, static-max has always a bit larger value
> > than "xen maximum". xl uses static-max to enforce limits on guests
> > but it is just an info for guest itself. "xen maximum" is a kind
> > of hard limit which could not be exceeded and is enforced on guest
> > by Xen hypervisor.
>
> The reason for xl using static-max is that in the absence of memory
> hotplug, attempting to raise a guest above static-max will not work.
> And this check takes effect in xl.

OK but now it is quiet difficult (or close to impossible) to know
in advance that a given guest supports memory hotplug or not. That
is why I think that static-max should be unconditionally changed or
the guest should write something in xenstore to inform that it supports
memory hotplug and relevant check should be waived.

> So that's not "enforcing a limit on guests".

OK, it should be phrased that xl checks that itself does not exceeds limits.

Daniel

Re: [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too

[Ian Jackson]

Daniel Kiper writes ("Re: [Xen-devel] [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too"):
> On Tue, Apr 09, 2013 at 02:10:53PM +0100, Ian Jackson wrote:
> > The reason for xl using static-max is that in the absence of memory
> > hotplug, attempting to raise a guest above static-max will not work.
> > And this check takes effect in xl.
> 
> OK but now it is quiet difficult (or close to impossible) to know
> in advance that a given guest supports memory hotplug or not.

Right.

> That is why I think that static-max should be unconditionally
> changed or the guest should write something in xenstore to inform
> that it supports memory hotplug and relevant check should be waived.

Right.  I don't see what any of that has to do with "xl mem-max"
though.  The purpose of xl mem-max is to update the guest's
enforcement limit.

I think it would be great for the guest to write something to xenstore
saying it supports memory hotplug.  How does the protocol work ?  The
guest writes "I support memory hotplug", and the tools then ignore
static-max and simply update the memory target with a new value ?  Do
the tools need to do anything else ?

Ian.

Re: [PATCH v3 3/3] xl: Improve xl documentation in regards to guest memory management

[Ian Jackson]

Daniel Kiper writes ("Re: [Xen-devel] [PATCH v3 3/3] xl: Improve xl documentation in regards to guest memory management"):
> On Mon, Apr 08, 2013 at 05:19:00PM +0100, Ian Jackson wrote:
> > Presumably hotplugged memory can be ballooned too.  So perhaps we need
> > another column ?
> 
> I think that it is better to add comment that after
> hotplugging memory could be ballooned down.

I'd prefer not to make the diagram wrong.  Perhaps it would be better
to annotate the "static maximum" to say that it might be increased by
memory hotplug.

> > It might be worth adding:
> >
> >   "Guests whose actual allocation exceeds the target may experience
> >    memory allocation failures."
> 
> ... or "Guests which attempts to exceed the target may experience
> memory allocation failures."

No, because a guest which is being asked to shrink will experience
memory allocation failures if it doesn't shrink monotonically.

Ian.

Re: [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too

[Daniel Kiper]

On Tue, Apr 09, 2013 at 03:18:10PM +0100, Ian Jackson wrote:
> Daniel Kiper writes ("Re: [Xen-devel] [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too"):
> > On Tue, Apr 09, 2013 at 02:10:53PM +0100, Ian Jackson wrote:
> > > The reason for xl using static-max is that in the absence of memory
> > > hotplug, attempting to raise a guest above static-max will not work.
> > > And this check takes effect in xl.
> >
> > OK but now it is quiet difficult (or close to impossible) to know
> > in advance that a given guest supports memory hotplug or not.
>
> Right.
>
> > That is why I think that static-max should be unconditionally
> > changed or the guest should write something in xenstore to inform
> > that it supports memory hotplug and relevant check should be waived.
>
> Right.  I don't see what any of that has to do with "xl mem-max"
> though.  The purpose of xl mem-max is to update the guest's
> enforcement limit.

...and static-max if you would like to use memory hotplug in real.
If not, then xl mem-set does not allow you to allocate more memory
to a guest than static-max.

> I think it would be great for the guest to write something to xenstore
> saying it supports memory hotplug.  How does the protocol work ?  The
> guest writes "I support memory hotplug", and the tools then ignore

More or less...

> static-max and simply update the memory target with a new value ?  Do
> the tools need to do anything else ?

I think that tools should not ignore static-max in any casy.
Memory target should not be never ever above static-max.

Daniel

Re: [PATCH v3 3/3] xl: Improve xl documentation in regards to guest memory management

[Daniel Kiper]

On Tue, Apr 09, 2013 at 03:33:58PM +0100, Ian Jackson wrote:
> Daniel Kiper writes ("Re: [Xen-devel] [PATCH v3 3/3] xl: Improve xl documentation in regards to guest memory management"):
> > On Mon, Apr 08, 2013 at 05:19:00PM +0100, Ian Jackson wrote:
> > > Presumably hotplugged memory can be ballooned too.  So perhaps we need
> > > another column ?
> >
> > I think that it is better to add comment that after
> > hotplugging memory could be ballooned down.
>
> I'd prefer not to make the diagram wrong.  Perhaps it would be better
> to annotate the "static maximum" to say that it might be increased by
> memory hotplug.

No, memory hotplug should not change "static maximum" in any case.
"static maximum" should define memory limit for a given guest.
If "static maximum" is above "build maximum" then memory hotplug
in the guest should be involved (if it is available).

> > > It might be worth adding:
> > >
> > >   "Guests whose actual allocation exceeds the target may experience
> > >    memory allocation failures."
> >
> > ... or "Guests which attempts to exceed the target may experience
> > memory allocation failures."
>
> No, because a guest which is being asked to shrink will experience
> memory allocation failures if it doesn't shrink monotonically.

IIRC if you shrink memory for a given guest and it does not have
memory balloon driver installed nothing happens. I did not tested
it but it would be crazy when host could steal random page from
the guest without its acceptance. It would lead to crash almost
immediately. That is why I wrote "attempts" because memory allocation
failures may happen when you increase reservation only.

Daniel

Re: [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too

[Ian Campbell]

On Tue, 2013-04-09 at 15:08 +0100, Daniel Kiper wrote:
> On Tue, Apr 09, 2013 at 02:10:53PM +0100, Ian Jackson wrote:
> > Daniel Kiper writes ("Re: [Xen-devel] [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too"):
> > > On Mon, Apr 08, 2013 at 05:21:27PM +0100, Ian Jackson wrote:
> > > > Doesn't this, together with your previous patch, conflate the "static
> > > > maximum" (ie, boot-time memory size which in the absence of memory
> > > > hotplug can never be exceeded), with the "xen maximum" (ie the
> > > > enforced memory limit) ?
> > > >
> > > > I thought that the static-max xenstore key was used for the former.
> > >
> > > To some extent. However, static-max has always a bit larger value
> > > than "xen maximum". xl uses static-max to enforce limits on guests
> > > but it is just an info for guest itself. "xen maximum" is a kind
> > > of hard limit which could not be exceeded and is enforced on guest
> > > by Xen hypervisor.
> >
> > The reason for xl using static-max is that in the absence of memory
> > hotplug, attempting to raise a guest above static-max will not work.
> > And this check takes effect in xl.
> 
> OK but now it is quiet difficult (or close to impossible) to know
> in advance that a given guest supports memory hotplug or not.

This is why I previously queried whether memory hotplug ought not to be
a conscious admin decision.

Re: [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too [and 1 more messages]

[Ian Jackson]

Daniel Kiper writes ("Re: [Xen-devel] [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too"):
> On Tue, Apr 09, 2013 at 03:18:10PM +0100, Ian Jackson wrote:
> > Daniel Kiper writes ("Re: [Xen-devel] [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too"):
> > > That is why I think that static-max should be unconditionally
> > > changed or the guest should write something in xenstore to inform
> > > that it supports memory hotplug and relevant check should be waived.
> >
> > Right.  I don't see what any of that has to do with "xl mem-max"
> > though.  The purpose of xl mem-max is to update the guest's
> > enforcement limit.
> 
> ...and static-max if you would like to use memory hotplug in real.
> If not, then xl mem-set does not allow you to allocate more memory
> to a guest than static-max.

You seem to be losing sight of the purpose of the static-max key in
xenstore.  It is for the benefit of the toolstack - and, indirectly,
the user - so that we know the maximum amount of memory we can ask the
guest to us.

Daniel Kiper writes ("Re: [Xen-devel] [PATCH v3 3/3] xl: Improve xl documentation in regards to guest memory management"):
> No, memory hotplug should not change "static maximum" in any case.

Why shouldn't it ?

> "static maximum" should define memory limit for a given guest.
> If "static maximum" is above "build maximum" then memory hotplug
> in the guest should be involved (if it is available).

If a guest supports memory hotplug, does it simply hotplug more memory
response to changes in target ?  Is there not a limit on how much
memory hotplug a guest can cope with ?

Perhaps the guest should advertise its actual maximum ?

> > No, because a guest which is being asked to shrink will experience
> > memory allocation failures if it doesn't shrink monotonically.
> 
> IIRC if you shrink memory for a given guest and it does not have
> memory balloon driver installed nothing happens.

Yes.

>   I did not tested it but it would be crazy when host could steal
> random page from the guest without its acceptance.

It can't steal memory from the guest.  But if the guest does any
operation which allocates memory, when it is above the limit, that
operation can fail.

So for example consider a guest with a balloon driver (but without
memory hotplug) which was booted with 1G and subsequently ballooned
down to 768M.

If the user says "xl mem-set blah 512M" then the limit and target are
both set to 512M.  If the guest's balloon driver, instead of shrinking
the guest as it is supposed to, tries to allieviate some memory
pressure inside the guest by allocating _more_ memory from Xen, that
allocation will fail.

Ian.

Re: [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too [and 1 more messages]

[George Dunlap]

On Wed, Apr 10, 2013 at 4:56 PM, Ian Jackson <Ian.Jackson@eu.citrix.com> wrote:
> Daniel Kiper writes ("Re: [Xen-devel] [PATCH v3 3/3] xl: Improve xl documentation in regards to guest memory management"):
>> No, memory hotplug should not change "static maximum" in any case.
>
> Why shouldn't it ?

FWIW, at least XenServer uses "static max" to mean "cannot be changed
except by rebooting".  It *also* uses *that* for HVM guests to say
"how much memory to advertize to the guest".  That is, if you boot a
VM with memory = X and static_max = Y, then it will put Y amount of
memory in the virtual e820 map, allocate X memory for the PoD, and
then rely on the balloon driver to balloon from Y down to X during
boot.

This is two slightly different meanings -- "how much to put in the
guest e820" and "how much maximum a guest is ever allowed to have".
The plain meaning of "static max" should be the second (i.e., can't be
changed dynamically at run time).  If we want a "max in the e820 at
build time" we should probably come up with a new term; and if "how
much the guest is ever allowed to have without rebooting" is not a
useful concept, we should probably just discard it.

 -George

Re: [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too

[Daniel Kiper]

On Wed, Apr 10, 2013 at 02:13:39PM +0100, Ian Campbell wrote:
> On Tue, 2013-04-09 at 15:08 +0100, Daniel Kiper wrote:
> > On Tue, Apr 09, 2013 at 02:10:53PM +0100, Ian Jackson wrote:
> > > Daniel Kiper writes ("Re: [Xen-devel] [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too"):
> > > > On Mon, Apr 08, 2013 at 05:21:27PM +0100, Ian Jackson wrote:
> > > > > Doesn't this, together with your previous patch, conflate the "static
> > > > > maximum" (ie, boot-time memory size which in the absence of memory
> > > > > hotplug can never be exceeded), with the "xen maximum" (ie the
> > > > > enforced memory limit) ?
> > > > >
> > > > > I thought that the static-max xenstore key was used for the former.
> > > >
> > > > To some extent. However, static-max has always a bit larger value
> > > > than "xen maximum". xl uses static-max to enforce limits on guests
> > > > but it is just an info for guest itself. "xen maximum" is a kind
> > > > of hard limit which could not be exceeded and is enforced on guest
> > > > by Xen hypervisor.
> > >
> > > The reason for xl using static-max is that in the absence of memory
> > > hotplug, attempting to raise a guest above static-max will not work.
> > > And this check takes effect in xl.
> >
> > OK but now it is quiet difficult (or close to impossible) to know
> > in advance that a given guest supports memory hotplug or not.
>
> This is why I previously queried whether memory hotplug ought not to be
> a conscious admin decision.

Sure but I do not understand how this is related to above.
Addtitionally, which admin do you mean? Guest or host?

Daniel

Re: [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too [and 1 more messages]

[Daniel Kiper]

On Wed, Apr 10, 2013 at 04:56:09PM +0100, Ian Jackson wrote:
> Daniel Kiper writes ("Re: [Xen-devel] [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too"):
> > On Tue, Apr 09, 2013 at 03:18:10PM +0100, Ian Jackson wrote:
> > > Daniel Kiper writes ("Re: [Xen-devel] [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too"):
> > > > That is why I think that static-max should be unconditionally
> > > > changed or the guest should write something in xenstore to inform
> > > > that it supports memory hotplug and relevant check should be waived.
> > >
> > > Right.  I don't see what any of that has to do with "xl mem-max"
> > > though.  The purpose of xl mem-max is to update the guest's
> > > enforcement limit.
> >
> > ...and static-max if you would like to use memory hotplug in real.
> > If not, then xl mem-set does not allow you to allocate more memory
> > to a guest than static-max.
>
> You seem to be losing sight of the purpose of the static-max key in
> xenstore.  It is for the benefit of the toolstack - and, indirectly,
> the user - so that we know the maximum amount of memory we can ask the
> guest to us.

OK, I understand that. However, static-max equal to maxmem makes sens
for a guest without memory hotplug. If you have memory hotplug and you
could not change static-max then this is an obstacle to use memory
hotplug in real. You could not raise target above static-max using
xl mem-set and you could not change target from guest because
of "xen maximum" too.

Now we have two options:
  - we could allow user to change static-max for a given guest by calling
    xl mem-max (my current solution); this way we change a bit meaning
    of static-max from "maximum amount of memory allowed for the guest
    (usualy all guest OS structures were prepared for this amount of memory
    but they do not need to be filled at boot time)" to "maximum amount
    of memory allowed for the guest at a given moment",
  - we could leave static-max as is and use "xen maximum" as "maximum
    amount of memory allowed for the guest at a given moment"; However,
    in this case comparison with static-max in libxl_set_memory_target()
    should be changed to comparison with "xen maximum".

[...]

> > > No, because a guest which is being asked to shrink will experience
> > > memory allocation failures if it doesn't shrink monotonically.
> >
> > IIRC if you shrink memory for a given guest and it does not have
> > memory balloon driver installed nothing happens.
>
> Yes.
>
> >   I did not tested it but it would be crazy when host could steal
> > random page from the guest without its acceptance.
>
> It can't steal memory from the guest.  But if the guest does any
> operation which allocates memory, when it is above the limit, that
> operation can fail.
>
> So for example consider a guest with a balloon driver (but without
> memory hotplug) which was booted with 1G and subsequently ballooned
> down to 768M.
>
> If the user says "xl mem-set blah 512M" then the limit and target are
> both set to 512M.  If the guest's balloon driver, instead of shrinking
> the guest as it is supposed to, tries to allieviate some memory
> pressure inside the guest by allocating _more_ memory from Xen, that
> allocation will fail.

I suppose it could be phrased like that:

Guests which attempts to exceed or exceeds the target may experience
memory allocation failures.

It takes in to account your case and my case when a guest attempts
to raise its allocation above target.

Daniel

Re: [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too [and 1 more messages]

[Daniel Kiper]

On Thu, Apr 11, 2013 at 11:15:15AM +0100, George Dunlap wrote:
> On Wed, Apr 10, 2013 at 4:56 PM, Ian Jackson <Ian.Jackson@eu.citrix.com> wrote:
> > Daniel Kiper writes ("Re: [Xen-devel] [PATCH v3 3/3] xl: Improve xl documentation in regards to guest memory management"):
> >> No, memory hotplug should not change "static maximum" in any case.
> >
> > Why shouldn't it ?
>
> FWIW, at least XenServer uses "static max" to mean "cannot be changed
> except by rebooting".  It *also* uses *that* for HVM guests to say
> "how much memory to advertize to the guest".  That is, if you boot a
> VM with memory = X and static_max = Y, then it will put Y amount of
> memory in the virtual e820 map, allocate X memory for the PoD, and
> then rely on the balloon driver to balloon from Y down to X during
> boot.
>
> This is two slightly different meanings -- "how much to put in the
> guest e820" and "how much maximum a guest is ever allowed to have".
> The plain meaning of "static max" should be the second (i.e., can't be
> changed dynamically at run time).  If we want a "max in the e820 at
> build time" we should probably come up with a new term; and if "how
> much the guest is ever allowed to have without rebooting" is not a
> useful concept, we should probably just discard it.

George, thanks for this clarification.

Daniel

Re: [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too

[Ian Campbell]

On Thu, 2013-04-11 at 12:20 +0100, Daniel Kiper wrote:
> On Wed, Apr 10, 2013 at 02:13:39PM +0100, Ian Campbell wrote:
> > On Tue, 2013-04-09 at 15:08 +0100, Daniel Kiper wrote:
> > > On Tue, Apr 09, 2013 at 02:10:53PM +0100, Ian Jackson wrote:
> > > > Daniel Kiper writes ("Re: [Xen-devel] [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too"):
> > > > > On Mon, Apr 08, 2013 at 05:21:27PM +0100, Ian Jackson wrote:
> > > > > > Doesn't this, together with your previous patch, conflate the "static
> > > > > > maximum" (ie, boot-time memory size which in the absence of memory
> > > > > > hotplug can never be exceeded), with the "xen maximum" (ie the
> > > > > > enforced memory limit) ?
> > > > > >
> > > > > > I thought that the static-max xenstore key was used for the former.
> > > > >
> > > > > To some extent. However, static-max has always a bit larger value
> > > > > than "xen maximum". xl uses static-max to enforce limits on guests
> > > > > but it is just an info for guest itself. "xen maximum" is a kind
> > > > > of hard limit which could not be exceeded and is enforced on guest
> > > > > by Xen hypervisor.
> > > >
> > > > The reason for xl using static-max is that in the absence of memory
> > > > hotplug, attempting to raise a guest above static-max will not work.
> > > > And this check takes effect in xl.
> > >
> > > OK but now it is quiet difficult (or close to impossible) to know
> > > in advance that a given guest supports memory hotplug or not.
> >
> > This is why I previously queried whether memory hotplug ought not to be
> > a conscious admin decision.
> 
> Sure but I do not understand how this is related to above.

Because an admin might know whether hotplug is (potentially) an option
for the domain.

> Addtitionally, which admin do you mean? Guest or host?

Host.

Ian.

Re: [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too [and 1 more messages]

[Ian Campbell]

On Thu, 2013-04-11 at 13:24 +0100, Daniel Kiper wrote:
> On Wed, Apr 10, 2013 at 04:56:09PM +0100, Ian Jackson wrote:
> > Daniel Kiper writes ("Re: [Xen-devel] [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too"):
> > > On Tue, Apr 09, 2013 at 03:18:10PM +0100, Ian Jackson wrote:
> > > > Daniel Kiper writes ("Re: [Xen-devel] [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too"):
> > > > > That is why I think that static-max should be unconditionally
> > > > > changed or the guest should write something in xenstore to inform
> > > > > that it supports memory hotplug and relevant check should be waived.
> > > >
> > > > Right.  I don't see what any of that has to do with "xl mem-max"
> > > > though.  The purpose of xl mem-max is to update the guest's
> > > > enforcement limit.
> > >
> > > ...and static-max if you would like to use memory hotplug in real.
> > > If not, then xl mem-set does not allow you to allocate more memory
> > > to a guest than static-max.
> >
> > You seem to be losing sight of the purpose of the static-max key in
> > xenstore.  It is for the benefit of the toolstack - and, indirectly,
> > the user - so that we know the maximum amount of memory we can ask the
> > guest to us.
> 
> OK, I understand that. However, static-max equal to maxmem makes sens
> for a guest without memory hotplug. If you have memory hotplug and you
> could not change static-max then this is an obstacle to use memory
> hotplug in real. You could not raise target above static-max using
> xl mem-set and you could not change target from guest because
> of "xen maximum" too.
> 
> Now we have two options:
>   - we could allow user to change static-max for a given guest by calling
>     xl mem-max (my current solution); this way we change a bit meaning
>     of static-max from "maximum amount of memory allowed for the guest
>     (usualy all guest OS structures were prepared for this amount of memory
>     but they do not need to be filled at boot time)" to "maximum amount
>     of memory allowed for the guest at a given moment",
>   - we could leave static-max as is and use "xen maximum" as "maximum
>     amount of memory allowed for the guest at a given moment"; However,
>     in this case comparison with static-max in libxl_set_memory_target()
>     should be changed to comparison with "xen maximum".

How does this stuff work for physical memory hotplug? Understanding that
might help us decide what admins expect (and is also directly relevant
to HVM memory hotplug).

In the e820 of a physical system memory which is actually present at
boot is obviously represented as E820_MEMORY, but how are the holes in
the memory map where DIMMs could subsequently be physically inserted
represented? Are they just "reserved" or is there a special "unpopulated
memory" type?

And on the PV kernel side how does this appear to the guest? If you boot
a "massively ballooned" guest (e.g. 1G out of max 1TB) does it
automatically switch to making the bulk of the difference a hotpluggable
region rather than a balloon region? What does "pre-ballooned" even mean
to a guest which supports memory hotplug?

Do the kernels support memory unplug?

Ian.

Re: [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too [and 1 more messages]

[Daniel Kiper]

On Thu, Apr 11, 2013 at 02:47:40PM +0100, Ian Campbell wrote:
> On Thu, 2013-04-11 at 13:24 +0100, Daniel Kiper wrote:

[...]

> > Now we have two options:
> >   - we could allow user to change static-max for a given guest by calling
> >     xl mem-max (my current solution); this way we change a bit meaning
> >     of static-max from "maximum amount of memory allowed for the guest
> >     (usualy all guest OS structures were prepared for this amount of memory
> >     but they do not need to be filled at boot time)" to "maximum amount
> >     of memory allowed for the guest at a given moment",
> >   - we could leave static-max as is and use "xen maximum" as "maximum
> >     amount of memory allowed for the guest at a given moment"; However,
> >     in this case comparison with static-max in libxl_set_memory_target()
> >     should be changed to comparison with "xen maximum".
>
> How does this stuff work for physical memory hotplug? Understanding that
> might help us decide what admins expect (and is also directly relevant
> to HVM memory hotplug).

Memory hotplug works in the same way in PV and HVM guest.

> In the e820 of a physical system memory which is actually present at
> boot is obviously represented as E820_MEMORY, but how are the holes in
> the memory map where DIMMs could subsequently be physically inserted
> represented? Are they just "reserved" or is there a special "unpopulated
> memory" type?

Region for hotplugged memory is not reserved in any way. Only
memory available at boot have e820 entries. After hotplugging
memory hardware places it at relevant address and informs system
about new memory and its config usualy via ACPI. System admin
must online new memory via sysfs interface.

Current memory hotplug implementation in balloon driver does
not use ACPI and establishes placement for new memory itself
(simply above max_pfn; algorithm is not perfect and it fails
in some cases; I am going to fix it). Other things work like
in physical case.

> And on the PV kernel side how does this appear to the guest? If you boot
> a "massively ballooned" guest (e.g. 1G out of max 1TB) does it
> automatically switch to making the bulk of the difference a hotpluggable
> region rather than a balloon region? What does "pre-ballooned" even mean
> to a guest which supports memory hotplug?

Guest is "massively ballooned". There is no automatic
change from balloon to memory hotplug.

> Do the kernels support memory unplug?

Linux Kernel supports memory unplug (hot remove) on baremetal.
In Xen guest case memory is ballooned down after memory hotplug.

Daniel

Re: [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too

[Daniel Kiper]

On Thu, Apr 11, 2013 at 02:35:53PM +0100, Ian Campbell wrote:
> On Thu, 2013-04-11 at 12:20 +0100, Daniel Kiper wrote:
> > On Wed, Apr 10, 2013 at 02:13:39PM +0100, Ian Campbell wrote:
> > > On Tue, 2013-04-09 at 15:08 +0100, Daniel Kiper wrote:
> > > > On Tue, Apr 09, 2013 at 02:10:53PM +0100, Ian Jackson wrote:
> > > > > Daniel Kiper writes ("Re: [Xen-devel] [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too"):
> > > > > > On Mon, Apr 08, 2013 at 05:21:27PM +0100, Ian Jackson wrote:
> > > > > > > Doesn't this, together with your previous patch, conflate the "static
> > > > > > > maximum" (ie, boot-time memory size which in the absence of memory
> > > > > > > hotplug can never be exceeded), with the "xen maximum" (ie the
> > > > > > > enforced memory limit) ?
> > > > > > >
> > > > > > > I thought that the static-max xenstore key was used for the former.
> > > > > >
> > > > > > To some extent. However, static-max has always a bit larger value
> > > > > > than "xen maximum". xl uses static-max to enforce limits on guests
> > > > > > but it is just an info for guest itself. "xen maximum" is a kind
> > > > > > of hard limit which could not be exceeded and is enforced on guest
> > > > > > by Xen hypervisor.
> > > > >
> > > > > The reason for xl using static-max is that in the absence of memory
> > > > > hotplug, attempting to raise a guest above static-max will not work.
> > > > > And this check takes effect in xl.
> > > >
> > > > OK but now it is quiet difficult (or close to impossible) to know
> > > > in advance that a given guest supports memory hotplug or not.
> > >
> > > This is why I previously queried whether memory hotplug ought not to be
> > > a conscious admin decision.
> >
> > Sure but I do not understand how this is related to above.
>
> Because an admin might know whether hotplug is (potentially) an option
> for the domain.
>
> > Addtitionally, which admin do you mean? Guest or host?
>
> Host.

I thought once that balloon driver with memory hotplug support
might store something in xenstore. Maybe it will be sufficient.

Daniel

Re: [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too [and 1 more messages]

[Daniel Kiper]

On Thu, Apr 11, 2013 at 04:28:43PM +0200, Daniel Kiper wrote:
> On Thu, Apr 11, 2013 at 02:47:40PM +0100, Ian Campbell wrote:
> > On Thu, 2013-04-11 at 13:24 +0100, Daniel Kiper wrote:
>
> [...]
>
> > > Now we have two options:
> > >   - we could allow user to change static-max for a given guest by calling
> > >     xl mem-max (my current solution); this way we change a bit meaning
> > >     of static-max from "maximum amount of memory allowed for the guest
> > >     (usualy all guest OS structures were prepared for this amount of memory
> > >     but they do not need to be filled at boot time)" to "maximum amount
> > >     of memory allowed for the guest at a given moment",
> > >   - we could leave static-max as is and use "xen maximum" as "maximum
> > >     amount of memory allowed for the guest at a given moment"; However,
> > >     in this case comparison with static-max in libxl_set_memory_target()
> > >     should be changed to comparison with "xen maximum".
> >
> > How does this stuff work for physical memory hotplug? Understanding that
> > might help us decide what admins expect (and is also directly relevant
> > to HVM memory hotplug).
>
> Memory hotplug works in the same way in PV and HVM guest.
>
> > In the e820 of a physical system memory which is actually present at
> > boot is obviously represented as E820_MEMORY, but how are the holes in
> > the memory map where DIMMs could subsequently be physically inserted
> > represented? Are they just "reserved" or is there a special "unpopulated
> > memory" type?
>
> Region for hotplugged memory is not reserved in any way. Only
> memory available at boot have e820 entries. After hotplugging
> memory hardware places it at relevant address and informs system
> about new memory and its config usualy via ACPI. System admin
> must online new memory via sysfs interface.
>
> Current memory hotplug implementation in balloon driver does
> not use ACPI and establishes placement for new memory itself
> (simply above max_pfn; algorithm is not perfect and it fails
> in some cases; I am going to fix it). Other things work like
> in physical case.
>
> > And on the PV kernel side how does this appear to the guest? If you boot
> > a "massively ballooned" guest (e.g. 1G out of max 1TB) does it
> > automatically switch to making the bulk of the difference a hotpluggable
> > region rather than a balloon region? What does "pre-ballooned" even mean
> > to a guest which supports memory hotplug?
>
> Guest is "massively ballooned". There is no automatic
> change from balloon to memory hotplug.
>
> > Do the kernels support memory unplug?
>
> Linux Kernel supports memory unplug (hot remove) on baremetal.
> In Xen guest case memory is ballooned down after memory hotplug.

Ian, any additional thoughts, comments, ...?
I would like to prepare next version of patch ASAP.
I hope it will be included in Xen 4.3.

Daniel

Re: [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too

[Daniel Kiper]

On Thu, Apr 11, 2013 at 04:32:39PM +0200, Daniel Kiper wrote:
> On Thu, Apr 11, 2013 at 02:35:53PM +0100, Ian Campbell wrote:
> > On Thu, 2013-04-11 at 12:20 +0100, Daniel Kiper wrote:
> > > On Wed, Apr 10, 2013 at 02:13:39PM +0100, Ian Campbell wrote:
> > > > On Tue, 2013-04-09 at 15:08 +0100, Daniel Kiper wrote:
> > > > > On Tue, Apr 09, 2013 at 02:10:53PM +0100, Ian Jackson wrote:
> > > > > > Daniel Kiper writes ("Re: [Xen-devel] [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too"):
> > > > > > > On Mon, Apr 08, 2013 at 05:21:27PM +0100, Ian Jackson wrote:
> > > > > > > > Doesn't this, together with your previous patch, conflate the "static
> > > > > > > > maximum" (ie, boot-time memory size which in the absence of memory
> > > > > > > > hotplug can never be exceeded), with the "xen maximum" (ie the
> > > > > > > > enforced memory limit) ?
> > > > > > > >
> > > > > > > > I thought that the static-max xenstore key was used for the former.
> > > > > > >
> > > > > > > To some extent. However, static-max has always a bit larger value
> > > > > > > than "xen maximum". xl uses static-max to enforce limits on guests
> > > > > > > but it is just an info for guest itself. "xen maximum" is a kind
> > > > > > > of hard limit which could not be exceeded and is enforced on guest
> > > > > > > by Xen hypervisor.
> > > > > >
> > > > > > The reason for xl using static-max is that in the absence of memory
> > > > > > hotplug, attempting to raise a guest above static-max will not work.
> > > > > > And this check takes effect in xl.
> > > > >
> > > > > OK but now it is quiet difficult (or close to impossible) to know
> > > > > in advance that a given guest supports memory hotplug or not.
> > > >
> > > > This is why I previously queried whether memory hotplug ought not to be
> > > > a conscious admin decision.
> > >
> > > Sure but I do not understand how this is related to above.
> >
> > Because an admin might know whether hotplug is (potentially) an option
> > for the domain.
> >
> > > Addtitionally, which admin do you mean? Guest or host?
> >
> > Host.
>
> I thought once that balloon driver with memory hotplug support
> might store something in xenstore. Maybe it will be sufficient.

Same as in previous email...

Daniel

Re: [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too [and 1 more messages]

[Ian Campbell]

On Tue, 2013-04-16 at 13:10 +0100, Daniel Kiper wrote:
> Ian, any additional thoughts, comments, ...?

Do you mean me or IanJ?

I think I'm still of the opinion that operations which may involve
memory hotplug on the guest side rather than just ballooning should be
explicit, at least at the libxl API layer. TBH I'm not sure what this
would look like since the current API doesn't really seem to suggest a
natural way to extend it like this.

At the xl layer I'm not so sure what the answer is but xl is easier to
change if we aren't happy with it.

Ian.

Re: [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too [and 1 more messages]

[Daniel Kiper]

On Tue, Apr 16, 2013 at 01:21:46PM +0100, Ian Campbell wrote:
> On Tue, 2013-04-16 at 13:10 +0100, Daniel Kiper wrote:
> > Ian, any additional thoughts, comments, ...?
>
> Do you mean me or IanJ?

You, but if IanJ has comments I am happy to hear them too...

> I think I'm still of the opinion that operations which may involve
> memory hotplug on the guest side rather than just ballooning should be
> explicit, at least at the libxl API layer. TBH I'm not sure what this
> would look like since the current API doesn't really seem to suggest a
> natural way to extend it like this.
>
> At the xl layer I'm not so sure what the answer is but xl is easier to
> change if we aren't happy with it.

I think that we should not care (from a host point of view) which mechanism
is used in a guest to increase its memory size. From the host point of view
in balloon case and in memory hotplug case we just allocate pages for the guest.
We do not care where they will be placed and when they will be used. That is it.

However, we should create a mechanism which allows the guest to allocate memory
above limit set at boot and not eat all host memory in uncontrolled manner
(e.g. using memory hotplug mechanism). Last is provided by xl now but former
is not. Currently there is no way to increase memory limit above memory limit
set at boot (maxmem). Now I think we should just decide which value should be
used as a cap reference: static-max or "xen maximum". Both are not perfect
and some recalculation is always needed to establish real cap internaly
in the guest.

Daniel

Re: [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too [and 1 more messages]

[Ian Jackson]

Daniel Kiper writes ("Re: [Xen-devel] [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too [and 1 more messages]"):
> I think that we should not care (from a host point of view) which
> mechanism is used in a guest to increase its memory size. From the
> host point of view in balloon case and in memory hotplug case we
> just allocate pages for the guest.  We do not care where they will
> be placed and when they will be used. That is it.

Yes, exactly.

> However, we should create a mechanism which allows the guest to
> allocate memory above limit set at boot and not eat all host memory
> in uncontrolled manner (e.g. using memory hotplug mechanism). Last
> is provided by xl now but former is not.

Yes.  That limit mechanism is the Xen memory limit, which by default
is set by xl mem-set and with your changes can be independently set
with xl mem-max.

> Currently there is no way to increase memory limit above memory
> limit set at boot (maxmem).

That's the wrong way to think about it.  Currently xl will allow
neither the target nor the xen maximum to be increased beyond the boot
maximum, because they think that won't work.

> Now I think we should just decide which value should be
> used as a cap reference: static-max or "xen maximum". Both are not
> perfect and some recalculation is always needed to establish real
> cap internaly in the guest.

I don't know what "cap reference" is.

It seems to me that if the guest could in principle do memory hotplug
without assistance from the toolstack, all that's needed is:

 * Guest advertises memory hotplug capability in xenstore
   (presumably it advertises a maximum figure, rather than
    just promising to be able to use any amount of memory, since
    most guests will be able to hotplug only a certain amount?)

 * When memory hotplug limit is advertised, libxl ignores static-max
   in xenstore in favour of the guest's advertised capability.
   xl mem-set (and if different, xl mem-max) can be told to set the
   guest's memory target (in xenstore), and the xen memory limit,
   to more than the boot time memory.  When this happens the guest's
   balloon/hotplug driver can hotplug memory appropriately to
   bring the guest's memory usage to the requested target.

Ian.