RE: [RFC][PATCH] 0/9 Populate-on-demand memory

["Tian, Kevin" <kevin.tian@intel.com>]

>From: George Dunlap
>Sent: Tuesday, December 23, 2008 8:55 PM
>BACKGROUND
>
>When non-PV domains boots, they typically read the e820 maps to
>determine how much memory they have, and then assume that much memory
>thereafter.  Memory requirements can be reduced using a balloon
>driver, but it cannot be increased past this initial value.

Isn't it also true for pv guest? Unless guest supports memory hotadd,
balloon driver always can't increase past the initial max memory. But
your patch is nice since more VMs can be created w/o below hard 
limitation at boot time.

>Currently, this means that a non-PV domain must be booted with the
>maximum amount of memory you want that VM every to be able to use.
>
>Populate-on-demand allows us to "boot ballooned", in the 
>following manner:
>* Mark the entire range of memory (memory_static_max aka maxmem) with
>a new p2m type, populate_on_demand, reporting memory_static_max in th
>e820 map.  No memory is allocated at this stage.
>* Allocate the "memory_dynamic_max" (aka "target") amount of memory
>for a "PoD cache".  This memory is kept on a separate list in the
>domain struct.
>* Boot the guest.
>* Populate the p2m table on-demand as it's accessed with pages from
>the PoD cache.
>* When the balloon driver loads, it inflates the balloon size to
>(maxmem - target), giving the memory back to Xen.  When this is
>accomplished, the "populate-on-demand" portion of boot is effectively
>finished.
>

Another tricky point could be with VT-d. If one guest page is used as 
DMA target before balloon driver is installed, and no early access on
that page (like start-of-day scrubber), then PoD action will not be triggered...
Not sure the possibility of such condition, but you may need to have 
some thought or guard on that. em... after more thinking, actually PoD 
pages may be alive even after balloon driver is installed. I guess before
coming up a solution you may add a check on whether target domain
has passthrough device to decide whether this feature is on on-the-fly.

PoD is anyhow a bit different from balloon driver, since the latter claims
ownership on ballooned pages which then will not be used as the DMA
target within guest.

>
>NB that this code is designed to work only in conjunction with a
>balloon driver.  If the balloon driver is not loaded, eventually all
>pages will be dirtied (non-zero), the emergency sweep will fail, and
>there will be no memory to back outstanding PoD pages.  When this
>happens, the domain will crash.

In that case, is it better to increase PoD target to configured max mem?
It looks uncomfortable to crash a domain just because some optimization
doesn't apply. :-)

Last, do you have any performance data on how this patch may impact
the boot process, or even some workload after login?

Thanks,
Kevin