[Xen-devel] reported memory usage does not match real memory usage

[Xen-devel] reported memory usage does not match real memory usage

[Olaf Hering]

[-- Attachment #1.1: Type: text/plain, Size: 1793 bytes --]

I was made aware of the fact that Xen apparently loses memory as soon as domUs are started. In my testing HVM domUs occupy much more memory than what was configured for them. The expected memory footprint for a PV domU seems to match the value in the domU config file.

My test host has 128GB and 8 cpus, dom0 is started with a fixed amount of memory. Before any domU is started, 1.4G are already "lost".

It seems for each HVM a certain amount of extra memory must be available. For a 100G HVM domU another 809M is required. With just 1 vcpu instead of 8 the amount of extra memory is reduced to 802M. With 32 vcpus it increases to 834M. Apparently each vcpu needs 1M extra memory.


Is there a formula to calculate that amount of extra memory, is this behavior documented somewhere?


Olaf

(XEN) System RAM: 131062MB (134208492kB)
 xl info | grep -i mem
total_memory           : 131062
free_memory            : 125551
xen_commandline        : loglvl=all guest_loglvl=all smt=1 console=com1 com1=57600 dom0_mem=4G

131062M - 125551M = 5511M used for just dom0
5511M - 4096M = 1415M lost?



pv domU, pvgrub2, mem=1024, vcpu=1
free_memory            : 124527
125551M - 124527M = 1024M, matches expectation


pv domU, pvgrub2, mem=65536, vcpu=8
free_memory            : 58990
124527M - 58990M = 65537M
65537M - 65536M = 1M extra?


fv domU, mem=32768, vcpu=8
free_memory            : 25957
58990M - 25957M = 33033M
33033M - 32768M =  256M extra?


stop all domUs
free_memory            : 125551


fv domU, mem=102400, vcpu=8
free_memory            : 22342
125551M - 22342M = 103209M
103209M - 102400M =  809M extra?


fv domU, mem=102400, vcpu=1
free_memory            : 22349
125551M - 22349M = 103202M
103202M - 102400M =  802M extra?


[-- Attachment #1.2: Digitale Signatur von OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] reported memory usage does not match real memory usage

[Olaf Hering]

[-- Attachment #1.1: Type: text/plain, Size: 1693 bytes --]

Am Wed, 12 Feb 2020 11:53:25 +0100
schrieb Olaf Hering <olaf@aepfle.de>:

> Is there a formula to calculate that amount of extra memory, is this behavior documented somewhere?


With the script below, the formula may look like this:
- each vcpu needs 1MB extra memory
- each GB of a HVM domU memory needs 8MB extra memory
- each HVM domU needs 2MB extra memory

I assume these 8MB per GB is needed for the EPT page tables.

In case this extra memory is indeed some obvious static value, it would be better to allocate it from the value specified in 'memory=' to make sure a domU uses (almost) exactly the value that was configured.

Olaf


domU='hvm'
free_memory='125551'
for memory in {1024..102400}
do
        test "$(( ${memory} % (4*1024) ))" = "0" || continue
        xl destroy "${domU}" &> /dev/null
        while test "`xl info | awk '/^free_memory/{print $3}'`" -lt "${free_memory}"
        do
                sleep 0.2
        done
        xl create -q -f '/netshare/domU.cfg' "name='${domU}'" "memory='${memory}'" "vcpus='1'"
        while sleep 0.1
        do
                state="`xl list "${domU}" | awk "/^"${domU}'/{print $5}'`"
                case "${state}" in
                        r?????) break ;;
                        ?b????) break ;;
                        *) ;;
                esac
        done
        actual_free_memory="`xl info | awk '/^free_memory/{print $3}'`"
        domU_used_memory="$(( ${free_memory} - ${actual_free_memory} ))"
        extra_memory="$(( ${domU_used_memory} - ${memory} ))"
        echo "${memory}/$((${memory}/1024)) ${domU_used_memory} ${extra_memory}: $((((${memory}/1024)*8)+2))"
done

[-- Attachment #1.2: Digitale Signatur von OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] reported memory usage does not match real memory usage

[Jan Beulich]

On 12.02.2020 15:22, Olaf Hering wrote:
> Am Wed, 12 Feb 2020 11:53:25 +0100
> schrieb Olaf Hering <olaf@aepfle.de>:
> 
>> Is there a formula to calculate that amount of extra memory, is this behavior documented somewhere?

See e.g. sh_min_allocation() and whatever its HAP counterpart is. But
this and hence ...

> With the script below, the formula may look like this:
> - each vcpu needs 1MB extra memory
> - each GB of a HVM domU memory needs 8MB extra memory
> - each HVM domU needs 2MB extra memory

... this is really an implementation details.

> I assume these 8MB per GB is needed for the EPT page tables.
> 
> In case this extra memory is indeed some obvious static value,
> it would be better to allocate it from the value specified in
> 'memory=' to make sure a domU uses (almost) exactly the value
> that was configured.

Well, the "memory=" is, afaik, supposed to (largely) represent
what the guest is going to "see" (not all of it may in fact be
visible through the E820 map the guest sees at boot, but that's
a different aspect). Along with it there also is
"shadow_memory=", but the implementation has always been growing
this value if it was too small, to avoid rendering guests
unstable. So switching to a model like you propose would likely
require a new guest config option (and then how to deal with
conflicts between old and new options would need to be spelled
out).

Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] reported memory usage does not match real memory usage

[Olaf Hering]

[-- Attachment #1.1: Type: text/plain, Size: 856 bytes --]

Am Thu, 13 Feb 2020 11:28:29 +0100
schrieb Jan Beulich <jbeulich@suse.com>:

> On 12.02.2020 15:22, Olaf Hering wrote:
> > With the script below, the formula may look like this:
> > - each vcpu needs 1MB extra memory
> > - each GB of a HVM domU memory needs 8MB extra memory
> > - each HVM domU needs 2MB extra memory  
> ... this is really an implementation details.


That might be true. But it is important to know such detail. Otherwise the value of "free_memory" becomes useless because it will be unknown if there are enough resources for the domU that is supposed to be started or received.

However, if the value of 'memory=' becomes the real footprint from which everything is allocated, future versions of Xen may need more extra memory. The domU may see suddenly less memory after migration, which is something unexpected.

Olaf

[-- Attachment #1.2: Digitale Signatur von OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel