PoD, 4.2, and current/maximum reservation

PoD, 4.2, and current/maximum reservation

[James Harper]

A user has pointed out a problem with GPLPV under Xen 4.2 when using PoD. I'm using the difference between XENMEM_maximum_reservation and XENMEM_current_reservation to tell me how much I should balloon down to account for PoD, but when the user has ballooned down to 1G (from 4Gb or 8GB), GPLPV logs the following:

13005008825593: XenPCI     XENMEM_maximum_reservation = 262400
13005008825593: XenPCI     XENMEM_current_reservation = 262136
13005008825609: XenPCI     Trying to give 1056 KB (1 MB) to Xen

What is the correct way to tell how much PoD memory there is under 4.2? Am I doing it wrong?

I balloon down as early as possible (before xenbus starts) to avoid windows going over its limit so I'm hoping I can determine the size of PoD memory just via hypercalls.

Thanks

James

Re: PoD, 4.2, and current/maximum reservation

[George Dunlap]

[-- Attachment #1.1: Type: text/plain, Size: 2283 bytes --]

On Mon, Feb 11, 2013 at 12:03 AM, James Harper <
james.harper@bendigoit.com.au> wrote:

> A user has pointed out a problem with GPLPV under Xen 4.2 when using PoD.
> I'm using the difference between XENMEM_maximum_reservation and
> XENMEM_current_reservation to tell me how much I should balloon down to
> account for PoD, but when the user has ballooned down to 1G (from 4Gb or
> 8GB), GPLPV logs the following:
>
> 13005008825593: XenPCI     XENMEM_maximum_reservation = 262400
> 13005008825593: XenPCI     XENMEM_current_reservation = 262136
> 13005008825609: XenPCI     Trying to give 1056 KB (1 MB) to Xen
>
> What is the correct way to tell how much PoD memory there is under 4.2? Am
> I doing it wrong?
>
> I balloon down as early as possible (before xenbus starts) to avoid
> windows going over its limit so I'm hoping I can determine the size of PoD
> memory just via hypercalls.
>

You shouldn't have to know anything specifically about PoD -- you should
just look at the balloon target for the guest written in xenstore.  The
idea was as much as possible for the toolstack and Xen to work together to
make it transparent to the balloon driver, in part because we expected to
be running legacy drivers.  The Citrix PV drivers don't do anything special
wrt PoD memory.  (Paul, please correct me if I'm wrong.)

WRT timing and xenbus, a couple of things:
* Windows does a scrub of all memory at start-of-day.  Especially on
multiple-vcpu systems, we were unable to start the balloon process early
enough to win the race against the scrubber, so we had to have ways of
"reclaiming" zeroed pages for the PoD pool.  What this means is that it's
not a matter of Windows *touching* memory, but of windows *dirtying* memory
that will lead to a problem.
* So there is a minimum amount of memory Windows needs to be able to make
it to the stage where the balloon driver can run.  When XenServer first
implemented DMC, the team did extensive testing to determine minimum values
above which Windows never crashed or hung, and (as I understand it) baked
those into the xapi toolstack as a "seatbelt" to prevent users from setting
the value too low.

Not sure if that helps in your particular case -- I think 1G was within the
limit, but I could be wrong.  Dave, any comments?

 -George

[-- Attachment #1.2: Type: text/html, Size: 2914 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: PoD, 4.2, and current/maximum reservation

[James Harper]

> 
> On Mon, Feb 11, 2013 at 12:03 AM, James Harper
> <james.harper@bendigoit.com.au> wrote:
> 
> 	A user has pointed out a problem with GPLPV under Xen 4.2 when
> using PoD. I'm using the difference between
> XENMEM_maximum_reservation and XENMEM_current_reservation to tell
> me how much I should balloon down to account for PoD, but when the user
> has ballooned down to 1G (from 4Gb or 8GB), GPLPV logs the following:
> 
> 	13005008825593: XenPCI     XENMEM_maximum_reservation =
> 262400
> 	13005008825593: XenPCI     XENMEM_current_reservation = 262136
> 	13005008825609: XenPCI     Trying to give 1056 KB (1 MB) to Xen
> 
> 	What is the correct way to tell how much PoD memory there is under
> 4.2? Am I doing it wrong?
> 
> 	I balloon down as early as possible (before xenbus starts) to avoid
> windows going over its limit so I'm hoping I can determine the size of PoD
> memory just via hypercalls.
> 
> 
> 
> You shouldn't have to know anything specifically about PoD -- you should just
> look at the balloon target for the guest written in xenstore.  The idea was as
> much as possible for the toolstack and Xen to work together to make it
> transparent to the balloon driver, in part because we expected to be running
> legacy drivers.  The Citrix PV drivers don't do anything special wrt PoD
> memory.  (Paul, please correct me if I'm wrong.)

So I should just balloon down to the current_reservation figure right?

> WRT timing and xenbus, a couple of things:
> 
> * Windows does a scrub of all memory at start-of-day.  Especially on multiple-
> vcpu systems, we were unable to start the balloon process early enough to
> win the race against the scrubber, so we had to have ways of "reclaiming"
> zeroed pages for the PoD pool.  What this means is that it's not a matter of
> Windows *touching* memory, but of windows *dirtying* memory that will
> lead to a problem.
> 
> * So there is a minimum amount of memory Windows needs to be able to
> make it to the stage where the balloon driver can run.  When XenServer first
> implemented DMC, the team did extensive testing to determine minimum
> values above which Windows never crashed or hung, and (as I understand it)
> baked those into the xapi toolstack as a "seatbelt" to prevent users from
> setting the value too low.
> 
> Not sure if that helps in your particular case -- I think 1G was within the limit,
> but I could be wrong.  Dave, any comments?
> 

I think I could go from 4GB down to 128MB reliably without crashing, although the resulting OS wasn't particularly usable :)

Thanks

James

Re: PoD, 4.2, and current/maximum reservation

[George Dunlap]

On 11/02/13 11:39, James Harper wrote:
>> On Mon, Feb 11, 2013 at 12:03 AM, James Harper
>> <james.harper@bendigoit.com.au> wrote:
>>
>> 	A user has pointed out a problem with GPLPV under Xen 4.2 when
>> using PoD. I'm using the difference between
>> XENMEM_maximum_reservation and XENMEM_current_reservation to tell
>> me how much I should balloon down to account for PoD, but when the user
>> has ballooned down to 1G (from 4Gb or 8GB), GPLPV logs the following:
>>
>> 	13005008825593: XenPCI     XENMEM_maximum_reservation =
>> 262400
>> 	13005008825593: XenPCI     XENMEM_current_reservation = 262136
>> 	13005008825609: XenPCI     Trying to give 1056 KB (1 MB) to Xen
>>
>> 	What is the correct way to tell how much PoD memory there is under
>> 4.2? Am I doing it wrong?
>>
>> 	I balloon down as early as possible (before xenbus starts) to avoid
>> windows going over its limit so I'm hoping I can determine the size of PoD
>> memory just via hypercalls.
>>
>>
>>
>> You shouldn't have to know anything specifically about PoD -- you should just
>> look at the balloon target for the guest written in xenstore.  The idea was as
>> much as possible for the toolstack and Xen to work together to make it
>> transparent to the balloon driver, in part because we expected to be running
>> legacy drivers.  The Citrix PV drivers don't do anything special wrt PoD
>> memory.  (Paul, please correct me if I'm wrong.)
> So I should just balloon down to the current_reservation figure right?

...I don't think so -- it looks like you're getting that from a 
hypercall, not from xenstore.  You want the normal balloon target value 
from xenstore.  (But I might be confused -- I'm not super familiar with 
the technical details of the ballooning codepath, just the overall 
principles.)

>> WRT timing and xenbus, a couple of things:
>>
>> * Windows does a scrub of all memory at start-of-day.  Especially on multiple-
>> vcpu systems, we were unable to start the balloon process early enough to
>> win the race against the scrubber, so we had to have ways of "reclaiming"
>> zeroed pages for the PoD pool.  What this means is that it's not a matter of
>> Windows *touching* memory, but of windows *dirtying* memory that will
>> lead to a problem.
>>
>> * So there is a minimum amount of memory Windows needs to be able to
>> make it to the stage where the balloon driver can run.  When XenServer first
>> implemented DMC, the team did extensive testing to determine minimum
>> values above which Windows never crashed or hung, and (as I understand it)
>> baked those into the xapi toolstack as a "seatbelt" to prevent users from
>> setting the value too low.
>>
>> Not sure if that helps in your particular case -- I think 1G was within the limit,
>> but I could be wrong.  Dave, any comments?
>>
> I think I could go from 4GB down to 128MB reliably without crashing, although the resulting OS wasn't particularly usable :)

I think what we found was that Windows required a certain amount of 
memory to *boot*.  If you gave it less than that, then either one of two 
things would happen:
* The processes dirtying RAM beat the balloon driver, and the guest 
crashed with a PoD-cache-empty error
* The balloon driver beat the processes dirtying ram, and the guest 
hung; presumably some service somewhere waiting for some memory to "free 
up".

In theory after boot you could reduce the memory below that threshold.  
But then what would you do if you needed to reboot the guest?  You'd 
have to go grab the memory from somewhere else, which is the situation 
we were trying to avoid with having PoD in the first place.  So the guy 
architecting DMC just made it a hard floor (again, as I understand it).

  -George

Re: PoD, 4.2, and current/maximum reservation

[Jan Beulich]

>>> On 11.02.13 at 12:40, George Dunlap <george.dunlap@eu.citrix.com> wrote:
> On 11/02/13 11:39, James Harper wrote:
>>> On Mon, Feb 11, 2013 at 12:03 AM, James Harper
>>> <james.harper@bendigoit.com.au> wrote:
>>>
>>> 	A user has pointed out a problem with GPLPV under Xen 4.2 when
>>> using PoD. I'm using the difference between
>>> XENMEM_maximum_reservation and XENMEM_current_reservation to tell
>>> me how much I should balloon down to account for PoD, but when the user
>>> has ballooned down to 1G (from 4Gb or 8GB), GPLPV logs the following:
>>>
>>> 	13005008825593: XenPCI     XENMEM_maximum_reservation =
>>> 262400
>>> 	13005008825593: XenPCI     XENMEM_current_reservation = 262136
>>> 	13005008825609: XenPCI     Trying to give 1056 KB (1 MB) to Xen
>>>
>>> 	What is the correct way to tell how much PoD memory there is under
>>> 4.2? Am I doing it wrong?
>>>
>>> 	I balloon down as early as possible (before xenbus starts) to avoid
>>> windows going over its limit so I'm hoping I can determine the size of PoD
>>> memory just via hypercalls.
>>>
>>>
>>>
>>> You shouldn't have to know anything specifically about PoD -- you should just
>>> look at the balloon target for the guest written in xenstore.  The idea was 
> as
>>> much as possible for the toolstack and Xen to work together to make it
>>> transparent to the balloon driver, in part because we expected to be running
>>> legacy drivers.  The Citrix PV drivers don't do anything special wrt PoD
>>> memory.  (Paul, please correct me if I'm wrong.)
>> So I should just balloon down to the current_reservation figure right?
> 
> ...I don't think so -- it looks like you're getting that from a 
> hypercall, not from xenstore.  You want the normal balloon target value 
> from xenstore.  (But I might be confused -- I'm not super familiar with 
> the technical details of the ballooning codepath, just the overall 
> principles.)

As James said, this ought to run without xenbus having initialized
(so it can be done as early as possible). Hence the need for doing
this via hypercalls. The "classic" Xen balloon driver does so too, btw.

Jan

Re: PoD, 4.2, and current/maximum reservation

[James Harper]

> >> So I should just balloon down to the current_reservation figure right?
> >
> > ...I don't think so -- it looks like you're getting that from a
> > hypercall, not from xenstore.  You want the normal balloon target value
> > from xenstore.  (But I might be confused -- I'm not super familiar with
> > the technical details of the ballooning codepath, just the overall
> > principles.)
> 
> As James said, this ought to run without xenbus having initialized
> (so it can be done as early as possible). Hence the need for doing
> this via hypercalls. The "classic" Xen balloon driver does so too, btw.
> 

So is my mistake simply that I'm assuming that 'maximum reservation' == 'total memory?

It occurs to me that I don't actually know how to get the total amount of memory from a windows driver... guess I'd better look it up!

James

Re: PoD, 4.2, and current/maximum reservation

[Paul Durrant]

> -----Original Message-----
> From: Jan Beulich [mailto:JBeulich@suse.com]
> Sent: 12 February 2013 09:54
> To: James Harper; George Dunlap
> Cc: Paul Durrant; Dave Scott; xen-devel@lists.xen.org
> Subject: Re: [Xen-devel] PoD, 4.2, and current/maximum reservation
> 
> >>> On 11.02.13 at 12:40, George Dunlap <george.dunlap@eu.citrix.com>
> wrote:
> > On 11/02/13 11:39, James Harper wrote:
> >>> On Mon, Feb 11, 2013 at 12:03 AM, James Harper
> >>> <james.harper@bendigoit.com.au> wrote:
> >>>
> >>> 	A user has pointed out a problem with GPLPV under Xen 4.2 when
> >>> using PoD. I'm using the difference between
> >>> XENMEM_maximum_reservation and XENMEM_current_reservation to
> tell
> >>> me how much I should balloon down to account for PoD, but when the
> user
> >>> has ballooned down to 1G (from 4Gb or 8GB), GPLPV logs the following:
> >>>
> >>> 	13005008825593: XenPCI     XENMEM_maximum_reservation =
> >>> 262400
> >>> 	13005008825593: XenPCI     XENMEM_current_reservation = 262136
> >>> 	13005008825609: XenPCI     Trying to give 1056 KB (1 MB) to Xen
> >>>
> >>> 	What is the correct way to tell how much PoD memory there is under
> >>> 4.2? Am I doing it wrong?
> >>>
> >>> 	I balloon down as early as possible (before xenbus starts) to avoid
> >>> windows going over its limit so I'm hoping I can determine the size of
> PoD
> >>> memory just via hypercalls.
> >>>
> >>>
> >>>
> >>> You shouldn't have to know anything specifically about PoD -- you should
> just
> >>> look at the balloon target for the guest written in xenstore.  The idea
> was
> > as
> >>> much as possible for the toolstack and Xen to work together to make it
> >>> transparent to the balloon driver, in part because we expected to be
> running
> >>> legacy drivers.  The Citrix PV drivers don't do anything special wrt PoD
> >>> memory.  (Paul, please correct me if I'm wrong.)
> >> So I should just balloon down to the current_reservation figure right?
> >
> > ...I don't think so -- it looks like you're getting that from a
> > hypercall, not from xenstore.  You want the normal balloon target value
> > from xenstore.  (But I might be confused -- I'm not super familiar with
> > the technical details of the ballooning codepath, just the overall
> > principles.)
> 
> As James said, this ought to run without xenbus having initialized
> (so it can be done as early as possible). Hence the need for doing
> this via hypercalls. The "classic" Xen balloon driver does so too, btw.
> 

We just use xenbus in the Citrix PV drivers and rely on the zero page sweep code in Xen to prevent a memory scrub from crashing the domain.

One problem with Windows can be SKU memory limits i.e. the OS cannot actually allocate from all the memory that is given to the domain so the target cannot necessarily be met.

  Paul

Re: PoD, 4.2, and current/maximum reservation

[James Harper]

> >
> > As James said, this ought to run without xenbus having initialized
> > (so it can be done as early as possible). Hence the need for doing
> > this via hypercalls. The "classic" Xen balloon driver does so too, btw.
> >
> 
> We just use xenbus in the Citrix PV drivers and rely on the zero page sweep
> code in Xen to prevent a memory scrub from crashing the domain.
> 

I was using the difference between current_reservation and maximum_reservation but maximum_reservation doesn't appear to return the value in 4.2 that it did in 4.1 (based on one user report). current_reservation still appears to though so as long as I figure out how much physical memory is in the system (harder than it seems but a requirement even if you are using xenbus) I should be back in business.

> One problem with Windows can be SKU memory limits i.e. the OS cannot
> actually allocate from all the memory that is given to the domain so the target
> cannot necessarily be met.
> 

Like giving memory=2GB, maxmem=6GB to an XP32 system and wondering why things don't work? I guess that's a case of don't do that :)

Windows has too many secrets.

James