what happens when a PoD page is touched?

what happens when a PoD page is touched?

[James Harper]

I'm finding that the time of boot and hibernation of Windows under
xen-4.0.2-rc3 when maxmem is set is a big problem - 40 seconds to
balloon down 512MB on my system. Hibernation is even worse with delays
of minutes or hours.

What happens when such a PoD page is touched? Does Xen or qemu handle
this?

Thanks

James

Re: what happens when a PoD page is touched?

[Tim Deegan]

At 11:16 +0100 on 15 May (1305458171), James Harper wrote:
> I'm finding that the time of boot and hibernation of Windows under
> xen-4.0.2-rc3 when maxmem is set is a big problem - 40 seconds to
> balloon down 512MB on my system. Hibernation is even worse with delays
> of minutes or hours.
> 
> What happens when such a PoD page is touched? Does Xen or qemu handle
> this?

Xen handles it.  When you touch a PoD page the frame is backed with a
fresh, zeroed page from the pool of PoD pages.  The slowdown comes when
the PoD pages run low; then Xen has to scan guest memory looking for 
pages that are all zeroes and reclaim them into the PoD pool.

This is all pretty unpleasant, as you can imagine.  The practical 
advice is:

- get your balloon driver loaded as soon as you possibly can, 
  so you can balloon down before the Windows page scrubber pointlessly 
  touches all of RAM (AIUI on SMP systems this is pretty hard; it may be
  that once we get UEFI firmware it will be eaiser);
- alloc pages to be ballooned using an interface that doesn't scrub them;
- don't be too aggressive about how much you overcommit. 

Ideally, once Hyper-V brings in an interface for dynamic memory
ballooning in guests, we can use that and avoid this whole rigmarole for
new windows version. 

CC'ing George, who knows this code best.  IIRC there were some tweaks to
this code for XenServer, which I hope are all now upstream.  Do you know
whether 4.1-testing has the full set or are they only in -unstable?

Cheers,

Tim.

-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, Xen Platform Team
Citrix Systems UK Ltd.  (Company #02937203, SL9 0BG)

RE: what happens when a PoD page is touched?

[Paul Durrant]

> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-
> bounces@lists.xensource.com] On Behalf Of Tim Deegan
> Sent: 16 May 2011 09:39
> To: James Harper
> Cc: George Dunlap; xen devel
> Subject: Re: [Xen-devel] what happens when a PoD page is touched?
> 
> At 11:16 +0100 on 15 May (1305458171), James Harper wrote:
> > I'm finding that the time of boot and hibernation of Windows under
> > xen-4.0.2-rc3 when maxmem is set is a big problem - 40 seconds to
> > balloon down 512MB on my system. Hibernation is even worse with
> delays
> > of minutes or hours.
> >

Are you sure that your hibernation slowdown is not caused by your frontend retrying failed block writes? Since hibernation knows nothing of what is allocated to the balloon it may try to dump ballooned out pages (possibly because they border pages containing useful info) which the storage backend will fail to grant map and the blkif request will be errored. If you retry then clearly you'll just get another error so if you have some sort of retry/timeout mechanism in your hiber driver it may will explain the slowness you are seeing.

  Paul

RE: what happens when a PoD page is touched?

[James Harper]

> 
> At 11:16 +0100 on 15 May (1305458171), James Harper wrote:
> > I'm finding that the time of boot and hibernation of Windows under
> > xen-4.0.2-rc3 when maxmem is set is a big problem - 40 seconds to
> > balloon down 512MB on my system. Hibernation is even worse with
delays
> > of minutes or hours.
> >
> > What happens when such a PoD page is touched? Does Xen or qemu
handle
> > this?
> 
> Xen handles it.  When you touch a PoD page the frame is backed with a
> fresh, zeroed page from the pool of PoD pages.  The slowdown comes
when
> the PoD pages run low; then Xen has to scan guest memory looking for
> pages that are all zeroes and reclaim them into the PoD pool.
> 
> This is all pretty unpleasant, as you can imagine.  The practical
> advice is:
> 
> - get your balloon driver loaded as soon as you possibly can,
>   so you can balloon down before the Windows page scrubber pointlessly
>   touches all of RAM (AIUI on SMP systems this is pretty hard; it may
be
>   that once we get UEFI firmware it will be eaiser);

Doing that pretty successfully. The only slowdown is when I balloon down
and have to zero the pages first (see below)

> - alloc pages to be ballooned using an interface that doesn't scrub
them;

That exists under Windows 2003 and newer (MmAllocatePagesForMdlEx with
the MM_DONT_ZERO_ALLOCATION flag), but not before that. But I thought
the pages had to be zeroed by us anyway as part of the contract? I tried
testing each page before freeing and the first 90% or so are already all
zeroes but the last few % of pages aren't necessarily.

> - don't be too aggressive about how much you overcommit.

:)

> Ideally, once Hyper-V brings in an interface for dynamic memory
> ballooning in guests, we can use that and avoid this whole rigmarole
for
> new windows version.
> 
> CC'ing George, who knows this code best.  IIRC there were some tweaks
to
> this code for XenServer, which I hope are all now upstream.  Do you
know
> whether 4.1-testing has the full set or are they only in -unstable?
> 

Is the delay only the first time a page is touched? If I allocate 1MB
worth of pages (I'm ballooning up/down in 1MB increments) and Windows
goes through and zero's the lot, will the sweep be invoked every time a
new page is hit (potentially just reusing one of the previously zeroed
pages)?

Thanks

James

RE: what happens when a PoD page is touched?

[Paul Durrant]

> -----Original Message-----
[snip]
> 
> Ideally, once Hyper-V brings in an interface for dynamic memory
> ballooning in guests, we can use that and avoid this whole rigmarole
> for new windows version.
> 

Yes, it would be nice if newer viridian-aware kernels did not zero-scrub memory at start of day. This was certainly not what we observed with original windows 7 kernels, which relied on the zero-scrubbing having been done where older kernels did not (particularly when setting up pagetables IIRC). That pre-dates ballooning in hyper-V of course so I guess it'd be worth examining the behaviour of an SP1 system at start of day.

  Paul

RE: what happens when a PoD page is touched?

[James Harper]

> > -----Original Message-----
> > From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-
> > bounces@lists.xensource.com] On Behalf Of Tim Deegan
> > Sent: 16 May 2011 09:39
> > To: James Harper
> > Cc: George Dunlap; xen devel
> > Subject: Re: [Xen-devel] what happens when a PoD page is touched?
> >
> > At 11:16 +0100 on 15 May (1305458171), James Harper wrote:
> > > I'm finding that the time of boot and hibernation of Windows under
> > > xen-4.0.2-rc3 when maxmem is set is a big problem - 40 seconds to
> > > balloon down 512MB on my system. Hibernation is even worse with
> > delays
> > > of minutes or hours.
> > >
> 
> Are you sure that your hibernation slowdown is not caused by your
frontend
> retrying failed block writes? Since hibernation knows nothing of what
is
> allocated to the balloon it may try to dump ballooned out pages
(possibly
> because they border pages containing useful info) which the storage
backend
> will fail to grant map and the blkif request will be errored. If you
retry
> then clearly you'll just get another error so if you have some sort of
> retry/timeout mechanism in your hiber driver it may will explain the
slowness
> you are seeing.
> 

I'm almost absolutely positive. What you are describing happens during
crash dumps and I allow for a small number of such errors (although not
enough for ballooned domains probably), but the hibernation file is
compressed so pages are never actually written directly, only the
compression buffers. I imagine that the delay is while it sweeps the
ballooned out area - the hibernate progress never appears to increases
during this time because 3GB of 0's most likely compresses remarkably
well and there is nothing to write to disk

James

RE: what happens when a PoD page is touched?

[James Harper]

> > -----Original Message-----
> [snip]
> >
> > Ideally, once Hyper-V brings in an interface for dynamic memory
> > ballooning in guests, we can use that and avoid this whole rigmarole
> > for new windows version.
> >
> 
> Yes, it would be nice if newer viridian-aware kernels did not
zero-scrub
> memory at start of day. This was certainly not what we observed with
original
> windows 7 kernels, which relied on the zero-scrubbing having been done
where
> older kernels did not (particularly when setting up pagetables IIRC).
That
> pre-dates ballooning in hyper-V of course so I guess it'd be worth
examining
> the behaviour of an SP1 system at start of day.
> 

I've not seen any slowdown on boot with any version of Windows until I
go to actually do the balloon down...

James

RE: what happens when a PoD page is touched?

[Paul Durrant]

> -----Original Message-----
> 
> I've not seen any slowdown on boot with any version of Windows until
> I go to actually do the balloon down...
> 

In that case I guess your slowdown is caused by touching the pages. IIRC you will encounter a sweep every time you touch a pod page and the cache is exhausted. The sweep should reap all zeroed pages and hopefully fill the cache so the sweep should not be invoked too often. Try using the non-touching allocator.

  Paul

Re: what happens when a PoD page is touched?

[Tim Deegan]

At 10:17 +0100 on 16 May (1305541030), James Harper wrote:
> > - alloc pages to be ballooned using an interface that doesn't scrub
> them;
> 
> That exists under Windows 2003 and newer (MmAllocatePagesForMdlEx with
> the MM_DONT_ZERO_ALLOCATION flag), but not before that. But I thought
> the pages had to be zeroed by us anyway as part of the contract? I tried
> testing each page before freeing and the first 90% or so are already all
> zeroes but the last few % of pages aren't necessarily.

You're not _required_ to zero them, but Xen won't scrub them before
giving them to another VM so if you care about the contents you need to
scrub them yourself.

We could add a flag to decrease_reservation to ask Xen to scrub the
pages for you (on your time, obviously).  That way Xen would be able to
DTRT with PoD pages instead of populating them, zeroing them and
removing them again. 

> Is the delay only the first time a page is touched? If I allocate 1MB
> worth of pages (I'm ballooning up/down in 1MB increments) and Windows
> goes through and zero's the lot, will the sweep be invoked every time a
> new page is hit (potentially just reusing one of the previously zeroed
> pages)?

AIUI the sweep is invoked when you run out of PoD pages, so not every
time, but more frequently as you use up more of your memory.

Tim.

-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, Xen Platform Team
Citrix Systems UK Ltd.  (Company #02937203, SL9 0BG)

Re: what happens when a PoD page is touched?

[George Dunlap]

On Mon, May 16, 2011 at 10:22 AM, James Harper
<james.harper@bendigoit.com.au> wrote:
> I've not seen any slowdown on boot with any version of Windows until I
> go to actually do the balloon down...

You mean, you don't see slowness on boot unless you boot with maxmem
!= memory (i.e,. in PoD mode)?

For one thing, after the balloon driver is finished with its initial
allocation, there shouldn't be any sweeping; if there is, then it may
be that you're not actually inflating the balloon as much as Xen
thinks you should.

Before the balloon driver is done, the emergency sweep could be an
issue.  We had a bunch of stuff in XenServer in our most recent
release to try to mitigate this, but it was all pretty hacky and
unsuitable for upstreaming.  I determined that it just needs to be
re-written in a way that actually made things Better; I haven't gotten
a chance to do that yet.

It looks like ATM the sweep is limited to doing 2MiB at a time; so if
you have a reasonably sized guest, every other superpage that gets hit
will cause another sweep -- potentially a big problem.

I have a patch that attempts to greedily re-grab pages after a sweep;
I'll check to see if it applies to tip, and if so, I'll send it to you
to see if that helps any.

 -George

RE: what happens when a PoD page is touched?

[James Harper]

> 
> > -----Original Message-----
> >
> > I've not seen any slowdown on boot with any version of Windows until
> > I go to actually do the balloon down...
> >
> 
> In that case I guess your slowdown is caused by touching the pages.
IIRC you
> will encounter a sweep every time you touch a pod page and the cache
is
> exhausted. The sweep should reap all zeroed pages and hopefully fill
the cache
> so the sweep should not be invoked too often. Try using the
non-touching
> allocator.
> 

So avoiding the sweep is the thing to do then?

Where does the sweep start? Does it sweep in physical address order from
lowest address to highest? If I allocated some memory with a low
physical address and zero'd it and then didn't touch it would that be
found first?

Alternatively, I balloon down 1MB of memory at a time - if I could set
aside 1MB of memory that was filled with 0's and could somehow tell xen
to use that memory first then it might speed things up too yes?

Thanks

James

RE: what happens when a PoD page is touched?

[James Harper]

> 
> On Mon, May 16, 2011 at 10:22 AM, James Harper
> <james.harper@bendigoit.com.au> wrote:
> > I've not seen any slowdown on boot with any version of Windows until
I
> > go to actually do the balloon down...
> 
> You mean, you don't see slowness on boot unless you boot with maxmem
> != memory (i.e,. in PoD mode)?

Not quite what I mean. If I boot with memory=512 and maxmem=4100, there
are no performance problems I have noticed until my balloon down driver
starts returning memory to xen very early in boot. In allocating this
memory, Windows zero's it before it gives it to me which means I am
touching 3.5GB of memory and presumably invoking the emergency sweep for
pretty much every page.

> 
> For one thing, after the balloon driver is finished with its initial
> allocation, there shouldn't be any sweeping; if there is, then it may
> be that you're not actually inflating the balloon as much as Xen
> thinks you should.
> 
> Before the balloon driver is done, the emergency sweep could be an
> issue.  We had a bunch of stuff in XenServer in our most recent
> release to try to mitigate this, but it was all pretty hacky and
> unsuitable for upstreaming.  I determined that it just needs to be
> re-written in a way that actually made things Better; I haven't gotten
> a chance to do that yet.

Yes it sounds like that might be the case.

> 
> It looks like ATM the sweep is limited to doing 2MiB at a time; so if
> you have a reasonably sized guest, every other superpage that gets hit
> will cause another sweep -- potentially a big problem.
> 
> I have a patch that attempts to greedily re-grab pages after a sweep;
> I'll check to see if it applies to tip, and if so, I'll send it to you
> to see if that helps any.
> 

Thanks

James

RE: what happens when a PoD page is touched?

[Paul Durrant]

> -----Original Message-----
> From: James Harper [mailto:james.harper@bendigoit.com.au]
> Sent: 17 May 2011 00:40
> To: Paul Durrant; Tim Deegan
> Cc: George Dunlap; xen devel
> Subject: RE: [Xen-devel] what happens when a PoD page is touched?
> 
> >
> > > -----Original Message-----
> > >
> > > I've not seen any slowdown on boot with any version of Windows
> until
> > > I go to actually do the balloon down...
> > >
> >
> > In that case I guess your slowdown is caused by touching the
> pages.
> IIRC you
> > will encounter a sweep every time you touch a pod page and the
> cache
> is
> > exhausted. The sweep should reap all zeroed pages and hopefully
> fill
> the cache
> > so the sweep should not be invoked too often. Try using the
> non-touching
> > allocator.
> >
> 
> So avoiding the sweep is the thing to do then?
> 

Yes. You clearly want to avoid the overhead of a sweep to populate pages that you're immediately going to hand back to Xen.

> Where does the sweep start? Does it sweep in physical address order
> from lowest address to highest? If I allocated some memory with a
> low physical address and zero'd it and then didn't touch it would
> that be found first?
> 

I can't remember off the top of my head. I think we sweep from low to high.

> Alternatively, I balloon down 1MB of memory at a time - if I could
> set aside 1MB of memory that was filled with 0's and could somehow
> tell xen to use that memory first then it might speed things up too
> yes?
> 

Why can't you just use an allocator that doesn't touch memory in the majority of cases. MmAllocatePagesForMdlEx() is available post 2k3-sp1 so it's really only XP that would be suffering sweeps anyway and you may be able to mitigate that by ballooning down in smaller chunks such that you fill the PoD cache just enough to avoid a sweep during in the next bunch of allocations.

  Paul

RE: what happens when a PoD page is touched?

[James Harper]

 
> > Alternatively, I balloon down 1MB of memory at a time - if I could
> > set aside 1MB of memory that was filled with 0's and could somehow
> > tell xen to use that memory first then it might speed things up too
> > yes?
> >
> 
> Why can't you just use an allocator that doesn't touch memory in the
majority
> of cases. MmAllocatePagesForMdlEx() is available post 2k3-sp1 so it's
really
> only XP that would be suffering sweeps anyway

I can't guarantee that all the pages I hand back are clean, and testing
shows that a small number of them aren't. In theory, anything that cares
about the data in its pages (eg an encrypted FS) would have cleaned them
before handing them back to windows but I'd rather clean them first.

Is there a way to tell if a page is currently populated? That would
allow me to only clean populated pages.

> and you may be able to mitigate
> that by ballooning down in smaller chunks such that you fill the PoD
cache
> just enough to avoid a sweep during in the next bunch of allocations.

I still can't quite get my head around why this happens at all... I
thought it would go like this:

1. Allocate 1MB of memory
2. Still under our limit so xen populates the pages when Windows clears
them
3. Hand them back to xen
4. Repeat

I'm doing that in a tight loop very early in boot. If I keep handing
back pages (and thus reducing my populated page count) why am I hitting
any PoD limit at all and invoking the page scavenging code? Windows
isn't doing anything else at this point, and even if it was, I'm the
boot driver so it has to wait for me before the boot can progress so
it's not like it would be consuming gigabytes of memory.

Thanks

James

Re: what happens when a PoD page is touched?

[George Dunlap]

On Tue, May 17, 2011 at 10:28 AM, Paul Durrant <Paul.Durrant@citrix.com> wrote:
>> Where does the sweep start? Does it sweep in physical address order
>> from lowest address to highest? If I allocated some memory with a
>> low physical address and zero'd it and then didn't touch it would
>> that be found first?
>>
>
> I can't remember off the top of my head. I think we sweep from low to high.

Just checked: we sweep high to low.  I believe that's because that's
the way the Windows scrubber goes.

 -George

Re: what happens when a PoD page is touched?

[George Dunlap]

On Tue, May 17, 2011 at 10:37 AM, James Harper
<james.harper@bendigoit.com.au> wrote:
> I still can't quite get my head around why this happens at all... I
> thought it would go like this:
>
> 1. Allocate 1MB of memory
> 2. Still under our limit so xen populates the pages when Windows clears
> them
> 3. Hand them back to xen
> 4. Repeat
>
> I'm doing that in a tight loop very early in boot. If I keep handing
> back pages (and thus reducing my populated page count) why am I hitting
> any PoD limit at all and invoking the page scavenging code? Windows
> isn't doing anything else at this point, and even if it was, I'm the
> boot driver so it has to wait for me before the boot can progress so
> it's not like it would be consuming gigabytes of memory.

Suppose for simplicity we're using 1G pages and not splintering them.

Suppose we boot a VM with 8G maxmem, 4G memory.

* Domain built
pages 0-7 are PoD, 4 pages in PoD pool.

* HVMLoader loaded into memory
Page 0 populated; 1-7 PoD, 3 pages in PoD pool

* Windows boots, starts scrubbing (from top of memory)
 + page 7 populated, 2 pages in PoD pool.
 + Page 6 populated, 1 page in PoD pool
 + Page 5 populated, 0 pages in PoD pool
 + Page 4 touched
  - emergency sweep marks page 7 PoD; page 4 populated
 + Page 3 touched
  - emergency sweep marks page 6 PoD; page 3 populated
 etc
 + page 1 touched
  - emergency sweep marks page 5 PoD; page 1 populated
 + Result at the end: pages 0-4 populated, 5-7 PoD, no pages in PoD pool


* Balloon driver starts.
 Let's suppose Windows happens to do all allocation from the PoD region.
 + Asks windows for 1 page
  - Windows chooses page 6, scrubs it.
  - emergency sweep marks page 4 PoD, populates page 6
  - Balloon driver hands page 6 back to Xen
  - page 5 marked empty; page put in PoD pool
 + Asks Windows for 1 more page
  - Windows chooses page 5, scrubs it.
  - Xen fills page 5 with the page in the PoD pool; PoD now empty
  - Balloon driver hands page 5 back to Xen
  - page 5 marked empty; page put in PoD pool
 + Repeat previous

So in theory, once you get your initial allocation from Windows, you
shouldn't have to do any sweeps.

The exception to this would be the case where you're allocating N
pages, where N > (used_pages - total allocation).

I guess try allocating fewer pages before returning them to Xen, and
seeing if that helps at all.

Hmm -- looks like I didn't upstream my PoD tracing patch, otherwise we
could see what was going on.  If you're interested in taking a look at
what's happening at a PoD level, I can try to dig it up and send it to
you.  (Probably need some porting because of the p2m restructuring Tim
Deegan has been doing.)

Peace,
 -George

RE: what happens when a PoD page is touched?

[James Harper]

> The exception to this would be the case where you're allocating N
> pages, where N > (used_pages - total allocation).
> 
> I guess try allocating fewer pages before returning them to Xen, and
> seeing if that helps at all.

I tried returning 64KB at a time (eg 16 pages) instead of 1MB but it
doesn't improve things. Xen should just repopulate every page I allocate
with the pages I just returned and things should fly along but it's not
working that way.

Just taking a step back for a sec, maybe I'm doing something else wrong.
My balloon down code allocates memory from Windows with
MmAllocatePagesForMdlEx then hands the pages to Xen with
XENMEM_decrease_reservation. Is that all I need to do? I notice that
Linux makes a call to set_phys_to_machine(INVALID_P2M_ENTRY) for each
page, even in the hvm case. Is that some function I need to mirror too
or is that just internal Linux housekeeping and not required under
Windows?

Thanks

James

RE: what happens when a PoD page is touched?

[James Harper]

> 
> > The exception to this would be the case where you're allocating N
> > pages, where N > (used_pages - total allocation).
> >
> > I guess try allocating fewer pages before returning them to Xen, and
> > seeing if that helps at all.
> 
> I tried returning 64KB at a time (eg 16 pages) instead of 1MB but it
> doesn't improve things. Xen should just repopulate every page I
allocate
> with the pages I just returned and things should fly along but it's
not
> working that way.
> 
> Just taking a step back for a sec, maybe I'm doing something else
wrong.
> My balloon down code allocates memory from Windows with
> MmAllocatePagesForMdlEx then hands the pages to Xen with
> XENMEM_decrease_reservation. Is that all I need to do? I notice that
> Linux makes a call to set_phys_to_machine(INVALID_P2M_ENTRY) for each
> page, even in the hvm case. Is that some function I need to mirror too
> or is that just internal Linux housekeeping and not required under
> Windows?
> 

Hmmm... I think I'm chasing the wrong problem here. I measured time
taken to allocate the memory, and then time taken to decrease
reservation, and the decrease reservation time is 6x longer than the
memory allocation time (as measured by the
TSC/KeQueryPerformanceCounter). That approximately equates to 18 seconds
to allocate and scrub 3.5GB of memory, and 111 seconds in calls to
decrease reservation. This measurement is consistent across a few boots
so I assume it's not an artefact of using the TSC under a virtualised
environment...

James

RE: what happens when a PoD page is touched?

[Jan Beulich]

>>> On 18.05.11 at 06:52, "James Harper" <james.harper@bendigoit.com.au> wrote:
> Just taking a step back for a sec, maybe I'm doing something else wrong.
> My balloon down code allocates memory from Windows with
> MmAllocatePagesForMdlEx then hands the pages to Xen with
> XENMEM_decrease_reservation. Is that all I need to do? I notice that
> Linux makes a call to set_phys_to_machine(INVALID_P2M_ENTRY) for each
> page, even in the hvm case. Is that some function I need to mirror too
> or is that just internal Linux housekeeping and not required under
> Windows?

Probably not: In the HVM case this function degenerates to just a
couple of BUG_ON()s.

Jan