HVMlite gains

HVMlite gains

[Luis R. Rodriguez]

While discussing HVMLite with a few people a few questions have come
up. Since I only really understand a few possible gains with the
current design I wanted to get clarificaiton on a few which I simply
have no clue if we stand to gain from them, or if its on the roadmap:

  a) Will context switches use the actual CR3 register?
  b) Will IOPL live in the actual FLAGS register?
  c) Will guest-usable CPU features should show up in CPUID, and will
features that shouldn't be used should *not* show up in CPUID. For
instance currently you happen to boot Xen 4.4.0 with a new Linux dom0
on a CPU that supports MPX what will happen? What about with HVMlite?
  d)  Will acking an interrupt use the standard APIC mechanism?  Do
any of the current Xen variants do that?
  e) Can timing use RDTSC?

 Luis

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: HVMlite gains

[Andrew Cooper]

On 15/03/2016 21:14, Luis R. Rodriguez wrote:
> While discussing HVMLite with a few people a few questions have come
> up. Since I only really understand a few possible gains with the
> current design I wanted to get clarificaiton on a few which I simply
> have no clue if we stand to gain from them, or if its on the roadmap:
>
>   a) Will context switches use the actual CR3 register?

Yes.  HVMLite will use fully the full array of hardware virtualisation
extensions, so gets its own pagetables and cr3.

>   b) Will IOPL live in the actual FLAGS register?

Yes

>   c) Will guest-usable CPU features should show up in CPUID, and will
> features that shouldn't be used should *not* show up in CPUID. For
> instance currently you happen to boot Xen 4.4.0 with a new Linux dom0
> on a CPU that supports MPX what will happen? What about with HVMlite?

I am desperately trying to fix the existing broken-ness.  PV is too far
beyond repair when it comes to the OSXSAVE bit, but the rest is fine. 
See my "x86: Improvements to cpuid handling for guests" series, v3 of
which was posted today.

With that series in place, a guest should always see correct features
(give or take some "fun" with masking, but at least the xen_cpuid() path
will still be correct).

>   d)  Will acking an interrupt use the standard APIC mechanism?  Do
> any of the current Xen variants do that?

Emulated APIC will be enabled by default. The "PV event" path will be
available as an alternative.

If a user desperately wishes to avoid the emulated APIC, they have the
option to disable it in the domain config.

>   e) Can timing use RDTSC?

I don't understand this question in the context of the others.  RDTSC
has (as far as I can tell) always been advertised and available for
guest use.  RDTSCP is a different matter, and I have half-fixed that
brokenness; it should now work correctly in HVM guests.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: HVMlite gains

[Andy Lutomirski]

On Tue, Mar 15, 2016 at 2:29 PM, Andrew Cooper
<andrew.cooper3@citrix.com> wrote:
> On 15/03/2016 21:14, Luis R. Rodriguez wrote:
>> While discussing HVMLite with a few people a few questions have come
>> up. Since I only really understand a few possible gains with the
>> current design I wanted to get clarificaiton on a few which I simply
>> have no clue if we stand to gain from them, or if its on the roadmap:
>>
>>   a) Will context switches use the actual CR3 register?
>
> Yes.  HVMLite will use fully the full array of hardware virtualisation
> extensions, so gets its own pagetables and cr3.
>
>>   b) Will IOPL live in the actual FLAGS register?
>
> Yes
>
>>   c) Will guest-usable CPU features should show up in CPUID, and will
>> features that shouldn't be used should *not* show up in CPUID. For
>> instance currently you happen to boot Xen 4.4.0 with a new Linux dom0
>> on a CPU that supports MPX what will happen? What about with HVMlite?
>
> I am desperately trying to fix the existing broken-ness.  PV is too far
> beyond repair when it comes to the OSXSAVE bit, but the rest is fine.
> See my "x86: Improvements to cpuid handling for guests" series, v3 of
> which was posted today.
>
> With that series in place, a guest should always see correct features
> (give or take some "fun" with masking, but at least the xen_cpuid() path
> will still be correct).
>
>>   d)  Will acking an interrupt use the standard APIC mechanism?  Do
>> any of the current Xen variants do that?
>
> Emulated APIC will be enabled by default. The "PV event" path will be
> available as an alternative.
>
> If a user desperately wishes to avoid the emulated APIC, they have the
> option to disable it in the domain config.
>
>>   e) Can timing use RDTSC?
>
> I don't understand this question in the context of the others.  RDTSC
> has (as far as I can tell) always been advertised and available for
> guest use.  RDTSCP is a different matter, and I have half-fixed that
> brokenness; it should now work correctly in HVM guests.
>

These questions mostly came from me, and they weren't necessarily
intended to make sense as a coherent whole :)  They were more of a
random collection of things I was wondering about to varying extents.

What I mean is:  if we point sched_clock at RDTSC and try to use the
regular TSC timesource in a guest, will it work reasonably well,
assuming that the underlying hardware supports it?  And, if the
underlying hardware doesn't support it (e.g. not constant / invariant
or no TSC offsetting available or similar), will the hypervisor tell
the guest this fact via CPUID so that the standard guest clocksource
code doesn't try to use a non-working TSC?

--Andy

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: HVMlite gains

[Boris Ostrovsky]

On 03/15/2016 05:36 PM, Andy Lutomirski wrote:
> On Tue, Mar 15, 2016 at 2:29 PM, Andrew Cooper
> <andrew.cooper3@citrix.com> wrote:
>> On 15/03/2016 21:14, Luis R. Rodriguez wrote:
>>> While discussing HVMLite with a few people a few questions have come
>>> up. Since I only really understand a few possible gains with the
>>> current design I wanted to get clarificaiton on a few which I simply
>>> have no clue if we stand to gain from them, or if its on the roadmap:
>>>
>>>    a) Will context switches use the actual CR3 register?
>> Yes.  HVMLite will use fully the full array of hardware virtualisation
>> extensions, so gets its own pagetables and cr3.
>>
>>>    b) Will IOPL live in the actual FLAGS register?
>> Yes
>>
>>>    c) Will guest-usable CPU features should show up in CPUID, and will
>>> features that shouldn't be used should *not* show up in CPUID. For
>>> instance currently you happen to boot Xen 4.4.0 with a new Linux dom0
>>> on a CPU that supports MPX what will happen? What about with HVMlite?
>> I am desperately trying to fix the existing broken-ness.  PV is too far
>> beyond repair when it comes to the OSXSAVE bit, but the rest is fine.
>> See my "x86: Improvements to cpuid handling for guests" series, v3 of
>> which was posted today.
>>
>> With that series in place, a guest should always see correct features
>> (give or take some "fun" with masking, but at least the xen_cpuid() path
>> will still be correct).
>>
>>>    d)  Will acking an interrupt use the standard APIC mechanism?  Do
>>> any of the current Xen variants do that?
>> Emulated APIC will be enabled by default. The "PV event" path will be
>> available as an alternative.
>>
>> If a user desperately wishes to avoid the emulated APIC, they have the
>> option to disable it in the domain config.
>>
>>>    e) Can timing use RDTSC?
>> I don't understand this question in the context of the others.  RDTSC
>> has (as far as I can tell) always been advertised and available for
>> guest use.  RDTSCP is a different matter, and I have half-fixed that
>> brokenness; it should now work correctly in HVM guests.
>>
> These questions mostly came from me, and they weren't necessarily
> intended to make sense as a coherent whole :)  They were more of a
> random collection of things I was wondering about to varying extents.
>
> What I mean is:  if we point sched_clock at RDTSC and try to use the
> regular TSC timesource in a guest, will it work reasonably well,
> assuming that the underlying hardware supports it?  And, if the
> underlying hardware doesn't support it (e.g. not constant / invariant
> or no TSC offsetting available or similar), will the hypervisor tell
> the guest this fact via CPUID so that the standard guest clocksource
> code doesn't try to use a non-working TSC?

Hypervisor typically clears TSC Invariant bit because the guest can 
migrate to a system with a different clock

http://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=xen/arch/x86/domain.c;h=a6d721bd48176f51c0d9dfb57099c8b7f52220c2;hb=HEAD#l2612

There are options (in guest config file) to have hypervisor set this flag.

-boris


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: HVMlite gains

[Andrew Cooper]

On 15/03/2016 21:36, Andy Lutomirski wrote:
>
>>>   e) Can timing use RDTSC?
>> I don't understand this question in the context of the others.  RDTSC
>> has (as far as I can tell) always been advertised and available for
>> guest use.  RDTSCP is a different matter, and I have half-fixed that
>> brokenness; it should now work correctly in HVM guests.
>>
> These questions mostly came from me, and they weren't necessarily
> intended to make sense as a coherent whole :)  They were more of a
> random collection of things I was wondering about to varying extents.
>
> What I mean is:  if we point sched_clock at RDTSC and try to use the
> regular TSC timesource in a guest, will it work reasonably well,
> assuming that the underlying hardware supports it?  And, if the
> underlying hardware doesn't support it (e.g. not constant / invariant
> or no TSC offsetting available or similar), will the hypervisor tell
> the guest this fact via CPUID so that the standard guest clocksource
> code doesn't try to use a non-working TSC?

In principle yes, but it is rather more complicated than that.

By default, if you want a guest to be migrateable and you can't
guarantee that you will have hardware TSC scaling support on every
future destination, you cannot advertise the TSC as stable to the
guest.  We err on the side of caution and don't advertise invariance by
default.

In practice, if you are running on anything vaguely modern, the TSC will
be reliable between migrates.

What the migration protocol currently lacks is a mechanism to identify
"This VM was advertised invariant TSC at frequency $X when it was
booted".  There is nominally a "no migrate" flag which can be set, at
which point invariance will be advertised if the host is capable. 
However, there is no way for the toolstack to query this, so nothing in
the migrate code checks or acts upon it.

Windows have worked around this limitation with the Viridian spec,
whereby the hypervisor can provide the current TSC frequency, and
promises that it won't change until the next suspend/resume, at which
point the frequency will be resampled.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: HVMlite gains

[Andy Lutomirski]

On Tue, Mar 15, 2016 at 2:50 PM, Andrew Cooper
<andrew.cooper3@citrix.com> wrote:
> On 15/03/2016 21:36, Andy Lutomirski wrote:
>>
>>>>   e) Can timing use RDTSC?
>>> I don't understand this question in the context of the others.  RDTSC
>>> has (as far as I can tell) always been advertised and available for
>>> guest use.  RDTSCP is a different matter, and I have half-fixed that
>>> brokenness; it should now work correctly in HVM guests.
>>>
>> These questions mostly came from me, and they weren't necessarily
>> intended to make sense as a coherent whole :)  They were more of a
>> random collection of things I was wondering about to varying extents.
>>
>> What I mean is:  if we point sched_clock at RDTSC and try to use the
>> regular TSC timesource in a guest, will it work reasonably well,
>> assuming that the underlying hardware supports it?  And, if the
>> underlying hardware doesn't support it (e.g. not constant / invariant
>> or no TSC offsetting available or similar), will the hypervisor tell
>> the guest this fact via CPUID so that the standard guest clocksource
>> code doesn't try to use a non-working TSC?
>
> In principle yes, but it is rather more complicated than that.
>
> By default, if you want a guest to be migrateable and you can't
> guarantee that you will have hardware TSC scaling support on every
> future destination, you cannot advertise the TSC as stable to the
> guest.  We err on the side of caution and don't advertise invariance by
> default.
>
> In practice, if you are running on anything vaguely modern, the TSC will
> be reliable between migrates.

By "reliable" do you mean monotonic and not horribly jumpy?  I thought
there was no shipping hardware with TSC scaling.

>
> What the migration protocol currently lacks is a mechanism to identify
> "This VM was advertised invariant TSC at frequency $X when it was
> booted".  There is nominally a "no migrate" flag which can be set, at
> which point invariance will be advertised if the host is capable.
> However, there is no way for the toolstack to query this, so nothing in
> the migrate code checks or acts upon it.
>
> Windows have worked around this limitation with the Viridian spec,
> whereby the hypervisor can provide the current TSC frequency, and
> promises that it won't change until the next suspend/resume, at which
> point the frequency will be resampled.
>

That's simpler and maybe even better than the pvclock design, at least
as implemented by KVM.  Sigh.

--Andy

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: HVMlite gains

[Andrew Cooper]

On 15/03/2016 21:52, Andy Lutomirski wrote:
> On Tue, Mar 15, 2016 at 2:50 PM, Andrew Cooper
> <andrew.cooper3@citrix.com> wrote:
>> On 15/03/2016 21:36, Andy Lutomirski wrote:
>>>>>   e) Can timing use RDTSC?
>>>> I don't understand this question in the context of the others.  RDTSC
>>>> has (as far as I can tell) always been advertised and available for
>>>> guest use.  RDTSCP is a different matter, and I have half-fixed that
>>>> brokenness; it should now work correctly in HVM guests.
>>>>
>>> These questions mostly came from me, and they weren't necessarily
>>> intended to make sense as a coherent whole :)  They were more of a
>>> random collection of things I was wondering about to varying extents.
>>>
>>> What I mean is:  if we point sched_clock at RDTSC and try to use the
>>> regular TSC timesource in a guest, will it work reasonably well,
>>> assuming that the underlying hardware supports it?  And, if the
>>> underlying hardware doesn't support it (e.g. not constant / invariant
>>> or no TSC offsetting available or similar), will the hypervisor tell
>>> the guest this fact via CPUID so that the standard guest clocksource
>>> code doesn't try to use a non-working TSC?
>> In principle yes, but it is rather more complicated than that.
>>
>> By default, if you want a guest to be migrateable and you can't
>> guarantee that you will have hardware TSC scaling support on every
>> future destination, you cannot advertise the TSC as stable to the
>> guest.  We err on the side of caution and don't advertise invariance by
>> default.
>>
>> In practice, if you are running on anything vaguely modern, the TSC will
>> be reliable between migrates.
> By "reliable" do you mean monotonic and not horribly jumpy?  I thought
> there was no shipping hardware with TSC scaling.

AMD have had TSC scaling for a long time (code added to Xen in 2011). 
Intel are the ones late to the party in this case.

There was a patch series from Joao around about Christmas "x86/time:
PVCLOCK_TSC_STABLE_BIT supportwhich identified several bugs with Xen's
TSC handling as visible in the PVCLK.  It would be nice to get those
bugs fixed.

>
>> What the migration protocol currently lacks is a mechanism to identify
>> "This VM was advertised invariant TSC at frequency $X when it was
>> booted".  There is nominally a "no migrate" flag which can be set, at
>> which point invariance will be advertised if the host is capable.
>> However, there is no way for the toolstack to query this, so nothing in
>> the migrate code checks or acts upon it.
>>
>> Windows have worked around this limitation with the Viridian spec,
>> whereby the hypervisor can provide the current TSC frequency, and
>> promises that it won't change until the next suspend/resume, at which
>> point the frequency will be resampled.
>>
> That's simpler and maybe even better than the pvclock design, at least
> as implemented by KVM.  Sigh.

Updates to that also need fixing.  PVCLOCK is a Xen ABI which was
borrowed by KVM then locally modified.

I believe the two are still compatible.

But yes - the Viridian way does appear substantially more sane.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: HVMlite gains

[Konrad Rzeszutek Wilk]

On Tue, Mar 15, 2016 at 02:14:15PM -0700, Luis R. Rodriguez wrote:
> While discussing HVMLite with a few people a few questions have come
> up. Since I only really understand a few possible gains with the
> current design I wanted to get clarificaiton on a few which I simply

Just think of baremetal without BIOS. Without PCI support (unless
needed).

> have no clue if we stand to gain from them, or if its on the roadmap:
> 
>   a) Will context switches use the actual CR3 register?

Yes.
>   b) Will IOPL live in the actual FLAGS register?

Yes.
>   c) Will guest-usable CPU features should show up in CPUID, and will
> features that shouldn't be used should *not* show up in CPUID. For

Correct.
> instance currently you happen to boot Xen 4.4.0 with a new Linux dom0
> on a CPU that supports MPX what will happen? What about with HVMlite?

It should boot just normally. Either PV dom0 guest or HVMLite dom0
guest (whenever that is operational).


>   d)  Will acking an interrupt use the standard APIC mechanism?  Do
> any of the current Xen variants do that?

Yes and no. It is disabled right now but it will be enabled as I want
to make sure we can use vAPIC in the guest.

You don't want to use emulated APIC mechanism as it incurs VMEXITs.

>   e) Can timing use RDTSC?

Yes (odd question? You could always use rdtsc). 
> 
>  Luis

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: HVMlite gains

[Andy Lutomirski]

On Mar 15, 2016 3:05 PM, "Andrew Cooper" <andrew.cooper3@citrix.com> wrote:
>
> On 15/03/2016 21:52, Andy Lutomirski wrote:
> > On Tue, Mar 15, 2016 at 2:50 PM, Andrew Cooper
> > <andrew.cooper3@citrix.com> wrote:
> >> On 15/03/2016 21:36, Andy Lutomirski wrote:
> >>>>>   e) Can timing use RDTSC?
> >>>> I don't understand this question in the context of the others.  RDTSC
> >>>> has (as far as I can tell) always been advertised and available for
> >>>> guest use.  RDTSCP is a different matter, and I have half-fixed that
> >>>> brokenness; it should now work correctly in HVM guests.
> >>>>
> >>> These questions mostly came from me, and they weren't necessarily
> >>> intended to make sense as a coherent whole :)  They were more of a
> >>> random collection of things I was wondering about to varying extents.
> >>>
> >>> What I mean is:  if we point sched_clock at RDTSC and try to use the
> >>> regular TSC timesource in a guest, will it work reasonably well,
> >>> assuming that the underlying hardware supports it?  And, if the
> >>> underlying hardware doesn't support it (e.g. not constant / invariant
> >>> or no TSC offsetting available or similar), will the hypervisor tell
> >>> the guest this fact via CPUID so that the standard guest clocksource
> >>> code doesn't try to use a non-working TSC?
> >> In principle yes, but it is rather more complicated than that.
> >>
> >> By default, if you want a guest to be migrateable and you can't
> >> guarantee that you will have hardware TSC scaling support on every
> >> future destination, you cannot advertise the TSC as stable to the
> >> guest.  We err on the side of caution and don't advertise invariance by
> >> default.
> >>
> >> In practice, if you are running on anything vaguely modern, the TSC will
> >> be reliable between migrates.
> > By "reliable" do you mean monotonic and not horribly jumpy?  I thought
> > there was no shipping hardware with TSC scaling.
>
> AMD have had TSC scaling for a long time (code added to Xen in 2011).
> Intel are the ones late to the party in this case.
>
> There was a patch series from Joao around about Christmas "x86/time:
> PVCLOCK_TSC_STABLE_BIT supportwhich identified several bugs with Xen's
> TSC handling as visible in the PVCLK.  It would be nice to get those
> bugs fixed.
>
> >
> >> What the migration protocol currently lacks is a mechanism to identify
> >> "This VM was advertised invariant TSC at frequency $X when it was
> >> booted".  There is nominally a "no migrate" flag which can be set, at
> >> which point invariance will be advertised if the host is capable.
> >> However, there is no way for the toolstack to query this, so nothing in
> >> the migrate code checks or acts upon it.
> >>
> >> Windows have worked around this limitation with the Viridian spec,
> >> whereby the hypervisor can provide the current TSC frequency, and
> >> promises that it won't change until the next suspend/resume, at which
> >> point the frequency will be resampled.
> >>
> > That's simpler and maybe even better than the pvclock design, at least
> > as implemented by KVM.  Sigh.
>
> Updates to that also need fixing.  PVCLOCK is a Xen ABI which was
> borrowed by KVM then locally modified.
>
> I believe the two are still compatible.
>
> But yes - the Viridian way does appear substantially more sane.
>

Hmm. Is migration synchronous enough for that approach to be reliable?
 That is, when the TSC frequency changes, is there some way that the
guest is guaranteed to be notified before it starts screwing up its
timing calculations?

> ~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: HVMlite gains

[Andrew Cooper]

On 16/03/16 03:18, Andy Lutomirski wrote:
> On Mar 15, 2016 3:05 PM, "Andrew Cooper" <andrew.cooper3@citrix.com> wrote:
>> On 15/03/2016 21:52, Andy Lutomirski wrote:
>>> On Tue, Mar 15, 2016 at 2:50 PM, Andrew Cooper
>>> <andrew.cooper3@citrix.com> wrote:
>>>> On 15/03/2016 21:36, Andy Lutomirski wrote:
>>>>>>>   e) Can timing use RDTSC?
>>>>>> I don't understand this question in the context of the others.  RDTSC
>>>>>> has (as far as I can tell) always been advertised and available for
>>>>>> guest use.  RDTSCP is a different matter, and I have half-fixed that
>>>>>> brokenness; it should now work correctly in HVM guests.
>>>>>>
>>>>> These questions mostly came from me, and they weren't necessarily
>>>>> intended to make sense as a coherent whole :)  They were more of a
>>>>> random collection of things I was wondering about to varying extents.
>>>>>
>>>>> What I mean is:  if we point sched_clock at RDTSC and try to use the
>>>>> regular TSC timesource in a guest, will it work reasonably well,
>>>>> assuming that the underlying hardware supports it?  And, if the
>>>>> underlying hardware doesn't support it (e.g. not constant / invariant
>>>>> or no TSC offsetting available or similar), will the hypervisor tell
>>>>> the guest this fact via CPUID so that the standard guest clocksource
>>>>> code doesn't try to use a non-working TSC?
>>>> In principle yes, but it is rather more complicated than that.
>>>>
>>>> By default, if you want a guest to be migrateable and you can't
>>>> guarantee that you will have hardware TSC scaling support on every
>>>> future destination, you cannot advertise the TSC as stable to the
>>>> guest.  We err on the side of caution and don't advertise invariance by
>>>> default.
>>>>
>>>> In practice, if you are running on anything vaguely modern, the TSC will
>>>> be reliable between migrates.
>>> By "reliable" do you mean monotonic and not horribly jumpy?  I thought
>>> there was no shipping hardware with TSC scaling.
>> AMD have had TSC scaling for a long time (code added to Xen in 2011).
>> Intel are the ones late to the party in this case.
>>
>> There was a patch series from Joao around about Christmas "x86/time:
>> PVCLOCK_TSC_STABLE_BIT supportwhich identified several bugs with Xen's
>> TSC handling as visible in the PVCLK.  It would be nice to get those
>> bugs fixed.
>>
>>>> What the migration protocol currently lacks is a mechanism to identify
>>>> "This VM was advertised invariant TSC at frequency $X when it was
>>>> booted".  There is nominally a "no migrate" flag which can be set, at
>>>> which point invariance will be advertised if the host is capable.
>>>> However, there is no way for the toolstack to query this, so nothing in
>>>> the migrate code checks or acts upon it.
>>>>
>>>> Windows have worked around this limitation with the Viridian spec,
>>>> whereby the hypervisor can provide the current TSC frequency, and
>>>> promises that it won't change until the next suspend/resume, at which
>>>> point the frequency will be resampled.
>>>>
>>> That's simpler and maybe even better than the pvclock design, at least
>>> as implemented by KVM.  Sigh.
>> Updates to that also need fixing.  PVCLOCK is a Xen ABI which was
>> borrowed by KVM then locally modified.
>>
>> I believe the two are still compatible.
>>
>> But yes - the Viridian way does appear substantially more sane.
>>
> Hmm. Is migration synchronous enough for that approach to be reliable?
>  That is, when the TSC frequency changes, is there some way that the
> guest is guaranteed to be notified before it starts screwing up its
> timing calculations?

For VMs which do not have any Xen PV drivers, it is possible to migrate
them without any cooperation at all.  In this case, there is no
practical way to indicate that the TSC frequency has changed.

For VMs which do have PV drivers, migration requires guest cooperation,
or memory corruption will occur (pre-existing shared mappings can't have
writes tracked on them, so the guest driver is required to replay its
control ring on the destination side).  In this case, the guest always
passes through xen_suspend().  HYPERVISOR_suspend() is called in the
source side, and returns on the destination.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel