[PATCH 0/14] Nested Virtualization: Overview

[PATCH 0/14] Nested Virtualization: Overview

[Christoph Egger]

Hi!

This patch series brings Nested Virtualization to Xen.
This is the third patch series. Improvements to the
previous patch submission:

- implement HVM-on-HVM (instead of SVM-on-HVM)
- move cpuid handling into tools (per Keir's request)

There might still be some nuances to fiddle with to make it
fit for VMX. Feedback from Intel is appreciated, therefore.

The patch series:

patch 01: add nestedhvm guest config option to the tools.
                  This is the only one patch touching the tools
patch 02: change local_event_delivery_* to take vcpu argument.
                  This prevents spurios xen crashes on guest shutdown/destroy
                  with nestedhvm enabled.
patch 03: Add data structures for nested virtualization.
patch 04: add nestedhvm function hooks, described in XenNestedHVM.pdf
patch 05: The heart of nested virtualization.
patch 06: Allow switch to paged real mode during vmrun emulation.
                  Emulate cr0 and cr4 when guest does not intercept them
                  (i.e. Hyper-V/Windows7, KVM)
patch 07: When injecting an exception into nested guest, inject
                  #VMEXIT into the guest if intercepted
patch 08: Allow guest to enable SVM in EFER
patch 09: Handle interrupts (generic part)
patch 10: SVM specific implementation for nested virtualization
patch 11: Handle interrupts (SVM specific)
patch 12: The piece of code that effectively turns nested virtualization on.
                  Use HVM_PARAM_* this time.
patch 13: Change p2m infrastructure to operate with per-p2m instead
                  of per-domain. Combined with patch 15, this allows to run
                  nested guest with hap-on-hap.
patch 14: Handle nested pagefault to enable hap-on-hap and handle
                  nested guest page-table-walks to emulate instructions
                  the guest does not intercept (i.e. WBINVD with Windows 7).



-- 
---to satisfy European Law for business letters:
Advanced Micro Devices GmbH
Einsteinring 24, 85609 Dornach b. Muenchen
Geschaeftsfuehrer: Alberto Bozzo, Andrew Bowd
Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen
Registergericht Muenchen, HRB Nr. 43632

RE: [PATCH 0/14] Nested Virtualization: Overview

[Dong, Eddie]

Hi, Chris:

Christoph Egger wrote:
> Hi!
> 
> This patch series brings Nested Virtualization to Xen.
> This is the third patch series. Improvements to the
> previous patch submission:
> 
> - implement HVM-on-HVM (instead of SVM-on-HVM)

Given that we don't have consensus on cross architecture nested virtualization support, I am doubting why this is urgent for now. I would prefer we make SVM-on-SVM and VMX-on-VMX work first. After that, if you prove SVM-on-VMX has real performance gain (which I doubt), we can see how to make a much generic effort to accomodate both natively nested virtualization and cross architecture nested virtualization. 

Drawing a picture which doesn't have a real usage with massive common code change is a kind of too much load for us now. Xen hvm_function table is a good example. Intel enabled VMX at very beginning of Xen HVM support, and SVM comes later on with a lot of code reuse from VMX side. Then the community and both side work together to make an API wrapper to reuse common code better and accomodate both architecture. I think we have to go similar path to make it work first.


> - move cpuid handling into tools (per Keir's request)
> 
> There might still be some nuances to fiddle with to make it
> fit for VMX. Feedback from Intel is appreciated, therefore.
> 


Thx, Eddie

Re: [PATCH 0/14] Nested Virtualization: Overview

[Christoph Egger]

On Tuesday 17 August 2010 08:04:20 Dong, Eddie wrote:
> Hi, Chris:
>
> Christoph Egger wrote:
> > Hi!
> >
> > This patch series brings Nested Virtualization to Xen.
> > This is the third patch series. Improvements to the
> > previous patch submission:
> >
> > - implement HVM-on-HVM (instead of SVM-on-HVM)
>
> Given that we don't have consensus on cross architecture nested
> virtualization support, I am doubting why this is urgent for now.

The reason to be "urgent" is not the time. This is the best way from
the software engineering side.

> I would prefer we make SVM-on-SVM and VMX-on-VMX work first. After that,
> if you prove SVM-on-VMX has real performance gain (which I doubt), we can
> see how to make a much generic effort to accomodate both natively nested
> virtualization and cross architecture nested virtualization.

Tim and Keir made clear they don't want to have two implementations after
I submitted my patch series the *first* time.

> Drawing a picture which doesn't have a real usage with massive common code
> change is a kind of too much load for us now. Xen hvm_function table is a
> good example. Intel enabled VMX at very beginning of Xen HVM support, and
> SVM comes later on with a lot of code reuse from VMX side. Then the
> community and both side work together to make an API wrapper to reuse
> common code better and accomodate both architecture. I think we have to go
> similar path to make it work first.

Tim remembers on this and said in this mail
http://lists.xensource.com/archives/html/xen-devel/2010-04/msg00812.html

"With HVM it has turned out to be quite a lot, but it's taken years of
reshuffling to pull code out into common HVM routines (and we're not there
yet)."

If I understand Tim correctly, the way you suggest is a "no".
(Added Tim on CC)

Christoph

>
> > - move cpuid handling into tools (per Keir's request)
> >
> > There might still be some nuances to fiddle with to make it
> > fit for VMX. Feedback from Intel is appreciated, therefore.
>
> Thx, Eddie



-- 
---to satisfy European Law for business letters:
Advanced Micro Devices GmbH
Einsteinring 24, 85609 Dornach b. Muenchen
Geschaeftsfuehrer: Alberto Bozzo, Andrew Bowd
Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen
Registergericht Muenchen, HRB Nr. 43632

Re: [PATCH 0/14] Nested Virtualization: Overview

[Keir Fraser]

On 17/08/2010 11:33, "Christoph Egger" <Christoph.Egger@amd.com> wrote:

>>> - implement HVM-on-HVM (instead of SVM-on-HVM)
>> 
>> Given that we don't have consensus on cross architecture nested
>> virtualization support, I am doubting why this is urgent for now.
> 
> The reason to be "urgent" is not the time. This is the best way from
> the software engineering side.
> 
>> I would prefer we make SVM-on-SVM and VMX-on-VMX work first. After that,
>> if you prove SVM-on-VMX has real performance gain (which I doubt), we can
>> see how to make a much generic effort to accomodate both natively nested
>> virtualization and cross architecture nested virtualization.
> 
> Tim and Keir made clear they don't want to have two implementations after
> I submitted my patch series the *first* time.

I think maybe this is an argument over two different things. To be clear we
want to support VMX-on-VMX and SVM-on-SVM. I assume this is what Christoph
means by HVM-on-HVM: any-like-on-like. In which case there is no
disagreement here.

Now, separately there is a debate to be had on how much code can be shared
in HVM-on-HVM, given the big differences between VMX and SVM. I would guess
that there will be at least things in the area of nested shadow and nested
HAP that can be shared, for example. Probably there is other stuff too. The
question is to what degree do we pursue that now rather than get divergent
stuff in tree and then go after it later. My mind isn't totally made up on
that; I don't know about Tim's.

 -- Keir

Re: [PATCH 0/14] Nested Virtualization: Overview

[Tim Deegan]

At 11:41 +0100 on 17 Aug (1282045318), Keir Fraser wrote:
> On 17/08/2010 11:33, "Christoph Egger" <Christoph.Egger@amd.com> wrote:
> > Tim and Keir made clear they don't want to have two implementations after
> > I submitted my patch series the *first* time.
> 
> I think maybe this is an argument over two different things. To be clear we
> want to support VMX-on-VMX and SVM-on-SVM. I assume this is what Christoph
> means by HVM-on-HVM: any-like-on-like. In which case there is no
> disagreement here.
> 
> Now, separately there is a debate to be had on how much code can be shared
> in HVM-on-HVM, given the big differences between VMX and SVM. I would guess
> that there will be at least things in the area of nested shadow and nested
> HAP that can be shared, for example. Probably there is other stuff too. The
> question is to what degree do we pursue that now rather than get divergent
> stuff in tree and then go after it later. My mind isn't totally made up on
> that; I don't know about Tim's.

The general tone of Christoph's latest patchset seems about right to me:
the concept of a VCPU being in nested HVM mode or not, the control
interface, and the bulk of the interrupt/exception injection logic seem
like they should be common from day one, unless there's a particular
reason not to.  The details of exactly how the nested VMC[BS] is
accessed and updated are of course arch-specific.

The two HAP-on-HAP designs are quite different but since all the EPT
code is already separate from the other p2m code that's OK.
Shadow-on-shadow and shadow-on-HAP ought to just work[tm] without any
extra moving parts.

I've no strong feelings about the details of the interface between the
common and the arch-specific code, but it seems like with a bit of
flexibility on both sides it could be made suitable for everyone.

Cheers,

Tim.

-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, XenServer Engineering
Citrix Systems UK Ltd.  (Company #02937203, SL9 0BG)