Future support of 5-level paging in Xen

Future support of 5-level paging in Xen

[Juergen Gross]

The first round of (very preliminary) patches for supporting the new
5-level paging of future Intel x86 processors [1] has been posted to
lkml:

https://lkml.org/lkml/2016/12/8/378

An explicit note has been added: "CONFIG_XEN is broken." and
"I would appreciate help with the code."

I think we should start a discussion what we want to do in future:

- are we going to support 5-level paging for PV guests?
- do we limit 5-level paging to PVH and HVM?


Juergen

[1]
https://software.intel.com/sites/default/files/managed/2b/80/5-level_paging_white_paper.pdf

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Future support of 5-level paging in Xen

[Andrew Cooper]

On 08/12/16 16:46, Juergen Gross wrote:
> The first round of (very preliminary) patches for supporting the new
> 5-level paging of future Intel x86 processors [1] has been posted to
> lkml:
>
> https://lkml.org/lkml/2016/12/8/378
>
> An explicit note has been added: "CONFIG_XEN is broken." and
> "I would appreciate help with the code."
>
> I think we should start a discussion what we want to do in future:
>
> - are we going to support 5-level paging for PV guests?
> - do we limit 5-level paging to PVH and HVM?

The 64bit PV ABI has 16TB of virtual address space just above the upper
48-canonical boundary.

Were Xen to support 5-level PV guests, we'd either leave the PV guest
kernel with exactly the same amount of higher half space as it currently
has, or we'd have to recompile Xen as properly position-independent and
use a different virtual range in different paging mode.

Another pain point is the quantity of virtual address space handed away
in the ABI.  We currently had 97% of the virtual address space away to
64bit PV guests, and frankly this is too much.  This is the wrong way
around when Xen has more management to do than the guest.  If we were to
go along the 5-level PV guests route, I will insist that there is a
rather more even split of virtual address space baked into the ABI.

However, a big question is whether any of this effort is worth doing, in
the light of PVH.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Future support of 5-level paging in Xen

[Stefano Stabellini]

On Thu, 8 Dec 2016, Andrew Cooper wrote:
> On 08/12/16 16:46, Juergen Gross wrote:
> > The first round of (very preliminary) patches for supporting the new
> > 5-level paging of future Intel x86 processors [1] has been posted to
> > lkml:
> >
> > https://lkml.org/lkml/2016/12/8/378
> >
> > An explicit note has been added: "CONFIG_XEN is broken." and
> > "I would appreciate help with the code."
> >
> > I think we should start a discussion what we want to do in future:
> >
> > - are we going to support 5-level paging for PV guests?
> > - do we limit 5-level paging to PVH and HVM?
> 
> The 64bit PV ABI has 16TB of virtual address space just above the upper
> 48-canonical boundary.
> 
> Were Xen to support 5-level PV guests, we'd either leave the PV guest
> kernel with exactly the same amount of higher half space as it currently
> has, or we'd have to recompile Xen as properly position-independent and
> use a different virtual range in different paging mode.
> 
> Another pain point is the quantity of virtual address space handed away
> in the ABI.  We currently had 97% of the virtual address space away to
> 64bit PV guests, and frankly this is too much.  This is the wrong way
> around when Xen has more management to do than the guest.  If we were to
> go along the 5-level PV guests route, I will insist that there is a
> rather more even split of virtual address space baked into the ABI.
> 
> However, a big question is whether any of this effort is worth doing, in
> the light of PVH.

With my Aporeto hat on, I'll say that given the overwhelming amount of
hardware available out there without virtualization support, this work
is worth doing. I am referring to all the public cloud virtual machines,
which can support Xen PV guests but cannot support PVH guests.

Of course even the largest virtual machine today (2TB on Amazon AFAIK)
is not close to reaching the current memory limit, but it's just a
matter of time.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Future support of 5-level paging in Xen

[Andrew Cooper]

On 08/12/2016 19:18, Stefano Stabellini wrote:
> On Thu, 8 Dec 2016, Andrew Cooper wrote:
>> On 08/12/16 16:46, Juergen Gross wrote:
>>> The first round of (very preliminary) patches for supporting the new
>>> 5-level paging of future Intel x86 processors [1] has been posted to
>>> lkml:
>>>
>>> https://lkml.org/lkml/2016/12/8/378
>>>
>>> An explicit note has been added: "CONFIG_XEN is broken." and
>>> "I would appreciate help with the code."
>>>
>>> I think we should start a discussion what we want to do in future:
>>>
>>> - are we going to support 5-level paging for PV guests?
>>> - do we limit 5-level paging to PVH and HVM?
>> The 64bit PV ABI has 16TB of virtual address space just above the upper
>> 48-canonical boundary.
>>
>> Were Xen to support 5-level PV guests, we'd either leave the PV guest
>> kernel with exactly the same amount of higher half space as it currently
>> has, or we'd have to recompile Xen as properly position-independent and
>> use a different virtual range in different paging mode.
>>
>> Another pain point is the quantity of virtual address space handed away
>> in the ABI.  We currently had 97% of the virtual address space away to
>> 64bit PV guests, and frankly this is too much.  This is the wrong way
>> around when Xen has more management to do than the guest.  If we were to
>> go along the 5-level PV guests route, I will insist that there is a
>> rather more even split of virtual address space baked into the ABI.
>>
>> However, a big question is whether any of this effort is worth doing, in
>> the light of PVH.
> With my Aporeto hat on, I'll say that given the overwhelming amount of
> hardware available out there without virtualization support, this work
> is worth doing. I am referring to all the public cloud virtual machines,
> which can support Xen PV guests but cannot support PVH guests.

Why is Xen supporting 5-level guests useful for running in a PV cloud
VM?  Xen doesn't run PV.

I am not suggesting that we avoid adding 5-level support to Xen.  We
should absolutely do that.  The question is only whether we extend the
PV ABI to support 5-level PV guests.  Conceptually, its very easy to
have a 5-level Xen but only supporting 4-level PV guests.

VT-x and SVM date from 2005/2006 and are now 10 years old.  I would be
surprised if you would find much hardware of this age in any cloud; you
can't by anything that old, and support contracts have probably run out
if you have owned that hardware for 10 years.

> Of course even the largest virtual machine today (2TB on Amazon AFAIK)
> is not close to reaching the current memory limit, but it's just a
> matter of time.

/me things Oracle will have something to say about this.  I'm sure there
was talk about VMs larger than this at previous hackathons.  XenServer
functions (ish, so long as you don't migrate) with 6TB VMs, although
starting and shutting them down feels like treacle.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Future support of 5-level paging in Xen

[Boris Ostrovsky]

On 12/08/2016 05:21 PM, Andrew Cooper wrote:
> On 08/12/2016 19:18, Stefano Stabellini wrote:

>
>> Of course even the largest virtual machine today (2TB on Amazon AFAIK)
>> is not close to reaching the current memory limit, but it's just a
>> matter of time.
>
> /me things Oracle will have something to say about this.  I'm sure there
> was talk about VMs larger than this at previous hackathons.  XenServer
> functions (ish, so long as you don't migrate) with 6TB VMs, although
> starting and shutting them down feels like treacle.



I've been working (on and off) with SGI to get one of their 32TB boxes 
to boot and I don't think that works. We've fixed a couple of bugs but I 
don't think Xen can boot with that much memory. We successfully booted 
with just under 8TB but couldn't do it with the full system. The machine 
has been taken from us for now so this work is on hold.

This is on OVM, which is 4.4-based, we haven't tried (IIRC) latest bits.

(BTW, speaking of slow starting and shutting down very large guests ---
have you or anyone else had a chance to look at this? My investigation 
initially pointed to scrubbing and then to an insane number of hypercall 
preemptions in relinquish_memory()).

-boris

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Future support of 5-level paging in Xen:wq

[Stefano Stabellini]

On Thu, 8 Dec 2016, Andrew Cooper wrote:
> On 08/12/2016 19:18, Stefano Stabellini wrote:
> > On Thu, 8 Dec 2016, Andrew Cooper wrote:
> >> On 08/12/16 16:46, Juergen Gross wrote:
> >>> The first round of (very preliminary) patches for supporting the new
> >>> 5-level paging of future Intel x86 processors [1] has been posted to
> >>> lkml:
> >>>
> >>> https://lkml.org/lkml/2016/12/8/378
> >>>
> >>> An explicit note has been added: "CONFIG_XEN is broken." and
> >>> "I would appreciate help with the code."
> >>>
> >>> I think we should start a discussion what we want to do in future:
> >>>
> >>> - are we going to support 5-level paging for PV guests?
> >>> - do we limit 5-level paging to PVH and HVM?
> >> The 64bit PV ABI has 16TB of virtual address space just above the upper
> >> 48-canonical boundary.
> >>
> >> Were Xen to support 5-level PV guests, we'd either leave the PV guest
> >> kernel with exactly the same amount of higher half space as it currently
> >> has, or we'd have to recompile Xen as properly position-independent and
> >> use a different virtual range in different paging mode.
> >>
> >> Another pain point is the quantity of virtual address space handed away
> >> in the ABI.  We currently had 97% of the virtual address space away to
> >> 64bit PV guests, and frankly this is too much.  This is the wrong way
> >> around when Xen has more management to do than the guest.  If we were to
> >> go along the 5-level PV guests route, I will insist that there is a
> >> rather more even split of virtual address space baked into the ABI.
> >>
> >> However, a big question is whether any of this effort is worth doing, in
> >> the light of PVH.
> > With my Aporeto hat on, I'll say that given the overwhelming amount of
> > hardware available out there without virtualization support, this work
> > is worth doing. I am referring to all the public cloud virtual machines,
> > which can support Xen PV guests but cannot support PVH guests.
> 
> Why is Xen supporting 5-level guests useful for running in a PV cloud
> VM?  Xen doesn't run PV.
> 
> I am not suggesting that we avoid adding 5-level support to Xen.  We
> should absolutely do that.  The question is only whether we extend the
> PV ABI to support 5-level PV guests.  Conceptually, its very easy to
> have a 5-level Xen but only supporting 4-level PV guests.
> 
> VT-x and SVM date from 2005/2006 and are now 10 years old.  I would be
> surprised if you would find much hardware of this age in any cloud; you
> can't by anything that old, and support contracts have probably run out
> if you have owned that hardware for 10 years.

I am thinking that in a couple of years, we might already find VMs so
large that to use all the memory in a nested virt scenario, we need
5-level PV abi support.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Future support of 5-level paging in Xen

[Andrew Cooper]

On 08/12/2016 23:40, Boris Ostrovsky wrote:
>
>
>>
>>> Of course even the largest virtual machine today (2TB on Amazon AFAIK)
>>> is not close to reaching the current memory limit, but it's just a
>>> matter of time.
>>
>> /me things Oracle will have something to say about this.  I'm sure there
>> was talk about VMs larger than this at previous hackathons.  XenServer
>> functions (ish, so long as you don't migrate) with 6TB VMs, although
>> starting and shutting them down feels like treacle.
>
> I've been working (on and off) with SGI to get one of their 32TB boxes
> to boot and I don't think that works. We've fixed a couple of bugs but
> I don't think Xen can boot with that much memory. We successfully
> booted with just under 8TB but couldn't do it with the full system.
> The machine has been taken from us for now so this work is on hold.
>
> This is on OVM, which is 4.4-based, we haven't tried (IIRC) latest bits.

Because 64bit PV guests get 97% of the virtual address space, Xen hits
highmem/lowmem problems at the 5TB boundary, which is where we run out
of virtual address space for the directmap.

Xen supports up to 16TB of RAM (32bits in struct page_info, for a total
of 44 bits of mfns), although last time I checked Xen was still unstable
if there was any RAM above the 5 TB boundary.  Jan did subsequently find
and fix an off-by-one error, and I haven't had occasion to re-test since.

If you enable CONFIG_BIGMEM (newer than 4.4 I think, but I don't
actually recall), Xen's virtual layout changes.  The directmap shrinks
to just 3.5TB, to make space for a frametable containing larger struct
page_info's with 64bit indicies.  This has a total supported limit of
123TB of RAM, due to virtual range allocated to the frametable.

When I observed this going wrong, it went wrong because
alloc_xenheap_page() handed back virtual addresses which creep into the
64bit PV kernels ABI range.  These virtual addresses are safe for Xen to
use in idle and hvm contexts, but not in PV context.

> (BTW, speaking of slow starting and shutting down very large guests ---
> have you or anyone else had a chance to look at this? My investigation
> initially pointed to scrubbing and then to an insane number of
> hypercall preemptions in relinquish_memory()).

This is another item I meant to re-engage on.  (Its on my todo list,
along with CPUID and nested virt, but looks like it is depending on my
whishlist item of several extra hours in the day to get some of the work
done in.)

Yes.  We should do something towards fixing that.  Current performance
measurements put a 1.5TB domain at ~14 minutes for the domain_kill
hypercall to complete.

I seem to recall some vague plans towards having per-node dirty-page
lists, scrubbing in idle context, and on-demand scrubbing at alloc-time
if the clean list is empty.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Future support of 5-level paging in Xen

[Boris Ostrovsky]

On 12/08/2016 07:20 PM, Andrew Cooper wrote:
> On 08/12/2016 23:40, Boris Ostrovsky wrote:
>>
>>
>>>
>>>> Of course even the largest virtual machine today (2TB on Amazon AFAIK)
>>>> is not close to reaching the current memory limit, but it's just a
>>>> matter of time.
>>>
>>> /me things Oracle will have something to say about this.  I'm sure there
>>> was talk about VMs larger than this at previous hackathons.  XenServer
>>> functions (ish, so long as you don't migrate) with 6TB VMs, although
>>> starting and shutting them down feels like treacle.
>>
>> I've been working (on and off) with SGI to get one of their 32TB boxes
>> to boot and I don't think that works. We've fixed a couple of bugs but
>> I don't think Xen can boot with that much memory. We successfully
>> booted with just under 8TB but couldn't do it with the full system.
>> The machine has been taken from us for now so this work is on hold.
>>
>> This is on OVM, which is 4.4-based, we haven't tried (IIRC) latest bits.
>
> Because 64bit PV guests get 97% of the virtual address space, Xen hits
> highmem/lowmem problems at the 5TB boundary, which is where we run out
> of virtual address space for the directmap.
>
> Xen supports up to 16TB of RAM (32bits in struct page_info, for a total
> of 44 bits of mfns), although last time I checked Xen was still unstable
> if there was any RAM above the 5 TB boundary.  Jan did subsequently find
> and fix an off-by-one error, and I haven't had occasion to re-test since.
>
> If you enable CONFIG_BIGMEM (newer than 4.4 I think, but I don't


And apparently we don't have that in the OVM version I am looking at. 
But I'll try the upstream bits when we get a chance to get on this box.


> actually recall), Xen's virtual layout changes.  The directmap shrinks
> to just 3.5TB, to make space for a frametable containing larger struct
> page_info's with 64bit indicies.  This has a total supported limit of
> 123TB of RAM, due to virtual range allocated to the frametable.
>
> When I observed this going wrong, it went wrong because
> alloc_xenheap_page() handed back virtual addresses which creep into the
> 64bit PV kernels ABI range.  These virtual addresses are safe for Xen to
> use in idle and hvm contexts, but not in PV context.
>
>> (BTW, speaking of slow starting and shutting down very large guests ---
>> have you or anyone else had a chance to look at this? My investigation
>> initially pointed to scrubbing and then to an insane number of
>> hypercall preemptions in relinquish_memory()).
>
> This is another item I meant to re-engage on.  (Its on my todo list,
> along with CPUID and nested virt, but looks like it is depending on my
> whishlist item of several extra hours in the day to get some of the work
> done in.)
>
> Yes.  We should do something towards fixing that.  Current performance
> measurements put a 1.5TB domain at ~14 minutes for the domain_kill
> hypercall to complete.
>
> I seem to recall some vague plans towards having per-node dirty-page
> lists, scrubbing in idle context, and on-demand scrubbing at alloc-time
> if the clean list is empty.


I have this (almost) working but then I found that the hypercall 
preemption was eating even more time than scrubbing and got distracted 
by that. And then by other things (I have attention span of a squirrel)

-boris

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Future support of 5-level paging in Xen:wq

[Juergen Gross]

On 09/12/16 00:50, Stefano Stabellini wrote:
> On Thu, 8 Dec 2016, Andrew Cooper wrote:
>> On 08/12/2016 19:18, Stefano Stabellini wrote:
>>> On Thu, 8 Dec 2016, Andrew Cooper wrote:
>>>> On 08/12/16 16:46, Juergen Gross wrote:
>>>>> The first round of (very preliminary) patches for supporting the new
>>>>> 5-level paging of future Intel x86 processors [1] has been posted to
>>>>> lkml:
>>>>>
>>>>> https://lkml.org/lkml/2016/12/8/378
>>>>>
>>>>> An explicit note has been added: "CONFIG_XEN is broken." and
>>>>> "I would appreciate help with the code."
>>>>>
>>>>> I think we should start a discussion what we want to do in future:
>>>>>
>>>>> - are we going to support 5-level paging for PV guests?
>>>>> - do we limit 5-level paging to PVH and HVM?
>>>> The 64bit PV ABI has 16TB of virtual address space just above the upper
>>>> 48-canonical boundary.
>>>>
>>>> Were Xen to support 5-level PV guests, we'd either leave the PV guest
>>>> kernel with exactly the same amount of higher half space as it currently
>>>> has, or we'd have to recompile Xen as properly position-independent and
>>>> use a different virtual range in different paging mode.
>>>>
>>>> Another pain point is the quantity of virtual address space handed away
>>>> in the ABI.  We currently had 97% of the virtual address space away to
>>>> 64bit PV guests, and frankly this is too much.  This is the wrong way
>>>> around when Xen has more management to do than the guest.  If we were to
>>>> go along the 5-level PV guests route, I will insist that there is a
>>>> rather more even split of virtual address space baked into the ABI.
>>>>
>>>> However, a big question is whether any of this effort is worth doing, in
>>>> the light of PVH.
>>> With my Aporeto hat on, I'll say that given the overwhelming amount of
>>> hardware available out there without virtualization support, this work
>>> is worth doing. I am referring to all the public cloud virtual machines,
>>> which can support Xen PV guests but cannot support PVH guests.
>>
>> Why is Xen supporting 5-level guests useful for running in a PV cloud
>> VM?  Xen doesn't run PV.
>>
>> I am not suggesting that we avoid adding 5-level support to Xen.  We
>> should absolutely do that.  The question is only whether we extend the
>> PV ABI to support 5-level PV guests.  Conceptually, its very easy to
>> have a 5-level Xen but only supporting 4-level PV guests.
>>
>> VT-x and SVM date from 2005/2006 and are now 10 years old.  I would be
>> surprised if you would find much hardware of this age in any cloud; you
>> can't by anything that old, and support contracts have probably run out
>> if you have owned that hardware for 10 years.
> 
> I am thinking that in a couple of years, we might already find VMs so
> large that to use all the memory in a nested virt scenario, we need
> 5-level PV abi support.
> 

No, I don't think so. I believe there will be no hardware capable of
5-level paging but without VMX/SVM support. Support of PVH/HVM for such
large guests should be enough. We don't need to extend PV which we want
to get rid of in Linux anyway, no?

Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Future support of 5-level paging in Xen

[Jan Beulich]

>>> On 09.12.16 at 00:40, <boris.ostrovsky@oracle.com> wrote:
> I've been working (on and off) with SGI to get one of their 32TB boxes 
> to boot and I don't think that works. We've fixed a couple of bugs but I 
> don't think Xen can boot with that much memory. We successfully booted 
> with just under 8TB but couldn't do it with the full system. The machine 
> has been taken from us for now so this work is on hold.

I've certainly seen logs of Xen successfully booting on a 24Tb system

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Future support of 5-level paging in Xen

[Jan Beulich]

>>> On 08.12.16 at 18:22, <andrew.cooper3@citrix.com> wrote:
> On 08/12/16 16:46, Juergen Gross wrote:
>> The first round of (very preliminary) patches for supporting the new
>> 5-level paging of future Intel x86 processors [1] has been posted to
>> lkml:
>>
>> https://lkml.org/lkml/2016/12/8/378 
>>
>> An explicit note has been added: "CONFIG_XEN is broken." and
>> "I would appreciate help with the code."
>>
>> I think we should start a discussion what we want to do in future:
>>
>> - are we going to support 5-level paging for PV guests?
>> - do we limit 5-level paging to PVH and HVM?
> 
> The 64bit PV ABI has 16TB of virtual address space just above the upper
> 48-canonical boundary.
> 
> Were Xen to support 5-level PV guests, we'd either leave the PV guest
> kernel with exactly the same amount of higher half space as it currently
> has, or we'd have to recompile Xen as properly position-independent and
> use a different virtual range in different paging mode.

Right; a first question though would be whether 5-level support
would be a build time selection (just like 32-bit PAE was long ago),
or runtime determined.

> Another pain point is the quantity of virtual address space handed away
> in the ABI.  We currently had 97% of the virtual address space away to
> 64bit PV guests, and frankly this is too much.  This is the wrong way
> around when Xen has more management to do than the guest.  If we were to
> go along the 5-level PV guests route, I will insist that there is a
> rather more even split of virtual address space baked into the ABI.

I agree, but we may face resistance from (Linux and other) kernel
folks.

> However, a big question is whether any of this effort is worth doing, in
> the light of PVH.

Much depends, I think, on how quickly this becomes a fully
supported feature.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Future support of 5-level paging in Xen

[Juergen Gross]

On 09/12/16 10:59, Jan Beulich wrote:
>>>> On 08.12.16 at 18:22, <andrew.cooper3@citrix.com> wrote:
>> On 08/12/16 16:46, Juergen Gross wrote:
>>> The first round of (very preliminary) patches for supporting the new
>>> 5-level paging of future Intel x86 processors [1] has been posted to
>>> lkml:
>>>
>>> https://lkml.org/lkml/2016/12/8/378 
>>>
>>> An explicit note has been added: "CONFIG_XEN is broken." and
>>> "I would appreciate help with the code."
>>>
>>> I think we should start a discussion what we want to do in future:
>>>
>>> - are we going to support 5-level paging for PV guests?
>>> - do we limit 5-level paging to PVH and HVM?
>>
>> The 64bit PV ABI has 16TB of virtual address space just above the upper
>> 48-canonical boundary.
>>
>> Were Xen to support 5-level PV guests, we'd either leave the PV guest
>> kernel with exactly the same amount of higher half space as it currently
>> has, or we'd have to recompile Xen as properly position-independent and
>> use a different virtual range in different paging mode.
> 
> Right; a first question though would be whether 5-level support
> would be a build time selection (just like 32-bit PAE was long ago),
> or runtime determined.

Guessing you mean Linux kernel here: the intention is to have one kernel
being capable to use 4- or 5-level paging in the end. Current patches
don't support this, they are using a Kconfig option to select the number
of page table levels.

>> Another pain point is the quantity of virtual address space handed away
>> in the ABI.  We currently had 97% of the virtual address space away to
>> 64bit PV guests, and frankly this is too much.  This is the wrong way
>> around when Xen has more management to do than the guest.  If we were to
>> go along the 5-level PV guests route, I will insist that there is a
>> rather more even split of virtual address space baked into the ABI.
> 
> I agree, but we may face resistance from (Linux and other) kernel
> folks.
> 
>> However, a big question is whether any of this effort is worth doing, in
>> the light of PVH.
> 
> Much depends, I think, on how quickly this becomes a fully
> supported feature.

"this" refers to PVH, I guess?


Juergen


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Future support of 5-level paging in Xen

[Jan Beulich]

>>> On 09.12.16 at 11:07, <JGross@suse.com> wrote:
> On 09/12/16 10:59, Jan Beulich wrote:
>> Right; a first question though would be whether 5-level support
>> would be a build time selection (just like 32-bit PAE was long ago),
>> or runtime determined.
> 
> Guessing you mean Linux kernel here: the intention is to have one kernel
> being capable to use 4- or 5-level paging in the end. Current patches
> don't support this, they are using a Kconfig option to select the number
> of page table levels.

Good to know, but the reference was really to Xen itself.

>>> However, a big question is whether any of this effort is worth doing, in
>>> the light of PVH.
>> 
>> Much depends, I think, on how quickly this becomes a fully
>> supported feature.
> 
> "this" refers to PVH, I guess?

Yes.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Future support of 5-level paging in Xen:wq

[Stefano Stabellini]

On Fri, 9 Dec 2016, Juergen Gross wrote:
> On 09/12/16 00:50, Stefano Stabellini wrote:
> > On Thu, 8 Dec 2016, Andrew Cooper wrote:
> >> On 08/12/2016 19:18, Stefano Stabellini wrote:
> >>> On Thu, 8 Dec 2016, Andrew Cooper wrote:
> >>>> On 08/12/16 16:46, Juergen Gross wrote:
> >>>>> The first round of (very preliminary) patches for supporting the new
> >>>>> 5-level paging of future Intel x86 processors [1] has been posted to
> >>>>> lkml:
> >>>>>
> >>>>> https://lkml.org/lkml/2016/12/8/378
> >>>>>
> >>>>> An explicit note has been added: "CONFIG_XEN is broken." and
> >>>>> "I would appreciate help with the code."
> >>>>>
> >>>>> I think we should start a discussion what we want to do in future:
> >>>>>
> >>>>> - are we going to support 5-level paging for PV guests?
> >>>>> - do we limit 5-level paging to PVH and HVM?
> >>>> The 64bit PV ABI has 16TB of virtual address space just above the upper
> >>>> 48-canonical boundary.
> >>>>
> >>>> Were Xen to support 5-level PV guests, we'd either leave the PV guest
> >>>> kernel with exactly the same amount of higher half space as it currently
> >>>> has, or we'd have to recompile Xen as properly position-independent and
> >>>> use a different virtual range in different paging mode.
> >>>>
> >>>> Another pain point is the quantity of virtual address space handed away
> >>>> in the ABI.  We currently had 97% of the virtual address space away to
> >>>> 64bit PV guests, and frankly this is too much.  This is the wrong way
> >>>> around when Xen has more management to do than the guest.  If we were to
> >>>> go along the 5-level PV guests route, I will insist that there is a
> >>>> rather more even split of virtual address space baked into the ABI.
> >>>>
> >>>> However, a big question is whether any of this effort is worth doing, in
> >>>> the light of PVH.
> >>> With my Aporeto hat on, I'll say that given the overwhelming amount of
> >>> hardware available out there without virtualization support, this work
> >>> is worth doing. I am referring to all the public cloud virtual machines,
> >>> which can support Xen PV guests but cannot support PVH guests.
> >>
> >> Why is Xen supporting 5-level guests useful for running in a PV cloud
> >> VM?  Xen doesn't run PV.
> >>
> >> I am not suggesting that we avoid adding 5-level support to Xen.  We
> >> should absolutely do that.  The question is only whether we extend the
> >> PV ABI to support 5-level PV guests.  Conceptually, its very easy to
> >> have a 5-level Xen but only supporting 4-level PV guests.
> >>
> >> VT-x and SVM date from 2005/2006 and are now 10 years old.  I would be
> >> surprised if you would find much hardware of this age in any cloud; you
> >> can't by anything that old, and support contracts have probably run out
> >> if you have owned that hardware for 10 years.
> > 
> > I am thinking that in a couple of years, we might already find VMs so
> > large that to use all the memory in a nested virt scenario, we need
> > 5-level PV abi support.
> > 
> 
> No, I don't think so. I believe there will be no hardware capable of
> 5-level paging but without VMX/SVM support. Support of PVH/HVM for such
> large guests should be enough. We don't need to extend PV which we want
> to get rid of in Linux anyway, no?

I am talking about nested virtualization when the L1 virtual machine
does not support nested VMX or SVM. No Amazon AWS virtual machines
support nested VMX, but it is possible to run Xen PV virtual machines on
top of any Amazon HVM instance.

When 5-level pagetable hardware will become available on Amazon AWS, it
might be possible to get virtual machines so large that in order to use
all the memory, you need to use 5-level pagetables in L1 Xen. In this
scenario, if we want to create a L2 virtual machine as large as possible
we will need support for 5-level page tables in the PV ABI.

Please correct me if I am wrong.


P.S.
I used the following terminology:

L0 Xen is the one running on the hardware
L1 virtual machine, is a VM created by L0 Xen. In this context this is
   an Amazon AWS HVM instance without nested VMX support.
L1 Xen is the one installed inside an L1 virtual machine
L2 virtual machine, is a VM created by L1 Xen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Future support of 5-level paging in Xen:wq

[Andrew Cooper]

On 09/12/16 19:31, Stefano Stabellini wrote:
> On Fri, 9 Dec 2016, Juergen Gross wrote:
>> On 09/12/16 00:50, Stefano Stabellini wrote:
>>> On Thu, 8 Dec 2016, Andrew Cooper wrote:
>>>> On 08/12/2016 19:18, Stefano Stabellini wrote:
>>>>> On Thu, 8 Dec 2016, Andrew Cooper wrote:
>>>>>> On 08/12/16 16:46, Juergen Gross wrote:
>>>>>>> The first round of (very preliminary) patches for supporting the new
>>>>>>> 5-level paging of future Intel x86 processors [1] has been posted to
>>>>>>> lkml:
>>>>>>>
>>>>>>> https://lkml.org/lkml/2016/12/8/378
>>>>>>>
>>>>>>> An explicit note has been added: "CONFIG_XEN is broken." and
>>>>>>> "I would appreciate help with the code."
>>>>>>>
>>>>>>> I think we should start a discussion what we want to do in future:
>>>>>>>
>>>>>>> - are we going to support 5-level paging for PV guests?
>>>>>>> - do we limit 5-level paging to PVH and HVM?
>>>>>> The 64bit PV ABI has 16TB of virtual address space just above the upper
>>>>>> 48-canonical boundary.
>>>>>>
>>>>>> Were Xen to support 5-level PV guests, we'd either leave the PV guest
>>>>>> kernel with exactly the same amount of higher half space as it currently
>>>>>> has, or we'd have to recompile Xen as properly position-independent and
>>>>>> use a different virtual range in different paging mode.
>>>>>>
>>>>>> Another pain point is the quantity of virtual address space handed away
>>>>>> in the ABI.  We currently had 97% of the virtual address space away to
>>>>>> 64bit PV guests, and frankly this is too much.  This is the wrong way
>>>>>> around when Xen has more management to do than the guest.  If we were to
>>>>>> go along the 5-level PV guests route, I will insist that there is a
>>>>>> rather more even split of virtual address space baked into the ABI.
>>>>>>
>>>>>> However, a big question is whether any of this effort is worth doing, in
>>>>>> the light of PVH.
>>>>> With my Aporeto hat on, I'll say that given the overwhelming amount of
>>>>> hardware available out there without virtualization support, this work
>>>>> is worth doing. I am referring to all the public cloud virtual machines,
>>>>> which can support Xen PV guests but cannot support PVH guests.
>>>> Why is Xen supporting 5-level guests useful for running in a PV cloud
>>>> VM?  Xen doesn't run PV.
>>>>
>>>> I am not suggesting that we avoid adding 5-level support to Xen.  We
>>>> should absolutely do that.  The question is only whether we extend the
>>>> PV ABI to support 5-level PV guests.  Conceptually, its very easy to
>>>> have a 5-level Xen but only supporting 4-level PV guests.
>>>>
>>>> VT-x and SVM date from 2005/2006 and are now 10 years old.  I would be
>>>> surprised if you would find much hardware of this age in any cloud; you
>>>> can't by anything that old, and support contracts have probably run out
>>>> if you have owned that hardware for 10 years.
>>> I am thinking that in a couple of years, we might already find VMs so
>>> large that to use all the memory in a nested virt scenario, we need
>>> 5-level PV abi support.
>>>
>> No, I don't think so. I believe there will be no hardware capable of
>> 5-level paging but without VMX/SVM support. Support of PVH/HVM for such
>> large guests should be enough. We don't need to extend PV which we want
>> to get rid of in Linux anyway, no?
> I am talking about nested virtualization when the L1 virtual machine
> does not support nested VMX or SVM. No Amazon AWS virtual machines
> support nested VMX, but it is possible to run Xen PV virtual machines on
> top of any Amazon HVM instance.
>
> When 5-level pagetable hardware will become available on Amazon AWS, it
> might be possible to get virtual machines so large that in order to use
> all the memory, you need to use 5-level pagetables in L1 Xen. In this
> scenario, if we want to create a L2 virtual machine as large as possible
> we will need support for 5-level page tables in the PV ABI.
>
> Please correct me if I am wrong.

That is a valid scenario, although I don't think its very likely to happen.

Intel currently tops out at 46 bits physical (64TB), according to the
whitepaper, while a lot of AMD hardware has 48 bits physical (256TB). I
dread to think how much AWS would charge you for that much RAM, or how
much Amazon would be charged to buy such a server in the first place.

This is more RAM that Xen can currently handle, and isn't going to
change without breaking the current ABI.

Also, given the rise of virtualisation-based security solutions even by
Microsoft themselves in Windows 10, I think the chances are good that
you will be able to get VMs with nested virt before being able to get
VMs large enough to need 5-level paging.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Future support of 5-level paging in Xen:wq

[Juergen Gross]

On 09/12/16 20:31, Stefano Stabellini wrote:
> On Fri, 9 Dec 2016, Juergen Gross wrote:
>> On 09/12/16 00:50, Stefano Stabellini wrote:
>>> On Thu, 8 Dec 2016, Andrew Cooper wrote:
>>>> On 08/12/2016 19:18, Stefano Stabellini wrote:
>>>>> On Thu, 8 Dec 2016, Andrew Cooper wrote:
>>>>>> On 08/12/16 16:46, Juergen Gross wrote:
>>>>>>> The first round of (very preliminary) patches for supporting the new
>>>>>>> 5-level paging of future Intel x86 processors [1] has been posted to
>>>>>>> lkml:
>>>>>>>
>>>>>>> https://lkml.org/lkml/2016/12/8/378
>>>>>>>
>>>>>>> An explicit note has been added: "CONFIG_XEN is broken." and
>>>>>>> "I would appreciate help with the code."
>>>>>>>
>>>>>>> I think we should start a discussion what we want to do in future:
>>>>>>>
>>>>>>> - are we going to support 5-level paging for PV guests?
>>>>>>> - do we limit 5-level paging to PVH and HVM?
>>>>>> The 64bit PV ABI has 16TB of virtual address space just above the upper
>>>>>> 48-canonical boundary.
>>>>>>
>>>>>> Were Xen to support 5-level PV guests, we'd either leave the PV guest
>>>>>> kernel with exactly the same amount of higher half space as it currently
>>>>>> has, or we'd have to recompile Xen as properly position-independent and
>>>>>> use a different virtual range in different paging mode.
>>>>>>
>>>>>> Another pain point is the quantity of virtual address space handed away
>>>>>> in the ABI.  We currently had 97% of the virtual address space away to
>>>>>> 64bit PV guests, and frankly this is too much.  This is the wrong way
>>>>>> around when Xen has more management to do than the guest.  If we were to
>>>>>> go along the 5-level PV guests route, I will insist that there is a
>>>>>> rather more even split of virtual address space baked into the ABI.
>>>>>>
>>>>>> However, a big question is whether any of this effort is worth doing, in
>>>>>> the light of PVH.
>>>>> With my Aporeto hat on, I'll say that given the overwhelming amount of
>>>>> hardware available out there without virtualization support, this work
>>>>> is worth doing. I am referring to all the public cloud virtual machines,
>>>>> which can support Xen PV guests but cannot support PVH guests.
>>>>
>>>> Why is Xen supporting 5-level guests useful for running in a PV cloud
>>>> VM?  Xen doesn't run PV.
>>>>
>>>> I am not suggesting that we avoid adding 5-level support to Xen.  We
>>>> should absolutely do that.  The question is only whether we extend the
>>>> PV ABI to support 5-level PV guests.  Conceptually, its very easy to
>>>> have a 5-level Xen but only supporting 4-level PV guests.
>>>>
>>>> VT-x and SVM date from 2005/2006 and are now 10 years old.  I would be
>>>> surprised if you would find much hardware of this age in any cloud; you
>>>> can't by anything that old, and support contracts have probably run out
>>>> if you have owned that hardware for 10 years.
>>>
>>> I am thinking that in a couple of years, we might already find VMs so
>>> large that to use all the memory in a nested virt scenario, we need
>>> 5-level PV abi support.
>>>
>>
>> No, I don't think so. I believe there will be no hardware capable of
>> 5-level paging but without VMX/SVM support. Support of PVH/HVM for such
>> large guests should be enough. We don't need to extend PV which we want
>> to get rid of in Linux anyway, no?
> 
> I am talking about nested virtualization when the L1 virtual machine
> does not support nested VMX or SVM. No Amazon AWS virtual machines
> support nested VMX, but it is possible to run Xen PV virtual machines on
> top of any Amazon HVM instance.
> 
> When 5-level pagetable hardware will become available on Amazon AWS, it
> might be possible to get virtual machines so large that in order to use
> all the memory, you need to use 5-level pagetables in L1 Xen. In this
> scenario, if we want to create a L2 virtual machine as large as possible
> we will need support for 5-level page tables in the PV ABI.
> 
> Please correct me if I am wrong.

So basically you are telling us that due to nested virt on AWS we will
never be able to drop PV support, right? Why are we pushing PVH then?


Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Future support of 5-level paging in Xen:wq

[Stefano Stabellini]

On Mon, 12 Dec 2016, Juergen Gross wrote:
> On 09/12/16 20:31, Stefano Stabellini wrote:
> > On Fri, 9 Dec 2016, Juergen Gross wrote:
> >> On 09/12/16 00:50, Stefano Stabellini wrote:
> >>> On Thu, 8 Dec 2016, Andrew Cooper wrote:
> >>>> On 08/12/2016 19:18, Stefano Stabellini wrote:
> >>>>> On Thu, 8 Dec 2016, Andrew Cooper wrote:
> >>>>>> On 08/12/16 16:46, Juergen Gross wrote:
> >>>>>>> The first round of (very preliminary) patches for supporting the new
> >>>>>>> 5-level paging of future Intel x86 processors [1] has been posted to
> >>>>>>> lkml:
> >>>>>>>
> >>>>>>> https://lkml.org/lkml/2016/12/8/378
> >>>>>>>
> >>>>>>> An explicit note has been added: "CONFIG_XEN is broken." and
> >>>>>>> "I would appreciate help with the code."
> >>>>>>>
> >>>>>>> I think we should start a discussion what we want to do in future:
> >>>>>>>
> >>>>>>> - are we going to support 5-level paging for PV guests?
> >>>>>>> - do we limit 5-level paging to PVH and HVM?
> >>>>>> The 64bit PV ABI has 16TB of virtual address space just above the upper
> >>>>>> 48-canonical boundary.
> >>>>>>
> >>>>>> Were Xen to support 5-level PV guests, we'd either leave the PV guest
> >>>>>> kernel with exactly the same amount of higher half space as it currently
> >>>>>> has, or we'd have to recompile Xen as properly position-independent and
> >>>>>> use a different virtual range in different paging mode.
> >>>>>>
> >>>>>> Another pain point is the quantity of virtual address space handed away
> >>>>>> in the ABI.  We currently had 97% of the virtual address space away to
> >>>>>> 64bit PV guests, and frankly this is too much.  This is the wrong way
> >>>>>> around when Xen has more management to do than the guest.  If we were to
> >>>>>> go along the 5-level PV guests route, I will insist that there is a
> >>>>>> rather more even split of virtual address space baked into the ABI.
> >>>>>>
> >>>>>> However, a big question is whether any of this effort is worth doing, in
> >>>>>> the light of PVH.
> >>>>> With my Aporeto hat on, I'll say that given the overwhelming amount of
> >>>>> hardware available out there without virtualization support, this work
> >>>>> is worth doing. I am referring to all the public cloud virtual machines,
> >>>>> which can support Xen PV guests but cannot support PVH guests.
> >>>>
> >>>> Why is Xen supporting 5-level guests useful for running in a PV cloud
> >>>> VM?  Xen doesn't run PV.
> >>>>
> >>>> I am not suggesting that we avoid adding 5-level support to Xen.  We
> >>>> should absolutely do that.  The question is only whether we extend the
> >>>> PV ABI to support 5-level PV guests.  Conceptually, its very easy to
> >>>> have a 5-level Xen but only supporting 4-level PV guests.
> >>>>
> >>>> VT-x and SVM date from 2005/2006 and are now 10 years old.  I would be
> >>>> surprised if you would find much hardware of this age in any cloud; you
> >>>> can't by anything that old, and support contracts have probably run out
> >>>> if you have owned that hardware for 10 years.
> >>>
> >>> I am thinking that in a couple of years, we might already find VMs so
> >>> large that to use all the memory in a nested virt scenario, we need
> >>> 5-level PV abi support.
> >>>
> >>
> >> No, I don't think so. I believe there will be no hardware capable of
> >> 5-level paging but without VMX/SVM support. Support of PVH/HVM for such
> >> large guests should be enough. We don't need to extend PV which we want
> >> to get rid of in Linux anyway, no?
> > 
> > I am talking about nested virtualization when the L1 virtual machine
> > does not support nested VMX or SVM. No Amazon AWS virtual machines
> > support nested VMX, but it is possible to run Xen PV virtual machines on
> > top of any Amazon HVM instance.
> > 
> > When 5-level pagetable hardware will become available on Amazon AWS, it
> > might be possible to get virtual machines so large that in order to use
> > all the memory, you need to use 5-level pagetables in L1 Xen. In this
> > scenario, if we want to create a L2 virtual machine as large as possible
> > we will need support for 5-level page tables in the PV ABI.
> > 
> > Please correct me if I am wrong.
> 
> So basically you are telling us that due to nested virt on AWS we will
> never be able to drop PV support, right?

That's right, although never say never :-)


> Why are we pushing PVH then?

Because it is significantly faster than PV on most workloads on real
hardware, while retaining the security properties of PV. Definitely a
vital option to have on platforms that have VMX or SVM.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Future support of 5-level paging in Xen:wq

[Konrad Rzeszutek Wilk]

> > So basically you are telling us that due to nested virt on AWS we will
> > never be able to drop PV support, right?
> 
> That's right, although never say never :-)

Unless cloud provides exposes VMX to the guests (nested virtualization).

> 
> 
> > Why are we pushing PVH then?
> 
> Because it is significantly faster than PV on most workloads on real
> hardware, while retaining the security properties of PV. Definitely a
> vital option to have on platforms that have VMX or SVM.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Future support of 5-level paging in Xen:wq

[Li, Liang Z]

Hi All,

We are now working on enabling 5 level paging & 5 level EPT for XEN. We need the community's opinions about the following aspects:

1.  Should we enable 5 level paging for PV guest ? (You are discussing)
2.  Should we make the 5 level paging and 4 level paging supported with one binary?
3.  When should the 5 level EPT be enabled?
4. Should we extend shadow page to support 5 level paging guest?

Opinions and comments are welcome!

Thanks!
Liang


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Future support of 5-level paging in Xen:wq

[Tian, Kevin]

> From: Li, Liang Z
> Sent: Tuesday, December 13, 2016 11:58 AM
> 
> Hi All,
> 
> We are now working on enabling 5 level paging & 5 level EPT for XEN. We need the
> community's opinions about the following aspects:
> 
> 1.  Should we enable 5 level paging for PV guest ? (You are discussing)
> 2.  Should we make the 5 level paging and 4 level paging supported with one binary?
> 3.  When should the 5 level EPT be enabled?

Isn't it as usual i.e. let's enable once the spec is published? Or could you
elaborate any concern of asking this question?

> 4. Should we extend shadow page to support 5 level paging guest?
> 

I don't think so. EPT has been there for so many years. We can assume
EPT available on new platform which supports 5-level.

Thanks
Kevin
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Future support of 5-level paging in Xen:wq

[Zhang, Xiong Y]

> > From: Li, Liang Z
> > Sent: Tuesday, December 13, 2016 11:58 AM
> >
> > Hi All,
> >
> > We are now working on enabling 5 level paging & 5 level EPT for XEN. We need
> the
> > community's opinions about the following aspects:
> >
> > 1.  Should we enable 5 level paging for PV guest ? (You are discussing)
> > 2.  Should we make the 5 level paging and 4 level paging supported with one
> binary?
[Zhang, Xiong Y] If we plan to enable 5 level paging at runtime, we have to reconstruct xen mm code first as some constants (especially in asm-x86/config.h) are defined static for 4 level paging and are used before __start_xen.
If we plan to enable 5 level paging with separate binary, things will be much easier with CONFIG option. 

> > 3.  When should the 5 level EPT be enabled?
> 
> Isn't it as usual i.e. let's enable once the spec is published? Or could you
> elaborate any concern of asking this question?
> 
> > 4. Should we extend shadow page to support 5 level paging guest?
> >
> 
> I don't think so. EPT has been there for so many years. We can assume
> EPT available on new platform which supports 5-level.
> 
> Thanks
> Kevin
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Future support of 5-level paging in Xen:wq

[Li, Liang Z]

> Subject: RE: [Xen-devel] Future support of 5-level paging in Xen:wq
> 
> > From: Li, Liang Z
> > Sent: Tuesday, December 13, 2016 11:58 AM
> >
> > Hi All,
> >
> > We are now working on enabling 5 level paging & 5 level EPT for XEN.
> > We need the community's opinions about the following aspects:
> >
> > 1.  Should we enable 5 level paging for PV guest ? (You are
> > discussing) 2.  Should we make the 5 level paging and 4 level paging
> supported with one binary?
> > 3.  When should the 5 level EPT be enabled?
> 
> Isn't it as usual i.e. let's enable once the spec is published? Or could you
> elaborate any concern of asking this question?
> 

The concern is the performance vs complexity . Walk the 5 level EPT is slower than the 4 level EPT because
 It needs to access the memory one more time.
We can chose to enable 5 level EPT as long as the CPU support, or only when it's needed, e.g.
 to support a guest with huge amount of  memory(beyond 48bit physical address can support),
or when the host has a small amount of memory present.

The former is simple and straight forward, and the latter is better for performance but add extra
mechanism  to choose the proper EPT level used. Which one to use may dependent on the performance
gap.

Thanks!
Liang 

> > 4. Should we extend shadow page to support 5 level paging guest?
> >
> 
> I don't think so. EPT has been there for so many years. We can assume EPT
> available on new platform which supports 5-level.
> 
> Thanks
> Kevin
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Future support of 5-level paging in Xen:wq

[Jan Beulich]

>>> On 13.12.16 at 04:57, <liang.z.li@intel.com> wrote:
> 1.  Should we enable 5 level paging for PV guest ? (You are discussing)

Personally I think we should, but as you say - we haven't settled on
this yet.

> 2.  Should we make the 5 level paging and 4 level paging supported with one binary?

May I suggest to make this dependent on (possibly among other
aspects) how much of an impact the switch from using constants
to using variables has on the various code paths, namely hot ones?
Quite possibly the most flexible approach would be a Kconfig choice
between 3 build flavors (static 4 levels, static 5 levels, runtime
determined), which I would hope to not end up in too ugly overall
source code.

> 3.  When should the 5 level EPT be enabled?

I think the suggestion to avoid 5 levels when not needed for a guest
is quite reasonable. In fact, if hardware allows it, using just 3 levels
for not too big guests could be an option, too.

> 4. Should we extend shadow page to support 5 level paging guest?

That to a large part depends on what we settle on for 1).

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Future support of 5-level paging in Xen:wq

[Andrew Cooper]

On 13/12/16 05:44, Li, Liang Z wrote:
>> Subject: RE: [Xen-devel] Future support of 5-level paging in Xen:wq
>>
>>> From: Li, Liang Z
>>> Sent: Tuesday, December 13, 2016 11:58 AM
>>>
>>> Hi All,
>>>
>>> We are now working on enabling 5 level paging & 5 level EPT for XEN.
>>> We need the community's opinions about the following aspects:
>>>
>>> 1.  Should we enable 5 level paging for PV guest ? (You are
>>> discussing) 2.  Should we make the 5 level paging and 4 level paging
>> supported with one binary?
>>> 3.  When should the 5 level EPT be enabled?
>> Isn't it as usual i.e. let's enable once the spec is published? Or could you
>> elaborate any concern of asking this question?
>>
> The concern is the performance vs complexity . Walk the 5 level EPT is slower than the 4 level EPT because
>  It needs to access the memory one more time.
> We can chose to enable 5 level EPT as long as the CPU support, or only when it's needed, e.g.
>  to support a guest with huge amount of  memory(beyond 48bit physical address can support),
> or when the host has a small amount of memory present.
>
> The former is simple and straight forward, and the latter is better for performance but add extra
> mechanism  to choose the proper EPT level used. Which one to use may dependent on the performance
> gap.

In this case, at the point that we support 5 level EPT, we are going to
have to support hardware with a walk length of 3 as well as 4, so we
will have code to deal with both.

There is nothing to be gained by using a walk length of 4 when 3 will
do, and a performance penalty for doing so.

I think it would be entirely reasonable (and hopefully straightforward)
to choose the walk length based on the toolstack-provided CPUID policy
maxphysaddr.  (Certainly more straightforward than getting Xen's memory
management code to understand 5-level plain pagetables.)

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Future support of 5-level paging in Xen:wq

[George Dunlap]

> On Dec 13, 2016, at 12:16 PM, Tian, Kevin <kevin.tian@intel.com> wrote:
> 
>> From: Li, Liang Z
>> Sent: Tuesday, December 13, 2016 11:58 AM
>> 
>> Hi All,
>> 
>> We are now working on enabling 5 level paging & 5 level EPT for XEN. We need the
>> community's opinions about the following aspects:
>> 
>> 1.  Should we enable 5 level paging for PV guest ? (You are discussing)
>> 2.  Should we make the 5 level paging and 4 level paging supported with one binary?
>> 3.  When should the 5 level EPT be enabled?
> 
> Isn't it as usual i.e. let's enable once the spec is published? Or could you
> elaborate any concern of asking this question?
> 
>> 4. Should we extend shadow page to support 5 level paging guest?
>> 
> 
> I don't think so. EPT has been there for so many years. We can assume
> EPT available on new platform which supports 5-level.

Because of the quadratic nature of HAP (i.e., number of lookups becomes N x M rather than just N), there are still workloads where shadow pagetables performs better than HAP.  On my list of things to do sometime when I have time:
* Implement native superpages for shadows (ie., the shadows themselves will have superpages)
* Implement a way of automatically switching between shadow and HAP based on the workload

As Jan says, the shadow case will be necessary if we do 5-level PV guests; and I think it is still an important use case even for HVM guests (although it certainly comes second to other features mentioned here).

 -George
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Future support of 5-level paging in Xen

[anshul makkar]

On 08/12/16 23:40, Boris Ostrovsky wrote:
>
>
> On 12/08/2016 05:21 PM, Andrew Cooper wrote:
>> On 08/12/2016 19:18, Stefano Stabellini wrote:
>
>>
>>> Of course even the largest virtual machine today (2TB on Amazon AFAIK)
>>> is not close to reaching the current memory limit, but it's just a
>>> matter of time.
>>
>> /me things Oracle will have something to say about this.  I'm sure there
>> was talk about VMs larger than this at previous hackathons. XenServer
>> functions (ish, so long as you don't migrate) with 6TB VMs, although
>> starting and shutting them down feels like treacle.
>
>
>
> I've been working (on and off) with SGI to get one of their 32TB boxes 
> to boot and I don't think that works. We've fixed a couple of bugs but 
> I don't think Xen can boot with that much memory. We successfully 
> booted with just under 8TB but couldn't do it with the full system. 
> The machine has been taken from us for now so this work is on hold.
>
> This is on OVM, which is 4.4-based, we haven't tried (IIRC) latest bits.
>
> (BTW, speaking of slow starting and shutting down very large guests ---
> have you or anyone else had a chance to look at this? My investigation 
> initially pointed to scrubbing and then to an insane number of 
> hypercall preemptions in relinquish_memory()).
>
I had a quick look at it when I was working on support for large guest 
and found that scrubbing was indeed one of the issue. Just haven't got 
time to look at it in more details. Hopefully in near future, might work 
on it.

> -boris
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Future support of 5-level paging in Xen

[Boris Ostrovsky]

On 01/03/2017 12:32 PM, anshul makkar wrote:
>
>
> On 08/12/16 23:40, Boris Ostrovsky wrote:
>>
>>
>> On 12/08/2016 05:21 PM, Andrew Cooper wrote:
>>> On 08/12/2016 19:18, Stefano Stabellini wrote:
>>
>>>
>>>> Of course even the largest virtual machine today (2TB on Amazon AFAIK)
>>>> is not close to reaching the current memory limit, but it's just a
>>>> matter of time.
>>>
>>> /me things Oracle will have something to say about this.  I'm sure
>>> there
>>> was talk about VMs larger than this at previous hackathons. XenServer
>>> functions (ish, so long as you don't migrate) with 6TB VMs, although
>>> starting and shutting them down feels like treacle.
>>
>>
>>
>> I've been working (on and off) with SGI to get one of their 32TB
>> boxes to boot and I don't think that works. We've fixed a couple of
>> bugs but I don't think Xen can boot with that much memory. We
>> successfully booted with just under 8TB but couldn't do it with the
>> full system. The machine has been taken from us for now so this work
>> is on hold.
>>
>> This is on OVM, which is 4.4-based, we haven't tried (IIRC) latest bits.
>>
>> (BTW, speaking of slow starting and shutting down very large guests ---
>> have you or anyone else had a chance to look at this? My
>> investigation initially pointed to scrubbing and then to an insane
>> number of hypercall preemptions in relinquish_memory()).
>>
> I had a quick look at it when I was working on support for large guest
> and found that scrubbing was indeed one of the issue. Just haven't got
> time to look at it in more details. Hopefully in near future, might
> work on it.

If you are interested I can share with you the prototype code that I
have for improving scrubbing performance. Ping me when/if you start
looking into this.

-boris

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel