issue with domains having close to or more than 512G

issue with domains having close to or more than 512G

[Jan Beulich]

Keir,

the ordering of elements in the initial virtual memory arrangement is part of
the official ABI, yet I would think some parts of it may need changing:

When the complete space occupied by the initial mapping exceeds 1G,
Linux won't cope at present, and obviously as soon as it exceeds 2G it
really never can't with the current layout. 1G is reached when there is
close to 512G available to the domain. I think that the only way to deal
with this is to move the initial p2m map out of that range, at least in the
case where it would cause the entire mapping to exceed 1G (or, if
moving start_info before p2m, then 2G could be considered the limit
here, but it would require fixes to Linux anyway).

Are you aware of kernel side dependencies on this layout other than the
initial page table setup needing to be made aware of the possibility of
the p2m map sitting other than between kernel start and initial page
tables end?

Thanks, Jan

Re: issue with domains having close to or more than 512G

[Keir Fraser]

I think we should keep the current layout up to the point that the required
address space wraps past zero. Beyond that we can flag a new layout.

Who's surprised that a domain won't boot with nearly 512GB of memory? It's
never been tested, and it's not immediately an argument for messing with the
memory layout, unless it clearly will not fit.

 -- Keir

On 18/12/2008 10:39, "Jan Beulich" <jbeulich@novell.com> wrote:

> Keir,
> 
> the ordering of elements in the initial virtual memory arrangement is part of
> the official ABI, yet I would think some parts of it may need changing:
> 
> When the complete space occupied by the initial mapping exceeds 1G,
> Linux won't cope at present, and obviously as soon as it exceeds 2G it
> really never can't with the current layout. 1G is reached when there is
> close to 512G available to the domain. I think that the only way to deal
> with this is to move the initial p2m map out of that range, at least in the
> case where it would cause the entire mapping to exceed 1G (or, if
> moving start_info before p2m, then 2G could be considered the limit
> here, but it would require fixes to Linux anyway).
> 
> Are you aware of kernel side dependencies on this layout other than the
> initial page table setup needing to be made aware of the possibility of
> the p2m map sitting other than between kernel start and initial page
> tables end?
> 
> Thanks, Jan
> 

Re: issue with domains having close to or more than 512G

[Jan Beulich]

>>> Keir Fraser <keir.fraser@eu.citrix.com> 18.12.08 12:20 >>>
>I think we should keep the current layout up to the point that the required
>address space wraps past zero. Beyond that we can flag a new layout.
>
>Who's surprised that a domain won't boot with nearly 512GB of memory? It's
>never been tested, and it's not immediately an argument for messing with the
>memory layout, unless it clearly will not fit.

Making wrapping the condition for a change doesn't mean Linux can go
unfixed, whereas if we set the condition lower (e.g. needing more than one
L3 table entry, which is where things currently break) the fixes in Linux can
be restricted to just make the new layout work. In any case, we got a bug
report on 1Tb not working with Dom0, so we need to do something about
it (beyond intermediately telling them to restrict Dom0 to about 500G,
and perhaps making older Xen simply not give out more than what Linux
currently can cope with).

But you didn't respond in any way to this:

>> Are you aware of kernel side dependencies on this layout other than the
>> initial page table setup needing to be made aware of the possibility of
>> the p2m map sitting other than between kernel start and initial page
>> tables end?

I was actually hoping to get some agreement on the route to go here, so
I could try to implement a solution acceptable to all. One of the issues I
can't deal with just by myself is that I'm not really clear what requirements
non-Linux Dom0 have. For Linux alone, I'd put the p2m map into the
address space covered by the second from last L4 table entry...

Jan

Re: issue with domains having close to or more than 512G

[Keir Fraser]

On 18/12/2008 12:51, "Jan Beulich" <jbeulich@novell.com> wrote:

> Making wrapping the condition for a change doesn't mean Linux can go
> unfixed, whereas if we set the condition lower (e.g. needing more than one
> L3 table entry, which is where things currently break) the fixes in Linux can
> be restricted to just make the new layout work. In any case, we got a bug
> report on 1Tb not working with Dom0, so we need to do something about
> it (beyond intermediately telling them to restrict Dom0 to about 500G,
> and perhaps making older Xen simply not give out more than what Linux
> currently can cope with).

It will mean that non-Linux PV guests may need 'fixing' for a new layout
when they otherwise wouldn't. Hence we should stretch the old layout as far
as it'll go, at least for non-Linux PV.

> I was actually hoping to get some agreement on the route to go here, so
> I could try to implement a solution acceptable to all. One of the issues I
> can't deal with just by myself is that I'm not really clear what requirements
> non-Linux Dom0 have. For Linux alone, I'd put the p2m map into the
> address space covered by the second from last L4 table entry...

If you simply want the p2m mapped put elsewhere, I think the guest should
advertise that location in its Elf headers. Then in fact the builder can
always use the new layout if it sees the new Elf header format. That allows
Linux to use the new layout in all cases (including the buggy case where
more than one L3 is used) while not needlessly risking breakage for
non-Linux guests.

 -- Keir