Re: Uses of /hypervisor memory range (was: FreeBSD/Xen/ARM issues)

[Julien Grall <julien@xen.org>]

Hi Elliott,

On 14/05/2021 03:42, Elliott Mitchell wrote:
> Upon thinking about it, this seems appropriate to bring to the attention
> of the Xen development list since it seems to have wider implications.
> 
> 
> On Wed, May 12, 2021 at 11:08:39AM +0100, Julien Grall wrote:
>> On 12/05/2021 03:37, Elliott Mitchell wrote:
>>>
>>> What about the approach to the grant-table/xenpv memory situation?
>>>
>>> As stated for a 768MB VM, Xen suggested a 16MB range.  I'm unsure whether
>>> that is strictly meant for grant-table use or is meant for any foreign
>>> memory mappings (Julien?).
>>
>> An OS is free to use it as it wants. However, there is no promise that:
>>     1) The region will not shrink
>>     2) The region will stay where it is
> 
> Issue is what is the intended use of the memory range allocated to
> /hypervisor in the device-tree on ARM?  What do the Xen developers plan
> for?  What is expected?

 From docs/misc/arm/device-tree/guest.txt:

"
- reg: specifies the base physical address and size of a region in
   memory where the grant table should be mapped to, using an
   HYPERVISOR_memory_op hypercall. The memory region is large enough to map
   the whole grant table (it is larger or equal to 
gnttab_max_grant_frames()).
   This property is unnecessary when booting Dom0 using ACPI.
"

Effectively, this is a known space in memory that is unallocated. Not 
all the guests will use it if they have a better way to find unallocated 
space.

> 
> 
> With FreeBSD, Julien Grall's attempt 5 years ago at getting Xen/ARM
> support treated the grant table as distinct from other foreign memory
> mappings.  Yet for the current code (which is oriented towards x86) it is
> rather easier to treat all foreign mappings the same.
> 
> Limiting foreign mappings to a total of 16MB for a 768MB domain is tight.

It is not clear to me whether you are referring to frontend or backend 
domain.

However, there is no relation between the size of a domain and how many 
foreign pages it will map. You can have a tiny backend (let say 128MB of 
RAM) that will handle a large domain (e.g. 2GB).

Instead, it depends on the maximum number of pages that will be mapped 
at a given point. If you are running a device emulator, then it is more 
convenient to try to keep as many foreign pages as possible mapped.

For PV backend (e.g. block, net), they tend to use grant mapping. Most 
of the time they are ephemeral (they last for the duration of the 
requests) but in some cases they will be kept mapped for the longer (for 
instance the block backend may support persistent grant).

> Was the /hypervisor range intended *strictly* for mapping grant-tables?

It was introduced to tell the OS a place where the grant-table could be 
conveniently mapped.

> Was it intended for the /hypervisor range to dynamically scale with the
> size of the domain? 
As per above, this doesn't depend on the size of the domain. Instead, 
this depends on what sort of the backend will be present in the domain.

> Was it intended for /hypervisor to grow over the
> years as hardware got cheaper?
I don't understand this question.

> Might it be better to deprecate the /hypervisor range and have domains
> allocate any available address space for foreign mappings?

It may be easy for FreeBSD to find available address space but so far 
this has not been the case in Linux (I haven't checked the latest 
version though).

To be clear, an OS is free to not use the range provided in /hypervisor 
(maybe this is not clear enough in the spec?). This was mostly 
introduced to overcome some issues we saw in Linux when Xen on Arm was 
introduced.

> 
> Should the FreeBSD implementation be treating grant tables as distinct
> from other foreign mappings?

Both require unallocated addres space to work. IIRC FreeBSD is able to 
find unallocated space easily, so I would recommend to use it.

> (is treating them the same likely to
> induce buggy behavior on x86?)

I will leave this answer to Roger.

Cheers,

-- 
Julien Grall