Uses of /hypervisor memory range (was: FreeBSD/Xen/ARM issues)

Uses of /hypervisor memory range (was: FreeBSD/Xen/ARM issues)

[Elliott Mitchell]

Upon thinking about it, this seems appropriate to bring to the attention
of the Xen development list since it seems to have wider implications.


On Wed, May 12, 2021 at 11:08:39AM +0100, Julien Grall wrote:
> On 12/05/2021 03:37, Elliott Mitchell wrote:
> > 
> > What about the approach to the grant-table/xenpv memory situation?
> > 
> > As stated for a 768MB VM, Xen suggested a 16MB range.  I'm unsure whether
> > that is strictly meant for grant-table use or is meant for any foreign
> > memory mappings (Julien?).
> 
> An OS is free to use it as it wants. However, there is no promise that:
>    1) The region will not shrink
>    2) The region will stay where it is

Issue is what is the intended use of the memory range allocated to
/hypervisor in the device-tree on ARM?  What do the Xen developers plan
for?  What is expected?


With FreeBSD, Julien Grall's attempt 5 years ago at getting Xen/ARM
support treated the grant table as distinct from other foreign memory
mappings.  Yet for the current code (which is oriented towards x86) it is
rather easier to treat all foreign mappings the same.

Limiting foreign mappings to a total of 16MB for a 768MB domain is tight.
Was the /hypervisor range intended *strictly* for mapping grant-tables?
Was it intended for the /hypervisor range to dynamically scale with the
size of the domain?  Was it intended for /hypervisor to grow over the
years as hardware got cheaper?

Might it be better to deprecate the /hypervisor range and have domains
allocate any available address space for foreign mappings?

Should the FreeBSD implementation be treating grant tables as distinct
from other foreign mappings?  (is treating them the same likely to
induce buggy behavior on x86?)


-- 
(\___(\___(\______          --=> 8-) EHM <=--          ______/)___/)___/)
 \BS (    |         ehem+sigmsg@m5p.com  PGP 87145445         |    )   /
  \_CS\   |  _____  -O #include <stddisclaimer.h> O-   _____  |   /  _/
8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445

Re: Uses of /hypervisor memory range (was: FreeBSD/Xen/ARM issues)

[Julien Grall]

Hi Elliott,

On 14/05/2021 03:42, Elliott Mitchell wrote:
> Upon thinking about it, this seems appropriate to bring to the attention
> of the Xen development list since it seems to have wider implications.
> 
> 
> On Wed, May 12, 2021 at 11:08:39AM +0100, Julien Grall wrote:
>> On 12/05/2021 03:37, Elliott Mitchell wrote:
>>>
>>> What about the approach to the grant-table/xenpv memory situation?
>>>
>>> As stated for a 768MB VM, Xen suggested a 16MB range.  I'm unsure whether
>>> that is strictly meant for grant-table use or is meant for any foreign
>>> memory mappings (Julien?).
>>
>> An OS is free to use it as it wants. However, there is no promise that:
>>     1) The region will not shrink
>>     2) The region will stay where it is
> 
> Issue is what is the intended use of the memory range allocated to
> /hypervisor in the device-tree on ARM?  What do the Xen developers plan
> for?  What is expected?

 From docs/misc/arm/device-tree/guest.txt:

"
- reg: specifies the base physical address and size of a region in
   memory where the grant table should be mapped to, using an
   HYPERVISOR_memory_op hypercall. The memory region is large enough to map
   the whole grant table (it is larger or equal to 
gnttab_max_grant_frames()).
   This property is unnecessary when booting Dom0 using ACPI.
"

Effectively, this is a known space in memory that is unallocated. Not 
all the guests will use it if they have a better way to find unallocated 
space.

> 
> 
> With FreeBSD, Julien Grall's attempt 5 years ago at getting Xen/ARM
> support treated the grant table as distinct from other foreign memory
> mappings.  Yet for the current code (which is oriented towards x86) it is
> rather easier to treat all foreign mappings the same.
> 
> Limiting foreign mappings to a total of 16MB for a 768MB domain is tight.

It is not clear to me whether you are referring to frontend or backend 
domain.

However, there is no relation between the size of a domain and how many 
foreign pages it will map. You can have a tiny backend (let say 128MB of 
RAM) that will handle a large domain (e.g. 2GB).

Instead, it depends on the maximum number of pages that will be mapped 
at a given point. If you are running a device emulator, then it is more 
convenient to try to keep as many foreign pages as possible mapped.

For PV backend (e.g. block, net), they tend to use grant mapping. Most 
of the time they are ephemeral (they last for the duration of the 
requests) but in some cases they will be kept mapped for the longer (for 
instance the block backend may support persistent grant).

> Was the /hypervisor range intended *strictly* for mapping grant-tables?

It was introduced to tell the OS a place where the grant-table could be 
conveniently mapped.

> Was it intended for the /hypervisor range to dynamically scale with the
> size of the domain? 
As per above, this doesn't depend on the size of the domain. Instead, 
this depends on what sort of the backend will be present in the domain.

> Was it intended for /hypervisor to grow over the
> years as hardware got cheaper?
I don't understand this question.

> Might it be better to deprecate the /hypervisor range and have domains
> allocate any available address space for foreign mappings?

It may be easy for FreeBSD to find available address space but so far 
this has not been the case in Linux (I haven't checked the latest 
version though).

To be clear, an OS is free to not use the range provided in /hypervisor 
(maybe this is not clear enough in the spec?). This was mostly 
introduced to overcome some issues we saw in Linux when Xen on Arm was 
introduced.

> 
> Should the FreeBSD implementation be treating grant tables as distinct
> from other foreign mappings?

Both require unallocated addres space to work. IIRC FreeBSD is able to 
find unallocated space easily, so I would recommend to use it.

> (is treating them the same likely to
> induce buggy behavior on x86?)

I will leave this answer to Roger.

Cheers,

-- 
Julien Grall

Re: Uses of /hypervisor memory range (was: FreeBSD/Xen/ARM issues)

[Roger Pau Monné]

On Fri, May 14, 2021 at 09:32:10AM +0100, Julien Grall wrote:
> Hi Elliott,
> 
> On 14/05/2021 03:42, Elliott Mitchell wrote:
> > Was it intended for the /hypervisor range to dynamically scale with the
> > size of the domain?
> As per above, this doesn't depend on the size of the domain. Instead, this
> depends on what sort of the backend will be present in the domain.

It should instead scale based on the total memory on the system, ie:
if your hardware has 4GB of RAM the unpopulated range should at least
be: 4GB - memory of the current domain, so that it could map any
possible page assigned to a different domain (and even then I'm not
sure we shouldn't account for duplicated mappings).

> > Might it be better to deprecate the /hypervisor range and have domains
> > allocate any available address space for foreign mappings?
> 
> It may be easy for FreeBSD to find available address space but so far this
> has not been the case in Linux (I haven't checked the latest version
> though).
> 
> To be clear, an OS is free to not use the range provided in /hypervisor
> (maybe this is not clear enough in the spec?). This was mostly introduced to
> overcome some issues we saw in Linux when Xen on Arm was introduced.
> 
> > 
> > Should the FreeBSD implementation be treating grant tables as distinct
> > from other foreign mappings?
> 
> Both require unallocated addres space to work. IIRC FreeBSD is able to find
> unallocated space easily, so I would recommend to use it.

I agree. I think the main issue here is that there seems to be some
bug (or behavior not understood properly) with the resource manager
on Arm that returns an error when requesting a region anywhere in the
memory address space, ie: [0, ~0].

> > (is treating them the same likely to
> > induce buggy behavior on x86?)
> 
> I will leave this answer to Roger.

x86 is already treating them the same by using xenmem_alloc to request
memory to map the grant table or foreign mappings, so there's no
change on x86 in that regard.

Maybe I'm not getting that last question right.

Roger.

Re: Uses of /hypervisor memory range (was: FreeBSD/Xen/ARM issues)

[Elliott Mitchell]

On Fri, May 14, 2021 at 09:32:10AM +0100, Julien Grall wrote:
> On 14/05/2021 03:42, Elliott Mitchell wrote:
> > 
> > Issue is what is the intended use of the memory range allocated to
> > /hypervisor in the device-tree on ARM?  What do the Xen developers plan
> > for?  What is expected?
> 
>  From docs/misc/arm/device-tree/guest.txt:
> 
> "
> - reg: specifies the base physical address and size of a region in
>    memory where the grant table should be mapped to, using an
>    HYPERVISOR_memory_op hypercall. The memory region is large enough to map
>    the whole grant table (it is larger or equal to 
> gnttab_max_grant_frames()).
>    This property is unnecessary when booting Dom0 using ACPI.
> "
> 
> Effectively, this is a known space in memory that is unallocated. Not 
> all the guests will use it if they have a better way to find unallocated 
> space.

The use of "should" is generally considered strong encouragement to do
so.  A warning $something is lurking here and you may regret it if you
recklessly disobey this without knowning what is going on behind the
scenes.

Whereas your language here suggests "can" is a better word since it is
simply a random unused address range.


> > Was the /hypervisor range intended *strictly* for mapping grant-tables?
> 
> It was introduced to tell the OS a place where the grant-table could be 
> conveniently mapped.

Yet this is strange.  If any $random unused address range is acceptable,
why bother suggesting a particular one?  If this is really purely the
OS's choice, why is Xen bothering to suggest a range at all?


> > Was it intended for /hypervisor to grow over the
> > years as hardware got cheaper?
> I don't understand this question.

Going to the trouble of suggesting a range points to something going on.
I'm looking for an explanation since strange choices might hint at
something unpleasant lurking below and I should watch where I step.


> > Might it be better to deprecate the /hypervisor range and have domains
> > allocate any available address space for foreign mappings?
> 
> It may be easy for FreeBSD to find available address space but so far 
> this has not been the case in Linux (I haven't checked the latest 
> version though).
> 
> To be clear, an OS is free to not use the range provided in /hypervisor 
> (maybe this is not clear enough in the spec?). This was mostly 
> introduced to overcome some issues we saw in Linux when Xen on Arm was 
> introduced.

Mind if I paraphrase this?

"this is a bring-up hack for Linux which hangs around since we haven't
felt any pressure to fix the underlying Linux issue"

Is that reasonable?


> > Should the FreeBSD implementation be treating grant tables as distinct
> > from other foreign mappings?
> 
> Both require unallocated addres space to work. IIRC FreeBSD is able to 
> find unallocated space easily, so I would recommend to use it.

That is supposed to be, but it appears there is presently a bug which has
broken the functionality on ARM.  As such, as a proper lazy developer if
I can abuse the /hypervisor address range for all foreign mappings, I
will.

My feeling is one of two things should happen with the /hypervisor
address range:

1>  OSes could be encouraged to use it for all foreign mappings.  The
range should be dynamic in some fashion.  There could be a handy way to
allow configuring the amount of address space thus reserved.

2>  The range should be declared deprecated.  Everyone should be put on
the same page that this was a quick hack for bringing up Xen/ARM/Linux,
and really it shouldn't have escaped.


> > (is treating them the same likely to
> > induce buggy behavior on x86?)
> 
> I will leave this answer to Roger.

This was directed towards *you*.  There is this thing here which looks
kind of odd in a vaguely unpleasant way.  I'm trying to figure out
whether I should embrace it, versus running away.



On Fri, May 14, 2021 at 12:07:53PM +0200, Roger Pau Monn?? wrote:
> On Fri, May 14, 2021 at 09:32:10AM +0100, Julien Grall wrote:
> > On 14/05/2021 03:42, Elliott Mitchell wrote:
> > > Was it intended for the /hypervisor range to dynamically scale with the
> > > size of the domain?
> > As per above, this doesn't depend on the size of the domain. Instead, this
> > depends on what sort of the backend will be present in the domain.
> 
> It should instead scale based on the total memory on the system, ie:
> if your hardware has 4GB of RAM the unpopulated range should at least
> be: 4GB - memory of the current domain, so that it could map any
> possible page assigned to a different domain (and even then I'm not
> sure we shouldn't account for duplicated mappings).

This would be approach #1 from above.  Going fully in this direction
seems reasonable if the entire Xen/ARM team is up for this approach.
Otherwise approach #2 also seems reasonable.  Problem is the current
situation seems an unreasonable hybrid.

> > > Should the FreeBSD implementation be treating grant tables as distinct
> > > from other foreign mappings?
> > 
> > Both require unallocated addres space to work. IIRC FreeBSD is able to find
> > unallocated space easily, so I would recommend to use it.
> 
> I agree. I think the main issue here is that there seems to be some
> bug (or behavior not understood properly) with the resource manager
> on Arm that returns an error when requesting a region anywhere in the
> memory address space, ie: [0, ~0].

I'm pretty sure there IS a bug, somewhere.  Question is whether it is in
the ARM nexus code, versus the xenpv code.  Thing is, as a lazy developer
I would love to avoid the task of fully diagnosing the bug by using an
alternative approach.

Alas, the alternative approach may not be viable longer term at which
point I want to force everyone to endure the hardship of getting this
fully fixed.   :-)


-- 
(\___(\___(\______          --=> 8-) EHM <=--          ______/)___/)___/)
 \BS (    |         ehem+sigmsg@m5p.com  PGP 87145445         |    )   /
  \_CS\   |  _____  -O #include <stddisclaimer.h> O-   _____  |   /  _/
8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445

Re: Uses of /hypervisor memory range (was: FreeBSD/Xen/ARM issues)

[Julien Grall]

(+ Andrew, + Stefano)

On 15/05/2021 02:18, Elliott Mitchell wrote:
> On Fri, May 14, 2021 at 09:32:10AM +0100, Julien Grall wrote:
>> On 14/05/2021 03:42, Elliott Mitchell wrote:
>>>
>>> Issue is what is the intended use of the memory range allocated to
>>> /hypervisor in the device-tree on ARM?  What do the Xen developers plan
>>> for?  What is expected?
>>
>>   From docs/misc/arm/device-tree/guest.txt:
>>
>> "
>> - reg: specifies the base physical address and size of a region in
>>     memory where the grant table should be mapped to, using an
>>     HYPERVISOR_memory_op hypercall. The memory region is large enough to map
>>     the whole grant table (it is larger or equal to
>> gnttab_max_grant_frames()).
>>     This property is unnecessary when booting Dom0 using ACPI.
>> "
>>
>> Effectively, this is a known space in memory that is unallocated. Not
>> all the guests will use it if they have a better way to find unallocated
>> space.
> 
> The use of "should" is generally considered strong encouragement to do
> so.  A warning $something is lurking here and you may regret it if you
> recklessly disobey this without knowning what is going on behind the
> scenes.

I thought a bit more over night. The potential trouble I can think of 
for a domU is the magic pages are not described in the DT.

I think every other regions should be discoverable from the DT (at least 
for a domU).

> 
> Whereas your language here suggests "can" is a better word since it is
> simply a random unused address range.
> 
> 
>>> Was the /hypervisor range intended *strictly* for mapping grant-tables?
>>
>> It was introduced to tell the OS a place where the grant-table could be
>> conveniently mapped.
> 
> Yet this is strange.  If any $random unused address range is acceptable,
> why bother suggesting a particular one?  If this is really purely the
> OS's choice, why is Xen bothering to suggest a range at all?

I have added Stefano who may have more historical context than what I 
wrote in my previous e-mail.

> 
> 
>>> Was it intended for /hypervisor to grow over the
>>> years as hardware got cheaper?
>> I don't understand this question.
> 
> Going to the trouble of suggesting a range points to something going on.
> I'm looking for an explanation since strange choices might hint at
> something unpleasant lurking below and I should watch where I step.
> 
> 
>>> Might it be better to deprecate the /hypervisor range and have domains
>>> allocate any available address space for foreign mappings?
>>
>> It may be easy for FreeBSD to find available address space but so far
>> this has not been the case in Linux (I haven't checked the latest
>> version though).
>>
>> To be clear, an OS is free to not use the range provided in /hypervisor
>> (maybe this is not clear enough in the spec?). This was mostly
>> introduced to overcome some issues we saw in Linux when Xen on Arm was
>> introduced.
> 
> Mind if I paraphrase this?
> 
> "this is a bring-up hack for Linux which hangs around since we haven't
> felt any pressure to fix the underlying Linux issue"
> 
> Is that reasonable?

Yes. I have revisited the problem a few times and every time I got stuck 
because not all the I/O regions where reported to Linux. So Linux would 
not be able to find a safe unallocated space.

> 
> 
>>> Should the FreeBSD implementation be treating grant tables as distinct
>>> from other foreign mappings?
>>
>> Both require unallocated addres space to work. IIRC FreeBSD is able to
>> find unallocated space easily, so I would recommend to use it.
> 
> That is supposed to be, but it appears there is presently a bug which has
> broken the functionality on ARM.  

Do you mind share some details?

> As such, as a proper lazy developer if
> I can abuse the /hypervisor address range for all foreign mappings, I
> will.

Are you aiming to support dom0 now?

> 
> My feeling is one of two things should happen with the /hypervisor
> address range:
> 
> 1>  OSes could be encouraged to use it for all foreign mappings.  The
> range should be dynamic in some fashion.  There could be a handy way to
> allow configuring the amount of address space thus reserved.

In the context of XSA-300 and virtio on Xen on Arm, we discussed about 
providing a revion for foreign mapping. The main trouble here is 
figuring out of the size, if you mess it up then you may break all the 
PV drivers.

If the problem is finding space, then I would like to suggest a 
different approach (I think I may have discussed it with Andrew). Xen is 
in maintaining the P2M for the guest and therefore now where are the 
unallocated space. How about introducing a new hypercall to ask for 
"unallocated space"?

This would not work for older hypervisor but you could use the RAM 
instead (as Linux does). This is has drawbacks (e.g. shattering 
superpage, reducing the amount of memory usuable...), but for 1> you 
would also need a hack for older Xen.

> 
> 2>  The range should be declared deprecated.  Everyone should be put on
> the same page that this was a quick hack for bringing up Xen/ARM/Linux,
> and really it shouldn't have escaped.

How about relaxing the wording instead?

> 
> 
>>> (is treating them the same likely to
>>> induce buggy behavior on x86?)
>>
>> I will leave this answer to Roger.
> 
> This was directed towards *you*.  There is this thing here which looks
> kind of odd in a vaguely unpleasant way.  I'm trying to figure out
> whether I should embrace it, versus running away.

I am not aware of any potential buggy behavior here. In both arm and 
x86, the requirement is to find unallocated address space (unless you 
want to waste RAM as Linux does).

Cheers,

-- 
Julien Grall

Re: Uses of /hypervisor memory range (was: FreeBSD/Xen/ARM issues)

[Stefano Stabellini]

On Sat, 15 May 2021, Julien Grall wrote:
> (+ Andrew, + Stefano)
> 
> On 15/05/2021 02:18, Elliott Mitchell wrote:
> > On Fri, May 14, 2021 at 09:32:10AM +0100, Julien Grall wrote:
> > > On 14/05/2021 03:42, Elliott Mitchell wrote:
> > > > 
> > > > Issue is what is the intended use of the memory range allocated to
> > > > /hypervisor in the device-tree on ARM?  What do the Xen developers plan
> > > > for?  What is expected?
> > > 
> > >   From docs/misc/arm/device-tree/guest.txt:
> > > 
> > > "
> > > - reg: specifies the base physical address and size of a region in
> > >     memory where the grant table should be mapped to, using an
> > >     HYPERVISOR_memory_op hypercall. The memory region is large enough to
> > > map
> > >     the whole grant table (it is larger or equal to
> > > gnttab_max_grant_frames()).
> > >     This property is unnecessary when booting Dom0 using ACPI.
> > > "
> > > 
> > > Effectively, this is a known space in memory that is unallocated. Not
> > > all the guests will use it if they have a better way to find unallocated
> > > space.
> > 
> > The use of "should" is generally considered strong encouragement to do
> > so.  A warning $something is lurking here and you may regret it if you
> > recklessly disobey this without knowning what is going on behind the
> > scenes.
> 
> I thought a bit more over night. The potential trouble I can think of for a
> domU is the magic pages are not described in the DT.
> 
> I think every other regions should be discoverable from the DT (at least for a
> domU).
> 
> > Whereas your language here suggests "can" is a better word since it is
> > simply a random unused address range.
> > 
> > 
> > > > Was the /hypervisor range intended *strictly* for mapping grant-tables?
> > > 
> > > It was introduced to tell the OS a place where the grant-table could be
> > > conveniently mapped.
> > 
> > Yet this is strange.  If any $random unused address range is acceptable,
> > why bother suggesting a particular one?  If this is really purely the
> > OS's choice, why is Xen bothering to suggest a range at all?
> 
> I have added Stefano who may have more historical context than what I wrote in
> my previous e-mail.

At the time when the binding was introduced, there was a long history of
issues in finding a "safe" place where to map the grant table. Consider
that on x86 not all "empty" regions are really empty, there can be
hidden MMIO regions and with PV guests there is not even a stage2
address translation to protect you.

To stay on the safe side, we wanted to have a way to provide a range
known to be safe so that Linux wouldn't have to worry about it.


> > > > Was it intended for /hypervisor to grow over the
> > > > years as hardware got cheaper?
> > > I don't understand this question.
> > 
> > Going to the trouble of suggesting a range points to something going on.
> > I'm looking for an explanation since strange choices might hint at
> > something unpleasant lurking below and I should watch where I step.
> > 
> > 
> > > > Might it be better to deprecate the /hypervisor range and have domains
> > > > allocate any available address space for foreign mappings?
> > > 
> > > It may be easy for FreeBSD to find available address space but so far
> > > this has not been the case in Linux (I haven't checked the latest
> > > version though).
> > > 
> > > To be clear, an OS is free to not use the range provided in /hypervisor
> > > (maybe this is not clear enough in the spec?). This was mostly
> > > introduced to overcome some issues we saw in Linux when Xen on Arm was
> > > introduced.
> > 
> > Mind if I paraphrase this?
> > 
> > "this is a bring-up hack for Linux which hangs around since we haven't
> > felt any pressure to fix the underlying Linux issue"
> > 
> > Is that reasonable?
> 
> Yes. I have revisited the problem a few times and every time I got stuck
> because not all the I/O regions where reported to Linux. So Linux would not be
> able to find a safe unallocated space.

Exactly!



> > > > Should the FreeBSD implementation be treating grant tables as distinct
> > > > from other foreign mappings?
> > > 
> > > Both require unallocated addres space to work. IIRC FreeBSD is able to
> > > find unallocated space easily, so I would recommend to use it.
> > 
> > That is supposed to be, but it appears there is presently a bug which has
> > broken the functionality on ARM.  
> 
> Do you mind share some details?
> 
> > As such, as a proper lazy developer if
> > I can abuse the /hypervisor address range for all foreign mappings, I
> > will.
> 
> Are you aiming to support dom0 now?

I guess it is not quite an abuse. This was meant to be a safe address
range to do special mappings, primarily for the grant table. Maybe it
could be used for other things too.

I would be open to the idea of extending the device tree description to
say that it can be used for other mappings as well.



> > My feeling is one of two things should happen with the /hypervisor
> > address range:
> > 
> > 1>  OSes could be encouraged to use it for all foreign mappings.  The
> > range should be dynamic in some fashion.  There could be a handy way to
> > allow configuring the amount of address space thus reserved.
> 
> In the context of XSA-300 and virtio on Xen on Arm, we discussed about
> providing a revion for foreign mapping. The main trouble here is figuring out
> of the size, if you mess it up then you may break all the PV drivers.
> 
> If the problem is finding space, then I would like to suggest a different
> approach (I think I may have discussed it with Andrew). Xen is in maintaining
> the P2M for the guest and therefore now where are the unallocated space. How
> about introducing a new hypercall to ask for "unallocated space"?
> 
> This would not work for older hypervisor but you could use the RAM instead (as
> Linux does). This is has drawbacks (e.g. shattering superpage, reducing the
> amount of memory usuable...), but for 1> you would also need a hack for older
> Xen.

I am starting to wonder if it would make sense to add a new device tree
binding to describe larger free region for foreign mapping rather then a
hypercall. It could be several GBs or even TBs in size. I like the idea
of having it device tree because after all this is static information.
I can see that a hypercall would also work and I am open to it but if
possible I think it would be better not to extend the hypercall
interface when there is a good alternative.


> > 2>  The range should be declared deprecated.  Everyone should be put on
> > the same page that this was a quick hack for bringing up Xen/ARM/Linux,
> > and really it shouldn't have escaped.
> 
> How about relaxing the wording instead?

Yes, I agree!

Re: Uses of /hypervisor memory range (was: FreeBSD/Xen/ARM issues)

[Julien Grall]

Hi Stefano,

On 20/05/2021 00:25, Stefano Stabellini wrote:
> On Sat, 15 May 2021, Julien Grall wrote:
>>> My feeling is one of two things should happen with the /hypervisor
>>> address range:
>>>
>>> 1>  OSes could be encouraged to use it for all foreign mappings.  The
>>> range should be dynamic in some fashion.  There could be a handy way to
>>> allow configuring the amount of address space thus reserved.
>>
>> In the context of XSA-300 and virtio on Xen on Arm, we discussed about
>> providing a revion for foreign mapping. The main trouble here is figuring out
>> of the size, if you mess it up then you may break all the PV drivers.
>>
>> If the problem is finding space, then I would like to suggest a different
>> approach (I think I may have discussed it with Andrew). Xen is in maintaining
>> the P2M for the guest and therefore now where are the unallocated space. How
>> about introducing a new hypercall to ask for "unallocated space"?
>>
>> This would not work for older hypervisor but you could use the RAM instead (as
>> Linux does). This is has drawbacks (e.g. shattering superpage, reducing the
>> amount of memory usuable...), but for 1> you would also need a hack for older
>> Xen.
> 
> I am starting to wonder if it would make sense to add a new device tree
> binding to describe larger free region for foreign mapping rather then a
> hypercall. It could be several GBs or even TBs in size. I like the idea
> of having it device tree because after all this is static information.
> I can see that a hypercall would also work and I am open to it but if
> possible I think it would be better not to extend the hypercall
> interface when there is a good alternative.

There are two issues with the Device-Tree approach:
   1) This doesn't work on ACPI
   2) It is not clear how to define the size of the region. An admin 
doesn't have the right information in hand to know how many mappings 
will be done (a backend may map multiple time the same page).

The advantage of the hypercall solution is the size is "virtually" 
unlimited and not based on a specific firmware.

Cheers,

-- 
Julien Grall

Re: Uses of /hypervisor memory range (was: FreeBSD/Xen/ARM issues)

[Stefano Stabellini]

On Thu, 20 May 2021, Julien Grall wrote:
> On 20/05/2021 00:25, Stefano Stabellini wrote:
> > On Sat, 15 May 2021, Julien Grall wrote:
> > > > My feeling is one of two things should happen with the /hypervisor
> > > > address range:
> > > > 
> > > > 1>  OSes could be encouraged to use it for all foreign mappings.  The
> > > > range should be dynamic in some fashion.  There could be a handy way to
> > > > allow configuring the amount of address space thus reserved.
> > > 
> > > In the context of XSA-300 and virtio on Xen on Arm, we discussed about
> > > providing a revion for foreign mapping. The main trouble here is figuring
> > > out
> > > of the size, if you mess it up then you may break all the PV drivers.
> > > 
> > > If the problem is finding space, then I would like to suggest a different
> > > approach (I think I may have discussed it with Andrew). Xen is in
> > > maintaining
> > > the P2M for the guest and therefore now where are the unallocated space.
> > > How
> > > about introducing a new hypercall to ask for "unallocated space"?
> > > 
> > > This would not work for older hypervisor but you could use the RAM instead
> > > (as
> > > Linux does). This is has drawbacks (e.g. shattering superpage, reducing
> > > the
> > > amount of memory usuable...), but for 1> you would also need a hack for
> > > older
> > > Xen.
> > 
> > I am starting to wonder if it would make sense to add a new device tree
> > binding to describe larger free region for foreign mapping rather then a
> > hypercall. It could be several GBs or even TBs in size. I like the idea
> > of having it device tree because after all this is static information.
> > I can see that a hypercall would also work and I am open to it but if
> > possible I think it would be better not to extend the hypercall
> > interface when there is a good alternative.
> 
> There are two issues with the Device-Tree approach:
>   1) This doesn't work on ACPI
>   2) It is not clear how to define the size of the region. An admin doesn't
> have the right information in hand to know how many mappings will be done (a
> backend may map multiple time the same page).
> 
> The advantage of the hypercall solution is the size is "virtually" unlimited
> and not based on a specific firmware.

Fair enough

Re: Uses of /hypervisor memory range (was: FreeBSD/Xen/ARM issues)

[Oleksandr Andrushchenko]

Hi, all!

What do we need in order to move on on this?

It seems this can be good to have for many use-cases around

Thank you,

Oleksandr

On 20.05.21 19:21, Stefano Stabellini wrote:
> On Thu, 20 May 2021, Julien Grall wrote:
>> On 20/05/2021 00:25, Stefano Stabellini wrote:
>>> On Sat, 15 May 2021, Julien Grall wrote:
>>>>> My feeling is one of two things should happen with the /hypervisor
>>>>> address range:
>>>>>
>>>>> 1>  OSes could be encouraged to use it for all foreign mappings.  The
>>>>> range should be dynamic in some fashion.  There could be a handy way to
>>>>> allow configuring the amount of address space thus reserved.
>>>> In the context of XSA-300 and virtio on Xen on Arm, we discussed about
>>>> providing a revion for foreign mapping. The main trouble here is figuring
>>>> out
>>>> of the size, if you mess it up then you may break all the PV drivers.
>>>>
>>>> If the problem is finding space, then I would like to suggest a different
>>>> approach (I think I may have discussed it with Andrew). Xen is in
>>>> maintaining
>>>> the P2M for the guest and therefore now where are the unallocated space.
>>>> How
>>>> about introducing a new hypercall to ask for "unallocated space"?
>>>>
>>>> This would not work for older hypervisor but you could use the RAM instead
>>>> (as
>>>> Linux does). This is has drawbacks (e.g. shattering superpage, reducing
>>>> the
>>>> amount of memory usuable...), but for 1> you would also need a hack for
>>>> older
>>>> Xen.
>>> I am starting to wonder if it would make sense to add a new device tree
>>> binding to describe larger free region for foreign mapping rather then a
>>> hypercall. It could be several GBs or even TBs in size. I like the idea
>>> of having it device tree because after all this is static information.
>>> I can see that a hypercall would also work and I am open to it but if
>>> possible I think it would be better not to extend the hypercall
>>> interface when there is a good alternative.
>> There are two issues with the Device-Tree approach:
>>    1) This doesn't work on ACPI
>>    2) It is not clear how to define the size of the region. An admin doesn't
>> have the right information in hand to know how many mappings will be done (a
>> backend may map multiple time the same page).
>>
>> The advantage of the hypercall solution is the size is "virtually" unlimited
>> and not based on a specific firmware.
> Fair enough
>

Re: Uses of /hypervisor memory range (was: FreeBSD/Xen/ARM issues)

[Julien Grall]

On 18/06/2021 13:19, Oleksandr Andrushchenko wrote:
> Hi, all!

Hi,

Sorry for the late reply.

> What do we need in order to move on on this?

The next steps would be
  1) Write a PoC to confirm the approach is feasible
  2) Write a design document

Unfortunately, I will not have time to work on it any time soon. Is it 
something you plan to work on?

Cheers,

-- 
Julien Grall

Re: Uses of /hypervisor memory range (was: FreeBSD/Xen/ARM issues)

[Oleksandr Tyshchenko]

[-- Attachment #1: Type: text/plain, Size: 1023 bytes --]

On Sat, Jul 3, 2021 at 8:18 PM Julien Grall <julien@xen.org> wrote:

>
>
> On 18/06/2021 13:19, Oleksandr Andrushchenko wrote:
> > Hi, all!
>
> Hi,
>

Hello Julien, all.
[sorry for the possible format issues]

I am sorry to jump into the conversation too late.



>
> Sorry for the late reply.
>
> > What do we need in order to move on on this?
>
> The next steps would be
>   1) Write a PoC to confirm the approach is feasible
>   2) Write a design document
>
> Unfortunately, I will not have time to work on it any time soon. Is it
> something you plan to work on?
>

I am working on this PoC on Xen on Arm right now taking into the account
the details that we initially discussed on the IRC, unfortunately, I had to
postpone this activity several times due to the limited bandwidth,
but this time, I hope, I will be able to finish this PoC without
interruption) I will try to share Xen+Linux changes in a week, so we will
able to discuss them.


>
> Cheers,
>
> --
> Julien Grall
>
>

-- 
Regards,

Oleksandr Tyshchenko

[-- Attachment #2: Type: text/html, Size: 2249 bytes --]