xen-devel.lists.xenproject.org archive mirror
 help / color / mirror / Atom feed
From: Elliott Mitchell <ehem+xen@m5p.com>
To: Julien Grall <julien@xen.org>
Cc: xen-devel@lists.xenproject.org,
	Roger Pau Monn?? <royger@freebsd.org>,
	Mitchell Horne <mhorne@freebsd.org>
Subject: Re: Uses of /hypervisor memory range (was: FreeBSD/Xen/ARM issues)
Date: Fri, 14 May 2021 18:18:04 -0700	[thread overview]
Message-ID: <YJ8hTE/JbJygtVAL@mattapan.m5p.com> (raw)
In-Reply-To: <93936406-574f-7fd0-53bf-3bafaa4b1947@xen.org>

On Fri, May 14, 2021 at 09:32:10AM +0100, Julien Grall wrote:
> On 14/05/2021 03:42, Elliott Mitchell wrote:
> > 
> > Issue is what is the intended use of the memory range allocated to
> > /hypervisor in the device-tree on ARM?  What do the Xen developers plan
> > for?  What is expected?
>  From docs/misc/arm/device-tree/guest.txt:
> "
> - reg: specifies the base physical address and size of a region in
>    memory where the grant table should be mapped to, using an
>    HYPERVISOR_memory_op hypercall. The memory region is large enough to map
>    the whole grant table (it is larger or equal to 
> gnttab_max_grant_frames()).
>    This property is unnecessary when booting Dom0 using ACPI.
> "
> Effectively, this is a known space in memory that is unallocated. Not 
> all the guests will use it if they have a better way to find unallocated 
> space.

The use of "should" is generally considered strong encouragement to do
so.  A warning $something is lurking here and you may regret it if you
recklessly disobey this without knowning what is going on behind the

Whereas your language here suggests "can" is a better word since it is
simply a random unused address range.

> > Was the /hypervisor range intended *strictly* for mapping grant-tables?
> It was introduced to tell the OS a place where the grant-table could be 
> conveniently mapped.

Yet this is strange.  If any $random unused address range is acceptable,
why bother suggesting a particular one?  If this is really purely the
OS's choice, why is Xen bothering to suggest a range at all?

> > Was it intended for /hypervisor to grow over the
> > years as hardware got cheaper?
> I don't understand this question.

Going to the trouble of suggesting a range points to something going on.
I'm looking for an explanation since strange choices might hint at
something unpleasant lurking below and I should watch where I step.

> > Might it be better to deprecate the /hypervisor range and have domains
> > allocate any available address space for foreign mappings?
> It may be easy for FreeBSD to find available address space but so far 
> this has not been the case in Linux (I haven't checked the latest 
> version though).
> To be clear, an OS is free to not use the range provided in /hypervisor 
> (maybe this is not clear enough in the spec?). This was mostly 
> introduced to overcome some issues we saw in Linux when Xen on Arm was 
> introduced.

Mind if I paraphrase this?

"this is a bring-up hack for Linux which hangs around since we haven't
felt any pressure to fix the underlying Linux issue"

Is that reasonable?

> > Should the FreeBSD implementation be treating grant tables as distinct
> > from other foreign mappings?
> Both require unallocated addres space to work. IIRC FreeBSD is able to 
> find unallocated space easily, so I would recommend to use it.

That is supposed to be, but it appears there is presently a bug which has
broken the functionality on ARM.  As such, as a proper lazy developer if
I can abuse the /hypervisor address range for all foreign mappings, I

My feeling is one of two things should happen with the /hypervisor
address range:

1>  OSes could be encouraged to use it for all foreign mappings.  The
range should be dynamic in some fashion.  There could be a handy way to
allow configuring the amount of address space thus reserved.

2>  The range should be declared deprecated.  Everyone should be put on
the same page that this was a quick hack for bringing up Xen/ARM/Linux,
and really it shouldn't have escaped.

> > (is treating them the same likely to
> > induce buggy behavior on x86?)
> I will leave this answer to Roger.

This was directed towards *you*.  There is this thing here which looks
kind of odd in a vaguely unpleasant way.  I'm trying to figure out
whether I should embrace it, versus running away.

On Fri, May 14, 2021 at 12:07:53PM +0200, Roger Pau Monn?? wrote:
> On Fri, May 14, 2021 at 09:32:10AM +0100, Julien Grall wrote:
> > On 14/05/2021 03:42, Elliott Mitchell wrote:
> > > Was it intended for the /hypervisor range to dynamically scale with the
> > > size of the domain?
> > As per above, this doesn't depend on the size of the domain. Instead, this
> > depends on what sort of the backend will be present in the domain.
> It should instead scale based on the total memory on the system, ie:
> if your hardware has 4GB of RAM the unpopulated range should at least
> be: 4GB - memory of the current domain, so that it could map any
> possible page assigned to a different domain (and even then I'm not
> sure we shouldn't account for duplicated mappings).

This would be approach #1 from above.  Going fully in this direction
seems reasonable if the entire Xen/ARM team is up for this approach.
Otherwise approach #2 also seems reasonable.  Problem is the current
situation seems an unreasonable hybrid.

> > > Should the FreeBSD implementation be treating grant tables as distinct
> > > from other foreign mappings?
> > 
> > Both require unallocated addres space to work. IIRC FreeBSD is able to find
> > unallocated space easily, so I would recommend to use it.
> I agree. I think the main issue here is that there seems to be some
> bug (or behavior not understood properly) with the resource manager
> on Arm that returns an error when requesting a region anywhere in the
> memory address space, ie: [0, ~0].

I'm pretty sure there IS a bug, somewhere.  Question is whether it is in
the ARM nexus code, versus the xenpv code.  Thing is, as a lazy developer
I would love to avoid the task of fully diagnosing the bug by using an
alternative approach.

Alas, the alternative approach may not be viable longer term at which
point I want to force everyone to endure the hardship of getting this
fully fixed.   :-)

(\___(\___(\______          --=> 8-) EHM <=--          ______/)___/)___/)
 \BS (    |         ehem+sigmsg@m5p.com  PGP 87145445         |    )   /
  \_CS\   |  _____  -O #include <stddisclaimer.h> O-   _____  |   /  _/
8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445

  parent reply	other threads:[~2021-05-15  1:18 UTC|newest]

Thread overview: 11+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
     [not found] <YIhSbkfShjN/gMCe@Air-de-Roger>
     [not found] ` <YIndyh0sRqcmcMim@mattapan.m5p.com>
     [not found]   ` <YIptpndhk6MOJFod@Air-de-Roger>
     [not found]     ` <YItwHirnih6iUtRS@mattapan.m5p.com>
     [not found]       ` <YIu80FNQHKS3+jVN@Air-de-Roger>
     [not found]         ` <YJDcDjjgCsQUdsZ7@mattapan.m5p.com>
     [not found]           ` <YJURGaqAVBSYnMRf@Air-de-Roger>
     [not found]             ` <YJYem5CW/97k/e5A@mattapan.m5p.com>
     [not found]               ` <YJs/YAgB8molh7e5@mattapan.m5p.com>
     [not found]                 ` <54427968-9b13-36e6-0001-27fb49f85635@xen.org>
2021-05-14  2:42                   ` Elliott Mitchell
2021-05-14  8:32                     ` Julien Grall
2021-05-14 10:07                       ` Roger Pau Monné
2021-05-15  1:18                       ` Elliott Mitchell [this message]
2021-05-15 10:08                         ` Julien Grall
2021-05-19 23:25                           ` Stefano Stabellini
2021-05-20  9:51                             ` Julien Grall
2021-05-20 16:21                               ` Stefano Stabellini
2021-06-18 12:19                                 ` Oleksandr Andrushchenko
2021-07-03 17:17                                   ` Julien Grall
2021-07-22 20:41                                     ` Oleksandr Tyshchenko

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=YJ8hTE/JbJygtVAL@mattapan.m5p.com \
    --to=ehem+xen@m5p.com \
    --cc=julien@xen.org \
    --cc=mhorne@freebsd.org \
    --cc=royger@freebsd.org \
    --cc=xen-devel@lists.xenproject.org \
    --subject='Re: Uses of /hypervisor memory range (was: FreeBSD/Xen/ARM issues)' \


* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).