hypercalls with 64-bit results

hypercalls with 64-bit results

[Jan Beulich]

All,

several years back do_memory_op() in libxc was changed to have "long"
return type. This is because some of the sub-ops return potentially
large values as the hypercall return value (i.e. not in an argument
structure field). This change, however, didn't have the intended
effect from all I can tell, which apparently manifests in the present
two remaining ovmf failures in the staging osstest flights. Anthony
tells me that ovmf as of not very long ago puts the shared info page
at a really high address, thus making the p2m of the guest very large.
Its size gets returned by XENMEM_maximum_gpfn, as function return
value.

Since hypercalls from the tool stack are based on ioctl(), and since
ioctl() has a return type of "int", I'm afraid there's no way we can
deal with this by adjusting function return types in the libraries.
Instead we appear to need either a new privcmd ioctl or new XENMEM_*
subops (for those cases where potentially large values get returned).

Until we manage to deal with this I wonder whether we should suggest
to the ovmf folks to undo that change. I'm anyway not really
convinced this aggressive enlarging of the p2m is a good idea. There
are a number of cases in the hypervisor where we try to reduce GFN
ranges based on this upper bound, and there in particular is a loop
in mem-sharing code going all the way up to that limit. EPT P2M
dumping also has such a loop.

Jan

Re: hypercalls with 64-bit results

[Andrew Cooper]

On 16/06/2021 17:04, Jan Beulich wrote:
> All,
>
> several years back do_memory_op() in libxc was changed to have "long"
> return type. This is because some of the sub-ops return potentially
> large values as the hypercall return value (i.e. not in an argument
> structure field). This change, however, didn't have the intended
> effect from all I can tell, which apparently manifests in the present
> two remaining ovmf failures in the staging osstest flights. Anthony
> tells me that ovmf as of not very long ago puts the shared info page
> at a really high address, thus making the p2m of the guest very large.
> Its size gets returned by XENMEM_maximum_gpfn, as function return
> value.
>
> Since hypercalls from the tool stack are based on ioctl(), and since
> ioctl() has a return type of "int", I'm afraid there's no way we can
> deal with this by adjusting function return types in the libraries.
> Instead we appear to need either a new privcmd ioctl or new XENMEM_*
> subops (for those cases where potentially large values get returned).
>
> Until we manage to deal with this I wonder whether we should suggest
> to the ovmf folks to undo that change. I'm anyway not really
> convinced this aggressive enlarging of the p2m is a good idea. There
> are a number of cases in the hypervisor where we try to reduce GFN
> ranges based on this upper bound, and there in particular is a loop
> in mem-sharing code going all the way up to that limit. EPT P2M
> dumping also has such a loop.

There are multiple things in here which are disappointing, but I think
they've mostly been known already.

But I do agree that this is very much another nail in the coffin of the
ioctl ABI.

For ABIv2, there are many changes needed, and this ioctl ABI was never
going to survive, for other reasons too.  Obviously, we can't wait for
ABIv2 to fix this immediate issue.

However, I think it might be reasonable to wait for ABIv2 until we can
reasonably support VMs larger than 8T(?).

For now, I'd agree with trying to undo the change in OVMF.

~Andrew

Re: hypercalls with 64-bit results

[Juergen Gross]

[-- Attachment #1.1.1: Type: text/plain, Size: 1162 bytes --]

On 16.06.21 18:04, Jan Beulich wrote:
> All,
> 
> several years back do_memory_op() in libxc was changed to have "long"
> return type. This is because some of the sub-ops return potentially
> large values as the hypercall return value (i.e. not in an argument
> structure field). This change, however, didn't have the intended
> effect from all I can tell, which apparently manifests in the present
> two remaining ovmf failures in the staging osstest flights. Anthony
> tells me that ovmf as of not very long ago puts the shared info page
> at a really high address, thus making the p2m of the guest very large.
> Its size gets returned by XENMEM_maximum_gpfn, as function return
> value.
> 
> Since hypercalls from the tool stack are based on ioctl(), and since
> ioctl() has a return type of "int", I'm afraid there's no way we can
> deal with this by adjusting function return types in the libraries.
> Instead we appear to need either a new privcmd ioctl or new XENMEM_*
> subops (for those cases where potentially large values get returned).

I think we can just use a multicall in libxc to wrap the affected
operations.


Juergen

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 3135 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

Re: hypercalls with 64-bit results

[Jan Beulich]

On 17.06.2021 06:55, Juergen Gross wrote:
> On 16.06.21 18:04, Jan Beulich wrote:
>> Since hypercalls from the tool stack are based on ioctl(), and since
>> ioctl() has a return type of "int", I'm afraid there's no way we can
>> deal with this by adjusting function return types in the libraries.
>> Instead we appear to need either a new privcmd ioctl or new XENMEM_*
>> subops (for those cases where potentially large values get returned).
> 
> I think we can just use a multicall in libxc to wrap the affected
> operations.

Hmm, we might, if we're happy for these to then not work in HVM domains
(PVH Dom0, which still is experimental only, or PVH/HVM DomU-s using
the libraries for some purpose), or if we finally wire up multicalls in
the HVM case (there ought to be a reason why they aren't, but I have no
idea what that is).

Jan

Re: hypercalls with 64-bit results

[Jan Beulich]

On 16.06.2021 20:15, Andrew Cooper wrote:
> On 16/06/2021 17:04, Jan Beulich wrote:
>> All,
>>
>> several years back do_memory_op() in libxc was changed to have "long"
>> return type. This is because some of the sub-ops return potentially
>> large values as the hypercall return value (i.e. not in an argument
>> structure field). This change, however, didn't have the intended
>> effect from all I can tell, which apparently manifests in the present
>> two remaining ovmf failures in the staging osstest flights. Anthony
>> tells me that ovmf as of not very long ago puts the shared info page
>> at a really high address, thus making the p2m of the guest very large.
>> Its size gets returned by XENMEM_maximum_gpfn, as function return
>> value.
>>
>> Since hypercalls from the tool stack are based on ioctl(), and since
>> ioctl() has a return type of "int", I'm afraid there's no way we can
>> deal with this by adjusting function return types in the libraries.
>> Instead we appear to need either a new privcmd ioctl or new XENMEM_*
>> subops (for those cases where potentially large values get returned).
>>
>> Until we manage to deal with this I wonder whether we should suggest
>> to the ovmf folks to undo that change. I'm anyway not really
>> convinced this aggressive enlarging of the p2m is a good idea. There
>> are a number of cases in the hypervisor where we try to reduce GFN
>> ranges based on this upper bound, and there in particular is a loop
>> in mem-sharing code going all the way up to that limit. EPT P2M
>> dumping also has such a loop.
> 
> There are multiple things in here which are disappointing, but I think
> they've mostly been known already.
> 
> But I do agree that this is very much another nail in the coffin of the
> ioctl ABI.
> 
> For ABIv2, there are many changes needed, and this ioctl ABI was never
> going to survive, for other reasons too.  Obviously, we can't wait for
> ABIv2 to fix this immediate issue.
> 
> However, I think it might be reasonable to wait for ABIv2 until we can
> reasonably support VMs larger than 8T(?).

But it's not just XENMEM_maximum_gpfn that's affected; that's just the
one pointing out the underlying issue. Plus if so, shouldn't we avoid
returning values that are going to be truncated (and, as can be seen
here, then get perhaps recognized as error codes up the call chain)?

> For now, I'd agree with trying to undo the change in OVMF.

Anthony, thoughts?

Jan

Re: hypercalls with 64-bit results

[Juergen Gross]

[-- Attachment #1.1.1: Type: text/plain, Size: 1041 bytes --]

On 17.06.21 10:00, Jan Beulich wrote:
> On 17.06.2021 06:55, Juergen Gross wrote:
>> On 16.06.21 18:04, Jan Beulich wrote:
>>> Since hypercalls from the tool stack are based on ioctl(), and since
>>> ioctl() has a return type of "int", I'm afraid there's no way we can
>>> deal with this by adjusting function return types in the libraries.
>>> Instead we appear to need either a new privcmd ioctl or new XENMEM_*
>>> subops (for those cases where potentially large values get returned).
>>
>> I think we can just use a multicall in libxc to wrap the affected
>> operations.
> 
> Hmm, we might, if we're happy for these to then not work in HVM domains
> (PVH Dom0, which still is experimental only, or PVH/HVM DomU-s using
> the libraries for some purpose), or if we finally wire up multicalls in
> the HVM case (there ought to be a reason why they aren't, but I have no
> idea what that is).

Me neither, especially as on Arm they are supported.

And TBH: PVH Dom0 without multicalls might be hard anyway.


Juergen

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 3135 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

Re: hypercalls with 64-bit results

[Jan Beulich]

On 17.06.2021 10:05, Juergen Gross wrote:
> On 17.06.21 10:00, Jan Beulich wrote:
>> On 17.06.2021 06:55, Juergen Gross wrote:
>>> On 16.06.21 18:04, Jan Beulich wrote:
>>>> Since hypercalls from the tool stack are based on ioctl(), and since
>>>> ioctl() has a return type of "int", I'm afraid there's no way we can
>>>> deal with this by adjusting function return types in the libraries.
>>>> Instead we appear to need either a new privcmd ioctl or new XENMEM_*
>>>> subops (for those cases where potentially large values get returned).
>>>
>>> I think we can just use a multicall in libxc to wrap the affected
>>> operations.
>>
>> Hmm, we might, if we're happy for these to then not work in HVM domains
>> (PVH Dom0, which still is experimental only, or PVH/HVM DomU-s using
>> the libraries for some purpose), or if we finally wire up multicalls in
>> the HVM case (there ought to be a reason why they aren't, but I have no
>> idea what that is).
> 
> Me neither, especially as on Arm they are supported.
> 
> And TBH: PVH Dom0 without multicalls might be hard anyway.

Okay, let me see whether, while trying to wire them up, I run into
particular issues.

Jan

Re: hypercalls with 64-bit results

[Roger Pau Monné]

On Wed, Jun 16, 2021 at 06:04:02PM +0200, Jan Beulich wrote:
> All,
> 
> several years back do_memory_op() in libxc was changed to have "long"
> return type. This is because some of the sub-ops return potentially
> large values as the hypercall return value (i.e. not in an argument
> structure field). This change, however, didn't have the intended
> effect from all I can tell, which apparently manifests in the present
> two remaining ovmf failures in the staging osstest flights. Anthony
> tells me that ovmf as of not very long ago puts the shared info page
> at a really high address, thus making the p2m of the guest very large.
> Its size gets returned by XENMEM_maximum_gpfn, as function return
> value.
> 
> Since hypercalls from the tool stack are based on ioctl(), and since
> ioctl() has a return type of "int", I'm afraid there's no way we can
> deal with this by adjusting function return types in the libraries.
> Instead we appear to need either a new privcmd ioctl or new XENMEM_*
> subops (for those cases where potentially large values get returned).

AFAICT NetBSD and FreeBSD are not affected by this issue as the
hypercall return value is propagated to the caller using a long field
in the ioctl structure payload for hypercalls.

osdep_hypercall however should be fixed in libs/call in order to
return a long instead of an int, and wrappers around it should also be
adjusted.

Roger.

Re: hypercalls with 64-bit results

[Anthony PERARD]

On Thu, Jun 17, 2021 at 10:03:50AM +0200, Jan Beulich wrote:
> But it's not just XENMEM_maximum_gpfn that's affected; that's just the
> one pointing out the underlying issue. Plus if so, shouldn't we avoid
> returning values that are going to be truncated (and, as can be seen
> here, then get perhaps recognized as error codes up the call chain)?
> 
> > For now, I'd agree with trying to undo the change in OVMF.
> 
> Anthony, thoughts?

I can map the shared_info page somewhere else, that fine. The hard part
is figuring out where. I can probably map it just after the guest RAM
(as described by the e820 from hvmloader or Xen).

-- 
Anthony PERARD

Re: hypercalls with 64-bit results

[Jan Beulich]

On 17.06.2021 17:03, Anthony PERARD wrote:
> On Thu, Jun 17, 2021 at 10:03:50AM +0200, Jan Beulich wrote:
>> But it's not just XENMEM_maximum_gpfn that's affected; that's just the
>> one pointing out the underlying issue. Plus if so, shouldn't we avoid
>> returning values that are going to be truncated (and, as can be seen
>> here, then get perhaps recognized as error codes up the call chain)?
>>
>>> For now, I'd agree with trying to undo the change in OVMF.
>>
>> Anthony, thoughts?
> 
> I can map the shared_info page somewhere else, that fine. The hard part
> is figuring out where. I can probably map it just after the guest RAM
> (as described by the e820 from hvmloader or Xen).

Can't you put it (not necessarily immediately) next to the LAPIC
page, for example? Or the IO-APIC one? Or somewhere in that area,
which normally OSes won't use for placing BARs?

Jan

Re: hypercalls with 64-bit results

[Jan Beulich]

On 17.06.2021 10:08, Jan Beulich wrote:
> On 17.06.2021 10:05, Juergen Gross wrote:
>> On 17.06.21 10:00, Jan Beulich wrote:
>>> On 17.06.2021 06:55, Juergen Gross wrote:
>>>> On 16.06.21 18:04, Jan Beulich wrote:
>>>>> Since hypercalls from the tool stack are based on ioctl(), and since
>>>>> ioctl() has a return type of "int", I'm afraid there's no way we can
>>>>> deal with this by adjusting function return types in the libraries.
>>>>> Instead we appear to need either a new privcmd ioctl or new XENMEM_*
>>>>> subops (for those cases where potentially large values get returned).
>>>>
>>>> I think we can just use a multicall in libxc to wrap the affected
>>>> operations.
>>>
>>> Hmm, we might, if we're happy for these to then not work in HVM domains
>>> (PVH Dom0, which still is experimental only, or PVH/HVM DomU-s using
>>> the libraries for some purpose), or if we finally wire up multicalls in
>>> the HVM case (there ought to be a reason why they aren't, but I have no
>>> idea what that is).
>>
>> Me neither, especially as on Arm they are supported.
>>
>> And TBH: PVH Dom0 without multicalls might be hard anyway.
> 
> Okay, let me see whether, while trying to wire them up, I run into
> particular issues.

And, at the risk of stating the obvious, this isn't going to help 32-bit
Dom0 at all, so (assuming the immediate issue wasn't going to be taken
care of at the ovmf side) would help only one of the two tests preventing
a push right now. (I have a patch set drafted, but it's yet to be tested.
I did find though that the tool stack already uses multicalls in one
special case.)

Jan