Design session PVH dom0

Design session PVH dom0

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 3110 bytes --]

Session description (by Jan):
In the course of working on an XSA I had to finally get PVH Dom0 work on at least one of my systems, in a minimal fashion. This had turned up a number of issues, some of which have since remained pending. Therefore I’d like to gain understanding on whether there is any future to this mode of Dom0 operation, and if so when it can be expected to be better than tech preview or even just experimental.

Open issues off the top of my head: - PCI pass-through / SR-IOV - passing of video settings to Dom0 kernel - serial console (at least when on a plug-in PCI card)


Jürgen: PVH dom0 would be nice, related to feature parity PV vs PVH (PCI passthrough etc)

George: gitlab has related tickets
https://gitlab.com/groups/xen-project/-/epics/2
When completed, PV could be called "legacy"

Stefano: dom0less could help with PV-less setup
      fully featured vPCI required

Roger: still need to support HVM way of passthrough with qemu, otherwise some devices via vPCI and some emulated for the same guest

Stefano: qemu is not certifiable, should be avoided in cerifiable configurations

Jürgen: use qemu for virtio

Stefano: ioreq server needs to work together with vPCI

George: move from just qemu to vPCI may move devices -> Windows BSOD

Jan: no problem with qemu coexisting with vPCI

Roger: some use cases still require qemu for passthrough (HVM), if some parts are special handling

George: GVT-g(?) is such case

Marek: actually, the ioreq server is in dom0 kernel, qemu only reserves slot

Jan: host bridge, root complex emulation currently is ARM specific

George: who is going to do the work?

Jan: that's why this session, no progress, or even patch review

Stefano: there was proposal from Julien about using platform hypercalls for [???]

George: track what's happening to related patches

Stefano: PVH dom0 is useful

Roger: EPAM already do vPCI for PVH dom0, minimal configuration for Q35

Stefano: what other gaps for PVH dom0?

George: see gitlab epic

Jan: hide serial cards - if dom0 doesn't enumerate it, something gets confused (overlap check?)
Jan: video information - for PV dom0 framebuffer info is in start_info, but no equivalent for PVH

Jürgen: use Linux boot config protocol?

Jan: Xen don't speak it

Stefano: EFI services?

Jan: dom0 doesn't have boot services access

Stefano: that was video output on RPi

Jan: do hypercall to obtain the info

Roger: PVH is a thing on firecracker

George: there are definitely more issues, but the big ones are the above

Jürgen: PVH dom0 performance?

Roger: it's bad; mostly relevant is qemu interfaces

George: only for safety certifications? performance penalty may be okay

Jürgen: hypercalls can be improved (virtual buffers?)

Stefano: litte sense in performance optimization before feature complete

Roger: limited capacity what we can work on

Stefano: safety certification may motivate the effort

George: sell it this way to AMD






-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Design session PVH dom0

[Juergen Gross]

[-- Attachment #1.1.1: Type: text/plain, Size: 2088 bytes --]

On 21.09.22 17:53, Marek Marczykowski-Górecki wrote:
> Session description (by Jan):
> In the course of working on an XSA I had to finally get PVH Dom0 work on at least one of my systems, in a minimal fashion. This had turned up a number of issues, some of which have since remained pending. Therefore I’d like to gain understanding on whether there is any future to this mode of Dom0 operation, and if so when it can be expected to be better than tech preview or even just experimental.

...

> Jürgen: PVH dom0 performance?
> 
> Roger: it's bad; mostly relevant is qemu interfaces
> 
> George: only for safety certifications? performance penalty may be okay
> 
> Jürgen: hypercalls can be improved (virtual buffers?)

Some more thoughts on this topic: Having hypercall variants with physically
addressed buffers will help, but there is an additional complexity: what
about hypercalls with really large buffers (e.g. the bitmap for modified
pages for guest migration). In order to avoid having to allocate huge
physically contiguous buffers for those purposes we'd probably need
something like scatter/gather lists for hypercall buffers. And those might
want to be supported in a generic way. Additionally: what if such a SG-list
would exceed the size of a page? The dirty bitmap of a guest with 64 GB of
RAM would already need 512 pages, so the SG-list for that bitmap would already
fill a complete page assuming only 8 byte for one SG-entry (which would limit
the general usability already).

My favorite solution would be some kind of buffer address qualifier for each
buffer (e.g. virtual, physical, SG-list, maybe nested SG-list). So the new
hypercalls would not mean "physical buffer addresses", but "qualified buffer
addresses". By requiring a minimum of 4-byte alignment for each buffer (can we
do that, at least for the new hypercalls?) this would leave the 2 lowest bits
of a buffer address for the new qualifier. If by any means an unaligned buffer
is needed sometimes, it could still be achieved via a single-entry SG-list.


Juergen

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 3149 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

Re: Design session PVH dom0

[Jan Beulich]

On 23.09.2022 10:20, Juergen Gross wrote:
> On 21.09.22 17:53, Marek Marczykowski-Górecki wrote:
>> Session description (by Jan):
>> In the course of working on an XSA I had to finally get PVH Dom0 work on at least one of my systems, in a minimal fashion. This had turned up a number of issues, some of which have since remained pending. Therefore I’d like to gain understanding on whether there is any future to this mode of Dom0 operation, and if so when it can be expected to be better than tech preview or even just experimental.
> 
> ...
> 
>> Jürgen: PVH dom0 performance?
>>
>> Roger: it's bad; mostly relevant is qemu interfaces
>>
>> George: only for safety certifications? performance penalty may be okay
>>
>> Jürgen: hypercalls can be improved (virtual buffers?)
> 
> Some more thoughts on this topic: Having hypercall variants with physically
> addressed buffers will help, but there is an additional complexity: what
> about hypercalls with really large buffers (e.g. the bitmap for modified
> pages for guest migration). In order to avoid having to allocate huge
> physically contiguous buffers for those purposes we'd probably need
> something like scatter/gather lists for hypercall buffers.

Not sure. I'd rather see us add new (sub)hypercalls for such non-standard
cases. E.g. the bitmap example you give would be amended by a new flavor
having the caller pass in an array of GFNs (perhaps, as you say, with
further indirection to deal with that array also growing large). I'd
really like to keep the common case simple.

> And those might
> want to be supported in a generic way. Additionally: what if such a SG-list
> would exceed the size of a page? The dirty bitmap of a guest with 64 GB of
> RAM would already need 512 pages, so the SG-list for that bitmap would already
> fill a complete page assuming only 8 byte for one SG-entry (which would limit
> the general usability already).
> 
> My favorite solution would be some kind of buffer address qualifier for each
> buffer (e.g. virtual, physical, SG-list, maybe nested SG-list). So the new
> hypercalls would not mean "physical buffer addresses", but "qualified buffer
> addresses". By requiring a minimum of 4-byte alignment for each buffer (can we
> do that, at least for the new hypercalls?) this would leave the 2 lowest bits
> of a buffer address for the new qualifier. If by any means an unaligned buffer
> is needed sometimes, it could still be achieved via a single-entry SG-list.

While this might be an option, I'm not sure I'd be really happy with such
re-use of the low address bits, nor with the implied further restriction
on buffer alignment (most struct-s we use are 4-byte aligned at least,
but I don't think it's all of them, plus we also have guest handles to
e.g. arrays of char).

Jan

Re: Design session PVH dom0

[Juergen Gross]

[-- Attachment #1.1.1: Type: text/plain, Size: 3617 bytes --]

On 26.09.22 09:53, Jan Beulich wrote:
> On 23.09.2022 10:20, Juergen Gross wrote:
>> On 21.09.22 17:53, Marek Marczykowski-Górecki wrote:
>>> Session description (by Jan):
>>> In the course of working on an XSA I had to finally get PVH Dom0 work on at least one of my systems, in a minimal fashion. This had turned up a number of issues, some of which have since remained pending. Therefore I’d like to gain understanding on whether there is any future to this mode of Dom0 operation, and if so when it can be expected to be better than tech preview or even just experimental.
>>
>> ...
>>
>>> Jürgen: PVH dom0 performance?
>>>
>>> Roger: it's bad; mostly relevant is qemu interfaces
>>>
>>> George: only for safety certifications? performance penalty may be okay
>>>
>>> Jürgen: hypercalls can be improved (virtual buffers?)
>>
>> Some more thoughts on this topic: Having hypercall variants with physically
>> addressed buffers will help, but there is an additional complexity: what
>> about hypercalls with really large buffers (e.g. the bitmap for modified
>> pages for guest migration). In order to avoid having to allocate huge
>> physically contiguous buffers for those purposes we'd probably need
>> something like scatter/gather lists for hypercall buffers.
> 
> Not sure. I'd rather see us add new (sub)hypercalls for such non-standard
> cases. E.g. the bitmap example you give would be amended by a new flavor
> having the caller pass in an array of GFNs (perhaps, as you say, with
> further indirection to deal with that array also growing large). I'd
> really like to keep the common case simple.

The question is how many hypercalls would be hit by the not common case.

Taking a quick glance I spotted:

- grant_table_op (subops setup_table and get_status_frames)
- memory_op (several sub-ops)
- multicall (main list of calls)
- console_io (console data)
- mmuext_op (some ops allow lists)
- xsm_op (not sure a buffer can span pages, but interface would allow it)
- physdev_op (subop set_iobitmap)
- hvm_op (altp2m handling)
- sysctl (multiple sub-ops)
- domctl (multiple sub-ops)
- hypfs (node data can exceed page size)

Do we really want to special case all of those?

>> And those might
>> want to be supported in a generic way. Additionally: what if such a SG-list
>> would exceed the size of a page? The dirty bitmap of a guest with 64 GB of
>> RAM would already need 512 pages, so the SG-list for that bitmap would already
>> fill a complete page assuming only 8 byte for one SG-entry (which would limit
>> the general usability already).
>>
>> My favorite solution would be some kind of buffer address qualifier for each
>> buffer (e.g. virtual, physical, SG-list, maybe nested SG-list). So the new
>> hypercalls would not mean "physical buffer addresses", but "qualified buffer
>> addresses". By requiring a minimum of 4-byte alignment for each buffer (can we
>> do that, at least for the new hypercalls?) this would leave the 2 lowest bits
>> of a buffer address for the new qualifier. If by any means an unaligned buffer
>> is needed sometimes, it could still be achieved via a single-entry SG-list.
> 
> While this might be an option, I'm not sure I'd be really happy with such
> re-use of the low address bits, nor with the implied further restriction
> on buffer alignment (most struct-s we use are 4-byte aligned at least,
> but I don't think it's all of them, plus we also have guest handles to
> e.g. arrays of char).

The unaligned cases could be handled dynamically via the single-entry
SG-list.


Juergen

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 3149 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

Re: Design session PVH dom0

[Jan Beulich]

On 26.09.2022 10:33, Juergen Gross wrote:
> On 26.09.22 09:53, Jan Beulich wrote:
>> On 23.09.2022 10:20, Juergen Gross wrote:
>>> On 21.09.22 17:53, Marek Marczykowski-Górecki wrote:
>>>> Session description (by Jan):
>>>> In the course of working on an XSA I had to finally get PVH Dom0 work on at least one of my systems, in a minimal fashion. This had turned up a number of issues, some of which have since remained pending. Therefore I’d like to gain understanding on whether there is any future to this mode of Dom0 operation, and if so when it can be expected to be better than tech preview or even just experimental.
>>>
>>> ...
>>>
>>>> Jürgen: PVH dom0 performance?
>>>>
>>>> Roger: it's bad; mostly relevant is qemu interfaces
>>>>
>>>> George: only for safety certifications? performance penalty may be okay
>>>>
>>>> Jürgen: hypercalls can be improved (virtual buffers?)
>>>
>>> Some more thoughts on this topic: Having hypercall variants with physically
>>> addressed buffers will help, but there is an additional complexity: what
>>> about hypercalls with really large buffers (e.g. the bitmap for modified
>>> pages for guest migration). In order to avoid having to allocate huge
>>> physically contiguous buffers for those purposes we'd probably need
>>> something like scatter/gather lists for hypercall buffers.
>>
>> Not sure. I'd rather see us add new (sub)hypercalls for such non-standard
>> cases. E.g. the bitmap example you give would be amended by a new flavor
>> having the caller pass in an array of GFNs (perhaps, as you say, with
>> further indirection to deal with that array also growing large). I'd
>> really like to keep the common case simple.
> 
> The question is how many hypercalls would be hit by the not common case.
> 
> Taking a quick glance I spotted:
> 
> - grant_table_op (subops setup_table and get_status_frames)
> - memory_op (several sub-ops)
> - multicall (main list of calls)
> - console_io (console data)
> - mmuext_op (some ops allow lists)
> - xsm_op (not sure a buffer can span pages, but interface would allow it)
> - physdev_op (subop set_iobitmap)
> - hvm_op (altp2m handling)
> - sysctl (multiple sub-ops)
> - domctl (multiple sub-ops)
> - hypfs (node data can exceed page size)
> 
> Do we really want to special case all of those?

Looking at the Linux uses of several of the ones covered by the top three
entries in your list I find they all use contiguous buffers already, to a
fair part because of limiting maximum number of entries. Therefore I don't
really think all of these would need special casing, until a reasonable
use case is demonstrated where such large lists are actually needed.

Jan

Re: Design session PVH dom0

[Juergen Gross]

[-- Attachment #1.1.1: Type: text/plain, Size: 3099 bytes --]

On 26.09.22 11:30, Jan Beulich wrote:
> On 26.09.2022 10:33, Juergen Gross wrote:
>> On 26.09.22 09:53, Jan Beulich wrote:
>>> On 23.09.2022 10:20, Juergen Gross wrote:
>>>> On 21.09.22 17:53, Marek Marczykowski-Górecki wrote:
>>>>> Session description (by Jan):
>>>>> In the course of working on an XSA I had to finally get PVH Dom0 work on at least one of my systems, in a minimal fashion. This had turned up a number of issues, some of which have since remained pending. Therefore I’d like to gain understanding on whether there is any future to this mode of Dom0 operation, and if so when it can be expected to be better than tech preview or even just experimental.
>>>>
>>>> ...
>>>>
>>>>> Jürgen: PVH dom0 performance?
>>>>>
>>>>> Roger: it's bad; mostly relevant is qemu interfaces
>>>>>
>>>>> George: only for safety certifications? performance penalty may be okay
>>>>>
>>>>> Jürgen: hypercalls can be improved (virtual buffers?)
>>>>
>>>> Some more thoughts on this topic: Having hypercall variants with physically
>>>> addressed buffers will help, but there is an additional complexity: what
>>>> about hypercalls with really large buffers (e.g. the bitmap for modified
>>>> pages for guest migration). In order to avoid having to allocate huge
>>>> physically contiguous buffers for those purposes we'd probably need
>>>> something like scatter/gather lists for hypercall buffers.
>>>
>>> Not sure. I'd rather see us add new (sub)hypercalls for such non-standard
>>> cases. E.g. the bitmap example you give would be amended by a new flavor
>>> having the caller pass in an array of GFNs (perhaps, as you say, with
>>> further indirection to deal with that array also growing large). I'd
>>> really like to keep the common case simple.
>>
>> The question is how many hypercalls would be hit by the not common case.
>>
>> Taking a quick glance I spotted:
>>
>> - grant_table_op (subops setup_table and get_status_frames)
>> - memory_op (several sub-ops)
>> - multicall (main list of calls)
>> - console_io (console data)
>> - mmuext_op (some ops allow lists)
>> - xsm_op (not sure a buffer can span pages, but interface would allow it)
>> - physdev_op (subop set_iobitmap)
>> - hvm_op (altp2m handling)
>> - sysctl (multiple sub-ops)
>> - domctl (multiple sub-ops)
>> - hypfs (node data can exceed page size)
>>
>> Do we really want to special case all of those?
> 
> Looking at the Linux uses of several of the ones covered by the top three
> entries in your list I find they all use contiguous buffers already, to a
> fair part because of limiting maximum number of entries. Therefore I don't
> really think all of these would need special casing, until a reasonable
> use case is demonstrated where such large lists are actually needed.

... and then there are the use cases of Xen tools via the privcmd driver.

Especially the tools side use cases might be able to be usable via a generic
framework in case we make the buffer qualification generic instead of special
casing it for the different hypercalls.


Juergen

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 3149 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

Re: Design session PVH dom0

[Jan Beulich]

On 26.09.2022 10:33, Juergen Gross wrote:
> On 26.09.22 09:53, Jan Beulich wrote:
>> On 23.09.2022 10:20, Juergen Gross wrote:
>>> My favorite solution would be some kind of buffer address qualifier for each
>>> buffer (e.g. virtual, physical, SG-list, maybe nested SG-list). So the new
>>> hypercalls would not mean "physical buffer addresses", but "qualified buffer
>>> addresses". By requiring a minimum of 4-byte alignment for each buffer (can we
>>> do that, at least for the new hypercalls?) this would leave the 2 lowest bits
>>> of a buffer address for the new qualifier. If by any means an unaligned buffer
>>> is needed sometimes, it could still be achieved via a single-entry SG-list.
>>
>> While this might be an option, I'm not sure I'd be really happy with such
>> re-use of the low address bits, nor with the implied further restriction
>> on buffer alignment (most struct-s we use are 4-byte aligned at least,
>> but I don't think it's all of them, plus we also have guest handles to
>> e.g. arrays of char).
> 
> The unaligned cases could be handled dynamically via the single-entry
> SG-list.

Can they? The first example you gave, the bitmap passed to collect the
output of XEN_DOMCTL_SHADOW_OP_{CLEAN,PEEK}, comes as a handle-of-uint8,
i.e. generally large but not necessarily aligned (even if in practice
the caller likely will pass a page aligned buffer of multiple pages in
size). If we introduced physical-address bases replacement sub-ops, I
think we would make the buffer described by an array of GFNs, not even
allowing sub-page alignment or size.

Jan

Re: Design session PVH dom0

[Juergen Gross]

[-- Attachment #1.1.1: Type: text/plain, Size: 1845 bytes --]

On 26.09.22 17:52, Jan Beulich wrote:
> On 26.09.2022 10:33, Juergen Gross wrote:
>> On 26.09.22 09:53, Jan Beulich wrote:
>>> On 23.09.2022 10:20, Juergen Gross wrote:
>>>> My favorite solution would be some kind of buffer address qualifier for each
>>>> buffer (e.g. virtual, physical, SG-list, maybe nested SG-list). So the new
>>>> hypercalls would not mean "physical buffer addresses", but "qualified buffer
>>>> addresses". By requiring a minimum of 4-byte alignment for each buffer (can we
>>>> do that, at least for the new hypercalls?) this would leave the 2 lowest bits
>>>> of a buffer address for the new qualifier. If by any means an unaligned buffer
>>>> is needed sometimes, it could still be achieved via a single-entry SG-list.
>>>
>>> While this might be an option, I'm not sure I'd be really happy with such
>>> re-use of the low address bits, nor with the implied further restriction
>>> on buffer alignment (most struct-s we use are 4-byte aligned at least,
>>> but I don't think it's all of them, plus we also have guest handles to
>>> e.g. arrays of char).
>>
>> The unaligned cases could be handled dynamically via the single-entry
>> SG-list.
> 
> Can they? The first example you gave, the bitmap passed to collect the
> output of XEN_DOMCTL_SHADOW_OP_{CLEAN,PEEK}, comes as a handle-of-uint8,
> i.e. generally large but not necessarily aligned (even if in practice
> the caller likely will pass a page aligned buffer of multiple pages in
> size). If we introduced physical-address bases replacement sub-ops, I
> think we would make the buffer described by an array of GFNs, not even
> allowing sub-page alignment or size.

In case the buffer is crossing page boundaries, the SG-list would need to
have more than one entry, of course (assuming the SG-list variant is chosen).


Juergen

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 3149 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

Re: Design session PVH dom0

[Jan Beulich]

On 27.09.2022 09:02, Juergen Gross wrote:
> On 26.09.22 17:52, Jan Beulich wrote:
>> On 26.09.2022 10:33, Juergen Gross wrote:
>>> On 26.09.22 09:53, Jan Beulich wrote:
>>>> On 23.09.2022 10:20, Juergen Gross wrote:
>>>>> My favorite solution would be some kind of buffer address qualifier for each
>>>>> buffer (e.g. virtual, physical, SG-list, maybe nested SG-list). So the new
>>>>> hypercalls would not mean "physical buffer addresses", but "qualified buffer
>>>>> addresses". By requiring a minimum of 4-byte alignment for each buffer (can we
>>>>> do that, at least for the new hypercalls?) this would leave the 2 lowest bits
>>>>> of a buffer address for the new qualifier. If by any means an unaligned buffer
>>>>> is needed sometimes, it could still be achieved via a single-entry SG-list.
>>>>
>>>> While this might be an option, I'm not sure I'd be really happy with such
>>>> re-use of the low address bits, nor with the implied further restriction
>>>> on buffer alignment (most struct-s we use are 4-byte aligned at least,
>>>> but I don't think it's all of them, plus we also have guest handles to
>>>> e.g. arrays of char).
>>>
>>> The unaligned cases could be handled dynamically via the single-entry
>>> SG-list.
>>
>> Can they? The first example you gave, the bitmap passed to collect the
>> output of XEN_DOMCTL_SHADOW_OP_{CLEAN,PEEK}, comes as a handle-of-uint8,
>> i.e. generally large but not necessarily aligned (even if in practice
>> the caller likely will pass a page aligned buffer of multiple pages in
>> size). If we introduced physical-address bases replacement sub-ops, I
>> think we would make the buffer described by an array of GFNs, not even
>> allowing sub-page alignment or size.
> 
> In case the buffer is crossing page boundaries, the SG-list would need to
> have more than one entry, of course (assuming the SG-list variant is chosen).

Of course, but that wasn't the point I was trying to get at. How would the
(generic, i.e. alignment unaware) copying logic know the low bits are not
a descriptor in this case? I'd rather not see us have e.g. both
copy_from_guest_pv() and copy_from_guest_pv_unaligned() (nor a 4th argument
to the former, to express the same).

Jan