xen summit 2023 design session: Using only stable interfaces for (o)xenstored

xen summit 2023 design session: Using only stable interfaces for (o)xenstored

[Edwin Torok]

Background: https://design-sessions.xenproject.org/uid/discussion/disc_MEXyUIXV6clI8n1kgzQ9/view

Possible solutions:
Andrew Cooper: use get domain info list instead of querying domains one-by-one to reduce number of hypercalls, even though Alejandro's patch makes the get_domainfo lookups O(1).
To avoid a hypercall a shared read-only memory page between Xen and Dom0, containing a bitmap of domain existence and shutdown state.
We have 32k domains max and this fits into a small number of 4K pages.
Hypervisor can update it using atomic set/clear-bit, Dom0 will memcpy and then check for differences
Bernhardt/Andrew: discussion about efficiency and find first bit set, XOR. A *lot* more efficient than using hypercalls anyway
Edwin: do I have to implement that for various architectures?
Andrew Cooper: already implemented for all arches. Some architectures can only do atomic updates on natural register width (RISC-V), but that is already handled
Edwin: how would the stable interface look like? Memory layout definition in the header and ..?
Andrew Cooper: use acquire_resource with a new resource type, already have stable xen libs for resource mapping

Christopher Clarke/Daniel Smith (virtually from chatroom): alternatives to bitmap, how is it indexed?
Andrew Coooper: indexed by domid, we only have 32k
Marcus Granado: what if we want to increase that?
Andrew Coooper: 16-bit domid is baked into so many ABIs that if we want to increase that then this API won't be the blocker. No need to worry about more than 32k domids.
Christopher Clark: how about using an Argo ring
Andrew Cooper: Xen must NOT block when delivering the notification and notification delivery cannot be lossy, cannot afford to drop events: toolstacks/xenstored critically rely on this information.
Christopher Clark: bip buffers: https://www.codeproject.com/Articles/3479/The-Bip-Buffer-The-Circular-Buffer-with-a-Twist as possible solution to avoid running out of space
Andrew Cooper: bitmap also avoids having to take the domctl lock and reduce background idle CPU usage
Edwin: on that topic: we should also fix xcp-rrdd so it doesn't shell out to xenpm every 5s to collect statistics...
Andrew Cooper: should be possible to use libxenctrl directly for that
Edwin Torok: there are other interfacs that would need to be stabilized (see xenopsd), but lets start small and fix just this one for now, which will result in an immediate improvement for (o)xenstored since this was the final blocker to getting off stable interfaces (there is one other usage, but that already has a patch posted/solution)

Edwin Torok: how about release cadence? Doesn't necessarily have to go into immediately next Xen release, but don't know in which phase the release is in right now
Andrew Cooper: probably about 6 weeks time to get into next release

Re: xen summit 2023 design session: Using only stable interfaces for (o)xenstored

[Juergen Gross]

[-- Attachment #1.1.1: Type: text/plain, Size: 3026 bytes --]

On 27.06.23 12:21, Edwin Torok wrote:
> Background: https://design-sessions.xenproject.org/uid/discussion/disc_MEXyUIXV6clI8n1kgzQ9/view
> 
> Possible solutions:
> Andrew Cooper: use get domain info list instead of querying domains one-by-one to reduce number of hypercalls, even though Alejandro's patch makes the get_domainfo lookups O(1).
> To avoid a hypercall a shared read-only memory page between Xen and Dom0, containing a bitmap of domain existence and shutdown state.
> We have 32k domains max and this fits into a small number of 4K pages.
> Hypervisor can update it using atomic set/clear-bit, Dom0 will memcpy and then check for differences
> Bernhardt/Andrew: discussion about efficiency and find first bit set, XOR. A *lot* more efficient than using hypercalls anyway
> Edwin: do I have to implement that for various architectures?
> Andrew Cooper: already implemented for all arches. Some architectures can only do atomic updates on natural register width (RISC-V), but that is already handled
> Edwin: how would the stable interface look like? Memory layout definition in the header and ..?
> Andrew Cooper: use acquire_resource with a new resource type, already have stable xen libs for resource mapping
> 
> Christopher Clarke/Daniel Smith (virtually from chatroom): alternatives to bitmap, how is it indexed?
> Andrew Coooper: indexed by domid, we only have 32k
> Marcus Granado: what if we want to increase that?
> Andrew Coooper: 16-bit domid is baked into so many ABIs that if we want to increase that then this API won't be the blocker. No need to worry about more than 32k domids.
> Christopher Clark: how about using an Argo ring
> Andrew Cooper: Xen must NOT block when delivering the notification and notification delivery cannot be lossy, cannot afford to drop events: toolstacks/xenstored critically rely on this information.
> Christopher Clark: bip buffers: https://www.codeproject.com/Articles/3479/The-Bip-Buffer-The-Circular-Buffer-with-a-Twist as possible solution to avoid running out of space
> Andrew Cooper: bitmap also avoids having to take the domctl lock and reduce background idle CPU usage
> Edwin: on that topic: we should also fix xcp-rrdd so it doesn't shell out to xenpm every 5s to collect statistics...
> Andrew Cooper: should be possible to use libxenctrl directly for that
> Edwin Torok: there are other interfacs that would need to be stabilized (see xenopsd), but lets start small and fix just this one for now, which will result in an immediate improvement for (o)xenstored since this was the final blocker to getting off stable interfaces (there is one other usage, but that already has a patch posted/solution)
> 
> Edwin Torok: how about release cadence? Doesn't necessarily have to go into immediately next Xen release, but don't know in which phase the release is in right now
> Andrew Cooper: probably about 6 weeks time to get into next release

See https://lists.xen.org/archives/html/xen-devel/2021-09/msg01098.html


Juergen


[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 3149 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]