[Xen-devel] [PATCH] docs/sphinx: todo/wishlist

[Xen-devel] [PATCH] docs/sphinx: todo/wishlist

[Andrew Cooper]

a.k.a. (at least in this form) Andrew's "work which might be offloadable to
someone else" list.

Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
---
CC: George Dunlap <George.Dunlap@eu.citrix.com>
CC: Ian Jackson <ian.jackson@citrix.com>
CC: Jan Beulich <JBeulich@suse.com>
CC: Stefano Stabellini <sstabellini@kernel.org>
CC: Tim Deegan <tim@xen.org>
CC: Wei Liu <wl@xen.org>
CC: Julien Grall <julien.grall@arm.com>
CC: Lars Kurth <lars.kurth@citrix.com>
CC: Paul Durrant <paul.durrant@citrix.com>
CC: Roger Pau Monné <roger.pau@citrix.com>

RFC for obvious reasons.

A rendered version of this can be found at:
https://andrewcoop-xen.readthedocs.io/en/docs-wishlist/misc/wishlist.html

During XenSummit in Chicago, it was expressed several times that having some
todo lists would be a benefit, to help coordinate work in related areas.

Here is an attempt to start one.  For now, it covers one single
item (xenstored's use of non-stable APIs) to get some feedback about the
general approach.  I have plenty to get stuck into in Xen itself if this way
of expressing them isn't deemed unacceptable.

As for the wishlist itself, I think it is important that it be restricted to
concrete actions (i.e. already partially groomed, if you speak agile), which
are identified problems, and suggested fixes.

In particular, I don't think it is appropriate to devolve into a bullet point
list of new features, or tasks like "document $whotsit".  It should be
restricted to things which are real problems, on existing systems, which have
some forward plan of action.  That way, any developer should be able to
cross-reference at least at a high level, and see if there are areas of
overlapping work, or whether a slightly tweaked approach might be suitable for
multiple areas.

Anyway - thoughts from the peanut gallery?
---
 docs/conf.py           | 10 +++++++++-
 docs/index.rst         |  9 +++++++++
 docs/misc/wishlist.rst | 53 ++++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 71 insertions(+), 1 deletion(-)
 create mode 100644 docs/misc/wishlist.rst

diff --git a/docs/conf.py b/docs/conf.py
index 73b7b9bfa2..a5765bf7f4 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -52,7 +52,7 @@
 # Add any Sphinx extension module names here, as strings. They can be
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
-extensions = []
+extensions = ["sphinx.ext.extlinks"]
 
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ['_templates']
@@ -191,3 +191,11 @@
 
 # A list of files that should not be packed into the epub file.
 epub_exclude_files = ['search.html']
+
+
+# -- Configuration for extlinks ----------------------------------------------
+
+extlinks = {
+    'xen-cs': ('https://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=%s',
+               "Xen c/s "),
+}
diff --git a/docs/index.rst b/docs/index.rst
index 31bb8927f2..9bbe6ee688 100644
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -45,3 +45,12 @@ kind of development environment.
   :maxdepth: 2
 
   hypervisor-guide/index
+
+
+Misc unsorted
+-------------
+
+.. toctree::
+  :maxdepth: 2
+
+  misc/wishlist
diff --git a/docs/misc/wishlist.rst b/docs/misc/wishlist.rst
new file mode 100644
index 0000000000..6cdb47d6e7
--- /dev/null
+++ b/docs/misc/wishlist.rst
@@ -0,0 +1,53 @@
+Development Wishlist
+====================
+
+Remove xenstored's dependencies on unstable interfaces
+------------------------------------------------------
+
+Various xenstored implementations use libxc for two purposes.  It would be a
+substantial advantage to move xenstored onto entirely stable interfaces, which
+disconnects it from the internal of the libxc.
+
+1. Foreign mapping of the store ring
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+This is obsolete since :xen-cs:`6a2de353a9` (2012) which allocated grant
+entries instead, to allow xenstored to function as a stub-domain without dom0
+permissions.  :xen-cs:`38eeb3864d` dropped foreign mapping for cxenstored.
+However, there are no OCaml bindings for libxengnttab.
+
+Work Items:
+
+* Minimal ``tools/ocaml/libs/xg/`` binding for ``tools/libs/gnttab/``.
+* Replicate :xen-cs:`38eeb3864d` for oxenstored as well.
+
+
+2. Figuring out which domain(s) have gone away
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Currently, the handling of domains is asymmetric.
+
+* When a domain is created, the toolstack explicitly sends an
+  ``XS_INTRODUCE(domid, store mfn, store evtchn)`` message to xenstored, to
+  cause xenstored to connect to the guest ring, and fire the
+  ``@introduceDomain`` watch.
+* When a domain is destroyed, Xen fires ``VIRQ_DOM_EXC`` which is bound by
+  xenstored, rather than the toolstack.  xenstored updates its idea of the
+  status of domains, and fires the ``@releaseDomain`` watch.
+
+Xenstored uses ``xc_domain_getinfo()``, to work out which domain(s) have gone
+away, and only cares about the shutdown status.
+
+Furthermore, ``@releaseDomain`` (like ``VIRQ_DOM_EXC``) is a single-bit
+message, which requires all listeners to evaluate whether the message applies
+to them or not.  This results in a flurry of ``xc_domain_getinfo()`` calls
+from multiple entities in the system, which all serialise on the domctl lock
+in Xen.
+
+Work Items:
+
+* Figure out how shutdown status can be expressed in a stable way from Xen.
+* Figure out if ``VIRQ_DOM_EXC`` and ``@releaseDomain`` can be extended to
+  carry at least a domid, to make domain shutdown scale better.
+* Figure out if ``VIRQ_DOM_EXC`` would better be bound by the toolstack,
+  rather than xenstored.
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH] docs/sphinx: todo/wishlist

[Juergen Gross]

On 22.07.19 21:20, Andrew Cooper wrote:
> a.k.a. (at least in this form) Andrew's "work which might be offloadable to
> someone else" list.
> 
> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
> ---
> CC: George Dunlap <George.Dunlap@eu.citrix.com>
> CC: Ian Jackson <ian.jackson@citrix.com>
> CC: Jan Beulich <JBeulich@suse.com>
> CC: Stefano Stabellini <sstabellini@kernel.org>
> CC: Tim Deegan <tim@xen.org>
> CC: Wei Liu <wl@xen.org>
> CC: Julien Grall <julien.grall@arm.com>
> CC: Lars Kurth <lars.kurth@citrix.com>
> CC: Paul Durrant <paul.durrant@citrix.com>
> CC: Roger Pau Monné <roger.pau@citrix.com>
> 
> RFC for obvious reasons.
> 
> A rendered version of this can be found at:
> https://andrewcoop-xen.readthedocs.io/en/docs-wishlist/misc/wishlist.html
> 
> During XenSummit in Chicago, it was expressed several times that having some
> todo lists would be a benefit, to help coordinate work in related areas.
> 
> Here is an attempt to start one.  For now, it covers one single
> item (xenstored's use of non-stable APIs) to get some feedback about the
> general approach.  I have plenty to get stuck into in Xen itself if this way
> of expressing them isn't deemed unacceptable.
> 
> As for the wishlist itself, I think it is important that it be restricted to
> concrete actions (i.e. already partially groomed, if you speak agile), which
> are identified problems, and suggested fixes.
> 
> In particular, I don't think it is appropriate to devolve into a bullet point
> list of new features, or tasks like "document $whotsit".  It should be
> restricted to things which are real problems, on existing systems, which have
> some forward plan of action.  That way, any developer should be able to
> cross-reference at least at a high level, and see if there are areas of
> overlapping work, or whether a slightly tweaked approach might be suitable for
> multiple areas.
> 
> Anyway - thoughts from the peanut gallery?
> ---
>   docs/conf.py           | 10 +++++++++-
>   docs/index.rst         |  9 +++++++++
>   docs/misc/wishlist.rst | 53 ++++++++++++++++++++++++++++++++++++++++++++++++++
>   3 files changed, 71 insertions(+), 1 deletion(-)
>   create mode 100644 docs/misc/wishlist.rst
> 
> diff --git a/docs/conf.py b/docs/conf.py
> index 73b7b9bfa2..a5765bf7f4 100644
> --- a/docs/conf.py
> +++ b/docs/conf.py
> @@ -52,7 +52,7 @@
>   # Add any Sphinx extension module names here, as strings. They can be
>   # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
>   # ones.
> -extensions = []
> +extensions = ["sphinx.ext.extlinks"]
>   
>   # Add any paths that contain templates here, relative to this directory.
>   templates_path = ['_templates']
> @@ -191,3 +191,11 @@
>   
>   # A list of files that should not be packed into the epub file.
>   epub_exclude_files = ['search.html']
> +
> +
> +# -- Configuration for extlinks ----------------------------------------------
> +
> +extlinks = {
> +    'xen-cs': ('https://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=%s',
> +               "Xen c/s "),
> +}
> diff --git a/docs/index.rst b/docs/index.rst
> index 31bb8927f2..9bbe6ee688 100644
> --- a/docs/index.rst
> +++ b/docs/index.rst
> @@ -45,3 +45,12 @@ kind of development environment.
>     :maxdepth: 2
>   
>     hypervisor-guide/index
> +
> +
> +Misc unsorted
> +-------------
> +
> +.. toctree::
> +  :maxdepth: 2
> +
> +  misc/wishlist
> diff --git a/docs/misc/wishlist.rst b/docs/misc/wishlist.rst
> new file mode 100644
> index 0000000000..6cdb47d6e7
> --- /dev/null
> +++ b/docs/misc/wishlist.rst
> @@ -0,0 +1,53 @@
> +Development Wishlist
> +====================
> +
> +Remove xenstored's dependencies on unstable interfaces
> +------------------------------------------------------
> +
> +Various xenstored implementations use libxc for two purposes.  It would be a
> +substantial advantage to move xenstored onto entirely stable interfaces, which
> +disconnects it from the internal of the libxc.
> +
> +1. Foreign mapping of the store ring
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +This is obsolete since :xen-cs:`6a2de353a9` (2012) which allocated grant
> +entries instead, to allow xenstored to function as a stub-domain without dom0
> +permissions.  :xen-cs:`38eeb3864d` dropped foreign mapping for cxenstored.
> +However, there are no OCaml bindings for libxengnttab.
> +
> +Work Items:
> +
> +* Minimal ``tools/ocaml/libs/xg/`` binding for ``tools/libs/gnttab/``.
> +* Replicate :xen-cs:`38eeb3864d` for oxenstored as well.
> +
> +
> +2. Figuring out which domain(s) have gone away
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +Currently, the handling of domains is asymmetric.
> +
> +* When a domain is created, the toolstack explicitly sends an
> +  ``XS_INTRODUCE(domid, store mfn, store evtchn)`` message to xenstored, to
> +  cause xenstored to connect to the guest ring, and fire the
> +  ``@introduceDomain`` watch.
> +* When a domain is destroyed, Xen fires ``VIRQ_DOM_EXC`` which is bound by
> +  xenstored, rather than the toolstack.  xenstored updates its idea of the
> +  status of domains, and fires the ``@releaseDomain`` watch.
> +
> +Xenstored uses ``xc_domain_getinfo()``, to work out which domain(s) have gone
> +away, and only cares about the shutdown status.
> +
> +Furthermore, ``@releaseDomain`` (like ``VIRQ_DOM_EXC``) is a single-bit
> +message, which requires all listeners to evaluate whether the message applies
> +to them or not.  This results in a flurry of ``xc_domain_getinfo()`` calls
> +from multiple entities in the system, which all serialise on the domctl lock
> +in Xen.
> +
> +Work Items:
> +
> +* Figure out how shutdown status can be expressed in a stable way from Xen.
> +* Figure out if ``VIRQ_DOM_EXC`` and ``@releaseDomain`` can be extended to
> +  carry at least a domid, to make domain shutdown scale better.

@releaseDomain (and @introduceDomain) can't be extended, we'd need to
add another watch path like @domainStatus/<domid>/<newState>. Xenstored
could advertise its capability to raise this watch in /tool/xenstored.

As VIRQ_DOM_EXC is just an event I don't see how the domid could be
passed by it. I guess we'd need e.g. a shared memory area which the
domain registered for VIRQ_DOM_EXC could map and which would contain a
bitmap (one bit per domain). The hypervisor would set the bit on a
status change and fire VIRQ_DOM_EXC, xenstored would look for a set
bit, clear it and read the status of the related domain.

> +* Figure out if ``VIRQ_DOM_EXC`` would better be bound by the toolstack,
> +  rather than xenstored.

No, I don't think so. This would need another daemon.


Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH] docs/sphinx: todo/wishlist

[Andrew Cooper]

On 23/07/2019 05:36, Juergen Gross wrote:
> On 22.07.19 21:20, Andrew Cooper wrote:
>> diff --git a/docs/misc/wishlist.rst b/docs/misc/wishlist.rst
>> new file mode 100644
>> index 0000000000..6cdb47d6e7
>> --- /dev/null
>> +++ b/docs/misc/wishlist.rst
>> @@ -0,0 +1,53 @@
>> +Development Wishlist
>> +====================
>> +
>> +Remove xenstored's dependencies on unstable interfaces
>> +------------------------------------------------------
>> +
>> +Various xenstored implementations use libxc for two purposes.  It
>> would be a
>> +substantial advantage to move xenstored onto entirely stable
>> interfaces, which
>> +disconnects it from the internal of the libxc.
>> +
>> +1. Foreign mapping of the store ring
>> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> +
>> +This is obsolete since :xen-cs:`6a2de353a9` (2012) which allocated
>> grant
>> +entries instead, to allow xenstored to function as a stub-domain
>> without dom0
>> +permissions.  :xen-cs:`38eeb3864d` dropped foreign mapping for
>> cxenstored.
>> +However, there are no OCaml bindings for libxengnttab.
>> +
>> +Work Items:
>> +
>> +* Minimal ``tools/ocaml/libs/xg/`` binding for ``tools/libs/gnttab/``.
>> +* Replicate :xen-cs:`38eeb3864d` for oxenstored as well.
>> +
>> +
>> +2. Figuring out which domain(s) have gone away
>> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> +
>> +Currently, the handling of domains is asymmetric.
>> +
>> +* When a domain is created, the toolstack explicitly sends an
>> +  ``XS_INTRODUCE(domid, store mfn, store evtchn)`` message to
>> xenstored, to
>> +  cause xenstored to connect to the guest ring, and fire the
>> +  ``@introduceDomain`` watch.
>> +* When a domain is destroyed, Xen fires ``VIRQ_DOM_EXC`` which is
>> bound by
>> +  xenstored, rather than the toolstack.  xenstored updates its idea
>> of the
>> +  status of domains, and fires the ``@releaseDomain`` watch.
>> +
>> +Xenstored uses ``xc_domain_getinfo()``, to work out which domain(s)
>> have gone
>> +away, and only cares about the shutdown status.
>> +
>> +Furthermore, ``@releaseDomain`` (like ``VIRQ_DOM_EXC``) is a single-bit
>> +message, which requires all listeners to evaluate whether the
>> message applies
>> +to them or not.  This results in a flurry of ``xc_domain_getinfo()``
>> calls
>> +from multiple entities in the system, which all serialise on the
>> domctl lock
>> +in Xen.
>> +
>> +Work Items:
>> +
>> +* Figure out how shutdown status can be expressed in a stable way
>> from Xen.
>> +* Figure out if ``VIRQ_DOM_EXC`` and ``@releaseDomain`` can be
>> extended to
>> +  carry at least a domid, to make domain shutdown scale better.
>
> @releaseDomain (and @introduceDomain) can't be extended, we'd need to
> add another watch path like @domainStatus/<domid>/<newState>. Xenstored
> could advertise its capability to raise this watch in /tool/xenstored.

I guess I was being a bit fast and loose with terminology.  I didn't
intend to imply "literally modify @{introduce,release}Domain", as they
are already fixed ABIs, but more to "compatibly build something which is
better".

That scheme would work for improved @releaseDomain, but it wouldn't work
for an improved introduce.  Introduce needs a single key to watch on,
which hands back the domid so you don't need to go searching for it.

>
> As VIRQ_DOM_EXC is just an event I don't see how the domid could be
> passed by it. I guess we'd need e.g. a shared memory area which the
> domain registered for VIRQ_DOM_EXC could map and which would contain a
> bitmap (one bit per domain). The hypervisor would set the bit on a
> status change and fire VIRQ_DOM_EXC, xenstored would look for a set
> bit, clear it and read the status of the related domain.

The point here is to avoid using xc_domain_getinfo() in the first place,
so there needs to be no "getting the status of the domain".

DOM_EXC is fired for domain_shutdown() only (but for reasons which
escape me, fired twice).  Given that a domid is a 15 bit number, a
bitmap of all domains does fit within a single 4k page.

All xenstored needs to know is that the domain shut down - nothing
else.  Given a mapping such as the above, xenstored, on getting
VIRQ_DOM_EXC would memcpy the page sideways, then loop over the page
looking for bits which have become set, notify for each of those
domains, and clear the bit in the page.

>
>> +* Figure out if ``VIRQ_DOM_EXC`` would better be bound by the
>> toolstack,
>> +  rather than xenstored.
>
> No, I don't think so. This would need another daemon.

There is exactly one toolstack which is nominally daemonless, and it
doesn't account for any substantial proportion of the Xen install-base,
worldwide.

The fact xl might not cope has no relevance to the technical discussion
of whether the current asymmetric setup is sane or not, and whether it
can, in principle, be improved.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH] docs/sphinx: todo/wishlist

[Andrew Cooper]

On 22/07/2019 20:20, Andrew Cooper wrote:
> a.k.a. (at least in this form) Andrew's "work which might be offloadable to
> someone else" list.
>
> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
> ---
> CC: George Dunlap <George.Dunlap@eu.citrix.com>
> CC: Ian Jackson <ian.jackson@citrix.com>
> CC: Jan Beulich <JBeulich@suse.com>
> CC: Stefano Stabellini <sstabellini@kernel.org>
> CC: Tim Deegan <tim@xen.org>
> CC: Wei Liu <wl@xen.org>
> CC: Julien Grall <julien.grall@arm.com>
> CC: Lars Kurth <lars.kurth@citrix.com>
> CC: Paul Durrant <paul.durrant@citrix.com>
> CC: Roger Pau Monné <roger.pau@citrix.com>
>
> RFC for obvious reasons.
>
> A rendered version of this can be found at:
> https://andrewcoop-xen.readthedocs.io/en/docs-wishlist/misc/wishlist.html
>
> During XenSummit in Chicago, it was expressed several times that having some
> todo lists would be a benefit, to help coordinate work in related areas.
>
> Here is an attempt to start one.  For now, it covers one single
> item (xenstored's use of non-stable APIs) to get some feedback about the
> general approach.  I have plenty to get stuck into in Xen itself if this way
> of expressing them isn't deemed unacceptable.
>
> As for the wishlist itself, I think it is important that it be restricted to
> concrete actions (i.e. already partially groomed, if you speak agile), which
> are identified problems, and suggested fixes.
>
> In particular, I don't think it is appropriate to devolve into a bullet point
> list of new features, or tasks like "document $whotsit".  It should be
> restricted to things which are real problems, on existing systems, which have
> some forward plan of action.  That way, any developer should be able to
> cross-reference at least at a high level, and see if there are areas of
> overlapping work, or whether a slightly tweaked approach might be suitable for
> multiple areas.
>
> Anyway - thoughts from the peanut gallery?
> ---
>  docs/conf.py           | 10 +++++++++-
>  docs/index.rst         |  9 +++++++++
>  docs/misc/wishlist.rst | 53 ++++++++++++++++++++++++++++++++++++++++++++++++++

It has been pointed out that calling this the wishlist is a poor name. 
A better name would be the technical debt list.  I won't resend for just
this, but please bear it in mind when considering the suggestion.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH] docs/sphinx: todo/wishlist

[Juergen Gross]

On 23.07.19 15:38, Andrew Cooper wrote:
> On 23/07/2019 05:36, Juergen Gross wrote:
>> On 22.07.19 21:20, Andrew Cooper wrote:
>>> diff --git a/docs/misc/wishlist.rst b/docs/misc/wishlist.rst
>>> new file mode 100644
>>> index 0000000000..6cdb47d6e7
>>> --- /dev/null
>>> +++ b/docs/misc/wishlist.rst
>>> @@ -0,0 +1,53 @@
>>> +Development Wishlist
>>> +====================
>>> +
>>> +Remove xenstored's dependencies on unstable interfaces
>>> +------------------------------------------------------
>>> +
>>> +Various xenstored implementations use libxc for two purposes.  It
>>> would be a
>>> +substantial advantage to move xenstored onto entirely stable
>>> interfaces, which
>>> +disconnects it from the internal of the libxc.
>>> +
>>> +1. Foreign mapping of the store ring
>>> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>> +
>>> +This is obsolete since :xen-cs:`6a2de353a9` (2012) which allocated
>>> grant
>>> +entries instead, to allow xenstored to function as a stub-domain
>>> without dom0
>>> +permissions.  :xen-cs:`38eeb3864d` dropped foreign mapping for
>>> cxenstored.
>>> +However, there are no OCaml bindings for libxengnttab.
>>> +
>>> +Work Items:
>>> +
>>> +* Minimal ``tools/ocaml/libs/xg/`` binding for ``tools/libs/gnttab/``.
>>> +* Replicate :xen-cs:`38eeb3864d` for oxenstored as well.
>>> +
>>> +
>>> +2. Figuring out which domain(s) have gone away
>>> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>> +
>>> +Currently, the handling of domains is asymmetric.
>>> +
>>> +* When a domain is created, the toolstack explicitly sends an
>>> +  ``XS_INTRODUCE(domid, store mfn, store evtchn)`` message to
>>> xenstored, to
>>> +  cause xenstored to connect to the guest ring, and fire the
>>> +  ``@introduceDomain`` watch.
>>> +* When a domain is destroyed, Xen fires ``VIRQ_DOM_EXC`` which is
>>> bound by
>>> +  xenstored, rather than the toolstack.  xenstored updates its idea
>>> of the
>>> +  status of domains, and fires the ``@releaseDomain`` watch.
>>> +
>>> +Xenstored uses ``xc_domain_getinfo()``, to work out which domain(s)
>>> have gone
>>> +away, and only cares about the shutdown status.
>>> +
>>> +Furthermore, ``@releaseDomain`` (like ``VIRQ_DOM_EXC``) is a single-bit
>>> +message, which requires all listeners to evaluate whether the
>>> message applies
>>> +to them or not.  This results in a flurry of ``xc_domain_getinfo()``
>>> calls
>>> +from multiple entities in the system, which all serialise on the
>>> domctl lock
>>> +in Xen.
>>> +
>>> +Work Items:
>>> +
>>> +* Figure out how shutdown status can be expressed in a stable way
>>> from Xen.
>>> +* Figure out if ``VIRQ_DOM_EXC`` and ``@releaseDomain`` can be
>>> extended to
>>> +  carry at least a domid, to make domain shutdown scale better.
>>
>> @releaseDomain (and @introduceDomain) can't be extended, we'd need to
>> add another watch path like @domainStatus/<domid>/<newState>. Xenstored
>> could advertise its capability to raise this watch in /tool/xenstored.
> 
> I guess I was being a bit fast and loose with terminology.  I didn't
> intend to imply "literally modify @{introduce,release}Domain", as they
> are already fixed ABIs, but more to "compatibly build something which is
> better".

Okay.

> 
> That scheme would work for improved @releaseDomain, but it wouldn't work
> for an improved introduce.  Introduce needs a single key to watch on,
> which hands back the domid so you don't need to go searching for it.

Yes, and? Its perfectly fine to set a watch firing if anything below
@domainStatus is changing.

> 
>>
>> As VIRQ_DOM_EXC is just an event I don't see how the domid could be
>> passed by it. I guess we'd need e.g. a shared memory area which the
>> domain registered for VIRQ_DOM_EXC could map and which would contain a
>> bitmap (one bit per domain). The hypervisor would set the bit on a
>> status change and fire VIRQ_DOM_EXC, xenstored would look for a set
>> bit, clear it and read the status of the related domain.
> 
> The point here is to avoid using xc_domain_getinfo() in the first place,
> so there needs to be no "getting the status of the domain".

I'd guess a single xc_domain_getinfo() in the tools wouldn't be so
problematic. The caller would know the domid already, so no need to
query all domains.

> DOM_EXC is fired for domain_shutdown() only (but for reasons which
> escape me, fired twice).  Given that a domid is a 15 bit number, a
> bitmap of all domains does fit within a single 4k page.

Firing twice is needed: first time for disconnecting all backends
and the second time for cleaning up when the domain is completely
gone.

> 
> All xenstored needs to know is that the domain shut down - nothing
> else.  Given a mapping such as the above, xenstored, on getting
> VIRQ_DOM_EXC would memcpy the page sideways, then loop over the page
> looking for bits which have become set, notify for each of those
> domains, and clear the bit in the page.

Right, that was the idea.

> 
>>
>>> +* Figure out if ``VIRQ_DOM_EXC`` would better be bound by the
>>> toolstack,
>>> +  rather than xenstored.
>>
>> No, I don't think so. This would need another daemon.
> 
> There is exactly one toolstack which is nominally daemonless, and it
> doesn't account for any substantial proportion of the Xen install-base,
> worldwide.
> 
> The fact xl might not cope has no relevance to the technical discussion
> of whether the current asymmetric setup is sane or not, and whether it
> can, in principle, be improved.

Today we have either libvirtd or a per-domain xl instance running. So no
central piece of code for that purpose.


Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH] docs/sphinx: todo/wishlist

[Andrew Cooper]

On 23/07/2019 16:30, Juergen Gross wrote:
> On 23.07.19 15:38, Andrew Cooper wrote:
>> On 23/07/2019 05:36, Juergen Gross wrote:
>>> On 22.07.19 21:20, Andrew Cooper wrote:
>>>> diff --git a/docs/misc/wishlist.rst b/docs/misc/wishlist.rst
>>>> new file mode 100644
>>>> index 0000000000..6cdb47d6e7
>>>> --- /dev/null
>>>> +++ b/docs/misc/wishlist.rst
>>>> @@ -0,0 +1,53 @@
>>>> +Development Wishlist
>>>> +====================
>>>> +
>>>> +Remove xenstored's dependencies on unstable interfaces
>>>> +------------------------------------------------------
>>>> +
>>>> +Various xenstored implementations use libxc for two purposes.  It
>>>> would be a
>>>> +substantial advantage to move xenstored onto entirely stable
>>>> interfaces, which
>>>> +disconnects it from the internal of the libxc.
>>>> +
>>>> +1. Foreign mapping of the store ring
>>>> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>> +
>>>> +This is obsolete since :xen-cs:`6a2de353a9` (2012) which allocated
>>>> grant
>>>> +entries instead, to allow xenstored to function as a stub-domain
>>>> without dom0
>>>> +permissions.  :xen-cs:`38eeb3864d` dropped foreign mapping for
>>>> cxenstored.
>>>> +However, there are no OCaml bindings for libxengnttab.
>>>> +
>>>> +Work Items:
>>>> +
>>>> +* Minimal ``tools/ocaml/libs/xg/`` binding for
>>>> ``tools/libs/gnttab/``.
>>>> +* Replicate :xen-cs:`38eeb3864d` for oxenstored as well.
>>>> +
>>>> +
>>>> +2. Figuring out which domain(s) have gone away
>>>> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>> +
>>>> +Currently, the handling of domains is asymmetric.
>>>> +
>>>> +* When a domain is created, the toolstack explicitly sends an
>>>> +  ``XS_INTRODUCE(domid, store mfn, store evtchn)`` message to
>>>> xenstored, to
>>>> +  cause xenstored to connect to the guest ring, and fire the
>>>> +  ``@introduceDomain`` watch.
>>>> +* When a domain is destroyed, Xen fires ``VIRQ_DOM_EXC`` which is
>>>> bound by
>>>> +  xenstored, rather than the toolstack.  xenstored updates its idea
>>>> of the
>>>> +  status of domains, and fires the ``@releaseDomain`` watch.
>>>> +
>>>> +Xenstored uses ``xc_domain_getinfo()``, to work out which domain(s)
>>>> have gone
>>>> +away, and only cares about the shutdown status.
>>>> +
>>>> +Furthermore, ``@releaseDomain`` (like ``VIRQ_DOM_EXC``) is a
>>>> single-bit
>>>> +message, which requires all listeners to evaluate whether the
>>>> message applies
>>>> +to them or not.  This results in a flurry of ``xc_domain_getinfo()``
>>>> calls
>>>> +from multiple entities in the system, which all serialise on the
>>>> domctl lock
>>>> +in Xen.
>>>> +
>>>> +Work Items:
>>>> +
>>>> +* Figure out how shutdown status can be expressed in a stable way
>>>> from Xen.
>>>> +* Figure out if ``VIRQ_DOM_EXC`` and ``@releaseDomain`` can be
>>>> extended to
>>>> +  carry at least a domid, to make domain shutdown scale better.
>>>
>>> @releaseDomain (and @introduceDomain) can't be extended, we'd need to
>>> add another watch path like @domainStatus/<domid>/<newState>. Xenstored
>>> could advertise its capability to raise this watch in /tool/xenstored.
>>
>> I guess I was being a bit fast and loose with terminology.  I didn't
>> intend to imply "literally modify @{introduce,release}Domain", as they
>> are already fixed ABIs, but more to "compatibly build something which is
>> better".
>
> Okay.
>
>>
>> That scheme would work for improved @releaseDomain, but it wouldn't work
>> for an improved introduce.  Introduce needs a single key to watch on,
>> which hands back the domid so you don't need to go searching for it.
>
> Yes, and? Its perfectly fine to set a watch firing if anything below
> @domainStatus is changing.

Hmm - that might work if no other information was put into domainStatus,
but would quickly become a scalability problem otherwise.

>
>>
>>>
>>> As VIRQ_DOM_EXC is just an event I don't see how the domid could be
>>> passed by it. I guess we'd need e.g. a shared memory area which the
>>> domain registered for VIRQ_DOM_EXC could map and which would contain a
>>> bitmap (one bit per domain). The hypervisor would set the bit on a
>>> status change and fire VIRQ_DOM_EXC, xenstored would look for a set
>>> bit, clear it and read the status of the related domain.
>>
>> The point here is to avoid using xc_domain_getinfo() in the first place,
>> so there needs to be no "getting the status of the domain".
>
> I'd guess a single xc_domain_getinfo() in the tools wouldn't be so
> problematic. The caller would know the domid already, so no need to
> query all domains.

Its still a problem when you've got 1000 Qemu's, they all get
@releaseDomain, and try to figure out if it is their own domain which
went away.

This shouldn't require taking the domctl lock in Xen 1000 times to
figure out, seeing as xenstored knows exactly which domain actually went
away.

>
>> DOM_EXC is fired for domain_shutdown() only (but for reasons which
>> escape me, fired twice).  Given that a domid is a 15 bit number, a
>> bitmap of all domains does fit within a single 4k page.
>
> Firing twice is needed: first time for disconnecting all backends
> and the second time for cleaning up when the domain is completely
> gone.

Do we really have pieces of code which count the the @releaseDomain's
for a specific domain?  Please say no...

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH] docs/sphinx: todo/wishlist

[Juergen Gross]

On 23.07.19 21:05, Andrew Cooper wrote:
> On 23/07/2019 16:30, Juergen Gross wrote:
>> On 23.07.19 15:38, Andrew Cooper wrote:
>>> On 23/07/2019 05:36, Juergen Gross wrote:
>>>> On 22.07.19 21:20, Andrew Cooper wrote:
>>>>> diff --git a/docs/misc/wishlist.rst b/docs/misc/wishlist.rst
>>>>> new file mode 100644
>>>>> index 0000000000..6cdb47d6e7
>>>>> --- /dev/null
>>>>> +++ b/docs/misc/wishlist.rst
>>>>> @@ -0,0 +1,53 @@
>>>>> +Development Wishlist
>>>>> +====================
>>>>> +
>>>>> +Remove xenstored's dependencies on unstable interfaces
>>>>> +------------------------------------------------------
>>>>> +
>>>>> +Various xenstored implementations use libxc for two purposes.  It
>>>>> would be a
>>>>> +substantial advantage to move xenstored onto entirely stable
>>>>> interfaces, which
>>>>> +disconnects it from the internal of the libxc.
>>>>> +
>>>>> +1. Foreign mapping of the store ring
>>>>> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>>> +
>>>>> +This is obsolete since :xen-cs:`6a2de353a9` (2012) which allocated
>>>>> grant
>>>>> +entries instead, to allow xenstored to function as a stub-domain
>>>>> without dom0
>>>>> +permissions.  :xen-cs:`38eeb3864d` dropped foreign mapping for
>>>>> cxenstored.
>>>>> +However, there are no OCaml bindings for libxengnttab.
>>>>> +
>>>>> +Work Items:
>>>>> +
>>>>> +* Minimal ``tools/ocaml/libs/xg/`` binding for
>>>>> ``tools/libs/gnttab/``.
>>>>> +* Replicate :xen-cs:`38eeb3864d` for oxenstored as well.
>>>>> +
>>>>> +
>>>>> +2. Figuring out which domain(s) have gone away
>>>>> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>>> +
>>>>> +Currently, the handling of domains is asymmetric.
>>>>> +
>>>>> +* When a domain is created, the toolstack explicitly sends an
>>>>> +  ``XS_INTRODUCE(domid, store mfn, store evtchn)`` message to
>>>>> xenstored, to
>>>>> +  cause xenstored to connect to the guest ring, and fire the
>>>>> +  ``@introduceDomain`` watch.
>>>>> +* When a domain is destroyed, Xen fires ``VIRQ_DOM_EXC`` which is
>>>>> bound by
>>>>> +  xenstored, rather than the toolstack.  xenstored updates its idea
>>>>> of the
>>>>> +  status of domains, and fires the ``@releaseDomain`` watch.
>>>>> +
>>>>> +Xenstored uses ``xc_domain_getinfo()``, to work out which domain(s)
>>>>> have gone
>>>>> +away, and only cares about the shutdown status.
>>>>> +
>>>>> +Furthermore, ``@releaseDomain`` (like ``VIRQ_DOM_EXC``) is a
>>>>> single-bit
>>>>> +message, which requires all listeners to evaluate whether the
>>>>> message applies
>>>>> +to them or not.  This results in a flurry of ``xc_domain_getinfo()``
>>>>> calls
>>>>> +from multiple entities in the system, which all serialise on the
>>>>> domctl lock
>>>>> +in Xen.
>>>>> +
>>>>> +Work Items:
>>>>> +
>>>>> +* Figure out how shutdown status can be expressed in a stable way
>>>>> from Xen.
>>>>> +* Figure out if ``VIRQ_DOM_EXC`` and ``@releaseDomain`` can be
>>>>> extended to
>>>>> +  carry at least a domid, to make domain shutdown scale better.
>>>>
>>>> @releaseDomain (and @introduceDomain) can't be extended, we'd need to
>>>> add another watch path like @domainStatus/<domid>/<newState>. Xenstored
>>>> could advertise its capability to raise this watch in /tool/xenstored.
>>>
>>> I guess I was being a bit fast and loose with terminology.  I didn't
>>> intend to imply "literally modify @{introduce,release}Domain", as they
>>> are already fixed ABIs, but more to "compatibly build something which is
>>> better".
>>
>> Okay.
>>
>>>
>>> That scheme would work for improved @releaseDomain, but it wouldn't work
>>> for an improved introduce.  Introduce needs a single key to watch on,
>>> which hands back the domid so you don't need to go searching for it.
>>
>> Yes, and? Its perfectly fine to set a watch firing if anything below
>> @domainStatus is changing.
> 
> Hmm - that might work if no other information was put into domainStatus,
> but would quickly become a scalability problem otherwise.
> 
>>
>>>
>>>>
>>>> As VIRQ_DOM_EXC is just an event I don't see how the domid could be
>>>> passed by it. I guess we'd need e.g. a shared memory area which the
>>>> domain registered for VIRQ_DOM_EXC could map and which would contain a
>>>> bitmap (one bit per domain). The hypervisor would set the bit on a
>>>> status change and fire VIRQ_DOM_EXC, xenstored would look for a set
>>>> bit, clear it and read the status of the related domain.
>>>
>>> The point here is to avoid using xc_domain_getinfo() in the first place,
>>> so there needs to be no "getting the status of the domain".
>>
>> I'd guess a single xc_domain_getinfo() in the tools wouldn't be so
>> problematic. The caller would know the domid already, so no need to
>> query all domains.
> 
> Its still a problem when you've got 1000 Qemu's, they all get
> @releaseDomain, and try to figure out if it is their own domain which
> went away.

They would receive @domainStatus/<domid> so they know which domid has
changed. They might even register for @domainStatus/<domid> only, so
they wouldn't get noticed about any status changes of other domains.

> 
> This shouldn't require taking the domctl lock in Xen 1000 times to
> figure out, seeing as xenstored knows exactly which domain actually went
> away.
> 
>>
>>> DOM_EXC is fired for domain_shutdown() only (but for reasons which
>>> escape me, fired twice).  Given that a domid is a 15 bit number, a
>>> bitmap of all domains does fit within a single 4k page.
>>
>> Firing twice is needed: first time for disconnecting all backends
>> and the second time for cleaning up when the domain is completely
>> gone.
> 
> Do we really have pieces of code which count the the @releaseDomain's
> for a specific domain?  Please say no...

I don't think its is counting, but I'm rather sure it is looking at the
domain state and will react differently for the two cases (see
domain_death_xswatch_callback() in tools/libxl/libxl_domain.c).


Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH] docs/sphinx: todo/wishlist

[Rich Persaud]

[-- Attachment #1.1: Type: text/plain, Size: 3361 bytes --]

> On Jul 22, 2019, at 15:20, Andrew Cooper <andrew.cooper3@citrix.com> wrote:
> 
> a.k.a. (at least in this form) Andrew's "work which might be offloadable to
> someone else" list.
> 
> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
> ---
> CC: George Dunlap <George.Dunlap@eu.citrix.com>
> CC: Ian Jackson <ian.jackson@citrix.com>
> CC: Jan Beulich <JBeulich@suse.com>
> CC: Stefano Stabellini <sstabellini@kernel.org>
> CC: Tim Deegan <tim@xen.org>
> CC: Wei Liu <wl@xen.org>
> CC: Julien Grall <julien.grall@arm.com>
> CC: Lars Kurth <lars.kurth@citrix.com>
> CC: Paul Durrant <paul.durrant@citrix.com>
> CC: Roger Pau Monné <roger.pau@citrix.com>
> 
> RFC for obvious reasons.
> 
> A rendered version of this can be found at:
> https://andrewcoop-xen.readthedocs.io/en/docs-wishlist/misc/wishlist.html
> 
> During XenSummit in Chicago, it was expressed several times that having some
> todo lists would be a benefit, to help coordinate work in related areas.
> 
> Here is an attempt to start one.  For now, it covers one single
> item (xenstored's use of non-stable APIs) to get some feedback about the
> general approach.  I have plenty to get stuck into in Xen itself if this way
> of expressing them isn't deemed unacceptable.
> 
> As for the wishlist itself, I think it is important that it be restricted to
> concrete actions (i.e. already partially groomed, if you speak agile), which
> are identified problems, and suggested fixes.
> 
> In particular, I don't think it is appropriate to devolve into a bullet point
> list of new features, or tasks like "document $whotsit".  It should be
> restricted to things which are real problems, on existing systems, which have
> some forward plan of action.  That way, any developer should be able to
> cross-reference at least at a high level, and see if there are areas of
> overlapping work, or whether a slightly tweaked approach might be suitable for
> multiple areas.
> 
> Anyway - thoughts from the peanut gallery?

Would you consider a permissive, documentation-oriented license, e.g. Creative Commons CC-BY 4.0, for Xen's Sphinx/RST documentation?
https://creativecommons.org/licenses/by/4.0/

As Xen moves beyond cloud computing into multi-vendor, edge/embedded supply chains [1], the audience and context for Xen's technical docs is expanding.  Beyond operating system user/dev/admin, there may be: nested hypervisor user/dev/admin, certification (FuSA), security, firmware/device/accelerator dev, processor architects, formal verification (e.g. TLA+ models), ecosystem building (e.g. blogs, books, videos, training, research) and commercial maintenance manuals for long-lived products with multiple Xen configs and embedded processors.

A permissive license would encourage reuse and tailoring of Xen docs.  With healthy OSS projects, there will remain an incentive to contribute long-lived improvements upstream, even if those improvements are not mandated by the CC license. The Xen wiki license is historically CC-BY-SA 3.0, so that content would be incompatible with CC-BY 4.0.  But Xen's Sphinx/RST docs appear to be focused on new content, so we have an opportunity to choose a license which reflects current community priorities.

Rich

[1] https://dornerworks.com/blog/high-performance-space-computing-platform-nasa-sbir


[-- Attachment #1.2: Type: text/html, Size: 5189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH] docs/sphinx: todo/wishlist

[Lars Kurth]

On 22/07/2019, 20:21, "Andrew Cooper" <andrew.cooper3@citrix.com> wrote:

    a.k.a. (at least in this form) Andrew's "work which might be offloadable to
    someone else" list.
    
    Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
    ---
    CC: George Dunlap <George.Dunlap@eu.citrix.com>
    CC: Ian Jackson <ian.jackson@citrix.com>
    CC: Jan Beulich <JBeulich@suse.com>
    CC: Stefano Stabellini <sstabellini@kernel.org>
    CC: Tim Deegan <tim@xen.org>
    CC: Wei Liu <wl@xen.org>
    CC: Julien Grall <julien.grall@arm.com>
    CC: Lars Kurth <lars.kurth@citrix.com>
    CC: Paul Durrant <paul.durrant@citrix.com>
    CC: Roger Pau Monné <roger.pau@citrix.com>
    
    RFC for obvious reasons.
    
    A rendered version of this can be found at:
    https://andrewcoop-xen.readthedocs.io/en/docs-wishlist/misc/wishlist.html
    
    During XenSummit in Chicago, it was expressed several times that having some
    todo lists would be a benefit, to help coordinate work in related areas.
    
    Here is an attempt to start one.  For now, it covers one single
    item (xenstored's use of non-stable APIs) to get some feedback about the
    general approach.  I have plenty to get stuck into in Xen itself if this way
    of expressing them isn't deemed unacceptable.
    
    As for the wishlist itself, I think it is important that it be restricted to
    concrete actions (i.e. already partially groomed, if you speak agile), which
    are identified problems, and suggested fixes.
    
    In particular, I don't think it is appropriate to devolve into a bullet point
    list of new features, or tasks like "document $whotsit".  It should be
    restricted to things which are real problems, on existing systems, which have
    some forward plan of action.  That way, any developer should be able to
    cross-reference at least at a high level, and see if there are areas of
    overlapping work, or whether a slightly tweaked approach might be suitable for
    multiple areas.
    
    Anyway - thoughts from the peanut gallery?

Hi Andy, sorry I had missed this. I think this is a good idea and worth experimenting with.

Regarding Rich's comment, I thought we already agreed that we can use a CC, but we have not decided how to do this in practical terms as we may be hoovering up content that in the past may have been published as GPL

Lars
 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH] docs/sphinx: todo/wishlist

[Ian Jackson]

Andrew Cooper writes ("Re: [PATCH] docs/sphinx: todo/wishlist"):
> It has been pointed out that calling this the wishlist is a poor name. 
> A better name would be the technical debt list.  I won't resend for just
> this, but please bear it in mind when considering the suggestion.

I approve of the idea of recording technical debt in-tree.

But, do we want it as a centralised list like this ?  I think we
currently have technical debt marked as comments nearby in the
relevant code or interface definitions.  That is often more natural
when developing.

How about instead of this, we invent a fixed string which we can grep
for, which we are to put next to poor areas of interface or code.

So in this case you would add a comment near @introduceDomain or
something.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH] docs/sphinx: todo/wishlist

[Andrew Cooper]

On 09/09/2019 12:02, Ian Jackson wrote:
> Andrew Cooper writes ("Re: [PATCH] docs/sphinx: todo/wishlist"):
>> It has been pointed out that calling this the wishlist is a poor name. 
>> A better name would be the technical debt list.  I won't resend for just
>> this, but please bear it in mind when considering the suggestion.
> I approve of the idea of recording technical debt in-tree.
>
> But, do we want it as a centralised list like this ?

Specifically, yes.

It allows for much easier identification of related areas of work, and
it ends up as a mostly-up-to-date list of small-to-large tasks, e.g. for
students looking for a semester project.

> I think we
> currently have technical debt marked as comments nearby in the
> relevant code or interface definitions.  That is often more natural
> when developing.

For the ~10 or so people working full-time on the project, perhaps. 
Very definitely not for people new to the project.

No-one, not even the main developers, are going to go and grep the
codebase for a todo list, and this is a sure-fire way of not improving
the status quo.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel