Re: [PATCH 1/2] docs/designs/launch: hyperlaunch design document

[Christopher Clark <christopher.w.clark@gmail.com>]

On Wed, Mar 24, 2021 at 1:01 AM Roger Pau Monné <roger.pau@citrix.com> wrote:
>
> On Tue, Mar 23, 2021 at 10:39:53AM -0700, Christopher Clark wrote:
> > On Thu, Mar 18, 2021 at 9:43 AM Roger Pau Monné <roger.pau@citrix.com> wrote:
> > >
> > > Just took a quick look at it.
> > >
> > > On Mon, Mar 15, 2021 at 11:18:13PM -0400, Daniel P. Smith wrote:
> > > > + +---------------+-----------+------------+-----------+-------------+---------------------+
> > > > + | **Xen Dom0**  | **Linux** | **Late**   | **Jail**  | **Xen**     | **Xen Hyperlaunch** |
> > > > + | **(Classic)** | **KVM**   | **HW Dom** | **house** | **dom0less**+---------+-----------+
> > > > + |               |           |            |           |             | Static  | Dynamic   |
> > > > + +===============+===========+============+===========+=============+=========+===========+
> > > > + | Hypervisor able to launch multiple VMs during host boot                                |
> > > > + +---------------+-----------+------------+-----------+-------------+---------+-----------+
> > > > + |               |           |            |     Y     |       Y     |    Y    |     Y     |
> > > > + +---------------+-----------+------------+-----------+-------------+---------+-----------+
> > > > + | Hypervisor supports Static Partitioning                                                |
> > > > + +---------------+-----------+------------+-----------+-------------+---------+-----------+
> > > > + |               |           |            |     Y     |       Y     |    Y    |           |
> > > > + +---------------+-----------+------------+-----------+-------------+---------+-----------+
> > > > + | Able to launch VMs dynamically after host boot                                         |
> > > > + +---------------+-----------+------------+-----------+-------------+---------+-----------+
> > > > + |       Y       |     Y     |      Y*    |     Y     |       Y*    |         |     Y     |
> > > > + +---------------+-----------+------------+-----------+-------------+---------+-----------+
> > > > + | Supports strong isolation between all VMs started at host boot                         |
> > > > + +---------------+-----------+------------+-----------+-------------+---------+-----------+
> > > > + |               |           |            |     Y     |       Y     |    Y    |     Y     |
> > > > + +---------------+-----------+------------+-----------+-------------+---------+-----------+
> > > > + | Enables flexible sequencing of VM start during host boot                               |
> > > > + +---------------+-----------+------------+-----------+-------------+---------+-----------+
> > > > + |               |           |            |           |             |    Y    |     Y     |
> > > > + +---------------+-----------+------------+-----------+-------------+---------+-----------+
> > > > + | Prevent all-powerful static root domain being launched at boot                         |
> > > > + +---------------+-----------+------------+-----------+-------------+---------+-----------+
> > > > + |               |           |            |           |       Y*    |    Y    |     Y     |
> > > > + +---------------+-----------+------------+-----------+-------------+---------+-----------+
> > > > + | Operates without a Highly-privileged management VM (eg. Dom0)                          |
> > > > + +---------------+-----------+------------+-----------+-------------+---------+-----------+
> > > > + |               |           |      Y*    |           |       Y*    |    Y    |     Y     |
> > > > + +---------------+-----------+------------+-----------+-------------+---------+-----------+
> > > > + | Operates without a privileged toolstack VM (Control Domain)                            |
> > > > + +---------------+-----------+------------+-----------+-------------+---------+-----------+
> > > > + |               |           |            |           |       Y*    |    Y    |           |
> > > > + +---------------+-----------+------------+-----------+-------------+---------+-----------+
> > > > + | Extensible VM configuration applied before launch of VMs at host boot                  |
> > > > + +---------------+-----------+------------+-----------+-------------+---------+-----------+
> > > > + |               |           |            |           |             |    Y    |     Y     |
> > > > + +---------------+-----------+------------+-----------+-------------+---------+-----------+
> > > > + | Flexible granular assignment of permissions and functions to VMs                       |
> > > > + +---------------+-----------+------------+-----------+-------------+---------+-----------+
> > > > + |               |           |            |           |             |    Y    |     Y     |
> > > > + +---------------+-----------+------------+-----------+-------------+---------+-----------+
> > > > + | Supports extensible VM measurement architecture for DRTM and attestation               |
> > > > + +---------------+-----------+------------+-----------+-------------+---------+-----------+
> > > > + |               |           |            |           |             |    Y    |     Y     |
> > > > + +---------------+-----------+------------+-----------+-------------+---------+-----------+
> > > > + | PCI passthrough configured at host boot                                                |
> > > > + +---------------+-----------+------------+-----------+-------------+---------+-----------+
> > > > + |               |           |            |           |             |    Y    |     Y     |
> > > > + +---------------+-----------+------------+-----------+-------------+---------+-----------+
> > >
> > > I'm curious about this, I assume this is done using vPCI so that
> > > there's no hardware domain (and user-space device model) involved for
> > > PCI passthrough?
> >
> > That would be an incorrect assumption. See below for why.
> >
> > > I'm also not sure how you are going to handle things like SR-IOV
> > > devices. Right now SR-IOV capability is setup and initialized by the
> > > hardware domain, and the new virtual devices are notified to Xen once
> > > setup is done. Do you plan to move those bits into Xen, so that it can
> > > setup and initialize the SR-IOV capability?
> >
> > While you could do it with the vPCI, as you point out this will not work
> > for SR-IOV. With hyperlaunch, these cases will require the use of a boot
> > domain, which is for all intents and purposes, a lightweight/restricted
> > toolstack domain.
> >
> > The boot domain will have to do the necessary operations to ensure that
> > when startup is finished, PCI passthrough will be successfully setup.
> > Note, this may have to include the boot domain unpausing the hardware
> > domain to help complete the setup before the boot domain can exit and
> > allow the remaining domains to come online.
>
> OK, I was expecting hyperlaunch to do all domain creation in the
> hypervisor.

That is my expectation too. It is what we've been planning for in our
work so far but we can work on explaining the steps involved in
constructing the domains more clearly.

> If you offload domain creation of guests with
> pci-passthrough devices to a control domain and/or hardware domain,
> I'm not sure I see the difference from normal domain creation, ie:
> it's no longer something specific to hyperlaunch, as I could achieve
> the same by using the existing xendomains init script.

So that's not what we've proposed, and hopefully not what we'll need to do.

Do you know if there is a need to perform work to support the
assignment of PCI devices at the point of domain creation (ie. in
domain_create), rather than handling it in a later step of domain
configuration, prior to the domain being started?

> Also you need a way to pass the configuration from the hypervisor into
> a control domain that would then wait for the hardware domain to come
> up and afterwards launch a guest with a pci-passthorugh device. The
> passing of this information from the hypervisor to the control domain
> would need to be done in an OS agnostic way if possible.

Ack. We have discussed a plan for surfacing the domain configuration
data from the Launch Control Module to the boot domain via either ACPI
tables or a Device Tree -- this needs to be added to the design
documents. Communicating the domain configuration information to the
control domain too also needs consideration. Thanks for raising it.

Earlier discussion notes were posted here:
https://lists.xenproject.org/archives/html/xen-devel/2020-07/msg00729.html

> Don't get me wrong, I don't think such approach is bad, I'm just
> unsure whether such functionality is really part of hyperlaunch, or
> instead something that you can achieve outside of hyperlaunch already.

I think that it will provide a new capability; will work on the docs
on how to better communicate how it does so.

thanks,

Christopher

>
> Thanks, Roger.