Re: [PATCH 1/2] docs/designs/launch: hyperlaunch design document

From: Stefano Stabellini <sstabellini@kernel.org>
To: Christopher Clark <christopher.w.clark@gmail.com>
Cc: "Roger Pau Monné" <roger.pau@citrix.com>,
	"Daniel P. Smith" <dpsmith@apertussolutions.com>,
	xen-devel <xen-devel@lists.xenproject.org>,
	"Andrew Cooper" <andrew.cooper3@citrix.com>,
	"Stefano Stabellini" <stefano.stabellini@xilinx.com>,
	"Julien Grall" <jgrall@amazon.com>,
	"Julien Grall" <Julien.grall.oss@gmail.com>,
	iwj@xenproject.org, "Wei Liu" <wl@xen.org>,
	"George Dunlap" <george.dunlap@citrix.com>,
	"Jan Beulich" <jbeulich@suse.com>,
	"Rich Persaud" <persaur@gmail.com>,
	"Bertrand Marquis" <Bertrand.Marquis@arm.com>,
	luca.fancellu@arm.com, paul@xen.org,
	"Adam Schwalm" <adam.schwalm@starlab.io>
Subject: Re: [PATCH 1/2] docs/designs/launch: hyperlaunch design document
Date: Wed, 24 Mar 2021 12:10:31 -0700 (PDT)	[thread overview]
Message-ID: <alpine.DEB.2.21.2103241206430.439@sstabellini-ThinkPad-T480s> (raw)
In-Reply-To: <CACMJ4GZMNfbCjTKR5ngQBMwwAfUnSOptP+UJV8BcNqKAUy9Row@mail.gmail.com>

On Wed, 24 Mar 2021, Christopher Clark wrote:
> > > > I'm also not sure how you are going to handle things like SR-IOV
> > > > devices. Right now SR-IOV capability is setup and initialized by the
> > > > hardware domain, and the new virtual devices are notified to Xen once
> > > > setup is done. Do you plan to move those bits into Xen, so that it can
> > > > setup and initialize the SR-IOV capability?
> > >
> > > While you could do it with the vPCI, as you point out this will not work
> > > for SR-IOV. With hyperlaunch, these cases will require the use of a boot
> > > domain, which is for all intents and purposes, a lightweight/restricted
> > > toolstack domain.
> > >
> > > The boot domain will have to do the necessary operations to ensure that
> > > when startup is finished, PCI passthrough will be successfully setup.
> > > Note, this may have to include the boot domain unpausing the hardware
> > > domain to help complete the setup before the boot domain can exit and
> > > allow the remaining domains to come online.
> >
> > OK, I was expecting hyperlaunch to do all domain creation in the
> > hypervisor.
> 
> That is my expectation too. It is what we've been planning for in our
> work so far but we can work on explaining the steps involved in
> constructing the domains more clearly.
> 
> > If you offload domain creation of guests with
> > pci-passthrough devices to a control domain and/or hardware domain,
> > I'm not sure I see the difference from normal domain creation, ie:
> > it's no longer something specific to hyperlaunch, as I could achieve
> > the same by using the existing xendomains init script.
> 
> So that's not what we've proposed, and hopefully not what we'll need to do.
> 
> Do you know if there is a need to perform work to support the
> assignment of PCI devices at the point of domain creation (ie. in
> domain_create), rather than handling it in a later step of domain
> configuration, prior to the domain being started?

Let's leave SR-IOV aside for a moment that can be a bit strange (if I
recall correctly sometimes VFs can only appear after the PF driver has
been loaded).

Just for my understanding, for regular PCI passthrough, why is it
important that the device assignment is done later by the control
domain, instead of straight away at domain creation by Xen?

I am fine either way but I meant to ask if it is just to make the
implementation easier or if there is some kind of policy involved.