Re: [PATCH 1/2] docs/designs/launch: hyperlaunch design document

From: Jan Beulich <jbeulich@suse.com>
To: Stefano Stabellini <sstabellini@kernel.org>,
	Christopher Clark <christopher.w.clark@gmail.com>
Cc: "Roger Pau Monné" <roger.pau@citrix.com>,
	"Daniel P. Smith" <dpsmith@apertussolutions.com>,
	xen-devel <xen-devel@lists.xenproject.org>,
	"Andrew Cooper" <andrew.cooper3@citrix.com>,
	"Stefano Stabellini" <stefano.stabellini@xilinx.com>,
	"Julien Grall" <jgrall@amazon.com>,
	"Julien Grall" <Julien.grall.oss@gmail.com>,
	iwj@xenproject.org, "Wei Liu" <wl@xen.org>,
	"George Dunlap" <george.dunlap@citrix.com>,
	"Rich Persaud" <persaur@gmail.com>,
	"Bertrand Marquis" <Bertrand.Marquis@arm.com>,
	luca.fancellu@arm.com, paul@xen.org,
	"Adam Schwalm" <adam.schwalm@starlab.io>
Subject: Re: [PATCH 1/2] docs/designs/launch: hyperlaunch design document
Date: Thu, 25 Mar 2021 09:07:10 +0100	[thread overview]
Message-ID: <95447252-c50e-9521-4c0d-b5297b22367d@suse.com> (raw)
In-Reply-To: <alpine.DEB.2.21.2103241206430.439@sstabellini-ThinkPad-T480s>

On 24.03.2021 20:10, Stefano Stabellini wrote:
> On Wed, 24 Mar 2021, Christopher Clark wrote:
>>>>> I'm also not sure how you are going to handle things like SR-IOV
>>>>> devices. Right now SR-IOV capability is setup and initialized by the
>>>>> hardware domain, and the new virtual devices are notified to Xen once
>>>>> setup is done. Do you plan to move those bits into Xen, so that it can
>>>>> setup and initialize the SR-IOV capability?
>>>>
>>>> While you could do it with the vPCI, as you point out this will not work
>>>> for SR-IOV. With hyperlaunch, these cases will require the use of a boot
>>>> domain, which is for all intents and purposes, a lightweight/restricted
>>>> toolstack domain.
>>>>
>>>> The boot domain will have to do the necessary operations to ensure that
>>>> when startup is finished, PCI passthrough will be successfully setup.
>>>> Note, this may have to include the boot domain unpausing the hardware
>>>> domain to help complete the setup before the boot domain can exit and
>>>> allow the remaining domains to come online.
>>>
>>> OK, I was expecting hyperlaunch to do all domain creation in the
>>> hypervisor.
>>
>> That is my expectation too. It is what we've been planning for in our
>> work so far but we can work on explaining the steps involved in
>> constructing the domains more clearly.
>>
>>> If you offload domain creation of guests with
>>> pci-passthrough devices to a control domain and/or hardware domain,
>>> I'm not sure I see the difference from normal domain creation, ie:
>>> it's no longer something specific to hyperlaunch, as I could achieve
>>> the same by using the existing xendomains init script.
>>
>> So that's not what we've proposed, and hopefully not what we'll need to do.
>>
>> Do you know if there is a need to perform work to support the
>> assignment of PCI devices at the point of domain creation (ie. in
>> domain_create), rather than handling it in a later step of domain
>> configuration, prior to the domain being started?
> 
> Let's leave SR-IOV aside for a moment that can be a bit strange (if I
> recall correctly sometimes VFs can only appear after the PF driver has
> been loaded).

Yes, and iirc beyond that the number of VFs can (in Linux at least)
also be controlled (read: changed) when the driver is already
loaded, via sysfs. I think such flexibility is desirable, as a
device's resources may get partitioned differently depending on the
number of VFs, and hence the number of VFs may affect individual
VFs' (and maybe also the PF's) performance. Otoh if the system as
a whole gets statically partitioned, one-time setup of VFs will of
course do.

I don't think though that SR-IOV can be left aside - whatever is to
be done here needs to take it into consideration from the very
beginning.

Jan