On Thu, 25 Mar 2021, Roger Pau Monné wrote: > On Thu, Mar 25, 2021 at 10:14:31AM +0100, George Dunlap wrote: > > > > > > > On Mar 25, 2021, at 8:32 AM, Roger Pau Monne wrote: > > > > > > On Wed, Mar 24, 2021 at 05:53:26AM -0700, Christopher Clark wrote: > > >> On Wed, Mar 24, 2021 at 1:01 AM Roger Pau Monné wrote: > > >>> > > >>> On Tue, Mar 23, 2021 at 10:39:53AM -0700, Christopher Clark wrote: > > >>>> On Thu, Mar 18, 2021 at 9:43 AM Roger Pau Monné wrote: > > >>> If you offload domain creation of guests with > > >>> pci-passthrough devices to a control domain and/or hardware domain, > > >>> I'm not sure I see the difference from normal domain creation, ie: > > >>> it's no longer something specific to hyperlaunch, as I could achieve > > >>> the same by using the existing xendomains init script. > > >> > > >> So that's not what we've proposed, and hopefully not what we'll need to do. > > >> > > >> Do you know if there is a need to perform work to support the > > >> assignment of PCI devices at the point of domain creation (ie. in > > >> domain_create), rather than handling it in a later step of domain > > >> configuration, prior to the domain being started? > > > > > > So while I think you could indeed create a domain from the hypervisor > > > in a paused state and attach the pci devices later from a > > > control/hardware domain, I don't see much benefit in doing it. If you > > > need to end up waiting for a control/hardware domain to attach the > > > devices and unpause you might as well do the whole domain creation > > > from such control/hardware domain. > > > > My understanding was that one of the primary advantages of domB was > > that you could compile and run arbitrary code in whatever language > > you wanted to, using already known tools. If *all* we want to do is > > to assign some pre-defined specific BDFs to specific domains, then > > sure, we could add that capability to Xen. > > Well, it's not so easy because we require QEMU or pciback ATM on x86 > in order to do pci passthrough to guests, so assigning BDFs to > specific domains from the hypervisor would need to be done using vPCI > (which is not yet ready for unprivileged guest usage) and only support > HVM kind of guests strictly. > > Doing passthrough to PV guests directly from the hypervisor is > impossible AFAICT without putting pciback inside of the hypervisor. > > > But suppose you wanted > > to have the flexibility to handle changes in hardware at boot time? > > “Scan through the PCI bus and assign anything that looks like a > > network card to domNet, and anything that looks like a USB > > controller to domUSB” is something you could easily do in domB, but > > would be way too complicated to add to Xen. > > Right, but then you might as well create the domain from domB instead > of doing it in the hypervisor? > > I'm not arguing about not using domB, I just don't see the benefit of > creating a paused domain from the hypervisor that then requires the > intervention of a further domain (domB) in order to finish creation. > Won't it be simpler to just create the domain and attach the pci > devices from domB? I think that the ability of creating multiple VMs from Xen is actually a very good one to have for a few reasons. It would align x86 with ARM, it would be useful in cases where PCI passthrough is not involved, and it is a powerful tool to have in our toolbox. I see that handling PCI passthrough at domain creation time can be difficult, so I think Christopher's solution is a good compromise. FYI for dom0less/ARM we have been discussing doing device assignment at creation time, but the idea was to provide the PCI topology in device tree to Xen to help with discovery.