Re: [PATCH v5 07/18] vfio-user: set qdev bus callbacks for remote machine

From: Jag Raman <jag.raman@oracle.com>
To: Stefan Hajnoczi <stefanha@redhat.com>
Cc: "eduardo@habkost.net" <eduardo@habkost.net>,
	"Elena Ufimtseva" <elena.ufimtseva@oracle.com>,
	"John Johnson" <john.g.johnson@oracle.com>,
	"berrange@redhat.com" <berrange@redhat.com>,
	"Beraldo Leal" <bleal@redhat.com>,
	"john.levon@nutanix.com" <john.levon@nutanix.com>,
	"mst@redhat.com" <mst@redhat.com>,
	"armbru@redhat.com" <armbru@redhat.com>,
	"quintela@redhat.com" <quintela@redhat.com>,
	qemu-devel <qemu-devel@nongnu.org>,
	"Philippe Mathieu-Daudé" <f4bug@amsat.org>,
	"Marc-André Lureau" <marcandre.lureau@gmail.com>,
	"thanos.makatos@nutanix.com" <thanos.makatos@nutanix.com>,
	"Paolo Bonzini" <pbonzini@redhat.com>,
	"eblake@redhat.com" <eblake@redhat.com>,
	"dgilbert@redhat.com" <dgilbert@redhat.com>
Subject: Re: [PATCH v5 07/18] vfio-user: set qdev bus callbacks for remote machine
Date: Tue, 25 Jan 2022 21:12:28 +0000	[thread overview]
Message-ID: <5C83E65E-33B6-4282-9276-D651B2E56592@oracle.com> (raw)
In-Reply-To: <Ye/UqqSippA8LTHK@stefanha-x1.localdomain>

> On Jan 25, 2022, at 5:44 AM, Stefan Hajnoczi <stefanha@redhat.com> wrote:
> 
> On Wed, Jan 19, 2022 at 04:41:56PM -0500, Jagannathan Raman wrote:
>> Signed-off-by: Elena Ufimtseva <elena.ufimtseva@oracle.com>
>> Signed-off-by: John G Johnson <john.g.johnson@oracle.com>
>> Signed-off-by: Jagannathan Raman <jag.raman@oracle.com>
>> ---
>> hw/remote/machine.c | 57 +++++++++++++++++++++++++++++++++++++++++++++
>> 1 file changed, 57 insertions(+)
>> 
>> diff --git a/hw/remote/machine.c b/hw/remote/machine.c
>> index 220ff01aa9..221a8430c1 100644
>> --- a/hw/remote/machine.c
>> +++ b/hw/remote/machine.c
>> @@ -22,6 +22,60 @@
>> #include "hw/pci/pci_host.h"
>> #include "hw/remote/iohub.h"
>> 
>> +static bool remote_machine_get_bus(const char *type, BusState **bus,
>> +                                   Error **errp)
>> +{
>> +    ERRP_GUARD();
>> +    RemoteMachineState *s = REMOTE_MACHINE(current_machine);
>> +    BusState *root_bus = NULL;
>> +    PCIBus *new_pci_bus = NULL;
>> +
>> +    if (!bus) {
>> +        error_setg(errp, "Invalid argument");
>> +        return false;
>> +    }
>> +
>> +    if (strcmp(type, TYPE_PCI_BUS) && strcmp(type, TYPE_PCI_BUS)) {
>> +        return true;
>> +    }
>> +
>> +    root_bus = qbus_find_recursive(sysbus_get_default(), NULL, TYPE_PCIE_BUS);
>> +    if (!root_bus) {
>> +        error_setg(errp, "Unable to find root PCI device");
>> +        return false;
>> +    }
>> +
>> +    new_pci_bus = pci_isol_bus_new(root_bus, type, errp);
>> +    if (!new_pci_bus) {
>> +        return false;
>> +    }
>> +
>> +    *bus = BUS(new_pci_bus);
>> +
>> +    pci_bus_irqs(new_pci_bus, remote_iohub_set_irq, remote_iohub_map_irq,
>> +                 &s->iohub, REMOTE_IOHUB_NB_PIRQS);
>> +
>> +    return true;
>> +}
> 
> Can the user create the same PCI bus via QMP commands? If so, then this

I think there is a way we could achieve it.

When I looked around, both the command line and the QMP didn’t have a direct
way to create a bus. However, there are some indirect ways. For example, the
TYPE_LSI53C895A device creates a SCSI bus to attach SCSI devices. Similarly,
there are some special PCI devices like TYPE_PCI_BRIDGE which create a
secondary PCI bus.

Similarly, we could implement a PCI device that creates a PCI bus with
isolated address spaces.

> is just a convenience that saves the extra step. Or is there some magic
> that cannot be done via QMP device_add?
> 
> I'm asking because there are 3 objects involved and I'd like to
> understand the lifecycle/dependencies:
> 1. The PCIDevice we wish to export.
> 2. The PCIBus with isolated address spaces that contains the PCIDevice.
> 3. The vfio-user server that exports a given PCIDevice.
> 
> Users can already create the PCIDevice via hotplug and the vfio-user
> server via object-add. So if there's no magic they could also create the
> PCI bus:
> 1. device_add ...some PCI bus stuff here...,id=isol-pci-bus0
> 2. device_add ...the PCIDevice...,bus=isol-pci-bus0,id=mydev
> 3. object-add x-vfio-user-server,device=mydev

We are able to do 2 & 3 already. We could introduce a PCI device that
creates an isolated PCI bus. That would cover step 1 outlined above.

> 
> Unplug would work in the reverse order.
> 
> It may be more convenient to automatically create a PCIBus when the
> PCIDevice is hotplugged, but this kind of magic also has drawbacks
> (hidden devices, naming collisions, etc).

OK, makes sense.

--
Jag

> 
>> +
>> +static bool remote_machine_put_bus(BusState *bus, Error **errp)
>> +{
>> +    PCIBus *pci_bus = NULL;
>> +
>> +    if (!bus) {
>> +        error_setg(errp, "Invalid argument");
>> +        return false;
>> +    }
>> +
>> +    if (!object_dynamic_cast(OBJECT(bus), TYPE_PCI_BUS)) {
>> +        return true;
>> +    }
>> +
>> +    pci_bus = PCI_BUS(bus);
>> +
>> +    return pci_isol_bus_free(pci_bus, errp);
>> +}
>> +
>> static void remote_machine_init(MachineState *machine)
>> {
>>     MemoryRegion *system_memory, *system_io, *pci_memory;
>> @@ -56,6 +110,9 @@ static void remote_machine_init(MachineState *machine)
>>                  &s->iohub, REMOTE_IOHUB_NB_PIRQS);
>> 
>>     qbus_set_hotplug_handler(BUS(pci_host->bus), OBJECT(s));
>> +
>> +    qdev_set_bus_cbs(remote_machine_get_bus, remote_machine_put_bus,
>> +                     &error_fatal);
>> }
>> 
>> static void remote_machine_pre_plug_cb(HotplugHandler *hotplug_dev,
>> -- 
>> 2.20.1
>> 