Re: [PATCH v5 03/18] pci: isolated address space for PCI bus

From: Stefan Hajnoczi <stefanha@redhat.com>
To: Alex Williamson <alex.williamson@redhat.com>
Cc: "eduardo@habkost.net" <eduardo@habkost.net>,
	"Elena Ufimtseva" <elena.ufimtseva@oracle.com>,
	"Jag Raman" <jag.raman@oracle.com>,
	"Beraldo Leal" <bleal@redhat.com>,
	"John Johnson" <john.g.johnson@oracle.com>,
	"Michael S. Tsirkin" <mst@redhat.com>,
	qemu-devel <qemu-devel@nongnu.org>,
	"armbru@redhat.com" <armbru@redhat.com>,
	"quintela@redhat.com" <quintela@redhat.com>,
	"Paolo Bonzini" <pbonzini@redhat.com>,
	"Marc-André Lureau" <marcandre.lureau@gmail.com>,
	"Dr. David Alan Gilbert" <dgilbert@redhat.com>,
	"thanos.makatos@nutanix.com" <thanos.makatos@nutanix.com>,
	"Daniel P. Berrangé" <berrange@redhat.com>,
	"Eric Blake" <eblake@redhat.com>,
	"john.levon@nutanix.com" <john.levon@nutanix.com>,
	"Philippe Mathieu-Daudé" <f4bug@amsat.org>
Subject: Re: [PATCH v5 03/18] pci: isolated address space for PCI bus
Date: Fri, 28 Jan 2022 09:18:08 +0000	[thread overview]
Message-ID: <YfO00O4sT2UgaW31@stefanha-x1.localdomain> (raw)
In-Reply-To: <20220127142253.21ab0025.alex.williamson@redhat.com>

[-- Attachment #1: Type: text/plain, Size: 1725 bytes --]

On Thu, Jan 27, 2022 at 02:22:53PM -0700, Alex Williamson wrote:
> If the goal here is to restrict DMA between devices, ie. peer-to-peer
> (p2p), why are we trying to re-invent what an IOMMU already does?

The issue Dave raised is that vfio-user servers run in separate
processses from QEMU with shared memory access to RAM but no direct
access to non-RAM MemoryRegions. The virtiofs DAX Window BAR is one
example of a non-RAM MemoryRegion that can be the source/target of DMA
requests.

I don't think IOMMUs solve this problem but luckily the vfio-user
protocol already has messages that vfio-user servers can use as a
fallback when DMA cannot be completed through the shared memory RAM
accesses.

> In
> fact, it seems like an IOMMU does this better in providing an IOVA
> address space per BDF.  Is the dynamic mapping overhead too much?  What
> physical hardware properties or specifications could we leverage to
> restrict p2p mappings to a device?  Should it be governed by machine
> type to provide consistency between devices?  Should each "isolated"
> bus be in a separate root complex?  Thanks,

There is a separate issue in this patch series regarding isolating the
address space where BAR accesses are made (i.e. the global
address_space_memory/io). When one process hosts multiple vfio-user
server instances (e.g. a software-defined network switch with multiple
ethernet devices) then each instance needs isolated memory and io address
spaces so that vfio-user clients don't cause collisions when they map
BARs to the same address.

I think the the separate root complex idea is a good solution. This
patch series takes a different approach by adding the concept of
isolated address spaces into hw/pci/.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]