Re: [PATCH v5 03/18] pci: isolated address space for PCI bus

From: Stefan Hajnoczi <stefanha@redhat.com>
To: "Michael S. Tsirkin" <mst@redhat.com>
Cc: "eduardo@habkost.net" <eduardo@habkost.net>,
	"Elena Ufimtseva" <elena.ufimtseva@oracle.com>,
	"John Johnson" <john.g.johnson@oracle.com>,
	"Jag Raman" <jag.raman@oracle.com>,
	"Beraldo Leal" <bleal@redhat.com>,
	"quintela@redhat.com" <quintela@redhat.com>,
	qemu-devel <qemu-devel@nongnu.org>,
	"armbru@redhat.com" <armbru@redhat.com>,
	"john.levon@nutanix.com" <john.levon@nutanix.com>,
	"Philippe Mathieu-Daudé" <f4bug@amsat.org>,
	"Dr. David Alan Gilbert" <dgilbert@redhat.com>,
	"Marc-André Lureau" <marcandre.lureau@gmail.com>,
	"Daniel P. Berrangé" <berrange@redhat.com>,
	"thanos.makatos@nutanix.com" <thanos.makatos@nutanix.com>,
	"Paolo Bonzini" <pbonzini@redhat.com>,
	"Eric Blake" <eblake@redhat.com>
Subject: Re: [PATCH v5 03/18] pci: isolated address space for PCI bus
Date: Thu, 27 Jan 2022 08:30:13 +0000	[thread overview]
Message-ID: <YfJYFbBVQzho7mGp@stefanha-x1.localdomain> (raw)
In-Reply-To: <20220126161120-mutt-send-email-mst@kernel.org>

[-- Attachment #1: Type: text/plain, Size: 4043 bytes --]

On Wed, Jan 26, 2022 at 04:13:33PM -0500, Michael S. Tsirkin wrote:
> On Wed, Jan 26, 2022 at 08:07:36PM +0000, Dr. David Alan Gilbert wrote:
> > * Stefan Hajnoczi (stefanha@redhat.com) wrote:
> > > On Wed, Jan 26, 2022 at 05:27:32AM +0000, Jag Raman wrote:
> > > > 
> > > > 
> > > > > On Jan 25, 2022, at 1:38 PM, Dr. David Alan Gilbert <dgilbert@redhat.com> wrote:
> > > > > 
> > > > > * Jag Raman (jag.raman@oracle.com) wrote:
> > > > >> 
> > > > >> 
> > > > >>> On Jan 19, 2022, at 7:12 PM, Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > >>> 
> > > > >>> On Wed, Jan 19, 2022 at 04:41:52PM -0500, Jagannathan Raman wrote:
> > > > >>>> Allow PCI buses to be part of isolated CPU address spaces. This has a
> > > > >>>> niche usage.
> > > > >>>> 
> > > > >>>> TYPE_REMOTE_MACHINE allows multiple VMs to house their PCI devices in
> > > > >>>> the same machine/server. This would cause address space collision as
> > > > >>>> well as be a security vulnerability. Having separate address spaces for
> > > > >>>> each PCI bus would solve this problem.
> > > > >>> 
> > > > >>> Fascinating, but I am not sure I understand. any examples?
> > > > >> 
> > > > >> Hi Michael!
> > > > >> 
> > > > >> multiprocess QEMU and vfio-user implement a client-server model to allow
> > > > >> out-of-process emulation of devices. The client QEMU, which makes ioctls
> > > > >> to the kernel and runs VCPUs, could attach devices running in a server
> > > > >> QEMU. The server QEMU needs access to parts of the client’s RAM to
> > > > >> perform DMA.
> > > > > 
> > > > > Do you ever have the opposite problem? i.e. when an emulated PCI device
> > > > 
> > > > That’s an interesting question.
> > > > 
> > > > > exposes a chunk of RAM-like space (frame buffer, or maybe a mapped file)
> > > > > that the client can see.  What happens if two emulated devices need to
> > > > > access each others emulated address space?
> > > > 
> > > > In this case, the kernel driver would map the destination’s chunk of internal RAM into
> > > > the DMA space of the source device. Then the source device could write to that
> > > > mapped address range, and the IOMMU should direct those writes to the
> > > > destination device.
> > > > 
> > > > I would like to take a closer look at the IOMMU implementation on how to achieve
> > > > this, and get back to you. I think the IOMMU would handle this. Could you please
> > > > point me to the IOMMU implementation you have in mind?
> > > 
> > > I don't know if the current vfio-user client/server patches already
> > > implement device-to-device DMA, but the functionality is supported by
> > > the vfio-user protocol.
> > > 
> > > Basically: if the DMA regions lookup inside the vfio-user server fails,
> > > fall back to VFIO_USER_DMA_READ/WRITE messages instead.
> > > https://github.com/nutanix/libvfio-user/blob/master/docs/vfio-user.rst#vfio-user-dma-read
> > > 
> > > Here is the flow:
> > > 1. The vfio-user server with device A sends a DMA read to QEMU.
> > > 2. QEMU finds the MemoryRegion associated with the DMA address and sees
> > >    it's a device.
> > >    a. If it's emulated inside the QEMU process then the normal
> > >       device emulation code kicks in.
> > >    b. If it's another vfio-user PCI device then the vfio-user PCI proxy
> > >       device forwards the DMA to the second vfio-user server's device B.
> > 
> > I'm starting to be curious if there's a way to persuade the guest kernel
> > to do it for us; in general is there a way to say to PCI devices that
> > they can only DMA to the host and not other PCI devices?
> 
> 
> But of course - this is how e.g. VFIO protects host PCI devices from
> each other when one of them is passed through to a VM.

Michael: Are you saying just turn on vIOMMU? :)

Devices in different VFIO groups have their own IOMMU context, so their
IOVA space is isolated. Just don't map other devices into the IOVA space
and those other devices will be inaccessible.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]