Re: [Qemu-devel] About virtio device hotplug in Q35! 【外域邮件.谨慎查阅】

[Alex Williamson <alex.williamson@redhat.com>]

On Wed, 26 Jul 2017 19:06:58 +0300
"Michael S. Tsirkin" <mst@redhat.com> wrote:

> On Wed, Jul 26, 2017 at 09:29:31AM -0600, Alex Williamson wrote:
> > On Wed, 26 Jul 2017 09:21:38 +0300
> > Marcel Apfelbaum <marcel@redhat.com> wrote:
> >   
> > > On 25/07/2017 11:53, 陈博 wrote:  
> > > > To accelerate data traversing between devices under the same PCIE Root 
> > > > Port or Switch.
> > > > 
> > > > See https://lists.nongnu.org/archive/html/qemu-devel/2017-07/msg07209.html
> > > >     
> > > 
> > > Hi,
> > > 
> > > It may be possible, but maybe PCIe Switch assignment is not
> > > the only way to go.
> > > 
> > > Adding Alex and Michael for their input on this matter.
> > > More info at:
> > > https://lists.nongnu.org/archive/html/qemu-devel/2017-07/msg07209.html  
> > 
> > I think you need to look at where the IOMMU is in the topology and what
> > address space the devices are working in when assigned to a VM to
> > realize that it doesn't make any sense to assign switch ports to a VM.
> > GPUs cannot do switch level peer to peer when assigned because they are
> > operating in an I/O virtual address space.  This is why we configure
> > ACS on downstream ports to prevent peer to peer.  Peer to peer
> > transactions must be forwarded upstream by the switch ports in order to
> > reach the IOMMU for translation.  Note however that we do populate peer
> > to peer mappings within the IOMMU, so if the hardware supports it, the
> > IOMMU can reflect the transaction back out to the I/O bus to reach the
> > other device without CPU involvement.
> > 
> > Therefore I think the better solution, if it encourages the NVIDIA
> > driver to do the right thing, is to use emulated switches.  Assigning
> > the physical switch would really do nothing more than make the PCIe link
> > information more correct in the VM, everything else about the switch
> > would be emulated.  Even still, unless you have an I/O topology which
> > integrates the IOMMU into the switch itself, the data flow still needs
> > to go all the way to the root complex to hit the IOMMU before being
> > reflected to the other device.  Direct peer to peer between downstream
> > switch ports operates in the wrong address space.  Thanks,
> > 
> > Alex  
> 
> That's true of course. What would make sense would be for
> hardware vendors to add ATS support to their cards.
> 
> Then peer to peer should be allowed by hypervisor for translated transactions.
> 
> Gives you the performance benefit without the security issues.
> 
> Does anyone know whether any hardware implements this?

GPUs often do implement ATS and the ACS DT (Direct Translated P2P)
capability should handle routing requests with the Address Type field
indicating a translated address directly between downstream ports.  DT
is however not part of the standard set of ACS bits that we enable.  It
seems like it might be fairly easy to poke the DT enable bit with
setpci from userspace to test whether this "just works", providing of
course you can get the driver to attempt to do peer to peer and ATS is
already functioning on the GPU.  If so, then we should look at where
in the code to do that enabling automatically.  Thanks,

Alex