Re: [Qemu-devel] issues of region cache and iommu reset

From: "Michael S. Tsirkin" <mst@redhat.com>
To: Peter Xu <peterx@redhat.com>
Cc: Jason Wang <jasowang@redhat.com>,
	Paolo Bonzini <pbonzini@redhat.com>,
	qemu devel <qemu-devel@nongnu.org>,
	Cornelia Huck <cornelia.huck@de.ibm.com>
Subject: Re: [Qemu-devel] issues of region cache and iommu reset
Date: Thu, 30 Mar 2017 00:38:16 +0300	[thread overview]
Message-ID: <20170330003316-mutt-send-email-mst@kernel.org> (raw)
In-Reply-To: <20170329081630.GF3152@pxdev.xzpeter.org>

On Wed, Mar 29, 2017 at 04:16:30PM +0800, Peter Xu wrote:
> On Wed, Mar 29, 2017 at 04:00:37PM +0800, Jason Wang wrote:
> > Hi:
> > 
> > I meet an issue when doing reboot for a guest with two virtio-net-pci cards
> > when iommu is enabled. What happens is:
> > 
> > 
> > 1) vtd was reset first
> 
> I'll add an extra question:
> 
> Currently with Q35 and VT-d, our system qtree looks like (simplified
> version):
> 
> bus: system bus
>   dev: intel-iommu
>   dev: q35-pcihost
>     bus: pcie.0
>       dev: pci device 1
>       dev: pci device 2
>       ...
> 
> Not sure whether it'll be clearer to switch to:
> 
> bus: system bus
>   dev: q35-pcihost
>     bus: iommu-scope 0
>       dev: intel-iommu
>         bus: pcie.0
>           dev: pci device 1
>           dev: pci device 2
>           ...
> 
> since logically the IOMMU device should be part of q35 pci-host?
> 
> Further, if we'll have more intel-iommus in the future in a single
> guest, we'll be able to have devices dangle under the specific iommu
> that it belongs. That'll be nice imho.
> 
> (Btw, I see that we are using qbus-qdev-qbus-... iterations to
>  describe the system tree. Why cannot we have qdev under another qdev?
>  Sorry if the question is stupid... Any pointers on explanations to
>  these qdev logics would be appreciated as well.)

Especially when you have many iommus, it's quite common to have iommus
be guest programmable with the bus range that they handle.

They are integrated devices, yes, but they simply aren't buses in the
real world.  I am guessing what happens on bare metal is that host
bridge resets the bus first and then resets itself
including the IOMMU.

> > 
> > 2) during the reset of virtio-net-pci #1, deletion of msix subregion will
> > cause a commit of all memory listeners
> > 
> > 3) virito-net-pci #2's region cache will be update, but since vtd has
> > already been reset, it can't get a valid mappings here
> > 
> > 
> > Any ideas on how to fix this? Need region cache be aware of IOMMU/IOTLB
> > state in this case? Or can we simply reset IOMMU as the last one?
> > 
> > Thanks
> > 
> 
> -- peterx