From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:33188) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aQcLH-0007jB-0l for qemu-devel@nongnu.org; Tue, 02 Feb 2016 09:54:08 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1aQcLC-0005LS-Qa for qemu-devel@nongnu.org; Tue, 02 Feb 2016 09:54:06 -0500 Received: from mx1.redhat.com ([209.132.183.28]:51850) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aQcLC-0005LL-Io for qemu-devel@nongnu.org; Tue, 02 Feb 2016 09:54:02 -0500 Message-ID: <1454424840.10542.18.camel@redhat.com> From: Alex Williamson Date: Tue, 02 Feb 2016 07:54:00 -0700 In-Reply-To: <1454413825.4788.49.camel@infradead.org> References: <1451994098-6972-1-git-send-email-kraxel@redhat.com> <1454009759.7183.7.camel@redhat.com> <1454051359.28516.28.camel@redhat.com> <1454090373.23148.11.camel@redhat.com> <003AAFE53969E14CB1F09B6FD68C3CD47BB681B7@ORSMSX106.amr.corp.intel.com> <1454262132.10542.1.camel@redhat.com> <003AAFE53969E14CB1F09B6FD68C3CD47BB6B475@ORSMSX106.amr.corp.intel.com> <1454413825.4788.49.camel@infradead.org> Content-Type: text/plain; charset="UTF-8" Mime-Version: 1.0 Content-Transfer-Encoding: quoted-printable Subject: Re: [Qemu-devel] [Xen-devel] [iGVT-g] [vfio-users] [PATCH v3 00/11] igd passthrough chipset tweaks List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: David Woodhouse , "Tian, Kevin" , "Kay, Allen M" , Gerd Hoffmann Cc: "igvt-g@ml01.01.org" , "xen-devel@lists.xensource.com" , Eduardo Habkost , Stefano Stabellini , "qemu-devel@nongnu.org" , Cao jin , "vfio-users@redhat.com" On Tue, 2016-02-02 at 11:50 +0000, David Woodhouse wrote: > On Tue, 2016-02-02 at 06:42 +0000, Tian, Kevin wrote: > > > From: Kay, Allen M > > > Sent: Tuesday, February 02, 2016 8:04 AM > > > >=C2=A0 > > > > David notes in the latter commit above: > > > >=C2=A0 > > > > "We should be able to successfully assign graphics devices to gue= sts too, as > > > > long as the initial handling of stolen memory is reconfigured app= ropriately." > > > >=C2=A0 > > > > What code is supposed to be doing that reconfiguration when a dev= ice is > > > > assigned?=C2=A0=C2=A0Clearly we don't have it yet, making assignm= ent of these devices > > > > very unsafe.=C2=A0=C2=A0It seems like vfio or IOMMU code=C2=A0=C2= =A0in the kernel needs device > > > > specific code to clear these settings to make it safe for userspa= ce, then > > > > perhaps VM BIOS support to reallocate.=C2=A0=C2=A0Is there any co= nsistency across IGD > > > > revisions for doing this?=C2=A0=C2=A0Is there a spec? > > > > Thanks, >=C2=A0 > I haven't ever successfully assigned an IGD device to a VM myself, but > my understanding was that it *has* been done. So when the code was > changed to prevent assignment of devices afflicted by RMRRs (except USB > where we know it's OK), I just added the integrated graphics to that > same exception as USB, to preserve the status quo ante. It had been successfully done on /Xen/, not on anything that actually mad= e use of that exclusion, so there was no status quo to preserve. > > I don't think stolen memory should be handled explicitly. If yes, it = should be > > listed as a RMRR region so general RMRR setup will cover it. But as A= llen > > pointed out, the whole RMRR becomes unnecessary if we target only sec= ondary > > device for IGD. >=C2=A0 > Perhaps the best option is *not* to have special cases in the IOMMU > code for "those devices which can safely be assigned despite RMRRs". >=C2=A0 > Instead, let's let the device driver =E2=80=94 or whatever =E2=80=94 te= ll the IOMMU > code when it's *stopped* the firmware from (ab)using the device's DMA > facilities. >=C2=A0 > So when the USB code does the handoff thing to quiesce the firmware's > access to USB and take over in the OS, it would call the IOMMU function > to revoke the RMRR for the USB controller. >=C2=A0 > And if/when the graphics driver resets its device into a state where > it's no longer accessing stolen memory and can be assigned to a VM, it > can also call that 'RMRR revoke' function. >=C2=A0 > Likewise, if we teach device drivers to cancel whatever abominations > the HP firmware tends to set up behind the OS's back on other PCI > devices, we can cancel the RMRRs for those too. >=C2=A0 > Then the IOMMU code has a simple choice and no special cases =E2=80=94 = we can > assign a device iff it has no active RMRR. I first glance I like it, but there's a problem, it assumes there is a host driver for the device that will permanently release the device from the RMRR even after the device is unbound.=C2=A0=C2=A0Currently we don't = have a requirement that the user must first bind the device to a native host driver, unbind it, and only then is it eligible for device assignment. In fact with GPUs we often blacklist the native driver or attach them directly to a stub driver to avoid the host driver.=C2=A0=C2=A0Maybe that= issue works itself out since the IOMMU won't allow access to the device without this step, but it means that i915 needs to be better than most graphics drivers when it comes to unbinding the device (which is not a very high bar).=C2=A0=C2=A0Of course as I've shown on IGD, it's not simpl= y a matter of declaring the RMRR unused, some reconfiguration of the device is necessary such that the guest driver doesn't try to start using that same reserved range.=C2=A0=C2=A0Thanks, Alex From mboxrd@z Thu Jan 1 00:00:00 1970 From: Alex Williamson Subject: Re: [Xen-devel] [iGVT-g] [vfio-users] [PATCH v3 00/11] igd passthrough chipset tweaks Date: Tue, 02 Feb 2016 07:54:00 -0700 Message-ID: <1454424840.10542.18.camel@redhat.com> References: <1451994098-6972-1-git-send-email-kraxel@redhat.com> <1454009759.7183.7.camel@redhat.com> <1454051359.28516.28.camel@redhat.com> <1454090373.23148.11.camel@redhat.com> <003AAFE53969E14CB1F09B6FD68C3CD47BB681B7@ORSMSX106.amr.corp.intel.com> <1454262132.10542.1.camel@redhat.com> <003AAFE53969E14CB1F09B6FD68C3CD47BB6B475@ORSMSX106.amr.corp.intel.com> <1454413825.4788.49.camel@infradead.org> Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Return-path: In-Reply-To: <1454413825.4788.49.camel@infradead.org> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+gceq-qemu-devel=gmane.org@nongnu.org Sender: qemu-devel-bounces+gceq-qemu-devel=gmane.org@nongnu.org To: David Woodhouse , "Tian, Kevin" , "Kay, Allen M" , Gerd Hoffmann Cc: "igvt-g@ml01.01.org" , "xen-devel@lists.xensource.com" , Eduardo Habkost , Stefano Stabellini , "qemu-devel@nongnu.org" , Cao jin , "vfio-users@redhat.com" List-Id: xen-devel@lists.xenproject.org On Tue, 2016-02-02 at 11:50 +0000, David Woodhouse wrote: > On Tue, 2016-02-02 at 06:42 +0000, Tian, Kevin wrote: > > > From: Kay, Allen M > > > Sent: Tuesday, February 02, 2016 8:04 AM > > > >=C2=A0 > > > > David notes in the latter commit above: > > > >=C2=A0 > > > > "We should be able to successfully assign graphics devices to gue= sts too, as > > > > long as the initial handling of stolen memory is reconfigured app= ropriately." > > > >=C2=A0 > > > > What code is supposed to be doing that reconfiguration when a dev= ice is > > > > assigned?=C2=A0=C2=A0Clearly we don't have it yet, making assignm= ent of these devices > > > > very unsafe.=C2=A0=C2=A0It seems like vfio or IOMMU code=C2=A0=C2= =A0in the kernel needs device > > > > specific code to clear these settings to make it safe for userspa= ce, then > > > > perhaps VM BIOS support to reallocate.=C2=A0=C2=A0Is there any co= nsistency across IGD > > > > revisions for doing this?=C2=A0=C2=A0Is there a spec? > > > > Thanks, >=C2=A0 > I haven't ever successfully assigned an IGD device to a VM myself, but > my understanding was that it *has* been done. So when the code was > changed to prevent assignment of devices afflicted by RMRRs (except USB > where we know it's OK), I just added the integrated graphics to that > same exception as USB, to preserve the status quo ante. It had been successfully done on /Xen/, not on anything that actually mad= e use of that exclusion, so there was no status quo to preserve. > > I don't think stolen memory should be handled explicitly. If yes, it = should be > > listed as a RMRR region so general RMRR setup will cover it. But as A= llen > > pointed out, the whole RMRR becomes unnecessary if we target only sec= ondary > > device for IGD. >=C2=A0 > Perhaps the best option is *not* to have special cases in the IOMMU > code for "those devices which can safely be assigned despite RMRRs". >=C2=A0 > Instead, let's let the device driver =E2=80=94 or whatever =E2=80=94 te= ll the IOMMU > code when it's *stopped* the firmware from (ab)using the device's DMA > facilities. >=C2=A0 > So when the USB code does the handoff thing to quiesce the firmware's > access to USB and take over in the OS, it would call the IOMMU function > to revoke the RMRR for the USB controller. >=C2=A0 > And if/when the graphics driver resets its device into a state where > it's no longer accessing stolen memory and can be assigned to a VM, it > can also call that 'RMRR revoke' function. >=C2=A0 > Likewise, if we teach device drivers to cancel whatever abominations > the HP firmware tends to set up behind the OS's back on other PCI > devices, we can cancel the RMRRs for those too. >=C2=A0 > Then the IOMMU code has a simple choice and no special cases =E2=80=94 = we can > assign a device iff it has no active RMRR. I first glance I like it, but there's a problem, it assumes there is a host driver for the device that will permanently release the device from the RMRR even after the device is unbound.=C2=A0=C2=A0Currently we don't = have a requirement that the user must first bind the device to a native host driver, unbind it, and only then is it eligible for device assignment. In fact with GPUs we often blacklist the native driver or attach them directly to a stub driver to avoid the host driver.=C2=A0=C2=A0Maybe that= issue works itself out since the IOMMU won't allow access to the device without this step, but it means that i915 needs to be better than most graphics drivers when it comes to unbinding the device (which is not a very high bar).=C2=A0=C2=A0Of course as I've shown on IGD, it's not simpl= y a matter of declaring the RMRR unused, some reconfiguration of the device is necessary such that the guest driver doesn't try to start using that same reserved range.=C2=A0=C2=A0Thanks, Alex