From mboxrd@z Thu Jan 1 00:00:00 1970 From: Benjamin Herrenschmidt Subject: Re: kvm PCI assignment & VFIO ramblings Date: Wed, 24 Aug 2011 09:35:37 +1000 Message-ID: <1314142537.30478.59.camel@pasglop> References: <20110802082848.GD29719@yookeroo.fritz.box> <1312308847.2653.467.camel@bling.home> <1312310121.2653.470.camel@bling.home> <20110803020422.GF29719@yookeroo.fritz.box> <4E3F9E33.5000706@redhat.com> <1312932258.4524.55.camel@bling.home> <1312944513.29273.28.camel@pasglop> <1313859105.6866.192.camel@x201.home> <20110822172508.GJ2079@amd.com> <1314047033.7662.39.camel@pasglop> <20110823131819.GO2079@amd.com> Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit Cc: Alexey Kardashevskiy , "kvm@vger.kernel.org" , Paul Mackerras , "linux-pci@vger.kernel.org" , qemu-devel , iommu , chrisw , Alex Williamson , Avi Kivity , linuxppc-dev , "benve@cisco.com" To: "Roedel, Joerg" Return-path: In-Reply-To: <20110823131819.GO2079@amd.com> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+gceq-qemu-devel=gmane.org@nongnu.org Sender: qemu-devel-bounces+gceq-qemu-devel=gmane.org@nongnu.org List-Id: kvm.vger.kernel.org On Tue, 2011-08-23 at 15:18 +0200, Roedel, Joerg wrote: > On Mon, Aug 22, 2011 at 05:03:53PM -0400, Benjamin Herrenschmidt wrote: > > > > > I am in favour of /dev/vfio/$GROUP. If multiple devices should be > > > assigned to a guest, there can also be an ioctl to bind a group to an > > > address-space of another group (certainly needs some care to not allow > > > that both groups belong to different processes). > > > > > > Btw, a problem we havn't talked about yet entirely is > > > driver-deassignment. User space can decide to de-assign the device from > > > vfio while a fd is open on it. With PCI there is no way to let this fail > > > (the .release function returns void last time i checked). Is this a > > > problem, and yes, how we handle that? > > > > We can treat it as a hard unplug (like a cardbus gone away). > > > > IE. Dispose of the direct mappings (switch to MMIO emulation) and return > > all ff's from reads (& ignore writes). > > > > Then send an unplug event via whatever mechanism the platform provides > > (ACPI hotplug controller on x86 for example, we haven't quite sorted out > > what to do on power for hotplug yet). > > Hmm, good idea. But as far as I know the hotplug-event needs to be in > the guest _before_ the device is actually unplugged (so that the guest > can unbind its driver first). That somehow brings back the sleep-idea > and the timeout in the .release function. That's for normal assisted hotplug, but don't we support hard hotplug ? I mean, things like cardbus, thunderbolt (if we ever support that) etc... will need it and some platforms do support hard hotplug of PCIe devices. (That's why drivers should never spin on MMIO waiting for a 1 bit to clear without a timeout :-) Cheers, Ben. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate.crashing.org (gate.crashing.org [63.228.1.57]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by ozlabs.org (Postfix) with ESMTPS id 1BD35B6F69 for ; Wed, 24 Aug 2011 09:36:33 +1000 (EST) Subject: Re: kvm PCI assignment & VFIO ramblings From: Benjamin Herrenschmidt To: "Roedel, Joerg" In-Reply-To: <20110823131819.GO2079@amd.com> References: <20110802082848.GD29719@yookeroo.fritz.box> <1312308847.2653.467.camel@bling.home> <1312310121.2653.470.camel@bling.home> <20110803020422.GF29719@yookeroo.fritz.box> <4E3F9E33.5000706@redhat.com> <1312932258.4524.55.camel@bling.home> <1312944513.29273.28.camel@pasglop> <1313859105.6866.192.camel@x201.home> <20110822172508.GJ2079@amd.com> <1314047033.7662.39.camel@pasglop> <20110823131819.GO2079@amd.com> Content-Type: text/plain; charset="UTF-8" Date: Wed, 24 Aug 2011 09:35:37 +1000 Message-ID: <1314142537.30478.59.camel@pasglop> Mime-Version: 1.0 Cc: Alexey Kardashevskiy , "kvm@vger.kernel.org" , Paul Mackerras , "linux-pci@vger.kernel.org" , qemu-devel , iommu , chrisw , Alex Williamson , Avi Kivity , Anthony Liguori , linuxppc-dev , "benve@cisco.com" List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , On Tue, 2011-08-23 at 15:18 +0200, Roedel, Joerg wrote: > On Mon, Aug 22, 2011 at 05:03:53PM -0400, Benjamin Herrenschmidt wrote: > > > > > I am in favour of /dev/vfio/$GROUP. If multiple devices should be > > > assigned to a guest, there can also be an ioctl to bind a group to an > > > address-space of another group (certainly needs some care to not allow > > > that both groups belong to different processes). > > > > > > Btw, a problem we havn't talked about yet entirely is > > > driver-deassignment. User space can decide to de-assign the device from > > > vfio while a fd is open on it. With PCI there is no way to let this fail > > > (the .release function returns void last time i checked). Is this a > > > problem, and yes, how we handle that? > > > > We can treat it as a hard unplug (like a cardbus gone away). > > > > IE. Dispose of the direct mappings (switch to MMIO emulation) and return > > all ff's from reads (& ignore writes). > > > > Then send an unplug event via whatever mechanism the platform provides > > (ACPI hotplug controller on x86 for example, we haven't quite sorted out > > what to do on power for hotplug yet). > > Hmm, good idea. But as far as I know the hotplug-event needs to be in > the guest _before_ the device is actually unplugged (so that the guest > can unbind its driver first). That somehow brings back the sleep-idea > and the timeout in the .release function. That's for normal assisted hotplug, but don't we support hard hotplug ? I mean, things like cardbus, thunderbolt (if we ever support that) etc... will need it and some platforms do support hard hotplug of PCIe devices. (That's why drivers should never spin on MMIO waiting for a 1 bit to clear without a timeout :-) Cheers, Ben. From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([140.186.70.92]:43614) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Qw0WN-0004zr-R4 for qemu-devel@nongnu.org; Tue, 23 Aug 2011 19:36:40 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Qw0WM-0005fg-PJ for qemu-devel@nongnu.org; Tue, 23 Aug 2011 19:36:39 -0400 Received: from gate.crashing.org ([63.228.1.57]:48071) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Qw0WM-0005fL-GK for qemu-devel@nongnu.org; Tue, 23 Aug 2011 19:36:38 -0400 From: Benjamin Herrenschmidt In-Reply-To: <20110823131819.GO2079@amd.com> References: <20110802082848.GD29719@yookeroo.fritz.box> <1312308847.2653.467.camel@bling.home> <1312310121.2653.470.camel@bling.home> <20110803020422.GF29719@yookeroo.fritz.box> <4E3F9E33.5000706@redhat.com> <1312932258.4524.55.camel@bling.home> <1312944513.29273.28.camel@pasglop> <1313859105.6866.192.camel@x201.home> <20110822172508.GJ2079@amd.com> <1314047033.7662.39.camel@pasglop> <20110823131819.GO2079@amd.com> Content-Type: text/plain; charset="UTF-8" Date: Wed, 24 Aug 2011 09:35:37 +1000 Message-ID: <1314142537.30478.59.camel@pasglop> Mime-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] kvm PCI assignment & VFIO ramblings List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: "Roedel, Joerg" Cc: Alexey Kardashevskiy , "kvm@vger.kernel.org" , Paul Mackerras , "linux-pci@vger.kernel.org" , qemu-devel , iommu , chrisw , Alex Williamson , Avi Kivity , linuxppc-dev , "benve@cisco.com" On Tue, 2011-08-23 at 15:18 +0200, Roedel, Joerg wrote: > On Mon, Aug 22, 2011 at 05:03:53PM -0400, Benjamin Herrenschmidt wrote: > > > > > I am in favour of /dev/vfio/$GROUP. If multiple devices should be > > > assigned to a guest, there can also be an ioctl to bind a group to an > > > address-space of another group (certainly needs some care to not allow > > > that both groups belong to different processes). > > > > > > Btw, a problem we havn't talked about yet entirely is > > > driver-deassignment. User space can decide to de-assign the device from > > > vfio while a fd is open on it. With PCI there is no way to let this fail > > > (the .release function returns void last time i checked). Is this a > > > problem, and yes, how we handle that? > > > > We can treat it as a hard unplug (like a cardbus gone away). > > > > IE. Dispose of the direct mappings (switch to MMIO emulation) and return > > all ff's from reads (& ignore writes). > > > > Then send an unplug event via whatever mechanism the platform provides > > (ACPI hotplug controller on x86 for example, we haven't quite sorted out > > what to do on power for hotplug yet). > > Hmm, good idea. But as far as I know the hotplug-event needs to be in > the guest _before_ the device is actually unplugged (so that the guest > can unbind its driver first). That somehow brings back the sleep-idea > and the timeout in the .release function. That's for normal assisted hotplug, but don't we support hard hotplug ? I mean, things like cardbus, thunderbolt (if we ever support that) etc... will need it and some platforms do support hard hotplug of PCIe devices. (That's why drivers should never spin on MMIO waiting for a 1 bit to clear without a timeout :-) Cheers, Ben.