From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Tian, Kevin" Subject: RE: [RFC PATCH v4 0/3] Add Mediated device support[was: Add vGPU support] Date: Fri, 27 May 2016 11:02:46 +0000 Message-ID: References: <1464119897-10844-1-git-send-email-kwankhede@nvidia.com> <20160525074356.52121ab8@ul30vt.home> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 8BIT Cc: Kirti Wankhede , "pbonzini@redhat.com" , "kraxel@redhat.com" , "cjia@nvidia.com" , "qemu-devel@nongnu.org" , "kvm@vger.kernel.org" , "Ruan, Shuai" , "Song, Jike" , "Lv, Zhiyuan" , "bjsdjshi@linux.vnet.ibm.com" To: Alex Williamson Return-path: Received: from mga11.intel.com ([192.55.52.93]:61124 "EHLO mga11.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750828AbcE0LCu convert rfc822-to-8bit (ORCPT ); Fri, 27 May 2016 07:02:50 -0400 In-Reply-To: <20160525074356.52121ab8@ul30vt.home> Content-Language: en-US Sender: kvm-owner@vger.kernel.org List-ID: > From: Alex Williamson [mailto:alex.williamson@redhat.com] > Sent: Wednesday, May 25, 2016 9:44 PM > > On Wed, 25 May 2016 07:13:58 +0000 > "Tian, Kevin" wrote: > > > > From: Kirti Wankhede [mailto:kwankhede@nvidia.com] > > > Sent: Wednesday, May 25, 2016 3:58 AM > > > > > > This series adds Mediated device support to v4.6 Linux host kernel. Purpose > > > of this series is to provide a common interface for mediated device > > > management that can be used by different devices. This series introduces > > > Mdev core module that create and manage mediated devices, VFIO based driver > > > for mediated PCI devices that are created by Mdev core module and update > > > VFIO type1 IOMMU module to support mediated devices. > > > > Thanks. "Mediated device" is more generic than previous one. :-) > > > > > > > > What's new in v4? > > > - Renamed 'vgpu' module to 'mdev' module that represent generic term > > > 'Mediated device'. > > > - Moved mdev directory to drivers/vfio directory as this is the extension > > > of VFIO APIs for mediated devices. > > > - Updated mdev driver to be flexible to register multiple types of drivers > > > to mdev_bus_type bus. > > > - Updated mdev core driver with mdev_put_device() and mdev_get_device() for > > > mediated devices. > > > > > > > > > > Just curious. In this version you move the whole mdev core under > > VFIO now. Sorry if I missed any agreement on this change. IIRC Alex > > doesn't want VFIO to manage mdev life-cycle directly. Instead VFIO is > > just a mdev driver on created mediated devices.... > > I did originally suggest keeping them separate, but as we've progressed > through the implementation, it's become more clear that the mediated > device interface is very much tied to the vfio interface, acting mostly > as a passthrough. So I thought it made sense to pull them together. > Still open to discussion of course. Thanks, > The main benefit of maintaining a separate mdev framework, IMHO, is to allow better support of both KVM and Xen. Xen doesn't work with VFIO today, because other VM's memory is not allocated from Dom0 which means VFIO within Dom0 doesn't has view/permission to control isolation for other VMs. However, after some thinking I think it might not be a big problem to combine VFIO/mdev together, if we extend Xen to just use VFIO for resource enumeration. In such model, VFIO still behaves as a single kernel portal to enumerate mediated devices to user space, but give up permission control to Qemu which will request a secure agent - Xen hypervisor - to ensure isolation of VM usage on mediated device (including EPT/IOMMU configuration). I'm not sure whether VFIO can support this usage today. It is somehow similar to channel io passthru in s390, where we also rely on Qemu to mediate ccw commands to ensure isolation. Maybe just some slight extension is required (e.g. not assume some API must be invoked). Of course Qemu side vfio code also need some change. If this can work, at least we can first put it as the enumeration interface for mediated device in Xen. In the future it may be extended to cover normal Xen PCI assignment as well instead of using sysfs to read PCI resource today. If above works, then we have a sound plan to enable mediated devices based on VFIO first for KVM, and then extend to Xen with reasonable effort. How do you think about it? Thanks Kevin From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:44075) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1b6FXb-0003ur-Dm for qemu-devel@nongnu.org; Fri, 27 May 2016 07:02:59 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1b6FXW-0004FA-DN for qemu-devel@nongnu.org; Fri, 27 May 2016 07:02:54 -0400 Received: from mga01.intel.com ([192.55.52.88]:57608) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1b6FXW-0004EE-2v for qemu-devel@nongnu.org; Fri, 27 May 2016 07:02:50 -0400 From: "Tian, Kevin" Date: Fri, 27 May 2016 11:02:46 +0000 Message-ID: References: <1464119897-10844-1-git-send-email-kwankhede@nvidia.com> <20160525074356.52121ab8@ul30vt.home> In-Reply-To: <20160525074356.52121ab8@ul30vt.home> Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [Qemu-devel] [RFC PATCH v4 0/3] Add Mediated device support[was: Add vGPU support] List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Alex Williamson Cc: Kirti Wankhede , "pbonzini@redhat.com" , "kraxel@redhat.com" , "cjia@nvidia.com" , "qemu-devel@nongnu.org" , "kvm@vger.kernel.org" , "Ruan, Shuai" , "Song, Jike" , "Lv, Zhiyuan" , "bjsdjshi@linux.vnet.ibm.com" > From: Alex Williamson [mailto:alex.williamson@redhat.com] > Sent: Wednesday, May 25, 2016 9:44 PM >=20 > On Wed, 25 May 2016 07:13:58 +0000 > "Tian, Kevin" wrote: >=20 > > > From: Kirti Wankhede [mailto:kwankhede@nvidia.com] > > > Sent: Wednesday, May 25, 2016 3:58 AM > > > > > > This series adds Mediated device support to v4.6 Linux host kernel. P= urpose > > > of this series is to provide a common interface for mediated device > > > management that can be used by different devices. This series introdu= ces > > > Mdev core module that create and manage mediated devices, VFIO based = driver > > > for mediated PCI devices that are created by Mdev core module and upd= ate > > > VFIO type1 IOMMU module to support mediated devices. > > > > Thanks. "Mediated device" is more generic than previous one. :-) > > > > > > > > What's new in v4? > > > - Renamed 'vgpu' module to 'mdev' module that represent generic term > > > 'Mediated device'. > > > - Moved mdev directory to drivers/vfio directory as this is the exten= sion > > > of VFIO APIs for mediated devices. > > > - Updated mdev driver to be flexible to register multiple types of dr= ivers > > > to mdev_bus_type bus. > > > - Updated mdev core driver with mdev_put_device() and mdev_get_device= () for > > > mediated devices. > > > > > > > > > > Just curious. In this version you move the whole mdev core under > > VFIO now. Sorry if I missed any agreement on this change. IIRC Alex > > doesn't want VFIO to manage mdev life-cycle directly. Instead VFIO is > > just a mdev driver on created mediated devices.... >=20 > I did originally suggest keeping them separate, but as we've progressed > through the implementation, it's become more clear that the mediated > device interface is very much tied to the vfio interface, acting mostly > as a passthrough. So I thought it made sense to pull them together. > Still open to discussion of course. Thanks, >=20 The main benefit of maintaining a separate mdev framework, IMHO, is to allow better support of both KVM and Xen. Xen doesn't work with VFIO today, because other VM's memory is not allocated from Dom0 which means VFIO within Dom0 doesn't has view/permission to control isolation=20 for other VMs. However, after some thinking I think it might not be a big problem to combine VFIO/mdev together, if we extend Xen to just use VFIO for resource enumeration. In such model, VFIO still behaves as a single=20 kernel portal to enumerate mediated devices to user space, but give up=20 permission control to Qemu which will request a secure agent - Xen hypervisor - to ensure isolation of VM usage on mediated device (including EPT/IOMMU configuration). I'm not sure whether VFIO can support this usage today. It is somehow=20 similar to channel io passthru in s390, where we also rely on Qemu to=20 mediate ccw commands to ensure isolation. Maybe just some slight=20 extension is required (e.g. not assume some API must be invoked). Of=20 course Qemu side vfio code also need some change. If this can work,=20 at least we can first put it as the enumeration interface for mediated=20 device in Xen. In the future it may be extended to cover normal Xen=20 PCI assignment as well instead of using sysfs to read PCI resource today. If above works, then we have a sound plan to enable mediated devices=20 based on VFIO first for KVM, and then extend to Xen with reasonable=20 effort. =20 How do you think about it? Thanks Kevin