From mboxrd@z Thu Jan 1 00:00:00 1970 From: Kirti Wankhede Subject: Re: VFIO based vGPU(was Re: [Announcement] 2015-Q3 release of XenGT - a Mediated ...) Date: Wed, 27 Jan 2016 13:36:43 +0530 Message-ID: <56A87A93.3000105@nvidia.com> References: <569C5071.6080004@intel.com> <1453092476.32741.67.camel@redhat.com> <569CA8AD.6070200@intel.com> <1453143919.32741.169.camel@redhat.com> <569F4C86.2070501@intel.com> <56A6083E.10703@intel.com> <1453757426.32741.614.camel@redhat.com> <20160126102003.GA14400@nvidia.com> <1453838773.15515.1.camel@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: QUOTED-PRINTABLE Cc: "Song, Jike" , Gerd Hoffmann , Paolo Bonzini , "Lv, Zhiyuan" , "Ruan, Shuai" , "kvm@vger.kernel.org" , qemu-devel , "igvt-g@lists.01.org" To: Alex Williamson , Neo Jia , "Tian, Kevin" Return-path: Received: from hqemgate16.nvidia.com ([216.228.121.65]:14978 "EHLO hqemgate16.nvidia.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751269AbcA0IGu convert rfc822-to-8bit (ORCPT ); Wed, 27 Jan 2016 03:06:50 -0500 In-Reply-To: <1453838773.15515.1.camel@redhat.com> Sender: kvm-owner@vger.kernel.org List-ID: On 1/27/2016 1:36 AM, Alex Williamson wrote: > On Tue, 2016-01-26 at 02:20 -0800, Neo Jia wrote: >> On Mon, Jan 25, 2016 at 09:45:14PM +0000, Tian, Kevin wrote: >>>> From: Alex Williamson [mailto:alex.williamson@redhat.com] >> =20 >> Hi Alex, Kevin and Jike, >> =20 >> (Seems I shouldn't use attachment, resend it again to the list, patc= hes are >> inline at the end) >> =20 >> Thanks for adding me to this technical discussion, a great opportuni= ty >> for us to design together which can bring both Intel and NVIDIA vGPU= solution to >> KVM platform. >> =20 >> Instead of directly jumping to the proposal that we have been workin= g on >> recently for NVIDIA vGPU on KVM, I think it is better for me to put = out couple >> quick comments / thoughts regarding the existing discussions on this= thread as >> fundamentally I think we are solving the same problem, DMA, interrup= t and MMIO. >> =20 >> Then we can look at what we have, hopefully we can reach some consen= sus soon. >> =20 >>> Yes, and since you're creating and destroying the vgpu here, this i= s >>> where I'd expect a struct device to be created and added to an IOMM= U >>> group. The lifecycle management should really include links betwee= n >>> the vGPU and physical GPU, which would be much, much easier to do w= ith >>> struct devices create here rather than at the point where we start >>> doing vfio "stuff". >> =20 >> Infact to keep vfio-vgpu to be more generic, vgpu device creation an= d management >> can be centralized and done in vfio-vgpu. That also include adding t= o IOMMU >> group and VFIO group. > Is this really a good idea? The concept of a vgpu is not unique to > vfio, we want vfio to be a driver for a vgpu, not an integral part of > the lifecycle of a vgpu. That certainly doesn't exclude adding > infrastructure to make lifecycle management of a vgpu more consistent > between drivers, but it should be done independently of vfio. I'll g= o > back to the SR-IOV model, vfio is often used with SR-IOV VFs, but vfi= o > does not create the VF, that's done in coordination with the PF makin= g > use of some PCI infrastructure for consistency between drivers. > > It seems like we need to take more advantage of the class and driver > core support to perhaps setup a vgpu bus and class with vfio-vgpu jus= t > being a driver for those devices. =46or device passthrough or SR-IOV model, PCI devices are created by PC= I=20 bus driver and from the probe routine each device is added in vfio grou= p. =46or vgpu, there should be a common module that create vgpu device, sa= y=20 vgpu module, add vgpu device to an IOMMU group and then add it to vfio=20 group. This module can handle management of vgpus. Advantage of keepin= g=20 this module a separate module than doing device creation in vendor=20 modules is to have generic interface for vgpu management, for example,=20 files /sys/class/vgpu/vgpu_start and /sys/class/vgpu/vgpu_shudown and=20 vgpu driver registration interface. In the patch, vgpu_dev.c + vgpu_sysfs.c form such vgpu module and=20 vgpu_vfio.c is for VFIO interface. Each vgpu device should be added to=20 vfio group, so vgpu_group_init() from vgpu_vfio.c should be called per=20 device. In the vgpu module, vgpu devices are created on request, so=20 vgpu_group_init() should be called explicitly for per vgpu device.=20 That=E2=80=99s why had merged the 2 modules, vgpu + vgpu_vfio to form= one vgpu=20 module. Vgpu_vfio would remain separate entity but merged with vgpu=20 module. Thanks, Kirti From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:60047) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aOLRQ-0002IL-S2 for qemu-devel@nongnu.org; Wed, 27 Jan 2016 03:27:05 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1aOLRL-0005O0-Rr for qemu-devel@nongnu.org; Wed, 27 Jan 2016 03:27:04 -0500 Received: from hqemgate16.nvidia.com ([216.228.121.65]:18029) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aOLRL-0005Ni-JQ for qemu-devel@nongnu.org; Wed, 27 Jan 2016 03:26:59 -0500 References: <569C5071.6080004@intel.com> <1453092476.32741.67.camel@redhat.com> <569CA8AD.6070200@intel.com> <1453143919.32741.169.camel@redhat.com> <569F4C86.2070501@intel.com> <56A6083E.10703@intel.com> <1453757426.32741.614.camel@redhat.com> <20160126102003.GA14400@nvidia.com> <1453838773.15515.1.camel@redhat.com> From: Kirti Wankhede Message-ID: <56A87A93.3000105@nvidia.com> Date: Wed, 27 Jan 2016 13:36:43 +0530 MIME-Version: 1.0 In-Reply-To: <1453838773.15515.1.camel@redhat.com> Content-Type: text/plain; charset="utf-8"; format=flowed Content-Transfer-Encoding: quoted-printable Subject: Re: [Qemu-devel] VFIO based vGPU(was Re: [Announcement] 2015-Q3 release of XenGT - a Mediated ...) List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Alex Williamson , Neo Jia , "Tian, Kevin" Cc: "Ruan, Shuai" , "Song, Jike" , "kvm@vger.kernel.org" , "igvt-g@lists.01.org" , qemu-devel , Gerd Hoffmann , Paolo Bonzini , "Lv, Zhiyuan" On 1/27/2016 1:36 AM, Alex Williamson wrote: > On Tue, 2016-01-26 at 02:20 -0800, Neo Jia wrote: >> On Mon, Jan 25, 2016 at 09:45:14PM +0000, Tian, Kevin wrote: >>>> From: Alex Williamson [mailto:alex.williamson@redhat.com] >> =20 >> Hi Alex, Kevin and Jike, >> =20 >> (Seems I shouldn't use attachment, resend it again to the list, patches = are >> inline at the end) >> =20 >> Thanks for adding me to this technical discussion, a great opportunity >> for us to design together which can bring both Intel and NVIDIA vGPU sol= ution to >> KVM platform. >> =20 >> Instead of directly jumping to the proposal that we have been working on >> recently for NVIDIA vGPU on KVM, I think it is better for me to put out = couple >> quick comments / thoughts regarding the existing discussions on this thr= ead as >> fundamentally I think we are solving the same problem, DMA, interrupt an= d MMIO. >> =20 >> Then we can look at what we have, hopefully we can reach some consensus = soon. >> =20 >>> Yes, and since you're creating and destroying the vgpu here, this is >>> where I'd expect a struct device to be created and added to an IOMMU >>> group. The lifecycle management should really include links between >>> the vGPU and physical GPU, which would be much, much easier to do with >>> struct devices create here rather than at the point where we start >>> doing vfio "stuff". >> =20 >> Infact to keep vfio-vgpu to be more generic, vgpu device creation and ma= nagement >> can be centralized and done in vfio-vgpu. That also include adding to IO= MMU >> group and VFIO group. > Is this really a good idea? The concept of a vgpu is not unique to > vfio, we want vfio to be a driver for a vgpu, not an integral part of > the lifecycle of a vgpu. That certainly doesn't exclude adding > infrastructure to make lifecycle management of a vgpu more consistent > between drivers, but it should be done independently of vfio. I'll go > back to the SR-IOV model, vfio is often used with SR-IOV VFs, but vfio > does not create the VF, that's done in coordination with the PF making > use of some PCI infrastructure for consistency between drivers. > > It seems like we need to take more advantage of the class and driver > core support to perhaps setup a vgpu bus and class with vfio-vgpu just > being a driver for those devices. For device passthrough or SR-IOV model, PCI devices are created by PCI=20 bus driver and from the probe routine each device is added in vfio group. For vgpu, there should be a common module that create vgpu device, say=20 vgpu module, add vgpu device to an IOMMU group and then add it to vfio=20 group. This module can handle management of vgpus. Advantage of keeping=20 this module a separate module than doing device creation in vendor=20 modules is to have generic interface for vgpu management, for example,=20 files /sys/class/vgpu/vgpu_start and /sys/class/vgpu/vgpu_shudown and=20 vgpu driver registration interface. In the patch, vgpu_dev.c + vgpu_sysfs.c form such vgpu module and=20 vgpu_vfio.c is for VFIO interface. Each vgpu device should be added to=20 vfio group, so vgpu_group_init() from vgpu_vfio.c should be called per=20 device. In the vgpu module, vgpu devices are created on request, so=20 vgpu_group_init() should be called explicitly for per vgpu device.=20 That=E2=80=99s why had merged the 2 modules, vgpu + vgpu_vfio to form one= vgpu=20 module. Vgpu_vfio would remain separate entity but merged with vgpu=20 module. Thanks, Kirti