From mboxrd@z Thu Jan 1 00:00:00 1970 From: Alex Williamson Subject: Re: [RFC PATCH kernel 0/5] powerpc/P9/vfio: Pass through NVIDIA Tesla V100 Date: Thu, 7 Jun 2018 16:15:41 -0600 Message-ID: <20180607161541.21df6434@w520.home> References: <20180607084420.29513-1-aik@ozlabs.ru> <20180607110409.5057ebac@w520.home> Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Cc: kvm@vger.kernel.org, Alexey Kardashevskiy , Alistair Popple , Ram Pai , kvm-ppc@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, David Gibson To: Benjamin Herrenschmidt Return-path: In-Reply-To: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: linuxppc-dev-bounces+glppe-linuxppc-embedded-2=m.gmane.org@lists.ozlabs.org Sender: "Linuxppc-dev" List-Id: kvm.vger.kernel.org On Fri, 08 Jun 2018 07:54:02 +1000 Benjamin Herrenschmidt wrote: > On Thu, 2018-06-07 at 11:04 -0600, Alex Williamson wrote: > > > > Can we back up and discuss whether the IOMMU grouping of NVLink > > connected devices makes sense? AIUI we have a PCI view of these > > devices and from that perspective they're isolated. That's the view of > > the device used to generate the grouping. However, not visible to us, > > these devices are interconnected via NVLink. What isolation properties > > does NVLink provide given that its entire purpose for existing seems to > > be to provide a high performance link for p2p between devices? > > Not entire. On POWER chips, we also have an nvlink between the device > and the CPU which is running significantly faster than PCIe. > > But yes, there are cross-links and those should probably be accounted > for in the grouping. Then after we fix the grouping, can we just let the host driver manage this coherent memory range and expose vGPUs to guests? The use case of assigning all 6 GPUs to one VM seems pretty limited. (Might need to convince NVIDIA to support more than a single vGPU per VM though) Thanks, Alex From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mx1.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4120H86d1tzF31D for ; Fri, 8 Jun 2018 08:15:47 +1000 (AEST) Date: Thu, 7 Jun 2018 16:15:41 -0600 From: Alex Williamson To: Benjamin Herrenschmidt Cc: Alexey Kardashevskiy , linuxppc-dev@lists.ozlabs.org, David Gibson , kvm-ppc@vger.kernel.org, Ram Pai , kvm@vger.kernel.org, Alistair Popple Subject: Re: [RFC PATCH kernel 0/5] powerpc/P9/vfio: Pass through NVIDIA Tesla V100 Message-ID: <20180607161541.21df6434@w520.home> In-Reply-To: References: <20180607084420.29513-1-aik@ozlabs.ru> <20180607110409.5057ebac@w520.home> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , On Fri, 08 Jun 2018 07:54:02 +1000 Benjamin Herrenschmidt wrote: > On Thu, 2018-06-07 at 11:04 -0600, Alex Williamson wrote: > > > > Can we back up and discuss whether the IOMMU grouping of NVLink > > connected devices makes sense? AIUI we have a PCI view of these > > devices and from that perspective they're isolated. That's the view of > > the device used to generate the grouping. However, not visible to us, > > these devices are interconnected via NVLink. What isolation properties > > does NVLink provide given that its entire purpose for existing seems to > > be to provide a high performance link for p2p between devices? > > Not entire. On POWER chips, we also have an nvlink between the device > and the CPU which is running significantly faster than PCIe. > > But yes, there are cross-links and those should probably be accounted > for in the grouping. Then after we fix the grouping, can we just let the host driver manage this coherent memory range and expose vGPUs to guests? The use case of assigning all 6 GPUs to one VM seems pretty limited. (Might need to convince NVIDIA to support more than a single vGPU per VM though) Thanks, Alex From mboxrd@z Thu Jan 1 00:00:00 1970 From: Alex Williamson Date: Thu, 07 Jun 2018 22:15:41 +0000 Subject: Re: [RFC PATCH kernel 0/5] powerpc/P9/vfio: Pass through NVIDIA Tesla V100 Message-Id: <20180607161541.21df6434@w520.home> List-Id: References: <20180607084420.29513-1-aik@ozlabs.ru> <20180607110409.5057ebac@w520.home> In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit To: Benjamin Herrenschmidt Cc: kvm@vger.kernel.org, Alexey Kardashevskiy , Alistair Popple , Ram Pai , kvm-ppc@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, David Gibson On Fri, 08 Jun 2018 07:54:02 +1000 Benjamin Herrenschmidt wrote: > On Thu, 2018-06-07 at 11:04 -0600, Alex Williamson wrote: > > > > Can we back up and discuss whether the IOMMU grouping of NVLink > > connected devices makes sense? AIUI we have a PCI view of these > > devices and from that perspective they're isolated. That's the view of > > the device used to generate the grouping. However, not visible to us, > > these devices are interconnected via NVLink. What isolation properties > > does NVLink provide given that its entire purpose for existing seems to > > be to provide a high performance link for p2p between devices? > > Not entire. On POWER chips, we also have an nvlink between the device > and the CPU which is running significantly faster than PCIe. > > But yes, there are cross-links and those should probably be accounted > for in the grouping. Then after we fix the grouping, can we just let the host driver manage this coherent memory range and expose vGPUs to guests? The use case of assigning all 6 GPUs to one VM seems pretty limited. (Might need to convince NVIDIA to support more than a single vGPU per VM though) Thanks, Alex