From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mx1.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ml01.01.org (Postfix) with ESMTPS id CF81220348602 for ; Wed, 9 May 2018 10:50:00 -0700 (PDT) Date: Wed, 9 May 2018 13:49:53 -0400 From: Jerome Glisse Subject: Re: [PATCH v4 04/14] PCI/P2PDMA: Clear ACS P2P flags for all devices behind switches Message-ID: <20180509174952.GC4140@redhat.com> References: <20180508133407.57a46902@w520.home> <5fc9b1c1-9208-06cc-0ec5-1f54c2520494@deltatee.com> <20180508141331.7cd737cb@w520.home> <20180508205005.GC15608@redhat.com> <7FFB9603-DF9F-4441-82E9-46037CB6C0DE@raithlin.com> <1775CC56-4651-422F-953A-18E024D3717C@raithlin.com> <20180509160722.GB4140@redhat.com> <366A8132-B88A-40F7-BDE3-DA542E45FC0C@raithlin.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <366A8132-B88A-40F7-BDE3-DA542E45FC0C@raithlin.com> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Errors-To: linux-nvdimm-bounces@lists.01.org Sender: "Linux-nvdimm" To: Stephen Bates Cc: Jens Axboe , Keith Busch , "linux-nvdimm@lists.01.org" , "linux-rdma@vger.kernel.org" , "linux-pci@vger.kernel.org" , "linux-kernel@vger.kernel.org" , "linux-nvme@lists.infradead.org" , Christoph Hellwig , "linux-block@vger.kernel.org" , Alex Williamson , Jason Gunthorpe , Bjorn Helgaas , Benjamin Herrenschmidt , Bjorn Helgaas , Max Gurtovoy , Christian =?iso-8859-1?Q?K=F6nig?= List-ID: On Wed, May 09, 2018 at 04:30:32PM +0000, Stephen Bates wrote: > Hi Jerome > = > > Now inside that page table you can point GPU virtual address > > to use GPU memory or use system memory. Those system memory entry can > > also be mark as ATS against a given PASID. > = > Thanks. This all makes sense. = > = > But do you have examples of this in a kernel driver (if so can you point = me too it) or is this all done via user-space? Based on my grepping of the = kernel code I see zero EP drivers using in-kernel ATS functionality right n= ow... > = As it is tie to PASID this is done using IOMMU so looks for caller of amd_iommu_bind_pasid() or intel_svm_bind_mm() in GPU the existing user is the AMD GPU driver see: drivers/gpu/drm/amd/ drivers/gpu/drm/amd/amdkfd/ drivers/gpu/drm/amd/amdgpu/ Lot of codes there. The GPU code details do not really matter for this discussions thought. You do not need to do much to use PASID. Cheers, J=E9r=F4me _______________________________________________ Linux-nvdimm mailing list Linux-nvdimm@lists.01.org https://lists.01.org/mailman/listinfo/linux-nvdimm From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Date: Wed, 9 May 2018 13:49:53 -0400 From: Jerome Glisse To: Stephen Bates Cc: Christian =?iso-8859-1?Q?K=F6nig?= , Logan Gunthorpe , Alex Williamson , Bjorn Helgaas , "linux-kernel@vger.kernel.org" , "linux-pci@vger.kernel.org" , "linux-nvme@lists.infradead.org" , "linux-rdma@vger.kernel.org" , "linux-nvdimm@lists.01.org" , "linux-block@vger.kernel.org" , Christoph Hellwig , Jens Axboe , Keith Busch , Sagi Grimberg , Bjorn Helgaas , Jason Gunthorpe , Max Gurtovoy , Dan Williams , Benjamin Herrenschmidt Subject: Re: [PATCH v4 04/14] PCI/P2PDMA: Clear ACS P2P flags for all devices behind switches Message-ID: <20180509174952.GC4140@redhat.com> References: <20180508133407.57a46902@w520.home> <5fc9b1c1-9208-06cc-0ec5-1f54c2520494@deltatee.com> <20180508141331.7cd737cb@w520.home> <20180508205005.GC15608@redhat.com> <7FFB9603-DF9F-4441-82E9-46037CB6C0DE@raithlin.com> <1775CC56-4651-422F-953A-18E024D3717C@raithlin.com> <20180509160722.GB4140@redhat.com> <366A8132-B88A-40F7-BDE3-DA542E45FC0C@raithlin.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 In-Reply-To: <366A8132-B88A-40F7-BDE3-DA542E45FC0C@raithlin.com> List-ID: On Wed, May 09, 2018 at 04:30:32PM +0000, Stephen Bates wrote: > Hi Jerome > > > Now inside that page table you can point GPU virtual address > > to use GPU memory or use system memory. Those system memory entry can > > also be mark as ATS against a given PASID. > > Thanks. This all makes sense. > > But do you have examples of this in a kernel driver (if so can you point me too it) or is this all done via user-space? Based on my grepping of the kernel code I see zero EP drivers using in-kernel ATS functionality right now... > As it is tie to PASID this is done using IOMMU so looks for caller of amd_iommu_bind_pasid() or intel_svm_bind_mm() in GPU the existing user is the AMD GPU driver see: drivers/gpu/drm/amd/ drivers/gpu/drm/amd/amdkfd/ drivers/gpu/drm/amd/amdgpu/ Lot of codes there. The GPU code details do not really matter for this discussions thought. You do not need to do much to use PASID. Cheers, J�r�me From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jerome Glisse Subject: Re: [PATCH v4 04/14] PCI/P2PDMA: Clear ACS P2P flags for all devices behind switches Date: Wed, 9 May 2018 13:49:53 -0400 Message-ID: <20180509174952.GC4140@redhat.com> References: <20180508133407.57a46902@w520.home> <5fc9b1c1-9208-06cc-0ec5-1f54c2520494@deltatee.com> <20180508141331.7cd737cb@w520.home> <20180508205005.GC15608@redhat.com> <7FFB9603-DF9F-4441-82E9-46037CB6C0DE@raithlin.com> <1775CC56-4651-422F-953A-18E024D3717C@raithlin.com> <20180509160722.GB4140@redhat.com> <366A8132-B88A-40F7-BDE3-DA542E45FC0C@raithlin.com> Mime-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Return-path: Content-Disposition: inline In-Reply-To: <366A8132-B88A-40F7-BDE3-DA542E45FC0C-pv7U853sEMVWk0Htik3J/w@public.gmane.org> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: linux-nvdimm-bounces-hn68Rpc1hR1g9hUCZPvPmw@public.gmane.org Sender: "Linux-nvdimm" To: Stephen Bates Cc: Jens Axboe , Keith Busch , "linux-nvdimm-hn68Rpc1hR1g9hUCZPvPmw@public.gmane.org" , "linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , "linux-pci-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , "linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , "linux-nvme-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org" , Christoph Hellwig , "linux-block-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , Alex Williamson , Jason Gunthorpe , Bjorn Helgaas , Benjamin Herrenschmidt , Bjorn Helgaas , Max Gurtovoy , Christian =?iso-8859-1?Q?K=F6nig?= List-Id: linux-rdma@vger.kernel.org On Wed, May 09, 2018 at 04:30:32PM +0000, Stephen Bates wrote: > Hi Jerome > = > > Now inside that page table you can point GPU virtual address > > to use GPU memory or use system memory. Those system memory entry can > > also be mark as ATS against a given PASID. > = > Thanks. This all makes sense. = > = > But do you have examples of this in a kernel driver (if so can you point = me too it) or is this all done via user-space? Based on my grepping of the = kernel code I see zero EP drivers using in-kernel ATS functionality right n= ow... > = As it is tie to PASID this is done using IOMMU so looks for caller of amd_iommu_bind_pasid() or intel_svm_bind_mm() in GPU the existing user is the AMD GPU driver see: drivers/gpu/drm/amd/ drivers/gpu/drm/amd/amdkfd/ drivers/gpu/drm/amd/amdgpu/ Lot of codes there. The GPU code details do not really matter for this discussions thought. You do not need to do much to use PASID. Cheers, J=E9r=F4me From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S935502AbeEIRuD (ORCPT ); Wed, 9 May 2018 13:50:03 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:50730 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S933555AbeEIRuA (ORCPT ); Wed, 9 May 2018 13:50:00 -0400 Date: Wed, 9 May 2018 13:49:53 -0400 From: Jerome Glisse To: Stephen Bates Cc: Christian =?iso-8859-1?Q?K=F6nig?= , Logan Gunthorpe , Alex Williamson , Bjorn Helgaas , "linux-kernel@vger.kernel.org" , "linux-pci@vger.kernel.org" , "linux-nvme@lists.infradead.org" , "linux-rdma@vger.kernel.org" , "linux-nvdimm@lists.01.org" , "linux-block@vger.kernel.org" , Christoph Hellwig , Jens Axboe , Keith Busch , Sagi Grimberg , Bjorn Helgaas , Jason Gunthorpe , Max Gurtovoy , Dan Williams , Benjamin Herrenschmidt Subject: Re: [PATCH v4 04/14] PCI/P2PDMA: Clear ACS P2P flags for all devices behind switches Message-ID: <20180509174952.GC4140@redhat.com> References: <20180508133407.57a46902@w520.home> <5fc9b1c1-9208-06cc-0ec5-1f54c2520494@deltatee.com> <20180508141331.7cd737cb@w520.home> <20180508205005.GC15608@redhat.com> <7FFB9603-DF9F-4441-82E9-46037CB6C0DE@raithlin.com> <1775CC56-4651-422F-953A-18E024D3717C@raithlin.com> <20180509160722.GB4140@redhat.com> <366A8132-B88A-40F7-BDE3-DA542E45FC0C@raithlin.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <366A8132-B88A-40F7-BDE3-DA542E45FC0C@raithlin.com> User-Agent: Mutt/1.9.2 (2017-12-15) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, May 09, 2018 at 04:30:32PM +0000, Stephen Bates wrote: > Hi Jerome > > > Now inside that page table you can point GPU virtual address > > to use GPU memory or use system memory. Those system memory entry can > > also be mark as ATS against a given PASID. > > Thanks. This all makes sense. > > But do you have examples of this in a kernel driver (if so can you point me too it) or is this all done via user-space? Based on my grepping of the kernel code I see zero EP drivers using in-kernel ATS functionality right now... > As it is tie to PASID this is done using IOMMU so looks for caller of amd_iommu_bind_pasid() or intel_svm_bind_mm() in GPU the existing user is the AMD GPU driver see: drivers/gpu/drm/amd/ drivers/gpu/drm/amd/amdkfd/ drivers/gpu/drm/amd/amdgpu/ Lot of codes there. The GPU code details do not really matter for this discussions thought. You do not need to do much to use PASID. Cheers, Jérôme From mboxrd@z Thu Jan 1 00:00:00 1970 From: jglisse@redhat.com (Jerome Glisse) Date: Wed, 9 May 2018 13:49:53 -0400 Subject: [PATCH v4 04/14] PCI/P2PDMA: Clear ACS P2P flags for all devices behind switches In-Reply-To: <366A8132-B88A-40F7-BDE3-DA542E45FC0C@raithlin.com> References: <20180508133407.57a46902@w520.home> <5fc9b1c1-9208-06cc-0ec5-1f54c2520494@deltatee.com> <20180508141331.7cd737cb@w520.home> <20180508205005.GC15608@redhat.com> <7FFB9603-DF9F-4441-82E9-46037CB6C0DE@raithlin.com> <1775CC56-4651-422F-953A-18E024D3717C@raithlin.com> <20180509160722.GB4140@redhat.com> <366A8132-B88A-40F7-BDE3-DA542E45FC0C@raithlin.com> Message-ID: <20180509174952.GC4140@redhat.com> On Wed, May 09, 2018@04:30:32PM +0000, Stephen Bates wrote: > Hi Jerome > > > Now inside that page table you can point GPU virtual address > > to use GPU memory or use system memory. Those system memory entry can > > also be mark as ATS against a given PASID. > > Thanks. This all makes sense. > > But do you have examples of this in a kernel driver (if so can you point me too it) or is this all done via user-space? Based on my grepping of the kernel code I see zero EP drivers using in-kernel ATS functionality right now... > As it is tie to PASID this is done using IOMMU so looks for caller of amd_iommu_bind_pasid() or intel_svm_bind_mm() in GPU the existing user is the AMD GPU driver see: drivers/gpu/drm/amd/ drivers/gpu/drm/amd/amdkfd/ drivers/gpu/drm/amd/amdgpu/ Lot of codes there. The GPU code details do not really matter for this discussions thought. You do not need to do much to use PASID. Cheers, J?r?me