From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Date: Thu, 5 Jan 2017 12:01:13 -0700 From: Jason Gunthorpe Subject: Re: Enabling peer to peer device transactions for PCIe devices Message-ID: <20170105190113.GA12587@obsidianresearch.com> References: <20170105183927.GA5324@gmail.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20170105183927.GA5324@gmail.com> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: linux-nvdimm-bounces@lists.01.org Sender: "Linux-nvdimm" To: Jerome Glisse Cc: david1.zhou@amd.com, "'linux-rdma@vger.kernel.org'" , "'linux-nvdimm@lists.01.org'" , Kuehling,, "Serguei , 'linux-kernel@vger.kernel.org'" , "'dri-devel@lists.freedesktop.org'" , "Koenig, Christian , hch@infradead.org, qiang.yu@amd.com, Sander, Ben" , "Suthikulpanit, Suravee , 'linux-pci@vger.kernel.org'" , Deucher,, "Paul , 'Linux-media@vger.kernel.org'" List-ID: On Thu, Jan 05, 2017 at 01:39:29PM -0500, Jerome Glisse wrote: > 1) peer-to-peer because of userspace specific API like NVidia GPU > direct (AMD is pushing its own similar API i just can't remember > marketing name). This does not happen through a vma, this happens > through specific device driver call going through device specific > ioctl on both side (GPU and RDMA). So both kernel driver are aware > of each others. Today you can only do user-initiated RDMA operations in conjection with a VMA. We'd need a really big and strong reason to create an entirely new non-VMA based memory handle scheme for RDMA. So my inclination is to just completely push back on this idea. You need a VMA to do RMA. GPUs need to create VMAs for the memory they want to RDMA from, even if the VMA handle just causes SIGBUS for any CPU access. Jason _______________________________________________ Linux-nvdimm mailing list Linux-nvdimm@lists.01.org https://lists.01.org/mailman/listinfo/linux-nvdimm From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jason Gunthorpe Subject: Re: Enabling peer to peer device transactions for PCIe devices Date: Thu, 5 Jan 2017 12:01:13 -0700 Message-ID: <20170105190113.GA12587@obsidianresearch.com> References: <20170105183927.GA5324@gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: Content-Disposition: inline In-Reply-To: <20170105183927.GA5324-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: linux-nvdimm-bounces-hn68Rpc1hR1g9hUCZPvPmw@public.gmane.org Sender: "Linux-nvdimm" To: Jerome Glisse Cc: david1.zhou-5C7GfCeVMHo@public.gmane.org, "'linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org'" , "'linux-nvdimm-hn68Rpc1hR1g9hUCZPvPmw@public.gmane.org'" , "Kuehling, Felix" , "Sagalovitch, Serguei" , "'linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org'" , "'dri-devel-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org'" , "Koenig, Christian" , hch-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org, qiang.yu-5C7GfCeVMHo@public.gmane.org, "Sander, Ben" , "Suthikulpanit, Suravee" , "'linux-pci-u79uwXL29TY76Z2rM5mHXA@public.gmane.org'" , "Deucher, Alexander" , "Blinzer, Paul" , "'Linux-media-u79uwXL29TY76Z2rM5mHXA@public.gmane.org'" List-Id: linux-rdma@vger.kernel.org On Thu, Jan 05, 2017 at 01:39:29PM -0500, Jerome Glisse wrote: > 1) peer-to-peer because of userspace specific API like NVidia GPU > direct (AMD is pushing its own similar API i just can't remember > marketing name). This does not happen through a vma, this happens > through specific device driver call going through device specific > ioctl on both side (GPU and RDMA). So both kernel driver are aware > of each others. Today you can only do user-initiated RDMA operations in conjection with a VMA. We'd need a really big and strong reason to create an entirely new non-VMA based memory handle scheme for RDMA. So my inclination is to just completely push back on this idea. You need a VMA to do RMA. GPUs need to create VMAs for the memory they want to RDMA from, even if the VMA handle just causes SIGBUS for any CPU access. Jason From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S937854AbdAETCH (ORCPT ); Thu, 5 Jan 2017 14:02:07 -0500 Received: from quartz.orcorp.ca ([184.70.90.242]:35921 "EHLO quartz.orcorp.ca" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1763438AbdAETBV (ORCPT ); Thu, 5 Jan 2017 14:01:21 -0500 Date: Thu, 5 Jan 2017 12:01:13 -0700 From: Jason Gunthorpe To: Jerome Glisse Cc: "Deucher, Alexander" , "'linux-kernel@vger.kernel.org'" , "'linux-rdma@vger.kernel.org'" , "'linux-nvdimm@lists.01.org'" , "'Linux-media@vger.kernel.org'" , "'dri-devel@lists.freedesktop.org'" , "'linux-pci@vger.kernel.org'" , "Kuehling, Felix" , "Sagalovitch, Serguei" , "Blinzer, Paul" , "Koenig, Christian" , "Suthikulpanit, Suravee" , "Sander, Ben" , hch@infradead.org, david1.zhou@amd.com, qiang.yu@amd.com Subject: Re: Enabling peer to peer device transactions for PCIe devices Message-ID: <20170105190113.GA12587@obsidianresearch.com> References: <20170105183927.GA5324@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20170105183927.GA5324@gmail.com> User-Agent: Mutt/1.5.24 (2015-08-30) X-Broken-Reverse-DNS: no host name found for IP address 10.0.0.156 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Thu, Jan 05, 2017 at 01:39:29PM -0500, Jerome Glisse wrote: > 1) peer-to-peer because of userspace specific API like NVidia GPU > direct (AMD is pushing its own similar API i just can't remember > marketing name). This does not happen through a vma, this happens > through specific device driver call going through device specific > ioctl on both side (GPU and RDMA). So both kernel driver are aware > of each others. Today you can only do user-initiated RDMA operations in conjection with a VMA. We'd need a really big and strong reason to create an entirely new non-VMA based memory handle scheme for RDMA. So my inclination is to just completely push back on this idea. You need a VMA to do RMA. GPUs need to create VMAs for the memory they want to RDMA from, even if the VMA handle just causes SIGBUS for any CPU access. Jason