From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-oi0-x233.google.com (mail-oi0-x233.google.com [IPv6:2607:f8b0:4003:c06::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ml01.01.org (Postfix) with ESMTPS id 3959081D6A for ; Tue, 22 Nov 2016 10:11:53 -0800 (PST) Received: by mail-oi0-x233.google.com with SMTP id w63so26996936oiw.0 for ; Tue, 22 Nov 2016 10:11:53 -0800 (PST) MIME-Version: 1.0 In-Reply-To: References: From: Dan Williams Date: Tue, 22 Nov 2016 10:11:51 -0800 Message-ID: Subject: Re: Enabling peer to peer device transactions for PCIe devices List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: linux-nvdimm-bounces@lists.01.org Sender: "Linux-nvdimm" To: "Deucher, Alexander" Cc: "Sagalovitch, Serguei" , "linux-nvdimm@lists.01.org" , "linux-rdma@vger.kernel.org" , "linux-pci@vger.kernel.org" , Kuehling,, "John , linux-kernel@vger.kernel.org" , "dri-devel@lists.freedesktop.org" , Koenig,, "Ben , Suthikulpanit, Suravee" , "Blinzer, Paul , Linux-media@vger.kernel.org" List-ID: On Mon, Nov 21, 2016 at 12:36 PM, Deucher, Alexander wrote: > This is certainly not the first time this has been brought up, but I'd like to try and get some consensus on the best way to move this forward. Allowing devices to talk directly improves performance and reduces latency by avoiding the use of staging buffers in system memory. Also in cases where both devices are behind a switch, it avoids the CPU entirely. Most current APIs (DirectGMA, PeerDirect, CUDA, HSA) that deal with this are pointer based. Ideally we'd be able to take a CPU virtual address and be able to get to a physical address taking into account IOMMUs, etc. Having struct pages for the memory would allow it to work more generally and wouldn't require as much explicit support in drivers that wanted to use it. > > Some use cases: > 1. Storage devices streaming directly to GPU device memory > 2. GPU device memory to GPU device memory streaming > 3. DVB/V4L/SDI devices streaming directly to GPU device memory > 4. DVB/V4L/SDI devices streaming directly to storage devices > > Here is a relatively simple example of how this could work for testing. This is obviously not a complete solution. > - Device memory will be registered with Linux memory sub-system by created corresponding struct page structures for device memory > - get_user_pages_fast() will return corresponding struct pages when CPU address points to the device memory > - put_page() will deal with struct pages for device memory > [..] > 4. iopmem > iopmem : A block device for PCIe memory (https://lwn.net/Articles/703895/) The change I suggest for this particular approach is to switch to "device-DAX" [1]. I.e. a character device for establishing DAX mappings rather than a block device plus a DAX filesystem. The pro of this approach is standard user pointers and struct pages rather than a new construct. The con is that this is done via an interface separate from the existing gpu and storage device. For example it would require a /dev/dax instance alongside a /dev/nvme interface, but I don't see that as a significant blocking concern. [1]: https://lists.01.org/pipermail/linux-nvdimm/2016-October/007496.html _______________________________________________ Linux-nvdimm mailing list Linux-nvdimm@lists.01.org https://lists.01.org/mailman/listinfo/linux-nvdimm From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dan Williams Subject: Re: Enabling peer to peer device transactions for PCIe devices Date: Tue, 22 Nov 2016 10:11:51 -0800 Message-ID: References: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: linux-nvdimm-bounces-hn68Rpc1hR1g9hUCZPvPmw@public.gmane.org Sender: "Linux-nvdimm" To: "Deucher, Alexander" Cc: "Sagalovitch, Serguei" , "linux-nvdimm-hn68Rpc1hR1g9hUCZPvPmw@public.gmane.org" , "linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , "linux-pci-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , "Kuehling, Felix" , "Bridgman, John" , "linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , "dri-devel-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org" , "Koenig, Christian" , "Sander, Ben" , "Suthikulpanit, Suravee" , "Blinzer, Paul" , "Linux-media-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" List-Id: linux-rdma@vger.kernel.org On Mon, Nov 21, 2016 at 12:36 PM, Deucher, Alexander wrote: > This is certainly not the first time this has been brought up, but I'd like to try and get some consensus on the best way to move this forward. Allowing devices to talk directly improves performance and reduces latency by avoiding the use of staging buffers in system memory. Also in cases where both devices are behind a switch, it avoids the CPU entirely. Most current APIs (DirectGMA, PeerDirect, CUDA, HSA) that deal with this are pointer based. Ideally we'd be able to take a CPU virtual address and be able to get to a physical address taking into account IOMMUs, etc. Having struct pages for the memory would allow it to work more generally and wouldn't require as much explicit support in drivers that wanted to use it. > > Some use cases: > 1. Storage devices streaming directly to GPU device memory > 2. GPU device memory to GPU device memory streaming > 3. DVB/V4L/SDI devices streaming directly to GPU device memory > 4. DVB/V4L/SDI devices streaming directly to storage devices > > Here is a relatively simple example of how this could work for testing. This is obviously not a complete solution. > - Device memory will be registered with Linux memory sub-system by created corresponding struct page structures for device memory > - get_user_pages_fast() will return corresponding struct pages when CPU address points to the device memory > - put_page() will deal with struct pages for device memory > [..] > 4. iopmem > iopmem : A block device for PCIe memory (https://lwn.net/Articles/703895/) The change I suggest for this particular approach is to switch to "device-DAX" [1]. I.e. a character device for establishing DAX mappings rather than a block device plus a DAX filesystem. The pro of this approach is standard user pointers and struct pages rather than a new construct. The con is that this is done via an interface separate from the existing gpu and storage device. For example it would require a /dev/dax instance alongside a /dev/nvme interface, but I don't see that as a significant blocking concern. [1]: https://lists.01.org/pipermail/linux-nvdimm/2016-October/007496.html From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S934315AbcKVSL4 (ORCPT ); Tue, 22 Nov 2016 13:11:56 -0500 Received: from mail-oi0-f47.google.com ([209.85.218.47]:34762 "EHLO mail-oi0-f47.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S934187AbcKVSLy (ORCPT ); Tue, 22 Nov 2016 13:11:54 -0500 MIME-Version: 1.0 In-Reply-To: References: From: Dan Williams Date: Tue, 22 Nov 2016 10:11:51 -0800 Message-ID: Subject: Re: Enabling peer to peer device transactions for PCIe devices To: "Deucher, Alexander" Cc: "linux-kernel@vger.kernel.org" , "linux-rdma@vger.kernel.org" , "linux-nvdimm@lists.01.org" , "Linux-media@vger.kernel.org" , "dri-devel@lists.freedesktop.org" , "linux-pci@vger.kernel.org" , "Bridgman, John" , "Kuehling, Felix" , "Sagalovitch, Serguei" , "Blinzer, Paul" , "Koenig, Christian" , "Suthikulpanit, Suravee" , "Sander, Ben" Content-Type: text/plain; charset=UTF-8 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by mail.home.local id uAMIC0NA032411 On Mon, Nov 21, 2016 at 12:36 PM, Deucher, Alexander wrote: > This is certainly not the first time this has been brought up, but I'd like to try and get some consensus on the best way to move this forward. Allowing devices to talk directly improves performance and reduces latency by avoiding the use of staging buffers in system memory. Also in cases where both devices are behind a switch, it avoids the CPU entirely. Most current APIs (DirectGMA, PeerDirect, CUDA, HSA) that deal with this are pointer based. Ideally we'd be able to take a CPU virtual address and be able to get to a physical address taking into account IOMMUs, etc. Having struct pages for the memory would allow it to work more generally and wouldn't require as much explicit support in drivers that wanted to use it. > > Some use cases: > 1. Storage devices streaming directly to GPU device memory > 2. GPU device memory to GPU device memory streaming > 3. DVB/V4L/SDI devices streaming directly to GPU device memory > 4. DVB/V4L/SDI devices streaming directly to storage devices > > Here is a relatively simple example of how this could work for testing. This is obviously not a complete solution. > - Device memory will be registered with Linux memory sub-system by created corresponding struct page structures for device memory > - get_user_pages_fast() will return corresponding struct pages when CPU address points to the device memory > - put_page() will deal with struct pages for device memory > [..] > 4. iopmem > iopmem : A block device for PCIe memory (https://lwn.net/Articles/703895/) The change I suggest for this particular approach is to switch to "device-DAX" [1]. I.e. a character device for establishing DAX mappings rather than a block device plus a DAX filesystem. The pro of this approach is standard user pointers and struct pages rather than a new construct. The con is that this is done via an interface separate from the existing gpu and storage device. For example it would require a /dev/dax instance alongside a /dev/nvme interface, but I don't see that as a significant blocking concern. [1]: https://lists.01.org/pipermail/linux-nvdimm/2016-October/007496.html From mboxrd@z Thu Jan 1 00:00:00 1970 Return-path: MIME-Version: 1.0 In-Reply-To: References: From: Dan Williams Date: Tue, 22 Nov 2016 10:11:51 -0800 Message-ID: Subject: Re: Enabling peer to peer device transactions for PCIe devices To: "Deucher, Alexander" Cc: "linux-kernel@vger.kernel.org" , "linux-rdma@vger.kernel.org" , "linux-nvdimm@lists.01.org" , "Linux-media@vger.kernel.org" , "dri-devel@lists.freedesktop.org" , "linux-pci@vger.kernel.org" , "Bridgman, John" , "Kuehling, Felix" , "Sagalovitch, Serguei" , "Blinzer, Paul" , "Koenig, Christian" , "Suthikulpanit, Suravee" , "Sander, Ben" Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Sender: linux-media-owner@vger.kernel.org List-ID: On Mon, Nov 21, 2016 at 12:36 PM, Deucher, Alexander wrote: > This is certainly not the first time this has been brought up, but I'd li= ke to try and get some consensus on the best way to move this forward. All= owing devices to talk directly improves performance and reduces latency by = avoiding the use of staging buffers in system memory. Also in cases where = both devices are behind a switch, it avoids the CPU entirely. Most current= APIs (DirectGMA, PeerDirect, CUDA, HSA) that deal with this are pointer ba= sed. Ideally we'd be able to take a CPU virtual address and be able to get= to a physical address taking into account IOMMUs, etc. Having struct page= s for the memory would allow it to work more generally and wouldn't require= as much explicit support in drivers that wanted to use it. > > Some use cases: > 1. Storage devices streaming directly to GPU device memory > 2. GPU device memory to GPU device memory streaming > 3. DVB/V4L/SDI devices streaming directly to GPU device memory > 4. DVB/V4L/SDI devices streaming directly to storage devices > > Here is a relatively simple example of how this could work for testing. = This is obviously not a complete solution. > - Device memory will be registered with Linux memory sub-system by create= d corresponding struct page structures for device memory > - get_user_pages_fast() will return corresponding struct pages when CPU = address points to the device memory > - put_page() will deal with struct pages for device memory > [..] > 4. iopmem > iopmem : A block device for PCIe memory (https://lwn.net/Articles/703895/= ) The change I suggest for this particular approach is to switch to "device-DAX" [1]. I.e. a character device for establishing DAX mappings rather than a block device plus a DAX filesystem. The pro of this approach is standard user pointers and struct pages rather than a new construct. The con is that this is done via an interface separate from the existing gpu and storage device. For example it would require a /dev/dax instance alongside a /dev/nvme interface, but I don't see that as a significant blocking concern. [1]: https://lists.01.org/pipermail/linux-nvdimm/2016-October/007496.html From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-oi0-f50.google.com ([209.85.218.50]:34858 "EHLO mail-oi0-f50.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S933839AbcKVSLx (ORCPT ); Tue, 22 Nov 2016 13:11:53 -0500 Received: by mail-oi0-f50.google.com with SMTP id b126so26944345oia.2 for ; Tue, 22 Nov 2016 10:11:53 -0800 (PST) MIME-Version: 1.0 In-Reply-To: References: From: Dan Williams Date: Tue, 22 Nov 2016 10:11:51 -0800 Message-ID: Subject: Re: Enabling peer to peer device transactions for PCIe devices To: "Deucher, Alexander" Cc: "linux-kernel@vger.kernel.org" , "linux-rdma@vger.kernel.org" , "linux-nvdimm@lists.01.org" , "Linux-media@vger.kernel.org" , "dri-devel@lists.freedesktop.org" , "linux-pci@vger.kernel.org" , "Bridgman, John" , "Kuehling, Felix" , "Sagalovitch, Serguei" , "Blinzer, Paul" , "Koenig, Christian" , "Suthikulpanit, Suravee" , "Sander, Ben" Content-Type: text/plain; charset=UTF-8 Sender: linux-pci-owner@vger.kernel.org List-ID: On Mon, Nov 21, 2016 at 12:36 PM, Deucher, Alexander wrote: > This is certainly not the first time this has been brought up, but I'd li= ke to try and get some consensus on the best way to move this forward. All= owing devices to talk directly improves performance and reduces latency by = avoiding the use of staging buffers in system memory. Also in cases where = both devices are behind a switch, it avoids the CPU entirely. Most current= APIs (DirectGMA, PeerDirect, CUDA, HSA) that deal with this are pointer ba= sed. Ideally we'd be able to take a CPU virtual address and be able to get= to a physical address taking into account IOMMUs, etc. Having struct page= s for the memory would allow it to work more generally and wouldn't require= as much explicit support in drivers that wanted to use it. > > Some use cases: > 1. Storage devices streaming directly to GPU device memory > 2. GPU device memory to GPU device memory streaming > 3. DVB/V4L/SDI devices streaming directly to GPU device memory > 4. DVB/V4L/SDI devices streaming directly to storage devices > > Here is a relatively simple example of how this could work for testing. = This is obviously not a complete solution. > - Device memory will be registered with Linux memory sub-system by create= d corresponding struct page structures for device memory > - get_user_pages_fast() will return corresponding struct pages when CPU = address points to the device memory > - put_page() will deal with struct pages for device memory > [..] > 4. iopmem > iopmem : A block device for PCIe memory (https://lwn.net/Articles/703895/= ) The change I suggest for this particular approach is to switch to "device-DAX" [1]. I.e. a character device for establishing DAX mappings rather than a block device plus a DAX filesystem. The pro of this approach is standard user pointers and struct pages rather than a new construct. The con is that this is done via an interface separate from the existing gpu and storage device. For example it would require a /dev/dax instance alongside a /dev/nvme interface, but I don't see that as a significant blocking concern. [1]: https://lists.01.org/pipermail/linux-nvdimm/2016-October/007496.html