Re: Enabling peer to peer device transactions for PCIe devices

From: Logan Gunthorpe <logang@deltatee.com>
To: Dan Williams <dan.j.williams@intel.com>,
	Jason Gunthorpe <jgunthorpe@obsidianresearch.com>
Cc: Stephen Bates <sbates@raithlin.com>,
	Haggai Eran <haggaie@mellanox.com>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"linux-rdma@vger.kernel.org" <linux-rdma@vger.kernel.org>,
	"linux-nvdimm@ml01.01.org" <linux-nvdimm@ml01.01.org>,
	"christian.koenig@amd.com" <christian.koenig@amd.com>,
	"Suravee.Suthikulpanit@amd.com" <suravee.suthikulpanit@amd.com>,
	"John.Bridgman@amd.com" <john.bridgman@amd.com>,
	"Alexander.Deucher@amd.com" <alexander.deucher@amd.com>,
	"Linux-media@vger.kernel.org" <linux-media@vger.kernel.org>,
	"dri-devel@lists.freedesktop.org"
	<dri-devel@lists.freedesktop.org>,
	Max Gurtovoy <maxg@mellanox.com>,
	"linux-pci@vger.kernel.org" <linux-pci@vger.kernel.org>,
	"serguei.sagalovitch@amd.com" <serguei.sagalovitch@amd.com>,
	"Paul.Blinzer@amd.com" <paul.blinzer@amd.com>,
	"Felix.Kuehling@amd.com" <felix.kuehling@amd.com>,
	"ben.sander@amd.com" <ben.sander@amd.com>
Subject: Re: Enabling peer to peer device transactions for PCIe devices
Date: Mon, 5 Dec 2016 11:39:13 -0700	[thread overview]
Message-ID: <a3a1c239-297d-c091-7758-54acdf00f74e@deltatee.com> (raw)
In-Reply-To: <CAPcyv4iEXwvtDbZgnWzdKU6uN_sOGmXH1KtW_Nws6kUftJUigQ@mail.gmail.com>

On 05/12/16 11:08 AM, Dan Williams wrote:
> I've already recommended that iopmem not be a block device and instead
> be a device-dax instance. I also don't think it should claim the PCI
> ID, rather the driver that wants to map one of its bars this way can
> register the memory region with the device-dax core.
>
> I'm not sure there are enough device drivers that want to do this to
> have it be a generic /sys/.../resource_dmableX capability. It still
> seems to be an exotic one-off type of configuration.

Yes, this is essentially my thinking. Except I think the userspace 
interface should really depend on the device itself. Device dax is a 
good  choice for many and I agree the block device approach wouldn't be 
ideal.

Specifically for NVME CMB: I think it would make a lot of sense to just 
hand out these mappings with an mmap call on /dev/nvmeX. I expect CMB 
buffers would be volatile and thus you wouldn't need to keep track of 
where in the BAR the region came from. Thus, the mmap call would just be 
an allocator from BAR memory. If device-dax were used, userspace would 
need to lookup which device-dax instance corresponds to which nvme drive.

Logan