RE: [RFC] libibverbs IB Device Memory support

From: Liran Liss <liranl-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
To: Christoph Lameter <cl-vYTEC60ixJUAvxtiuMwx3w@public.gmane.org>,
	Ahmad Omary <ahmad-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Cc: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>,
	Jason Gunthorpe
	<jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>,
	ahmad omary <ahmad151084-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>,
	"linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org"
	<linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>,
	Yishai Hadas <yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>,
	Tzahi Oved <tzahio-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>,
	Alex Rosenbaum <alexr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>,
	Ariel Levkovich <lariel-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>,
	Christoph Hellwig <hch-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>
Subject: RE: [RFC] libibverbs IB Device Memory support
Date: Thu, 8 Jun 2017 13:10:34 +0000	[thread overview]
Message-ID: <HE1PR0501MB2812BC7D5A7824FFD77B4D75B1C90@HE1PR0501MB2812.eurprd05.prod.outlook.com> (raw)
In-Reply-To: <alpine.DEB.2.20.1706061022270.16819-wcBtFHqTun5QOdAKl3ChDw@public.gmane.org>

> From: Christoph Lameter [mailto:cl-vYTEC60ixJUAvxtiuMwx3w@public.gmane.org]
> Sent: Tuesday, June 06, 2017 6:30 PM

Hi Chistoph,

> The OS needs to provide the protection if that is the case and then it probably is
> not a HPC device. We are talking about RDMA high end devices here. The design
> here is for performance and low latency. I dont know of any devices in use in
> HPC or in HFT that have these tiny memory sizes.
> Mostly these devices are already engineeded for mmapping.
> 
> Is this for some kind of embedded device?
> 

In our case, this is the HCA.
Device memory could be a scarce resource, and even allocating a 4K page per process could wasteful in some systems.
An allocation API allows allocations of any size.

> > Note that the device memory does not necessary have to be mapped to
> > the CPU= .
> > i.e. is not necessary accessible by PCI, and can only be accessed by RDMA.
> > This is why we can't use MMAP for all cases  and a dedicated
> > allocation and  copy functions are needed.
> 
> Can we come up with some sort of ioctl API then to write to the devices
> inaccessible memory? There must be other drives outside of the RDMA tree that
> have similar requirements and that may already have implemented some
> version of it.
> 
> This seems to be a very specialized application that may be device specific. ioctls
> are usually used then.

The host to dev memcopy is often a user-space only action, so ioctl() doesn't work for us.
The memcpy API allows efficient data transfer regardless of the implementation.

I also don't think that always mandating a direct mmap() is the solution.
First, not in all cases such a mapping exists.
Some devices have only limited BAR space, which can only be dynamically mapped to a small window at a time. In this case, you might need to move the window before the access.
Other device might not be able to map device memory to the CPU at all.

Even if a direct mapping is possible, you don't expose mmap()'d IO memory directly to the application.
There are details such as optimizing for write-combining, locking, and flushing that the application shouldn't be aware of.
--Liran
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html