Re: [RFC] libibverbs IB Device Memory support

[Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>]

[-- Attachment #1: Type: text/plain, Size: 1646 bytes --]

On Mon, Jun 05, 2017 at 11:17:49AM -0600, Jason Gunthorpe wrote:
> On Mon, Jun 05, 2017 at 08:08:25PM +0300, Leon Romanovsky wrote:
> > On Mon, Jun 05, 2017 at 11:44:00AM -0500, Christoph Lameter wrote:
> > > On Wed, 10 May 2017, ahmad omary wrote:
> > >
> > > > We have considered using mmap(), but As the size of device memory may be limited
> > > > ,the way to access it from host cpu may differ from vendor to vendor, due to
> > > > the 4K (page) aligment limitation of mmap() and the need not to directly
> > > > allow user to access the device memory, there is a need for a wrapper access
> > > > methods API that allows allocating and managing chunks that are smaller than
> > > > 4KB and not necessarily aligned to 4KB (page size).
> > >
> > > Why are 4k sized chunks a problem given that there are megabytes of memory
> > > in these devices? We are using various adapters already with an mmapped
> > > solution here.
> >
> > Ahmad presented use case where he needs access to small objects
> > (semaphores) in large scale (MPI). 1MB in the granularity of 4k will give us
> > 256 chunks only, and it is definitely not enough.
>
> Is 256 chunks per machine not enough? A single process could carve out
> smaller regions from the 4k kernel allocation.

It is rough calculation for 1MB, when I asked Ahmad about this
limitation (4K) he explained to me that exposed device memory is less
than 1MB.

Thanks

>
> Jason
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]