Re: [LSF/MM TOPIC] get_user_pages() for PCI BAR Memory

[Jason Gunthorpe <jgg@mellanox.com>]

On Sat, Feb 08, 2020 at 02:10:59PM +0100, Christian König wrote:
> > For patch sets, we've seen a number of attempts so far, but little has
> > been merged yet. Common elements of past discussions have been:
> >   - Building struct page for BAR memory
> >   - Stuffing BAR memory into scatter/gather lists, bios and skbs
> >   - DMA mapping BAR memory
> >   - Referencing BAR memory without a struct page
> >   - Managing lifetime of BAR memory across multiple drivers
> 
> I can only repeat Jérôme that this most likely will never work correctly
> with get_user_pages().

I suppose I'm using 'get_user_pages()' as something of a placeholder
here to refer to the existing family of kernel DMA consumers that call
get_user_pages to work on VMA backed process visible memory.

We have to have something like get_user_pages() because the kernel
call-sites are fundamentally only dealing with userspace VA. That is
how their uAPIs are designed, and we want to keep them working.

So, if something doesn't fit into get_user_pages(), ie because it
doesn't have a VMA in the first place, then that is some other
discussion. DMA buf seems like a pretty good answer.

> E.g. you have memory which is not even CPU addressable, but can be shared
> between GPUs using XGMI, NVLink, SLI etc....

For this kind of memory if it is mapped into a VMA with
DEVICE_PRIVATE, as Jerome has imagined, then it would be part of this
discussion.

> So we need to figure out how express DMA addresses outside of the CPU
> address space first before we can even think about something like extending
> get_user_pages() for P2P in an HMM scenario.

Why? This is discussion is not exclusively for GPU. We have many use
cases that do not have CPU invisible memory to worry about, and I
don't think defining how DMA mapping works for cpu-invisible
interconnect overlaps with figuring out how to make get_user_pages
work with existing ZONE_DEVICE memory types.

ie the challenge here is how to deliver the required information to
the p2pdma subsystem so a get_user_pages() call site can do a DMA map.

Improving the p2pdma subsystem to handle more complex cases like CPU
invisible memory and interconnect is a different topic, I think :)

Regards,
Jason