Re: [RFC 0/4] RFC - Coherent Device Memory (Not for inclusion)

[John Hubbard <jhubbard@nvidia.com>]

On 05/01/2017 02:04 PM, Reza Arbab wrote:
> On Mon, May 01, 2017 at 01:41:55PM -0700, John Hubbard wrote:
>> 1. A way to move pages between NUMA nodes, both virtual address and physical 
>> address-based, from kernel mode.
> 
> Jérôme's migrate_vma() and migrate_dma() should have this covered, including 
> DMA-accelerated copy.

Yes, that's good. I wasn't sure from this discussion here if either or both of those 
would be used, but now I see.

Are those APIs ready for moving pages between NUMA nodes? As there is no NUMA node 
id in the API, are we relying on the pages' membership (using each page and updating 
which node it is on)?

> 
>> 5. Something to handle the story of bringing NUMA nodes online and putting them 
>> back offline, given that they require a device driver that may not yet have been 
>> loaded. There are a few minor missing bits there.
> 
> This has been prototyped with the driver doing memory hotplug/hotremove. Could you 
> elaborate a little on what you feel is missing?
> 

We just worked through how to deal with this in our driver, and I remember feeling 
worried about the way NUMA nodes can only be put online via a user space action 
(through sysfs). It seemed like you'd want to do that from kernel as well, when a 
device driver gets loaded.

I was also uneasy about user space trying to bring a node online before the 
associated device driver was loaded, and I think it would be nice to be sure that 
that whole story is looked at.

The theme here is that driver load/unload is, today, independent from the NUMA node 
online/offline, and that's a problem. Not a huge one, though, just worth enumerating 
here.

thanks
john h

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>