Re: Interacting with coherent memory on external devices

From: Jerome Glisse <j.glisse@gmail.com>
To: Christoph Lameter <cl@linux.com>
Cc: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>,
	linux-kernel@vger.kernel.org, linux-mm@kvack.org,
	jglisse@redhat.com, mgorman@suse.de, aarcange@redhat.com,
	riel@redhat.com, airlied@redhat.com, benh@kernel.crashing.org,
	aneesh.kumar@linux.vnet.ibm.com,
	Cameron Buschardt <cabuschardt@nvidia.com>,
	Mark Hairgrove <mhairgrove@nvidia.com>,
	Geoffrey Gerfin <ggerfin@nvidia.com>,
	John McKenna <jmckenna@nvidia.com>,
	akpm@linux-foundation.org
Subject: Re: Interacting with coherent memory on external devices
Date: Wed, 22 Apr 2015 13:07:38 -0400	[thread overview]
Message-ID: <20150422170737.GB4062@gmail.com> (raw)
In-Reply-To: <alpine.DEB.2.11.1504221105130.24979@gentwo.org>

On Wed, Apr 22, 2015 at 11:16:49AM -0500, Christoph Lameter wrote:
> On Wed, 22 Apr 2015, Paul E. McKenney wrote:
> 
> > I completely agree that some critically important use cases, such as
> > yours, will absolutely require that the application explicitly choose
> > memory placement and have the memory stay there.
> 
> 
> 
> Most of what you are trying to do here is already there and has been done.
> GPU memory is accessible. NICs work etc etc. All without CAPI. What
> exactly are the benefits of CAPI? Is driver simplification? Reduction of
> overhead? If so then the measures proposed are a bit radical and
> may result in just the opposite.
> 

No, what Paul is trying to do, and what i am trying to do with HMM, does
not exist. This is share address space btw CPU and GPU/accelerator and
leveraging GPU local memory transparently at the same time.

Today world is GPU have different address space and complex data structure
like list or tree need to be replicated accross different address space.
You might not care for this but for lot of application this is a show
stopper and the outcome is using GPU is too complex because of that.

Now if you have the exact same address space then structure you have on
the CPU are exactly view in the same way on the GPU and you can start
porting library to leverage GPU without having to change a single line of
code inside many many many applications. It is also lot easier to debug
things as you do not have to strungly with two distinct address space.

Finaly, leveraging transparently the local GPU memory is the only way to
reach the full potential of the GPU. GPU are all about bandwidth and GPU
local memory have bandwidth far greater than any system memory i know
about. Here again if you can transparently leverage this memory without
the application ever needing to know about such subtlety.

But again let me stress that application that want to be in control will
stay in control. If you want to make the decission yourself about where
things should end up then nothing in all we are proposing will preclude
you from doing that. Please just think about others people application,
not just yours, they are a lot of others thing in the world and they do
not want to be as close to the metal as you want to be. We just want to
accomodate the largest number of use case.

Cheers,
Jérôme