From: Anshuman Khandual <khandual@linux.vnet.ibm.com>
To: Bob Liu <liubo95@huawei.com>, Jerome Glisse <jglisse@redhat.com>
Cc: Anshuman Khandual <khandual@linux.vnet.ibm.com>,
Michal Hocko <mhocko@kernel.org>, Mel Gorman <mgorman@suse.de>,
linux-kernel@vger.kernel.org, linux-mm@kvack.org, vbabka@suse.cz,
minchan@kernel.org, aneesh.kumar@linux.vnet.ibm.com,
bsingharora@gmail.com, srikar@linux.vnet.ibm.com,
haren@linux.vnet.ibm.com, dave.hansen@intel.com,
"dan.j.williams@intel.com; jhubbard"@nvidia.com
Subject: Re: [PATCH V3 0/4] Define coherent device memory node
Date: Mon, 27 Feb 2017 11:11:39 +0530 [thread overview]
Message-ID: <67a068c6-6970-aae1-9a1e-847e322f0e39@linux.vnet.ibm.com> (raw)
In-Reply-To: <fc486de7-81ce-6953-3e56-90f45a2e5527@huawei.com>
On 02/27/2017 07:26 AM, Bob Liu wrote:
> On 2017/2/24 12:53, Jerome Glisse wrote:
>> On Fri, Feb 24, 2017 at 09:06:19AM +0800, Bob Liu wrote:
>>> On 2017/2/21 21:39, Anshuman Khandual wrote:
>>>> On 02/21/2017 04:41 PM, Michal Hocko wrote:
>>>>> On Fri 17-02-17 17:11:57, Anshuman Khandual wrote:
>>>>> [...]
>>>>>> * User space using mbind() to get CDM memory is an additional benefit
>>>>>> we get by making the CDM plug in as a node and be part of the buddy
>>>>>> allocator. But the over all idea from the user space point of view
>>>>>> is that the application can allocate any generic buffer and try to
>>>>>> use the buffer either from the CPU side or from the device without
>>>>>> knowing about where the buffer is really mapped physically. That
>>>>>> gives a seamless and transparent view to the user space where CPU
>>>>>> compute and possible device based compute can work together. This
>>>>>> is not possible through a driver allocated buffer.
>>>>>
>>>>> But how are you going to define any policy around that. Who is allowed
>>>>
>>>> The user space VMA can define the policy with a mbind(MPOL_BIND) call
>>>> with CDM/CDMs in the nodemask.
>>>>
>>>>> to allocate and how much of this "special memory". Is it possible that
>>>>
>>>> Any user space application with mbind(MPOL_BIND) call with CDM/CDMs in
>>>> the nodemask can allocate from the CDM memory. "How much" gets controlled
>>>> by how we fault from CPU and the default behavior of the buddy allocator.
>>>>
>>>>> we will eventually need some access control mechanism? If yes then mbind
>>>>
>>>> No access control mechanism is needed. If an application wants to use
>>>> CDM memory by specifying in the mbind() it can. Nothing prevents it
>>>> from using the CDM memory.
>>>>
>>>>> is really not suitable interface to (ab)use. Also what should happen if
>>>>> the mbind mentions only CDM memory and that is depleted?
>>>>
>>>> IIUC *only CDM* cannot be requested from user space as there are no user
>>>> visible interface which can translate to __GFP_THISNODE. MPOL_BIND with
>>>> CDM in the nodemask will eventually pick a FALLBACK zonelist which will
>>>> have zones of the system including CDM ones. If the resultant CDM zones
>>>> run out of memory, we fail the allocation request as usual.
>>>>
>>>>>
>>>>> Could you also explain why the transparent view is really better than
>>>>> using a device specific mmap (aka CDM awareness)?
>>>>
>>>> Okay with a transparent view, we can achieve a control flow of application
>>>> like the following.
>>>>
>>>> (1) Allocate a buffer: alloc_buffer(buf, size)
>>>> (2) CPU compute on buffer: cpu_compute(buf, size)
>>>> (3) Device compute on buffer: device_compute(buf, size)
>>>> (4) CPU compute on buffer: cpu_compute(buf, size)
>>>> (5) Release the buffer: release_buffer(buf, size)
>>>>
>>>> With assistance from a device specific driver, the actual page mapping of
>>>> the buffer can change between system RAM and device memory depending on
>>>> which side is accessing at a given point. This will be achieved through
>>>> driver initiated migrations.
>>>>
>>>
>>> Sorry, I'm a bit confused here.
>>> What's the difference with the Heterogeneous memory management?
>>> Which also "allows to use device memory transparently inside any process
>>> without any modifications to process program code."
>>
>> HMM is first and foremost for platform (like Intel) where CPU can not
>> access device memory in cache coherent way or at all. CDM is for more
>> advance platform with a system bus that allow the CPU to access device
>> memory in cache coherent way.
>>
>> Hence CDM was design to integrate more closely in existing concept like
>> NUMA. From my point of view it is like another level in the memory
>> hierarchy. Nowaday you have local node memory and other node memory.
>> In not too distant future you will have fast CPU on die memory, local
>> memory (you beloved DDR3/DDR4), slightly slower but gigantic persistant
>> memory and also device memory (all those local to a node).
>>
>> On top of that you will still have the regular NUMA hierarchy between
>> nodes. But each node will have its own local hierarchy of memory.
>>
>> CDM wants to integrate with existing memory hinting API and i believe
>> this is needed to get some experience with how end user might want to
>> use this to fine tune their application.
>>
>> Some bit of HMM are generic and will be reuse by CDM, for instance the
>> DMA capable memory migration helpers. Wether they can also share HMM
>> approach of using ZONE_DEVICE is yet to be proven but it comes with
>> limitations (can't be on lru or have device lru) that might hinder a
>> closer integration of CDM memory with many aspect of kernel mm.
>>
>>
>> This is my own view and it likely differ in some way from the view of
>> the people behind CDM :)
>>
>
> Got it, thank you for the kindly explanation.
> And also thank you, John.
Thanks Jerome and John for helping out with the detailed explanation.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
prev parent reply other threads:[~2017-02-27 5:41 UTC|newest]
Thread overview: 43+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-02-15 12:07 [PATCH V3 0/4] Define coherent device memory node Anshuman Khandual
2017-02-15 12:07 ` [PATCH V3 1/4] mm: Define coherent device memory (CDM) node Anshuman Khandual
2017-02-17 14:05 ` Bob Liu
2017-02-21 10:20 ` Anshuman Khandual
2017-02-15 12:07 ` [PATCH V3 2/4] mm: Enable HugeTLB allocation isolation for CDM nodes Anshuman Khandual
2017-02-15 12:07 ` [PATCH V3 3/4] mm: Add new parameter to get_page_from_freelist() function Anshuman Khandual
2017-02-15 12:07 ` [PATCH V3 4/4] mm: Enable Buddy allocation isolation for CDM nodes Anshuman Khandual
2017-02-15 18:20 ` [PATCH V3 0/4] Define coherent device memory node Mel Gorman
2017-02-16 22:14 ` Balbir Singh
2017-02-17 9:33 ` Mel Gorman
2017-02-21 2:57 ` Balbir Singh
2017-03-01 2:42 ` Balbir Singh
2017-03-01 9:55 ` Mel Gorman
2017-03-01 10:59 ` Balbir Singh
2017-03-08 9:04 ` Anshuman Khandual
2017-03-08 9:21 ` [PATCH 1/2] mm: Change generic FALLBACK zonelist creation process Anshuman Khandual
2017-03-08 11:07 ` John Hubbard
2017-03-14 13:33 ` Anshuman Khandual
2017-03-15 4:10 ` John Hubbard
2017-03-08 9:21 ` [PATCH 2/2] mm: Change mbind(MPOL_BIND) implementation for CDM nodes Anshuman Khandual
2017-02-17 11:41 ` [PATCH V3 0/4] Define coherent device memory node Anshuman Khandual
2017-02-17 13:32 ` Mel Gorman
2017-02-21 13:09 ` Anshuman Khandual
2017-02-21 20:14 ` Jerome Glisse
2017-02-23 8:14 ` Anshuman Khandual
2017-02-23 15:27 ` Jerome Glisse
2017-02-22 9:29 ` Michal Hocko
2017-02-22 14:59 ` Jerome Glisse
2017-02-22 16:54 ` Michal Hocko
2017-03-06 5:48 ` Anshuman Khandual
2017-02-23 8:52 ` Anshuman Khandual
2017-02-23 15:57 ` Mel Gorman
2017-03-06 5:12 ` Anshuman Khandual
2017-02-21 11:11 ` Michal Hocko
2017-02-21 13:39 ` Anshuman Khandual
2017-02-22 9:50 ` Michal Hocko
2017-02-23 6:52 ` Anshuman Khandual
2017-03-05 12:39 ` Anshuman Khandual
2017-02-24 1:06 ` Bob Liu
2017-02-24 4:39 ` John Hubbard
2017-02-24 4:53 ` Jerome Glisse
2017-02-27 1:56 ` Bob Liu
2017-02-27 5:41 ` Anshuman Khandual [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=67a068c6-6970-aae1-9a1e-847e322f0e39@linux.vnet.ibm.com \
--to=khandual@linux.vnet.ibm.com \
--cc="dan.j.williams@intel.com; jhubbard"@nvidia.com \
--cc=aneesh.kumar@linux.vnet.ibm.com \
--cc=bsingharora@gmail.com \
--cc=dave.hansen@intel.com \
--cc=haren@linux.vnet.ibm.com \
--cc=jglisse@redhat.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=liubo95@huawei.com \
--cc=mgorman@suse.de \
--cc=mhocko@kernel.org \
--cc=minchan@kernel.org \
--cc=srikar@linux.vnet.ibm.com \
--cc=vbabka@suse.cz \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).