From: Bob Liu <liubo95@huawei.com>
To: Anshuman Khandual <khandual@linux.vnet.ibm.com>,
Michal Hocko <mhocko@kernel.org>
Cc: Mel Gorman <mgorman@suse.de>,
linux-kernel@vger.kernel.org, linux-mm@kvack.org, vbabka@suse.cz,
minchan@kernel.org, aneesh.kumar@linux.vnet.ibm.com,
bsingharora@gmail.com, srikar@linux.vnet.ibm.com,
haren@linux.vnet.ibm.com, jglisse@redhat.com,
dave.hansen@intel.com, dan.j.williams@intel.com
Subject: Re: [PATCH V3 0/4] Define coherent device memory node
Date: Fri, 24 Feb 2017 09:06:19 +0800 [thread overview]
Message-ID: <60b3dd35-a802-ba93-c2c5-d6b2b3dd72ea@huawei.com> (raw)
In-Reply-To: <890fb824-d1f0-3711-4fe6-d6ddf29a0d80@linux.vnet.ibm.com>
On 2017/2/21 21:39, Anshuman Khandual wrote:
> On 02/21/2017 04:41 PM, Michal Hocko wrote:
>> On Fri 17-02-17 17:11:57, Anshuman Khandual wrote:
>> [...]
>>> * User space using mbind() to get CDM memory is an additional benefit
>>> we get by making the CDM plug in as a node and be part of the buddy
>>> allocator. But the over all idea from the user space point of view
>>> is that the application can allocate any generic buffer and try to
>>> use the buffer either from the CPU side or from the device without
>>> knowing about where the buffer is really mapped physically. That
>>> gives a seamless and transparent view to the user space where CPU
>>> compute and possible device based compute can work together. This
>>> is not possible through a driver allocated buffer.
>>
>> But how are you going to define any policy around that. Who is allowed
>
> The user space VMA can define the policy with a mbind(MPOL_BIND) call
> with CDM/CDMs in the nodemask.
>
>> to allocate and how much of this "special memory". Is it possible that
>
> Any user space application with mbind(MPOL_BIND) call with CDM/CDMs in
> the nodemask can allocate from the CDM memory. "How much" gets controlled
> by how we fault from CPU and the default behavior of the buddy allocator.
>
>> we will eventually need some access control mechanism? If yes then mbind
>
> No access control mechanism is needed. If an application wants to use
> CDM memory by specifying in the mbind() it can. Nothing prevents it
> from using the CDM memory.
>
>> is really not suitable interface to (ab)use. Also what should happen if
>> the mbind mentions only CDM memory and that is depleted?
>
> IIUC *only CDM* cannot be requested from user space as there are no user
> visible interface which can translate to __GFP_THISNODE. MPOL_BIND with
> CDM in the nodemask will eventually pick a FALLBACK zonelist which will
> have zones of the system including CDM ones. If the resultant CDM zones
> run out of memory, we fail the allocation request as usual.
>
>>
>> Could you also explain why the transparent view is really better than
>> using a device specific mmap (aka CDM awareness)?
>
> Okay with a transparent view, we can achieve a control flow of application
> like the following.
>
> (1) Allocate a buffer: alloc_buffer(buf, size)
> (2) CPU compute on buffer: cpu_compute(buf, size)
> (3) Device compute on buffer: device_compute(buf, size)
> (4) CPU compute on buffer: cpu_compute(buf, size)
> (5) Release the buffer: release_buffer(buf, size)
>
> With assistance from a device specific driver, the actual page mapping of
> the buffer can change between system RAM and device memory depending on
> which side is accessing at a given point. This will be achieved through
> driver initiated migrations.
>
Sorry, I'm a bit confused here.
What's the difference with the Heterogeneous memory management?
Which also "allows to use device memory transparently inside any process
without any modifications to process program code."
Thanks,
-Bob
>>
>>> * The placement of the memory on the buffer can happen on system memory
>>> when the CPU faults while accessing it. But a driver can manage the
>>> migration between system RAM and CDM memory once the buffer is being
>>> used from CPU and the device interchangeably. As you have mentioned
>>> driver will have more information about where which part of the buffer
>>> should be placed at any point of time and it can make it happen with
>>> migration. So both allocation and placement are decided by the driver
>>> during runtime. CDM provides the framework for this can kind device
>>> assisted compute and driver managed memory placements.
>>>
>>> * If any application is not using CDM memory for along time placed on
>>> its buffer and another application is forced to fallback on system
>>> RAM when it really wanted is CDM, the driver can detect these kind
>>> of situations through memory access patterns on the device HW and
>>> take necessary migration decisions.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
next prev parent reply other threads:[~2017-02-24 1:09 UTC|newest]
Thread overview: 43+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-02-15 12:07 [PATCH V3 0/4] Define coherent device memory node Anshuman Khandual
2017-02-15 12:07 ` [PATCH V3 1/4] mm: Define coherent device memory (CDM) node Anshuman Khandual
2017-02-17 14:05 ` Bob Liu
2017-02-21 10:20 ` Anshuman Khandual
2017-02-15 12:07 ` [PATCH V3 2/4] mm: Enable HugeTLB allocation isolation for CDM nodes Anshuman Khandual
2017-02-15 12:07 ` [PATCH V3 3/4] mm: Add new parameter to get_page_from_freelist() function Anshuman Khandual
2017-02-15 12:07 ` [PATCH V3 4/4] mm: Enable Buddy allocation isolation for CDM nodes Anshuman Khandual
2017-02-15 18:20 ` [PATCH V3 0/4] Define coherent device memory node Mel Gorman
2017-02-16 22:14 ` Balbir Singh
2017-02-17 9:33 ` Mel Gorman
2017-02-21 2:57 ` Balbir Singh
2017-03-01 2:42 ` Balbir Singh
2017-03-01 9:55 ` Mel Gorman
2017-03-01 10:59 ` Balbir Singh
2017-03-08 9:04 ` Anshuman Khandual
2017-03-08 9:21 ` [PATCH 1/2] mm: Change generic FALLBACK zonelist creation process Anshuman Khandual
2017-03-08 11:07 ` John Hubbard
2017-03-14 13:33 ` Anshuman Khandual
2017-03-15 4:10 ` John Hubbard
2017-03-08 9:21 ` [PATCH 2/2] mm: Change mbind(MPOL_BIND) implementation for CDM nodes Anshuman Khandual
2017-02-17 11:41 ` [PATCH V3 0/4] Define coherent device memory node Anshuman Khandual
2017-02-17 13:32 ` Mel Gorman
2017-02-21 13:09 ` Anshuman Khandual
2017-02-21 20:14 ` Jerome Glisse
2017-02-23 8:14 ` Anshuman Khandual
2017-02-23 15:27 ` Jerome Glisse
2017-02-22 9:29 ` Michal Hocko
2017-02-22 14:59 ` Jerome Glisse
2017-02-22 16:54 ` Michal Hocko
2017-03-06 5:48 ` Anshuman Khandual
2017-02-23 8:52 ` Anshuman Khandual
2017-02-23 15:57 ` Mel Gorman
2017-03-06 5:12 ` Anshuman Khandual
2017-02-21 11:11 ` Michal Hocko
2017-02-21 13:39 ` Anshuman Khandual
2017-02-22 9:50 ` Michal Hocko
2017-02-23 6:52 ` Anshuman Khandual
2017-03-05 12:39 ` Anshuman Khandual
2017-02-24 1:06 ` Bob Liu [this message]
2017-02-24 4:39 ` John Hubbard
2017-02-24 4:53 ` Jerome Glisse
2017-02-27 1:56 ` Bob Liu
2017-02-27 5:41 ` Anshuman Khandual
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=60b3dd35-a802-ba93-c2c5-d6b2b3dd72ea@huawei.com \
--to=liubo95@huawei.com \
--cc=aneesh.kumar@linux.vnet.ibm.com \
--cc=bsingharora@gmail.com \
--cc=dan.j.williams@intel.com \
--cc=dave.hansen@intel.com \
--cc=haren@linux.vnet.ibm.com \
--cc=jglisse@redhat.com \
--cc=khandual@linux.vnet.ibm.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=mgorman@suse.de \
--cc=mhocko@kernel.org \
--cc=minchan@kernel.org \
--cc=srikar@linux.vnet.ibm.com \
--cc=vbabka@suse.cz \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).