From: "Huang, Ying" <ying.huang@intel.com>
To: "Aneesh Kumar K.V" <aneesh.kumar@linux.ibm.com>
Cc: Vishal Verma <vishal.l.verma@intel.com>,
Andrew Morton <akpm@linux-foundation.org>,
David Hildenbrand <david@redhat.com>,
Oscar Salvador <osalvador@suse.de>,
Dan Williams <dan.j.williams@intel.com>,
Dave Jiang <dave.jiang@intel.com>,
linux-kernel@vger.kernel.org, linux-mm@kvack.org,
nvdimm@lists.linux.dev, linux-cxl@vger.kernel.org,
Dave Hansen <dave.hansen@linux.intel.com>,
Jonathan Cameron <Jonathan.Cameron@Huawei.com>,
Jeff Moyer <jmoyer@redhat.com>
Subject: Re: [PATCH v2 2/3] mm/memory_hotplug: split memmap_on_memory requests across memblocks
Date: Mon, 24 Jul 2023 11:16:28 +0800 [thread overview]
Message-ID: <87351e2e43.fsf@yhuang6-desk2.ccr.corp.intel.com> (raw)
In-Reply-To: <87a5vmadcw.fsf@linux.ibm.com> (Aneesh Kumar K. V.'s message of "Sun, 23 Jul 2023 20:23:19 +0530")
"Aneesh Kumar K.V" <aneesh.kumar@linux.ibm.com> writes:
> Vishal Verma <vishal.l.verma@intel.com> writes:
>
>> The MHP_MEMMAP_ON_MEMORY flag for hotplugged memory is currently
>> restricted to 'memblock_size' chunks of memory being added. Adding a
>> larger span of memory precludes memmap_on_memory semantics.
>>
>> For users of hotplug such as kmem, large amounts of memory might get
>> added from the CXL subsystem. In some cases, this amount may exceed the
>> available 'main memory' to store the memmap for the memory being added.
>> In this case, it is useful to have a way to place the memmap on the
>> memory being added, even if it means splitting the addition into
>> memblock-sized chunks.
>>
>> Change add_memory_resource() to loop over memblock-sized chunks of
>> memory if caller requested memmap_on_memory, and if other conditions for
>> it are met,. Teach try_remove_memory() to also expect that a memory
>> range being removed might have been split up into memblock sized chunks,
>> and to loop through those as needed.
>>
>> Cc: Andrew Morton <akpm@linux-foundation.org>
>> Cc: David Hildenbrand <david@redhat.com>
>> Cc: Oscar Salvador <osalvador@suse.de>
>> Cc: Dan Williams <dan.j.williams@intel.com>
>> Cc: Dave Jiang <dave.jiang@intel.com>
>> Cc: Dave Hansen <dave.hansen@linux.intel.com>
>> Cc: Huang Ying <ying.huang@intel.com>
>> Suggested-by: David Hildenbrand <david@redhat.com>
>> Signed-off-by: Vishal Verma <vishal.l.verma@intel.com>
>> ---
>> mm/memory_hotplug.c | 154 +++++++++++++++++++++++++++++++---------------------
>> 1 file changed, 91 insertions(+), 63 deletions(-)
>>
>> diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
>> index e9bcacbcbae2..20456f0d28e6 100644
>> --- a/mm/memory_hotplug.c
>> +++ b/mm/memory_hotplug.c
>> @@ -1286,6 +1286,35 @@ bool mhp_supports_memmap_on_memory(unsigned long size)
>> }
>> EXPORT_SYMBOL_GPL(mhp_supports_memmap_on_memory);
>>
>> +static int add_memory_create_devices(int nid, struct memory_group *group,
>> + u64 start, u64 size, mhp_t mhp_flags)
>> +{
>> + struct mhp_params params = { .pgprot = pgprot_mhp(PAGE_KERNEL) };
>> + struct vmem_altmap mhp_altmap = {};
>> + int ret;
>> +
>> + if ((mhp_flags & MHP_MEMMAP_ON_MEMORY)) {
>> + mhp_altmap.free = PHYS_PFN(size);
>> + mhp_altmap.base_pfn = PHYS_PFN(start);
>> + params.altmap = &mhp_altmap;
>> + }
>> +
>> + /* call arch's memory hotadd */
>> + ret = arch_add_memory(nid, start, size, ¶ms);
>> + if (ret < 0)
>> + return ret;
>> +
>> + /* create memory block devices after memory was added */
>> + ret = create_memory_block_devices(start, size, mhp_altmap.alloc,
>> + group);
>> + if (ret) {
>> + arch_remove_memory(start, size, NULL);
>> + return ret;
>> + }
>> +
>> + return 0;
>> +}
>> +
>> /*
>> * NOTE: The caller must call lock_device_hotplug() to serialize hotplug
>> * and online/offline operations (triggered e.g. by sysfs).
>> @@ -1294,11 +1323,10 @@ EXPORT_SYMBOL_GPL(mhp_supports_memmap_on_memory);
>> */
>> int __ref add_memory_resource(int nid, struct resource *res, mhp_t mhp_flags)
>> {
>> - struct mhp_params params = { .pgprot = pgprot_mhp(PAGE_KERNEL) };
>> + unsigned long memblock_size = memory_block_size_bytes();
>> enum memblock_flags memblock_flags = MEMBLOCK_NONE;
>> - struct vmem_altmap mhp_altmap = {};
>> struct memory_group *group = NULL;
>> - u64 start, size;
>> + u64 start, size, cur_start;
>> bool new_node = false;
>> int ret;
>>
>> @@ -1339,27 +1367,20 @@ int __ref add_memory_resource(int nid, struct resource *res, mhp_t mhp_flags)
>> /*
>> * Self hosted memmap array
>> */
>> - if (mhp_flags & MHP_MEMMAP_ON_MEMORY) {
>> - if (!mhp_supports_memmap_on_memory(size)) {
>> - ret = -EINVAL;
>> + if ((mhp_flags & MHP_MEMMAP_ON_MEMORY) &&
>> + mhp_supports_memmap_on_memory(memblock_size)) {
>> + for (cur_start = start; cur_start < start + size;
>> + cur_start += memblock_size) {
>> + ret = add_memory_create_devices(nid, group, cur_start,
>> + memblock_size,
>> + mhp_flags);
>> + if (ret)
>> + goto error;
>> + }
>
> We should handle the below error details here.
>
> 1) If we hit an error after some blocks got added, should we iterate over rest of the dev_dax->nr_range.
> 2) With some blocks added if we return a failure here, we remove the
> resource in dax_kmem. Is that ok?
>
> IMHO error handling with partial creation of memory blocks in a resource range should be
> documented with this change.
Or, should we remove all added memory blocks upon error?
--
Best Regards,
Huang, Ying
next prev parent reply other threads:[~2023-07-24 3:18 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-07-20 7:14 [PATCH v2 0/3] mm: use memmap_on_memory semantics for dax/kmem Vishal Verma
2023-07-20 7:14 ` [PATCH v2 1/3] mm/memory_hotplug: Export symbol mhp_supports_memmap_on_memory() Vishal Verma
2023-07-24 6:00 ` Huang, Ying
2023-07-20 7:14 ` [PATCH v2 2/3] mm/memory_hotplug: split memmap_on_memory requests across memblocks Vishal Verma
2023-07-21 12:20 ` Aneesh Kumar K.V
2023-07-23 14:53 ` Aneesh Kumar K.V
2023-07-24 3:16 ` Huang, Ying [this message]
2023-08-02 6:02 ` Verma, Vishal L
2023-08-14 6:04 ` Huang, Ying
2023-07-24 5:54 ` Huang, Ying
2023-08-02 6:08 ` Verma, Vishal L
2023-08-14 6:45 ` Huang, Ying
2023-08-14 7:20 ` David Hildenbrand
2023-07-20 7:14 ` [PATCH v2 3/3] dax/kmem: allow kmem to add memory with memmap_on_memory Vishal Verma
2023-07-25 15:54 ` [PATCH v2 0/3] mm: use memmap_on_memory semantics for dax/kmem David Hildenbrand
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=87351e2e43.fsf@yhuang6-desk2.ccr.corp.intel.com \
--to=ying.huang@intel.com \
--cc=Jonathan.Cameron@Huawei.com \
--cc=akpm@linux-foundation.org \
--cc=aneesh.kumar@linux.ibm.com \
--cc=dan.j.williams@intel.com \
--cc=dave.hansen@linux.intel.com \
--cc=dave.jiang@intel.com \
--cc=david@redhat.com \
--cc=jmoyer@redhat.com \
--cc=linux-cxl@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=nvdimm@lists.linux.dev \
--cc=osalvador@suse.de \
--cc=vishal.l.verma@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).