From: Tang Chen <tangchen@cn.fujitsu.com>
To: Glauber Costa <glommer@parallels.com>
Cc: linux-ia64@vger.kernel.org, linux-sh@vger.kernel.org,
linux-mm@kvack.org, paulus@samba.org, hpa@zytor.com,
sparclinux@vger.kernel.org, cl@linux.com,
linux-s390@vger.kernel.org, x86@kernel.org,
linux-acpi@vger.kernel.org, isimatu.yasuaki@jp.fujitsu.com,
linfeng@cn.fujitsu.com, mgorman@suse.de,
kosaki.motohiro@jp.fujitsu.com, rientjes@google.com,
liuj97@gmail.com, len.brown@intel.com,
Miao Xie <miaox@cn.fujitsu.com>,
Wen Congyang <wency@cn.fujitsu.com>,
cmetcalf@tilera.com, wujianguo@huawei.com, yinghai@kernel.org,
KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>,
laijs@cn.fujitsu.com, linux-kernel@vger.kernel.org,
minchan.kim@gmail.com, akpm@linux-foundation.org,
linuxppc-dev@lists.ozlabs.org
Subject: Re: [PATCH v5 01/14] memory-hotplug: try to offline the memory twice to avoid dependence
Date: Wed, 06 Feb 2013 18:10:37 +0800 [thread overview]
Message-ID: <51122C1D.5020002@cn.fujitsu.com> (raw)
In-Reply-To: <51121FB7.1070205@cn.fujitsu.com>
On 02/06/2013 05:17 PM, Tang Chen wrote:
> Hi all,
>
> On 02/06/2013 11:07 AM, Tang Chen wrote:
>> Hi Glauber, all,
>>
>> An old thing I want to discuss with you. :)
>>
>> On 01/09/2013 11:09 PM, Glauber Costa wrote:
>>>>>> memory can't be offlined when CONFIG_MEMCG is selected.
>>>>>> For example: there is a memory device on node 1. The address range
>>>>>> is [1G, 1.5G). You will find 4 new directories memory8, memory9,
>>>>>> memory10,
>>>>>> and memory11 under the directory /sys/devices/system/memory/.
>>>>>>
>>>>>> If CONFIG_MEMCG is selected, we will allocate memory to store page
>>>>>> cgroup
>>>>>> when we online pages. When we online memory8, the memory stored
>>>>>> page cgroup
>>>>>> is not provided by this memory device. But when we online memory9,
>>>>>> the memory
>>>>>> stored page cgroup may be provided by memory8. So we can't offline
>>>>>> memory8
>>>>>> now. We should offline the memory in the reversed order.
>>>>>>
>>>>>> When the memory device is hotremoved, we will auto offline memory
>>>>>> provided
>>>>>> by this memory device. But we don't know which memory is onlined
>>>>>> first, so
>>>>>> offlining memory may fail. In such case, iterate twice to offline
>>>>>> the memory.
>>>>>> 1st iterate: offline every non primary memory block.
>>>>>> 2nd iterate: offline primary (i.e. first added) memory block.
>>>>>>
>>>>>> This idea is suggested by KOSAKI Motohiro.
>>>>>>
>>>>>> Signed-off-by: Wen Congyang<wency@cn.fujitsu.com>
>>>>>
>>>>> Maybe there is something here that I am missing - I admit that I came
>>>>> late to this one, but this really sounds like a very ugly hack, that
>>>>> really has no place in here.
>>>>>
>>>>> Retrying, of course, may make sense, if we have reasonable belief that
>>>>> we may now succeed. If this is the case, you need to document - in the
>>>>> code - while is that.
>>>>>
>>>>> The memcg argument, however, doesn't really cut it. Why can't we make
>>>>> all page_cgroup allocations local to the node they are describing? If
>>>>> memcg is the culprit here, we should fix it, and not retry. If
>>>>> there is
>>>>> still any benefit in retrying, then we retry being very specific
>>>>> about why.
>>>>
>>>> We try to make all page_cgroup allocations local to the node they are
>>>> describing
>>>> now. If the memory is the first memory onlined in this node, we will
>>>> allocate
>>>> it from the other node.
>>>>
>>>> For example, node1 has 4 memory blocks: 8-11, and we online it from 8
>>>> to 11
>>>> 1. memory block 8, page_cgroup allocations are in the other nodes
>>>> 2. memory block 9, page_cgroup allocations are in memory block 8
>>>>
>>>> So we should offline memory block 9 first. But we don't know in which
>>>> order
>>>> the user online the memory block.
>>>>
>>>> I think we can modify memcg like this:
>>>> allocate the memory from the memory block they are describing
>>>>
>>>> I am not sure it is OK to do so.
>>>
>>> I don't see a reason why not.
>>>
>>> You would have to tweak a bit the lookup function for page_cgroup, but
>>> assuming you will always have the pfns and limits, it should be easy
>>> to do.
>>>
>>> I think the only tricky part is that today we have a single
>>> node_page_cgroup, and we would of course have to have one per memory
>>> block. My assumption is that the number of memory blocks is limited and
>>> likely not very big. So even a static array would do.
>>>
>>
>> About the idea "allocate the memory from the memory block they are
>> describing",
>>
>> online_pages()
>> |-->memory_notify(MEM_GOING_ONLINE, &arg) ----------- memory of this
>> section is not in buddy yet.
>> |-->page_cgroup_callback()
>> |-->online_page_cgroup()
>> |-->init_section_page_cgroup()
>> |-->alloc_page_cgroup() --------- allocate page_cgroup from buddy system.
>>
>> When onlining pages, we allocate page_cgroup from buddy. And the being
>> onlined pages are not in
>> buddy yet. I think we can reserve some memory in the section for
>> page_cgroup, and return all the
>> rest to the buddy.
>>
>> But when the system is booting,
>>
>> start_kernel()
>> |-->setup_arch()
>> |-->mm_init()
>> | |-->mem_init()
>> | |-->numa_free_all_bootmem() -------------- all the pages are in buddy
>> system.
>> |-->page_cgroup_init()
>> |-->init_section_page_cgroup()
>> |-->alloc_page_cgroup() ------------------ I don't know how to reserve
>> memory in each section.
>>
>> So any idea about how to deal with it when the system is booting please?
>>
>
> How about this way.
>
> 1) Add a new flag PAGE_CGROUP_INFO, like SECTION_INFO and MIX_SECTION_INFO.
> 2) In sparse_init(), reserve some beginning pages of each section as
> bootmem.
Hi all,
After digging into bootmem code, I met another problem.
memblock allocates memory from high address to low address, using
memblock.current_limit
to remember where the upper limit is. What I am doing will produce a lot
of fragments,
and the memory will be non-contiguous. So we need to modify memblock again.
I don't think it's a good idea. How do you think ?
Thanks. :)
> 3) In register_page_bootmem_info_section(), set these pages as
> page->lru.next = PAGE_CGROUP_INFO;
>
> Then these pages will not go to buddy system.
>
> But I do worry about the fragment problem because part of each section will
> be used in the very beginning.
>
> Thanks. :)
>
>>
>> And one more question, a memory section is 128MB in Linux. If we reserve
>> part of the them for page_cgroup,
>> then anyone who wants to allocate a contiguous memory larger than 128MB,
>> it will fail, right ?
>> Is it OK ?
>>
>> Thanks. :)
>>
>>
>>
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-acpi" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-acpi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
next prev parent reply other threads:[~2013-02-06 10:11 UTC|newest]
Thread overview: 47+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-12-24 12:09 [PATCH v5 00/14] memory-hotplug: hot-remove physical memory Tang Chen
2012-12-24 12:09 ` [PATCH v5 01/14] memory-hotplug: try to offline the memory twice to avoid dependence Tang Chen
2012-12-25 8:35 ` Glauber Costa
2012-12-30 5:58 ` Wen Congyang
2013-01-09 15:09 ` Glauber Costa
2013-01-10 1:38 ` Tang Chen
2013-02-06 3:07 ` Tang Chen
2013-02-06 9:17 ` Tang Chen
2013-02-06 10:10 ` Tang Chen [this message]
2013-02-06 14:24 ` Glauber Costa
2013-02-07 7:56 ` Tang Chen
2012-12-26 3:02 ` Kamezawa Hiroyuki
2012-12-30 5:49 ` Wen Congyang
2012-12-24 12:09 ` [PATCH v5 02/14] memory-hotplug: check whether all memory blocks are offlined or not when removing memory Tang Chen
2012-12-26 3:10 ` Kamezawa Hiroyuki
2012-12-27 3:10 ` Tang Chen
2012-12-24 12:09 ` [PATCH v5 03/14] memory-hotplug: remove redundant codes Tang Chen
2012-12-26 3:20 ` Kamezawa Hiroyuki
2012-12-27 3:09 ` Tang Chen
2012-12-24 12:09 ` [PATCH v5 04/14] memory-hotplug: remove /sys/firmware/memmap/X sysfs Tang Chen
2012-12-26 3:30 ` Kamezawa Hiroyuki
2012-12-27 3:09 ` Tang Chen
2013-01-02 14:24 ` Christoph Lameter
2012-12-24 12:09 ` [PATCH v5 05/14] memory-hotplug: introduce new function arch_remove_memory() for removing page table depends on architecture Tang Chen
2012-12-26 3:37 ` Kamezawa Hiroyuki
2012-12-24 12:09 ` [PATCH v5 06/14] memory-hotplug: implement register_page_bootmem_info_section of sparse-vmemmap Tang Chen
2012-12-25 8:09 ` Jianguo Wu
2012-12-26 3:21 ` Tang Chen
2012-12-24 12:09 ` [PATCH v5 07/14] memory-hotplug: move pgdat_resize_lock into sparse_remove_one_section() Tang Chen
2012-12-26 3:47 ` Kamezawa Hiroyuki
2012-12-26 6:20 ` Tang Chen
2012-12-24 12:09 ` [PATCH v5 08/14] memory-hotplug: Common APIs to support page tables hot-remove Tang Chen
2012-12-25 8:17 ` Jianguo Wu
2012-12-26 2:49 ` Tang Chen
2012-12-26 3:11 ` Tang Chen
2012-12-26 3:19 ` Tang Chen
2012-12-24 12:09 ` [PATCH v5 09/14] memory-hotplug: remove page table of x86_64 architecture Tang Chen
2012-12-24 12:09 ` [PATCH v5 10/14] memory-hotplug: remove memmap of sparse-vmemmap Tang Chen
2012-12-24 12:09 ` [PATCH v5 11/14] memory-hotplug: Integrated __remove_section() of CONFIG_SPARSEMEM_VMEMMAP Tang Chen
2012-12-24 12:09 ` [PATCH v5 12/14] memory-hotplug: memory_hotplug: clear zone when removing the memory Tang Chen
2012-12-24 12:09 ` [PATCH v5 13/14] memory-hotplug: remove sysfs file of node Tang Chen
2012-12-24 12:09 ` [PATCH v5 14/14] memory-hotplug: free node_data when a node is offlined Tang Chen
2012-12-26 3:55 ` Kamezawa Hiroyuki
2012-12-27 12:16 ` Wen Congyang
2012-12-28 0:28 ` Kamezawa Hiroyuki
2012-12-30 6:02 ` Wen Congyang
2013-01-07 5:30 ` Kamezawa Hiroyuki
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=51122C1D.5020002@cn.fujitsu.com \
--to=tangchen@cn.fujitsu.com \
--cc=akpm@linux-foundation.org \
--cc=cl@linux.com \
--cc=cmetcalf@tilera.com \
--cc=glommer@parallels.com \
--cc=hpa@zytor.com \
--cc=isimatu.yasuaki@jp.fujitsu.com \
--cc=kamezawa.hiroyu@jp.fujitsu.com \
--cc=kosaki.motohiro@jp.fujitsu.com \
--cc=laijs@cn.fujitsu.com \
--cc=len.brown@intel.com \
--cc=linfeng@cn.fujitsu.com \
--cc=linux-acpi@vger.kernel.org \
--cc=linux-ia64@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=linux-s390@vger.kernel.org \
--cc=linux-sh@vger.kernel.org \
--cc=linuxppc-dev@lists.ozlabs.org \
--cc=liuj97@gmail.com \
--cc=mgorman@suse.de \
--cc=miaox@cn.fujitsu.com \
--cc=minchan.kim@gmail.com \
--cc=paulus@samba.org \
--cc=rientjes@google.com \
--cc=sparclinux@vger.kernel.org \
--cc=wency@cn.fujitsu.com \
--cc=wujianguo@huawei.com \
--cc=x86@kernel.org \
--cc=yinghai@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).