From: Kamezawa Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
To: Glauber Costa <glommer@parallels.com>
Cc: linux-ia64@vger.kernel.org, linux-sh@vger.kernel.org,
Tang Chen <tangchen@cn.fujitsu.com>,
linux-mm@kvack.org, paulus@samba.org, hpa@zytor.com,
sparclinux@vger.kernel.org, cl@linux.com,
linux-s390@vger.kernel.org, x86@kernel.org,
linux-acpi@vger.kernel.org, isimatu.yasuaki@jp.fujitsu.com,
linfeng@cn.fujitsu.com, mgorman@suse.de,
kosaki.motohiro@jp.fujitsu.com, rientjes@google.com,
len.brown@intel.com, wency@cn.fujitsu.com, cmetcalf@tilera.com,
wujianguo@huawei.com, yinghai@kernel.org, laijs@cn.fujitsu.com,
linux-kernel@vger.kernel.org, minchan.kim@gmail.com,
Andrew Morton <akpm@linux-foundation.org>,
linuxppc-dev@lists.ozlabs.org
Subject: Re: [PATCH v6 00/15] memory-hotplug: hot-remove physical memory
Date: Thu, 10 Jan 2013 17:23:07 +0900 [thread overview]
Message-ID: <50EE7A6B.7020005@jp.fujitsu.com> (raw)
In-Reply-To: <50EE73DE.30208@parallels.com>
(2013/01/10 16:55), Glauber Costa wrote:
> On 01/10/2013 11:31 AM, Kamezawa Hiroyuki wrote:
>> (2013/01/10 16:14), Glauber Costa wrote:
>>> On 01/10/2013 06:17 AM, Tang Chen wrote:
>>>>>> Note: if the memory provided by the memory device is used by the
>>>>>> kernel, it
>>>>>> can't be offlined. It is not a bug.
>>>>>
>>>>> Right. But how often does this happen in testing? In other words,
>>>>> please provide an overall description of how well memory hot-remove is
>>>>> presently operating. Is it reliable? What is the success rate in
>>>>> real-world situations?
>>>>
>>>> We test the hot-remove functionality mostly with movable_online used.
>>>> And the memory used by kernel is not allowed to be removed.
>>>
>>> Can you try doing this using cpusets configured to hardwall ?
>>> It is my understanding that the object allocators will try hard not to
>>> allocate anything outside the walls defined by cpuset. Which means that
>>> if you have one process per node, and they are hardwalled, your kernel
>>> memory will be spread evenly among the machine. With a big enough load,
>>> they should eventually be present in all blocks.
>>>
>>
>> I'm sorry I couldn't catch your point.
>> Do you want to confirm whether cpuset can work enough instead of
>> ZONE_MOVABLE ?
>> Or Do you want to confirm whether ZONE_MOVABLE will not work if it's
>> used with cpuset ?
>>
>>
> No, I am not proposing to use cpuset do tackle the problem. I am just
> wondering if you would still have high success rates with cpusets in use
> with hardwalls. This is just one example of a workload that would spread
> kernel memory around quite heavily.
>
> So this is just me trying to understand the limitations of the mechanism.
>
Hm, okay. In my undestanding, if the whole memory of a node is configured as
MOVABLE, no kernel memory will not be allocated in the node because zonelist
will not match. So, if cpuset is used with hardwalls, user will see -ENOMEM or OOM,
I guess. even fork() will fail if fallback-to-other-node is not allowed.
If it's configure as ZONE_NORMAL, you need to pray for offlining memory.
AFAIK, IBM's ppc? has 16MB section size. So, some of sections can be offlined
even if they are configured as ZONE_NORMAL. For them, placement of offlined
memory is not important because it's virtualized by LPAR, they don't try
to remove DIMM, they just want to increase/decrease amount of memory.
It's an another approach.
But here, we(fujitsu) tries to remove a system board/DIMM.
So, configuring the whole memory of a node as ZONE_MOVABLE and tries to guarantee
DIMM as removable.
>> IMHO, I don't think shrink_slab() can kill all objects in a node even
>> if they are some caches. We need more study for doing that.
>>
>
> Indeed, shrink_slab can only kill cached objects. They, however, are
> usually a very big part of kernel memory. I wonder though if in case of
> failure, it is worth it to try at least one shrink pass before you give up.
>
Yeah, now, his (our) approach is never allowing kernel memory on a node to be
hot-removed by ZONE_MOVABLE. So, shrink_slab()'s effect will not be seen.
If other brave guys tries to use ZONE_NORMAL for hot-pluggable DIMM, I see,
it's worth triying.
How about checking the target memsection is in NORMAL or in MOVABLE at
hot-removing ? If NORMAL, shrink_slab() will be worth to be called.
BTW, shrink_slab() is now node/zone aware ? If not, fixing that first will
be better direction I guess.
Thanks,
-Kame
next prev parent reply other threads:[~2013-01-10 8:24 UTC|newest]
Thread overview: 67+ messages / expand[flat|nested] mbox.gz Atom feed top
2013-01-09 9:32 [PATCH v6 00/15] memory-hotplug: hot-remove physical memory Tang Chen
2013-01-09 9:32 ` [PATCH v6 01/15] memory-hotplug: try to offline the memory twice to avoid dependence Tang Chen
2013-01-09 9:32 ` [PATCH v6 02/15] memory-hotplug: check whether all memory blocks are offlined or not when removing memory Tang Chen
2013-01-09 23:11 ` Andrew Morton
2013-01-10 5:56 ` Tang Chen
2013-01-09 9:32 ` [PATCH v6 03/15] memory-hotplug: remove redundant codes Tang Chen
2013-01-09 9:32 ` [PATCH v6 04/15] memory-hotplug: remove /sys/firmware/memmap/X sysfs Tang Chen
2013-01-09 22:49 ` Andrew Morton
2013-01-10 6:07 ` Tang Chen
2013-01-09 23:19 ` Andrew Morton
2013-01-10 6:15 ` Tang Chen
2013-01-09 9:32 ` [PATCH v6 05/15] memory-hotplug: introduce new function arch_remove_memory() for removing page table depends on architecture Tang Chen
2013-01-09 22:50 ` Andrew Morton
2013-01-10 2:25 ` Tang Chen
2013-01-09 9:32 ` [PATCH v6 06/15] memory-hotplug: implement register_page_bootmem_info_section of sparse-vmemmap Tang Chen
2013-01-09 9:32 ` [PATCH v6 07/15] memory-hotplug: move pgdat_resize_lock into sparse_remove_one_section() Tang Chen
2013-01-09 9:32 ` [PATCH v6 08/15] memory-hotplug: Common APIs to support page tables hot-remove Tang Chen
2013-01-29 13:02 ` Simon Jeons
2013-01-30 1:53 ` Jianguo Wu
2013-01-30 2:13 ` Simon Jeons
2013-01-29 13:04 ` Simon Jeons
2013-01-30 2:16 ` Tang Chen
2013-01-30 3:27 ` Simon Jeons
2013-01-30 5:55 ` Tang Chen
2013-01-30 7:32 ` Simon Jeons
2013-02-04 23:04 ` Andrew Morton
2013-01-09 9:32 ` [PATCH v6 09/15] memory-hotplug: remove page table of x86_64 architecture Tang Chen
2013-01-09 9:32 ` [PATCH v6 10/15] memory-hotplug: remove memmap of sparse-vmemmap Tang Chen
2013-01-09 9:32 ` [PATCH v6 11/15] memory-hotplug: Integrated __remove_section() of CONFIG_SPARSEMEM_VMEMMAP Tang Chen
2013-01-09 9:32 ` [PATCH v6 12/15] memory-hotplug: memory_hotplug: clear zone when removing the memory Tang Chen
2013-01-09 9:32 ` [PATCH v6 13/15] memory-hotplug: remove sysfs file of node Tang Chen
2013-01-09 9:32 ` [PATCH v6 14/15] memory-hotplug: free node_data when a node is offlined Tang Chen
2013-01-09 9:32 ` [PATCH v6 15/15] memory-hotplug: Do not allocate pdgat if it was not freed when offline Tang Chen
2013-01-09 22:23 ` [PATCH v6 00/15] memory-hotplug: hot-remove physical memory Andrew Morton
2013-01-10 2:17 ` Tang Chen
2013-01-10 7:14 ` Glauber Costa
2013-01-10 7:31 ` Kamezawa Hiroyuki
2013-01-10 7:55 ` Glauber Costa
2013-01-10 8:23 ` Kamezawa Hiroyuki [this message]
2013-01-10 8:36 ` Glauber Costa
2013-01-10 8:39 ` Kamezawa Hiroyuki
2013-01-09 23:33 ` Andrew Morton
2013-01-10 2:18 ` Tang Chen
2013-01-29 12:52 ` Simon Jeons
2013-01-30 2:32 ` Tang Chen
2013-01-30 2:48 ` Simon Jeons
2013-01-30 3:00 ` Tang Chen
2013-01-30 10:15 ` Tang Chen
2013-01-30 10:18 ` Tang Chen
2013-01-31 1:22 ` Simon Jeons
2013-01-31 3:31 ` Tang Chen
2013-01-31 6:19 ` Simon Jeons
2013-01-31 7:10 ` Tang Chen
2013-01-31 8:17 ` Simon Jeons
2013-01-31 8:48 ` Simon Jeons
2013-01-31 9:44 ` Tang Chen
2013-01-31 10:38 ` Simon Jeons
2013-02-01 1:32 ` Jianguo Wu
2013-02-01 1:36 ` Simon Jeons
2013-02-01 1:57 ` Jianguo Wu
2013-02-01 2:06 ` Simon Jeons
2013-02-01 2:18 ` Jianguo Wu
2013-02-01 1:57 ` Tang Chen
2013-02-01 2:17 ` Simon Jeons
2013-02-01 2:42 ` Tang Chen
2013-02-01 3:06 ` Simon Jeons
2013-02-01 3:39 ` Tang Chen
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=50EE7A6B.7020005@jp.fujitsu.com \
--to=kamezawa.hiroyu@jp.fujitsu.com \
--cc=akpm@linux-foundation.org \
--cc=cl@linux.com \
--cc=cmetcalf@tilera.com \
--cc=glommer@parallels.com \
--cc=hpa@zytor.com \
--cc=isimatu.yasuaki@jp.fujitsu.com \
--cc=kosaki.motohiro@jp.fujitsu.com \
--cc=laijs@cn.fujitsu.com \
--cc=len.brown@intel.com \
--cc=linfeng@cn.fujitsu.com \
--cc=linux-acpi@vger.kernel.org \
--cc=linux-ia64@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=linux-s390@vger.kernel.org \
--cc=linux-sh@vger.kernel.org \
--cc=linuxppc-dev@lists.ozlabs.org \
--cc=mgorman@suse.de \
--cc=minchan.kim@gmail.com \
--cc=paulus@samba.org \
--cc=rientjes@google.com \
--cc=sparclinux@vger.kernel.org \
--cc=tangchen@cn.fujitsu.com \
--cc=wency@cn.fujitsu.com \
--cc=wujianguo@huawei.com \
--cc=x86@kernel.org \
--cc=yinghai@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).