From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jiang Liu Subject: Re: [RFC PATCH v3 0/13] memory-hotplug : hot-remove physical memory Date: Wed, 11 Jul 2012 08:21:05 +0800 Message-ID: <4FFCC6F1.5060908@gmail.com> References: <4FFAB0A2.8070304@jp.fujitsu.com> <4FFBFCAC.4010007@jp.fujitsu.com> <4FFC5D43.7040206@gmail.com> <4FFCC438.4080004@jp.fujitsu.com> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Return-path: Received: from mail-gg0-f174.google.com ([209.85.161.174]:54803 "EHLO mail-gg0-f174.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932123Ab2GKAVQ (ORCPT ); Tue, 10 Jul 2012 20:21:16 -0400 In-Reply-To: <4FFCC438.4080004@jp.fujitsu.com> Sender: linux-acpi-owner@vger.kernel.org List-Id: linux-acpi@vger.kernel.org To: Yasuaki Ishimatsu Cc: Christoph Lameter , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-acpi@vger.kernel.org, rientjes@google.com, len.brown@intel.com, benh@kernel.crashing.org, paulus@samba.org, minchan.kim@gmail.com, akpm@linux-foundation.org, kosaki.motohiro@jp.fujitsu.com, wency@cn.fujitsu.com On 07/11/2012 08:09 AM, Yasuaki Ishimatsu wrote: > Hi Jiang, > > 2012/07/11 1:50, Jiang Liu wrote: >> On 07/10/2012 05:58 PM, Yasuaki Ishimatsu wrote: >>> Hi Christoph, >>> >>> 2012/07/10 0:18, Christoph Lameter wrote: >>>> >>>> On Mon, 9 Jul 2012, Yasuaki Ishimatsu wrote: >>>> >>>>> Even if you apply these patches, you cannot remove the physical memory >>>>> completely since these patches are still under development. I want you to >>>>> cooperate to improve the physical memory hot-remove. So please review these >>>>> patches and give your comment/idea. >>>> >>>> Could you at least give a method on how you want to do physical memory >>>> removal? >>> >>> We plan to release a dynamic hardware partitionable system. It will be >>> able to hot remove/add a system board which included memory and cpu. >>> But as you know, Linux does not support memory hot-remove on x86 box. >>> So I try to develop it. >>> >>> Current plan to hot remove system board is to use container driver. >>> Thus I define the system board in ACPI DSDT table as a container device. >>> It have supported hot-add a container device. And if container device >>> has _EJ0 ACPI method, "eject" file to remove the container device is >>> prepared as follow: >>> >>> # ls -l /sys/bus/acpi/devices/ACPI0004\:01/eject >>> --w-------. 1 root root 4096 Jul 10 18:19 /sys/bus/acpi/devices/ACPI0004:01/eject >>> >>> When I hot-remove the container device, I echo 1 to the file as follow: >>> >>> #echo 1 > /sys/bus/acpi/devices/ACPI0004\:02/eject >>> >>> Then acpi_bus_trim() is called. And it calls acpi_memory_device_remove() >>> for removing memory device. But the code does not do nothing. >>> So I developed the continuation of the function. >>> >>>> You would have to remove all objects from the range you want to >>>> physically remove. That is only possible under special circumstances and >>>> with a limited set of objects. Even if you exclusively use ZONE_MOVEABLE >>>> you still may get cases where pages are pinned for a long time. >>> >>> I know it. So my memory hot-remove plan is as follows: >>> >>> 1. hot-added a system board >>> All memory which included the system board is offline. >>> >>> 2. online the memory as removable page >>> The function has not supported yet. It is being developed by Lai as follow: >>> http://lkml.indiana.edu/hypermail/linux/kernel/1207.0/01478.html >>> If it is supported, I will be able to create movable memory. >>> >>> 3. hot-remove the memory by container device's eject file >> We have implemented a prototype to do physical node (mem + CPU + IOH) hotplug >> for Itanium and is now porting it to x86. But with currently solution, memory >> hotplug functionality may cause 10-20% performance decrease because we concentrate >> all DMA/Normal memory to the first NUMA node, and all other NUMA nodes only >> hosts ZONE_MOVABLE. We are working on solution to minimize the performance >> drop now. > > Thank you for your interesting response. > > I have a question. How do you move all other NUMA nodes to ZONE_MOVABLE? > To use ZONE_MOVABLE, we need to use boot options like kernelcore or movablecore. > But it is not enough, since the requested amount is spread evenly throughout > all nodes in the system. So I think we do not have way to move all other NUMA > node to ZONE_MOVABLE. We have modified the ZONE_MOVABLE spreading and bootmem allocation. If the kernelcore or movablecore kernel parameters are present, we follow current behavior. If those parameter are absent and the platform supports physical hotplug, we will concentrate DMA/NORMAL memory to specific nodes. > > Thanks, > Yasuaki Ishimatsu > >> >>> >>> Thanks, >>> Yasuaki Ishimatsu >>> >>>> >>>> I am not sure that these patches are useful unless we know where you are >>>> going with this. If we end up with a situation where we still cannot >>>> remove physical memory then this patchset is not helpful. >>> >>> >>> >> >> > > > From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx205.postini.com [74.125.245.205]) by kanga.kvack.org (Postfix) with SMTP id C130D6B0073 for ; Tue, 10 Jul 2012 20:21:16 -0400 (EDT) Received: by yenr5 with SMTP id r5so743723yen.14 for ; Tue, 10 Jul 2012 17:21:15 -0700 (PDT) Message-ID: <4FFCC6F1.5060908@gmail.com> Date: Wed, 11 Jul 2012 08:21:05 +0800 From: Jiang Liu MIME-Version: 1.0 Subject: Re: [RFC PATCH v3 0/13] memory-hotplug : hot-remove physical memory References: <4FFAB0A2.8070304@jp.fujitsu.com> <4FFBFCAC.4010007@jp.fujitsu.com> <4FFC5D43.7040206@gmail.com> <4FFCC438.4080004@jp.fujitsu.com> In-Reply-To: <4FFCC438.4080004@jp.fujitsu.com> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: Yasuaki Ishimatsu Cc: Christoph Lameter , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-acpi@vger.kernel.org, rientjes@google.com, len.brown@intel.com, benh@kernel.crashing.org, paulus@samba.org, minchan.kim@gmail.com, akpm@linux-foundation.org, kosaki.motohiro@jp.fujitsu.com, wency@cn.fujitsu.com On 07/11/2012 08:09 AM, Yasuaki Ishimatsu wrote: > Hi Jiang, > > 2012/07/11 1:50, Jiang Liu wrote: >> On 07/10/2012 05:58 PM, Yasuaki Ishimatsu wrote: >>> Hi Christoph, >>> >>> 2012/07/10 0:18, Christoph Lameter wrote: >>>> >>>> On Mon, 9 Jul 2012, Yasuaki Ishimatsu wrote: >>>> >>>>> Even if you apply these patches, you cannot remove the physical memory >>>>> completely since these patches are still under development. I want you to >>>>> cooperate to improve the physical memory hot-remove. So please review these >>>>> patches and give your comment/idea. >>>> >>>> Could you at least give a method on how you want to do physical memory >>>> removal? >>> >>> We plan to release a dynamic hardware partitionable system. It will be >>> able to hot remove/add a system board which included memory and cpu. >>> But as you know, Linux does not support memory hot-remove on x86 box. >>> So I try to develop it. >>> >>> Current plan to hot remove system board is to use container driver. >>> Thus I define the system board in ACPI DSDT table as a container device. >>> It have supported hot-add a container device. And if container device >>> has _EJ0 ACPI method, "eject" file to remove the container device is >>> prepared as follow: >>> >>> # ls -l /sys/bus/acpi/devices/ACPI0004\:01/eject >>> --w-------. 1 root root 4096 Jul 10 18:19 /sys/bus/acpi/devices/ACPI0004:01/eject >>> >>> When I hot-remove the container device, I echo 1 to the file as follow: >>> >>> #echo 1 > /sys/bus/acpi/devices/ACPI0004\:02/eject >>> >>> Then acpi_bus_trim() is called. And it calls acpi_memory_device_remove() >>> for removing memory device. But the code does not do nothing. >>> So I developed the continuation of the function. >>> >>>> You would have to remove all objects from the range you want to >>>> physically remove. That is only possible under special circumstances and >>>> with a limited set of objects. Even if you exclusively use ZONE_MOVEABLE >>>> you still may get cases where pages are pinned for a long time. >>> >>> I know it. So my memory hot-remove plan is as follows: >>> >>> 1. hot-added a system board >>> All memory which included the system board is offline. >>> >>> 2. online the memory as removable page >>> The function has not supported yet. It is being developed by Lai as follow: >>> http://lkml.indiana.edu/hypermail/linux/kernel/1207.0/01478.html >>> If it is supported, I will be able to create movable memory. >>> >>> 3. hot-remove the memory by container device's eject file >> We have implemented a prototype to do physical node (mem + CPU + IOH) hotplug >> for Itanium and is now porting it to x86. But with currently solution, memory >> hotplug functionality may cause 10-20% performance decrease because we concentrate >> all DMA/Normal memory to the first NUMA node, and all other NUMA nodes only >> hosts ZONE_MOVABLE. We are working on solution to minimize the performance >> drop now. > > Thank you for your interesting response. > > I have a question. How do you move all other NUMA nodes to ZONE_MOVABLE? > To use ZONE_MOVABLE, we need to use boot options like kernelcore or movablecore. > But it is not enough, since the requested amount is spread evenly throughout > all nodes in the system. So I think we do not have way to move all other NUMA > node to ZONE_MOVABLE. We have modified the ZONE_MOVABLE spreading and bootmem allocation. If the kernelcore or movablecore kernel parameters are present, we follow current behavior. If those parameter are absent and the platform supports physical hotplug, we will concentrate DMA/NORMAL memory to specific nodes. > > Thanks, > Yasuaki Ishimatsu > >> >>> >>> Thanks, >>> Yasuaki Ishimatsu >>> >>>> >>>> I am not sure that these patches are useful unless we know where you are >>>> going with this. If we end up with a situation where we still cannot >>>> remove physical memory then this patchset is not helpful. >>> >>> >>> >> >> > > > -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-yw0-f43.google.com (mail-yw0-f43.google.com [209.85.213.43]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority" (not verified)) by ozlabs.org (Postfix) with ESMTPS id 65E652C0200 for ; Wed, 11 Jul 2012 10:21:18 +1000 (EST) Received: by yhl10 with SMTP id 10so1157637yhl.16 for ; Tue, 10 Jul 2012 17:21:15 -0700 (PDT) Message-ID: <4FFCC6F1.5060908@gmail.com> Date: Wed, 11 Jul 2012 08:21:05 +0800 From: Jiang Liu MIME-Version: 1.0 To: Yasuaki Ishimatsu Subject: Re: [RFC PATCH v3 0/13] memory-hotplug : hot-remove physical memory References: <4FFAB0A2.8070304@jp.fujitsu.com> <4FFBFCAC.4010007@jp.fujitsu.com> <4FFC5D43.7040206@gmail.com> <4FFCC438.4080004@jp.fujitsu.com> In-Reply-To: <4FFCC438.4080004@jp.fujitsu.com> Content-Type: text/plain; charset=ISO-8859-1 Cc: len.brown@intel.com, wency@cn.fujitsu.com, linux-acpi@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, paulus@samba.org, minchan.kim@gmail.com, kosaki.motohiro@jp.fujitsu.com, rientjes@google.com, Christoph Lameter , linuxppc-dev@lists.ozlabs.org, akpm@linux-foundation.org List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , On 07/11/2012 08:09 AM, Yasuaki Ishimatsu wrote: > Hi Jiang, > > 2012/07/11 1:50, Jiang Liu wrote: >> On 07/10/2012 05:58 PM, Yasuaki Ishimatsu wrote: >>> Hi Christoph, >>> >>> 2012/07/10 0:18, Christoph Lameter wrote: >>>> >>>> On Mon, 9 Jul 2012, Yasuaki Ishimatsu wrote: >>>> >>>>> Even if you apply these patches, you cannot remove the physical memory >>>>> completely since these patches are still under development. I want you to >>>>> cooperate to improve the physical memory hot-remove. So please review these >>>>> patches and give your comment/idea. >>>> >>>> Could you at least give a method on how you want to do physical memory >>>> removal? >>> >>> We plan to release a dynamic hardware partitionable system. It will be >>> able to hot remove/add a system board which included memory and cpu. >>> But as you know, Linux does not support memory hot-remove on x86 box. >>> So I try to develop it. >>> >>> Current plan to hot remove system board is to use container driver. >>> Thus I define the system board in ACPI DSDT table as a container device. >>> It have supported hot-add a container device. And if container device >>> has _EJ0 ACPI method, "eject" file to remove the container device is >>> prepared as follow: >>> >>> # ls -l /sys/bus/acpi/devices/ACPI0004\:01/eject >>> --w-------. 1 root root 4096 Jul 10 18:19 /sys/bus/acpi/devices/ACPI0004:01/eject >>> >>> When I hot-remove the container device, I echo 1 to the file as follow: >>> >>> #echo 1 > /sys/bus/acpi/devices/ACPI0004\:02/eject >>> >>> Then acpi_bus_trim() is called. And it calls acpi_memory_device_remove() >>> for removing memory device. But the code does not do nothing. >>> So I developed the continuation of the function. >>> >>>> You would have to remove all objects from the range you want to >>>> physically remove. That is only possible under special circumstances and >>>> with a limited set of objects. Even if you exclusively use ZONE_MOVEABLE >>>> you still may get cases where pages are pinned for a long time. >>> >>> I know it. So my memory hot-remove plan is as follows: >>> >>> 1. hot-added a system board >>> All memory which included the system board is offline. >>> >>> 2. online the memory as removable page >>> The function has not supported yet. It is being developed by Lai as follow: >>> http://lkml.indiana.edu/hypermail/linux/kernel/1207.0/01478.html >>> If it is supported, I will be able to create movable memory. >>> >>> 3. hot-remove the memory by container device's eject file >> We have implemented a prototype to do physical node (mem + CPU + IOH) hotplug >> for Itanium and is now porting it to x86. But with currently solution, memory >> hotplug functionality may cause 10-20% performance decrease because we concentrate >> all DMA/Normal memory to the first NUMA node, and all other NUMA nodes only >> hosts ZONE_MOVABLE. We are working on solution to minimize the performance >> drop now. > > Thank you for your interesting response. > > I have a question. How do you move all other NUMA nodes to ZONE_MOVABLE? > To use ZONE_MOVABLE, we need to use boot options like kernelcore or movablecore. > But it is not enough, since the requested amount is spread evenly throughout > all nodes in the system. So I think we do not have way to move all other NUMA > node to ZONE_MOVABLE. We have modified the ZONE_MOVABLE spreading and bootmem allocation. If the kernelcore or movablecore kernel parameters are present, we follow current behavior. If those parameter are absent and the platform supports physical hotplug, we will concentrate DMA/NORMAL memory to specific nodes. > > Thanks, > Yasuaki Ishimatsu > >> >>> >>> Thanks, >>> Yasuaki Ishimatsu >>> >>>> >>>> I am not sure that these patches are useful unless we know where you are >>>> going with this. If we end up with a situation where we still cannot >>>> remove physical memory then this patchset is not helpful. >>> >>> >>> >> >> > > >