From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jiang Liu Subject: Re: [RFC PATCH v3 0/13] memory-hotplug : hot-remove physical memory Date: Wed, 11 Jul 2012 00:50:11 +0800 Message-ID: <4FFC5D43.7040206@gmail.com> References: <4FFAB0A2.8070304@jp.fujitsu.com> <4FFBFCAC.4010007@jp.fujitsu.com> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Return-path: Received: from mail-gg0-f174.google.com ([209.85.161.174]:61452 "EHLO mail-gg0-f174.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754038Ab2GJQu1 (ORCPT ); Tue, 10 Jul 2012 12:50:27 -0400 In-Reply-To: <4FFBFCAC.4010007@jp.fujitsu.com> Sender: linux-acpi-owner@vger.kernel.org List-Id: linux-acpi@vger.kernel.org To: Yasuaki Ishimatsu Cc: Christoph Lameter , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-acpi@vger.kernel.org, rientjes@google.com, len.brown@intel.com, benh@kernel.crashing.org, paulus@samba.org, minchan.kim@gmail.com, akpm@linux-foundation.org, kosaki.motohiro@jp.fujitsu.com, wency@cn.fujitsu.com On 07/10/2012 05:58 PM, Yasuaki Ishimatsu wrote: > Hi Christoph, > > 2012/07/10 0:18, Christoph Lameter wrote: >> >> On Mon, 9 Jul 2012, Yasuaki Ishimatsu wrote: >> >>> Even if you apply these patches, you cannot remove the physical memory >>> completely since these patches are still under development. I want you to >>> cooperate to improve the physical memory hot-remove. So please review these >>> patches and give your comment/idea. >> >> Could you at least give a method on how you want to do physical memory >> removal? > > We plan to release a dynamic hardware partitionable system. It will be > able to hot remove/add a system board which included memory and cpu. > But as you know, Linux does not support memory hot-remove on x86 box. > So I try to develop it. > > Current plan to hot remove system board is to use container driver. > Thus I define the system board in ACPI DSDT table as a container device. > It have supported hot-add a container device. And if container device > has _EJ0 ACPI method, "eject" file to remove the container device is > prepared as follow: > > # ls -l /sys/bus/acpi/devices/ACPI0004\:01/eject > --w-------. 1 root root 4096 Jul 10 18:19 /sys/bus/acpi/devices/ACPI0004:01/eject > > When I hot-remove the container device, I echo 1 to the file as follow: > > #echo 1 > /sys/bus/acpi/devices/ACPI0004\:02/eject > > Then acpi_bus_trim() is called. And it calls acpi_memory_device_remove() > for removing memory device. But the code does not do nothing. > So I developed the continuation of the function. > >> You would have to remove all objects from the range you want to >> physically remove. That is only possible under special circumstances and >> with a limited set of objects. Even if you exclusively use ZONE_MOVEABLE >> you still may get cases where pages are pinned for a long time. > > I know it. So my memory hot-remove plan is as follows: > > 1. hot-added a system board > All memory which included the system board is offline. > > 2. online the memory as removable page > The function has not supported yet. It is being developed by Lai as follow: > http://lkml.indiana.edu/hypermail/linux/kernel/1207.0/01478.html > If it is supported, I will be able to create movable memory. > > 3. hot-remove the memory by container device's eject file We have implemented a prototype to do physical node (mem + CPU + IOH) hotplug for Itanium and is now porting it to x86. But with currently solution, memory hotplug functionality may cause 10-20% performance decrease because we concentrate all DMA/Normal memory to the first NUMA node, and all other NUMA nodes only hosts ZONE_MOVABLE. We are working on solution to minimize the performance drop now. > > Thanks, > Yasuaki Ishimatsu > >> >> I am not sure that these patches are useful unless we know where you are >> going with this. If we end up with a situation where we still cannot >> remove physical memory then this patchset is not helpful. > > > From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx118.postini.com [74.125.245.118]) by kanga.kvack.org (Postfix) with SMTP id D34746B006C for ; Tue, 10 Jul 2012 12:50:27 -0400 (EDT) Received: by ggm4 with SMTP id 4so248621ggm.14 for ; Tue, 10 Jul 2012 09:50:27 -0700 (PDT) Message-ID: <4FFC5D43.7040206@gmail.com> Date: Wed, 11 Jul 2012 00:50:11 +0800 From: Jiang Liu MIME-Version: 1.0 Subject: Re: [RFC PATCH v3 0/13] memory-hotplug : hot-remove physical memory References: <4FFAB0A2.8070304@jp.fujitsu.com> <4FFBFCAC.4010007@jp.fujitsu.com> In-Reply-To: <4FFBFCAC.4010007@jp.fujitsu.com> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: Yasuaki Ishimatsu Cc: Christoph Lameter , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-acpi@vger.kernel.org, rientjes@google.com, len.brown@intel.com, benh@kernel.crashing.org, paulus@samba.org, minchan.kim@gmail.com, akpm@linux-foundation.org, kosaki.motohiro@jp.fujitsu.com, wency@cn.fujitsu.com On 07/10/2012 05:58 PM, Yasuaki Ishimatsu wrote: > Hi Christoph, > > 2012/07/10 0:18, Christoph Lameter wrote: >> >> On Mon, 9 Jul 2012, Yasuaki Ishimatsu wrote: >> >>> Even if you apply these patches, you cannot remove the physical memory >>> completely since these patches are still under development. I want you to >>> cooperate to improve the physical memory hot-remove. So please review these >>> patches and give your comment/idea. >> >> Could you at least give a method on how you want to do physical memory >> removal? > > We plan to release a dynamic hardware partitionable system. It will be > able to hot remove/add a system board which included memory and cpu. > But as you know, Linux does not support memory hot-remove on x86 box. > So I try to develop it. > > Current plan to hot remove system board is to use container driver. > Thus I define the system board in ACPI DSDT table as a container device. > It have supported hot-add a container device. And if container device > has _EJ0 ACPI method, "eject" file to remove the container device is > prepared as follow: > > # ls -l /sys/bus/acpi/devices/ACPI0004\:01/eject > --w-------. 1 root root 4096 Jul 10 18:19 /sys/bus/acpi/devices/ACPI0004:01/eject > > When I hot-remove the container device, I echo 1 to the file as follow: > > #echo 1 > /sys/bus/acpi/devices/ACPI0004\:02/eject > > Then acpi_bus_trim() is called. And it calls acpi_memory_device_remove() > for removing memory device. But the code does not do nothing. > So I developed the continuation of the function. > >> You would have to remove all objects from the range you want to >> physically remove. That is only possible under special circumstances and >> with a limited set of objects. Even if you exclusively use ZONE_MOVEABLE >> you still may get cases where pages are pinned for a long time. > > I know it. So my memory hot-remove plan is as follows: > > 1. hot-added a system board > All memory which included the system board is offline. > > 2. online the memory as removable page > The function has not supported yet. It is being developed by Lai as follow: > http://lkml.indiana.edu/hypermail/linux/kernel/1207.0/01478.html > If it is supported, I will be able to create movable memory. > > 3. hot-remove the memory by container device's eject file We have implemented a prototype to do physical node (mem + CPU + IOH) hotplug for Itanium and is now porting it to x86. But with currently solution, memory hotplug functionality may cause 10-20% performance decrease because we concentrate all DMA/Normal memory to the first NUMA node, and all other NUMA nodes only hosts ZONE_MOVABLE. We are working on solution to minimize the performance drop now. > > Thanks, > Yasuaki Ishimatsu > >> >> I am not sure that these patches are useful unless we know where you are >> going with this. If we end up with a situation where we still cannot >> remove physical memory then this patchset is not helpful. > > > -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-yx0-f179.google.com (mail-yx0-f179.google.com [209.85.213.179]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority" (not verified)) by ozlabs.org (Postfix) with ESMTPS id 3C55C2C01FA for ; Wed, 11 Jul 2012 02:50:29 +1000 (EST) Received: by yenr13 with SMTP id r13so195118yen.38 for ; Tue, 10 Jul 2012 09:50:27 -0700 (PDT) Message-ID: <4FFC5D43.7040206@gmail.com> Date: Wed, 11 Jul 2012 00:50:11 +0800 From: Jiang Liu MIME-Version: 1.0 To: Yasuaki Ishimatsu Subject: Re: [RFC PATCH v3 0/13] memory-hotplug : hot-remove physical memory References: <4FFAB0A2.8070304@jp.fujitsu.com> <4FFBFCAC.4010007@jp.fujitsu.com> In-Reply-To: <4FFBFCAC.4010007@jp.fujitsu.com> Content-Type: text/plain; charset=ISO-8859-1 Cc: len.brown@intel.com, wency@cn.fujitsu.com, linux-acpi@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, paulus@samba.org, minchan.kim@gmail.com, kosaki.motohiro@jp.fujitsu.com, rientjes@google.com, Christoph Lameter , linuxppc-dev@lists.ozlabs.org, akpm@linux-foundation.org List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , On 07/10/2012 05:58 PM, Yasuaki Ishimatsu wrote: > Hi Christoph, > > 2012/07/10 0:18, Christoph Lameter wrote: >> >> On Mon, 9 Jul 2012, Yasuaki Ishimatsu wrote: >> >>> Even if you apply these patches, you cannot remove the physical memory >>> completely since these patches are still under development. I want you to >>> cooperate to improve the physical memory hot-remove. So please review these >>> patches and give your comment/idea. >> >> Could you at least give a method on how you want to do physical memory >> removal? > > We plan to release a dynamic hardware partitionable system. It will be > able to hot remove/add a system board which included memory and cpu. > But as you know, Linux does not support memory hot-remove on x86 box. > So I try to develop it. > > Current plan to hot remove system board is to use container driver. > Thus I define the system board in ACPI DSDT table as a container device. > It have supported hot-add a container device. And if container device > has _EJ0 ACPI method, "eject" file to remove the container device is > prepared as follow: > > # ls -l /sys/bus/acpi/devices/ACPI0004\:01/eject > --w-------. 1 root root 4096 Jul 10 18:19 /sys/bus/acpi/devices/ACPI0004:01/eject > > When I hot-remove the container device, I echo 1 to the file as follow: > > #echo 1 > /sys/bus/acpi/devices/ACPI0004\:02/eject > > Then acpi_bus_trim() is called. And it calls acpi_memory_device_remove() > for removing memory device. But the code does not do nothing. > So I developed the continuation of the function. > >> You would have to remove all objects from the range you want to >> physically remove. That is only possible under special circumstances and >> with a limited set of objects. Even if you exclusively use ZONE_MOVEABLE >> you still may get cases where pages are pinned for a long time. > > I know it. So my memory hot-remove plan is as follows: > > 1. hot-added a system board > All memory which included the system board is offline. > > 2. online the memory as removable page > The function has not supported yet. It is being developed by Lai as follow: > http://lkml.indiana.edu/hypermail/linux/kernel/1207.0/01478.html > If it is supported, I will be able to create movable memory. > > 3. hot-remove the memory by container device's eject file We have implemented a prototype to do physical node (mem + CPU + IOH) hotplug for Itanium and is now porting it to x86. But with currently solution, memory hotplug functionality may cause 10-20% performance decrease because we concentrate all DMA/Normal memory to the first NUMA node, and all other NUMA nodes only hosts ZONE_MOVABLE. We are working on solution to minimize the performance drop now. > > Thanks, > Yasuaki Ishimatsu > >> >> I am not sure that these patches are useful unless we know where you are >> going with this. If we end up with a situation where we still cannot >> remove physical memory then this patchset is not helpful. > > >