From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753207AbdGJGpq (ORCPT ); Mon, 10 Jul 2017 02:45:46 -0400 Received: from mx2.suse.de ([195.135.220.15]:48276 "EHLO mx1.suse.de" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1752295AbdGJGpp (ORCPT ); Mon, 10 Jul 2017 02:45:45 -0400 Date: Mon, 10 Jul 2017 08:45:40 +0200 From: Michal Hocko To: Vlastimil Babka Cc: linux-mm@kvack.org, Andrew Morton , Mel Gorman , Andrea Arcangeli , Reza Arbab , Yasuaki Ishimatsu , qiuxishi@huawei.com, Kani Toshimitsu , slaoub@gmail.com, Joonsoo Kim , Daniel Kiper , Igor Mammedov , Vitaly Kuznetsov , Wei Yang , LKML , Linux API Subject: Re: [PATCH 2/2] mm, memory_hotplug: remove zone restrictions Message-ID: <20170710064540.GA19185@dhcp22.suse.cz> References: <20170629073509.623-1-mhocko@kernel.org> <20170629073509.623-3-mhocko@kernel.org> <64e889ae-24ab-b845-5751-978a76dd0dd9@suse.cz> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <64e889ae-24ab-b845-5751-978a76dd0dd9@suse.cz> User-Agent: Mutt/1.5.23 (2014-03-12) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Fri 07-07-17 17:02:59, Vlastimil Babka wrote: > [+CC linux-api] > > On 06/29/2017 09:35 AM, Michal Hocko wrote: > > From: Michal Hocko > > > > Historically we have enforced that any kernel zone (e.g ZONE_NORMAL) has > > to precede the Movable zone in the physical memory range. The purpose of > > the movable zone is, however, not bound to any physical memory restriction. > > It merely defines a class of migrateable and reclaimable memory. > > > > There are users (e.g. CMA) who might want to reserve specific physical > > memory ranges for their own purpose. Moreover our pfn walkers have to be > > prepared for zones overlapping in the physical range already because we > > do support interleaving NUMA nodes and therefore zones can interleave as > > well. This means we can allow each memory block to be associated with a > > different zone. > > > > Loosen the current onlining semantic and allow explicit onlining type on > > any memblock. That means that online_{kernel,movable} will be allowed > > regardless of the physical address of the memblock as long as it is > > offline of course. This might result in moveble zone overlapping with > > other kernel zones. Default onlining then becomes a bit tricky but still > > sensible. echo online > memoryXY/state will online the given block to > > 1) the default zone if the given range is outside of any zone > > 2) the enclosing zone if such a zone doesn't interleave with > > any other zone > > 3) the default zone if more zones interleave for this range > > where default zone is movable zone only if movable_node is enabled > > otherwise it is a kernel zone. > > > > Here is an example of the semantic with (movable_node is not present but > > it work in an analogous way). We start with following memblocks, all of > > them offline > > memory34/valid_zones:Normal Movable > > memory35/valid_zones:Normal Movable > > memory36/valid_zones:Normal Movable > > memory37/valid_zones:Normal Movable > > memory38/valid_zones:Normal Movable > > memory39/valid_zones:Normal Movable > > memory40/valid_zones:Normal Movable > > memory41/valid_zones:Normal Movable > > > > Now, we online block 34 in default mode and block 37 as movable > > root@test1:/sys/devices/system/node/node1# echo online > memory34/state > > root@test1:/sys/devices/system/node/node1# echo online_movable > memory37/state > > memory34/valid_zones:Normal > > memory35/valid_zones:Normal Movable > > memory36/valid_zones:Normal Movable > > memory37/valid_zones:Movable > > memory38/valid_zones:Normal Movable > > memory39/valid_zones:Normal Movable > > memory40/valid_zones:Normal Movable > > memory41/valid_zones:Normal Movable > > Hm so previously, blocks 37-41 would only allow Movable at this point, right? yes > Shouldn't we still default to Movable for them? We might be breaking some > existing userspace here. I do not think so. Prior to this merge window f1dd2cd13c4b ("mm, memory_hotplug: do not associate hotadded memory to zones until online") we allowed only the last offline or the adjacent to existing movable memory block to be onlined movable. So the above wasn't possible. I doubt we have grown a new user since the rework has been merged but if you think we should make sure nothing like that happens then we should probably merge this patch in this release cycle. > IMHO onlining new memory past existing blocks is more common use case than > onlining memory between two blocks that are already online? I am not really sure. It is quite common to online and offline within an existing zones for the memory ballooning. I do not know what kind of online operation they use but using the default online operation has historically preserved the zone so I would be really reluctant to change that. > I also agree with Wei Yang that it's rather fuzzy that a zone that has been > completely offlined will affect the defaults for the next onlining just because > it has some spanned range, which is however empty of actual populated memory. I am sorry but I still do not see why. The zone is not empty. It has a range spanned. It just doesn't have any pages online. I really fail to see how that is different from zones with large offline holes. > Maybe it would simplest for everyone to just default to Normal, except > movable_node? That's if we decide that the potential breakage I > described above is a non-issue. This would break the usecase where the memory is onlined a certain type initially and the offline/online it later on demand for ballooning. I wish this could be more clear but the default onlining has been fuzzy since the movable online has been introduced and it is hard to buil something really clear since then. The proposed semantic is the most clean I could come up with but I am open to any suggestions that wouldn't break existing usage. -- Michal Hocko SUSE Labs From mboxrd@z Thu Jan 1 00:00:00 1970 From: Michal Hocko Subject: Re: [PATCH 2/2] mm, memory_hotplug: remove zone restrictions Date: Mon, 10 Jul 2017 08:45:40 +0200 Message-ID: <20170710064540.GA19185@dhcp22.suse.cz> References: <20170629073509.623-1-mhocko@kernel.org> <20170629073509.623-3-mhocko@kernel.org> <64e889ae-24ab-b845-5751-978a76dd0dd9@suse.cz> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Return-path: Content-Disposition: inline In-Reply-To: <64e889ae-24ab-b845-5751-978a76dd0dd9@suse.cz> Sender: owner-linux-mm@kvack.org To: Vlastimil Babka Cc: linux-mm@kvack.org, Andrew Morton , Mel Gorman , Andrea Arcangeli , Reza Arbab , Yasuaki Ishimatsu , qiuxishi@huawei.com, Kani Toshimitsu , slaoub@gmail.com, Joonsoo Kim , Daniel Kiper , Igor Mammedov , Vitaly Kuznetsov , Wei Yang , LKML , Linux API List-Id: linux-api@vger.kernel.org On Fri 07-07-17 17:02:59, Vlastimil Babka wrote: > [+CC linux-api] > > On 06/29/2017 09:35 AM, Michal Hocko wrote: > > From: Michal Hocko > > > > Historically we have enforced that any kernel zone (e.g ZONE_NORMAL) has > > to precede the Movable zone in the physical memory range. The purpose of > > the movable zone is, however, not bound to any physical memory restriction. > > It merely defines a class of migrateable and reclaimable memory. > > > > There are users (e.g. CMA) who might want to reserve specific physical > > memory ranges for their own purpose. Moreover our pfn walkers have to be > > prepared for zones overlapping in the physical range already because we > > do support interleaving NUMA nodes and therefore zones can interleave as > > well. This means we can allow each memory block to be associated with a > > different zone. > > > > Loosen the current onlining semantic and allow explicit onlining type on > > any memblock. That means that online_{kernel,movable} will be allowed > > regardless of the physical address of the memblock as long as it is > > offline of course. This might result in moveble zone overlapping with > > other kernel zones. Default onlining then becomes a bit tricky but still > > sensible. echo online > memoryXY/state will online the given block to > > 1) the default zone if the given range is outside of any zone > > 2) the enclosing zone if such a zone doesn't interleave with > > any other zone > > 3) the default zone if more zones interleave for this range > > where default zone is movable zone only if movable_node is enabled > > otherwise it is a kernel zone. > > > > Here is an example of the semantic with (movable_node is not present but > > it work in an analogous way). We start with following memblocks, all of > > them offline > > memory34/valid_zones:Normal Movable > > memory35/valid_zones:Normal Movable > > memory36/valid_zones:Normal Movable > > memory37/valid_zones:Normal Movable > > memory38/valid_zones:Normal Movable > > memory39/valid_zones:Normal Movable > > memory40/valid_zones:Normal Movable > > memory41/valid_zones:Normal Movable > > > > Now, we online block 34 in default mode and block 37 as movable > > root@test1:/sys/devices/system/node/node1# echo online > memory34/state > > root@test1:/sys/devices/system/node/node1# echo online_movable > memory37/state > > memory34/valid_zones:Normal > > memory35/valid_zones:Normal Movable > > memory36/valid_zones:Normal Movable > > memory37/valid_zones:Movable > > memory38/valid_zones:Normal Movable > > memory39/valid_zones:Normal Movable > > memory40/valid_zones:Normal Movable > > memory41/valid_zones:Normal Movable > > Hm so previously, blocks 37-41 would only allow Movable at this point, right? yes > Shouldn't we still default to Movable for them? We might be breaking some > existing userspace here. I do not think so. Prior to this merge window f1dd2cd13c4b ("mm, memory_hotplug: do not associate hotadded memory to zones until online") we allowed only the last offline or the adjacent to existing movable memory block to be onlined movable. So the above wasn't possible. I doubt we have grown a new user since the rework has been merged but if you think we should make sure nothing like that happens then we should probably merge this patch in this release cycle. > IMHO onlining new memory past existing blocks is more common use case than > onlining memory between two blocks that are already online? I am not really sure. It is quite common to online and offline within an existing zones for the memory ballooning. I do not know what kind of online operation they use but using the default online operation has historically preserved the zone so I would be really reluctant to change that. > I also agree with Wei Yang that it's rather fuzzy that a zone that has been > completely offlined will affect the defaults for the next onlining just because > it has some spanned range, which is however empty of actual populated memory. I am sorry but I still do not see why. The zone is not empty. It has a range spanned. It just doesn't have any pages online. I really fail to see how that is different from zones with large offline holes. > Maybe it would simplest for everyone to just default to Normal, except > movable_node? That's if we decide that the potential breakage I > described above is a non-issue. This would break the usecase where the memory is onlined a certain type initially and the offline/online it later on demand for ballooning. I wish this could be more clear but the default onlining has been fuzzy since the movable online has been introduced and it is hard to buil something really clear since then. The proposed semantic is the most clean I could come up with but I am open to any suggestions that wouldn't break existing usage. -- Michal Hocko SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org