From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5BB4AC54EE9 for ; Mon, 19 Sep 2022 09:00:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 77CD5940007; Mon, 19 Sep 2022 05:00:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 72CED6B0073; Mon, 19 Sep 2022 05:00:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5CDCE940007; Mon, 19 Sep 2022 05:00:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 4C9CB6B0072 for ; Mon, 19 Sep 2022 05:00:23 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 0E0D614047D for ; Mon, 19 Sep 2022 09:00:23 +0000 (UTC) X-FDA: 79928238726.10.E5959F5 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf25.hostedemail.com (Postfix) with ESMTP id 50E3AA00DE for ; Mon, 19 Sep 2022 09:00:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1663578021; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=f8vJrGjQeilqRFYE1RxARCjX6qJaMbpyS//5x6x7ndo=; b=hw2HO6ndgFQGzK+IufliCNxPIfNREa9z84WabP2z66JhWhO+U1Fz6b3F8s08Ok81m5ilOz hXaVOxHaGKOqZDoAmAPFKcsYTix8D64GCfZlFjEwr9r2OyvgQQ/Zzi+65+ggXymOebPLLv Oi03w0+nd/4RdT29iiihtnSOkf53gbA= Received: from mail-wm1-f69.google.com (mail-wm1-f69.google.com [209.85.128.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-149-buSlz7HONpC61zckQoLIVg-1; Mon, 19 Sep 2022 05:00:18 -0400 X-MC-Unique: buSlz7HONpC61zckQoLIVg-1 Received: by mail-wm1-f69.google.com with SMTP id v128-20020a1cac86000000b003b33fab37e8so14914201wme.0 for ; Mon, 19 Sep 2022 02:00:18 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:in-reply-to:subject:organization:from :references:cc:to:content-language:user-agent:mime-version:date :message-id:x-gm-message-state:from:to:cc:subject:date; bh=f8vJrGjQeilqRFYE1RxARCjX6qJaMbpyS//5x6x7ndo=; b=HzGEjam3M+xe5GXgEAPfPiYrIsGPFE8nynnaP/PEWyBQmzASNT0CkrBGsaAP2ploOD v/CkSuMPxItiD23QyVFLXmW6g3lZU3/FI7cNVDmExw/rDNMGfM6USEwQszq4hZ803kCB oq+CEXazwOqVlXEsZlolQj4BGKD3UpWdA89503SfzNRKIB7dSI54t96cHoFlEG85M35g x4AWmdPmRsvJkLYGWBxKRLHXnq5u9FSrhRLWxBkVFfUS9a/bAzA6UotdqKq9kAntdMmg 5G4XrgRmGsYraGtky66uiPFyBse+NgGhf7id/hfFZ+yKCefVIE/EKU6mGwIF7/6J6ycE 9W0A== X-Gm-Message-State: ACrzQf2kig1iUQjeruvKO9v1MdXatDan0N1VCyNC2clCgsm5CIyNnwx1 x2/x+jVlia4usLOfvsNVe0m+9hExszHyo0rmKBpiDCETSOfIGkCfmRLYZ+iCvjFLgmVrTLp1EdE cuv6SuGMUjwo= X-Received: by 2002:adf:fac1:0:b0:226:d790:7644 with SMTP id a1-20020adffac1000000b00226d7907644mr9702009wrs.471.1663578017568; Mon, 19 Sep 2022 02:00:17 -0700 (PDT) X-Google-Smtp-Source: AMsMyM7rznsvQQ2aFPHDwwQoZOzqCdtHUk+1F3DZsMptAy7fZIeeIGMjN1cCROUuWBeOapmd+ic45w== X-Received: by 2002:adf:fac1:0:b0:226:d790:7644 with SMTP id a1-20020adffac1000000b00226d7907644mr9701986wrs.471.1663578017299; Mon, 19 Sep 2022 02:00:17 -0700 (PDT) Received: from ?IPV6:2003:cb:c703:c100:c136:f914:345f:f5f3? (p200300cbc703c100c136f914345ff5f3.dip0.t-ipconnect.de. [2003:cb:c703:c100:c136:f914:345f:f5f3]) by smtp.gmail.com with ESMTPSA id k38-20020a05600c1ca600b003b31fc77407sm13688025wms.30.2022.09.19.02.00.15 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 19 Sep 2022 02:00:16 -0700 (PDT) Message-ID: Date: Mon, 19 Sep 2022 11:00:15 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.2.1 To: Doug Berger , Andrew Morton Cc: Jonathan Corbet , Rob Herring , Krzysztof Kozlowski , Frank Rowand , Mike Kravetz , Muchun Song , Mike Rapoport , Christoph Hellwig , Marek Szyprowski , Robin Murphy , Borislav Petkov , "Paul E. McKenney" , Neeraj Upadhyay , Randy Dunlap , Damien Le Moal , Florian Fainelli , Zi Yan , Oscar Salvador , Hari Bathini , Kees Cook , - , KOSAKI Motohiro , Mel Gorman , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, devicetree@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux.dev References: <20220913195508.3511038-1-opendmb@gmail.com> From: David Hildenbrand Organization: Red Hat Subject: Re: [PATCH 00/21] mm: introduce Designated Movable Blocks In-Reply-To: <20220913195508.3511038-1-opendmb@gmail.com> X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Language: en-US Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1663578022; a=rsa-sha256; cv=none; b=nG7MuQLf83Dd+8/X7Jeg1XE+PMNnRwocqL86L5kjGAT5VsuQraZII0Bqo814AKZjBIKs0M rkPWYwj4t2DJGpwylP1XWlUFyQvFsHgzCKLR0PL6EWJmNuLBjErZU5dOlp0GmB+8eJdkUw lDJ9JaTqcvQgGACtL5SihvlMCeqZjU8= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=hw2HO6nd; spf=pass (imf25.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1663578022; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=f8vJrGjQeilqRFYE1RxARCjX6qJaMbpyS//5x6x7ndo=; b=2EjLF8ufdRllIt4qkoErZkHrcqcsEtzMg8hemuL/bebiMyxzSlVUo7CYBzPxIkOlmqMpUO 5TnAZe7hHffh4B6kzY0ovG47ZmRliU/yDqn7WNqBddbP0jXiSz8rAB6u8+fK5JZxUC+kqP vQmNzsxmNjzu6+tNdRIIECg21bNQ/js= Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=hw2HO6nd; spf=pass (imf25.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Stat-Signature: rs8c5apkb943ztq6jeg6tadfbgasiagj X-Rspam-User: X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 50E3AA00DE X-HE-Tag: 1663578022-516502 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hi Dough, I have some high-level questions. > MOTIVATION: > Some Broadcom devices (e.g. 7445, 7278) contain multiple memory > controllers with each mapped in a different address range within > a Uniform Memory Architecture. Some users of these systems have How large are these areas typically? How large are they in comparison to other memory in the system? How is this memory currently presented to the system? > expressed the desire to locate ZONE_MOVABLE memory on each > memory controller to allow user space intensive processing to > make better use of the additional memory bandwidth. Can you share some more how exactly ZONE_MOVABLE would help here to make better use of the memory bandwidth? > Unfortunately, the historical monotonic layout of zones would > mean that if the lowest addressed memory controller contains > ZONE_MOVABLE memory then all of the memory available from > memory controllers at higher addresses must also be in the > ZONE_MOVABLE zone. This would force all kernel memory accesses > onto the lowest addressed memory controller and significantly > reduce the amount of memory available for non-movable > allocations. We do have code that relies on zones during boot to not overlap within a single node. > > The main objective of this patch set is therefore to allow a > block of memory to be designated as part of the ZONE_MOVABLE > zone where it will always only be used by the kernel page > allocator to satisfy requests for movable pages. The term > Designated Movable Block is introduced here to represent such a > block. The favored implementation allows modification of the Sorry to say, but that term is rather suboptimal to describe what you are doing here. You simply have some system RAM you'd want to have managed by ZONE_MOVABLE, no? > 'movablecore' kernel parameter to allow specification of a base > address and support for multiple blocks. The existing > 'movablecore' mechanisms are retained. Other mechanisms based on > device tree are also included in this set. > > BACKGROUND: > NUMA architectures support distributing movablecore memory > across each node, but it is undesirable to introduce the > overhead and complexities of NUMA on systems that don't have a > Non-Uniform Memory Architecture. How exactly would that look like? I think I am missing something :) > > Commit 342332e6a925 ("mm/page_alloc.c: introduce kernelcore=mirror option") > also depends on zone overlap to support sytems with multiple > mirrored ranges. IIRC, zones will not overlap within a single node. > > Commit c6f03e2903c9 ("mm, memory_hotplug: remove zone restrictions") > embraced overlapped zones for memory hotplug. Yes, after boot. > > This commit set follows their lead to allow the ZONE_MOVABLE > zone to overlap other zones while spanning the pages from the > lowest Designated Movable Block to the end of the node. > Designated Movable Blocks are made absent from overlapping zones > and present within the ZONE_MOVABLE zone. > > I initially investigated an implementation using a Designated > Movable migrate type in line with comments[1] made by Mel Gorman > regarding a "sticky" MIGRATE_MOVABLE type to avoid using > ZONE_MOVABLE. However, this approach was riskier since it was > much more instrusive on the allocation paths. Ultimately, the > progress made by the memory hotplug folks to expand the > ZONE_MOVABLE functionality convinced me to follow this approach. > > OPPORTUNITIES: > There have been many attempts to modify the behavior of the > kernel page allocators use of CMA regions. This implementation > of Designated Movable Blocks creates an opportunity to repurpose > the CMA allocator to operate on ZONE_MOVABLE memory that the > kernel page allocator can use more agressively, without > affecting the existing CMA implementation. It is hoped that the > "shared-dmb-pool" approach included here will be useful in cases > where memory sharing is more important than allocation latency. > > CMA introduced a paradigm where multiple allocators could > operate on the same region of memory, and that paradigm can be > extended to Designated Movable Blocks as well. I was interested > in using kernel resource management as a mechanism for exposing > Designated Movable Block resources (e.g. /proc/iomem) that would > be used by the kernel page allocator like any other ZONE_MOVABLE > memory, but could be claimed by an alternative allocator (e.g. > CMA). Unfortunately, this becomes complicated because the kernel > resource implementation varies materially across different > architectures and I do not require this capability so I have > deferred that. Why can't we simply designate these regions as CMA regions? Why do we have to start using ZONE_MOVABLE for them? -- Thanks, David / dhildenb