From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8BF1BC6FA82 for ; Fri, 23 Sep 2022 11:20:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C023380008; Fri, 23 Sep 2022 07:20:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BB2F880007; Fri, 23 Sep 2022 07:20:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A53E180008; Fri, 23 Sep 2022 07:20:11 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 968B880007 for ; Fri, 23 Sep 2022 07:20:11 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 3E252A104C for ; Fri, 23 Sep 2022 11:20:11 +0000 (UTC) X-FDA: 79943106222.21.AE14ADE Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by imf21.hostedemail.com (Postfix) with ESMTP id B66311C0010 for ; Fri, 23 Sep 2022 11:20:10 +0000 (UTC) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 43CE2B8272B; Fri, 23 Sep 2022 11:20:09 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 15548C433D6; Fri, 23 Sep 2022 11:19:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1663932008; bh=lzkcfA8lwY431cGoFaoDpZ054pCCOOU4vMeBaieOALo=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=CFFgso3bs6An9nGUhg/DG6R6EN9niEPLr2bcvNvSIZeH5Ca/4zD5HH3MAQwNJ5mc9 +Hc567HPd6/20p6YaNX/Zvv6xT/ckjT9jLPQvD/WGz8mFXi9wWGc+5xCpFYC4T4XZw J2QCchDnSDhacfqP44aIabXsMgDbC1DrzUvn9pDZ1hvVgL7+QgzJhX1ywAwW83QAff 5tr++tEQUnIFI/6oMtIWKFC+RtQAUJUrpM3ED/pvVcV6Y/GAKzQaMKcRFEC7DztYOH Au8CuF0rW5ZOStR6byQQubKH+EDX0QcQvObkvdsoCIqJ2ivv7nIVDarHsRqR3Li0Ww tr6LGpBwCOQ2w== Date: Fri, 23 Sep 2022 14:19:47 +0300 From: Mike Rapoport To: Doug Berger Cc: David Hildenbrand , Andrew Morton , Jonathan Corbet , Rob Herring , Krzysztof Kozlowski , Frank Rowand , Mike Kravetz , Muchun Song , Christoph Hellwig , Marek Szyprowski , Robin Murphy , Borislav Petkov , "Paul E. McKenney" , Neeraj Upadhyay , Randy Dunlap , Damien Le Moal , Florian Fainelli , Zi Yan , Oscar Salvador , Hari Bathini , Kees Cook , - , KOSAKI Motohiro , Mel Gorman , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, devicetree@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux.dev Subject: Re: [PATCH 00/21] mm: introduce Designated Movable Blocks Message-ID: References: <20220913195508.3511038-1-opendmb@gmail.com> <02561695-df44-4df6-c486-1431bf152650@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <02561695-df44-4df6-c486-1431bf152650@gmail.com> ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1663932010; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=IPhYUpZauKikAfVdOEQJLqhLVZF2+o75+8CfBOsurN0=; b=UiR0dggYPmMZNf3EAL8qle7NGPm54riKzIqfyuVDoDVN+1SPHVPWD89coXSXDTbq7AiFzT zdTZmckFyZ2kGUtRETAkg7KyPbSNWDAASefnjgNXaJ7KaZBbPRZKFq/+mDTKSai+J8Yg9U xUy5ybVU/Jql7CjPgtZDLAO2rr+FkYU= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=CFFgso3b; spf=pass (imf21.hostedemail.com: domain of rppt@kernel.org designates 145.40.68.75 as permitted sender) smtp.mailfrom=rppt@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1663932010; a=rsa-sha256; cv=none; b=tjZBSyeXUT+noAjKOmP7gdVpzSpwVT2WcysijjDTEMZ8qFW7UKyan8gA/tHvPUAFmeGRuv /089EjDG3scQ1jrHo1XyuyF3kudOqXcOoypQdcU/JaDDxEYOOMOGKsZMeGLXV/uehGEtE2 sBvRRzVJYpcnPBasev3uC3QsCCPkQbE= X-Stat-Signature: 7mr1qx4urdwbtehgrm7rcmjuxafusung X-Rspamd-Queue-Id: B66311C0010 X-Rspam-User: Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=CFFgso3b; spf=pass (imf21.hostedemail.com: domain of rppt@kernel.org designates 145.40.68.75 as permitted sender) smtp.mailfrom=rppt@kernel.org; dmarc=pass (policy=none) header.from=kernel.org X-Rspamd-Server: rspam03 X-HE-Tag: 1663932010-762576 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hi Doug, I only had time to skim through the patches and before diving in I'd like to clarify a few things. On Mon, Sep 19, 2022 at 06:03:55PM -0700, Doug Berger wrote: > On 9/19/2022 2:00 AM, David Hildenbrand wrote: > > > > How is this memory currently presented to the system? > > The 7278 device has four ARMv8 CPU cores in an SMP cluster and two memory > controllers (MEMCs). Each MEMC is capable of controlling up to 8GB of DRAM. > An example 7278 system might have 1GB on each controller, so an arm64 kernel > might see 1GB on MEMC0 at 0x40000000-0x7FFFFFFF and 1GB on MEMC1 at > 0x300000000-0x33FFFFFFF. > > The base capability described in commits 7-15 of this V1 patch set is to > allow a 'movablecore' block to be created at a particular base address > rather than solely at the end of addressable memory. I think this capability is only useful when there is non-uniform access to different memory ranges. Otherwise it wouldn't matter where the movable pages reside. The system you describe looks quite NUMA to me, with two memory controllers, each for accessing a partial range of the available memory. > > > expressed the desire to locate ZONE_MOVABLE memory on each > > > memory controller to allow user space intensive processing to > > > make better use of the additional memory bandwidth. > > > > Can you share some more how exactly ZONE_MOVABLE would help here to make > > better use of the memory bandwidth? > > ZONE_MOVABLE memory is effectively unusable by the kernel. It can be used by > user space applications through both the page allocator and the Hugetlbfs. > If a large 'movablecore' allocation is defined and it can only be located at > the end of addressable memory then it will always be located on MEMC1 of a > 7278 system. This will create a tendency for user space accesses to consume > more bandwidth on the MEMC1 memory controller and kernel space accesses to > consume more bandwidth on MEMC0. A more even distribution of ZONE_MOVABLE > memory between the available memory controllers in theory makes more memory > bandwidth available to user space intensive loads. The theory makes perfect sense, but is there any practical evidence of improvement? Some benchmark results that illustrate the difference would be nice. > > > BACKGROUND: > > > NUMA architectures support distributing movablecore memory > > > across each node, but it is undesirable to introduce the > > > overhead and complexities of NUMA on systems that don't have a > > > Non-Uniform Memory Architecture. > > > > How exactly would that look like? I think I am missing something :) > > The notion would be to consider each memory controller as a separate node, > but as stated it is not desirable. Why? > Thanks for your consideration, > Dough Baker ... I mean Doug Berger :). -- Sincerely yours, Mike.