From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id CCEEFC433EF for ; Tue, 22 Mar 2022 04:55:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:Subject:Message-ID:Date:From: In-Reply-To:References:MIME-Version:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=edANiyF1FwGFGptHURzKzVl0x0olVGFPDNiMIVRxLbA=; b=xlJmA1FOrL0kl3 /L39v1mskyblaFfmTPm20GpSxt6WzKUAx8lOvDQAdsjZs0GNXvmlBnF7nuRvB5wn9pT/vA5xzTccy ZB6f5dj2xH9v4n+roWCBvQq5IEJMCHe7UbhqziIDP289QQPSF5FLa2ZXLZYNHrTPRDxfoJKgKvkV3 CJDBvQJzl9twq2hhxdlJUX9gb+pQeqIs1q8aD67ghd6heERi79XyW/JzdKbgyZXErnmQtqE6Tc4Tf rUgmKc7EgX8d1hLl2v7J6rw6iirdyXW3n88H3tgUiUP72NzxJBHU9pht+2ZoQoTX3AEMCODT703oe pppswY/hWfs+qmoV4GYQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nWWVv-009zt7-8z; Tue, 22 Mar 2022 04:52:59 +0000 Received: from mail-vk1-xa33.google.com ([2607:f8b0:4864:20::a33]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nWWVq-009zri-QQ for linux-arm-kernel@lists.infradead.org; Tue, 22 Mar 2022 04:52:56 +0000 Received: by mail-vk1-xa33.google.com with SMTP id j204so882281vkj.8 for ; Mon, 21 Mar 2022 21:52:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=WZwFAXeypL3jPipKNgPJxQINJj6JMXunijbszepaSjU=; b=YFh0z8aRY3bgF45d2I8rgI3ST09NIq0Hofz3dyojfGq9mke2zfjLTsefMvQvUBTJgm 1qnKwVGrEXKXSj0hr1XpDupQwV09DvcSlZv2oyo79KqZp8KW5tKNq+0f1ASQPMGtcfJj P5fQrQ8ujRVKAEfEuFzSHk0AfcCg6JSuVh6LG1bUk+O7UOrgYohfOEL9uBkgKuz4Wreu Q8KI8EdVa4vu6DTWcfdyaVSbtYeXcWRWcrhBF6Lx8WDBw43iNsBdBH0EcHw0kye8ODpH XYWP2nIJloYZp0SmdNuOj8C9hJ23ZC46NCbzbhzmgg4s3AxfY1MAHIflZ+Ntz3noPvvn it6A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=WZwFAXeypL3jPipKNgPJxQINJj6JMXunijbszepaSjU=; b=kEOUl324i+YgeFu6YvM8p8Ok5MLVfX9G+NE0Tj1EVeLSTkqYgphZkVV0awYmkVKN8S tPX6wAcxzideybGsMrtkR+loaqMZN/94czj8+kQK5wOfyysP4jwcs87llINcOhiZlanI YlcjZY/5HR5k1GKG8AEL9QXvDsDRMGEMgurIAmbaG//NFGTqHvllS6Tit4Nl3HZuBvMT WtstpBOk0+8ehQ+aP6tU7QSf4smslhngjCfaJ1afENMqbzhjO9Jr/eEaKNbKTQ+vkEOF eAak8gKq1clcHDS+BGlAmA78A0nxPHt8X4u/Xnqqa7CfFCtqpBfoMGj5G/NFyOtUN/W6 x9rQ== X-Gm-Message-State: AOAM532aoUl5r8iYhJhwAWWfObWAJ0PsEigtTDJSvGXFrjCSEnC0GA88 etv6zilFVDRBKtoT0tORKDxy5EjrQTLIHCUQ+ZvSKg== X-Google-Smtp-Source: ABdhPJxFR1nglzJK4RoUAy7iy7C1My9l/amg903cMnO5caRvyJNwsaMuy9GXoP8bTaVJxjBoEyZx6dpVzJR7dnuPQ7o= X-Received: by 2002:a05:6122:887:b0:332:699e:7e67 with SMTP id 7-20020a056122088700b00332699e7e67mr9362524vkf.35.1647924773467; Mon, 21 Mar 2022 21:52:53 -0700 (PDT) MIME-Version: 1.0 References: <20220309021230.721028-1-yuzhao@google.com> <20220309021230.721028-6-yuzhao@google.com> <875yoh552i.fsf@yhuang6-desk2.ccr.corp.intel.com> In-Reply-To: From: Yu Zhao Date: Mon, 21 Mar 2022 22:52:42 -0600 Message-ID: Subject: Re: [PATCH v9 05/14] mm: multi-gen LRU: groundwork To: Prarit Bhargava , Justin Forbes Cc: Andi Kleen , kernel-team@lists.ubuntu.com, Vaibhav Jain , Rik van Riel , Mel Gorman , Catalin Marinas , Johannes Weiner , Aneesh Kumar , Brian Geffon , "open list:DOCUMENTATION" , Jesse Barnes , Sofia Trinh , "Huang, Ying" , linux-kernel , Steven Barrett , Shuang Zhai , Donald Carr , Oleksandr Natalenko , =?UTF-8?Q?Holger_Hoffst=C3=A4tte?= , Will Deacon , Dave Hansen , Jonathan Corbet , Mike Rapoport , Andrew Morton , Jens Axboe , Hillf Danton , Michal Hocko , kernel , Suleiman Souhlal , Daniel Byrne , "the arch/x86 maintainers" , Konstantin Kharlamov , Matthew Wilcox , Linus Torvalds , Michael Larabel , Linux-MM , Kernel Page Reclaim v2 , Jan Alexander Steffens , Linux ARM X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220321_215254_868130_06F05D3A X-CRM114-Status: GOOD ( 32.56 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Mon, Mar 21, 2022 at 1:18 PM Prarit Bhargava wrote: > > On 3/21/22 14:58, Justin Forbes wrote: > > On Mon, Mar 14, 2022 at 4:30 AM Yu Zhao wrote: > >> > >> On Mon, Mar 14, 2022 at 2:09 AM Huang, Ying wrote: > >>> > >>> Hi, Yu, > >>> > >>> Yu Zhao writes: > >>>> diff --git a/mm/Kconfig b/mm/Kconfig > >>>> index 3326ee3903f3..747ab1690bcf 100644 > >>>> --- a/mm/Kconfig > >>>> +++ b/mm/Kconfig > >>>> @@ -892,6 +892,16 @@ config ANON_VMA_NAME > >>>> area from being merged with adjacent virtual memory areas due to the > >>>> difference in their name. > >>>> > >>>> +# the multi-gen LRU { > >>>> +config LRU_GEN > >>>> + bool "Multi-Gen LRU" > >>>> + depends on MMU > >>>> + # the following options can use up the spare bits in page flags > >>>> + depends on !MAXSMP && (64BIT || !SPARSEMEM || SPARSEMEM_VMEMMAP) > >>> > >>> LRU_GEN depends on !MAXSMP. So, What is the maximum NR_CPUS supported > >>> by LRU_GEN? > >> > >> LRU_GEN doesn't really care about NR_CPUS. IOW, it doesn't impose a > >> max number. The dependency is with NODES_SHIFT selected by MAXSMP: > >> default "10" if MAXSMP > >> This combined with LAST_CPUPID_SHIFT can exhaust the spare bits in page flags. > >> > >> MAXSMP is meant for kernel developers to test their code, and it > >> should not be used in production [1]. But some distros unfortunately > >> ship kernels built with this option, e.g., Fedora and Ubuntu. And > >> their users reported build errors to me after they applied MGLRU on > >> those kernels ("Not enough bits in page flags"). Let me add Fedora and > >> Ubuntu to this thread. > >> > >> Fedora and Ubuntu, > >> > >> Could you please clarify if there is a reason to ship kernels built > >> with MAXSMP? Otherwise, please consider disabling this option. Thanks. > >> > >> As per above, MAXSMP enables ridiculously large numbers of CPUs and > >> NUMA nodes for testing purposes. It is detrimental to performance, > >> e.g., CPUMASK_OFFSTACK. > > > > It was enabled for Fedora, and RHEL because we did need more than 512 > > CPUs, originally only in RHEL until SGI (years ago) complained that > > they were testing very large machines with Fedora. The testing done > > on RHEL showed that the performance impact was minimal. For a very > > long time we had MAXSMP off and carried a patch which allowed us to > > turn on CPUMASK_OFFSTACK without debugging because there was supposed > > to be "something else" coming. In 2019 we gave up, dropped that patch > > and just turned on MAXSMP. > > > > I do not have any metrics for how often someone runs Fedora on a > > ridiculously large machine these days, but I would guess that number > > is not 0. > > It is not 0. I've seen data from large systems (1000+ logical threads) > that are running Fedora albeit with a modified Fedora kernel. > > Additionally the max limit for CPUS in RHEL is 1792, however, we have > recently had a request to *double* that to 3584. You should just assume > that number will continue to increase. Good to know. Thanks. >From the standpoint of overhead, I'd consider NR_CPUS=4096 and NODES_SHIFT=7 as the next step, before going with MAXSMP. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel