From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.5 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,NICE_REPLY_A,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2A2EDC433DB for ; Fri, 22 Jan 2021 12:04:02 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 8E625238EE for ; Fri, 22 Jan 2021 12:04:01 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8E625238EE Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 973206B0008; Fri, 22 Jan 2021 07:04:00 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9224C6B000A; Fri, 22 Jan 2021 07:04:00 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 81D8E6B000C; Fri, 22 Jan 2021 07:04:00 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0097.hostedemail.com [216.40.44.97]) by kanga.kvack.org (Postfix) with ESMTP id 6B1006B0008 for ; Fri, 22 Jan 2021 07:04:00 -0500 (EST) Received: from smtpin02.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 3163E52B3 for ; Fri, 22 Jan 2021 12:04:00 +0000 (UTC) X-FDA: 77733277440.02.name57_17127982756b Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin02.hostedemail.com (Postfix) with ESMTP id 0FAAD100A11AA for ; Fri, 22 Jan 2021 12:04:00 +0000 (UTC) X-HE-Tag: name57_17127982756b X-Filterd-Recvd-Size: 4776 Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by imf38.hostedemail.com (Postfix) with ESMTP for ; Fri, 22 Jan 2021 12:03:59 +0000 (UTC) X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (unknown [195.135.221.27]) by mx2.suse.de (Postfix) with ESMTP id 24929ABDA; Fri, 22 Jan 2021 12:03:58 +0000 (UTC) Subject: Re: [RFC PATCH v0] mm/slub: Let number of online CPUs determine the slub page order To: Vincent Guittot Cc: Christoph Lameter , Bharata B Rao , linux-kernel , linux-mm@kvack.org, David Rientjes , Joonsoo Kim , Andrew Morton , guro@fb.com, Shakeel Butt , Johannes Weiner , aneesh.kumar@linux.ibm.com, Jann Horn , Michal Hocko References: <20201118082759.1413056-1-bharata@linux.ibm.com> <20210121053003.GB2587010@in.ibm.com> From: Vlastimil Babka Message-ID: <786571e7-b9a2-4cdb-06d5-aa4a4b439b7e@suse.cz> Date: Fri, 22 Jan 2021 13:03:57 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.6.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 1/22/21 9:03 AM, Vincent Guittot wrote: > On Thu, 21 Jan 2021 at 19:19, Vlastimil Babka wrote: >> >> On 1/21/21 11:01 AM, Christoph Lameter wrote: >> > On Thu, 21 Jan 2021, Bharata B Rao wrote: >> > >> >> > The problem is that calculate_order() is called a number of times >> >> > before secondaries CPUs are booted and it returns 1 instead of 224. >> >> > This makes the use of num_online_cpus() irrelevant for those cases >> >> > >> >> > After adding in my command line "slub_min_objects=36" which equals to >> >> > 4 * (fls(num_online_cpus()) + 1) with a correct num_online_cpus == 224 >> >> > , the regression diseapears: >> >> > >> >> > 9 iterations of hackbench -l 16000 -g 16: 3.201sec (+/- 0.90%) >> >> I'm surprised that hackbench is that sensitive to slab performance, anyway. It's >> supposed to be a scheduler benchmark? What exactly is going on? >> > > From hackbench description: > Hackbench is both a benchmark and a stress test for the Linux kernel > scheduler. It's main > job is to create a specified number of pairs of schedulable > entities (either threads or > traditional processes) which communicate via either sockets or > pipes and time how long it > takes for each pair to send data back and forth. Yep, so I wonder which slab entities this is stressing that much. >> Things would be easier if we could trust *on all arches* either >> >> - num_present_cpus() to count what the hardware really physically has during >> boot, even if not yet onlined, at the time we init slab. This would still not >> handle later hotplug (probably mostly in a VM scenario, not that somebody would >> bring bunch of actual new cpu boards to a running bare metal system?). >> >> - num_possible_cpus()/nr_cpu_ids not to be excessive (broken BIOS?) on systems >> where it's not really possible to plug more CPU's. In a VM scenario we could >> still have an opposite problem, where theoretically "anything is possible" but >> the virtual cpus are never added later. > > On all the system that I have tested num_possible_cpus()/nr_cpu_ids > were correctly initialized > > large arm64 acpi system > small arm64 DT based system > VM on x86 system So it's just powerpc that has this issue with too large nr_cpu_ids? Is it caused by bios or the hypervisor? How does num_present_cpus() look there? What about heuristic: - num_online_cpus() > 1 - we trust that and use it - otherwise nr_cpu_ids Would that work? Too arbitrary? >> We could also start questioning the very assumption that number of cpus should >> affect slab page size in the first place. Should it? After all, each CPU will >> have one or more slab pages privately cached, as we discuss in the other >> thread... So why make the slab pages also larger? >> >> > Or the num_online_cpus needs to be up to date earlier. Why does this issue >> > not occur on x86? Does x86 have an up to date num_online_cpus earlier? >> > >> > >> >