From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8194FC433E0 for ; Fri, 22 Jan 2021 13:17:24 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5120B23428 for ; Fri, 22 Jan 2021 13:17:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727442AbhAVNRX (ORCPT ); Fri, 22 Jan 2021 08:17:23 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36452 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727401AbhAVNRF (ORCPT ); Fri, 22 Jan 2021 08:17:05 -0500 Received: from mail-lj1-x231.google.com (mail-lj1-x231.google.com [IPv6:2a00:1450:4864:20::231]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A4542C06174A for ; Fri, 22 Jan 2021 05:16:24 -0800 (PST) Received: by mail-lj1-x231.google.com with SMTP id f2so1255586ljp.11 for ; Fri, 22 Jan 2021 05:16:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=1v2FAdAIM+FduQLY9fgZwr7tF6/Isi2KlND7ig2PLqM=; b=w65sROKBqsor9zfqRpfltZ34/wM853Cw2NXBuRcrS5fYxv/d2nceN68WoCJYvFVGoh krEbKtoXibVHrjj1toHnXg9oBfFeOT2V8UTDDtG84gaRlJk0VPT2XhXzHi9HKL9/Byn3 zTOV+lhC4xDAgmZQZczTdURgv4OnednJAug7hzkH63lMFprGrvad1FHUV1DO3N5/Y2ut R1ilFeGGEuJQE251rWv1/Q1ukRwGMg1UadZSwxTDfmYucOZbuoaRajZ5K3+DU/FFOcoq H+UMa41hui9M/D7u6I+yXBmO/oY31aSQyWQTRSCxtj38sDe3NqP5W6HHGOTpzA0TdSsM xNxQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=1v2FAdAIM+FduQLY9fgZwr7tF6/Isi2KlND7ig2PLqM=; b=CiGudiPJvpHP91n936BjwxvcyQoDN7dArAscKy8pqaivkA1vL4pfx7Rq5rp2FDl/3Z +bwGC5gLWLfZTuG028xJ6aD7EY4Yjq+fQcNlMRluKt8FKwlBB+VDpEnmvzPSdITv9K8V 0uRKnOCfK2aFNhGaSmIczhUJcyqBUB9NKtn5okJOicJwEIQmmUajoxhG35biqtNII2Dj ZS1KX9KCwDHJIUPiSXgFT5JuAVY8PYhrOyKKUenLzpE+6kVblbnSzMVzsur1miR7lnz1 t5xDGt8O8bBeCwOWp9bPWvXSgMWYKXpn4QdmUx0B6b6a7xgW1NmXwOuKzw6LauL6LwPb lozA== X-Gm-Message-State: AOAM532XUmZ9B3YiidWV0tzFvEd6QuHvdYp5k4lPcwHr7sJFsv5482P7 Yusu6MjmJIk30ccLTuKTiIl5Wsj8WP2WQtYB7k7KxsV8twRmXQ== X-Google-Smtp-Source: ABdhPJxjtYcNvePVfz3PyQvJmmzO6M4fKzwNdZqKua6XH8jtTIXlHJqEXPINzkevucgcTd+1sQSvY97jqlAeG8Lnsx4= X-Received: by 2002:a2e:b6cc:: with SMTP id m12mr837848ljo.401.1611321383077; Fri, 22 Jan 2021 05:16:23 -0800 (PST) MIME-Version: 1.0 References: <20201118082759.1413056-1-bharata@linux.ibm.com> <20210121053003.GB2587010@in.ibm.com> <786571e7-b9a2-4cdb-06d5-aa4a4b439b7e@suse.cz> In-Reply-To: <786571e7-b9a2-4cdb-06d5-aa4a4b439b7e@suse.cz> From: Vincent Guittot Date: Fri, 22 Jan 2021 14:16:11 +0100 Message-ID: Subject: Re: [RFC PATCH v0] mm/slub: Let number of online CPUs determine the slub page order To: Vlastimil Babka Cc: Christoph Lameter , Bharata B Rao , linux-kernel , linux-mm@kvack.org, David Rientjes , Joonsoo Kim , Andrew Morton , guro@fb.com, Shakeel Butt , Johannes Weiner , aneesh.kumar@linux.ibm.com, Jann Horn , Michal Hocko Content-Type: text/plain; charset="UTF-8" Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Fri, 22 Jan 2021 at 13:03, Vlastimil Babka wrote: > > On 1/22/21 9:03 AM, Vincent Guittot wrote: > > On Thu, 21 Jan 2021 at 19:19, Vlastimil Babka wrote: > >> > >> On 1/21/21 11:01 AM, Christoph Lameter wrote: > >> > On Thu, 21 Jan 2021, Bharata B Rao wrote: > >> > > >> >> > The problem is that calculate_order() is called a number of times > >> >> > before secondaries CPUs are booted and it returns 1 instead of 224. > >> >> > This makes the use of num_online_cpus() irrelevant for those cases > >> >> > > >> >> > After adding in my command line "slub_min_objects=36" which equals to > >> >> > 4 * (fls(num_online_cpus()) + 1) with a correct num_online_cpus == 224 > >> >> > , the regression diseapears: > >> >> > > >> >> > 9 iterations of hackbench -l 16000 -g 16: 3.201sec (+/- 0.90%) > >> > >> I'm surprised that hackbench is that sensitive to slab performance, anyway. It's > >> supposed to be a scheduler benchmark? What exactly is going on? > >> > > > > From hackbench description: > > Hackbench is both a benchmark and a stress test for the Linux kernel > > scheduler. It's main > > job is to create a specified number of pairs of schedulable > > entities (either threads or > > traditional processes) which communicate via either sockets or > > pipes and time how long it > > takes for each pair to send data back and forth. > > Yep, so I wonder which slab entities this is stressing that much. > > >> Things would be easier if we could trust *on all arches* either > >> > >> - num_present_cpus() to count what the hardware really physically has during > >> boot, even if not yet onlined, at the time we init slab. This would still not > >> handle later hotplug (probably mostly in a VM scenario, not that somebody would > >> bring bunch of actual new cpu boards to a running bare metal system?). > >> > >> - num_possible_cpus()/nr_cpu_ids not to be excessive (broken BIOS?) on systems > >> where it's not really possible to plug more CPU's. In a VM scenario we could > >> still have an opposite problem, where theoretically "anything is possible" but > >> the virtual cpus are never added later. > > > > On all the system that I have tested num_possible_cpus()/nr_cpu_ids > > were correctly initialized > > > > large arm64 acpi system > > small arm64 DT based system > > VM on x86 system > > So it's just powerpc that has this issue with too large nr_cpu_ids? Is it caused > by bios or the hypervisor? How does num_present_cpus() look there? num_present_cpus() starts to 1 until secondary cpus boot in the arm64 case > > What about heuristic: > - num_online_cpus() > 1 - we trust that and use it > - otherwise nr_cpu_ids > Would that work? Too arbitrary? > > > >> We could also start questioning the very assumption that number of cpus should > >> affect slab page size in the first place. Should it? After all, each CPU will > >> have one or more slab pages privately cached, as we discuss in the other > >> thread... So why make the slab pages also larger? > >> > >> > Or the num_online_cpus needs to be up to date earlier. Why does this issue > >> > not occur on x86? Does x86 have an up to date num_online_cpus earlier? > >> > > >> > > >> > > > From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EBAA6C433E0 for ; Fri, 22 Jan 2021 13:16:26 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 6FA1823106 for ; Fri, 22 Jan 2021 13:16:26 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6FA1823106 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id D72ED6B000C; Fri, 22 Jan 2021 08:16:25 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id CFB4E6B000D; Fri, 22 Jan 2021 08:16:25 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B4D2B6B000E; Fri, 22 Jan 2021 08:16:25 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0159.hostedemail.com [216.40.44.159]) by kanga.kvack.org (Postfix) with ESMTP id 98F4A6B000C for ; Fri, 22 Jan 2021 08:16:25 -0500 (EST) Received: from smtpin14.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 5B88C1814065D for ; Fri, 22 Jan 2021 13:16:25 +0000 (UTC) X-FDA: 77733459930.14.event64_00033622756c Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin14.hostedemail.com (Postfix) with ESMTP id 3677318229818 for ; Fri, 22 Jan 2021 13:16:25 +0000 (UTC) X-HE-Tag: event64_00033622756c X-Filterd-Recvd-Size: 6622 Received: from mail-lj1-f177.google.com (mail-lj1-f177.google.com [209.85.208.177]) by imf47.hostedemail.com (Postfix) with ESMTP for ; Fri, 22 Jan 2021 13:16:24 +0000 (UTC) Received: by mail-lj1-f177.google.com with SMTP id u11so6412528ljo.13 for ; Fri, 22 Jan 2021 05:16:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=1v2FAdAIM+FduQLY9fgZwr7tF6/Isi2KlND7ig2PLqM=; b=w65sROKBqsor9zfqRpfltZ34/wM853Cw2NXBuRcrS5fYxv/d2nceN68WoCJYvFVGoh krEbKtoXibVHrjj1toHnXg9oBfFeOT2V8UTDDtG84gaRlJk0VPT2XhXzHi9HKL9/Byn3 zTOV+lhC4xDAgmZQZczTdURgv4OnednJAug7hzkH63lMFprGrvad1FHUV1DO3N5/Y2ut R1ilFeGGEuJQE251rWv1/Q1ukRwGMg1UadZSwxTDfmYucOZbuoaRajZ5K3+DU/FFOcoq H+UMa41hui9M/D7u6I+yXBmO/oY31aSQyWQTRSCxtj38sDe3NqP5W6HHGOTpzA0TdSsM xNxQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=1v2FAdAIM+FduQLY9fgZwr7tF6/Isi2KlND7ig2PLqM=; b=S58aHon39M5TTpDZ3hjpxA2Awf+db4FUPyWxliuRUnjAZARgLJRajLVvI0xMJaoleP n/vC1Z+q141qO8z1E8lZ6QNxsTzj4t1rvAOMczOUdlltta28mpeq/iuAy+Sksd2g6xxY fltpuV/4STBO0HxrNnWJ7ysCrjbmws4LORlrzWF0JfTnflL7dtdgSJj4z8rp5AzOlHmT j4nQ/DaSGuHgulSrXKkWvGm4pXL4s9qGYhqMfJcAQxDsUoc6apOstP/xEqQvntgONSo5 6Pr+wBl30wYH4meAQIH9lYVpcI707JaNm8AVgmAn1yEPdcnaECiHc7dTCqtRDFt2BgTm bdag== X-Gm-Message-State: AOAM532IN9f2n902e1mtDkqlGfoEled2NWVGrnkV1HjMqKoPLMxb7kSb eE6589LOhRRKsCkTi4/6RcwMdTonKNXsVAXQPHIKzQ== X-Google-Smtp-Source: ABdhPJxjtYcNvePVfz3PyQvJmmzO6M4fKzwNdZqKua6XH8jtTIXlHJqEXPINzkevucgcTd+1sQSvY97jqlAeG8Lnsx4= X-Received: by 2002:a2e:b6cc:: with SMTP id m12mr837848ljo.401.1611321383077; Fri, 22 Jan 2021 05:16:23 -0800 (PST) MIME-Version: 1.0 References: <20201118082759.1413056-1-bharata@linux.ibm.com> <20210121053003.GB2587010@in.ibm.com> <786571e7-b9a2-4cdb-06d5-aa4a4b439b7e@suse.cz> In-Reply-To: <786571e7-b9a2-4cdb-06d5-aa4a4b439b7e@suse.cz> From: Vincent Guittot Date: Fri, 22 Jan 2021 14:16:11 +0100 Message-ID: Subject: Re: [RFC PATCH v0] mm/slub: Let number of online CPUs determine the slub page order To: Vlastimil Babka Cc: Christoph Lameter , Bharata B Rao , linux-kernel , linux-mm@kvack.org, David Rientjes , Joonsoo Kim , Andrew Morton , guro@fb.com, Shakeel Butt , Johannes Weiner , aneesh.kumar@linux.ibm.com, Jann Horn , Michal Hocko Content-Type: text/plain; charset="UTF-8" X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Fri, 22 Jan 2021 at 13:03, Vlastimil Babka wrote: > > On 1/22/21 9:03 AM, Vincent Guittot wrote: > > On Thu, 21 Jan 2021 at 19:19, Vlastimil Babka wrote: > >> > >> On 1/21/21 11:01 AM, Christoph Lameter wrote: > >> > On Thu, 21 Jan 2021, Bharata B Rao wrote: > >> > > >> >> > The problem is that calculate_order() is called a number of times > >> >> > before secondaries CPUs are booted and it returns 1 instead of 224. > >> >> > This makes the use of num_online_cpus() irrelevant for those cases > >> >> > > >> >> > After adding in my command line "slub_min_objects=36" which equals to > >> >> > 4 * (fls(num_online_cpus()) + 1) with a correct num_online_cpus == 224 > >> >> > , the regression diseapears: > >> >> > > >> >> > 9 iterations of hackbench -l 16000 -g 16: 3.201sec (+/- 0.90%) > >> > >> I'm surprised that hackbench is that sensitive to slab performance, anyway. It's > >> supposed to be a scheduler benchmark? What exactly is going on? > >> > > > > From hackbench description: > > Hackbench is both a benchmark and a stress test for the Linux kernel > > scheduler. It's main > > job is to create a specified number of pairs of schedulable > > entities (either threads or > > traditional processes) which communicate via either sockets or > > pipes and time how long it > > takes for each pair to send data back and forth. > > Yep, so I wonder which slab entities this is stressing that much. > > >> Things would be easier if we could trust *on all arches* either > >> > >> - num_present_cpus() to count what the hardware really physically has during > >> boot, even if not yet onlined, at the time we init slab. This would still not > >> handle later hotplug (probably mostly in a VM scenario, not that somebody would > >> bring bunch of actual new cpu boards to a running bare metal system?). > >> > >> - num_possible_cpus()/nr_cpu_ids not to be excessive (broken BIOS?) on systems > >> where it's not really possible to plug more CPU's. In a VM scenario we could > >> still have an opposite problem, where theoretically "anything is possible" but > >> the virtual cpus are never added later. > > > > On all the system that I have tested num_possible_cpus()/nr_cpu_ids > > were correctly initialized > > > > large arm64 acpi system > > small arm64 DT based system > > VM on x86 system > > So it's just powerpc that has this issue with too large nr_cpu_ids? Is it caused > by bios or the hypervisor? How does num_present_cpus() look there? num_present_cpus() starts to 1 until secondary cpus boot in the arm64 case > > What about heuristic: > - num_online_cpus() > 1 - we trust that and use it > - otherwise nr_cpu_ids > Would that work? Too arbitrary? > > > >> We could also start questioning the very assumption that number of cpus should > >> affect slab page size in the first place. Should it? After all, each CPU will > >> have one or more slab pages privately cached, as we discuss in the other > >> thread... So why make the slab pages also larger? > >> > >> > Or the num_online_cpus needs to be up to date earlier. Why does this issue > >> > not occur on x86? Does x86 have an up to date num_online_cpus earlier? > >> > > >> > > >> > > >