From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B20D1C61DA4 for ; Fri, 10 Mar 2023 00:05:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1162E6B0072; Thu, 9 Mar 2023 19:05:54 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0C5916B0074; Thu, 9 Mar 2023 19:05:54 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id ECEAA6B0075; Thu, 9 Mar 2023 19:05:53 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id DDA566B0072 for ; Thu, 9 Mar 2023 19:05:53 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 958011612DB for ; Fri, 10 Mar 2023 00:05:53 +0000 (UTC) X-FDA: 80551045386.16.D14866A Received: from mail-ed1-f45.google.com (mail-ed1-f45.google.com [209.85.208.45]) by imf04.hostedemail.com (Postfix) with ESMTP id BC7054000A for ; Fri, 10 Mar 2023 00:05:51 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="s/togCRF"; spf=pass (imf04.hostedemail.com: domain of zokeefe@google.com designates 209.85.208.45 as permitted sender) smtp.mailfrom=zokeefe@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678406751; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=TOQ2JwrgXFV05O9tRlxq9mUvvX5cwHSFjxQP49GBiII=; b=hA4/uW9g4EFvQgnMqB/2I1vm7DmlGeovus4U17M52NdqPcv3yx1Y+tYAvVd6dVFOW3l07E 0Sh0faryr+hgn0iz3riGzFmZgmUV+/C0Amrm0QBVOKVItF6TiLwMJmBlzzbPcpB36hrIZv 1zSytzpOhoWmG0+5C5gEBOVCZ2eDazE= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="s/togCRF"; spf=pass (imf04.hostedemail.com: domain of zokeefe@google.com designates 209.85.208.45 as permitted sender) smtp.mailfrom=zokeefe@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678406751; a=rsa-sha256; cv=none; b=AfvqE3Kn8XtmCILtmXZLEm4hQIap3lc9akBLYyprRgUWqvV8WdcN5sC9ielXaJJWJ0MBe2 6e6xr/g09jUO6atcMxJEXGN8keFNsGrLdXLHhdPLYlReCXYAQbjuzr2yFfVAGSUPoOPV1I hqRmrroYxWWfDSn6atHoAejKh1vDFaU= Received: by mail-ed1-f45.google.com with SMTP id ec29so13899563edb.6 for ; Thu, 09 Mar 2023 16:05:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1678406750; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=TOQ2JwrgXFV05O9tRlxq9mUvvX5cwHSFjxQP49GBiII=; b=s/togCRF/U8G5aJGup6+UUy1vUxgTglMyHmabUkcmccX9d1MOs200hR0ZT7xD0I0tR D0vbyE7VzURzFPRaW+7fNph0qGIDKejB9vhKBfXCETcj3GBtSZ4YCHJXdoaGaQVl2vNt /AMYSrr4m/26Yc5Sm8sugPnCFGCTvH6SZ4rcnjjTgFYZ6QqI6NlFEqv4z9sHNpSAHAxC kZ0e4NETFqkqAR5eqD5orN1/V+uox5TZSucGENw1KFZVMehbKNITeU4f8ImpWYvTxmYH EinLcP1hhyyOxjgoCFvZKmFIy8rPm6oOdn3pLqjOcAjG0EWnwzkY8d/r9Rry53bJAh4G UAlg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678406750; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=TOQ2JwrgXFV05O9tRlxq9mUvvX5cwHSFjxQP49GBiII=; b=tiGiyb6EcO0E/JehyiRlic+cXzbMkVZ3NC5SQQQHOPMA83MRnyf8dy3OvzxNpBUSZ5 Vw42EdCaV/4zhf8oC/q6Gztub3Bm047SxGMld54TIacxPWyjV9gbHa2ygXbZdo1WWwoP BpaopcoZs5SzfNsYscxNUOsPTEShLAwzxb4JDEv0YesavVQ9WH4oIUx9Ji0cOcCw/c6l SOHPJD8q++wpktfRsbel9SpisuNp1hADdkElPZjIV6q+plSNIVDZwWBJCJFim+YmrtU1 l5qc1g/eGAe2+C07NDjLcxKQoYG8Quv1OFE4hg15A0LN27B0oFrdgKPGvUtDDRxVHup8 1c/A== X-Gm-Message-State: AO0yUKX6JfJUAkqaWbmM3lfD0Ak3vCBZnm7iKcW8EeUZPRVq1j0NKksy KEt4l5Th6HzCDZJJYGDqPvkUpXGR0KTpXnjmSdxxcA== X-Google-Smtp-Source: AK7set/v4KpK5OJys67IvSPsijPpyucP+9dcxti8BzahB/SHxHqNo5m2OHgr4lu195qf+XV87sPuXopsDYEaFZ8Ur5M= X-Received: by 2002:a17:906:7803:b0:8db:b5c1:7203 with SMTP id u3-20020a170906780300b008dbb5c17203mr12143219ejm.11.1678406749957; Thu, 09 Mar 2023 16:05:49 -0800 (PST) MIME-Version: 1.0 References: <20230306235730.GA31451@monkey> <20230307004049.GC4956@monkey> <20230308190206.GA4005@monkey> <20230309233340.GC3700@monkey> In-Reply-To: <20230309233340.GC3700@monkey> From: "Zach O'Keefe" Date: Thu, 9 Mar 2023 16:05:13 -0800 Message-ID: Subject: Re: THP backed thread stacks To: Mike Kravetz Cc: Peter Xu , David Hildenbrand , Rik van Riel , Mike Rapoport , linux-mm@kvack.org, linux-kernel@vger.kernel.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: BC7054000A X-Stat-Signature: 71jz95tf9jgg1kyrhwfubfaetoizh7ei X-HE-Tag: 1678406751-216808 X-HE-Meta: U2FsdGVkX182TXhSCoPt9s81utepG6i4JETcTycJ8OPQq1MgKiSjhXxDXrjZtOGBCRAv92XVkV5fkNc14osZSEMPl+jAlE4rDtpL+i/sX+4+VOlPIOC13Ds00q67QGW7OPw/KLcp63cT28rLdrd1kcBX0+0onhraowtJnnFlHbhsh0JbpFdoudg02W4JO+axs0Xin59BN0PY+8pfX2HLWQkx044jARtiRJIosF1u2pC+2Jwy43AlhsMjsVQsUlDpWH0tU21vifiPpr9YAxu7vtidfAdSO3F5WAWegJD6S2zWC+/l6UeemW5NaMRSv7el0MqoVfF0q/ETgVq0M5iZYgixltr+tArlr9hd3bcxrxzreI4BxHNhdGxWA4aYFT4+2eClhzg4BLCFwKF7OsRfp6VIsG1i+DW5fJujktQLURXCFusR6KtOsDVwireCNirHu6LZ7NT6zDD0anX3EFfREefUgAUK7YiqFcDhttfZzg5BO4/G+AL8ZB4M1AY1AUh94mmnQuNvc5uZMunbYheK8+VYm8Wiowd4X84y656wHwManKuylXVNxwQSzxCl8BzjGkDSYCNUOsq5F6aoGoLKiG8I57Lx0WdPZ+xGP7OSve3HY9dT/K+f1uXFqgLjklig0hGatOu2YFVvcHGSOH7iLT8Ux9mgu2FZCXv+zZS1aLMfeqDpWco22SmAEibBwfPhz1jR457nxvqlsWJoNdd4Q0NXW1ig7AXqf2rroQLX7QDlK2CTkjb3crY+n9PJaRjuxh35BpN/Wt/WCkB9qL/viNWF8MIvg2jPyUMhrhBD0OAKtor+1UNyRNPrDf5bkc0K2zFGsSuoCFn8ck/ySR7WwblI7LOxjEU/ZgI2Gfs1u4RcAuzR+QFe7rv1/wLHPwG+BEQyFdpgaAs/8UFrGDMlSvXM3jQBREy8MXVFxmOiPotSDXYlCQLi2tIqjnh/W6zEL0Q7NjnGEvIX/GN3Z3a OW+EAeWo V5FKm5qlZQrou2AiDtG9LvMTRHSmMGgrVhPPnbUa/APj7mFXXbsyGLwODuJc902MQ13RtWgIEgk9AVYVqoyzjstjhHcO92JlMHkbf/gAT+PiNCNF/03LzsF1g2dY/PdMoIFoIjqwubQf9XaoXFDKUrWbqvH6BOGgu1TfS3wjyUjpuePPwLloKjdM4cZEtNjMUliS7nD2Dl5gLB7AiZ3pKxnf18A6qEK2j4OGgry0U/9BDgqP7px/xxjK7hYMLC2LILe98oUOJ2VtmQto= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, Mar 9, 2023 at 3:33=E2=80=AFPM Mike Kravetz wrote: > > On 03/09/23 14:38, Zach O'Keefe wrote: > > On Wed, Mar 8, 2023 at 11:02=E2=80=AFAM Mike Kravetz wrote: > > > > > > On 03/06/23 16:40, Mike Kravetz wrote: > > > > On 03/06/23 19:15, Peter Xu wrote: > > > > > On Mon, Mar 06, 2023 at 03:57:30PM -0800, Mike Kravetz wrote: > > > > > > > > > > > > Just wondering if there is anything better or more selective th= at can be > > > > > > done? Does it make sense to have THP backed stacks by default?= If not, > > > > > > who would be best at disabling? A couple thoughts: > > > > > > - The kernel could disable huge pages on stacks. libpthread/gl= ibc pass > > > > > > the unused flag MAP_STACK. We could key off this and disable= huge pages. > > > > > > However, I'm sure there is somebody somewhere today that is g= etting better > > > > > > performance because they have huge pages backing their stacks= . > > > > > > - We could push this to glibc/libpthreads and have them use > > > > > > MADV_NOHUGEPAGE on thread stacks. However, this also has the= potential > > > > > > of regressing performance if somebody somewhere is getting be= tter > > > > > > performance due to huge pages. > > > > > > > > > > Yes it seems it's always not safe to change a default behavior to= me. > > > > > > > > > > For stack I really can't tell why it must be different here. I a= ssume the > > > > > problem is the wasted space and it exaggerates easily with N-thre= ads. But > > > > > IIUC it'll be the same as thp to normal memories iiuc, e.g., ther= e can be a > > > > > per-thread mmap() of 2MB even if only 4K is used each, then if su= ch mmap() > > > > > is populated by THP for each thread there'll also be a huge waste= . > > > > > > I may be alone in my thinking here, but it seems that stacks by their= nature > > > are not generally good candidates for huge pages. I am just thinking= about > > > the 'normal' use case where stacks contain local function data and ar= guments. > > > Am I missing something, or are huge pages really a benefit here? > > > > > > Of course, I can imagine some thread with a large amount of frequentl= y > > > accessed data allocated on it's stack which could benefit from huge > > > pages. But, this seems to be an exception rather than the rule. > > > > > > I understand the argument that THP always means always and everywhere= . > > > It just seems that thread stacks may be 'special enough' to consider > > > disabling by default > > > > Just my drive-by 2c, but would agree with you here (at least wrt > > hugepages not being good candidates, in general). A user mmap()'ing > > memory has a lot more (direct) control over how they fault / utilize > > the memory: you know when you're running out of space and can map more > > space as needed. For these stacks, you're setting the stack size to > > 2MB just as a precaution so you can avoid overflow -- AFAIU there is > > no intention of using the whole mapping (and looking at some data, > > it's very likely you won't come close). > > > > That said, why bother setting stack attribute to 2MiB in size if there > > isn't some intention of possibly being THP-backed? Moreover, how did > > it happen that the mappings were always hugepage-aligned here? > > I do not have the details as to why the Java group chose 2MB for stack > size. My 'guess' is that they are trying to save on virtual space (altho= ugh > that seems silly). 2MB is actually reducing the default size. The > default pthread stack size on my desktop (fedora) is 8MB [..] Oh, that's interesting -- I did not know that. That's huge. > [..] This also is > a nice multiple of THP size. > > I think the hugepage alignment in their environment was somewhat luck. > One suggestion made was to change stack size to avoid alignment and > hugepage usage. That 'works' but seems kind of hackish. That was my first thought, if the alignment was purely due to luck, and not somebody manually specifying it. Agreed it's kind of hackish if anyone can get bit by this by sheer luck. > Also, David H pointed out the somewhat recent commit to align sufficientl= y > large mappings to THP boundaries. This is going to make all stacks huge > page aligned. I think that change was reverted by Linus in commit 0ba09b173387 ("Revert "mm: align larger anonymous mappings on THP boundaries""), until it's perf regressions were better understood -- and I haven't seen a revamp of it. > -- > Mike Kravetz