From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F19E2C6FD19 for ; Fri, 10 Mar 2023 22:02:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 528A16B0071; Fri, 10 Mar 2023 17:02:39 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4D9AF6B0072; Fri, 10 Mar 2023 17:02:39 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3A19E6B0074; Fri, 10 Mar 2023 17:02:39 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 2D42F6B0071 for ; Fri, 10 Mar 2023 17:02:39 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id F2A1CC077F for ; Fri, 10 Mar 2023 22:02:38 +0000 (UTC) X-FDA: 80554363638.02.D90CA9B Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) by imf22.hostedemail.com (Postfix) with ESMTP id 0887DC0012 for ; Fri, 10 Mar 2023 22:02:36 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b="Fnl0nq/R"; spf=pass (imf22.hostedemail.com: domain of shy828301@gmail.com designates 209.85.214.175 as permitted sender) smtp.mailfrom=shy828301@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678485757; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=pl9+UnJsuGtSL6Htf5pmFF2VpYUF7uQsk//bCajkwMA=; b=kVK5NJ4FDgcQtyVtwNOjiCIzR5tzv2haemTWNi6/cL/5xDIochg9WTUc0e9Ah0BkeyZkCH Wo+ovOleAr36fpp/hMW1SxIEb5yZYdsDLiKTqC3cFPPWymHCNMXrn1YeFPmdelOG+PmzGm aP+aq14mo92W6gOQuNnI2md0TFUDLfI= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b="Fnl0nq/R"; spf=pass (imf22.hostedemail.com: domain of shy828301@gmail.com designates 209.85.214.175 as permitted sender) smtp.mailfrom=shy828301@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678485757; a=rsa-sha256; cv=none; b=VXP7bfWKwlVZtsOv5z1ENprafvRhXV/lF1fCnrk1KBrBhDg4EJrXbaZXa7A2e/OMxiDcch wrtxoMEiGu2QTOJBFyLjq3NSjICFxVMmJ0SOUGcDo2ACo60lwu0MJCOm1wT2LTFruKfNbv jWeN5DFI3pRLsFUIPLsZRxnwvgmKH7M= Received: by mail-pl1-f175.google.com with SMTP id h8so7070108plf.10 for ; Fri, 10 Mar 2023 14:02:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1678485756; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=pl9+UnJsuGtSL6Htf5pmFF2VpYUF7uQsk//bCajkwMA=; b=Fnl0nq/R0PR8v0EG9jkEbvFgb3uDWFfT/Q6iPUq/0uH3Zvv/NvCKB3krHXm9LccxFu YyV4m+roG57TGhT0gI35AOJ8cPJ420MqCsRi0iHsJV/fswg9d8O1jOvOMTohjmS+4DYf 7G6Hrt+FrMhoOBblRkGvxQoHh4DCrR18Eo4MxC5Rxxf1p8BxjTwIL8w00KKJihPjVN5A /2NLhT2I2Cr/FlKc2U1zeKg58DoHzy67/cQKPu/CfDIo9u3HQG5gl3P0D1Ke7HBuRPQ9 w2alaLB8XALXK7wrf+Gq14wqbN79UwYOgJ6bYfywogEqKTSYtKqiLVTDWBJrxdwpaueQ wcsA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678485756; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=pl9+UnJsuGtSL6Htf5pmFF2VpYUF7uQsk//bCajkwMA=; b=fI+bL2nF+dOPHMAvfr0cj/vjWI6/SsT+r/wPjrvNVZmPbtearzgTqj5AOLnNrc5yKf SoHAdPTBxKHPyUFoLiafVwh25HwPm8DctYd5/QCFfLx5TUbM26MkQG7ug1LyIoU+e5Oq hhSVmbtaOxO3z88kBvKu4YtSlcRt1ijFdtssG7dx6bbeSpCD3aoHWoCMuu5Qshyamkh1 5Ko8uK7vpgP/HDYrYAiUPn5ByOAJIvs028OnH+JpSbgRbe/awLvZXJW2dljLIMp/WC8i 99OWiEbV4S9WNO9fzMEowMSu6sY6+fVqL2aBtKUeHgJvJenijH6oXFrgFMDzPGenV7sH NlAQ== X-Gm-Message-State: AO0yUKUa17tLy2eXTGmQrRP4T/3H2IfB1okL27fKe3Plb5DP5KXhi8et xWXvpPzTS9kAINmhryZgtnwbtz7ZskQrD4mK2EE= X-Google-Smtp-Source: AK7set9293PF3gtVSLcSeKP7/GQJRUx7rj9DrZRTCwLJ7jGarBoSu/FzASHeQcoehpR0SaIfWNpVvSibgsUsbagRsLU= X-Received: by 2002:a17:903:26d4:b0:19a:f9d5:684d with SMTP id jg20-20020a17090326d400b0019af9d5684dmr9923676plb.8.1678485755776; Fri, 10 Mar 2023 14:02:35 -0800 (PST) MIME-Version: 1.0 References: <20230306235730.GA31451@monkey> <20230307004049.GC4956@monkey> <20230308190206.GA4005@monkey> <20230309233340.GC3700@monkey> In-Reply-To: From: Yang Shi Date: Fri, 10 Mar 2023 14:02:23 -0800 Message-ID: Subject: Re: THP backed thread stacks To: "Zach O'Keefe" Cc: Mike Kravetz , Peter Xu , David Hildenbrand , Rik van Riel , Mike Rapoport , linux-mm@kvack.org, linux-kernel@vger.kernel.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 0887DC0012 X-Stat-Signature: hapz7jj1ay4mcos73wnczw6qmrhpqtc7 X-HE-Tag: 1678485756-691434 X-HE-Meta: U2FsdGVkX1/tW/swaYC/aU2OGH2Jaz/4682hwEsbqRSMPVoROwHeN2QmWCQt24mYZob7beuYa/nPveYdA4BCfq1E8u9heTo18jMr7ri01j7K4QDRy4peJK+tz8Io0lxe49lvSjS0d55WTBbQP8d1blAhtn3hH/bygTK7AeDFihD/yFKVPpwUNS3hW14H5MvBpU5PEMeRDWP+eg8NH6ZeRmuP6qORUIJaAj3FEbe8GKPmMoa41JkFkaeUmibte5GPYAflrQGlrc1Apg6gxxUlGxAZk7kgOShLIZYdeq2JrR5IS/RFqYLaadkiq5FxDKQUz7vNhq2JrgHI9Dc3lZn7L1TotJk2dWh0+Wd1+9HTElx9UDnl8tlfV278YYP5WEskzJux/M9Reru0+kmQvTGNhcH+A+ONFzwDojZjIWrE4uaMT9Mvnzy7DynqAacB6n8FgZ3rKhcDa7UF/urwYkEjVkhPBhvl/D3aA8PD1DfPhnOsl8KwSz37cd2XmLKI/0/Ne1pvqNJqbF9NwntnQaHf9Wqp8sCtf0MVnPKIGwORB90UZxs+3WpzodhpLZzfyT4uqyb/WZCYbikw11slGTv3P9Ik3LcZGdX9mghrCZF2yO91MFWfM3JC4z5b8pn8mW2i0fmDFbNUkbKFiuslVwrolY/VWm82pIKYl8FjNfTH2tJbBTWVMit3d1CNEhpoWAVqGSqwHYEBy0/k+at8Cdw2UlUCFnCa8SVYKWhcehpCjvM4Uk9DuzhZXAuAl/TlqFbLJ9Bjwmv4LZB53zniyxVkXaqpQo6aBXsILe7SxR4EMDhUmu6nv/5UaM6PRtorrLE0wkieUx0kvA2mvSvG2fSaygycFJijUPVHKA7Lyecyt5UG+NnbRP8WBc1RD2tzisBj7sm8ueY9+kEcMPRd9A+0AFwhlU1RBmN2YLzF2rD4mairfNFdUHbHF1PtXW+WJEUfTbz51QkZ28aN6k7Endo dqbmJaw+ 5djl5xpJTjnypr/WFoAK8rUJVfPQg6Fuzw+TM1ak2bryHhhe4GTAjR2L7bbW6abomjNEobO5AdB5RC0ow77uOYPnwpYcXK6hGkbrI9F6ifSnsQTcDQFPNmP7/62IG7wfjZ463VrDEI53zeIPJr74YKbdjSb3Hh0eLAo71wR4myG02BMhSBcYfgkHGwH+52u3FrgIbkraX+M+HnPZ9EwtrbZKn9LdcvSKqzSpyyMvQxFPlH9b6ZKw3F+KQu+Hy5sKeqqr1ys5NSuVWY2tjqCiP/vFmfIBaw9CW1ZFYIqAObi3913UfgiS6hE2fhl5mdF+gJH4tEpYQ5v+4hGl4pdgbTv38m5XtnnjCEoh2u2qKA4mL9rhZF/8plWgoZB/Pxzm0w4iW9Pu6gKKn28tJVKwfoYgFVAVCh7kTvCMd X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, Mar 9, 2023 at 4:05=E2=80=AFPM Zach O'Keefe wr= ote: > > On Thu, Mar 9, 2023 at 3:33=E2=80=AFPM Mike Kravetz wrote: > > > > On 03/09/23 14:38, Zach O'Keefe wrote: > > > On Wed, Mar 8, 2023 at 11:02=E2=80=AFAM Mike Kravetz wrote: > > > > > > > > On 03/06/23 16:40, Mike Kravetz wrote: > > > > > On 03/06/23 19:15, Peter Xu wrote: > > > > > > On Mon, Mar 06, 2023 at 03:57:30PM -0800, Mike Kravetz wrote: > > > > > > > > > > > > > > Just wondering if there is anything better or more selective = that can be > > > > > > > done? Does it make sense to have THP backed stacks by defaul= t? If not, > > > > > > > who would be best at disabling? A couple thoughts: > > > > > > > - The kernel could disable huge pages on stacks. libpthread/= glibc pass > > > > > > > the unused flag MAP_STACK. We could key off this and disab= le huge pages. > > > > > > > However, I'm sure there is somebody somewhere today that is= getting better > > > > > > > performance because they have huge pages backing their stac= ks. > > > > > > > - We could push this to glibc/libpthreads and have them use > > > > > > > MADV_NOHUGEPAGE on thread stacks. However, this also has t= he potential > > > > > > > of regressing performance if somebody somewhere is getting = better > > > > > > > performance due to huge pages. > > > > > > > > > > > > Yes it seems it's always not safe to change a default behavior = to me. > > > > > > > > > > > > For stack I really can't tell why it must be different here. I= assume the > > > > > > problem is the wasted space and it exaggerates easily with N-th= reads. But > > > > > > IIUC it'll be the same as thp to normal memories iiuc, e.g., th= ere can be a > > > > > > per-thread mmap() of 2MB even if only 4K is used each, then if = such mmap() > > > > > > is populated by THP for each thread there'll also be a huge was= te. > > > > > > > > I may be alone in my thinking here, but it seems that stacks by the= ir nature > > > > are not generally good candidates for huge pages. I am just thinki= ng about > > > > the 'normal' use case where stacks contain local function data and = arguments. > > > > Am I missing something, or are huge pages really a benefit here? > > > > > > > > Of course, I can imagine some thread with a large amount of frequen= tly > > > > accessed data allocated on it's stack which could benefit from huge > > > > pages. But, this seems to be an exception rather than the rule. > > > > > > > > I understand the argument that THP always means always and everywhe= re. > > > > It just seems that thread stacks may be 'special enough' to conside= r > > > > disabling by default > > > > > > Just my drive-by 2c, but would agree with you here (at least wrt > > > hugepages not being good candidates, in general). A user mmap()'ing > > > memory has a lot more (direct) control over how they fault / utilize > > > the memory: you know when you're running out of space and can map mor= e > > > space as needed. For these stacks, you're setting the stack size to > > > 2MB just as a precaution so you can avoid overflow -- AFAIU there is > > > no intention of using the whole mapping (and looking at some data, > > > it's very likely you won't come close). > > > > > > That said, why bother setting stack attribute to 2MiB in size if ther= e > > > isn't some intention of possibly being THP-backed? Moreover, how did > > > it happen that the mappings were always hugepage-aligned here? > > > > I do not have the details as to why the Java group chose 2MB for stack > > size. My 'guess' is that they are trying to save on virtual space (alt= hough > > that seems silly). 2MB is actually reducing the default size. The > > default pthread stack size on my desktop (fedora) is 8MB [..] > > Oh, that's interesting -- I did not know that. That's huge. > > > [..] This also is > > a nice multiple of THP size. > > > > I think the hugepage alignment in their environment was somewhat luck. > > One suggestion made was to change stack size to avoid alignment and > > hugepage usage. That 'works' but seems kind of hackish. > > That was my first thought, if the alignment was purely due to luck, > and not somebody manually specifying it. Agreed it's kind of hackish > if anyone can get bit by this by sheer luck. > > > Also, David H pointed out the somewhat recent commit to align sufficien= tly > > large mappings to THP boundaries. This is going to make all stacks hug= e > > page aligned. > > I think that change was reverted by Linus in commit 0ba09b173387 > ("Revert "mm: align larger anonymous mappings on THP boundaries""), > until it's perf regressions were better understood -- and I haven't > seen a revamp of it. The regression has been fixed and it is not related to this commit. I suggested Andrew to resurrect this commit a couple of months ago, but it has not been. > > > -- > > Mike Kravetz >