From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E7B03C678D5 for ; Tue, 7 Mar 2023 13:36:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 53F46280001; Tue, 7 Mar 2023 08:36:56 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4EE066B0072; Tue, 7 Mar 2023 08:36:56 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3B728280001; Tue, 7 Mar 2023 08:36:56 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 2A59C6B0071 for ; Tue, 7 Mar 2023 08:36:56 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id ECE4980A5A for ; Tue, 7 Mar 2023 13:36:55 +0000 (UTC) X-FDA: 80542202790.14.F9352AC Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf26.hostedemail.com (Postfix) with ESMTP id 3C4D3140003 for ; Tue, 7 Mar 2023 13:36:54 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=vDcIIeHL; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf26.hostedemail.com: domain of rppt@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=rppt@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678196214; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=dQRd5Ew5AlU7NMZURRS5Fao1/FjkoLGh++V2ZVV+AKw=; b=3+ntSNDsMVMgTZbZ+KYKCNzzrAbkn5JjHe8EMSQIn29ha/urU9GNCbHlaj9pDN4l3TsHcU OnluiHq7VE6bwfeGlgKh0sRG/IoJPaT2sYr02D7i74d1NqFxTev2cUvvjgKXdW7Lq5RjGy ohX2ajDdAc1amB+hVe3gE1gMgl49hmg= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=vDcIIeHL; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf26.hostedemail.com: domain of rppt@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=rppt@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678196214; a=rsa-sha256; cv=none; b=ew+c428htFWSMpgsEyLjPyVsqWRffyNv5OTEEESBOXCSrmP5gV729mdeparJoK3QzQfGmJ 5tRVHNmku4PFuIdsGOTLle92SlxebdA/ZcF7ltfpMCrHsmyLYFLflSf4o5o2/nreroaZEx UpEGD4zupwvnPm1cphA9GjHsI8Yx5tE= Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 2164A6144B; Tue, 7 Mar 2023 13:36:53 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8B82DC433EF; Tue, 7 Mar 2023 13:36:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1678196211; bh=kVKeC2EhtKsgrAU3QzvKrOOsQ+sn27416WSfb4QStJU=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=vDcIIeHLgDvJ2Qam8i1aV+8JWgsuVVXgZb0F6Yz2Sa33GgpyNSVtdDB8HwmYpLY1N +ZWXqlXfanaFDjA7xEZNmhBVdjHDRfvkoC8x3n8iadUvbtAhnTgHbXlfX9LojgUfKW WO9jmRrEAwpvy0sbYMIoocFnfV2ww7ymrL5uZG9ygHTRM4Qi86+lCZXwahaXA7ktyb h9bTo8PSwdJgYPMDF/D5L+f0BFtMQ0G4VbkoFjezjR9uJoAdms8LRQRanUJ0qBJjom psaWqHzoSJQ+BNEJkSX2VE7+zwHJykc0Ujy2AslAJkOaiEEvDuCttByrnJeKNmvP98 tRM4C+ldLj68w== Date: Tue, 7 Mar 2023 15:36:40 +0200 From: Mike Rapoport To: Mike Kravetz Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: Re: THP backed thread stacks Message-ID: References: <20230306235730.GA31451@monkey> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20230306235730.GA31451@monkey> X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 3C4D3140003 X-Stat-Signature: da9rxmqiq51frn9er33kmdza8mwmojst X-HE-Tag: 1678196214-742882 X-HE-Meta: U2FsdGVkX1/ecVfzsyDD6NwuzuWvjMXSICO0fgFTWBSiAJr6UHLf6UHXQ9UQJhIuf5D6cKQiyEdgqbb91GkkCob52srNhHnRmDG2g0NCJp9oOzoVk/UcAZQDztiKK3KIbdZXA/9QC3VNVMDRX7ZtjbR8LSCHHLEpzE+49luyDwuYUHiTTzMQw/Ir88ewn73totiy2efaXWd+PKy4JDtvlrSzVBhAfu1g2EuovEnaD/7RHEX97bD/lCc9blA+li5DpMHe+HmE0Ox1jMTPJQ/Ia0LqFoy5lcujgPVeMIBzhbEDj/6nyhjwPZoF/lRaQ+BH2koPfiE/JJih33hB3GFqBQDofa6chdYtJMIhNqGmiOwpLa64Z52E/lyqgwlCKoFc1ZjA6y9sOsWONBI8Ljj5A+6pjimMwgcRs5bsqjqDlq06gVpKnWbHGIIfZ0PKXvFFRHuBB2lpThZ93VMJPwQdjMenmzf5O7Z1fI3A8a5idsZlMoNhVdKOrzLtH3khbnbRQ9EhH/w0cVbsMlHWeik+0d/5XBhg3LByh7Da4VfZTXEw1oLST0r2aqiZYlDc1Fjww3n9+k7+JVbl2i2cSu3/ucXQB/kBcGoKW2auJmSvbFDiyfclTONIjJ5Lg/vPgPgSGhwqZRJvgwhjh0ZfPRu4O00N/EYmYGQ6DkoFjB9EHziMR5WlatBP6SPkNID0vvcg1JHvfjC/WzlnDDSdQpLbJVqRApC1wzH7R4XFw+HCjvc+jtXwl0H+xURvOZz6W0vnErW1RRE+RlA2B4PxLRoHiqNGUalcdv85wsUNk6Dj13qpnpqorz3VUh6xpIXDB9MHXUYLug0eL6BCogi062FzEUry/hKOqVYQT5RCPc/FiE3Y6dFwzdvPbFm8SHLwBVQ5QZgUYL4U3VHCV9P+BBQ+m/m6TANXwAT/Mu9UWSx5kJ2K1/UtwCrBgHnnczjntgJIePQlE3YChwC4NBCB3kq UrSXLEie vuq9ybL3HJFJB6EI/1tga2Xy4lsAMPrUext5EPFQdJMADKHhwj3zozVgiFPf+G3lQCFWBhsFr75zlfI+j3Vy6gmm6C161qOJ7hegHb1S0PEIXASyQNw94gRLD+UjsZJkkD1v4OSw3K1ZKVWQ3vqGSPp/MTcY96+lXDvYBy8wpwMbqV57s6XvhFxPkHHg9NcCjLa8FORg6EXSETUs9KCMVh5J6dZSTIgTwjGJHzdhVNVZvT6e9ve0eyUGTyATL0SR66+bvU+BlGkUcYrw0glMGCl5kC8jnUqA7JU6ccUEDNZJzkZdbBgzzsn+J4Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hi Mike, On Mon, Mar 06, 2023 at 03:57:30PM -0800, Mike Kravetz wrote: > One of our product teams recently experienced 'memory bloat' in their > environment. The application in this environment is the JVM which > creates hundreds of threads. Threads are ultimately created via > pthread_create which also creates the thread stacks. pthread attributes > are modified so that stacks are 2MB in size. It just so happens that > due to allocation patterns, all their stacks are at 2MB boundaries. The > system has THP always set, so a huge page is allocated at the first > (write) fault when libpthread initializes the stack. > > It would seem that this is expected behavior. If you set THP always, > you may get huge pages anywhere. > > However, I can't help but think that backing stacks with huge pages by > default may not be the right thing to do. Stacks by their very nature > grow in somewhat unpredictable ways over time. Using a large virtual > space so that memory is allocated as needed is the desired behavior. > > The only way to address their 'memory bloat' via thread stacks today is > by switching THP to madvise. > > Just wondering if there is anything better or more selective that can be > done? Does it make sense to have THP backed stacks by default? If not, > who would be best at disabling? A couple thoughts: > - The kernel could disable huge pages on stacks. libpthread/glibc pass > the unused flag MAP_STACK. We could key off this and disable huge pages. > However, I'm sure there is somebody somewhere today that is getting better > performance because they have huge pages backing their stacks. > - We could push this to glibc/libpthreads and have them use > MADV_NOHUGEPAGE on thread stacks. However, this also has the potential > of regressing performance if somebody somewhere is getting better > performance due to huge pages. > - Other thoughts? Push this to the application? :) Something like pthread_attr_getstack() + madvice(MADV_NOHUGEPAGE) will do the job, no? > Perhaps this is just expected behavior of THP always which is unfortunate > in this situation. > -- > Mike Kravetz > -- Sincerely yours, Mike.