From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2D55DEB64D9 for ; Mon, 10 Jul 2023 08:45:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233236AbjGJIpA (ORCPT ); Mon, 10 Jul 2023 04:45:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51940 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233293AbjGJInA (ORCPT ); Mon, 10 Jul 2023 04:43:00 -0400 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id E1A5319A5 for ; Mon, 10 Jul 2023 01:41:44 -0700 (PDT) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 219322B; Mon, 10 Jul 2023 01:42:02 -0700 (PDT) Received: from [10.57.77.63] (unknown [10.57.77.63]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 2505B3F740; Mon, 10 Jul 2023 01:41:18 -0700 (PDT) Message-ID: Date: Mon, 10 Jul 2023 09:41:16 +0100 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0) Gecko/20100101 Thunderbird/102.12.0 Subject: Re: [PATCH v2 4/5] mm: FLEXIBLE_THP for improved performance To: David Hildenbrand , Matthew Wilcox Cc: "Huang, Ying" , Andrew Morton , "Kirill A. Shutemov" , Yin Fengwei , Yu Zhao , Catalin Marinas , Will Deacon , Anshuman Khandual , Yang Shi , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org References: <20230703135330.1865927-1-ryan.roberts@arm.com> <20230703135330.1865927-5-ryan.roberts@arm.com> <87edlkgnfa.fsf@yhuang6-desk2.ccr.corp.intel.com> <44e60630-5e9d-c8df-ab79-cb0767de680e@arm.com> <524bacd2-4a47-2b8b-6685-c46e31a01631@redhat.com> <1e406f04-78ef-6573-e1f1-f0d0e0d5246a@redhat.com> <9dd036a8-9ba3-0cc4-b791-cb3178237728@arm.com> From: Ryan Roberts In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 07/07/2023 20:06, David Hildenbrand wrote: >>>> I still feel that it would be better for the thp and large anon folio controls >>>> to be independent though - what's the argument for tying them together? >>> >>> Thinking about desired 2 MiB flexible THP on aarch64 (64k kernel) vs, 2 MiB PMD >>> THP on aarch64 (4k kernel), how are they any different? Just the way they are >>> mapped ... >> >> The last patch in the series shows my current approach to that: >> >> int arch_wants_pte_order(struct vm_area_struct *vma) >> { >>     if (hugepage_vma_check(vma, vma->vm_flags, false, true, true)) >>         return CONFIG_ARM64_PTE_ORDER_THP; <<< always the contpte size >>     else >>         return CONFIG_ARM64_PTE_ORDER_NOTHP; <<< limited to 64K >> } >> >> But Yu has raised concerns that this type of policy needs to be in the core mm. >> So we could have the arch blindly return the preferred order from HW perspective >> (which would be contpte size for arm64). Then for !hugepage_vma_check(), mm >> could take the min of that value and some determined "acceptable" limit (which >> in my mind is 64K ;-). > > Yeah, it's really tricky. Because why should arm64 with 64k base pages *not* > return 2MiB (which is one possible cont-pte size IIRC) ? > > I share the idea that 64k might *currently* on *some platforms* be a reasonable > choice. But that's where the "fun" begins. > >> >>> >>> It's easy to say "64k vs. 2 MiB" is a difference and we want separate controls, >>> but how is "2MiB vs. 2 MiB" different? >>> >>> Having that said, I think we have to make up our mind how much control we want >>> to give user space. Again, the "2MiB vs. 2 MiB" case nicely shows that it's not >>> trivial: memory waste is a real issue on some systems where we limit THP to >>> madvise(). >>> >>> >>> Just throwing it out for discussing: >>> >>> What about keeping the "all / madvise / never" semantics (and MADV_NOHUGEPAGE >>> ...) but having an additional config knob that specifies in which cases we >>> *still* allow flexible THP even though the system was configured for "madvise". >>> >>> I can't come up with a good name for that, but something like >>> "max_auto_size=64k" could be something reasonable to set. We could have an >>> arch+hw specific default. >> >> Ahha, yes, that's essentially what I have above. I personally also like the idea >> of the limit being an absolute value rather than an order. Although I know Yu >> feels differently (see [1]). > > Exposed to user space I think it should be a human-readable value. Inside the > kernel, I don't particularly care. My point was less about human-readable vs not. It was about expressing a value that is relative to the base page size vs expressing a value that is independent of base page size. If the concern is about limiting internal fragmentation, I think its the absolute size that matters. > > (Having databases/VMs on arch64 with 64k in mind) I think it might be > interesting to have something like the following: > > thp=madvise > max_auto_size=64k/128k/256k > > > So in MADV_HUGEPAGE VMAs (such as under QEMU), we'd happily take any flexible > THP, especially ones < PMD THP (512 MiB) as well. 2 MiB or 4 MiB THP? sure, give > them to my VM. You're barely going to find 512 MiB THP either way in practice .... > > But for the remainder of my system, just do something reasonable and don't go > crazy on the memory waste. Yep, we're on the same page. I've got a v3 that's almost ready to go, based on Yu's prevuous round of review. I'm going to encorporate this mechanism into it then post hopefully later in the week. Now I just need to figure out a decent name for the max_auto_size control... > > > I'll try reading all the previous discussions next week. > From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 389B4EB64DC for ; Mon, 10 Jul 2023 08:42:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:From:References:Cc:To: Subject:MIME-Version:Date:Message-ID:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=jDOyfrE9Sel2YPkmxsMod/As3Br7oGRxpG7lIvRyCSk=; b=rnJvk3SrIrBykf 8/M+tncZOx76xBIgXaVxcbIQcnC72S3L7g1FhJIK+kcr6ssyMbt04Xcwh4Saaok6ZtyWQ376r22Mz jhrOsde112QTfdvDGRvM4Gh8AfhNDzExwnd7ORSTBBdJVyw2URNHEAV01Kd8WKRJzv5OPPLL34rY/ gxZDtZIBHqPi/hCWEUSfKSI53HnYJ4NdtWaIaYCcbP8pzLk90kvYadVmrs7irmvc2X2+ZZj0yw0vV R6OTOelGdIWpXWbeR2v+qM4nlAnCHRTggmagjOVS5Kmm/yqey2ZZvPlsLN/L3u5TtIXEDDJMwpndi Emx94mEi9XGcgV3/IW+g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qImSZ-00AseF-1E; Mon, 10 Jul 2023 08:41:31 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qImSQ-00AsUx-2R for linux-arm-kernel@lists.infradead.org; Mon, 10 Jul 2023 08:41:25 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 219322B; Mon, 10 Jul 2023 01:42:02 -0700 (PDT) Received: from [10.57.77.63] (unknown [10.57.77.63]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 2505B3F740; Mon, 10 Jul 2023 01:41:18 -0700 (PDT) Message-ID: Date: Mon, 10 Jul 2023 09:41:16 +0100 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0) Gecko/20100101 Thunderbird/102.12.0 Subject: Re: [PATCH v2 4/5] mm: FLEXIBLE_THP for improved performance To: David Hildenbrand , Matthew Wilcox Cc: "Huang, Ying" , Andrew Morton , "Kirill A. Shutemov" , Yin Fengwei , Yu Zhao , Catalin Marinas , Will Deacon , Anshuman Khandual , Yang Shi , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org References: <20230703135330.1865927-1-ryan.roberts@arm.com> <20230703135330.1865927-5-ryan.roberts@arm.com> <87edlkgnfa.fsf@yhuang6-desk2.ccr.corp.intel.com> <44e60630-5e9d-c8df-ab79-cb0767de680e@arm.com> <524bacd2-4a47-2b8b-6685-c46e31a01631@redhat.com> <1e406f04-78ef-6573-e1f1-f0d0e0d5246a@redhat.com> <9dd036a8-9ba3-0cc4-b791-cb3178237728@arm.com> From: Ryan Roberts In-Reply-To: X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230710_014122_896172_43690FFB X-CRM114-Status: GOOD ( 29.00 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org T24gMDcvMDcvMjAyMyAyMDowNiwgRGF2aWQgSGlsZGVuYnJhbmQgd3JvdGU6Cj4+Pj4gSSBzdGls bCBmZWVsIHRoYXQgaXQgd291bGQgYmUgYmV0dGVyIGZvciB0aGUgdGhwIGFuZCBsYXJnZSBhbm9u IGZvbGlvIGNvbnRyb2xzCj4+Pj4gdG8gYmUgaW5kZXBlbmRlbnQgdGhvdWdoIC0gd2hhdCdzIHRo ZSBhcmd1bWVudCBmb3IgdHlpbmcgdGhlbSB0b2dldGhlcj8KPj4+Cj4+PiBUaGlua2luZyBhYm91 dCBkZXNpcmVkIDIgTWlCIGZsZXhpYmxlIFRIUCBvbiBhYXJjaDY0ICg2NGsga2VybmVsKSB2cywg MiBNaUIgUE1ECj4+PiBUSFAgb24gYWFyY2g2NCAoNGsga2VybmVsKSwgaG93IGFyZSB0aGV5IGFu eSBkaWZmZXJlbnQ/IEp1c3QgdGhlIHdheSB0aGV5IGFyZQo+Pj4gbWFwcGVkIC4uLgo+Pgo+PiBU aGUgbGFzdCBwYXRjaCBpbiB0aGUgc2VyaWVzIHNob3dzIG15IGN1cnJlbnQgYXBwcm9hY2ggdG8g dGhhdDoKPj4KPj4gaW50IGFyY2hfd2FudHNfcHRlX29yZGVyKHN0cnVjdCB2bV9hcmVhX3N0cnVj dCAqdm1hKQo+PiB7Cj4+IMKgwqDCoMKgaWYgKGh1Z2VwYWdlX3ZtYV9jaGVjayh2bWEsIHZtYS0+ dm1fZmxhZ3MsIGZhbHNlLCB0cnVlLCB0cnVlKSkKPj4gwqDCoMKgwqDCoMKgwqAgcmV0dXJuIENP TkZJR19BUk02NF9QVEVfT1JERVJfVEhQOyA8PDwgYWx3YXlzIHRoZSBjb250cHRlIHNpemUKPj4g wqDCoMKgwqBlbHNlCj4+IMKgwqDCoMKgwqDCoMKgIHJldHVybiBDT05GSUdfQVJNNjRfUFRFX09S REVSX05PVEhQOyA8PDwgbGltaXRlZCB0byA2NEsKPj4gfQo+Pgo+PiBCdXQgWXUgaGFzIHJhaXNl ZCBjb25jZXJucyB0aGF0IHRoaXMgdHlwZSBvZiBwb2xpY3kgbmVlZHMgdG8gYmUgaW4gdGhlIGNv cmUgbW0uCj4+IFNvIHdlIGNvdWxkIGhhdmUgdGhlIGFyY2ggYmxpbmRseSByZXR1cm4gdGhlIHBy ZWZlcnJlZCBvcmRlciBmcm9tIEhXIHBlcnNwZWN0aXZlCj4+ICh3aGljaCB3b3VsZCBiZSBjb250 cHRlIHNpemUgZm9yIGFybTY0KS4gVGhlbiBmb3IgIWh1Z2VwYWdlX3ZtYV9jaGVjaygpLCBtbQo+ PiBjb3VsZCB0YWtlIHRoZSBtaW4gb2YgdGhhdCB2YWx1ZSBhbmQgc29tZSBkZXRlcm1pbmVkICJh Y2NlcHRhYmxlIiBsaW1pdCAod2hpY2gKPj4gaW4gbXkgbWluZCBpcyA2NEsgOy0pLgo+IAo+IFll YWgsIGl0J3MgcmVhbGx5IHRyaWNreS4gQmVjYXVzZSB3aHkgc2hvdWxkIGFybTY0IHdpdGggNjRr IGJhc2UgcGFnZXMgKm5vdCoKPiByZXR1cm4gMk1pQiAod2hpY2ggaXMgb25lIHBvc3NpYmxlIGNv bnQtcHRlIHNpemUgSUlSQykgPwo+IAo+IEkgc2hhcmUgdGhlIGlkZWEgdGhhdCA2NGsgbWlnaHQg KmN1cnJlbnRseSogb24gKnNvbWUgcGxhdGZvcm1zKiBiZSBhIHJlYXNvbmFibGUKPiBjaG9pY2Uu IEJ1dCB0aGF0J3Mgd2hlcmUgdGhlICJmdW4iIGJlZ2lucy4KPiAKPj4KPj4+Cj4+PiBJdCdzIGVh c3kgdG8gc2F5ICI2NGsgdnMuIDIgTWlCIiBpcyBhIGRpZmZlcmVuY2UgYW5kIHdlIHdhbnQgc2Vw YXJhdGUgY29udHJvbHMsCj4+PiBidXQgaG93IGlzICIyTWlCIHZzLiAyIE1pQiIgZGlmZmVyZW50 Pwo+Pj4KPj4+IEhhdmluZyB0aGF0IHNhaWQsIEkgdGhpbmsgd2UgaGF2ZSB0byBtYWtlIHVwIG91 ciBtaW5kIGhvdyBtdWNoIGNvbnRyb2wgd2Ugd2FudAo+Pj4gdG8gZ2l2ZSB1c2VyIHNwYWNlLiBB Z2FpbiwgdGhlICIyTWlCIHZzLiAyIE1pQiIgY2FzZSBuaWNlbHkgc2hvd3MgdGhhdCBpdCdzIG5v dAo+Pj4gdHJpdmlhbDogbWVtb3J5IHdhc3RlIGlzIGEgcmVhbCBpc3N1ZSBvbiBzb21lIHN5c3Rl bXMgd2hlcmUgd2UgbGltaXQgVEhQIHRvCj4+PiBtYWR2aXNlKCkuCj4+Pgo+Pj4KPj4+IEp1c3Qg dGhyb3dpbmcgaXQgb3V0IGZvciBkaXNjdXNzaW5nOgo+Pj4KPj4+IFdoYXQgYWJvdXQga2VlcGlu ZyB0aGUgImFsbCAvIG1hZHZpc2UgLyBuZXZlciIgc2VtYW50aWNzIChhbmQgTUFEVl9OT0hVR0VQ QUdFCj4+PiAuLi4pIGJ1dCBoYXZpbmcgYW4gYWRkaXRpb25hbCBjb25maWcga25vYiB0aGF0IHNw ZWNpZmllcyBpbiB3aGljaCBjYXNlcyB3ZQo+Pj4gKnN0aWxsKiBhbGxvdyBmbGV4aWJsZSBUSFAg ZXZlbiB0aG91Z2ggdGhlIHN5c3RlbSB3YXMgY29uZmlndXJlZCBmb3IgIm1hZHZpc2UiLgo+Pj4K Pj4+IEkgY2FuJ3QgY29tZSB1cCB3aXRoIGEgZ29vZCBuYW1lIGZvciB0aGF0LCBidXQgc29tZXRo aW5nIGxpa2UKPj4+ICJtYXhfYXV0b19zaXplPTY0ayIgY291bGQgYmUgc29tZXRoaW5nIHJlYXNv bmFibGUgdG8gc2V0LiBXZSBjb3VsZCBoYXZlIGFuCj4+PiBhcmNoK2h3IHNwZWNpZmljIGRlZmF1 bHQuCj4+Cj4+IEFoaGEsIHllcywgdGhhdCdzIGVzc2VudGlhbGx5IHdoYXQgSSBoYXZlIGFib3Zl LiBJIHBlcnNvbmFsbHkgYWxzbyBsaWtlIHRoZSBpZGVhCj4+IG9mIHRoZSBsaW1pdCBiZWluZyBh biBhYnNvbHV0ZSB2YWx1ZSByYXRoZXIgdGhhbiBhbiBvcmRlci4gQWx0aG91Z2ggSSBrbm93IFl1 Cj4+IGZlZWxzIGRpZmZlcmVudGx5IChzZWUgWzFdKS4KPiAKPiBFeHBvc2VkIHRvIHVzZXIgc3Bh Y2UgSSB0aGluayBpdCBzaG91bGQgYmUgYSBodW1hbi1yZWFkYWJsZSB2YWx1ZS4gSW5zaWRlIHRo ZQo+IGtlcm5lbCwgSSBkb24ndCBwYXJ0aWN1bGFybHkgY2FyZS4KCk15IHBvaW50IHdhcyBsZXNz IGFib3V0IGh1bWFuLXJlYWRhYmxlIHZzIG5vdC4gSXQgd2FzIGFib3V0IGV4cHJlc3NpbmcgYSB2 YWx1ZQp0aGF0IGlzIHJlbGF0aXZlIHRvIHRoZSBiYXNlIHBhZ2Ugc2l6ZSB2cyBleHByZXNzaW5n IGEgdmFsdWUgdGhhdCBpcyBpbmRlcGVuZGVudApvZiBiYXNlIHBhZ2Ugc2l6ZS4gSWYgdGhlIGNv bmNlcm4gaXMgYWJvdXQgbGltaXRpbmcgaW50ZXJuYWwgZnJhZ21lbnRhdGlvbiwgSQp0aGluayBp dHMgdGhlIGFic29sdXRlIHNpemUgdGhhdCBtYXR0ZXJzLgoKPiAKPiAoSGF2aW5nIGRhdGFiYXNl cy9WTXMgb24gYXJjaDY0IHdpdGggNjRrIGluIG1pbmQpIEkgdGhpbmsgaXQgbWlnaHQgYmUKPiBp bnRlcmVzdGluZyB0byBoYXZlIHNvbWV0aGluZyBsaWtlIHRoZSBmb2xsb3dpbmc6Cj4gCj4gdGhw PW1hZHZpc2UKPiBtYXhfYXV0b19zaXplPTY0ay8xMjhrLzI1NmsKPiAKPiAKPiBTbyBpbiBNQURW X0hVR0VQQUdFIFZNQXMgKHN1Y2ggYXMgdW5kZXIgUUVNVSksIHdlJ2QgaGFwcGlseSB0YWtlIGFu eSBmbGV4aWJsZQo+IFRIUCwgZXNwZWNpYWxseSBvbmVzIDwgUE1EIFRIUCAoNTEyIE1pQikgYXMg d2VsbC4gMiBNaUIgb3IgNCBNaUIgVEhQPyBzdXJlLCBnaXZlCj4gdGhlbSB0byBteSBWTS4gWW91 J3JlIGJhcmVseSBnb2luZyB0byBmaW5kIDUxMiBNaUIgVEhQIGVpdGhlciB3YXkgaW4gcHJhY3Rp Y2UgLi4uLgo+IAo+IEJ1dCBmb3IgdGhlIHJlbWFpbmRlciBvZiBteSBzeXN0ZW0sIGp1c3QgZG8g c29tZXRoaW5nIHJlYXNvbmFibGUgYW5kIGRvbid0IGdvCj4gY3Jhenkgb24gdGhlIG1lbW9yeSB3 YXN0ZS4KClllcCwgd2UncmUgb24gdGhlIHNhbWUgcGFnZS4gSSd2ZSBnb3QgYSB2MyB0aGF0J3Mg YWxtb3N0IHJlYWR5IHRvIGdvLCBiYXNlZCBvbgpZdSdzIHByZXZ1b3VzIHJvdW5kIG9mIHJldmll dy4gSSdtIGdvaW5nIHRvIGVuY29ycG9yYXRlIHRoaXMgbWVjaGFuaXNtIGludG8gaXQKdGhlbiBw b3N0IGhvcGVmdWxseSBsYXRlciBpbiB0aGUgd2Vlay4gTm93IEkganVzdCBuZWVkIHRvIGZpZ3Vy ZSBvdXQgYSBkZWNlbnQKbmFtZSBmb3IgdGhlIG1heF9hdXRvX3NpemUgY29udHJvbC4uLgoKPiAK PiAKPiBJJ2xsIHRyeSByZWFkaW5nIGFsbCB0aGUgcHJldmlvdXMgZGlzY3Vzc2lvbnMgbmV4dCB3 ZWVrLgo+IAoKCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f CmxpbnV4LWFybS1rZXJuZWwgbWFpbGluZyBsaXN0CmxpbnV4LWFybS1rZXJuZWxAbGlzdHMuaW5m cmFkZWFkLm9yZwpodHRwOi8vbGlzdHMuaW5mcmFkZWFkLm9yZy9tYWlsbWFuL2xpc3RpbmZvL2xp bnV4LWFybS1rZXJuZWwK