From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 52669C43334 for ; Fri, 24 Jun 2022 17:37:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8C0638E024A; Fri, 24 Jun 2022 13:37:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 826568E0244; Fri, 24 Jun 2022 13:37:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 626888E024A; Fri, 24 Jun 2022 13:37:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 47F1F8E0244 for ; Fri, 24 Jun 2022 13:37:28 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id CF71033577 for ; Fri, 24 Jun 2022 17:37:27 +0000 (UTC) X-FDA: 79613836134.20.C624E99 Received: from mail-ua1-f73.google.com (mail-ua1-f73.google.com [209.85.222.73]) by imf12.hostedemail.com (Postfix) with ESMTP id 6FDBC4001D for ; Fri, 24 Jun 2022 17:37:27 +0000 (UTC) Received: by mail-ua1-f73.google.com with SMTP id l7-20020ab053c7000000b003791afd560cso1010426uaa.21 for ; Fri, 24 Jun 2022 10:37:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=9NP33zy0A1MX3vEGrpVIKeFhmsIPx2kWV4buJ37voF4=; b=ZKOKH7vAAREURJ1l8IWF0iXVyZJmt/59w23N609lJlz66JMzYMime9PnmzUaf55ztT vzLMK8s9/fXSSxWnYgNDHKf7LqeMXxNL7KAl5SuDjU3NQJJ60yl5tOO73MRzJCUdunKz 7Lgn3pxPgvpWylL4Lwm2BTczrJtzLYwEI/PWM2Cho00mXJipoxojyLs3eWxEQPPDnMB8 Oxl49ZR9JG0HGfk3WVBD25dYMN0YwU/lye70tN93BwsbS6nd07nY55SUbsCU9gTsjkxw 9k4o1zcXpunPWXdWooquHfqVjKqZmpuvtPY3r1RM36B/t5rz9uVN89mK3zfIIDtHV9hY WUTw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=9NP33zy0A1MX3vEGrpVIKeFhmsIPx2kWV4buJ37voF4=; b=qN16YwzqyjcV1hV5BuXMHtCai7/S0tuEOIrtIKppfDyve0l9o2BYjqz/c1YyIoSZ+g 0FdCToqCzbZUMLRnHcHYymmMIXQPuYirhsVN/MNHa2pPTGPk85EwPQ82mVxtqtR9oeJv FUM7tBAKTg5UVDaolFBGYglplIZRJta7g/eng4hLHsV9nVxpfZStsQh9hvjnKcRiIveT 6hEJvS3Rm5xTB2Xr7Lo8Wl2paP0nGjz/ytzz/qJhVvc1rqoQ0kqjoWWlIh0rzGukCvlO YPwXaIwI0OkHVWqgCxVeKNeydSiZHFhp3FIoqIkT8lVGy+hxpyY0A/L7UhG1/KGqV3GU 3uWQ== X-Gm-Message-State: AJIora+6MCeAYm5OiwR3pMosCVUg22uhic+RjcflwBtbFHbUHX6SpizR BFOIduJoDjp5w77v54kLw1SRexZ/79lGvO7j X-Google-Smtp-Source: AGRyM1ud2VpyFSoYe3jfqlLqsCZHAGiyta2WrrFtmqkuRYO0f2uYFJE5sTDJIrzr1fNDbhgGC1mFcexvMmBaf2W2 X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:2a4f]) (user=jthoughton job=sendgmr) by 2002:a67:ef4a:0:b0:354:6546:8b97 with SMTP id k10-20020a67ef4a000000b0035465468b97mr8260702vsr.83.1656092246792; Fri, 24 Jun 2022 10:37:26 -0700 (PDT) Date: Fri, 24 Jun 2022 17:36:42 +0000 In-Reply-To: <20220624173656.2033256-1-jthoughton@google.com> Message-Id: <20220624173656.2033256-13-jthoughton@google.com> Mime-Version: 1.0 References: <20220624173656.2033256-1-jthoughton@google.com> X-Mailer: git-send-email 2.37.0.rc0.161.g10f37bed90-goog Subject: [RFC PATCH 12/26] hugetlb: add HugeTLB splitting functionality From: James Houghton To: Mike Kravetz , Muchun Song , Peter Xu Cc: David Hildenbrand , David Rientjes , Axel Rasmussen , Mina Almasry , Jue Wang , Manish Mishra , "Dr . David Alan Gilbert" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, James Houghton Content-Type: text/plain; charset="UTF-8" ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1656092247; a=rsa-sha256; cv=none; b=sX0K3abSYvcomtUXOWYEUaaO1PFXaT9aPV7oo4tAGfOwm/DejMASY2BVrpsebNeLxBP2h7 YizpVOS7TEHKzhJjcDRTuY2drx0VjDV6GXAFx0WL8qS5WObA0Bya/zz/WAyvyE3aAOYxcT Po5LXT8+hthbLw06LmX2p6s8+CFgBok= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1656092247; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9NP33zy0A1MX3vEGrpVIKeFhmsIPx2kWV4buJ37voF4=; b=R7m4jA7d3VXulY24sGsGdiVgfcUHtYIyOQxNsxUhFd37Q8UwizyEAzIxOGFWHdDKLhZSA2 6Bfk0VjswUWNENpSyHlHZSf0vzLP5TcVnywXc14KFFkhBqQq+QYjy6CJiVc0xKtUV7iN4Y L9dzNtiZmIDRDC2ISmo5dJHhMtpiMik= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=ZKOKH7vA; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf12.hostedemail.com: domain of 3Vva1YgoKCDccmahnZamhgZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--jthoughton.bounces.google.com designates 209.85.222.73 as permitted sender) smtp.mailfrom=3Vva1YgoKCDccmahnZamhgZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--jthoughton.bounces.google.com X-Stat-Signature: atqtih7exrbfpnjp177nxxr3dumrji69 X-Rspamd-Queue-Id: 6FDBC4001D Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=ZKOKH7vA; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf12.hostedemail.com: domain of 3Vva1YgoKCDccmahnZamhgZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--jthoughton.bounces.google.com designates 209.85.222.73 as permitted sender) smtp.mailfrom=3Vva1YgoKCDccmahnZamhgZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--jthoughton.bounces.google.com X-Rspam-User: X-Rspamd-Server: rspam04 X-HE-Tag: 1656092247-666214 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The new function, hugetlb_split_to_shift, will optimally split the page table to map a particular address at a particular granularity. This is useful for punching a hole in the mapping and for mapping small sections of a HugeTLB page (via UFFDIO_CONTINUE, for example). Signed-off-by: James Houghton --- mm/hugetlb.c | 122 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 122 insertions(+) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 3ec2a921ee6f..eaffe7b4f67c 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -102,6 +102,18 @@ struct mutex *hugetlb_fault_mutex_table ____cacheline_aligned_in_smp; /* Forward declaration */ static int hugetlb_acct_memory(struct hstate *h, long delta); +/* + * Find the subpage that corresponds to `addr` in `hpage`. + */ +static struct page *hugetlb_find_subpage(struct hstate *h, struct page *hpage, + unsigned long addr) +{ + size_t idx = (addr & ~huge_page_mask(h))/PAGE_SIZE; + + BUG_ON(idx >= pages_per_huge_page(h)); + return &hpage[idx]; +} + static inline bool subpool_is_free(struct hugepage_subpool *spool) { if (spool->count) @@ -7044,6 +7056,116 @@ static unsigned int __shift_for_hstate(struct hstate *h) for ((tmp_h) = hstate; (shift) = __shift_for_hstate(tmp_h), \ (tmp_h) <= &hstates[hugetlb_max_hstate]; \ (tmp_h)++) + +/* + * Given a particular address, split the HugeTLB PTE that currently maps it + * so that, for the given address, the PTE that maps it is `desired_shift`. + * This function will always split the HugeTLB PTE optimally. + * + * For example, given a HugeTLB 1G page that is mapped from VA 0 to 1G. If we + * call this function with addr=0 and desired_shift=PAGE_SHIFT, will result in + * these changes to the page table: + * 1. The PUD will be split into 2M PMDs. + * 2. The first PMD will be split again into 4K PTEs. + */ +static int hugetlb_split_to_shift(struct mm_struct *mm, struct vm_area_struct *vma, + const struct hugetlb_pte *hpte, + unsigned long addr, unsigned long desired_shift) +{ + unsigned long start, end, curr; + unsigned long desired_sz = 1UL << desired_shift; + struct hstate *h = hstate_vma(vma); + int ret; + struct hugetlb_pte new_hpte; + struct mmu_notifier_range range; + struct page *hpage = NULL; + struct page *subpage; + pte_t old_entry; + struct mmu_gather tlb; + + BUG_ON(!hpte->ptep); + BUG_ON(hugetlb_pte_size(hpte) == desired_sz); + + start = addr & hugetlb_pte_mask(hpte); + end = start + hugetlb_pte_size(hpte); + + i_mmap_assert_write_locked(vma->vm_file->f_mapping); + + BUG_ON(!hpte->ptep); + /* This function only works if we are looking at a leaf-level PTE. */ + BUG_ON(!hugetlb_pte_none(hpte) && !hugetlb_pte_present_leaf(hpte)); + + /* + * Clear the PTE so that we will allocate the PT structures when + * walking the page table. + */ + old_entry = huge_ptep_get_and_clear(mm, start, hpte->ptep); + + if (!huge_pte_none(old_entry)) + hpage = pte_page(old_entry); + + BUG_ON(!IS_ALIGNED(start, desired_sz)); + BUG_ON(!IS_ALIGNED(end, desired_sz)); + + for (curr = start; curr < end;) { + struct hstate *tmp_h; + unsigned int shift; + + for_each_hgm_shift(h, tmp_h, shift) { + unsigned long sz = 1UL << shift; + + if (!IS_ALIGNED(curr, sz) || curr + sz > end) + continue; + /* + * If we are including `addr`, we need to make sure + * splitting down to the correct size. Go to a smaller + * size if we are not. + */ + if (curr <= addr && curr + sz > addr && + shift > desired_shift) + continue; + + /* + * Continue the page table walk to the level we want, + * allocate PT structures as we go. + */ + hugetlb_pte_copy(&new_hpte, hpte); + ret = hugetlb_walk_to(mm, &new_hpte, curr, sz, + /*stop_at_none=*/false); + if (ret) + goto err; + BUG_ON(hugetlb_pte_size(&new_hpte) != sz); + if (hpage) { + pte_t new_entry; + + subpage = hugetlb_find_subpage(h, hpage, curr); + new_entry = make_huge_pte_with_shift(vma, subpage, + huge_pte_write(old_entry), + shift); + set_huge_pte_at(mm, curr, new_hpte.ptep, new_entry); + } + curr += sz; + goto next; + } + /* We couldn't find a size that worked. */ + BUG(); +next: + continue; + } + + mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, vma, vma->vm_mm, + start, end); + mmu_notifier_invalidate_range_start(&range); + return 0; +err: + tlb_gather_mmu(&tlb, mm); + /* Free any newly allocated page table entries. */ + hugetlb_free_range(&tlb, hpte, start, curr); + /* Restore the old entry. */ + set_huge_pte_at(mm, start, hpte->ptep, old_entry); + tlb_finish_mmu(&tlb); + return ret; +} #endif /* CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING */ /* -- 2.37.0.rc0.161.g10f37bed90-goog