From: James Houghton <jthoughton@google.com>
To: Mike Kravetz <mike.kravetz@oracle.com>,
Muchun Song <songmuchun@bytedance.com>,
Peter Xu <peterx@redhat.com>,
Andrew Morton <akpm@linux-foundation.org>
Cc: David Hildenbrand <david@redhat.com>,
David Rientjes <rientjes@google.com>,
Axel Rasmussen <axelrasmussen@google.com>,
Mina Almasry <almasrymina@google.com>,
"Zach O'Keefe" <zokeefe@google.com>,
Manish Mishra <manish.mishra@nutanix.com>,
Naoya Horiguchi <naoya.horiguchi@nec.com>,
"Dr . David Alan Gilbert" <dgilbert@redhat.com>,
"Matthew Wilcox (Oracle)" <willy@infradead.org>,
Vlastimil Babka <vbabka@suse.cz>,
Baolin Wang <baolin.wang@linux.alibaba.com>,
Miaohe Lin <linmiaohe@huawei.com>, Yang Shi <shy828301@gmail.com>,
Frank van der Linden <fvdl@google.com>,
Jiaqi Yan <jiaqiyan@google.com>,
linux-mm@kvack.org, linux-kernel@vger.kernel.org,
James Houghton <jthoughton@google.com>
Subject: [PATCH v2 12/46] hugetlb: add hugetlb_alloc_pmd and hugetlb_alloc_pte
Date: Sat, 18 Feb 2023 00:27:45 +0000 [thread overview]
Message-ID: <20230218002819.1486479-13-jthoughton@google.com> (raw)
In-Reply-To: <20230218002819.1486479-1-jthoughton@google.com>
These functions are used to allocate new PTEs below the hstate PTE. This
will be used by hugetlb_walk_step, which implements stepping forwards in
a HugeTLB high-granularity page table walk.
The reasons that we don't use the standard pmd_alloc/pte_alloc*
functions are:
1) This prevents us from accidentally overwriting swap entries or
attempting to use swap entries as present non-leaf PTEs (see
pmd_alloc(); we assume that !pte_none means pte_present and
non-leaf).
2) Locking hugetlb PTEs can different than regular PTEs. (Although, as
implemented right now, locking is the same.)
3) We can maintain compatibility with CONFIG_HIGHPTE. That is, HugeTLB
HGM won't use HIGHPTE, but the kernel can still be built with it,
and other mm code will use it.
When GENERAL_HUGETLB supports P4D-based hugepages, we will need to
implement hugetlb_pud_alloc to implement hugetlb_walk_step.
Signed-off-by: James Houghton <jthoughton@google.com>
diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h
index eeacadf3272b..9d839519c875 100644
--- a/include/linux/hugetlb.h
+++ b/include/linux/hugetlb.h
@@ -72,6 +72,11 @@ unsigned long hugetlb_pte_mask(const struct hugetlb_pte *hpte)
bool hugetlb_pte_present_leaf(const struct hugetlb_pte *hpte, pte_t pte);
+pmd_t *hugetlb_alloc_pmd(struct mm_struct *mm, struct hugetlb_pte *hpte,
+ unsigned long addr);
+pte_t *hugetlb_alloc_pte(struct mm_struct *mm, struct hugetlb_pte *hpte,
+ unsigned long addr);
+
struct hugepage_subpool {
spinlock_t lock;
long count;
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 6c74adff43b6..bb424cdf79e4 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -483,6 +483,120 @@ static bool has_same_uncharge_info(struct file_region *rg,
#endif
}
+/*
+ * hugetlb_alloc_pmd -- Allocate or find a PMD beneath a PUD-level hpte.
+ *
+ * This is meant to be used to implement hugetlb_walk_step when one must go to
+ * step down to a PMD. Different architectures may implement hugetlb_walk_step
+ * differently, but hugetlb_alloc_pmd and hugetlb_alloc_pte are architecture-
+ * independent.
+ *
+ * Returns:
+ * On success: the pointer to the PMD. This should be placed into a
+ * hugetlb_pte. @hpte is not changed.
+ * ERR_PTR(-EINVAL): hpte is not PUD-level
+ * ERR_PTR(-EEXIST): there is a non-leaf and non-empty PUD in @hpte
+ * ERR_PTR(-ENOMEM): could not allocate the new PMD
+ */
+pmd_t *hugetlb_alloc_pmd(struct mm_struct *mm, struct hugetlb_pte *hpte,
+ unsigned long addr)
+{
+ spinlock_t *ptl = hugetlb_pte_lockptr(hpte);
+ pmd_t *new;
+ pud_t *pudp;
+ pud_t pud;
+
+ if (hpte->level != HUGETLB_LEVEL_PUD)
+ return ERR_PTR(-EINVAL);
+
+ pudp = (pud_t *)hpte->ptep;
+retry:
+ pud = READ_ONCE(*pudp);
+ if (likely(pud_present(pud)))
+ return unlikely(pud_leaf(pud))
+ ? ERR_PTR(-EEXIST)
+ : pmd_offset(pudp, addr);
+ else if (!pud_none(pud))
+ /*
+ * Not present and not none means that a swap entry lives here,
+ * and we can't get rid of it.
+ */
+ return ERR_PTR(-EEXIST);
+
+ new = pmd_alloc_one(mm, addr);
+ if (!new)
+ return ERR_PTR(-ENOMEM);
+
+ spin_lock(ptl);
+ if (!pud_same(pud, *pudp)) {
+ spin_unlock(ptl);
+ pmd_free(mm, new);
+ goto retry;
+ }
+
+ mm_inc_nr_pmds(mm);
+ smp_wmb(); /* See comment in pmd_install() */
+ pud_populate(mm, pudp, new);
+ spin_unlock(ptl);
+ return pmd_offset(pudp, addr);
+}
+
+/*
+ * hugetlb_alloc_pte -- Allocate a PTE beneath a pmd_none PMD-level hpte.
+ *
+ * See the comment above hugetlb_alloc_pmd.
+ */
+pte_t *hugetlb_alloc_pte(struct mm_struct *mm, struct hugetlb_pte *hpte,
+ unsigned long addr)
+{
+ spinlock_t *ptl = hugetlb_pte_lockptr(hpte);
+ pgtable_t new;
+ pmd_t *pmdp;
+ pmd_t pmd;
+
+ if (hpte->level != HUGETLB_LEVEL_PMD)
+ return ERR_PTR(-EINVAL);
+
+ pmdp = (pmd_t *)hpte->ptep;
+retry:
+ pmd = READ_ONCE(*pmdp);
+ if (likely(pmd_present(pmd)))
+ return unlikely(pmd_leaf(pmd))
+ ? ERR_PTR(-EEXIST)
+ : pte_offset_kernel(pmdp, addr);
+ else if (!pmd_none(pmd))
+ /*
+ * Not present and not none means that a swap entry lives here,
+ * and we can't get rid of it.
+ */
+ return ERR_PTR(-EEXIST);
+
+ /*
+ * With CONFIG_HIGHPTE, calling `pte_alloc_one` directly may result
+ * in page tables being allocated in high memory, needing a kmap to
+ * access. Instead, we call __pte_alloc_one directly with
+ * GFP_PGTABLE_USER to prevent these PTEs being allocated in high
+ * memory.
+ */
+ new = __pte_alloc_one(mm, GFP_PGTABLE_USER);
+ if (!new)
+ return ERR_PTR(-ENOMEM);
+
+ spin_lock(ptl);
+ if (!pmd_same(pmd, *pmdp)) {
+ spin_unlock(ptl);
+ pgtable_pte_page_dtor(new);
+ __free_page(new);
+ goto retry;
+ }
+
+ mm_inc_nr_ptes(mm);
+ smp_wmb(); /* See comment in pmd_install() */
+ pmd_populate(mm, pmdp, new);
+ spin_unlock(ptl);
+ return pte_offset_kernel(pmdp, addr);
+}
+
static void coalesce_file_region(struct resv_map *resv, struct file_region *rg)
{
struct file_region *nrg, *prg;
--
2.39.2.637.g21b0678d19-goog
next prev parent reply other threads:[~2023-02-18 0:29 UTC|newest]
Thread overview: 96+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-02-18 0:27 [PATCH v2 00/46] hugetlb: introduce HugeTLB high-granularity mapping James Houghton
2023-02-18 0:27 ` [PATCH v2 01/46] hugetlb: don't set PageUptodate for UFFDIO_CONTINUE James Houghton
2023-02-18 0:41 ` Mina Almasry
2023-02-21 15:59 ` James Houghton
2023-02-21 19:33 ` Mike Kravetz
2023-02-21 19:58 ` James Houghton
2023-02-18 0:27 ` [PATCH v2 02/46] hugetlb: remove mk_huge_pte; it is unused James Houghton
2023-02-18 0:27 ` [PATCH v2 03/46] hugetlb: remove redundant pte_mkhuge in migration path James Houghton
2023-02-18 0:27 ` [PATCH v2 04/46] hugetlb: only adjust address ranges when VMAs want PMD sharing James Houghton
2023-02-18 1:10 ` Mina Almasry
2023-02-18 0:27 ` [PATCH v2 05/46] rmap: hugetlb: switch from page_dup_file_rmap to page_add_file_rmap James Houghton
2023-03-02 1:06 ` Jiaqi Yan
2023-03-02 15:44 ` James Houghton
2023-03-02 16:43 ` James Houghton
2023-03-02 19:22 ` Mike Kravetz
2023-02-18 0:27 ` [PATCH v2 06/46] hugetlb: add CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING James Houghton
2023-02-18 0:27 ` [PATCH v2 07/46] mm: add VM_HUGETLB_HGM VMA flag James Houghton
2023-02-24 22:35 ` Mike Kravetz
2023-02-18 0:27 ` [PATCH v2 08/46] hugetlb: add HugeTLB HGM enablement helpers James Houghton
2023-02-18 1:40 ` Mina Almasry
2023-02-21 16:16 ` James Houghton
2023-02-24 23:08 ` Mike Kravetz
2023-02-18 0:27 ` [PATCH v2 09/46] mm: add MADV_SPLIT to enable HugeTLB HGM James Houghton
2023-02-18 1:58 ` Mina Almasry
2023-02-21 16:33 ` James Houghton
2023-02-24 23:25 ` Mike Kravetz
2023-02-27 15:14 ` James Houghton
2023-02-18 0:27 ` [PATCH v2 10/46] hugetlb: make huge_pte_lockptr take an explicit shift argument James Houghton
2023-02-18 0:27 ` [PATCH v2 11/46] hugetlb: add hugetlb_pte to track HugeTLB page table entries James Houghton
2023-02-18 5:24 ` Mina Almasry
2023-02-21 16:36 ` James Houghton
2023-02-25 0:09 ` Mike Kravetz
2023-02-18 0:27 ` James Houghton [this message]
2023-02-18 17:46 ` [PATCH v2 12/46] hugetlb: add hugetlb_alloc_pmd and hugetlb_alloc_pte kernel test robot
2023-02-27 19:16 ` Mike Kravetz
2023-02-27 19:31 ` James Houghton
2023-02-18 0:27 ` [PATCH v2 13/46] hugetlb: add hugetlb_hgm_walk and hugetlb_walk_step James Houghton
2023-02-18 7:43 ` kernel test robot
2023-02-18 18:07 ` kernel test robot
2023-02-21 17:09 ` James Houghton
2023-02-28 22:14 ` Mike Kravetz
2023-02-28 23:03 ` James Houghton
2023-02-18 0:27 ` [PATCH v2 14/46] hugetlb: split PTE markers when doing HGM walks James Houghton
2023-02-18 19:49 ` kernel test robot
2023-02-28 22:48 ` Mike Kravetz
2023-02-18 0:27 ` [PATCH v2 15/46] hugetlb: add make_huge_pte_with_shift James Houghton
2023-02-22 21:14 ` Mina Almasry
2023-02-22 22:53 ` James Houghton
2023-02-18 0:27 ` [PATCH v2 16/46] hugetlb: make default arch_make_huge_pte understand small mappings James Houghton
2023-02-22 21:17 ` Mina Almasry
2023-02-22 22:52 ` James Houghton
2023-02-28 23:02 ` Mike Kravetz
2023-02-18 0:27 ` [PATCH v2 17/46] hugetlbfs: do a full walk to check if vma maps a page James Houghton
2023-02-22 15:46 ` James Houghton
2023-02-28 23:52 ` Mike Kravetz
2023-02-18 0:27 ` [PATCH v2 18/46] hugetlb: add HGM support to __unmap_hugepage_range James Houghton
2023-02-18 0:27 ` [PATCH v2 19/46] hugetlb: add HGM support to hugetlb_change_protection James Houghton
2023-02-18 0:27 ` [PATCH v2 20/46] hugetlb: add HGM support to follow_hugetlb_page James Houghton
2023-02-18 0:27 ` [PATCH v2 21/46] hugetlb: add HGM support to hugetlb_follow_page_mask James Houghton
2023-02-18 0:27 ` [PATCH v2 22/46] hugetlb: add HGM support to copy_hugetlb_page_range James Houghton
2023-02-24 17:39 ` James Houghton
2023-02-18 0:27 ` [PATCH v2 23/46] hugetlb: add HGM support to move_hugetlb_page_tables James Houghton
2023-02-18 0:27 ` [PATCH v2 24/46] hugetlb: add HGM support to hugetlb_fault and hugetlb_no_page James Houghton
2023-02-18 0:27 ` [PATCH v2 25/46] hugetlb: use struct hugetlb_pte for walk_hugetlb_range James Houghton
2023-02-18 0:27 ` [PATCH v2 26/46] mm: rmap: provide pte_order in page_vma_mapped_walk James Houghton
2023-02-18 0:28 ` [PATCH v2 27/46] mm: rmap: update try_to_{migrate,unmap} to handle mapcount for HGM James Houghton
2023-02-18 0:28 ` [PATCH v2 28/46] mm: rmap: in try_to_{migrate,unmap}, check head page for hugetlb page flags James Houghton
2023-02-18 0:28 ` [PATCH v2 29/46] hugetlb: update page_vma_mapped to do high-granularity walks James Houghton
2023-02-18 0:28 ` [PATCH v2 30/46] hugetlb: add high-granularity migration support James Houghton
2023-02-18 0:28 ` [PATCH v2 31/46] hugetlb: sort hstates in hugetlb_init_hstates James Houghton
2023-02-18 0:28 ` [PATCH v2 32/46] hugetlb: add for_each_hgm_shift James Houghton
2023-02-18 0:28 ` [PATCH v2 33/46] hugetlb: userfaultfd: add support for high-granularity UFFDIO_CONTINUE James Houghton
2023-02-18 0:28 ` [PATCH v2 34/46] hugetlb: add MADV_COLLAPSE for hugetlb James Houghton
2023-02-18 0:28 ` [PATCH v2 35/46] hugetlb: add check to prevent refcount overflow via HGM James Houghton
2023-02-24 17:42 ` James Houghton
2023-02-24 18:05 ` James Houghton
2023-02-18 0:28 ` [PATCH v2 36/46] hugetlb: remove huge_pte_lock and huge_pte_lockptr James Houghton
2023-02-18 0:28 ` [PATCH v2 37/46] hugetlb: replace make_huge_pte with make_huge_pte_with_shift James Houghton
2023-02-18 0:28 ` [PATCH v2 38/46] mm: smaps: add stats for HugeTLB mapping size James Houghton
2023-02-18 0:28 ` [PATCH v2 39/46] hugetlb: x86: enable high-granularity mapping for x86_64 James Houghton
2023-02-18 0:28 ` [PATCH v2 40/46] docs: hugetlb: update hugetlb and userfaultfd admin-guides with HGM info James Houghton
2023-02-18 0:28 ` [PATCH v2 41/46] docs: proc: include information about HugeTLB HGM James Houghton
2023-02-18 0:28 ` [PATCH v2 42/46] selftests/mm: add HugeTLB HGM to userfaultfd selftest James Houghton
2023-02-18 0:28 ` [PATCH v2 43/46] KVM: selftests: add HugeTLB HGM to KVM demand paging selftest James Houghton
2023-02-18 0:28 ` [PATCH v2 44/46] selftests/mm: add anon and shared hugetlb to migration test James Houghton
2023-02-18 0:28 ` [PATCH v2 45/46] selftests/mm: add hugetlb HGM test to migration selftest James Houghton
2023-02-18 0:28 ` [PATCH v2 46/46] selftests/mm: add HGM UFFDIO_CONTINUE and hwpoison tests James Houghton
2023-02-24 17:37 ` James Houghton
2023-02-21 21:46 ` [PATCH v2 00/46] hugetlb: introduce HugeTLB high-granularity mapping Mike Kravetz
2023-02-22 15:48 ` David Hildenbrand
2023-02-22 20:57 ` Mina Almasry
2023-02-23 9:07 ` David Hildenbrand
2023-02-23 15:53 ` James Houghton
2023-02-23 16:17 ` David Hildenbrand
2023-02-23 18:33 ` Dr. David Alan Gilbert
2023-02-23 18:25 ` Mike Kravetz
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20230218002819.1486479-13-jthoughton@google.com \
--to=jthoughton@google.com \
--cc=akpm@linux-foundation.org \
--cc=almasrymina@google.com \
--cc=axelrasmussen@google.com \
--cc=baolin.wang@linux.alibaba.com \
--cc=david@redhat.com \
--cc=dgilbert@redhat.com \
--cc=fvdl@google.com \
--cc=jiaqiyan@google.com \
--cc=linmiaohe@huawei.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=manish.mishra@nutanix.com \
--cc=mike.kravetz@oracle.com \
--cc=naoya.horiguchi@nec.com \
--cc=peterx@redhat.com \
--cc=rientjes@google.com \
--cc=shy828301@gmail.com \
--cc=songmuchun@bytedance.com \
--cc=vbabka@suse.cz \
--cc=willy@infradead.org \
--cc=zokeefe@google.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).