From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.3 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 872A4C2D0A8 for ; Mon, 28 Sep 2020 17:56:45 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 1658722204 for ; Mon, 28 Sep 2020 17:56:38 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=sent.com header.i=@sent.com header.b="gkWZtUub"; dkim=temperror (0-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="Leevw8rq" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1658722204 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=sent.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id AAFFA90001A; Mon, 28 Sep 2020 13:55:32 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A5A8890001F; Mon, 28 Sep 2020 13:55:32 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8D95E900020; Mon, 28 Sep 2020 13:55:32 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0124.hostedemail.com [216.40.44.124]) by kanga.kvack.org (Postfix) with ESMTP id 6F6A790001F for ; Mon, 28 Sep 2020 13:55:32 -0400 (EDT) Received: from smtpin02.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 36323824999B for ; Mon, 28 Sep 2020 17:55:32 +0000 (UTC) X-FDA: 77313222504.02.watch04_1300cf827183 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin02.hostedemail.com (Postfix) with ESMTP id 03AB6101FA54A for ; Mon, 28 Sep 2020 17:55:31 +0000 (UTC) X-HE-Tag: watch04_1300cf827183 X-Filterd-Recvd-Size: 12947 Received: from wnew3-smtp.messagingengine.com (wnew3-smtp.messagingengine.com [64.147.123.17]) by imf13.hostedemail.com (Postfix) with ESMTP for ; Mon, 28 Sep 2020 17:55:31 +0000 (UTC) Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailnew.west.internal (Postfix) with ESMTP id 4DB7FEB8; Mon, 28 Sep 2020 13:55:29 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute4.internal (MEProxy); Mon, 28 Sep 2020 13:55:30 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sent.com; h=from :to:cc:subject:date:message-id:in-reply-to:references:reply-to :mime-version:content-transfer-encoding; s=fm1; bh=Nj/H1bhUbJdbY KZHa/71xGaBODpP4brVRKFdfTzhT6c=; b=gkWZtUubFT06rXcTnth5omC+yd5DX RyZJL4TCSVIbCb0v7g3+WlQ0Y7fqvw7Hubqd1pPUgMpDuD8QYF7zAP5zs1TauCHF 3GcXAZTfxcxY9pLUAXi3HDPJ9HAKM1LGN4VzyGgnYeDcvRfq0Tspd+ZqOVdi3aYi mdiLmvI5MiYaEF+AKIieaRJ8RDiexxgVPNFd098rcYxi4xVZxkRV7QtspBcVg6XQ LR1QBEeNaoZGWaVc8eJKJyNqyeGIt6UmIcVYZ1uETD5AH4I4VI9fpb/tSraMOOs6 dW/kNnEDBbEMHeDo4HprH2TsrtNhzsrpTZQ838hUHfgj2dC/1ceW3gY/A== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:reply-to:subject :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; bh=Nj/H1bhUbJdbYKZHa/71xGaBODpP4brVRKFdfTzhT6c=; b=Leevw8rq LQ5I5XbmbI+exRs8SO2qBixGTGP+sjKvn9XOx6mD6VQIrxf1fTuFvcAsAqymQXYR AaVcMwk/NFCcG2unWpnhw2+cHK0/iMgM5R31bcG8b47D/vEZ9PW3zfe4zCZJxPXd /ZnHtphbVlKC4cdBL7WUK8B4YpQihTPtqkPFvy2d1+hmf95kC9QAgglknFQChWUM P3DiQqTpknHJ/mfQ/QvXos+zKGdg1NaSHzvFz+TAcDAjtmeDHE4PMvGpsQt6JqlJ mNadnWBQSmNKOCjf/+DneDkoR6N8Uqg1JBEkGRjHaGPvE1ur/aOyRwS1LkiEHut5 DU+QrhYcvGCSEA== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedujedrvdeigdeliecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvffufffkofgjfhhrggfgsedtkeertdertddtnecuhfhrohhmpegkihcujggr nhcuoeiiihdrhigrnhesshgvnhhtrdgtohhmqeenucggtffrrghtthgvrhhnpeduhfffve ektdduhfdutdfgtdekkedvhfetuedufedtgffgvdevleehheevjefgtdenucfkphepuddv rdegiedruddtiedrudeigeenucevlhhushhtvghrufhiiigvpeduvdenucfrrghrrghmpe hmrghilhhfrhhomhepiihirdihrghnsehsvghnthdrtghomh X-ME-Proxy: Received: from nvrsysarch6.NVidia.COM (unknown [12.46.106.164]) by mail.messagingengine.com (Postfix) with ESMTPA id 595303064683; Mon, 28 Sep 2020 13:55:28 -0400 (EDT) From: Zi Yan To: linux-mm@kvack.org Cc: "Kirill A . Shutemov" , Roman Gushchin , Rik van Riel , Matthew Wilcox , Shakeel Butt , Yang Shi , Jason Gunthorpe , Mike Kravetz , Michal Hocko , David Hildenbrand , William Kucharski , Andrea Arcangeli , John Hubbard , David Nellans , linux-kernel@vger.kernel.org, Zi Yan Subject: [RFC PATCH v2 29/30] mm: thp: use cma reservation for pud thp allocation. Date: Mon, 28 Sep 2020 13:54:27 -0400 Message-Id: <20200928175428.4110504-30-zi.yan@sent.com> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20200928175428.4110504-1-zi.yan@sent.com> References: <20200928175428.4110504-1-zi.yan@sent.com> Reply-To: Zi Yan MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Zi Yan Sharing hugepage_cma reservation with hugetlb for pud thp allocaiton. The reserved cma regions still can be used for moveable page allocations. During 1GB page split, all subpages are cleared from the CMA bitmap, since they are no more 1GB pages and will be freed via the normal path instead of cma_release(). Signed-off-by: Zi Yan --- include/linux/cma.h | 3 +++ include/linux/huge_mm.h | 10 ++++++++++ mm/cma.c | 31 +++++++++++++++++++++++++++++++ mm/huge_memory.c | 34 ++++++++++++++++++++++++++++++++++ mm/hugetlb.c | 21 +-------------------- mm/mempolicy.c | 14 +++++++++++++- mm/page_alloc.c | 29 +++++++++++++++++++++++++++++ 7 files changed, 121 insertions(+), 21 deletions(-) diff --git a/include/linux/cma.h b/include/linux/cma.h index 9989d580c2a7..c299b62b3a7a 100644 --- a/include/linux/cma.h +++ b/include/linux/cma.h @@ -48,6 +48,9 @@ extern struct page *cma_alloc(struct cma *cma, size_t c= ount, unsigned int align, bool no_warn); extern bool cma_release(struct cma *cma, const struct page *pages, unsig= ned int count); =20 +extern bool cma_clear_bitmap_if_in_range(struct cma *cma, const struct p= age *page, + unsigned int count); + extern int cma_for_each_area(int (*it)(struct cma *cma, void *data), voi= d *data); =20 extern void cma_reserve(int min_order, unsigned long requested_size, diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index 0d0f9cf25aeb..163b244d9acd 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -24,6 +24,8 @@ extern struct page *follow_trans_huge_pud(struct vm_are= a_struct *vma, unsigned long addr, pud_t *pud, unsigned int flags); +extern struct page *alloc_thp_pud_page(int nid); +extern bool free_thp_pud_page(struct page *page, int order); #else static inline void huge_pud_set_accessed(struct vm_fault *vmf, pud_t ori= g_pud) { @@ -43,6 +45,14 @@ struct page *follow_trans_huge_pud(struct vm_area_stru= ct *vma, { return NULL; } +struct page *alloc_thp_pud_page(int nid) +{ + return NULL; +} +extern bool free_thp_pud_page(struct page *page, int order); +{ + return false; +} #endif =20 extern vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf, pmd_t orig_p= md); diff --git a/mm/cma.c b/mm/cma.c index 1a9d997fa5ab..c595aad61f58 100644 --- a/mm/cma.c +++ b/mm/cma.c @@ -532,6 +532,37 @@ bool cma_release(struct cma *cma, const struct page = *pages, unsigned int count) return true; } =20 +/** + * cma_clear_bitmap_if_in_range() - clear bitmap for a given page + * @cma: Contiguous memory region for which the allocation is performe= d. + * @pages: Allocated pages. + * @count: Number of allocated pages. + * + * This function clears bitmap of memory allocated by cma_alloc(). + * It returns false when provided pages do not belong to contiguous area= and + * true otherwise. + */ +bool cma_clear_bitmap_if_in_range(struct cma *cma, const struct page *pa= ges, + unsigned int count) +{ + unsigned long pfn; + + if (!cma || !pages) + return false; + + pfn =3D page_to_pfn(pages); + + if (pfn < cma->base_pfn || pfn >=3D cma->base_pfn + cma->count) + return false; + + if (pfn + count > cma->base_pfn + cma->count) + return false; + + cma_clear_bitmap(cma, pfn, count); + + return true; +} + int cma_for_each_area(int (*it)(struct cma *cma, void *data), void *data= ) { int i; diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 20ecffc27396..910e51f35910 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -33,6 +33,7 @@ #include #include #include +#include =20 #include #include @@ -62,6 +63,10 @@ static struct shrinker deferred_split_shrinker; static atomic_t huge_zero_refcount; struct page *huge_zero_page __read_mostly; =20 +#ifdef CONFIG_CMA +extern struct cma *hugepage_cma[MAX_NUMNODES]; +#endif + bool transparent_hugepage_enabled(struct vm_area_struct *vma) { /* The addr is used to check if the vma size fits */ @@ -2498,6 +2503,17 @@ static void __split_huge_pud_page(struct page *pag= e, struct list_head *list, /* no file-back page support yet */ VM_BUG_ON(!PageAnon(page)); =20 + /* + * clear cma bitmap when we split pud page so the subpages can be freed + * as normal pages + */ + if (IS_ENABLED(CONFIG_CMA)) { + struct cma *cma =3D hugepage_cma[page_to_nid(head)]; + + VM_BUG_ON(!cma_clear_bitmap_if_in_range(cma, head, + thp_nr_pages(head))); + } + for (i =3D HPAGE_PUD_NR - HPAGE_PMD_NR; i >=3D 1; i -=3D HPAGE_PMD_NR) __split_huge_pud_page_tail(head, i, lruvec, list); =20 @@ -3732,3 +3748,21 @@ void remove_migration_pmd(struct page_vma_mapped_w= alk *pvmw, struct page *new) update_mmu_cache_pmd(vma, address, pvmw->pmd); } #endif + +struct page *alloc_thp_pud_page(int nid) +{ + struct page *page =3D NULL; +#ifdef CONFIG_CMA + page =3D cma_alloc(hugepage_cma[nid], HPAGE_PUD_NR, HPAGE_PUD_ORDER, tr= ue); +#endif + return page; +} + +bool free_thp_pud_page(struct page *page, int order) +{ + bool ret =3D false; +#ifdef CONFIG_CMA + ret =3D cma_release(hugepage_cma[page_to_nid(page)], page, 1<