From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AC233C433EF for ; Tue, 31 May 2022 15:06:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3A4B56B007D; Tue, 31 May 2022 11:06:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 32E966B007E; Tue, 31 May 2022 11:06:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1F6866B0080; Tue, 31 May 2022 11:06:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 0A4906B007D for ; Tue, 31 May 2022 11:06:31 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id CFC453533F for ; Tue, 31 May 2022 15:06:30 +0000 (UTC) X-FDA: 79526364540.20.F9882AA Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf03.hostedemail.com (Postfix) with ESMTP id A1B1520052 for ; Tue, 31 May 2022 15:06:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=TSKydin8CSjzrQk8aEZX2Nw1h5HXz6kdLwwIhB8/XBM=; b=Q4Ojy7HdsM3SwrPq/Qr2C+3gxF k55ZKje9FDTiwbm+oyXL550KyftMx7L00OcSBoIM8QzB16TWxNb3bKIJkD9KuUQMWSxGQaOMvoRtD GY44QwgMNGjJZtLHlcwAAo2/FYjQ8T5RHgfGZZmapG+Pn3nrWnhDLfWTvOQCBbQFi6y7zHh61u+JT hVlNQclsWPPkI+hiMYRb2/Bggz8eY1Idn6pOG60poKC8b1v9S59tbW2JEOEzxvZY8MA9gBv9XWaLL 0cQSlrY7vnWnrWqoJS1i4NPEC2PMXtQxw0H9SYcvrb4ATBlQkjvGu9+C1+6O1Gw3SiqjVy9vo0Xp9 iaJ2rrhA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1nw3Rl-005T1Y-GT; Tue, 31 May 2022 15:06:13 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" Subject: [PATCH 4/6] mm/page_alloc: Add alloc_frozen_pages() Date: Tue, 31 May 2022 16:06:09 +0100 Message-Id: <20220531150611.1303156-5-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220531150611.1303156-1-willy@infradead.org> References: <20220531150611.1303156-1-willy@infradead.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: A1B1520052 X-Rspam-User: X-Stat-Signature: pzo5efqwtry54151y9in9sechbs1jf8i Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=Q4Ojy7Hd; dmarc=none; spf=none (imf03.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-HE-Tag: 1654009575-818056 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Provide an interface to allocate pages from the page allocator without incrementing their refcount. This saves an atomic operation on free, which may be beneficial to some users (eg slab). Signed-off-by: Matthew Wilcox (Oracle) --- mm/internal.h | 11 +++++++++ mm/mempolicy.c | 61 ++++++++++++++++++++++++++++++------------------- mm/page_alloc.c | 18 +++++++++++---- 3 files changed, 63 insertions(+), 27 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index f1c0dab2b98e..bf70ee2e38e9 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -362,9 +362,20 @@ extern void post_alloc_hook(struct page *page, unsigned int order, gfp_t gfp_flags); extern int user_min_free_kbytes; +struct page *__alloc_frozen_pages(gfp_t, unsigned int order, int nid, + nodemask_t *); void free_frozen_pages(struct page *, unsigned int order); void free_unref_page_list(struct list_head *list); +#ifdef CONFIG_NUMA +struct page *alloc_frozen_pages(gfp_t, unsigned int order); +#else +static inline struct page *alloc_frozen_pages(gfp_t gfp, unsigned int order) +{ + return __alloc_frozen_pages(gfp, order, numa_node_id(), NULL); +} +#endif + extern void zone_pcp_update(struct zone *zone, int cpu_online); extern void zone_pcp_reset(struct zone *zone); extern void zone_pcp_disable(struct zone *zone); diff --git a/mm/mempolicy.c b/mm/mempolicy.c index d39b01fd52fe..ac7c45d0f7dc 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -2102,7 +2102,7 @@ static struct page *alloc_page_interleave(gfp_t gfp, unsigned order, { struct page *page; - page = __alloc_pages(gfp, order, nid, NULL); + page = __alloc_frozen_pages(gfp, order, nid, NULL); /* skip NUMA_INTERLEAVE_HIT counter update if numa stats is disabled */ if (!static_branch_likely(&vm_numa_stat_key)) return page; @@ -2128,9 +2128,9 @@ static struct page *alloc_pages_preferred_many(gfp_t gfp, unsigned int order, */ preferred_gfp = gfp | __GFP_NOWARN; preferred_gfp &= ~(__GFP_DIRECT_RECLAIM | __GFP_NOFAIL); - page = __alloc_pages(preferred_gfp, order, nid, &pol->nodes); + page = __alloc_frozen_pages(preferred_gfp, order, nid, &pol->nodes); if (!page) - page = __alloc_pages(gfp, order, nid, NULL); + page = __alloc_frozen_pages(gfp, order, nid, NULL); return page; } @@ -2169,8 +2169,11 @@ struct folio *vma_alloc_folio(gfp_t gfp, int order, struct vm_area_struct *vma, mpol_cond_put(pol); gfp |= __GFP_COMP; page = alloc_page_interleave(gfp, order, nid); - if (page && order > 1) - prep_transhuge_page(page); + if (page) { + set_page_refcounted(page); + if (order > 1) + prep_transhuge_page(page); + } folio = (struct folio *)page; goto out; } @@ -2182,8 +2185,11 @@ struct folio *vma_alloc_folio(gfp_t gfp, int order, struct vm_area_struct *vma, gfp |= __GFP_COMP; page = alloc_pages_preferred_many(gfp, order, node, pol); mpol_cond_put(pol); - if (page && order > 1) - prep_transhuge_page(page); + if (page) { + set_page_refcounted(page); + if (order > 1) + prep_transhuge_page(page); + } folio = (struct folio *)page; goto out; } @@ -2237,21 +2243,7 @@ struct folio *vma_alloc_folio(gfp_t gfp, int order, struct vm_area_struct *vma, } EXPORT_SYMBOL(vma_alloc_folio); -/** - * alloc_pages - Allocate pages. - * @gfp: GFP flags. - * @order: Power of two of number of pages to allocate. - * - * Allocate 1 << @order contiguous pages. The physical address of the - * first page is naturally aligned (eg an order-3 allocation will be aligned - * to a multiple of 8 * PAGE_SIZE bytes). The NUMA policy of the current - * process is honoured when in process context. - * - * Context: Can be called from any context, providing the appropriate GFP - * flags are used. - * Return: The page on success or NULL if allocation fails. - */ -struct page *alloc_pages(gfp_t gfp, unsigned order) +struct page *alloc_frozen_pages(gfp_t gfp, unsigned order) { struct mempolicy *pol = &default_policy; struct page *page; @@ -2269,12 +2261,35 @@ struct page *alloc_pages(gfp_t gfp, unsigned order) page = alloc_pages_preferred_many(gfp, order, policy_node(gfp, pol, numa_node_id()), pol); else - page = __alloc_pages(gfp, order, + page = __alloc_frozen_pages(gfp, order, policy_node(gfp, pol, numa_node_id()), policy_nodemask(gfp, pol)); return page; } + +/** + * alloc_pages - Allocate pages. + * @gfp: GFP flags. + * @order: Power of two of number of pages to allocate. + * + * Allocate 1 << @order contiguous pages. The physical address of the + * first page is naturally aligned (eg an order-3 allocation will be aligned + * to a multiple of 8 * PAGE_SIZE bytes). The NUMA policy of the current + * process is honoured when in process context. + * + * Context: Can be called from any context, providing the appropriate GFP + * flags are used. + * Return: The page on success or NULL if allocation fails. + */ +struct page *alloc_pages(gfp_t gfp, unsigned order) +{ + struct page *page = alloc_frozen_pages(gfp, order); + + if (page) + set_page_refcounted(page); + return page; +} EXPORT_SYMBOL(alloc_pages); struct folio *folio_alloc(gfp_t gfp, unsigned order) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 825922000781..49d8f04d14ef 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2390,7 +2390,6 @@ inline void post_alloc_hook(struct page *page, unsigned int order, bool init_tags = init && (gfp_flags & __GFP_ZEROTAGS); set_page_private(page, 0); - set_page_refcounted(page); arch_alloc_page(page, order); debug_pagealloc_map_pages(page, 1 << order); @@ -5386,8 +5385,8 @@ EXPORT_SYMBOL_GPL(__alloc_pages_bulk); /* * This is the 'heart' of the zoned buddy allocator. */ -struct page *__alloc_pages(gfp_t gfp, unsigned int order, int preferred_nid, - nodemask_t *nodemask) +struct page *__alloc_frozen_pages(gfp_t gfp, unsigned int order, + int preferred_nid, nodemask_t *nodemask) { struct page *page; unsigned int alloc_flags = ALLOC_WMARK_LOW; @@ -5440,7 +5439,7 @@ struct page *__alloc_pages(gfp_t gfp, unsigned int order, int preferred_nid, out: if (memcg_kmem_enabled() && (gfp & __GFP_ACCOUNT) && page && unlikely(__memcg_kmem_charge_page(page, gfp, order) != 0)) { - __free_pages(page, order); + free_frozen_pages(page, order); page = NULL; } @@ -5448,6 +5447,17 @@ struct page *__alloc_pages(gfp_t gfp, unsigned int order, int preferred_nid, return page; } + +struct page *__alloc_pages(gfp_t gfp, unsigned int order, int preferred_nid, + nodemask_t *nodemask) +{ + struct page *page; + + page = __alloc_frozen_pages(gfp, order, preferred_nid, nodemask); + if (page) + set_page_refcounted(page); + return page; +} EXPORT_SYMBOL(__alloc_pages); struct folio *__folio_alloc(gfp_t gfp, unsigned int order, int preferred_nid, -- 2.34.1