From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6F8E3C433F5 for ; Mon, 29 Nov 2021 22:50:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235540AbhK2WyG (ORCPT ); Mon, 29 Nov 2021 17:54:06 -0500 Received: from mail-qt1-f178.google.com ([209.85.160.178]:44017 "EHLO mail-qt1-f178.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235713AbhK2WxG (ORCPT ); Mon, 29 Nov 2021 17:53:06 -0500 Received: by mail-qt1-f178.google.com with SMTP id q14so18265189qtx.10; Mon, 29 Nov 2021 14:49:46 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=oq7rSvRN8ZmF8826TeWA1nFVHMF74d3/NsU+2BMgrzA=; b=QyWasYPklsbKpNLnuhQAfkow9UuPzTC6g/dIr3OAC/szR6t0pyV0Eqp2gIAgSV8/wz kqVrdZ/9nawbYPuLhWwW8UzwS5l9lqTX8uILLnSZ2nV+amXV7kNmtVED1StmaAWrcj8s UuDtCNhxtC1X2YfUYfwVNNw1d0fA0BSJ3rYYnN0E4lzuR9B0QZh5op/nq1pf02ylWVy1 LEpJLmGeFmUNIXkqEBPRxGUIOnXMsDxwS6guJszi0eW4gAr8ZILn1ibfPpmNsGNMhkDi 4vX8/YWv8SRfa/BGrpYmeXMGTIPU7N0B4+3hpWZZ9tjt3/Ipx7qdAoPxjpyuMKbz8PEw m8kQ== X-Gm-Message-State: AOAM531vmJM2Sm4eI/CSORznBnBEgo6J7PmhypM2LQxrkbPcKs0g7MIP p7UGk3BlHn5p2hMdzCLoH9c= X-Google-Smtp-Source: ABdhPJxnPvgLuExnZ/ccsT7KqYyls4Id19XqxozgRZPGiHFhvLeIoUIbisNi5P/Pe4vmzv7DzMoClA== X-Received: by 2002:a05:622a:24c:: with SMTP id c12mr38014924qtx.483.1638226186251; Mon, 29 Nov 2021 14:49:46 -0800 (PST) Received: from fedora (pool-173-68-57-129.nycmny.fios.verizon.net. [173.68.57.129]) by smtp.gmail.com with ESMTPSA id r16sm9636543qkp.42.2021.11.29.14.49.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 29 Nov 2021 14:49:45 -0800 (PST) Date: Mon, 29 Nov 2021 17:49:43 -0500 From: Dennis Zhou To: Kefeng Wang Cc: akpm@linux-foundation.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, tj@kernel.org, gregkh@linuxfoundation.org, cl@linux.com, catalin.marinas@arm.com, will@kernel.org, tsbogend@alpha.franken.de, mpe@ellerman.id.au, benh@kernel.crashing.org, paulus@samba.org, paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, davem@davemloft.net, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, linux-arm-kernel@lists.infradead.org, linux-ia64@vger.kernel.org, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, sparclinux@vger.kernel.org, x86@kernel.org Subject: Re: [PATCH RFC 4/4] mm: percpu: Add generic pcpu_populate_pte() function Message-ID: References: <20211121093557.139034-1-wangkefeng.wang@huawei.com> <20211121093557.139034-5-wangkefeng.wang@huawei.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20211121093557.139034-5-wangkefeng.wang@huawei.com> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Sun, Nov 21, 2021 at 05:35:57PM +0800, Kefeng Wang wrote: > When NEED_PER_CPU_PAGE_FIRST_CHUNK enabled, we need a function to > populate pte, add a generic pcpu populate pte function and switch > to use it. > > Signed-off-by: Kefeng Wang > --- > arch/powerpc/kernel/setup_64.c | 47 +-------------------- > arch/sparc/kernel/smp_64.c | 57 +------------------------ > arch/x86/kernel/setup_percpu.c | 5 +-- > drivers/base/arch_numa.c | 51 +--------------------- > include/linux/percpu.h | 5 +-- > mm/percpu.c | 77 +++++++++++++++++++++++++++++++--- > 6 files changed, 79 insertions(+), 163 deletions(-) > > diff --git a/arch/powerpc/kernel/setup_64.c b/arch/powerpc/kernel/setup_64.c > index 364b1567f822..1a17828af77f 100644 > --- a/arch/powerpc/kernel/setup_64.c > +++ b/arch/powerpc/kernel/setup_64.c > @@ -788,51 +788,6 @@ static int pcpu_cpu_distance(unsigned int from, unsigned int to) > unsigned long __per_cpu_offset[NR_CPUS] __read_mostly; > EXPORT_SYMBOL(__per_cpu_offset); > > -static void __init pcpu_populate_pte(unsigned long addr) > -{ > - pgd_t *pgd = pgd_offset_k(addr); > - p4d_t *p4d; > - pud_t *pud; > - pmd_t *pmd; > - > - p4d = p4d_offset(pgd, addr); > - if (p4d_none(*p4d)) { > - pud_t *new; > - > - new = memblock_alloc(PUD_TABLE_SIZE, PUD_TABLE_SIZE); > - if (!new) > - goto err_alloc; > - p4d_populate(&init_mm, p4d, new); > - } > - > - pud = pud_offset(p4d, addr); > - if (pud_none(*pud)) { > - pmd_t *new; > - > - new = memblock_alloc(PMD_TABLE_SIZE, PMD_TABLE_SIZE); > - if (!new) > - goto err_alloc; > - pud_populate(&init_mm, pud, new); > - } > - > - pmd = pmd_offset(pud, addr); > - if (!pmd_present(*pmd)) { > - pte_t *new; > - > - new = memblock_alloc(PTE_TABLE_SIZE, PTE_TABLE_SIZE); > - if (!new) > - goto err_alloc; > - pmd_populate_kernel(&init_mm, pmd, new); > - } > - > - return; > - > -err_alloc: > - panic("%s: Failed to allocate %lu bytes align=%lx from=%lx\n", > - __func__, PAGE_SIZE, PAGE_SIZE, PAGE_SIZE); > -} > - > - > void __init setup_per_cpu_areas(void) > { > const size_t dyn_size = PERCPU_MODULE_RESERVE + PERCPU_DYNAMIC_RESERVE; > @@ -861,7 +816,7 @@ void __init setup_per_cpu_areas(void) > } > > if (rc < 0) > - rc = pcpu_page_first_chunk(0, pcpu_cpu_to_node, pcpu_populate_pte); > + rc = pcpu_page_first_chunk(0, pcpu_cpu_to_node); > if (rc < 0) > panic("cannot initialize percpu area (err=%d)", rc); > > diff --git a/arch/sparc/kernel/smp_64.c b/arch/sparc/kernel/smp_64.c > index 198dadddb75d..00dffe2d834b 100644 > --- a/arch/sparc/kernel/smp_64.c > +++ b/arch/sparc/kernel/smp_64.c > @@ -1534,59 +1534,6 @@ static int __init pcpu_cpu_distance(unsigned int from, unsigned int to) > return REMOTE_DISTANCE; > } > > -static void __init pcpu_populate_pte(unsigned long addr) > -{ > - pgd_t *pgd = pgd_offset_k(addr); > - p4d_t *p4d; > - pud_t *pud; > - pmd_t *pmd; > - > - if (pgd_none(*pgd)) { > - pud_t *new; > - > - new = memblock_alloc_from(PAGE_SIZE, PAGE_SIZE, PAGE_SIZE); > - if (!new) > - goto err_alloc; > - pgd_populate(&init_mm, pgd, new); > - } > - > - p4d = p4d_offset(pgd, addr); > - if (p4d_none(*p4d)) { > - pud_t *new; > - > - new = memblock_alloc_from(PAGE_SIZE, PAGE_SIZE, PAGE_SIZE); > - if (!new) > - goto err_alloc; > - p4d_populate(&init_mm, p4d, new); > - } > - > - pud = pud_offset(p4d, addr); > - if (pud_none(*pud)) { > - pmd_t *new; > - > - new = memblock_alloc_from(PAGE_SIZE, PAGE_SIZE, PAGE_SIZE); > - if (!new) > - goto err_alloc; > - pud_populate(&init_mm, pud, new); > - } > - > - pmd = pmd_offset(pud, addr); > - if (!pmd_present(*pmd)) { > - pte_t *new; > - > - new = memblock_alloc_from(PAGE_SIZE, PAGE_SIZE, PAGE_SIZE); > - if (!new) > - goto err_alloc; > - pmd_populate_kernel(&init_mm, pmd, new); > - } > - > - return; > - > -err_alloc: > - panic("%s: Failed to allocate %lu bytes align=%lx from=%lx\n", > - __func__, PAGE_SIZE, PAGE_SIZE, PAGE_SIZE); > -} > - > void __init setup_per_cpu_areas(void) > { > unsigned long delta; > @@ -1604,9 +1551,7 @@ void __init setup_per_cpu_areas(void) > pcpu_fc_names[pcpu_chosen_fc], rc); > } > if (rc < 0) > - rc = pcpu_page_first_chunk(PERCPU_MODULE_RESERVE, > - cpu_to_node, > - pcpu_populate_pte); > + rc = pcpu_page_first_chunk(PERCPU_MODULE_RESERVE, cpu_to_node); > if (rc < 0) > panic("cannot initialize percpu area (err=%d)", rc); > > diff --git a/arch/x86/kernel/setup_percpu.c b/arch/x86/kernel/setup_percpu.c > index cd672bd46241..4eadbe45078e 100644 > --- a/arch/x86/kernel/setup_percpu.c > +++ b/arch/x86/kernel/setup_percpu.c > @@ -101,7 +101,7 @@ static int __init pcpu_cpu_to_node(int cpu) > return IS_ENABLED(CONFIG_NUMA) ? early_cpu_to_node(cpu) : NUMA_NO_NODE; > } > > -static void __init pcpup_populate_pte(unsigned long addr) > +void __init pcpu_populate_pte(unsigned long addr) > { > populate_extra_pte(addr); > } > @@ -163,8 +163,7 @@ void __init setup_per_cpu_areas(void) > } > if (rc < 0) > rc = pcpu_page_first_chunk(PERCPU_FIRST_CHUNK_RESERVE, > - pcpu_cpu_to_node, > - pcpup_populate_pte); > + pcpu_cpu_to_node); x86 has it's own implementation that differs for 32 bit. I'm not confident this is correct to drop in as a replacement for x86, so I'd prefer to keep populate_pte_fn() around. > if (rc < 0) > panic("cannot initialize percpu area (err=%d)", rc); > > diff --git a/drivers/base/arch_numa.c b/drivers/base/arch_numa.c > index 23a10cc36165..eaa31e567d1e 100644 > --- a/drivers/base/arch_numa.c > +++ b/drivers/base/arch_numa.c > @@ -14,7 +14,6 @@ > #include > > #include > -#include > > struct pglist_data *node_data[MAX_NUMNODES] __read_mostly; > EXPORT_SYMBOL(node_data); > @@ -155,52 +154,6 @@ static int __init pcpu_cpu_distance(unsigned int from, unsigned int to) > return node_distance(early_cpu_to_node(from), early_cpu_to_node(to)); > } > > -#ifdef CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK > -static void __init pcpu_populate_pte(unsigned long addr) > -{ > - pgd_t *pgd = pgd_offset_k(addr); > - p4d_t *p4d; > - pud_t *pud; > - pmd_t *pmd; > - > - p4d = p4d_offset(pgd, addr); > - if (p4d_none(*p4d)) { > - pud_t *new; > - > - new = memblock_alloc(PAGE_SIZE, PAGE_SIZE); > - if (!new) > - goto err_alloc; > - p4d_populate(&init_mm, p4d, new); > - } > - > - pud = pud_offset(p4d, addr); > - if (pud_none(*pud)) { > - pmd_t *new; > - > - new = memblock_alloc(PAGE_SIZE, PAGE_SIZE); > - if (!new) > - goto err_alloc; > - pud_populate(&init_mm, pud, new); > - } > - > - pmd = pmd_offset(pud, addr); > - if (!pmd_present(*pmd)) { > - pte_t *new; > - > - new = memblock_alloc(PAGE_SIZE, PAGE_SIZE); > - if (!new) > - goto err_alloc; > - pmd_populate_kernel(&init_mm, pmd, new); > - } > - > - return; > - > -err_alloc: > - panic("%s: Failed to allocate %lu bytes align=%lx from=%lx\n", > - __func__, PAGE_SIZE, PAGE_SIZE, PAGE_SIZE); > -} > -#endif > - > void __init setup_per_cpu_areas(void) > { > unsigned long delta; > @@ -225,9 +178,7 @@ void __init setup_per_cpu_areas(void) > > #ifdef CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK > if (rc < 0) > - rc = pcpu_page_first_chunk(PERCPU_MODULE_RESERVE, > - early_cpu_to_node, > - pcpu_populate_pte); > + rc = pcpu_page_first_chunk(PERCPU_MODULE_RESERVE, early_cpu_to_node); > #endif > if (rc < 0) > panic("Failed to initialize percpu areas (err=%d).", rc); > diff --git a/include/linux/percpu.h b/include/linux/percpu.h > index d73c97ef4ff4..f1ec5ad1351c 100644 > --- a/include/linux/percpu.h > +++ b/include/linux/percpu.h > @@ -95,7 +95,6 @@ extern const char * const pcpu_fc_names[PCPU_FC_NR]; > extern enum pcpu_fc pcpu_chosen_fc; > > typedef int (pcpu_fc_cpu_to_node_fn_t)(int cpu); > -typedef void (*pcpu_fc_populate_pte_fn_t)(unsigned long addr); > typedef int (pcpu_fc_cpu_distance_fn_t)(unsigned int from, unsigned int to); > > extern struct pcpu_alloc_info * __init pcpu_alloc_alloc_info(int nr_groups, > @@ -113,9 +112,9 @@ extern int __init pcpu_embed_first_chunk(size_t reserved_size, size_t dyn_size, > #endif > > #ifdef CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK > +void __init pcpu_populate_pte(unsigned long addr); > extern int __init pcpu_page_first_chunk(size_t reserved_size, > - pcpu_fc_cpu_to_node_fn_t cpu_to_nd_fn, > - pcpu_fc_populate_pte_fn_t populate_pte_fn); > + pcpu_fc_cpu_to_node_fn_t cpu_to_nd_fn); > #endif > > extern void __percpu *__alloc_reserved_percpu(size_t size, size_t align) __alloc_size(1); > diff --git a/mm/percpu.c b/mm/percpu.c > index efaa1cbaf73d..d907daed04eb 100644 > --- a/mm/percpu.c > +++ b/mm/percpu.c > @@ -3162,11 +3162,80 @@ int __init pcpu_embed_first_chunk(size_t reserved_size, size_t dyn_size, > #endif /* BUILD_EMBED_FIRST_CHUNK */ > > #ifdef BUILD_PAGE_FIRST_CHUNK > +#include > + > +#ifndef P4D_TABLE_SIZE > +#define P4D_TABLE_SIZE PAGE_SIZE > +#endif > + > +#ifndef PUD_TABLE_SIZE > +#define PUD_TABLE_SIZE PAGE_SIZE > +#endif > + > +#ifndef PMD_TABLE_SIZE > +#define PMD_TABLE_SIZE PAGE_SIZE > +#endif > + > +#ifndef PTE_TABLE_SIZE > +#define PTE_TABLE_SIZE PAGE_SIZE > +#endif > +void __init __weak pcpu_populate_pte(unsigned long addr) > +{ > + pgd_t *pgd = pgd_offset_k(addr); > + p4d_t *p4d; > + pud_t *pud; > + pmd_t *pmd; > + > + if (pgd_none(*pgd)) { > + p4d_t *new; > + > + new = memblock_alloc_from(P4D_TABLE_SIZE, P4D_TABLE_SIZE, PAGE_SIZE); It's unnecessary to specify a min_addr to memblock_alloc_from() as it won't allocate 0 anyway. So please use memblock_alloc() instead. > + if (!new) > + goto err_alloc; > + pgd_populate(&init_mm, pgd, new); > + } > + > + p4d = p4d_offset(pgd, addr); > + if (p4d_none(*p4d)) { > + pud_t *new; > + > + new = memblock_alloc_from(PUD_TABLE_SIZE, PUD_TABLE_SIZE, PAGE_SIZE); See above. > + if (!new) > + goto err_alloc; > + p4d_populate(&init_mm, p4d, new); > + } > + > + pud = pud_offset(p4d, addr); > + if (pud_none(*pud)) { > + pmd_t *new; > + > + new = memblock_alloc_from(PMD_TABLE_SIZE, PMD_TABLE_SIZE, PAGE_SIZE); See above. > + if (!new) > + goto err_alloc; > + pud_populate(&init_mm, pud, new); > + } > + > + pmd = pmd_offset(pud, addr); > + if (!pmd_present(*pmd)) { > + pte_t *new; > + > + new = memblock_alloc_from(PTE_TABLE_SIZE, PTE_TABLE_SIZE, PAGE_SIZE); See above. > + if (!new) > + goto err_alloc; > + pmd_populate_kernel(&init_mm, pmd, new); > + } > + > + return; > + > +err_alloc: > + panic("%s: Failed to allocate %lu bytes align=%lx from=%lx\n", > + __func__, PAGE_SIZE, PAGE_SIZE, PAGE_SIZE); > +} > + > /** > * pcpu_page_first_chunk - map the first chunk using PAGE_SIZE pages > * @reserved_size: the size of reserved percpu area in bytes > * @cpu_to_nd_fn: callback to convert cpu to it's node, optional > - * @populate_pte_fn: function to populate pte > * > * This is a helper to ease setting up page-remapped first percpu > * chunk and can be called where pcpu_setup_first_chunk() is expected. > @@ -3177,9 +3246,7 @@ int __init pcpu_embed_first_chunk(size_t reserved_size, size_t dyn_size, > * RETURNS: > * 0 on success, -errno on failure. > */ > -int __init pcpu_page_first_chunk(size_t reserved_size, > - pcpu_fc_cpu_to_node_fn_t cpu_to_nd_fn, > - pcpu_fc_populate_pte_fn_t populate_pte_fn) > +int __init pcpu_page_first_chunk(size_t reserved_size, pcpu_fc_cpu_to_node_fn_t cpu_to_nd_fn) > { > static struct vm_struct vm; > struct pcpu_alloc_info *ai; > @@ -3243,7 +3310,7 @@ int __init pcpu_page_first_chunk(size_t reserved_size, > (unsigned long)vm.addr + unit * ai->unit_size; > > for (i = 0; i < unit_pages; i++) > - populate_pte_fn(unit_addr + (i << PAGE_SHIFT)); > + pcpu_populate_pte(unit_addr + (i << PAGE_SHIFT)); > > /* pte already populated, the following shouldn't fail */ > rc = __pcpu_map_pages(unit_addr, &pages[unit * unit_pages], > -- > 2.26.2 > From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 89ED4C433F5 for ; Mon, 29 Nov 2021 22:50:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=1ZQO//Zc37n6la/PRt9CkAiFWgJ86zDGESI6ClBS9VA=; b=Md5mTXAxOnwSP0 H0yo++CHuexAgD3s8qhGiVPtocKIntHBaNEOhh1mYlD+lgjc67BRkkGExObwjmC1iFSQQGcYVfqYM CB0vGfWfMfw/aIR+RrCnwQPXMcEyrbsQG5a9dXM/XCpokfhqQow2+AmLaXOGLtCRCCPIeAm2ZZrDQ 8e+vCUDMMMKGub+F3Fqj0SBb3YdDw8+PlxTcUvm8V75TKMhj4+wkq5Cce1Z1HYKHdt3aVb9xXRYsl scRWoi5BGTiXa1efdtA5CDd4jQM1rL4t+Pf8JlnHX484/7xNFlx0oqAS9pYg7mQeXGfNlwXNa7qdz Y0RbmML+tDAz5eKM/hnA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1mrpTH-002xrR-F4; Mon, 29 Nov 2021 22:50:03 +0000 Received: from mail-qt1-f176.google.com ([209.85.160.176]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1mrpT1-002xmj-Fp; Mon, 29 Nov 2021 22:49:49 +0000 Received: by mail-qt1-f176.google.com with SMTP id j17so18309084qtx.2; Mon, 29 Nov 2021 14:49:47 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=oq7rSvRN8ZmF8826TeWA1nFVHMF74d3/NsU+2BMgrzA=; b=WvhZlYdJ+8VeX2crtY30iWKzRVVrm7dgy/mXtjlo2Mvmz0MW65Lg567J6eK+V3OaBk 3P6vb5H8vs6K2j8C5oN1bqcGeCZe3FLeaPf3XzQnxO7InFQuDMKhimFNzJCVoqPLLJav cPunfSzNQZ6itEUsIPdPDJz9cGBEwR30sKhA6uiGPXWhMUGZqvJ0ww5O/Xb2O7xeiXnd vFMZ1g1OStqiSgQJdiZa4BLby91GlpKxNaRFhFKOWDAC9KjBM4VQ82d/mBavbI/pLj65 l7WqlMW4PJkq3kSTNZm8mD0YRUNKwHZkgKr+PXEW9Gj7TMGWOnNRSwxTF5LhGUt2ONop kqUw== X-Gm-Message-State: AOAM533wB7jRC0aTWW7IFf6i+grzJ19wY6aLR4BPOoWiHlpLcFWwugTs thHpzsAvoOvx1+xUya+fuCc= X-Google-Smtp-Source: ABdhPJxnPvgLuExnZ/ccsT7KqYyls4Id19XqxozgRZPGiHFhvLeIoUIbisNi5P/Pe4vmzv7DzMoClA== X-Received: by 2002:a05:622a:24c:: with SMTP id c12mr38014924qtx.483.1638226186251; Mon, 29 Nov 2021 14:49:46 -0800 (PST) Received: from fedora (pool-173-68-57-129.nycmny.fios.verizon.net. [173.68.57.129]) by smtp.gmail.com with ESMTPSA id r16sm9636543qkp.42.2021.11.29.14.49.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 29 Nov 2021 14:49:45 -0800 (PST) Date: Mon, 29 Nov 2021 17:49:43 -0500 From: Dennis Zhou To: Kefeng Wang Cc: akpm@linux-foundation.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, tj@kernel.org, gregkh@linuxfoundation.org, cl@linux.com, catalin.marinas@arm.com, will@kernel.org, tsbogend@alpha.franken.de, mpe@ellerman.id.au, benh@kernel.crashing.org, paulus@samba.org, paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, davem@davemloft.net, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, linux-arm-kernel@lists.infradead.org, linux-ia64@vger.kernel.org, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, sparclinux@vger.kernel.org, x86@kernel.org Subject: Re: [PATCH RFC 4/4] mm: percpu: Add generic pcpu_populate_pte() function Message-ID: References: <20211121093557.139034-1-wangkefeng.wang@huawei.com> <20211121093557.139034-5-wangkefeng.wang@huawei.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20211121093557.139034-5-wangkefeng.wang@huawei.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20211129_144947_578744_03D2CE4B X-CRM114-Status: GOOD ( 29.33 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org On Sun, Nov 21, 2021 at 05:35:57PM +0800, Kefeng Wang wrote: > When NEED_PER_CPU_PAGE_FIRST_CHUNK enabled, we need a function to > populate pte, add a generic pcpu populate pte function and switch > to use it. > > Signed-off-by: Kefeng Wang > --- > arch/powerpc/kernel/setup_64.c | 47 +-------------------- > arch/sparc/kernel/smp_64.c | 57 +------------------------ > arch/x86/kernel/setup_percpu.c | 5 +-- > drivers/base/arch_numa.c | 51 +--------------------- > include/linux/percpu.h | 5 +-- > mm/percpu.c | 77 +++++++++++++++++++++++++++++++--- > 6 files changed, 79 insertions(+), 163 deletions(-) > > diff --git a/arch/powerpc/kernel/setup_64.c b/arch/powerpc/kernel/setup_64.c > index 364b1567f822..1a17828af77f 100644 > --- a/arch/powerpc/kernel/setup_64.c > +++ b/arch/powerpc/kernel/setup_64.c > @@ -788,51 +788,6 @@ static int pcpu_cpu_distance(unsigned int from, unsigned int to) > unsigned long __per_cpu_offset[NR_CPUS] __read_mostly; > EXPORT_SYMBOL(__per_cpu_offset); > > -static void __init pcpu_populate_pte(unsigned long addr) > -{ > - pgd_t *pgd = pgd_offset_k(addr); > - p4d_t *p4d; > - pud_t *pud; > - pmd_t *pmd; > - > - p4d = p4d_offset(pgd, addr); > - if (p4d_none(*p4d)) { > - pud_t *new; > - > - new = memblock_alloc(PUD_TABLE_SIZE, PUD_TABLE_SIZE); > - if (!new) > - goto err_alloc; > - p4d_populate(&init_mm, p4d, new); > - } > - > - pud = pud_offset(p4d, addr); > - if (pud_none(*pud)) { > - pmd_t *new; > - > - new = memblock_alloc(PMD_TABLE_SIZE, PMD_TABLE_SIZE); > - if (!new) > - goto err_alloc; > - pud_populate(&init_mm, pud, new); > - } > - > - pmd = pmd_offset(pud, addr); > - if (!pmd_present(*pmd)) { > - pte_t *new; > - > - new = memblock_alloc(PTE_TABLE_SIZE, PTE_TABLE_SIZE); > - if (!new) > - goto err_alloc; > - pmd_populate_kernel(&init_mm, pmd, new); > - } > - > - return; > - > -err_alloc: > - panic("%s: Failed to allocate %lu bytes align=%lx from=%lx\n", > - __func__, PAGE_SIZE, PAGE_SIZE, PAGE_SIZE); > -} > - > - > void __init setup_per_cpu_areas(void) > { > const size_t dyn_size = PERCPU_MODULE_RESERVE + PERCPU_DYNAMIC_RESERVE; > @@ -861,7 +816,7 @@ void __init setup_per_cpu_areas(void) > } > > if (rc < 0) > - rc = pcpu_page_first_chunk(0, pcpu_cpu_to_node, pcpu_populate_pte); > + rc = pcpu_page_first_chunk(0, pcpu_cpu_to_node); > if (rc < 0) > panic("cannot initialize percpu area (err=%d)", rc); > > diff --git a/arch/sparc/kernel/smp_64.c b/arch/sparc/kernel/smp_64.c > index 198dadddb75d..00dffe2d834b 100644 > --- a/arch/sparc/kernel/smp_64.c > +++ b/arch/sparc/kernel/smp_64.c > @@ -1534,59 +1534,6 @@ static int __init pcpu_cpu_distance(unsigned int from, unsigned int to) > return REMOTE_DISTANCE; > } > > -static void __init pcpu_populate_pte(unsigned long addr) > -{ > - pgd_t *pgd = pgd_offset_k(addr); > - p4d_t *p4d; > - pud_t *pud; > - pmd_t *pmd; > - > - if (pgd_none(*pgd)) { > - pud_t *new; > - > - new = memblock_alloc_from(PAGE_SIZE, PAGE_SIZE, PAGE_SIZE); > - if (!new) > - goto err_alloc; > - pgd_populate(&init_mm, pgd, new); > - } > - > - p4d = p4d_offset(pgd, addr); > - if (p4d_none(*p4d)) { > - pud_t *new; > - > - new = memblock_alloc_from(PAGE_SIZE, PAGE_SIZE, PAGE_SIZE); > - if (!new) > - goto err_alloc; > - p4d_populate(&init_mm, p4d, new); > - } > - > - pud = pud_offset(p4d, addr); > - if (pud_none(*pud)) { > - pmd_t *new; > - > - new = memblock_alloc_from(PAGE_SIZE, PAGE_SIZE, PAGE_SIZE); > - if (!new) > - goto err_alloc; > - pud_populate(&init_mm, pud, new); > - } > - > - pmd = pmd_offset(pud, addr); > - if (!pmd_present(*pmd)) { > - pte_t *new; > - > - new = memblock_alloc_from(PAGE_SIZE, PAGE_SIZE, PAGE_SIZE); > - if (!new) > - goto err_alloc; > - pmd_populate_kernel(&init_mm, pmd, new); > - } > - > - return; > - > -err_alloc: > - panic("%s: Failed to allocate %lu bytes align=%lx from=%lx\n", > - __func__, PAGE_SIZE, PAGE_SIZE, PAGE_SIZE); > -} > - > void __init setup_per_cpu_areas(void) > { > unsigned long delta; > @@ -1604,9 +1551,7 @@ void __init setup_per_cpu_areas(void) > pcpu_fc_names[pcpu_chosen_fc], rc); > } > if (rc < 0) > - rc = pcpu_page_first_chunk(PERCPU_MODULE_RESERVE, > - cpu_to_node, > - pcpu_populate_pte); > + rc = pcpu_page_first_chunk(PERCPU_MODULE_RESERVE, cpu_to_node); > if (rc < 0) > panic("cannot initialize percpu area (err=%d)", rc); > > diff --git a/arch/x86/kernel/setup_percpu.c b/arch/x86/kernel/setup_percpu.c > index cd672bd46241..4eadbe45078e 100644 > --- a/arch/x86/kernel/setup_percpu.c > +++ b/arch/x86/kernel/setup_percpu.c > @@ -101,7 +101,7 @@ static int __init pcpu_cpu_to_node(int cpu) > return IS_ENABLED(CONFIG_NUMA) ? early_cpu_to_node(cpu) : NUMA_NO_NODE; > } > > -static void __init pcpup_populate_pte(unsigned long addr) > +void __init pcpu_populate_pte(unsigned long addr) > { > populate_extra_pte(addr); > } > @@ -163,8 +163,7 @@ void __init setup_per_cpu_areas(void) > } > if (rc < 0) > rc = pcpu_page_first_chunk(PERCPU_FIRST_CHUNK_RESERVE, > - pcpu_cpu_to_node, > - pcpup_populate_pte); > + pcpu_cpu_to_node); x86 has it's own implementation that differs for 32 bit. I'm not confident this is correct to drop in as a replacement for x86, so I'd prefer to keep populate_pte_fn() around. > if (rc < 0) > panic("cannot initialize percpu area (err=%d)", rc); > > diff --git a/drivers/base/arch_numa.c b/drivers/base/arch_numa.c > index 23a10cc36165..eaa31e567d1e 100644 > --- a/drivers/base/arch_numa.c > +++ b/drivers/base/arch_numa.c > @@ -14,7 +14,6 @@ > #include > > #include > -#include > > struct pglist_data *node_data[MAX_NUMNODES] __read_mostly; > EXPORT_SYMBOL(node_data); > @@ -155,52 +154,6 @@ static int __init pcpu_cpu_distance(unsigned int from, unsigned int to) > return node_distance(early_cpu_to_node(from), early_cpu_to_node(to)); > } > > -#ifdef CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK > -static void __init pcpu_populate_pte(unsigned long addr) > -{ > - pgd_t *pgd = pgd_offset_k(addr); > - p4d_t *p4d; > - pud_t *pud; > - pmd_t *pmd; > - > - p4d = p4d_offset(pgd, addr); > - if (p4d_none(*p4d)) { > - pud_t *new; > - > - new = memblock_alloc(PAGE_SIZE, PAGE_SIZE); > - if (!new) > - goto err_alloc; > - p4d_populate(&init_mm, p4d, new); > - } > - > - pud = pud_offset(p4d, addr); > - if (pud_none(*pud)) { > - pmd_t *new; > - > - new = memblock_alloc(PAGE_SIZE, PAGE_SIZE); > - if (!new) > - goto err_alloc; > - pud_populate(&init_mm, pud, new); > - } > - > - pmd = pmd_offset(pud, addr); > - if (!pmd_present(*pmd)) { > - pte_t *new; > - > - new = memblock_alloc(PAGE_SIZE, PAGE_SIZE); > - if (!new) > - goto err_alloc; > - pmd_populate_kernel(&init_mm, pmd, new); > - } > - > - return; > - > -err_alloc: > - panic("%s: Failed to allocate %lu bytes align=%lx from=%lx\n", > - __func__, PAGE_SIZE, PAGE_SIZE, PAGE_SIZE); > -} > -#endif > - > void __init setup_per_cpu_areas(void) > { > unsigned long delta; > @@ -225,9 +178,7 @@ void __init setup_per_cpu_areas(void) > > #ifdef CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK > if (rc < 0) > - rc = pcpu_page_first_chunk(PERCPU_MODULE_RESERVE, > - early_cpu_to_node, > - pcpu_populate_pte); > + rc = pcpu_page_first_chunk(PERCPU_MODULE_RESERVE, early_cpu_to_node); > #endif > if (rc < 0) > panic("Failed to initialize percpu areas (err=%d).", rc); > diff --git a/include/linux/percpu.h b/include/linux/percpu.h > index d73c97ef4ff4..f1ec5ad1351c 100644 > --- a/include/linux/percpu.h > +++ b/include/linux/percpu.h > @@ -95,7 +95,6 @@ extern const char * const pcpu_fc_names[PCPU_FC_NR]; > extern enum pcpu_fc pcpu_chosen_fc; > > typedef int (pcpu_fc_cpu_to_node_fn_t)(int cpu); > -typedef void (*pcpu_fc_populate_pte_fn_t)(unsigned long addr); > typedef int (pcpu_fc_cpu_distance_fn_t)(unsigned int from, unsigned int to); > > extern struct pcpu_alloc_info * __init pcpu_alloc_alloc_info(int nr_groups, > @@ -113,9 +112,9 @@ extern int __init pcpu_embed_first_chunk(size_t reserved_size, size_t dyn_size, > #endif > > #ifdef CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK > +void __init pcpu_populate_pte(unsigned long addr); > extern int __init pcpu_page_first_chunk(size_t reserved_size, > - pcpu_fc_cpu_to_node_fn_t cpu_to_nd_fn, > - pcpu_fc_populate_pte_fn_t populate_pte_fn); > + pcpu_fc_cpu_to_node_fn_t cpu_to_nd_fn); > #endif > > extern void __percpu *__alloc_reserved_percpu(size_t size, size_t align) __alloc_size(1); > diff --git a/mm/percpu.c b/mm/percpu.c > index efaa1cbaf73d..d907daed04eb 100644 > --- a/mm/percpu.c > +++ b/mm/percpu.c > @@ -3162,11 +3162,80 @@ int __init pcpu_embed_first_chunk(size_t reserved_size, size_t dyn_size, > #endif /* BUILD_EMBED_FIRST_CHUNK */ > > #ifdef BUILD_PAGE_FIRST_CHUNK > +#include > + > +#ifndef P4D_TABLE_SIZE > +#define P4D_TABLE_SIZE PAGE_SIZE > +#endif > + > +#ifndef PUD_TABLE_SIZE > +#define PUD_TABLE_SIZE PAGE_SIZE > +#endif > + > +#ifndef PMD_TABLE_SIZE > +#define PMD_TABLE_SIZE PAGE_SIZE > +#endif > + > +#ifndef PTE_TABLE_SIZE > +#define PTE_TABLE_SIZE PAGE_SIZE > +#endif > +void __init __weak pcpu_populate_pte(unsigned long addr) > +{ > + pgd_t *pgd = pgd_offset_k(addr); > + p4d_t *p4d; > + pud_t *pud; > + pmd_t *pmd; > + > + if (pgd_none(*pgd)) { > + p4d_t *new; > + > + new = memblock_alloc_from(P4D_TABLE_SIZE, P4D_TABLE_SIZE, PAGE_SIZE); It's unnecessary to specify a min_addr to memblock_alloc_from() as it won't allocate 0 anyway. So please use memblock_alloc() instead. > + if (!new) > + goto err_alloc; > + pgd_populate(&init_mm, pgd, new); > + } > + > + p4d = p4d_offset(pgd, addr); > + if (p4d_none(*p4d)) { > + pud_t *new; > + > + new = memblock_alloc_from(PUD_TABLE_SIZE, PUD_TABLE_SIZE, PAGE_SIZE); See above. > + if (!new) > + goto err_alloc; > + p4d_populate(&init_mm, p4d, new); > + } > + > + pud = pud_offset(p4d, addr); > + if (pud_none(*pud)) { > + pmd_t *new; > + > + new = memblock_alloc_from(PMD_TABLE_SIZE, PMD_TABLE_SIZE, PAGE_SIZE); See above. > + if (!new) > + goto err_alloc; > + pud_populate(&init_mm, pud, new); > + } > + > + pmd = pmd_offset(pud, addr); > + if (!pmd_present(*pmd)) { > + pte_t *new; > + > + new = memblock_alloc_from(PTE_TABLE_SIZE, PTE_TABLE_SIZE, PAGE_SIZE); See above. > + if (!new) > + goto err_alloc; > + pmd_populate_kernel(&init_mm, pmd, new); > + } > + > + return; > + > +err_alloc: > + panic("%s: Failed to allocate %lu bytes align=%lx from=%lx\n", > + __func__, PAGE_SIZE, PAGE_SIZE, PAGE_SIZE); > +} > + > /** > * pcpu_page_first_chunk - map the first chunk using PAGE_SIZE pages > * @reserved_size: the size of reserved percpu area in bytes > * @cpu_to_nd_fn: callback to convert cpu to it's node, optional > - * @populate_pte_fn: function to populate pte > * > * This is a helper to ease setting up page-remapped first percpu > * chunk and can be called where pcpu_setup_first_chunk() is expected. > @@ -3177,9 +3246,7 @@ int __init pcpu_embed_first_chunk(size_t reserved_size, size_t dyn_size, > * RETURNS: > * 0 on success, -errno on failure. > */ > -int __init pcpu_page_first_chunk(size_t reserved_size, > - pcpu_fc_cpu_to_node_fn_t cpu_to_nd_fn, > - pcpu_fc_populate_pte_fn_t populate_pte_fn) > +int __init pcpu_page_first_chunk(size_t reserved_size, pcpu_fc_cpu_to_node_fn_t cpu_to_nd_fn) > { > static struct vm_struct vm; > struct pcpu_alloc_info *ai; > @@ -3243,7 +3310,7 @@ int __init pcpu_page_first_chunk(size_t reserved_size, > (unsigned long)vm.addr + unit * ai->unit_size; > > for (i = 0; i < unit_pages; i++) > - populate_pte_fn(unit_addr + (i << PAGE_SHIFT)); > + pcpu_populate_pte(unit_addr + (i << PAGE_SHIFT)); > > /* pte already populated, the following shouldn't fail */ > rc = __pcpu_map_pages(unit_addr, &pages[unit * unit_pages], > -- > 2.26.2 > _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3FDCBC433FE for ; Mon, 29 Nov 2021 22:50:15 +0000 (UTC) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4J30tF4Kycz30R6 for ; Tue, 30 Nov 2021 09:50:13 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=209.85.160.176; helo=mail-qt1-f176.google.com; envelope-from=dennisszhou@gmail.com; receiver=) Received: from mail-qt1-f176.google.com (mail-qt1-f176.google.com [209.85.160.176]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4J30sn0gTvz2xXC for ; Tue, 30 Nov 2021 09:49:48 +1100 (AEDT) Received: by mail-qt1-f176.google.com with SMTP id v22so18287796qtx.8 for ; Mon, 29 Nov 2021 14:49:48 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=oq7rSvRN8ZmF8826TeWA1nFVHMF74d3/NsU+2BMgrzA=; b=dDdBRYyhEnmlmAt2Lyr5KZy7nBErF/vhDGMKJtog25jFQc5fMXNmVssVd46SVrWV46 pYCV++gOn2CcjZf1yLyuid7xFxSQFhfwqUdRwnmGCz5wEU/Jxi//wvpV6Jj7shMly6WH 73JBQoaUxz/VpA3L6+H8GbRRuc7hoDpSsnyUh6uoideq/y4xzA6Ibl33aRXzvTsCPw2O s8fuJvC/YP3ENHDJArlX3xcrz0Sdi0Edh0RpKW8BYVhwJ1Sk4n9LmW3M9Z2rYEg48bFU y1jyvzlg+C3SWbdxiyHDH1/KmUeK7Js8cqZm2z+nzSCqinGxHNGqIt21j9YOh7whS3Nv n+ZQ== X-Gm-Message-State: AOAM530+J/gUqD8xiraMbIHlMqLg5FD3tUhx5YrcrlukG3svN/BoYHSo qxHwSjtnpzim9kZn70grZ8Y= X-Google-Smtp-Source: ABdhPJxnPvgLuExnZ/ccsT7KqYyls4Id19XqxozgRZPGiHFhvLeIoUIbisNi5P/Pe4vmzv7DzMoClA== X-Received: by 2002:a05:622a:24c:: with SMTP id c12mr38014924qtx.483.1638226186251; Mon, 29 Nov 2021 14:49:46 -0800 (PST) Received: from fedora (pool-173-68-57-129.nycmny.fios.verizon.net. [173.68.57.129]) by smtp.gmail.com with ESMTPSA id r16sm9636543qkp.42.2021.11.29.14.49.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 29 Nov 2021 14:49:45 -0800 (PST) Date: Mon, 29 Nov 2021 17:49:43 -0500 From: Dennis Zhou To: Kefeng Wang Subject: Re: [PATCH RFC 4/4] mm: percpu: Add generic pcpu_populate_pte() function Message-ID: References: <20211121093557.139034-1-wangkefeng.wang@huawei.com> <20211121093557.139034-5-wangkefeng.wang@huawei.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20211121093557.139034-5-wangkefeng.wang@huawei.com> X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-ia64@vger.kernel.org, dave.hansen@linux.intel.com, linux-mips@vger.kernel.org, linux-mm@kvack.org, paulus@samba.org, hpa@zytor.com, sparclinux@vger.kernel.org, cl@linux.com, will@kernel.org, linux-riscv@lists.infradead.org, x86@kernel.org, mingo@redhat.com, catalin.marinas@arm.com, aou@eecs.berkeley.edu, bp@alien8.de, paul.walmsley@sifive.com, tglx@linutronix.de, linux-arm-kernel@lists.infradead.org, tsbogend@alpha.franken.de, gregkh@linuxfoundation.org, linux-kernel@vger.kernel.org, palmer@dabbelt.com, tj@kernel.org, akpm@linux-foundation.org, linuxppc-dev@lists.ozlabs.org, davem@davemloft.net Errors-To: linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Sender: "Linuxppc-dev" On Sun, Nov 21, 2021 at 05:35:57PM +0800, Kefeng Wang wrote: > When NEED_PER_CPU_PAGE_FIRST_CHUNK enabled, we need a function to > populate pte, add a generic pcpu populate pte function and switch > to use it. > > Signed-off-by: Kefeng Wang > --- > arch/powerpc/kernel/setup_64.c | 47 +-------------------- > arch/sparc/kernel/smp_64.c | 57 +------------------------ > arch/x86/kernel/setup_percpu.c | 5 +-- > drivers/base/arch_numa.c | 51 +--------------------- > include/linux/percpu.h | 5 +-- > mm/percpu.c | 77 +++++++++++++++++++++++++++++++--- > 6 files changed, 79 insertions(+), 163 deletions(-) > > diff --git a/arch/powerpc/kernel/setup_64.c b/arch/powerpc/kernel/setup_64.c > index 364b1567f822..1a17828af77f 100644 > --- a/arch/powerpc/kernel/setup_64.c > +++ b/arch/powerpc/kernel/setup_64.c > @@ -788,51 +788,6 @@ static int pcpu_cpu_distance(unsigned int from, unsigned int to) > unsigned long __per_cpu_offset[NR_CPUS] __read_mostly; > EXPORT_SYMBOL(__per_cpu_offset); > > -static void __init pcpu_populate_pte(unsigned long addr) > -{ > - pgd_t *pgd = pgd_offset_k(addr); > - p4d_t *p4d; > - pud_t *pud; > - pmd_t *pmd; > - > - p4d = p4d_offset(pgd, addr); > - if (p4d_none(*p4d)) { > - pud_t *new; > - > - new = memblock_alloc(PUD_TABLE_SIZE, PUD_TABLE_SIZE); > - if (!new) > - goto err_alloc; > - p4d_populate(&init_mm, p4d, new); > - } > - > - pud = pud_offset(p4d, addr); > - if (pud_none(*pud)) { > - pmd_t *new; > - > - new = memblock_alloc(PMD_TABLE_SIZE, PMD_TABLE_SIZE); > - if (!new) > - goto err_alloc; > - pud_populate(&init_mm, pud, new); > - } > - > - pmd = pmd_offset(pud, addr); > - if (!pmd_present(*pmd)) { > - pte_t *new; > - > - new = memblock_alloc(PTE_TABLE_SIZE, PTE_TABLE_SIZE); > - if (!new) > - goto err_alloc; > - pmd_populate_kernel(&init_mm, pmd, new); > - } > - > - return; > - > -err_alloc: > - panic("%s: Failed to allocate %lu bytes align=%lx from=%lx\n", > - __func__, PAGE_SIZE, PAGE_SIZE, PAGE_SIZE); > -} > - > - > void __init setup_per_cpu_areas(void) > { > const size_t dyn_size = PERCPU_MODULE_RESERVE + PERCPU_DYNAMIC_RESERVE; > @@ -861,7 +816,7 @@ void __init setup_per_cpu_areas(void) > } > > if (rc < 0) > - rc = pcpu_page_first_chunk(0, pcpu_cpu_to_node, pcpu_populate_pte); > + rc = pcpu_page_first_chunk(0, pcpu_cpu_to_node); > if (rc < 0) > panic("cannot initialize percpu area (err=%d)", rc); > > diff --git a/arch/sparc/kernel/smp_64.c b/arch/sparc/kernel/smp_64.c > index 198dadddb75d..00dffe2d834b 100644 > --- a/arch/sparc/kernel/smp_64.c > +++ b/arch/sparc/kernel/smp_64.c > @@ -1534,59 +1534,6 @@ static int __init pcpu_cpu_distance(unsigned int from, unsigned int to) > return REMOTE_DISTANCE; > } > > -static void __init pcpu_populate_pte(unsigned long addr) > -{ > - pgd_t *pgd = pgd_offset_k(addr); > - p4d_t *p4d; > - pud_t *pud; > - pmd_t *pmd; > - > - if (pgd_none(*pgd)) { > - pud_t *new; > - > - new = memblock_alloc_from(PAGE_SIZE, PAGE_SIZE, PAGE_SIZE); > - if (!new) > - goto err_alloc; > - pgd_populate(&init_mm, pgd, new); > - } > - > - p4d = p4d_offset(pgd, addr); > - if (p4d_none(*p4d)) { > - pud_t *new; > - > - new = memblock_alloc_from(PAGE_SIZE, PAGE_SIZE, PAGE_SIZE); > - if (!new) > - goto err_alloc; > - p4d_populate(&init_mm, p4d, new); > - } > - > - pud = pud_offset(p4d, addr); > - if (pud_none(*pud)) { > - pmd_t *new; > - > - new = memblock_alloc_from(PAGE_SIZE, PAGE_SIZE, PAGE_SIZE); > - if (!new) > - goto err_alloc; > - pud_populate(&init_mm, pud, new); > - } > - > - pmd = pmd_offset(pud, addr); > - if (!pmd_present(*pmd)) { > - pte_t *new; > - > - new = memblock_alloc_from(PAGE_SIZE, PAGE_SIZE, PAGE_SIZE); > - if (!new) > - goto err_alloc; > - pmd_populate_kernel(&init_mm, pmd, new); > - } > - > - return; > - > -err_alloc: > - panic("%s: Failed to allocate %lu bytes align=%lx from=%lx\n", > - __func__, PAGE_SIZE, PAGE_SIZE, PAGE_SIZE); > -} > - > void __init setup_per_cpu_areas(void) > { > unsigned long delta; > @@ -1604,9 +1551,7 @@ void __init setup_per_cpu_areas(void) > pcpu_fc_names[pcpu_chosen_fc], rc); > } > if (rc < 0) > - rc = pcpu_page_first_chunk(PERCPU_MODULE_RESERVE, > - cpu_to_node, > - pcpu_populate_pte); > + rc = pcpu_page_first_chunk(PERCPU_MODULE_RESERVE, cpu_to_node); > if (rc < 0) > panic("cannot initialize percpu area (err=%d)", rc); > > diff --git a/arch/x86/kernel/setup_percpu.c b/arch/x86/kernel/setup_percpu.c > index cd672bd46241..4eadbe45078e 100644 > --- a/arch/x86/kernel/setup_percpu.c > +++ b/arch/x86/kernel/setup_percpu.c > @@ -101,7 +101,7 @@ static int __init pcpu_cpu_to_node(int cpu) > return IS_ENABLED(CONFIG_NUMA) ? early_cpu_to_node(cpu) : NUMA_NO_NODE; > } > > -static void __init pcpup_populate_pte(unsigned long addr) > +void __init pcpu_populate_pte(unsigned long addr) > { > populate_extra_pte(addr); > } > @@ -163,8 +163,7 @@ void __init setup_per_cpu_areas(void) > } > if (rc < 0) > rc = pcpu_page_first_chunk(PERCPU_FIRST_CHUNK_RESERVE, > - pcpu_cpu_to_node, > - pcpup_populate_pte); > + pcpu_cpu_to_node); x86 has it's own implementation that differs for 32 bit. I'm not confident this is correct to drop in as a replacement for x86, so I'd prefer to keep populate_pte_fn() around. > if (rc < 0) > panic("cannot initialize percpu area (err=%d)", rc); > > diff --git a/drivers/base/arch_numa.c b/drivers/base/arch_numa.c > index 23a10cc36165..eaa31e567d1e 100644 > --- a/drivers/base/arch_numa.c > +++ b/drivers/base/arch_numa.c > @@ -14,7 +14,6 @@ > #include > > #include > -#include > > struct pglist_data *node_data[MAX_NUMNODES] __read_mostly; > EXPORT_SYMBOL(node_data); > @@ -155,52 +154,6 @@ static int __init pcpu_cpu_distance(unsigned int from, unsigned int to) > return node_distance(early_cpu_to_node(from), early_cpu_to_node(to)); > } > > -#ifdef CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK > -static void __init pcpu_populate_pte(unsigned long addr) > -{ > - pgd_t *pgd = pgd_offset_k(addr); > - p4d_t *p4d; > - pud_t *pud; > - pmd_t *pmd; > - > - p4d = p4d_offset(pgd, addr); > - if (p4d_none(*p4d)) { > - pud_t *new; > - > - new = memblock_alloc(PAGE_SIZE, PAGE_SIZE); > - if (!new) > - goto err_alloc; > - p4d_populate(&init_mm, p4d, new); > - } > - > - pud = pud_offset(p4d, addr); > - if (pud_none(*pud)) { > - pmd_t *new; > - > - new = memblock_alloc(PAGE_SIZE, PAGE_SIZE); > - if (!new) > - goto err_alloc; > - pud_populate(&init_mm, pud, new); > - } > - > - pmd = pmd_offset(pud, addr); > - if (!pmd_present(*pmd)) { > - pte_t *new; > - > - new = memblock_alloc(PAGE_SIZE, PAGE_SIZE); > - if (!new) > - goto err_alloc; > - pmd_populate_kernel(&init_mm, pmd, new); > - } > - > - return; > - > -err_alloc: > - panic("%s: Failed to allocate %lu bytes align=%lx from=%lx\n", > - __func__, PAGE_SIZE, PAGE_SIZE, PAGE_SIZE); > -} > -#endif > - > void __init setup_per_cpu_areas(void) > { > unsigned long delta; > @@ -225,9 +178,7 @@ void __init setup_per_cpu_areas(void) > > #ifdef CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK > if (rc < 0) > - rc = pcpu_page_first_chunk(PERCPU_MODULE_RESERVE, > - early_cpu_to_node, > - pcpu_populate_pte); > + rc = pcpu_page_first_chunk(PERCPU_MODULE_RESERVE, early_cpu_to_node); > #endif > if (rc < 0) > panic("Failed to initialize percpu areas (err=%d).", rc); > diff --git a/include/linux/percpu.h b/include/linux/percpu.h > index d73c97ef4ff4..f1ec5ad1351c 100644 > --- a/include/linux/percpu.h > +++ b/include/linux/percpu.h > @@ -95,7 +95,6 @@ extern const char * const pcpu_fc_names[PCPU_FC_NR]; > extern enum pcpu_fc pcpu_chosen_fc; > > typedef int (pcpu_fc_cpu_to_node_fn_t)(int cpu); > -typedef void (*pcpu_fc_populate_pte_fn_t)(unsigned long addr); > typedef int (pcpu_fc_cpu_distance_fn_t)(unsigned int from, unsigned int to); > > extern struct pcpu_alloc_info * __init pcpu_alloc_alloc_info(int nr_groups, > @@ -113,9 +112,9 @@ extern int __init pcpu_embed_first_chunk(size_t reserved_size, size_t dyn_size, > #endif > > #ifdef CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK > +void __init pcpu_populate_pte(unsigned long addr); > extern int __init pcpu_page_first_chunk(size_t reserved_size, > - pcpu_fc_cpu_to_node_fn_t cpu_to_nd_fn, > - pcpu_fc_populate_pte_fn_t populate_pte_fn); > + pcpu_fc_cpu_to_node_fn_t cpu_to_nd_fn); > #endif > > extern void __percpu *__alloc_reserved_percpu(size_t size, size_t align) __alloc_size(1); > diff --git a/mm/percpu.c b/mm/percpu.c > index efaa1cbaf73d..d907daed04eb 100644 > --- a/mm/percpu.c > +++ b/mm/percpu.c > @@ -3162,11 +3162,80 @@ int __init pcpu_embed_first_chunk(size_t reserved_size, size_t dyn_size, > #endif /* BUILD_EMBED_FIRST_CHUNK */ > > #ifdef BUILD_PAGE_FIRST_CHUNK > +#include > + > +#ifndef P4D_TABLE_SIZE > +#define P4D_TABLE_SIZE PAGE_SIZE > +#endif > + > +#ifndef PUD_TABLE_SIZE > +#define PUD_TABLE_SIZE PAGE_SIZE > +#endif > + > +#ifndef PMD_TABLE_SIZE > +#define PMD_TABLE_SIZE PAGE_SIZE > +#endif > + > +#ifndef PTE_TABLE_SIZE > +#define PTE_TABLE_SIZE PAGE_SIZE > +#endif > +void __init __weak pcpu_populate_pte(unsigned long addr) > +{ > + pgd_t *pgd = pgd_offset_k(addr); > + p4d_t *p4d; > + pud_t *pud; > + pmd_t *pmd; > + > + if (pgd_none(*pgd)) { > + p4d_t *new; > + > + new = memblock_alloc_from(P4D_TABLE_SIZE, P4D_TABLE_SIZE, PAGE_SIZE); It's unnecessary to specify a min_addr to memblock_alloc_from() as it won't allocate 0 anyway. So please use memblock_alloc() instead. > + if (!new) > + goto err_alloc; > + pgd_populate(&init_mm, pgd, new); > + } > + > + p4d = p4d_offset(pgd, addr); > + if (p4d_none(*p4d)) { > + pud_t *new; > + > + new = memblock_alloc_from(PUD_TABLE_SIZE, PUD_TABLE_SIZE, PAGE_SIZE); See above. > + if (!new) > + goto err_alloc; > + p4d_populate(&init_mm, p4d, new); > + } > + > + pud = pud_offset(p4d, addr); > + if (pud_none(*pud)) { > + pmd_t *new; > + > + new = memblock_alloc_from(PMD_TABLE_SIZE, PMD_TABLE_SIZE, PAGE_SIZE); See above. > + if (!new) > + goto err_alloc; > + pud_populate(&init_mm, pud, new); > + } > + > + pmd = pmd_offset(pud, addr); > + if (!pmd_present(*pmd)) { > + pte_t *new; > + > + new = memblock_alloc_from(PTE_TABLE_SIZE, PTE_TABLE_SIZE, PAGE_SIZE); See above. > + if (!new) > + goto err_alloc; > + pmd_populate_kernel(&init_mm, pmd, new); > + } > + > + return; > + > +err_alloc: > + panic("%s: Failed to allocate %lu bytes align=%lx from=%lx\n", > + __func__, PAGE_SIZE, PAGE_SIZE, PAGE_SIZE); > +} > + > /** > * pcpu_page_first_chunk - map the first chunk using PAGE_SIZE pages > * @reserved_size: the size of reserved percpu area in bytes > * @cpu_to_nd_fn: callback to convert cpu to it's node, optional > - * @populate_pte_fn: function to populate pte > * > * This is a helper to ease setting up page-remapped first percpu > * chunk and can be called where pcpu_setup_first_chunk() is expected. > @@ -3177,9 +3246,7 @@ int __init pcpu_embed_first_chunk(size_t reserved_size, size_t dyn_size, > * RETURNS: > * 0 on success, -errno on failure. > */ > -int __init pcpu_page_first_chunk(size_t reserved_size, > - pcpu_fc_cpu_to_node_fn_t cpu_to_nd_fn, > - pcpu_fc_populate_pte_fn_t populate_pte_fn) > +int __init pcpu_page_first_chunk(size_t reserved_size, pcpu_fc_cpu_to_node_fn_t cpu_to_nd_fn) > { > static struct vm_struct vm; > struct pcpu_alloc_info *ai; > @@ -3243,7 +3310,7 @@ int __init pcpu_page_first_chunk(size_t reserved_size, > (unsigned long)vm.addr + unit * ai->unit_size; > > for (i = 0; i < unit_pages; i++) > - populate_pte_fn(unit_addr + (i << PAGE_SHIFT)); > + pcpu_populate_pte(unit_addr + (i << PAGE_SHIFT)); > > /* pte already populated, the following shouldn't fail */ > rc = __pcpu_map_pages(unit_addr, &pages[unit * unit_pages], > -- > 2.26.2 > From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 127C8C433EF for ; Mon, 29 Nov 2021 22:51:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=9vJdgGabKfy8Za/Rr6mLzZokKuaQdoDA2CM2JxGXFl0=; b=29pwU2mo/sCLeD a+Tw5dihhCp4XdLSv/DY3T8sf/mAqu/8Rd6DBsPdxDrvo0oVIW2lZ5gJs6UeC1MGHhkgS6DMM2RfV +bFeP+r08GFH93vvNLWctZ/Equ0yM/VuFaBb9ho2Kv3+ngBtNrcu1OBUeHURwVUy6dbcQZ9qRv3IL a6B24UxnJuqAVV7cjOesSUYzMF1+yhHoux+n7rwRHJ7VORmTIceodngiMrvXNzkuy4Wl3GHYrSIJ7 MPqeQyO7SEIzcXDciYinLkbu0DA+Q7bPuW+U229d3tjFS8Y6JITLEvNfV3bpEPRk6AAxeMDyANJNE jlMfJieZRFjxvETejiNA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1mrpT6-002xnc-2D; Mon, 29 Nov 2021 22:49:52 +0000 Received: from mail-qt1-f176.google.com ([209.85.160.176]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1mrpT1-002xmj-Fp; Mon, 29 Nov 2021 22:49:49 +0000 Received: by mail-qt1-f176.google.com with SMTP id j17so18309084qtx.2; Mon, 29 Nov 2021 14:49:47 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=oq7rSvRN8ZmF8826TeWA1nFVHMF74d3/NsU+2BMgrzA=; b=WvhZlYdJ+8VeX2crtY30iWKzRVVrm7dgy/mXtjlo2Mvmz0MW65Lg567J6eK+V3OaBk 3P6vb5H8vs6K2j8C5oN1bqcGeCZe3FLeaPf3XzQnxO7InFQuDMKhimFNzJCVoqPLLJav cPunfSzNQZ6itEUsIPdPDJz9cGBEwR30sKhA6uiGPXWhMUGZqvJ0ww5O/Xb2O7xeiXnd vFMZ1g1OStqiSgQJdiZa4BLby91GlpKxNaRFhFKOWDAC9KjBM4VQ82d/mBavbI/pLj65 l7WqlMW4PJkq3kSTNZm8mD0YRUNKwHZkgKr+PXEW9Gj7TMGWOnNRSwxTF5LhGUt2ONop kqUw== X-Gm-Message-State: AOAM533wB7jRC0aTWW7IFf6i+grzJ19wY6aLR4BPOoWiHlpLcFWwugTs thHpzsAvoOvx1+xUya+fuCc= X-Google-Smtp-Source: ABdhPJxnPvgLuExnZ/ccsT7KqYyls4Id19XqxozgRZPGiHFhvLeIoUIbisNi5P/Pe4vmzv7DzMoClA== X-Received: by 2002:a05:622a:24c:: with SMTP id c12mr38014924qtx.483.1638226186251; Mon, 29 Nov 2021 14:49:46 -0800 (PST) Received: from fedora (pool-173-68-57-129.nycmny.fios.verizon.net. [173.68.57.129]) by smtp.gmail.com with ESMTPSA id r16sm9636543qkp.42.2021.11.29.14.49.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 29 Nov 2021 14:49:45 -0800 (PST) Date: Mon, 29 Nov 2021 17:49:43 -0500 From: Dennis Zhou To: Kefeng Wang Cc: akpm@linux-foundation.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, tj@kernel.org, gregkh@linuxfoundation.org, cl@linux.com, catalin.marinas@arm.com, will@kernel.org, tsbogend@alpha.franken.de, mpe@ellerman.id.au, benh@kernel.crashing.org, paulus@samba.org, paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, davem@davemloft.net, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, linux-arm-kernel@lists.infradead.org, linux-ia64@vger.kernel.org, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, sparclinux@vger.kernel.org, x86@kernel.org Subject: Re: [PATCH RFC 4/4] mm: percpu: Add generic pcpu_populate_pte() function Message-ID: References: <20211121093557.139034-1-wangkefeng.wang@huawei.com> <20211121093557.139034-5-wangkefeng.wang@huawei.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20211121093557.139034-5-wangkefeng.wang@huawei.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20211129_144947_578744_03D2CE4B X-CRM114-Status: GOOD ( 29.33 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Sun, Nov 21, 2021 at 05:35:57PM +0800, Kefeng Wang wrote: > When NEED_PER_CPU_PAGE_FIRST_CHUNK enabled, we need a function to > populate pte, add a generic pcpu populate pte function and switch > to use it. > > Signed-off-by: Kefeng Wang > --- > arch/powerpc/kernel/setup_64.c | 47 +-------------------- > arch/sparc/kernel/smp_64.c | 57 +------------------------ > arch/x86/kernel/setup_percpu.c | 5 +-- > drivers/base/arch_numa.c | 51 +--------------------- > include/linux/percpu.h | 5 +-- > mm/percpu.c | 77 +++++++++++++++++++++++++++++++--- > 6 files changed, 79 insertions(+), 163 deletions(-) > > diff --git a/arch/powerpc/kernel/setup_64.c b/arch/powerpc/kernel/setup_64.c > index 364b1567f822..1a17828af77f 100644 > --- a/arch/powerpc/kernel/setup_64.c > +++ b/arch/powerpc/kernel/setup_64.c > @@ -788,51 +788,6 @@ static int pcpu_cpu_distance(unsigned int from, unsigned int to) > unsigned long __per_cpu_offset[NR_CPUS] __read_mostly; > EXPORT_SYMBOL(__per_cpu_offset); > > -static void __init pcpu_populate_pte(unsigned long addr) > -{ > - pgd_t *pgd = pgd_offset_k(addr); > - p4d_t *p4d; > - pud_t *pud; > - pmd_t *pmd; > - > - p4d = p4d_offset(pgd, addr); > - if (p4d_none(*p4d)) { > - pud_t *new; > - > - new = memblock_alloc(PUD_TABLE_SIZE, PUD_TABLE_SIZE); > - if (!new) > - goto err_alloc; > - p4d_populate(&init_mm, p4d, new); > - } > - > - pud = pud_offset(p4d, addr); > - if (pud_none(*pud)) { > - pmd_t *new; > - > - new = memblock_alloc(PMD_TABLE_SIZE, PMD_TABLE_SIZE); > - if (!new) > - goto err_alloc; > - pud_populate(&init_mm, pud, new); > - } > - > - pmd = pmd_offset(pud, addr); > - if (!pmd_present(*pmd)) { > - pte_t *new; > - > - new = memblock_alloc(PTE_TABLE_SIZE, PTE_TABLE_SIZE); > - if (!new) > - goto err_alloc; > - pmd_populate_kernel(&init_mm, pmd, new); > - } > - > - return; > - > -err_alloc: > - panic("%s: Failed to allocate %lu bytes align=%lx from=%lx\n", > - __func__, PAGE_SIZE, PAGE_SIZE, PAGE_SIZE); > -} > - > - > void __init setup_per_cpu_areas(void) > { > const size_t dyn_size = PERCPU_MODULE_RESERVE + PERCPU_DYNAMIC_RESERVE; > @@ -861,7 +816,7 @@ void __init setup_per_cpu_areas(void) > } > > if (rc < 0) > - rc = pcpu_page_first_chunk(0, pcpu_cpu_to_node, pcpu_populate_pte); > + rc = pcpu_page_first_chunk(0, pcpu_cpu_to_node); > if (rc < 0) > panic("cannot initialize percpu area (err=%d)", rc); > > diff --git a/arch/sparc/kernel/smp_64.c b/arch/sparc/kernel/smp_64.c > index 198dadddb75d..00dffe2d834b 100644 > --- a/arch/sparc/kernel/smp_64.c > +++ b/arch/sparc/kernel/smp_64.c > @@ -1534,59 +1534,6 @@ static int __init pcpu_cpu_distance(unsigned int from, unsigned int to) > return REMOTE_DISTANCE; > } > > -static void __init pcpu_populate_pte(unsigned long addr) > -{ > - pgd_t *pgd = pgd_offset_k(addr); > - p4d_t *p4d; > - pud_t *pud; > - pmd_t *pmd; > - > - if (pgd_none(*pgd)) { > - pud_t *new; > - > - new = memblock_alloc_from(PAGE_SIZE, PAGE_SIZE, PAGE_SIZE); > - if (!new) > - goto err_alloc; > - pgd_populate(&init_mm, pgd, new); > - } > - > - p4d = p4d_offset(pgd, addr); > - if (p4d_none(*p4d)) { > - pud_t *new; > - > - new = memblock_alloc_from(PAGE_SIZE, PAGE_SIZE, PAGE_SIZE); > - if (!new) > - goto err_alloc; > - p4d_populate(&init_mm, p4d, new); > - } > - > - pud = pud_offset(p4d, addr); > - if (pud_none(*pud)) { > - pmd_t *new; > - > - new = memblock_alloc_from(PAGE_SIZE, PAGE_SIZE, PAGE_SIZE); > - if (!new) > - goto err_alloc; > - pud_populate(&init_mm, pud, new); > - } > - > - pmd = pmd_offset(pud, addr); > - if (!pmd_present(*pmd)) { > - pte_t *new; > - > - new = memblock_alloc_from(PAGE_SIZE, PAGE_SIZE, PAGE_SIZE); > - if (!new) > - goto err_alloc; > - pmd_populate_kernel(&init_mm, pmd, new); > - } > - > - return; > - > -err_alloc: > - panic("%s: Failed to allocate %lu bytes align=%lx from=%lx\n", > - __func__, PAGE_SIZE, PAGE_SIZE, PAGE_SIZE); > -} > - > void __init setup_per_cpu_areas(void) > { > unsigned long delta; > @@ -1604,9 +1551,7 @@ void __init setup_per_cpu_areas(void) > pcpu_fc_names[pcpu_chosen_fc], rc); > } > if (rc < 0) > - rc = pcpu_page_first_chunk(PERCPU_MODULE_RESERVE, > - cpu_to_node, > - pcpu_populate_pte); > + rc = pcpu_page_first_chunk(PERCPU_MODULE_RESERVE, cpu_to_node); > if (rc < 0) > panic("cannot initialize percpu area (err=%d)", rc); > > diff --git a/arch/x86/kernel/setup_percpu.c b/arch/x86/kernel/setup_percpu.c > index cd672bd46241..4eadbe45078e 100644 > --- a/arch/x86/kernel/setup_percpu.c > +++ b/arch/x86/kernel/setup_percpu.c > @@ -101,7 +101,7 @@ static int __init pcpu_cpu_to_node(int cpu) > return IS_ENABLED(CONFIG_NUMA) ? early_cpu_to_node(cpu) : NUMA_NO_NODE; > } > > -static void __init pcpup_populate_pte(unsigned long addr) > +void __init pcpu_populate_pte(unsigned long addr) > { > populate_extra_pte(addr); > } > @@ -163,8 +163,7 @@ void __init setup_per_cpu_areas(void) > } > if (rc < 0) > rc = pcpu_page_first_chunk(PERCPU_FIRST_CHUNK_RESERVE, > - pcpu_cpu_to_node, > - pcpup_populate_pte); > + pcpu_cpu_to_node); x86 has it's own implementation that differs for 32 bit. I'm not confident this is correct to drop in as a replacement for x86, so I'd prefer to keep populate_pte_fn() around. > if (rc < 0) > panic("cannot initialize percpu area (err=%d)", rc); > > diff --git a/drivers/base/arch_numa.c b/drivers/base/arch_numa.c > index 23a10cc36165..eaa31e567d1e 100644 > --- a/drivers/base/arch_numa.c > +++ b/drivers/base/arch_numa.c > @@ -14,7 +14,6 @@ > #include > > #include > -#include > > struct pglist_data *node_data[MAX_NUMNODES] __read_mostly; > EXPORT_SYMBOL(node_data); > @@ -155,52 +154,6 @@ static int __init pcpu_cpu_distance(unsigned int from, unsigned int to) > return node_distance(early_cpu_to_node(from), early_cpu_to_node(to)); > } > > -#ifdef CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK > -static void __init pcpu_populate_pte(unsigned long addr) > -{ > - pgd_t *pgd = pgd_offset_k(addr); > - p4d_t *p4d; > - pud_t *pud; > - pmd_t *pmd; > - > - p4d = p4d_offset(pgd, addr); > - if (p4d_none(*p4d)) { > - pud_t *new; > - > - new = memblock_alloc(PAGE_SIZE, PAGE_SIZE); > - if (!new) > - goto err_alloc; > - p4d_populate(&init_mm, p4d, new); > - } > - > - pud = pud_offset(p4d, addr); > - if (pud_none(*pud)) { > - pmd_t *new; > - > - new = memblock_alloc(PAGE_SIZE, PAGE_SIZE); > - if (!new) > - goto err_alloc; > - pud_populate(&init_mm, pud, new); > - } > - > - pmd = pmd_offset(pud, addr); > - if (!pmd_present(*pmd)) { > - pte_t *new; > - > - new = memblock_alloc(PAGE_SIZE, PAGE_SIZE); > - if (!new) > - goto err_alloc; > - pmd_populate_kernel(&init_mm, pmd, new); > - } > - > - return; > - > -err_alloc: > - panic("%s: Failed to allocate %lu bytes align=%lx from=%lx\n", > - __func__, PAGE_SIZE, PAGE_SIZE, PAGE_SIZE); > -} > -#endif > - > void __init setup_per_cpu_areas(void) > { > unsigned long delta; > @@ -225,9 +178,7 @@ void __init setup_per_cpu_areas(void) > > #ifdef CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK > if (rc < 0) > - rc = pcpu_page_first_chunk(PERCPU_MODULE_RESERVE, > - early_cpu_to_node, > - pcpu_populate_pte); > + rc = pcpu_page_first_chunk(PERCPU_MODULE_RESERVE, early_cpu_to_node); > #endif > if (rc < 0) > panic("Failed to initialize percpu areas (err=%d).", rc); > diff --git a/include/linux/percpu.h b/include/linux/percpu.h > index d73c97ef4ff4..f1ec5ad1351c 100644 > --- a/include/linux/percpu.h > +++ b/include/linux/percpu.h > @@ -95,7 +95,6 @@ extern const char * const pcpu_fc_names[PCPU_FC_NR]; > extern enum pcpu_fc pcpu_chosen_fc; > > typedef int (pcpu_fc_cpu_to_node_fn_t)(int cpu); > -typedef void (*pcpu_fc_populate_pte_fn_t)(unsigned long addr); > typedef int (pcpu_fc_cpu_distance_fn_t)(unsigned int from, unsigned int to); > > extern struct pcpu_alloc_info * __init pcpu_alloc_alloc_info(int nr_groups, > @@ -113,9 +112,9 @@ extern int __init pcpu_embed_first_chunk(size_t reserved_size, size_t dyn_size, > #endif > > #ifdef CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK > +void __init pcpu_populate_pte(unsigned long addr); > extern int __init pcpu_page_first_chunk(size_t reserved_size, > - pcpu_fc_cpu_to_node_fn_t cpu_to_nd_fn, > - pcpu_fc_populate_pte_fn_t populate_pte_fn); > + pcpu_fc_cpu_to_node_fn_t cpu_to_nd_fn); > #endif > > extern void __percpu *__alloc_reserved_percpu(size_t size, size_t align) __alloc_size(1); > diff --git a/mm/percpu.c b/mm/percpu.c > index efaa1cbaf73d..d907daed04eb 100644 > --- a/mm/percpu.c > +++ b/mm/percpu.c > @@ -3162,11 +3162,80 @@ int __init pcpu_embed_first_chunk(size_t reserved_size, size_t dyn_size, > #endif /* BUILD_EMBED_FIRST_CHUNK */ > > #ifdef BUILD_PAGE_FIRST_CHUNK > +#include > + > +#ifndef P4D_TABLE_SIZE > +#define P4D_TABLE_SIZE PAGE_SIZE > +#endif > + > +#ifndef PUD_TABLE_SIZE > +#define PUD_TABLE_SIZE PAGE_SIZE > +#endif > + > +#ifndef PMD_TABLE_SIZE > +#define PMD_TABLE_SIZE PAGE_SIZE > +#endif > + > +#ifndef PTE_TABLE_SIZE > +#define PTE_TABLE_SIZE PAGE_SIZE > +#endif > +void __init __weak pcpu_populate_pte(unsigned long addr) > +{ > + pgd_t *pgd = pgd_offset_k(addr); > + p4d_t *p4d; > + pud_t *pud; > + pmd_t *pmd; > + > + if (pgd_none(*pgd)) { > + p4d_t *new; > + > + new = memblock_alloc_from(P4D_TABLE_SIZE, P4D_TABLE_SIZE, PAGE_SIZE); It's unnecessary to specify a min_addr to memblock_alloc_from() as it won't allocate 0 anyway. So please use memblock_alloc() instead. > + if (!new) > + goto err_alloc; > + pgd_populate(&init_mm, pgd, new); > + } > + > + p4d = p4d_offset(pgd, addr); > + if (p4d_none(*p4d)) { > + pud_t *new; > + > + new = memblock_alloc_from(PUD_TABLE_SIZE, PUD_TABLE_SIZE, PAGE_SIZE); See above. > + if (!new) > + goto err_alloc; > + p4d_populate(&init_mm, p4d, new); > + } > + > + pud = pud_offset(p4d, addr); > + if (pud_none(*pud)) { > + pmd_t *new; > + > + new = memblock_alloc_from(PMD_TABLE_SIZE, PMD_TABLE_SIZE, PAGE_SIZE); See above. > + if (!new) > + goto err_alloc; > + pud_populate(&init_mm, pud, new); > + } > + > + pmd = pmd_offset(pud, addr); > + if (!pmd_present(*pmd)) { > + pte_t *new; > + > + new = memblock_alloc_from(PTE_TABLE_SIZE, PTE_TABLE_SIZE, PAGE_SIZE); See above. > + if (!new) > + goto err_alloc; > + pmd_populate_kernel(&init_mm, pmd, new); > + } > + > + return; > + > +err_alloc: > + panic("%s: Failed to allocate %lu bytes align=%lx from=%lx\n", > + __func__, PAGE_SIZE, PAGE_SIZE, PAGE_SIZE); > +} > + > /** > * pcpu_page_first_chunk - map the first chunk using PAGE_SIZE pages > * @reserved_size: the size of reserved percpu area in bytes > * @cpu_to_nd_fn: callback to convert cpu to it's node, optional > - * @populate_pte_fn: function to populate pte > * > * This is a helper to ease setting up page-remapped first percpu > * chunk and can be called where pcpu_setup_first_chunk() is expected. > @@ -3177,9 +3246,7 @@ int __init pcpu_embed_first_chunk(size_t reserved_size, size_t dyn_size, > * RETURNS: > * 0 on success, -errno on failure. > */ > -int __init pcpu_page_first_chunk(size_t reserved_size, > - pcpu_fc_cpu_to_node_fn_t cpu_to_nd_fn, > - pcpu_fc_populate_pte_fn_t populate_pte_fn) > +int __init pcpu_page_first_chunk(size_t reserved_size, pcpu_fc_cpu_to_node_fn_t cpu_to_nd_fn) > { > static struct vm_struct vm; > struct pcpu_alloc_info *ai; > @@ -3243,7 +3310,7 @@ int __init pcpu_page_first_chunk(size_t reserved_size, > (unsigned long)vm.addr + unit * ai->unit_size; > > for (i = 0; i < unit_pages; i++) > - populate_pte_fn(unit_addr + (i << PAGE_SHIFT)); > + pcpu_populate_pte(unit_addr + (i << PAGE_SHIFT)); > > /* pte already populated, the following shouldn't fail */ > rc = __pcpu_map_pages(unit_addr, &pages[unit * unit_pages], > -- > 2.26.2 > _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dennis Zhou Date: Mon, 29 Nov 2021 22:49:43 +0000 Subject: Re: [PATCH RFC 4/4] mm: percpu: Add generic pcpu_populate_pte() function Message-Id: List-Id: References: <20211121093557.139034-1-wangkefeng.wang@huawei.com> <20211121093557.139034-5-wangkefeng.wang@huawei.com> In-Reply-To: <20211121093557.139034-5-wangkefeng.wang@huawei.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit To: Kefeng Wang Cc: akpm@linux-foundation.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, tj@kernel.org, gregkh@linuxfoundation.org, cl@linux.com, catalin.marinas@arm.com, will@kernel.org, tsbogend@alpha.franken.de, mpe@ellerman.id.au, benh@kernel.crashing.org, paulus@samba.org, paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, davem@davemloft.net, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, linux-arm-kernel@lists.infradead.org, linux-ia64@vger.kernel.org, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, sparclinux@vger.kernel.org, x86@kernel.org On Sun, Nov 21, 2021 at 05:35:57PM +0800, Kefeng Wang wrote: > When NEED_PER_CPU_PAGE_FIRST_CHUNK enabled, we need a function to > populate pte, add a generic pcpu populate pte function and switch > to use it. > > Signed-off-by: Kefeng Wang > --- > arch/powerpc/kernel/setup_64.c | 47 +-------------------- > arch/sparc/kernel/smp_64.c | 57 +------------------------ > arch/x86/kernel/setup_percpu.c | 5 +-- > drivers/base/arch_numa.c | 51 +--------------------- > include/linux/percpu.h | 5 +-- > mm/percpu.c | 77 +++++++++++++++++++++++++++++++--- > 6 files changed, 79 insertions(+), 163 deletions(-) > > diff --git a/arch/powerpc/kernel/setup_64.c b/arch/powerpc/kernel/setup_64.c > index 364b1567f822..1a17828af77f 100644 > --- a/arch/powerpc/kernel/setup_64.c > +++ b/arch/powerpc/kernel/setup_64.c > @@ -788,51 +788,6 @@ static int pcpu_cpu_distance(unsigned int from, unsigned int to) > unsigned long __per_cpu_offset[NR_CPUS] __read_mostly; > EXPORT_SYMBOL(__per_cpu_offset); > > -static void __init pcpu_populate_pte(unsigned long addr) > -{ > - pgd_t *pgd = pgd_offset_k(addr); > - p4d_t *p4d; > - pud_t *pud; > - pmd_t *pmd; > - > - p4d = p4d_offset(pgd, addr); > - if (p4d_none(*p4d)) { > - pud_t *new; > - > - new = memblock_alloc(PUD_TABLE_SIZE, PUD_TABLE_SIZE); > - if (!new) > - goto err_alloc; > - p4d_populate(&init_mm, p4d, new); > - } > - > - pud = pud_offset(p4d, addr); > - if (pud_none(*pud)) { > - pmd_t *new; > - > - new = memblock_alloc(PMD_TABLE_SIZE, PMD_TABLE_SIZE); > - if (!new) > - goto err_alloc; > - pud_populate(&init_mm, pud, new); > - } > - > - pmd = pmd_offset(pud, addr); > - if (!pmd_present(*pmd)) { > - pte_t *new; > - > - new = memblock_alloc(PTE_TABLE_SIZE, PTE_TABLE_SIZE); > - if (!new) > - goto err_alloc; > - pmd_populate_kernel(&init_mm, pmd, new); > - } > - > - return; > - > -err_alloc: > - panic("%s: Failed to allocate %lu bytes align=%lx from=%lx\n", > - __func__, PAGE_SIZE, PAGE_SIZE, PAGE_SIZE); > -} > - > - > void __init setup_per_cpu_areas(void) > { > const size_t dyn_size = PERCPU_MODULE_RESERVE + PERCPU_DYNAMIC_RESERVE; > @@ -861,7 +816,7 @@ void __init setup_per_cpu_areas(void) > } > > if (rc < 0) > - rc = pcpu_page_first_chunk(0, pcpu_cpu_to_node, pcpu_populate_pte); > + rc = pcpu_page_first_chunk(0, pcpu_cpu_to_node); > if (rc < 0) > panic("cannot initialize percpu area (err=%d)", rc); > > diff --git a/arch/sparc/kernel/smp_64.c b/arch/sparc/kernel/smp_64.c > index 198dadddb75d..00dffe2d834b 100644 > --- a/arch/sparc/kernel/smp_64.c > +++ b/arch/sparc/kernel/smp_64.c > @@ -1534,59 +1534,6 @@ static int __init pcpu_cpu_distance(unsigned int from, unsigned int to) > return REMOTE_DISTANCE; > } > > -static void __init pcpu_populate_pte(unsigned long addr) > -{ > - pgd_t *pgd = pgd_offset_k(addr); > - p4d_t *p4d; > - pud_t *pud; > - pmd_t *pmd; > - > - if (pgd_none(*pgd)) { > - pud_t *new; > - > - new = memblock_alloc_from(PAGE_SIZE, PAGE_SIZE, PAGE_SIZE); > - if (!new) > - goto err_alloc; > - pgd_populate(&init_mm, pgd, new); > - } > - > - p4d = p4d_offset(pgd, addr); > - if (p4d_none(*p4d)) { > - pud_t *new; > - > - new = memblock_alloc_from(PAGE_SIZE, PAGE_SIZE, PAGE_SIZE); > - if (!new) > - goto err_alloc; > - p4d_populate(&init_mm, p4d, new); > - } > - > - pud = pud_offset(p4d, addr); > - if (pud_none(*pud)) { > - pmd_t *new; > - > - new = memblock_alloc_from(PAGE_SIZE, PAGE_SIZE, PAGE_SIZE); > - if (!new) > - goto err_alloc; > - pud_populate(&init_mm, pud, new); > - } > - > - pmd = pmd_offset(pud, addr); > - if (!pmd_present(*pmd)) { > - pte_t *new; > - > - new = memblock_alloc_from(PAGE_SIZE, PAGE_SIZE, PAGE_SIZE); > - if (!new) > - goto err_alloc; > - pmd_populate_kernel(&init_mm, pmd, new); > - } > - > - return; > - > -err_alloc: > - panic("%s: Failed to allocate %lu bytes align=%lx from=%lx\n", > - __func__, PAGE_SIZE, PAGE_SIZE, PAGE_SIZE); > -} > - > void __init setup_per_cpu_areas(void) > { > unsigned long delta; > @@ -1604,9 +1551,7 @@ void __init setup_per_cpu_areas(void) > pcpu_fc_names[pcpu_chosen_fc], rc); > } > if (rc < 0) > - rc = pcpu_page_first_chunk(PERCPU_MODULE_RESERVE, > - cpu_to_node, > - pcpu_populate_pte); > + rc = pcpu_page_first_chunk(PERCPU_MODULE_RESERVE, cpu_to_node); > if (rc < 0) > panic("cannot initialize percpu area (err=%d)", rc); > > diff --git a/arch/x86/kernel/setup_percpu.c b/arch/x86/kernel/setup_percpu.c > index cd672bd46241..4eadbe45078e 100644 > --- a/arch/x86/kernel/setup_percpu.c > +++ b/arch/x86/kernel/setup_percpu.c > @@ -101,7 +101,7 @@ static int __init pcpu_cpu_to_node(int cpu) > return IS_ENABLED(CONFIG_NUMA) ? early_cpu_to_node(cpu) : NUMA_NO_NODE; > } > > -static void __init pcpup_populate_pte(unsigned long addr) > +void __init pcpu_populate_pte(unsigned long addr) > { > populate_extra_pte(addr); > } > @@ -163,8 +163,7 @@ void __init setup_per_cpu_areas(void) > } > if (rc < 0) > rc = pcpu_page_first_chunk(PERCPU_FIRST_CHUNK_RESERVE, > - pcpu_cpu_to_node, > - pcpup_populate_pte); > + pcpu_cpu_to_node); x86 has it's own implementation that differs for 32 bit. I'm not confident this is correct to drop in as a replacement for x86, so I'd prefer to keep populate_pte_fn() around. > if (rc < 0) > panic("cannot initialize percpu area (err=%d)", rc); > > diff --git a/drivers/base/arch_numa.c b/drivers/base/arch_numa.c > index 23a10cc36165..eaa31e567d1e 100644 > --- a/drivers/base/arch_numa.c > +++ b/drivers/base/arch_numa.c > @@ -14,7 +14,6 @@ > #include > > #include > -#include > > struct pglist_data *node_data[MAX_NUMNODES] __read_mostly; > EXPORT_SYMBOL(node_data); > @@ -155,52 +154,6 @@ static int __init pcpu_cpu_distance(unsigned int from, unsigned int to) > return node_distance(early_cpu_to_node(from), early_cpu_to_node(to)); > } > > -#ifdef CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK > -static void __init pcpu_populate_pte(unsigned long addr) > -{ > - pgd_t *pgd = pgd_offset_k(addr); > - p4d_t *p4d; > - pud_t *pud; > - pmd_t *pmd; > - > - p4d = p4d_offset(pgd, addr); > - if (p4d_none(*p4d)) { > - pud_t *new; > - > - new = memblock_alloc(PAGE_SIZE, PAGE_SIZE); > - if (!new) > - goto err_alloc; > - p4d_populate(&init_mm, p4d, new); > - } > - > - pud = pud_offset(p4d, addr); > - if (pud_none(*pud)) { > - pmd_t *new; > - > - new = memblock_alloc(PAGE_SIZE, PAGE_SIZE); > - if (!new) > - goto err_alloc; > - pud_populate(&init_mm, pud, new); > - } > - > - pmd = pmd_offset(pud, addr); > - if (!pmd_present(*pmd)) { > - pte_t *new; > - > - new = memblock_alloc(PAGE_SIZE, PAGE_SIZE); > - if (!new) > - goto err_alloc; > - pmd_populate_kernel(&init_mm, pmd, new); > - } > - > - return; > - > -err_alloc: > - panic("%s: Failed to allocate %lu bytes align=%lx from=%lx\n", > - __func__, PAGE_SIZE, PAGE_SIZE, PAGE_SIZE); > -} > -#endif > - > void __init setup_per_cpu_areas(void) > { > unsigned long delta; > @@ -225,9 +178,7 @@ void __init setup_per_cpu_areas(void) > > #ifdef CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK > if (rc < 0) > - rc = pcpu_page_first_chunk(PERCPU_MODULE_RESERVE, > - early_cpu_to_node, > - pcpu_populate_pte); > + rc = pcpu_page_first_chunk(PERCPU_MODULE_RESERVE, early_cpu_to_node); > #endif > if (rc < 0) > panic("Failed to initialize percpu areas (err=%d).", rc); > diff --git a/include/linux/percpu.h b/include/linux/percpu.h > index d73c97ef4ff4..f1ec5ad1351c 100644 > --- a/include/linux/percpu.h > +++ b/include/linux/percpu.h > @@ -95,7 +95,6 @@ extern const char * const pcpu_fc_names[PCPU_FC_NR]; > extern enum pcpu_fc pcpu_chosen_fc; > > typedef int (pcpu_fc_cpu_to_node_fn_t)(int cpu); > -typedef void (*pcpu_fc_populate_pte_fn_t)(unsigned long addr); > typedef int (pcpu_fc_cpu_distance_fn_t)(unsigned int from, unsigned int to); > > extern struct pcpu_alloc_info * __init pcpu_alloc_alloc_info(int nr_groups, > @@ -113,9 +112,9 @@ extern int __init pcpu_embed_first_chunk(size_t reserved_size, size_t dyn_size, > #endif > > #ifdef CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK > +void __init pcpu_populate_pte(unsigned long addr); > extern int __init pcpu_page_first_chunk(size_t reserved_size, > - pcpu_fc_cpu_to_node_fn_t cpu_to_nd_fn, > - pcpu_fc_populate_pte_fn_t populate_pte_fn); > + pcpu_fc_cpu_to_node_fn_t cpu_to_nd_fn); > #endif > > extern void __percpu *__alloc_reserved_percpu(size_t size, size_t align) __alloc_size(1); > diff --git a/mm/percpu.c b/mm/percpu.c > index efaa1cbaf73d..d907daed04eb 100644 > --- a/mm/percpu.c > +++ b/mm/percpu.c > @@ -3162,11 +3162,80 @@ int __init pcpu_embed_first_chunk(size_t reserved_size, size_t dyn_size, > #endif /* BUILD_EMBED_FIRST_CHUNK */ > > #ifdef BUILD_PAGE_FIRST_CHUNK > +#include > + > +#ifndef P4D_TABLE_SIZE > +#define P4D_TABLE_SIZE PAGE_SIZE > +#endif > + > +#ifndef PUD_TABLE_SIZE > +#define PUD_TABLE_SIZE PAGE_SIZE > +#endif > + > +#ifndef PMD_TABLE_SIZE > +#define PMD_TABLE_SIZE PAGE_SIZE > +#endif > + > +#ifndef PTE_TABLE_SIZE > +#define PTE_TABLE_SIZE PAGE_SIZE > +#endif > +void __init __weak pcpu_populate_pte(unsigned long addr) > +{ > + pgd_t *pgd = pgd_offset_k(addr); > + p4d_t *p4d; > + pud_t *pud; > + pmd_t *pmd; > + > + if (pgd_none(*pgd)) { > + p4d_t *new; > + > + new = memblock_alloc_from(P4D_TABLE_SIZE, P4D_TABLE_SIZE, PAGE_SIZE); It's unnecessary to specify a min_addr to memblock_alloc_from() as it won't allocate 0 anyway. So please use memblock_alloc() instead. > + if (!new) > + goto err_alloc; > + pgd_populate(&init_mm, pgd, new); > + } > + > + p4d = p4d_offset(pgd, addr); > + if (p4d_none(*p4d)) { > + pud_t *new; > + > + new = memblock_alloc_from(PUD_TABLE_SIZE, PUD_TABLE_SIZE, PAGE_SIZE); See above. > + if (!new) > + goto err_alloc; > + p4d_populate(&init_mm, p4d, new); > + } > + > + pud = pud_offset(p4d, addr); > + if (pud_none(*pud)) { > + pmd_t *new; > + > + new = memblock_alloc_from(PMD_TABLE_SIZE, PMD_TABLE_SIZE, PAGE_SIZE); See above. > + if (!new) > + goto err_alloc; > + pud_populate(&init_mm, pud, new); > + } > + > + pmd = pmd_offset(pud, addr); > + if (!pmd_present(*pmd)) { > + pte_t *new; > + > + new = memblock_alloc_from(PTE_TABLE_SIZE, PTE_TABLE_SIZE, PAGE_SIZE); See above. > + if (!new) > + goto err_alloc; > + pmd_populate_kernel(&init_mm, pmd, new); > + } > + > + return; > + > +err_alloc: > + panic("%s: Failed to allocate %lu bytes align=%lx from=%lx\n", > + __func__, PAGE_SIZE, PAGE_SIZE, PAGE_SIZE); > +} > + > /** > * pcpu_page_first_chunk - map the first chunk using PAGE_SIZE pages > * @reserved_size: the size of reserved percpu area in bytes > * @cpu_to_nd_fn: callback to convert cpu to it's node, optional > - * @populate_pte_fn: function to populate pte > * > * This is a helper to ease setting up page-remapped first percpu > * chunk and can be called where pcpu_setup_first_chunk() is expected. > @@ -3177,9 +3246,7 @@ int __init pcpu_embed_first_chunk(size_t reserved_size, size_t dyn_size, > * RETURNS: > * 0 on success, -errno on failure. > */ > -int __init pcpu_page_first_chunk(size_t reserved_size, > - pcpu_fc_cpu_to_node_fn_t cpu_to_nd_fn, > - pcpu_fc_populate_pte_fn_t populate_pte_fn) > +int __init pcpu_page_first_chunk(size_t reserved_size, pcpu_fc_cpu_to_node_fn_t cpu_to_nd_fn) > { > static struct vm_struct vm; > struct pcpu_alloc_info *ai; > @@ -3243,7 +3310,7 @@ int __init pcpu_page_first_chunk(size_t reserved_size, > (unsigned long)vm.addr + unit * ai->unit_size; > > for (i = 0; i < unit_pages; i++) > - populate_pte_fn(unit_addr + (i << PAGE_SHIFT)); > + pcpu_populate_pte(unit_addr + (i << PAGE_SHIFT)); > > /* pte already populated, the following shouldn't fail */ > rc = __pcpu_map_pages(unit_addr, &pages[unit * unit_pages], > -- > 2.26.2 >