From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 06C03C433E0 for ; Wed, 17 Mar 2021 06:25:53 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 7F37664F8C for ; Wed, 17 Mar 2021 06:25:52 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7F37664F8C Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 0EF836B0083; Wed, 17 Mar 2021 02:25:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0C6D66B0085; Wed, 17 Mar 2021 02:25:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E5B6C8D0001; Wed, 17 Mar 2021 02:25:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0115.hostedemail.com [216.40.44.115]) by kanga.kvack.org (Postfix) with ESMTP id C2B356B0083 for ; Wed, 17 Mar 2021 02:25:51 -0400 (EDT) Received: from smtpin20.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 80D858249980 for ; Wed, 17 Mar 2021 06:25:51 +0000 (UTC) X-FDA: 77928380502.20.D871587 Received: from mail-pj1-f54.google.com (mail-pj1-f54.google.com [209.85.216.54]) by imf13.hostedemail.com (Postfix) with ESMTP id D1E2EE0011CE for ; Wed, 17 Mar 2021 06:25:50 +0000 (UTC) Received: by mail-pj1-f54.google.com with SMTP id x7-20020a17090a2b07b02900c0ea793940so2654999pjc.2 for ; Tue, 16 Mar 2021 23:25:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=1uU5bcxzx01P8dHEfjYViSMSkdrX5rHnhOTXMkO32oE=; b=Dj87U9gZev1Rr5H6OFRHf6eB+X0ny+W2T+OPJf0i/fOrJmxO3O48S9Vx0Mbw6Gq6NX gfPOFq5B7JTJrzXg18w7QBdVeT+YXh3gNrSw25zVXb6ta4pysEGsbSEriZc/k2fEtFh9 t7E5K3yCYvFBW0G1nF3YU7MsJ0thlF3SKnEhydrd/GBiX3zAOo0o1vJDWgERNqSjd3zY EjfQeI25/XpweODnWb8UdiQdM5IwydmQ74Scg0xDwuUXCyLXdQhDWLBrpLzOGBvk2POz FceEazvOBdPhVPkXW+xYqQt1Lglvvv2EWea6GEhmIbSfe7RurmDdjhAEh4LZyN8LtnqI 6UmQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=1uU5bcxzx01P8dHEfjYViSMSkdrX5rHnhOTXMkO32oE=; b=bQ0smCTh9lQqgmoPoAzO9rah7/df+7hekJb9RA0ptOYVcT7Xvelx3nu8RFnS11zov3 OLXYUil4HoyWIpWqyNYkHGPBt23oHybEYPLwig++fqzVTL3y1ldSHi5Nl/xqJ4e7lSXF 5OUvHO7tS9gjSfeIFcdI8QgX4QzUDsvehPOVVLaOuL1EcFf65FZ9MaHJl76drmCM2Wbh QasmML3ecADItOGGoAOkfRqVtPrb6/R2LgU8dt330khCpjnlRRye/Mr98ZWV79y828KM lgZ2u8ntN3uChLf8i3GGF9B+xq9uN5FLgzJd+Ja5Vpb++jehVuAbVDImxnxKRVcS8OrE TlrA== X-Gm-Message-State: AOAM5330NZcRW13abjZmnLMj/b5eeUYDOay2GGirTwL3KqeMrtVGjMnN gLg0SEXBBXMZG1711UtTtuKAT1AziXY= X-Google-Smtp-Source: ABdhPJxulUfRkKo4BI6n1Nb1qSvY1AXLNKJ0oLKh3JtExGE7ySc2ERfh6WLBwjpZzO7oJObIeRqEDg== X-Received: by 2002:a17:902:7888:b029:e6:b94d:c72 with SMTP id q8-20020a1709027888b02900e6b94d0c72mr3106960pll.8.1615962349361; Tue, 16 Mar 2021 23:25:49 -0700 (PDT) Received: from bobo.ozlabs.ibm.com (58-6-239-121.tpgi.com.au. [58.6.239.121]) by smtp.gmail.com with ESMTPSA id s19sm17959620pfh.168.2021.03.16.23.25.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Mar 2021 23:25:48 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org, Andrew Morton Cc: Nicholas Piggin , linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, Jonathan Cameron , Christoph Hellwig , Christophe Leroy , Rick Edgecombe , Ding Tianhong Subject: [PATCH v13 13/14] mm/vmalloc: Hugepage vmalloc mappings Date: Wed, 17 Mar 2021 16:24:01 +1000 Message-Id: <20210317062402.533919-14-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20210317062402.533919-1-npiggin@gmail.com> References: <20210317062402.533919-1-npiggin@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: D1E2EE0011CE X-Stat-Signature: 6aawy6umos7hffqi54cejfcx4hz58mzi Received-SPF: none (gmail.com>: No applicable sender policy available) receiver=imf13; identity=mailfrom; envelope-from=""; helo=mail-pj1-f54.google.com; client-ip=209.85.216.54 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1615962350-262181 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Support huge page vmalloc mappings. Config option HAVE_ARCH_HUGE_VMALLOC enables support on architectures that define HAVE_ARCH_HUGE_VMAP and supports PMD sized vmap mappings. vmalloc will attempt to allocate PMD-sized pages if allocating PMD size or larger, and fall back to small pages if that was unsuccessful. Architectures must ensure that any arch specific vmalloc allocations that require PAGE_SIZE mappings (e.g., module allocations vs strict module rwx) use the VM_NOHUGE flag to inhibit larger mappings. This can result in more internal fragmentation and memory overhead for a given allocation, an option nohugevmalloc is added to disable at boot. Signed-off-by: Nicholas Piggin --- arch/Kconfig | 11 ++ include/linux/vmalloc.h | 21 ++++ mm/page_alloc.c | 5 +- mm/vmalloc.c | 216 +++++++++++++++++++++++++++++++--------- 4 files changed, 206 insertions(+), 47 deletions(-) diff --git a/arch/Kconfig b/arch/Kconfig index ecfd3520b676..b347102f2984 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -785,6 +785,17 @@ config HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD config HAVE_ARCH_HUGE_VMAP bool =20 +# +# Archs that select this would be capable of PMD-sized vmaps (i.e., +# arch_vmap_pmd_supported() returns true), and they must make no assump= tions +# that vmalloc memory is mapped with PAGE_SIZE ptes. The VM_NO_HUGE_VMA= P flag +# can be used to prohibit arch-specific allocations from using hugepage= s to +# help with this (e.g., modules may require it). +# +config HAVE_ARCH_HUGE_VMALLOC + depends on HAVE_ARCH_HUGE_VMAP + bool + config ARCH_WANT_HUGE_PMD_SHARE bool =20 diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index 1f6844e2670a..8341964e6eb5 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -26,6 +26,7 @@ struct notifier_block; /* in notifier.h */ #define VM_KASAN 0x00000080 /* has allocated kasan shadow memory *= / #define VM_FLUSH_RESET_PERMS 0x00000100 /* reset direct map and flush TL= B on unmap, can't be freed in atomic context */ #define VM_MAP_PUT_PAGES 0x00000200 /* put pages and free array in vfree= */ +#define VM_NO_HUGE_VMAP 0x00000400 /* force PAGE_SIZE pte mapping */ =20 /* * VM_KASAN is used slighly differently depending on CONFIG_KASAN_VMALLO= C. @@ -54,6 +55,9 @@ struct vm_struct { unsigned long size; unsigned long flags; struct page **pages; +#ifdef CONFIG_HAVE_ARCH_HUGE_VMALLOC + unsigned int page_order; +#endif unsigned int nr_pages; phys_addr_t phys_addr; const void *caller; @@ -188,6 +192,22 @@ void free_vm_area(struct vm_struct *area); extern struct vm_struct *remove_vm_area(const void *addr); extern struct vm_struct *find_vm_area(const void *addr); =20 +static inline bool is_vm_area_hugepages(const void *addr) +{ + /* + * This may not 100% tell if the area is mapped with > PAGE_SIZE + * page table entries, if for some reason the architecture indicates + * larger sizes are available but decides not to use them, nothing + * prevents that. This only indicates the size of the physical page + * allocated in the vmalloc layer. + */ +#ifdef CONFIG_HAVE_ARCH_HUGE_VMALLOC + return find_vm_area(addr)->page_order > 0; +#else + return false; +#endif +} + #ifdef CONFIG_MMU int vmap_range(unsigned long addr, unsigned long end, phys_addr_t phys_addr, pgprot_t prot, @@ -205,6 +225,7 @@ static inline void set_vm_flush_reset_perms(void *add= r) if (vm) vm->flags |=3D VM_FLUSH_RESET_PERMS; } + #else static inline int map_kernel_range_noflush(unsigned long start, unsigned long size, diff --git a/mm/page_alloc.c b/mm/page_alloc.c index cfc72873961d..2e2042f39b8b 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -72,6 +72,7 @@ #include #include #include +#include =20 #include #include @@ -8222,6 +8223,7 @@ void *__init alloc_large_system_hash(const char *ta= blename, void *table =3D NULL; gfp_t gfp_flags; bool virt; + bool huge; =20 /* allow the kernel cmdline to have a say */ if (!numentries) { @@ -8289,6 +8291,7 @@ void *__init alloc_large_system_hash(const char *ta= blename, } else if (get_order(size) >=3D MAX_ORDER || hashdist) { table =3D __vmalloc(size, gfp_flags); virt =3D true; + huge =3D is_vm_area_hugepages(table); } else { /* * If bucketsize is not a power-of-two, we may free @@ -8305,7 +8308,7 @@ void *__init alloc_large_system_hash(const char *ta= blename, =20 pr_info("%s hash table entries: %ld (order: %d, %lu bytes, %s)\n", tablename, 1UL << log2qty, ilog2(size) - PAGE_SHIFT, size, - virt ? "vmalloc" : "linear"); + virt ? (huge ? "vmalloc hugepage" : "vmalloc") : "linear"); =20 if (_hash_shift) *_hash_shift =3D log2qty; diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 9455dba58b0e..87e3bec51ddb 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -42,6 +42,19 @@ #include "internal.h" #include "pgalloc-track.h" =20 +#ifdef CONFIG_HAVE_ARCH_HUGE_VMALLOC +static bool __ro_after_init vmap_allow_huge =3D true; + +static int __init set_nohugevmalloc(char *str) +{ + vmap_allow_huge =3D false; + return 0; +} +early_param("nohugevmalloc", set_nohugevmalloc); +#else /* CONFIG_HAVE_ARCH_HUGE_VMALLOC */ +static const bool vmap_allow_huge =3D false; +#endif /* CONFIG_HAVE_ARCH_HUGE_VMALLOC */ + bool is_vmalloc_addr(const void *x) { unsigned long addr =3D (unsigned long)x; @@ -483,31 +496,12 @@ static int vmap_pages_p4d_range(pgd_t *pgd, unsigne= d long addr, return 0; } =20 -/** - * map_kernel_range_noflush - map kernel VM area with the specified page= s - * @addr: start of the VM area to map - * @size: size of the VM area to map - * @prot: page protection flags to use - * @pages: pages to map - * - * Map PFN_UP(@size) pages at @addr. The VM area @addr and @size specif= y should - * have been allocated using get_vm_area() and its friends. - * - * NOTE: - * This function does NOT do any cache flushing. The caller is responsi= ble for - * calling flush_cache_vmap() on to-be-mapped areas before calling this - * function. - * - * RETURNS: - * 0 on success, -errno on failure. - */ -int map_kernel_range_noflush(unsigned long addr, unsigned long size, - pgprot_t prot, struct page **pages) +static int vmap_small_pages_range_noflush(unsigned long addr, unsigned l= ong end, + pgprot_t prot, struct page **pages) { unsigned long start =3D addr; - unsigned long end =3D addr + size; - unsigned long next; pgd_t *pgd; + unsigned long next; int err =3D 0; int nr =3D 0; pgtbl_mod_mask mask =3D 0; @@ -529,6 +523,66 @@ int map_kernel_range_noflush(unsigned long addr, uns= igned long size, return 0; } =20 +static int vmap_pages_range_noflush(unsigned long addr, unsigned long en= d, + pgprot_t prot, struct page **pages, unsigned int page_shift) +{ + unsigned int i, nr =3D (end - addr) >> PAGE_SHIFT; + + WARN_ON(page_shift < PAGE_SHIFT); + + if (!IS_ENABLED(CONFIG_HAVE_ARCH_HUGE_VMALLOC) || + page_shift =3D=3D PAGE_SHIFT) + return vmap_small_pages_range_noflush(addr, end, prot, pages); + + for (i =3D 0; i < nr; i +=3D 1U << (page_shift - PAGE_SHIFT)) { + int err; + + err =3D vmap_range_noflush(addr, addr + (1UL << page_shift), + __pa(page_address(pages[i])), prot, + page_shift); + if (err) + return err; + + addr +=3D 1UL << page_shift; + } + + return 0; +} + +static int vmap_pages_range(unsigned long addr, unsigned long end, + pgprot_t prot, struct page **pages, unsigned int page_shift) +{ + int err; + + err =3D vmap_pages_range_noflush(addr, end, prot, pages, page_shift); + flush_cache_vmap(addr, end); + return err; +} + +/** + * map_kernel_range_noflush - map kernel VM area with the specified page= s + * @addr: start of the VM area to map + * @size: size of the VM area to map + * @prot: page protection flags to use + * @pages: pages to map + * + * Map PFN_UP(@size) pages at @addr. The VM area @addr and @size specif= y should + * have been allocated using get_vm_area() and its friends. + * + * NOTE: + * This function does NOT do any cache flushing. The caller is responsi= ble for + * calling flush_cache_vmap() on to-be-mapped areas before calling this + * function. + * + * RETURNS: + * 0 on success, -errno on failure. + */ +int map_kernel_range_noflush(unsigned long addr, unsigned long size, + pgprot_t prot, struct page **pages) +{ + return vmap_pages_range_noflush(addr, addr + size, prot, pages, PAGE_SH= IFT); +} + int map_kernel_range(unsigned long start, unsigned long size, pgprot_t p= rot, struct page **pages) { @@ -2112,6 +2166,24 @@ EXPORT_SYMBOL(vm_map_ram); =20 static struct vm_struct *vmlist __initdata; =20 +static inline unsigned int vm_area_page_order(struct vm_struct *vm) +{ +#ifdef CONFIG_HAVE_ARCH_HUGE_VMALLOC + return vm->page_order; +#else + return 0; +#endif +} + +static inline void set_vm_area_page_order(struct vm_struct *vm, unsigned= int order) +{ +#ifdef CONFIG_HAVE_ARCH_HUGE_VMALLOC + vm->page_order =3D order; +#else + BUG_ON(order !=3D 0); +#endif +} + /** * vm_area_add_early - add vmap area early during boot * @vm: vm_struct to add @@ -2422,6 +2494,7 @@ static inline void set_area_direct_map(const struct= vm_struct *area, { int i; =20 + /* HUGE_VMALLOC passes small pages to set_direct_map */ for (i =3D 0; i < area->nr_pages; i++) if (page_address(area->pages[i])) set_direct_map(area->pages[i]); @@ -2431,6 +2504,7 @@ static inline void set_area_direct_map(const struct= vm_struct *area, static void vm_remove_mappings(struct vm_struct *area, int deallocate_pa= ges) { unsigned long start =3D ULONG_MAX, end =3D 0; + unsigned int page_order =3D vm_area_page_order(area); int flush_reset =3D area->flags & VM_FLUSH_RESET_PERMS; int flush_dmap =3D 0; int i; @@ -2455,11 +2529,14 @@ static void vm_remove_mappings(struct vm_struct *= area, int deallocate_pages) * map. Find the start and end range of the direct mappings to make sur= e * the vm_unmap_aliases() flush includes the direct map. */ - for (i =3D 0; i < area->nr_pages; i++) { + for (i =3D 0; i < area->nr_pages; i +=3D 1U << page_order) { unsigned long addr =3D (unsigned long)page_address(area->pages[i]); if (addr) { + unsigned long page_size; + + page_size =3D PAGE_SIZE << page_order; start =3D min(addr, start); - end =3D max(addr + PAGE_SIZE, end); + end =3D max(addr + page_size, end); flush_dmap =3D 1; } } @@ -2500,13 +2577,14 @@ static void __vunmap(const void *addr, int deallo= cate_pages) vm_remove_mappings(area, deallocate_pages); =20 if (deallocate_pages) { + unsigned int page_order =3D vm_area_page_order(area); int i; =20 - for (i =3D 0; i < area->nr_pages; i++) { + for (i =3D 0; i < area->nr_pages; i +=3D 1U << page_order) { struct page *page =3D area->pages[i]; =20 BUG_ON(!page); - __free_pages(page, 0); + __free_pages(page, page_order); } atomic_long_sub(area->nr_pages, &nr_vmalloc_pages); =20 @@ -2697,15 +2775,19 @@ EXPORT_SYMBOL_GPL(vmap_pfn); #endif /* CONFIG_VMAP_PFN */ =20 static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, - pgprot_t prot, int node) + pgprot_t prot, unsigned int page_shift, + int node) { const gfp_t nested_gfp =3D (gfp_mask & GFP_RECLAIM_MASK) | __GFP_ZERO; - unsigned int nr_pages =3D get_vm_area_size(area) >> PAGE_SHIFT; + unsigned long addr =3D (unsigned long)area->addr; + unsigned long size =3D get_vm_area_size(area); unsigned long array_size; - unsigned int i; + unsigned int nr_small_pages =3D size >> PAGE_SHIFT; + unsigned int page_order; struct page **pages; + unsigned int i; =20 - array_size =3D (unsigned long)nr_pages * sizeof(struct page *); + array_size =3D (unsigned long)nr_small_pages * sizeof(struct page *); gfp_mask |=3D __GFP_NOWARN; if (!(gfp_mask & (GFP_DMA | GFP_DMA32))) gfp_mask |=3D __GFP_HIGHMEM; @@ -2724,30 +2806,38 @@ static void *__vmalloc_area_node(struct vm_struct= *area, gfp_t gfp_mask, } =20 area->pages =3D pages; - area->nr_pages =3D nr_pages; + area->nr_pages =3D nr_small_pages; + set_vm_area_page_order(area, page_shift - PAGE_SHIFT); =20 - for (i =3D 0; i < area->nr_pages; i++) { - struct page *page; + page_order =3D vm_area_page_order(area); =20 - if (node =3D=3D NUMA_NO_NODE) - page =3D alloc_page(gfp_mask); - else - page =3D alloc_pages_node(node, gfp_mask, 0); + /* + * Careful, we allocate and map page_order pages, but tracking is done + * per PAGE_SIZE page so as to keep the vm_struct APIs independent of + * the physical/mapped size. + */ + for (i =3D 0; i < area->nr_pages; i +=3D 1U << page_order) { + struct page *page; + int p; =20 + /* Compound pages required for remap_vmalloc_page */ + page =3D alloc_pages_node(node, gfp_mask | __GFP_COMP, page_order); if (unlikely(!page)) { /* Successfully allocated i pages, free them in __vfree() */ area->nr_pages =3D i; atomic_long_add(area->nr_pages, &nr_vmalloc_pages); goto fail; } - area->pages[i] =3D page; + + for (p =3D 0; p < (1U << page_order); p++) + area->pages[i + p] =3D page + p; + if (gfpflags_allow_blocking(gfp_mask)) cond_resched(); } atomic_long_add(area->nr_pages, &nr_vmalloc_pages); =20 - if (map_kernel_range((unsigned long)area->addr, get_vm_area_size(area), - prot, pages) < 0) + if (vmap_pages_range(addr, addr + size, prot, pages, page_shift) < 0) goto fail; =20 return area->addr; @@ -2755,7 +2845,7 @@ static void *__vmalloc_area_node(struct vm_struct *= area, gfp_t gfp_mask, fail: warn_alloc(gfp_mask, NULL, "vmalloc: allocation failure, allocated %ld of %ld bytes", - (area->nr_pages*PAGE_SIZE), area->size); + (area->nr_pages*PAGE_SIZE), size); __vfree(area->addr); return NULL; } @@ -2786,19 +2876,43 @@ void *__vmalloc_node_range(unsigned long size, un= signed long align, struct vm_struct *area; void *addr; unsigned long real_size =3D size; + unsigned long real_align =3D align; + unsigned int shift =3D PAGE_SHIFT; =20 - size =3D PAGE_ALIGN(size); if (!size || (size >> PAGE_SHIFT) > totalram_pages()) goto fail; =20 - area =3D __get_vm_area_node(real_size, align, VM_ALLOC | VM_UNINITIALIZ= ED | + if (vmap_allow_huge && !(vm_flags & VM_NO_HUGE_VMAP) && + arch_vmap_pmd_supported(prot)) { + unsigned long size_per_node; + + /* + * Try huge pages. Only try for PAGE_KERNEL allocations, + * others like modules don't yet expect huge pages in + * their allocations due to apply_to_page_range not + * supporting them. + */ + + size_per_node =3D size; + if (node =3D=3D NUMA_NO_NODE) + size_per_node /=3D num_online_nodes(); + if (size_per_node >=3D PMD_SIZE) { + shift =3D PMD_SHIFT; + align =3D max(real_align, 1UL << shift); + size =3D ALIGN(real_size, 1UL << shift); + } + } + +again: + size =3D PAGE_ALIGN(size); + area =3D __get_vm_area_node(size, align, VM_ALLOC | VM_UNINITIALIZED | vm_flags, start, end, node, gfp_mask, caller); if (!area) goto fail; =20 - addr =3D __vmalloc_area_node(area, gfp_mask, prot, node); + addr =3D __vmalloc_area_node(area, gfp_mask, prot, shift, node); if (!addr) - return NULL; + goto fail; =20 /* * In this function, newly allocated vm_struct has VM_UNINITIALIZED @@ -2812,8 +2926,18 @@ void *__vmalloc_node_range(unsigned long size, uns= igned long align, return addr; =20 fail: - warn_alloc(gfp_mask, NULL, + if (shift > PAGE_SHIFT) { + shift =3D PAGE_SHIFT; + align =3D real_align; + size =3D real_size; + goto again; + } + + if (!area) { + /* Warn for area allocation, page allocations already warn */ + warn_alloc(gfp_mask, NULL, "vmalloc: allocation failure: %lu bytes", real_size); + } return NULL; } =20 --=20 2.23.0