From: Uladzislau Rezki <urezki@gmail.com>
To: Matthew Wilcox <willy@infradead.org>
Cc: Uladzislau Rezki <urezki@gmail.com>,
Andrew Morton <akpm@linux-foundation.org>,
Christoph Lameter <cl@linux.com>,
Pekka Enberg <penberg@kernel.org>,
David Rientjes <rientjes@google.com>,
Joonsoo Kim <iamjoonsoo.kim@lge.com>,
Vlastimil Babka <vbabka@suse.cz>,
linux-mm@kvack.org, linux-kernel@vger.kernel.org,
Nicholas Piggin <npiggin@gmail.com>
Subject: Re: [PATCH 2/2] mm/vmalloc: Use kvmalloc to allocate the table of pages
Date: Tue, 23 Mar 2021 14:39:48 +0100 [thread overview]
Message-ID: <20210323133948.GA10046@pc638.lan> (raw)
In-Reply-To: <20210323123913.GD1719932@casper.infradead.org>
On Tue, Mar 23, 2021 at 12:39:13PM +0000, Matthew Wilcox wrote:
> On Tue, Mar 23, 2021 at 01:04:36PM +0100, Uladzislau Rezki wrote:
> > On Mon, Mar 22, 2021 at 11:03:11PM +0000, Matthew Wilcox wrote:
> > > I suspect the vast majority of the time is spent calling alloc_pages_node()
> > > 1024 times. Have you looked at Mel's patch to do ... well, exactly what
> > > vmalloc() wants?
> > >
> > <snip>
> > - __vmalloc_node_range
> > - 45.25% __alloc_pages_nodemask
> > - 37.59% get_page_from_freelist
> [...]
> > - 44.61% 0xffffffffc047348d
> > - __vunmap
> > - 35.56% free_unref_page
>
> Hmm! I hadn't been thinking about the free side of things.
> Does this make a difference?
>
> diff --git a/mm/vmalloc.c b/mm/vmalloc.c
> index 4f5f8c907897..61d5b769fea0 100644
> --- a/mm/vmalloc.c
> +++ b/mm/vmalloc.c
> @@ -2277,16 +2277,8 @@ static void __vunmap(const void *addr, int deallocate_pages)
> vm_remove_mappings(area, deallocate_pages);
>
> if (deallocate_pages) {
> - int i;
> -
> - for (i = 0; i < area->nr_pages; i++) {
> - struct page *page = area->pages[i];
> -
> - BUG_ON(!page);
> - __free_pages(page, 0);
> - }
> + release_pages(area->pages, area->nr_pages);
> atomic_long_sub(area->nr_pages, &nr_vmalloc_pages);
> -
> kvfree(area->pages);
> }
>
Will check it today!
> release_pages does a bunch of checks that are unnecessary ... we could
> probably just do:
>
> LIST_HEAD(pages_to_free);
>
> for (i = 0; i < area->nr_pages; i++) {
> struct page *page = area->pages[i];
> if (put_page_testzero(page))
> list_add(&page->lru, &pages_to_free);
> }
> free_unref_page_list(&pages_to_free);
>
> but let's see if the provided interface gets us the performance we want.
>
> > Reviewed-by: Uladzislau Rezki (Sony) <urezki@gmail.com>
> >
> > Thanks!
>
> Thank you!
You are welcome. A small nit:
CC mm/vmalloc.o
mm/vmalloc.c: In function ‘__vmalloc_area_node’:
mm/vmalloc.c:2492:14: warning: passing argument 4 of ‘kvmalloc_node_caller’ makes integer from pointer without a cast [-Wint-conversion]
area->caller);
~~~~^~~~~~~~
In file included from mm/vmalloc.c:12:
./include/linux/mm.h:782:7: note: expected ‘long unsigned int’ but argument is of type ‘const void *’
void *kvmalloc_node_caller(size_t size, gfp_t flags, int node,
<snip>
diff --git a/mm/vmalloc.c b/mm/vmalloc.c
index 8a202ba263f6..ee6fa44983bc 100644
--- a/mm/vmalloc.c
+++ b/mm/vmalloc.c
@@ -2489,7 +2489,7 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask,
/* Please note that the recursion is strictly bounded. */
pages = kvmalloc_node_caller(array_size, nested_gfp, node,
- area->caller);
+ (unsigned long) area->caller);
if (!pages) {
free_vm_area(area);
return NULL;
<snip>
As for the bulk-array interface. I have checked the:
git://git.kernel.org/pub/scm/linux/kernel/git/mel/linux.git mm-bulk-rebase-v6r2
applied the patch that is in question + below one:
<snip>
@@ -2503,25 +2498,13 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask,
area->pages = pages;
area->nr_pages = nr_pages;
- for (i = 0; i < area->nr_pages; i++) {
- struct page *page;
-
- if (node == NUMA_NO_NODE)
- page = alloc_page(gfp_mask);
- else
- page = alloc_pages_node(node, gfp_mask, 0);
-
- if (unlikely(!page)) {
- /* Successfully allocated i pages, free them in __vfree() */
- area->nr_pages = i;
- atomic_long_add(area->nr_pages, &nr_vmalloc_pages);
- goto fail;
- }
- area->pages[i] = page;
- if (gfpflags_allow_blocking(gfp_mask))
- cond_resched();
+ ret = alloc_pages_bulk_array(gfp_mask, area->nr_pages, area->pages);
+ if (ret == nr_pages)
+ atomic_long_add(area->nr_pages, &nr_vmalloc_pages);
+ else {
+ area->nr_pages = ret;
+ goto fail;
}
- atomic_long_add(area->nr_pages, &nr_vmalloc_pages);
<snip>
single CPU, 4MB allocation, 1000000 avg: 70639437 usec
single CPU, 4MB allocation, 1000000 avg: 89218654 usec
and now we get ~21% delta. That is very good :)
--
Vlad Rezki
next prev parent reply other threads:[~2021-03-23 13:39 UTC|newest]
Thread overview: 11+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-03-22 19:38 [PATCH 1/2] mm/util: Add kvmalloc_node_caller Matthew Wilcox (Oracle)
2021-03-22 19:38 ` [PATCH 2/2] mm/vmalloc: Use kvmalloc to allocate the table of pages Matthew Wilcox (Oracle)
2021-03-22 22:36 ` Uladzislau Rezki
2021-03-22 23:03 ` Matthew Wilcox
2021-03-23 12:04 ` Uladzislau Rezki
2021-03-23 12:39 ` Matthew Wilcox
2021-03-23 13:39 ` Uladzislau Rezki [this message]
2021-03-23 14:07 ` Matthew Wilcox
2021-03-23 20:49 ` Uladzislau Rezki
2021-03-23 20:39 ` Uladzislau Rezki
2021-03-24 18:41 ` Uladzislau Rezki
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210323133948.GA10046@pc638.lan \
--to=urezki@gmail.com \
--cc=akpm@linux-foundation.org \
--cc=cl@linux.com \
--cc=iamjoonsoo.kim@lge.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=npiggin@gmail.com \
--cc=penberg@kernel.org \
--cc=rientjes@google.com \
--cc=vbabka@suse.cz \
--cc=willy@infradead.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).