From: Konstantin Khlebnikov <khlebnikov@yandex-team.ru>
To: Dave Hansen <dave.hansen@intel.com>,
linux-kernel@vger.kernel.org,
Christoph Hellwig <hch@infradead.org>,
linux-mm@kvack.org, Andy Lutomirski <luto@kernel.org>,
Andrew Morton <akpm@linux-foundation.org>
Subject: Re: [PATCH 3/4] kernel/fork: switch vmapped stack callation to __vmalloc_area()
Date: Tue, 23 Jan 2018 16:57:21 +0300 [thread overview]
Message-ID: <5c19630f-7466-676d-dbbc-a5668c91cbcd@yandex-team.ru> (raw)
In-Reply-To: <151670492913.658225.2758351129158778856.stgit@buzz>
# stress-ng --clone 100 -t 10s --metrics-brief
at 32-core machine shows boost 35000 -> 36000 bogo ops
Patch 4/4 is a kind of RFC.
Actually per-cpu cache of preallocated stacks works faster than buddy allocator thus
performance boots for it happens only at completely insane rate of clones.
On 23.01.2018 13:55, Konstantin Khlebnikov wrote:
> This gives as pointer vm_struct without calling find_vm_area().
>
> And fix comment about that task holds cache of vm area: this cache used
> for retrieving actual stack pages, freeing is done by vfree_deferred().
>
> Signed-off-by: Konstantin Khlebnikov <khlebnikov@yandex-team.ru>
> ---
> kernel/fork.c | 37 +++++++++++++++----------------------
> 1 file changed, 15 insertions(+), 22 deletions(-)
>
> diff --git a/kernel/fork.c b/kernel/fork.c
> index 2295fc69717f..457c9151f3c8 100644
> --- a/kernel/fork.c
> +++ b/kernel/fork.c
> @@ -204,39 +204,32 @@ static int free_vm_stack_cache(unsigned int cpu)
> static unsigned long *alloc_thread_stack_node(struct task_struct *tsk, int node)
> {
> #ifdef CONFIG_VMAP_STACK
> - void *stack;
> + struct vm_struct *stack;
> int i;
>
> for (i = 0; i < NR_CACHED_STACKS; i++) {
> - struct vm_struct *s;
> -
> - s = this_cpu_xchg(cached_stacks[i], NULL);
> -
> - if (!s)
> + stack = this_cpu_xchg(cached_stacks[i], NULL);
> + if (!stack)
> continue;
>
> #ifdef CONFIG_DEBUG_KMEMLEAK
> /* Clear stale pointers from reused stack. */
> - memset(s->addr, 0, THREAD_SIZE);
> + memset(stack->addr, 0, THREAD_SIZE);
> #endif
> - tsk->stack_vm_area = s;
> - return s->addr;
> + tsk->stack_vm_area = stack;
> + return stack->addr;
> }
>
> - stack = __vmalloc_node_range(THREAD_SIZE, THREAD_ALIGN,
> - VMALLOC_START, VMALLOC_END,
> - THREADINFO_GFP,
> - PAGE_KERNEL,
> - 0, node, __builtin_return_address(0));
> + stack = __vmalloc_area(THREAD_SIZE, THREAD_ALIGN,
> + VMALLOC_START, VMALLOC_END,
> + THREADINFO_GFP, PAGE_KERNEL,
> + 0, node, __builtin_return_address(0));
> + if (unlikely(!stack))
> + return NULL;
>
> - /*
> - * We can't call find_vm_area() in interrupt context, and
> - * free_thread_stack() can be called in interrupt context,
> - * so cache the vm_struct.
> - */
> - if (stack)
> - tsk->stack_vm_area = find_vm_area(stack);
> - return stack;
> + /* Cache the vm_struct for stack to page conversions. */
> + tsk->stack_vm_area = stack;
> + return stack->addr;
> #else
> struct page *page = alloc_pages_node(node, THREADINFO_GFP,
> THREAD_SIZE_ORDER);
>
next prev parent reply other threads:[~2018-01-23 13:57 UTC|newest]
Thread overview: 24+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-01-23 10:55 [PATCH 1/4] vmalloc: add vm_flags argument to internal __vmalloc_node() Konstantin Khlebnikov
2018-01-23 10:55 ` [PATCH 2/4] vmalloc: add __vmalloc_area() Konstantin Khlebnikov
2018-01-23 10:55 ` [PATCH 3/4] kernel/fork: switch vmapped stack callation to __vmalloc_area() Konstantin Khlebnikov
2018-01-23 13:57 ` Konstantin Khlebnikov [this message]
2018-02-21 0:16 ` Andrew Morton
2018-02-21 7:23 ` Konstantin Khlebnikov
2018-02-21 16:35 ` Andy Lutomirski
2018-01-23 10:55 ` [PATCH 4/4] kernel/fork: add option to use virtually mapped stacks as fallback Konstantin Khlebnikov
2018-02-21 15:42 ` Use higher-order pages in vmalloc Matthew Wilcox
2018-02-21 16:11 ` Andy Lutomirski
2018-02-21 16:50 ` Matthew Wilcox
2018-02-21 16:16 ` Dave Hansen
2018-02-21 17:01 ` Matthew Wilcox
2018-02-22 6:59 ` Michal Hocko
2018-02-22 12:22 ` Matthew Wilcox
2018-02-22 13:36 ` Michal Hocko
2018-02-22 19:01 ` Andy Lutomirski
2018-02-22 19:19 ` Dave Hansen
2018-02-22 19:27 ` Andy Lutomirski
2018-02-22 19:36 ` Dave Hansen
2018-02-23 12:13 ` Michal Hocko
2018-03-01 18:16 ` Eric Dumazet
2018-02-21 12:24 ` [PATCH 1/4] vmalloc: add vm_flags argument to internal __vmalloc_node() Matthew Wilcox
2018-02-21 12:39 ` Andrey Ryabinin
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=5c19630f-7466-676d-dbbc-a5668c91cbcd@yandex-team.ru \
--to=khlebnikov@yandex-team.ru \
--cc=akpm@linux-foundation.org \
--cc=dave.hansen@intel.com \
--cc=hch@infradead.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=luto@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).