From: Suren Baghdasaryan <surenb@google.com>
To: Matthew Wilcox <willy@infradead.org>
Cc: "Liam R. Howlett" <liam.howlett@oracle.com>,
Michal Hocko <mhocko@suse.com>,
Andrew Morton <akpm@linux-foundation.org>,
David Rientjes <rientjes@google.com>,
Johannes Weiner <hannes@cmpxchg.org>,
Roman Gushchin <guro@fb.com>, Rik van Riel <riel@surriel.com>,
Minchan Kim <minchan@kernel.org>,
Christian Brauner <christian@brauner.io>,
Christoph Hellwig <hch@infradead.org>,
Oleg Nesterov <oleg@redhat.com>,
David Hildenbrand <david@redhat.com>,
Jann Horn <jannh@google.com>, Shakeel Butt <shakeelb@google.com>,
Andy Lutomirski <luto@kernel.org>,
Christian Brauner <christian.brauner@ubuntu.com>,
Florian Weimer <fweimer@redhat.com>,
Jan Engelhardt <jengelh@inai.de>,
Linux API <linux-api@vger.kernel.org>,
linux-mm <linux-mm@kvack.org>,
LKML <linux-kernel@vger.kernel.org>,
kernel-team <kernel-team@android.com>,
"Kirill A. Shutemov" <kirill@shutemov.name>,
Andrea Arcangeli <aarcange@redhat.com>
Subject: Re: [PATCH 1/1] mm: prevent a race between process_mrelease and exit_mmap
Date: Wed, 27 Oct 2021 10:42:29 -0700 [thread overview]
Message-ID: <CAJuCfpFP-57JkWhDAN4T6VtPboSV4LGqipHMU4j+wJKU45yjYg@mail.gmail.com> (raw)
In-Reply-To: <YXmNaoV4dBTOJ3+w@casper.infradead.org>
On Wed, Oct 27, 2021 at 10:35 AM Matthew Wilcox <willy@infradead.org> wrote:
>
> On Wed, Oct 27, 2021 at 09:08:21AM -0700, Suren Baghdasaryan wrote:
> > Unconditional mmap_write_lock around free_pgtables in exit_mmap seems
> > to me the most semantically correct way forward and the pushback is on
> > the basis of regressing performance of the exit path. I would like to
> > measure that regression to confirm this. I don't have access to a big
> > machine but will ask someone in another Google team to try the test
> > Michal wrote here
> > https://lore.kernel.org/all/20170725142626.GJ26723@dhcp22.suse.cz/ on
> > a server with and without a custom patch.
>
> Sorry to hijack this, but could you ask that team to also test this
> patch? I think there's probably a good-sized win here, but I have no
> profiles to share at this point. I've only done light testing, and
> it may have bugs.
>
> NB: I only did the exit() path here. fork() conversion is left as an
> exercise for the reader^W^W Liam.
To clarify, this patch does not change the mmap_write_lock portion of
exit_mmap. Do you want to test it in isolation or with the locking
changes in exit_mmap I mentioned?
>
> From 5f9daa14a5e58c86a73eccf59abe23d131004926 Mon Sep 17 00:00:00 2001
> From: "Matthew Wilcox (Oracle)" <willy@infradead.org>
> Date: Wed, 27 Oct 2021 13:28:35 -0400
> Subject: [PATCH] mm: Add vmavec
>
> The vmavec lets us allocate and free batches of VMAs instead of
> one at a time. Should improve fork() and exit() performance.
>
> Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
> ---
> include/linux/vmavec.h | 38 ++++++++++++++++++++++++++++++++++++++
> kernel/fork.c | 17 +++++++++++++++++
> mm/mmap.c | 30 ++++++++++++++++++++++--------
> 3 files changed, 77 insertions(+), 8 deletions(-)
> create mode 100644 include/linux/vmavec.h
>
> diff --git a/include/linux/vmavec.h b/include/linux/vmavec.h
> new file mode 100644
> index 000000000000..8a324e2e1258
> --- /dev/null
> +++ b/include/linux/vmavec.h
> @@ -0,0 +1,38 @@
> +/*
> + * A vma vector is an array of vm_area_structs, with a counter.
> + */
> +
> +struct vm_area_struct;
> +
> +#define VMAVEC_SIZE 15
> +
> +struct vmavec {
> + unsigned char nr;
> + void *vmas[VMAVEC_SIZE];
> +};
> +
> +#define VMAVEC(name) struct vmavec name = { }
> +
> +static inline bool vmavec_full(struct vmavec *vmavec)
> +{
> + return vmavec->nr == VMAVEC_SIZE;
> +}
> +
> +static inline bool vmavec_empty(struct vmavec *vmavec)
> +{
> + return vmavec->nr == 0;
> +}
> +
> +static inline
> +void vmavec_push(struct vmavec *vmavec, struct vm_area_struct *vma)
> +{
> + vmavec->vmas[vmavec->nr++] = vma;
> +}
> +
> +static inline struct vm_area_struct *vmavec_pop(struct vmavec *vmavec)
> +{
> + return vmavec->vmas[--vmavec->nr];
> +}
> +
> +void vm_area_free_vec(struct vmavec *vmavec);
> +void vm_area_alloc_vec(struct mm_struct *mm, struct vmavec *vmavec);
> diff --git a/kernel/fork.c b/kernel/fork.c
> index 38681ad44c76..ea7e8bd00be8 100644
> --- a/kernel/fork.c
> +++ b/kernel/fork.c
> @@ -97,6 +97,7 @@
> #include <linux/scs.h>
> #include <linux/io_uring.h>
> #include <linux/bpf.h>
> +#include <linux/vmavec.h>
>
> #include <asm/pgalloc.h>
> #include <linux/uaccess.h>
> @@ -375,6 +376,22 @@ void vm_area_free(struct vm_area_struct *vma)
> kmem_cache_free(vm_area_cachep, vma);
> }
>
> +void vm_area_alloc_vec(struct mm_struct *mm, struct vmavec *vmavec)
> +{
> + int i;
> +
> + vmavec->nr = kmem_cache_alloc_bulk(vm_area_cachep, GFP_KERNEL,
> + VMAVEC_SIZE, vmavec->vmas);
> + for (i = 0; i < vmavec->nr; i++)
> + vma_init(vmavec->vmas[i], mm);
> +}
> +
> +void vm_area_free_vec(struct vmavec *vmavec)
> +{
> + kmem_cache_free_bulk(vm_area_cachep, vmavec->nr, vmavec->vmas);
> + vmavec->nr = 0;
> +}
> +
> static void account_kernel_stack(struct task_struct *tsk, int account)
> {
> void *stack = task_stack_page(tsk);
> diff --git a/mm/mmap.c b/mm/mmap.c
> index 88dcc5c25225..bff4e94eec8c 100644
> --- a/mm/mmap.c
> +++ b/mm/mmap.c
> @@ -47,6 +47,7 @@
> #include <linux/pkeys.h>
> #include <linux/oom.h>
> #include <linux/sched/mm.h>
> +#include <linux/vmavec.h>
>
> #include <linux/uaccess.h>
> #include <asm/cacheflush.h>
> @@ -172,19 +173,24 @@ void unlink_file_vma(struct vm_area_struct *vma)
> }
> }
>
> -/*
> - * Close a vm structure and free it, returning the next.
> - */
> -static struct vm_area_struct *remove_vma(struct vm_area_struct *vma)
> +static void __remove_vma(struct vm_area_struct *vma)
> {
> - struct vm_area_struct *next = vma->vm_next;
> -
> might_sleep();
> if (vma->vm_ops && vma->vm_ops->close)
> vma->vm_ops->close(vma);
> if (vma->vm_file)
> fput(vma->vm_file);
> mpol_put(vma_policy(vma));
> +}
> +
> +/*
> + * Close a vm structure and free it, returning the next.
> + */
> +static struct vm_area_struct *remove_vma(struct vm_area_struct *vma)
> +{
> + struct vm_area_struct *next = vma->vm_next;
> +
> + __remove_vma(vma);
> vm_area_free(vma);
> return next;
> }
> @@ -3125,6 +3131,7 @@ void exit_mmap(struct mm_struct *mm)
> {
> struct mmu_gather tlb;
> struct vm_area_struct *vma;
> + VMAVEC(vmavec);
> unsigned long nr_accounted = 0;
>
> /* mm's last user has gone, and its about to be pulled down */
> @@ -3179,9 +3186,16 @@ void exit_mmap(struct mm_struct *mm)
> while (vma) {
> if (vma->vm_flags & VM_ACCOUNT)
> nr_accounted += vma_pages(vma);
> - vma = remove_vma(vma);
> - cond_resched();
> + __remove_vma(vma);
> + vmavec_push(&vmavec, vma);
> + vma = vma->vm_next;
> + if (vmavec_full(&vmavec)) {
> + vm_area_free_vec(&vmavec);
> + cond_resched();
> + }
> }
> + if (!vmavec_empty(&vmavec))
> + vm_area_free_vec(&vmavec);
> vm_unacct_memory(nr_accounted);
> }
>
> --
> 2.33.0
>
next prev parent reply other threads:[~2021-10-27 17:42 UTC|newest]
Thread overview: 32+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-10-22 1:46 [PATCH 1/1] mm: prevent a race between process_mrelease and exit_mmap Suren Baghdasaryan
2021-10-22 2:24 ` Andrew Morton
2021-10-22 5:23 ` Suren Baghdasaryan
2021-10-22 8:03 ` Michal Hocko
2021-10-22 11:32 ` Matthew Wilcox
2021-10-22 12:04 ` Michal Hocko
2021-10-22 17:38 ` Suren Baghdasaryan
2021-10-27 16:08 ` Suren Baghdasaryan
2021-10-27 17:33 ` Matthew Wilcox
2021-10-27 17:42 ` Suren Baghdasaryan [this message]
2021-10-27 17:51 ` Matthew Wilcox
2021-10-27 18:00 ` Suren Baghdasaryan
2021-10-29 13:03 ` Michal Hocko
2021-10-29 16:07 ` Suren Baghdasaryan
2021-11-01 8:37 ` Michal Hocko
2021-11-01 15:44 ` Suren Baghdasaryan
2021-11-01 19:59 ` Suren Baghdasaryan
2021-11-02 7:58 ` Michal Hocko
2021-11-02 15:14 ` Suren Baghdasaryan
2021-11-09 19:01 ` Suren Baghdasaryan
2021-11-09 19:26 ` Michal Hocko
2021-11-09 19:37 ` Suren Baghdasaryan
2021-11-09 19:50 ` Michal Hocko
2021-11-09 20:02 ` Suren Baghdasaryan
2021-11-09 20:10 ` Michal Hocko
2021-11-09 21:10 ` Suren Baghdasaryan
2021-11-11 1:49 ` Suren Baghdasaryan
2021-11-11 9:20 ` Michal Hocko
2021-11-11 15:02 ` Suren Baghdasaryan
2021-11-12 8:58 ` Michal Hocko
2021-11-12 16:00 ` Suren Baghdasaryan
2021-11-09 19:41 ` Michal Hocko
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CAJuCfpFP-57JkWhDAN4T6VtPboSV4LGqipHMU4j+wJKU45yjYg@mail.gmail.com \
--to=surenb@google.com \
--cc=aarcange@redhat.com \
--cc=akpm@linux-foundation.org \
--cc=christian.brauner@ubuntu.com \
--cc=christian@brauner.io \
--cc=david@redhat.com \
--cc=fweimer@redhat.com \
--cc=guro@fb.com \
--cc=hannes@cmpxchg.org \
--cc=hch@infradead.org \
--cc=jannh@google.com \
--cc=jengelh@inai.de \
--cc=kernel-team@android.com \
--cc=kirill@shutemov.name \
--cc=liam.howlett@oracle.com \
--cc=linux-api@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=luto@kernel.org \
--cc=mhocko@suse.com \
--cc=minchan@kernel.org \
--cc=oleg@redhat.com \
--cc=riel@surriel.com \
--cc=rientjes@google.com \
--cc=shakeelb@google.com \
--cc=willy@infradead.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).