From: Suren Baghdasaryan <surenb@google.com> To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan <surenb@google.com> Subject: [PATCH v3 16/35] mm/mmap: write-lock VMAs before merging, splitting or expanding them Date: Wed, 15 Feb 2023 21:17:31 -0800 [thread overview] Message-ID: <20230216051750.3125598-17-surenb@google.com> (raw) In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Decisions about whether VMAs can be merged, split or expanded must be made while VMAs are protected from the changes which can affect that decision. For example, merge_vma uses vma->anon_vma in its decision whether the VMA can be merged. Meanwhile, page fault handler changes vma->anon_vma during COW operation. Write-lock all VMAs which might be affected by a merge or split operation before making decision how such operations should be performed. Signed-off-by: Suren Baghdasaryan <surenb@google.com> --- mm/mmap.c | 23 ++++++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-) diff --git a/mm/mmap.c b/mm/mmap.c index c5f2ddf17b87..ec2f8d0af280 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -269,8 +269,11 @@ SYSCALL_DEFINE1(brk, unsigned long, brk) */ vma_iter_init(&vmi, mm, oldbrk); next = vma_find(&vmi, newbrk + PAGE_SIZE + stack_guard_gap); - if (next && newbrk + PAGE_SIZE > vm_start_gap(next)) - goto out; + if (next) { + vma_start_write(next); + if (newbrk + PAGE_SIZE > vm_start_gap(next)) + goto out; + } brkvma = vma_prev_limit(&vmi, mm->start_brk); /* Ok, looks good - let it rip. */ @@ -912,10 +915,17 @@ struct vm_area_struct *vma_merge(struct vma_iterator *vmi, struct mm_struct *mm, if (vm_flags & VM_SPECIAL) return NULL; + if (prev) + vma_start_write(prev); next = find_vma(mm, prev ? prev->vm_end : 0); + if (next) + vma_start_write(next); mid = next; - if (next && next->vm_end == end) /* cases 6, 7, 8 */ + if (next && next->vm_end == end) { /* cases 6, 7, 8 */ next = find_vma(mm, next->vm_end); + if (next) + vma_start_write(next); + } /* verify some invariant that must be enforced by the caller */ VM_WARN_ON(prev && addr <= prev->vm_start); @@ -2163,6 +2173,7 @@ int __split_vma(struct vma_iterator *vmi, struct vm_area_struct *vma, WARN_ON(vma->vm_start >= addr); WARN_ON(vma->vm_end <= addr); + vma_start_write(vma); if (vma->vm_ops && vma->vm_ops->may_split) { err = vma->vm_ops->may_split(vma, addr); if (err) @@ -2518,6 +2529,8 @@ unsigned long mmap_region(struct file *file, unsigned long addr, /* Attempt to expand an old mapping */ /* Check next */ + if (next) + vma_start_write(next); if (next && next->vm_start == end && !vma_policy(next) && can_vma_merge_before(next, vm_flags, NULL, file, pgoff+pglen, NULL_VM_UFFD_CTX, NULL)) { @@ -2527,6 +2540,8 @@ unsigned long mmap_region(struct file *file, unsigned long addr, } /* Check prev */ + if (prev) + vma_start_write(prev); if (prev && prev->vm_end == addr && !vma_policy(prev) && (vma ? can_vma_merge_after(prev, vm_flags, vma->anon_vma, file, pgoff, vma->vm_userfaultfd_ctx, NULL) : @@ -2900,6 +2915,8 @@ static int do_brk_flags(struct vma_iterator *vmi, struct vm_area_struct *vma, if (security_vm_enough_memory_mm(mm, len >> PAGE_SHIFT)) return -ENOMEM; + if (vma) + vma_start_write(vma); /* * Expand the existing vma if possible; Note that singular lists do not * occur after forking, so the expand will only happen on new VMAs. -- 2.39.1
WARNING: multiple messages have this Message-ID (diff)
From: Suren Baghdasaryan <surenb@google.com> To: akpm@linux-foundation.org Cc: michel@lespinasse.org, joelaf@google.com, songliubraving@fb.com, mhocko@suse.com, leewalsh@google.com, david@redhat.com, peterz@infradead.org, bigeasy@linutronix.de, peterx@redhat.com, dhowells@redhat.com, linux-mm@kvack.org, edumazet@google.com, jglisse@google.com, punit.agrawal@bytedance.com, will@kernel.org, arjunroy@google.com, chriscli@google.com, dave@stgolabs.net, minchan@google.com, x86@kernel.org, hughd@google.com, willy@infradead.org, gurua@google.com, mingo@redhat.com, linux-arm-kernel@lists.infradead.org, rientjes@google.com, axelrasmussen@google.com, kernel-team@android.com, michalechner92@googlemail.com, soheil@google.com, paulmck@kernel.org, jannh@google.com, liam.howlett@oracle.com, shakeelb@google.com, luto@kernel.org, gthelen@google.com, ldufour@linux.ibm.com, Suren Baghdasaryan <surenb@google.com>, vbabka@suse.cz, posk@google.com, lstoakes@gmail.com, peterjung1337@gmail.com, linuxppc-dev@lists.ozlabs.org, kent.overstreet@linux.dev, linux-kernel@vger.kernel.org, hannes@cmpxchg.org, tatashin@google.com, mgorman@techsingularity.net, rppt@kernel.org Subject: [PATCH v3 16/35] mm/mmap: write-lock VMAs before merging, splitting or expanding them Date: Wed, 15 Feb 2023 21:17:31 -0800 [thread overview] Message-ID: <20230216051750.3125598-17-surenb@google.com> (raw) In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Decisions about whether VMAs can be merged, split or expanded must be made while VMAs are protected from the changes which can affect that decision. For example, merge_vma uses vma->anon_vma in its decision whether the VMA can be merged. Meanwhile, page fault handler changes vma->anon_vma during COW operation. Write-lock all VMAs which might be affected by a merge or split operation before making decision how such operations should be performed. Signed-off-by: Suren Baghdasaryan <surenb@google.com> --- mm/mmap.c | 23 ++++++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-) diff --git a/mm/mmap.c b/mm/mmap.c index c5f2ddf17b87..ec2f8d0af280 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -269,8 +269,11 @@ SYSCALL_DEFINE1(brk, unsigned long, brk) */ vma_iter_init(&vmi, mm, oldbrk); next = vma_find(&vmi, newbrk + PAGE_SIZE + stack_guard_gap); - if (next && newbrk + PAGE_SIZE > vm_start_gap(next)) - goto out; + if (next) { + vma_start_write(next); + if (newbrk + PAGE_SIZE > vm_start_gap(next)) + goto out; + } brkvma = vma_prev_limit(&vmi, mm->start_brk); /* Ok, looks good - let it rip. */ @@ -912,10 +915,17 @@ struct vm_area_struct *vma_merge(struct vma_iterator *vmi, struct mm_struct *mm, if (vm_flags & VM_SPECIAL) return NULL; + if (prev) + vma_start_write(prev); next = find_vma(mm, prev ? prev->vm_end : 0); + if (next) + vma_start_write(next); mid = next; - if (next && next->vm_end == end) /* cases 6, 7, 8 */ + if (next && next->vm_end == end) { /* cases 6, 7, 8 */ next = find_vma(mm, next->vm_end); + if (next) + vma_start_write(next); + } /* verify some invariant that must be enforced by the caller */ VM_WARN_ON(prev && addr <= prev->vm_start); @@ -2163,6 +2173,7 @@ int __split_vma(struct vma_iterator *vmi, struct vm_area_struct *vma, WARN_ON(vma->vm_start >= addr); WARN_ON(vma->vm_end <= addr); + vma_start_write(vma); if (vma->vm_ops && vma->vm_ops->may_split) { err = vma->vm_ops->may_split(vma, addr); if (err) @@ -2518,6 +2529,8 @@ unsigned long mmap_region(struct file *file, unsigned long addr, /* Attempt to expand an old mapping */ /* Check next */ + if (next) + vma_start_write(next); if (next && next->vm_start == end && !vma_policy(next) && can_vma_merge_before(next, vm_flags, NULL, file, pgoff+pglen, NULL_VM_UFFD_CTX, NULL)) { @@ -2527,6 +2540,8 @@ unsigned long mmap_region(struct file *file, unsigned long addr, } /* Check prev */ + if (prev) + vma_start_write(prev); if (prev && prev->vm_end == addr && !vma_policy(prev) && (vma ? can_vma_merge_after(prev, vm_flags, vma->anon_vma, file, pgoff, vma->vm_userfaultfd_ctx, NULL) : @@ -2900,6 +2915,8 @@ static int do_brk_flags(struct vma_iterator *vmi, struct vm_area_struct *vma, if (security_vm_enough_memory_mm(mm, len >> PAGE_SHIFT)) return -ENOMEM; + if (vma) + vma_start_write(vma); /* * Expand the existing vma if possible; Note that singular lists do not * occur after forking, so the expand will only happen on new VMAs. -- 2.39.1
next prev parent reply other threads:[~2023-02-16 5:20 UTC|newest] Thread overview: 141+ messages / expand[flat|nested] mbox.gz Atom feed top 2023-02-16 5:17 [PATCH v3 00/35] Per-VMA locks Suren Baghdasaryan 2023-02-16 5:17 ` Suren Baghdasaryan 2023-02-16 5:17 ` [PATCH v3 01/35] maple_tree: Be more cautious about dead nodes Suren Baghdasaryan 2023-02-16 5:17 ` Suren Baghdasaryan 2023-02-16 5:17 ` [PATCH v3 02/35] maple_tree: Detect dead nodes in mas_start() Suren Baghdasaryan 2023-02-16 5:17 ` Suren Baghdasaryan 2023-02-16 5:17 ` [PATCH v3 03/35] maple_tree: Fix freeing of nodes in rcu mode Suren Baghdasaryan 2023-02-16 5:17 ` Suren Baghdasaryan 2023-02-16 5:17 ` [PATCH v3 04/35] maple_tree: remove extra smp_wmb() from mas_dead_leaves() Suren Baghdasaryan 2023-02-16 5:17 ` Suren Baghdasaryan 2023-02-16 5:17 ` [PATCH v3 05/35] maple_tree: Fix write memory barrier of nodes once dead for RCU mode Suren Baghdasaryan 2023-02-16 5:17 ` Suren Baghdasaryan 2023-02-16 5:17 ` [PATCH v3 06/35] maple_tree: Add smp_rmb() to dead node detection Suren Baghdasaryan 2023-02-16 5:17 ` Suren Baghdasaryan 2023-02-16 5:17 ` [PATCH v3 07/35] maple_tree: Add RCU lock checking to rcu callback functions Suren Baghdasaryan 2023-02-16 5:17 ` Suren Baghdasaryan 2023-02-16 5:17 ` [PATCH v3 08/35] mm: Enable maple tree RCU mode by default Suren Baghdasaryan 2023-02-16 5:17 ` Suren Baghdasaryan 2023-02-16 5:17 ` [PATCH v3 09/35] mm: introduce CONFIG_PER_VMA_LOCK Suren Baghdasaryan 2023-02-16 5:17 ` Suren Baghdasaryan 2023-02-16 5:17 ` [PATCH v3 10/35] mm: rcu safe VMA freeing Suren Baghdasaryan 2023-02-16 5:17 ` Suren Baghdasaryan 2023-02-16 5:17 ` [PATCH v3 11/35] mm: move mmap_lock assert function definitions Suren Baghdasaryan 2023-02-16 5:17 ` Suren Baghdasaryan 2023-02-16 5:17 ` [PATCH v3 12/35] mm: add per-VMA lock and helper functions to control it Suren Baghdasaryan 2023-02-16 5:17 ` Suren Baghdasaryan 2023-02-16 5:17 ` [PATCH v3 13/35] mm: mark VMA as being written when changing vm_flags Suren Baghdasaryan 2023-02-16 5:17 ` Suren Baghdasaryan 2023-02-16 5:17 ` [PATCH v3 14/35] mm/mmap: move VMA locking before vma_adjust_trans_huge call Suren Baghdasaryan 2023-02-16 5:17 ` Suren Baghdasaryan 2023-02-16 5:17 ` [PATCH v3 15/35] mm/khugepaged: write-lock VMA while collapsing a huge page Suren Baghdasaryan 2023-02-16 5:17 ` Suren Baghdasaryan 2023-02-16 5:17 ` Suren Baghdasaryan [this message] 2023-02-16 5:17 ` [PATCH v3 16/35] mm/mmap: write-lock VMAs before merging, splitting or expanding them Suren Baghdasaryan 2023-02-23 14:51 ` Hyeonggon Yoo 2023-02-23 14:51 ` Hyeonggon Yoo 2023-02-23 14:59 ` Hyeonggon Yoo 2023-02-23 14:59 ` Hyeonggon Yoo 2023-02-23 17:46 ` Suren Baghdasaryan 2023-02-23 17:46 ` Suren Baghdasaryan 2023-02-16 5:17 ` [PATCH v3 17/35] mm/mmap: write-lock VMA before shrinking or expanding it Suren Baghdasaryan 2023-02-16 5:17 ` Suren Baghdasaryan 2023-02-23 20:20 ` Liam R. Howlett 2023-02-23 20:20 ` Liam R. Howlett 2023-02-23 20:20 ` Liam R. Howlett 2023-02-23 20:28 ` Liam R. Howlett 2023-02-23 20:28 ` Liam R. Howlett 2023-02-23 21:16 ` Suren Baghdasaryan 2023-02-24 1:46 ` Liam R. Howlett 2023-02-24 1:46 ` Liam R. Howlett 2023-02-24 1:46 ` Liam R. Howlett 2023-02-24 2:06 ` Suren Baghdasaryan 2023-02-24 16:14 ` Liam R. Howlett 2023-02-24 16:14 ` Liam R. Howlett 2023-02-24 16:14 ` Liam R. Howlett 2023-02-24 16:19 ` Suren Baghdasaryan 2023-02-27 17:33 ` Suren Baghdasaryan 2023-02-16 5:17 ` [PATCH v3 18/35] mm/mremap: write-lock VMA while remapping it to a new address range Suren Baghdasaryan 2023-02-16 5:17 ` Suren Baghdasaryan 2023-02-16 5:17 ` [PATCH v3 19/35] mm: write-lock VMAs before removing them from VMA tree Suren Baghdasaryan 2023-02-16 5:17 ` Suren Baghdasaryan 2023-02-16 5:17 ` [PATCH v3 20/35] mm: conditionally write-lock VMA in free_pgtables Suren Baghdasaryan 2023-02-16 5:17 ` Suren Baghdasaryan 2023-02-16 5:17 ` [PATCH v3 21/35] mm/mmap: write-lock adjacent VMAs if they can grow into unmapped area Suren Baghdasaryan 2023-02-16 5:17 ` Suren Baghdasaryan 2023-02-16 15:34 ` Liam R. Howlett 2023-02-16 15:34 ` Liam R. Howlett 2023-02-16 15:34 ` Liam R. Howlett 2023-02-16 19:36 ` Suren Baghdasaryan 2023-02-17 14:50 ` Liam R. Howlett 2023-02-17 14:50 ` Liam R. Howlett 2023-02-17 14:50 ` Liam R. Howlett 2023-02-17 15:54 ` Suren Baghdasaryan 2023-02-16 5:17 ` [PATCH v3 22/35] kernel/fork: assert no VMA readers during its destruction Suren Baghdasaryan 2023-02-16 5:17 ` Suren Baghdasaryan 2023-02-16 5:17 ` [PATCH v3 23/35] mm/mmap: prevent pagefault handler from racing with mmu_notifier registration Suren Baghdasaryan 2023-02-16 5:17 ` Suren Baghdasaryan 2023-02-23 20:06 ` Liam R. Howlett 2023-02-23 20:06 ` Liam R. Howlett 2023-02-23 20:06 ` Liam R. Howlett 2023-02-23 20:29 ` Suren Baghdasaryan 2023-02-16 5:17 ` [PATCH v3 24/35] mm: introduce vma detached flag Suren Baghdasaryan 2023-02-16 5:17 ` Suren Baghdasaryan 2023-02-23 20:08 ` Liam R. Howlett 2023-02-23 20:08 ` Liam R. Howlett 2023-02-23 20:08 ` Liam R. Howlett 2023-02-23 20:34 ` Suren Baghdasaryan 2023-02-16 5:17 ` [PATCH v3 25/35] mm: introduce lock_vma_under_rcu to be used from arch-specific code Suren Baghdasaryan 2023-02-16 5:17 ` Suren Baghdasaryan 2023-02-16 5:17 ` [PATCH v3 26/35] mm: fall back to mmap_lock if vma->anon_vma is not yet set Suren Baghdasaryan 2023-02-16 5:17 ` Suren Baghdasaryan 2023-02-16 15:44 ` Matthew Wilcox 2023-02-16 15:44 ` Matthew Wilcox 2023-02-16 15:44 ` Matthew Wilcox 2023-02-16 19:43 ` Suren Baghdasaryan 2023-02-16 19:43 ` Suren Baghdasaryan 2023-02-16 19:43 ` Suren Baghdasaryan 2023-02-17 2:14 ` Suren Baghdasaryan 2023-02-17 2:14 ` Suren Baghdasaryan 2023-02-17 2:14 ` Suren Baghdasaryan 2023-02-17 10:21 ` Hyeonggon Yoo 2023-02-17 10:21 ` Hyeonggon Yoo 2023-02-17 16:13 ` Suren Baghdasaryan 2023-02-17 16:13 ` Suren Baghdasaryan 2023-02-17 18:49 ` Hyeonggon Yoo 2023-02-17 18:49 ` Hyeonggon Yoo 2023-02-17 16:05 ` Matthew Wilcox 2023-02-17 16:05 ` Matthew Wilcox 2023-02-17 16:05 ` Matthew Wilcox 2023-02-17 16:10 ` Suren Baghdasaryan 2023-02-17 16:10 ` Suren Baghdasaryan 2023-02-17 16:10 ` Suren Baghdasaryan 2023-04-03 19:49 ` Matthew Wilcox 2023-04-03 19:49 ` Matthew Wilcox 2023-04-03 19:49 ` Matthew Wilcox 2023-02-16 5:17 ` [PATCH v3 27/35] mm: add FAULT_FLAG_VMA_LOCK flag Suren Baghdasaryan 2023-02-16 5:17 ` Suren Baghdasaryan 2023-02-16 5:17 ` [PATCH v3 28/35] mm: prevent do_swap_page from handling page faults under VMA lock Suren Baghdasaryan 2023-02-16 5:17 ` Suren Baghdasaryan 2023-02-16 5:17 ` [PATCH v3 29/35] mm: prevent userfaults to be handled under per-vma lock Suren Baghdasaryan 2023-02-16 5:17 ` Suren Baghdasaryan 2023-02-16 5:17 ` [PATCH v3 30/35] mm: introduce per-VMA lock statistics Suren Baghdasaryan 2023-02-16 5:17 ` Suren Baghdasaryan 2023-02-16 5:17 ` [PATCH v3 31/35] x86/mm: try VMA lock-based page fault handling first Suren Baghdasaryan 2023-02-16 5:17 ` Suren Baghdasaryan 2023-02-16 5:17 ` [PATCH v3 32/35] arm64/mm: " Suren Baghdasaryan 2023-02-16 5:17 ` Suren Baghdasaryan 2023-02-16 5:17 ` [PATCH v3 33/35] powerc/mm: " Suren Baghdasaryan 2023-02-16 5:17 ` Suren Baghdasaryan 2023-02-16 5:17 ` [PATCH v3 34/35] mm/mmap: free vm_area_struct without call_rcu in exit_mmap Suren Baghdasaryan 2023-02-16 5:17 ` Suren Baghdasaryan 2023-02-16 5:17 ` [PATCH v3 35/35] mm: separate vma->lock from vm_area_struct Suren Baghdasaryan 2023-02-16 5:17 ` Suren Baghdasaryan 2023-02-24 9:21 ` [PATCH v3 00/35] Per-VMA locks freak07 2023-02-24 9:21 ` freak07 2023-02-24 9:21 ` freak07 2023-02-27 16:50 ` Davidlohr Bueso 2023-02-27 16:50 ` Davidlohr Bueso 2023-02-27 16:50 ` Davidlohr Bueso 2023-02-27 17:22 ` Suren Baghdasaryan 2023-02-27 17:22 ` Suren Baghdasaryan
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20230216051750.3125598-17-surenb@google.com \ --to=surenb@google.com \ --cc=akpm@linux-foundation.org \ --cc=arjunroy@google.com \ --cc=axelrasmussen@google.com \ --cc=bigeasy@linutronix.de \ --cc=chriscli@google.com \ --cc=dave@stgolabs.net \ --cc=david@redhat.com \ --cc=dhowells@redhat.com \ --cc=edumazet@google.com \ --cc=gthelen@google.com \ --cc=gurua@google.com \ --cc=hannes@cmpxchg.org \ --cc=hughd@google.com \ --cc=jannh@google.com \ --cc=jglisse@google.com \ --cc=joelaf@google.com \ --cc=kent.overstreet@linux.dev \ --cc=kernel-team@android.com \ --cc=ldufour@linux.ibm.com \ --cc=leewalsh@google.com \ --cc=liam.howlett@oracle.com \ --cc=linux-arm-kernel@lists.infradead.org \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-mm@kvack.org \ --cc=linuxppc-dev@lists.ozlabs.org \ --cc=lstoakes@gmail.com \ --cc=luto@kernel.org \ --cc=mgorman@techsingularity.net \ --cc=mhocko@suse.com \ --cc=michalechner92@googlemail.com \ --cc=michel@lespinasse.org \ --cc=minchan@google.com \ --cc=mingo@redhat.com \ --cc=paulmck@kernel.org \ --cc=peterjung1337@gmail.com \ --cc=peterx@redhat.com \ --cc=peterz@infradead.org \ --cc=posk@google.com \ --cc=punit.agrawal@bytedance.com \ --cc=rientjes@google.com \ --cc=rppt@kernel.org \ --cc=shakeelb@google.com \ --cc=soheil@google.com \ --cc=songliubraving@fb.com \ --cc=tatashin@google.com \ --cc=vbabka@suse.cz \ --cc=will@kernel.org \ --cc=willy@infradead.org \ --cc=x86@kernel.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.