From: Jerome Glisse <jglisse@redhat.com>
To: Laurent Dufour <ldufour@linux.ibm.com>
Cc: jack@suse.cz, sergey.senozhatsky.work@gmail.com,
peterz@infradead.org, Will Deacon <will.deacon@arm.com>,
mhocko@kernel.org, linux-mm@kvack.org, paulus@samba.org,
Punit Agrawal <punitagrawal@gmail.com>,
hpa@zytor.com, Michel Lespinasse <walken@google.com>,
Alexei Starovoitov <alexei.starovoitov@gmail.com>,
Andrea Arcangeli <aarcange@redhat.com>,
ak@linux.intel.com, Minchan Kim <minchan@kernel.org>,
aneesh.kumar@linux.ibm.com, x86@kernel.org,
Matthew Wilcox <willy@infradead.org>,
Daniel Jordan <daniel.m.jordan@oracle.com>,
Ingo Molnar <mingo@redhat.com>,
David Rientjes <rientjes@google.com>,
paulmck@linux.vnet.ibm.com, Haiyan Song <haiyanx.song@intel.com>,
npiggin@gmail.com, sj38.park@gmail.com, dave@stgolabs.net,
kemi.wang@intel.com, kirill@shutemov.name,
Thomas Gleixner <tglx@linutronix.de>,
zhong jiang <zhongjiang@huawei.com>,
Ganesh Mahendran <opensource.ganesh@gmail.com>,
Yang Shi <yang.shi@linux.alibaba.com>,
Mike Rapoport <rppt@linux.ibm.com>,
linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org,
Sergey Senozhatsky <sergey.senozhatsky@gmail.com>,
vinayak menon <vinayakm.list@gmail.com>,
akpm@linux-foundation.org, Tim Chen <tim.c.chen@linux.intel.com>,
haren@linux.vnet.ibm.com
Subject: Re: [PATCH v12 07/31] mm: make pte_unmap_same compatible with SPF
Date: Thu, 18 Apr 2019 18:10:04 -0400 [thread overview]
Message-ID: <20190418221004.GG11645@redhat.com> (raw)
In-Reply-To: <20190416134522.17540-8-ldufour@linux.ibm.com>
On Tue, Apr 16, 2019 at 03:44:58PM +0200, Laurent Dufour wrote:
> pte_unmap_same() is making the assumption that the page table are still
> around because the mmap_sem is held.
> This is no more the case when running a speculative page fault and
> additional check must be made to ensure that the final page table are still
> there.
>
> This is now done by calling pte_spinlock() to check for the VMA's
> consistency while locking for the page tables.
>
> This is requiring passing a vm_fault structure to pte_unmap_same() which is
> containing all the needed parameters.
>
> As pte_spinlock() may fail in the case of a speculative page fault, if the
> VMA has been touched in our back, pte_unmap_same() should now return 3
> cases :
> 1. pte are the same (0)
> 2. pte are different (VM_FAULT_PTNOTSAME)
> 3. a VMA's changes has been detected (VM_FAULT_RETRY)
>
> The case 2 is handled by the introduction of a new VM_FAULT flag named
> VM_FAULT_PTNOTSAME which is then trapped in cow_user_page().
> If VM_FAULT_RETRY is returned, it is passed up to the callers to retry the
> page fault while holding the mmap_sem.
>
> Acked-by: David Rientjes <rientjes@google.com>
> Signed-off-by: Laurent Dufour <ldufour@linux.ibm.com>
Reviewed-by: Jérôme Glisse <jglisse@redhat.com>
> ---
> include/linux/mm_types.h | 6 +++++-
> mm/memory.c | 37 +++++++++++++++++++++++++++----------
> 2 files changed, 32 insertions(+), 11 deletions(-)
>
> diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
> index 8ec38b11b361..fd7d38ee2e33 100644
> --- a/include/linux/mm_types.h
> +++ b/include/linux/mm_types.h
> @@ -652,6 +652,8 @@ typedef __bitwise unsigned int vm_fault_t;
> * @VM_FAULT_NEEDDSYNC: ->fault did not modify page tables and needs
> * fsync() to complete (for synchronous page faults
> * in DAX)
> + * @VM_FAULT_PTNOTSAME Page table entries have changed during a
> + * speculative page fault handling.
> * @VM_FAULT_HINDEX_MASK: mask HINDEX value
> *
> */
> @@ -669,6 +671,7 @@ enum vm_fault_reason {
> VM_FAULT_FALLBACK = (__force vm_fault_t)0x000800,
> VM_FAULT_DONE_COW = (__force vm_fault_t)0x001000,
> VM_FAULT_NEEDDSYNC = (__force vm_fault_t)0x002000,
> + VM_FAULT_PTNOTSAME = (__force vm_fault_t)0x004000,
> VM_FAULT_HINDEX_MASK = (__force vm_fault_t)0x0f0000,
> };
>
> @@ -693,7 +696,8 @@ enum vm_fault_reason {
> { VM_FAULT_RETRY, "RETRY" }, \
> { VM_FAULT_FALLBACK, "FALLBACK" }, \
> { VM_FAULT_DONE_COW, "DONE_COW" }, \
> - { VM_FAULT_NEEDDSYNC, "NEEDDSYNC" }
> + { VM_FAULT_NEEDDSYNC, "NEEDDSYNC" }, \
> + { VM_FAULT_PTNOTSAME, "PTNOTSAME" }
>
> struct vm_special_mapping {
> const char *name; /* The name, e.g. "[vdso]". */
> diff --git a/mm/memory.c b/mm/memory.c
> index 221ccdf34991..d5bebca47d98 100644
> --- a/mm/memory.c
> +++ b/mm/memory.c
> @@ -2094,21 +2094,29 @@ static inline bool pte_map_lock(struct vm_fault *vmf)
> * parts, do_swap_page must check under lock before unmapping the pte and
> * proceeding (but do_wp_page is only called after already making such a check;
> * and do_anonymous_page can safely check later on).
> + *
> + * pte_unmap_same() returns:
> + * 0 if the PTE are the same
> + * VM_FAULT_PTNOTSAME if the PTE are different
> + * VM_FAULT_RETRY if the VMA has changed in our back during
> + * a speculative page fault handling.
> */
> -static inline int pte_unmap_same(struct mm_struct *mm, pmd_t *pmd,
> - pte_t *page_table, pte_t orig_pte)
> +static inline vm_fault_t pte_unmap_same(struct vm_fault *vmf)
> {
> - int same = 1;
> + int ret = 0;
> +
> #if defined(CONFIG_SMP) || defined(CONFIG_PREEMPT)
> if (sizeof(pte_t) > sizeof(unsigned long)) {
> - spinlock_t *ptl = pte_lockptr(mm, pmd);
> - spin_lock(ptl);
> - same = pte_same(*page_table, orig_pte);
> - spin_unlock(ptl);
> + if (pte_spinlock(vmf)) {
> + if (!pte_same(*vmf->pte, vmf->orig_pte))
> + ret = VM_FAULT_PTNOTSAME;
> + spin_unlock(vmf->ptl);
> + } else
> + ret = VM_FAULT_RETRY;
> }
> #endif
> - pte_unmap(page_table);
> - return same;
> + pte_unmap(vmf->pte);
> + return ret;
> }
>
> static inline void cow_user_page(struct page *dst, struct page *src, unsigned long va, struct vm_area_struct *vma)
> @@ -2714,8 +2722,17 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
> int exclusive = 0;
> vm_fault_t ret = 0;
>
> - if (!pte_unmap_same(vma->vm_mm, vmf->pmd, vmf->pte, vmf->orig_pte))
> + ret = pte_unmap_same(vmf);
> + if (ret) {
> + /*
> + * If pte != orig_pte, this means another thread did the
> + * swap operation in our back.
> + * So nothing else to do.
> + */
> + if (ret == VM_FAULT_PTNOTSAME)
> + ret = 0;
> goto out;
> + }
>
> entry = pte_to_swp_entry(vmf->orig_pte);
> if (unlikely(non_swap_entry(entry))) {
> --
> 2.21.0
>
next prev parent reply other threads:[~2019-04-18 22:12 UTC|newest]
Thread overview: 98+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-04-16 13:44 [PATCH v12 00/31] Speculative page faults Laurent Dufour
2019-04-16 13:44 ` [PATCH v12 01/31] mm: introduce CONFIG_SPECULATIVE_PAGE_FAULT Laurent Dufour
2019-04-18 21:47 ` Jerome Glisse
2019-04-23 15:21 ` Laurent Dufour
2019-04-16 13:44 ` [PATCH v12 02/31] x86/mm: define ARCH_SUPPORTS_SPECULATIVE_PAGE_FAULT Laurent Dufour
2019-04-18 21:48 ` Jerome Glisse
2019-04-16 13:44 ` [PATCH v12 03/31] powerpc/mm: set ARCH_SUPPORTS_SPECULATIVE_PAGE_FAULT Laurent Dufour
2019-04-18 21:49 ` Jerome Glisse
2019-04-16 13:44 ` [PATCH v12 04/31] arm64/mm: define ARCH_SUPPORTS_SPECULATIVE_PAGE_FAULT Laurent Dufour
2019-04-16 14:27 ` Mark Rutland
2019-04-16 14:31 ` Laurent Dufour
2019-04-16 14:41 ` Mark Rutland
2019-04-18 21:51 ` Jerome Glisse
2019-04-23 15:36 ` Laurent Dufour
2019-04-23 16:19 ` Mark Rutland
2019-04-24 10:34 ` Laurent Dufour
2019-04-16 13:44 ` [PATCH v12 05/31] mm: prepare for FAULT_FLAG_SPECULATIVE Laurent Dufour
2019-04-18 22:04 ` Jerome Glisse
2019-04-23 15:45 ` Laurent Dufour
2019-04-16 13:44 ` [PATCH v12 06/31] mm: introduce pte_spinlock " Laurent Dufour
2019-04-18 22:05 ` Jerome Glisse
2019-04-16 13:44 ` [PATCH v12 07/31] mm: make pte_unmap_same compatible with SPF Laurent Dufour
2019-04-18 22:10 ` Jerome Glisse [this message]
2019-04-23 15:43 ` Matthew Wilcox
2019-04-23 15:47 ` Laurent Dufour
2019-04-16 13:44 ` [PATCH v12 08/31] mm: introduce INIT_VMA() Laurent Dufour
2019-04-18 22:22 ` Jerome Glisse
2019-04-16 13:45 ` [PATCH v12 09/31] mm: VMA sequence count Laurent Dufour
2019-04-18 22:48 ` Jerome Glisse
2019-04-19 15:45 ` Laurent Dufour
2019-04-22 15:51 ` Jerome Glisse
2019-04-16 13:45 ` [PATCH v12 10/31] mm: protect VMA modifications using " Laurent Dufour
2019-04-22 19:43 ` Jerome Glisse
2019-04-16 13:45 ` [PATCH v12 11/31] mm: protect mremap() against SPF hanlder Laurent Dufour
2019-04-22 19:51 ` Jerome Glisse
2019-04-23 15:51 ` Laurent Dufour
2019-04-16 13:45 ` [PATCH v12 12/31] mm: protect SPF handler against anon_vma changes Laurent Dufour
2019-04-22 19:53 ` Jerome Glisse
2019-04-16 13:45 ` [PATCH v12 13/31] mm: cache some VMA fields in the vm_fault structure Laurent Dufour
2019-04-22 20:06 ` Jerome Glisse
2019-04-16 13:45 ` [PATCH v12 14/31] mm/migrate: Pass vm_fault pointer to migrate_misplaced_page() Laurent Dufour
2019-04-22 20:09 ` Jerome Glisse
2019-04-16 13:45 ` [PATCH v12 15/31] mm: introduce __lru_cache_add_active_or_unevictable Laurent Dufour
2019-04-22 20:11 ` Jerome Glisse
2019-04-16 13:45 ` [PATCH v12 16/31] mm: introduce __vm_normal_page() Laurent Dufour
2019-04-22 20:15 ` Jerome Glisse
2019-04-16 13:45 ` [PATCH v12 17/31] mm: introduce __page_add_new_anon_rmap() Laurent Dufour
2019-04-22 20:18 ` Jerome Glisse
2019-04-16 13:45 ` [PATCH v12 18/31] mm: protect against PTE changes done by dup_mmap() Laurent Dufour
2019-04-22 20:32 ` Jerome Glisse
2019-04-24 10:33 ` Laurent Dufour
2019-04-16 13:45 ` [PATCH v12 19/31] mm: protect the RB tree with a sequence lock Laurent Dufour
2019-04-22 20:33 ` Jerome Glisse
2019-04-16 13:45 ` [PATCH v12 20/31] mm: introduce vma reference counter Laurent Dufour
2019-04-22 20:36 ` Jerome Glisse
2019-04-24 14:26 ` Laurent Dufour
2019-04-16 13:45 ` [PATCH v12 21/31] mm: Introduce find_vma_rcu() Laurent Dufour
2019-04-22 20:57 ` Jerome Glisse
2019-04-24 14:39 ` Laurent Dufour
2019-04-23 9:27 ` Peter Zijlstra
2019-04-23 18:13 ` Davidlohr Bueso
2019-04-24 7:57 ` Laurent Dufour
2019-04-16 13:45 ` [PATCH v12 22/31] mm: provide speculative fault infrastructure Laurent Dufour
2019-04-22 21:26 ` Jerome Glisse
2019-04-24 14:56 ` Laurent Dufour
2019-04-24 15:13 ` Jerome Glisse
2019-04-16 13:45 ` [PATCH v12 23/31] mm: don't do swap readahead during speculative page fault Laurent Dufour
2019-04-22 21:36 ` Jerome Glisse
2019-04-24 14:57 ` Laurent Dufour
2019-04-16 13:45 ` [PATCH v12 24/31] mm: adding speculative page fault failure trace events Laurent Dufour
2019-04-16 13:45 ` [PATCH v12 25/31] perf: add a speculative page fault sw event Laurent Dufour
2019-04-16 13:45 ` [PATCH v12 26/31] perf tools: add support for the SPF perf event Laurent Dufour
2019-04-16 13:45 ` [PATCH v12 27/31] mm: add speculative page fault vmstats Laurent Dufour
2019-04-16 13:45 ` [PATCH v12 28/31] x86/mm: add speculative pagefault handling Laurent Dufour
2019-04-16 13:45 ` [PATCH v12 29/31] powerpc/mm: add speculative page fault Laurent Dufour
2019-04-16 13:45 ` [PATCH v12 30/31] arm64/mm: " Laurent Dufour
2019-04-16 13:45 ` [PATCH v12 31/31] mm: Add a speculative page fault switch in sysctl Laurent Dufour
2019-04-22 21:29 ` [PATCH v12 00/31] Speculative page faults Michel Lespinasse
2019-04-23 9:38 ` Peter Zijlstra
2019-04-24 7:33 ` Laurent Dufour
2019-04-27 1:53 ` Michel Lespinasse
2019-04-23 10:47 ` Michal Hocko
2019-04-23 12:41 ` Matthew Wilcox
2019-04-23 12:48 ` Peter Zijlstra
2019-04-23 13:42 ` Michal Hocko
2019-04-24 18:01 ` Laurent Dufour
2019-04-27 6:00 ` Michel Lespinasse
2019-04-23 11:35 ` Anshuman Khandual
2019-06-06 6:51 ` Haiyan Song
2019-06-14 8:37 ` Laurent Dufour
2019-06-14 8:44 ` Laurent Dufour
2019-06-20 8:19 ` Haiyan Song
2020-07-06 9:25 ` Chinwen Chang
2020-07-06 12:27 ` Laurent Dufour
2020-07-07 5:31 ` Chinwen Chang
2020-12-14 2:03 ` Joel Fernandes
2020-12-14 9:36 ` Laurent Dufour
2020-12-14 18:10 ` Joel Fernandes
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20190418221004.GG11645@redhat.com \
--to=jglisse@redhat.com \
--cc=aarcange@redhat.com \
--cc=ak@linux.intel.com \
--cc=akpm@linux-foundation.org \
--cc=alexei.starovoitov@gmail.com \
--cc=aneesh.kumar@linux.ibm.com \
--cc=daniel.m.jordan@oracle.com \
--cc=dave@stgolabs.net \
--cc=haiyanx.song@intel.com \
--cc=haren@linux.vnet.ibm.com \
--cc=hpa@zytor.com \
--cc=jack@suse.cz \
--cc=kemi.wang@intel.com \
--cc=kirill@shutemov.name \
--cc=ldufour@linux.ibm.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=linuxppc-dev@lists.ozlabs.org \
--cc=mhocko@kernel.org \
--cc=minchan@kernel.org \
--cc=mingo@redhat.com \
--cc=npiggin@gmail.com \
--cc=opensource.ganesh@gmail.com \
--cc=paulmck@linux.vnet.ibm.com \
--cc=paulus@samba.org \
--cc=peterz@infradead.org \
--cc=punitagrawal@gmail.com \
--cc=rientjes@google.com \
--cc=rppt@linux.ibm.com \
--cc=sergey.senozhatsky.work@gmail.com \
--cc=sergey.senozhatsky@gmail.com \
--cc=sj38.park@gmail.com \
--cc=tglx@linutronix.de \
--cc=tim.c.chen@linux.intel.com \
--cc=vinayakm.list@gmail.com \
--cc=walken@google.com \
--cc=will.deacon@arm.com \
--cc=willy@infradead.org \
--cc=x86@kernel.org \
--cc=yang.shi@linux.alibaba.com \
--cc=zhongjiang@huawei.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).