From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A942AC43461 for ; Wed, 7 Apr 2021 01:45:59 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 4D03C610CE for ; Wed, 7 Apr 2021 01:45:59 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4D03C610CE Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 667EB8E0012; Tue, 6 Apr 2021 21:45:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AE7AD6B008A; Tue, 6 Apr 2021 21:45:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DD43F8E0007; Tue, 6 Apr 2021 21:45:09 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0073.hostedemail.com [216.40.44.73]) by kanga.kvack.org (Postfix) with ESMTP id 976558D0005 for ; Tue, 6 Apr 2021 21:45:08 -0400 (EDT) Received: from smtpin19.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 52FAC180ACEE6 for ; Wed, 7 Apr 2021 01:45:08 +0000 (UTC) X-FDA: 78003877896.19.6EBBE39 Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf26.hostedemail.com (Postfix) with ESMTP id 8CDCC40002C1 for ; Wed, 7 Apr 2021 01:45:05 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-ed; t=1617759902; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=8COoG1kyMAyzmcjmtCVhUuZfAMmUT556bb2lAEkl2Jk=; b=PHxlIUq8Gez4GGg1T1vZB3f/iGsF8+rBmKPA5lGRVNAhdo4q0AcUCTl4jG0ud1ymS98BV pXuyifekidg4jc1BQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-rsa; t=1617759902; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=8COoG1kyMAyzmcjmtCVhUuZfAMmUT556bb2lAEkl2Jk=; b=Xj0qGgONLfNk5zEaM5IEBqutABXQbilGjAJuQCGy/NJElYzNQNWVz/qj9s1C6jR2zMKsp CjgSDeSQCsFCM+kAwSicmrOf/jMLccvhMVOhjghOl1FsBmbeoi8ks/+cPqSCRaAA0S5gXBo wH6j2k0bx/8iypailFquYrsYVPFz097+Hi4V3IjA75FYXoRM5H3/nt2VmtdgV28sESHbThT nPem9Matb4e4goClyC4hD+hIaDWZ5mKOjFOWWWxDQ0KasRhTnPowHxV9A95siPdM+tgbJcM rKG/7yOOiDxRufdYxGz6E0iQRg9rMSq43GJ4K/9MnRhNIYT5b2usXjV3CPcA== Received: from zeus.lespinasse.org (zeus.lespinasse.org [10.0.0.150]) by server.lespinasse.org (Postfix) with ESMTPS id E7B4016031A; Tue, 6 Apr 2021 18:45:02 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id D9CBE19F31D; Tue, 6 Apr 2021 18:45:02 -0700 (PDT) From: Michel Lespinasse To: Linux-MM Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Rom Lemarchand , Linux-Kernel , Michel Lespinasse Subject: [RFC PATCH 19/37] mm: implement speculative handling in wp_page_copy() Date: Tue, 6 Apr 2021 18:44:44 -0700 Message-Id: <20210407014502.24091-20-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210407014502.24091-1-michel@lespinasse.org> References: <20210407014502.24091-1-michel@lespinasse.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 8CDCC40002C1 X-Stat-Signature: 7x6fys8xdjyrt4xmxh9ia8fc5zq4foda X-Rspamd-Server: rspam02 Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf26; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617759905-508584 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Change wp_page_copy() to handle the speculative case. This involves aborting speculative faults if they have to allocate an anon_vma, and using pte_map_lock() instead of pte_offset_map_lock() to complete the page fault. Also change call sites to clear vmf->pte after unmapping the page table, in order to satisfy pte_map_lock()'s preconditions. Signed-off-by: Michel Lespinasse --- mm/memory.c | 31 ++++++++++++++++++++++--------- 1 file changed, 22 insertions(+), 9 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index eea72bd78d06..547d9d0ee962 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -2911,20 +2911,27 @@ static vm_fault_t wp_page_copy(struct vm_fault *v= mf) pte_t entry; int page_copied =3D 0; struct mmu_notifier_range range; + vm_fault_t ret =3D VM_FAULT_OOM; =20 - if (unlikely(anon_vma_prepare(vma))) - goto oom; + if (unlikely(!vma->anon_vma)) { + if (vmf->flags & FAULT_FLAG_SPECULATIVE) { + ret =3D VM_FAULT_RETRY; + goto out; + } + if (__anon_vma_prepare(vma)) + goto out; + } =20 if (is_zero_pfn(pte_pfn(vmf->orig_pte))) { new_page =3D alloc_zeroed_user_highpage_movable(vma, vmf->address); if (!new_page) - goto oom; + goto out; } else { new_page =3D alloc_page_vma(GFP_HIGHUSER_MOVABLE, vma, vmf->address); if (!new_page) - goto oom; + goto out; =20 if (!cow_user_page(new_page, old_page, vmf)) { /* @@ -2941,7 +2948,7 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf= ) } =20 if (mem_cgroup_charge(new_page, mm, GFP_KERNEL)) - goto oom_free_new; + goto out_free_new; cgroup_throttle_swaprate(new_page, GFP_KERNEL); =20 __SetPageUptodate(new_page); @@ -2954,7 +2961,11 @@ static vm_fault_t wp_page_copy(struct vm_fault *vm= f) /* * Re-check the pte - we dropped the lock */ - vmf->pte =3D pte_offset_map_lock(mm, vmf->pmd, vmf->address, &vmf->ptl)= ; + if (!pte_map_lock(vmf)) { + ret =3D VM_FAULT_RETRY; + /* put_page() will uncharge the page */ + goto out_free_new; + } if (likely(pte_same(*vmf->pte, vmf->orig_pte))) { if (old_page) { if (!PageAnon(old_page)) { @@ -3042,12 +3053,12 @@ static vm_fault_t wp_page_copy(struct vm_fault *v= mf) put_page(old_page); } return page_copied ? VM_FAULT_WRITE : 0; -oom_free_new: +out_free_new: put_page(new_page); -oom: +out: if (old_page) put_page(old_page); - return VM_FAULT_OOM; + return ret; } =20 /** @@ -3190,6 +3201,7 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf) return wp_pfn_shared(vmf); =20 pte_unmap_unlock(vmf->pte, vmf->ptl); + vmf->pte =3D NULL; return wp_page_copy(vmf); } =20 @@ -3228,6 +3240,7 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf) get_page(vmf->page); =20 pte_unmap_unlock(vmf->pte, vmf->ptl); + vmf->pte =3D NULL; return wp_page_copy(vmf); } =20 --=20 2.20.1