From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6580CC433B4 for ; Wed, 7 Apr 2021 01:56:42 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 247D46128D for ; Wed, 7 Apr 2021 01:56:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233464AbhDGB4t (ORCPT ); Tue, 6 Apr 2021 21:56:49 -0400 Received: from server.lespinasse.org ([63.205.204.226]:33301 "EHLO server.lespinasse.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229751AbhDGB4r (ORCPT ); Tue, 6 Apr 2021 21:56:47 -0400 DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-ed; t=1617759903; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=XkD4f0ULJymIOIeAqVxcBhuCK1FNqNRxb6Ygkc1ApCs=; b=T+2gjG7c+P1JGIgyykqnKaQiD2Wtt2L+tO+pRnvCt2JO1FKjXP26ZCM6hHIjsZg0F91Pv IUbkovCOPxRyi8dAA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-rsa; t=1617759903; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=XkD4f0ULJymIOIeAqVxcBhuCK1FNqNRxb6Ygkc1ApCs=; b=RTBwrQVEuGgTfXMpMfbrh/D5mOUGsMVm80V0u7rXaSdXDkPKYPpLKgLTiCRWVRop96qn6 0H69mSmJu7OkmjqPwNX7g+7lzVmBTBOnkqtihAaT+U/O2gxSb293SkAW7EIfEHbmGHMYZWk pOscXZjW7Z0CSgRkvL7L15AFZtvdGsJOlu2QZxqeLxSHccd/OsdnNeVdL56bmw0Rd1+uEX3 VukmTQUvytixFC1AIzrmU1IDhnsHFdOkwUHs5+5/3GbtMI8p+2GamYUAQkZO2nK072EtqZp iyjY45FT2zBP25QHQ+U8WOHV240vWPDjZQGCB3bXmbBMsDK2fiPRHyEjp86w== Received: from zeus.lespinasse.org (zeus.lespinasse.org [IPv6:fd00::150:0]) by server.lespinasse.org (Postfix) with ESMTPS id 35EA7160531; Tue, 6 Apr 2021 18:45:03 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id 26A2B19F31D; Tue, 6 Apr 2021 18:45:03 -0700 (PDT) From: Michel Lespinasse To: Linux-MM Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Rom Lemarchand , Linux-Kernel , Michel Lespinasse Subject: [RFC PATCH 28/37] mm: implement speculative handling in filemap_map_pages() Date: Tue, 6 Apr 2021 18:44:53 -0700 Message-Id: <20210407014502.24091-29-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210407014502.24091-1-michel@lespinasse.org> References: <20210407014502.24091-1-michel@lespinasse.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org In the speculative case, we know the page table already exists, and it must be locked with pte_map_lock(). In the case where no page is found for the given address, return VM_FAULT_RETRY which will abort the fault before we get into the vm_ops->fault() callback. This is fine because if filemap_map_pages does not find the page in page cache, vm_ops->fault() will not either. Initialize addr and last_pgoff to correspond to the pte at the original fault address (which was mapped with pte_map_lock()), rather than the pte at start_pgoff. The choice of initial values doesn't matter as they will all be adjusted together before use, so they just need to be consistent with each other, and using the original fault address and pte allows us to reuse pte_map_lock() without any changes to it. Signed-off-by: Michel Lespinasse --- mm/filemap.c | 27 ++++++++++++++++----------- 1 file changed, 16 insertions(+), 11 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 6e8505fe5df9..d496771749e6 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -3136,25 +3136,31 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf, struct vm_area_struct *vma = vmf->vma; struct file *file = vma->vm_file; struct address_space *mapping = file->f_mapping; - pgoff_t last_pgoff = start_pgoff; + pgoff_t last_pgoff; unsigned long addr; XA_STATE(xas, &mapping->i_pages, start_pgoff); struct page *head, *page; unsigned int mmap_miss = READ_ONCE(file->f_ra.mmap_miss); - vm_fault_t ret = 0; + vm_fault_t ret = (vmf->flags & FAULT_FLAG_SPECULATIVE) ? + VM_FAULT_RETRY : 0; - rcu_read_lock(); + /* filemap_map_pages() is called within an rcu read lock already. */ head = first_map_page(mapping, &xas, end_pgoff); if (!head) - goto out; + return ret; - if (filemap_map_pmd(vmf, head)) { - ret = VM_FAULT_NOPAGE; - goto out; + if (!(vmf->flags & FAULT_FLAG_SPECULATIVE) && + filemap_map_pmd(vmf, head)) + return VM_FAULT_NOPAGE; + + if (!pte_map_lock(vmf)) { + unlock_page(head); + put_page(head); + return VM_FAULT_RETRY; } + addr = vmf->address; + last_pgoff = vmf->pgoff; - addr = vma->vm_start + ((start_pgoff - vma->vm_pgoff) << PAGE_SHIFT); - vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, addr, &vmf->ptl); do { page = find_subpage(head, xas.xa_index); if (PageHWPoison(page)) @@ -3184,8 +3190,7 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf, put_page(head); } while ((head = next_map_page(mapping, &xas, end_pgoff)) != NULL); pte_unmap_unlock(vmf->pte, vmf->ptl); -out: - rcu_read_unlock(); + vmf->pte = NULL; WRITE_ONCE(file->f_ra.mmap_miss, mmap_miss); return ret; } -- 2.20.1