From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 542A6C433DB for ; Fri, 15 Jan 2021 17:10:26 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 05736238EE for ; Fri, 15 Jan 2021 17:10:26 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 05736238EE Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id EAA7A8D01B5; Fri, 15 Jan 2021 12:10:08 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E08B08D01B2; Fri, 15 Jan 2021 12:10:08 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C84288D01B5; Fri, 15 Jan 2021 12:10:08 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0092.hostedemail.com [216.40.44.92]) by kanga.kvack.org (Postfix) with ESMTP id B1EDB8D01B2 for ; Fri, 15 Jan 2021 12:10:08 -0500 (EST) Received: from smtpin13.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id E87C08E75 for ; Fri, 15 Jan 2021 17:10:07 +0000 (UTC) X-FDA: 77708647254.13.spoon82_5d175d527531 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin13.hostedemail.com (Postfix) with ESMTP id C0D49181D02AD for ; Fri, 15 Jan 2021 17:10:07 +0000 (UTC) X-HE-Tag: spoon82_5d175d527531 X-Filterd-Recvd-Size: 11949 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by imf01.hostedemail.com (Postfix) with ESMTP for ; Fri, 15 Jan 2021 17:10:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1610730606; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=GCz2Z9Zgad3gqL1m2VPHTbAcPly4nG/hCXs5cYStaH0=; b=UAbfGnIhThxCZVE3B4ufHxJEBiyKHpsCzEkX+D253eFEJ++qubPA/uHkT1Uxqu/jKubfnY WaXMCid+87+7bzph7qI/BDo0HYlSojuhiOgfl4aPIgwXHav9V+zRAjnrvsnyk1SDY9CNFE Z2bg86yR6IlfO7RTS9YxDCT3Cl98Pq4= Received: from mail-qk1-f200.google.com (mail-qk1-f200.google.com [209.85.222.200]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-443-Fv_ftd7iMeavvr_Hf3ypag-1; Fri, 15 Jan 2021 12:10:05 -0500 X-MC-Unique: Fv_ftd7iMeavvr_Hf3ypag-1 Received: by mail-qk1-f200.google.com with SMTP id l138so8617689qke.4 for ; Fri, 15 Jan 2021 09:10:04 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=GCz2Z9Zgad3gqL1m2VPHTbAcPly4nG/hCXs5cYStaH0=; b=UzCrN4OsuzQoflptN6ghdH6fm6MvMHgbO8Jue3tBx4E/HiObf/wL6me72VrgY0rq+2 fKQNqOTvQGFIe5bkBIQABg4aEnIeEWcPCgb1OYWona/eIaKNcwwCOp7tqTht9vVeyy8j 7lgvpRHeJZOu8//OA8iGSuX8p7AFE19neBPRUF6EJRoWgesafdjTzIfoeqZ0QMlDmhRf 0N0MODsF1LxuE7wc8i52mHz5ARGvddLQWqRVkL+nBgqG8KWDqwkyUr1Vob2z1wchnivg G6z3n0TFCXGGErfapVZAmtb7aOy6HpQl1Lk1oTaGVNqqdssll4g64trjDiBjd7ae35Gy hzWg== X-Gm-Message-State: AOAM531kiJ2L48+mEd733+swmOQ6XJc7mNto8a9Y3SbaKU9RfApWvTbq RB4ZFtX5RissOiDuGVQcfAnixeotw3AdA0hAULU9sCWwMhyLFsP5GgNLaq6wrqcjPRo7aaMEJhb 2TUrXjur/WL4= X-Received: by 2002:a37:c0b:: with SMTP id 11mr13568851qkm.32.1610730604178; Fri, 15 Jan 2021 09:10:04 -0800 (PST) X-Google-Smtp-Source: ABdhPJxR8gnzle+XvdiEUHLdFl5e93T23UAPGki6VkBtygLlYVM2vhnQhPTVW8Iyu6c62gDqNjWM3w== X-Received: by 2002:a37:c0b:: with SMTP id 11mr13568805qkm.32.1610730603894; Fri, 15 Jan 2021 09:10:03 -0800 (PST) Received: from localhost.localdomain ([142.126.83.202]) by smtp.gmail.com with ESMTPSA id d123sm5187840qke.95.2021.01.15.09.10.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 15 Jan 2021 09:10:03 -0800 (PST) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Mike Rapoport , Mike Kravetz , peterx@redhat.com, Jerome Glisse , "Kirill A . Shutemov" , Hugh Dickins , Axel Rasmussen , Matthew Wilcox , Andrew Morton , Andrea Arcangeli , Nadav Amit Subject: [PATCH RFC 28/30] hugetlb/userfaultfd: Only drop uffd-wp special pte if required Date: Fri, 15 Jan 2021 12:09:05 -0500 Message-Id: <20210115170907.24498-29-peterx@redhat.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210115170907.24498-1-peterx@redhat.com> References: <20210115170907.24498-1-peterx@redhat.com> MIME-Version: 1.0 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=peterx@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Just like what we've done with shmem uffd-wp special ptes, we shouldn't d= rop uffd-wp special swap pte for hugetlb too, only if we're going to unmap th= e whole vma, or we're punching a hole with safe locks held. For example, remove_inode_hugepages() is safe to drop uffd-wp ptes, becau= se it has taken hugetlb fault mutex so that no concurrent page fault would trig= ger. While the call to hugetlb_vmdelete_list() in hugetlbfs_punch_hole() is no= t safe. That's why the previous call will be with ZAP_FLAG_DROP_FILE_UFFD_= WP, while the latter one won't be able to. Signed-off-by: Peter Xu --- fs/hugetlbfs/inode.c | 15 +++++++++------ include/linux/hugetlb.h | 13 ++++++++----- mm/hugetlb.c | 27 +++++++++++++++++++++------ mm/memory.c | 5 ++++- 4 files changed, 42 insertions(+), 18 deletions(-) diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c index b5c109703daa..f9ff2ba5e47b 100644 --- a/fs/hugetlbfs/inode.c +++ b/fs/hugetlbfs/inode.c @@ -399,7 +399,8 @@ static void remove_huge_page(struct page *page) } =20 static void -hugetlb_vmdelete_list(struct rb_root_cached *root, pgoff_t start, pgoff_= t end) +hugetlb_vmdelete_list(struct rb_root_cached *root, pgoff_t start, pgoff_= t end, + unsigned long zap_flags) { struct vm_area_struct *vma; =20 @@ -432,7 +433,7 @@ hugetlb_vmdelete_list(struct rb_root_cached *root, pg= off_t start, pgoff_t end) } =20 unmap_hugepage_range(vma, vma->vm_start + v_offset, v_end, - NULL); + NULL, zap_flags); } } =20 @@ -513,7 +514,8 @@ static void remove_inode_hugepages(struct inode *inod= e, loff_t lstart, mutex_lock(&hugetlb_fault_mutex_table[hash]); hugetlb_vmdelete_list(&mapping->i_mmap, index * pages_per_huge_page(h), - (index + 1) * pages_per_huge_page(h)); + (index + 1) * pages_per_huge_page(h), + ZAP_FLAG_DROP_FILE_UFFD_WP); i_mmap_unlock_write(mapping); } =20 @@ -579,7 +581,8 @@ static int hugetlb_vmtruncate(struct inode *inode, lo= ff_t offset) i_mmap_lock_write(mapping); i_size_write(inode, offset); if (!RB_EMPTY_ROOT(&mapping->i_mmap.rb_root)) - hugetlb_vmdelete_list(&mapping->i_mmap, pgoff, 0); + hugetlb_vmdelete_list(&mapping->i_mmap, pgoff, 0, + ZAP_FLAG_DROP_FILE_UFFD_WP); i_mmap_unlock_write(mapping); remove_inode_hugepages(inode, offset, LLONG_MAX); return 0; @@ -613,8 +616,8 @@ static long hugetlbfs_punch_hole(struct inode *inode,= loff_t offset, loff_t len) i_mmap_lock_write(mapping); if (!RB_EMPTY_ROOT(&mapping->i_mmap.rb_root)) hugetlb_vmdelete_list(&mapping->i_mmap, - hole_start >> PAGE_SHIFT, - hole_end >> PAGE_SHIFT); + hole_start >> PAGE_SHIFT, + hole_end >> PAGE_SHIFT, 0); i_mmap_unlock_write(mapping); remove_inode_hugepages(inode, hole_start, hole_end); inode_unlock(inode); diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 8841d118f45b..93f3c46439b2 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -121,14 +121,15 @@ long follow_hugetlb_page(struct mm_struct *, struct= vm_area_struct *, unsigned long *, unsigned long *, long, unsigned int, int *); void unmap_hugepage_range(struct vm_area_struct *, - unsigned long, unsigned long, struct page *); + unsigned long, unsigned long, struct page *, + unsigned long); void __unmap_hugepage_range_final(struct mmu_gather *tlb, struct vm_area_struct *vma, unsigned long start, unsigned long end, - struct page *ref_page); + struct page *ref_page, unsigned long zap_flags); void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struc= t *vma, unsigned long start, unsigned long end, - struct page *ref_page); + struct page *ref_page, unsigned long zap_flags); void hugetlb_report_meminfo(struct seq_file *); int hugetlb_report_node_meminfo(char *buf, int len, int nid); void hugetlb_show_meminfo(void); @@ -353,14 +354,16 @@ static inline unsigned long hugetlb_change_protecti= on( =20 static inline void __unmap_hugepage_range_final(struct mmu_gather *tlb, struct vm_area_struct *vma, unsigned long start, - unsigned long end, struct page *ref_page) + unsigned long end, struct page *ref_page, + unsigned long zap_flags) { BUG(); } =20 static inline void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct *vma, unsigned long start, - unsigned long end, struct page *ref_page) + unsigned long end, struct page *ref_page, + unsigned long zap_flags) { BUG(); } diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 7959fb4b1633..731a26617673 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3864,7 +3864,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, = struct mm_struct *src, =20 void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struc= t *vma, unsigned long start, unsigned long end, - struct page *ref_page) + struct page *ref_page, unsigned long zap_flags) { struct mm_struct *mm =3D vma->vm_mm; unsigned long address; @@ -3916,6 +3916,19 @@ void __unmap_hugepage_range(struct mmu_gather *tlb= , struct vm_area_struct *vma, continue; } =20 + if (unlikely(is_swap_special_pte(pte))) { + WARN_ON_ONCE(!pte_swp_uffd_wp_special(pte)); + /* + * Only drop the special swap uffd-wp pte if + * e.g. unmapping a vma or punching a hole (with proper + * lock held so that concurrent page fault won't happen). + */ + if (zap_flags & ZAP_FLAG_DROP_FILE_UFFD_WP) + huge_pte_clear(mm, address, ptep, sz); + spin_unlock(ptl); + continue; + } + /* * Migrating hugepage or HWPoisoned hugepage is already * unmapped and its refcount is dropped, so just clear pte here. @@ -3967,9 +3980,10 @@ void __unmap_hugepage_range(struct mmu_gather *tlb= , struct vm_area_struct *vma, =20 void __unmap_hugepage_range_final(struct mmu_gather *tlb, struct vm_area_struct *vma, unsigned long start, - unsigned long end, struct page *ref_page) + unsigned long end, struct page *ref_page, + unsigned long zap_flags) { - __unmap_hugepage_range(tlb, vma, start, end, ref_page); + __unmap_hugepage_range(tlb, vma, start, end, ref_page, zap_flags); =20 /* * Clear this flag so that x86's huge_pmd_share page_table_shareable @@ -3985,7 +3999,8 @@ void __unmap_hugepage_range_final(struct mmu_gather= *tlb, } =20 void unmap_hugepage_range(struct vm_area_struct *vma, unsigned long star= t, - unsigned long end, struct page *ref_page) + unsigned long end, struct page *ref_page, + unsigned long zap_flags) { struct mm_struct *mm; struct mmu_gather tlb; @@ -4004,7 +4019,7 @@ void unmap_hugepage_range(struct vm_area_struct *vm= a, unsigned long start, mm =3D vma->vm_mm; =20 tlb_gather_mmu(&tlb, mm, tlb_start, tlb_end); - __unmap_hugepage_range(&tlb, vma, start, end, ref_page); + __unmap_hugepage_range(&tlb, vma, start, end, ref_page, zap_flags); tlb_finish_mmu(&tlb, tlb_start, tlb_end); } =20 @@ -4059,7 +4074,7 @@ static void unmap_ref_private(struct mm_struct *mm,= struct vm_area_struct *vma, */ if (!is_vma_resv_set(iter_vma, HPAGE_RESV_OWNER)) unmap_hugepage_range(iter_vma, address, - address + huge_page_size(h), page); + address + huge_page_size(h), page, 0); } i_mmap_unlock_write(mapping); } diff --git a/mm/memory.c b/mm/memory.c index 59d56f57ba2c..993ec7a7961a 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1499,8 +1499,11 @@ static void unmap_single_vma(struct mmu_gather *tl= b, * safe to do nothing in this case. */ if (vma->vm_file) { + unsigned long zap_flags =3D details ? + details->zap_flags : 0; i_mmap_lock_write(vma->vm_file->f_mapping); - __unmap_hugepage_range_final(tlb, vma, start, end, NULL); + __unmap_hugepage_range_final(tlb, vma, start, end, + NULL, zap_flags); i_mmap_unlock_write(vma->vm_file->f_mapping); } } else --=20 2.26.2