From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 90A3EC5517A for ; Mon, 26 Oct 2020 04:14:41 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id F1ECA2073A for ; Mon, 26 Oct 2020 04:14:40 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="EGs1p8sF" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org F1ECA2073A Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 41D6F6B0073; Mon, 26 Oct 2020 00:14:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1F05F6B0075; Mon, 26 Oct 2020 00:14:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0DA0F6B007B; Mon, 26 Oct 2020 00:14:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0131.hostedemail.com [216.40.44.131]) by kanga.kvack.org (Postfix) with ESMTP id BE1156B0074 for ; Mon, 26 Oct 2020 00:14:27 -0400 (EDT) Received: from smtpin12.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 73F6E3624 for ; Mon, 26 Oct 2020 04:14:27 +0000 (UTC) X-FDA: 77412759774.12.flesh31_1617bf027270 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin12.hostedemail.com (Postfix) with ESMTP id 57C0E1800EC4D for ; Mon, 26 Oct 2020 04:14:27 +0000 (UTC) X-HE-Tag: flesh31_1617bf027270 X-Filterd-Recvd-Size: 10741 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf34.hostedemail.com (Postfix) with ESMTP for ; Mon, 26 Oct 2020 04:14:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=D4HUkF63j7GrFkU84geEXagljZ4wF1EtZdc5p5/p3pw=; b=EGs1p8sFgOKa6qgZad9xlY0Cju i1oJNd/8uuYAxaImSQmGz4DGuVqT09wW762qsH/nnroe2ZS0R1o26nrzOdp2FVfx0VHNWzG/8TlRm jwXNvNwf+Xe6k88yOc5VJ6DpiuZZdSJz1zVVzZvaQfxSVH8V8ZVeFfo4QSDQVwPjuSlUdB+skjl7U dIw0m+gE/vDoQz03YAYSi0TIc/dV1zgAEeFTak4YvqwZTzww0ZpUm1dxlUCQuyDD9YY2S1kpQG7OY WA8lbigI6x04OVb29uo675bCb7Yt+3yKRDBV3RQWa0/1XOTbTD03ESIHQKtDDtXsFBtrvj0uDXmCN eDr4RiRA==; Received: from willy by casper.infradead.org with local (Exim 4.92.3 #3 (Red Hat Linux)) id 1kWttb-0006a3-8x; Mon, 26 Oct 2020 04:14:11 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , linux-fsdevel@vger.kernel.org, Andrew Morton , Hugh Dickins , Johannes Weiner , Yang Shi , Dave Chinner , linux-kernel@vger.kernel.org, Jan Kara , William Kucharski Subject: [PATCH v3 05/12] mm: Add and use find_lock_entries Date: Mon, 26 Oct 2020 04:14:01 +0000 Message-Id: <20201026041408.25230-6-willy@infradead.org> X-Mailer: git-send-email 2.21.3 In-Reply-To: <20201026041408.25230-1-willy@infradead.org> References: <20201026041408.25230-1-willy@infradead.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: We have three functions (shmem_undo_range(), truncate_inode_pages_range() and invalidate_mapping_pages()) which want exactly this function, so add it to filemap.c. Before this patch, shmem_undo_range() would split any compound page which overlaps either end of the range being punched in both the first and second loops through the address space. After this patch, that functionality is left for the second loop, which is arguably more appropriate since the first loop is supposed to run through all the pages quickly, and splitting a page can sleep. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Jan Kara Reviewed-by: William Kucharski --- mm/filemap.c | 57 ++++++++++++++++++++++++++++++++ mm/internal.h | 2 ++ mm/shmem.c | 22 +++---------- mm/truncate.c | 91 +++++++-------------------------------------------- 4 files changed, 75 insertions(+), 97 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 3a55d258d9f2..9a33d1b8cef6 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -1952,6 +1952,63 @@ unsigned find_get_entries(struct address_space *ma= pping, return ret; } =20 +/** + * find_lock_entries - Find a batch of pagecache entries. + * @mapping: The address_space to search. + * @start: The starting page cache index. + * @end: The final page index (inclusive). + * @pvec: Where the resulting entries are placed. + * @indices: The cache indices of the entries in @pvec. + * + * find_lock_entries() will return a batch of entries from @mapping. + * Swap, shadow and DAX entries are included. Pages are returned + * locked and with an incremented refcount. Pages which are locked by + * somebody else or under writeback are skipped. Only the head page of + * a THP is returned. Pages which are partially outside the range are + * not returned. + * + * The entries have ascending indexes. The indices may not be consecuti= ve + * due to not-present entries, THP pages, pages which could not be locke= d + * or pages under writeback. + * + * Return: The number of entries which were found. + */ +unsigned find_lock_entries(struct address_space *mapping, pgoff_t start, + pgoff_t end, struct pagevec *pvec, pgoff_t *indices) +{ + XA_STATE(xas, &mapping->i_pages, start); + struct page *page; + + rcu_read_lock(); + while ((page =3D xas_find_get_entry(&xas, end, XA_PRESENT))) { + if (!xa_is_value(page)) { + if (page->index < start) + goto put; + VM_BUG_ON_PAGE(page->index !=3D xas.xa_index, page); + if (page->index + thp_nr_pages(page) - 1 > end) + goto put; + if (!trylock_page(page)) + goto put; + if (page->mapping !=3D mapping || PageWriteback(page)) + goto unlock; + } + indices[pvec->nr] =3D xas.xa_index; + if (!pagevec_add(pvec, page)) + break; + goto next; +unlock: + unlock_page(page); +put: + put_page(page); +next: + if (!xa_is_value(page) && PageTransHuge(page)) + xas_set(&xas, page->index + thp_nr_pages(page)); + } + rcu_read_unlock(); + + return pagevec_count(pvec); +} + /** * find_get_pages_range - gang pagecache lookup * @mapping: The address_space to search diff --git a/mm/internal.h b/mm/internal.h index c43ccdddb0f6..8d79f4d21eaf 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -62,6 +62,8 @@ static inline void force_page_cache_readahead(struct ad= dress_space *mapping, =20 struct page *find_get_entry(struct address_space *mapping, pgoff_t index= ); struct page *find_lock_entry(struct address_space *mapping, pgoff_t inde= x); +unsigned find_lock_entries(struct address_space *mapping, pgoff_t start, + pgoff_t end, struct pagevec *pvec, pgoff_t *indices); =20 /** * page_evictable - test whether a page is evictable diff --git a/mm/shmem.c b/mm/shmem.c index 726cede653f5..ef34271cad2d 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -907,12 +907,8 @@ static void shmem_undo_range(struct inode *inode, lo= ff_t lstart, loff_t lend, =20 pagevec_init(&pvec); index =3D start; - while (index < end) { - pvec.nr =3D find_get_entries(mapping, index, - min(end - index, (pgoff_t)PAGEVEC_SIZE), - pvec.pages, indices); - if (!pvec.nr) - break; + while (index < end && find_lock_entries(mapping, index, end - 1, + &pvec, indices)) { for (i =3D 0; i < pagevec_count(&pvec); i++) { struct page *page =3D pvec.pages[i]; =20 @@ -927,18 +923,10 @@ static void shmem_undo_range(struct inode *inode, l= off_t lstart, loff_t lend, index, page); continue; } + index +=3D thp_nr_pages(page) - 1; =20 - VM_BUG_ON_PAGE(page_to_pgoff(page) !=3D index, page); - - if (!trylock_page(page)) - continue; - - if ((!unfalloc || !PageUptodate(page)) && - page_mapping(page) =3D=3D mapping) { - VM_BUG_ON_PAGE(PageWriteback(page), page); - if (shmem_punch_compound(page, start, end)) - truncate_inode_page(mapping, page); - } + if (!unfalloc || !PageUptodate(page)) + truncate_inode_page(mapping, page); unlock_page(page); } pagevec_remove_exceptionals(&pvec); diff --git a/mm/truncate.c b/mm/truncate.c index 18cec39a9f53..3c6b6d5a0046 100644 --- a/mm/truncate.c +++ b/mm/truncate.c @@ -326,51 +326,19 @@ void truncate_inode_pages_range(struct address_spac= e *mapping, =20 pagevec_init(&pvec); index =3D start; - while (index < end && pagevec_lookup_entries(&pvec, mapping, index, - min(end - index, (pgoff_t)PAGEVEC_SIZE), - indices)) { - /* - * Pagevec array has exceptional entries and we may also fail - * to lock some pages. So we store pages that can be deleted - * in a new pagevec. - */ - struct pagevec locked_pvec; - - pagevec_init(&locked_pvec); - for (i =3D 0; i < pagevec_count(&pvec); i++) { - struct page *page =3D pvec.pages[i]; - - /* We rely upon deletion not changing page->index */ - index =3D indices[i]; - if (index >=3D end) - break; - - if (xa_is_value(page)) - continue; - - if (!trylock_page(page)) - continue; - WARN_ON(page_to_index(page) !=3D index); - if (PageWriteback(page)) { - unlock_page(page); - continue; - } - if (page->mapping !=3D mapping) { - unlock_page(page); - continue; - } - pagevec_add(&locked_pvec, page); - } - for (i =3D 0; i < pagevec_count(&locked_pvec); i++) - truncate_cleanup_page(mapping, locked_pvec.pages[i]); - delete_from_page_cache_batch(mapping, &locked_pvec); - for (i =3D 0; i < pagevec_count(&locked_pvec); i++) - unlock_page(locked_pvec.pages[i]); + while (index < end && find_lock_entries(mapping, index, end - 1, + &pvec, indices)) { + index =3D indices[pagevec_count(&pvec) - 1] + 1; truncate_exceptional_pvec_entries(mapping, &pvec, indices, end); + for (i =3D 0; i < pagevec_count(&pvec); i++) + truncate_cleanup_page(mapping, pvec.pages[i]); + delete_from_page_cache_batch(mapping, &pvec); + for (i =3D 0; i < pagevec_count(&pvec); i++) + unlock_page(pvec.pages[i]); pagevec_release(&pvec); cond_resched(); - index++; } + if (partial_start) { struct page *page =3D find_lock_page(mapping, start - 1); if (page) { @@ -539,9 +507,7 @@ unsigned long __invalidate_mapping_pages(struct addre= ss_space *mapping, int i; =20 pagevec_init(&pvec); - while (index <=3D end && pagevec_lookup_entries(&pvec, mapping, index, - min(end - index, (pgoff_t)PAGEVEC_SIZE - 1) + 1, - indices)) { + while (find_lock_entries(mapping, index, end, &pvec, indices)) { for (i =3D 0; i < pagevec_count(&pvec); i++) { struct page *page =3D pvec.pages[i]; =20 @@ -555,39 +521,7 @@ unsigned long __invalidate_mapping_pages(struct addr= ess_space *mapping, page); continue; } - - if (!trylock_page(page)) - continue; - - WARN_ON(page_to_index(page) !=3D index); - - /* Middle of THP: skip */ - if (PageTransTail(page)) { - unlock_page(page); - continue; - } else if (PageTransHuge(page)) { - index +=3D HPAGE_PMD_NR - 1; - i +=3D HPAGE_PMD_NR - 1; - /* - * 'end' is in the middle of THP. Don't - * invalidate the page as the part outside of - * 'end' could be still useful. - */ - if (index > end) { - unlock_page(page); - continue; - } - - /* Take a pin outside pagevec */ - get_page(page); - - /* - * Drop extra pins before trying to invalidate - * the huge page. - */ - pagevec_remove_exceptionals(&pvec); - pagevec_release(&pvec); - } + index +=3D thp_nr_pages(page) - 1; =20 ret =3D invalidate_inode_page(page); unlock_page(page); @@ -601,9 +535,6 @@ unsigned long __invalidate_mapping_pages(struct addre= ss_space *mapping, if (nr_pagevec) (*nr_pagevec)++; } - - if (PageTransHuge(page)) - put_page(page); count +=3D ret; } pagevec_remove_exceptionals(&pvec); --=20 2.28.0