From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.5 required=3.0 tests=DKIM_INVALID,DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 41925C35254 for ; Wed, 19 Feb 2020 21:01:17 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id E061324654 for ; Wed, 19 Feb 2020 21:01:16 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="WXBgTKQs" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E061324654 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id C66AA6B0007; Wed, 19 Feb 2020 16:01:08 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id BC6F06B000A; Wed, 19 Feb 2020 16:01:08 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AB8A16B000C; Wed, 19 Feb 2020 16:01:08 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0253.hostedemail.com [216.40.44.253]) by kanga.kvack.org (Postfix) with ESMTP id 8A2B36B0007 for ; Wed, 19 Feb 2020 16:01:08 -0500 (EST) Received: from smtpin10.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 491BE180AD815 for ; Wed, 19 Feb 2020 21:01:08 +0000 (UTC) X-FDA: 76508096616.10.sleep35_9ddc2381fb17 X-HE-Tag: sleep35_9ddc2381fb17 X-Filterd-Recvd-Size: 7040 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf26.hostedemail.com (Postfix) with ESMTP for ; Wed, 19 Feb 2020 21:01:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=vhLH5Sbrak3JAHbVrH4ikkRxZI58fwVlV1VnoATDnKM=; b=WXBgTKQseU8E34FxKQI0F2BT+o CQ7O4Q4Hr6JgCjCHNJs0UaY4bivB07YB9Mp447bJYCwoplvd1D+OL7cnRtYs35XBf3evBAD69gnvb 12ucE1OpTnyHYjLZ7m5HeP+SJoRKvp6sFMkk9vat3VTmGpEhSNVll6sVlBYIdZaUY9OBgstb2XvXi 38RTWxE3tC1OZRty3qOOKhJ4OJbTOu4lTQbG8cFdd90DPceHh455tUlFgOrmyoxcNaTaUFVMASCWz rnV8O9HzXEmt07Cb2ZhBfuAC9Lr/oNscwOBJAe9YnjUwO0l6BmNfmFOUEZSGqv+oEI97+rO6oMQ90 sDELCpfA==; Received: from willy by bombadil.infradead.org with local (Exim 4.92.3 #3 (Red Hat Linux)) id 1j4WSv-0008Tm-1k; Wed, 19 Feb 2020 21:01:05 +0000 From: Matthew Wilcox To: linux-fsdevel@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-btrfs@vger.kernel.org, linux-erofs@lists.ozlabs.org, linux-ext4@vger.kernel.org, linux-f2fs-devel@lists.sourceforge.net, cluster-devel@redhat.com, ocfs2-devel@oss.oracle.com, linux-xfs@vger.kernel.org Subject: [PATCH v7 09/24] mm: Put readahead pages in cache earlier Date: Wed, 19 Feb 2020 13:00:48 -0800 Message-Id: <20200219210103.32400-10-willy@infradead.org> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200219210103.32400-1-willy@infradead.org> References: <20200219210103.32400-1-willy@infradead.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Matthew Wilcox (Oracle)" When populating the page cache for readahead, mappings that use ->readpages must populate the page cache themselves as the pages are passed on a linked list which would normally be used for the page cache's LRU. For mappings that use ->readpage or the upcoming ->readahead method= , we can put the pages into the page cache as soon as they're allocated, which solves a race between readahead and direct IO. It also lets us remove the gfp argument from read_pages(). Use the new readahead_page() API to implement the repeated calls to ->readpage(), just like most filesystems will. This iterator also supports huge pages, even though none of the filesystems have been converted to use them yet. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/pagemap.h | 20 +++++++++++++++++ mm/readahead.c | 48 +++++++++++++++++++++++++---------------- 2 files changed, 49 insertions(+), 19 deletions(-) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 55fcea0249e6..4989d330fada 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -647,8 +647,28 @@ struct readahead_control { /* private: use the readahead_* accessors instead */ pgoff_t _index; unsigned int _nr_pages; + unsigned int _batch_count; }; =20 +static inline struct page *readahead_page(struct readahead_control *rac) +{ + struct page *page; + + BUG_ON(rac->_batch_count > rac->_nr_pages); + rac->_nr_pages -=3D rac->_batch_count; + rac->_index +=3D rac->_batch_count; + rac->_batch_count =3D 0; + + if (!rac->_nr_pages) + return NULL; + + page =3D xa_load(&rac->mapping->i_pages, rac->_index); + VM_BUG_ON_PAGE(!PageLocked(page), page); + rac->_batch_count =3D hpage_nr_pages(page); + + return page; +} + /* The number of pages in this readahead block */ static inline unsigned int readahead_count(struct readahead_control *rac= ) { diff --git a/mm/readahead.c b/mm/readahead.c index 83df5c061d33..aaa209559ba2 100644 --- a/mm/readahead.c +++ b/mm/readahead.c @@ -113,15 +113,14 @@ int read_cache_pages(struct address_space *mapping,= struct list_head *pages, =20 EXPORT_SYMBOL(read_cache_pages); =20 -static void read_pages(struct readahead_control *rac, struct list_head *= pages, - gfp_t gfp) +static void read_pages(struct readahead_control *rac, struct list_head *= pages) { const struct address_space_operations *aops =3D rac->mapping->a_ops; + struct page *page; struct blk_plug plug; - unsigned page_idx; =20 if (!readahead_count(rac)) - return; + goto out; =20 blk_start_plug(&plug); =20 @@ -130,23 +129,23 @@ static void read_pages(struct readahead_control *ra= c, struct list_head *pages, readahead_count(rac)); /* Clean up the remaining pages */ put_pages_list(pages); - goto out; - } - - for (page_idx =3D 0; page_idx < readahead_count(rac); page_idx++) { - struct page *page =3D lru_to_page(pages); - list_del(&page->lru); - if (!add_to_page_cache_lru(page, rac->mapping, page->index, - gfp)) + rac->_index +=3D rac->_nr_pages; + rac->_nr_pages =3D 0; + } else { + while ((page =3D readahead_page(rac))) { aops->readpage(rac->file, page); - put_page(page); + put_page(page); + } } =20 -out: blk_finish_plug(&plug); =20 BUG_ON(!list_empty(pages)); - rac->_nr_pages =3D 0; + BUG_ON(readahead_count(rac)); + +out: + /* If we were called due to a conflicting page, skip over it */ + rac->_index++; } =20 /* @@ -165,9 +164,11 @@ void __do_page_cache_readahead(struct address_space = *mapping, LIST_HEAD(page_pool); loff_t isize =3D i_size_read(inode); gfp_t gfp_mask =3D readahead_gfp_mask(mapping); + bool use_list =3D mapping->a_ops->readpages; struct readahead_control rac =3D { .mapping =3D mapping, .file =3D filp, + ._index =3D index, ._nr_pages =3D 0, }; unsigned long i; @@ -184,6 +185,8 @@ void __do_page_cache_readahead(struct address_space *= mapping, if (index + i > end_index) break; =20 + BUG_ON(index + i !=3D rac._index + rac._nr_pages); + page =3D xa_load(&mapping->i_pages, index + i); if (page && !xa_is_value(page)) { /* @@ -191,15 +194,22 @@ void __do_page_cache_readahead(struct address_space= *mapping, * contiguous pages before continuing with the next * batch. */ - read_pages(&rac, &page_pool, gfp_mask); + read_pages(&rac, &page_pool); continue; } =20 page =3D __page_cache_alloc(gfp_mask); if (!page) break; - page->index =3D index + i; - list_add(&page->lru, &page_pool); + if (use_list) { + page->index =3D index + i; + list_add(&page->lru, &page_pool); + } else if (add_to_page_cache_lru(page, mapping, index + i, + gfp_mask) < 0) { + put_page(page); + read_pages(&rac, &page_pool); + continue; + } if (i =3D=3D nr_to_read - lookahead_size) SetPageReadahead(page); rac._nr_pages++; @@ -210,7 +220,7 @@ void __do_page_cache_readahead(struct address_space *= mapping, * uptodate then the caller will launch readpage again, and * will then handle the error. */ - read_pages(&rac, &page_pool, gfp_mask); + read_pages(&rac, &page_pool); } =20 /* --=20 2.25.0