[PATCH 00/48] Folios for 5.17

[PATCH 00/48] Folios for 5.17

[Matthew Wilcox (Oracle)]

I was trying to get all the way to adding large folios to the page cache,
but I ran out of time.  I think the changes in this batch of patches
are worth adding, even without large folio support for filesystems other
than tmpfs.

The big change here is the last patch which switches from storing
large folios in the page cache as 2^N identical entries to using the
XArray support for multi-index entries.  As the changelog says, this
fixes a bug that can occur when doing (eg) msync() or sync_file_range().

Some parts of this have been sent before.  The first patch is already
in Andrew's tree, but I included it here so the build bots don't freak
out about not being able to apply this patch series.  The folio_batch
(replacement for pagevec) is new.  I also fixed a few bugs.

This all passes xfstests with no new failures on both xfs and tmpfs.
I intend to put all this into for-next tomorrow.

Matthew Wilcox (Oracle) (48):
  filemap: Remove PageHWPoison check from next_uptodate_page()
  fs/writeback: Convert inode_switch_wbs_work_fn to folios
  mm/doc: Add documentation for folio_test_uptodate
  mm/writeback: Improve __folio_mark_dirty() comment
  pagevec: Add folio_batch
  iov_iter: Add copy_folio_to_iter()
  iov_iter: Convert iter_xarray to use folios
  mm: Add folio_test_pmd_mappable()
  filemap: Add folio_put_wait_locked()
  filemap: Convert page_cache_delete to take a folio
  filemap: Add filemap_unaccount_folio()
  filemap: Convert tracing of page cache operations to folio
  filemap: Add filemap_remove_folio and __filemap_remove_folio
  filemap: Convert find_get_entry to return a folio
  filemap: Remove thp_contains()
  filemap: Convert filemap_get_read_batch to use folios
  filemap: Convert find_get_pages_contig to folios
  filemap: Convert filemap_read_page to take a folio
  filemap: Convert filemap_create_page to folio
  filemap: Convert filemap_range_uptodate to folios
  readahead: Convert page_cache_async_ra() to take a folio
  readahead: Convert page_cache_ra_unbounded to folios
  filemap: Convert do_async_mmap_readahead to take a folio
  filemap: Convert filemap_fault to folio
  filemap: Add read_cache_folio and read_mapping_folio
  filemap: Convert filemap_get_pages to use folios
  filemap: Convert page_cache_delete_batch to folios
  filemap: Use folios in next_uptodate_page
  filemap: Use a folio in filemap_map_pages
  filemap: Use a folio in filemap_page_mkwrite
  filemap: Add filemap_release_folio()
  truncate: Add truncate_cleanup_folio()
  mm: Add unmap_mapping_folio()
  shmem: Convert part of shmem_undo_range() to use a folio
  truncate,shmem: Add truncate_inode_folio()
  truncate: Skip known-truncated indices
  truncate: Convert invalidate_inode_pages2_range() to use a folio
  truncate: Add invalidate_complete_folio2()
  filemap: Convert filemap_read() to use a folio
  filemap: Convert filemap_get_read_batch() to use a folio_batch
  filemap: Return only folios from find_get_entries()
  mm: Convert find_lock_entries() to use a folio_batch
  mm: Remove pagevec_remove_exceptionals()
  fs: Convert vfs_dedupe_file_range_compare to folios
  truncate: Convert invalidate_inode_pages2_range to folios
  truncate,shmem: Handle truncates that split large folios
  XArray: Add xas_advance()
  mm: Use multi-index entries in the page cache

 fs/fs-writeback.c              |  24 +-
 fs/remap_range.c               | 116 ++--
 include/linux/huge_mm.h        |  14 +
 include/linux/mm.h             |  68 +--
 include/linux/page-flags.h     |  13 +-
 include/linux/pagemap.h        |  59 +-
 include/linux/pagevec.h        |  61 ++-
 include/linux/uio.h            |   7 +
 include/linux/xarray.h         |  18 +
 include/trace/events/filemap.h |  32 +-
 lib/iov_iter.c                 |  29 +-
 lib/xarray.c                   |   6 +-
 mm/filemap.c                   | 967 ++++++++++++++++-----------------
 mm/folio-compat.c              |  11 +
 mm/huge_memory.c               |  20 +-
 mm/internal.h                  |  35 +-
 mm/khugepaged.c                |  12 +-
 mm/memory.c                    |  27 +-
 mm/migrate.c                   |  29 +-
 mm/page-writeback.c            |   6 +-
 mm/readahead.c                 |  24 +-
 mm/shmem.c                     | 174 +++---
 mm/swap.c                      |  26 +-
 mm/truncate.c                  | 305 ++++++-----
 24 files changed, 1100 insertions(+), 983 deletions(-)

-- 
2.33.0

[PATCH 01/48] filemap: Remove PageHWPoison check from next_uptodate_page()

[Matthew Wilcox (Oracle)]

Pages are individually marked as suffering from hardware poisoning.
Checking that the head page is not hardware poisoned doesn't make
sense; we might be after a subpage.  We check each page individually
before we use it, so this was an optimisation gone wrong.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 mm/filemap.c | 2 --
 1 file changed, 2 deletions(-)

diff --git a/mm/filemap.c b/mm/filemap.c
index daa0e23a6ee6..39c4c46c6133 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -3253,8 +3253,6 @@ static struct page *next_uptodate_page(struct page *page,
 			goto skip;
 		if (!PageUptodate(page) || PageReadahead(page))
 			goto skip;
-		if (PageHWPoison(page))
-			goto skip;
 		if (!trylock_page(page))
 			goto skip;
 		if (page->mapping != mapping)
-- 
2.33.0

[PATCH 02/48] fs/writeback: Convert inode_switch_wbs_work_fn to folios

[Matthew Wilcox (Oracle)]

This gets the statistics correct by modifying the counters by the
number of pages in the folio instead of by 1.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 fs/fs-writeback.c | 24 +++++++++++++-----------
 1 file changed, 13 insertions(+), 11 deletions(-)

diff --git a/fs/fs-writeback.c b/fs/fs-writeback.c
index 67f0e88eed01..4f680f848c8b 100644
--- a/fs/fs-writeback.c
+++ b/fs/fs-writeback.c
@@ -372,7 +372,7 @@ static bool inode_do_switch_wbs(struct inode *inode,
 {
 	struct address_space *mapping = inode->i_mapping;
 	XA_STATE(xas, &mapping->i_pages, 0);
-	struct page *page;
+	struct folio *folio;
 	bool switched = false;
 
 	spin_lock(&inode->i_lock);
@@ -389,21 +389,23 @@ static bool inode_do_switch_wbs(struct inode *inode,
 
 	/*
 	 * Count and transfer stats.  Note that PAGECACHE_TAG_DIRTY points
-	 * to possibly dirty pages while PAGECACHE_TAG_WRITEBACK points to
-	 * pages actually under writeback.
+	 * to possibly dirty folios while PAGECACHE_TAG_WRITEBACK points to
+	 * folios actually under writeback.
 	 */
-	xas_for_each_marked(&xas, page, ULONG_MAX, PAGECACHE_TAG_DIRTY) {
-		if (PageDirty(page)) {
-			dec_wb_stat(old_wb, WB_RECLAIMABLE);
-			inc_wb_stat(new_wb, WB_RECLAIMABLE);
+	xas_for_each_marked(&xas, folio, ULONG_MAX, PAGECACHE_TAG_DIRTY) {
+		if (folio_test_dirty(folio)) {
+			long nr = folio_nr_pages(folio);
+			wb_stat_mod(old_wb, WB_RECLAIMABLE, -nr);
+			wb_stat_mod(new_wb, WB_RECLAIMABLE, nr);
 		}
 	}
 
 	xas_set(&xas, 0);
-	xas_for_each_marked(&xas, page, ULONG_MAX, PAGECACHE_TAG_WRITEBACK) {
-		WARN_ON_ONCE(!PageWriteback(page));
-		dec_wb_stat(old_wb, WB_WRITEBACK);
-		inc_wb_stat(new_wb, WB_WRITEBACK);
+	xas_for_each_marked(&xas, folio, ULONG_MAX, PAGECACHE_TAG_WRITEBACK) {
+		long nr = folio_nr_pages(folio);
+		WARN_ON_ONCE(!folio_test_writeback(folio));
+		wb_stat_mod(old_wb, WB_WRITEBACK, -nr);
+		wb_stat_mod(new_wb, WB_WRITEBACK, nr);
 	}
 
 	if (mapping_tagged(mapping, PAGECACHE_TAG_WRITEBACK)) {
-- 
2.33.0

[PATCH 03/48] mm/doc: Add documentation for folio_test_uptodate

[Matthew Wilcox (Oracle)]

Move the PG_uptodate documentation to be documentation for
folio_test_uptodate() and expand on it a little.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 include/linux/page-flags.h | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)

diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h
index b5f14d581113..b3d353d537e2 100644
--- a/include/linux/page-flags.h
+++ b/include/linux/page-flags.h
@@ -68,9 +68,6 @@
  * might lose their PG_swapbacked flag when they simply can be dropped (e.g. as
  * a result of MADV_FREE).
  *
- * PG_uptodate tells whether the page's contents is valid.  When a read
- * completes, the page becomes uptodate, unless a disk I/O error happened.
- *
  * PG_referenced, PG_reclaim are used for page reclaim for anonymous and
  * file-backed pagecache (see mm/vmscan.c).
  *
@@ -615,6 +612,16 @@ TESTPAGEFLAG_FALSE(Ksm, ksm)
 
 u64 stable_page_flags(struct page *page);
 
+/**
+ * folio_test_uptodate - Is this folio up to date?
+ * @folio: The folio.
+ *
+ * The uptodate flag is set on a folio when every byte in the folio is
+ * at least as new as the corresponding bytes on storage.  Anonymous
+ * and CoW folios are always uptodate.  If the folio is not uptodate,
+ * some of the bytes in it may be; see the is_partially_uptodate()
+ * address_space operation.
+ */
 static inline bool folio_test_uptodate(struct folio *folio)
 {
 	bool ret = test_bit(PG_uptodate, folio_flags(folio, 0));
-- 
2.33.0

[PATCH 04/48] mm/writeback: Improve __folio_mark_dirty() comment

[Matthew Wilcox (Oracle)]

Add some notes about how this function needs to be called.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 mm/page-writeback.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/mm/page-writeback.c b/mm/page-writeback.c
index a613f8ef6a02..91d163f8d36b 100644
--- a/mm/page-writeback.c
+++ b/mm/page-writeback.c
@@ -2496,7 +2496,11 @@ void folio_account_cleaned(struct folio *folio, struct address_space *mapping,
  * If warn is true, then emit a warning if the folio is not uptodate and has
  * not been truncated.
  *
- * The caller must hold lock_page_memcg().
+ * The caller must hold lock_page_memcg().  Most callers have the folio
+ * locked.  A few have the folio blocked from truncation through other
+ * means (eg zap_page_range() has it mapped and is holding the page table
+ * lock).  This can also be called from mark_buffer_dirty(), which I
+ * cannot prove is always protected against truncate.
  */
 void __folio_mark_dirty(struct folio *folio, struct address_space *mapping,
 			     int warn)
-- 
2.33.0

[PATCH 05/48] pagevec: Add folio_batch

[Matthew Wilcox (Oracle)]

The folio_batch is the same as the pagevec, except that it is typed
to contain folios and not pages.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 include/linux/pagevec.h | 63 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 63 insertions(+)

diff --git a/include/linux/pagevec.h b/include/linux/pagevec.h
index 7f3f19065a9f..4483e6ad7607 100644
--- a/include/linux/pagevec.h
+++ b/include/linux/pagevec.h
@@ -15,6 +15,7 @@
 #define PAGEVEC_SIZE	15
 
 struct page;
+struct folio;
 struct address_space;
 
 struct pagevec {
@@ -81,4 +82,66 @@ static inline void pagevec_release(struct pagevec *pvec)
 		__pagevec_release(pvec);
 }
 
+/**
+ * struct folio_batch - A collection of folios.
+ *
+ * The folio_batch is used to amortise the cost of retrieving and
+ * operating on a set of folios.  The order of folios in the batch is
+ * not considered important.  Some users of the folio_batch store
+ * "exceptional" entries in it which can be removed by calling
+ * folio_batch_remove_exceptionals().
+ */
+struct folio_batch {
+	unsigned char nr;
+	unsigned char aux[3];
+	struct folio *folios[PAGEVEC_SIZE];
+};
+
+/**
+ * folio_batch_init() - Initialise a batch of folios
+ * @fbatch: The folio batch.
+ *
+ * A freshly initialised folio_batch contains zero folios.
+ */
+static inline void folio_batch_init(struct folio_batch *fbatch)
+{
+	fbatch->nr = 0;
+}
+
+static inline unsigned int folio_batch_count(struct folio_batch *fbatch)
+{
+	return fbatch->nr;
+}
+
+static inline unsigned int fbatch_space(struct folio_batch *fbatch)
+{
+	return PAGEVEC_SIZE - fbatch->nr;
+}
+
+/**
+ * folio_batch_add() - Add a folio to a batch.
+ * @fbatch: The folio batch.
+ * @folio: The folio to add.
+ *
+ * The folio is added to the end of the batch.
+ * The batch must have previously been initialised using folio_batch_init().
+ *
+ * Return: The number of slots still available.
+ */
+static inline unsigned folio_batch_add(struct folio_batch *fbatch,
+		struct folio *folio)
+{
+	fbatch->folios[fbatch->nr++] = folio;
+	return fbatch_space(fbatch);
+}
+
+static inline void folio_batch_release(struct folio_batch *fbatch)
+{
+	pagevec_release((struct pagevec *)fbatch);
+}
+
+static inline void folio_batch_remove_exceptionals(struct folio_batch *fbatch)
+{
+	pagevec_remove_exceptionals((struct pagevec *)fbatch);
+}
 #endif /* _LINUX_PAGEVEC_H */
-- 
2.33.0

[PATCH 06/48] iov_iter: Add copy_folio_to_iter()

[Matthew Wilcox (Oracle)]

This wrapper around copy_page_to_iter() works because copy_page_to_iter()
handles compound pages correctly.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 include/linux/uio.h | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/include/linux/uio.h b/include/linux/uio.h
index 6350354f97e9..8479cf46b5b1 100644
--- a/include/linux/uio.h
+++ b/include/linux/uio.h
@@ -10,6 +10,7 @@
 #include <uapi/linux/uio.h>
 
 struct page;
+struct folio;
 struct pipe_inode_info;
 
 struct kvec {
@@ -146,6 +147,12 @@ size_t _copy_to_iter(const void *addr, size_t bytes, struct iov_iter *i);
 size_t _copy_from_iter(void *addr, size_t bytes, struct iov_iter *i);
 size_t _copy_from_iter_nocache(void *addr, size_t bytes, struct iov_iter *i);
 
+static inline size_t copy_folio_to_iter(struct folio *folio, size_t offset,
+		size_t bytes, struct iov_iter *i)
+{
+	return copy_page_to_iter((struct page *)folio, offset, bytes, i);
+}
+
 static __always_inline __must_check
 size_t copy_to_iter(const void *addr, size_t bytes, struct iov_iter *i)
 {
-- 
2.33.0

[PATCH 07/48] iov_iter: Convert iter_xarray to use folios

[Matthew Wilcox (Oracle)]

Take advantage of how kmap_local_folio() works to simplify the loop.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 lib/iov_iter.c | 29 +++++++++++++----------------
 1 file changed, 13 insertions(+), 16 deletions(-)

diff --git a/lib/iov_iter.c b/lib/iov_iter.c
index 66a740e6e153..03cf43b003a0 100644
--- a/lib/iov_iter.c
+++ b/lib/iov_iter.c
@@ -69,42 +69,39 @@
 #define iterate_xarray(i, n, base, len, __off, STEP) {		\
 	__label__ __out;					\
 	size_t __off = 0;					\
-	struct page *head = NULL;				\
+	struct folio *folio;					\
 	loff_t start = i->xarray_start + i->iov_offset;		\
-	unsigned offset = start % PAGE_SIZE;			\
 	pgoff_t index = start / PAGE_SIZE;			\
-	int j;							\
-								\
 	XA_STATE(xas, i->xarray, index);			\
 								\
+	len = PAGE_SIZE - offset_in_page(start);		\
 	rcu_read_lock();					\
-	xas_for_each(&xas, head, ULONG_MAX) {			\
+	xas_for_each(&xas, folio, ULONG_MAX) {			\
 		unsigned left;					\
-		if (xas_retry(&xas, head))			\
+		size_t offset = offset_in_folio(folio, start + __off);	\
+		if (xas_retry(&xas, folio))			\
 			continue;				\
-		if (WARN_ON(xa_is_value(head)))			\
+		if (WARN_ON(xa_is_value(folio)))		\
 			break;					\
-		if (WARN_ON(PageHuge(head)))			\
+		if (WARN_ON(folio_test_hugetlb(folio)))		\
 			break;					\
-		for (j = (head->index < index) ? index - head->index : 0; \
-		     j < thp_nr_pages(head); j++) {		\
-			void *kaddr = kmap_local_page(head + j);	\
-			base = kaddr + offset;			\
-			len = PAGE_SIZE - offset;		\
+		while (offset < folio_size(folio)) {		\
+			base = kmap_local_folio(folio, offset);	\
 			len = min(n, len);			\
 			left = (STEP);				\
-			kunmap_local(kaddr);			\
+			kunmap_local(base);			\
 			len -= left;				\
 			__off += len;				\
 			n -= len;				\
 			if (left || n == 0)			\
 				goto __out;			\
-			offset = 0;				\
+			offset += len;				\
+			len = PAGE_SIZE;			\
 		}						\
 	}							\
 __out:								\
 	rcu_read_unlock();					\
-	i->iov_offset += __off;						\
+	i->iov_offset += __off;					\
 	n = __off;						\
 }
 
-- 
2.33.0

[PATCH 08/48] mm: Add folio_test_pmd_mappable()

[Matthew Wilcox (Oracle)]

Add a predicate to determine if the folio might be mapped by a PMD entry.
If CONFIG_TRANSPARENT_HUGEPAGE is disabled, we know it can't be, even
if it's large enough.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 include/linux/huge_mm.h | 14 ++++++++++++++
 include/linux/mm.h      | 42 ++++++++++++++++++++---------------------
 2 files changed, 35 insertions(+), 21 deletions(-)

diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h
index f280f33ff223..e4c18ba8d3bf 100644
--- a/include/linux/huge_mm.h
+++ b/include/linux/huge_mm.h
@@ -274,6 +274,15 @@ static inline int thp_nr_pages(struct page *page)
 	return 1;
 }
 
+/**
+ * folio_test_pmd_mappable - Can we map this folio with a PMD?
+ * @folio: The folio to test
+ */
+static inline bool folio_test_pmd_mappable(struct folio *folio)
+{
+	return folio_order(folio) >= HPAGE_PMD_ORDER;
+}
+
 struct page *follow_devmap_pmd(struct vm_area_struct *vma, unsigned long addr,
 		pmd_t *pmd, int flags, struct dev_pagemap **pgmap);
 struct page *follow_devmap_pud(struct vm_area_struct *vma, unsigned long addr,
@@ -339,6 +348,11 @@ static inline int thp_nr_pages(struct page *page)
 	return 1;
 }
 
+static inline bool folio_test_pmd_mappable(struct folio *folio)
+{
+	return false;
+}
+
 static inline bool __transparent_hugepage_enabled(struct vm_area_struct *vma)
 {
 	return false;
diff --git a/include/linux/mm.h b/include/linux/mm.h
index a7e4a9e7d807..72ca04f16711 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -714,6 +714,27 @@ int vma_is_stack_for_current(struct vm_area_struct *vma);
 struct mmu_gather;
 struct inode;
 
+static inline unsigned int compound_order(struct page *page)
+{
+	if (!PageHead(page))
+		return 0;
+	return page[1].compound_order;
+}
+
+/**
+ * folio_order - The allocation order of a folio.
+ * @folio: The folio.
+ *
+ * A folio is composed of 2^order pages.  See get_order() for the definition
+ * of order.
+ *
+ * Return: The order of the folio.
+ */
+static inline unsigned int folio_order(struct folio *folio)
+{
+	return compound_order(&folio->page);
+}
+
 #include <linux/huge_mm.h>
 
 /*
@@ -906,27 +927,6 @@ static inline void destroy_compound_page(struct page *page)
 	compound_page_dtors[page[1].compound_dtor](page);
 }
 
-static inline unsigned int compound_order(struct page *page)
-{
-	if (!PageHead(page))
-		return 0;
-	return page[1].compound_order;
-}
-
-/**
- * folio_order - The allocation order of a folio.
- * @folio: The folio.
- *
- * A folio is composed of 2^order pages.  See get_order() for the definition
- * of order.
- *
- * Return: The order of the folio.
- */
-static inline unsigned int folio_order(struct folio *folio)
-{
-	return compound_order(&folio->page);
-}
-
 static inline bool hpage_pincount_available(struct page *page)
 {
 	/*
-- 
2.33.0

[PATCH 09/48] filemap: Add folio_put_wait_locked()

[Matthew Wilcox (Oracle)]

Convert all three callers of put_and_wait_on_page_locked() to
folio_put_wait_locked().  This shrinks the kernel overall by 19 bytes.
filemap_update_page() shrinks by 19 bytes while __migration_entry_wait()
is unchanged.  folio_put_wait_locked() is 14 bytes smaller than
put_and_wait_on_page_locked(), but pmd_migration_entry_wait() grows by
14 bytes.  It removes the assumption from pmd_migration_entry_wait()
that pages cannot be larger than a PMD (which is true today, but
may be interesting to explore in the future).

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 include/linux/pagemap.h |  2 +-
 mm/filemap.c            | 27 +++++++++++++++------------
 mm/migrate.c            | 21 ++++++++++-----------
 3 files changed, 26 insertions(+), 24 deletions(-)

diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
index 605246452305..841f7ba62d7d 100644
--- a/include/linux/pagemap.h
+++ b/include/linux/pagemap.h
@@ -868,7 +868,7 @@ static inline int wait_on_page_locked_killable(struct page *page)
 	return folio_wait_locked_killable(page_folio(page));
 }
 
-int put_and_wait_on_page_locked(struct page *page, int state);
+int folio_put_wait_locked(struct folio *folio, int state);
 void wait_on_page_writeback(struct page *page);
 void folio_wait_writeback(struct folio *folio);
 int folio_wait_writeback_killable(struct folio *folio);
diff --git a/mm/filemap.c b/mm/filemap.c
index 39c4c46c6133..5dd3c6e39c9f 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -1259,10 +1259,10 @@ enum behavior {
 			 * __folio_lock() waiting on then setting PG_locked.
 			 */
 	SHARED,		/* Hold ref to page and check the bit when woken, like
-			 * wait_on_page_writeback() waiting on PG_writeback.
+			 * folio_wait_writeback() waiting on PG_writeback.
 			 */
 	DROP,		/* Drop ref to page before wait, no check when woken,
-			 * like put_and_wait_on_page_locked() on PG_locked.
+			 * like folio_put_wait_locked() on PG_locked.
 			 */
 };
 
@@ -1439,22 +1439,21 @@ int folio_wait_bit_killable(struct folio *folio, int bit_nr)
 EXPORT_SYMBOL(folio_wait_bit_killable);
 
 /**
- * put_and_wait_on_page_locked - Drop a reference and wait for it to be unlocked
- * @page: The page to wait for.
+ * folio_put_wait_locked - Drop a reference and wait for it to be unlocked
+ * @folio: The folio to wait for.
  * @state: The sleep state (TASK_KILLABLE, TASK_UNINTERRUPTIBLE, etc).
  *
- * The caller should hold a reference on @page.  They expect the page to
+ * The caller should hold a reference on @folio.  They expect the page to
  * become unlocked relatively soon, but do not wish to hold up migration
- * (for example) by holding the reference while waiting for the page to
+ * (for example) by holding the reference while waiting for the folio to
  * come unlocked.  After this function returns, the caller should not
- * dereference @page.
+ * dereference @folio.
  *
- * Return: 0 if the page was unlocked or -EINTR if interrupted by a signal.
+ * Return: 0 if the folio was unlocked or -EINTR if interrupted by a signal.
  */
-int put_and_wait_on_page_locked(struct page *page, int state)
+int folio_put_wait_locked(struct folio *folio, int state)
 {
-	return folio_wait_bit_common(page_folio(page), PG_locked, state,
-			DROP);
+	return folio_wait_bit_common(folio, PG_locked, state, DROP);
 }
 
 /**
@@ -2447,7 +2446,11 @@ static int filemap_update_page(struct kiocb *iocb,
 			goto unlock_mapping;
 		if (!(iocb->ki_flags & IOCB_WAITQ)) {
 			filemap_invalidate_unlock_shared(mapping);
-			put_and_wait_on_page_locked(&folio->page, TASK_KILLABLE);
+			/*
+			 * This is where we usually end up waiting for a
+			 * previously submitted readahead to finish.
+			 */
+			folio_put_wait_locked(folio, TASK_KILLABLE);
 			return AOP_TRUNCATED_PAGE;
 		}
 		error = __folio_lock_async(folio, iocb->ki_waitq);
diff --git a/mm/migrate.c b/mm/migrate.c
index cf25b00f03c8..311638177536 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -291,7 +291,7 @@ void __migration_entry_wait(struct mm_struct *mm, pte_t *ptep,
 {
 	pte_t pte;
 	swp_entry_t entry;
-	struct page *page;
+	struct folio *folio;
 
 	spin_lock(ptl);
 	pte = *ptep;
@@ -302,18 +302,17 @@ void __migration_entry_wait(struct mm_struct *mm, pte_t *ptep,
 	if (!is_migration_entry(entry))
 		goto out;
 
-	page = pfn_swap_entry_to_page(entry);
-	page = compound_head(page);
+	folio = page_folio(pfn_swap_entry_to_page(entry));
 
 	/*
 	 * Once page cache replacement of page migration started, page_count
-	 * is zero; but we must not call put_and_wait_on_page_locked() without
-	 * a ref. Use get_page_unless_zero(), and just fault again if it fails.
+	 * is zero; but we must not call folio_put_wait_locked() without
+	 * a ref. Use folio_try_get(), and just fault again if it fails.
 	 */
-	if (!get_page_unless_zero(page))
+	if (!folio_try_get(folio))
 		goto out;
 	pte_unmap_unlock(ptep, ptl);
-	put_and_wait_on_page_locked(page, TASK_UNINTERRUPTIBLE);
+	folio_put_wait_locked(folio, TASK_UNINTERRUPTIBLE);
 	return;
 out:
 	pte_unmap_unlock(ptep, ptl);
@@ -338,16 +337,16 @@ void migration_entry_wait_huge(struct vm_area_struct *vma,
 void pmd_migration_entry_wait(struct mm_struct *mm, pmd_t *pmd)
 {
 	spinlock_t *ptl;
-	struct page *page;
+	struct folio *folio;
 
 	ptl = pmd_lock(mm, pmd);
 	if (!is_pmd_migration_entry(*pmd))
 		goto unlock;
-	page = pfn_swap_entry_to_page(pmd_to_swp_entry(*pmd));
-	if (!get_page_unless_zero(page))
+	folio = page_folio(pfn_swap_entry_to_page(pmd_to_swp_entry(*pmd)));
+	if (!folio_try_get(folio))
 		goto unlock;
 	spin_unlock(ptl);
-	put_and_wait_on_page_locked(page, TASK_UNINTERRUPTIBLE);
+	folio_put_wait_locked(folio, TASK_UNINTERRUPTIBLE);
 	return;
 unlock:
 	spin_unlock(ptl);
-- 
2.33.0

[PATCH 10/48] filemap: Convert page_cache_delete to take a folio

[Matthew Wilcox (Oracle)]

It was already assuming a head page, so this is a straightforward
conversion.  Convert the one caller to call page_folio(), even though
it must currently be passing in a head page.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 mm/filemap.c | 22 +++++++++++-----------
 1 file changed, 11 insertions(+), 11 deletions(-)

diff --git a/mm/filemap.c b/mm/filemap.c
index 5dd3c6e39c9f..38fb26e16b85 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -121,27 +121,26 @@
  */
 
 static void page_cache_delete(struct address_space *mapping,
-				   struct page *page, void *shadow)
+				   struct folio *folio, void *shadow)
 {
-	XA_STATE(xas, &mapping->i_pages, page->index);
-	unsigned int nr = 1;
+	XA_STATE(xas, &mapping->i_pages, folio->index);
+	long nr = 1;
 
 	mapping_set_update(&xas, mapping);
 
 	/* hugetlb pages are represented by a single entry in the xarray */
-	if (!PageHuge(page)) {
-		xas_set_order(&xas, page->index, compound_order(page));
-		nr = compound_nr(page);
+	if (!folio_test_hugetlb(folio)) {
+		xas_set_order(&xas, folio->index, folio_order(folio));
+		nr = folio_nr_pages(folio);
 	}
 
-	VM_BUG_ON_PAGE(!PageLocked(page), page);
-	VM_BUG_ON_PAGE(PageTail(page), page);
-	VM_BUG_ON_PAGE(nr != 1 && shadow, page);
+	VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio);
+	VM_BUG_ON_FOLIO(nr != 1 && shadow, folio);
 
 	xas_store(&xas, shadow);
 	xas_init_marks(&xas);
 
-	page->mapping = NULL;
+	folio->mapping = NULL;
 	/* Leave page->index set: truncation lookup relies upon it */
 	mapping->nrpages -= nr;
 }
@@ -223,12 +222,13 @@ static void unaccount_page_cache_page(struct address_space *mapping,
  */
 void __delete_from_page_cache(struct page *page, void *shadow)
 {
+	struct folio *folio = page_folio(page);
 	struct address_space *mapping = page->mapping;
 
 	trace_mm_filemap_delete_from_page_cache(page);
 
 	unaccount_page_cache_page(mapping, page);
-	page_cache_delete(mapping, page, shadow);
+	page_cache_delete(mapping, folio, shadow);
 }
 
 static void page_cache_free_page(struct address_space *mapping,
-- 
2.33.0

[PATCH 11/48] filemap: Add filemap_unaccount_folio()

[Matthew Wilcox (Oracle)]

Replace unaccount_page_cache_page() with filemap_unaccount_folio().
The bug handling path could be a bit more robust (eg taking into account
the mapcounts of tail pages), but it's really never supposed to happen.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 include/linux/pagemap.h |  5 ---
 mm/filemap.c            | 70 ++++++++++++++++++++---------------------
 2 files changed, 35 insertions(+), 40 deletions(-)

diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
index 841f7ba62d7d..077b6f378666 100644
--- a/include/linux/pagemap.h
+++ b/include/linux/pagemap.h
@@ -884,11 +884,6 @@ static inline void __set_page_dirty(struct page *page,
 }
 void folio_account_cleaned(struct folio *folio, struct address_space *mapping,
 			  struct bdi_writeback *wb);
-static inline void account_page_cleaned(struct page *page,
-		struct address_space *mapping, struct bdi_writeback *wb)
-{
-	return folio_account_cleaned(page_folio(page), mapping, wb);
-}
 void __folio_cancel_dirty(struct folio *folio);
 static inline void folio_cancel_dirty(struct folio *folio)
 {
diff --git a/mm/filemap.c b/mm/filemap.c
index 38fb26e16b85..600b8c921a67 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -145,74 +145,74 @@ static void page_cache_delete(struct address_space *mapping,
 	mapping->nrpages -= nr;
 }
 
-static void unaccount_page_cache_page(struct address_space *mapping,
-				      struct page *page)
+static void filemap_unaccount_folio(struct address_space *mapping,
+		struct folio *folio)
 {
-	int nr;
+	long nr;
 
 	/*
 	 * if we're uptodate, flush out into the cleancache, otherwise
 	 * invalidate any existing cleancache entries.  We can't leave
 	 * stale data around in the cleancache once our page is gone
 	 */
-	if (PageUptodate(page) && PageMappedToDisk(page))
-		cleancache_put_page(page);
+	if (folio_test_uptodate(folio) && folio_test_mappedtodisk(folio))
+		cleancache_put_page(&folio->page);
 	else
-		cleancache_invalidate_page(mapping, page);
+		cleancache_invalidate_page(mapping, &folio->page);
 
-	VM_BUG_ON_PAGE(PageTail(page), page);
-	VM_BUG_ON_PAGE(page_mapped(page), page);
-	if (!IS_ENABLED(CONFIG_DEBUG_VM) && unlikely(page_mapped(page))) {
+	VM_BUG_ON_FOLIO(folio_mapped(folio), folio);
+	if (!IS_ENABLED(CONFIG_DEBUG_VM) && unlikely(folio_mapped(folio))) {
 		int mapcount;
 
 		pr_alert("BUG: Bad page cache in process %s  pfn:%05lx\n",
-			 current->comm, page_to_pfn(page));
-		dump_page(page, "still mapped when deleted");
+			 current->comm, folio_pfn(folio));
+		dump_page(&folio->page, "still mapped when deleted");
 		dump_stack();
 		add_taint(TAINT_BAD_PAGE, LOCKDEP_NOW_UNRELIABLE);
 
-		mapcount = page_mapcount(page);
+		mapcount = page_mapcount(&folio->page);
 		if (mapping_exiting(mapping) &&
-		    page_count(page) >= mapcount + 2) {
+		    folio_ref_count(folio) >= mapcount + 2) {
 			/*
 			 * All vmas have already been torn down, so it's
-			 * a good bet that actually the page is unmapped,
+			 * a good bet that actually the folio is unmapped,
 			 * and we'd prefer not to leak it: if we're wrong,
 			 * some other bad page check should catch it later.
 			 */
-			page_mapcount_reset(page);
-			page_ref_sub(page, mapcount);
+			page_mapcount_reset(&folio->page);
+			folio_ref_sub(folio, mapcount);
 		}
 	}
 
-	/* hugetlb pages do not participate in page cache accounting. */
-	if (PageHuge(page))
+	/* hugetlb folios do not participate in page cache accounting. */
+	if (folio_test_hugetlb(folio))
 		return;
 
-	nr = thp_nr_pages(page);
+	nr = folio_nr_pages(folio);
 
-	__mod_lruvec_page_state(page, NR_FILE_PAGES, -nr);
-	if (PageSwapBacked(page)) {
-		__mod_lruvec_page_state(page, NR_SHMEM, -nr);
-		if (PageTransHuge(page))
-			__mod_lruvec_page_state(page, NR_SHMEM_THPS, -nr);
-	} else if (PageTransHuge(page)) {
-		__mod_lruvec_page_state(page, NR_FILE_THPS, -nr);
+	__lruvec_stat_mod_folio(folio, NR_FILE_PAGES, -nr);
+	if (folio_test_swapbacked(folio)) {
+		__lruvec_stat_mod_folio(folio, NR_SHMEM, -nr);
+		if (folio_test_pmd_mappable(folio))
+			__lruvec_stat_mod_folio(folio, NR_SHMEM_THPS, -nr);
+	} else if (folio_test_pmd_mappable(folio)) {
+		__lruvec_stat_mod_folio(folio, NR_FILE_THPS, -nr);
 		filemap_nr_thps_dec(mapping);
 	}
 
 	/*
-	 * At this point page must be either written or cleaned by
-	 * truncate.  Dirty page here signals a bug and loss of
+	 * At this point folio must be either written or cleaned by
+	 * truncate.  Dirty folio here signals a bug and loss of
 	 * unwritten data.
 	 *
-	 * This fixes dirty accounting after removing the page entirely
-	 * but leaves PageDirty set: it has no effect for truncated
-	 * page and anyway will be cleared before returning page into
+	 * This fixes dirty accounting after removing the folio entirely
+	 * but leaves the dirty flag set: it has no effect for truncated
+	 * folio and anyway will be cleared before returning folio to
 	 * buddy allocator.
 	 */
-	if (WARN_ON_ONCE(PageDirty(page)))
-		account_page_cleaned(page, mapping, inode_to_wb(mapping->host));
+	if (WARN_ON_ONCE(folio_test_dirty(folio)))
+		folio_account_cleaned(folio, mapping,
+					inode_to_wb(mapping->host));
 }
 
 /*
@@ -227,7 +227,7 @@ void __delete_from_page_cache(struct page *page, void *shadow)
 
 	trace_mm_filemap_delete_from_page_cache(page);
 
-	unaccount_page_cache_page(mapping, page);
+	filemap_unaccount_folio(mapping, folio);
 	page_cache_delete(mapping, folio, shadow);
 }
 
@@ -348,7 +348,7 @@ void delete_from_page_cache_batch(struct address_space *mapping,
 	for (i = 0; i < pagevec_count(pvec); i++) {
 		trace_mm_filemap_delete_from_page_cache(pvec->pages[i]);
 
-		unaccount_page_cache_page(mapping, pvec->pages[i]);
+		filemap_unaccount_folio(mapping, page_folio(pvec->pages[i]));
 	}
 	page_cache_delete_batch(mapping, pvec);
 	xa_unlock_irq(&mapping->i_pages);
-- 
2.33.0

[PATCH 12/48] filemap: Convert tracing of page cache operations to folio

[Matthew Wilcox (Oracle)]

Pass the folio instead of a page.  The page was already implicitly a
folio as it accessed page->mapping directly.  Add the order of the folio
to the tracepoint, as this is important information.  Also drop printing
the address of the struct page as the pfn provides better information
than the struct page address.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 include/trace/events/filemap.h | 32 +++++++++++++++++---------------
 mm/filemap.c                   |  9 +++++----
 2 files changed, 22 insertions(+), 19 deletions(-)

diff --git a/include/trace/events/filemap.h b/include/trace/events/filemap.h
index c47b63db124e..46c89c1e460c 100644
--- a/include/trace/events/filemap.h
+++ b/include/trace/events/filemap.h
@@ -15,43 +15,45 @@
 
 DECLARE_EVENT_CLASS(mm_filemap_op_page_cache,
 
-	TP_PROTO(struct page *page),
+	TP_PROTO(struct folio *folio),
 
-	TP_ARGS(page),
+	TP_ARGS(folio),
 
 	TP_STRUCT__entry(
 		__field(unsigned long, pfn)
 		__field(unsigned long, i_ino)
 		__field(unsigned long, index)
 		__field(dev_t, s_dev)
+		__field(unsigned char, order)
 	),
 
 	TP_fast_assign(
-		__entry->pfn = page_to_pfn(page);
-		__entry->i_ino = page->mapping->host->i_ino;
-		__entry->index = page->index;
-		if (page->mapping->host->i_sb)
-			__entry->s_dev = page->mapping->host->i_sb->s_dev;
+		__entry->pfn = folio_pfn(folio);
+		__entry->i_ino = folio->mapping->host->i_ino;
+		__entry->index = folio->index;
+		if (folio->mapping->host->i_sb)
+			__entry->s_dev = folio->mapping->host->i_sb->s_dev;
 		else
-			__entry->s_dev = page->mapping->host->i_rdev;
+			__entry->s_dev = folio->mapping->host->i_rdev;
+		__entry->order = folio_order(folio);
 	),
 
-	TP_printk("dev %d:%d ino %lx page=%p pfn=0x%lx ofs=%lu",
+	TP_printk("dev %d:%d ino %lx pfn=0x%lx ofs=%lu order=%u",
 		MAJOR(__entry->s_dev), MINOR(__entry->s_dev),
 		__entry->i_ino,
-		pfn_to_page(__entry->pfn),
 		__entry->pfn,
-		__entry->index << PAGE_SHIFT)
+		__entry->index << PAGE_SHIFT,
+		__entry->order)
 );
 
 DEFINE_EVENT(mm_filemap_op_page_cache, mm_filemap_delete_from_page_cache,
-	TP_PROTO(struct page *page),
-	TP_ARGS(page)
+	TP_PROTO(struct folio *folio),
+	TP_ARGS(folio)
 	);
 
 DEFINE_EVENT(mm_filemap_op_page_cache, mm_filemap_add_to_page_cache,
-	TP_PROTO(struct page *page),
-	TP_ARGS(page)
+	TP_PROTO(struct folio *folio),
+	TP_ARGS(folio)
 	);
 
 TRACE_EVENT(filemap_set_wb_err,
diff --git a/mm/filemap.c b/mm/filemap.c
index 600b8c921a67..bcdc8bb4d2c8 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -225,7 +225,7 @@ void __delete_from_page_cache(struct page *page, void *shadow)
 	struct folio *folio = page_folio(page);
 	struct address_space *mapping = page->mapping;
 
-	trace_mm_filemap_delete_from_page_cache(page);
+	trace_mm_filemap_delete_from_page_cache(folio);
 
 	filemap_unaccount_folio(mapping, folio);
 	page_cache_delete(mapping, folio, shadow);
@@ -346,9 +346,10 @@ void delete_from_page_cache_batch(struct address_space *mapping,
 	spin_lock(&mapping->host->i_lock);
 	xa_lock_irq(&mapping->i_pages);
 	for (i = 0; i < pagevec_count(pvec); i++) {
-		trace_mm_filemap_delete_from_page_cache(pvec->pages[i]);
+		struct folio *folio = page_folio(pvec->pages[i]);
 
-		filemap_unaccount_folio(mapping, page_folio(pvec->pages[i]));
+		trace_mm_filemap_delete_from_page_cache(folio);
+		filemap_unaccount_folio(mapping, folio);
 	}
 	page_cache_delete_batch(mapping, pvec);
 	xa_unlock_irq(&mapping->i_pages);
@@ -959,7 +960,7 @@ noinline int __filemap_add_folio(struct address_space *mapping,
 		goto error;
 	}
 
-	trace_mm_filemap_add_to_page_cache(&folio->page);
+	trace_mm_filemap_add_to_page_cache(folio);
 	return 0;
 error:
 	folio->mapping = NULL;
-- 
2.33.0

[PATCH 13/48] filemap: Add filemap_remove_folio and __filemap_remove_folio

[Matthew Wilcox (Oracle)]

Reimplement __delete_from_page_cache() as a wrapper around
__filemap_remove_folio() and delete_from_page_cache() as a wrapper
around filemap_remove_folio().  Remove the EXPORT_SYMBOL as
delete_from_page_cache() was not used by any in-tree modules.
Convert page_cache_free_page() into filemap_free_folio().

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 include/linux/pagemap.h |  9 +++++++--
 mm/filemap.c            | 43 +++++++++++++++++++----------------------
 mm/folio-compat.c       |  5 +++++
 3 files changed, 32 insertions(+), 25 deletions(-)

diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
index 077b6f378666..3f26b191ede3 100644
--- a/include/linux/pagemap.h
+++ b/include/linux/pagemap.h
@@ -930,8 +930,13 @@ int add_to_page_cache_lru(struct page *page, struct address_space *mapping,
 		pgoff_t index, gfp_t gfp);
 int filemap_add_folio(struct address_space *mapping, struct folio *folio,
 		pgoff_t index, gfp_t gfp);
-extern void delete_from_page_cache(struct page *page);
-extern void __delete_from_page_cache(struct page *page, void *shadow);
+void filemap_remove_folio(struct folio *folio);
+void delete_from_page_cache(struct page *page);
+void __filemap_remove_folio(struct folio *folio, void *shadow);
+static inline void __delete_from_page_cache(struct page *page, void *shadow)
+{
+	__filemap_remove_folio(page_folio(page), shadow);
+}
 void replace_page_cache_page(struct page *old, struct page *new);
 void delete_from_page_cache_batch(struct address_space *mapping,
 				  struct pagevec *pvec);
diff --git a/mm/filemap.c b/mm/filemap.c
index bcdc8bb4d2c8..4fe845b30f33 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -220,58 +220,55 @@ static void filemap_unaccount_folio(struct address_space *mapping,
  * sure the page is locked and that nobody else uses it - or that usage
  * is safe.  The caller must hold the i_pages lock.
  */
-void __delete_from_page_cache(struct page *page, void *shadow)
+void __filemap_remove_folio(struct folio *folio, void *shadow)
 {
-	struct folio *folio = page_folio(page);
-	struct address_space *mapping = page->mapping;
+	struct address_space *mapping = folio->mapping;
 
 	trace_mm_filemap_delete_from_page_cache(folio);
-
 	filemap_unaccount_folio(mapping, folio);
 	page_cache_delete(mapping, folio, shadow);
 }
 
-static void page_cache_free_page(struct address_space *mapping,
-				struct page *page)
+static void filemap_free_folio(struct address_space *mapping,
+				struct folio *folio)
 {
 	void (*freepage)(struct page *);
 
 	freepage = mapping->a_ops->freepage;
 	if (freepage)
-		freepage(page);
+		freepage(&folio->page);
 
-	if (PageTransHuge(page) && !PageHuge(page)) {
-		page_ref_sub(page, thp_nr_pages(page));
-		VM_BUG_ON_PAGE(page_count(page) <= 0, page);
+	if (folio_test_large(folio) && !folio_test_hugetlb(folio)) {
+		folio_ref_sub(folio, folio_nr_pages(folio));
+		VM_BUG_ON_FOLIO(folio_ref_count(folio) <= 0, folio);
 	} else {
-		put_page(page);
+		folio_put(folio);
 	}
 }
 
 /**
- * delete_from_page_cache - delete page from page cache
- * @page: the page which the kernel is trying to remove from page cache
+ * filemap_remove_folio - Remove folio from page cache.
+ * @folio: The folio.
  *
- * This must be called only on pages that have been verified to be in the page
- * cache and locked.  It will never put the page into the free list, the caller
- * has a reference on the page.
+ * This must be called only on folios that are locked and have been
+ * verified to be in the page cache.  It will never put the folio into
+ * the free list because the caller has a reference on the page.
  */
-void delete_from_page_cache(struct page *page)
+void filemap_remove_folio(struct folio *folio)
 {
-	struct address_space *mapping = page_mapping(page);
+	struct address_space *mapping = folio->mapping;
 
-	BUG_ON(!PageLocked(page));
+	BUG_ON(!folio_test_locked(folio));
 	spin_lock(&mapping->host->i_lock);
 	xa_lock_irq(&mapping->i_pages);
-	__delete_from_page_cache(page, NULL);
+	__filemap_remove_folio(folio, NULL);
 	xa_unlock_irq(&mapping->i_pages);
 	if (mapping_shrinkable(mapping))
 		inode_add_lru(mapping->host);
 	spin_unlock(&mapping->host->i_lock);
 
-	page_cache_free_page(mapping, page);
+	filemap_free_folio(mapping, folio);
 }
-EXPORT_SYMBOL(delete_from_page_cache);
 
 /*
  * page_cache_delete_batch - delete several pages from page cache
@@ -358,7 +355,7 @@ void delete_from_page_cache_batch(struct address_space *mapping,
 	spin_unlock(&mapping->host->i_lock);
 
 	for (i = 0; i < pagevec_count(pvec); i++)
-		page_cache_free_page(mapping, pvec->pages[i]);
+		filemap_free_folio(mapping, page_folio(pvec->pages[i]));
 }
 
 int filemap_check_errors(struct address_space *mapping)
diff --git a/mm/folio-compat.c b/mm/folio-compat.c
index 5b6ae1da314e..749a695b4217 100644
--- a/mm/folio-compat.c
+++ b/mm/folio-compat.c
@@ -140,3 +140,8 @@ struct page *grab_cache_page_write_begin(struct address_space *mapping,
 			mapping_gfp_mask(mapping));
 }
 EXPORT_SYMBOL(grab_cache_page_write_begin);
+
+void delete_from_page_cache(struct page *page)
+{
+	return filemap_remove_folio(page_folio(page));
+}
-- 
2.33.0

[PATCH 14/48] filemap: Convert find_get_entry to return a folio

[Matthew Wilcox (Oracle)]

Convert callers to cope.  Saves 580 bytes of kernel text; all five
callers are reduced in size.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 mm/filemap.c | 146 +++++++++++++++++++++++++--------------------------
 1 file changed, 72 insertions(+), 74 deletions(-)

diff --git a/mm/filemap.c b/mm/filemap.c
index 4fe845b30f33..2a51ec720e9e 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -1976,37 +1976,36 @@ struct folio *__filemap_get_folio(struct address_space *mapping, pgoff_t index,
 }
 EXPORT_SYMBOL(__filemap_get_folio);
 
-static inline struct page *find_get_entry(struct xa_state *xas, pgoff_t max,
+static inline struct folio *find_get_entry(struct xa_state *xas, pgoff_t max,
 		xa_mark_t mark)
 {
-	struct page *page;
+	struct folio *folio;
 
 retry:
 	if (mark == XA_PRESENT)
-		page = xas_find(xas, max);
+		folio = xas_find(xas, max);
 	else
-		page = xas_find_marked(xas, max, mark);
+		folio = xas_find_marked(xas, max, mark);
 
-	if (xas_retry(xas, page))
+	if (xas_retry(xas, folio))
 		goto retry;
 	/*
 	 * A shadow entry of a recently evicted page, a swap
 	 * entry from shmem/tmpfs or a DAX entry.  Return it
 	 * without attempting to raise page count.
 	 */
-	if (!page || xa_is_value(page))
-		return page;
+	if (!folio || xa_is_value(folio))
+		return folio;
 
-	if (!page_cache_get_speculative(page))
+	if (!folio_try_get_rcu(folio))
 		goto reset;
 
-	/* Has the page moved or been split? */
-	if (unlikely(page != xas_reload(xas))) {
-		put_page(page);
+	if (unlikely(folio != xas_reload(xas))) {
+		folio_put(folio);
 		goto reset;
 	}
 
-	return page;
+	return folio;
 reset:
 	xas_reset(xas);
 	goto retry;
@@ -2042,19 +2041,20 @@ unsigned find_get_entries(struct address_space *mapping, pgoff_t start,
 		pgoff_t end, struct pagevec *pvec, pgoff_t *indices)
 {
 	XA_STATE(xas, &mapping->i_pages, start);
-	struct page *page;
+	struct folio *folio;
 	unsigned int ret = 0;
 	unsigned nr_entries = PAGEVEC_SIZE;
 
 	rcu_read_lock();
-	while ((page = find_get_entry(&xas, end, XA_PRESENT))) {
+	while ((folio = find_get_entry(&xas, end, XA_PRESENT)) != NULL) {
+		struct page *page = &folio->page;
 		/*
 		 * Terminate early on finding a THP, to allow the caller to
 		 * handle it all at once; but continue if this is hugetlbfs.
 		 */
-		if (!xa_is_value(page) && PageTransHuge(page) &&
-				!PageHuge(page)) {
-			page = find_subpage(page, xas.xa_index);
+		if (!xa_is_value(folio) && folio_test_large(folio) &&
+				!folio_test_hugetlb(folio)) {
+			page = folio_file_page(folio, xas.xa_index);
 			nr_entries = ret + 1;
 		}
 
@@ -2078,15 +2078,14 @@ unsigned find_get_entries(struct address_space *mapping, pgoff_t start,
  * @indices:	The cache indices of the entries in @pvec.
  *
  * find_lock_entries() will return a batch of entries from @mapping.
- * Swap, shadow and DAX entries are included.  Pages are returned
- * locked and with an incremented refcount.  Pages which are locked by
- * somebody else or under writeback are skipped.  Only the head page of
- * a THP is returned.  Pages which are partially outside the range are
- * not returned.
+ * Swap, shadow and DAX entries are included.  Folios are returned
+ * locked and with an incremented refcount.  Folios which are locked
+ * by somebody else or under writeback are skipped.  Folios which are
+ * partially outside the range are not returned.
  *
  * The entries have ascending indexes.  The indices may not be consecutive
- * due to not-present entries, THP pages, pages which could not be locked
- * or pages under writeback.
+ * due to not-present entries, large folios, folios which could not be
+ * locked or folios under writeback.
  *
  * Return: The number of entries which were found.
  */
@@ -2094,37 +2093,36 @@ unsigned find_lock_entries(struct address_space *mapping, pgoff_t start,
 		pgoff_t end, struct pagevec *pvec, pgoff_t *indices)
 {
 	XA_STATE(xas, &mapping->i_pages, start);
-	struct page *page;
+	struct folio *folio;
 
 	rcu_read_lock();
-	while ((page = find_get_entry(&xas, end, XA_PRESENT))) {
-		if (!xa_is_value(page)) {
-			if (page->index < start)
+	while ((folio = find_get_entry(&xas, end, XA_PRESENT))) {
+		if (!xa_is_value(folio)) {
+			if (folio->index < start)
 				goto put;
-			if (page->index + thp_nr_pages(page) - 1 > end)
+			if (folio->index + folio_nr_pages(folio) - 1 > end)
 				goto put;
-			if (!trylock_page(page))
+			if (!folio_trylock(folio))
 				goto put;
-			if (page->mapping != mapping || PageWriteback(page))
+			if (folio->mapping != mapping ||
+			    folio_test_writeback(folio))
 				goto unlock;
-			VM_BUG_ON_PAGE(!thp_contains(page, xas.xa_index),
-					page);
+			VM_BUG_ON_FOLIO(!folio_contains(folio, xas.xa_index),
+					folio);
 		}
 		indices[pvec->nr] = xas.xa_index;
-		if (!pagevec_add(pvec, page))
+		if (!pagevec_add(pvec, &folio->page))
 			break;
 		goto next;
 unlock:
-		unlock_page(page);
+		folio_unlock(folio);
 put:
-		put_page(page);
+		folio_put(folio);
 next:
-		if (!xa_is_value(page) && PageTransHuge(page)) {
-			unsigned int nr_pages = thp_nr_pages(page);
-
-			/* Final THP may cross MAX_LFS_FILESIZE on 32-bit */
-			xas_set(&xas, page->index + nr_pages);
-			if (xas.xa_index < nr_pages)
+		if (!xa_is_value(folio) && folio_test_large(folio)) {
+			xas_set(&xas, folio->index + folio_nr_pages(folio));
+			/* Did we wrap on 32-bit? */
+			if (!xas.xa_index)
 				break;
 		}
 	}
@@ -2159,19 +2157,19 @@ unsigned find_get_pages_range(struct address_space *mapping, pgoff_t *start,
 			      struct page **pages)
 {
 	XA_STATE(xas, &mapping->i_pages, *start);
-	struct page *page;
+	struct folio *folio;
 	unsigned ret = 0;
 
 	if (unlikely(!nr_pages))
 		return 0;
 
 	rcu_read_lock();
-	while ((page = find_get_entry(&xas, end, XA_PRESENT))) {
+	while ((folio = find_get_entry(&xas, end, XA_PRESENT))) {
 		/* Skip over shadow, swap and DAX entries */
-		if (xa_is_value(page))
+		if (xa_is_value(folio))
 			continue;
 
-		pages[ret] = find_subpage(page, xas.xa_index);
+		pages[ret] = folio_file_page(folio, xas.xa_index);
 		if (++ret == nr_pages) {
 			*start = xas.xa_index + 1;
 			goto out;
@@ -2268,25 +2266,25 @@ unsigned find_get_pages_range_tag(struct address_space *mapping, pgoff_t *index,
 			struct page **pages)
 {
 	XA_STATE(xas, &mapping->i_pages, *index);
-	struct page *page;
+	struct folio *folio;
 	unsigned ret = 0;
 
 	if (unlikely(!nr_pages))
 		return 0;
 
 	rcu_read_lock();
-	while ((page = find_get_entry(&xas, end, tag))) {
+	while ((folio = find_get_entry(&xas, end, tag))) {
 		/*
 		 * Shadow entries should never be tagged, but this iteration
 		 * is lockless so there is a window for page reclaim to evict
 		 * a page we saw tagged.  Skip over it.
 		 */
-		if (xa_is_value(page))
+		if (xa_is_value(folio))
 			continue;
 
-		pages[ret] = page;
+		pages[ret] = &folio->page;
 		if (++ret == nr_pages) {
-			*index = page->index + thp_nr_pages(page);
+			*index = folio->index + folio_nr_pages(folio);
 			goto out;
 		}
 	}
@@ -2794,44 +2792,44 @@ generic_file_read_iter(struct kiocb *iocb, struct iov_iter *iter)
 }
 EXPORT_SYMBOL(generic_file_read_iter);
 
-static inline loff_t page_seek_hole_data(struct xa_state *xas,
-		struct address_space *mapping, struct page *page,
+static inline loff_t folio_seek_hole_data(struct xa_state *xas,
+		struct address_space *mapping, struct folio *folio,
 		loff_t start, loff_t end, bool seek_data)
 {
 	const struct address_space_operations *ops = mapping->a_ops;
 	size_t offset, bsz = i_blocksize(mapping->host);
 
-	if (xa_is_value(page) || PageUptodate(page))
+	if (xa_is_value(folio) || folio_test_uptodate(folio))
 		return seek_data ? start : end;
 	if (!ops->is_partially_uptodate)
 		return seek_data ? end : start;
 
 	xas_pause(xas);
 	rcu_read_unlock();
-	lock_page(page);
-	if (unlikely(page->mapping != mapping))
+	folio_lock(folio);
+	if (unlikely(folio->mapping != mapping))
 		goto unlock;
 
-	offset = offset_in_thp(page, start) & ~(bsz - 1);
+	offset = offset_in_folio(folio, start) & ~(bsz - 1);
 
 	do {
-		if (ops->is_partially_uptodate(page, offset, bsz) == seek_data)
+		if (ops->is_partially_uptodate(&folio->page, offset, bsz) ==
+							seek_data)
 			break;
 		start = (start + bsz) & ~(bsz - 1);
 		offset += bsz;
-	} while (offset < thp_size(page));
+	} while (offset < folio_size(folio));
 unlock:
-	unlock_page(page);
+	folio_unlock(folio);
 	rcu_read_lock();
 	return start;
 }
 
-static inline
-unsigned int seek_page_size(struct xa_state *xas, struct page *page)
+static inline size_t seek_folio_size(struct xa_state *xas, struct folio *folio)
 {
-	if (xa_is_value(page))
+	if (xa_is_value(folio))
 		return PAGE_SIZE << xa_get_order(xas->xa, xas->xa_index);
-	return thp_size(page);
+	return folio_size(folio);
 }
 
 /**
@@ -2858,15 +2856,15 @@ loff_t mapping_seek_hole_data(struct address_space *mapping, loff_t start,
 	XA_STATE(xas, &mapping->i_pages, start >> PAGE_SHIFT);
 	pgoff_t max = (end - 1) >> PAGE_SHIFT;
 	bool seek_data = (whence == SEEK_DATA);
-	struct page *page;
+	struct folio *folio;
 
 	if (end <= start)
 		return -ENXIO;
 
 	rcu_read_lock();
-	while ((page = find_get_entry(&xas, max, XA_PRESENT))) {
+	while ((folio = find_get_entry(&xas, max, XA_PRESENT))) {
 		loff_t pos = (u64)xas.xa_index << PAGE_SHIFT;
-		unsigned int seek_size;
+		size_t seek_size;
 
 		if (start < pos) {
 			if (!seek_data)
@@ -2874,9 +2872,9 @@ loff_t mapping_seek_hole_data(struct address_space *mapping, loff_t start,
 			start = pos;
 		}
 
-		seek_size = seek_page_size(&xas, page);
-		pos = round_up(pos + 1, seek_size);
-		start = page_seek_hole_data(&xas, mapping, page, start, pos,
+		seek_size = seek_folio_size(&xas, folio);
+		pos = round_up((u64)pos + 1, seek_size);
+		start = folio_seek_hole_data(&xas, mapping, folio, start, pos,
 				seek_data);
 		if (start < pos)
 			goto unlock;
@@ -2884,15 +2882,15 @@ loff_t mapping_seek_hole_data(struct address_space *mapping, loff_t start,
 			break;
 		if (seek_size > PAGE_SIZE)
 			xas_set(&xas, pos >> PAGE_SHIFT);
-		if (!xa_is_value(page))
-			put_page(page);
+		if (!xa_is_value(folio))
+			folio_put(folio);
 	}
 	if (seek_data)
 		start = -ENXIO;
 unlock:
 	rcu_read_unlock();
-	if (page && !xa_is_value(page))
-		put_page(page);
+	if (folio && !xa_is_value(folio))
+		folio_put(folio);
 	if (start > end)
 		return end;
 	return start;
-- 
2.33.0

[PATCH 15/48] filemap: Remove thp_contains()

[Matthew Wilcox (Oracle)]

This function is now unused, so delete it.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 include/linux/pagemap.h | 9 ---------
 1 file changed, 9 deletions(-)

diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
index 3f26b191ede3..8c2cad7f0c36 100644
--- a/include/linux/pagemap.h
+++ b/include/linux/pagemap.h
@@ -512,15 +512,6 @@ static inline struct page *grab_cache_page_nowait(struct address_space *mapping,
 			mapping_gfp_mask(mapping));
 }
 
-/* Does this page contain this index? */
-static inline bool thp_contains(struct page *head, pgoff_t index)
-{
-	/* HugeTLBfs indexes the page cache in units of hpage_size */
-	if (PageHuge(head))
-		return head->index == index;
-	return page_index(head) == (index & ~(thp_nr_pages(head) - 1UL));
-}
-
 #define swapcache_index(folio)	__page_file_index(&(folio)->page)
 
 /**
-- 
2.33.0

[PATCH 16/48] filemap: Convert filemap_get_read_batch to use folios

[Matthew Wilcox (Oracle)]

The page cache only stores folios, never tail pages.  Saves 29 bytes
due to removing calls to compound_head().

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 mm/filemap.c | 23 +++++++++++------------
 1 file changed, 11 insertions(+), 12 deletions(-)

diff --git a/mm/filemap.c b/mm/filemap.c
index 2a51ec720e9e..120df74f3c7c 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -2339,32 +2339,31 @@ static void filemap_get_read_batch(struct address_space *mapping,
 		pgoff_t index, pgoff_t max, struct pagevec *pvec)
 {
 	XA_STATE(xas, &mapping->i_pages, index);
-	struct page *head;
+	struct folio *folio;
 
 	rcu_read_lock();
-	for (head = xas_load(&xas); head; head = xas_next(&xas)) {
-		if (xas_retry(&xas, head))
+	for (folio = xas_load(&xas); folio; folio = xas_next(&xas)) {
+		if (xas_retry(&xas, folio))
 			continue;
-		if (xas.xa_index > max || xa_is_value(head))
+		if (xas.xa_index > max || xa_is_value(folio))
 			break;
-		if (!page_cache_get_speculative(head))
+		if (!folio_try_get_rcu(folio))
 			goto retry;
 
-		/* Has the page moved or been split? */
-		if (unlikely(head != xas_reload(&xas)))
+		if (unlikely(folio != xas_reload(&xas)))
 			goto put_page;
 
-		if (!pagevec_add(pvec, head))
+		if (!pagevec_add(pvec, &folio->page))
 			break;
-		if (!PageUptodate(head))
+		if (!folio_test_uptodate(folio))
 			break;
-		if (PageReadahead(head))
+		if (folio_test_readahead(folio))
 			break;
-		xas.xa_index = head->index + thp_nr_pages(head) - 1;
+		xas.xa_index = folio->index + folio_nr_pages(folio) - 1;
 		xas.xa_offset = (xas.xa_index >> xas.xa_shift) & XA_CHUNK_MASK;
 		continue;
 put_page:
-		put_page(head);
+		folio_put(folio);
 retry:
 		xas_reset(&xas);
 	}
-- 
2.33.0

[PATCH 17/48] filemap: Convert find_get_pages_contig to folios

[Matthew Wilcox (Oracle)]

None of the callers of find_get_pages_contig() want tail pages.  They all
use order-0 pages today, but if they were converted, they'd want folios.
So just remove the call to find_subpage() instead of replacing it with
folio_page().

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 mm/filemap.c | 17 ++++++++---------
 1 file changed, 8 insertions(+), 9 deletions(-)

diff --git a/mm/filemap.c b/mm/filemap.c
index 120df74f3c7c..33e638f1ca34 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -2208,36 +2208,35 @@ unsigned find_get_pages_contig(struct address_space *mapping, pgoff_t index,
 			       unsigned int nr_pages, struct page **pages)
 {
 	XA_STATE(xas, &mapping->i_pages, index);
-	struct page *page;
+	struct folio *folio;
 	unsigned int ret = 0;
 
 	if (unlikely(!nr_pages))
 		return 0;
 
 	rcu_read_lock();
-	for (page = xas_load(&xas); page; page = xas_next(&xas)) {
-		if (xas_retry(&xas, page))
+	for (folio = xas_load(&xas); folio; folio = xas_next(&xas)) {
+		if (xas_retry(&xas, folio))
 			continue;
 		/*
 		 * If the entry has been swapped out, we can stop looking.
 		 * No current caller is looking for DAX entries.
 		 */
-		if (xa_is_value(page))
+		if (xa_is_value(folio))
 			break;
 
-		if (!page_cache_get_speculative(page))
+		if (!folio_try_get_rcu(folio))
 			goto retry;
 
-		/* Has the page moved or been split? */
-		if (unlikely(page != xas_reload(&xas)))
+		if (unlikely(folio != xas_reload(&xas)))
 			goto put_page;
 
-		pages[ret] = find_subpage(page, xas.xa_index);
+		pages[ret] = &folio->page;
 		if (++ret == nr_pages)
 			break;
 		continue;
 put_page:
-		put_page(page);
+		folio_put(folio);
 retry:
 		xas_reset(&xas);
 	}
-- 
2.33.0

[PATCH 18/48] filemap: Convert filemap_read_page to take a folio

[Matthew Wilcox (Oracle)]

One of the callers already had a folio; the other two grow by a few
bytes, but filemap_read_page() shrinks by 50 bytes for a net reduction
of 27 bytes.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 mm/filemap.c | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)

diff --git a/mm/filemap.c b/mm/filemap.c
index 33e638f1ca34..581f9fdb3406 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -2369,8 +2369,8 @@ static void filemap_get_read_batch(struct address_space *mapping,
 	rcu_read_unlock();
 }
 
-static int filemap_read_page(struct file *file, struct address_space *mapping,
-		struct page *page)
+static int filemap_read_folio(struct file *file, struct address_space *mapping,
+		struct folio *folio)
 {
 	int error;
 
@@ -2379,16 +2379,16 @@ static int filemap_read_page(struct file *file, struct address_space *mapping,
 	 * eg. multipath errors.  PG_error will be set again if readpage
 	 * fails.
 	 */
-	ClearPageError(page);
+	folio_clear_error(folio);
 	/* Start the actual read. The read will unlock the page. */
-	error = mapping->a_ops->readpage(file, page);
+	error = mapping->a_ops->readpage(file, &folio->page);
 	if (error)
 		return error;
 
-	error = wait_on_page_locked_killable(page);
+	error = folio_wait_locked_killable(folio);
 	if (error)
 		return error;
-	if (PageUptodate(page))
+	if (folio_test_uptodate(folio))
 		return 0;
 	shrink_readahead_size_eio(&file->f_ra);
 	return -EIO;
@@ -2464,7 +2464,7 @@ static int filemap_update_page(struct kiocb *iocb,
 	if (iocb->ki_flags & (IOCB_NOIO | IOCB_NOWAIT | IOCB_WAITQ))
 		goto unlock;
 
-	error = filemap_read_page(iocb->ki_filp, mapping, &folio->page);
+	error = filemap_read_folio(iocb->ki_filp, mapping, folio);
 	goto unlock_mapping;
 unlock:
 	folio_unlock(folio);
@@ -2506,7 +2506,7 @@ static int filemap_create_page(struct file *file,
 	if (error)
 		goto error;
 
-	error = filemap_read_page(file, mapping, page);
+	error = filemap_read_folio(file, mapping, page_folio(page));
 	if (error)
 		goto error;
 
@@ -3168,7 +3168,7 @@ vm_fault_t filemap_fault(struct vm_fault *vmf)
 	 * and we need to check for errors.
 	 */
 	fpin = maybe_unlock_mmap_for_io(vmf, fpin);
-	error = filemap_read_page(file, mapping, page);
+	error = filemap_read_folio(file, mapping, page_folio(page));
 	if (fpin)
 		goto out_retry;
 	put_page(page);
-- 
2.33.0

[PATCH 19/48] filemap: Convert filemap_create_page to folio

[Matthew Wilcox (Oracle)]

This is all internal to filemap and saves 100 bytes of text.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 mm/filemap.c | 39 ++++++++++++++++++++-------------------
 1 file changed, 20 insertions(+), 19 deletions(-)

diff --git a/mm/filemap.c b/mm/filemap.c
index 581f9fdb3406..b044afef78ef 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -2475,47 +2475,48 @@ static int filemap_update_page(struct kiocb *iocb,
 	return error;
 }
 
-static int filemap_create_page(struct file *file,
+static int filemap_create_folio(struct file *file,
 		struct address_space *mapping, pgoff_t index,
 		struct pagevec *pvec)
 {
-	struct page *page;
+	struct folio *folio;
 	int error;
 
-	page = page_cache_alloc(mapping);
-	if (!page)
+	folio = filemap_alloc_folio(mapping_gfp_mask(mapping), 0);
+	if (!folio)
 		return -ENOMEM;
 
 	/*
-	 * Protect against truncate / hole punch. Grabbing invalidate_lock here
-	 * assures we cannot instantiate and bring uptodate new pagecache pages
-	 * after evicting page cache during truncate and before actually
-	 * freeing blocks.  Note that we could release invalidate_lock after
-	 * inserting the page into page cache as the locked page would then be
-	 * enough to synchronize with hole punching. But there are code paths
-	 * such as filemap_update_page() filling in partially uptodate pages or
-	 * ->readpages() that need to hold invalidate_lock while mapping blocks
-	 * for IO so let's hold the lock here as well to keep locking rules
-	 * simple.
+	 * Protect against truncate / hole punch. Grabbing invalidate_lock
+	 * here assures we cannot instantiate and bring uptodate new
+	 * pagecache folios after evicting page cache during truncate
+	 * and before actually freeing blocks.	Note that we could
+	 * release invalidate_lock after inserting the folio into
+	 * the page cache as the locked folio would then be enough to
+	 * synchronize with hole punching. But there are code paths
+	 * such as filemap_update_page() filling in partially uptodate
+	 * pages or ->readpages() that need to hold invalidate_lock
+	 * while mapping blocks for IO so let's hold the lock here as
+	 * well to keep locking rules simple.
 	 */
 	filemap_invalidate_lock_shared(mapping);
-	error = add_to_page_cache_lru(page, mapping, index,
+	error = filemap_add_folio(mapping, folio, index,
 			mapping_gfp_constraint(mapping, GFP_KERNEL));
 	if (error == -EEXIST)
 		error = AOP_TRUNCATED_PAGE;
 	if (error)
 		goto error;
 
-	error = filemap_read_folio(file, mapping, page_folio(page));
+	error = filemap_read_folio(file, mapping, folio);
 	if (error)
 		goto error;
 
 	filemap_invalidate_unlock_shared(mapping);
-	pagevec_add(pvec, page);
+	pagevec_add(pvec, &folio->page);
 	return 0;
 error:
 	filemap_invalidate_unlock_shared(mapping);
-	put_page(page);
+	folio_put(folio);
 	return error;
 }
 
@@ -2557,7 +2558,7 @@ static int filemap_get_pages(struct kiocb *iocb, struct iov_iter *iter,
 	if (!pagevec_count(pvec)) {
 		if (iocb->ki_flags & (IOCB_NOWAIT | IOCB_WAITQ))
 			return -EAGAIN;
-		err = filemap_create_page(filp, mapping,
+		err = filemap_create_folio(filp, mapping,
 				iocb->ki_pos >> PAGE_SHIFT, pvec);
 		if (err == AOP_TRUNCATED_PAGE)
 			goto retry;
-- 
2.33.0

[PATCH 20/48] filemap: Convert filemap_range_uptodate to folios

[Matthew Wilcox (Oracle)]

The only caller was already passing a head page, so this simply avoids
a call to compound_head().

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 mm/filemap.c | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)

diff --git a/mm/filemap.c b/mm/filemap.c
index b044afef78ef..c4f887c277d0 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -2395,29 +2395,29 @@ static int filemap_read_folio(struct file *file, struct address_space *mapping,
 }
 
 static bool filemap_range_uptodate(struct address_space *mapping,
-		loff_t pos, struct iov_iter *iter, struct page *page)
+		loff_t pos, struct iov_iter *iter, struct folio *folio)
 {
 	int count;
 
-	if (PageUptodate(page))
+	if (folio_test_uptodate(folio))
 		return true;
 	/* pipes can't handle partially uptodate pages */
 	if (iov_iter_is_pipe(iter))
 		return false;
 	if (!mapping->a_ops->is_partially_uptodate)
 		return false;
-	if (mapping->host->i_blkbits >= (PAGE_SHIFT + thp_order(page)))
+	if (mapping->host->i_blkbits >= folio_shift(folio))
 		return false;
 
 	count = iter->count;
-	if (page_offset(page) > pos) {
-		count -= page_offset(page) - pos;
+	if (folio_pos(folio) > pos) {
+		count -= folio_pos(folio) - pos;
 		pos = 0;
 	} else {
-		pos -= page_offset(page);
+		pos -= folio_pos(folio);
 	}
 
-	return mapping->a_ops->is_partially_uptodate(page, pos, count);
+	return mapping->a_ops->is_partially_uptodate(&folio->page, pos, count);
 }
 
 static int filemap_update_page(struct kiocb *iocb,
@@ -2457,7 +2457,7 @@ static int filemap_update_page(struct kiocb *iocb,
 		goto unlock;
 
 	error = 0;
-	if (filemap_range_uptodate(mapping, iocb->ki_pos, iter, &folio->page))
+	if (filemap_range_uptodate(mapping, iocb->ki_pos, iter, folio))
 		goto unlock;
 
 	error = -EAGAIN;
-- 
2.33.0

[PATCH 21/48] readahead: Convert page_cache_async_ra() to take a folio

[Matthew Wilcox (Oracle)]

Using the folio here avoids checking whether it's a tail page.
This patch mostly just enables some of the following patches.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 include/linux/pagemap.h | 4 ++--
 mm/readahead.c          | 6 +++---
 2 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
index 8c2cad7f0c36..30302be6977f 100644
--- a/include/linux/pagemap.h
+++ b/include/linux/pagemap.h
@@ -993,7 +993,7 @@ struct readahead_control {
 void page_cache_ra_unbounded(struct readahead_control *,
 		unsigned long nr_to_read, unsigned long lookahead_count);
 void page_cache_sync_ra(struct readahead_control *, unsigned long req_count);
-void page_cache_async_ra(struct readahead_control *, struct page *,
+void page_cache_async_ra(struct readahead_control *, struct folio *,
 		unsigned long req_count);
 void readahead_expand(struct readahead_control *ractl,
 		      loff_t new_start, size_t new_len);
@@ -1040,7 +1040,7 @@ void page_cache_async_readahead(struct address_space *mapping,
 		struct page *page, pgoff_t index, unsigned long req_count)
 {
 	DEFINE_READAHEAD(ractl, file, ra, mapping, index);
-	page_cache_async_ra(&ractl, page, req_count);
+	page_cache_async_ra(&ractl, page_folio(page), req_count);
 }
 
 static inline struct folio *__readahead_folio(struct readahead_control *ractl)
diff --git a/mm/readahead.c b/mm/readahead.c
index 6ae5693de28c..e48e78641772 100644
--- a/mm/readahead.c
+++ b/mm/readahead.c
@@ -581,7 +581,7 @@ void page_cache_sync_ra(struct readahead_control *ractl,
 EXPORT_SYMBOL_GPL(page_cache_sync_ra);
 
 void page_cache_async_ra(struct readahead_control *ractl,
-		struct page *page, unsigned long req_count)
+		struct folio *folio, unsigned long req_count)
 {
 	/* no read-ahead */
 	if (!ractl->ra->ra_pages)
@@ -590,10 +590,10 @@ void page_cache_async_ra(struct readahead_control *ractl,
 	/*
 	 * Same bit is used for PG_readahead and PG_reclaim.
 	 */
-	if (PageWriteback(page))
+	if (folio_test_writeback(folio))
 		return;
 
-	ClearPageReadahead(page);
+	folio_clear_readahead(folio);
 
 	/*
 	 * Defer asynchronous read-ahead on IO congestion.
-- 
2.33.0

[PATCH 22/48] readahead: Convert page_cache_ra_unbounded to folios

[Matthew Wilcox (Oracle)]

This saves 99 bytes of kernel text.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 mm/readahead.c | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)

diff --git a/mm/readahead.c b/mm/readahead.c
index e48e78641772..cf0dcf89eb69 100644
--- a/mm/readahead.c
+++ b/mm/readahead.c
@@ -196,9 +196,9 @@ void page_cache_ra_unbounded(struct readahead_control *ractl,
 	 * Preallocate as many pages as we will need.
 	 */
 	for (i = 0; i < nr_to_read; i++) {
-		struct page *page = xa_load(&mapping->i_pages, index + i);
+		struct folio *folio = xa_load(&mapping->i_pages, index + i);
 
-		if (page && !xa_is_value(page)) {
+		if (folio && !xa_is_value(folio)) {
 			/*
 			 * Page already present?  Kick off the current batch
 			 * of contiguous pages before continuing with the
@@ -212,21 +212,21 @@ void page_cache_ra_unbounded(struct readahead_control *ractl,
 			continue;
 		}
 
-		page = __page_cache_alloc(gfp_mask);
-		if (!page)
+		folio = filemap_alloc_folio(gfp_mask, 0);
+		if (!folio)
 			break;
 		if (mapping->a_ops->readpages) {
-			page->index = index + i;
-			list_add(&page->lru, &page_pool);
-		} else if (add_to_page_cache_lru(page, mapping, index + i,
+			folio->index = index + i;
+			list_add(&folio->lru, &page_pool);
+		} else if (filemap_add_folio(mapping, folio, index + i,
 					gfp_mask) < 0) {
-			put_page(page);
+			folio_put(folio);
 			read_pages(ractl, &page_pool, true);
 			i = ractl->_index + ractl->_nr_pages - index - 1;
 			continue;
 		}
 		if (i == nr_to_read - lookahead_size)
-			SetPageReadahead(page);
+			folio_set_readahead(folio);
 		ractl->_nr_pages++;
 	}
 
-- 
2.33.0

[PATCH 23/48] filemap: Convert do_async_mmap_readahead to take a folio

[Matthew Wilcox (Oracle)]

Call page_cache_async_ra() directly instead of indirecting through
page_cache_async_readahead().

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 mm/filemap.c | 15 ++++++++-------
 1 file changed, 8 insertions(+), 7 deletions(-)

diff --git a/mm/filemap.c b/mm/filemap.c
index c4f887c277d0..0838b08557f5 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -3001,25 +3001,25 @@ static struct file *do_sync_mmap_readahead(struct vm_fault *vmf)
  * was pinned if we have to drop the mmap_lock in order to do IO.
  */
 static struct file *do_async_mmap_readahead(struct vm_fault *vmf,
-					    struct page *page)
+					    struct folio *folio)
 {
 	struct file *file = vmf->vma->vm_file;
 	struct file_ra_state *ra = &file->f_ra;
-	struct address_space *mapping = file->f_mapping;
+	DEFINE_READAHEAD(ractl, file, ra, file->f_mapping, vmf->pgoff);
 	struct file *fpin = NULL;
 	unsigned int mmap_miss;
-	pgoff_t offset = vmf->pgoff;
 
 	/* If we don't want any read-ahead, don't bother */
 	if (vmf->vma->vm_flags & VM_RAND_READ || !ra->ra_pages)
 		return fpin;
+
 	mmap_miss = READ_ONCE(ra->mmap_miss);
 	if (mmap_miss)
 		WRITE_ONCE(ra->mmap_miss, --mmap_miss);
-	if (PageReadahead(page)) {
+
+	if (folio_test_readahead(folio)) {
 		fpin = maybe_unlock_mmap_for_io(vmf, fpin);
-		page_cache_async_readahead(mapping, ra, file,
-					   page, offset, ra->ra_pages);
+		page_cache_async_ra(&ractl, folio, ra->ra_pages);
 	}
 	return fpin;
 }
@@ -3069,12 +3069,13 @@ vm_fault_t filemap_fault(struct vm_fault *vmf)
 	 */
 	page = find_get_page(mapping, offset);
 	if (likely(page)) {
+		struct folio *folio = page_folio(page);
 		/*
 		 * We found the page, so try async readahead before waiting for
 		 * the lock.
 		 */
 		if (!(vmf->flags & FAULT_FLAG_TRIED))
-			fpin = do_async_mmap_readahead(vmf, page);
+			fpin = do_async_mmap_readahead(vmf, folio);
 		if (unlikely(!PageUptodate(page))) {
 			filemap_invalidate_lock_shared(mapping);
 			mapping_locked = true;
-- 
2.33.0

[PATCH 24/48] filemap: Convert filemap_fault to folio

[Matthew Wilcox (Oracle)]

Instead of converting back-and-forth between the actual page and
the head page, just convert once at the end of the function where we
set the vmf->page.  Saves 241 bytes of text, or 15% of the size of
filemap_fault().

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 mm/filemap.c | 77 +++++++++++++++++++++++++---------------------------
 1 file changed, 37 insertions(+), 40 deletions(-)

diff --git a/mm/filemap.c b/mm/filemap.c
index 0838b08557f5..fc0f1d9904d2 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -2898,21 +2898,20 @@ loff_t mapping_seek_hole_data(struct address_space *mapping, loff_t start,
 #ifdef CONFIG_MMU
 #define MMAP_LOTSAMISS  (100)
 /*
- * lock_page_maybe_drop_mmap - lock the page, possibly dropping the mmap_lock
+ * lock_folio_maybe_drop_mmap - lock the page, possibly dropping the mmap_lock
  * @vmf - the vm_fault for this fault.
- * @page - the page to lock.
+ * @folio - the folio to lock.
  * @fpin - the pointer to the file we may pin (or is already pinned).
  *
- * This works similar to lock_page_or_retry in that it can drop the mmap_lock.
- * It differs in that it actually returns the page locked if it returns 1 and 0
- * if it couldn't lock the page.  If we did have to drop the mmap_lock then fpin
- * will point to the pinned file and needs to be fput()'ed at a later point.
+ * This works similar to lock_folio_or_retry in that it can drop the
+ * mmap_lock.  It differs in that it actually returns the folio locked
+ * if it returns 1 and 0 if it couldn't lock the folio.  If we did have
+ * to drop the mmap_lock then fpin will point to the pinned file and
+ * needs to be fput()'ed at a later point.
  */
-static int lock_page_maybe_drop_mmap(struct vm_fault *vmf, struct page *page,
+static int lock_folio_maybe_drop_mmap(struct vm_fault *vmf, struct folio *folio,
 				     struct file **fpin)
 {
-	struct folio *folio = page_folio(page);
-
 	if (folio_trylock(folio))
 		return 1;
 
@@ -3038,7 +3037,7 @@ static struct file *do_async_mmap_readahead(struct vm_fault *vmf,
  * vma->vm_mm->mmap_lock must be held on entry.
  *
  * If our return value has VM_FAULT_RETRY set, it's because the mmap_lock
- * may be dropped before doing I/O or by lock_page_maybe_drop_mmap().
+ * may be dropped before doing I/O or by lock_folio_maybe_drop_mmap().
  *
  * If our return value does not have VM_FAULT_RETRY set, the mmap_lock
  * has not been released.
@@ -3054,29 +3053,27 @@ vm_fault_t filemap_fault(struct vm_fault *vmf)
 	struct file *fpin = NULL;
 	struct address_space *mapping = file->f_mapping;
 	struct inode *inode = mapping->host;
-	pgoff_t offset = vmf->pgoff;
-	pgoff_t max_off;
-	struct page *page;
+	pgoff_t max_idx, index = vmf->pgoff;
+	struct folio *folio;
 	vm_fault_t ret = 0;
 	bool mapping_locked = false;
 
-	max_off = DIV_ROUND_UP(i_size_read(inode), PAGE_SIZE);
-	if (unlikely(offset >= max_off))
+	max_idx = DIV_ROUND_UP(i_size_read(inode), PAGE_SIZE);
+	if (unlikely(index >= max_idx))
 		return VM_FAULT_SIGBUS;
 
 	/*
 	 * Do we have something in the page cache already?
 	 */
-	page = find_get_page(mapping, offset);
-	if (likely(page)) {
-		struct folio *folio = page_folio(page);
+	folio = filemap_get_folio(mapping, index);
+	if (likely(folio)) {
 		/*
 		 * We found the page, so try async readahead before waiting for
 		 * the lock.
 		 */
 		if (!(vmf->flags & FAULT_FLAG_TRIED))
 			fpin = do_async_mmap_readahead(vmf, folio);
-		if (unlikely(!PageUptodate(page))) {
+		if (unlikely(!folio_test_uptodate(folio))) {
 			filemap_invalidate_lock_shared(mapping);
 			mapping_locked = true;
 		}
@@ -3088,17 +3085,17 @@ vm_fault_t filemap_fault(struct vm_fault *vmf)
 		fpin = do_sync_mmap_readahead(vmf);
 retry_find:
 		/*
-		 * See comment in filemap_create_page() why we need
+		 * See comment in filemap_create_folio() why we need
 		 * invalidate_lock
 		 */
 		if (!mapping_locked) {
 			filemap_invalidate_lock_shared(mapping);
 			mapping_locked = true;
 		}
-		page = pagecache_get_page(mapping, offset,
+		folio = __filemap_get_folio(mapping, index,
 					  FGP_CREAT|FGP_FOR_MMAP,
 					  vmf->gfp_mask);
-		if (!page) {
+		if (!folio) {
 			if (fpin)
 				goto out_retry;
 			filemap_invalidate_unlock_shared(mapping);
@@ -3106,22 +3103,22 @@ vm_fault_t filemap_fault(struct vm_fault *vmf)
 		}
 	}
 
-	if (!lock_page_maybe_drop_mmap(vmf, page, &fpin))
+	if (!lock_folio_maybe_drop_mmap(vmf, folio, &fpin))
 		goto out_retry;
 
 	/* Did it get truncated? */
-	if (unlikely(compound_head(page)->mapping != mapping)) {
-		unlock_page(page);
-		put_page(page);
+	if (unlikely(folio->mapping != mapping)) {
+		folio_unlock(folio);
+		folio_put(folio);
 		goto retry_find;
 	}
-	VM_BUG_ON_PAGE(page_to_pgoff(page) != offset, page);
+	VM_BUG_ON_FOLIO(!folio_contains(folio, index), folio);
 
 	/*
 	 * We have a locked page in the page cache, now we need to check
 	 * that it's up-to-date. If not, it is going to be due to an error.
 	 */
-	if (unlikely(!PageUptodate(page))) {
+	if (unlikely(!folio_test_uptodate(folio))) {
 		/*
 		 * The page was in cache and uptodate and now it is not.
 		 * Strange but possible since we didn't hold the page lock all
@@ -3129,8 +3126,8 @@ vm_fault_t filemap_fault(struct vm_fault *vmf)
 		 * try again.
 		 */
 		if (!mapping_locked) {
-			unlock_page(page);
-			put_page(page);
+			folio_unlock(folio);
+			folio_put(folio);
 			goto retry_find;
 		}
 		goto page_not_uptodate;
@@ -3142,7 +3139,7 @@ vm_fault_t filemap_fault(struct vm_fault *vmf)
 	 * redo the fault.
 	 */
 	if (fpin) {
-		unlock_page(page);
+		folio_unlock(folio);
 		goto out_retry;
 	}
 	if (mapping_locked)
@@ -3152,14 +3149,14 @@ vm_fault_t filemap_fault(struct vm_fault *vmf)
 	 * Found the page and have a reference on it.
 	 * We must recheck i_size under page lock.
 	 */
-	max_off = DIV_ROUND_UP(i_size_read(inode), PAGE_SIZE);
-	if (unlikely(offset >= max_off)) {
-		unlock_page(page);
-		put_page(page);
+	max_idx = DIV_ROUND_UP(i_size_read(inode), PAGE_SIZE);
+	if (unlikely(index >= max_idx)) {
+		folio_unlock(folio);
+		folio_put(folio);
 		return VM_FAULT_SIGBUS;
 	}
 
-	vmf->page = page;
+	vmf->page = folio_file_page(folio, index);
 	return ret | VM_FAULT_LOCKED;
 
 page_not_uptodate:
@@ -3170,10 +3167,10 @@ vm_fault_t filemap_fault(struct vm_fault *vmf)
 	 * and we need to check for errors.
 	 */
 	fpin = maybe_unlock_mmap_for_io(vmf, fpin);
-	error = filemap_read_folio(file, mapping, page_folio(page));
+	error = filemap_read_folio(file, mapping, folio);
 	if (fpin)
 		goto out_retry;
-	put_page(page);
+	folio_put(folio);
 
 	if (!error || error == AOP_TRUNCATED_PAGE)
 		goto retry_find;
@@ -3187,8 +3184,8 @@ vm_fault_t filemap_fault(struct vm_fault *vmf)
 	 * re-find the vma and come back and find our hopefully still populated
 	 * page.
 	 */
-	if (page)
-		put_page(page);
+	if (folio)
+		folio_put(folio);
 	if (mapping_locked)
 		filemap_invalidate_unlock_shared(mapping);
 	if (fpin)
-- 
2.33.0

[PATCH 25/48] filemap: Add read_cache_folio and read_mapping_folio

[Matthew Wilcox (Oracle)]

Reimplement read_cache_page() as a wrapper around read_cache_folio().
Saves over 400 bytes of text from do_read_cache_folio() which more
thn makes up for the extra 100 bytes of text added to the various
wrapper functions.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 include/linux/pagemap.h | 12 +++++-
 mm/filemap.c            | 95 +++++++++++++++++++++--------------------
 2 files changed, 59 insertions(+), 48 deletions(-)

diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
index 30302be6977f..7bef50ea5435 100644
--- a/include/linux/pagemap.h
+++ b/include/linux/pagemap.h
@@ -629,8 +629,10 @@ static inline struct page *grab_cache_page(struct address_space *mapping,
 	return find_or_create_page(mapping, index, mapping_gfp_mask(mapping));
 }
 
-extern struct page * read_cache_page(struct address_space *mapping,
-				pgoff_t index, filler_t *filler, void *data);
+struct folio *read_cache_folio(struct address_space *, pgoff_t index,
+		filler_t *filler, void *data);
+struct page *read_cache_page(struct address_space *, pgoff_t index,
+		filler_t *filler, void *data);
 extern struct page * read_cache_page_gfp(struct address_space *mapping,
 				pgoff_t index, gfp_t gfp_mask);
 extern int read_cache_pages(struct address_space *mapping,
@@ -642,6 +644,12 @@ static inline struct page *read_mapping_page(struct address_space *mapping,
 	return read_cache_page(mapping, index, NULL, data);
 }
 
+static inline struct folio *read_mapping_folio(struct address_space *mapping,
+				pgoff_t index, void *data)
+{
+	return read_cache_folio(mapping, index, NULL, data);
+}
+
 /*
  * Get index of the page within radix-tree (but not for hugetlb pages).
  * (TODO: remove once hugetlb pages will have ->index in PAGE_SIZE)
diff --git a/mm/filemap.c b/mm/filemap.c
index fc0f1d9904d2..f34dda0a7627 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -3418,35 +3418,20 @@ EXPORT_SYMBOL(filemap_page_mkwrite);
 EXPORT_SYMBOL(generic_file_mmap);
 EXPORT_SYMBOL(generic_file_readonly_mmap);
 
-static struct page *wait_on_page_read(struct page *page)
+static struct folio *do_read_cache_folio(struct address_space *mapping,
+		pgoff_t index, filler_t filler, void *data, gfp_t gfp)
 {
-	if (!IS_ERR(page)) {
-		wait_on_page_locked(page);
-		if (!PageUptodate(page)) {
-			put_page(page);
-			page = ERR_PTR(-EIO);
-		}
-	}
-	return page;
-}
-
-static struct page *do_read_cache_page(struct address_space *mapping,
-				pgoff_t index,
-				int (*filler)(void *, struct page *),
-				void *data,
-				gfp_t gfp)
-{
-	struct page *page;
+	struct folio *folio;
 	int err;
 repeat:
-	page = find_get_page(mapping, index);
-	if (!page) {
-		page = __page_cache_alloc(gfp);
-		if (!page)
+	folio = filemap_get_folio(mapping, index);
+	if (!folio) {
+		folio = filemap_alloc_folio(gfp, 0);
+		if (!folio)
 			return ERR_PTR(-ENOMEM);
-		err = add_to_page_cache_lru(page, mapping, index, gfp);
+		err = filemap_add_folio(mapping, folio, index, gfp);
 		if (unlikely(err)) {
-			put_page(page);
+			folio_put(folio);
 			if (err == -EEXIST)
 				goto repeat;
 			/* Presumably ENOMEM for xarray node */
@@ -3455,21 +3440,24 @@ static struct page *do_read_cache_page(struct address_space *mapping,
 
 filler:
 		if (filler)
-			err = filler(data, page);
+			err = filler(data, &folio->page);
 		else
-			err = mapping->a_ops->readpage(data, page);
+			err = mapping->a_ops->readpage(data, &folio->page);
 
 		if (err < 0) {
-			put_page(page);
+			folio_put(folio);
 			return ERR_PTR(err);
 		}
 
-		page = wait_on_page_read(page);
-		if (IS_ERR(page))
-			return page;
+		folio_wait_locked(folio);
+		if (!folio_test_uptodate(folio)) {
+			folio_put(folio);
+			return ERR_PTR(-EIO);
+		}
+
 		goto out;
 	}
-	if (PageUptodate(page))
+	if (folio_test_uptodate(folio))
 		goto out;
 
 	/*
@@ -3503,23 +3491,23 @@ static struct page *do_read_cache_page(struct address_space *mapping,
 	 * avoid spurious serialisations and wakeups when multiple processes
 	 * wait on the same page for IO to complete.
 	 */
-	wait_on_page_locked(page);
-	if (PageUptodate(page))
+	folio_wait_locked(folio);
+	if (folio_test_uptodate(folio))
 		goto out;
 
 	/* Distinguish between all the cases under the safety of the lock */
-	lock_page(page);
+	folio_lock(folio);
 
 	/* Case c or d, restart the operation */
-	if (!page->mapping) {
-		unlock_page(page);
-		put_page(page);
+	if (!folio->mapping) {
+		folio_unlock(folio);
+		folio_put(folio);
 		goto repeat;
 	}
 
 	/* Someone else locked and filled the page in a very small window */
-	if (PageUptodate(page)) {
-		unlock_page(page);
+	if (folio_test_uptodate(folio)) {
+		folio_unlock(folio);
 		goto out;
 	}
 
@@ -3529,16 +3517,16 @@ static struct page *do_read_cache_page(struct address_space *mapping,
 	 * Clear page error before actual read, PG_error will be
 	 * set again if read page fails.
 	 */
-	ClearPageError(page);
+	folio_clear_error(folio);
 	goto filler;
 
 out:
-	mark_page_accessed(page);
-	return page;
+	folio_mark_accessed(folio);
+	return folio;
 }
 
 /**
- * read_cache_page - read into page cache, fill it if needed
+ * read_cache_folio - read into page cache, fill it if needed
  * @mapping:	the page's address_space
  * @index:	the page index
  * @filler:	function to perform the read
@@ -3553,10 +3541,25 @@ static struct page *do_read_cache_page(struct address_space *mapping,
  *
  * Return: up to date page on success, ERR_PTR() on failure.
  */
+struct folio *read_cache_folio(struct address_space *mapping, pgoff_t index,
+		filler_t filler, void *data)
+{
+	return do_read_cache_folio(mapping, index, filler, data,
+			mapping_gfp_mask(mapping));
+}
+EXPORT_SYMBOL(read_cache_folio);
+
+static struct page *do_read_cache_page(struct address_space *mapping,
+		pgoff_t index, filler_t *filler, void *data, gfp_t gfp)
+{
+	struct folio *folio = read_cache_folio(mapping, index, filler, data);
+	if (IS_ERR(folio))
+		return &folio->page;
+	return folio_file_page(folio, index);
+}
+
 struct page *read_cache_page(struct address_space *mapping,
-				pgoff_t index,
-				int (*filler)(void *, struct page *),
-				void *data)
+				pgoff_t index, filler_t *filler, void *data)
 {
 	return do_read_cache_page(mapping, index, filler, data,
 			mapping_gfp_mask(mapping));
-- 
2.33.0

[PATCH 26/48] filemap: Convert filemap_get_pages to use folios

[Matthew Wilcox (Oracle)]

This saves a few calls to compound_head(), including one in
filemap_update_page().  Shrinks the kernel by 78 bytes.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 mm/filemap.c | 24 ++++++++++++------------
 1 file changed, 12 insertions(+), 12 deletions(-)

diff --git a/mm/filemap.c b/mm/filemap.c
index f34dda0a7627..d191a4fd758a 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -2422,9 +2422,8 @@ static bool filemap_range_uptodate(struct address_space *mapping,
 
 static int filemap_update_page(struct kiocb *iocb,
 		struct address_space *mapping, struct iov_iter *iter,
-		struct page *page)
+		struct folio *folio)
 {
-	struct folio *folio = page_folio(page);
 	int error;
 
 	if (iocb->ki_flags & IOCB_NOWAIT) {
@@ -2521,13 +2520,14 @@ static int filemap_create_folio(struct file *file,
 }
 
 static int filemap_readahead(struct kiocb *iocb, struct file *file,
-		struct address_space *mapping, struct page *page,
+		struct address_space *mapping, struct folio *folio,
 		pgoff_t last_index)
 {
+	DEFINE_READAHEAD(ractl, file, &file->f_ra, mapping, folio->index);
+
 	if (iocb->ki_flags & IOCB_NOIO)
 		return -EAGAIN;
-	page_cache_async_readahead(mapping, &file->f_ra, file, page,
-			page->index, last_index - page->index);
+	page_cache_async_ra(&ractl, folio, last_index - folio->index);
 	return 0;
 }
 
@@ -2539,7 +2539,7 @@ static int filemap_get_pages(struct kiocb *iocb, struct iov_iter *iter,
 	struct file_ra_state *ra = &filp->f_ra;
 	pgoff_t index = iocb->ki_pos >> PAGE_SHIFT;
 	pgoff_t last_index;
-	struct page *page;
+	struct folio *folio;
 	int err = 0;
 
 	last_index = DIV_ROUND_UP(iocb->ki_pos + iter->count, PAGE_SIZE);
@@ -2565,16 +2565,16 @@ static int filemap_get_pages(struct kiocb *iocb, struct iov_iter *iter,
 		return err;
 	}
 
-	page = pvec->pages[pagevec_count(pvec) - 1];
-	if (PageReadahead(page)) {
-		err = filemap_readahead(iocb, filp, mapping, page, last_index);
+	folio = page_folio(pvec->pages[pagevec_count(pvec) - 1]);
+	if (folio_test_readahead(folio)) {
+		err = filemap_readahead(iocb, filp, mapping, folio, last_index);
 		if (err)
 			goto err;
 	}
-	if (!PageUptodate(page)) {
+	if (!folio_test_uptodate(folio)) {
 		if ((iocb->ki_flags & IOCB_WAITQ) && pagevec_count(pvec) > 1)
 			iocb->ki_flags |= IOCB_NOWAIT;
-		err = filemap_update_page(iocb, mapping, iter, page);
+		err = filemap_update_page(iocb, mapping, iter, folio);
 		if (err)
 			goto err;
 	}
@@ -2582,7 +2582,7 @@ static int filemap_get_pages(struct kiocb *iocb, struct iov_iter *iter,
 	return 0;
 err:
 	if (err < 0)
-		put_page(page);
+		folio_put(folio);
 	if (likely(--pvec->nr))
 		return 0;
 	if (err == AOP_TRUNCATED_PAGE)
-- 
2.33.0

[PATCH 27/48] filemap: Convert page_cache_delete_batch to folios

[Matthew Wilcox (Oracle)]

Saves one call to compound_head() and reduces text size by 15 bytes.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 mm/filemap.c | 20 ++++++++++----------
 1 file changed, 10 insertions(+), 10 deletions(-)

diff --git a/mm/filemap.c b/mm/filemap.c
index d191a4fd758a..3ea81adbabd8 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -290,15 +290,15 @@ static void page_cache_delete_batch(struct address_space *mapping,
 	XA_STATE(xas, &mapping->i_pages, pvec->pages[0]->index);
 	int total_pages = 0;
 	int i = 0;
-	struct page *page;
+	struct folio *folio;
 
 	mapping_set_update(&xas, mapping);
-	xas_for_each(&xas, page, ULONG_MAX) {
+	xas_for_each(&xas, folio, ULONG_MAX) {
 		if (i >= pagevec_count(pvec))
 			break;
 
 		/* A swap/dax/shadow entry got inserted? Skip it. */
-		if (xa_is_value(page))
+		if (xa_is_value(folio))
 			continue;
 		/*
 		 * A page got inserted in our range? Skip it. We have our
@@ -307,16 +307,16 @@ static void page_cache_delete_batch(struct address_space *mapping,
 		 * means our page has been removed, which shouldn't be
 		 * possible because we're holding the PageLock.
 		 */
-		if (page != pvec->pages[i]) {
-			VM_BUG_ON_PAGE(page->index > pvec->pages[i]->index,
-					page);
+		if (&folio->page != pvec->pages[i]) {
+			VM_BUG_ON_FOLIO(folio->index >
+						pvec->pages[i]->index, folio);
 			continue;
 		}
 
-		WARN_ON_ONCE(!PageLocked(page));
+		WARN_ON_ONCE(!folio_test_locked(folio));
 
-		if (page->index == xas.xa_index)
-			page->mapping = NULL;
+		if (folio->index == xas.xa_index)
+			folio->mapping = NULL;
 		/* Leave page->index set: truncation lookup relies on it */
 
 		/*
@@ -324,7 +324,7 @@ static void page_cache_delete_batch(struct address_space *mapping,
 		 * page or the index is of the last sub-page of this compound
 		 * page.
 		 */
-		if (page->index + compound_nr(page) - 1 == xas.xa_index)
+		if (folio->index + folio_nr_pages(folio) - 1 == xas.xa_index)
 			i++;
 		xas_store(&xas, NULL);
 		total_pages++;
-- 
2.33.0

[PATCH 28/48] filemap: Use folios in next_uptodate_page

[Matthew Wilcox (Oracle)]

This saves 105 bytes of text.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 mm/filemap.c | 30 +++++++++++++++---------------
 1 file changed, 15 insertions(+), 15 deletions(-)

diff --git a/mm/filemap.c b/mm/filemap.c
index 3ea81adbabd8..47880ec789f4 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -3227,43 +3227,43 @@ static bool filemap_map_pmd(struct vm_fault *vmf, struct page *page)
 	return false;
 }
 
-static struct page *next_uptodate_page(struct page *page,
+static struct page *next_uptodate_page(struct folio *folio,
 				       struct address_space *mapping,
 				       struct xa_state *xas, pgoff_t end_pgoff)
 {
 	unsigned long max_idx;
 
 	do {
-		if (!page)
+		if (!folio)
 			return NULL;
-		if (xas_retry(xas, page))
+		if (xas_retry(xas, folio))
 			continue;
-		if (xa_is_value(page))
+		if (xa_is_value(folio))
 			continue;
-		if (PageLocked(page))
+		if (folio_test_locked(folio))
 			continue;
-		if (!page_cache_get_speculative(page))
+		if (!folio_try_get_rcu(folio))
 			continue;
 		/* Has the page moved or been split? */
-		if (unlikely(page != xas_reload(xas)))
+		if (unlikely(folio != xas_reload(xas)))
 			goto skip;
-		if (!PageUptodate(page) || PageReadahead(page))
+		if (!folio_test_uptodate(folio) || folio_test_readahead(folio))
 			goto skip;
-		if (!trylock_page(page))
+		if (!folio_trylock(folio))
 			goto skip;
-		if (page->mapping != mapping)
+		if (folio->mapping != mapping)
 			goto unlock;
-		if (!PageUptodate(page))
+		if (!folio_test_uptodate(folio))
 			goto unlock;
 		max_idx = DIV_ROUND_UP(i_size_read(mapping->host), PAGE_SIZE);
 		if (xas->xa_index >= max_idx)
 			goto unlock;
-		return page;
+		return &folio->page;
 unlock:
-		unlock_page(page);
+		folio_unlock(folio);
 skip:
-		put_page(page);
-	} while ((page = xas_next_entry(xas, end_pgoff)) != NULL);
+		folio_put(folio);
+	} while ((folio = xas_next_entry(xas, end_pgoff)) != NULL);
 
 	return NULL;
 }
-- 
2.33.0

[PATCH 29/48] filemap: Use a folio in filemap_map_pages

[Matthew Wilcox (Oracle)]

Saves 61 bytes due to fewer calls to compound_head().

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 mm/filemap.c | 27 ++++++++++++++-------------
 1 file changed, 14 insertions(+), 13 deletions(-)

diff --git a/mm/filemap.c b/mm/filemap.c
index 47880ec789f4..8cca04a79808 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -3227,7 +3227,7 @@ static bool filemap_map_pmd(struct vm_fault *vmf, struct page *page)
 	return false;
 }
 
-static struct page *next_uptodate_page(struct folio *folio,
+static struct folio *next_uptodate_page(struct folio *folio,
 				       struct address_space *mapping,
 				       struct xa_state *xas, pgoff_t end_pgoff)
 {
@@ -3258,7 +3258,7 @@ static struct page *next_uptodate_page(struct folio *folio,
 		max_idx = DIV_ROUND_UP(i_size_read(mapping->host), PAGE_SIZE);
 		if (xas->xa_index >= max_idx)
 			goto unlock;
-		return &folio->page;
+		return folio;
 unlock:
 		folio_unlock(folio);
 skip:
@@ -3268,7 +3268,7 @@ static struct page *next_uptodate_page(struct folio *folio,
 	return NULL;
 }
 
-static inline struct page *first_map_page(struct address_space *mapping,
+static inline struct folio *first_map_page(struct address_space *mapping,
 					  struct xa_state *xas,
 					  pgoff_t end_pgoff)
 {
@@ -3276,7 +3276,7 @@ static inline struct page *first_map_page(struct address_space *mapping,
 				  mapping, xas, end_pgoff);
 }
 
-static inline struct page *next_map_page(struct address_space *mapping,
+static inline struct folio *next_map_page(struct address_space *mapping,
 					 struct xa_state *xas,
 					 pgoff_t end_pgoff)
 {
@@ -3293,16 +3293,17 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf,
 	pgoff_t last_pgoff = start_pgoff;
 	unsigned long addr;
 	XA_STATE(xas, &mapping->i_pages, start_pgoff);
-	struct page *head, *page;
+	struct folio *folio;
+	struct page *page;
 	unsigned int mmap_miss = READ_ONCE(file->f_ra.mmap_miss);
 	vm_fault_t ret = 0;
 
 	rcu_read_lock();
-	head = first_map_page(mapping, &xas, end_pgoff);
-	if (!head)
+	folio = first_map_page(mapping, &xas, end_pgoff);
+	if (!folio)
 		goto out;
 
-	if (filemap_map_pmd(vmf, head)) {
+	if (filemap_map_pmd(vmf, &folio->page)) {
 		ret = VM_FAULT_NOPAGE;
 		goto out;
 	}
@@ -3310,7 +3311,7 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf,
 	addr = vma->vm_start + ((start_pgoff - vma->vm_pgoff) << PAGE_SHIFT);
 	vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, addr, &vmf->ptl);
 	do {
-		page = find_subpage(head, xas.xa_index);
+		page = folio_file_page(folio, xas.xa_index);
 		if (PageHWPoison(page))
 			goto unlock;
 
@@ -3331,12 +3332,12 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf,
 		do_set_pte(vmf, page, addr);
 		/* no need to invalidate: a not-present page won't be cached */
 		update_mmu_cache(vma, addr, vmf->pte);
-		unlock_page(head);
+		folio_unlock(folio);
 		continue;
 unlock:
-		unlock_page(head);
-		put_page(head);
-	} while ((head = next_map_page(mapping, &xas, end_pgoff)) != NULL);
+		folio_unlock(folio);
+		folio_put(folio);
+	} while ((folio = next_map_page(mapping, &xas, end_pgoff)) != NULL);
 	pte_unmap_unlock(vmf->pte, vmf->ptl);
 out:
 	rcu_read_unlock();
-- 
2.33.0

[PATCH 30/48] filemap: Use a folio in filemap_page_mkwrite

[Matthew Wilcox (Oracle)]

This fixes a bug for tail pages.  They always have a NULL mapping, so
the check would fail and we would never mark the folio as dirty.
Ends up growing the kernel by 19 bytes although there will be fewer
calls to compound_head() dynamically.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 mm/filemap.c | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)

diff --git a/mm/filemap.c b/mm/filemap.c
index 8cca04a79808..4ae9d5befffa 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -3349,24 +3349,24 @@ EXPORT_SYMBOL(filemap_map_pages);
 vm_fault_t filemap_page_mkwrite(struct vm_fault *vmf)
 {
 	struct address_space *mapping = vmf->vma->vm_file->f_mapping;
-	struct page *page = vmf->page;
+	struct folio *folio = page_folio(vmf->page);
 	vm_fault_t ret = VM_FAULT_LOCKED;
 
 	sb_start_pagefault(mapping->host->i_sb);
 	file_update_time(vmf->vma->vm_file);
-	lock_page(page);
-	if (page->mapping != mapping) {
-		unlock_page(page);
+	folio_lock(folio);
+	if (folio->mapping != mapping) {
+		folio_unlock(folio);
 		ret = VM_FAULT_NOPAGE;
 		goto out;
 	}
 	/*
-	 * We mark the page dirty already here so that when freeze is in
+	 * We mark the folio dirty already here so that when freeze is in
 	 * progress, we are guaranteed that writeback during freezing will
-	 * see the dirty page and writeprotect it again.
+	 * see the dirty folio and writeprotect it again.
 	 */
-	set_page_dirty(page);
-	wait_for_stable_page(page);
+	folio_mark_dirty(folio);
+	folio_wait_stable(folio);
 out:
 	sb_end_pagefault(mapping->host->i_sb);
 	return ret;
-- 
2.33.0

[PATCH 31/48] filemap: Add filemap_release_folio()

[Matthew Wilcox (Oracle)]

Reimplement try_to_release_page() as a wrapper around
filemap_release_folio().

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 include/linux/mm.h      |  1 -
 include/linux/pagemap.h |  2 ++
 mm/filemap.c            | 39 +++++++++++++++++++--------------------
 mm/folio-compat.c       |  6 ++++++
 4 files changed, 27 insertions(+), 21 deletions(-)

diff --git a/include/linux/mm.h b/include/linux/mm.h
index 72ca04f16711..145f045b0ddc 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -1970,7 +1970,6 @@ int get_kernel_pages(const struct kvec *iov, int nr_pages, int write,
 			struct page **pages);
 struct page *get_dump_page(unsigned long addr);
 
-extern int try_to_release_page(struct page * page, gfp_t gfp_mask);
 extern void do_invalidatepage(struct page *page, unsigned int offset,
 			      unsigned int length);
 
diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
index 7bef50ea5435..eb6e58e106c8 100644
--- a/include/linux/pagemap.h
+++ b/include/linux/pagemap.h
@@ -939,6 +939,8 @@ static inline void __delete_from_page_cache(struct page *page, void *shadow)
 void replace_page_cache_page(struct page *old, struct page *new);
 void delete_from_page_cache_batch(struct address_space *mapping,
 				  struct pagevec *pvec);
+int try_to_release_page(struct page *page, gfp_t gfp);
+bool filemap_release_folio(struct folio *folio, gfp_t gfp);
 loff_t mapping_seek_hole_data(struct address_space *, loff_t start, loff_t end,
 		int whence);
 
diff --git a/mm/filemap.c b/mm/filemap.c
index 4ae9d5befffa..7a418f0012e5 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -3920,33 +3920,32 @@ ssize_t generic_file_write_iter(struct kiocb *iocb, struct iov_iter *from)
 EXPORT_SYMBOL(generic_file_write_iter);
 
 /**
- * try_to_release_page() - release old fs-specific metadata on a page
+ * filemap_release_folio() - Release fs-specific metadata on a folio.
+ * @folio: The folio which the kernel is trying to free.
+ * @gfp: Memory allocation flags (and I/O mode).
  *
- * @page: the page which the kernel is trying to free
- * @gfp_mask: memory allocation flags (and I/O mode)
+ * The address_space is trying to release any data attached to a folio
+ * (presumably at folio->private).
  *
- * The address_space is to try to release any data against the page
- * (presumably at page->private).
+ * This will also be called if the private_2 flag is set on a page,
+ * indicating that the folio has other metadata associated with it.
  *
- * This may also be called if PG_fscache is set on a page, indicating that the
- * page is known to the local caching routines.
+ * The @gfp argument specifies whether I/O may be performed to release
+ * this page (__GFP_IO), and whether the call may block
+ * (__GFP_RECLAIM & __GFP_FS).
  *
- * The @gfp_mask argument specifies whether I/O may be performed to release
- * this page (__GFP_IO), and whether the call may block (__GFP_RECLAIM & __GFP_FS).
- *
- * Return: %1 if the release was successful, otherwise return zero.
+ * Return: %true if the release was successful, otherwise %false.
  */
-int try_to_release_page(struct page *page, gfp_t gfp_mask)
+bool filemap_release_folio(struct folio *folio, gfp_t gfp)
 {
-	struct address_space * const mapping = page->mapping;
+	struct address_space * const mapping = folio->mapping;
 
-	BUG_ON(!PageLocked(page));
-	if (PageWriteback(page))
-		return 0;
+	BUG_ON(!folio_test_locked(folio));
+	if (folio_test_writeback(folio))
+		return false;
 
 	if (mapping && mapping->a_ops->releasepage)
-		return mapping->a_ops->releasepage(page, gfp_mask);
-	return try_to_free_buffers(page);
+		return mapping->a_ops->releasepage(&folio->page, gfp);
+	return try_to_free_buffers(&folio->page);
 }
-
-EXPORT_SYMBOL(try_to_release_page);
+EXPORT_SYMBOL(filemap_release_folio);
diff --git a/mm/folio-compat.c b/mm/folio-compat.c
index 749a695b4217..749555a232a8 100644
--- a/mm/folio-compat.c
+++ b/mm/folio-compat.c
@@ -145,3 +145,9 @@ void delete_from_page_cache(struct page *page)
 {
 	return filemap_remove_folio(page_folio(page));
 }
+
+int try_to_release_page(struct page *page, gfp_t gfp)
+{
+	return filemap_release_folio(page_folio(page), gfp);
+}
+EXPORT_SYMBOL(try_to_release_page);
-- 
2.33.0

[PATCH 32/48] truncate: Add truncate_cleanup_folio()

[Matthew Wilcox (Oracle)]

Convert both callers of truncate_cleanup_page() to use
truncate_cleanup_folio() instead.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 mm/truncate.c | 21 +++++++++++----------
 1 file changed, 11 insertions(+), 10 deletions(-)

diff --git a/mm/truncate.c b/mm/truncate.c
index cc83a3f7c1ad..ab86b07c1e9c 100644
--- a/mm/truncate.c
+++ b/mm/truncate.c
@@ -177,21 +177,21 @@ void do_invalidatepage(struct page *page, unsigned int offset,
  * its lock, b) when a concurrent invalidate_mapping_pages got there first and
  * c) when tmpfs swizzles a page between a tmpfs inode and swapper_space.
  */
-static void truncate_cleanup_page(struct page *page)
+static void truncate_cleanup_folio(struct folio *folio)
 {
-	if (page_mapped(page))
-		unmap_mapping_page(page);
+	if (folio_mapped(folio))
+		unmap_mapping_page(&folio->page);
 
-	if (page_has_private(page))
-		do_invalidatepage(page, 0, thp_size(page));
+	if (folio_has_private(folio))
+		do_invalidatepage(&folio->page, 0, folio_size(folio));
 
 	/*
 	 * Some filesystems seem to re-dirty the page even after
 	 * the VM has canceled the dirty bit (eg ext3 journaling).
 	 * Hence dirty accounting check is placed after invalidation.
 	 */
-	cancel_dirty_page(page);
-	ClearPageMappedToDisk(page);
+	folio_cancel_dirty(folio);
+	folio_clear_mappedtodisk(folio);
 }
 
 /*
@@ -220,13 +220,14 @@ invalidate_complete_page(struct address_space *mapping, struct page *page)
 
 int truncate_inode_page(struct address_space *mapping, struct page *page)
 {
+	struct folio *folio = page_folio(page);
 	VM_BUG_ON_PAGE(PageTail(page), page);
 
 	if (page->mapping != mapping)
 		return -EIO;
 
-	truncate_cleanup_page(page);
-	delete_from_page_cache(page);
+	truncate_cleanup_folio(folio);
+	filemap_remove_folio(folio);
 	return 0;
 }
 
@@ -332,7 +333,7 @@ void truncate_inode_pages_range(struct address_space *mapping,
 		index = indices[pagevec_count(&pvec) - 1] + 1;
 		truncate_exceptional_pvec_entries(mapping, &pvec, indices);
 		for (i = 0; i < pagevec_count(&pvec); i++)
-			truncate_cleanup_page(pvec.pages[i]);
+			truncate_cleanup_folio(page_folio(pvec.pages[i]));
 		delete_from_page_cache_batch(mapping, &pvec);
 		for (i = 0; i < pagevec_count(&pvec); i++)
 			unlock_page(pvec.pages[i]);
-- 
2.33.0

[PATCH 33/48] mm: Add unmap_mapping_folio()

[Matthew Wilcox (Oracle)]

Convert both callers of unmap_mapping_page() to call unmap_mapping_folio()
instead.  Also move zap_details from linux/mm.h to mm/internal.h

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 include/linux/mm.h | 24 ------------------------
 mm/internal.h      | 25 ++++++++++++++++++++++++-
 mm/memory.c        | 27 +++++++++++++--------------
 mm/truncate.c      |  4 ++--
 4 files changed, 39 insertions(+), 41 deletions(-)

diff --git a/include/linux/mm.h b/include/linux/mm.h
index 145f045b0ddc..c9cdb26802fb 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -1825,28 +1825,6 @@ static inline bool can_do_mlock(void) { return false; }
 extern int user_shm_lock(size_t, struct ucounts *);
 extern void user_shm_unlock(size_t, struct ucounts *);
 
-/*
- * Parameter block passed down to zap_pte_range in exceptional cases.
- */
-struct zap_details {
-	struct address_space *zap_mapping;	/* Check page->mapping if set */
-	struct page *single_page;		/* Locked page to be unmapped */
-};
-
-/*
- * We set details->zap_mappings when we want to unmap shared but keep private
- * pages. Return true if skip zapping this page, false otherwise.
- */
-static inline bool
-zap_skip_check_mapping(struct zap_details *details, struct page *page)
-{
-	if (!details || !page)
-		return false;
-
-	return details->zap_mapping &&
-	    (details->zap_mapping != page_rmapping(page));
-}
-
 struct page *vm_normal_page(struct vm_area_struct *vma, unsigned long addr,
 			     pte_t pte);
 struct page *vm_normal_page_pmd(struct vm_area_struct *vma, unsigned long addr,
@@ -1892,7 +1870,6 @@ extern vm_fault_t handle_mm_fault(struct vm_area_struct *vma,
 extern int fixup_user_fault(struct mm_struct *mm,
 			    unsigned long address, unsigned int fault_flags,
 			    bool *unlocked);
-void unmap_mapping_page(struct page *page);
 void unmap_mapping_pages(struct address_space *mapping,
 		pgoff_t start, pgoff_t nr, bool even_cows);
 void unmap_mapping_range(struct address_space *mapping,
@@ -1913,7 +1890,6 @@ static inline int fixup_user_fault(struct mm_struct *mm, unsigned long address,
 	BUG();
 	return -EFAULT;
 }
-static inline void unmap_mapping_page(struct page *page) { }
 static inline void unmap_mapping_pages(struct address_space *mapping,
 		pgoff_t start, pgoff_t nr, bool even_cows) { }
 static inline void unmap_mapping_range(struct address_space *mapping,
diff --git a/mm/internal.h b/mm/internal.h
index 3b79a5c9427a..3f359f4830da 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -74,6 +74,28 @@ static inline bool can_madv_lru_vma(struct vm_area_struct *vma)
 	return !(vma->vm_flags & (VM_LOCKED|VM_HUGETLB|VM_PFNMAP));
 }
 
+/*
+ * Parameter block passed down to zap_pte_range in exceptional cases.
+ */
+struct zap_details {
+	struct address_space *zap_mapping;	/* Check page->mapping if set */
+	struct folio *single_folio;	/* Locked folio to be unmapped */
+};
+
+/*
+ * We set details->zap_mappings when we want to unmap shared but keep private
+ * pages. Return true if skip zapping this page, false otherwise.
+ */
+static inline bool
+zap_skip_check_mapping(struct zap_details *details, struct page *page)
+{
+	if (!details || !page)
+		return false;
+
+	return details->zap_mapping &&
+	    (details->zap_mapping != page_rmapping(page));
+}
+
 void unmap_page_range(struct mmu_gather *tlb,
 			     struct vm_area_struct *vma,
 			     unsigned long addr, unsigned long end,
@@ -388,6 +410,7 @@ void __vma_link_list(struct mm_struct *mm, struct vm_area_struct *vma,
 void __vma_unlink_list(struct mm_struct *mm, struct vm_area_struct *vma);
 
 #ifdef CONFIG_MMU
+void unmap_mapping_folio(struct folio *folio);
 extern long populate_vma_page_range(struct vm_area_struct *vma,
 		unsigned long start, unsigned long end, int *locked);
 extern long faultin_vma_page_range(struct vm_area_struct *vma,
@@ -491,8 +514,8 @@ static inline struct file *maybe_unlock_mmap_for_io(struct vm_fault *vmf,
 	}
 	return fpin;
 }
-
 #else /* !CONFIG_MMU */
+static inline void unmap_mapping_folio(struct folio *folio) { }
 static inline void clear_page_mlock(struct page *page) { }
 static inline void mlock_vma_page(struct page *page) { }
 static inline void vunmap_range_noflush(unsigned long start, unsigned long end)
diff --git a/mm/memory.c b/mm/memory.c
index 8f1de811a1dc..a86027026f2a 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -1443,8 +1443,8 @@ static inline unsigned long zap_pmd_range(struct mmu_gather *tlb,
 			else if (zap_huge_pmd(tlb, vma, pmd, addr))
 				goto next;
 			/* fall through */
-		} else if (details && details->single_page &&
-			   PageTransCompound(details->single_page) &&
+		} else if (details && details->single_folio &&
+			   folio_test_pmd_mappable(details->single_folio) &&
 			   next - addr == HPAGE_PMD_SIZE && pmd_none(*pmd)) {
 			spinlock_t *ptl = pmd_lock(tlb->mm, pmd);
 			/*
@@ -3332,31 +3332,30 @@ static inline void unmap_mapping_range_tree(struct rb_root_cached *root,
 }
 
 /**
- * unmap_mapping_page() - Unmap single page from processes.
- * @page: The locked page to be unmapped.
+ * unmap_mapping_folio() - Unmap single folio from processes.
+ * @folio: The locked folio to be unmapped.
  *
- * Unmap this page from any userspace process which still has it mmaped.
+ * Unmap this folio from any userspace process which still has it mmaped.
  * Typically, for efficiency, the range of nearby pages has already been
  * unmapped by unmap_mapping_pages() or unmap_mapping_range().  But once
- * truncation or invalidation holds the lock on a page, it may find that
- * the page has been remapped again: and then uses unmap_mapping_page()
+ * truncation or invalidation holds the lock on a folio, it may find that
+ * the page has been remapped again: and then uses unmap_mapping_folio()
  * to unmap it finally.
  */
-void unmap_mapping_page(struct page *page)
+void unmap_mapping_folio(struct folio *folio)
 {
-	struct address_space *mapping = page->mapping;
+	struct address_space *mapping = folio->mapping;
 	struct zap_details details = { };
 	pgoff_t	first_index;
 	pgoff_t	last_index;
 
-	VM_BUG_ON(!PageLocked(page));
-	VM_BUG_ON(PageTail(page));
+	VM_BUG_ON(!folio_test_locked(folio));
 
-	first_index = page->index;
-	last_index = page->index + thp_nr_pages(page) - 1;
+	first_index = folio->index;
+	last_index = folio->index + folio_nr_pages(folio) - 1;
 
 	details.zap_mapping = mapping;
-	details.single_page = page;
+	details.single_folio = folio;
 
 	i_mmap_lock_write(mapping);
 	if (unlikely(!RB_EMPTY_ROOT(&mapping->i_mmap.rb_root)))
diff --git a/mm/truncate.c b/mm/truncate.c
index ab86b07c1e9c..c98feea75a10 100644
--- a/mm/truncate.c
+++ b/mm/truncate.c
@@ -180,7 +180,7 @@ void do_invalidatepage(struct page *page, unsigned int offset,
 static void truncate_cleanup_folio(struct folio *folio)
 {
 	if (folio_mapped(folio))
-		unmap_mapping_page(&folio->page);
+		unmap_mapping_folio(folio);
 
 	if (folio_has_private(folio))
 		do_invalidatepage(&folio->page, 0, folio_size(folio));
@@ -670,7 +670,7 @@ int invalidate_inode_pages2_range(struct address_space *mapping,
 			wait_on_page_writeback(page);
 
 			if (page_mapped(page))
-				unmap_mapping_page(page);
+				unmap_mapping_folio(page_folio(page));
 			BUG_ON(page_mapped(page));
 
 			ret2 = do_launder_page(mapping, page);
-- 
2.33.0

[PATCH 34/48] shmem: Convert part of shmem_undo_range() to use a folio

[Matthew Wilcox (Oracle)]

find_lock_entries() never returns tail pages.  We cannot use page_folio()
here as the pagevec may also contain swap entries, so simply cast.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 mm/shmem.c | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/mm/shmem.c b/mm/shmem.c
index 18f93c2d68f1..40da9075374b 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -936,22 +936,22 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, loff_t lend,
 	while (index < end && find_lock_entries(mapping, index, end - 1,
 			&pvec, indices)) {
 		for (i = 0; i < pagevec_count(&pvec); i++) {
-			struct page *page = pvec.pages[i];
+			struct folio *folio = (struct folio *)pvec.pages[i];
 
 			index = indices[i];
 
-			if (xa_is_value(page)) {
+			if (xa_is_value(folio)) {
 				if (unfalloc)
 					continue;
 				nr_swaps_freed += !shmem_free_swap(mapping,
-								index, page);
+								index, folio);
 				continue;
 			}
-			index += thp_nr_pages(page) - 1;
+			index += folio_nr_pages(folio) - 1;
 
-			if (!unfalloc || !PageUptodate(page))
-				truncate_inode_page(mapping, page);
-			unlock_page(page);
+			if (!unfalloc || !folio_test_uptodate(folio))
+				truncate_inode_page(mapping, &folio->page);
+			folio_unlock(folio);
 		}
 		pagevec_remove_exceptionals(&pvec);
 		pagevec_release(&pvec);
-- 
2.33.0

[PATCH 35/48] truncate,shmem: Add truncate_inode_folio()

[Matthew Wilcox (Oracle)]

Convert all callers of truncate_inode_page() to call
truncate_inode_folio() instead, and move the declaration to mm/internal.h.
Move the assertion that the caller is not passing in a tail page to
generic_error_remove_page().  We can't entirely remove the struct page
from the callers yet because the page pointer in the pvec might be a
shadow/dax/swap entry instead of actually a page.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 include/linux/mm.h |  1 -
 mm/internal.h      |  1 +
 mm/shmem.c         |  5 +++--
 mm/truncate.c      | 23 ++++++++++++-----------
 4 files changed, 16 insertions(+), 14 deletions(-)

diff --git a/include/linux/mm.h b/include/linux/mm.h
index c9cdb26802fb..d8b7d7ed14dd 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -1859,7 +1859,6 @@ extern void truncate_pagecache(struct inode *inode, loff_t new);
 extern void truncate_setsize(struct inode *inode, loff_t newsize);
 void pagecache_isize_extended(struct inode *inode, loff_t from, loff_t to);
 void truncate_pagecache_range(struct inode *inode, loff_t offset, loff_t end);
-int truncate_inode_page(struct address_space *mapping, struct page *page);
 int generic_error_remove_page(struct address_space *mapping, struct page *page);
 int invalidate_inode_page(struct page *page);
 
diff --git a/mm/internal.h b/mm/internal.h
index 3f359f4830da..b05515d3b07b 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -113,6 +113,7 @@ static inline void force_page_cache_readahead(struct address_space *mapping,
 
 unsigned find_lock_entries(struct address_space *mapping, pgoff_t start,
 		pgoff_t end, struct pagevec *pvec, pgoff_t *indices);
+int truncate_inode_folio(struct address_space *mapping, struct folio *folio);
 
 /**
  * folio_evictable - Test whether a folio is evictable.
diff --git a/mm/shmem.c b/mm/shmem.c
index 40da9075374b..dbef008fb6e5 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -950,7 +950,7 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, loff_t lend,
 			index += folio_nr_pages(folio) - 1;
 
 			if (!unfalloc || !folio_test_uptodate(folio))
-				truncate_inode_page(mapping, &folio->page);
+				truncate_inode_folio(mapping, folio);
 			folio_unlock(folio);
 		}
 		pagevec_remove_exceptionals(&pvec);
@@ -1027,7 +1027,8 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, loff_t lend,
 				}
 				VM_BUG_ON_PAGE(PageWriteback(page), page);
 				if (shmem_punch_compound(page, start, end))
-					truncate_inode_page(mapping, page);
+					truncate_inode_folio(mapping,
+							     page_folio(page));
 				else if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)) {
 					/* Wipe the page and don't get stuck */
 					clear_highpage(page);
diff --git a/mm/truncate.c b/mm/truncate.c
index c98feea75a10..0000424fc56b 100644
--- a/mm/truncate.c
+++ b/mm/truncate.c
@@ -218,12 +218,9 @@ invalidate_complete_page(struct address_space *mapping, struct page *page)
 	return ret;
 }
 
-int truncate_inode_page(struct address_space *mapping, struct page *page)
+int truncate_inode_folio(struct address_space *mapping, struct folio *folio)
 {
-	struct folio *folio = page_folio(page);
-	VM_BUG_ON_PAGE(PageTail(page), page);
-
-	if (page->mapping != mapping)
+	if (folio->mapping != mapping)
 		return -EIO;
 
 	truncate_cleanup_folio(folio);
@@ -236,6 +233,8 @@ int truncate_inode_page(struct address_space *mapping, struct page *page)
  */
 int generic_error_remove_page(struct address_space *mapping, struct page *page)
 {
+	VM_BUG_ON_PAGE(PageTail(page), page);
+
 	if (!mapping)
 		return -EINVAL;
 	/*
@@ -244,7 +243,7 @@ int generic_error_remove_page(struct address_space *mapping, struct page *page)
 	 */
 	if (!S_ISREG(mapping->host->i_mode))
 		return -EIO;
-	return truncate_inode_page(mapping, page);
+	return truncate_inode_folio(mapping, page_folio(page));
 }
 EXPORT_SYMBOL(generic_error_remove_page);
 
@@ -395,18 +394,20 @@ void truncate_inode_pages_range(struct address_space *mapping,
 
 		for (i = 0; i < pagevec_count(&pvec); i++) {
 			struct page *page = pvec.pages[i];
+			struct folio *folio;
 
 			/* We rely upon deletion not changing page->index */
 			index = indices[i];
 
 			if (xa_is_value(page))
 				continue;
+			folio = page_folio(page);
 
-			lock_page(page);
-			WARN_ON(page_to_index(page) != index);
-			wait_on_page_writeback(page);
-			truncate_inode_page(mapping, page);
-			unlock_page(page);
+			folio_lock(folio);
+			VM_BUG_ON_FOLIO(!folio_contains(folio, index), folio);
+			folio_wait_writeback(folio);
+			truncate_inode_folio(mapping, folio);
+			folio_unlock(folio);
 		}
 		truncate_exceptional_pvec_entries(mapping, &pvec, indices);
 		pagevec_release(&pvec);
-- 
2.33.0

[PATCH 36/48] truncate: Skip known-truncated indices

[Matthew Wilcox (Oracle)]

If we've truncated an entire folio, we can skip over all the indices
covered by this folio.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 mm/truncate.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/mm/truncate.c b/mm/truncate.c
index 0000424fc56b..0df420c1cf5b 100644
--- a/mm/truncate.c
+++ b/mm/truncate.c
@@ -408,6 +408,7 @@ void truncate_inode_pages_range(struct address_space *mapping,
 			folio_wait_writeback(folio);
 			truncate_inode_folio(mapping, folio);
 			folio_unlock(folio);
+			index = folio_index(folio) + folio_nr_pages(folio) - 1;
 		}
 		truncate_exceptional_pvec_entries(mapping, &pvec, indices);
 		pagevec_release(&pvec);
-- 
2.33.0

[PATCH 37/48] truncate: Convert invalidate_inode_pages2_range() to use a folio

[Matthew Wilcox (Oracle)]

If we're going to unmap a folio, we have to be sure to unmap the entire
folio, not just the part of it which lies after the search index.

We cannot yet remove the struct page from invalidate_inode_pages2_range()
because the page pointer in the pvec might be a shadow/dax/swap entry
instead of actually a page.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 mm/truncate.c | 31 +++++++++++++++++--------------
 1 file changed, 17 insertions(+), 14 deletions(-)

diff --git a/mm/truncate.c b/mm/truncate.c
index 0df420c1cf5b..ef6980b240e2 100644
--- a/mm/truncate.c
+++ b/mm/truncate.c
@@ -642,8 +642,9 @@ int invalidate_inode_pages2_range(struct address_space *mapping,
 	while (find_get_entries(mapping, index, end, &pvec, indices)) {
 		for (i = 0; i < pagevec_count(&pvec); i++) {
 			struct page *page = pvec.pages[i];
+			struct folio *folio;
 
-			/* We rely upon deletion not changing page->index */
+			/* We rely upon deletion not changing folio->index */
 			index = indices[i];
 
 			if (xa_is_value(page)) {
@@ -652,10 +653,11 @@ int invalidate_inode_pages2_range(struct address_space *mapping,
 					ret = -EBUSY;
 				continue;
 			}
+			folio = page_folio(page);
 
-			if (!did_range_unmap && page_mapped(page)) {
+			if (!did_range_unmap && folio_mapped(folio)) {
 				/*
-				 * If page is mapped, before taking its lock,
+				 * If folio is mapped, before taking its lock,
 				 * zap the rest of the file in one hit.
 				 */
 				unmap_mapping_pages(mapping, index,
@@ -663,26 +665,27 @@ int invalidate_inode_pages2_range(struct address_space *mapping,
 				did_range_unmap = 1;
 			}
 
-			lock_page(page);
-			WARN_ON(page_to_index(page) != index);
-			if (page->mapping != mapping) {
-				unlock_page(page);
+			folio_lock(folio);
+			VM_BUG_ON_FOLIO(!folio_contains(folio, index), folio);
+			if (folio->mapping != mapping) {
+				folio_unlock(folio);
 				continue;
 			}
-			wait_on_page_writeback(page);
+			folio_wait_writeback(folio);
 
-			if (page_mapped(page))
-				unmap_mapping_folio(page_folio(page));
-			BUG_ON(page_mapped(page));
+			if (folio_mapped(folio))
+				unmap_mapping_folio(folio);
+			BUG_ON(folio_mapped(folio));
 
-			ret2 = do_launder_page(mapping, page);
+			ret2 = do_launder_page(mapping, &folio->page);
 			if (ret2 == 0) {
-				if (!invalidate_complete_page2(mapping, page))
+				if (!invalidate_complete_page2(mapping,
+								&folio->page))
 					ret2 = -EBUSY;
 			}
 			if (ret2 < 0)
 				ret = ret2;
-			unlock_page(page);
+			folio_unlock(folio);
 		}
 		pagevec_remove_exceptionals(&pvec);
 		pagevec_release(&pvec);
-- 
2.33.0

[PATCH 38/48] truncate: Add invalidate_complete_folio2()

[Matthew Wilcox (Oracle)]

Convert invalidate_complete_page2() to invalidate_complete_folio2().
Use filemap_free_folio() to free the page instead of calling ->freepage
manually.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 mm/filemap.c  |  3 +--
 mm/internal.h |  1 +
 mm/truncate.c | 23 ++++++++++-------------
 3 files changed, 12 insertions(+), 15 deletions(-)

diff --git a/mm/filemap.c b/mm/filemap.c
index 7a418f0012e5..fb3cdb7aeffc 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -229,8 +229,7 @@ void __filemap_remove_folio(struct folio *folio, void *shadow)
 	page_cache_delete(mapping, folio, shadow);
 }
 
-static void filemap_free_folio(struct address_space *mapping,
-				struct folio *folio)
+void filemap_free_folio(struct address_space *mapping, struct folio *folio)
 {
 	void (*freepage)(struct page *);
 
diff --git a/mm/internal.h b/mm/internal.h
index b05515d3b07b..d3c7b35934ed 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -113,6 +113,7 @@ static inline void force_page_cache_readahead(struct address_space *mapping,
 
 unsigned find_lock_entries(struct address_space *mapping, pgoff_t start,
 		pgoff_t end, struct pagevec *pvec, pgoff_t *indices);
+void filemap_free_folio(struct address_space *mapping, struct folio *folio);
 int truncate_inode_folio(struct address_space *mapping, struct folio *folio);
 
 /**
diff --git a/mm/truncate.c b/mm/truncate.c
index ef6980b240e2..5370094641d6 100644
--- a/mm/truncate.c
+++ b/mm/truncate.c
@@ -571,31 +571,29 @@ void invalidate_mapping_pagevec(struct address_space *mapping,
  * shrink_page_list() has a temp ref on them, or because they're transiently
  * sitting in the lru_cache_add() pagevecs.
  */
-static int
-invalidate_complete_page2(struct address_space *mapping, struct page *page)
+static int invalidate_complete_folio2(struct address_space *mapping,
+					struct folio *folio)
 {
-	if (page->mapping != mapping)
+	if (folio->mapping != mapping)
 		return 0;
 
-	if (page_has_private(page) && !try_to_release_page(page, GFP_KERNEL))
+	if (folio_has_private(folio) &&
+	    !filemap_release_folio(folio, GFP_KERNEL))
 		return 0;
 
 	spin_lock(&mapping->host->i_lock);
 	xa_lock_irq(&mapping->i_pages);
-	if (PageDirty(page))
+	if (folio_test_dirty(folio))
 		goto failed;
 
-	BUG_ON(page_has_private(page));
-	__delete_from_page_cache(page, NULL);
+	BUG_ON(folio_has_private(folio));
+	__filemap_remove_folio(folio, NULL);
 	xa_unlock_irq(&mapping->i_pages);
 	if (mapping_shrinkable(mapping))
 		inode_add_lru(mapping->host);
 	spin_unlock(&mapping->host->i_lock);
 
-	if (mapping->a_ops->freepage)
-		mapping->a_ops->freepage(page);
-
-	put_page(page);	/* pagecache ref */
+	filemap_free_folio(mapping, folio);
 	return 1;
 failed:
 	xa_unlock_irq(&mapping->i_pages);
@@ -679,8 +677,7 @@ int invalidate_inode_pages2_range(struct address_space *mapping,
 
 			ret2 = do_launder_page(mapping, &folio->page);
 			if (ret2 == 0) {
-				if (!invalidate_complete_page2(mapping,
-								&folio->page))
+				if (!invalidate_complete_folio2(mapping, folio))
 					ret2 = -EBUSY;
 			}
 			if (ret2 < 0)
-- 
2.33.0

[PATCH 39/48] filemap: Convert filemap_read() to use a folio

[Matthew Wilcox (Oracle)]

We know the pagevec always contains folios, but use page_folio() anyway
instead of casting.  Removes a few calls to legacy functions.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 mm/filemap.c | 28 ++++++++++++----------------
 1 file changed, 12 insertions(+), 16 deletions(-)

diff --git a/mm/filemap.c b/mm/filemap.c
index fb3cdb7aeffc..91399027b349 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -2668,30 +2668,26 @@ ssize_t filemap_read(struct kiocb *iocb, struct iov_iter *iter,
 			mark_page_accessed(pvec.pages[0]);
 
 		for (i = 0; i < pagevec_count(&pvec); i++) {
-			struct page *page = pvec.pages[i];
-			size_t page_size = thp_size(page);
-			size_t offset = iocb->ki_pos & (page_size - 1);
+			struct folio *folio = page_folio(pvec.pages[i]);
+			size_t fsize = folio_size(folio);
+			size_t offset = iocb->ki_pos & (fsize - 1);
 			size_t bytes = min_t(loff_t, end_offset - iocb->ki_pos,
-					     page_size - offset);
+					     fsize - offset);
 			size_t copied;
 
-			if (end_offset < page_offset(page))
+			if (end_offset < folio_pos(folio))
 				break;
 			if (i > 0)
-				mark_page_accessed(page);
+				folio_mark_accessed(folio);
 			/*
-			 * If users can be writing to this page using arbitrary
-			 * virtual addresses, take care about potential aliasing
-			 * before reading the page on the kernel side.
+			 * If users can be writing to this folio using arbitrary
+			 * virtual addresses, take care of potential aliasing
+			 * before reading the folio on the kernel side.
 			 */
-			if (writably_mapped) {
-				int j;
-
-				for (j = 0; j < thp_nr_pages(page); j++)
-					flush_dcache_page(page + j);
-			}
+			if (writably_mapped)
+				flush_dcache_folio(folio);
 
-			copied = copy_page_to_iter(page, offset, bytes, iter);
+			copied = copy_folio_to_iter(folio, offset, bytes, iter);
 
 			already_read += copied;
 			iocb->ki_pos += copied;
-- 
2.33.0

[PATCH 40/48] filemap: Convert filemap_get_read_batch() to use a folio_batch

[Matthew Wilcox (Oracle)]

This change ripples all the way through the filemap_read() call chain and
removes a lot of messing about converting folios to pages and back again.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 mm/filemap.c | 65 ++++++++++++++++++++++++++--------------------------
 1 file changed, 33 insertions(+), 32 deletions(-)

diff --git a/mm/filemap.c b/mm/filemap.c
index 91399027b349..38726ca96f0e 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -2325,16 +2325,16 @@ static void shrink_readahead_size_eio(struct file_ra_state *ra)
 }
 
 /*
- * filemap_get_read_batch - Get a batch of pages for read
+ * filemap_get_read_batch - Get a batch of folios for read
  *
- * Get a batch of pages which represent a contiguous range of bytes
- * in the file.  No tail pages will be returned.  If @index is in the
- * middle of a THP, the entire THP will be returned.  The last page in
- * the batch may have Readahead set or be not Uptodate so that the
- * caller can take the appropriate action.
+ * Get a batch of folios which represent a contiguous range of bytes in
+ * the file.  No exceptional entries will be returned.  If @index is in
+ * the middle of a folio, the entire folio will be returned.  The last
+ * folio in the batch may have the readahead flag set or the uptodate flag
+ * clear so that the caller can take the appropriate action.
  */
 static void filemap_get_read_batch(struct address_space *mapping,
-		pgoff_t index, pgoff_t max, struct pagevec *pvec)
+		pgoff_t index, pgoff_t max, struct folio_batch *fbatch)
 {
 	XA_STATE(xas, &mapping->i_pages, index);
 	struct folio *folio;
@@ -2349,9 +2349,9 @@ static void filemap_get_read_batch(struct address_space *mapping,
 			goto retry;
 
 		if (unlikely(folio != xas_reload(&xas)))
-			goto put_page;
+			goto put_folio;
 
-		if (!pagevec_add(pvec, &folio->page))
+		if (!folio_batch_add(fbatch, folio))
 			break;
 		if (!folio_test_uptodate(folio))
 			break;
@@ -2360,7 +2360,7 @@ static void filemap_get_read_batch(struct address_space *mapping,
 		xas.xa_index = folio->index + folio_nr_pages(folio) - 1;
 		xas.xa_offset = (xas.xa_index >> xas.xa_shift) & XA_CHUNK_MASK;
 		continue;
-put_page:
+put_folio:
 		folio_put(folio);
 retry:
 		xas_reset(&xas);
@@ -2475,7 +2475,7 @@ static int filemap_update_page(struct kiocb *iocb,
 
 static int filemap_create_folio(struct file *file,
 		struct address_space *mapping, pgoff_t index,
-		struct pagevec *pvec)
+		struct folio_batch *fbatch)
 {
 	struct folio *folio;
 	int error;
@@ -2510,7 +2510,7 @@ static int filemap_create_folio(struct file *file,
 		goto error;
 
 	filemap_invalidate_unlock_shared(mapping);
-	pagevec_add(pvec, &folio->page);
+	folio_batch_add(fbatch, folio);
 	return 0;
 error:
 	filemap_invalidate_unlock_shared(mapping);
@@ -2531,7 +2531,7 @@ static int filemap_readahead(struct kiocb *iocb, struct file *file,
 }
 
 static int filemap_get_pages(struct kiocb *iocb, struct iov_iter *iter,
-		struct pagevec *pvec)
+		struct folio_batch *fbatch)
 {
 	struct file *filp = iocb->ki_filp;
 	struct address_space *mapping = filp->f_mapping;
@@ -2546,32 +2546,33 @@ static int filemap_get_pages(struct kiocb *iocb, struct iov_iter *iter,
 	if (fatal_signal_pending(current))
 		return -EINTR;
 
-	filemap_get_read_batch(mapping, index, last_index, pvec);
-	if (!pagevec_count(pvec)) {
+	filemap_get_read_batch(mapping, index, last_index, fbatch);
+	if (!folio_batch_count(fbatch)) {
 		if (iocb->ki_flags & IOCB_NOIO)
 			return -EAGAIN;
 		page_cache_sync_readahead(mapping, ra, filp, index,
 				last_index - index);
-		filemap_get_read_batch(mapping, index, last_index, pvec);
+		filemap_get_read_batch(mapping, index, last_index, fbatch);
 	}
-	if (!pagevec_count(pvec)) {
+	if (!folio_batch_count(fbatch)) {
 		if (iocb->ki_flags & (IOCB_NOWAIT | IOCB_WAITQ))
 			return -EAGAIN;
 		err = filemap_create_folio(filp, mapping,
-				iocb->ki_pos >> PAGE_SHIFT, pvec);
+				iocb->ki_pos >> PAGE_SHIFT, fbatch);
 		if (err == AOP_TRUNCATED_PAGE)
 			goto retry;
 		return err;
 	}
 
-	folio = page_folio(pvec->pages[pagevec_count(pvec) - 1]);
+	folio = fbatch->folios[folio_batch_count(fbatch) - 1];
 	if (folio_test_readahead(folio)) {
 		err = filemap_readahead(iocb, filp, mapping, folio, last_index);
 		if (err)
 			goto err;
 	}
 	if (!folio_test_uptodate(folio)) {
-		if ((iocb->ki_flags & IOCB_WAITQ) && pagevec_count(pvec) > 1)
+		if ((iocb->ki_flags & IOCB_WAITQ) &&
+		    folio_batch_count(fbatch) > 1)
 			iocb->ki_flags |= IOCB_NOWAIT;
 		err = filemap_update_page(iocb, mapping, iter, folio);
 		if (err)
@@ -2582,7 +2583,7 @@ static int filemap_get_pages(struct kiocb *iocb, struct iov_iter *iter,
 err:
 	if (err < 0)
 		folio_put(folio);
-	if (likely(--pvec->nr))
+	if (likely(--fbatch->nr))
 		return 0;
 	if (err == AOP_TRUNCATED_PAGE)
 		goto retry;
@@ -2609,7 +2610,7 @@ ssize_t filemap_read(struct kiocb *iocb, struct iov_iter *iter,
 	struct file_ra_state *ra = &filp->f_ra;
 	struct address_space *mapping = filp->f_mapping;
 	struct inode *inode = mapping->host;
-	struct pagevec pvec;
+	struct folio_batch fbatch;
 	int i, error = 0;
 	bool writably_mapped;
 	loff_t isize, end_offset;
@@ -2620,7 +2621,7 @@ ssize_t filemap_read(struct kiocb *iocb, struct iov_iter *iter,
 		return 0;
 
 	iov_iter_truncate(iter, inode->i_sb->s_maxbytes);
-	pagevec_init(&pvec);
+	folio_batch_init(&fbatch);
 
 	do {
 		cond_resched();
@@ -2636,7 +2637,7 @@ ssize_t filemap_read(struct kiocb *iocb, struct iov_iter *iter,
 		if (unlikely(iocb->ki_pos >= i_size_read(inode)))
 			break;
 
-		error = filemap_get_pages(iocb, iter, &pvec);
+		error = filemap_get_pages(iocb, iter, &fbatch);
 		if (error < 0)
 			break;
 
@@ -2650,7 +2651,7 @@ ssize_t filemap_read(struct kiocb *iocb, struct iov_iter *iter,
 		 */
 		isize = i_size_read(inode);
 		if (unlikely(iocb->ki_pos >= isize))
-			goto put_pages;
+			goto put_folios;
 		end_offset = min_t(loff_t, isize, iocb->ki_pos + iter->count);
 
 		/*
@@ -2665,10 +2666,10 @@ ssize_t filemap_read(struct kiocb *iocb, struct iov_iter *iter,
 		 */
 		if (iocb->ki_pos >> PAGE_SHIFT !=
 		    ra->prev_pos >> PAGE_SHIFT)
-			mark_page_accessed(pvec.pages[0]);
+			folio_mark_accessed(fbatch.folios[0]);
 
-		for (i = 0; i < pagevec_count(&pvec); i++) {
-			struct folio *folio = page_folio(pvec.pages[i]);
+		for (i = 0; i < folio_batch_count(&fbatch); i++) {
+			struct folio *folio = fbatch.folios[i];
 			size_t fsize = folio_size(folio);
 			size_t offset = iocb->ki_pos & (fsize - 1);
 			size_t bytes = min_t(loff_t, end_offset - iocb->ki_pos,
@@ -2698,10 +2699,10 @@ ssize_t filemap_read(struct kiocb *iocb, struct iov_iter *iter,
 				break;
 			}
 		}
-put_pages:
-		for (i = 0; i < pagevec_count(&pvec); i++)
-			put_page(pvec.pages[i]);
-		pagevec_reinit(&pvec);
+put_folios:
+		for (i = 0; i < folio_batch_count(&fbatch); i++)
+			folio_put(fbatch.folios[i]);
+		folio_batch_init(&fbatch);
 	} while (iov_iter_count(iter) && iocb->ki_pos < isize && !error);
 
 	file_accessed(filp);
-- 
2.33.0

[PATCH 41/48] filemap: Return only folios from find_get_entries()

[Matthew Wilcox (Oracle)]

The callers have all been converted to work on folios, so convert
find_get_entries() to return a batch of folios instead of pages.
We also now return multiple large folios in a single call.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
Reviewed-by: Jan Kara <jack@suse.cz>
Reviewed-by: William Kucharski <william.kucharski@oracle.com>
---
 include/linux/pagemap.h |  2 --
 mm/filemap.c            | 43 +++++++++++------------------------------
 mm/internal.h           |  4 ++++
 mm/shmem.c              | 36 +++++++++++++++++++---------------
 mm/truncate.c           | 43 +++++++++++++++++++++++------------------
 5 files changed, 59 insertions(+), 69 deletions(-)

diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
index eb6e58e106c8..d2259a1da51c 100644
--- a/include/linux/pagemap.h
+++ b/include/linux/pagemap.h
@@ -592,8 +592,6 @@ static inline struct page *find_subpage(struct page *head, pgoff_t index)
 	return head + (index & (thp_nr_pages(head) - 1));
 }
 
-unsigned find_get_entries(struct address_space *mapping, pgoff_t start,
-		pgoff_t end, struct pagevec *pvec, pgoff_t *indices);
 unsigned find_get_pages_range(struct address_space *mapping, pgoff_t *start,
 			pgoff_t end, unsigned int nr_pages,
 			struct page **pages);
diff --git a/mm/filemap.c b/mm/filemap.c
index 38726ca96f0e..4f00412d72d3 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -2015,57 +2015,36 @@ static inline struct folio *find_get_entry(struct xa_state *xas, pgoff_t max,
  * @mapping:	The address_space to search
  * @start:	The starting page cache index
  * @end:	The final page index (inclusive).
- * @pvec:	Where the resulting entries are placed.
+ * @fbatch:	Where the resulting entries are placed.
  * @indices:	The cache indices corresponding to the entries in @entries
  *
  * find_get_entries() will search for and return a batch of entries in
- * the mapping.  The entries are placed in @pvec.  find_get_entries()
- * takes a reference on any actual pages it returns.
+ * the mapping.  The entries are placed in @fbatch.  find_get_entries()
+ * takes a reference on any actual folios it returns.
  *
- * The search returns a group of mapping-contiguous page cache entries
- * with ascending indexes.  There may be holes in the indices due to
- * not-present pages.
+ * The entries have ascending indexes.  The indices may not be consecutive
+ * due to not-present entries or large folios.
  *
- * Any shadow entries of evicted pages, or swap entries from
+ * Any shadow entries of evicted folios, or swap entries from
  * shmem/tmpfs, are included in the returned array.
  *
- * If it finds a Transparent Huge Page, head or tail, find_get_entries()
- * stops at that page: the caller is likely to have a better way to handle
- * the compound page as a whole, and then skip its extent, than repeatedly
- * calling find_get_entries() to return all its tails.
- *
- * Return: the number of pages and shadow entries which were found.
+ * Return: The number of entries which were found.
  */
 unsigned find_get_entries(struct address_space *mapping, pgoff_t start,
-		pgoff_t end, struct pagevec *pvec, pgoff_t *indices)
+		pgoff_t end, struct folio_batch *fbatch, pgoff_t *indices)
 {
 	XA_STATE(xas, &mapping->i_pages, start);
 	struct folio *folio;
-	unsigned int ret = 0;
-	unsigned nr_entries = PAGEVEC_SIZE;
 
 	rcu_read_lock();
 	while ((folio = find_get_entry(&xas, end, XA_PRESENT)) != NULL) {
-		struct page *page = &folio->page;
-		/*
-		 * Terminate early on finding a THP, to allow the caller to
-		 * handle it all at once; but continue if this is hugetlbfs.
-		 */
-		if (!xa_is_value(folio) && folio_test_large(folio) &&
-				!folio_test_hugetlb(folio)) {
-			page = folio_file_page(folio, xas.xa_index);
-			nr_entries = ret + 1;
-		}
-
-		indices[ret] = xas.xa_index;
-		pvec->pages[ret] = page;
-		if (++ret == nr_entries)
+		indices[fbatch->nr] = xas.xa_index;
+		if (!folio_batch_add(fbatch, folio))
 			break;
 	}
 	rcu_read_unlock();
 
-	pvec->nr = ret;
-	return ret;
+	return folio_batch_count(fbatch);
 }
 
 /**
diff --git a/mm/internal.h b/mm/internal.h
index d3c7b35934ed..36ad6ffe53bf 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -12,6 +12,8 @@
 #include <linux/pagemap.h>
 #include <linux/tracepoint-defs.h>
 
+struct folio_batch;
+
 /*
  * The set of flags that only affect watermark checking and reclaim
  * behaviour. This is used by the MM to obey the caller constraints
@@ -113,6 +115,8 @@ static inline void force_page_cache_readahead(struct address_space *mapping,
 
 unsigned find_lock_entries(struct address_space *mapping, pgoff_t start,
 		pgoff_t end, struct pagevec *pvec, pgoff_t *indices);
+unsigned find_get_entries(struct address_space *mapping, pgoff_t start,
+		pgoff_t end, struct folio_batch *fbatch, pgoff_t *indices);
 void filemap_free_folio(struct address_space *mapping, struct folio *folio);
 int truncate_inode_folio(struct address_space *mapping, struct folio *folio);
 
diff --git a/mm/shmem.c b/mm/shmem.c
index dbef008fb6e5..e909c163fb38 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -920,6 +920,7 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, loff_t lend,
 	unsigned int partial_start = lstart & (PAGE_SIZE - 1);
 	unsigned int partial_end = (lend + 1) & (PAGE_SIZE - 1);
 	struct pagevec pvec;
+	struct folio_batch fbatch;
 	pgoff_t indices[PAGEVEC_SIZE];
 	long nr_swaps_freed = 0;
 	pgoff_t index;
@@ -987,11 +988,12 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, loff_t lend,
 	if (start >= end)
 		return;
 
+	folio_batch_init(&fbatch);
 	index = start;
 	while (index < end) {
 		cond_resched();
 
-		if (!find_get_entries(mapping, index, end - 1, &pvec,
+		if (!find_get_entries(mapping, index, end - 1, &fbatch,
 				indices)) {
 			/* If all gone or hole-punch or unfalloc, we're done */
 			if (index == start || end != -1)
@@ -1000,14 +1002,14 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, loff_t lend,
 			index = start;
 			continue;
 		}
-		for (i = 0; i < pagevec_count(&pvec); i++) {
-			struct page *page = pvec.pages[i];
+		for (i = 0; i < folio_batch_count(&fbatch); i++) {
+			struct folio *folio = fbatch.folios[i];
 
 			index = indices[i];
-			if (xa_is_value(page)) {
+			if (xa_is_value(folio)) {
 				if (unfalloc)
 					continue;
-				if (shmem_free_swap(mapping, index, page)) {
+				if (shmem_free_swap(mapping, index, folio)) {
 					/* Swap was replaced by page: retry */
 					index--;
 					break;
@@ -1016,33 +1018,35 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, loff_t lend,
 				continue;
 			}
 
-			lock_page(page);
+			folio_lock(folio);
 
-			if (!unfalloc || !PageUptodate(page)) {
-				if (page_mapping(page) != mapping) {
+			if (!unfalloc || !folio_test_uptodate(folio)) {
+				struct page *page = folio_file_page(folio,
+									index);
+				if (folio_mapping(folio) != mapping) {
 					/* Page was replaced by swap: retry */
-					unlock_page(page);
+					folio_unlock(folio);
 					index--;
 					break;
 				}
-				VM_BUG_ON_PAGE(PageWriteback(page), page);
+				VM_BUG_ON_FOLIO(folio_test_writeback(folio),
+						folio);
 				if (shmem_punch_compound(page, start, end))
-					truncate_inode_folio(mapping,
-							     page_folio(page));
+					truncate_inode_folio(mapping, folio);
 				else if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)) {
 					/* Wipe the page and don't get stuck */
 					clear_highpage(page);
 					flush_dcache_page(page);
-					set_page_dirty(page);
+					folio_mark_dirty(folio);
 					if (index <
 					    round_up(start, HPAGE_PMD_NR))
 						start = index + 1;
 				}
 			}
-			unlock_page(page);
+			folio_unlock(folio);
 		}
-		pagevec_remove_exceptionals(&pvec);
-		pagevec_release(&pvec);
+		folio_batch_remove_exceptionals(&fbatch);
+		folio_batch_release(&fbatch);
 		index++;
 	}
 
diff --git a/mm/truncate.c b/mm/truncate.c
index 5370094641d6..357af144df63 100644
--- a/mm/truncate.c
+++ b/mm/truncate.c
@@ -108,6 +108,13 @@ static void truncate_exceptional_pvec_entries(struct address_space *mapping,
 	pvec->nr = j;
 }
 
+static void truncate_folio_batch_exceptionals(struct address_space *mapping,
+				struct folio_batch *fbatch, pgoff_t *indices)
+{
+	truncate_exceptional_pvec_entries(mapping, (struct pagevec *)fbatch,
+						indices);
+}
+
 /*
  * Invalidate exceptional entry if easily possible. This handles exceptional
  * entries for invalidate_inode_pages().
@@ -297,6 +304,7 @@ void truncate_inode_pages_range(struct address_space *mapping,
 	unsigned int	partial_start;	/* inclusive */
 	unsigned int	partial_end;	/* exclusive */
 	struct pagevec	pvec;
+	struct folio_batch fbatch;
 	pgoff_t		indices[PAGEVEC_SIZE];
 	pgoff_t		index;
 	int		i;
@@ -379,10 +387,11 @@ void truncate_inode_pages_range(struct address_space *mapping,
 	if (start >= end)
 		goto out;
 
+	folio_batch_init(&fbatch);
 	index = start;
 	for ( ; ; ) {
 		cond_resched();
-		if (!find_get_entries(mapping, index, end - 1, &pvec,
+		if (!find_get_entries(mapping, index, end - 1, &fbatch,
 				indices)) {
 			/* If all gone from start onwards, we're done */
 			if (index == start)
@@ -392,16 +401,14 @@ void truncate_inode_pages_range(struct address_space *mapping,
 			continue;
 		}
 
-		for (i = 0; i < pagevec_count(&pvec); i++) {
-			struct page *page = pvec.pages[i];
-			struct folio *folio;
+		for (i = 0; i < folio_batch_count(&fbatch); i++) {
+			struct folio *folio = fbatch.folios[i];
 
 			/* We rely upon deletion not changing page->index */
 			index = indices[i];
 
-			if (xa_is_value(page))
+			if (xa_is_value(folio))
 				continue;
-			folio = page_folio(page);
 
 			folio_lock(folio);
 			VM_BUG_ON_FOLIO(!folio_contains(folio, index), folio);
@@ -410,8 +417,8 @@ void truncate_inode_pages_range(struct address_space *mapping,
 			folio_unlock(folio);
 			index = folio_index(folio) + folio_nr_pages(folio) - 1;
 		}
-		truncate_exceptional_pvec_entries(mapping, &pvec, indices);
-		pagevec_release(&pvec);
+		truncate_folio_batch_exceptionals(mapping, &fbatch, indices);
+		folio_batch_release(&fbatch);
 		index++;
 	}
 
@@ -625,7 +632,7 @@ int invalidate_inode_pages2_range(struct address_space *mapping,
 				  pgoff_t start, pgoff_t end)
 {
 	pgoff_t indices[PAGEVEC_SIZE];
-	struct pagevec pvec;
+	struct folio_batch fbatch;
 	pgoff_t index;
 	int i;
 	int ret = 0;
@@ -635,23 +642,21 @@ int invalidate_inode_pages2_range(struct address_space *mapping,
 	if (mapping_empty(mapping))
 		goto out;
 
-	pagevec_init(&pvec);
+	folio_batch_init(&fbatch);
 	index = start;
-	while (find_get_entries(mapping, index, end, &pvec, indices)) {
-		for (i = 0; i < pagevec_count(&pvec); i++) {
-			struct page *page = pvec.pages[i];
-			struct folio *folio;
+	while (find_get_entries(mapping, index, end, &fbatch, indices)) {
+		for (i = 0; i < folio_batch_count(&fbatch); i++) {
+			struct folio *folio = fbatch.folios[i];
 
 			/* We rely upon deletion not changing folio->index */
 			index = indices[i];
 
-			if (xa_is_value(page)) {
+			if (xa_is_value(folio)) {
 				if (!invalidate_exceptional_entry2(mapping,
-								   index, page))
+						index, folio))
 					ret = -EBUSY;
 				continue;
 			}
-			folio = page_folio(page);
 
 			if (!did_range_unmap && folio_mapped(folio)) {
 				/*
@@ -684,8 +689,8 @@ int invalidate_inode_pages2_range(struct address_space *mapping,
 				ret = ret2;
 			folio_unlock(folio);
 		}
-		pagevec_remove_exceptionals(&pvec);
-		pagevec_release(&pvec);
+		folio_batch_remove_exceptionals(&fbatch);
+		folio_batch_release(&fbatch);
 		cond_resched();
 		index++;
 	}
-- 
2.33.0

[PATCH 42/48] mm: Convert find_lock_entries() to use a folio_batch

[Matthew Wilcox (Oracle)]

find_lock_entries() already only returned the head page of folios, so
convert it to return a folio_batch instead of a pagevec.  That cascades
through converting truncate_inode_pages_range() to
delete_from_page_cache_batch() and page_cache_delete_batch().

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 include/linux/pagemap.h |  4 +--
 mm/filemap.c            | 60 ++++++++++++++++++------------------
 mm/internal.h           |  2 +-
 mm/shmem.c              | 14 ++++-----
 mm/truncate.c           | 67 ++++++++++++++++++-----------------------
 5 files changed, 67 insertions(+), 80 deletions(-)

diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
index d2259a1da51c..6e038811f4c8 100644
--- a/include/linux/pagemap.h
+++ b/include/linux/pagemap.h
@@ -16,7 +16,7 @@
 #include <linux/hardirq.h> /* for in_interrupt() */
 #include <linux/hugetlb_inline.h>
 
-struct pagevec;
+struct folio_batch;
 
 static inline bool mapping_empty(struct address_space *mapping)
 {
@@ -936,7 +936,7 @@ static inline void __delete_from_page_cache(struct page *page, void *shadow)
 }
 void replace_page_cache_page(struct page *old, struct page *new);
 void delete_from_page_cache_batch(struct address_space *mapping,
-				  struct pagevec *pvec);
+				  struct folio_batch *fbatch);
 int try_to_release_page(struct page *page, gfp_t gfp);
 bool filemap_release_folio(struct folio *folio, gfp_t gfp);
 loff_t mapping_seek_hole_data(struct address_space *, loff_t start, loff_t end,
diff --git a/mm/filemap.c b/mm/filemap.c
index 4f00412d72d3..89a10624e361 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -270,30 +270,29 @@ void filemap_remove_folio(struct folio *folio)
 }
 
 /*
- * page_cache_delete_batch - delete several pages from page cache
- * @mapping: the mapping to which pages belong
- * @pvec: pagevec with pages to delete
+ * page_cache_delete_batch - delete several folios from page cache
+ * @mapping: the mapping to which folios belong
+ * @fbatch: batch of folios to delete
  *
- * The function walks over mapping->i_pages and removes pages passed in @pvec
- * from the mapping. The function expects @pvec to be sorted by page index
- * and is optimised for it to be dense.
- * It tolerates holes in @pvec (mapping entries at those indices are not
- * modified). The function expects only THP head pages to be present in the
- * @pvec.
+ * The function walks over mapping->i_pages and removes folios passed in
+ * @fbatch from the mapping. The function expects @fbatch to be sorted
+ * by page index and is optimised for it to be dense.
+ * It tolerates holes in @fbatch (mapping entries at those indices are not
+ * modified).
  *
  * The function expects the i_pages lock to be held.
  */
 static void page_cache_delete_batch(struct address_space *mapping,
-			     struct pagevec *pvec)
+			     struct folio_batch *fbatch)
 {
-	XA_STATE(xas, &mapping->i_pages, pvec->pages[0]->index);
+	XA_STATE(xas, &mapping->i_pages, fbatch->folios[0]->index);
 	int total_pages = 0;
 	int i = 0;
 	struct folio *folio;
 
 	mapping_set_update(&xas, mapping);
 	xas_for_each(&xas, folio, ULONG_MAX) {
-		if (i >= pagevec_count(pvec))
+		if (i >= folio_batch_count(fbatch))
 			break;
 
 		/* A swap/dax/shadow entry got inserted? Skip it. */
@@ -306,9 +305,9 @@ static void page_cache_delete_batch(struct address_space *mapping,
 		 * means our page has been removed, which shouldn't be
 		 * possible because we're holding the PageLock.
 		 */
-		if (&folio->page != pvec->pages[i]) {
+		if (folio != fbatch->folios[i]) {
 			VM_BUG_ON_FOLIO(folio->index >
-						pvec->pages[i]->index, folio);
+					fbatch->folios[i]->index, folio);
 			continue;
 		}
 
@@ -316,12 +315,11 @@ static void page_cache_delete_batch(struct address_space *mapping,
 
 		if (folio->index == xas.xa_index)
 			folio->mapping = NULL;
-		/* Leave page->index set: truncation lookup relies on it */
+		/* Leave folio->index set: truncation lookup relies on it */
 
 		/*
-		 * Move to the next page in the vector if this is a regular
-		 * page or the index is of the last sub-page of this compound
-		 * page.
+		 * Move to the next folio in the batch if this is a regular
+		 * folio or the index is of the last sub-page of this folio.
 		 */
 		if (folio->index + folio_nr_pages(folio) - 1 == xas.xa_index)
 			i++;
@@ -332,29 +330,29 @@ static void page_cache_delete_batch(struct address_space *mapping,
 }
 
 void delete_from_page_cache_batch(struct address_space *mapping,
-				  struct pagevec *pvec)
+				  struct folio_batch *fbatch)
 {
 	int i;
 
-	if (!pagevec_count(pvec))
+	if (!folio_batch_count(fbatch))
 		return;
 
 	spin_lock(&mapping->host->i_lock);
 	xa_lock_irq(&mapping->i_pages);
-	for (i = 0; i < pagevec_count(pvec); i++) {
-		struct folio *folio = page_folio(pvec->pages[i]);
+	for (i = 0; i < folio_batch_count(fbatch); i++) {
+		struct folio *folio = fbatch->folios[i];
 
 		trace_mm_filemap_delete_from_page_cache(folio);
 		filemap_unaccount_folio(mapping, folio);
 	}
-	page_cache_delete_batch(mapping, pvec);
+	page_cache_delete_batch(mapping, fbatch);
 	xa_unlock_irq(&mapping->i_pages);
 	if (mapping_shrinkable(mapping))
 		inode_add_lru(mapping->host);
 	spin_unlock(&mapping->host->i_lock);
 
-	for (i = 0; i < pagevec_count(pvec); i++)
-		filemap_free_folio(mapping, page_folio(pvec->pages[i]));
+	for (i = 0; i < folio_batch_count(fbatch); i++)
+		filemap_free_folio(mapping, fbatch->folios[i]);
 }
 
 int filemap_check_errors(struct address_space *mapping)
@@ -2052,8 +2050,8 @@ unsigned find_get_entries(struct address_space *mapping, pgoff_t start,
  * @mapping:	The address_space to search.
  * @start:	The starting page cache index.
  * @end:	The final page index (inclusive).
- * @pvec:	Where the resulting entries are placed.
- * @indices:	The cache indices of the entries in @pvec.
+ * @fbatch:	Where the resulting entries are placed.
+ * @indices:	The cache indices of the entries in @fbatch.
  *
  * find_lock_entries() will return a batch of entries from @mapping.
  * Swap, shadow and DAX entries are included.  Folios are returned
@@ -2068,7 +2066,7 @@ unsigned find_get_entries(struct address_space *mapping, pgoff_t start,
  * Return: The number of entries which were found.
  */
 unsigned find_lock_entries(struct address_space *mapping, pgoff_t start,
-		pgoff_t end, struct pagevec *pvec, pgoff_t *indices)
+		pgoff_t end, struct folio_batch *fbatch, pgoff_t *indices)
 {
 	XA_STATE(xas, &mapping->i_pages, start);
 	struct folio *folio;
@@ -2088,8 +2086,8 @@ unsigned find_lock_entries(struct address_space *mapping, pgoff_t start,
 			VM_BUG_ON_FOLIO(!folio_contains(folio, xas.xa_index),
 					folio);
 		}
-		indices[pvec->nr] = xas.xa_index;
-		if (!pagevec_add(pvec, &folio->page))
+		indices[fbatch->nr] = xas.xa_index;
+		if (!folio_batch_add(fbatch, folio))
 			break;
 		goto next;
 unlock:
@@ -2106,7 +2104,7 @@ unsigned find_lock_entries(struct address_space *mapping, pgoff_t start,
 	}
 	rcu_read_unlock();
 
-	return pagevec_count(pvec);
+	return folio_batch_count(fbatch);
 }
 
 /**
diff --git a/mm/internal.h b/mm/internal.h
index 36ad6ffe53bf..7759d4ff3323 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -114,7 +114,7 @@ static inline void force_page_cache_readahead(struct address_space *mapping,
 }
 
 unsigned find_lock_entries(struct address_space *mapping, pgoff_t start,
-		pgoff_t end, struct pagevec *pvec, pgoff_t *indices);
+		pgoff_t end, struct folio_batch *fbatch, pgoff_t *indices);
 unsigned find_get_entries(struct address_space *mapping, pgoff_t start,
 		pgoff_t end, struct folio_batch *fbatch, pgoff_t *indices);
 void filemap_free_folio(struct address_space *mapping, struct folio *folio);
diff --git a/mm/shmem.c b/mm/shmem.c
index e909c163fb38..bbfa2d05e787 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -919,7 +919,6 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, loff_t lend,
 	pgoff_t end = (lend + 1) >> PAGE_SHIFT;
 	unsigned int partial_start = lstart & (PAGE_SIZE - 1);
 	unsigned int partial_end = (lend + 1) & (PAGE_SIZE - 1);
-	struct pagevec pvec;
 	struct folio_batch fbatch;
 	pgoff_t indices[PAGEVEC_SIZE];
 	long nr_swaps_freed = 0;
@@ -932,12 +931,12 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, loff_t lend,
 	if (info->fallocend > start && info->fallocend <= end && !unfalloc)
 		info->fallocend = start;
 
-	pagevec_init(&pvec);
+	folio_batch_init(&fbatch);
 	index = start;
 	while (index < end && find_lock_entries(mapping, index, end - 1,
-			&pvec, indices)) {
-		for (i = 0; i < pagevec_count(&pvec); i++) {
-			struct folio *folio = (struct folio *)pvec.pages[i];
+			&fbatch, indices)) {
+		for (i = 0; i < folio_batch_count(&fbatch); i++) {
+			struct folio *folio = fbatch.folios[i];
 
 			index = indices[i];
 
@@ -954,8 +953,8 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, loff_t lend,
 				truncate_inode_folio(mapping, folio);
 			folio_unlock(folio);
 		}
-		pagevec_remove_exceptionals(&pvec);
-		pagevec_release(&pvec);
+		folio_batch_remove_exceptionals(&fbatch);
+		folio_batch_release(&fbatch);
 		cond_resched();
 		index++;
 	}
@@ -988,7 +987,6 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, loff_t lend,
 	if (start >= end)
 		return;
 
-	folio_batch_init(&fbatch);
 	index = start;
 	while (index < end) {
 		cond_resched();
diff --git a/mm/truncate.c b/mm/truncate.c
index 357af144df63..e7f5762c43d3 100644
--- a/mm/truncate.c
+++ b/mm/truncate.c
@@ -56,11 +56,11 @@ static void clear_shadow_entry(struct address_space *mapping, pgoff_t index,
 
 /*
  * Unconditionally remove exceptional entries. Usually called from truncate
- * path. Note that the pagevec may be altered by this function by removing
+ * path. Note that the folio_batch may be altered by this function by removing
  * exceptional entries similar to what pagevec_remove_exceptionals does.
  */
-static void truncate_exceptional_pvec_entries(struct address_space *mapping,
-				struct pagevec *pvec, pgoff_t *indices)
+static void truncate_folio_batch_exceptionals(struct address_space *mapping,
+				struct folio_batch *fbatch, pgoff_t *indices)
 {
 	int i, j;
 	bool dax;
@@ -69,11 +69,11 @@ static void truncate_exceptional_pvec_entries(struct address_space *mapping,
 	if (shmem_mapping(mapping))
 		return;
 
-	for (j = 0; j < pagevec_count(pvec); j++)
-		if (xa_is_value(pvec->pages[j]))
+	for (j = 0; j < folio_batch_count(fbatch); j++)
+		if (xa_is_value(fbatch->folios[j]))
 			break;
 
-	if (j == pagevec_count(pvec))
+	if (j == folio_batch_count(fbatch))
 		return;
 
 	dax = dax_mapping(mapping);
@@ -82,12 +82,12 @@ static void truncate_exceptional_pvec_entries(struct address_space *mapping,
 		xa_lock_irq(&mapping->i_pages);
 	}
 
-	for (i = j; i < pagevec_count(pvec); i++) {
-		struct page *page = pvec->pages[i];
+	for (i = j; i < folio_batch_count(fbatch); i++) {
+		struct folio *folio = fbatch->folios[i];
 		pgoff_t index = indices[i];
 
-		if (!xa_is_value(page)) {
-			pvec->pages[j++] = page;
+		if (!xa_is_value(folio)) {
+			fbatch->folios[j++] = folio;
 			continue;
 		}
 
@@ -96,7 +96,7 @@ static void truncate_exceptional_pvec_entries(struct address_space *mapping,
 			continue;
 		}
 
-		__clear_shadow_entry(mapping, index, page);
+		__clear_shadow_entry(mapping, index, folio);
 	}
 
 	if (!dax) {
@@ -105,14 +105,7 @@ static void truncate_exceptional_pvec_entries(struct address_space *mapping,
 			inode_add_lru(mapping->host);
 		spin_unlock(&mapping->host->i_lock);
 	}
-	pvec->nr = j;
-}
-
-static void truncate_folio_batch_exceptionals(struct address_space *mapping,
-				struct folio_batch *fbatch, pgoff_t *indices)
-{
-	truncate_exceptional_pvec_entries(mapping, (struct pagevec *)fbatch,
-						indices);
+	fbatch->nr = j;
 }
 
 /*
@@ -303,7 +296,6 @@ void truncate_inode_pages_range(struct address_space *mapping,
 	pgoff_t		end;		/* exclusive */
 	unsigned int	partial_start;	/* inclusive */
 	unsigned int	partial_end;	/* exclusive */
-	struct pagevec	pvec;
 	struct folio_batch fbatch;
 	pgoff_t		indices[PAGEVEC_SIZE];
 	pgoff_t		index;
@@ -333,18 +325,18 @@ void truncate_inode_pages_range(struct address_space *mapping,
 	else
 		end = (lend + 1) >> PAGE_SHIFT;
 
-	pagevec_init(&pvec);
+	folio_batch_init(&fbatch);
 	index = start;
 	while (index < end && find_lock_entries(mapping, index, end - 1,
-			&pvec, indices)) {
-		index = indices[pagevec_count(&pvec) - 1] + 1;
-		truncate_exceptional_pvec_entries(mapping, &pvec, indices);
-		for (i = 0; i < pagevec_count(&pvec); i++)
-			truncate_cleanup_folio(page_folio(pvec.pages[i]));
-		delete_from_page_cache_batch(mapping, &pvec);
-		for (i = 0; i < pagevec_count(&pvec); i++)
-			unlock_page(pvec.pages[i]);
-		pagevec_release(&pvec);
+			&fbatch, indices)) {
+		index = indices[folio_batch_count(&fbatch) - 1] + 1;
+		truncate_folio_batch_exceptionals(mapping, &fbatch, indices);
+		for (i = 0; i < folio_batch_count(&fbatch); i++)
+			truncate_cleanup_folio(fbatch.folios[i]);
+		delete_from_page_cache_batch(mapping, &fbatch);
+		for (i = 0; i < folio_batch_count(&fbatch); i++)
+			folio_unlock(fbatch.folios[i]);
+		folio_batch_release(&fbatch);
 		cond_resched();
 	}
 
@@ -387,7 +379,6 @@ void truncate_inode_pages_range(struct address_space *mapping,
 	if (start >= end)
 		goto out;
 
-	folio_batch_init(&fbatch);
 	index = start;
 	for ( ; ; ) {
 		cond_resched();
@@ -489,16 +480,16 @@ static unsigned long __invalidate_mapping_pages(struct address_space *mapping,
 		pgoff_t start, pgoff_t end, unsigned long *nr_pagevec)
 {
 	pgoff_t indices[PAGEVEC_SIZE];
-	struct pagevec pvec;
+	struct folio_batch fbatch;
 	pgoff_t index = start;
 	unsigned long ret;
 	unsigned long count = 0;
 	int i;
 
-	pagevec_init(&pvec);
-	while (find_lock_entries(mapping, index, end, &pvec, indices)) {
-		for (i = 0; i < pagevec_count(&pvec); i++) {
-			struct page *page = pvec.pages[i];
+	folio_batch_init(&fbatch);
+	while (find_lock_entries(mapping, index, end, &fbatch, indices)) {
+		for (i = 0; i < folio_batch_count(&fbatch); i++) {
+			struct page *page = &fbatch.folios[i]->page;
 
 			/* We rely upon deletion not changing page->index */
 			index = indices[i];
@@ -525,8 +516,8 @@ static unsigned long __invalidate_mapping_pages(struct address_space *mapping,
 			}
 			count += ret;
 		}
-		pagevec_remove_exceptionals(&pvec);
-		pagevec_release(&pvec);
+		folio_batch_remove_exceptionals(&fbatch);
+		folio_batch_release(&fbatch);
 		cond_resched();
 		index++;
 	}
-- 
2.33.0

[PATCH 43/48] mm: Remove pagevec_remove_exceptionals()

[Matthew Wilcox (Oracle)]

All of its callers now call folio_batch_remove_exceptionals().

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 include/linux/pagevec.h |  6 +-----
 mm/swap.c               | 26 +++++++++++++-------------
 mm/truncate.c           |  2 +-
 3 files changed, 15 insertions(+), 19 deletions(-)

diff --git a/include/linux/pagevec.h b/include/linux/pagevec.h
index 4483e6ad7607..7d3494f7fb70 100644
--- a/include/linux/pagevec.h
+++ b/include/linux/pagevec.h
@@ -26,7 +26,6 @@ struct pagevec {
 
 void __pagevec_release(struct pagevec *pvec);
 void __pagevec_lru_add(struct pagevec *pvec);
-void pagevec_remove_exceptionals(struct pagevec *pvec);
 unsigned pagevec_lookup_range(struct pagevec *pvec,
 			      struct address_space *mapping,
 			      pgoff_t *start, pgoff_t end);
@@ -140,8 +139,5 @@ static inline void folio_batch_release(struct folio_batch *fbatch)
 	pagevec_release((struct pagevec *)fbatch);
 }
 
-static inline void folio_batch_remove_exceptionals(struct folio_batch *fbatch)
-{
-	pagevec_remove_exceptionals((struct pagevec *)fbatch);
-}
+void folio_batch_remove_exceptionals(struct folio_batch *fbatch);
 #endif /* _LINUX_PAGEVEC_H */
diff --git a/mm/swap.c b/mm/swap.c
index e8c9dc6d0377..74f6b311d7ee 100644
--- a/mm/swap.c
+++ b/mm/swap.c
@@ -1077,24 +1077,24 @@ void __pagevec_lru_add(struct pagevec *pvec)
 }
 
 /**
- * pagevec_remove_exceptionals - pagevec exceptionals pruning
- * @pvec:	The pagevec to prune
+ * folio_batch_remove_exceptionals() - Prune non-folios from a batch.
+ * @fbatch: The batch to prune
  *
- * find_get_entries() fills both pages and XArray value entries (aka
- * exceptional entries) into the pagevec.  This function prunes all
- * exceptionals from @pvec without leaving holes, so that it can be
- * passed on to page-only pagevec operations.
+ * find_get_entries() fills a batch with both folios and shadow/swap/DAX
+ * entries.  This function prunes all the non-folio entries from @fbatch
+ * without leaving holes, so that it can be passed on to folio-only batch
+ * operations.
  */
-void pagevec_remove_exceptionals(struct pagevec *pvec)
+void folio_batch_remove_exceptionals(struct folio_batch *fbatch)
 {
-	int i, j;
+	unsigned int i, j;
 
-	for (i = 0, j = 0; i < pagevec_count(pvec); i++) {
-		struct page *page = pvec->pages[i];
-		if (!xa_is_value(page))
-			pvec->pages[j++] = page;
+	for (i = 0, j = 0; i < folio_batch_count(fbatch); i++) {
+		struct folio *folio = fbatch->folios[i];
+		if (!xa_is_value(folio))
+			fbatch->folios[j++] = folio;
 	}
-	pvec->nr = j;
+	fbatch->nr = j;
 }
 
 /**
diff --git a/mm/truncate.c b/mm/truncate.c
index e7f5762c43d3..a1113b0abb30 100644
--- a/mm/truncate.c
+++ b/mm/truncate.c
@@ -57,7 +57,7 @@ static void clear_shadow_entry(struct address_space *mapping, pgoff_t index,
 /*
  * Unconditionally remove exceptional entries. Usually called from truncate
  * path. Note that the folio_batch may be altered by this function by removing
- * exceptional entries similar to what pagevec_remove_exceptionals does.
+ * exceptional entries similar to what folio_batch_remove_exceptionals() does.
  */
 static void truncate_folio_batch_exceptionals(struct address_space *mapping,
 				struct folio_batch *fbatch, pgoff_t *indices)
-- 
2.33.0

[PATCH 44/48] fs: Convert vfs_dedupe_file_range_compare to folios

[Matthew Wilcox (Oracle)]

We still only operate on a single page of data at a time due to using
kmap().  A more complex implementation would work on each page in a folio,
but it's not clear that such a complex implementation would be worthwhile.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 fs/remap_range.c | 116 ++++++++++++++++++++++-------------------------
 1 file changed, 55 insertions(+), 61 deletions(-)

diff --git a/fs/remap_range.c b/fs/remap_range.c
index 6d4a9beaa097..231159682907 100644
--- a/fs/remap_range.c
+++ b/fs/remap_range.c
@@ -146,41 +146,41 @@ static int generic_remap_check_len(struct inode *inode_in,
 }
 
 /* Read a page's worth of file data into the page cache. */
-static struct page *vfs_dedupe_get_page(struct inode *inode, loff_t offset)
+static struct folio *vfs_dedupe_get_folio(struct inode *inode, loff_t pos)
 {
-	struct page *page;
+	struct folio *folio;
 
-	page = read_mapping_page(inode->i_mapping, offset >> PAGE_SHIFT, NULL);
-	if (IS_ERR(page))
-		return page;
-	if (!PageUptodate(page)) {
-		put_page(page);
+	folio = read_mapping_folio(inode->i_mapping, pos >> PAGE_SHIFT, NULL);
+	if (IS_ERR(folio))
+		return folio;
+	if (!folio_test_uptodate(folio)) {
+		folio_put(folio);
 		return ERR_PTR(-EIO);
 	}
-	return page;
+	return folio;
 }
 
 /*
- * Lock two pages, ensuring that we lock in offset order if the pages are from
- * the same file.
+ * Lock two folios, ensuring that we lock in offset order if the folios
+ * are from the same file.
  */
-static void vfs_lock_two_pages(struct page *page1, struct page *page2)
+static void vfs_lock_two_folios(struct folio *folio1, struct folio *folio2)
 {
 	/* Always lock in order of increasing index. */
-	if (page1->index > page2->index)
-		swap(page1, page2);
+	if (folio1->index > folio2->index)
+		swap(folio1, folio2);
 
-	lock_page(page1);
-	if (page1 != page2)
-		lock_page(page2);
+	folio_lock(folio1);
+	if (folio1 != folio2)
+		folio_lock(folio2);
 }
 
-/* Unlock two pages, being careful not to unlock the same page twice. */
-static void vfs_unlock_two_pages(struct page *page1, struct page *page2)
+/* Unlock two folios, being careful not to unlock the same folio twice. */
+static void vfs_unlock_two_folios(struct folio *folio1, struct folio *folio2)
 {
-	unlock_page(page1);
-	if (page1 != page2)
-		unlock_page(page2);
+	folio_unlock(folio1);
+	if (folio1 != folio2)
+		folio_unlock(folio2);
 }
 
 /*
@@ -188,77 +188,71 @@ static void vfs_unlock_two_pages(struct page *page1, struct page *page2)
  * Caller must have locked both inodes to prevent write races.
  */
 static int vfs_dedupe_file_range_compare(struct inode *src, loff_t srcoff,
-					 struct inode *dest, loff_t destoff,
+					 struct inode *dest, loff_t dstoff,
 					 loff_t len, bool *is_same)
 {
-	loff_t src_poff;
-	loff_t dest_poff;
-	void *src_addr;
-	void *dest_addr;
-	struct page *src_page;
-	struct page *dest_page;
-	loff_t cmp_len;
-	bool same;
-	int error;
-
-	error = -EINVAL;
-	same = true;
+	bool same = true;
+	int error = -EINVAL;
+
 	while (len) {
-		src_poff = srcoff & (PAGE_SIZE - 1);
-		dest_poff = destoff & (PAGE_SIZE - 1);
-		cmp_len = min(PAGE_SIZE - src_poff,
-			      PAGE_SIZE - dest_poff);
+		struct folio *src_folio, *dst_folio;
+		void *src_addr, *dst_addr;
+		loff_t cmp_len = min(PAGE_SIZE - offset_in_page(srcoff),
+				     PAGE_SIZE - offset_in_page(dstoff));
+
 		cmp_len = min(cmp_len, len);
 		if (cmp_len <= 0)
 			goto out_error;
 
-		src_page = vfs_dedupe_get_page(src, srcoff);
-		if (IS_ERR(src_page)) {
-			error = PTR_ERR(src_page);
+		src_folio = vfs_dedupe_get_folio(src, srcoff);
+		if (IS_ERR(src_folio)) {
+			error = PTR_ERR(src_folio);
 			goto out_error;
 		}
-		dest_page = vfs_dedupe_get_page(dest, destoff);
-		if (IS_ERR(dest_page)) {
-			error = PTR_ERR(dest_page);
-			put_page(src_page);
+		dst_folio = vfs_dedupe_get_folio(dest, dstoff);
+		if (IS_ERR(dst_folio)) {
+			error = PTR_ERR(dst_folio);
+			folio_put(src_folio);
 			goto out_error;
 		}
 
-		vfs_lock_two_pages(src_page, dest_page);
+		vfs_lock_two_folios(src_folio, dst_folio);
 
 		/*
-		 * Now that we've locked both pages, make sure they're still
+		 * Now that we've locked both folios, make sure they're still
 		 * mapped to the file data we're interested in.  If not,
 		 * someone is invalidating pages on us and we lose.
 		 */
-		if (!PageUptodate(src_page) || !PageUptodate(dest_page) ||
-		    src_page->mapping != src->i_mapping ||
-		    dest_page->mapping != dest->i_mapping) {
+		if (!folio_test_uptodate(src_folio) || !folio_test_uptodate(dst_folio) ||
+		    src_folio->mapping != src->i_mapping ||
+		    dst_folio->mapping != dest->i_mapping) {
 			same = false;
 			goto unlock;
 		}
 
-		src_addr = kmap_atomic(src_page);
-		dest_addr = kmap_atomic(dest_page);
+		src_addr = kmap_local_folio(src_folio,
+					offset_in_folio(src_folio, srcoff));
+		dst_addr = kmap_local_folio(dst_folio,
+					offset_in_folio(dst_folio, dstoff));
 
-		flush_dcache_page(src_page);
-		flush_dcache_page(dest_page);
+		flush_dcache_folio(src_folio);
+		flush_dcache_folio(dst_folio);
 
-		if (memcmp(src_addr + src_poff, dest_addr + dest_poff, cmp_len))
+		if (memcmp(src_addr, dst_addr, cmp_len))
 			same = false;
 
-		kunmap_atomic(dest_addr);
-		kunmap_atomic(src_addr);
+		kunmap_local(dst_addr);
+		kunmap_local(src_addr);
 unlock:
-		vfs_unlock_two_pages(src_page, dest_page);
-		put_page(dest_page);
-		put_page(src_page);
+		vfs_unlock_two_folios(src_folio, dst_folio);
+		folio_put(dst_folio);
+		folio_put(src_folio);
 
 		if (!same)
 			break;
 
 		srcoff += cmp_len;
-		destoff += cmp_len;
+		dstoff += cmp_len;
 		len -= cmp_len;
 	}
 
-- 
2.33.0

[PATCH 45/48] truncate: Convert invalidate_inode_pages2_range to folios

[Matthew Wilcox (Oracle)]

If we're going to unmap a folio, we have to be sure to unmap the entire
folio, not just the part of it which lies after the search index.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 mm/truncate.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/mm/truncate.c b/mm/truncate.c
index a1113b0abb30..2d1dae085acb 100644
--- a/mm/truncate.c
+++ b/mm/truncate.c
@@ -599,13 +599,13 @@ static int invalidate_complete_folio2(struct address_space *mapping,
 	return 0;
 }
 
-static int do_launder_page(struct address_space *mapping, struct page *page)
+static int do_launder_folio(struct address_space *mapping, struct folio *folio)
 {
-	if (!PageDirty(page))
+	if (!folio_test_dirty(folio))
 		return 0;
-	if (page->mapping != mapping || mapping->a_ops->launder_page == NULL)
+	if (folio->mapping != mapping || mapping->a_ops->launder_page == NULL)
 		return 0;
-	return mapping->a_ops->launder_page(page);
+	return mapping->a_ops->launder_page(&folio->page);
 }
 
 /**
@@ -671,7 +671,7 @@ int invalidate_inode_pages2_range(struct address_space *mapping,
 				unmap_mapping_folio(folio);
 			BUG_ON(folio_mapped(folio));
 
-			ret2 = do_launder_page(mapping, &folio->page);
+			ret2 = do_launder_folio(mapping, folio);
 			if (ret2 == 0) {
 				if (!invalidate_complete_folio2(mapping, folio))
 					ret2 = -EBUSY;
-- 
2.33.0

[PATCH 46/48] truncate,shmem: Handle truncates that split large folios

[Matthew Wilcox (Oracle)]

Handle folio splitting in the parts of the truncation functions which
already handle partial pages.  Factor all that code out into a new
function called truncate_inode_partial_folio().

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
Reviewed-by: Jan Kara <jack@suse.cz>
Reviewed-by: William Kucharski <william.kucharski@oracle.com>
---
 mm/internal.h |   2 +
 mm/shmem.c    | 107 ++++++++++++++++++---------------------------
 mm/truncate.c | 118 ++++++++++++++++++++++++++++++++------------------
 3 files changed, 120 insertions(+), 107 deletions(-)

diff --git a/mm/internal.h b/mm/internal.h
index 7759d4ff3323..e989d8ceec91 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -119,6 +119,8 @@ unsigned find_get_entries(struct address_space *mapping, pgoff_t start,
 		pgoff_t end, struct folio_batch *fbatch, pgoff_t *indices);
 void filemap_free_folio(struct address_space *mapping, struct folio *folio);
 int truncate_inode_folio(struct address_space *mapping, struct folio *folio);
+bool truncate_inode_partial_folio(struct folio *folio, loff_t start,
+		loff_t end);
 
 /**
  * folio_evictable - Test whether a folio is evictable.
diff --git a/mm/shmem.c b/mm/shmem.c
index bbfa2d05e787..7f0b07845c1f 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -151,6 +151,19 @@ int shmem_getpage(struct inode *inode, pgoff_t index,
 		mapping_gfp_mask(inode->i_mapping), NULL, NULL, NULL);
 }
 
+int shmem_get_folio(struct inode *inode, pgoff_t index,
+		struct folio **foliop, enum sgp_type sgp)
+{
+	struct page *page = NULL;
+	int ret = shmem_getpage(inode, index, &page, sgp);
+
+	if (page)
+		*foliop = page_folio(page);
+	else
+		*foliop = NULL;
+	return ret;
+}
+
 static inline struct shmem_sb_info *SHMEM_SB(struct super_block *sb)
 {
 	return sb->s_fs_info;
@@ -880,32 +893,6 @@ void shmem_unlock_mapping(struct address_space *mapping)
 	}
 }
 
-/*
- * Check whether a hole-punch or truncation needs to split a huge page,
- * returning true if no split was required, or the split has been successful.
- *
- * Eviction (or truncation to 0 size) should never need to split a huge page;
- * but in rare cases might do so, if shmem_undo_range() failed to trylock on
- * head, and then succeeded to trylock on tail.
- *
- * A split can only succeed when there are no additional references on the
- * huge page: so the split below relies upon find_get_entries() having stopped
- * when it found a subpage of the huge page, without getting further references.
- */
-static bool shmem_punch_compound(struct page *page, pgoff_t start, pgoff_t end)
-{
-	if (!PageTransCompound(page))
-		return true;
-
-	/* Just proceed to delete a huge page wholly within the range punched */
-	if (PageHead(page) &&
-	    page->index >= start && page->index + HPAGE_PMD_NR <= end)
-		return true;
-
-	/* Try to split huge page, so we can truly punch the hole or truncate */
-	return split_huge_page(page) >= 0;
-}
-
 /*
  * Remove range of pages and swap entries from page cache, and free them.
  * If !unfalloc, truncate or punch hole; if unfalloc, undo failed fallocate.
@@ -917,13 +904,13 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, loff_t lend,
 	struct shmem_inode_info *info = SHMEM_I(inode);
 	pgoff_t start = (lstart + PAGE_SIZE - 1) >> PAGE_SHIFT;
 	pgoff_t end = (lend + 1) >> PAGE_SHIFT;
-	unsigned int partial_start = lstart & (PAGE_SIZE - 1);
-	unsigned int partial_end = (lend + 1) & (PAGE_SIZE - 1);
 	struct folio_batch fbatch;
 	pgoff_t indices[PAGEVEC_SIZE];
+	struct folio *folio;
 	long nr_swaps_freed = 0;
 	pgoff_t index;
 	int i;
+	bool partial_end;
 
 	if (lend == -1)
 		end = -1;	/* unsigned, so actually very big */
@@ -959,33 +946,34 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, loff_t lend,
 		index++;
 	}
 
-	if (partial_start) {
-		struct page *page = NULL;
-		shmem_getpage(inode, start - 1, &page, SGP_READ);
-		if (page) {
-			unsigned int top = PAGE_SIZE;
-			if (start > end) {
-				top = partial_end;
-				partial_end = 0;
-			}
-			zero_user_segment(page, partial_start, top);
-			set_page_dirty(page);
-			unlock_page(page);
-			put_page(page);
+	partial_end = ((lend + 1) % PAGE_SIZE) > 0;
+	shmem_get_folio(inode, lstart >> PAGE_SHIFT, &folio, SGP_READ);
+	if (folio) {
+		bool same_page;
+
+		same_page = lend < folio_pos(folio) + folio_size(folio);
+		if (same_page)
+			partial_end = false;
+		folio_mark_dirty(folio);
+		if (!truncate_inode_partial_folio(folio, lstart, lend)) {
+			start = folio->index + folio_nr_pages(folio);
+			if (same_page)
+				end = folio->index;
 		}
+		folio_unlock(folio);
+		folio_put(folio);
+		folio = NULL;
 	}
-	if (partial_end) {
-		struct page *page = NULL;
-		shmem_getpage(inode, end, &page, SGP_READ);
-		if (page) {
-			zero_user_segment(page, 0, partial_end);
-			set_page_dirty(page);
-			unlock_page(page);
-			put_page(page);
-		}
+
+	if (partial_end)
+		shmem_get_folio(inode, end, &folio, SGP_READ);
+	if (folio) {
+		folio_mark_dirty(folio);
+		if (!truncate_inode_partial_folio(folio, lstart, lend))
+			end = folio->index;
+		folio_unlock(folio);
+		folio_put(folio);
 	}
-	if (start >= end)
-		return;
 
 	index = start;
 	while (index < end) {
@@ -1019,8 +1007,6 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, loff_t lend,
 			folio_lock(folio);
 
 			if (!unfalloc || !folio_test_uptodate(folio)) {
-				struct page *page = folio_file_page(folio,
-									index);
 				if (folio_mapping(folio) != mapping) {
 					/* Page was replaced by swap: retry */
 					folio_unlock(folio);
@@ -1029,18 +1015,9 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, loff_t lend,
 				}
 				VM_BUG_ON_FOLIO(folio_test_writeback(folio),
 						folio);
-				if (shmem_punch_compound(page, start, end))
-					truncate_inode_folio(mapping, folio);
-				else if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)) {
-					/* Wipe the page and don't get stuck */
-					clear_highpage(page);
-					flush_dcache_page(page);
-					folio_mark_dirty(folio);
-					if (index <
-					    round_up(start, HPAGE_PMD_NR))
-						start = index + 1;
-				}
+				truncate_inode_folio(mapping, folio);
 			}
+			index = folio->index + folio_nr_pages(folio) - 1;
 			folio_unlock(folio);
 		}
 		folio_batch_remove_exceptionals(&fbatch);
diff --git a/mm/truncate.c b/mm/truncate.c
index 2d1dae085acb..336c8d099efa 100644
--- a/mm/truncate.c
+++ b/mm/truncate.c
@@ -228,6 +228,58 @@ int truncate_inode_folio(struct address_space *mapping, struct folio *folio)
 	return 0;
 }
 
+/*
+ * Handle partial folios.  The folio may be entirely within the
+ * range if a split has raced with us.  If not, we zero the part of the
+ * folio that's within the [start, end] range, and then split the folio if
+ * it's large.  split_page_range() will discard pages which now lie beyond
+ * i_size, and we rely on the caller to discard pages which lie within a
+ * newly created hole.
+ *
+ * Returns false if splitting failed so the caller can avoid
+ * discarding the entire folio which is stubbornly unsplit.
+ */
+bool truncate_inode_partial_folio(struct folio *folio, loff_t start, loff_t end)
+{
+	loff_t pos = folio_pos(folio);
+	unsigned int offset, length;
+
+	if (pos < start)
+		offset = start - pos;
+	else
+		offset = 0;
+	length = folio_size(folio);
+	if (pos + length <= (u64)end)
+		length = length - offset;
+	else
+		length = end + 1 - pos - offset;
+
+	folio_wait_writeback(folio);
+	if (length == folio_size(folio)) {
+		truncate_inode_folio(folio->mapping, folio);
+		return true;
+	}
+
+	/*
+	 * We may be zeroing pages we're about to discard, but it avoids
+	 * doing a complex calculation here, and then doing the zeroing
+	 * anyway if the page split fails.
+	 */
+	folio_zero_range(folio, offset, length);
+
+	cleancache_invalidate_page(folio->mapping, &folio->page);
+	if (folio_has_private(folio))
+		do_invalidatepage(&folio->page, offset, length);
+	if (!folio_test_large(folio))
+		return true;
+	if (split_huge_page(&folio->page) == 0)
+		return true;
+	if (folio_test_dirty(folio))
+		return false;
+	truncate_inode_folio(folio->mapping, folio);
+	return true;
+}
+
 /*
  * Used to get rid of pages on hardware memory corruption.
  */
@@ -294,20 +346,16 @@ void truncate_inode_pages_range(struct address_space *mapping,
 {
 	pgoff_t		start;		/* inclusive */
 	pgoff_t		end;		/* exclusive */
-	unsigned int	partial_start;	/* inclusive */
-	unsigned int	partial_end;	/* exclusive */
 	struct folio_batch fbatch;
 	pgoff_t		indices[PAGEVEC_SIZE];
 	pgoff_t		index;
 	int		i;
+	struct folio *	folio;
+	bool partial_end;
 
 	if (mapping_empty(mapping))
 		goto out;
 
-	/* Offsets within partial pages */
-	partial_start = lstart & (PAGE_SIZE - 1);
-	partial_end = (lend + 1) & (PAGE_SIZE - 1);
-
 	/*
 	 * 'start' and 'end' always covers the range of pages to be fully
 	 * truncated. Partial pages are covered with 'partial_start' at the
@@ -340,47 +388,33 @@ void truncate_inode_pages_range(struct address_space *mapping,
 		cond_resched();
 	}
 
-	if (partial_start) {
-		struct page *page = find_lock_page(mapping, start - 1);
-		if (page) {
-			unsigned int top = PAGE_SIZE;
-			if (start > end) {
-				/* Truncation within a single page */
-				top = partial_end;
-				partial_end = 0;
-			}
-			wait_on_page_writeback(page);
-			zero_user_segment(page, partial_start, top);
-			cleancache_invalidate_page(mapping, page);
-			if (page_has_private(page))
-				do_invalidatepage(page, partial_start,
-						  top - partial_start);
-			unlock_page(page);
-			put_page(page);
+	partial_end = ((lend + 1) % PAGE_SIZE) > 0;
+	folio = __filemap_get_folio(mapping, lstart >> PAGE_SHIFT, FGP_LOCK, 0);
+	if (folio) {
+		bool same_folio = lend < folio_pos(folio) + folio_size(folio);
+		if (same_folio)
+			partial_end = false;
+		if (!truncate_inode_partial_folio(folio, lstart, lend)) {
+			start = folio->index + folio_nr_pages(folio);
+			if (same_folio)
+				end = folio->index;
 		}
+		folio_unlock(folio);
+		folio_put(folio);
+		folio = NULL;
 	}
-	if (partial_end) {
-		struct page *page = find_lock_page(mapping, end);
-		if (page) {
-			wait_on_page_writeback(page);
-			zero_user_segment(page, 0, partial_end);
-			cleancache_invalidate_page(mapping, page);
-			if (page_has_private(page))
-				do_invalidatepage(page, 0,
-						  partial_end);
-			unlock_page(page);
-			put_page(page);
-		}
+
+	if (partial_end)
+		folio = __filemap_get_folio(mapping, end, FGP_LOCK, 0);
+	if (folio) {
+		if (!truncate_inode_partial_folio(folio, lstart, lend))
+			end = folio->index;
+		folio_unlock(folio);
+		folio_put(folio);
 	}
-	/*
-	 * If the truncation happened within a single page no pages
-	 * will be released, just zeroed, so we can bail out now.
-	 */
-	if (start >= end)
-		goto out;
 
 	index = start;
-	for ( ; ; ) {
+	while (index < end) {
 		cond_resched();
 		if (!find_get_entries(mapping, index, end - 1, &fbatch,
 				indices)) {
-- 
2.33.0

[PATCH 47/48] XArray: Add xas_advance()

[Matthew Wilcox (Oracle)]

Add a new helper function to help iterate over multi-index entries.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 include/linux/xarray.h | 18 ++++++++++++++++++
 lib/xarray.c           |  6 +++---
 2 files changed, 21 insertions(+), 3 deletions(-)

diff --git a/include/linux/xarray.h b/include/linux/xarray.h
index a91e3d90df8a..d6d5da6ed735 100644
--- a/include/linux/xarray.h
+++ b/include/linux/xarray.h
@@ -1580,6 +1580,24 @@ static inline void xas_set(struct xa_state *xas, unsigned long index)
 	xas->xa_node = XAS_RESTART;
 }
 
+/**
+ * xas_advance() - Skip over sibling entries.
+ * @xas: XArray operation state.
+ * @index: Index of last sibling entry.
+ *
+ * Move the operation state to refer to the last sibling entry.
+ * This is useful for loops that normally want to see sibling
+ * entries but sometimes want to skip them.  Use xas_set() if you
+ * want to move to an index which is not part of this entry.
+ */
+static inline void xas_advance(struct xa_state *xas, unsigned long index)
+{
+	unsigned char shift = xas_is_node(xas) ? xas->xa_node->shift : 0;
+
+	xas->xa_index = index;
+	xas->xa_offset = (index >> shift) & XA_CHUNK_MASK;
+}
+
 /**
  * xas_set_order() - Set up XArray operation state for a multislot entry.
  * @xas: XArray operation state.
diff --git a/lib/xarray.c b/lib/xarray.c
index f5d8f54907b4..6f47f6375808 100644
--- a/lib/xarray.c
+++ b/lib/xarray.c
@@ -157,7 +157,7 @@ static void xas_move_index(struct xa_state *xas, unsigned long offset)
 	xas->xa_index += offset << shift;
 }
 
-static void xas_advance(struct xa_state *xas)
+static void xas_next_offset(struct xa_state *xas)
 {
 	xas->xa_offset++;
 	xas_move_index(xas, xas->xa_offset);
@@ -1250,7 +1250,7 @@ void *xas_find(struct xa_state *xas, unsigned long max)
 		xas->xa_offset = ((xas->xa_index - 1) & XA_CHUNK_MASK) + 1;
 	}
 
-	xas_advance(xas);
+	xas_next_offset(xas);
 
 	while (xas->xa_node && (xas->xa_index <= max)) {
 		if (unlikely(xas->xa_offset == XA_CHUNK_SIZE)) {
@@ -1268,7 +1268,7 @@ void *xas_find(struct xa_state *xas, unsigned long max)
 		if (entry && !xa_is_sibling(entry))
 			return entry;
 
-		xas_advance(xas);
+		xas_next_offset(xas);
 	}
 
 	if (!xas->xa_node)
-- 
2.33.0

[PATCH 48/48] mm: Use multi-index entries in the page cache

[Matthew Wilcox (Oracle)]

We currently store large folios as 2^N consecutive entries.  While this
consumes rather more memory than necessary, it also turns out to be buggy.
A writeback operation which starts within a tail page of a dirty folio will
not write back the folio as the xarray's dirty bit is only set on the
head index.  With multi-index entries, the dirty bit will be found no
matter where in the folio the operation starts.

This does end up simplifying the page cache slightly, although not as
much as I had hoped.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 include/linux/pagemap.h | 10 -------
 mm/filemap.c            | 61 ++++++++++++++++++++++++++---------------
 mm/huge_memory.c        | 20 +++++++++++---
 mm/khugepaged.c         | 12 +++++++-
 mm/migrate.c            |  8 ------
 mm/shmem.c              | 16 ++++-------
 6 files changed, 72 insertions(+), 55 deletions(-)

diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
index 6e038811f4c8..704cb1b4b15d 100644
--- a/include/linux/pagemap.h
+++ b/include/linux/pagemap.h
@@ -1125,16 +1125,6 @@ static inline unsigned int __readahead_batch(struct readahead_control *rac,
 		VM_BUG_ON_PAGE(PageTail(page), page);
 		array[i++] = page;
 		rac->_batch_count += thp_nr_pages(page);
-
-		/*
-		 * The page cache isn't using multi-index entries yet,
-		 * so the xas cursor needs to be manually moved to the
-		 * next index.  This can be removed once the page cache
-		 * is converted.
-		 */
-		if (PageHead(page))
-			xas_set(&xas, rac->_index + rac->_batch_count);
-
 		if (i == array_sz)
 			break;
 	}
diff --git a/mm/filemap.c b/mm/filemap.c
index 89a10624e361..9b5b2d962c37 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -135,7 +135,6 @@ static void page_cache_delete(struct address_space *mapping,
 	}
 
 	VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio);
-	VM_BUG_ON_FOLIO(nr != 1 && shadow, folio);
 
 	xas_store(&xas, shadow);
 	xas_init_marks(&xas);
@@ -286,7 +285,7 @@ static void page_cache_delete_batch(struct address_space *mapping,
 			     struct folio_batch *fbatch)
 {
 	XA_STATE(xas, &mapping->i_pages, fbatch->folios[0]->index);
-	int total_pages = 0;
+	long total_pages = 0;
 	int i = 0;
 	struct folio *folio;
 
@@ -313,18 +312,12 @@ static void page_cache_delete_batch(struct address_space *mapping,
 
 		WARN_ON_ONCE(!folio_test_locked(folio));
 
-		if (folio->index == xas.xa_index)
-			folio->mapping = NULL;
+		folio->mapping = NULL;
 		/* Leave folio->index set: truncation lookup relies on it */
 
-		/*
-		 * Move to the next folio in the batch if this is a regular
-		 * folio or the index is of the last sub-page of this folio.
-		 */
-		if (folio->index + folio_nr_pages(folio) - 1 == xas.xa_index)
-			i++;
+		i++;
 		xas_store(&xas, NULL);
-		total_pages++;
+		total_pages += folio_nr_pages(folio);
 	}
 	mapping->nrpages -= total_pages;
 }
@@ -2089,24 +2082,27 @@ unsigned find_lock_entries(struct address_space *mapping, pgoff_t start,
 		indices[fbatch->nr] = xas.xa_index;
 		if (!folio_batch_add(fbatch, folio))
 			break;
-		goto next;
+		continue;
 unlock:
 		folio_unlock(folio);
 put:
 		folio_put(folio);
-next:
-		if (!xa_is_value(folio) && folio_test_large(folio)) {
-			xas_set(&xas, folio->index + folio_nr_pages(folio));
-			/* Did we wrap on 32-bit? */
-			if (!xas.xa_index)
-				break;
-		}
 	}
 	rcu_read_unlock();
 
 	return folio_batch_count(fbatch);
 }
 
+static inline
+bool folio_more_pages(struct folio *folio, pgoff_t index, pgoff_t max)
+{
+	if (!folio_test_large(folio) || folio_test_hugetlb(folio))
+		return false;
+	if (index >= max)
+		return false;
+	return index < folio->index + folio_nr_pages(folio) - 1;
+}
+
 /**
  * find_get_pages_range - gang pagecache lookup
  * @mapping:	The address_space to search
@@ -2145,11 +2141,17 @@ unsigned find_get_pages_range(struct address_space *mapping, pgoff_t *start,
 		if (xa_is_value(folio))
 			continue;
 
+again:
 		pages[ret] = folio_file_page(folio, xas.xa_index);
 		if (++ret == nr_pages) {
 			*start = xas.xa_index + 1;
 			goto out;
 		}
+		if (folio_more_pages(folio, xas.xa_index, end)) {
+			xas.xa_index++;
+			folio_ref_inc(folio);
+			goto again;
+		}
 	}
 
 	/*
@@ -2207,9 +2209,15 @@ unsigned find_get_pages_contig(struct address_space *mapping, pgoff_t index,
 		if (unlikely(folio != xas_reload(&xas)))
 			goto put_page;
 
-		pages[ret] = &folio->page;
+again:
+		pages[ret] = folio_file_page(folio, xas.xa_index);
 		if (++ret == nr_pages)
 			break;
+		if (folio_more_pages(folio, xas.xa_index, ULONG_MAX)) {
+			xas.xa_index++;
+			folio_ref_inc(folio);
+			goto again;
+		}
 		continue;
 put_page:
 		folio_put(folio);
@@ -2334,8 +2342,7 @@ static void filemap_get_read_batch(struct address_space *mapping,
 			break;
 		if (folio_test_readahead(folio))
 			break;
-		xas.xa_index = folio->index + folio_nr_pages(folio) - 1;
-		xas.xa_offset = (xas.xa_index >> xas.xa_shift) & XA_CHUNK_MASK;
+		xas_advance(&xas, folio->index + folio_nr_pages(folio) - 1);
 		continue;
 put_folio:
 		folio_put(folio);
@@ -3284,6 +3291,7 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf,
 	addr = vma->vm_start + ((start_pgoff - vma->vm_pgoff) << PAGE_SHIFT);
 	vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, addr, &vmf->ptl);
 	do {
+again:
 		page = folio_file_page(folio, xas.xa_index);
 		if (PageHWPoison(page))
 			goto unlock;
@@ -3305,9 +3313,18 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf,
 		do_set_pte(vmf, page, addr);
 		/* no need to invalidate: a not-present page won't be cached */
 		update_mmu_cache(vma, addr, vmf->pte);
+		if (folio_more_pages(folio, xas.xa_index, end_pgoff)) {
+			xas.xa_index++;
+			folio_ref_inc(folio);
+			goto again;
+		}
 		folio_unlock(folio);
 		continue;
 unlock:
+		if (folio_more_pages(folio, xas.xa_index, end_pgoff)) {
+			xas.xa_index++;
+			goto again;
+		}
 		folio_unlock(folio);
 		folio_put(folio);
 	} while ((folio = next_map_page(mapping, &xas, end_pgoff)) != NULL);
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index e5483347291c..e57af3a0af43 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -2614,6 +2614,7 @@ int split_huge_page_to_list(struct page *page, struct list_head *list)
 {
 	struct page *head = compound_head(page);
 	struct deferred_split *ds_queue = get_deferred_split_queue(head);
+	XA_STATE(xas, &head->mapping->i_pages, head->index);
 	struct anon_vma *anon_vma = NULL;
 	struct address_space *mapping = NULL;
 	int extra_pins, ret;
@@ -2678,16 +2679,24 @@ int split_huge_page_to_list(struct page *page, struct list_head *list)
 
 	unmap_page(head);
 
+	if (mapping) {
+		xas_split_alloc(&xas, head, compound_order(head),
+				mapping_gfp_mask(mapping) & GFP_RECLAIM_MASK);
+		if (xas_error(&xas)) {
+			ret = xas_error(&xas);
+			goto out_unlock;
+		}
+	}
+
 	/* block interrupt reentry in xa_lock and spinlock */
 	local_irq_disable();
 	if (mapping) {
-		XA_STATE(xas, &mapping->i_pages, page_index(head));
-
 		/*
 		 * Check if the head page is present in page cache.
 		 * We assume all tail are present too, if head is there.
 		 */
-		xa_lock(&mapping->i_pages);
+		xas_lock(&xas);
+		xas_reset(&xas);
 		if (xas_load(&xas) != head)
 			goto fail;
 	}
@@ -2703,6 +2712,7 @@ int split_huge_page_to_list(struct page *page, struct list_head *list)
 		if (mapping) {
 			int nr = thp_nr_pages(head);
 
+			xas_split(&xas, head, thp_order(head));
 			if (PageSwapBacked(head)) {
 				__mod_lruvec_page_state(head, NR_SHMEM_THPS,
 							-nr);
@@ -2719,7 +2729,7 @@ int split_huge_page_to_list(struct page *page, struct list_head *list)
 		spin_unlock(&ds_queue->split_queue_lock);
 fail:
 		if (mapping)
-			xa_unlock(&mapping->i_pages);
+			xas_unlock(&xas);
 		local_irq_enable();
 		remap_page(head, thp_nr_pages(head));
 		ret = -EBUSY;
@@ -2733,6 +2743,8 @@ int split_huge_page_to_list(struct page *page, struct list_head *list)
 	if (mapping)
 		i_mmap_unlock_read(mapping);
 out:
+	/* Free any memory we didn't use */
+	xas_nomem(&xas, 0);
 	count_vm_event(!ret ? THP_SPLIT_PAGE : THP_SPLIT_PAGE_FAILED);
 	return ret;
 }
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index e99101162f1a..2e1911cc3466 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -1667,7 +1667,10 @@ static void collapse_file(struct mm_struct *mm,
 	}
 	count_memcg_page_event(new_page, THP_COLLAPSE_ALLOC);
 
-	/* This will be less messy when we use multi-index entries */
+	/*
+	 * Ensure we have slots for all the pages in the range.  This is
+	 * almost certainly a no-op because most of the pages must be present
+	 */
 	do {
 		xas_lock_irq(&xas);
 		xas_create_range(&xas);
@@ -1892,6 +1895,9 @@ static void collapse_file(struct mm_struct *mm,
 			__mod_lruvec_page_state(new_page, NR_SHMEM, nr_none);
 	}
 
+	/* Join all the small entries into a single multi-index entry */
+	xas_set_order(&xas, start, HPAGE_PMD_ORDER);
+	xas_store(&xas, new_page);
 xa_locked:
 	xas_unlock_irq(&xas);
 xa_unlocked:
@@ -2013,6 +2019,10 @@ static void khugepaged_scan_file(struct mm_struct *mm,
 			continue;
 		}
 
+		/*
+		 * XXX: khugepaged should compact smaller compound pages
+		 * into a PMD sized page
+		 */
 		if (PageTransCompound(page)) {
 			result = SCAN_PAGE_COMPOUND;
 			break;
diff --git a/mm/migrate.c b/mm/migrate.c
index 311638177536..7079e6b7dbe7 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -433,14 +433,6 @@ int folio_migrate_mapping(struct address_space *mapping,
 	}
 
 	xas_store(&xas, newfolio);
-	if (nr > 1) {
-		int i;
-
-		for (i = 1; i < nr; i++) {
-			xas_next(&xas);
-			xas_store(&xas, newfolio);
-		}
-	}
 
 	/*
 	 * Drop cache reference from old page by unfreezing
diff --git a/mm/shmem.c b/mm/shmem.c
index 7f0b07845c1f..4f80cf4c74d3 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -707,7 +707,6 @@ static int shmem_add_to_page_cache(struct page *page,
 				   struct mm_struct *charge_mm)
 {
 	XA_STATE_ORDER(xas, &mapping->i_pages, index, compound_order(page));
-	unsigned long i = 0;
 	unsigned long nr = compound_nr(page);
 	int error;
 
@@ -736,18 +735,15 @@ static int shmem_add_to_page_cache(struct page *page,
 	do {
 		void *entry;
 		xas_lock_irq(&xas);
-		entry = xas_find_conflict(&xas);
-		if (entry != expected)
+		while ((entry = xas_find_conflict(&xas)) != NULL) {
+			if (entry == expected)
+				continue;
 			xas_set_err(&xas, -EEXIST);
-		xas_create_range(&xas);
-		if (xas_error(&xas))
 			goto unlock;
-next:
-		xas_store(&xas, page);
-		if (++i < nr) {
-			xas_next(&xas);
-			goto next;
 		}
+		xas_store(&xas, page);
+		if (xas_error(&xas))
+			goto unlock;
 		if (PageTransHuge(page)) {
 			count_vm_event(THP_FILE_ALLOC);
 			__mod_lruvec_page_state(page, NR_SHMEM_THPS, nr);
-- 
2.33.0

Re: [PATCH 42/48] mm: Convert find_lock_entries() to use a folio_batch

[kernel test robot]

Hi "Matthew,

Thank you for the patch! Perhaps something to improve:

[auto build test WARNING on hnaz-mm/master]
[also build test WARNING on rostedt-trace/for-next linus/master v5.16-rc4]
[cannot apply to next-20211208]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]

url:    https://github.com/0day-ci/linux/commits/Matthew-Wilcox-Oracle/Folios-for-5-17/20211208-122734
base:   https://github.com/hnaz/linux-mm master
config: alpha-allyesconfig (https://download.01.org/0day-ci/archive/20211208/202112081952.NHF8MX2L-lkp@intel.com/config)
compiler: alpha-linux-gcc (GCC) 11.2.0
reproduce (this is a W=1 build):
        wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
        chmod +x ~/bin/make.cross
        # https://github.com/0day-ci/linux/commit/b883ee2b43293c901ea31f233d1596f255e0dcb9
        git remote add linux-review https://github.com/0day-ci/linux
        git fetch --no-tags linux-review Matthew-Wilcox-Oracle/Folios-for-5-17/20211208-122734
        git checkout b883ee2b43293c901ea31f233d1596f255e0dcb9
        # save the config file to linux build tree
        mkdir build_dir
        COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-11.2.0 make.cross O=build_dir ARCH=alpha SHELL=/bin/bash fs/

If you fix the issue, kindly add following tag as appropriate
Reported-by: kernel test robot <lkp@intel.com>

All warnings (new ones prefixed by >>):

   In file included from fs/f2fs/dir.c:13:
>> fs/f2fs/f2fs.h:4055:67: warning: 'struct pagevec' declared inside parameter list will not be visible outside of this definition or declaration
    4055 | bool f2fs_all_cluster_page_loaded(struct compress_ctx *cc, struct pagevec *pvec,
         |                                                                   ^~~~~~~


vim +4055 fs/f2fs/f2fs.h

4c8ff7095bef64 Chao Yu       2019-11-01  4034  
4c8ff7095bef64 Chao Yu       2019-11-01  4035  /*
4c8ff7095bef64 Chao Yu       2019-11-01  4036   * compress.c
4c8ff7095bef64 Chao Yu       2019-11-01  4037   */
4c8ff7095bef64 Chao Yu       2019-11-01  4038  #ifdef CONFIG_F2FS_FS_COMPRESSION
4c8ff7095bef64 Chao Yu       2019-11-01  4039  bool f2fs_is_compressed_page(struct page *page);
4c8ff7095bef64 Chao Yu       2019-11-01  4040  struct page *f2fs_compress_control_page(struct page *page);
4c8ff7095bef64 Chao Yu       2019-11-01  4041  int f2fs_prepare_compress_overwrite(struct inode *inode,
4c8ff7095bef64 Chao Yu       2019-11-01  4042  			struct page **pagep, pgoff_t index, void **fsdata);
4c8ff7095bef64 Chao Yu       2019-11-01  4043  bool f2fs_compress_write_end(struct inode *inode, void *fsdata,
4c8ff7095bef64 Chao Yu       2019-11-01  4044  					pgoff_t index, unsigned copied);
3265d3db1f1639 Chao Yu       2020-03-18  4045  int f2fs_truncate_partial_cluster(struct inode *inode, u64 from, bool lock);
4c8ff7095bef64 Chao Yu       2019-11-01  4046  void f2fs_compress_write_end_io(struct bio *bio, struct page *page);
4c8ff7095bef64 Chao Yu       2019-11-01  4047  bool f2fs_is_compress_backend_ready(struct inode *inode);
5e6bbde9598230 Chao Yu       2020-04-08  4048  int f2fs_init_compress_mempool(void);
5e6bbde9598230 Chao Yu       2020-04-08  4049  void f2fs_destroy_compress_mempool(void);
6ce19aff0b8cd3 Chao Yu       2021-05-20  4050  void f2fs_decompress_cluster(struct decompress_io_ctx *dic);
6ce19aff0b8cd3 Chao Yu       2021-05-20  4051  void f2fs_end_read_compressed_page(struct page *page, bool failed,
6ce19aff0b8cd3 Chao Yu       2021-05-20  4052  							block_t blkaddr);
4c8ff7095bef64 Chao Yu       2019-11-01  4053  bool f2fs_cluster_is_empty(struct compress_ctx *cc);
4c8ff7095bef64 Chao Yu       2019-11-01  4054  bool f2fs_cluster_can_merge_page(struct compress_ctx *cc, pgoff_t index);
2ce5eeadf5d8d9 Andrew Morton 2021-10-28 @4055  bool f2fs_all_cluster_page_loaded(struct compress_ctx *cc, struct pagevec *pvec,
2ce5eeadf5d8d9 Andrew Morton 2021-10-28  4056  				int index, int nr_pages);
bbe1da7e34ac5a Chao Yu       2021-08-06  4057  bool f2fs_sanity_check_cluster(struct dnode_of_data *dn);
4c8ff7095bef64 Chao Yu       2019-11-01  4058  void f2fs_compress_ctx_add_page(struct compress_ctx *cc, struct page *page);
4c8ff7095bef64 Chao Yu       2019-11-01  4059  int f2fs_write_multi_pages(struct compress_ctx *cc,
4c8ff7095bef64 Chao Yu       2019-11-01  4060  						int *submitted,
4c8ff7095bef64 Chao Yu       2019-11-01  4061  						struct writeback_control *wbc,
4c8ff7095bef64 Chao Yu       2019-11-01  4062  						enum iostat_type io_type);
4c8ff7095bef64 Chao Yu       2019-11-01  4063  int f2fs_is_compressed_cluster(struct inode *inode, pgoff_t index);
94afd6d6e52531 Chao Yu       2021-08-04  4064  void f2fs_update_extent_tree_range_compressed(struct inode *inode,
94afd6d6e52531 Chao Yu       2021-08-04  4065  				pgoff_t fofs, block_t blkaddr, unsigned int llen,
94afd6d6e52531 Chao Yu       2021-08-04  4066  				unsigned int c_len);
4c8ff7095bef64 Chao Yu       2019-11-01  4067  int f2fs_read_multi_pages(struct compress_ctx *cc, struct bio **bio_ret,
4c8ff7095bef64 Chao Yu       2019-11-01  4068  				unsigned nr_pages, sector_t *last_block_in_bio,
0683728adab251 Chao Yu       2020-02-18  4069  				bool is_readahead, bool for_write);
4c8ff7095bef64 Chao Yu       2019-11-01  4070  struct decompress_io_ctx *f2fs_alloc_dic(struct compress_ctx *cc);
7f59b277f79e8a Eric Biggers  2021-01-04  4071  void f2fs_decompress_end_io(struct decompress_io_ctx *dic, bool failed);
7f59b277f79e8a Eric Biggers  2021-01-04  4072  void f2fs_put_page_dic(struct page *page);
94afd6d6e52531 Chao Yu       2021-08-04  4073  unsigned int f2fs_cluster_blocks_are_contiguous(struct dnode_of_data *dn);
4c8ff7095bef64 Chao Yu       2019-11-01  4074  int f2fs_init_compress_ctx(struct compress_ctx *cc);
8bfbfb0ddd706b Chao Yu       2021-05-10  4075  void f2fs_destroy_compress_ctx(struct compress_ctx *cc, bool reuse);
4c8ff7095bef64 Chao Yu       2019-11-01  4076  void f2fs_init_compress_info(struct f2fs_sb_info *sbi);
6ce19aff0b8cd3 Chao Yu       2021-05-20  4077  int f2fs_init_compress_inode(struct f2fs_sb_info *sbi);
6ce19aff0b8cd3 Chao Yu       2021-05-20  4078  void f2fs_destroy_compress_inode(struct f2fs_sb_info *sbi);
31083031709eea Chao Yu       2020-09-14  4079  int f2fs_init_page_array_cache(struct f2fs_sb_info *sbi);
31083031709eea Chao Yu       2020-09-14  4080  void f2fs_destroy_page_array_cache(struct f2fs_sb_info *sbi);
c68d6c88302250 Chao Yu       2020-09-14  4081  int __init f2fs_init_compress_cache(void);
c68d6c88302250 Chao Yu       2020-09-14  4082  void f2fs_destroy_compress_cache(void);
6ce19aff0b8cd3 Chao Yu       2021-05-20  4083  struct address_space *COMPRESS_MAPPING(struct f2fs_sb_info *sbi);
6ce19aff0b8cd3 Chao Yu       2021-05-20  4084  void f2fs_invalidate_compress_page(struct f2fs_sb_info *sbi, block_t blkaddr);
6ce19aff0b8cd3 Chao Yu       2021-05-20  4085  void f2fs_cache_compressed_page(struct f2fs_sb_info *sbi, struct page *page,
6ce19aff0b8cd3 Chao Yu       2021-05-20  4086  						nid_t ino, block_t blkaddr);
6ce19aff0b8cd3 Chao Yu       2021-05-20  4087  bool f2fs_load_compressed_page(struct f2fs_sb_info *sbi, struct page *page,
6ce19aff0b8cd3 Chao Yu       2021-05-20  4088  								block_t blkaddr);
6ce19aff0b8cd3 Chao Yu       2021-05-20  4089  void f2fs_invalidate_compress_pages(struct f2fs_sb_info *sbi, nid_t ino);
5ac443e26a0964 Daeho Jeong   2021-03-15  4090  #define inc_compr_inode_stat(inode)					\
5ac443e26a0964 Daeho Jeong   2021-03-15  4091  	do {								\
5ac443e26a0964 Daeho Jeong   2021-03-15  4092  		struct f2fs_sb_info *sbi = F2FS_I_SB(inode);		\
5ac443e26a0964 Daeho Jeong   2021-03-15  4093  		sbi->compr_new_inode++;					\
5ac443e26a0964 Daeho Jeong   2021-03-15  4094  	} while (0)
5ac443e26a0964 Daeho Jeong   2021-03-15  4095  #define add_compr_block_stat(inode, blocks)				\
5ac443e26a0964 Daeho Jeong   2021-03-15  4096  	do {								\
5ac443e26a0964 Daeho Jeong   2021-03-15  4097  		struct f2fs_sb_info *sbi = F2FS_I_SB(inode);		\
5ac443e26a0964 Daeho Jeong   2021-03-15  4098  		int diff = F2FS_I(inode)->i_cluster_size - blocks;	\
5ac443e26a0964 Daeho Jeong   2021-03-15  4099  		sbi->compr_written_block += blocks;			\
5ac443e26a0964 Daeho Jeong   2021-03-15  4100  		sbi->compr_saved_block += diff;				\
5ac443e26a0964 Daeho Jeong   2021-03-15  4101  	} while (0)
4c8ff7095bef64 Chao Yu       2019-11-01  4102  #else
4c8ff7095bef64 Chao Yu       2019-11-01  4103  static inline bool f2fs_is_compressed_page(struct page *page) { return false; }
4c8ff7095bef64 Chao Yu       2019-11-01  4104  static inline bool f2fs_is_compress_backend_ready(struct inode *inode)
4c8ff7095bef64 Chao Yu       2019-11-01  4105  {
4c8ff7095bef64 Chao Yu       2019-11-01  4106  	if (!f2fs_compressed_file(inode))
4c8ff7095bef64 Chao Yu       2019-11-01  4107  		return true;
4c8ff7095bef64 Chao Yu       2019-11-01  4108  	/* not support compression */
4c8ff7095bef64 Chao Yu       2019-11-01  4109  	return false;
4c8ff7095bef64 Chao Yu       2019-11-01  4110  }
4c8ff7095bef64 Chao Yu       2019-11-01  4111  static inline struct page *f2fs_compress_control_page(struct page *page)
4c8ff7095bef64 Chao Yu       2019-11-01  4112  {
4c8ff7095bef64 Chao Yu       2019-11-01  4113  	WARN_ON_ONCE(1);
4c8ff7095bef64 Chao Yu       2019-11-01  4114  	return ERR_PTR(-EINVAL);
4c8ff7095bef64 Chao Yu       2019-11-01  4115  }
5e6bbde9598230 Chao Yu       2020-04-08  4116  static inline int f2fs_init_compress_mempool(void) { return 0; }
5e6bbde9598230 Chao Yu       2020-04-08  4117  static inline void f2fs_destroy_compress_mempool(void) { }
6ce19aff0b8cd3 Chao Yu       2021-05-20  4118  static inline void f2fs_decompress_cluster(struct decompress_io_ctx *dic) { }
6ce19aff0b8cd3 Chao Yu       2021-05-20  4119  static inline void f2fs_end_read_compressed_page(struct page *page,
6ce19aff0b8cd3 Chao Yu       2021-05-20  4120  						bool failed, block_t blkaddr)
7f59b277f79e8a Eric Biggers  2021-01-04  4121  {
7f59b277f79e8a Eric Biggers  2021-01-04  4122  	WARN_ON_ONCE(1);
7f59b277f79e8a Eric Biggers  2021-01-04  4123  }
7f59b277f79e8a Eric Biggers  2021-01-04  4124  static inline void f2fs_put_page_dic(struct page *page)
7f59b277f79e8a Eric Biggers  2021-01-04  4125  {
7f59b277f79e8a Eric Biggers  2021-01-04  4126  	WARN_ON_ONCE(1);
7f59b277f79e8a Eric Biggers  2021-01-04  4127  }
94afd6d6e52531 Chao Yu       2021-08-04  4128  static inline unsigned int f2fs_cluster_blocks_are_contiguous(struct dnode_of_data *dn) { return 0; }
bbe1da7e34ac5a Chao Yu       2021-08-06  4129  static inline bool f2fs_sanity_check_cluster(struct dnode_of_data *dn) { return false; }
6ce19aff0b8cd3 Chao Yu       2021-05-20  4130  static inline int f2fs_init_compress_inode(struct f2fs_sb_info *sbi) { return 0; }
6ce19aff0b8cd3 Chao Yu       2021-05-20  4131  static inline void f2fs_destroy_compress_inode(struct f2fs_sb_info *sbi) { }
31083031709eea Chao Yu       2020-09-14  4132  static inline int f2fs_init_page_array_cache(struct f2fs_sb_info *sbi) { return 0; }
31083031709eea Chao Yu       2020-09-14  4133  static inline void f2fs_destroy_page_array_cache(struct f2fs_sb_info *sbi) { }
c68d6c88302250 Chao Yu       2020-09-14  4134  static inline int __init f2fs_init_compress_cache(void) { return 0; }
c68d6c88302250 Chao Yu       2020-09-14  4135  static inline void f2fs_destroy_compress_cache(void) { }
6ce19aff0b8cd3 Chao Yu       2021-05-20  4136  static inline void f2fs_invalidate_compress_page(struct f2fs_sb_info *sbi,
6ce19aff0b8cd3 Chao Yu       2021-05-20  4137  				block_t blkaddr) { }
6ce19aff0b8cd3 Chao Yu       2021-05-20  4138  static inline void f2fs_cache_compressed_page(struct f2fs_sb_info *sbi,
6ce19aff0b8cd3 Chao Yu       2021-05-20  4139  				struct page *page, nid_t ino, block_t blkaddr) { }
6ce19aff0b8cd3 Chao Yu       2021-05-20  4140  static inline bool f2fs_load_compressed_page(struct f2fs_sb_info *sbi,
6ce19aff0b8cd3 Chao Yu       2021-05-20  4141  				struct page *page, block_t blkaddr) { return false; }
6ce19aff0b8cd3 Chao Yu       2021-05-20  4142  static inline void f2fs_invalidate_compress_pages(struct f2fs_sb_info *sbi,
6ce19aff0b8cd3 Chao Yu       2021-05-20  4143  							nid_t ino) { }
5ac443e26a0964 Daeho Jeong   2021-03-15  4144  #define inc_compr_inode_stat(inode)		do { } while (0)
94afd6d6e52531 Chao Yu       2021-08-04  4145  static inline void f2fs_update_extent_tree_range_compressed(struct inode *inode,
94afd6d6e52531 Chao Yu       2021-08-04  4146  				pgoff_t fofs, block_t blkaddr, unsigned int llen,
94afd6d6e52531 Chao Yu       2021-08-04  4147  				unsigned int c_len) { }
4c8ff7095bef64 Chao Yu       2019-11-01  4148  #endif
4c8ff7095bef64 Chao Yu       2019-11-01  4149  

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org

Re: [PATCH 42/48] mm: Convert find_lock_entries() to use a folio_batch

[kernel test robot]

[-- Attachment #1: Type: text/plain, Size: 13092 bytes --]

Hi "Matthew,

Thank you for the patch! Perhaps something to improve:

[auto build test WARNING on hnaz-mm/master]
[also build test WARNING on rostedt-trace/for-next linus/master v5.16-rc4]
[cannot apply to next-20211208]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]

url:    https://github.com/0day-ci/linux/commits/Matthew-Wilcox-Oracle/Folios-for-5-17/20211208-122734
base:   https://github.com/hnaz/linux-mm master
config: alpha-allyesconfig (https://download.01.org/0day-ci/archive/20211208/202112081952.NHF8MX2L-lkp(a)intel.com/config)
compiler: alpha-linux-gcc (GCC) 11.2.0
reproduce (this is a W=1 build):
        wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
        chmod +x ~/bin/make.cross
        # https://github.com/0day-ci/linux/commit/b883ee2b43293c901ea31f233d1596f255e0dcb9
        git remote add linux-review https://github.com/0day-ci/linux
        git fetch --no-tags linux-review Matthew-Wilcox-Oracle/Folios-for-5-17/20211208-122734
        git checkout b883ee2b43293c901ea31f233d1596f255e0dcb9
        # save the config file to linux build tree
        mkdir build_dir
        COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-11.2.0 make.cross O=build_dir ARCH=alpha SHELL=/bin/bash fs/

If you fix the issue, kindly add following tag as appropriate
Reported-by: kernel test robot <lkp@intel.com>

All warnings (new ones prefixed by >>):

   In file included from fs/f2fs/dir.c:13:
>> fs/f2fs/f2fs.h:4055:67: warning: 'struct pagevec' declared inside parameter list will not be visible outside of this definition or declaration
    4055 | bool f2fs_all_cluster_page_loaded(struct compress_ctx *cc, struct pagevec *pvec,
         |                                                                   ^~~~~~~


vim +4055 fs/f2fs/f2fs.h

4c8ff7095bef64 Chao Yu       2019-11-01  4034  
4c8ff7095bef64 Chao Yu       2019-11-01  4035  /*
4c8ff7095bef64 Chao Yu       2019-11-01  4036   * compress.c
4c8ff7095bef64 Chao Yu       2019-11-01  4037   */
4c8ff7095bef64 Chao Yu       2019-11-01  4038  #ifdef CONFIG_F2FS_FS_COMPRESSION
4c8ff7095bef64 Chao Yu       2019-11-01  4039  bool f2fs_is_compressed_page(struct page *page);
4c8ff7095bef64 Chao Yu       2019-11-01  4040  struct page *f2fs_compress_control_page(struct page *page);
4c8ff7095bef64 Chao Yu       2019-11-01  4041  int f2fs_prepare_compress_overwrite(struct inode *inode,
4c8ff7095bef64 Chao Yu       2019-11-01  4042  			struct page **pagep, pgoff_t index, void **fsdata);
4c8ff7095bef64 Chao Yu       2019-11-01  4043  bool f2fs_compress_write_end(struct inode *inode, void *fsdata,
4c8ff7095bef64 Chao Yu       2019-11-01  4044  					pgoff_t index, unsigned copied);
3265d3db1f1639 Chao Yu       2020-03-18  4045  int f2fs_truncate_partial_cluster(struct inode *inode, u64 from, bool lock);
4c8ff7095bef64 Chao Yu       2019-11-01  4046  void f2fs_compress_write_end_io(struct bio *bio, struct page *page);
4c8ff7095bef64 Chao Yu       2019-11-01  4047  bool f2fs_is_compress_backend_ready(struct inode *inode);
5e6bbde9598230 Chao Yu       2020-04-08  4048  int f2fs_init_compress_mempool(void);
5e6bbde9598230 Chao Yu       2020-04-08  4049  void f2fs_destroy_compress_mempool(void);
6ce19aff0b8cd3 Chao Yu       2021-05-20  4050  void f2fs_decompress_cluster(struct decompress_io_ctx *dic);
6ce19aff0b8cd3 Chao Yu       2021-05-20  4051  void f2fs_end_read_compressed_page(struct page *page, bool failed,
6ce19aff0b8cd3 Chao Yu       2021-05-20  4052  							block_t blkaddr);
4c8ff7095bef64 Chao Yu       2019-11-01  4053  bool f2fs_cluster_is_empty(struct compress_ctx *cc);
4c8ff7095bef64 Chao Yu       2019-11-01  4054  bool f2fs_cluster_can_merge_page(struct compress_ctx *cc, pgoff_t index);
2ce5eeadf5d8d9 Andrew Morton 2021-10-28 @4055  bool f2fs_all_cluster_page_loaded(struct compress_ctx *cc, struct pagevec *pvec,
2ce5eeadf5d8d9 Andrew Morton 2021-10-28  4056  				int index, int nr_pages);
bbe1da7e34ac5a Chao Yu       2021-08-06  4057  bool f2fs_sanity_check_cluster(struct dnode_of_data *dn);
4c8ff7095bef64 Chao Yu       2019-11-01  4058  void f2fs_compress_ctx_add_page(struct compress_ctx *cc, struct page *page);
4c8ff7095bef64 Chao Yu       2019-11-01  4059  int f2fs_write_multi_pages(struct compress_ctx *cc,
4c8ff7095bef64 Chao Yu       2019-11-01  4060  						int *submitted,
4c8ff7095bef64 Chao Yu       2019-11-01  4061  						struct writeback_control *wbc,
4c8ff7095bef64 Chao Yu       2019-11-01  4062  						enum iostat_type io_type);
4c8ff7095bef64 Chao Yu       2019-11-01  4063  int f2fs_is_compressed_cluster(struct inode *inode, pgoff_t index);
94afd6d6e52531 Chao Yu       2021-08-04  4064  void f2fs_update_extent_tree_range_compressed(struct inode *inode,
94afd6d6e52531 Chao Yu       2021-08-04  4065  				pgoff_t fofs, block_t blkaddr, unsigned int llen,
94afd6d6e52531 Chao Yu       2021-08-04  4066  				unsigned int c_len);
4c8ff7095bef64 Chao Yu       2019-11-01  4067  int f2fs_read_multi_pages(struct compress_ctx *cc, struct bio **bio_ret,
4c8ff7095bef64 Chao Yu       2019-11-01  4068  				unsigned nr_pages, sector_t *last_block_in_bio,
0683728adab251 Chao Yu       2020-02-18  4069  				bool is_readahead, bool for_write);
4c8ff7095bef64 Chao Yu       2019-11-01  4070  struct decompress_io_ctx *f2fs_alloc_dic(struct compress_ctx *cc);
7f59b277f79e8a Eric Biggers  2021-01-04  4071  void f2fs_decompress_end_io(struct decompress_io_ctx *dic, bool failed);
7f59b277f79e8a Eric Biggers  2021-01-04  4072  void f2fs_put_page_dic(struct page *page);
94afd6d6e52531 Chao Yu       2021-08-04  4073  unsigned int f2fs_cluster_blocks_are_contiguous(struct dnode_of_data *dn);
4c8ff7095bef64 Chao Yu       2019-11-01  4074  int f2fs_init_compress_ctx(struct compress_ctx *cc);
8bfbfb0ddd706b Chao Yu       2021-05-10  4075  void f2fs_destroy_compress_ctx(struct compress_ctx *cc, bool reuse);
4c8ff7095bef64 Chao Yu       2019-11-01  4076  void f2fs_init_compress_info(struct f2fs_sb_info *sbi);
6ce19aff0b8cd3 Chao Yu       2021-05-20  4077  int f2fs_init_compress_inode(struct f2fs_sb_info *sbi);
6ce19aff0b8cd3 Chao Yu       2021-05-20  4078  void f2fs_destroy_compress_inode(struct f2fs_sb_info *sbi);
31083031709eea Chao Yu       2020-09-14  4079  int f2fs_init_page_array_cache(struct f2fs_sb_info *sbi);
31083031709eea Chao Yu       2020-09-14  4080  void f2fs_destroy_page_array_cache(struct f2fs_sb_info *sbi);
c68d6c88302250 Chao Yu       2020-09-14  4081  int __init f2fs_init_compress_cache(void);
c68d6c88302250 Chao Yu       2020-09-14  4082  void f2fs_destroy_compress_cache(void);
6ce19aff0b8cd3 Chao Yu       2021-05-20  4083  struct address_space *COMPRESS_MAPPING(struct f2fs_sb_info *sbi);
6ce19aff0b8cd3 Chao Yu       2021-05-20  4084  void f2fs_invalidate_compress_page(struct f2fs_sb_info *sbi, block_t blkaddr);
6ce19aff0b8cd3 Chao Yu       2021-05-20  4085  void f2fs_cache_compressed_page(struct f2fs_sb_info *sbi, struct page *page,
6ce19aff0b8cd3 Chao Yu       2021-05-20  4086  						nid_t ino, block_t blkaddr);
6ce19aff0b8cd3 Chao Yu       2021-05-20  4087  bool f2fs_load_compressed_page(struct f2fs_sb_info *sbi, struct page *page,
6ce19aff0b8cd3 Chao Yu       2021-05-20  4088  								block_t blkaddr);
6ce19aff0b8cd3 Chao Yu       2021-05-20  4089  void f2fs_invalidate_compress_pages(struct f2fs_sb_info *sbi, nid_t ino);
5ac443e26a0964 Daeho Jeong   2021-03-15  4090  #define inc_compr_inode_stat(inode)					\
5ac443e26a0964 Daeho Jeong   2021-03-15  4091  	do {								\
5ac443e26a0964 Daeho Jeong   2021-03-15  4092  		struct f2fs_sb_info *sbi = F2FS_I_SB(inode);		\
5ac443e26a0964 Daeho Jeong   2021-03-15  4093  		sbi->compr_new_inode++;					\
5ac443e26a0964 Daeho Jeong   2021-03-15  4094  	} while (0)
5ac443e26a0964 Daeho Jeong   2021-03-15  4095  #define add_compr_block_stat(inode, blocks)				\
5ac443e26a0964 Daeho Jeong   2021-03-15  4096  	do {								\
5ac443e26a0964 Daeho Jeong   2021-03-15  4097  		struct f2fs_sb_info *sbi = F2FS_I_SB(inode);		\
5ac443e26a0964 Daeho Jeong   2021-03-15  4098  		int diff = F2FS_I(inode)->i_cluster_size - blocks;	\
5ac443e26a0964 Daeho Jeong   2021-03-15  4099  		sbi->compr_written_block += blocks;			\
5ac443e26a0964 Daeho Jeong   2021-03-15  4100  		sbi->compr_saved_block += diff;				\
5ac443e26a0964 Daeho Jeong   2021-03-15  4101  	} while (0)
4c8ff7095bef64 Chao Yu       2019-11-01  4102  #else
4c8ff7095bef64 Chao Yu       2019-11-01  4103  static inline bool f2fs_is_compressed_page(struct page *page) { return false; }
4c8ff7095bef64 Chao Yu       2019-11-01  4104  static inline bool f2fs_is_compress_backend_ready(struct inode *inode)
4c8ff7095bef64 Chao Yu       2019-11-01  4105  {
4c8ff7095bef64 Chao Yu       2019-11-01  4106  	if (!f2fs_compressed_file(inode))
4c8ff7095bef64 Chao Yu       2019-11-01  4107  		return true;
4c8ff7095bef64 Chao Yu       2019-11-01  4108  	/* not support compression */
4c8ff7095bef64 Chao Yu       2019-11-01  4109  	return false;
4c8ff7095bef64 Chao Yu       2019-11-01  4110  }
4c8ff7095bef64 Chao Yu       2019-11-01  4111  static inline struct page *f2fs_compress_control_page(struct page *page)
4c8ff7095bef64 Chao Yu       2019-11-01  4112  {
4c8ff7095bef64 Chao Yu       2019-11-01  4113  	WARN_ON_ONCE(1);
4c8ff7095bef64 Chao Yu       2019-11-01  4114  	return ERR_PTR(-EINVAL);
4c8ff7095bef64 Chao Yu       2019-11-01  4115  }
5e6bbde9598230 Chao Yu       2020-04-08  4116  static inline int f2fs_init_compress_mempool(void) { return 0; }
5e6bbde9598230 Chao Yu       2020-04-08  4117  static inline void f2fs_destroy_compress_mempool(void) { }
6ce19aff0b8cd3 Chao Yu       2021-05-20  4118  static inline void f2fs_decompress_cluster(struct decompress_io_ctx *dic) { }
6ce19aff0b8cd3 Chao Yu       2021-05-20  4119  static inline void f2fs_end_read_compressed_page(struct page *page,
6ce19aff0b8cd3 Chao Yu       2021-05-20  4120  						bool failed, block_t blkaddr)
7f59b277f79e8a Eric Biggers  2021-01-04  4121  {
7f59b277f79e8a Eric Biggers  2021-01-04  4122  	WARN_ON_ONCE(1);
7f59b277f79e8a Eric Biggers  2021-01-04  4123  }
7f59b277f79e8a Eric Biggers  2021-01-04  4124  static inline void f2fs_put_page_dic(struct page *page)
7f59b277f79e8a Eric Biggers  2021-01-04  4125  {
7f59b277f79e8a Eric Biggers  2021-01-04  4126  	WARN_ON_ONCE(1);
7f59b277f79e8a Eric Biggers  2021-01-04  4127  }
94afd6d6e52531 Chao Yu       2021-08-04  4128  static inline unsigned int f2fs_cluster_blocks_are_contiguous(struct dnode_of_data *dn) { return 0; }
bbe1da7e34ac5a Chao Yu       2021-08-06  4129  static inline bool f2fs_sanity_check_cluster(struct dnode_of_data *dn) { return false; }
6ce19aff0b8cd3 Chao Yu       2021-05-20  4130  static inline int f2fs_init_compress_inode(struct f2fs_sb_info *sbi) { return 0; }
6ce19aff0b8cd3 Chao Yu       2021-05-20  4131  static inline void f2fs_destroy_compress_inode(struct f2fs_sb_info *sbi) { }
31083031709eea Chao Yu       2020-09-14  4132  static inline int f2fs_init_page_array_cache(struct f2fs_sb_info *sbi) { return 0; }
31083031709eea Chao Yu       2020-09-14  4133  static inline void f2fs_destroy_page_array_cache(struct f2fs_sb_info *sbi) { }
c68d6c88302250 Chao Yu       2020-09-14  4134  static inline int __init f2fs_init_compress_cache(void) { return 0; }
c68d6c88302250 Chao Yu       2020-09-14  4135  static inline void f2fs_destroy_compress_cache(void) { }
6ce19aff0b8cd3 Chao Yu       2021-05-20  4136  static inline void f2fs_invalidate_compress_page(struct f2fs_sb_info *sbi,
6ce19aff0b8cd3 Chao Yu       2021-05-20  4137  				block_t blkaddr) { }
6ce19aff0b8cd3 Chao Yu       2021-05-20  4138  static inline void f2fs_cache_compressed_page(struct f2fs_sb_info *sbi,
6ce19aff0b8cd3 Chao Yu       2021-05-20  4139  				struct page *page, nid_t ino, block_t blkaddr) { }
6ce19aff0b8cd3 Chao Yu       2021-05-20  4140  static inline bool f2fs_load_compressed_page(struct f2fs_sb_info *sbi,
6ce19aff0b8cd3 Chao Yu       2021-05-20  4141  				struct page *page, block_t blkaddr) { return false; }
6ce19aff0b8cd3 Chao Yu       2021-05-20  4142  static inline void f2fs_invalidate_compress_pages(struct f2fs_sb_info *sbi,
6ce19aff0b8cd3 Chao Yu       2021-05-20  4143  							nid_t ino) { }
5ac443e26a0964 Daeho Jeong   2021-03-15  4144  #define inc_compr_inode_stat(inode)		do { } while (0)
94afd6d6e52531 Chao Yu       2021-08-04  4145  static inline void f2fs_update_extent_tree_range_compressed(struct inode *inode,
94afd6d6e52531 Chao Yu       2021-08-04  4146  				pgoff_t fofs, block_t blkaddr, unsigned int llen,
94afd6d6e52531 Chao Yu       2021-08-04  4147  				unsigned int c_len) { }
4c8ff7095bef64 Chao Yu       2019-11-01  4148  #endif
4c8ff7095bef64 Chao Yu       2019-11-01  4149  

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org

Re: [PATCH 42/48] mm: Convert find_lock_entries() to use a folio_batch

[Matthew Wilcox]

On Wed, Dec 08, 2021 at 07:29:33PM +0800, kernel test robot wrote:
> config: alpha-allyesconfig (https://download.01.org/0day-ci/archive/20211208/202112081952.NHF8MX2L-lkp@intel.com/config)
> compiler: alpha-linux-gcc (GCC) 11.2.0

Thanks.  Strangely, it doesn't reproduce on x86 allmodconfig.

Re: [PATCH 42/48] mm: Convert find_lock_entries() to use a folio_batch

[Matthew Wilcox]

[-- Attachment #1: Type: text/plain, Size: 299 bytes --]

On Wed, Dec 08, 2021 at 07:29:33PM +0800, kernel test robot wrote:
> config: alpha-allyesconfig (https://download.01.org/0day-ci/archive/20211208/202112081952.NHF8MX2L-lkp(a)intel.com/config)
> compiler: alpha-linux-gcc (GCC) 11.2.0

Thanks.  Strangely, it doesn't reproduce on x86 allmodconfig.

Re: [PATCH 46/48] truncate,shmem: Handle truncates that split large folios

[Matthew Wilcox]

On Wed, Dec 08, 2021 at 04:22:54AM +0000, Matthew Wilcox (Oracle) wrote:
> @@ -917,13 +904,13 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, loff_t lend,
>  	struct shmem_inode_info *info = SHMEM_I(inode);
>  	pgoff_t start = (lstart + PAGE_SIZE - 1) >> PAGE_SHIFT;
>  	pgoff_t end = (lend + 1) >> PAGE_SHIFT;
> -	unsigned int partial_start = lstart & (PAGE_SIZE - 1);
> -	unsigned int partial_end = (lend + 1) & (PAGE_SIZE - 1);
>  	struct folio_batch fbatch;
>  	pgoff_t indices[PAGEVEC_SIZE];
> +	struct folio *folio;

This turns a couple of other definitions of struct folio in this
function into shadowed definitions.  We don't have -Wshadow turned
on, so I didn't notice until doing more patch review this morning.
I'm going to fold in this patch:

+++ b/mm/shmem.c
@@ -919,7 +919,7 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, loff_t lend,
        while (index < end && find_lock_entries(mapping, index, end - 1,
                        &fbatch, indices)) {
                for (i = 0; i < folio_batch_count(&fbatch); i++) {
-                       struct folio *folio = fbatch.folios[i];
+                       folio = fbatch.folios[i];

                        index = indices[i];

@@ -985,7 +985,7 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, loff_t lend,
                        continue;
                }
                for (i = 0; i < folio_batch_count(&fbatch); i++) {
-                       struct folio *folio = fbatch.folios[i];
+                       folio = fbatch.folios[i];

                        index = indices[i];
                        if (xa_is_value(folio)) {

Re: [PATCH 01/48] filemap: Remove PageHWPoison check from next_uptodate_page()

[Christoph Hellwig]

On Wed, Dec 08, 2021 at 04:22:09AM +0000, Matthew Wilcox (Oracle) wrote:
> Pages are individually marked as suffering from hardware poisoning.
> Checking that the head page is not hardware poisoned doesn't make
> sense; we might be after a subpage.  We check each page individually
> before we use it, so this was an optimisation gone wrong.

Looks good,

Reviewed-by: Christoph Hellwig <hch@lst.de>

Re: [PATCH 02/48] fs/writeback: Convert inode_switch_wbs_work_fn to folios

[Christoph Hellwig]

On Wed, Dec 08, 2021 at 04:22:10AM +0000, Matthew Wilcox (Oracle) wrote:
> This gets the statistics correct by modifying the counters by the
> number of pages in the folio instead of by 1.

We can't actually hit this for a multi-page folio yet, can we?
So this should be more of a prep patch?

Otherwise looks good:

Reviewed-by: Christoph Hellwig <hch@lst.de>

Re: [PATCH 03/48] mm/doc: Add documentation for folio_test_uptodate

[Christoph Hellwig]

Do changes to comment get a doc prefix in the subject now?

Otherwise looks good:

Reviewed-by: Christoph Hellwig <hch@lst.de>

Re: [PATCH 04/48] mm/writeback: Improve __folio_mark_dirty() comment

[Christoph Hellwig]

Looks good,

Reviewed-by: Christoph Hellwig <hch@lst.de>

Re: [PATCH 05/48] pagevec: Add folio_batch

[Christoph Hellwig]

On Wed, Dec 08, 2021 at 04:22:13AM +0000, Matthew Wilcox (Oracle) wrote:
> +static inline void folio_batch_release(struct folio_batch *fbatch)
> +{
> +	pagevec_release((struct pagevec *)fbatch);
> +}
> +
> +static inline void folio_batch_remove_exceptionals(struct folio_batch *fbatch)
> +{
> +	pagevec_remove_exceptionals((struct pagevec *)fbatch);
> +}

I think these casts need documentation, both here and at the
struct folio_batch and struct pagevec definitions.

Alternatively I wonder if a union in stuct pagevec so that it can store
folios or pages might be the better option.

Re: [PATCH 06/48] iov_iter: Add copy_folio_to_iter()

[Christoph Hellwig]

On Wed, Dec 08, 2021 at 04:22:14AM +0000, Matthew Wilcox (Oracle) wrote:
> +static inline size_t copy_folio_to_iter(struct folio *folio, size_t offset,
> +		size_t bytes, struct iov_iter *i)
> +{
> +	return copy_page_to_iter((struct page *)folio, offset, bytes, i);
> +}

I think we had this 2 or three series ago, but these open coded casts
are a bad idea.  I think we need a proper and well documented helper for
them.

Re: [PATCH 07/48] iov_iter: Convert iter_xarray to use folios

[Christoph Hellwig]

> +		size_t offset = offset_in_folio(folio, start + __off);	\
> +		if (xas_retry(&xas, folio))			\
>  			continue;				\
> +		if (WARN_ON(xa_is_value(folio)))		\
>  			break;					\
> +		if (WARN_ON(folio_test_hugetlb(folio)))		\
>  			break;					\
> +		while (offset < folio_size(folio)) {		\

Nit: I'd be tempted to use a for loop on offset here.

Otherwise looks good:

Reviewed-by: Christoph Hellwig <hch@lst.de>

Re: [PATCH 08/48] mm: Add folio_test_pmd_mappable()

[Christoph Hellwig]

On Wed, Dec 08, 2021 at 04:22:16AM +0000, Matthew Wilcox (Oracle) wrote:
> Add a predicate to determine if the folio might be mapped by a PMD entry.
> If CONFIG_TRANSPARENT_HUGEPAGE is disabled, we know it can't be, even
> if it's large enough.

Looks good,

Reviewed-by: Christoph Hellwig <hch@lst.de>

Re: [PATCH 09/48] filemap: Add folio_put_wait_locked()

[Christoph Hellwig]

> -int put_and_wait_on_page_locked(struct page *page, int state);
> +int folio_put_wait_locked(struct folio *folio, int state);

This could actually move to mm/internal.h.

Otherwise looks good:

Reviewed-by: Christoph Hellwig <hch@lst.de>

Re: [PATCH 10/48] filemap: Convert page_cache_delete to take a folio

[Christoph Hellwig]

On Wed, Dec 08, 2021 at 04:22:18AM +0000, Matthew Wilcox (Oracle) wrote:
> It was already assuming a head page, so this is a straightforward
> conversion.  Convert the one caller to call page_folio(), even though
> it must currently be passing in a head page.

Looks good,

Reviewed-by: Christoph Hellwig <hch@lst.de>

Re: [PATCH 11/48] filemap: Add filemap_unaccount_folio()

[Christoph Hellwig]

Looks good,

Reviewed-by: Christoph Hellwig <hch@lst.de>

Re: [PATCH 12/48] filemap: Convert tracing of page cache operations to folio

[Christoph Hellwig]

Looks good,

Reviewed-by: Christoph Hellwig <hch@lst.de>

Re: [PATCH 13/48] filemap: Add filemap_remove_folio and __filemap_remove_folio

[Christoph Hellwig]

On Wed, Dec 08, 2021 at 04:22:21AM +0000, Matthew Wilcox (Oracle) wrote:
> Reimplement __delete_from_page_cache() as a wrapper around
> __filemap_remove_folio() and delete_from_page_cache() as a wrapper
> around filemap_remove_folio().  Remove the EXPORT_SYMBOL as
> delete_from_page_cache() was not used by any in-tree modules.
> Convert page_cache_free_page() into filemap_free_folio().
> 
> Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>

Looks good,

Reviewed-by: Christoph Hellwig <hch@lst.de>

Re: [PATCH 14/48] filemap: Convert find_get_entry to return a folio

[Christoph Hellwig]

On Wed, Dec 08, 2021 at 04:22:22AM +0000, Matthew Wilcox (Oracle) wrote:
> Convert callers to cope.  Saves 580 bytes of kernel text; all five
> callers are reduced in size.

Looks good,

Reviewed-by: Christoph Hellwig <hch@lst.de>

Re: [PATCH 15/48] filemap: Remove thp_contains()

[Christoph Hellwig]

On Wed, Dec 08, 2021 at 04:22:23AM +0000, Matthew Wilcox (Oracle) wrote:
> This function is now unused, so delete it.

Looks good,

Reviewed-by: Christoph Hellwig <hch@lst.de>

Re: [PATCH 16/48] filemap: Convert filemap_get_read_batch to use folios

[Christoph Hellwig]

On Wed, Dec 08, 2021 at 04:22:24AM +0000, Matthew Wilcox (Oracle) wrote:
> The page cache only stores folios, never tail pages.  Saves 29 bytes
> due to removing calls to compound_head().

Looksgood:

Reviewed-by: Christoph Hellwig <hch@lst.de>

Re: [PATCH 17/48] filemap: Convert find_get_pages_contig to folios

[Christoph Hellwig]

On Wed, Dec 08, 2021 at 04:22:25AM +0000, Matthew Wilcox (Oracle) wrote:
> None of the callers of find_get_pages_contig() want tail pages.  They all
> use order-0 pages today, but if they were converted, they'd want folios.
> So just remove the call to find_subpage() instead of replacing it with
> folio_page().

Looks good,

Reviewed-by: Christoph Hellwig <hch@lst.de>

Re: [PATCH 18/48] filemap: Convert filemap_read_page to take a folio

[Christoph Hellwig]

On Wed, Dec 08, 2021 at 04:22:26AM +0000, Matthew Wilcox (Oracle) wrote:
> One of the callers already had a folio; the other two grow by a few
> bytes, but filemap_read_page() shrinks by 50 bytes for a net reduction
> of 27 bytes.

Looks good,

Reviewed-by: Christoph Hellwig <hch@lst.de>

Re: [PATCH 19/48] filemap: Convert filemap_create_page to folio

[Christoph Hellwig]

On Wed, Dec 08, 2021 at 04:22:27AM +0000, Matthew Wilcox (Oracle) wrote:
> This is all internal to filemap and saves 100 bytes of text.

Looks good,

Reviewed-by: Christoph Hellwig <hch@lst.de>

Re: [PATCH 20/48] filemap: Convert filemap_range_uptodate to folios

[Christoph Hellwig]

On Wed, Dec 08, 2021 at 04:22:28AM +0000, Matthew Wilcox (Oracle) wrote:
> The only caller was already passing a head page, so this simply avoids
> a call to compound_head().

Looks good,

Reviewed-by: Christoph Hellwig <hch@lst.de>

Re: [PATCH 21/48] readahead: Convert page_cache_async_ra() to take a folio

[Christoph Hellwig]

On Wed, Dec 08, 2021 at 04:22:29AM +0000, Matthew Wilcox (Oracle) wrote:
> Using the folio here avoids checking whether it's a tail page.
> This patch mostly just enables some of the following patches.

Looks good,

Reviewed-by: Christoph Hellwig <hch@lst.de>

Re: [PATCH 22/48] readahead: Convert page_cache_ra_unbounded to folios

[Christoph Hellwig]

On Wed, Dec 08, 2021 at 04:22:30AM +0000, Matthew Wilcox (Oracle) wrote:
> This saves 99 bytes of kernel text.
> 
> Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>

Looks good,

Reviewed-by: Christoph Hellwig <hch@lst.de>

Re: [PATCH 23/48] filemap: Convert do_async_mmap_readahead to take a folio

[Christoph Hellwig]

On Wed, Dec 08, 2021 at 04:22:31AM +0000, Matthew Wilcox (Oracle) wrote:
> Call page_cache_async_ra() directly instead of indirecting through
> page_cache_async_readahead().
> 
> Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>

Looks good,

Reviewed-by: Christoph Hellwig <hch@lst.de>

Re: [PATCH 24/48] filemap: Convert filemap_fault to folio

[Christoph Hellwig]

Looks good,

Reviewed-by: Christoph Hellwig <hch@lst.de>

Re: [PATCH 33/48] mm: Add unmap_mapping_folio()

[Christoph Hellwig]

On Wed, Dec 08, 2021 at 04:22:41AM +0000, Matthew Wilcox (Oracle) wrote:
> Convert both callers of unmap_mapping_page() to call unmap_mapping_folio()
> instead.  Also move zap_details from linux/mm.h to mm/internal.h

In fact it could even move to mm/memory.c as no one needs it outside of
that file. __oom_reap_task_mm always passes a NULL zap_details argument
to unmap_page_range.

Otherwise looks good,

Reviewed-by: Christoph Hellwig <hch@lst.de>

Re: [PATCH 34/48] shmem: Convert part of shmem_undo_range() to use a folio

[Christoph Hellwig]

On Wed, Dec 08, 2021 at 04:22:42AM +0000, Matthew Wilcox (Oracle) wrote:
> find_lock_entries() never returns tail pages.  We cannot use page_folio()
> here as the pagevec may also contain swap entries, so simply cast.

Ugg.  At least this seems to be temporary, so:

Reviewed-by: Christoph Hellwig <hch@lst.de>

Re: [PATCH 25/48] filemap: Add read_cache_folio and read_mapping_folio

[Christoph Hellwig]

> thn makes up for the extra 100 bytes of text added to the various

s/thn/

>  	return read_cache_page(mapping, index, NULL, data);
>  }
>  
> +static inline struct folio *read_mapping_folio(struct address_space *mapping,
> +				pgoff_t index, void *data)
> +{
> +	return read_cache_folio(mapping, index, NULL, data);
> +}

Is there much of a point in this wrapper?

> +static struct page *do_read_cache_page(struct address_space *mapping,
> +		pgoff_t index, filler_t *filler, void *data, gfp_t gfp)
> +{
> +	struct folio *folio = read_cache_folio(mapping, index, filler, data);
> +	if (IS_ERR(folio))
> +		return &folio->page;
> +	return folio_file_page(folio, index);
> +}

This drops the gfp on the floor.

Re: [PATCH 26/48] filemap: Convert filemap_get_pages to use folios

[Christoph Hellwig]

Looks good,

Reviewed-by: Christoph Hellwig <hch@lst.de>

Re: [PATCH 27/48] filemap: Convert page_cache_delete_batch to folios

[Christoph Hellwig]

On Wed, Dec 08, 2021 at 04:22:35AM +0000, Matthew Wilcox (Oracle) wrote:
> Saves one call to compound_head() and reduces text size by 15 bytes.

Looks good,

Reviewed-by: Christoph Hellwig <hch@lst.de>

Re: [PATCH 28/48] filemap: Use folios in next_uptodate_page

[Christoph Hellwig]

On Wed, Dec 08, 2021 at 04:22:36AM +0000, Matthew Wilcox (Oracle) wrote:
> This saves 105 bytes of text.

Looks good,

Reviewed-by: Christoph Hellwig <hch@lst.de>

Re: [PATCH 29/48] filemap: Use a folio in filemap_map_pages

[Christoph Hellwig]

On Wed, Dec 08, 2021 at 04:22:37AM +0000, Matthew Wilcox (Oracle) wrote:
> Saves 61 bytes due to fewer calls to compound_head().

Looks good,

Reviewed-by: Christoph Hellwig <hch@lst.de>

Re: [PATCH 30/48] filemap: Use a folio in filemap_page_mkwrite

[Christoph Hellwig]

On Wed, Dec 08, 2021 at 04:22:38AM +0000, Matthew Wilcox (Oracle) wrote:
> This fixes a bug for tail pages.  They always have a NULL mapping, so
> the check would fail and we would never mark the folio as dirty.
> Ends up growing the kernel by 19 bytes although there will be fewer
> calls to compound_head() dynamically.

Looks good,

Reviewed-by: Christoph Hellwig <hch@lst.de>

Re: [PATCH 31/48] filemap: Add filemap_release_folio()

[Christoph Hellwig]

On Wed, Dec 08, 2021 at 04:22:39AM +0000, Matthew Wilcox (Oracle) wrote:
> Reimplement try_to_release_page() as a wrapper around
> filemap_release_folio().

Looks good,

Reviewed-by: Christoph Hellwig <hch@lst.de>

Re: [PATCH 32/48] truncate: Add truncate_cleanup_folio()

[Christoph Hellwig]

On Wed, Dec 08, 2021 at 04:22:40AM +0000, Matthew Wilcox (Oracle) wrote:
> Convert both callers of truncate_cleanup_page() to use
> truncate_cleanup_folio() instead.

Looks good,

Reviewed-by: Christoph Hellwig <hch@lst.de>

Re: [PATCH 35/48] truncate,shmem: Add truncate_inode_folio()

[Christoph Hellwig]

On Wed, Dec 08, 2021 at 04:22:43AM +0000, Matthew Wilcox (Oracle) wrote:
> Convert all callers of truncate_inode_page() to call
> truncate_inode_folio() instead, and move the declaration to mm/internal.h.
> Move the assertion that the caller is not passing in a tail page to
> generic_error_remove_page().  We can't entirely remove the struct page
> from the callers yet because the page pointer in the pvec might be a
> shadow/dax/swap entry instead of actually a page.
> 
> Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>

Looks good,

Reviewed-by: Christoph Hellwig <hch@lst.de>

Re: [PATCH 36/48] truncate: Skip known-truncated indices

[Christoph Hellwig]

On Wed, Dec 08, 2021 at 04:22:44AM +0000, Matthew Wilcox (Oracle) wrote:
> If we've truncated an entire folio, we can skip over all the indices
> covered by this folio.
> 
> Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>

Looks good,

Reviewed-by: Christoph Hellwig <hch@lst.de>

Re: [PATCH 37/48] truncate: Convert invalidate_inode_pages2_range() to use a folio

[Christoph Hellwig]

On Wed, Dec 08, 2021 at 04:22:45AM +0000, Matthew Wilcox (Oracle) wrote:
> If we're going to unmap a folio, we have to be sure to unmap the entire
> folio, not just the part of it which lies after the search index.
> 
> We cannot yet remove the struct page from invalidate_inode_pages2_range()
> because the page pointer in the pvec might be a shadow/dax/swap entry
> instead of actually a page.
> 
> Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>

Looks good,

Reviewed-by: Christoph Hellwig <hch@lst.de>

Re: [PATCH 38/48] truncate: Add invalidate_complete_folio2()

[Christoph Hellwig]

On Wed, Dec 08, 2021 at 04:22:46AM +0000, Matthew Wilcox (Oracle) wrote:
> Convert invalidate_complete_page2() to invalidate_complete_folio2().
> Use filemap_free_folio() to free the page instead of calling ->freepage
> manually.

Looks good:

Reviewed-by: Christoph Hellwig <hch@lst.de>

Although eventually we need to get rid of this silly *2 naming here.

Re: [PATCH 39/48] filemap: Convert filemap_read() to use a folio

[Christoph Hellwig]

On Wed, Dec 08, 2021 at 04:22:47AM +0000, Matthew Wilcox (Oracle) wrote:
>  		for (i = 0; i < pagevec_count(&pvec); i++) {
> -			struct page *page = pvec.pages[i];
> -			size_t page_size = thp_size(page);
> -			size_t offset = iocb->ki_pos & (page_size - 1);
> +			struct folio *folio = page_folio(pvec.pages[i]);
> +			size_t fsize = folio_size(folio);

Any reason for fsize vs folio_size?

Otherwise looks good:

Reviewed-by: Christoph Hellwig <hch@lst.de>

Re: [PATCH 40/48] filemap: Convert filemap_get_read_batch() to use a folio_batch

[Christoph Hellwig]

On Wed, Dec 08, 2021 at 04:22:48AM +0000, Matthew Wilcox (Oracle) wrote:
> This change ripples all the way through the filemap_read() call chain and
> removes a lot of messing about converting folios to pages and back again.

Looks good,

Reviewed-by: Christoph Hellwig <hch@lst.de>

Re: [PATCH 41/48] filemap: Return only folios from find_get_entries()

[Christoph Hellwig]

On Wed, Dec 08, 2021 at 04:22:49AM +0000, Matthew Wilcox (Oracle) wrote:
> The callers have all been converted to work on folios, so convert
> find_get_entries() to return a batch of folios instead of pages.
> We also now return multiple large folios in a single call.
> 
> Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
> Reviewed-by: Jan Kara <jack@suse.cz>
> Reviewed-by: William Kucharski <william.kucharski@oracle.com>

Looks good,

Reviewed-by: Christoph Hellwig <hch@lst.de>

Re: [PATCH 42/48] mm: Convert find_lock_entries() to use a folio_batch

[Christoph Hellwig]

Looks good (modulo the include chain thing):

Reviewed-by: Christoph Hellwig <hch@lst.de>

Re: [PATCH 43/48] mm: Remove pagevec_remove_exceptionals()

[Christoph Hellwig]

Looks good,

Reviewed-by: Christoph Hellwig <hch@lst.de>

Re: [PATCH 44/48] fs: Convert vfs_dedupe_file_range_compare to folios

[Christoph Hellwig]

On Wed, Dec 08, 2021 at 04:22:52AM +0000, Matthew Wilcox (Oracle) wrote:
> We still only operate on a single page of data at a time due to using
> kmap().  A more complex implementation would work on each page in a folio,
> but it's not clear that such a complex implementation would be worthwhile.

It sounds simple enough to be worthwhile.  But no need to do it here:

Reviewed-by: Christoph Hellwig <hch@lst.de>

Re: [PATCH 45/48] truncate: Convert invalidate_inode_pages2_range to folios

[Christoph Hellwig]

On Wed, Dec 08, 2021 at 04:22:53AM +0000, Matthew Wilcox (Oracle) wrote:
> If we're going to unmap a folio, we have to be sure to unmap the entire
> folio, not just the part of it which lies after the search index.
> 
> Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>

Looks good,

Reviewed-by: Christoph Hellwig <hch@lst.de>

Re: [PATCH 47/48] XArray: Add xas_advance()

[Christoph Hellwig]

On Wed, Dec 08, 2021 at 04:22:55AM +0000, Matthew Wilcox (Oracle) wrote:
> Add a new helper function to help iterate over multi-index entries.
> 
> Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>

Looks good,

Reviewed-by: Christoph Hellwig <hch@lst.de>

Re: [PATCH 46/48] truncate,shmem: Handle truncates that split large folios

[Christoph Hellwig]

> +	partial_end = ((lend + 1) % PAGE_SIZE) > 0;

The > 0 reads a little odd as we can never get a negative value here.

> +	shmem_get_folio(inode, lstart >> PAGE_SHIFT, &folio, SGP_READ);
> +	if (folio) {
> +		bool same_page;
> +
> +		same_page = lend < folio_pos(folio) + folio_size(folio);
> +		if (same_page)
> +			partial_end = false;
> +		folio_mark_dirty(folio);
> +		if (!truncate_inode_partial_folio(folio, lstart, lend)) {
> +			start = folio->index + folio_nr_pages(folio);
> +			if (same_page)
> +				end = folio->index;
>  		}
> +		folio_unlock(folio);
> +		folio_put(folio);
> +		folio = NULL;

The entire logic after the shmem_get_folio call is duplicated in
truncate_inode_pages_range.  Shouldn't we ty to combine that by moving it
into a helper?

If we can't merge it I think same_page shoud be renamed to same_folio as in
truncate_inode_pages_range.

>  	pgoff_t		start;		/* inclusive */
>  	pgoff_t		end;		/* exclusive */
> -	unsigned int	partial_start;	/* inclusive */
> -	unsigned int	partial_end;	/* exclusive */
>  	struct folio_batch fbatch;
>  	pgoff_t		indices[PAGEVEC_SIZE];
>  	pgoff_t		index;
>  	int		i;
> +	struct folio *	folio;

Weird indentation after the star.

> +	bool partial_end;

And the style here in general does not seem to match the existing code.

Re: [PATCH 48/48] mm: Use multi-index entries in the page cache

[Christoph Hellwig]

On Wed, Dec 08, 2021 at 04:22:56AM +0000, Matthew Wilcox (Oracle) wrote:
> We currently store large folios as 2^N consecutive entries.  While this
> consumes rather more memory than necessary, it also turns out to be buggy.
> A writeback operation which starts within a tail page of a dirty folio will
> not write back the folio as the xarray's dirty bit is only set on the
> head index.  With multi-index entries, the dirty bit will be found no
> matter where in the folio the operation starts.
> 
> This does end up simplifying the page cache slightly, although not as
> much as I had hoped.

This looks sensible to me, but I'm not xarray expert.

So the only thing I have to offer is a superficial nit:

> +static inline
> +bool folio_more_pages(struct folio *folio, pgoff_t index, pgoff_t max)

Weird identation indentation here.

Re: [PATCH 02/48] fs/writeback: Convert inode_switch_wbs_work_fn to folios

[Matthew Wilcox]

On Wed, Dec 22, 2021 at 10:50:20PM -0800, Christoph Hellwig wrote:
> On Wed, Dec 08, 2021 at 04:22:10AM +0000, Matthew Wilcox (Oracle) wrote:
> > This gets the statistics correct by modifying the counters by the
> > number of pages in the folio instead of by 1.
> 
> We can't actually hit this for a multi-page folio yet, can we?
> So this should be more of a prep patch?
> 
> Otherwise looks good:
> 
> Reviewed-by: Christoph Hellwig <hch@lst.de>

Right, even by the end of this series, we don't actually create large
folios.  I was hoping to get to that point, but there's still a reliance
on CONFIG_TRANSPARENT_HUGEPAGE that I need to get rid of.

Re: [PATCH 05/48] pagevec: Add folio_batch

[Matthew Wilcox]

On Wed, Dec 22, 2021 at 10:54:27PM -0800, Christoph Hellwig wrote:
> On Wed, Dec 08, 2021 at 04:22:13AM +0000, Matthew Wilcox (Oracle) wrote:
> > +static inline void folio_batch_release(struct folio_batch *fbatch)
> > +{
> > +	pagevec_release((struct pagevec *)fbatch);
> > +}
> > +
> > +static inline void folio_batch_remove_exceptionals(struct folio_batch *fbatch)
> > +{
> > +	pagevec_remove_exceptionals((struct pagevec *)fbatch);
> > +}
> 
> I think these casts need documentation, both here and at the
> struct folio_batch and struct pagevec definitions.
> 
> Alternatively I wonder if a union in stuct pagevec so that it can store
> folios or pages might be the better option.

I tried that way first, but then the caller & callee need to agree
whether they're storing folios or pages in the pagevec.  And that's kind
of why we have types.

pagevec_remove_exceptionals() goes away by the end of this series.
pagevec_release() will take longer to remove.  What documentation
do you want to see?

Re: [PATCH 06/48] iov_iter: Add copy_folio_to_iter()

[Matthew Wilcox]

On Wed, Dec 22, 2021 at 10:55:20PM -0800, Christoph Hellwig wrote:
> On Wed, Dec 08, 2021 at 04:22:14AM +0000, Matthew Wilcox (Oracle) wrote:
> > +static inline size_t copy_folio_to_iter(struct folio *folio, size_t offset,
> > +		size_t bytes, struct iov_iter *i)
> > +{
> > +	return copy_page_to_iter((struct page *)folio, offset, bytes, i);
> > +}
> 
> I think we had this 2 or three series ago, but these open coded casts
> are a bad idea.  I think we need a proper and well documented helper for
> them.

I don't want to see casts between folio & page in most code, but I think
having it in this kind of helper is fine.  It literally exists in order
to document that the callee handles folios correctly.

Re: [PATCH 07/48] iov_iter: Convert iter_xarray to use folios

[Matthew Wilcox]

On Wed, Dec 22, 2021 at 10:57:28PM -0800, Christoph Hellwig wrote:
> > +		size_t offset = offset_in_folio(folio, start + __off);	\
> > +		if (xas_retry(&xas, folio))			\
> >  			continue;				\
> > +		if (WARN_ON(xa_is_value(folio)))		\
> >  			break;					\
> > +		if (WARN_ON(folio_test_hugetlb(folio)))		\
> >  			break;					\
> > +		while (offset < folio_size(folio)) {		\
> 
> Nit: I'd be tempted to use a for loop on offset here.

Dave, this is your code originally ... any opinions?

> Otherwise looks good:
> 
> Reviewed-by: Christoph Hellwig <hch@lst.de>

Re: [PATCH 09/48] filemap: Add folio_put_wait_locked()

[Matthew Wilcox]

On Wed, Dec 22, 2021 at 11:00:40PM -0800, Christoph Hellwig wrote:
> > -int put_and_wait_on_page_locked(struct page *page, int state);
> > +int folio_put_wait_locked(struct folio *folio, int state);
> 
> This could actually move to mm/internal.h.

It doesn't have any callers outside mm/ for now (and actually a couple
of the callers are going away), but conceptually it's part of the
lock_page() family of functions.  Some other places might benefit from
waiting without the reference held.

Re: [PATCH 25/48] filemap: Add read_cache_folio and read_mapping_folio

[Matthew Wilcox]

On Thu, Dec 23, 2021 at 08:39:59AM +0100, Christoph Hellwig wrote:
> > thn makes up for the extra 100 bytes of text added to the various
> 
> s/thn/
> 
> >  	return read_cache_page(mapping, index, NULL, data);
> >  }
> >  
> > +static inline struct folio *read_mapping_folio(struct address_space *mapping,
> > +				pgoff_t index, void *data)
> > +{
> > +	return read_cache_folio(mapping, index, NULL, data);
> > +}
> 
> Is there much of a point in this wrapper?

Comparing read_mapping_page() vs read_cache_page(), 4 callers vs ~50.
It's definitely the preferred form, so I'd rather keep it.

> > +static struct page *do_read_cache_page(struct address_space *mapping,
> > +		pgoff_t index, filler_t *filler, void *data, gfp_t gfp)
> > +{
> > +	struct folio *folio = read_cache_folio(mapping, index, filler, data);
> > +	if (IS_ERR(folio))
> > +		return &folio->page;
> > +	return folio_file_page(folio, index);
> > +}
> 
> This drops the gfp on the floor.

Oops.  Will fix.

Re: [PATCH 07/48] iov_iter: Convert iter_xarray to use folios

[David Howells]

Christoph Hellwig <hch@infradead.org> wrote:

> > +		size_t offset = offset_in_folio(folio, start + __off);	\
> > +		if (xas_retry(&xas, folio))			\
> >  			continue;				\
> > +		if (WARN_ON(xa_is_value(folio)))		\
> >  			break;					\
> > +		if (WARN_ON(folio_test_hugetlb(folio)))		\
> >  			break;					\
> > +		while (offset < folio_size(folio)) {		\
> 
> Nit: I'd be tempted to use a for loop on offset here.

A while-loop makes more sense here.  The adjustment you need for offset
(ie. len) is overwritten after offset is altered at the end of the loop:

> +			offset += len;				\
> +			len = PAGE_SIZE;			\
>  		}						\

So you'd have to move both of these into the for-incrementor expression, in
addition to moving in the initialiser expression, making the thing less
readable.

David

Re: [PATCH 25/48] filemap: Add read_cache_folio and read_mapping_folio

[Matthew Wilcox]

On Thu, Dec 23, 2021 at 03:18:34PM +0000, Matthew Wilcox wrote:
> > > +static struct page *do_read_cache_page(struct address_space *mapping,
> > > +		pgoff_t index, filler_t *filler, void *data, gfp_t gfp)
> > > +{
> > > +	struct folio *folio = read_cache_folio(mapping, index, filler, data);
> > > +	if (IS_ERR(folio))
> > > +		return &folio->page;
> > > +	return folio_file_page(folio, index);
> > > +}
> > 
> > This drops the gfp on the floor.
> 
> Oops.  Will fix.

For the record, this fix:

-       struct folio *folio = read_cache_folio(mapping, index, filler, data);
+       struct folio *folio;
+
+       folio = do_read_cache_folio(mapping, index, filler, data, gfp);

Re: [PATCH 25/48] filemap: Add read_cache_folio and read_mapping_folio

[Matthew Wilcox]

On Wed, Dec 08, 2021 at 04:22:33AM +0000, Matthew Wilcox (Oracle) wrote:
> @@ -3503,23 +3491,23 @@ static struct page *do_read_cache_page(struct address_space *mapping,
>  	 * avoid spurious serialisations and wakeups when multiple processes
>  	 * wait on the same page for IO to complete.
>  	 */
> -	wait_on_page_locked(page);
> -	if (PageUptodate(page))
> +	folio_wait_locked(folio);
> +	if (folio_test_uptodate(folio))
>  		goto out;
>  
>  	/* Distinguish between all the cases under the safety of the lock */
> -	lock_page(page);
> +	folio_lock(folio);
>  
>  	/* Case c or d, restart the operation */
> -	if (!page->mapping) {
> -		unlock_page(page);
> -		put_page(page);
> +	if (!folio->mapping) {
> +		folio_unlock(folio);
> +		folio_put(folio);
>  		goto repeat;
>  	}

Re-reviewing this patch after Christoph's feedback, I believe it is
wrong to sleep with the refcount elevated, waiting for someone else to
unlock the page.  Doing that prevents anyone from splitting the folio,
which can be annoying.

So I'm thinking about adding this patch (as a follow-on).  It brings
the code closer to the read_iter path, which is always a good thing.
The comment was going to be made untrue by this patch, and I tried
rewriting it, but concluded ultimately that it was more distracting than
informative.

diff --git a/mm/filemap.c b/mm/filemap.c
index 6f541d931a4c..33077c264d79 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -3451,45 +3451,12 @@ static struct folio *do_read_cache_folio(struct address_space *mapping,
 	if (folio_test_uptodate(folio))
 		goto out;
 
-	/*
-	 * Page is not up to date and may be locked due to one of the following
-	 * case a: Page is being filled and the page lock is held
-	 * case b: Read/write error clearing the page uptodate status
-	 * case c: Truncation in progress (page locked)
-	 * case d: Reclaim in progress
-	 *
-	 * Case a, the page will be up to date when the page is unlocked.
-	 *    There is no need to serialise on the page lock here as the page
-	 *    is pinned so the lock gives no additional protection. Even if the
-	 *    page is truncated, the data is still valid if PageUptodate as
-	 *    it's a race vs truncate race.
-	 * Case b, the page will not be up to date
-	 * Case c, the page may be truncated but in itself, the data may still
-	 *    be valid after IO completes as it's a read vs truncate race. The
-	 *    operation must restart if the page is not uptodate on unlock but
-	 *    otherwise serialising on page lock to stabilise the mapping gives
-	 *    no additional guarantees to the caller as the page lock is
-	 *    released before return.
-	 * Case d, similar to truncation. If reclaim holds the page lock, it
-	 *    will be a race with remove_mapping that determines if the mapping
-	 *    is valid on unlock but otherwise the data is valid and there is
-	 *    no need to serialise with page lock.
-	 *
-	 * As the page lock gives no additional guarantee, we optimistically
-	 * wait on the page to be unlocked and check if it's up to date and
-	 * use the page if it is. Otherwise, the page lock is required to
-	 * distinguish between the different cases. The motivation is that we
-	 * avoid spurious serialisations and wakeups when multiple processes
-	 * wait on the same page for IO to complete.
-	 */
-	folio_wait_locked(folio);
-	if (folio_test_uptodate(folio))
-		goto out;
-
-	/* Distinguish between all the cases under the safety of the lock */
-	folio_lock(folio);
+	if (!folio_trylock(folio)) {
+		folio_put_wait_locked(folio, TASK_UNINTERRUPTIBLE);
+		goto repeat;
+	}
 
-	/* Case c or d, restart the operation */
+	/* Folio was truncated from mapping */
 	if (!folio->mapping) {
 		folio_unlock(folio);
 		folio_put(folio);

Re: [PATCH 05/48] pagevec: Add folio_batch

[Christoph Hellwig]

On Thu, Dec 23, 2021 at 02:18:40PM +0000, Matthew Wilcox wrote:
> > I think these casts need documentation, both here and at the
> > struct folio_batch and struct pagevec definitions.
> > 
> > Alternatively I wonder if a union in stuct pagevec so that it can store
> > folios or pages might be the better option.
> 
> I tried that way first, but then the caller & callee need to agree
> whether they're storing folios or pages in the pagevec.  And that's kind
> of why we have types.
> 
> pagevec_remove_exceptionals() goes away by the end of this series.
> pagevec_release() will take longer to remove.  What documentation
> do you want to see?

Mostly comments at the pagevec and folio_batch definitions that they
need to match because of these two functions, and then maybe a
backreference from the casts to the definitions.

Re: [PATCH 07/48] iov_iter: Convert iter_xarray to use folios

[Christoph Hellwig]

On Thu, Dec 23, 2021 at 03:24:21PM +0000, David Howells wrote:
> > > +		while (offset < folio_size(folio)) {		\
> > 
> > Nit: I'd be tempted to use a for loop on offset here.
> 
> A while-loop makes more sense here.  The adjustment you need for offset
> (ie. len) is overwritten after offset is altered at the end of the loop:
> 
> > +			offset += len;				\
> > +			len = PAGE_SIZE;			\
> >  		}						\
> 
> So you'd have to move both of these into the for-incrementor expression, in
> addition to moving in the initialiser expression, making the thing less
> readable.

Ok.

Re: [PATCH 00/48] Folios for 5.17

[William Kucharski]

Consolidated multiple review comments into one email, the majority are nits at
best:

[PATCH 04/48]:

An obnoxiously pendantic English grammar nit:

+ * lock).  This can also be called from mark_buffer_dirty(), which I

The period should be inside the paren, e.g.: "lock.)"


[PATCH 05/48]:

+       unsigned char aux[3];

I'd like to see an explanation of why this is "3."

+static inline void folio_batch_init(struct folio_batch *fbatch)
+{
+       fbatch->nr = 0;
+}
+
+static inline unsigned int folio_batch_count(struct folio_batch *fbatch)
+{
+       return fbatch->nr;
+}
+
+static inline unsigned int fbatch_space(struct folio_batch *fbatch)
+{
+       return PAGEVEC_SIZE - fbatch->nr;
+}
+
+/**
+ * folio_batch_add() - Add a folio to a batch.
+ * @fbatch: The folio batch.
+ * @folio: The folio to add.
+ *
+ * The folio is added to the end of the batch.
+ * The batch must have previously been initialised using folio_batch_init().
+ *
+ * Return: The number of slots still available.
+ */
+static inline unsigned folio_batch_add(struct folio_batch *fbatch,
+               struct folio *folio)
+{
+       fbatch->folios[fbatch->nr++] = folio;

Is there any need to validate fbatch in these inlines?

[PATCH 07/48]:

+       xas_for_each(&xas, folio, ULONG_MAX) {                  \
                unsigned left;                                  \
-               if (xas_retry(&xas, head))                      \
+               size_t offset = offset_in_folio(folio, start + __off);  \
+               if (xas_retry(&xas, folio))                     \
                        continue;                               \
-               if (WARN_ON(xa_is_value(head)))                 \
+               if (WARN_ON(xa_is_value(folio)))                \
                        break;                                  \
-               if (WARN_ON(PageHuge(head)))                    \
+               if (WARN_ON(folio_test_hugetlb(folio)))         \
                        break;                                  \
-               for (j = (head->index < index) ? index - head->index : 0; \
-                    j < thp_nr_pages(head); j++) {             \
-                       void *kaddr = kmap_local_page(head + j);        \
-                       base = kaddr + offset;                  \
-                       len = PAGE_SIZE - offset;               \
+               while (offset < folio_size(folio)) {            \

Since offset is not actually used until after a bunch of error conditions
may exit or restart the loop, and isn't used at all in between, defer
its calculation until just before its first use in the "while."

[PATCH 25/48]:

Whether you use your follow-on patch sent to Christop or defer it until later
as mentioned in your followup email, either approach is fine with me.

Otherwise it all looks good.

For the series:

Reviewed-by: William Kucharski <william.kucharski@oracle.com>


> On Dec 7, 2021, at 9:22 PM, Matthew Wilcox (Oracle) <willy@infradead.org> wrote:
> 
> I was trying to get all the way to adding large folios to the page cache,
> but I ran out of time.  I think the changes in this batch of patches
> are worth adding, even without large folio support for filesystems other
> than tmpfs.
> 
> The big change here is the last patch which switches from storing
> large folios in the page cache as 2^N identical entries to using the
> XArray support for multi-index entries.  As the changelog says, this
> fixes a bug that can occur when doing (eg) msync() or sync_file_range().
> 
> Some parts of this have been sent before.  The first patch is already
> in Andrew's tree, but I included it here so the build bots don't freak
> out about not being able to apply this patch series.  The folio_batch
> (replacement for pagevec) is new.  I also fixed a few bugs.
> 
> This all passes xfstests with no new failures on both xfs and tmpfs.
> I intend to put all this into for-next tomorrow.
> 
> Matthew Wilcox (Oracle) (48):
>  filemap: Remove PageHWPoison check from next_uptodate_page()
>  fs/writeback: Convert inode_switch_wbs_work_fn to folios
>  mm/doc: Add documentation for folio_test_uptodate
>  mm/writeback: Improve __folio_mark_dirty() comment
>  pagevec: Add folio_batch
>  iov_iter: Add copy_folio_to_iter()
>  iov_iter: Convert iter_xarray to use folios
>  mm: Add folio_test_pmd_mappable()
>  filemap: Add folio_put_wait_locked()
>  filemap: Convert page_cache_delete to take a folio
>  filemap: Add filemap_unaccount_folio()
>  filemap: Convert tracing of page cache operations to folio
>  filemap: Add filemap_remove_folio and __filemap_remove_folio
>  filemap: Convert find_get_entry to return a folio
>  filemap: Remove thp_contains()
>  filemap: Convert filemap_get_read_batch to use folios
>  filemap: Convert find_get_pages_contig to folios
>  filemap: Convert filemap_read_page to take a folio
>  filemap: Convert filemap_create_page to folio
>  filemap: Convert filemap_range_uptodate to folios
>  readahead: Convert page_cache_async_ra() to take a folio
>  readahead: Convert page_cache_ra_unbounded to folios
>  filemap: Convert do_async_mmap_readahead to take a folio
>  filemap: Convert filemap_fault to folio
>  filemap: Add read_cache_folio and read_mapping_folio
>  filemap: Convert filemap_get_pages to use folios
>  filemap: Convert page_cache_delete_batch to folios
>  filemap: Use folios in next_uptodate_page
>  filemap: Use a folio in filemap_map_pages
>  filemap: Use a folio in filemap_page_mkwrite
>  filemap: Add filemap_release_folio()
>  truncate: Add truncate_cleanup_folio()
>  mm: Add unmap_mapping_folio()
>  shmem: Convert part of shmem_undo_range() to use a folio
>  truncate,shmem: Add truncate_inode_folio()
>  truncate: Skip known-truncated indices
>  truncate: Convert invalidate_inode_pages2_range() to use a folio
>  truncate: Add invalidate_complete_folio2()
>  filemap: Convert filemap_read() to use a folio
>  filemap: Convert filemap_get_read_batch() to use a folio_batch
>  filemap: Return only folios from find_get_entries()
>  mm: Convert find_lock_entries() to use a folio_batch
>  mm: Remove pagevec_remove_exceptionals()
>  fs: Convert vfs_dedupe_file_range_compare to folios
>  truncate: Convert invalidate_inode_pages2_range to folios
>  truncate,shmem: Handle truncates that split large folios
>  XArray: Add xas_advance()
>  mm: Use multi-index entries in the page cache
> 
> fs/fs-writeback.c              |  24 +-
> fs/remap_range.c               | 116 ++--
> include/linux/huge_mm.h        |  14 +
> include/linux/mm.h             |  68 +--
> include/linux/page-flags.h     |  13 +-
> include/linux/pagemap.h        |  59 +-
> include/linux/pagevec.h        |  61 ++-
> include/linux/uio.h            |   7 +
> include/linux/xarray.h         |  18 +
> include/trace/events/filemap.h |  32 +-
> lib/iov_iter.c                 |  29 +-
> lib/xarray.c                   |   6 +-
> mm/filemap.c                   | 967 ++++++++++++++++-----------------
> mm/folio-compat.c              |  11 +
> mm/huge_memory.c               |  20 +-
> mm/internal.h                  |  35 +-
> mm/khugepaged.c                |  12 +-
> mm/memory.c                    |  27 +-
> mm/migrate.c                   |  29 +-
> mm/page-writeback.c            |   6 +-
> mm/readahead.c                 |  24 +-
> mm/shmem.c                     | 174 +++---
> mm/swap.c                      |  26 +-
> mm/truncate.c                  | 305 ++++++-----
> 24 files changed, 1100 insertions(+), 983 deletions(-)
> 
> -- 
> 2.33.0
> 
> 

Re: [PATCH 39/48] filemap: Convert filemap_read() to use a folio

[Matthew Wilcox]

On Thu, Dec 23, 2021 at 09:22:02AM +0100, Christoph Hellwig wrote:
> On Wed, Dec 08, 2021 at 04:22:47AM +0000, Matthew Wilcox (Oracle) wrote:
> >  		for (i = 0; i < pagevec_count(&pvec); i++) {
> > -			struct page *page = pvec.pages[i];
> > -			size_t page_size = thp_size(page);
> > -			size_t offset = iocb->ki_pos & (page_size - 1);
> > +			struct folio *folio = page_folio(pvec.pages[i]);
> > +			size_t fsize = folio_size(folio);
> 
> Any reason for fsize vs folio_size?

  CC      mm/filemap.o
../mm/filemap.c: In function ‘filemap_read’:
../mm/filemap.c:2672:45: error: called object ‘folio_size’ is not a function or function pointer
 2672 |                         size_t folio_size = folio_size(folio);
      |                                             ^~~~~~~~~~
../mm/filemap.c:2672:32: note: declared here
 2672 |                         size_t folio_size = folio_size(folio);
      |                                ^~~~~~~~~~

Re: [PATCH 33/48] mm: Add unmap_mapping_folio()

[Matthew Wilcox]

On Thu, Dec 23, 2021 at 08:36:45AM +0100, Christoph Hellwig wrote:
> On Wed, Dec 08, 2021 at 04:22:41AM +0000, Matthew Wilcox (Oracle) wrote:
> > Convert both callers of unmap_mapping_page() to call unmap_mapping_folio()
> > instead.  Also move zap_details from linux/mm.h to mm/internal.h
> 
> In fact it could even move to mm/memory.c as no one needs it outside of
> that file. __oom_reap_task_mm always passes a NULL zap_details argument
> to unmap_page_range.

Umm ... no?

static inline bool
zap_skip_check_mapping(struct zap_details *details, struct page *page)
{
        if (!details || !page)
                return false;

        return details->zap_mapping &&
            (details->zap_mapping != page_rmapping(page));
}

Re: [PATCH 00/48] Folios for 5.17

[Matthew Wilcox]

On Wed, Dec 08, 2021 at 04:22:08AM +0000, Matthew Wilcox (Oracle) wrote:
> This all passes xfstests with no new failures on both xfs and tmpfs.
> I intend to put all this into for-next tomorrow.

As a result of Christoph's review, here's the diff.  I don't
think it's worth re-posting the entire patch series.


diff --git a/include/linux/pagevec.h b/include/linux/pagevec.h
index 7d3494f7fb70..dda8d5868c81 100644
--- a/include/linux/pagevec.h
+++ b/include/linux/pagevec.h
@@ -18,6 +18,7 @@ struct page;
 struct folio;
 struct address_space;
 
+/* Layout must match folio_batch */
 struct pagevec {
 	unsigned char nr;
 	bool percpu_pvec_drained;
@@ -85,17 +86,22 @@ static inline void pagevec_release(struct pagevec *pvec)
  * struct folio_batch - A collection of folios.
  *
  * The folio_batch is used to amortise the cost of retrieving and
- * operating on a set of folios.  The order of folios in the batch is
- * not considered important.  Some users of the folio_batch store
- * "exceptional" entries in it which can be removed by calling
- * folio_batch_remove_exceptionals().
+ * operating on a set of folios.  The order of folios in the batch may be
+ * significant (eg delete_from_page_cache_batch()).  Some users of the
+ * folio_batch store "exceptional" entries in it which can be removed
+ * by calling folio_batch_remove_exceptionals().
  */
 struct folio_batch {
 	unsigned char nr;
-	unsigned char aux[3];
+	bool percpu_pvec_drained;
 	struct folio *folios[PAGEVEC_SIZE];
 };
 
+/* Layout must match pagevec */
+static_assert(sizeof(struct pagevec) == sizeof(struct folio_batch));
+static_assert(offsetof(struct pagevec, pages) ==
+		offsetof(struct folio_batch, folios));
+
 /**
  * folio_batch_init() - Initialise a batch of folios
  * @fbatch: The folio batch.
diff --git a/include/linux/uio.h b/include/linux/uio.h
index 8479cf46b5b1..781d98d96611 100644
--- a/include/linux/uio.h
+++ b/include/linux/uio.h
@@ -150,7 +150,7 @@ size_t _copy_from_iter_nocache(void *addr, size_t bytes, struct iov_iter *i);
 static inline size_t copy_folio_to_iter(struct folio *folio, size_t offset,
 		size_t bytes, struct iov_iter *i)
 {
-	return copy_page_to_iter((struct page *)folio, offset, bytes, i);
+	return copy_page_to_iter(&folio->page, offset, bytes, i);
 }
 
 static __always_inline __must_check
diff --git a/mm/filemap.c b/mm/filemap.c
index 9b5b2d962c37..33077c264d79 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -3451,45 +3451,12 @@ static struct folio *do_read_cache_folio(struct address_space *mapping,
 	if (folio_test_uptodate(folio))
 		goto out;
 
-	/*
-	 * Page is not up to date and may be locked due to one of the following
-	 * case a: Page is being filled and the page lock is held
-	 * case b: Read/write error clearing the page uptodate status
-	 * case c: Truncation in progress (page locked)
-	 * case d: Reclaim in progress
-	 *
-	 * Case a, the page will be up to date when the page is unlocked.
-	 *    There is no need to serialise on the page lock here as the page
-	 *    is pinned so the lock gives no additional protection. Even if the
-	 *    page is truncated, the data is still valid if PageUptodate as
-	 *    it's a race vs truncate race.
-	 * Case b, the page will not be up to date
-	 * Case c, the page may be truncated but in itself, the data may still
-	 *    be valid after IO completes as it's a read vs truncate race. The
-	 *    operation must restart if the page is not uptodate on unlock but
-	 *    otherwise serialising on page lock to stabilise the mapping gives
-	 *    no additional guarantees to the caller as the page lock is
-	 *    released before return.
-	 * Case d, similar to truncation. If reclaim holds the page lock, it
-	 *    will be a race with remove_mapping that determines if the mapping
-	 *    is valid on unlock but otherwise the data is valid and there is
-	 *    no need to serialise with page lock.
-	 *
-	 * As the page lock gives no additional guarantee, we optimistically
-	 * wait on the page to be unlocked and check if it's up to date and
-	 * use the page if it is. Otherwise, the page lock is required to
-	 * distinguish between the different cases. The motivation is that we
-	 * avoid spurious serialisations and wakeups when multiple processes
-	 * wait on the same page for IO to complete.
-	 */
-	folio_wait_locked(folio);
-	if (folio_test_uptodate(folio))
-		goto out;
-
-	/* Distinguish between all the cases under the safety of the lock */
-	folio_lock(folio);
+	if (!folio_trylock(folio)) {
+		folio_put_wait_locked(folio, TASK_UNINTERRUPTIBLE);
+		goto repeat;
+	}
 
-	/* Case c or d, restart the operation */
+	/* Folio was truncated from mapping */
 	if (!folio->mapping) {
 		folio_unlock(folio);
 		folio_put(folio);
@@ -3543,7 +3510,9 @@ EXPORT_SYMBOL(read_cache_folio);
 static struct page *do_read_cache_page(struct address_space *mapping,
 		pgoff_t index, filler_t *filler, void *data, gfp_t gfp)
 {
-	struct folio *folio = read_cache_folio(mapping, index, filler, data);
+	struct folio *folio;
+
+	folio = do_read_cache_folio(mapping, index, filler, data, gfp);
 	if (IS_ERR(folio))
 		return &folio->page;
 	return folio_file_page(folio, index);
diff --git a/mm/shmem.c b/mm/shmem.c
index 735404fdfcc3..6cd3af0addc1 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -942,18 +942,18 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, loff_t lend,
 		index++;
 	}
 
-	partial_end = ((lend + 1) % PAGE_SIZE) > 0;
+	partial_end = ((lend + 1) % PAGE_SIZE) != 0;
 	shmem_get_folio(inode, lstart >> PAGE_SHIFT, &folio, SGP_READ);
 	if (folio) {
-		bool same_page;
+		bool same_folio;
 
-		same_page = lend < folio_pos(folio) + folio_size(folio);
-		if (same_page)
+		same_folio = lend < folio_pos(folio) + folio_size(folio);
+		if (same_folio)
 			partial_end = false;
 		folio_mark_dirty(folio);
 		if (!truncate_inode_partial_folio(folio, lstart, lend)) {
 			start = folio->index + folio_nr_pages(folio);
-			if (same_page)
+			if (same_folio)
 				end = folio->index;
 		}
 		folio_unlock(folio);
diff --git a/mm/truncate.c b/mm/truncate.c
index 336c8d099efa..749aac71fda5 100644
--- a/mm/truncate.c
+++ b/mm/truncate.c
@@ -350,8 +350,8 @@ void truncate_inode_pages_range(struct address_space *mapping,
 	pgoff_t		indices[PAGEVEC_SIZE];
 	pgoff_t		index;
 	int		i;
-	struct folio *	folio;
-	bool partial_end;
+	struct folio	*folio;
+	bool		partial_end;
 
 	if (mapping_empty(mapping))
 		goto out;
@@ -388,7 +388,7 @@ void truncate_inode_pages_range(struct address_space *mapping,
 		cond_resched();
 	}
 
-	partial_end = ((lend + 1) % PAGE_SIZE) > 0;
+	partial_end = ((lend + 1) % PAGE_SIZE) != 0;
 	folio = __filemap_get_folio(mapping, lstart >> PAGE_SHIFT, FGP_LOCK, 0);
 	if (folio) {
 		bool same_folio = lend < folio_pos(folio) + folio_size(folio);

Re: [PATCH 00/48] Folios for 5.17

[William Kucharski]

Looks good, addresses one of my comments as well.

Again:

Reviewed-by: William Kucharski <william.kucharski@oracle.com>

> On Jan 2, 2022, at 9:19 AM, Matthew Wilcox <willy@infradead.org> wrote:
> 
> On Wed, Dec 08, 2021 at 04:22:08AM +0000, Matthew Wilcox (Oracle) wrote:
>> This all passes xfstests with no new failures on both xfs and tmpfs.
>> I intend to put all this into for-next tomorrow.
> 
> As a result of Christoph's review, here's the diff.  I don't
> think it's worth re-posting the entire patch series.
> 
> 
> diff --git a/include/linux/pagevec.h b/include/linux/pagevec.h
> index 7d3494f7fb70..dda8d5868c81 100644
> --- a/include/linux/pagevec.h
> +++ b/include/linux/pagevec.h
> @@ -18,6 +18,7 @@ struct page;
> struct folio;
> struct address_space;
> 
> +/* Layout must match folio_batch */
> struct pagevec {
> 	unsigned char nr;
> 	bool percpu_pvec_drained;
> @@ -85,17 +86,22 @@ static inline void pagevec_release(struct pagevec *pvec)
>  * struct folio_batch - A collection of folios.
>  *
>  * The folio_batch is used to amortise the cost of retrieving and
> - * operating on a set of folios.  The order of folios in the batch is
> - * not considered important.  Some users of the folio_batch store
> - * "exceptional" entries in it which can be removed by calling
> - * folio_batch_remove_exceptionals().
> + * operating on a set of folios.  The order of folios in the batch may be
> + * significant (eg delete_from_page_cache_batch()).  Some users of the
> + * folio_batch store "exceptional" entries in it which can be removed
> + * by calling folio_batch_remove_exceptionals().
>  */
> struct folio_batch {
> 	unsigned char nr;
> -	unsigned char aux[3];
> +	bool percpu_pvec_drained;
> 	struct folio *folios[PAGEVEC_SIZE];
> };
> 
> +/* Layout must match pagevec */
> +static_assert(sizeof(struct pagevec) == sizeof(struct folio_batch));
> +static_assert(offsetof(struct pagevec, pages) ==
> +		offsetof(struct folio_batch, folios));
> +
> /**
>  * folio_batch_init() - Initialise a batch of folios
>  * @fbatch: The folio batch.
> diff --git a/include/linux/uio.h b/include/linux/uio.h
> index 8479cf46b5b1..781d98d96611 100644
> --- a/include/linux/uio.h
> +++ b/include/linux/uio.h
> @@ -150,7 +150,7 @@ size_t _copy_from_iter_nocache(void *addr, size_t bytes, struct iov_iter *i);
> static inline size_t copy_folio_to_iter(struct folio *folio, size_t offset,
> 		size_t bytes, struct iov_iter *i)
> {
> -	return copy_page_to_iter((struct page *)folio, offset, bytes, i);
> +	return copy_page_to_iter(&folio->page, offset, bytes, i);
> }
> 
> static __always_inline __must_check
> diff --git a/mm/filemap.c b/mm/filemap.c
> index 9b5b2d962c37..33077c264d79 100644
> --- a/mm/filemap.c
> +++ b/mm/filemap.c
> @@ -3451,45 +3451,12 @@ static struct folio *do_read_cache_folio(struct address_space *mapping,
> 	if (folio_test_uptodate(folio))
> 		goto out;
> 
> -	/*
> -	 * Page is not up to date and may be locked due to one of the following
> -	 * case a: Page is being filled and the page lock is held
> -	 * case b: Read/write error clearing the page uptodate status
> -	 * case c: Truncation in progress (page locked)
> -	 * case d: Reclaim in progress
> -	 *
> -	 * Case a, the page will be up to date when the page is unlocked.
> -	 *    There is no need to serialise on the page lock here as the page
> -	 *    is pinned so the lock gives no additional protection. Even if the
> -	 *    page is truncated, the data is still valid if PageUptodate as
> -	 *    it's a race vs truncate race.
> -	 * Case b, the page will not be up to date
> -	 * Case c, the page may be truncated but in itself, the data may still
> -	 *    be valid after IO completes as it's a read vs truncate race. The
> -	 *    operation must restart if the page is not uptodate on unlock but
> -	 *    otherwise serialising on page lock to stabilise the mapping gives
> -	 *    no additional guarantees to the caller as the page lock is
> -	 *    released before return.
> -	 * Case d, similar to truncation. If reclaim holds the page lock, it
> -	 *    will be a race with remove_mapping that determines if the mapping
> -	 *    is valid on unlock but otherwise the data is valid and there is
> -	 *    no need to serialise with page lock.
> -	 *
> -	 * As the page lock gives no additional guarantee, we optimistically
> -	 * wait on the page to be unlocked and check if it's up to date and
> -	 * use the page if it is. Otherwise, the page lock is required to
> -	 * distinguish between the different cases. The motivation is that we
> -	 * avoid spurious serialisations and wakeups when multiple processes
> -	 * wait on the same page for IO to complete.
> -	 */
> -	folio_wait_locked(folio);
> -	if (folio_test_uptodate(folio))
> -		goto out;
> -
> -	/* Distinguish between all the cases under the safety of the lock */
> -	folio_lock(folio);
> +	if (!folio_trylock(folio)) {
> +		folio_put_wait_locked(folio, TASK_UNINTERRUPTIBLE);
> +		goto repeat;
> +	}
> 
> -	/* Case c or d, restart the operation */
> +	/* Folio was truncated from mapping */
> 	if (!folio->mapping) {
> 		folio_unlock(folio);
> 		folio_put(folio);
> @@ -3543,7 +3510,9 @@ EXPORT_SYMBOL(read_cache_folio);
> static struct page *do_read_cache_page(struct address_space *mapping,
> 		pgoff_t index, filler_t *filler, void *data, gfp_t gfp)
> {
> -	struct folio *folio = read_cache_folio(mapping, index, filler, data);
> +	struct folio *folio;
> +
> +	folio = do_read_cache_folio(mapping, index, filler, data, gfp);
> 	if (IS_ERR(folio))
> 		return &folio->page;
> 	return folio_file_page(folio, index);
> diff --git a/mm/shmem.c b/mm/shmem.c
> index 735404fdfcc3..6cd3af0addc1 100644
> --- a/mm/shmem.c
> +++ b/mm/shmem.c
> @@ -942,18 +942,18 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, loff_t lend,
> 		index++;
> 	}
> 
> -	partial_end = ((lend + 1) % PAGE_SIZE) > 0;
> +	partial_end = ((lend + 1) % PAGE_SIZE) != 0;
> 	shmem_get_folio(inode, lstart >> PAGE_SHIFT, &folio, SGP_READ);
> 	if (folio) {
> -		bool same_page;
> +		bool same_folio;
> 
> -		same_page = lend < folio_pos(folio) + folio_size(folio);
> -		if (same_page)
> +		same_folio = lend < folio_pos(folio) + folio_size(folio);
> +		if (same_folio)
> 			partial_end = false;
> 		folio_mark_dirty(folio);
> 		if (!truncate_inode_partial_folio(folio, lstart, lend)) {
> 			start = folio->index + folio_nr_pages(folio);
> -			if (same_page)
> +			if (same_folio)
> 				end = folio->index;
> 		}
> 		folio_unlock(folio);
> diff --git a/mm/truncate.c b/mm/truncate.c
> index 336c8d099efa..749aac71fda5 100644
> --- a/mm/truncate.c
> +++ b/mm/truncate.c
> @@ -350,8 +350,8 @@ void truncate_inode_pages_range(struct address_space *mapping,
> 	pgoff_t		indices[PAGEVEC_SIZE];
> 	pgoff_t		index;
> 	int		i;
> -	struct folio *	folio;
> -	bool partial_end;
> +	struct folio	*folio;
> +	bool		partial_end;
> 
> 	if (mapping_empty(mapping))
> 		goto out;
> @@ -388,7 +388,7 @@ void truncate_inode_pages_range(struct address_space *mapping,
> 		cond_resched();
> 	}
> 
> -	partial_end = ((lend + 1) % PAGE_SIZE) > 0;
> +	partial_end = ((lend + 1) % PAGE_SIZE) != 0;
> 	folio = __filemap_get_folio(mapping, lstart >> PAGE_SHIFT, FGP_LOCK, 0);
> 	if (folio) {
> 		bool same_folio = lend < folio_pos(folio) + folio_size(folio);
> 

Re: [PATCH 00/48] Folios for 5.17

[Matthew Wilcox]

Sorry I missed this while travelling.

On Sun, Dec 26, 2021 at 10:26:23PM +0000, William Kucharski wrote:
> Consolidated multiple review comments into one email, the majority are nits at
> best:
> 
> [PATCH 04/48]:
> 
> An obnoxiously pendantic English grammar nit:
> 
> + * lock).  This can also be called from mark_buffer_dirty(), which I
> 
> The period should be inside the paren, e.g.: "lock.)"

That's at least debatable.  The full context here is:

 [...] A few have the folio blocked from truncation through other
+ * means (eg zap_page_range() has it mapped and is holding the page table
+ * lock).

According to AP Style, the period goes outside the paren in this case:
https://blog.apastyle.org/apastyle/2013/03/punctuation-junction-periods-and-parentheses.html

I'm sure you can find an authority to support always placing a period
inside a paren, but we don't have a controlling authority for how to
punctuate our documentation.  I'm great fun at parties when I get going
on the subject of the Oxford comma.

> [PATCH 05/48]:
> 
> +       unsigned char aux[3];
> 
> I'd like to see an explanation of why this is "3."

I got rid of it ... for now ;-)

> +static inline void folio_batch_init(struct folio_batch *fbatch)
> +{
> +       fbatch->nr = 0;
> +}
> +
> +static inline unsigned int folio_batch_count(struct folio_batch *fbatch)
> +{
> +       return fbatch->nr;
> +}
> +
> +static inline unsigned int fbatch_space(struct folio_batch *fbatch)
> +{
> +       return PAGEVEC_SIZE - fbatch->nr;
> +}
> +
> +/**
> + * folio_batch_add() - Add a folio to a batch.
> + * @fbatch: The folio batch.
> + * @folio: The folio to add.
> + *
> + * The folio is added to the end of the batch.
> + * The batch must have previously been initialised using folio_batch_init().
> + *
> + * Return: The number of slots still available.
> + */
> +static inline unsigned folio_batch_add(struct folio_batch *fbatch,
> +               struct folio *folio)
> +{
> +       fbatch->folios[fbatch->nr++] = folio;
> 
> Is there any need to validate fbatch in these inlines?

I don't think so?  At least, there's no validation for the pagevec
equivalents.  I'd be open to adding something cheap if it's likely to
catch a bug someone's likely to introduce.

> [PATCH 07/48]:
> 
> +       xas_for_each(&xas, folio, ULONG_MAX) {                  \
>                 unsigned left;                                  \
> -               if (xas_retry(&xas, head))                      \
> +               size_t offset = offset_in_folio(folio, start + __off);  \
> +               if (xas_retry(&xas, folio))                     \
>                         continue;                               \
> -               if (WARN_ON(xa_is_value(head)))                 \
> +               if (WARN_ON(xa_is_value(folio)))                \
>                         break;                                  \
> -               if (WARN_ON(PageHuge(head)))                    \
> +               if (WARN_ON(folio_test_hugetlb(folio)))         \
>                         break;                                  \
> -               for (j = (head->index < index) ? index - head->index : 0; \
> -                    j < thp_nr_pages(head); j++) {             \
> -                       void *kaddr = kmap_local_page(head + j);        \
> -                       base = kaddr + offset;                  \
> -                       len = PAGE_SIZE - offset;               \
> +               while (offset < folio_size(folio)) {            \
> 
> Since offset is not actually used until after a bunch of error conditions
> may exit or restart the loop, and isn't used at all in between, defer
> its calculation until just before its first use in the "while."

Hmm.  Those conditions aren't likely to occur, but ... now that you
mention it, checking xa_is_value() after using folio as if it's not a
value is Wrong.  So I'm going to fold in this:

@@ -78,13 +78,14 @@
        rcu_read_lock();                                        \
        xas_for_each(&xas, folio, ULONG_MAX) {                  \
                unsigned left;                                  \
-               size_t offset = offset_in_folio(folio, start + __off);  \
+               size_t offset;                                  \
                if (xas_retry(&xas, folio))                     \
                        continue;                               \
                if (WARN_ON(xa_is_value(folio)))                \
                        break;                                  \
                if (WARN_ON(folio_test_hugetlb(folio)))         \
                        break;                                  \
+               offset = offset_in_folio(folio, start + __off); \
                while (offset < folio_size(folio)) {            \
                        base = kmap_local_folio(folio, offset); \
                        len = min(n, len);                      \

> Reviewed-by: William Kucharski <william.kucharski@oracle.com>

Thanks.  I'll go through and add that in, then push again.

Re: [PATCH 00/48] Folios for 5.17

[Hugh Dickins]

On Sun, 2 Jan 2022, Matthew Wilcox wrote:
> On Wed, Dec 08, 2021 at 04:22:08AM +0000, Matthew Wilcox (Oracle) wrote:
> > This all passes xfstests with no new failures on both xfs and tmpfs.
> > I intend to put all this into for-next tomorrow.
> 
> As a result of Christoph's review, here's the diff.  I don't
> think it's worth re-posting the entire patch series.

Yes, please don't re-post.  I've just spent a few days testing and
fixing the shmem end of it (as I did in Nov 2020 - but I've kept
closer to your intent this time), three patches follow:

 mm/shmem.c    |   58 ++++++++++++++++++++++++++------------------------
 mm/truncate.c |   15 ++++++------
 2 files changed, 38 insertions(+), 35 deletions(-)

Those patches are against next-20211224, not based on your latest diff;
but I think your shmem.c and truncate.c updates can just be replaced by
mine (yes, I too changed "same_page" to "same_folio").

Hugh

Re: [PATCH 00/48] Folios for 5.17

[Matthew Wilcox]

On Sun, Jan 02, 2022 at 04:19:41PM +0000, Matthew Wilcox wrote:
> +++ b/include/linux/uio.h
> @@ -150,7 +150,7 @@ size_t _copy_from_iter_nocache(void *addr, size_t bytes, struct iov_iter *i);
>  static inline size_t copy_folio_to_iter(struct folio *folio, size_t offset,
>  		size_t bytes, struct iov_iter *i)
>  {
> -	return copy_page_to_iter((struct page *)folio, offset, bytes, i);
> +	return copy_page_to_iter(&folio->page, offset, bytes, i);
>  }

Mmmpf.  I should have tested this more thoroughly.  Some files currently
include uio.h without including mm_types.h.  So this on top fixes
the build:

+++ b/include/linux/uio.h
@@ -7,10 +7,10 @@

 #include <linux/kernel.h>
 #include <linux/thread_info.h>
+#include <linux/mm_types.h>
 #include <uapi/linux/uio.h>

 struct page;
-struct folio;
 struct pipe_inode_info;

 struct kvec {

Re: [PATCH 33/48] mm: Add unmap_mapping_folio()

[Christoph Hellwig]

On Sun, Jan 02, 2022 at 04:11:08PM +0000, Matthew Wilcox wrote:
> On Thu, Dec 23, 2021 at 08:36:45AM +0100, Christoph Hellwig wrote:
> > On Wed, Dec 08, 2021 at 04:22:41AM +0000, Matthew Wilcox (Oracle) wrote:
> > > Convert both callers of unmap_mapping_page() to call unmap_mapping_folio()
> > > instead.  Also move zap_details from linux/mm.h to mm/internal.h
> > 
> > In fact it could even move to mm/memory.c as no one needs it outside of
> > that file. __oom_reap_task_mm always passes a NULL zap_details argument
> > to unmap_page_range.
> 
> Umm ... no?
> 
> static inline bool
> zap_skip_check_mapping(struct zap_details *details, struct page *page)
> {
>         if (!details || !page)
>                 return false;
> 
>         return details->zap_mapping &&
>             (details->zap_mapping != page_rmapping(page));
> }

And now check where zap_skip_check_mapping is actually called..

Re: [PATCH 00/48] Folios for 5.17

[Christoph Hellwig]

On Sun, Jan 02, 2022 at 04:19:41PM +0000, Matthew Wilcox wrote:
> On Wed, Dec 08, 2021 at 04:22:08AM +0000, Matthew Wilcox (Oracle) wrote:
> > This all passes xfstests with no new failures on both xfs and tmpfs.
> > I intend to put all this into for-next tomorrow.
> 
> As a result of Christoph's review, here's the diff.  I don't
> think it's worth re-posting the entire patch series.

Thanks, the changes look good to me.

Re: [PATCH 00/48] Folios for 5.17

[William Kucharski]

Thanks.

I like your changes and will defer to you on the period and Oxford comma. :-)

> On Jan 2, 2022, at 18:27, Matthew Wilcox <willy@infradead.org> wrote:
> 
> Sorry I missed this while travelling.
> 
>> On Sun, Dec 26, 2021 at 10:26:23PM +0000, William Kucharski wrote:
>> Consolidated multiple review comments into one email, the majority are nits at
>> best:
>> 
>> [PATCH 04/48]:
>> 
>> An obnoxiously pendantic English grammar nit:
>> 
>> + * lock).  This can also be called from mark_buffer_dirty(), which I
>> 
>> The period should be inside the paren, e.g.: "lock.)"
> 
> That's at least debatable.  The full context here is:
> 
> [...] A few have the folio blocked from truncation through other
> + * means (eg zap_page_range() has it mapped and is holding the page table
> + * lock).
> 
> According to AP Style, the period goes outside the paren in this case:
> https://blog.apastyle.org/apastyle/2013/03/punctuation-junction-periods-and-parentheses.html
> 
> I'm sure you can find an authority to support always placing a period
> inside a paren, but we don't have a controlling authority for how to
> punctuate our documentation.  I'm great fun at parties when I get going
> on the subject of the Oxford comma.
> 
>> [PATCH 05/48]:
>> 
>> +       unsigned char aux[3];
>> 
>> I'd like to see an explanation of why this is "3."
> 
> I got rid of it ... for now ;-)
> 
>> +static inline void folio_batch_init(struct folio_batch *fbatch)
>> +{
>> +       fbatch->nr = 0;
>> +}
>> +
>> +static inline unsigned int folio_batch_count(struct folio_batch *fbatch)
>> +{
>> +       return fbatch->nr;
>> +}
>> +
>> +static inline unsigned int fbatch_space(struct folio_batch *fbatch)
>> +{
>> +       return PAGEVEC_SIZE - fbatch->nr;
>> +}
>> +
>> +/**
>> + * folio_batch_add() - Add a folio to a batch.
>> + * @fbatch: The folio batch.
>> + * @folio: The folio to add.
>> + *
>> + * The folio is added to the end of the batch.
>> + * The batch must have previously been initialised using folio_batch_init().
>> + *
>> + * Return: The number of slots still available.
>> + */
>> +static inline unsigned folio_batch_add(struct folio_batch *fbatch,
>> +               struct folio *folio)
>> +{
>> +       fbatch->folios[fbatch->nr++] = folio;
>> 
>> Is there any need to validate fbatch in these inlines?
> 
> I don't think so?  At least, there's no validation for the pagevec
> equivalents.  I'd be open to adding something cheap if it's likely to
> catch a bug someone's likely to introduce.
> 
>> [PATCH 07/48]:
>> 
>> +       xas_for_each(&xas, folio, ULONG_MAX) {                  \
>>                unsigned left;                                  \
>> -               if (xas_retry(&xas, head))                      \
>> +               size_t offset = offset_in_folio(folio, start + __off);  \
>> +               if (xas_retry(&xas, folio))                     \
>>                        continue;                               \
>> -               if (WARN_ON(xa_is_value(head)))                 \
>> +               if (WARN_ON(xa_is_value(folio)))                \
>>                        break;                                  \
>> -               if (WARN_ON(PageHuge(head)))                    \
>> +               if (WARN_ON(folio_test_hugetlb(folio)))         \
>>                        break;                                  \
>> -               for (j = (head->index < index) ? index - head->index : 0; \
>> -                    j < thp_nr_pages(head); j++) {             \
>> -                       void *kaddr = kmap_local_page(head + j);        \
>> -                       base = kaddr + offset;                  \
>> -                       len = PAGE_SIZE - offset;               \
>> +               while (offset < folio_size(folio)) {            \
>> 
>> Since offset is not actually used until after a bunch of error conditions
>> may exit or restart the loop, and isn't used at all in between, defer
>> its calculation until just before its first use in the "while."
> 
> Hmm.  Those conditions aren't likely to occur, but ... now that you
> mention it, checking xa_is_value() after using folio as if it's not a
> value is Wrong.  So I'm going to fold in this:
> 
> @@ -78,13 +78,14 @@
>        rcu_read_lock();                                        \
>        xas_for_each(&xas, folio, ULONG_MAX) {                  \
>                unsigned left;                                  \
> -               size_t offset = offset_in_folio(folio, start + __off);  \
> +               size_t offset;                                  \
>                if (xas_retry(&xas, folio))                     \
>                        continue;                               \
>                if (WARN_ON(xa_is_value(folio)))                \
>                        break;                                  \
>                if (WARN_ON(folio_test_hugetlb(folio)))         \
>                        break;                                  \
> +               offset = offset_in_folio(folio, start + __off); \
>                while (offset < folio_size(folio)) {            \
>                        base = kmap_local_folio(folio, offset); \
>                        len = min(n, len);                      \
> 
>> Reviewed-by: William Kucharski <william.kucharski@oracle.com>
> 
> Thanks.  I'll go through and add that in, then push again.
> 

Re: [PATCH 00/48] Folios for 5.17

[Matthew Wilcox]

On Sun, Jan 02, 2022 at 04:19:41PM +0000, Matthew Wilcox wrote:
> On Wed, Dec 08, 2021 at 04:22:08AM +0000, Matthew Wilcox (Oracle) wrote:
> > This all passes xfstests with no new failures on both xfs and tmpfs.
> > I intend to put all this into for-next tomorrow.
> 
> As a result of Christoph's review, here's the diff.  I don't
> think it's worth re-posting the entire patch series.

After further review and integrating Hugh's fixes, here's what
I've just updated the for-next tree with.  A little late, but that's
this time of year ...

diff --git a/mm/internal.h b/mm/internal.h
index e989d8ceec91..26af8a5a5be3 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -76,28 +76,7 @@ static inline bool can_madv_lru_vma(struct vm_area_struct *vma)
 	return !(vma->vm_flags & (VM_LOCKED|VM_HUGETLB|VM_PFNMAP));
 }
 
-/*
- * Parameter block passed down to zap_pte_range in exceptional cases.
- */
-struct zap_details {
-	struct address_space *zap_mapping;	/* Check page->mapping if set */
-	struct folio *single_folio;	/* Locked folio to be unmapped */
-};
-
-/*
- * We set details->zap_mappings when we want to unmap shared but keep private
- * pages. Return true if skip zapping this page, false otherwise.
- */
-static inline bool
-zap_skip_check_mapping(struct zap_details *details, struct page *page)
-{
-	if (!details || !page)
-		return false;
-
-	return details->zap_mapping &&
-	    (details->zap_mapping != page_rmapping(page));
-}
-
+struct zap_details;
 void unmap_page_range(struct mmu_gather *tlb,
 			     struct vm_area_struct *vma,
 			     unsigned long addr, unsigned long end,
diff --git a/mm/memory.c b/mm/memory.c
index a86027026f2a..23f2f1300d42 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -1304,6 +1304,28 @@ copy_page_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma)
 	return ret;
 }
 
+/*
+ * Parameter block passed down to zap_pte_range in exceptional cases.
+ */
+struct zap_details {
+	struct address_space *zap_mapping;	/* Check page->mapping if set */
+	struct folio *single_folio;	/* Locked folio to be unmapped */
+};
+
+/*
+ * We set details->zap_mapping when we want to unmap shared but keep private
+ * pages. Return true if skip zapping this page, false otherwise.
+ */
+static inline bool
+zap_skip_check_mapping(struct zap_details *details, struct page *page)
+{
+	if (!details || !page)
+		return false;
+
+	return details->zap_mapping &&
+		(details->zap_mapping != page_rmapping(page));
+}
+
 static unsigned long zap_pte_range(struct mmu_gather *tlb,
 				struct vm_area_struct *vma, pmd_t *pmd,
 				unsigned long addr, unsigned long end,
diff --git a/mm/shmem.c b/mm/shmem.c
index 637de21ff40b..28d627444a24 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -151,19 +151,6 @@ int shmem_getpage(struct inode *inode, pgoff_t index,
 		mapping_gfp_mask(inode->i_mapping), NULL, NULL, NULL);
 }
 
-static int shmem_get_folio(struct inode *inode, pgoff_t index,
-		struct folio **foliop, enum sgp_type sgp)
-{
-	struct page *page = NULL;
-	int ret = shmem_getpage(inode, index, &page, sgp);
-
-	if (page)
-		*foliop = page_folio(page);
-	else
-		*foliop = NULL;
-	return ret;
-}
-
 static inline struct shmem_sb_info *SHMEM_SB(struct super_block *sb)
 {
 	return sb->s_fs_info;
@@ -890,6 +877,28 @@ void shmem_unlock_mapping(struct address_space *mapping)
 	}
 }
 
+static struct folio *shmem_get_partial_folio(struct inode *inode, pgoff_t index)
+{
+	struct folio *folio;
+	struct page *page;
+
+	/*
+	 * At first avoid shmem_getpage(,,,SGP_READ): that fails
+	 * beyond i_size, and reports fallocated pages as holes.
+	 */
+	folio = __filemap_get_folio(inode->i_mapping, index,
+					FGP_ENTRY | FGP_LOCK, 0);
+	if (!xa_is_value(folio))
+		return folio;
+	/*
+	 * But read a page back from swap if any of it is within i_size
+	 * (although in some cases this is just a waste of time).
+	 */
+	page = NULL;
+	shmem_getpage(inode, index, &page, SGP_READ);
+	return page ? page_folio(page) : NULL;
+}
+
 /*
  * Remove range of pages and swap entries from page cache, and free them.
  * If !unfalloc, truncate or punch hole; if unfalloc, undo failed fallocate.
@@ -904,10 +913,10 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, loff_t lend,
 	struct folio_batch fbatch;
 	pgoff_t indices[PAGEVEC_SIZE];
 	struct folio *folio;
+	bool same_folio;
 	long nr_swaps_freed = 0;
 	pgoff_t index;
 	int i;
-	bool partial_end;
 
 	if (lend == -1)
 		end = -1;	/* unsigned, so actually very big */
@@ -943,14 +952,10 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, loff_t lend,
 		index++;
 	}
 
-	partial_end = ((lend + 1) % PAGE_SIZE) != 0;
-	shmem_get_folio(inode, lstart >> PAGE_SHIFT, &folio, SGP_READ);
+	same_folio = (lstart >> PAGE_SHIFT) == (lend >> PAGE_SHIFT);
+	folio = shmem_get_partial_folio(inode, lstart >> PAGE_SHIFT);
 	if (folio) {
-		bool same_folio;
-
 		same_folio = lend < folio_pos(folio) + folio_size(folio);
-		if (same_folio)
-			partial_end = false;
 		folio_mark_dirty(folio);
 		if (!truncate_inode_partial_folio(folio, lstart, lend)) {
 			start = folio->index + folio_nr_pages(folio);
@@ -962,8 +967,8 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, loff_t lend,
 		folio = NULL;
 	}
 
-	if (partial_end)
-		shmem_get_folio(inode, end, &folio, SGP_READ);
+	if (!same_folio)
+		folio = shmem_get_partial_folio(inode, lend >> PAGE_SHIFT);
 	if (folio) {
 		folio_mark_dirty(folio);
 		if (!truncate_inode_partial_folio(folio, lstart, lend))
diff --git a/mm/truncate.c b/mm/truncate.c
index 749aac71fda5..5c87cdc70e7b 100644
--- a/mm/truncate.c
+++ b/mm/truncate.c
@@ -351,7 +351,7 @@ void truncate_inode_pages_range(struct address_space *mapping,
 	pgoff_t		index;
 	int		i;
 	struct folio	*folio;
-	bool		partial_end;
+	bool		same_folio;
 
 	if (mapping_empty(mapping))
 		goto out;
@@ -388,12 +388,10 @@ void truncate_inode_pages_range(struct address_space *mapping,
 		cond_resched();
 	}
 
-	partial_end = ((lend + 1) % PAGE_SIZE) != 0;
+	same_folio = (lstart >> PAGE_SHIFT) == (lend >> PAGE_SHIFT);
 	folio = __filemap_get_folio(mapping, lstart >> PAGE_SHIFT, FGP_LOCK, 0);
 	if (folio) {
-		bool same_folio = lend < folio_pos(folio) + folio_size(folio);
-		if (same_folio)
-			partial_end = false;
+		same_folio = lend < folio_pos(folio) + folio_size(folio);
 		if (!truncate_inode_partial_folio(folio, lstart, lend)) {
 			start = folio->index + folio_nr_pages(folio);
 			if (same_folio)
@@ -404,8 +402,9 @@ void truncate_inode_pages_range(struct address_space *mapping,
 		folio = NULL;
 	}
 
-	if (partial_end)
-		folio = __filemap_get_folio(mapping, end, FGP_LOCK, 0);
+	if (!same_folio)
+		folio = __filemap_get_folio(mapping, lend >> PAGE_SHIFT,
+						FGP_LOCK, 0);
 	if (folio) {
 		if (!truncate_inode_partial_folio(folio, lstart, lend))
 			end = folio->index;

Re: [PATCH 00/48] Folios for 5.17

[Hugh Dickins]

On Sat, 8 Jan 2022, Matthew Wilcox wrote:
> On Sun, Jan 02, 2022 at 04:19:41PM +0000, Matthew Wilcox wrote:
> > On Wed, Dec 08, 2021 at 04:22:08AM +0000, Matthew Wilcox (Oracle) wrote:
> > > This all passes xfstests with no new failures on both xfs and tmpfs.
> > > I intend to put all this into for-next tomorrow.
> > 
> > As a result of Christoph's review, here's the diff.  I don't
> > think it's worth re-posting the entire patch series.
> 
> After further review and integrating Hugh's fixes, here's what
> I've just updated the for-next tree with.  A little late, but that's
> this time of year ...

I don't see any fix to shmem_add_to_page_cache() in this diff, my 3/3
shmem: Fix "Unused swap" messages - I'm not sure whether you decided
my fix has to be adjusted or not, but some fix is needed there.

Hugh

Re: [PATCH 00/48] Folios for 5.17

[Matthew Wilcox]

On Sat, Jan 08, 2022 at 08:47:49AM -0800, Hugh Dickins wrote:
> On Sat, 8 Jan 2022, Matthew Wilcox wrote:
> > On Sun, Jan 02, 2022 at 04:19:41PM +0000, Matthew Wilcox wrote:
> > > On Wed, Dec 08, 2021 at 04:22:08AM +0000, Matthew Wilcox (Oracle) wrote:
> > > > This all passes xfstests with no new failures on both xfs and tmpfs.
> > > > I intend to put all this into for-next tomorrow.
> > > 
> > > As a result of Christoph's review, here's the diff.  I don't
> > > think it's worth re-posting the entire patch series.
> > 
> > After further review and integrating Hugh's fixes, here's what
> > I've just updated the for-next tree with.  A little late, but that's
> > this time of year ...
> 
> I don't see any fix to shmem_add_to_page_cache() in this diff, my 3/3
> shmem: Fix "Unused swap" messages - I'm not sure whether you decided
> my fix has to be adjusted or not, but some fix is needed there.

I pushed that earlier because I had more confidence in my understanding
of that patch.  Here's what's currently in for-next:

@@ -721,20 +720,18 @@ static int shmem_add_to_page_cache(struct page *page,
        cgroup_throttle_swaprate(page, gfp);

        do {
-               void *entry;
                xas_lock_irq(&xas);
-               entry = xas_find_conflict(&xas);
-               if (entry != expected)
+               if (expected != xas_find_conflict(&xas)) {
+                       xas_set_err(&xas, -EEXIST);
+                       goto unlock;
+               }
+               if (expected && xas_find_conflict(&xas)) {
                        xas_set_err(&xas, -EEXIST);
-               xas_create_range(&xas);
-               if (xas_error(&xas))
                        goto unlock;
-next:
-               xas_store(&xas, page);
-               if (++i < nr) {
-                       xas_next(&xas);
-                       goto next;
                }
+               xas_store(&xas, page);
+               if (xas_error(&xas))
+                       goto unlock;
                if (PageTransHuge(page)) {
                        count_vm_event(THP_FILE_ALLOC);
                        __mod_lruvec_page_state(page, NR_SHMEM_THPS, nr);

Re: [PATCH 00/48] Folios for 5.17

[Hugh Dickins]

On Sat, 8 Jan 2022, Matthew Wilcox wrote:
> On Sat, Jan 08, 2022 at 08:47:49AM -0800, Hugh Dickins wrote:
> > On Sat, 8 Jan 2022, Matthew Wilcox wrote:
> > > On Sun, Jan 02, 2022 at 04:19:41PM +0000, Matthew Wilcox wrote:
> > > > On Wed, Dec 08, 2021 at 04:22:08AM +0000, Matthew Wilcox (Oracle) wrote:
> > > > > This all passes xfstests with no new failures on both xfs and tmpfs.
> > > > > I intend to put all this into for-next tomorrow.
> > > > 
> > > > As a result of Christoph's review, here's the diff.  I don't
> > > > think it's worth re-posting the entire patch series.
> > > 
> > > After further review and integrating Hugh's fixes, here's what
> > > I've just updated the for-next tree with.  A little late, but that's
> > > this time of year ...
> > 
> > I don't see any fix to shmem_add_to_page_cache() in this diff, my 3/3
> > shmem: Fix "Unused swap" messages - I'm not sure whether you decided
> > my fix has to be adjusted or not, but some fix is needed there.
> 
> I pushed that earlier because I had more confidence in my understanding
> of that patch.  Here's what's currently in for-next:

Okay, thanks: I tried that variant when you proposed it, and it worked fine.

Hugh