linux-fsdevel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [PATCH v2 00/46] Folio-enabling the page cache
@ 2021-06-22 12:15 Matthew Wilcox (Oracle)
  2021-06-22 12:15 ` [PATCH v2 01/46] mm: Add folio_to_pfn() Matthew Wilcox (Oracle)
                   ` (45 more replies)
  0 siblings, 46 replies; 128+ messages in thread
From: Matthew Wilcox (Oracle) @ 2021-06-22 12:15 UTC (permalink / raw)
  To: akpm; +Cc: Matthew Wilcox (Oracle), linux-fsdevel, linux-mm, linux-kernel

These are all the patches I've collected to date which enable filesystems
to be converted to use folios.  After applying these patches (on top of
folio v12), I have an iomap (ie xfs/zonefs) conversion.  I would expect
filesystems to convert one-by-one, rather than converting all callers of
(say) set_page_writeback() to call folio_start_writeback().

The biggest chunk of this is teaching the writeback code that folios may
be larger than a single page, so there's no (or little) code reduction
from these patches.  Instead it takes us to where we can start preparing
filesystems to see multi-page folios.

Matthew Wilcox (Oracle) (46):
  mm: Add folio_to_pfn()
  mm: Add folio_rmapping()
  mm: Add kmap_local_folio()
  mm: Add flush_dcache_folio()
  mm: Add arch_make_folio_accessible()
  mm: Add folio_young() and folio_idle()
  mm/workingset: Convert workingset_activation to take a folio
  mm/swap: Add folio_activate()
  mm/swap: Add folio_mark_accessed()
  mm/rmap: Add folio_mkclean()
  mm/memcg: Remove 'page' parameter to mem_cgroup_charge_statistics()
  mm/memcg: Use the node id in mem_cgroup_update_tree()
  mm/memcg: Convert commit_charge() to take a folio
  mm/memcg: Add folio_charge_cgroup()
  mm/memcg: Add folio_uncharge_cgroup()
  mm/memcg: Add folio_migrate_cgroup()
  mm/memcg: Convert mem_cgroup_track_foreign_dirty_slowpath() to folio
  mm/migrate: Add folio_migrate_mapping()
  mm/migrate: Add folio_migrate_flags()
  mm/migrate: Add folio_migrate_copy()
  mm/writeback: Rename __add_wb_stat() to wb_stat_mod()
  flex_proportions: Allow N events instead of 1
  mm/writeback: Change __wb_writeout_inc() to __wb_writeout_add()
  mm/writeback: Add __folio_end_writeback()
  mm/writeback: Add folio_start_writeback()
  mm/writeback: Add folio_mark_dirty()
  mm/writeback: Add __folio_mark_dirty()
  mm/writeback: Add filemap_dirty_folio()
  mm/writeback: Add folio_account_cleaned()
  mm/writeback: Add folio_cancel_dirty()
  mm/writeback: Add folio_clear_dirty_for_io()
  mm/writeback: Add folio_account_redirty()
  mm/writeback: Add folio_redirty_for_writepage()
  mm/filemap: Add i_blocks_per_folio()
  mm/filemap: Add folio_mkwrite_check_truncate()
  mm/filemap: Add readahead_folio()
  mm/workingset: Convert workingset_refault() to take a folio
  mm: Add folio_evictable()
  mm/lru: Convert __pagevec_lru_add_fn to take a folio
  mm/lru: Add folio_add_lru()
  mm/page_alloc: Add folio allocation functions
  mm/filemap: Add filemap_alloc_folio
  mm/filemap: Add filemap_add_folio
  mm/filemap: Convert mapping_get_entry to return a folio
  mm/filemap: Add filemap_get_folio
  mm/filemap: Add FGP_STABLE

 .../admin-guide/cgroup-v1/memcg_test.rst      |   2 +-
 Documentation/core-api/cachetlb.rst           |   6 +
 arch/nds32/include/asm/cacheflush.h           |   1 +
 fs/jfs/jfs_metapage.c                         |   1 +
 include/asm-generic/cacheflush.h              |   6 +
 include/linux/backing-dev.h                   |   6 +-
 include/linux/flex_proportions.h              |   9 +-
 include/linux/gfp.h                           |  22 +-
 include/linux/highmem-internal.h              |  11 +
 include/linux/highmem.h                       |  38 ++
 include/linux/ksm.h                           |   4 +-
 include/linux/memcontrol.h                    |  28 +-
 include/linux/migrate.h                       |   4 +
 include/linux/mm.h                            |  51 +--
 include/linux/page-flags.h                    |  20 +-
 include/linux/page_idle.h                     |  99 +++--
 include/linux/page_owner.h                    |   8 +-
 include/linux/pagemap.h                       | 195 ++++++---
 include/linux/rmap.h                          |  10 +-
 include/linux/swap.h                          |  10 +-
 include/linux/writeback.h                     |   9 +-
 include/trace/events/writeback.h              |   8 +-
 kernel/bpf/verifier.c                         |   2 +-
 lib/flex_proportions.c                        |  28 +-
 mm/filemap.c                                  | 240 +++++------
 mm/folio-compat.c                             |  98 +++++
 mm/internal.h                                 |  35 +-
 mm/ksm.c                                      |  31 +-
 mm/memcontrol.c                               | 124 +++---
 mm/memory.c                                   |   3 +-
 mm/mempolicy.c                                |  10 +
 mm/migrate.c                                  | 242 +++++------
 mm/page-writeback.c                           | 383 ++++++++++--------
 mm/page_alloc.c                               |  12 +
 mm/page_owner.c                               |  10 +-
 mm/rmap.c                                     |  12 +-
 mm/shmem.c                                    |   5 +-
 mm/swap.c                                     | 137 ++++---
 mm/swap_state.c                               |   2 +-
 mm/util.c                                     |  33 +-
 mm/workingset.c                               |  44 +-
 41 files changed, 1158 insertions(+), 841 deletions(-)

-- 
2.30.2


^ permalink raw reply	[flat|nested] 128+ messages in thread

* [PATCH v2 01/46] mm: Add folio_to_pfn()
  2021-06-22 12:15 [PATCH v2 00/46] Folio-enabling the page cache Matthew Wilcox (Oracle)
@ 2021-06-22 12:15 ` Matthew Wilcox (Oracle)
  2021-06-23  7:49   ` Christoph Hellwig
  2021-06-22 12:15 ` [PATCH v2 02/46] mm: Add folio_rmapping() Matthew Wilcox (Oracle)
                   ` (44 subsequent siblings)
  45 siblings, 1 reply; 128+ messages in thread
From: Matthew Wilcox (Oracle) @ 2021-06-22 12:15 UTC (permalink / raw)
  To: akpm; +Cc: Matthew Wilcox (Oracle), linux-fsdevel, linux-mm, linux-kernel

The pfn of a folio is the pfn of its head page.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 include/linux/mm.h | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/include/linux/mm.h b/include/linux/mm.h
index 9b7030d1899f..2c7b6ae1d3fc 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -1619,6 +1619,11 @@ static inline unsigned long page_to_section(const struct page *page)
 }
 #endif
 
+static inline unsigned long folio_to_pfn(struct folio *folio)
+{
+	return page_to_pfn(&folio->page);
+}
+
 /* MIGRATE_CMA and ZONE_MOVABLE do not allow pin pages */
 #ifdef CONFIG_MIGRATION
 static inline bool is_pinnable_page(struct page *page)
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 128+ messages in thread

* [PATCH v2 02/46] mm: Add folio_rmapping()
  2021-06-22 12:15 [PATCH v2 00/46] Folio-enabling the page cache Matthew Wilcox (Oracle)
  2021-06-22 12:15 ` [PATCH v2 01/46] mm: Add folio_to_pfn() Matthew Wilcox (Oracle)
@ 2021-06-22 12:15 ` Matthew Wilcox (Oracle)
  2021-06-23  7:56   ` Christoph Hellwig
  2021-06-22 12:15 ` [PATCH v2 03/46] mm: Add kmap_local_folio() Matthew Wilcox (Oracle)
                   ` (43 subsequent siblings)
  45 siblings, 1 reply; 128+ messages in thread
From: Matthew Wilcox (Oracle) @ 2021-06-22 12:15 UTC (permalink / raw)
  To: akpm; +Cc: Matthew Wilcox (Oracle), linux-fsdevel, linux-mm, linux-kernel

Convert __page_rmapping to folio_rmapping and move it to mm/internal.h.
It's only a couple of instructions (load and mask), so it's definitely
going to be cheaper to inline it than call it.  Leave page_rmapping
out of line.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 mm/internal.h |  7 +++++++
 mm/util.c     | 20 ++++----------------
 2 files changed, 11 insertions(+), 16 deletions(-)

diff --git a/mm/internal.h b/mm/internal.h
index 76ddcf55012c..3e70121c71c7 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -34,6 +34,13 @@
 
 void page_writeback_init(void);
 
+static inline void *folio_rmapping(struct folio *folio)
+{
+	unsigned long mapping = (unsigned long)folio->mapping;
+
+	return (void *)(mapping & ~PAGE_MAPPING_FLAGS);
+}
+
 vm_fault_t do_swap_page(struct vm_fault *vmf);
 void folio_rotate_reclaimable(struct folio *folio);
 
diff --git a/mm/util.c b/mm/util.c
index a8766e7f1b7f..0ba3a56c2c90 100644
--- a/mm/util.c
+++ b/mm/util.c
@@ -635,21 +635,10 @@ void kvfree_sensitive(const void *addr, size_t len)
 }
 EXPORT_SYMBOL(kvfree_sensitive);
 
-static inline void *__page_rmapping(struct page *page)
-{
-	unsigned long mapping;
-
-	mapping = (unsigned long)page->mapping;
-	mapping &= ~PAGE_MAPPING_FLAGS;
-
-	return (void *)mapping;
-}
-
 /* Neutral page->mapping pointer to address_space or anon_vma or other */
 void *page_rmapping(struct page *page)
 {
-	page = compound_head(page);
-	return __page_rmapping(page);
+	return folio_rmapping(page_folio(page));
 }
 
 /**
@@ -680,13 +669,12 @@ EXPORT_SYMBOL(folio_mapped);
 
 struct anon_vma *page_anon_vma(struct page *page)
 {
-	unsigned long mapping;
+	struct folio *folio = page_folio(page);
+	unsigned long mapping = (unsigned long)folio->mapping;
 
-	page = compound_head(page);
-	mapping = (unsigned long)page->mapping;
 	if ((mapping & PAGE_MAPPING_FLAGS) != PAGE_MAPPING_ANON)
 		return NULL;
-	return __page_rmapping(page);
+	return folio_rmapping(folio);
 }
 
 /**
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 128+ messages in thread

* [PATCH v2 03/46] mm: Add kmap_local_folio()
  2021-06-22 12:15 [PATCH v2 00/46] Folio-enabling the page cache Matthew Wilcox (Oracle)
  2021-06-22 12:15 ` [PATCH v2 01/46] mm: Add folio_to_pfn() Matthew Wilcox (Oracle)
  2021-06-22 12:15 ` [PATCH v2 02/46] mm: Add folio_rmapping() Matthew Wilcox (Oracle)
@ 2021-06-22 12:15 ` Matthew Wilcox (Oracle)
  2021-06-23  7:58   ` Christoph Hellwig
  2021-06-22 12:15 ` [PATCH v2 04/46] mm: Add flush_dcache_folio() Matthew Wilcox (Oracle)
                   ` (42 subsequent siblings)
  45 siblings, 1 reply; 128+ messages in thread
From: Matthew Wilcox (Oracle) @ 2021-06-22 12:15 UTC (permalink / raw)
  To: akpm; +Cc: Matthew Wilcox (Oracle), linux-fsdevel, linux-mm, linux-kernel

This allows us to map a portion of a folio.  Callers can only expect
to access up to the next page boundary.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 include/linux/highmem-internal.h | 11 +++++++++
 include/linux/highmem.h          | 38 ++++++++++++++++++++++++++++++++
 2 files changed, 49 insertions(+)

diff --git a/include/linux/highmem-internal.h b/include/linux/highmem-internal.h
index 7902c7d8b55f..d5d6f930ae1d 100644
--- a/include/linux/highmem-internal.h
+++ b/include/linux/highmem-internal.h
@@ -73,6 +73,12 @@ static inline void *kmap_local_page(struct page *page)
 	return __kmap_local_page_prot(page, kmap_prot);
 }
 
+static inline void *kmap_local_folio(struct folio *folio, size_t offset)
+{
+	struct page *page = folio_page(folio, offset / PAGE_SIZE);
+	return __kmap_local_page_prot(page, kmap_prot) + offset % PAGE_SIZE;
+}
+
 static inline void *kmap_local_page_prot(struct page *page, pgprot_t prot)
 {
 	return __kmap_local_page_prot(page, prot);
@@ -160,6 +166,11 @@ static inline void *kmap_local_page(struct page *page)
 	return page_address(page);
 }
 
+static inline void *kmap_local_folio(struct folio *folio, size_t offset)
+{
+	return page_address(&folio->page) + offset;
+}
+
 static inline void *kmap_local_page_prot(struct page *page, pgprot_t prot)
 {
 	return kmap_local_page(page);
diff --git a/include/linux/highmem.h b/include/linux/highmem.h
index 832b49b50c7b..c7c664e0315b 100644
--- a/include/linux/highmem.h
+++ b/include/linux/highmem.h
@@ -96,6 +96,44 @@ static inline void kmap_flush_unused(void);
  */
 static inline void *kmap_local_page(struct page *page);
 
+/**
+ * kmap_local_folio - Map a page in this folio for temporary usage
+ * @folio:	The folio to be mapped.
+ * @offset:	The byte offset within the folio.
+ *
+ * Returns: The virtual address of the mapping
+ *
+ * Can be invoked from any context.
+ *
+ * Requires careful handling when nesting multiple mappings because the map
+ * management is stack based. The unmap has to be in the reverse order of
+ * the map operation:
+ *
+ * addr1 = kmap_local_folio(page1, offset1);
+ * addr2 = kmap_local_folio(page2, offset2);
+ * ...
+ * kunmap_local(addr2);
+ * kunmap_local(addr1);
+ *
+ * Unmapping addr1 before addr2 is invalid and causes malfunction.
+ *
+ * Contrary to kmap() mappings the mapping is only valid in the context of
+ * the caller and cannot be handed to other contexts.
+ *
+ * On CONFIG_HIGHMEM=n kernels and for low memory pages this returns the
+ * virtual address of the direct mapping. Only real highmem pages are
+ * temporarily mapped.
+ *
+ * While it is significantly faster than kmap() for the higmem case it
+ * comes with restrictions about the pointer validity. Only use when really
+ * necessary.
+ *
+ * On HIGHMEM enabled systems mapping a highmem page has the side effect of
+ * disabling migration in order to keep the virtual address stable across
+ * preemption. No caller of kmap_local_folio() can rely on this side effect.
+ */
+static inline void *kmap_local_folio(struct folio *folio, size_t offset);
+
 /**
  * kmap_atomic - Atomically map a page for temporary usage - Deprecated!
  * @page:	Pointer to the page to be mapped
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 128+ messages in thread

* [PATCH v2 04/46] mm: Add flush_dcache_folio()
  2021-06-22 12:15 [PATCH v2 00/46] Folio-enabling the page cache Matthew Wilcox (Oracle)
                   ` (2 preceding siblings ...)
  2021-06-22 12:15 ` [PATCH v2 03/46] mm: Add kmap_local_folio() Matthew Wilcox (Oracle)
@ 2021-06-22 12:15 ` Matthew Wilcox (Oracle)
  2021-06-22 12:15 ` [PATCH v2 05/46] mm: Add arch_make_folio_accessible() Matthew Wilcox (Oracle)
                   ` (41 subsequent siblings)
  45 siblings, 0 replies; 128+ messages in thread
From: Matthew Wilcox (Oracle) @ 2021-06-22 12:15 UTC (permalink / raw)
  To: akpm; +Cc: Matthew Wilcox (Oracle), linux-fsdevel, linux-mm, linux-kernel

This is a default implementation which calls flush_dcache_page() on
each page in the folio.  If architectures can do better, they should
implement their own version of it.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 Documentation/core-api/cachetlb.rst |  6 ++++++
 arch/nds32/include/asm/cacheflush.h |  1 +
 include/asm-generic/cacheflush.h    |  6 ++++++
 mm/util.c                           | 13 +++++++++++++
 4 files changed, 26 insertions(+)

diff --git a/Documentation/core-api/cachetlb.rst b/Documentation/core-api/cachetlb.rst
index fe4290e26729..29682f69a915 100644
--- a/Documentation/core-api/cachetlb.rst
+++ b/Documentation/core-api/cachetlb.rst
@@ -325,6 +325,12 @@ maps this page at its virtual address.
 			dirty.  Again, see sparc64 for examples of how
 			to deal with this.
 
+  ``void flush_dcache_folio(struct folio *folio)``
+	This function is called under the same circumstances as
+	flush_dcache_page().  It allows the architecture to
+	optimise for flushing the entire folio of pages instead
+	of flushing one page at a time.
+
   ``void copy_to_user_page(struct vm_area_struct *vma, struct page *page,
   unsigned long user_vaddr, void *dst, void *src, int len)``
   ``void copy_from_user_page(struct vm_area_struct *vma, struct page *page,
diff --git a/arch/nds32/include/asm/cacheflush.h b/arch/nds32/include/asm/cacheflush.h
index 7d6824f7c0e8..f10d13af4ae5 100644
--- a/arch/nds32/include/asm/cacheflush.h
+++ b/arch/nds32/include/asm/cacheflush.h
@@ -38,6 +38,7 @@ void flush_anon_page(struct vm_area_struct *vma,
 
 #define ARCH_HAS_FLUSH_KERNEL_DCACHE_PAGE
 void flush_kernel_dcache_page(struct page *page);
+void flush_dcache_folio(struct folio *folio);
 void flush_kernel_vmap_range(void *addr, int size);
 void invalidate_kernel_vmap_range(void *addr, int size);
 #define flush_dcache_mmap_lock(mapping)   xa_lock_irq(&(mapping)->i_pages)
diff --git a/include/asm-generic/cacheflush.h b/include/asm-generic/cacheflush.h
index 4a674db4e1fa..fedc0dfa4877 100644
--- a/include/asm-generic/cacheflush.h
+++ b/include/asm-generic/cacheflush.h
@@ -49,9 +49,15 @@ static inline void flush_cache_page(struct vm_area_struct *vma,
 static inline void flush_dcache_page(struct page *page)
 {
 }
+
+static inline void flush_dcache_folio(struct folio *folio) { }
 #define ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE 0
+#define ARCH_IMPLEMENTS_FLUSH_DCACHE_FOLIO
 #endif
 
+#ifndef ARCH_IMPLEMENTS_FLUSH_DCACHE_FOLIO
+void flush_dcache_folio(struct folio *folio);
+#endif
 
 #ifndef flush_dcache_mmap_lock
 static inline void flush_dcache_mmap_lock(struct address_space *mapping)
diff --git a/mm/util.c b/mm/util.c
index 0ba3a56c2c90..cae22295c41f 100644
--- a/mm/util.c
+++ b/mm/util.c
@@ -1007,3 +1007,16 @@ void mem_dump_obj(void *object)
 }
 EXPORT_SYMBOL_GPL(mem_dump_obj);
 #endif
+
+#ifndef ARCH_IMPLEMENTS_FLUSH_DCACHE_FOLIO
+void flush_dcache_folio(struct folio *folio)
+{
+	unsigned int n = folio_nr_pages(folio);
+
+	do {
+		n--;
+		flush_dcache_page(folio_page(folio, n));
+	} while (n);
+}
+EXPORT_SYMBOL(flush_dcache_folio);
+#endif
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 128+ messages in thread

* [PATCH v2 05/46] mm: Add arch_make_folio_accessible()
  2021-06-22 12:15 [PATCH v2 00/46] Folio-enabling the page cache Matthew Wilcox (Oracle)
                   ` (3 preceding siblings ...)
  2021-06-22 12:15 ` [PATCH v2 04/46] mm: Add flush_dcache_folio() Matthew Wilcox (Oracle)
@ 2021-06-22 12:15 ` Matthew Wilcox (Oracle)
  2021-06-23  8:00   ` Christoph Hellwig
  2021-06-22 12:15 ` [PATCH v2 06/46] mm: Add folio_young() and folio_idle() Matthew Wilcox (Oracle)
                   ` (40 subsequent siblings)
  45 siblings, 1 reply; 128+ messages in thread
From: Matthew Wilcox (Oracle) @ 2021-06-22 12:15 UTC (permalink / raw)
  To: akpm; +Cc: Matthew Wilcox (Oracle), linux-fsdevel, linux-mm, linux-kernel

As a default implementation, call arch_make_page_accessible n times.
If an architecture can do better, it can override this.

Also move the default implementation of arch_make_page_accessible()
from gfp.h to mm.h.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 include/linux/gfp.h |  6 ------
 include/linux/mm.h  | 21 +++++++++++++++++++++
 2 files changed, 21 insertions(+), 6 deletions(-)

diff --git a/include/linux/gfp.h b/include/linux/gfp.h
index 11da8af06704..a503d928e684 100644
--- a/include/linux/gfp.h
+++ b/include/linux/gfp.h
@@ -508,12 +508,6 @@ static inline void arch_free_page(struct page *page, int order) { }
 #ifndef HAVE_ARCH_ALLOC_PAGE
 static inline void arch_alloc_page(struct page *page, int order) { }
 #endif
-#ifndef HAVE_ARCH_MAKE_PAGE_ACCESSIBLE
-static inline int arch_make_page_accessible(struct page *page)
-{
-	return 0;
-}
-#endif
 
 struct page *__alloc_pages(gfp_t gfp, unsigned int order, int preferred_nid,
 		nodemask_t *nodemask);
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 2c7b6ae1d3fc..5609095ffcac 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -1719,6 +1719,27 @@ static inline size_t folio_size(struct folio *folio)
 	return PAGE_SIZE << folio_order(folio);
 }
 
+#ifndef HAVE_ARCH_MAKE_PAGE_ACCESSIBLE
+static inline int arch_make_page_accessible(struct page *page)
+{
+	return 0;
+}
+#endif
+
+#ifndef HAVE_ARCH_MAKE_FOLIO_ACCESSIBLE
+static inline int arch_make_folio_accessible(struct folio *folio)
+{
+	int ret, i;
+	for (i = 0; i < folio_nr_pages(folio); i++) {
+		ret = arch_make_page_accessible(folio_page(folio, i));
+		if (ret)
+			break;
+	}
+
+	return ret;
+}
+#endif
+
 /*
  * Some inline functions in vmstat.h depend on page_zone()
  */
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 128+ messages in thread

* [PATCH v2 06/46] mm: Add folio_young() and folio_idle()
  2021-06-22 12:15 [PATCH v2 00/46] Folio-enabling the page cache Matthew Wilcox (Oracle)
                   ` (4 preceding siblings ...)
  2021-06-22 12:15 ` [PATCH v2 05/46] mm: Add arch_make_folio_accessible() Matthew Wilcox (Oracle)
@ 2021-06-22 12:15 ` Matthew Wilcox (Oracle)
  2021-06-22 12:15 ` [PATCH v2 07/46] mm/workingset: Convert workingset_activation to take a folio Matthew Wilcox (Oracle)
                   ` (39 subsequent siblings)
  45 siblings, 0 replies; 128+ messages in thread
From: Matthew Wilcox (Oracle) @ 2021-06-22 12:15 UTC (permalink / raw)
  To: akpm
  Cc: Matthew Wilcox (Oracle),
	linux-fsdevel, linux-mm, linux-kernel, Vlastimil Babka,
	William Kucharski, Christoph Hellwig

Idle page tracking is handled through page_ext on 32-bit architectures.
Add folio equivalents for 32-bit and move all the page compatibility
parts to common code.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
Acked-by: Vlastimil Babka <vbabka@suse.cz>
Reviewed-by: William Kucharski <william.kucharski@oracle.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
---
 include/linux/page_idle.h | 99 +++++++++++++++++++--------------------
 1 file changed, 49 insertions(+), 50 deletions(-)

diff --git a/include/linux/page_idle.h b/include/linux/page_idle.h
index 1e894d34bdce..bd957e818558 100644
--- a/include/linux/page_idle.h
+++ b/include/linux/page_idle.h
@@ -8,46 +8,16 @@
 
 #ifdef CONFIG_IDLE_PAGE_TRACKING
 
-#ifdef CONFIG_64BIT
-static inline bool page_is_young(struct page *page)
-{
-	return PageYoung(page);
-}
-
-static inline void set_page_young(struct page *page)
-{
-	SetPageYoung(page);
-}
-
-static inline bool test_and_clear_page_young(struct page *page)
-{
-	return TestClearPageYoung(page);
-}
-
-static inline bool page_is_idle(struct page *page)
-{
-	return PageIdle(page);
-}
-
-static inline void set_page_idle(struct page *page)
-{
-	SetPageIdle(page);
-}
-
-static inline void clear_page_idle(struct page *page)
-{
-	ClearPageIdle(page);
-}
-#else /* !CONFIG_64BIT */
+#ifndef CONFIG_64BIT
 /*
  * If there is not enough space to store Idle and Young bits in page flags, use
  * page ext flags instead.
  */
 extern struct page_ext_operations page_idle_ops;
 
-static inline bool page_is_young(struct page *page)
+static inline bool folio_young(struct folio *folio)
 {
-	struct page_ext *page_ext = lookup_page_ext(page);
+	struct page_ext *page_ext = lookup_page_ext(&folio->page);
 
 	if (unlikely(!page_ext))
 		return false;
@@ -55,9 +25,9 @@ static inline bool page_is_young(struct page *page)
 	return test_bit(PAGE_EXT_YOUNG, &page_ext->flags);
 }
 
-static inline void set_page_young(struct page *page)
+static inline void folio_set_young_flag(struct folio *folio)
 {
-	struct page_ext *page_ext = lookup_page_ext(page);
+	struct page_ext *page_ext = lookup_page_ext(&folio->page);
 
 	if (unlikely(!page_ext))
 		return;
@@ -65,9 +35,9 @@ static inline void set_page_young(struct page *page)
 	set_bit(PAGE_EXT_YOUNG, &page_ext->flags);
 }
 
-static inline bool test_and_clear_page_young(struct page *page)
+static inline bool folio_test_clear_young_flag(struct folio *folio)
 {
-	struct page_ext *page_ext = lookup_page_ext(page);
+	struct page_ext *page_ext = lookup_page_ext(&folio->page);
 
 	if (unlikely(!page_ext))
 		return false;
@@ -75,9 +45,9 @@ static inline bool test_and_clear_page_young(struct page *page)
 	return test_and_clear_bit(PAGE_EXT_YOUNG, &page_ext->flags);
 }
 
-static inline bool page_is_idle(struct page *page)
+static inline bool folio_idle(struct folio *folio)
 {
-	struct page_ext *page_ext = lookup_page_ext(page);
+	struct page_ext *page_ext = lookup_page_ext(&folio->page);
 
 	if (unlikely(!page_ext))
 		return false;
@@ -85,9 +55,9 @@ static inline bool page_is_idle(struct page *page)
 	return test_bit(PAGE_EXT_IDLE, &page_ext->flags);
 }
 
-static inline void set_page_idle(struct page *page)
+static inline void folio_set_idle_flag(struct folio *folio)
 {
-	struct page_ext *page_ext = lookup_page_ext(page);
+	struct page_ext *page_ext = lookup_page_ext(&folio->page);
 
 	if (unlikely(!page_ext))
 		return;
@@ -95,46 +65,75 @@ static inline void set_page_idle(struct page *page)
 	set_bit(PAGE_EXT_IDLE, &page_ext->flags);
 }
 
-static inline void clear_page_idle(struct page *page)
+static inline void folio_clear_idle_flag(struct folio *folio)
 {
-	struct page_ext *page_ext = lookup_page_ext(page);
+	struct page_ext *page_ext = lookup_page_ext(&folio->page);
 
 	if (unlikely(!page_ext))
 		return;
 
 	clear_bit(PAGE_EXT_IDLE, &page_ext->flags);
 }
-#endif /* CONFIG_64BIT */
+#endif /* !CONFIG_64BIT */
 
 #else /* !CONFIG_IDLE_PAGE_TRACKING */
 
-static inline bool page_is_young(struct page *page)
+static inline bool folio_young(struct folio *folio)
 {
 	return false;
 }
 
-static inline void set_page_young(struct page *page)
+static inline void folio_set_young_flag(struct folio *folio)
 {
 }
 
-static inline bool test_and_clear_page_young(struct page *page)
+static inline bool folio_test_clear_young_flag(struct folio *folio)
 {
 	return false;
 }
 
-static inline bool page_is_idle(struct page *page)
+static inline bool folio_idle(struct folio *folio)
 {
 	return false;
 }
 
-static inline void set_page_idle(struct page *page)
+static inline void folio_set_idle_flag(struct folio *folio)
 {
 }
 
-static inline void clear_page_idle(struct page *page)
+static inline void folio_clear_idle_flag(struct folio *folio)
 {
 }
 
 #endif /* CONFIG_IDLE_PAGE_TRACKING */
 
+static inline bool page_is_young(struct page *page)
+{
+	return folio_young(page_folio(page));
+}
+
+static inline void set_page_young(struct page *page)
+{
+	folio_set_young_flag(page_folio(page));
+}
+
+static inline bool test_and_clear_page_young(struct page *page)
+{
+	return folio_test_clear_young_flag(page_folio(page));
+}
+
+static inline bool page_is_idle(struct page *page)
+{
+	return folio_idle(page_folio(page));
+}
+
+static inline void set_page_idle(struct page *page)
+{
+	folio_set_idle_flag(page_folio(page));
+}
+
+static inline void clear_page_idle(struct page *page)
+{
+	folio_clear_idle_flag(page_folio(page));
+}
 #endif /* _LINUX_MM_PAGE_IDLE_H */
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 128+ messages in thread

* [PATCH v2 07/46] mm/workingset: Convert workingset_activation to take a folio
  2021-06-22 12:15 [PATCH v2 00/46] Folio-enabling the page cache Matthew Wilcox (Oracle)
                   ` (5 preceding siblings ...)
  2021-06-22 12:15 ` [PATCH v2 06/46] mm: Add folio_young() and folio_idle() Matthew Wilcox (Oracle)
@ 2021-06-22 12:15 ` Matthew Wilcox (Oracle)
  2021-06-23  8:02   ` Christoph Hellwig
  2021-06-22 12:15 ` [PATCH v2 08/46] mm/swap: Add folio_activate() Matthew Wilcox (Oracle)
                   ` (38 subsequent siblings)
  45 siblings, 1 reply; 128+ messages in thread
From: Matthew Wilcox (Oracle) @ 2021-06-22 12:15 UTC (permalink / raw)
  To: akpm; +Cc: Matthew Wilcox (Oracle), linux-fsdevel, linux-mm, linux-kernel

This function already assumed it was being passed a head page.  No real
change here, except that thp_nr_pages() compiles away on kernels with
THP compiled out while folio_nr_pages() is always present.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 include/linux/swap.h |  2 +-
 mm/swap.c            |  2 +-
 mm/workingset.c      | 10 +++++-----
 3 files changed, 7 insertions(+), 7 deletions(-)

diff --git a/include/linux/swap.h b/include/linux/swap.h
index 76b2338ef24d..8e0118b25bdc 100644
--- a/include/linux/swap.h
+++ b/include/linux/swap.h
@@ -324,7 +324,7 @@ static inline swp_entry_t folio_swap_entry(struct folio *folio)
 void workingset_age_nonresident(struct lruvec *lruvec, unsigned long nr_pages);
 void *workingset_eviction(struct page *page, struct mem_cgroup *target_memcg);
 void workingset_refault(struct page *page, void *shadow);
-void workingset_activation(struct page *page);
+void workingset_activation(struct folio *folio);
 
 /* Only track the nodes of mappings with shadow entries */
 void workingset_update_node(struct xa_node *node);
diff --git a/mm/swap.c b/mm/swap.c
index b01352821583..00d1d781c1c3 100644
--- a/mm/swap.c
+++ b/mm/swap.c
@@ -447,7 +447,7 @@ void mark_page_accessed(struct page *page)
 		else
 			__lru_cache_activate_page(page);
 		ClearPageReferenced(page);
-		workingset_activation(page);
+		workingset_activation(page_folio(page));
 	}
 	if (page_is_idle(page))
 		clear_page_idle(page);
diff --git a/mm/workingset.c b/mm/workingset.c
index b7cdeca5a76d..37cdcda96afd 100644
--- a/mm/workingset.c
+++ b/mm/workingset.c
@@ -390,9 +390,9 @@ void workingset_refault(struct page *page, void *shadow)
 
 /**
  * workingset_activation - note a page activation
- * @page: page that is being activated
+ * @folio: Folio that is being activated.
  */
-void workingset_activation(struct page *page)
+void workingset_activation(struct folio *folio)
 {
 	struct mem_cgroup *memcg;
 	struct lruvec *lruvec;
@@ -405,11 +405,11 @@ void workingset_activation(struct page *page)
 	 * XXX: See workingset_refault() - this should return
 	 * root_mem_cgroup even for !CONFIG_MEMCG.
 	 */
-	memcg = page_memcg_rcu(page);
+	memcg = page_memcg_rcu(&folio->page);
 	if (!mem_cgroup_disabled() && !memcg)
 		goto out;
-	lruvec = mem_cgroup_page_lruvec(page, page_pgdat(page));
-	workingset_age_nonresident(lruvec, thp_nr_pages(page));
+	lruvec = mem_cgroup_folio_lruvec(folio);
+	workingset_age_nonresident(lruvec, folio_nr_pages(folio));
 out:
 	rcu_read_unlock();
 }
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 128+ messages in thread

* [PATCH v2 08/46] mm/swap: Add folio_activate()
  2021-06-22 12:15 [PATCH v2 00/46] Folio-enabling the page cache Matthew Wilcox (Oracle)
                   ` (6 preceding siblings ...)
  2021-06-22 12:15 ` [PATCH v2 07/46] mm/workingset: Convert workingset_activation to take a folio Matthew Wilcox (Oracle)
@ 2021-06-22 12:15 ` Matthew Wilcox (Oracle)
  2021-06-23  8:04   ` Christoph Hellwig
  2021-06-22 12:15 ` [PATCH v2 09/46] mm/swap: Add folio_mark_accessed() Matthew Wilcox (Oracle)
                   ` (37 subsequent siblings)
  45 siblings, 1 reply; 128+ messages in thread
From: Matthew Wilcox (Oracle) @ 2021-06-22 12:15 UTC (permalink / raw)
  To: akpm; +Cc: Matthew Wilcox (Oracle), linux-fsdevel, linux-mm, linux-kernel

This replaces activate_page() and eliminates lots of calls to
compound_head().  Saves net 118 bytes of kernel text.  There are still
some redundant calls to page_folio() here which will be removed when
pagevec_lru_move_fn() is converted to use folios.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 mm/swap.c | 42 +++++++++++++++++++++++-------------------
 1 file changed, 23 insertions(+), 19 deletions(-)

diff --git a/mm/swap.c b/mm/swap.c
index 00d1d781c1c3..1d528c8f1cf4 100644
--- a/mm/swap.c
+++ b/mm/swap.c
@@ -319,15 +319,15 @@ void lru_note_cost_page(struct page *page)
 		      page_is_file_lru(page), thp_nr_pages(page));
 }
 
-static void __activate_page(struct page *page, struct lruvec *lruvec)
+static void __folio_activate(struct folio *folio, struct lruvec *lruvec)
 {
-	if (!PageActive(page) && !PageUnevictable(page)) {
-		int nr_pages = thp_nr_pages(page);
+	if (!folio_active(folio) && !folio_unevictable(folio)) {
+		int nr_pages = folio_nr_pages(folio);
 
-		del_page_from_lru_list(page, lruvec);
-		SetPageActive(page);
-		add_page_to_lru_list(page, lruvec);
-		trace_mm_lru_activate(page);
+		folio_del_from_lru_list(folio, lruvec);
+		folio_set_active_flag(folio);
+		folio_add_to_lru_list(folio, lruvec);
+		trace_mm_lru_activate(&folio->page);
 
 		__count_vm_events(PGACTIVATE, nr_pages);
 		__count_memcg_events(lruvec_memcg(lruvec), PGACTIVATE,
@@ -336,6 +336,11 @@ static void __activate_page(struct page *page, struct lruvec *lruvec)
 }
 
 #ifdef CONFIG_SMP
+static void __activate_page(struct page *page, struct lruvec *lruvec)
+{
+	return __folio_activate(page_folio(page), lruvec);
+}
+
 static void activate_page_drain(int cpu)
 {
 	struct pagevec *pvec = &per_cpu(lru_pvecs.activate_page, cpu);
@@ -349,16 +354,16 @@ static bool need_activate_page_drain(int cpu)
 	return pagevec_count(&per_cpu(lru_pvecs.activate_page, cpu)) != 0;
 }
 
-static void activate_page(struct page *page)
+static void folio_activate(struct folio *folio)
 {
-	page = compound_head(page);
-	if (PageLRU(page) && !PageActive(page) && !PageUnevictable(page)) {
+	if (folio_lru(folio) && !folio_active(folio) &&
+	    !folio_unevictable(folio)) {
 		struct pagevec *pvec;
 
+		folio_get(folio);
 		local_lock(&lru_pvecs.lock);
 		pvec = this_cpu_ptr(&lru_pvecs.activate_page);
-		get_page(page);
-		if (pagevec_add_and_need_flush(pvec, page))
+		if (pagevec_add_and_need_flush(pvec, &folio->page))
 			pagevec_lru_move_fn(pvec, __activate_page);
 		local_unlock(&lru_pvecs.lock);
 	}
@@ -369,16 +374,15 @@ static inline void activate_page_drain(int cpu)
 {
 }
 
-static void activate_page(struct page *page)
+static void folio_activate(struct folio *folio)
 {
 	struct lruvec *lruvec;
 
-	page = compound_head(page);
-	if (TestClearPageLRU(page)) {
-		lruvec = lock_page_lruvec_irq(page);
-		__activate_page(page, lruvec);
+	if (folio_test_clear_lru_flag(folio)) {
+		lruvec = folio_lock_lruvec_irq(folio);
+		__folio_activate(folio, lruvec);
 		unlock_page_lruvec_irq(lruvec);
-		SetPageLRU(page);
+		folio_set_lru_flag(folio);
 	}
 }
 #endif
@@ -443,7 +447,7 @@ void mark_page_accessed(struct page *page)
 		 * LRU on the next drain.
 		 */
 		if (PageLRU(page))
-			activate_page(page);
+			folio_activate(page_folio(page));
 		else
 			__lru_cache_activate_page(page);
 		ClearPageReferenced(page);
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 128+ messages in thread

* [PATCH v2 09/46] mm/swap: Add folio_mark_accessed()
  2021-06-22 12:15 [PATCH v2 00/46] Folio-enabling the page cache Matthew Wilcox (Oracle)
                   ` (7 preceding siblings ...)
  2021-06-22 12:15 ` [PATCH v2 08/46] mm/swap: Add folio_activate() Matthew Wilcox (Oracle)
@ 2021-06-22 12:15 ` Matthew Wilcox (Oracle)
  2021-06-23  8:07   ` Christoph Hellwig
  2021-06-22 12:15 ` [PATCH v2 10/46] mm/rmap: Add folio_mkclean() Matthew Wilcox (Oracle)
                   ` (36 subsequent siblings)
  45 siblings, 1 reply; 128+ messages in thread
From: Matthew Wilcox (Oracle) @ 2021-06-22 12:15 UTC (permalink / raw)
  To: akpm; +Cc: Matthew Wilcox (Oracle), linux-fsdevel, linux-mm, linux-kernel

Convert mark_page_accessed() to folio_mark_accessed().  It already
operated on the entire compound page, but now we can avoid calling
compound_head quite so many times.  Shrinks the function from 424 bytes
to 295 bytes (shrinking by 129 bytes).  The compatibility wrapper is 30
bytes, plus the 8 bytes for the exported symbol means the kernel shrinks
by 91 bytes.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 include/linux/swap.h |  3 ++-
 mm/folio-compat.c    |  7 +++++++
 mm/swap.c            | 34 ++++++++++++++++------------------
 3 files changed, 25 insertions(+), 19 deletions(-)

diff --git a/include/linux/swap.h b/include/linux/swap.h
index 8e0118b25bdc..d1cb67cdb476 100644
--- a/include/linux/swap.h
+++ b/include/linux/swap.h
@@ -346,7 +346,8 @@ extern void lru_note_cost(struct lruvec *lruvec, bool file,
 			  unsigned int nr_pages);
 extern void lru_note_cost_page(struct page *);
 extern void lru_cache_add(struct page *);
-extern void mark_page_accessed(struct page *);
+void mark_page_accessed(struct page *);
+void folio_mark_accessed(struct folio *);
 
 extern atomic_t lru_disable_count;
 
diff --git a/mm/folio-compat.c b/mm/folio-compat.c
index 7044fcc8a8aa..a374747ae1c6 100644
--- a/mm/folio-compat.c
+++ b/mm/folio-compat.c
@@ -5,6 +5,7 @@
  */
 
 #include <linux/pagemap.h>
+#include <linux/swap.h>
 
 struct address_space *page_mapping(struct page *page)
 {
@@ -41,3 +42,9 @@ bool page_mapped(struct page *page)
 	return folio_mapped(page_folio(page));
 }
 EXPORT_SYMBOL(page_mapped);
+
+void mark_page_accessed(struct page *page)
+{
+	folio_mark_accessed(page_folio(page));
+}
+EXPORT_SYMBOL(mark_page_accessed);
diff --git a/mm/swap.c b/mm/swap.c
index 1d528c8f1cf4..53422f6b7db1 100644
--- a/mm/swap.c
+++ b/mm/swap.c
@@ -387,7 +387,7 @@ static void folio_activate(struct folio *folio)
 }
 #endif
 
-static void __lru_cache_activate_page(struct page *page)
+static void __lru_cache_activate_folio(struct folio *folio)
 {
 	struct pagevec *pvec;
 	int i;
@@ -408,8 +408,8 @@ static void __lru_cache_activate_page(struct page *page)
 	for (i = pagevec_count(pvec) - 1; i >= 0; i--) {
 		struct page *pagevec_page = pvec->pages[i];
 
-		if (pagevec_page == page) {
-			SetPageActive(page);
+		if (pagevec_page == &folio->page) {
+			folio_set_active_flag(folio);
 			break;
 		}
 	}
@@ -427,36 +427,34 @@ static void __lru_cache_activate_page(struct page *page)
  * When a newly allocated page is not yet visible, so safe for non-atomic ops,
  * __SetPageReferenced(page) may be substituted for mark_page_accessed(page).
  */
-void mark_page_accessed(struct page *page)
+void folio_mark_accessed(struct folio *folio)
 {
-	page = compound_head(page);
-
-	if (!PageReferenced(page)) {
-		SetPageReferenced(page);
-	} else if (PageUnevictable(page)) {
+	if (!folio_referenced(folio)) {
+		folio_set_referenced_flag(folio);
+	} else if (folio_unevictable(folio)) {
 		/*
 		 * Unevictable pages are on the "LRU_UNEVICTABLE" list. But,
 		 * this list is never rotated or maintained, so marking an
 		 * evictable page accessed has no effect.
 		 */
-	} else if (!PageActive(page)) {
+	} else if (!folio_active(folio)) {
 		/*
 		 * If the page is on the LRU, queue it for activation via
 		 * lru_pvecs.activate_page. Otherwise, assume the page is on a
 		 * pagevec, mark it active and it'll be moved to the active
 		 * LRU on the next drain.
 		 */
-		if (PageLRU(page))
-			folio_activate(page_folio(page));
+		if (folio_lru(folio))
+			folio_activate(folio);
 		else
-			__lru_cache_activate_page(page);
-		ClearPageReferenced(page);
-		workingset_activation(page_folio(page));
+			__lru_cache_activate_folio(folio);
+		folio_clear_referenced_flag(folio);
+		workingset_activation(folio);
 	}
-	if (page_is_idle(page))
-		clear_page_idle(page);
+	if (folio_idle(folio))
+		folio_clear_idle_flag(folio);
 }
-EXPORT_SYMBOL(mark_page_accessed);
+EXPORT_SYMBOL(folio_mark_accessed);
 
 /**
  * lru_cache_add - add a page to a page list
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 128+ messages in thread

* [PATCH v2 10/46] mm/rmap: Add folio_mkclean()
  2021-06-22 12:15 [PATCH v2 00/46] Folio-enabling the page cache Matthew Wilcox (Oracle)
                   ` (8 preceding siblings ...)
  2021-06-22 12:15 ` [PATCH v2 09/46] mm/swap: Add folio_mark_accessed() Matthew Wilcox (Oracle)
@ 2021-06-22 12:15 ` Matthew Wilcox (Oracle)
  2021-06-23  8:08   ` Christoph Hellwig
  2021-06-22 12:15 ` [PATCH v2 11/46] mm/memcg: Remove 'page' parameter to mem_cgroup_charge_statistics() Matthew Wilcox (Oracle)
                   ` (35 subsequent siblings)
  45 siblings, 1 reply; 128+ messages in thread
From: Matthew Wilcox (Oracle) @ 2021-06-22 12:15 UTC (permalink / raw)
  To: akpm; +Cc: Matthew Wilcox (Oracle), linux-fsdevel, linux-mm, linux-kernel

Transform page_mkclean() into folio_mkclean() and add a page_mkclean()
wrapper around folio_mkclean().

folio_mkclean is 15 bytes smaller than page_mkclean, but the kernel
is enlarged by 33 bytes due to inlining page_folio() into each caller.
This will go away once the callers are converted to use folio_mkclean().

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 include/linux/rmap.h | 10 ++++++----
 mm/rmap.c            | 12 ++++++------
 2 files changed, 12 insertions(+), 10 deletions(-)

diff --git a/include/linux/rmap.h b/include/linux/rmap.h
index def5c62c93b3..edb006bc4159 100644
--- a/include/linux/rmap.h
+++ b/include/linux/rmap.h
@@ -233,7 +233,7 @@ unsigned long page_address_in_vma(struct page *, struct vm_area_struct *);
  *
  * returns the number of cleaned PTEs.
  */
-int page_mkclean(struct page *);
+int folio_mkclean(struct folio *);
 
 /*
  * called in munlock()/munmap() path to check for other vmas holding
@@ -291,12 +291,14 @@ static inline int page_referenced(struct page *page, int is_locked,
 
 #define try_to_unmap(page, refs) false
 
-static inline int page_mkclean(struct page *page)
+static inline int folio_mkclean(struct folio *folio)
 {
 	return 0;
 }
-
-
 #endif	/* CONFIG_MMU */
 
+static inline int page_mkclean(struct page *page)
+{
+	return folio_mkclean(page_folio(page));
+}
 #endif	/* _LINUX_RMAP_H */
diff --git a/mm/rmap.c b/mm/rmap.c
index 693a610e181d..e29dbbc880d7 100644
--- a/mm/rmap.c
+++ b/mm/rmap.c
@@ -983,7 +983,7 @@ static bool invalid_mkclean_vma(struct vm_area_struct *vma, void *arg)
 	return true;
 }
 
-int page_mkclean(struct page *page)
+int folio_mkclean(struct folio *folio)
 {
 	int cleaned = 0;
 	struct address_space *mapping;
@@ -993,20 +993,20 @@ int page_mkclean(struct page *page)
 		.invalid_vma = invalid_mkclean_vma,
 	};
 
-	BUG_ON(!PageLocked(page));
+	BUG_ON(!folio_locked(folio));
 
-	if (!page_mapped(page))
+	if (!folio_mapped(folio))
 		return 0;
 
-	mapping = page_mapping(page);
+	mapping = folio_mapping(folio);
 	if (!mapping)
 		return 0;
 
-	rmap_walk(page, &rwc);
+	rmap_walk(&folio->page, &rwc);
 
 	return cleaned;
 }
-EXPORT_SYMBOL_GPL(page_mkclean);
+EXPORT_SYMBOL_GPL(folio_mkclean);
 
 /**
  * page_move_anon_rmap - move a page to our anon_vma
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 128+ messages in thread

* [PATCH v2 11/46] mm/memcg: Remove 'page' parameter to mem_cgroup_charge_statistics()
  2021-06-22 12:15 [PATCH v2 00/46] Folio-enabling the page cache Matthew Wilcox (Oracle)
                   ` (9 preceding siblings ...)
  2021-06-22 12:15 ` [PATCH v2 10/46] mm/rmap: Add folio_mkclean() Matthew Wilcox (Oracle)
@ 2021-06-22 12:15 ` Matthew Wilcox (Oracle)
  2021-06-23  8:09   ` Christoph Hellwig
  2021-06-25  7:58   ` Michal Hocko
  2021-06-22 12:15 ` [PATCH v2 12/46] mm/memcg: Use the node id in mem_cgroup_update_tree() Matthew Wilcox (Oracle)
                   ` (34 subsequent siblings)
  45 siblings, 2 replies; 128+ messages in thread
From: Matthew Wilcox (Oracle) @ 2021-06-22 12:15 UTC (permalink / raw)
  To: akpm; +Cc: Matthew Wilcox (Oracle), linux-fsdevel, linux-mm, linux-kernel

The last use of 'page' was removed by commit 468c398233da ("mm:
memcontrol: switch to native NR_ANON_THPS counter"), so we can now remove
the parameter from the function.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 mm/memcontrol.c | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)

diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 64ada9e650a5..1204c6a0c671 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -814,7 +814,6 @@ static unsigned long memcg_events_local(struct mem_cgroup *memcg, int event)
 }
 
 static void mem_cgroup_charge_statistics(struct mem_cgroup *memcg,
-					 struct page *page,
 					 int nr_pages)
 {
 	/* pagein of a big page is an event. So, ignore page size */
@@ -5504,9 +5503,9 @@ static int mem_cgroup_move_account(struct page *page,
 	ret = 0;
 
 	local_irq_disable();
-	mem_cgroup_charge_statistics(to, page, nr_pages);
+	mem_cgroup_charge_statistics(to, nr_pages);
 	memcg_check_events(to, page);
-	mem_cgroup_charge_statistics(from, page, -nr_pages);
+	mem_cgroup_charge_statistics(from, -nr_pages);
 	memcg_check_events(from, page);
 	local_irq_enable();
 out_unlock:
@@ -6527,7 +6526,7 @@ static int __mem_cgroup_charge(struct page *page, struct mem_cgroup *memcg,
 	commit_charge(page, memcg);
 
 	local_irq_disable();
-	mem_cgroup_charge_statistics(memcg, page, nr_pages);
+	mem_cgroup_charge_statistics(memcg, nr_pages);
 	memcg_check_events(memcg, page);
 	local_irq_enable();
 out:
@@ -6814,7 +6813,7 @@ void mem_cgroup_migrate(struct page *oldpage, struct page *newpage)
 	commit_charge(newpage, memcg);
 
 	local_irq_save(flags);
-	mem_cgroup_charge_statistics(memcg, newpage, nr_pages);
+	mem_cgroup_charge_statistics(memcg, nr_pages);
 	memcg_check_events(memcg, newpage);
 	local_irq_restore(flags);
 }
@@ -7044,7 +7043,7 @@ void mem_cgroup_swapout(struct page *page, swp_entry_t entry)
 	 * only synchronisation we have for updating the per-CPU variables.
 	 */
 	VM_BUG_ON(!irqs_disabled());
-	mem_cgroup_charge_statistics(memcg, page, -nr_entries);
+	mem_cgroup_charge_statistics(memcg, -nr_entries);
 	memcg_check_events(memcg, page);
 
 	css_put(&memcg->css);
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 128+ messages in thread

* [PATCH v2 12/46] mm/memcg: Use the node id in mem_cgroup_update_tree()
  2021-06-22 12:15 [PATCH v2 00/46] Folio-enabling the page cache Matthew Wilcox (Oracle)
                   ` (10 preceding siblings ...)
  2021-06-22 12:15 ` [PATCH v2 11/46] mm/memcg: Remove 'page' parameter to mem_cgroup_charge_statistics() Matthew Wilcox (Oracle)
@ 2021-06-22 12:15 ` Matthew Wilcox (Oracle)
  2021-06-23  8:12   ` Christoph Hellwig
  2021-06-22 12:15 ` [PATCH v2 13/46] mm/memcg: Convert commit_charge() to take a folio Matthew Wilcox (Oracle)
                   ` (33 subsequent siblings)
  45 siblings, 1 reply; 128+ messages in thread
From: Matthew Wilcox (Oracle) @ 2021-06-22 12:15 UTC (permalink / raw)
  To: akpm; +Cc: Matthew Wilcox (Oracle), linux-fsdevel, linux-mm, linux-kernel

Hoist the page_to_nid() call from mem_cgroup_page_nodeinfo() into
mem_cgroup_update_tree().  That lets us call soft_limit_tree_node()
and delete soft_limit_tree_from_page() altogether.  Saves 42
bytes of kernel text on my config.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 mm/memcontrol.c | 17 ++++-------------
 1 file changed, 4 insertions(+), 13 deletions(-)

diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 1204c6a0c671..7423cb11eb88 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -453,10 +453,8 @@ ino_t page_cgroup_ino(struct page *page)
 }
 
 static struct mem_cgroup_per_node *
-mem_cgroup_page_nodeinfo(struct mem_cgroup *memcg, struct page *page)
+mem_cgroup_nodeinfo(struct mem_cgroup *memcg, int nid)
 {
-	int nid = page_to_nid(page);
-
 	return memcg->nodeinfo[nid];
 }
 
@@ -466,14 +464,6 @@ soft_limit_tree_node(int nid)
 	return soft_limit_tree.rb_tree_per_node[nid];
 }
 
-static struct mem_cgroup_tree_per_node *
-soft_limit_tree_from_page(struct page *page)
-{
-	int nid = page_to_nid(page);
-
-	return soft_limit_tree.rb_tree_per_node[nid];
-}
-
 static void __mem_cgroup_insert_exceeded(struct mem_cgroup_per_node *mz,
 					 struct mem_cgroup_tree_per_node *mctz,
 					 unsigned long new_usage_in_excess)
@@ -549,8 +539,9 @@ static void mem_cgroup_update_tree(struct mem_cgroup *memcg, struct page *page)
 	unsigned long excess;
 	struct mem_cgroup_per_node *mz;
 	struct mem_cgroup_tree_per_node *mctz;
+	int nid = page_to_nid(page);
 
-	mctz = soft_limit_tree_from_page(page);
+	mctz = soft_limit_tree_node(nid);
 	if (!mctz)
 		return;
 	/*
@@ -558,7 +549,7 @@ static void mem_cgroup_update_tree(struct mem_cgroup *memcg, struct page *page)
 	 * because their event counter is not touched.
 	 */
 	for (; memcg; memcg = parent_mem_cgroup(memcg)) {
-		mz = mem_cgroup_page_nodeinfo(memcg, page);
+		mz = mem_cgroup_nodeinfo(memcg, nid);
 		excess = soft_limit_excess(memcg);
 		/*
 		 * We have to update the tree if mz is on RB-tree or
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 128+ messages in thread

* [PATCH v2 13/46] mm/memcg: Convert commit_charge() to take a folio
  2021-06-22 12:15 [PATCH v2 00/46] Folio-enabling the page cache Matthew Wilcox (Oracle)
                   ` (11 preceding siblings ...)
  2021-06-22 12:15 ` [PATCH v2 12/46] mm/memcg: Use the node id in mem_cgroup_update_tree() Matthew Wilcox (Oracle)
@ 2021-06-22 12:15 ` Matthew Wilcox (Oracle)
  2021-06-23  8:13   ` Christoph Hellwig
  2021-06-25  8:11   ` Michal Hocko
  2021-06-22 12:15 ` [PATCH v2 14/46] mm/memcg: Add folio_charge_cgroup() Matthew Wilcox (Oracle)
                   ` (32 subsequent siblings)
  45 siblings, 2 replies; 128+ messages in thread
From: Matthew Wilcox (Oracle) @ 2021-06-22 12:15 UTC (permalink / raw)
  To: akpm; +Cc: Matthew Wilcox (Oracle), linux-fsdevel, linux-mm, linux-kernel

The memcg_data is only set on the head page, so enforce that by
typing it as a folio.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 mm/memcontrol.c | 27 +++++++++++++--------------
 1 file changed, 13 insertions(+), 14 deletions(-)

diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 7423cb11eb88..7939e4e9118d 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -2700,9 +2700,9 @@ static void cancel_charge(struct mem_cgroup *memcg, unsigned int nr_pages)
 }
 #endif
 
-static void commit_charge(struct page *page, struct mem_cgroup *memcg)
+static void commit_charge(struct folio *folio, struct mem_cgroup *memcg)
 {
-	VM_BUG_ON_PAGE(page_memcg(page), page);
+	VM_BUG_ON_FOLIO(folio_memcg(folio), folio);
 	/*
 	 * Any of the following ensures page's memcg stability:
 	 *
@@ -2711,7 +2711,7 @@ static void commit_charge(struct page *page, struct mem_cgroup *memcg)
 	 * - lock_page_memcg()
 	 * - exclusive reference
 	 */
-	page->memcg_data = (unsigned long)memcg;
+	folio->memcg_data = (unsigned long)memcg;
 }
 
 static struct mem_cgroup *get_mem_cgroup_from_objcg(struct obj_cgroup *objcg)
@@ -6506,7 +6506,8 @@ void mem_cgroup_calculate_protection(struct mem_cgroup *root,
 static int __mem_cgroup_charge(struct page *page, struct mem_cgroup *memcg,
 			       gfp_t gfp)
 {
-	unsigned int nr_pages = thp_nr_pages(page);
+	struct folio *folio = page_folio(page);
+	unsigned int nr_pages = folio_nr_pages(folio);
 	int ret;
 
 	ret = try_charge(memcg, gfp, nr_pages);
@@ -6514,7 +6515,7 @@ static int __mem_cgroup_charge(struct page *page, struct mem_cgroup *memcg,
 		goto out;
 
 	css_get(&memcg->css);
-	commit_charge(page, memcg);
+	commit_charge(folio, memcg);
 
 	local_irq_disable();
 	mem_cgroup_charge_statistics(memcg, nr_pages);
@@ -6771,21 +6772,21 @@ void mem_cgroup_uncharge_list(struct list_head *page_list)
  */
 void mem_cgroup_migrate(struct page *oldpage, struct page *newpage)
 {
+	struct folio *newfolio = page_folio(newpage);
 	struct mem_cgroup *memcg;
-	unsigned int nr_pages;
+	unsigned int nr_pages = folio_nr_pages(newfolio);
 	unsigned long flags;
 
 	VM_BUG_ON_PAGE(!PageLocked(oldpage), oldpage);
-	VM_BUG_ON_PAGE(!PageLocked(newpage), newpage);
-	VM_BUG_ON_PAGE(PageAnon(oldpage) != PageAnon(newpage), newpage);
-	VM_BUG_ON_PAGE(PageTransHuge(oldpage) != PageTransHuge(newpage),
-		       newpage);
+	VM_BUG_ON_FOLIO(!folio_locked(newfolio), newfolio);
+	VM_BUG_ON_FOLIO(PageAnon(oldpage) != folio_anon(newfolio), newfolio);
+	VM_BUG_ON_FOLIO(compound_nr(oldpage) != nr_pages, newfolio);
 
 	if (mem_cgroup_disabled())
 		return;
 
 	/* Page cache replacement: new page already charged? */
-	if (page_memcg(newpage))
+	if (folio_memcg(newfolio))
 		return;
 
 	memcg = page_memcg(oldpage);
@@ -6794,14 +6795,12 @@ void mem_cgroup_migrate(struct page *oldpage, struct page *newpage)
 		return;
 
 	/* Force-charge the new page. The old one will be freed soon */
-	nr_pages = thp_nr_pages(newpage);
-
 	page_counter_charge(&memcg->memory, nr_pages);
 	if (do_memsw_account())
 		page_counter_charge(&memcg->memsw, nr_pages);
 
 	css_get(&memcg->css);
-	commit_charge(newpage, memcg);
+	commit_charge(newfolio, memcg);
 
 	local_irq_save(flags);
 	mem_cgroup_charge_statistics(memcg, nr_pages);
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 128+ messages in thread

* [PATCH v2 14/46] mm/memcg: Add folio_charge_cgroup()
  2021-06-22 12:15 [PATCH v2 00/46] Folio-enabling the page cache Matthew Wilcox (Oracle)
                   ` (12 preceding siblings ...)
  2021-06-22 12:15 ` [PATCH v2 13/46] mm/memcg: Convert commit_charge() to take a folio Matthew Wilcox (Oracle)
@ 2021-06-22 12:15 ` Matthew Wilcox (Oracle)
  2021-06-23  8:15   ` Christoph Hellwig
  2021-06-22 12:15 ` [PATCH v2 15/46] mm/memcg: Add folio_uncharge_cgroup() Matthew Wilcox (Oracle)
                   ` (31 subsequent siblings)
  45 siblings, 1 reply; 128+ messages in thread
From: Matthew Wilcox (Oracle) @ 2021-06-22 12:15 UTC (permalink / raw)
  To: akpm; +Cc: Matthew Wilcox (Oracle), linux-fsdevel, linux-mm, linux-kernel

mem_cgroup_charge() already assumed it was being passed a non-tail
page (and looking at the callers, that's true; it's called for freshly
allocated pages).  The only real change here is that folio_nr_pages()
doesn't compile away like thp_nr_pages() does as folio support
is not conditional on transparent hugepage support.  Reimplement
mem_cgroup_charge() as a wrapper around folio_charge_cgroup().

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 include/linux/memcontrol.h |  8 ++++++++
 mm/folio-compat.c          |  7 +++++++
 mm/memcontrol.c            | 26 +++++++++++++-------------
 3 files changed, 28 insertions(+), 13 deletions(-)

diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
index 4460ff0e70a1..a50e5cee6d2c 100644
--- a/include/linux/memcontrol.h
+++ b/include/linux/memcontrol.h
@@ -704,6 +704,8 @@ static inline bool mem_cgroup_below_min(struct mem_cgroup *memcg)
 		page_counter_read(&memcg->memory);
 }
 
+int folio_charge_cgroup(struct folio *, struct mm_struct *, gfp_t);
+
 int mem_cgroup_charge(struct page *page, struct mm_struct *mm, gfp_t gfp_mask);
 int mem_cgroup_swapin_charge_page(struct page *page, struct mm_struct *mm,
 				  gfp_t gfp, swp_entry_t entry);
@@ -1216,6 +1218,12 @@ static inline bool mem_cgroup_below_min(struct mem_cgroup *memcg)
 	return false;
 }
 
+static inline int folio_charge_cgroup(struct folio *folio,
+		struct mm_struct *mm, gfp_t gfp)
+{
+	return 0;
+}
+
 static inline int mem_cgroup_charge(struct page *page, struct mm_struct *mm,
 				    gfp_t gfp_mask)
 {
diff --git a/mm/folio-compat.c b/mm/folio-compat.c
index a374747ae1c6..1d71b8b587f8 100644
--- a/mm/folio-compat.c
+++ b/mm/folio-compat.c
@@ -48,3 +48,10 @@ void mark_page_accessed(struct page *page)
 	folio_mark_accessed(page_folio(page));
 }
 EXPORT_SYMBOL(mark_page_accessed);
+
+#ifdef CONFIG_MEMCG
+int mem_cgroup_charge(struct page *page, struct mm_struct *mm, gfp_t gfp)
+{
+	return folio_charge_cgroup(page_folio(page), mm, gfp);
+}
+#endif
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 7939e4e9118d..69638f84d11b 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -6503,10 +6503,9 @@ void mem_cgroup_calculate_protection(struct mem_cgroup *root,
 			atomic_long_read(&parent->memory.children_low_usage)));
 }
 
-static int __mem_cgroup_charge(struct page *page, struct mem_cgroup *memcg,
+static int __mem_cgroup_charge(struct folio *folio, struct mem_cgroup *memcg,
 			       gfp_t gfp)
 {
-	struct folio *folio = page_folio(page);
 	unsigned int nr_pages = folio_nr_pages(folio);
 	int ret;
 
@@ -6519,26 +6518,26 @@ static int __mem_cgroup_charge(struct page *page, struct mem_cgroup *memcg,
 
 	local_irq_disable();
 	mem_cgroup_charge_statistics(memcg, nr_pages);
-	memcg_check_events(memcg, page);
+	memcg_check_events(memcg, &folio->page);
 	local_irq_enable();
 out:
 	return ret;
 }
 
 /**
- * mem_cgroup_charge - charge a newly allocated page to a cgroup
- * @page: page to charge
- * @mm: mm context of the victim
- * @gfp_mask: reclaim mode
+ * folio_charge_cgroup - Charge a newly allocated folio to a cgroup.
+ * @folio: Folio to charge.
+ * @mm: mm context of the allocating task.
+ * @gfp: reclaim mode
  *
- * Try to charge @page to the memcg that @mm belongs to, reclaiming
- * pages according to @gfp_mask if necessary.
+ * Try to charge @folio to the memcg that @mm belongs to, reclaiming
+ * pages according to @gfp if necessary.
  *
- * Do not use this for pages allocated for swapin.
+ * Do not use this for folios allocated for swapin.
  *
  * Returns 0 on success. Otherwise, an error code is returned.
  */
-int mem_cgroup_charge(struct page *page, struct mm_struct *mm, gfp_t gfp_mask)
+int folio_charge_cgroup(struct folio *folio, struct mm_struct *mm, gfp_t gfp)
 {
 	struct mem_cgroup *memcg;
 	int ret;
@@ -6547,7 +6546,7 @@ int mem_cgroup_charge(struct page *page, struct mm_struct *mm, gfp_t gfp_mask)
 		return 0;
 
 	memcg = get_mem_cgroup_from_mm(mm);
-	ret = __mem_cgroup_charge(page, memcg, gfp_mask);
+	ret = __mem_cgroup_charge(folio, memcg, gfp);
 	css_put(&memcg->css);
 
 	return ret;
@@ -6568,6 +6567,7 @@ int mem_cgroup_charge(struct page *page, struct mm_struct *mm, gfp_t gfp_mask)
 int mem_cgroup_swapin_charge_page(struct page *page, struct mm_struct *mm,
 				  gfp_t gfp, swp_entry_t entry)
 {
+	struct folio *folio = page_folio(page);
 	struct mem_cgroup *memcg;
 	unsigned short id;
 	int ret;
@@ -6582,7 +6582,7 @@ int mem_cgroup_swapin_charge_page(struct page *page, struct mm_struct *mm,
 		memcg = get_mem_cgroup_from_mm(mm);
 	rcu_read_unlock();
 
-	ret = __mem_cgroup_charge(page, memcg, gfp);
+	ret = __mem_cgroup_charge(folio, memcg, gfp);
 
 	css_put(&memcg->css);
 	return ret;
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 128+ messages in thread

* [PATCH v2 15/46] mm/memcg: Add folio_uncharge_cgroup()
  2021-06-22 12:15 [PATCH v2 00/46] Folio-enabling the page cache Matthew Wilcox (Oracle)
                   ` (13 preceding siblings ...)
  2021-06-22 12:15 ` [PATCH v2 14/46] mm/memcg: Add folio_charge_cgroup() Matthew Wilcox (Oracle)
@ 2021-06-22 12:15 ` Matthew Wilcox (Oracle)
  2021-06-23  8:16   ` Christoph Hellwig
  2021-06-25  8:25   ` Michal Hocko
  2021-06-22 12:15 ` [PATCH v2 16/46] mm/memcg: Add folio_migrate_cgroup() Matthew Wilcox (Oracle)
                   ` (30 subsequent siblings)
  45 siblings, 2 replies; 128+ messages in thread
From: Matthew Wilcox (Oracle) @ 2021-06-22 12:15 UTC (permalink / raw)
  To: akpm; +Cc: Matthew Wilcox (Oracle), linux-fsdevel, linux-mm, linux-kernel

Reimplement mem_cgroup_uncharge() as a wrapper around
folio_uncharge_cgroup().

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 include/linux/memcontrol.h |  5 +++++
 mm/folio-compat.c          |  5 +++++
 mm/memcontrol.c            | 14 +++++++-------
 3 files changed, 17 insertions(+), 7 deletions(-)

diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
index a50e5cee6d2c..d4b2bc939eee 100644
--- a/include/linux/memcontrol.h
+++ b/include/linux/memcontrol.h
@@ -705,6 +705,7 @@ static inline bool mem_cgroup_below_min(struct mem_cgroup *memcg)
 }
 
 int folio_charge_cgroup(struct folio *, struct mm_struct *, gfp_t);
+void folio_uncharge_cgroup(struct folio *);
 
 int mem_cgroup_charge(struct page *page, struct mm_struct *mm, gfp_t gfp_mask);
 int mem_cgroup_swapin_charge_page(struct page *page, struct mm_struct *mm,
@@ -1224,6 +1225,10 @@ static inline int folio_charge_cgroup(struct folio *folio,
 	return 0;
 }
 
+static inline void folio_uncharge_cgroup(struct folio *folio)
+{
+}
+
 static inline int mem_cgroup_charge(struct page *page, struct mm_struct *mm,
 				    gfp_t gfp_mask)
 {
diff --git a/mm/folio-compat.c b/mm/folio-compat.c
index 1d71b8b587f8..d229b979b00d 100644
--- a/mm/folio-compat.c
+++ b/mm/folio-compat.c
@@ -54,4 +54,9 @@ int mem_cgroup_charge(struct page *page, struct mm_struct *mm, gfp_t gfp)
 {
 	return folio_charge_cgroup(page_folio(page), mm, gfp);
 }
+
+void mem_cgroup_uncharge(struct page *page)
+{
+	folio_uncharge_cgroup(page_folio(page));
+}
 #endif
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 69638f84d11b..a6befc0843e7 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -6717,24 +6717,24 @@ static void uncharge_page(struct page *page, struct uncharge_gather *ug)
 }
 
 /**
- * mem_cgroup_uncharge - uncharge a page
- * @page: page to uncharge
+ * folio_uncharge_cgroup - Uncharge a folio.
+ * @folio: Folio to uncharge.
  *
- * Uncharge a page previously charged with mem_cgroup_charge().
+ * Uncharge a folio previously charged with folio_charge_cgroup().
  */
-void mem_cgroup_uncharge(struct page *page)
+void folio_uncharge_cgroup(struct folio *folio)
 {
 	struct uncharge_gather ug;
 
 	if (mem_cgroup_disabled())
 		return;
 
-	/* Don't touch page->lru of any random page, pre-check: */
-	if (!page_memcg(page))
+	/* Don't touch folio->lru of any random page, pre-check: */
+	if (!folio_memcg(folio))
 		return;
 
 	uncharge_gather_clear(&ug);
-	uncharge_page(page, &ug);
+	uncharge_page(&folio->page, &ug);
 	uncharge_batch(&ug);
 }
 
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 128+ messages in thread

* [PATCH v2 16/46] mm/memcg: Add folio_migrate_cgroup()
  2021-06-22 12:15 [PATCH v2 00/46] Folio-enabling the page cache Matthew Wilcox (Oracle)
                   ` (14 preceding siblings ...)
  2021-06-22 12:15 ` [PATCH v2 15/46] mm/memcg: Add folio_uncharge_cgroup() Matthew Wilcox (Oracle)
@ 2021-06-22 12:15 ` Matthew Wilcox (Oracle)
  2021-06-23  8:19   ` Christoph Hellwig
  2021-06-22 12:15 ` [PATCH v2 17/46] mm/memcg: Convert mem_cgroup_track_foreign_dirty_slowpath() to folio Matthew Wilcox (Oracle)
                   ` (29 subsequent siblings)
  45 siblings, 1 reply; 128+ messages in thread
From: Matthew Wilcox (Oracle) @ 2021-06-22 12:15 UTC (permalink / raw)
  To: akpm; +Cc: Matthew Wilcox (Oracle), linux-fsdevel, linux-mm, linux-kernel

Convert all callers of mem_cgroup_migrate() to call folio_migrate_cgroup()
instead.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 .../admin-guide/cgroup-v1/memcg_test.rst      |  2 +-
 include/linux/memcontrol.h                    |  5 ++-
 mm/filemap.c                                  |  4 ++-
 mm/memcontrol.c                               | 31 +++++++++----------
 mm/migrate.c                                  |  4 ++-
 mm/shmem.c                                    |  5 ++-
 6 files changed, 28 insertions(+), 23 deletions(-)

diff --git a/Documentation/admin-guide/cgroup-v1/memcg_test.rst b/Documentation/admin-guide/cgroup-v1/memcg_test.rst
index 45b94f7b3beb..686beda647d0 100644
--- a/Documentation/admin-guide/cgroup-v1/memcg_test.rst
+++ b/Documentation/admin-guide/cgroup-v1/memcg_test.rst
@@ -129,7 +129,7 @@ Under below explanation, we assume CONFIG_MEM_RES_CTRL_SWAP=y.
 7. Page Migration
 =================
 
-	mem_cgroup_migrate()
+	folio_migrate_cgroup()
 
 8. LRU
 ======
diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
index d4b2bc939eee..8158c16f8097 100644
--- a/include/linux/memcontrol.h
+++ b/include/linux/memcontrol.h
@@ -706,6 +706,7 @@ static inline bool mem_cgroup_below_min(struct mem_cgroup *memcg)
 
 int folio_charge_cgroup(struct folio *, struct mm_struct *, gfp_t);
 void folio_uncharge_cgroup(struct folio *);
+void folio_migrate_cgroup(struct folio *old, struct folio *new);
 
 int mem_cgroup_charge(struct page *page, struct mm_struct *mm, gfp_t gfp_mask);
 int mem_cgroup_swapin_charge_page(struct page *page, struct mm_struct *mm,
@@ -715,8 +716,6 @@ void mem_cgroup_swapin_uncharge_swap(swp_entry_t entry);
 void mem_cgroup_uncharge(struct page *page);
 void mem_cgroup_uncharge_list(struct list_head *page_list);
 
-void mem_cgroup_migrate(struct page *oldpage, struct page *newpage);
-
 /**
  * mem_cgroup_lruvec - get the lru list vector for a memcg & node
  * @memcg: memcg of the wanted lruvec
@@ -1253,7 +1252,7 @@ static inline void mem_cgroup_uncharge_list(struct list_head *page_list)
 {
 }
 
-static inline void mem_cgroup_migrate(struct page *old, struct page *new)
+static inline void folio_migrate_cgroup(struct page *old, struct page *new)
 {
 }
 
diff --git a/mm/filemap.c b/mm/filemap.c
index 7b0e4d0e4741..4b2698e5e8e2 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -817,6 +817,8 @@ EXPORT_SYMBOL(file_write_and_wait_range);
  */
 void replace_page_cache_page(struct page *old, struct page *new)
 {
+	struct folio *fold = page_folio(old);
+	struct folio *fnew = page_folio(new);
 	struct address_space *mapping = old->mapping;
 	void (*freepage)(struct page *) = mapping->a_ops->freepage;
 	pgoff_t offset = old->index;
@@ -831,7 +833,7 @@ void replace_page_cache_page(struct page *old, struct page *new)
 	new->mapping = mapping;
 	new->index = offset;
 
-	mem_cgroup_migrate(old, new);
+	folio_migrate_cgroup(fold, fnew);
 
 	xas_lock_irqsave(&xas, flags);
 	xas_store(&xas, new);
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index a6befc0843e7..a9857e091455 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -5410,7 +5410,7 @@ static int mem_cgroup_move_account(struct page *page,
 	VM_BUG_ON(compound && !PageTransHuge(page));
 
 	/*
-	 * Prevent mem_cgroup_migrate() from looking at
+	 * Prevent folio_migrate_cgroup() from looking at
 	 * page's memory cgroup of its source page while we change it.
 	 */
 	ret = -EBUSY;
@@ -6761,40 +6761,39 @@ void mem_cgroup_uncharge_list(struct list_head *page_list)
 }
 
 /**
- * mem_cgroup_migrate - charge a page's replacement
- * @oldpage: currently circulating page
- * @newpage: replacement page
+ * folio_migrate_cgroup - charge a folio's replacement
+ * @oldfolio: currently circulating folio
+ * @newfolio: replacement folio
  *
- * Charge @newpage as a replacement page for @oldpage. @oldpage will
+ * Charge @newfolio as a replacement folio for @oldfolio. @oldfolio will
  * be uncharged upon free.
  *
- * Both pages must be locked, @newpage->mapping must be set up.
+ * Both folios must be locked, @newfolio->mapping must be set up.
  */
-void mem_cgroup_migrate(struct page *oldpage, struct page *newpage)
+void folio_migrate_cgroup(struct folio *old, struct folio *newfolio)
 {
-	struct folio *newfolio = page_folio(newpage);
 	struct mem_cgroup *memcg;
 	unsigned int nr_pages = folio_nr_pages(newfolio);
 	unsigned long flags;
 
-	VM_BUG_ON_PAGE(!PageLocked(oldpage), oldpage);
+	VM_BUG_ON_FOLIO(!folio_locked(old), old);
 	VM_BUG_ON_FOLIO(!folio_locked(newfolio), newfolio);
-	VM_BUG_ON_FOLIO(PageAnon(oldpage) != folio_anon(newfolio), newfolio);
-	VM_BUG_ON_FOLIO(compound_nr(oldpage) != nr_pages, newfolio);
+	VM_BUG_ON_FOLIO(folio_anon(old) != folio_anon(newfolio), newfolio);
+	VM_BUG_ON_FOLIO(folio_nr_pages(old) != nr_pages, newfolio);
 
 	if (mem_cgroup_disabled())
 		return;
 
-	/* Page cache replacement: new page already charged? */
+	/* Page cache replacement: new folio already charged? */
 	if (folio_memcg(newfolio))
 		return;
 
-	memcg = page_memcg(oldpage);
-	VM_WARN_ON_ONCE_PAGE(!memcg, oldpage);
+	memcg = folio_memcg(old);
+	VM_WARN_ON_ONCE_FOLIO(!memcg, old);
 	if (!memcg)
 		return;
 
-	/* Force-charge the new page. The old one will be freed soon */
+	/* Force-charge the new folio. The old one will be freed soon */
 	page_counter_charge(&memcg->memory, nr_pages);
 	if (do_memsw_account())
 		page_counter_charge(&memcg->memsw, nr_pages);
@@ -6804,7 +6803,7 @@ void mem_cgroup_migrate(struct page *oldpage, struct page *newpage)
 
 	local_irq_save(flags);
 	mem_cgroup_charge_statistics(memcg, nr_pages);
-	memcg_check_events(memcg, newpage);
+	memcg_check_events(memcg, &newfolio->page);
 	local_irq_restore(flags);
 }
 
diff --git a/mm/migrate.c b/mm/migrate.c
index b234c3f3acb7..fff63e139767 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -581,6 +581,8 @@ static void copy_huge_page(struct page *dst, struct page *src)
  */
 void migrate_page_states(struct page *newpage, struct page *page)
 {
+	struct folio *folio = page_folio(page);
+	struct folio *newfolio = page_folio(newpage);
 	int cpupid;
 
 	if (PageError(page))
@@ -645,7 +647,7 @@ void migrate_page_states(struct page *newpage, struct page *page)
 	copy_page_owner(page, newpage);
 
 	if (!PageHuge(page))
-		mem_cgroup_migrate(page, newpage);
+		folio_migrate_cgroup(folio, newfolio);
 }
 EXPORT_SYMBOL(migrate_page_states);
 
diff --git a/mm/shmem.c b/mm/shmem.c
index 5d46611cba8d..efc77a7e19bd 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -1619,6 +1619,7 @@ static int shmem_replace_page(struct page **pagep, gfp_t gfp,
 				struct shmem_inode_info *info, pgoff_t index)
 {
 	struct page *oldpage, *newpage;
+	struct folio *old, *new;
 	struct address_space *swap_mapping;
 	swp_entry_t entry;
 	pgoff_t swap_index;
@@ -1655,7 +1656,9 @@ static int shmem_replace_page(struct page **pagep, gfp_t gfp,
 	xa_lock_irq(&swap_mapping->i_pages);
 	error = shmem_replace_entry(swap_mapping, swap_index, oldpage, newpage);
 	if (!error) {
-		mem_cgroup_migrate(oldpage, newpage);
+		old = page_folio(oldpage);
+		new = page_folio(newpage);
+		folio_migrate_cgroup(old, new);
 		__inc_lruvec_page_state(newpage, NR_FILE_PAGES);
 		__dec_lruvec_page_state(oldpage, NR_FILE_PAGES);
 	}
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 128+ messages in thread

* [PATCH v2 17/46] mm/memcg: Convert mem_cgroup_track_foreign_dirty_slowpath() to folio
  2021-06-22 12:15 [PATCH v2 00/46] Folio-enabling the page cache Matthew Wilcox (Oracle)
                   ` (15 preceding siblings ...)
  2021-06-22 12:15 ` [PATCH v2 16/46] mm/memcg: Add folio_migrate_cgroup() Matthew Wilcox (Oracle)
@ 2021-06-22 12:15 ` Matthew Wilcox (Oracle)
  2021-06-23  8:21   ` Christoph Hellwig
  2021-06-22 12:15 ` [PATCH v2 18/46] mm/migrate: Add folio_migrate_mapping() Matthew Wilcox (Oracle)
                   ` (28 subsequent siblings)
  45 siblings, 1 reply; 128+ messages in thread
From: Matthew Wilcox (Oracle) @ 2021-06-22 12:15 UTC (permalink / raw)
  To: akpm; +Cc: Matthew Wilcox (Oracle), linux-fsdevel, linux-mm, linux-kernel

The page was only being used for the memcg and to gather trace
information, so this is a simple conversion.  The only caller of
mem_cgroup_track_foreign_dirty() will be converted to folios in a later
patch, so doing this now makes that patch simpler.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 include/linux/memcontrol.h       | 7 ++++---
 include/trace/events/writeback.h | 8 ++++----
 mm/memcontrol.c                  | 6 +++---
 3 files changed, 11 insertions(+), 10 deletions(-)

diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
index 8158c16f8097..00693cb48b5d 100644
--- a/include/linux/memcontrol.h
+++ b/include/linux/memcontrol.h
@@ -1635,17 +1635,18 @@ void mem_cgroup_wb_stats(struct bdi_writeback *wb, unsigned long *pfilepages,
 			 unsigned long *pheadroom, unsigned long *pdirty,
 			 unsigned long *pwriteback);
 
-void mem_cgroup_track_foreign_dirty_slowpath(struct page *page,
+void mem_cgroup_track_foreign_dirty_slowpath(struct folio *folio,
 					     struct bdi_writeback *wb);
 
 static inline void mem_cgroup_track_foreign_dirty(struct page *page,
 						  struct bdi_writeback *wb)
 {
+	struct folio *folio = page_folio(page);
 	if (mem_cgroup_disabled())
 		return;
 
-	if (unlikely(&page_memcg(page)->css != wb->memcg_css))
-		mem_cgroup_track_foreign_dirty_slowpath(page, wb);
+	if (unlikely(&folio_memcg(folio)->css != wb->memcg_css))
+		mem_cgroup_track_foreign_dirty_slowpath(folio, wb);
 }
 
 void mem_cgroup_flush_foreign(struct bdi_writeback *wb);
diff --git a/include/trace/events/writeback.h b/include/trace/events/writeback.h
index 1efa463c4979..80b24801bbf7 100644
--- a/include/trace/events/writeback.h
+++ b/include/trace/events/writeback.h
@@ -235,9 +235,9 @@ TRACE_EVENT(inode_switch_wbs,
 
 TRACE_EVENT(track_foreign_dirty,
 
-	TP_PROTO(struct page *page, struct bdi_writeback *wb),
+	TP_PROTO(struct folio *folio, struct bdi_writeback *wb),
 
-	TP_ARGS(page, wb),
+	TP_ARGS(folio, wb),
 
 	TP_STRUCT__entry(
 		__array(char,		name, 32)
@@ -249,7 +249,7 @@ TRACE_EVENT(track_foreign_dirty,
 	),
 
 	TP_fast_assign(
-		struct address_space *mapping = page_mapping(page);
+		struct address_space *mapping = folio_mapping(folio);
 		struct inode *inode = mapping ? mapping->host : NULL;
 
 		strscpy_pad(__entry->name, bdi_dev_name(wb->bdi), 32);
@@ -257,7 +257,7 @@ TRACE_EVENT(track_foreign_dirty,
 		__entry->ino		= inode ? inode->i_ino : 0;
 		__entry->memcg_id	= wb->memcg_css->id;
 		__entry->cgroup_ino	= __trace_wb_assign_cgroup(wb);
-		__entry->page_cgroup_ino = cgroup_ino(page_memcg(page)->css.cgroup);
+		__entry->page_cgroup_ino = cgroup_ino(folio_memcg(folio)->css.cgroup);
 	),
 
 	TP_printk("bdi %s[%llu]: ino=%lu memcg_id=%u cgroup_ino=%lu page_cgroup_ino=%lu",
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index a9857e091455..64eff07f0c4c 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -4394,17 +4394,17 @@ void mem_cgroup_wb_stats(struct bdi_writeback *wb, unsigned long *pfilepages,
  * As being wrong occasionally doesn't matter, updates and accesses to the
  * records are lockless and racy.
  */
-void mem_cgroup_track_foreign_dirty_slowpath(struct page *page,
+void mem_cgroup_track_foreign_dirty_slowpath(struct folio *folio,
 					     struct bdi_writeback *wb)
 {
-	struct mem_cgroup *memcg = page_memcg(page);
+	struct mem_cgroup *memcg = folio_memcg(folio);
 	struct memcg_cgwb_frn *frn;
 	u64 now = get_jiffies_64();
 	u64 oldest_at = now;
 	int oldest = -1;
 	int i;
 
-	trace_track_foreign_dirty(page, wb);
+	trace_track_foreign_dirty(folio, wb);
 
 	/*
 	 * Pick the slot to use.  If there is already a slot for @wb, keep
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 128+ messages in thread

* [PATCH v2 18/46] mm/migrate: Add folio_migrate_mapping()
  2021-06-22 12:15 [PATCH v2 00/46] Folio-enabling the page cache Matthew Wilcox (Oracle)
                   ` (16 preceding siblings ...)
  2021-06-22 12:15 ` [PATCH v2 17/46] mm/memcg: Convert mem_cgroup_track_foreign_dirty_slowpath() to folio Matthew Wilcox (Oracle)
@ 2021-06-22 12:15 ` Matthew Wilcox (Oracle)
  2021-06-23  8:22   ` Christoph Hellwig
  2021-06-22 12:15 ` [PATCH v2 19/46] mm/migrate: Add folio_migrate_flags() Matthew Wilcox (Oracle)
                   ` (27 subsequent siblings)
  45 siblings, 1 reply; 128+ messages in thread
From: Matthew Wilcox (Oracle) @ 2021-06-22 12:15 UTC (permalink / raw)
  To: akpm; +Cc: Matthew Wilcox (Oracle), linux-fsdevel, linux-mm, linux-kernel

Reimplement migrate_page_move_mapping() as a wrapper around
folio_migrate_mapping().  Saves 193 bytes of kernel text.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 include/linux/migrate.h |  2 +
 mm/folio-compat.c       | 11 ++++++
 mm/migrate.c            | 85 +++++++++++++++++++++--------------------
 3 files changed, 57 insertions(+), 41 deletions(-)

diff --git a/include/linux/migrate.h b/include/linux/migrate.h
index 4bb4e519e3f5..a4ff65e9c1e3 100644
--- a/include/linux/migrate.h
+++ b/include/linux/migrate.h
@@ -51,6 +51,8 @@ extern int migrate_huge_page_move_mapping(struct address_space *mapping,
 				  struct page *newpage, struct page *page);
 extern int migrate_page_move_mapping(struct address_space *mapping,
 		struct page *newpage, struct page *page, int extra_count);
+int folio_migrate_mapping(struct address_space *mapping,
+		struct folio *newfolio, struct folio *folio, int extra_count);
 #else
 
 static inline void putback_movable_pages(struct list_head *l) {}
diff --git a/mm/folio-compat.c b/mm/folio-compat.c
index d229b979b00d..25c2269655f4 100644
--- a/mm/folio-compat.c
+++ b/mm/folio-compat.c
@@ -4,6 +4,7 @@
  * eventually.
  */
 
+#include <linux/migrate.h>
 #include <linux/pagemap.h>
 #include <linux/swap.h>
 
@@ -60,3 +61,13 @@ void mem_cgroup_uncharge(struct page *page)
 	folio_uncharge_cgroup(page_folio(page));
 }
 #endif
+
+#ifdef CONFIG_MIGRATION
+int migrate_page_move_mapping(struct address_space *mapping,
+		struct page *newpage, struct page *page, int extra_count)
+{
+	return folio_migrate_mapping(mapping, page_folio(newpage),
+					page_folio(page), extra_count);
+}
+EXPORT_SYMBOL(migrate_page_move_mapping);
+#endif
diff --git a/mm/migrate.c b/mm/migrate.c
index fff63e139767..b668970acd11 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -355,7 +355,7 @@ static int expected_page_refs(struct address_space *mapping, struct page *page)
 	 */
 	expected_count += is_device_private_page(page);
 	if (mapping)
-		expected_count += thp_nr_pages(page) + page_has_private(page);
+		expected_count += compound_nr(page) + page_has_private(page);
 
 	return expected_count;
 }
@@ -368,74 +368,75 @@ static int expected_page_refs(struct address_space *mapping, struct page *page)
  * 2 for pages with a mapping
  * 3 for pages with a mapping and PagePrivate/PagePrivate2 set.
  */
-int migrate_page_move_mapping(struct address_space *mapping,
-		struct page *newpage, struct page *page, int extra_count)
+int folio_migrate_mapping(struct address_space *mapping,
+		struct folio *newfolio, struct folio *folio, int extra_count)
 {
-	XA_STATE(xas, &mapping->i_pages, page_index(page));
+	XA_STATE(xas, &mapping->i_pages, folio_index(folio));
 	struct zone *oldzone, *newzone;
 	int dirty;
-	int expected_count = expected_page_refs(mapping, page) + extra_count;
-	int nr = thp_nr_pages(page);
+	int expected_count = expected_page_refs(mapping, &folio->page) + extra_count;
+	int nr = folio_nr_pages(folio);
 
 	if (!mapping) {
 		/* Anonymous page without mapping */
-		if (page_count(page) != expected_count)
+		if (folio_ref_count(folio) != expected_count)
 			return -EAGAIN;
 
 		/* No turning back from here */
-		newpage->index = page->index;
-		newpage->mapping = page->mapping;
-		if (PageSwapBacked(page))
-			__SetPageSwapBacked(newpage);
+		newfolio->index = folio->index;
+		newfolio->mapping = folio->mapping;
+		if (folio_swapbacked(folio))
+			__folio_set_swapbacked_flag(newfolio);
 
 		return MIGRATEPAGE_SUCCESS;
 	}
 
-	oldzone = page_zone(page);
-	newzone = page_zone(newpage);
+	oldzone = folio_zone(folio);
+	newzone = folio_zone(newfolio);
 
 	xas_lock_irq(&xas);
-	if (page_count(page) != expected_count || xas_load(&xas) != page) {
+	if (folio_ref_count(folio) != expected_count ||
+	    xas_load(&xas) != folio) {
 		xas_unlock_irq(&xas);
 		return -EAGAIN;
 	}
 
-	if (!page_ref_freeze(page, expected_count)) {
+	if (!folio_ref_freeze(folio, expected_count)) {
 		xas_unlock_irq(&xas);
 		return -EAGAIN;
 	}
 
 	/*
-	 * Now we know that no one else is looking at the page:
+	 * Now we know that no one else is looking at the folio:
 	 * no turning back from here.
 	 */
-	newpage->index = page->index;
-	newpage->mapping = page->mapping;
-	page_ref_add(newpage, nr); /* add cache reference */
-	if (PageSwapBacked(page)) {
-		__SetPageSwapBacked(newpage);
-		if (PageSwapCache(page)) {
-			SetPageSwapCache(newpage);
-			set_page_private(newpage, page_private(page));
+	newfolio->index = folio->index;
+	newfolio->mapping = folio->mapping;
+	folio_ref_add(newfolio, nr); /* add cache reference */
+	if (folio_swapbacked(folio)) {
+		__folio_set_swapbacked_flag(newfolio);
+		if (folio_swapcache(folio)) {
+			folio_set_swapcache_flag(newfolio);
+			newfolio->private = folio_get_private(folio);
 		}
 	} else {
-		VM_BUG_ON_PAGE(PageSwapCache(page), page);
+		VM_BUG_ON_FOLIO(folio_swapcache(folio), folio);
 	}
 
 	/* Move dirty while page refs frozen and newpage not yet exposed */
-	dirty = PageDirty(page);
+	dirty = folio_dirty(folio);
 	if (dirty) {
-		ClearPageDirty(page);
-		SetPageDirty(newpage);
+		folio_clear_dirty_flag(folio);
+		folio_set_dirty_flag(newfolio);
 	}
 
-	xas_store(&xas, newpage);
-	if (PageTransHuge(page)) {
+	xas_store(&xas, newfolio);
+	if (nr > 1) {
 		int i;
 
 		for (i = 1; i < nr; i++) {
 			xas_next(&xas);
-			xas_store(&xas, newpage);
+			xas_store(&xas, newfolio);
 		}
 	}
 
@@ -444,7 +445,7 @@ int migrate_page_move_mapping(struct address_space *mapping,
 	 * to one less reference.
 	 * We know this isn't the last reference.
 	 */
-	page_ref_unfreeze(page, expected_count - nr);
+	folio_ref_unfreeze(folio, expected_count - nr);
 
 	xas_unlock(&xas);
 	/* Leave irq disabled to prevent preemption while updating stats */
@@ -463,18 +464,18 @@ int migrate_page_move_mapping(struct address_space *mapping,
 		struct lruvec *old_lruvec, *new_lruvec;
 		struct mem_cgroup *memcg;
 
-		memcg = page_memcg(page);
+		memcg = folio_memcg(folio);
 		old_lruvec = mem_cgroup_lruvec(memcg, oldzone->zone_pgdat);
 		new_lruvec = mem_cgroup_lruvec(memcg, newzone->zone_pgdat);
 
 		__mod_lruvec_state(old_lruvec, NR_FILE_PAGES, -nr);
 		__mod_lruvec_state(new_lruvec, NR_FILE_PAGES, nr);
-		if (PageSwapBacked(page) && !PageSwapCache(page)) {
+		if (folio_swapbacked(folio) && !folio_swapcache(folio)) {
 			__mod_lruvec_state(old_lruvec, NR_SHMEM, -nr);
 			__mod_lruvec_state(new_lruvec, NR_SHMEM, nr);
 		}
 #ifdef CONFIG_SWAP
-		if (PageSwapCache(page)) {
+		if (folio_swapcache(folio)) {
 			__mod_lruvec_state(old_lruvec, NR_SWAPCACHE, -nr);
 			__mod_lruvec_state(new_lruvec, NR_SWAPCACHE, nr);
 		}
@@ -490,11 +491,11 @@ int migrate_page_move_mapping(struct address_space *mapping,
 
 	return MIGRATEPAGE_SUCCESS;
 }
-EXPORT_SYMBOL(migrate_page_move_mapping);
+EXPORT_SYMBOL(folio_migrate_mapping);
 
 /*
  * The expected number of remaining references is the same as that
- * of migrate_page_move_mapping().
+ * of folio_migrate_mapping().
  */
 int migrate_huge_page_move_mapping(struct address_space *mapping,
 				   struct page *newpage, struct page *page)
@@ -603,7 +604,7 @@ void migrate_page_states(struct page *newpage, struct page *page)
 	if (PageMappedToDisk(page))
 		SetPageMappedToDisk(newpage);
 
-	/* Move dirty on pages not done by migrate_page_move_mapping() */
+	/* Move dirty on pages not done by folio_migrate_mapping() */
 	if (PageDirty(page))
 		SetPageDirty(newpage);
 
@@ -676,11 +677,13 @@ int migrate_page(struct address_space *mapping,
 		struct page *newpage, struct page *page,
 		enum migrate_mode mode)
 {
+	struct folio *newfolio = page_folio(newpage);
+	struct folio *folio = page_folio(page);
 	int rc;
 
-	BUG_ON(PageWriteback(page));	/* Writeback must be complete */
+	BUG_ON(folio_writeback(folio));	/* Writeback must be complete */
 
-	rc = migrate_page_move_mapping(mapping, newpage, page, 0);
+	rc = folio_migrate_mapping(mapping, newfolio, folio, 0);
 
 	if (rc != MIGRATEPAGE_SUCCESS)
 		return rc;
@@ -2536,7 +2539,7 @@ static void migrate_vma_collect(struct migrate_vma *migrate)
  * @page: struct page to check
  *
  * Pinned pages cannot be migrated. This is the same test as in
- * migrate_page_move_mapping(), except that here we allow migration of a
+ * folio_migrate_mapping(), except that here we allow migration of a
  * ZONE_DEVICE page.
  */
 static bool migrate_vma_check_page(struct page *page)
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 128+ messages in thread

* [PATCH v2 19/46] mm/migrate: Add folio_migrate_flags()
  2021-06-22 12:15 [PATCH v2 00/46] Folio-enabling the page cache Matthew Wilcox (Oracle)
                   ` (17 preceding siblings ...)
  2021-06-22 12:15 ` [PATCH v2 18/46] mm/migrate: Add folio_migrate_mapping() Matthew Wilcox (Oracle)
@ 2021-06-22 12:15 ` Matthew Wilcox (Oracle)
  2021-06-23  8:28   ` Christoph Hellwig
  2021-06-22 12:15 ` [PATCH v2 20/46] mm/migrate: Add folio_migrate_copy() Matthew Wilcox (Oracle)
                   ` (26 subsequent siblings)
  45 siblings, 1 reply; 128+ messages in thread
From: Matthew Wilcox (Oracle) @ 2021-06-22 12:15 UTC (permalink / raw)
  To: akpm; +Cc: Matthew Wilcox (Oracle), linux-fsdevel, linux-mm, linux-kernel

Turn migrate_page_states() into a wrapper around folio_migrate_flags().
Also convert two functions only called from folio_migrate_flags() to
be folio-based.  ksm_migrate_page() becomes folio_migrate_ksm() and
copy_page_owner() becomes folio_copy_owner().  folio_migrate_flags()
alone shrinks by two thirds -- 1967 bytes down to 642 bytes.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 include/linux/ksm.h        |  4 +-
 include/linux/migrate.h    |  1 +
 include/linux/page_owner.h |  8 ++--
 mm/folio-compat.c          |  6 +++
 mm/ksm.c                   | 31 ++++++++------
 mm/migrate.c               | 85 +++++++++++++++++++-------------------
 mm/page_owner.c            | 10 ++---
 7 files changed, 79 insertions(+), 66 deletions(-)

diff --git a/include/linux/ksm.h b/include/linux/ksm.h
index 161e8164abcf..a38a5bca1ba5 100644
--- a/include/linux/ksm.h
+++ b/include/linux/ksm.h
@@ -52,7 +52,7 @@ struct page *ksm_might_need_to_copy(struct page *page,
 			struct vm_area_struct *vma, unsigned long address);
 
 void rmap_walk_ksm(struct page *page, struct rmap_walk_control *rwc);
-void ksm_migrate_page(struct page *newpage, struct page *oldpage);
+void folio_migrate_ksm(struct folio *newfolio, struct folio *folio);
 
 #else  /* !CONFIG_KSM */
 
@@ -83,7 +83,7 @@ static inline void rmap_walk_ksm(struct page *page,
 {
 }
 
-static inline void ksm_migrate_page(struct page *newpage, struct page *oldpage)
+static inline void folio_migrate_ksm(struct folio *newfolio, struct folio *old)
 {
 }
 #endif /* CONFIG_MMU */
diff --git a/include/linux/migrate.h b/include/linux/migrate.h
index a4ff65e9c1e3..7993faffa46d 100644
--- a/include/linux/migrate.h
+++ b/include/linux/migrate.h
@@ -51,6 +51,7 @@ extern int migrate_huge_page_move_mapping(struct address_space *mapping,
 				  struct page *newpage, struct page *page);
 extern int migrate_page_move_mapping(struct address_space *mapping,
 		struct page *newpage, struct page *page, int extra_count);
+void folio_migrate_flags(struct folio *newfolio, struct folio *folio);
 int folio_migrate_mapping(struct address_space *mapping,
 		struct folio *newfolio, struct folio *folio, int extra_count);
 #else
diff --git a/include/linux/page_owner.h b/include/linux/page_owner.h
index 719bfe5108c5..43c638c51c1f 100644
--- a/include/linux/page_owner.h
+++ b/include/linux/page_owner.h
@@ -12,7 +12,7 @@ extern void __reset_page_owner(struct page *page, unsigned int order);
 extern void __set_page_owner(struct page *page,
 			unsigned int order, gfp_t gfp_mask);
 extern void __split_page_owner(struct page *page, unsigned int nr);
-extern void __copy_page_owner(struct page *oldpage, struct page *newpage);
+extern void __folio_copy_owner(struct folio *newfolio, struct folio *old);
 extern void __set_page_owner_migrate_reason(struct page *page, int reason);
 extern void __dump_page_owner(const struct page *page);
 extern void pagetypeinfo_showmixedcount_print(struct seq_file *m,
@@ -36,10 +36,10 @@ static inline void split_page_owner(struct page *page, unsigned int nr)
 	if (static_branch_unlikely(&page_owner_inited))
 		__split_page_owner(page, nr);
 }
-static inline void copy_page_owner(struct page *oldpage, struct page *newpage)
+static inline void folio_copy_owner(struct folio *newfolio, struct folio *old)
 {
 	if (static_branch_unlikely(&page_owner_inited))
-		__copy_page_owner(oldpage, newpage);
+		__folio_copy_owner(newfolio, old);
 }
 static inline void set_page_owner_migrate_reason(struct page *page, int reason)
 {
@@ -63,7 +63,7 @@ static inline void split_page_owner(struct page *page,
 			unsigned int order)
 {
 }
-static inline void copy_page_owner(struct page *oldpage, struct page *newpage)
+static inline void folio_copy_owner(struct folio *newfolio, struct folio *folio)
 {
 }
 static inline void set_page_owner_migrate_reason(struct page *page, int reason)
diff --git a/mm/folio-compat.c b/mm/folio-compat.c
index 25c2269655f4..f0ac904d396f 100644
--- a/mm/folio-compat.c
+++ b/mm/folio-compat.c
@@ -70,4 +70,10 @@ int migrate_page_move_mapping(struct address_space *mapping,
 					page_folio(page), extra_count);
 }
 EXPORT_SYMBOL(migrate_page_move_mapping);
+
+void migrate_page_states(struct page *newpage, struct page *page)
+{
+	folio_migrate_flags(page_folio(newpage), page_folio(page));
+}
+EXPORT_SYMBOL(migrate_page_states);
 #endif
diff --git a/mm/ksm.c b/mm/ksm.c
index 2f3aaeb34a42..edbae596ce31 100644
--- a/mm/ksm.c
+++ b/mm/ksm.c
@@ -755,7 +755,7 @@ static struct page *get_ksm_page(struct stable_node *stable_node,
 	/*
 	 * We come here from above when page->mapping or !PageSwapCache
 	 * suggests that the node is stale; but it might be under migration.
-	 * We need smp_rmb(), matching the smp_wmb() in ksm_migrate_page(),
+	 * We need smp_rmb(), matching the smp_wmb() in folio_migrate_ksm(),
 	 * before checking whether node->kpfn has been changed.
 	 */
 	smp_rmb();
@@ -856,9 +856,14 @@ static int unmerge_ksm_pages(struct vm_area_struct *vma,
 	return err;
 }
 
+static inline struct stable_node *folio_stable_node(struct folio *folio)
+{
+	return folio_ksm(folio) ? folio_rmapping(folio) : NULL;
+}
+
 static inline struct stable_node *page_stable_node(struct page *page)
 {
-	return PageKsm(page) ? page_rmapping(page) : NULL;
+	return folio_stable_node(page_folio(page));
 }
 
 static inline void set_page_stable_node(struct page *page,
@@ -2662,26 +2667,26 @@ void rmap_walk_ksm(struct page *page, struct rmap_walk_control *rwc)
 }
 
 #ifdef CONFIG_MIGRATION
-void ksm_migrate_page(struct page *newpage, struct page *oldpage)
+void folio_migrate_ksm(struct folio *newfolio, struct folio *folio)
 {
 	struct stable_node *stable_node;
 
-	VM_BUG_ON_PAGE(!PageLocked(oldpage), oldpage);
-	VM_BUG_ON_PAGE(!PageLocked(newpage), newpage);
-	VM_BUG_ON_PAGE(newpage->mapping != oldpage->mapping, newpage);
+	VM_BUG_ON_FOLIO(!folio_locked(folio), folio);
+	VM_BUG_ON_FOLIO(!folio_locked(newfolio), newfolio);
+	VM_BUG_ON_FOLIO(newfolio->mapping != folio->mapping, newfolio);
 
-	stable_node = page_stable_node(newpage);
+	stable_node = folio_stable_node(folio);
 	if (stable_node) {
-		VM_BUG_ON_PAGE(stable_node->kpfn != page_to_pfn(oldpage), oldpage);
-		stable_node->kpfn = page_to_pfn(newpage);
+		VM_BUG_ON_FOLIO(stable_node->kpfn != folio_to_pfn(folio), folio);
+		stable_node->kpfn = folio_to_pfn(newfolio);
 		/*
-		 * newpage->mapping was set in advance; now we need smp_wmb()
+		 * newfolio->mapping was set in advance; now we need smp_wmb()
 		 * to make sure that the new stable_node->kpfn is visible
-		 * to get_ksm_page() before it can see that oldpage->mapping
-		 * has gone stale (or that PageSwapCache has been cleared).
+		 * to get_ksm_page() before it can see that folio->mapping
+		 * has gone stale (or that folio_swapcache has been cleared).
 		 */
 		smp_wmb();
-		set_page_stable_node(oldpage, NULL);
+		set_page_stable_node(&folio->page, NULL);
 	}
 }
 #endif /* CONFIG_MIGRATION */
diff --git a/mm/migrate.c b/mm/migrate.c
index b668970acd11..8b289156f4c6 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -578,79 +578,80 @@ static void copy_huge_page(struct page *dst, struct page *src)
 }
 
 /*
- * Copy the page to its new location
+ * Copy the flags and some other ancillary information
  */
-void migrate_page_states(struct page *newpage, struct page *page)
+void folio_migrate_flags(struct folio *newfolio, struct folio *folio)
 {
-	struct folio *folio = page_folio(page);
-	struct folio *newfolio = page_folio(newpage);
 	int cpupid;
 
-	if (PageError(page))
-		SetPageError(newpage);
-	if (PageReferenced(page))
-		SetPageReferenced(newpage);
-	if (PageUptodate(page))
-		SetPageUptodate(newpage);
-	if (TestClearPageActive(page)) {
-		VM_BUG_ON_PAGE(PageUnevictable(page), page);
-		SetPageActive(newpage);
-	} else if (TestClearPageUnevictable(page))
-		SetPageUnevictable(newpage);
-	if (PageWorkingset(page))
-		SetPageWorkingset(newpage);
-	if (PageChecked(page))
-		SetPageChecked(newpage);
-	if (PageMappedToDisk(page))
-		SetPageMappedToDisk(newpage);
+	if (folio_error(folio))
+		folio_set_error_flag(newfolio);
+	if (folio_referenced(folio))
+		folio_set_referenced_flag(newfolio);
+	if (folio_uptodate(folio))
+		folio_mark_uptodate(newfolio);
+	if (folio_test_clear_active_flag(folio)) {
+		VM_BUG_ON_FOLIO(folio_unevictable(folio), folio);
+		folio_set_active_flag(newfolio);
+	} else if (folio_test_clear_unevictable_flag(folio))
+		folio_set_unevictable_flag(newfolio);
+	if (folio_workingset(folio))
+		folio_set_workingset_flag(newfolio);
+	if (folio_checked(folio))
+		folio_set_checked_flag(newfolio);
+	if (folio_mappedtodisk(folio))
+		folio_set_mappedtodisk_flag(newfolio);
 
 	/* Move dirty on pages not done by folio_migrate_mapping() */
-	if (PageDirty(page))
-		SetPageDirty(newpage);
+	if (folio_dirty(folio))
+		folio_set_dirty_flag(newfolio);
 
-	if (page_is_young(page))
-		set_page_young(newpage);
-	if (page_is_idle(page))
-		set_page_idle(newpage);
+	if (folio_young(folio))
+		folio_set_young_flag(newfolio);
+	if (folio_idle(folio))
+		folio_set_idle_flag(newfolio);
 
 	/*
 	 * Copy NUMA information to the new page, to prevent over-eager
 	 * future migrations of this same page.
 	 */
-	cpupid = page_cpupid_xchg_last(page, -1);
-	page_cpupid_xchg_last(newpage, cpupid);
+	cpupid = page_cpupid_xchg_last(&folio->page, -1);
+	page_cpupid_xchg_last(&newfolio->page, cpupid);
 
-	ksm_migrate_page(newpage, page);
+	folio_migrate_ksm(newfolio, folio);
 	/*
 	 * Please do not reorder this without considering how mm/ksm.c's
 	 * get_ksm_page() depends upon ksm_migrate_page() and PageSwapCache().
 	 */
-	if (PageSwapCache(page))
-		ClearPageSwapCache(page);
-	ClearPagePrivate(page);
-	set_page_private(page, 0);
+	if (folio_swapcache(folio))
+		folio_clear_swapcache_flag(folio);
+	folio_clear_private_flag(folio);
+
+	/* page->private contains hugetlb specific flags */
+	if (!folio_hugetlb(folio))
+		folio->private = NULL;
 
 	/*
 	 * If any waiters have accumulated on the new page then
 	 * wake them up.
 	 */
-	if (PageWriteback(newpage))
-		end_page_writeback(newpage);
+	if (folio_writeback(newfolio))
+		folio_end_writeback(newfolio);
 
 	/*
 	 * PG_readahead shares the same bit with PG_reclaim.  The above
 	 * end_page_writeback() may clear PG_readahead mistakenly, so set the
 	 * bit after that.
 	 */
-	if (PageReadahead(page))
-		SetPageReadahead(newpage);
+	if (folio_readahead(folio))
+		folio_set_readahead_flag(newfolio);
 
-	copy_page_owner(page, newpage);
+	folio_copy_owner(folio, newfolio);
 
-	if (!PageHuge(page))
+	if (!folio_hugetlb(folio))
 		folio_migrate_cgroup(folio, newfolio);
 }
-EXPORT_SYMBOL(migrate_page_states);
+EXPORT_SYMBOL(folio_migrate_flags);
 
 void migrate_page_copy(struct page *newpage, struct page *page)
 {
@@ -691,7 +692,7 @@ int migrate_page(struct address_space *mapping,
 	if (mode != MIGRATE_SYNC_NO_COPY)
 		migrate_page_copy(newpage, page);
 	else
-		migrate_page_states(newpage, page);
+		folio_migrate_flags(newfolio, folio);
 	return MIGRATEPAGE_SUCCESS;
 }
 EXPORT_SYMBOL(migrate_page);
diff --git a/mm/page_owner.c b/mm/page_owner.c
index f51a57e92aa3..23bfb074ca3f 100644
--- a/mm/page_owner.c
+++ b/mm/page_owner.c
@@ -210,10 +210,10 @@ void __split_page_owner(struct page *page, unsigned int nr)
 	}
 }
 
-void __copy_page_owner(struct page *oldpage, struct page *newpage)
+void __folio_copy_owner(struct folio *newfolio, struct folio *old)
 {
-	struct page_ext *old_ext = lookup_page_ext(oldpage);
-	struct page_ext *new_ext = lookup_page_ext(newpage);
+	struct page_ext *old_ext = lookup_page_ext(&old->page);
+	struct page_ext *new_ext = lookup_page_ext(&newfolio->page);
 	struct page_owner *old_page_owner, *new_page_owner;
 
 	if (unlikely(!old_ext || !new_ext))
@@ -231,11 +231,11 @@ void __copy_page_owner(struct page *oldpage, struct page *newpage)
 	new_page_owner->free_ts_nsec = old_page_owner->ts_nsec;
 
 	/*
-	 * We don't clear the bit on the oldpage as it's going to be freed
+	 * We don't clear the bit on the old folio as it's going to be freed
 	 * after migration. Until then, the info can be useful in case of
 	 * a bug, and the overall stats will be off a bit only temporarily.
 	 * Also, migrate_misplaced_transhuge_page() can still fail the
-	 * migration and then we want the oldpage to retain the info. But
+	 * migration and then we want the old folio to retain the info. But
 	 * in that case we also don't need to explicitly clear the info from
 	 * the new page, which will be freed.
 	 */
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 128+ messages in thread

* [PATCH v2 20/46] mm/migrate: Add folio_migrate_copy()
  2021-06-22 12:15 [PATCH v2 00/46] Folio-enabling the page cache Matthew Wilcox (Oracle)
                   ` (18 preceding siblings ...)
  2021-06-22 12:15 ` [PATCH v2 19/46] mm/migrate: Add folio_migrate_flags() Matthew Wilcox (Oracle)
@ 2021-06-22 12:15 ` Matthew Wilcox (Oracle)
  2021-06-23  8:35   ` Christoph Hellwig
  2021-06-22 12:15 ` [PATCH v2 21/46] mm/writeback: Rename __add_wb_stat() to wb_stat_mod() Matthew Wilcox (Oracle)
                   ` (25 subsequent siblings)
  45 siblings, 1 reply; 128+ messages in thread
From: Matthew Wilcox (Oracle) @ 2021-06-22 12:15 UTC (permalink / raw)
  To: akpm; +Cc: Matthew Wilcox (Oracle), linux-fsdevel, linux-mm, linux-kernel

Combine the THP, hugetlb and base page routines together into a simple
loop.  It does iterate the pages backwards, but real CPUs can prefetch
a backwards walk.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 include/linux/migrate.h |  1 +
 mm/folio-compat.c       |  6 ++++
 mm/migrate.c            | 68 ++++++++---------------------------------
 3 files changed, 19 insertions(+), 56 deletions(-)

diff --git a/include/linux/migrate.h b/include/linux/migrate.h
index 7993faffa46d..cb67692e659a 100644
--- a/include/linux/migrate.h
+++ b/include/linux/migrate.h
@@ -52,6 +52,7 @@ extern int migrate_huge_page_move_mapping(struct address_space *mapping,
 extern int migrate_page_move_mapping(struct address_space *mapping,
 		struct page *newpage, struct page *page, int extra_count);
 void folio_migrate_flags(struct folio *newfolio, struct folio *folio);
+void folio_migrate_copy(struct folio *newfolio, struct folio *folio);
 int folio_migrate_mapping(struct address_space *mapping,
 		struct folio *newfolio, struct folio *folio, int extra_count);
 #else
diff --git a/mm/folio-compat.c b/mm/folio-compat.c
index f0ac904d396f..316c912e49e0 100644
--- a/mm/folio-compat.c
+++ b/mm/folio-compat.c
@@ -76,4 +76,10 @@ void migrate_page_states(struct page *newpage, struct page *page)
 	folio_migrate_flags(page_folio(newpage), page_folio(page));
 }
 EXPORT_SYMBOL(migrate_page_states);
+
+void migrate_page_copy(struct page *newpage, struct page *page)
+{
+	folio_migrate_copy(page_folio(newpage), page_folio(page));
+}
+EXPORT_SYMBOL(migrate_page_copy);
 #endif
diff --git a/mm/migrate.c b/mm/migrate.c
index 8b289156f4c6..bc1362f367ae 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -529,54 +529,6 @@ int migrate_huge_page_move_mapping(struct address_space *mapping,
 	return MIGRATEPAGE_SUCCESS;
 }
 
-/*
- * Gigantic pages are so large that we do not guarantee that page++ pointer
- * arithmetic will work across the entire page.  We need something more
- * specialized.
- */
-static void __copy_gigantic_page(struct page *dst, struct page *src,
-				int nr_pages)
-{
-	int i;
-	struct page *dst_base = dst;
-	struct page *src_base = src;
-
-	for (i = 0; i < nr_pages; ) {
-		cond_resched();
-		copy_highpage(dst, src);
-
-		i++;
-		dst = mem_map_next(dst, dst_base, i);
-		src = mem_map_next(src, src_base, i);
-	}
-}
-
-static void copy_huge_page(struct page *dst, struct page *src)
-{
-	int i;
-	int nr_pages;
-
-	if (PageHuge(src)) {
-		/* hugetlbfs page */
-		struct hstate *h = page_hstate(src);
-		nr_pages = pages_per_huge_page(h);
-
-		if (unlikely(nr_pages > MAX_ORDER_NR_PAGES)) {
-			__copy_gigantic_page(dst, src, nr_pages);
-			return;
-		}
-	} else {
-		/* thp page */
-		BUG_ON(!PageTransHuge(src));
-		nr_pages = thp_nr_pages(src);
-	}
-
-	for (i = 0; i < nr_pages; i++) {
-		cond_resched();
-		copy_highpage(dst + i, src + i);
-	}
-}
-
 /*
  * Copy the flags and some other ancillary information
  */
@@ -653,16 +605,20 @@ void folio_migrate_flags(struct folio *newfolio, struct folio *folio)
 }
 EXPORT_SYMBOL(folio_migrate_flags);
 
-void migrate_page_copy(struct page *newpage, struct page *page)
+void folio_migrate_copy(struct folio *newfolio, struct folio *folio)
 {
-	if (PageHuge(page) || PageTransHuge(page))
-		copy_huge_page(newpage, page);
-	else
-		copy_highpage(newpage, page);
+	unsigned int i = folio_nr_pages(folio) - 1;
 
-	migrate_page_states(newpage, page);
+	copy_highpage(folio_page(newfolio, i), folio_page(folio, i));
+	while (i-- > 0) {
+		cond_resched();
+		/* folio_page() handles discontinuities in memmap */
+		copy_highpage(folio_page(newfolio, i), folio_page(folio, i));
+	}
+
+	folio_migrate_flags(newfolio, folio);
 }
-EXPORT_SYMBOL(migrate_page_copy);
+EXPORT_SYMBOL(folio_migrate_copy);
 
 /************************************************************
  *                    Migration functions
@@ -690,7 +646,7 @@ int migrate_page(struct address_space *mapping,
 		return rc;
 
 	if (mode != MIGRATE_SYNC_NO_COPY)
-		migrate_page_copy(newpage, page);
+		folio_migrate_copy(newfolio, folio);
 	else
 		folio_migrate_flags(newfolio, folio);
 	return MIGRATEPAGE_SUCCESS;
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 128+ messages in thread

* [PATCH v2 21/46] mm/writeback: Rename __add_wb_stat() to wb_stat_mod()
  2021-06-22 12:15 [PATCH v2 00/46] Folio-enabling the page cache Matthew Wilcox (Oracle)
                   ` (19 preceding siblings ...)
  2021-06-22 12:15 ` [PATCH v2 20/46] mm/migrate: Add folio_migrate_copy() Matthew Wilcox (Oracle)
@ 2021-06-22 12:15 ` Matthew Wilcox (Oracle)
  2021-06-23  8:37   ` Christoph Hellwig
  2021-06-22 12:15 ` [PATCH v2 22/46] flex_proportions: Allow N events instead of 1 Matthew Wilcox (Oracle)
                   ` (24 subsequent siblings)
  45 siblings, 1 reply; 128+ messages in thread
From: Matthew Wilcox (Oracle) @ 2021-06-22 12:15 UTC (permalink / raw)
  To: akpm; +Cc: Matthew Wilcox (Oracle), linux-fsdevel, linux-mm, linux-kernel

Make this look like the newly renamed vmstat functions.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 include/linux/backing-dev.h | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/include/linux/backing-dev.h b/include/linux/backing-dev.h
index 44df4fcef65c..a852876bb6e2 100644
--- a/include/linux/backing-dev.h
+++ b/include/linux/backing-dev.h
@@ -64,7 +64,7 @@ static inline bool bdi_has_dirty_io(struct backing_dev_info *bdi)
 	return atomic_long_read(&bdi->tot_write_bandwidth);
 }
 
-static inline void __add_wb_stat(struct bdi_writeback *wb,
+static inline void wb_stat_mod(struct bdi_writeback *wb,
 				 enum wb_stat_item item, s64 amount)
 {
 	percpu_counter_add_batch(&wb->stat[item], amount, WB_STAT_BATCH);
@@ -72,12 +72,12 @@ static inline void __add_wb_stat(struct bdi_writeback *wb,
 
 static inline void inc_wb_stat(struct bdi_writeback *wb, enum wb_stat_item item)
 {
-	__add_wb_stat(wb, item, 1);
+	wb_stat_mod(wb, item, 1);
 }
 
 static inline void dec_wb_stat(struct bdi_writeback *wb, enum wb_stat_item item)
 {
-	__add_wb_stat(wb, item, -1);
+	wb_stat_mod(wb, item, -1);
 }
 
 static inline s64 wb_stat(struct bdi_writeback *wb, enum wb_stat_item item)
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 128+ messages in thread

* [PATCH v2 22/46] flex_proportions: Allow N events instead of 1
  2021-06-22 12:15 [PATCH v2 00/46] Folio-enabling the page cache Matthew Wilcox (Oracle)
                   ` (20 preceding siblings ...)
  2021-06-22 12:15 ` [PATCH v2 21/46] mm/writeback: Rename __add_wb_stat() to wb_stat_mod() Matthew Wilcox (Oracle)
@ 2021-06-22 12:15 ` Matthew Wilcox (Oracle)
  2021-06-23  8:41   ` Christoph Hellwig
  2021-06-22 12:15 ` [PATCH v2 23/46] mm/writeback: Change __wb_writeout_inc() to __wb_writeout_add() Matthew Wilcox (Oracle)
                   ` (23 subsequent siblings)
  45 siblings, 1 reply; 128+ messages in thread
From: Matthew Wilcox (Oracle) @ 2021-06-22 12:15 UTC (permalink / raw)
  To: akpm; +Cc: Matthew Wilcox (Oracle), linux-fsdevel, linux-mm, linux-kernel

When batching events (such as writing back N pages in a single I/O), it
is better to do one flex_proportion operation instead of N.  There is
only one caller of __fprop_inc_percpu_max(), and it's the one we're
going to change in the next patch, so rename it instead of adding a
compatibility wrapper.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 include/linux/flex_proportions.h |  9 +++++----
 lib/flex_proportions.c           | 28 +++++++++++++++++++---------
 mm/page-writeback.c              |  4 ++--
 3 files changed, 26 insertions(+), 15 deletions(-)

diff --git a/include/linux/flex_proportions.h b/include/linux/flex_proportions.h
index c12df59d3f5f..3e378b1fb0bc 100644
--- a/include/linux/flex_proportions.h
+++ b/include/linux/flex_proportions.h
@@ -83,9 +83,10 @@ struct fprop_local_percpu {
 
 int fprop_local_init_percpu(struct fprop_local_percpu *pl, gfp_t gfp);
 void fprop_local_destroy_percpu(struct fprop_local_percpu *pl);
-void __fprop_inc_percpu(struct fprop_global *p, struct fprop_local_percpu *pl);
-void __fprop_inc_percpu_max(struct fprop_global *p, struct fprop_local_percpu *pl,
-			    int max_frac);
+void __fprop_add_percpu(struct fprop_global *p, struct fprop_local_percpu *pl,
+		long nr);
+void __fprop_add_percpu_max(struct fprop_global *p,
+		struct fprop_local_percpu *pl, int max_frac, long nr);
 void fprop_fraction_percpu(struct fprop_global *p,
 	struct fprop_local_percpu *pl, unsigned long *numerator,
 	unsigned long *denominator);
@@ -96,7 +97,7 @@ void fprop_inc_percpu(struct fprop_global *p, struct fprop_local_percpu *pl)
 	unsigned long flags;
 
 	local_irq_save(flags);
-	__fprop_inc_percpu(p, pl);
+	__fprop_add_percpu(p, pl, 1);
 	local_irq_restore(flags);
 }
 
diff --git a/lib/flex_proportions.c b/lib/flex_proportions.c
index 451543937524..53e7eb1dd76c 100644
--- a/lib/flex_proportions.c
+++ b/lib/flex_proportions.c
@@ -217,11 +217,12 @@ static void fprop_reflect_period_percpu(struct fprop_global *p,
 }
 
 /* Event of type pl happened */
-void __fprop_inc_percpu(struct fprop_global *p, struct fprop_local_percpu *pl)
+void __fprop_add_percpu(struct fprop_global *p, struct fprop_local_percpu *pl,
+		long nr)
 {
 	fprop_reflect_period_percpu(p, pl);
-	percpu_counter_add_batch(&pl->events, 1, PROP_BATCH);
-	percpu_counter_add(&p->events, 1);
+	percpu_counter_add_batch(&pl->events, nr, PROP_BATCH);
+	percpu_counter_add(&p->events, nr);
 }
 
 void fprop_fraction_percpu(struct fprop_global *p,
@@ -253,20 +254,29 @@ void fprop_fraction_percpu(struct fprop_global *p,
 }
 
 /*
- * Like __fprop_inc_percpu() except that event is counted only if the given
+ * Like __fprop_add_percpu() except that event is counted only if the given
  * type has fraction smaller than @max_frac/FPROP_FRAC_BASE
  */
-void __fprop_inc_percpu_max(struct fprop_global *p,
-			    struct fprop_local_percpu *pl, int max_frac)
+void __fprop_add_percpu_max(struct fprop_global *p,
+		struct fprop_local_percpu *pl, int max_frac, long nr)
 {
 	if (unlikely(max_frac < FPROP_FRAC_BASE)) {
 		unsigned long numerator, denominator;
+		s64 tmp;
 
 		fprop_fraction_percpu(p, pl, &numerator, &denominator);
-		if (numerator >
-		    (((u64)denominator) * max_frac) >> FPROP_FRAC_SHIFT)
+		/* Adding 'nr' to fraction exceeds max_frac/FPROP_FRAC_BASE? */
+		tmp = (u64)denominator * max_frac -
+					((u64)numerator << FPROP_FRAC_SHIFT);
+		if (tmp < 0) {
+			/* Maximum fraction already exceeded? */
 			return;
+		} else if (tmp < nr * (FPROP_FRAC_BASE - max_frac)) {
+			/* Add just enough for the fraction to saturate */
+			nr = div_u64(tmp + FPROP_FRAC_BASE - max_frac - 1,
+					FPROP_FRAC_BASE - max_frac);
+		}
 	}
 
-	__fprop_inc_percpu(p, pl);
+	__fprop_add_percpu(p, pl, nr);
 }
diff --git a/mm/page-writeback.c b/mm/page-writeback.c
index 7ad5538f2489..8079d5ef19df 100644
--- a/mm/page-writeback.c
+++ b/mm/page-writeback.c
@@ -571,8 +571,8 @@ static void wb_domain_writeout_inc(struct wb_domain *dom,
 				   struct fprop_local_percpu *completions,
 				   unsigned int max_prop_frac)
 {
-	__fprop_inc_percpu_max(&dom->completions, completions,
-			       max_prop_frac);
+	__fprop_add_percpu_max(&dom->completions, completions,
+			       max_prop_frac, 1);
 	/* First event after period switching was turned off? */
 	if (unlikely(!dom->period_time)) {
 		/*
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 128+ messages in thread

* [PATCH v2 23/46] mm/writeback: Change __wb_writeout_inc() to __wb_writeout_add()
  2021-06-22 12:15 [PATCH v2 00/46] Folio-enabling the page cache Matthew Wilcox (Oracle)
                   ` (21 preceding siblings ...)
  2021-06-22 12:15 ` [PATCH v2 22/46] flex_proportions: Allow N events instead of 1 Matthew Wilcox (Oracle)
@ 2021-06-22 12:15 ` Matthew Wilcox (Oracle)
  2021-06-23  8:45   ` Christoph Hellwig
  2021-06-22 12:15 ` [PATCH v2 24/46] mm/writeback: Add __folio_end_writeback() Matthew Wilcox (Oracle)
                   ` (22 subsequent siblings)
  45 siblings, 1 reply; 128+ messages in thread
From: Matthew Wilcox (Oracle) @ 2021-06-22 12:15 UTC (permalink / raw)
  To: akpm; +Cc: Matthew Wilcox (Oracle), linux-fsdevel, linux-mm, linux-kernel

Allow for accounting N pages at once instead of one page at a time.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 mm/page-writeback.c | 22 +++++++++++-----------
 1 file changed, 11 insertions(+), 11 deletions(-)

diff --git a/mm/page-writeback.c b/mm/page-writeback.c
index 8079d5ef19df..dc66ff78f033 100644
--- a/mm/page-writeback.c
+++ b/mm/page-writeback.c
@@ -567,12 +567,12 @@ static unsigned long wp_next_time(unsigned long cur_time)
 	return cur_time;
 }
 
-static void wb_domain_writeout_inc(struct wb_domain *dom,
+static void wb_domain_writeout_add(struct wb_domain *dom,
 				   struct fprop_local_percpu *completions,
-				   unsigned int max_prop_frac)
+				   unsigned int max_prop_frac, long nr)
 {
 	__fprop_add_percpu_max(&dom->completions, completions,
-			       max_prop_frac, 1);
+			       max_prop_frac, nr);
 	/* First event after period switching was turned off? */
 	if (unlikely(!dom->period_time)) {
 		/*
@@ -590,18 +590,18 @@ static void wb_domain_writeout_inc(struct wb_domain *dom,
  * Increment @wb's writeout completion count and the global writeout
  * completion count. Called from test_clear_page_writeback().
  */
-static inline void __wb_writeout_inc(struct bdi_writeback *wb)
+static inline void __wb_writeout_add(struct bdi_writeback *wb, long nr)
 {
 	struct wb_domain *cgdom;
 
-	inc_wb_stat(wb, WB_WRITTEN);
-	wb_domain_writeout_inc(&global_wb_domain, &wb->completions,
-			       wb->bdi->max_prop_frac);
+	wb_stat_mod(wb, WB_WRITTEN, nr);
+	wb_domain_writeout_add(&global_wb_domain, &wb->completions,
+			       wb->bdi->max_prop_frac, nr);
 
 	cgdom = mem_cgroup_wb_domain(wb);
 	if (cgdom)
-		wb_domain_writeout_inc(cgdom, wb_memcg_completions(wb),
-				       wb->bdi->max_prop_frac);
+		wb_domain_writeout_add(cgdom, wb_memcg_completions(wb),
+				       wb->bdi->max_prop_frac, nr);
 }
 
 void wb_writeout_inc(struct bdi_writeback *wb)
@@ -609,7 +609,7 @@ void wb_writeout_inc(struct bdi_writeback *wb)
 	unsigned long flags;
 
 	local_irq_save(flags);
-	__wb_writeout_inc(wb);
+	__wb_writeout_add(wb, 1);
 	local_irq_restore(flags);
 }
 EXPORT_SYMBOL_GPL(wb_writeout_inc);
@@ -2747,7 +2747,7 @@ int test_clear_page_writeback(struct page *page)
 				struct bdi_writeback *wb = inode_to_wb(inode);
 
 				dec_wb_stat(wb, WB_WRITEBACK);
-				__wb_writeout_inc(wb);
+				__wb_writeout_add(wb, 1);
 			}
 		}
 
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 128+ messages in thread

* [PATCH v2 24/46] mm/writeback: Add __folio_end_writeback()
  2021-06-22 12:15 [PATCH v2 00/46] Folio-enabling the page cache Matthew Wilcox (Oracle)
                   ` (22 preceding siblings ...)
  2021-06-22 12:15 ` [PATCH v2 23/46] mm/writeback: Change __wb_writeout_inc() to __wb_writeout_add() Matthew Wilcox (Oracle)
@ 2021-06-22 12:15 ` Matthew Wilcox (Oracle)
  2021-06-23  9:15   ` Christoph Hellwig
  2021-06-22 12:15 ` [PATCH v2 25/46] mm/writeback: Add folio_start_writeback() Matthew Wilcox (Oracle)
                   ` (21 subsequent siblings)
  45 siblings, 1 reply; 128+ messages in thread
From: Matthew Wilcox (Oracle) @ 2021-06-22 12:15 UTC (permalink / raw)
  To: akpm; +Cc: Matthew Wilcox (Oracle), linux-fsdevel, linux-mm, linux-kernel

test_clear_page_writeback() is actually an mm-internal function, although
it's named as if it's a pagecache function.  Move it to mm/internal.h,
rename it to __folio_end_writeback() and change the return type to bool.

The conversion from page to folio is mostly about accounting the number
of pages being written back, although it does eliminate a couple of
calls to compound_head().

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 include/linux/page-flags.h |  1 -
 mm/filemap.c               |  2 +-
 mm/internal.h              |  1 +
 mm/page-writeback.c        | 29 +++++++++++++++--------------
 4 files changed, 17 insertions(+), 16 deletions(-)

diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h
index 50df34886537..bdf807c98736 100644
--- a/include/linux/page-flags.h
+++ b/include/linux/page-flags.h
@@ -646,7 +646,6 @@ static __always_inline void SetPageUptodate(struct page *page)
 
 CLEARPAGEFLAG(Uptodate, uptodate, PF_NO_TAIL)
 
-int test_clear_page_writeback(struct page *page);
 int __test_set_page_writeback(struct page *page, bool keep_write);
 
 #define test_set_page_writeback(page)			\
diff --git a/mm/filemap.c b/mm/filemap.c
index 4b2698e5e8e2..dd360721c72b 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -1535,7 +1535,7 @@ void folio_end_writeback(struct folio *folio)
 	 * reused before the folio_wake().
 	 */
 	folio_get(folio);
-	if (!test_clear_page_writeback(&folio->page))
+	if (!__folio_end_writeback(folio))
 		BUG();
 
 	smp_mb__after_atomic();
diff --git a/mm/internal.h b/mm/internal.h
index 3e70121c71c7..d7013df5d1f0 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -43,6 +43,7 @@ static inline void *folio_rmapping(struct folio *folio)
 
 vm_fault_t do_swap_page(struct vm_fault *vmf);
 void folio_rotate_reclaimable(struct folio *folio);
+bool __folio_end_writeback(struct folio *folio);
 
 void free_pgtables(struct mmu_gather *tlb, struct vm_area_struct *start_vma,
 		unsigned long floor, unsigned long ceiling);
diff --git a/mm/page-writeback.c b/mm/page-writeback.c
index dc66ff78f033..9213b8b6d50b 100644
--- a/mm/page-writeback.c
+++ b/mm/page-writeback.c
@@ -588,7 +588,7 @@ static void wb_domain_writeout_add(struct wb_domain *dom,
 
 /*
  * Increment @wb's writeout completion count and the global writeout
- * completion count. Called from test_clear_page_writeback().
+ * completion count. Called from __folio_end_writeback().
  */
 static inline void __wb_writeout_add(struct bdi_writeback *wb, long nr)
 {
@@ -2727,27 +2727,28 @@ int clear_page_dirty_for_io(struct page *page)
 }
 EXPORT_SYMBOL(clear_page_dirty_for_io);
 
-int test_clear_page_writeback(struct page *page)
+bool __folio_end_writeback(struct folio *folio)
 {
-	struct address_space *mapping = page_mapping(page);
-	int ret;
+	long nr = folio_nr_pages(folio);
+	struct address_space *mapping = folio_mapping(folio);
+	bool ret;
 
-	lock_page_memcg(page);
+	lock_folio_memcg(folio);
 	if (mapping && mapping_use_writeback_tags(mapping)) {
 		struct inode *inode = mapping->host;
 		struct backing_dev_info *bdi = inode_to_bdi(inode);
 		unsigned long flags;
 
 		xa_lock_irqsave(&mapping->i_pages, flags);
-		ret = TestClearPageWriteback(page);
+		ret = folio_test_clear_writeback_flag(folio);
 		if (ret) {
-			__xa_clear_mark(&mapping->i_pages, page_index(page),
+			__xa_clear_mark(&mapping->i_pages, folio_index(folio),
 						PAGECACHE_TAG_WRITEBACK);
 			if (bdi->capabilities & BDI_CAP_WRITEBACK_ACCT) {
 				struct bdi_writeback *wb = inode_to_wb(inode);
 
-				dec_wb_stat(wb, WB_WRITEBACK);
-				__wb_writeout_add(wb, 1);
+				wb_stat_mod(wb, WB_WRITEBACK, -nr);
+				__wb_writeout_add(wb, nr);
 			}
 		}
 
@@ -2757,14 +2758,14 @@ int test_clear_page_writeback(struct page *page)
 
 		xa_unlock_irqrestore(&mapping->i_pages, flags);
 	} else {
-		ret = TestClearPageWriteback(page);
+		ret = folio_test_clear_writeback_flag(folio);
 	}
 	if (ret) {
-		dec_lruvec_page_state(page, NR_WRITEBACK);
-		dec_zone_page_state(page, NR_ZONE_WRITE_PENDING);
-		inc_node_page_state(page, NR_WRITTEN);
+		lruvec_stat_mod_folio(folio, NR_WRITEBACK, -nr);
+		zone_stat_mod_folio(folio, NR_ZONE_WRITE_PENDING, -nr);
+		node_stat_mod_folio(folio, NR_WRITTEN, nr);
 	}
-	unlock_page_memcg(page);
+	unlock_folio_memcg(folio);
 	return ret;
 }
 
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 128+ messages in thread

* [PATCH v2 25/46] mm/writeback: Add folio_start_writeback()
  2021-06-22 12:15 [PATCH v2 00/46] Folio-enabling the page cache Matthew Wilcox (Oracle)
                   ` (23 preceding siblings ...)
  2021-06-22 12:15 ` [PATCH v2 24/46] mm/writeback: Add __folio_end_writeback() Matthew Wilcox (Oracle)
@ 2021-06-22 12:15 ` Matthew Wilcox (Oracle)
  2021-06-23  9:18   ` Christoph Hellwig
  2021-06-22 12:15 ` [PATCH v2 26/46] mm/writeback: Add folio_mark_dirty() Matthew Wilcox (Oracle)
                   ` (20 subsequent siblings)
  45 siblings, 1 reply; 128+ messages in thread
From: Matthew Wilcox (Oracle) @ 2021-06-22 12:15 UTC (permalink / raw)
  To: akpm; +Cc: Matthew Wilcox (Oracle), linux-fsdevel, linux-mm, linux-kernel

Rename set_page_writeback() to folio_start_writeback() to match
folio_end_writeback().  Do not bother with wrappers that return void;
callers are perfectly capable of ignoring return values.

Add wrappers for set_page_writeback(), set_page_writeback_keepwrite() and
test_set_page_writeback() for compatibililty with existing filesystems.
The main advantage of this patch is getting the statistics right,
although it does eliminate a couple of calls to compound_head().

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 include/linux/page-flags.h | 19 +++++++++---------
 mm/folio-compat.c          |  6 ++++++
 mm/page-writeback.c        | 40 ++++++++++++++++++++------------------
 3 files changed, 37 insertions(+), 28 deletions(-)

diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h
index bdf807c98736..e40a3c6387cc 100644
--- a/include/linux/page-flags.h
+++ b/include/linux/page-flags.h
@@ -646,21 +646,22 @@ static __always_inline void SetPageUptodate(struct page *page)
 
 CLEARPAGEFLAG(Uptodate, uptodate, PF_NO_TAIL)
 
-int __test_set_page_writeback(struct page *page, bool keep_write);
+bool __folio_start_writeback(struct folio *folio, bool keep_write);
+bool set_page_writeback(struct page *page);
 
-#define test_set_page_writeback(page)			\
-	__test_set_page_writeback(page, false)
-#define test_set_page_writeback_keepwrite(page)	\
-	__test_set_page_writeback(page, true)
+#define folio_start_writeback(folio)			\
+	__folio_start_writeback(folio, false)
+#define folio_start_writeback_keepwrite(folio)	\
+	__folio_start_writeback(folio, true)
 
-static inline void set_page_writeback(struct page *page)
+static inline void set_page_writeback_keepwrite(struct page *page)
 {
-	test_set_page_writeback(page);
+	folio_start_writeback_keepwrite(page_folio(page));
 }
 
-static inline void set_page_writeback_keepwrite(struct page *page)
+static inline bool test_set_page_writeback(struct page *page)
 {
-	test_set_page_writeback_keepwrite(page);
+	return set_page_writeback(page);
 }
 
 __PAGEFLAG(Head, head, PF_ANY) CLEARPAGEFLAG(Head, head, PF_ANY)
diff --git a/mm/folio-compat.c b/mm/folio-compat.c
index 316c912e49e0..db9190fa6bf5 100644
--- a/mm/folio-compat.c
+++ b/mm/folio-compat.c
@@ -83,3 +83,9 @@ void migrate_page_copy(struct page *newpage, struct page *page)
 }
 EXPORT_SYMBOL(migrate_page_copy);
 #endif
+
+bool set_page_writeback(struct page *page)
+{
+	return folio_start_writeback(page_folio(page));
+}
+EXPORT_SYMBOL(set_page_writeback);
diff --git a/mm/page-writeback.c b/mm/page-writeback.c
index 9213b8b6d50b..ffb55ca2ddd5 100644
--- a/mm/page-writeback.c
+++ b/mm/page-writeback.c
@@ -2769,21 +2769,23 @@ bool __folio_end_writeback(struct folio *folio)
 	return ret;
 }
 
-int __test_set_page_writeback(struct page *page, bool keep_write)
+bool __folio_start_writeback(struct folio *folio, bool keep_write)
 {
-	struct address_space *mapping = page_mapping(page);
-	int ret, access_ret;
+	long nr = folio_nr_pages(folio);
+	struct address_space *mapping = folio_mapping(folio);
+	bool ret;
+	int access_ret;
 
-	lock_page_memcg(page);
+	lock_folio_memcg(folio);
 	if (mapping && mapping_use_writeback_tags(mapping)) {
-		XA_STATE(xas, &mapping->i_pages, page_index(page));
+		XA_STATE(xas, &mapping->i_pages, folio_index(folio));
 		struct inode *inode = mapping->host;
 		struct backing_dev_info *bdi = inode_to_bdi(inode);
 		unsigned long flags;
 
 		xas_lock_irqsave(&xas, flags);
 		xas_load(&xas);
-		ret = TestSetPageWriteback(page);
+		ret = folio_test_set_writeback_flag(folio);
 		if (!ret) {
 			bool on_wblist;
 
@@ -2792,40 +2794,40 @@ int __test_set_page_writeback(struct page *page, bool keep_write)
 
 			xas_set_mark(&xas, PAGECACHE_TAG_WRITEBACK);
 			if (bdi->capabilities & BDI_CAP_WRITEBACK_ACCT)
-				inc_wb_stat(inode_to_wb(inode), WB_WRITEBACK);
+				wb_stat_mod(inode_to_wb(inode), WB_WRITEBACK,
+						nr);
 
 			/*
-			 * We can come through here when swapping anonymous
-			 * pages, so we don't necessarily have an inode to track
-			 * for sync.
+			 * We can come through here when swapping
+			 * anonymous folios, so we don't necessarily
+			 * have an inode to track for sync.
 			 */
 			if (mapping->host && !on_wblist)
 				sb_mark_inode_writeback(mapping->host);
 		}
-		if (!PageDirty(page))
+		if (!folio_dirty(folio))
 			xas_clear_mark(&xas, PAGECACHE_TAG_DIRTY);
 		if (!keep_write)
 			xas_clear_mark(&xas, PAGECACHE_TAG_TOWRITE);
 		xas_unlock_irqrestore(&xas, flags);
 	} else {
-		ret = TestSetPageWriteback(page);
+		ret = folio_test_set_writeback_flag(folio);
 	}
 	if (!ret) {
-		inc_lruvec_page_state(page, NR_WRITEBACK);
-		inc_zone_page_state(page, NR_ZONE_WRITE_PENDING);
+		lruvec_stat_mod_folio(folio, NR_WRITEBACK, nr);
+		zone_stat_mod_folio(folio, NR_ZONE_WRITE_PENDING, nr);
 	}
-	unlock_page_memcg(page);
-	access_ret = arch_make_page_accessible(page);
+	unlock_folio_memcg(folio);
+	access_ret = arch_make_folio_accessible(folio);
 	/*
 	 * If writeback has been triggered on a page that cannot be made
 	 * accessible, it is too late to recover here.
 	 */
-	VM_BUG_ON_PAGE(access_ret != 0, page);
+	VM_BUG_ON_FOLIO(access_ret != 0, folio);
 
 	return ret;
-
 }
-EXPORT_SYMBOL(__test_set_page_writeback);
+EXPORT_SYMBOL(__folio_start_writeback);
 
 /**
  * folio_wait_writeback - Wait for a folio to finish writeback.
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 128+ messages in thread

* [PATCH v2 26/46] mm/writeback: Add folio_mark_dirty()
  2021-06-22 12:15 [PATCH v2 00/46] Folio-enabling the page cache Matthew Wilcox (Oracle)
                   ` (24 preceding siblings ...)
  2021-06-22 12:15 ` [PATCH v2 25/46] mm/writeback: Add folio_start_writeback() Matthew Wilcox (Oracle)
@ 2021-06-22 12:15 ` Matthew Wilcox (Oracle)
  2021-06-23  9:21   ` Christoph Hellwig
  2021-06-22 12:15 ` [PATCH v2 27/46] mm/writeback: Add __folio_mark_dirty() Matthew Wilcox (Oracle)
                   ` (19 subsequent siblings)
  45 siblings, 1 reply; 128+ messages in thread
From: Matthew Wilcox (Oracle) @ 2021-06-22 12:15 UTC (permalink / raw)
  To: akpm; +Cc: Matthew Wilcox (Oracle), linux-fsdevel, linux-mm, linux-kernel

Reimplement set_page_dirty() as a wrapper around folio_mark_dirty().
There is no change to filesystems as they were already being called
with the compound_head of the page being marked dirty.  We avoid
several calls to compound_head(), both statically (through
using folio_dirty() instead of PageDirty() and dynamically by
calling folio_mapping() instead of page_mapping().

Also return bool instead of int to show the range of values actually
returned, and add kernel-doc.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 include/linux/mm.h  |  3 ++-
 mm/folio-compat.c   |  6 ++++++
 mm/page-writeback.c | 35 +++++++++++++++++++----------------
 3 files changed, 27 insertions(+), 17 deletions(-)

diff --git a/include/linux/mm.h b/include/linux/mm.h
index 5609095ffcac..3c8dfcb56fa5 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -1988,7 +1988,8 @@ int redirty_page_for_writepage(struct writeback_control *wbc,
 				struct page *page);
 void account_page_cleaned(struct page *page, struct address_space *mapping,
 			  struct bdi_writeback *wb);
-int set_page_dirty(struct page *page);
+bool folio_mark_dirty(struct folio *folio);
+bool set_page_dirty(struct page *page);
 int set_page_dirty_lock(struct page *page);
 void __cancel_dirty_page(struct page *page);
 static inline void cancel_dirty_page(struct page *page)
diff --git a/mm/folio-compat.c b/mm/folio-compat.c
index db9190fa6bf5..60780ceca363 100644
--- a/mm/folio-compat.c
+++ b/mm/folio-compat.c
@@ -89,3 +89,9 @@ bool set_page_writeback(struct page *page)
 	return folio_start_writeback(page_folio(page));
 }
 EXPORT_SYMBOL(set_page_writeback);
+
+bool set_page_dirty(struct page *page)
+{
+	return folio_mark_dirty(page_folio(page));
+}
+EXPORT_SYMBOL(set_page_dirty);
diff --git a/mm/page-writeback.c b/mm/page-writeback.c
index ffb55ca2ddd5..73b937955cc1 100644
--- a/mm/page-writeback.c
+++ b/mm/page-writeback.c
@@ -2560,18 +2560,21 @@ int redirty_page_for_writepage(struct writeback_control *wbc, struct page *page)
 }
 EXPORT_SYMBOL(redirty_page_for_writepage);
 
-/*
- * Dirty a page.
+/**
+ * folio_mark_dirty - Mark a folio as being modified.
+ * @folio: The folio.
+ *
+ * For folios with a mapping this should be done under the page lock
+ * for the benefit of asynchronous memory errors who prefer a consistent
+ * dirty state. This rule can be broken in some special cases,
+ * but should be better not to.
  *
- * For pages with a mapping this should be done under the page lock for the
- * benefit of asynchronous memory errors who prefer a consistent dirty state.
- * This rule can be broken in some special cases, but should be better not to.
+ * Return: True if the folio was newly dirtied, false if it was already dirty.
  */
-int set_page_dirty(struct page *page)
+bool folio_mark_dirty(struct folio *folio)
 {
-	struct address_space *mapping = page_mapping(page);
+	struct address_space *mapping = folio_mapping(folio);
 
-	page = compound_head(page);
 	if (likely(mapping)) {
 		/*
 		 * readahead/lru_deactivate_page could remain
@@ -2583,17 +2586,17 @@ int set_page_dirty(struct page *page)
 		 * it will confuse readahead and make it restart the size rampup
 		 * process. But it's a trivial problem.
 		 */
-		if (PageReclaim(page))
-			ClearPageReclaim(page);
-		return mapping->a_ops->set_page_dirty(page);
+		if (folio_reclaim(folio))
+			folio_clear_reclaim_flag(folio);
+		return mapping->a_ops->set_page_dirty(&folio->page);
 	}
-	if (!PageDirty(page)) {
-		if (!TestSetPageDirty(page))
-			return 1;
+	if (!folio_dirty(folio)) {
+		if (!folio_test_set_dirty_flag(folio))
+			return true;
 	}
-	return 0;
+	return false;
 }
-EXPORT_SYMBOL(set_page_dirty);
+EXPORT_SYMBOL(folio_mark_dirty);
 
 /*
  * set_page_dirty() is racy if the caller has no reference against
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 128+ messages in thread

* [PATCH v2 27/46] mm/writeback: Add __folio_mark_dirty()
  2021-06-22 12:15 [PATCH v2 00/46] Folio-enabling the page cache Matthew Wilcox (Oracle)
                   ` (25 preceding siblings ...)
  2021-06-22 12:15 ` [PATCH v2 26/46] mm/writeback: Add folio_mark_dirty() Matthew Wilcox (Oracle)
@ 2021-06-22 12:15 ` Matthew Wilcox (Oracle)
  2021-06-23  9:27   ` Christoph Hellwig
  2021-06-22 12:15 ` [PATCH v2 28/46] mm/writeback: Add filemap_dirty_folio() Matthew Wilcox (Oracle)
                   ` (18 subsequent siblings)
  45 siblings, 1 reply; 128+ messages in thread
From: Matthew Wilcox (Oracle) @ 2021-06-22 12:15 UTC (permalink / raw)
  To: akpm; +Cc: Matthew Wilcox (Oracle), linux-fsdevel, linux-mm, linux-kernel

Turn __set_page_dirty() into a wrapper around __folio_mark_dirty() (which
can directly cast from page to folio because we know that set_page_dirty()
calls filesystems with the head page).  Convert account_page_dirtied()
into folio_account_dirtied() and account the number of pages in the folio.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 include/linux/memcontrol.h |  5 ++---
 include/linux/pagemap.h    |  7 ++++++-
 mm/page-writeback.c        | 41 +++++++++++++++++++-------------------
 3 files changed, 29 insertions(+), 24 deletions(-)

diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
index 00693cb48b5d..f9f05724db3b 100644
--- a/include/linux/memcontrol.h
+++ b/include/linux/memcontrol.h
@@ -1638,10 +1638,9 @@ void mem_cgroup_wb_stats(struct bdi_writeback *wb, unsigned long *pfilepages,
 void mem_cgroup_track_foreign_dirty_slowpath(struct folio *folio,
 					     struct bdi_writeback *wb);
 
-static inline void mem_cgroup_track_foreign_dirty(struct page *page,
+static inline void mem_cgroup_track_foreign_dirty(struct folio *folio,
 						  struct bdi_writeback *wb)
 {
-	struct folio *folio = page_folio(page);
 	if (mem_cgroup_disabled())
 		return;
 
@@ -1666,7 +1665,7 @@ static inline void mem_cgroup_wb_stats(struct bdi_writeback *wb,
 {
 }
 
-static inline void mem_cgroup_track_foreign_dirty(struct page *page,
+static inline void mem_cgroup_track_foreign_dirty(struct folio *folio,
 						  struct bdi_writeback *wb)
 {
 }
diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
index 22d756d56404..e6a9756293aa 100644
--- a/include/linux/pagemap.h
+++ b/include/linux/pagemap.h
@@ -772,8 +772,13 @@ void end_page_writeback(struct page *page);
 void folio_end_writeback(struct folio *folio);
 void wait_for_stable_page(struct page *page);
 void folio_wait_stable(struct folio *folio);
+void __folio_mark_dirty(struct folio *folio, struct address_space *, int warn);
+static inline void __set_page_dirty(struct page *page,
+		struct address_space *mapping, int warn)
+{
+	__folio_mark_dirty((struct folio *)page, mapping, warn);
+}
 
-void __set_page_dirty(struct page *, struct address_space *, int warn);
 int __set_page_dirty_nobuffers(struct page *page);
 int __set_page_dirty_no_writeback(struct page *page);
 
diff --git a/mm/page-writeback.c b/mm/page-writeback.c
index 73b937955cc1..a7989870b171 100644
--- a/mm/page-writeback.c
+++ b/mm/page-writeback.c
@@ -2417,29 +2417,30 @@ EXPORT_SYMBOL(__set_page_dirty_no_writeback);
  *
  * NOTE: This relies on being atomic wrt interrupts.
  */
-static void account_page_dirtied(struct page *page,
+static void folio_account_dirtied(struct folio *folio,
 		struct address_space *mapping)
 {
 	struct inode *inode = mapping->host;
 
-	trace_writeback_dirty_page(page, mapping);
+	trace_writeback_dirty_page(&folio->page, mapping);
 
 	if (mapping_can_writeback(mapping)) {
 		struct bdi_writeback *wb;
+		long nr = folio_nr_pages(folio);
 
-		inode_attach_wb(inode, page);
+		inode_attach_wb(inode, &folio->page);
 		wb = inode_to_wb(inode);
 
-		__inc_lruvec_page_state(page, NR_FILE_DIRTY);
-		__inc_zone_page_state(page, NR_ZONE_WRITE_PENDING);
-		__inc_node_page_state(page, NR_DIRTIED);
-		inc_wb_stat(wb, WB_RECLAIMABLE);
-		inc_wb_stat(wb, WB_DIRTIED);
-		task_io_account_write(PAGE_SIZE);
-		current->nr_dirtied++;
-		this_cpu_inc(bdp_ratelimits);
+		__lruvec_stat_mod_folio(folio, NR_FILE_DIRTY, nr);
+		__zone_stat_mod_folio(folio, NR_ZONE_WRITE_PENDING, nr);
+		__node_stat_mod_folio(folio, NR_DIRTIED, nr);
+		wb_stat_mod(wb, WB_RECLAIMABLE, nr);
+		wb_stat_mod(wb, WB_DIRTIED, nr);
+		task_io_account_write(nr * PAGE_SIZE);
+		current->nr_dirtied += nr;
+		__this_cpu_add(bdp_ratelimits, nr);
 
-		mem_cgroup_track_foreign_dirty(page, wb);
+		mem_cgroup_track_foreign_dirty(folio, wb);
 	}
 }
 
@@ -2460,24 +2461,24 @@ void account_page_cleaned(struct page *page, struct address_space *mapping,
 }
 
 /*
- * Mark the page dirty, and set it dirty in the page cache, and mark the inode
- * dirty.
+ * Mark the folio dirty, and set it dirty in the page cache, and mark
+ * the inode dirty.
  *
- * If warn is true, then emit a warning if the page is not uptodate and has
+ * If warn is true, then emit a warning if the folio is not uptodate and has
  * not been truncated.
  *
  * The caller must hold lock_page_memcg().
  */
-void __set_page_dirty(struct page *page, struct address_space *mapping,
+void __folio_mark_dirty(struct folio *folio, struct address_space *mapping,
 			     int warn)
 {
 	unsigned long flags;
 
 	xa_lock_irqsave(&mapping->i_pages, flags);
-	if (page->mapping) {	/* Race with truncate? */
-		WARN_ON_ONCE(warn && !PageUptodate(page));
-		account_page_dirtied(page, mapping);
-		__xa_set_mark(&mapping->i_pages, page_index(page),
+	if (folio->mapping) {	/* Race with truncate? */
+		WARN_ON_ONCE(warn && !folio_uptodate(folio));
+		folio_account_dirtied(folio, mapping);
+		__xa_set_mark(&mapping->i_pages, folio_index(folio),
 				PAGECACHE_TAG_DIRTY);
 	}
 	xa_unlock_irqrestore(&mapping->i_pages, flags);
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 128+ messages in thread

* [PATCH v2 28/46] mm/writeback: Add filemap_dirty_folio()
  2021-06-22 12:15 [PATCH v2 00/46] Folio-enabling the page cache Matthew Wilcox (Oracle)
                   ` (26 preceding siblings ...)
  2021-06-22 12:15 ` [PATCH v2 27/46] mm/writeback: Add __folio_mark_dirty() Matthew Wilcox (Oracle)
@ 2021-06-22 12:15 ` Matthew Wilcox (Oracle)
  2021-06-23  9:32   ` Christoph Hellwig
  2021-06-22 12:15 ` [PATCH v2 29/46] mm/writeback: Add folio_account_cleaned() Matthew Wilcox (Oracle)
                   ` (17 subsequent siblings)
  45 siblings, 1 reply; 128+ messages in thread
From: Matthew Wilcox (Oracle) @ 2021-06-22 12:15 UTC (permalink / raw)
  To: akpm; +Cc: Matthew Wilcox (Oracle), linux-fsdevel, linux-mm, linux-kernel

Reimplement __set_page_dirty_nobuffers() as a wrapper around
filemap_dirty_folio().  This can use a cast to struct folio
because we know that the ->set_page_dirty address space op
is always called with a page pointer that happens to also be
a folio pointer.  Saves 7 bytes of kernel text.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 include/linux/writeback.h |  1 +
 mm/page-writeback.c       | 64 ++++++++++++++++++++++-----------------
 2 files changed, 37 insertions(+), 28 deletions(-)

diff --git a/include/linux/writeback.h b/include/linux/writeback.h
index 8e5c5bb16e2d..aa372f6d2b55 100644
--- a/include/linux/writeback.h
+++ b/include/linux/writeback.h
@@ -398,6 +398,7 @@ void writeback_set_ratelimit(void);
 void tag_pages_for_writeback(struct address_space *mapping,
 			     pgoff_t start, pgoff_t end);
 
+bool filemap_dirty_folio(struct address_space *mapping, struct folio *folio);
 void account_page_redirty(struct page *page);
 
 void sb_mark_inode_writeback(struct inode *inode);
diff --git a/mm/page-writeback.c b/mm/page-writeback.c
index a7989870b171..64b989eff9f5 100644
--- a/mm/page-writeback.c
+++ b/mm/page-writeback.c
@@ -2484,39 +2484,47 @@ void __folio_mark_dirty(struct folio *folio, struct address_space *mapping,
 	xa_unlock_irqrestore(&mapping->i_pages, flags);
 }
 
-/*
- * For address_spaces which do not use buffers.  Just tag the page as dirty in
- * the xarray.
- *
- * This is also used when a single buffer is being dirtied: we want to set the
- * page dirty in that case, but not all the buffers.  This is a "bottom-up"
- * dirtying, whereas __set_page_dirty_buffers() is a "top-down" dirtying.
- *
- * The caller must ensure this doesn't race with truncation.  Most will simply
- * hold the page lock, but e.g. zap_pte_range() calls with the page mapped and
- * the pte lock held, which also locks out truncation.
+/**
+ * filemap_dirty_folio - Mark a folio dirty for filesystems which do not use buffer_heads.
+ * @mapping: Address space this folio belongs to.
+ * @folio: Folio to be marked as dirty.
+ *
+ * Filesystems which do not use buffer heads should call this function
+ * from their set_page_dirty address space operation.  It ignores the
+ * contents of folio_private(), so if the filesystem marks individual
+ * blocks as dirty, the filesystem should handle that itself.
+ *
+ * This is also sometimes used by filesystems which use buffer_heads when
+ * a single buffer is being dirtied: we want to set the folio dirty in
+ * that case, but not all the buffers.  This is a "bottom-up" dirtying,
+ * whereas __set_page_dirty_buffers() is a "top-down" dirtying.
+ *
+ * The caller must ensure this doesn't race with truncation.  Most will
+ * simply hold the folio lock, but e.g. zap_pte_range() calls with the
+ * folio mapped and the pte lock held, which also locks out truncation.
  */
-int __set_page_dirty_nobuffers(struct page *page)
+bool filemap_dirty_folio(struct address_space *mapping, struct folio *folio)
 {
-	lock_page_memcg(page);
-	if (!TestSetPageDirty(page)) {
-		struct address_space *mapping = page_mapping(page);
+	lock_folio_memcg(folio);
+	if (folio_test_set_dirty_flag(folio)) {
+		unlock_folio_memcg(folio);
+		return false;
+	}
 
-		if (!mapping) {
-			unlock_page_memcg(page);
-			return 1;
-		}
-		__set_page_dirty(page, mapping, !PagePrivate(page));
-		unlock_page_memcg(page);
+	__folio_mark_dirty(folio, mapping, !folio_private(folio));
+	unlock_folio_memcg(folio);
 
-		if (mapping->host) {
-			/* !PageAnon && !swapper_space */
-			__mark_inode_dirty(mapping->host, I_DIRTY_PAGES);
-		}
-		return 1;
+	if (mapping->host) {
+		/* !PageAnon && !swapper_space */
+		__mark_inode_dirty(mapping->host, I_DIRTY_PAGES);
 	}
-	unlock_page_memcg(page);
-	return 0;
+	return true;
+}
+EXPORT_SYMBOL(filemap_dirty_folio);
+
+int __set_page_dirty_nobuffers(struct page *page)
+{
+	return filemap_dirty_folio(page_mapping(page), (struct folio *)page);
 }
 EXPORT_SYMBOL(__set_page_dirty_nobuffers);
 
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 128+ messages in thread

* [PATCH v2 29/46] mm/writeback: Add folio_account_cleaned()
  2021-06-22 12:15 [PATCH v2 00/46] Folio-enabling the page cache Matthew Wilcox (Oracle)
                   ` (27 preceding siblings ...)
  2021-06-22 12:15 ` [PATCH v2 28/46] mm/writeback: Add filemap_dirty_folio() Matthew Wilcox (Oracle)
@ 2021-06-22 12:15 ` Matthew Wilcox (Oracle)
  2021-06-23  9:36   ` Christoph Hellwig
  2021-06-22 12:15 ` [PATCH v2 30/46] mm/writeback: Add folio_cancel_dirty() Matthew Wilcox (Oracle)
                   ` (16 subsequent siblings)
  45 siblings, 1 reply; 128+ messages in thread
From: Matthew Wilcox (Oracle) @ 2021-06-22 12:15 UTC (permalink / raw)
  To: akpm; +Cc: Matthew Wilcox (Oracle), linux-fsdevel, linux-mm, linux-kernel

Get the statistics right; compound pages were being accounted as a
single page.  Also move the declaration to filemap.h since this is
part of the page cache.  Add a wrapper for account_page_cleaned().

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 include/linux/mm.h      |  3 ---
 include/linux/pagemap.h |  7 +++++++
 mm/page-writeback.c     | 11 ++++++-----
 3 files changed, 13 insertions(+), 8 deletions(-)

diff --git a/include/linux/mm.h b/include/linux/mm.h
index 3c8dfcb56fa5..2ccf294afc3e 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -39,7 +39,6 @@ struct anon_vma_chain;
 struct file_ra_state;
 struct user_struct;
 struct writeback_control;
-struct bdi_writeback;
 struct pt_regs;
 
 extern int sysctl_page_lock_unfairness;
@@ -1986,8 +1985,6 @@ extern void do_invalidatepage(struct page *page, unsigned int offset,
 
 int redirty_page_for_writepage(struct writeback_control *wbc,
 				struct page *page);
-void account_page_cleaned(struct page *page, struct address_space *mapping,
-			  struct bdi_writeback *wb);
 bool folio_mark_dirty(struct folio *folio);
 bool set_page_dirty(struct page *page);
 int set_page_dirty_lock(struct page *page);
diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
index e6a9756293aa..084fca551e60 100644
--- a/include/linux/pagemap.h
+++ b/include/linux/pagemap.h
@@ -778,6 +778,13 @@ static inline void __set_page_dirty(struct page *page,
 {
 	__folio_mark_dirty((struct folio *)page, mapping, warn);
 }
+void folio_account_cleaned(struct folio *folio, struct address_space *mapping,
+			  struct bdi_writeback *wb);
+static inline void account_page_cleaned(struct page *page,
+		struct address_space *mapping, struct bdi_writeback *wb)
+{
+	return folio_account_cleaned(page_folio(page), mapping, wb);
+}
 
 int __set_page_dirty_nobuffers(struct page *page);
 int __set_page_dirty_no_writeback(struct page *page);
diff --git a/mm/page-writeback.c b/mm/page-writeback.c
index 64b989eff9f5..cf48ac5b85f6 100644
--- a/mm/page-writeback.c
+++ b/mm/page-writeback.c
@@ -2449,14 +2449,15 @@ static void folio_account_dirtied(struct folio *folio,
  *
  * Caller must hold lock_page_memcg().
  */
-void account_page_cleaned(struct page *page, struct address_space *mapping,
+void folio_account_cleaned(struct folio *folio, struct address_space *mapping,
 			  struct bdi_writeback *wb)
 {
 	if (mapping_can_writeback(mapping)) {
-		dec_lruvec_page_state(page, NR_FILE_DIRTY);
-		dec_zone_page_state(page, NR_ZONE_WRITE_PENDING);
-		dec_wb_stat(wb, WB_RECLAIMABLE);
-		task_io_account_cancelled_write(PAGE_SIZE);
+		long nr = folio_nr_pages(folio);
+		lruvec_stat_mod_folio(folio, NR_FILE_DIRTY, -nr);
+		zone_stat_mod_folio(folio, NR_ZONE_WRITE_PENDING, -nr);
+		wb_stat_mod(wb, WB_RECLAIMABLE, -nr);
+		task_io_account_cancelled_write(folio_size(folio));
 	}
 }
 
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 128+ messages in thread

* [PATCH v2 30/46] mm/writeback: Add folio_cancel_dirty()
  2021-06-22 12:15 [PATCH v2 00/46] Folio-enabling the page cache Matthew Wilcox (Oracle)
                   ` (28 preceding siblings ...)
  2021-06-22 12:15 ` [PATCH v2 29/46] mm/writeback: Add folio_account_cleaned() Matthew Wilcox (Oracle)
@ 2021-06-22 12:15 ` Matthew Wilcox (Oracle)
  2021-06-23  9:39   ` Christoph Hellwig
  2021-06-22 12:15 ` [PATCH v2 31/46] mm/writeback: Add folio_clear_dirty_for_io() Matthew Wilcox (Oracle)
                   ` (15 subsequent siblings)
  45 siblings, 1 reply; 128+ messages in thread
From: Matthew Wilcox (Oracle) @ 2021-06-22 12:15 UTC (permalink / raw)
  To: akpm; +Cc: Matthew Wilcox (Oracle), linux-fsdevel, linux-mm, linux-kernel

Turn __cancel_dirty_page() into __folio_cancel_dirty() and add wrappers.
Move the prototypes into pagemap.h since this is page cache functionality.
Saves 44 bytes of kernel text in total; 33 bytes from __folio_cancel_dirty
and 11 from two callers of cancel_dirty_page().

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 include/linux/mm.h      |  7 -------
 include/linux/pagemap.h | 11 +++++++++++
 mm/page-writeback.c     | 16 ++++++++--------
 3 files changed, 19 insertions(+), 15 deletions(-)

diff --git a/include/linux/mm.h b/include/linux/mm.h
index 2ccf294afc3e..f0b0779c75cd 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -1988,13 +1988,6 @@ int redirty_page_for_writepage(struct writeback_control *wbc,
 bool folio_mark_dirty(struct folio *folio);
 bool set_page_dirty(struct page *page);
 int set_page_dirty_lock(struct page *page);
-void __cancel_dirty_page(struct page *page);
-static inline void cancel_dirty_page(struct page *page)
-{
-	/* Avoid atomic ops, locking, etc. when not actually needed. */
-	if (PageDirty(page))
-		__cancel_dirty_page(page);
-}
 int clear_page_dirty_for_io(struct page *page);
 
 int get_cmdline(struct task_struct *task, char *buffer, int buflen);
diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
index 084fca551e60..1dc8ab5651c3 100644
--- a/include/linux/pagemap.h
+++ b/include/linux/pagemap.h
@@ -785,6 +785,17 @@ static inline void account_page_cleaned(struct page *page,
 {
 	return folio_account_cleaned(page_folio(page), mapping, wb);
 }
+void __folio_cancel_dirty(struct folio *folio);
+static inline void folio_cancel_dirty(struct folio *folio)
+{
+	/* Avoid atomic ops, locking, etc. when not actually needed. */
+	if (folio_dirty(folio))
+		__folio_cancel_dirty(folio);
+}
+static inline void cancel_dirty_page(struct page *page)
+{
+	folio_cancel_dirty(page_folio(page));
+}
 
 int __set_page_dirty_nobuffers(struct page *page);
 int __set_page_dirty_no_writeback(struct page *page);
diff --git a/mm/page-writeback.c b/mm/page-writeback.c
index cf48ac5b85f6..a508c5629c15 100644
--- a/mm/page-writeback.c
+++ b/mm/page-writeback.c
@@ -2642,28 +2642,28 @@ EXPORT_SYMBOL(set_page_dirty_lock);
  * page without actually doing it through the VM. Can you say "ext3 is
  * horribly ugly"? Thought you could.
  */
-void __cancel_dirty_page(struct page *page)
+void __folio_cancel_dirty(struct folio *folio)
 {
-	struct address_space *mapping = page_mapping(page);
+	struct address_space *mapping = folio_mapping(folio);
 
 	if (mapping_can_writeback(mapping)) {
 		struct inode *inode = mapping->host;
 		struct bdi_writeback *wb;
 		struct wb_lock_cookie cookie = {};
 
-		lock_page_memcg(page);
+		lock_folio_memcg(folio);
 		wb = unlocked_inode_to_wb_begin(inode, &cookie);
 
-		if (TestClearPageDirty(page))
-			account_page_cleaned(page, mapping, wb);
+		if (folio_test_clear_dirty_flag(folio))
+			folio_account_cleaned(folio, mapping, wb);
 
 		unlocked_inode_to_wb_end(inode, &cookie);
-		unlock_page_memcg(page);
+		unlock_folio_memcg(folio);
 	} else {
-		ClearPageDirty(page);
+		folio_clear_dirty_flag(folio);
 	}
 }
-EXPORT_SYMBOL(__cancel_dirty_page);
+EXPORT_SYMBOL(__folio_cancel_dirty);
 
 /*
  * Clear a page's dirty flag, while caring for dirty memory accounting.
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 128+ messages in thread

* [PATCH v2 31/46] mm/writeback: Add folio_clear_dirty_for_io()
  2021-06-22 12:15 [PATCH v2 00/46] Folio-enabling the page cache Matthew Wilcox (Oracle)
                   ` (29 preceding siblings ...)
  2021-06-22 12:15 ` [PATCH v2 30/46] mm/writeback: Add folio_cancel_dirty() Matthew Wilcox (Oracle)
@ 2021-06-22 12:15 ` Matthew Wilcox (Oracle)
  2021-06-23  9:43   ` Christoph Hellwig
  2021-06-22 12:15 ` [PATCH v2 32/46] mm/writeback: Add folio_account_redirty() Matthew Wilcox (Oracle)
                   ` (14 subsequent siblings)
  45 siblings, 1 reply; 128+ messages in thread
From: Matthew Wilcox (Oracle) @ 2021-06-22 12:15 UTC (permalink / raw)
  To: akpm; +Cc: Matthew Wilcox (Oracle), linux-fsdevel, linux-mm, linux-kernel

Transform clear_page_dirty_for_io() into folio_clear_dirty_for_io()
and add a compatibility wrapper.  Also move the declaration to pagemap.h
as this is page cache functionality that doesn't need to be used by the
rest of the kernel.

Increases the size of the kernel by 79 bytes.  While we remove a few
calls to compound_head(), we add a call to folio_nr_pages() to get the
stats correct.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 include/linux/mm.h      |  1 -
 include/linux/pagemap.h |  2 ++
 mm/folio-compat.c       |  6 ++++
 mm/page-writeback.c     | 63 +++++++++++++++++++++--------------------
 4 files changed, 40 insertions(+), 32 deletions(-)

diff --git a/include/linux/mm.h b/include/linux/mm.h
index f0b0779c75cd..dce7593de256 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -1988,7 +1988,6 @@ int redirty_page_for_writepage(struct writeback_control *wbc,
 bool folio_mark_dirty(struct folio *folio);
 bool set_page_dirty(struct page *page);
 int set_page_dirty_lock(struct page *page);
-int clear_page_dirty_for_io(struct page *page);
 
 int get_cmdline(struct task_struct *task, char *buffer, int buflen);
 
diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
index 1dc8ab5651c3..31edfa891987 100644
--- a/include/linux/pagemap.h
+++ b/include/linux/pagemap.h
@@ -796,6 +796,8 @@ static inline void cancel_dirty_page(struct page *page)
 {
 	folio_cancel_dirty(page_folio(page));
 }
+bool folio_clear_dirty_for_io(struct folio *folio);
+bool clear_page_dirty_for_io(struct page *page);
 
 int __set_page_dirty_nobuffers(struct page *page);
 int __set_page_dirty_no_writeback(struct page *page);
diff --git a/mm/folio-compat.c b/mm/folio-compat.c
index 60780ceca363..bdd2cee01e4c 100644
--- a/mm/folio-compat.c
+++ b/mm/folio-compat.c
@@ -95,3 +95,9 @@ bool set_page_dirty(struct page *page)
 	return folio_mark_dirty(page_folio(page));
 }
 EXPORT_SYMBOL(set_page_dirty);
+
+bool clear_page_dirty_for_io(struct page *page)
+{
+	return folio_clear_dirty_for_io(page_folio(page));
+}
+EXPORT_SYMBOL(clear_page_dirty_for_io);
diff --git a/mm/page-writeback.c b/mm/page-writeback.c
index a508c5629c15..5fc1137d2c86 100644
--- a/mm/page-writeback.c
+++ b/mm/page-writeback.c
@@ -2666,25 +2666,25 @@ void __folio_cancel_dirty(struct folio *folio)
 EXPORT_SYMBOL(__folio_cancel_dirty);
 
 /*
- * Clear a page's dirty flag, while caring for dirty memory accounting.
- * Returns true if the page was previously dirty.
- *
- * This is for preparing to put the page under writeout.  We leave the page
- * tagged as dirty in the xarray so that a concurrent write-for-sync
- * can discover it via a PAGECACHE_TAG_DIRTY walk.  The ->writepage
- * implementation will run either set_page_writeback() or set_page_dirty(),
- * at which stage we bring the page's dirty flag and xarray dirty tag
- * back into sync.
- *
- * This incoherency between the page's dirty flag and xarray tag is
- * unfortunate, but it only exists while the page is locked.
+ * Clear a folio's dirty flag, while caring for dirty memory accounting.
+ * Returns true if the folio was previously dirty.
+ *
+ * This is for preparing to put the folio under writeout.  We leave
+ * the folio tagged as dirty in the xarray so that a concurrent
+ * write-for-sync can discover it via a PAGECACHE_TAG_DIRTY walk.
+ * The ->writepage implementation will run either folio_start_writeback()
+ * or folio_mark_dirty(), at which stage we bring the folio's dirty flag
+ * and xarray dirty tag back into sync.
+ *
+ * This incoherency between the folio's dirty flag and xarray tag is
+ * unfortunate, but it only exists while the folio is locked.
  */
-int clear_page_dirty_for_io(struct page *page)
+bool folio_clear_dirty_for_io(struct folio *folio)
 {
-	struct address_space *mapping = page_mapping(page);
-	int ret = 0;
+	struct address_space *mapping = folio_mapping(folio);
+	bool ret = false;
 
-	VM_BUG_ON_PAGE(!PageLocked(page), page);
+	VM_BUG_ON_FOLIO(!folio_locked(folio), folio);
 
 	if (mapping && mapping_can_writeback(mapping)) {
 		struct inode *inode = mapping->host;
@@ -2697,48 +2697,49 @@ int clear_page_dirty_for_io(struct page *page)
 		 * We use this sequence to make sure that
 		 *  (a) we account for dirty stats properly
 		 *  (b) we tell the low-level filesystem to
-		 *      mark the whole page dirty if it was
+		 *      mark the whole folio dirty if it was
 		 *      dirty in a pagetable. Only to then
-		 *  (c) clean the page again and return 1 to
+		 *  (c) clean the folio again and return 1 to
 		 *      cause the writeback.
 		 *
 		 * This way we avoid all nasty races with the
 		 * dirty bit in multiple places and clearing
 		 * them concurrently from different threads.
 		 *
-		 * Note! Normally the "set_page_dirty(page)"
+		 * Note! Normally the "folio_mark_dirty(folio)"
 		 * has no effect on the actual dirty bit - since
 		 * that will already usually be set. But we
 		 * need the side effects, and it can help us
 		 * avoid races.
 		 *
-		 * We basically use the page "master dirty bit"
+		 * We basically use the folio "master dirty bit"
 		 * as a serialization point for all the different
 		 * threads doing their things.
 		 */
-		if (page_mkclean(page))
-			set_page_dirty(page);
+		if (folio_mkclean(folio))
+			folio_mark_dirty(folio);
 		/*
 		 * We carefully synchronise fault handlers against
-		 * installing a dirty pte and marking the page dirty
+		 * installing a dirty pte and marking the folio dirty
 		 * at this point.  We do this by having them hold the
-		 * page lock while dirtying the page, and pages are
+		 * page lock while dirtying the folio, and folios are
 		 * always locked coming in here, so we get the desired
 		 * exclusion.
 		 */
 		wb = unlocked_inode_to_wb_begin(inode, &cookie);
-		if (TestClearPageDirty(page)) {
-			dec_lruvec_page_state(page, NR_FILE_DIRTY);
-			dec_zone_page_state(page, NR_ZONE_WRITE_PENDING);
-			dec_wb_stat(wb, WB_RECLAIMABLE);
-			ret = 1;
+		if (folio_test_clear_dirty_flag(folio)) {
+			long nr = folio_nr_pages(folio);
+			lruvec_stat_mod_folio(folio, NR_FILE_DIRTY, -nr);
+			zone_stat_mod_folio(folio, NR_ZONE_WRITE_PENDING, -nr);
+			wb_stat_mod(wb, WB_RECLAIMABLE, -nr);
+			ret = true;
 		}
 		unlocked_inode_to_wb_end(inode, &cookie);
 		return ret;
 	}
-	return TestClearPageDirty(page);
+	return folio_test_clear_dirty_flag(folio);
 }
-EXPORT_SYMBOL(clear_page_dirty_for_io);
+EXPORT_SYMBOL(folio_clear_dirty_for_io);
 
 bool __folio_end_writeback(struct folio *folio)
 {
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 128+ messages in thread

* [PATCH v2 32/46] mm/writeback: Add folio_account_redirty()
  2021-06-22 12:15 [PATCH v2 00/46] Folio-enabling the page cache Matthew Wilcox (Oracle)
                   ` (30 preceding siblings ...)
  2021-06-22 12:15 ` [PATCH v2 31/46] mm/writeback: Add folio_clear_dirty_for_io() Matthew Wilcox (Oracle)
@ 2021-06-22 12:15 ` Matthew Wilcox (Oracle)
  2021-06-23  9:44   ` Christoph Hellwig
  2021-06-22 12:15 ` [PATCH v2 33/46] mm/writeback: Add folio_redirty_for_writepage() Matthew Wilcox (Oracle)
                   ` (13 subsequent siblings)
  45 siblings, 1 reply; 128+ messages in thread
From: Matthew Wilcox (Oracle) @ 2021-06-22 12:15 UTC (permalink / raw)
  To: akpm; +Cc: Matthew Wilcox (Oracle), linux-fsdevel, linux-mm, linux-kernel

Account the number of pages in the folio that we're redirtying.
Turn account_page_dirty() into a wrapper around it.  Also turn
the comment on folio_account_redirty() into kernel-doc and
edit it slightly so it makes sense to its potential callers.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 include/linux/writeback.h |  6 +++++-
 mm/page-writeback.c       | 32 +++++++++++++++++++-------------
 2 files changed, 24 insertions(+), 14 deletions(-)

diff --git a/include/linux/writeback.h b/include/linux/writeback.h
index aa372f6d2b55..9883dbf8b763 100644
--- a/include/linux/writeback.h
+++ b/include/linux/writeback.h
@@ -399,7 +399,11 @@ void tag_pages_for_writeback(struct address_space *mapping,
 			     pgoff_t start, pgoff_t end);
 
 bool filemap_dirty_folio(struct address_space *mapping, struct folio *folio);
-void account_page_redirty(struct page *page);
+void folio_account_redirty(struct folio *folio);
+static inline void account_page_redirty(struct page *page)
+{
+	folio_account_redirty(page_folio(page));
+}
 
 void sb_mark_inode_writeback(struct inode *inode);
 void sb_clear_inode_writeback(struct inode *inode);
diff --git a/mm/page-writeback.c b/mm/page-writeback.c
index 5fc1137d2c86..d10c7ad4229b 100644
--- a/mm/page-writeback.c
+++ b/mm/page-writeback.c
@@ -1089,7 +1089,7 @@ static void wb_update_write_bandwidth(struct bdi_writeback *wb,
 	 * write_bandwidth = ---------------------------------------------------
 	 *                                          period
 	 *
-	 * @written may have decreased due to account_page_redirty().
+	 * @written may have decreased due to folio_account_redirty().
 	 * Avoid underflowing @bw calculation.
 	 */
 	bw = written - min(written, wb->written_stamp);
@@ -2529,30 +2529,36 @@ int __set_page_dirty_nobuffers(struct page *page)
 }
 EXPORT_SYMBOL(__set_page_dirty_nobuffers);
 
-/*
- * Call this whenever redirtying a page, to de-account the dirty counters
- * (NR_DIRTIED, WB_DIRTIED, tsk->nr_dirtied), so that they match the written
- * counters (NR_WRITTEN, WB_WRITTEN) in long term. The mismatches will lead to
- * systematic errors in balanced_dirty_ratelimit and the dirty pages position
- * control.
+/**
+ * folio_account_redirty - Manually account for redirtying a page.
+ * @folio: The folio which is being redirtied.
+ *
+ * Most filesystems should call folio_redirty_for_writepage() instead
+ * of this fuction.  If your filesystem is doing writeback outside the
+ * context of a writeback_control(), it can call this when redirtying
+ * a folio, to de-account the dirty counters (NR_DIRTIED, WB_DIRTIED,
+ * tsk->nr_dirtied), so that they match the written counters (NR_WRITTEN,
+ * WB_WRITTEN) in long term. The mismatches will lead to systematic errors
+ * in balanced_dirty_ratelimit and the dirty pages position control.
  */
-void account_page_redirty(struct page *page)
+void folio_account_redirty(struct folio *folio)
 {
-	struct address_space *mapping = page->mapping;
+	struct address_space *mapping = folio->mapping;
 
 	if (mapping && mapping_can_writeback(mapping)) {
 		struct inode *inode = mapping->host;
 		struct bdi_writeback *wb;
 		struct wb_lock_cookie cookie = {};
+		unsigned nr = folio_nr_pages(folio);
 
 		wb = unlocked_inode_to_wb_begin(inode, &cookie);
-		current->nr_dirtied--;
-		dec_node_page_state(page, NR_DIRTIED);
-		dec_wb_stat(wb, WB_DIRTIED);
+		current->nr_dirtied -= nr;
+		node_stat_mod_folio(folio, NR_DIRTIED, -nr);
+		wb_stat_mod(wb, WB_DIRTIED, -nr);
 		unlocked_inode_to_wb_end(inode, &cookie);
 	}
 }
-EXPORT_SYMBOL(account_page_redirty);
+EXPORT_SYMBOL(folio_account_redirty);
 
 /*
  * When a writepage implementation decides that it doesn't want to write this
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 128+ messages in thread

* [PATCH v2 33/46] mm/writeback: Add folio_redirty_for_writepage()
  2021-06-22 12:15 [PATCH v2 00/46] Folio-enabling the page cache Matthew Wilcox (Oracle)
                   ` (31 preceding siblings ...)
  2021-06-22 12:15 ` [PATCH v2 32/46] mm/writeback: Add folio_account_redirty() Matthew Wilcox (Oracle)
@ 2021-06-22 12:15 ` Matthew Wilcox (Oracle)
  2021-06-23  9:45   ` Christoph Hellwig
  2021-06-22 12:15 ` [PATCH v2 34/46] mm/filemap: Add i_blocks_per_folio() Matthew Wilcox (Oracle)
                   ` (12 subsequent siblings)
  45 siblings, 1 reply; 128+ messages in thread
From: Matthew Wilcox (Oracle) @ 2021-06-22 12:15 UTC (permalink / raw)
  To: akpm; +Cc: Matthew Wilcox (Oracle), linux-fsdevel, linux-mm, linux-kernel

Reimplement redirty_page_for_writepage() as a wrapper around
folio_redirty_for_writepage().  Account the number of pages in the
folio, add kernel-doc and move the prototype to writeback.h.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 fs/jfs/jfs_metapage.c     |  1 +
 include/linux/mm.h        |  4 ----
 include/linux/writeback.h |  2 ++
 mm/folio-compat.c         |  7 +++++++
 mm/page-writeback.c       | 30 ++++++++++++++++++++----------
 5 files changed, 30 insertions(+), 14 deletions(-)

diff --git a/fs/jfs/jfs_metapage.c b/fs/jfs/jfs_metapage.c
index 176580f54af9..104ae698443e 100644
--- a/fs/jfs/jfs_metapage.c
+++ b/fs/jfs/jfs_metapage.c
@@ -13,6 +13,7 @@
 #include <linux/buffer_head.h>
 #include <linux/mempool.h>
 #include <linux/seq_file.h>
+#include <linux/writeback.h>
 #include "jfs_incore.h"
 #include "jfs_superblock.h"
 #include "jfs_filsys.h"
diff --git a/include/linux/mm.h b/include/linux/mm.h
index dce7593de256..d25ff74cf9e1 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -36,9 +36,7 @@
 struct mempolicy;
 struct anon_vma;
 struct anon_vma_chain;
-struct file_ra_state;
 struct user_struct;
-struct writeback_control;
 struct pt_regs;
 
 extern int sysctl_page_lock_unfairness;
@@ -1983,8 +1981,6 @@ extern int try_to_release_page(struct page * page, gfp_t gfp_mask);
 extern void do_invalidatepage(struct page *page, unsigned int offset,
 			      unsigned int length);
 
-int redirty_page_for_writepage(struct writeback_control *wbc,
-				struct page *page);
 bool folio_mark_dirty(struct folio *folio);
 bool set_page_dirty(struct page *page);
 int set_page_dirty_lock(struct page *page);
diff --git a/include/linux/writeback.h b/include/linux/writeback.h
index 9883dbf8b763..bdf933f74a1d 100644
--- a/include/linux/writeback.h
+++ b/include/linux/writeback.h
@@ -404,6 +404,8 @@ static inline void account_page_redirty(struct page *page)
 {
 	folio_account_redirty(page_folio(page));
 }
+bool folio_redirty_for_writepage(struct writeback_control *, struct folio *);
+bool redirty_page_for_writepage(struct writeback_control *, struct page *);
 
 void sb_mark_inode_writeback(struct inode *inode);
 void sb_clear_inode_writeback(struct inode *inode);
diff --git a/mm/folio-compat.c b/mm/folio-compat.c
index bdd2cee01e4c..8f35ff91ee15 100644
--- a/mm/folio-compat.c
+++ b/mm/folio-compat.c
@@ -101,3 +101,10 @@ bool clear_page_dirty_for_io(struct page *page)
 	return folio_clear_dirty_for_io(page_folio(page));
 }
 EXPORT_SYMBOL(clear_page_dirty_for_io);
+
+bool redirty_page_for_writepage(struct writeback_control *wbc,
+		struct page *page)
+{
+	return folio_redirty_for_writepage(wbc, page_folio(page));
+}
+EXPORT_SYMBOL(redirty_page_for_writepage);
diff --git a/mm/page-writeback.c b/mm/page-writeback.c
index d10c7ad4229b..80bf36128fdd 100644
--- a/mm/page-writeback.c
+++ b/mm/page-writeback.c
@@ -2560,21 +2560,31 @@ void folio_account_redirty(struct folio *folio)
 }
 EXPORT_SYMBOL(folio_account_redirty);
 
-/*
- * When a writepage implementation decides that it doesn't want to write this
- * page for some reason, it should redirty the locked page via
- * redirty_page_for_writepage() and it should then unlock the page and return 0
+/**
+ * folio_redirty_for_writepage - Decline to write a dirty folio.
+ * @wbc: The writeback control.
+ * @folio: The folio.
+ *
+ * When a writepage implementation decides that it doesn't want to write
+ * @folio for some reason, it should call this function, unlock @folio and
+ * return 0.
+ *
+ * Return: True if we redirtied the folio.  False if someone else dirtied
+ * it first.
  */
-int redirty_page_for_writepage(struct writeback_control *wbc, struct page *page)
+bool folio_redirty_for_writepage(struct writeback_control *wbc,
+		struct folio *folio)
 {
-	int ret;
+	bool ret;
+	unsigned nr = folio_nr_pages(folio);
+
+	wbc->pages_skipped += nr;
+	ret = filemap_dirty_folio(folio->mapping, folio);
+	folio_account_redirty(folio);
 
-	wbc->pages_skipped++;
-	ret = __set_page_dirty_nobuffers(page);
-	account_page_redirty(page);
 	return ret;
 }
-EXPORT_SYMBOL(redirty_page_for_writepage);
+EXPORT_SYMBOL(folio_redirty_for_writepage);
 
 /**
  * folio_mark_dirty - Mark a folio as being modified.
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 128+ messages in thread

* [PATCH v2 34/46] mm/filemap: Add i_blocks_per_folio()
  2021-06-22 12:15 [PATCH v2 00/46] Folio-enabling the page cache Matthew Wilcox (Oracle)
                   ` (32 preceding siblings ...)
  2021-06-22 12:15 ` [PATCH v2 33/46] mm/writeback: Add folio_redirty_for_writepage() Matthew Wilcox (Oracle)
@ 2021-06-22 12:15 ` Matthew Wilcox (Oracle)
  2021-06-23  9:46   ` Christoph Hellwig
  2021-06-22 12:15 ` [PATCH v2 35/46] mm/filemap: Add folio_mkwrite_check_truncate() Matthew Wilcox (Oracle)
                   ` (11 subsequent siblings)
  45 siblings, 1 reply; 128+ messages in thread
From: Matthew Wilcox (Oracle) @ 2021-06-22 12:15 UTC (permalink / raw)
  To: akpm; +Cc: Matthew Wilcox (Oracle), linux-fsdevel, linux-mm, linux-kernel

Reimplement i_blocks_per_page() as a wrapper around i_blocks_per_folio().

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 include/linux/pagemap.h | 18 ++++++++++++------
 1 file changed, 12 insertions(+), 6 deletions(-)

diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
index 31edfa891987..c30db827b65d 100644
--- a/include/linux/pagemap.h
+++ b/include/linux/pagemap.h
@@ -1149,19 +1149,25 @@ static inline int page_mkwrite_check_truncate(struct page *page,
 }
 
 /**
- * i_blocks_per_page - How many blocks fit in this page.
+ * i_blocks_per_folio - How many blocks fit in this folio.
  * @inode: The inode which contains the blocks.
- * @page: The page (head page if the page is a THP).
+ * @folio: The folio.
  *
- * If the block size is larger than the size of this page, return zero.
+ * If the block size is larger than the size of this folio, return zero.
  *
- * Context: The caller should hold a refcount on the page to prevent it
+ * Context: The caller should hold a refcount on the folio to prevent it
  * from being split.
- * Return: The number of filesystem blocks covered by this page.
+ * Return: The number of filesystem blocks covered by this folio.
  */
+static inline
+unsigned int i_blocks_per_folio(struct inode *inode, struct folio *folio)
+{
+	return folio_size(folio) >> inode->i_blkbits;
+}
+
 static inline
 unsigned int i_blocks_per_page(struct inode *inode, struct page *page)
 {
-	return thp_size(page) >> inode->i_blkbits;
+	return i_blocks_per_folio(inode, page_folio(page));
 }
 #endif /* _LINUX_PAGEMAP_H */
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 128+ messages in thread

* [PATCH v2 35/46] mm/filemap: Add folio_mkwrite_check_truncate()
  2021-06-22 12:15 [PATCH v2 00/46] Folio-enabling the page cache Matthew Wilcox (Oracle)
                   ` (33 preceding siblings ...)
  2021-06-22 12:15 ` [PATCH v2 34/46] mm/filemap: Add i_blocks_per_folio() Matthew Wilcox (Oracle)
@ 2021-06-22 12:15 ` Matthew Wilcox (Oracle)
  2021-06-23  9:47   ` Christoph Hellwig
  2021-06-22 12:15 ` [PATCH v2 36/46] mm/filemap: Add readahead_folio() Matthew Wilcox (Oracle)
                   ` (10 subsequent siblings)
  45 siblings, 1 reply; 128+ messages in thread
From: Matthew Wilcox (Oracle) @ 2021-06-22 12:15 UTC (permalink / raw)
  To: akpm; +Cc: Matthew Wilcox (Oracle), linux-fsdevel, linux-mm, linux-kernel

This is the folio equivalent of page_mkwrite_check_truncate().

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 include/linux/pagemap.h | 28 ++++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)

diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
index c30db827b65d..14f0c5260234 100644
--- a/include/linux/pagemap.h
+++ b/include/linux/pagemap.h
@@ -1120,6 +1120,34 @@ static inline unsigned long dir_pages(struct inode *inode)
 			       PAGE_SHIFT;
 }
 
+/**
+ * folio_mkwrite_check_truncate - check if folio was truncated
+ * @folio: the folio to check
+ * @inode: the inode to check the folio against
+ *
+ * Return: the number of bytes in the folio up to EOF,
+ * or -EFAULT if the folio was truncated.
+ */
+static inline ssize_t folio_mkwrite_check_truncate(struct folio *folio,
+					      struct inode *inode)
+{
+	loff_t size = i_size_read(inode);
+	pgoff_t index = size >> PAGE_SHIFT;
+	size_t offset = offset_in_folio(folio, size);
+
+	if (!folio->mapping)
+		return -EFAULT;
+
+	/* folio is wholly inside EOF */
+	if (folio_next_index(folio) - 1 < index)
+		return folio_size(folio);
+	/* folio is wholly past EOF */
+	if (folio->index > index || !offset)
+		return -EFAULT;
+	/* folio is partially inside EOF */
+	return offset;
+}
+
 /**
  * page_mkwrite_check_truncate - check if page was truncated
  * @page: the page to check
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 128+ messages in thread

* [PATCH v2 36/46] mm/filemap: Add readahead_folio()
  2021-06-22 12:15 [PATCH v2 00/46] Folio-enabling the page cache Matthew Wilcox (Oracle)
                   ` (34 preceding siblings ...)
  2021-06-22 12:15 ` [PATCH v2 35/46] mm/filemap: Add folio_mkwrite_check_truncate() Matthew Wilcox (Oracle)
@ 2021-06-22 12:15 ` Matthew Wilcox (Oracle)
  2021-06-23  9:50   ` Christoph Hellwig
  2021-06-22 12:15 ` [PATCH v2 37/46] mm/workingset: Convert workingset_refault() to take a folio Matthew Wilcox (Oracle)
                   ` (9 subsequent siblings)
  45 siblings, 1 reply; 128+ messages in thread
From: Matthew Wilcox (Oracle) @ 2021-06-22 12:15 UTC (permalink / raw)
  To: akpm; +Cc: Matthew Wilcox (Oracle), linux-fsdevel, linux-mm, linux-kernel

The pointers stored in the page cache are folios, by definition.
This change comes with a behaviour change -- callers of readahead_folio()
are no longer required to put the page reference themselves.  This matches
how readpage works, rather than matching how readpages used to work.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 include/linux/pagemap.h | 53 +++++++++++++++++++++++++++++------------
 1 file changed, 38 insertions(+), 15 deletions(-)

diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
index 14f0c5260234..c1df4c569148 100644
--- a/include/linux/pagemap.h
+++ b/include/linux/pagemap.h
@@ -987,33 +987,56 @@ void page_cache_async_readahead(struct address_space *mapping,
 	page_cache_async_ra(&ractl, page, req_count);
 }
 
+static inline struct folio *__readahead_folio(struct readahead_control *ractl)
+{
+	struct folio *folio;
+
+	BUG_ON(ractl->_batch_count > ractl->_nr_pages);
+	ractl->_nr_pages -= ractl->_batch_count;
+	ractl->_index += ractl->_batch_count;
+
+	if (!ractl->_nr_pages) {
+		ractl->_batch_count = 0;
+		return NULL;
+	}
+
+	folio = xa_load(&ractl->mapping->i_pages, ractl->_index);
+	VM_BUG_ON_FOLIO(!folio_locked(folio), folio);
+	ractl->_batch_count = folio_nr_pages(folio);
+
+	return folio;
+}
+
 /**
  * readahead_page - Get the next page to read.
- * @rac: The current readahead request.
+ * @ractl: The current readahead request.
  *
  * Context: The page is locked and has an elevated refcount.  The caller
  * should decreases the refcount once the page has been submitted for I/O
  * and unlock the page once all I/O to that page has completed.
  * Return: A pointer to the next page, or %NULL if we are done.
  */
-static inline struct page *readahead_page(struct readahead_control *rac)
+static inline struct page *readahead_page(struct readahead_control *ractl)
 {
-	struct page *page;
+	struct folio *folio = __readahead_folio(ractl);
 
-	BUG_ON(rac->_batch_count > rac->_nr_pages);
-	rac->_nr_pages -= rac->_batch_count;
-	rac->_index += rac->_batch_count;
-
-	if (!rac->_nr_pages) {
-		rac->_batch_count = 0;
-		return NULL;
-	}
+	return &folio->page;
+}
 
-	page = xa_load(&rac->mapping->i_pages, rac->_index);
-	VM_BUG_ON_PAGE(!PageLocked(page), page);
-	rac->_batch_count = thp_nr_pages(page);
+/**
+ * readahead_folio - Get the next folio to read.
+ * @ractl: The current readahead request.
+ *
+ * Context: The folio is locked.  The caller should unlock the folio once
+ * all I/O to that folio has completed.
+ * Return: A pointer to the next folio, or %NULL if we are done.
+ */
+static inline struct folio *readahead_folio(struct readahead_control *ractl)
+{
+	struct folio *folio = __readahead_folio(ractl);
 
-	return page;
+	folio_put(folio);
+	return folio;
 }
 
 static inline unsigned int __readahead_batch(struct readahead_control *rac,
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 128+ messages in thread

* [PATCH v2 37/46] mm/workingset: Convert workingset_refault() to take a folio
  2021-06-22 12:15 [PATCH v2 00/46] Folio-enabling the page cache Matthew Wilcox (Oracle)
                   ` (35 preceding siblings ...)
  2021-06-22 12:15 ` [PATCH v2 36/46] mm/filemap: Add readahead_folio() Matthew Wilcox (Oracle)
@ 2021-06-22 12:15 ` Matthew Wilcox (Oracle)
  2021-06-23  9:52   ` Christoph Hellwig
  2021-06-22 12:15 ` [PATCH v2 38/46] mm: Add folio_evictable() Matthew Wilcox (Oracle)
                   ` (8 subsequent siblings)
  45 siblings, 1 reply; 128+ messages in thread
From: Matthew Wilcox (Oracle) @ 2021-06-22 12:15 UTC (permalink / raw)
  To: akpm; +Cc: Matthew Wilcox (Oracle), linux-fsdevel, linux-mm, linux-kernel

This nets us 178 bytes of savings from removing calls to compound_head.
The three callers all grow a little, but each of them will be converted
to use folios soon, so that's fine.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 include/linux/swap.h |  4 ++--
 mm/filemap.c         |  2 +-
 mm/memory.c          |  3 ++-
 mm/swap.c            |  6 +++---
 mm/swap_state.c      |  2 +-
 mm/workingset.c      | 34 +++++++++++++++++-----------------
 6 files changed, 26 insertions(+), 25 deletions(-)

diff --git a/include/linux/swap.h b/include/linux/swap.h
index d1cb67cdb476..35d3dba422a8 100644
--- a/include/linux/swap.h
+++ b/include/linux/swap.h
@@ -323,7 +323,7 @@ static inline swp_entry_t folio_swap_entry(struct folio *folio)
 /* linux/mm/workingset.c */
 void workingset_age_nonresident(struct lruvec *lruvec, unsigned long nr_pages);
 void *workingset_eviction(struct page *page, struct mem_cgroup *target_memcg);
-void workingset_refault(struct page *page, void *shadow);
+void workingset_refault(struct folio *folio, void *shadow);
 void workingset_activation(struct folio *folio);
 
 /* Only track the nodes of mappings with shadow entries */
@@ -344,7 +344,7 @@ extern unsigned long nr_free_buffer_pages(void);
 /* linux/mm/swap.c */
 extern void lru_note_cost(struct lruvec *lruvec, bool file,
 			  unsigned int nr_pages);
-extern void lru_note_cost_page(struct page *);
+extern void lru_note_cost_folio(struct folio *);
 extern void lru_cache_add(struct page *);
 void mark_page_accessed(struct page *);
 void folio_mark_accessed(struct folio *);
diff --git a/mm/filemap.c b/mm/filemap.c
index dd360721c72b..673094ce67da 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -981,7 +981,7 @@ int add_to_page_cache_lru(struct page *page, struct address_space *mapping,
 		 */
 		WARN_ON_ONCE(PageActive(page));
 		if (!(gfp_mask & __GFP_WRITE) && shadow)
-			workingset_refault(page, shadow);
+			workingset_refault(page_folio(page), shadow);
 		lru_cache_add(page);
 	}
 	return ret;
diff --git a/mm/memory.c b/mm/memory.c
index a8f0ddd9e84a..d37f0f898f65 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -3364,7 +3364,8 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
 
 				shadow = get_shadow_from_swap_cache(entry);
 				if (shadow)
-					workingset_refault(page, shadow);
+					workingset_refault(page_folio(page),
+								shadow);
 
 				lru_cache_add(page);
 
diff --git a/mm/swap.c b/mm/swap.c
index 53422f6b7db1..2ed00cfd03ac 100644
--- a/mm/swap.c
+++ b/mm/swap.c
@@ -313,10 +313,10 @@ void lru_note_cost(struct lruvec *lruvec, bool file, unsigned int nr_pages)
 	} while ((lruvec = parent_lruvec(lruvec)));
 }
 
-void lru_note_cost_page(struct page *page)
+void lru_note_cost_folio(struct folio *folio)
 {
-	lru_note_cost(mem_cgroup_page_lruvec(page, page_pgdat(page)),
-		      page_is_file_lru(page), thp_nr_pages(page));
+	lru_note_cost(mem_cgroup_folio_lruvec(folio),
+		      folio_is_file_lru(folio), folio_nr_pages(folio));
 }
 
 static void __folio_activate(struct folio *folio, struct lruvec *lruvec)
diff --git a/mm/swap_state.c b/mm/swap_state.c
index 272ea2108c9d..1c8e8b3aa10b 100644
--- a/mm/swap_state.c
+++ b/mm/swap_state.c
@@ -503,7 +503,7 @@ struct page *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask,
 	mem_cgroup_swapin_uncharge_swap(entry);
 
 	if (shadow)
-		workingset_refault(page, shadow);
+		workingset_refault(page_folio(page), shadow);
 
 	/* Caller will initiate read into locked page */
 	lru_cache_add(page);
diff --git a/mm/workingset.c b/mm/workingset.c
index 37cdcda96afd..2c8f211daafe 100644
--- a/mm/workingset.c
+++ b/mm/workingset.c
@@ -271,17 +271,17 @@ void *workingset_eviction(struct page *page, struct mem_cgroup *target_memcg)
 }
 
 /**
- * workingset_refault - evaluate the refault of a previously evicted page
- * @page: the freshly allocated replacement page
- * @shadow: shadow entry of the evicted page
+ * workingset_refault - evaluate the refault of a previously evicted folio
+ * @page: the freshly allocated replacement folio
+ * @shadow: shadow entry of the evicted folio
  *
  * Calculates and evaluates the refault distance of the previously
- * evicted page in the context of the node and the memcg whose memory
+ * evicted folio in the context of the node and the memcg whose memory
  * pressure caused the eviction.
  */
-void workingset_refault(struct page *page, void *shadow)
+void workingset_refault(struct folio *folio, void *shadow)
 {
-	bool file = page_is_file_lru(page);
+	bool file = folio_is_file_lru(folio);
 	struct mem_cgroup *eviction_memcg;
 	struct lruvec *eviction_lruvec;
 	unsigned long refault_distance;
@@ -299,10 +299,10 @@ void workingset_refault(struct page *page, void *shadow)
 	rcu_read_lock();
 	/*
 	 * Look up the memcg associated with the stored ID. It might
-	 * have been deleted since the page's eviction.
+	 * have been deleted since the folio's eviction.
 	 *
 	 * Note that in rare events the ID could have been recycled
-	 * for a new cgroup that refaults a shared page. This is
+	 * for a new cgroup that refaults a shared folio. This is
 	 * impossible to tell from the available data. However, this
 	 * should be a rare and limited disturbance, and activations
 	 * are always speculative anyway. Ultimately, it's the aging
@@ -338,14 +338,14 @@ void workingset_refault(struct page *page, void *shadow)
 	refault_distance = (refault - eviction) & EVICTION_MASK;
 
 	/*
-	 * The activation decision for this page is made at the level
+	 * The activation decision for this folio is made at the level
 	 * where the eviction occurred, as that is where the LRU order
-	 * during page reclaim is being determined.
+	 * during folio reclaim is being determined.
 	 *
-	 * However, the cgroup that will own the page is the one that
+	 * However, the cgroup that will own the folio is the one that
 	 * is actually experiencing the refault event.
 	 */
-	memcg = page_memcg(page);
+	memcg = folio_memcg(folio);
 	lruvec = mem_cgroup_lruvec(memcg, pgdat);
 
 	inc_lruvec_state(lruvec, WORKINGSET_REFAULT_BASE + file);
@@ -373,15 +373,15 @@ void workingset_refault(struct page *page, void *shadow)
 	if (refault_distance > workingset_size)
 		goto out;
 
-	SetPageActive(page);
-	workingset_age_nonresident(lruvec, thp_nr_pages(page));
+	folio_set_active_flag(folio);
+	workingset_age_nonresident(lruvec, folio_nr_pages(folio));
 	inc_lruvec_state(lruvec, WORKINGSET_ACTIVATE_BASE + file);
 
-	/* Page was active prior to eviction */
+	/* Folio was active prior to eviction */
 	if (workingset) {
-		SetPageWorkingset(page);
+		folio_set_workingset_flag(folio);
 		/* XXX: Move to lru_cache_add() when it supports new vs putback */
-		lru_note_cost_page(page);
+		lru_note_cost_folio(folio);
 		inc_lruvec_state(lruvec, WORKINGSET_RESTORE_BASE + file);
 	}
 out:
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 128+ messages in thread

* [PATCH v2 38/46] mm: Add folio_evictable()
  2021-06-22 12:15 [PATCH v2 00/46] Folio-enabling the page cache Matthew Wilcox (Oracle)
                   ` (36 preceding siblings ...)
  2021-06-22 12:15 ` [PATCH v2 37/46] mm/workingset: Convert workingset_refault() to take a folio Matthew Wilcox (Oracle)
@ 2021-06-22 12:15 ` Matthew Wilcox (Oracle)
  2021-06-23  9:54   ` Christoph Hellwig
  2021-06-22 12:15 ` [PATCH v2 39/46] mm/lru: Convert __pagevec_lru_add_fn to take a folio Matthew Wilcox (Oracle)
                   ` (7 subsequent siblings)
  45 siblings, 1 reply; 128+ messages in thread
From: Matthew Wilcox (Oracle) @ 2021-06-22 12:15 UTC (permalink / raw)
  To: akpm; +Cc: Matthew Wilcox (Oracle), linux-fsdevel, linux-mm, linux-kernel

This is the folio equivalent of page_evictable().  Unfortunately, it's
different from !folio_unevictable(), but I think it's used in places
where you have to be a VM expert and can reasonably be expected to know
the difference.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 mm/internal.h | 27 +++++++++++++++++++--------
 1 file changed, 19 insertions(+), 8 deletions(-)

diff --git a/mm/internal.h b/mm/internal.h
index d7013df5d1f0..3b0ab6b5457c 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -72,17 +72,28 @@ unsigned find_lock_entries(struct address_space *mapping, pgoff_t start,
 		pgoff_t end, struct pagevec *pvec, pgoff_t *indices);
 
 /**
- * page_evictable - test whether a page is evictable
- * @page: the page to test
+ * folio_evictable - Test whether a folio is evictable.
+ * @folio: The folio to test.
  *
- * Test whether page is evictable--i.e., should be placed on active/inactive
- * lists vs unevictable list.
- *
- * Reasons page might not be evictable:
- * (1) page's mapping marked unevictable
- * (2) page is part of an mlocked VMA
+ * Test whether @folio is evictable -- i.e., should be placed on
+ * active/inactive lists vs unevictable list.
  *
+ * Reasons folio might not be evictable:
+ * 1. page's mapping marked unevictable
+ * 2. page is part of an mlocked VMA
  */
+static inline bool folio_evictable(struct folio *folio)
+{
+	bool ret;
+
+	/* Prevent address_space of inode and swap cache from being freed */
+	rcu_read_lock();
+	ret = !mapping_unevictable(folio_mapping(folio)) &&
+			!folio_mlocked(folio);
+	rcu_read_unlock();
+	return ret;
+}
+
 static inline bool page_evictable(struct page *page)
 {
 	bool ret;
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 128+ messages in thread

* [PATCH v2 39/46] mm/lru: Convert __pagevec_lru_add_fn to take a folio
  2021-06-22 12:15 [PATCH v2 00/46] Folio-enabling the page cache Matthew Wilcox (Oracle)
                   ` (37 preceding siblings ...)
  2021-06-22 12:15 ` [PATCH v2 38/46] mm: Add folio_evictable() Matthew Wilcox (Oracle)
@ 2021-06-22 12:15 ` Matthew Wilcox (Oracle)
  2021-06-23  9:55   ` Christoph Hellwig
  2021-06-22 12:15 ` [PATCH v2 40/46] mm/lru: Add folio_add_lru() Matthew Wilcox (Oracle)
                   ` (6 subsequent siblings)
  45 siblings, 1 reply; 128+ messages in thread
From: Matthew Wilcox (Oracle) @ 2021-06-22 12:15 UTC (permalink / raw)
  To: akpm; +Cc: Matthew Wilcox (Oracle), linux-fsdevel, linux-mm, linux-kernel

This saves five calls to compound_head(), totalling 60 bytes of text.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 mm/swap.c | 35 ++++++++++++++++++-----------------
 1 file changed, 18 insertions(+), 17 deletions(-)

diff --git a/mm/swap.c b/mm/swap.c
index 2ed00cfd03ac..f3f1ee9f8616 100644
--- a/mm/swap.c
+++ b/mm/swap.c
@@ -998,17 +998,18 @@ void __pagevec_release(struct pagevec *pvec)
 }
 EXPORT_SYMBOL(__pagevec_release);
 
-static void __pagevec_lru_add_fn(struct page *page, struct lruvec *lruvec)
+static void __pagevec_lru_add_fn(struct folio *folio, struct lruvec *lruvec)
 {
-	int was_unevictable = TestClearPageUnevictable(page);
-	int nr_pages = thp_nr_pages(page);
+	int was_unevictable = folio_test_clear_unevictable_flag(folio);
+	int nr_pages = folio_nr_pages(folio);
 
-	VM_BUG_ON_PAGE(PageLRU(page), page);
+	VM_BUG_ON_FOLIO(folio_lru(folio), folio);
 
 	/*
-	 * Page becomes evictable in two ways:
+	 * Folio becomes evictable in two ways:
 	 * 1) Within LRU lock [munlock_vma_page() and __munlock_pagevec()].
-	 * 2) Before acquiring LRU lock to put the page to correct LRU and then
+	 * 2) Before acquiring LRU lock to put the folio on the correct LRU
+	 *    and then
 	 *   a) do PageLRU check with lock [check_move_unevictable_pages]
 	 *   b) do PageLRU check before lock [clear_page_mlock]
 	 *
@@ -1017,10 +1018,10 @@ static void __pagevec_lru_add_fn(struct page *page, struct lruvec *lruvec)
 	 *
 	 * #0: __pagevec_lru_add_fn		#1: clear_page_mlock
 	 *
-	 * SetPageLRU()				TestClearPageMlocked()
+	 * folio_set_lru_flag()			folio_test_clear_mlocked_flag()
 	 * smp_mb() // explicit ordering	// above provides strict
 	 *					// ordering
-	 * PageMlocked()			PageLRU()
+	 * folio_mlocked()			folio_lru()
 	 *
 	 *
 	 * if '#1' does not observe setting of PG_lru by '#0' and fails
@@ -1031,21 +1032,21 @@ static void __pagevec_lru_add_fn(struct page *page, struct lruvec *lruvec)
 	 * looking at the same page) and the evictable page will be stranded
 	 * in an unevictable LRU.
 	 */
-	SetPageLRU(page);
+	folio_set_lru_flag(folio);
 	smp_mb__after_atomic();
 
-	if (page_evictable(page)) {
+	if (folio_evictable(folio)) {
 		if (was_unevictable)
 			__count_vm_events(UNEVICTABLE_PGRESCUED, nr_pages);
 	} else {
-		ClearPageActive(page);
-		SetPageUnevictable(page);
+		folio_clear_active_flag(folio);
+		folio_set_unevictable_flag(folio);
 		if (!was_unevictable)
 			__count_vm_events(UNEVICTABLE_PGCULLED, nr_pages);
 	}
 
-	add_page_to_lru_list(page, lruvec);
-	trace_mm_lru_insertion(page);
+	folio_add_to_lru_list(folio, lruvec);
+	trace_mm_lru_insertion(&folio->page);
 }
 
 /*
@@ -1059,10 +1060,10 @@ void __pagevec_lru_add(struct pagevec *pvec)
 	unsigned long flags = 0;
 
 	for (i = 0; i < pagevec_count(pvec); i++) {
-		struct page *page = pvec->pages[i];
+		struct folio *folio = page_folio(pvec->pages[i]);
 
-		lruvec = relock_page_lruvec_irqsave(page, lruvec, &flags);
-		__pagevec_lru_add_fn(page, lruvec);
+		lruvec = folio_relock_lruvec_irqsave(folio, lruvec, &flags);
+		__pagevec_lru_add_fn(folio, lruvec);
 	}
 	if (lruvec)
 		unlock_page_lruvec_irqrestore(lruvec, flags);
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 128+ messages in thread

* [PATCH v2 40/46] mm/lru: Add folio_add_lru()
  2021-06-22 12:15 [PATCH v2 00/46] Folio-enabling the page cache Matthew Wilcox (Oracle)
                   ` (38 preceding siblings ...)
  2021-06-22 12:15 ` [PATCH v2 39/46] mm/lru: Convert __pagevec_lru_add_fn to take a folio Matthew Wilcox (Oracle)
@ 2021-06-22 12:15 ` Matthew Wilcox (Oracle)
  2021-06-23  9:56   ` Christoph Hellwig
  2021-06-22 12:15 ` [PATCH v2 41/46] mm/page_alloc: Add folio allocation functions Matthew Wilcox (Oracle)
                   ` (5 subsequent siblings)
  45 siblings, 1 reply; 128+ messages in thread
From: Matthew Wilcox (Oracle) @ 2021-06-22 12:15 UTC (permalink / raw)
  To: akpm; +Cc: Matthew Wilcox (Oracle), linux-fsdevel, linux-mm, linux-kernel

Reimplement lru_cache_add() as a wrapper around folio_add_lru().
Saves 159 bytes of kernel text due to removing calls to compound_head().

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 include/linux/swap.h |  1 +
 mm/folio-compat.c    |  6 ++++++
 mm/swap.c            | 22 +++++++++++-----------
 3 files changed, 18 insertions(+), 11 deletions(-)

diff --git a/include/linux/swap.h b/include/linux/swap.h
index 35d3dba422a8..87bfabd69132 100644
--- a/include/linux/swap.h
+++ b/include/linux/swap.h
@@ -345,6 +345,7 @@ extern unsigned long nr_free_buffer_pages(void);
 extern void lru_note_cost(struct lruvec *lruvec, bool file,
 			  unsigned int nr_pages);
 extern void lru_note_cost_folio(struct folio *);
+extern void folio_add_lru(struct folio *);
 extern void lru_cache_add(struct page *);
 void mark_page_accessed(struct page *);
 void folio_mark_accessed(struct folio *);
diff --git a/mm/folio-compat.c b/mm/folio-compat.c
index 8f35ff91ee15..0a8bdd3f8d91 100644
--- a/mm/folio-compat.c
+++ b/mm/folio-compat.c
@@ -108,3 +108,9 @@ bool redirty_page_for_writepage(struct writeback_control *wbc,
 	return folio_redirty_for_writepage(wbc, page_folio(page));
 }
 EXPORT_SYMBOL(redirty_page_for_writepage);
+
+void lru_cache_add(struct page *page)
+{
+	folio_add_lru(page_folio(page));
+}
+EXPORT_SYMBOL(lru_cache_add);
diff --git a/mm/swap.c b/mm/swap.c
index f3f1ee9f8616..4dfc4bb6ba09 100644
--- a/mm/swap.c
+++ b/mm/swap.c
@@ -457,29 +457,29 @@ void folio_mark_accessed(struct folio *folio)
 EXPORT_SYMBOL(folio_mark_accessed);
 
 /**
- * lru_cache_add - add a page to a page list
- * @page: the page to be added to the LRU.
+ * folio_add_lru - Add a folio to an LRU list.
+ * @folio: The folio to be added to the LRU.
  *
- * Queue the page for addition to the LRU via pagevec. The decision on whether
+ * Queue the folio for addition to the LRU. The decision on whether
  * to add the page to the [in]active [file|anon] list is deferred until the
- * pagevec is drained. This gives a chance for the caller of lru_cache_add()
- * have the page added to the active list using mark_page_accessed().
+ * pagevec is drained. This gives a chance for the caller of folio_add_lru()
+ * have the folio added to the active list using folio_mark_accessed().
  */
-void lru_cache_add(struct page *page)
+void folio_add_lru(struct folio *folio)
 {
 	struct pagevec *pvec;
 
-	VM_BUG_ON_PAGE(PageActive(page) && PageUnevictable(page), page);
-	VM_BUG_ON_PAGE(PageLRU(page), page);
+	VM_BUG_ON_FOLIO(folio_active(folio) && folio_unevictable(folio), folio);
+	VM_BUG_ON_FOLIO(folio_lru(folio), folio);
 
-	get_page(page);
+	folio_get(folio);
 	local_lock(&lru_pvecs.lock);
 	pvec = this_cpu_ptr(&lru_pvecs.lru_add);
-	if (pagevec_add_and_need_flush(pvec, page))
+	if (pagevec_add_and_need_flush(pvec, &folio->page))
 		__pagevec_lru_add(pvec);
 	local_unlock(&lru_pvecs.lock);
 }
-EXPORT_SYMBOL(lru_cache_add);
+EXPORT_SYMBOL(folio_add_lru);
 
 /**
  * lru_cache_add_inactive_or_unevictable
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 128+ messages in thread

* [PATCH v2 41/46] mm/page_alloc: Add folio allocation functions
  2021-06-22 12:15 [PATCH v2 00/46] Folio-enabling the page cache Matthew Wilcox (Oracle)
                   ` (39 preceding siblings ...)
  2021-06-22 12:15 ` [PATCH v2 40/46] mm/lru: Add folio_add_lru() Matthew Wilcox (Oracle)
@ 2021-06-22 12:15 ` Matthew Wilcox (Oracle)
  2021-06-23  9:58   ` Christoph Hellwig
  2021-06-22 12:15 ` [PATCH v2 42/46] mm/filemap: Add filemap_alloc_folio Matthew Wilcox (Oracle)
                   ` (4 subsequent siblings)
  45 siblings, 1 reply; 128+ messages in thread
From: Matthew Wilcox (Oracle) @ 2021-06-22 12:15 UTC (permalink / raw)
  To: akpm; +Cc: Matthew Wilcox (Oracle), linux-fsdevel, linux-mm, linux-kernel

The __alloc_folio(), __alloc_folio_node() and alloc_folio() functions
are mostly for type safety, but they also ensure that the page allocator
allocates a compound page and initialises the deferred list if the page
is large enough to have one.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 include/linux/gfp.h | 16 ++++++++++++++++
 mm/mempolicy.c      | 10 ++++++++++
 mm/page_alloc.c     | 12 ++++++++++++
 3 files changed, 38 insertions(+)

diff --git a/include/linux/gfp.h b/include/linux/gfp.h
index a503d928e684..76086c798cb1 100644
--- a/include/linux/gfp.h
+++ b/include/linux/gfp.h
@@ -511,6 +511,8 @@ static inline void arch_alloc_page(struct page *page, int order) { }
 
 struct page *__alloc_pages(gfp_t gfp, unsigned int order, int preferred_nid,
 		nodemask_t *nodemask);
+struct folio *__alloc_folio(gfp_t gfp, unsigned int order, int preferred_nid,
+		nodemask_t *nodemask);
 
 unsigned long __alloc_pages_bulk(gfp_t gfp, int preferred_nid,
 				nodemask_t *nodemask, int nr_pages,
@@ -543,6 +545,15 @@ __alloc_pages_node(int nid, gfp_t gfp_mask, unsigned int order)
 	return __alloc_pages(gfp_mask, order, nid, NULL);
 }
 
+static inline
+struct folio *__alloc_folio_node(gfp_t gfp, unsigned int order, int nid)
+{
+	VM_BUG_ON(nid < 0 || nid >= MAX_NUMNODES);
+	VM_WARN_ON((gfp & __GFP_THISNODE) && !node_online(nid));
+
+	return __alloc_folio(gfp, order, nid, NULL);
+}
+
 /*
  * Allocate pages, preferring the node given as nid. When nid == NUMA_NO_NODE,
  * prefer the current CPU's closest node. Otherwise node must be valid and
@@ -559,6 +570,7 @@ static inline struct page *alloc_pages_node(int nid, gfp_t gfp_mask,
 
 #ifdef CONFIG_NUMA
 struct page *alloc_pages(gfp_t gfp, unsigned int order);
+struct folio *alloc_folio(gfp_t gfp, unsigned order);
 extern struct page *alloc_pages_vma(gfp_t gfp_mask, int order,
 			struct vm_area_struct *vma, unsigned long addr,
 			int node, bool hugepage);
@@ -569,6 +581,10 @@ static inline struct page *alloc_pages(gfp_t gfp_mask, unsigned int order)
 {
 	return alloc_pages_node(numa_node_id(), gfp_mask, order);
 }
+static inline struct folio *alloc_folio(gfp_t gfp, unsigned int order)
+{
+	return __alloc_folio_node(gfp, order, numa_node_id());
+}
 #define alloc_pages_vma(gfp_mask, order, vma, addr, node, false)\
 	alloc_pages(gfp_mask, order)
 #define alloc_hugepage_vma(gfp_mask, vma, addr, order) \
diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index d79fa299b70c..382fec380f28 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -2277,6 +2277,16 @@ struct page *alloc_pages(gfp_t gfp, unsigned order)
 }
 EXPORT_SYMBOL(alloc_pages);
 
+struct folio *alloc_folio(gfp_t gfp, unsigned order)
+{
+	struct page *page = alloc_pages(gfp | __GFP_COMP, order);
+
+	if (page && order > 1)
+		prep_transhuge_page(page);
+	return (struct folio *)page;
+}
+EXPORT_SYMBOL(alloc_folio);
+
 int vma_dup_policy(struct vm_area_struct *src, struct vm_area_struct *dst)
 {
 	struct mempolicy *pol = mpol_dup(vma_policy(src));
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 6f2e131d08b5..b466d0aaaa18 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -5225,6 +5225,18 @@ struct page *__alloc_pages(gfp_t gfp, unsigned int order, int preferred_nid,
 }
 EXPORT_SYMBOL(__alloc_pages);
 
+struct folio *__alloc_folio(gfp_t gfp, unsigned int order, int preferred_nid,
+		nodemask_t *nodemask)
+{
+	struct page *page = __alloc_pages(gfp | __GFP_COMP, order,
+			preferred_nid, nodemask);
+
+	if (page && order > 1)
+		prep_transhuge_page(page);
+	return (struct folio *)page;
+}
+EXPORT_SYMBOL(__alloc_folio);
+
 /*
  * Common helper functions. Never use with __GFP_HIGHMEM because the returned
  * address cannot represent highmem pages. Use alloc_pages and then kmap if
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 128+ messages in thread

* [PATCH v2 42/46] mm/filemap: Add filemap_alloc_folio
  2021-06-22 12:15 [PATCH v2 00/46] Folio-enabling the page cache Matthew Wilcox (Oracle)
                   ` (40 preceding siblings ...)
  2021-06-22 12:15 ` [PATCH v2 41/46] mm/page_alloc: Add folio allocation functions Matthew Wilcox (Oracle)
@ 2021-06-22 12:15 ` Matthew Wilcox (Oracle)
  2021-06-23  9:59   ` Christoph Hellwig
  2021-06-22 12:15 ` [PATCH v2 43/46] mm/filemap: Add filemap_add_folio Matthew Wilcox (Oracle)
                   ` (3 subsequent siblings)
  45 siblings, 1 reply; 128+ messages in thread
From: Matthew Wilcox (Oracle) @ 2021-06-22 12:15 UTC (permalink / raw)
  To: akpm; +Cc: Matthew Wilcox (Oracle), linux-fsdevel, linux-mm, linux-kernel

Reimplement __page_cache_alloc as a wrapper around filemap_alloc_folio
to allow filesystems to be converted at our leisure.  Increases
kernel text size by 133 bytes, mostly in cachefiles_read_backing_file().
pagecache_get_page() shrinks by 32 bytes, though.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 include/linux/pagemap.h | 11 ++++++++---
 mm/filemap.c            | 14 +++++++-------
 2 files changed, 15 insertions(+), 10 deletions(-)

diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
index c1df4c569148..7637cc9333c9 100644
--- a/include/linux/pagemap.h
+++ b/include/linux/pagemap.h
@@ -262,14 +262,19 @@ static inline void *detach_page_private(struct page *page)
 }
 
 #ifdef CONFIG_NUMA
-extern struct page *__page_cache_alloc(gfp_t gfp);
+struct folio *filemap_alloc_folio(gfp_t gfp, unsigned int order);
 #else
-static inline struct page *__page_cache_alloc(gfp_t gfp)
+static inline struct folio *filemap_alloc_folio(gfp_t gfp, unsigned int order)
 {
-	return alloc_pages(gfp, 0);
+	return alloc_folio(gfp, order);
 }
 #endif
 
+static inline struct page *__page_cache_alloc(gfp_t gfp)
+{
+	return &filemap_alloc_folio(gfp, 0)->page;
+}
+
 static inline struct page *page_cache_alloc(struct address_space *x)
 {
 	return __page_cache_alloc(mapping_gfp_mask(x));
diff --git a/mm/filemap.c b/mm/filemap.c
index 673094ce67da..4debb11ecc3e 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -989,24 +989,24 @@ int add_to_page_cache_lru(struct page *page, struct address_space *mapping,
 EXPORT_SYMBOL_GPL(add_to_page_cache_lru);
 
 #ifdef CONFIG_NUMA
-struct page *__page_cache_alloc(gfp_t gfp)
+struct folio *filemap_alloc_folio(gfp_t gfp, unsigned int order)
 {
 	int n;
-	struct page *page;
+	struct folio *folio;
 
 	if (cpuset_do_page_mem_spread()) {
 		unsigned int cpuset_mems_cookie;
 		do {
 			cpuset_mems_cookie = read_mems_allowed_begin();
 			n = cpuset_mem_spread_node();
-			page = __alloc_pages_node(n, gfp, 0);
-		} while (!page && read_mems_allowed_retry(cpuset_mems_cookie));
+			folio = __alloc_folio_node(gfp, order, n);
+		} while (!folio && read_mems_allowed_retry(cpuset_mems_cookie));
 
-		return page;
+		return folio;
 	}
-	return alloc_pages(gfp, 0);
+	return alloc_folio(gfp, order);
 }
-EXPORT_SYMBOL(__page_cache_alloc);
+EXPORT_SYMBOL(filemap_alloc_folio);
 #endif
 
 /*
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 128+ messages in thread

* [PATCH v2 43/46] mm/filemap: Add filemap_add_folio
  2021-06-22 12:15 [PATCH v2 00/46] Folio-enabling the page cache Matthew Wilcox (Oracle)
                   ` (41 preceding siblings ...)
  2021-06-22 12:15 ` [PATCH v2 42/46] mm/filemap: Add filemap_alloc_folio Matthew Wilcox (Oracle)
@ 2021-06-22 12:15 ` Matthew Wilcox (Oracle)
  2021-06-23 11:30   ` Christoph Hellwig
  2021-06-22 12:15 ` [PATCH v2 44/46] mm/filemap: Convert mapping_get_entry to return a folio Matthew Wilcox (Oracle)
                   ` (2 subsequent siblings)
  45 siblings, 1 reply; 128+ messages in thread
From: Matthew Wilcox (Oracle) @ 2021-06-22 12:15 UTC (permalink / raw)
  To: akpm; +Cc: Matthew Wilcox (Oracle), linux-fsdevel, linux-mm, linux-kernel

Pages being added to the page cache should already be folios, so
just cast the page to a folio in the add_to_page_cache_lru() wrapper.
Saves 96 bytes of text.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 include/linux/mm.h      |  7 -----
 include/linux/pagemap.h | 16 ++++++++--
 kernel/bpf/verifier.c   |  2 +-
 mm/filemap.c            | 69 ++++++++++++++++++++---------------------
 4 files changed, 47 insertions(+), 47 deletions(-)

diff --git a/include/linux/mm.h b/include/linux/mm.h
index d25ff74cf9e1..4ad03f4a9376 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -223,13 +223,6 @@ int overcommit_kbytes_handler(struct ctl_table *, int, void *, size_t *,
 		loff_t *);
 int overcommit_policy_handler(struct ctl_table *, int, void *, size_t *,
 		loff_t *);
-/*
- * Any attempt to mark this function as static leads to build failure
- * when CONFIG_DEBUG_INFO_BTF is enabled because __add_to_page_cache_locked()
- * is referred to by BPF code. This must be visible for error injection.
- */
-int __add_to_page_cache_locked(struct page *page, struct address_space *mapping,
-		pgoff_t index, gfp_t gfp, void **shadowp);
 
 #if defined(CONFIG_SPARSEMEM) && !defined(CONFIG_SPARSEMEM_VMEMMAP)
 #define nth_page(page,n) pfn_to_page(page_to_pfn((page)) + (n))
diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
index 7637cc9333c9..b0c1d24fb01b 100644
--- a/include/linux/pagemap.h
+++ b/include/linux/pagemap.h
@@ -876,9 +876,9 @@ static inline int fault_in_pages_readable(const char __user *uaddr, int size)
 }
 
 int add_to_page_cache_locked(struct page *page, struct address_space *mapping,
-				pgoff_t index, gfp_t gfp_mask);
-int add_to_page_cache_lru(struct page *page, struct address_space *mapping,
-				pgoff_t index, gfp_t gfp_mask);
+				pgoff_t index, gfp_t gfp);
+int filemap_add_folio(struct address_space *mapping, struct folio *folio,
+				pgoff_t index, gfp_t gfp);
 extern void delete_from_page_cache(struct page *page);
 extern void __delete_from_page_cache(struct page *page, void *shadow);
 void replace_page_cache_page(struct page *old, struct page *new);
@@ -903,6 +903,16 @@ static inline int add_to_page_cache(struct page *page,
 	return error;
 }
 
+static inline int add_to_page_cache_lru(struct page *page,
+		struct address_space *mapping, pgoff_t index, gfp_t gfp)
+{
+	return filemap_add_folio(mapping, (struct folio *)page, index, gfp);
+}
+
+/* Must be non-static for BPF error injection */
+int __filemap_add_folio(struct address_space *mapping, struct folio *folio,
+		pgoff_t index, gfp_t gfp, void **shadowp);
+
 /**
  * struct readahead_control - Describes a readahead request.
  *
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 94ba5163d4c5..cab4d64c1809 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -12962,7 +12962,7 @@ BTF_SET_START(btf_non_sleepable_error_inject)
 /* Three functions below can be called from sleepable and non-sleepable context.
  * Assume non-sleepable from bpf safety point of view.
  */
-BTF_ID(func, __add_to_page_cache_locked)
+BTF_ID(func, __filemap_add_folio)
 BTF_ID(func, should_fail_alloc_page)
 BTF_ID(func, should_failslab)
 BTF_SET_END(btf_non_sleepable_error_inject)
diff --git a/mm/filemap.c b/mm/filemap.c
index 4debb11ecc3e..a174f8ce87ea 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -855,26 +855,24 @@ void replace_page_cache_page(struct page *old, struct page *new)
 }
 EXPORT_SYMBOL_GPL(replace_page_cache_page);
 
-noinline int __add_to_page_cache_locked(struct page *page,
-					struct address_space *mapping,
-					pgoff_t offset, gfp_t gfp,
-					void **shadowp)
+noinline int __filemap_add_folio(struct address_space *mapping,
+		struct folio *folio, pgoff_t index, gfp_t gfp, void **shadowp)
 {
-	XA_STATE(xas, &mapping->i_pages, offset);
-	int huge = PageHuge(page);
+	XA_STATE(xas, &mapping->i_pages, index);
+	int huge = folio_hugetlb(folio);
 	int error;
 	bool charged = false;
 
-	VM_BUG_ON_PAGE(!PageLocked(page), page);
-	VM_BUG_ON_PAGE(PageSwapBacked(page), page);
+	VM_BUG_ON_FOLIO(!folio_locked(folio), folio);
+	VM_BUG_ON_FOLIO(folio_swapbacked(folio), folio);
 	mapping_set_update(&xas, mapping);
 
-	get_page(page);
-	page->mapping = mapping;
-	page->index = offset;
+	folio_get(folio);
+	folio->mapping = mapping;
+	folio->index = index;
 
 	if (!huge) {
-		error = mem_cgroup_charge(page, current->mm, gfp);
+		error = folio_charge_cgroup(folio, current->mm, gfp);
 		if (error)
 			goto error;
 		charged = true;
@@ -886,7 +884,7 @@ noinline int __add_to_page_cache_locked(struct page *page,
 		unsigned int order = xa_get_order(xas.xa, xas.xa_index);
 		void *entry, *old = NULL;
 
-		if (order > thp_order(page))
+		if (order > folio_order(folio))
 			xas_split_alloc(&xas, xa_load(xas.xa, xas.xa_index),
 					order, gfp);
 		xas_lock_irq(&xas);
@@ -903,13 +901,13 @@ noinline int __add_to_page_cache_locked(struct page *page,
 				*shadowp = old;
 			/* entry may have been split before we acquired lock */
 			order = xa_get_order(xas.xa, xas.xa_index);
-			if (order > thp_order(page)) {
+			if (order > folio_order(folio)) {
 				xas_split(&xas, old, order);
 				xas_reset(&xas);
 			}
 		}
 
-		xas_store(&xas, page);
+		xas_store(&xas, folio);
 		if (xas_error(&xas))
 			goto unlock;
 
@@ -917,7 +915,7 @@ noinline int __add_to_page_cache_locked(struct page *page,
 
 		/* hugetlb pages do not participate in page cache accounting */
 		if (!huge)
-			__inc_lruvec_page_state(page, NR_FILE_PAGES);
+			__lruvec_stat_add_folio(folio, NR_FILE_PAGES);
 unlock:
 		xas_unlock_irq(&xas);
 	} while (xas_nomem(&xas, gfp));
@@ -925,19 +923,19 @@ noinline int __add_to_page_cache_locked(struct page *page,
 	if (xas_error(&xas)) {
 		error = xas_error(&xas);
 		if (charged)
-			mem_cgroup_uncharge(page);
+			folio_uncharge_cgroup(folio);
 		goto error;
 	}
 
-	trace_mm_filemap_add_to_page_cache(page);
+	trace_mm_filemap_add_to_page_cache(&folio->page);
 	return 0;
 error:
-	page->mapping = NULL;
+	folio->mapping = NULL;
 	/* Leave page->index set: truncation relies upon it */
-	put_page(page);
+	folio_put(folio);
 	return error;
 }
-ALLOW_ERROR_INJECTION(__add_to_page_cache_locked, ERRNO);
+ALLOW_ERROR_INJECTION(__filemap_add_folio, ERRNO);
 
 /**
  * add_to_page_cache_locked - add a locked page to the pagecache
@@ -954,39 +952,38 @@ ALLOW_ERROR_INJECTION(__add_to_page_cache_locked, ERRNO);
 int add_to_page_cache_locked(struct page *page, struct address_space *mapping,
 		pgoff_t offset, gfp_t gfp_mask)
 {
-	return __add_to_page_cache_locked(page, mapping, offset,
+	return __filemap_add_folio(mapping, page_folio(page), offset,
 					  gfp_mask, NULL);
 }
 EXPORT_SYMBOL(add_to_page_cache_locked);
 
-int add_to_page_cache_lru(struct page *page, struct address_space *mapping,
-				pgoff_t offset, gfp_t gfp_mask)
+int filemap_add_folio(struct address_space *mapping, struct folio *folio,
+				pgoff_t index, gfp_t gfp)
 {
 	void *shadow = NULL;
 	int ret;
 
-	__SetPageLocked(page);
-	ret = __add_to_page_cache_locked(page, mapping, offset,
-					 gfp_mask, &shadow);
+	__folio_set_locked_flag(folio);
+	ret = __filemap_add_folio(mapping, folio, index, gfp, &shadow);
 	if (unlikely(ret))
-		__ClearPageLocked(page);
+		__folio_clear_locked_flag(folio);
 	else {
 		/*
-		 * The page might have been evicted from cache only
+		 * The folio might have been evicted from cache only
 		 * recently, in which case it should be activated like
-		 * any other repeatedly accessed page.
-		 * The exception is pages getting rewritten; evicting other
+		 * any other repeatedly accessed folio.
+		 * The exception is folios getting rewritten; evicting other
 		 * data from the working set, only to cache data that will
 		 * get overwritten with something else, is a waste of memory.
 		 */
-		WARN_ON_ONCE(PageActive(page));
-		if (!(gfp_mask & __GFP_WRITE) && shadow)
-			workingset_refault(page_folio(page), shadow);
-		lru_cache_add(page);
+		WARN_ON_ONCE(folio_active(folio));
+		if (!(gfp & __GFP_WRITE) && shadow)
+			workingset_refault(folio, shadow);
+		folio_add_lru(folio);
 	}
 	return ret;
 }
-EXPORT_SYMBOL_GPL(add_to_page_cache_lru);
+EXPORT_SYMBOL_GPL(filemap_add_folio);
 
 #ifdef CONFIG_NUMA
 struct folio *filemap_alloc_folio(gfp_t gfp, unsigned int order)
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 128+ messages in thread

* [PATCH v2 44/46] mm/filemap: Convert mapping_get_entry to return a folio
  2021-06-22 12:15 [PATCH v2 00/46] Folio-enabling the page cache Matthew Wilcox (Oracle)
                   ` (42 preceding siblings ...)
  2021-06-22 12:15 ` [PATCH v2 43/46] mm/filemap: Add filemap_add_folio Matthew Wilcox (Oracle)
@ 2021-06-22 12:15 ` Matthew Wilcox (Oracle)
  2021-06-23 11:32   ` Christoph Hellwig
  2021-06-22 12:15 ` [PATCH v2 45/46] mm/filemap: Add filemap_get_folio Matthew Wilcox (Oracle)
  2021-06-22 12:15 ` [PATCH v2 46/46] mm/filemap: Add FGP_STABLE Matthew Wilcox (Oracle)
  45 siblings, 1 reply; 128+ messages in thread
From: Matthew Wilcox (Oracle) @ 2021-06-22 12:15 UTC (permalink / raw)
  To: akpm; +Cc: Matthew Wilcox (Oracle), linux-fsdevel, linux-mm, linux-kernel

The pagecache only contains folios, so indicate that this is definitely
not a tail page.  Shrinks mapping_get_entry() by 56 bytes, but grows
pagecache_get_page() by 21 bytes as gcc makes slightly different hot/cold
code decisions.  A net reduction of 35 bytes of text.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 mm/filemap.c | 38 +++++++++++++++++---------------------
 1 file changed, 17 insertions(+), 21 deletions(-)

diff --git a/mm/filemap.c b/mm/filemap.c
index a174f8ce87ea..7d0b51f30223 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -1754,49 +1754,43 @@ EXPORT_SYMBOL(page_cache_prev_miss);
  * @mapping: the address_space to search
  * @index: The page cache index.
  *
- * Looks up the page cache slot at @mapping & @index.  If there is a
- * page cache page, the head page is returned with an increased refcount.
+ * Looks up the page cache entry at @mapping & @index.  If it is a folio,
+ * it is returned with an increased refcount.  If it is a shadow entry
+ * of a previously evicted folio, or a swap entry from shmem/tmpfs,
+ * it is returned without further action.
  *
- * If the slot holds a shadow entry of a previously evicted page, or a
- * swap entry from shmem/tmpfs, it is returned.
- *
- * Return: The head page or shadow entry, %NULL if nothing is found.
+ * Return: The folio, swap or shadow entry, %NULL if nothing is found.
  */
-static struct page *mapping_get_entry(struct address_space *mapping,
+static struct folio *mapping_get_entry(struct address_space *mapping,
 		pgoff_t index)
 {
 	XA_STATE(xas, &mapping->i_pages, index);
-	struct page *page;
+	struct folio *folio;
 
 	rcu_read_lock();
 repeat:
 	xas_reset(&xas);
-	page = xas_load(&xas);
-	if (xas_retry(&xas, page))
+	folio = xas_load(&xas);
+	if (xas_retry(&xas, folio))
 		goto repeat;
 	/*
 	 * A shadow entry of a recently evicted page, or a swap entry from
 	 * shmem/tmpfs.  Return it without attempting to raise page count.
 	 */
-	if (!page || xa_is_value(page))
+	if (!folio || xa_is_value(folio))
 		goto out;
 
-	if (!page_cache_get_speculative(page))
+	if (!folio_try_get_rcu(folio))
 		goto repeat;
 
-	/*
-	 * Has the page moved or been split?
-	 * This is part of the lockless pagecache protocol. See
-	 * include/linux/pagemap.h for details.
-	 */
-	if (unlikely(page != xas_reload(&xas))) {
-		put_page(page);
+	if (unlikely(folio != xas_reload(&xas))) {
+		folio_put(folio);
 		goto repeat;
 	}
 out:
 	rcu_read_unlock();
 
-	return page;
+	return folio;
 }
 
 /**
@@ -1836,10 +1830,12 @@ static struct page *mapping_get_entry(struct address_space *mapping,
 struct page *pagecache_get_page(struct address_space *mapping, pgoff_t index,
 		int fgp_flags, gfp_t gfp_mask)
 {
+	struct folio *folio;
 	struct page *page;
 
 repeat:
-	page = mapping_get_entry(mapping, index);
+	folio = mapping_get_entry(mapping, index);
+	page = &folio->page;
 	if (xa_is_value(page)) {
 		if (fgp_flags & FGP_ENTRY)
 			return page;
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 128+ messages in thread

* [PATCH v2 45/46] mm/filemap: Add filemap_get_folio
  2021-06-22 12:15 [PATCH v2 00/46] Folio-enabling the page cache Matthew Wilcox (Oracle)
                   ` (43 preceding siblings ...)
  2021-06-22 12:15 ` [PATCH v2 44/46] mm/filemap: Convert mapping_get_entry to return a folio Matthew Wilcox (Oracle)
@ 2021-06-22 12:15 ` Matthew Wilcox (Oracle)
  2021-06-23 11:39   ` Christoph Hellwig
  2021-06-22 12:15 ` [PATCH v2 46/46] mm/filemap: Add FGP_STABLE Matthew Wilcox (Oracle)
  45 siblings, 1 reply; 128+ messages in thread
From: Matthew Wilcox (Oracle) @ 2021-06-22 12:15 UTC (permalink / raw)
  To: akpm; +Cc: Matthew Wilcox (Oracle), linux-fsdevel, linux-mm, linux-kernel

filemap_get_folio() is a replacement for find_get_page().
Turn pagecache_get_page() into a wrapper around __filemap_get_folio().
Remove find_lock_head() as this use case is now covered by
filemap_get_folio().

Reduces overall kernel size by 209 bytes.  __filemap_get_folio() is
316 bytes shorter than pagecache_get_page() was, but the new
pagecache_get_page() is 99 bytes.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 include/linux/pagemap.h | 41 +++++++++----------
 mm/filemap.c            | 90 +++++++++++++++++++----------------------
 mm/folio-compat.c       | 12 ++++++
 3 files changed, 74 insertions(+), 69 deletions(-)

diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
index b0c1d24fb01b..669bdebe2861 100644
--- a/include/linux/pagemap.h
+++ b/include/linux/pagemap.h
@@ -302,8 +302,26 @@ pgoff_t page_cache_prev_miss(struct address_space *mapping,
 #define FGP_HEAD		0x00000080
 #define FGP_ENTRY		0x00000100
 
-struct page *pagecache_get_page(struct address_space *mapping, pgoff_t offset,
-		int fgp_flags, gfp_t cache_gfp_mask);
+struct folio *__filemap_get_folio(struct address_space *mapping, pgoff_t index,
+		int fgp_flags, gfp_t gfp);
+struct page *pagecache_get_page(struct address_space *mapping, pgoff_t index,
+		int fgp_flags, gfp_t gfp);
+
+/**
+ * filemap_get_folio - Find and get a folio.
+ * @mapping: The address_space to search.
+ * @index: The page index.
+ *
+ * Looks up the page cache entry at @mapping & @index.  If a folio is
+ * present, it is returned with an increased refcount.
+ *
+ * Otherwise, %NULL is returned.
+ */
+static inline struct folio *filemap_get_folio(struct address_space *mapping,
+					pgoff_t index)
+{
+	return __filemap_get_folio(mapping, index, 0, 0);
+}
 
 /**
  * find_get_page - find and get a page reference
@@ -346,25 +364,6 @@ static inline struct page *find_lock_page(struct address_space *mapping,
 	return pagecache_get_page(mapping, index, FGP_LOCK, 0);
 }
 
-/**
- * find_lock_head - Locate, pin and lock a pagecache page.
- * @mapping: The address_space to search.
- * @index: The page index.
- *
- * Looks up the page cache entry at @mapping & @index.  If there is a
- * page cache page, its head page is returned locked and with an increased
- * refcount.
- *
- * Context: May sleep.
- * Return: A struct page which is !PageTail, or %NULL if there is no page
- * in the cache for this index.
- */
-static inline struct page *find_lock_head(struct address_space *mapping,
-					pgoff_t index)
-{
-	return pagecache_get_page(mapping, index, FGP_LOCK | FGP_HEAD, 0);
-}
-
 /**
  * find_or_create_page - locate or add a pagecache page
  * @mapping: the page's address_space
diff --git a/mm/filemap.c b/mm/filemap.c
index 7d0b51f30223..e431461279c3 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -1794,95 +1794,89 @@ static struct folio *mapping_get_entry(struct address_space *mapping,
 }
 
 /**
- * pagecache_get_page - Find and get a reference to a page.
+ * __filemap_get_folio - Find and get a reference to a folio.
  * @mapping: The address_space to search.
  * @index: The page index.
- * @fgp_flags: %FGP flags modify how the page is returned.
- * @gfp_mask: Memory allocation flags to use if %FGP_CREAT is specified.
+ * @fgp_flags: %FGP flags modify how the folio is returned.
+ * @gfp: Memory allocation flags to use if %FGP_CREAT is specified.
  *
  * Looks up the page cache entry at @mapping & @index.
  *
  * @fgp_flags can be zero or more of these flags:
  *
- * * %FGP_ACCESSED - The page will be marked accessed.
- * * %FGP_LOCK - The page is returned locked.
- * * %FGP_HEAD - If the page is present and a THP, return the head page
- *   rather than the exact page specified by the index.
+ * * %FGP_ACCESSED - The folio will be marked accessed.
+ * * %FGP_LOCK - The folio is returned locked.
  * * %FGP_ENTRY - If there is a shadow / swap / DAX entry, return it
- *   instead of allocating a new page to replace it.
+ *   instead of allocating a new folio to replace it.
  * * %FGP_CREAT - If no page is present then a new page is allocated using
- *   @gfp_mask and added to the page cache and the VM's LRU list.
+ *   @gfp and added to the page cache and the VM's LRU list.
  *   The page is returned locked and with an increased refcount.
  * * %FGP_FOR_MMAP - The caller wants to do its own locking dance if the
  *   page is already in cache.  If the page was allocated, unlock it before
  *   returning so the caller can do the same dance.
- * * %FGP_WRITE - The page will be written
- * * %FGP_NOFS - __GFP_FS will get cleared in gfp mask
- * * %FGP_NOWAIT - Don't get blocked by page lock
+ * * %FGP_WRITE - The page will be written to by the caller.
+ * * %FGP_NOFS - __GFP_FS will get cleared in gfp.
+ * * %FGP_NOWAIT - Don't get blocked by page lock.
  *
  * If %FGP_LOCK or %FGP_CREAT are specified then the function may sleep even
  * if the %GFP flags specified for %FGP_CREAT are atomic.
  *
  * If there is a page cache page, it is returned with an increased refcount.
  *
- * Return: The found page or %NULL otherwise.
+ * Return: The found folio or %NULL otherwise.
  */
-struct page *pagecache_get_page(struct address_space *mapping, pgoff_t index,
-		int fgp_flags, gfp_t gfp_mask)
+struct folio *__filemap_get_folio(struct address_space *mapping, pgoff_t index,
+		int fgp_flags, gfp_t gfp)
 {
 	struct folio *folio;
-	struct page *page;
 
 repeat:
 	folio = mapping_get_entry(mapping, index);
-	page = &folio->page;
-	if (xa_is_value(page)) {
+	if (xa_is_value(folio)) {
 		if (fgp_flags & FGP_ENTRY)
-			return page;
-		page = NULL;
+			return folio;
+		folio = NULL;
 	}
-	if (!page)
+	if (!folio)
 		goto no_page;
 
 	if (fgp_flags & FGP_LOCK) {
 		if (fgp_flags & FGP_NOWAIT) {
-			if (!trylock_page(page)) {
-				put_page(page);
+			if (!folio_trylock(folio)) {
+				folio_put(folio);
 				return NULL;
 			}
 		} else {
-			lock_page(page);
+			folio_lock(folio);
 		}
 
 		/* Has the page been truncated? */
-		if (unlikely(page->mapping != mapping)) {
-			unlock_page(page);
-			put_page(page);
+		if (unlikely(folio->mapping != mapping)) {
+			folio_unlock(folio);
+			folio_put(folio);
 			goto repeat;
 		}
-		VM_BUG_ON_PAGE(!thp_contains(page, index), page);
+		VM_BUG_ON_FOLIO(!folio_contains(folio, index), folio);
 	}
 
 	if (fgp_flags & FGP_ACCESSED)
-		mark_page_accessed(page);
+		folio_mark_accessed(folio);
 	else if (fgp_flags & FGP_WRITE) {
 		/* Clear idle flag for buffer write */
-		if (page_is_idle(page))
-			clear_page_idle(page);
+		if (folio_idle(folio))
+			folio_clear_idle_flag(folio);
 	}
-	if (!(fgp_flags & FGP_HEAD))
-		page = find_subpage(page, index);
 
 no_page:
-	if (!page && (fgp_flags & FGP_CREAT)) {
+	if (!folio && (fgp_flags & FGP_CREAT)) {
 		int err;
 		if ((fgp_flags & FGP_WRITE) && mapping_can_writeback(mapping))
-			gfp_mask |= __GFP_WRITE;
+			gfp |= __GFP_WRITE;
 		if (fgp_flags & FGP_NOFS)
-			gfp_mask &= ~__GFP_FS;
+			gfp &= ~__GFP_FS;
 
-		page = __page_cache_alloc(gfp_mask);
-		if (!page)
+		folio = filemap_alloc_folio(gfp, 0);
+		if (!folio)
 			return NULL;
 
 		if (WARN_ON_ONCE(!(fgp_flags & (FGP_LOCK | FGP_FOR_MMAP))))
@@ -1890,27 +1884,27 @@ struct page *pagecache_get_page(struct address_space *mapping, pgoff_t index,
 
 		/* Init accessed so avoid atomic mark_page_accessed later */
 		if (fgp_flags & FGP_ACCESSED)
-			__SetPageReferenced(page);
+			__folio_set_referenced_flag(folio);
 
-		err = add_to_page_cache_lru(page, mapping, index, gfp_mask);
+		err = filemap_add_folio(mapping, folio, index, gfp);
 		if (unlikely(err)) {
-			put_page(page);
-			page = NULL;
+			folio_put(folio);
+			folio = NULL;
 			if (err == -EEXIST)
 				goto repeat;
 		}
 
 		/*
-		 * add_to_page_cache_lru locks the page, and for mmap we expect
-		 * an unlocked page.
+		 * filemap_add_folio locks the page, and for mmap
+		 * we expect an unlocked page.
 		 */
-		if (page && (fgp_flags & FGP_FOR_MMAP))
-			unlock_page(page);
+		if (folio && (fgp_flags & FGP_FOR_MMAP))
+			folio_unlock(folio);
 	}
 
-	return page;
+	return folio;
 }
-EXPORT_SYMBOL(pagecache_get_page);
+EXPORT_SYMBOL(__filemap_get_folio);
 
 static inline struct page *find_get_entry(struct xa_state *xas, pgoff_t max,
 		xa_mark_t mark)
diff --git a/mm/folio-compat.c b/mm/folio-compat.c
index 0a8bdd3f8d91..78365eaee7d3 100644
--- a/mm/folio-compat.c
+++ b/mm/folio-compat.c
@@ -114,3 +114,15 @@ void lru_cache_add(struct page *page)
 	folio_add_lru(page_folio(page));
 }
 EXPORT_SYMBOL(lru_cache_add);
+
+struct page *pagecache_get_page(struct address_space *mapping, pgoff_t index,
+		int fgp_flags, gfp_t gfp)
+{
+	struct folio *folio;
+
+	folio = __filemap_get_folio(mapping, index, fgp_flags, gfp);
+	if ((fgp_flags & FGP_HEAD) || !folio || xa_is_value(folio))
+		return &folio->page;
+	return folio_file_page(folio, index);
+}
+EXPORT_SYMBOL(pagecache_get_page);
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 128+ messages in thread

* [PATCH v2 46/46] mm/filemap: Add FGP_STABLE
  2021-06-22 12:15 [PATCH v2 00/46] Folio-enabling the page cache Matthew Wilcox (Oracle)
                   ` (44 preceding siblings ...)
  2021-06-22 12:15 ` [PATCH v2 45/46] mm/filemap: Add filemap_get_folio Matthew Wilcox (Oracle)
@ 2021-06-22 12:15 ` Matthew Wilcox (Oracle)
  2021-06-23 11:43   ` Christoph Hellwig
  45 siblings, 1 reply; 128+ messages in thread
From: Matthew Wilcox (Oracle) @ 2021-06-22 12:15 UTC (permalink / raw)
  To: akpm; +Cc: Matthew Wilcox (Oracle), linux-fsdevel, linux-mm, linux-kernel

Allow filemap_get_folio() to wait for writeback to complete (if the
filesystem wants that behaviour).  This is the folio equivalent of
grab_cache_page_write_begin(), which is moved into the folio-compat
file as a reminder to migrate all the code using it.  This paves the
way for getting rid of AOP_FLAG_NOFS once grab_cache_page_write_begin()
is removed.

Kernel grows by 11 bytes.  filemap_get_folio() grows by 33 bytes
but grab_cache_page_write_begin() shrinks by 22 bytes to make up
for it.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 include/linux/pagemap.h |  1 +
 mm/filemap.c            | 25 +++----------------------
 mm/folio-compat.c       | 13 +++++++++++++
 3 files changed, 17 insertions(+), 22 deletions(-)

diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
index 669bdebe2861..865b7784e99b 100644
--- a/include/linux/pagemap.h
+++ b/include/linux/pagemap.h
@@ -301,6 +301,7 @@ pgoff_t page_cache_prev_miss(struct address_space *mapping,
 #define FGP_FOR_MMAP		0x00000040
 #define FGP_HEAD		0x00000080
 #define FGP_ENTRY		0x00000100
+#define FGP_STABLE		0x00000200
 
 struct folio *__filemap_get_folio(struct address_space *mapping, pgoff_t index,
 		int fgp_flags, gfp_t gfp);
diff --git a/mm/filemap.c b/mm/filemap.c
index e431461279c3..4951fe8787d8 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -1817,6 +1817,7 @@ static struct folio *mapping_get_entry(struct address_space *mapping,
  * * %FGP_WRITE - The page will be written to by the caller.
  * * %FGP_NOFS - __GFP_FS will get cleared in gfp.
  * * %FGP_NOWAIT - Don't get blocked by page lock.
+ * * %FGP_STABLE - Wait for the folio to be stable (finished writeback)
  *
  * If %FGP_LOCK or %FGP_CREAT are specified then the function may sleep even
  * if the %GFP flags specified for %FGP_CREAT are atomic.
@@ -1867,6 +1868,8 @@ struct folio *__filemap_get_folio(struct address_space *mapping, pgoff_t index,
 			folio_clear_idle_flag(folio);
 	}
 
+	if (fgp_flags & FGP_STABLE)
+		folio_wait_stable(folio);
 no_page:
 	if (!folio && (fgp_flags & FGP_CREAT)) {
 		int err;
@@ -3590,28 +3593,6 @@ generic_file_direct_write(struct kiocb *iocb, struct iov_iter *from)
 }
 EXPORT_SYMBOL(generic_file_direct_write);
 
-/*
- * Find or create a page at the given pagecache position. Return the locked
- * page. This function is specifically for buffered writes.
- */
-struct page *grab_cache_page_write_begin(struct address_space *mapping,
-					pgoff_t index, unsigned flags)
-{
-	struct page *page;
-	int fgp_flags = FGP_LOCK|FGP_WRITE|FGP_CREAT;
-
-	if (flags & AOP_FLAG_NOFS)
-		fgp_flags |= FGP_NOFS;
-
-	page = pagecache_get_page(mapping, index, fgp_flags,
-			mapping_gfp_mask(mapping));
-	if (page)
-		wait_for_stable_page(page);
-
-	return page;
-}
-EXPORT_SYMBOL(grab_cache_page_write_begin);
-
 ssize_t generic_perform_write(struct file *file,
 				struct iov_iter *i, loff_t pos)
 {
diff --git a/mm/folio-compat.c b/mm/folio-compat.c
index 78365eaee7d3..206bedd621d0 100644
--- a/mm/folio-compat.c
+++ b/mm/folio-compat.c
@@ -115,6 +115,7 @@ void lru_cache_add(struct page *page)
 }
 EXPORT_SYMBOL(lru_cache_add);
 
+noinline
 struct page *pagecache_get_page(struct address_space *mapping, pgoff_t index,
 		int fgp_flags, gfp_t gfp)
 {
@@ -126,3 +127,15 @@ struct page *pagecache_get_page(struct address_space *mapping, pgoff_t index,
 	return folio_file_page(folio, index);
 }
 EXPORT_SYMBOL(pagecache_get_page);
+
+struct page *grab_cache_page_write_begin(struct address_space *mapping,
+					pgoff_t index, unsigned flags)
+{
+	unsigned fgp_flags = FGP_LOCK | FGP_WRITE | FGP_CREAT | FGP_STABLE;
+
+	if (flags & AOP_FLAG_NOFS)
+		fgp_flags |= FGP_NOFS;
+	return pagecache_get_page(mapping, index, fgp_flags,
+			mapping_gfp_mask(mapping));
+}
+EXPORT_SYMBOL(grab_cache_page_write_begin);
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 01/46] mm: Add folio_to_pfn()
  2021-06-22 12:15 ` [PATCH v2 01/46] mm: Add folio_to_pfn() Matthew Wilcox (Oracle)
@ 2021-06-23  7:49   ` Christoph Hellwig
  2021-06-24 15:12     ` Matthew Wilcox
  0 siblings, 1 reply; 128+ messages in thread
From: Christoph Hellwig @ 2021-06-23  7:49 UTC (permalink / raw)
  To: Matthew Wilcox (Oracle); +Cc: akpm, linux-fsdevel, linux-mm, linux-kernel

On Tue, Jun 22, 2021 at 01:15:06PM +0100, Matthew Wilcox (Oracle) wrote:
> The pfn of a folio is the pfn of its head page.
> 
> Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>

Maybe add a kerneldoc comment stating that?

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 02/46] mm: Add folio_rmapping()
  2021-06-22 12:15 ` [PATCH v2 02/46] mm: Add folio_rmapping() Matthew Wilcox (Oracle)
@ 2021-06-23  7:56   ` Christoph Hellwig
  2021-06-24 15:51     ` Matthew Wilcox
  0 siblings, 1 reply; 128+ messages in thread
From: Christoph Hellwig @ 2021-06-23  7:56 UTC (permalink / raw)
  To: Matthew Wilcox (Oracle); +Cc: akpm, linux-fsdevel, linux-mm, linux-kernel

> +static inline void *folio_rmapping(struct folio *folio)

This name, just like the old one is not exaclty descriptive.  I guess the
r stands for raw somehow?  As a casual contributor to the fringes of the
MM I would have no idea when to use it.

All this of course also applies to the existing (__)page_rmapping, but
maybe this is a good time to sort it out.

>  
>  struct anon_vma *page_anon_vma(struct page *page)
>  {
> +	struct folio *folio = page_folio(page);
> +	unsigned long mapping = (unsigned long)folio->mapping;
>  
>  	if ((mapping & PAGE_MAPPING_FLAGS) != PAGE_MAPPING_ANON)
>  		return NULL;
> +	return folio_rmapping(folio);

It feelds kinda silly to not just open code folio_rmapping here
given that we alredy went half the way. 

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 03/46] mm: Add kmap_local_folio()
  2021-06-22 12:15 ` [PATCH v2 03/46] mm: Add kmap_local_folio() Matthew Wilcox (Oracle)
@ 2021-06-23  7:58   ` Christoph Hellwig
  0 siblings, 0 replies; 128+ messages in thread
From: Christoph Hellwig @ 2021-06-23  7:58 UTC (permalink / raw)
  To: Matthew Wilcox (Oracle); +Cc: akpm, linux-fsdevel, linux-mm, linux-kernel

On Tue, Jun 22, 2021 at 01:15:08PM +0100, Matthew Wilcox (Oracle) wrote:
> This allows us to map a portion of a folio.  Callers can only expect
> to access up to the next page boundary.
> 
> Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>

This looks sensible to me:

Reviewed-by: Christoph Hellwig <hch@lst.de>

While we're at it:  flushing incoherent mappings for use with kmap
has been a complete mess.  I've been trying to fix this with
kunmap_local_dirty, but that could use another review, as well as a
folio version probably give that one prime use case of folios is to have
them mapped into userspace.

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 05/46] mm: Add arch_make_folio_accessible()
  2021-06-22 12:15 ` [PATCH v2 05/46] mm: Add arch_make_folio_accessible() Matthew Wilcox (Oracle)
@ 2021-06-23  8:00   ` Christoph Hellwig
  2021-06-24 15:57     ` Matthew Wilcox
  0 siblings, 1 reply; 128+ messages in thread
From: Christoph Hellwig @ 2021-06-23  8:00 UTC (permalink / raw)
  To: Matthew Wilcox (Oracle); +Cc: akpm, linux-fsdevel, linux-mm, linux-kernel

On Tue, Jun 22, 2021 at 01:15:10PM +0100, Matthew Wilcox (Oracle) wrote:
> As a default implementation, call arch_make_page_accessible n times.
> If an architecture can do better, it can override this.
> 
> Also move the default implementation of arch_make_page_accessible()
> from gfp.h to mm.h.

Can we wait with introducing arch hooks until we have an actual user
lined up?

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 07/46] mm/workingset: Convert workingset_activation to take a folio
  2021-06-22 12:15 ` [PATCH v2 07/46] mm/workingset: Convert workingset_activation to take a folio Matthew Wilcox (Oracle)
@ 2021-06-23  8:02   ` Christoph Hellwig
  0 siblings, 0 replies; 128+ messages in thread
From: Christoph Hellwig @ 2021-06-23  8:02 UTC (permalink / raw)
  To: Matthew Wilcox (Oracle); +Cc: akpm, linux-fsdevel, linux-mm, linux-kernel

On Tue, Jun 22, 2021 at 01:15:12PM +0100, Matthew Wilcox (Oracle) wrote:
> This function already assumed it was being passed a head page.  No real
> change here, except that thp_nr_pages() compiles away on kernels with
> THP compiled out while folio_nr_pages() is always present.
> 
> Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>

Looks good,

Reviewed-by: Christoph Hellwig <hch@lst.de>

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 08/46] mm/swap: Add folio_activate()
  2021-06-22 12:15 ` [PATCH v2 08/46] mm/swap: Add folio_activate() Matthew Wilcox (Oracle)
@ 2021-06-23  8:04   ` Christoph Hellwig
  0 siblings, 0 replies; 128+ messages in thread
From: Christoph Hellwig @ 2021-06-23  8:04 UTC (permalink / raw)
  To: Matthew Wilcox (Oracle); +Cc: akpm, linux-fsdevel, linux-mm, linux-kernel

On Tue, Jun 22, 2021 at 01:15:13PM +0100, Matthew Wilcox (Oracle) wrote:
> This replaces activate_page() and eliminates lots of calls to
> compound_head().  Saves net 118 bytes of kernel text.  There are still
> some redundant calls to page_folio() here which will be removed when
> pagevec_lru_move_fn() is converted to use folios.
> 
> Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>

Looks good,

Reviewed-by: Christoph Hellwig <hch@lst.de>

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 09/46] mm/swap: Add folio_mark_accessed()
  2021-06-22 12:15 ` [PATCH v2 09/46] mm/swap: Add folio_mark_accessed() Matthew Wilcox (Oracle)
@ 2021-06-23  8:07   ` Christoph Hellwig
  0 siblings, 0 replies; 128+ messages in thread
From: Christoph Hellwig @ 2021-06-23  8:07 UTC (permalink / raw)
  To: Matthew Wilcox (Oracle); +Cc: akpm, linux-fsdevel, linux-mm, linux-kernel

On Tue, Jun 22, 2021 at 01:15:14PM +0100, Matthew Wilcox (Oracle) wrote:
> Convert mark_page_accessed() to folio_mark_accessed().  It already
> operated on the entire compound page, but now we can avoid calling
> compound_head quite so many times.  Shrinks the function from 424 bytes
> to 295 bytes (shrinking by 129 bytes).  The compatibility wrapper is 30
> bytes, plus the 8 bytes for the exported symbol means the kernel shrinks
> by 91 bytes.

Looks good,

Reviewed-by: Christoph Hellwig <hch@lst.de>

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 10/46] mm/rmap: Add folio_mkclean()
  2021-06-22 12:15 ` [PATCH v2 10/46] mm/rmap: Add folio_mkclean() Matthew Wilcox (Oracle)
@ 2021-06-23  8:08   ` Christoph Hellwig
  0 siblings, 0 replies; 128+ messages in thread
From: Christoph Hellwig @ 2021-06-23  8:08 UTC (permalink / raw)
  To: Matthew Wilcox (Oracle); +Cc: akpm, linux-fsdevel, linux-mm, linux-kernel

On Tue, Jun 22, 2021 at 01:15:15PM +0100, Matthew Wilcox (Oracle) wrote:
> Transform page_mkclean() into folio_mkclean() and add a page_mkclean()
> wrapper around folio_mkclean().
> 
> folio_mkclean is 15 bytes smaller than page_mkclean, but the kernel
> is enlarged by 33 bytes due to inlining page_folio() into each caller.
> This will go away once the callers are converted to use folio_mkclean().
> 
> Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>

Looks good,

Reviewed-by: Christoph Hellwig <hch@lst.de>

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 11/46] mm/memcg: Remove 'page' parameter to mem_cgroup_charge_statistics()
  2021-06-22 12:15 ` [PATCH v2 11/46] mm/memcg: Remove 'page' parameter to mem_cgroup_charge_statistics() Matthew Wilcox (Oracle)
@ 2021-06-23  8:09   ` Christoph Hellwig
  2021-06-25  7:58   ` Michal Hocko
  1 sibling, 0 replies; 128+ messages in thread
From: Christoph Hellwig @ 2021-06-23  8:09 UTC (permalink / raw)
  To: Matthew Wilcox (Oracle); +Cc: akpm, linux-fsdevel, linux-mm, linux-kernel

On Tue, Jun 22, 2021 at 01:15:16PM +0100, Matthew Wilcox (Oracle) wrote:
> The last use of 'page' was removed by commit 468c398233da ("mm:
> memcontrol: switch to native NR_ANON_THPS counter"), so we can now remove
> the parameter from the function.
> 
> Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>

Looks good,

Reviewed-by: Christoph Hellwig <hch@lst.de>

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 12/46] mm/memcg: Use the node id in mem_cgroup_update_tree()
  2021-06-22 12:15 ` [PATCH v2 12/46] mm/memcg: Use the node id in mem_cgroup_update_tree() Matthew Wilcox (Oracle)
@ 2021-06-23  8:12   ` Christoph Hellwig
  2021-06-24 16:19     ` Matthew Wilcox
  0 siblings, 1 reply; 128+ messages in thread
From: Christoph Hellwig @ 2021-06-23  8:12 UTC (permalink / raw)
  To: Matthew Wilcox (Oracle); +Cc: akpm, linux-fsdevel, linux-mm, linux-kernel

On Tue, Jun 22, 2021 at 01:15:17PM +0100, Matthew Wilcox (Oracle) wrote:
>  static struct mem_cgroup_per_node *
> -mem_cgroup_page_nodeinfo(struct mem_cgroup *memcg, struct page *page)
> +mem_cgroup_nodeinfo(struct mem_cgroup *memcg, int nid)
>  {
> -	int nid = page_to_nid(page);
> -
>  	return memcg->nodeinfo[nid];
>  }

I'd just kill this function entirely and open code it into the only
caller

> -	mctz = soft_limit_tree_from_page(page);
> +	mctz = soft_limit_tree_node(nid);

And while were at it, soft_limit_tree_node seems like a completely
pointless helper that does nothing but obsfucating the code.  While
you touch this area it might be worth to spin another patch to just
remove it as well.

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 13/46] mm/memcg: Convert commit_charge() to take a folio
  2021-06-22 12:15 ` [PATCH v2 13/46] mm/memcg: Convert commit_charge() to take a folio Matthew Wilcox (Oracle)
@ 2021-06-23  8:13   ` Christoph Hellwig
  2021-06-25  8:11   ` Michal Hocko
  1 sibling, 0 replies; 128+ messages in thread
From: Christoph Hellwig @ 2021-06-23  8:13 UTC (permalink / raw)
  To: Matthew Wilcox (Oracle); +Cc: akpm, linux-fsdevel, linux-mm, linux-kernel

On Tue, Jun 22, 2021 at 01:15:18PM +0100, Matthew Wilcox (Oracle) wrote:
> The memcg_data is only set on the head page, so enforce that by
> typing it as a folio.
> 
> Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>

Looks good,

Reviewed-by: Christoph Hellwig <hch@lst.de>

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 14/46] mm/memcg: Add folio_charge_cgroup()
  2021-06-22 12:15 ` [PATCH v2 14/46] mm/memcg: Add folio_charge_cgroup() Matthew Wilcox (Oracle)
@ 2021-06-23  8:15   ` Christoph Hellwig
  2021-06-24 16:42     ` Matthew Wilcox
  0 siblings, 1 reply; 128+ messages in thread
From: Christoph Hellwig @ 2021-06-23  8:15 UTC (permalink / raw)
  To: Matthew Wilcox (Oracle); +Cc: akpm, linux-fsdevel, linux-mm, linux-kernel

On Tue, Jun 22, 2021 at 01:15:19PM +0100, Matthew Wilcox (Oracle) wrote:
> mem_cgroup_charge() already assumed it was being passed a non-tail
> page (and looking at the callers, that's true; it's called for freshly
> allocated pages).  The only real change here is that folio_nr_pages()
> doesn't compile away like thp_nr_pages() does as folio support
> is not conditional on transparent hugepage support.  Reimplement
> mem_cgroup_charge() as a wrapper around folio_charge_cgroup().

Maybe rename __mem_cgroup_charge to __folio_charge_cgroup as well?

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 15/46] mm/memcg: Add folio_uncharge_cgroup()
  2021-06-22 12:15 ` [PATCH v2 15/46] mm/memcg: Add folio_uncharge_cgroup() Matthew Wilcox (Oracle)
@ 2021-06-23  8:16   ` Christoph Hellwig
  2021-06-25  8:25   ` Michal Hocko
  1 sibling, 0 replies; 128+ messages in thread
From: Christoph Hellwig @ 2021-06-23  8:16 UTC (permalink / raw)
  To: Matthew Wilcox (Oracle); +Cc: akpm, linux-fsdevel, linux-mm, linux-kernel

On Tue, Jun 22, 2021 at 01:15:20PM +0100, Matthew Wilcox (Oracle) wrote:
> Reimplement mem_cgroup_uncharge() as a wrapper around
> folio_uncharge_cgroup().

Looks good,

Reviewed-by: Christoph Hellwig <hch@lst.de>

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 16/46] mm/memcg: Add folio_migrate_cgroup()
  2021-06-22 12:15 ` [PATCH v2 16/46] mm/memcg: Add folio_migrate_cgroup() Matthew Wilcox (Oracle)
@ 2021-06-23  8:19   ` Christoph Hellwig
  0 siblings, 0 replies; 128+ messages in thread
From: Christoph Hellwig @ 2021-06-23  8:19 UTC (permalink / raw)
  To: Matthew Wilcox (Oracle); +Cc: akpm, linux-fsdevel, linux-mm, linux-kernel

On Tue, Jun 22, 2021 at 01:15:21PM +0100, Matthew Wilcox (Oracle) wrote:
> Convert all callers of mem_cgroup_migrate() to call folio_migrate_cgroup()
> instead.
> 
> Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>

Looks good,

Reviewed-by: Christoph Hellwig <hch@lst.de>

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 17/46] mm/memcg: Convert mem_cgroup_track_foreign_dirty_slowpath() to folio
  2021-06-22 12:15 ` [PATCH v2 17/46] mm/memcg: Convert mem_cgroup_track_foreign_dirty_slowpath() to folio Matthew Wilcox (Oracle)
@ 2021-06-23  8:21   ` Christoph Hellwig
  2021-06-24 17:37     ` Matthew Wilcox
  0 siblings, 1 reply; 128+ messages in thread
From: Christoph Hellwig @ 2021-06-23  8:21 UTC (permalink / raw)
  To: Matthew Wilcox (Oracle); +Cc: akpm, linux-fsdevel, linux-mm, linux-kernel

Looks good,

Reviewed-by: Christoph Hellwig <hch@lst.de>

Although I wish we could come up with a shorter name for
mem_cgroup_track_foreign_dirty_slowpath somehow..

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 18/46] mm/migrate: Add folio_migrate_mapping()
  2021-06-22 12:15 ` [PATCH v2 18/46] mm/migrate: Add folio_migrate_mapping() Matthew Wilcox (Oracle)
@ 2021-06-23  8:22   ` Christoph Hellwig
  0 siblings, 0 replies; 128+ messages in thread
From: Christoph Hellwig @ 2021-06-23  8:22 UTC (permalink / raw)
  To: Matthew Wilcox (Oracle); +Cc: akpm, linux-fsdevel, linux-mm, linux-kernel

On Tue, Jun 22, 2021 at 01:15:23PM +0100, Matthew Wilcox (Oracle) wrote:
> Reimplement migrate_page_move_mapping() as a wrapper around
> folio_migrate_mapping().  Saves 193 bytes of kernel text.

Looks good,

Reviewed-by: Christoph Hellwig <hch@lst.de>

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 19/46] mm/migrate: Add folio_migrate_flags()
  2021-06-22 12:15 ` [PATCH v2 19/46] mm/migrate: Add folio_migrate_flags() Matthew Wilcox (Oracle)
@ 2021-06-23  8:28   ` Christoph Hellwig
  2021-06-24 17:55     ` Matthew Wilcox
  0 siblings, 1 reply; 128+ messages in thread
From: Christoph Hellwig @ 2021-06-23  8:28 UTC (permalink / raw)
  To: Matthew Wilcox (Oracle); +Cc: akpm, linux-fsdevel, linux-mm, linux-kernel

>  	/*
>  	 * Please do not reorder this without considering how mm/ksm.c's
>  	 * get_ksm_page() depends upon ksm_migrate_page() and PageSwapCache().
>  	 */
> -	if (PageSwapCache(page))
> -		ClearPageSwapCache(page);
> -	ClearPagePrivate(page);
> -	set_page_private(page, 0);
> +	if (folio_swapcache(folio))
> +		folio_clear_swapcache_flag(folio);
> +	folio_clear_private_flag(folio);
> +
> +	/* page->private contains hugetlb specific flags */
> +	if (!folio_hugetlb(folio))
> +		folio->private = NULL;

Ymmm. Dosn't the ->private handling change now?  Given that you
added a comment it seems intentional, but I do not understand why
it changes as part of the conversion.

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 20/46] mm/migrate: Add folio_migrate_copy()
  2021-06-22 12:15 ` [PATCH v2 20/46] mm/migrate: Add folio_migrate_copy() Matthew Wilcox (Oracle)
@ 2021-06-23  8:35   ` Christoph Hellwig
  2021-06-24 18:02     ` Matthew Wilcox
  0 siblings, 1 reply; 128+ messages in thread
From: Christoph Hellwig @ 2021-06-23  8:35 UTC (permalink / raw)
  To: Matthew Wilcox (Oracle); +Cc: akpm, linux-fsdevel, linux-mm, linux-kernel

> +void folio_migrate_copy(struct folio *newfolio, struct folio *folio)
>  {
> +	unsigned int i = folio_nr_pages(folio) - 1;
>  
> +	copy_highpage(folio_page(newfolio, i), folio_page(folio, i));
> +	while (i-- > 0) {
> +		cond_resched()a
> +		/* folio_page() handles discontinuities in memmap */
> +		copy_highpage(folio_page(newfolio, i), folio_page(folio, i));
> +	}
> +

What is the advantage of copying backwards here to start with?

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 21/46] mm/writeback: Rename __add_wb_stat() to wb_stat_mod()
  2021-06-22 12:15 ` [PATCH v2 21/46] mm/writeback: Rename __add_wb_stat() to wb_stat_mod() Matthew Wilcox (Oracle)
@ 2021-06-23  8:37   ` Christoph Hellwig
  0 siblings, 0 replies; 128+ messages in thread
From: Christoph Hellwig @ 2021-06-23  8:37 UTC (permalink / raw)
  To: Matthew Wilcox (Oracle); +Cc: akpm, linux-fsdevel, linux-mm, linux-kernel

Looks good,

Reviewed-by: Christoph Hellwig <hch@lst.de>

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 22/46] flex_proportions: Allow N events instead of 1
  2021-06-22 12:15 ` [PATCH v2 22/46] flex_proportions: Allow N events instead of 1 Matthew Wilcox (Oracle)
@ 2021-06-23  8:41   ` Christoph Hellwig
  0 siblings, 0 replies; 128+ messages in thread
From: Christoph Hellwig @ 2021-06-23  8:41 UTC (permalink / raw)
  To: Matthew Wilcox (Oracle); +Cc: akpm, linux-fsdevel, linux-mm, linux-kernel

On Tue, Jun 22, 2021 at 01:15:27PM +0100, Matthew Wilcox (Oracle) wrote:
> When batching events (such as writing back N pages in a single I/O), it
> is better to do one flex_proportion operation instead of N.  There is
> only one caller of __fprop_inc_percpu_max(), and it's the one we're
> going to change in the next patch, so rename it instead of adding a
> compatibility wrapper.
> 
> Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>

Looks good,

Reviewed-by: Christoph Hellwig <hch@lst.de>

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 23/46] mm/writeback: Change __wb_writeout_inc() to __wb_writeout_add()
  2021-06-22 12:15 ` [PATCH v2 23/46] mm/writeback: Change __wb_writeout_inc() to __wb_writeout_add() Matthew Wilcox (Oracle)
@ 2021-06-23  8:45   ` Christoph Hellwig
  0 siblings, 0 replies; 128+ messages in thread
From: Christoph Hellwig @ 2021-06-23  8:45 UTC (permalink / raw)
  To: Matthew Wilcox (Oracle); +Cc: akpm, linux-fsdevel, linux-mm, linux-kernel

On Tue, Jun 22, 2021 at 01:15:28PM +0100, Matthew Wilcox (Oracle) wrote:
> Allow for accounting N pages at once instead of one page at a time.

Looks good,

Reviewed-by: Christoph Hellwig <hch@lst.de>

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 24/46] mm/writeback: Add __folio_end_writeback()
  2021-06-22 12:15 ` [PATCH v2 24/46] mm/writeback: Add __folio_end_writeback() Matthew Wilcox (Oracle)
@ 2021-06-23  9:15   ` Christoph Hellwig
  2021-06-24 18:20     ` Matthew Wilcox
  0 siblings, 1 reply; 128+ messages in thread
From: Christoph Hellwig @ 2021-06-23  9:15 UTC (permalink / raw)
  To: Matthew Wilcox (Oracle); +Cc: akpm, linux-fsdevel, linux-mm, linux-kernel

On Tue, Jun 22, 2021 at 01:15:29PM +0100, Matthew Wilcox (Oracle) wrote:
> test_clear_page_writeback() is actually an mm-internal function, although
> it's named as if it's a pagecache function.  Move it to mm/internal.h,
> rename it to __folio_end_writeback() and change the return type to bool.
> 
> The conversion from page to folio is mostly about accounting the number
> of pages being written back, although it does eliminate a couple of
> calls to compound_head().

While this looks good, I think the whole abstraction is wrong.  I think
test_clear_page_writeback should just be merged into it's only caller.

But if that is somehow not on the table this change looks ok:

Reviewed-by: Christoph Hellwig <hch@lst.de>

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 25/46] mm/writeback: Add folio_start_writeback()
  2021-06-22 12:15 ` [PATCH v2 25/46] mm/writeback: Add folio_start_writeback() Matthew Wilcox (Oracle)
@ 2021-06-23  9:18   ` Christoph Hellwig
  2021-06-24 18:33     ` Matthew Wilcox
  0 siblings, 1 reply; 128+ messages in thread
From: Christoph Hellwig @ 2021-06-23  9:18 UTC (permalink / raw)
  To: Matthew Wilcox (Oracle); +Cc: akpm, linux-fsdevel, linux-mm, linux-kernel

> +static inline void set_page_writeback_keepwrite(struct page *page)
>  {
> +	folio_start_writeback_keepwrite(page_folio(page));
>  }
>  
> +static inline bool test_set_page_writeback(struct page *page)
>  {
> +	return set_page_writeback(page);
>  }

Shouldn't these be in folio-compat.c as well?

Otherwise looks good:

Reviewed-by: Christoph Hellwig <hch@lst.de>

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 26/46] mm/writeback: Add folio_mark_dirty()
  2021-06-22 12:15 ` [PATCH v2 26/46] mm/writeback: Add folio_mark_dirty() Matthew Wilcox (Oracle)
@ 2021-06-23  9:21   ` Christoph Hellwig
  0 siblings, 0 replies; 128+ messages in thread
From: Christoph Hellwig @ 2021-06-23  9:21 UTC (permalink / raw)
  To: Matthew Wilcox (Oracle); +Cc: akpm, linux-fsdevel, linux-mm, linux-kernel

Looks good,

Reviewed-by: Christoph Hellwig <hch@lst.de>

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 27/46] mm/writeback: Add __folio_mark_dirty()
  2021-06-22 12:15 ` [PATCH v2 27/46] mm/writeback: Add __folio_mark_dirty() Matthew Wilcox (Oracle)
@ 2021-06-23  9:27   ` Christoph Hellwig
  2021-06-24 18:37     ` Matthew Wilcox
  0 siblings, 1 reply; 128+ messages in thread
From: Christoph Hellwig @ 2021-06-23  9:27 UTC (permalink / raw)
  To: Matthew Wilcox (Oracle); +Cc: akpm, linux-fsdevel, linux-mm, linux-kernel

On Tue, Jun 22, 2021 at 01:15:32PM +0100, Matthew Wilcox (Oracle) wrote:
> Turn __set_page_dirty() into a wrapper around __folio_mark_dirty() (which
> can directly cast from page to folio because we know that set_page_dirty()
> calls filesystems with the head page).  Convert account_page_dirtied()
> into folio_account_dirtied() and account the number of pages in the folio.

Is it really worth micro-optimizing a transitional function like that?
I'd rather eat the overhead of the compound_page() call over adding hacky
casts like this.

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 28/46] mm/writeback: Add filemap_dirty_folio()
  2021-06-22 12:15 ` [PATCH v2 28/46] mm/writeback: Add filemap_dirty_folio() Matthew Wilcox (Oracle)
@ 2021-06-23  9:32   ` Christoph Hellwig
  0 siblings, 0 replies; 128+ messages in thread
From: Christoph Hellwig @ 2021-06-23  9:32 UTC (permalink / raw)
  To: Matthew Wilcox (Oracle); +Cc: akpm, linux-fsdevel, linux-mm, linux-kernel

On Tue, Jun 22, 2021 at 01:15:33PM +0100, Matthew Wilcox (Oracle) wrote:
> Reimplement __set_page_dirty_nobuffers() as a wrapper around
> filemap_dirty_folio().  This can use a cast to struct folio
> because we know that the ->set_page_dirty address space op
> is always called with a page pointer that happens to also be
> a folio pointer.  Saves 7 bytes of kernel text.

Modulo the cast comment from the last patch this looks fine:

Reviewed-by: Christoph Hellwig <hch@lst.de>

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 29/46] mm/writeback: Add folio_account_cleaned()
  2021-06-22 12:15 ` [PATCH v2 29/46] mm/writeback: Add folio_account_cleaned() Matthew Wilcox (Oracle)
@ 2021-06-23  9:36   ` Christoph Hellwig
  2021-06-24 20:06     ` Matthew Wilcox
  0 siblings, 1 reply; 128+ messages in thread
From: Christoph Hellwig @ 2021-06-23  9:36 UTC (permalink / raw)
  To: Matthew Wilcox (Oracle); +Cc: akpm, linux-fsdevel, linux-mm, linux-kernel

On Tue, Jun 22, 2021 at 01:15:34PM +0100, Matthew Wilcox (Oracle) wrote:
> Get the statistics right; compound pages were being accounted as a
> single page.

Maybe reword this a little to document the existing function that got
it wrong, and why it did not matter before.

Otherwise looks good:

Reviewed-by: Christoph Hellwig <hch@lst.de>

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 30/46] mm/writeback: Add folio_cancel_dirty()
  2021-06-22 12:15 ` [PATCH v2 30/46] mm/writeback: Add folio_cancel_dirty() Matthew Wilcox (Oracle)
@ 2021-06-23  9:39   ` Christoph Hellwig
  0 siblings, 0 replies; 128+ messages in thread
From: Christoph Hellwig @ 2021-06-23  9:39 UTC (permalink / raw)
  To: Matthew Wilcox (Oracle); +Cc: akpm, linux-fsdevel, linux-mm, linux-kernel

Looks good,

Reviewed-by: Christoph Hellwig <hch@lst.de>

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 31/46] mm/writeback: Add folio_clear_dirty_for_io()
  2021-06-22 12:15 ` [PATCH v2 31/46] mm/writeback: Add folio_clear_dirty_for_io() Matthew Wilcox (Oracle)
@ 2021-06-23  9:43   ` Christoph Hellwig
  2021-06-24 20:09     ` Matthew Wilcox
  0 siblings, 1 reply; 128+ messages in thread
From: Christoph Hellwig @ 2021-06-23  9:43 UTC (permalink / raw)
  To: Matthew Wilcox (Oracle); +Cc: akpm, linux-fsdevel, linux-mm, linux-kernel

On Tue, Jun 22, 2021 at 01:15:36PM +0100, Matthew Wilcox (Oracle) wrote:
> Transform clear_page_dirty_for_io() into folio_clear_dirty_for_io()
> and add a compatibility wrapper.  Also move the declaration to pagemap.h
> as this is page cache functionality that doesn't need to be used by the
> rest of the kernel.
> 
> Increases the size of the kernel by 79 bytes.  While we remove a few
> calls to compound_head(), we add a call to folio_nr_pages() to get the
> stats correct.

... for the eventual support of multi-page folios.

Otherwise looks good:

Reviewed-by: Christoph Hellwig <hch@lst.de>

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 32/46] mm/writeback: Add folio_account_redirty()
  2021-06-22 12:15 ` [PATCH v2 32/46] mm/writeback: Add folio_account_redirty() Matthew Wilcox (Oracle)
@ 2021-06-23  9:44   ` Christoph Hellwig
  0 siblings, 0 replies; 128+ messages in thread
From: Christoph Hellwig @ 2021-06-23  9:44 UTC (permalink / raw)
  To: Matthew Wilcox (Oracle); +Cc: akpm, linux-fsdevel, linux-mm, linux-kernel

On Tue, Jun 22, 2021 at 01:15:37PM +0100, Matthew Wilcox (Oracle) wrote:
> Account the number of pages in the folio that we're redirtying.
> Turn account_page_dirty() into a wrapper around it.  Also turn
> the comment on folio_account_redirty() into kernel-doc and
> edit it slightly so it makes sense to its potential callers.
> 
> Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>

Looks good,

Reviewed-by: Christoph Hellwig <hch@lst.de>

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 33/46] mm/writeback: Add folio_redirty_for_writepage()
  2021-06-22 12:15 ` [PATCH v2 33/46] mm/writeback: Add folio_redirty_for_writepage() Matthew Wilcox (Oracle)
@ 2021-06-23  9:45   ` Christoph Hellwig
  0 siblings, 0 replies; 128+ messages in thread
From: Christoph Hellwig @ 2021-06-23  9:45 UTC (permalink / raw)
  To: Matthew Wilcox (Oracle); +Cc: akpm, linux-fsdevel, linux-mm, linux-kernel

On Tue, Jun 22, 2021 at 01:15:38PM +0100, Matthew Wilcox (Oracle) wrote:
> Reimplement redirty_page_for_writepage() as a wrapper around
> folio_redirty_for_writepage().  Account the number of pages in the
> folio, add kernel-doc and move the prototype to writeback.h.
> 
> Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>

Looks good,

Reviewed-by: Christoph Hellwig <hch@lst.de>

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 34/46] mm/filemap: Add i_blocks_per_folio()
  2021-06-22 12:15 ` [PATCH v2 34/46] mm/filemap: Add i_blocks_per_folio() Matthew Wilcox (Oracle)
@ 2021-06-23  9:46   ` Christoph Hellwig
  0 siblings, 0 replies; 128+ messages in thread
From: Christoph Hellwig @ 2021-06-23  9:46 UTC (permalink / raw)
  To: Matthew Wilcox (Oracle); +Cc: akpm, linux-fsdevel, linux-mm, linux-kernel

On Tue, Jun 22, 2021 at 01:15:39PM +0100, Matthew Wilcox (Oracle) wrote:
> Reimplement i_blocks_per_page() as a wrapper around i_blocks_per_folio().
> 
> Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
> ---
>  include/linux/pagemap.h | 18 ++++++++++++------
>  1 file changed, 12 insertions(+), 6 deletions(-)
> 
> diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
> index 31edfa891987..c30db827b65d 100644
> --- a/include/linux/pagemap.h
> +++ b/include/linux/pagemap.h
> @@ -1149,19 +1149,25 @@ static inline int page_mkwrite_check_truncate(struct page *page,
>  }
>  
>  /**
> - * i_blocks_per_page - How many blocks fit in this page.
> + * i_blocks_per_folio - How many blocks fit in this folio.
>   * @inode: The inode which contains the blocks.
> - * @page: The page (head page if the page is a THP).
> + * @folio: The folio.
>   *
> - * If the block size is larger than the size of this page, return zero.
> + * If the block size is larger than the size of this folio, return zero.
>   *
> - * Context: The caller should hold a refcount on the page to prevent it
> + * Context: The caller should hold a refcount on the folio to prevent it
>   * from being split.
> - * Return: The number of filesystem blocks covered by this page.
> + * Return: The number of filesystem blocks covered by this folio.
>   */
> +static inline
> +unsigned int i_blocks_per_folio(struct inode *inode, struct folio *folio)

Weirdo formatting (same as i_blocks_per_page, but I'd still rather avoid
it).

Otherwise looks good:

Reviewed-by: Christoph Hellwig <hch@lst.de>

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 35/46] mm/filemap: Add folio_mkwrite_check_truncate()
  2021-06-22 12:15 ` [PATCH v2 35/46] mm/filemap: Add folio_mkwrite_check_truncate() Matthew Wilcox (Oracle)
@ 2021-06-23  9:47   ` Christoph Hellwig
  2021-06-23 13:19     ` Matthew Wilcox
  0 siblings, 1 reply; 128+ messages in thread
From: Christoph Hellwig @ 2021-06-23  9:47 UTC (permalink / raw)
  To: Matthew Wilcox (Oracle); +Cc: akpm, linux-fsdevel, linux-mm, linux-kernel

On Tue, Jun 22, 2021 at 01:15:40PM +0100, Matthew Wilcox (Oracle) wrote:
> This is the folio equivalent of page_mkwrite_check_truncate().
> 
> Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>

Any reason that page_mkwrite_check_truncate isn't turned into a wrapper?

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 36/46] mm/filemap: Add readahead_folio()
  2021-06-22 12:15 ` [PATCH v2 36/46] mm/filemap: Add readahead_folio() Matthew Wilcox (Oracle)
@ 2021-06-23  9:50   ` Christoph Hellwig
  2021-06-24 23:46     ` Matthew Wilcox
  0 siblings, 1 reply; 128+ messages in thread
From: Christoph Hellwig @ 2021-06-23  9:50 UTC (permalink / raw)
  To: Matthew Wilcox (Oracle); +Cc: akpm, linux-fsdevel, linux-mm, linux-kernel

On Tue, Jun 22, 2021 at 01:15:41PM +0100, Matthew Wilcox (Oracle) wrote:
> The pointers stored in the page cache are folios, by definition.
> This change comes with a behaviour change -- callers of readahead_folio()
> are no longer required to put the page reference themselves.  This matches
> how readpage works, rather than matching how readpages used to work.

The way this stores and retrieves different but compatible types from the
same xarray is a little nasty.  But I guess we'll have to live with it for
now, so:

Reviewed-by: Christoph Hellwig <hch@lst.de>

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 37/46] mm/workingset: Convert workingset_refault() to take a folio
  2021-06-22 12:15 ` [PATCH v2 37/46] mm/workingset: Convert workingset_refault() to take a folio Matthew Wilcox (Oracle)
@ 2021-06-23  9:52   ` Christoph Hellwig
  0 siblings, 0 replies; 128+ messages in thread
From: Christoph Hellwig @ 2021-06-23  9:52 UTC (permalink / raw)
  To: Matthew Wilcox (Oracle); +Cc: akpm, linux-fsdevel, linux-mm, linux-kernel

Looks good,

Reviewed-by: Christoph Hellwig <hch@lst.de>

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 38/46] mm: Add folio_evictable()
  2021-06-22 12:15 ` [PATCH v2 38/46] mm: Add folio_evictable() Matthew Wilcox (Oracle)
@ 2021-06-23  9:54   ` Christoph Hellwig
  2021-06-24 23:50     ` Matthew Wilcox
  0 siblings, 1 reply; 128+ messages in thread
From: Christoph Hellwig @ 2021-06-23  9:54 UTC (permalink / raw)
  To: Matthew Wilcox (Oracle); +Cc: akpm, linux-fsdevel, linux-mm, linux-kernel

> + * Reasons folio might not be evictable:
> + * 1. page's mapping marked unevictable
> + * 2. page is part of an mlocked VMA

s/page/folio/?

Otherwise looks good:

Reviewed-by: Christoph Hellwig <hch@lst.de>

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 39/46] mm/lru: Convert __pagevec_lru_add_fn to take a folio
  2021-06-22 12:15 ` [PATCH v2 39/46] mm/lru: Convert __pagevec_lru_add_fn to take a folio Matthew Wilcox (Oracle)
@ 2021-06-23  9:55   ` Christoph Hellwig
  0 siblings, 0 replies; 128+ messages in thread
From: Christoph Hellwig @ 2021-06-23  9:55 UTC (permalink / raw)
  To: Matthew Wilcox (Oracle); +Cc: akpm, linux-fsdevel, linux-mm, linux-kernel

On Tue, Jun 22, 2021 at 01:15:44PM +0100, Matthew Wilcox (Oracle) wrote:
> This saves five calls to compound_head(), totalling 60 bytes of text.
> 
> Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>

Looks good,

Reviewed-by: Christoph Hellwig <hch@lst.de>

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 40/46] mm/lru: Add folio_add_lru()
  2021-06-22 12:15 ` [PATCH v2 40/46] mm/lru: Add folio_add_lru() Matthew Wilcox (Oracle)
@ 2021-06-23  9:56   ` Christoph Hellwig
  0 siblings, 0 replies; 128+ messages in thread
From: Christoph Hellwig @ 2021-06-23  9:56 UTC (permalink / raw)
  To: Matthew Wilcox (Oracle); +Cc: akpm, linux-fsdevel, linux-mm, linux-kernel

On Tue, Jun 22, 2021 at 01:15:45PM +0100, Matthew Wilcox (Oracle) wrote:
> Reimplement lru_cache_add() as a wrapper around folio_add_lru().
> Saves 159 bytes of kernel text due to removing calls to compound_head().
> 
> Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>

Looks good,

Reviewed-by: Christoph Hellwig <hch@lst.de>

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 41/46] mm/page_alloc: Add folio allocation functions
  2021-06-22 12:15 ` [PATCH v2 41/46] mm/page_alloc: Add folio allocation functions Matthew Wilcox (Oracle)
@ 2021-06-23  9:58   ` Christoph Hellwig
  2021-06-25  1:06     ` Matthew Wilcox
  0 siblings, 1 reply; 128+ messages in thread
From: Christoph Hellwig @ 2021-06-23  9:58 UTC (permalink / raw)
  To: Matthew Wilcox (Oracle); +Cc: akpm, linux-fsdevel, linux-mm, linux-kernel

On Tue, Jun 22, 2021 at 01:15:46PM +0100, Matthew Wilcox (Oracle) wrote:
> +static inline
> +struct folio *__alloc_folio_node(gfp_t gfp, unsigned int order, int nid)

Weirdo prototype formatting.

Otherwise looks good (assuming we grow callers):

Reviewed-by: Christoph Hellwig <hch@lst.de>

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 42/46] mm/filemap: Add filemap_alloc_folio
  2021-06-22 12:15 ` [PATCH v2 42/46] mm/filemap: Add filemap_alloc_folio Matthew Wilcox (Oracle)
@ 2021-06-23  9:59   ` Christoph Hellwig
  0 siblings, 0 replies; 128+ messages in thread
From: Christoph Hellwig @ 2021-06-23  9:59 UTC (permalink / raw)
  To: Matthew Wilcox (Oracle); +Cc: akpm, linux-fsdevel, linux-mm, linux-kernel

On Tue, Jun 22, 2021 at 01:15:47PM +0100, Matthew Wilcox (Oracle) wrote:
> Reimplement __page_cache_alloc as a wrapper around filemap_alloc_folio
> to allow filesystems to be converted at our leisure.  Increases
> kernel text size by 133 bytes, mostly in cachefiles_read_backing_file().
> pagecache_get_page() shrinks by 32 bytes, though.
> 
> Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>

Looks good,

Reviewed-by: Christoph Hellwig <hch@lst.de>

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 43/46] mm/filemap: Add filemap_add_folio
  2021-06-22 12:15 ` [PATCH v2 43/46] mm/filemap: Add filemap_add_folio Matthew Wilcox (Oracle)
@ 2021-06-23 11:30   ` Christoph Hellwig
  2021-06-25  1:57     ` Matthew Wilcox
  0 siblings, 1 reply; 128+ messages in thread
From: Christoph Hellwig @ 2021-06-23 11:30 UTC (permalink / raw)
  To: Matthew Wilcox (Oracle); +Cc: akpm, linux-fsdevel, linux-mm, linux-kernel

On Tue, Jun 22, 2021 at 01:15:48PM +0100, Matthew Wilcox (Oracle) wrote:
> Pages being added to the page cache should already be folios, so
> just cast the page to a folio in the add_to_page_cache_lru() wrapper.
> Saves 96 bytes of text.

modulo the casting:

Reviewed-by: Christoph Hellwig <hch@lst.de>

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 44/46] mm/filemap: Convert mapping_get_entry to return a folio
  2021-06-22 12:15 ` [PATCH v2 44/46] mm/filemap: Convert mapping_get_entry to return a folio Matthew Wilcox (Oracle)
@ 2021-06-23 11:32   ` Christoph Hellwig
  2021-06-25  3:29     ` Matthew Wilcox
  0 siblings, 1 reply; 128+ messages in thread
From: Christoph Hellwig @ 2021-06-23 11:32 UTC (permalink / raw)
  To: Matthew Wilcox (Oracle); +Cc: akpm, linux-fsdevel, linux-mm, linux-kernel

On Tue, Jun 22, 2021 at 01:15:49PM +0100, Matthew Wilcox (Oracle) wrote:
> - * Return: The head page or shadow entry, %NULL if nothing is found.
> + * Return: The folio, swap or shadow entry, %NULL if nothing is found.

This (old and new) reads a little weird, given that it returns a
struct folio, even if that happens to be a magic entry.

Otherwise looks good:

Reviewed-by: Christoph Hellwig <hch@lst.de>

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 45/46] mm/filemap: Add filemap_get_folio
  2021-06-22 12:15 ` [PATCH v2 45/46] mm/filemap: Add filemap_get_folio Matthew Wilcox (Oracle)
@ 2021-06-23 11:39   ` Christoph Hellwig
  0 siblings, 0 replies; 128+ messages in thread
From: Christoph Hellwig @ 2021-06-23 11:39 UTC (permalink / raw)
  To: Matthew Wilcox (Oracle); +Cc: akpm, linux-fsdevel, linux-mm, linux-kernel

>  /**
> - * pagecache_get_page - Find and get a reference to a page.
> + * __filemap_get_folio - Find and get a reference to a folio.

Maybe filemap_get_folio_flags is a better name? 

Otherwise looks good:

Reviewed-by: Christoph Hellwig <hch@lst.de>

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 46/46] mm/filemap: Add FGP_STABLE
  2021-06-22 12:15 ` [PATCH v2 46/46] mm/filemap: Add FGP_STABLE Matthew Wilcox (Oracle)
@ 2021-06-23 11:43   ` Christoph Hellwig
  2021-06-23 12:15     ` Matthew Wilcox
  0 siblings, 1 reply; 128+ messages in thread
From: Christoph Hellwig @ 2021-06-23 11:43 UTC (permalink / raw)
  To: Matthew Wilcox (Oracle); +Cc: akpm, linux-fsdevel, linux-mm, linux-kernel

On Tue, Jun 22, 2021 at 01:15:51PM +0100, Matthew Wilcox (Oracle) wrote:
> Allow filemap_get_folio() to wait for writeback to complete (if the
> filesystem wants that behaviour).  This is the folio equivalent of
> grab_cache_page_write_begin(), which is moved into the folio-compat
> file as a reminder to migrate all the code using it.  This paves the
> way for getting rid of AOP_FLAG_NOFS once grab_cache_page_write_begin()
> is removed.

We actually should kill FGP_NOFS as well by switching everything over
to memalloc_nofs_{save, restore} eventually, given how error prone
all these manual flags settings are.

> diff --git a/mm/folio-compat.c b/mm/folio-compat.c
> index 78365eaee7d3..206bedd621d0 100644
> --- a/mm/folio-compat.c
> +++ b/mm/folio-compat.c
> @@ -115,6 +115,7 @@ void lru_cache_add(struct page *page)
>  }
>  EXPORT_SYMBOL(lru_cache_add);
>  
> +noinline
>  struct page *pagecache_get_page(struct address_space *mapping, pgoff_t index,
>  		int fgp_flags, gfp_t gfp)

How did that sneak in here?

Otherwise:

Reviewed-by: Christoph Hellwig <hch@lst.de>


^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 46/46] mm/filemap: Add FGP_STABLE
  2021-06-23 11:43   ` Christoph Hellwig
@ 2021-06-23 12:15     ` Matthew Wilcox
  0 siblings, 0 replies; 128+ messages in thread
From: Matthew Wilcox @ 2021-06-23 12:15 UTC (permalink / raw)
  To: Christoph Hellwig; +Cc: akpm, linux-fsdevel, linux-mm, linux-kernel

On Wed, Jun 23, 2021 at 01:43:30PM +0200, Christoph Hellwig wrote:
> On Tue, Jun 22, 2021 at 01:15:51PM +0100, Matthew Wilcox (Oracle) wrote:
> > Allow filemap_get_folio() to wait for writeback to complete (if the
> > filesystem wants that behaviour).  This is the folio equivalent of
> > grab_cache_page_write_begin(), which is moved into the folio-compat
> > file as a reminder to migrate all the code using it.  This paves the
> > way for getting rid of AOP_FLAG_NOFS once grab_cache_page_write_begin()
> > is removed.
> 
> We actually should kill FGP_NOFS as well by switching everything over
> to memalloc_nofs_{save, restore} eventually, given how error prone
> all these manual flags settings are.

Well, yes, but it's been four years and we still have over 1100 uses of
GFP_NOFS.  Until someone takes on that Augean Stables, we're going to need
FGP_NOFS.  I added that context to the readahead path in f2c817bed58d,
but of course that doesn't let me remove any uses of GFP_NOFS.

> > diff --git a/mm/folio-compat.c b/mm/folio-compat.c
> > index 78365eaee7d3..206bedd621d0 100644
> > --- a/mm/folio-compat.c
> > +++ b/mm/folio-compat.c
> > @@ -115,6 +115,7 @@ void lru_cache_add(struct page *page)
> >  }
> >  EXPORT_SYMBOL(lru_cache_add);
> >  
> > +noinline
> >  struct page *pagecache_get_page(struct address_space *mapping, pgoff_t index,
> >  		int fgp_flags, gfp_t gfp)
> 
> How did that sneak in here?

Without it, pagecache_get_page() gets inlined by
grab_cache_page_write_begin() which is just too much code.

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 35/46] mm/filemap: Add folio_mkwrite_check_truncate()
  2021-06-23  9:47   ` Christoph Hellwig
@ 2021-06-23 13:19     ` Matthew Wilcox
  0 siblings, 0 replies; 128+ messages in thread
From: Matthew Wilcox @ 2021-06-23 13:19 UTC (permalink / raw)
  To: Christoph Hellwig; +Cc: akpm, linux-fsdevel, linux-mm, linux-kernel

On Wed, Jun 23, 2021 at 11:47:58AM +0200, Christoph Hellwig wrote:
> On Tue, Jun 22, 2021 at 01:15:40PM +0100, Matthew Wilcox (Oracle) wrote:
> > This is the folio equivalent of page_mkwrite_check_truncate().
> > 
> > Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
> 
> Any reason that page_mkwrite_check_truncate isn't turned into a wrapper?

It'd introduce an extra call to page_folio() for no actual benefit

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 01/46] mm: Add folio_to_pfn()
  2021-06-23  7:49   ` Christoph Hellwig
@ 2021-06-24 15:12     ` Matthew Wilcox
  2021-06-28  6:18       ` Christoph Hellwig
  0 siblings, 1 reply; 128+ messages in thread
From: Matthew Wilcox @ 2021-06-24 15:12 UTC (permalink / raw)
  To: Christoph Hellwig; +Cc: akpm, linux-fsdevel, linux-mm, linux-kernel

On Wed, Jun 23, 2021 at 08:49:55AM +0100, Christoph Hellwig wrote:
> On Tue, Jun 22, 2021 at 01:15:06PM +0100, Matthew Wilcox (Oracle) wrote:
> > The pfn of a folio is the pfn of its head page.
> > 
> > Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
> 
> Maybe add a kerneldoc comment stating that?

/**
 * folio_to_pfn - Return the Page Frame Number of a folio.
 * @folio: The folio.
 *
 * A folio may contain multiple pages.  The pages have consecutive
 * Page Frame Numbers.
 *
 * Return: The Page Frame Number of the first page in the folio.
 */


^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 02/46] mm: Add folio_rmapping()
  2021-06-23  7:56   ` Christoph Hellwig
@ 2021-06-24 15:51     ` Matthew Wilcox
  0 siblings, 0 replies; 128+ messages in thread
From: Matthew Wilcox @ 2021-06-24 15:51 UTC (permalink / raw)
  To: Christoph Hellwig; +Cc: akpm, linux-fsdevel, linux-mm, linux-kernel

On Wed, Jun 23, 2021 at 09:56:58AM +0200, Christoph Hellwig wrote:
> > +static inline void *folio_rmapping(struct folio *folio)
> 
> This name, just like the old one is not exaclty descriptive.  I guess the
> r stands for raw somehow?  As a casual contributor to the fringes of the
> MM I would have no idea when to use it.
> 
> All this of course also applies to the existing (__)page_rmapping, but
> maybe this is a good time to sort it out.

Yes, good point.  I don't like the name rmapping either, since
we already have rmap which has nothing to do with this.  I'll
leave page_rmapping() alone for now; no need to add that churn.
I think they all become calls to folio_raw_mapping() later.

> >  
> >  struct anon_vma *page_anon_vma(struct page *page)
> >  {
> > +	struct folio *folio = page_folio(page);
> > +	unsigned long mapping = (unsigned long)folio->mapping;
> >  
> >  	if ((mapping & PAGE_MAPPING_FLAGS) != PAGE_MAPPING_ANON)
> >  		return NULL;
> > +	return folio_rmapping(folio);
> 
> It feelds kinda silly to not just open code folio_rmapping here
> given that we alredy went half the way. 

Yeah, I thought about that too.  Done:

-       return folio_rmapping(folio);
+       return (void *)(mapping - PAGE_MAPPING_ANON);


^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 05/46] mm: Add arch_make_folio_accessible()
  2021-06-23  8:00   ` Christoph Hellwig
@ 2021-06-24 15:57     ` Matthew Wilcox
  2021-06-28  6:21       ` Christoph Hellwig
  0 siblings, 1 reply; 128+ messages in thread
From: Matthew Wilcox @ 2021-06-24 15:57 UTC (permalink / raw)
  To: Christoph Hellwig; +Cc: akpm, linux-fsdevel, linux-mm, linux-kernel

On Wed, Jun 23, 2021 at 10:00:37AM +0200, Christoph Hellwig wrote:
> On Tue, Jun 22, 2021 at 01:15:10PM +0100, Matthew Wilcox (Oracle) wrote:
> > As a default implementation, call arch_make_page_accessible n times.
> > If an architecture can do better, it can override this.
> > 
> > Also move the default implementation of arch_make_page_accessible()
> > from gfp.h to mm.h.
> 
> Can we wait with introducing arch hooks until we have an actual user
> lined up?

This one gets used in __folio_end_writeback() which is patch 24 in this
series.

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 12/46] mm/memcg: Use the node id in mem_cgroup_update_tree()
  2021-06-23  8:12   ` Christoph Hellwig
@ 2021-06-24 16:19     ` Matthew Wilcox
  2021-06-25  8:05       ` Michal Hocko
  0 siblings, 1 reply; 128+ messages in thread
From: Matthew Wilcox @ 2021-06-24 16:19 UTC (permalink / raw)
  To: Christoph Hellwig
  Cc: akpm, linux-fsdevel, linux-mm, linux-kernel, Johannes Weiner,
	Michal Hocko, Vladimir Davydov, cgroups

On Wed, Jun 23, 2021 at 10:12:40AM +0200, Christoph Hellwig wrote:
> On Tue, Jun 22, 2021 at 01:15:17PM +0100, Matthew Wilcox (Oracle) wrote:
> >  static struct mem_cgroup_per_node *
> > -mem_cgroup_page_nodeinfo(struct mem_cgroup *memcg, struct page *page)
> > +mem_cgroup_nodeinfo(struct mem_cgroup *memcg, int nid)
> >  {
> > -	int nid = page_to_nid(page);
> > -
> >  	return memcg->nodeinfo[nid];
> >  }
> 
> I'd just kill this function entirely and open code it into the only
> caller

Done.

> > -	mctz = soft_limit_tree_from_page(page);
> > +	mctz = soft_limit_tree_node(nid);
> 
> And while were at it, soft_limit_tree_node seems like a completely
> pointless helper that does nothing but obsfucating the code.  While
> you touch this area it might be worth to spin another patch to just
> remove it as well.

I'm scared that if I touch this file too much, people will start to
think I know something about memcgs.  Happy to add it on; cc'ing
maintainers.

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 14/46] mm/memcg: Add folio_charge_cgroup()
  2021-06-23  8:15   ` Christoph Hellwig
@ 2021-06-24 16:42     ` Matthew Wilcox
  2021-06-25  8:22       ` Michal Hocko
  0 siblings, 1 reply; 128+ messages in thread
From: Matthew Wilcox @ 2021-06-24 16:42 UTC (permalink / raw)
  To: Christoph Hellwig; +Cc: akpm, linux-fsdevel, linux-mm, linux-kernel

On Wed, Jun 23, 2021 at 10:15:20AM +0200, Christoph Hellwig wrote:
> On Tue, Jun 22, 2021 at 01:15:19PM +0100, Matthew Wilcox (Oracle) wrote:
> > mem_cgroup_charge() already assumed it was being passed a non-tail
> > page (and looking at the callers, that's true; it's called for freshly
> > allocated pages).  The only real change here is that folio_nr_pages()
> > doesn't compile away like thp_nr_pages() does as folio support
> > is not conditional on transparent hugepage support.  Reimplement
> > mem_cgroup_charge() as a wrapper around folio_charge_cgroup().
> 
> Maybe rename __mem_cgroup_charge to __folio_charge_cgroup as well?

Oh, yeah, should have done that.  Thanks.

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 17/46] mm/memcg: Convert mem_cgroup_track_foreign_dirty_slowpath() to folio
  2021-06-23  8:21   ` Christoph Hellwig
@ 2021-06-24 17:37     ` Matthew Wilcox
  2021-06-28  6:24       ` Christoph Hellwig
  0 siblings, 1 reply; 128+ messages in thread
From: Matthew Wilcox @ 2021-06-24 17:37 UTC (permalink / raw)
  To: Christoph Hellwig; +Cc: akpm, linux-fsdevel, linux-mm, linux-kernel

On Wed, Jun 23, 2021 at 10:21:26AM +0200, Christoph Hellwig wrote:
> Looks good,
> 
> Reviewed-by: Christoph Hellwig <hch@lst.de>
> 
> Although I wish we could come up with a shorter name for
> mem_cgroup_track_foreign_dirty_slowpath somehow..

It is quite grotesque!

How about folio_track_foreign_writeback() as a replacement name for
mem_cgroup_track_foreign_dirty() and have it call
__folio_track_foreign_writeback()?

Although 'foreign' tends to be used in MM to mean "wrong NUMA node",
so maybe that's misleading.  folio_track_dirty_cgroup()?
folio_mark_dirty_cgroup()?  (the last to be read in context of
__set_page_dirty() being renamed to __folio_mark_dirty())

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 19/46] mm/migrate: Add folio_migrate_flags()
  2021-06-23  8:28   ` Christoph Hellwig
@ 2021-06-24 17:55     ` Matthew Wilcox
  0 siblings, 0 replies; 128+ messages in thread
From: Matthew Wilcox @ 2021-06-24 17:55 UTC (permalink / raw)
  To: Christoph Hellwig; +Cc: akpm, linux-fsdevel, linux-mm, linux-kernel

On Wed, Jun 23, 2021 at 10:28:52AM +0200, Christoph Hellwig wrote:
> >  	/*
> >  	 * Please do not reorder this without considering how mm/ksm.c's
> >  	 * get_ksm_page() depends upon ksm_migrate_page() and PageSwapCache().
> >  	 */
> > -	if (PageSwapCache(page))
> > -		ClearPageSwapCache(page);
> > -	ClearPagePrivate(page);
> > -	set_page_private(page, 0);
> > +	if (folio_swapcache(folio))
> > +		folio_clear_swapcache_flag(folio);
> > +	folio_clear_private_flag(folio);
> > +
> > +	/* page->private contains hugetlb specific flags */
> > +	if (!folio_hugetlb(folio))
> > +		folio->private = NULL;
> 
> Ymmm. Dosn't the ->private handling change now?  Given that you
> added a comment it seems intentional, but I do not understand why
> it changes as part of the conversion.

Oooh.  I was based on linux-next, and Linus asked me to stop doing that.
So I rebased on -rc4 (ish), but I inadvertently brought back some of the
changes which are currently in mmotm.  This one is from:

    mm: hugetlb: alloc the vmemmap pages associated with each HugeTLB page

which contains:

-       set_page_private(page, 0);
+
+       /* page->private contains hugetlb specific flags */
+       if (!PageHuge(page))
+               set_page_private(page, 0);

So, good catch, glad you're reviewing it so closely.  I'll fix this up
as part of rebasing this patch set on top of 14-rc1.  Obviously this
second patch set isn't for merging during this merge window, but I do
hope it can go into 5.15's merge window.

The only API change I intentionally brought back was:

    mm: memcontrol: remove the pgdata parameter of mem_cgroup_page_lruvec

You'll notice that my patches have:

+static inline struct lruvec *mem_cgroup_folio_lruvec(struct folio *folio)
+{
+       return mem_cgroup_page_lruvec(&folio->page, folio_pgdat(folio));
+}

where mmotm has:

-static inline struct lruvec *mem_cgroup_page_lruvec(struct page *page,
-                                               struct pglist_data *pgdat)
+static inline struct lruvec *mem_cgroup_page_lruvec(struct page *page)
 {
+       pg_data_t *pgdat = page_pgdat(page);


Probably I should have cherry-picked those prereq patches into my tree
to ease the rebasing (like I did our changes to set_page_dirty), but
I'm not 100% confident I'll get all their prereq patches.

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 20/46] mm/migrate: Add folio_migrate_copy()
  2021-06-23  8:35   ` Christoph Hellwig
@ 2021-06-24 18:02     ` Matthew Wilcox
  2021-06-28  6:26       ` Christoph Hellwig
  0 siblings, 1 reply; 128+ messages in thread
From: Matthew Wilcox @ 2021-06-24 18:02 UTC (permalink / raw)
  To: Christoph Hellwig; +Cc: akpm, linux-fsdevel, linux-mm, linux-kernel

On Wed, Jun 23, 2021 at 10:35:00AM +0200, Christoph Hellwig wrote:
> > +void folio_migrate_copy(struct folio *newfolio, struct folio *folio)
> >  {
> > +	unsigned int i = folio_nr_pages(folio) - 1;
> >  
> > +	copy_highpage(folio_page(newfolio, i), folio_page(folio, i));
> > +	while (i-- > 0) {
> > +		cond_resched()a
> > +		/* folio_page() handles discontinuities in memmap */
> > +		copy_highpage(folio_page(newfolio, i), folio_page(folio, i));
> > +	}
> > +
> 
> What is the advantage of copying backwards here to start with?

Easier to write the loop this way?  I suppose we could do it as ...

	unsigned int i, nr = folio_nr_pages(folio);

	for (i = 0; i < nr; i++) {
		/* folio_page() handles discontinuities in memmap */
		copy_highpage(folio_page(newfolio, i), folio_page(folio, i));
		cond_resched();
	}

I'm not really bothered.  As long as we don't call folio_nr_pages() for
each iteration of the loop ... I've actually been wondering about
marking that as __pure, but I don't quite have the nerve to do it yet.

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 24/46] mm/writeback: Add __folio_end_writeback()
  2021-06-23  9:15   ` Christoph Hellwig
@ 2021-06-24 18:20     ` Matthew Wilcox
  2021-06-28  6:33       ` Christoph Hellwig
  0 siblings, 1 reply; 128+ messages in thread
From: Matthew Wilcox @ 2021-06-24 18:20 UTC (permalink / raw)
  To: Christoph Hellwig; +Cc: akpm, linux-fsdevel, linux-mm, linux-kernel

On Wed, Jun 23, 2021 at 11:15:55AM +0200, Christoph Hellwig wrote:
> On Tue, Jun 22, 2021 at 01:15:29PM +0100, Matthew Wilcox (Oracle) wrote:
> > test_clear_page_writeback() is actually an mm-internal function, although
> > it's named as if it's a pagecache function.  Move it to mm/internal.h,
> > rename it to __folio_end_writeback() and change the return type to bool.
> > 
> > The conversion from page to folio is mostly about accounting the number
> > of pages being written back, although it does eliminate a couple of
> > calls to compound_head().
> 
> While this looks good, I think the whole abstraction is wrong.  I think
> test_clear_page_writeback should just be merged into it's only caller.

I'm not opposed to doing that, but something else has to get
un-static'ed in order to make that happen.

folio_end_writeback (exported, filemap.c)
 -> folio_wake (static, filemap.c)
     -> folio_wake_bit (static, filemap.c)
 -> __folio_end_writeback (non-static, page-writeback.c)
     -> __wb_writeout_add (static, page-writeback.c)

I'm not sure there's an obviously better split than where it is right
now.

> But if that is somehow not on the table this change looks ok:
> 
> Reviewed-by: Christoph Hellwig <hch@lst.de>

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 25/46] mm/writeback: Add folio_start_writeback()
  2021-06-23  9:18   ` Christoph Hellwig
@ 2021-06-24 18:33     ` Matthew Wilcox
  0 siblings, 0 replies; 128+ messages in thread
From: Matthew Wilcox @ 2021-06-24 18:33 UTC (permalink / raw)
  To: Christoph Hellwig; +Cc: akpm, linux-fsdevel, linux-mm, linux-kernel

On Wed, Jun 23, 2021 at 11:18:04AM +0200, Christoph Hellwig wrote:
> > +static inline void set_page_writeback_keepwrite(struct page *page)
> >  {
> > +	folio_start_writeback_keepwrite(page_folio(page));
> >  }
> >  
> > +static inline bool test_set_page_writeback(struct page *page)
> >  {
> > +	return set_page_writeback(page);
> >  }
> 
> Shouldn't these be in folio-compat.c as well?

Thought about it.  We only have one caller of
set_page_writeback_keepwrite(), so it may as well get inlined there.
And test_set_page_writeback() is just a renaming ... I'd rather
replace the callers of test_set_page_writeback() with calls to
the new set_page_writeback() than move it into folio-compat.

> Otherwise looks good:
> 
> Reviewed-by: Christoph Hellwig <hch@lst.de>

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 27/46] mm/writeback: Add __folio_mark_dirty()
  2021-06-23  9:27   ` Christoph Hellwig
@ 2021-06-24 18:37     ` Matthew Wilcox
  2021-06-28  6:03       ` Christoph Hellwig
  0 siblings, 1 reply; 128+ messages in thread
From: Matthew Wilcox @ 2021-06-24 18:37 UTC (permalink / raw)
  To: Christoph Hellwig; +Cc: akpm, linux-fsdevel, linux-mm, linux-kernel

On Wed, Jun 23, 2021 at 11:27:12AM +0200, Christoph Hellwig wrote:
> On Tue, Jun 22, 2021 at 01:15:32PM +0100, Matthew Wilcox (Oracle) wrote:
> > Turn __set_page_dirty() into a wrapper around __folio_mark_dirty() (which
> > can directly cast from page to folio because we know that set_page_dirty()
> > calls filesystems with the head page).  Convert account_page_dirtied()
> > into folio_account_dirtied() and account the number of pages in the folio.
> 
> Is it really worth micro-optimizing a transitional function like that?
> I'd rather eat the overhead of the compound_page() call over adding hacky
> casts like this.

Fair enough.  There's only three calls to it and one of them goes away
this series.

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 29/46] mm/writeback: Add folio_account_cleaned()
  2021-06-23  9:36   ` Christoph Hellwig
@ 2021-06-24 20:06     ` Matthew Wilcox
  0 siblings, 0 replies; 128+ messages in thread
From: Matthew Wilcox @ 2021-06-24 20:06 UTC (permalink / raw)
  To: Christoph Hellwig; +Cc: akpm, linux-fsdevel, linux-mm, linux-kernel

On Wed, Jun 23, 2021 at 11:36:14AM +0200, Christoph Hellwig wrote:
> On Tue, Jun 22, 2021 at 01:15:34PM +0100, Matthew Wilcox (Oracle) wrote:
> > Get the statistics right; compound pages were being accounted as a
> > single page.
> 
> Maybe reword this a little to document the existing function that got
> it wrong, and why it did not matter before.

Get the statistics right; compound pages were being accounted as a
single page.  This didn't matter before now as no filesystem which
supported compound pages did writeback.  Also move the declaration
to filemap.h since this is part of the page cache.  Add a wrapper for
account_page_cleaned().


^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 31/46] mm/writeback: Add folio_clear_dirty_for_io()
  2021-06-23  9:43   ` Christoph Hellwig
@ 2021-06-24 20:09     ` Matthew Wilcox
  0 siblings, 0 replies; 128+ messages in thread
From: Matthew Wilcox @ 2021-06-24 20:09 UTC (permalink / raw)
  To: Christoph Hellwig; +Cc: akpm, linux-fsdevel, linux-mm, linux-kernel

On Wed, Jun 23, 2021 at 11:43:20AM +0200, Christoph Hellwig wrote:
> On Tue, Jun 22, 2021 at 01:15:36PM +0100, Matthew Wilcox (Oracle) wrote:
> > Transform clear_page_dirty_for_io() into folio_clear_dirty_for_io()
> > and add a compatibility wrapper.  Also move the declaration to pagemap.h
> > as this is page cache functionality that doesn't need to be used by the
> > rest of the kernel.
> > 
> > Increases the size of the kernel by 79 bytes.  While we remove a few
> > calls to compound_head(), we add a call to folio_nr_pages() to get the
> > stats correct.
> 
> ... for the eventual support of multi-page folios.

Added.

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 36/46] mm/filemap: Add readahead_folio()
  2021-06-23  9:50   ` Christoph Hellwig
@ 2021-06-24 23:46     ` Matthew Wilcox
  0 siblings, 0 replies; 128+ messages in thread
From: Matthew Wilcox @ 2021-06-24 23:46 UTC (permalink / raw)
  To: Christoph Hellwig; +Cc: akpm, linux-fsdevel, linux-mm, linux-kernel

On Wed, Jun 23, 2021 at 11:50:04AM +0200, Christoph Hellwig wrote:
> On Tue, Jun 22, 2021 at 01:15:41PM +0100, Matthew Wilcox (Oracle) wrote:
> > The pointers stored in the page cache are folios, by definition.
> > This change comes with a behaviour change -- callers of readahead_folio()
> > are no longer required to put the page reference themselves.  This matches
> > how readpage works, rather than matching how readpages used to work.
> 
> The way this stores and retrieves different but compatible types from the
> same xarray is a little nasty.  But I guess we'll have to live with it for
> now, so:

I think that's mostly fixed up by the end of this series.  I think
there's still a few bits which are currently postponed to series 4
(eg uses of __page_cache_alloc followed by add_to_page_cache_lru).

> Reviewed-by: Christoph Hellwig <hch@lst.de>

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 38/46] mm: Add folio_evictable()
  2021-06-23  9:54   ` Christoph Hellwig
@ 2021-06-24 23:50     ` Matthew Wilcox
  0 siblings, 0 replies; 128+ messages in thread
From: Matthew Wilcox @ 2021-06-24 23:50 UTC (permalink / raw)
  To: Christoph Hellwig; +Cc: akpm, linux-fsdevel, linux-mm, linux-kernel

On Wed, Jun 23, 2021 at 11:54:06AM +0200, Christoph Hellwig wrote:
> > + * Reasons folio might not be evictable:
> > + * 1. page's mapping marked unevictable
> > + * 2. page is part of an mlocked VMA
> 
> s/page/folio/?

Thanks.

 * Reasons folio might not be evictable:
 * 1. folio's mapping marked unevictable
 * 2. One of the pages in the folio is part of an mlocked VMA

> Otherwise looks good:
> 
> Reviewed-by: Christoph Hellwig <hch@lst.de>

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 41/46] mm/page_alloc: Add folio allocation functions
  2021-06-23  9:58   ` Christoph Hellwig
@ 2021-06-25  1:06     ` Matthew Wilcox
  0 siblings, 0 replies; 128+ messages in thread
From: Matthew Wilcox @ 2021-06-25  1:06 UTC (permalink / raw)
  To: Christoph Hellwig; +Cc: akpm, linux-fsdevel, linux-mm, linux-kernel

On Wed, Jun 23, 2021 at 11:58:02AM +0200, Christoph Hellwig wrote:
> On Tue, Jun 22, 2021 at 01:15:46PM +0100, Matthew Wilcox (Oracle) wrote:
> > +static inline
> > +struct folio *__alloc_folio_node(gfp_t gfp, unsigned int order, int nid)
> 
> Weirdo prototype formatting.
> 
> Otherwise looks good (assuming we grow callers):

The next patch adds filemap_alloc_folio() which uses these.  That then
gets used in "mm/filemap: Add filemap_get_folio", which is the patch
after that.

Now that I look at this patch again, I wonder if this shouldn't be
folio_alloc(), __folio_alloc() and __folio_alloc_node.  And even though
I have no users yet, completing the set with folio_alloc_node() might
be a good idea.

> Reviewed-by: Christoph Hellwig <hch@lst.de>

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 43/46] mm/filemap: Add filemap_add_folio
  2021-06-23 11:30   ` Christoph Hellwig
@ 2021-06-25  1:57     ` Matthew Wilcox
  0 siblings, 0 replies; 128+ messages in thread
From: Matthew Wilcox @ 2021-06-25  1:57 UTC (permalink / raw)
  To: Christoph Hellwig; +Cc: akpm, linux-fsdevel, linux-mm, linux-kernel

On Wed, Jun 23, 2021 at 01:30:04PM +0200, Christoph Hellwig wrote:
> On Tue, Jun 22, 2021 at 01:15:48PM +0100, Matthew Wilcox (Oracle) wrote:
> > Pages being added to the page cache should already be folios, so
> > just cast the page to a folio in the add_to_page_cache_lru() wrapper.
> > Saves 96 bytes of text.
> 
> modulo the casting:

ok.  Moved add_to_page_cache_lru() into folio-compat.c, added the call
to page_folio() and also added:

        if (!huge) {
+               VM_BUG_ON_FOLIO(index & (folio_nr_pages(folio) - 1), folio);
                error = folio_charge_cgroup(folio, current->mm, gfp);

as we don't want pages added at an unaligned index in the file.

> Reviewed-by: Christoph Hellwig <hch@lst.de>

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 44/46] mm/filemap: Convert mapping_get_entry to return a folio
  2021-06-23 11:32   ` Christoph Hellwig
@ 2021-06-25  3:29     ` Matthew Wilcox
  2021-06-28  6:07       ` Christoph Hellwig
  0 siblings, 1 reply; 128+ messages in thread
From: Matthew Wilcox @ 2021-06-25  3:29 UTC (permalink / raw)
  To: Christoph Hellwig; +Cc: akpm, linux-fsdevel, linux-mm, linux-kernel

On Wed, Jun 23, 2021 at 01:32:39PM +0200, Christoph Hellwig wrote:
> On Tue, Jun 22, 2021 at 01:15:49PM +0100, Matthew Wilcox (Oracle) wrote:
> > - * Return: The head page or shadow entry, %NULL if nothing is found.
> > + * Return: The folio, swap or shadow entry, %NULL if nothing is found.
> 
> This (old and new) reads a little weird, given that it returns a
> struct folio, even if that happens to be a magic entry.

Yeah.  How about this?

- * Return: The head page or shadow entry, %NULL if nothing is found.
+ * Return: The folio, swap or shadow entry, %NULL if nothing is found.
  */
-static struct page *mapping_get_entry(struct address_space *mapping,
-               pgoff_t index)
+static void *mapping_get_entry(struct address_space *mapping, pgoff_t index)
 {

I still use a struct folio in mapping_get_entry(), but this means that
pagecache_get_page() doesn't change in this patch.

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 11/46] mm/memcg: Remove 'page' parameter to mem_cgroup_charge_statistics()
  2021-06-22 12:15 ` [PATCH v2 11/46] mm/memcg: Remove 'page' parameter to mem_cgroup_charge_statistics() Matthew Wilcox (Oracle)
  2021-06-23  8:09   ` Christoph Hellwig
@ 2021-06-25  7:58   ` Michal Hocko
  1 sibling, 0 replies; 128+ messages in thread
From: Michal Hocko @ 2021-06-25  7:58 UTC (permalink / raw)
  To: Matthew Wilcox (Oracle); +Cc: akpm, linux-fsdevel, linux-mm, linux-kernel

On Tue 22-06-21 13:15:16, Matthew Wilcox wrote:
> The last use of 'page' was removed by commit 468c398233da ("mm:
> memcontrol: switch to native NR_ANON_THPS counter"), so we can now remove
> the parameter from the function.
> 
> Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>

Acked-by: Michal Hocko <mhocko@suse.com>
Thanks!

> ---
>  mm/memcontrol.c | 11 +++++------
>  1 file changed, 5 insertions(+), 6 deletions(-)
> 
> diff --git a/mm/memcontrol.c b/mm/memcontrol.c
> index 64ada9e650a5..1204c6a0c671 100644
> --- a/mm/memcontrol.c
> +++ b/mm/memcontrol.c
> @@ -814,7 +814,6 @@ static unsigned long memcg_events_local(struct mem_cgroup *memcg, int event)
>  }
>  
>  static void mem_cgroup_charge_statistics(struct mem_cgroup *memcg,
> -					 struct page *page,
>  					 int nr_pages)
>  {
>  	/* pagein of a big page is an event. So, ignore page size */
> @@ -5504,9 +5503,9 @@ static int mem_cgroup_move_account(struct page *page,
>  	ret = 0;
>  
>  	local_irq_disable();
> -	mem_cgroup_charge_statistics(to, page, nr_pages);
> +	mem_cgroup_charge_statistics(to, nr_pages);
>  	memcg_check_events(to, page);
> -	mem_cgroup_charge_statistics(from, page, -nr_pages);
> +	mem_cgroup_charge_statistics(from, -nr_pages);
>  	memcg_check_events(from, page);
>  	local_irq_enable();
>  out_unlock:
> @@ -6527,7 +6526,7 @@ static int __mem_cgroup_charge(struct page *page, struct mem_cgroup *memcg,
>  	commit_charge(page, memcg);
>  
>  	local_irq_disable();
> -	mem_cgroup_charge_statistics(memcg, page, nr_pages);
> +	mem_cgroup_charge_statistics(memcg, nr_pages);
>  	memcg_check_events(memcg, page);
>  	local_irq_enable();
>  out:
> @@ -6814,7 +6813,7 @@ void mem_cgroup_migrate(struct page *oldpage, struct page *newpage)
>  	commit_charge(newpage, memcg);
>  
>  	local_irq_save(flags);
> -	mem_cgroup_charge_statistics(memcg, newpage, nr_pages);
> +	mem_cgroup_charge_statistics(memcg, nr_pages);
>  	memcg_check_events(memcg, newpage);
>  	local_irq_restore(flags);
>  }
> @@ -7044,7 +7043,7 @@ void mem_cgroup_swapout(struct page *page, swp_entry_t entry)
>  	 * only synchronisation we have for updating the per-CPU variables.
>  	 */
>  	VM_BUG_ON(!irqs_disabled());
> -	mem_cgroup_charge_statistics(memcg, page, -nr_entries);
> +	mem_cgroup_charge_statistics(memcg, -nr_entries);
>  	memcg_check_events(memcg, page);
>  
>  	css_put(&memcg->css);
> -- 
> 2.30.2

-- 
Michal Hocko
SUSE Labs

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 12/46] mm/memcg: Use the node id in mem_cgroup_update_tree()
  2021-06-24 16:19     ` Matthew Wilcox
@ 2021-06-25  8:05       ` Michal Hocko
  0 siblings, 0 replies; 128+ messages in thread
From: Michal Hocko @ 2021-06-25  8:05 UTC (permalink / raw)
  To: Matthew Wilcox
  Cc: Christoph Hellwig, akpm, linux-fsdevel, linux-mm, linux-kernel,
	Johannes Weiner, Vladimir Davydov, cgroups

On Thu 24-06-21 17:19:53, Matthew Wilcox wrote:
> On Wed, Jun 23, 2021 at 10:12:40AM +0200, Christoph Hellwig wrote:
> > On Tue, Jun 22, 2021 at 01:15:17PM +0100, Matthew Wilcox (Oracle) wrote:
> > >  static struct mem_cgroup_per_node *
> > > -mem_cgroup_page_nodeinfo(struct mem_cgroup *memcg, struct page *page)
> > > +mem_cgroup_nodeinfo(struct mem_cgroup *memcg, int nid)
> > >  {
> > > -	int nid = page_to_nid(page);
> > > -
> > >  	return memcg->nodeinfo[nid];
> > >  }
> > 
> > I'd just kill this function entirely and open code it into the only
> > caller
> 
> Done.

This makes sense.

> > > -	mctz = soft_limit_tree_from_page(page);
> > > +	mctz = soft_limit_tree_node(nid);
> > 
> > And while were at it, soft_limit_tree_node seems like a completely
> > pointless helper that does nothing but obsfucating the code.  While
> > you touch this area it might be worth to spin another patch to just
> > remove it as well.

Yeah, the whole soft limit reclaim code is kinda pain to even look at.
Opencoding those two will certainly not make it worse so fine with me.

> I'm scared that if I touch this file too much, people will start to
> think I know something about memcgs.  Happy to add it on; cc'ing
> maintainers.

get_maintainers will surely notice ;)

-- 
Michal Hocko
SUSE Labs

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 13/46] mm/memcg: Convert commit_charge() to take a folio
  2021-06-22 12:15 ` [PATCH v2 13/46] mm/memcg: Convert commit_charge() to take a folio Matthew Wilcox (Oracle)
  2021-06-23  8:13   ` Christoph Hellwig
@ 2021-06-25  8:11   ` Michal Hocko
  1 sibling, 0 replies; 128+ messages in thread
From: Michal Hocko @ 2021-06-25  8:11 UTC (permalink / raw)
  To: Matthew Wilcox (Oracle); +Cc: akpm, linux-fsdevel, linux-mm, linux-kernel

On Tue 22-06-21 13:15:18, Matthew Wilcox wrote:
> The memcg_data is only set on the head page, so enforce that by
> typing it as a folio.
> 
> Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>

Acked-by: Michal Hocko <mhocko@suse.com>
Thanks!

> ---
>  mm/memcontrol.c | 27 +++++++++++++--------------
>  1 file changed, 13 insertions(+), 14 deletions(-)
> 
> diff --git a/mm/memcontrol.c b/mm/memcontrol.c
> index 7423cb11eb88..7939e4e9118d 100644
> --- a/mm/memcontrol.c
> +++ b/mm/memcontrol.c
> @@ -2700,9 +2700,9 @@ static void cancel_charge(struct mem_cgroup *memcg, unsigned int nr_pages)
>  }
>  #endif
>  
> -static void commit_charge(struct page *page, struct mem_cgroup *memcg)
> +static void commit_charge(struct folio *folio, struct mem_cgroup *memcg)
>  {
> -	VM_BUG_ON_PAGE(page_memcg(page), page);
> +	VM_BUG_ON_FOLIO(folio_memcg(folio), folio);
>  	/*
>  	 * Any of the following ensures page's memcg stability:
>  	 *
> @@ -2711,7 +2711,7 @@ static void commit_charge(struct page *page, struct mem_cgroup *memcg)
>  	 * - lock_page_memcg()
>  	 * - exclusive reference
>  	 */
> -	page->memcg_data = (unsigned long)memcg;
> +	folio->memcg_data = (unsigned long)memcg;
>  }
>  
>  static struct mem_cgroup *get_mem_cgroup_from_objcg(struct obj_cgroup *objcg)
> @@ -6506,7 +6506,8 @@ void mem_cgroup_calculate_protection(struct mem_cgroup *root,
>  static int __mem_cgroup_charge(struct page *page, struct mem_cgroup *memcg,
>  			       gfp_t gfp)
>  {
> -	unsigned int nr_pages = thp_nr_pages(page);
> +	struct folio *folio = page_folio(page);
> +	unsigned int nr_pages = folio_nr_pages(folio);
>  	int ret;
>  
>  	ret = try_charge(memcg, gfp, nr_pages);
> @@ -6514,7 +6515,7 @@ static int __mem_cgroup_charge(struct page *page, struct mem_cgroup *memcg,
>  		goto out;
>  
>  	css_get(&memcg->css);
> -	commit_charge(page, memcg);
> +	commit_charge(folio, memcg);
>  
>  	local_irq_disable();
>  	mem_cgroup_charge_statistics(memcg, nr_pages);
> @@ -6771,21 +6772,21 @@ void mem_cgroup_uncharge_list(struct list_head *page_list)
>   */
>  void mem_cgroup_migrate(struct page *oldpage, struct page *newpage)
>  {
> +	struct folio *newfolio = page_folio(newpage);
>  	struct mem_cgroup *memcg;
> -	unsigned int nr_pages;
> +	unsigned int nr_pages = folio_nr_pages(newfolio);
>  	unsigned long flags;
>  
>  	VM_BUG_ON_PAGE(!PageLocked(oldpage), oldpage);
> -	VM_BUG_ON_PAGE(!PageLocked(newpage), newpage);
> -	VM_BUG_ON_PAGE(PageAnon(oldpage) != PageAnon(newpage), newpage);
> -	VM_BUG_ON_PAGE(PageTransHuge(oldpage) != PageTransHuge(newpage),
> -		       newpage);
> +	VM_BUG_ON_FOLIO(!folio_locked(newfolio), newfolio);
> +	VM_BUG_ON_FOLIO(PageAnon(oldpage) != folio_anon(newfolio), newfolio);
> +	VM_BUG_ON_FOLIO(compound_nr(oldpage) != nr_pages, newfolio);
>  
>  	if (mem_cgroup_disabled())
>  		return;
>  
>  	/* Page cache replacement: new page already charged? */
> -	if (page_memcg(newpage))
> +	if (folio_memcg(newfolio))
>  		return;
>  
>  	memcg = page_memcg(oldpage);
> @@ -6794,14 +6795,12 @@ void mem_cgroup_migrate(struct page *oldpage, struct page *newpage)
>  		return;
>  
>  	/* Force-charge the new page. The old one will be freed soon */
> -	nr_pages = thp_nr_pages(newpage);
> -
>  	page_counter_charge(&memcg->memory, nr_pages);
>  	if (do_memsw_account())
>  		page_counter_charge(&memcg->memsw, nr_pages);
>  
>  	css_get(&memcg->css);
> -	commit_charge(newpage, memcg);
> +	commit_charge(newfolio, memcg);
>  
>  	local_irq_save(flags);
>  	mem_cgroup_charge_statistics(memcg, nr_pages);
> -- 
> 2.30.2

-- 
Michal Hocko
SUSE Labs

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 14/46] mm/memcg: Add folio_charge_cgroup()
  2021-06-24 16:42     ` Matthew Wilcox
@ 2021-06-25  8:22       ` Michal Hocko
  2021-06-25 11:34         ` Matthew Wilcox
  0 siblings, 1 reply; 128+ messages in thread
From: Michal Hocko @ 2021-06-25  8:22 UTC (permalink / raw)
  To: Matthew Wilcox
  Cc: Christoph Hellwig, akpm, linux-fsdevel, linux-mm, linux-kernel

On Thu 24-06-21 17:42:58, Matthew Wilcox wrote:
> On Wed, Jun 23, 2021 at 10:15:20AM +0200, Christoph Hellwig wrote:
> > On Tue, Jun 22, 2021 at 01:15:19PM +0100, Matthew Wilcox (Oracle) wrote:
> > > mem_cgroup_charge() already assumed it was being passed a non-tail
> > > page (and looking at the callers, that's true; it's called for freshly
> > > allocated pages).  The only real change here is that folio_nr_pages()
> > > doesn't compile away like thp_nr_pages() does as folio support
> > > is not conditional on transparent hugepage support.  Reimplement
> > > mem_cgroup_charge() as a wrapper around folio_charge_cgroup().
> > 
> > Maybe rename __mem_cgroup_charge to __folio_charge_cgroup as well?
> 
> Oh, yeah, should have done that.  Thanks.

I would stick with __mem_cgroup_charge here. Not that I would insist but the
folio nature is quite obvious from the parameter already.

Btw. memcg_check_events doesn't really need the page argument. A nid
should be sufficient and your earlier patch is already touching the
softlimit code so maybe it would be worth changing this page -> folio ->
page back and forth.

-- 
Michal Hocko
SUSE Labs

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 15/46] mm/memcg: Add folio_uncharge_cgroup()
  2021-06-22 12:15 ` [PATCH v2 15/46] mm/memcg: Add folio_uncharge_cgroup() Matthew Wilcox (Oracle)
  2021-06-23  8:16   ` Christoph Hellwig
@ 2021-06-25  8:25   ` Michal Hocko
  2021-06-25 11:21     ` Matthew Wilcox
  1 sibling, 1 reply; 128+ messages in thread
From: Michal Hocko @ 2021-06-25  8:25 UTC (permalink / raw)
  To: Matthew Wilcox (Oracle); +Cc: akpm, linux-fsdevel, linux-mm, linux-kernel

On Tue 22-06-21 13:15:20, Matthew Wilcox wrote:
> Reimplement mem_cgroup_uncharge() as a wrapper around
> folio_uncharge_cgroup().
> 
> Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>

Similar to the previous patch. Is there any reason why we cannot simply
stick with mem_cgroup_{un}charge and only change the parameter to folio?

> ---
>  include/linux/memcontrol.h |  5 +++++
>  mm/folio-compat.c          |  5 +++++
>  mm/memcontrol.c            | 14 +++++++-------
>  3 files changed, 17 insertions(+), 7 deletions(-)
> 
> diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
> index a50e5cee6d2c..d4b2bc939eee 100644
> --- a/include/linux/memcontrol.h
> +++ b/include/linux/memcontrol.h
> @@ -705,6 +705,7 @@ static inline bool mem_cgroup_below_min(struct mem_cgroup *memcg)
>  }
>  
>  int folio_charge_cgroup(struct folio *, struct mm_struct *, gfp_t);
> +void folio_uncharge_cgroup(struct folio *);
>  
>  int mem_cgroup_charge(struct page *page, struct mm_struct *mm, gfp_t gfp_mask);
>  int mem_cgroup_swapin_charge_page(struct page *page, struct mm_struct *mm,
> @@ -1224,6 +1225,10 @@ static inline int folio_charge_cgroup(struct folio *folio,
>  	return 0;
>  }
>  
> +static inline void folio_uncharge_cgroup(struct folio *folio)
> +{
> +}
> +
>  static inline int mem_cgroup_charge(struct page *page, struct mm_struct *mm,
>  				    gfp_t gfp_mask)
>  {
> diff --git a/mm/folio-compat.c b/mm/folio-compat.c
> index 1d71b8b587f8..d229b979b00d 100644
> --- a/mm/folio-compat.c
> +++ b/mm/folio-compat.c
> @@ -54,4 +54,9 @@ int mem_cgroup_charge(struct page *page, struct mm_struct *mm, gfp_t gfp)
>  {
>  	return folio_charge_cgroup(page_folio(page), mm, gfp);
>  }
> +
> +void mem_cgroup_uncharge(struct page *page)
> +{
> +	folio_uncharge_cgroup(page_folio(page));
> +}
>  #endif
> diff --git a/mm/memcontrol.c b/mm/memcontrol.c
> index 69638f84d11b..a6befc0843e7 100644
> --- a/mm/memcontrol.c
> +++ b/mm/memcontrol.c
> @@ -6717,24 +6717,24 @@ static void uncharge_page(struct page *page, struct uncharge_gather *ug)
>  }
>  
>  /**
> - * mem_cgroup_uncharge - uncharge a page
> - * @page: page to uncharge
> + * folio_uncharge_cgroup - Uncharge a folio.
> + * @folio: Folio to uncharge.
>   *
> - * Uncharge a page previously charged with mem_cgroup_charge().
> + * Uncharge a folio previously charged with folio_charge_cgroup().
>   */
> -void mem_cgroup_uncharge(struct page *page)
> +void folio_uncharge_cgroup(struct folio *folio)
>  {
>  	struct uncharge_gather ug;
>  
>  	if (mem_cgroup_disabled())
>  		return;
>  
> -	/* Don't touch page->lru of any random page, pre-check: */
> -	if (!page_memcg(page))
> +	/* Don't touch folio->lru of any random page, pre-check: */
> +	if (!folio_memcg(folio))
>  		return;
>  
>  	uncharge_gather_clear(&ug);
> -	uncharge_page(page, &ug);
> +	uncharge_page(&folio->page, &ug);
>  	uncharge_batch(&ug);
>  }
>  
> -- 
> 2.30.2

-- 
Michal Hocko
SUSE Labs

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 15/46] mm/memcg: Add folio_uncharge_cgroup()
  2021-06-25  8:25   ` Michal Hocko
@ 2021-06-25 11:21     ` Matthew Wilcox
  2021-06-25 13:21       ` Michal Hocko
  0 siblings, 1 reply; 128+ messages in thread
From: Matthew Wilcox @ 2021-06-25 11:21 UTC (permalink / raw)
  To: Michal Hocko; +Cc: akpm, linux-fsdevel, linux-mm, linux-kernel

On Fri, Jun 25, 2021 at 10:25:44AM +0200, Michal Hocko wrote:
> On Tue 22-06-21 13:15:20, Matthew Wilcox wrote:
> > Reimplement mem_cgroup_uncharge() as a wrapper around
> > folio_uncharge_cgroup().
> > 
> > Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
> 
> Similar to the previous patch. Is there any reason why we cannot simply
> stick with mem_cgroup_{un}charge and only change the parameter to folio?

There are a dozen callers of mem_cgroup_charge() and most of them
aren't quite ready to convert to folios at this point in the patch
series.  So either we need a new name for the variant that takes a
folio, or we need to play fun games with _Generic to allow
mem_cgroup_charge() to take either a folio or a page, or we convert
all callers to open-code their call to page_folio, like this:

-	if (mem_cgroup_charge(vmf->cow_page, vma->vm_mm, GFP_KERNEL)) {
+	if (mem_cgroup_charge(page_folio(vmf->cow_page), vma->vm_mm,
+			GFP_KERNEL)) {

I've generally gone with creating compat functions to minimise the
merge conflicts when people are adding new callers or changing code near
existing ones.  But if you don't like the new name, we have options.

> > ---
> >  include/linux/memcontrol.h |  5 +++++
> >  mm/folio-compat.c          |  5 +++++
> >  mm/memcontrol.c            | 14 +++++++-------
> >  3 files changed, 17 insertions(+), 7 deletions(-)
> > 
> > diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
> > index a50e5cee6d2c..d4b2bc939eee 100644
> > --- a/include/linux/memcontrol.h
> > +++ b/include/linux/memcontrol.h
> > @@ -705,6 +705,7 @@ static inline bool mem_cgroup_below_min(struct mem_cgroup *memcg)
> >  }
> >  
> >  int folio_charge_cgroup(struct folio *, struct mm_struct *, gfp_t);
> > +void folio_uncharge_cgroup(struct folio *);
> >  
> >  int mem_cgroup_charge(struct page *page, struct mm_struct *mm, gfp_t gfp_mask);
> >  int mem_cgroup_swapin_charge_page(struct page *page, struct mm_struct *mm,
> > @@ -1224,6 +1225,10 @@ static inline int folio_charge_cgroup(struct folio *folio,
> >  	return 0;
> >  }
> >  
> > +static inline void folio_uncharge_cgroup(struct folio *folio)
> > +{
> > +}
> > +
> >  static inline int mem_cgroup_charge(struct page *page, struct mm_struct *mm,
> >  				    gfp_t gfp_mask)
> >  {
> > diff --git a/mm/folio-compat.c b/mm/folio-compat.c
> > index 1d71b8b587f8..d229b979b00d 100644
> > --- a/mm/folio-compat.c
> > +++ b/mm/folio-compat.c
> > @@ -54,4 +54,9 @@ int mem_cgroup_charge(struct page *page, struct mm_struct *mm, gfp_t gfp)
> >  {
> >  	return folio_charge_cgroup(page_folio(page), mm, gfp);
> >  }
> > +
> > +void mem_cgroup_uncharge(struct page *page)
> > +{
> > +	folio_uncharge_cgroup(page_folio(page));
> > +}
> >  #endif
> > diff --git a/mm/memcontrol.c b/mm/memcontrol.c
> > index 69638f84d11b..a6befc0843e7 100644
> > --- a/mm/memcontrol.c
> > +++ b/mm/memcontrol.c
> > @@ -6717,24 +6717,24 @@ static void uncharge_page(struct page *page, struct uncharge_gather *ug)
> >  }
> >  
> >  /**
> > - * mem_cgroup_uncharge - uncharge a page
> > - * @page: page to uncharge
> > + * folio_uncharge_cgroup - Uncharge a folio.
> > + * @folio: Folio to uncharge.
> >   *
> > - * Uncharge a page previously charged with mem_cgroup_charge().
> > + * Uncharge a folio previously charged with folio_charge_cgroup().
> >   */
> > -void mem_cgroup_uncharge(struct page *page)
> > +void folio_uncharge_cgroup(struct folio *folio)
> >  {
> >  	struct uncharge_gather ug;
> >  
> >  	if (mem_cgroup_disabled())
> >  		return;
> >  
> > -	/* Don't touch page->lru of any random page, pre-check: */
> > -	if (!page_memcg(page))
> > +	/* Don't touch folio->lru of any random page, pre-check: */
> > +	if (!folio_memcg(folio))
> >  		return;
> >  
> >  	uncharge_gather_clear(&ug);
> > -	uncharge_page(page, &ug);
> > +	uncharge_page(&folio->page, &ug);
> >  	uncharge_batch(&ug);
> >  }
> >  
> > -- 
> > 2.30.2
> 
> -- 
> Michal Hocko
> SUSE Labs

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 14/46] mm/memcg: Add folio_charge_cgroup()
  2021-06-25  8:22       ` Michal Hocko
@ 2021-06-25 11:34         ` Matthew Wilcox
  0 siblings, 0 replies; 128+ messages in thread
From: Matthew Wilcox @ 2021-06-25 11:34 UTC (permalink / raw)
  To: Michal Hocko
  Cc: Christoph Hellwig, akpm, linux-fsdevel, linux-mm, linux-kernel

On Fri, Jun 25, 2021 at 10:22:35AM +0200, Michal Hocko wrote:
> On Thu 24-06-21 17:42:58, Matthew Wilcox wrote:
> > On Wed, Jun 23, 2021 at 10:15:20AM +0200, Christoph Hellwig wrote:
> > > On Tue, Jun 22, 2021 at 01:15:19PM +0100, Matthew Wilcox (Oracle) wrote:
> > > > mem_cgroup_charge() already assumed it was being passed a non-tail
> > > > page (and looking at the callers, that's true; it's called for freshly
> > > > allocated pages).  The only real change here is that folio_nr_pages()
> > > > doesn't compile away like thp_nr_pages() does as folio support
> > > > is not conditional on transparent hugepage support.  Reimplement
> > > > mem_cgroup_charge() as a wrapper around folio_charge_cgroup().
> > > 
> > > Maybe rename __mem_cgroup_charge to __folio_charge_cgroup as well?
> > 
> > Oh, yeah, should have done that.  Thanks.
> 
> I would stick with __mem_cgroup_charge here. Not that I would insist but the
> folio nature is quite obvious from the parameter already.
> 
> Btw. memcg_check_events doesn't really need the page argument. A nid
> should be sufficient and your earlier patch is already touching the
> softlimit code so maybe it would be worth changing this page -> folio ->
> page back and forth.

I'm not a huge fan of that 'dummy_page' component of uncharge_gather,
so replacing that with nid makes sense.  I'll juggle these patches a bit
and work that in.  Thanks!

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 15/46] mm/memcg: Add folio_uncharge_cgroup()
  2021-06-25 11:21     ` Matthew Wilcox
@ 2021-06-25 13:21       ` Michal Hocko
  0 siblings, 0 replies; 128+ messages in thread
From: Michal Hocko @ 2021-06-25 13:21 UTC (permalink / raw)
  To: Matthew Wilcox; +Cc: akpm, linux-fsdevel, linux-mm, linux-kernel

On Fri 25-06-21 12:21:32, Matthew Wilcox wrote:
> On Fri, Jun 25, 2021 at 10:25:44AM +0200, Michal Hocko wrote:
> > On Tue 22-06-21 13:15:20, Matthew Wilcox wrote:
> > > Reimplement mem_cgroup_uncharge() as a wrapper around
> > > folio_uncharge_cgroup().
> > > 
> > > Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
> > 
> > Similar to the previous patch. Is there any reason why we cannot simply
> > stick with mem_cgroup_{un}charge and only change the parameter to folio?
> 
> There are a dozen callers of mem_cgroup_charge() and most of them
> aren't quite ready to convert to folios at this point in the patch
> series.  So either we need a new name for the variant that takes a
> folio, or we need to play fun games with _Generic to allow
> mem_cgroup_charge() to take either a folio or a page, or we convert
> all callers to open-code their call to page_folio, like this:
> 
> -	if (mem_cgroup_charge(vmf->cow_page, vma->vm_mm, GFP_KERNEL)) {
> +	if (mem_cgroup_charge(page_folio(vmf->cow_page), vma->vm_mm,
> +			GFP_KERNEL)) {
> 
> I've generally gone with creating compat functions to minimise the
> merge conflicts when people are adding new callers or changing code near
> existing ones.  But if you don't like the new name, we have options.

Well, I will not insist because I can see how the conversion is PITA in
general.
mem_cgroup_charge should be something to be added very often so if you
do not mind I would go with your above example of direct usage of
page_folio() rather than wrappers.

Thanks!
-- 
Michal Hocko
SUSE Labs

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 27/46] mm/writeback: Add __folio_mark_dirty()
  2021-06-24 18:37     ` Matthew Wilcox
@ 2021-06-28  6:03       ` Christoph Hellwig
  2021-06-28 15:43         ` Matthew Wilcox
  0 siblings, 1 reply; 128+ messages in thread
From: Christoph Hellwig @ 2021-06-28  6:03 UTC (permalink / raw)
  To: Matthew Wilcox
  Cc: Christoph Hellwig, akpm, linux-fsdevel, linux-mm, linux-kernel

On Thu, Jun 24, 2021 at 07:37:30PM +0100, Matthew Wilcox wrote:
> On Wed, Jun 23, 2021 at 11:27:12AM +0200, Christoph Hellwig wrote:
> > On Tue, Jun 22, 2021 at 01:15:32PM +0100, Matthew Wilcox (Oracle) wrote:
> > > Turn __set_page_dirty() into a wrapper around __folio_mark_dirty() (which
> > > can directly cast from page to folio because we know that set_page_dirty()
> > > calls filesystems with the head page).  Convert account_page_dirtied()
> > > into folio_account_dirtied() and account the number of pages in the folio.
> > 
> > Is it really worth micro-optimizing a transitional function like that?
> > I'd rather eat the overhead of the compound_page() call over adding hacky
> > casts like this.
> 
> Fair enough.  There's only three calls to it and one of them goes away
> this series.

The other option would be a helper that asserts a page is not a tail
page and then do the cast to document the assumptions.

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 44/46] mm/filemap: Convert mapping_get_entry to return a folio
  2021-06-25  3:29     ` Matthew Wilcox
@ 2021-06-28  6:07       ` Christoph Hellwig
  0 siblings, 0 replies; 128+ messages in thread
From: Christoph Hellwig @ 2021-06-28  6:07 UTC (permalink / raw)
  To: Matthew Wilcox
  Cc: Christoph Hellwig, akpm, linux-fsdevel, linux-mm, linux-kernel

On Fri, Jun 25, 2021 at 04:29:47AM +0100, Matthew Wilcox wrote:
> > > - * Return: The head page or shadow entry, %NULL if nothing is found.
> > > + * Return: The folio, swap or shadow entry, %NULL if nothing is found.
> > 
> > This (old and new) reads a little weird, given that it returns a
> > struct folio, even if that happens to be a magic entry.
> 
> Yeah.  How about this?
> 
> - * Return: The head page or shadow entry, %NULL if nothing is found.
> + * Return: The folio, swap or shadow entry, %NULL if nothing is found.
>   */
> -static struct page *mapping_get_entry(struct address_space *mapping,
> -               pgoff_t index)
> +static void *mapping_get_entry(struct address_space *mapping, pgoff_t index)
>  {
> 
> I still use a struct folio in mapping_get_entry(), but this means that
> pagecache_get_page() doesn't change in this patch.

Much better, thanks.

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 01/46] mm: Add folio_to_pfn()
  2021-06-24 15:12     ` Matthew Wilcox
@ 2021-06-28  6:18       ` Christoph Hellwig
  0 siblings, 0 replies; 128+ messages in thread
From: Christoph Hellwig @ 2021-06-28  6:18 UTC (permalink / raw)
  To: Matthew Wilcox
  Cc: Christoph Hellwig, akpm, linux-fsdevel, linux-mm, linux-kernel

On Thu, Jun 24, 2021 at 04:12:05PM +0100, Matthew Wilcox wrote:
> On Wed, Jun 23, 2021 at 08:49:55AM +0100, Christoph Hellwig wrote:
> > On Tue, Jun 22, 2021 at 01:15:06PM +0100, Matthew Wilcox (Oracle) wrote:
> > > The pfn of a folio is the pfn of its head page.
> > > 
> > > Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
> > 
> > Maybe add a kerneldoc comment stating that?
> 
> /**
>  * folio_to_pfn - Return the Page Frame Number of a folio.
>  * @folio: The folio.
>  *
>  * A folio may contain multiple pages.  The pages have consecutive
>  * Page Frame Numbers.
>  *
>  * Return: The Page Frame Number of the first page in the folio.
>  */

Looks fine.

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 05/46] mm: Add arch_make_folio_accessible()
  2021-06-24 15:57     ` Matthew Wilcox
@ 2021-06-28  6:21       ` Christoph Hellwig
  2021-06-28 14:07         ` Matthew Wilcox
  0 siblings, 1 reply; 128+ messages in thread
From: Christoph Hellwig @ 2021-06-28  6:21 UTC (permalink / raw)
  To: Matthew Wilcox
  Cc: Christoph Hellwig, akpm, linux-fsdevel, linux-mm, linux-kernel

On Thu, Jun 24, 2021 at 04:57:29PM +0100, Matthew Wilcox wrote:
> On Wed, Jun 23, 2021 at 10:00:37AM +0200, Christoph Hellwig wrote:
> > On Tue, Jun 22, 2021 at 01:15:10PM +0100, Matthew Wilcox (Oracle) wrote:
> > > As a default implementation, call arch_make_page_accessible n times.
> > > If an architecture can do better, it can override this.
> > > 
> > > Also move the default implementation of arch_make_page_accessible()
> > > from gfp.h to mm.h.
> > 
> > Can we wait with introducing arch hooks until we have an actual user
> > lined up?
> 
> This one gets used in __folio_end_writeback() which is patch 24 in this
> series.

With arch hook I mean the ifdef to allow the architeture to override
the folio function.  Same for the previous patch, btw.

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 17/46] mm/memcg: Convert mem_cgroup_track_foreign_dirty_slowpath() to folio
  2021-06-24 17:37     ` Matthew Wilcox
@ 2021-06-28  6:24       ` Christoph Hellwig
  0 siblings, 0 replies; 128+ messages in thread
From: Christoph Hellwig @ 2021-06-28  6:24 UTC (permalink / raw)
  To: Matthew Wilcox
  Cc: Christoph Hellwig, akpm, linux-fsdevel, linux-mm, linux-kernel

On Thu, Jun 24, 2021 at 06:37:47PM +0100, Matthew Wilcox wrote:
> On Wed, Jun 23, 2021 at 10:21:26AM +0200, Christoph Hellwig wrote:
> > Looks good,
> > 
> > Reviewed-by: Christoph Hellwig <hch@lst.de>
> > 
> > Although I wish we could come up with a shorter name for
> > mem_cgroup_track_foreign_dirty_slowpath somehow..
> 
> It is quite grotesque!
> 
> How about folio_track_foreign_writeback() as a replacement name for
> mem_cgroup_track_foreign_dirty() and have it call
> __folio_track_foreign_writeback()?
> 
> Although 'foreign' tends to be used in MM to mean "wrong NUMA node",
> so maybe that's misleading.  folio_track_dirty_cgroup()?
> folio_mark_dirty_cgroup()?  (the last to be read in context of
> __set_page_dirty() being renamed to __folio_mark_dirty())

That all sounds reasonable to me, hopefully someone more attached to
this code can pick one.

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 20/46] mm/migrate: Add folio_migrate_copy()
  2021-06-24 18:02     ` Matthew Wilcox
@ 2021-06-28  6:26       ` Christoph Hellwig
  0 siblings, 0 replies; 128+ messages in thread
From: Christoph Hellwig @ 2021-06-28  6:26 UTC (permalink / raw)
  To: Matthew Wilcox
  Cc: Christoph Hellwig, akpm, linux-fsdevel, linux-mm, linux-kernel

On Thu, Jun 24, 2021 at 07:02:23PM +0100, Matthew Wilcox wrote:
> > What is the advantage of copying backwards here to start with?
> 
> Easier to write the loop this way?  I suppose we could do it as ...
> 
> 	unsigned int i, nr = folio_nr_pages(folio);
> 
> 	for (i = 0; i < nr; i++) {
> 		/* folio_page() handles discontinuities in memmap */
> 		copy_highpage(folio_page(newfolio, i), folio_page(folio, i));
> 		cond_resched();
> 	}

I'd prefer that if there is no obvious downside.

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 24/46] mm/writeback: Add __folio_end_writeback()
  2021-06-24 18:20     ` Matthew Wilcox
@ 2021-06-28  6:33       ` Christoph Hellwig
  0 siblings, 0 replies; 128+ messages in thread
From: Christoph Hellwig @ 2021-06-28  6:33 UTC (permalink / raw)
  To: Matthew Wilcox
  Cc: Christoph Hellwig, akpm, linux-fsdevel, linux-mm, linux-kernel

On Thu, Jun 24, 2021 at 07:20:12PM +0100, Matthew Wilcox wrote:
> > While this looks good, I think the whole abstraction is wrong.  I think
> > test_clear_page_writeback should just be merged into it's only caller.
> 
> I'm not opposed to doing that, but something else has to get
> un-static'ed in order to make that happen.
> 
> folio_end_writeback (exported, filemap.c)
>  -> folio_wake (static, filemap.c)
>      -> folio_wake_bit (static, filemap.c)
>  -> __folio_end_writeback (non-static, page-writeback.c)
>      -> __wb_writeout_add (static, page-writeback.c)
> 
> I'm not sure there's an obviously better split than where it is right
> now.

Ok, let's ignore that whole mess for now.  There is plenty bigger fish to
fry.

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 05/46] mm: Add arch_make_folio_accessible()
  2021-06-28  6:21       ` Christoph Hellwig
@ 2021-06-28 14:07         ` Matthew Wilcox
  0 siblings, 0 replies; 128+ messages in thread
From: Matthew Wilcox @ 2021-06-28 14:07 UTC (permalink / raw)
  To: Christoph Hellwig; +Cc: akpm, linux-fsdevel, linux-mm, linux-kernel

On Mon, Jun 28, 2021 at 07:21:01AM +0100, Christoph Hellwig wrote:
> On Thu, Jun 24, 2021 at 04:57:29PM +0100, Matthew Wilcox wrote:
> > On Wed, Jun 23, 2021 at 10:00:37AM +0200, Christoph Hellwig wrote:
> > > On Tue, Jun 22, 2021 at 01:15:10PM +0100, Matthew Wilcox (Oracle) wrote:
> > > > As a default implementation, call arch_make_page_accessible n times.
> > > > If an architecture can do better, it can override this.
> > > > 
> > > > Also move the default implementation of arch_make_page_accessible()
> > > > from gfp.h to mm.h.
> > > 
> > > Can we wait with introducing arch hooks until we have an actual user
> > > lined up?
> > 
> > This one gets used in __folio_end_writeback() which is patch 24 in this
> > series.
> 
> With arch hook I mean the ifdef to allow the architeture to override
> the folio function.  Same for the previous patch, btw.

Ah.  Actually, I hope that all architectures override this.  Ideally
'accessible' and 'dcache flush needed' would be per-folio flags, set only
on the head page, but the different architectures are inconsistent about
this.  So I've gone with "safe and slow" for the default, and maybe when
all architectures have decided that they'd rather be fast than safe, we
can fix this up.  As you know, I want to get rid of tail pages eventually,
so I'm trying to enable other people to do parts of that work for me.

^ permalink raw reply	[flat|nested] 128+ messages in thread

* Re: [PATCH v2 27/46] mm/writeback: Add __folio_mark_dirty()
  2021-06-28  6:03       ` Christoph Hellwig
@ 2021-06-28 15:43         ` Matthew Wilcox
  0 siblings, 0 replies; 128+ messages in thread
From: Matthew Wilcox @ 2021-06-28 15:43 UTC (permalink / raw)
  To: Christoph Hellwig; +Cc: akpm, linux-fsdevel, linux-mm, linux-kernel

On Mon, Jun 28, 2021 at 07:03:26AM +0100, Christoph Hellwig wrote:
> On Thu, Jun 24, 2021 at 07:37:30PM +0100, Matthew Wilcox wrote:
> > On Wed, Jun 23, 2021 at 11:27:12AM +0200, Christoph Hellwig wrote:
> > > On Tue, Jun 22, 2021 at 01:15:32PM +0100, Matthew Wilcox (Oracle) wrote:
> > > > Turn __set_page_dirty() into a wrapper around __folio_mark_dirty() (which
> > > > can directly cast from page to folio because we know that set_page_dirty()
> > > > calls filesystems with the head page).  Convert account_page_dirtied()
> > > > into folio_account_dirtied() and account the number of pages in the folio.
> > > 
> > > Is it really worth micro-optimizing a transitional function like that?
> > > I'd rather eat the overhead of the compound_page() call over adding hacky
> > > casts like this.
> > 
> > Fair enough.  There's only three calls to it and one of them goes away
> > this series.
> 
> The other option would be a helper that asserts a page is not a tail
> page and then do the cast to document the assumptions.

btw, every call to folio_flags() checks !PageTail:

        struct page *page = &folio->page;

        VM_BUG_ON_PGFLAGS(PageTail(page), page);

now, that's not going to be turned on for regular builds, but it does
give us a _lot_ of runtime assertions that somebody hasn't cast a tail
page to a folio.

^ permalink raw reply	[flat|nested] 128+ messages in thread

end of thread, other threads:[~2021-06-28 15:54 UTC | newest]

Thread overview: 128+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-06-22 12:15 [PATCH v2 00/46] Folio-enabling the page cache Matthew Wilcox (Oracle)
2021-06-22 12:15 ` [PATCH v2 01/46] mm: Add folio_to_pfn() Matthew Wilcox (Oracle)
2021-06-23  7:49   ` Christoph Hellwig
2021-06-24 15:12     ` Matthew Wilcox
2021-06-28  6:18       ` Christoph Hellwig
2021-06-22 12:15 ` [PATCH v2 02/46] mm: Add folio_rmapping() Matthew Wilcox (Oracle)
2021-06-23  7:56   ` Christoph Hellwig
2021-06-24 15:51     ` Matthew Wilcox
2021-06-22 12:15 ` [PATCH v2 03/46] mm: Add kmap_local_folio() Matthew Wilcox (Oracle)
2021-06-23  7:58   ` Christoph Hellwig
2021-06-22 12:15 ` [PATCH v2 04/46] mm: Add flush_dcache_folio() Matthew Wilcox (Oracle)
2021-06-22 12:15 ` [PATCH v2 05/46] mm: Add arch_make_folio_accessible() Matthew Wilcox (Oracle)
2021-06-23  8:00   ` Christoph Hellwig
2021-06-24 15:57     ` Matthew Wilcox
2021-06-28  6:21       ` Christoph Hellwig
2021-06-28 14:07         ` Matthew Wilcox
2021-06-22 12:15 ` [PATCH v2 06/46] mm: Add folio_young() and folio_idle() Matthew Wilcox (Oracle)
2021-06-22 12:15 ` [PATCH v2 07/46] mm/workingset: Convert workingset_activation to take a folio Matthew Wilcox (Oracle)
2021-06-23  8:02   ` Christoph Hellwig
2021-06-22 12:15 ` [PATCH v2 08/46] mm/swap: Add folio_activate() Matthew Wilcox (Oracle)
2021-06-23  8:04   ` Christoph Hellwig
2021-06-22 12:15 ` [PATCH v2 09/46] mm/swap: Add folio_mark_accessed() Matthew Wilcox (Oracle)
2021-06-23  8:07   ` Christoph Hellwig
2021-06-22 12:15 ` [PATCH v2 10/46] mm/rmap: Add folio_mkclean() Matthew Wilcox (Oracle)
2021-06-23  8:08   ` Christoph Hellwig
2021-06-22 12:15 ` [PATCH v2 11/46] mm/memcg: Remove 'page' parameter to mem_cgroup_charge_statistics() Matthew Wilcox (Oracle)
2021-06-23  8:09   ` Christoph Hellwig
2021-06-25  7:58   ` Michal Hocko
2021-06-22 12:15 ` [PATCH v2 12/46] mm/memcg: Use the node id in mem_cgroup_update_tree() Matthew Wilcox (Oracle)
2021-06-23  8:12   ` Christoph Hellwig
2021-06-24 16:19     ` Matthew Wilcox
2021-06-25  8:05       ` Michal Hocko
2021-06-22 12:15 ` [PATCH v2 13/46] mm/memcg: Convert commit_charge() to take a folio Matthew Wilcox (Oracle)
2021-06-23  8:13   ` Christoph Hellwig
2021-06-25  8:11   ` Michal Hocko
2021-06-22 12:15 ` [PATCH v2 14/46] mm/memcg: Add folio_charge_cgroup() Matthew Wilcox (Oracle)
2021-06-23  8:15   ` Christoph Hellwig
2021-06-24 16:42     ` Matthew Wilcox
2021-06-25  8:22       ` Michal Hocko
2021-06-25 11:34         ` Matthew Wilcox
2021-06-22 12:15 ` [PATCH v2 15/46] mm/memcg: Add folio_uncharge_cgroup() Matthew Wilcox (Oracle)
2021-06-23  8:16   ` Christoph Hellwig
2021-06-25  8:25   ` Michal Hocko
2021-06-25 11:21     ` Matthew Wilcox
2021-06-25 13:21       ` Michal Hocko
2021-06-22 12:15 ` [PATCH v2 16/46] mm/memcg: Add folio_migrate_cgroup() Matthew Wilcox (Oracle)
2021-06-23  8:19   ` Christoph Hellwig
2021-06-22 12:15 ` [PATCH v2 17/46] mm/memcg: Convert mem_cgroup_track_foreign_dirty_slowpath() to folio Matthew Wilcox (Oracle)
2021-06-23  8:21   ` Christoph Hellwig
2021-06-24 17:37     ` Matthew Wilcox
2021-06-28  6:24       ` Christoph Hellwig
2021-06-22 12:15 ` [PATCH v2 18/46] mm/migrate: Add folio_migrate_mapping() Matthew Wilcox (Oracle)
2021-06-23  8:22   ` Christoph Hellwig
2021-06-22 12:15 ` [PATCH v2 19/46] mm/migrate: Add folio_migrate_flags() Matthew Wilcox (Oracle)
2021-06-23  8:28   ` Christoph Hellwig
2021-06-24 17:55     ` Matthew Wilcox
2021-06-22 12:15 ` [PATCH v2 20/46] mm/migrate: Add folio_migrate_copy() Matthew Wilcox (Oracle)
2021-06-23  8:35   ` Christoph Hellwig
2021-06-24 18:02     ` Matthew Wilcox
2021-06-28  6:26       ` Christoph Hellwig
2021-06-22 12:15 ` [PATCH v2 21/46] mm/writeback: Rename __add_wb_stat() to wb_stat_mod() Matthew Wilcox (Oracle)
2021-06-23  8:37   ` Christoph Hellwig
2021-06-22 12:15 ` [PATCH v2 22/46] flex_proportions: Allow N events instead of 1 Matthew Wilcox (Oracle)
2021-06-23  8:41   ` Christoph Hellwig
2021-06-22 12:15 ` [PATCH v2 23/46] mm/writeback: Change __wb_writeout_inc() to __wb_writeout_add() Matthew Wilcox (Oracle)
2021-06-23  8:45   ` Christoph Hellwig
2021-06-22 12:15 ` [PATCH v2 24/46] mm/writeback: Add __folio_end_writeback() Matthew Wilcox (Oracle)
2021-06-23  9:15   ` Christoph Hellwig
2021-06-24 18:20     ` Matthew Wilcox
2021-06-28  6:33       ` Christoph Hellwig
2021-06-22 12:15 ` [PATCH v2 25/46] mm/writeback: Add folio_start_writeback() Matthew Wilcox (Oracle)
2021-06-23  9:18   ` Christoph Hellwig
2021-06-24 18:33     ` Matthew Wilcox
2021-06-22 12:15 ` [PATCH v2 26/46] mm/writeback: Add folio_mark_dirty() Matthew Wilcox (Oracle)
2021-06-23  9:21   ` Christoph Hellwig
2021-06-22 12:15 ` [PATCH v2 27/46] mm/writeback: Add __folio_mark_dirty() Matthew Wilcox (Oracle)
2021-06-23  9:27   ` Christoph Hellwig
2021-06-24 18:37     ` Matthew Wilcox
2021-06-28  6:03       ` Christoph Hellwig
2021-06-28 15:43         ` Matthew Wilcox
2021-06-22 12:15 ` [PATCH v2 28/46] mm/writeback: Add filemap_dirty_folio() Matthew Wilcox (Oracle)
2021-06-23  9:32   ` Christoph Hellwig
2021-06-22 12:15 ` [PATCH v2 29/46] mm/writeback: Add folio_account_cleaned() Matthew Wilcox (Oracle)
2021-06-23  9:36   ` Christoph Hellwig
2021-06-24 20:06     ` Matthew Wilcox
2021-06-22 12:15 ` [PATCH v2 30/46] mm/writeback: Add folio_cancel_dirty() Matthew Wilcox (Oracle)
2021-06-23  9:39   ` Christoph Hellwig
2021-06-22 12:15 ` [PATCH v2 31/46] mm/writeback: Add folio_clear_dirty_for_io() Matthew Wilcox (Oracle)
2021-06-23  9:43   ` Christoph Hellwig
2021-06-24 20:09     ` Matthew Wilcox
2021-06-22 12:15 ` [PATCH v2 32/46] mm/writeback: Add folio_account_redirty() Matthew Wilcox (Oracle)
2021-06-23  9:44   ` Christoph Hellwig
2021-06-22 12:15 ` [PATCH v2 33/46] mm/writeback: Add folio_redirty_for_writepage() Matthew Wilcox (Oracle)
2021-06-23  9:45   ` Christoph Hellwig
2021-06-22 12:15 ` [PATCH v2 34/46] mm/filemap: Add i_blocks_per_folio() Matthew Wilcox (Oracle)
2021-06-23  9:46   ` Christoph Hellwig
2021-06-22 12:15 ` [PATCH v2 35/46] mm/filemap: Add folio_mkwrite_check_truncate() Matthew Wilcox (Oracle)
2021-06-23  9:47   ` Christoph Hellwig
2021-06-23 13:19     ` Matthew Wilcox
2021-06-22 12:15 ` [PATCH v2 36/46] mm/filemap: Add readahead_folio() Matthew Wilcox (Oracle)
2021-06-23  9:50   ` Christoph Hellwig
2021-06-24 23:46     ` Matthew Wilcox
2021-06-22 12:15 ` [PATCH v2 37/46] mm/workingset: Convert workingset_refault() to take a folio Matthew Wilcox (Oracle)
2021-06-23  9:52   ` Christoph Hellwig
2021-06-22 12:15 ` [PATCH v2 38/46] mm: Add folio_evictable() Matthew Wilcox (Oracle)
2021-06-23  9:54   ` Christoph Hellwig
2021-06-24 23:50     ` Matthew Wilcox
2021-06-22 12:15 ` [PATCH v2 39/46] mm/lru: Convert __pagevec_lru_add_fn to take a folio Matthew Wilcox (Oracle)
2021-06-23  9:55   ` Christoph Hellwig
2021-06-22 12:15 ` [PATCH v2 40/46] mm/lru: Add folio_add_lru() Matthew Wilcox (Oracle)
2021-06-23  9:56   ` Christoph Hellwig
2021-06-22 12:15 ` [PATCH v2 41/46] mm/page_alloc: Add folio allocation functions Matthew Wilcox (Oracle)
2021-06-23  9:58   ` Christoph Hellwig
2021-06-25  1:06     ` Matthew Wilcox
2021-06-22 12:15 ` [PATCH v2 42/46] mm/filemap: Add filemap_alloc_folio Matthew Wilcox (Oracle)
2021-06-23  9:59   ` Christoph Hellwig
2021-06-22 12:15 ` [PATCH v2 43/46] mm/filemap: Add filemap_add_folio Matthew Wilcox (Oracle)
2021-06-23 11:30   ` Christoph Hellwig
2021-06-25  1:57     ` Matthew Wilcox
2021-06-22 12:15 ` [PATCH v2 44/46] mm/filemap: Convert mapping_get_entry to return a folio Matthew Wilcox (Oracle)
2021-06-23 11:32   ` Christoph Hellwig
2021-06-25  3:29     ` Matthew Wilcox
2021-06-28  6:07       ` Christoph Hellwig
2021-06-22 12:15 ` [PATCH v2 45/46] mm/filemap: Add filemap_get_folio Matthew Wilcox (Oracle)
2021-06-23 11:39   ` Christoph Hellwig
2021-06-22 12:15 ` [PATCH v2 46/46] mm/filemap: Add FGP_STABLE Matthew Wilcox (Oracle)
2021-06-23 11:43   ` Christoph Hellwig
2021-06-23 12:15     ` Matthew Wilcox

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).