[PATCH 0/4] Convert memcontrol charge moving to use folios

[PATCH 0/4] Convert memcontrol charge moving to use folios

[Matthew Wilcox (Oracle)]

No part of these patches should change behaviour; all the called functions
already convert from page to folio, so this ought to simply be a reduction
in the number of calls to compound_head().

Matthew Wilcox (Oracle) (4):
  memcg: Convert mem_cgroup_move_charge_pte_range() to use a folio
  memcg: Return the folio in union mc_target
  memcg: Use a folio in get_mctgt_type
  memcg: Use a folio in get_mctgt_type_thp

 mm/memcontrol.c | 88 ++++++++++++++++++++++++++-----------------------
 1 file changed, 46 insertions(+), 42 deletions(-)

-- 
2.43.0

[PATCH 1/4] memcg: Convert mem_cgroup_move_charge_pte_range() to use a folio

[Matthew Wilcox (Oracle)]

Remove many calls to compound_head() by calling page_folio() once at
the start of each stanza which receives a struct page from 'target'.
There should be no change in behaviour here as all the called functions
start out by converting the page to its folio.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 mm/memcontrol.c | 49 ++++++++++++++++++++++++-------------------------
 1 file changed, 24 insertions(+), 25 deletions(-)

diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index c4c422c81f93..c04bda961165 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -5932,23 +5932,22 @@ static struct page *mc_handle_file_pte(struct vm_area_struct *vma,
 }
 
 /**
- * mem_cgroup_move_account - move account of the page
- * @page: the page
+ * mem_cgroup_move_account - move account of the folio
+ * @folio: The folio.
  * @compound: charge the page as compound or small page
- * @from: mem_cgroup which the page is moved from.
- * @to:	mem_cgroup which the page is moved to. @from != @to.
+ * @from: mem_cgroup which the folio is moved from.
+ * @to:	mem_cgroup which the folio is moved to. @from != @to.
  *
- * The page must be locked and not on the LRU.
+ * The folio must be locked and not on the LRU.
  *
  * This function doesn't do "charge" to new cgroup and doesn't do "uncharge"
  * from old cgroup.
  */
-static int mem_cgroup_move_account(struct page *page,
+static int mem_cgroup_move_account(struct folio *folio,
 				   bool compound,
 				   struct mem_cgroup *from,
 				   struct mem_cgroup *to)
 {
-	struct folio *folio = page_folio(page);
 	struct lruvec *from_vec, *to_vec;
 	struct pglist_data *pgdat;
 	unsigned int nr_pages = compound ? folio_nr_pages(folio) : 1;
@@ -6398,7 +6397,7 @@ static int mem_cgroup_move_charge_pte_range(pmd_t *pmd,
 	spinlock_t *ptl;
 	enum mc_target_type target_type;
 	union mc_target target;
-	struct page *page;
+	struct folio *folio;
 
 	ptl = pmd_trans_huge_lock(pmd, vma);
 	if (ptl) {
@@ -6408,26 +6407,26 @@ static int mem_cgroup_move_charge_pte_range(pmd_t *pmd,
 		}
 		target_type = get_mctgt_type_thp(vma, addr, *pmd, &target);
 		if (target_type == MC_TARGET_PAGE) {
-			page = target.page;
-			if (isolate_lru_page(page)) {
-				if (!mem_cgroup_move_account(page, true,
+			folio = page_folio(target.page);
+			if (folio_isolate_lru(folio)) {
+				if (!mem_cgroup_move_account(folio, true,
 							     mc.from, mc.to)) {
 					mc.precharge -= HPAGE_PMD_NR;
 					mc.moved_charge += HPAGE_PMD_NR;
 				}
-				putback_lru_page(page);
+				folio_putback_lru(folio);
 			}
-			unlock_page(page);
-			put_page(page);
+			folio_unlock(folio);
+			folio_put(folio);
 		} else if (target_type == MC_TARGET_DEVICE) {
-			page = target.page;
-			if (!mem_cgroup_move_account(page, true,
+			folio = page_folio(target.page);
+			if (!mem_cgroup_move_account(folio, true,
 						     mc.from, mc.to)) {
 				mc.precharge -= HPAGE_PMD_NR;
 				mc.moved_charge += HPAGE_PMD_NR;
 			}
-			unlock_page(page);
-			put_page(page);
+			folio_unlock(folio);
+			folio_put(folio);
 		}
 		spin_unlock(ptl);
 		return 0;
@@ -6450,28 +6449,28 @@ static int mem_cgroup_move_charge_pte_range(pmd_t *pmd,
 			device = true;
 			fallthrough;
 		case MC_TARGET_PAGE:
-			page = target.page;
+			folio = page_folio(target.page);
 			/*
 			 * We can have a part of the split pmd here. Moving it
 			 * can be done but it would be too convoluted so simply
 			 * ignore such a partial THP and keep it in original
 			 * memcg. There should be somebody mapping the head.
 			 */
-			if (PageTransCompound(page))
+			if (folio_test_large(folio))
 				goto put;
-			if (!device && !isolate_lru_page(page))
+			if (!device && !folio_isolate_lru(folio))
 				goto put;
-			if (!mem_cgroup_move_account(page, false,
+			if (!mem_cgroup_move_account(folio, false,
 						mc.from, mc.to)) {
 				mc.precharge--;
 				/* we uncharge from mc.from later. */
 				mc.moved_charge++;
 			}
 			if (!device)
-				putback_lru_page(page);
+				folio_putback_lru(folio);
 put:			/* get_mctgt_type() gets & locks the page */
-			unlock_page(page);
-			put_page(page);
+			folio_unlock(folio);
+			folio_put(folio);
 			break;
 		case MC_TARGET_SWAP:
 			ent = target.ent;
-- 
2.43.0

Re: [PATCH 1/4] memcg: Convert mem_cgroup_move_charge_pte_range() to use a folio

[Muchun Song]

> On Jan 12, 2024, at 02:12, Matthew Wilcox (Oracle) <willy@infradead.org> wrote:
> 
> Remove many calls to compound_head() by calling page_folio() once at
> the start of each stanza which receives a struct page from 'target'.
> There should be no change in behaviour here as all the called functions
> start out by converting the page to its folio.
> 
> Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>

Reviewed-by: Muchun Song <muchun.song@linux.dev>

Thanks.

[PATCH 2/4] memcg: Return the folio in union mc_target

[Matthew Wilcox (Oracle)]

All users of target.page convert it to the folio, so we can just return
the folio directly and save a few calls to compound_head().

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 mm/memcontrol.c | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index c04bda961165..d14fe0740b37 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -5840,7 +5840,7 @@ static int mem_cgroup_do_precharge(unsigned long count)
 }
 
 union mc_target {
-	struct page	*page;
+	struct folio	*folio;
 	swp_entry_t	ent;
 };
 
@@ -6062,7 +6062,7 @@ static int mem_cgroup_move_account(struct folio *folio,
  * Return:
  * * MC_TARGET_NONE - If the pte is not a target for move charge.
  * * MC_TARGET_PAGE - If the page corresponding to this pte is a target for
- *   move charge. If @target is not NULL, the page is stored in target->page
+ *   move charge. If @target is not NULL, the folio is stored in target->folio
  *   with extra refcnt taken (Caller should release it).
  * * MC_TARGET_SWAP - If the swap entry corresponding to this pte is a
  *   target for charge migration.  If @target is not NULL, the entry is
@@ -6127,7 +6127,7 @@ static enum mc_target_type get_mctgt_type(struct vm_area_struct *vma,
 			    is_device_coherent_page(page))
 				ret = MC_TARGET_DEVICE;
 			if (target)
-				target->page = page;
+				target->folio = page_folio(page);
 		}
 		if (!ret || !target) {
 			if (target)
@@ -6177,7 +6177,7 @@ static enum mc_target_type get_mctgt_type_thp(struct vm_area_struct *vma,
 				put_page(page);
 				return MC_TARGET_NONE;
 			}
-			target->page = page;
+			target->folio = page_folio(page);
 		}
 	}
 	return ret;
@@ -6407,7 +6407,7 @@ static int mem_cgroup_move_charge_pte_range(pmd_t *pmd,
 		}
 		target_type = get_mctgt_type_thp(vma, addr, *pmd, &target);
 		if (target_type == MC_TARGET_PAGE) {
-			folio = page_folio(target.page);
+			folio = target.folio;
 			if (folio_isolate_lru(folio)) {
 				if (!mem_cgroup_move_account(folio, true,
 							     mc.from, mc.to)) {
@@ -6419,7 +6419,7 @@ static int mem_cgroup_move_charge_pte_range(pmd_t *pmd,
 			folio_unlock(folio);
 			folio_put(folio);
 		} else if (target_type == MC_TARGET_DEVICE) {
-			folio = page_folio(target.page);
+			folio = target.folio;
 			if (!mem_cgroup_move_account(folio, true,
 						     mc.from, mc.to)) {
 				mc.precharge -= HPAGE_PMD_NR;
@@ -6449,7 +6449,7 @@ static int mem_cgroup_move_charge_pte_range(pmd_t *pmd,
 			device = true;
 			fallthrough;
 		case MC_TARGET_PAGE:
-			folio = page_folio(target.page);
+			folio = target.folio;
 			/*
 			 * We can have a part of the split pmd here. Moving it
 			 * can be done but it would be too convoluted so simply
-- 
2.43.0

Re: [PATCH 2/4] memcg: Return the folio in union mc_target

[Muchun Song]

> On Jan 12, 2024, at 02:12, Matthew Wilcox (Oracle) <willy@infradead.org> wrote:
> 
> All users of target.page convert it to the folio, so we can just return
> the folio directly and save a few calls to compound_head().
> 
> Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>

Reviewed-by: Muchun Song <muchun.song@linux.dev>

Thanks.

[PATCH 3/4] memcg: Use a folio in get_mctgt_type

[Matthew Wilcox (Oracle)]

Replace seven calls to compound_head() with one.  We still use the
page as page_mapped() is different from folio_mapped().

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 mm/memcontrol.c | 23 +++++++++++++----------
 1 file changed, 13 insertions(+), 10 deletions(-)

diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index d14fe0740b37..b6096c34b3e4 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -6076,6 +6076,7 @@ static enum mc_target_type get_mctgt_type(struct vm_area_struct *vma,
 		unsigned long addr, pte_t ptent, union mc_target *target)
 {
 	struct page *page = NULL;
+	struct folio *folio;
 	enum mc_target_type ret = MC_TARGET_NONE;
 	swp_entry_t ent = { .val = 0 };
 
@@ -6090,9 +6091,11 @@ static enum mc_target_type get_mctgt_type(struct vm_area_struct *vma,
 	else if (is_swap_pte(ptent))
 		page = mc_handle_swap_pte(vma, ptent, &ent);
 
+	if (page)
+		folio = page_folio(page);
 	if (target && page) {
-		if (!trylock_page(page)) {
-			put_page(page);
+		if (!folio_trylock(folio)) {
+			folio_put(folio);
 			return ret;
 		}
 		/*
@@ -6107,8 +6110,8 @@ static enum mc_target_type get_mctgt_type(struct vm_area_struct *vma,
 		 * Alas, skip moving the page in this case.
 		 */
 		if (!pte_present(ptent) && page_mapped(page)) {
-			unlock_page(page);
-			put_page(page);
+			folio_unlock(folio);
+			folio_put(folio);
 			return ret;
 		}
 	}
@@ -6121,18 +6124,18 @@ static enum mc_target_type get_mctgt_type(struct vm_area_struct *vma,
 		 * mem_cgroup_move_account() checks the page is valid or
 		 * not under LRU exclusion.
 		 */
-		if (page_memcg(page) == mc.from) {
+		if (folio_memcg(folio) == mc.from) {
 			ret = MC_TARGET_PAGE;
-			if (is_device_private_page(page) ||
-			    is_device_coherent_page(page))
+			if (folio_is_device_private(folio) ||
+			    folio_is_device_coherent(folio))
 				ret = MC_TARGET_DEVICE;
 			if (target)
-				target->folio = page_folio(page);
+				target->folio = folio;
 		}
 		if (!ret || !target) {
 			if (target)
-				unlock_page(page);
-			put_page(page);
+				folio_unlock(folio);
+			folio_put(folio);
 		}
 	}
 	/*
-- 
2.43.0

Re: [PATCH 3/4] memcg: Use a folio in get_mctgt_type

[Johannes Weiner]

On Thu, Jan 11, 2024 at 06:12:18PM +0000, Matthew Wilcox (Oracle) wrote:
> Replace seven calls to compound_head() with one.  We still use the
> page as page_mapped() is different from folio_mapped().

Only seven, eh? :) For the 4 patches:

Acked-by: Johannes Weiner <hannes@cmpxchg.org>

Thanks

Re: [PATCH 3/4] memcg: Use a folio in get_mctgt_type

[Muchun Song]

> On Jan 12, 2024, at 02:12, Matthew Wilcox (Oracle) <willy@infradead.org> wrote:
> 
> Replace seven calls to compound_head() with one.  We still use the
> page as page_mapped() is different from folio_mapped().
> 
> Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>

Reviewed-by: Muchun Song <muchun.song@linux.dev>

Thanks.

[PATCH 4/4] memcg: Use a folio in get_mctgt_type_thp

[Matthew Wilcox (Oracle)]

Replace five calls to compound_head() with one.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 mm/memcontrol.c | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)

diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index b6096c34b3e4..935f48c4d399 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -6161,6 +6161,7 @@ static enum mc_target_type get_mctgt_type_thp(struct vm_area_struct *vma,
 		unsigned long addr, pmd_t pmd, union mc_target *target)
 {
 	struct page *page = NULL;
+	struct folio *folio;
 	enum mc_target_type ret = MC_TARGET_NONE;
 
 	if (unlikely(is_swap_pmd(pmd))) {
@@ -6170,17 +6171,18 @@ static enum mc_target_type get_mctgt_type_thp(struct vm_area_struct *vma,
 	}
 	page = pmd_page(pmd);
 	VM_BUG_ON_PAGE(!page || !PageHead(page), page);
+	folio = page_folio(page);
 	if (!(mc.flags & MOVE_ANON))
 		return ret;
-	if (page_memcg(page) == mc.from) {
+	if (folio_memcg(folio) == mc.from) {
 		ret = MC_TARGET_PAGE;
 		if (target) {
-			get_page(page);
-			if (!trylock_page(page)) {
-				put_page(page);
+			folio_get(folio);
+			if (!folio_trylock(folio)) {
+				folio_put(folio);
 				return MC_TARGET_NONE;
 			}
-			target->folio = page_folio(page);
+			target->folio = folio;
 		}
 	}
 	return ret;
-- 
2.43.0

Re: [PATCH 4/4] memcg: Use a folio in get_mctgt_type_thp

[Roman Gushchin]

On Thu, Jan 11, 2024 at 06:12:19PM +0000, Matthew Wilcox (Oracle) wrote:
> Replace five calls to compound_head() with one.
> 
> Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>

Reviewed-by: Roman Gushchin <roman.gushchin@linux.dev>
for the series.

Looks good, thanks!

Re: [PATCH 4/4] memcg: Use a folio in get_mctgt_type_thp

[Muchun Song]

> On Jan 12, 2024, at 02:12, Matthew Wilcox (Oracle) <willy@infradead.org> wrote:
> 
> Replace five calls to compound_head() with one.
> 
> Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>

Reviewed-by: Muchun Song <muchun.song@linux.dev>

Thanks.

Re: [PATCH 0/4] Convert memcontrol charge moving to use folios

[Shakeel Butt]

On Thu, Jan 11, 2024 at 10:12 AM Matthew Wilcox (Oracle)
<willy@infradead.org> wrote:
>
> No part of these patches should change behaviour; all the called functions
> already convert from page to folio, so this ought to simply be a reduction
> in the number of calls to compound_head().
>
> Matthew Wilcox (Oracle) (4):
>   memcg: Convert mem_cgroup_move_charge_pte_range() to use a folio
>   memcg: Return the folio in union mc_target
>   memcg: Use a folio in get_mctgt_type
>   memcg: Use a folio in get_mctgt_type_thp
>
>  mm/memcontrol.c | 88 ++++++++++++++++++++++++++-----------------------
>  1 file changed, 46 insertions(+), 42 deletions(-)
>

For the series:

Acked-by: Shakeel Butt <shakeelb@google.com>

Re: [PATCH 0/4] Convert memcontrol charge moving to use folios

[Michal Hocko]

Hi,

On Thu 11-01-24 18:12:15, Matthew Wilcox wrote:
> No part of these patches should change behaviour; all the called functions
> already convert from page to folio, so this ought to simply be a reduction
> in the number of calls to compound_head().
> 
> Matthew Wilcox (Oracle) (4):
>   memcg: Convert mem_cgroup_move_charge_pte_range() to use a folio
>   memcg: Return the folio in union mc_target
>   memcg: Use a folio in get_mctgt_type
>   memcg: Use a folio in get_mctgt_type_thp
> 
>  mm/memcontrol.c | 88 ++++++++++++++++++++++++++-----------------------
>  1 file changed, 46 insertions(+), 42 deletions(-)

sorry, I have missed this before. Thanks for the conversion, all seem
fine. The code is rarely executed because it is not enabled by cgroup
v2. But I do agree that it nicer that it fits to the new folio scheme
after these patches.

Acked-by: Michal Hocko <mhocko@suse.com>

Thanks!
-- 
Michal Hocko
SUSE Labs