[PATCH 0/3] mm/page_reporting: Make page reporting work on arm64 with 64KB page size

[PATCH 0/3] mm/page_reporting: Make page reporting work on arm64 with 64KB page size

[Gavin Shan]

The page reporting threshold is currently equal to @pageblock_order, which
is 13 and 512MB on arm64 with 64KB base page size selected. The page
reporting won't be triggered if the freeing page can't come up with a free
area like that huge. The condition is hard to be met, especially when the
system memory becomes fragmented.

This series intends to solve the issue by having page reporting threshold
as 5 (2MB) on arm64 with 64KB base page size. The patches are organized as:

   PATCH[1/3] introduces variable (@page_reporting_order) to replace original
              macro (PAGE_REPORTING_MIN_ORDER). It's also exported so that it
              can be adjusted at runtime.
   PATCH[2/3] renames PAGE_REPORTING_MIN_ORDER with PAGE_REPORTING_ORDER and
              allows architecture to specify its own version.
   PATCH[3/3] defines PAGE_REPORTING_ORDER to 5, corresponding to 2MB in size,
              on arm64 when 64KB base page size is selected. It's still same
              as to @pageblock_order for other architectures and cases.

Gavin Shan (3):
  mm/page_reporting: Allow to set reporting order
  mm/page_reporting: Allow architecture to select reporting order
  arm64: mm: Specify smaller page reporting order

 Documentation/admin-guide/kernel-parameters.txt |  6 ++++++
 arch/arm64/include/asm/page.h                   | 13 +++++++++++++
 mm/page_reporting.c                             |  8 ++++++--
 mm/page_reporting.h                             | 10 +++++++---
 4 files changed, 32 insertions(+), 5 deletions(-)

-- 
2.23.0

[PATCH 1/3] mm/page_reporting: Allow to set reporting order

[Gavin Shan]

The macro PAGE_REPORTING_MIN_ORDER is defined as the page reporting
threshold. Without a variable, it's impossible to be changed at runtime.

This introduces variable (@page_reporting_order) so that the threshold
can be changed at runtime.

Suggested-by: David Hildenbrand <david@redhat.com>
Signed-off-by: Gavin Shan <gshan@redhat.com>
---
 Documentation/admin-guide/kernel-parameters.txt | 6 ++++++
 mm/page_reporting.c                             | 8 ++++++--
 mm/page_reporting.h                             | 3 ++-
 3 files changed, 14 insertions(+), 3 deletions(-)

diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index cb89dbdedc46..566c4b9af3cd 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -3566,6 +3566,12 @@
 			off: turn off poisoning (default)
 			on: turn on poisoning
 
+	page_reporting.page_reporting_order=
+			[KNL] Minimal page reporting order
+			Format: <integer>
+			Adjust the minimal page reporting order. The page
+			reporting is disabled when it exceeds (MAX_ORDER-1).
+
 	panic=		[KNL] Kernel behaviour on panic: delay <timeout>
 			timeout > 0: seconds before rebooting
 			timeout = 0: wait forever
diff --git a/mm/page_reporting.c b/mm/page_reporting.c
index c50d93ffa252..759b29f6e845 100644
--- a/mm/page_reporting.c
+++ b/mm/page_reporting.c
@@ -10,6 +10,10 @@
 #include "page_reporting.h"
 #include "internal.h"
 
+unsigned int page_reporting_order = PAGE_REPORTING_MIN_ORDER;
+module_param(page_reporting_order, uint, 0644);
+MODULE_PARM_DESC(page_reporting_order, "Set page reporting order");
+
 #define PAGE_REPORTING_DELAY	(2 * HZ)
 static struct page_reporting_dev_info __rcu *pr_dev_info __read_mostly;
 
@@ -229,7 +233,7 @@ page_reporting_process_zone(struct page_reporting_dev_info *prdev,
 
 	/* Generate minimum watermark to be able to guarantee progress */
 	watermark = low_wmark_pages(zone) +
-		    (PAGE_REPORTING_CAPACITY << PAGE_REPORTING_MIN_ORDER);
+		    (PAGE_REPORTING_CAPACITY << page_reporting_order);
 
 	/*
 	 * Cancel request if insufficient free memory or if we failed
@@ -239,7 +243,7 @@ page_reporting_process_zone(struct page_reporting_dev_info *prdev,
 		return err;
 
 	/* Process each free list starting from lowest order/mt */
-	for (order = PAGE_REPORTING_MIN_ORDER; order < MAX_ORDER; order++) {
+	for (order = page_reporting_order; order < MAX_ORDER; order++) {
 		for (mt = 0; mt < MIGRATE_TYPES; mt++) {
 			/* We do not pull pages from the isolate free list */
 			if (is_migrate_isolate(mt))
diff --git a/mm/page_reporting.h b/mm/page_reporting.h
index 2c385dd4ddbd..19d58a590cc2 100644
--- a/mm/page_reporting.h
+++ b/mm/page_reporting.h
@@ -14,6 +14,7 @@
 
 #ifdef CONFIG_PAGE_REPORTING
 DECLARE_STATIC_KEY_FALSE(page_reporting_enabled);
+extern unsigned int page_reporting_order;
 void __page_reporting_notify(void);
 
 static inline bool page_reported(struct page *page)
@@ -38,7 +39,7 @@ static inline void page_reporting_notify_free(unsigned int order)
 		return;
 
 	/* Determine if we have crossed reporting threshold */
-	if (order < PAGE_REPORTING_MIN_ORDER)
+	if (order < page_reporting_order)
 		return;
 
 	/* This will add a few cycles, but should be called infrequently */
-- 
2.23.0

[PATCH 2/3] mm/page_reporting: Allow architecture to select reporting order

[Gavin Shan]

This renames PAGE_REPORTING_MIN_ORDER to PAGE_REPORTING_ORDER, to
match with the variable name (@page_reporting_order). Its value is
stick to @pageblock_order if architecture doesn't specify it. This
is going to be used by arm64 when 64KB base page size is selected.

Signed-off-by: Gavin Shan <gshan@redhat.com>
---
 mm/page_reporting.c | 2 +-
 mm/page_reporting.h | 7 +++++--
 2 files changed, 6 insertions(+), 3 deletions(-)

diff --git a/mm/page_reporting.c b/mm/page_reporting.c
index 759b29f6e845..d4d9f5265d6b 100644
--- a/mm/page_reporting.c
+++ b/mm/page_reporting.c
@@ -10,7 +10,7 @@
 #include "page_reporting.h"
 #include "internal.h"
 
-unsigned int page_reporting_order = PAGE_REPORTING_MIN_ORDER;
+unsigned int page_reporting_order = PAGE_REPORTING_ORDER;
 module_param(page_reporting_order, uint, 0644);
 MODULE_PARM_DESC(page_reporting_order, "Set page reporting order");
 
diff --git a/mm/page_reporting.h b/mm/page_reporting.h
index 19d58a590cc2..91688dd30abe 100644
--- a/mm/page_reporting.h
+++ b/mm/page_reporting.h
@@ -10,9 +10,12 @@
 #include <linux/pgtable.h>
 #include <linux/scatterlist.h>
 
-#define PAGE_REPORTING_MIN_ORDER	pageblock_order
-
 #ifdef CONFIG_PAGE_REPORTING
+
+#ifndef PAGE_REPORTING_ORDER
+#define PAGE_REPORTING_ORDER pageblock_order
+#endif
+
 DECLARE_STATIC_KEY_FALSE(page_reporting_enabled);
 extern unsigned int page_reporting_order;
 void __page_reporting_notify(void);
-- 
2.23.0

[PATCH 3/3] arm64: mm: Specify smaller page reporting order

[Gavin Shan]

The page reporting order is stick to @pageblock_order and its size
is 512MB when 64KB base page size is chosen. It also means the
page reporting won't be started if current freeing page can't come
up with a 512MB free area. It's hard when the system memory becomes
fragmented.

This specifies the page reporting order to 5 when the 64KB base page
size is chosen, corresponding to huge page size with 4KB base page
size, so that page reporting has more chance to work.

Cc: Anshuman Khandual <anshuman.khandual@arm.com>
Cc: Will Deacon <will@kernel.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: linux-arm-kernel@lists.infradead.org
Signed-off-by: Gavin Shan <gshan@redhat.com>
---
 arch/arm64/include/asm/page.h | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/arch/arm64/include/asm/page.h b/arch/arm64/include/asm/page.h
index 012cffc574e8..b8386e7d7d47 100644
--- a/arch/arm64/include/asm/page.h
+++ b/arch/arm64/include/asm/page.h
@@ -18,6 +18,19 @@
 struct page;
 struct vm_area_struct;
 
+/*
+ * The page reporting won't be started if the freeing page can't come up
+ * with a huge page, which is 512MB with 64KB base page size. It's hard
+ * to have 512MB free area when the memory becomes fragmented, stop page
+ * reporting from working properly. So we choose smaller page reporting
+ * order, which is same to the huge page size (2MB) with 4KB base page
+ * size. However, more overheads will be introduced because page reporting
+ * will be running more frequently.
+ */
+#if defined(CONFIG_PAGE_REPORTING) && defined(CONFIG_ARM64_64K_PAGES)
+#define PAGE_REPORTING_ORDER 5
+#endif
+
 extern void copy_page(void *to, const void *from);
 extern void clear_page(void *to);
 
-- 
2.23.0

Re: [PATCH 0/3] mm/page_reporting: Make page reporting work on arm64 with 64KB page size

[Alexander Duyck]

So the question I would have is what is the use case for this? It
seems like you don't have to deal with the guest native page size
issues since you are willing to break up what would otherwise be THP
pages on the guest, and the fact that you are willing to go down to
2MB pages which happens to align with the host THP page size for x86
makes me wonder if that is actually the environment you are running
in.

Rather than having the guest control this it might make sense to look
at adding an interface so that the page_reporting_register function
and the page_reporting_dev_info struct could be used to report and
configure the minimum page size that the host can support for the page
reporting. With that the host could then guarantee that it isn't going
to hurt performance by splitting pages on the host and risk hurting
the virtualization performance.

Also you would benefit by looking into the callers of
page_reporting_register as there are more than just the virtio balloon
that are consuming it. Odds are HyperV won't care about an ARM64
architecture, but your change would essentially disable it outright
which is why I think this might be better to address via the consumers
of page reporting rather than trying to address it in page reporting
itself.

Thanks,

- Alex

On Sun, Jun 20, 2021 at 8:11 PM Gavin Shan <gshan@redhat.com> wrote:
>
> The page reporting threshold is currently equal to @pageblock_order, which
> is 13 and 512MB on arm64 with 64KB base page size selected. The page
> reporting won't be triggered if the freeing page can't come up with a free
> area like that huge. The condition is hard to be met, especially when the
> system memory becomes fragmented.
>
> This series intends to solve the issue by having page reporting threshold
> as 5 (2MB) on arm64 with 64KB base page size. The patches are organized as:
>
>    PATCH[1/3] introduces variable (@page_reporting_order) to replace original
>               macro (PAGE_REPORTING_MIN_ORDER). It's also exported so that it
>               can be adjusted at runtime.
>    PATCH[2/3] renames PAGE_REPORTING_MIN_ORDER with PAGE_REPORTING_ORDER and
>               allows architecture to specify its own version.
>    PATCH[3/3] defines PAGE_REPORTING_ORDER to 5, corresponding to 2MB in size,
>               on arm64 when 64KB base page size is selected. It's still same
>               as to @pageblock_order for other architectures and cases.
>
> Gavin Shan (3):
>   mm/page_reporting: Allow to set reporting order
>   mm/page_reporting: Allow architecture to select reporting order
>   arm64: mm: Specify smaller page reporting order
>
>  Documentation/admin-guide/kernel-parameters.txt |  6 ++++++
>  arch/arm64/include/asm/page.h                   | 13 +++++++++++++
>  mm/page_reporting.c                             |  8 ++++++--
>  mm/page_reporting.h                             | 10 +++++++---
>  4 files changed, 32 insertions(+), 5 deletions(-)
>
> --
> 2.23.0
>

RE: [PATCH 0/3] mm/page_reporting: Make page reporting work on arm64 with 64KB page size

[Michael Kelley]

From: Alexander Duyck <alexander.duyck@gmail.com> Sent: Monday, June 21, 2021 7:02 AM
> To: Gavin Shan <gshan@redhat.com>
> Cc: linux-mm <linux-mm@kvack.org>; LKML <linux-kernel@vger.kernel.org>; Andrew
> Morton <akpm@linux-foundation.org>; David Hildenbrand <david@redhat.com>;
> Anshuman Khandual <anshuman.khandual@arm.com>; Catalin Marinas
> <catalin.marinas@arm.com>; Will Deacon <will@kernel.org>; shan.gavin@gmail.com
> Subject: Re: [PATCH 0/3] mm/page_reporting: Make page reporting work on arm64 with
> 64KB page size
> 
> So the question I would have is what is the use case for this? It
> seems like you don't have to deal with the guest native page size
> issues since you are willing to break up what would otherwise be THP
> pages on the guest, and the fact that you are willing to go down to
> 2MB pages which happens to align with the host THP page size for x86
> makes me wonder if that is actually the environment you are running
> in.
> 
> Rather than having the guest control this it might make sense to look
> at adding an interface so that the page_reporting_register function
> and the page_reporting_dev_info struct could be used to report and
> configure the minimum page size that the host can support for the page
> reporting. With that the host could then guarantee that it isn't going
> to hurt performance by splitting pages on the host and risk hurting
> the virtualization performance.
> 
> Also you would benefit by looking into the callers of
> page_reporting_register as there are more than just the virtio balloon
> that are consuming it. Odds are HyperV won't care about an ARM64
> architecture, 

FWIW, Hyper-V *does* care about ARM64.  It's already in use by
the Windows Subsystem for Linux VM that's part of Windows 10
on ARM64 hardware.  We're working to get the code accepted
upstream.

Michael

> but your change would essentially disable it outright
> which is why I think this might be better to address via the consumers
> of page reporting rather than trying to address it in page reporting
> itself.
> 
> Thanks,
> 
> - Alex
> 
> On Sun, Jun 20, 2021 at 8:11 PM Gavin Shan <gshan@redhat.com> wrote:
> >
> > The page reporting threshold is currently equal to @pageblock_order, which
> > is 13 and 512MB on arm64 with 64KB base page size selected. The page
> > reporting won't be triggered if the freeing page can't come up with a free
> > area like that huge. The condition is hard to be met, especially when the
> > system memory becomes fragmented.
> >
> > This series intends to solve the issue by having page reporting threshold
> > as 5 (2MB) on arm64 with 64KB base page size. The patches are organized as:
> >
> >    PATCH[1/3] introduces variable (@page_reporting_order) to replace original
> >               macro (PAGE_REPORTING_MIN_ORDER). It's also exported so that it
> >               can be adjusted at runtime.
> >    PATCH[2/3] renames PAGE_REPORTING_MIN_ORDER with PAGE_REPORTING_ORDER
> and
> >               allows architecture to specify its own version.
> >    PATCH[3/3] defines PAGE_REPORTING_ORDER to 5, corresponding to 2MB in size,
> >               on arm64 when 64KB base page size is selected. It's still same
> >               as to @pageblock_order for other architectures and cases.
> >
> > Gavin Shan (3):
> >   mm/page_reporting: Allow to set reporting order
> >   mm/page_reporting: Allow architecture to select reporting order
> >   arm64: mm: Specify smaller page reporting order
> >
> >  Documentation/admin-guide/kernel-parameters.txt |  6 ++++++
> >  arch/arm64/include/asm/page.h                   | 13 +++++++++++++
> >  mm/page_reporting.c                             |  8 ++++++--
> >  mm/page_reporting.h                             | 10 +++++++---
> >  4 files changed, 32 insertions(+), 5 deletions(-)
> >
> > --
> > 2.23.0
> >

Re: ** POTENTIAL FRAUD ALERT - RED HAT ** RE: [PATCH 0/3] mm/page_reporting: Make page reporting work on arm64 with 64KB page size

[Gavin Shan]

On 6/22/21 2:06 AM, Michael Kelley wrote:
> From: Alexander Duyck <alexander.duyck@gmail.com> Sent: Monday, June 21, 2021 7:02 AM
>> To: Gavin Shan <gshan@redhat.com>
>> Cc: linux-mm <linux-mm@kvack.org>; LKML <linux-kernel@vger.kernel.org>; Andrew
>> Morton <akpm@linux-foundation.org>; David Hildenbrand <david@redhat.com>;
>> Anshuman Khandual <anshuman.khandual@arm.com>; Catalin Marinas
>> <catalin.marinas@arm.com>; Will Deacon <will@kernel.org>; shan.gavin@gmail.com
>> Subject: Re: [PATCH 0/3] mm/page_reporting: Make page reporting work on arm64 with
>> 64KB page size
>>
>> So the question I would have is what is the use case for this? It
>> seems like you don't have to deal with the guest native page size
>> issues since you are willing to break up what would otherwise be THP
>> pages on the guest, and the fact that you are willing to go down to
>> 2MB pages which happens to align with the host THP page size for x86
>> makes me wonder if that is actually the environment you are running
>> in.
>>
>> Rather than having the guest control this it might make sense to look
>> at adding an interface so that the page_reporting_register function
>> and the page_reporting_dev_info struct could be used to report and
>> configure the minimum page size that the host can support for the page
>> reporting. With that the host could then guarantee that it isn't going
>> to hurt performance by splitting pages on the host and risk hurting
>> the virtualization performance.
>>
>> Also you would benefit by looking into the callers of
>> page_reporting_register as there are more than just the virtio balloon
>> that are consuming it. Odds are HyperV won't care about an ARM64
>> architecture,
> 
> FWIW, Hyper-V *does* care about ARM64.  It's already in use by
> the Windows Subsystem for Linux VM that's part of Windows 10
> on ARM64 hardware.  We're working to get the code accepted
> upstream.
> 

Michael, thanks for your confirmation. As the issue found on 64KB guest
when memory balloon is used, lets resolve the case first. I will look
into Hyper-V case later if you agree. It won't be difficult to fix the
same issue for Hyper-V after the solution is figured out for memory
balloon.

Thanks,
Gavin

> 
>> but your change would essentially disable it outright
>> which is why I think this might be better to address via the consumers
>> of page reporting rather than trying to address it in page reporting
>> itself.
>>
>> Thanks,
>>
>> - Alex
>>
>> On Sun, Jun 20, 2021 at 8:11 PM Gavin Shan <gshan@redhat.com> wrote:
>>>
>>> The page reporting threshold is currently equal to @pageblock_order, which
>>> is 13 and 512MB on arm64 with 64KB base page size selected. The page
>>> reporting won't be triggered if the freeing page can't come up with a free
>>> area like that huge. The condition is hard to be met, especially when the
>>> system memory becomes fragmented.
>>>
>>> This series intends to solve the issue by having page reporting threshold
>>> as 5 (2MB) on arm64 with 64KB base page size. The patches are organized as:
>>>
>>>     PATCH[1/3] introduces variable (@page_reporting_order) to replace original
>>>                macro (PAGE_REPORTING_MIN_ORDER). It's also exported so that it
>>>                can be adjusted at runtime.
>>>     PATCH[2/3] renames PAGE_REPORTING_MIN_ORDER with PAGE_REPORTING_ORDER
>> and
>>>                allows architecture to specify its own version.
>>>     PATCH[3/3] defines PAGE_REPORTING_ORDER to 5, corresponding to 2MB in size,
>>>                on arm64 when 64KB base page size is selected. It's still same
>>>                as to @pageblock_order for other architectures and cases.
>>>
>>> Gavin Shan (3):
>>>    mm/page_reporting: Allow to set reporting order
>>>    mm/page_reporting: Allow architecture to select reporting order
>>>    arm64: mm: Specify smaller page reporting order
>>>
>>>   Documentation/admin-guide/kernel-parameters.txt |  6 ++++++
>>>   arch/arm64/include/asm/page.h                   | 13 +++++++++++++
>>>   mm/page_reporting.c                             |  8 ++++++--
>>>   mm/page_reporting.h                             | 10 +++++++---
>>>   4 files changed, 32 insertions(+), 5 deletions(-)
>>>
>>> --
>>> 2.23.0
>>>

Re: [PATCH 0/3] mm/page_reporting: Make page reporting work on arm64 with 64KB page size

[Gavin Shan]

On 6/22/21 12:02 AM, Alexander Duyck wrote:
> So the question I would have is what is the use case for this? It
> seems like you don't have to deal with the guest native page size
> issues since you are willing to break up what would otherwise be THP
> pages on the guest, and the fact that you are willing to go down to
> 2MB pages which happens to align with the host THP page size for x86
> makes me wonder if that is actually the environment you are running
> in.
> 
> Rather than having the guest control this it might make sense to look
> at adding an interface so that the page_reporting_register function
> and the page_reporting_dev_info struct could be used to report and
> configure the minimum page size that the host can support for the page
> reporting. With that the host could then guarantee that it isn't going
> to hurt performance by splitting pages on the host and risk hurting
> the virtualization performance.
> 
> Also you would benefit by looking into the callers of
> page_reporting_register as there are more than just the virtio balloon
> that are consuming it. Odds are HyperV won't care about an ARM64
> architecture, but your change would essentially disable it outright
> which is why I think this might be better to address via the consumers
> of page reporting rather than trying to address it in page reporting
> itself.
> 

Alex, the issue was initially found on guest with 64KB base page size
when memory balloon is used. The same issue isn't found on guest with
4KB base page size. Both guests can be running on host with 4KB or
64KB base page size. Besides, the code changes are specific to ARM64,
meaning they don't affect other architectures.

(1) If we have the combination of guest.64KB and host.4KB, the 2MB page
     reporting order is still avoid splitting the THP on host.
(2) If we have the combination of guest.64KB and host.64KB, the THP splitting
     can't be avoided.

After thinking about it, I agree to reuse page_report_register() and
"struct page_reporting_dev_info" to have the page reporting order. In
this way, the PAGE_REPORTING_CAPACITY can be provided by consumer in
future either. I will have these updates in v2 patches.

Thanks,
Gavin

> 
> On Sun, Jun 20, 2021 at 8:11 PM Gavin Shan <gshan@redhat.com> wrote:
>>
>> The page reporting threshold is currently equal to @pageblock_order, which
>> is 13 and 512MB on arm64 with 64KB base page size selected. The page
>> reporting won't be triggered if the freeing page can't come up with a free
>> area like that huge. The condition is hard to be met, especially when the
>> system memory becomes fragmented.
>>
>> This series intends to solve the issue by having page reporting threshold
>> as 5 (2MB) on arm64 with 64KB base page size. The patches are organized as:
>>
>>     PATCH[1/3] introduces variable (@page_reporting_order) to replace original
>>                macro (PAGE_REPORTING_MIN_ORDER). It's also exported so that it
>>                can be adjusted at runtime.
>>     PATCH[2/3] renames PAGE_REPORTING_MIN_ORDER with PAGE_REPORTING_ORDER and
>>                allows architecture to specify its own version.
>>     PATCH[3/3] defines PAGE_REPORTING_ORDER to 5, corresponding to 2MB in size,
>>                on arm64 when 64KB base page size is selected. It's still same
>>                as to @pageblock_order for other architectures and cases.
>>
>> Gavin Shan (3):
>>    mm/page_reporting: Allow to set reporting order
>>    mm/page_reporting: Allow architecture to select reporting order
>>    arm64: mm: Specify smaller page reporting order
>>
>>   Documentation/admin-guide/kernel-parameters.txt |  6 ++++++
>>   arch/arm64/include/asm/page.h                   | 13 +++++++++++++
>>   mm/page_reporting.c                             |  8 ++++++--
>>   mm/page_reporting.h                             | 10 +++++++---
>>   4 files changed, 32 insertions(+), 5 deletions(-)
>>
>> --
>> 2.23.0
>>
>