* [PATCH v1 0/4] Enable >0 order folio memory compaction
@ 2023-11-13 17:01 Zi Yan
2023-11-13 17:01 ` [PATCH v1 1/4] mm/compaction: enable compacting >0 order folios Zi Yan
` (5 more replies)
0 siblings, 6 replies; 21+ messages in thread
From: Zi Yan @ 2023-11-13 17:01 UTC (permalink / raw)
To: linux-mm, linux-kernel
Cc: Zi Yan, Huang, Ying, Ryan Roberts, Andrew Morton,
Matthew Wilcox (Oracle),
David Hildenbrand, Yin, Fengwei, Yu Zhao, Vlastimil Babka,
Kirill A . Shutemov, Johannes Weiner, Baolin Wang, Kemeng Shi,
Mel Gorman, Rohan Puri, Mcgrof Chamberlain, Adam Manzanares,
Vishal Moola (Oracle)
From: Zi Yan <ziy@nvidia.com>
Hi all,
This patchset enables >0 order folio memory compaction, which is one of
the prerequisitions for large folio support[1]. It is on top of
mm-everything-2023-10-21-02-40 (also applies to mm-everything-2023-11-10-20-42
without conflicts).
Per patch vm-scalability results on a 8-CPU (Intel Xeon E5-2650 v4 @ 2.20GHz)
16G VM are at the bottom and based on mm-everything-2023-10-21-02-40.
Each has 5 runs. Highlights:
1. mmap-pread-seq-mt: -18.3% if split folios during compaction and -7.7%
if migrating folios during compaction.
2. lru-file-readtwice: +41.5% if split folios during compaction and
+30.4% if migrating folios during compaction.
3. lru-file-mmap-read: +35.1% if split folios during compaction and
+32.3% if migrating folios during compaction.
I will run experiments on bare metal and report results later.
Changelog
===
From RFC:
1. Enabled >0 order folio compaction in the first patch by splitting all
to-be-migrated folios. (per Huang, Ying)
2. Stopped isolating compound pages with order greater than cc->order
to avoid wasting effort, since cc->order gives a hint that no free pages
with order greater than it exist, thus migrating the compound pages will fail.
(per Baolin Wang)
3. Retained the folio check within lru lock. (per Baolin Wang)
4. Made isolate_freepages_block() generate order-sorted multi lists.
(per Johannes Weiner)
Overview
===
To support >0 order folio compaction, the patchset changes how free pages used
for migration are kept during compaction. Free pages used to be split into
order-0 pages that are post allocation processed (i.e., PageBuddy flag cleared,
page order stored in page->private is zeroed, and page reference is set to 1).
Now all free pages are kept in a MAX_ORDER+1 array of page lists based
on their order without post allocation process. When migrate_pages() asks for
a new page, one of the free pages, based on the requested page order, is
then processed and given out.
Optimizations
===
1. Free page split is added to increase migration success rate in case
a source page does not have a matched free page in the free page lists.
Free page merge is possible but not implemented, since existing
PFN-based buddy page merge algorithm requires the identification of
buddy pages, but free pages kept for memory compaction cannot have
PageBuddy set to avoid confusing other PFN scanners.
2. Sort source pages in ascending order before migration is added to
reduce free page split. Otherwise, high order free pages might be
prematurely split, causing undesired high order folio migration failures.
TODOs
===
1. Refactor free page post allocation and free page preparation code so
that compaction_alloc() and compaction_free() can call functions instead
of hard coding.
2. Make isolate_migratepages_block() produce a sorted list for
compaction to avoid sort_folio_by_order() in Patch 4.
Feel free to give comments and ask questions.
Thanks.
vm-scalability results
===
=========================================================================================
compiler/kconfig/rootfs/runtime/tbox_group/test/testcase:
gcc-13/defconfig/debian/300s/qemu-vm/mmap-xread-seq-mt/vm-scalability
commit:
6.6.0-rc4-mm-everything-2023-10-21-02-40+
6.6.0-rc4-split-folio-in-compaction+
6.6.0-rc4-folio-migration-in-compaction+
6.6.0-rc4-folio-migration-free-page-split+
6.6.0-rc4-folio-migration-free-page-split-sort-src+
6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f
---------------- --------------------------- --------------------------- --------------------------- ---------------------------
%stddev %change %stddev %change %stddev %change %stddev %change %stddev
\ | \ | \ | \ | \
12896955 +2.7% 13249322 -4.0% 12385175 ± 5% +1.1% 13033951 -0.4% 12845698 vm-scalability.throughput
=========================================================================================
compiler/kconfig/rootfs/runtime/tbox_group/test/testcase:
gcc-13/defconfig/debian/300s/qemu-vm/mmap-pread-seq/vm-scalability
commit:
6.6.0-rc4-mm-everything-2023-10-21-02-40+
6.6.0-rc4-split-folio-in-compaction+
6.6.0-rc4-folio-migration-in-compaction+
6.6.0-rc4-folio-migration-free-page-split+
6.6.0-rc4-folio-migration-free-page-split-sort-src+
6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f
---------------- --------------------------- --------------------------- --------------------------- ---------------------------
%stddev %change %stddev %change %stddev %change %stddev %change %stddev
\ | \ | \ | \ | \
14626665 -5.5% 13825048 -1.7% 14384837 -1.9% 14345958 -2.0% 14338834 vm-scalability.throughput
=========================================================================================
compiler/kconfig/rootfs/runtime/tbox_group/test/testcase:
gcc-13/defconfig/debian/300s/qemu-vm/mmap-pread-seq-mt/vm-scalability
commit:
6.6.0-rc4-mm-everything-2023-10-21-02-40+
6.6.0-rc4-split-folio-in-compaction+
6.6.0-rc4-folio-migration-in-compaction+
6.6.0-rc4-folio-migration-free-page-split+
6.6.0-rc4-folio-migration-free-page-split-sort-src+
6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f
---------------- --------------------------- --------------------------- --------------------------- ---------------------------
%stddev %change %stddev %change %stddev %change %stddev %change %stddev
\ | \ | \ | \ | \
14314960 -18.3% 11691969 ± 2% -7.7% 13217795 -8.8% 13057395 -10.8% 12764204 vm-scalability.throughput
=========================================================================================
compiler/kconfig/rootfs/runtime/tbox_group/test/testcase:
gcc-13/defconfig/debian/300s/qemu-vm/lru-file-readtwice/vm-scalability
commit:
6.6.0-rc4-mm-everything-2023-10-21-02-40+
6.6.0-rc4-split-folio-in-compaction+
6.6.0-rc4-folio-migration-in-compaction+
6.6.0-rc4-folio-migration-free-page-split+
6.6.0-rc4-folio-migration-free-page-split-sort-src+
6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f
---------------- --------------------------- --------------------------- --------------------------- ---------------------------
%stddev %change %stddev %change %stddev %change %stddev %change %stddev
\ | \ | \ | \ | \
6498203 ± 6% +41.5% 9193356 +30.4% 8472024 +30.0% 8449054 ± 3% +32.3% 8596781 ± 2% vm-scalability.throughput
=========================================================================================
compiler/kconfig/rootfs/runtime/tbox_group/test/testcase:
gcc-13/defconfig/debian/300s/qemu-vm/lru-file-mmap-read/vm-scalability
commit:
6.6.0-rc4-mm-everything-2023-10-21-02-40+
6.6.0-rc4-split-folio-in-compaction+
6.6.0-rc4-folio-migration-in-compaction+
6.6.0-rc4-folio-migration-free-page-split+
6.6.0-rc4-folio-migration-free-page-split-sort-src+
6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f
---------------- --------------------------- --------------------------- --------------------------- ---------------------------
%stddev %change %stddev %change %stddev %change %stddev %change %stddev
\ | \ | \ | \ | \
5440732 ± 10% +35.1% 7352631 ± 9% +32.3% 7200622 ± 5% +27.0% 6909673 ± 4% +25.9% 6849852 ± 5% vm-scalability.throughput
=========================================================================================
compiler/kconfig/rootfs/runtime/tbox_group/test/testcase:
gcc-13/defconfig/debian/300s/qemu-vm/lru-file-mmap-read-rand/vm-scalability
commit:
6.6.0-rc4-mm-everything-2023-10-21-02-40+
6.6.0-rc4-split-folio-in-compaction+
6.6.0-rc4-folio-migration-in-compaction+
6.6.0-rc4-folio-migration-free-page-split+
6.6.0-rc4-folio-migration-free-page-split-sort-src+
6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f
---------------- --------------------------- --------------------------- --------------------------- ---------------------------
%stddev %change %stddev %change %stddev %change %stddev %change %stddev
\ | \ | \ | \ | \
1627989 +0.3% 1632141 +1.8% 1656877 +0.1% 1629816 -0.2% 1625213 vm-scalability.throughput
=========================================================================================
compiler/kconfig/rootfs/runtime/size/tbox_group/test/testcase:
gcc-13/defconfig/debian/300s/8T/qemu-vm/anon-wx-seq-mt/vm-scalability
commit:
6.6.0-rc4-mm-everything-2023-10-21-02-40+
6.6.0-rc4-split-folio-in-compaction+
6.6.0-rc4-folio-migration-in-compaction+
6.6.0-rc4-folio-migration-free-page-split+
6.6.0-rc4-folio-migration-free-page-split-sort-src+
6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f
---------------- --------------------------- --------------------------- --------------------------- ---------------------------
%stddev %change %stddev %change %stddev %change %stddev %change %stddev
\ | \ | \ | \ | \
6738043 +1.4% 6835096 +1.4% 6834329 +1.0% 6805401 +1.2% 6820637 vm-scalability.throughput
=========================================================================================
compiler/kconfig/rootfs/runtime/size/tbox_group/test/testcase:
gcc-13/defconfig/debian/300s/8T/qemu-vm/anon-w-seq-mt/vm-scalability
commit:
6.6.0-rc4-mm-everything-2023-10-21-02-40+
6.6.0-rc4-split-folio-in-compaction+
6.6.0-rc4-folio-migration-in-compaction+
6.6.0-rc4-folio-migration-free-page-split+
6.6.0-rc4-folio-migration-free-page-split-sort-src+
6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f
---------------- --------------------------- --------------------------- --------------------------- ---------------------------
%stddev %change %stddev %change %stddev %change %stddev %change %stddev
\ | \ | \ | \ | \
7644894 +0.2% 7661425 +0.4% 7676929 +1.0% 7719218 +0.3% 7664658 vm-scalability.throughput
=========================================================================================
compiler/kconfig/rootfs/runtime/size/tbox_group/test/testcase:
gcc-13/defconfig/debian/300s/512G/qemu-vm/anon-wx-rand-mt/vm-scalability
commit:
6.6.0-rc4-mm-everything-2023-10-21-02-40+
6.6.0-rc4-split-folio-in-compaction+
6.6.0-rc4-folio-migration-in-compaction+
6.6.0-rc4-folio-migration-free-page-split+
6.6.0-rc4-folio-migration-free-page-split-sort-src+
6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f
---------------- --------------------------- --------------------------- --------------------------- ---------------------------
%stddev %change %stddev %change %stddev %change %stddev %change %stddev
\ | \ | \ | \ | \
521941 +3.3% 539254 +3.3% 539213 +1.0% 527048 +3.4% 539588 vm-scalability.throughput
=========================================================================================
compiler/kconfig/rootfs/runtime/size/tbox_group/test/testcase:
gcc-13/defconfig/debian/300s/512G/qemu-vm/anon-w-rand/vm-scalability
commit:
6.6.0-rc4-mm-everything-2023-10-21-02-40+
6.6.0-rc4-split-folio-in-compaction+
6.6.0-rc4-folio-migration-in-compaction+
6.6.0-rc4-folio-migration-free-page-split+
6.6.0-rc4-folio-migration-free-page-split-sort-src+
6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f
---------------- --------------------------- --------------------------- --------------------------- ---------------------------
%stddev %change %stddev %change %stddev %change %stddev %change %stddev
\ | \ | \ | \ | \
481028 +4.3% 501876 +2.4% 492713 +0.4% 482829 ± 2% +3.5% 497649 ± 2% vm-scalability.throughput
=========================================================================================
compiler/kconfig/rootfs/runtime/size/tbox_group/test/testcase:
gcc-13/defconfig/debian/300s/512G/qemu-vm/anon-w-rand-mt/vm-scalability
commit:
6.6.0-rc4-mm-everything-2023-10-21-02-40+
6.6.0-rc4-split-folio-in-compaction+
6.6.0-rc4-folio-migration-in-compaction+
6.6.0-rc4-folio-migration-free-page-split+
6.6.0-rc4-folio-migration-free-page-split-sort-src+
6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f
---------------- --------------------------- --------------------------- --------------------------- ---------------------------
%stddev %change %stddev %change %stddev %change %stddev %change %stddev
\ | \ | \ | \ | \
477063 +4.0% 496373 +4.0% 496283 +1.4% 483966 +3.2% 492216 ± 2% vm-scalability.throughput
=========================================================================================
compiler/kconfig/rootfs/runtime/size/tbox_group/test/testcase:
gcc-13/defconfig/debian/300s/512G/qemu-vm/anon-cow-rand/vm-scalability
commit:
6.6.0-rc4-mm-everything-2023-10-21-02-40+
6.6.0-rc4-split-folio-in-compaction+
6.6.0-rc4-folio-migration-in-compaction+
6.6.0-rc4-folio-migration-free-page-split+
6.6.0-rc4-folio-migration-free-page-split-sort-src+
6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f
---------------- --------------------------- --------------------------- --------------------------- ---------------------------
%stddev %change %stddev %change %stddev %change %stddev %change %stddev
\ | \ | \ | \ | \
458073 +3.3% 473056 +2.6% 469835 +0.3% 459266 +3.0% 471735 vm-scalability.throughput
=========================================================================================
compiler/kconfig/rootfs/runtime/size/tbox_group/test/testcase:
gcc-13/defconfig/debian/300s/512G/qemu-vm/anon-cow-rand-mt/vm-scalability
commit:
6.6.0-rc4-mm-everything-2023-10-21-02-40+
6.6.0-rc4-split-folio-in-compaction+
6.6.0-rc4-folio-migration-in-compaction+
6.6.0-rc4-folio-migration-free-page-split+
6.6.0-rc4-folio-migration-free-page-split-sort-src+
6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f
---------------- --------------------------- --------------------------- --------------------------- ---------------------------
%stddev %change %stddev %change %stddev %change %stddev %change %stddev
\ | \ | \ | \ | \
468158 +0.8% 472115 +1.4% 474547 +1.1% 473307 +2.1% 478173 vm-scalability.throughput
=========================================================================================
compiler/kconfig/rootfs/runtime/size/tbox_group/test/testcase:
gcc-13/defconfig/debian/300s/2T/qemu-vm/shm-xread-seq/vm-scalability
commit:
6.6.0-rc4-mm-everything-2023-10-21-02-40+
6.6.0-rc4-split-folio-in-compaction+
6.6.0-rc4-folio-migration-in-compaction+
6.6.0-rc4-folio-migration-free-page-split+
6.6.0-rc4-folio-migration-free-page-split-sort-src+
6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f
---------------- --------------------------- --------------------------- --------------------------- ---------------------------
%stddev %change %stddev %change %stddev %change %stddev %change %stddev
\ | \ | \ | \ | \
3489946 +1.4% 3540036 +0.4% 3505413 +1.7% 3549852 +1.4% 3539309 vm-scalability.throughput
=========================================================================================
compiler/kconfig/rootfs/runtime/size/tbox_group/test/testcase:
gcc-13/defconfig/debian/300s/2T/qemu-vm/shm-xread-seq-mt/vm-scalability
commit:
6.6.0-rc4-mm-everything-2023-10-21-02-40+
6.6.0-rc4-split-folio-in-compaction+
6.6.0-rc4-folio-migration-in-compaction+
6.6.0-rc4-folio-migration-free-page-split+
6.6.0-rc4-folio-migration-free-page-split-sort-src+
6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f
---------------- --------------------------- --------------------------- --------------------------- ---------------------------
%stddev %change %stddev %change %stddev %change %stddev %change %stddev
\ | \ | \ | \ | \
2621783 +1.0% 2647480 +0.6% 2636399 +0.7% 2641150 +0.1% 2624301 vm-scalability.throughput
=========================================================================================
compiler/kconfig/rootfs/runtime/size/tbox_group/test/testcase:
gcc-13/defconfig/debian/300s/2T/qemu-vm/shm-pread-seq/vm-scalability
commit:
6.6.0-rc4-mm-everything-2023-10-21-02-40+
6.6.0-rc4-split-folio-in-compaction+
6.6.0-rc4-folio-migration-in-compaction+
6.6.0-rc4-folio-migration-free-page-split+
6.6.0-rc4-folio-migration-free-page-split-sort-src+
6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f
---------------- --------------------------- --------------------------- --------------------------- ---------------------------
%stddev %change %stddev %change %stddev %change %stddev %change %stddev
\ | \ | \ | \ | \
3504293 +1.5% 3557664 -0.4% 3489639 +0.4% 3516842 +0.8% 3533944 vm-scalability.throughput
=========================================================================================
compiler/kconfig/rootfs/runtime/size/tbox_group/test/testcase:
gcc-13/defconfig/debian/300s/256G/qemu-vm/msync/vm-scalability
commit:
6.6.0-rc4-mm-everything-2023-10-21-02-40+
6.6.0-rc4-split-folio-in-compaction+
6.6.0-rc4-folio-migration-in-compaction+
6.6.0-rc4-folio-migration-free-page-split+
6.6.0-rc4-folio-migration-free-page-split-sort-src+
6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f
---------------- --------------------------- --------------------------- --------------------------- ---------------------------
%stddev %change %stddev %change %stddev %change %stddev %change %stddev
\ | \ | \ | \ | \
919378 ± 6% +3.8% 954028 ± 10% +7.2% 985920 ± 6% +7.4% 987077 ± 5% +14.2% 1049473 ± 2% vm-scalability.throughput
=========================================================================================
compiler/kconfig/rootfs/runtime/size/tbox_group/test/testcase:
gcc-13/defconfig/debian/300s/256G/qemu-vm/msync-mt/vm-scalability
commit:
6.6.0-rc4-mm-everything-2023-10-21-02-40+
6.6.0-rc4-split-folio-in-compaction+
6.6.0-rc4-folio-migration-in-compaction+
6.6.0-rc4-folio-migration-free-page-split+
6.6.0-rc4-folio-migration-free-page-split-sort-src+
6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f
---------------- --------------------------- --------------------------- --------------------------- ---------------------------
%stddev %change %stddev %change %stddev %change %stddev %change %stddev
\ | \ | \ | \ | \
2398063 +2.0% 2446844 -1.5% 2361096 ± 2% -3.1% 2324346 ± 4% -1.0% 2375055 ± 4% vm-scalability.throughput
=========================================================================================
compiler/kconfig/rootfs/runtime/size/tbox_group/test/testcase:
gcc-13/defconfig/debian/300s/128G/qemu-vm/truncate/vm-scalability
commit:
6.6.0-rc4-mm-everything-2023-10-21-02-40+
6.6.0-rc4-split-folio-in-compaction+
6.6.0-rc4-folio-migration-in-compaction+
6.6.0-rc4-folio-migration-free-page-split+
6.6.0-rc4-folio-migration-free-page-split-sort-src+
6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f
---------------- --------------------------- --------------------------- --------------------------- ---------------------------
%stddev %change %stddev %change %stddev %change %stddev %change %stddev
\ | \ | \ | \ | \
2.084e+08 ± 2% +1.5% 2.114e+08 ± 3% +6.5% 2.22e+08 ± 3% -2.3% 2.035e+08 ± 2% -2.1% 2.04e+08 ± 5% vm-scalability.throughput
[1] https://lore.kernel.org/linux-mm/f8d47176-03a8-99bf-a813-b5942830fd73@arm.com/
Zi Yan (4):
mm/compaction: enable compacting >0 order folios.
mm/compaction: add support for >0 order folio memory compaction.
mm/compaction: optimize >0 order folio compaction with free page
split.
mm/compaction: optimize >0 order folio compaction by sorting source
pages.
mm/compaction.c | 280 ++++++++++++++++++++++++++++++++++++------------
mm/internal.h | 7 +-
2 files changed, 217 insertions(+), 70 deletions(-)
--
2.42.0
^ permalink raw reply [flat|nested] 21+ messages in thread
* [PATCH v1 1/4] mm/compaction: enable compacting >0 order folios.
2023-11-13 17:01 [PATCH v1 0/4] Enable >0 order folio memory compaction Zi Yan
@ 2023-11-13 17:01 ` Zi Yan
2023-11-13 18:30 ` Matthew Wilcox
2023-11-20 9:18 ` Baolin Wang
2023-11-13 17:01 ` [PATCH v1 2/4] mm/compaction: add support for >0 order folio memory compaction Zi Yan
` (4 subsequent siblings)
5 siblings, 2 replies; 21+ messages in thread
From: Zi Yan @ 2023-11-13 17:01 UTC (permalink / raw)
To: linux-mm, linux-kernel
Cc: Zi Yan, Huang, Ying, Ryan Roberts, Andrew Morton,
Matthew Wilcox (Oracle),
David Hildenbrand, Yin, Fengwei, Yu Zhao, Vlastimil Babka,
Kirill A . Shutemov, Johannes Weiner, Baolin Wang, Kemeng Shi,
Mel Gorman, Rohan Puri, Mcgrof Chamberlain, Adam Manzanares,
Vishal Moola (Oracle)
From: Zi Yan <ziy@nvidia.com>
migrate_pages() supports >0 order folio migration and during compaction,
even if compaction_alloc() cannot provide >0 order free pages,
migrate_pages() can split the source page and try to migrate the base pages
from the split. It can be a baseline and start point for adding support for
compacting >0 order folios.
Suggested-by: Huang Ying <ying.huang@intel.com>
Signed-off-by: Zi Yan <ziy@nvidia.com>
---
mm/compaction.c | 57 ++++++++++++++++++++++++++++++++++++-------------
1 file changed, 42 insertions(+), 15 deletions(-)
diff --git a/mm/compaction.c b/mm/compaction.c
index 01ba298739dd..5217dd35b493 100644
--- a/mm/compaction.c
+++ b/mm/compaction.c
@@ -816,6 +816,21 @@ static bool too_many_isolated(struct compact_control *cc)
return too_many;
}
+/*
+ * 1. if the page order is larger than or equal to target_order (i.e.,
+ * cc->order and when it is not -1 for global compaction), skip it since
+ * target_order already indicates no free page with larger than target_order
+ * exists and later migrating it will most likely fail;
+ *
+ * 2. compacting > pageblock_order pages does not improve memory fragmentation,
+ * skip them;
+ */
+static bool skip_isolation_on_order(int order, int target_order)
+{
+ return (target_order != -1 && order >= target_order) ||
+ order >= pageblock_order;
+}
+
/**
* isolate_migratepages_block() - isolate all migrate-able pages within
* a single pageblock
@@ -1009,7 +1024,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,
/*
* Regardless of being on LRU, compound pages such as THP and
* hugetlbfs are not to be compacted unless we are attempting
- * an allocation much larger than the huge page size (eg CMA).
+ * an allocation larger than the compound page size.
* We can potentially save a lot of iterations if we skip them
* at once. The check is racy, but we can consider only valid
* values and the only danger is skipping too much.
@@ -1017,11 +1032,18 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,
if (PageCompound(page) && !cc->alloc_contig) {
const unsigned int order = compound_order(page);
- if (likely(order <= MAX_ORDER)) {
- low_pfn += (1UL << order) - 1;
- nr_scanned += (1UL << order) - 1;
+ /*
+ * Skip based on page order and compaction target order
+ * and skip hugetlbfs pages.
+ */
+ if (skip_isolation_on_order(order, cc->order) ||
+ PageHuge(page)) {
+ if (order <= MAX_ORDER) {
+ low_pfn += (1UL << order) - 1;
+ nr_scanned += (1UL << order) - 1;
+ }
+ goto isolate_fail;
}
- goto isolate_fail;
}
/*
@@ -1144,17 +1166,18 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,
goto isolate_abort;
}
}
+ }
- /*
- * folio become large since the non-locked check,
- * and it's on LRU.
- */
- if (unlikely(folio_test_large(folio) && !cc->alloc_contig)) {
- low_pfn += folio_nr_pages(folio) - 1;
- nr_scanned += folio_nr_pages(folio) - 1;
- folio_set_lru(folio);
- goto isolate_fail_put;
- }
+ /*
+ * Check LRU folio order under the lock
+ */
+ if (unlikely(skip_isolation_on_order(folio_order(folio),
+ cc->order) &&
+ !cc->alloc_contig)) {
+ low_pfn += folio_nr_pages(folio) - 1;
+ nr_scanned += folio_nr_pages(folio) - 1;
+ folio_set_lru(folio);
+ goto isolate_fail_put;
}
/* The folio is taken off the LRU */
@@ -1764,6 +1787,10 @@ static struct folio *compaction_alloc(struct folio *src, unsigned long data)
struct compact_control *cc = (struct compact_control *)data;
struct folio *dst;
+ /* this makes migrate_pages() split the source page and retry */
+ if (folio_order(src) > 0)
+ return NULL;
+
if (list_empty(&cc->freepages)) {
isolate_freepages(cc);
--
2.42.0
^ permalink raw reply related [flat|nested] 21+ messages in thread
* [PATCH v1 2/4] mm/compaction: add support for >0 order folio memory compaction.
2023-11-13 17:01 [PATCH v1 0/4] Enable >0 order folio memory compaction Zi Yan
2023-11-13 17:01 ` [PATCH v1 1/4] mm/compaction: enable compacting >0 order folios Zi Yan
@ 2023-11-13 17:01 ` Zi Yan
2024-01-09 15:18 ` Ryan Roberts
2023-11-13 17:01 ` [PATCH v1 3/4] mm/compaction: optimize >0 order folio compaction with free page split Zi Yan
` (3 subsequent siblings)
5 siblings, 1 reply; 21+ messages in thread
From: Zi Yan @ 2023-11-13 17:01 UTC (permalink / raw)
To: linux-mm, linux-kernel
Cc: Zi Yan, Huang, Ying, Ryan Roberts, Andrew Morton,
Matthew Wilcox (Oracle),
David Hildenbrand, Yin, Fengwei, Yu Zhao, Vlastimil Babka,
Kirill A . Shutemov, Johannes Weiner, Baolin Wang, Kemeng Shi,
Mel Gorman, Rohan Puri, Mcgrof Chamberlain, Adam Manzanares,
Vishal Moola (Oracle)
From: Zi Yan <ziy@nvidia.com>
Before, memory compaction only migrates order-0 folios and skips >0 order
folios. This commit adds support for >0 order folio compaction by keeping
isolated free pages at their original size without splitting them into
order-0 pages and using them directly during migration process.
What is different from the prior implementation:
1. All isolated free pages are kept in a MAX_ORDER+1 array of page lists,
where each page list stores free pages in the same order.
2. All free pages are not post_alloc_hook() processed nor buddy pages,
although their orders are stored in first page's private like buddy
pages.
3. During migration, in new page allocation time (i.e., in
compaction_alloc()), free pages are then processed by post_alloc_hook().
When migration fails and a new page is returned (i.e., in
compaction_free()), free pages are restored by reversing the
post_alloc_hook() operations.
Step 3 is done for a latter optimization that splitting and/or merging free
pages during compaction becomes easier.
Signed-off-by: Zi Yan <ziy@nvidia.com>
---
mm/compaction.c | 160 ++++++++++++++++++++++++++++++------------------
mm/internal.h | 7 ++-
2 files changed, 108 insertions(+), 59 deletions(-)
diff --git a/mm/compaction.c b/mm/compaction.c
index 5217dd35b493..ec6b5cc7e907 100644
--- a/mm/compaction.c
+++ b/mm/compaction.c
@@ -66,45 +66,64 @@ static inline void count_compact_events(enum vm_event_item item, long delta)
#define COMPACTION_HPAGE_ORDER (PMD_SHIFT - PAGE_SHIFT)
#endif
-static unsigned long release_freepages(struct list_head *freelist)
+static void init_page_list(struct page_list *p)
{
- struct page *page, *next;
- unsigned long high_pfn = 0;
-
- list_for_each_entry_safe(page, next, freelist, lru) {
- unsigned long pfn = page_to_pfn(page);
- list_del(&page->lru);
- __free_page(page);
- if (pfn > high_pfn)
- high_pfn = pfn;
- }
-
- return high_pfn;
+ INIT_LIST_HEAD(&p->pages);
+ p->nr_pages = 0;
}
-static void split_map_pages(struct list_head *list)
+static void split_map_pages(struct page_list *freepages)
{
unsigned int i, order, nr_pages;
struct page *page, *next;
LIST_HEAD(tmp_list);
- list_for_each_entry_safe(page, next, list, lru) {
- list_del(&page->lru);
+ for (order = 0; order <= MAX_ORDER; order++) {
+ freepages[order].nr_pages = 0;
+
+ list_for_each_entry_safe(page, next, &freepages[order].pages, lru) {
+ list_del(&page->lru);
- order = page_private(page);
- nr_pages = 1 << order;
+ nr_pages = 1 << order;
- post_alloc_hook(page, order, __GFP_MOVABLE);
- if (order)
- split_page(page, order);
+ post_alloc_hook(page, order, __GFP_MOVABLE);
+ if (order)
+ split_page(page, order);
- for (i = 0; i < nr_pages; i++) {
- list_add(&page->lru, &tmp_list);
- page++;
+ for (i = 0; i < nr_pages; i++) {
+ list_add(&page->lru, &tmp_list);
+ page++;
+ }
+ freepages[order].nr_pages += nr_pages;
}
+ list_splice(&tmp_list, &freepages[order].pages);
}
+}
- list_splice(&tmp_list, list);
+static unsigned long release_free_list(struct page_list *freepages)
+{
+ int order;
+ unsigned long high_pfn = 0;
+
+ for (order = 0; order <= MAX_ORDER; order++) {
+ struct page *page, *next;
+
+ list_for_each_entry_safe(page, next, &freepages[order].pages, lru) {
+ unsigned long pfn = page_to_pfn(page);
+
+ list_del(&page->lru);
+ /*
+ * Convert free pages into post allocation pages, so
+ * that we can free them via __free_page.
+ */
+ post_alloc_hook(page, order, __GFP_MOVABLE);
+ __free_pages(page, order);
+ if (pfn > high_pfn)
+ high_pfn = pfn;
+ }
+ freepages[order].nr_pages = 0;
+ }
+ return high_pfn;
}
#ifdef CONFIG_COMPACTION
@@ -583,7 +602,7 @@ static bool compact_unlock_should_abort(spinlock_t *lock,
static unsigned long isolate_freepages_block(struct compact_control *cc,
unsigned long *start_pfn,
unsigned long end_pfn,
- struct list_head *freelist,
+ struct page_list *freelist,
unsigned int stride,
bool strict)
{
@@ -657,7 +676,8 @@ static unsigned long isolate_freepages_block(struct compact_control *cc,
nr_scanned += isolated - 1;
total_isolated += isolated;
cc->nr_freepages += isolated;
- list_add_tail(&page->lru, freelist);
+ list_add_tail(&page->lru, &freelist[order].pages);
+ freelist[order].nr_pages++;
if (!strict && cc->nr_migratepages <= cc->nr_freepages) {
blockpfn += isolated;
@@ -722,7 +742,11 @@ isolate_freepages_range(struct compact_control *cc,
unsigned long start_pfn, unsigned long end_pfn)
{
unsigned long isolated, pfn, block_start_pfn, block_end_pfn;
- LIST_HEAD(freelist);
+ int order;
+ struct page_list tmp_freepages[MAX_ORDER + 1];
+
+ for (order = 0; order <= MAX_ORDER; order++)
+ init_page_list(&tmp_freepages[order]);
pfn = start_pfn;
block_start_pfn = pageblock_start_pfn(pfn);
@@ -753,7 +777,7 @@ isolate_freepages_range(struct compact_control *cc,
break;
isolated = isolate_freepages_block(cc, &isolate_start_pfn,
- block_end_pfn, &freelist, 0, true);
+ block_end_pfn, tmp_freepages, 0, true);
/*
* In strict mode, isolate_freepages_block() returns 0 if
@@ -770,15 +794,15 @@ isolate_freepages_range(struct compact_control *cc,
*/
}
- /* __isolate_free_page() does not map the pages */
- split_map_pages(&freelist);
-
if (pfn < end_pfn) {
/* Loop terminated early, cleanup. */
- release_freepages(&freelist);
+ release_free_list(tmp_freepages);
return 0;
}
+ /* __isolate_free_page() does not map the pages */
+ split_map_pages(tmp_freepages);
+
/* We don't use freelists for anything. */
return pfn;
}
@@ -1462,7 +1486,7 @@ fast_isolate_around(struct compact_control *cc, unsigned long pfn)
if (!page)
return;
- isolate_freepages_block(cc, &start_pfn, end_pfn, &cc->freepages, 1, false);
+ isolate_freepages_block(cc, &start_pfn, end_pfn, cc->freepages, 1, false);
/* Skip this pageblock in the future as it's full or nearly full */
if (start_pfn == end_pfn && !cc->no_set_skip_hint)
@@ -1591,7 +1615,7 @@ static void fast_isolate_freepages(struct compact_control *cc)
nr_scanned += nr_isolated - 1;
total_isolated += nr_isolated;
cc->nr_freepages += nr_isolated;
- list_add_tail(&page->lru, &cc->freepages);
+ list_add_tail(&page->lru, &cc->freepages[order].pages);
count_compact_events(COMPACTISOLATED, nr_isolated);
} else {
/* If isolation fails, abort the search */
@@ -1665,13 +1689,12 @@ static void isolate_freepages(struct compact_control *cc)
unsigned long isolate_start_pfn; /* exact pfn we start at */
unsigned long block_end_pfn; /* end of current pageblock */
unsigned long low_pfn; /* lowest pfn scanner is able to scan */
- struct list_head *freelist = &cc->freepages;
unsigned int stride;
/* Try a small search of the free lists for a candidate */
fast_isolate_freepages(cc);
if (cc->nr_freepages)
- goto splitmap;
+ return;
/*
* Initialise the free scanner. The starting point is where we last
@@ -1731,7 +1754,7 @@ static void isolate_freepages(struct compact_control *cc)
/* Found a block suitable for isolating free pages from. */
nr_isolated = isolate_freepages_block(cc, &isolate_start_pfn,
- block_end_pfn, freelist, stride, false);
+ block_end_pfn, cc->freepages, stride, false);
/* Update the skip hint if the full pageblock was scanned */
if (isolate_start_pfn == block_end_pfn)
@@ -1772,10 +1795,6 @@ static void isolate_freepages(struct compact_control *cc)
* and the loop terminated due to isolate_start_pfn < low_pfn
*/
cc->free_pfn = isolate_start_pfn;
-
-splitmap:
- /* __isolate_free_page() does not map the pages */
- split_map_pages(freelist);
}
/*
@@ -1786,23 +1805,22 @@ static struct folio *compaction_alloc(struct folio *src, unsigned long data)
{
struct compact_control *cc = (struct compact_control *)data;
struct folio *dst;
+ int order = folio_order(src);
- /* this makes migrate_pages() split the source page and retry */
- if (folio_order(src) > 0)
- return NULL;
-
- if (list_empty(&cc->freepages)) {
+ if (!cc->freepages[order].nr_pages) {
isolate_freepages(cc);
-
- if (list_empty(&cc->freepages))
+ if (!cc->freepages[order].nr_pages)
return NULL;
}
- dst = list_entry(cc->freepages.next, struct folio, lru);
+ dst = list_first_entry(&cc->freepages[order].pages, struct folio, lru);
+ cc->freepages[order].nr_pages--;
list_del(&dst->lru);
- cc->nr_freepages--;
-
- return dst;
+ post_alloc_hook(&dst->page, order, __GFP_MOVABLE);
+ if (order)
+ prep_compound_page(&dst->page, order);
+ cc->nr_freepages -= 1 << order;
+ return page_rmappable_folio(&dst->page);
}
/*
@@ -1813,9 +1831,34 @@ static struct folio *compaction_alloc(struct folio *src, unsigned long data)
static void compaction_free(struct folio *dst, unsigned long data)
{
struct compact_control *cc = (struct compact_control *)data;
+ int order = folio_order(dst);
+ struct page *page = &dst->page;
+
+ if (order) {
+ int i;
- list_add(&dst->lru, &cc->freepages);
- cc->nr_freepages++;
+ page[1].flags &= ~PAGE_FLAGS_SECOND;
+ for (i = 1; i < (1 << order); i++) {
+ page[i].mapping = NULL;
+ clear_compound_head(&page[i]);
+ page[i].flags &= ~PAGE_FLAGS_CHECK_AT_PREP;
+ }
+
+ }
+ /* revert post_alloc_hook() operations */
+ page->mapping = NULL;
+ page->flags &= ~PAGE_FLAGS_CHECK_AT_PREP;
+ set_page_count(page, 0);
+ page_mapcount_reset(page);
+ reset_page_owner(page, order);
+ page_table_check_free(page, order);
+ arch_free_page(page, order);
+ set_page_private(page, order);
+ INIT_LIST_HEAD(&dst->lru);
+
+ list_add(&dst->lru, &cc->freepages[order].pages);
+ cc->freepages[order].nr_pages++;
+ cc->nr_freepages += 1 << order;
}
/* possible outcome of isolate_migratepages */
@@ -2439,6 +2482,7 @@ compact_zone(struct compact_control *cc, struct capture_control *capc)
const bool sync = cc->mode != MIGRATE_ASYNC;
bool update_cached;
unsigned int nr_succeeded = 0;
+ int order;
/*
* These counters track activities during zone compaction. Initialize
@@ -2448,7 +2492,8 @@ compact_zone(struct compact_control *cc, struct capture_control *capc)
cc->total_free_scanned = 0;
cc->nr_migratepages = 0;
cc->nr_freepages = 0;
- INIT_LIST_HEAD(&cc->freepages);
+ for (order = 0; order <= MAX_ORDER; order++)
+ init_page_list(&cc->freepages[order]);
INIT_LIST_HEAD(&cc->migratepages);
cc->migratetype = gfp_migratetype(cc->gfp_mask);
@@ -2634,7 +2679,7 @@ compact_zone(struct compact_control *cc, struct capture_control *capc)
* so we don't leave any returned pages behind in the next attempt.
*/
if (cc->nr_freepages > 0) {
- unsigned long free_pfn = release_freepages(&cc->freepages);
+ unsigned long free_pfn = release_free_list(cc->freepages);
cc->nr_freepages = 0;
VM_BUG_ON(free_pfn == 0);
@@ -2653,7 +2698,6 @@ compact_zone(struct compact_control *cc, struct capture_control *capc)
trace_mm_compaction_end(cc, start_pfn, end_pfn, sync, ret);
- VM_BUG_ON(!list_empty(&cc->freepages));
VM_BUG_ON(!list_empty(&cc->migratepages));
return ret;
diff --git a/mm/internal.h b/mm/internal.h
index 8450562744cf..46c8bb94ecbb 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -473,6 +473,11 @@ int split_free_page(struct page *free_page,
/*
* in mm/compaction.c
*/
+
+struct page_list {
+ struct list_head pages;
+ unsigned long nr_pages;
+};
/*
* compact_control is used to track pages being migrated and the free pages
* they are being migrated to during memory compaction. The free_pfn starts
@@ -481,7 +486,7 @@ int split_free_page(struct page *free_page,
* completes when free_pfn <= migrate_pfn
*/
struct compact_control {
- struct list_head freepages; /* List of free pages to migrate to */
+ struct page_list freepages[MAX_ORDER + 1]; /* List of free pages to migrate to */
struct list_head migratepages; /* List of pages being migrated */
unsigned int nr_freepages; /* Number of isolated free pages */
unsigned int nr_migratepages; /* Number of pages to migrate */
--
2.42.0
^ permalink raw reply related [flat|nested] 21+ messages in thread
* [PATCH v1 3/4] mm/compaction: optimize >0 order folio compaction with free page split.
2023-11-13 17:01 [PATCH v1 0/4] Enable >0 order folio memory compaction Zi Yan
2023-11-13 17:01 ` [PATCH v1 1/4] mm/compaction: enable compacting >0 order folios Zi Yan
2023-11-13 17:01 ` [PATCH v1 2/4] mm/compaction: add support for >0 order folio memory compaction Zi Yan
@ 2023-11-13 17:01 ` Zi Yan
2023-11-22 10:26 ` Ryan Roberts
2023-11-13 17:01 ` [PATCH v1 4/4] mm/compaction: optimize >0 order folio compaction by sorting source pages Zi Yan
` (2 subsequent siblings)
5 siblings, 1 reply; 21+ messages in thread
From: Zi Yan @ 2023-11-13 17:01 UTC (permalink / raw)
To: linux-mm, linux-kernel
Cc: Zi Yan, Huang, Ying, Ryan Roberts, Andrew Morton,
Matthew Wilcox (Oracle),
David Hildenbrand, Yin, Fengwei, Yu Zhao, Vlastimil Babka,
Kirill A . Shutemov, Johannes Weiner, Baolin Wang, Kemeng Shi,
Mel Gorman, Rohan Puri, Mcgrof Chamberlain, Adam Manzanares,
Vishal Moola (Oracle)
From: Zi Yan <ziy@nvidia.com>
During migration in a memory compaction, free pages are placed in an array
of page lists based on their order. But the desired free page order (i.e.,
the order of a source page) might not be always present, thus leading to
migration failures. Split a high order free pages when source migration
page has a lower order to increase migration successful rate.
Note: merging free pages when a migration fails and a lower order free
page is returned via compaction_free() is possible, but there is too much
work. Since the free pages are not buddy pages, it is hard to identify
these free pages using existing PFN-based page merging algorithm.
Signed-off-by: Zi Yan <ziy@nvidia.com>
---
mm/compaction.c | 40 +++++++++++++++++++++++++++++++++++++++-
1 file changed, 39 insertions(+), 1 deletion(-)
diff --git a/mm/compaction.c b/mm/compaction.c
index ec6b5cc7e907..9c083e6b399a 100644
--- a/mm/compaction.c
+++ b/mm/compaction.c
@@ -1806,9 +1806,46 @@ static struct folio *compaction_alloc(struct folio *src, unsigned long data)
struct compact_control *cc = (struct compact_control *)data;
struct folio *dst;
int order = folio_order(src);
+ bool has_isolated_pages = false;
+again:
if (!cc->freepages[order].nr_pages) {
- isolate_freepages(cc);
+ int i;
+
+ for (i = order + 1; i <= MAX_ORDER; i++) {
+ if (cc->freepages[i].nr_pages) {
+ struct page *freepage =
+ list_first_entry(&cc->freepages[i].pages,
+ struct page, lru);
+
+ int start_order = i;
+ unsigned long size = 1 << start_order;
+
+ list_del(&freepage->lru);
+ cc->freepages[i].nr_pages--;
+
+ while (start_order > order) {
+ start_order--;
+ size >>= 1;
+
+ list_add(&freepage[size].lru,
+ &cc->freepages[start_order].pages);
+ cc->freepages[start_order].nr_pages++;
+ set_page_private(&freepage[size], start_order);
+ }
+ post_alloc_hook(freepage, order, __GFP_MOVABLE);
+ if (order)
+ prep_compound_page(freepage, order);
+ dst = page_folio(freepage);
+ goto done;
+ }
+ }
+ if (!has_isolated_pages) {
+ isolate_freepages(cc);
+ has_isolated_pages = true;
+ goto again;
+ }
+
if (!cc->freepages[order].nr_pages)
return NULL;
}
@@ -1819,6 +1856,7 @@ static struct folio *compaction_alloc(struct folio *src, unsigned long data)
post_alloc_hook(&dst->page, order, __GFP_MOVABLE);
if (order)
prep_compound_page(&dst->page, order);
+done:
cc->nr_freepages -= 1 << order;
return page_rmappable_folio(&dst->page);
}
--
2.42.0
^ permalink raw reply related [flat|nested] 21+ messages in thread
* [PATCH v1 4/4] mm/compaction: optimize >0 order folio compaction by sorting source pages.
2023-11-13 17:01 [PATCH v1 0/4] Enable >0 order folio memory compaction Zi Yan
` (2 preceding siblings ...)
2023-11-13 17:01 ` [PATCH v1 3/4] mm/compaction: optimize >0 order folio compaction with free page split Zi Yan
@ 2023-11-13 17:01 ` Zi Yan
2023-11-21 15:46 ` [PATCH v1 0/4] Enable >0 order folio memory compaction Ryan Roberts
2023-11-24 14:58 ` Ryan Roberts
5 siblings, 0 replies; 21+ messages in thread
From: Zi Yan @ 2023-11-13 17:01 UTC (permalink / raw)
To: linux-mm, linux-kernel
Cc: Zi Yan, Huang, Ying, Ryan Roberts, Andrew Morton,
Matthew Wilcox (Oracle),
David Hildenbrand, Yin, Fengwei, Yu Zhao, Vlastimil Babka,
Kirill A . Shutemov, Johannes Weiner, Baolin Wang, Kemeng Shi,
Mel Gorman, Rohan Puri, Mcgrof Chamberlain, Adam Manzanares,
Vishal Moola (Oracle)
From: Zi Yan <ziy@nvidia.com>
It should maximize high order free page use and minimize free page splits.
It might be useful before free page merging is implemented.
Signed-off-by: Zi Yan <ziy@nvidia.com>
---
mm/compaction.c | 33 +++++++++++++++++++++++++++++++++
1 file changed, 33 insertions(+)
diff --git a/mm/compaction.c b/mm/compaction.c
index 9c083e6b399a..91809bee5422 100644
--- a/mm/compaction.c
+++ b/mm/compaction.c
@@ -127,6 +127,37 @@ static unsigned long release_free_list(struct page_list *freepages)
}
#ifdef CONFIG_COMPACTION
+
+static void sort_folios_by_order(struct list_head *pages)
+{
+ struct page_list page_list[MAX_ORDER + 1];
+ int order;
+ struct folio *folio, *next;
+
+ for (order = 0; order <= MAX_ORDER; order++)
+ init_page_list(&page_list[order]);
+
+ list_for_each_entry_safe(folio, next, pages, lru) {
+ order = folio_order(folio);
+
+ if (order > MAX_ORDER)
+ continue;
+
+ list_move(&folio->lru, &page_list[order].pages);
+ page_list[order].nr_pages++;
+ }
+
+ for (order = MAX_ORDER; order >= 0; order--) {
+ if (page_list[order].nr_pages) {
+
+ list_for_each_entry_safe(folio, next,
+ &page_list[order].pages, lru) {
+ list_move_tail(&folio->lru, pages);
+ }
+ }
+ }
+}
+
bool PageMovable(struct page *page)
{
const struct movable_operations *mops;
@@ -2639,6 +2670,8 @@ compact_zone(struct compact_control *cc, struct capture_control *capc)
pageblock_start_pfn(cc->migrate_pfn - 1));
}
+ sort_folios_by_order(&cc->migratepages);
+
err = migrate_pages(&cc->migratepages, compaction_alloc,
compaction_free, (unsigned long)cc, cc->mode,
MR_COMPACTION, &nr_succeeded);
--
2.42.0
^ permalink raw reply related [flat|nested] 21+ messages in thread
* Re: [PATCH v1 1/4] mm/compaction: enable compacting >0 order folios.
2023-11-13 17:01 ` [PATCH v1 1/4] mm/compaction: enable compacting >0 order folios Zi Yan
@ 2023-11-13 18:30 ` Matthew Wilcox
2023-11-13 19:22 ` Zi Yan
2023-11-20 9:18 ` Baolin Wang
1 sibling, 1 reply; 21+ messages in thread
From: Matthew Wilcox @ 2023-11-13 18:30 UTC (permalink / raw)
To: Zi Yan
Cc: linux-mm, linux-kernel, Huang, Ying, Ryan Roberts, Andrew Morton,
David Hildenbrand, Yin, Fengwei, Yu Zhao, Vlastimil Babka,
Kirill A . Shutemov, Johannes Weiner, Baolin Wang, Kemeng Shi,
Mel Gorman, Rohan Puri, Mcgrof Chamberlain, Adam Manzanares,
Vishal Moola (Oracle)
On Mon, Nov 13, 2023 at 12:01:54PM -0500, Zi Yan wrote:
> + /* this makes migrate_pages() split the source page and retry */
> + if (folio_order(src) > 0)
> + return NULL;
Nit: folio_test_large() is more efficient than folio_order() > 0.
The former simply tests the bit, while the second tests the bit, then
loads folio->_order to check it's >0. We know it will be, but there's
no way to tell gcc that if the bit is set, this value is definitely not 0.
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v1 1/4] mm/compaction: enable compacting >0 order folios.
2023-11-13 18:30 ` Matthew Wilcox
@ 2023-11-13 19:22 ` Zi Yan
0 siblings, 0 replies; 21+ messages in thread
From: Zi Yan @ 2023-11-13 19:22 UTC (permalink / raw)
To: Matthew Wilcox
Cc: linux-mm, linux-kernel, "Huang, Ying",
Ryan Roberts, Andrew Morton, David Hildenbrand, "Yin,
Fengwei",
Yu Zhao, Vlastimil Babka, "Kirill A . Shutemov",
Johannes Weiner, Baolin Wang, Kemeng Shi, Mel Gorman, Rohan Puri,
Mcgrof Chamberlain, Adam Manzanares,
"Vishal Moola (Oracle)"
[-- Attachment #1: Type: text/plain, Size: 606 bytes --]
On 13 Nov 2023, at 13:30, Matthew Wilcox wrote:
> On Mon, Nov 13, 2023 at 12:01:54PM -0500, Zi Yan wrote:
>> + /* this makes migrate_pages() split the source page and retry */
>> + if (folio_order(src) > 0)
>> + return NULL;
>
> Nit: folio_test_large() is more efficient than folio_order() > 0.
> The former simply tests the bit, while the second tests the bit, then
> loads folio->_order to check it's >0. We know it will be, but there's
> no way to tell gcc that if the bit is set, this value is definitely not 0.
Got it. Make sense. Will change it in next version. Thanks.
--
Best Regards,
Yan, Zi
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 854 bytes --]
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v1 1/4] mm/compaction: enable compacting >0 order folios.
2023-11-13 17:01 ` [PATCH v1 1/4] mm/compaction: enable compacting >0 order folios Zi Yan
2023-11-13 18:30 ` Matthew Wilcox
@ 2023-11-20 9:18 ` Baolin Wang
2023-11-20 14:05 ` Zi Yan
1 sibling, 1 reply; 21+ messages in thread
From: Baolin Wang @ 2023-11-20 9:18 UTC (permalink / raw)
To: Zi Yan, linux-mm, linux-kernel
Cc: Huang, Ying, Ryan Roberts, Andrew Morton, Matthew Wilcox (Oracle),
David Hildenbrand, Yin, Fengwei, Yu Zhao, Vlastimil Babka,
Kirill A . Shutemov, Johannes Weiner, Kemeng Shi, Mel Gorman,
Rohan Puri, Mcgrof Chamberlain, Adam Manzanares,
Vishal Moola (Oracle)
On 11/14/2023 1:01 AM, Zi Yan wrote:
> From: Zi Yan <ziy@nvidia.com>
>
> migrate_pages() supports >0 order folio migration and during compaction,
> even if compaction_alloc() cannot provide >0 order free pages,
> migrate_pages() can split the source page and try to migrate the base pages
> from the split. It can be a baseline and start point for adding support for
> compacting >0 order folios.
>
> Suggested-by: Huang Ying <ying.huang@intel.com>
> Signed-off-by: Zi Yan <ziy@nvidia.com>
> ---
> mm/compaction.c | 57 ++++++++++++++++++++++++++++++++++++-------------
> 1 file changed, 42 insertions(+), 15 deletions(-)
>
> diff --git a/mm/compaction.c b/mm/compaction.c
> index 01ba298739dd..5217dd35b493 100644
> --- a/mm/compaction.c
> +++ b/mm/compaction.c
> @@ -816,6 +816,21 @@ static bool too_many_isolated(struct compact_control *cc)
> return too_many;
> }
>
> +/*
> + * 1. if the page order is larger than or equal to target_order (i.e.,
> + * cc->order and when it is not -1 for global compaction), skip it since
> + * target_order already indicates no free page with larger than target_order
> + * exists and later migrating it will most likely fail;
> + *
> + * 2. compacting > pageblock_order pages does not improve memory fragmentation,
> + * skip them;
> + */
> +static bool skip_isolation_on_order(int order, int target_order)
> +{
> + return (target_order != -1 && order >= target_order) ||
> + order >= pageblock_order;
> +}
> +
> /**
> * isolate_migratepages_block() - isolate all migrate-able pages within
> * a single pageblock
> @@ -1009,7 +1024,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,
> /*
> * Regardless of being on LRU, compound pages such as THP and
> * hugetlbfs are not to be compacted unless we are attempting
> - * an allocation much larger than the huge page size (eg CMA).
> + * an allocation larger than the compound page size.
> * We can potentially save a lot of iterations if we skip them
> * at once. The check is racy, but we can consider only valid
> * values and the only danger is skipping too much.
> @@ -1017,11 +1032,18 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,
> if (PageCompound(page) && !cc->alloc_contig) {
> const unsigned int order = compound_order(page);
>
> - if (likely(order <= MAX_ORDER)) {
> - low_pfn += (1UL << order) - 1;
> - nr_scanned += (1UL << order) - 1;
> + /*
> + * Skip based on page order and compaction target order
> + * and skip hugetlbfs pages.
> + */
> + if (skip_isolation_on_order(order, cc->order) ||
> + PageHuge(page)) {
> + if (order <= MAX_ORDER) {
> + low_pfn += (1UL << order) - 1;
> + nr_scanned += (1UL << order) - 1;
> + }
> + goto isolate_fail;
> }
> - goto isolate_fail;
> }
>
> /*
> @@ -1144,17 +1166,18 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,
> goto isolate_abort;
> }
> }
> + }
>
> - /*
> - * folio become large since the non-locked check,
> - * and it's on LRU.
> - */
> - if (unlikely(folio_test_large(folio) && !cc->alloc_contig)) {
> - low_pfn += folio_nr_pages(folio) - 1;
> - nr_scanned += folio_nr_pages(folio) - 1;
> - folio_set_lru(folio);
> - goto isolate_fail_put;
> - }
> + /*
> + * Check LRU folio order under the lock
> + */
> + if (unlikely(skip_isolation_on_order(folio_order(folio),
> + cc->order) &&
> + !cc->alloc_contig)) {
> + low_pfn += folio_nr_pages(folio) - 1;
> + nr_scanned += folio_nr_pages(folio) - 1;
> + folio_set_lru(folio);
> + goto isolate_fail_put;
> }
Why was this part moved out of the 'if (lruvec != locked)' block? If we
hold the lru lock, then we do not need to check again, right?
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v1 1/4] mm/compaction: enable compacting >0 order folios.
2023-11-20 9:18 ` Baolin Wang
@ 2023-11-20 14:05 ` Zi Yan
0 siblings, 0 replies; 21+ messages in thread
From: Zi Yan @ 2023-11-20 14:05 UTC (permalink / raw)
To: Baolin Wang
Cc: linux-mm, linux-kernel, "Huang, Ying",
Ryan Roberts, Andrew Morton, "Matthew Wilcox (Oracle)",
David Hildenbrand, "Yin, Fengwei",
Yu Zhao, Vlastimil Babka, "Kirill A . Shutemov",
Johannes Weiner, Kemeng Shi, Mel Gorman, Rohan Puri,
Mcgrof Chamberlain, Adam Manzanares,
"Vishal Moola (Oracle)"
[-- Attachment #1: Type: text/plain, Size: 1163 bytes --]
>> @@ -1144,17 +1166,18 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,
>> goto isolate_abort;
>> }
>> }
>> + }
>> - /*
>> - * folio become large since the non-locked check,
>> - * and it's on LRU.
>> - */
>> - if (unlikely(folio_test_large(folio) && !cc->alloc_contig)) {
>> - low_pfn += folio_nr_pages(folio) - 1;
>> - nr_scanned += folio_nr_pages(folio) - 1;
>> - folio_set_lru(folio);
>> - goto isolate_fail_put;
>> - }
>> + /*
>> + * Check LRU folio order under the lock
>> + */
>> + if (unlikely(skip_isolation_on_order(folio_order(folio),
>> + cc->order) &&
>> + !cc->alloc_contig)) {
>> + low_pfn += folio_nr_pages(folio) - 1;
>> + nr_scanned += folio_nr_pages(folio) - 1;
>> + folio_set_lru(folio);
>> + goto isolate_fail_put;
>> }
>
> Why was this part moved out of the 'if (lruvec != locked)' block? If we hold the lru lock, then we do not need to check again, right?
Probably I messed this up during rebase. Thank you for pointing this out.
Will fix it in the next version.
--
Best Regards,
Yan, Zi
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 854 bytes --]
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v1 0/4] Enable >0 order folio memory compaction
2023-11-13 17:01 [PATCH v1 0/4] Enable >0 order folio memory compaction Zi Yan
` (3 preceding siblings ...)
2023-11-13 17:01 ` [PATCH v1 4/4] mm/compaction: optimize >0 order folio compaction by sorting source pages Zi Yan
@ 2023-11-21 15:46 ` Ryan Roberts
2023-11-21 16:45 ` Zi Yan
2023-11-24 14:58 ` Ryan Roberts
5 siblings, 1 reply; 21+ messages in thread
From: Ryan Roberts @ 2023-11-21 15:46 UTC (permalink / raw)
To: Zi Yan, linux-mm, linux-kernel
Cc: Huang, Ying, Andrew Morton, Matthew Wilcox (Oracle),
David Hildenbrand, Yin, Fengwei, Yu Zhao, Vlastimil Babka,
Kirill A . Shutemov, Johannes Weiner, Baolin Wang, Kemeng Shi,
Mel Gorman, Rohan Puri, Mcgrof Chamberlain, Adam Manzanares,
Vishal Moola (Oracle)
On 13/11/2023 17:01, Zi Yan wrote:
> From: Zi Yan <ziy@nvidia.com>
>
> Hi all,
>
> This patchset enables >0 order folio memory compaction, which is one of
> the prerequisitions for large folio support[1]. It is on top of
> mm-everything-2023-10-21-02-40 (also applies to mm-everything-2023-11-10-20-42
> without conflicts).
>
> Per patch vm-scalability results on a 8-CPU (Intel Xeon E5-2650 v4 @ 2.20GHz)
> 16G VM are at the bottom and based on mm-everything-2023-10-21-02-40.
> Each has 5 runs. Highlights:
>
> 1. mmap-pread-seq-mt: -18.3% if split folios during compaction and -7.7%
> if migrating folios during compaction.
>
> 2. lru-file-readtwice: +41.5% if split folios during compaction and
> +30.4% if migrating folios during compaction.
>
> 3. lru-file-mmap-read: +35.1% if split folios during compaction and
> +32.3% if migrating folios during compaction.
>
> I will run experiments on bare metal and report results later.
>
>
> Changelog
> ===
>
> From RFC:
> 1. Enabled >0 order folio compaction in the first patch by splitting all
> to-be-migrated folios. (per Huang, Ying)
>
> 2. Stopped isolating compound pages with order greater than cc->order
> to avoid wasting effort, since cc->order gives a hint that no free pages
> with order greater than it exist, thus migrating the compound pages will fail.
> (per Baolin Wang)
>
> 3. Retained the folio check within lru lock. (per Baolin Wang)
>
> 4. Made isolate_freepages_block() generate order-sorted multi lists.
> (per Johannes Weiner)
>
>
> Overview
> ===
>
> To support >0 order folio compaction, the patchset changes how free pages used
> for migration are kept during compaction. Free pages used to be split into
> order-0 pages that are post allocation processed (i.e., PageBuddy flag cleared,
> page order stored in page->private is zeroed, and page reference is set to 1).
> Now all free pages are kept in a MAX_ORDER+1 array of page lists based
> on their order without post allocation process. When migrate_pages() asks for
> a new page, one of the free pages, based on the requested page order, is
> then processed and given out.
>
>
> Optimizations
> ===
>
> 1. Free page split is added to increase migration success rate in case
> a source page does not have a matched free page in the free page lists.
> Free page merge is possible but not implemented, since existing
> PFN-based buddy page merge algorithm requires the identification of
> buddy pages, but free pages kept for memory compaction cannot have
> PageBuddy set to avoid confusing other PFN scanners.
>
> 2. Sort source pages in ascending order before migration is added to
> reduce free page split. Otherwise, high order free pages might be
> prematurely split, causing undesired high order folio migration failures.
>
>
> TODOs
> ===
>
> 1. Refactor free page post allocation and free page preparation code so
> that compaction_alloc() and compaction_free() can call functions instead
> of hard coding.
>
> 2. Make isolate_migratepages_block() produce a sorted list for
> compaction to avoid sort_folio_by_order() in Patch 4.
>
>
> Feel free to give comments and ask questions.
>
> Thanks.
>
> vm-scalability results
> ===
>
> =========================================================================================
> compiler/kconfig/rootfs/runtime/tbox_group/test/testcase:
> gcc-13/defconfig/debian/300s/qemu-vm/mmap-xread-seq-mt/vm-scalability
>
> commit:
> 6.6.0-rc4-mm-everything-2023-10-21-02-40+
> 6.6.0-rc4-split-folio-in-compaction+
> 6.6.0-rc4-folio-migration-in-compaction+
> 6.6.0-rc4-folio-migration-free-page-split+
> 6.6.0-rc4-folio-migration-free-page-split-sort-src+
>
> 6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f
> ---------------- --------------------------- --------------------------- --------------------------- ---------------------------
> %stddev %change %stddev %change %stddev %change %stddev %change %stddev
> \ | \ | \ | \ | \
> 12896955 +2.7% 13249322 -4.0% 12385175 ± 5% +1.1% 13033951 -0.4% 12845698 vm-scalability.throughput
Hi Zi,
Are you able to add any commentary to these results as I'm struggling to
interpret them; Is a positive or negative change better (are they times or
rates?). What are the stddev values? The title suggests percent but the values
are huge - I'm trying to understand what the error bars look like - are the
swings real or noise?
Thanks,
Ryan
>
> =========================================================================================
> compiler/kconfig/rootfs/runtime/tbox_group/test/testcase:
> gcc-13/defconfig/debian/300s/qemu-vm/mmap-pread-seq/vm-scalability
>
> commit:
> 6.6.0-rc4-mm-everything-2023-10-21-02-40+
> 6.6.0-rc4-split-folio-in-compaction+
> 6.6.0-rc4-folio-migration-in-compaction+
> 6.6.0-rc4-folio-migration-free-page-split+
> 6.6.0-rc4-folio-migration-free-page-split-sort-src+
>
> 6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f
> ---------------- --------------------------- --------------------------- --------------------------- ---------------------------
> %stddev %change %stddev %change %stddev %change %stddev %change %stddev
> \ | \ | \ | \ | \
> 14626665 -5.5% 13825048 -1.7% 14384837 -1.9% 14345958 -2.0% 14338834 vm-scalability.throughput
>
> =========================================================================================
> compiler/kconfig/rootfs/runtime/tbox_group/test/testcase:
> gcc-13/defconfig/debian/300s/qemu-vm/mmap-pread-seq-mt/vm-scalability
>
> commit:
> 6.6.0-rc4-mm-everything-2023-10-21-02-40+
> 6.6.0-rc4-split-folio-in-compaction+
> 6.6.0-rc4-folio-migration-in-compaction+
> 6.6.0-rc4-folio-migration-free-page-split+
> 6.6.0-rc4-folio-migration-free-page-split-sort-src+
>
> 6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f
> ---------------- --------------------------- --------------------------- --------------------------- ---------------------------
> %stddev %change %stddev %change %stddev %change %stddev %change %stddev
> \ | \ | \ | \ | \
> 14314960 -18.3% 11691969 ± 2% -7.7% 13217795 -8.8% 13057395 -10.8% 12764204 vm-scalability.throughput
>
> =========================================================================================
> compiler/kconfig/rootfs/runtime/tbox_group/test/testcase:
> gcc-13/defconfig/debian/300s/qemu-vm/lru-file-readtwice/vm-scalability
>
> commit:
> 6.6.0-rc4-mm-everything-2023-10-21-02-40+
> 6.6.0-rc4-split-folio-in-compaction+
> 6.6.0-rc4-folio-migration-in-compaction+
> 6.6.0-rc4-folio-migration-free-page-split+
> 6.6.0-rc4-folio-migration-free-page-split-sort-src+
>
> 6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f
> ---------------- --------------------------- --------------------------- --------------------------- ---------------------------
> %stddev %change %stddev %change %stddev %change %stddev %change %stddev
> \ | \ | \ | \ | \
> 6498203 ± 6% +41.5% 9193356 +30.4% 8472024 +30.0% 8449054 ± 3% +32.3% 8596781 ± 2% vm-scalability.throughput
>
> =========================================================================================
> compiler/kconfig/rootfs/runtime/tbox_group/test/testcase:
> gcc-13/defconfig/debian/300s/qemu-vm/lru-file-mmap-read/vm-scalability
>
> commit:
> 6.6.0-rc4-mm-everything-2023-10-21-02-40+
> 6.6.0-rc4-split-folio-in-compaction+
> 6.6.0-rc4-folio-migration-in-compaction+
> 6.6.0-rc4-folio-migration-free-page-split+
> 6.6.0-rc4-folio-migration-free-page-split-sort-src+
>
> 6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f
> ---------------- --------------------------- --------------------------- --------------------------- ---------------------------
> %stddev %change %stddev %change %stddev %change %stddev %change %stddev
> \ | \ | \ | \ | \
> 5440732 ± 10% +35.1% 7352631 ± 9% +32.3% 7200622 ± 5% +27.0% 6909673 ± 4% +25.9% 6849852 ± 5% vm-scalability.throughput
>
> =========================================================================================
> compiler/kconfig/rootfs/runtime/tbox_group/test/testcase:
> gcc-13/defconfig/debian/300s/qemu-vm/lru-file-mmap-read-rand/vm-scalability
>
> commit:
> 6.6.0-rc4-mm-everything-2023-10-21-02-40+
> 6.6.0-rc4-split-folio-in-compaction+
> 6.6.0-rc4-folio-migration-in-compaction+
> 6.6.0-rc4-folio-migration-free-page-split+
> 6.6.0-rc4-folio-migration-free-page-split-sort-src+
>
> 6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f
> ---------------- --------------------------- --------------------------- --------------------------- ---------------------------
> %stddev %change %stddev %change %stddev %change %stddev %change %stddev
> \ | \ | \ | \ | \
> 1627989 +0.3% 1632141 +1.8% 1656877 +0.1% 1629816 -0.2% 1625213 vm-scalability.throughput
>
> =========================================================================================
> compiler/kconfig/rootfs/runtime/size/tbox_group/test/testcase:
> gcc-13/defconfig/debian/300s/8T/qemu-vm/anon-wx-seq-mt/vm-scalability
>
> commit:
> 6.6.0-rc4-mm-everything-2023-10-21-02-40+
> 6.6.0-rc4-split-folio-in-compaction+
> 6.6.0-rc4-folio-migration-in-compaction+
> 6.6.0-rc4-folio-migration-free-page-split+
> 6.6.0-rc4-folio-migration-free-page-split-sort-src+
>
> 6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f
> ---------------- --------------------------- --------------------------- --------------------------- ---------------------------
> %stddev %change %stddev %change %stddev %change %stddev %change %stddev
> \ | \ | \ | \ | \
> 6738043 +1.4% 6835096 +1.4% 6834329 +1.0% 6805401 +1.2% 6820637 vm-scalability.throughput
>
> =========================================================================================
> compiler/kconfig/rootfs/runtime/size/tbox_group/test/testcase:
> gcc-13/defconfig/debian/300s/8T/qemu-vm/anon-w-seq-mt/vm-scalability
>
> commit:
> 6.6.0-rc4-mm-everything-2023-10-21-02-40+
> 6.6.0-rc4-split-folio-in-compaction+
> 6.6.0-rc4-folio-migration-in-compaction+
> 6.6.0-rc4-folio-migration-free-page-split+
> 6.6.0-rc4-folio-migration-free-page-split-sort-src+
>
> 6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f
> ---------------- --------------------------- --------------------------- --------------------------- ---------------------------
> %stddev %change %stddev %change %stddev %change %stddev %change %stddev
> \ | \ | \ | \ | \
> 7644894 +0.2% 7661425 +0.4% 7676929 +1.0% 7719218 +0.3% 7664658 vm-scalability.throughput
>
>
> =========================================================================================
> compiler/kconfig/rootfs/runtime/size/tbox_group/test/testcase:
> gcc-13/defconfig/debian/300s/512G/qemu-vm/anon-wx-rand-mt/vm-scalability
>
> commit:
> 6.6.0-rc4-mm-everything-2023-10-21-02-40+
> 6.6.0-rc4-split-folio-in-compaction+
> 6.6.0-rc4-folio-migration-in-compaction+
> 6.6.0-rc4-folio-migration-free-page-split+
> 6.6.0-rc4-folio-migration-free-page-split-sort-src+
>
> 6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f
> ---------------- --------------------------- --------------------------- --------------------------- ---------------------------
> %stddev %change %stddev %change %stddev %change %stddev %change %stddev
> \ | \ | \ | \ | \
> 521941 +3.3% 539254 +3.3% 539213 +1.0% 527048 +3.4% 539588 vm-scalability.throughput
>
> =========================================================================================
> compiler/kconfig/rootfs/runtime/size/tbox_group/test/testcase:
> gcc-13/defconfig/debian/300s/512G/qemu-vm/anon-w-rand/vm-scalability
>
> commit:
> 6.6.0-rc4-mm-everything-2023-10-21-02-40+
> 6.6.0-rc4-split-folio-in-compaction+
> 6.6.0-rc4-folio-migration-in-compaction+
> 6.6.0-rc4-folio-migration-free-page-split+
> 6.6.0-rc4-folio-migration-free-page-split-sort-src+
>
> 6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f
> ---------------- --------------------------- --------------------------- --------------------------- ---------------------------
> %stddev %change %stddev %change %stddev %change %stddev %change %stddev
> \ | \ | \ | \ | \
> 481028 +4.3% 501876 +2.4% 492713 +0.4% 482829 ± 2% +3.5% 497649 ± 2% vm-scalability.throughput
>
> =========================================================================================
> compiler/kconfig/rootfs/runtime/size/tbox_group/test/testcase:
> gcc-13/defconfig/debian/300s/512G/qemu-vm/anon-w-rand-mt/vm-scalability
>
> commit:
> 6.6.0-rc4-mm-everything-2023-10-21-02-40+
> 6.6.0-rc4-split-folio-in-compaction+
> 6.6.0-rc4-folio-migration-in-compaction+
> 6.6.0-rc4-folio-migration-free-page-split+
> 6.6.0-rc4-folio-migration-free-page-split-sort-src+
>
> 6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f
> ---------------- --------------------------- --------------------------- --------------------------- ---------------------------
> %stddev %change %stddev %change %stddev %change %stddev %change %stddev
> \ | \ | \ | \ | \
> 477063 +4.0% 496373 +4.0% 496283 +1.4% 483966 +3.2% 492216 ± 2% vm-scalability.throughput
>
> =========================================================================================
> compiler/kconfig/rootfs/runtime/size/tbox_group/test/testcase:
> gcc-13/defconfig/debian/300s/512G/qemu-vm/anon-cow-rand/vm-scalability
>
> commit:
> 6.6.0-rc4-mm-everything-2023-10-21-02-40+
> 6.6.0-rc4-split-folio-in-compaction+
> 6.6.0-rc4-folio-migration-in-compaction+
> 6.6.0-rc4-folio-migration-free-page-split+
> 6.6.0-rc4-folio-migration-free-page-split-sort-src+
>
> 6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f
> ---------------- --------------------------- --------------------------- --------------------------- ---------------------------
> %stddev %change %stddev %change %stddev %change %stddev %change %stddev
> \ | \ | \ | \ | \
> 458073 +3.3% 473056 +2.6% 469835 +0.3% 459266 +3.0% 471735 vm-scalability.throughput
>
> =========================================================================================
> compiler/kconfig/rootfs/runtime/size/tbox_group/test/testcase:
> gcc-13/defconfig/debian/300s/512G/qemu-vm/anon-cow-rand-mt/vm-scalability
>
> commit:
> 6.6.0-rc4-mm-everything-2023-10-21-02-40+
> 6.6.0-rc4-split-folio-in-compaction+
> 6.6.0-rc4-folio-migration-in-compaction+
> 6.6.0-rc4-folio-migration-free-page-split+
> 6.6.0-rc4-folio-migration-free-page-split-sort-src+
>
> 6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f
> ---------------- --------------------------- --------------------------- --------------------------- ---------------------------
> %stddev %change %stddev %change %stddev %change %stddev %change %stddev
> \ | \ | \ | \ | \
> 468158 +0.8% 472115 +1.4% 474547 +1.1% 473307 +2.1% 478173 vm-scalability.throughput
>
> =========================================================================================
> compiler/kconfig/rootfs/runtime/size/tbox_group/test/testcase:
> gcc-13/defconfig/debian/300s/2T/qemu-vm/shm-xread-seq/vm-scalability
>
> commit:
> 6.6.0-rc4-mm-everything-2023-10-21-02-40+
> 6.6.0-rc4-split-folio-in-compaction+
> 6.6.0-rc4-folio-migration-in-compaction+
> 6.6.0-rc4-folio-migration-free-page-split+
> 6.6.0-rc4-folio-migration-free-page-split-sort-src+
>
> 6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f
> ---------------- --------------------------- --------------------------- --------------------------- ---------------------------
> %stddev %change %stddev %change %stddev %change %stddev %change %stddev
> \ | \ | \ | \ | \
> 3489946 +1.4% 3540036 +0.4% 3505413 +1.7% 3549852 +1.4% 3539309 vm-scalability.throughput
>
> =========================================================================================
> compiler/kconfig/rootfs/runtime/size/tbox_group/test/testcase:
> gcc-13/defconfig/debian/300s/2T/qemu-vm/shm-xread-seq-mt/vm-scalability
>
> commit:
> 6.6.0-rc4-mm-everything-2023-10-21-02-40+
> 6.6.0-rc4-split-folio-in-compaction+
> 6.6.0-rc4-folio-migration-in-compaction+
> 6.6.0-rc4-folio-migration-free-page-split+
> 6.6.0-rc4-folio-migration-free-page-split-sort-src+
>
> 6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f
> ---------------- --------------------------- --------------------------- --------------------------- ---------------------------
> %stddev %change %stddev %change %stddev %change %stddev %change %stddev
> \ | \ | \ | \ | \
> 2621783 +1.0% 2647480 +0.6% 2636399 +0.7% 2641150 +0.1% 2624301 vm-scalability.throughput
>
> =========================================================================================
> compiler/kconfig/rootfs/runtime/size/tbox_group/test/testcase:
> gcc-13/defconfig/debian/300s/2T/qemu-vm/shm-pread-seq/vm-scalability
>
> commit:
> 6.6.0-rc4-mm-everything-2023-10-21-02-40+
> 6.6.0-rc4-split-folio-in-compaction+
> 6.6.0-rc4-folio-migration-in-compaction+
> 6.6.0-rc4-folio-migration-free-page-split+
> 6.6.0-rc4-folio-migration-free-page-split-sort-src+
>
> 6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f
> ---------------- --------------------------- --------------------------- --------------------------- ---------------------------
> %stddev %change %stddev %change %stddev %change %stddev %change %stddev
> \ | \ | \ | \ | \
> 3504293 +1.5% 3557664 -0.4% 3489639 +0.4% 3516842 +0.8% 3533944 vm-scalability.throughput
>
> =========================================================================================
> compiler/kconfig/rootfs/runtime/size/tbox_group/test/testcase:
> gcc-13/defconfig/debian/300s/256G/qemu-vm/msync/vm-scalability
>
> commit:
> 6.6.0-rc4-mm-everything-2023-10-21-02-40+
> 6.6.0-rc4-split-folio-in-compaction+
> 6.6.0-rc4-folio-migration-in-compaction+
> 6.6.0-rc4-folio-migration-free-page-split+
> 6.6.0-rc4-folio-migration-free-page-split-sort-src+
>
> 6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f
> ---------------- --------------------------- --------------------------- --------------------------- ---------------------------
> %stddev %change %stddev %change %stddev %change %stddev %change %stddev
> \ | \ | \ | \ | \
> 919378 ± 6% +3.8% 954028 ± 10% +7.2% 985920 ± 6% +7.4% 987077 ± 5% +14.2% 1049473 ± 2% vm-scalability.throughput
>
> =========================================================================================
> compiler/kconfig/rootfs/runtime/size/tbox_group/test/testcase:
> gcc-13/defconfig/debian/300s/256G/qemu-vm/msync-mt/vm-scalability
>
> commit:
> 6.6.0-rc4-mm-everything-2023-10-21-02-40+
> 6.6.0-rc4-split-folio-in-compaction+
> 6.6.0-rc4-folio-migration-in-compaction+
> 6.6.0-rc4-folio-migration-free-page-split+
> 6.6.0-rc4-folio-migration-free-page-split-sort-src+
>
> 6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f
> ---------------- --------------------------- --------------------------- --------------------------- ---------------------------
> %stddev %change %stddev %change %stddev %change %stddev %change %stddev
> \ | \ | \ | \ | \
> 2398063 +2.0% 2446844 -1.5% 2361096 ± 2% -3.1% 2324346 ± 4% -1.0% 2375055 ± 4% vm-scalability.throughput
>
> =========================================================================================
> compiler/kconfig/rootfs/runtime/size/tbox_group/test/testcase:
> gcc-13/defconfig/debian/300s/128G/qemu-vm/truncate/vm-scalability
>
> commit:
> 6.6.0-rc4-mm-everything-2023-10-21-02-40+
> 6.6.0-rc4-split-folio-in-compaction+
> 6.6.0-rc4-folio-migration-in-compaction+
> 6.6.0-rc4-folio-migration-free-page-split+
> 6.6.0-rc4-folio-migration-free-page-split-sort-src+
>
> 6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f
> ---------------- --------------------------- --------------------------- --------------------------- ---------------------------
> %stddev %change %stddev %change %stddev %change %stddev %change %stddev
> \ | \ | \ | \ | \
> 2.084e+08 ± 2% +1.5% 2.114e+08 ± 3% +6.5% 2.22e+08 ± 3% -2.3% 2.035e+08 ± 2% -2.1% 2.04e+08 ± 5% vm-scalability.throughput
>
>
>
> [1] https://lore.kernel.org/linux-mm/f8d47176-03a8-99bf-a813-b5942830fd73@arm.com/
>
> Zi Yan (4):
> mm/compaction: enable compacting >0 order folios.
> mm/compaction: add support for >0 order folio memory compaction.
> mm/compaction: optimize >0 order folio compaction with free page
> split.
> mm/compaction: optimize >0 order folio compaction by sorting source
> pages.
>
> mm/compaction.c | 280 ++++++++++++++++++++++++++++++++++++------------
> mm/internal.h | 7 +-
> 2 files changed, 217 insertions(+), 70 deletions(-)
>
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v1 0/4] Enable >0 order folio memory compaction
2023-11-21 15:46 ` [PATCH v1 0/4] Enable >0 order folio memory compaction Ryan Roberts
@ 2023-11-21 16:45 ` Zi Yan
2023-11-21 17:11 ` Ryan Roberts
0 siblings, 1 reply; 21+ messages in thread
From: Zi Yan @ 2023-11-21 16:45 UTC (permalink / raw)
To: Ryan Roberts
Cc: linux-mm, linux-kernel, "Huang, Ying",
Andrew Morton, "Matthew Wilcox (Oracle)",
David Hildenbrand, "Yin, Fengwei",
Yu Zhao, Vlastimil Babka, "Kirill A . Shutemov",
Johannes Weiner, Baolin Wang, Kemeng Shi, Mel Gorman, Rohan Puri,
Mcgrof Chamberlain, Adam Manzanares,
"Vishal Moola (Oracle)"
[-- Attachment #1: Type: text/plain, Size: 2385 bytes --]
On 21 Nov 2023, at 10:46, Ryan Roberts wrote:
>>
>> vm-scalability results
>> ===
>>
>> =========================================================================================
>> compiler/kconfig/rootfs/runtime/tbox_group/test/testcase:
>> gcc-13/defconfig/debian/300s/qemu-vm/mmap-xread-seq-mt/vm-scalability
>>
>> commit:
>> 6.6.0-rc4-mm-everything-2023-10-21-02-40+
>> 6.6.0-rc4-split-folio-in-compaction+
>> 6.6.0-rc4-folio-migration-in-compaction+
>> 6.6.0-rc4-folio-migration-free-page-split+
>> 6.6.0-rc4-folio-migration-free-page-split-sort-src+
>>
>> 6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f
>> ---------------- --------------------------- --------------------------- --------------------------- ---------------------------
>> %stddev %change %stddev %change %stddev %change %stddev %change %stddev
>> \ | \ | \ | \ | \
>> 12896955 +2.7% 13249322 -4.0% 12385175 ± 5% +1.1% 13033951 -0.4% 12845698 vm-scalability.throughput
>
> Hi Zi,
>
> Are you able to add any commentary to these results as I'm struggling to
> interpret them; Is a positive or negative change better (are they times or
> rates?). What are the stddev values? The title suggests percent but the values
> are huge - I'm trying to understand what the error bars look like - are the
> swings real or noise?
The metric is vm-scalability.throughput, so the larger the better. Some %stddev
are not present since they are too small. For 6.6.0-rc4-folio-migration-in-compaction+,
%stddev is greater than %change, so the change might be noise.
Also, I talked to DavidH in last THP Cabal meeting about this. He suggested that
there are a lot of noise in vm-scalability like what I have here and I should
run more iterations and on bare metal. I am currently rerun them on a baremetal
and more iterations on the existing VM and report the results later. Please
note that the runs really take some time.
In addition, I will find other fragmentation-related benchmarks, so we can see
the impact on memory fragmentation.
--
Best Regards,
Yan, Zi
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 854 bytes --]
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v1 0/4] Enable >0 order folio memory compaction
2023-11-21 16:45 ` Zi Yan
@ 2023-11-21 17:11 ` Ryan Roberts
2024-01-02 20:50 ` Zi Yan
0 siblings, 1 reply; 21+ messages in thread
From: Ryan Roberts @ 2023-11-21 17:11 UTC (permalink / raw)
To: Zi Yan
Cc: linux-mm, linux-kernel, Huang, Ying, Andrew Morton,
Matthew Wilcox (Oracle),
David Hildenbrand, Yin, Fengwei, Yu Zhao, Vlastimil Babka,
Kirill A . Shutemov, Johannes Weiner, Baolin Wang, Kemeng Shi,
Mel Gorman, Rohan Puri, Mcgrof Chamberlain, Adam Manzanares,
Vishal Moola (Oracle)
On 21/11/2023 16:45, Zi Yan wrote:
> On 21 Nov 2023, at 10:46, Ryan Roberts wrote:
>
>>>
>>> vm-scalability results
>>> ===
>>>
>>> =========================================================================================
>>> compiler/kconfig/rootfs/runtime/tbox_group/test/testcase:
>>> gcc-13/defconfig/debian/300s/qemu-vm/mmap-xread-seq-mt/vm-scalability
>>>
>>> commit:
>>> 6.6.0-rc4-mm-everything-2023-10-21-02-40+
>>> 6.6.0-rc4-split-folio-in-compaction+
>>> 6.6.0-rc4-folio-migration-in-compaction+
>>> 6.6.0-rc4-folio-migration-free-page-split+
>>> 6.6.0-rc4-folio-migration-free-page-split-sort-src+
>>>
>>> 6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f
>>> ---------------- --------------------------- --------------------------- --------------------------- ---------------------------
>>> %stddev %change %stddev %change %stddev %change %stddev %change %stddev
>>> \ | \ | \ | \ | \
>>> 12896955 +2.7% 13249322 -4.0% 12385175 ± 5% +1.1% 13033951 -0.4% 12845698 vm-scalability.throughput
>>
>> Hi Zi,
>>
>> Are you able to add any commentary to these results as I'm struggling to
>> interpret them; Is a positive or negative change better (are they times or
>> rates?). What are the stddev values? The title suggests percent but the values
>> are huge - I'm trying to understand what the error bars look like - are the
>> swings real or noise?
>
> The metric is vm-scalability.throughput, so the larger the better. Some %stddev
> are not present since they are too small. For 6.6.0-rc4-folio-migration-in-compaction+,
> %stddev is greater than %change, so the change might be noise.
Ahh got it - thanks!
>
> Also, I talked to DavidH in last THP Cabal meeting about this. He suggested that
> there are a lot of noise in vm-scalability like what I have here and I should
> run more iterations and on bare metal. I am currently rerun them on a baremetal
> and more iterations on the existing VM and report the results later. Please
> note that the runs really take some time.
Ahh ok, I'll wait for the bare metal numbers and will disregard these for now.
Thanks!
>
> In addition, I will find other fragmentation-related benchmarks, so we can see
> the impact on memory fragmentation.
>
> --
> Best Regards,
> Yan, Zi
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v1 3/4] mm/compaction: optimize >0 order folio compaction with free page split.
2023-11-13 17:01 ` [PATCH v1 3/4] mm/compaction: optimize >0 order folio compaction with free page split Zi Yan
@ 2023-11-22 10:26 ` Ryan Roberts
2023-11-22 14:35 ` Zi Yan
0 siblings, 1 reply; 21+ messages in thread
From: Ryan Roberts @ 2023-11-22 10:26 UTC (permalink / raw)
To: Zi Yan, linux-mm, linux-kernel
Cc: Huang, Ying, Andrew Morton, Matthew Wilcox (Oracle),
David Hildenbrand, Yin, Fengwei, Yu Zhao, Vlastimil Babka,
Kirill A . Shutemov, Johannes Weiner, Baolin Wang, Kemeng Shi,
Mel Gorman, Rohan Puri, Mcgrof Chamberlain, Adam Manzanares,
Vishal Moola (Oracle)
On 13/11/2023 17:01, Zi Yan wrote:
> From: Zi Yan <ziy@nvidia.com>
>
> During migration in a memory compaction, free pages are placed in an array
> of page lists based on their order. But the desired free page order (i.e.,
> the order of a source page) might not be always present, thus leading to
> migration failures. Split a high order free pages when source migration
> page has a lower order to increase migration successful rate.
>
> Note: merging free pages when a migration fails and a lower order free
> page is returned via compaction_free() is possible, but there is too much
> work. Since the free pages are not buddy pages, it is hard to identify
> these free pages using existing PFN-based page merging algorithm.
>
> Signed-off-by: Zi Yan <ziy@nvidia.com>
> ---
> mm/compaction.c | 40 +++++++++++++++++++++++++++++++++++++++-
> 1 file changed, 39 insertions(+), 1 deletion(-)
>
> diff --git a/mm/compaction.c b/mm/compaction.c
> index ec6b5cc7e907..9c083e6b399a 100644
> --- a/mm/compaction.c
> +++ b/mm/compaction.c
> @@ -1806,9 +1806,46 @@ static struct folio *compaction_alloc(struct folio *src, unsigned long data)
> struct compact_control *cc = (struct compact_control *)data;
> struct folio *dst;
> int order = folio_order(src);
> + bool has_isolated_pages = false;
>
> +again:
> if (!cc->freepages[order].nr_pages) {
> - isolate_freepages(cc);
> + int i;
> +
> + for (i = order + 1; i <= MAX_ORDER; i++) {
> + if (cc->freepages[i].nr_pages) {
> + struct page *freepage =
> + list_first_entry(&cc->freepages[i].pages,
> + struct page, lru);
> +
> + int start_order = i;
> + unsigned long size = 1 << start_order;
> +
> + list_del(&freepage->lru);
> + cc->freepages[i].nr_pages--;
> +
> + while (start_order > order) {
> + start_order--;
> + size >>= 1;
> +
> + list_add(&freepage[size].lru,
> + &cc->freepages[start_order].pages);
> + cc->freepages[start_order].nr_pages++;
> + set_page_private(&freepage[size], start_order);
> + }
> + post_alloc_hook(freepage, order, __GFP_MOVABLE);
> + if (order)
> + prep_compound_page(freepage, order);
> + dst = page_folio(freepage);
> + goto done;
Perhaps just do:
dst = (struct folio *)freepage;
goto done;
then move done: up a couple of statements below, so that post_alloc_hook() and
prep_compound_page() are always done below in common path? Although perhaps the
cast is frowned upon, you're already making the assumption that page and folio
are interchangable the way you call list_first_entry().
> + }
> + }
> + if (!has_isolated_pages) {
> + isolate_freepages(cc);
> + has_isolated_pages = true;
> + goto again;
> + }
> +
> if (!cc->freepages[order].nr_pages)
> return NULL;
> }
> @@ -1819,6 +1856,7 @@ static struct folio *compaction_alloc(struct folio *src, unsigned long data)
> post_alloc_hook(&dst->page, order, __GFP_MOVABLE);
> if (order)
> prep_compound_page(&dst->page, order);
> +done:
> cc->nr_freepages -= 1 << order;
> return page_rmappable_folio(&dst->page);
> }
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v1 3/4] mm/compaction: optimize >0 order folio compaction with free page split.
2023-11-22 10:26 ` Ryan Roberts
@ 2023-11-22 14:35 ` Zi Yan
0 siblings, 0 replies; 21+ messages in thread
From: Zi Yan @ 2023-11-22 14:35 UTC (permalink / raw)
To: Ryan Roberts
Cc: linux-mm, linux-kernel, "Huang, Ying",
Andrew Morton, "Matthew Wilcox (Oracle)",
David Hildenbrand, "Yin, Fengwei",
Yu Zhao, Vlastimil Babka, "Kirill A . Shutemov",
Johannes Weiner, Baolin Wang, Kemeng Shi, Mel Gorman, Rohan Puri,
Mcgrof Chamberlain, Adam Manzanares,
"Vishal Moola (Oracle)"
[-- Attachment #1: Type: text/plain, Size: 3430 bytes --]
On 22 Nov 2023, at 5:26, Ryan Roberts wrote:
> On 13/11/2023 17:01, Zi Yan wrote:
>> From: Zi Yan <ziy@nvidia.com>
>>
>> During migration in a memory compaction, free pages are placed in an array
>> of page lists based on their order. But the desired free page order (i.e.,
>> the order of a source page) might not be always present, thus leading to
>> migration failures. Split a high order free pages when source migration
>> page has a lower order to increase migration successful rate.
>>
>> Note: merging free pages when a migration fails and a lower order free
>> page is returned via compaction_free() is possible, but there is too much
>> work. Since the free pages are not buddy pages, it is hard to identify
>> these free pages using existing PFN-based page merging algorithm.
>>
>> Signed-off-by: Zi Yan <ziy@nvidia.com>
>> ---
>> mm/compaction.c | 40 +++++++++++++++++++++++++++++++++++++++-
>> 1 file changed, 39 insertions(+), 1 deletion(-)
>>
>> diff --git a/mm/compaction.c b/mm/compaction.c
>> index ec6b5cc7e907..9c083e6b399a 100644
>> --- a/mm/compaction.c
>> +++ b/mm/compaction.c
>> @@ -1806,9 +1806,46 @@ static struct folio *compaction_alloc(struct folio *src, unsigned long data)
>> struct compact_control *cc = (struct compact_control *)data;
>> struct folio *dst;
>> int order = folio_order(src);
>> + bool has_isolated_pages = false;
>>
>> +again:
>> if (!cc->freepages[order].nr_pages) {
>> - isolate_freepages(cc);
>> + int i;
>> +
>> + for (i = order + 1; i <= MAX_ORDER; i++) {
>> + if (cc->freepages[i].nr_pages) {
>> + struct page *freepage =
>> + list_first_entry(&cc->freepages[i].pages,
>> + struct page, lru);
>> +
>> + int start_order = i;
>> + unsigned long size = 1 << start_order;
>> +
>> + list_del(&freepage->lru);
>> + cc->freepages[i].nr_pages--;
>> +
>> + while (start_order > order) {
>> + start_order--;
>> + size >>= 1;
>> +
>> + list_add(&freepage[size].lru,
>> + &cc->freepages[start_order].pages);
>> + cc->freepages[start_order].nr_pages++;
>> + set_page_private(&freepage[size], start_order);
>> + }
>> + post_alloc_hook(freepage, order, __GFP_MOVABLE);
>> + if (order)
>> + prep_compound_page(freepage, order);
>> + dst = page_folio(freepage);
>> + goto done;
>
> Perhaps just do:
>
> dst = (struct folio *)freepage;
> goto done;
>
> then move done: up a couple of statements below, so that post_alloc_hook() and
> prep_compound_page() are always done below in common path? Although perhaps the
Sure. Thanks for the suggestion.
> cast is frowned upon, you're already making the assumption that page and folio
> are interchangable the way you call list_first_entry().
To save the _compound_head() in page_folio()? OK.
>
>> + }
>> + }
>> + if (!has_isolated_pages) {
>> + isolate_freepages(cc);
>> + has_isolated_pages = true;
>> + goto again;
>> + }
>> +
>> if (!cc->freepages[order].nr_pages)
>> return NULL;
>> }
>> @@ -1819,6 +1856,7 @@ static struct folio *compaction_alloc(struct folio *src, unsigned long data)
>> post_alloc_hook(&dst->page, order, __GFP_MOVABLE);
>> if (order)
>> prep_compound_page(&dst->page, order);
>> +done:
>> cc->nr_freepages -= 1 << order;
>> return page_rmappable_folio(&dst->page);
>> }
--
Best Regards,
Yan, Zi
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 854 bytes --]
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v1 0/4] Enable >0 order folio memory compaction
2023-11-13 17:01 [PATCH v1 0/4] Enable >0 order folio memory compaction Zi Yan
` (4 preceding siblings ...)
2023-11-21 15:46 ` [PATCH v1 0/4] Enable >0 order folio memory compaction Ryan Roberts
@ 2023-11-24 14:58 ` Ryan Roberts
5 siblings, 0 replies; 21+ messages in thread
From: Ryan Roberts @ 2023-11-24 14:58 UTC (permalink / raw)
To: Zi Yan, linux-mm, linux-kernel
Cc: Huang, Ying, Andrew Morton, Matthew Wilcox (Oracle),
David Hildenbrand, Yin, Fengwei, Yu Zhao, Vlastimil Babka,
Kirill A . Shutemov, Johannes Weiner, Baolin Wang, Kemeng Shi,
Mel Gorman, Rohan Puri, Mcgrof Chamberlain, Adam Manzanares,
Vishal Moola (Oracle)
On 13/11/2023 17:01, Zi Yan wrote:
> From: Zi Yan <ziy@nvidia.com>
>
> Hi all,
>
> This patchset enables >0 order folio memory compaction, which is one of
> the prerequisitions for large folio support[1]. It is on top of
> mm-everything-2023-10-21-02-40 (also applies to mm-everything-2023-11-10-20-42
> without conflicts).
>
> Per patch vm-scalability results on a 8-CPU (Intel Xeon E5-2650 v4 @ 2.20GHz)
> 16G VM are at the bottom and based on mm-everything-2023-10-21-02-40.
> Each has 5 runs. Highlights:
>
> 1. mmap-pread-seq-mt: -18.3% if split folios during compaction and -7.7%
> if migrating folios during compaction.
>
> 2. lru-file-readtwice: +41.5% if split folios during compaction and
> +30.4% if migrating folios during compaction.
>
> 3. lru-file-mmap-read: +35.1% if split folios during compaction and
> +32.3% if migrating folios during compaction.
>
> I will run experiments on bare metal and report results later.
Hi Zi,
I've taken this series into a backport on v6.1 that I maintain for my folios
stuff - I had to do a fair bit of massaging to get it ported, so its been a good
way to review the code. With the exception of the issue that Baolin has already
pointed out, it looks good to me! So, assuming the forthcoming perf numbers
don't show any problem:
Reviewed-by: Ryan Roberts <ryan.roberts@arm.com>
>
>
> Changelog
> ===
>
> From RFC:
> 1. Enabled >0 order folio compaction in the first patch by splitting all
> to-be-migrated folios. (per Huang, Ying)
>
> 2. Stopped isolating compound pages with order greater than cc->order
> to avoid wasting effort, since cc->order gives a hint that no free pages
> with order greater than it exist, thus migrating the compound pages will fail.
> (per Baolin Wang)
>
> 3. Retained the folio check within lru lock. (per Baolin Wang)
>
> 4. Made isolate_freepages_block() generate order-sorted multi lists.
> (per Johannes Weiner)
>
>
> Overview
> ===
>
> To support >0 order folio compaction, the patchset changes how free pages used
> for migration are kept during compaction. Free pages used to be split into
> order-0 pages that are post allocation processed (i.e., PageBuddy flag cleared,
> page order stored in page->private is zeroed, and page reference is set to 1).
> Now all free pages are kept in a MAX_ORDER+1 array of page lists based
> on their order without post allocation process. When migrate_pages() asks for
> a new page, one of the free pages, based on the requested page order, is
> then processed and given out.
>
>
> Optimizations
> ===
>
> 1. Free page split is added to increase migration success rate in case
> a source page does not have a matched free page in the free page lists.
> Free page merge is possible but not implemented, since existing
> PFN-based buddy page merge algorithm requires the identification of
> buddy pages, but free pages kept for memory compaction cannot have
> PageBuddy set to avoid confusing other PFN scanners.
>
> 2. Sort source pages in ascending order before migration is added to
> reduce free page split. Otherwise, high order free pages might be
> prematurely split, causing undesired high order folio migration failures.
>
>
> TODOs
> ===
>
> 1. Refactor free page post allocation and free page preparation code so
> that compaction_alloc() and compaction_free() can call functions instead
> of hard coding.
>
> 2. Make isolate_migratepages_block() produce a sorted list for
> compaction to avoid sort_folio_by_order() in Patch 4.
>
>
> Feel free to give comments and ask questions.
>
> Thanks.
>
> vm-scalability results
> ===
>
> =========================================================================================
> compiler/kconfig/rootfs/runtime/tbox_group/test/testcase:
> gcc-13/defconfig/debian/300s/qemu-vm/mmap-xread-seq-mt/vm-scalability
>
> commit:
> 6.6.0-rc4-mm-everything-2023-10-21-02-40+
> 6.6.0-rc4-split-folio-in-compaction+
> 6.6.0-rc4-folio-migration-in-compaction+
> 6.6.0-rc4-folio-migration-free-page-split+
> 6.6.0-rc4-folio-migration-free-page-split-sort-src+
>
> 6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f
> ---------------- --------------------------- --------------------------- --------------------------- ---------------------------
> %stddev %change %stddev %change %stddev %change %stddev %change %stddev
> \ | \ | \ | \ | \
> 12896955 +2.7% 13249322 -4.0% 12385175 ± 5% +1.1% 13033951 -0.4% 12845698 vm-scalability.throughput
>
> =========================================================================================
> compiler/kconfig/rootfs/runtime/tbox_group/test/testcase:
> gcc-13/defconfig/debian/300s/qemu-vm/mmap-pread-seq/vm-scalability
>
> commit:
> 6.6.0-rc4-mm-everything-2023-10-21-02-40+
> 6.6.0-rc4-split-folio-in-compaction+
> 6.6.0-rc4-folio-migration-in-compaction+
> 6.6.0-rc4-folio-migration-free-page-split+
> 6.6.0-rc4-folio-migration-free-page-split-sort-src+
>
> 6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f
> ---------------- --------------------------- --------------------------- --------------------------- ---------------------------
> %stddev %change %stddev %change %stddev %change %stddev %change %stddev
> \ | \ | \ | \ | \
> 14626665 -5.5% 13825048 -1.7% 14384837 -1.9% 14345958 -2.0% 14338834 vm-scalability.throughput
>
> =========================================================================================
> compiler/kconfig/rootfs/runtime/tbox_group/test/testcase:
> gcc-13/defconfig/debian/300s/qemu-vm/mmap-pread-seq-mt/vm-scalability
>
> commit:
> 6.6.0-rc4-mm-everything-2023-10-21-02-40+
> 6.6.0-rc4-split-folio-in-compaction+
> 6.6.0-rc4-folio-migration-in-compaction+
> 6.6.0-rc4-folio-migration-free-page-split+
> 6.6.0-rc4-folio-migration-free-page-split-sort-src+
>
> 6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f
> ---------------- --------------------------- --------------------------- --------------------------- ---------------------------
> %stddev %change %stddev %change %stddev %change %stddev %change %stddev
> \ | \ | \ | \ | \
> 14314960 -18.3% 11691969 ± 2% -7.7% 13217795 -8.8% 13057395 -10.8% 12764204 vm-scalability.throughput
>
> =========================================================================================
> compiler/kconfig/rootfs/runtime/tbox_group/test/testcase:
> gcc-13/defconfig/debian/300s/qemu-vm/lru-file-readtwice/vm-scalability
>
> commit:
> 6.6.0-rc4-mm-everything-2023-10-21-02-40+
> 6.6.0-rc4-split-folio-in-compaction+
> 6.6.0-rc4-folio-migration-in-compaction+
> 6.6.0-rc4-folio-migration-free-page-split+
> 6.6.0-rc4-folio-migration-free-page-split-sort-src+
>
> 6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f
> ---------------- --------------------------- --------------------------- --------------------------- ---------------------------
> %stddev %change %stddev %change %stddev %change %stddev %change %stddev
> \ | \ | \ | \ | \
> 6498203 ± 6% +41.5% 9193356 +30.4% 8472024 +30.0% 8449054 ± 3% +32.3% 8596781 ± 2% vm-scalability.throughput
>
> =========================================================================================
> compiler/kconfig/rootfs/runtime/tbox_group/test/testcase:
> gcc-13/defconfig/debian/300s/qemu-vm/lru-file-mmap-read/vm-scalability
>
> commit:
> 6.6.0-rc4-mm-everything-2023-10-21-02-40+
> 6.6.0-rc4-split-folio-in-compaction+
> 6.6.0-rc4-folio-migration-in-compaction+
> 6.6.0-rc4-folio-migration-free-page-split+
> 6.6.0-rc4-folio-migration-free-page-split-sort-src+
>
> 6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f
> ---------------- --------------------------- --------------------------- --------------------------- ---------------------------
> %stddev %change %stddev %change %stddev %change %stddev %change %stddev
> \ | \ | \ | \ | \
> 5440732 ± 10% +35.1% 7352631 ± 9% +32.3% 7200622 ± 5% +27.0% 6909673 ± 4% +25.9% 6849852 ± 5% vm-scalability.throughput
>
> =========================================================================================
> compiler/kconfig/rootfs/runtime/tbox_group/test/testcase:
> gcc-13/defconfig/debian/300s/qemu-vm/lru-file-mmap-read-rand/vm-scalability
>
> commit:
> 6.6.0-rc4-mm-everything-2023-10-21-02-40+
> 6.6.0-rc4-split-folio-in-compaction+
> 6.6.0-rc4-folio-migration-in-compaction+
> 6.6.0-rc4-folio-migration-free-page-split+
> 6.6.0-rc4-folio-migration-free-page-split-sort-src+
>
> 6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f
> ---------------- --------------------------- --------------------------- --------------------------- ---------------------------
> %stddev %change %stddev %change %stddev %change %stddev %change %stddev
> \ | \ | \ | \ | \
> 1627989 +0.3% 1632141 +1.8% 1656877 +0.1% 1629816 -0.2% 1625213 vm-scalability.throughput
>
> =========================================================================================
> compiler/kconfig/rootfs/runtime/size/tbox_group/test/testcase:
> gcc-13/defconfig/debian/300s/8T/qemu-vm/anon-wx-seq-mt/vm-scalability
>
> commit:
> 6.6.0-rc4-mm-everything-2023-10-21-02-40+
> 6.6.0-rc4-split-folio-in-compaction+
> 6.6.0-rc4-folio-migration-in-compaction+
> 6.6.0-rc4-folio-migration-free-page-split+
> 6.6.0-rc4-folio-migration-free-page-split-sort-src+
>
> 6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f
> ---------------- --------------------------- --------------------------- --------------------------- ---------------------------
> %stddev %change %stddev %change %stddev %change %stddev %change %stddev
> \ | \ | \ | \ | \
> 6738043 +1.4% 6835096 +1.4% 6834329 +1.0% 6805401 +1.2% 6820637 vm-scalability.throughput
>
> =========================================================================================
> compiler/kconfig/rootfs/runtime/size/tbox_group/test/testcase:
> gcc-13/defconfig/debian/300s/8T/qemu-vm/anon-w-seq-mt/vm-scalability
>
> commit:
> 6.6.0-rc4-mm-everything-2023-10-21-02-40+
> 6.6.0-rc4-split-folio-in-compaction+
> 6.6.0-rc4-folio-migration-in-compaction+
> 6.6.0-rc4-folio-migration-free-page-split+
> 6.6.0-rc4-folio-migration-free-page-split-sort-src+
>
> 6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f
> ---------------- --------------------------- --------------------------- --------------------------- ---------------------------
> %stddev %change %stddev %change %stddev %change %stddev %change %stddev
> \ | \ | \ | \ | \
> 7644894 +0.2% 7661425 +0.4% 7676929 +1.0% 7719218 +0.3% 7664658 vm-scalability.throughput
>
>
> =========================================================================================
> compiler/kconfig/rootfs/runtime/size/tbox_group/test/testcase:
> gcc-13/defconfig/debian/300s/512G/qemu-vm/anon-wx-rand-mt/vm-scalability
>
> commit:
> 6.6.0-rc4-mm-everything-2023-10-21-02-40+
> 6.6.0-rc4-split-folio-in-compaction+
> 6.6.0-rc4-folio-migration-in-compaction+
> 6.6.0-rc4-folio-migration-free-page-split+
> 6.6.0-rc4-folio-migration-free-page-split-sort-src+
>
> 6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f
> ---------------- --------------------------- --------------------------- --------------------------- ---------------------------
> %stddev %change %stddev %change %stddev %change %stddev %change %stddev
> \ | \ | \ | \ | \
> 521941 +3.3% 539254 +3.3% 539213 +1.0% 527048 +3.4% 539588 vm-scalability.throughput
>
> =========================================================================================
> compiler/kconfig/rootfs/runtime/size/tbox_group/test/testcase:
> gcc-13/defconfig/debian/300s/512G/qemu-vm/anon-w-rand/vm-scalability
>
> commit:
> 6.6.0-rc4-mm-everything-2023-10-21-02-40+
> 6.6.0-rc4-split-folio-in-compaction+
> 6.6.0-rc4-folio-migration-in-compaction+
> 6.6.0-rc4-folio-migration-free-page-split+
> 6.6.0-rc4-folio-migration-free-page-split-sort-src+
>
> 6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f
> ---------------- --------------------------- --------------------------- --------------------------- ---------------------------
> %stddev %change %stddev %change %stddev %change %stddev %change %stddev
> \ | \ | \ | \ | \
> 481028 +4.3% 501876 +2.4% 492713 +0.4% 482829 ± 2% +3.5% 497649 ± 2% vm-scalability.throughput
>
> =========================================================================================
> compiler/kconfig/rootfs/runtime/size/tbox_group/test/testcase:
> gcc-13/defconfig/debian/300s/512G/qemu-vm/anon-w-rand-mt/vm-scalability
>
> commit:
> 6.6.0-rc4-mm-everything-2023-10-21-02-40+
> 6.6.0-rc4-split-folio-in-compaction+
> 6.6.0-rc4-folio-migration-in-compaction+
> 6.6.0-rc4-folio-migration-free-page-split+
> 6.6.0-rc4-folio-migration-free-page-split-sort-src+
>
> 6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f
> ---------------- --------------------------- --------------------------- --------------------------- ---------------------------
> %stddev %change %stddev %change %stddev %change %stddev %change %stddev
> \ | \ | \ | \ | \
> 477063 +4.0% 496373 +4.0% 496283 +1.4% 483966 +3.2% 492216 ± 2% vm-scalability.throughput
>
> =========================================================================================
> compiler/kconfig/rootfs/runtime/size/tbox_group/test/testcase:
> gcc-13/defconfig/debian/300s/512G/qemu-vm/anon-cow-rand/vm-scalability
>
> commit:
> 6.6.0-rc4-mm-everything-2023-10-21-02-40+
> 6.6.0-rc4-split-folio-in-compaction+
> 6.6.0-rc4-folio-migration-in-compaction+
> 6.6.0-rc4-folio-migration-free-page-split+
> 6.6.0-rc4-folio-migration-free-page-split-sort-src+
>
> 6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f
> ---------------- --------------------------- --------------------------- --------------------------- ---------------------------
> %stddev %change %stddev %change %stddev %change %stddev %change %stddev
> \ | \ | \ | \ | \
> 458073 +3.3% 473056 +2.6% 469835 +0.3% 459266 +3.0% 471735 vm-scalability.throughput
>
> =========================================================================================
> compiler/kconfig/rootfs/runtime/size/tbox_group/test/testcase:
> gcc-13/defconfig/debian/300s/512G/qemu-vm/anon-cow-rand-mt/vm-scalability
>
> commit:
> 6.6.0-rc4-mm-everything-2023-10-21-02-40+
> 6.6.0-rc4-split-folio-in-compaction+
> 6.6.0-rc4-folio-migration-in-compaction+
> 6.6.0-rc4-folio-migration-free-page-split+
> 6.6.0-rc4-folio-migration-free-page-split-sort-src+
>
> 6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f
> ---------------- --------------------------- --------------------------- --------------------------- ---------------------------
> %stddev %change %stddev %change %stddev %change %stddev %change %stddev
> \ | \ | \ | \ | \
> 468158 +0.8% 472115 +1.4% 474547 +1.1% 473307 +2.1% 478173 vm-scalability.throughput
>
> =========================================================================================
> compiler/kconfig/rootfs/runtime/size/tbox_group/test/testcase:
> gcc-13/defconfig/debian/300s/2T/qemu-vm/shm-xread-seq/vm-scalability
>
> commit:
> 6.6.0-rc4-mm-everything-2023-10-21-02-40+
> 6.6.0-rc4-split-folio-in-compaction+
> 6.6.0-rc4-folio-migration-in-compaction+
> 6.6.0-rc4-folio-migration-free-page-split+
> 6.6.0-rc4-folio-migration-free-page-split-sort-src+
>
> 6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f
> ---------------- --------------------------- --------------------------- --------------------------- ---------------------------
> %stddev %change %stddev %change %stddev %change %stddev %change %stddev
> \ | \ | \ | \ | \
> 3489946 +1.4% 3540036 +0.4% 3505413 +1.7% 3549852 +1.4% 3539309 vm-scalability.throughput
>
> =========================================================================================
> compiler/kconfig/rootfs/runtime/size/tbox_group/test/testcase:
> gcc-13/defconfig/debian/300s/2T/qemu-vm/shm-xread-seq-mt/vm-scalability
>
> commit:
> 6.6.0-rc4-mm-everything-2023-10-21-02-40+
> 6.6.0-rc4-split-folio-in-compaction+
> 6.6.0-rc4-folio-migration-in-compaction+
> 6.6.0-rc4-folio-migration-free-page-split+
> 6.6.0-rc4-folio-migration-free-page-split-sort-src+
>
> 6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f
> ---------------- --------------------------- --------------------------- --------------------------- ---------------------------
> %stddev %change %stddev %change %stddev %change %stddev %change %stddev
> \ | \ | \ | \ | \
> 2621783 +1.0% 2647480 +0.6% 2636399 +0.7% 2641150 +0.1% 2624301 vm-scalability.throughput
>
> =========================================================================================
> compiler/kconfig/rootfs/runtime/size/tbox_group/test/testcase:
> gcc-13/defconfig/debian/300s/2T/qemu-vm/shm-pread-seq/vm-scalability
>
> commit:
> 6.6.0-rc4-mm-everything-2023-10-21-02-40+
> 6.6.0-rc4-split-folio-in-compaction+
> 6.6.0-rc4-folio-migration-in-compaction+
> 6.6.0-rc4-folio-migration-free-page-split+
> 6.6.0-rc4-folio-migration-free-page-split-sort-src+
>
> 6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f
> ---------------- --------------------------- --------------------------- --------------------------- ---------------------------
> %stddev %change %stddev %change %stddev %change %stddev %change %stddev
> \ | \ | \ | \ | \
> 3504293 +1.5% 3557664 -0.4% 3489639 +0.4% 3516842 +0.8% 3533944 vm-scalability.throughput
>
> =========================================================================================
> compiler/kconfig/rootfs/runtime/size/tbox_group/test/testcase:
> gcc-13/defconfig/debian/300s/256G/qemu-vm/msync/vm-scalability
>
> commit:
> 6.6.0-rc4-mm-everything-2023-10-21-02-40+
> 6.6.0-rc4-split-folio-in-compaction+
> 6.6.0-rc4-folio-migration-in-compaction+
> 6.6.0-rc4-folio-migration-free-page-split+
> 6.6.0-rc4-folio-migration-free-page-split-sort-src+
>
> 6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f
> ---------------- --------------------------- --------------------------- --------------------------- ---------------------------
> %stddev %change %stddev %change %stddev %change %stddev %change %stddev
> \ | \ | \ | \ | \
> 919378 ± 6% +3.8% 954028 ± 10% +7.2% 985920 ± 6% +7.4% 987077 ± 5% +14.2% 1049473 ± 2% vm-scalability.throughput
>
> =========================================================================================
> compiler/kconfig/rootfs/runtime/size/tbox_group/test/testcase:
> gcc-13/defconfig/debian/300s/256G/qemu-vm/msync-mt/vm-scalability
>
> commit:
> 6.6.0-rc4-mm-everything-2023-10-21-02-40+
> 6.6.0-rc4-split-folio-in-compaction+
> 6.6.0-rc4-folio-migration-in-compaction+
> 6.6.0-rc4-folio-migration-free-page-split+
> 6.6.0-rc4-folio-migration-free-page-split-sort-src+
>
> 6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f
> ---------------- --------------------------- --------------------------- --------------------------- ---------------------------
> %stddev %change %stddev %change %stddev %change %stddev %change %stddev
> \ | \ | \ | \ | \
> 2398063 +2.0% 2446844 -1.5% 2361096 ± 2% -3.1% 2324346 ± 4% -1.0% 2375055 ± 4% vm-scalability.throughput
>
> =========================================================================================
> compiler/kconfig/rootfs/runtime/size/tbox_group/test/testcase:
> gcc-13/defconfig/debian/300s/128G/qemu-vm/truncate/vm-scalability
>
> commit:
> 6.6.0-rc4-mm-everything-2023-10-21-02-40+
> 6.6.0-rc4-split-folio-in-compaction+
> 6.6.0-rc4-folio-migration-in-compaction+
> 6.6.0-rc4-folio-migration-free-page-split+
> 6.6.0-rc4-folio-migration-free-page-split-sort-src+
>
> 6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f
> ---------------- --------------------------- --------------------------- --------------------------- ---------------------------
> %stddev %change %stddev %change %stddev %change %stddev %change %stddev
> \ | \ | \ | \ | \
> 2.084e+08 ± 2% +1.5% 2.114e+08 ± 3% +6.5% 2.22e+08 ± 3% -2.3% 2.035e+08 ± 2% -2.1% 2.04e+08 ± 5% vm-scalability.throughput
>
>
>
> [1] https://lore.kernel.org/linux-mm/f8d47176-03a8-99bf-a813-b5942830fd73@arm.com/
>
> Zi Yan (4):
> mm/compaction: enable compacting >0 order folios.
> mm/compaction: add support for >0 order folio memory compaction.
> mm/compaction: optimize >0 order folio compaction with free page
> split.
> mm/compaction: optimize >0 order folio compaction by sorting source
> pages.
>
> mm/compaction.c | 280 ++++++++++++++++++++++++++++++++++++------------
> mm/internal.h | 7 +-
> 2 files changed, 217 insertions(+), 70 deletions(-)
>
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v1 0/4] Enable >0 order folio memory compaction
2023-11-21 17:11 ` Ryan Roberts
@ 2024-01-02 20:50 ` Zi Yan
2024-01-03 9:12 ` Ryan Roberts
0 siblings, 1 reply; 21+ messages in thread
From: Zi Yan @ 2024-01-02 20:50 UTC (permalink / raw)
To: Ryan Roberts
Cc: linux-mm, linux-kernel, "Huang, Ying",
Andrew Morton, "Matthew Wilcox (Oracle)",
David Hildenbrand, "Yin, Fengwei",
Yu Zhao, Vlastimil Babka, "Kirill A . Shutemov",
Johannes Weiner, Baolin Wang, Kemeng Shi, Mel Gorman, Rohan Puri,
Mcgrof Chamberlain, Adam Manzanares,
"Vishal Moola (Oracle)"
[-- Attachment #1: Type: text/plain, Size: 13181 bytes --]
On 21 Nov 2023, at 12:11, Ryan Roberts wrote:
> On 21/11/2023 16:45, Zi Yan wrote:
>> On 21 Nov 2023, at 10:46, Ryan Roberts wrote:
>>
>>>>
>>>> vm-scalability results
>>>> ===
>>>>
>>>> =========================================================================================
>>>> compiler/kconfig/rootfs/runtime/tbox_group/test/testcase:
>>>> gcc-13/defconfig/debian/300s/qemu-vm/mmap-xread-seq-mt/vm-scalability
>>>>
>>>> commit:
>>>> 6.6.0-rc4-mm-everything-2023-10-21-02-40+
>>>> 6.6.0-rc4-split-folio-in-compaction+
>>>> 6.6.0-rc4-folio-migration-in-compaction+
>>>> 6.6.0-rc4-folio-migration-free-page-split+
>>>> 6.6.0-rc4-folio-migration-free-page-split-sort-src+
>>>>
>>>> 6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f
>>>> ---------------- --------------------------- --------------------------- --------------------------- ---------------------------
>>>> %stddev %change %stddev %change %stddev %change %stddev %change %stddev
>>>> \ | \ | \ | \ | \
>>>> 12896955 +2.7% 13249322 -4.0% 12385175 ± 5% +1.1% 13033951 -0.4% 12845698 vm-scalability.throughput
>>>
>>> Hi Zi,
>>>
>>> Are you able to add any commentary to these results as I'm struggling to
>>> interpret them; Is a positive or negative change better (are they times or
>>> rates?). What are the stddev values? The title suggests percent but the values
>>> are huge - I'm trying to understand what the error bars look like - are the
>>> swings real or noise?
>>
>> The metric is vm-scalability.throughput, so the larger the better. Some %stddev
>> are not present since they are too small. For 6.6.0-rc4-folio-migration-in-compaction+,
>> %stddev is greater than %change, so the change might be noise.
>
> Ahh got it - thanks!
>
>>
>> Also, I talked to DavidH in last THP Cabal meeting about this. He suggested that
>> there are a lot of noise in vm-scalability like what I have here and I should
>> run more iterations and on bare metal. I am currently rerun them on a baremetal
>> and more iterations on the existing VM and report the results later. Please
>> note that the runs really take some time.
>
> Ahh ok, I'll wait for the bare metal numbers and will disregard these for now.
> Thanks!
It seems that the unexpected big mmap-pread-seq-mt perf drop came from the mistake I
made in patch 1. After fixing that, mmap-pread-seq-mt perf only drops 0.5%. The new
results on top of 6.7.0-rc1-mm-everything-2023-11-15-00-17 are at the end of the email.
I am preparing v2 and will send it out soon.
=========================================================================================
compiler/kconfig/rootfs/runtime/tbox_group/test/testcase:
gcc-13/defconfig/debian/300s/qemu-vm/mmap-xread-seq-mt/vm-scalability
commit:
6.7.0-rc1-mm-everything-2023-11-15-00-17+
6.7.0-rc1-split-folio-in-compaction+
6.7.0-rc1-folio-migration-in-compaction+
6.7.0-rc1-folio-migration-free-page-split+
6.7.0-rc1-folio-migration-free-page-split-sort-src+
6.7.0-rc1-mm-eve 6.7.0-rc1-split-folio-in-co 6.7.0-rc1-folio-migration-i 6.7.0-rc1-folio-migration-f 6.7.0-rc1-folio-migration-f
---------------- --------------------------- --------------------------- --------------------------- ---------------------------
%stddev %change %stddev %change %stddev %change %stddev %change %stddev
\ | \ | \ | \ | \
13041962 +16.1% 15142976 +5.0% 13690666 ± 6% +6.7% 13920441 +5.5% 13762582 vm-scalability.throughput
=========================================================================================
compiler/kconfig/rootfs/runtime/tbox_group/test/testcase:
gcc-13/defconfig/debian/300s/qemu-vm/mmap-pread-seq/vm-scalability
commit:
6.7.0-rc1-mm-everything-2023-11-15-00-17+
6.7.0-rc1-split-folio-in-compaction+
6.7.0-rc1-folio-migration-in-compaction+
6.7.0-rc1-folio-migration-free-page-split+
6.7.0-rc1-folio-migration-free-page-split-sort-src+
6.7.0-rc1-mm-eve 6.7.0-rc1-split-folio-in-co 6.7.0-rc1-folio-migration-i 6.7.0-rc1-folio-migration-f 6.7.0-rc1-folio-migration-f
---------------- --------------------------- --------------------------- --------------------------- ---------------------------
%stddev %change %stddev %change %stddev %change %stddev %change %stddev
\ | \ | \ | \ | \
14998168 -1.0% 14852803 -0.7% 14890569 -0.3% 14946766 -0.4% 14943302 vm-scalability.throughput
=========================================================================================
compiler/kconfig/rootfs/runtime/tbox_group/test/testcase:
gcc-13/defconfig/debian/300s/qemu-vm/lru-file-readtwice/vm-scalability
commit:
6.7.0-rc1-mm-everything-2023-11-15-00-17+
6.7.0-rc1-split-folio-in-compaction+
6.7.0-rc1-folio-migration-in-compaction+
6.7.0-rc1-folio-migration-free-page-split+
6.7.0-rc1-folio-migration-free-page-split-sort-src+
6.7.0-rc1-mm-eve 6.7.0-rc1-split-folio-in-co 6.7.0-rc1-folio-migration-i 6.7.0-rc1-folio-migration-f 6.7.0-rc1-folio-migration-f
---------------- --------------------------- --------------------------- --------------------------- ---------------------------
%stddev %change %stddev %change %stddev %change %stddev %change %stddev
\ | \ | \ | \ | \
6750930 ± 7% +41.5% 9549570 ± 2% +31.6% 8883545 ± 2% +33.1% 8982606 ± 2% +30.7% 8821667 ± 3% vm-scalability.throughput
=========================================================================================
compiler/kconfig/rootfs/runtime/tbox_group/test/testcase:
gcc-13/defconfig/debian/300s/qemu-vm/lru-file-mmap-read/vm-scalability
commit:
6.7.0-rc1-mm-everything-2023-11-15-00-17+
6.7.0-rc1-split-folio-in-compaction+
6.7.0-rc1-folio-migration-in-compaction+
6.7.0-rc1-folio-migration-free-page-split+
6.7.0-rc1-folio-migration-free-page-split-sort-src+
6.7.0-rc1-mm-eve 6.7.0-rc1-split-folio-in-co 6.7.0-rc1-folio-migration-i 6.7.0-rc1-folio-migration-f 6.7.0-rc1-folio-migration-f
---------------- --------------------------- --------------------------- --------------------------- ---------------------------
%stddev %change %stddev %change %stddev %change %stddev %change %stddev
\ | \ | \ | \ | \
5818610 ± 14% +43.1% 8329118 ± 2% +26.0% 7331427 ± 4% +23.2% 7170418 ± 4% +22.8% 7147458 ± 4% vm-scalability.throughput
=========================================================================================
compiler/kconfig/rootfs/runtime/tbox_group/test/testcase:
gcc-13/defconfig/debian/300s/qemu-vm/anon-r-rand/vm-scalability
commit:
6.7.0-rc1-mm-everything-2023-11-15-00-17+
6.7.0-rc1-split-folio-in-compaction+
6.7.0-rc1-folio-migration-in-compaction+
6.7.0-rc1-folio-migration-free-page-split+
6.7.0-rc1-folio-migration-free-page-split-sort-src+
6.7.0-rc1-mm-eve 6.7.0-rc1-split-folio-in-co 6.7.0-rc1-folio-migration-i 6.7.0-rc1-folio-migration-f 6.7.0-rc1-folio-migration-f
---------------- --------------------------- --------------------------- --------------------------- ---------------------------
%stddev %change %stddev %change %stddev %change %stddev %change %stddev
\ | \ | \ | \ | \
1973808 -0.2% 1969617 -0.5% 1964376 +0.3% 1979245 +1.0% 1993677 vm-scalability.throughput
=========================================================================================
compiler/kconfig/rootfs/runtime/size/tbox_group/test/testcase:
gcc-13/defconfig/debian/300s/512G/qemu-vm/anon-w-rand/vm-scalability
commit:
6.7.0-rc1-mm-everything-2023-11-15-00-17+
6.7.0-rc1-split-folio-in-compaction+
6.7.0-rc1-folio-migration-in-compaction+
6.7.0-rc1-folio-migration-free-page-split+
6.7.0-rc1-folio-migration-free-page-split-sort-src+
6.7.0-rc1-mm-eve 6.7.0-rc1-split-folio-in-co 6.7.0-rc1-folio-migration-i 6.7.0-rc1-folio-migration-f 6.7.0-rc1-folio-migration-f
---------------- --------------------------- --------------------------- --------------------------- ---------------------------
%stddev %change %stddev %change %stddev %change %stddev %change %stddev
\ | \ | \ | \ | \
479486 +0.3% 481036 +0.0% 479580 +2.3% 490310 +1.6% 487107 vm-scalability.throughput
=========================================================================================
compiler/kconfig/rootfs/runtime/size/tbox_group/test/testcase:
gcc-13/defconfig/debian/300s/512G/qemu-vm/anon-cow-rand/vm-scalability
commit:
6.7.0-rc1-mm-everything-2023-11-15-00-17+
6.7.0-rc1-split-folio-in-compaction+
6.7.0-rc1-folio-migration-in-compaction+
6.7.0-rc1-folio-migration-free-page-split+
6.7.0-rc1-folio-migration-free-page-split-sort-src+
6.7.0-rc1-mm-eve 6.7.0-rc1-split-folio-in-co 6.7.0-rc1-folio-migration-i 6.7.0-rc1-folio-migration-f 6.7.0-rc1-folio-migration-f
---------------- --------------------------- --------------------------- --------------------------- ---------------------------
%stddev %change %stddev %change %stddev %change %stddev %change %stddev
\ | \ | \ | \ | \
453237 +1.2% 458580 +0.5% 455455 +2.6% 464830 +2.6% 465211 vm-scalability.throughput
=========================================================================================
compiler/kconfig/rootfs/runtime/size/tbox_group/test/testcase:
gcc-13/defconfig/debian/300s/256G/qemu-vm/msync/vm-scalability
commit:
6.7.0-rc1-mm-everything-2023-11-15-00-17+
6.7.0-rc1-split-folio-in-compaction+
6.7.0-rc1-folio-migration-in-compaction+
6.7.0-rc1-folio-migration-free-page-split+
6.7.0-rc1-folio-migration-free-page-split-sort-src+
6.7.0-rc1-mm-eve 6.7.0-rc1-split-folio-in-co 6.7.0-rc1-folio-migration-i 6.7.0-rc1-folio-migration-f 6.7.0-rc1-folio-migration-f
---------------- --------------------------- --------------------------- --------------------------- ---------------------------
%stddev %change %stddev %change %stddev %change %stddev %change %stddev
\ | \ | \ | \ | \
1074404 ± 5% +78.7% 1920268 +78.7% 1920270 +77.2% 1903612 +78.0% 1912223 vm-scalability.throughput
=========================================================================================
compiler/kconfig/rootfs/runtime/size/tbox_group/test/testcase:
gcc-13/defconfig/debian/300s/256G/qemu-vm/msync-mt/vm-scalability
commit:
6.7.0-rc1-mm-everything-2023-11-15-00-17+
6.7.0-rc1-split-folio-in-compaction+
6.7.0-rc1-folio-migration-in-compaction+
6.7.0-rc1-folio-migration-free-page-split+
6.7.0-rc1-folio-migration-free-page-split-sort-src+
6.7.0-rc1-mm-eve 6.7.0-rc1-split-folio-in-co 6.7.0-rc1-folio-migration-i 6.7.0-rc1-folio-migration-f 6.7.0-rc1-folio-migration-f
---------------- --------------------------- --------------------------- --------------------------- ---------------------------
%stddev %change %stddev %change %stddev %change %stddev %change %stddev
\ | \ | \ | \ | \
2568988 +5.3% 2706227 +5.3% 2706214 +6.3% 2729716 +5.3% 2704539 vm-scalability.throughput
--
Best Regards,
Yan, Zi
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 854 bytes --]
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v1 0/4] Enable >0 order folio memory compaction
2024-01-02 20:50 ` Zi Yan
@ 2024-01-03 9:12 ` Ryan Roberts
2024-01-03 15:51 ` Zi Yan
0 siblings, 1 reply; 21+ messages in thread
From: Ryan Roberts @ 2024-01-03 9:12 UTC (permalink / raw)
To: Zi Yan
Cc: linux-mm, linux-kernel, Huang, Ying, Andrew Morton,
Matthew Wilcox (Oracle),
David Hildenbrand, Yin, Fengwei, Yu Zhao, Vlastimil Babka,
Kirill A . Shutemov, Johannes Weiner, Baolin Wang, Kemeng Shi,
Mel Gorman, Rohan Puri, Mcgrof Chamberlain, Adam Manzanares,
Vishal Moola (Oracle)
On 02/01/2024 20:50, Zi Yan wrote:
> On 21 Nov 2023, at 12:11, Ryan Roberts wrote:
>
>> On 21/11/2023 16:45, Zi Yan wrote:
>>> On 21 Nov 2023, at 10:46, Ryan Roberts wrote:
>>>
>>>>>
>>>>> vm-scalability results
>>>>> ===
>>>>>
>>>>> =========================================================================================
>>>>> compiler/kconfig/rootfs/runtime/tbox_group/test/testcase:
>>>>> gcc-13/defconfig/debian/300s/qemu-vm/mmap-xread-seq-mt/vm-scalability
>>>>>
>>>>> commit:
>>>>> 6.6.0-rc4-mm-everything-2023-10-21-02-40+
>>>>> 6.6.0-rc4-split-folio-in-compaction+
>>>>> 6.6.0-rc4-folio-migration-in-compaction+
>>>>> 6.6.0-rc4-folio-migration-free-page-split+
>>>>> 6.6.0-rc4-folio-migration-free-page-split-sort-src+
>>>>>
>>>>> 6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f
>>>>> ---------------- --------------------------- --------------------------- --------------------------- ---------------------------
>>>>> %stddev %change %stddev %change %stddev %change %stddev %change %stddev
>>>>> \ | \ | \ | \ | \
>>>>> 12896955 +2.7% 13249322 -4.0% 12385175 ± 5% +1.1% 13033951 -0.4% 12845698 vm-scalability.throughput
>>>>
>>>> Hi Zi,
>>>>
>>>> Are you able to add any commentary to these results as I'm struggling to
>>>> interpret them; Is a positive or negative change better (are they times or
>>>> rates?). What are the stddev values? The title suggests percent but the values
>>>> are huge - I'm trying to understand what the error bars look like - are the
>>>> swings real or noise?
>>>
>>> The metric is vm-scalability.throughput, so the larger the better. Some %stddev
>>> are not present since they are too small. For 6.6.0-rc4-folio-migration-in-compaction+,
>>> %stddev is greater than %change, so the change might be noise.
>>
>> Ahh got it - thanks!
>>
>>>
>>> Also, I talked to DavidH in last THP Cabal meeting about this. He suggested that
>>> there are a lot of noise in vm-scalability like what I have here and I should
>>> run more iterations and on bare metal. I am currently rerun them on a baremetal
>>> and more iterations on the existing VM and report the results later. Please
>>> note that the runs really take some time.
>>
>> Ahh ok, I'll wait for the bare metal numbers and will disregard these for now.
>> Thanks!
>
> It seems that the unexpected big mmap-pread-seq-mt perf drop came from the mistake I
> made in patch 1. After fixing that, mmap-pread-seq-mt perf only drops 0.5%. The new
> results on top of 6.7.0-rc1-mm-everything-2023-11-15-00-17 are at the end of the email.
Good news! I don't see the results for mmap-pread-seq-mt below - perhaps you
forgot to include it?
>
> I am preparing v2 and will send it out soon.
>
> =========================================================================================
> compiler/kconfig/rootfs/runtime/tbox_group/test/testcase:
> gcc-13/defconfig/debian/300s/qemu-vm/mmap-xread-seq-mt/vm-scalability
>
> commit:
> 6.7.0-rc1-mm-everything-2023-11-15-00-17+
> 6.7.0-rc1-split-folio-in-compaction+
> 6.7.0-rc1-folio-migration-in-compaction+
> 6.7.0-rc1-folio-migration-free-page-split+
> 6.7.0-rc1-folio-migration-free-page-split-sort-src+
>
> 6.7.0-rc1-mm-eve 6.7.0-rc1-split-folio-in-co 6.7.0-rc1-folio-migration-i 6.7.0-rc1-folio-migration-f 6.7.0-rc1-folio-migration-f
> ---------------- --------------------------- --------------------------- --------------------------- ---------------------------
> %stddev %change %stddev %change %stddev %change %stddev %change %stddev
> \ | \ | \ | \ | \
> 13041962 +16.1% 15142976 +5.0% 13690666 ± 6% +6.7% 13920441 +5.5% 13762582 vm-scalability.throughput
I'm still not sure I'm interpretting this correctly; is %change always relative
to 6.7.0-rc1-mm-everything-2023-11-15-00-17 or is it relative to the previous
commit?
If the former, then it looks like splitting the folios is actually faster than
migrating them whole?
>
> =========================================================================================
> compiler/kconfig/rootfs/runtime/tbox_group/test/testcase:
> gcc-13/defconfig/debian/300s/qemu-vm/mmap-pread-seq/vm-scalability
>
> commit:
> 6.7.0-rc1-mm-everything-2023-11-15-00-17+
> 6.7.0-rc1-split-folio-in-compaction+
> 6.7.0-rc1-folio-migration-in-compaction+
> 6.7.0-rc1-folio-migration-free-page-split+
> 6.7.0-rc1-folio-migration-free-page-split-sort-src+
>
> 6.7.0-rc1-mm-eve 6.7.0-rc1-split-folio-in-co 6.7.0-rc1-folio-migration-i 6.7.0-rc1-folio-migration-f 6.7.0-rc1-folio-migration-f
> ---------------- --------------------------- --------------------------- --------------------------- ---------------------------
> %stddev %change %stddev %change %stddev %change %stddev %change %stddev
> \ | \ | \ | \ | \
> 14998168 -1.0% 14852803 -0.7% 14890569 -0.3% 14946766 -0.4% 14943302 vm-scalability.throughput
>
> =========================================================================================
> compiler/kconfig/rootfs/runtime/tbox_group/test/testcase:
> gcc-13/defconfig/debian/300s/qemu-vm/lru-file-readtwice/vm-scalability
>
> commit:
> 6.7.0-rc1-mm-everything-2023-11-15-00-17+
> 6.7.0-rc1-split-folio-in-compaction+
> 6.7.0-rc1-folio-migration-in-compaction+
> 6.7.0-rc1-folio-migration-free-page-split+
> 6.7.0-rc1-folio-migration-free-page-split-sort-src+
>
> 6.7.0-rc1-mm-eve 6.7.0-rc1-split-folio-in-co 6.7.0-rc1-folio-migration-i 6.7.0-rc1-folio-migration-f 6.7.0-rc1-folio-migration-f
> ---------------- --------------------------- --------------------------- --------------------------- ---------------------------
> %stddev %change %stddev %change %stddev %change %stddev %change %stddev
> \ | \ | \ | \ | \
> 6750930 ± 7% +41.5% 9549570 ± 2% +31.6% 8883545 ± 2% +33.1% 8982606 ± 2% +30.7% 8821667 ± 3% vm-scalability.throughput
>
> =========================================================================================
> compiler/kconfig/rootfs/runtime/tbox_group/test/testcase:
> gcc-13/defconfig/debian/300s/qemu-vm/lru-file-mmap-read/vm-scalability
>
> commit:
> 6.7.0-rc1-mm-everything-2023-11-15-00-17+
> 6.7.0-rc1-split-folio-in-compaction+
> 6.7.0-rc1-folio-migration-in-compaction+
> 6.7.0-rc1-folio-migration-free-page-split+
> 6.7.0-rc1-folio-migration-free-page-split-sort-src+
>
> 6.7.0-rc1-mm-eve 6.7.0-rc1-split-folio-in-co 6.7.0-rc1-folio-migration-i 6.7.0-rc1-folio-migration-f 6.7.0-rc1-folio-migration-f
> ---------------- --------------------------- --------------------------- --------------------------- ---------------------------
> %stddev %change %stddev %change %stddev %change %stddev %change %stddev
> \ | \ | \ | \ | \
> 5818610 ± 14% +43.1% 8329118 ± 2% +26.0% 7331427 ± 4% +23.2% 7170418 ± 4% +22.8% 7147458 ± 4% vm-scalability.throughput
>
> =========================================================================================
> compiler/kconfig/rootfs/runtime/tbox_group/test/testcase:
> gcc-13/defconfig/debian/300s/qemu-vm/anon-r-rand/vm-scalability
>
> commit:
> 6.7.0-rc1-mm-everything-2023-11-15-00-17+
> 6.7.0-rc1-split-folio-in-compaction+
> 6.7.0-rc1-folio-migration-in-compaction+
> 6.7.0-rc1-folio-migration-free-page-split+
> 6.7.0-rc1-folio-migration-free-page-split-sort-src+
>
> 6.7.0-rc1-mm-eve 6.7.0-rc1-split-folio-in-co 6.7.0-rc1-folio-migration-i 6.7.0-rc1-folio-migration-f 6.7.0-rc1-folio-migration-f
> ---------------- --------------------------- --------------------------- --------------------------- ---------------------------
> %stddev %change %stddev %change %stddev %change %stddev %change %stddev
> \ | \ | \ | \ | \
> 1973808 -0.2% 1969617 -0.5% 1964376 +0.3% 1979245 +1.0% 1993677 vm-scalability.throughput
>
> =========================================================================================
> compiler/kconfig/rootfs/runtime/size/tbox_group/test/testcase:
> gcc-13/defconfig/debian/300s/512G/qemu-vm/anon-w-rand/vm-scalability
>
> commit:
> 6.7.0-rc1-mm-everything-2023-11-15-00-17+
> 6.7.0-rc1-split-folio-in-compaction+
> 6.7.0-rc1-folio-migration-in-compaction+
> 6.7.0-rc1-folio-migration-free-page-split+
> 6.7.0-rc1-folio-migration-free-page-split-sort-src+
>
> 6.7.0-rc1-mm-eve 6.7.0-rc1-split-folio-in-co 6.7.0-rc1-folio-migration-i 6.7.0-rc1-folio-migration-f 6.7.0-rc1-folio-migration-f
> ---------------- --------------------------- --------------------------- --------------------------- ---------------------------
> %stddev %change %stddev %change %stddev %change %stddev %change %stddev
> \ | \ | \ | \ | \
> 479486 +0.3% 481036 +0.0% 479580 +2.3% 490310 +1.6% 487107 vm-scalability.throughput
>
> =========================================================================================
> compiler/kconfig/rootfs/runtime/size/tbox_group/test/testcase:
> gcc-13/defconfig/debian/300s/512G/qemu-vm/anon-cow-rand/vm-scalability
>
> commit:
> 6.7.0-rc1-mm-everything-2023-11-15-00-17+
> 6.7.0-rc1-split-folio-in-compaction+
> 6.7.0-rc1-folio-migration-in-compaction+
> 6.7.0-rc1-folio-migration-free-page-split+
> 6.7.0-rc1-folio-migration-free-page-split-sort-src+
>
> 6.7.0-rc1-mm-eve 6.7.0-rc1-split-folio-in-co 6.7.0-rc1-folio-migration-i 6.7.0-rc1-folio-migration-f 6.7.0-rc1-folio-migration-f
> ---------------- --------------------------- --------------------------- --------------------------- ---------------------------
> %stddev %change %stddev %change %stddev %change %stddev %change %stddev
> \ | \ | \ | \ | \
> 453237 +1.2% 458580 +0.5% 455455 +2.6% 464830 +2.6% 465211 vm-scalability.throughput
>
> =========================================================================================
> compiler/kconfig/rootfs/runtime/size/tbox_group/test/testcase:
> gcc-13/defconfig/debian/300s/256G/qemu-vm/msync/vm-scalability
>
> commit:
> 6.7.0-rc1-mm-everything-2023-11-15-00-17+
> 6.7.0-rc1-split-folio-in-compaction+
> 6.7.0-rc1-folio-migration-in-compaction+
> 6.7.0-rc1-folio-migration-free-page-split+
> 6.7.0-rc1-folio-migration-free-page-split-sort-src+
>
> 6.7.0-rc1-mm-eve 6.7.0-rc1-split-folio-in-co 6.7.0-rc1-folio-migration-i 6.7.0-rc1-folio-migration-f 6.7.0-rc1-folio-migration-f
> ---------------- --------------------------- --------------------------- --------------------------- ---------------------------
> %stddev %change %stddev %change %stddev %change %stddev %change %stddev
> \ | \ | \ | \ | \
> 1074404 ± 5% +78.7% 1920268 +78.7% 1920270 +77.2% 1903612 +78.0% 1912223 vm-scalability.throughput
>
> =========================================================================================
> compiler/kconfig/rootfs/runtime/size/tbox_group/test/testcase:
> gcc-13/defconfig/debian/300s/256G/qemu-vm/msync-mt/vm-scalability
>
> commit:
> 6.7.0-rc1-mm-everything-2023-11-15-00-17+
> 6.7.0-rc1-split-folio-in-compaction+
> 6.7.0-rc1-folio-migration-in-compaction+
> 6.7.0-rc1-folio-migration-free-page-split+
> 6.7.0-rc1-folio-migration-free-page-split-sort-src+
>
> 6.7.0-rc1-mm-eve 6.7.0-rc1-split-folio-in-co 6.7.0-rc1-folio-migration-i 6.7.0-rc1-folio-migration-f 6.7.0-rc1-folio-migration-f
> ---------------- --------------------------- --------------------------- --------------------------- ---------------------------
> %stddev %change %stddev %change %stddev %change %stddev %change %stddev
> \ | \ | \ | \ | \
> 2568988 +5.3% 2706227 +5.3% 2706214 +6.3% 2729716 +5.3% 2704539 vm-scalability.throughput
>
>
> --
> Best Regards,
> Yan, Zi
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v1 0/4] Enable >0 order folio memory compaction
2024-01-03 9:12 ` Ryan Roberts
@ 2024-01-03 15:51 ` Zi Yan
2024-01-05 22:56 ` Zi Yan
0 siblings, 1 reply; 21+ messages in thread
From: Zi Yan @ 2024-01-03 15:51 UTC (permalink / raw)
To: Ryan Roberts
Cc: linux-mm, linux-kernel, "Huang, Ying",
Andrew Morton, "Matthew Wilcox (Oracle)",
David Hildenbrand, "Yin, Fengwei",
Yu Zhao, Vlastimil Babka, "Kirill A . Shutemov",
Johannes Weiner, Baolin Wang, Kemeng Shi, Mel Gorman, Rohan Puri,
Mcgrof Chamberlain, Adam Manzanares,
"Vishal Moola (Oracle)"
[-- Attachment #1: Type: text/plain, Size: 14529 bytes --]
On 3 Jan 2024, at 4:12, Ryan Roberts wrote:
> On 02/01/2024 20:50, Zi Yan wrote:
>> On 21 Nov 2023, at 12:11, Ryan Roberts wrote:
>>
>>> On 21/11/2023 16:45, Zi Yan wrote:
>>>> On 21 Nov 2023, at 10:46, Ryan Roberts wrote:
>>>>
>>>>>>
>>>>>> vm-scalability results
>>>>>> ===
>>>>>>
>>>>>> =========================================================================================
>>>>>> compiler/kconfig/rootfs/runtime/tbox_group/test/testcase:
>>>>>> gcc-13/defconfig/debian/300s/qemu-vm/mmap-xread-seq-mt/vm-scalability
>>>>>>
>>>>>> commit:
>>>>>> 6.6.0-rc4-mm-everything-2023-10-21-02-40+
>>>>>> 6.6.0-rc4-split-folio-in-compaction+
>>>>>> 6.6.0-rc4-folio-migration-in-compaction+
>>>>>> 6.6.0-rc4-folio-migration-free-page-split+
>>>>>> 6.6.0-rc4-folio-migration-free-page-split-sort-src+
>>>>>>
>>>>>> 6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f
>>>>>> ---------------- --------------------------- --------------------------- --------------------------- ---------------------------
>>>>>> %stddev %change %stddev %change %stddev %change %stddev %change %stddev
>>>>>> \ | \ | \ | \ | \
>>>>>> 12896955 +2.7% 13249322 -4.0% 12385175 ± 5% +1.1% 13033951 -0.4% 12845698 vm-scalability.throughput
>>>>>
>>>>> Hi Zi,
>>>>>
>>>>> Are you able to add any commentary to these results as I'm struggling to
>>>>> interpret them; Is a positive or negative change better (are they times or
>>>>> rates?). What are the stddev values? The title suggests percent but the values
>>>>> are huge - I'm trying to understand what the error bars look like - are the
>>>>> swings real or noise?
>>>>
>>>> The metric is vm-scalability.throughput, so the larger the better. Some %stddev
>>>> are not present since they are too small. For 6.6.0-rc4-folio-migration-in-compaction+,
>>>> %stddev is greater than %change, so the change might be noise.
>>>
>>> Ahh got it - thanks!
>>>
>>>>
>>>> Also, I talked to DavidH in last THP Cabal meeting about this. He suggested that
>>>> there are a lot of noise in vm-scalability like what I have here and I should
>>>> run more iterations and on bare metal. I am currently rerun them on a baremetal
>>>> and more iterations on the existing VM and report the results later. Please
>>>> note that the runs really take some time.
>>>
>>> Ahh ok, I'll wait for the bare metal numbers and will disregard these for now.
>>> Thanks!
>>
>> It seems that the unexpected big mmap-pread-seq-mt perf drop came from the mistake I
>> made in patch 1. After fixing that, mmap-pread-seq-mt perf only drops 0.5%. The new
>> results on top of 6.7.0-rc1-mm-everything-2023-11-15-00-17 are at the end of the email.
>
> Good news! I don't see the results for mmap-pread-seq-mt below - perhaps you
> forgot to include it?
The stats below only shows significant changes and mmap-pread-seq-mt delta is less
than 5%, thus it is not shown.
>
>>
>> I am preparing v2 and will send it out soon.
>>
>> =========================================================================================
>> compiler/kconfig/rootfs/runtime/tbox_group/test/testcase:
>> gcc-13/defconfig/debian/300s/qemu-vm/mmap-xread-seq-mt/vm-scalability
>>
>> commit:
>> 6.7.0-rc1-mm-everything-2023-11-15-00-17+
>> 6.7.0-rc1-split-folio-in-compaction+
>> 6.7.0-rc1-folio-migration-in-compaction+
>> 6.7.0-rc1-folio-migration-free-page-split+
>> 6.7.0-rc1-folio-migration-free-page-split-sort-src+
>>
>> 6.7.0-rc1-mm-eve 6.7.0-rc1-split-folio-in-co 6.7.0-rc1-folio-migration-i 6.7.0-rc1-folio-migration-f 6.7.0-rc1-folio-migration-f
>> ---------------- --------------------------- --------------------------- --------------------------- ---------------------------
>> %stddev %change %stddev %change %stddev %change %stddev %change %stddev
>> \ | \ | \ | \ | \
>> 13041962 +16.1% 15142976 +5.0% 13690666 ± 6% +6.7% 13920441 +5.5% 13762582 vm-scalability.throughput
>
> I'm still not sure I'm interpretting this correctly; is %change always relative
> to 6.7.0-rc1-mm-everything-2023-11-15-00-17 or is it relative to the previous
> commit?
The former, always relative to 6.7.0-rc1-mm-everything-2023-11-15-00-17.
>
> If the former, then it looks like splitting the folios is actually faster than
> migrating them whole?
Yes, I will look into it when I am preparing the next version.
>
>
>>
>> =========================================================================================
>> compiler/kconfig/rootfs/runtime/tbox_group/test/testcase:
>> gcc-13/defconfig/debian/300s/qemu-vm/mmap-pread-seq/vm-scalability
>>
>> commit:
>> 6.7.0-rc1-mm-everything-2023-11-15-00-17+
>> 6.7.0-rc1-split-folio-in-compaction+
>> 6.7.0-rc1-folio-migration-in-compaction+
>> 6.7.0-rc1-folio-migration-free-page-split+
>> 6.7.0-rc1-folio-migration-free-page-split-sort-src+
>>
>> 6.7.0-rc1-mm-eve 6.7.0-rc1-split-folio-in-co 6.7.0-rc1-folio-migration-i 6.7.0-rc1-folio-migration-f 6.7.0-rc1-folio-migration-f
>> ---------------- --------------------------- --------------------------- --------------------------- ---------------------------
>> %stddev %change %stddev %change %stddev %change %stddev %change %stddev
>> \ | \ | \ | \ | \
>> 14998168 -1.0% 14852803 -0.7% 14890569 -0.3% 14946766 -0.4% 14943302 vm-scalability.throughput
>>
>> =========================================================================================
>> compiler/kconfig/rootfs/runtime/tbox_group/test/testcase:
>> gcc-13/defconfig/debian/300s/qemu-vm/lru-file-readtwice/vm-scalability
>>
>> commit:
>> 6.7.0-rc1-mm-everything-2023-11-15-00-17+
>> 6.7.0-rc1-split-folio-in-compaction+
>> 6.7.0-rc1-folio-migration-in-compaction+
>> 6.7.0-rc1-folio-migration-free-page-split+
>> 6.7.0-rc1-folio-migration-free-page-split-sort-src+
>>
>> 6.7.0-rc1-mm-eve 6.7.0-rc1-split-folio-in-co 6.7.0-rc1-folio-migration-i 6.7.0-rc1-folio-migration-f 6.7.0-rc1-folio-migration-f
>> ---------------- --------------------------- --------------------------- --------------------------- ---------------------------
>> %stddev %change %stddev %change %stddev %change %stddev %change %stddev
>> \ | \ | \ | \ | \
>> 6750930 ± 7% +41.5% 9549570 ± 2% +31.6% 8883545 ± 2% +33.1% 8982606 ± 2% +30.7% 8821667 ± 3% vm-scalability.throughput
>>
>> =========================================================================================
>> compiler/kconfig/rootfs/runtime/tbox_group/test/testcase:
>> gcc-13/defconfig/debian/300s/qemu-vm/lru-file-mmap-read/vm-scalability
>>
>> commit:
>> 6.7.0-rc1-mm-everything-2023-11-15-00-17+
>> 6.7.0-rc1-split-folio-in-compaction+
>> 6.7.0-rc1-folio-migration-in-compaction+
>> 6.7.0-rc1-folio-migration-free-page-split+
>> 6.7.0-rc1-folio-migration-free-page-split-sort-src+
>>
>> 6.7.0-rc1-mm-eve 6.7.0-rc1-split-folio-in-co 6.7.0-rc1-folio-migration-i 6.7.0-rc1-folio-migration-f 6.7.0-rc1-folio-migration-f
>> ---------------- --------------------------- --------------------------- --------------------------- ---------------------------
>> %stddev %change %stddev %change %stddev %change %stddev %change %stddev
>> \ | \ | \ | \ | \
>> 5818610 ± 14% +43.1% 8329118 ± 2% +26.0% 7331427 ± 4% +23.2% 7170418 ± 4% +22.8% 7147458 ± 4% vm-scalability.throughput
>>
>> =========================================================================================
>> compiler/kconfig/rootfs/runtime/tbox_group/test/testcase:
>> gcc-13/defconfig/debian/300s/qemu-vm/anon-r-rand/vm-scalability
>>
>> commit:
>> 6.7.0-rc1-mm-everything-2023-11-15-00-17+
>> 6.7.0-rc1-split-folio-in-compaction+
>> 6.7.0-rc1-folio-migration-in-compaction+
>> 6.7.0-rc1-folio-migration-free-page-split+
>> 6.7.0-rc1-folio-migration-free-page-split-sort-src+
>>
>> 6.7.0-rc1-mm-eve 6.7.0-rc1-split-folio-in-co 6.7.0-rc1-folio-migration-i 6.7.0-rc1-folio-migration-f 6.7.0-rc1-folio-migration-f
>> ---------------- --------------------------- --------------------------- --------------------------- ---------------------------
>> %stddev %change %stddev %change %stddev %change %stddev %change %stddev
>> \ | \ | \ | \ | \
>> 1973808 -0.2% 1969617 -0.5% 1964376 +0.3% 1979245 +1.0% 1993677 vm-scalability.throughput
>>
>> =========================================================================================
>> compiler/kconfig/rootfs/runtime/size/tbox_group/test/testcase:
>> gcc-13/defconfig/debian/300s/512G/qemu-vm/anon-w-rand/vm-scalability
>>
>> commit:
>> 6.7.0-rc1-mm-everything-2023-11-15-00-17+
>> 6.7.0-rc1-split-folio-in-compaction+
>> 6.7.0-rc1-folio-migration-in-compaction+
>> 6.7.0-rc1-folio-migration-free-page-split+
>> 6.7.0-rc1-folio-migration-free-page-split-sort-src+
>>
>> 6.7.0-rc1-mm-eve 6.7.0-rc1-split-folio-in-co 6.7.0-rc1-folio-migration-i 6.7.0-rc1-folio-migration-f 6.7.0-rc1-folio-migration-f
>> ---------------- --------------------------- --------------------------- --------------------------- ---------------------------
>> %stddev %change %stddev %change %stddev %change %stddev %change %stddev
>> \ | \ | \ | \ | \
>> 479486 +0.3% 481036 +0.0% 479580 +2.3% 490310 +1.6% 487107 vm-scalability.throughput
>>
>> =========================================================================================
>> compiler/kconfig/rootfs/runtime/size/tbox_group/test/testcase:
>> gcc-13/defconfig/debian/300s/512G/qemu-vm/anon-cow-rand/vm-scalability
>>
>> commit:
>> 6.7.0-rc1-mm-everything-2023-11-15-00-17+
>> 6.7.0-rc1-split-folio-in-compaction+
>> 6.7.0-rc1-folio-migration-in-compaction+
>> 6.7.0-rc1-folio-migration-free-page-split+
>> 6.7.0-rc1-folio-migration-free-page-split-sort-src+
>>
>> 6.7.0-rc1-mm-eve 6.7.0-rc1-split-folio-in-co 6.7.0-rc1-folio-migration-i 6.7.0-rc1-folio-migration-f 6.7.0-rc1-folio-migration-f
>> ---------------- --------------------------- --------------------------- --------------------------- ---------------------------
>> %stddev %change %stddev %change %stddev %change %stddev %change %stddev
>> \ | \ | \ | \ | \
>> 453237 +1.2% 458580 +0.5% 455455 +2.6% 464830 +2.6% 465211 vm-scalability.throughput
>>
>> =========================================================================================
>> compiler/kconfig/rootfs/runtime/size/tbox_group/test/testcase:
>> gcc-13/defconfig/debian/300s/256G/qemu-vm/msync/vm-scalability
>>
>> commit:
>> 6.7.0-rc1-mm-everything-2023-11-15-00-17+
>> 6.7.0-rc1-split-folio-in-compaction+
>> 6.7.0-rc1-folio-migration-in-compaction+
>> 6.7.0-rc1-folio-migration-free-page-split+
>> 6.7.0-rc1-folio-migration-free-page-split-sort-src+
>>
>> 6.7.0-rc1-mm-eve 6.7.0-rc1-split-folio-in-co 6.7.0-rc1-folio-migration-i 6.7.0-rc1-folio-migration-f 6.7.0-rc1-folio-migration-f
>> ---------------- --------------------------- --------------------------- --------------------------- ---------------------------
>> %stddev %change %stddev %change %stddev %change %stddev %change %stddev
>> \ | \ | \ | \ | \
>> 1074404 ± 5% +78.7% 1920268 +78.7% 1920270 +77.2% 1903612 +78.0% 1912223 vm-scalability.throughput
>>
>> =========================================================================================
>> compiler/kconfig/rootfs/runtime/size/tbox_group/test/testcase:
>> gcc-13/defconfig/debian/300s/256G/qemu-vm/msync-mt/vm-scalability
>>
>> commit:
>> 6.7.0-rc1-mm-everything-2023-11-15-00-17+
>> 6.7.0-rc1-split-folio-in-compaction+
>> 6.7.0-rc1-folio-migration-in-compaction+
>> 6.7.0-rc1-folio-migration-free-page-split+
>> 6.7.0-rc1-folio-migration-free-page-split-sort-src+
>>
>> 6.7.0-rc1-mm-eve 6.7.0-rc1-split-folio-in-co 6.7.0-rc1-folio-migration-i 6.7.0-rc1-folio-migration-f 6.7.0-rc1-folio-migration-f
>> ---------------- --------------------------- --------------------------- --------------------------- ---------------------------
>> %stddev %change %stddev %change %stddev %change %stddev %change %stddev
>> \ | \ | \ | \ | \
>> 2568988 +5.3% 2706227 +5.3% 2706214 +6.3% 2729716 +5.3% 2704539 vm-scalability.throughput
>>
>>
>> --
>> Best Regards,
>> Yan, Zi
--
Best Regards,
Yan, Zi
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 854 bytes --]
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v1 0/4] Enable >0 order folio memory compaction
2024-01-03 15:51 ` Zi Yan
@ 2024-01-05 22:56 ` Zi Yan
0 siblings, 0 replies; 21+ messages in thread
From: Zi Yan @ 2024-01-05 22:56 UTC (permalink / raw)
To: Ryan Roberts
Cc: linux-mm, linux-kernel, "Huang, Ying",
Andrew Morton, "Matthew Wilcox (Oracle)",
David Hildenbrand, "Yin, Fengwei",
Yu Zhao, Vlastimil Babka, "Kirill A . Shutemov",
Johannes Weiner, Baolin Wang, Kemeng Shi, Mel Gorman, Rohan Puri,
Mcgrof Chamberlain, Adam Manzanares,
"Vishal Moola (Oracle)"
[-- Attachment #1: Type: text/plain, Size: 5490 bytes --]
On 3 Jan 2024, at 10:51, Zi Yan wrote:
> On 3 Jan 2024, at 4:12, Ryan Roberts wrote:
>
>> On 02/01/2024 20:50, Zi Yan wrote:
>>> On 21 Nov 2023, at 12:11, Ryan Roberts wrote:
>>>
>>>> On 21/11/2023 16:45, Zi Yan wrote:
>>>>> On 21 Nov 2023, at 10:46, Ryan Roberts wrote:
>>>>>
>>>>>>>
>>>>>>> vm-scalability results
>>>>>>> ===
>>>>>>>
>>>>>>> =========================================================================================
>>>>>>> compiler/kconfig/rootfs/runtime/tbox_group/test/testcase:
>>>>>>> gcc-13/defconfig/debian/300s/qemu-vm/mmap-xread-seq-mt/vm-scalability
>>>>>>>
>>>>>>> commit:
>>>>>>> 6.6.0-rc4-mm-everything-2023-10-21-02-40+
>>>>>>> 6.6.0-rc4-split-folio-in-compaction+
>>>>>>> 6.6.0-rc4-folio-migration-in-compaction+
>>>>>>> 6.6.0-rc4-folio-migration-free-page-split+
>>>>>>> 6.6.0-rc4-folio-migration-free-page-split-sort-src+
>>>>>>>
>>>>>>> 6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f
>>>>>>> ---------------- --------------------------- --------------------------- --------------------------- ---------------------------
>>>>>>> %stddev %change %stddev %change %stddev %change %stddev %change %stddev
>>>>>>> \ | \ | \ | \ | \
>>>>>>> 12896955 +2.7% 13249322 -4.0% 12385175 ± 5% +1.1% 13033951 -0.4% 12845698 vm-scalability.throughput
>>>>>>
>>>>>> Hi Zi,
>>>>>>
>>>>>> Are you able to add any commentary to these results as I'm struggling to
>>>>>> interpret them; Is a positive or negative change better (are they times or
>>>>>> rates?). What are the stddev values? The title suggests percent but the values
>>>>>> are huge - I'm trying to understand what the error bars look like - are the
>>>>>> swings real or noise?
>>>>>
>>>>> The metric is vm-scalability.throughput, so the larger the better. Some %stddev
>>>>> are not present since they are too small. For 6.6.0-rc4-folio-migration-in-compaction+,
>>>>> %stddev is greater than %change, so the change might be noise.
>>>>
>>>> Ahh got it - thanks!
>>>>
>>>>>
>>>>> Also, I talked to DavidH in last THP Cabal meeting about this. He suggested that
>>>>> there are a lot of noise in vm-scalability like what I have here and I should
>>>>> run more iterations and on bare metal. I am currently rerun them on a baremetal
>>>>> and more iterations on the existing VM and report the results later. Please
>>>>> note that the runs really take some time.
>>>>
>>>> Ahh ok, I'll wait for the bare metal numbers and will disregard these for now.
>>>> Thanks!
>>>
>>> It seems that the unexpected big mmap-pread-seq-mt perf drop came from the mistake I
>>> made in patch 1. After fixing that, mmap-pread-seq-mt perf only drops 0.5%. The new
>>> results on top of 6.7.0-rc1-mm-everything-2023-11-15-00-17 are at the end of the email.
>>
>> Good news! I don't see the results for mmap-pread-seq-mt below - perhaps you
>> forgot to include it?
>
> The stats below only shows significant changes and mmap-pread-seq-mt delta is less
> than 5%, thus it is not shown.
>
>>
>>>
>>> I am preparing v2 and will send it out soon.
>>>
>>> =========================================================================================
>>> compiler/kconfig/rootfs/runtime/tbox_group/test/testcase:
>>> gcc-13/defconfig/debian/300s/qemu-vm/mmap-xread-seq-mt/vm-scalability
>>>
>>> commit:
>>> 6.7.0-rc1-mm-everything-2023-11-15-00-17+
>>> 6.7.0-rc1-split-folio-in-compaction+
>>> 6.7.0-rc1-folio-migration-in-compaction+
>>> 6.7.0-rc1-folio-migration-free-page-split+
>>> 6.7.0-rc1-folio-migration-free-page-split-sort-src+
>>>
>>> 6.7.0-rc1-mm-eve 6.7.0-rc1-split-folio-in-co 6.7.0-rc1-folio-migration-i 6.7.0-rc1-folio-migration-f 6.7.0-rc1-folio-migration-f
>>> ---------------- --------------------------- --------------------------- --------------------------- ---------------------------
>>> %stddev %change %stddev %change %stddev %change %stddev %change %stddev
>>> \ | \ | \ | \ | \
>>> 13041962 +16.1% 15142976 +5.0% 13690666 ± 6% +6.7% 13920441 +5.5% 13762582 vm-scalability.throughput
>>
>> I'm still not sure I'm interpretting this correctly; is %change always relative
>> to 6.7.0-rc1-mm-everything-2023-11-15-00-17 or is it relative to the previous
>> commit?
>
> The former, always relative to 6.7.0-rc1-mm-everything-2023-11-15-00-17.
>
>>
>> If the former, then it looks like splitting the folios is actually faster than
>> migrating them whole?
>
> Yes, I will look into it when I am preparing the next version.
>
The reason seems to be that compaction ends early when migrating folios as a whole.
It happens when a order-0 folio is migrated and there is no order-0 free page,
then migrate_pages() returns -ENOMEM making compact_zone() stop compaction (for
higher order folios, they would be split). This should be fixed by enabling
free page split optimization, but the perf number does not say so. Let me dig more.
--
Best Regards,
Yan, Zi
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 854 bytes --]
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v1 2/4] mm/compaction: add support for >0 order folio memory compaction.
2023-11-13 17:01 ` [PATCH v1 2/4] mm/compaction: add support for >0 order folio memory compaction Zi Yan
@ 2024-01-09 15:18 ` Ryan Roberts
2024-01-09 15:25 ` Zi Yan
0 siblings, 1 reply; 21+ messages in thread
From: Ryan Roberts @ 2024-01-09 15:18 UTC (permalink / raw)
To: Zi Yan, linux-mm, linux-kernel
Cc: Huang, Ying, Andrew Morton, Matthew Wilcox (Oracle),
David Hildenbrand, Yin, Fengwei, Yu Zhao, Vlastimil Babka,
Kirill A . Shutemov, Johannes Weiner, Baolin Wang, Kemeng Shi,
Mel Gorman, Rohan Puri, Mcgrof Chamberlain, Adam Manzanares,
Vishal Moola (Oracle)
On 13/11/2023 17:01, Zi Yan wrote:
> From: Zi Yan <ziy@nvidia.com>
>
> Before, memory compaction only migrates order-0 folios and skips >0 order
> folios. This commit adds support for >0 order folio compaction by keeping
> isolated free pages at their original size without splitting them into
> order-0 pages and using them directly during migration process.
>
> What is different from the prior implementation:
> 1. All isolated free pages are kept in a MAX_ORDER+1 array of page lists,
> where each page list stores free pages in the same order.
> 2. All free pages are not post_alloc_hook() processed nor buddy pages,
> although their orders are stored in first page's private like buddy
> pages.
> 3. During migration, in new page allocation time (i.e., in
> compaction_alloc()), free pages are then processed by post_alloc_hook().
> When migration fails and a new page is returned (i.e., in
> compaction_free()), free pages are restored by reversing the
> post_alloc_hook() operations.
>
> Step 3 is done for a latter optimization that splitting and/or merging free
> pages during compaction becomes easier.
>
> Signed-off-by: Zi Yan <ziy@nvidia.com>
> ---
> mm/compaction.c | 160 ++++++++++++++++++++++++++++++------------------
> mm/internal.h | 7 ++-
> 2 files changed, 108 insertions(+), 59 deletions(-)
>
> diff --git a/mm/compaction.c b/mm/compaction.c
> index 5217dd35b493..ec6b5cc7e907 100644
> --- a/mm/compaction.c
> +++ b/mm/compaction.c
> @@ -66,45 +66,64 @@ static inline void count_compact_events(enum vm_event_item item, long delta)
> #define COMPACTION_HPAGE_ORDER (PMD_SHIFT - PAGE_SHIFT)
> #endif
>
> -static unsigned long release_freepages(struct list_head *freelist)
> +static void init_page_list(struct page_list *p)
> {
> - struct page *page, *next;
> - unsigned long high_pfn = 0;
> -
> - list_for_each_entry_safe(page, next, freelist, lru) {
> - unsigned long pfn = page_to_pfn(page);
> - list_del(&page->lru);
> - __free_page(page);
> - if (pfn > high_pfn)
> - high_pfn = pfn;
> - }
> -
> - return high_pfn;
> + INIT_LIST_HEAD(&p->pages);
> + p->nr_pages = 0;
> }
>
> -static void split_map_pages(struct list_head *list)
> +static void split_map_pages(struct page_list *freepages)
> {
> unsigned int i, order, nr_pages;
> struct page *page, *next;
> LIST_HEAD(tmp_list);
>
> - list_for_each_entry_safe(page, next, list, lru) {
> - list_del(&page->lru);
> + for (order = 0; order <= MAX_ORDER; order++) {
> + freepages[order].nr_pages = 0;
> +
> + list_for_each_entry_safe(page, next, &freepages[order].pages, lru) {
> + list_del(&page->lru);
>
> - order = page_private(page);
> - nr_pages = 1 << order;
> + nr_pages = 1 << order;
>
> - post_alloc_hook(page, order, __GFP_MOVABLE);
> - if (order)
> - split_page(page, order);
> + post_alloc_hook(page, order, __GFP_MOVABLE);
> + if (order)
> + split_page(page, order);
>
> - for (i = 0; i < nr_pages; i++) {
> - list_add(&page->lru, &tmp_list);
> - page++;
> + for (i = 0; i < nr_pages; i++) {
> + list_add(&page->lru, &tmp_list);
> + page++;
> + }
> + freepages[order].nr_pages += nr_pages;
> }
> + list_splice(&tmp_list, &freepages[order].pages);
I think this should be list_splice_init() since you are reusing tmp_list in each
iteration of the outer loop?
Thanks,
Ryan
> }
> +}
>
> - list_splice(&tmp_list, list);
> +static unsigned long release_free_list(struct page_list *freepages)
> +{
> + int order;
> + unsigned long high_pfn = 0;
> +
> + for (order = 0; order <= MAX_ORDER; order++) {
> + struct page *page, *next;
> +
> + list_for_each_entry_safe(page, next, &freepages[order].pages, lru) {
> + unsigned long pfn = page_to_pfn(page);
> +
> + list_del(&page->lru);
> + /*
> + * Convert free pages into post allocation pages, so
> + * that we can free them via __free_page.
> + */
> + post_alloc_hook(page, order, __GFP_MOVABLE);
> + __free_pages(page, order);
> + if (pfn > high_pfn)
> + high_pfn = pfn;
> + }
> + freepages[order].nr_pages = 0;
> + }
> + return high_pfn;
> }
>
> #ifdef CONFIG_COMPACTION
> @@ -583,7 +602,7 @@ static bool compact_unlock_should_abort(spinlock_t *lock,
> static unsigned long isolate_freepages_block(struct compact_control *cc,
> unsigned long *start_pfn,
> unsigned long end_pfn,
> - struct list_head *freelist,
> + struct page_list *freelist,
> unsigned int stride,
> bool strict)
> {
> @@ -657,7 +676,8 @@ static unsigned long isolate_freepages_block(struct compact_control *cc,
> nr_scanned += isolated - 1;
> total_isolated += isolated;
> cc->nr_freepages += isolated;
> - list_add_tail(&page->lru, freelist);
> + list_add_tail(&page->lru, &freelist[order].pages);
> + freelist[order].nr_pages++;
>
> if (!strict && cc->nr_migratepages <= cc->nr_freepages) {
> blockpfn += isolated;
> @@ -722,7 +742,11 @@ isolate_freepages_range(struct compact_control *cc,
> unsigned long start_pfn, unsigned long end_pfn)
> {
> unsigned long isolated, pfn, block_start_pfn, block_end_pfn;
> - LIST_HEAD(freelist);
> + int order;
> + struct page_list tmp_freepages[MAX_ORDER + 1];
> +
> + for (order = 0; order <= MAX_ORDER; order++)
> + init_page_list(&tmp_freepages[order]);
>
> pfn = start_pfn;
> block_start_pfn = pageblock_start_pfn(pfn);
> @@ -753,7 +777,7 @@ isolate_freepages_range(struct compact_control *cc,
> break;
>
> isolated = isolate_freepages_block(cc, &isolate_start_pfn,
> - block_end_pfn, &freelist, 0, true);
> + block_end_pfn, tmp_freepages, 0, true);
>
> /*
> * In strict mode, isolate_freepages_block() returns 0 if
> @@ -770,15 +794,15 @@ isolate_freepages_range(struct compact_control *cc,
> */
> }
>
> - /* __isolate_free_page() does not map the pages */
> - split_map_pages(&freelist);
> -
> if (pfn < end_pfn) {
> /* Loop terminated early, cleanup. */
> - release_freepages(&freelist);
> + release_free_list(tmp_freepages);
> return 0;
> }
>
> + /* __isolate_free_page() does not map the pages */
> + split_map_pages(tmp_freepages);
> +
> /* We don't use freelists for anything. */
> return pfn;
> }
> @@ -1462,7 +1486,7 @@ fast_isolate_around(struct compact_control *cc, unsigned long pfn)
> if (!page)
> return;
>
> - isolate_freepages_block(cc, &start_pfn, end_pfn, &cc->freepages, 1, false);
> + isolate_freepages_block(cc, &start_pfn, end_pfn, cc->freepages, 1, false);
>
> /* Skip this pageblock in the future as it's full or nearly full */
> if (start_pfn == end_pfn && !cc->no_set_skip_hint)
> @@ -1591,7 +1615,7 @@ static void fast_isolate_freepages(struct compact_control *cc)
> nr_scanned += nr_isolated - 1;
> total_isolated += nr_isolated;
> cc->nr_freepages += nr_isolated;
> - list_add_tail(&page->lru, &cc->freepages);
> + list_add_tail(&page->lru, &cc->freepages[order].pages);
> count_compact_events(COMPACTISOLATED, nr_isolated);
> } else {
> /* If isolation fails, abort the search */
> @@ -1665,13 +1689,12 @@ static void isolate_freepages(struct compact_control *cc)
> unsigned long isolate_start_pfn; /* exact pfn we start at */
> unsigned long block_end_pfn; /* end of current pageblock */
> unsigned long low_pfn; /* lowest pfn scanner is able to scan */
> - struct list_head *freelist = &cc->freepages;
> unsigned int stride;
>
> /* Try a small search of the free lists for a candidate */
> fast_isolate_freepages(cc);
> if (cc->nr_freepages)
> - goto splitmap;
> + return;
>
> /*
> * Initialise the free scanner. The starting point is where we last
> @@ -1731,7 +1754,7 @@ static void isolate_freepages(struct compact_control *cc)
>
> /* Found a block suitable for isolating free pages from. */
> nr_isolated = isolate_freepages_block(cc, &isolate_start_pfn,
> - block_end_pfn, freelist, stride, false);
> + block_end_pfn, cc->freepages, stride, false);
>
> /* Update the skip hint if the full pageblock was scanned */
> if (isolate_start_pfn == block_end_pfn)
> @@ -1772,10 +1795,6 @@ static void isolate_freepages(struct compact_control *cc)
> * and the loop terminated due to isolate_start_pfn < low_pfn
> */
> cc->free_pfn = isolate_start_pfn;
> -
> -splitmap:
> - /* __isolate_free_page() does not map the pages */
> - split_map_pages(freelist);
> }
>
> /*
> @@ -1786,23 +1805,22 @@ static struct folio *compaction_alloc(struct folio *src, unsigned long data)
> {
> struct compact_control *cc = (struct compact_control *)data;
> struct folio *dst;
> + int order = folio_order(src);
>
> - /* this makes migrate_pages() split the source page and retry */
> - if (folio_order(src) > 0)
> - return NULL;
> -
> - if (list_empty(&cc->freepages)) {
> + if (!cc->freepages[order].nr_pages) {
> isolate_freepages(cc);
> -
> - if (list_empty(&cc->freepages))
> + if (!cc->freepages[order].nr_pages)
> return NULL;
> }
>
> - dst = list_entry(cc->freepages.next, struct folio, lru);
> + dst = list_first_entry(&cc->freepages[order].pages, struct folio, lru);
> + cc->freepages[order].nr_pages--;
> list_del(&dst->lru);
> - cc->nr_freepages--;
> -
> - return dst;
> + post_alloc_hook(&dst->page, order, __GFP_MOVABLE);
> + if (order)
> + prep_compound_page(&dst->page, order);
> + cc->nr_freepages -= 1 << order;
> + return page_rmappable_folio(&dst->page);
> }
>
> /*
> @@ -1813,9 +1831,34 @@ static struct folio *compaction_alloc(struct folio *src, unsigned long data)
> static void compaction_free(struct folio *dst, unsigned long data)
> {
> struct compact_control *cc = (struct compact_control *)data;
> + int order = folio_order(dst);
> + struct page *page = &dst->page;
> +
> + if (order) {
> + int i;
>
> - list_add(&dst->lru, &cc->freepages);
> - cc->nr_freepages++;
> + page[1].flags &= ~PAGE_FLAGS_SECOND;
> + for (i = 1; i < (1 << order); i++) {
> + page[i].mapping = NULL;
> + clear_compound_head(&page[i]);
> + page[i].flags &= ~PAGE_FLAGS_CHECK_AT_PREP;
> + }
> +
> + }
> + /* revert post_alloc_hook() operations */
> + page->mapping = NULL;
> + page->flags &= ~PAGE_FLAGS_CHECK_AT_PREP;
> + set_page_count(page, 0);
> + page_mapcount_reset(page);
> + reset_page_owner(page, order);
> + page_table_check_free(page, order);
> + arch_free_page(page, order);
> + set_page_private(page, order);
> + INIT_LIST_HEAD(&dst->lru);
> +
> + list_add(&dst->lru, &cc->freepages[order].pages);
> + cc->freepages[order].nr_pages++;
> + cc->nr_freepages += 1 << order;
> }
>
> /* possible outcome of isolate_migratepages */
> @@ -2439,6 +2482,7 @@ compact_zone(struct compact_control *cc, struct capture_control *capc)
> const bool sync = cc->mode != MIGRATE_ASYNC;
> bool update_cached;
> unsigned int nr_succeeded = 0;
> + int order;
>
> /*
> * These counters track activities during zone compaction. Initialize
> @@ -2448,7 +2492,8 @@ compact_zone(struct compact_control *cc, struct capture_control *capc)
> cc->total_free_scanned = 0;
> cc->nr_migratepages = 0;
> cc->nr_freepages = 0;
> - INIT_LIST_HEAD(&cc->freepages);
> + for (order = 0; order <= MAX_ORDER; order++)
> + init_page_list(&cc->freepages[order]);
> INIT_LIST_HEAD(&cc->migratepages);
>
> cc->migratetype = gfp_migratetype(cc->gfp_mask);
> @@ -2634,7 +2679,7 @@ compact_zone(struct compact_control *cc, struct capture_control *capc)
> * so we don't leave any returned pages behind in the next attempt.
> */
> if (cc->nr_freepages > 0) {
> - unsigned long free_pfn = release_freepages(&cc->freepages);
> + unsigned long free_pfn = release_free_list(cc->freepages);
>
> cc->nr_freepages = 0;
> VM_BUG_ON(free_pfn == 0);
> @@ -2653,7 +2698,6 @@ compact_zone(struct compact_control *cc, struct capture_control *capc)
>
> trace_mm_compaction_end(cc, start_pfn, end_pfn, sync, ret);
>
> - VM_BUG_ON(!list_empty(&cc->freepages));
> VM_BUG_ON(!list_empty(&cc->migratepages));
>
> return ret;
> diff --git a/mm/internal.h b/mm/internal.h
> index 8450562744cf..46c8bb94ecbb 100644
> --- a/mm/internal.h
> +++ b/mm/internal.h
> @@ -473,6 +473,11 @@ int split_free_page(struct page *free_page,
> /*
> * in mm/compaction.c
> */
> +
> +struct page_list {
> + struct list_head pages;
> + unsigned long nr_pages;
> +};
> /*
> * compact_control is used to track pages being migrated and the free pages
> * they are being migrated to during memory compaction. The free_pfn starts
> @@ -481,7 +486,7 @@ int split_free_page(struct page *free_page,
> * completes when free_pfn <= migrate_pfn
> */
> struct compact_control {
> - struct list_head freepages; /* List of free pages to migrate to */
> + struct page_list freepages[MAX_ORDER + 1]; /* List of free pages to migrate to */
> struct list_head migratepages; /* List of pages being migrated */
> unsigned int nr_freepages; /* Number of isolated free pages */
> unsigned int nr_migratepages; /* Number of pages to migrate */
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v1 2/4] mm/compaction: add support for >0 order folio memory compaction.
2024-01-09 15:18 ` Ryan Roberts
@ 2024-01-09 15:25 ` Zi Yan
0 siblings, 0 replies; 21+ messages in thread
From: Zi Yan @ 2024-01-09 15:25 UTC (permalink / raw)
To: Ryan Roberts
Cc: linux-mm, linux-kernel, "Huang, Ying",
Andrew Morton, "Matthew Wilcox (Oracle)",
David Hildenbrand, "Yin, Fengwei",
Yu Zhao, Vlastimil Babka, "Kirill A . Shutemov",
Johannes Weiner, Baolin Wang, Kemeng Shi, Mel Gorman, Rohan Puri,
Mcgrof Chamberlain, Adam Manzanares,
"Vishal Moola (Oracle)"
[-- Attachment #1: Type: text/plain, Size: 3648 bytes --]
On 9 Jan 2024, at 10:18, Ryan Roberts wrote:
> On 13/11/2023 17:01, Zi Yan wrote:
>> From: Zi Yan <ziy@nvidia.com>
>>
>> Before, memory compaction only migrates order-0 folios and skips >0 order
>> folios. This commit adds support for >0 order folio compaction by keeping
>> isolated free pages at their original size without splitting them into
>> order-0 pages and using them directly during migration process.
>>
>> What is different from the prior implementation:
>> 1. All isolated free pages are kept in a MAX_ORDER+1 array of page lists,
>> where each page list stores free pages in the same order.
>> 2. All free pages are not post_alloc_hook() processed nor buddy pages,
>> although their orders are stored in first page's private like buddy
>> pages.
>> 3. During migration, in new page allocation time (i.e., in
>> compaction_alloc()), free pages are then processed by post_alloc_hook().
>> When migration fails and a new page is returned (i.e., in
>> compaction_free()), free pages are restored by reversing the
>> post_alloc_hook() operations.
>>
>> Step 3 is done for a latter optimization that splitting and/or merging free
>> pages during compaction becomes easier.
>>
>> Signed-off-by: Zi Yan <ziy@nvidia.com>
>> ---
>> mm/compaction.c | 160 ++++++++++++++++++++++++++++++------------------
>> mm/internal.h | 7 ++-
>> 2 files changed, 108 insertions(+), 59 deletions(-)
>>
>> diff --git a/mm/compaction.c b/mm/compaction.c
>> index 5217dd35b493..ec6b5cc7e907 100644
>> --- a/mm/compaction.c
>> +++ b/mm/compaction.c
>> @@ -66,45 +66,64 @@ static inline void count_compact_events(enum vm_event_item item, long delta)
>> #define COMPACTION_HPAGE_ORDER (PMD_SHIFT - PAGE_SHIFT)
>> #endif
>>
>> -static unsigned long release_freepages(struct list_head *freelist)
>> +static void init_page_list(struct page_list *p)
>> {
>> - struct page *page, *next;
>> - unsigned long high_pfn = 0;
>> -
>> - list_for_each_entry_safe(page, next, freelist, lru) {
>> - unsigned long pfn = page_to_pfn(page);
>> - list_del(&page->lru);
>> - __free_page(page);
>> - if (pfn > high_pfn)
>> - high_pfn = pfn;
>> - }
>> -
>> - return high_pfn;
>> + INIT_LIST_HEAD(&p->pages);
>> + p->nr_pages = 0;
>> }
>>
>> -static void split_map_pages(struct list_head *list)
>> +static void split_map_pages(struct page_list *freepages)
>> {
>> unsigned int i, order, nr_pages;
>> struct page *page, *next;
>> LIST_HEAD(tmp_list);
>>
>> - list_for_each_entry_safe(page, next, list, lru) {
>> - list_del(&page->lru);
>> + for (order = 0; order <= MAX_ORDER; order++) {
>> + freepages[order].nr_pages = 0;
>> +
>> + list_for_each_entry_safe(page, next, &freepages[order].pages, lru) {
>> + list_del(&page->lru);
>>
>> - order = page_private(page);
>> - nr_pages = 1 << order;
>> + nr_pages = 1 << order;
>>
>> - post_alloc_hook(page, order, __GFP_MOVABLE);
>> - if (order)
>> - split_page(page, order);
>> + post_alloc_hook(page, order, __GFP_MOVABLE);
>> + if (order)
>> + split_page(page, order);
>>
>> - for (i = 0; i < nr_pages; i++) {
>> - list_add(&page->lru, &tmp_list);
>> - page++;
>> + for (i = 0; i < nr_pages; i++) {
>> + list_add(&page->lru, &tmp_list);
>> + page++;
>> + }
>> + freepages[order].nr_pages += nr_pages;
>> }
>> + list_splice(&tmp_list, &freepages[order].pages);
>
> I think this should be list_splice_init() since you are reusing tmp_list in each
> iteration of the outer loop?
Right. Will fix it in the next version. Thanks.
--
Best Regards,
Yan, Zi
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 854 bytes --]
^ permalink raw reply [flat|nested] 21+ messages in thread
end of thread, other threads:[~2024-01-09 15:25 UTC | newest]
Thread overview: 21+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-11-13 17:01 [PATCH v1 0/4] Enable >0 order folio memory compaction Zi Yan
2023-11-13 17:01 ` [PATCH v1 1/4] mm/compaction: enable compacting >0 order folios Zi Yan
2023-11-13 18:30 ` Matthew Wilcox
2023-11-13 19:22 ` Zi Yan
2023-11-20 9:18 ` Baolin Wang
2023-11-20 14:05 ` Zi Yan
2023-11-13 17:01 ` [PATCH v1 2/4] mm/compaction: add support for >0 order folio memory compaction Zi Yan
2024-01-09 15:18 ` Ryan Roberts
2024-01-09 15:25 ` Zi Yan
2023-11-13 17:01 ` [PATCH v1 3/4] mm/compaction: optimize >0 order folio compaction with free page split Zi Yan
2023-11-22 10:26 ` Ryan Roberts
2023-11-22 14:35 ` Zi Yan
2023-11-13 17:01 ` [PATCH v1 4/4] mm/compaction: optimize >0 order folio compaction by sorting source pages Zi Yan
2023-11-21 15:46 ` [PATCH v1 0/4] Enable >0 order folio memory compaction Ryan Roberts
2023-11-21 16:45 ` Zi Yan
2023-11-21 17:11 ` Ryan Roberts
2024-01-02 20:50 ` Zi Yan
2024-01-03 9:12 ` Ryan Roberts
2024-01-03 15:51 ` Zi Yan
2024-01-05 22:56 ` Zi Yan
2023-11-24 14:58 ` Ryan Roberts
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).