From: Zi Yan <zi.yan@sent.com>
To: linux-mm@kvack.org, linux-kernel@vger.kernel.org
Cc: Zi Yan <ziy@nvidia.com>, "Huang, Ying" <ying.huang@intel.com>,
Ryan Roberts <ryan.roberts@arm.com>,
Andrew Morton <akpm@linux-foundation.org>,
"Matthew Wilcox (Oracle)" <willy@infradead.org>,
David Hildenbrand <david@redhat.com>,
"Yin, Fengwei" <fengwei.yin@intel.com>,
Yu Zhao <yuzhao@google.com>, Vlastimil Babka <vbabka@suse.cz>,
"Kirill A . Shutemov" <kirill.shutemov@linux.intel.com>,
Johannes Weiner <hannes@cmpxchg.org>,
Baolin Wang <baolin.wang@linux.alibaba.com>,
Kemeng Shi <shikemeng@huaweicloud.com>,
Mel Gorman <mgorman@techsingularity.net>,
Rohan Puri <rohan.puri15@gmail.com>,
Mcgrof Chamberlain <mcgrof@kernel.org>,
Adam Manzanares <a.manzanares@samsung.com>,
"Vishal Moola (Oracle)" <vishal.moola@gmail.com>
Subject: [PATCH v2 0/3] Enable >0 order folio memory compaction
Date: Mon, 22 Jan 2024 22:46:32 -0500 [thread overview]
Message-ID: <20240123034636.1095672-1-zi.yan@sent.com> (raw)
From: Zi Yan <ziy@nvidia.com>
Hi all,
This patchset enables >0 order folio memory compaction, which is one of
the prerequisitions for large folio support[1]. It is on top of
mm-everything-2024-01-18-22-21.
I am aware of that split free pages is necessary for folio
migration in compaction, since if >0 order free pages are never split
and no order-0 free page is scanned, compaction will end prematurely due
to migration returns -ENOMEM. Free page split becomes a must instead of
an optimization.
Some applications from vm-scalability show different performance trends
on default LRU and CONFIG_LRU_GEN from patch 1 (split folio during compaction),
to patch 2 (folio migration during compaction), to patch 3 (folio
migration during compaction with free page split). I am looking into it.
lkp ncompare results (with >5% delta) for default LRU and CONFIG_LRU_GEN are
shown at the bottom (on a 8-CPU (Intel Xeon E5-2650 v4 @ 2.20GHz) 16G VM).
Changelog
===
From V1 [2]:
1. Used folio_test_large() instead of folio_order() > 0. (per Matthew
Wilcox)
2. Fixed code rebase error. (per Baolin Wang)
3. Used list_split_init() instead of list_split(). (per Ryan Boberts)
4. Added free_pages_prepare_fpi_none() to avoid duplicate free page code
in compaction_free().
5. Dropped source page order sorting patch.
From RFC [1]:
1. Enabled >0 order folio compaction in the first patch by splitting all
to-be-migrated folios. (per Huang, Ying)
2. Stopped isolating compound pages with order greater than cc->order
to avoid wasting effort, since cc->order gives a hint that no free pages
with order greater than it exist, thus migrating the compound pages will fail.
(per Baolin Wang)
3. Retained the folio check within lru lock. (per Baolin Wang)
4. Made isolate_freepages_block() generate order-sorted multi lists.
(per Johannes Weiner)
Overview
===
To support >0 order folio compaction, the patchset changes how free pages used
for migration are kept during compaction. Free pages used to be split into
order-0 pages that are post allocation processed (i.e., PageBuddy flag cleared,
page order stored in page->private is zeroed, and page reference is set to 1).
Now all free pages are kept in a MAX_ORDER+1 array of page lists based
on their order without post allocation process. When migrate_pages() asks for
a new page, one of the free pages, based on the requested page order, is
then processed and given out.
Feel free to give comments and ask questions.
Thanks.
[1] https://lore.kernel.org/linux-mm/20230912162815.440749-1-zi.yan@sent.com/
[2] https://lore.kernel.org/linux-mm/20231113170157.280181-1-zi.yan@sent.com/
vm-scalability results on CONFIG_LRU_GEN
===
=========================================================================================
compiler/kconfig/rootfs/runtime/tbox_group/test/testcase:
gcc-13/defconfig/debian/300s/qemu-vm/small-allocs/vm-scalability
commit:
6.7.0-rc4+
6.7.0-rc4-split-folio-in-compaction+
6.7.0-rc4-folio-migration-in-compaction+
6.7.0-rc4-folio-migration-free-page-split+
6.7.0-rc4+ 6.7.0-rc4-split-folio-in-co 6.7.0-rc4-folio-migration-i 6.7.0-rc4-folio-migration-f
---------------- --------------------------- --------------------------- ---------------------------
%stddev %change %stddev %change %stddev %change %stddev
\ | \ | \ | \
2024326 +35.5% 2743772 ± 41% +364.0% 9392198 ± 35% +31.0% 2651634 vm-scalability.throughput
=========================================================================================
compiler/kconfig/rootfs/runtime/tbox_group/test/testcase:
gcc-13/defconfig/debian/300s/qemu-vm/small-allocs-mt/vm-scalability
commit:
6.7.0-rc4+
6.7.0-rc4-split-folio-in-compaction+
6.7.0-rc4-folio-migration-in-compaction+
6.7.0-rc4-folio-migration-free-page-split+
6.7.0-rc4+ 6.7.0-rc4-split-folio-in-co 6.7.0-rc4-folio-migration-i 6.7.0-rc4-folio-migration-f
---------------- --------------------------- --------------------------- ---------------------------
%stddev %change %stddev %change %stddev %change %stddev
\ | \ | \ | \
1450189 +0.9% 1463418 +30.4% 1891610 ± 22% +0.3% 1454100 vm-scalability.throughput
=========================================================================================
compiler/kconfig/rootfs/runtime/tbox_group/test/testcase:
gcc-13/defconfig/debian/300s/qemu-vm/mmap-xread-seq-mt/vm-scalability
commit:
6.7.0-rc4+
6.7.0-rc4-split-folio-in-compaction+
6.7.0-rc4-folio-migration-in-compaction+
6.7.0-rc4-folio-migration-free-page-split+
6.7.0-rc4+ 6.7.0-rc4-split-folio-in-co 6.7.0-rc4-folio-migration-i 6.7.0-rc4-folio-migration-f
---------------- --------------------------- --------------------------- ---------------------------
%stddev %change %stddev %change %stddev %change %stddev
\ | \ | \ | \
14428848 ± 27% -51.7% 6963308 ± 73% +13.5% 16372621 +11.2% 16046511 vm-scalability.throughput
=========================================================================================
compiler/kconfig/rootfs/runtime/tbox_group/test/testcase:
gcc-13/defconfig/debian/300s/qemu-vm/mmap-pread-seq/vm-scalability
commit:
6.7.0-rc4+
6.7.0-rc4-split-folio-in-compaction+
6.7.0-rc4-folio-migration-in-compaction+
6.7.0-rc4-folio-migration-free-page-split+
6.7.0-rc4+ 6.7.0-rc4-split-folio-in-co 6.7.0-rc4-folio-migration-i 6.7.0-rc4-folio-migration-f
---------------- --------------------------- --------------------------- ---------------------------
%stddev %change %stddev %change %stddev %change %stddev
\ | \ | \ | \
13569502 ± 24% -45.9% 7340064 ± 59% +12.3% 15240531 +10.4% 14983705 vm-scalability.throughput
=========================================================================================
compiler/kconfig/rootfs/runtime/tbox_group/test/testcase:
gcc-13/defconfig/debian/300s/qemu-vm/mmap-pread-seq-mt/vm-scalability
commit:
6.7.0-rc4+
6.7.0-rc4-split-folio-in-compaction+
6.7.0-rc4-folio-migration-in-compaction+
6.7.0-rc4-folio-migration-free-page-split+
6.7.0-rc4+ 6.7.0-rc4-split-folio-in-co 6.7.0-rc4-folio-migration-i 6.7.0-rc4-folio-migration-f
---------------- --------------------------- --------------------------- ---------------------------
%stddev %change %stddev %change %stddev %change %stddev
\ | \ | \ | \
13305823 ± 24% -45.1% 7299664 ± 56% +12.5% 14974725 +10.4% 14695963 vm-scalability.throughput
=========================================================================================
compiler/kconfig/rootfs/runtime/tbox_group/test/testcase:
gcc-13/defconfig/debian/300s/qemu-vm/lru-file-readtwice/vm-scalability
commit:
6.7.0-rc4+
6.7.0-rc4-split-folio-in-compaction+
6.7.0-rc4-folio-migration-in-compaction+
6.7.0-rc4-folio-migration-free-page-split+
6.7.0-rc4+ 6.7.0-rc4-split-folio-in-co 6.7.0-rc4-folio-migration-i 6.7.0-rc4-folio-migration-f
---------------- --------------------------- --------------------------- ---------------------------
%stddev %change %stddev %change %stddev %change %stddev
\ | \ | \ | \
13244376 ± 28% +54.2% 20425838 ± 23% -4.4% 12660113 ± 3% -9.0% 12045809 ± 3% vm-scalability.throughput
=========================================================================================
compiler/kconfig/rootfs/runtime/tbox_group/test/testcase:
gcc-13/defconfig/debian/300s/qemu-vm/lru-file-mmap-read/vm-scalability
commit:
6.7.0-rc4+
6.7.0-rc4-split-folio-in-compaction+
6.7.0-rc4-folio-migration-in-compaction+
6.7.0-rc4-folio-migration-free-page-split+
6.7.0-rc4+ 6.7.0-rc4-split-folio-in-co 6.7.0-rc4-folio-migration-i 6.7.0-rc4-folio-migration-f
---------------- --------------------------- --------------------------- ---------------------------
%stddev %change %stddev %change %stddev %change %stddev
\ | \ | \ | \
7021425 ± 11% -20.9% 5556751 ± 19% +14.8% 8057811 ± 3% +9.4% 7678613 ± 4% vm-scalability.throughput
=========================================================================================
compiler/kconfig/rootfs/runtime/size/tbox_group/test/testcase:
gcc-13/defconfig/debian/300s/256G/qemu-vm/msync/vm-scalability
commit:
6.7.0-rc4+
6.7.0-rc4-split-folio-in-compaction+
6.7.0-rc4-folio-migration-in-compaction+
6.7.0-rc4-folio-migration-free-page-split+
6.7.0-rc4+ 6.7.0-rc4-split-folio-in-co 6.7.0-rc4-folio-migration-i 6.7.0-rc4-folio-migration-f
---------------- --------------------------- --------------------------- ---------------------------
%stddev %change %stddev %change %stddev %change %stddev
\ | \ | \ | \
1208994 ±137% +263.5% 4394683 ± 49% -49.4% 611204 ± 6% -48.1% 627937 ± 13% vm-scalability.throughput
vm-scalability results on default LRU (with -no-mglru suffix)
===
=========================================================================================
compiler/kconfig/rootfs/runtime/tbox_group/test/testcase:
gcc-13/defconfig/debian/300s/qemu-vm/lru-file-readtwice/vm-scalability
commit:
6.7.0-rc4-no-mglru+
6.7.0-rc4-split-folio-in-compaction-no-mglru+
6.7.0-rc4-folio-migration-in-compaction-no-mglru+
6.7.0-rc4-folio-migration-free-page-split-no-mglru+
6.7.0-rc4-no-mgl 6.7.0-rc4-split-folio-in-co 6.7.0-rc4-folio-migration-i 6.7.0-rc4-folio-migration-f
---------------- --------------------------- --------------------------- ---------------------------
%stddev %change %stddev %change %stddev %change %stddev
\ | \ | \ | \
8412072 ± 3% +32.1% 11114537 ± 41% +3.5% 8703491 ± 3% +1.5% 8536343 ± 3% vm-scalability.throughput
=========================================================================================
compiler/kconfig/rootfs/runtime/tbox_group/test/testcase:
gcc-13/defconfig/debian/300s/qemu-vm/lru-file-mmap-read/vm-scalability
commit:
6.7.0-rc4-no-mglru+
6.7.0-rc4-split-folio-in-compaction-no-mglru+
6.7.0-rc4-folio-migration-in-compaction-no-mglru+
6.7.0-rc4-folio-migration-free-page-split-no-mglru+
6.7.0-rc4-no-mgl 6.7.0-rc4-split-folio-in-co 6.7.0-rc4-folio-migration-i 6.7.0-rc4-folio-migration-f
---------------- --------------------------- --------------------------- ---------------------------
%stddev %change %stddev %change %stddev %change %stddev
\ | \ | \ | \
7095358 +10.8% 7863635 ± 16% +5.5% 7484110 +1.5% 7200666 ± 4% vm-scalability.throughput
Zi Yan (3):
mm/compaction: enable compacting >0 order folios.
mm/compaction: add support for >0 order folio memory compaction.
mm/compaction: optimize >0 order folio compaction with free page
split.
mm/compaction.c | 218 ++++++++++++++++++++++++++++++++++--------------
mm/internal.h | 9 +-
mm/page_alloc.c | 6 ++
3 files changed, 169 insertions(+), 64 deletions(-)
--
2.43.0
next reply other threads:[~2024-01-23 3:47 UTC|newest]
Thread overview: 8+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-01-23 3:46 Zi Yan [this message]
2024-01-23 3:46 ` [PATCH v2 1/3] mm/compaction: enable compacting >0 order folios Zi Yan
2024-01-26 2:29 ` Baolin Wang
2024-01-23 3:46 ` [PATCH v2 2/3] mm/compaction: add support for >0 order folio memory compaction Zi Yan
2024-01-29 18:32 ` Zi Yan
2024-01-23 3:46 ` [PATCH v2 3/3] mm/compaction: optimize >0 order folio compaction with free page split Zi Yan
2024-01-26 9:03 ` [PATCH v2 0/3] Enable >0 order folio memory compaction Baolin Wang
2024-01-26 14:22 ` Zi Yan
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20240123034636.1095672-1-zi.yan@sent.com \
--to=zi.yan@sent.com \
--cc=a.manzanares@samsung.com \
--cc=akpm@linux-foundation.org \
--cc=baolin.wang@linux.alibaba.com \
--cc=david@redhat.com \
--cc=fengwei.yin@intel.com \
--cc=hannes@cmpxchg.org \
--cc=kirill.shutemov@linux.intel.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=mcgrof@kernel.org \
--cc=mgorman@techsingularity.net \
--cc=rohan.puri15@gmail.com \
--cc=ryan.roberts@arm.com \
--cc=shikemeng@huaweicloud.com \
--cc=vbabka@suse.cz \
--cc=vishal.moola@gmail.com \
--cc=willy@infradead.org \
--cc=ying.huang@intel.com \
--cc=yuzhao@google.com \
--cc=ziy@nvidia.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).