nvdimm.lists.linux.dev archive mirror
 help / color / mirror / Atom feed
* [PATCH RFC 0/9] mm, sparse-vmemmap: Introduce compound pagemaps
@ 2020-12-08 17:28 Joao Martins
  2020-12-08 17:28 ` [PATCH RFC 1/9] memremap: add ZONE_DEVICE support for compound pages Joao Martins
                   ` (12 more replies)
  0 siblings, 13 replies; 67+ messages in thread
From: Joao Martins @ 2020-12-08 17:28 UTC (permalink / raw)
  To: linux-mm
  Cc: linux-nvdimm, Matthew Wilcox, Jason Gunthorpe, Muchun Song,
	Mike Kravetz, Andrew Morton, Joao Martins

Hey,

This small series, attempts at minimizing 'struct page' overhead by
pursuing a similar approach as Muchun Song series "Free some vmemmap
pages of hugetlb page"[0] but applied to devmap/ZONE_DEVICE. 

[0] https://lore.kernel.org/linux-mm/20201130151838.11208-1-songmuchun@bytedance.com/

The link above describes it quite nicely, but the idea is to reuse tail
page vmemmap areas, particular the area which only describes tail pages.
So a vmemmap page describes 64 struct pages, and the first page for a given
ZONE_DEVICE vmemmap would contain the head page and 63 tail pages. The second
vmemmap page would contain only tail pages, and that's what gets reused across
the rest of the subsection/section. The bigger the page size, the bigger the
savings (2M hpage -> save 6 vmemmap pages; 1G hpage -> save 4094 vmemmap pages).

In terms of savings, per 1Tb of memory, the struct page cost would go down
with compound pagemap:

* with 2M pages we lose 4G instead of 16G (0.39% instead of 1.5% of total memory)
* with 1G pages we lose 8MB instead of 16G (0.0007% instead of 1.5% of total memory)

Along the way I've extended it past 'struct page' overhead *trying* to address a
few performance issues we knew about for pmem, specifically on the
{pin,get}_user_pages* function family with device-dax vmas which are really
slow even of the fast variants. THP is great on -fast variants but all except
hugetlbfs perform rather poorly on non-fast gup.

So to summarize what the series does:

Patches 1-5: Much like Muchun series, we reuse tail page areas across a given
page size (namely @align was referred by remaining memremap/dax code) and
enabling of memremap to initialize the ZONE_DEVICE pages as compound pages or a
given @align order. The main difference though, is that contrary to the hugetlbfs
series, there's no vmemmap for the area, because we are onlining it. IOW no
freeing of pages of already initialized vmemmap like the case for hugetlbfs,
which simplifies the logic (besides not being arch-specific). After these,
there's quite visible region bootstrap of pmem memmap given that we would
initialize fewer struct pages depending on the page size.

    NVDIMM namespace bootstrap improves from ~750ms to ~190ms/<=1ms on emulated NVDIMMs
    with 2M and 1G respectivally. The net gain in improvement is similarly observed
    in proportion when running on actual NVDIMMs.

Patch 6 - 8: Optimize grabbing/release a page refcount changes given that we
are working with compound pages i.e. we do 1 increment/decrement to the head
page for a given set of N subpages compared as opposed to N individual writes.
{get,pin}_user_pages_fast() for zone_device with compound pagemap consequently
improves considerably, and unpin_user_pages() improves as well when passed a
set of consecutive pages:

                                           before          after
    (get_user_pages_fast 1G;2M page size) ~75k  us -> ~3.2k ; ~5.2k us
    (pin_user_pages_fast 1G;2M page size) ~125k us -> ~3.4k ; ~5.5k us

The RDMA patch (patch 8/9) is to demonstrate the improvement for an existing
user. For unpin_user_pages() we have an additional test to demonstrate the
improvement.  The test performs MR reg/unreg continuously and measuring its
rate for a given period. So essentially ib_mem_get and ib_mem_release being
stress tested which at the end of day means: pin_user_pages_longterm() and
unpin_user_pages() for a scatterlist:

    Before:
    159 rounds in 5.027 sec: 31617.923 usec / round (device-dax)
    466 rounds in 5.009 sec: 10748.456 usec / round (hugetlbfs)
	        
    After:
    305 rounds in 5.010 sec: 16426.047 usec / round (device-dax)
    1073 rounds in 5.004 sec: 4663.622 usec / round (hugetlbfs)

Patch 9: Improves {pin,get}_user_pages() and its longterm counterpart. It
is very experimental, and I imported most of follow_hugetlb_page(), except
that we do the same trick as gup-fast. In doing the patch I feel this batching
should live in follow_page_mask() and having that being changed to return a set
of pages/something-else when walking over PMD/PUDs for THP / devmap pages. This
patch then brings the previous test of mr reg/unreg (above) on parity
between device-dax and hugetlbfs.

Some of the patches are a little fresh/WIP (specially patch 3 and 9) and we are
still running tests. Hence the RFC, asking for comments and general direction
of the work before continuing.

Patches apply on top of linux-next tag next-20201208 (commit a9e26cb5f261).

Comments and suggestions very much appreciated!

Thanks,
	Joao

Joao Martins (9):
  memremap: add ZONE_DEVICE support for compound pages
  sparse-vmemmap: Consolidate arguments in vmemmap section populate
  sparse-vmemmap: Reuse vmemmap areas for a given page size
  mm/page_alloc: Reuse tail struct pages for compound pagemaps
  device-dax: Compound pagemap support
  mm/gup: Grab head page refcount once for group of subpages
  mm/gup: Decrement head page once for group of subpages
  RDMA/umem: batch page unpin in __ib_mem_release()
  mm: Add follow_devmap_page() for devdax vmas

 drivers/dax/device.c           |  54 ++++++---
 drivers/infiniband/core/umem.c |  25 +++-
 include/linux/huge_mm.h        |   4 +
 include/linux/memory_hotplug.h |  16 ++-
 include/linux/memremap.h       |   2 +
 include/linux/mm.h             |   6 +-
 mm/gup.c                       | 130 ++++++++++++++++-----
 mm/huge_memory.c               | 202 +++++++++++++++++++++++++++++++++
 mm/memory_hotplug.c            |  13 ++-
 mm/memremap.c                  |  13 ++-
 mm/page_alloc.c                |  28 ++++-
 mm/sparse-vmemmap.c            |  97 +++++++++++++---
 mm/sparse.c                    |  16 +--
 13 files changed, 531 insertions(+), 75 deletions(-)

-- 
2.17.1
_______________________________________________
Linux-nvdimm mailing list -- linux-nvdimm@lists.01.org
To unsubscribe send an email to linux-nvdimm-leave@lists.01.org

^ permalink raw reply	[flat|nested] 67+ messages in thread

end of thread, other threads:[~2021-03-12  5:54 UTC | newest]

Thread overview: 67+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-12-08 17:28 [PATCH RFC 0/9] mm, sparse-vmemmap: Introduce compound pagemaps Joao Martins
2020-12-08 17:28 ` [PATCH RFC 1/9] memremap: add ZONE_DEVICE support for compound pages Joao Martins
2020-12-09  5:59   ` John Hubbard
2020-12-09  6:33     ` Matthew Wilcox
2020-12-09 13:12       ` Joao Martins
2021-02-20  1:43     ` Dan Williams
2021-02-22 11:24       ` Joao Martins
2021-02-22 20:37         ` Dan Williams
2021-02-23 15:46           ` Joao Martins
2021-02-23 16:50             ` Dan Williams
2021-02-23 17:18               ` Joao Martins
2021-02-23 18:18                 ` Dan Williams
2021-03-10 18:12           ` Joao Martins
2021-03-12  5:54             ` Dan Williams
2021-02-20  1:24   ` Dan Williams
2021-02-22 11:09     ` Joao Martins
2020-12-08 17:28 ` [PATCH RFC 2/9] sparse-vmemmap: Consolidate arguments in vmemmap section populate Joao Martins
2020-12-09  6:16   ` John Hubbard
2020-12-09 13:51     ` Joao Martins
2021-02-20  1:49   ` Dan Williams
2021-02-22 11:26     ` Joao Martins
2020-12-08 17:28 ` [PATCH RFC 3/9] sparse-vmemmap: Reuse vmemmap areas for a given mhp_params::align Joao Martins
2020-12-08 17:38   ` Joao Martins
2020-12-08 17:28 ` [PATCH RFC 3/9] sparse-vmemmap: Reuse vmemmap areas for a given page size Joao Martins
2021-02-20  3:34   ` Dan Williams
2021-02-22 11:42     ` Joao Martins
2021-02-22 22:40       ` Dan Williams
2021-02-23 15:46         ` Joao Martins
2020-12-08 17:28 ` [PATCH RFC 4/9] mm/page_alloc: Reuse tail struct pages for compound pagemaps Joao Martins
2021-02-20  6:17   ` Dan Williams
2021-02-22 12:01     ` Joao Martins
2020-12-08 17:28 ` [PATCH RFC 5/9] device-dax: Compound pagemap support Joao Martins
2020-12-08 17:28 ` [PATCH RFC 6/9] mm/gup: Grab head page refcount once for group of subpages Joao Martins
2020-12-09  4:40   ` John Hubbard
2020-12-09 13:44     ` Joao Martins
     [not found]   ` <20201208194905.GQ5487@ziepe.ca>
2020-12-09 11:05     ` Joao Martins
     [not found]       ` <20201209151505.GV5487@ziepe.ca>
2020-12-09 16:02         ` Joao Martins
     [not found]           ` <20201209162438.GW5487@ziepe.ca>
2020-12-09 17:27             ` Joao Martins
2020-12-09 18:14             ` Matthew Wilcox
2020-12-10 15:43               ` Joao Martins
2020-12-08 17:28 ` [PATCH RFC 7/9] mm/gup: Decrement head page " Joao Martins
     [not found]   ` <20201208193446.GP5487@ziepe.ca>
2020-12-09  5:06     ` John Hubbard
2020-12-09 12:17     ` Joao Martins
2020-12-17 19:05     ` Joao Martins
     [not found]       ` <20201217200530.GK5487@ziepe.ca>
2020-12-17 22:34         ` Joao Martins
2020-12-19  2:06         ` John Hubbard
2020-12-19 13:10           ` Joao Martins
2020-12-08 17:29 ` [PATCH RFC 8/9] RDMA/umem: batch page unpin in __ib_mem_release() Joao Martins
2020-12-09  5:18   ` John Hubbard
     [not found]   ` <20201208192935.GA1908088@ziepe.ca>
2020-12-09 10:59     ` Joao Martins
2020-12-19 13:15       ` Joao Martins
2020-12-08 17:29 ` [PATCH RFC 9/9] mm: Add follow_devmap_page() for devdax vmas Joao Martins
2020-12-09  5:23   ` John Hubbard
     [not found]   ` <20201208195754.GR5487@ziepe.ca>
2020-12-09  8:05     ` Christoph Hellwig
2020-12-09 11:19     ` Joao Martins
2020-12-09  9:38 ` [PATCH RFC 0/9] mm, sparse-vmemmap: Introduce compound pagemaps David Hildenbrand
2020-12-09  9:52 ` [External] " Muchun Song
2021-02-20  1:18 ` Dan Williams
2021-02-22 11:06   ` Joao Martins
2021-02-22 14:32     ` Joao Martins
2021-02-23 16:28   ` Joao Martins
2021-02-23 16:44     ` Dan Williams
2021-02-23 17:15       ` Joao Martins
2021-02-23 18:15         ` Dan Williams
     [not found]       ` <20210223185435.GO2643399@ziepe.ca>
2021-02-23 22:48         ` Dan Williams
     [not found]           ` <20210223230723.GP2643399@ziepe.ca>
2021-02-24  0:14             ` Dan Williams
     [not found]               ` <20210224010017.GQ2643399@ziepe.ca>
2021-02-24  1:32                 ` Dan Williams

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).