From: Joao Martins <joao.m.martins@oracle.com> To: linux-mm@kvack.org Cc: linux-nvdimm@lists.01.org, Matthew Wilcox <willy@infradead.org>, Jason Gunthorpe <jgg@ziepe.ca>, Muchun Song <songmuchun@bytedance.com>, Mike Kravetz <mike.kravetz@oracle.com>, Andrew Morton <akpm@linux-foundation.org>, Joao Martins <joao.m.martins@oracle.com> Subject: [PATCH v1 04/11] mm/memremap: add ZONE_DEVICE support for compound pages Date: Thu, 25 Mar 2021 23:09:31 +0000 [thread overview] Message-ID: <20210325230938.30752-5-joao.m.martins@oracle.com> (raw) In-Reply-To: <20210325230938.30752-1-joao.m.martins@oracle.com> Add a new align property for struct dev_pagemap which specifies that a pagemap is composed of a set of compound pages of size @align, instead of base pages. When these pages are initialised, most are initialised as tail pages instead of order-0 pages. For certain ZONE_DEVICE users like device-dax which have a fixed page size, this creates an opportunity to optimize GUP and GUP-fast walkers, treating it the same way as THP or hugetlb pages. Signed-off-by: Joao Martins <joao.m.martins@oracle.com> --- include/linux/memremap.h | 13 +++++++++++++ mm/memremap.c | 8 ++++++-- mm/page_alloc.c | 24 +++++++++++++++++++++++- 3 files changed, 42 insertions(+), 3 deletions(-) diff --git a/include/linux/memremap.h b/include/linux/memremap.h index b46f63dcaed3..bb28d82dda5e 100644 --- a/include/linux/memremap.h +++ b/include/linux/memremap.h @@ -114,6 +114,7 @@ struct dev_pagemap { struct completion done; enum memory_type type; unsigned int flags; + unsigned long align; const struct dev_pagemap_ops *ops; void *owner; int nr_range; @@ -130,6 +131,18 @@ static inline struct vmem_altmap *pgmap_altmap(struct dev_pagemap *pgmap) return NULL; } +static inline unsigned long pgmap_align(struct dev_pagemap *pgmap) +{ + if (!pgmap || !pgmap->align) + return PAGE_SIZE; + return pgmap->align; +} + +static inline unsigned long pgmap_pfn_align(struct dev_pagemap *pgmap) +{ + return PHYS_PFN(pgmap_align(pgmap)); +} + #ifdef CONFIG_ZONE_DEVICE bool pfn_zone_device_reserved(unsigned long pfn); void *memremap_pages(struct dev_pagemap *pgmap, int nid); diff --git a/mm/memremap.c b/mm/memremap.c index 805d761740c4..d160853670c4 100644 --- a/mm/memremap.c +++ b/mm/memremap.c @@ -318,8 +318,12 @@ static int pagemap_range(struct dev_pagemap *pgmap, struct mhp_params *params, memmap_init_zone_device(&NODE_DATA(nid)->node_zones[ZONE_DEVICE], PHYS_PFN(range->start), PHYS_PFN(range_len(range)), pgmap); - percpu_ref_get_many(pgmap->ref, pfn_end(pgmap, range_id) - - pfn_first(pgmap, range_id)); + if (pgmap_align(pgmap) > PAGE_SIZE) + percpu_ref_get_many(pgmap->ref, (pfn_end(pgmap, range_id) + - pfn_first(pgmap, range_id)) / pgmap_pfn_align(pgmap)); + else + percpu_ref_get_many(pgmap->ref, pfn_end(pgmap, range_id) + - pfn_first(pgmap, range_id)); return 0; err_add_memory: diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 58974067bbd4..3a77f9e43f3a 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -6285,6 +6285,8 @@ void __ref memmap_init_zone_device(struct zone *zone, unsigned long pfn, end_pfn = start_pfn + nr_pages; struct pglist_data *pgdat = zone->zone_pgdat; struct vmem_altmap *altmap = pgmap_altmap(pgmap); + unsigned int pfn_align = pgmap_pfn_align(pgmap); + unsigned int order_align = order_base_2(pfn_align); unsigned long zone_idx = zone_idx(zone); unsigned long start = jiffies; int nid = pgdat->node_id; @@ -6302,10 +6304,30 @@ void __ref memmap_init_zone_device(struct zone *zone, nr_pages = end_pfn - start_pfn; } - for (pfn = start_pfn; pfn < end_pfn; pfn++) { + for (pfn = start_pfn; pfn < end_pfn; pfn += pfn_align) { struct page *page = pfn_to_page(pfn); + unsigned long i; __init_zone_device_page(page, pfn, zone_idx, nid, pgmap); + + if (pfn_align == 1) + continue; + + __SetPageHead(page); + + for (i = 1; i < pfn_align; i++) { + __init_zone_device_page(page + i, pfn + i, zone_idx, + nid, pgmap); + prep_compound_tail(page, i); + + /* + * The first and second tail pages need to + * initialized first, hence the head page is + * prepared last. + */ + if (i == 2) + prep_compound_head(page, order_align); + } } pr_info("%s initialised %lu pages in %ums\n", __func__, -- 2.17.1 _______________________________________________ Linux-nvdimm mailing list -- linux-nvdimm@lists.01.org To unsubscribe send an email to linux-nvdimm-leave@lists.01.org
WARNING: multiple messages have this Message-ID (diff)
From: Joao Martins <joao.m.martins@oracle.com> To: linux-mm@kvack.org Cc: Dan Williams <dan.j.williams@intel.com>, Ira Weiny <ira.weiny@intel.com>, linux-nvdimm@lists.01.org, Matthew Wilcox <willy@infradead.org>, Jason Gunthorpe <jgg@ziepe.ca>, Jane Chu <jane.chu@oracle.com>, Muchun Song <songmuchun@bytedance.com>, Mike Kravetz <mike.kravetz@oracle.com>, Andrew Morton <akpm@linux-foundation.org>, Joao Martins <joao.m.martins@oracle.com> Subject: [PATCH v1 04/11] mm/memremap: add ZONE_DEVICE support for compound pages Date: Thu, 25 Mar 2021 23:09:31 +0000 [thread overview] Message-ID: <20210325230938.30752-5-joao.m.martins@oracle.com> (raw) In-Reply-To: <20210325230938.30752-1-joao.m.martins@oracle.com> Add a new align property for struct dev_pagemap which specifies that a pagemap is composed of a set of compound pages of size @align, instead of base pages. When these pages are initialised, most are initialised as tail pages instead of order-0 pages. For certain ZONE_DEVICE users like device-dax which have a fixed page size, this creates an opportunity to optimize GUP and GUP-fast walkers, treating it the same way as THP or hugetlb pages. Signed-off-by: Joao Martins <joao.m.martins@oracle.com> --- include/linux/memremap.h | 13 +++++++++++++ mm/memremap.c | 8 ++++++-- mm/page_alloc.c | 24 +++++++++++++++++++++++- 3 files changed, 42 insertions(+), 3 deletions(-) diff --git a/include/linux/memremap.h b/include/linux/memremap.h index b46f63dcaed3..bb28d82dda5e 100644 --- a/include/linux/memremap.h +++ b/include/linux/memremap.h @@ -114,6 +114,7 @@ struct dev_pagemap { struct completion done; enum memory_type type; unsigned int flags; + unsigned long align; const struct dev_pagemap_ops *ops; void *owner; int nr_range; @@ -130,6 +131,18 @@ static inline struct vmem_altmap *pgmap_altmap(struct dev_pagemap *pgmap) return NULL; } +static inline unsigned long pgmap_align(struct dev_pagemap *pgmap) +{ + if (!pgmap || !pgmap->align) + return PAGE_SIZE; + return pgmap->align; +} + +static inline unsigned long pgmap_pfn_align(struct dev_pagemap *pgmap) +{ + return PHYS_PFN(pgmap_align(pgmap)); +} + #ifdef CONFIG_ZONE_DEVICE bool pfn_zone_device_reserved(unsigned long pfn); void *memremap_pages(struct dev_pagemap *pgmap, int nid); diff --git a/mm/memremap.c b/mm/memremap.c index 805d761740c4..d160853670c4 100644 --- a/mm/memremap.c +++ b/mm/memremap.c @@ -318,8 +318,12 @@ static int pagemap_range(struct dev_pagemap *pgmap, struct mhp_params *params, memmap_init_zone_device(&NODE_DATA(nid)->node_zones[ZONE_DEVICE], PHYS_PFN(range->start), PHYS_PFN(range_len(range)), pgmap); - percpu_ref_get_many(pgmap->ref, pfn_end(pgmap, range_id) - - pfn_first(pgmap, range_id)); + if (pgmap_align(pgmap) > PAGE_SIZE) + percpu_ref_get_many(pgmap->ref, (pfn_end(pgmap, range_id) + - pfn_first(pgmap, range_id)) / pgmap_pfn_align(pgmap)); + else + percpu_ref_get_many(pgmap->ref, pfn_end(pgmap, range_id) + - pfn_first(pgmap, range_id)); return 0; err_add_memory: diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 58974067bbd4..3a77f9e43f3a 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -6285,6 +6285,8 @@ void __ref memmap_init_zone_device(struct zone *zone, unsigned long pfn, end_pfn = start_pfn + nr_pages; struct pglist_data *pgdat = zone->zone_pgdat; struct vmem_altmap *altmap = pgmap_altmap(pgmap); + unsigned int pfn_align = pgmap_pfn_align(pgmap); + unsigned int order_align = order_base_2(pfn_align); unsigned long zone_idx = zone_idx(zone); unsigned long start = jiffies; int nid = pgdat->node_id; @@ -6302,10 +6304,30 @@ void __ref memmap_init_zone_device(struct zone *zone, nr_pages = end_pfn - start_pfn; } - for (pfn = start_pfn; pfn < end_pfn; pfn++) { + for (pfn = start_pfn; pfn < end_pfn; pfn += pfn_align) { struct page *page = pfn_to_page(pfn); + unsigned long i; __init_zone_device_page(page, pfn, zone_idx, nid, pgmap); + + if (pfn_align == 1) + continue; + + __SetPageHead(page); + + for (i = 1; i < pfn_align; i++) { + __init_zone_device_page(page + i, pfn + i, zone_idx, + nid, pgmap); + prep_compound_tail(page, i); + + /* + * The first and second tail pages need to + * initialized first, hence the head page is + * prepared last. + */ + if (i == 2) + prep_compound_head(page, order_align); + } } pr_info("%s initialised %lu pages in %ums\n", __func__, -- 2.17.1
next prev parent reply other threads:[~2021-03-25 23:10 UTC|newest] Thread overview: 108+ messages / expand[flat|nested] mbox.gz Atom feed top 2021-03-25 23:09 [PATCH v1 00/11] mm, sparse-vmemmap: Introduce compound pagemaps Joao Martins 2021-03-25 23:09 ` Joao Martins 2021-03-25 23:09 ` [PATCH v1 01/11] memory-failure: fetch compound_head after pgmap_pfn_valid() Joao Martins 2021-03-25 23:09 ` Joao Martins 2021-04-24 0:12 ` Dan Williams 2021-04-24 0:12 ` Dan Williams 2021-04-24 19:00 ` Joao Martins 2021-04-24 19:00 ` Joao Martins 2021-03-25 23:09 ` [PATCH v1 02/11] mm/page_alloc: split prep_compound_page into head and tail subparts Joao Martins 2021-03-25 23:09 ` Joao Martins 2021-04-24 0:16 ` Dan Williams 2021-04-24 0:16 ` Dan Williams 2021-04-24 19:05 ` Joao Martins 2021-04-24 19:05 ` Joao Martins 2021-03-25 23:09 ` [PATCH v1 03/11] mm/page_alloc: refactor memmap_init_zone_device() page init Joao Martins 2021-03-25 23:09 ` Joao Martins 2021-04-24 0:18 ` Dan Williams 2021-04-24 0:18 ` Dan Williams 2021-04-24 19:05 ` Joao Martins 2021-04-24 19:05 ` Joao Martins 2021-03-25 23:09 ` Joao Martins [this message] 2021-03-25 23:09 ` [PATCH v1 04/11] mm/memremap: add ZONE_DEVICE support for compound pages Joao Martins 2021-05-05 18:44 ` Dan Williams 2021-05-05 18:44 ` Dan Williams 2021-05-05 18:58 ` Matthew Wilcox 2021-05-05 18:58 ` Matthew Wilcox 2021-05-05 19:49 ` Joao Martins 2021-05-05 19:49 ` Joao Martins 2021-05-05 22:20 ` Dan Williams 2021-05-05 22:20 ` Dan Williams 2021-05-05 22:36 ` Joao Martins 2021-05-05 22:36 ` Joao Martins 2021-05-05 23:03 ` Dan Williams 2021-05-05 23:03 ` Dan Williams 2021-05-06 10:12 ` Joao Martins 2021-05-06 10:12 ` Joao Martins 2021-05-18 17:27 ` Joao Martins 2021-05-18 17:27 ` Joao Martins 2021-05-18 19:56 ` Jane Chu 2021-05-18 19:56 ` Jane Chu 2021-05-19 11:29 ` Joao Martins 2021-05-19 11:29 ` Joao Martins 2021-05-19 18:36 ` Jane Chu 2021-06-07 20:17 ` Dan Williams 2021-06-07 20:47 ` Joao Martins 2021-06-07 21:00 ` Joao Martins 2021-06-07 21:57 ` Dan Williams 2021-05-06 8:05 ` Aneesh Kumar K.V 2021-05-06 8:05 ` Aneesh Kumar K.V 2021-05-06 10:23 ` Joao Martins 2021-05-06 10:23 ` Joao Martins 2021-05-06 11:43 ` Matthew Wilcox 2021-05-06 11:43 ` Matthew Wilcox 2021-05-06 12:15 ` Joao Martins 2021-05-06 12:15 ` Joao Martins 2021-03-25 23:09 ` [PATCH v1 05/11] mm/sparse-vmemmap: add a pgmap argument to section activation Joao Martins 2021-03-25 23:09 ` Joao Martins 2021-05-05 22:34 ` Dan Williams 2021-05-05 22:34 ` Dan Williams 2021-05-05 22:37 ` Joao Martins 2021-05-05 22:37 ` Joao Martins 2021-05-05 23:14 ` Dan Williams 2021-05-05 23:14 ` Dan Williams 2021-05-06 10:24 ` Joao Martins 2021-05-06 10:24 ` Joao Martins 2021-03-25 23:09 ` [PATCH v1 06/11] mm/sparse-vmemmap: refactor vmemmap_populate_basepages() Joao Martins 2021-03-25 23:09 ` Joao Martins 2021-05-05 22:43 ` Dan Williams 2021-05-05 22:43 ` Dan Williams 2021-05-06 10:27 ` Joao Martins 2021-05-06 10:27 ` Joao Martins 2021-05-06 18:36 ` Joao Martins 2021-05-06 18:36 ` Joao Martins 2021-03-25 23:09 ` [PATCH v1 07/11] mm/sparse-vmemmap: populate compound pagemaps Joao Martins 2021-03-25 23:09 ` Joao Martins 2021-05-06 1:18 ` Dan Williams 2021-05-06 1:18 ` Dan Williams 2021-05-06 11:01 ` Joao Martins 2021-05-06 11:01 ` Joao Martins 2021-05-10 19:19 ` Dan Williams 2021-05-10 19:19 ` Dan Williams 2021-05-13 18:45 ` Joao Martins 2021-05-13 18:45 ` Joao Martins 2021-06-16 15:05 ` Joao Martins 2021-06-16 23:35 ` Dan Williams 2021-03-25 23:09 ` [PATCH v1 08/11] mm/sparse-vmemmap: use hugepages for PUD " Joao Martins 2021-03-25 23:09 ` Joao Martins 2021-06-01 19:30 ` Dan Williams 2021-06-07 12:02 ` Joao Martins 2021-06-07 19:47 ` Dan Williams 2021-03-25 23:09 ` [PATCH v1 09/11] mm/page_alloc: reuse tail struct pages for " Joao Martins 2021-03-25 23:09 ` Joao Martins 2021-06-01 23:35 ` Dan Williams 2021-06-07 13:48 ` Joao Martins 2021-06-07 19:32 ` Dan Williams 2021-06-14 18:41 ` Joao Martins 2021-06-14 23:07 ` Dan Williams 2021-03-25 23:09 ` [PATCH v1 10/11] device-dax: compound pagemap support Joao Martins 2021-03-25 23:09 ` Joao Martins 2021-06-02 0:36 ` Dan Williams 2021-06-07 13:59 ` Joao Martins 2021-03-25 23:09 ` [PATCH v1 11/11] mm/gup: grab head page refcount once for group of subpages Joao Martins 2021-03-25 23:09 ` Joao Martins 2021-06-02 1:05 ` Dan Williams 2021-06-07 15:21 ` Joao Martins 2021-06-07 19:22 ` Dan Williams 2021-04-01 9:38 ` [PATCH v1 00/11] mm, sparse-vmemmap: Introduce compound pagemaps Joao Martins 2021-04-01 9:38 ` Joao Martins
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20210325230938.30752-5-joao.m.martins@oracle.com \ --to=joao.m.martins@oracle.com \ --cc=akpm@linux-foundation.org \ --cc=jgg@ziepe.ca \ --cc=linux-mm@kvack.org \ --cc=linux-nvdimm@lists.01.org \ --cc=mike.kravetz@oracle.com \ --cc=songmuchun@bytedance.com \ --cc=willy@infradead.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.