From: Mel Gorman <mgorman@techsingularity.net> To: Andrew Morton <akpm@linux-foundation.org> Cc: Vlastimil Babka <vbabka@suse.cz>, Jesper Dangaard Brouer <brouer@redhat.com>, Linux-MM <linux-mm@kvack.org>, LKML <linux-kernel@vger.kernel.org>, Mel Gorman <mgorman@techsingularity.net> Subject: [PATCH 16/28] mm, page_alloc: Move __GFP_HARDWALL modifications out of the fastpath Date: Fri, 15 Apr 2016 10:07:43 +0100 [thread overview] Message-ID: <1460711275-1130-4-git-send-email-mgorman@techsingularity.net> (raw) In-Reply-To: <1460711275-1130-1-git-send-email-mgorman@techsingularity.net> __GFP_HARDWALL only has meaning in the context of cpusets but the fast path always applies the flag on the first attempt. Move the manipulations into the cpuset paths where they will be masked by a static branch in the common case. With the other micro-optimisations in this series combined, the impact on a page allocator microbenchmark is 4.6.0-rc2 4.6.0-rc2 decstat-v1r20 micro-v1r20 Min alloc-odr0-1 381.00 ( 0.00%) 377.00 ( 1.05%) Min alloc-odr0-2 275.00 ( 0.00%) 273.00 ( 0.73%) Min alloc-odr0-4 229.00 ( 0.00%) 226.00 ( 1.31%) Min alloc-odr0-8 199.00 ( 0.00%) 196.00 ( 1.51%) Min alloc-odr0-16 186.00 ( 0.00%) 183.00 ( 1.61%) Min alloc-odr0-32 179.00 ( 0.00%) 175.00 ( 2.23%) Min alloc-odr0-64 174.00 ( 0.00%) 172.00 ( 1.15%) Min alloc-odr0-128 172.00 ( 0.00%) 170.00 ( 1.16%) Min alloc-odr0-256 181.00 ( 0.00%) 183.00 ( -1.10%) Min alloc-odr0-512 193.00 ( 0.00%) 191.00 ( 1.04%) Min alloc-odr0-1024 201.00 ( 0.00%) 199.00 ( 1.00%) Min alloc-odr0-2048 206.00 ( 0.00%) 204.00 ( 0.97%) Min alloc-odr0-4096 212.00 ( 0.00%) 210.00 ( 0.94%) Min alloc-odr0-8192 215.00 ( 0.00%) 213.00 ( 0.93%) Min alloc-odr0-16384 216.00 ( 0.00%) 214.00 ( 0.93%) Signed-off-by: Mel Gorman <mgorman@techsingularity.net> --- mm/page_alloc.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 9ef2f4ab9ca5..4a364e318873 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -3353,7 +3353,7 @@ __alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order, struct page *page; unsigned int cpuset_mems_cookie; unsigned int alloc_flags = ALLOC_WMARK_LOW|ALLOC_FAIR; - gfp_t alloc_mask; /* The gfp_t that was actually used for allocation */ + gfp_t alloc_mask = gfp_mask; /* The gfp_t that was actually used for allocation */ struct alloc_context ac = { .high_zoneidx = gfp_zone(gfp_mask), .zonelist = zonelist, @@ -3362,6 +3362,7 @@ __alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order, }; if (cpusets_enabled()) { + alloc_mask |= __GFP_HARDWALL; alloc_flags |= ALLOC_CPUSET; if (!ac.nodemask) ac.nodemask = &cpuset_current_mems_allowed; @@ -3389,7 +3390,6 @@ __alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order, ac.classzone_idx = zonelist_zone_idx(preferred_zoneref); /* First allocation attempt */ - alloc_mask = gfp_mask|__GFP_HARDWALL; page = get_page_from_freelist(alloc_mask, order, alloc_flags, &ac); if (unlikely(!page)) { /* @@ -3414,8 +3414,10 @@ __alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order, * the mask is being updated. If a page allocation is about to fail, * check if the cpuset changed during allocation and if so, retry. */ - if (unlikely(!page && read_mems_allowed_retry(cpuset_mems_cookie))) + if (unlikely(!page && read_mems_allowed_retry(cpuset_mems_cookie))) { + alloc_mask = gfp_mask; goto retry_cpuset; + } return page; } -- 2.6.4
WARNING: multiple messages have this Message-ID (diff)
From: Mel Gorman <mgorman@techsingularity.net> To: Andrew Morton <akpm@linux-foundation.org> Cc: Vlastimil Babka <vbabka@suse.cz>, Jesper Dangaard Brouer <brouer@redhat.com>, Linux-MM <linux-mm@kvack.org>, LKML <linux-kernel@vger.kernel.org>, Mel Gorman <mgorman@techsingularity.net> Subject: [PATCH 16/28] mm, page_alloc: Move __GFP_HARDWALL modifications out of the fastpath Date: Fri, 15 Apr 2016 10:07:43 +0100 [thread overview] Message-ID: <1460711275-1130-4-git-send-email-mgorman@techsingularity.net> (raw) In-Reply-To: <1460711275-1130-1-git-send-email-mgorman@techsingularity.net> __GFP_HARDWALL only has meaning in the context of cpusets but the fast path always applies the flag on the first attempt. Move the manipulations into the cpuset paths where they will be masked by a static branch in the common case. With the other micro-optimisations in this series combined, the impact on a page allocator microbenchmark is 4.6.0-rc2 4.6.0-rc2 decstat-v1r20 micro-v1r20 Min alloc-odr0-1 381.00 ( 0.00%) 377.00 ( 1.05%) Min alloc-odr0-2 275.00 ( 0.00%) 273.00 ( 0.73%) Min alloc-odr0-4 229.00 ( 0.00%) 226.00 ( 1.31%) Min alloc-odr0-8 199.00 ( 0.00%) 196.00 ( 1.51%) Min alloc-odr0-16 186.00 ( 0.00%) 183.00 ( 1.61%) Min alloc-odr0-32 179.00 ( 0.00%) 175.00 ( 2.23%) Min alloc-odr0-64 174.00 ( 0.00%) 172.00 ( 1.15%) Min alloc-odr0-128 172.00 ( 0.00%) 170.00 ( 1.16%) Min alloc-odr0-256 181.00 ( 0.00%) 183.00 ( -1.10%) Min alloc-odr0-512 193.00 ( 0.00%) 191.00 ( 1.04%) Min alloc-odr0-1024 201.00 ( 0.00%) 199.00 ( 1.00%) Min alloc-odr0-2048 206.00 ( 0.00%) 204.00 ( 0.97%) Min alloc-odr0-4096 212.00 ( 0.00%) 210.00 ( 0.94%) Min alloc-odr0-8192 215.00 ( 0.00%) 213.00 ( 0.93%) Min alloc-odr0-16384 216.00 ( 0.00%) 214.00 ( 0.93%) Signed-off-by: Mel Gorman <mgorman@techsingularity.net> --- mm/page_alloc.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 9ef2f4ab9ca5..4a364e318873 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -3353,7 +3353,7 @@ __alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order, struct page *page; unsigned int cpuset_mems_cookie; unsigned int alloc_flags = ALLOC_WMARK_LOW|ALLOC_FAIR; - gfp_t alloc_mask; /* The gfp_t that was actually used for allocation */ + gfp_t alloc_mask = gfp_mask; /* The gfp_t that was actually used for allocation */ struct alloc_context ac = { .high_zoneidx = gfp_zone(gfp_mask), .zonelist = zonelist, @@ -3362,6 +3362,7 @@ __alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order, }; if (cpusets_enabled()) { + alloc_mask |= __GFP_HARDWALL; alloc_flags |= ALLOC_CPUSET; if (!ac.nodemask) ac.nodemask = &cpuset_current_mems_allowed; @@ -3389,7 +3390,6 @@ __alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order, ac.classzone_idx = zonelist_zone_idx(preferred_zoneref); /* First allocation attempt */ - alloc_mask = gfp_mask|__GFP_HARDWALL; page = get_page_from_freelist(alloc_mask, order, alloc_flags, &ac); if (unlikely(!page)) { /* @@ -3414,8 +3414,10 @@ __alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order, * the mask is being updated. If a page allocation is about to fail, * check if the cpuset changed during allocation and if so, retry. */ - if (unlikely(!page && read_mems_allowed_retry(cpuset_mems_cookie))) + if (unlikely(!page && read_mems_allowed_retry(cpuset_mems_cookie))) { + alloc_mask = gfp_mask; goto retry_cpuset; + } return page; } -- 2.6.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
next prev parent reply other threads:[~2016-04-15 9:08 UTC|newest] Thread overview: 160+ messages / expand[flat|nested] mbox.gz Atom feed top 2016-04-15 8:58 [PATCH 00/28] Optimise page alloc/free fast paths v3 Mel Gorman 2016-04-15 8:58 ` Mel Gorman 2016-04-15 8:58 ` [PATCH 01/28] mm, page_alloc: Only check PageCompound for high-order pages Mel Gorman 2016-04-15 8:58 ` Mel Gorman 2016-04-25 9:33 ` Vlastimil Babka 2016-04-25 9:33 ` Vlastimil Babka 2016-04-26 10:33 ` Mel Gorman 2016-04-26 10:33 ` Mel Gorman 2016-04-26 11:20 ` Vlastimil Babka 2016-04-26 11:20 ` Vlastimil Babka 2016-04-15 8:58 ` [PATCH 02/28] mm, page_alloc: Use new PageAnonHead helper in the free page fast path Mel Gorman 2016-04-15 8:58 ` Mel Gorman 2016-04-25 9:56 ` Vlastimil Babka 2016-04-25 9:56 ` Vlastimil Babka 2016-04-15 8:58 ` [PATCH 03/28] mm, page_alloc: Reduce branches in zone_statistics Mel Gorman 2016-04-15 8:58 ` Mel Gorman 2016-04-25 11:15 ` Vlastimil Babka 2016-04-25 11:15 ` Vlastimil Babka 2016-04-15 8:58 ` [PATCH 04/28] mm, page_alloc: Inline zone_statistics Mel Gorman 2016-04-15 8:58 ` Mel Gorman 2016-04-25 11:17 ` Vlastimil Babka 2016-04-25 11:17 ` Vlastimil Babka 2016-04-15 8:58 ` [PATCH 05/28] mm, page_alloc: Inline the fast path of the zonelist iterator Mel Gorman 2016-04-15 8:58 ` Mel Gorman 2016-04-25 14:50 ` Vlastimil Babka 2016-04-25 14:50 ` Vlastimil Babka 2016-04-26 10:30 ` Mel Gorman 2016-04-26 10:30 ` Mel Gorman 2016-04-26 11:05 ` Vlastimil Babka 2016-04-26 11:05 ` Vlastimil Babka 2016-04-15 8:58 ` [PATCH 06/28] mm, page_alloc: Use __dec_zone_state for order-0 page allocation Mel Gorman 2016-04-15 8:58 ` Mel Gorman 2016-04-26 11:25 ` Vlastimil Babka 2016-04-26 11:25 ` Vlastimil Babka 2016-04-15 8:58 ` [PATCH 07/28] mm, page_alloc: Avoid unnecessary zone lookups during pageblock operations Mel Gorman 2016-04-15 8:58 ` Mel Gorman 2016-04-26 11:29 ` Vlastimil Babka 2016-04-26 11:29 ` Vlastimil Babka 2016-04-15 8:59 ` [PATCH 08/28] mm, page_alloc: Convert alloc_flags to unsigned Mel Gorman 2016-04-15 8:59 ` Mel Gorman 2016-04-26 11:31 ` Vlastimil Babka 2016-04-26 11:31 ` Vlastimil Babka 2016-04-15 8:59 ` [PATCH 09/28] mm, page_alloc: Convert nr_fair_skipped to bool Mel Gorman 2016-04-15 8:59 ` Mel Gorman 2016-04-26 11:37 ` Vlastimil Babka 2016-04-26 11:37 ` Vlastimil Babka 2016-04-15 8:59 ` [PATCH 10/28] mm, page_alloc: Remove unnecessary local variable in get_page_from_freelist Mel Gorman 2016-04-15 8:59 ` Mel Gorman 2016-04-26 11:38 ` Vlastimil Babka 2016-04-26 11:38 ` Vlastimil Babka 2016-04-15 8:59 ` [PATCH 11/28] mm, page_alloc: Remove unnecessary initialisation " Mel Gorman 2016-04-15 8:59 ` Mel Gorman 2016-04-26 11:39 ` Vlastimil Babka 2016-04-26 11:39 ` Vlastimil Babka 2016-04-15 9:07 ` [PATCH 13/28] mm, page_alloc: Remove redundant check for empty zonelist Mel Gorman 2016-04-15 9:07 ` Mel Gorman 2016-04-15 9:07 ` [PATCH 14/28] mm, page_alloc: Simplify last cpupid reset Mel Gorman 2016-04-15 9:07 ` Mel Gorman 2016-04-26 13:30 ` Vlastimil Babka 2016-04-26 13:30 ` Vlastimil Babka 2016-04-15 9:07 ` [PATCH 15/28] mm, page_alloc: Move might_sleep_if check to the allocator slowpath Mel Gorman 2016-04-15 9:07 ` Mel Gorman 2016-04-26 13:41 ` Vlastimil Babka 2016-04-26 13:41 ` Vlastimil Babka 2016-04-26 14:50 ` Mel Gorman 2016-04-26 14:50 ` Mel Gorman 2016-04-26 15:16 ` Vlastimil Babka 2016-04-26 15:16 ` Vlastimil Babka 2016-04-26 16:29 ` Mel Gorman 2016-04-26 16:29 ` Mel Gorman 2016-04-15 9:07 ` Mel Gorman [this message] 2016-04-15 9:07 ` [PATCH 16/28] mm, page_alloc: Move __GFP_HARDWALL modifications out of the fastpath Mel Gorman 2016-04-26 14:13 ` Vlastimil Babka 2016-04-26 14:13 ` Vlastimil Babka 2016-04-15 9:07 ` [PATCH 17/28] mm, page_alloc: Check once if a zone has isolated pageblocks Mel Gorman 2016-04-15 9:07 ` Mel Gorman 2016-04-26 14:27 ` Vlastimil Babka 2016-04-26 14:27 ` Vlastimil Babka 2016-04-15 9:07 ` [PATCH 18/28] mm, page_alloc: Shorten the page allocator fast path Mel Gorman 2016-04-15 9:07 ` Mel Gorman 2016-04-26 15:23 ` Vlastimil Babka 2016-04-26 15:23 ` Vlastimil Babka 2016-04-15 9:07 ` [PATCH 19/28] mm, page_alloc: Reduce cost of fair zone allocation policy retry Mel Gorman 2016-04-15 9:07 ` Mel Gorman 2016-04-26 17:24 ` Vlastimil Babka 2016-04-26 17:24 ` Vlastimil Babka 2016-04-15 9:07 ` [PATCH 20/28] mm, page_alloc: Shortcut watermark checks for order-0 pages Mel Gorman 2016-04-15 9:07 ` Mel Gorman 2016-04-26 17:32 ` Vlastimil Babka 2016-04-26 17:32 ` Vlastimil Babka 2016-04-15 9:07 ` [PATCH 21/28] mm, page_alloc: Avoid looking up the first zone in a zonelist twice Mel Gorman 2016-04-15 9:07 ` Mel Gorman 2016-04-26 17:46 ` Vlastimil Babka 2016-04-26 17:46 ` Vlastimil Babka 2016-04-15 9:07 ` [PATCH 22/28] mm, page_alloc: Remove field from alloc_context Mel Gorman 2016-04-15 9:07 ` Mel Gorman 2016-04-15 9:07 ` [PATCH 23/28] mm, page_alloc: Check multiple page fields with a single branch Mel Gorman 2016-04-15 9:07 ` Mel Gorman 2016-04-26 18:41 ` Vlastimil Babka 2016-04-26 18:41 ` Vlastimil Babka 2016-04-27 10:07 ` Mel Gorman 2016-04-27 10:07 ` Mel Gorman 2016-04-15 9:07 ` [PATCH 24/28] mm, page_alloc: Remove unnecessary variable from free_pcppages_bulk Mel Gorman 2016-04-15 9:07 ` Mel Gorman 2016-04-26 18:43 ` Vlastimil Babka 2016-04-26 18:43 ` Vlastimil Babka 2016-04-15 9:07 ` [PATCH 25/28] mm, page_alloc: Inline pageblock lookup in page free fast paths Mel Gorman 2016-04-15 9:07 ` Mel Gorman 2016-04-26 19:10 ` Vlastimil Babka 2016-04-26 19:10 ` Vlastimil Babka 2016-04-15 9:07 ` [PATCH 26/28] cpuset: use static key better and convert to new API Mel Gorman 2016-04-15 9:07 ` Mel Gorman 2016-04-26 19:49 ` Vlastimil Babka 2016-04-26 19:49 ` Vlastimil Babka 2016-04-15 9:07 ` [PATCH 27/28] mm, page_alloc: Defer debugging checks of freed pages until a PCP drain Mel Gorman 2016-04-15 9:07 ` Mel Gorman 2016-04-27 11:59 ` Vlastimil Babka 2016-04-27 11:59 ` Vlastimil Babka 2016-04-27 12:01 ` [PATCH 1/3] mm, page_alloc: un-inline the bad part of free_pages_check Vlastimil Babka 2016-04-27 12:01 ` Vlastimil Babka 2016-04-27 12:01 ` [PATCH 2/3] mm, page_alloc: pull out side effects from free_pages_check Vlastimil Babka 2016-04-27 12:01 ` Vlastimil Babka 2016-04-27 12:41 ` Mel Gorman 2016-04-27 12:41 ` Mel Gorman 2016-04-27 13:00 ` Vlastimil Babka 2016-04-27 13:00 ` Vlastimil Babka 2016-04-27 12:01 ` [PATCH 3/3] mm, page_alloc: don't duplicate code in free_pcp_prepare Vlastimil Babka 2016-04-27 12:01 ` Vlastimil Babka 2016-04-27 12:37 ` [PATCH 1/3] mm, page_alloc: un-inline the bad part of free_pages_check Mel Gorman 2016-04-27 12:37 ` Mel Gorman 2016-04-27 12:53 ` Vlastimil Babka 2016-04-27 12:53 ` Vlastimil Babka 2016-04-15 9:07 ` [PATCH 28/28] mm, page_alloc: Defer debugging checks of pages allocated from the PCP Mel Gorman 2016-04-15 9:07 ` Mel Gorman 2016-04-27 14:06 ` Vlastimil Babka 2016-04-27 14:06 ` Vlastimil Babka 2016-04-27 15:31 ` Mel Gorman 2016-04-27 15:31 ` Mel Gorman 2016-05-17 6:41 ` Naoya Horiguchi 2016-05-17 6:41 ` Naoya Horiguchi 2016-05-18 7:51 ` Vlastimil Babka 2016-05-18 7:51 ` Vlastimil Babka 2016-05-18 7:55 ` Vlastimil Babka 2016-05-18 7:55 ` Vlastimil Babka 2016-05-18 8:49 ` Mel Gorman 2016-05-18 8:49 ` Mel Gorman 2016-04-26 12:04 ` [PATCH 13/28] mm, page_alloc: Remove redundant check for empty zonelist Vlastimil Babka 2016-04-26 12:04 ` Vlastimil Babka 2016-04-26 13:00 ` Mel Gorman 2016-04-26 13:00 ` Mel Gorman 2016-04-26 19:11 ` Andrew Morton 2016-04-26 19:11 ` Andrew Morton 2016-04-15 12:44 ` [PATCH 00/28] Optimise page alloc/free fast paths v3 Jesper Dangaard Brouer 2016-04-15 12:44 ` Jesper Dangaard Brouer 2016-04-15 13:08 ` Mel Gorman 2016-04-15 13:08 ` Mel Gorman 2016-04-16 7:21 ` [PATCH 12/28] mm, page_alloc: Remove unnecessary initialisation from __alloc_pages_nodemask() Mel Gorman 2016-04-16 7:21 ` Mel Gorman 2016-04-26 11:41 ` Vlastimil Babka 2016-04-26 11:41 ` Vlastimil Babka
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=1460711275-1130-4-git-send-email-mgorman@techsingularity.net \ --to=mgorman@techsingularity.net \ --cc=akpm@linux-foundation.org \ --cc=brouer@redhat.com \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-mm@kvack.org \ --cc=vbabka@suse.cz \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.