From: David Rientjes <firstname.lastname@example.org> To: Mel Gorman <email@example.com> Cc: Michal Hocko <firstname.lastname@example.org>, Andrew Morton <email@example.com>, Vlastimil Babka <firstname.lastname@example.org>, Andrea Argangeli <email@example.com>, Zi Yan <firstname.lastname@example.org>, Stefan Priebe - Profihost AG <email@example.com>, "Kirill A. Shutemov" <firstname.lastname@example.org>, email@example.com, LKML <firstname.lastname@example.org>, Andrea Arcangeli <email@example.com>, Stable tree <firstname.lastname@example.org>, Michal Hocko <email@example.com> Subject: Re: [PATCH 1/2] mm: thp: relax __GFP_THISNODE for MADV_HUGEPAGE mappings Date: Fri, 5 Oct 2018 13:35:15 -0700 (PDT) Message-ID: <alpine.DEB.firstname.lastname@example.org> (raw) In-Reply-To: <20181005073854.GB6931@suse.de> On Fri, 5 Oct 2018, Mel Gorman wrote: > > This causes, on average, a 13.9% access latency regression on Haswell, and > > the regression would likely be more severe on Naples and Rome. > > > > That assumes that fragmentation prevents easy allocation which may very > well be the case. While it would be great that compaction or the page > allocator could be further improved to deal with fragmentation, it's > outside the scope of this patch. > Hi Mel, The regression that Andrea is working on, correct me if I'm wrong, is heavy reclaim and swapping activity that is trying to desperately allocate local hugepages when the local node is fragmented based on advice provided by MADV_HUGEPAGE. Why is it ever appropriate to do heavy reclaim and swap activity to allocate a transparent hugepage? This is exactly what the __GFP_NORETRY check for high-order allocations is attempting to avoid, and it explicitly states that it is for thp faults. The fact that we lost __GFP_NORERY for thp allocations for all settings, including the default setting, other than yours (setting of "always") is what I'm focusing on. There is no guarantee that this activity will free an entire pageblock or that it is even worthwhile. Why is thp memory ever being allocated without __GFP_NORETRY as the page allocator expects? That aside, removing __GFP_THISNODE can make the fault latency much worse if remote notes are fragmented and/or reclaim has the inability to free contiguous memory, which it likely cannot. This is where I measured over 40% fault latency regression from Linus's tree with this patch on a fragmnented system where order-9 memory is neither available from node 0 or node 1 on Haswell. > > There exist libraries that allow the .text segment of processes to be > > remapped to memory backed by transparent hugepages and use MADV_HUGEPAGE > > to stress local compaction to defragment node local memory for hugepages > > at startup. > > That is taking advantage of a co-incidence of the implementation. > MADV_HUGEPAGE is *advice* that huge pages be used, not what the locality > is. A hint for strong locality preferences should be separate advice > (madvise) or a separate memory policy. Doing that is outside the context > of this patch but nothing stops you introducing such a policy or madvise, > whichever you think would be best for the libraries to consume (I'm only > aware of libhugetlbfs but there might be others). > The behavior that MADV_HUGEPAGE specifies is certainly not clearly defined, unfortunately. The way that an application writer may read it, as we have, is that it will make a stronger attempt at allocating a hugepage at fault. This actually works quite well when the allocation correctly has __GFP_NORETRY, as it's supposed to, and compaction is MIGRATE_ASYNC. So rather than focusing on what MADV_HUGEPAGE has meant over the past 2+ years of kernels that we have implemented based on, or what it meant prior to that, is a fundamental question of the purpose of direct reclaim and swap activity that had always been precluded before __GFP_NORETRY was removed in a thp allocation. I don't think anybody in this thread wants 14% remote access latency regression if we allocate remotely or 40% fault latency regression when remote nodes are fragmented as well. Removing __GFP_THISNODE only helps when remote memory is not fragmented, otherwise it multiplies the problem as I've shown. The numbers that you provide while using the non-default option to mimick MADV_HUGEPAGE mappings but also use __GFP_NORETRY makes the actual source of the problem quite easy to identify: there is an inconsistency in the thp gfp mask and the page allocator implementation. > > The cost, including the statistics Mel gathered, is > > acceptable for these processes: they are not concerned with startup cost, > > they are concerned only with optimal access latency while they are > > running. > > > > Then such applications at startup have the option of setting > zone_reclaim_mode during initialisation assuming a privileged helper > can be created. That would be somewhat heavy handed and a longer-term > solution would still be to create a proper memory policy of madvise flag > for those libraries. > We *never* want to use zone_reclaim_mode for these allocations, that would be even worse, we do not want to reclaim because we have a very unlikely chance of making pageblocks free without the involvement of compaction. We want to trigger memory compaction with a well-bounded cost that MIGRATE_ASYNC provides and then fail.
next prev parent reply index Thread overview: 74+ messages / expand[flat|nested] mbox.gz Atom feed top 2018-09-25 12:03 [PATCH 0/2] thp nodereclaim fixes Michal Hocko 2018-09-25 12:03 ` [PATCH 1/2] mm: thp: relax __GFP_THISNODE for MADV_HUGEPAGE mappings Michal Hocko 2018-09-25 12:20 ` Mel Gorman 2018-09-25 12:30 ` Michal Hocko 2018-10-04 20:16 ` David Rientjes 2018-10-04 21:10 ` Andrea Arcangeli 2018-10-04 23:05 ` David Rientjes 2018-10-06 3:19 ` Andrea Arcangeli 2018-10-05 7:38 ` Mel Gorman 2018-10-05 20:35 ` David Rientjes [this message] 2018-10-05 23:21 ` Andrea Arcangeli 2018-10-08 20:41 ` David Rientjes 2018-10-09 9:48 ` Mel Gorman 2018-10-09 12:27 ` Michal Hocko 2018-10-09 13:00 ` Mel Gorman 2018-10-09 14:25 ` Michal Hocko 2018-10-09 15:16 ` Mel Gorman 2018-10-09 23:03 ` Andrea Arcangeli 2018-10-10 21:19 ` David Rientjes 2018-10-15 22:30 ` David Rientjes 2018-10-15 22:44 ` Andrew Morton 2018-10-15 23:19 ` Andrea Arcangeli 2018-10-22 20:54 ` David Rientjes 2018-10-16 7:46 ` Mel Gorman 2018-10-16 22:37 ` Andrew Morton 2018-10-16 23:11 ` Andrea Arcangeli 2018-10-16 23:16 ` Andrew Morton 2018-10-17 7:08 ` Michal Hocko 2018-10-17 9:00 ` Mel Gorman 2018-10-22 21:04 ` David Rientjes 2018-10-23 1:27 ` Zi Yan 2018-10-28 21:45 ` David Rientjes 2018-10-23 7:57 ` Mel Gorman 2018-10-23 8:38 ` Mel Gorman 2018-10-15 22:57 ` Andrea Arcangeli 2018-10-22 20:45 ` David Rientjes 2018-10-09 22:17 ` David Rientjes 2018-10-09 22:51 ` Andrea Arcangeli 2018-10-10 7:54 ` Vlastimil Babka 2018-10-10 21:00 ` David Rientjes 2018-10-09 13:08 ` Vlastimil Babka 2018-10-09 22:21 ` Andrea Arcangeli 2018-10-29 5:17 ` Balbir Singh 2018-10-29 9:00 ` Michal Hocko 2018-10-29 9:42 ` Balbir Singh 2018-10-29 10:08 ` Michal Hocko 2018-10-29 10:56 ` Andrea Arcangeli 2018-09-25 12:03 ` [PATCH 2/2] mm, thp: consolidate THP gfp handling into alloc_hugepage_direct_gfpmask Michal Hocko 2018-09-26 13:30 ` Kirill A. Shutemov 2018-09-26 14:17 ` Michal Hocko 2018-09-26 14:22 ` Michal Hocko 2018-10-19 2:11 ` Andrew Morton 2018-10-19 8:06 ` Michal Hocko 2018-10-22 13:27 ` Vlastimil Babka 2018-10-24 23:17 ` Andrew Morton 2018-10-25 4:56 ` Vlastimil Babka 2018-10-25 16:14 ` Michal Hocko 2018-10-25 16:18 ` Andrew Morton 2018-10-25 16:45 ` Michal Hocko 2018-10-22 13:15 ` Vlastimil Babka 2018-10-22 13:30 ` Michal Hocko 2018-10-22 13:35 ` Vlastimil Babka 2018-10-22 13:46 ` Michal Hocko 2018-10-22 13:53 ` Vlastimil Babka 2018-10-04 20:17 ` David Rientjes 2018-10-04 21:49 ` Zi Yan 2018-10-09 12:36 ` Michal Hocko 2018-09-26 13:08 ` linux-mm@ archive on lore.kernel.org (Was: [PATCH 0/2] thp nodereclaim fixes) Kirill A. Shutemov 2018-09-26 13:14 ` Michal Hocko 2018-09-26 22:22 ` Andrew Morton 2018-09-26 23:08 ` Mel Gorman 2018-09-27 0:47 ` Konstantin Ryabitsev 2018-09-26 15:25 ` Konstantin Ryabitsev 2018-09-27 11:30 ` Kirill A. Shutemov
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=alpine.DEB.email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
Linux-mm Archive on lore.kernel.org Archives are clonable: git clone --mirror https://lore.kernel.org/linux-mm/0 linux-mm/git/0.git # If you have public-inbox 1.1+ installed, you may # initialize and index your mirror using the following commands: public-inbox-init -V2 linux-mm linux-mm/ https://lore.kernel.org/linux-mm \ firstname.lastname@example.org public-inbox-index linux-mm Example config snippet for mirrors Newsgroup available over NNTP: nntp://nntp.lore.kernel.org/org.kvack.linux-mm AGPL code for this site: git clone https://public-inbox.org/public-inbox.git