Re: hugetlb pages not accounted for in rss

[David Rientjes <rientjes@google.com>]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2617 bytes --]

On Thu, 30 Jul 2015, Jörn Engel wrote:

> > If I want to track hugetlb usage on a per-task basis, do I then need to
> > create one cgroup per task?
> > 

I think this would only be used for debugging or testing, but if you have 
root and are trying to organize processes into a hugetlb_cgroup hierarchy, 
presumably you would just look at smaps and find each thread's hugetlb 
memory usage and not bother.

> Maybe some background is useful.  I would absolutely love to use
> transparent hugepages.  They are absolutely perfect in every respect,
> except for performance.  With transparent hugepages we get higher
> latencies.  Small pages are unacceptable, so we are forced to use
> non-transparent hugepages.
> 

Believe me, we are on the same page that way :)  We still deploy 
configurations with hugetlb memory because we need to meet certain 
allocation requirements and it is only possible to do at boot.

With regard to the performance of thp, I can think of two things that are 
affecting you:

 - allocation cost

   Async memory compaction in the page fault path for thp memory is very
   lightweight and it happily falls back to using small pages instead.
   Memory compaction is always being improved upon and there is on-going
   work to do memory compaction both periodically and in the background to
   keep fragmentation low.  The ultimate goal would be to remove async
   compaction entirely from the thp page fault path and rely on 
   improvements to memory compaction such that we have a great allocation
   success rate and less cost when we fail.

 - NUMA cost

   Until very recently, thp pages could easily be allocated remotely
   instead of small pages locally.  That has since been improved and we
   only allocate thp locally and then fallback to small pages locally
   first.  Khugepaged can still migrate memory remotely, but it will
   allocate the hugepage on the node where the majority of smallpages
   are from.

> The part of our system that uses small pages is pretty much constant,
> while total system memory follows Moore's law.  When possible we even
> try to shrink that part.  Hugepages already dominate today and things
> will get worse.
> 

I wrote a patchset, hugepages overcommit, that allows unmapped hugetlb 
pages to be freed in oom conditions before calling the oom killer up to a 
certain threshold and then kickoff a background thread to try to 
reallocate them.  The idea is to keep the hugetlb pool as large as 
possible up to oom and then only reclaim what is needed and then try to 
reallocate them.  Not sure if it would help your particular usecase or 
not.