Re: Prerequisites for Large Anon Folios

[David Hildenbrand <david@redhat.com>]

On 30.08.23 12:44, Ryan Roberts wrote:
> Hi All,
> 

Hi Ryan,

I'll be back from vacation next Wednesday.

Note that I asked David R. to have large anon folios as topic for the 
next bi-weekly mm meeting.

There, we should discuss things like
* naming
* accounting (/proc/meminfo)
* required toggles (especially, to ways to disable it, as we want to
   keep toggles minimal)

David R. raised that there are certainly workloads where the additional 
memory overhead is usually not acceptable. So it will be valuable to get 
input from others.

> 
> I want to get serious about getting large anon folios merged. To do that, there
> are a number of outstanding prerequistes. I'm hoping the respective owners may
> be able to provide an update on progress?

I shared some details in the last meeting when you were on vacation :)

High level update below.

[...]

>>
>> - item:
>>      shared vs exclusive mappings
>>
>>    priority:
>>      prerequisite
>>
>>    description: >-
>>      New mechanism to allow us to easily determine precisely whether a given
>>      folio is mapped exclusively or shared between multiple processes. Required
>>      for (from David H):
>>
>>      (1) Detecting shared folios, to not mess with them while they are shared.
>>      MADV_PAGEOUT, user-triggered page migration, NUMA hinting, khugepaged ...
>>      replace cases where folio_estimated_sharers() == 1 would currently be the
>>      best we can do (and in some cases, page_mapcount() == 1).
>>
>>      (2) COW improvements for PTE-mapped large anon folios after fork(). Before
>>      fork(), PageAnonExclusive would have been reliable, after fork() it's not.
>>
>>      For (1), "MADV_PAGEOUT" maps to the "madvise" item captured in this list. I
>>      *think* "NUMA hinting" maps to "numa balancing" (but need confirmation!).
>>      "user-triggered page migration" and "khugepaged" not yet captured (would
>>      appreciate someone fleshing it out). I previously understood migration to be
>>      working for large folios - is "user-triggered page migration" some specific
>>      aspect that does not work?
>>
>>      For (2), this relates to Large Anon Folio enhancements which I plan to
>>      tackle after we get the basic series merged.
>>
>>    links:
>>      - 'email thread: Mapcount games: "exclusive mapped" vs. "mapped shared"'
>>
>>    location:
>>      - shrink_folio_list()
>>
>>    assignee:
>>      David Hildenbrand <david@redhat.com>
> 
> Any comment on this David? I think the last comment I saw was that you were
> planning to start an implementation a couple of weeks back? Did that get anywhere?

The math should be solid at this point and I had a simple prototype 
running -- including fairly clean COW reuse handling.

I started cleaning it all up before my vacation. I'll first need the 
total mapcount (which I sent), and might have to implement rmap patching 
during THP split (easy), but I first have to do more measurements.

Willies patches to free up space in the first tail page will be 
required. In addition, my patches to free up ->private in tail pages for 
THP_SWAP. Both things on their way upstream.

Based on that, I need a bit spinlock to protect the total 
mapcount+tracking data. There are things to measure (contention) and 
optimize (why even care about tracking shared vs. exclusive if it's 
pretty guaranteed to always be shared -- for example, shared libraries).

So it looks reasonable at this point, but I'll have to look into 
possible contentions and optimizations once I have the basics 
implemented cleanly.

It's a shame we cannot get the subpage mapcount out of the way 
immediately, then it wouldn't be "additional tracking" but "different 
tracking" :)

Once back from vacation, I'm planning on prioritizing this. Shouldn't 
take ages to get it cleaned up. Measurements and optimizations might 
take a bit longer.

[...]


>>
>>    assignee:
>>      Yin, Fengwei <fengwei.yin@intel.com>
> 
> As I understand it: initial solution based on folio_estimated_sharers() has gone
> into v6.5. Have a dependecy on David's precise shared vs exclusive work for an

shared vs. exclusive in place would replace folio_estimated_sharers() 
users and most sub-page mapcount users.

> improved solution. And I think you mentioned you are planning to do a change
> that avoids splitting a large folio if it is entirely covered by the range?

[..]
>>
>> - item:
>>      numa balancing
>>
>>    priority:
>>      prerequisite
>>
>>    description: >-
>>      Large, pte-mapped folios are ignored by numa-balancing code. Commit comment
>>      (e81c480): "We're going to have THP mapped with PTEs. It will confuse
>>      numabalancing. Let's skip them for now." Likely depends on "shared vs
>>      exclusive mappings". >>
>>    links: []
>>
>>    location:
>>      - do_numa_page()
>>
>>    assignee:
>>      <none>
>>
> 
> Vaguely sounded like David might be planning to tackle this as part of his work
> on "shared vs exclusive mappings" ("NUMA hinting"??). David?

It should be easy to handle it based on that. Similarly, khugepaged IIRC.

-- 
Cheers,

David / dhildenb