[RFC] mm: MADV_COLLAPSE semantics

["Zach O'Keefe" <zokeefe@google.com>]

Hey All,

I'm sending this out before the v6 of "mm: userspace hugepage
collapse" for the purposes of aligning on and finalizing the semantics
of the proposed MADV_COLLAPSE madvise(2) mode.

Background:

So far, thanks to everyone's input, we've aligned on:
- MADV_COLLAPSE specifies its own hugepage allocation semantics (it
allows direct reclaim/compaction).
- MADV_COLLAPSE ignores khugepaged heuristics
(/sys/kernel/mm/transparent_hugepage/khugepaged/max_pte_* and
young/referenced page requirements).

In terms of THP _eligibility_, in v5 it was proposed that
MADV_COLLAPSE follow existing THP eligibility semantics
(/sys/kernel/mm/transparent_hugepage/enabled + the VMA flags of the
VMA being collapsed)[1].

However, Rongwei Wang kindly pointed out that the useability of
process_madvise(MADV_COLLAPSE) on a system in "madvise" THP mode was
limited. I agreed to include process_madvise(2) support for
MADV_[NO]HUGEPAGE in v6, but following a discussion with David H., I
think that was a mistake.  Namely, as David kindly pointed out, there
exist programs that don't
work with THP and have good reason to disable it. The example
provided was postcopy life migration in QEMU, which explicitly
disables THP right before faulting in any pages.

Idea: MADV_COLLAPSE should respect VM_NOHUGEPAGE and "never" THP mode,
but otherwise would attempt to collapse.

Why? If someone(*), somewhere told us not to use THPs, then don't
override that decision. Otherwise, this is an explicit, safe(**)
request made on behalf of ourselves, or by a CAP_SYS_ADMIN process,
and shouldn't be blocked by interfaces meant to guide the
"transparent" part of THPs.

Other options considered:

I considered variations of setting VM_HUGEPAGE only if calling on
behalf of self or if VM_NOHUGEPAGE is not set. However, I didn't like
this because there isn't a way to undo the operation: If we supported
process_madvise(MADV_NOHUGEPAGE), we would have to let the application
unclear VM_NOHUGEPAGE because outside processes can't/shouldn't. It
would have to require some *new* madvise mode like MADV_CLEARHUGEPAGE
(that would fail if calling on behalf of another process and
VM_NOHUGEPAGE set) to clear VM_[NO]HUGEPAGE.

A possible downside to the proposed approach is that, if in "madvise"
THP mode and collapsing a VMA not marked VM_HUGEPAGE, it's now the
caller's responsibility to monitor and recollapse this memory back
into THPs. However, in practice this likely means an explicit
MADV_DONTNEED (please let me know if there are other important cases
here), and presumably it's the caller's job to do the monitoring anyway.

Thanks again for taking the time to read / provide input here. I think
this is the last point to clear up before releasing a v6 that should
hopefully have all the functionality we need.

Best,
Zach

---

(*) If we could verify that "never" THP mode was used _only_ for
debugging, then I'd actually opt to ignore "never" in MADV_COLLAPSE.
It's the last dependency MADV_COLLAPSE has on sysfs THP interface and
would provide a convenient way to test/debug MADV_COLLAPSE with
khugepaged / at-fault disabled.
(**) I suppose there could exist applications that see THP "madvise"
mode, never call MADV_HUGEPAGE, and so assume THPs will never be
found.

[1] https://lore.kernel.org/linux-mm/20220504214437.2850685-1-zokeefe@google.com/
[2] https://lore.kernel.org/linux-mm/502a3ced-f3c6-7117-3b24-d80d204d66ee@linux.alibaba.com/