Re: [RFC 0/4] mm: Introduce lazy exec permission setting on a page

[Catalin Marinas <catalin.marinas@arm.com>]

On Thu, Feb 14, 2019 at 09:38:44AM +0100, Michal Hocko wrote:
> On Thu 14-02-19 11:34:09, Anshuman Khandual wrote:
> > On 02/13/2019 09:08 PM, Michal Hocko wrote:
> > > Are there any numbers to show the optimization impact?
> > 
> > This series transfers execution cost linearly with nr_pages from migration path
> > to subsequent exec access path for normal, THP and HugeTLB pages. The experiment
> > is on mainline kernel (1f947a7a011fcceb14cb912f548) along with some patches for
> > HugeTLB and THP migration enablement on arm64 platform.
> 
> Please make sure that these numbers are in the changelog. I am also
> missing an explanation why this is an overal win. Why should we pay
> on the later access rather than the migration which is arguably a slower
> path. What is the usecase that benefits from the cost shift?

Originally the investigation started because of a regression we had
sending IPIs on each set_pte_at(PROT_EXEC). This has been fixed
separately, so the original value of this patchset has been diminished.

Trying to frame the problem, let's analyse the overall cost of migration
+ execute. Removing other invariants like cost of the initial mapping of
the pages or the mapping of new pages after migration, we have:

M - number of mapped executable pages just before migration
N - number of previously mapped pages that will be executed after
    migration (N <= M)
D - cost of migrating page data
I - cost of I-cache maintenance for a page
F - cost of an instruction fault (handle_mm_fault() + set_pte_at()
    without the actual I-cache maintenance)

Tc - total migration cost current kernel (including executing)
Tp - total migration cost patched kernel (including executing)

  Tc = M * (D + I)
  Tp = M * D + N * (F + I)

To be useful, we want this patchset to lead to:

  Tp < Tc

Simplifying:

  M * D + N * (F + I) < M * (D + I)
  ...
  F < I * (M - N) / N

So the question is, in a *real-world* scenario, what proportion of the
mapped executable pages would still be executed from after migration.
I'd leave this as a task for Anshuman to investigate and come up with
some numbers (and it's fine if it's just in the noise, we won't need
this patchset).

Also note that there are ARM CPU implementations that don't need I-cache
maintenance (the I side can snoop the D side), so for those this
patchset introducing an additional cost. But we can make the decision in
the arch code via pte_mklazyexec().

We implemented something similar in arm64 KVM (d0e22b4ac3ba "KVM:
arm/arm64: Limit icache invalidation to prefetch aborts") but the
use-case was different: previously KVM considered all pages executable
though the vast majority were only data pages in guests.

-- 
Catalin