From mboxrd@z Thu Jan 1 00:00:00 1970 From: David Goldblatt Subject: Re: [PATCH v2 0/2] mm: introduce MAP_FIXED_SAFE Date: Wed, 13 Dec 2017 17:35:48 -0800 Message-ID: References: <20171213092550.2774-1-mhocko@kernel.org> <20171213163210.6a16ccf8753b74a6982ef5b6@linux-foundation.org> Mime-Version: 1.0 Content-Type: multipart/alternative; boundary="001a11424216604b00056042e777" Return-path: In-Reply-To: <20171213163210.6a16ccf8753b74a6982ef5b6@linux-foundation.org> Sender: owner-linux-mm@kvack.org To: Andrew Morton Cc: Michal Hocko , linux-api@vger.kernel.org, Khalid Aziz , Michael Ellerman , Russell King - ARM Linux , Andrea Arcangeli , linux-mm@kvack.org, LKML , linux-arch@vger.kernel.org, Florian Weimer , John Hubbard , Matthew Wilcox , Abdul Haleem , Joel Stanley , Kees Cook , Michal Hocko , trasz@freebsd.org, Jason Evans List-Id: linux-api@vger.kernel.org --001a11424216604b00056042e777 Content-Type: text/plain; charset="UTF-8" (+cc the jemalloc jasone; -cc,+bcc the Google jasone). The only time we would want MAP_FIXED (or rather, a non-broken variant) is in the middle of trying to expand an allocation in place; "atomic address range probing in the multithreaded programs" describes our use case pretty well. That's in a pathway that usually fails; it's pretty far down on our kernel mmap enhancements wish-list. On Wed, Dec 13, 2017 at 4:32 PM, Andrew Morton wrote: > On Wed, 13 Dec 2017 10:25:48 +0100 Michal Hocko wrote: > > > > > Hi, > > I am resending with some minor updates based on Michael's review and > > ask for inclusion. There haven't been any fundamental objections for > > the RFC [1] nor the previous version [2]. The biggest discussion > > revolved around the naming. There were many suggestions flowing > > around MAP_REQUIRED, MAP_EXACT, MAP_FIXED_NOCLOBBER, MAP_AT_ADDR, > > MAP_FIXED_NOREPLACE etc... > > I like MAP_FIXED_CAREFUL :) > > > I am afraid we can bikeshed this to death and there will still be > > somebody finding yet another better name. Therefore I've decided to > > stick with my original MAP_FIXED_SAFE. Why? Well, because it keeps the > > MAP_FIXED prefix which should be recognized by developers and _SAFE > > suffix should also be clear that all dangerous side effects of the old > > MAP_FIXED are gone. > > > > If somebody _really_ hates this then feel free to nack and resubmit > > with a different name you can find a consensus for. I am sorry to be > > stubborn here but I would rather have this merged than go over few more > > iterations changing the name just because it seems like a good idea > > now. My experience tells me that chances are that the name will turn out > > to be "suboptimal" anyway over time. > > > > Some more background: > > This has started as a follow up discussion [3][4] resulting in the > > runtime failure caused by hardening patch [5] which removes MAP_FIXED > > from the elf loader because MAP_FIXED is inherently dangerous as it > > might silently clobber an existing underlying mapping (e.g. stack). The > > reason for the failure is that some architectures enforce an alignment > > for the given address hint without MAP_FIXED used (e.g. for shared or > > file backed mappings). > > > > One way around this would be excluding those archs which do alignment > > tricks from the hardening [6]. The patch is really trivial but it has > > been objected, rightfully so, that this screams for a more generic > > solution. We basically want a non-destructive MAP_FIXED. > > > > The first patch introduced MAP_FIXED_SAFE which enforces the given > > address but unlike MAP_FIXED it fails with EEXIST if the given range > > conflicts with an existing one. The flag is introduced as a completely > > new one rather than a MAP_FIXED extension because of the backward > > compatibility. We really want a never-clobber semantic even on older > > kernels which do not recognize the flag. Unfortunately mmap sucks wrt. > > flags evaluation because we do not EINVAL on unknown flags. On those > > kernels we would simply use the traditional hint based semantic so the > > caller can still get a different address (which sucks) but at least not > > silently corrupt an existing mapping. I do not see a good way around > > that. Except we won't export expose the new semantic to the userspace at > > all. > > > > It seems there are users who would like to have something like that. > > Jemalloc has been mentioned by Michael Ellerman [7] > > http://lkml.kernel.org/r/87efp1w7vy.fsf@concordia.ellerman.id.au. > > It would be useful to get feedback from jemalloc developers (please). > I'll add some cc's. > > > > Florian Weimer has mentioned the following: > > : glibc ld.so currently maps DSOs without hints. This means that the > kernel > > : will map right next to each other, and the offsets between them a > completely > > : predictable. We would like to change that and supply a random address > in a > > : window of the address space. If there is a conflict, we do not want > the > > : kernel to pick a non-random address. Instead, we would try again with a > > : random address. > > > > John Hubbard has mentioned CUDA example > > : a) Searches /proc//maps for a "suitable" region of available > > : VA space. "Suitable" generally means it has to have a base address > > : within a certain limited range (a particular device model might > > : have odd limitations, for example), it has to be large enough, and > > : alignment has to be large enough (again, various devices may have > > : constraints that lead us to do this). > > : > > : This is of course subject to races with other threads in the process. > > : > > : Let's say it finds a region starting at va. > > : > > : b) Next it does: > > : p = mmap(va, ...) > > : > > : *without* setting MAP_FIXED, of course (so va is just a hint), to > > : attempt to safely reserve that region. If p != va, then in most cases, > > : this is a failure (almost certainly due to another thread getting a > > : mapping from that region before we did), and so this layer now has to > > : call munmap(), before returning a "failure: retry" to upper layers. > > : > > : IMPROVEMENT: --> if instead, we could call this: > > : > > : p = mmap(va, ... MAP_FIXED_SAFE ...) > > : > > : , then we could skip the munmap() call upon failure. This > > : is a small thing, but it is useful here. (Thanks to Piotr > > : Jaroszynski and Mark Hairgrove for helping me get that detail > > : exactly right, btw.) > > : > > : c) After that, CUDA suballocates from p, via: > > : > > : q = mmap(sub_region_start, ... MAP_FIXED ...) > > : > > : Interestingly enough, "freeing" is also done via MAP_FIXED, and > > : setting PROT_NONE to the subregion. Anyway, I just included (c) for > > : general interest. > > > > Atomic address range probing in the multithreaded programs in general > > sounds like an interesting thing to me. > > > > The second patch simply replaces MAP_FIXED use in elf loader by > > MAP_FIXED_SAFE. I believe other places which rely on MAP_FIXED should > > follow. Actually real MAP_FIXED usages should be docummented properly > > and they should be more of an exception. > > --001a11424216604b00056042e777 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
(+cc the jemalloc jasone; -cc,+bcc the Google jasone).
The only time we would want MAP_FIXED (or rather, a non-bro= ken variant) is in the middle of trying to expand an allocation in place; &= quot;atomic address range probing in the m= ultithreaded programs" describes our use case pretty well. That's = in a pathway that usually fails; it's pretty far down on our kernel mma= p enhancements wish-list.

=
On Wed, Dec 13, 2017 at 4:32 PM, Andrew Morton <= span dir=3D"ltr"><akpm@linux-foundation.org> wrote:
On Wed, 13 Dec 2017 10:25:48 +0100 Michal Hocko <mhocko@kernel.org> wrote:

>
> Hi,
> I am resending with some minor updates based on Michael's review a= nd
> ask for inclusion. There haven't been any fundamental objections f= or
> the RFC [1] nor the previous version [2].=C2=A0 The biggest discussion=
> revolved around the naming. There were many suggestions flowing
> around MAP_REQUIRED, MAP_EXACT, MAP_FIXED_NOCLOBBER, MAP_AT_ADDR,
> MAP_FIXED_NOREPLACE etc...

I like MAP_FIXED_CAREFUL :)

> I am afraid we can bikeshed this to death and there will still be
> somebody finding yet another better name. Therefore I've decided t= o
> stick with my original MAP_FIXED_SAFE. Why? Well, because it keeps the=
> MAP_FIXED prefix which should be recognized by developers and _SAFE > suffix should also be clear that all dangerous side effects of the old=
> MAP_FIXED are gone.
>
> If somebody _really_ hates this then feel free to nack and resubmit > with a different name you can find a consensus for. I am sorry to be > stubborn here but I would rather have this merged than go over few mor= e
> iterations changing the name just because it seems like a good idea > now. My experience tells me that chances are that the name will turn o= ut
> to be "suboptimal" anyway over time.
>
> Some more background:
> This has started as a follow up discussion [3][4] resulting in the
> runtime failure caused by hardening patch [5] which removes MAP_FIXED<= br> > from the elf loader because MAP_FIXED is inherently dangerous as it > might silently clobber an existing underlying mapping (e.g. stack). Th= e
> reason for the failure is that some architectures enforce an alignment=
> for the given address hint without MAP_FIXED used (e.g. for shared or<= br> > file backed mappings).
>
> One way around this would be excluding those archs which do alignment<= br> > tricks from the hardening [6]. The patch is really trivial but it has<= br> > been objected, rightfully so, that this screams for a more generic
> solution. We basically want a non-destructive MAP_FIXED.
>
> The first patch introduced MAP_FIXED_SAFE which enforces the given
> address but unlike MAP_FIXED it fails with EEXIST if the given range > conflicts with an existing one. The flag is introduced as a completely=
> new one rather than a MAP_FIXED extension because of the backward
> compatibility. We really want a never-clobber semantic even on older > kernels which do not recognize the flag. Unfortunately mmap sucks wrt.=
> flags evaluation because we do not EINVAL on unknown flags. On those > kernels we would simply use the traditional hint based semantic so the=
> caller can still get a different address (which sucks) but at least no= t
> silently corrupt an existing mapping. I do not see a good way around > that. Except we won't export expose the new semantic to the usersp= ace at
> all.
>
> It seems there are users who would like to have something like that. > Jemalloc has been mentioned by Michael Ellerman [7]

http://lkml.kernel.org/r/87efp1= w7vy.fsf@concordia.ellerman.id.au.

It would be useful to get feedback from jemalloc developers (please).
I'll add some cc's.


> Florian Weimer has mentioned the following:
> : glibc ld.so currently maps DSOs without hints.=C2=A0 This means that= the kernel
> : will map right next to each other, and the offsets between them a co= mpletely
> : predictable.=C2=A0 We would like to change that and supply a random = address in a
> : window of the address space.=C2=A0 If there is a conflict, we do not= want the
> : kernel to pick a non-random address. Instead, we would try again wit= h a
> : random address.
>
> John Hubbard has mentioned CUDA example
> : a) Searches /proc/<pid>/maps for a "suitable" region= of available
> : VA space.=C2=A0 "Suitable" generally means it has to have = a base address
> : within a certain limited range (a particular device model might
> : have odd limitations, for example), it has to be large enough, and > : alignment has to be large enough (again, various devices may have > : constraints that lead us to do this).
> :
> : This is of course subject to races with other threads in the process= .
> :
> : Let's say it finds a region starting at va.
> :
> : b) Next it does:
> :=C2=A0 =C2=A0 =C2=A0p =3D mmap(va, ...)
> :
> : *without* setting MAP_FIXED, of course (so va is just a hint), to > : attempt to safely reserve that region. If p !=3D va, then in most ca= ses,
> : this is a failure (almost certainly due to another thread getting a<= br> > : mapping from that region before we did), and so this layer now has t= o
> : call munmap(), before returning a "failure: retry" to uppe= r layers.
> :
> :=C2=A0 =C2=A0 =C2=A0IMPROVEMENT: --> if instead, we could call thi= s:
> :
> :=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0p =3D mmap(va, ... MA= P_FIXED_SAFE ...)
> :
> :=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0, then we could skip the munmap() c= all upon failure. This
> :=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0is a small thing, but it is useful = here. (Thanks to Piotr
> :=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Jaroszynski and Mark Hairgrove for = helping me get that detail
> :=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0exactly right, btw.)
> :
> : c) After that, CUDA suballocates from p, via:
> :
> :=C2=A0 =C2=A0 =C2=A0 q =3D mmap(sub_region_start, ... MAP_FIXED ...)<= br> > :
> : Interestingly enough, "freeing" is also done via MAP_FIXED= , and
> : setting PROT_NONE to the subregion. Anyway, I just included (c) for<= br> > : general interest.
>
> Atomic address range probing in the multithreaded programs in general<= br> > sounds like an interesting thing to me.
>
> The second patch simply replaces MAP_FIXED use in elf loader by
> MAP_FIXED_SAFE. I believe other places which rely on MAP_FIXED should<= br> > follow. Actually real MAP_FIXED usages should be docummented properly<= br> > and they should be more of an exception.


--001a11424216604b00056042e777-- -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org