ucontext, kernel vs. userspace (glibc)

ucontext, kernel vs. userspace (glibc)

[Benjamin Herrenschmidt]

Hi Folks !

So I'm discovering arm64 intricacies and today, as I was looking at SVE
support (in the context of distro glibc backports.. don't ask), I
noticed that glibc has no provision for dealing with kernel generated
ucontext's in its {get,set,swap}_context functions...

(It says so explicitly in the code unless I misunderstood).

So one thing we did to "solve" this on ppc64 a while ago was to create
a swapcontext syscall which can operate as all 3 operations (you can
have NULL arguments), which also handles the sigprocmask (bonus:
atomically with the context get/set from a userspace perspective).

Would it make sense to do something similar on aarch64 ? (And have
glibc then exploit it).

The hard-to-solve thing is the case where the SVE context spills
outside of the ucontext itself, in the extra room on the stack, since
programs that "now" about ucontext will not have allocated space for
that, so that's more/less a lost cause already.

Cheers,
Ben.



_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: ucontext, kernel vs. userspace (glibc)

[Catalin Marinas]

Hi Ben,

On Mon, Aug 30, 2021 at 08:40:03PM +1000, Benjamin Herrenschmidt wrote:
> So I'm discovering arm64 intricacies and today, as I was looking at SVE
> support (in the context of distro glibc backports.. don't ask), I
> noticed that glibc has no provision for dealing with kernel generated
> ucontext's in its {get,set,swap}_context functions...
> 
> (It says so explicitly in the code unless I misunderstood).
> 
> So one thing we did to "solve" this on ppc64 a while ago was to create
> a swapcontext syscall which can operate as all 3 operations (you can
> have NULL arguments), which also handles the sigprocmask (bonus:
> atomically with the context get/set from a userspace perspective).
> 
> Would it make sense to do something similar on aarch64 ? (And have
> glibc then exploit it).
> 
> The hard-to-solve thing is the case where the SVE context spills
> outside of the ucontext itself, in the extra room on the stack, since
> programs that "now" about ucontext will not have allocated space for
> that, so that's more/less a lost cause already.

I haven't fully parsed your email yet but adding a Mark B for SVE and
Szabolcs for glibc (and removing Dave who left Arm recently).

-- 
Catalin

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: ucontext, kernel vs. userspace (glibc)

[Mark Brown]

[-- Attachment #1.1: Type: text/plain, Size: 1448 bytes --]

On Mon, Aug 30, 2021 at 08:40:03PM +1000, Benjamin Herrenschmidt wrote:

> So I'm discovering arm64 intricacies and today, as I was looking at SVE
> support (in the context of distro glibc backports.. don't ask), I
> noticed that glibc has no provision for dealing with kernel generated
> ucontext's in its {get,set,swap}_context functions...

> (It says so explicitly in the code unless I misunderstood).

> So one thing we did to "solve" this on ppc64 a while ago was to create
> a swapcontext syscall which can operate as all 3 operations (you can
> have NULL arguments), which also handles the sigprocmask (bonus:
> atomically with the context get/set from a userspace perspective).

> Would it make sense to do something similar on aarch64 ? (And have
> glibc then exploit it).

I think the usefulness of such an interface is mainly a question for
userspace - I don't immediately see any issue with implementing it if
it's useful to people.

> The hard-to-solve thing is the case where the SVE context spills
> outside of the ucontext itself, in the extra room on the stack, since
> programs that "now" about ucontext will not have allocated space for
> that, so that's more/less a lost cause already.

You can figure out the maximum possible size for a context so it would
be possible to define a mechanism for pointing to extra data I guess but
yeah, it's going to be a problem when we start seeing systems with large
enough register state.

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[-- Attachment #2: Type: text/plain, Size: 176 bytes --]

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: ucontext, kernel vs. userspace (glibc)

[Benjamin Herrenschmidt]

On Thu, 2021-09-02 at 13:42 +0100, Mark Brown wrote:
> On Mon, Aug 30, 2021 at 08:40:03PM +1000, Benjamin Herrenschmidt wrote:
> 
> > So I'm discovering arm64 intricacies and today, as I was looking at SVE
> > support (in the context of distro glibc backports.. don't ask), I
> > noticed that glibc has no provision for dealing with kernel generated
> > ucontext's in its {get,set,swap}_context functions...
> > (It says so explicitly in the code unless I misunderstood).
> > So one thing we did to "solve" this on ppc64 a while ago was to create
> > a swapcontext syscall which can operate as all 3 operations (you can
> > have NULL arguments), which also handles the sigprocmask (bonus:
> > atomically with the context get/set from a userspace perspective).
> > Would it make sense to do something similar on aarch64 ? (And have
> > glibc then exploit it).
> 
> I think the usefulness of such an interface is mainly a question for
> userspace - I don't immediately see any issue with implementing it if
> it's useful to people.

Well, the problem as far as I can tell is that the glibc implementation
of these today. They support "FPSIMD" but that's about it (so no SVE or
anything else) along with a comment:

	/* Check for FP SIMD context.  We don't support restoring
	   contexts created by the kernel, so this context must have
	   been created by getcontext.  Hence we can rely on the
	   first extension block being the FP SIMD context.  */

That said, a bit of reading around seems to indicate that the
expecation of being able to setcontext() back to a signal handler
generated context has been deprecated by the standard and broken on x86
for a while in Linux, so I suppose that is less of an issue.

That said, there is still some advantage in letting the kernel
implement these as it would allow the kernel to support various
"extensions" such as SVE (as long as there is room) transparently
without having to change glibc.

In fact, isn't it possible for glibc to define its own ucontext
structure for applications to use that can potentially have a larger
reserved area ? By passing that size to the syscall, you can
essentially get userspace ready for future extensions... within limits.

> > The hard-to-solve thing is the case where the SVE context spills
> > outside of the ucontext itself, in the extra room on the stack, since
> > programs that "now" about ucontext will not have allocated space for
> > that, so that's more/less a lost cause already.
> 
> You can figure out the maximum possible size for a context so it would
> be possible to define a mechanism for pointing to extra data I guess but
> yeah, it's going to be a problem when we start seeing systems with large
> enough register state.

Extra data for userspace generated ucontext's isn't going to fly much,
there's really no "place" to put it (those things can be part of
structures etc...) and no "hook" to allocate/free sub structures.

So you need whatever struct ucontext is used in userspace to be big
enough.

That said, I think the current one might be enough for sve512 (I need
to check) and we could have glibc define something much bigger (16KB ?)
without much damage I suspect.

Nagyu ? What do you think ?

Cheers,
Ben.



_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: ucontext, kernel vs. userspace (glibc)

[Szabolcs Nagy]

The 09/03/2021 17:14, Benjamin Herrenschmidt wrote:
> On Thu, 2021-09-02 at 13:42 +0100, Mark Brown wrote:
> > On Mon, Aug 30, 2021 at 08:40:03PM +1000, Benjamin Herrenschmidt wrote:
> > 
> > > So I'm discovering arm64 intricacies and today, as I was looking at SVE
> > > support (in the context of distro glibc backports.. don't ask), I
> > > noticed that glibc has no provision for dealing with kernel generated
> > > ucontext's in its {get,set,swap}_context functions...
> > > (It says so explicitly in the code unless I misunderstood).
> > > So one thing we did to "solve" this on ppc64 a while ago was to create
> > > a swapcontext syscall which can operate as all 3 operations (you can
> > > have NULL arguments), which also handles the sigprocmask (bonus:
> > > atomically with the context get/set from a userspace perspective).
> > > Would it make sense to do something similar on aarch64 ? (And have
> > > glibc then exploit it).
> > 
> > I think the usefulness of such an interface is mainly a question for
> > userspace - I don't immediately see any issue with implementing it if
> > it's useful to people.
> 
> Well, the problem as far as I can tell is that the glibc implementation
> of these today. They support "FPSIMD" but that's about it (so no SVE or
> anything else) along with a comment:
> 
> 	/* Check for FP SIMD context.  We don't support restoring
> 	   contexts created by the kernel, so this context must have
> 	   been created by getcontext.  Hence we can rely on the
> 	   first extension block being the FP SIMD context.  */
> 
> That said, a bit of reading around seems to indicate that the
> expecation of being able to setcontext() back to a signal handler
> generated context has been deprecated by the standard and broken on x86
> for a while in Linux, so I suppose that is less of an issue.

yes, setcontext is not expected to work with kernel
context.

> 
> That said, there is still some advantage in letting the kernel
> implement these as it would allow the kernel to support various
> "extensions" such as SVE (as long as there is room) transparently
> without having to change glibc.
> 
> In fact, isn't it possible for glibc to define its own ucontext
> structure for applications to use that can potentially have a larger
> reserved area ? By passing that size to the syscall, you can
> essentially get userspace ready for future extensions... within limits.

i think this can be a discussion for libc-alpha, but
i don't think there is interest in using the libc
context functions with kernel signal contexts,
that turned out to be problematic historically.
but if there is a use-case that can be discussed.

> 
> > > The hard-to-solve thing is the case where the SVE context spills
> > > outside of the ucontext itself, in the extra room on the stack, since
> > > programs that "now" about ucontext will not have allocated space for
> > > that, so that's more/less a lost cause already.
> > 
> > You can figure out the maximum possible size for a context so it would
> > be possible to define a mechanism for pointing to extra data I guess but
> > yeah, it's going to be a problem when we start seeing systems with large
> > enough register state.
> 
> Extra data for userspace generated ucontext's isn't going to fly much,
> there's really no "place" to put it (those things can be part of
> structures etc...) and no "hook" to allocate/free sub structures.
> 
> So you need whatever struct ucontext is used in userspace to be big
> enough.
> 
> That said, I think the current one might be enough for sve512 (I need
> to check) and we could have glibc define something much bigger (16KB ?)
> without much damage I suspect.
> 
> Nagyu ? What do you think ?

i think we only want to change set/get/swapcontext
if there is a use-case for this. currently only
a small bit of fp state has to be saved/restored.
and there can be security concerns since we have
features like bti that limits where one can jump
(arbitrary pc in setcontext does not work).

> 
> Cheers,
> Ben.
> 

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: ucontext, kernel vs. userspace (glibc)

[Mark Brown]

[-- Attachment #1.1: Type: text/plain, Size: 2347 bytes --]

On Fri, Sep 03, 2021 at 05:14:28PM +1000, Benjamin Herrenschmidt wrote:

> Well, the problem as far as I can tell is that the glibc implementation
> of these today. They support "FPSIMD" but that's about it (so no SVE or
> anything else) along with a comment:

> 	/* Check for FP SIMD context.  We don't support restoring
> 	   contexts created by the kernel, so this context must have
> 	   been created by getcontext.  Hence we can rely on the
> 	   first extension block being the FP SIMD context.  */

The kernel does generate a FPSIMD context in addition to any SVE context
for compatibility, though that doesn't mean you can actually fully
restore them successfully with glibc.  For contexts generated by glibc
note that unless a function signature involves a SVE type the SVE
register contents are caller saved:

   https://github.com/ARM-software/abi-aa/blob/main/aapcs64/aapcs64.rst#id48

so glibc will be fine just ignoring SVE when it generates contexts given
that it'll be doing that inside a non-SVE function call.

Like Szabolcs says BTI is going to cause issues restoring kernel
generated contexts even without extensions like SVE adding extra
register state.

> > You can figure out the maximum possible size for a context so it would
> > be possible to define a mechanism for pointing to extra data I guess but
> > yeah, it's going to be a problem when we start seeing systems with large
> > enough register state.

> Extra data for userspace generated ucontext's isn't going to fly much,
> there's really no "place" to put it (those things can be part of
> structures etc...) and no "hook" to allocate/free sub structures.

> So you need whatever struct ucontext is used in userspace to be big
> enough.

Indeed, it'd have to be joined up with an increase of the userspace
ucontext.

> That said, I think the current one might be enough for sve512 (I need
> to check) and we could have glibc define something much bigger (16KB ?)
> without much damage I suspect.

Yes, the current context should be big enough for 512 bit SVE - that's
why the kernel clamps the default SVE vector length to 512 bits, so we
sidestep these issues by default even if the user happens to have a
system that can do larger vector lengths.  Not an immediate issue with
actual hardware in any case, though that code currently kicks in on
qemu's cpu=max.

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[-- Attachment #2: Type: text/plain, Size: 176 bytes --]

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel