Re: [PATCH] prctl.2: Add description of Intel MPX calls

Re: [PATCH] prctl.2: Add description of Intel MPX calls

[Michael Kerrisk (man-pages)]

On 01/09/2015 07:36 PM, Dave Hansen wrote:
> From: Dave Hansen <dave.hansen@intel.com>
> 
> The 3.19 kernel will have support for Intel MPX, including a pair
> of new prctl() calls for enabling and disabling the kernel's
> management of the "bounds tables".  Add some descriptions of the
> interface.
> 
> The kernel patches were written by myself and another Intel
> developer.

Thanks for the revision, Dave.

Applied (with a few light edits).

Cheers,

Michael


> Signed-off-by: Dave Hansen <dave.hansen@intel.com>
> Cc: Qiaowei Ren <qiaowei.ren@intel.com>
> Cc: linux-man@vger.kernel.org
> Cc: linux-kernel@vger.kernel.org
> ---
>  man2/prctl.2 | 63 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 63 insertions(+)
> 
> diff --git a/man2/prctl.2 b/man2/prctl.2
> index 4efabcf..dd0448e 100644
> --- a/man2/prctl.2
> +++ b/man2/prctl.2
> @@ -47,6 +47,7 @@
>  .\"                             PR_GET_TIMERSLACK
>  .\" 2013-01-10 Kees Cook, document PR_SET_PTRACER
>  .\" 2012-02-04 Michael kerrisk, document PR_{SET,GET}_CHILD_SUBREAPER
> +.\" 2014-11-10 Dave Hansen, document PR_MPX_{EN,DIS}ABLE_MANAGEMENT
>  .\"
>  .\"
>  .TH PRCTL 2 2014-04-14 "Linux" "Linux Programmer's Manual"
> @@ -799,6 +800,59 @@ in a process life time.
>  Any further attempts will be rejected.
>  This should help system administrators monitor unusual
>  symbolic-link transitions over all processes running on a system.
> +.TP
> +.BR PR_MPX_ENABLE_MANAGEMENT / PR_MPX_DISABLE_MANAGEMENT " (since Linux 3.19) "
> +.\" commit fe3d197f84319d3bce379a9c0dc17b1f48ad358c
> +.\" See also http://lwn.net/Articles/582712/
> +.\" See also https://gcc.gnu.org/wiki/Intel%20MPX%20support%20in%20the%20GCC%20compiler
> +
> +Control the kernel's management of Memory Protection eXtensions (MPX)
> +bounds tables.
> +
> +MPX is a hardware-assisted mechanism for performing bounds checking on
> +pointers.  It consists of a set of registers storing bounds information
> +and a set of special instruction prefixes that tell the CPU on which
> +instructions it should do bounds enforcement.   There is a limited
> +number of these registers and when there are more pointers than
> +registers, their contents must be "spilled" in to a set of tables.
> +These tables are called "bounds tables" and these prctl()s control
> +whether the kernel manages their allocation and freeing.
> +
> +When management is enabled, the kernel will take over allocation
> +and freeing of the bounds tables.  It does this by trapping the #BR
> +exceptions that result at first use of missing bounds tables and
> +instead of delivering the exception to userspace, it allocates the
> +table and populates the bounds directory with the location of the
> +new table.  For freeing, the kernel checks to see if bounds tables are
> +present for memory which is not allocated, and frees them if so.
> +
> +Before enabling MPX management using PR_MPX_ENABLE_MANAGEMENT,
> +the application must first have allocated a user-space buffer for
> +the bounds directory and placed the location of that directory in
> +the bndcfgu register.
> +
> +These calls will fail if the CPU or kernel does not support MPX.
> +Kernel support for MPX is enabled via the
> +.BR CONFIG_X86_INTEL_MPX
> +configuration option.
> +You can check whether the CPU supports MPX by looking for the 'mpx'
> +CPUID bit, like with the following command:
> +
> +	cat /proc/cpuinfo | grep ' mpx '
> +
> +A thread may not switch in or out of long (64-bit) mode while MPX is
> +enabled.
> +
> +All threads in a process are affected by these calls.
> +
> +The child of a fork(2) will inherit the state of MPX management.
> +
> +During an execve(2) call, MPX management will be reset to a state
> +as if PR_MPX_DISABLE_MANAGEMENT had been called.
> +
> +For further information on Intel MPX, see the kernel source file
> +.IR Documentation/x86/intel_mpx.txt .
> +
>  .RE
>  .\"
>  .SH RETURN VALUE
> @@ -1062,6 +1116,15 @@ is not valid.
>  .\" is
>  .\" .BR PR_SET_SECCOMP ,
>  .\" and secure computing mode is already 1.
> +.TP
> +.B ENXIO
> +.I option
> +is PR_MPX_ENABLE_MANAGEMENT and the kernel was unable to enable
> +management.  Ensure the kernel and processor have MPX support.
> +.TP
> +.B ENXIO
> +.I option
> +is PR_MPX_DISABLE_MANAGEMENT and the kernel or CPU does not support MPX.
>  .SH VERSIONS
>  The
>  .BR prctl ()
> 


-- 
Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
Linux/UNIX System Programming Training: http://man7.org/training/

Re: [PATCH] prctl.2: Add description of Intel MPX calls

[Dave Hansen]

On 01/08/2015 02:27 AM, Michael Kerrisk (man-pages) wrote:
> Thanks for the patch. Quite a number of queries below. Could you
> address those and resubmit?

Absolutely, thanks for the detailed review!

I'll address most of your comments in the new patch.

>> +The application must have allocated virtual space for the bounds
>> +directory and placed its location in bndcfgu before enabling management.
> 
> s/bndcfgu/the bndcfgu register/
> 
> So, let me make sure I understood the previous sentence correctly.
> Is this a good formulation:
> 
>     Before enabling MPX management using PR_MPX_ENABLE_MANAGEMENT, 
>     the application must first have allocated a user-space buffer for
>     the bounds directory and placed the location of that directory in
>     the bndcfgu register.
> ?
> 
> If so, I think the patch could be reworded closer to that text, since
> it makes things a little clearer.

That looks good to me.

> By the way, what size should allocated for the buffer? And what can
> wrong if the allocated buffer is too small?

4MB on 32-bit and 2GB on 64-bit.

If the buffer is too small, the hardware might attempt to read a
non-allocated memory area (and SEGV).  The kernel would notice that the
buffer is too small probably during the first bounds fault touching the
unallocated area of the table.

>> +Once management is enabled, the bounds directory may not be moved.
> 
> A few things are not so clear to me, even after reading Jon's article
> and Documentation/x86/intel_mpx.txt. In particular, who has responsibility
> for populating the bounds directory, and what is its format? Is this 
> all invisible to the application? (Because I'm not quite clear on these
> points, I'm not sure if anything should be said in the man page.)

This is all best spelled out in the documentation available here:

	https://software.intel.com/en-us/intel-isa-extensions

The compiler generates code which loads the bounds values in to the
bounds registers.  There are special instructions that "spill" the
bounds registers in to the tables.

So, the instructions in the application do it, but it's transparent and
doesn't require any source modifications.

>>  .P
>>  The following options are available since Linux 3.5.
>>  .\" commit fe8c7f5cbf91124987106faa3bdf0c8b955c4cf7
> 
> Could you add some text to the patch to explain the ENXIO error?

Sure.

> What are the semantics of the MPX state with respect to the
> child of a fork(2)? (Probably, the man page needs to detail this.)

The tables are inherited just as any other memory contents are.  The
register contents are inherited just as any other register would be.

> What happens to the MPX state during an execve(2)? (Probably, 
> the man page needs to detail this.)

The tables go away at exec just like any other memory.  The BNDCFGU
register (the one holding the location of the tables) is zeroed.

> Is there a way to discover the current state (enabled/disabled)
> of MPX management? (It appears not, but I would like to confirm.)

There is not.  It's not something I really expect an app to want to turn
on and off or need to query.  It might not be a bad idea to expose it in
/proc somehow.

Also, FWIW, you could always attach with a debugger and check out the
BNDCFGU contents if you _practically_ wanted to check for MPX being
used.  It wouldn't tell you if bounds management is enabled, but it
would be a really good indicator.

> I see that the prctl() operations fail to check that unused arguments
> are zero. See my patch sent in a separate mail to LKML et al.

OK, fair enough.

Re: [PATCH] prctl.2: Add description of Intel MPX calls

[Michael Kerrisk (man-pages)]

[CC += Qiaowei Ren, in case they might comment]

Hello Dave,

Thanks for the patch. Quite a number of queries below. Could you
address those and resubmit?

On 12/15/2014 11:52 PM, Dave Hansen wrote:
> From: Dave Hansen <dave.hansen@intel.com>
> 
> (apologies if this is a duplicate mail, I didn't see my first one
>  hit either of the lists, so I assume Intel's mail relays ate it)

(Actually, both mails did arrive.)

> The 3.19 kernel will have support for Intel MPX, including a pair
> of new prctl() calls for enabling and disabling the kernel's
> management of the "bounds tables".  Add some descriptions of the
> interface.
> 
> The kernel patches were written by myself and another Intel
> developer.

(Thanks -- getting that piece of info is always helpful when
I consider how carefully I need to review a patch.)

> Signed-off-by: Dave Hansen <dave.hansen@intel.com>
> Cc: linux-man@vger.kernel.org
> Cc: linux-kernel@vger.kernel.org
> ---
>  man2/prctl.2 | 16 ++++++++++++++++
>  1 file changed, 16 insertions(+)
> 
> diff --git a/man2/prctl.2 b/man2/prctl.2
> index 4efabcf..6555d7a 100644
> --- a/man2/prctl.2
> +++ b/man2/prctl.2
> @@ -47,6 +47,7 @@
>  .\"                             PR_GET_TIMERSLACK
>  .\" 2013-01-10 Kees Cook, document PR_SET_PTRACER
>  .\" 2012-02-04 Michael kerrisk, document PR_{SET,GET}_CHILD_SUBREAPER
> +.\" 2014-11-10 Dave Hansen, Document PR_MPX_{EN,DIS}ABLE_MANAGEMENT
>  .\"
>  .\"
>  .TH PRCTL 2 2014-04-14 "Linux" "Linux Programmer's Manual"
> @@ -737,6 +738,21 @@ value.

It looks like this patch is misplaced in the source. Probably best to
place the new text just above the RETURN VALUE section.

>  The requirements for the address are the same as for the
>  .BR PR_SET_MM_START_BRK
>  option.
> +.TP
> +.BR PR_MPX_ENABLE_MANAGEMENT / PR_MPX_DISABLE_MANAGEMENT

Make this:

.BR PR_MPX_ENABLE_MANAGEMENT / PR_MPX_DISABLE_MANAGEMENT " (since Linux 3.19)"

And then just after please add the following source comments
(for handy reference on future page edits):

.\" commit fe3d197f84319d3bce379a9c0dc17b1f48ad358c
.\" See also http://lwn.net/Articles/582712/
.\" See also https://gcc.gnu.org/wiki/Intel%20MPX%20support%20in%20the%20GCC%20compiler

> +
> +Control the kernel's management of Intel Memory Protection eXtensions

s/Memory Protection eXtensions/Memory Protection eXtensions (MPX)/

> +tables.  When management is enabled, the kernel will take over allocation
> +and freeing of MPX bounds tables.

In between or after the two preceding sentences, I think it would be good 
to some text explaining why MPX is useful. Something along the lines of 
this text from Jon Corbet's article at http://lwn.net/Articles/582712/,
though you may want to cut down, update, or improve the text.

[[
MPX is a hardware-assisted mechanism for performing bounds checking 
on pointer accesses. The hardware, following direction from software, 
maintains a table of pointers in use and the range of accessible 
memory (the "bounds") associated with each. Whenever a pointer is 
dereferenced, special instructions can be used to ensure that the 
program is accessing memory within the range specified for that 
particular pointer.

When a bounds violation is detected, the processor will trap into 
the kernel. The kernel, in turn, will turn the trap into a SIGSEGV 
signal to be delivered to the application, similar to other types 
of memory access errors. Applications that look at the siginfo 
structure passed to the signal handler from the kernel will be able 
to recognize a bounds error by checking the si_code field for the new 
SEGV_BNDERR value. The offending address will be stored in si_addr, 
while the bounds in effect at the time of the trap will be stored 
in si_lower and si_upper.  
]]

> +
> +The application must have allocated virtual space for the bounds
> +directory and placed its location in bndcfgu before enabling management.

s/bndcfgu/the bndcfgu register/

So, let me make sure I understood the previous sentence correctly.
Is this a good formulation:

    Before enabling MPX management using PR_MPX_ENABLE_MANAGEMENT, 
    the application must first have allocated a user-space buffer for
    the bounds directory and placed the location of that directory in
    the bndcfgu register.
?

If so, I think the patch could be reworded closer to that text, since
it makes things a little clearer.

By the way, what size should allocated for the buffer? And what can
wrong if the allocated buffer is too small?

> +Once management is enabled, the bounds directory may not be moved.

A few things are not so clear to me, even after reading Jon's article
and Documentation/x86/intel_mpx.txt. In particular, who has responsibility
for populating the bounds directory, and what is its format? Is this 
all invisible to the application? (Because I'm not quite clear on these
points, I'm not sure if anything should be said in the man page.)

> +
> +These calls will fail if MPX support is not present in the CPU or the

I suggest:
s/if MPX support is not present in the CPU/if the CPU does not support MPX/

> +kernel does not had support for MPX enabled.

s/had/have/

And add the following:

Kernel support for MPX is enabled via the
.BR CONFIG_X86_INTEL_MPX
configuration option.
You can check whether the CPU supports MPX using the following command:

    cat /proc/cpuinfo | grep ' mpx '

> +
> +All threads in a process are affected by these calls.

Please add here:

For further information on Intel MPX, see the kernel source file
.IR Documentation/x86/intel_mpx.txt .

>  .P
>  The following options are available since Linux 3.5.
>  .\" commit fe8c7f5cbf91124987106faa3bdf0c8b955c4cf7

Could you add some text to the patch to explain the ENXIO error?

What are the semantics of the MPX state with respect to the
child of a fork(2)? (Probably, the man page needs to detail this.)

What happens to the MPX state during an execve(2)? (Probably, 
the man page needs to detail this.)

Is there a way to discover the current state (enabled/disabled)
of MPX management? (It appears not, but I would like to confirm.)

I see that the prctl() operations fail to check that unused arguments
are zero. See my patch sent in a separate mail to LKML et al.

Cheers,

Michael


-- 
Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
Linux/UNIX System Programming Training: http://man7.org/training/