um: PTRACE_SETREGSET failure with XSTATE on Kabylake CPU

um: PTRACE_SETREGSET failure with XSTATE on Kabylake CPU

[Thomas Meyer]

Hi,

I finally did figure out where in the host kernel the ptrace syscall
fails with -EFAULT.

In arch/x86/kernel/fpu/regset.c:130:

114 int xstateregs_set(struct task_struct *target, const struct user_regset *regset,
115                   unsigned int pos, unsigned int count,
116                   const void *kbuf, const void __user *ubuf)
117 {
118         struct fpu *fpu = &target->thread.fpu;
119         struct xregs_state *xsave;
120         int ret;
121 
122         if (!boot_cpu_has(X86_FEATURE_XSAVE))
123                 return -ENODEV;
124 
125         pr_info("in xstateregs_set");
126 
127         /*
128          * A whole standard-format XSAVE buffer is needed:
129          */
130         if ((pos != 0) || (count < fpu_user_xstate_size)) {
131                 pr_info("EFAULT from xstateregs_set");
132->               pr_info("pos = %i, count = %i, fpu_user_xstate_size= %i\n", pos, count, fpu_user_xstate_size);
133                 return -EFAULT;
134         }

Sadly I had to fallback to debugging by printk because kgdb/qemu
gdbstub, all didn't work for some unknown reason :-(

output is:
[   69.598349] EFAULT from xstateregs_set
[   69.598350] pos = 0, count = 832, fpu_user_xstate_size= 1088

calling code is in arch/x86/um/os-Linux/registers.c:

 49 int restore_fp_registers(int pid, unsigned long *fp_regs)
 50 {
 51         struct iovec iov;
 52 
 53         if (have_xstate_support) {
 54                 iov.iov_base = fp_regs;
 55                 iov.iov_len = sizeof(struct _xstate);
 56                 if (ptrace(PTRACE_SETREGSET, pid, NT_X86_XSTATE, &iov) < 0)
 57  ->                     return -errno;
 58                 return 0;
 59         } else {
 60                 return restore_i387_registers(pid, fp_regs);
 61         }
 62 }

it looks like _xstate is too short for above operation, I wonder why
PTRACE_GETREGSET works without a warning of too short size.

with kind regards
thomas

Re: um: PTRACE_SETREGSET failure with XSTATE on Kabylake CPU

[Richard Weinberger]

Thomas,

Am 20.06.2017 um 03:56 schrieb Thomas Meyer:
> Hi,
> 
> I finally did figure out where in the host kernel the ptrace syscall
> fails with -EFAULT.

Nice! Thanks a lot for digging into this. I still had no chance to setup
Ipv6 to connect to your host and figure myself. ;-\

> In arch/x86/kernel/fpu/regset.c:130:
> 
> 114 int xstateregs_set(struct task_struct *target, const struct user_regset *regset,
> 115                   unsigned int pos, unsigned int count,
> 116                   const void *kbuf, const void __user *ubuf)
> 117 {
> 118         struct fpu *fpu = &target->thread.fpu;
> 119         struct xregs_state *xsave;
> 120         int ret;
> 121 
> 122         if (!boot_cpu_has(X86_FEATURE_XSAVE))
> 123                 return -ENODEV;
> 124 
> 125         pr_info("in xstateregs_set");
> 126 
> 127         /*
> 128          * A whole standard-format XSAVE buffer is needed:
> 129          */
> 130         if ((pos != 0) || (count < fpu_user_xstate_size)) {
> 131                 pr_info("EFAULT from xstateregs_set");
> 132->               pr_info("pos = %i, count = %i, fpu_user_xstate_size= %i\n", pos, count, fpu_user_xstate_size);
> 133                 return -EFAULT;
> 134         }
> 
> Sadly I had to fallback to debugging by printk because kgdb/qemu
> gdbstub, all didn't work for some unknown reason :-(

As always. printk is best debugger ever. ;-)

> output is:
> [   69.598349] EFAULT from xstateregs_set
> [   69.598350] pos = 0, count = 832, fpu_user_xstate_size= 1088
> 
> calling code is in arch/x86/um/os-Linux/registers.c:
> 
>  49 int restore_fp_registers(int pid, unsigned long *fp_regs)
>  50 {
>  51         struct iovec iov;
>  52 
>  53         if (have_xstate_support) {
>  54                 iov.iov_base = fp_regs;
>  55                 iov.iov_len = sizeof(struct _xstate);
>  56                 if (ptrace(PTRACE_SETREGSET, pid, NT_X86_XSTATE, &iov) < 0)
>  57  ->                     return -errno;
>  58                 return 0;
>  59         } else {
>  60                 return restore_i387_registers(pid, fp_regs);
>  61         }
>  62 }
> 
> it looks like _xstate is too short for above operation, I wonder why
> PTRACE_GETREGSET works without a warning of too short size.

Does PTRACE_GETREGSET return a size? Maybe we have to take this into account.
It could be that your host CPU has a smaller set.
Also check whether PTRACE_SETREGSET always fails.

Thanks,
//richard

Re: um: PTRACE_SETREGSET failure with XSTATE on Kabylake CPU

[Thomas Meyer]

Am Dienstag, den 20.06.2017, 08:58 +0200 schrieb Richard Weinberger:
> Thomas,
> 
> Am 20.06.2017 um 03:56 schrieb Thomas Meyer:
> > Hi,
> > 
> > I finally did figure out where in the host kernel the ptrace
> > syscall
> > fails with -EFAULT.
> 
> Nice! Thanks a lot for digging into this. I still had no chance to
> setup
> Ipv6 to connect to your host and figure myself. ;-\
> 
> > In arch/x86/kernel/fpu/regset.c:130:
> > 
> > 114 int xstateregs_set(struct task_struct *target, const struct
> > user_regset *regset,
> > 115                   unsigned int pos, unsigned int count,
> > 116                   const void *kbuf, const void __user *ubuf)
> > 117 {
> > 118         struct fpu *fpu = &target->thread.fpu;
> > 119         struct xregs_state *xsave;
> > 120         int ret;
> > 121 
> > 122         if (!boot_cpu_has(X86_FEATURE_XSAVE))
> > 123                 return -ENODEV;
> > 124 
> > 125         pr_info("in xstateregs_set");
> > 126 
> > 127         /*
> > 128          * A whole standard-format XSAVE buffer is needed:
> > 129          */
> > 130         if ((pos != 0) || (count < fpu_user_xstate_size)) {
> > 131                 pr_info("EFAULT from xstateregs_set");
> > 132->               pr_info("pos = %i, count = %i,
> > fpu_user_xstate_size= %i\n", pos, count, fpu_user_xstate_size);
> > 133                 return -EFAULT;
> > 134         }
> > 
> > Sadly I had to fallback to debugging by printk because kgdb/qemu
> > gdbstub, all didn't work for some unknown reason :-(
> 
> As always. printk is best debugger ever. ;-)
> 
> > output is:
> > [   69.598349] EFAULT from xstateregs_set
> > [   69.598350] pos = 0, count = 832, fpu_user_xstate_size= 1088
> > 
> > calling code is in arch/x86/um/os-Linux/registers.c:
> > 
> >  49 int restore_fp_registers(int pid, unsigned long *fp_regs)
> >  50 {
> >  51         struct iovec iov;
> >  52 
> >  53         if (have_xstate_support) {
> >  54                 iov.iov_base = fp_regs;
> >  55                 iov.iov_len = sizeof(struct _xstate);
> >  56                 if (ptrace(PTRACE_SETREGSET, pid,
> > NT_X86_XSTATE, &iov) < 0)
> >  57  ->                     return -errno;
> >  58                 return 0;
> >  59         } else {
> >  60                 return restore_i387_registers(pid, fp_regs);
> >  61         }
> >  62 }
> > 
> > it looks like _xstate is too short for above operation, I wonder
> > why
> > PTRACE_GETREGSET works without a warning of too short size.
> 
> Does PTRACE_GETREGSET return a size?

Yes, it returns 832. the size of struct _xstate.

> Maybe we have to take this into account.
> It could be that your host CPU has a smaller set.
> Also check whether PTRACE_SETREGSET always fails.

In UML the first userspace ptrace always fails, so init get's killed.

The check "count < fpu_user_xstate_size" was introduced by commit:

commit 91c3dba7dbc199191272f4a9863f86ea3bfd679f
Author: Yu-cheng Yu <yu-cheng.yu@intel.com>
Date:   Fri Jun 17 13:07:17 2016 -0700

    x86/fpu/xstate: Fix PTRACE frames for XSAVES
    
    XSAVES uses compacted format and is a kernel instruction. The kernel
    should use standard-format, non-supervisor state data for PTRACE.

So to summarize:

- PTRACE_GETREGSET with NT_X86_XSTATE gets 832 and return 832, with no
error.

- PTRACE_SETREGSET get 832 (sizeof struct _xstate) but wants at least
1088, otherwise it will fail with -EFAULT (why not -EINVAL?)

Ideas?

> 
> Thanks,
> //richard

Re: um: PTRACE_SETREGSET failure with XSTATE on Kabylake CPU

[Richard Weinberger]

[adding x86 folks]

Am 20.06.2017 um 10:49 schrieb Thomas Meyer:
> Am Dienstag, den 20.06.2017, 08:58 +0200 schrieb Richard Weinberger:
>> Thomas,
>>
>> Am 20.06.2017 um 03:56 schrieb Thomas Meyer:
>>> Hi,
>>>
>>> I finally did figure out where in the host kernel the ptrace
>>> syscall
>>> fails with -EFAULT.
>>
>> Nice! Thanks a lot for digging into this. I still had no chance to
>> setup
>> Ipv6 to connect to your host and figure myself. ;-\
>>
>>> In arch/x86/kernel/fpu/regset.c:130:
>>>
>>> 114 int xstateregs_set(struct task_struct *target, const struct
>>> user_regset *regset,
>>> 115                   unsigned int pos, unsigned int count,
>>> 116                   const void *kbuf, const void __user *ubuf)
>>> 117 {
>>> 118         struct fpu *fpu = &target->thread.fpu;
>>> 119         struct xregs_state *xsave;
>>> 120         int ret;
>>> 121 
>>> 122         if (!boot_cpu_has(X86_FEATURE_XSAVE))
>>> 123                 return -ENODEV;
>>> 124 
>>> 125         pr_info("in xstateregs_set");
>>> 126 
>>> 127         /*
>>> 128          * A whole standard-format XSAVE buffer is needed:
>>> 129          */
>>> 130         if ((pos != 0) || (count < fpu_user_xstate_size)) {
>>> 131                 pr_info("EFAULT from xstateregs_set");
>>> 132->               pr_info("pos = %i, count = %i,
>>> fpu_user_xstate_size= %i\n", pos, count, fpu_user_xstate_size);
>>> 133                 return -EFAULT;
>>> 134         }
>>>
>>> Sadly I had to fallback to debugging by printk because kgdb/qemu
>>> gdbstub, all didn't work for some unknown reason :-(
>>
>> As always. printk is best debugger ever. ;-)
>>
>>> output is:
>>> [   69.598349] EFAULT from xstateregs_set
>>> [   69.598350] pos = 0, count = 832, fpu_user_xstate_size= 1088
>>>
>>> calling code is in arch/x86/um/os-Linux/registers.c:
>>>
>>>  49 int restore_fp_registers(int pid, unsigned long *fp_regs)
>>>  50 {
>>>  51         struct iovec iov;
>>>  52 
>>>  53         if (have_xstate_support) {
>>>  54                 iov.iov_base = fp_regs;
>>>  55                 iov.iov_len = sizeof(struct _xstate);
>>>  56                 if (ptrace(PTRACE_SETREGSET, pid,
>>> NT_X86_XSTATE, &iov) < 0)
>>>  57  ->                     return -errno;
>>>  58                 return 0;
>>>  59         } else {
>>>  60                 return restore_i387_registers(pid, fp_regs);
>>>  61         }
>>>  62 }
>>>
>>> it looks like _xstate is too short for above operation, I wonder
>>> why
>>> PTRACE_GETREGSET works without a warning of too short size.
>>
>> Does PTRACE_GETREGSET return a size?
> 
> Yes, it returns 832. the size of struct _xstate.
> 
>> Maybe we have to take this into account.
>> It could be that your host CPU has a smaller set.
>> Also check whether PTRACE_SETREGSET always fails.
> 
> In UML the first userspace ptrace always fails, so init get's killed.
> 
> The check "count < fpu_user_xstate_size" was introduced by commit:
> 
> commit 91c3dba7dbc199191272f4a9863f86ea3bfd679f
> Author: Yu-cheng Yu <yu-cheng.yu@intel.com>
> Date:   Fri Jun 17 13:07:17 2016 -0700
> 
>     x86/fpu/xstate: Fix PTRACE frames for XSAVES
>     
>     XSAVES uses compacted format and is a kernel instruction. The kernel
>     should use standard-format, non-supervisor state data for PTRACE.
> 
> So to summarize:
> 
> - PTRACE_GETREGSET with NT_X86_XSTATE gets 832 and return 832, with no
> error.
> 
> - PTRACE_SETREGSET get 832 (sizeof struct _xstate) but wants at least
> 1088, otherwise it will fail with -EFAULT (why not -EINVAL?)
> 
> Ideas?
> 
>>
>> Thanks,
>> //richard

Re: um: PTRACE_SETREGSET failure with XSTATE on Kabylake CPU

[Yu-cheng Yu]

On Tue, 2017-06-20 at 11:05 +0200, Richard Weinberger wrote:
> [adding x86 folks]
> 
> Am 20.06.2017 um 10:49 schrieb Thomas Meyer:
> > 
> > In UML the first userspace ptrace always fails, so init get's killed.
> > 
> > The check "count < fpu_user_xstate_size" was introduced by commit:
> > 
> > commit 91c3dba7dbc199191272f4a9863f86ea3bfd679f
> > Author: Yu-cheng Yu <yu-cheng.yu@intel.com>
> > Date:   Fri Jun 17 13:07:17 2016 -0700
> > 
> >     x86/fpu/xstate: Fix PTRACE frames for XSAVES
> >     
> >     XSAVES uses compacted format and is a kernel instruction. The kernel
> >     should use standard-format, non-supervisor state data for PTRACE.
> > 
> > So to summarize:
> > 
> > - PTRACE_GETREGSET with NT_X86_XSTATE gets 832 and return 832, with no
> > error.
> > 
> > - PTRACE_SETREGSET get 832 (sizeof struct _xstate) but wants at least
> > 1088, otherwise it will fail with -EFAULT (why not -EINVAL?)
> > 
> > Ideas?

We considered allowing a partial XSAVE buffer for PTRACE_SETREGSET, but
it was that the XSAVE instruction requires a full-size buffer led to
this choice.  Using a smaller buffer for XSAVE causes a fault.

Yu-cheng

Re: um: PTRACE_SETREGSET failure with XSTATE on Kabylake CPU

[Richard Weinberger]

Yu-cheng,

Am 20.06.2017 um 20:04 schrieb Yu-cheng Yu:
>>> So to summarize:
>>>
>>> - PTRACE_GETREGSET with NT_X86_XSTATE gets 832 and return 832, with no
>>> error.
>>>
>>> - PTRACE_SETREGSET get 832 (sizeof struct _xstate) but wants at least
>>> 1088, otherwise it will fail with -EFAULT (why not -EINVAL?)
>>>
>>> Ideas?
> 
> We considered allowing a partial XSAVE buffer for PTRACE_SETREGSET, but
> it was that the XSAVE instruction requires a full-size buffer led to
> this choice.  Using a smaller buffer for XSAVE causes a fault.

So, this code is not supposed to work?

iov.iov_base = fp_regs;
iov.iov_len = sizeof(struct _xstate);
ptrace(PTRACE_GETREGSET, pid, NT_X86_XSTATE, &iov);
ptrace(PTRACE_SETREGSET, pid, NT_X86_XSTATE, &iov);

This is what UML does and on Thomas's new Laptop PTRACE_SETREGSET is failing.

Thanks,
//richard

Re: um: PTRACE_SETREGSET failure with XSTATE on Kabylake CPU

[Richard Weinberger]

Yu-cheng,

Am 20.06.2017 um 20:17 schrieb Richard Weinberger:
> Yu-cheng,
> 
> Am 20.06.2017 um 20:04 schrieb Yu-cheng Yu:
>>>> So to summarize:
>>>>
>>>> - PTRACE_GETREGSET with NT_X86_XSTATE gets 832 and return 832, with no
>>>> error.
>>>>
>>>> - PTRACE_SETREGSET get 832 (sizeof struct _xstate) but wants at least
>>>> 1088, otherwise it will fail with -EFAULT (why not -EINVAL?)
>>>>
>>>> Ideas?
>>
>> We considered allowing a partial XSAVE buffer for PTRACE_SETREGSET, but
>> it was that the XSAVE instruction requires a full-size buffer led to
>> this choice.  Using a smaller buffer for XSAVE causes a fault.
> 
> So, this code is not supposed to work?
> 
> iov.iov_base = fp_regs;
> iov.iov_len = sizeof(struct _xstate);
> ptrace(PTRACE_GETREGSET, pid, NT_X86_XSTATE, &iov);
> ptrace(PTRACE_SETREGSET, pid, NT_X86_XSTATE, &iov);
> 
> This is what UML does and on Thomas's new Laptop PTRACE_SETREGSET is failing.

Hmm, I think we need to do what gdb does, it uses a buffer of size X86_XSTATE_MAX_SIZE.

Thanks,
//richard

Re: um: PTRACE_SETREGSET failure with XSTATE on Kabylake CPU

[Yu-cheng Yu]

On Tue, 2017-06-20 at 20:59 +0200, Richard Weinberger wrote:
> Yu-cheng,
> 
> Am 20.06.2017 um 20:17 schrieb Richard Weinberger:
> > Yu-cheng,
> > 
> > Am 20.06.2017 um 20:04 schrieb Yu-cheng Yu:
> >>>> So to summarize:
> >>>>
> >>>> - PTRACE_GETREGSET with NT_X86_XSTATE gets 832 and return 832, with no
> >>>> error.
> >>>>
> >>>> - PTRACE_SETREGSET get 832 (sizeof struct _xstate) but wants at least
> >>>> 1088, otherwise it will fail with -EFAULT (why not -EINVAL?)
> >>>>
> >>>> Ideas?
> >>
> >> We considered allowing a partial XSAVE buffer for PTRACE_SETREGSET, but
> >> it was that the XSAVE instruction requires a full-size buffer led to
> >> this choice.  Using a smaller buffer for XSAVE causes a fault.
> > 
> > So, this code is not supposed to work?
> > 
> > iov.iov_base = fp_regs;
> > iov.iov_len = sizeof(struct _xstate);
> > ptrace(PTRACE_GETREGSET, pid, NT_X86_XSTATE, &iov);
> > ptrace(PTRACE_SETREGSET, pid, NT_X86_XSTATE, &iov);
> > 
> > This is what UML does and on Thomas's new Laptop PTRACE_SETREGSET is failing.
> 
> Hmm, I think we need to do what gdb does, it uses a buffer of size X86_XSTATE_MAX_SIZE.
> 

Linux kernel determines XSAVE buffer size from CPUID:
http://elixir.free-electrons.com/linux/latest/source/arch/x86/kernel/fpu/xstate.c#L626

GDB has a fixed X86_XSTATE_MAX_SIZE of 2688.  That can become an issue.

Yu-cheng

Re: um: PTRACE_SETREGSET failure with XSTATE on Kabylake CPU

[Thomas Meyer]

[-- Attachment #1: Type: text/plain, Size: 1563 bytes --]


> Am 20.06.2017 um 21:53 schrieb Yu-cheng Yu <yu-cheng.yu@intel.com>:
> 
>> On Tue, 2017-06-20 at 20:59 +0200, Richard Weinberger wrote:
>> Yu-cheng,
>> 
>>> Am 20.06.2017 um 20:17 schrieb Richard Weinberger:
>>> Yu-cheng,
>>> 
>>> Am 20.06.2017 um 20:04 schrieb Yu-cheng Yu:
>>>>>> So to summarize:
>>>>>> 
>>>>>> - PTRACE_GETREGSET with NT_X86_XSTATE gets 832 and return 832, with no
>>>>>> error.
>>>>>> 
>>>>>> - PTRACE_SETREGSET get 832 (sizeof struct _xstate) but wants at least
>>>>>> 1088, otherwise it will fail with -EFAULT (why not -EINVAL?)
>>>>>> 
>>>>>> Ideas?
>>>> 
>>>> We considered allowing a partial XSAVE buffer for PTRACE_SETREGSET, but
>>>> it was that the XSAVE instruction requires a full-size buffer led to
>>>> this choice.  Using a smaller buffer for XSAVE causes a fault.
>>> 
>>> So, this code is not supposed to work?
>>> 
>>> iov.iov_base = fp_regs;
>>> iov.iov_len = sizeof(struct _xstate);
>>> ptrace(PTRACE_GETREGSET, pid, NT_X86_XSTATE, &iov);
>>> ptrace(PTRACE_SETREGSET, pid, NT_X86_XSTATE, &iov);
>>> 
>>> This is what UML does and on Thomas's new Laptop PTRACE_SETREGSET is failing.
>> 
>> Hmm, I think we need to do what gdb does, it uses a buffer of size X86_XSTATE_MAX_SIZE.
> 
> Linux kernel determines XSAVE buffer size from CPUID:
> http://elixir.free-electrons.com/linux/latest/source/arch/x86/kernel/fpu/xstate.c#L626

Hi,

Is there a user space API to get this value?

> 
> GDB has a fixed X86_XSTATE_MAX_SIZE of 2688.  That can become an issue.
> 
> Yu-cheng
> 
> 

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5334 bytes --]