From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C606AC433DB for ; Tue, 16 Mar 2021 15:15:30 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 548B66508F for ; Tue, 16 Mar 2021 15:15:30 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 548B66508F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=intel.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id DF0BA6B0071; Tue, 16 Mar 2021 11:15:29 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DC5BC8D0002; Tue, 16 Mar 2021 11:15:29 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C3FE16B0075; Tue, 16 Mar 2021 11:15:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0239.hostedemail.com [216.40.44.239]) by kanga.kvack.org (Postfix) with ESMTP id 9FE436B0071 for ; Tue, 16 Mar 2021 11:15:29 -0400 (EDT) Received: from smtpin11.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 5D30E180AD83E for ; Tue, 16 Mar 2021 15:15:29 +0000 (UTC) X-FDA: 77926086378.11.0284580 Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by imf02.hostedemail.com (Postfix) with ESMTP id 3E17D4080F68 for ; Tue, 16 Mar 2021 15:15:22 +0000 (UTC) IronPort-SDR: ASGhcVcQckZlm2SaOnyCxgTUngHHu4KBYWpF62wFRSqzB4+ImgzJzPprXY7/WG9I4YqxvhDKXp 46fU6KGgDGvg== X-IronPort-AV: E=McAfee;i="6000,8403,9924"; a="169190334" X-IronPort-AV: E=Sophos;i="5.81,251,1610438400"; d="scan'208";a="169190334" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Mar 2021 08:11:33 -0700 IronPort-SDR: Mw0Pewq14bswbB92Ynxu5yvvO2dqfzacvV1i/Yck7OpZxofl2d0j62ZKG1Jvqnav29XwamPBpH /GVf15vfaCng== X-IronPort-AV: E=Sophos;i="5.81,251,1610438400"; d="scan'208";a="405570323" Received: from yyu32-desk.sc.intel.com ([143.183.136.146]) by fmsmga008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Mar 2021 08:11:32 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue , Dave Martin , Weijiang Yang , Pengfei Xu , Haitao Huang Cc: Yu-cheng Yu Subject: [PATCH v23 23/28] x86/cet/shstk: Handle signals for shadow stack Date: Tue, 16 Mar 2021 08:10:49 -0700 Message-Id: <20210316151054.5405-24-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20210316151054.5405-1-yu-cheng.yu@intel.com> References: <20210316151054.5405-1-yu-cheng.yu@intel.com> MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 3E17D4080F68 X-Stat-Signature: ty9z4ux5s51q4sxknu4r16na88abi9wf Received-SPF: none (intel.com>: No applicable sender policy available) receiver=imf02; identity=mailfrom; envelope-from=""; helo=mga17.intel.com; client-ip=192.55.52.151 X-HE-DKIM-Result: none/none X-HE-Tag: 1615907722-765176 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: To deliver a signal, create a shadow stack restore token and put the toke= n and the signal restorer address on the shadow stack. For sigreturn, veri= fy the token and restore from it the shadow stack pointer. A shadow stack restore token marks a restore point of the shadow stack, a= nd the address in a token must point directly above the token, which is with= in the same shadow stack. This is distinctively different from other pointe= rs on the shadow stack; those pointers point to executable code area. In sigreturn, restoring from a token ensures the target address is the location pointed by the token. Introduce WRUSS, which is a kernel-mode instruction but writes directly t= o user shadow stack. It is used to construct the user signal stack as described above. Currently there is no systematic facility for extending a signal context. Introduce a signal context extension 'struct sc_ext', which is used to sa= ve shadow stack restore token address and WAIT_ENDBR status. WAIT_ENDBR wil= l be introduced later in the Indirect Branch Tracking (IBT) series, but add that into sc_ext now to keep the struct stable in case the IBT series is applied later. Signed-off-by: Yu-cheng Yu Reviewed-by: Kees Cook --- arch/x86/ia32/ia32_signal.c | 17 +++ arch/x86/include/asm/cet.h | 8 ++ arch/x86/include/asm/fpu/internal.h | 10 ++ arch/x86/include/asm/special_insns.h | 32 ++++++ arch/x86/include/uapi/asm/sigcontext.h | 9 ++ arch/x86/kernel/cet.c | 152 +++++++++++++++++++++++++ arch/x86/kernel/fpu/signal.c | 100 ++++++++++++++++ arch/x86/kernel/signal.c | 10 ++ 8 files changed, 338 insertions(+) diff --git a/arch/x86/ia32/ia32_signal.c b/arch/x86/ia32/ia32_signal.c index 5e3d9b7fd5fb..aee3e367e184 100644 --- a/arch/x86/ia32/ia32_signal.c +++ b/arch/x86/ia32/ia32_signal.c @@ -35,6 +35,7 @@ #include #include #include +#include =20 static inline void reload_segments(struct sigcontext_32 *sc) { @@ -205,6 +206,7 @@ static void __user *get_sigframe(struct ksignal *ksig= , struct pt_regs *regs, void __user **fpstate) { unsigned long sp, fx_aligned, math_size; + void __user *restorer =3D NULL; =20 /* Default to using normal stack */ sp =3D regs->sp; @@ -218,8 +220,23 @@ static void __user *get_sigframe(struct ksignal *ksi= g, struct pt_regs *regs, ksig->ka.sa.sa_restorer) sp =3D (unsigned long) ksig->ka.sa.sa_restorer; =20 + if (ksig->ka.sa.sa_flags & SA_RESTORER) { + restorer =3D ksig->ka.sa.sa_restorer; + } else if (current->mm->context.vdso) { + if (ksig->ka.sa.sa_flags & SA_SIGINFO) + restorer =3D current->mm->context.vdso + + vdso_image_32.sym___kernel_rt_sigreturn; + else + restorer =3D current->mm->context.vdso + + vdso_image_32.sym___kernel_sigreturn; + } + sp =3D fpu__alloc_mathframe(sp, 1, &fx_aligned, &math_size); *fpstate =3D (struct _fpstate_32 __user *) sp; + + if (save_cet_to_sigframe(1, *fpstate, (unsigned long)restorer)) + return (void __user *)-1L; + if (copy_fpstate_to_sigframe(*fpstate, (void __user *)fx_aligned, math_size) < 0) return (void __user *) -1L; diff --git a/arch/x86/include/asm/cet.h b/arch/x86/include/asm/cet.h index 5750fbcbb952..73435856ce54 100644 --- a/arch/x86/include/asm/cet.h +++ b/arch/x86/include/asm/cet.h @@ -6,6 +6,8 @@ #include =20 struct task_struct; +struct sc_ext; + /* * Per-thread CET status */ @@ -18,9 +20,15 @@ struct cet_status { int cet_setup_shstk(void); void cet_disable_shstk(void); void cet_free_shstk(struct task_struct *p); +int cet_verify_rstor_token(bool ia32, unsigned long ssp, unsigned long *= new_ssp); +void cet_restore_signal(struct sc_ext *sc); +int cet_setup_signal(bool ia32, unsigned long rstor, struct sc_ext *sc); #else static inline void cet_disable_shstk(void) {} static inline void cet_free_shstk(struct task_struct *p) {} +static inline void cet_restore_signal(struct sc_ext *sc) { return; } +static inline int cet_setup_signal(bool ia32, unsigned long rstor, + struct sc_ext *sc) { return -EINVAL; } #endif =20 #endif /* __ASSEMBLY__ */ diff --git a/arch/x86/include/asm/fpu/internal.h b/arch/x86/include/asm/f= pu/internal.h index 8d33ad80704f..2c1f59ebe9d8 100644 --- a/arch/x86/include/asm/fpu/internal.h +++ b/arch/x86/include/asm/fpu/internal.h @@ -443,6 +443,16 @@ static inline void copy_kernel_to_fpregs(union fpreg= s_state *fpstate) __copy_kernel_to_fpregs(fpstate, -1); } =20 +#ifdef CONFIG_X86_CET +extern int save_cet_to_sigframe(int ia32, void __user *fp, + unsigned long restorer); +#else +static inline int save_cet_to_sigframe(int ia32, void __user *fp, + unsigned long restorer) +{ + return 0; +} +#endif extern int copy_fpstate_to_sigframe(void __user *buf, void __user *fp, i= nt size); =20 /* diff --git a/arch/x86/include/asm/special_insns.h b/arch/x86/include/asm/= special_insns.h index 1d3cbaef4bb7..cc7b3c999366 100644 --- a/arch/x86/include/asm/special_insns.h +++ b/arch/x86/include/asm/special_insns.h @@ -234,6 +234,38 @@ static inline void clwb(volatile void *__p) : [pax] "a" (p)); } =20 +#ifdef CONFIG_X86_CET +#if defined(CONFIG_IA32_EMULATION) || defined(CONFIG_X86_X32) +static inline int write_user_shstk_32(unsigned long addr, unsigned int v= al) +{ + asm_volatile_goto("1: wrussd %1, (%0)\n" + _ASM_EXTABLE(1b, %l[fail]) + :: "r" (addr), "r" (val) + :: fail); + return 0; +fail: + return -EPERM; +} +#else +static inline int write_user_shstk_32(unsigned long addr, unsigned int v= al) +{ + WARN_ONCE(1, "%s used but not supported.\n", __func__); + return -EFAULT; +} +#endif + +static inline int write_user_shstk_64(unsigned long addr, unsigned long = val) +{ + asm_volatile_goto("1: wrussq %1, (%0)\n" + _ASM_EXTABLE(1b, %l[fail]) + :: "r" (addr), "r" (val) + :: fail); + return 0; +fail: + return -EPERM; +} +#endif /* CONFIG_X86_CET */ + #define nop() asm volatile ("nop") =20 static inline void serialize(void) diff --git a/arch/x86/include/uapi/asm/sigcontext.h b/arch/x86/include/ua= pi/asm/sigcontext.h index 844d60eb1882..cf2d55db3be4 100644 --- a/arch/x86/include/uapi/asm/sigcontext.h +++ b/arch/x86/include/uapi/asm/sigcontext.h @@ -196,6 +196,15 @@ struct _xstate { /* New processor state extensions go here: */ }; =20 +/* + * Located at the end of sigcontext->fpstate, aligned to 8. + */ +struct sc_ext { + unsigned long total_size; + unsigned long ssp; + unsigned long wait_endbr; +}; + /* * The 32-bit signal frame: */ diff --git a/arch/x86/kernel/cet.c b/arch/x86/kernel/cet.c index d25a03215984..08e43d9b5176 100644 --- a/arch/x86/kernel/cet.c +++ b/arch/x86/kernel/cet.c @@ -19,6 +19,8 @@ #include #include #include +#include +#include =20 static void start_update_msrs(void) { @@ -72,6 +74,80 @@ static unsigned long alloc_shstk(unsigned long size, i= nt flags) return addr; } =20 +#define TOKEN_MODE_MASK 3UL +#define TOKEN_MODE_64 1UL +#define IS_TOKEN_64(token) (((token) & TOKEN_MODE_MASK) =3D=3D TOKEN_MOD= E_64) +#define IS_TOKEN_32(token) (((token) & TOKEN_MODE_MASK) =3D=3D 0) + +/* + * Verify the restore token at the address of 'ssp' is + * valid and then set shadow stack pointer according to the + * token. + */ +int cet_verify_rstor_token(bool ia32, unsigned long ssp, + unsigned long *new_ssp) +{ + unsigned long token; + + *new_ssp =3D 0; + + if (!IS_ALIGNED(ssp, 8)) + return -EINVAL; + + if (get_user(token, (unsigned long __user *)ssp)) + return -EFAULT; + + /* Is 64-bit mode flag correct? */ + if (!ia32 && !IS_TOKEN_64(token)) + return -EINVAL; + else if (ia32 && !IS_TOKEN_32(token)) + return -EINVAL; + + token &=3D ~TOKEN_MODE_MASK; + + /* + * Restore address properly aligned? + */ + if ((!ia32 && !IS_ALIGNED(token, 8)) || !IS_ALIGNED(token, 4)) + return -EINVAL; + + /* + * Token was placed properly? + */ + if (((ALIGN_DOWN(token, 8) - 8) !=3D ssp) || token >=3D TASK_SIZE_MAX) + return -EINVAL; + + *new_ssp =3D token; + return 0; +} + +/* + * Create a restore token on the shadow stack. + * A token is always 8-byte and aligned to 8. + */ +static int create_rstor_token(bool ia32, unsigned long ssp, + unsigned long *new_ssp) +{ + unsigned long addr; + + *new_ssp =3D 0; + + if ((!ia32 && !IS_ALIGNED(ssp, 8)) || !IS_ALIGNED(ssp, 4)) + return -EINVAL; + + addr =3D ALIGN_DOWN(ssp, 8) - 8; + + /* Is the token for 64-bit? */ + if (!ia32) + ssp |=3D TOKEN_MODE_64; + + if (write_user_shstk_64(addr, ssp)) + return -EFAULT; + + *new_ssp =3D addr; + return 0; +} + int cet_setup_shstk(void) { unsigned long addr, size; @@ -145,3 +221,79 @@ void cet_free_shstk(struct task_struct *tsk) cet->shstk_base =3D 0; cet->shstk_size =3D 0; } + +/* + * Called from __fpu__restore_sig() and XSAVES buffer is protected by + * set_thread_flag(TIF_NEED_FPU_LOAD) in the slow path. + */ +void cet_restore_signal(struct sc_ext *sc_ext) +{ + struct cet_user_state *cet_user_state; + struct cet_status *cet =3D ¤t->thread.cet; + u64 msr_val =3D 0; + + if (!static_cpu_has(X86_FEATURE_SHSTK)) + return; + + cet_user_state =3D get_xsave_addr(¤t->thread.fpu.state.xsave, + XFEATURE_CET_USER); + if (!cet_user_state) + return; + + if (cet->shstk_size) { + if (test_thread_flag(TIF_NEED_FPU_LOAD)) + cet_user_state->user_ssp =3D sc_ext->ssp; + else + wrmsrl(MSR_IA32_PL3_SSP, sc_ext->ssp); + + msr_val |=3D CET_SHSTK_EN; + } + + if (test_thread_flag(TIF_NEED_FPU_LOAD)) + cet_user_state->user_cet =3D msr_val; + else + wrmsrl(MSR_IA32_U_CET, msr_val); +} + +/* + * Setup the shadow stack for the signal handler: first, + * create a restore token to keep track of the current ssp, + * and then the return address of the signal handler. + */ +int cet_setup_signal(bool ia32, unsigned long rstor_addr, struct sc_ext = *sc_ext) +{ + struct cet_status *cet =3D ¤t->thread.cet; + unsigned long ssp =3D 0, new_ssp =3D 0; + int err; + + if (cet->shstk_size) { + if (!rstor_addr) + return -EINVAL; + + ssp =3D cet_get_shstk_addr(); + err =3D create_rstor_token(ia32, ssp, &new_ssp); + if (err) + return err; + + if (ia32) { + ssp =3D new_ssp - sizeof(u32); + err =3D write_user_shstk_32(ssp, (unsigned int)rstor_addr); + } else { + ssp =3D new_ssp - sizeof(u64); + err =3D write_user_shstk_64(ssp, rstor_addr); + } + + if (err) + return err; + + sc_ext->ssp =3D new_ssp; + } + + if (ssp) { + start_update_msrs(); + wrmsrl(MSR_IA32_PL3_SSP, ssp); + end_update_msrs(); + } + + return 0; +} diff --git a/arch/x86/kernel/fpu/signal.c b/arch/x86/kernel/fpu/signal.c index a4ec65317a7f..270e4649f435 100644 --- a/arch/x86/kernel/fpu/signal.c +++ b/arch/x86/kernel/fpu/signal.c @@ -52,6 +52,74 @@ static inline int check_for_xstate(struct fxregs_state= __user *buf, return 0; } =20 +#ifdef CONFIG_X86_CET +int save_cet_to_sigframe(int ia32, void __user *fp, unsigned long restor= er) +{ + int err =3D 0; + + if (!current->thread.cet.shstk_size) + return 0; + + if (fp) { + struct sc_ext ext =3D {}; + + err =3D cet_setup_signal(ia32, restorer, &ext); + if (!err) { + void __user *p =3D fp; + + ext.total_size =3D sizeof(ext); + + if (ia32) + p +=3D sizeof(struct fregs_state); + + p +=3D fpu_user_xstate_size + FP_XSTATE_MAGIC2_SIZE; + p =3D (void __user *)ALIGN((unsigned long)p, 8); + + if (copy_to_user(p, &ext, sizeof(ext))) + return -EFAULT; + } + } + + return err; +} + +static int get_cet_from_sigframe(int ia32, void __user *fp, struct sc_ex= t *ext) +{ + int err =3D 0; + + memset(ext, 0, sizeof(*ext)); + + if (!current->thread.cet.shstk_size) + return 0; + + if (fp) { + void __user *p =3D fp; + + if (ia32) + p +=3D sizeof(struct fregs_state); + + p +=3D fpu_user_xstate_size + FP_XSTATE_MAGIC2_SIZE; + p =3D (void __user *)ALIGN((unsigned long)p, 8); + + if (copy_from_user(ext, p, sizeof(*ext))) + return -EFAULT; + + if (ext->total_size !=3D sizeof(*ext)) + return -EFAULT; + + if (current->thread.cet.shstk_size) + err =3D cet_verify_rstor_token(ia32, ext->ssp, &ext->ssp); + } + + return err; +} +#else +static int get_cet_from_sigframe(int ia32, void __user *fp, struct sc_ex= t *ext) +{ + return 0; +} +#endif + /* * Signal frame handlers. */ @@ -295,6 +363,7 @@ static int __fpu__restore_sig(void __user *buf, void = __user *buf_fx, int size) struct task_struct *tsk =3D current; struct fpu *fpu =3D &tsk->thread.fpu; struct user_i387_ia32_struct env; + struct sc_ext sc_ext; u64 user_xfeatures =3D 0; int fx_only =3D 0; int ret =3D 0; @@ -335,6 +404,10 @@ static int __fpu__restore_sig(void __user *buf, void= __user *buf_fx, int size) if ((unsigned long)buf_fx % 64) fx_only =3D 1; =20 + ret =3D get_cet_from_sigframe(ia32_fxstate, buf, &sc_ext); + if (ret) + return ret; + if (!ia32_fxstate) { /* * Attempt to restore the FPU registers directly from user @@ -349,6 +422,8 @@ static int __fpu__restore_sig(void __user *buf, void = __user *buf_fx, int size) pagefault_enable(); if (!ret) { =20 + cet_restore_signal(&sc_ext); + /* * Restore supervisor states: previous context switch * etc has done XSAVES and saved the supervisor states @@ -423,6 +498,8 @@ static int __fpu__restore_sig(void __user *buf, void = __user *buf_fx, int size) if (unlikely(init_bv)) copy_kernel_to_xregs(&init_fpstate.xsave, init_bv); =20 + cet_restore_signal(&sc_ext); + /* * Restore previously saved supervisor xstates along with * copied-in user xstates. @@ -491,12 +568,35 @@ int fpu__restore_sig(void __user *buf, int ia32_fra= me) return __fpu__restore_sig(buf, buf_fx, size); } =20 +#ifdef CONFIG_X86_CET +static unsigned long fpu__alloc_sigcontext_ext(unsigned long sp) +{ + struct cet_status *cet =3D ¤t->thread.cet; + + /* + * sigcontext_ext is at: fpu + fpu_user_xstate_size + + * FP_XSTATE_MAGIC2_SIZE, then aligned to 8. + */ + if (cet->shstk_size) + sp -=3D (sizeof(struct sc_ext) + 8); + + return sp; +} +#else +static unsigned long fpu__alloc_sigcontext_ext(unsigned long sp) +{ + return sp; +} +#endif + unsigned long fpu__alloc_mathframe(unsigned long sp, int ia32_frame, unsigned long *buf_fx, unsigned long *size) { unsigned long frame_size =3D xstate_sigframe_size(); =20 + sp =3D fpu__alloc_sigcontext_ext(sp); + *buf_fx =3D sp =3D round_down(sp - frame_size, 64); if (ia32_frame && use_fxsr()) { frame_size +=3D sizeof(struct fregs_state); diff --git a/arch/x86/kernel/signal.c b/arch/x86/kernel/signal.c index ea794a083c44..1807379f1d86 100644 --- a/arch/x86/kernel/signal.c +++ b/arch/x86/kernel/signal.c @@ -46,6 +46,7 @@ #include #include #include +#include =20 #ifdef CONFIG_X86_64 /* @@ -239,6 +240,9 @@ get_sigframe(struct k_sigaction *ka, struct pt_regs *= regs, size_t frame_size, unsigned long buf_fx =3D 0; int onsigstack =3D on_sig_stack(sp); int ret; +#ifdef CONFIG_X86_64 + void __user *restorer =3D NULL; +#endif =20 /* redzone */ if (IS_ENABLED(CONFIG_X86_64)) @@ -270,6 +274,12 @@ get_sigframe(struct k_sigaction *ka, struct pt_regs = *regs, size_t frame_size, if (onsigstack && !likely(on_sig_stack(sp))) return (void __user *)-1L; =20 +#ifdef CONFIG_X86_64 + if (ka->sa.sa_flags & SA_RESTORER) + restorer =3D ka->sa.sa_restorer; + ret =3D save_cet_to_sigframe(0, *fpstate, (unsigned long)restorer); +#endif + /* save i387 and extended state */ ret =3D copy_fpstate_to_sigframe(*fpstate, (void __user *)buf_fx, math_= size); if (ret < 0) --=20 2.21.0