From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 66CA4C76196 for ; Sun, 19 Mar 2023 00:16:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0258028001B; Sat, 18 Mar 2023 20:16:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E5563280001; Sat, 18 Mar 2023 20:16:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C103A28001B; Sat, 18 Mar 2023 20:16:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 85C0D280001 for ; Sat, 18 Mar 2023 20:16:46 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 5A6B880CAD for ; Sun, 19 Mar 2023 00:16:46 +0000 (UTC) X-FDA: 80583732012.09.4047C83 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by imf21.hostedemail.com (Postfix) with ESMTP id 20E8E1C0018 for ; Sun, 19 Mar 2023 00:16:43 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=ag6yogTs; spf=pass (imf21.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.115 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679185004; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references:dkim-signature; bh=o3Rz70ck6nM+CTFRpD6v5ZMhHD7aV88XHo5GuS7VB1o=; b=XBSAvkrZ4tL6ZsXCF8sxm1aIZWkLr/UoKo90ddLQLm3a3l/r1Xzgen4vD8Ress5TuAK7s4 7zo0jP9BQwpFsxLIE1+TauxwXTJzJVqBvKlXVCfs3VaqLGSGGpRSyykpFkIUnSHBvIfYtq OIRnx6hKPHcRTQ+zYqoQDQUQdaE0Pkk= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=ag6yogTs; spf=pass (imf21.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.115 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1679185004; a=rsa-sha256; cv=none; b=SxTa7BvVtsZcVYCvoHzsGAs4JrawlqPXLT+haCXSr69nH4IepRn/dZjqR/GKRlzMcZQcX4 oN5hcAPhg78vszVouhQHi9g/LV6kNl3n1TUIqTvVZsiPUNYLYhnszb67AEJkOT3zd6nyW0 M7yFvrA03uZkURPSBoLDljxuM1ZIVig= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1679185004; x=1710721004; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=qFnC8BjwG9ajYhpSUihMC4CLPCFBVSbSaq8Aw+3zJPY=; b=ag6yogTsZbZuW1DUHkTjXOpJzuDAXCL5LGr6xf4PayfzMNWj9cjpB1R0 31m2mP+oKhCEf+MxfysvRj+plueAypUFz8C7g7XoQd71TyPS/sGmgjNK5 wHZAlujV0DcA+gaJNk1FlEz4KV3uf2crqCtnvvvWEmbJAHyBUxc7rv9oR 2iMQy/x1/gDMalKsElXGuXbBeQLaPtFaGJdvvbe7X61LoMn9XmrzI3LzI Phfeq0GwqHsb53uVlJwZZAR6XPcC5vdAw2Qkzmni6asmd4kFsUv53FQus uPvozB5ERjoIiMFapkTU0kkpjKv1M62rDM7KpxiaFQj0c2xUqYsH77YL7 A==; X-IronPort-AV: E=McAfee;i="6600,9927,10653"; a="338491387" X-IronPort-AV: E=Sophos;i="5.98,272,1673942400"; d="scan'208";a="338491387" Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 Mar 2023 17:16:43 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10653"; a="749672927" X-IronPort-AV: E=Sophos;i="5.98,272,1673942400"; d="scan'208";a="749672927" Received: from bmahatwo-mobl1.gar.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.135.34.5]) by fmsmga004-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 Mar 2023 17:16:41 -0700 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com, david@redhat.com, debug@rivosinc.com, szabolcs.nagy@arm.com Cc: rick.p.edgecombe@intel.com, Yu-cheng Yu Subject: [PATCH v8 29/40] x86/shstk: Add user-mode shadow stack support Date: Sat, 18 Mar 2023 17:15:24 -0700 Message-Id: <20230319001535.23210-30-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230319001535.23210-1-rick.p.edgecombe@intel.com> References: <20230319001535.23210-1-rick.p.edgecombe@intel.com> X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 20E8E1C0018 X-Rspam-User: X-Stat-Signature: ict4xjdqrmsxdqwu9bu4z36pjh674uc4 X-HE-Tag: 1679185003-601331 X-HE-Meta: U2FsdGVkX1+S2FBELCi2hexOd5OIDqngz8dG1ReUawVDBfBtViEBPxIpisXHDC0Vr7QkdXx8K0MIOj+sAy80qK3m3TlVgByLCy1SwGAdyUYeaeBmrwjHw4VBcmaQrvC9DQ1fui3QJVLL953m33Qiaw4EFweMgDaPMmCr1YYWafUkSuDJxpFQkH7w0yTwo3lyfDqNlX19PW0kGMapfPB/jcA60iIJdSOoDgcdHTgqRzMVGZb5NU1GTG85I2xBiZSi70dweTSCHIOyzeY/Vlzk6ekVso4Q4ZBk4ENFF6jfWFHkMCF38NILAzA/2FcSMm6PpP212npN4iDuWwvPSKDFj6Z/vF6IFzwvQkyGvDanKexsrvo4/Zzuj6jMKlSNvM+9ofyUnLOoOg96GhdCa0isJq/o5ouLKmNvrIc/llRbkz37fuhC4+4Wv97qK0jylC0YQT2oaM/ioPc6i5AXXAJ88DK2FpKnwEW3Kvkh3PGpDDzDtfqfv5tsTmnkbYtmGnXX5c8Br8hgSTnPrUGNym19XeXvalZ8IvNoOKnYoCtckZKBIBAdzK3IBA2mtb7sa9DBb0CXA+M5f7od1DDdTni82Rr8J1TFf829kqr1nkZUS/mNZnYATmT4XIrUg43SuPd8fL39u/JMNcPYNME5CetU0aCVHOpGz/POnunUj7XgcqhC1J3BnExF+Id7snIyE2+xraO7WSAlS4+ryGB/rZvDdSFOwrY3jlvYklfuf/5mkOrvGk/KtRsf7BWbZZOtHfna7UsMdi0Hi8r9TXYveteCV0QiOh5Jh8/GAddpL1jFozx6vyWJMZp5jfLRSGKRV6SDvVUNClLOLsAeYcgG5wydvpswjs5OyIW2Diwo9BrvviarLloj6siOY04NNiYYAXXUDtI2kRUUAro8t5zWpo0ibQpqj72jW9ixf1m/Daurs+JL0NTXsWx0UPt9c/GUt3hIxy6cbMfGjid/Jlutj9B yAjEyNPH DkWclCvXRr81lRthA4sl6Ucs7nGvaLtYz4nTUrfa01os2y1Wnu9gXtguBongTVnIZH9Vy8oIyxQtRcBb8xjsQiHx4h+GPf0evb975T1gusA9bBRao51pNTtJ7jZuOVoo21hl6Kl6jyeaX5PYNKwPCiv2CJFMVdllmmmvJsTbT9URG6S45wGp7HT//QJYGaWZjdk5F2zN/qbcXE39SaDUX5aBrWpkihMGDkWiDsgzxMEW0hjZpEMSX8qncg9JvhR1Khahzlxyx1JHbNW2kyh6HyxI2wcp76+pR43EtuLKTOhRYC4UPGa7VXbzoh7CtcQ+PFjHzde4PQN2gSOSdXOaW3YtE8rUq7APE8GxA X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Introduce basic shadow stack enabling/disabling/allocation routines. A task's shadow stack is allocated from memory with VM_SHADOW_STACK flag and has a fixed size of min(RLIMIT_STACK, 4GB). Keep the task's shadow stack address and size in thread_struct. This will be copied when cloning new threads, but needs to be cleared during exec, so add a function to do this. 32 bit shadow stack is not expected to have many users and it will complicate the signal implementation. So do not support IA32 emulation or x32. Co-developed-by: Yu-cheng Yu Signed-off-by: Yu-cheng Yu Signed-off-by: Rick Edgecombe Reviewed-by: Kees Cook Acked-by: Mike Rapoport (IBM) Tested-by: Pengfei Xu Tested-by: John Allen Tested-by: Kees Cook --- v7: - Add explanation for not supporting 32 bit in commit log (Boris) v5: - Switch to EOPNOTSUPP - Use MAP_ABOVE4G - Move set_clr_bits_msrl() to patch where it is first used v4: - Just set MSR_IA32_U_CET when disabling shadow stack, since we don't have IBT yet. (Peterz) v3: - Use define for set_clr_bits_msrl() (Kees) - Make some functions static (Kees) - Change feature_foo() to features_foo() (Kees) - Centralize shadow stack size rlimit checks (Kees) - Disable x32 support --- arch/x86/include/asm/processor.h | 2 + arch/x86/include/asm/shstk.h | 7 ++ arch/x86/include/uapi/asm/prctl.h | 3 + arch/x86/kernel/shstk.c | 145 ++++++++++++++++++++++++++++++ 4 files changed, 157 insertions(+) diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h index bd16e012b3e9..ff98cd6d5af2 100644 --- a/arch/x86/include/asm/processor.h +++ b/arch/x86/include/asm/processor.h @@ -479,6 +479,8 @@ struct thread_struct { #ifdef CONFIG_X86_USER_SHADOW_STACK unsigned long features; unsigned long features_locked; + + struct thread_shstk shstk; #endif /* Floating point and extended processor state */ diff --git a/arch/x86/include/asm/shstk.h b/arch/x86/include/asm/shstk.h index ec753809f074..2b1f7c9b9995 100644 --- a/arch/x86/include/asm/shstk.h +++ b/arch/x86/include/asm/shstk.h @@ -8,12 +8,19 @@ struct task_struct; #ifdef CONFIG_X86_USER_SHADOW_STACK +struct thread_shstk { + u64 base; + u64 size; +}; + long shstk_prctl(struct task_struct *task, int option, unsigned long features); void reset_thread_features(void); +void shstk_free(struct task_struct *p); #else static inline long shstk_prctl(struct task_struct *task, int option, unsigned long arg2) { return -EINVAL; } static inline void reset_thread_features(void) {} +static inline void shstk_free(struct task_struct *p) {} #endif /* CONFIG_X86_USER_SHADOW_STACK */ #endif /* __ASSEMBLY__ */ diff --git a/arch/x86/include/uapi/asm/prctl.h b/arch/x86/include/uapi/asm/prctl.h index b2b3b7200b2d..7dfd9dc00509 100644 --- a/arch/x86/include/uapi/asm/prctl.h +++ b/arch/x86/include/uapi/asm/prctl.h @@ -26,4 +26,7 @@ #define ARCH_SHSTK_DISABLE 0x5002 #define ARCH_SHSTK_LOCK 0x5003 +/* ARCH_SHSTK_ features bits */ +#define ARCH_SHSTK_SHSTK (1ULL << 0) + #endif /* _ASM_X86_PRCTL_H */ diff --git a/arch/x86/kernel/shstk.c b/arch/x86/kernel/shstk.c index 41ed6552e0a5..3cb85224d856 100644 --- a/arch/x86/kernel/shstk.c +++ b/arch/x86/kernel/shstk.c @@ -8,14 +8,159 @@ #include #include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include #include +static bool features_enabled(unsigned long features) +{ + return current->thread.features & features; +} + +static void features_set(unsigned long features) +{ + current->thread.features |= features; +} + +static void features_clr(unsigned long features) +{ + current->thread.features &= ~features; +} + +static unsigned long alloc_shstk(unsigned long size) +{ + int flags = MAP_ANONYMOUS | MAP_PRIVATE | MAP_ABOVE4G; + struct mm_struct *mm = current->mm; + unsigned long addr, unused; + + mmap_write_lock(mm); + addr = do_mmap(NULL, addr, size, PROT_READ, flags, + VM_SHADOW_STACK | VM_WRITE, 0, &unused, NULL); + + mmap_write_unlock(mm); + + return addr; +} + +static unsigned long adjust_shstk_size(unsigned long size) +{ + if (size) + return PAGE_ALIGN(size); + + return PAGE_ALIGN(min_t(unsigned long long, rlimit(RLIMIT_STACK), SZ_4G)); +} + +static void unmap_shadow_stack(u64 base, u64 size) +{ + while (1) { + int r; + + r = vm_munmap(base, size); + + /* + * vm_munmap() returns -EINTR when mmap_lock is held by + * something else, and that lock should not be held for a + * long time. Retry it for the case. + */ + if (r == -EINTR) { + cond_resched(); + continue; + } + + /* + * For all other types of vm_munmap() failure, either the + * system is out of memory or there is bug. + */ + WARN_ON_ONCE(r); + break; + } +} + +static int shstk_setup(void) +{ + struct thread_shstk *shstk = ¤t->thread.shstk; + unsigned long addr, size; + + /* Already enabled */ + if (features_enabled(ARCH_SHSTK_SHSTK)) + return 0; + + /* Also not supported for 32 bit and x32 */ + if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK) || in_32bit_syscall()) + return -EOPNOTSUPP; + + size = adjust_shstk_size(0); + addr = alloc_shstk(size); + if (IS_ERR_VALUE(addr)) + return PTR_ERR((void *)addr); + + fpregs_lock_and_load(); + wrmsrl(MSR_IA32_PL3_SSP, addr + size); + wrmsrl(MSR_IA32_U_CET, CET_SHSTK_EN); + fpregs_unlock(); + + shstk->base = addr; + shstk->size = size; + features_set(ARCH_SHSTK_SHSTK); + + return 0; +} + void reset_thread_features(void) { + memset(¤t->thread.shstk, 0, sizeof(struct thread_shstk)); current->thread.features = 0; current->thread.features_locked = 0; } +void shstk_free(struct task_struct *tsk) +{ + struct thread_shstk *shstk = &tsk->thread.shstk; + + if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK) || + !features_enabled(ARCH_SHSTK_SHSTK)) + return; + + if (!tsk->mm) + return; + + unmap_shadow_stack(shstk->base, shstk->size); +} + +static int shstk_disable(void) +{ + if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK)) + return -EOPNOTSUPP; + + /* Already disabled? */ + if (!features_enabled(ARCH_SHSTK_SHSTK)) + return 0; + + fpregs_lock_and_load(); + /* Disable WRSS too when disabling shadow stack */ + wrmsrl(MSR_IA32_U_CET, 0); + wrmsrl(MSR_IA32_PL3_SSP, 0); + fpregs_unlock(); + + shstk_free(current); + features_clr(ARCH_SHSTK_SHSTK); + + return 0; +} + long shstk_prctl(struct task_struct *task, int option, unsigned long features) { if (option == ARCH_SHSTK_LOCK) { -- 2.17.1