From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753240AbbGMT7n (ORCPT ); Mon, 13 Jul 2015 15:59:43 -0400 Received: from mail-db3on0064.outbound.protection.outlook.com ([157.55.234.64]:62937 "EHLO emea01-db3-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1753004AbbGMT6X (ORCPT ); Mon, 13 Jul 2015 15:58:23 -0400 Authentication-Results: spf=fail (sender IP is 12.216.194.146) smtp.mailfrom=ezchip.com; ezchip.com; dkim=none (message not signed) header.d=none; From: Chris Metcalf To: Gilad Ben Yossef , Steven Rostedt , Ingo Molnar , Peter Zijlstra , Andrew Morton , Rik van Riel , Tejun Heo , Frederic Weisbecker , Thomas Gleixner , "Paul E. McKenney" , Christoph Lameter , Viresh Kumar , Catalin Marinas , Will Deacon , , , CC: Chris Metcalf Subject: [PATCH v4 2/5] nohz: support PR_CPU_ISOLATED_STRICT mode Date: Mon, 13 Jul 2015 15:57:58 -0400 Message-ID: <1436817481-8732-3-git-send-email-cmetcalf@ezchip.com> X-Mailer: git-send-email 2.1.2 In-Reply-To: <1436817481-8732-1-git-send-email-cmetcalf@ezchip.com> References: <1436817481-8732-1-git-send-email-cmetcalf@ezchip.com> X-EOPAttributedMessage: 0 X-Microsoft-Exchange-Diagnostics: 1;DB3FFO11OLC001;1:i+ll54BfVrUiKZgFeNaGF20RxiqRYD++M2etvu5VtlXrOFwsfNzmdhL1KgjW6lmnpTn1mW7cZIDnx6k6ea/pfpvDRm8i/R/0esWnsDrMhXQETfQjMbX7QnedH3Wkfb2OeVGaXKmWh3j+/d7fKpyS67t6bDydqNEx7TJoyoGM+7JBby2l4wCrXuCtso4Mh5VOUYgBjMZahzzNNACAaesZb/aE8FRRpVXL6+mB/EVLI8IcPUqFsF4F5srDKRIC5+WJ7Fzg6jmfX0KUm6upein5nEE0SaUnpFww7CeUK7pVdcqmifhMacAArAlgvwC8Q/V3ZL3PHbE+UTUUHEQNT1L9UXS9pUrzT80OUKvU2s8NPGY5/qTJVig0jOVSNAY7SbHRXknEy1uqcE3sYMRS6RfmoQ== X-Forefront-Antispam-Report: CIP:12.216.194.146;CTRY:US;IPV:NLI;EFV:NLI;SFV:NSPM;SFS:(10009020)(6009001)(2980300002)(339900001)(189002)(199003)(62966003)(42186005)(189998001)(6806004)(19580395003)(85426001)(48376002)(50466002)(5003940100001)(2201001)(86362001)(19580405001)(92566002)(2950100001)(87936001)(50986999)(76176999)(77156002)(107886002)(105606002)(104016003)(5001960100002)(106466001)(5001770100001)(36756003)(46102003)(229853001)(33646002)(47776003)(50226001)(921003)(1121003)(4001430100001);DIR:OUT;SFP:1101;SCL:1;SRVR:AM2PR02MB0419;H:ld-1.internal.tilera.com;FPR:;SPF:Fail;MLV:sfv;MX:1;A:1;LANG:en; MIME-Version: 1.0 Content-Type: text/plain X-Microsoft-Exchange-Diagnostics: 1;AM2PR02MB0419;2:AYw2eKE8hCZWob4VyJG/nVadXItYmc52IskQ7AIV5ReEMw06FemKvtIXZaTTlQb0;3:BLxct5iooQIYPnJFMjbKaAHahO/DyuFJU1s+7wkQ0yx9Eutz/qx60uFsmXQxaiC7wQLXU1CAXqlYKzfrQc38vFV8GBnHR0x/pURFKYDmbWhwisd82hIOgiQKW9P5SqRlQlTY7W4LQjkHNkp0BTc14UXdbvRjWkD+IL5/rnd6UjNQfRrDqBQPEOEDMFA5pDEhDOkDS+FXm24kXwDWoNCqBsnO0W4+KuXxeT9FHZ6QZrHoD6aw0fXSgvqG89+53KUh;25:cFIz4+SgwitJf0IdDG16yX0mg8eLwAlBstnFVMHHNbfSMstLjh1NO+D0CjIcalz6J/F+f9ktYmR/ZPtYeuq0jiRfZsKJXVVqlqlxdDY/TOaHG/oTjsELtr9DJdxJHwr4VXeqnukaKoLeX4+51NIur19ZC4AQ/9+F2q4vLRJovf2Fg9PbFpiDLzjXsUS0XQzcQyaK0ekobbebQ71fii3eTAb5hq/Mi3/djWEM6QFiOP5g44zWQ7/SVpUQQ6Yju/Byej58Rclya0A3cbPEf+spCg==;20:/1q5W3aMVvd6N02sIe4MZOrBUuxi+3rHOt+AwXfEqXNgAk5141+LhE2icNh2adKmYPfxVkoNf4aMZoSVppkOLt3ZmjtaslhsVNsBrL8FyXNUH6Qt3zs2ptKo7Xf0PQiFmVGw9z8wIPcyhbRH5azzY8cTccFyKXPC8hAgIEi7GvM= X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:AM2PR02MB0419; AM2PR02MB0419: X-MS-Exchange-Organization-RulesExecuted X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(601004)(5005006)(3002001);SRVR:AM2PR02MB0419;BCL:0;PCL:0;RULEID:;SRVR:AM2PR02MB0419; X-Microsoft-Exchange-Diagnostics: 1;AM2PR02MB0419;4:C/tPp4B/p5ypbaxZOGBNq7hO8/RMAbNJshRIRF49F6e5xIrOL4wmdd08Vp9amPZY67lCRn8aMAcJnDBpOhyAdBByFBabicD5/aUHp5l3dfAEo3Fk8755rFzRzwGqtMkeHPxthsaorstRBj5ckFT0vgL5VPKxEtAVR2IFI2jPZ9PlXejmMyJaw34FZzdVLMVNDVuE0B0LEKaaT7536yJKMqDVAk6B66TXvBjZt6bHLreeuW1PqDGXu0XyUvhHvq1EoxASvFdGs6c1dm9ypym/OmC83Wt4kL2xdkIBDpMzLec= X-Forefront-PRVS: 0636271852 X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1;AM2PR02MB0419;23:8i+1XDyjuWUcIwy4zCo57sPBYFTSF/95YM4Ul0DNz?= =?us-ascii?Q?sFUNb4u7zEi8IrWq0v065tafKTbakoPXMIlhRlBh3SWsKulWYPfkUpwjsI9W?= =?us-ascii?Q?AmISAxuaFI3Wu3r+HMBPUhrAgOs8He2MuB9+E/MgLVgYNhiMjnJS/dl6c+95?= =?us-ascii?Q?+pBzlMaWPDV3lzvnETeXSF5CMrO8r/QLeq9fuH6PUJn+iqe/Qz+KA22JM+sY?= =?us-ascii?Q?b2QOqKKVCwCsEq3W1b2qyV69qcMlciFWL7bcq2FAwt1u8XhNTz1btuaDDtsK?= =?us-ascii?Q?4JalTa860gKkc+oWDY01w5SHdN0d1ZS27gzngvn44g68CIIaOSW8AhBCI2lV?= =?us-ascii?Q?nNypKwgkBzyKt6YgGMKkyk28jMleZUd8OKVnbu2/yFoJ298wCZxLTpWfh7Xx?= =?us-ascii?Q?7Yt+3ATwagZLISpcXYh1aQyXcudWcQH0nKsEVVeqrvmDW/qa/Bm1+zYfaiIX?= =?us-ascii?Q?NXlf3jo5MFJiC5627svsY4URzan6YQFfDHyqHnW/oxyRGr54PpQ2A4O44Yka?= =?us-ascii?Q?NZDVCk58BT2dw6JKsxHPuVcS/ANpfnk5zXE87lKFjM6nbSXNejgwRA1ZH6lS?= =?us-ascii?Q?utaAE+L6B2uX1TmevTu4vLrD0xbTdKfdD5dVL2TMlD3H/24AmScthhp6NBI7?= =?us-ascii?Q?47sotlXaH6kEqdqwp0abz89rCHqYoG5HHPy3TgdC2JIefhuSiVf7fRl8V/SP?= =?us-ascii?Q?V+Quu3KN0Dt+7DpuZhcxc53K3wzZOTF7aVVO8l7c6S75wIF0PRfQFbEa6g9j?= =?us-ascii?Q?cUbZVJgsRacPAG5ZFrG31qzZvxoy6HXDg7GO5JLhG3yWfunRe3N5hJfOQLrO?= =?us-ascii?Q?iIgiKpd7gfjMqq45lNIBKW/nEwjxEvaqawJLBALDt65/py+eGqup19olMGf5?= =?us-ascii?Q?M/krefLGdD9kJPCt0EMMGXY+UYedPj5Tuhegkmy5J0VgdgjDf1cq2YVZHMla?= =?us-ascii?Q?RZdfFVj5/L8DqmFRtCwAQOVy4AFpJhtwGB+j167KK+dyA+Kqy+CtB+Auwjuy?= =?us-ascii?Q?3jUmDI4bVHnU6dksuu2rq91?= X-Microsoft-Exchange-Diagnostics: 1;AM2PR02MB0419;5:qhh3cQOQeZxC0dBwqX72wa3and/kTfJCfLEkUvUxrFANSJ5Z8WjySSyufIgdgf1YpEBMngxEjIfRR0zXXznYsicDGpUXT1Vf7J6GvflhMx/iUQMDGmq+BEBiVC4z1bvrHqeaYeuM2hA6LwZF4VaZDw==;24:9KU4L884d/SkPujdzMmoAtr2cK24Lq/jC5YirLE1UcuNBbSoQ677sWvVyPUlqgiU5d9ane1R7VPHjwcDdj7qHHEaFkO7gYLHPu9TX1QrllE=;20:fQE0vA3fx8n0G1w8L6YFFNTtKgysnefTpG5zcamPVKx7zCfwPShfvMKjUCpISiN7s7hfbVHKofJgecG6Vm+t0A== SpamDiagnosticOutput: 1:23 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: ezchip.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Jul 2015 19:58:18.2719 (UTC) X-MS-Exchange-CrossTenant-Id: 0fc16e0a-3cd3-4092-8b2f-0a42cff122c3 X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=0fc16e0a-3cd3-4092-8b2f-0a42cff122c3;Ip=[12.216.194.146];Helo=[ld-1.internal.tilera.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM2PR02MB0419 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org With cpu_isolated mode, the task is in principle guaranteed not to be interrupted by the kernel, but only if it behaves. In particular, if it enters the kernel via system call, page fault, or any of a number of other synchronous traps, it may be unexpectedly exposed to long latencies. Add a simple flag that puts the process into a state where any such kernel entry is fatal. To allow the state to be entered and exited, we add an internal bit to current->cpu_isolated_flags that is set when prctl() sets the flags. We check the bit on syscall entry as well as on any exception_enter(). The prctl() syscall is ignored to allow clearing the bit again later, and exit/exit_group are ignored to allow exiting the task without a pointless signal killing you as you try to do so. This change adds the syscall-detection hooks only for x86, arm64, and tile. The signature of context_tracking_exit() changes to report whether we, in fact, are exiting back to user space, so that we can track user exceptions properly separately from other kernel entries. Signed-off-by: Chris Metcalf --- arch/arm64/kernel/ptrace.c | 4 ++++ arch/tile/kernel/ptrace.c | 6 +++++- arch/x86/kernel/ptrace.c | 2 ++ include/linux/context_tracking.h | 11 ++++++++--- include/linux/tick.h | 16 ++++++++++++++++ include/uapi/linux/prctl.h | 1 + kernel/context_tracking.c | 9 ++++++--- kernel/time/tick-sched.c | 38 ++++++++++++++++++++++++++++++++++++++ 8 files changed, 80 insertions(+), 7 deletions(-) diff --git a/arch/arm64/kernel/ptrace.c b/arch/arm64/kernel/ptrace.c index d882b833dbdb..7315b1579cbd 100644 --- a/arch/arm64/kernel/ptrace.c +++ b/arch/arm64/kernel/ptrace.c @@ -1150,6 +1150,10 @@ static void tracehook_report_syscall(struct pt_regs *regs, asmlinkage int syscall_trace_enter(struct pt_regs *regs) { + /* Ensure we report cpu_isolated violations in all circumstances. */ + if (test_thread_flag(TIF_NOHZ) && tick_nohz_cpu_isolated_strict()) + tick_nohz_cpu_isolated_syscall(regs->syscallno); + /* Do the secure computing check first; failures should be fast. */ if (secure_computing() == -1) return -1; diff --git a/arch/tile/kernel/ptrace.c b/arch/tile/kernel/ptrace.c index f84eed8243da..d4e43a13bab1 100644 --- a/arch/tile/kernel/ptrace.c +++ b/arch/tile/kernel/ptrace.c @@ -259,8 +259,12 @@ int do_syscall_trace_enter(struct pt_regs *regs) * If TIF_NOHZ is set, we are required to call user_exit() before * doing anything that could touch RCU. */ - if (work & _TIF_NOHZ) + if (work & _TIF_NOHZ) { user_exit(); + if (tick_nohz_cpu_isolated_strict()) + tick_nohz_cpu_isolated_syscall( + regs->regs[TREG_SYSCALL_NR]); + } if (work & _TIF_SYSCALL_TRACE) { if (tracehook_report_syscall_entry(regs)) diff --git a/arch/x86/kernel/ptrace.c b/arch/x86/kernel/ptrace.c index 9be72bc3613f..860f346977e2 100644 --- a/arch/x86/kernel/ptrace.c +++ b/arch/x86/kernel/ptrace.c @@ -1479,6 +1479,8 @@ unsigned long syscall_trace_enter_phase1(struct pt_regs *regs, u32 arch) if (work & _TIF_NOHZ) { user_exit(); work &= ~_TIF_NOHZ; + if (tick_nohz_cpu_isolated_strict()) + tick_nohz_cpu_isolated_syscall(regs->orig_ax); } #ifdef CONFIG_SECCOMP diff --git a/include/linux/context_tracking.h b/include/linux/context_tracking.h index b96bd299966f..8b994e2a0330 100644 --- a/include/linux/context_tracking.h +++ b/include/linux/context_tracking.h @@ -3,6 +3,7 @@ #include #include +#include #include #include @@ -11,7 +12,7 @@ extern void context_tracking_cpu_set(int cpu); extern void context_tracking_enter(enum ctx_state state); -extern void context_tracking_exit(enum ctx_state state); +extern bool context_tracking_exit(enum ctx_state state); extern void context_tracking_user_enter(void); extern void context_tracking_user_exit(void); @@ -35,8 +36,12 @@ static inline enum ctx_state exception_enter(void) return 0; prev_ctx = this_cpu_read(context_tracking.state); - if (prev_ctx != CONTEXT_KERNEL) - context_tracking_exit(prev_ctx); + if (prev_ctx != CONTEXT_KERNEL) { + if (context_tracking_exit(prev_ctx)) { + if (tick_nohz_cpu_isolated_strict()) + tick_nohz_cpu_isolated_exception(); + } + } return prev_ctx; } diff --git a/include/linux/tick.h b/include/linux/tick.h index cb5569181359..f79f6945f762 100644 --- a/include/linux/tick.h +++ b/include/linux/tick.h @@ -157,6 +157,8 @@ extern void tick_nohz_full_kick_cpu(int cpu); extern void tick_nohz_full_kick_all(void); extern void __tick_nohz_task_switch(struct task_struct *tsk); extern void tick_nohz_cpu_isolated_enter(void); +extern void tick_nohz_cpu_isolated_syscall(int nr); +extern void tick_nohz_cpu_isolated_exception(void); #else static inline bool tick_nohz_full_enabled(void) { return false; } static inline bool tick_nohz_full_cpu(int cpu) { return false; } @@ -168,6 +170,8 @@ static inline void tick_nohz_full_kick_all(void) { } static inline void __tick_nohz_task_switch(struct task_struct *tsk) { } static inline bool tick_nohz_is_cpu_isolated(void) { return false; } static inline void tick_nohz_cpu_isolated_enter(void) { } +static inline void tick_nohz_cpu_isolated_syscall(int nr) { } +static inline void tick_nohz_cpu_isolated_exception(void) { } #endif static inline bool is_housekeeping_cpu(int cpu) @@ -200,4 +204,16 @@ static inline void tick_nohz_task_switch(struct task_struct *tsk) __tick_nohz_task_switch(tsk); } +static inline bool tick_nohz_cpu_isolated_strict(void) +{ +#ifdef CONFIG_NO_HZ_FULL + if (tick_nohz_full_cpu(smp_processor_id()) && + (current->cpu_isolated_flags & + (PR_CPU_ISOLATED_ENABLE | PR_CPU_ISOLATED_STRICT)) == + (PR_CPU_ISOLATED_ENABLE | PR_CPU_ISOLATED_STRICT)) + return true; +#endif + return false; +} + #endif diff --git a/include/uapi/linux/prctl.h b/include/uapi/linux/prctl.h index edb40b6b84db..0c11238a84fb 100644 --- a/include/uapi/linux/prctl.h +++ b/include/uapi/linux/prctl.h @@ -194,5 +194,6 @@ struct prctl_mm_map { #define PR_SET_CPU_ISOLATED 47 #define PR_GET_CPU_ISOLATED 48 # define PR_CPU_ISOLATED_ENABLE (1 << 0) +# define PR_CPU_ISOLATED_STRICT (1 << 1) #endif /* _LINUX_PRCTL_H */ diff --git a/kernel/context_tracking.c b/kernel/context_tracking.c index f9de3ee12723..fd051ea290ee 100644 --- a/kernel/context_tracking.c +++ b/kernel/context_tracking.c @@ -147,15 +147,16 @@ NOKPROBE_SYMBOL(context_tracking_user_enter); * This call supports re-entrancy. This way it can be called from any exception * handler without needing to know if we came from userspace or not. */ -void context_tracking_exit(enum ctx_state state) +bool context_tracking_exit(enum ctx_state state) { unsigned long flags; + bool from_user = false; if (!context_tracking_is_enabled()) - return; + return false; if (in_interrupt()) - return; + return false; local_irq_save(flags); if (!context_tracking_recursion_enter()) @@ -169,6 +170,7 @@ void context_tracking_exit(enum ctx_state state) */ rcu_user_exit(); if (state == CONTEXT_USER) { + from_user = true; vtime_user_exit(current); trace_user_exit(0); } @@ -178,6 +180,7 @@ void context_tracking_exit(enum ctx_state state) context_tracking_recursion_exit(); out_irq_restore: local_irq_restore(flags); + return from_user; } NOKPROBE_SYMBOL(context_tracking_exit); EXPORT_SYMBOL_GPL(context_tracking_exit); diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c index 4cf093c012d1..9f495c7c7dc2 100644 --- a/kernel/time/tick-sched.c +++ b/kernel/time/tick-sched.c @@ -27,6 +27,7 @@ #include #include +#include #include "tick-internal.h" @@ -446,6 +447,43 @@ void tick_nohz_cpu_isolated_enter(void) } } +static void kill_cpu_isolated_strict_task(void) +{ + dump_stack(); + current->cpu_isolated_flags &= ~PR_CPU_ISOLATED_ENABLE; + send_sig(SIGKILL, current, 1); +} + +/* + * This routine is called from syscall entry (with the syscall number + * passed in) if the _STRICT flag is set. + */ +void tick_nohz_cpu_isolated_syscall(int syscall) +{ + /* Ignore prctl() syscalls or any task exit. */ + switch (syscall) { + case __NR_prctl: + case __NR_exit: + case __NR_exit_group: + return; + } + + pr_warn("%s/%d: cpu_isolated strict mode violated by syscall %d\n", + current->comm, current->pid, syscall); + kill_cpu_isolated_strict_task(); +} + +/* + * This routine is called from any userspace exception if the _STRICT + * flag is set. + */ +void tick_nohz_cpu_isolated_exception(void) +{ + pr_warn("%s/%d: cpu_isolated strict mode violated by exception\n", + current->comm, current->pid); + kill_cpu_isolated_strict_task(); +} + #endif /* -- 2.1.2 From mboxrd@z Thu Jan 1 00:00:00 1970 From: Chris Metcalf Subject: [PATCH v4 2/5] nohz: support PR_CPU_ISOLATED_STRICT mode Date: Mon, 13 Jul 2015 15:57:58 -0400 Message-ID: <1436817481-8732-3-git-send-email-cmetcalf@ezchip.com> References: <1436817481-8732-1-git-send-email-cmetcalf@ezchip.com> Mime-Version: 1.0 Content-Type: text/plain Return-path: In-Reply-To: <1436817481-8732-1-git-send-email-cmetcalf@ezchip.com> Sender: linux-kernel-owner@vger.kernel.org To: Gilad Ben Yossef , Steven Rostedt , Ingo Molnar , Peter Zijlstra , Andrew Morton , Rik van Riel , Tejun Heo , Frederic Weisbecker , Thomas Gleixner , "Paul E. McKenney" , Christoph Lameter , Viresh Kumar , Catalin Marinas , Will Deacon , linux-doc@vger.kernel.org, linux-api@vger.kernel.org, linux-kernel@vger.kernel.org Cc: Chris Metcalf List-Id: linux-api@vger.kernel.org With cpu_isolated mode, the task is in principle guaranteed not to be interrupted by the kernel, but only if it behaves. In particular, if it enters the kernel via system call, page fault, or any of a number of other synchronous traps, it may be unexpectedly exposed to long latencies. Add a simple flag that puts the process into a state where any such kernel entry is fatal. To allow the state to be entered and exited, we add an internal bit to current->cpu_isolated_flags that is set when prctl() sets the flags. We check the bit on syscall entry as well as on any exception_enter(). The prctl() syscall is ignored to allow clearing the bit again later, and exit/exit_group are ignored to allow exiting the task without a pointless signal killing you as you try to do so. This change adds the syscall-detection hooks only for x86, arm64, and tile. The signature of context_tracking_exit() changes to report whether we, in fact, are exiting back to user space, so that we can track user exceptions properly separately from other kernel entries. Signed-off-by: Chris Metcalf --- arch/arm64/kernel/ptrace.c | 4 ++++ arch/tile/kernel/ptrace.c | 6 +++++- arch/x86/kernel/ptrace.c | 2 ++ include/linux/context_tracking.h | 11 ++++++++--- include/linux/tick.h | 16 ++++++++++++++++ include/uapi/linux/prctl.h | 1 + kernel/context_tracking.c | 9 ++++++--- kernel/time/tick-sched.c | 38 ++++++++++++++++++++++++++++++++++++++ 8 files changed, 80 insertions(+), 7 deletions(-) diff --git a/arch/arm64/kernel/ptrace.c b/arch/arm64/kernel/ptrace.c index d882b833dbdb..7315b1579cbd 100644 --- a/arch/arm64/kernel/ptrace.c +++ b/arch/arm64/kernel/ptrace.c @@ -1150,6 +1150,10 @@ static void tracehook_report_syscall(struct pt_regs *regs, asmlinkage int syscall_trace_enter(struct pt_regs *regs) { + /* Ensure we report cpu_isolated violations in all circumstances. */ + if (test_thread_flag(TIF_NOHZ) && tick_nohz_cpu_isolated_strict()) + tick_nohz_cpu_isolated_syscall(regs->syscallno); + /* Do the secure computing check first; failures should be fast. */ if (secure_computing() == -1) return -1; diff --git a/arch/tile/kernel/ptrace.c b/arch/tile/kernel/ptrace.c index f84eed8243da..d4e43a13bab1 100644 --- a/arch/tile/kernel/ptrace.c +++ b/arch/tile/kernel/ptrace.c @@ -259,8 +259,12 @@ int do_syscall_trace_enter(struct pt_regs *regs) * If TIF_NOHZ is set, we are required to call user_exit() before * doing anything that could touch RCU. */ - if (work & _TIF_NOHZ) + if (work & _TIF_NOHZ) { user_exit(); + if (tick_nohz_cpu_isolated_strict()) + tick_nohz_cpu_isolated_syscall( + regs->regs[TREG_SYSCALL_NR]); + } if (work & _TIF_SYSCALL_TRACE) { if (tracehook_report_syscall_entry(regs)) diff --git a/arch/x86/kernel/ptrace.c b/arch/x86/kernel/ptrace.c index 9be72bc3613f..860f346977e2 100644 --- a/arch/x86/kernel/ptrace.c +++ b/arch/x86/kernel/ptrace.c @@ -1479,6 +1479,8 @@ unsigned long syscall_trace_enter_phase1(struct pt_regs *regs, u32 arch) if (work & _TIF_NOHZ) { user_exit(); work &= ~_TIF_NOHZ; + if (tick_nohz_cpu_isolated_strict()) + tick_nohz_cpu_isolated_syscall(regs->orig_ax); } #ifdef CONFIG_SECCOMP diff --git a/include/linux/context_tracking.h b/include/linux/context_tracking.h index b96bd299966f..8b994e2a0330 100644 --- a/include/linux/context_tracking.h +++ b/include/linux/context_tracking.h @@ -3,6 +3,7 @@ #include #include +#include #include #include @@ -11,7 +12,7 @@ extern void context_tracking_cpu_set(int cpu); extern void context_tracking_enter(enum ctx_state state); -extern void context_tracking_exit(enum ctx_state state); +extern bool context_tracking_exit(enum ctx_state state); extern void context_tracking_user_enter(void); extern void context_tracking_user_exit(void); @@ -35,8 +36,12 @@ static inline enum ctx_state exception_enter(void) return 0; prev_ctx = this_cpu_read(context_tracking.state); - if (prev_ctx != CONTEXT_KERNEL) - context_tracking_exit(prev_ctx); + if (prev_ctx != CONTEXT_KERNEL) { + if (context_tracking_exit(prev_ctx)) { + if (tick_nohz_cpu_isolated_strict()) + tick_nohz_cpu_isolated_exception(); + } + } return prev_ctx; } diff --git a/include/linux/tick.h b/include/linux/tick.h index cb5569181359..f79f6945f762 100644 --- a/include/linux/tick.h +++ b/include/linux/tick.h @@ -157,6 +157,8 @@ extern void tick_nohz_full_kick_cpu(int cpu); extern void tick_nohz_full_kick_all(void); extern void __tick_nohz_task_switch(struct task_struct *tsk); extern void tick_nohz_cpu_isolated_enter(void); +extern void tick_nohz_cpu_isolated_syscall(int nr); +extern void tick_nohz_cpu_isolated_exception(void); #else static inline bool tick_nohz_full_enabled(void) { return false; } static inline bool tick_nohz_full_cpu(int cpu) { return false; } @@ -168,6 +170,8 @@ static inline void tick_nohz_full_kick_all(void) { } static inline void __tick_nohz_task_switch(struct task_struct *tsk) { } static inline bool tick_nohz_is_cpu_isolated(void) { return false; } static inline void tick_nohz_cpu_isolated_enter(void) { } +static inline void tick_nohz_cpu_isolated_syscall(int nr) { } +static inline void tick_nohz_cpu_isolated_exception(void) { } #endif static inline bool is_housekeeping_cpu(int cpu) @@ -200,4 +204,16 @@ static inline void tick_nohz_task_switch(struct task_struct *tsk) __tick_nohz_task_switch(tsk); } +static inline bool tick_nohz_cpu_isolated_strict(void) +{ +#ifdef CONFIG_NO_HZ_FULL + if (tick_nohz_full_cpu(smp_processor_id()) && + (current->cpu_isolated_flags & + (PR_CPU_ISOLATED_ENABLE | PR_CPU_ISOLATED_STRICT)) == + (PR_CPU_ISOLATED_ENABLE | PR_CPU_ISOLATED_STRICT)) + return true; +#endif + return false; +} + #endif diff --git a/include/uapi/linux/prctl.h b/include/uapi/linux/prctl.h index edb40b6b84db..0c11238a84fb 100644 --- a/include/uapi/linux/prctl.h +++ b/include/uapi/linux/prctl.h @@ -194,5 +194,6 @@ struct prctl_mm_map { #define PR_SET_CPU_ISOLATED 47 #define PR_GET_CPU_ISOLATED 48 # define PR_CPU_ISOLATED_ENABLE (1 << 0) +# define PR_CPU_ISOLATED_STRICT (1 << 1) #endif /* _LINUX_PRCTL_H */ diff --git a/kernel/context_tracking.c b/kernel/context_tracking.c index f9de3ee12723..fd051ea290ee 100644 --- a/kernel/context_tracking.c +++ b/kernel/context_tracking.c @@ -147,15 +147,16 @@ NOKPROBE_SYMBOL(context_tracking_user_enter); * This call supports re-entrancy. This way it can be called from any exception * handler without needing to know if we came from userspace or not. */ -void context_tracking_exit(enum ctx_state state) +bool context_tracking_exit(enum ctx_state state) { unsigned long flags; + bool from_user = false; if (!context_tracking_is_enabled()) - return; + return false; if (in_interrupt()) - return; + return false; local_irq_save(flags); if (!context_tracking_recursion_enter()) @@ -169,6 +170,7 @@ void context_tracking_exit(enum ctx_state state) */ rcu_user_exit(); if (state == CONTEXT_USER) { + from_user = true; vtime_user_exit(current); trace_user_exit(0); } @@ -178,6 +180,7 @@ void context_tracking_exit(enum ctx_state state) context_tracking_recursion_exit(); out_irq_restore: local_irq_restore(flags); + return from_user; } NOKPROBE_SYMBOL(context_tracking_exit); EXPORT_SYMBOL_GPL(context_tracking_exit); diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c index 4cf093c012d1..9f495c7c7dc2 100644 --- a/kernel/time/tick-sched.c +++ b/kernel/time/tick-sched.c @@ -27,6 +27,7 @@ #include #include +#include #include "tick-internal.h" @@ -446,6 +447,43 @@ void tick_nohz_cpu_isolated_enter(void) } } +static void kill_cpu_isolated_strict_task(void) +{ + dump_stack(); + current->cpu_isolated_flags &= ~PR_CPU_ISOLATED_ENABLE; + send_sig(SIGKILL, current, 1); +} + +/* + * This routine is called from syscall entry (with the syscall number + * passed in) if the _STRICT flag is set. + */ +void tick_nohz_cpu_isolated_syscall(int syscall) +{ + /* Ignore prctl() syscalls or any task exit. */ + switch (syscall) { + case __NR_prctl: + case __NR_exit: + case __NR_exit_group: + return; + } + + pr_warn("%s/%d: cpu_isolated strict mode violated by syscall %d\n", + current->comm, current->pid, syscall); + kill_cpu_isolated_strict_task(); +} + +/* + * This routine is called from any userspace exception if the _STRICT + * flag is set. + */ +void tick_nohz_cpu_isolated_exception(void) +{ + pr_warn("%s/%d: cpu_isolated strict mode violated by exception\n", + current->comm, current->pid); + kill_cpu_isolated_strict_task(); +} + #endif /* -- 2.1.2