From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S934217AbbI1PR6 (ORCPT ); Mon, 28 Sep 2015 11:17:58 -0400 Received: from mail-db3on0055.outbound.protection.outlook.com ([157.55.234.55]:31158 "EHLO emea01-db3-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S934024AbbI1PRv (ORCPT ); Mon, 28 Sep 2015 11:17:51 -0400 Authentication-Results: spf=fail (sender IP is 12.216.194.146) smtp.mailfrom=ezchip.com; ezchip.com; dkim=none (message not signed) header.d=none;ezchip.com; dmarc=none action=none header.from=ezchip.com; From: Chris Metcalf To: Gilad Ben Yossef , Steven Rostedt , Ingo Molnar , Peter Zijlstra , Andrew Morton , "Rik van Riel" , Tejun Heo , Frederic Weisbecker , Thomas Gleixner , "Paul E. McKenney" , Christoph Lameter , Viresh Kumar , Catalin Marinas , Will Deacon , Andy Lutomirski , , , CC: Chris Metcalf Subject: [PATCH v7 02/11] task_isolation: add initial support Date: Mon, 28 Sep 2015 11:17:17 -0400 Message-ID: <1443453446-7827-3-git-send-email-cmetcalf@ezchip.com> X-Mailer: git-send-email 2.1.2 In-Reply-To: <1443453446-7827-1-git-send-email-cmetcalf@ezchip.com> References: <1443453446-7827-1-git-send-email-cmetcalf@ezchip.com> X-EOPAttributedMessage: 0 X-Microsoft-Exchange-Diagnostics: 1;DB3FFO11FD022;1:JSdyILNvv8Z+NbmgqHX8439SMFDeMld184hXiwoQsEEl8mONeS7YjIsdjfy6P1FXAztGKTOxaIcTgbXDVFdypb+N6jUqcGq8zTPUTm0mxuKp4urVxM+LvdY4sN+TsdQCMG0PAf/MYiXeSOmlfVT+CLVoTAjX8TzOS4ZzMfFPcUU4YBbEvCm8b463LaGAiO9+M+T/CTRM03wYpUAA6TZFJZNDyrLJl+47vWuCivV3xISHL++DZ/NI+g29NHzcoO/jFc2PJEdiFE8xjOC5UqYDb5XL7ZDkWwZXxMC4RyvYSPaIeU3Q0mO0gJueGGKQxvWxgyUrpEbp4nFpezBgfcwjuB9TVDh5ZMWbXLeiemx5jzs= X-Forefront-Antispam-Report: CIP:12.216.194.146;CTRY:US;IPV:NLI;EFV:NLI;SFV:NSPM;SFS:(10009020)(6009001)(2980300002)(1109001)(1110001)(339900001)(199003)(189002)(64706001)(33646002)(2201001)(19580405001)(5001770100001)(5001860100001)(97736004)(50226001)(47776003)(92566002)(19580395003)(11100500001)(86362001)(5007970100001)(5001830100001)(48376002)(104016004)(50466002)(5003940100001)(77156002)(46102003)(4001540100001)(87936001)(105606002)(81156007)(50986999)(6806005)(76176999)(2950100001)(107886002)(5001960100002)(106476002)(106466001)(62966003)(189998001)(68736005)(106356001)(229853001)(36756003)(85426001)(42186005)(9376005)(921003)(2101003)(4001430100001)(83996005)(1121003);DIR:OUT;SFP:1101;SCL:1;SRVR:VI1PR02MB0783;H:ld-1.internal.tilera.com;FPR:;SPF:Fail;PTR:InfoNoRecords;A:1;MX:1;LANG:en; MIME-Version: 1.0 Content-Type: text/plain X-Microsoft-Exchange-Diagnostics: 1;VI1PR02MB0783;2:ztwBu3ASQv0aesDngbrfIEYGHwhRDOUKBKnWUpJdW/RmilxF2xw4xKd2I/tA6W3TOVYE0/HytzVTDaXJrS6tMyYd/akz+gb2gGrezsKVly+AAjzItXysUJMOT6odH/w3p/B9+2rIrHxRQhne3SUwQGwDCKrq1NDbgygXjI+0m6M=;3:VxhLGfsqihp5eJkehMB4RBG9XtQO22NSXsdR7mQvMA2r+p8CoapY5NReRiwxvwKRvIw4x6GKmc6+jQ0hXfI+cDQ9BF/g9Rsg96a+zHETxm6WjvMwuDaA9Uaj4RRsoEWyDtwDNCV2siZd6s0fWJbmgInXcKTbFc8tgrzFdIsH72zee48h7j08trw6F7GVJkKFoNaklbhdafTof8DT31IBJsBzM6Yf/jgvrLjhzO/yZPAiHrT7ArGyfNL6zzG2hQx9;25:unYZTRDQ+7/52Ii4rOJ3ylRwqUE3pHWFFvq7CU7CFuH8ObwFpH1Kw2ozgLrty7VmrZ7Ax0awX9jRmiwvFUjl1b2Zct9ehMNOqfrN+DCR3nUaQ863Ew3dBJ3X+9DTNvsfu9zBu/rJTVLxRJCLnAdgswwAEOlkUwgOG41juOOw/JKVk5XvOO3xg6+u0pDn9hCoWa/HliMJ7Q8PGl1gfw1BK6csO8FNZFlu3p+ZUnV33X75LvMMr+9T+Nul7qxmWx2dVxRrHClOgNQtiFN9IX1Qtg==;20:+YcZQg+cX+Hzf4U6Yh/fIv//swkfhiq9p8sGfNgmw/MT5pEn12wmdf0rXbz5biIfoZETc4MPddB0VNqsZdwzAE9BamrPCyDAEvXr9V8bn3Wv/7GNySXKvPRL//5nlnu4ciSNRjfS2JAvW55z9Exnl7JuhjDwBHSIUVmyYCtneZo= X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:VI1PR02MB0783; X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(601004)(2401047)(5005006)(520078)(8121501046)(3002001);SRVR:VI1PR02MB0783;BCL:0;PCL:0;RULEID:;SRVR:VI1PR02MB0783; X-Microsoft-Exchange-Diagnostics: 1;VI1PR02MB0783;4:OYrwpAQu/tQYq0MX7CNL9YN4LoK5Q5vqede1CnNuh7MwPD+S0WO/732ne42h4vpr5NYj6/ACYX/e5QjyIW67ZCS9qUMEnCeeRJXVZ1ypoWRFgkbNzc2W1loyCahZYYd6GOOv+hFNp7LGviNnHQQUDeG6IkxcEnHHjrRv5oFXmfp1u2c2Ov+G6pFtpP2q8AYKuqUWeejTvriVo03ntv0NbwxECZ8To5NYlxy9XSQUh1E6Pfk0NICkv9DMLjYGKY7SgmkncD1b8qfBB7S3wWJamIaO6g1j4GeraB+29KT0udr13PxhH82DBM9F22+p6pvqHw6GCrACCCE8Ynh9v6C1+gUhGGsAylArehzHmJsJL0E= X-Forefront-PRVS: 0713BC207F X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1;VI1PR02MB0783;23:ebhDUZvPN0MrHAoccE92wy9lqL6L6eZ+DQnoupne6?= =?us-ascii?Q?qZVuA9GY6stLzreiWAADslZaWO+2k7pQY9BqxeRiINMC7caZ+DbA3I4Hdfoy?= =?us-ascii?Q?XIXCrx0bCDwJeyBJDPxlBfKf7a2DazvbL99rvL8jf0Hx9u6XsBD/zrjDvi3r?= =?us-ascii?Q?s4bgNvPypvveFGOLBY6nQAOuyJPSD2vRFE7OyP064IKx7qfO+RGyEiZGEYML?= =?us-ascii?Q?977/NGeiuTsSHa8R8y8iqMjDf/KN5IJBFCp2F+V1Mp/xt6a32A4HYcEUos3S?= =?us-ascii?Q?pRosSUAE2DtHwIWVFt9Nw3llLe5RRWDnSJZmzmGmQuwx5adbG/WKgBSts49E?= =?us-ascii?Q?PJkvL6MJyJTRUYSOTvuvHo3n+D+RaIIY4Q2cDsXyupJf8PDCnjm4J6AUfWiG?= =?us-ascii?Q?TLDzDOw1LtkDq7A2pEsZn/S4+6oFMjnXZ5sN5ZUGxx8tFnyeOVDLD9bsPWIR?= =?us-ascii?Q?okjVsGImMEACxX/n14O0Rsgzh/nHn8LGl7bwfGTiePkuUNG8PLTbEhxgn76x?= =?us-ascii?Q?m/X/Vr3YU5bgoH5S/YwOaTGRUNVV1NbVWlJxqUqdgO9UH2AFxg4Y3+bJY1VQ?= =?us-ascii?Q?Kw0hf+30wnJO+c03iBfRqOyI0hBSLw+hU4GAhsVrQoCdQ6csFU+UaWRpneEt?= =?us-ascii?Q?UaZ/q6fyO+WibAAdCqSPtyksdVIMSoaxywLySV/zCn6McYzH9RmpgljDFE/c?= =?us-ascii?Q?S8Ty0EdgwO+MgwlLJOxsi240oJnGqJ/3YTTH4bXpkpUzIz8AgOaTmePqcFcU?= =?us-ascii?Q?zeEgaUs7YK27uF4gHl4DPHMBJCSxq1tQjePVhIeEF1I6d44QssvMxBGmYCkQ?= =?us-ascii?Q?4kSyzoeA47K4ezI9zuBwPqIXBvk+3f9koFhhDtOc2259ZRqLFiP5Jj24aiad?= =?us-ascii?Q?YRe+Dvv/KBuVcIs0ftvryGM4IQr5ZfyPrlhfaAoJwSdcmOmR7Hw2asYkhr3p?= =?us-ascii?Q?gWxSKMWRsDw5zJmzEyNPd+EI6YmeVR0aBU8yUmwVcO4yK18QvSFPe+dQnwDd?= =?us-ascii?Q?PfLWQ5mRdvE8vKMen4JVAAb+HJEcsQS46ub9FRlKpSTNNNo3ymMn+E6hZ0R6?= =?us-ascii?Q?J0E8pxR1G9/zinM4IOs16Pz58LJ3wynkONABnmztT4UXsJq0O6VDjBGRlT40?= =?us-ascii?Q?pck8VFiXgqVMgiO6bbO3JsSDxP4hs/PEcjjQ0RbQwfJnhqNYhjSMJDEsBWwG?= =?us-ascii?Q?WW2M9lQ+6OKYV/sLCrN4hp0Nq8B/emyIYpzc3MaFUnvlDJgHUXsjk3uiewyy?= =?us-ascii?Q?UTEq8cr7D1V6EdrdSAMP12/vW8Vb8lfNosCkMQJmU4eJTYdUmQmlhn5sR4H1?= =?us-ascii?Q?rOY1/XdztW08MElwQ8AKUEST2HfGZ7wrl6lMC39QfRk5oM49+jWmV3+SsnEe?= =?us-ascii?Q?VO6jJG7ILRDrfwHR+5fvwiQCNjs7PwnSy3oBu4TXyXhv2Y1EJ3N5+19WQPTW?= =?us-ascii?Q?I517bEfxA=3D=3D?= X-Microsoft-Exchange-Diagnostics: 1;VI1PR02MB0783;5:JzUiZy/PfIKcmEe939enX9iN2wYEO7gxAgBjajlgdXULOyU5i4FkVkenGicMOuw3K43xTtZXjDb4w0GlFGM++vLWpGEB/3d+hY/s2LwHvbtWavJ+8jXQxQ7El5tuOk+q3gTmHGmvYNzhBAdIr7iUcg==;24:49LcSgEdSojshq8xOL0NTtqXX5+6h7Ph85l/gQ3JFeovynSXXzaBdyQ9yBCvFG67GSCCTrRZWR0GSkBsP3JBsFAqaUaXnbmrdz2HHffmTsI=;20:/GNYhnLWMJ9O2jqqVCKk66BW99818tKKDrbtzqwssaOrBHskbyijwy43GwngSlmamE7WfsSdyBFjokKrUwMKCg== SpamDiagnosticOutput: 1:23 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: ezchip.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 Sep 2015 15:17:44.7822 (UTC) X-MS-Exchange-CrossTenant-Id: 0fc16e0a-3cd3-4092-8b2f-0a42cff122c3 X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=0fc16e0a-3cd3-4092-8b2f-0a42cff122c3;Ip=[12.216.194.146];Helo=[ld-1.internal.tilera.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR02MB0783 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org The existing nohz_full mode is designed as a "soft" isolation mode that makes tradeoffs to minimize userspace interruptions while still attempting to avoid overheads in the kernel entry/exit path, to provide 100% kernel semantics, etc. However, some applications require a "hard" commitment from the kernel to avoid interruptions, in particular userspace device driver style applications, such as high-speed networking code. This change introduces a framework to allow applications to elect to have the "hard" semantics as needed, specifying prctl(PR_SET_TASK_ISOLATION, PR_TASK_ISOLATION_ENABLE) to do so. Subsequent commits will add additional flags and additional semantics. The kernel must be built with the new TASK_ISOLATION Kconfig flag to enable this mode, and the kernel booted with an appropriate nohz_full=CPULIST boot argument. The "task_isolation" state is then indicated by setting a new task struct field, task_isolation_flag, to the value passed by prctl(). When the _ENABLE bit is set for a task, and it is returning to userspace on a nohz_full core, it calls the new task_isolation_enter() routine to take additional actions to help the task avoid being interrupted in the future. Initially, there are only three actions taken. First, the task calls lru_add_drain() to prevent being interrupted by a subsequent lru_add_drain_all() call on another core. Then, it calls quiet_vmstat() to quieten the vmstat worker to avoid a follow-on interrupt. Finally, the code checks for pending timer interrupts and quiesces until they are no longer pending. As a result, sys calls (and page faults, etc.) can be inordinately slow. However, this quiescing guarantees that no unexpected interrupts will occur, even if the application intentionally calls into the kernel. The task_isolation_enter() routine must be called just before the hard return to userspace, so it is appropriately placed in the prepare_exit_to_usermode() routine for an individual architecture or some comparable location. Separate patches that follow provide these changes for x86, arm64, and tile. Signed-off-by: Chris Metcalf --- include/linux/isolation.h | 24 +++++++++++++++ include/linux/sched.h | 3 ++ include/uapi/linux/prctl.h | 5 +++ init/Kconfig | 20 ++++++++++++ kernel/Makefile | 1 + kernel/isolation.c | 77 ++++++++++++++++++++++++++++++++++++++++++++++ kernel/sys.c | 8 +++++ 7 files changed, 138 insertions(+) create mode 100644 include/linux/isolation.h create mode 100644 kernel/isolation.c diff --git a/include/linux/isolation.h b/include/linux/isolation.h new file mode 100644 index 000000000000..fd04011b1c1e --- /dev/null +++ b/include/linux/isolation.h @@ -0,0 +1,24 @@ +/* + * Task isolation related global functions + */ +#ifndef _LINUX_ISOLATION_H +#define _LINUX_ISOLATION_H + +#include +#include + +#ifdef CONFIG_TASK_ISOLATION +static inline bool task_isolation_enabled(void) +{ + return tick_nohz_full_cpu(smp_processor_id()) && + (current->task_isolation_flags & PR_TASK_ISOLATION_ENABLE); +} + +extern void task_isolation_enter(void); +extern void task_isolation_wait(void); +#else +static inline bool task_isolation_enabled(void) { return false; } +static inline void task_isolation_enter(void) { } +#endif + +#endif diff --git a/include/linux/sched.h b/include/linux/sched.h index a4ab9daa387c..bd2dc26948a6 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1800,6 +1800,9 @@ struct task_struct { unsigned long task_state_change; #endif int pagefault_disabled; +#ifdef CONFIG_TASK_ISOLATION + unsigned int task_isolation_flags; +#endif /* CPU-specific state of this task */ struct thread_struct thread; /* diff --git a/include/uapi/linux/prctl.h b/include/uapi/linux/prctl.h index a8d0759a9e40..67224df4b559 100644 --- a/include/uapi/linux/prctl.h +++ b/include/uapi/linux/prctl.h @@ -197,4 +197,9 @@ struct prctl_mm_map { # define PR_CAP_AMBIENT_LOWER 3 # define PR_CAP_AMBIENT_CLEAR_ALL 4 +/* Enable/disable or query task_isolation mode for NO_HZ_FULL kernels. */ +#define PR_SET_TASK_ISOLATION 48 +#define PR_GET_TASK_ISOLATION 49 +# define PR_TASK_ISOLATION_ENABLE (1 << 0) + #endif /* _LINUX_PRCTL_H */ diff --git a/init/Kconfig b/init/Kconfig index c24b6f767bf0..4ff7f052059a 100644 --- a/init/Kconfig +++ b/init/Kconfig @@ -787,6 +787,26 @@ config RCU_EXPEDITE_BOOT endmenu # "RCU Subsystem" +config TASK_ISOLATION + bool "Provide hard CPU isolation from the kernel on demand" + depends on NO_HZ_FULL + help + Allow userspace processes to place themselves on nohz_full + cores and run prctl(PR_SET_TASK_ISOLATION) to "isolate" + themselves from the kernel. On return to userspace, + isolated tasks will first arrange that no future kernel + activity will interrupt the task while the task is running + in userspace. This "hard" isolation from the kernel is + required for userspace tasks that are running hard real-time + tasks in userspace, such as a 10 Gbit network driver in userspace. + + Without this option, but with NO_HZ_FULL enabled, the kernel + will make a best-faith, "soft" effort to shield a single userspace + process from interrupts, but makes no guarantees. + + You should say "N" unless you are intending to run a + high-performance userspace driver or similar task. + config BUILD_BIN2C bool default n diff --git a/kernel/Makefile b/kernel/Makefile index 53abf008ecb3..693a2ba35679 100644 --- a/kernel/Makefile +++ b/kernel/Makefile @@ -103,6 +103,7 @@ obj-$(CONFIG_TORTURE_TEST) += torture.o obj-$(CONFIG_MEMBARRIER) += membarrier.o obj-$(CONFIG_HAS_IOMEM) += memremap.o +obj-$(CONFIG_TASK_ISOLATION) += isolation.o $(obj)/configs.o: $(obj)/config_data.h diff --git a/kernel/isolation.c b/kernel/isolation.c new file mode 100644 index 000000000000..6ace866c69f6 --- /dev/null +++ b/kernel/isolation.c @@ -0,0 +1,77 @@ +/* + * linux/kernel/isolation.c + * + * Implementation for task isolation. + * + * Distributed under GPLv2. + */ + +#include +#include +#include +#include +#include "time/tick-sched.h" + +/* + * Rather than continuously polling for the next_event in the + * tick_cpu_device, architectures can provide a method to save power + * by sleeping until an interrupt arrives. + * + * Note that it must be guaranteed for a particular architecture + * that if next_event is not KTIME_MAX, then a timer interrupt will + * occur, otherwise the sleep may never awaken. + */ +void __weak task_isolation_wait(void) +{ + cpu_relax(); +} + +/* + * We normally return immediately to userspace. + * + * In task_isolation mode we wait until no more interrupts are + * pending. Otherwise we nap with interrupts enabled and wait for the + * next interrupt to fire, then loop back and retry. + * + * Note that if you schedule two task_isolation processes on the same + * core, neither will ever leave the kernel, and one will have to be + * killed manually. Otherwise in situations where another process is + * in the runqueue on this cpu, this task will just wait for that + * other task to go idle before returning to user space. + */ +void task_isolation_enter(void) +{ + struct clock_event_device *dev = + __this_cpu_read(tick_cpu_device.evtdev); + struct task_struct *task = current; + unsigned long start = jiffies; + bool warned = false; + + if (WARN_ON(irqs_disabled())) + local_irq_enable(); + + /* Drain the pagevecs to avoid unnecessary IPI flushes later. */ + lru_add_drain(); + + /* Quieten the vmstat worker so it won't interrupt us. */ + quiet_vmstat(); + + while (READ_ONCE(dev->next_event.tv64) != KTIME_MAX) { + if (!warned && (jiffies - start) >= (5 * HZ)) { + pr_warn("%s/%d: cpu %d: task_isolation task blocked for %ld seconds\n", + task->comm, task->pid, smp_processor_id(), + (jiffies - start) / HZ); + warned = true; + } + cond_resched(); + if (test_thread_flag(TIF_SIGPENDING)) + break; + task_isolation_wait(); + } + if (warned) { + pr_warn("%s/%d: cpu %d: task_isolation task unblocked after %ld seconds\n", + task->comm, task->pid, smp_processor_id(), + (jiffies - start) / HZ); + dump_stack(); + } +} diff --git a/kernel/sys.c b/kernel/sys.c index fa2f2f671a5c..a2c6eb1d4ad9 100644 --- a/kernel/sys.c +++ b/kernel/sys.c @@ -2266,6 +2266,14 @@ SYSCALL_DEFINE5(prctl, int, option, unsigned long, arg2, unsigned long, arg3, case PR_GET_FP_MODE: error = GET_FP_MODE(me); break; +#ifdef CONFIG_TASK_ISOLATION + case PR_SET_TASK_ISOLATION: + me->task_isolation_flags = arg2; + break; + case PR_GET_TASK_ISOLATION: + error = me->task_isolation_flags; + break; +#endif default: error = -EINVAL; break; -- 2.1.2 From mboxrd@z Thu Jan 1 00:00:00 1970 From: Chris Metcalf Subject: [PATCH v7 02/11] task_isolation: add initial support Date: Mon, 28 Sep 2015 11:17:17 -0400 Message-ID: <1443453446-7827-3-git-send-email-cmetcalf@ezchip.com> References: <1443453446-7827-1-git-send-email-cmetcalf@ezchip.com> Mime-Version: 1.0 Content-Type: text/plain Return-path: In-Reply-To: <1443453446-7827-1-git-send-email-cmetcalf@ezchip.com> Sender: linux-doc-owner@vger.kernel.org To: Gilad Ben Yossef , Steven Rostedt , Ingo Molnar , Peter Zijlstra , Andrew Morton , Rik van Riel , Tejun Heo , Frederic Weisbecker , Thomas Gleixner , "Paul E. McKenney" , Christoph Lameter , Viresh Kumar , Catalin Marinas , Will Deacon , Andy Lutomirski , linux-doc@vger.kernel.org, linux-api@vger.kernel.org, linux-kernel@vger.kernel.org Cc: Chris Metcalf List-Id: linux-api@vger.kernel.org The existing nohz_full mode is designed as a "soft" isolation mode that makes tradeoffs to minimize userspace interruptions while still attempting to avoid overheads in the kernel entry/exit path, to provide 100% kernel semantics, etc. However, some applications require a "hard" commitment from the kernel to avoid interruptions, in particular userspace device driver style applications, such as high-speed networking code. This change introduces a framework to allow applications to elect to have the "hard" semantics as needed, specifying prctl(PR_SET_TASK_ISOLATION, PR_TASK_ISOLATION_ENABLE) to do so. Subsequent commits will add additional flags and additional semantics. The kernel must be built with the new TASK_ISOLATION Kconfig flag to enable this mode, and the kernel booted with an appropriate nohz_full=CPULIST boot argument. The "task_isolation" state is then indicated by setting a new task struct field, task_isolation_flag, to the value passed by prctl(). When the _ENABLE bit is set for a task, and it is returning to userspace on a nohz_full core, it calls the new task_isolation_enter() routine to take additional actions to help the task avoid being interrupted in the future. Initially, there are only three actions taken. First, the task calls lru_add_drain() to prevent being interrupted by a subsequent lru_add_drain_all() call on another core. Then, it calls quiet_vmstat() to quieten the vmstat worker to avoid a follow-on interrupt. Finally, the code checks for pending timer interrupts and quiesces until they are no longer pending. As a result, sys calls (and page faults, etc.) can be inordinately slow. However, this quiescing guarantees that no unexpected interrupts will occur, even if the application intentionally calls into the kernel. The task_isolation_enter() routine must be called just before the hard return to userspace, so it is appropriately placed in the prepare_exit_to_usermode() routine for an individual architecture or some comparable location. Separate patches that follow provide these changes for x86, arm64, and tile. Signed-off-by: Chris Metcalf --- include/linux/isolation.h | 24 +++++++++++++++ include/linux/sched.h | 3 ++ include/uapi/linux/prctl.h | 5 +++ init/Kconfig | 20 ++++++++++++ kernel/Makefile | 1 + kernel/isolation.c | 77 ++++++++++++++++++++++++++++++++++++++++++++++ kernel/sys.c | 8 +++++ 7 files changed, 138 insertions(+) create mode 100644 include/linux/isolation.h create mode 100644 kernel/isolation.c diff --git a/include/linux/isolation.h b/include/linux/isolation.h new file mode 100644 index 000000000000..fd04011b1c1e --- /dev/null +++ b/include/linux/isolation.h @@ -0,0 +1,24 @@ +/* + * Task isolation related global functions + */ +#ifndef _LINUX_ISOLATION_H +#define _LINUX_ISOLATION_H + +#include +#include + +#ifdef CONFIG_TASK_ISOLATION +static inline bool task_isolation_enabled(void) +{ + return tick_nohz_full_cpu(smp_processor_id()) && + (current->task_isolation_flags & PR_TASK_ISOLATION_ENABLE); +} + +extern void task_isolation_enter(void); +extern void task_isolation_wait(void); +#else +static inline bool task_isolation_enabled(void) { return false; } +static inline void task_isolation_enter(void) { } +#endif + +#endif diff --git a/include/linux/sched.h b/include/linux/sched.h index a4ab9daa387c..bd2dc26948a6 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1800,6 +1800,9 @@ struct task_struct { unsigned long task_state_change; #endif int pagefault_disabled; +#ifdef CONFIG_TASK_ISOLATION + unsigned int task_isolation_flags; +#endif /* CPU-specific state of this task */ struct thread_struct thread; /* diff --git a/include/uapi/linux/prctl.h b/include/uapi/linux/prctl.h index a8d0759a9e40..67224df4b559 100644 --- a/include/uapi/linux/prctl.h +++ b/include/uapi/linux/prctl.h @@ -197,4 +197,9 @@ struct prctl_mm_map { # define PR_CAP_AMBIENT_LOWER 3 # define PR_CAP_AMBIENT_CLEAR_ALL 4 +/* Enable/disable or query task_isolation mode for NO_HZ_FULL kernels. */ +#define PR_SET_TASK_ISOLATION 48 +#define PR_GET_TASK_ISOLATION 49 +# define PR_TASK_ISOLATION_ENABLE (1 << 0) + #endif /* _LINUX_PRCTL_H */ diff --git a/init/Kconfig b/init/Kconfig index c24b6f767bf0..4ff7f052059a 100644 --- a/init/Kconfig +++ b/init/Kconfig @@ -787,6 +787,26 @@ config RCU_EXPEDITE_BOOT endmenu # "RCU Subsystem" +config TASK_ISOLATION + bool "Provide hard CPU isolation from the kernel on demand" + depends on NO_HZ_FULL + help + Allow userspace processes to place themselves on nohz_full + cores and run prctl(PR_SET_TASK_ISOLATION) to "isolate" + themselves from the kernel. On return to userspace, + isolated tasks will first arrange that no future kernel + activity will interrupt the task while the task is running + in userspace. This "hard" isolation from the kernel is + required for userspace tasks that are running hard real-time + tasks in userspace, such as a 10 Gbit network driver in userspace. + + Without this option, but with NO_HZ_FULL enabled, the kernel + will make a best-faith, "soft" effort to shield a single userspace + process from interrupts, but makes no guarantees. + + You should say "N" unless you are intending to run a + high-performance userspace driver or similar task. + config BUILD_BIN2C bool default n diff --git a/kernel/Makefile b/kernel/Makefile index 53abf008ecb3..693a2ba35679 100644 --- a/kernel/Makefile +++ b/kernel/Makefile @@ -103,6 +103,7 @@ obj-$(CONFIG_TORTURE_TEST) += torture.o obj-$(CONFIG_MEMBARRIER) += membarrier.o obj-$(CONFIG_HAS_IOMEM) += memremap.o +obj-$(CONFIG_TASK_ISOLATION) += isolation.o $(obj)/configs.o: $(obj)/config_data.h diff --git a/kernel/isolation.c b/kernel/isolation.c new file mode 100644 index 000000000000..6ace866c69f6 --- /dev/null +++ b/kernel/isolation.c @@ -0,0 +1,77 @@ +/* + * linux/kernel/isolation.c + * + * Implementation for task isolation. + * + * Distributed under GPLv2. + */ + +#include +#include +#include +#include +#include "time/tick-sched.h" + +/* + * Rather than continuously polling for the next_event in the + * tick_cpu_device, architectures can provide a method to save power + * by sleeping until an interrupt arrives. + * + * Note that it must be guaranteed for a particular architecture + * that if next_event is not KTIME_MAX, then a timer interrupt will + * occur, otherwise the sleep may never awaken. + */ +void __weak task_isolation_wait(void) +{ + cpu_relax(); +} + +/* + * We normally return immediately to userspace. + * + * In task_isolation mode we wait until no more interrupts are + * pending. Otherwise we nap with interrupts enabled and wait for the + * next interrupt to fire, then loop back and retry. + * + * Note that if you schedule two task_isolation processes on the same + * core, neither will ever leave the kernel, and one will have to be + * killed manually. Otherwise in situations where another process is + * in the runqueue on this cpu, this task will just wait for that + * other task to go idle before returning to user space. + */ +void task_isolation_enter(void) +{ + struct clock_event_device *dev = + __this_cpu_read(tick_cpu_device.evtdev); + struct task_struct *task = current; + unsigned long start = jiffies; + bool warned = false; + + if (WARN_ON(irqs_disabled())) + local_irq_enable(); + + /* Drain the pagevecs to avoid unnecessary IPI flushes later. */ + lru_add_drain(); + + /* Quieten the vmstat worker so it won't interrupt us. */ + quiet_vmstat(); + + while (READ_ONCE(dev->next_event.tv64) != KTIME_MAX) { + if (!warned && (jiffies - start) >= (5 * HZ)) { + pr_warn("%s/%d: cpu %d: task_isolation task blocked for %ld seconds\n", + task->comm, task->pid, smp_processor_id(), + (jiffies - start) / HZ); + warned = true; + } + cond_resched(); + if (test_thread_flag(TIF_SIGPENDING)) + break; + task_isolation_wait(); + } + if (warned) { + pr_warn("%s/%d: cpu %d: task_isolation task unblocked after %ld seconds\n", + task->comm, task->pid, smp_processor_id(), + (jiffies - start) / HZ); + dump_stack(); + } +} diff --git a/kernel/sys.c b/kernel/sys.c index fa2f2f671a5c..a2c6eb1d4ad9 100644 --- a/kernel/sys.c +++ b/kernel/sys.c @@ -2266,6 +2266,14 @@ SYSCALL_DEFINE5(prctl, int, option, unsigned long, arg2, unsigned long, arg3, case PR_GET_FP_MODE: error = GET_FP_MODE(me); break; +#ifdef CONFIG_TASK_ISOLATION + case PR_SET_TASK_ISOLATION: + me->task_isolation_flags = arg2; + break; + case PR_GET_TASK_ISOLATION: + error = me->task_isolation_flags; + break; +#endif default: error = -EINVAL; break; -- 2.1.2