From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1756778AbcA2SS1 (ORCPT ); Fri, 29 Jan 2016 13:18:27 -0500 Received: from mail-am1on0061.outbound.protection.outlook.com ([157.56.112.61]:3440 "EHLO emea01-am1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1756742AbcA2SSS (ORCPT ); Fri, 29 Jan 2016 13:18:18 -0500 Authentication-Results: vger.kernel.org; dkim=none (message not signed) header.d=none;vger.kernel.org; dmarc=none action=none header.from=ezchip.com; Subject: Re: [PATCH v9 04/13] task_isolation: add initial support To: Frederic Weisbecker References: <1451936091-29247-1-git-send-email-cmetcalf@ezchip.com> <1451936091-29247-5-git-send-email-cmetcalf@ezchip.com> <20160119154214.GA2722@lerouge> <569EA050.5030106@ezchip.com> <20160128002801.GA14313@lerouge> CC: Gilad Ben Yossef , Steven Rostedt , Ingo Molnar , Peter Zijlstra , Andrew Morton , Rik van Riel , Tejun Heo , Thomas Gleixner , "Paul E. McKenney" , Christoph Lameter , Viresh Kumar , Catalin Marinas , Will Deacon , Andy Lutomirski , , , From: Chris Metcalf Message-ID: <56ABACDD.5090500@ezchip.com> Date: Fri, 29 Jan 2016 13:18:05 -0500 User-Agent: Mozilla/5.0 (X11; Linux i686 on x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.5.1 MIME-Version: 1.0 In-Reply-To: <20160128002801.GA14313@lerouge> Content-Type: text/plain; charset="windows-1252"; format=flowed Content-Transfer-Encoding: 7bit X-Originating-IP: [12.216.194.146] X-ClientProxiedBy: BLUPR0101CA0038.prod.exchangelabs.com (2a01:111:e400:52e8::48) To AM3PR02MB116.eurprd02.prod.outlook.com (2a01:111:e400:8807::19) X-Microsoft-Exchange-Diagnostics: 1;AM3PR02MB116;2:85/HhOXsxciFk2GDSreNkHAMYN0NL6Mt476OuMVFf8AP4Cv8hInpQHjP2M3jzE7/rJr407KRpFPbeoesbq4EHy2Xc4O4q9yHdKa9pdIjgME45sG+8Kr52eS9eVvdygMyyXyFlI9J8+FTGdp36PfVLQ==;3:UFlBvjOLgUikZCkGbNSoaPwfvpuv1fjpRgSwV0iIX622NS1bnlSokduR5VPNIThNYn1vHGC6BR+JUq9p4UF4r4/wYGu4tYKx8FaGAjLZP8O6ggYN2qpPhxKiNYUgqzpT;25:hTwbfpnobDIhvCNuttxuzY2xiEb7Vdf/lSzpsEHgdjR5aDQZrHjKBVZlECaYZ83Ox9hUAxV4jrVnx9RbyBbR7qo+QVLK6I7SG0NKq6yEZQgvEy2ijTAyb75USNpjSABw6dyz89vkR3tGtJj+4tPnTqQRKJBn+JpMQIeu9E9iIuV3rr/H51jpfbwlipOQ+/bhyinb8fO4/lALylj4Fn8WtjjV9MqEnJG7Zi4kKwhfWTui1gDWmoYWhAElLX5UdoKc;20:yxPeZtSpHwxkRyU3QI9y/Bo/mWaLRHanPJXeLPDusqk35dolaMxsLFx+pffbh5TwQGoQNQjPwPPdTJenR1jkDDRzKUMozRJ9xBR7n0T6zt4qiPrZKKMYotSS0LxrTvunkSUhWAnCV8pTaoTJuGtcwDcGU53sWIh1vGYohZNCnNc= X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:AM3PR02MB116; X-MS-Office365-Filtering-Correlation-Id: 977ad6fa-fe9e-4d9b-99ef-08d328d88eb9 X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(601004)(2401047)(8121501046)(5005006)(10201501046)(3002001);SRVR:AM3PR02MB116;BCL:0;PCL:0;RULEID:;SRVR:AM3PR02MB116; X-Microsoft-Exchange-Diagnostics: 1;AM3PR02MB116;4:P9YOJAv7Cmb0uSzgSOyITIB6Wyu0v2bVJ6g29HlrUoEF47afSMW16pS9VRAR9RhG1Lcr+U/lGsNjmhbns2qQVyifo5cKm82FgDQ1XQNHipnT+xjJjkXiKCnFWV3zM+FawW+Y8XvTCpvr9MKKapYUumXhOuAmcEMg5ncFRdLoMmVz0I+cYjEt4I6gLjEqtY1BhxWpfamMxmYF2QzmaaiktZwrMO+uwaN9VMZX7WaEZWv4P4ZZfEElA5ZLXZkvYL7L/l+E0er3NlA8zus1fSPe0qxnnu55aPsCqYQ4/FozjpETqyDbk+dYCUxqU+TZNkD9EehBzbBogoPi2HKFS7Pd+xr9dPJI9Haw7VOM2+vYrfHsG4/lnPt7fJErY9XSwum2 X-Forefront-PRVS: 083691450C X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10009020)(6049001)(6009001)(377454003)(24454002)(479174004)(43544003)(59896002)(54356999)(33656002)(64126003)(5008740100001)(36756003)(122386002)(1096002)(3470700001)(586003)(40100003)(3846002)(4326007)(92566002)(6116002)(2906002)(42186005)(47776003)(80316001)(76176999)(230700001)(110136002)(19580395003)(83506001)(65816999)(87266999)(50466002)(4001350100001)(93886004)(23746002)(65956001)(15975445007)(65806001)(77096005)(1411001)(189998001)(86362001)(5001960100002)(66066001)(5004730100002)(50986999)(2950100001)(18886065003);DIR:OUT;SFP:1101;SCL:1;SRVR:AM3PR02MB116;H:[10.7.0.41];FPR:;SPF:None;MLV:sfv;LANG:en; X-Microsoft-Exchange-Diagnostics: =?Windows-1252?Q?1;AM3PR02MB116;23:ZR+943EzUgV8CunmoYX5V8N+lk4sHd09pcm4tz?= =?Windows-1252?Q?4Ncon3QzUXhzB0UKwMUQueoqDuaxnfNs9W30Lr5DHEUn7wHtAxvxWnZW?= =?Windows-1252?Q?kn/rpRLVZ0CU4WCwHXMY7HbL1rlhOWUaoBKTPEOsHVhmtoI88Uwq80nB?= =?Windows-1252?Q?eSyEd2mmenykR7oIQFuYLJVKcsMSEN5h/Ure189WKr02xGHgk3kMMjAM?= =?Windows-1252?Q?na9CT+QjG1KfFG8R0C08HYJ3jv3xTIZjZPzb9HFuYHfej8GFSwdkyDZ5?= =?Windows-1252?Q?iF46flh1mBQgqMldq99b8MLkUvHjTnmdtMRZmOW/A/2w+sJx0a877/U/?= =?Windows-1252?Q?AA7PgNKoU6dhUqEyyeRE92pAcq1RHiwUJdv+dxgcRpPtF6dzL5ezfbMV?= =?Windows-1252?Q?HuNXrRvb2kzDibb5GXV+UhkVkzWAeMX1VOX6mRJstEHVD1gY7Uku1BYI?= =?Windows-1252?Q?daRlChHmPwI9Rh7aE4oGVfoAsjbsTiFoDfQqQm9dM0mwiwqeWj74nWge?= =?Windows-1252?Q?W0sz9YYAfaskZDle3tLjoDpNFh5xuPsYKXpA/zpPoZhr3kB0OwumlDm2?= =?Windows-1252?Q?b49z12PbNj09la5eWjwm4FmEQXodqxIgF64lmmyBIy6n1I8u4i4X4q1w?= =?Windows-1252?Q?eAyu4xiN6VawDaBuGzl6StAVYgjaAsjyD9/ZZibNgoc8iyuSyMNwxKwP?= =?Windows-1252?Q?+1wdgcIx/gYIzYAUCq9Oo0kK7eAU+lJQBfRFW6hJiweZRx2I5zUrxiF0?= =?Windows-1252?Q?k4X7ZJueVkTWyLzS2Cej+ywDrB4vRFOSuc/nxI34kCfHstcpHrD0pGm8?= =?Windows-1252?Q?pinMFDlTmM4C7zWAqd4l5J4sTUFHY7xC1/WaR8SLbE7YPNv8cq61ZE+J?= =?Windows-1252?Q?fZuQyEeLM1j5Dg2lEKQInhE61U/Rk/4E6ND+NIVmGwb1Ya4dwrC7QKNZ?= =?Windows-1252?Q?hIBZ8tD0Ya76NW2vEQ5d57TY6ARfMv8vVn5PMrkEkW1rxWwGZk2yZqLJ?= =?Windows-1252?Q?HgJEZLZRe5q6nyTYmPQI5SPx4FjvBBk2KDFPL7lAizlzw6rczXIVcdKP?= =?Windows-1252?Q?puIOwGjJO5qcnW00NqVVM6B4jlaljckhEArKB3sKms4UfOAcYxKfbivu?= =?Windows-1252?Q?vjRjFDI+9/9okz17YIsUxVcuDZnZ4a/9Xk9XueYGrQObjcEqXpxnVTPU?= =?Windows-1252?Q?xyBarYPL5kCxMzRAH7Ja/DT7zqJsF2j9j5iHMXMY2GMnuQAb2lvWF5ps?= =?Windows-1252?Q?5FwEwn6zWHxikHJxLIzeeMGitpPbEc0vtlN3gBc1B0ZsBJWIBrYExvDM?= =?Windows-1252?Q?DCu+sy2gRbiUPV2Vzv9ZDwX4nlrzzX3att3znjJsk1+InxCYIgge07Sz?= =?Windows-1252?Q?vwgLhItzuR?= X-Microsoft-Exchange-Diagnostics: 1;AM3PR02MB116;5:GGxSdIW97tmRuriPHb4DOn4eVd7s3CzgXMVFCTSTyg1ohZh66pUzGWsBol8b+NWU6aECOIMJjPpqAP4/1lHtlT7r3va6E8THVUVjmaWrPJDe/RN5bkAPSmew+/l7D3KcYxLqbkubym4mLw69OfG4QQ==;24:OryAVq8f1nH2yfTbbnXU5ul3XYbm19YdXrqp2Lb7fidFXHEJrcsIPAv9UyQAujN3OVvibrQGaxfRd1EnIyfU01go9Nvt56Ma6xK14a8ApME= SpamDiagnosticOutput: 1:23 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: ezchip.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 29 Jan 2016 18:18:14.1043 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM3PR02MB116 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 01/27/2016 07:28 PM, Frederic Weisbecker wrote: > On Tue, Jan 19, 2016 at 03:45:04PM -0500, Chris Metcalf wrote: >> You asked what happens if nohz_full= is given as well, which is a very >> good question. Perhaps the right answer is to have an early_initcall >> that suppresses task isolation on any cores that lost their nohz_full >> or isolcpus status due to later boot command line arguments (and >> generate a console warning, obviously). > I'd rather imagine that the final nohz full cpumask is "nohz_full=" | "task_isolation=" > That's the easiest way to deal with and both nohz and task isolation can call > a common initializer that takes care of the allocation and add the cpus to the mask. I like it! And by the same token, the final isolcpus cpumask is "isolcpus=" | "task_isolation="? That seems like we'd want to do it to keep things parallel. >>>> +bool _task_isolation_ready(void) >>>> +{ >>>> + WARN_ON_ONCE(!irqs_disabled()); >>>> + >>>> + /* If we need to drain the LRU cache, we're not ready. */ >>>> + if (lru_add_drain_needed(smp_processor_id())) >>>> + return false; >>>> + >>>> + /* If vmstats need updating, we're not ready. */ >>>> + if (!vmstat_idle()) >>>> + return false; >>>> + >>>> + /* Request rescheduling unless we are in full dynticks mode. */ >>>> + if (!tick_nohz_tick_stopped()) { >>>> + set_tsk_need_resched(current); >>> I'm not sure doing this will help getting the tick to get stopped. >> Well, I don't know that there is anything else we CAN do, right? If there's >> another task that can run, great - it may be that that's why full dynticks >> isn't happening yet. Or, it might be that we're waiting for an RCU tick and >> there's nothing else we can do, in which case we basically spend our time >> going around through the scheduler code and back out to the >> task_isolation_ready() test, but again, there's really nothing else more >> useful we can be doing at this point. Once the RCU tick fires (or whatever >> it was that was preventing full dynticks from engaging), we will pass this >> test and return to user space. > There is nothing at all you can do and setting TIF_RESCHED won't help either. > If there is another task that can run, the scheduler takes care of resched > by itself :-) The problem is that the scheduler will only take care of resched at a later time, typically when we get a timer interrupt later. By invoking the scheduler here, we allow any tasks that are ready to run to run immediately, rather than waiting for an interrupt to wake the scheduler. Plenty of places in the kernel just call schedule() directly when they are waiting. Since we're waiting here regardless, we might as well immediately get any other runnable tasks dealt with. We could also just return "false" in _task_isolation_ready(), and then check tick_nohz_tick_stopped() in _task_isolation_enter() and if false, call schedule() explicitly there, but that seems a little more roundabout. Admittedly it's more usual to see kernel code call schedule() directly to yield the processor, but in this case I'm not convinced it's cleaner given we're already in a loop where the caller is checking TIF_RESCHED and then calling schedule() when it's set. -- Chris Metcalf, EZChip Semiconductor http://www.ezchip.com From mboxrd@z Thu Jan 1 00:00:00 1970 From: Chris Metcalf Subject: Re: [PATCH v9 04/13] task_isolation: add initial support Date: Fri, 29 Jan 2016 13:18:05 -0500 Message-ID: <56ABACDD.5090500@ezchip.com> References: <1451936091-29247-1-git-send-email-cmetcalf@ezchip.com> <1451936091-29247-5-git-send-email-cmetcalf@ezchip.com> <20160119154214.GA2722@lerouge> <569EA050.5030106@ezchip.com> <20160128002801.GA14313@lerouge> Mime-Version: 1.0 Content-Type: text/plain; charset="windows-1252"; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <20160128002801.GA14313@lerouge> Sender: linux-kernel-owner@vger.kernel.org To: Frederic Weisbecker Cc: Gilad Ben Yossef , Steven Rostedt , Ingo Molnar , Peter Zijlstra , Andrew Morton , Rik van Riel , Tejun Heo , Thomas Gleixner , "Paul E. McKenney" , Christoph Lameter , Viresh Kumar , Catalin Marinas , Will Deacon , Andy Lutomirski , linux-doc@vger.kernel.org, linux-api@vger.kernel.org, linux-kernel@vger.kernel.org List-Id: linux-api@vger.kernel.org On 01/27/2016 07:28 PM, Frederic Weisbecker wrote: > On Tue, Jan 19, 2016 at 03:45:04PM -0500, Chris Metcalf wrote: >> You asked what happens if nohz_full= is given as well, which is a very >> good question. Perhaps the right answer is to have an early_initcall >> that suppresses task isolation on any cores that lost their nohz_full >> or isolcpus status due to later boot command line arguments (and >> generate a console warning, obviously). > I'd rather imagine that the final nohz full cpumask is "nohz_full=" | "task_isolation=" > That's the easiest way to deal with and both nohz and task isolation can call > a common initializer that takes care of the allocation and add the cpus to the mask. I like it! And by the same token, the final isolcpus cpumask is "isolcpus=" | "task_isolation="? That seems like we'd want to do it to keep things parallel. >>>> +bool _task_isolation_ready(void) >>>> +{ >>>> + WARN_ON_ONCE(!irqs_disabled()); >>>> + >>>> + /* If we need to drain the LRU cache, we're not ready. */ >>>> + if (lru_add_drain_needed(smp_processor_id())) >>>> + return false; >>>> + >>>> + /* If vmstats need updating, we're not ready. */ >>>> + if (!vmstat_idle()) >>>> + return false; >>>> + >>>> + /* Request rescheduling unless we are in full dynticks mode. */ >>>> + if (!tick_nohz_tick_stopped()) { >>>> + set_tsk_need_resched(current); >>> I'm not sure doing this will help getting the tick to get stopped. >> Well, I don't know that there is anything else we CAN do, right? If there's >> another task that can run, great - it may be that that's why full dynticks >> isn't happening yet. Or, it might be that we're waiting for an RCU tick and >> there's nothing else we can do, in which case we basically spend our time >> going around through the scheduler code and back out to the >> task_isolation_ready() test, but again, there's really nothing else more >> useful we can be doing at this point. Once the RCU tick fires (or whatever >> it was that was preventing full dynticks from engaging), we will pass this >> test and return to user space. > There is nothing at all you can do and setting TIF_RESCHED won't help either. > If there is another task that can run, the scheduler takes care of resched > by itself :-) The problem is that the scheduler will only take care of resched at a later time, typically when we get a timer interrupt later. By invoking the scheduler here, we allow any tasks that are ready to run to run immediately, rather than waiting for an interrupt to wake the scheduler. Plenty of places in the kernel just call schedule() directly when they are waiting. Since we're waiting here regardless, we might as well immediately get any other runnable tasks dealt with. We could also just return "false" in _task_isolation_ready(), and then check tick_nohz_tick_stopped() in _task_isolation_enter() and if false, call schedule() explicitly there, but that seems a little more roundabout. Admittedly it's more usual to see kernel code call schedule() directly to yield the processor, but in this case I'm not convinced it's cleaner given we're already in a loop where the caller is checking TIF_RESCHED and then calling schedule() when it's set. -- Chris Metcalf, EZChip Semiconductor http://www.ezchip.com