From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S933039AbcI3Tc7 (ORCPT ); Fri, 30 Sep 2016 15:32:59 -0400 Received: from mail-db5eur01on0085.outbound.protection.outlook.com ([104.47.2.85]:23317 "EHLO EUR01-DB5-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1751076AbcI3Tck (ORCPT ); Fri, 30 Sep 2016 15:32:40 -0400 Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=cmetcalf@mellanox.com; Subject: Re: [PATCH v15 04/13] task_isolation: add initial support To: Andy Lutomirski References: <1471382376-5443-1-git-send-email-cmetcalf@mellanox.com> <1471382376-5443-5-git-send-email-cmetcalf@mellanox.com> <20160829163352.GV10153@twins.programming.kicks-ass.net> <20160830075854.GZ10153@twins.programming.kicks-ass.net> CC: Thomas Gleixner , Christoph Lameter , Michal Hocko , Gilad Ben Yossef , Andrew Morton , Linux API , Viresh Kumar , "Ingo Molnar" , Steven Rostedt , Tejun Heo , Will Deacon , Rik van Riel , Frederic Weisbecker , "Paul E. McKenney" , "linux-mm@kvack.org" , "linux-kernel@vger.kernel.org" , Catalin Marinas , Peter Zijlstra From: Chris Metcalf Message-ID: Date: Fri, 30 Sep 2016 12:59:33 -0400 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.3.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset="utf-8"; format=flowed Content-Transfer-Encoding: 7bit X-Originating-IP: [12.216.194.146] X-ClientProxiedBy: CY1PR14CA0071.namprd14.prod.outlook.com (10.164.65.167) To VI1PR0501MB2766.eurprd05.prod.outlook.com (10.172.11.16) X-MS-Office365-Filtering-Correlation-Id: fda11373-eaf7-404e-3996-08d3e953355d X-Microsoft-Exchange-Diagnostics: 1;VI1PR0501MB2766;2:IQLOM4gJUSCD3eaYk0acwhIKR8fVVxZo0kKaFyAcWNBD1hgjmNjGBcogob1xNJDuMCoesw2/sl6brqeEH3g0Dg/k4Hvq136wTDqO5dPk0Pky0WbKgRuRIwHn6NIuZB96A7N4uc3hTaU7R5x1Ih8un2F7ckM94bP0JEbXrXWTDj+L/oZBiNGag1FOVDIA9N+dkJxcIDiLTJxux7wpMCbdLg==;3:L3TZ36HFz3kPYquW21yNKbo0lxoFkK52mANVobsQvFNyWx4JZBTR0R+nezu8QHB7FWZA5NcQy3Ybx67YZPeKaQ6zl1MyNQ58Ihc6mAlFobG2LzC8W9lwRgfRz6mdOJTS0rkn8uojCrxrf6XkKsuZ8A== X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:VI1PR0501MB2766; X-Microsoft-Exchange-Diagnostics: 1;VI1PR0501MB2766;25:vUmqKggm24f1FVNwb7AOlAIDm6AJksyVJyhvtu3HLI8HUOS141XV0WKFPsnBdG+gi6vHis41QDPHkWsBEwT5n+jKUWGtt54ORamGuS/YjlEIos/cRPxtwIoMbVpMNs6KJiMVZeBd61EJGxqdAsHF541gUEFgGGjpN5u0pdfHd9T6g2hYnQ4xkZkiIIX7OMlOWvEWjFxEZxjvXBk2uCPX4ZmTeK6Kn11nw30/r3fAilAEDdle6x8hjCfX/ql5LbdphOMsGMEJqpvtqIkZ42Fhdbm44SdR20CMsf3eYpvxD2w16gsWuo3J41wCeRyHSxFewyyWsSV1PTLj6dKZNtO/EoVKmgu5jdjot3ZU5LJLCAO1pgiz5jzkTw3gsuc5jdaX7v4xiTl7cLp/FmTUa2L3yiPrAweBDA5mFcEaE/9mTQMWe8wWGlDND/qWE0UolQUoNLVoaH/cElBBhxC1HTw0dBuDYULZHgjMXB6X1Ip7w4i5UKdBQaEbmfCDWeJ6TrDt4UtDVSjo7kuegSwX2nzZCtWQApfKpYjhm0JWmlv8/7QmeqtnVqfFjOGFwh4KVVRH1VqtajdFPfvjPufD7TNf4Rg0YlOpOw0YF5/mPMR6FOD3zkeajFhtoNYy9/o3Rz2GAz9nijqpFbBQ4B20fBRCutQNGnqtGvwmFNmVocMXdJH21Xs6KiXcxbgb7+m4eEUfy8iAtgXjDrP2dg0yUIuwgXw14hgss9dxiyCJ/PxKcMSdOQmarPCoRAF05is55ffpz1NL8GaTyZpc2e1xE2MRWDlhDOl+TeyehlOtalTo0ypPs7kCK8pDsWQUhZncO2yose96bpCJnvUOmDKAxqVOgeFpJBfAZkYeaaU5Q61qXmE= X-Microsoft-Exchange-Diagnostics: 1;VI1PR0501MB2766;31:N1cp8Z/7EHZoy6yiVmbs+QBTwRVTP91vbtEFxeaRKWr8i34inDWxXMgDkG3w/i0csDWw9s4zERydzg7iKv+wRZ0dIXCJ8VbuLE8kwvShyIcMG8qJxBgdaLoPTvM4lSKz44Dt8rYXUpGwvxp87Pk7Q9mIXIlJfRNKRa5088WGluGK7iEKsYwy8PGPKHhDZceOrT/qhMHWZTdZVZwcIqHQHp8IL329yuIBtvXKpxFw7qc8EJBRpKqPfBI27K6q4+Lk;20:Saw7hlxd73cfgHj2uhQv4+7AltAgC3iMXhGlzo/3qfXb60+6TTAu5F4s5VyTOlY4K5JdeJhBn8aTxkpRwTi15U0OgvqOuQ142DIhkDDqmidEGX7Vjx+Ulosq/kBUC106Ssh2+Jg6unM9SOsIaf7gUUxIbDdISLRkNFm4ycL42+4EvMdoXdyuqW6XlafjHItfWNAaJoeoEseIqnr6poB1+s20rqsUZdd2po26n17AWa8cRPpcWAwzi4lIRe+nSn1oin3DarcJuKOoLnS6bjEihe8AydKv/sRTEDP/JAHTzp49f6A/nQQmMBKmoALKctD6W/7din1bJigyH6ebHwbY6YgM436Nou/oBVBNeg1PWcZjRVQjcq7G/xFL4hfy/qK0q+v+xZkUUhi4rI6iULxGsf6BShyioNDYnd2ReknYRL5EWHxpWEthfIQmtnwbNao7SjgRb0/7kyMTAp135bRol8jE6fAu11qv3VdbSXAeAgwtDW4/eyhVkVzCSr1jd87c X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(171992500451332); X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(6040176)(601004)(2401047)(8121501046)(5005006)(3002001)(10201501046)(6055026);SRVR:VI1PR0501MB2766;BCL:0;PCL:0;RULEID:;SRVR:VI1PR0501MB2766; X-Microsoft-Exchange-Diagnostics: 1;VI1PR0501MB2766;4:IWlc17+qnSSFeoddFWmnIjKVCEk6ZmaAZ1TZUIBkkfRsWhf9/iFX2yDrfnZ4nHWpFMnKuxkhmOx1HLU+C+Btp3ycjxk/bnhaOm4ExHy5irkh4FAHGSQoEVtthCXRVzKEymuvDhUiZ+ZIKL5s3241RRSZ2a97ueUaUrwIwCAAd7LDzqA7hxujmd4Qn2sTjQ93EKreOUHpUosJKSCnDN/4drSTU68sSze3UN9WvjdinufTgdd5qr1Ww06DGmDKcI4CYut8z8/34FyZxtVr6XYQZ4CCBfJtLh0Yq1d+P1dvgEQ9fvv6akYC6K9HrTPlJwiSRheJNpXN0jWordiPVLpgyFN410E17DJ022MBJQiu0il4lr1XzIxeBtwOtJrYKQsamfIUfH5muGsdIx8S/nR/6g26Ojar/Z6ZIBtCSw8JJNbs+noRCngqYlOQZKG5f6iEogl9iYKiPAC6Ea0Rmgrn2A== X-Forefront-PRVS: 008184426E X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10009020)(4630300001)(6009001)(6049001)(7916002)(189002)(199003)(24454002)(377454003)(36756003)(4001350100001)(105586002)(97736004)(50986999)(189998001)(23676002)(76176999)(8676002)(19580405001)(93886004)(2906002)(81156014)(7416002)(31696002)(54356999)(6116002)(230700001)(3846002)(81166006)(42186005)(19580395003)(4326007)(65826007)(33646002)(110136003)(6666003)(68736007)(77096005)(65956001)(64126003)(5660300001)(2950100002)(6916009)(106356001)(7846002)(31686004)(83506001)(47776003)(7736002)(50466002)(15975445007)(101416001)(66066001)(92566002)(86362001)(305945005)(586003)(65806001)(18886065003);DIR:OUT;SFP:1101;SCL:1;SRVR:VI1PR0501MB2766;H:[10.15.7.181];FPR:;SPF:None;PTR:InfoNoRecords;MX:1;A:1;LANG:en; X-Microsoft-Exchange-Diagnostics: =?utf-8?B?MTtWSTFQUjA1MDFNQjI3NjY7MjM6Wjc5enhpVlQxa3NzVVB1d3ZuREI4T3lV?= =?utf-8?B?YXZkeE1xR1hDU083UjFRRGRCY3JPNzhUNHBjWFVpMXVhTkNUblNpNWVTUHhv?= =?utf-8?B?SHNmbXN6d1lHN3Z2TnBObjZRUkg5VlZIOHZWeHRjVUFaeW01VTB3akFueXFK?= =?utf-8?B?MFZYMlpTSjRiQjNsWUxwYldHaWdRUEJOcHNubnk2YTZLRGNQRGcydFlQSHlX?= =?utf-8?B?MDJBcFdsVmJqRnoyRHlsOHhlcU1WRjcyd3VjQll3Z0doNGlXNnJvcWdXenZQ?= =?utf-8?B?NkQ1UThoVVJoSWducm5uQ1kvQnI0M1kxOTl4aldYYzM2NFA5VzRLdGVDcTdE?= =?utf-8?B?Vm5KdmRNd2k0NTk0TVA4bkNPYlZlWEZnTWFWeVRkQjRJN0Z5VDFUdTZvNklH?= =?utf-8?B?Tm9wRFoxd0NmbXRKQ0FuamVLZ0EwbUI0ZVJweVpCeDZrT2lGRUhHQTFCbkRK?= =?utf-8?B?UHR4RTNUdjBUbnpXZWdDNnVoY2FxQXk4bzlKQnd2TFlqYUhjaEVmdkMrTSs4?= =?utf-8?B?VW03SUh0TjlxNUZ2OFdYK2tpQTdyeXQ4SkZvRzhLTkpPeU50QytCd0RpakEv?= =?utf-8?B?VnpJUHBRL1ZJalVKMlkxUXgwVnNhRm5uOVFZN3hVbHFIaFJ4cXZpT25WblQx?= =?utf-8?B?RDNqOHl5di9qN0hOcFNKbGRnajUxNlBXUUtubWtaN3hTSDFqRGZLYklJaE9W?= =?utf-8?B?aDFIRFBWQmdiRnBwajdCNDZkVmVpaXFNQlJqdU15NnNrTGZub2N3ZjJTZFgz?= =?utf-8?B?RU9WUGFVamMrQTVFQjRyVVpiZ3FsMU9zRDk1aytKWkdBU1NqYTR4TUNpYjJv?= =?utf-8?B?ZEhITXpwbGZwRGVNTk9vMUJ4aW10bnhXWTNHUGpvRWpNQ2U4SXZRbStsQWQ2?= =?utf-8?B?Um50WU42aHdrT2w4Sm5GT2E3Y1JWUnJYOXZHT0JDWEZXbTA3Lzg2Vk9UNXZB?= =?utf-8?B?VVRSSXpBeG9KVkJFVWhZVFR3Q0MrZ25hM1ZRZXBCb0pGWDVKM2lZeTFTcUZH?= =?utf-8?B?cE8yeWNOd3hnREg3OWg0bENlaG55dWxpbldPek54RnR2a0x2bWF3MnJ5dGNM?= =?utf-8?B?VXowZ1FjV2tCMnZZeURUbW16cm13Y2dFZVkvRVBOZ1Noanl3T3JDWHRJRTBR?= =?utf-8?B?U1IzWXhSQVFVaWphaFFpcy90VVQxeU1KUjY4aTF4Y0FNWGZsWjVIN2FQbUNW?= =?utf-8?B?dnNRNWRIY3FaRTZoNXhPWWxpL1c3ZkhsK0JzV3p5cVlMaGsycVRlR2NndU9i?= =?utf-8?B?NDRnNFQwZUdzclV5bURCT3NhNWRDQ05GRzhFdHB3Y0JjZnBVd0t3aHhocC9n?= =?utf-8?B?Z3huRW5BRHcyOVBZQ0xPQkxBdXExZjR6aWg3b3lTSFpZZmc2MzhjWHFpRGlS?= =?utf-8?B?QmN6ZWE2TXpyKzY0azR3WFc3ZXQ4NXNrTU9weGpPSzRrY0FJMXJwNnZPeGVy?= =?utf-8?B?VnV3dFdEOWxEdk5EMnMrdFkxQms5Wm1jSklRRTZ4MXZyOEhSOStsek8reFlo?= =?utf-8?B?NUJmbzZZNWFXdDUvVDMwR0MxemFwNXdQNFR3eTY0SWM5QU9rckJmRkEyRUZk?= =?utf-8?B?NmdzdjJBY3R4YVVEalRUQjgvdkFhMG0weDBNTWduUG53emNxdWMrSEVvZDlk?= =?utf-8?B?TThyZ2ZvV3pQVUt0ZTN4alh5YjJVa3RBcjRaSmR1LzJiQ1dzVUZWWFU5M3Nl?= =?utf-8?B?bmRFWmhOTk1xUm9jd1lzTXlWV01IS3Y3RDY2RFdNQlZYZ2hSZ0VSKzU4cmRZ?= =?utf-8?B?RkIySmUrZ2NFenc1YWc4aEJvK21Sc3lDbG9zajFaNE5oQ3VkSW5VRmE4VllC?= =?utf-8?B?QU1Rc3BSazdjK1RwTVRDTmdkUFdpOXN2eFdOakg4SkVvenFYMDdOQjBaRFo0?= =?utf-8?B?Z2Rzb2s1M0s5ZkNEMDZYOTdWZE1wUEE3a2lMTUZDQ1Ywa1YySnJpN0IvbGw2?= =?utf-8?B?UzBDcXRYK1k5UkE9PQ==?= X-Microsoft-Exchange-Diagnostics: 1;VI1PR0501MB2766;6:y08USTjt+rcoO6CW/L8DqYLax0JvM8SvSsfQwUf6nkI/YgcQJeepcYjXLsioLxhi5eol9D9vyCGs66q17Z/WwoHqYexoZos9QFms1bXdL6OkJ6eg46jNHhte1bmdV+n1GfIaXxGK63dk43LMYBA3Sm1d0ZjLc0llwEruOjJA8b/mH08TfDtW/dRSmf5KLhfR7KWRLpjVUyv2QhDYLJYnXMc0C7DwR6nnKHjOPjC/MoeancBQOEUfTKsDNt8Vxvomd5nizqRLIIEAGf53BsWXeElJGn8Rh4FmaoIKQy/mkPJj4C2/LC4m5fFJN14rI85iC6r6haFqwa10sXIJkWDrWAxku+d11RxZK6kk74tnM6Y=;5:khmrGslXvFl/huGgIShPIq2UP/EjypiAbYbCtYHCEgdS+ubcw/9+SIodWVFjaHgORuUD6ZWjFZq4NLUlTGr9ecYx3TUczxjS4CjZOthculk/NKwu1PoIYqvUsBmqyuVHxtznqn0RiEE+KiTikaNxaPP2qEqWgtlXIoTytOjWMW0=;24:dk/vmOL6stbP7ju2XE59qFJqcuZWX8eAZHapSVLLo+5WZVa+6RG/bNXJBA+F0bgJW+HWjwcj8Nuc7FaWmR2cYliiGc53D5w5+PBgHRDpvO4= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1;VI1PR0501MB2766;7:rlNYXc9orgcT3wU/VzAop+OcTEWWqkakT6XtY8M2zz5W8jDeGe1kRKqC26zElgzk1s2rPzpewHFwBXfSb9yDCbMGBimaVz0seEyC27bEiBUHEwS3nN5DP2BuYL09fDW8z3cBDJfn7GafDutuEp7m6zi6DtumjXzNolZ+zdlrwfzX+e4vWdpGv2K8cXt8M51oxwO0xSnl1rPI3g8l1eymG2mmzbm5X5IXvBfAQra+rf3wdsJs1V9nGzSHYDsekXW4+kxfeQWfUuNZHmlSy8RF/rLYxCHynElISiop3Rec04hHBrLR8dCA2/OkCy2xp8E82P0diCRJP0SKCjqLdgwuVA== X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Sep 2016 16:59:53.0242 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR0501MB2766 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 8/30/2016 2:43 PM, Andy Lutomirski wrote: > On Aug 30, 2016 10:02 AM, "Chris Metcalf" wrote: >> We really want to run task isolation last, so we can guarantee that >> all the isolation prerequisites are met (dynticks stopped, per-cpu lru >> cache empty, etc). But achieving that state can require enabling >> interrupts - most obviously if we have to schedule, e.g. for vmstat >> clearing or whatnot (see the cond_resched in refresh_cpu_vm_stats), or >> just while waiting for that last dyntick interrupt to occur. I'm also >> not sure that even something as simple as draining the per-cpu lru >> cache can be done holding interrupts disabled throughout - certainly >> there's a !SMP code path there that just re-enables interrupts >> unconditionally, which gives me pause. >> >> At any rate at that point you need to retest for signals, resched, >> etc, all as usual, and then you need to recheck the task isolation >> prerequisites once more. >> >> I may be missing something here, but it's really not obvious to me >> that there's a way to do this without having task isolation integrated >> into the usual return-to-userspace loop. > What if we did it the other way around: set a percpu flag saying > "going quiescent; disallow new deferred work", then finish all > existing work and return to userspace. Then, on the next entry, clear > that flag. With the flag set, vmstat would just flush anything that > it accumulates immediately, nothing would be added to the LRU list, > etc. Thinking about this some more, I was struck by an even simpler way to approach this. What if we just said that on task isolation cores, no kernel subsystem should do something that would require a future interruption? So vmstat would just always sync immediately on task isolation cores, the mm subsystem wouldn't use per-cpu LRU stuff on task isolation cores, etc. That way we don't have to worry about the status of those things as we are returning to userspace for a task isolation process, since it's just always kept "pristine". The task-isolation setting per-core is not user-customizable, and the task-stealing scheduler doesn't even run there, so it's not like any processes will land there and be in a position to complain about the performance overhead of having no deferred work being created... -- Chris Metcalf, Mellanox Technologies http://www.mellanox.com