From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752330AbbJLQzu (ORCPT ); Mon, 12 Oct 2015 12:55:50 -0400 Received: from mail-am1on0095.outbound.protection.outlook.com ([157.56.112.95]:36929 "EHLO emea01-am1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1751866AbbJLQzr (ORCPT ); Mon, 12 Oct 2015 12:55:47 -0400 Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=cmetcalf@ezchip.com; Subject: Re: [PATCH] nohz: Revert "nohz: Set isolcpus when nohz_full is set" To: , Frederic Weisbecker References: <1444663283-30068-1-git-send-email-fweisbec@gmail.com> <20151012153202.GB3910@linux.vnet.ibm.com> <20151012162001.GA32228@lerouge> <20151012165353.GF3910@linux.vnet.ibm.com> CC: Ingo Molnar , LKML , Peter Zijlstra , Mike Galbraith , Dave Jones , Thomas Gleixner , Oleg Nesterov , Christoph Lameter , Alexey Dobriyan , Rik van Riel , Andrew Morton From: Chris Metcalf Message-ID: <561BE5FC.1020300@ezchip.com> Date: Mon, 12 Oct 2015 12:55:24 -0400 User-Agent: Mozilla/5.0 (X11; Linux i686 on x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 In-Reply-To: <20151012165353.GF3910@linux.vnet.ibm.com> Content-Type: text/plain; charset="windows-1252"; format=flowed Content-Transfer-Encoding: 7bit X-Originating-IP: [12.216.194.146] X-ClientProxiedBy: CY1PR1201CA0016.namprd12.prod.outlook.com (25.169.17.154) To HE1PR02MB0777.eurprd02.prod.outlook.com (25.161.118.141) X-Microsoft-Exchange-Diagnostics: 1;HE1PR02MB0777;2:xZDkyqSEkywKdm1VGRttHxKTrH6O4Jw5Yw24qE1pb5ynOG76qHTp72fTLgOsuW7dpW1x9LyWzPtanyETwhX7NFEw6OLa7M/Yjqnb2Ox2+5MwDV97y8xmcU+U7BXMi11hs2KDJExYdIWA6xY25lNjt+9nPkwkEc0WN2XnUSWfp/k=;3:5hMkVEesT1xLIDrRVw9MKtp7CxjIpdWaSFqN1razQuqETHVRFzv9NBZkqcNU2LGAWZiwzYY+OPWzvxAgSqQEVLuh/sCWoJ0E5C17K2qrDtqfHz5MrNrmFpVLDtXpzROhibwgSxVkvUCf8bGsL4c4Ig==;25:HtHgCkZMw3js83u2/92KMHUjaK4Mcb2lcd5fM2yYQ/D2ulKP22WLZu2sDlyxlw1ALHYgX6qT/zEtrm/rkzA8Ee6vEYKN1+B/qzF4VLZRt2LA5uemep77wpVO5uFcgeQ7Fx8m5uVxHvKwMMvNigANozyBfsz/Asc2FSIrPBhaSle48IxdmUgbUV+HRnIMuKXbZCi6/KdcJe5CLSXV6oF0zSmeceCBQ0BR1E5yGh2BuGAKjayRDFTCr4HTrRPRUWmCUyJnTsS8SgQAZHXuQNVGTQ==;20:klXJMTAudYinfiQSl+gAMHE5tX+1fcOWWMYKX70nI73fMvEZE84AqaJ4c5gg9KmL74PtLXkL3xltX64HN5LI3c/Md6/SEH7Spa6a97BjBWEHUuk6CRQ+OYI6MTQO1OZ1teQMaZbNqbTDC+inbgEM5pMfDbw3omecYODRS1Q59A0= X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:HE1PR02MB0777; X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(601004)(2401047)(5005006)(520078)(8121501046)(3002001);SRVR:HE1PR02MB0777;BCL:0;PCL:0;RULEID:;SRVR:HE1PR02MB0777; X-Microsoft-Exchange-Diagnostics: 1;HE1PR02MB0777;4:YjF18NUbD5fDNUkR66IxCpnML1Hw2toVBShPPMN60CVHo5Jqf0t+m0y8zf5hc0W0wUaoLana1gM0/VZPhNj90k9w7q9MikOuv1UW/NycFbDQ9eQVR9+ROBnfn1hUNDdjmWEMDqnlo8R2KKwN3++1GYDo87LxcUCMb3i+xwJ02rCXURftyPBBk+9PDCcV3hWsmvbm/Vnvw0/3a8BdGn4HhbIEF9+fSRPI2QWniHGxbIh13wShdlb9S9iB0Spavydre421siv7hKkhGT7XYwGyEF67SLZ6gViOEQgRhYrcuQKnw3ROZ8CSVCy5FoMkCZsPQwVxo6IDxoTGw++Nywvau/v24OniYOY4d6o/zeZfe48= X-Forefront-PRVS: 0727122FC6 X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10009020)(6009001)(6049001)(199003)(377454003)(189002)(479174004)(24454002)(93886004)(15975445007)(50986999)(83506001)(19580395003)(77096005)(36756003)(33656002)(87976001)(46102003)(54356999)(23746002)(19580405001)(87266999)(76176999)(65816999)(101416001)(66066001)(92566002)(64706001)(64126003)(42186005)(65956001)(47776003)(2950100001)(105586002)(5001920100001)(65806001)(575784001)(5001770100001)(106356001)(97736004)(40100003)(122386002)(86362001)(5007970100001)(5008740100001)(189998001)(59896002)(50466002)(4001350100001)(80316001)(81156007)(5001960100002)(5004730100002)(18886065003);DIR:OUT;SFP:1101;SCL:1;SRVR:HE1PR02MB0777;H:[10.7.0.41];FPR:;SPF:None;PTR:InfoNoRecords;A:1;MX:1;LANG:en; X-Microsoft-Exchange-Diagnostics: =?Windows-1252?Q?1;HE1PR02MB0777;23:mC/Fe58jjY4tQAUa19LIHkM7metUYPY62LSPG?= =?Windows-1252?Q?X1HwOmHzvmdYongeOtvcLIWumhQRSilG+K/8naHQthclVig+oEUDP6SF?= =?Windows-1252?Q?9sbychLEOEvw8p81lZJzXdwS3cdLjCdGc88drVaWsZSn85bioPULukRz?= =?Windows-1252?Q?cAdU9foruWslQfQprtkZ8MUxcKRJEyZSw+7R7lVXLThfgH7tb8GV3dTc?= =?Windows-1252?Q?hGAK999cRvLyVtIxuPoFyGyLGqp62I8WRs8cm+Fk/SKGShcqULjHpwhC?= =?Windows-1252?Q?0wNt1jOHIWYz+dqYTB/PJATMSvnH42bo0vsf2HWMn0TDlFyUGnROaNim?= =?Windows-1252?Q?ysBdAFXdUUQRgiWBAn6eenAd37dIBzpH4QPENzDe7KVxsg2gdMXYgyP5?= =?Windows-1252?Q?rbj/kuWgas1WhDDlUXMrqIFRIED1qKVbXzKftPYd/26kYxoqPRCklQ3/?= =?Windows-1252?Q?ns+zInFuxDUJ+r0zUCPhIw9jWslm7MtEXM7RluqfiOmI327vVX0kr842?= =?Windows-1252?Q?w4tXIILJg52IjWyDsYCK/nZtV91kbWfGvhhCJkt+X71g0EOVTL4kHfut?= =?Windows-1252?Q?RkF3H1hJb971Dxz/tFJqqMZgc5eJpcEr0sBE/HGriZG3MIT65iSf/jZf?= =?Windows-1252?Q?QHDlIdczBG10rdA0ffw0CptCztbw12wM/tw//Z4BHaL0JZwt8zR0U3kK?= =?Windows-1252?Q?yzkOnpslURZunLFnYzZzwiJ8Ku/fYI1uJ/3UBVnqAHQ/dZ7ofa07bzEM?= =?Windows-1252?Q?ThHWEdC+DgK54iiQBV8RAnNqyX3GFo6cPja2iFgbF40RILP64RILxJbp?= =?Windows-1252?Q?WGLldSdcYKjXhQwUek0jqvK1BaG1+NjHHoQ4GYtVto2N7fqEMXkIZ/sU?= =?Windows-1252?Q?CrL+QSGpk0/rGCH0u4PWcL43fL0BSMWNyX83+cRQUEVk5Ntibgqesexk?= =?Windows-1252?Q?ekemDIk6nSyNLepUlgF6ugxx2kOR/jEURceTCWQ5KspN2SgcKAc43ZFZ?= =?Windows-1252?Q?Vh8vg2lfKxB1XlDfqa0FIX3CgT0f6SxMghzCPNDDL6pxLpFGo17Yw6X/?= =?Windows-1252?Q?p7iCqETa9W0miXnbOu+ZujwmP86NGDLJFOAdCy9VOr+N9lkF/sE45+AJ?= =?Windows-1252?Q?S2pj8X+q0e7zXHrUPs2/mBxp3hZQ93kV77Fc2y6eiqbCkW1M/ao8AUex?= =?Windows-1252?Q?Ehq1iShFdUzj2EV5h+4H5IKSsGCWWk/ysbEYe2RnikApZnw0bfqn7F2J?= =?Windows-1252?Q?98niVqulspWFhSx573s9grRF98J9zeFaW0PsLMdjsuVLAhzxnyxAo0RR?= =?Windows-1252?Q?Xb3wOMKznMpeFGf3dLucHVfIY2nSErwubgxVTQeA/cOt9Xxybuk0Vxer?= =?Windows-1252?Q?UvJ7rQqOmjL73PdhqxBBqWqE+nsMP5ZQ0a0F105ZiFKEFeMG+o4m+J+l?= =?Windows-1252?Q?drpsozAqdkdwbgvr6eqG5+p8sn2BBxrS/+TiikiA75Q7tiQ5rC5iRt3x?= =?Windows-1252?Q?MQxdAD4pHPcDUcNPCkTuETvdo1A?= X-Microsoft-Exchange-Diagnostics: 1;HE1PR02MB0777;5:QG6jRzmNMoOssUQx4iFa0U6jNDcP7sD82d/o62C44/H0Fvohvo+JKO9xHLXtsoQCqCnoO1EpQF3OKLRZMdtwkn/Gbt3cmRLLIBYQTsRnT0bopkZaI0fIV3YpD9UVfmY+5WQQDVLYsJbrlaDnrpyWJA==;24:hJl0Ht5YklmJZ8GJXz4+MkjA1IfxUqL1pKeRYaykkXKF81Lf9AH4gfauAmuoAzHeQn2QkLLx2SClwJ+52rDSjqZTG0VJ5AGfoS48SahBf1c=;20:4CbQHoOsflxGDBdF9kPEHBHtXy1xCs7ZG7LAKB7FhLt1c2FPkDH+lvjgin3KrS/mgDYjSgfPxq1tQzE/i2Nm1w== SpamDiagnosticOutput: 1:23 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: ezchip.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Oct 2015 16:55:40.0581 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: HE1PR02MB0777 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 10/12/2015 12:53 PM, Paul E. McKenney wrote: > On Mon, Oct 12, 2015 at 06:20:03PM +0200, Frederic Weisbecker wrote: >> On Mon, Oct 12, 2015 at 08:32:02AM -0700, Paul E. McKenney wrote: >>> On Mon, Oct 12, 2015 at 05:21:23PM +0200, Frederic Weisbecker wrote: >>>> This reverts commit 8cb9764fc88b41db11f251e8b2a0d006578b7eb4. >>>> >>>> We assumed that nohz full users always want scheduler isolation on full >>>> dynticks CPUs, therefore we included nohz full CPUs on cpu_isolated_map. >>>> This means that tasks run by default on CPUs outside the nohz_full range >>>> unless their affinity is explicity overwritten. >>>> >>>> This suits pure isolation workloads but when the machine is needed to >>>> run common workloads, the available sets of CPUs to run common tasks >>>> becomes reduced. >>>> >>>> We reach an extreme case when CONFIG_NO_HZ_FULL_ALL is enabled as it >>>> leaves only CPU 0 for non-isolation tasks, which makes people think that >>>> their supercomputer regressed to 90's UP. >>>> >>>> Some nohz full users appear to be interested in running normal workloads >>>> either before or after an isolation workload. Nohz full isn't optimized >>>> toward normal workloads but it's still better than UP performance. >>>> >>>> We are reaching a limitation in kernel presets here. Lets revert this >>>> cpu_isolated_map inclusion and let userspace do its own scheduler >>>> isolation using cpusets or explicit affinity settings. >>>> >>>> Reported-by: Ingo Molnar >>>> Reported-by: Mike Galbraith >>>> Cc: Chris Metcalf >>>> Cc: Rik van Riel >>>> Cc: Christoph Lameter >>>> Cc: Mike Galbraith >>>> Cc: Peter Zijlstra >>>> Cc: Dave Jones >>>> Cc: Oleg Nesterov >>>> Cc: Paul E. McKenney >>>> Cc: Thomas Gleixner >>>> Cc: Ingo Molnar >>>> Cc: Alexey Dobriyan >>>> Cc: Andrew Morton >>>> Signed-off-by: Frederic Weisbecker >>>> --- >>>> kernel/sched/core.c | 3 --- >>>> 1 file changed, 3 deletions(-) >>>> >>>> diff --git a/kernel/sched/core.c b/kernel/sched/core.c >>>> index 6159531..3c35b5f 100644 >>>> --- a/kernel/sched/core.c >>>> +++ b/kernel/sched/core.c >>>> @@ -7238,9 +7238,6 @@ void __init sched_init_smp(void) >>>> alloc_cpumask_var(&non_isolated_cpus, GFP_KERNEL); >>>> alloc_cpumask_var(&fallback_doms, GFP_KERNEL); >>>> >>>> - /* nohz_full won't take effect without isolating the cpus. */ >>>> - tick_nohz_full_add_cpus_to(cpu_isolated_map); >>>> - >>> Why not make this controlled by a boot parameter? That preserves >>> the ease of use for those needing it, but avoids problems from people >>> doing "make randconfig". >> Well it is already. As you pass nohz_full=1-32, you can pass as well isolcpus=1-32 > True enough. Not sure that having to repeat the CPU list twice qualifies as > "easy to use", though. Why not a nohz_full_iso or some such that isolates > whatever CPUs you specified? Is it worth starting to think about grouping things under the "task isolation" model somehow? "task_isolation_cpus=1-31" or some such for this, and then that just sets up the nohz_full and isolcpus options under the hood? -- Chris Metcalf, EZChip Semiconductor http://www.ezchip.com