From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753458AbdHIPeu (ORCPT ); Wed, 9 Aug 2017 11:34:50 -0400 Received: from bombadil.infradead.org ([65.50.211.133]:56381 "EHLO bombadil.infradead.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752608AbdHIPet (ORCPT ); Wed, 9 Aug 2017 11:34:49 -0400 Date: Wed, 9 Aug 2017 17:34:37 +0200 From: Peter Zijlstra To: "Ofer Levi(SW)" Cc: "rusty@rustcorp.com.au" , "mingo@redhat.com" , "Vineet.Gupta1@synopsys.com" , "linux-kernel@vger.kernel.org" , Tejun Heo Subject: Re: hotplug support for arch/arc/plat-eznps platform Message-ID: <20170809153437.gngof7hiiny2zv3j@hirez.programming.kicks-ass.net> References: <20170807083354.ptkehfql54xydsvm@hirez.programming.kicks-ass.net> <20170807151020.h2u45tx64ccee64a@hirez.programming.kicks-ass.net> <20170808101624.qdszwko4ajy7qka3@hirez.programming.kicks-ass.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: NeoMutt/20170609 (1.8.3) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, Aug 09, 2017 at 03:19:02PM +0000, Ofer Levi(SW) wrote: > I appreciate your effort and detailed reply, however I'm still experiencing performance hit at > partition_sched_domains(). It seems the issue is due to the large magnitude of cpus. > I used he suggested method 2, patched in the diffs and used the command line switch isolcpus to > kill load-balancing. > It did save few hundredth of a sec per cpu. When I limited number of available cpus > (using present and possible cpus ) to 48, it did reduced dramatically this function execution time: > > With 4K available cpus : > [ 48.890000] ## CPU16 LIVE ##: Executing Code... > [ 48.910000] partition_sched_domains start > [ 49.360000] partition_sched_domains end > > With 48 available cpus: > [ 36.950000] ## CPU16 LIVE ##: Executing Code... > [ 36.950000] partition_sched_domains start > [ 36.960000] partition_sched_domains end > > Note that I currently use kernel version: 4.8.0.17.0600.00.0000, if this has any influence. > Would appreciate your thoughts. > Does something like this cure things? It seems we're doing a possible_cpus iteration for sysctl cruft, and that will most certainly hurt on your little toy :-) Not sure what the more generic solution to that would be, but the below avoids it for isolcpus. --- --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -85,6 +85,7 @@ int sysctl_sched_rt_runtime = 950000; /* CPUs with isolated domains */ cpumask_var_t cpu_isolated_map; +cpumask_var_t non_isolated_cpus; /* * __task_rq_lock - lock the rq @p resides on. @@ -5685,8 +5686,6 @@ static inline void sched_init_smt(void) void __init sched_init_smp(void) { - cpumask_var_t non_isolated_cpus; - alloc_cpumask_var(&non_isolated_cpus, GFP_KERNEL); sched_init_numa(); @@ -5697,17 +5696,17 @@ void __init sched_init_smp(void) * happen. */ mutex_lock(&sched_domains_mutex); - sched_init_domains(cpu_active_mask); cpumask_andnot(non_isolated_cpus, cpu_possible_mask, cpu_isolated_map); if (cpumask_empty(non_isolated_cpus)) cpumask_set_cpu(smp_processor_id(), non_isolated_cpus); + + sched_init_domains(cpu_active_mask); mutex_unlock(&sched_domains_mutex); /* Move init over to a non-isolated CPU */ if (set_cpus_allowed_ptr(current, non_isolated_cpus) < 0) BUG(); sched_init_granularity(); - free_cpumask_var(non_isolated_cpus); init_sched_rt_class(); init_sched_dl_class(); --- a/kernel/sched/debug.c +++ b/kernel/sched/debug.c @@ -327,6 +327,8 @@ static struct ctl_table *sd_alloc_ctl_cp return table; } +extern cpumask_var_t non_isolated_cpus; + static struct ctl_table_header *sd_sysctl_header; void register_sched_domain_sysctl(void) { @@ -340,7 +342,7 @@ void register_sched_domain_sysctl(void) if (entry == NULL) return; - for_each_possible_cpu(i) { + for_each_cpu(i, non_isolated_cpus) { snprintf(buf, 32, "cpu%d", i); entry->procname = kstrdup(buf, GFP_KERNEL); entry->mode = 0555;