From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752181AbdHJHkW (ORCPT ); Thu, 10 Aug 2017 03:40:22 -0400 Received: from mail-eopbgr20088.outbound.protection.outlook.com ([40.107.2.88]:22776 "EHLO EUR02-VE1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1750868AbdHJHkV (ORCPT ); Thu, 10 Aug 2017 03:40:21 -0400 From: "Ofer Levi(SW)" To: Peter Zijlstra CC: "rusty@rustcorp.com.au" , "mingo@redhat.com" , "Vineet.Gupta1@synopsys.com" , "linux-kernel@vger.kernel.org" , Tejun Heo Subject: RE: hotplug support for arch/arc/plat-eznps platform Thread-Topic: hotplug support for arch/arc/plat-eznps platform Thread-Index: AdMMYSqGmAraesgzTdut3hkUP9fMggC9ry0AAApSJUAAA4Y/AAAfBi3QAAkAdgAAO4R8UAAB4zKAACGMIdA= Date: Thu, 10 Aug 2017 07:40:16 +0000 Message-ID: References: <20170807083354.ptkehfql54xydsvm@hirez.programming.kicks-ass.net> <20170807151020.h2u45tx64ccee64a@hirez.programming.kicks-ass.net> <20170808101624.qdszwko4ajy7qka3@hirez.programming.kicks-ass.net> <20170809153437.gngof7hiiny2zv3j@hirez.programming.kicks-ass.net> In-Reply-To: <20170809153437.gngof7hiiny2zv3j@hirez.programming.kicks-ass.net> Accept-Language: he-IL, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=oferle@mellanox.com; x-originating-ip: [193.47.165.251] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1;VI1PR0501MB2110;6:m5YGKrsRq6nObsa1ZPRwdjUncmMEO+XFPaJFQp3ELRcGfMUUPaCiaFmS94e+wVUeeVpf7ZUWjL4rXYCEOkULHt2MdNequJCbmfhWdYc4woOUOtEwUBjAwDfmSEC2pGX2FC6mRuUjymNBpx+uH4p98s1tt1CFpIgbKMqM86kEWdPfDb3P88Nr3Bxx0Bp1HEAfDoP4L9JJv8xGbu82dAbiJCtlBij6+e63YPaFTI954jWB5LvPIFutDuGBgLnhhi+I93PqOnkQEo1oZfhDwyXp7Q0M6kaJkp0OD9q/QZmm1lCy/5jtmfYpa8hmydj4CbQQKvMcs405HwDqCsLlF1HviQ==;5:RcQmIFCH4E/gQTQH6HGMro/kHhQNA/qRwaC2sWTEbrds/MC3hWlOmG1SA8SonlUg7Ykm9rmpP9LKU98ucvg1HcoozSTECTX1xgyTThduTEdtSJQZoIy/Px/F2rh1mbq2mRkHXvGhxYjzV2E9LVKWgw==;24:QktXxZ0DN4XDkoMvYVTsiOiYPxlikhTc+y15U6gO7ANYK0PPYX699WsKXaxLTYxRWHRObJHH5+JItCy3vUmnPp6RCM7NMWoegsR3CBEer28=;7:PTSCl9FTPHqnW2Adiwop6qL+T6QXzmL99+1MZNjBR4o9jfdJieD6bQVS3yQRKEidBE3VUyWKz33q9cicf/lP4uKPearnD0GcezBj8PSgiKJvXYzZWnwEDuuVLW5dcbegoHKmcO2Egy/o35hK4nyDmhsbKQ7YAP8XMjcMcznXrrqBQrp1I3UajblQdBqxCVdAcGl39vFD8p0JtSTijVshrgVhIW+VlAI8TIuZdYn9N0I= x-ms-exchange-antispam-srfa-diagnostics: SSOS; x-ms-office365-filtering-correlation-id: 42cef0f9-30f1-47be-02e3-08d4dfc30b3c x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:(300000500095)(300135000095)(300000501095)(300135300095)(22001)(300000502095)(300135100095)(2017030254152)(300000503095)(300135400095)(48565401081)(2017052603031)(201703131423075)(201703031133081)(201702281549075)(300000504095)(300135200095)(300000505095)(300135600095)(300000506095)(300135500095);SRVR:VI1PR0501MB2110; x-ms-traffictypediagnostic: VI1PR0501MB2110: x-exchange-antispam-report-test: UriScan:(9452136761055)(46150409022019); x-microsoft-antispam-prvs: x-exchange-antispam-report-cfa-test: BCL:0;PCL:0;RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6040450)(601004)(2401047)(5005006)(8121501046)(3002001)(10201501046)(93006095)(93001095)(100000703101)(100105400095)(6055026)(6041248)(20161123558100)(20161123562025)(20161123564025)(20161123560025)(20161123555025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(6072148)(201708071742011)(100000704101)(100105200095)(100000705101)(100105500095);SRVR:VI1PR0501MB2110;BCL:0;PCL:0;RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095);SRVR:VI1PR0501MB2110; x-forefront-prvs: 03950F25EC x-forefront-antispam-report: SFV:NSPM;SFS:(10009020)(6009001)(39840400002)(39860400002)(39450400003)(39850400002)(39410400002)(39400400002)(24454002)(189002)(199003)(57704003)(377454003)(13464003)(3660700001)(7736002)(81166006)(305945005)(3280700002)(229853002)(106356001)(6916009)(2950100002)(105586002)(68736007)(54356999)(110136004)(8676002)(6116002)(6246003)(81156014)(76176999)(38730400002)(102836003)(3846002)(6506006)(33656002)(50986999)(101416001)(5250100002)(2900100001)(6436002)(97736004)(4326008)(66066001)(478600001)(14454004)(93886004)(5660300001)(7696004)(189998001)(53546010)(99286003)(55016002)(2906002)(86362001)(8936002)(74316002)(9686003)(54906002)(25786009)(53936002);DIR:OUT;SFP:1101;SCL:1;SRVR:VI1PR0501MB2110;H:VI1PR0501MB2110.eurprd05.prod.outlook.com;FPR:;SPF:None;PTR:InfoNoRecords;MX:1;A:1;LANG:en; spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-originalarrivaltime: 10 Aug 2017 07:40:16.5244 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR0501MB2110 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by nfs id v7A7eT2R027077 Well, this definitely have pleased the little toy :) Thank you. I really appreciate your time and effort. If I may, one more newbie question. What do I need to do for the two patches to find their way into formal kernel code? Thanks -Ofer > -----Original Message----- > From: Peter Zijlstra [mailto:peterz@infradead.org] > Sent: Wednesday, August 9, 2017 6:35 PM > To: Ofer Levi(SW) > Cc: rusty@rustcorp.com.au; mingo@redhat.com; > Vineet.Gupta1@synopsys.com; linux-kernel@vger.kernel.org; Tejun Heo > > Subject: Re: hotplug support for arch/arc/plat-eznps platform > > On Wed, Aug 09, 2017 at 03:19:02PM +0000, Ofer Levi(SW) wrote: > > I appreciate your effort and detailed reply, however I'm still > > experiencing performance hit at partition_sched_domains(). It seems the > issue is due to the large magnitude of cpus. > > I used he suggested method 2, patched in the diffs and used the > > command line switch isolcpus to kill load-balancing. > > It did save few hundredth of a sec per cpu. When I limited number of > > available cpus (using present and possible cpus ) to 48, it did reduced > dramatically this function execution time: > > > > With 4K available cpus : > > [ 48.890000] ## CPU16 LIVE ##: Executing Code... > > [ 48.910000] partition_sched_domains start > > [ 49.360000] partition_sched_domains end > > > > With 48 available cpus: > > [ 36.950000] ## CPU16 LIVE ##: Executing Code... > > [ 36.950000] partition_sched_domains start > > [ 36.960000] partition_sched_domains end > > > > Note that I currently use kernel version: 4.8.0.17.0600.00.0000, if this has > any influence. > > Would appreciate your thoughts. > > > > Does something like this cure things? It seems we're doing a possible_cpus > iteration for sysctl cruft, and that will most certainly hurt on your little toy :-) > > Not sure what the more generic solution to that would be, but the below > avoids it for isolcpus. > > --- > --- a/kernel/sched/core.c > +++ b/kernel/sched/core.c > @@ -85,6 +85,7 @@ int sysctl_sched_rt_runtime = 950000; > > /* CPUs with isolated domains */ > cpumask_var_t cpu_isolated_map; > +cpumask_var_t non_isolated_cpus; > > /* > * __task_rq_lock - lock the rq @p resides on. > @@ -5685,8 +5686,6 @@ static inline void sched_init_smt(void) > > void __init sched_init_smp(void) > { > - cpumask_var_t non_isolated_cpus; > - > alloc_cpumask_var(&non_isolated_cpus, GFP_KERNEL); > > sched_init_numa(); > @@ -5697,17 +5696,17 @@ void __init sched_init_smp(void) > * happen. > */ > mutex_lock(&sched_domains_mutex); > - sched_init_domains(cpu_active_mask); > cpumask_andnot(non_isolated_cpus, cpu_possible_mask, > cpu_isolated_map); > if (cpumask_empty(non_isolated_cpus)) > cpumask_set_cpu(smp_processor_id(), non_isolated_cpus); > + > + sched_init_domains(cpu_active_mask); > mutex_unlock(&sched_domains_mutex); > > /* Move init over to a non-isolated CPU */ > if (set_cpus_allowed_ptr(current, non_isolated_cpus) < 0) > BUG(); > sched_init_granularity(); > - free_cpumask_var(non_isolated_cpus); > > init_sched_rt_class(); > init_sched_dl_class(); > --- a/kernel/sched/debug.c > +++ b/kernel/sched/debug.c > @@ -327,6 +327,8 @@ static struct ctl_table *sd_alloc_ctl_cp > return table; > } > > +extern cpumask_var_t non_isolated_cpus; > + > static struct ctl_table_header *sd_sysctl_header; void > register_sched_domain_sysctl(void) > { > @@ -340,7 +342,7 @@ void register_sched_domain_sysctl(void) > if (entry == NULL) > return; > > - for_each_possible_cpu(i) { > + for_each_cpu(i, non_isolated_cpus) { > snprintf(buf, 32, "cpu%d", i); > entry->procname = kstrdup(buf, GFP_KERNEL); > entry->mode = 0555;