From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.8 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 16CABC433DF for ; Thu, 9 Jul 2020 10:14:20 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id DF6D520663 for ; Thu, 9 Jul 2020 10:14:19 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="OmDlpJ0f" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726605AbgGIKOR (ORCPT ); Thu, 9 Jul 2020 06:14:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35316 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726897AbgGIKOM (ORCPT ); Thu, 9 Jul 2020 06:14:12 -0400 Received: from mail-pl1-x641.google.com (mail-pl1-x641.google.com [IPv6:2607:f8b0:4864:20::641]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DD836C061A0B for ; Thu, 9 Jul 2020 03:14:11 -0700 (PDT) Received: by mail-pl1-x641.google.com with SMTP id d10so654801pls.5 for ; Thu, 09 Jul 2020 03:14:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=yaA0NjXT9ejZD1re0oAP+uDPoFeFqxhvljIg3jQh5Fg=; b=OmDlpJ0f/w2TVYbbJkMRYc+V1bqmuarLLa/EookrkuDVPdoxPOaoTwW2FdIo6o18Qq Y+f7Ph0V8W36eKNfyxE7Un2XDMm3T84cN37TqFgz7ABiKCscy8ezDUBDx4fQupfS91nf Y3q6n/AVJfmt1rtTTKaHkRVv0fpTAeqPdSwvSaMxnvC4ei7816Eobex9wc6X9aqOjHH6 rc1avRejayCXlWqTvHDnp0tg6NUjMBqXqbAvvUhLapddHReM7Q53MLxnz+9iqJQSItsQ 5EohM8Mt8Pof2Qtt4SNLRxX4EFpkKCHnlxYifo5cuHuohDjqF+frQPpsGDDJh5htpIhv RZPg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=yaA0NjXT9ejZD1re0oAP+uDPoFeFqxhvljIg3jQh5Fg=; b=tC3s2/W52mDZqjKX/ZcWXkWIKfFEt4zoNA+YjpP5p18Kr6sD8GFnXP6fbl3w9XEkwl 5APpJx94g1459aIqPD7Ndo0eCMO0qr19K8BVFFgl7fDnqizD6c0hx328FJoyaUgG/mmo T1oR8fFzEjaqTbyt3M9UzHz7jwn9pVY4ZtbWzc4mjkzoe2B8pIdJR6uWj/yNUxQuVNDb qF36cH9Z15/CKz+0a6BqiDxkiJql4uR5PVnT0Ux09gSsed78U7BQ9zpIkVQ9rUhdbgts X+Q/CQkBByGIeyxJFhiVjwTI3klU38e6oCOjkXcZlUnZyL+9BXlSTAN/unm3eDK9TJcc TzLQ== X-Gm-Message-State: AOAM533+a1/QWV6kPuKaR01MJaoxJpycS0vJC3sKcSXKVXGuK5e5sFIB tx/bcmvTH/GXuOo7vtB3nhy3Vg== X-Google-Smtp-Source: ABdhPJy8HyrcoQ6VzKHy2wCN8zHGO7OHZaG7NF7HUVcRbgP1JwrDTgqUnW0Scs6eadbC6ftvWS1FLQ== X-Received: by 2002:a17:902:a40c:: with SMTP id p12mr43077013plq.51.1594289651367; Thu, 09 Jul 2020 03:14:11 -0700 (PDT) Received: from localhost ([122.172.40.201]) by smtp.gmail.com with ESMTPSA id w1sm2135716pfq.53.2020.07.09.03.14.10 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 09 Jul 2020 03:14:10 -0700 (PDT) From: Viresh Kumar To: Catalin Marinas , Will Deacon , Sudeep Holla , Greg Kroah-Hartman , "Rafael J. Wysocki" Cc: Viresh Kumar , Ionela Voinescu , Peter Puhov , Vincent Guittot , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: [RFC 2/3] topology: Provide generic implementation of arch_freq_counters_available() Date: Thu, 9 Jul 2020 15:43:34 +0530 Message-Id: X-Mailer: git-send-email 2.25.0.rc1.19.g042ed3e048af In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org arch_freq_counters_available() is implemented only for ARM AMU hardware right now and this patch attempts to make it generic enough so other parts of the kernel can also register their own implementation of topology_scale_freq_tick() routine. Signed-off-by: Viresh Kumar --- arch/arm64/include/asm/topology.h | 7 --- arch/arm64/kernel/topology.c | 81 +++++++++++++++---------------- drivers/base/arch_topology.c | 43 ++++++++++++++-- include/linux/arch_topology.h | 5 +- 4 files changed, 81 insertions(+), 55 deletions(-) diff --git a/arch/arm64/include/asm/topology.h b/arch/arm64/include/asm/topology.h index 0cc835ddfcd1..93cca6a8cde3 100644 --- a/arch/arm64/include/asm/topology.h +++ b/arch/arm64/include/asm/topology.h @@ -16,14 +16,7 @@ int pcibus_to_node(struct pci_bus *bus); #include -#ifdef CONFIG_ARM64_AMU_EXTN -/* - * Replace task scheduler's default counter-based - * frequency-invariance scale factor setting. - */ -void topology_scale_freq_tick(void); #define arch_scale_freq_tick topology_scale_freq_tick -#endif /* CONFIG_ARM64_AMU_EXTN */ /* Replace task scheduler's default frequency-invariant accounting */ #define arch_scale_freq_capacity topology_get_freq_scale diff --git a/arch/arm64/kernel/topology.c b/arch/arm64/kernel/topology.c index 74fde35b56ef..97741da31b6d 100644 --- a/arch/arm64/kernel/topology.c +++ b/arch/arm64/kernel/topology.c @@ -188,6 +188,41 @@ bool amu_counters_supported(void) cpumask_equal(valid_cpus, cpu_present_mask); } +void amu_scale_freq_tick(void) +{ + u64 prev_core_cnt, prev_const_cnt; + u64 core_cnt, const_cnt, scale; + + const_cnt = read_sysreg_s(SYS_AMEVCNTR0_CONST_EL0); + core_cnt = read_sysreg_s(SYS_AMEVCNTR0_CORE_EL0); + prev_const_cnt = this_cpu_read(arch_const_cycles_prev); + prev_core_cnt = this_cpu_read(arch_core_cycles_prev); + + if (unlikely(core_cnt <= prev_core_cnt || + const_cnt <= prev_const_cnt)) + goto store_and_exit; + + /* + * /\core arch_max_freq_scale + * scale = ------- * -------------------- + * /\const SCHED_CAPACITY_SCALE + * + * See setup_freq_invariance() for details on + * arch_max_freq_scale and the use of SCHED_CAPACITY_SHIFT. + */ + scale = core_cnt - prev_core_cnt; + scale *= this_cpu_read(arch_max_freq_scale); + scale = div64_u64(scale >> SCHED_CAPACITY_SHIFT, + const_cnt - prev_const_cnt); + + scale = min_t(unsigned long, scale, SCHED_CAPACITY_SCALE); + this_cpu_write(freq_scale, (unsigned long)scale); + +store_and_exit: + this_cpu_write(arch_core_cycles_prev, core_cnt); + this_cpu_write(arch_const_cycles_prev, const_cnt); +} + static int __init early_init_amu_fie(void) { int cpu; @@ -230,9 +265,11 @@ static int __init late_init_amu_fie(void) if (!cpumask_empty(amu_fie_cpus)) { pr_info("CPUs[%*pbl]: counters will be used for FIE.", cpumask_pr_args(amu_fie_cpus)); - static_branch_enable(&amu_fie_key); + if (!topology_set_scale_freq_tick(amu_scale_freq_tick, amu_fie_cpus)) + pr_info("Registered amu_scale_freq_tick()\n"); } + free_cpumask_var(amu_fie_cpus); return 0; } late_initcall_sync(late_init_amu_fie); @@ -242,48 +279,6 @@ bool arch_freq_counters_available(struct cpumask *cpus) return amu_freq_invariant() && cpumask_subset(cpus, amu_fie_cpus); } - -void topology_scale_freq_tick(void) -{ - u64 prev_core_cnt, prev_const_cnt; - u64 core_cnt, const_cnt, scale; - int cpu = smp_processor_id(); - - if (!amu_freq_invariant()) - return; - - if (!cpumask_test_cpu(cpu, amu_fie_cpus)) - return; - - const_cnt = read_sysreg_s(SYS_AMEVCNTR0_CONST_EL0); - core_cnt = read_sysreg_s(SYS_AMEVCNTR0_CORE_EL0); - prev_const_cnt = this_cpu_read(arch_const_cycles_prev); - prev_core_cnt = this_cpu_read(arch_core_cycles_prev); - - if (unlikely(core_cnt <= prev_core_cnt || - const_cnt <= prev_const_cnt)) - goto store_and_exit; - - /* - * /\core arch_max_freq_scale - * scale = ------- * -------------------- - * /\const SCHED_CAPACITY_SCALE - * - * See setup_freq_invariance() for details on - * arch_max_freq_scale and the use of SCHED_CAPACITY_SHIFT. - */ - scale = core_cnt - prev_core_cnt; - scale *= this_cpu_read(arch_max_freq_scale); - scale = div64_u64(scale >> SCHED_CAPACITY_SHIFT, - const_cnt - prev_const_cnt); - - scale = min_t(unsigned long, scale, SCHED_CAPACITY_SCALE); - this_cpu_write(freq_scale, (unsigned long)scale); - -store_and_exit: - this_cpu_write(arch_core_cycles_prev, core_cnt); - this_cpu_write(arch_const_cycles_prev, const_cnt); -} #else bool amu_counters_supported(void) { diff --git a/drivers/base/arch_topology.c b/drivers/base/arch_topology.c index 4d0a0038b476..3820109864c1 100644 --- a/drivers/base/arch_topology.c +++ b/drivers/base/arch_topology.c @@ -21,11 +21,48 @@ #include #include -__weak bool arch_freq_counters_available(struct cpumask *cpus) +static void (*scale_freq_tick)(void); +static cpumask_var_t freq_tick_cpus; + +int topology_set_scale_freq_tick(void *ptr, const struct cpumask *cpus) { - return false; + if (scale_freq_tick) + return -EBUSY; + + if (!zalloc_cpumask_var(&freq_tick_cpus, GFP_KERNEL)) + return -ENOMEM; + + cpumask_copy(freq_tick_cpus, cpus); + scale_freq_tick = ptr; + return 0; } +EXPORT_SYMBOL_GPL(topology_set_scale_freq_tick); + +void topology_remove_scale_freq_tick(void *ptr) +{ + if (scale_freq_tick == ptr) { + free_cpumask_var(freq_tick_cpus); + scale_freq_tick = NULL; + } else { + pr_err("%s: Invalid argument\n", __func__); + } +} +EXPORT_SYMBOL_GPL(topology_remove_scale_freq_tick); + +void topology_scale_freq_tick(void) +{ + if (scale_freq_tick && + cpumask_test_cpu(raw_smp_processor_id(), freq_tick_cpus)) + scale_freq_tick(); +} + +static bool support_scale_freq_tick(struct cpumask *cpus) +{ + return scale_freq_tick && cpumask_subset(cpus, freq_tick_cpus); +} + DEFINE_PER_CPU(unsigned long, freq_scale) = SCHED_CAPACITY_SCALE; +EXPORT_SYMBOL_GPL(freq_scale); void arch_set_freq_scale(struct cpumask *cpus, unsigned long cur_freq, unsigned long max_freq) @@ -38,7 +75,7 @@ void arch_set_freq_scale(struct cpumask *cpus, unsigned long cur_freq, * want to update the scale factor with information from CPUFREQ. * Instead the scale factor will be updated from arch_scale_freq_tick. */ - if (arch_freq_counters_available(cpus)) + if (support_scale_freq_tick(cpus)) return; scale = (cur_freq << SCHED_CAPACITY_SHIFT) / max_freq; diff --git a/include/linux/arch_topology.h b/include/linux/arch_topology.h index 0566cb3314ef..6305148c8aa0 100644 --- a/include/linux/arch_topology.h +++ b/include/linux/arch_topology.h @@ -10,6 +10,9 @@ void topology_normalize_cpu_scale(void); int topology_update_cpu_topology(void); +void topology_scale_freq_tick(void); +int topology_set_scale_freq_tick(void *ptr, const struct cpumask *cpus); +void topology_remove_scale_freq_tick(void *ptr); struct device_node; bool topology_parse_cpu_capacity(struct device_node *cpu_node, int cpu); @@ -30,8 +33,6 @@ static inline unsigned long topology_get_freq_scale(int cpu) return per_cpu(freq_scale, cpu); } -bool arch_freq_counters_available(struct cpumask *cpus); - DECLARE_PER_CPU(unsigned long, thermal_pressure); static inline unsigned long topology_get_thermal_pressure(int cpu) -- 2.25.0.rc1.19.g042ed3e048af From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.8 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3491DC433E0 for ; Thu, 9 Jul 2020 10:15:55 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 06EF420674 for ; Thu, 9 Jul 2020 10:15:55 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="J3oe3hOG"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="OmDlpJ0f" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 06EF420674 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=QyNx3aU9ledWm9yavXcDhzvxS2nLt2gnFFC1OCUrLys=; b=J3oe3hOGbCDLWank0U+26JUZ0 xpaKZZ93NHjaLFhRjEcMGo1KCFXTkriIoHPBM7lYHRR/kpTD2vEeEqJIGRC320KyBCMDxIOmUGxKG ZmwtD+c2knhBuLPcKBByhIURf6h+5ZgNeaE//gSYiFOnVzMyXryK6bbU9MdeRxoXL7pziLiQYQ/kD 8EM8jsn60SLVygPva571sS5XgW/XconB+ebr2QdByGDVdW4p6O+ZDiakiWZSZLz8pRy33fV9YF24a 8Zhx4t60ekayGqyxTx2tEOt10Ze4Vna2GDYgNOwHcguHjGTrSVsFpidwgLvUAr9niL6a+ZzrFXt83 HBc36lqZQ==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jtTZM-0004ZM-4x; Thu, 09 Jul 2020 10:14:20 +0000 Received: from mail-pl1-x644.google.com ([2607:f8b0:4864:20::644]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1jtTZF-0004WO-Bt for linux-arm-kernel@lists.infradead.org; Thu, 09 Jul 2020 10:14:14 +0000 Received: by mail-pl1-x644.google.com with SMTP id b9so652329plx.6 for ; Thu, 09 Jul 2020 03:14:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=yaA0NjXT9ejZD1re0oAP+uDPoFeFqxhvljIg3jQh5Fg=; b=OmDlpJ0f/w2TVYbbJkMRYc+V1bqmuarLLa/EookrkuDVPdoxPOaoTwW2FdIo6o18Qq Y+f7Ph0V8W36eKNfyxE7Un2XDMm3T84cN37TqFgz7ABiKCscy8ezDUBDx4fQupfS91nf Y3q6n/AVJfmt1rtTTKaHkRVv0fpTAeqPdSwvSaMxnvC4ei7816Eobex9wc6X9aqOjHH6 rc1avRejayCXlWqTvHDnp0tg6NUjMBqXqbAvvUhLapddHReM7Q53MLxnz+9iqJQSItsQ 5EohM8Mt8Pof2Qtt4SNLRxX4EFpkKCHnlxYifo5cuHuohDjqF+frQPpsGDDJh5htpIhv RZPg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=yaA0NjXT9ejZD1re0oAP+uDPoFeFqxhvljIg3jQh5Fg=; b=FdTaj8QGJWkbVxX8Sd1FOQ14XpuvT0i1KlcJa3g7oaYkcf8+xPsFRRvsdNWifnzb5j AunBWrCRDVfX1iw/mOHdN7iO3763jMGtzVum3CN8C4D8F5Qdqbc8tCcC+1xq5SnIIaZv EiHbtgxHTQB2MhDcFJJjkSArnomPQG12fpTxrt8BMbAbvW433EIYcGbia6rrYPlNz7T7 8JcO5/ZmHTd00gpF2u8B1yyUmUPxhn6qOIT5ML6rNHwV11mmRcBSZG7fI7k6D7Uel6Wa u+qj9G33gOQFVCCucgpXodvaQw4X/l4qk/Klv0JZhgdg75GeoQbx066loW+lNzsK1ZRi +UxQ== X-Gm-Message-State: AOAM533sbUm0z+kSLvMhsxeFHdkmV1NuGKGGyalKiqqV+DgvCKCs+Wh1 knoIH9UxfMrNdVABSLi3pSjFfQ== X-Google-Smtp-Source: ABdhPJy8HyrcoQ6VzKHy2wCN8zHGO7OHZaG7NF7HUVcRbgP1JwrDTgqUnW0Scs6eadbC6ftvWS1FLQ== X-Received: by 2002:a17:902:a40c:: with SMTP id p12mr43077013plq.51.1594289651367; Thu, 09 Jul 2020 03:14:11 -0700 (PDT) Received: from localhost ([122.172.40.201]) by smtp.gmail.com with ESMTPSA id w1sm2135716pfq.53.2020.07.09.03.14.10 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 09 Jul 2020 03:14:10 -0700 (PDT) From: Viresh Kumar To: Catalin Marinas , Will Deacon , Sudeep Holla , Greg Kroah-Hartman , "Rafael J. Wysocki" Subject: [RFC 2/3] topology: Provide generic implementation of arch_freq_counters_available() Date: Thu, 9 Jul 2020 15:43:34 +0530 Message-Id: X-Mailer: git-send-email 2.25.0.rc1.19.g042ed3e048af In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200709_061413_421087_E0DE9E24 X-CRM114-Status: GOOD ( 19.36 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Vincent Guittot , Viresh Kumar , linux-kernel@vger.kernel.org, Peter Puhov , Ionela Voinescu , linux-arm-kernel@lists.infradead.org Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org arch_freq_counters_available() is implemented only for ARM AMU hardware right now and this patch attempts to make it generic enough so other parts of the kernel can also register their own implementation of topology_scale_freq_tick() routine. Signed-off-by: Viresh Kumar --- arch/arm64/include/asm/topology.h | 7 --- arch/arm64/kernel/topology.c | 81 +++++++++++++++---------------- drivers/base/arch_topology.c | 43 ++++++++++++++-- include/linux/arch_topology.h | 5 +- 4 files changed, 81 insertions(+), 55 deletions(-) diff --git a/arch/arm64/include/asm/topology.h b/arch/arm64/include/asm/topology.h index 0cc835ddfcd1..93cca6a8cde3 100644 --- a/arch/arm64/include/asm/topology.h +++ b/arch/arm64/include/asm/topology.h @@ -16,14 +16,7 @@ int pcibus_to_node(struct pci_bus *bus); #include -#ifdef CONFIG_ARM64_AMU_EXTN -/* - * Replace task scheduler's default counter-based - * frequency-invariance scale factor setting. - */ -void topology_scale_freq_tick(void); #define arch_scale_freq_tick topology_scale_freq_tick -#endif /* CONFIG_ARM64_AMU_EXTN */ /* Replace task scheduler's default frequency-invariant accounting */ #define arch_scale_freq_capacity topology_get_freq_scale diff --git a/arch/arm64/kernel/topology.c b/arch/arm64/kernel/topology.c index 74fde35b56ef..97741da31b6d 100644 --- a/arch/arm64/kernel/topology.c +++ b/arch/arm64/kernel/topology.c @@ -188,6 +188,41 @@ bool amu_counters_supported(void) cpumask_equal(valid_cpus, cpu_present_mask); } +void amu_scale_freq_tick(void) +{ + u64 prev_core_cnt, prev_const_cnt; + u64 core_cnt, const_cnt, scale; + + const_cnt = read_sysreg_s(SYS_AMEVCNTR0_CONST_EL0); + core_cnt = read_sysreg_s(SYS_AMEVCNTR0_CORE_EL0); + prev_const_cnt = this_cpu_read(arch_const_cycles_prev); + prev_core_cnt = this_cpu_read(arch_core_cycles_prev); + + if (unlikely(core_cnt <= prev_core_cnt || + const_cnt <= prev_const_cnt)) + goto store_and_exit; + + /* + * /\core arch_max_freq_scale + * scale = ------- * -------------------- + * /\const SCHED_CAPACITY_SCALE + * + * See setup_freq_invariance() for details on + * arch_max_freq_scale and the use of SCHED_CAPACITY_SHIFT. + */ + scale = core_cnt - prev_core_cnt; + scale *= this_cpu_read(arch_max_freq_scale); + scale = div64_u64(scale >> SCHED_CAPACITY_SHIFT, + const_cnt - prev_const_cnt); + + scale = min_t(unsigned long, scale, SCHED_CAPACITY_SCALE); + this_cpu_write(freq_scale, (unsigned long)scale); + +store_and_exit: + this_cpu_write(arch_core_cycles_prev, core_cnt); + this_cpu_write(arch_const_cycles_prev, const_cnt); +} + static int __init early_init_amu_fie(void) { int cpu; @@ -230,9 +265,11 @@ static int __init late_init_amu_fie(void) if (!cpumask_empty(amu_fie_cpus)) { pr_info("CPUs[%*pbl]: counters will be used for FIE.", cpumask_pr_args(amu_fie_cpus)); - static_branch_enable(&amu_fie_key); + if (!topology_set_scale_freq_tick(amu_scale_freq_tick, amu_fie_cpus)) + pr_info("Registered amu_scale_freq_tick()\n"); } + free_cpumask_var(amu_fie_cpus); return 0; } late_initcall_sync(late_init_amu_fie); @@ -242,48 +279,6 @@ bool arch_freq_counters_available(struct cpumask *cpus) return amu_freq_invariant() && cpumask_subset(cpus, amu_fie_cpus); } - -void topology_scale_freq_tick(void) -{ - u64 prev_core_cnt, prev_const_cnt; - u64 core_cnt, const_cnt, scale; - int cpu = smp_processor_id(); - - if (!amu_freq_invariant()) - return; - - if (!cpumask_test_cpu(cpu, amu_fie_cpus)) - return; - - const_cnt = read_sysreg_s(SYS_AMEVCNTR0_CONST_EL0); - core_cnt = read_sysreg_s(SYS_AMEVCNTR0_CORE_EL0); - prev_const_cnt = this_cpu_read(arch_const_cycles_prev); - prev_core_cnt = this_cpu_read(arch_core_cycles_prev); - - if (unlikely(core_cnt <= prev_core_cnt || - const_cnt <= prev_const_cnt)) - goto store_and_exit; - - /* - * /\core arch_max_freq_scale - * scale = ------- * -------------------- - * /\const SCHED_CAPACITY_SCALE - * - * See setup_freq_invariance() for details on - * arch_max_freq_scale and the use of SCHED_CAPACITY_SHIFT. - */ - scale = core_cnt - prev_core_cnt; - scale *= this_cpu_read(arch_max_freq_scale); - scale = div64_u64(scale >> SCHED_CAPACITY_SHIFT, - const_cnt - prev_const_cnt); - - scale = min_t(unsigned long, scale, SCHED_CAPACITY_SCALE); - this_cpu_write(freq_scale, (unsigned long)scale); - -store_and_exit: - this_cpu_write(arch_core_cycles_prev, core_cnt); - this_cpu_write(arch_const_cycles_prev, const_cnt); -} #else bool amu_counters_supported(void) { diff --git a/drivers/base/arch_topology.c b/drivers/base/arch_topology.c index 4d0a0038b476..3820109864c1 100644 --- a/drivers/base/arch_topology.c +++ b/drivers/base/arch_topology.c @@ -21,11 +21,48 @@ #include #include -__weak bool arch_freq_counters_available(struct cpumask *cpus) +static void (*scale_freq_tick)(void); +static cpumask_var_t freq_tick_cpus; + +int topology_set_scale_freq_tick(void *ptr, const struct cpumask *cpus) { - return false; + if (scale_freq_tick) + return -EBUSY; + + if (!zalloc_cpumask_var(&freq_tick_cpus, GFP_KERNEL)) + return -ENOMEM; + + cpumask_copy(freq_tick_cpus, cpus); + scale_freq_tick = ptr; + return 0; } +EXPORT_SYMBOL_GPL(topology_set_scale_freq_tick); + +void topology_remove_scale_freq_tick(void *ptr) +{ + if (scale_freq_tick == ptr) { + free_cpumask_var(freq_tick_cpus); + scale_freq_tick = NULL; + } else { + pr_err("%s: Invalid argument\n", __func__); + } +} +EXPORT_SYMBOL_GPL(topology_remove_scale_freq_tick); + +void topology_scale_freq_tick(void) +{ + if (scale_freq_tick && + cpumask_test_cpu(raw_smp_processor_id(), freq_tick_cpus)) + scale_freq_tick(); +} + +static bool support_scale_freq_tick(struct cpumask *cpus) +{ + return scale_freq_tick && cpumask_subset(cpus, freq_tick_cpus); +} + DEFINE_PER_CPU(unsigned long, freq_scale) = SCHED_CAPACITY_SCALE; +EXPORT_SYMBOL_GPL(freq_scale); void arch_set_freq_scale(struct cpumask *cpus, unsigned long cur_freq, unsigned long max_freq) @@ -38,7 +75,7 @@ void arch_set_freq_scale(struct cpumask *cpus, unsigned long cur_freq, * want to update the scale factor with information from CPUFREQ. * Instead the scale factor will be updated from arch_scale_freq_tick. */ - if (arch_freq_counters_available(cpus)) + if (support_scale_freq_tick(cpus)) return; scale = (cur_freq << SCHED_CAPACITY_SHIFT) / max_freq; diff --git a/include/linux/arch_topology.h b/include/linux/arch_topology.h index 0566cb3314ef..6305148c8aa0 100644 --- a/include/linux/arch_topology.h +++ b/include/linux/arch_topology.h @@ -10,6 +10,9 @@ void topology_normalize_cpu_scale(void); int topology_update_cpu_topology(void); +void topology_scale_freq_tick(void); +int topology_set_scale_freq_tick(void *ptr, const struct cpumask *cpus); +void topology_remove_scale_freq_tick(void *ptr); struct device_node; bool topology_parse_cpu_capacity(struct device_node *cpu_node, int cpu); @@ -30,8 +33,6 @@ static inline unsigned long topology_get_freq_scale(int cpu) return per_cpu(freq_scale, cpu); } -bool arch_freq_counters_available(struct cpumask *cpus); - DECLARE_PER_CPU(unsigned long, thermal_pressure); static inline unsigned long topology_get_thermal_pressure(int cpu) -- 2.25.0.rc1.19.g042ed3e048af _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel