From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-11.3 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH, MAILING_LIST_MULTI,NICE_REPLY_A,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4B529C64E7A for ; Tue, 17 Nov 2020 13:17:34 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id B3A472225B for ; Tue, 17 Nov 2020 13:17:31 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="tRCIV2cV" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B3A472225B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Type: Content-Transfer-Encoding:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:Date:Message-ID:From: References:To:Subject:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=BRYvY7Rm/56GjCZEDU/omQ+KSJBHp1TQzLAqnHRRlF4=; b=tRCIV2cVcE2/xHycvzcJ9a173 mij6HovwotrQkdk53nfUHVhTkjKpK8cwKiF2UJQTZUXIHsMmgLlul1PawDpDWWcC6+Ohr9ogqOsSb E963DlqJe6kA5qseXrIGh5D4zshdRbjzd+lcE/zJF4d6CC0Nm1k0mveh1rzP8zS1IVO7OGS6ouSvt PkDTJ3XIMhZl0HpV9dyGQHBYxfQs4s9J41fRoa1QrsjlXG1SzWKO3IATzp23W+jhWwONOUhs1XT63 QMPOTMYO0TSa7ZorR0KMF65hJEvWT8m9Q4rzUpV1d9LPvUBP0vRaZjhofUYRqZ31KPPIB6wab+hbe gBCBcB6Qw==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kf0q5-0000lt-QC; Tue, 17 Nov 2020 13:16:05 +0000 Received: from foss.arm.com ([217.140.110.172]) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kf0q2-0000kQ-SW for linux-arm-kernel@lists.infradead.org; Tue, 17 Nov 2020 13:16:04 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 854AA101E; Tue, 17 Nov 2020 05:16:00 -0800 (PST) Received: from [10.57.25.49] (unknown [10.57.25.49]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 8C0C63F719; Tue, 17 Nov 2020 05:15:58 -0800 (PST) Subject: Re: [PATCH v2 4/4] powercap/drivers/dtpm: Add CPU energy model based support To: Daniel Lezcano References: <20201116152649.11482-1-daniel.lezcano@linaro.org> <20201116152649.11482-5-daniel.lezcano@linaro.org> From: Lukasz Luba Message-ID: Date: Tue, 17 Nov 2020 13:15:56 +0000 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.9.0 MIME-Version: 1.0 In-Reply-To: <20201116152649.11482-5-daniel.lezcano@linaro.org> Content-Language: en-US X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201117_081603_078349_BAD68A4D X-CRM114-Status: GOOD ( 40.90 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: ulf.hansson@linaro.org, linux-pm@vger.kernel.org, rjw@rjwysocki.net, linux-kernel@vger.kernel.org, ilina@codeaurora.org, rkumbako@codeaurora.org, rui.zhang@intel.com, linux-arm-kernel@lists.infradead.org Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="us-ascii"; Format="flowed" Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Hi Daniel, Only one small comment regarding the setup of 'power_limit'. On 11/16/20 3:26 PM, Daniel Lezcano wrote: > With the powercap dtpm controller, we are able to plug devices with > power limitation features in the tree. > > The following patch introduces the CPU power limitation based on the > energy model and the performance states. > > The power limitation is done at the performance domain level. If some > CPUs are unplugged, the corresponding power will be subtracted from > the performance domain total power. > > It is up to the platform to initialize the dtpm tree and add the CPU. > > Here is an example to create a simple tree with one root node called > "pkg" and the CPU's performance domains. > > static int dtpm_register_pkg(struct dtpm_descr *descr) > { > struct dtpm *pkg; > int ret; > > pkg = dtpm_alloc(); > if (!pkg) > return -ENOMEM; > > ret = dtpm_register_parent(descr->name, pkg, descr->parent); > if (ret) > return ret; > > return dtpm_register_cpu(pkg); > } > > static struct dtpm_descr descr = { > .name = "pkg", > .init = dtpm_register_pkg, > }; > DTPM_DECLARE(descr); > > Signed-off-by: Daniel Lezcano > --- > drivers/powercap/Kconfig | 7 + > drivers/powercap/Makefile | 1 + > drivers/powercap/dtpm_cpu.c | 282 ++++++++++++++++++++++++++++++++++++ > include/linux/cpuhotplug.h | 1 + > include/linux/dtpm.h | 3 + > 5 files changed, 294 insertions(+) > create mode 100644 drivers/powercap/dtpm_cpu.c > > diff --git a/drivers/powercap/Kconfig b/drivers/powercap/Kconfig > index cc1953bd8bed..20b4325c6161 100644 > --- a/drivers/powercap/Kconfig > +++ b/drivers/powercap/Kconfig > @@ -49,4 +49,11 @@ config DTPM > help > This enables support for the power capping for the dynamic > thermal power management userspace engine. > + > +config DTPM_CPU > + bool "Add CPU power capping based on the energy model" > + depends on DTPM && ENERGY_MODEL > + help > + This enables support for CPU power limitation based on > + energy model. > endif > diff --git a/drivers/powercap/Makefile b/drivers/powercap/Makefile > index 6482ac52054d..fabcf388a8d3 100644 > --- a/drivers/powercap/Makefile > +++ b/drivers/powercap/Makefile > @@ -1,5 +1,6 @@ > # SPDX-License-Identifier: GPL-2.0-only > obj-$(CONFIG_DTPM) += dtpm.o > +obj-$(CONFIG_DTPM_CPU) += dtpm_cpu.o > obj-$(CONFIG_POWERCAP) += powercap_sys.o > obj-$(CONFIG_INTEL_RAPL_CORE) += intel_rapl_common.o > obj-$(CONFIG_INTEL_RAPL) += intel_rapl_msr.o > diff --git a/drivers/powercap/dtpm_cpu.c b/drivers/powercap/dtpm_cpu.c > new file mode 100644 > index 000000000000..6bff5f27d891 > --- /dev/null > +++ b/drivers/powercap/dtpm_cpu.c > @@ -0,0 +1,282 @@ > +// SPDX-License-Identifier: GPL-2.0-only > +/* > + * Copyright 2020 Linaro Limited > + * > + * Author: Daniel Lezcano > + * > + * The DTPM CPU is based on the energy model. It hooks the CPU in the > + * DTPM tree which in turns update the power number by propagating the > + * power number from the CPU energy model information to the parents. > + * > + * The association between the power and the performance state, allows > + * to set the power of the CPU at the OPP granularity. > + * > + * The CPU hotplug is supported and the power numbers will be updated > + * if a CPU is hot plugged / unplugged. > + */ > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > + > +static struct dtpm *__parent; > + > +static DEFINE_PER_CPU(struct dtpm *, dtpm_per_cpu); > + > +struct dtpm_cpu { > + struct freq_qos_request qos_req; > + int cpu; > +}; > + > +/* > + * When a new CPU is inserted at hotplug or boot time, add the power > + * contribution and update the dtpm tree. > + */ > +static int power_add(struct dtpm *dtpm, struct em_perf_domain *em) > +{ > + u64 power_min, power_max; > + > + power_min = em->table[0].power; > + power_min *= MICROWATT_PER_MILLIWATT; > + power_min += dtpm->power_min; > + > + power_max = em->table[em->nr_perf_states - 1].power; > + power_max *= MICROWATT_PER_MILLIWATT; > + power_max += dtpm->power_max; > + > + return dtpm_update_power(dtpm, power_min, power_max); > +} > + > +/* > + * When a CPU is unplugged, remove its power contribution from the > + * dtpm tree. > + */ > +static int power_sub(struct dtpm *dtpm, struct em_perf_domain *em) > +{ > + u64 power_min, power_max; > + > + power_min = em->table[0].power; > + power_min *= MICROWATT_PER_MILLIWATT; > + power_min = dtpm->power_min - power_min; > + > + power_max = em->table[em->nr_perf_states - 1].power; > + power_max *= MICROWATT_PER_MILLIWATT; > + power_max = dtpm->power_max - power_max; > + > + return dtpm_update_power(dtpm, power_min, power_max); > +} > + > +static int set_pd_power_limit(struct powercap_zone *pcz, int cid, > + u64 power_limit) > +{ > + struct dtpm *dtpm = to_dtpm(pcz); > + struct dtpm_cpu *dtpm_cpu = dtpm->private; > + struct em_perf_domain *pd; > + struct cpumask cpus; > + unsigned long freq; > + u64 power; > + int i, nr_cpus; > + > + spin_lock(&dtpm->lock); > + > + power_limit = clamp_val(power_limit, dtpm->power_min, dtpm->power_max); > + > + pd = em_cpu_get(dtpm_cpu->cpu); > + > + cpumask_and(&cpus, cpu_online_mask, to_cpumask(pd->cpus)); > + > + nr_cpus = cpumask_weight(&cpus); > + > + for (i = 0; i < pd->nr_perf_states; i++) { > + > + power = pd->table[i].power * MICROWATT_PER_MILLIWATT * nr_cpus; > + > + if (power > power_limit) > + break; > + } > + > + freq = pd->table[i - 1].frequency; > + > + dtpm->power_limit = pd->table[i - 1].power * > + MICROWATT_PER_MILLIWATT * nr_cpus; > + > + spin_unlock(&dtpm->lock); > + > + freq_qos_update_request(&dtpm_cpu->qos_req, freq); > + > + return 0; > +} > + > +static int get_pd_power_limit(struct powercap_zone *pcz, int cid, u64 *data) > +{ > + struct dtpm *dtpm = to_dtpm(pcz); > + > + spin_lock(&dtpm->lock); > + *data = dtpm->power_limit; > + spin_unlock(&dtpm->lock); > + > + return 0; > +} > + > +static int get_pd_power_uw(struct powercap_zone *pcz, u64 *power_uw) > +{ > + struct dtpm *dtpm = to_dtpm(pcz); > + struct dtpm_cpu *dtpm_cpu = dtpm->private; > + struct em_perf_domain *pd; > + unsigned long freq; > + int i, nr_cpus; > + > + freq = cpufreq_quick_get(dtpm_cpu->cpu); > + pd = em_cpu_get(dtpm_cpu->cpu); > + nr_cpus = cpumask_weight(to_cpumask(pd->cpus)); > + > + for (i = 0; i < pd->nr_perf_states; i++) { > + > + if (pd->table[i].frequency < freq) > + continue; > + > + *power_uw = pd->table[i].power * > + MICROWATT_PER_MILLIWATT * nr_cpus; > + > + return 0; > + } > + > + return -EINVAL; > +} > + > +static int cpu_release_zone(struct powercap_zone *pcz) > +{ > + struct dtpm *dtpm = to_dtpm(pcz); > + struct dtpm_cpu *dtpm_cpu = dtpm->private; > + > + freq_qos_remove_request(&dtpm_cpu->qos_req); > + kfree(dtpm_cpu); > + > + return dtpm_release_zone(pcz); > +} > + > +static struct powercap_zone_constraint_ops pd_constraint_ops = { > + .set_power_limit_uw = set_pd_power_limit, > + .get_power_limit_uw = get_pd_power_limit, > +}; > + > +static struct powercap_zone_ops pd_zone_ops = { > + .get_power_uw = get_pd_power_uw, > + .release = cpu_release_zone, > +}; > + > +static int cpuhp_dtpm_cpu_offline(unsigned int cpu) > +{ > + struct cpufreq_policy *policy; > + struct em_perf_domain *pd; > + struct dtpm *dtpm; > + > + policy = cpufreq_cpu_get(cpu); > + > + if (!policy) > + return 0; > + > + pd = em_cpu_get(cpu); > + if (!pd) > + return -EINVAL; > + > + dtpm = per_cpu(dtpm_per_cpu, cpu); > + > + power_sub(dtpm, pd); > + > + if (cpumask_weight(policy->cpus) != 1) > + return 0; > + > + for_each_cpu(cpu, policy->related_cpus) > + per_cpu(dtpm_per_cpu, cpu) = NULL; > + > + dtpm_unregister(dtpm); > + > + return 0; > +} > + > +static int cpuhp_dtpm_cpu_online(unsigned int cpu) > +{ > + struct dtpm *dtpm; > + struct dtpm_cpu *dtpm_cpu; > + struct cpufreq_policy *policy; > + struct em_perf_domain *pd; > + char name[CPUFREQ_NAME_LEN]; > + int ret; > + > + policy = cpufreq_cpu_get(cpu); > + > + if (!policy) > + return 0; > + > + pd = em_cpu_get(cpu); > + if (!pd) > + return -EINVAL; > + > + dtpm = per_cpu(dtpm_per_cpu, cpu); > + if (dtpm) > + return power_add(dtpm, pd); The dtpm->power_limit is not incremented in this path, when a new CPU joins the cluster. Is it correct? Or maybe we need something like: ------------------------------>8--------------------- if (dtpm) { ret = power_add(dtpm, pd); if (!ret) dtpm->power_limit = dtpm->power_max; return ret; } ------------------------8<--------------- The power_max should be updated after successful power_add(). It would disturb user set value in power_limit, though (described below). > + > + dtpm = dtpm_alloc(); > + if (!dtpm) > + return -EINVAL; > + > + dtpm_cpu = kzalloc(sizeof(dtpm_cpu), GFP_KERNEL); > + if (!dtpm_cpu) { > + kfree(dtpm); > + return -ENOMEM; > + } > + > + dtpm->private = dtpm_cpu; > + dtpm_cpu->cpu = cpu; > + > + for_each_cpu(cpu, policy->related_cpus) > + per_cpu(dtpm_per_cpu, cpu) = dtpm; > + > + ret = power_add(dtpm, pd); > + if (ret) > + goto out_kfree_dtpm_cpu; > + > + dtpm->power_limit = dtpm->power_max; Here, the power_limit will be set only once with power_max for a single CPU. I am not sure, but maybe we can simple say: dtpm->power_limit = dtpm->power_max * cpumask_weight(policy->related_cpus) an avoid touching it later (?) Because this function can be called in runtime, when the power_limit was already set by userspace, the hotpluging in/out/in... CPU shouldn't change this limit. Regards, Lukasz _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel