* [PATCH v3 0/2] Enable EAS for CPPC/ACPI based systems @ 2022-04-25 12:38 Pierre Gondois 2022-04-25 12:38 ` [PATCH v3 1/2] cpufreq: CPPC: Add per_cpu efficiency_class Pierre Gondois ` (2 more replies) 0 siblings, 3 replies; 9+ messages in thread From: Pierre Gondois @ 2022-04-25 12:38 UTC (permalink / raw) To: linux-kernel Cc: Ionela.Voinescu, Lukasz.Luba, Morten.Rasmussen, Dietmar.Eggemann, maz, daniel.lezcano, Pierre Gondois, Catalin Marinas, Will Deacon, Rafael J. Wysocki, Viresh Kumar, Mark Rutland, Ard Biesheuvel, Fuad Tabba, Valentin Schneider, Rob Herring, Lee Jones, linux-arm-kernel, linux-pm v3: - Remove efficiency_class_populated variable. [Viresh] - Remove patch "cpufreq: CPPC: Add cppc_cpufreq_search_cpu_data" and access cpu_data through policy->driver_data. [Viresh] - arm64 code only acked by Catalin [Catalin] v2: - Remove inline hint of cppc_cpufreq_search_cpu_data(). [Mark] - Use EXPORT_SYMBOL_GPL() instead of EXPORT_SYMBOL(). [Mark] - Use a bitmap to squeeze CPU efficiency class values. [Mark] 0. Overview The current Energy Model (EM) for CPUs requires knowledge about CPU performance states and their power consumption. Both of these information is not available for ACPI based systems. In ACPI, describing power efficiency of CPUs can be done through the following arm specific field: ACPI 6.4, s5.2.12.14 "GIC CPU Interface (GICC) Structure", "Processor Power Efficiency Class field": Describes the relative power efficiency of the associated pro- cessor. Lower efficiency class numbers are more efficient than higher ones (e.g. efficiency class 0 should be treated as more efficient than efficiency class 1). However, absolute values of this number have no meaning: 2 isn't necessarily half as efficient as 1. Add an 'efficiency_class' field to describe the relative power efficiency of CPUs. CPUs relying on this field will have performance states (power and frequency values) artificially created. Such EM will be referred to as an artificial EM. The artificial EM is used for the CPPC driver. 1. Dependencies This patch-set is based on linux-next. 2. Testing This patch-set has been tested on a Juno-r2 and a Pixel4. Two types of tests were done: energy testing, and performance testing. The energy testing was done with 2 sets of tasks: - homogeneous tasks (#Tasks at 5% utilization and 16ms period) - heterogeneous tasks (#Tasks at 5|10|15% utilization and 16ms period). If a test has 3 tasks, then there is one with each utilization (1 at 5%, 1 at 10%, 1 at 15%). Tasks spawn on the biggest CPU(s) of the platform. If there are multiple big CPUs, tasks spawn alternatively on big CPUs. 2.1. Juno-r2 testing The Juno-r2 has 6 CPUs: - 4 little [0, 3-5], max_capa=383 - 2 big [1-2], max_capa=1024 Base kernel is v5.17-rc5. 2.1.1. Energy testing The tests were done on: - a system using a DT and the scmi cpufreq driver. Comparison is done between no-EAS and EAS. - a system using ACPI and the cppc cpufreq driver. Comparison is done between CPPC-no-EAS and CPPC-EAS. CPPC-EAS uses the artificial EM. Energy numbers come from the Juno energy counter, by summing little and big clusters energy spending. There has been 5 iterations of each test. Lower energy spending is better. 2.1.1.1. Homogeneous tasks Energy results (Joules): +--------+-------------------+-----------------------------+ | | no-EAS | EAS | +--------+---------+---------+-------------------+---------+ | #Tasks | Mean | ci(+/-) | Mean | ci(+/-) | +--------+---------+---------+-------------------+---------+ | 10 | 7.89 | 0.26 | 6.99 (-11.36) | 0.49 | | 20 | 13.42 | 0.32 | 13.42 ( -0.02) | 0.08 | | 30 | 21.43 | 0.98 | 21.62 ( +0.87) | 0.63 | | 40 | 30.03 | 0.82 | 30.31 ( +0.94) | 0.37 | | 50 | 43.19 | 0.56 | 43.50 ( +0.72) | 0.52 | +--------+---------+---------+-------------------+---------+ +--------+-------------------+-----------------------------+ | | CPPC-no-EAS | CPPC-EAS | +--------+---------+---------+-------------------+---------+ | #Tasks | Mean | ci(+/-) | Mean | ci(+/-) | +--------+---------+---------+-------------------+---------+ | 10 | 7.86 | 0.37 | 5.64 (-28.23) | 0.05 | | 20 | 13.36 | 0.20 | 10.92 (-18.31) | 0.31 | | 30 | 19.28 | 0.34 | 18.30 ( -5.07) | 0.64 | | 40 | 28.33 | 0.59 | 27.13 ( -4.23) | 0.42 | | 50 | 40.78 | 0.58 | 40.77 ( -0.04) | 0.45 | +--------+---------+---------+-------------------+---------+ Missed activations were measured while comparing CPPC-no-EAS/CPPC-EAS energy values. They were of 0.00% for all tests and both configurations. Missed activations start to appear in a significant number starting from ~70 tasks. 2.1.1.2. Heterogeneous tasks Energy results (Joules): +--------+-------------------+-----------------------------+ | | no-EAS | EAS | +--------+---------+---------+-------------------+---------+ | #Tasks | Mean | ci(+/-) | Mean | ci(+/-) | +--------+---------+---------+-------------------+---------+ | 3 | 5.25 | 0.50 | 4.58 (-12.82%) | 0.07 | | 9 | 12.30 | 0.28 | 11.45 ( -6.97%) | 0.34 | | 15 | 20.06 | 1.32 | 20.60 ( 2.66%) | 1.00 | | 21 | 30.03 | 0.63 | 30.07 ( 0.12%) | 0.41 | +--------+---------+---------+-------------------+---------+ +--------+-------------------+-----------------------------+ | | CPPC-no-EAS | CPPC-EAS | +--------+---------+---------+-------------------+---------+ | #Tasks | Mean | ci(+/-) | Mean | ci(+/-) | +--------+---------+---------+-------------------+---------+ | 3 | 4.58 | 0.31 | 3.65 (-20.31%) | 0.05 | | 9 | 11.53 | 0.20 | 9.23 (-19.97%) | 0.22 | | 15 | 19.19 | 0.16 | 18.33 ( -4.49%) | 0.71 | | 21 | 29.07 | 0.29 | 29.06 ( -0.01%) | 0.08 | +--------+---------+---------+-------------------+---------+ Missed activations were measured while comparing CPPC-no-EAS/CPPC-EAS energy values. They were of 0.00% for all tests and both configurations. Missed activations start to appear in a significant number starting from ~36 tasks. 2.1.1.3. Analysis: The artificial EM often shows better energy gains than the EM, especially for small loads. Indeed, the artificial power values show a huge energy gain by placing tasks on little CPUs. The 6% margin is always reached, so tasks are easily placed on little CPUs. The margin is not always reached with real power values, leading to tasks staying on big CPUs. 2.1.2. Performance testing 10 iterations of HackBench with the "--pipe --thread" options and 1000 loops. Compared value is the testing time in seconds. A lower timing is better. +----------------+-------------------+---------------------------+ | | CPPC-no-EAS | CPPC-EAS | +--------+-------+---------+---------+-----------------+---------+ | Groups | Tasks | Mean | ci(+/-) | Mean | ci(+/-) | +--------+-------+---------+---------+-----------------+---------+ | 1 | 40 | 2.39 | 0.19 | 2.39 (-0.24%) | 0.07 | | 2 | 80 | 5.56 | 0.48 | 5.28 (-5.02%) | 0.42 | | 4 | 160 | 12.15 | 0.84 | 12.06 (-0.80%) | 0.48 | | 8 | 320 | 23.03 | 0.94 | 23.12 (+0.36%) | 0.70 | +--------+-------+---------+---------+-----------------+---------+ The performance is overall sligthly better, but stays in the margin or error. 2.2. Pixel4 testing Pixel4 has 7 CPUs: - 4 little [0-3], max_capa=261 - 3 medium [4-6], max_capa=861 - 1 big [7], max_capa=1024 Base kernel is android-10.0.0_r0.81. The performance states advertised in the DT were modified with performance states that would be generated by this patch-set. The artificial EM was set such as little CPUs > medium CPUs > big CPU, meaning little CPUs are the most energy efficient. Comparing the power/capacity ratio, little CPUs' performance states are all more energy efficient than the medium CPUs' performance states. This is wrong when comparing medium and big CPUs. 2.2.1. Energy testing The 2 sets of tests (heterogeneous/homogeneous) were tested while registering battery voltage and current (power is obtained by multiplying them). Voltage is averaged over a rolling period of ~11s and current over a period of ~6s. Usb-C cable is plugged in but alimentation is cut. Pixel4 is on airplane mode. The tests lasts 120s, the first 50s and last 10s are trimmed as the power is slowly raising to reach a plateau. Are compared: - android with EAS (but NO_FIND_BEST_TARGET is set): echo ENERGY_AWARE > /sys/kernel/debug/sched_features echo NO_FIND_BEST_TARGET > /sys/kernel/debug/sched_features - android without EAS: echo NO_ENERGY_AWARE > /sys/kernel/debug/sched_features - android with the artificial energy model Lower energy spending is better. 2.2.1.2. Homogeneous tasks Energy results (in uW): +--------+-------------------+-----------------------------+ | | Without EAS | With EAS | +--------+---------+---------+-------------------+---------+ | #Tasks | Mean | ci(+/-) | Mean | ci(+/-) | +--------+---------+---------+-------------------+---------+ | 10 | 6.21+05 | 3.12+02 | 5.09+05 (-18.01%) | 2.18+03 | | 20 | 9.12+05 | 9.71+02 | 7.91+05 (-13.26%) | 9.92+02 | | 30 | 1.25+06 | 2.02+03 | 1.09+06 (-12.12%) | 2.00+03 | | 40 | 2.05+06 | 5.15+03 | 1.38+06 (-32.36%) | 1.21+03 | | 50 | 3.03+06 | 6.94+03 | 1.89+06 (-37.44%) | 3.21+03 | +--------+---------+---------+-------------------+---------+ +--------+-------------------+-----------------------------+ | | Without EAS | With patch | +--------+---------+---------+-------------------+---------+ | #Tasks | Mean | ci(+/-) | Mean | ci(+/-) | +--------+---------+---------+-------------------+---------+ | 10 | 6.21+05 | 3.12+02 | 4.39+05 (-29.29%) | 5.63+02 | | 20 | 9.12+05 | 9.71+02 | 7.30+05 (-19.90%) | 1.98+03 | | 30 | 1.25+06 | 2.02+03 | 1.01+06 (-18.60%) | 1.72+03 | | 40 | 2.05+06 | 5.15+03 | 1.38+06 (-32.60%) | 3.93+03 | | 50 | 3.03+06 | 6.94+03 | 2.05+06 (-32.08%) | 1.25+04 | +--------+---------+---------+-------------------+---------+ 2.2.1.2. Heterogeneous tasks Energy results (in uW): +--------+-------------------+-----------------------------+ | | Without EAS | With EAS | +--------+---------+---------+-------------------+---------+ | #Tasks | Mean | ci(+/-) | Mean | ci(+/-) | +--------+---------+---------+-------------------+---------+ | 3 | 5.14+05 | 1.06+03 | 3.76+05 (-26.82%) | 4.58+02 | | 9 | 8.52+05 | 1.18+03 | 7.25+05 (-14.96%) | 1.39+03 | | 15 | 1.42+06 | 3.14+03 | 1.20+06 (-15.41%) | 1.06+04 | | 21 | 2.73+06 | 3.49+03 | 1.49+06 (-45.47%) | 3.43+03 | | 27 | 3.17+06 | 6.92+03 | 2.42+06 (-23.77%) | 8.43+03 | +--------+---------+---------+-------------------+---------+ +--------+-------------------+-----------------------------+ | | Without EAS | With patch | +--------+---------+---------+-------------------+---------+ | #Tasks | Mean | ci(+/-) | Mean | ci(+/-) | +--------+---------+---------+-------------------+---------+ | 3 | 5.14+05 | 1.06+03 | 3.82+05 (-25.70%) | 7.67+02 | | 9 | 8.52+05 | 1.18+03 | 7.05+05 (-17.30%) | 9.79+02 | | 15 | 1.42+06 | 3.14+03 | 1.05+06 (-26.00%) | 1.15+03 | | 21 | 2.73+06 | 3.49+03 | 1.53+06 (-43.68%) | 2.23+03 | | 27 | 3.17+06 | 6.92+03 | 2.86+06 ( -9.77%) | 4.26+03 | +--------+---------+---------+-------------------+---------+ 2.2.1.2. Analysis Similarly to Juno, the artificial performance states show a huge gain to place tasks on small CPUs, leading to better energy results. 2.2.2. Performance testing 10 iterations of PcMark. Compared value is the final score (PcmaWorkv3Score). A bigger score is better. +----------------+-------------------------+-------------------------+ | Without EAS | With EAS | With patch | +------+---------+---------------+---------+---------------+---------+ | Mean | ci(+/-) | Mean | ci(+/-) | Mean | ci(+/-) | +------+---------+---------------+---------+---------------+---------+ | 8026 | 86 | 8003 | 74 | 7840 (-2.00%) | 104 | +------+---------+---------------+---------+---------------+---------+ Performance is lower, but still in the margin of error. 3. Summary The artificial performance states show overall better energy results and a small performance decrease. They lead to a more aggressive task placement on the most energy efficient CPUs, and this explains the results. Pierre Gondois (2): cpufreq: CPPC: Add per_cpu efficiency_class cpufreq: CPPC: Register EM based on efficiency class information arch/arm64/kernel/smp.c | 1 + drivers/cpufreq/cppc_cpufreq.c | 186 +++++++++++++++++++++++++++++++++ 2 files changed, 187 insertions(+) -- 2.25.1 ^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH v3 1/2] cpufreq: CPPC: Add per_cpu efficiency_class 2022-04-25 12:38 [PATCH v3 0/2] Enable EAS for CPPC/ACPI based systems Pierre Gondois @ 2022-04-25 12:38 ` Pierre Gondois 2022-04-25 12:38 ` [PATCH v3 2/2] cpufreq: CPPC: Register EM based on efficiency class information Pierre Gondois 2022-04-26 3:08 ` [PATCH v3 0/2] Enable EAS for CPPC/ACPI based systems Viresh Kumar 2 siblings, 0 replies; 9+ messages in thread From: Pierre Gondois @ 2022-04-25 12:38 UTC (permalink / raw) To: linux-kernel Cc: Ionela.Voinescu, Lukasz.Luba, Morten.Rasmussen, Dietmar.Eggemann, maz, daniel.lezcano, Pierre Gondois, Catalin Marinas, Will Deacon, Rafael J. Wysocki, Viresh Kumar, Mark Rutland, Ard Biesheuvel, Fuad Tabba, Valentin Schneider, Lee Jones, Rob Herring, linux-arm-kernel, linux-pm From: Pierre Gondois <Pierre.Gondois@arm.com> In ACPI, describing power efficiency of CPUs can be done through the following arm specific field: ACPI 6.4, s5.2.12.14 'GIC CPU Interface (GICC) Structure', 'Processor Power Efficiency Class field': Describes the relative power efficiency of the associated pro- cessor. Lower efficiency class numbers are more efficient than higher ones (e.g. efficiency class 0 should be treated as more efficient than efficiency class 1). However, absolute values of this number have no meaning: 2 isn’t necessarily half as efficient as 1. The efficiency_class field is stored in the GicC structure of the ACPI MADT table and it's currently supported in Linux for arm64 only. Thus, this new functionality is introduced for arm64 only. To allow the cppc_cpufreq driver to know and preprocess the efficiency_class values of all the CPUs, add a per_cpu efficiency_class variable to store them. At least 2 different efficiency classes must be present, otherwise there is no use in creating an Energy Model. The efficiency_class values are squeezed in [0:#efficiency_class-1] while conserving the order. For instance, efficiency classes of: [111, 212, 250] will be mapped to: [0 (was 111), 1 (was 212), 2 (was 250)]. Each policy being independently registered in the driver, populating the per_cpu efficiency_class is done only once at the driver initialization. This prevents from having each policy re-searching the efficiency_class values of other CPUs. The EM will be registered in a following patch. The patch also exports acpi_cpu_get_madt_gicc() to fetch the GicC structure of the ACPI MADT table for each CPU. Acked-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Pierre Gondois <Pierre.Gondois@arm.com> --- arch/arm64/kernel/smp.c | 1 + drivers/cpufreq/cppc_cpufreq.c | 42 ++++++++++++++++++++++++++++++++++ 2 files changed, 43 insertions(+) diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c index 3b46041f2b97..62ed361a4376 100644 --- a/arch/arm64/kernel/smp.c +++ b/arch/arm64/kernel/smp.c @@ -512,6 +512,7 @@ struct acpi_madt_generic_interrupt *acpi_cpu_get_madt_gicc(int cpu) { return &cpu_madt_gicc[cpu]; } +EXPORT_SYMBOL_GPL(acpi_cpu_get_madt_gicc); /* * acpi_map_gic_cpu_interface - parse processor MADT entry diff --git a/drivers/cpufreq/cppc_cpufreq.c b/drivers/cpufreq/cppc_cpufreq.c index 82d370ae6a4a..3cd05651707d 100644 --- a/drivers/cpufreq/cppc_cpufreq.c +++ b/drivers/cpufreq/cppc_cpufreq.c @@ -420,12 +420,53 @@ static unsigned int cppc_cpufreq_get_transition_delay_us(unsigned int cpu) return cppc_get_transition_latency(cpu) / NSEC_PER_USEC; } +static DEFINE_PER_CPU(unsigned int, efficiency_class); + +static int populate_efficiency_class(void) +{ + struct acpi_madt_generic_interrupt *gicc; + DECLARE_BITMAP(used_classes, 256) = {}; + int class, cpu, index; + + for_each_possible_cpu(cpu) { + gicc = acpi_cpu_get_madt_gicc(cpu); + class = gicc->efficiency_class; + bitmap_set(used_classes, class, 1); + } + + if (bitmap_weight(used_classes, 256) <= 1) { + pr_debug("Efficiency classes are all equal (=%d). " + "No EM registered", class); + return -EINVAL; + } + + /* + * Squeeze efficiency class values on [0:#efficiency_class-1]. + * Values are per spec in [0:255]. + */ + index = 0; + for_each_set_bit(class, used_classes, 256) { + for_each_possible_cpu(cpu) { + gicc = acpi_cpu_get_madt_gicc(cpu); + if (gicc->efficiency_class == class) + per_cpu(efficiency_class, cpu) = index; + } + index++; + } + + return 0; +} + #else static unsigned int cppc_cpufreq_get_transition_delay_us(unsigned int cpu) { return cppc_get_transition_latency(cpu) / NSEC_PER_USEC; } +static int populate_efficiency_class(void) +{ + return 0; +} #endif @@ -742,6 +783,7 @@ static int __init cppc_cpufreq_init(void) cppc_check_hisi_workaround(); cppc_freq_invariance_init(); + populate_efficiency_class(); ret = cpufreq_register_driver(&cppc_cpufreq_driver); if (ret) -- 2.25.1 ^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH v3 2/2] cpufreq: CPPC: Register EM based on efficiency class information 2022-04-25 12:38 [PATCH v3 0/2] Enable EAS for CPPC/ACPI based systems Pierre Gondois 2022-04-25 12:38 ` [PATCH v3 1/2] cpufreq: CPPC: Add per_cpu efficiency_class Pierre Gondois @ 2022-04-25 12:38 ` Pierre Gondois 2022-05-30 6:42 ` Shaokun Zhang 2022-04-26 3:08 ` [PATCH v3 0/2] Enable EAS for CPPC/ACPI based systems Viresh Kumar 2 siblings, 1 reply; 9+ messages in thread From: Pierre Gondois @ 2022-04-25 12:38 UTC (permalink / raw) To: linux-kernel Cc: Ionela.Voinescu, Lukasz.Luba, Morten.Rasmussen, Dietmar.Eggemann, maz, daniel.lezcano, Pierre Gondois, Catalin Marinas, Will Deacon, Rafael J. Wysocki, Viresh Kumar, Mark Rutland, Ard Biesheuvel, Fuad Tabba, Phil Auld, Rob Herring, Valentin Schneider, linux-arm-kernel, linux-pm From: Pierre Gondois <Pierre.Gondois@arm.com> Performance states and energy consumption values are not advertised in ACPI. In the GicC structure of the MADT table, the "Processor Power Efficiency Class field" (called efficiency class from now) allows to describe the relative energy efficiency of CPUs. To leverage the EM and EAS, the CPPC driver creates a set of artificial performance states and registers them in the Energy Model (EM), such as: - Every 20 capacity unit, a performance state is created. - The energy cost of each performance state gradually increases. No power value is generated as only the cost is used in the EM. During task placement, a task can raise the frequency of its whole pd. This can make EAS place a task on a pd with CPUs that are individually less energy efficient. As cost values are artificial, and to place tasks on CPUs with the lower efficiency class, a gap in cost values is generated for adjacent efficiency classes. E.g.: - efficiency class = 0, capacity is in [0-1024], so cost values are in [0: 51] (one performance state every 20 capacity unit) - efficiency class = 1, capacity is in [0-1024], cost values are in [1*gap+0: 1*gap+51]. The value of the cost gap is chosen to absorb a the energy of 4 CPUs at their maximum capacity. This means that between: 1- a pd of 4 CPUs, each of them being used at almost their full capacity. Their efficiency class is N. 2- a CPU using almost none of its capacity. Its efficiency class is N+1 EAS will choose the first option. This patch also populates the (struct cpufreq_driver).register_em callback if the valid efficiency_class ACPI values are provided. Signed-off-by: Pierre Gondois <Pierre.Gondois@arm.com> --- drivers/cpufreq/cppc_cpufreq.c | 144 +++++++++++++++++++++++++++++++++ 1 file changed, 144 insertions(+) diff --git a/drivers/cpufreq/cppc_cpufreq.c b/drivers/cpufreq/cppc_cpufreq.c index 3cd05651707d..3eaa23d1aaf5 100644 --- a/drivers/cpufreq/cppc_cpufreq.c +++ b/drivers/cpufreq/cppc_cpufreq.c @@ -421,6 +421,134 @@ static unsigned int cppc_cpufreq_get_transition_delay_us(unsigned int cpu) } static DEFINE_PER_CPU(unsigned int, efficiency_class); +static void cppc_cpufreq_register_em(struct cpufreq_policy *policy); + +/* Create an artificial performance state every CPPC_EM_CAP_STEP capacity unit. */ +#define CPPC_EM_CAP_STEP (20) +/* Increase the cost value by CPPC_EM_COST_STEP every performance state. */ +#define CPPC_EM_COST_STEP (1) +/* Add a cost gap correspnding to the energy of 4 CPUs. */ +#define CPPC_EM_COST_GAP (4 * SCHED_CAPACITY_SCALE * CPPC_EM_COST_STEP \ + / CPPC_EM_CAP_STEP) + +static unsigned int get_perf_level_count(struct cpufreq_policy *policy) +{ + struct cppc_perf_caps *perf_caps; + unsigned int min_cap, max_cap; + struct cppc_cpudata *cpu_data; + int cpu = policy->cpu; + + cpu_data = policy->driver_data; + perf_caps = &cpu_data->perf_caps; + max_cap = arch_scale_cpu_capacity(cpu); + min_cap = div_u64(max_cap * perf_caps->lowest_perf, perf_caps->highest_perf); + if ((min_cap == 0) || (max_cap < min_cap)) + return 0; + return 1 + max_cap / CPPC_EM_CAP_STEP - min_cap / CPPC_EM_CAP_STEP; +} + +/* + * The cost is defined as: + * cost = power * max_frequency / frequency + */ +static inline unsigned long compute_cost(int cpu, int step) +{ + return CPPC_EM_COST_GAP * per_cpu(efficiency_class, cpu) + + step * CPPC_EM_COST_STEP; +} + +static int cppc_get_cpu_power(struct device *cpu_dev, + unsigned long *power, unsigned long *KHz) +{ + unsigned long perf_step, perf_prev, perf, perf_check; + unsigned int min_step, max_step, step, step_check; + unsigned long prev_freq = *KHz; + unsigned int min_cap, max_cap; + struct cpufreq_policy *policy; + + struct cppc_perf_caps *perf_caps; + struct cppc_cpudata *cpu_data; + + policy = cpufreq_cpu_get_raw(cpu_dev->id); + cpu_data = policy->driver_data; + perf_caps = &cpu_data->perf_caps; + max_cap = arch_scale_cpu_capacity(cpu_dev->id); + min_cap = div_u64(max_cap * perf_caps->lowest_perf, + perf_caps->highest_perf); + + perf_step = CPPC_EM_CAP_STEP * perf_caps->highest_perf / max_cap; + min_step = min_cap / CPPC_EM_CAP_STEP; + max_step = max_cap / CPPC_EM_CAP_STEP; + + perf_prev = cppc_cpufreq_khz_to_perf(cpu_data, *KHz); + step = perf_prev / perf_step; + + if (step > max_step) + return -EINVAL; + + if (min_step == max_step) { + step = max_step; + perf = perf_caps->highest_perf; + } else if (step < min_step) { + step = min_step; + perf = perf_caps->lowest_perf; + } else { + step++; + if (step == max_step) + perf = perf_caps->highest_perf; + else + perf = step * perf_step; + } + + *KHz = cppc_cpufreq_perf_to_khz(cpu_data, perf); + perf_check = cppc_cpufreq_khz_to_perf(cpu_data, *KHz); + step_check = perf_check / perf_step; + + /* + * To avoid bad integer approximation, check that new frequency value + * increased and that the new frequency will be converted to the + * desired step value. + */ + while ((*KHz == prev_freq) || (step_check != step)) { + perf++; + *KHz = cppc_cpufreq_perf_to_khz(cpu_data, perf); + perf_check = cppc_cpufreq_khz_to_perf(cpu_data, *KHz); + step_check = perf_check / perf_step; + } + + /* + * With an artificial EM, only the cost value is used. Still the power + * is populated such as 0 < power < EM_MAX_POWER. This allows to add + * more sense to the artificial performance states. + */ + *power = compute_cost(cpu_dev->id, step); + + return 0; +} + +static int cppc_get_cpu_cost(struct device *cpu_dev, unsigned long KHz, + unsigned long *cost) +{ + unsigned long perf_step, perf_prev; + struct cppc_perf_caps *perf_caps; + struct cpufreq_policy *policy; + struct cppc_cpudata *cpu_data; + unsigned int max_cap; + int step; + + policy = cpufreq_cpu_get_raw(cpu_dev->id); + cpu_data = policy->driver_data; + perf_caps = &cpu_data->perf_caps; + max_cap = arch_scale_cpu_capacity(cpu_dev->id); + + perf_prev = cppc_cpufreq_khz_to_perf(cpu_data, KHz); + perf_step = CPPC_EM_CAP_STEP * perf_caps->highest_perf / max_cap; + step = perf_prev / perf_step; + + *cost = compute_cost(cpu_dev->id, step); + + return 0; +} static int populate_efficiency_class(void) { @@ -453,10 +581,23 @@ static int populate_efficiency_class(void) } index++; } + cppc_cpufreq_driver.register_em = cppc_cpufreq_register_em; return 0; } +static void cppc_cpufreq_register_em(struct cpufreq_policy *policy) +{ + struct cppc_cpudata *cpu_data; + struct em_data_callback em_cb = + EM_ADV_DATA_CB(cppc_get_cpu_power, cppc_get_cpu_cost); + + cpu_data = policy->driver_data; + em_dev_register_perf_domain(get_cpu_device(policy->cpu), + get_perf_level_count(policy), &em_cb, + cpu_data->shared_cpu_map, 0); +} + #else static unsigned int cppc_cpufreq_get_transition_delay_us(unsigned int cpu) @@ -467,6 +608,9 @@ static int populate_efficiency_class(void) { return 0; } +static void cppc_cpufreq_register_em(struct cpufreq_policy *policy) +{ +} #endif -- 2.25.1 ^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [PATCH v3 2/2] cpufreq: CPPC: Register EM based on efficiency class information 2022-04-25 12:38 ` [PATCH v3 2/2] cpufreq: CPPC: Register EM based on efficiency class information Pierre Gondois @ 2022-05-30 6:42 ` Shaokun Zhang 0 siblings, 0 replies; 9+ messages in thread From: Shaokun Zhang @ 2022-05-30 6:42 UTC (permalink / raw) To: Pierre Gondois, linux-kernel Cc: Ionela.Voinescu, Lukasz.Luba, Morten.Rasmussen, Dietmar.Eggemann, maz, daniel.lezcano, Catalin Marinas, Will Deacon, Rafael J. Wysocki, Viresh Kumar, Mark Rutland, Ard Biesheuvel, Fuad Tabba, Phil Auld, Rob Herring, Valentin Schneider, linux-arm-kernel, linux-pm Hi, There is a warning on arm64 platform when CONFIG_ENERGY_MODEL is not set: drivers/cpufreq/cppc_cpufreq.c:550:12: error: ‘cppc_get_cpu_cost’ defined but not used [-Werror=unused-function] 550 | static int cppc_get_cpu_cost(struct device *cpu_dev, unsigned long KHz, | ^~~~~~~~~~~~~~~~~ drivers/cpufreq/cppc_cpufreq.c:481:12: error: ‘cppc_get_cpu_power’ defined but not used [-Werror=unused-function] 481 | static int cppc_get_cpu_power(struct device *cpu_dev, | ^~~~~~~~~~~~~~~~~~ Thanks, Shaokun On 2022/4/25 20:38, Pierre Gondois wrote: > From: Pierre Gondois <Pierre.Gondois@arm.com> > > Performance states and energy consumption values are not advertised > in ACPI. In the GicC structure of the MADT table, the "Processor > Power Efficiency Class field" (called efficiency class from now) > allows to describe the relative energy efficiency of CPUs. > > To leverage the EM and EAS, the CPPC driver creates a set of > artificial performance states and registers them in the Energy Model > (EM), such as: > - Every 20 capacity unit, a performance state is created. > - The energy cost of each performance state gradually increases. > No power value is generated as only the cost is used in the EM. > > During task placement, a task can raise the frequency of its whole > pd. This can make EAS place a task on a pd with CPUs that are > individually less energy efficient. > As cost values are artificial, and to place tasks on CPUs with the > lower efficiency class, a gap in cost values is generated for adjacent > efficiency classes. > E.g.: > - efficiency class = 0, capacity is in [0-1024], so cost values > are in [0: 51] (one performance state every 20 capacity unit) > - efficiency class = 1, capacity is in [0-1024], cost values > are in [1*gap+0: 1*gap+51]. > > The value of the cost gap is chosen to absorb a the energy of 4 CPUs > at their maximum capacity. This means that between: > 1- a pd of 4 CPUs, each of them being used at almost their full > capacity. Their efficiency class is N. > 2- a CPU using almost none of its capacity. Its efficiency class is > N+1 > EAS will choose the first option. > > This patch also populates the (struct cpufreq_driver).register_em > callback if the valid efficiency_class ACPI values are provided. > > Signed-off-by: Pierre Gondois <Pierre.Gondois@arm.com> > --- > drivers/cpufreq/cppc_cpufreq.c | 144 +++++++++++++++++++++++++++++++++ > 1 file changed, 144 insertions(+) > > diff --git a/drivers/cpufreq/cppc_cpufreq.c b/drivers/cpufreq/cppc_cpufreq.c > index 3cd05651707d..3eaa23d1aaf5 100644 > --- a/drivers/cpufreq/cppc_cpufreq.c > +++ b/drivers/cpufreq/cppc_cpufreq.c > @@ -421,6 +421,134 @@ static unsigned int cppc_cpufreq_get_transition_delay_us(unsigned int cpu) > } > > static DEFINE_PER_CPU(unsigned int, efficiency_class); > +static void cppc_cpufreq_register_em(struct cpufreq_policy *policy); > + > +/* Create an artificial performance state every CPPC_EM_CAP_STEP capacity unit. */ > +#define CPPC_EM_CAP_STEP (20) > +/* Increase the cost value by CPPC_EM_COST_STEP every performance state. */ > +#define CPPC_EM_COST_STEP (1) > +/* Add a cost gap correspnding to the energy of 4 CPUs. */ > +#define CPPC_EM_COST_GAP (4 * SCHED_CAPACITY_SCALE * CPPC_EM_COST_STEP \ > + / CPPC_EM_CAP_STEP) > + > +static unsigned int get_perf_level_count(struct cpufreq_policy *policy) > +{ > + struct cppc_perf_caps *perf_caps; > + unsigned int min_cap, max_cap; > + struct cppc_cpudata *cpu_data; > + int cpu = policy->cpu; > + > + cpu_data = policy->driver_data; > + perf_caps = &cpu_data->perf_caps; > + max_cap = arch_scale_cpu_capacity(cpu); > + min_cap = div_u64(max_cap * perf_caps->lowest_perf, perf_caps->highest_perf); > + if ((min_cap == 0) || (max_cap < min_cap)) > + return 0; > + return 1 + max_cap / CPPC_EM_CAP_STEP - min_cap / CPPC_EM_CAP_STEP; > +} > + > +/* > + * The cost is defined as: > + * cost = power * max_frequency / frequency > + */ > +static inline unsigned long compute_cost(int cpu, int step) > +{ > + return CPPC_EM_COST_GAP * per_cpu(efficiency_class, cpu) + > + step * CPPC_EM_COST_STEP; > +} > + > +static int cppc_get_cpu_power(struct device *cpu_dev, > + unsigned long *power, unsigned long *KHz) > +{ > + unsigned long perf_step, perf_prev, perf, perf_check; > + unsigned int min_step, max_step, step, step_check; > + unsigned long prev_freq = *KHz; > + unsigned int min_cap, max_cap; > + struct cpufreq_policy *policy; > + > + struct cppc_perf_caps *perf_caps; > + struct cppc_cpudata *cpu_data; > + > + policy = cpufreq_cpu_get_raw(cpu_dev->id); > + cpu_data = policy->driver_data; > + perf_caps = &cpu_data->perf_caps; > + max_cap = arch_scale_cpu_capacity(cpu_dev->id); > + min_cap = div_u64(max_cap * perf_caps->lowest_perf, > + perf_caps->highest_perf); > + > + perf_step = CPPC_EM_CAP_STEP * perf_caps->highest_perf / max_cap; > + min_step = min_cap / CPPC_EM_CAP_STEP; > + max_step = max_cap / CPPC_EM_CAP_STEP; > + > + perf_prev = cppc_cpufreq_khz_to_perf(cpu_data, *KHz); > + step = perf_prev / perf_step; > + > + if (step > max_step) > + return -EINVAL; > + > + if (min_step == max_step) { > + step = max_step; > + perf = perf_caps->highest_perf; > + } else if (step < min_step) { > + step = min_step; > + perf = perf_caps->lowest_perf; > + } else { > + step++; > + if (step == max_step) > + perf = perf_caps->highest_perf; > + else > + perf = step * perf_step; > + } > + > + *KHz = cppc_cpufreq_perf_to_khz(cpu_data, perf); > + perf_check = cppc_cpufreq_khz_to_perf(cpu_data, *KHz); > + step_check = perf_check / perf_step; > + > + /* > + * To avoid bad integer approximation, check that new frequency value > + * increased and that the new frequency will be converted to the > + * desired step value. > + */ > + while ((*KHz == prev_freq) || (step_check != step)) { > + perf++; > + *KHz = cppc_cpufreq_perf_to_khz(cpu_data, perf); > + perf_check = cppc_cpufreq_khz_to_perf(cpu_data, *KHz); > + step_check = perf_check / perf_step; > + } > + > + /* > + * With an artificial EM, only the cost value is used. Still the power > + * is populated such as 0 < power < EM_MAX_POWER. This allows to add > + * more sense to the artificial performance states. > + */ > + *power = compute_cost(cpu_dev->id, step); > + > + return 0; > +} > + > +static int cppc_get_cpu_cost(struct device *cpu_dev, unsigned long KHz, > + unsigned long *cost) > +{ > + unsigned long perf_step, perf_prev; > + struct cppc_perf_caps *perf_caps; > + struct cpufreq_policy *policy; > + struct cppc_cpudata *cpu_data; > + unsigned int max_cap; > + int step; > + > + policy = cpufreq_cpu_get_raw(cpu_dev->id); > + cpu_data = policy->driver_data; > + perf_caps = &cpu_data->perf_caps; > + max_cap = arch_scale_cpu_capacity(cpu_dev->id); > + > + perf_prev = cppc_cpufreq_khz_to_perf(cpu_data, KHz); > + perf_step = CPPC_EM_CAP_STEP * perf_caps->highest_perf / max_cap; > + step = perf_prev / perf_step; > + > + *cost = compute_cost(cpu_dev->id, step); > + > + return 0; > +} > > static int populate_efficiency_class(void) > { > @@ -453,10 +581,23 @@ static int populate_efficiency_class(void) > } > index++; > } > + cppc_cpufreq_driver.register_em = cppc_cpufreq_register_em; > > return 0; > } > > +static void cppc_cpufreq_register_em(struct cpufreq_policy *policy) > +{ > + struct cppc_cpudata *cpu_data; > + struct em_data_callback em_cb = > + EM_ADV_DATA_CB(cppc_get_cpu_power, cppc_get_cpu_cost); > + > + cpu_data = policy->driver_data; > + em_dev_register_perf_domain(get_cpu_device(policy->cpu), > + get_perf_level_count(policy), &em_cb, > + cpu_data->shared_cpu_map, 0); > +} > + > #else > > static unsigned int cppc_cpufreq_get_transition_delay_us(unsigned int cpu) > @@ -467,6 +608,9 @@ static int populate_efficiency_class(void) > { > return 0; > } > +static void cppc_cpufreq_register_em(struct cpufreq_policy *policy) > +{ > +} > #endif > > > ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH v3 0/2] Enable EAS for CPPC/ACPI based systems 2022-04-25 12:38 [PATCH v3 0/2] Enable EAS for CPPC/ACPI based systems Pierre Gondois 2022-04-25 12:38 ` [PATCH v3 1/2] cpufreq: CPPC: Add per_cpu efficiency_class Pierre Gondois 2022-04-25 12:38 ` [PATCH v3 2/2] cpufreq: CPPC: Register EM based on efficiency class information Pierre Gondois @ 2022-04-26 3:08 ` Viresh Kumar 2022-04-26 6:37 ` Viresh Kumar 2 siblings, 1 reply; 9+ messages in thread From: Viresh Kumar @ 2022-04-26 3:08 UTC (permalink / raw) To: Pierre Gondois Cc: linux-kernel, Ionela.Voinescu, Lukasz.Luba, Morten.Rasmussen, Dietmar.Eggemann, maz, daniel.lezcano, Catalin Marinas, Will Deacon, Rafael J. Wysocki, Mark Rutland, Ard Biesheuvel, Fuad Tabba, Valentin Schneider, Rob Herring, Lee Jones, linux-arm-kernel, linux-pm On 25-04-22, 14:38, Pierre Gondois wrote: > v3: > - Remove efficiency_class_populated variable. [Viresh] > - Remove patch "cpufreq: CPPC: Add cppc_cpufreq_search_cpu_data" > and access cpu_data through policy->driver_data. [Viresh] > - arm64 code only acked by Catalin [Catalin] Applied. Thanks. -- viresh ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH v3 0/2] Enable EAS for CPPC/ACPI based systems 2022-04-26 3:08 ` [PATCH v3 0/2] Enable EAS for CPPC/ACPI based systems Viresh Kumar @ 2022-04-26 6:37 ` Viresh Kumar 2022-04-26 7:10 ` Pierre Gondois 0 siblings, 1 reply; 9+ messages in thread From: Viresh Kumar @ 2022-04-26 6:37 UTC (permalink / raw) To: Pierre Gondois Cc: linux-kernel, Ionela.Voinescu, Lukasz.Luba, Morten.Rasmussen, Dietmar.Eggemann, maz, daniel.lezcano, Catalin Marinas, Will Deacon, Rafael J. Wysocki, Mark Rutland, Ard Biesheuvel, Fuad Tabba, Valentin Schneider, Rob Herring, Lee Jones, linux-arm-kernel, linux-pm On 26-04-22, 08:38, Viresh Kumar wrote: > On 25-04-22, 14:38, Pierre Gondois wrote: > > v3: > > - Remove efficiency_class_populated variable. [Viresh] > > - Remove patch "cpufreq: CPPC: Add cppc_cpufreq_search_cpu_data" > > and access cpu_data through policy->driver_data. [Viresh] > > - arm64 code only acked by Catalin [Catalin] > > Applied. Thanks. Removed, build failures: https://gitlab.com/vireshk/pmko/-/jobs/2375905218 Log: https://builds.tuxbuild.com/28Jos1GBXfT5Cl3HoBKKwmM8Ddk/build.log /builds/linux/drivers/cpufreq/cppc_cpufreq.c: In function 'cppc_cpufreq_register_em': 368/builds/linux/drivers/cpufreq/cppc_cpufreq.c:593:3: error: implicit declaration of function 'EM_ADV_DATA_CB'; did you mean 'EM_DATA_CB'? [-Werror=implicit-function-declaration] 369 593 | EM_ADV_DATA_CB(cppc_get_cpu_power, cppc_get_cpu_cost); 370 | ^~~~~~~~~~~~~~ 371 | EM_DATA_CB 372/builds/linux/drivers/cpufreq/cppc_cpufreq.c:593:3: error: invalid initializer 373cc1: some warnings being treated as errors 374 -- viresh ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH v3 0/2] Enable EAS for CPPC/ACPI based systems 2022-04-26 6:37 ` Viresh Kumar @ 2022-04-26 7:10 ` Pierre Gondois 2022-04-26 7:12 ` Viresh Kumar 0 siblings, 1 reply; 9+ messages in thread From: Pierre Gondois @ 2022-04-26 7:10 UTC (permalink / raw) To: Viresh Kumar Cc: linux-kernel, Ionela.Voinescu, Lukasz.Luba, Morten.Rasmussen, Dietmar.Eggemann, maz, daniel.lezcano, Catalin Marinas, Will Deacon, Rafael J. Wysocki, Mark Rutland, Ard Biesheuvel, Fuad Tabba, Valentin Schneider, Rob Herring, Lee Jones, linux-arm-kernel, linux-pm Hello Viresh, The 2 patches are relying on Lukasz' patch-set at: https://lkml.org/lkml/2022/3/21/282 The serie was accepted by Rafael (cf. https://lkml.org/lkml/2022/4/13/701) and is currently in linux-next. More specifically, the missing patch causing the build failure is: 'PM: EM: Add .get_cost() callback' From what I see, the branch cpufreq/arm/linux-next (from your repo) that was used in the CI is based on v5.18-rc1 and doesn't have Lukasz' patches. Should we wait for the patches to be in a rc version, or is there a process for this kind of case ? Regards, Pierre On 4/26/22 08:37, Viresh Kumar wrote: > On 26-04-22, 08:38, Viresh Kumar wrote: >> On 25-04-22, 14:38, Pierre Gondois wrote: >>> v3: >>> - Remove efficiency_class_populated variable. [Viresh] >>> - Remove patch "cpufreq: CPPC: Add cppc_cpufreq_search_cpu_data" >>> and access cpu_data through policy->driver_data. [Viresh] >>> - arm64 code only acked by Catalin [Catalin] >> >> Applied. Thanks. > > Removed, build failures: > > https://gitlab.com/vireshk/pmko/-/jobs/2375905218 > > Log: https://builds.tuxbuild.com/28Jos1GBXfT5Cl3HoBKKwmM8Ddk/build.log > > /builds/linux/drivers/cpufreq/cppc_cpufreq.c: In function 'cppc_cpufreq_register_em': > 368/builds/linux/drivers/cpufreq/cppc_cpufreq.c:593:3: error: implicit declaration of function 'EM_ADV_DATA_CB'; did you mean 'EM_DATA_CB'? [-Werror=implicit-function-declaration] > 369 593 | EM_ADV_DATA_CB(cppc_get_cpu_power, cppc_get_cpu_cost); > 370 | ^~~~~~~~~~~~~~ > 371 | EM_DATA_CB > 372/builds/linux/drivers/cpufreq/cppc_cpufreq.c:593:3: error: invalid initializer > 373cc1: some warnings being treated as errors > 374 > ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH v3 0/2] Enable EAS for CPPC/ACPI based systems 2022-04-26 7:10 ` Pierre Gondois @ 2022-04-26 7:12 ` Viresh Kumar 2022-05-06 19:02 ` Rafael J. Wysocki 0 siblings, 1 reply; 9+ messages in thread From: Viresh Kumar @ 2022-04-26 7:12 UTC (permalink / raw) To: Pierre Gondois Cc: linux-kernel, Ionela.Voinescu, Lukasz.Luba, Morten.Rasmussen, Dietmar.Eggemann, maz, daniel.lezcano, Catalin Marinas, Will Deacon, Rafael J. Wysocki, Mark Rutland, Ard Biesheuvel, Fuad Tabba, Valentin Schneider, Rob Herring, Lee Jones, linux-arm-kernel, linux-pm On 26-04-22, 09:10, Pierre Gondois wrote: > Hello Viresh, > The 2 patches are relying on Lukasz' patch-set at: > https://lkml.org/lkml/2022/3/21/282 > The serie was accepted by Rafael (cf. https://lkml.org/lkml/2022/4/13/701) and > is currently in linux-next. More specifically, the missing patch causing the > build failure is: 'PM: EM: Add .get_cost() callback' > > From what I see, the branch cpufreq/arm/linux-next (from your repo) that was > used in the CI is based on v5.18-rc1 and doesn't have Lukasz' patches. Should > we wait for the patches to be in a rc version, or is there a process for this > kind of case ? Ok. Rafael: Please pick these patches directly. Acked-by: Viresh Kumar <viresh.kumar@linaro.org> -- viresh ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH v3 0/2] Enable EAS for CPPC/ACPI based systems 2022-04-26 7:12 ` Viresh Kumar @ 2022-05-06 19:02 ` Rafael J. Wysocki 0 siblings, 0 replies; 9+ messages in thread From: Rafael J. Wysocki @ 2022-05-06 19:02 UTC (permalink / raw) To: Viresh Kumar Cc: Pierre Gondois, Linux Kernel Mailing List, Ionela Voinescu, Lukasz Luba, Morten Rasmussen, Dietmar Eggemann, Marc Zyngier, Daniel Lezcano, Catalin Marinas, Will Deacon, Rafael J. Wysocki, Mark Rutland, Ard Biesheuvel, Fuad Tabba, Valentin Schneider, Rob Herring, Lee Jones, Linux ARM, Linux PM On Tue, Apr 26, 2022 at 9:12 AM Viresh Kumar <viresh.kumar@linaro.org> wrote: > > On 26-04-22, 09:10, Pierre Gondois wrote: > > Hello Viresh, > > The 2 patches are relying on Lukasz' patch-set at: > > https://lkml.org/lkml/2022/3/21/282 > > The serie was accepted by Rafael (cf. https://lkml.org/lkml/2022/4/13/701) and > > is currently in linux-next. More specifically, the missing patch causing the > > build failure is: 'PM: EM: Add .get_cost() callback' > > > > From what I see, the branch cpufreq/arm/linux-next (from your repo) that was > > used in the CI is based on v5.18-rc1 and doesn't have Lukasz' patches. Should > > we wait for the patches to be in a rc version, or is there a process for this > > kind of case ? > > Ok. > > Rafael: Please pick these patches directly. > > Acked-by: Viresh Kumar <viresh.kumar@linaro.org> Done, thanks! ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2022-05-30 6:42 UTC | newest] Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2022-04-25 12:38 [PATCH v3 0/2] Enable EAS for CPPC/ACPI based systems Pierre Gondois 2022-04-25 12:38 ` [PATCH v3 1/2] cpufreq: CPPC: Add per_cpu efficiency_class Pierre Gondois 2022-04-25 12:38 ` [PATCH v3 2/2] cpufreq: CPPC: Register EM based on efficiency class information Pierre Gondois 2022-05-30 6:42 ` Shaokun Zhang 2022-04-26 3:08 ` [PATCH v3 0/2] Enable EAS for CPPC/ACPI based systems Viresh Kumar 2022-04-26 6:37 ` Viresh Kumar 2022-04-26 7:10 ` Pierre Gondois 2022-04-26 7:12 ` Viresh Kumar 2022-05-06 19:02 ` Rafael J. Wysocki
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).