* Re: [RFC v2 1/2] cpuidle: Extract IPI based and timer based wakeup latency from idle states
@ 2021-04-02 9:03 kernel test robot
2021-04-02 9:03 ` [PATCH] cpuidle: fix debugfs_simple_attr.cocci warnings kernel test robot
0 siblings, 1 reply; 3+ messages in thread
From: kernel test robot @ 2021-04-02 9:03 UTC (permalink / raw)
To: kbuild
[-- Attachment #1: Type: text/plain, Size: 1653 bytes --]
CC: kbuild-all(a)lists.01.org
In-Reply-To: <20210401114504.13466-2-psampat@linux.ibm.com>
References: <20210401114504.13466-2-psampat@linux.ibm.com>
TO: Pratik Rajesh Sampat <psampat@linux.ibm.com>
Hi Pratik,
[FYI, it's a private test report for your RFC patch.]
[auto build test WARNING on kselftest/next]
[also build test WARNING on pm/linux-next linus/master daniel.lezcano/clockevents/next v5.12-rc5 next-20210401]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/0day-ci/linux/commits/Pratik-Rajesh-Sampat/cpuidle-Extract-IPI-based-and-timer-based-wakeup-latency-from-idle-states/20210402-015842
base: https://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest.git next
:::::: branch date: 15 hours ago
:::::: commit date: 15 hours ago
config: i386-allmodconfig (attached as .config)
compiler: gcc-9 (Debian 9.3.0-22) 9.3.0
If you fix the issue, kindly add following tag as appropriate
Reported-by: kernel test robot <lkp@intel.com>
Reported-by: Julia Lawall <julia.lawall@lip6.fr>
cocci warnings: (new ones prefixed by >>)
>> drivers/cpuidle/test-cpuidle_latency.c:95:0-23: WARNING: ipi_ops should be defined with DEFINE_DEBUGFS_ATTRIBUTE
>> drivers/cpuidle/test-cpuidle_latency.c:108:0-23: WARNING: timeout_ops should be defined with DEFINE_DEBUGFS_ATTRIBUTE
Please review and possibly fold the followup patch.
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
[-- Attachment #2: config.gz --]
[-- Type: application/gzip, Size: 65514 bytes --]
^ permalink raw reply [flat|nested] 3+ messages in thread
* [PATCH] cpuidle: fix debugfs_simple_attr.cocci warnings 2021-04-02 9:03 [RFC v2 1/2] cpuidle: Extract IPI based and timer based wakeup latency from idle states kernel test robot @ 2021-04-02 9:03 ` kernel test robot 0 siblings, 0 replies; 3+ messages in thread From: kernel test robot @ 2021-04-02 9:03 UTC (permalink / raw) To: kbuild [-- Attachment #1: Type: text/plain, Size: 3254 bytes --] CC: kbuild-all(a)lists.01.org In-Reply-To: <20210401114504.13466-2-psampat@linux.ibm.com> References: <20210401114504.13466-2-psampat@linux.ibm.com> TO: Pratik Rajesh Sampat <psampat@linux.ibm.com> From: kernel test robot <lkp@intel.com> drivers/cpuidle/test-cpuidle_latency.c:95:0-23: WARNING: ipi_ops should be defined with DEFINE_DEBUGFS_ATTRIBUTE drivers/cpuidle/test-cpuidle_latency.c:108:0-23: WARNING: timeout_ops should be defined with DEFINE_DEBUGFS_ATTRIBUTE Use DEFINE_DEBUGFS_ATTRIBUTE rather than DEFINE_SIMPLE_ATTRIBUTE for debugfs files. Semantic patch information: Rationale: DEFINE_SIMPLE_ATTRIBUTE + debugfs_create_file() imposes some significant overhead as compared to DEFINE_DEBUGFS_ATTRIBUTE + debugfs_create_file_unsafe(). Generated by: scripts/coccinelle/api/debugfs/debugfs_simple_attr.cocci CC: Pratik Rajesh Sampat <psampat@linux.ibm.com> Reported-by: kernel test robot <lkp@intel.com> Signed-off-by: kernel test robot <lkp@intel.com> --- url: https://github.com/0day-ci/linux/commits/Pratik-Rajesh-Sampat/cpuidle-Extract-IPI-based-and-timer-based-wakeup-latency-from-idle-states/20210402-015842 base: https://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest.git next :::::: branch date: 15 hours ago :::::: commit date: 15 hours ago Please take the patch only if it's a positive warning. Thanks! test-cpuidle_latency.c | 19 +++++++------------ 1 file changed, 7 insertions(+), 12 deletions(-) --- a/drivers/cpuidle/test-cpuidle_latency.c +++ b/drivers/cpuidle/test-cpuidle_latency.c @@ -92,7 +92,7 @@ static int cpu_write_op(void *data, u64 run_smp_call_function_test(value); return 0; } -DEFINE_SIMPLE_ATTRIBUTE(ipi_ops, cpu_read_op, cpu_write_op, "%llu\n"); +DEFINE_DEBUGFS_ATTRIBUTE(ipi_ops, cpu_read_op, cpu_write_op, "%llu\n"); static int timeout_read_op(void *data, u64 *timeout) { @@ -105,7 +105,8 @@ static int timeout_write_op(void *data, run_timer_test(value); return 0; } -DEFINE_SIMPLE_ATTRIBUTE(timeout_ops, timeout_read_op, timeout_write_op, "%llu\n"); +DEFINE_DEBUGFS_ATTRIBUTE(timeout_ops, timeout_read_op, timeout_write_op, + "%llu\n"); static int __init latency_init(void) { @@ -116,11 +117,8 @@ static int __init latency_init(void) pr_alert("latency_test: failed to create /sys/kernel/debug/latency_test\n"); return -1; } - temp = debugfs_create_file("ipi_cpu_dest", - 0666, - dir, - NULL, - &ipi_ops); + temp = debugfs_create_file_unsafe("ipi_cpu_dest", 0666, dir, NULL, + &ipi_ops); if (!temp) { pr_alert("latency_test: failed to create /sys/kernel/debug/ipi_cpu_dest\n"); return -1; @@ -128,11 +126,8 @@ static int __init latency_init(void) debugfs_create_u64("ipi_latency_ns", 0444, dir, &ipi_wakeup.latency_ns); debugfs_create_u32("ipi_cpu_src", 0444, dir, &ipi_wakeup.src_cpu); - temp = debugfs_create_file("timeout_expected_ns", - 0666, - dir, - NULL, - &timeout_ops); + temp = debugfs_create_file_unsafe("timeout_expected_ns", 0666, dir, + NULL, &timeout_ops); if (!temp) { pr_alert("latency_test: failed to create /sys/kernel/debug/timeout_expected_ns\n"); return -1; ^ permalink raw reply [flat|nested] 3+ messages in thread
* [RFC v2 0/2] CPU-Idle latency selftest framework @ 2021-04-01 11:45 Pratik Rajesh Sampat 2021-04-01 11:45 ` [RFC v2 1/2] cpuidle: Extract IPI based and timer based wakeup latency from idle states Pratik Rajesh Sampat 0 siblings, 1 reply; 3+ messages in thread From: Pratik Rajesh Sampat @ 2021-04-01 11:45 UTC (permalink / raw) To: rjw, daniel.lezcano, shuah, dsmythies, ego, svaidy, linux-pm, linux-kernel, linux-kselftest, pratik.r.sampat, psampat Changelog RFC v1-->v2 The timer based test produces run to run variance on some intel based systems that sport a mechansim of "C-state pre-wake" which can pre-wake a CPU from an idle state when timers are armed. Hence invoking the timer tests is now parameterized for systems and architectures that don't support pre-wakeup logic and need granular timer measurements along with IPI results. This RFC does not yet support treating of CPU 0s idle states differently especially as reported on Intel systems. More understanding is needed on systems to determine if only CPU 0 is treated differently of if they are more CPUs that cannot have its idle state properties changed. RFC v1: https://lkml.org/lkml/2021/3/15/492 --- A kernel module + userspace driver to estimate the wakeup latency caused by going into stop states. The motivation behind this program is to find significant deviations behind advertised latency and residency values. The patchset measures latencies for two kinds of events. IPIs and Timers As this is a software-only mechanism, there will additional latencies of the kernel-firmware-hardware interactions. To account for that, the program also measures a baseline latency on a 100 percent loaded CPU and the latencies achieved must be in view relative to that. To achieve this, we introduce a kernel module and expose its control knobs through the debugfs interface that the selftests can engage with. The kernel module provides the following interfaces within /sys/kernel/debug/latency_test/ for, IPI test: ipi_cpu_dest = Destination CPU for the IPI ipi_cpu_src = Origin of the IPI ipi_latency_ns = Measured latency time in ns Timeout test: timeout_cpu_src = CPU on which the timer to be queued timeout_expected_ns = Timer duration timeout_diff_ns = Difference of actual duration vs expected timer Sample output on a POWER9 system is as follows: # --IPI Latency Test--- # Baseline Average IPI latency(ns): 3114 # Observed Average IPI latency(ns) - State0: 3265 # Observed Average IPI latency(ns) - State1: 3507 # Observed Average IPI latency(ns) - State2: 3739 # Observed Average IPI latency(ns) - State3: 3807 # Observed Average IPI latency(ns) - State4: 17070 # Observed Average IPI latency(ns) - State5: 1038174 # Observed Average IPI latency(ns) - State6: 1068784 # # --Timeout Latency Test-- # Baseline Average timeout diff(ns): 1420 # Observed Average timeout diff(ns) - State0: 1640 # Observed Average timeout diff(ns) - State1: 1764 # Observed Average timeout diff(ns) - State2: 1715 # Observed Average timeout diff(ns) - State3: 1845 # Observed Average timeout diff(ns) - State4: 16581 # Observed Average timeout diff(ns) - State5: 939977 # Observed Average timeout diff(ns) - State6: 1073024 Things to keep in mind: 1. This kernel module + bash driver does not guarantee idleness on a core when the IPI and the Timer is armed. It only invokes sleep and hopes that the core is idle once the IPI/Timer is invoked onto it. Hence this program must be run on a completely idle system for best results 2. Even on a completely idle system, there maybe book-keeping tasks or jitter tasks that can run on the core we want idle. This can create outliers in the latency measurement. Thankfully, these outliers should be large enough to easily weed them out. 3. A userspace only selftest variant was also sent out as RFC based on suggestions over the previous patchset to simply the kernel complexeity. However, a userspace only approach had more noise in the latency measurement due to userspace-kernel interactions which led to run to run variance and a lesser accurate test. Another downside of the nature of a userspace program is that it takes orders of magnitude longer to complete a full system test compared to the kernel framework. RFC patch: https://lkml.org/lkml/2020/9/2/356 4. For Intel Systems, the Timer based latencies don't exactly give out the measure of idle latencies. This is because of a hardware optimization mechanism that pre-arms a CPU when a timer is set to wakeup. That doesn't make this metric useless for Intel systems, it just means that is measuring IPI/Timer responding latency rather than idle wakeup latencies. (Source: https://lkml.org/lkml/2020/9/2/610) For solution to this problem, a hardware based latency analyzer is devised by Artem Bityutskiy from Intel. https://youtu.be/Opk92aQyvt0?t=8266 https://intel.github.io/wult/ Pratik Rajesh Sampat (2): cpuidle: Extract IPI based and timer based wakeup latency from idle states selftest/cpuidle: Add support for cpuidle latency measurement drivers/cpuidle/Makefile | 1 + drivers/cpuidle/test-cpuidle_latency.c | 157 ++++++++++ lib/Kconfig.debug | 10 + tools/testing/selftests/Makefile | 1 + tools/testing/selftests/cpuidle/Makefile | 6 + tools/testing/selftests/cpuidle/cpuidle.sh | 323 +++++++++++++++++++++ tools/testing/selftests/cpuidle/settings | 2 + 7 files changed, 500 insertions(+) create mode 100644 drivers/cpuidle/test-cpuidle_latency.c create mode 100644 tools/testing/selftests/cpuidle/Makefile create mode 100755 tools/testing/selftests/cpuidle/cpuidle.sh create mode 100644 tools/testing/selftests/cpuidle/settings -- 2.17.1 ^ permalink raw reply [flat|nested] 3+ messages in thread
* [RFC v2 1/2] cpuidle: Extract IPI based and timer based wakeup latency from idle states 2021-04-01 11:45 [RFC v2 0/2] CPU-Idle latency selftest framework Pratik Rajesh Sampat @ 2021-04-01 11:45 ` Pratik Rajesh Sampat 0 siblings, 0 replies; 3+ messages in thread From: Pratik Rajesh Sampat @ 2021-04-01 11:45 UTC (permalink / raw) To: rjw, daniel.lezcano, shuah, dsmythies, ego, svaidy, linux-pm, linux-kernel, linux-kselftest, pratik.r.sampat, psampat Introduce a mechanism to fire directed IPIs from a specified source CPU to a specified target CPU and measure the difference in time incurred on wakeup. Also, introduce a mechanism to queue a HR timer on a specified CPU and subsequently measure the time taken to wakeup the CPU. Finally define a simple debugfs interface to control the knobs to fire the IPI and Timer events on specified CPU and view their incurred idle wakeup latencies. Signed-off-by: Pratik Rajesh Sampat <psampat@linux.ibm.com> --- drivers/cpuidle/Makefile | 1 + drivers/cpuidle/test-cpuidle_latency.c | 157 +++++++++++++++++++++++++ lib/Kconfig.debug | 10 ++ 3 files changed, 168 insertions(+) create mode 100644 drivers/cpuidle/test-cpuidle_latency.c diff --git a/drivers/cpuidle/Makefile b/drivers/cpuidle/Makefile index 26bbc5e74123..3b4ee06a9164 100644 --- a/drivers/cpuidle/Makefile +++ b/drivers/cpuidle/Makefile @@ -8,6 +8,7 @@ obj-$(CONFIG_ARCH_NEEDS_CPU_IDLE_COUPLED) += coupled.o obj-$(CONFIG_DT_IDLE_STATES) += dt_idle_states.o obj-$(CONFIG_ARCH_HAS_CPU_RELAX) += poll_state.o obj-$(CONFIG_HALTPOLL_CPUIDLE) += cpuidle-haltpoll.o +obj-$(CONFIG_IDLE_LATENCY_SELFTEST) += test-cpuidle_latency.o ################################################################################## # ARM SoC drivers diff --git a/drivers/cpuidle/test-cpuidle_latency.c b/drivers/cpuidle/test-cpuidle_latency.c new file mode 100644 index 000000000000..f138011ac225 --- /dev/null +++ b/drivers/cpuidle/test-cpuidle_latency.c @@ -0,0 +1,157 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* + * Module-based API test facility for cpuidle latency using IPIs and timers + */ + +#include <linux/debugfs.h> +#include <linux/kernel.h> +#include <linux/module.h> + +/* + * IPI based wakeup latencies + * Measure time taken for a CPU to wakeup on a IPI sent from another CPU + * The latency measured also includes the latency of sending the IPI + */ +struct latency { + unsigned int src_cpu; + unsigned int dest_cpu; + ktime_t time_start; + ktime_t time_end; + u64 latency_ns; +} ipi_wakeup; + +static void measure_latency(void *info) +{ + struct latency *v; + ktime_t time_diff; + + v = (struct latency *)info; + v->time_end = ktime_get(); + time_diff = ktime_sub(v->time_end, v->time_start); + v->latency_ns = ktime_to_ns(time_diff); +} + +void run_smp_call_function_test(unsigned int cpu) +{ + ipi_wakeup.src_cpu = smp_processor_id(); + ipi_wakeup.dest_cpu = cpu; + ipi_wakeup.time_start = ktime_get(); + smp_call_function_single(cpu, measure_latency, &ipi_wakeup, 1); +} + +/* + * Timer based wakeup latencies + * Measure time taken for a CPU to wakeup on a timer being armed and fired + */ +struct timer_data { + unsigned int src_cpu; + u64 timeout; + ktime_t time_start; + ktime_t time_end; + struct hrtimer timer; + u64 timeout_diff_ns; +} timer_wakeup; + +static enum hrtimer_restart timer_called(struct hrtimer *hrtimer) +{ + struct timer_data *w; + ktime_t time_diff; + + w = container_of(hrtimer, struct timer_data, timer); + w->time_end = ktime_get(); + + time_diff = ktime_sub(w->time_end, w->time_start); + time_diff = ktime_sub(time_diff, ns_to_ktime(w->timeout)); + w->timeout_diff_ns = ktime_to_ns(time_diff); + return HRTIMER_NORESTART; +} + +static void run_timer_test(unsigned int ns) +{ + hrtimer_init(&timer_wakeup.timer, CLOCK_MONOTONIC, + HRTIMER_MODE_REL); + timer_wakeup.timer.function = timer_called; + timer_wakeup.src_cpu = smp_processor_id(); + timer_wakeup.timeout = ns; + timer_wakeup.time_start = ktime_get(); + + hrtimer_start(&timer_wakeup.timer, ns_to_ktime(ns), + HRTIMER_MODE_REL_PINNED); +} + +static struct dentry *dir; + +static int cpu_read_op(void *data, u64 *dest_cpu) +{ + *dest_cpu = ipi_wakeup.dest_cpu; + return 0; +} + +static int cpu_write_op(void *data, u64 value) +{ + run_smp_call_function_test(value); + return 0; +} +DEFINE_SIMPLE_ATTRIBUTE(ipi_ops, cpu_read_op, cpu_write_op, "%llu\n"); + +static int timeout_read_op(void *data, u64 *timeout) +{ + *timeout = timer_wakeup.timeout; + return 0; +} + +static int timeout_write_op(void *data, u64 value) +{ + run_timer_test(value); + return 0; +} +DEFINE_SIMPLE_ATTRIBUTE(timeout_ops, timeout_read_op, timeout_write_op, "%llu\n"); + +static int __init latency_init(void) +{ + struct dentry *temp; + + dir = debugfs_create_dir("latency_test", 0); + if (!dir) { + pr_alert("latency_test: failed to create /sys/kernel/debug/latency_test\n"); + return -1; + } + temp = debugfs_create_file("ipi_cpu_dest", + 0666, + dir, + NULL, + &ipi_ops); + if (!temp) { + pr_alert("latency_test: failed to create /sys/kernel/debug/ipi_cpu_dest\n"); + return -1; + } + debugfs_create_u64("ipi_latency_ns", 0444, dir, &ipi_wakeup.latency_ns); + debugfs_create_u32("ipi_cpu_src", 0444, dir, &ipi_wakeup.src_cpu); + + temp = debugfs_create_file("timeout_expected_ns", + 0666, + dir, + NULL, + &timeout_ops); + if (!temp) { + pr_alert("latency_test: failed to create /sys/kernel/debug/timeout_expected_ns\n"); + return -1; + } + debugfs_create_u64("timeout_diff_ns", 0444, dir, &timer_wakeup.timeout_diff_ns); + debugfs_create_u32("timeout_cpu_src", 0444, dir, &timer_wakeup.src_cpu); + pr_info("Latency Test module loaded\n"); + return 0; +} + +static void __exit latency_cleanup(void) +{ + pr_info("Cleaning up Latency Test module.\n"); + debugfs_remove_recursive(dir); +} + +module_init(latency_init); +module_exit(latency_cleanup); + +MODULE_LICENSE("GPL"); +MODULE_AUTHOR("IBM Corporation"); +MODULE_DESCRIPTION("Measuring idle latency for IPIs and Timers"); diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index 2779c29d9981..60fa46a99a4f 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -1513,6 +1513,16 @@ config DEBUG_KOBJECT If you say Y here, some extra kobject debugging messages will be sent to the syslog. +config IDLE_LATENCY_SELFTEST + tristate "Cpuidle latency selftests" + depends on CPU_IDLE + help + This option provides a kernel module that runs tests using the IPI and + timers to measure latency. + + Say M if you want these self tests to build as a module. + Say N if you are unsure. + config DEBUG_KOBJECT_RELEASE bool "kobject release debugging" depends on DEBUG_OBJECTS_TIMERS -- 2.17.1 ^ permalink raw reply related [flat|nested] 3+ messages in thread
end of thread, other threads:[~2021-04-02 9:03 UTC | newest] Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-04-02 9:03 [RFC v2 1/2] cpuidle: Extract IPI based and timer based wakeup latency from idle states kernel test robot 2021-04-02 9:03 ` [PATCH] cpuidle: fix debugfs_simple_attr.cocci warnings kernel test robot -- strict thread matches above, loose matches on Subject: below -- 2021-04-01 11:45 [RFC v2 0/2] CPU-Idle latency selftest framework Pratik Rajesh Sampat 2021-04-01 11:45 ` [RFC v2 1/2] cpuidle: Extract IPI based and timer based wakeup latency from idle states Pratik Rajesh Sampat
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.